Date: Sun, 12 May 2013 00:13:52 -0700 From: "Marc G. Fournier" <scrappy@hub.org> To: freebsd-fs@freebsd.org, Rick Macklem <rmacklem@uoguelph.ca> Subject: Re: NFS Performance issue against NetApp Message-ID: <518F4130.6080201@hub.org> In-Reply-To: <518EFE05.8010100@hub.org> References: <968416157.282645.1368232366317.JavaMail.root@erie.cs.uoguelph.ca> <518EFE05.8010100@hub.org>
next in thread | previous in thread | raw e-mail | index | archive | help
'k, here is on Linux ... this is right after rebooting the server, doing
a mount and running the startup once:
Client rpc stats:
calls retrans authrefrsh
40602 0 40609
Client nfs v3:
null getattr setattr lookup access readlink
0 0% 13000 32% 5 0% 6140 15% 6741 16%
0 0%
read write create mkdir symlink mknod
3556 8% 6711 16% 3743 9% 307 0% 0 0%
0 0%
remove rmdir rename link readdir readdirplus
1 0% 0 0% 0 0% 0 0% 16 0%
380 0%
fsstat fsinfo pathconf commit
0 0% 2 0% 1 0% 0 0%
One thing to note is that both Linux/FreeBSD have
"rsize=65536,wsize=65536" ... but there are 63x as many reads / 34x as
many writes on FreeBSD as on Linux ... ?
Just noticed this on the FreeBSD stats:
Rpc Info:
TimedOut Invalid X Replies Retries Requests
0 0 0 0 818479
818k Retries? Is that normal ... ?
Also, the NetApp volumes being used here are not shared ... there are no
other clients mounting these, and the Linux/FreeBSD volumes are seperate
... same size, same jboss install, same configuration, same war file ...
I could mount /vol/linux_jboss onto the FreeBSD, or /vol/freebsd_jboss
onto the Linux, and they would load the same way ... in fact, the jboss
install itself was done onto the FreeBSD and copied over to the Linux
... and both are using OpenJDK7 ... I tried to make it as identical as I
could ...
On 2013-05-11 7:27 PM, Marc G. Fournier wrote:
>
> With
>
> vfs.nfs.noconsist=3 ... 385595ms
>
> nfsstat -z before startup, nfsstat -c after:
>
> Client Info:
> Rpc Counts:
> Getattr Setattr Lookup Readlink Read Write Create
> Remove
> 332594 5 17238 0 224426 231137
> 3743 1
> Rename Link Symlink Mkdir Rmdir Readdir
> RdirPlus Access
> 0 0 0 307 0 71 0 8447
> Mknod Fsstat Fsinfo PathConf Commit
> 0 509 0 0 0
> Rpc Info:
> TimedOut Invalid X Replies Retries Requests
> 0 0 0 0 818479
> Cache Info:
> Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW
> Hits Misses
> 608296 332596 526200 17245 -95425 224426 13178
> 231137
> BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs
> Hits Misses
> 0 0 1050 55 502 7
> 543340 8448
>
>
> ============
>
> vfs.nfs.noconsist=2 ... 392201ms
>
> Client Info:
> Rpc Counts:
> Getattr Setattr Lookup Readlink Read Write Create
> Remove
> 332557 5 17228 0 224421 231131
> 3743 1
> Rename Link Symlink Mkdir Rmdir Readdir
> RdirPlus Access
> 0 0 0 307 0 72 0 8430
> Mknod Fsstat Fsinfo PathConf Commit
> 0 502 0 0 0
> Rpc Info:
> TimedOut Invalid X Replies Retries Requests
> 0 0 0 0 818395
> Cache Info:
> Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW
> Hits Misses
> 607834 332557 525801 17231 -95401 224421 13178
> 231131
> BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs
> Hits Misses
> 0 0 1028 56 502 0
> 542925 8431
>
>
> ============
> vfs.nfs.noconsist=0 ... 391622ms
>
>
> Client Info:
> Rpc Counts:
> Getattr Setattr Lookup Readlink Read Write Create
> Remove
> 236122 5 17221 0 230575 230823
> 3743 1
> Rename Link Symlink Mkdir Rmdir Readdir
> RdirPlus Access
> 0 0 0 307 0 71 0 8425
> Mknod Fsstat Fsinfo PathConf Commit
> 0 516 0 0 0
> Rpc Info:
> TimedOut Invalid X Replies Retries Requests
> 0 0 0 0 727799
> Cache Info:
> Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW
> Hits Misses
> 711860 236124 526549 17225 -101525 230490 13178
> 230823
> BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs
> Hits Misses
> 0 0 1057 55 516 0
> 543709 8425
>
>
> I checked a second time with nonconsist=0, and the nfsstat -c values
> seem to come out pretty much the same ...
>
> I'm going to head down to the office and try again with Solaris (I'd
> have to re-install, since I used that system for the Solaris), and see
> what nfsstat -c results I get out of that ... will post a followup on
> this when completed ...
>
>
>
> On 2013-05-10 5:32 PM, Rick Macklem wrote:
>> Marc G. Fournier wrote:
>>> FYI … I just installed Solaris 11 onto the same hardware and ran the
>>> same test … so far, I'm seeing:
>>>
>>> Linux @ ~30s
>>> Solaris @ ~44s
>>>
>>> OpenBSD @ ~200s
>>> FreeBSD @ ~240s
>>>
>>> I've even tried FreeBSD 8.3 just to see if maybe its as 'newish' issue
>>> … same as 9.x … I could see Linux 'cutting corners', but
>>> Oracle/Solaris too … ?
>>>
>> The three client implementations (BSD, Linux, Solaris) were developed
>> independently and, as such, will all implement somewaht different
>> caching algorithms (the RFCs specify what goes on the wire, but say
>> little w.r.t. client side caching).
>>
>> I have a attached a patch that might be useful for determining if
>> the client side buffer cache consistency algorithm in FreeBSD is
>> causing the slow startup of jboss. Do not run this patch on a
>> production system, since it pretty well disables all buffer cache
>> coherency (ie. if another client modifies a file, the patched client
>> won't notice and will continue to cache stale file data).
>>
>> If the patch does speed up startup of jboss significantly, you can
>> use the sysctl:
>> vfs.nfs.noconsist
>> to check for which coherency check is involved by decreasing the
>> value for the sysctl by 1 and then trying a startup again. (When
>> vfs.nfs.noconsist=0, normal cache coherency will be applied.)
>>
>> I have no idea if buffer cache coherency is a factor, but trying
>> the attached patch might determine if it is.
>>
>> Note that you have never posted updated "nfsstat -c" values.
>> (Remember that what you posted indicated 88 RPCs, which seemed
>> bogus.) Finding out if FreeBSD does a lot more of certain RPCs
>> that Linux/Solaris might help isolate what is going on.
>>
>> rick
>>
>>> On 2013-05-03, at 04:50 , Mark Felder <feld@feld.me> wrote:
>>>
>>>> On Thu, 02 May 2013 18:43:17 -0500, Marc G. Fournier
>>>> <scrappy@hub.org> wrote:
>>>>
>>>>> Hadn't thought to do so with Linux, but …
>>>>> Linux ……. 20732ms, 20117ms, 20935ms, 20130ms, 20560ms
>>>>> FreeBSD .. 28996ms, 24794ms, 24702ms, 23311ms, 24153ms
>>>> Please make sure both platforms are using similar atime settings. I
>>>> think most distros use ext4 with diratime by default. I'd just do
>>>> noatime on both platforms to be safe.
>>>> _______________________________________________
>>>> freebsd-fs@freebsd.org mailing list
>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>> To unsubscribe, send any mail to
>>>> "freebsd-fs-unsubscribe@freebsd.org"
>>> _______________________________________________
>>> freebsd-fs@freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
>
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?518F4130.6080201>