Date: Thu, 24 Nov 2016 11:42:41 -0700 From: Alan Somers <asomers@freebsd.org> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: Konstantin Belousov <kostikbel@gmail.com>, FreeBSD CURRENT <freebsd-current@freebsd.org> Subject: Re: NFSv4 performance degradation with 12.0-CURRENT client Message-ID: <CAOtMX2hBXAJN_udED-u5%2B6UznR2%2BW88xgb=RqKSZL65Z3%2BcKOw@mail.gmail.com> In-Reply-To: <YTXPR01MB0189E0B1DB5B16EE6B388B7DDDB60@YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM> References: <CAOtMX2jJ2XoQyVG1c04QL7NTJn1pg38s=XEgecE38ea0QoFAOw@mail.gmail.com> <20161124090811.GO54029@kib.kiev.ua> <YTXPR01MB0189E0B1DB5B16EE6B388B7DDDB60@YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Nov 24, 2016 at 5:53 AM, Rick Macklem <rmacklem@uoguelph.ca> wrote:
>
> On Wed, Nov 23, 2016 at 10:17:25PM -0700, Alan Somers wrote:
>> I have a FreeBSD 10.3-RELEASE-p12 server exporting its home
>> directories over both NFSv3 and NFSv4. I have a TrueOS client (based
>> on 12.0-CURRENT on the drm-next-4.7 branch, built on 28-October)
>> mounting the home directories over NFSv4. At first, everything is
>> fine and performance is good. But if the client does a buildworld
>> using sources on NFS and locally stored objects, performance slowly
>> degrades. The degradation is most noticeable with metadata-heavy
>> operations. For example, "ls -l" in a directory with 153 files takes
>> less than 0.1 seconds right after booting. But the longer the
>> buildworld goes on, the slower it gets. Eventually that same "ls -l"
>> takes 19 seconds. When the home directories are mounted over NFSv3
>> instead, I see no degradation.
>>
>> top shows negligible CPU consumption on the server, and very high
>> consumption on the client when using NFSv4 (nearly 100%). The
>> NFS-using process is spending almost all of its time in system mode,
>> and dtrace shows that almost all of its time is spent in
>> ncl_getpages().
>>
> A couple of things you could do when it slow (as well as what Kostik suggested):
> - nfsstat -c -e on client and nfsstat -e -s on server, to see what RPCs are being done
> and how quickly. (nfsstat -s -e will also show you how big the DRC is, although a
> large DRC should show up as increased CPU consumption on the server)
> - capture packets with tcpdump -s 0 -w test.pcap host <other-one>
> - then you can email me test.pcap as an attachment. I can look at it in wireshark
> and see if there seem to protocol and/or TCP issues. (You can look at in wireshark
> yourself, the look for NFS4ERR_xxx, TCP segment retransmits...)
>
> If you are using either "intr" or "soft" on the mounts, try without those mount options.
> (The Bugs section of mount_nfs recommends against using them. If an RPC fails due to
> these options, something called a seqid# can be "out of sync" between client/server and
> that causes serious problems.)
> --> These seqid#s are not used by NFSv4.1, so you could try that by adding
> "minorversion=1" to your mount options.
>
> Good luck with it, rick
I've reproduced the issue on stock FreeBSD 12, and I've also learned
that nullfs is a required factor. Doing the buildworld directly on
the NFS mount doesn't cause any slowdown, but doing a buildworld on
the nullfs copy of the NFS mount does. The slowdown affects the base
NFS mount as well as the nullfs copy. Here is the nfsstat output for
both server and client duing "ls -al" on the client:
nfsstat -e -s -z
Server Info:
Getattr Setattr Lookup Readlink Read Write Create Remove
800 0 121 0 0 2 0 0
Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access
0 0 0 0 0 0 0 8
Mknod Fsstat Fsinfo PathConf Commit LookupP SetClId SetClIdCf
0 0 0 0 1 3 0 0
Open OpenAttr OpenDwnGr OpenCfrm DelePurge DeleRet GetFH Lock
0 0 0 0 0 0 123 0
LockT LockU Close Verify NVerify PutFH PutPubFH PutRootFH
0 0 0 0 0 674 0 0
Renew RestoreFH SaveFH Secinfo RelLckOwn V4Create
0 0 0 0 0 0
Server:
Retfailed Faults Clients
0 0 0
OpenOwner Opens LockOwner Locks Delegs
0 0 0 0 0
Server Cache Stats:
Inprog Idem Non-idem Misses CacheSize TCPPeak
0 0 0 674 16738 16738
nfsstat -e -c -z
Client Info:
Rpc Counts:
Getattr Setattr Lookup Readlink Read Write Create Remove
60 0 119 0 0 0 0 0
Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access
0 0 0 0 0 0 0 3
Mknod Fsstat Fsinfo PathConf Commit SetClId SetClIdCf Lock
0 0 0 0 0 0 0 0
LockT LockU Open OpenCfr
0 0 0 0
OpenOwner Opens LockOwner Locks Delegs LocalOwn LocalOpen LocalLOwn
5638 141453 0 0 0 0 0 0
LocalLock
0
Rpc Info:
TimedOut Invalid X Replies Retries Requests
0 0 0 0 662
Cache Info:
Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits Misses
1275 58 837 121 0 0 0 0
BioRLHits Misses BioD Hits Misses DirE Hits Misses
1 0 6 0 1 0
And here are the most popular stack traces of "ls -al", as observed by
dtrace. The number beneath each stack is the number of times dtrace
observed that exact stack:
kernel`bcmp+0x21
kernel`vinactive+0xc6
kernel`vputx+0x30e
kernel`kern_statat+0x165
kernel`sys_fstatat+0x2c
kernel`amd64_syscall+0x314
kernel`vputx+0x30e
kernel`NDFREE+0xaa
kernel`sys___acl_get_link+0x82
kernel`amd64_syscall+0x314
kernel`0xffffffff80eb95fb
96
kernel`nfscl_doclose+0x383
kernel`vinactive+0xc6
kernel`vputx+0x30e
kernel`NDFREE+0xaa
kernel`sys___acl_get_link+0x82
kernel`amd64_syscall+0x314
kernel`0xffffffff80eb95fb
183
kernel`nfscl_doclose+0x383
kernel`vinactive+0xc6
kernel`vputx+0x30e
kernel`kern_statat+0x165
kernel`sys_fstatat+0x2c
kernel`amd64_syscall+0x314
kernel`0xffffffff80eb95fb
189
kernel`lock_delay+0x52
kernel`nfs_lookup+0x337
kernel`VOP_LOOKUP_APV+0xda
kernel`lookup+0x6a2
kernel`namei+0x57e
kernel`sys___acl_get_link+0x55
kernel`amd64_syscall+0x314
kernel`0xffffffff80eb95fb
194
kernel`lock_delay+0x52
kernel`ncl_getattrcache+0x28
kernel`nfs_getattr+0x92
kernel`VOP_GETATTR_APV+0xda
kernel`vn_stat+0xa3
kernel`kern_statat+0xde
kernel`sys_fstatat+0x2c
kernel`amd64_syscall+0x314
kernel`0xffffffff80eb95fb
196
What role could nullfs be playing?
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2hBXAJN_udED-u5%2B6UznR2%2BW88xgb=RqKSZL65Z3%2BcKOw>