Site Navigation

Date:      Tue, 06 Jan 2015 08:25:40 +0000
From:      "=?utf-8?B?TG/Dr2MgQmxvdA==?=" <loic.blot@unix-experience.fr>
To:        "Rick Macklem" <rmacklem@uoguelph.ca>
Cc:        freebsd-fs@freebsd.org
Subject:   Re: High Kernel Load with nfsv4
Message-ID:  <f8c1e0380b1c195ce3c63bc41fafb9e1@mail.unix-experience.fr>
In-Reply-To: <2093433467.6650515.1420514242874.JavaMail.root@uoguelph.ca>
References:  <2093433467.6650515.1420514242874.JavaMail.root@uoguelph.ca>

---

next in thread | previous in thread | raw e-mail | index | archive | help

---

Hi Rick,

i saw that some people has issues with igb cards with NFS
For example: http://freebsd.1045724.n5.nabble.com/NFS-over-LAGG-lacp-poor-performance-td5906349.html 

can my problem be related ? I use igb with default queue number. Here are my vmstat -i outputs

Server side:

interrupt                          total       rate
irq1: atkbd0                          18          0
irq20: ehci1                     2790134          2
irq21: ehci0                     2547642          2
cpu0:timer                      36299188         35
irq264: ciss0                    6352476          6
irq265: igb0:que 0               2716692          2
irq266: igb0:que 1              32205278         31
irq267: igb0:que 2              38395109         37
irq268: igb0:que 3               1413468          1
irq269: igb0:que 4              39207930         38
irq270: igb0:que 5               1622715          1
irq271: igb0:que 6               1634676          1
irq272: igb0:que 7               1190123          1
irq273: igb0:link                      2          0
cpu1:timer                      14074423         13
cpu8:timer                      12204739         11
cpu9:timer                      11384192         11
cpu3:timer                      10461566         10
cpu4:timer                      12785103         12
cpu6:timer                      10739344         10
cpu5:timer                      10978294         10
cpu7:timer                      10599705         10
cpu2:timer                      13998891         13
cpu10:timer                     11602361         11
cpu11:timer                     11568523         11
Total                          296772592        290

And client side:
interrupt                          total       rate
irq9: acpi0                            4          0
irq22: ehci1                      950519          2
irq23: ehci0                     1865060          4
cpu0:timer                     248128035        546
irq268: mfi0                      406896          0
irq269: igb0:que 0               2510556          5
irq270: igb0:que 1               2825336          6
irq271: igb0:que 2               2092958          4
irq272: igb0:que 3               1960849          4
irq273: igb0:que 4               2645369          5
irq274: igb0:que 5               2735187          6
irq275: igb0:que 6               2290531          5
irq276: igb0:que 7               2384370          5
irq277: igb0:link                      2          0
irq287: igb2:que 0               1465051          3
irq288: igb2:que 1                856381          1
irq289: igb2:que 2                809318          1
irq290: igb2:que 3                897154          1
irq291: igb2:que 4                875755          1
irq292: igb2:que 5              35866117         78
irq293: igb2:que 6                846517          1
irq294: igb2:que 7                857979          1
irq295: igb2:link                      2          0
irq296: igb3:que 0                535212          1
irq297: igb3:que 1                454359          1
irq298: igb3:que 2                454142          1
irq299: igb3:que 3                454623          1
irq300: igb3:que 4                456297          1
irq301: igb3:que 5                455482          1
irq302: igb3:que 6                456128          1
irq303: igb3:que 7                454680          1
irq304: igb3:link                      3          0
irq305: ahci0                         75          0
cpu1:timer                     257233702        566
cpu13:timer                    255603184        562
cpu7:timer                     258492826        569
cpu12:timer                    255819351        563
cpu6:timer                     258493465        569
cpu15:timer                    254694003        560
cpu3:timer                     258171320        568
cpu22:timer                    256506877        564
cpu5:timer                     253401435        558
cpu16:timer                    255412360        562
cpu11:timer                    257318013        566
cpu20:timer                    253648060        558
cpu2:timer                     257864543        567
cpu17:timer                    261828899        576
cpu9:timer                     257497326        567
cpu18:timer                    258451190        569
cpu8:timer                     257784504        567
cpu14:timer                    254923723        561
cpu10:timer                    257265498        566
cpu19:timer                    258775946        569
cpu4:timer                     256368658        564
cpu23:timer                    255050534        561
cpu21:timer                    257663842        567
Total                         6225260206      13710

Please note igb2 on client side is the dedicated link for NFSv4

Regards,

Loïc Blot,
UNIX Systems, Network and Security Engineer
http://www.unix-experience.fr

6 janvier 2015 04:17 "Rick Macklem" <rmacklem@uoguelph.ca> a écrit: 
> Loic Blot wrote:
> 
>> Hi Rick,
>> nfsstat -e -s don't show usefull datas on server.
> 
> Well, as far as I know, it returns valid information.
> (See below.)
> 
>> Server Info:
>> Getattr Setattr Lookup Readlink Read Write Create
>> Remove
>> 26935254 16911 5755728 302 2334920 3673866 0
>> 328332
>> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus
>> Access
>> 77980 28 0 0 3 8900 3
>> 1806052
>> Mknod Fsstat Fsinfo PathConf Commit LookupP SetClId
>> SetClIdCf
>> 1 1095 0 0 614377 8172 8
>> 8
>> Open OpenAttr OpenDwnGr OpenCfrm DelePurge DeleRet GetFH
>> Lock
>> 1595299 0 44145 1495 0 0 5197490
>> 635015
>> LockT LockU Close Verify NVerify PutFH PutPubFH
>> PutRootFH
>> 0 614919 1270938 0 0 22688676 0
>> 5
>> Renew RestoreFH SaveFH Secinfo RelLckOwn V4Create
>> 42104 197606 275820 0 143 4578
>> Server:
>> Retfailed Faults Clients
>> 0 0 6
>> OpenOwner Opens LockOwner Locks Delegs
>> 32335 145448 204 181 0
> 
> Well, 145448 Opens are a lot of Open files. Each of these uses
> a kernel malloc'd data structure that is linked into multiple
> linked lists.
> 
> The question is..why aren't these Opens being closed?
> Since FreeBSD does I/O on an mmap'd file after closing it,
> the FreeBSD NFSv4 client is forced to delay doing Close RPCs
> until the vnode is VOP_INACTIVE()/VOP_RECLAIM()'d. (The
> VOP_RECLAIM() case is needed, since VOP_INACTIVE() isn't
> guaranteed to be called.)
> 
> Since there were about 1.5 million Opens and 1.27 million
> Closes, it does appear that Opens are being Closed.
> Now, I'm not sure I would have imagined 1.5million file Opens
> in a few days. My guess is this is the bottleneck.
> 
> I'd suggest that you do:
> # nfsstat -e -c
> on each of the NFSv4 clients and see how many Opens/client
> there are. I vaguely remember an upper limit in the client,
> but can't remember what it is set to.
> --> I suspect the client Open/Lock limit needs to be increased.
> (I can't remember if the server also has a limit, but I
> think it does.)
> Then the size of the hash tables used to search the Opens
> may also need to be increased a lot.
> 
> Also, I'd suggest you take a look at whatever apps. are
> running on the client(s) and try to figure out why they
> are Opening so many files?
> 
> My guess is that the client(s) are gettig bogged down by all
> these Opens.
> 
>> Server Cache Stats:
>> Inprog Idem Non-idem Misses CacheSize TCPPeak
>> 0 0 1 15082947 60 16522
>> 
>> Only GetAttr and Lookup increase and it's only every 4-5 seconds and
>> only +2 to +5 into theses values.
>> 
>> Now on client, if i take four processes stack i got
>> 
>> PID TID COMM TDNAME KSTACK
>> 63170 102547 mv - mi_switch+0xe1
>> turnstile_wait+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65
>> nfs_lookup+0x3d0 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4
>> vn_open_cred+0x21d kern_openat+0x26f amd64_syscall+0x351
>> Xfast_syscall+0xfb
>> 
>> Another mv:
>> 63140 101738 mv - mi_switch+0xe1
>> turnstile_wait+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65
>> nfs_lookup+0x3d0 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4
>> kern_statat_vnhook+0xae sys_lstat+0x30 amd64_syscall+0x351
>> Xfast_syscall+0xfb
>> 
>> 62070 102170 sendmail - mi_switch+0xe1
>> sleepq_timedwait+0x3a _sleep+0x26e clnt_vc_call+0x666
>> clnt_reconnect_call+0x4fa newnfs_request+0xa8c nfscl_request+0x72
>> nfsrpc_lookup+0x1fb nfs_lookup+0x508 VOP_LOOKUP_APV+0xa1
>> lookup+0x59c namei+0x4d4 kern_statat_vnhook+0xae sys_lstat+0x30
>> amd64_syscall+0x351 Xfast_syscall+0xfb
>> 
>> 63200 100930 mv - mi_switch+0xe1
>> turnstile_wait+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65
>> nfs_lookup+0x3d0 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4
>> kern_statat_vnhook+0xae sys_lstat+0x30 amd64_syscall+0x351
>> Xfast_syscall+0xfb
> 
> The above simply says that thread 102710 is waiting for a Lookup
> reply from the server and the other 3 are waiting for the mutex
> that protects the state structures in the client. (I suspect
> some other thread in the client is wading through the Open list,
> if a single client has a lot of these 145K Opens.)
> 
>> When client is in this state, server was doing nothing special
>> (procstat -kk)
>> 
>> PID TID COMM TDNAME KSTACK
>> 895 100538 nfsd nfsd: master mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10
>> _cv_timedwait_sig_sbt+0x18b svc_run_internal+0x4a1 svc_run+0x1de
>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c
>> amd64_syscall+0x351 Xfast_syscall+0xfb
>> 895 100568 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100569 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100570 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100571 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100572 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100573 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100575 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100576 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100577 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100578 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100579 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100580 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100581 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100582 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100583 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100584 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100585 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100586 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100587 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100588 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100589 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100590 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100592 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100593 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100594 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100595 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100596 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100597 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100598 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100599 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100600 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100602 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100603 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100604 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100605 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100606 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100607 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100608 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100609 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100610 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100611 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100612 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100613 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100614 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100615 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100617 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100618 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100619 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100621 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100622 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100623 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100624 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100625 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100626 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100627 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100628 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100629 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100630 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100631 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100632 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100633 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100634 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100635 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100636 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100638 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100639 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100640 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100641 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100642 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100643 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100644 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100645 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100646 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100647 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100648 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100649 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100651 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100652 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100653 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100654 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100655 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100656 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100657 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100658 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100659 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100661 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100662 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100684 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100685 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100686 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100797 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100798 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100799 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100800 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 895 100801 nfsd nfsd: service mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 
>> I really think it's a client side problem, maybe a lookup problem.
>> 
>> Regards,
>> 
>> Loïc Blot,
>> UNIX Systems, Network and Security Engineer
>> http://www.unix-experience.fr
>> 
>> 5 janvier 2015 14:35 "Rick Macklem" <rmacklem@uoguelph.ca> a écrit:
>>> Loic Blot wrote:
>>> 
>>>> Hi,
>>>> happy new year Rick and @freebsd-fs.
>>>> 
>>>> After some days, i looked my NFSv4.1 mount. At server start it was
>>>> calm, but after 4 days, here is the top stat...
>>>> 
>>>> CPU: 0.0% user, 0.0% nice, 100% system, 0.0% interrupt, 0.0%
>>>> idle
>>>> 
>>>> Definitively i think it's a problem on client side. What can i
>>>> look
>>>> into running kernel to resolve this issue ?
>>> 
>>> Well, I'd start with:
>>> # nfsstat -e -s
>>> - run repeatedly on the server (once every N seconds in a loop).
>>> Then look at the output, comparing the counts and see which RPCs
>>> are being performed by the client(s). You are looking for which
>>> RPCs are being done a lot. (If one RPC is almost 100% of the load,
>>> then it might be a client/caching issue for whatever that RPC is
>>> doing.)
>>> 
>>> Also look at the Open/Lock counts near the end of the output.
>>> If the # of Opens/Locks is large, it may be possible to reduce the
>>> CPU overheads by using larger hash tables.
>>> 
>>> Then you need to profile the server kernel to see where the CPU
>>> is being used.
>>> Hopefully someone else can fill you in on how to do that, because
>>> I'll admit I don't know how to.
>>> Basically you are looking to see if the CPU is being used in
>>> the NFS server code or ZFS.
>>> 
>>> Good luck with it, rick
>>> 
>>>> Regards,
>>>> 
>>>> Loïc Blot,
>>>> UNIX Systems, Network and Security Engineer
>>>> http://www.unix-experience.fr
>>>> 
>>>> 30 décembre 2014 16:16 "Loïc Blot" <loic.blot@unix-experience.fr>
>>>> a
>>>> écrit:
>>>>> Hi Rick,
>>>>> i upgraded my jail host from FreeBSD 9.3 to 10.1 to use NFS v4.1
>>>>> (mountoptions:
>>>>> rw,rsize=32768,wsize=32768,tcp,nfsv4,minorversion=1)
>>>>> 
>>>>> Performance is quite stable but it's slow. Not as slow as before
>>>>> but slow... services was launched
>>>>> but no client are using them and system CPU % was 10-50%.
>>>>> 
>>>>> I don't see anything on NFSv4.1 server, it's perfectly stable and
>>>>> functionnal.
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> Loïc Blot,
>>>>> UNIX Systems, Network and Security Engineer
>>>>> http://www.unix-experience.fr
>>>>> 
>>>>> 23 décembre 2014 00:20 "Rick Macklem" <rmacklem@uoguelph.ca> a
>>>>> écrit:
>>>>> 
>>>>>> Loic Blot wrote:
>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> To clarify because of our exchanges. Here are the current
>>>>>>> sysctl
>>>>>>> options for server:
>>>>>>> 
>>>>>>> vfs.nfsd.enable_nobodycheck=0
>>>>>>> vfs.nfsd.enable_nogroupcheck=0
>>>>>>> 
>>>>>>> vfs.nfsd.maxthreads=200
>>>>>>> vfs.nfsd.tcphighwater=10000
>>>>>>> vfs.nfsd.tcpcachetimeo=300
>>>>>>> vfs.nfsd.server_min_nfsvers=4
>>>>>>> 
>>>>>>> kern.maxvnodes=10000000
>>>>>>> kern.ipc.maxsockbuf=4194304
>>>>>>> net.inet.tcp.sendbuf_max=4194304
>>>>>>> net.inet.tcp.recvbuf_max=4194304
>>>>>>> 
>>>>>>> vfs.lookup_shared=0
>>>>>>> 
>>>>>>> Regards,
>>>>>>> 
>>>>>>> Loïc Blot,
>>>>>>> UNIX Systems, Network and Security Engineer
>>>>>>> http://www.unix-experience.fr
>>>>>>> 
>>>>>>> 22 décembre 2014 09:42 "Loïc Blot"
>>>>>>> <loic.blot@unix-experience.fr>
>>>>>>> a
>>>>>>> écrit:
>>>>>>> 
>>>>>>> Hi Rick,
>>>>>>> my 5 jails runs this weekend and now i have some stats on this
>>>>>>> monday.
>>>>>>> 
>>>>>>> Hopefully deadlock was fixed, yeah, but everything isn't good
>>>>>>> :(
>>>>>>> 
>>>>>>> On NFSv4 server (FreeBSD 10.1) system uses 35% CPU
>>>>>>> 
>>>>>>> As i can see this is because of nfsd:
>>>>>>> 
>>>>>>> 918 root 96 20 0 12352K 3372K rpcsvc 6 51.4H
>>>>>>> 273.68% nfsd: server (nfsd)
>>>>>>> 
>>>>>>> If i look at dmesg i see:
>>>>>>> nfsd server cache flooded, try increasing vfs.nfsd.tcphighwater
>>>>>> 
>>>>>> Well, you have a couple of choices:
>>>>>> 1 - Use NFSv4.1 (add "minorversion=1" to your mount options).
>>>>>> (NFSv4.1 avoids use of the DRC and instead uses something
>>>>>> called sessions. See below.)
>>>>>> OR
>>>>>> 
>>>>>>> vfs.nfsd.tcphighwater was set to 10000, i increase it to 15000
>>>>>> 
>>>>>> 2 - Bump vfs.nfsd.tcphighwater way up, until you no longer see
>>>>>> "nfs server cache flooded" messages. (I think Garrett Wollman
>>>>>> uses
>>>>>> 100000. (You may still see quite a bit of CPU overheads.)
>>>>>> 
>>>>>> OR
>>>>>> 
>>>>>> 3 - Set vfs.nfsd.cachetcp=0 (which disables the DRC and gets rid
>>>>>> of the CPU overheads). However, there is a risk of data
>>>>>> corruption
>>>>>> if you have a client->server network partitioning of a moderate
>>>>>> duration, because a non-idempotent RPC may get redone, becasue
>>>>>> the client times out waiting for a reply. If a non-idempotent
>>>>>> RPC gets done twice on the server, data corruption can happen.
>>>>>> (The DRC provides improved correctness, but does add overhead.)
>>>>>> 
>>>>>> If #1 works for you, it is the preferred solution, since
>>>>>> Sessions
>>>>>> in NFSv4.1 solves the correctness problem in a good, space bound
>>>>>> way. A session basically has N (usually 32 or 64) slots and only
>>>>>> allows one outstanding RPC/slot. As such, it can cache the
>>>>>> previous
>>>>>> reply for each slot (32 or 64 of them) and guarantee "exactly
>>>>>> once"
>>>>>> RPC semantics.
>>>>>> 
>>>>>> rick
>>>>>> 
>>>>>>> Here is 'nfsstat -s' output:
>>>>>>> 
>>>>>>> Server Info:
>>>>>>> Getattr Setattr Lookup Readlink Read Write Create
>>>>>>> Remove
>>>>>>> 12600652 1812 2501097 156 1386423 1983729 123
>>>>>>> 162067
>>>>>>> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus
>>>>>>> Access
>>>>>>> 36762 9 0 0 0 3147 0
>>>>>>> 623524
>>>>>>> Mknod Fsstat Fsinfo PathConf Commit
>>>>>>> 0 0 0 0 328117
>>>>>>> Server Ret-Failed
>>>>>>> 0
>>>>>>> Server Faults
>>>>>>> 0
>>>>>>> Server Cache Stats:
>>>>>>> Inprog Idem Non-idem Misses
>>>>>>> 0 0 0 12635512
>>>>>>> Server Write Gathering:
>>>>>>> WriteOps WriteRPC Opsaved
>>>>>>> 1983729 1983729 0
>>>>>>> 
>>>>>>> And here is 'procstat -kk' for nfsd (server)
>>>>>>> 
>>>>>>> 918 100528 nfsd nfsd: master mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10
>>>>>>> _cv_timedwait_sig_sbt+0x18b svc_run_internal+0x4a1
>>>>>>> svc_run+0x1de
>>>>>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c
>>>>>>> amd64_syscall+0x351 Xfast_syscall+0xfb
>>>>>>> 918 100568 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100569 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100570 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100571 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100572 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100573 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100574 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100575 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100576 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100577 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100578 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100579 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100580 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100581 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100582 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100583 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100584 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100585 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100586 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100587 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100588 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100589 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100590 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100591 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100592 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100593 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100594 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100595 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100596 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100597 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100598 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100599 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100600 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100601 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100602 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100603 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100604 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100605 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100606 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100607 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100608 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100609 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100610 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100611 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100612 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100613 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100614 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100615 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100616 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100617 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100618 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100619 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100620 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100621 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100622 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100623 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100624 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100625 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100626 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100627 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100628 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100629 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100630 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100631 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100632 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100633 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100634 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100635 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100636 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100637 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100638 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100639 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100640 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100641 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100642 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100643 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100644 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100645 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100646 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100647 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100648 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100649 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100650 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100651 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100652 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100653 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100654 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100655 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100656 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100657 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100658 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100659 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100660 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100661 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100662 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> ---
>>>>>>> 
>>>>>>> Now if we look at client (FreeBSD 9.3)
>>>>>>> 
>>>>>>> We see system was very busy and do many and many interrupts
>>>>>>> 
>>>>>>> CPU: 0.0% user, 0.0% nice, 37.8% system, 51.2% interrupt, 11.0%
>>>>>>> idle
>>>>>>> 
>>>>>>> A look at process list shows that there are many sendmail
>>>>>>> process
>>>>>>> in
>>>>>>> state nfstry
>>>>>>> 
>>>>>>> nfstry 18 32:27 0.88% sendmail: Queue runner@00:30:00 for
>>>>>>> /var/spool/clientm
>>>>>>> 
>>>>>>> Here is 'nfsstat -c' output:
>>>>>>> 
>>>>>>> Client Info:
>>>>>>> Rpc Counts:
>>>>>>> Getattr Setattr Lookup Readlink Read Write Create
>>>>>>> Remove
>>>>>>> 1051347 1724 2494481 118 903902 1901285 162676
>>>>>>> 161899
>>>>>>> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus
>>>>>>> Access
>>>>>>> 36744 2 0 114 40 3131 0
>>>>>>> 544136
>>>>>>> Mknod Fsstat Fsinfo PathConf Commit
>>>>>>> 9 0 0 0 245821
>>>>>>> Rpc Info:
>>>>>>> TimedOut Invalid X Replies Retries Requests
>>>>>>> 0 0 0 0 8356557
>>>>>>> Cache Info:
>>>>>>> Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits
>>>>>>> Misses
>>>>>>> 108754455 491475 54229224 2437229 46814561 821723 5132123
>>>>>>> 1871871
>>>>>>> BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits
>>>>>>> Misses
>>>>>>> 144035 118 53736 2753 27813 1 57238839
>>>>>>> 544205
>>>>>>> 
>>>>>>> If you need more things, tell me, i let the PoC in this state.
>>>>>>> 
>>>>>>> Thanks
>>>>>>> 
>>>>>>> Regards,
>>>>>>> 
>>>>>>> Loïc Blot,
>>>>>>> UNIX Systems, Network and Security Engineer
>>>>>>> http://www.unix-experience.fr
>>>>>>> 
>>>>>>> 21 décembre 2014 01:33 "Rick Macklem" <rmacklem@uoguelph.ca> a
>>>>>>> écrit:
>>>>>>> 
>>>>>>> Loic Blot wrote:
>>>>>>> 
>>>>>>>> Hi Rick,
>>>>>>>> ok, i don't need locallocks, i haven't understand option was
>>>>>>>> for
>>>>>>>> that
>>>>>>>> usage, i removed it.
>>>>>>>> I do more tests on monday.
>>>>>>>> Thanks for the deadlock fix, for other people :)
>>>>>>> 
>>>>>>> Good. Please let us know if running with
>>>>>>> vfs.nfsd.enable_locallocks=0
>>>>>>> gets rid of the deadlocks? (I think it fixes the one you saw.)
>>>>>>> 
>>>>>>> On the performance side, you might also want to try different
>>>>>>> values
>>>>>>> of
>>>>>>> readahead, if the Linux client has such a mount option. (With
>>>>>>> the
>>>>>>> NFSv4-ZFS sequential vs random I/O heuristic, I have no idea
>>>>>>> what
>>>>>>> the
>>>>>>> optimal readahead value would be.)
>>>>>>> 
>>>>>>> Good luck with it and please let us know how it goes, rick
>>>>>>> ps: I now have a patch to fix the deadlock when
>>>>>>> vfs.nfsd.enable_locallocks=1
>>>>>>> is set. I'll post it for anyone who is interested after I put
>>>>>>> it
>>>>>>> through some testing.
>>>>>>> 
>>>>>>> --
>>>>>>> Best regards,
>>>>>>> Loïc BLOT,
>>>>>>> UNIX systems, security and network engineer
>>>>>>> http://www.unix-experience.fr
>>>>>>> 
>>>>>>> Le jeudi 18 décembre 2014 à 19:46 -0500, Rick Macklem a écrit :
>>>>>>> 
>>>>>>> Loic Blot wrote:
>>>>>>>> Hi rick,
>>>>>>>> i tried to start a LXC container on Debian Squeeze from my
>>>>>>>> freebsd
>>>>>>>> ZFS+NFSv4 server and i also have a deadlock on nfsd
>>>>>>>> (vfs.lookup_shared=0). Deadlock procs each time i launch a
>>>>>>>> squeeze
>>>>>>>> container, it seems (3 tries, 3 fails).
>>>>>>> 
>>>>>>> Well, I`ll take a look at this `procstat -kk`, but the only
>>>>>>> thing
>>>>>>> I`ve seen posted w.r.t. avoiding deadlocks in ZFS is to not use
>>>>>>> nullfs. (I have no idea if you are using any nullfs mounts, but
>>>>>>> if so, try getting rid of them.)
>>>>>>> 
>>>>>>> Here`s a high level post about the ZFS and vnode locking
>>>>>>> problem,
>>>>>>> but there is no patch available, as far as I know.
>>>>>>> 
>>>>>>> http://docs.FreeBSD.org/cgi/mid.cgi?54739F41.8030407
>>>>>>> 
>>>>>>> rick
>>>>>>> 
>>>>>>> 921 - D 0:00.02 nfsd: server (nfsd)
>>>>>>> 
>>>>>>> Here is the procstat -kk
>>>>>>> 
>>>>>>> PID TID COMM TDNAME KSTACK
>>>>>>> 921 100538 nfsd nfsd: master mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad
>>>>>>> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554
>>>>>>> svc_run_internal+0xc77 svc_run+0x1de nfsrvd_nfsd+0x1ca
>>>>>>> nfssvc_nfsd+0x107 sys_nfssvc+0x9c
>>>>>>> 921 100572 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100573 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100574 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100575 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100576 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100577 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100578 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100579 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100580 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100581 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100582 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100583 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100584 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100585 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100586 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100587 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100588 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100589 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100590 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100591 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100592 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100593 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100594 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100595 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100596 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100597 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100598 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100599 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100600 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100601 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100602 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100603 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100604 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100605 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100606 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100607 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100608 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100609 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100610 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100611 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100612 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100613 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100614 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100615 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100616 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1
>>>>>>> nfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 921 100617 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100618 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 921 100619 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100620 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100621 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100622 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100623 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100624 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100625 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100626 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100627 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100628 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100629 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100630 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100631 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100632 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100633 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100634 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100635 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100636 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100637 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100638 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100639 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100640 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100641 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100642 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100643 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100644 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100645 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100646 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100647 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100648 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100649 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100650 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100651 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100652 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100653 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100654 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100655 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100656 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100657 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100658 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100659 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100660 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100661 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100662 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100663 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100664 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100665 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>> _cv_wait_sig+0x16a
>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 921 100666 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8
>>>>>>> nfsrvd_dorpc+0xc76
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 
>>>>>>> Regards,
>>>>>>> 
>>>>>>> Loïc Blot,
>>>>>>> UNIX Systems, Network and Security Engineer
>>>>>>> http://www.unix-experience.fr
>>>>>>> 
>>>>>>> 15 décembre 2014 15:18 "Rick Macklem" <rmacklem@uoguelph.ca> a
>>>>>>> écrit:
>>>>>>> 
>>>>>>> Loic Blot wrote:
>>>>>>> 
>>>>>>>> For more informations, here is procstat -kk on nfsd, if you
>>>>>>>> need
>>>>>>>> more
>>>>>>>> hot datas, tell me.
>>>>>>>> 
>>>>>>>> Regards, PID TID COMM TDNAME KSTACK
>>>>>>>> 918 100529 nfsd nfsd: master mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_run+0x1de
>>>>>>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c
>>>>>>>> amd64_syscall+0x351
>>>>>>> 
>>>>>>> Well, most of the threads are stuck like this one, waiting for
>>>>>>> a
>>>>>>> vnode
>>>>>>> lock in ZFS. All of them appear to be in zfs_fhtovp().
>>>>>>> I`m not a ZFS guy, so I can`t help much. I`ll try changing the
>>>>>>> subject line
>>>>>>> to include ZFS vnode lock, so maybe the ZFS guys will take a
>>>>>>> look.
>>>>>>> 
>>>>>>> The only thing I`ve seen suggested is trying:
>>>>>>> sysctl vfs.lookup_shared=0
>>>>>>> to disable shared vop_lookup()s. Apparently zfs_lookup()
>>>>>>> doesn`t
>>>>>>> obey the vnode locking rules for lookup and rename, according
>>>>>>> to
>>>>>>> the posting I saw.
>>>>>>> 
>>>>>>> I`ve added a couple of comments about the other threads below,
>>>>>>> but
>>>>>>> they are all either waiting for an RPC request or waiting for
>>>>>>> the
>>>>>>> threads stuck on the ZFS vnode lock to complete.
>>>>>>> 
>>>>>>> rick
>>>>>>> 
>>>>>>>> 918 100564 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>> fork_trampoline+0xe
>>>>>>> 
>>>>>>> Fyi, this thread is just waiting for an RPC to arrive. (Normal)
>>>>>>> 
>>>>>>>> 918 100565 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>> fork_trampoline+0xe
>>>>>>>> 918 100566 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>> fork_trampoline+0xe
>>>>>>>> 918 100567 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>> fork_trampoline+0xe
>>>>>>>> 918 100568 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>> fork_trampoline+0xe
>>>>>>>> 918 100569 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>> fork_trampoline+0xe
>>>>>>>> 918 100570 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>> fork_trampoline+0xe
>>>>>>>> 918 100571 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100572 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>>> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8
>>>>>>>> nfsrvd_dorpc+0xc76
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 
>>>>>>> This one (and a few others) are waiting for the nfsv4_lock.
>>>>>>> This
>>>>>>> happens
>>>>>>> because other threads are stuck with RPCs in progress. (ie. The
>>>>>>> ones
>>>>>>> waiting on the vnode lock in zfs_fhtovp().)
>>>>>>> For these, the RPC needs to lock out other threads to do the
>>>>>>> operation,
>>>>>>> so it waits for the nfsv4_lock() which can exclusively lock the
>>>>>>> NFSv4
>>>>>>> data structures once all other nfsd threads complete their RPCs
>>>>>>> in
>>>>>>> progress.
>>>>>>> 
>>>>>>>> 918 100573 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 
>>>>>>> Same as above.
>>>>>>> 
>>>>>>>> 918 100574 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100575 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100576 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100577 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100578 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100579 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100580 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100581 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100582 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100583 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100584 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100585 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100586 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100587 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100588 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100589 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100590 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100591 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100592 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100593 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100594 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100595 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100596 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100597 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100598 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100599 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100600 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100601 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100602 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100603 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100604 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100605 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100606 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>> 918 100607 nfsd nfsd: service mi_switch+0xe1
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>> svc_thread_start+0xb
>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 
>>>>>>> Lots more waiting for the ZFS vnode lock in zfs_fhtovp().
>>>>>>> 
>>>>>>> 918 100608 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1
>>>>>>> nfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100609 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100610 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad
>>>>>>> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554
>>>>>>> svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a
>>>>>>> fork_trampoline+0xe
>>>>>>> 918 100611 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100612 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100613 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100614 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100615 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100616 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100617 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100618 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100619 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100620 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100621 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100622 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100623 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100624 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100625 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100626 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100627 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100628 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100629 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100630 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100631 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100632 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100633 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100634 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100635 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100636 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100637 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100638 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100639 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100640 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100641 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100642 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100643 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100644 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100645 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100646 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100647 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100648 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100649 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100650 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100651 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100652 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100653 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100654 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100655 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100656 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100657 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 918 100658 nfsd nfsd: service mi_switch+0xe1
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>> zfs_fhtovp+0x38d
>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>> svc_thread_start+0xb
>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>> 
>>>>>>> Loïc Blot,
>>>>>>> UNIX Systems, Network and Security Engineer
>>>>>>> http://www.unix-experience.fr
>>>>>>> 
>>>>>>> 15 décembre 2014 13:29 "Loïc Blot"
>>>>>>> <loic.blot@unix-experience.fr>
>>>>>>> a
>>>>>>> écrit:
>>>>>>> 
>>>>>>> Hmmm...
>>>>>>> now i'm experiencing a deadlock.
>>>>>>> 
>>>>>>> 0 918 915 0 21 0 12352 3372 zfs D - 1:48.64 nfsd: server
>>>>>>> (nfsd)
>>>>>>> 
>>>>>>> the only issue was to reboot the server, but after rebooting
>>>>>>> deadlock arrives a second time when i
>>>>>>> start my jails over NFS.
>>>>>>> 
>>>>>>> Regards,
>>>>>>> 
>>>>>>> Loïc Blot,
>>>>>>> UNIX Systems, Network and Security Engineer
>>>>>>> http://www.unix-experience.fr
>>>>>>> 
>>>>>>> 15 décembre 2014 10:07 "Loïc Blot"
>>>>>>> <loic.blot@unix-experience.fr>
>>>>>>> a
>>>>>>> écrit:
>>>>>>> 
>>>>>>> Hi Rick,
>>>>>>> after talking with my N+1, NFSv4 is required on our
>>>>>>> infrastructure.
>>>>>>> I tried to upgrade NFSv4+ZFS
>>>>>>> server from 9.3 to 10.1, i hope this will resolve some
>>>>>>> issues...
>>>>>>> 
>>>>>>> Regards,
>>>>>>> 
>>>>>>> Loïc Blot,
>>>>>>> UNIX Systems, Network and Security Engineer
>>>>>>> http://www.unix-experience.fr
>>>>>>> 
>>>>>>> 10 décembre 2014 15:36 "Loïc Blot"
>>>>>>> <loic.blot@unix-experience.fr>
>>>>>>> a
>>>>>>> écrit:
>>>>>>> 
>>>>>>> Hi Rick,
>>>>>>> thanks for your suggestion.
>>>>>>> For my locking bug, rpc.lockd is stucked in rpcrecv state on
>>>>>>> the
>>>>>>> server. kill -9 doesn't affect the
>>>>>>> process, it's blocked.... (State: Ds)
>>>>>>> 
>>>>>>> for the performances
>>>>>>> 
>>>>>>> NFSv3: 60Mbps
>>>>>>> NFSv4: 45Mbps
>>>>>>> Regards,
>>>>>>> 
>>>>>>> Loïc Blot,
>>>>>>> UNIX Systems, Network and Security Engineer
>>>>>>> http://www.unix-experience.fr
>>>>>>> 
>>>>>>> 10 décembre 2014 13:56 "Rick Macklem" <rmacklem@uoguelph.ca>
>>>>>>> a
>>>>>>> écrit:
>>>>>>> 
>>>>>>> Loic Blot wrote:
>>>>>>> 
>>>>>>>> Hi Rick,
>>>>>>>> I'm trying NFSv3.
>>>>>>>> Some jails are starting very well but now i have an issue
>>>>>>>> with
>>>>>>>> lockd
>>>>>>>> after some minutes:
>>>>>>>> 
>>>>>>>> nfs server 10.10.X.8:/jails: lockd not responding
>>>>>>>> nfs server 10.10.X.8:/jails lockd is alive again
>>>>>>>> 
>>>>>>>> I look at mbuf, but i seems there is no problem.
>>>>>>> 
>>>>>>> Well, if you need locks to be visible across multiple
>>>>>>> clients,
>>>>>>> then
>>>>>>> I'm afraid you are stuck with using NFSv4 and the
>>>>>>> performance
>>>>>>> you
>>>>>>> get
>>>>>>> from it. (There is no way to do file handle affinity for
>>>>>>> NFSv4
>>>>>>> because
>>>>>>> the read and write ops are buried in the compound RPC and
>>>>>>> not
>>>>>>> easily
>>>>>>> recognized.)
>>>>>>> 
>>>>>>> If the locks don't need to be visible across multiple
>>>>>>> clients,
>>>>>>> I'd
>>>>>>> suggest trying the "nolockd" option with nfsv3.
>>>>>>> 
>>>>>>>> Here is my rc.conf on server:
>>>>>>>> 
>>>>>>>> nfs_server_enable="YES"
>>>>>>>> nfsv4_server_enable="YES"
>>>>>>>> nfsuserd_enable="YES"
>>>>>>>> nfsd_server_flags="-u -t -n 256"
>>>>>>>> mountd_enable="YES"
>>>>>>>> mountd_flags="-r"
>>>>>>>> nfsuserd_flags="-usertimeout 0 -force 20"
>>>>>>>> rpcbind_enable="YES"
>>>>>>>> rpc_lockd_enable="YES"
>>>>>>>> rpc_statd_enable="YES"
>>>>>>>> 
>>>>>>>> Here is the client:
>>>>>>>> 
>>>>>>>> nfsuserd_enable="YES"
>>>>>>>> nfsuserd_flags="-usertimeout 0 -force 20"
>>>>>>>> nfscbd_enable="YES"
>>>>>>>> rpc_lockd_enable="YES"
>>>>>>>> rpc_statd_enable="YES"
>>>>>>>> 
>>>>>>>> Have you got an idea ?
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> 
>>>>>>>> Loïc Blot,
>>>>>>>> UNIX Systems, Network and Security Engineer
>>>>>>>> http://www.unix-experience.fr
>>>>>>>> 
>>>>>>>> 9 décembre 2014 04:31 "Rick Macklem" <rmacklem@uoguelph.ca>
>>>>>>>> a
>>>>>>>> écrit:
>>>>>>>>> Loic Blot wrote:
>>>>>>>>> 
>>>>>>>>>> Hi rick,
>>>>>>>>>> 
>>>>>>>>>> I waited 3 hours (no lag at jail launch) and now I do:
>>>>>>>>>> sysrc
>>>>>>>>>> memcached_flags="-v -m 512"
>>>>>>>>>> Command was very very slow...
>>>>>>>>>> 
>>>>>>>>>> Here is a dd over NFS:
>>>>>>>>>> 
>>>>>>>>>> 601062912 bytes transferred in 21.060679 secs (28539579
>>>>>>>>>> bytes/sec)
>>>>>>>>> 
>>>>>>>>> Can you try the same read using an NFSv3 mount?
>>>>>>>>> (If it runs much faster, you have probably been bitten by
>>>>>>>>> the
>>>>>>>>> ZFS
>>>>>>>>> "sequential vs random" read heuristic which I've been told
>>>>>>>>> things
>>>>>>>>> NFS is doing "random" reads without file handle affinity.
>>>>>>>>> File
>>>>>>>>> handle affinity is very hard to do for NFSv4, so it isn't
>>>>>>>>> done.)
>>>>>>> 
>>>>>>> I was actually suggesting that you try the "dd" over nfsv3
>>>>>>> to
>>>>>>> see
>>>>>>> how
>>>>>>> the performance compared with nfsv4. If you do that, please
>>>>>>> post
>>>>>>> the
>>>>>>> comparable results.
>>>>>>> 
>>>>>>> Someday I would like to try and get ZFS's sequential vs
>>>>>>> random
>>>>>>> read
>>>>>>> heuristic modified and any info on what difference in
>>>>>>> performance
>>>>>>> that
>>>>>>> might make for NFS would be useful.
>>>>>>> 
>>>>>>> rick
>>>>>>> 
>>>>>>> rick
>>>>>>> 
>>>>>>> This is quite slow...
>>>>>>> 
>>>>>>> You can found some nfsstat below (command isn't finished
>>>>>>> yet)
>>>>>>> 
>>>>>>> nfsstat -c -w 1
>>>>>>> 
>>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 4 0 0 0 0 0 16 0
>>>>>>> 2 0 0 0 0 0 17 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 4 0 0 0 0 4 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 4 0 0 0 0 0 3 0
>>>>>>> 0 0 0 0 0 0 3 0
>>>>>>> 37 10 0 8 0 0 14 1
>>>>>>> 18 16 0 4 1 2 4 0
>>>>>>> 78 91 0 82 6 12 30 0
>>>>>>> 19 18 0 2 2 4 2 0
>>>>>>> 0 0 0 0 2 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 1 0 0 0 0 1 0
>>>>>>> 4 6 0 0 6 0 3 0
>>>>>>> 2 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 1 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 1 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 6 108 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 98 54 0 86 11 0 25 0
>>>>>>> 36 24 0 39 25 0 10 1
>>>>>>> 67 8 0 63 63 0 41 0
>>>>>>> 34 0 0 35 34 0 0 0
>>>>>>> 75 0 0 75 77 0 0 0
>>>>>>> 34 0 0 35 35 0 0 0
>>>>>>> 75 0 0 74 76 0 0 0
>>>>>>> 33 0 0 34 33 0 0 0
>>>>>>> 0 0 0 0 5 0 0 0
>>>>>>> 0 0 0 0 0 0 6 0
>>>>>>> 11 0 0 0 0 0 11 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 17 0 0 0 0 1 0
>>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>>>>>> 4 5 0 0 0 0 12 0
>>>>>>> 2 0 0 0 0 0 26 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 4 0 0 0 0 4 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 4 0 0 0 0 0 2 0
>>>>>>> 2 0 0 0 0 0 24 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 4 0 0 0 0 0 7 0
>>>>>>> 2 1 0 0 0 0 1 0
>>>>>>> 0 0 0 0 2 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 6 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 4 6 0 0 0 0 3 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 2 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 4 71 0 0 0 0 0 0
>>>>>>> 0 1 0 0 0 0 0 0
>>>>>>> 2 36 0 0 0 0 1 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 1 0 0 0 0 0 1 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 79 6 0 79 79 0 2 0
>>>>>>> 25 0 0 25 26 0 6 0
>>>>>>> 43 18 0 39 46 0 23 0
>>>>>>> 36 0 0 36 36 0 31 0
>>>>>>> 68 1 0 66 68 0 0 0
>>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>>>>>> 36 0 0 36 36 0 0 0
>>>>>>> 48 0 0 48 49 0 0 0
>>>>>>> 20 0 0 20 20 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 3 14 0 1 0 0 11 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 0 4 0 0 0 0 4 0
>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>> 4 22 0 0 0 0 16 0
>>>>>>> 2 0 0 0 0 0 23 0
>>>>>>> 
>>>>>>> Regards,
>>>>>>> 
>>>>>>> Loïc Blot,
>>>>>>> UNIX Systems, Network and Security Engineer
>>>>>>> http://www.unix-experience.fr
>>>>>>> 
>>>>>>> 8 décembre 2014 09:36 "Loïc Blot"
>>>>>>> <loic.blot@unix-experience.fr> a
>>>>>>> écrit:
>>>>>>>> Hi Rick,
>>>>>>>> I stopped the jails this week-end and started it this
>>>>>>>> morning,
>>>>>>>> i'll
>>>>>>>> give you some stats this week.
>>>>>>>> 
>>>>>>>> Here is my nfsstat -m output (with your rsize/wsize
>>>>>>>> tweaks)
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> nfsv4,tcp,resvport,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negna
>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> etimeo=60,rsize=32768,wsize=32768,readdirsize=32768,readahead=1,wcommitsize=773136,timeout=120,retra
>>>>>> 
>>>>>>> s=2147483647
>>>>>>> 
>>>>>>> On server side my disks are on a raid controller which show a
>>>>>>> 512b
>>>>>>> volume and write performances
>>>>>>> are very honest (dd if=/dev/zero of=/jails/test.dd bs=4096
>>>>>>> count=100000000 => 450MBps)
>>>>>>> 
>>>>>>> Regards,
>>>>>>> 
>>>>>>> Loïc Blot,
>>>>>>> UNIX Systems, Network and Security Engineer
>>>>>>> http://www.unix-experience.fr
>>>>>>> 
>>>>>>> 5 décembre 2014 15:14 "Rick Macklem" <rmacklem@uoguelph.ca> a
>>>>>>> écrit:
>>>>>>> 
>>>>>>> Loic Blot wrote:
>>>>>>> 
>>>>>>> Hi,
>>>>>>> i'm trying to create a virtualisation environment based on
>>>>>>> jails.
>>>>>>> Those jails are stored under a big ZFS pool on a FreeBSD
>>>>>>> 9.3
>>>>>>> which
>>>>>>> export a NFSv4 volume. This NFSv4 volume was mounted on a
>>>>>>> big
>>>>>>> hypervisor (2 Xeon E5v3 + 128GB memory and 8 ports (but
>>>>>>> only 1
>>>>>>> was
>>>>>>> used at this time).
>>>>>>> 
>>>>>>> The problem is simple, my hypervisors runs 6 jails (used 1%
>>>>>>> cpu
>>>>>>> and
>>>>>>> 10GB RAM approximatively and less than 1MB bandwidth) and
>>>>>>> works
>>>>>>> fine at start but the system slows down and after 2-3 days
>>>>>>> become
>>>>>>> unusable. When i look at top command i see 80-100% on
>>>>>>> system
>>>>>>> and
>>>>>>> commands are very very slow. Many process are tagged with
>>>>>>> nfs_cl*.
>>>>>>> 
>>>>>>> To be honest, I would expect the slowness to be because of
>>>>>>> slow
>>>>>>> response
>>>>>>> from the NFSv4 server, but if you do:
>>>>>>> # ps axHl
>>>>>>> on a client when it is slow and post that, it would give us
>>>>>>> some
>>>>>>> more
>>>>>>> information on where the client side processes are sitting.
>>>>>>> If you also do something like:
>>>>>>> # nfsstat -c -w 1
>>>>>>> and let it run for a while, that should show you how many
>>>>>>> RPCs
>>>>>>> are
>>>>>>> being done and which ones.
>>>>>>> 
>>>>>>> # nfsstat -m
>>>>>>> will show you what your mount is actually using.
>>>>>>> The only mount option I can suggest trying is
>>>>>>> "rsize=32768,wsize=32768",
>>>>>>> since some network environments have difficulties with 64K.
>>>>>>> 
>>>>>>> There are a few things you can try on the NFSv4 server side,
>>>>>>> if
>>>>>>> it
>>>>>>> appears
>>>>>>> that the clients are generating a large RPC load.
>>>>>>> - disabling the DRC cache for TCP by setting
>>>>>>> vfs.nfsd.cachetcp=0
>>>>>>> - If the server is seeing a large write RPC load, then
>>>>>>> "sync=disabled"
>>>>>>> might help, although it does run a risk of data loss when
>>>>>>> the
>>>>>>> server
>>>>>>> crashes.
>>>>>>> Then there are a couple of other ZFS related things (I'm not
>>>>>>> a
>>>>>>> ZFS
>>>>>>> guy,
>>>>>>> but these have shown up on the mailing lists).
>>>>>>> - make sure your volumes are 4K aligned and ashift=12 (in
>>>>>>> case a
>>>>>>> drive
>>>>>>> that uses 4K sectors is pretending to be 512byte sectored)
>>>>>>> - never run over 70-80% full if write performance is an
>>>>>>> issue
>>>>>>> - use a zil on an SSD with good write performance
>>>>>>> 
>>>>>>> The only NFSv4 thing I can tell you is that it is known that
>>>>>>> ZFS's
>>>>>>> algorithm for determining sequential vs random I/O fails for
>>>>>>> NFSv4
>>>>>>> during writing and this can be a performance hit. The only
>>>>>>> workaround
>>>>>>> is to use NFSv3 mounts, since file handle affinity
>>>>>>> apparently
>>>>>>> fixes
>>>>>>> the problem and this is only done for NFSv3.
>>>>>>> 
>>>>>>> rick
>>>>>>> 
>>>>>>> I saw that there are TSO issues with igb then i'm trying to
>>>>>>> disable
>>>>>>> it with sysctl but the situation wasn't solved.
>>>>>>> 
>>>>>>> Someone has got ideas ? I can give you more informations if
>>>>>>> you
>>>>>>> need.
>>>>>>> 
>>>>>>> Thanks in advance.
>>>>>>> Regards,
>>>>>>> 
>>>>>>> Loïc Blot,
>>>>>>> UNIX Systems, Network and Security Engineer
>>>>>>> http://www.unix-experience.fr
>>>>>>> _______________________________________________
>>>>>>> freebsd-fs@freebsd.org mailing list
>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>>>>> To unsubscribe, send any mail to
>>>>>>> "freebsd-fs-unsubscribe@freebsd.org"
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> freebsd-fs@freebsd.org mailing list
>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>>>>> To unsubscribe, send any mail to
>>>>>>> "freebsd-fs-unsubscribe@freebsd.org"
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> freebsd-fs@freebsd.org mailing list
>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>>>>> To unsubscribe, send any mail to
>>>>>>> "freebsd-fs-unsubscribe@freebsd.org"
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> freebsd-fs@freebsd.org mailing list
>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>>>>> To unsubscribe, send any mail to
>>>>>>> "freebsd-fs-unsubscribe@freebsd.org"
>>>>>>> _______________________________________________
>>>>>>> freebsd-fs@freebsd.org mailing list
>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>>>>> To unsubscribe, send any mail to
>>>>>>> "freebsd-fs-unsubscribe@freebsd.org"
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> freebsd-fs@freebsd.org mailing list
>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>>>>> To unsubscribe, send any mail to
>>>>>>> "freebsd-fs-unsubscribe@freebsd.org"
>>>>> 
>>>>> _______________________________________________
>>>>> freebsd-fs@freebsd.org mailing list
>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>>> To unsubscribe, send any mail to
>>>>> "freebsd-fs-unsubscribe@freebsd.org"

---

Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?f8c1e0380b1c195ce3c63bc41fafb9e1>

Legal Notices | © 1995-2025 The FreeBSD Project. All rights reserved.

Contact