Date: Wed, 14 Jan 2015 09:08:23 +0000 From: "=?utf-8?B?TG/Dr2MgQmxvdA==?=" <loic.blot@unix-experience.fr> To: "Rick Macklem" <rmacklem@uoguelph.ca> Cc: freebsd-fs@freebsd.org Subject: Re: High Kernel Load with nfsv4 Message-ID: <4426a915c891b0faf962f1975c4dee1e@mail.unix-experience.fr> In-Reply-To: <1566336890.7368425.1420586936307.JavaMail.root@uoguelph.ca> References: <1566336890.7368425.1420586936307.JavaMail.root@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Rick, you fix doesn't help, i'm sorry. It improves the NFS server speed, but don't remove the problem. I think have loader.conf variables for there options could be great. I have wrote a cpp bench for testing my NFS server configuration but i can't reproduce the problem whereas the bench let many file descriptors opened (randomly). Regards, Loïc Blot, UNIX Systems, Network and Security Engineer http://www.unix-experience.fr 7 janvier 2015 00:29 "Rick Macklem" <rmacklem@uoguelph.ca> a écrit: > Loic Blot wrote: > >> Hi Rick, >> >> i saw that some people has issues with igb cards with NFS >> For example: >> http://freebsd.1045724.n5.nabble.com/NFS-over-LAGG-lacp-poor-performance-td5906349.html >> >> can my problem be related ? I use igb with default queue number. Here >> are my vmstat -i outputs > > I have no idea. Maybe someone familiar with this will respond? > > I do think that the large # of NFSv4 Opens (which are actually a form of lock) > could be a factor. The client and server has to search those lists for a match > for many NFSv4 operations, including all reads/writes. > > On the server side, the default hash table sizes are very small. This is in > part that I did testing on 256Mbyte i386 systems, so that the values were > safe for such a machine. > I'd suggest you increase the following on the server's kernel. > In sys/fs/nfs/nfs.h: > NFSSTATEHASHSIZE - This one is in every client header, so if you have a large# > of clients, you don't want to increase it too much. However > for a fairly large server handling not too many clients, I'd > try something like 1000 instead of 10. > (I just tried 100 on the small i386 laptop I have handy and it > seemed ok for a small test.) > NFSLOCKHASHSIZE - This one is a single global table, so I'd bump it way up, > 20000 maybe? > In sys/fs/nfsport.h: > NFSRV_V4STATELIMIT - The comment notes that the default of 500000 seems safe for > a 256Mbyte i386, so I'd bump it to something like 2000000 > for your case. > You will have to rebuild a kernel from sources after editing these values and > boot it on the server. Maybe these should become tunables so building a kernel > isn't necessary? > > I looked and there isn't much that can be done in the client. At this point, > the open_owners and opens are single lists for a client (a mount point on a > client machine for FreeBSD). If you post what you get for "nfsstat -e -c" > on a typical client in your setup, that would tell me if it is the open_owners > (which I suspect) or opens that will be a long list. (I would have to code > a patch to make either of these a hash table instead of a single linked > list. I should do this. It was on my to-do list, but got forgotten.;-) > > rick > >> Server side: >> >> interrupt total rate >> irq1: atkbd0 18 0 >> irq20: ehci1 2790134 2 >> irq21: ehci0 2547642 2 >> cpu0:timer 36299188 35 >> irq264: ciss0 6352476 6 >> irq265: igb0:que 0 2716692 2 >> irq266: igb0:que 1 32205278 31 >> irq267: igb0:que 2 38395109 37 >> irq268: igb0:que 3 1413468 1 >> irq269: igb0:que 4 39207930 38 >> irq270: igb0:que 5 1622715 1 >> irq271: igb0:que 6 1634676 1 >> irq272: igb0:que 7 1190123 1 >> irq273: igb0:link 2 0 >> cpu1:timer 14074423 13 >> cpu8:timer 12204739 11 >> cpu9:timer 11384192 11 >> cpu3:timer 10461566 10 >> cpu4:timer 12785103 12 >> cpu6:timer 10739344 10 >> cpu5:timer 10978294 10 >> cpu7:timer 10599705 10 >> cpu2:timer 13998891 13 >> cpu10:timer 11602361 11 >> cpu11:timer 11568523 11 >> Total 296772592 290 >> >> And client side: >> interrupt total rate >> irq9: acpi0 4 0 >> irq22: ehci1 950519 2 >> irq23: ehci0 1865060 4 >> cpu0:timer 248128035 546 >> irq268: mfi0 406896 0 >> irq269: igb0:que 0 2510556 5 >> irq270: igb0:que 1 2825336 6 >> irq271: igb0:que 2 2092958 4 >> irq272: igb0:que 3 1960849 4 >> irq273: igb0:que 4 2645369 5 >> irq274: igb0:que 5 2735187 6 >> irq275: igb0:que 6 2290531 5 >> irq276: igb0:que 7 2384370 5 >> irq277: igb0:link 2 0 >> irq287: igb2:que 0 1465051 3 >> irq288: igb2:que 1 856381 1 >> irq289: igb2:que 2 809318 1 >> irq290: igb2:que 3 897154 1 >> irq291: igb2:que 4 875755 1 >> irq292: igb2:que 5 35866117 78 >> irq293: igb2:que 6 846517 1 >> irq294: igb2:que 7 857979 1 >> irq295: igb2:link 2 0 >> irq296: igb3:que 0 535212 1 >> irq297: igb3:que 1 454359 1 >> irq298: igb3:que 2 454142 1 >> irq299: igb3:que 3 454623 1 >> irq300: igb3:que 4 456297 1 >> irq301: igb3:que 5 455482 1 >> irq302: igb3:que 6 456128 1 >> irq303: igb3:que 7 454680 1 >> irq304: igb3:link 3 0 >> irq305: ahci0 75 0 >> cpu1:timer 257233702 566 >> cpu13:timer 255603184 562 >> cpu7:timer 258492826 569 >> cpu12:timer 255819351 563 >> cpu6:timer 258493465 569 >> cpu15:timer 254694003 560 >> cpu3:timer 258171320 568 >> cpu22:timer 256506877 564 >> cpu5:timer 253401435 558 >> cpu16:timer 255412360 562 >> cpu11:timer 257318013 566 >> cpu20:timer 253648060 558 >> cpu2:timer 257864543 567 >> cpu17:timer 261828899 576 >> cpu9:timer 257497326 567 >> cpu18:timer 258451190 569 >> cpu8:timer 257784504 567 >> cpu14:timer 254923723 561 >> cpu10:timer 257265498 566 >> cpu19:timer 258775946 569 >> cpu4:timer 256368658 564 >> cpu23:timer 255050534 561 >> cpu21:timer 257663842 567 >> Total 6225260206 13710 >> >> Please note igb2 on client side is the dedicated link for NFSv4 >> >> Regards, >> >> Loïc Blot, >> UNIX Systems, Network and Security Engineer >> http://www.unix-experience.fr >> >> 6 janvier 2015 04:17 "Rick Macklem" <rmacklem@uoguelph.ca> a écrit: >>> Loic Blot wrote: >>> >>>> Hi Rick, >>>> nfsstat -e -s don't show usefull datas on server. >>> >>> Well, as far as I know, it returns valid information. >>> (See below.) >>> >>>> Server Info: >>>> Getattr Setattr Lookup Readlink Read Write Create >>>> Remove >>>> 26935254 16911 5755728 302 2334920 3673866 0 >>>> 328332 >>>> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus >>>> Access >>>> 77980 28 0 0 3 8900 3 >>>> 1806052 >>>> Mknod Fsstat Fsinfo PathConf Commit LookupP SetClId >>>> SetClIdCf >>>> 1 1095 0 0 614377 8172 8 >>>> 8 >>>> Open OpenAttr OpenDwnGr OpenCfrm DelePurge DeleRet GetFH >>>> Lock >>>> 1595299 0 44145 1495 0 0 5197490 >>>> 635015 >>>> LockT LockU Close Verify NVerify PutFH PutPubFH >>>> PutRootFH >>>> 0 614919 1270938 0 0 22688676 0 >>>> 5 >>>> Renew RestoreFH SaveFH Secinfo RelLckOwn V4Create >>>> 42104 197606 275820 0 143 4578 >>>> Server: >>>> Retfailed Faults Clients >>>> 0 0 6 >>>> OpenOwner Opens LockOwner Locks Delegs >>>> 32335 145448 204 181 0 >>> >>> Well, 145448 Opens are a lot of Open files. Each of these uses >>> a kernel malloc'd data structure that is linked into multiple >>> linked lists. >>> >>> The question is..why aren't these Opens being closed? >>> Since FreeBSD does I/O on an mmap'd file after closing it, >>> the FreeBSD NFSv4 client is forced to delay doing Close RPCs >>> until the vnode is VOP_INACTIVE()/VOP_RECLAIM()'d. (The >>> VOP_RECLAIM() case is needed, since VOP_INACTIVE() isn't >>> guaranteed to be called.) >>> >>> Since there were about 1.5 million Opens and 1.27 million >>> Closes, it does appear that Opens are being Closed. >>> Now, I'm not sure I would have imagined 1.5million file Opens >>> in a few days. My guess is this is the bottleneck. >>> >>> I'd suggest that you do: >>> # nfsstat -e -c >>> on each of the NFSv4 clients and see how many Opens/client >>> there are. I vaguely remember an upper limit in the client, >>> but can't remember what it is set to. >>> --> I suspect the client Open/Lock limit needs to be increased. >>> (I can't remember if the server also has a limit, but I >>> think it does.) >>> Then the size of the hash tables used to search the Opens >>> may also need to be increased a lot. >>> >>> Also, I'd suggest you take a look at whatever apps. are >>> running on the client(s) and try to figure out why they >>> are Opening so many files? >>> >>> My guess is that the client(s) are gettig bogged down by all >>> these Opens. >>> >>>> Server Cache Stats: >>>> Inprog Idem Non-idem Misses CacheSize TCPPeak >>>> 0 0 1 15082947 60 16522 >>>> >>>> Only GetAttr and Lookup increase and it's only every 4-5 seconds >>>> and >>>> only +2 to +5 into theses values. >>>> >>>> Now on client, if i take four processes stack i got >>>> >>>> PID TID COMM TDNAME KSTACK >>>> 63170 102547 mv - mi_switch+0xe1 >>>> turnstile_wait+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65 >>>> nfs_lookup+0x3d0 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4 >>>> vn_open_cred+0x21d kern_openat+0x26f amd64_syscall+0x351 >>>> Xfast_syscall+0xfb >>>> >>>> Another mv: >>>> 63140 101738 mv - mi_switch+0xe1 >>>> turnstile_wait+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65 >>>> nfs_lookup+0x3d0 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4 >>>> kern_statat_vnhook+0xae sys_lstat+0x30 amd64_syscall+0x351 >>>> Xfast_syscall+0xfb >>>> >>>> 62070 102170 sendmail - mi_switch+0xe1 >>>> sleepq_timedwait+0x3a _sleep+0x26e clnt_vc_call+0x666 >>>> clnt_reconnect_call+0x4fa newnfs_request+0xa8c nfscl_request+0x72 >>>> nfsrpc_lookup+0x1fb nfs_lookup+0x508 VOP_LOOKUP_APV+0xa1 >>>> lookup+0x59c namei+0x4d4 kern_statat_vnhook+0xae sys_lstat+0x30 >>>> amd64_syscall+0x351 Xfast_syscall+0xfb >>>> >>>> 63200 100930 mv - mi_switch+0xe1 >>>> turnstile_wait+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65 >>>> nfs_lookup+0x3d0 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4 >>>> kern_statat_vnhook+0xae sys_lstat+0x30 amd64_syscall+0x351 >>>> Xfast_syscall+0xfb >>> >>> The above simply says that thread 102710 is waiting for a Lookup >>> reply from the server and the other 3 are waiting for the mutex >>> that protects the state structures in the client. (I suspect >>> some other thread in the client is wading through the Open list, >>> if a single client has a lot of these 145K Opens.) >>> >>>> When client is in this state, server was doing nothing special >>>> (procstat -kk) >>>> >>>> PID TID COMM TDNAME KSTACK >>>> 895 100538 nfsd nfsd: master mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 >>>> _cv_timedwait_sig_sbt+0x18b svc_run_internal+0x4a1 svc_run+0x1de >>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c >>>> amd64_syscall+0x351 Xfast_syscall+0xfb >>>> 895 100568 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100569 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100570 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100571 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100572 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100573 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100575 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100576 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100577 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100578 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100579 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100580 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100581 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100582 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100583 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100584 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100585 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100586 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100587 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100588 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100589 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100590 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100592 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100593 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100594 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100595 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100596 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100597 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100598 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100599 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100600 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100602 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100603 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100604 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100605 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100606 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100607 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100608 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100609 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100610 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100611 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100612 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100613 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100614 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100615 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100617 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100618 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100619 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100621 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100622 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100623 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100624 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100625 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100626 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100627 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100628 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100629 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100630 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100631 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100632 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100633 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100634 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100635 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100636 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100638 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100639 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100640 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100641 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100642 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100643 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100644 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100645 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100646 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100647 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100648 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100649 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100651 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100652 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100653 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100654 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100655 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100656 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100657 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100658 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100659 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100661 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100662 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100684 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100685 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100686 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100797 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100798 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100799 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100800 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> 895 100801 nfsd nfsd: service mi_switch+0xe1 >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>> fork_trampoline+0xe >>>> >>>> I really think it's a client side problem, maybe a lookup problem. >>>> >>>> Regards, >>>> >>>> Loïc Blot, >>>> UNIX Systems, Network and Security Engineer >>>> http://www.unix-experience.fr >>>> >>>> 5 janvier 2015 14:35 "Rick Macklem" <rmacklem@uoguelph.ca> a >>>> écrit: >>>>> Loic Blot wrote: >>>>> >>>>>> Hi, >>>>>> happy new year Rick and @freebsd-fs. >>>>>> >>>>>> After some days, i looked my NFSv4.1 mount. At server start it >>>>>> was >>>>>> calm, but after 4 days, here is the top stat... >>>>>> >>>>>> CPU: 0.0% user, 0.0% nice, 100% system, 0.0% interrupt, 0.0% >>>>>> idle >>>>>> >>>>>> Definitively i think it's a problem on client side. What can i >>>>>> look >>>>>> into running kernel to resolve this issue ? >>>>> >>>>> Well, I'd start with: >>>>> # nfsstat -e -s >>>>> - run repeatedly on the server (once every N seconds in a loop). >>>>> Then look at the output, comparing the counts and see which RPCs >>>>> are being performed by the client(s). You are looking for which >>>>> RPCs are being done a lot. (If one RPC is almost 100% of the >>>>> load, >>>>> then it might be a client/caching issue for whatever that RPC is >>>>> doing.) >>>>> >>>>> Also look at the Open/Lock counts near the end of the output. >>>>> If the # of Opens/Locks is large, it may be possible to reduce >>>>> the >>>>> CPU overheads by using larger hash tables. >>>>> >>>>> Then you need to profile the server kernel to see where the CPU >>>>> is being used. >>>>> Hopefully someone else can fill you in on how to do that, because >>>>> I'll admit I don't know how to. >>>>> Basically you are looking to see if the CPU is being used in >>>>> the NFS server code or ZFS. >>>>> >>>>> Good luck with it, rick >>>>> >>>>>> Regards, >>>>>> >>>>>> Loïc Blot, >>>>>> UNIX Systems, Network and Security Engineer >>>>>> http://www.unix-experience.fr >>>>>> >>>>>> 30 décembre 2014 16:16 "Loïc Blot" >>>>>> <loic.blot@unix-experience.fr> >>>>>> a >>>>>> écrit: >>>>>>> Hi Rick, >>>>>>> i upgraded my jail host from FreeBSD 9.3 to 10.1 to use NFS >>>>>>> v4.1 >>>>>>> (mountoptions: >>>>>>> rw,rsize=32768,wsize=32768,tcp,nfsv4,minorversion=1) >>>>>>> >>>>>>> Performance is quite stable but it's slow. Not as slow as >>>>>>> before >>>>>>> but slow... services was launched >>>>>>> but no client are using them and system CPU % was 10-50%. >>>>>>> >>>>>>> I don't see anything on NFSv4.1 server, it's perfectly stable >>>>>>> and >>>>>>> functionnal. >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Loïc Blot, >>>>>>> UNIX Systems, Network and Security Engineer >>>>>>> http://www.unix-experience.fr >>>>>>> >>>>>>> 23 décembre 2014 00:20 "Rick Macklem" <rmacklem@uoguelph.ca> a >>>>>>> écrit: >>>>>>> >>>>>>>> Loic Blot wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> To clarify because of our exchanges. Here are the current >>>>>>>>> sysctl >>>>>>>>> options for server: >>>>>>>>> >>>>>>>>> vfs.nfsd.enable_nobodycheck=0 >>>>>>>>> vfs.nfsd.enable_nogroupcheck=0 >>>>>>>>> >>>>>>>>> vfs.nfsd.maxthreads=200 >>>>>>>>> vfs.nfsd.tcphighwater=10000 >>>>>>>>> vfs.nfsd.tcpcachetimeo=300 >>>>>>>>> vfs.nfsd.server_min_nfsvers=4 >>>>>>>>> >>>>>>>>> kern.maxvnodes=10000000 >>>>>>>>> kern.ipc.maxsockbuf=4194304 >>>>>>>>> net.inet.tcp.sendbuf_max=4194304 >>>>>>>>> net.inet.tcp.recvbuf_max=4194304 >>>>>>>>> >>>>>>>>> vfs.lookup_shared=0 >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Loïc Blot, >>>>>>>>> UNIX Systems, Network and Security Engineer >>>>>>>>> http://www.unix-experience.fr >>>>>>>>> >>>>>>>>> 22 décembre 2014 09:42 "Loïc Blot" >>>>>>>>> <loic.blot@unix-experience.fr> >>>>>>>>> a >>>>>>>>> écrit: >>>>>>>>> >>>>>>>>> Hi Rick, >>>>>>>>> my 5 jails runs this weekend and now i have some stats on >>>>>>>>> this >>>>>>>>> monday. >>>>>>>>> >>>>>>>>> Hopefully deadlock was fixed, yeah, but everything isn't good >>>>>>>>> :( >>>>>>>>> >>>>>>>>> On NFSv4 server (FreeBSD 10.1) system uses 35% CPU >>>>>>>>> >>>>>>>>> As i can see this is because of nfsd: >>>>>>>>> >>>>>>>>> 918 root 96 20 0 12352K 3372K rpcsvc 6 51.4H >>>>>>>>> 273.68% nfsd: server (nfsd) >>>>>>>>> >>>>>>>>> If i look at dmesg i see: >>>>>>>>> nfsd server cache flooded, try increasing >>>>>>>>> vfs.nfsd.tcphighwater >>>>>>>> >>>>>>>> Well, you have a couple of choices: >>>>>>>> 1 - Use NFSv4.1 (add "minorversion=1" to your mount options). >>>>>>>> (NFSv4.1 avoids use of the DRC and instead uses something >>>>>>>> called sessions. See below.) >>>>>>>> OR >>>>>>>> >>>>>>>>> vfs.nfsd.tcphighwater was set to 10000, i increase it to >>>>>>>>> 15000 >>>>>>>> >>>>>>>> 2 - Bump vfs.nfsd.tcphighwater way up, until you no longer see >>>>>>>> "nfs server cache flooded" messages. (I think Garrett Wollman >>>>>>>> uses >>>>>>>> 100000. (You may still see quite a bit of CPU overheads.) >>>>>>>> >>>>>>>> OR >>>>>>>> >>>>>>>> 3 - Set vfs.nfsd.cachetcp=0 (which disables the DRC and gets >>>>>>>> rid >>>>>>>> of the CPU overheads). However, there is a risk of data >>>>>>>> corruption >>>>>>>> if you have a client->server network partitioning of a >>>>>>>> moderate >>>>>>>> duration, because a non-idempotent RPC may get redone, becasue >>>>>>>> the client times out waiting for a reply. If a non-idempotent >>>>>>>> RPC gets done twice on the server, data corruption can happen. >>>>>>>> (The DRC provides improved correctness, but does add >>>>>>>> overhead.) >>>>>>>> >>>>>>>> If #1 works for you, it is the preferred solution, since >>>>>>>> Sessions >>>>>>>> in NFSv4.1 solves the correctness problem in a good, space >>>>>>>> bound >>>>>>>> way. A session basically has N (usually 32 or 64) slots and >>>>>>>> only >>>>>>>> allows one outstanding RPC/slot. As such, it can cache the >>>>>>>> previous >>>>>>>> reply for each slot (32 or 64 of them) and guarantee "exactly >>>>>>>> once" >>>>>>>> RPC semantics. >>>>>>>> >>>>>>>> rick >>>>>>>> >>>>>>>>> Here is 'nfsstat -s' output: >>>>>>>>> >>>>>>>>> Server Info: >>>>>>>>> Getattr Setattr Lookup Readlink Read Write Create >>>>>>>>> Remove >>>>>>>>> 12600652 1812 2501097 156 1386423 1983729 123 >>>>>>>>> 162067 >>>>>>>>> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus >>>>>>>>> Access >>>>>>>>> 36762 9 0 0 0 3147 0 >>>>>>>>> 623524 >>>>>>>>> Mknod Fsstat Fsinfo PathConf Commit >>>>>>>>> 0 0 0 0 328117 >>>>>>>>> Server Ret-Failed >>>>>>>>> 0 >>>>>>>>> Server Faults >>>>>>>>> 0 >>>>>>>>> Server Cache Stats: >>>>>>>>> Inprog Idem Non-idem Misses >>>>>>>>> 0 0 0 12635512 >>>>>>>>> Server Write Gathering: >>>>>>>>> WriteOps WriteRPC Opsaved >>>>>>>>> 1983729 1983729 0 >>>>>>>>> >>>>>>>>> And here is 'procstat -kk' for nfsd (server) >>>>>>>>> >>>>>>>>> 918 100528 nfsd nfsd: master mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 >>>>>>>>> _cv_timedwait_sig_sbt+0x18b svc_run_internal+0x4a1 >>>>>>>>> svc_run+0x1de >>>>>>>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c >>>>>>>>> amd64_syscall+0x351 Xfast_syscall+0xfb >>>>>>>>> 918 100568 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100569 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100570 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100571 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100572 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100573 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100574 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100575 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100576 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100577 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100578 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100579 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100580 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100581 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100582 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100583 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100584 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100585 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100586 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100587 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100588 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100589 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100590 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100591 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100592 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100593 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100594 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100595 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100596 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100597 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100598 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100599 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100600 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100601 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100602 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100603 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100604 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100605 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100606 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100607 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100608 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100609 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100610 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100611 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100612 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100613 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100614 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100615 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100616 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100617 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100618 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100619 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100620 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100621 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100622 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100623 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100624 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100625 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100626 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100627 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100628 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100629 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100630 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100631 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100632 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100633 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100634 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100635 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100636 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100637 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100638 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100639 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100640 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100641 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100642 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100643 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100644 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100645 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100646 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100647 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100648 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100649 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100650 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100651 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100652 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100653 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100654 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100655 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100656 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100657 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100658 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100659 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100660 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100661 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100662 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> --- >>>>>>>>> >>>>>>>>> Now if we look at client (FreeBSD 9.3) >>>>>>>>> >>>>>>>>> We see system was very busy and do many and many interrupts >>>>>>>>> >>>>>>>>> CPU: 0.0% user, 0.0% nice, 37.8% system, 51.2% interrupt, >>>>>>>>> 11.0% >>>>>>>>> idle >>>>>>>>> >>>>>>>>> A look at process list shows that there are many sendmail >>>>>>>>> process >>>>>>>>> in >>>>>>>>> state nfstry >>>>>>>>> >>>>>>>>> nfstry 18 32:27 0.88% sendmail: Queue runner@00:30:00 for >>>>>>>>> /var/spool/clientm >>>>>>>>> >>>>>>>>> Here is 'nfsstat -c' output: >>>>>>>>> >>>>>>>>> Client Info: >>>>>>>>> Rpc Counts: >>>>>>>>> Getattr Setattr Lookup Readlink Read Write Create >>>>>>>>> Remove >>>>>>>>> 1051347 1724 2494481 118 903902 1901285 162676 >>>>>>>>> 161899 >>>>>>>>> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus >>>>>>>>> Access >>>>>>>>> 36744 2 0 114 40 3131 0 >>>>>>>>> 544136 >>>>>>>>> Mknod Fsstat Fsinfo PathConf Commit >>>>>>>>> 9 0 0 0 245821 >>>>>>>>> Rpc Info: >>>>>>>>> TimedOut Invalid X Replies Retries Requests >>>>>>>>> 0 0 0 0 8356557 >>>>>>>>> Cache Info: >>>>>>>>> Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits >>>>>>>>> Misses >>>>>>>>> 108754455 491475 54229224 2437229 46814561 821723 5132123 >>>>>>>>> 1871871 >>>>>>>>> BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits >>>>>>>>> Misses >>>>>>>>> 144035 118 53736 2753 27813 1 57238839 >>>>>>>>> 544205 >>>>>>>>> >>>>>>>>> If you need more things, tell me, i let the PoC in this >>>>>>>>> state. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Loïc Blot, >>>>>>>>> UNIX Systems, Network and Security Engineer >>>>>>>>> http://www.unix-experience.fr >>>>>>>>> >>>>>>>>> 21 décembre 2014 01:33 "Rick Macklem" <rmacklem@uoguelph.ca> >>>>>>>>> a >>>>>>>>> écrit: >>>>>>>>> >>>>>>>>> Loic Blot wrote: >>>>>>>>> >>>>>>>>>> Hi Rick, >>>>>>>>>> ok, i don't need locallocks, i haven't understand option was >>>>>>>>>> for >>>>>>>>>> that >>>>>>>>>> usage, i removed it. >>>>>>>>>> I do more tests on monday. >>>>>>>>>> Thanks for the deadlock fix, for other people :) >>>>>>>>> >>>>>>>>> Good. Please let us know if running with >>>>>>>>> vfs.nfsd.enable_locallocks=0 >>>>>>>>> gets rid of the deadlocks? (I think it fixes the one you >>>>>>>>> saw.) >>>>>>>>> >>>>>>>>> On the performance side, you might also want to try different >>>>>>>>> values >>>>>>>>> of >>>>>>>>> readahead, if the Linux client has such a mount option. (With >>>>>>>>> the >>>>>>>>> NFSv4-ZFS sequential vs random I/O heuristic, I have no idea >>>>>>>>> what >>>>>>>>> the >>>>>>>>> optimal readahead value would be.) >>>>>>>>> >>>>>>>>> Good luck with it and please let us know how it goes, rick >>>>>>>>> ps: I now have a patch to fix the deadlock when >>>>>>>>> vfs.nfsd.enable_locallocks=1 >>>>>>>>> is set. I'll post it for anyone who is interested after I put >>>>>>>>> it >>>>>>>>> through some testing. >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Best regards, >>>>>>>>> Loïc BLOT, >>>>>>>>> UNIX systems, security and network engineer >>>>>>>>> http://www.unix-experience.fr >>>>>>>>> >>>>>>>>> Le jeudi 18 décembre 2014 à 19:46 -0500, Rick Macklem a écrit >>>>>>>>> : >>>>>>>>> >>>>>>>>> Loic Blot wrote: >>>>>>>>>> Hi rick, >>>>>>>>>> i tried to start a LXC container on Debian Squeeze from my >>>>>>>>>> freebsd >>>>>>>>>> ZFS+NFSv4 server and i also have a deadlock on nfsd >>>>>>>>>> (vfs.lookup_shared=0). Deadlock procs each time i launch a >>>>>>>>>> squeeze >>>>>>>>>> container, it seems (3 tries, 3 fails). >>>>>>>>> >>>>>>>>> Well, I`ll take a look at this `procstat -kk`, but the only >>>>>>>>> thing >>>>>>>>> I`ve seen posted w.r.t. avoiding deadlocks in ZFS is to not >>>>>>>>> use >>>>>>>>> nullfs. (I have no idea if you are using any nullfs mounts, >>>>>>>>> but >>>>>>>>> if so, try getting rid of them.) >>>>>>>>> >>>>>>>>> Here`s a high level post about the ZFS and vnode locking >>>>>>>>> problem, >>>>>>>>> but there is no patch available, as far as I know. >>>>>>>>> >>>>>>>>> http://docs.FreeBSD.org/cgi/mid.cgi?54739F41.8030407 >>>>>>>>> >>>>>>>>> rick >>>>>>>>> >>>>>>>>> 921 - D 0:00.02 nfsd: server (nfsd) >>>>>>>>> >>>>>>>>> Here is the procstat -kk >>>>>>>>> >>>>>>>>> PID TID COMM TDNAME KSTACK >>>>>>>>> 921 100538 nfsd nfsd: master mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad >>>>>>>>> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554 >>>>>>>>> svc_run_internal+0xc77 svc_run+0x1de nfsrvd_nfsd+0x1ca >>>>>>>>> nfssvc_nfsd+0x107 sys_nfssvc+0x9c >>>>>>>>> 921 100572 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100573 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100574 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100575 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100576 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100577 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100578 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100579 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100580 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100581 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100582 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100583 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100584 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100585 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100586 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100587 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100588 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100589 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100590 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100591 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100592 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100593 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100594 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100595 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100596 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100597 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100598 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100599 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100600 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100601 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100602 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100603 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100604 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100605 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100606 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100607 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100608 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100609 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100610 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100611 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100612 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100613 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100614 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100615 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100616 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >>>>>>>>> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f >>>>>>>>> nfsrvd_lock+0x5b1 >>>>>>>>> nfsrvd_dorpc+0xec6 nfssvc_program+0x554 >>>>>>>>> svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 921 100617 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100618 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 >>>>>>>>> svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 921 100619 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100620 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100621 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100622 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100623 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100624 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100625 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100626 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100627 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100628 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100629 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100630 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100631 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100632 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100633 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100634 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100635 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100636 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100637 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100638 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100639 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100640 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100641 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100642 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100643 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100644 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100645 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100646 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100647 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100648 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100649 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100650 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100651 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100652 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100653 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100654 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100655 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100656 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100657 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100658 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100659 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100660 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100661 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100662 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100663 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100664 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100665 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 921 100666 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >>>>>>>>> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8 >>>>>>>>> nfsrvd_dorpc+0xc76 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Loïc Blot, >>>>>>>>> UNIX Systems, Network and Security Engineer >>>>>>>>> http://www.unix-experience.fr >>>>>>>>> >>>>>>>>> 15 décembre 2014 15:18 "Rick Macklem" <rmacklem@uoguelph.ca> >>>>>>>>> a >>>>>>>>> écrit: >>>>>>>>> >>>>>>>>> Loic Blot wrote: >>>>>>>>> >>>>>>>>>> For more informations, here is procstat -kk on nfsd, if you >>>>>>>>>> need >>>>>>>>>> more >>>>>>>>>> hot datas, tell me. >>>>>>>>>> >>>>>>>>>> Regards, PID TID COMM TDNAME KSTACK >>>>>>>>>> 918 100529 nfsd nfsd: master mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_run+0x1de >>>>>>>>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c >>>>>>>>>> amd64_syscall+0x351 >>>>>>>>> >>>>>>>>> Well, most of the threads are stuck like this one, waiting >>>>>>>>> for >>>>>>>>> a >>>>>>>>> vnode >>>>>>>>> lock in ZFS. All of them appear to be in zfs_fhtovp(). >>>>>>>>> I`m not a ZFS guy, so I can`t help much. I`ll try changing >>>>>>>>> the >>>>>>>>> subject line >>>>>>>>> to include ZFS vnode lock, so maybe the ZFS guys will take a >>>>>>>>> look. >>>>>>>>> >>>>>>>>> The only thing I`ve seen suggested is trying: >>>>>>>>> sysctl vfs.lookup_shared=0 >>>>>>>>> to disable shared vop_lookup()s. Apparently zfs_lookup() >>>>>>>>> doesn`t >>>>>>>>> obey the vnode locking rules for lookup and rename, according >>>>>>>>> to >>>>>>>>> the posting I saw. >>>>>>>>> >>>>>>>>> I`ve added a couple of comments about the other threads >>>>>>>>> below, >>>>>>>>> but >>>>>>>>> they are all either waiting for an RPC request or waiting for >>>>>>>>> the >>>>>>>>> threads stuck on the ZFS vnode lock to complete. >>>>>>>>> >>>>>>>>> rick >>>>>>>>> >>>>>>>>>> 918 100564 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>>> fork_trampoline+0xe >>>>>>>>> >>>>>>>>> Fyi, this thread is just waiting for an RPC to arrive. >>>>>>>>> (Normal) >>>>>>>>> >>>>>>>>>> 918 100565 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>>> fork_trampoline+0xe >>>>>>>>>> 918 100566 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>>> fork_trampoline+0xe >>>>>>>>>> 918 100567 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>>> fork_trampoline+0xe >>>>>>>>>> 918 100568 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>>> fork_trampoline+0xe >>>>>>>>>> 918 100569 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>>> fork_trampoline+0xe >>>>>>>>>> 918 100570 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>>>>>>>>> _cv_wait_sig+0x16a >>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>>>>>>>>> fork_trampoline+0xe >>>>>>>>>> 918 100571 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 >>>>>>>>>> svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100572 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >>>>>>>>>> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8 >>>>>>>>>> nfsrvd_dorpc+0xc76 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> >>>>>>>>> This one (and a few others) are waiting for the nfsv4_lock. >>>>>>>>> This >>>>>>>>> happens >>>>>>>>> because other threads are stuck with RPCs in progress. (ie. >>>>>>>>> The >>>>>>>>> ones >>>>>>>>> waiting on the vnode lock in zfs_fhtovp().) >>>>>>>>> For these, the RPC needs to lock out other threads to do the >>>>>>>>> operation, >>>>>>>>> so it waits for the nfsv4_lock() which can exclusively lock >>>>>>>>> the >>>>>>>>> NFSv4 >>>>>>>>> data structures once all other nfsd threads complete their >>>>>>>>> RPCs >>>>>>>>> in >>>>>>>>> progress. >>>>>>>>> >>>>>>>>>> 918 100573 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 >>>>>>>>>> svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> >>>>>>>>> Same as above. >>>>>>>>> >>>>>>>>>> 918 100574 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100575 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100576 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100577 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100578 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100579 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100580 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100581 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100582 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100583 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100584 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100585 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100586 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100587 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100588 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100589 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100590 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100591 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100592 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100593 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100594 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100595 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100596 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100597 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100598 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100599 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100600 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100601 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100602 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100603 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100604 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100605 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100606 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>>> 918 100607 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>>> svc_thread_start+0xb >>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> >>>>>>>>> Lots more waiting for the ZFS vnode lock in zfs_fhtovp(). >>>>>>>>> >>>>>>>>> 918 100608 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >>>>>>>>> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f >>>>>>>>> nfsrvd_lock+0x5b1 >>>>>>>>> nfsrvd_dorpc+0xec6 nfssvc_program+0x554 >>>>>>>>> svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100609 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100610 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad >>>>>>>>> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554 >>>>>>>>> svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a >>>>>>>>> fork_trampoline+0xe >>>>>>>>> 918 100611 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 >>>>>>>>> svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100612 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 >>>>>>>>> svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100613 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 >>>>>>>>> svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100614 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 >>>>>>>>> svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100615 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 >>>>>>>>> svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100616 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 >>>>>>>>> svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100617 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 >>>>>>>>> svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100618 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 >>>>>>>>> svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100619 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100620 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100621 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 >>>>>>>>> svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100622 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100623 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 >>>>>>>>> svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100624 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100625 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100626 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100627 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100628 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100629 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100630 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100631 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100632 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100633 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100634 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100635 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100636 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100637 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100638 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100639 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100640 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100641 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100642 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100643 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100644 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100645 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100646 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100647 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100648 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100649 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100650 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100651 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100652 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100653 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100654 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100655 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100656 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100657 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> 918 100658 nfsd nfsd: service mi_switch+0xe1 >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>>>>>>>> zfs_fhtovp+0x38d >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 >>>>>>>>> svc_thread_start+0xb >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe >>>>>>>>> >>>>>>>>> Loïc Blot, >>>>>>>>> UNIX Systems, Network and Security Engineer >>>>>>>>> http://www.unix-experience.fr >>>>>>>>> >>>>>>>>> 15 décembre 2014 13:29 "Loïc Blot" >>>>>>>>> <loic.blot@unix-experience.fr> >>>>>>>>> a >>>>>>>>> écrit: >>>>>>>>> >>>>>>>>> Hmmm... >>>>>>>>> now i'm experiencing a deadlock. >>>>>>>>> >>>>>>>>> 0 918 915 0 21 0 12352 3372 zfs D - 1:48.64 nfsd: server >>>>>>>>> (nfsd) >>>>>>>>> >>>>>>>>> the only issue was to reboot the server, but after rebooting >>>>>>>>> deadlock arrives a second time when i >>>>>>>>> start my jails over NFS. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Loïc Blot, >>>>>>>>> UNIX Systems, Network and Security Engineer >>>>>>>>> http://www.unix-experience.fr >>>>>>>>> >>>>>>>>> 15 décembre 2014 10:07 "Loïc Blot" >>>>>>>>> <loic.blot@unix-experience.fr> >>>>>>>>> a >>>>>>>>> écrit: >>>>>>>>> >>>>>>>>> Hi Rick, >>>>>>>>> after talking with my N+1, NFSv4 is required on our >>>>>>>>> infrastructure. >>>>>>>>> I tried to upgrade NFSv4+ZFS >>>>>>>>> server from 9.3 to 10.1, i hope this will resolve some >>>>>>>>> issues... >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Loïc Blot, >>>>>>>>> UNIX Systems, Network and Security Engineer >>>>>>>>> http://www.unix-experience.fr >>>>>>>>> >>>>>>>>> 10 décembre 2014 15:36 "Loïc Blot" >>>>>>>>> <loic.blot@unix-experience.fr> >>>>>>>>> a >>>>>>>>> écrit: >>>>>>>>> >>>>>>>>> Hi Rick, >>>>>>>>> thanks for your suggestion. >>>>>>>>> For my locking bug, rpc.lockd is stucked in rpcrecv state on >>>>>>>>> the >>>>>>>>> server. kill -9 doesn't affect the >>>>>>>>> process, it's blocked.... (State: Ds) >>>>>>>>> >>>>>>>>> for the performances >>>>>>>>> >>>>>>>>> NFSv3: 60Mbps >>>>>>>>> NFSv4: 45Mbps >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Loïc Blot, >>>>>>>>> UNIX Systems, Network and Security Engineer >>>>>>>>> http://www.unix-experience.fr >>>>>>>>> >>>>>>>>> 10 décembre 2014 13:56 "Rick Macklem" <rmacklem@uoguelph.ca> >>>>>>>>> a >>>>>>>>> écrit: >>>>>>>>> >>>>>>>>> Loic Blot wrote: >>>>>>>>> >>>>>>>>>> Hi Rick, >>>>>>>>>> I'm trying NFSv3. >>>>>>>>>> Some jails are starting very well but now i have an issue >>>>>>>>>> with >>>>>>>>>> lockd >>>>>>>>>> after some minutes: >>>>>>>>>> >>>>>>>>>> nfs server 10.10.X.8:/jails: lockd not responding >>>>>>>>>> nfs server 10.10.X.8:/jails lockd is alive again >>>>>>>>>> >>>>>>>>>> I look at mbuf, but i seems there is no problem. >>>>>>>>> >>>>>>>>> Well, if you need locks to be visible across multiple >>>>>>>>> clients, >>>>>>>>> then >>>>>>>>> I'm afraid you are stuck with using NFSv4 and the >>>>>>>>> performance >>>>>>>>> you >>>>>>>>> get >>>>>>>>> from it. (There is no way to do file handle affinity for >>>>>>>>> NFSv4 >>>>>>>>> because >>>>>>>>> the read and write ops are buried in the compound RPC and >>>>>>>>> not >>>>>>>>> easily >>>>>>>>> recognized.) >>>>>>>>> >>>>>>>>> If the locks don't need to be visible across multiple >>>>>>>>> clients, >>>>>>>>> I'd >>>>>>>>> suggest trying the "nolockd" option with nfsv3. >>>>>>>>> >>>>>>>>>> Here is my rc.conf on server: >>>>>>>>>> >>>>>>>>>> nfs_server_enable="YES" >>>>>>>>>> nfsv4_server_enable="YES" >>>>>>>>>> nfsuserd_enable="YES" >>>>>>>>>> nfsd_server_flags="-u -t -n 256" >>>>>>>>>> mountd_enable="YES" >>>>>>>>>> mountd_flags="-r" >>>>>>>>>> nfsuserd_flags="-usertimeout 0 -force 20" >>>>>>>>>> rpcbind_enable="YES" >>>>>>>>>> rpc_lockd_enable="YES" >>>>>>>>>> rpc_statd_enable="YES" >>>>>>>>>> >>>>>>>>>> Here is the client: >>>>>>>>>> >>>>>>>>>> nfsuserd_enable="YES" >>>>>>>>>> nfsuserd_flags="-usertimeout 0 -force 20" >>>>>>>>>> nfscbd_enable="YES" >>>>>>>>>> rpc_lockd_enable="YES" >>>>>>>>>> rpc_statd_enable="YES" >>>>>>>>>> >>>>>>>>>> Have you got an idea ? >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> Loïc Blot, >>>>>>>>>> UNIX Systems, Network and Security Engineer >>>>>>>>>> http://www.unix-experience.fr >>>>>>>>>> >>>>>>>>>> 9 décembre 2014 04:31 "Rick Macklem" <rmacklem@uoguelph.ca> >>>>>>>>>> a >>>>>>>>>> écrit: >>>>>>>>>>> Loic Blot wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi rick, >>>>>>>>>>>> >>>>>>>>>>>> I waited 3 hours (no lag at jail launch) and now I do: >>>>>>>>>>>> sysrc >>>>>>>>>>>> memcached_flags="-v -m 512" >>>>>>>>>>>> Command was very very slow... >>>>>>>>>>>> >>>>>>>>>>>> Here is a dd over NFS: >>>>>>>>>>>> >>>>>>>>>>>> 601062912 bytes transferred in 21.060679 secs (28539579 >>>>>>>>>>>> bytes/sec) >>>>>>>>>>> >>>>>>>>>>> Can you try the same read using an NFSv3 mount? >>>>>>>>>>> (If it runs much faster, you have probably been bitten by >>>>>>>>>>> the >>>>>>>>>>> ZFS >>>>>>>>>>> "sequential vs random" read heuristic which I've been told >>>>>>>>>>> things >>>>>>>>>>> NFS is doing "random" reads without file handle affinity. >>>>>>>>>>> File >>>>>>>>>>> handle affinity is very hard to do for NFSv4, so it isn't >>>>>>>>>>> done.) >>>>>>>>> >>>>>>>>> I was actually suggesting that you try the "dd" over nfsv3 >>>>>>>>> to >>>>>>>>> see >>>>>>>>> how >>>>>>>>> the performance compared with nfsv4. If you do that, please >>>>>>>>> post >>>>>>>>> the >>>>>>>>> comparable results. >>>>>>>>> >>>>>>>>> Someday I would like to try and get ZFS's sequential vs >>>>>>>>> random >>>>>>>>> read >>>>>>>>> heuristic modified and any info on what difference in >>>>>>>>> performance >>>>>>>>> that >>>>>>>>> might make for NFS would be useful. >>>>>>>>> >>>>>>>>> rick >>>>>>>>> >>>>>>>>> rick >>>>>>>>> >>>>>>>>> This is quite slow... >>>>>>>>> >>>>>>>>> You can found some nfsstat below (command isn't finished >>>>>>>>> yet) >>>>>>>>> >>>>>>>>> nfsstat -c -w 1 >>>>>>>>> >>>>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 4 0 0 0 0 0 16 0 >>>>>>>>> 2 0 0 0 0 0 17 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 4 0 0 0 0 4 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 4 0 0 0 0 0 3 0 >>>>>>>>> 0 0 0 0 0 0 3 0 >>>>>>>>> 37 10 0 8 0 0 14 1 >>>>>>>>> 18 16 0 4 1 2 4 0 >>>>>>>>> 78 91 0 82 6 12 30 0 >>>>>>>>> 19 18 0 2 2 4 2 0 >>>>>>>>> 0 0 0 0 2 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 1 0 0 0 0 1 0 >>>>>>>>> 4 6 0 0 6 0 3 0 >>>>>>>>> 2 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 1 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 1 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 6 108 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 98 54 0 86 11 0 25 0 >>>>>>>>> 36 24 0 39 25 0 10 1 >>>>>>>>> 67 8 0 63 63 0 41 0 >>>>>>>>> 34 0 0 35 34 0 0 0 >>>>>>>>> 75 0 0 75 77 0 0 0 >>>>>>>>> 34 0 0 35 35 0 0 0 >>>>>>>>> 75 0 0 74 76 0 0 0 >>>>>>>>> 33 0 0 34 33 0 0 0 >>>>>>>>> 0 0 0 0 5 0 0 0 >>>>>>>>> 0 0 0 0 0 0 6 0 >>>>>>>>> 11 0 0 0 0 0 11 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 17 0 0 0 0 1 0 >>>>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir >>>>>>>>> 4 5 0 0 0 0 12 0 >>>>>>>>> 2 0 0 0 0 0 26 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 4 0 0 0 0 4 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 4 0 0 0 0 0 2 0 >>>>>>>>> 2 0 0 0 0 0 24 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 4 0 0 0 0 0 7 0 >>>>>>>>> 2 1 0 0 0 0 1 0 >>>>>>>>> 0 0 0 0 2 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 6 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 4 6 0 0 0 0 3 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 2 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 4 71 0 0 0 0 0 0 >>>>>>>>> 0 1 0 0 0 0 0 0 >>>>>>>>> 2 36 0 0 0 0 1 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 1 0 0 0 0 0 1 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 79 6 0 79 79 0 2 0 >>>>>>>>> 25 0 0 25 26 0 6 0 >>>>>>>>> 43 18 0 39 46 0 23 0 >>>>>>>>> 36 0 0 36 36 0 31 0 >>>>>>>>> 68 1 0 66 68 0 0 0 >>>>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir >>>>>>>>> 36 0 0 36 36 0 0 0 >>>>>>>>> 48 0 0 48 49 0 0 0 >>>>>>>>> 20 0 0 20 20 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 3 14 0 1 0 0 11 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 0 4 0 0 0 0 4 0 >>>>>>>>> 0 0 0 0 0 0 0 0 >>>>>>>>> 4 22 0 0 0 0 16 0 >>>>>>>>> 2 0 0 0 0 0 23 0 >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Loïc Blot, >>>>>>>>> UNIX Systems, Network and Security Engineer >>>>>>>>> http://www.unix-experience.fr >>>>>>>>> >>>>>>>>> 8 décembre 2014 09:36 "Loïc Blot" >>>>>>>>> <loic.blot@unix-experience.fr> a >>>>>>>>> écrit: >>>>>>>>>> Hi Rick, >>>>>>>>>> I stopped the jails this week-end and started it this >>>>>>>>>> morning, >>>>>>>>>> i'll >>>>>>>>>> give you some stats this week. >>>>>>>>>> >>>>>>>>>> Here is my nfsstat -m output (with your rsize/wsize >>>>>>>>>> tweaks) >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > nfsv4,tcp,resvport,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negna >>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > etimeo=60,rsize=32768,wsize=32768,readdirsize=32768,readahead=1,wcommitsize=773136,timeout=120,retra >>>>>>>> >>>>>>>>> s=2147483647 >>>>>>>>> >>>>>>>>> On server side my disks are on a raid controller which show a >>>>>>>>> 512b >>>>>>>>> volume and write performances >>>>>>>>> are very honest (dd if=/dev/zero of=/jails/test.dd bs=4096 >>>>>>>>> count=100000000 => 450MBps) >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Loïc Blot, >>>>>>>>> UNIX Systems, Network and Security Engineer >>>>>>>>> http://www.unix-experience.fr >>>>>>>>> >>>>>>>>> 5 décembre 2014 15:14 "Rick Macklem" <rmacklem@uoguelph.ca> a >>>>>>>>> écrit: >>>>>>>>> >>>>>>>>> Loic Blot wrote: >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> i'm trying to create a virtualisation environment based on >>>>>>>>> jails. >>>>>>>>> Those jails are stored under a big ZFS pool on a FreeBSD >>>>>>>>> 9.3 >>>>>>>>> which >>>>>>>>> export a NFSv4 volume. This NFSv4 volume was mounted on a >>>>>>>>> big >>>>>>>>> hypervisor (2 Xeon E5v3 + 128GB memory and 8 ports (but >>>>>>>>> only 1 >>>>>>>>> was >>>>>>>>> used at this time). >>>>>>>>> >>>>>>>>> The problem is simple, my hypervisors runs 6 jails (used 1% >>>>>>>>> cpu >>>>>>>>> and >>>>>>>>> 10GB RAM approximatively and less than 1MB bandwidth) and >>>>>>>>> works >>>>>>>>> fine at start but the system slows down and after 2-3 days >>>>>>>>> become >>>>>>>>> unusable. When i look at top command i see 80-100% on >>>>>>>>> system >>>>>>>>> and >>>>>>>>> commands are very very slow. Many process are tagged with >>>>>>>>> nfs_cl*. >>>>>>>>> >>>>>>>>> To be honest, I would expect the slowness to be because of >>>>>>>>> slow >>>>>>>>> response >>>>>>>>> from the NFSv4 server, but if you do: >>>>>>>>> # ps axHl >>>>>>>>> on a client when it is slow and post that, it would give us >>>>>>>>> some >>>>>>>>> more >>>>>>>>> information on where the client side processes are sitting. >>>>>>>>> If you also do something like: >>>>>>>>> # nfsstat -c -w 1 >>>>>>>>> and let it run for a while, that should show you how many >>>>>>>>> RPCs >>>>>>>>> are >>>>>>>>> being done and which ones. >>>>>>>>> >>>>>>>>> # nfsstat -m >>>>>>>>> will show you what your mount is actually using. >>>>>>>>> The only mount option I can suggest trying is >>>>>>>>> "rsize=32768,wsize=32768", >>>>>>>>> since some network environments have difficulties with 64K. >>>>>>>>> >>>>>>>>> There are a few things you can try on the NFSv4 server side, >>>>>>>>> if >>>>>>>>> it >>>>>>>>> appears >>>>>>>>> that the clients are generating a large RPC load. >>>>>>>>> - disabling the DRC cache for TCP by setting >>>>>>>>> vfs.nfsd.cachetcp=0 >>>>>>>>> - If the server is seeing a large write RPC load, then >>>>>>>>> "sync=disabled" >>>>>>>>> might help, although it does run a risk of data loss when >>>>>>>>> the >>>>>>>>> server >>>>>>>>> crashes. >>>>>>>>> Then there are a couple of other ZFS related things (I'm not >>>>>>>>> a >>>>>>>>> ZFS >>>>>>>>> guy, >>>>>>>>> but these have shown up on the mailing lists). >>>>>>>>> - make sure your volumes are 4K aligned and ashift=12 (in >>>>>>>>> case a >>>>>>>>> drive >>>>>>>>> that uses 4K sectors is pretending to be 512byte sectored) >>>>>>>>> - never run over 70-80% full if write performance is an >>>>>>>>> issue >>>>>>>>> - use a zil on an SSD with good write performance >>>>>>>>> >>>>>>>>> The only NFSv4 thing I can tell you is that it is known that >>>>>>>>> ZFS's >>>>>>>>> algorithm for determining sequential vs random I/O fails for >>>>>>>>> NFSv4 >>>>>>>>> during writing and this can be a performance hit. The only >>>>>>>>> workaround >>>>>>>>> is to use NFSv3 mounts, since file handle affinity >>>>>>>>> apparently >>>>>>>>> fixes >>>>>>>>> the problem and this is only done for NFSv3. >>>>>>>>> >>>>>>>>> rick >>>>>>>>> >>>>>>>>> I saw that there are TSO issues with igb then i'm trying to >>>>>>>>> disable >>>>>>>>> it with sysctl but the situation wasn't solved. >>>>>>>>> >>>>>>>>> Someone has got ideas ? I can give you more informations if >>>>>>>>> you >>>>>>>>> need. >>>>>>>>> >>>>>>>>> Thanks in advance. >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Loïc Blot, >>>>>>>>> UNIX Systems, Network and Security Engineer >>>>>>>>> http://www.unix-experience.fr >>>>>>>>> _______________________________________________ >>>>>>>>> freebsd-fs@freebsd.org mailing list >>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>>>>>>> To unsubscribe, send any mail to >>>>>>>>> "freebsd-fs-unsubscribe@freebsd.org" >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> freebsd-fs@freebsd.org mailing list >>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>>>>>>> To unsubscribe, send any mail to >>>>>>>>> "freebsd-fs-unsubscribe@freebsd.org" >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> freebsd-fs@freebsd.org mailing list >>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>>>>>>> To unsubscribe, send any mail to >>>>>>>>> "freebsd-fs-unsubscribe@freebsd.org" >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> freebsd-fs@freebsd.org mailing list >>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>>>>>>> To unsubscribe, send any mail to >>>>>>>>> "freebsd-fs-unsubscribe@freebsd.org" >>>>>>>>> _______________________________________________ >>>>>>>>> freebsd-fs@freebsd.org mailing list >>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>>>>>>> To unsubscribe, send any mail to >>>>>>>>> "freebsd-fs-unsubscribe@freebsd.org" >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> freebsd-fs@freebsd.org mailing list >>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>>>>>>> To unsubscribe, send any mail to >>>>>>>>> "freebsd-fs-unsubscribe@freebsd.org" >>>>>>> >>>>>>> _______________________________________________ >>>>>>> freebsd-fs@freebsd.org mailing list >>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>>>>> To unsubscribe, send any mail to >>>>>>> "freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4426a915c891b0faf962f1975c4dee1e>