Date: Tue, 06 Jan 2015 08:25:40 +0000 From: "=?utf-8?B?TG/Dr2MgQmxvdA==?=" <loic.blot@unix-experience.fr> To: "Rick Macklem" <rmacklem@uoguelph.ca> Cc: freebsd-fs@freebsd.org Subject: Re: High Kernel Load with nfsv4 Message-ID: <f8c1e0380b1c195ce3c63bc41fafb9e1@mail.unix-experience.fr> In-Reply-To: <2093433467.6650515.1420514242874.JavaMail.root@uoguelph.ca> References: <2093433467.6650515.1420514242874.JavaMail.root@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Rick,=0A=0Ai saw that some people has issues with igb cards with NFS= =0AFor example: http://freebsd.1045724.n5.nabble.com/NFS-over-LAGG-lacp-p= oor-performance-td5906349.html =0A=0Acan my problem be related ? I use ig= b with default queue number. Here are my vmstat -i outputs=0A=0AServer si= de:=0A=0Ainterrupt total rate=0Airq1: atkb= d0 18 0=0Airq20: ehci1 = 2790134 2=0Airq21: ehci0 2547642 = 2=0Acpu0:timer 36299188 35=0Airq264: ciss0= 6352476 6=0Airq265: igb0:que 0 = 2716692 2=0Airq266: igb0:que 1 32205278 31= =0Airq267: igb0:que 2 38395109 37=0Airq268: igb0:que= 3 1413468 1=0Airq269: igb0:que 4 392= 07930 38=0Airq270: igb0:que 5 1622715 1=0A= irq271: igb0:que 6 1634676 1=0Airq272: igb0:que 7 = 1190123 1=0Airq273: igb0:link = 2 0=0Acpu1:timer 14074423 13=0Acpu= 8:timer 12204739 11=0Acpu9:timer = 11384192 11=0Acpu3:timer 10461566 = 10=0Acpu4:timer 12785103 12=0Acpu6:t= imer 10739344 10=0Acpu5:timer = 10978294 10=0Acpu7:timer 10599705 = 10=0Acpu2:timer 13998891 13=0Acpu10:tim= er 11602361 11=0Acpu11:timer = 11568523 11=0ATotal 296772592 = 290=0A=0AAnd client side:=0Ainterrupt total = rate=0Airq9: acpi0 4 0=0Airq22: eh= ci1 950519 2=0Airq23: ehci0 = 1865060 4=0Acpu0:timer 248128035 = 546=0Airq268: mfi0 406896 0=0Airq269: igb0= :que 0 2510556 5=0Airq270: igb0:que 1 = 2825336 6=0Airq271: igb0:que 2 2092958 = 4=0Airq272: igb0:que 3 1960849 4=0Airq273: igb0:qu= e 4 2645369 5=0Airq274: igb0:que 5 2= 735187 6=0Airq275: igb0:que 6 2290531 5= =0Airq276: igb0:que 7 2384370 5=0Airq277: igb0:lin= k 2 0=0Airq287: igb2:que 0 14= 65051 3=0Airq288: igb2:que 1 856381 1=0A= irq289: igb2:que 2 809318 1=0Airq290: igb2:que 3 = 897154 1=0Airq291: igb2:que 4 8757= 55 1=0Airq292: igb2:que 5 35866117 78=0Airq= 293: igb2:que 6 846517 1=0Airq294: igb2:que 7 = 857979 1=0Airq295: igb2:link 2 = 0=0Airq296: igb3:que 0 535212 1=0Airq297= : igb3:que 1 454359 1=0Airq298: igb3:que 2 = 454142 1=0Airq299: igb3:que 3 454623 = 1=0Airq300: igb3:que 4 456297 1=0Airq301: i= gb3:que 5 455482 1=0Airq302: igb3:que 6 = 456128 1=0Airq303: igb3:que 7 454680 = 1=0Airq304: igb3:link 3 0=0Airq305: ahci= 0 75 0=0Acpu1:timer = 257233702 566=0Acpu13:timer 255603184 56= 2=0Acpu7:timer 258492826 569=0Acpu12:timer = 255819351 563=0Acpu6:timer 258= 493465 569=0Acpu15:timer 254694003 560= =0Acpu3:timer 258171320 568=0Acpu22:timer = 256506877 564=0Acpu5:timer 2534= 01435 558=0Acpu16:timer 255412360 562=0A= cpu11:timer 257318013 566=0Acpu20:timer = 253648060 558=0Acpu2:timer 2578645= 43 567=0Acpu17:timer 261828899 576=0Acpu= 9:timer 257497326 567=0Acpu18:timer = 258451190 569=0Acpu8:timer 257784504 = 567=0Acpu14:timer 254923723 561=0Acpu10:= timer 257265498 566=0Acpu19:timer = 258775946 569=0Acpu4:timer 256368658 = 564=0Acpu23:timer 255050534 561=0Acpu21:tim= er 257663842 567=0ATotal = 6225260206 13710=0A=0APlease note igb2 on client side is the dedic= ated link for NFSv4=0A=0ARegards,=0A=0ALo=C3=AFc Blot,=0AUNIX Systems, Ne= twork and Security Engineer=0Ahttp://www.unix-experience.fr=0A=0A6 janvie= r 2015 04:17 "Rick Macklem" <rmacklem@uoguelph.ca> a =C3=A9crit: =0A> Loi= c Blot wrote:=0A> =0A>> Hi Rick,=0A>> nfsstat -e -s don't show usefull da= tas on server.=0A> =0A> Well, as far as I know, it returns valid informat= ion.=0A> (See below.)=0A> =0A>> Server Info:=0A>> Getattr Setattr Lookup = Readlink Read Write Create=0A>> Remove=0A>> 26935254 16911 5755728 302 23= 34920 3673866 0=0A>> 328332=0A>> Rename Link Symlink Mkdir Rmdir Readdir = RdirPlus=0A>> Access=0A>> 77980 28 0 0 3 8900 3=0A>> 1806052=0A>> Mknod F= sstat Fsinfo PathConf Commit LookupP SetClId=0A>> SetClIdCf=0A>> 1 1095 0= 0 614377 8172 8=0A>> 8=0A>> Open OpenAttr OpenDwnGr OpenCfrm DelePurge D= eleRet GetFH=0A>> Lock=0A>> 1595299 0 44145 1495 0 0 5197490=0A>> 635015= =0A>> LockT LockU Close Verify NVerify PutFH PutPubFH=0A>> PutRootFH=0A>>= 0 614919 1270938 0 0 22688676 0=0A>> 5=0A>> Renew RestoreFH SaveFH Secin= fo RelLckOwn V4Create=0A>> 42104 197606 275820 0 143 4578=0A>> Server:=0A= >> Retfailed Faults Clients=0A>> 0 0 6=0A>> OpenOwner Opens LockOwner Loc= ks Delegs=0A>> 32335 145448 204 181 0=0A> =0A> Well, 145448 Opens are a l= ot of Open files. Each of these uses=0A> a kernel malloc'd data structure= that is linked into multiple=0A> linked lists.=0A> =0A> The question is.= .why aren't these Opens being closed?=0A> Since FreeBSD does I/O on an mm= ap'd file after closing it,=0A> the FreeBSD NFSv4 client is forced to del= ay doing Close RPCs=0A> until the vnode is VOP_INACTIVE()/VOP_RECLAIM()'d= . (The=0A> VOP_RECLAIM() case is needed, since VOP_INACTIVE() isn't=0A> g= uaranteed to be called.)=0A> =0A> Since there were about 1.5 million Open= s and 1.27 million=0A> Closes, it does appear that Opens are being Closed= .=0A> Now, I'm not sure I would have imagined 1.5million file Opens=0A> i= n a few days. My guess is this is the bottleneck.=0A> =0A> I'd suggest th= at you do:=0A> # nfsstat -e -c=0A> on each of the NFSv4 clients and see h= ow many Opens/client=0A> there are. I vaguely remember an upper limit in = the client,=0A> but can't remember what it is set to.=0A> --> I suspect t= he client Open/Lock limit needs to be increased.=0A> (I can't remember if= the server also has a limit, but I=0A> think it does.)=0A> Then the size= of the hash tables used to search the Opens=0A> may also need to be incr= eased a lot.=0A> =0A> Also, I'd suggest you take a look at whatever apps.= are=0A> running on the client(s) and try to figure out why they=0A> are = Opening so many files?=0A> =0A> My guess is that the client(s) are gettig= bogged down by all=0A> these Opens.=0A> =0A>> Server Cache Stats:=0A>> I= nprog Idem Non-idem Misses CacheSize TCPPeak=0A>> 0 0 1 15082947 60 16522= =0A>> =0A>> Only GetAttr and Lookup increase and it's only every 4-5 seco= nds and=0A>> only +2 to +5 into theses values.=0A>> =0A>> Now on client, = if i take four processes stack i got=0A>> =0A>> PID TID COMM TDNAME KSTAC= K=0A>> 63170 102547 mv - mi_switch+0xe1=0A>> turnstile_wait+0x42a __mtx_l= ock_sleep+0x253 nfscl_nodeleg+0x65=0A>> nfs_lookup+0x3d0 VOP_LOOKUP_APV+0= xa1 lookup+0x59c namei+0x4d4=0A>> vn_open_cred+0x21d kern_openat+0x26f am= d64_syscall+0x351=0A>> Xfast_syscall+0xfb=0A>> =0A>> Another mv:=0A>> 631= 40 101738 mv - mi_switch+0xe1=0A>> turnstile_wait+0x42a __mtx_lock_sleep+= 0x253 nfscl_nodeleg+0x65=0A>> nfs_lookup+0x3d0 VOP_LOOKUP_APV+0xa1 lookup= +0x59c namei+0x4d4=0A>> kern_statat_vnhook+0xae sys_lstat+0x30 amd64_sysc= all+0x351=0A>> Xfast_syscall+0xfb=0A>> =0A>> 62070 102170 sendmail - mi_s= witch+0xe1=0A>> sleepq_timedwait+0x3a _sleep+0x26e clnt_vc_call+0x666=0A>= > clnt_reconnect_call+0x4fa newnfs_request+0xa8c nfscl_request+0x72=0A>> = nfsrpc_lookup+0x1fb nfs_lookup+0x508 VOP_LOOKUP_APV+0xa1=0A>> lookup+0x59= c namei+0x4d4 kern_statat_vnhook+0xae sys_lstat+0x30=0A>> amd64_syscall+0= x351 Xfast_syscall+0xfb=0A>> =0A>> 63200 100930 mv - mi_switch+0xe1=0A>> = turnstile_wait+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65=0A>> nfs_l= ookup+0x3d0 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4=0A>> kern_statat= _vnhook+0xae sys_lstat+0x30 amd64_syscall+0x351=0A>> Xfast_syscall+0xfb= =0A> =0A> The above simply says that thread 102710 is waiting for a Looku= p=0A> reply from the server and the other 3 are waiting for the mutex=0A>= that protects the state structures in the client. (I suspect=0A> some ot= her thread in the client is wading through the Open list,=0A> if a single= client has a lot of these 145K Opens.)=0A> =0A>> When client is in this = state, server was doing nothing special=0A>> (procstat -kk)=0A>> =0A>> PI= D TID COMM TDNAME KSTACK=0A>> 895 100538 nfsd nfsd: master mi_switch+0xe1= =0A>> sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10=0A>> _cv_timedw= ait_sig_sbt+0x18b svc_run_internal+0x4a1 svc_run+0x1de=0A>> nfsrvd_nfsd+0= x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c=0A>> amd64_syscall+0x351 Xfast_sys= call+0xfb=0A>> 895 100568 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_c= atch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe= =0A>> 895 100569 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_sign= als+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 = 100570 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab s= leepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100571 nfs= d nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait= _sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+= 0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100572 nfsd nfsd: se= rvice mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _= cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100573 nfsd nfsd: service mi_s= witch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_si= g+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>> fork_trampoline+0xe=0A>> 895 100575 nfsd nfsd: service mi_switch+0x= e1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a= =0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> for= k_trampoline+0xe=0A>> 895 100576 nfsd nfsd: service mi_switch+0xe1=0A>> s= leepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_= run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoli= ne+0xe=0A>> 895 100577 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catc= h_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>= > 895 100578 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+= 0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 1005= 79 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleep= q_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_= start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100580 nfsd nf= sd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig= +0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100581 nfsd nfsd: servic= e mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_w= ait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>> fork_trampoline+0xe=0A>> 895 100582 nfsd nfsd: service mi_switc= h+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x= 16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> = fork_trampoline+0xe=0A>> 895 100583 nfsd nfsd: service mi_switch+0xe1=0A>= > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> s= vc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_tramp= oline+0xe=0A>> 895 100584 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_c= atch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe= =0A>> 895 100585 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_sign= als+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 = 100586 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab s= leepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100587 nfs= d nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait= _sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+= 0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100588 nfsd nfsd: se= rvice mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _= cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100589 nfsd nfsd: service mi_s= witch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_si= g+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>> fork_trampoline+0xe=0A>> 895 100590 nfsd nfsd: service mi_switch+0x= e1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a= =0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> for= k_trampoline+0xe=0A>> 895 100592 nfsd nfsd: service mi_switch+0xe1=0A>> s= leepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_= run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoli= ne+0xe=0A>> 895 100593 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catc= h_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>= > 895 100594 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+= 0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 1005= 95 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleep= q_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_= start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100596 nfsd nf= sd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig= +0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100597 nfsd nfsd: servic= e mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_w= ait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>> fork_trampoline+0xe=0A>> 895 100598 nfsd nfsd: service mi_switc= h+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x= 16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> = fork_trampoline+0xe=0A>> 895 100599 nfsd nfsd: service mi_switch+0xe1=0A>= > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> s= vc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_tramp= oline+0xe=0A>> 895 100600 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_c= atch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe= =0A>> 895 100602 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_sign= als+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 = 100603 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab s= leepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100604 nfs= d nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait= _sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+= 0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100605 nfsd nfsd: se= rvice mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _= cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100606 nfsd nfsd: service mi_s= witch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_si= g+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>> fork_trampoline+0xe=0A>> 895 100607 nfsd nfsd: service mi_switch+0x= e1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a= =0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> for= k_trampoline+0xe=0A>> 895 100608 nfsd nfsd: service mi_switch+0xe1=0A>> s= leepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_= run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoli= ne+0xe=0A>> 895 100609 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catc= h_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>= > 895 100610 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+= 0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 1006= 11 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleep= q_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_= start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100612 nfsd nf= sd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig= +0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100613 nfsd nfsd: servic= e mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_w= ait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>> fork_trampoline+0xe=0A>> 895 100614 nfsd nfsd: service mi_switc= h+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x= 16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> = fork_trampoline+0xe=0A>> 895 100615 nfsd nfsd: service mi_switch+0xe1=0A>= > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> s= vc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_tramp= oline+0xe=0A>> 895 100617 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_c= atch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe= =0A>> 895 100618 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_sign= als+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 = 100619 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab s= leepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100621 nfs= d nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait= _sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+= 0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100622 nfsd nfsd: se= rvice mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _= cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100623 nfsd nfsd: service mi_s= witch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_si= g+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>> fork_trampoline+0xe=0A>> 895 100624 nfsd nfsd: service mi_switch+0x= e1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a= =0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> for= k_trampoline+0xe=0A>> 895 100625 nfsd nfsd: service mi_switch+0xe1=0A>> s= leepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_= run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoli= ne+0xe=0A>> 895 100626 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catc= h_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>= > 895 100627 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+= 0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 1006= 28 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleep= q_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_= start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100629 nfsd nf= sd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig= +0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100630 nfsd nfsd: servic= e mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_w= ait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>> fork_trampoline+0xe=0A>> 895 100631 nfsd nfsd: service mi_switc= h+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x= 16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> = fork_trampoline+0xe=0A>> 895 100632 nfsd nfsd: service mi_switch+0xe1=0A>= > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> s= vc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_tramp= oline+0xe=0A>> 895 100633 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_c= atch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe= =0A>> 895 100634 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_sign= als+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 = 100635 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab s= leepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100636 nfs= d nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait= _sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+= 0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100638 nfsd nfsd: se= rvice mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _= cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100639 nfsd nfsd: service mi_s= witch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_si= g+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>> fork_trampoline+0xe=0A>> 895 100640 nfsd nfsd: service mi_switch+0x= e1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a= =0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> for= k_trampoline+0xe=0A>> 895 100641 nfsd nfsd: service mi_switch+0xe1=0A>> s= leepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_= run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoli= ne+0xe=0A>> 895 100642 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catc= h_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>= > 895 100643 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+= 0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 1006= 44 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleep= q_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_= start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100645 nfsd nf= sd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig= +0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100646 nfsd nfsd: servic= e mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_w= ait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>> fork_trampoline+0xe=0A>> 895 100647 nfsd nfsd: service mi_switc= h+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x= 16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> = fork_trampoline+0xe=0A>> 895 100648 nfsd nfsd: service mi_switch+0xe1=0A>= > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> s= vc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_tramp= oline+0xe=0A>> 895 100649 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_c= atch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe= =0A>> 895 100651 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_sign= als+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 = 100652 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab s= leepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100653 nfs= d nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait= _sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+= 0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100654 nfsd nfsd: se= rvice mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _= cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100655 nfsd nfsd: service mi_s= witch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_si= g+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>> fork_trampoline+0xe=0A>> 895 100656 nfsd nfsd: service mi_switch+0x= e1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a= =0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> for= k_trampoline+0xe=0A>> 895 100657 nfsd nfsd: service mi_switch+0xe1=0A>> s= leepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_= run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoli= ne+0xe=0A>> 895 100658 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catc= h_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>= > 895 100659 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+= 0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 1006= 61 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleep= q_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_= start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100662 nfsd nf= sd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig= +0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100684 nfsd nfsd: servic= e mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_w= ait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>> fork_trampoline+0xe=0A>> 895 100685 nfsd nfsd: service mi_switc= h+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x= 16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> = fork_trampoline+0xe=0A>> 895 100686 nfsd nfsd: service mi_switch+0xe1=0A>= > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> s= vc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_tramp= oline+0xe=0A>> 895 100797 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_c= atch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe= =0A>> 895 100798 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_sign= als+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 = 100799 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab s= leepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100800 nfs= d nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait= _sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+= 0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 895 100801 nfsd nfsd: se= rvice mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _= cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>> fork_trampoline+0xe=0A>> =0A>> I really think it's a client= side problem, maybe a lookup problem.=0A>> =0A>> Regards,=0A>> =0A>> Lo= =C3=AFc Blot,=0A>> UNIX Systems, Network and Security Engineer=0A>> http:= //www.unix-experience.fr=0A>> =0A>> 5 janvier 2015 14:35 "Rick Macklem" <= rmacklem@uoguelph.ca> a =C3=A9crit:=0A>>> Loic Blot wrote:=0A>>> =0A>>>> = Hi,=0A>>>> happy new year Rick and @freebsd-fs.=0A>>>> =0A>>>> After some= days, i looked my NFSv4.1 mount. At server start it was=0A>>>> calm, but= after 4 days, here is the top stat...=0A>>>> =0A>>>> CPU: 0.0% user, 0.0= % nice, 100% system, 0.0% interrupt, 0.0%=0A>>>> idle=0A>>>> =0A>>>> Defi= nitively i think it's a problem on client side. What can i=0A>>>> look=0A= >>>> into running kernel to resolve this issue ?=0A>>> =0A>>> Well, I'd s= tart with:=0A>>> # nfsstat -e -s=0A>>> - run repeatedly on the server (on= ce every N seconds in a loop).=0A>>> Then look at the output, comparing t= he counts and see which RPCs=0A>>> are being performed by the client(s). = You are looking for which=0A>>> RPCs are being done a lot. (If one RPC is= almost 100% of the load,=0A>>> then it might be a client/caching issue f= or whatever that RPC is=0A>>> doing.)=0A>>> =0A>>> Also look at the Open/= Lock counts near the end of the output.=0A>>> If the # of Opens/Locks is = large, it may be possible to reduce the=0A>>> CPU overheads by using larg= er hash tables.=0A>>> =0A>>> Then you need to profile the server kernel t= o see where the CPU=0A>>> is being used.=0A>>> Hopefully someone else can= fill you in on how to do that, because=0A>>> I'll admit I don't know how= to.=0A>>> Basically you are looking to see if the CPU is being used in= =0A>>> the NFS server code or ZFS.=0A>>> =0A>>> Good luck with it, rick= =0A>>> =0A>>>> Regards,=0A>>>> =0A>>>> Lo=C3=AFc Blot,=0A>>>> UNIX System= s, Network and Security Engineer=0A>>>> http://www.unix-experience.fr=0A>= >>> =0A>>>> 30 d=C3=A9cembre 2014 16:16 "Lo=C3=AFc Blot" <loic.blot@unix-= experience.fr>=0A>>>> a=0A>>>> =C3=A9crit:=0A>>>>> Hi Rick,=0A>>>>> i upg= raded my jail host from FreeBSD 9.3 to 10.1 to use NFS v4.1=0A>>>>> (moun= toptions:=0A>>>>> rw,rsize=3D32768,wsize=3D32768,tcp,nfsv4,minorversion= =3D1)=0A>>>>> =0A>>>>> Performance is quite stable but it's slow. Not as = slow as before=0A>>>>> but slow... services was launched=0A>>>>> but no c= lient are using them and system CPU % was 10-50%.=0A>>>>> =0A>>>>> I don'= t see anything on NFSv4.1 server, it's perfectly stable and=0A>>>>> funct= ionnal.=0A>>>>> =0A>>>>> Regards,=0A>>>>> =0A>>>>> Lo=C3=AFc Blot,=0A>>>>= > UNIX Systems, Network and Security Engineer=0A>>>>> http://www.unix-exp= erience.fr=0A>>>>> =0A>>>>> 23 d=C3=A9cembre 2014 00:20 "Rick Macklem" <r= macklem@uoguelph.ca> a=0A>>>>> =C3=A9crit:=0A>>>>> =0A>>>>>> Loic Blot wr= ote:=0A>>>>>> =0A>>>>>>> Hi,=0A>>>>>>> =0A>>>>>>> To clarify because of o= ur exchanges. Here are the current=0A>>>>>>> sysctl=0A>>>>>>> options for= server:=0A>>>>>>> =0A>>>>>>> vfs.nfsd.enable_nobodycheck=3D0=0A>>>>>>> v= fs.nfsd.enable_nogroupcheck=3D0=0A>>>>>>> =0A>>>>>>> vfs.nfsd.maxthreads= =3D200=0A>>>>>>> vfs.nfsd.tcphighwater=3D10000=0A>>>>>>> vfs.nfsd.tcpcach= etimeo=3D300=0A>>>>>>> vfs.nfsd.server_min_nfsvers=3D4=0A>>>>>>> =0A>>>>>= >> kern.maxvnodes=3D10000000=0A>>>>>>> kern.ipc.maxsockbuf=3D4194304=0A>>= >>>>> net.inet.tcp.sendbuf_max=3D4194304=0A>>>>>>> net.inet.tcp.recvbuf_m= ax=3D4194304=0A>>>>>>> =0A>>>>>>> vfs.lookup_shared=3D0=0A>>>>>>> =0A>>>>= >>> Regards,=0A>>>>>>> =0A>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>> UNIX Systems,= Network and Security Engineer=0A>>>>>>> http://www.unix-experience.fr=0A= >>>>>>> =0A>>>>>>> 22 d=C3=A9cembre 2014 09:42 "Lo=C3=AFc Blot"=0A>>>>>>>= <loic.blot@unix-experience.fr>=0A>>>>>>> a=0A>>>>>>> =C3=A9crit:=0A>>>>>= >> =0A>>>>>>> Hi Rick,=0A>>>>>>> my 5 jails runs this weekend and now i h= ave some stats on this=0A>>>>>>> monday.=0A>>>>>>> =0A>>>>>>> Hopefully d= eadlock was fixed, yeah, but everything isn't good=0A>>>>>>> :(=0A>>>>>>>= =0A>>>>>>> On NFSv4 server (FreeBSD 10.1) system uses 35% CPU=0A>>>>>>> = =0A>>>>>>> As i can see this is because of nfsd:=0A>>>>>>> =0A>>>>>>> 918= root 96 20 0 12352K 3372K rpcsvc 6 51.4H=0A>>>>>>> 273.68% nfsd: server = (nfsd)=0A>>>>>>> =0A>>>>>>> If i look at dmesg i see:=0A>>>>>>> nfsd serv= er cache flooded, try increasing vfs.nfsd.tcphighwater=0A>>>>>> =0A>>>>>>= Well, you have a couple of choices:=0A>>>>>> 1 - Use NFSv4.1 (add "minor= version=3D1" to your mount options).=0A>>>>>> (NFSv4.1 avoids use of the = DRC and instead uses something=0A>>>>>> called sessions. See below.)=0A>>= >>>> OR=0A>>>>>> =0A>>>>>>> vfs.nfsd.tcphighwater was set to 10000, i inc= rease it to 15000=0A>>>>>> =0A>>>>>> 2 - Bump vfs.nfsd.tcphighwater way u= p, until you no longer see=0A>>>>>> "nfs server cache flooded" messages. = (I think Garrett Wollman=0A>>>>>> uses=0A>>>>>> 100000. (You may still se= e quite a bit of CPU overheads.)=0A>>>>>> =0A>>>>>> OR=0A>>>>>> =0A>>>>>>= 3 - Set vfs.nfsd.cachetcp=3D0 (which disables the DRC and gets rid=0A>>>= >>> of the CPU overheads). However, there is a risk of data=0A>>>>>> corr= uption=0A>>>>>> if you have a client->server network partitioning of a mo= derate=0A>>>>>> duration, because a non-idempotent RPC may get redone, be= casue=0A>>>>>> the client times out waiting for a reply. If a non-idempot= ent=0A>>>>>> RPC gets done twice on the server, data corruption can happe= n.=0A>>>>>> (The DRC provides improved correctness, but does add overhead= .)=0A>>>>>> =0A>>>>>> If #1 works for you, it is the preferred solution, = since=0A>>>>>> Sessions=0A>>>>>> in NFSv4.1 solves the correctness proble= m in a good, space bound=0A>>>>>> way. A session basically has N (usually= 32 or 64) slots and only=0A>>>>>> allows one outstanding RPC/slot. As su= ch, it can cache the=0A>>>>>> previous=0A>>>>>> reply for each slot (32 o= r 64 of them) and guarantee "exactly=0A>>>>>> once"=0A>>>>>> RPC semantic= s.=0A>>>>>> =0A>>>>>> rick=0A>>>>>> =0A>>>>>>> Here is 'nfsstat -s' outpu= t:=0A>>>>>>> =0A>>>>>>> Server Info:=0A>>>>>>> Getattr Setattr Lookup Rea= dlink Read Write Create=0A>>>>>>> Remove=0A>>>>>>> 12600652 1812 2501097 = 156 1386423 1983729 123=0A>>>>>>> 162067=0A>>>>>>> Rename Link Symlink Mk= dir Rmdir Readdir RdirPlus=0A>>>>>>> Access=0A>>>>>>> 36762 9 0 0 0 3147 = 0=0A>>>>>>> 623524=0A>>>>>>> Mknod Fsstat Fsinfo PathConf Commit=0A>>>>>>= > 0 0 0 0 328117=0A>>>>>>> Server Ret-Failed=0A>>>>>>> 0=0A>>>>>>> Server= Faults=0A>>>>>>> 0=0A>>>>>>> Server Cache Stats:=0A>>>>>>> Inprog Idem N= on-idem Misses=0A>>>>>>> 0 0 0 12635512=0A>>>>>>> Server Write Gathering:= =0A>>>>>>> WriteOps WriteRPC Opsaved=0A>>>>>>> 1983729 1983729 0=0A>>>>>>= > =0A>>>>>>> And here is 'procstat -kk' for nfsd (server)=0A>>>>>>> =0A>>= >>>>> 918 100528 nfsd nfsd: master mi_switch+0xe1=0A>>>>>>> sleepq_catch_= signals+0xab sleepq_timedwait_sig+0x10=0A>>>>>>> _cv_timedwait_sig_sbt+0x= 18b svc_run_internal+0x4a1=0A>>>>>>> svc_run+0x1de=0A>>>>>>> nfsrvd_nfsd+= 0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c=0A>>>>>>> amd64_syscall+0x351 Xfa= st_syscall+0xfb=0A>>>>>>> 918 100568 nfsd nfsd: service mi_switch+0xe1=0A= >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_= sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100569 nfsd nfsd: servi= ce mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100= 570 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0= xe=0A>>>>>>> 918 100571 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>= >>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>= >> fork_trampoline+0xe=0A>>>>>>> 918 100572 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _c= v_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fo= rk_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100573 nfsd nfsd= : service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_= sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc= _thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> = 918 100574 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signa= ls+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_ru= n_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_tramp= oline+0xe=0A>>>>>>> 918 100575 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>= > sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x= 16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100576 nfsd nfsd: service mi= _switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>= >>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_star= t+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100577 n= fsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab slee= pq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0= x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A= >>>>>>> 918 100578 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_cat= ch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>= > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fo= rk_trampoline+0xe=0A>>>>>>> 918 100579 nfsd nfsd: service mi_switch+0xe1= =0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wa= it_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100580 nfsd nfsd: se= rvice mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 = 100581 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_in= ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampolin= e+0xe=0A>>>>>>> 918 100582 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sl= eepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a= =0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>= >>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100583 nfsd nfsd: service mi_swi= tch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>= > _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0x= b fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100584 nfsd = nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_w= ait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e= svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>= >>> 918 100585 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_s= ignals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> sv= c_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_t= rampoline+0xe=0A>>>>>>> 918 100586 nfsd nfsd: service mi_switch+0xe1=0A>>= >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_si= g+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0= x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100587 nfsd nfsd: service= mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100= 588 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0= xe=0A>>>>>>> 918 100589 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>= >>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>= >> fork_trampoline+0xe=0A>>>>>>> 918 100590 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _c= v_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fo= rk_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100591 nfsd nfsd= : service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_= sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc= _thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> = 918 100592 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signa= ls+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_ru= n_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_tramp= oline+0xe=0A>>>>>>> 918 100593 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>= > sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x= 16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100594 nfsd nfsd: service mi= _switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>= >>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_star= t+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100595 n= fsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab slee= pq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0= x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A= >>>>>>> 918 100596 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_cat= ch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>= > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fo= rk_trampoline+0xe=0A>>>>>>> 918 100597 nfsd nfsd: service mi_switch+0xe1= =0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wa= it_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100598 nfsd nfsd: se= rvice mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 = 100599 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_in= ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampolin= e+0xe=0A>>>>>>> 918 100600 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sl= eepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a= =0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>= >>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100601 nfsd nfsd: service mi_swi= tch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>= > _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0x= b fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100602 nfsd = nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_w= ait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e= svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>= >>> 918 100603 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_s= ignals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> sv= c_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_t= rampoline+0xe=0A>>>>>>> 918 100604 nfsd nfsd: service mi_switch+0xe1=0A>>= >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_si= g+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0= x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100605 nfsd nfsd: service= mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100= 606 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0= xe=0A>>>>>>> 918 100607 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>= >>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>= >> fork_trampoline+0xe=0A>>>>>>> 918 100608 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _c= v_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fo= rk_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100609 nfsd nfsd= : service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_= sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc= _thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> = 918 100610 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signa= ls+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_ru= n_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_tramp= oline+0xe=0A>>>>>>> 918 100611 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>= > sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x= 16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100612 nfsd nfsd: service mi= _switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>= >>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_star= t+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100613 n= fsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab slee= pq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0= x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A= >>>>>>> 918 100614 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_cat= ch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>= > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fo= rk_trampoline+0xe=0A>>>>>>> 918 100615 nfsd nfsd: service mi_switch+0xe1= =0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wa= it_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100616 nfsd nfsd: se= rvice mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 = 100617 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_in= ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampolin= e+0xe=0A>>>>>>> 918 100618 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sl= eepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a= =0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>= >>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100619 nfsd nfsd: service mi_swi= tch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>= > _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0x= b fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100620 nfsd = nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_w= ait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e= svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>= >>> 918 100621 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_s= ignals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> sv= c_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_t= rampoline+0xe=0A>>>>>>> 918 100622 nfsd nfsd: service mi_switch+0xe1=0A>>= >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_si= g+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0= x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100623 nfsd nfsd: service= mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100= 624 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0= xe=0A>>>>>>> 918 100625 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>= >>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>= >> fork_trampoline+0xe=0A>>>>>>> 918 100626 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _c= v_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fo= rk_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100627 nfsd nfsd= : service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_= sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc= _thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> = 918 100628 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signa= ls+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_ru= n_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_tramp= oline+0xe=0A>>>>>>> 918 100629 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>= > sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x= 16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100630 nfsd nfsd: service mi= _switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>= >>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_star= t+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100631 n= fsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab slee= pq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0= x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A= >>>>>>> 918 100632 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_cat= ch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>= > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fo= rk_trampoline+0xe=0A>>>>>>> 918 100633 nfsd nfsd: service mi_switch+0xe1= =0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wa= it_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100634 nfsd nfsd: se= rvice mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 = 100635 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_in= ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampolin= e+0xe=0A>>>>>>> 918 100636 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sl= eepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a= =0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>= >>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100637 nfsd nfsd: service mi_swi= tch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>= > _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0x= b fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100638 nfsd = nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_w= ait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e= svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>= >>> 918 100639 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_s= ignals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> sv= c_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_t= rampoline+0xe=0A>>>>>>> 918 100640 nfsd nfsd: service mi_switch+0xe1=0A>>= >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_si= g+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0= x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100641 nfsd nfsd: service= mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100= 642 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0= xe=0A>>>>>>> 918 100643 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>= >>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>= >> fork_trampoline+0xe=0A>>>>>>> 918 100644 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _c= v_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fo= rk_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100645 nfsd nfsd= : service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_= sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc= _thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> = 918 100646 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signa= ls+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_ru= n_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_tramp= oline+0xe=0A>>>>>>> 918 100647 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>= > sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x= 16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100648 nfsd nfsd: service mi= _switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>= >>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_star= t+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100649 n= fsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab slee= pq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0= x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A= >>>>>>> 918 100650 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_cat= ch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>= > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fo= rk_trampoline+0xe=0A>>>>>>> 918 100651 nfsd nfsd: service mi_switch+0xe1= =0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wa= it_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100652 nfsd nfsd: se= rvice mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 = 100653 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_in= ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampolin= e+0xe=0A>>>>>>> 918 100654 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sl= eepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a= =0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>= >>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100655 nfsd nfsd: service mi_swi= tch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>= > _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0x= b fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100656 nfsd = nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_w= ait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e= svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>= >>> 918 100657 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_s= ignals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> sv= c_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_t= rampoline+0xe=0A>>>>>>> 918 100658 nfsd nfsd: service mi_switch+0xe1=0A>>= >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_si= g+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0= x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100659 nfsd nfsd: service= mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 918 100= 660 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0= xe=0A>>>>>>> 918 100661 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>= >>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>= >> fork_trampoline+0xe=0A>>>>>>> 918 100662 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _c= v_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fo= rk_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> ---=0A>>>>>>> =0A>>>= >>>> Now if we look at client (FreeBSD 9.3)=0A>>>>>>> =0A>>>>>>> We see s= ystem was very busy and do many and many interrupts=0A>>>>>>> =0A>>>>>>> = CPU: 0.0% user, 0.0% nice, 37.8% system, 51.2% interrupt, 11.0%=0A>>>>>>>= idle=0A>>>>>>> =0A>>>>>>> A look at process list shows that there are ma= ny sendmail=0A>>>>>>> process=0A>>>>>>> in=0A>>>>>>> state nfstry=0A>>>>>= >> =0A>>>>>>> nfstry 18 32:27 0.88% sendmail: Queue runner@00:30:00 for= =0A>>>>>>> /var/spool/clientm=0A>>>>>>> =0A>>>>>>> Here is 'nfsstat -c' o= utput:=0A>>>>>>> =0A>>>>>>> Client Info:=0A>>>>>>> Rpc Counts:=0A>>>>>>> = Getattr Setattr Lookup Readlink Read Write Create=0A>>>>>>> Remove=0A>>>>= >>> 1051347 1724 2494481 118=20903902 1901285 162676=0A>>>>>>> 161899=0A>= >>>>>> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus=0A>>>>>>> Access= =0A>>>>>>> 36744 2 0 114 40 3131 0=0A>>>>>>> 544136=0A>>>>>>> Mknod Fssta= t Fsinfo PathConf Commit=0A>>>>>>> 9 0 0 0 245821=0A>>>>>>> Rpc Info:=0A>= >>>>>> TimedOut Invalid X Replies Retries Requests=0A>>>>>>> 0 0 0 0 8356= 557=0A>>>>>>> Cache Info:=0A>>>>>>> Attr Hits Misses Lkup Hits Misses Bio= R Hits Misses BioW Hits=0A>>>>>>> Misses=0A>>>>>>> 108754455 491475 54229= 224 2437229 46814561 821723 5132123=0A>>>>>>> 1871871=0A>>>>>>> BioRLHits= Misses BioD Hits Misses DirE Hits Misses Accs Hits=0A>>>>>>> Misses=0A>>= >>>>> 144035 118 53736 2753 27813 1 57238839=0A>>>>>>> 544205=0A>>>>>>> = =0A>>>>>>> If you need more things, tell me, i let the PoC in this state.= =0A>>>>>>> =0A>>>>>>> Thanks=0A>>>>>>> =0A>>>>>>> Regards,=0A>>>>>>> =0A>= >>>>>> Lo=C3=AFc Blot,=0A>>>>>>> UNIX Systems, Network and Security Engin= eer=0A>>>>>>> http://www.unix-experience.fr=0A>>>>>>>; =0A>>>>>>> 21 d=C3= =A9cembre 2014 01:33 "Rick Macklem" <rmacklem@uoguelph.ca> a=0A>>>>>>> = =C3=A9crit:=0A>>>>>>> =0A>>>>>>> Loic Blot wrote:=0A>>>>>>> =0A>>>>>>>> H= i Rick,=0A>>>>>>>> ok, i don't need locallocks, i haven't understand opti= on was=0A>>>>>>>> for=0A>>>>>>>> that=0A>>>>>>>> usage, i removed it.=0A>= >>>>>>> I do more tests on monday.=0A>>>>>>>> Thanks for the deadlock fix= , for other people :)=0A>>>>>>> =0A>>>>>>> Good. Please let us know if ru= nning with=0A>>>>>>> vfs.nfsd.enable_locallocks=3D0=0A>>>>>>> gets rid of= the deadlocks? (I think it fixes the one you saw.)=0A>>>>>>> =0A>>>>>>> = On the performance side, you might also want to try different=0A>>>>>>> v= alues=0A>>>>>>> of=0A>>>>>>> readahead, if the Linux client has such a mo= unt option. (With=0A>>>>>>> the=0A>>>>>>> NFSv4-ZFS sequential vs random = I/O heuristic, I have no idea=0A>>>>>>> what=0A>>>>>>> the=0A>>>>>>> opti= mal readahead value would be.)=0A>>>>>>> =0A>>>>>>> Good luck with it and= please let us know how it goes, rick=0A>>>>>>> ps: I now have a patch to= fix the deadlock when=0A>>>>>>> vfs.nfsd.enable_locallocks=3D1=0A>>>>>>>= is set. I'll post it for anyone who is interested after I put=0A>>>>>>> = it=0A>>>>>>> through some testing.=0A>>>>>>> =0A>>>>>>> --=0A>>>>>>> Best= regards,=0A>>>>>>> Lo=C3=AFc BLOT,=0A>>>>>>> UNIX systems, security and = network engineer=0A>>>>>>> http://www.unix-experience.fr=0A>>>>>>>; =0A>>>= >>>> Le jeudi 18 d=C3=A9cembre 2014 =C3=A0 19:46 -0500, Rick Macklem a = =C3=A9crit :=0A>>>>>>> =0A>>>>>>> Loic Blot wrote:=0A>>>>>>>> Hi rick,=0A= >>>>>>>> i tried to start a LXC container on Debian Squeeze from my=0A>>>= >>>>> freebsd=0A>>>>>>>> ZFS+NFSv4 server and i also have a deadlock on n= fsd=0A>>>>>>>> (vfs.lookup_shared=3D0). Deadlock procs each time i launch= a=0A>>>>>>>> squeeze=0A>>>>>>>> container, it seems (3 tries, 3 fails).= =0A>>>>>>> =0A>>>>>>> Well, I`ll take a look at this `procstat -kk`, but = the only=0A>>>>>>> thing=0A>>>>>>> I`ve seen posted w.r.t. avoiding deadl= ocks in ZFS is to not use=0A>>>>>>> nullfs. (I have no idea if you are us= ing any nullfs mounts, but=0A>>>>>>> if so, try getting rid of them.)=0A>= >>>>>> =0A>>>>>>> Here`s a high level post about the ZFS and vnode lockin= g=0A>>>>>>> problem,=0A>>>>>>> but there is no patch available, as far as= I know.=0A>>>>>>> =0A>>>>>>> http://docs.FreeBSD.org/cgi/mid.cgi?54739F4= 1.8030407=0A>>>>>>> =0A>>>>>>> rick=0A>>>>>>> =0A>>>>>>> 921 - D 0:00.02 = nfsd: server (nfsd)=0A>>>>>>> =0A>>>>>>> Here is the procstat -kk=0A>>>>>= >> =0A>>>>>>> PID TID COMM TDNAME KSTACK=0A>>>>>>> 921 100538 nfsd nfsd: = master mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_= args+0xc9e=0A>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A= >>>>>>> nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad=0A>= >>>>>> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554=0A>>>>>= >> svc_run_internal+0xc77 svc_run+0x1de nfsrvd_nfsd+0x1ca=0A>>>>>>> nfssv= c_nfsd+0x107 sys_nfssvc+0x9c=0A>>>>>>> 921 100572 nfsd nfsd: service mi_s= witch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>= >>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+= 0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100573 nfs= d nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq= _wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>= >>>>> 921 100574 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch= _signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> = svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork= _trampoline+0xe=0A>>>>>>> 921 100575 nfsd nfsd: service mi_switch+0xe1=0A= >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_= sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100576 nfsd nfsd: servi= ce mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100= 577 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0= xe=0A>>>>>>> 921 100578 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>= >>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>= >> fork_trampoline+0xe=0A>>>>>>> 921 100579 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _c= v_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fo= rk_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100580 nfsd nfsd= : service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_= sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc= _thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> = 921 100581 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signa= ls+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_ru= n_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_tramp= oline+0xe=0A>>>>>>> 921 100582 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>= > sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x= 16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100583 nfsd nfsd: service mi= _switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>= >>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_star= t+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100584 n= fsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab slee= pq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0= x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A= >>>>>>> 921 100585 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_cat= ch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>= > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fo= rk_trampoline+0xe=0A>>>>>>> 921 100586 nfsd nfsd: service mi_switch+0xe1= =0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wa= it_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100587 nfsd nfsd: se= rvice mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 = 100588 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_in= ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampolin= e+0xe=0A>>>>>>> 921 100589 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sl= eepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a= =0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>= >>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100590 nfsd nfsd: service mi_swi= tch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>= > _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0x= b fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100591 nfsd = nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_w= ait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e= svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>= >>> 921 100592 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_s= ignals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> sv= c_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_t= rampoline+0xe=0A>>>>>>> 921 100593 nfsd nfsd: service mi_switch+0xe1=0A>>= >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_si= g+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0= x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100594 nfsd nfsd: service= mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100= 595 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0= xe=0A>>>>>>> 921 100596 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>= >>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>= >> fork_trampoline+0xe=0A>>>>>>> 921 100597 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _c= v_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fo= rk_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100598 nfsd nfsd= : service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_= sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc= _thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> = 921 100599 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signa= ls+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_ru= n_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_tramp= oline+0xe=0A>>>>>>> 921 100600 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>= > sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x= 16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100601 nfsd nfsd: service mi= _switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>= >>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_star= t+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100602 n= fsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab slee= pq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0= x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A= >>>>>>> 921 100603 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_cat= ch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>= > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fo= rk_trampoline+0xe=0A>>>>>>> 921 100604 nfsd nfsd: service mi_switch+0xe1= =0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wa= it_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100605 nfsd nfsd: se= rvice mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 = 100606 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_in= ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampolin= e+0xe=0A>>>>>>> 921 100607 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sl= eepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a= =0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>= >>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100608 nfsd nfsd: service mi_swi= tch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>= > _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0x= b fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100609 nfsd = nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_w= ait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e= svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>= >>> 921 100610 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_s= ignals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> sv= c_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_t= rampoline+0xe=0A>>>>>>> 921 100611 nfsd nfsd: service mi_switch+0xe1=0A>>= >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_si= g+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0= x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100612 nfsd nfsd: service= mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100= 613 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0= xe=0A>>>>>>> 921 100614 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>= >>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>= >> fork_trampoline+0xe=0A>>>>>>> 921 100615 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _c= v_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fo= rk_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100616 nfsd nfsd= : service mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmslee= p+0x66 nfsv4_lock+0x9b=0A>>>>>>> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0= x21f nfsrvd_lock+0x5b1=0A>>>>>>> nfsrvd_dorpc+0xec6 nfssvc_program+0x554 = svc_run_internal+0xc77=0A>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork= _trampoline+0xe=0A>>>>>>> 921 100617 nfsd nfsd: service mi_switch+0xe1=0A= >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_= sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100618 nfsd nfsd: servi= ce mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 = nfsv4_lock+0x9b=0A>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run= _internal+0xc77=0A>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampo= line+0xe=0A>>>>>>> 921 100619 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>= sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x1= 6a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100620 nfsd nfsd: service mi= _switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>= >>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_star= t+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100621 n= fsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab slee= pq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0= x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A= >>>>>>> 921 100622 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_cat= ch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>= > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fo= rk_trampoline+0xe=0A>>>>>>> 921 100623 nfsd nfsd: service mi_switch+0xe1= =0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wa= it_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100624 nfsd nfsd: se= rvice mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 = 100625 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_in= ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampolin= e+0xe=0A>>>>>>> 921 100626 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sl= eepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a= =0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>= >>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100627 nfsd nfsd: service mi_swi= tch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>= > _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0x= b fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100628 nfsd = nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_w= ait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e= svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>= >>> 921 100629 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_s= ignals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> sv= c_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_t= rampoline+0xe=0A>>>>>>> 921 100630 nfsd nfsd: service mi_switch+0xe1=0A>>= >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_si= g+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0= x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100631 nfsd nfsd: service= mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100= 632 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0= xe=0A>>>>>>> 921 100633 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>= >>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>= >> fork_trampoline+0xe=0A>>>>>>> 921 100634 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _c= v_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fo= rk_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100635 nfsd nfsd= : service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_= sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc= _thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> = 921 100636 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signa= ls+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_ru= n_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_tramp= oline+0xe=0A>>>>>>> 921 100637 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>= > sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x= 16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100638 nfsd nfsd: service mi= _switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>= >>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_star= t+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100639 n= fsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab slee= pq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0= x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A= >>>>>>> 921 100640 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_cat= ch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>= > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fo= rk_trampoline+0xe=0A>>>>>>> 921 100641 nfsd nfsd: service mi_switch+0xe1= =0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wa= it_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100642 nfsd nfsd: se= rvice mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 = 100643 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_in= ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampolin= e+0xe=0A>>>>>>> 921 100644 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sl= eepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a= =0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>= >>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100645 nfsd nfsd: service mi_swi= tch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>= > _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0x= b fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100646 nfsd = nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_w= ait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e= svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>= >>> 921 100647 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_s= ignals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> sv= c_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_t= rampoline+0xe=0A>>>>>>> 921 100648 nfsd nfsd: service mi_switch+0xe1=0A>>= >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_si= g+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0= x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100649 nfsd nfsd: service= mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100= 650 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0= xe=0A>>>>>>> 921 100651 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>= >>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>= >> fork_trampoline+0xe=0A>>>>>>> 921 100652 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _c= v_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fo= rk_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100653 nfsd nfsd= : service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_= sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc= _thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> = 921 100654 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signa= ls+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_ru= n_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_tramp= oline+0xe=0A>>>>>>> 921 100655 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>= > sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x= 16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100656 nfsd nfsd: service mi= _switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>= >>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_star= t+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100657 n= fsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab slee= pq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0= x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A= >>>>>>> 921 100658 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_cat= ch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>= > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fo= rk_trampoline+0xe=0A>>>>>>> 921 100659 nfsd nfsd: service mi_switch+0xe1= =0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wa= it_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100660 nfsd nfsd: se= rvice mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 = 100661 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_in= ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampolin= e+0xe=0A>>>>>>> 921 100662 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sl= eepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a= =0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>= >>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100663 nfsd nfsd: service mi_swi= tch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>= > _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e svc_thread_start+0x= b fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>>> 921 100664 nfsd = nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_signals+0xab sleepq_w= ait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> svc_run_internal+0x87e= svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>= >>> 921 100665 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_catch_s= ignals+0xab sleepq_wait_sig+0xf=0A>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>> sv= c_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_t= rampoline+0xe=0A>>>>>>> 921 100666 nfsd nfsd: service mi_switch+0xe1=0A>>= >>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>= >>> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8=0A>>>>>>> nfsrvd_dorpc+= 0xc76=0A>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>> svc= _thread_start+0xb=0A>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>> = =0A>>>>>>> Regards,=0A>>>>>>> =0A>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>> UNIX S= ystems, Network and Security Engineer=0A>>>>>>> http://www.unix-experienc= e.fr=0A>>>>>>> =0A>>>>>>> 15 d=C3=A9cembre 2014 15:18 "Rick Macklem" <rma= cklem@uoguelph.ca> a=0A>>>>>>> =C3=A9crit:=0A>>>>>>> =0A>>>>>>> Loic Blot= wrote:=0A>>>>>>> =0A>>>>>>>> For more informations, here is procstat -kk= on nfsd, if you=0A>>>>>>>> need=0A>>>>>>>> more=0A>>>>>>>> hot datas, te= ll me.=0A>>>>>>>> =0A>>>>>>>> Regards, PID TID COMM TDNAME KSTACK=0A>>>>>= >>> 918 100529 nfsd nfsd: master mi_switch+0xe1=0A>>>>>>>> sleepq_wait+0x= 3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>> vop_stdlock+0x3c VOP_LOC= K1_APV+0xab _vn_lock+0x43=0A>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>> nfsvno_f= htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>> nfssvc_program+= 0x554 svc_run_internal+0xc77 svc_run+0x1de=0A>>>>>>>> nfsrvd_nfsd+0x1ca n= fssvc_nfsd+0x107 sys_nfssvc+0x9c=0A>>>>>>>> amd64_syscall+0x351=0A>>>>>>>= =0A>>>>>>> Well, most of the threads are stuck like this one, waiting fo= r=0A>>>>>>> a=0A>>>>>>> vnode=0A>>>>>>> lock in ZFS. All of them appear t= o be in zfs_fhtovp().=0A>>>>>>> I`m not a ZFS guy, so I can`t help much. = I`ll try changing the=0A>>>>>>> subject line=0A>>>>>>> to include ZFS vno= de lock, so maybe the ZFS guys will take a=0A>>>>>>> look.=0A>>>>>>> =0A>= >>>>>> The only thing I`ve seen suggested is trying:=0A>>>>>>> sysctl vfs= .lookup_shared=3D0=0A>>>>>>> to disable shared vop_lookup()s. Apparently = zfs_lookup()=0A>>>>>>> doesn`t=0A>>>>>>> obey the vnode locking rules for= lookup and rename, according=0A>>>>>>> to=0A>>>>>>> the posting I saw.= =0A>>>>>>> =0A>>>>>>> I`ve added a couple of comments about the other thr= eads below,=0A>>>>>>> but=0A>>>>>>> they are all either waiting for an RP= C request or waiting for=0A>>>>>>> the=0A>>>>>>> threads stuck on the ZFS= vnode lock to complete.=0A>>>>>>> =0A>>>>>>> rick=0A>>>>>>> =0A>>>>>>>> = 918 100564 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>> sleepq_catch_sign= als+0xab sleepq_wait_sig+0xf=0A>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>> fork_t= rampoline+0xe=0A>>>>>>> =0A>>>>>>> Fyi, this thread is just waiting for a= n RPC to arrive. (Normal)=0A>>>>>>> =0A>>>>>>>> 918 100565 nfsd nfsd: ser= vice mi_switch+0xe1=0A>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf=0A>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>> svc_run_internal+0x87e svc_t= hread_start+0xb fork_exit+0x9a=0A>>>>>>>> fork_trampoline+0xe=0A>>>>>>>> = 918 100566 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>> sleepq_catch_sign= als+0xab sleepq_wait_sig+0xf=0A>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>> fork_t= rampoline+0xe=0A>>>>>>>> 918 100567 nfsd nfsd: service mi_switch+0xe1=0A>= >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>> _cv_wait= _sig+0x16a=0A>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_ex= it+0x9a=0A>>>>>>>> fork_trampoline+0xe=0A>>>>>>>> 918 100568 nfsd nfsd: s= ervice mi_switch+0xe1=0A>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_si= g+0xf=0A>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>> svc_run_internal+0x87e svc= _thread_start+0xb fork_exit+0x9a=0A>>>>>>>> fork_trampoline+0xe=0A>>>>>>>= > 918 100569 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>> sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf=0A>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>> s= vc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>> fork= _trampoline+0xe=0A>>>>>>>> 918 100570 nfsd nfsd: service mi_switch+0xe1= =0A>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>> _cv_= wait_sig+0x16a=0A>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb for= k_exit+0x9a=0A>>>>>>>> fork_trampoline+0xe=0A>>>>>>>> 918 100571 nfsd nfs= d: service mi_switch+0xe1=0A>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsl= eep+0x66 nfsv4_lock+0x9b=0A>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x5= 54 svc_run_internal+0xc77=0A>>>>>>>> svc_thread_start+0xb fork_exit+0x9a = fork_trampoline+0xe=0A>>>>>>>> 918 100572 nfsd nfsd: service mi_switch+0x= e1=0A>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9= b=0A>>>>>>>> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8=0A>>>>>>>> nfs= rvd_dorpc+0xc76=0A>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A= >>>>>>>> svc_thread_start+0xb=0A>>>>>>>> fork_exit+0x9a fork_trampoline+0= xe=0A>>>>>>> =0A>>>>>>> This one (and a few others) are waiting for the n= fsv4_lock.=0A>>>>>>> This=0A>>>>>>> happens=0A>>>>>>> because other threa= ds are stuck with RPCs in progress. (ie. The=0A>>>>>>> ones=0A>>>>>>> wai= ting on the vnode lock in zfs_fhtovp().)=0A>>>>>>> For these, the RPC nee= ds to lock out other threads to do the=0A>>>>>>> operation,=0A>>>>>>> so = it waits for the nfsv4_lock() which can exclusively lock the=0A>>>>>>> NF= Sv4=0A>>>>>>> data structures once all other nfsd threads complete their = RPCs=0A>>>>>>> in=0A>>>>>>> progress.=0A>>>>>>> =0A>>>>>>>> 918 100573 nf= sd nfsd: service mi_switch+0xe1=0A>>>>>>>> sleepq_wait+0x3a _sleep+0x287 = nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_progr= am+0x554 svc_run_internal+0xc77=0A>>>>>>>> svc_thread_start+0xb fork_exit= +0x9a fork_trampoline+0xe=0A>>>>>>> =0A>>>>>>> Same as above.=0A>>>>>>> = =0A>>>>>>>> 918 100574 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>> sleep= q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>> vop_stdlock+0x3= c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>= nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>> nfssvc= _program+0x554 svc_run_internal+0xc77=0A>>>>>>>> svc_thread_start+0xb=0A>= >>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>> 918 100575 nfsd nfs= d: service mi_switch+0xe1=0A>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __loc= kmgr_args+0x902=0A>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0= x43=0A>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp= +0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>> nfssvc_program+0x554 svc_run_internal= +0xc77=0A>>>>>>>> svc_thread_start+0xb=0A>>>>>>>> fork_exit+0x9a fork_tra= mpoline+0xe=0A>>>>>>>> 918 100576 nfsd nfsd: service mi_switch+0xe1=0A>>>= >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>> vop_= stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>> zfs_fhtovp+0x38d= =0A>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>= >>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>> svc_thread_st= art+0xb=0A>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>> 918 1005= 77 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>> sleepq_wait+0x3a sleeplk+= 0x15d __lockmgr_args+0x902=0A>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab= _vn_lock+0x43=0A>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>> nfsvno_fhtovp+0x7c = nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>> nfssvc_program+0x554 svc_r= un_internal+0xc77=0A>>>>>>>> svc_thread_start+0xb=0A>>>>>>>> fork_exit+0x= 9a fork_trampoline+0xe=0A>>>>>>>> 918 100578 nfsd nfsd: service mi_switch= +0xe1=0A>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>= >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>> zfs_f= htovp+0x38d=0A>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0= x917=0A>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>> sv= c_thread_start+0xb=0A>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>= >> 918 100579 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>> sleepq_wait+0x= 3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>> vop_stdlock+0x3c VOP_LOC= K1_APV+0xab _vn_lock+0x43=0A>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>> nfsvno_f= htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>> nfssvc_program+= 0x554 svc_run_internal+0xc77=0A>>>>>>>> svc_thread_start+0xb=0A>>>>>>>> f= ork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>> 918 100580 nfsd nfsd: servic= e mi_switch+0xe1=0A>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args= +0x902=0A>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>= >>>>> zfs_fhtovp+0x38d=0A>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfs= rvd_dorpc+0x917=0A>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A= >>>>>>>> svc_thread_start+0xb=0A>>>>>>>> fork_exit+0x9a fork_trampoline+0= xe=0A>>>>>>>> 918 100581 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>> sle= epq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>> vop_stdlock+0= x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>= >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>> nfss= vc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>> svc_thread_start+0xb= =0A>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>> 918 100582 nfsd= nfsd: service mi_switch+0xe1=0A>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d _= _lockmgr_args+0x902=0A>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lo= ck+0x43=0A>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fh= tovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>> nfssvc_program+0x554 svc_run_inte= rnal+0xc77=0A>>>>>>>> svc_thread_start+0xb=0A>>>>>>>> fork_exit+0x9a fork= _trampoline+0xe=0A>>>>>>>> 918 100583 nfsd nfsd: service mi_switch+0xe1= =0A>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>= > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>> zfs_fhtovp= +0x38d=0A>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917= =0A>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>> svc_th= read_start+0xb=0A>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>> 9= 18 100584 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>> sleepq_wait+0x3a s= leeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_A= PV+0xab _vn_lock+0x43=0A>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>> nfsvno_fhtov= p+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>> nfssvc_program+0x55= 4 svc_run_internal+0xc77=0A>>>>>>>> svc_thread_start+0xb=0A>>>>>>>> fork_= exit+0x9a fork_trampoline+0xe=0A>>>>>>>> 918 100585 nfsd nfsd: service mi= _switch+0xe1=0A>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x9= 02=0A>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>= > zfs_fhtovp+0x38d=0A>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_= dorpc+0x917=0A>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>= >>>> svc_thread_start+0xb=0A>>>>>>>> fork_exit+0x9a fork_trampoline+0xe= =0A>>>>>>>> 918 100586 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>> sleep= q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>> vop_stdlock+0x3= c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>= nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>> nfssvc= _program+0x554 svc_run_internal+0xc77=0A>>>>>>>> svc_thread_start+0xb=0A>= >>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>> 918 100587 nfsd nfs= d: service mi_switch+0xe1=0A>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __loc= kmgr_args+0x902=0A>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0= x43=0A>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp= +0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>> nfssvc_program+0x554 svc_run_internal= +0xc77=0A>>>>>>>> svc_thread_start+0xb=0A>>>>>>>> fork_exit+0x9a fork_tra= mpoline+0xe=0A>>>>>>>> 918 100588 nfsd nfsd: service mi_switch+0xe1=0A>>>= >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>> vop_= stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>> zfs_fhtovp+0x38d= =0A>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>= >>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>> svc_thread_st= art+0xb=0A>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>> 918 1005= 89 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>> sleepq_wait+0x3a sleeplk+= 0x15d __lockmgr_args+0x902=0A>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab= _vn_lock+0x43=0A>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>> nfsvno_fhtovp+0x7c = nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>> nfssvc_program+0x554 svc_r= un_internal+0xc77=0A>>>>>>>> svc_thread_start+0xb=0A>>>>>>>> fork_exit+0x= 9a fork_trampoline+0xe=0A>>>>>>>> 918 100590 nfsd nfsd: service mi_switch= +0xe1=0A>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>= >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>> zfs_f= htovp+0x38d=0A>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0= x917=0A>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>> sv= c_thread_start+0xb=0A>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>= >> 918 100591 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>> sleepq_wait+0x= 3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>> vop_stdlock+0x3c VOP_LOC= K1_APV+0xab _vn_lock+0x43=0A>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>> nfsvno_f= htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>> nfssvc_program+= 0x554 svc_run_internal+0xc77=0A>>>>>>>> svc_thread_start+0xb=0A>>>>>>>> f= ork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>> 918 100592 nfsd nfsd: servic= e mi_switch+0xe1=0A>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args= +0x902=0A>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>= >>>>> zfs_fhtovp+0x38d=0A>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfs= rvd_dorpc+0x917=0A>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A= >>>>>>>> svc_thread_start+0xb=0A>>>>>>>> fork_exit+0x9a fork_trampoline+0= xe=0A>>>>>>>> 918 100593 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>> sle= epq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>> vop_stdlock+0= x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>= >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>> nfss= vc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>> svc_thread_start+0xb= =0A>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>> 918 100594 nfsd= nfsd: service mi_switch+0xe1=0A>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d _= _lockmgr_args+0x902=0A>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lo= ck+0x43=0A>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fh= tovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>> nfssvc_program+0x554 svc_run_inte= rnal+0xc77=0A>>>>>>>> svc_thread_start+0xb=0A>>>>>>>> fork_exit+0x9a fork= _trampoline+0xe=0A>>>>>>>> 918 100595 nfsd nfsd: service mi_switch+0xe1= =0A>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>= > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>> zfs_fhtovp= +0x38d=0A>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917= =0A>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>> svc_th= read_start+0xb=0A>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>> 9= 18 100596 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>> sleepq_wait+0x3a s= leeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_A= PV+0xab _vn_lock+0x43=0A>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>> nfsvno_fhtov= p+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>> nfssvc_program+0x55= 4 svc_run_internal+0xc77=0A>>>>>>>> svc_thread_start+0xb=0A>>>>>>>> fork_= exit+0x9a fork_trampoline+0xe=0A>>>>>>>> 918 100597 nfsd nfsd: service mi= _switch+0xe1=0A>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x9= 02=0A>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>= > zfs_fhtovp+0x38d=0A>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_= dorpc+0x917=0A>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>= >>>> svc_thread_start+0xb=0A>>>>>>>> fork_exit+0x9a fork_trampoline+0xe= =0A>>>>>>>> 918 100598 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>> sleep= q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>> vop_stdlock+0x3= c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>= nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>> nfssvc= _program+0x554 svc_run_internal+0xc77=0A>>>>>>>> svc_thread_start+0xb=0A>= >>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>> 918 100599 nfsd nfs= d: service mi_switch+0xe1=0A>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __loc= kmgr_args+0x902=0A>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0= x43=0A>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp= +0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>> nfssvc_program+0x554 svc_run_internal= +0xc77=0A>>>>>>>> svc_thread_start+0xb=0A>>>>>>>> fork_exit+0x9a fork_tra= mpoline+0xe=0A>>>>>>>> 918 100600 nfsd nfsd: service mi_switch+0xe1=0A>>>= >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>> vop_= stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>> zfs_fhtovp+0x38d= =0A>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>= >>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>> svc_thread_st= art+0xb=0A>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>> 918 1006= 01 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>> sleepq_wait+0x3a sleeplk+= 0x15d __lockmgr_args+0x902=0A>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab= _vn_lock+0x43=0A>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>> nfsvno_fhtovp+0x7c = nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>> nfssvc_program+0x554 svc_r= un_internal+0xc77=0A>>>>>>>> svc_thread_start+0xb=0A>>>>>>>> fork_exit+0x= 9a fork_trampoline+0xe=0A>>>>>>>> 918 100602 nfsd nfsd: service mi_switch= +0xe1=0A>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>= >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>> zfs_f= htovp+0x38d=0A>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0= x917=0A>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>> sv= c_thread_start+0xb=0A>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>= >> 918 100603 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>> sleepq_wait+0x= 3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>> vop_stdlock+0x3c VOP_LOC= K1_APV+0xab _vn_lock+0x43=0A>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>> nfsvno_f= htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>> nfssvc_program+= 0x554 svc_run_internal+0xc77=0A>>>>>>>> svc_thread_start+0xb=0A>>>>>>>> f= ork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>> 918 100604 nfsd nfsd: servic= e mi_switch+0xe1=0A>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args= +0x902=0A>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>= >>>>> zfs_fhtovp+0x38d=0A>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfs= rvd_dorpc+0x917=0A>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A= >>>>>>>> svc_thread_start+0xb=0A>>>>>>>> fork_exit+0x9a fork_trampoline+0= xe=0A>>>>>>>> 918 100605 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>> sle= epq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>> vop_stdlock+0= x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>= >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>> nfss= vc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>> svc_thread_start+0xb= =0A>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>> 918 100606 nfsd= nfsd: service mi_switch+0xe1=0A>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d _= _lockmgr_args+0x902=0A>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lo= ck+0x43=0A>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fh= tovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>> nfssvc_program+0x554 svc_run_inte= rnal+0xc77=0A>>>>>>>> svc_thread_start+0xb=0A>>>>>>>> fork_exit+0x9a fork= _trampoline+0xe=0A>>>>>>>> 918 100607 nfsd nfsd: service mi_switch+0xe1= =0A>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>= > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>> zfs_fhtovp= +0x38d=0A>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917= =0A>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>> svc_th= read_start+0xb=0A>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>> = =0A>>>>>>> Lots more waiting for the ZFS vnode lock in zfs_fhtovp().=0A>>= >>>>> =0A>>>>>>> 918 100608 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> s= leepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>>> nfs= rv_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1=0A>>>>>>> nfs= rvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>> svc= _thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>> 918 100609= nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3a sleeplk+0x1= 5d __lockmgr_args+0x902=0A>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn= _lock+0x43=0A>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>> nfsvno_fhtovp+0x7c nfsd_f= htovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>> nfssvc_program+0x554 svc_run_inte= rnal+0xc77=0A>>>>>>> svc_thread_start+0xb=0A>>>>>>> fork_exit+0x9a fork_t= rampoline+0xe=0A>>>>>>> 918 100610 nfsd nfsd: service mi_switch+0xe1=0A>>= >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e=0A>>>>>>> vop_s= tdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>> nfsvno_advlock+0x1= 19 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad=0A>>>>>>> nfsrvd_locku+0x283 = nfsrvd_dorpc+0xec6 nfssvc_program+0x554=0A>>>>>>> svc_run_internal+0xc77 = svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>> fork_trampoline+0xe=0A>>>>>= >> 918 100611 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3= a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>>> nfsrvd_dorpc+0x31= 6 nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>> svc_thread_start+= 0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>> 918 100612 nfsd nfsd: se= rvice mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x= 66 nfsv4_lock+0x9b=0A>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_= run_internal+0xc77=0A>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_tra= mpoline+0xe=0A>>>>>>> 918 100613 nfsd nfsd: service mi_switch+0xe1=0A>>>>= >>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>>= > nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>= > svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>> 918 1= 00614 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3a _sleep= +0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>>> nfsrvd_dorpc+0x316 nfssvc= _program+0x554 svc_run_internal+0xc77=0A>>>>>>> svc_thread_start+0xb fork= _exit+0x9a fork_trampoline+0xe=0A>>>>>>> 918 100615 nfsd nfsd: service mi= _switch+0xe1=0A>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4= _lock+0x9b=0A>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_inte= rnal+0xc77=0A>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+= 0xe=0A>>>>>>> 918 100616 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> slee= pq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>>> nfsrvd= _dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>> svc_th= read_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>> 918 100617 nf= sd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3a _sleep+0x287 n= fsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program= +0x554 svc_run_internal+0xc77=0A>>>>>>> svc_thread_start+0xb fork_exit+0x= 9a fork_trampoline+0xe=0A>>>>>>> 918 100618 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x= 9b=0A>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc= 77=0A>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>= >>>>> 918 100619 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_wait+= 0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>> vop_stdlock+0x3c VOP_LO= CK1_APV+0xab _vn_lock+0x43=0A>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>> nfsvno_fh= tovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>> nfssvc_program+0x= 554 svc_run_internal+0xc77=0A>>>>>>> svc_thread_start+0xb=0A>>>>>>> fork_= exit+0x9a fork_trampoline+0xe=0A>>>>>>> 918 100620 nfsd nfsd: service mi_= switch+0xe1=0A>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902= =0A>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>> zf= s_fhtovp+0x38d=0A>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc= +0x917=0A>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>> sv= c_thread_start+0xb=0A>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>= 918 100621 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3a = _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>>> nfsrvd_dorpc+0x316 = nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>> svc_thread_start+0x= b fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>> 918 100622 nfsd nfsd: serv= ice mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_arg= s+0x902=0A>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>= >>>> zfs_fhtovp+0x38d=0A>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrv= d_dorpc+0x917=0A>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>= >>>> svc_thread_start+0xb=0A>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A= >>>>>>> 918 100623 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_wai= t+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>>> nfsrvd_dorpc= +0x316 nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>> svc_thread_s= tart+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>> 918 100624 nfsd nfs= d: service mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lock= mgr_args+0x902=0A>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x4= 3=0A>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc= 8 nfsrvd_dorpc+0x917=0A>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc7= 7=0A>>>>>>> svc_thread_start+0xb=0A>>>>>>>=20fork_exit+0x9a fork_trampoli= ne+0xe=0A>>>>>>> 918 100625 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> s= leepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>> vop_stdlock+= 0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>= > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>> nfssvc= _program+0x554 svc_run_internal+0xc77=0A>>>>>>> svc_thread_start+0xb=0A>>= >>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>> 918 100626 nfsd nfsd: = service mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr= _args+0x902=0A>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43= =0A>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8= nfsrvd_dorpc+0x917=0A>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77= =0A>>>>>>> svc_thread_start+0xb=0A>>>>>>> fork_exit+0x9a fork_trampoline+= 0xe=0A>>>>>>> 918 100627 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> slee= pq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>> vop_stdlock+0x3= c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>> n= fsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>> nfssvc_pr= ogram+0x554 svc_run_internal+0xc77=0A>>>>>>> svc_thread_start+0xb=0A>>>>>= >> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>> 918 100628 nfsd nfsd: ser= vice mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_ar= gs+0x902=0A>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>= >>>>> zfs_fhtovp+0x38d=0A>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsr= vd_dorpc+0x917=0A>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>= >>>>> svc_thread_start+0xb=0A>>>>>>> fork_exit+0x9a fork_trampoline+0xe= =0A>>>>>>> 918 100629 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_= wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>> vop_stdlock+0x3c V= OP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>> nfsv= no_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>> nfssvc_progr= am+0x554 svc_run_internal+0xc77=0A>>>>>>> svc_thread_start+0xb=0A>>>>>>> = fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>> 918 100630 nfsd nfsd: servic= e mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+= 0x902=0A>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>= >> zfs_fhtovp+0x38d=0A>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_= dorpc+0x917=0A>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>= >> svc_thread_start+0xb=0A>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>= >>>>> 918 100631 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_wait+= 0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>> vop_stdlock+0x3c VOP_LO= CK1_APV+0xab _vn_lock+0x43=0A>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>> nfsvno_fh= tovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>> nfssvc_program+0x= 554 svc_run_internal+0xc77=0A>>>>>>> svc_thread_start+0xb=0A>>>>>>> fork_= exit+0x9a fork_trampoline+0xe=0A>>>>>>> 918 100632 nfsd nfsd: service mi_= switch+0xe1=0A>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902= =0A>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>> zf= s_fhtovp+0x38d=0A>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc= +0x917=0A>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>> sv= c_thread_start+0xb=0A>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>= 918 100633 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3a = sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>> vop_stdlock+0x3c VOP_LOCK1_A= PV+0xab _vn_lock+0x43=0A>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>> nfsvno_fhtovp+= 0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>> nfssvc_program+0x554 s= vc_run_internal+0xc77=0A>>>>>>> svc_thread_start+0xb=0A>>>>>>> fork_exit+= 0x9a fork_trampoline+0xe=0A>>>>>>> 918 100634 nfsd nfsd: service mi_switc= h+0xe1=0A>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>= >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>> zfs_fht= ovp+0x38d=0A>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x91= 7=0A>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>> svc_thr= ead_start+0xb=0A>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>> 918 = 100635 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3a sleep= lk+0x15d __lockmgr_args+0x902=0A>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0x= ab _vn_lock+0x43=0A>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>> nfsvno_fhtovp+0x7c = nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>> nfssvc_program+0x554 svc_ru= n_internal+0xc77=0A>>>>>>> svc_thread_start+0xb=0A>>>>>>> fork_exit+0x9a = fork_trampoline+0xe=0A>>>>>>> 918 100636 nfsd nfsd: service mi_switch+0xe= 1=0A>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>= vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>> zfs_fhtovp+0= x38d=0A>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>= >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>> svc_thread_s= tart+0xb=0A>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>> 918 10063= 7 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3a sleeplk+0x= 15d __lockmgr_args+0x902=0A>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _v= n_lock+0x43=0A>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>> nfsvno_fhtovp+0x7c nfsd_= fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>> nfssvc_program+0x554 svc_run_int= ernal+0xc77=0A>>>>>>> svc_thread_start+0xb=0A>>>>>>> fork_exit+0x9a fork_= trampoline+0xe=0A>>>>>>> 918 100638 nfsd nfsd: service mi_switch+0xe1=0A>= >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>> vop_= stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>> zfs_fhtovp+0x38d= =0A>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>= >> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>> svc_thread_start= +0xb=0A>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>> 918 100639 nf= sd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3a sleeplk+0x15d = __lockmgr_args+0x902=0A>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lo= ck+0x43=0A>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhto= vp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>> nfssvc_program+0x554 svc_run_interna= l+0xc77=0A>>>>>>> svc_thread_start+0xb=0A>>>>>>> fork_exit+0x9a fork_tram= poline+0xe=0A>>>>>>> 918 100640 nfsd nfsd: service mi_switch+0xe1=0A>>>>>= >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>> vop_stdl= ock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>> zfs_fhtovp+0x38d=0A>>= >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>> nf= ssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>> svc_thread_start+0xb= =0A>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>> 918 100641 nfsd n= fsd: service mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lo= ckmgr_args+0x902=0A>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0= x43=0A>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0= xc8 nfsrvd_dorpc+0x917=0A>>>>>>> nfssvc_program+0x554 svc_run_internal+0x= c77=0A>>>>>>> svc_thread_start+0xb=0A>>>>>>> fork_exit+0x9a fork_trampoli= ne+0xe=0A>>>>>>> 918 100642 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> s= leepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>> vop_stdlock+= 0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>= > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>> nfssvc= _program+0x554 svc_run_internal+0xc77=0A>>>>>>> svc_thread_start+0xb=0A>>= >>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>> 918 100643 nfsd nfsd: = service mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr= _args+0x902=0A>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43= =0A>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8= nfsrvd_dorpc+0x917=0A>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77= =0A>>>>>>> svc_thread_start+0xb=0A>>>>>>> fork_exit+0x9a fork_trampoline+= 0xe=0A>>>>>>> 918 100644 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> slee= pq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>> vop_stdlock+0x3= c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>> n= fsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>> nfssvc_pr= ogram+0x554 svc_run_internal+0xc77=0A>>>>>>> svc_thread_start+0xb=0A>>>>>= >> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>> 918 100645 nfsd nfsd: ser= vice mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_ar= gs+0x902=0A>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>= >>>>> zfs_fhtovp+0x38d=0A>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsr= vd_dorpc+0x917=0A>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>= >>>>> svc_thread_start+0xb=0A>>>>>>> fork_exit+0x9a fork_trampoline+0xe= =0A>>>>>>> 918 100646 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_= wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>> vop_stdlock+0x3c V= OP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>> nfsv= no_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>> nfssvc_progr= am+0x554 svc_run_internal+0xc77=0A>>>>>>> svc_thread_start+0xb=0A>>>>>>> = fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>> 918 100647 nfsd nfsd: servic= e mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+= 0x902=0A>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>= >> zfs_fhtovp+0x38d=0A>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_= dorpc+0x917=0A>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>= >> svc_thread_start+0xb=0A>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>= >>>>> 918 100648 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_wait+= 0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>> vop_stdlock+0x3c VOP_LO= CK1_APV+0xab _vn_lock+0x43=0A>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>> nfsvno_fh= tovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>> nfssvc_program+0x= 554 svc_run_internal+0xc77=0A>>>>>>> svc_thread_start+0xb=0A>>>>>>> fork_= exit+0x9a fork_trampoline+0xe=0A>>>>>>> 918 100649 nfsd nfsd: service mi_= switch+0xe1=0A>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902= =0A>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>> zf= s_fhtovp+0x38d=0A>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc= +0x917=0A>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>> sv= c_thread_start+0xb=0A>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>= 918 100650 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3a = sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>> vop_stdlock+0x3c VOP_LOCK1_A= PV+0xab _vn_lock+0x43=0A>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>> nfsvno_fhtovp+= 0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>> nfssvc_program+0x554 s= vc_run_internal+0xc77=0A>>>>>>> svc_thread_start+0xb=0A>>>>>>> fork_exit+= 0x9a fork_trampoline+0xe=0A>>>>>>> 918 100651 nfsd nfsd: service mi_switc= h+0xe1=0A>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>= >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>> zfs_fht= ovp+0x38d=0A>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x91= 7=0A>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>> svc_thr= ead_start+0xb=0A>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>> 918 = 100652 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3a sleep= lk+0x15d __lockmgr_args+0x902=0A>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0x= ab _vn_lock+0x43=0A>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>> nfsvno_fhtovp+0x7c = nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>> nfssvc_program+0x554 svc_ru= n_internal+0xc77=0A>>>>>>> svc_thread_start+0xb=0A>>>>>>> fork_exit+0x9a = fork_trampoline+0xe=0A>>>>>>> 918 100653 nfsd nfsd: service mi_switch+0xe= 1=0A>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>= vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>> zfs_fhtovp+0= x38d=0A>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>= >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>> svc_thread_s= tart+0xb=0A>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>> 918 10065= 4 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3a sleeplk+0x= 15d __lockmgr_args+0x902=0A>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _v= n_lock+0x43=0A>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>> nfsvno_fhtovp+0x7c nfsd_= fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>> nfssvc_program+0x554 svc_run_int= ernal+0xc77=0A>>>>>>> svc_thread_start+0xb=0A>>>>>>> fork_exit+0x9a fork_= trampoline+0xe=0A>>>>>>> 918 100655 nfsd nfsd: service mi_switch+0xe1=0A>= >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>> vop_= stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>> zfs_fhtovp+0x38d= =0A>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>= >> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>> svc_thread_start= +0xb=0A>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>> 918 100656 nf= sd nfsd: service mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3a sleeplk+0x15d = __lockmgr_args+0x902=0A>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lo= ck+0x43=0A>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhto= vp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>> nfssvc_program+0x554 svc_run_interna= l+0xc77=0A>>>>>>> svc_thread_start+0xb=0A>>>>>>> fork_exit+0x9a fork_tram= poline+0xe=0A>>>>>>> 918 100657 nfsd nfsd: service mi_switch+0xe1=0A>>>>>= >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>> vop_stdl= ock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>> zfs_fhtovp+0x38d=0A>>= >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>> nf= ssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>> svc_thread_start+0xb= =0A>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>> 918 100658 nfsd n= fsd: service mi_switch+0xe1=0A>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lo= ckmgr_args+0x902=0A>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0= x43=0A>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0= xc8 nfsrvd_dorpc+0x917=0A>>>>>>> nfssvc_program+0x554 svc_run_internal+0x= c77=0A>>>>>>> svc_thread_start+0xb=0A>>>>>>> fork_exit+0x9a fork_trampoli= ne+0xe=0A>>>>>>> =0A>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>> UNIX Systems, Netwo= rk and Security Engineer=0A>>>>>>> http://www.unix-experience.fr=0A>>>>>>= > =0A>>>>>>> 15 d=C3=A9cembre 2014 13:29 "Lo=C3=AFc Blot"=0A>>>>>>> <loic= .blot@unix-experience.fr>=0A>>>>>>> a=0A>>>>>>> =C3=A9crit:=0A>>>>>>> =0A= >>>>>>> Hmmm...=0A>>>>>>> now i'm experiencing a deadlock.=0A>>>>>>> =0A>= >>>>>> 0 918 915 0 21 0 12352 3372 zfs D - 1:48.64 nfsd: server=0A>>>>>>>= (nfsd)=0A>>>>>>> =0A>>>>>>> the only issue was to reboot the server, but= after rebooting=0A>>>>>>> deadlock arrives a second time when i=0A>>>>>>= > start my jails over NFS.=0A>>>>>>> =0A>>>>>>> Regards,=0A>>>>>>> =0A>>>= >>>> Lo=C3=AFc Blot,=0A>>>>>>> UNIX Systems, Network and Security Enginee= r=0A>>>>>>> http://www.unix-experience.fr=0A>>>>>>>; =0A>>>>>>> 15 d=C3=A9= cembre 2014 10:07 "Lo=C3=AFc Blot"=0A>>>>>>> <loic.blot@unix-experience.f= r>=0A>>>>>>> a=0A>>>>>>> =C3=A9crit:=0A>>>>>>> =0A>>>>>>> Hi Rick,=0A>>>>= >>> after talking with my N+1, NFSv4 is required on our=0A>>>>>>> infrast= ructure.=0A>>>>>>> I tried to upgrade NFSv4+ZFS=0A>>>>>>> server from 9.3= to 10.1, i hope this will resolve some=0A>>>>>>> issues...=0A>>>>>>> =0A= >>>>>>> Regards,=0A>>>>>>> =0A>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>> UNIX Syst= ems, Network and Security Engineer=0A>>>>>>> http://www.unix-experience.f= r=0A>>>>>>> =0A>>>>>>> 10 d=C3=A9cembre 2014 15:36 "Lo=C3=AFc Blot"=0A>>>= >>>> <loic.blot@unix-experience.fr>=0A>>>>>>> a=0A>>>>>>> =C3=A9crit:=0A>= >>>>>> =0A>>>>>>> Hi Rick,=0A>>>>>>> thanks for your suggestion.=0A>>>>>>= > For my locking bug, rpc.lockd is stucked in rpcrecv state on=0A>>>>>>> = the=0A>>>>>>> server. kill -9 doesn't affect the=0A>>>>>>> process, it's = blocked.... (State: Ds)=0A>>>>>>> =0A>>>>>>> for the performances=0A>>>>>= >> =0A>>>>>>> NFSv3: 60Mbps=0A>>>>>>> NFSv4: 45Mbps=0A>>>>>>> Regards,=0A= >>>>>>> =0A>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>> UNIX Systems, Network and Se= curity Engineer=0A>>>>>>> http://www.unix-experience.fr=0A>>>>>>>; =0A>>>>= >>> 10 d=C3=A9cembre 2014 13:56 "Rick Macklem" <rmacklem@uoguelph.ca>=0A>= >>>>>> a=0A>>>>>>> =C3=A9crit:=0A>>>>>>> =0A>>>>>>> Loic Blot wrote:=0A>>= >>>>> =0A>>>>>>>> Hi Rick,=0A>>>>>>>> I'm trying NFSv3.=0A>>>>>>>> Some j= ails are starting very well but now i have an issue=0A>>>>>>>> with=0A>>>= >>>>> lockd=0A>>>>>>>> after some minutes:=0A>>>>>>>> =0A>>>>>>>> nfs ser= ver 10.10.X.8:/jails: lockd not responding=0A>>>>>>>> nfs server 10.10.X.= 8:/jails lockd is alive again=0A>>>>>>>> =0A>>>>>>>> I look at mbuf, but = i seems there is no problem.=0A>>>>>>> =0A>>>>>>> Well, if you need locks= to be visible across multiple=0A>>>>>>> clients,=0A>>>>>>> then=0A>>>>>>= > I'm afraid you are stuck with using NFSv4 and the=0A>>>>>>> performance= =0A>>>>>>> you=0A>>>>>>> get=0A>>>>>>> from it. (There is no way to do fi= le handle affinity for=0A>>>>>>> NFSv4=0A>>>>>>> because=0A>>>>>>> the re= ad and write ops are buried in the compound RPC and=0A>>>>>>> not=0A>>>>>= >> easily=0A>>>>>>> recognized.)=0A>>>>>>> =0A>>>>>>> If the locks don't = need to be visible across multiple=0A>>>>>>> clients,=0A>>>>>>> I'd=0A>>>= >>>> suggest trying the "nolockd" option with nfsv3.=0A>>>>>>> =0A>>>>>>>= > Here is my rc.conf on server:=0A>>>>>>>> =0A>>>>>>>> nfs_server_enable= =3D"YES"=0A>>>>>>>> nfsv4_server_enable=3D"YES"=0A>>>>>>>> nfsuserd_enabl= e=3D"YES"=0A>>>>>>>> nfsd_server_flags=3D"-u -t -n 256"=0A>>>>>>>> mountd= _enable=3D"YES"=0A>>>>>>>> mountd_flags=3D"-r"=0A>>>>>>>> nfsuserd_flags= =3D"-usertimeout 0 -force 20"=0A>>>>>>>> rpcbind_enable=3D"YES"=0A>>>>>>>= > rpc_lockd_enable=3D"YES"=0A>>>>>>>> rpc_statd_enable=3D"YES"=0A>>>>>>>>= =0A>>>>>>>> Here is the client:=0A>>>>>>>> =0A>>>>>>>> nfsuserd_enable= =3D"YES"=0A>>>>>>>> nfsuserd_flags=3D"-usertimeout 0 -force 20"=0A>>>>>>>= > nfscbd_enable=3D"YES"=0A>>>>>>>> rpc_lockd_enable=3D"YES"=0A>>>>>>>> rp= c_statd_enable=3D"YES"=0A>>>>>>>> =0A>>>>>>>> Have you got an idea ?=0A>>= >>>>>> =0A>>>>>>>> Regards,=0A>>>>>>>> =0A>>>>>>>> Lo=C3=AFc Blot,=0A>>>>= >>>> UNIX Systems, Network and Security Engineer=0A>>>>>>>> http://www.un= ix-experience.fr=0A>>>>>>>> =0A>>>>>>>> 9 d=C3=A9cembre 2014 04:31 "Rick = Macklem" <rmacklem@uoguelph.ca>=0A>>>>>>>> a=0A>>>>>>>> =C3=A9crit:=0A>>>= >>>>>> Loic Blot wrote:=0A>>>>>>>>> =0A>>>>>>>>>> Hi rick,=0A>>>>>>>>>> = =0A>>>>>>>>>> I waited 3 hours (no lag at jail launch) and now I do:=0A>>= >>>>>>>> sysrc=0A>>>>>>>>>> memcached_flags=3D"-v -m 512"=0A>>>>>>>>>> Co= mmand was very very slow...=0A>>>>>>>>>> =0A>>>>>>>>>> Here is a dd over = NFS:=0A>>>>>>>>>> =0A>>>>>>>>>> 601062912 bytes transferred in 21.060679 = secs (28539579=0A>>>>>>>>>> bytes/sec)=0A>>>>>>>>> =0A>>>>>>>>> Can you t= ry the same read using an NFSv3 mount?=0A>>>>>>>>> (If it runs much faste= r, you have probably been bitten by=0A>>>>>>>>> the=0A>>>>>>>>> ZFS=0A>>>= >>>>>> "sequential vs random" read heuristic which I've been told=0A>>>>>= >>>> things=0A>>>>>>>>> NFS is doing "random" reads without file handle a= ffinity.=0A>>>>>>>>> File=0A>>>>>>>>> handle affinity is very hard to do = for NFSv4, so it isn't=0A>>>>>>>>> done.)=0A>>>>>>> =0A>>>>>>> I was actu= ally suggesting that you try the "dd" over nfsv3=0A>>>>>>> to=0A>>>>>>> s= ee=0A>>>>>>> how=0A>>>>>>> the performance compared with nfsv4. If you do= that, please=0A>>>>>>> post=0A>>>>>>> the=0A>>>>>>> comparable results.= =0A>>>>>>> =0A>>>>>>> Someday I would like to try and get ZFS's sequentia= l vs=0A>>>>>>> random=0A>>>>>>> read=0A>>>>>>> heuristic modified and any= info on what difference in=0A>>>>>>> performance=0A>>>>>>> that=0A>>>>>>= > might make for NFS would be useful.=0A>>>>>>> =0A>>>>>>> rick=0A>>>>>>>= =0A>>>>>>> rick=0A>>>>>>> =0A>>>>>>> This is quite slow...=0A>>>>>>> =0A= >>>>>>> You can found some nfsstat below (command isn't finished=0A>>>>>>= > yet)=0A>>>>>>> =0A>>>>>>> nfsstat -c -w 1=0A>>>>>>> =0A>>>>>>> GtAttr L= ookup Rdlink Read Write Rename Access Rddir=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>= >>>>>> 4 0 0 0 0 0 16 0=0A>>>>>>> 2 0 0 0 0 0 17 0=0A>>>>>>> 0 0 0 0 0 0 = 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0= 0 0 0 0=0A>>>>>>> 0 4 0 0 0 0 4 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 = 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>= >> 4 0 0 0 0 0 3 0=0A>>>>>>> 0 0 0 0 0 0 3 0=0A>>>>>>> 37 10 0 8 0 0 14 1= =0A>>>>>>> 18 16 0 4 1 2 4 0=0A>>>>>>> 78 91 0 82 6 12 30 0=0A>>>>>>> 19 = 18 0 2 2 4 2 0=0A>>>>>>> 0 0 0 0 2 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>= >>> GtAttr Lookup Rdlink Read Write Rename Access Rddir=0A>>>>>>> 0 0 0 0= 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 = 1 0 0 0 0 1 0=0A>>>>>>> 4 6 0 0 6 0 3 0=0A>>>>>>> 2 0 0 0 0 0 0 0=0A>>>>>= >> 0 0 0 0 0 0 0 0=0A>>>>>>> 1 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 1 0 0 0=0A= >>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0= 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 = 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 6 1= 08 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>= >>> GtAttr Lookup Rdlink Read Write Rename Access Rddir=0A>>>>>>> 0 0 0 0= 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 = 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>= >> 0 0 0 0 0 0 0 0=0A>>>>>>> 98 54 0 86 11 0 25 0=0A>>>>>>> 36 24 0 39 25= 0 10 1=0A>>>>>>> 67 8 0 63 63 0 41 0=0A>>>>>>> 34 0 0 35 34 0 0 0=0A>>>>= >>> 75 0 0 75 77 0 0 0=0A>>>>>>> 34 0 0 35 35 0 0 0=0A>>>>>>> 75 0 0 74 7= 6 0 0 0=0A>>>>>>> 33 0 0 34 33 0 0 0=0A>>>>>>> 0 0 0 0 5 0 0 0=0A>>>>>>> = 0 0 0 0 0 0 6 0=0A>>>>>>> 11 0 0 0 0 0 11 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>= >>>>>> 0 17 0 0 0 0 1 0=0A>>>>>>> GtAttr Lookup Rdlink Read Write Rename = Access Rddir=0A>>>>>>> 4 5 0 0 0 0 12 0=0A>>>>>>> 2 0 0 0 0 0 26 0=0A>>>>= >>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0= =0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 4 0 0 0 = 0 4 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0= 0 0 0 0 0=0A>>>>>>> 4 0 0 0 0 0 2 0=0A>>>>>>> 2 0 0 0 0 0 24 0=0A>>>>>>>= 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>= >>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0= =0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> GtAttr Lookup Rdlink Read Write Rena= me Access Rddir=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>= >>>> 4 0 0 0 0 0 7 0=0A>>>>>>> 2 1 0 0 0 0 1 0=0A>>>>>>> 0 0 0 0 2 0 0 0= =0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 6 0 0 0=0A>>>>>>> 0 0 0 0 0 = 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0= 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> = 4 6 0 0 0 0 3 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 2 0 0 0 0 0 0 0=0A>>>= >>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0= =0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> GtAttr Lookup Rdlink Read Write Rena= me Access Rddir=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>= >>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0= =0A>>>>>>> 4 71 0 0 0 0 0 0=0A>>>>>>> 0 1 0 0 0 0 0 0=0A>>>>>>> 2 36 0 0 = 0 0 1 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0= 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 1 0 0 0 0 0 1 0=0A>>>>>>= > 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 79 6 0 79 79 0 2 0= =0A>>>>>>> 25 0 0 25 26 0 6 0=0A>>>>>>> 43 18 0 39 46 0 23 0=0A>>>>>>> 36= 0 0 36 36 0 31 0=0A>>>>>>> 68 1 0 66 68 0 0 0=0A>>>>>>> GtAttr Lookup Rd= link Read Write Rename Access Rddir=0A>>>>>>> 36 0 0 36 36 0 0 0=0A>>>>>>= > 48 0 0 48 49 0 0 0=0A>>>>>>> 20 0 0 20 20 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0= 0=0A>>>>>>> 3 14 0 1 0 0 11 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 = 0 0 0 0 0=0A>>>>>>> 0 4 0 0 0 0 4 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 4= 22 0 0 0 0 16 0=0A>>>>>>> 2 0 0 0 0 0 23 0=0A>>>>>>> =0A>>>>>>> Regards,= =0A>>>>>>> =0A>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>> UNIX Systems, Network and= Security Engineer=0A>>>>>>> http://www.unix-experience.fr=0A>>>>>>>; =0A>= >>>>>> 8 d=C3=A9cembre 2014 09:36 "Lo=C3=AFc Blot"=0A>>>>>>> <loic.blot@u= nix-experience.fr> a=0A>>>>>>> =C3=A9crit:=0A>>>>>>>> Hi Rick,=0A>>>>>>>>= I stopped the jails this week-end and started it this=0A>>>>>>>> morning= ,=0A>>>>>>>> i'll=0A>>>>>>>> give you some stats this week.=0A>>>>>>>> = =0A>>>>>>>> Here is my nfsstat -m output (with your rsize/wsize=0A>>>>>>>= > tweaks)=0A>>>>>> =0A>>>>>> =0A>>>>> =0A>>>> =0A>>> =0A>> =0A> nfsv4,tcp= ,resvport,hard,cto,sec=3Dsys,acdirmin=3D3,acdirmax=3D60,acregmin=3D5,acre= gmax=3D60,nametimeo=3D60,negna=0A>>>>>> =0A>>>>>>> =0A>>>>>> =0A>>>>>> = =0A>>>>> =0A>>>> =0A>>> =0A>> =0A> etimeo=3D60,rsize=3D32768,wsize=3D3276= 8,readdirsize=3D32768,readahead=3D1,wcommitsize=3D773136,timeout=3D120,re= tra=0A>>>>>> =0A>>>>>>> s=3D2147483647=0A>>>>>>> =0A>>>>>>> On server sid= e my disks are on a raid controller which show a=0A>>>>>>> 512b=0A>>>>>>>= volume and write performances=0A>>>>>>> are very honest (dd if=3D/dev/ze= ro of=3D/jails/test.dd bs=3D4096=0A>>>>>>> count=3D100000000 =3D> 450MBps= )=0A>>>>>>> =0A>>>>>>> Regards,=0A>>>>>>> =0A>>>>>>> Lo=C3=AFc Blot,=0A>>= >>>>> UNIX Systems, Network and Security Engineer=0A>>>>>>> http://www.un= ix-experience.fr=0A>>>>>>> =0A>>>>>>> 5 d=C3=A9cembre 2014 15:14 "Rick Ma= cklem" <rmacklem@uoguelph.ca> a=0A>>>>>>> =C3=A9crit:=0A>>>>>>> =0A>>>>>>= > Loic Blot wrote:=0A>>>>>>> =0A>>>>>>> Hi,=0A>>>>>>> i'm trying to creat= e a virtualisation environment based on=0A>>>>>>> jails.=0A>>>>>>> Those = jails are stored under a big ZFS pool on a FreeBSD=0A>>>>>>> 9.3=0A>>>>>>= > which=0A>>>>>>> export a NFSv4 volume. This NFSv4 volume was mounted on= a=0A>>>>>>> big=0A>>>>>>> hypervisor (2 Xeon E5v3 + 128GB memory and 8 p= orts (but=0A>>>>>>> only 1=0A>>>>>>> was=0A>>>>>>> used at this time).=0A= >>>>>>> =0A>>>>>>> The problem is simple, my hypervisors runs 6 jails (us= ed 1%=0A>>>>>>> cpu=0A>>>>>>> and=0A>>>>>>> 10GB RAM approximatively and = less than 1MB bandwidth) and=0A>>>>>>> works=0A>>>>>>> fine at start but = the system slows down and after 2-3 days=0A>>>>>>> become=0A>>>>>>> unusa= ble. When i look at top command i see 80-100% on=0A>>>>>>> system=0A>>>>>= >> and=0A>>>>>>> commands are very very slow. Many process are tagged wit= h=0A>>>>>>> nfs_cl*.=0A>>>>>>> =0A>>>>>>> To be honest, I would expect th= e slowness to be because of=0A>>>>>>> slow=0A>>>>>>> response=0A>>>>>>> f= rom the NFSv4 server, but if you do:=0A>>>>>>> # ps axHl=0A>>>>>>> on a c= lient when it is slow and post that, it would give us=0A>>>>>>> some=0A>>= >>>>> more=0A>>>>>>> information on where the client side processes are s= itting.=0A>>>>>>> If you also do something like:=0A>>>>>>> # nfsstat -c -= w 1=0A>>>>>>> and let it run for a while, that should show you how=20many= =0A>>>>>>> RPCs=0A>>>>>>> are=0A>>>>>>> being done and which ones.=0A>>>>= >>> =0A>>>>>>> # nfsstat -m=0A>>>>>>> will show you what your mount is ac= tually using.=0A>>>>>>> The only mount option I can suggest trying is=0A>= >>>>>> "rsize=3D32768,wsize=3D32768",=0A>>>>>>> since some network enviro= nments have difficulties with 64K.=0A>>>>>>> =0A>>>>>>> There are a few t= hings you can try on the NFSv4 server side,=0A>>>>>>> if=0A>>>>>>> it=0A>= >>>>>> appears=0A>>>>>>> that the clients are generating a large RPC load= .=0A>>>>>>> - disabling the DRC cache for TCP by setting=0A>>>>>>> vfs.nf= sd.cachetcp=3D0=0A>>>>>>> - If the server is seeing a large write RPC loa= d, then=0A>>>>>>> "sync=3Ddisabled"=0A>>>>>>> might help, although it doe= s run a risk of data loss when=0A>>>>>>> the=0A>>>>>>> server=0A>>>>>>> c= rashes.=0A>>>>>>> Then there are a couple of other ZFS related things (I'= m not=0A>>>>>>> a=0A>>>>>>> ZFS=0A>>>>>>> guy,=0A>>>>>>> but these have s= hown up on the mailing lists).=0A>>>>>>> - make sure your volumes are 4K = aligned and ashift=3D12 (in=0A>>>>>>> case a=0A>>>>>>> drive=0A>>>>>>> th= at uses 4K sectors is pretending to be 512byte sectored)=0A>>>>>>> - neve= r run over 70-80% full if write performance is an=0A>>>>>>> issue=0A>>>>>= >> - use a zil on an SSD with good write performance=0A>>>>>>> =0A>>>>>>>= The only NFSv4 thing I can tell you is that it is known that=0A>>>>>>> Z= FS's=0A>>>>>>> algorithm for determining sequential vs random I/O fails f= or=0A>>>>>>> NFSv4=0A>>>>>>> during writing and this can be a performance= hit. The only=0A>>>>>>> workaround=0A>>>>>>> is to use NFSv3 mounts, sin= ce file handle affinity=0A>>>>>>> apparently=0A>>>>>>> fixes=0A>>>>>>> th= e problem and this is only done for NFSv3.=0A>>>>>>> =0A>>>>>>> rick=0A>>= >>>>> =0A>>>>>>> I saw that there are TSO issues with igb then i'm trying= to=0A>>>>>>> disable=0A>>>>>>> it with sysctl but the situation wasn't s= olved.=0A>>>>>>> =0A>>>>>>> Someone has got ideas ? I can give you more i= nformations if=0A>>>>>>> you=0A>>>>>>> need.=0A>>>>>>> =0A>>>>>>> Thanks = in advance.=0A>>>>>>> Regards,=0A>>>>>>> =0A>>>>>>> Lo=C3=AFc Blot,=0A>>>= >>>> UNIX Systems, Network and Security Engineer=0A>>>>>>> http://www.uni= x-experience.fr=0A>>>>>>> _______________________________________________= =0A>>>>>>> freebsd-fs@freebsd.org mailing list=0A>>>>>>> http://lists.fre= ebsd.org/mailman/listinfo/freebsd-fs=0A>>>>>>> To unsubscribe, send any m= ail to=0A>>>>>>> "freebsd-fs-unsubscribe@freebsd.org"=0A>>>>>>> =0A>>>>>>= > _______________________________________________=0A>>>>>>> freebsd-fs@fr= eebsd.org mailing list=0A>>>>>>> http://lists.freebsd.org/mailman/listinf= o/freebsd-fs=0A>>>>>>> To unsubscribe, send any mail to=0A>>>>>>> "freebs= d-fs-unsubscribe@freebsd.org"=0A>>>>>>> =0A>>>>>>> ______________________= _________________________=0A>>>>>>> freebsd-fs@freebsd.org mailing list= =0A>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A>>>>>>>= To unsubscribe, send any mail to=0A>>>>>>> "freebsd-fs-unsubscribe@freeb= sd.org"=0A>>>>>>> =0A>>>>>>> ____________________________________________= ___=0A>>>>>>> freebsd-fs@freebsd.org mailing list=0A>>>>>>> http://lists.= freebsd.org/mailman/listinfo/freebsd-fs=0A>>>>>>> To unsubscribe, send an= y mail to=0A>>>>>>> "freebsd-fs-unsubscribe@freebsd.org"=0A>>>>>>> ______= _________________________________________=0A>>>>>>> freebsd-fs@freebsd.or= g mailing list=0A>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebs= d-fs=0A>>>>>>> To unsubscribe, send any mail to=0A>>>>>>> "freebsd-fs-uns= ubscribe@freebsd.org"=0A>>>>>>> =0A>>>>>>> ______________________________= _________________=0A>>>>>>> freebsd-fs@freebsd.org mailing list=0A>>>>>>>= http://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A>>>>>>>; To unsubs= cribe, send any mail to=0A>>>>>>> "freebsd-fs-unsubscribe@freebsd.org"=0A= >>>>> =0A>>>>> _______________________________________________=0A>>>>> fr= eebsd-fs@freebsd.org mailing list=0A>>>>> http://lists.freebsd.org/mailma= n/listinfo/freebsd-fs=0A>>>>> To unsubscribe, send any mail to=0A>>>>> "f= reebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?f8c1e0380b1c195ce3c63bc41fafb9e1>