Date: Mon, 05 Jan 2015 11:16:21 +0000 From: "=?utf-8?B?TG/Dr2MgQmxvdA==?=" <loic.blot@unix-experience.fr> To: "Rick Macklem" <rmacklem@uoguelph.ca> Cc: freebsd-fs@freebsd.org Subject: Re: High Kernel Load with nfsv4 Message-ID: <87a8b6ab243024e553b2baba30537b92@mail.unix-experience.fr> In-Reply-To: <82f8ef92c6bf5f2323bae5ce5e4e2394@mail.unix-experience.fr> References: <82f8ef92c6bf5f2323bae5ce5e4e2394@mail.unix-experience.fr> <1479765128.1118136.1419290403230.JavaMail.root@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi,=0Ahappy new year Rick and @freebsd-fs.=0A=0AAfter some days, i looked= my NFSv4.1 mount. At server start it was calm, but after 4 days, here is= the top stat...=0A=0ACPU: 0.0% user, 0.0% nice, 100% system, 0.0% in= terrupt, 0.0% idle=0A=0ADefinitively i think it's a problem on client si= de. What can i look into running kernel to resolve this issue ?=0A=0A=0AR= egards,=0A=0ALo=C3=AFc Blot,=0AUNIX Systems, Network and Security Enginee= r=0Ahttp://www.unix-experience.fr=0A=0A30 d=C3=A9cembre 2014 16:16 "Lo=C3= =AFc Blot" <loic.blot@unix-experience.fr> a =C3=A9crit: =0A> Hi Rick,=0A>= i upgraded my jail host from FreeBSD 9.3 to 10.1 to use NFS v4.1 (mounto= ptions:=0A> rw,rsize=3D32768,wsize=3D32768,tcp,nfsv4,minorversion=3D1)=0A= > =0A> Performance is quite stable but it's slow. Not as slow as before b= ut slow... services was launched=0A> but no client are using them and sys= tem CPU % was 10-50%.=0A> =0A> I don't see anything on NFSv4.1 server, it= 's perfectly stable and functionnal.=0A> =0A> Regards,=0A> =0A> Lo=C3=AFc= Blot,=0A> UNIX Systems, Network and Security Engineer=0A> http://www.uni= x-experience.fr=0A> =0A> 23 d=C3=A9cembre 2014 00:20 "Rick Macklem" <rmac= klem@uoguelph.ca> a =C3=A9crit:=0A> =0A>> Loic Blot wrote:=0A>> =0A>>> Hi= ,=0A>>> =0A>>> To clarify because of our exchanges. Here are the current = sysctl=0A>>> options for server:=0A>>> =0A>>> vfs.nfsd.enable_nobodycheck= =3D0=0A>>> vfs.nfsd.enable_nogroupcheck=3D0=0A>>> =0A>>> vfs.nfsd.maxthre= ads=3D200=0A>>> vfs.nfsd.tcphighwater=3D10000=0A>>> vfs.nfsd.tcpcachetime= o=3D300=0A>>> vfs.nfsd.server_min_nfsvers=3D4=0A>>> =0A>>> kern.maxvnodes= =3D10000000=0A>>> kern.ipc.maxsockbuf=3D4194304=0A>>> net.inet.tcp.sendbu= f_max=3D4194304=0A>>> net.inet.tcp.recvbuf_max=3D4194304=0A>>> =0A>>> vfs= .lookup_shared=3D0=0A>>> =0A>>> Regards,=0A>>> =0A>>> Lo=C3=AFc Blot,=0A>= >> UNIX Systems, Network and Security Engineer=0A>>> http://www.unix-expe= rience.fr=0A>>> =0A>>> 22 d=C3=A9cembre 2014 09:42 "Lo=C3=AFc Blot" <loic= .blot@unix-experience.fr> a=0A>>> =C3=A9crit:=0A>>> =0A>>> Hi Rick,=0A>>>= my 5 jails runs this weekend and now i have some stats on this=0A>>> mon= day.=0A>>> =0A>>> Hopefully deadlock was fixed, yeah, but everything isn'= t good :(=0A>>> =0A>>> On NFSv4 server (FreeBSD 10.1) system uses 35% CPU= =0A>>> =0A>>> As i can see this is because of nfsd:=0A>>> =0A>>> 918 root= 96 20 0 12352K 3372K rpcsvc 6 51.4H=0A>>> 273.68% nfsd: server (nfsd)=0A= >>> =0A>>> If i look at dmesg i see:=0A>>> nfsd server cache flooded, try= increasing vfs.nfsd.tcphighwater=0A>> =0A>> Well, you have a couple of c= hoices:=0A>> 1 - Use NFSv4.1 (add "minorversion=3D1" to your mount option= s).=0A>> (NFSv4.1 avoids use of the DRC and instead uses something=0A>> c= alled sessions. See below.)=0A>> OR=0A>> =0A>>> vfs.nfsd.tcphighwater was= set to 10000, i increase it to 15000=0A>> =0A>> 2 - Bump vfs.nfsd.tcphig= hwater way up, until you no longer see=0A>> "nfs server cache flooded" me= ssages. (I think Garrett Wollman uses=0A>> 100000. (You may still see qui= te a bit of CPU overheads.)=0A>> =0A>> OR=0A>> =0A>> 3 - Set vfs.nfsd.cac= hetcp=3D0 (which disables the DRC and gets rid=0A>> of the CPU overheads)= . However, there is a risk of data corruption=0A>> if you have a client->= server network partitioning of a moderate=0A>> duration, because a non-id= empotent RPC may get redone, becasue=0A>> the client times out waiting fo= r a reply. If a non-idempotent=0A>> RPC gets done twice on the server, da= ta corruption can happen.=0A>> (The DRC provides improved correctness, bu= t does add overhead.)=0A>> =0A>> If #1 works for you, it is the preferred= solution, since Sessions=0A>> in NFSv4.1 solves the correctness problem = in a good, space bound=0A>> way. A session basically has N (usually 32 or= 64) slots and only=0A>> allows one outstanding RPC/slot. As such, it can= cache the previous=0A>> reply for each slot (32 or 64 of them) and guara= ntee "exactly once"=0A>> RPC semantics.=0A>> =0A>> rick=0A>> =0A>>> Here = is 'nfsstat -s' output:=0A>>> =0A>>> Server Info:=0A>>> Getattr Setattr L= ookup Readlink Read Write Create=0A>>> Remove=0A>>> 12600652 1812 2501097= 156 1386423 1983729 123=0A>>> 162067=0A>>> Rename Link Symlink Mkdir Rmd= ir Readdir RdirPlus=0A>>> Access=0A>>> 36762 9 0 0 0 3147 0=0A>>> 623524= =0A>>> Mknod Fsstat Fsinfo PathConf Commit=0A>>> 0 0 0 0 328117=0A>>> Ser= ver Ret-Failed=0A>>> 0=0A>>> Server Faults=0A>>> 0=0A>>> Server Cache Sta= ts:=0A>>> Inprog Idem Non-idem Misses=0A>>> 0 0 0 12635512=0A>>> Server W= rite Gathering:=0A>>> WriteOps WriteRPC Opsaved=0A>>> 1983729 1983729 0= =0A>>> =0A>>> And here is 'procstat -kk' for nfsd (server)=0A>>> =0A>>> 9= 18 100528 nfsd nfsd: master mi_switch+0xe1=0A>>> sleepq_catch_signals+0xa= b sleepq_timedwait_sig+0x10=0A>>> _cv_timedwait_sig_sbt+0x18b svc_run_int= ernal+0x4a1 svc_run+0x1de=0A>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_n= fssvc+0x9c=0A>>> amd64_syscall+0x351 Xfast_syscall+0xfb=0A>>> 918 100568 = nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_= wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_s= tart+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100569 nfsd n= fsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_s= ig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0= xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100570 nfsd nfsd: s= ervice mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb for= k_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100571 nfsd nfsd: service= mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_w= ait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100572 nfsd nfsd: service mi_sw= itch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_si= g+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>> fork_trampoline+0xe=0A>>> 918 100573 nfsd nfsd: service mi_switch+= 0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1= 6a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>= fork_trampoline+0xe=0A>>> 918 100574 nfsd nfsd: service mi_switch+0xe1= =0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a= =0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> f= ork_trampoline+0xe=0A>>> 918 100575 nfsd nfsd: service mi_switch+0xe1=0A>= >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>= svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_tr= ampoline+0xe=0A>>> 918 100576 nfsd nfsd: service mi_switch+0xe1=0A>>> sle= epq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_r= un_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoli= ne+0xe=0A>>> 918 100577 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_ca= tch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe= =0A>>> 918 100578 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+= 0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>>= 918 100579 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+= 0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 1= 00580 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab s= leepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_th= read_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100581 = nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_= wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_s= tart+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100582 nfsd n= fsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_s= ig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0= xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100583 nfsd nfsd: s= ervice mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb for= k_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100584 nfsd nfsd: service= mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_w= ait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100585 nfsd nfsd: service mi_sw= itch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_si= g+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>> fork_trampoline+0xe=0A>>> 918 100586 nfsd nfsd: service mi_switch+= 0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1= 6a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>= fork_trampoline+0xe=0A>>> 918 100587 nfsd nfsd: service mi_switch+0xe1= =0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a= =0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> f= ork_trampoline+0xe=0A>>> 918 100588 nfsd nfsd: service mi_switch+0xe1=0A>= >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>= svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_tr= ampoline+0xe=0A>>> 918 100589 nfsd nfsd: service mi_switch+0xe1=0A>>> sle= epq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_r= un_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoli= ne+0xe=0A>>> 918 100590 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_ca= tch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe= =0A>>> 918 100591 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+= 0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>>= 918 100592 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+= 0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 1= 00593 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab s= leepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_th= read_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100594 = nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_= wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_s= tart+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100595 nfsd n= fsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_s= ig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0= xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100596 nfsd nfsd: s= ervice mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb for= k_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100597 nfsd nfsd: service= mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_w= ait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100598 nfsd nfsd: service mi_sw= itch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_si= g+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>> fork_trampoline+0xe=0A>>> 918 100599 nfsd nfsd: service mi_switch+= 0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1= 6a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>= fork_trampoline+0xe=0A>>> 918 100600 nfsd nfsd: service mi_switch+0xe1= =0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a= =0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> f= ork_trampoline+0xe=0A>>> 918 100601 nfsd nfsd: service mi_switch+0xe1=0A>= >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>= svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_tr= ampoline+0xe=0A>>> 918 100602 nfsd nfsd: service mi_switch+0xe1=0A>>> sle= epq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_r= un_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoli= ne+0xe=0A>>> 918 100603 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_ca= tch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe= =0A>>> 918 100604 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+= 0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>>= 918 100605 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+= 0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 1= 00606 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab s= leepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_th= read_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100607 = nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_= wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_s= tart+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100608 nfsd n= fsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_s= ig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0= xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100609 nfsd nfsd: s= ervice mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb for= k_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100610 nfsd nfsd: service= mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_w= ait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100611 nfsd nfsd: service mi_sw= itch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_si= g+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>> fork_trampoline+0xe=0A>>> 918 100612 nfsd nfsd: service mi_switch+= 0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1= 6a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>= fork_trampoline+0xe=0A>>> 918 100613 nfsd nfsd: service mi_switch+0xe1= =0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a= =0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> f= ork_trampoline+0xe=0A>>> 918 100614 nfsd nfsd: service mi_switch+0xe1=0A>= >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>= svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_tr= ampoline+0xe=0A>>> 918 100615 nfsd nfsd: service mi_switch+0xe1=0A>>> sle= epq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_r= un_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoli= ne+0xe=0A>>> 918 100616 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_ca= tch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe= =0A>>> 918 100617 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+= 0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>>= 918 100618 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+= 0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 1= 00619 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab s= leepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_th= read_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100620 = nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_= wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_s= tart+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100621 nfsd n= fsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_s= ig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0= xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100622 nfsd nfsd: s= ervice mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb for= k_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100623 nfsd nfsd: service= mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_w= ait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100624 nfsd nfsd: service mi_sw= itch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_si= g+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>> fork_trampoline+0xe=0A>>> 918 100625 nfsd nfsd: service mi_switch+= 0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1= 6a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>= fork_trampoline+0xe=0A>>> 918 100626 nfsd nfsd: service mi_switch+0xe1= =0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a= =0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> f= ork_trampoline+0xe=0A>>> 918 100627 nfsd nfsd: service mi_switch+0xe1=0A>= >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>= svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_tr= ampoline+0xe=0A>>> 918 100628 nfsd nfsd: service mi_switch+0xe1=0A>>> sle= epq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_r= un_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoli= ne+0xe=0A>>> 918 100629 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_ca= tch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe= =0A>>> 918 100630 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+= 0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>>= 918 100631 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+= 0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 1= 00632 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab s= leepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_th= read_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100633 = nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_= wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_s= tart+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100634 nfsd n= fsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_s= ig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0= xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100635 nfsd nfsd: s= ervice mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb for= k_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100636 nfsd nfsd: service= mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_w= ait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100637 nfsd nfsd: service mi_sw= itch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_si= g+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>> fork_trampoline+0xe=0A>>> 918 100638 nfsd nfsd: service mi_switch+= 0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1= 6a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>= fork_trampoline+0xe=0A>>> 918 100639 nfsd nfsd: service mi_switch+0xe1= =0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a= =0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> f= ork_trampoline+0xe=0A>>> 918 100640 nfsd nfsd: service mi_switch+0xe1=0A>= >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>= svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_tr= ampoline+0xe=0A>>> 918 100641 nfsd nfsd: service mi_switch+0xe1=0A>>> sle= epq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_r= un_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoli= ne+0xe=0A>>> 918 100642 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_ca= tch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe= =0A>>> 918 100643 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+= 0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>>= 918 100644 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+= 0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 1= 00645 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab s= leepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_th= read_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100646 = nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_= wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_s= tart+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100647 nfsd n= fsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_s= ig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0= xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100648 nfsd nfsd: s= ervice mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb for= k_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100649 nfsd nfsd: service= mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_w= ait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100650 nfsd nfsd: service mi_sw= itch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_si= g+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>> fork_trampoline+0xe=0A>>> 918 100651 nfsd nfsd: service mi_switch+= 0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1= 6a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>= fork_trampoline+0xe=0A>>> 918 100652 nfsd nfsd: service mi_switch+0xe1= =0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a= =0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> f= ork_trampoline+0xe=0A>>> 918 100653 nfsd nfsd: service mi_switch+0xe1=0A>= >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>= svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_tr= ampoline+0xe=0A>>> 918 100654 nfsd nfsd: service mi_switch+0xe1=0A>>> sle= epq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_r= un_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoli= ne+0xe=0A>>> 918 100655 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_ca= tch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe= =0A>>> 918 100656 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+= 0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>>= 918 100657 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+= 0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 1= 00658 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab s= leepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_th= read_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100659 = nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_= wait_sig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_s= tart+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100660 nfsd n= fsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_s= ig+0xf _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0= xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100661 nfsd nfsd: s= ervice mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb for= k_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100662 nfsd nfsd: service= mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_w= ait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a=0A>>> fork_trampoline+0xe=0A>>> ---=0A>>> =0A>>> Now if we look at = client (FreeBSD 9.3)=0A>>> =0A>>> We see system was very busy and do many= and many interrupts=0A>>> =0A>>> CPU: 0.0% user, 0.0% nice, 37.8% system= , 51.2% interrupt, 11.0%=0A>>> idle=0A>>> =0A>>> A look at process list s= hows that there are many sendmail process in=0A>>> state nfstry=0A>>> =0A= >>> nfstry 18 32:27 0.88% sendmail: Queue runner@00:30:00 for=0A>>> /var/= spool/clientm=0A>>> =0A>>> Here is 'nfsstat -c' output:=0A>>> =0A>>> Clie= nt Info:=0A>>> Rpc Counts:=0A>>> Getattr Setattr Lookup Readlink Read Wri= te Create=0A>>> Remove=0A>>> 1051347 1724 2494481 118 903902 1901285 1626= 76=0A>>> 161899=0A>>> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus=0A= >>> Access=0A>>> 36744 2 0 114 40 3131 0=0A>>> 544136=0A>>> Mknod Fsstat = Fsinfo PathConf Commit=0A>>> 9 0 0 0 245821=0A>>> Rpc Info:=0A>>> TimedOu= t Invalid X Replies Retries Requests=0A>>> 0 0 0 0 8356557=0A>>> Cache In= fo:=0A>>> Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits=0A= >>> Misses=0A>>> 108754455 491475 54229224 2437229 46814561 821723 513212= 3=0A>>> 1871871=0A>>> BioRLHits Misses BioD Hits Misses DirE Hits Misses = Accs Hits=0A>>> Misses=0A>>> 144035 118 53736 2753 27813 1 57238839=0A>>>= 544205=0A>>> =0A>>> If you need more things, tell me, i let the PoC in t= his state.=0A>>> =0A>>> Thanks=0A>>> =0A>>> Regards,=0A>>> =0A>>> Lo=C3= =AFc Blot,=0A>>> UNIX Systems, Network and Security Engineer=0A>>> http:/= /www.unix-experience.fr=0A>>> =0A>>> 21 d=C3=A9cembre 2014 01:33 "Rick Ma= cklem" <rmacklem@uoguelph.ca> a =C3=A9crit:=0A>>> =0A>>> Loic Blot wrote:= =0A>>> =0A>>>> Hi Rick,=0A>>>> ok, i don't need locallocks, i haven't und= erstand option was for=0A>>>> that=0A>>>> usage, i removed it.=0A>>>> I d= o more tests on monday.=0A>>>> Thanks for the deadlock fix, for other peo= ple :)=0A>>> =0A>>> Good. Please let us know if running with vfs.nfsd.ena= ble_locallocks=3D0=0A>>> gets rid of the deadlocks? (I think it fixes the= one you saw.)=0A>>> =0A>>> On the performance side, you might also want = to try different values=0A>>> of=0A>>> readahead, if the Linux client has= such a mount option. (With the=0A>>> NFSv4-ZFS sequential vs random I/O = heuristic, I have no idea what the=0A>>> optimal readahead value would be= .)=0A>>> =0A>>> Good luck with it and please let us know how it goes, ric= k=0A>>> ps: I now have a patch to fix the deadlock when=0A>>> vfs.nfsd.en= able_locallocks=3D1=0A>>> is set. I'll post it for anyone who is interest= ed after I put it=0A>>> through some testing.=0A>>> =0A>>> --=0A>>> Best = regards,=0A>>> Lo=C3=AFc BLOT,=0A>>> UNIX systems, security and network e= ngineer=0A>>> http://www.unix-experience.fr=0A>>>; =0A>>> Le jeudi 18 d=C3= =A9cembre 2014 =C3=A0 19:46 -0500, Rick Macklem a =C3=A9crit :=0A>>> =0A>= >> Loic Blot wrote: =0A>>>> Hi rick,=0A>>>> i tried to start a LXC contai= ner on Debian Squeeze from my=0A>>>> freebsd=0A>>>> ZFS+NFSv4 server and = i also have a deadlock on nfsd=0A>>>> (vfs.lookup_shared=3D0). Deadlock p= rocs each time i launch a=0A>>>> squeeze=0A>>>> container, it seems (3 tr= ies, 3 fails).=0A>>> =0A>>> Well, I`ll take a look at this `procstat -kk`= , but the only thing=0A>>> I`ve seen posted w.r.t. avoiding deadlocks in = ZFS is to not use=0A>>> nullfs. (I have no idea if you are using any null= fs mounts, but=0A>>> if so, try getting rid of them.)=0A>>> =0A>>> Here`s= a high level post about the ZFS and vnode locking problem,=0A>>> but the= re is no patch available, as far as I know.=0A>>> =0A>>> http://docs.Free= BSD.org/cgi/mid.cgi?54739F41.8030407=0A>>> =0A>>> rick=0A>>> =0A>>> 921 -= D 0:00.02 nfsd: server (nfsd)=0A>>> =0A>>> Here is the procstat -kk=0A>>= > =0A>>> PID TID COMM TDNAME KSTACK=0A>>> 921 100538 nfsd nfsd: master mi= _switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e=0A= >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> nfsvno_advloc= k+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad=0A>>> nfsrvd_locku+0x283= nfsrvd_dorpc+0xec6 nfssvc_program+0x554=0A>>> svc_run_internal+0xc77 svc= _run+0x1de nfsrvd_nfsd+0x1ca=0A>>> nfssvc_nfsd+0x107 sys_nfssvc+0x9c=0A>>= > 921 100572 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100573 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100574 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100575 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100576 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100577 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100578 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100579 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100580 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100581 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100582 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100583 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100584 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100585 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100586 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100587 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100588 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100589 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100590 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100591 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100592 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100593 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100594 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100595 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100596 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100597 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100598 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100599 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100600 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100601 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100602 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100603 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100604 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100605 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100606 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100607 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100608 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100609 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100610 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100611 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100612 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100613 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100614 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100615 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>= > 921 100616 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a _sl= eep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>> nfsrv_getlockfile+0x179 nf= srv_lockctrl+0x21f nfsrvd_lock+0x5b1=0A>>> nfsrvd_dorpc+0xec6 nfssvc_prog= ram+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb fork_exit+0x9= a fork_trampoline+0xe=0A>>> 921 100617 nfsd nfsd: service mi_switch+0xe1= =0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0= x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>= >> fork_trampoline+0xe=0A>>> 921 100618 nfsd nfsd: service mi_switch+0xe1= =0A>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>= nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc= _thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>> 921 100619 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100620 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100621 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100622 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100623 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100624 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100625 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100626 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100627 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100628 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100629 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100630 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100631 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100632 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100633 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100634 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100635 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100636 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100637 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100638 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100639 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100640 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100641 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100642 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100643 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100644 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100645 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100646 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100647 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100648 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100649 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100650 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100651 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100652 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100653 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100654 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100655 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100656 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100657 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100658 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100659 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100660 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100661 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100662 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100663 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100664 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100665 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 921 100666 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a _sleep+0x287 nfsmsl= eep+0x66 nfsv4_lock+0x9b=0A>>> nfsrv_setclient+0xbd nfsrvd_setclientid+0x= 3c8=0A>>> nfsrvd_dorpc+0xc76=0A>>> nfssvc_program+0x554 svc_run_internal+= 0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe= =0A>>> =0A>>> Regards,=0A>>> =0A>>> Lo=C3=AFc Blot,=0A>>> UNIX Systems, N= etwork and Security Engineer=0A>>> http://www.unix-experience.fr=0A>>>; = =0A>>> 15 d=C3=A9cembre 2014 15:18 "Rick Macklem" <rmacklem@uoguelph.ca> = a=0A>>> =C3=A9crit:=0A>>> =0A>>> Loic Blot wrote:=0A>>> =0A>>>> For more = informations, here is procstat -kk on nfsd, if you=0A>>>> need=0A>>>> mor= e=0A>>>> hot datas, tell me.=0A>>>> =0A>>>> Regards, PID TID COMM TDNAME = KSTACK=0A>>>> 918 100529 nfsd nfsd: master mi_switch+0xe1=0A>>>> sleepq_w= ait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>> vop_stdlock+0x3c VOP_L= OCK1_APV+0xab _vn_lock+0x43=0A>>>> zfs_fhtovp+0x38d=0A>>>> nfsvno_fhtovp+= 0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>> nfssvc_program+0x554 svc_= run_internal+0xc77 svc_run+0x1de=0A>>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x1= 07 sys_nfssvc+0x9c=0A>>>> amd64_syscall+0x351=0A>>> =0A>>> Well, most of = the threads are stuck like this one, waiting for=0A>>> a=0A>>> vnode=0A>>= > lock in ZFS. All of them appear to be in zfs_fhtovp().=0A>>> I`m not a = ZFS guy, so I can`t help much. I`ll try changing the=0A>>> subject line= =0A>>> to include ZFS vnode lock, so maybe the ZFS guys will take a=0A>>>= look.=0A>>> =0A>>> The only thing I`ve seen suggested is trying:=0A>>> s= ysctl vfs.lookup_shared=3D0=0A>>> to disable shared vop_lookup()s. Appare= ntly zfs_lookup()=0A>>> doesn`t=0A>>> obey the vnode locking rules for lo= okup and rename, according=0A>>> to=0A>>> the posting I saw.=0A>>> =0A>>>= I`ve added a couple of comments about the other threads below,=0A>>> but= =0A>>> they are all either waiting for an RPC request or waiting for=0A>>= > the=0A>>> threads stuck on the ZFS vnode lock to complete.=0A>>> =0A>>>= rick=0A>>> =0A>>>> 918 100564 nfsd nfsd: service mi_switch+0xe1=0A>>>> s= leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A= >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fo= rk_trampoline+0xe=0A>>> =0A>>> Fyi, this thread is just waiting for an RP= C to arrive. (Normal)=0A>>> =0A>>>> 918 100565 nfsd nfsd: service mi_swit= ch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_w= ait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exi= t+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 918 100566 nfsd nfsd: service mi= _switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> = _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb for= k_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 918 100567 nfsd nfsd: servi= ce mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A= >>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0x= b fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 918 100568 nfsd nfsd: = service mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0= xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_sta= rt+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 918 100569 nfsd n= fsd: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_= sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 918 100570 n= fsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_= wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_= thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 918 100= 571 nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_wait+0x3a _sleep+0x28= 7 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>> nfsrvd_dorpc+0x316 nfssvc_program= +0x554 svc_run_internal+0xc77=0A>>>> svc_thread_start+0xb fork_exit+0x9a = fork_trampoline+0xe=0A>>>> 918 100572 nfsd nfsd: service mi_switch+0xe1= =0A>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>= >> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8=0A>>>> nfsrvd_dorpc+0xc7= 6=0A>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>> svc_thread_st= art+0xb=0A>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>> =0A>>> This one (= and a few others) are waiting for the nfsv4_lock.=0A>>> This=0A>>> happen= s=0A>>> because other threads are stuck with RPCs in progress. (ie. The= =0A>>> ones=0A>>> waiting on the vnode lock in zfs_fhtovp().)=0A>>> For t= hese, the RPC needs to lock out other threads to do the=0A>>> operation,= =0A>>> so it waits for the nfsv4_lock() which can exclusively lock the=0A= >>> NFSv4=0A>>> data structures once all other nfsd threads complete thei= r RPCs=0A>>> in=0A>>> progress.=0A>>> =0A>>>> 918 100573 nfsd nfsd: servi= ce mi_switch+0xe1=0A>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfs= v4_lock+0x9b=0A>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_inter= nal+0xc77=0A>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe= =0A>>> =0A>>> Same as above.=0A>>> =0A>>>> 918 100574 nfsd nfsd: service = mi_switch+0xe1=0A>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902= =0A>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>> zfs_fhto= vp+0x38d=0A>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A= >>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>> svc_thread_start+= 0xb=0A>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>> 918 100575 nfsd nfsd= : service mi_switch+0xe1=0A>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_= args+0x902=0A>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>= > zfs_fhtovp+0x38d=0A>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorp= c+0x917=0A>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>> svc_thr= ead_start+0xb=0A>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>> 918 100576= nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_wait+0x3a sleeplk+0x15d = __lockmgr_args+0x902=0A>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+= 0x43=0A>>>> zfs_fhtovp+0x38d=0A>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 n= fsrvd_dorpc+0x917=0A>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>= >> svc_thread_start+0xb=0A>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>> = 918 100577 nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_wait+0x3a slee= plk+0x15d __lockmgr_args+0x902=0A>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab= _vn_lock+0x43=0A>>>> zfs_fhtovp+0x38d=0A>>>> nfsvno_fhtovp+0x7c nfsd_fht= ovp+0xc8 nfsrvd_dorpc+0x917=0A>>>> nfssvc_program+0x554 svc_run_internal+= 0xc77=0A>>>> svc_thread_start+0xb=0A>>>> fork_exit+0x9a fork_trampoline+0= xe=0A>>>> 918 100578 nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_wait= +0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>> vop_stdlock+0x3c VOP_LOCK= 1_APV+0xab _vn_lock+0x43=0A>>>> zfs_fhtovp+0x38d=0A>>>> nfsvno_fhtovp+0x7= c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>> nfssvc_program+0x554 svc_run= _internal+0xc77=0A>>>> svc_thread_start+0xb=0A>>>> fork_exit+0x9a fork_tr= ampoline+0xe=0A>>>> 918 100579 nfsd nfsd: service mi_switch+0xe1=0A>>>> s= leepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>> vop_stdlock+0x3= c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>> zfs_fhtovp+0x38d=0A>>>> nfsvno_= fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>> nfssvc_program+0x5= 54 svc_run_internal+0xc77=0A>>>> svc_thread_start+0xb=0A>>>> fork_exit+0x= 9a fork_trampoline+0xe=0A>>>> 918 100580 nfsd nfsd: service mi_switch+0xe= 1=0A>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>> vop_s= tdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>> zfs_fhtovp+0x38d=0A>>= >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>> nfssvc_p= rogram+0x554 svc_run_internal+0xc77=0A>>>> svc_thread_start+0xb=0A>>>> fo= rk_exit+0x9a fork_trampoline+0xe=0A>>>> 918 100581 nfsd nfsd: service mi_= switch+0xe1=0A>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A= >>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>> zfs_fhtovp+= 0x38d=0A>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>= > nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>> svc_thread_start+0xb= =0A>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>> 918 100582 nfsd nfsd: s= ervice mi_switch+0xe1=0A>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_arg= s+0x902=0A>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>> z= fs_fhtovp+0x38d=0A>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0= x917=0A>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>> svc_thread= _start+0xb=0A>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>> 918 100583 nf= sd nfsd: service mi_switch+0xe1=0A>>>> sleepq_wait+0x3a sleeplk+0x15d __l= ockmgr_args+0x902=0A>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x4= 3=0A>>>> zfs_fhtovp+0x38d=0A>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsr= vd_dorpc+0x917=0A>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>> = svc_thread_start+0xb=0A>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>> 918= 100584 nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_wait+0x3a sleeplk= +0x15d __lockmgr_args+0x902=0A>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _v= n_lock+0x43=0A>>>> zfs_fhtovp+0x38d=0A>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp= +0xc8 nfsrvd_dorpc+0x917=0A>>>> nfssvc_program+0x554 svc_run_internal+0xc= 77=0A>>>> svc_thread_start+0xb=0A>>>> fork_exit+0x9a fork_trampoline+0xe= =0A>>>> 918 100585 nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_wait+0= x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>> vop_stdlock+0x3c VOP_LOCK1_= APV+0xab _vn_lock+0x43=0A>>>> zfs_fhtovp+0x38d=0A>>>> nfsvno_fhtovp+0x7c = nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>> nfssvc_program+0x554 svc_run_i= nternal+0xc77=0A>>>> svc_thread_start+0xb=0A>>>> fork_exit+0x9a fork_tram= poline+0xe=0A>>>> 918 100586 nfsd nfsd: service mi_switch+0xe1=0A>>>> sle= epq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>> vop_stdlock+0x3c = VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>> zfs_fhtovp+0x38d=0A>>>> nfsvno_fh= tovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>> nfssvc_program+0x554= svc_run_internal+0xc77=0A>>>> svc_thread_start+0xb=0A>>>> fork_exit+0x9a= fork_trampoline+0xe=0A>>>> 918 100587 nfsd nfsd: service mi_switch+0xe1= =0A>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>> vop_st= dlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>> zfs_fhtovp+0x38d=0A>>>= > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>> nfssvc_pr= ogram+0x554 svc_run_internal+0xc77=0A>>>> svc_thread_start+0xb=0A>>>> for= k_exit+0x9a fork_trampoline+0xe=0A>>>> 918 100588 nfsd nfsd: service mi_s= witch+0xe1=0A>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>= >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>> zfs_fhtovp+0= x38d=0A>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>= nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>> svc_thread_start+0xb= =0A>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>> 918 100589 nfsd nfsd: s= ervice mi_switch+0xe1=0A>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_arg= s+0x902=0A>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>> z= fs_fhtovp+0x38d=0A>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0= x917=0A>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>> svc_thread= _start+0xb=0A>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>> 918 100590 nf= sd nfsd: service mi_switch+0xe1=0A>>>> sleepq_wait+0x3a sleeplk+0x15d __l= ockmgr_args+0x902=0A>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x4= 3=0A>>>> zfs_fhtovp+0x38d=0A>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsr= vd_dorpc+0x917=0A>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>> = svc_thread_start+0xb=0A>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>> 918= 100591 nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_wait+0x3a sleeplk= +0x15d __lockmgr_args+0x902=0A>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _v= n_lock+0x43=0A>>>> zfs_fhtovp+0x38d=0A>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp= +0xc8 nfsrvd_dorpc+0x917=0A>>>> nfssvc_program+0x554 svc_run_internal+0xc= 77=0A>>>> svc_thread_start+0xb=0A>>>> fork_exit+0x9a fork_trampoline+0xe= =0A>>>> 918 100592 nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_wait+0= x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>> vop_stdlock+0x3c VOP_LOCK1_= APV+0xab _vn_lock+0x43=0A>>>> zfs_fhtovp+0x38d=0A>>>> nfsvno_fhtovp+0x7c = nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>> nfssvc_program+0x554 svc_run_i= nternal+0xc77=0A>>>> svc_thread_start+0xb=0A>>>> fork_exit+0x9a fork_tram= poline+0xe=0A>>>> 918 100593 nfsd nfsd: service mi_switch+0xe1=0A>>>> sle= epq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>> vop_stdlock+0x3c = VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>> zfs_fhtovp+0x38d=0A>>>> nfsvno_fh= tovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>> nfssvc_program+0x554= svc_run_internal+0xc77=0A>>>> svc_thread_start+0xb=0A>>>> fork_exit+0x9a= fork_trampoline+0xe=0A>>>> 918 100594 nfsd nfsd: service mi_switch+0xe1= =0A>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>> vop_st= dlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>> zfs_fhtovp+0x38d=0A>>>= > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>> nfssvc_pr= ogram+0x554 svc_run_internal+0xc77=0A>>>> svc_thread_start+0xb=0A>>>> for= k_exit+0x9a fork_trampoline+0xe=0A>>>> 918 100595 nfsd nfsd: service mi_s= witch+0xe1=0A>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>= >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>> zfs_fhtovp+0= x38d=0A>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>= nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>> svc_thread_start+0xb= =0A>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>> 918 100596 nfsd nfsd: s= ervice mi_switch+0xe1=0A>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_arg= s+0x902=0A>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>> z= fs_fhtovp+0x38d=0A>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0= x917=0A>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>> svc_thread= _start+0xb=0A>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>> 918 100597 nf= sd nfsd: service mi_switch+0xe1=0A>>>> sleepq_wait+0x3a sleeplk+0x15d __l= ockmgr_args+0x902=0A>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x4= 3=0A>>>> zfs_fhtovp+0x38d=0A>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsr= vd_dorpc+0x917=0A>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>> = svc_thread_start+0xb=0A>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>> 918= 100598 nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_wait+0x3a sleeplk= +0x15d __lockmgr_args+0x902=0A>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _v= n_lock+0x43=0A>>>> zfs_fhtovp+0x38d=0A>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp= +0xc8 nfsrvd_dorpc+0x917=0A>>>> nfssvc_program+0x554 svc_run_internal+0xc= 77=0A>>>> svc_thread_start+0xb=0A>>>> fork_exit+0x9a fork_trampoline+0xe= =0A>>>> 918 100599 nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_wait+0= x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>> vop_stdlock+0x3c VOP_LOCK1_= APV+0xab _vn_lock+0x43=0A>>>> zfs_fhtovp+0x38d=0A>>>> nfsvno_fhtovp+0x7c = nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>> nfssvc_program+0x554 svc_run_i= nternal+0xc77=0A>>>> svc_thread_start+0xb=0A>>>> fork_exit+0x9a fork_tram= poline+0xe=0A>>>> 918 100600 nfsd nfsd: service mi_switch+0xe1=0A>>>> sle= epq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>> vop_stdlock+0x3c = VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>> zfs_fhtovp+0x38d=0A>>>> nfsvno_fh= tovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>> nfssvc_program+0x554= svc_run_internal+0xc77=0A>>>> svc_thread_start+0xb=0A>>>> fork_exit+0x9a= fork_trampoline+0xe=0A>>>> 918 100601 nfsd nfsd: service mi_switch+0xe1= =0A>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>> vop_st= dlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>> zfs_fhtovp+0x38d=0A>>>= > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>> nfssvc_pr= ogram+0x554 svc_run_internal+0xc77=0A>>>> svc_thread_start+0xb=0A>>>> for= k_exit+0x9a fork_trampoline+0xe=0A>>>> 918 100602 nfsd nfsd: service mi_s= witch+0xe1=0A>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>= >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>> zfs_fhtovp+0= x38d=0A>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>= nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>> svc_thread_start+0xb= =0A>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>> 918 100603 nfsd nfsd: s= ervice mi_switch+0xe1=0A>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_arg= s+0x902=0A>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>> z= fs_fhtovp+0x38d=0A>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0= x917=0A>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>> svc_thread= _start+0xb=0A>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>> 918 100604 nf= sd nfsd: service mi_switch+0xe1=0A>>>> sleepq_wait+0x3a sleeplk+0x15d __l= ockmgr_args+0x902=0A>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x4= 3=0A>>>> zfs_fhtovp+0x38d=0A>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsr= vd_dorpc+0x917=0A>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>> = svc_thread_start+0xb=0A>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>> 918= 100605 nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_wait+0x3a sleeplk= +0x15d __lockmgr_args+0x902=0A>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _v= n_lock+0x43=0A>>>> zfs_fhtovp+0x38d=0A>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp= +0xc8 nfsrvd_dorpc+0x917=0A>>>> nfssvc_program+0x554 svc_run_internal+0xc= 77=0A>>>> svc_thread_start+0xb=0A>>>> fork_exit+0x9a fork_trampoline+0xe= =0A>>>> 918 100606 nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_wait+0= x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>> vop_stdlock+0x3c VOP_LOCK1_= APV+0xab _vn_lock+0x43=0A>>>> zfs_fhtovp+0x38d=0A>>>> nfsvno_fhtovp+0x7c = nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>> nfssvc_program+0x554 svc_run_i= nternal+0xc77=0A>>>> svc_thread_start+0xb=0A>>>> fork_exit+0x9a fork_tram= poline+0xe=0A>>>> 918 100607 nfsd nfsd: service mi_switch+0xe1=0A>>>> sle= epq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>> vop_stdlock+0x3c = VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>> zfs_fhtovp+0x38d=0A>>>> nfsvno_fh= tovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>> nfssvc_program+0x554= svc_run_internal+0xc77=0A>>>> svc_thread_start+0xb=0A>>>> fork_exit+0x9a= fork_trampoline+0xe=0A>>> =0A>>> Lots more waiting for the ZFS vnode loc= k in zfs_fhtovp().=0A>>> =0A>>> 918 100608 nfsd nfsd: service mi_switch+0= xe1=0A>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A= >>> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1=0A>>> = nfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc_= thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 100609 nfsd= nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lock= mgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A= >>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dor= pc+0x917=0A>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc_thre= ad_start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 100610 nf= sd nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lo= ckmgr_args+0xc9e=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43= =0A>>> nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad=0A>>= > nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554=0A>>> svc_ru= n_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampolin= e+0xe=0A>>> 918 100611 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_wai= t+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>> nfsrvd_dorpc+0x3= 16 nfssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb= fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 100612 nfsd nfsd: service m= i_switch+0xe1=0A>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lo= ck+0x9b=0A>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0x= c77=0A>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>> 9= 18 100613 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a _sleep= +0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>> nfsrvd_dorpc+0x316 nfssvc_pro= gram+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb fork_exit+0x= 9a fork_trampoline+0xe=0A>>> 918 100614 nfsd nfsd: service mi_switch+0xe1= =0A>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>= nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc= _thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 100615 nfs= d nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a _sleep+0x287 nfsmsl= eep+0x66 nfsv4_lock+0x9b=0A>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 sv= c_run_internal+0xc77=0A>>> svc_thread_start+0xb fork_exit+0x9a fork_tramp= oline+0xe=0A>>> 918 100616 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq= _wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>> nfsrvd_dorpc= +0x316 nfssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start= +0xb fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 100617 nfsd nfsd: servi= ce mi_switch+0xe1=0A>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv= 4_lock+0x9b=0A>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_interna= l+0xc77=0A>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>= >> 918 100618 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a _s= leep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>> nfsrvd_dorpc+0x316 nfssvc= _program+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb fork_exi= t+0x9a fork_trampoline+0xe=0A>>> 918 100619 nfsd nfsd: service mi_switch+= 0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_= stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>= > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_pro= gram+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_e= xit+0x9a fork_trampoline+0xe=0A>>> 918 100620 nfsd nfsd: service mi_switc= h+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vo= p_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A= >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_p= rogram+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork= _exit+0x9a fork_trampoline+0xe=0A>>> 918 100621 nfsd nfsd: service mi_swi= tch+0xe1=0A>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x= 9b=0A>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77= =0A>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 = 100622 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0= x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_l= ock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8= nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>= >> svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>> 91= 8 100623 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a _sleep+= 0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>> nfsrvd_dorpc+0x316 nfssvc_prog= ram+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb fork_exit+0x9= a fork_trampoline+0xe=0A>>> 918 100624 nfsd nfsd: service mi_switch+0xe1= =0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdl= ock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nf= svno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program= +0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+= 0x9a fork_trampoline+0xe=0A>>> 918 100625 nfsd nfsd: service mi_switch+0x= e1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_st= dlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> = nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_progr= am+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exi= t+0x9a fork_trampoline+0xe=0A>>> 918 100626 nfsd nfsd: service mi_switch+= 0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_= stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>= > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_pro= gram+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_e= xit+0x9a fork_trampoline+0xe=0A>>> 918 100627 nfsd nfsd: service mi_switc= h+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vo= p_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A= >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_p= rogram+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork= _exit+0x9a fork_trampoline+0xe=0A>>> 918 100628 nfsd nfsd: service mi_swi= tch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> = vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d= =0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssv= c_program+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> f= ork_exit+0x9a fork_trampoline+0xe=0A>>> 918 100629 nfsd nfsd: service mi_= switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>= >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x3= 8d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfs= svc_program+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>>= fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 100630 nfsd nfsd: service m= i_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902= =0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp= +0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>= nfssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb= =0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 100631 nfsd nfsd: ser= vice mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0= x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fh= tovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917= =0A>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start= +0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 100632 nfsd nfsd:= service mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_ar= gs+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zf= s_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9= 17=0A>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc_thread_sta= rt+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 100633 nfsd nfs= d: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_= args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> = zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0= x917=0A>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc_thread_s= tart+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 100634 nfsd n= fsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmg= r_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>= > zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc= +0x917=0A>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc_thread= _start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 100635 nfsd= nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lock= mgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A= >>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dor= pc+0x917=0A>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc_thre= ad_start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 100636 nf= sd nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lo= ckmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43= =0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_= dorpc+0x917=0A>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc_t= hread_start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 100637= nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d _= _lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x= 43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrv= d_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc= _thread_start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 1006= 38 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d= __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+= 0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfs= rvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>> s= vc_thread_start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 10= 0639 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x1= 5d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_loc= k+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 n= fsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>= svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 = 100640 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0= x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_l= ock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8= nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>= >> svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>> 91= 8 100641 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk= +0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn= _lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0x= c8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_run_internal+0xc77= =0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>= > 918 100642 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a sle= eplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab= _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtov= p+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_run_internal+0xc= 77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A= >>> 918 100643 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a s= leeplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0x= ab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fht= ovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_run_internal+0= xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe= =0A>>> 918 100644 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3= a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV= +0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_= fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_run_interna= l+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0= xe=0A>>> 918 100645 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0= x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_A= PV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfs= d_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_run_inter= nal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_trampoline= +0xe=0A>>> 918 100646 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_wait= +0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1= _APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c n= fsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_run_int= ernal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_trampoli= ne+0xe=0A>>> 918 100647 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_wa= it+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOC= K1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c= nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_run_i= nternal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_trampo= line+0xe=0A>>> 918 100648 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_= wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_L= OCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x= 7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_run= _internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_tram= poline+0xe=0A>>> 918 100649 nfsd nfsd: service mi_switch+0xe1=0A>>> sleep= q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP= _LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+= 0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_r= un_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_tr= ampoline+0xe=0A>>> 918 100650 nfsd nfsd: service mi_switch+0xe1=0A>>> sle= epq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c V= OP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtov= p+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc= _run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_= trampoline+0xe=0A>>> 918 100651 nfsd nfsd: service mi_switch+0xe1=0A>>> s= leepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c= VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fht= ovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 s= vc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a for= k_trampoline+0xe=0A>>> 918 100652 nfsd nfsd: service mi_switch+0xe1=0A>>>= sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x= 3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_f= htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554= svc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a f= ork_trampoline+0xe=0A>>> 918 100653 nfsd nfsd: service mi_switch+0xe1=0A>= >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+= 0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno= _fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x5= 54 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a= fork_trampoline+0xe=0A>>> 918 100654 nfsd nfsd: service mi_switch+0xe1= =0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdl= ock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nf= svno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program= +0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+= 0x9a fork_trampoline+0xe=0A>>> 918 100655 nfsd nfsd: service mi_switch+0x= e1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_st= dlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> = nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_progr= am+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exi= t+0x9a fork_trampoline+0xe=0A>>> 918 100656 nfsd nfsd: service mi_switch+= 0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_= stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>= > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_pro= gram+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_e= xit+0x9a fork_trampoline+0xe=0A>>> 918 100657 nfsd nfsd: service mi_switc= h+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vo= p_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A= >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_p= rogram+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork= _exit+0x9a fork_trampoline+0xe=0A>>> 918 100658 nfsd nfsd: service mi_swi= tch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> = vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d= =0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssv= c_program+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> f= ork_exit+0x9a fork_trampoline+0xe=0A>>> =0A>>> Lo=C3=AFc Blot,=0A>>> UNIX= Systems, Network and Security Engineer=0A>>> http://www.unix-experience.= fr=0A>>> =0A>>> 15 d=C3=A9cembre 2014 13:29 "Lo=C3=AFc Blot"=0A>>> <loic.= blot@unix-experience.fr>=0A>>> a=0A>>> =C3=A9crit:=0A>>> =0A>>> Hmmm...= =0A>>> now i'm experiencing a deadlock.=0A>>> =0A>>> 0 918 915 0 21 0 123= 52 3372 zfs D - 1:48.64 nfsd: server=0A>>> (nfsd)=0A>>> =0A>>> the only i= ssue was to reboot the server, but after rebooting=0A>>> deadlock arrives= a second time when i=0A>>> start my jails over NFS.=0A>>> =0A>>> Regards= ,=0A>>> =0A>>> Lo=C3=AFc Blot,=0A>>> UNIX Systems, Network and Security E= ngineer=0A>>> http://www.unix-experience.fr=0A>>>; =0A>>> 15 d=C3=A9cembre= 2014 10:07 "Lo=C3=AFc Blot"=0A>>> <loic.blot@unix-experience.fr>=0A>>> a= =0A>>> =C3=A9crit:=0A>>> =0A>>> Hi Rick,=0A>>> after talking with my N+1,= NFSv4 is required on our=0A>>> infrastructure.=0A>>> I tried to upgrade = NFSv4+ZFS=0A>>> server from 9.3 to 10.1, i hope this will resolve some=0A= >>> issues...=0A>>> =0A>>> Regards,=0A>>> =0A>>> Lo=C3=AFc Blot,=0A>>> UN= IX Systems, Network and Security Engineer=0A>>> http://www.unix-experienc= e.fr=0A>>> =0A>>> 10 d=C3=A9cembre 2014 15:36 "Lo=C3=AFc Blot"=0A>>> <loi= c.blot@unix-experience.fr>=0A>>> a=0A>>> =C3=A9crit:=0A>>> =0A>>> Hi Rick= ,=0A>>> thanks for your suggestion.=0A>>> For my locking bug, rpc.lockd i= s stucked in rpcrecv state on=0A>>> the=0A>>> server. kill -9 doesn't aff= ect the=0A>>> process, it's blocked.... (State: Ds)=0A>>> =0A>>> for the = performances=0A>>> =0A>>> NFSv3: 60Mbps=0A>>> NFSv4: 45Mbps=0A>>> Regards= ,=0A>>> =0A>>> Lo=C3=AFc Blot,=0A>>> UNIX Systems, Network and Security E= ngineer=0A>>> http://www.unix-experience.fr=0A>>>; =0A>>> 10 d=C3=A9cembre= 2014 13:56 "Rick Macklem" <rmacklem@uoguelph.ca>=0A>>> a=0A>>> =C3=A9cri= t:=0A>>> =0A>>> Loic Blot wrote:=0A>>> =0A>>>> Hi Rick,=0A>>>> I'm trying= NFSv3.=0A>>>> Some jails are starting very well but now i have an issue= =0A>>>> with=0A>>>> lockd=0A>>>> after some minutes:=0A>>>> =0A>>>> nfs s= erver 10.10.X.8:/jails: lockd not responding=0A>>>> nfs server 10.10.X.8:= /jails lockd is alive again=0A>>>> =0A>>>> I look at mbuf, but i seems th= ere is no problem.=0A>>> =0A>>> Well, if you need locks to be visible acr= oss multiple=0A>>> clients,=0A>>> then=0A>>> I'm afraid you are stuck wit= h using NFSv4 and the=0A>>> performance=0A>>> you=0A>>> get=0A>>> from it= . (There is no way to do file handle affinity for=0A>>> NFSv4=0A>>> becau= se=0A>>> the read and write ops are buried in the compound RPC and=0A>>> = not=0A>>> easily=0A>>> recognized.)=0A>>> =0A>>> If the locks don't need = to be visible across multiple=0A>>> clients,=0A>>> I'd=0A>>> suggest tryi= ng the "nolockd" option with nfsv3.=0A>>> =0A>>>> Here is my rc.conf on s= erver:=0A>>>> =0A>>>> nfs_server_enable=3D"YES"=0A>>>> nfsv4_server_enabl= e=3D"YES"=0A>>>> nfsuserd_enable=3D"YES"=0A>>>> nfsd_server_flags=3D"-u -= t -n 256"=0A>>>> mountd_enable=3D"YES"=0A>>>> mountd_flags=3D"-r"=0A>>>> = nfsuserd_flags=3D"-usertimeout 0 -force 20"=0A>>>> rpcbind_enable=3D"YES"= =0A>>>> rpc_lockd_enable=3D"YES"=0A>>>> rpc_statd_enable=3D"YES"=0A>>>> = =0A>>>> Here is the client:=0A>>>> =0A>>>> nfsuserd_enable=3D"YES"=0A>>>>= nfsuserd_flags=3D"-usertimeout 0 -force 20"=0A>>>> nfscbd_enable=3D"YES"= =0A>>>> rpc_lockd_enable=3D"YES"=0A>>>> rpc_statd_enable=3D"YES"=0A>>>> = =0A>>>> Have you got an idea ?=0A>>>> =0A>>>> Regards,=0A>>>> =0A>>>> Lo= =C3=AFc Blot,=0A>>>> UNIX Systems, Network and Security Engineer=0A>>>> h= ttp://www.unix-experience.fr=0A>>>> =0A>>>> 9 d=C3=A9cembre 2014 04:31 "R= ick Macklem" <rmacklem@uoguelph.ca>=0A>>>> a=0A>>>> =C3=A9crit: =0A>>>>> = Loic Blot wrote:=0A>>>>> =0A>>>>>> Hi rick,=0A>>>>>> =0A>>>>>> I waited 3= hours (no lag at jail launch) and now I do:=0A>>>>>> sysrc=0A>>>>>> memc= ached_flags=3D"-v -m 512"=0A>>>>>> Command was very very slow...=0A>>>>>>= =0A>>>>>> Here is a dd over NFS:=0A>>>>>> =0A>>>>>> 601062912 bytes tran= sferred in 21.060679 secs (28539579=0A>>>>>> bytes/sec)=0A>>>>> =0A>>>>> = Can you try the same read using an NFSv3 mount?=0A>>>>> (If it runs much = faster, you have probably been bitten by=0A>>>>> the=0A>>>>> ZFS=0A>>>>> = "sequential vs random" read heuristic which I've been told=0A>>>>> things= =0A>>>>> NFS is doing "random" reads without file handle affinity.=0A>>>>= > File=0A>>>>> handle affinity is very hard to do for NFSv4, so it isn't= =0A>>>>> done.)=0A>>> =0A>>> I was actually suggesting that you try the "= dd" over nfsv3=0A>>> to=0A>>> see=0A>>> how=0A>>> the performance compare= d with nfsv4. If you do that, please=0A>>> post=0A>>> the=0A>>> comparabl= e results.=0A>>> =0A>>> Someday I would like to try and get ZFS's sequent= ial vs=0A>>> random=0A>>> read=0A>>> heuristic modified and any info on w= hat difference in=0A>>> performance=0A>>> that=0A>>> might make for NFS w= ould be useful.=0A>>> =0A>>> rick=0A>>> =0A>>> rick=0A>>> =0A>>> This is = quite slow...=0A>>> =0A>>> You can found some nfsstat below (command isn'= t finished=0A>>> yet)=0A>>> =0A>>> nfsstat -c -w 1=0A>>> =0A>>> GtAttr Lo= okup Rdlink Read Write Rename Access Rddir=0A>>> 0 0 0 0 0 0 0 0=0A>>> 4 = 0 0 0 0 0 16 0=0A>>> 2 0 0 0 0 0 17 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0= 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 4 0 0 0 0 4 = 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>>= 0 0 0 0 0 0 0 0=0A>>> 4 0 0 0 0 0 3 0=0A>>> 0 0 0 0 0 0 3 0=0A>>> 37 10 = 0 8 0 0 14 1=0A>>> 18 16 0 4 1 2 4 0=0A>>> 78 91 0 82 6 12 30 0=0A>>> 19 = 18 0 2 2 4 2 0=0A>>> 0 0 0 0 2 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> GtAttr L= ookup Rdlink Read Write Rename Access Rddir=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0= 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 1 0 0 0 0 1 0=0A>>> 4 6 0 0 = 6 0 3 0=0A>>> 2 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 1 0 0 0 0 0 0 0= =0A>>> 0 0 0 0 1 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> = 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0= 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 6 108 0 0 0 0 = 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> GtAttr Lookup Rdlin= k Read Write Rename Access Rddir=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 = 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>= >> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 98 54 0 86 11 0 25 0=0A>>>= 36 24 0 39 25 0 10 1=0A>>> 67 8 0 63 63 0 41 0=0A>>> 34 0 0 35 34 0 0 0= =0A>>> 75 0 0 75 77 0 0 0=0A>>> 34 0 0 35 35 0 0 0=0A>>> 75 0 0 74 76 0 0= 0=0A>>> 33 0 0 34 33 0 0 0=0A>>> 0 0 0 0 5 0 0 0=0A>>> 0 0 0 0 0 0 6 0= =0A>>> 11 0 0 0 0 0 11 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 17 0 0 0 0 1 0=0A>= >> GtAttr Lookup Rdlink Read Write Rename Access Rddir=0A>>> 4 5 0 0 0 0 = 12 0=0A>>> 2 0 0 0 0 0 26 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0= =0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> = 0 4 0 0 0 0 4 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0= 0 0 0 0=0A>>> 4 0 0 0 0 0 2 0=0A>>> 2 0 0 0 0 0 24 0=0A>>> 0 0 0 0 0 0 0= 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>= > 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> GtAtt= r Lookup Rdlink Read Write Rename Access Rddir=0A>>> 0 0 0 0 0 0 0 0=0A>>= > 0 0 0 0 0 0 0 0=0A>>> 4 0 0 0 0 0 7 0=0A>>> 2 1 0 0 0 0 1 0=0A>>> 0 0 0= 0 2 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 6 0 0 0=0A>>> 0 0 0 0 0 0 = 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>= >> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 4 6 0 0 0 0 3 0=0A>>> 0 0 = 0 0 0 0 0 0=0A>>> 2 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0= 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> GtAttr Lookup Rdli= nk Read Write Rename Access Rddir=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0= 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A= >>> 4 71 0 0 0 0 0 0=0A>>> 0 1 0 0 0 0 0 0=0A>>> 2 36 0 0 0 0 1 0=0A>>> 0= 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 = 0 0 0 0=0A>>> 1 0 0 0 0 0 1 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0= =0A>>> 79 6 0 79 79 0 2 0=0A>>> 25 0 0 25 26 0 6 0=0A>>> 43 18 0 39 46 0 = 23 0=0A>>> 36 0 0 36 36 0 31 0=0A>>> 68 1 0 66 68 0 0 0=0A>>> GtAttr Look= up Rdlink Read Write Rename Access Rddir=0A>>> 36 0 0 36 36 0 0 0=0A>>> 4= 8 0 0 48 49 0 0 0=0A>>> 20 0 0 20 20 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 3 = 14 0 1 0 0 11 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 0 4 0 0= 0 0 4 0=0A>>> 0 0 0 0 0 0 0 0=0A>>> 4 22 0 0 0 0 16 0=0A>>> 2 0 0 0 0 0 = 23 0=0A>>> =0A>>> Regards,=0A>>> =0A>>> Lo=C3=AFc Blot,=0A>>> UNIX System= s, Network and Security Engineer=0A>>> http://www.unix-experience.fr=0A>>= > =0A>>> 8 d=C3=A9cembre 2014 09:36 "Lo=C3=AFc Blot"=0A>>> <loic.blot@uni= x-experience.fr> a=0A>>> =C3=A9crit: =0A>>>> Hi Rick,=0A>>>> I stopped th= e jails this week-end and started it this=0A>>>> morning,=0A>>>> i'll=0A>= >>> give you some stats this week.=0A>>>> =0A>>>> Here is my nfsstat -m o= utput (with your rsize/wsize=0A>>>> tweaks)=0A>> =0A>> =0A> nfsv4,tcp,res= vport,hard,cto,sec=3Dsys,acdirmin=3D3,acdirmax=3D60,acregmin=3D5,acregmax= =3D60,nametimeo=3D60,negna=0A>> =0A>>> =0A>> =0A>> =0A> etimeo=3D60,rsize= =3D32768,wsize=3D32768,readdirsize=3D32768,readahead=3D1,wcommitsize=3D77= 3136,timeout=3D120,retra=0A>> =0A>>> s=3D2147483647=0A>>> =0A>>> On serve= r side my disks are on a raid controller which show a=0A>>> 512b=0A>>> vo= lume and write performances=0A>>> are very honest (dd if=3D/dev/zero of= =3D/jails/test.dd bs=3D4096=0A>>> count=3D100000000 =3D> 450MBps)=0A>>> = =0A>>> Regards,=0A>>> =0A>>> Lo=C3=AFc Blot,=0A>>> UNIX Systems, Network = and Security Engineer=0A>>> http://www.unix-experience.fr=0A>>>; =0A>>> 5 = d=C3=A9cembre 2014 15:14 "Rick Macklem" <rmacklem@uoguelph.ca> a=0A>>> = =C3=A9crit:=0A>>> =0A>>> Loic Blot wrote:=0A>>> =0A>>> Hi,=0A>>> i'm tryi= ng to create a virtualisation environment based on=0A>>> jails.=0A>>> Tho= se jails are stored under a big ZFS pool on a FreeBSD=0A>>> 9.3=0A>>> whi= ch=0A>>> export a NFSv4 volume. This NFSv4 volume was mounted on a=0A>>> = big=0A>>> hypervisor (2 Xeon E5v3 + 128GB memory and 8 ports (but=0A>>> o= nly 1=0A>>> was=0A>>> used at this time).=0A>>> =0A>>> The problem is sim= ple, my hypervisors runs 6 jails (used 1%=0A>>> cpu=0A>>> and=0A>>> 10GB = RAM approximatively and less than 1MB bandwidth) and=0A>>> works=0A>>> fi= ne at start but the system slows down and after 2-3 days=0A>>> become=0A>= >> unusable. When i look at top command i see 80-100% on=0A>>> system=0A>= >> and=0A>>> commands are very very slow. Many process are tagged with=0A= >>> nfs_cl*.=0A>>> =0A>>> To be honest, I would expect the slowness to be= because of=0A>>> slow=0A>>> response=0A>>> from the NFSv4 server, but if= you do:=0A>>> # ps axHl=0A>>> on a client when it is slow and post that,= it would give us=0A>>> some=0A>>> more=0A>>> information on where the cl= ient side processes are sitting.=0A>>> If you also do something like:=0A>= >> # nfsstat -c -w 1=0A>>> and let it run for a while, that should show y= ou how many=0A>>> RPCs=0A>>> are=0A>>> being done and which ones.=0A>>> = =0A>>> # nfsstat -m=0A>>> will show you what your mount is actually using= .=0A>>> The only mount option I can suggest trying is=0A>>> "rsize=3D3276= 8,wsize=3D32768",=0A>>> since some network environments have difficulties= with 64K.=0A>>> =0A>>> There are a few things you can try on the NFSv4 s= erver side,=0A>>> if=0A>>> it=0A>>> appears=0A>>> that the clients are ge= nerating a large RPC load.=0A>>> - disabling the DRC cache for TCP by set= ting=0A>>> vfs.nfsd.cachetcp=3D0=0A>>> - If the server is seeing a large = write RPC load, then=0A>>> "sync=3Ddisabled"=0A>>> might help, although i= t does run a risk of data loss when=0A>>> the=0A>>> server=0A>>> crashes.= =0A>>> Then there are a couple of other ZFS related things (I'm not=0A>>>= a=0A>>> ZFS=0A>>> guy,=0A>>> but these have shown up on the mailing list= s).=0A>>> - make sure your volumes are 4K aligned and ashift=3D12 (in=0A>= >> case a=0A>>> drive=0A>>> that uses 4K sectors is pretending to be 512b= yte sectored)=0A>>> - never run over 70-80% full if write performance is = an=0A>>> issue=0A>>> - use a zil on an SSD with good write performance=0A= >>> =0A>>> The only NFSv4 thing I can tell you is that it is known that= =0A>>> ZFS's=0A>>> algorithm for determining sequential vs random I/O fai= ls for=0A>>> NFSv4=0A>>> during writing and this can be a performance hit= . The only=0A>>> workaround=0A>>> is to use NFSv3 mounts, since file hand= le affinity=0A>>> apparently=0A>>> fixes=0A>>> the problem and this is on= ly done for NFSv3.=0A>>> =0A>>> rick=0A>>> =0A>>> I saw that there are TS= O issues with igb then i'm trying to=0A>>> disable=0A>>> it with sysctl b= ut the situation wasn't solved.=0A>>> =0A>>> Someone has got ideas ? I ca= n give you more informations if=0A>>> you=0A>>> need.=0A>>> =0A>>> Thanks= in advance.=0A>>> Regards,=0A>>> =0A>>> Lo=C3=AFc Blot,=0A>>> UNIX Syste= ms, Network and Security Engineer=0A>>> http://www.unix-experience.fr=0A>= >> _______________________________________________=0A>>> freebsd-fs@freeb= sd.org mailing list=0A>>> http://lists.freebsd.org/mailman/listinfo/freeb= sd-fs=0A>>> To unsubscribe, send any mail to=0A>>> "freebsd-fs-unsubscrib= e@freebsd.org"=0A>>> =0A>>> _____________________________________________= __=0A>>> freebsd-fs@freebsd.org mailing list=0A>>> http://lists.freebsd.o= rg/mailman/listinfo/freebsd-fs=0A>>> To unsubscribe, send any mail to=0A>= >> "freebsd-fs-unsubscribe@freebsd.org"=0A>>> =0A>>> ____________________= ___________________________=0A>>> freebsd-fs@freebsd.org mailing list=0A>= >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A>>>; To unsubscr= ibe, send any mail to=0A>>> "freebsd-fs-unsubscribe@freebsd.org"=0A>>> = =0A>>> _______________________________________________=0A>>> freebsd-fs@f= reebsd.org mailing list=0A>>> http://lists.freebsd.org/mailman/listinfo/f= reebsd-fs=0A>>> To unsubscribe, send any mail to=0A>>> "freebsd-fs-unsubs= cribe@freebsd.org"=0A>>> _______________________________________________= =0A>>> freebsd-fs@freebsd.org mailing list=0A>>> http://lists.freebsd.org= /mailman/listinfo/freebsd-fs=0A>>> To unsubscribe, send any mail to=0A>>>= "freebsd-fs-unsubscribe@freebsd.org"=0A>>> =0A>>> ______________________= _________________________=0A>>> freebsd-fs@freebsd.org mailing list=0A>>>= http://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A>>>; To unsubscrib= e, send any mail to "freebsd-fs-unsubscribe@freebsd.org"=0A> =0A> _______= ________________________________________=0A> freebsd-fs@freebsd.org maili= ng list=0A> http://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A>; To u= nsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?87a8b6ab243024e553b2baba30537b92>