Date: Tue, 30 Dec 2014 15:07:37 +0000 From: "=?utf-8?B?TG/Dr2MgQmxvdA==?=" <loic.blot@unix-experience.fr> To: "Rick Macklem" <rmacklem@uoguelph.ca> Cc: freebsd-fs@freebsd.org Subject: Re: ZFS vnode lock deadlock in zfs_fhtovp was: High Kernel Load with nfsv4 Message-ID: <82f8ef92c6bf5f2323bae5ce5e4e2394@mail.unix-experience.fr> In-Reply-To: <1479765128.1118136.1419290403230.JavaMail.root@uoguelph.ca> References: <1479765128.1118136.1419290403230.JavaMail.root@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Rick,=0Ai upgraded my jail host from FreeBSD 9.3 to 10.1 to use NFS v4= .1 (mountoptions: rw,rsize=3D32768,wsize=3D32768,tcp,nfsv4,minorversion= =3D1)=0A=0APerformance is quite stable but it's slow. Not as slow as befo= re but slow... services was launched but no client are using them and sys= tem CPU % was 10-50%.=0A=0AI don't see anything on NFSv4.1 server, it's p= erfectly stable and functionnal.=0A=0ARegards,=0A=0ALo=C3=AFc Blot,=0AUNI= X Systems, Network and Security Engineer=0Ahttp://www.unix-experience.fr= =0A=0A23 d=C3=A9cembre 2014 00:20 "Rick Macklem" <rmacklem@uoguelph.ca> a= =C3=A9crit: =0A> Loic Blot wrote:=0A> =0A>> Hi,=0A>> =0A>> To clarify be= cause of our exchanges. Here are the current sysctl=0A>> options for serv= er:=0A>> =0A>> vfs.nfsd.enable_nobodycheck=3D0=0A>> vfs.nfsd.enable_nogro= upcheck=3D0=0A>> =0A>> vfs.nfsd.maxthreads=3D200=0A>> vfs.nfsd.tcphighwat= er=3D10000=0A>> vfs.nfsd.tcpcachetimeo=3D300=0A>> vfs.nfsd.server_min_nfs= vers=3D4=0A>> =0A>> kern.maxvnodes=3D10000000=0A>> kern.ipc.maxsockbuf=3D= 4194304=0A>> net.inet.tcp.sendbuf_max=3D4194304=0A>> net.inet.tcp.recvbuf= _max=3D4194304=0A>> =0A>> vfs.lookup_shared=3D0=0A>> =0A>> Regards,=0A>> = =0A>> Lo=C3=AFc Blot,=0A>> UNIX Systems, Network and Security Engineer=0A= >> http://www.unix-experience.fr=0A>>; =0A>> 22 d=C3=A9cembre 2014 09:42 "= Lo=C3=AFc Blot" <loic.blot@unix-experience.fr> a=0A>> =C3=A9crit:=0A>> = =0A>> Hi Rick,=0A>> my 5 jails runs this weekend and now i have some stat= s on this=0A>> monday.=0A>> =0A>> Hopefully deadlock was fixed, yeah, but= everything isn't good :(=0A>> =0A>> On NFSv4 server (FreeBSD 10.1) syste= m uses 35% CPU=0A>> =0A>> As i can see this is because of nfsd:=0A>> =0A>= > 918 root 96 20 0 12352K 3372K rpcsvc 6 51.4H=0A>> 273= .68% nfsd: server (nfsd)=0A>> =0A>> If i look at dmesg i see:=0A>> nfsd s= erver cache flooded, try increasing vfs.nfsd.tcphighwater=0A> =0A> Well, = you have a couple of choices:=0A> 1 - Use NFSv4.1 (add "minorversion=3D1"= to your mount options).=0A> (NFSv4.1 avoids use of the DRC and instead u= ses something=0A> called sessions. See below.)=0A> OR=0A> =0A>> vfs.nfsd.= tcphighwater was set to 10000, i increase it to 15000=0A> =0A> 2 - Bump v= fs.nfsd.tcphighwater way up, until you no longer see=0A> "nfs server cach= e flooded" messages. (I think Garrett Wollman uses=0A> 100000. (You may s= till see quite a bit of CPU overheads.)=0A> =0A> OR=0A> =0A> 3 - Set vfs.= nfsd.cachetcp=3D0 (which disables the DRC and gets rid=0A> of the CPU ove= rheads). However, there is a risk of data corruption=0A> if you have a cl= ient->server network partitioning of a moderate=0A> duration, because a n= on-idempotent RPC may get redone, becasue=0A> the client times out waitin= g for a reply. If a non-idempotent=0A> RPC gets done twice on the server,= data corruption can happen.=0A> (The DRC provides improved correctness, = but does add overhead.)=0A> =0A> If #1 works for you, it is the preferred= solution, since Sessions=0A> in NFSv4.1 solves the correctness problem i= n a good, space bound=0A> way. A session basically has N (usually 32 or 6= 4) slots and only=0A> allows one outstanding RPC/slot. As such, it can ca= che the previous=0A> reply for each slot (32 or 64 of them) and guarantee= "exactly once"=0A> RPC semantics.=0A> =0A> rick=0A> =0A>> Here is 'nfsst= at -s' output:=0A>> =0A>> Server Info:=0A>> Getattr Setattr Lookup = Readlink Read Write Create=0A>> Remove=0A>> 12600652 181= 2 2501097 156 1386423 1983729 123=0A>> 162067=0A>> Rena= me Link Symlink Mkdir Rmdir Readdir RdirPlus=0A>> Acces= s=0A>> 36762 9 0 0 0 3147 0= =0A>> 623524=0A>> Mknod Fsstat Fsinfo PathConf Commit=0A>> 0 = 0 0 0 328117=0A>> Server Ret-Failed=0A>> 0=0A>> = Server Faults=0A>> 0=0A>> Server Cache Stats:=0A>> Inprog Idem Non-= idem Misses=0A>> 0 0 0 12635512=0A>> Server Write Gat= hering:=0A>> WriteOps WriteRPC Opsaved=0A>> 1983729 1983729 = 0=0A>> =0A>> And here is 'procstat -kk' for nfsd (server)=0A>> =0A>> 918 = 100528 nfsd nfsd: master mi_switch+0xe1=0A>> sleepq_catch= _signals+0xab sleepq_timedwait_sig+0x10=0A>> _cv_timedwait_sig_sbt+0x18b = svc_run_internal+0x4a1 svc_run+0x1de=0A>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0= x107 sys_nfssvc+0x9c=0A>> amd64_syscall+0x351 Xfast_syscall+0xfb=0A>> 918= 100568 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catc= h_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>= > 918 100569 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq= _catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_i= nternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0x= e=0A>> 918 100570 nfsd nfsd: service mi_switch+0xe1=0A>> s= leepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_= run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoli= ne+0xe=0A>> 918 100571 nfsd nfsd: service mi_switch+0xe1= =0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A= >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_t= rampoline+0xe=0A>> 918 100572 nfsd nfsd: service mi_switch= +0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1= 6a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> f= ork_trampoline+0xe=0A>> 918 100573 nfsd nfsd: service mi_s= witch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_si= g+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>> fork_trampoline+0xe=0A>> 918 100574 nfsd nfsd: service = mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_= wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a=0A>> fork_trampoline+0xe=0A>> 918 100575 nfsd nfsd: ser= vice mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork= _exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100576 nfsd nfsd= : service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_si= g+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb= fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100577 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wa= it_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_star= t+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100578 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab slee= pq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100579 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_t= hread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100580 n= fsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100= 581 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0= x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 91= 8 100582 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_cat= ch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A= >> 918 100583 nfsd nfsd: service mi_switch+0xe1=0A>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_= internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0= xe=0A>> 918 100584 nfsd nfsd: service mi_switch+0xe1=0A>> = sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampol= ine+0xe=0A>> 918 100585 nfsd nfsd: service mi_switch+0xe1= =0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A= >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_t= rampoline+0xe=0A>> 918 100586 nfsd nfsd: service mi_switch= +0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1= 6a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> f= ork_trampoline+0xe=0A>> 918 100587 nfsd=20=20=20=20=20=20=20=20 nfsd:= service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig= +0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100588 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start= +0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100589 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleep= q_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_= start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100590 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab = sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_th= read_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100591 nf= sd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+= 0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 1005= 92 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_sig= nals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x= 87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918= 100593 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catc= h_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>= > 918 100594 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq= _catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_i= nternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0x= e=0A>> 918 100595 nfsd nfsd: service mi_switch+0xe1=0A>> s= leepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_= run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoli= ne+0xe=0A>> 918 100596 nfsd nfsd: service mi_switch+0xe1= =0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A= >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_t= rampoline+0xe=0A>> 918 100597 nfsd nfsd: service mi_switch= +0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1= 6a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> f= ork_trampoline+0xe=0A>> 918 100598 nfsd nfsd: service mi_s= witch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_si= g+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>> fork_trampoline+0xe=0A>> 918 100599 nfsd nfsd: service = mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_= wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a=0A>> fork_trampoline+0xe=0A>> 918 100600 nfsd nfsd: ser= vice mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork= _exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100601 nfsd nfsd= : service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_si= g+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb= fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100602 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wa= it_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_star= t+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100603 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab slee= pq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100604 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_t= hread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100605 n= fsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100= 606 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0= x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 91= 8 100607 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_cat= ch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A= >> 918 100608 nfsd nfsd: service mi_switch+0xe1=0A>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_= internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0= xe=0A>> 918 100609 nfsd nfsd: service mi_switch+0xe1=0A>> = sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampol= ine+0xe=0A>> 918 100610 nfsd nfsd: service mi_switch+0xe1= =0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A= >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_t= rampoline+0xe=0A>> 918 100611 nfsd nfsd: service mi_switch= +0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1= 6a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> f= ork_trampoline+0xe=0A>> 918 100612 nfsd nfsd: service mi_s= witch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_si= g+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>> fork_trampoline+0xe=0A>> 918 100613 nfsd nfsd: service = mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_= wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a=0A>> fork_trampoline+0xe=0A>> 918 100614 nfsd nfsd: ser= vice mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork= _exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100615 nfsd nfsd= : service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_si= g+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb= fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100616 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wa= it_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_star= t+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100617 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab slee= pq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100618 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_t= hread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100619 n= fsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100= 620 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0= x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 91= 8 100621 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_cat= ch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A= >> 918 100622 nfsd nfsd: service mi_switch+0xe1=0A>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_= internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0= xe=0A>> 918 100623 nfsd nfsd: service mi_switch+0xe1=0A>> = sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampol= ine+0xe=0A>> 918 100624 nfsd nfsd: service mi_switch+0xe1= =0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A= >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_t= rampoline+0xe=0A>> 918 100625 nfsd nfsd: service mi_switch= +0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1= 6a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> f= ork_trampoline+0xe=0A>> 918 100626 nfsd nfsd: service mi_s= witch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_si= g+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>> fork_trampoline+0xe=0A>> 918 100627 nfsd nfsd: service = mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_= wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a=0A>> fork_trampoline+0xe=0A>> 918 100628 nfsd nfsd: ser= vice mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork= _exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100629 nfsd nfsd= : service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_si= g+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb= fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100630 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wa= it_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_star= t+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100631 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab slee= pq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100632 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_t= hread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100633 n= fsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100= 634 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0= x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 91= 8 100635 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_cat= ch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A= >> 918 100636 nfsd nfsd: service mi_switch+0xe1=0A>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_= internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0= xe=0A>> 918 100637 nfsd nfsd: service mi_switch+0xe1=0A>> = sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampol= ine+0xe=0A>> 918 100638 nfsd nfsd: service mi_switch+0xe1= =0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A= >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_t= rampoline+0xe=0A>> 918 100639 nfsd nfsd: service mi_switch= +0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1= 6a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> f= ork_trampoline+0xe=0A>> 918 100640 nfsd nfsd: service mi_s= witch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_si= g+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>> fork_trampoline+0xe=0A>> 918 100641 nfsd nfsd: service = mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_= wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a=0A>> fork_trampoline+0xe=0A>> 918 100642 nfsd nfsd: ser= vice mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork= _exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100643 nfsd nfsd= : service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_si= g+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb= fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100644 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wa= it_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_star= t+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100645 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab slee= pq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100646 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_t= hread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100647 n= fsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100= 648 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0= x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 91= 8 100649 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_cat= ch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A= >> 918 100650 nfsd nfsd: service mi_switch+0xe1=0A>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_= internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0= xe=0A>> 918 100651 nfsd nfsd: service mi_switch+0xe1=0A>> = sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampol= ine+0xe=0A>> 918 100652 nfsd nfsd: service mi_switch+0xe1= =0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A= >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_t= rampoline+0xe=0A>> 918 100653 nfsd nfsd: service mi_switch= +0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1= 6a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> f= ork_trampoline+0xe=0A>> 918 100654 nfsd nfsd: service mi_s= witch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_si= g+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>> fork_trampoline+0xe=0A>> 918 100655 nfsd nfsd: service = mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_= wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a=0A>> fork_trampoline+0xe=0A>> 918 100656 nfsd nfsd: ser= vice mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork= _exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100657 nfsd nfsd= : service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_si= g+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb= fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100658 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wa= it_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_star= t+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100659 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab slee= pq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100660 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_t= hread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100661 n= fsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100= 662 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0= x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> --= -=0A>> =0A>> Now if we look at client (FreeBSD 9.3)=0A>> =0A>> We see sys= tem was very busy and do many and many interrupts=0A>> =0A>> CPU: 0.0% u= ser, 0.0% nice, 37.8% system, 51.2% interrupt, 11.0%=0A>> idle=0A>> =0A>= > A look at process list shows that there are many sendmail process in=0A= >> state nfstry=0A>> =0A>> nfstry 18 32:27 0.88% sendmail: Queue runne= r@00:30:00 for=0A>> /var/spool/clientm=0A>> =0A>> Here is 'nfsstat -c' ou= tput:=0A>> =0A>> Client Info:=0A>> Rpc Counts:=0A>> Getattr Setattr = Lookup Readlink Read Write Create=0A>> Remove=0A>> 1051347 = 1724 2494481 118 903902 1901285 162676=0A>> 161899=0A= >> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus=0A>= > Access=0A>> 36744 2 0 114 40 3131 = 0=0A>> 544136=0A>> Mknod Fsstat Fsinfo PathConf Commit=0A>= > 9 0 0 0 245821=0A>> Rpc Info:=0A>> TimedOut = Invalid X Replies Retries Requests=0A>> 0 0 0 = 0 8356557=0A>> Cache Info:=0A>> Attr Hits Misses Lkup Hits Miss= es BioR Hits Misses BioW Hits=0A>> Misses=0A>> 108754455 491475 54= 229224 2437229 46814561 821723 5132123=0A>> 1871871=0A>> BioRLHit= s Misses BioD Hits Misses DirE Hits Misses Accs Hits=0A>> Misses= =0A>> 144035 118 53736 2753 27813 1 57238839= =0A>> 544205=0A>> =0A>> If you need more things, tell me, i let the PoC i= n this state.=0A>> =0A>> Thanks=0A>> =0A>> Regards,=0A>> =0A>> Lo=C3=AFc = Blot,=0A>> UNIX Systems, Network and Security Engineer=0A>> http://www.un= ix-experience.fr=0A>> =0A>> 21 d=C3=A9cembre 2014 01:33 "Rick Macklem" <r= macklem@uoguelph.ca> a =C3=A9crit:=0A>> =0A>> Loic Blot wrote:=0A>> =0A>>= > Hi Rick,=0A>>> ok, i don't need locallocks, i haven't understand option= was for=0A>>> that=0A>>> usage, i removed it.=0A>>> I do more tests on m= onday.=0A>>> Thanks for the deadlock fix, for other people :)=0A>> =0A>> = Good. Please let us know if running with vfs.nfsd.enable_locallocks=3D0= =0A>> gets rid of the deadlocks? (I think it fixes the one you saw.)=0A>>= =0A>> On the performance side, you might also want to try different valu= es=0A>> of=0A>> readahead, if the Linux client has such a mount option. (= With the=0A>> NFSv4-ZFS sequential vs random I/O heuristic, I have no ide= a what the=0A>> optimal readahead value would be.)=0A>> =0A>> Good luck w= ith it and please let us know how it goes, rick=0A>> ps: I now have a pat= ch to fix the deadlock when=0A>> vfs.nfsd.enable_locallocks=3D1=0A>> is s= et. I'll post it for anyone who is interested after I put it=0A>> through= some testing.=0A>> =0A>> --=0A>> Best regards,=0A>> Lo=C3=AFc BLOT,=0A>>= UNIX systems, security and network engineer=0A>> http://www.unix-experie= nce.fr=0A>> =0A>> Le jeudi 18 d=C3=A9cembre 2014 =C3=A0 19:46 -0500, Rick= Macklem a =C3=A9crit :=0A>> =0A>> Loic Blot wrote:=0A>>> Hi rick,=0A>>> = i tried to start a LXC container on Debian Squeeze from my=0A>>> freebsd= =0A>>> ZFS+NFSv4 server and i also have a deadlock on nfsd=0A>>> (vfs.loo= kup_shared=3D0). Deadlock procs each time i launch a=0A>>> squeeze=0A>>> = container, it seems (3 tries, 3 fails).=0A>> =0A>> Well, I`ll take a look= at this `procstat -kk`, but the only thing=0A>> I`ve seen posted w.r.t. = avoiding deadlocks in ZFS is to not use=0A>> nullfs. (I have no idea if y= ou are using any nullfs mounts, but=0A>> if so, try getting rid of them.)= =0A>> =0A>> Here`s a high level post about the ZFS and vnode locking prob= lem,=0A>> but there is no patch available, as far as I know.=0A>> =0A>> h= ttp://docs.FreeBSD.org/cgi/mid.cgi?54739F41.8030407=0A>> =0A>> rick=0A>> = =0A>> 921 - D 0:00.02 nfsd: server (nfsd)=0A>> =0A>> Here is the p= rocstat -kk=0A>> =0A>> PID TID COMM TDNAME KSTAC= K=0A>> 921 100538 nfsd nfsd: master mi_switch+0xe1=0A>> s= leepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e=0A>> vop_stdlock+0x3c = VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>> nfsvno_advlock+0x119 nfsrv_dolocal+= 0x84 nfsrv_lockctrl+0x14ad=0A>> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfs= svc_program+0x554=0A>> svc_run_internal+0xc77 svc_run+0x1de nfsrvd_nfsd+0= x1ca=0A>> nfssvc_nfsd+0x107 sys_nfssvc+0x9c=0A>> 921 100572 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq= _wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100573 nfs= d nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 = 100574 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch= _signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_in= ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe= =0A>> 921 100575 nfsd nfsd: service mi_switch+0xe1=0A>> sl= eepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> = svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_tram= poline+0xe=0A>> 921 100576 nfsd nfsd: service mi_switch+0x= e1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0= x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>= fork_trampoline+0xe=0A>> 921 100577 nfsd nfsd: service mi= _switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_= wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a=0A>> fork_trampoline+0xe=0A>> 921 100578 nfsd nfsd: ser= vice mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb= fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100579 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wa= it_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100580 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100= 581 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A= >> 921 100582 nfsd nfsd: service mi_switch+0xe1=0A>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampol= ine+0xe=0A>> 921 100583 nfsd nfsd: service mi_switch+0xe1= =0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x1= 6a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> f= ork_trampoline+0xe=0A>> 921 100584 nfsd nfsd: service mi_s= witch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wa= it_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0= x9a=0A>> fork_trampoline+0xe=0A>> 921 100585 nfsd nfsd: servi= ce mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb= fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100586 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wa= it_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100587 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100= 588 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A= >> 921 100589 nfsd nfsd: service mi_switch+0xe1=0A>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampol= ine+0xe=0A>> 921 100590 nfsd nfsd: service mi_switch+0xe1= =0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x1= 6a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> f= ork_trampoline+0xe=0A>> 921 100591 nfsd nfsd: service mi_s= witch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wa= it_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0= x9a=0A>> fork_trampoline+0xe=0A>> 921 100592 nfsd nfsd: servi= ce mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb= fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100593 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wa= it_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100594 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100= 595 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A= >> 921 100596 nfsd nfsd: service mi_switch+0xe1=0A>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampol= ine+0xe=0A>> 921 100597 nfsd nfsd: service mi_switch+0xe1= =0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x1= 6a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> f= ork_trampoline+0xe=0A>> 921 100598 nfsd nfsd: service mi_s= witch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wa= it_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0= x9a=0A>> fork_trampoline+0xe=0A>> 921 100599 nfsd nfsd: servi= ce mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb= fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100600 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wa= it_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100601 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100= 602 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A= >> 921 100603 nfsd nfsd: service mi_switch+0xe1=0A>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampol= ine+0xe=0A>> 921 100604 nfsd nfsd: service mi_switch+0xe1= =0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x1= 6a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> f= ork_trampoline+0xe=0A>> 921 100605 nfsd nfsd: service mi_s= witch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wa= it_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0= x9a=0A>> fork_trampoline+0xe=0A>> 921 100606 nfsd nfsd: servi= ce mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb= fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100607 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wa= it_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100608 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100= 609 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A= >> 921 100610 nfsd nfsd: service mi_switch+0xe1=0A>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampol= ine+0xe=0A>> 921 100611 nfsd nfsd: service mi_switch+0xe1= =0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x1= 6a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> f= ork_trampoline+0xe=0A>> 921 100612 nfsd nfsd: service mi_s= witch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wa= it_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0= x9a=0A>> fork_trampoline+0xe=0A>> 921 100613 nfsd nfsd: servi= ce mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb= fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100614 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wa= it_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100615 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100= 616 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3= a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>> nfsrv_getlockfile+0x17= 9 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1=0A>> nfsrvd_dorpc+0xec6 nfssvc_p= rogram+0x554 svc_run_internal+0xc77=0A>> svc_thread_start+0xb fork_exit+0= x9a fork_trampoline+0xe=0A>> 921 100617 nfsd nfsd: service = mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _= cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100618 nfsd nfsd: = service mi_switch+0xe1=0A>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x= 66 nfsv4_lock+0x9b=0A>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_i= nternal+0xc77=0A>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0x= e=0A>> 921 100619 nfsd nfsd: service mi_switch+0xe1=0A>> s= leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>>= svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_tra= mpoline+0xe=0A>> 921 100620 nfsd nfsd: service mi_switch+0= xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+= 0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>= > fork_trampoline+0xe=0A>> 921 100621 nfsd nfsd: service m= i_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv= _wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exi= t+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100622 nfsd nfsd: se= rvice mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0x= f=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0x= b fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100623 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_w= ait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100624 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xa= b sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e= svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 10= 0625 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_s= ignals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_inte= rnal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe= =0A>> 921 100626 nfsd nfsd: service mi_switch+0xe1=0A>> sl= eepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> = svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_tram= poline+0xe=0A>> 921 100627 nfsd nfsd: service mi_switch+0x= e1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0= x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>= fork_trampoline+0xe=0A>> 921 100628 nfsd nfsd: service=20=20= mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> = _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_= exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100629 nfsd nfsd:= service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig= +0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start= +0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100630 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleep= q_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_th= read_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100631 nf= sd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+= 0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x= 87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921= 100632 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catc= h_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_i= nternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0x= e=0A>> 921 100633 nfsd nfsd: service mi_switch+0xe1=0A>> s= leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>>= svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_tra= mpoline+0xe=0A>> 921 100634 nfsd nfsd: service mi_switch+0= xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+= 0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>= > fork_trampoline+0xe=0A>> 921 100635 nfsd nfsd: service m= i_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv= _wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exi= t+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100636 nfsd nfsd: se= rvice mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0x= f=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0x= b fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100637 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_w= ait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100638 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xa= b sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e= svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 10= 0639 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_s= ignals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_inte= rnal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe= =0A>> 921 100640 nfsd nfsd: service mi_switch+0xe1=0A>> sl= eepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> = svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_tram= poline+0xe=0A>> 921 100641 nfsd nfsd: service mi_switch+0x= e1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0= x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>= fork_trampoline+0xe=0A>> 921 100642 nfsd nfsd: service mi= _switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_= wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a=0A>> fork_trampoline+0xe=0A>> 921 100643 nfsd nfsd: ser= vice mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb= fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100644 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wa= it_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100645 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100= 646 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A= >> 921 100647 nfsd nfsd: service mi_switch+0xe1=0A>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampol= ine+0xe=0A>> 921 100648 nfsd nfsd: service mi_switch+0xe1= =0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x1= 6a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> f= ork_trampoline+0xe=0A>> 921 100649 nfsd nfsd: service mi_s= witch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wa= it_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0= x9a=0A>> fork_trampoline+0xe=0A>> 921 100650 nfsd nfsd: servi= ce mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb= fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100651 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wa= it_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100652 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100= 653 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A= >> 921 100654 nfsd nfsd: service mi_switch+0xe1=0A>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampol= ine+0xe=0A>> 921 100655 nfsd nfsd: service mi_switch+0xe1= =0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x1= 6a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> f= ork_trampoline+0xe=0A>> 921 100656 nfsd nfsd: service mi_s= witch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wa= it_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0= x9a=0A>> fork_trampoline+0xe=0A>> 921 100657 nfsd nfsd: servi= ce mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb= fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100658 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wa= it_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100659 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100= 660 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A= >> 921 100661 nfsd nfsd: service mi_switch+0xe1=0A>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_trampol= ine+0xe=0A>> 921 100662 nfsd nfsd: service mi_switch+0xe1= =0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wait_sig+0x1= 6a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> f= ork_trampoline+0xe=0A>> 921 100663 nfsd nfsd: service mi_s= witch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>> _cv_wa= it_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0= x9a=0A>> fork_trampoline+0xe=0A>> 921 100664 nfsd nfsd: servi= ce mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb= fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100665 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wa= it_sig+0xf=0A>> _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 921 100666 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a _sleep+0= x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>> nfsrv_setclient+0xbd nfsrvd_setc= lientid+0x3c8=0A>> nfsrvd_dorpc+0xc76=0A>> nfssvc_program+0x554 svc_run_i= nternal+0xc77=0A>> svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoli= ne+0xe=0A>> =0A>> Regards,=0A>> =0A>> Lo=C3=AFc Blot,=0A>> UNIX Systems, = Network and Security Engineer=0A>> http://www.unix-experience.fr=0A>>; =0A= >> 15 d=C3=A9cembre 2014 15:18 "Rick Macklem" <rmacklem@uoguelph.ca> a=0A= >> =C3=A9crit:=0A>> =0A>> Loic Blot wrote:=0A>> =0A>>> For more informati= ons, here is procstat -kk on nfsd, if you=0A>>> need=0A>>> more=0A>>> hot= datas, tell me.=0A>>> =0A>>> Regards, PID TID COMM TDNAM= E KSTACK=0A>>> 918 100529 nfsd nfsd: master mi_= switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>= >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x3= 8d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfs= svc_program+0x554 svc_run_internal+0xc77 svc_run+0x1de=0A>>> nfsrvd_nfsd+= 0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c=0A>>> amd64_syscall+0x351=0A>> = =0A>> Well, most of the threads are stuck like this one, waiting for=0A>>= a=0A>> vnode=0A>> lock in ZFS. All of them appear to be in zfs_fhtovp().= =0A>> I`m not a ZFS guy, so I can`t help much. I`ll try changing the=0A>>= subject line=0A>> to include ZFS vnode lock, so maybe the ZFS guys will = take a=0A>> look.=0A>> =0A>> The only thing I`ve seen suggested is trying= :=0A>> sysctl vfs.lookup_shared=3D0=0A>> to disable shared vop_lookup()s.= Apparently zfs_lookup()=0A>> doesn`t=0A>> obey the vnode locking rules f= or lookup and rename, according=0A>> to=0A>> the posting I saw.=0A>> =0A>= > I`ve added a couple of comments about the other threads below,=0A>> but= =0A>> they are all either waiting for an RPC request or waiting for=0A>> = the=0A>> threads stuck on the ZFS vnode lock to complete.=0A>> =0A>> rick= =0A>> =0A>>> 918 100564 nfsd nfsd: service mi_switch+0xe1= =0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0= x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>= >> fork_trampoline+0xe=0A>> =0A>> Fyi, this thread is just waiting for an= RPC to arrive. (Normal)=0A>> =0A>>> 918 100565 nfsd nfsd: se= rvice mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0= xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start= +0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100566 nfsd = nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sl= eepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 10= 0567 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_catch_= signals+0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_i= nternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> fork_trampoline+0= xe=0A>>> 918 100568 nfsd nfsd: service mi_switch+0xe1=0A>>= > sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>> _cv_wait_sig+0x16a= =0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>> f= ork_trampoline+0xe=0A>>> 918 100569 nfsd nfsd: service mi_= switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>> _cv= _wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_start+0xb fork_ex= it+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100570 nfsd nfsd:= service mi_switch+0xe1=0A>>> sleepq_catch_signals+0xab sleepq_wait_si= g+0xf=0A>>> _cv_wait_sig+0x16a=0A>>> svc_run_internal+0x87e svc_thread_st= art+0xb fork_exit+0x9a=0A>>> fork_trampoline+0xe=0A>>> 918 100571 nfsd = nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a _sleep+0= x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>> nfsrvd_dorpc+0x316 nfssvc_progr= am+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb fork_exit+0x9a= fork_trampoline+0xe=0A>>> 918 100572 nfsd nfsd: service m= i_switch+0xe1=0A>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lo= ck+0x9b=0A>>> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8=0A>>> nfsrvd_= dorpc+0xc76=0A>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc_t= hread_start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>> =0A>> This = one (and a few others) are waiting for the nfsv4_lock.=0A>> This=0A>> hap= pens=0A>> because other threads are stuck with RPCs in progress. (ie. The= =0A>> ones=0A>> waiting on the vnode lock in zfs_fhtovp().)=0A>> For thes= e, the RPC needs to lock out other threads to do the=0A>> operation,=0A>>= so it waits for the nfsv4_lock() which can exclusively lock the=0A>> NFS= v4=0A>> data structures once all other nfsd threads complete their RPCs= =0A>> in=0A>> progress.=0A>> =0A>>> 918 100573 nfsd nfsd: ser= vice mi_switch+0xe1=0A>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66= nfsv4_lock+0x9b=0A>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_in= ternal+0xc77=0A>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0x= e=0A>> =0A>> Same as above.=0A>> =0A>>> 918 100574 nfsd nfsd:= service mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr= _args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>= zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+= 0x917=0A>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc_thread_= start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 100575 nfsd = nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleepl= k+0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _v= n_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0= xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_run_internal+0xc77= =0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>= > 918 100576 nfsd nfsd: service mi_switch+0xe1=0A>>> sleep= q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP= _LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+= 0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_r= un_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_tr= ampoline+0xe=0A>>> 918 100577 nfsd nfsd: service mi_switch= +0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop= _stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>= >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_pr= ogram+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_= exit+0x9a fork_trampoline+0xe=0A>>> 918 100578 nfsd nfsd: ser= vice mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_arg= s+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs= _fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x91= 7=0A>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc_thread_star= t+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 100579 nfsd = nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x= 15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lo= ck+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 = nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>= > svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>> 918= 100580 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_wai= t+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK= 1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c = nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_run_in= ternal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_trampol= ine+0xe=0A>>> 918 100581 nfsd nfsd: service mi_switch+0xe1= =0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdl= ock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nf= svno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program= +0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+= 0x9a fork_trampoline+0xe=0A>>> 918 100582 nfsd nfsd: service = mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x9= 02=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhto= vp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>= >> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb= =0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 100583 nfsd = nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d _= _lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x= 43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrv= d_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc= _thread_start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 1005= 84 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3= a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV= +0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_= fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_run_interna= l+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0= xe=0A>>> 918 100585 nfsd nfsd: service mi_switch+0xe1=0A>>= > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0= x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_= fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x55= 4 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a = fork_trampoline+0xe=0A>>> 918 100586 nfsd nfsd: service mi= _switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A= >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x= 38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nf= ssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>= > fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 100587 nfsd nf= sd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lock= mgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A= >>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dor= pc+0x917=0A>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc_thre= ad_start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 100588 nf= sd nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a sle= eplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab= _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtov= p+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_run_internal+0xc= 77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A= >>> 918 100589 nfsd nfsd: service mi_switch+0xe1=0A>>> sle= epq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c V= OP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtov= p+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc= _run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_= trampoline+0xe=0A>>> 918 100590 nfsd nfsd: service mi_swit= ch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> v= op_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d= =0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssv= c_program+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> f= ork_exit+0x9a fork_trampoline+0xe=0A>>> 918 100591 nfsd nfsd:= service mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr= _args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>= zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+= 0x917=0A>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc_thread_= start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 100592 nfsd = nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleepl= k+0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _v= n_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0= xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_run_internal+0xc77= =0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>= > 918 100593 nfsd nfsd: service mi_switch+0xe1=0A>>> sleep= q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP= _LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+= 0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_r= un_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_tr= ampoline+0xe=0A>>> 918 100594 nfsd nfsd: service mi_switch= +0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop= _stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>= >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_pr= ogram+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_= exit+0x9a fork_trampoline+0xe=0A>>> 918 100595 nfsd nfsd: ser= vice mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_arg= s+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs= _fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x91= 7=0A>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc_thread_star= t+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 100596 nfsd = nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x= 15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lo= ck+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 = nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>= > svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>> 918= 100597 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_wai= t+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK= 1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c = nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_run_in= ternal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_trampol= ine+0xe=0A>>> 918 100598 nfsd nfsd: service mi_switch+0xe1= =0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdl= ock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nf= svno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program= +0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+= 0x9a fork_trampoline+0xe=0A>>> 918 100599 nfsd nfsd: service = mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x9= 02=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhto= vp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>= >> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb= =0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 100600 nfsd = nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d _= _lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x= 43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrv= d_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc= _thread_start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 1006= 01 nfsd nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3= a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV= +0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_= fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_run_interna= l+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0= xe=0A>>> 918 100602 nfsd nfsd: service mi_switch+0xe1=0A>>= > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0= x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_= fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x55= 4 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a = fork_trampoline+0xe=0A>>> 918 100603 nfsd nfsd: service mi= _switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A= >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x= 38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nf= ssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>= > fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 100604 nfsd nf= sd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lock= mgr_args+0x902=0A>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A= >>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dor= pc+0x917=0A>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>> svc_thre= ad_start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe=0A>>> 918 100605 nf= sd nfsd: service mi_switch+0xe1=0A>>> sleepq_wait+0x3a sle= eplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3c=20VOP_LOCK1_APV+0x= ab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fhtovp+0x7c nfsd_fht= ovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 svc_run_internal+0= xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a fork_trampoline+0xe= =0A>>> 918 100606 nfsd nfsd: service mi_switch+0xe1=0A>>> = sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>> vop_stdlock+0x3= c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38d=0A>>> nfsvno_fh= tovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfssvc_program+0x554 = svc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> fork_exit+0x9a fo= rk_trampoline+0xe=0A>>> 918 100607 nfsd nfsd: service mi_s= witch+0xe1=0A>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>= > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>> zfs_fhtovp+0x38= d=0A>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>> nfss= vc_program+0x554 svc_run_internal+0xc77=0A>>> svc_thread_start+0xb=0A>>> = fork_exit+0x9a fork_trampoline+0xe=0A>> =0A>> Lots more waiting for the Z= FS vnode lock in zfs_fhtovp().=0A>> =0A>> 918 100608 nfsd nfs= d: service mi_switch+0xe1=0A>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep= +0x66 nfsv4_lock+0x9b=0A>> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f n= fsrvd_lock+0x5b1=0A>> nfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_int= ernal+0xc77=0A>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe= =0A>> 918 100609 nfsd nfsd: service mi_switch+0xe1=0A>> sl= eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V= OP_LOCK1_APV+0xab _vn_lock+0x43=0A>> zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+= 0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_ru= n_internal+0xc77=0A>> svc_thread_start+0xb=0A>> fork_exit+0x9a fork_tramp= oline+0xe=0A>> 918 100610 nfsd nfsd: service mi_switch+0xe= 1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e=0A>> vop_stdlo= ck+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>> nfsvno_advlock+0x119 nfsrv_= dolocal+0x84 nfsrv_lockctrl+0x14ad=0A>> nfsrvd_locku+0x283 nfsrvd_dorpc+0= xec6 nfssvc_program+0x554=0A>> svc_run_internal+0xc77 svc_thread_start+0x= b fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100611 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a _sleep+0x287 nfsm= sleep+0x66 nfsv4_lock+0x9b=0A>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 s= vc_run_internal+0xc77=0A>> svc_thread_start+0xb fork_exit+0x9a fork_tramp= oline+0xe=0A>> 918 100612 nfsd nfsd: service mi_switch+0xe= 1=0A>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>> = nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77=0A>> svc_t= hread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100613 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a _sleep+0= x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>> nfsrvd_dorpc+0x316 nfssvc_progra= m+0x554 svc_run_internal+0xc77=0A>> svc_thread_start+0xb fork_exit+0x9a f= ork_trampoline+0xe=0A>> 918 100614 nfsd nfsd: service mi_s= witch+0xe1=0A>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0= x9b=0A>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77= =0A>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>> 918 10= 0615 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x= 3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>> nfsrvd_dorpc+0x316 nf= ssvc_program+0x554 svc_run_internal+0xc77=0A>> svc_thread_start+0xb fork_= exit+0x9a fork_trampoline+0xe=0A>> 918 100616 nfsd nfsd: serv= ice mi_switch+0xe1=0A>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 n= fsv4_lock+0x9b=0A>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_inter= nal+0xc77=0A>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A= >> 918 100617 nfsd nfsd: service mi_switch+0xe1=0A>> sleep= q_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>> nfsrvd_dorpc= +0x316 nfssvc_program+0x554 svc_run_internal+0xc77=0A>> svc_thread_start+= 0xb fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100618 nfsd n= fsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a _sleep+0x287 nfsmsle= ep+0x66 nfsv4_lock+0x9b=0A>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_= run_internal+0xc77=0A>> svc_thread_start+0xb fork_exit+0x9a fork_trampoli= ne+0xe=0A>> 918 100619 nfsd nfsd: service mi_switch+0xe1= =0A>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdloc= k+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>> zfs_fhtovp+0x38d=0A>> nfsvno= _fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x55= 4 svc_run_internal+0xc77=0A>> svc_thread_start+0xb=0A>> fork_exit+0x9a fo= rk_trampoline+0xe=0A>> 918 100620 nfsd nfsd: service mi_sw= itch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> v= op_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>> zfs_fhtovp+0x38d=0A= >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_pro= gram+0x554 svc_run_internal+0xc77=0A>> svc_thread_start+0xb=0A>> fork_exi= t+0x9a fork_trampoline+0xe=0A>> 918 100621 nfsd nfsd: service= mi_switch+0xe1=0A>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv= 4_lock+0x9b=0A>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal= +0xc77=0A>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>> = 918 100622 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_w= ait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOC= K1_APV+0xab _vn_lock+0x43=0A>> zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c n= fsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_inte= rnal+0xc77=0A>> svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+= 0xe=0A>> 918 100623 nfsd nfsd: service mi_switch+0xe1=0A>>= sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>> nfsrvd= _dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77=0A>> svc_thread_= start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100624 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d = __lockmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x= 43=0A>> zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_= dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>> svc_thr= ead_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100625 nfs= d nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleep= lk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _v= n_lock+0x43=0A>> zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc= 8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>= > svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 1= 00626 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0= x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_AP= V+0xab _vn_lock+0x43=0A>> zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_f= htovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_internal+= 0xc77=0A>> svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe= =0A>> 918 100627 nfsd nfsd: service mi_switch+0xe1=0A>> sl= eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V= OP_LOCK1_APV+0xab _vn_lock+0x43=0A>> zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+= 0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_ru= n_internal+0xc77=0A>> svc_thread_start+0xb=0A>> fork_exit+0x9a fork_tramp= oline+0xe=0A>> 918 100628 nfsd nfsd: service mi_switch+0xe= 1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlo= ck+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>> zfs_fhtovp+0x38d=0A>> nfsvn= o_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x5= 54 svc_run_internal+0xc77=0A>> svc_thread_start+0xb=0A>> fork_exit+0x9a f= ork_trampoline+0xe=0A>> 918 100629 nfsd nfsd: service mi_s= witch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> = vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>> zfs_fhtovp+0x38d= =0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_= program+0x554 svc_run_internal+0xc77=0A>> svc_thread_start+0xb=0A>> fork_= exit+0x9a fork_trampoline+0xe=0A>> 918 100630 nfsd nfsd: serv= ice mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+= 0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>> zfs_fht= ovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>= > nfssvc_program+0x554 svc_run_internal+0xc77=0A>> svc_thread_start+0xb= =0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918=20100631 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __= lockmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43= =0A>> zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_do= rpc+0x917=0A>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>> svc_threa= d_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100632 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk= +0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_= lock+0x43=0A>> zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 = nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>> = svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100= 633 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3= a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+= 0xab _vn_lock+0x43=0A>> zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fht= ovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_internal+0x= c77=0A>> svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>= > 918 100634 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq= _wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_L= OCK1_APV+0xab _vn_lock+0x43=0A>> zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c= nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_in= ternal+0xc77=0A>> svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampolin= e+0xe=0A>> 918 100635 nfsd nfsd: service mi_switch+0xe1=0A= >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0= x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>> zfs_fhtovp+0x38d=0A>> nfsvno_fh= tovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 s= vc_run_internal+0xc77=0A>> svc_thread_start+0xb=0A>> fork_exit+0x9a fork_= trampoline+0xe=0A>> 918 100636 nfsd nfsd: service mi_switc= h+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_= stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>> zfs_fhtovp+0x38d=0A>> = nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_progra= m+0x554 svc_run_internal+0xc77=0A>> svc_thread_start+0xb=0A>> fork_exit+0= x9a fork_trampoline+0xe=0A>> 918 100637 nfsd nfsd: service = mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902= =0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>> zfs_fhtovp+0= x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfs= svc_program+0x554 svc_run_internal+0xc77=0A>> svc_thread_start+0xb=0A>> f= ork_exit+0x9a fork_trampoline+0xe=0A>> 918 100638 nfsd nfsd: = service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_a= rgs+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>> zfs= _fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917= =0A>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>> svc_thread_start+0= xb=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100639 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __= lockmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43= =0A>> zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_do= rpc+0x917=0A>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>> svc_threa= d_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100640 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk= +0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_= lock+0x43=0A>> zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 = nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>> = svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100= 641 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3= a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+= 0xab _vn_lock+0x43=0A>> zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fht= ovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_internal+0x= c77=0A>> svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>= > 918 100642 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq= _wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_L= OCK1_APV+0xab _vn_lock+0x43=0A>> zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c= nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_in= ternal+0xc77=0A>> svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampolin= e+0xe=0A>> 918 100643 nfsd nfsd: service mi_switch+0xe1=0A= >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0= x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>> zfs_fhtovp+0x38d=0A>> nfsvno_fh= tovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 s= vc_run_internal+0xc77=0A>> svc_thread_start+0xb=0A>> fork_exit+0x9a fork_= trampoline+0xe=0A>> 918 100644 nfsd nfsd: service mi_switc= h+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_= stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>> zfs_fhtovp+0x38d=0A>> = nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_progra= m+0x554 svc_run_internal+0xc77=0A>> svc_thread_start+0xb=0A>> fork_exit+0= x9a fork_trampoline+0xe=0A>> 918 100645 nfsd nfsd: service = mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902= =0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>> zfs_fhtovp+0= x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfs= svc_program+0x554 svc_run_internal+0xc77=0A>> svc_thread_start+0xb=0A>> f= ork_exit+0x9a fork_trampoline+0xe=0A>> 918 100646 nfsd nfsd: = service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_a= rgs+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>> zfs= _fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917= =0A>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>> svc_thread_start+0= xb=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100647 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __= lockmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43= =0A>> zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_do= rpc+0x917=0A>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>> svc_threa= d_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100648 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk= +0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_= lock+0x43=0A>> zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 = nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>> = svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100= 649 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3= a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+= 0xab _vn_lock+0x43=0A>> zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fht= ovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_internal+0x= c77=0A>> svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>= > 918 100650 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq= _wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_L= OCK1_APV+0xab _vn_lock+0x43=0A>> zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c= nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_in= ternal+0xc77=0A>> svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampolin= e+0xe=0A>> 918 100651 nfsd nfsd: service mi_switch+0xe1=0A= >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0= x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>> zfs_fhtovp+0x38d=0A>> nfsvno_fh= tovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 s= vc_run_internal+0xc77=0A>> svc_thread_start+0xb=0A>> fork_exit+0x9a fork_= trampoline+0xe=0A>> 918 100652 nfsd nfsd: service mi_switc= h+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_= stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>> zfs_fhtovp+0x38d=0A>> = nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_progra= m+0x554 svc_run_internal+0xc77=0A>> svc_thread_start+0xb=0A>> fork_exit+0= x9a fork_trampoline+0xe=0A>> 918 100653 nfsd nfsd: service = mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902= =0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>> zfs_fhtovp+0= x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfs= svc_program+0x554 svc_run_internal+0xc77=0A>> svc_thread_start+0xb=0A>>= =20fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100654 nfsd nf= sd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lockm= gr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>= zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0= x917=0A>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>> svc_thread_sta= rt+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100655 nfsd = nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15= d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+= 0x43=0A>> zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrv= d_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>> svc_t= hread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100656 n= fsd nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sle= eplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab = _vn_lock+0x43=0A>> zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0= xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_internal+0xc77= =0A>> svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 9= 18 100657 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_wa= it+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK= 1_APV+0xab _vn_lock+0x43=0A>> zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nf= sd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_inter= nal+0xc77=0A>> svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0= xe=0A>> 918 100658 nfsd nfsd: service mi_switch+0xe1=0A>> = sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c= VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>> zfs_fhtovp+0x38d=0A>> nfsvno_fhtov= p+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_= run_internal+0xc77=0A>> svc_thread_start+0xb=0A>> fork_exit+0x9a fork_tra= mpoline+0xe=0A>> =0A>> Lo=C3=AFc Blot,=0A>> UNIX Systems, Network and Sec= urity Engineer=0A>> http://www.unix-experience.fr=0A>>; =0A>> 15 d=C3=A9ce= mbre 2014 13:29 "Lo=C3=AFc Blot"=0A>> <loic.blot@unix-experience.fr>=0A>>= a=0A>> =C3=A9crit:=0A>> =0A>> Hmmm...=0A>> now i'm experiencing a deadlo= ck.=0A>> =0A>> 0 918 915 0 21 0 12352 3372 zfs D - 1:48.64 nfsd: server= =0A>> (nfsd)=0A>> =0A>> the only issue was to reboot the server, but afte= r rebooting=0A>> deadlock arrives a second time when i=0A>> start my jail= s over NFS.=0A>> =0A>> Regards,=0A>> =0A>> Lo=C3=AFc Blot,=0A>> UNIX Syst= ems, Network and Security Engineer=0A>> http://www.unix-experience.fr=0A>= > =0A>> 15 d=C3=A9cembre 2014 10:07 "Lo=C3=AFc Blot"=0A>> <loic.blot@unix= -experience.fr>=0A>> a=0A>> =C3=A9crit:=0A>> =0A>> Hi Rick,=0A>> after ta= lking with my N+1, NFSv4 is required on our=0A>> infrastructure.=0A>> I t= ried to upgrade NFSv4+ZFS=0A>> server from 9.3 to 10.1, i hope this will = resolve some=0A>> issues...=0A>> =0A>> Regards,=0A>> =0A>> Lo=C3=AFc Blot= ,=0A>> UNIX Systems, Network and Security Engineer=0A>> http://www.unix-e= xperience.fr=0A>> =0A>> 10 d=C3=A9cembre 2014 15:36 "Lo=C3=AFc Blot"=0A>>= <loic.blot@unix-experience.fr>=0A>> a=0A>> =C3=A9crit:=0A>> =0A>> Hi Ric= k,=0A>> thanks for your suggestion.=0A>> For my locking bug, rpc.lockd is= stucked in rpcrecv state on=0A>> the=0A>> server. kill -9 doesn't affect= the=0A>> process, it's blocked.... (State: Ds)=0A>> =0A>> for the perfor= mances=0A>> =0A>> NFSv3: 60Mbps=0A>> NFSv4: 45Mbps=0A>> Regards,=0A>> =0A= >> Lo=C3=AFc Blot,=0A>> UNIX Systems, Network and Security Engineer=0A>> = http://www.unix-experience.fr=0A>>; =0A>> 10 d=C3=A9cembre 2014 13:56 "Ric= k Macklem" <rmacklem@uoguelph.ca>=0A>> a=0A>> =C3=A9crit:=0A>> =0A>> Loic= Blot wrote:=0A>> =0A>>> Hi Rick,=0A>>> I'm trying NFSv3.=0A>>> Some jail= s are starting very well but now i have an issue=0A>>> with=0A>>> lockd= =0A>>> after some minutes:=0A>>> =0A>>> nfs server 10.10.X.8:/jails: lock= d not responding=0A>>> nfs server 10.10.X.8:/jails lockd is alive again= =0A>>> =0A>>> I look at mbuf, but i seems there is no problem.=0A>> =0A>>= Well, if you need locks to be visible across multiple=0A>> clients,=0A>>= then=0A>> I'm afraid you are stuck with using NFSv4 and the=0A>> perform= ance=0A>> you=0A>> get=0A>> from it. (There is no way to do file handle a= ffinity for=0A>> NFSv4=0A>> because=0A>> the read and write ops are burie= d in the compound RPC and=0A>> not=0A>> easily=0A>> recognized.)=0A>> =0A= >> If the locks don't need to be visible across multiple=0A>> clients,=0A= >> I'd=0A>> suggest trying the "nolockd" option with nfsv3.=0A>> =0A>>> H= ere is my rc.conf on server:=0A>>> =0A>>> nfs_server_enable=3D"YES"=0A>>>= nfsv4_server_enable=3D"YES"=0A>>> nfsuserd_enable=3D"YES"=0A>>> nfsd_ser= ver_flags=3D"-u -t -n 256"=0A>>> mountd_enable=3D"YES"=0A>>> mountd_flags= =3D"-r"=0A>>> nfsuserd_flags=3D"-usertimeout 0 -force 20"=0A>>> rpcbind_e= nable=3D"YES"=0A>>> rpc_lockd_enable=3D"YES"=0A>>> rpc_statd_enable=3D"YE= S"=0A>>> =0A>>> Here is the client:=0A>>> =0A>>> nfsuserd_enable=3D"YES"= =0A>>> nfsuserd_flags=3D"-usertimeout 0 -force 20"=0A>>> nfscbd_enable=3D= "YES"=0A>>> rpc_lockd_enable=3D"YES"=0A>>> rpc_statd_enable=3D"YES"=0A>>>= =0A>>> Have you got an idea ?=0A>>> =0A>>> Regards,=0A>>> =0A>>> Lo=C3= =AFc Blot,=0A>>> UNIX Systems, Network and Security Engineer=0A>>> http:/= /www.unix-experience.fr=0A>>> =0A>>> 9 d=C3=A9cembre 2014 04:31 "Rick Mac= klem" <rmacklem@uoguelph.ca>=0A>>> a=0A>>> =C3=A9crit:=0A>>>> Loic Blot w= rote:=0A>>>> =0A>>>>> Hi rick,=0A>>>>> =0A>>>>> I waited 3 hours (no lag = at jail launch) and now I do:=0A>>>>> sysrc=0A>>>>> memcached_flags=3D"-v= -m 512"=0A>>>>> Command was very very slow...=0A>>>>> =0A>>>>> Here is a= dd over NFS:=0A>>>>> =0A>>>>> 601062912 bytes transferred in 21.060679 s= ecs (28539579=0A>>>>> bytes/sec)=0A>>>> =0A>>>> Can you try the same read= using an NFSv3 mount?=0A>>>> (If it runs much faster, you have probably = been bitten by=0A>>>> the=0A>>>> ZFS=0A>>>> "sequential vs random" read h= euristic which I've been told=0A>>>> things=0A>>>> NFS is doing "random" = reads without file handle affinity.=0A>>>> File=0A>>>> handle affinity is= very hard to do for NFSv4, so it isn't=0A>>>> done.)=0A>> =0A>> I was ac= tually suggesting that you try the "dd" over nfsv3=0A>> to=0A>> see=0A>> = how=0A>> the performance compared with nfsv4. If you do that, please=0A>>= post=0A>> the=0A>> comparable results.=0A>> =0A>> Someday I would like t= o try and get ZFS's sequential vs=0A>> random=0A>> read=0A>> heuristic mo= dified and any info on what difference in=0A>> performance=0A>> that=0A>>= might make for NFS would be useful.=0A>> =0A>> rick=0A>> =0A>> rick=0A>>= =0A>> This is quite slow...=0A>> =0A>> You can found some nfsstat below = (command isn't finished=0A>> yet)=0A>> =0A>> nfsstat -c -w 1=0A>> =0A>> G= tAttr Lookup Rdlink Read Write Rename Access Rddir=0A>> 0 0 0 0 0 0 0 0= =0A>> 4 0 0 0 0 0 16 0=0A>> 2 0 0 0 0 0 17 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 = 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 4 0 0 0 0 = 4 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0= 0 0 0 0 0 0 0=0A>> 4 0 0 0 0 0 3 0=0A>> 0 0 0 0 0 0 3 0=0A>> 37 10 0 8 0= 0 14 1=0A>> 18 16 0 4 1 2 4 0=0A>> 78 91 0 82 6 12 30 0=0A>> 19 18 0 2 2= 4 2 0=0A>> 0 0 0 0 2 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> GtAttr Lookup Rdlin= k Read Write Rename Access Rddir=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 = 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 1 0 0 0 0 1 0=0A>> 4 6 0 0 6 0 3 0=0A>> 2 0= 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 1 0 0 0 0 0 0 0=0A>> 0 0 0 0 1 0 0= 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 = 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 = 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 6 108 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>>= 0 0 0 0 0 0 0 0=0A>> GtAttr Lookup Rdlink Read Write Rename Access Rddir= =0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 = 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 = 0=0A>> 98 54 0 86 11 0 25 0=0A>> 36 24 0 39 25 0 10 1=0A>> 67 8 0 63 63 0= 41 0=0A>> 34 0 0 35 34 0 0 0=0A>> 75 0 0 75 77 0 0 0=0A>> 34 0 0 35 35 0= 0 0=0A>> 75 0 0 74 76 0 0 0=0A>> 33 0 0 34 33 0 0 0=0A>> 0 0 0 0 5 0 0 0= =0A>> 0 0 0 0 0 0 6 0=0A>> 11 0 0 0 0 0 11 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 = 17 0 0 0 0 1 0=0A>> GtAttr Lookup Rdlink Read Write Rename Access Rddir= =0A>> 4 5 0 0 0 0 12 0=0A>> 2 0 0 0 0 0 26 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 = 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 = 0 0=0A>> 0 4 0 0 0 0 4 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0= 0 0 0 0 0 0 0=0A>> 4 0 0 0 0 0 2 0=0A>> 2 0 0 0 0 0 24 0=0A>> 0 0 0 0 0 = 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>>= 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> GtAttr Lo= okup Rdlink Read Write Rename Access Rddir=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 = 0 0 0 0 0 0=0A>> 4 0 0 0 0 0 7 0=0A>> 2 1 0 0 0 0 1 0=0A>> 0 0 0 0 2 0 0 = 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 6 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0= 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>>=200 0 0 0 0 0= 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 4 6 0 0 0 0 3 0=0A>> 0 0 0 0 0 0 0 0=0A>> = 2 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 = 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> GtAttr Lookup Rdlink Read Write Rename Ac= cess Rddir=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0= =0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 4 71 0 0 0 0 0 0=0A>> 0 1= 0 0 0 0 0 0=0A>> 2 36 0 0 0 0 1 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 = 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 1 0 0 0 0 0 1 0=0A>> 0= 0 0 0 0 0 0 0=0A>> 0 0 0 0 0 0 0 0=0A>> 79 6 0 79 79 0 2 0=0A>> 25 0 0 2= 5 26 0 6 0=0A>> 43 18 0 39 46 0 23 0=0A>> 36 0 0 36 36 0 31 0=0A>> 68 1 0= 66 68 0 0 0=0A>> GtAttr Lookup Rdlink Read Write Rename Access Rddir=0A>= > 36 0 0 36 36 0 0 0=0A>> 48 0 0 48 49 0 0 0=0A>> 20 0 0 20 20 0 0 0=0A>>= 0 0 0 0 0 0 0 0=0A>> 3 14 0 1 0 0 11 0=0A>> 0 0 0 0 0 0 0 0=0A>> 0 0 0 0= 0 0 0 0=0A>> 0 4 0 0 0 0 4 0=0A>> 0 0 0 0 0 0 0 0=0A>> 4 22 0 0 0 0 16 0= =0A>> 2 0 0 0 0 0 23 0=0A>> =0A>> Regards,=0A>> =0A>> Lo=C3=AFc Blot,=0A>= > UNIX Systems, Network and Security Engineer=0A>> http://www.unix-experi= ence.fr=0A>> =0A>> 8 d=C3=A9cembre 2014 09:36 "Lo=C3=AFc Blot"=0A>> <loic= .blot@unix-experience.fr> a=0A>> =C3=A9crit:=0A>>> Hi Rick,=0A>>> I stopp= ed the jails this week-end and started it this=0A>>> morning,=0A>>> i'll= =0A>>> give you some stats this week.=0A>>> =0A>>> Here is my nfsstat -m = output (with your rsize/wsize=0A>>> tweaks)=0A>> =0A>> =0A> nfsv4,tcp,res= vport,hard,cto,sec=3Dsys,acdirmin=3D3,acdirmax=3D60,acregmin=3D5,acregmax= =3D60,nametimeo=3D60,negna=0A>> =0A>> =0A> etimeo=3D60,rsize=3D32768,wsiz= e=3D32768,readdirsize=3D32768,readahead=3D1,wcommitsize=3D773136,timeout= =3D120,retra=0A>> =0A>> s=3D2147483647=0A>> =0A>> On server side my disks= are on a raid controller which show a=0A>> 512b=0A>> volume and write pe= rformances=0A>> are very honest (dd if=3D/dev/zero of=3D/jails/test.dd bs= =3D4096=0A>> count=3D100000000 =3D> 450MBps)=0A>> =0A>> Regards,=0A>> =0A= >> Lo=C3=AFc Blot,=0A>> UNIX Systems, Network and Security Engineer=0A>> = http://www.unix-experience.fr=0A>>; =0A>> 5 d=C3=A9cembre 2014 15:14 "Rick= Macklem" <rmacklem@uoguelph.ca> a=0A>> =C3=A9crit:=0A>> =0A>> Loic Blot = wrote:=0A>> =0A>> Hi,=0A>> i'm trying to create a virtualisation environm= ent based on=0A>> jails.=0A>> Those jails are stored under a big ZFS pool= on a FreeBSD=0A>> 9.3=0A>> which=0A>> export a NFSv4 volume. This NFSv4 = volume was mounted on a=0A>> big=0A>> hypervisor (2 Xeon E5v3 + 128GB mem= ory and 8 ports (but=0A>> only 1=0A>> was=0A>> used at this time).=0A>> = =0A>> The problem is simple, my hypervisors runs 6 jails (used 1%=0A>> cp= u=0A>> and=0A>> 10GB RAM approximatively and less than 1MB bandwidth) and= =0A>> works=0A>> fine at start but the system slows down and after 2-3 da= ys=0A>> become=0A>> unusable. When i look at top command i see 80-100% on= =0A>> system=0A>> and=0A>> commands are very very slow. Many process are = tagged with=0A>> nfs_cl*.=0A>> =0A>> To be honest, I would expect the slo= wness to be because of=0A>> slow=0A>> response=0A>> from the NFSv4 server= , but if you do:=0A>> # ps axHl=0A>> on a client when it is slow and post= that, it would give us=0A>> some=0A>> more=0A>> information on where the= client side processes are sitting.=0A>> If you also do something like:= =0A>> # nfsstat -c -w 1=0A>> and let it run for a while, that should show= you how many=0A>> RPCs=0A>> are=0A>> being done and which ones.=0A>> =0A= >> # nfsstat -m=0A>> will show you what your mount is actually using.=0A>= > The only mount option I can suggest trying is=0A>> "rsize=3D32768,wsize= =3D32768",=0A>> since some network environments have difficulties with 64= K.=0A>> =0A>> There are a few things you can try on the NFSv4 server side= ,=0A>> if=0A>> it=0A>> appears=0A>> that the clients are generating a lar= ge RPC load.=0A>> - disabling the DRC cache for TCP by setting=0A>> vfs.n= fsd.cachetcp=3D0=0A>> - If the server is seeing a large write RPC load, t= hen=0A>> "sync=3Ddisabled"=0A>> might help, although it does run a risk o= f data loss when=0A>> the=0A>> server=0A>> crashes.=0A>> Then there are a= couple of other ZFS related things (I'm not=0A>> a=0A>> ZFS=0A>> guy,=0A= >> but these have shown up on the mailing lists).=0A>> - make sure your v= olumes are 4K aligned and ashift=3D12 (in=0A>> case a=0A>> drive=0A>> tha= t uses 4K sectors is pretending to be 512byte sectored)=0A>> - never run = over 70-80% full if write performance is an=0A>> issue=0A>> - use a zil o= n an SSD with good write performance=0A>> =0A>> The only NFSv4 thing I ca= n tell you is that it is known that=0A>> ZFS's=0A>> algorithm for determi= ning sequential vs random I/O fails for=0A>> NFSv4=0A>> during writing an= d this can be a performance hit. The only=0A>> workaround=0A>> is to use = NFSv3 mounts, since file handle affinity=0A>> apparently=0A>> fixes=0A>> = the problem and this is only done for NFSv3.=0A>> =0A>> rick=0A>> =0A>> I= saw that there are TSO issues with igb then i'm trying to=0A>> disable= =0A>> it with sysctl but the situation wasn't solved.=0A>> =0A>> Someone = has got ideas ? I can give you more informations if=0A>> you=0A>> need.= =0A>> =0A>> Thanks in advance.=0A>> Regards,=0A>> =0A>> Lo=C3=AFc Blot,= =0A>> UNIX Systems, Network and Security Engineer=0A>> http://www.unix-ex= perience.fr=0A>> _______________________________________________=0A>> fre= ebsd-fs@freebsd.org mailing list=0A>> http://lists.freebsd.org/mailman/li= stinfo/freebsd-fs=0A>> To unsubscribe, send any mail to=0A>> "freebsd-fs-= unsubscribe@freebsd.org"=0A>> =0A>> _____________________________________= __________=0A>> freebsd-fs@freebsd.org mailing list=0A>> http://lists.fre= ebsd.org/mailman/listinfo/freebsd-fs=0A>> To unsubscribe, send any mail t= o=0A>> "freebsd-fs-unsubscribe@freebsd.org"=0A>> =0A>> __________________= _____________________________=0A>> freebsd-fs@freebsd.org mailing list=0A= >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A>>; To unsubscri= be, send any mail to=0A>> "freebsd-fs-unsubscribe@freebsd.org"=0A>> =0A>>= _______________________________________________=0A>> freebsd-fs@freebsd.= org mailing list=0A>> http://lists.freebsd.org/mailman/listinfo/freebsd-f= s=0A>> To unsubscribe, send any mail to=0A>> "freebsd-fs-unsubscribe@free= bsd.org"=0A>> _______________________________________________=0A>> freebs= d-fs@freebsd.org mailing list=0A>> http://lists.freebsd.org/mailman/listi= nfo/freebsd-fs=0A>> To unsubscribe, send any mail to=0A>> "freebsd-fs-uns= ubscribe@freebsd.org"=0A>> =0A>> ________________________________________= _______=0A>> freebsd-fs@freebsd.org mailing list=0A>> http://lists.freebs= d.org/mailman/listinfo/freebsd-fs=0A>> To unsubscribe, send any mail to "= freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?82f8ef92c6bf5f2323bae5ce5e4e2394>