Date: Tue, 30 Dec 2014 15:07:37 +0000 From: "=?utf-8?B?TG/Dr2MgQmxvdA==?=" <loic.blot@unix-experience.fr> To: "Rick Macklem" <rmacklem@uoguelph.ca> Cc: freebsd-fs@freebsd.org Subject: Re: ZFS vnode lock deadlock in zfs_fhtovp was: High Kernel Load with nfsv4 Message-ID: <82f8ef92c6bf5f2323bae5ce5e4e2394@mail.unix-experience.fr> In-Reply-To: <1479765128.1118136.1419290403230.JavaMail.root@uoguelph.ca> References: <1479765128.1118136.1419290403230.JavaMail.root@uoguelph.ca>
index | next in thread | previous in thread | raw e-mail
Hi Rick, i upgraded my jail host from FreeBSD 9.3 to 10.1 to use NFS v4.1 (mountoptions: rw,rsize=32768,wsize=32768,tcp,nfsv4,minorversion=1) Performance is quite stable but it's slow. Not as slow as before but slow... services was launched but no client are using them and system CPU % was 10-50%. I don't see anything on NFSv4.1 server, it's perfectly stable and functionnal. Regards, Loïc Blot, UNIX Systems, Network and Security Engineer http://www.unix-experience.fr 23 décembre 2014 00:20 "Rick Macklem" <rmacklem@uoguelph.ca> a écrit: > Loic Blot wrote: > >> Hi, >> >> To clarify because of our exchanges. Here are the current sysctl >> options for server: >> >> vfs.nfsd.enable_nobodycheck=0 >> vfs.nfsd.enable_nogroupcheck=0 >> >> vfs.nfsd.maxthreads=200 >> vfs.nfsd.tcphighwater=10000 >> vfs.nfsd.tcpcachetimeo=300 >> vfs.nfsd.server_min_nfsvers=4 >> >> kern.maxvnodes=10000000 >> kern.ipc.maxsockbuf=4194304 >> net.inet.tcp.sendbuf_max=4194304 >> net.inet.tcp.recvbuf_max=4194304 >> >> vfs.lookup_shared=0 >> >> Regards, >> >> Loïc Blot, >> UNIX Systems, Network and Security Engineer >> http://www.unix-experience.fr >> >> 22 décembre 2014 09:42 "Loïc Blot" <loic.blot@unix-experience.fr> a >> écrit: >> >> Hi Rick, >> my 5 jails runs this weekend and now i have some stats on this >> monday. >> >> Hopefully deadlock was fixed, yeah, but everything isn't good :( >> >> On NFSv4 server (FreeBSD 10.1) system uses 35% CPU >> >> As i can see this is because of nfsd: >> >> 918 root 96 20 0 12352K 3372K rpcsvc 6 51.4H >> 273.68% nfsd: server (nfsd) >> >> If i look at dmesg i see: >> nfsd server cache flooded, try increasing vfs.nfsd.tcphighwater > > Well, you have a couple of choices: > 1 - Use NFSv4.1 (add "minorversion=1" to your mount options). > (NFSv4.1 avoids use of the DRC and instead uses something > called sessions. See below.) > OR > >> vfs.nfsd.tcphighwater was set to 10000, i increase it to 15000 > > 2 - Bump vfs.nfsd.tcphighwater way up, until you no longer see > "nfs server cache flooded" messages. (I think Garrett Wollman uses > 100000. (You may still see quite a bit of CPU overheads.) > > OR > > 3 - Set vfs.nfsd.cachetcp=0 (which disables the DRC and gets rid > of the CPU overheads). However, there is a risk of data corruption > if you have a client->server network partitioning of a moderate > duration, because a non-idempotent RPC may get redone, becasue > the client times out waiting for a reply. If a non-idempotent > RPC gets done twice on the server, data corruption can happen. > (The DRC provides improved correctness, but does add overhead.) > > If #1 works for you, it is the preferred solution, since Sessions > in NFSv4.1 solves the correctness problem in a good, space bound > way. A session basically has N (usually 32 or 64) slots and only > allows one outstanding RPC/slot. As such, it can cache the previous > reply for each slot (32 or 64 of them) and guarantee "exactly once" > RPC semantics. > > rick > >> Here is 'nfsstat -s' output: >> >> Server Info: >> Getattr Setattr Lookup Readlink Read Write Create >> Remove >> 12600652 1812 2501097 156 1386423 1983729 123 >> 162067 >> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus >> Access >> 36762 9 0 0 0 3147 0 >> 623524 >> Mknod Fsstat Fsinfo PathConf Commit >> 0 0 0 0 328117 >> Server Ret-Failed >> 0 >> Server Faults >> 0 >> Server Cache Stats: >> Inprog Idem Non-idem Misses >> 0 0 0 12635512 >> Server Write Gathering: >> WriteOps WriteRPC Opsaved >> 1983729 1983729 0 >> >> And here is 'procstat -kk' for nfsd (server) >> >> 918 100528 nfsd nfsd: master mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 >> _cv_timedwait_sig_sbt+0x18b svc_run_internal+0x4a1 svc_run+0x1de >> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c >> amd64_syscall+0x351 Xfast_syscall+0xfb >> 918 100568 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100569 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100570 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100571 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100572 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100573 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100574 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100575 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100576 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100577 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100578 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100579 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100580 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100581 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100582 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100583 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100584 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100585 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100586 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100587 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100588 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100589 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100590 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100591 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100592 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100593 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100594 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100595 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100596 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100597 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100598 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100599 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100600 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100601 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100602 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100603 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100604 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100605 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100606 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100607 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100608 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100609 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100610 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100611 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100612 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100613 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100614 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100615 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100616 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100617 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100618 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100619 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100620 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100621 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100622 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100623 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100624 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100625 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100626 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100627 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100628 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100629 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100630 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100631 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100632 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100633 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100634 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100635 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100636 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100637 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100638 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100639 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100640 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100641 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100642 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100643 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100644 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100645 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100646 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100647 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100648 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100649 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100650 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100651 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100652 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100653 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100654 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100655 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100656 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100657 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100658 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100659 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100660 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100661 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100662 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> --- >> >> Now if we look at client (FreeBSD 9.3) >> >> We see system was very busy and do many and many interrupts >> >> CPU: 0.0% user, 0.0% nice, 37.8% system, 51.2% interrupt, 11.0% >> idle >> >> A look at process list shows that there are many sendmail process in >> state nfstry >> >> nfstry 18 32:27 0.88% sendmail: Queue runner@00:30:00 for >> /var/spool/clientm >> >> Here is 'nfsstat -c' output: >> >> Client Info: >> Rpc Counts: >> Getattr Setattr Lookup Readlink Read Write Create >> Remove >> 1051347 1724 2494481 118 903902 1901285 162676 >> 161899 >> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus >> Access >> 36744 2 0 114 40 3131 0 >> 544136 >> Mknod Fsstat Fsinfo PathConf Commit >> 9 0 0 0 245821 >> Rpc Info: >> TimedOut Invalid X Replies Retries Requests >> 0 0 0 0 8356557 >> Cache Info: >> Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits >> Misses >> 108754455 491475 54229224 2437229 46814561 821723 5132123 >> 1871871 >> BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits >> Misses >> 144035 118 53736 2753 27813 1 57238839 >> 544205 >> >> If you need more things, tell me, i let the PoC in this state. >> >> Thanks >> >> Regards, >> >> Loïc Blot, >> UNIX Systems, Network and Security Engineer >> http://www.unix-experience.fr >> >> 21 décembre 2014 01:33 "Rick Macklem" <rmacklem@uoguelph.ca> a écrit: >> >> Loic Blot wrote: >> >>> Hi Rick, >>> ok, i don't need locallocks, i haven't understand option was for >>> that >>> usage, i removed it. >>> I do more tests on monday. >>> Thanks for the deadlock fix, for other people :) >> >> Good. Please let us know if running with vfs.nfsd.enable_locallocks=0 >> gets rid of the deadlocks? (I think it fixes the one you saw.) >> >> On the performance side, you might also want to try different values >> of >> readahead, if the Linux client has such a mount option. (With the >> NFSv4-ZFS sequential vs random I/O heuristic, I have no idea what the >> optimal readahead value would be.) >> >> Good luck with it and please let us know how it goes, rick >> ps: I now have a patch to fix the deadlock when >> vfs.nfsd.enable_locallocks=1 >> is set. I'll post it for anyone who is interested after I put it >> through some testing. >> >> -- >> Best regards, >> Loïc BLOT, >> UNIX systems, security and network engineer >> http://www.unix-experience.fr >> >> Le jeudi 18 décembre 2014 à 19:46 -0500, Rick Macklem a écrit : >> >> Loic Blot wrote: >>> Hi rick, >>> i tried to start a LXC container on Debian Squeeze from my >>> freebsd >>> ZFS+NFSv4 server and i also have a deadlock on nfsd >>> (vfs.lookup_shared=0). Deadlock procs each time i launch a >>> squeeze >>> container, it seems (3 tries, 3 fails). >> >> Well, I`ll take a look at this `procstat -kk`, but the only thing >> I`ve seen posted w.r.t. avoiding deadlocks in ZFS is to not use >> nullfs. (I have no idea if you are using any nullfs mounts, but >> if so, try getting rid of them.) >> >> Here`s a high level post about the ZFS and vnode locking problem, >> but there is no patch available, as far as I know. >> >> http://docs.FreeBSD.org/cgi/mid.cgi?54739F41.8030407 >> >> rick >> >> 921 - D 0:00.02 nfsd: server (nfsd) >> >> Here is the procstat -kk >> >> PID TID COMM TDNAME KSTACK >> 921 100538 nfsd nfsd: master mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad >> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554 >> svc_run_internal+0xc77 svc_run+0x1de nfsrvd_nfsd+0x1ca >> nfssvc_nfsd+0x107 sys_nfssvc+0x9c >> 921 100572 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100573 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100574 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100575 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100576 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100577 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100578 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100579 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100580 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100581 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100582 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100583 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100584 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100585 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100586 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100587 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100588 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100589 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100590 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100591 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100592 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100593 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100594 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100595 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100596 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100597 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100598 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100599 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100600 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100601 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100602 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100603 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100604 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100605 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100606 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100607 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100608 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100609 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100610 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100611 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100612 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100613 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100614 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100615 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100616 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1 >> nfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >> 921 100617 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100618 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >> 921 100619 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100620 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100621 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100622 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100623 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100624 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100625 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100626 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100627 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100628 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100629 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100630 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100631 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100632 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100633 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100634 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100635 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100636 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100637 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100638 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100639 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100640 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100641 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100642 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100643 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100644 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100645 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100646 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100647 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100648 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100649 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100650 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100651 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100652 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100653 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100654 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100655 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100656 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100657 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100658 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100659 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100660 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100661 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100662 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100663 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100664 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100665 nfsd nfsd: service mi_switch+0xe1 >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >> _cv_wait_sig+0x16a >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 921 100666 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8 >> nfsrvd_dorpc+0xc76 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> >> Regards, >> >> Loïc Blot, >> UNIX Systems, Network and Security Engineer >> http://www.unix-experience.fr >> >> 15 décembre 2014 15:18 "Rick Macklem" <rmacklem@uoguelph.ca> a >> écrit: >> >> Loic Blot wrote: >> >>> For more informations, here is procstat -kk on nfsd, if you >>> need >>> more >>> hot datas, tell me. >>> >>> Regards, PID TID COMM TDNAME KSTACK >>> 918 100529 nfsd nfsd: master mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_run+0x1de >>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c >>> amd64_syscall+0x351 >> >> Well, most of the threads are stuck like this one, waiting for >> a >> vnode >> lock in ZFS. All of them appear to be in zfs_fhtovp(). >> I`m not a ZFS guy, so I can`t help much. I`ll try changing the >> subject line >> to include ZFS vnode lock, so maybe the ZFS guys will take a >> look. >> >> The only thing I`ve seen suggested is trying: >> sysctl vfs.lookup_shared=0 >> to disable shared vop_lookup()s. Apparently zfs_lookup() >> doesn`t >> obey the vnode locking rules for lookup and rename, according >> to >> the posting I saw. >> >> I`ve added a couple of comments about the other threads below, >> but >> they are all either waiting for an RPC request or waiting for >> the >> threads stuck on the ZFS vnode lock to complete. >> >> rick >> >>> 918 100564 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>> _cv_wait_sig+0x16a >>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>> fork_trampoline+0xe >> >> Fyi, this thread is just waiting for an RPC to arrive. (Normal) >> >>> 918 100565 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>> _cv_wait_sig+0x16a >>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>> fork_trampoline+0xe >>> 918 100566 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>> _cv_wait_sig+0x16a >>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>> fork_trampoline+0xe >>> 918 100567 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>> _cv_wait_sig+0x16a >>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>> fork_trampoline+0xe >>> 918 100568 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>> _cv_wait_sig+0x16a >>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>> fork_trampoline+0xe >>> 918 100569 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>> _cv_wait_sig+0x16a >>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>> fork_trampoline+0xe >>> 918 100570 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf >>> _cv_wait_sig+0x16a >>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a >>> fork_trampoline+0xe >>> 918 100571 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >>> 918 100572 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >>> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8 >>> nfsrvd_dorpc+0xc76 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >> >> This one (and a few others) are waiting for the nfsv4_lock. >> This >> happens >> because other threads are stuck with RPCs in progress. (ie. The >> ones >> waiting on the vnode lock in zfs_fhtovp().) >> For these, the RPC needs to lock out other threads to do the >> operation, >> so it waits for the nfsv4_lock() which can exclusively lock the >> NFSv4 >> data structures once all other nfsd threads complete their RPCs >> in >> progress. >> >>> 918 100573 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >> >> Same as above. >> >>> 918 100574 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100575 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100576 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100577 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100578 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100579 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100580 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100581 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100582 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100583 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100584 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100585 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100586 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100587 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100588 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100589 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100590 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100591 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100592 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100593 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100594 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100595 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100596 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100597 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100598 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100599 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100600 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100601 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100602 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100603 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100604 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100605 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100606 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >>> 918 100607 nfsd nfsd: service mi_switch+0xe1 >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >>> zfs_fhtovp+0x38d >>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >>> nfssvc_program+0x554 svc_run_internal+0xc77 >>> svc_thread_start+0xb >>> fork_exit+0x9a fork_trampoline+0xe >> >> Lots more waiting for the ZFS vnode lock in zfs_fhtovp(). >> >> 918 100608 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1 >> nfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >> 918 100609 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100610 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad >> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554 >> svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a >> fork_trampoline+0xe >> 918 100611 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >> 918 100612 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >> 918 100613 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >> 918 100614 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >> 918 100615 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >> 918 100616 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >> 918 100617 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >> 918 100618 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >> 918 100619 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100620 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100621 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >> 918 100622 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100623 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b >> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >> 918 100624 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100625 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100626 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100627 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100628 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100629 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100630 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100631 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100632 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100633 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100634 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100635 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100636 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100637 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100638 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100639 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100640 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100641 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100642 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100643 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100644 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100645 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100646 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100647 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100648 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100649 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100650 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100651 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100652 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100653 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100654 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100655 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100656 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100657 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> 918 100658 nfsd nfsd: service mi_switch+0xe1 >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 >> zfs_fhtovp+0x38d >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 >> nfssvc_program+0x554 svc_run_internal+0xc77 >> svc_thread_start+0xb >> fork_exit+0x9a fork_trampoline+0xe >> >> Loïc Blot, >> UNIX Systems, Network and Security Engineer >> http://www.unix-experience.fr >> >> 15 décembre 2014 13:29 "Loïc Blot" >> <loic.blot@unix-experience.fr> >> a >> écrit: >> >> Hmmm... >> now i'm experiencing a deadlock. >> >> 0 918 915 0 21 0 12352 3372 zfs D - 1:48.64 nfsd: server >> (nfsd) >> >> the only issue was to reboot the server, but after rebooting >> deadlock arrives a second time when i >> start my jails over NFS. >> >> Regards, >> >> Loïc Blot, >> UNIX Systems, Network and Security Engineer >> http://www.unix-experience.fr >> >> 15 décembre 2014 10:07 "Loïc Blot" >> <loic.blot@unix-experience.fr> >> a >> écrit: >> >> Hi Rick, >> after talking with my N+1, NFSv4 is required on our >> infrastructure. >> I tried to upgrade NFSv4+ZFS >> server from 9.3 to 10.1, i hope this will resolve some >> issues... >> >> Regards, >> >> Loïc Blot, >> UNIX Systems, Network and Security Engineer >> http://www.unix-experience.fr >> >> 10 décembre 2014 15:36 "Loïc Blot" >> <loic.blot@unix-experience.fr> >> a >> écrit: >> >> Hi Rick, >> thanks for your suggestion. >> For my locking bug, rpc.lockd is stucked in rpcrecv state on >> the >> server. kill -9 doesn't affect the >> process, it's blocked.... (State: Ds) >> >> for the performances >> >> NFSv3: 60Mbps >> NFSv4: 45Mbps >> Regards, >> >> Loïc Blot, >> UNIX Systems, Network and Security Engineer >> http://www.unix-experience.fr >> >> 10 décembre 2014 13:56 "Rick Macklem" <rmacklem@uoguelph.ca> >> a >> écrit: >> >> Loic Blot wrote: >> >>> Hi Rick, >>> I'm trying NFSv3. >>> Some jails are starting very well but now i have an issue >>> with >>> lockd >>> after some minutes: >>> >>> nfs server 10.10.X.8:/jails: lockd not responding >>> nfs server 10.10.X.8:/jails lockd is alive again >>> >>> I look at mbuf, but i seems there is no problem. >> >> Well, if you need locks to be visible across multiple >> clients, >> then >> I'm afraid you are stuck with using NFSv4 and the >> performance >> you >> get >> from it. (There is no way to do file handle affinity for >> NFSv4 >> because >> the read and write ops are buried in the compound RPC and >> not >> easily >> recognized.) >> >> If the locks don't need to be visible across multiple >> clients, >> I'd >> suggest trying the "nolockd" option with nfsv3. >> >>> Here is my rc.conf on server: >>> >>> nfs_server_enable="YES" >>> nfsv4_server_enable="YES" >>> nfsuserd_enable="YES" >>> nfsd_server_flags="-u -t -n 256" >>> mountd_enable="YES" >>> mountd_flags="-r" >>> nfsuserd_flags="-usertimeout 0 -force 20" >>> rpcbind_enable="YES" >>> rpc_lockd_enable="YES" >>> rpc_statd_enable="YES" >>> >>> Here is the client: >>> >>> nfsuserd_enable="YES" >>> nfsuserd_flags="-usertimeout 0 -force 20" >>> nfscbd_enable="YES" >>> rpc_lockd_enable="YES" >>> rpc_statd_enable="YES" >>> >>> Have you got an idea ? >>> >>> Regards, >>> >>> Loïc Blot, >>> UNIX Systems, Network and Security Engineer >>> http://www.unix-experience.fr >>> >>> 9 décembre 2014 04:31 "Rick Macklem" <rmacklem@uoguelph.ca> >>> a >>> écrit: >>>> Loic Blot wrote: >>>> >>>>> Hi rick, >>>>> >>>>> I waited 3 hours (no lag at jail launch) and now I do: >>>>> sysrc >>>>> memcached_flags="-v -m 512" >>>>> Command was very very slow... >>>>> >>>>> Here is a dd over NFS: >>>>> >>>>> 601062912 bytes transferred in 21.060679 secs (28539579 >>>>> bytes/sec) >>>> >>>> Can you try the same read using an NFSv3 mount? >>>> (If it runs much faster, you have probably been bitten by >>>> the >>>> ZFS >>>> "sequential vs random" read heuristic which I've been told >>>> things >>>> NFS is doing "random" reads without file handle affinity. >>>> File >>>> handle affinity is very hard to do for NFSv4, so it isn't >>>> done.) >> >> I was actually suggesting that you try the "dd" over nfsv3 >> to >> see >> how >> the performance compared with nfsv4. If you do that, please >> post >> the >> comparable results. >> >> Someday I would like to try and get ZFS's sequential vs >> random >> read >> heuristic modified and any info on what difference in >> performance >> that >> might make for NFS would be useful. >> >> rick >> >> rick >> >> This is quite slow... >> >> You can found some nfsstat below (command isn't finished >> yet) >> >> nfsstat -c -w 1 >> >> GtAttr Lookup Rdlink Read Write Rename Access Rddir >> 0 0 0 0 0 0 0 0 >> 4 0 0 0 0 0 16 0 >> 2 0 0 0 0 0 17 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 4 0 0 0 0 4 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 4 0 0 0 0 0 3 0 >> 0 0 0 0 0 0 3 0 >> 37 10 0 8 0 0 14 1 >> 18 16 0 4 1 2 4 0 >> 78 91 0 82 6 12 30 0 >> 19 18 0 2 2 4 2 0 >> 0 0 0 0 2 0 0 0 >> 0 0 0 0 0 0 0 0 >> GtAttr Lookup Rdlink Read Write Rename Access Rddir >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 1 0 0 0 0 1 0 >> 4 6 0 0 6 0 3 0 >> 2 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 1 0 0 0 0 0 0 0 >> 0 0 0 0 1 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 6 108 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> GtAttr Lookup Rdlink Read Write Rename Access Rddir >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 98 54 0 86 11 0 25 0 >> 36 24 0 39 25 0 10 1 >> 67 8 0 63 63 0 41 0 >> 34 0 0 35 34 0 0 0 >> 75 0 0 75 77 0 0 0 >> 34 0 0 35 35 0 0 0 >> 75 0 0 74 76 0 0 0 >> 33 0 0 34 33 0 0 0 >> 0 0 0 0 5 0 0 0 >> 0 0 0 0 0 0 6 0 >> 11 0 0 0 0 0 11 0 >> 0 0 0 0 0 0 0 0 >> 0 17 0 0 0 0 1 0 >> GtAttr Lookup Rdlink Read Write Rename Access Rddir >> 4 5 0 0 0 0 12 0 >> 2 0 0 0 0 0 26 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 4 0 0 0 0 4 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 4 0 0 0 0 0 2 0 >> 2 0 0 0 0 0 24 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> GtAttr Lookup Rdlink Read Write Rename Access Rddir >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 4 0 0 0 0 0 7 0 >> 2 1 0 0 0 0 1 0 >> 0 0 0 0 2 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 6 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 4 6 0 0 0 0 3 0 >> 0 0 0 0 0 0 0 0 >> 2 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> GtAttr Lookup Rdlink Read Write Rename Access Rddir >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 4 71 0 0 0 0 0 0 >> 0 1 0 0 0 0 0 0 >> 2 36 0 0 0 0 1 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 1 0 0 0 0 0 1 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 79 6 0 79 79 0 2 0 >> 25 0 0 25 26 0 6 0 >> 43 18 0 39 46 0 23 0 >> 36 0 0 36 36 0 31 0 >> 68 1 0 66 68 0 0 0 >> GtAttr Lookup Rdlink Read Write Rename Access Rddir >> 36 0 0 36 36 0 0 0 >> 48 0 0 48 49 0 0 0 >> 20 0 0 20 20 0 0 0 >> 0 0 0 0 0 0 0 0 >> 3 14 0 1 0 0 11 0 >> 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 >> 0 4 0 0 0 0 4 0 >> 0 0 0 0 0 0 0 0 >> 4 22 0 0 0 0 16 0 >> 2 0 0 0 0 0 23 0 >> >> Regards, >> >> Loïc Blot, >> UNIX Systems, Network and Security Engineer >> http://www.unix-experience.fr >> >> 8 décembre 2014 09:36 "Loïc Blot" >> <loic.blot@unix-experience.fr> a >> écrit: >>> Hi Rick, >>> I stopped the jails this week-end and started it this >>> morning, >>> i'll >>> give you some stats this week. >>> >>> Here is my nfsstat -m output (with your rsize/wsize >>> tweaks) >> >> > nfsv4,tcp,resvport,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negna >> >> > etimeo=60,rsize=32768,wsize=32768,readdirsize=32768,readahead=1,wcommitsize=773136,timeout=120,retra >> >> s=2147483647 >> >> On server side my disks are on a raid controller which show a >> 512b >> volume and write performances >> are very honest (dd if=/dev/zero of=/jails/test.dd bs=4096 >> count=100000000 => 450MBps) >> >> Regards, >> >> Loïc Blot, >> UNIX Systems, Network and Security Engineer >> http://www.unix-experience.fr >> >> 5 décembre 2014 15:14 "Rick Macklem" <rmacklem@uoguelph.ca> a >> écrit: >> >> Loic Blot wrote: >> >> Hi, >> i'm trying to create a virtualisation environment based on >> jails. >> Those jails are stored under a big ZFS pool on a FreeBSD >> 9.3 >> which >> export a NFSv4 volume. This NFSv4 volume was mounted on a >> big >> hypervisor (2 Xeon E5v3 + 128GB memory and 8 ports (but >> only 1 >> was >> used at this time). >> >> The problem is simple, my hypervisors runs 6 jails (used 1% >> cpu >> and >> 10GB RAM approximatively and less than 1MB bandwidth) and >> works >> fine at start but the system slows down and after 2-3 days >> become >> unusable. When i look at top command i see 80-100% on >> system >> and >> commands are very very slow. Many process are tagged with >> nfs_cl*. >> >> To be honest, I would expect the slowness to be because of >> slow >> response >> from the NFSv4 server, but if you do: >> # ps axHl >> on a client when it is slow and post that, it would give us >> some >> more >> information on where the client side processes are sitting. >> If you also do something like: >> # nfsstat -c -w 1 >> and let it run for a while, that should show you how many >> RPCs >> are >> being done and which ones. >> >> # nfsstat -m >> will show you what your mount is actually using. >> The only mount option I can suggest trying is >> "rsize=32768,wsize=32768", >> since some network environments have difficulties with 64K. >> >> There are a few things you can try on the NFSv4 server side, >> if >> it >> appears >> that the clients are generating a large RPC load. >> - disabling the DRC cache for TCP by setting >> vfs.nfsd.cachetcp=0 >> - If the server is seeing a large write RPC load, then >> "sync=disabled" >> might help, although it does run a risk of data loss when >> the >> server >> crashes. >> Then there are a couple of other ZFS related things (I'm not >> a >> ZFS >> guy, >> but these have shown up on the mailing lists). >> - make sure your volumes are 4K aligned and ashift=12 (in >> case a >> drive >> that uses 4K sectors is pretending to be 512byte sectored) >> - never run over 70-80% full if write performance is an >> issue >> - use a zil on an SSD with good write performance >> >> The only NFSv4 thing I can tell you is that it is known that >> ZFS's >> algorithm for determining sequential vs random I/O fails for >> NFSv4 >> during writing and this can be a performance hit. The only >> workaround >> is to use NFSv3 mounts, since file handle affinity >> apparently >> fixes >> the problem and this is only done for NFSv3. >> >> rick >> >> I saw that there are TSO issues with igb then i'm trying to >> disable >> it with sysctl but the situation wasn't solved. >> >> Someone has got ideas ? I can give you more informations if >> you >> need. >> >> Thanks in advance. >> Regards, >> >> Loïc Blot, >> UNIX Systems, Network and Security Engineer >> http://www.unix-experience.fr >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to >> "freebsd-fs-unsubscribe@freebsd.org" >> >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to >> "freebsd-fs-unsubscribe@freebsd.org" >> >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to >> "freebsd-fs-unsubscribe@freebsd.org" >> >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to >> "freebsd-fs-unsubscribe@freebsd.org" >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to >> "freebsd-fs-unsubscribe@freebsd.org" >> >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?82f8ef92c6bf5f2323bae5ce5e4e2394>