From owner-freebsd-fs@FreeBSD.ORG Mon Dec 22 23:20:14 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 73A5BAB4 for ; Mon, 22 Dec 2014 23:20:14 +0000 (UTC) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id EE17964A7C for ; Mon, 22 Dec 2014 23:20:13 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ag0FAP2lmFSDaFve/2dsb2JhbABbg1hYBIMAw0MKhSZKAoEsAQEBAQF9hAwBAQEDAQEBARcBCAQnIAsFFhgCAg0ZAikBCSYGCAIFBAEaAgSIAwgNuhSVfAEBAQEBBQEBAQEBAQEBARmBIY4AAQEbATMHgi07EYEwBYlHiAiDHoMjMII0gjODQIQugzkigX8fgW4gMQEBBYEFOX4BAQE X-IronPort-AV: E=Sophos;i="5.07,627,1413259200"; d="scan'208";a="180375957" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-annu.net.uoguelph.ca with ESMTP; 22 Dec 2014 18:20:03 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 3C95DE7956; Mon, 22 Dec 2014 18:20:03 -0500 (EST) Date: Mon, 22 Dec 2014 18:20:03 -0500 (EST) From: Rick Macklem To: =?utf-8?B?TG/Dr2M=?= Blot Message-ID: <1479765128.1118136.1419290403230.JavaMail.root@uoguelph.ca> In-Reply-To: <811d455b0bcaeb43711e8108c96d4f2b@mail.unix-experience.fr> Subject: Re: ZFS vnode lock deadlock in zfs_fhtovp was: High Kernel Load with nfsv4 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.95.11] X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Dec 2014 23:20:14 -0000 Loic Blot wrote: > Hi, >=20 > To clarify because of our exchanges. Here are the current sysctl > options for server: >=20 > vfs.nfsd.enable_nobodycheck=3D0 > vfs.nfsd.enable_nogroupcheck=3D0 >=20 > vfs.nfsd.maxthreads=3D200 > vfs.nfsd.tcphighwater=3D10000 > vfs.nfsd.tcpcachetimeo=3D300 > vfs.nfsd.server_min_nfsvers=3D4 >=20 > kern.maxvnodes=3D10000000 > kern.ipc.maxsockbuf=3D4194304 > net.inet.tcp.sendbuf_max=3D4194304 > net.inet.tcp.recvbuf_max=3D4194304 >=20 > vfs.lookup_shared=3D0 >=20 > Regards, >=20 > Lo=C3=AFc Blot, > UNIX Systems, Network and Security Engineer > http://www.unix-experience.fr >=20 > 22 d=C3=A9cembre 2014 09:42 "Lo=C3=AFc Blot" a > =C3=A9crit: >=20 > Hi Rick, > my 5 jails runs this weekend and now i have some stats on this > monday. >=20 > Hopefully deadlock was fixed, yeah, but everything isn't good :( >=20 > On NFSv4 server (FreeBSD 10.1) system uses 35% CPU >=20 > As i can see this is because of nfsd: >=20 > 918 root 96 20 0 12352K 3372K rpcsvc 6 51.4H > 273.68% nfsd: server (nfsd) >=20 > If i look at dmesg i see: > nfsd server cache flooded, try increasing vfs.nfsd.tcphighwater >=20 Well, you have a couple of choices: 1 - Use NFSv4.1 (add "minorversion=3D1" to your mount options). (NFSv4.1 avoids use of the DRC and instead uses something called sessions. See below.) OR > vfs.nfsd.tcphighwater was set to 10000, i increase it to 15000 >=20 2 - Bump vfs.nfsd.tcphighwater way up, until you no longer see "nfs server cache flooded" messages. (I think Garrett Wollman uses 100000. (You may still see quite a bit of CPU overheads.) OR 3 - Set vfs.nfsd.cachetcp=3D0 (which disables the DRC and gets rid of the CPU overheads). However, there is a risk of data corruption if you have a client->server network partitioning of a moderate duration, because a non-idempotent RPC may get redone, becasue the client times out waiting for a reply. If a non-idempotent RPC gets done twice on the server, data corruption can happen. (The DRC provides improved correctness, but does add overhead.) If #1 works for you, it is the preferred solution, since Sessions in NFSv4.1 solves the correctness problem in a good, space bound way. A session basically has N (usually 32 or 64) slots and only allows one outstanding RPC/slot. As such, it can cache the previous reply for each slot (32 or 64 of them) and guarantee "exactly once" RPC semantics. rick > Here is 'nfsstat -s' output: >=20 > Server Info: > Getattr Setattr Lookup Readlink Read Write Create > Remove > 12600652 1812 2501097 156 1386423 1983729 123 > 162067 > Rename Link Symlink Mkdir Rmdir Readdir RdirPlus > Access > 36762 9 0 0 0 3147 0 > 623524 > Mknod Fsstat Fsinfo PathConf Commit > 0 0 0 0 328117 > Server Ret-Failed > 0 > Server Faults > 0 > Server Cache Stats: > Inprog Idem Non-idem Misses > 0 0 0 12635512 > Server Write Gathering: > WriteOps WriteRPC Opsaved > 1983729 1983729 0 >=20 > And here is 'procstat -kk' for nfsd (server) >=20 > 918 100528 nfsd nfsd: master mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 > _cv_timedwait_sig_sbt+0x18b svc_run_internal+0x4a1 svc_run+0x1de > nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c > amd64_syscall+0x351 Xfast_syscall+0xfb > 918 100568 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100569 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100570 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100571 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100572 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100573 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100574 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100575 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100576 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100577 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100578 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100579 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100580 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100581 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100582 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100583 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100584 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100585 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100586 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100587 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100588 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100589 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100590 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100591 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100592 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100593 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100594 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100595 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100596 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100597 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100598 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100599 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100600 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100601 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100602 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100603 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100604 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100605 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100606 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100607 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100608 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100609 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100610 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100611 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100612 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100613 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100614 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100615 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100616 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100617 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100618 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100619 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100620 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100621 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100622 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100623 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100624 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100625 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100626 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100627 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100628 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100629 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100630 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100631 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100632 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100633 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100634 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100635 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100636 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100637 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100638 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100639 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100640 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100641 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100642 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100643 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100644 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100645 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100646 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100647 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100648 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100649 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100650 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100651 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100652 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100653 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100654 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100655 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100656 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100657 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100658 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100659 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100660 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100661 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100662 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > --- >=20 > Now if we look at client (FreeBSD 9.3) >=20 > We see system was very busy and do many and many interrupts >=20 > CPU: 0.0% user, 0.0% nice, 37.8% system, 51.2% interrupt, 11.0% > idle >=20 > A look at process list shows that there are many sendmail process in > state nfstry >=20 > nfstry 18 32:27 0.88% sendmail: Queue runner@00:30:00 for > /var/spool/clientm >=20 > Here is 'nfsstat -c' output: >=20 > Client Info: > Rpc Counts: > Getattr Setattr Lookup Readlink Read Write Create > Remove > 1051347 1724 2494481 118 903902 1901285 162676 > 161899 > Rename Link Symlink Mkdir Rmdir Readdir RdirPlus > Access > 36744 2 0 114 40 3131 0 > 544136 > Mknod Fsstat Fsinfo PathConf Commit > 9 0 0 0 245821 > Rpc Info: > TimedOut Invalid X Replies Retries Requests > 0 0 0 0 8356557 > Cache Info: > Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits > Misses > 108754455 491475 54229224 2437229 46814561 821723 5132123 > 1871871 > BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits > Misses > 144035 118 53736 2753 27813 1 57238839 > 544205 >=20 > If you need more things, tell me, i let the PoC in this state. >=20 > Thanks >=20 > Regards, >=20 > Lo=C3=AFc Blot, > UNIX Systems, Network and Security Engineer > http://www.unix-experience.fr >=20 > 21 d=C3=A9cembre 2014 01:33 "Rick Macklem" a =C3= =A9crit: >=20 >=20 > Loic Blot wrote: >=20 > > Hi Rick, > > ok, i don't need locallocks, i haven't understand option was for > > that > > usage, i removed it. > > I do more tests on monday. > > Thanks for the deadlock fix, for other people :) >=20 > Good. Please let us know if running with vfs.nfsd.enable_locallocks=3D0 > gets rid of the deadlocks? (I think it fixes the one you saw.) >=20 > On the performance side, you might also want to try different values > of > readahead, if the Linux client has such a mount option. (With the > NFSv4-ZFS sequential vs random I/O heuristic, I have no idea what the > optimal readahead value would be.) >=20 > Good luck with it and please let us know how it goes, rick > ps: I now have a patch to fix the deadlock when > vfs.nfsd.enable_locallocks=3D1 > is set. I'll post it for anyone who is interested after I put it > through some testing. >=20 >=20 > -- > Best regards, > Lo=C3=AFc BLOT, > UNIX systems, security and network engineer > http://www.unix-experience.fr >=20 > Le jeudi 18 d=C3=A9cembre 2014 =C3=A0 19:46 -0500, Rick Macklem a =C3=A9c= rit : >=20 > Loic Blot wrote: > > Hi rick, > > i tried to start a LXC container on Debian Squeeze from my > > freebsd > > ZFS+NFSv4 server and i also have a deadlock on nfsd > > (vfs.lookup_shared=3D0). Deadlock procs each time i launch a > > squeeze > > container, it seems (3 tries, 3 fails). >=20 > Well, I`ll take a look at this `procstat -kk`, but the only thing > I`ve seen posted w.r.t. avoiding deadlocks in ZFS is to not use > nullfs. (I have no idea if you are using any nullfs mounts, but > if so, try getting rid of them.) >=20 > Here`s a high level post about the ZFS and vnode locking problem, > but there is no patch available, as far as I know. >=20 > http://docs.FreeBSD.org/cgi/mid.cgi?54739F41.8030407 >=20 > rick >=20 >=20 > 921 - D 0:00.02 nfsd: server (nfsd) >=20 > Here is the procstat -kk >=20 > PID TID COMM TDNAME KSTACK > 921 100538 nfsd nfsd: master mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad > nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554 > svc_run_internal+0xc77 svc_run+0x1de nfsrvd_nfsd+0x1ca > nfssvc_nfsd+0x107 sys_nfssvc+0x9c > 921 100572 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100573 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100574 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100575 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100576 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100577 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100578 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100579 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100580 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100581 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100582 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100583 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100584 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100585 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100586 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100587 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100588 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100589 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100590 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100591 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100592 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100593 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100594 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100595 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100596 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100597 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100598 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100599 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100600 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100601 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100602 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100603 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100604 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100605 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100606 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100607 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100608 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100609 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100610 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100611 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100612 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100613 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100614 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100615 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100616 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b > nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1 > nfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe > 921 100617 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100618 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b > nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe > 921 100619 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100620 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100621 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100622 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100623 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100624 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100625 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100626 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100627 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100628 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100629 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100630 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100631 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100632 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100633 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100634 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100635 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100636 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100637 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100638 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100639 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100640 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100641 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100642 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100643 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100644 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100645 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100646 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100647 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100648 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100649 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100650 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100651 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100652 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100653 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100654 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100655 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100656 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100657 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100658 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100659 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100660 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100661 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100662 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100663 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100664 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100665 nfsd nfsd: service mi_switch+0xe1 > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > _cv_wait_sig+0x16a > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 921 100666 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b > nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8 > nfsrvd_dorpc+0xc76 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe >=20 > Regards, >=20 > Lo=C3=AFc Blot, > UNIX Systems, Network and Security Engineer > http://www.unix-experience.fr >=20 > 15 d=C3=A9cembre 2014 15:18 "Rick Macklem" a > =C3=A9crit: >=20 > Loic Blot wrote: >=20 > > For more informations, here is procstat -kk on nfsd, if you > > need > > more > > hot datas, tell me. > >=20 > > Regards, PID TID COMM TDNAME KSTACK > > 918 100529 nfsd nfsd: master mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 svc_run+0x1de > > nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c > > amd64_syscall+0x351 >=20 > Well, most of the threads are stuck like this one, waiting for > a > vnode > lock in ZFS. All of them appear to be in zfs_fhtovp(). > I`m not a ZFS guy, so I can`t help much. I`ll try changing the > subject line > to include ZFS vnode lock, so maybe the ZFS guys will take a > look. >=20 > The only thing I`ve seen suggested is trying: > sysctl vfs.lookup_shared=3D0 > to disable shared vop_lookup()s. Apparently zfs_lookup() > doesn`t > obey the vnode locking rules for lookup and rename, according > to > the posting I saw. >=20 > I`ve added a couple of comments about the other threads below, > but > they are all either waiting for an RPC request or waiting for > the > threads stuck on the ZFS vnode lock to complete. >=20 > rick >=20 > > 918 100564 nfsd nfsd: service mi_switch+0xe1 > > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > > _cv_wait_sig+0x16a > > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > > fork_trampoline+0xe >=20 > Fyi, this thread is just waiting for an RPC to arrive. (Normal) >=20 > > 918 100565 nfsd nfsd: service mi_switch+0xe1 > > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > > _cv_wait_sig+0x16a > > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > > fork_trampoline+0xe > > 918 100566 nfsd nfsd: service mi_switch+0xe1 > > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > > _cv_wait_sig+0x16a > > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > > fork_trampoline+0xe > > 918 100567 nfsd nfsd: service mi_switch+0xe1 > > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > > _cv_wait_sig+0x16a > > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > > fork_trampoline+0xe > > 918 100568 nfsd nfsd: service mi_switch+0xe1 > > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > > _cv_wait_sig+0x16a > > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > > fork_trampoline+0xe > > 918 100569 nfsd nfsd: service mi_switch+0xe1 > > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > > _cv_wait_sig+0x16a > > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > > fork_trampoline+0xe > > 918 100570 nfsd nfsd: service mi_switch+0xe1 > > sleepq_catch_signals+0xab sleepq_wait_sig+0xf > > _cv_wait_sig+0x16a > > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a > > fork_trampoline+0xe > > 918 100571 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b > > nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe > > 918 100572 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b > > nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8 > > nfsrvd_dorpc+0xc76 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe >=20 > This one (and a few others) are waiting for the nfsv4_lock. > This > happens > because other threads are stuck with RPCs in progress. (ie. The > ones > waiting on the vnode lock in zfs_fhtovp().) > For these, the RPC needs to lock out other threads to do the > operation, > so it waits for the nfsv4_lock() which can exclusively lock the > NFSv4 > data structures once all other nfsd threads complete their RPCs > in > progress. >=20 > > 918 100573 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b > > nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe >=20 > Same as above. >=20 > > 918 100574 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100575 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100576 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100577 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100578 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100579 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100580 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100581 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100582 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100583 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100584 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100585 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100586 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100587 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100588 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100589 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100590 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100591 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100592 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100593 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100594 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100595 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100596 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100597 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100598 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100599 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100600 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100601 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100602 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100603 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100604 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100605 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100606 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe > > 918 100607 nfsd nfsd: service mi_switch+0xe1 > > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > > zfs_fhtovp+0x38d > > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > > nfssvc_program+0x554 svc_run_internal+0xc77 > > svc_thread_start+0xb > > fork_exit+0x9a fork_trampoline+0xe >=20 > Lots more waiting for the ZFS vnode lock in zfs_fhtovp(). >=20 >=20 > 918 100608 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b > nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1 > nfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe > 918 100609 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100610 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad > nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554 > svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a > fork_trampoline+0xe > 918 100611 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b > nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe > 918 100612 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b > nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe > 918 100613 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b > nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe > 918 100614 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b > nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe > 918 100615 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b > nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe > 918 100616 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b > nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe > 918 100617 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b > nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe > 918 100618 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b > nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe > 918 100619 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100620 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100621 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b > nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe > 918 100622 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100623 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b > nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe > 918 100624 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100625 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100626 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100627 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100628 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100629 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100630 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100631 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100632 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100633 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100634 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100635 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100636 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100637 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100638 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100639 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100640 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100641 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100642 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100643 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100644 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100645 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100646 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100647 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100648 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100649 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100650 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100651 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100652 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100653 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100654 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100655 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100656 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100657 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe > 918 100658 nfsd nfsd: service mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 > zfs_fhtovp+0x38d > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 > nfssvc_program+0x554 svc_run_internal+0xc77 > svc_thread_start+0xb > fork_exit+0x9a fork_trampoline+0xe >=20 > Lo=C3=AFc Blot, > UNIX Systems, Network and Security Engineer > http://www.unix-experience.fr >=20 > 15 d=C3=A9cembre 2014 13:29 "Lo=C3=AFc Blot" > > a > =C3=A9crit: >=20 > Hmmm... > now i'm experiencing a deadlock. >=20 > 0 918 915 0 21 0 12352 3372 zfs D - 1:48.64 nfsd: server > (nfsd) >=20 > the only issue was to reboot the server, but after rebooting > deadlock arrives a second time when i > start my jails over NFS. >=20 > Regards, >=20 > Lo=C3=AFc Blot, > UNIX Systems, Network and Security Engineer > http://www.unix-experience.fr >=20 > 15 d=C3=A9cembre 2014 10:07 "Lo=C3=AFc Blot" > > a > =C3=A9crit: >=20 > Hi Rick, > after talking with my N+1, NFSv4 is required on our > infrastructure. > I tried to upgrade NFSv4+ZFS > server from 9.3 to 10.1, i hope this will resolve some > issues... >=20 > Regards, >=20 > Lo=C3=AFc Blot, > UNIX Systems, Network and Security Engineer > http://www.unix-experience.fr >=20 > 10 d=C3=A9cembre 2014 15:36 "Lo=C3=AFc Blot" > > a > =C3=A9crit: >=20 > Hi Rick, > thanks for your suggestion. > For my locking bug, rpc.lockd is stucked in rpcrecv state on > the > server. kill -9 doesn't affect the > process, it's blocked.... (State: Ds) >=20 > for the performances >=20 > NFSv3: 60Mbps > NFSv4: 45Mbps > Regards, >=20 > Lo=C3=AFc Blot, > UNIX Systems, Network and Security Engineer > http://www.unix-experience.fr >=20 > 10 d=C3=A9cembre 2014 13:56 "Rick Macklem" > a > =C3=A9crit: >=20 >=20 > Loic Blot wrote: >=20 > > Hi Rick, > > I'm trying NFSv3. > > Some jails are starting very well but now i have an issue > > with > > lockd > > after some minutes: > >=20 > > nfs server 10.10.X.8:/jails: lockd not responding > > nfs server 10.10.X.8:/jails lockd is alive again > >=20 > > I look at mbuf, but i seems there is no problem. >=20 > Well, if you need locks to be visible across multiple > clients, > then > I'm afraid you are stuck with using NFSv4 and the > performance > you > get > from it. (There is no way to do file handle affinity for > NFSv4 > because > the read and write ops are buried in the compound RPC and > not > easily > recognized.) >=20 > If the locks don't need to be visible across multiple > clients, > I'd > suggest trying the "nolockd" option with nfsv3. >=20 > > Here is my rc.conf on server: > >=20 > > nfs_server_enable=3D"YES" > > nfsv4_server_enable=3D"YES" > > nfsuserd_enable=3D"YES" > > nfsd_server_flags=3D"-u -t -n 256" > > mountd_enable=3D"YES" > > mountd_flags=3D"-r" > > nfsuserd_flags=3D"-usertimeout 0 -force 20" > > rpcbind_enable=3D"YES" > > rpc_lockd_enable=3D"YES" > > rpc_statd_enable=3D"YES" > >=20 > > Here is the client: > >=20 > > nfsuserd_enable=3D"YES" > > nfsuserd_flags=3D"-usertimeout 0 -force 20" > > nfscbd_enable=3D"YES" > > rpc_lockd_enable=3D"YES" > > rpc_statd_enable=3D"YES" > >=20 > > Have you got an idea ? > >=20 > > Regards, > >=20 > > Lo=C3=AFc Blot, > > UNIX Systems, Network and Security Engineer > > http://www.unix-experience.fr > >=20 > > 9 d=C3=A9cembre 2014 04:31 "Rick Macklem" > > a > > =C3=A9crit: > >> Loic Blot wrote: > >>=20 > >>> Hi rick, > >>>=20 > >>> I waited 3 hours (no lag at jail launch) and now I do: > >>> sysrc > >>> memcached_flags=3D"-v -m 512" > >>> Command was very very slow... > >>>=20 > >>> Here is a dd over NFS: > >>>=20 > >>> 601062912 bytes transferred in 21.060679 secs (28539579 > >>> bytes/sec) > >>=20 > >> Can you try the same read using an NFSv3 mount? > >> (If it runs much faster, you have probably been bitten by > >> the > >> ZFS > >> "sequential vs random" read heuristic which I've been told > >> things > >> NFS is doing "random" reads without file handle affinity. > >> File > >> handle affinity is very hard to do for NFSv4, so it isn't > >> done.) >=20 > I was actually suggesting that you try the "dd" over nfsv3 > to > see > how > the performance compared with nfsv4. If you do that, please > post > the > comparable results. >=20 > Someday I would like to try and get ZFS's sequential vs > random > read > heuristic modified and any info on what difference in > performance > that > might make for NFS would be useful. >=20 > rick >=20 >=20 >=20 >=20 > rick >=20 >=20 > This is quite slow... >=20 > You can found some nfsstat below (command isn't finished > yet) >=20 > nfsstat -c -w 1 >=20 > GtAttr Lookup Rdlink Read Write Rename Access Rddir > 0 0 0 0 0 0 0 0 > 4 0 0 0 0 0 16 0 > 2 0 0 0 0 0 17 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 4 0 0 0 0 4 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 4 0 0 0 0 0 3 0 > 0 0 0 0 0 0 3 0 > 37 10 0 8 0 0 14 1 > 18 16 0 4 1 2 4 0 > 78 91 0 82 6 12 30 0 > 19 18 0 2 2 4 2 0 > 0 0 0 0 2 0 0 0 > 0 0 0 0 0 0 0 0 > GtAttr Lookup Rdlink Read Write Rename Access Rddir > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 1 0 0 0 0 1 0 > 4 6 0 0 6 0 3 0 > 2 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 1 0 0 0 0 0 0 0 > 0 0 0 0 1 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 6 108 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > GtAttr Lookup Rdlink Read Write Rename Access Rddir > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 98 54 0 86 11 0 25 0 > 36 24 0 39 25 0 10 1 > 67 8 0 63 63 0 41 0 > 34 0 0 35 34 0 0 0 > 75 0 0 75 77 0 0 0 > 34 0 0 35 35 0 0 0 > 75 0 0 74 76 0 0 0 > 33 0 0 34 33 0 0 0 > 0 0 0 0 5 0 0 0 > 0 0 0 0 0 0 6 0 > 11 0 0 0 0 0 11 0 > 0 0 0 0 0 0 0 0 > 0 17 0 0 0 0 1 0 > GtAttr Lookup Rdlink Read Write Rename Access Rddir > 4 5 0 0 0 0 12 0 > 2 0 0 0 0 0 26 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 4 0 0 0 0 4 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 4 0 0 0 0 0 2 0 > 2 0 0 0 0 0 24 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > GtAttr Lookup Rdlink Read Write Rename Access Rddir > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 4 0 0 0 0 0 7 0 > 2 1 0 0 0 0 1 0 > 0 0 0 0 2 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 6 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 4 6 0 0 0 0 3 0 > 0 0 0 0 0 0 0 0 > 2 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > GtAttr Lookup Rdlink Read Write Rename Access Rddir > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 4 71 0 0 0 0 0 0 > 0 1 0 0 0 0 0 0 > 2 36 0 0 0 0 1 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 1 0 0 0 0 0 1 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 79 6 0 79 79 0 2 0 > 25 0 0 25 26 0 6 0 > 43 18 0 39 46 0 23 0 > 36 0 0 36 36 0 31 0 > 68 1 0 66 68 0 0 0 > GtAttr Lookup Rdlink Read Write Rename Access Rddir > 36 0 0 36 36 0 0 0 > 48 0 0 48 49 0 0 0 > 20 0 0 20 20 0 0 0 > 0 0 0 0 0 0 0 0 > 3 14 0 1 0 0 11 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 4 0 0 0 0 4 0 > 0 0 0 0 0 0 0 0 > 4 22 0 0 0 0 16 0 > 2 0 0 0 0 0 23 0 >=20 > Regards, >=20 > Lo=C3=AFc Blot, > UNIX Systems, Network and Security Engineer > http://www.unix-experience.fr >=20 > 8 d=C3=A9cembre 2014 09:36 "Lo=C3=AFc Blot" > a > =C3=A9crit: > > Hi Rick, > > I stopped the jails this week-end and started it this > > morning, > > i'll > > give you some stats this week. > >=20 > > Here is my nfsstat -m output (with your rsize/wsize > > tweaks) >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 > nfsv4,tcp,resvport,hard,cto,sec=3Dsys,acdirmin=3D3,acdirmax=3D60,acregmin= =3D5,acregmax=3D60,nametimeo=3D60,negna >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 > etimeo=3D60,rsize=3D32768,wsize=3D32768,readdirsize=3D32768,readahead=3D1= ,wcommitsize=3D773136,timeout=3D120,retra >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 > s=3D2147483647 >=20 > On server side my disks are on a raid controller which show a > 512b > volume and write performances > are very honest (dd if=3D/dev/zero of=3D/jails/test.dd bs=3D4096 > count=3D100000000 =3D> 450MBps) >=20 > Regards, >=20 > Lo=C3=AFc Blot, > UNIX Systems, Network and Security Engineer > http://www.unix-experience.fr >=20 > 5 d=C3=A9cembre 2014 15:14 "Rick Macklem" a > =C3=A9crit: >=20 >=20 > Loic Blot wrote: >=20 >=20 > Hi, > i'm trying to create a virtualisation environment based on > jails. > Those jails are stored under a big ZFS pool on a FreeBSD > 9.3 > which > export a NFSv4 volume. This NFSv4 volume was mounted on a > big > hypervisor (2 Xeon E5v3 + 128GB memory and 8 ports (but > only 1 > was > used at this time). >=20 > The problem is simple, my hypervisors runs 6 jails (used 1% > cpu > and > 10GB RAM approximatively and less than 1MB bandwidth) and > works > fine at start but the system slows down and after 2-3 days > become > unusable. When i look at top command i see 80-100% on > system > and > commands are very very slow. Many process are tagged with > nfs_cl*. >=20 >=20 > To be honest, I would expect the slowness to be because of > slow > response > from the NFSv4 server, but if you do: > # ps axHl > on a client when it is slow and post that, it would give us > some > more > information on where the client side processes are sitting. > If you also do something like: > # nfsstat -c -w 1 > and let it run for a while, that should show you how many > RPCs > are > being done and which ones. >=20 > # nfsstat -m > will show you what your mount is actually using. > The only mount option I can suggest trying is > "rsize=3D32768,wsize=3D32768", > since some network environments have difficulties with 64K. >=20 > There are a few things you can try on the NFSv4 server side, > if > it > appears > that the clients are generating a large RPC load. > - disabling the DRC cache for TCP by setting > vfs.nfsd.cachetcp=3D0 > - If the server is seeing a large write RPC load, then > "sync=3Ddisabled" > might help, although it does run a risk of data loss when > the > server > crashes. > Then there are a couple of other ZFS related things (I'm not > a > ZFS > guy, > but these have shown up on the mailing lists). > - make sure your volumes are 4K aligned and ashift=3D12 (in > case a > drive > that uses 4K sectors is pretending to be 512byte sectored) > - never run over 70-80% full if write performance is an > issue > - use a zil on an SSD with good write performance >=20 > The only NFSv4 thing I can tell you is that it is known that > ZFS's > algorithm for determining sequential vs random I/O fails for > NFSv4 > during writing and this can be a performance hit. The only > workaround > is to use NFSv3 mounts, since file handle affinity > apparently > fixes > the problem and this is only done for NFSv3. >=20 > rick >=20 >=20 > I saw that there are TSO issues with igb then i'm trying to > disable > it with sysctl but the situation wasn't solved. >=20 > Someone has got ideas ? I can give you more informations if > you > need. >=20 > Thanks in advance. > Regards, >=20 > Lo=C3=AFc Blot, > UNIX Systems, Network and Security Engineer > http://www.unix-experience.fr > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to > "freebsd-fs-unsubscribe@freebsd.org" >=20 >=20 >=20 >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to > "freebsd-fs-unsubscribe@freebsd.org" >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to > "freebsd-fs-unsubscribe@freebsd.org" >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to > "freebsd-fs-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to > "freebsd-fs-unsubscribe@freebsd.org" >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >=20 >=20 >=20 >=20