Site Navigation

Date:      Mon, 22 Dec 2014 18:20:03 -0500 (EST)
From:      Rick Macklem <rmacklem@uoguelph.ca>
To:        =?utf-8?B?TG/Dr2M=?= Blot <loic.blot@unix-experience.fr>
Cc:        freebsd-fs@freebsd.org
Subject:   Re: ZFS vnode lock deadlock in zfs_fhtovp was: High Kernel Load with nfsv4
Message-ID:  <1479765128.1118136.1419290403230.JavaMail.root@uoguelph.ca>
In-Reply-To: <811d455b0bcaeb43711e8108c96d4f2b@mail.unix-experience.fr>

---

next in thread | previous in thread | raw e-mail | index | archive | help

---

Loic Blot wrote:
> Hi,
>=20
> To clarify because of our exchanges. Here are the current sysctl
> options for server:
>=20
> vfs.nfsd.enable_nobodycheck=3D0
> vfs.nfsd.enable_nogroupcheck=3D0
>=20
> vfs.nfsd.maxthreads=3D200
> vfs.nfsd.tcphighwater=3D10000
> vfs.nfsd.tcpcachetimeo=3D300
> vfs.nfsd.server_min_nfsvers=3D4
>=20
> kern.maxvnodes=3D10000000
> kern.ipc.maxsockbuf=3D4194304
> net.inet.tcp.sendbuf_max=3D4194304
> net.inet.tcp.recvbuf_max=3D4194304
>=20
> vfs.lookup_shared=3D0
>=20
> Regards,
>=20
> Lo=C3=AFc Blot,
> UNIX Systems, Network and Security Engineer
> http://www.unix-experience.fr
>=20
> 22 d=C3=A9cembre 2014 09:42 "Lo=C3=AFc Blot" <loic.blot@unix-experience.f=
r> a
> =C3=A9crit:
>=20
> Hi Rick,
> my 5 jails runs this weekend and now i have some stats on this
> monday.
>=20
> Hopefully deadlock was fixed, yeah, but everything isn't good :(
>=20
> On NFSv4 server (FreeBSD 10.1) system uses 35% CPU
>=20
> As i can see this is because of nfsd:
>=20
> 918 root            96  20    0 12352K  3372K rpcsvc  6  51.4H
> 273.68% nfsd: server (nfsd)
>=20
> If i look at dmesg i see:
> nfsd server cache flooded, try increasing vfs.nfsd.tcphighwater
>=20
Well, you have a couple of choices:
1 - Use NFSv4.1 (add "minorversion=3D1" to your mount options).
    (NFSv4.1 avoids use of the DRC and instead uses something
     called sessions. See below.)
OR

> vfs.nfsd.tcphighwater was set to 10000, i increase it to 15000
>=20
2 - Bump vfs.nfsd.tcphighwater way up, until you no longer see
    "nfs server cache flooded" messages. (I think Garrett Wollman uses
    100000. (You may still see quite a bit of CPU overheads.)

OR

3 - Set vfs.nfsd.cachetcp=3D0 (which disables the DRC and gets rid
    of the CPU overheads). However, there is a risk of data corruption
    if you have a client->server network partitioning of a moderate
    duration, because a non-idempotent RPC may get redone, becasue
    the client times out waiting for a reply. If a non-idempotent
    RPC gets done twice on the server, data corruption can happen.
    (The DRC provides improved correctness, but does add overhead.)

If #1 works for you, it is the preferred solution, since Sessions
in NFSv4.1 solves the correctness problem in a good, space bound
way. A session basically has N (usually 32 or 64) slots and only
allows one outstanding RPC/slot. As such, it can cache the previous
reply for each slot (32 or 64 of them) and guarantee "exactly once"
RPC semantics.

rick

> Here is 'nfsstat -s' output:
>=20
> Server Info:
> Getattr   Setattr    Lookup  Readlink      Read     Write    Create
>    Remove
> 12600652      1812   2501097       156   1386423   1983729       123
>    162067
> Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus
>    Access
> 36762         9         0         0         0      3147         0
>    623524
> Mknod    Fsstat    Fsinfo  PathConf    Commit
> 0         0         0         0    328117
> Server Ret-Failed
> 0
> Server Faults
> 0
> Server Cache Stats:
> Inprog      Idem  Non-idem    Misses
> 0         0         0  12635512
> Server Write Gathering:
> WriteOps  WriteRPC   Opsaved
> 1983729   1983729         0
>=20
> And here is 'procstat -kk' for nfsd (server)
>=20
> 918 100528 nfsd             nfsd: master     mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10
> _cv_timedwait_sig_sbt+0x18b svc_run_internal+0x4a1 svc_run+0x1de
> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c
> amd64_syscall+0x351 Xfast_syscall+0xfb
> 918 100568 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100569 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100570 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100571 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100572 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100573 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100574 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100575 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100576 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100577 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100578 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100579 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100580 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100581 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100582 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100583 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100584 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100585 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100586 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100587 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100588 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100589 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100590 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100591 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100592 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100593 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100594 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100595 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100596 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100597 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100598 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100599 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100600 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100601 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100602 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100603 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100604 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100605 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100606 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100607 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100608 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100609 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100610 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100611 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100612 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100613 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100614 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100615 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100616 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100617 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100618 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100619 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100620 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100621 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100622 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100623 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100624 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100625 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100626 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100627 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100628 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100629 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100630 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100631 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100632 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100633 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100634 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100635 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100636 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100637 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100638 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100639 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100640 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100641 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100642 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100643 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100644 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100645 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100646 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100647 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100648 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100649 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100650 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100651 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100652 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100653 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100654 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100655 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100656 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100657 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100658 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100659 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100660 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100661 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100662 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> ---
>=20
> Now if we look at client (FreeBSD 9.3)
>=20
> We see system was very busy and do many and many interrupts
>=20
> CPU:  0.0% user,  0.0% nice, 37.8% system, 51.2% interrupt, 11.0%
> idle
>=20
> A look at process list shows that there are many sendmail process in
> state nfstry
>=20
> nfstry 18  32:27   0.88% sendmail: Queue runner@00:30:00 for
> /var/spool/clientm
>=20
> Here is 'nfsstat -c' output:
>=20
> Client Info:
> Rpc Counts:
> Getattr   Setattr    Lookup  Readlink      Read     Write    Create
>    Remove
> 1051347      1724   2494481       118    903902   1901285    162676
>    161899
> Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus
>    Access
> 36744         2         0       114        40      3131         0
>    544136
> Mknod    Fsstat    Fsinfo  PathConf    Commit
> 9         0         0         0    245821
> Rpc Info:
> TimedOut   Invalid X Replies   Retries  Requests
> 0         0         0         0   8356557
> Cache Info:
> Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits
>    Misses
> 108754455    491475  54229224   2437229  46814561    821723   5132123
>   1871871
> BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses Accs Hits
>    Misses
> 144035       118     53736      2753     27813         1  57238839
>    544205
>=20
> If you need more things, tell me, i let the PoC in this state.
>=20
> Thanks
>=20
> Regards,
>=20
> Lo=C3=AFc Blot,
> UNIX Systems, Network and Security Engineer
> http://www.unix-experience.fr
>=20
> 21 d=C3=A9cembre 2014 01:33 "Rick Macklem" <rmacklem@uoguelph.ca> a =C3=
=A9crit:
>=20
>=20
> Loic Blot wrote:
>=20
> > Hi Rick,
> > ok, i don't need locallocks, i haven't understand option was for
> > that
> > usage, i removed it.
> > I do more tests on monday.
> > Thanks for the deadlock fix, for other people :)
>=20
> Good. Please let us know if running with vfs.nfsd.enable_locallocks=3D0
> gets rid of the deadlocks? (I think it fixes the one you saw.)
>=20
> On the performance side, you might also want to try different values
> of
> readahead, if the Linux client has such a mount option. (With the
> NFSv4-ZFS sequential vs random I/O heuristic, I have no idea what the
> optimal readahead value would be.)
>=20
> Good luck with it and please let us know how it goes, rick
> ps: I now have a patch to fix the deadlock when
> vfs.nfsd.enable_locallocks=3D1
> is set. I'll post it for anyone who is interested after I put it
> through some testing.
>=20
>=20
> --
> Best regards,
> Lo=C3=AFc BLOT,
> UNIX systems, security and network engineer
> http://www.unix-experience.fr
>=20
> Le jeudi 18 d=C3=A9cembre 2014 =C3=A0 19:46 -0500, Rick Macklem a =C3=A9c=
rit :
>=20
> Loic Blot wrote:
> > Hi rick,
> > i tried to start a LXC container on Debian Squeeze from my
> > freebsd
> > ZFS+NFSv4 server and i also have a deadlock on nfsd
> > (vfs.lookup_shared=3D0). Deadlock procs each time i launch a
> > squeeze
> > container, it seems (3 tries, 3 fails).
>=20
> Well, I`ll take a look at this `procstat -kk`, but the only thing
> I`ve seen posted w.r.t. avoiding deadlocks in ZFS is to not use
> nullfs. (I have no idea if you are using any nullfs mounts, but
> if so, try getting rid of them.)
>=20
> Here`s a high level post about the ZFS and vnode locking problem,
> but there is no patch available, as far as I know.
>=20
> http://docs.FreeBSD.org/cgi/mid.cgi?54739F41.8030407
>=20
> rick
>=20
>=20
> 921  -  D      0:00.02 nfsd: server (nfsd)
>=20
> Here is the procstat -kk
>=20
> PID    TID COMM             TDNAME           KSTACK
> 921 100538 nfsd             nfsd: master     mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad
> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554
> svc_run_internal+0xc77 svc_run+0x1de nfsrvd_nfsd+0x1ca
> nfssvc_nfsd+0x107 sys_nfssvc+0x9c
> 921 100572 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100573 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100574 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100575 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100576 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100577 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100578 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100579 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100580 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100581 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100582 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100583 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100584 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100585 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100586 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100587 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100588 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100589 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100590 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100591 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100592 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100593 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100594 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100595 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100596 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100597 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100598 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100599 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100600 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100601 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100602 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100603 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100604 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100605 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100606 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100607 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100608 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100609 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100610 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100611 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100612 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100613 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100614 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100615 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100616 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1
> nfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> 921 100617 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100618 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> 921 100619 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100620 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100621 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100622 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100623 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100624 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100625 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100626 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100627 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100628 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100629 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100630 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100631 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100632 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100633 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100634 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100635 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100636 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100637 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100638 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100639 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100640 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100641 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100642 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100643 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100644 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100645 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100646 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100647 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100648 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100649 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100650 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100651 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100652 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100653 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100654 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100655 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100656 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100657 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100658 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100659 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100660 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100661 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100662 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100663 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100664 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100665 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100666 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8
> nfsrvd_dorpc+0xc76
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
>=20
> Regards,
>=20
> Lo=C3=AFc Blot,
> UNIX Systems, Network and Security Engineer
> http://www.unix-experience.fr
>=20
> 15 d=C3=A9cembre 2014 15:18 "Rick Macklem" <rmacklem@uoguelph.ca> a
> =C3=A9crit:
>=20
> Loic Blot wrote:
>=20
> > For more informations, here is procstat -kk on nfsd, if you
> > need
> > more
> > hot datas, tell me.
> >=20
> > Regards,  PID    TID COMM             TDNAME           KSTACK
> > 918 100529 nfsd             nfsd: master     mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77 svc_run+0x1de
> > nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c
> > amd64_syscall+0x351
>=20
> Well, most of the threads are stuck like this one, waiting for
> a
> vnode
> lock in ZFS. All of them appear to be in zfs_fhtovp().
> I`m not a ZFS guy, so I can`t help much. I`ll try changing the
> subject line
> to include ZFS vnode lock, so maybe the ZFS guys will take a
> look.
>=20
> The only thing I`ve seen suggested is trying:
> sysctl vfs.lookup_shared=3D0
> to disable shared vop_lookup()s. Apparently zfs_lookup()
> doesn`t
> obey the vnode locking rules for lookup and rename, according
> to
> the posting I saw.
>=20
> I`ve added a couple of comments about the other threads below,
> but
> they are all either waiting for an RPC request or waiting for
> the
> threads stuck on the ZFS vnode lock to complete.
>=20
> rick
>=20
> > 918 100564 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> > _cv_wait_sig+0x16a
> > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> > fork_trampoline+0xe
>=20
> Fyi, this thread is just waiting for an RPC to arrive. (Normal)
>=20
> > 918 100565 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> > _cv_wait_sig+0x16a
> > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> > fork_trampoline+0xe
> > 918 100566 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> > _cv_wait_sig+0x16a
> > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> > fork_trampoline+0xe
> > 918 100567 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> > _cv_wait_sig+0x16a
> > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> > fork_trampoline+0xe
> > 918 100568 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> > _cv_wait_sig+0x16a
> > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> > fork_trampoline+0xe
> > 918 100569 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> > _cv_wait_sig+0x16a
> > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> > fork_trampoline+0xe
> > 918 100570 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> > _cv_wait_sig+0x16a
> > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> > fork_trampoline+0xe
> > 918 100571 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> > nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> > 918 100572 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> > nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8
> > nfsrvd_dorpc+0xc76
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
>=20
> This one (and a few others) are waiting for the nfsv4_lock.
> This
> happens
> because other threads are stuck with RPCs in progress. (ie. The
> ones
> waiting on the vnode lock in zfs_fhtovp().)
> For these, the RPC needs to lock out other threads to do the
> operation,
> so it waits for the nfsv4_lock() which can exclusively lock the
> NFSv4
> data structures once all other nfsd threads complete their RPCs
> in
> progress.
>=20
> > 918 100573 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> > nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>=20
> Same as above.
>=20
> > 918 100574 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100575 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100576 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100577 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100578 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100579 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100580 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100581 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100582 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100583 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100584 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100585 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100586 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100587 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100588 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100589 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100590 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100591 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100592 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100593 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100594 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100595 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100596 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100597 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100598 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100599 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100600 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100601 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100602 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100603 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100604 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100605 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100606 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
> > 918 100607 nfsd             nfsd: service    mi_switch+0xe1
> > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> > vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> > zfs_fhtovp+0x38d
> > nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> > nfssvc_program+0x554 svc_run_internal+0xc77
> > svc_thread_start+0xb
> > fork_exit+0x9a fork_trampoline+0xe
>=20
> Lots more waiting for the ZFS vnode lock in zfs_fhtovp().
>=20
>=20
> 918 100608 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1
> nfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> 918 100609 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100610 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad
> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554
> svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100611 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> 918 100612 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> 918 100613 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> 918 100614 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> 918 100615 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> 918 100616 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> 918 100617 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> 918 100618 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> 918 100619 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100620 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100621 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> 918 100622 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100623 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> 918 100624 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100625 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100626 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100627 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100628 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100629 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100630 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100631 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100632 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100633 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100634 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100635 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100636 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100637 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100638 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100639 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100640 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100641 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100642 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100643 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100644 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100645 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100646 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100647 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100648 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100649 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100650 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100651 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100652 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100653 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100654 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100655 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100656 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100657 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100658 nfsd             nfsd: service    mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
>=20
> Lo=C3=AFc Blot,
> UNIX Systems, Network and Security Engineer
> http://www.unix-experience.fr
>=20
> 15 d=C3=A9cembre 2014 13:29 "Lo=C3=AFc Blot"
> <loic.blot@unix-experience.fr>
> a
> =C3=A9crit:
>=20
> Hmmm...
> now i'm experiencing a deadlock.
>=20
> 0 918 915 0 21 0 12352 3372 zfs D - 1:48.64 nfsd: server
> (nfsd)
>=20
> the only issue was to reboot the server, but after rebooting
> deadlock arrives a second time when i
> start my jails over NFS.
>=20
> Regards,
>=20
> Lo=C3=AFc Blot,
> UNIX Systems, Network and Security Engineer
> http://www.unix-experience.fr
>=20
> 15 d=C3=A9cembre 2014 10:07 "Lo=C3=AFc Blot"
> <loic.blot@unix-experience.fr>
> a
> =C3=A9crit:
>=20
> Hi Rick,
> after talking with my N+1, NFSv4 is required on our
> infrastructure.
> I tried to upgrade NFSv4+ZFS
> server from 9.3 to 10.1, i hope this will resolve some
> issues...
>=20
> Regards,
>=20
> Lo=C3=AFc Blot,
> UNIX Systems, Network and Security Engineer
> http://www.unix-experience.fr
>=20
> 10 d=C3=A9cembre 2014 15:36 "Lo=C3=AFc Blot"
> <loic.blot@unix-experience.fr>
> a
> =C3=A9crit:
>=20
> Hi Rick,
> thanks for your suggestion.
> For my locking bug, rpc.lockd is stucked in rpcrecv state on
> the
> server. kill -9 doesn't affect the
> process, it's blocked.... (State: Ds)
>=20
> for the performances
>=20
> NFSv3: 60Mbps
> NFSv4: 45Mbps
> Regards,
>=20
> Lo=C3=AFc Blot,
> UNIX Systems, Network and Security Engineer
> http://www.unix-experience.fr
>=20
> 10 d=C3=A9cembre 2014 13:56 "Rick Macklem" <rmacklem@uoguelph.ca>
> a
> =C3=A9crit:
>=20
>=20
> Loic Blot wrote:
>=20
> > Hi Rick,
> > I'm trying NFSv3.
> > Some jails are starting very well but now i have an issue
> > with
> > lockd
> > after some minutes:
> >=20
> > nfs server 10.10.X.8:/jails: lockd not responding
> > nfs server 10.10.X.8:/jails lockd is alive again
> >=20
> > I look at mbuf, but i seems there is no problem.
>=20
> Well, if you need locks to be visible across multiple
> clients,
> then
> I'm afraid you are stuck with using NFSv4 and the
> performance
> you
> get
> from it. (There is no way to do file handle affinity for
> NFSv4
> because
> the read and write ops are buried in the compound RPC and
> not
> easily
> recognized.)
>=20
> If the locks don't need to be visible across multiple
> clients,
> I'd
> suggest trying the "nolockd" option with nfsv3.
>=20
> > Here is my rc.conf on server:
> >=20
> > nfs_server_enable=3D"YES"
> > nfsv4_server_enable=3D"YES"
> > nfsuserd_enable=3D"YES"
> > nfsd_server_flags=3D"-u -t -n 256"
> > mountd_enable=3D"YES"
> > mountd_flags=3D"-r"
> > nfsuserd_flags=3D"-usertimeout 0 -force 20"
> > rpcbind_enable=3D"YES"
> > rpc_lockd_enable=3D"YES"
> > rpc_statd_enable=3D"YES"
> >=20
> > Here is the client:
> >=20
> > nfsuserd_enable=3D"YES"
> > nfsuserd_flags=3D"-usertimeout 0 -force 20"
> > nfscbd_enable=3D"YES"
> > rpc_lockd_enable=3D"YES"
> > rpc_statd_enable=3D"YES"
> >=20
> > Have you got an idea ?
> >=20
> > Regards,
> >=20
> > Lo=C3=AFc Blot,
> > UNIX Systems, Network and Security Engineer
> > http://www.unix-experience.fr
> >=20
> > 9 d=C3=A9cembre 2014 04:31 "Rick Macklem" <rmacklem@uoguelph.ca>
> > a
> > =C3=A9crit:
> >> Loic Blot wrote:
> >>=20
> >>> Hi rick,
> >>>=20
> >>> I waited 3 hours (no lag at jail launch) and now I do:
> >>> sysrc
> >>> memcached_flags=3D"-v -m 512"
> >>> Command was very very slow...
> >>>=20
> >>> Here is a dd over NFS:
> >>>=20
> >>> 601062912 bytes transferred in 21.060679 secs (28539579
> >>> bytes/sec)
> >>=20
> >> Can you try the same read using an NFSv3 mount?
> >> (If it runs much faster, you have probably been bitten by
> >> the
> >> ZFS
> >> "sequential vs random" read heuristic which I've been told
> >> things
> >> NFS is doing "random" reads without file handle affinity.
> >> File
> >> handle affinity is very hard to do for NFSv4, so it isn't
> >> done.)
>=20
> I was actually suggesting that you try the "dd" over nfsv3
> to
> see
> how
> the performance compared with nfsv4. If you do that, please
> post
> the
> comparable results.
>=20
> Someday I would like to try and get ZFS's sequential vs
> random
> read
> heuristic modified and any info on what difference in
> performance
> that
> might make for NFS would be useful.
>=20
> rick
>=20
>=20
>=20
>=20
> rick
>=20
>=20
> This is quite slow...
>=20
> You can found some nfsstat below (command isn't finished
> yet)
>=20
> nfsstat -c -w 1
>=20
> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> 0 0 0 0 0 0 0 0
> 4 0 0 0 0 0 16 0
> 2 0 0 0 0 0 17 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 4 0 0 0 0 4 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 4 0 0 0 0 0 3 0
> 0 0 0 0 0 0 3 0
> 37 10 0 8 0 0 14 1
> 18 16 0 4 1 2 4 0
> 78 91 0 82 6 12 30 0
> 19 18 0 2 2 4 2 0
> 0 0 0 0 2 0 0 0
> 0 0 0 0 0 0 0 0
> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 1 0 0 0 0 1 0
> 4 6 0 0 6 0 3 0
> 2 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 1 0 0 0 0 0 0 0
> 0 0 0 0 1 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 6 108 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 98 54 0 86 11 0 25 0
> 36 24 0 39 25 0 10 1
> 67 8 0 63 63 0 41 0
> 34 0 0 35 34 0 0 0
> 75 0 0 75 77 0 0 0
> 34 0 0 35 35 0 0 0
> 75 0 0 74 76 0 0 0
> 33 0 0 34 33 0 0 0
> 0 0 0 0 5 0 0 0
> 0 0 0 0 0 0 6 0
> 11 0 0 0 0 0 11 0
> 0 0 0 0 0 0 0 0
> 0 17 0 0 0 0 1 0
> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> 4 5 0 0 0 0 12 0
> 2 0 0 0 0 0 26 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 4 0 0 0 0 4 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 4 0 0 0 0 0 2 0
> 2 0 0 0 0 0 24 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 4 0 0 0 0 0 7 0
> 2 1 0 0 0 0 1 0
> 0 0 0 0 2 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 6 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 4 6 0 0 0 0 3 0
> 0 0 0 0 0 0 0 0
> 2 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 4 71 0 0 0 0 0 0
> 0 1 0 0 0 0 0 0
> 2 36 0 0 0 0 1 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 1 0 0 0 0 0 1 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 79 6 0 79 79 0 2 0
> 25 0 0 25 26 0 6 0
> 43 18 0 39 46 0 23 0
> 36 0 0 36 36 0 31 0
> 68 1 0 66 68 0 0 0
> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> 36 0 0 36 36 0 0 0
> 48 0 0 48 49 0 0 0
> 20 0 0 20 20 0 0 0
> 0 0 0 0 0 0 0 0
> 3 14 0 1 0 0 11 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 4 0 0 0 0 4 0
> 0 0 0 0 0 0 0 0
> 4 22 0 0 0 0 16 0
> 2 0 0 0 0 0 23 0
>=20
> Regards,
>=20
> Lo=C3=AFc Blot,
> UNIX Systems, Network and Security Engineer
> http://www.unix-experience.fr
>=20
> 8 d=C3=A9cembre 2014 09:36 "Lo=C3=AFc Blot"
> <loic.blot@unix-experience.fr> a
> =C3=A9crit:
> > Hi Rick,
> > I stopped the jails this week-end and started it this
> > morning,
> > i'll
> > give you some stats this week.
> >=20
> > Here is my nfsstat -m output (with your rsize/wsize
> > tweaks)
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
> nfsv4,tcp,resvport,hard,cto,sec=3Dsys,acdirmin=3D3,acdirmax=3D60,acregmin=
=3D5,acregmax=3D60,nametimeo=3D60,negna
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
> etimeo=3D60,rsize=3D32768,wsize=3D32768,readdirsize=3D32768,readahead=3D1=
,wcommitsize=3D773136,timeout=3D120,retra
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
> s=3D2147483647
>=20
> On server side my disks are on a raid controller which show a
> 512b
> volume and write performances
> are very honest (dd if=3D/dev/zero of=3D/jails/test.dd bs=3D4096
> count=3D100000000 =3D> 450MBps)
>=20
> Regards,
>=20
> Lo=C3=AFc Blot,
> UNIX Systems, Network and Security Engineer
> http://www.unix-experience.fr
>=20
> 5 d=C3=A9cembre 2014 15:14 "Rick Macklem" <rmacklem@uoguelph.ca> a
> =C3=A9crit:
>=20
>=20
> Loic Blot wrote:
>=20
>=20
> Hi,
> i'm trying to create a virtualisation environment based on
> jails.
> Those jails are stored under a big ZFS pool on a FreeBSD
> 9.3
> which
> export a NFSv4 volume. This NFSv4 volume was mounted on a
> big
> hypervisor (2 Xeon E5v3 + 128GB memory and 8 ports (but
> only 1
> was
> used at this time).
>=20
> The problem is simple, my hypervisors runs 6 jails (used 1%
> cpu
> and
> 10GB RAM approximatively and less than 1MB bandwidth) and
> works
> fine at start but the system slows down and after 2-3 days
> become
> unusable. When i look at top command i see 80-100% on
> system
> and
> commands are very very slow. Many process are tagged with
> nfs_cl*.
>=20
>=20
> To be honest, I would expect the slowness to be because of
> slow
> response
> from the NFSv4 server, but if you do:
> # ps axHl
> on a client when it is slow and post that, it would give us
> some
> more
> information on where the client side processes are sitting.
> If you also do something like:
> # nfsstat -c -w 1
> and let it run for a while, that should show you how many
> RPCs
> are
> being done and which ones.
>=20
> # nfsstat -m
> will show you what your mount is actually using.
> The only mount option I can suggest trying is
> "rsize=3D32768,wsize=3D32768",
> since some network environments have difficulties with 64K.
>=20
> There are a few things you can try on the NFSv4 server side,
> if
> it
> appears
> that the clients are generating a large RPC load.
> - disabling the DRC cache for TCP by setting
> vfs.nfsd.cachetcp=3D0
> - If the server is seeing a large write RPC load, then
> "sync=3Ddisabled"
> might help, although it does run a risk of data loss when
> the
> server
> crashes.
> Then there are a couple of other ZFS related things (I'm not
> a
> ZFS
> guy,
> but these have shown up on the mailing lists).
> - make sure your volumes are 4K aligned and ashift=3D12 (in
> case a
> drive
> that uses 4K sectors is pretending to be 512byte sectored)
> - never run over 70-80% full if write performance is an
> issue
> - use a zil on an SSD with good write performance
>=20
> The only NFSv4 thing I can tell you is that it is known that
> ZFS's
> algorithm for determining sequential vs random I/O fails for
> NFSv4
> during writing and this can be a performance hit. The only
> workaround
> is to use NFSv3 mounts, since file handle affinity
> apparently
> fixes
> the problem and this is only done for NFSv3.
>=20
> rick
>=20
>=20
> I saw that there are TSO issues with igb then i'm trying to
> disable
> it with sysctl but the situation wasn't solved.
>=20
> Someone has got ideas ? I can give you more informations if
> you
> need.
>=20
> Thanks in advance.
> Regards,
>=20
> Lo=C3=AFc Blot,
> UNIX Systems, Network and Security Engineer
> http://www.unix-experience.fr
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to
> "freebsd-fs-unsubscribe@freebsd.org"
>=20
>=20
>=20
>=20
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to
> "freebsd-fs-unsubscribe@freebsd.org"
>=20
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to
> "freebsd-fs-unsubscribe@freebsd.org"
>=20
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to
> "freebsd-fs-unsubscribe@freebsd.org"
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to
> "freebsd-fs-unsubscribe@freebsd.org"
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
>=20
>=20
>=20
>=20

---

Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1479765128.1118136.1419290403230.JavaMail.root>

Legal Notices | © 1995-2024 The FreeBSD Project. All rights reserved.

Contact