From owner-freebsd-fs@FreeBSD.ORG  Tue Jan  6 23:28:59 2015
Return-Path: <owner-freebsd-fs@FreeBSD.ORG>
Delivered-To: freebsd-fs@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id 3215CB41
 for <freebsd-fs@freebsd.org>; Tue,  6 Jan 2015 23:28:59 +0000 (UTC)
Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca
 [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 6B1301028
 for <freebsd-fs@freebsd.org>; Tue,  6 Jan 2015 23:28:58 +0000 (UTC)
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: Ap4EANxurFSDaFve/2dsb2JhbADSaQICAQ
X-IronPort-AV: E=Sophos;i="5.07,711,1413259200"; d="scan'208";a="182085200"
Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca)
 ([131.104.91.222])
 by esa-jnhn.mail.uoguelph.ca with ESMTP; 06 Jan 2015 18:28:58 -0500
Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1])
 by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 5346EB4129;
 Tue,  6 Jan 2015 18:28:56 -0500 (EST)
Date: Tue, 6 Jan 2015 18:28:56 -0500 (EST)
From: Rick Macklem <rmacklem@uoguelph.ca>
To: =?utf-8?B?TG/Dr2M=?= Blot <loic.blot@unix-experience.fr>
Message-ID: <1566336890.7368425.1420586936307.JavaMail.root@uoguelph.ca>
In-Reply-To: <f8c1e0380b1c195ce3c63bc41fafb9e1@mail.unix-experience.fr>
Subject: Re: High Kernel Load with nfsv4
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Originating-IP: [172.17.95.12]
X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926)
Cc: freebsd-fs@freebsd.org
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-fs>,
 <mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
 <mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 06 Jan 2015 23:28:59 -0000

Loic Blot wrote:
> Hi Rick,
>=20
> i saw that some people has issues with igb cards with NFS
> For example:
> http://freebsd.1045724.n5.nabble.com/NFS-over-LAGG-lacp-poor-performance-=
td5906349.html
>=20
> can my problem be related ? I use igb with default queue number. Here
> are my vmstat -i outputs
>=20
I have no idea. Maybe someone familiar with this will respond?

I do think that the large # of NFSv4 Opens (which are actually a form of lo=
ck)
could be a factor. The client and server has to search those lists for a ma=
tch
for many NFSv4 operations, including all reads/writes.

On the server side, the default hash table sizes are very small. This is in
part that I did testing on 256Mbyte i386 systems, so that the values were
safe for such a machine.
I'd suggest you increase the following on the server's kernel.
In sys/fs/nfs/nfs.h:
NFSSTATEHASHSIZE - This one is in every client header, so if you have a lar=
ge#
                   of clients, you don't want to increase it too much. Howe=
ver
                   for a fairly large server handling not too many clients,=
 I'd
                   try something like 1000 instead of 10.
                   (I just tried 100 on the small i386 laptop I have handy =
and it
                    seemed ok for a small test.)
NFSLOCKHASHSIZE - This one is a single global table, so I'd bump it way up,
                   20000 maybe?
In sys/fs/nfsport.h:
NFSRV_V4STATELIMIT - The comment notes that the default of 500000 seems saf=
e for
                     a 256Mbyte i386, so I'd bump it to something like 2000=
000
                     for your case.
You will have to rebuild a kernel from sources after editing these values a=
nd
boot it on the server. Maybe these should become tunables so building a ker=
nel
isn't necessary?

I looked and there isn't much that can be done in the client. At this point=
,
the open_owners and opens are single lists for a client (a mount point on a
client machine for FreeBSD). If you post what you get for "nfsstat -e -c"
on a typical client in your setup, that would tell me if it is the open_own=
ers
(which I suspect) or opens that will be a long list. (I would have to code
 a patch to make either of these a hash table instead of a single linked
 list. I should do this. It was on my to-do list, but got forgotten.;-)

rick

> Server side:
>=20
> interrupt                          total       rate
> irq1: atkbd0                          18          0
> irq20: ehci1                     2790134          2
> irq21: ehci0                     2547642          2
> cpu0:timer                      36299188         35
> irq264: ciss0                    6352476          6
> irq265: igb0:que 0               2716692          2
> irq266: igb0:que 1              32205278         31
> irq267: igb0:que 2              38395109         37
> irq268: igb0:que 3               1413468          1
> irq269: igb0:que 4              39207930         38
> irq270: igb0:que 5               1622715          1
> irq271: igb0:que 6               1634676          1
> irq272: igb0:que 7               1190123          1
> irq273: igb0:link                      2          0
> cpu1:timer                      14074423         13
> cpu8:timer                      12204739         11
> cpu9:timer                      11384192         11
> cpu3:timer                      10461566         10
> cpu4:timer                      12785103         12
> cpu6:timer                      10739344         10
> cpu5:timer                      10978294         10
> cpu7:timer                      10599705         10
> cpu2:timer                      13998891         13
> cpu10:timer                     11602361         11
> cpu11:timer                     11568523         11
> Total                          296772592        290
>=20
> And client side:
> interrupt                          total       rate
> irq9: acpi0                            4          0
> irq22: ehci1                      950519          2
> irq23: ehci0                     1865060          4
> cpu0:timer                     248128035        546
> irq268: mfi0                      406896          0
> irq269: igb0:que 0               2510556          5
> irq270: igb0:que 1               2825336          6
> irq271: igb0:que 2               2092958          4
> irq272: igb0:que 3               1960849          4
> irq273: igb0:que 4               2645369          5
> irq274: igb0:que 5               2735187          6
> irq275: igb0:que 6               2290531          5
> irq276: igb0:que 7               2384370          5
> irq277: igb0:link                      2          0
> irq287: igb2:que 0               1465051          3
> irq288: igb2:que 1                856381          1
> irq289: igb2:que 2                809318          1
> irq290: igb2:que 3                897154          1
> irq291: igb2:que 4                875755          1
> irq292: igb2:que 5              35866117         78
> irq293: igb2:que 6                846517          1
> irq294: igb2:que 7                857979          1
> irq295: igb2:link                      2          0
> irq296: igb3:que 0                535212          1
> irq297: igb3:que 1                454359          1
> irq298: igb3:que 2                454142          1
> irq299: igb3:que 3                454623          1
> irq300: igb3:que 4                456297          1
> irq301: igb3:que 5                455482          1
> irq302: igb3:que 6                456128          1
> irq303: igb3:que 7                454680          1
> irq304: igb3:link                      3          0
> irq305: ahci0                         75          0
> cpu1:timer                     257233702        566
> cpu13:timer                    255603184        562
> cpu7:timer                     258492826        569
> cpu12:timer                    255819351        563
> cpu6:timer                     258493465        569
> cpu15:timer                    254694003        560
> cpu3:timer                     258171320        568
> cpu22:timer                    256506877        564
> cpu5:timer                     253401435        558
> cpu16:timer                    255412360        562
> cpu11:timer                    257318013        566
> cpu20:timer                    253648060        558
> cpu2:timer                     257864543        567
> cpu17:timer                    261828899        576
> cpu9:timer                     257497326        567
> cpu18:timer                    258451190        569
> cpu8:timer                     257784504        567
> cpu14:timer                    254923723        561
> cpu10:timer                    257265498        566
> cpu19:timer                    258775946        569
> cpu4:timer                     256368658        564
> cpu23:timer                    255050534        561
> cpu21:timer                    257663842        567
> Total                         6225260206      13710
>=20
> Please note igb2 on client side is the dedicated link for NFSv4
>=20
> Regards,
>=20
> Lo=C3=AFc Blot,
> UNIX Systems, Network and Security Engineer
> http://www.unix-experience.fr
>=20
> 6 janvier 2015 04:17 "Rick Macklem" <rmacklem@uoguelph.ca> a =C3=A9crit:
> > Loic Blot wrote:
> >=20
> >> Hi Rick,
> >> nfsstat -e -s don't show usefull datas on server.
> >=20
> > Well, as far as I know, it returns valid information.
> > (See below.)
> >=20
> >> Server Info:
> >> Getattr Setattr Lookup Readlink Read Write Create
> >> Remove
> >> 26935254 16911 5755728 302 2334920 3673866 0
> >> 328332
> >> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus
> >> Access
> >> 77980 28 0 0 3 8900 3
> >> 1806052
> >> Mknod Fsstat Fsinfo PathConf Commit LookupP SetClId
> >> SetClIdCf
> >> 1 1095 0 0 614377 8172 8
> >> 8
> >> Open OpenAttr OpenDwnGr OpenCfrm DelePurge DeleRet GetFH
> >> Lock
> >> 1595299 0 44145 1495 0 0 5197490
> >> 635015
> >> LockT LockU Close Verify NVerify PutFH PutPubFH
> >> PutRootFH
> >> 0 614919 1270938 0 0 22688676 0
> >> 5
> >> Renew RestoreFH SaveFH Secinfo RelLckOwn V4Create
> >> 42104 197606 275820 0 143 4578
> >> Server:
> >> Retfailed Faults Clients
> >> 0 0 6
> >> OpenOwner Opens LockOwner Locks Delegs
> >> 32335 145448 204 181 0
> >=20
> > Well, 145448 Opens are a lot of Open files. Each of these uses
> > a kernel malloc'd data structure that is linked into multiple
> > linked lists.
> >=20
> > The question is..why aren't these Opens being closed?
> > Since FreeBSD does I/O on an mmap'd file after closing it,
> > the FreeBSD NFSv4 client is forced to delay doing Close RPCs
> > until the vnode is VOP_INACTIVE()/VOP_RECLAIM()'d. (The
> > VOP_RECLAIM() case is needed, since VOP_INACTIVE() isn't
> > guaranteed to be called.)
> >=20
> > Since there were about 1.5 million Opens and 1.27 million
> > Closes, it does appear that Opens are being Closed.
> > Now, I'm not sure I would have imagined 1.5million file Opens
> > in a few days. My guess is this is the bottleneck.
> >=20
> > I'd suggest that you do:
> > # nfsstat -e -c
> > on each of the NFSv4 clients and see how many Opens/client
> > there are. I vaguely remember an upper limit in the client,
> > but can't remember what it is set to.
> > --> I suspect the client Open/Lock limit needs to be increased.
> > (I can't remember if the server also has a limit, but I
> > think it does.)
> > Then the size of the hash tables used to search the Opens
> > may also need to be increased a lot.
> >=20
> > Also, I'd suggest you take a look at whatever apps. are
> > running on the client(s) and try to figure out why they
> > are Opening so many files?
> >=20
> > My guess is that the client(s) are gettig bogged down by all
> > these Opens.
> >=20
> >> Server Cache Stats:
> >> Inprog Idem Non-idem Misses CacheSize TCPPeak
> >> 0 0 1 15082947 60 16522
> >>=20
> >> Only GetAttr and Lookup increase and it's only every 4-5 seconds
> >> and
> >> only +2 to +5 into theses values.
> >>=20
> >> Now on client, if i take four processes stack i got
> >>=20
> >> PID TID COMM TDNAME KSTACK
> >> 63170 102547 mv - mi_switch+0xe1
> >> turnstile_wait+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65
> >> nfs_lookup+0x3d0 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4
> >> vn_open_cred+0x21d kern_openat+0x26f amd64_syscall+0x351
> >> Xfast_syscall+0xfb
> >>=20
> >> Another mv:
> >> 63140 101738 mv - mi_switch+0xe1
> >> turnstile_wait+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65
> >> nfs_lookup+0x3d0 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4
> >> kern_statat_vnhook+0xae sys_lstat+0x30 amd64_syscall+0x351
> >> Xfast_syscall+0xfb
> >>=20
> >> 62070 102170 sendmail - mi_switch+0xe1
> >> sleepq_timedwait+0x3a _sleep+0x26e clnt_vc_call+0x666
> >> clnt_reconnect_call+0x4fa newnfs_request+0xa8c nfscl_request+0x72
> >> nfsrpc_lookup+0x1fb nfs_lookup+0x508 VOP_LOOKUP_APV+0xa1
> >> lookup+0x59c namei+0x4d4 kern_statat_vnhook+0xae sys_lstat+0x30
> >> amd64_syscall+0x351 Xfast_syscall+0xfb
> >>=20
> >> 63200 100930 mv - mi_switch+0xe1
> >> turnstile_wait+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65
> >> nfs_lookup+0x3d0 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4
> >> kern_statat_vnhook+0xae sys_lstat+0x30 amd64_syscall+0x351
> >> Xfast_syscall+0xfb
> >=20
> > The above simply says that thread 102710 is waiting for a Lookup
> > reply from the server and the other 3 are waiting for the mutex
> > that protects the state structures in the client. (I suspect
> > some other thread in the client is wading through the Open list,
> > if a single client has a lot of these 145K Opens.)
> >=20
> >> When client is in this state, server was doing nothing special
> >> (procstat -kk)
> >>=20
> >> PID TID COMM TDNAME KSTACK
> >> 895 100538 nfsd nfsd: master mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10
> >> _cv_timedwait_sig_sbt+0x18b svc_run_internal+0x4a1 svc_run+0x1de
> >> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c
> >> amd64_syscall+0x351 Xfast_syscall+0xfb
> >> 895 100568 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100569 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100570 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100571 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100572 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100573 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100575 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100576 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100577 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100578 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100579 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100580 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100581 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100582 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100583 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100584 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100585 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100586 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100587 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100588 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100589 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100590 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100592 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100593 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100594 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100595 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100596 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100597 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100598 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100599 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100600 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100602 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100603 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100604 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100605 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100606 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100607 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100608 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100609 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100610 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100611 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100612 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100613 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100614 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100615 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100617 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100618 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100619 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100621 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100622 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100623 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100624 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100625 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100626 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100627 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100628 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100629 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100630 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100631 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100632 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100633 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100634 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100635 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100636 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100638 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100639 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100640 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100641 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100642 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100643 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100644 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100645 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100646 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100647 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100648 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100649 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100651 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100652 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100653 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100654 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100655 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100656 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100657 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100658 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100659 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100661 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100662 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100684 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100685 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100686 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100797 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100798 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100799 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100800 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 895 100801 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >>=20
> >> I really think it's a client side problem, maybe a lookup problem.
> >>=20
> >> Regards,
> >>=20
> >> Lo=C3=AFc Blot,
> >> UNIX Systems, Network and Security Engineer
> >> http://www.unix-experience.fr
> >>=20
> >> 5 janvier 2015 14:35 "Rick Macklem" <rmacklem@uoguelph.ca> a
> >> =C3=A9crit:
> >>> Loic Blot wrote:
> >>>=20
> >>>> Hi,
> >>>> happy new year Rick and @freebsd-fs.
> >>>>=20
> >>>> After some days, i looked my NFSv4.1 mount. At server start it
> >>>> was
> >>>> calm, but after 4 days, here is the top stat...
> >>>>=20
> >>>> CPU: 0.0% user, 0.0% nice, 100% system, 0.0% interrupt, 0.0%
> >>>> idle
> >>>>=20
> >>>> Definitively i think it's a problem on client side. What can i
> >>>> look
> >>>> into running kernel to resolve this issue ?
> >>>=20
> >>> Well, I'd start with:
> >>> # nfsstat -e -s
> >>> - run repeatedly on the server (once every N seconds in a loop).
> >>> Then look at the output, comparing the counts and see which RPCs
> >>> are being performed by the client(s). You are looking for which
> >>> RPCs are being done a lot. (If one RPC is almost 100% of the
> >>> load,
> >>> then it might be a client/caching issue for whatever that RPC is
> >>> doing.)
> >>>=20
> >>> Also look at the Open/Lock counts near the end of the output.
> >>> If the # of Opens/Locks is large, it may be possible to reduce
> >>> the
> >>> CPU overheads by using larger hash tables.
> >>>=20
> >>> Then you need to profile the server kernel to see where the CPU
> >>> is being used.
> >>> Hopefully someone else can fill you in on how to do that, because
> >>> I'll admit I don't know how to.
> >>> Basically you are looking to see if the CPU is being used in
> >>> the NFS server code or ZFS.
> >>>=20
> >>> Good luck with it, rick
> >>>=20
> >>>> Regards,
> >>>>=20
> >>>> Lo=C3=AFc Blot,
> >>>> UNIX Systems, Network and Security Engineer
> >>>> http://www.unix-experience.fr
> >>>>=20
> >>>> 30 d=C3=A9cembre 2014 16:16 "Lo=C3=AFc Blot"
> >>>> <loic.blot@unix-experience.fr>
> >>>> a
> >>>> =C3=A9crit:
> >>>>> Hi Rick,
> >>>>> i upgraded my jail host from FreeBSD 9.3 to 10.1 to use NFS
> >>>>> v4.1
> >>>>> (mountoptions:
> >>>>> rw,rsize=3D32768,wsize=3D32768,tcp,nfsv4,minorversion=3D1)
> >>>>>=20
> >>>>> Performance is quite stable but it's slow. Not as slow as
> >>>>> before
> >>>>> but slow... services was launched
> >>>>> but no client are using them and system CPU % was 10-50%.
> >>>>>=20
> >>>>> I don't see anything on NFSv4.1 server, it's perfectly stable
> >>>>> and
> >>>>> functionnal.
> >>>>>=20
> >>>>> Regards,
> >>>>>=20
> >>>>> Lo=C3=AFc Blot,
> >>>>> UNIX Systems, Network and Security Engineer
> >>>>> http://www.unix-experience.fr
> >>>>>=20
> >>>>> 23 d=C3=A9cembre 2014 00:20 "Rick Macklem" <rmacklem@uoguelph.ca> a
> >>>>> =C3=A9crit:
> >>>>>=20
> >>>>>> Loic Blot wrote:
> >>>>>>=20
> >>>>>>> Hi,
> >>>>>>>=20
> >>>>>>> To clarify because of our exchanges. Here are the current
> >>>>>>> sysctl
> >>>>>>> options for server:
> >>>>>>>=20
> >>>>>>> vfs.nfsd.enable_nobodycheck=3D0
> >>>>>>> vfs.nfsd.enable_nogroupcheck=3D0
> >>>>>>>=20
> >>>>>>> vfs.nfsd.maxthreads=3D200
> >>>>>>> vfs.nfsd.tcphighwater=3D10000
> >>>>>>> vfs.nfsd.tcpcachetimeo=3D300
> >>>>>>> vfs.nfsd.server_min_nfsvers=3D4
> >>>>>>>=20
> >>>>>>> kern.maxvnodes=3D10000000
> >>>>>>> kern.ipc.maxsockbuf=3D4194304
> >>>>>>> net.inet.tcp.sendbuf_max=3D4194304
> >>>>>>> net.inet.tcp.recvbuf_max=3D4194304
> >>>>>>>=20
> >>>>>>> vfs.lookup_shared=3D0
> >>>>>>>=20
> >>>>>>> Regards,
> >>>>>>>=20
> >>>>>>> Lo=C3=AFc Blot,
> >>>>>>> UNIX Systems, Network and Security Engineer
> >>>>>>> http://www.unix-experience.fr
> >>>>>>>=20
> >>>>>>> 22 d=C3=A9cembre 2014 09:42 "Lo=C3=AFc Blot"
> >>>>>>> <loic.blot@unix-experience.fr>
> >>>>>>> a
> >>>>>>> =C3=A9crit:
> >>>>>>>=20
> >>>>>>> Hi Rick,
> >>>>>>> my 5 jails runs this weekend and now i have some stats on
> >>>>>>> this
> >>>>>>> monday.
> >>>>>>>=20
> >>>>>>> Hopefully deadlock was fixed, yeah, but everything isn't good
> >>>>>>> :(
> >>>>>>>=20
> >>>>>>> On NFSv4 server (FreeBSD 10.1) system uses 35% CPU
> >>>>>>>=20
> >>>>>>> As i can see this is because of nfsd:
> >>>>>>>=20
> >>>>>>> 918 root 96 20 0 12352K 3372K rpcsvc 6 51.4H
> >>>>>>> 273.68% nfsd: server (nfsd)
> >>>>>>>=20
> >>>>>>> If i look at dmesg i see:
> >>>>>>> nfsd server cache flooded, try increasing
> >>>>>>> vfs.nfsd.tcphighwater
> >>>>>>=20
> >>>>>> Well, you have a couple of choices:
> >>>>>> 1 - Use NFSv4.1 (add "minorversion=3D1" to your mount options).
> >>>>>> (NFSv4.1 avoids use of the DRC and instead uses something
> >>>>>> called sessions. See below.)
> >>>>>> OR
> >>>>>>=20
> >>>>>>> vfs.nfsd.tcphighwater was set to 10000, i increase it to
> >>>>>>> 15000
> >>>>>>=20
> >>>>>> 2 - Bump vfs.nfsd.tcphighwater way up, until you no longer see
> >>>>>> "nfs server cache flooded" messages. (I think Garrett Wollman
> >>>>>> uses
> >>>>>> 100000. (You may still see quite a bit of CPU overheads.)
> >>>>>>=20
> >>>>>> OR
> >>>>>>=20
> >>>>>> 3 - Set vfs.nfsd.cachetcp=3D0 (which disables the DRC and gets
> >>>>>> rid
> >>>>>> of the CPU overheads). However, there is a risk of data
> >>>>>> corruption
> >>>>>> if you have a client->server network partitioning of a
> >>>>>> moderate
> >>>>>> duration, because a non-idempotent RPC may get redone, becasue
> >>>>>> the client times out waiting for a reply. If a non-idempotent
> >>>>>> RPC gets done twice on the server, data corruption can happen.
> >>>>>> (The DRC provides improved correctness, but does add
> >>>>>> overhead.)
> >>>>>>=20
> >>>>>> If #1 works for you, it is the preferred solution, since
> >>>>>> Sessions
> >>>>>> in NFSv4.1 solves the correctness problem in a good, space
> >>>>>> bound
> >>>>>> way. A session basically has N (usually 32 or 64) slots and
> >>>>>> only
> >>>>>> allows one outstanding RPC/slot. As such, it can cache the
> >>>>>> previous
> >>>>>> reply for each slot (32 or 64 of them) and guarantee "exactly
> >>>>>> once"
> >>>>>> RPC semantics.
> >>>>>>=20
> >>>>>> rick
> >>>>>>=20
> >>>>>>> Here is 'nfsstat -s' output:
> >>>>>>>=20
> >>>>>>> Server Info:
> >>>>>>> Getattr Setattr Lookup Readlink Read Write Create
> >>>>>>> Remove
> >>>>>>> 12600652 1812 2501097 156 1386423 1983729 123
> >>>>>>> 162067
> >>>>>>> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus
> >>>>>>> Access
> >>>>>>> 36762 9 0 0 0 3147 0
> >>>>>>> 623524
> >>>>>>> Mknod Fsstat Fsinfo PathConf Commit
> >>>>>>> 0 0 0 0 328117
> >>>>>>> Server Ret-Failed
> >>>>>>> 0
> >>>>>>> Server Faults
> >>>>>>> 0
> >>>>>>> Server Cache Stats:
> >>>>>>> Inprog Idem Non-idem Misses
> >>>>>>> 0 0 0 12635512
> >>>>>>> Server Write Gathering:
> >>>>>>> WriteOps WriteRPC Opsaved
> >>>>>>> 1983729 1983729 0
> >>>>>>>=20
> >>>>>>> And here is 'procstat -kk' for nfsd (server)
> >>>>>>>=20
> >>>>>>> 918 100528 nfsd nfsd: master mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10
> >>>>>>> _cv_timedwait_sig_sbt+0x18b svc_run_internal+0x4a1
> >>>>>>> svc_run+0x1de
> >>>>>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c
> >>>>>>> amd64_syscall+0x351 Xfast_syscall+0xfb
> >>>>>>> 918 100568 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100569 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100570 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100571 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100572 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100573 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100574 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100575 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100576 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100577 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100578 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100579 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100580 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100581 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100582 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100583 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100584 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100585 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100586 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100587 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100588 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100589 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100590 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100591 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100592 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100593 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100594 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100595 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100596 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100597 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100598 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100599 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100600 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100601 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100602 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100603 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100604 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100605 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100606 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100607 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100608 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100609 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100610 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100611 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100612 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100613 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100614 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100615 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100616 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100617 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100618 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100619 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100620 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100621 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100622 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100623 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100624 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100625 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100626 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100627 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100628 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100629 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100630 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100631 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100632 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100633 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100634 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100635 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100636 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100637 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100638 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100639 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100640 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100641 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100642 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100643 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100644 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100645 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100646 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100647 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100648 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100649 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100650 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100651 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100652 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100653 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100654 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100655 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100656 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100657 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100658 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100659 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100660 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100661 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100662 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> ---
> >>>>>>>=20
> >>>>>>> Now if we look at client (FreeBSD 9.3)
> >>>>>>>=20
> >>>>>>> We see system was very busy and do many and many interrupts
> >>>>>>>=20
> >>>>>>> CPU: 0.0% user, 0.0% nice, 37.8% system, 51.2% interrupt,
> >>>>>>> 11.0%
> >>>>>>> idle
> >>>>>>>=20
> >>>>>>> A look at process list shows that there are many sendmail
> >>>>>>> process
> >>>>>>> in
> >>>>>>> state nfstry
> >>>>>>>=20
> >>>>>>> nfstry 18 32:27 0.88% sendmail: Queue runner@00:30:00 for
> >>>>>>> /var/spool/clientm
> >>>>>>>=20
> >>>>>>> Here is 'nfsstat -c' output:
> >>>>>>>=20
> >>>>>>> Client Info:
> >>>>>>> Rpc Counts:
> >>>>>>> Getattr Setattr Lookup Readlink Read Write Create
> >>>>>>> Remove
> >>>>>>> 1051347 1724 2494481 118 903902 1901285 162676
> >>>>>>> 161899
> >>>>>>> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus
> >>>>>>> Access
> >>>>>>> 36744 2 0 114 40 3131 0
> >>>>>>> 544136
> >>>>>>> Mknod Fsstat Fsinfo PathConf Commit
> >>>>>>> 9 0 0 0 245821
> >>>>>>> Rpc Info:
> >>>>>>> TimedOut Invalid X Replies Retries Requests
> >>>>>>> 0 0 0 0 8356557
> >>>>>>> Cache Info:
> >>>>>>> Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits
> >>>>>>> Misses
> >>>>>>> 108754455 491475 54229224 2437229 46814561 821723 5132123
> >>>>>>> 1871871
> >>>>>>> BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits
> >>>>>>> Misses
> >>>>>>> 144035 118 53736 2753 27813 1 57238839
> >>>>>>> 544205
> >>>>>>>=20
> >>>>>>> If you need more things, tell me, i let the PoC in this
> >>>>>>> state.
> >>>>>>>=20
> >>>>>>> Thanks
> >>>>>>>=20
> >>>>>>> Regards,
> >>>>>>>=20
> >>>>>>> Lo=C3=AFc Blot,
> >>>>>>> UNIX Systems, Network and Security Engineer
> >>>>>>> http://www.unix-experience.fr
> >>>>>>>=20
> >>>>>>> 21 d=C3=A9cembre 2014 01:33 "Rick Macklem" <rmacklem@uoguelph.ca>
> >>>>>>> a
> >>>>>>> =C3=A9crit:
> >>>>>>>=20
> >>>>>>> Loic Blot wrote:
> >>>>>>>=20
> >>>>>>>> Hi Rick,
> >>>>>>>> ok, i don't need locallocks, i haven't understand option was
> >>>>>>>> for
> >>>>>>>> that
> >>>>>>>> usage, i removed it.
> >>>>>>>> I do more tests on monday.
> >>>>>>>> Thanks for the deadlock fix, for other people :)
> >>>>>>>=20
> >>>>>>> Good. Please let us know if running with
> >>>>>>> vfs.nfsd.enable_locallocks=3D0
> >>>>>>> gets rid of the deadlocks? (I think it fixes the one you
> >>>>>>> saw.)
> >>>>>>>=20
> >>>>>>> On the performance side, you might also want to try different
> >>>>>>> values
> >>>>>>> of
> >>>>>>> readahead, if the Linux client has such a mount option. (With
> >>>>>>> the
> >>>>>>> NFSv4-ZFS sequential vs random I/O heuristic, I have no idea
> >>>>>>> what
> >>>>>>> the
> >>>>>>> optimal readahead value would be.)
> >>>>>>>=20
> >>>>>>> Good luck with it and please let us know how it goes, rick
> >>>>>>> ps: I now have a patch to fix the deadlock when
> >>>>>>> vfs.nfsd.enable_locallocks=3D1
> >>>>>>> is set. I'll post it for anyone who is interested after I put
> >>>>>>> it
> >>>>>>> through some testing.
> >>>>>>>=20
> >>>>>>> --
> >>>>>>> Best regards,
> >>>>>>> Lo=C3=AFc BLOT,
> >>>>>>> UNIX systems, security and network engineer
> >>>>>>> http://www.unix-experience.fr
> >>>>>>>=20
> >>>>>>> Le jeudi 18 d=C3=A9cembre 2014 =C3=A0 19:46 -0500, Rick Macklem a=
 =C3=A9crit
> >>>>>>> :
> >>>>>>>=20
> >>>>>>> Loic Blot wrote:
> >>>>>>>> Hi rick,
> >>>>>>>> i tried to start a LXC container on Debian Squeeze from my
> >>>>>>>> freebsd
> >>>>>>>> ZFS+NFSv4 server and i also have a deadlock on nfsd
> >>>>>>>> (vfs.lookup_shared=3D0). Deadlock procs each time i launch a
> >>>>>>>> squeeze
> >>>>>>>> container, it seems (3 tries, 3 fails).
> >>>>>>>=20
> >>>>>>> Well, I`ll take a look at this `procstat -kk`, but the only
> >>>>>>> thing
> >>>>>>> I`ve seen posted w.r.t. avoiding deadlocks in ZFS is to not
> >>>>>>> use
> >>>>>>> nullfs. (I have no idea if you are using any nullfs mounts,
> >>>>>>> but
> >>>>>>> if so, try getting rid of them.)
> >>>>>>>=20
> >>>>>>> Here`s a high level post about the ZFS and vnode locking
> >>>>>>> problem,
> >>>>>>> but there is no patch available, as far as I know.
> >>>>>>>=20
> >>>>>>> http://docs.FreeBSD.org/cgi/mid.cgi?54739F41.8030407
> >>>>>>>=20
> >>>>>>> rick
> >>>>>>>=20
> >>>>>>> 921 - D 0:00.02 nfsd: server (nfsd)
> >>>>>>>=20
> >>>>>>> Here is the procstat -kk
> >>>>>>>=20
> >>>>>>> PID TID COMM TDNAME KSTACK
> >>>>>>> 921 100538 nfsd nfsd: master mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad
> >>>>>>> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554
> >>>>>>> svc_run_internal+0xc77 svc_run+0x1de nfsrvd_nfsd+0x1ca
> >>>>>>> nfssvc_nfsd+0x107 sys_nfssvc+0x9c
> >>>>>>> 921 100572 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100573 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100574 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100575 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100576 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100577 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100578 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100579 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100580 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100581 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100582 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100583 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100584 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100585 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100586 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100587 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100588 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100589 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100590 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100591 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100592 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100593 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100594 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100595 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100596 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100597 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100598 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100599 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100600 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100601 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100602 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100603 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100604 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100605 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100606 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100607 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100608 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100609 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100610 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100611 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100612 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100613 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100614 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100615 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100616 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>>>> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f
> >>>>>>> nfsrvd_lock+0x5b1
> >>>>>>> nfsrvd_dorpc+0xec6 nfssvc_program+0x554
> >>>>>>> svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 921 100617 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100618 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554
> >>>>>>> svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 921 100619 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100620 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100621 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100622 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100623 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100624 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100625 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100626 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100627 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100628 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100629 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100630 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100631 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100632 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100633 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100634 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100635 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100636 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100637 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100638 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100639 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100640 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100641 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100642 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100643 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100644 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100645 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100646 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100647 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100648 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100649 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100650 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100651 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100652 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100653 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100654 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100655 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100656 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100657 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100658 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100659 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100660 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100661 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100662 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100663 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100664 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100665 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>> _cv_wait_sig+0x16a
> >>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 921 100666 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>>>> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8
> >>>>>>> nfsrvd_dorpc+0xc76
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>=20
> >>>>>>> Regards,
> >>>>>>>=20
> >>>>>>> Lo=C3=AFc Blot,
> >>>>>>> UNIX Systems, Network and Security Engineer
> >>>>>>> http://www.unix-experience.fr
> >>>>>>>=20
> >>>>>>> 15 d=C3=A9cembre 2014 15:18 "Rick Macklem" <rmacklem@uoguelph.ca>
> >>>>>>> a
> >>>>>>> =C3=A9crit:
> >>>>>>>=20
> >>>>>>> Loic Blot wrote:
> >>>>>>>=20
> >>>>>>>> For more informations, here is procstat -kk on nfsd, if you
> >>>>>>>> need
> >>>>>>>> more
> >>>>>>>> hot datas, tell me.
> >>>>>>>>=20
> >>>>>>>> Regards, PID TID COMM TDNAME KSTACK
> >>>>>>>> 918 100529 nfsd nfsd: master mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_run+0x1de
> >>>>>>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c
> >>>>>>>> amd64_syscall+0x351
> >>>>>>>=20
> >>>>>>> Well, most of the threads are stuck like this one, waiting
> >>>>>>> for
> >>>>>>> a
> >>>>>>> vnode
> >>>>>>> lock in ZFS. All of them appear to be in zfs_fhtovp().
> >>>>>>> I`m not a ZFS guy, so I can`t help much. I`ll try changing
> >>>>>>> the
> >>>>>>> subject line
> >>>>>>> to include ZFS vnode lock, so maybe the ZFS guys will take a
> >>>>>>> look.
> >>>>>>>=20
> >>>>>>> The only thing I`ve seen suggested is trying:
> >>>>>>> sysctl vfs.lookup_shared=3D0
> >>>>>>> to disable shared vop_lookup()s. Apparently zfs_lookup()
> >>>>>>> doesn`t
> >>>>>>> obey the vnode locking rules for lookup and rename, according
> >>>>>>> to
> >>>>>>> the posting I saw.
> >>>>>>>=20
> >>>>>>> I`ve added a couple of comments about the other threads
> >>>>>>> below,
> >>>>>>> but
> >>>>>>> they are all either waiting for an RPC request or waiting for
> >>>>>>> the
> >>>>>>> threads stuck on the ZFS vnode lock to complete.
> >>>>>>>=20
> >>>>>>> rick
> >>>>>>>=20
> >>>>>>>> 918 100564 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>>> _cv_wait_sig+0x16a
> >>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>>> fork_trampoline+0xe
> >>>>>>>=20
> >>>>>>> Fyi, this thread is just waiting for an RPC to arrive.
> >>>>>>> (Normal)
> >>>>>>>=20
> >>>>>>>> 918 100565 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>>> _cv_wait_sig+0x16a
> >>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>>> fork_trampoline+0xe
> >>>>>>>> 918 100566 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>>> _cv_wait_sig+0x16a
> >>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>>> fork_trampoline+0xe
> >>>>>>>> 918 100567 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>>> _cv_wait_sig+0x16a
> >>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>>> fork_trampoline+0xe
> >>>>>>>> 918 100568 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>>> _cv_wait_sig+0x16a
> >>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>>> fork_trampoline+0xe
> >>>>>>>> 918 100569 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>>> _cv_wait_sig+0x16a
> >>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>>> fork_trampoline+0xe
> >>>>>>>> 918 100570 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>>>> _cv_wait_sig+0x16a
> >>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>>>> fork_trampoline+0xe
> >>>>>>>> 918 100571 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554
> >>>>>>>> svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100572 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>>>>> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8
> >>>>>>>> nfsrvd_dorpc+0xc76
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>=20
> >>>>>>> This one (and a few others) are waiting for the nfsv4_lock.
> >>>>>>> This
> >>>>>>> happens
> >>>>>>> because other threads are stuck with RPCs in progress. (ie.
> >>>>>>> The
> >>>>>>> ones
> >>>>>>> waiting on the vnode lock in zfs_fhtovp().)
> >>>>>>> For these, the RPC needs to lock out other threads to do the
> >>>>>>> operation,
> >>>>>>> so it waits for the nfsv4_lock() which can exclusively lock
> >>>>>>> the
> >>>>>>> NFSv4
> >>>>>>> data structures once all other nfsd threads complete their
> >>>>>>> RPCs
> >>>>>>> in
> >>>>>>> progress.
> >>>>>>>=20
> >>>>>>>> 918 100573 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554
> >>>>>>>> svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>=20
> >>>>>>> Same as above.
> >>>>>>>=20
> >>>>>>>> 918 100574 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100575 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100576 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100577 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100578 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100579 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100580 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100581 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100582 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100583 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100584 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100585 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100586 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100587 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100588 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100589 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100590 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100591 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100592 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100593 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100594 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100595 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100596 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100597 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100598 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100599 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100600 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100601 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100602 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100603 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100604 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100605 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100606 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>> 918 100607 nfsd nfsd: service mi_switch+0xe1
> >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>>> zfs_fhtovp+0x38d
> >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>>> svc_thread_start+0xb
> >>>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>=20
> >>>>>>> Lots more waiting for the ZFS vnode lock in zfs_fhtovp().
> >>>>>>>=20
> >>>>>>> 918 100608 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>>>> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f
> >>>>>>> nfsrvd_lock+0x5b1
> >>>>>>> nfsrvd_dorpc+0xec6 nfssvc_program+0x554
> >>>>>>> svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100609 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100610 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad
> >>>>>>> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554
> >>>>>>> svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a
> >>>>>>> fork_trampoline+0xe
> >>>>>>> 918 100611 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554
> >>>>>>> svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100612 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554
> >>>>>>> svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100613 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554
> >>>>>>> svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100614 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554
> >>>>>>> svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100615 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554
> >>>>>>> svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100616 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554
> >>>>>>> svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100617 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554
> >>>>>>> svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100618 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554
> >>>>>>> svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100619 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100620 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100621 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554
> >>>>>>> svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100622 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100623 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554
> >>>>>>> svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100624 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100625 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100626 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100627 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100628 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100629 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100630 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100631 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100632 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100633 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100634 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100635 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100636 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100637 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100638 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100639 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100640 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100641 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100642 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100643 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100644 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100645 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100646 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100647 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100648 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100649 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100650 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100651 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100652 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100653 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100654 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100655 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100656 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100657 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>> 918 100658 nfsd nfsd: service mi_switch+0xe1
> >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>>> zfs_fhtovp+0x38d
> >>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>>> svc_thread_start+0xb
> >>>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>>>=20
> >>>>>>> Lo=C3=AFc Blot,
> >>>>>>> UNIX Systems, Network and Security Engineer
> >>>>>>> http://www.unix-experience.fr
> >>>>>>>=20
> >>>>>>> 15 d=C3=A9cembre 2014 13:29 "Lo=C3=AFc Blot"
> >>>>>>> <loic.blot@unix-experience.fr>
> >>>>>>> a
> >>>>>>> =C3=A9crit:
> >>>>>>>=20
> >>>>>>> Hmmm...
> >>>>>>> now i'm experiencing a deadlock.
> >>>>>>>=20
> >>>>>>> 0 918 915 0 21 0 12352 3372 zfs D - 1:48.64 nfsd: server
> >>>>>>> (nfsd)
> >>>>>>>=20
> >>>>>>> the only issue was to reboot the server, but after rebooting
> >>>>>>> deadlock arrives a second time when i
> >>>>>>> start my jails over NFS.
> >>>>>>>=20
> >>>>>>> Regards,
> >>>>>>>=20
> >>>>>>> Lo=C3=AFc Blot,
> >>>>>>> UNIX Systems, Network and Security Engineer
> >>>>>>> http://www.unix-experience.fr
> >>>>>>>=20
> >>>>>>> 15 d=C3=A9cembre 2014 10:07 "Lo=C3=AFc Blot"
> >>>>>>> <loic.blot@unix-experience.fr>
> >>>>>>> a
> >>>>>>> =C3=A9crit:
> >>>>>>>=20
> >>>>>>> Hi Rick,
> >>>>>>> after talking with my N+1, NFSv4 is required on our
> >>>>>>> infrastructure.
> >>>>>>> I tried to upgrade NFSv4+ZFS
> >>>>>>> server from 9.3 to 10.1, i hope this will resolve some
> >>>>>>> issues...
> >>>>>>>=20
> >>>>>>> Regards,
> >>>>>>>=20
> >>>>>>> Lo=C3=AFc Blot,
> >>>>>>> UNIX Systems, Network and Security Engineer
> >>>>>>> http://www.unix-experience.fr
> >>>>>>>=20
> >>>>>>> 10 d=C3=A9cembre 2014 15:36 "Lo=C3=AFc Blot"
> >>>>>>> <loic.blot@unix-experience.fr>
> >>>>>>> a
> >>>>>>> =C3=A9crit:
> >>>>>>>=20
> >>>>>>> Hi Rick,
> >>>>>>> thanks for your suggestion.
> >>>>>>> For my locking bug, rpc.lockd is stucked in rpcrecv state on
> >>>>>>> the
> >>>>>>> server. kill -9 doesn't affect the
> >>>>>>> process, it's blocked.... (State: Ds)
> >>>>>>>=20
> >>>>>>> for the performances
> >>>>>>>=20
> >>>>>>> NFSv3: 60Mbps
> >>>>>>> NFSv4: 45Mbps
> >>>>>>> Regards,
> >>>>>>>=20
> >>>>>>> Lo=C3=AFc Blot,
> >>>>>>> UNIX Systems, Network and Security Engineer
> >>>>>>> http://www.unix-experience.fr
> >>>>>>>=20
> >>>>>>> 10 d=C3=A9cembre 2014 13:56 "Rick Macklem" <rmacklem@uoguelph.ca>
> >>>>>>> a
> >>>>>>> =C3=A9crit:
> >>>>>>>=20
> >>>>>>> Loic Blot wrote:
> >>>>>>>=20
> >>>>>>>> Hi Rick,
> >>>>>>>> I'm trying NFSv3.
> >>>>>>>> Some jails are starting very well but now i have an issue
> >>>>>>>> with
> >>>>>>>> lockd
> >>>>>>>> after some minutes:
> >>>>>>>>=20
> >>>>>>>> nfs server 10.10.X.8:/jails: lockd not responding
> >>>>>>>> nfs server 10.10.X.8:/jails lockd is alive again
> >>>>>>>>=20
> >>>>>>>> I look at mbuf, but i seems there is no problem.
> >>>>>>>=20
> >>>>>>> Well, if you need locks to be visible across multiple
> >>>>>>> clients,
> >>>>>>> then
> >>>>>>> I'm afraid you are stuck with using NFSv4 and the
> >>>>>>> performance
> >>>>>>> you
> >>>>>>> get
> >>>>>>> from it. (There is no way to do file handle affinity for
> >>>>>>> NFSv4
> >>>>>>> because
> >>>>>>> the read and write ops are buried in the compound RPC and
> >>>>>>> not
> >>>>>>> easily
> >>>>>>> recognized.)
> >>>>>>>=20
> >>>>>>> If the locks don't need to be visible across multiple
> >>>>>>> clients,
> >>>>>>> I'd
> >>>>>>> suggest trying the "nolockd" option with nfsv3.
> >>>>>>>=20
> >>>>>>>> Here is my rc.conf on server:
> >>>>>>>>=20
> >>>>>>>> nfs_server_enable=3D"YES"
> >>>>>>>> nfsv4_server_enable=3D"YES"
> >>>>>>>> nfsuserd_enable=3D"YES"
> >>>>>>>> nfsd_server_flags=3D"-u -t -n 256"
> >>>>>>>> mountd_enable=3D"YES"
> >>>>>>>> mountd_flags=3D"-r"
> >>>>>>>> nfsuserd_flags=3D"-usertimeout 0 -force 20"
> >>>>>>>> rpcbind_enable=3D"YES"
> >>>>>>>> rpc_lockd_enable=3D"YES"
> >>>>>>>> rpc_statd_enable=3D"YES"
> >>>>>>>>=20
> >>>>>>>> Here is the client:
> >>>>>>>>=20
> >>>>>>>> nfsuserd_enable=3D"YES"
> >>>>>>>> nfsuserd_flags=3D"-usertimeout 0 -force 20"
> >>>>>>>> nfscbd_enable=3D"YES"
> >>>>>>>> rpc_lockd_enable=3D"YES"
> >>>>>>>> rpc_statd_enable=3D"YES"
> >>>>>>>>=20
> >>>>>>>> Have you got an idea ?
> >>>>>>>>=20
> >>>>>>>> Regards,
> >>>>>>>>=20
> >>>>>>>> Lo=C3=AFc Blot,
> >>>>>>>> UNIX Systems, Network and Security Engineer
> >>>>>>>> http://www.unix-experience.fr
> >>>>>>>>=20
> >>>>>>>> 9 d=C3=A9cembre 2014 04:31 "Rick Macklem" <rmacklem@uoguelph.ca>
> >>>>>>>> a
> >>>>>>>> =C3=A9crit:
> >>>>>>>>> Loic Blot wrote:
> >>>>>>>>>=20
> >>>>>>>>>> Hi rick,
> >>>>>>>>>>=20
> >>>>>>>>>> I waited 3 hours (no lag at jail launch) and now I do:
> >>>>>>>>>> sysrc
> >>>>>>>>>> memcached_flags=3D"-v -m 512"
> >>>>>>>>>> Command was very very slow...
> >>>>>>>>>>=20
> >>>>>>>>>> Here is a dd over NFS:
> >>>>>>>>>>=20
> >>>>>>>>>> 601062912 bytes transferred in 21.060679 secs (28539579
> >>>>>>>>>> bytes/sec)
> >>>>>>>>>=20
> >>>>>>>>> Can you try the same read using an NFSv3 mount?
> >>>>>>>>> (If it runs much faster, you have probably been bitten by
> >>>>>>>>> the
> >>>>>>>>> ZFS
> >>>>>>>>> "sequential vs random" read heuristic which I've been told
> >>>>>>>>> things
> >>>>>>>>> NFS is doing "random" reads without file handle affinity.
> >>>>>>>>> File
> >>>>>>>>> handle affinity is very hard to do for NFSv4, so it isn't
> >>>>>>>>> done.)
> >>>>>>>=20
> >>>>>>> I was actually suggesting that you try the "dd" over nfsv3
> >>>>>>> to
> >>>>>>> see
> >>>>>>> how
> >>>>>>> the performance compared with nfsv4. If you do that, please
> >>>>>>> post
> >>>>>>> the
> >>>>>>> comparable results.
> >>>>>>>=20
> >>>>>>> Someday I would like to try and get ZFS's sequential vs
> >>>>>>> random
> >>>>>>> read
> >>>>>>> heuristic modified and any info on what difference in
> >>>>>>> performance
> >>>>>>> that
> >>>>>>> might make for NFS would be useful.
> >>>>>>>=20
> >>>>>>> rick
> >>>>>>>=20
> >>>>>>> rick
> >>>>>>>=20
> >>>>>>> This is quite slow...
> >>>>>>>=20
> >>>>>>> You can found some nfsstat below (command isn't finished
> >>>>>>> yet)
> >>>>>>>=20
> >>>>>>> nfsstat -c -w 1
> >>>>>>>=20
> >>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 4 0 0 0 0 0 16 0
> >>>>>>> 2 0 0 0 0 0 17 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 4 0 0 0 0 4 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 4 0 0 0 0 0 3 0
> >>>>>>> 0 0 0 0 0 0 3 0
> >>>>>>> 37 10 0 8 0 0 14 1
> >>>>>>> 18 16 0 4 1 2 4 0
> >>>>>>> 78 91 0 82 6 12 30 0
> >>>>>>> 19 18 0 2 2 4 2 0
> >>>>>>> 0 0 0 0 2 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 1 0 0 0 0 1 0
> >>>>>>> 4 6 0 0 6 0 3 0
> >>>>>>> 2 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 1 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 1 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 6 108 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 98 54 0 86 11 0 25 0
> >>>>>>> 36 24 0 39 25 0 10 1
> >>>>>>> 67 8 0 63 63 0 41 0
> >>>>>>> 34 0 0 35 34 0 0 0
> >>>>>>> 75 0 0 75 77 0 0 0
> >>>>>>> 34 0 0 35 35 0 0 0
> >>>>>>> 75 0 0 74 76 0 0 0
> >>>>>>> 33 0 0 34 33 0 0 0
> >>>>>>> 0 0 0 0 5 0 0 0
> >>>>>>> 0 0 0 0 0 0 6 0
> >>>>>>> 11 0 0 0 0 0 11 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 17 0 0 0 0 1 0
> >>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> >>>>>>> 4 5 0 0 0 0 12 0
> >>>>>>> 2 0 0 0 0 0 26 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 4 0 0 0 0 4 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 4 0 0 0 0 0 2 0
> >>>>>>> 2 0 0 0 0 0 24 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 4 0 0 0 0 0 7 0
> >>>>>>> 2 1 0 0 0 0 1 0
> >>>>>>> 0 0 0 0 2 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 6 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 4 6 0 0 0 0 3 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 2 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 4 71 0 0 0 0 0 0
> >>>>>>> 0 1 0 0 0 0 0 0
> >>>>>>> 2 36 0 0 0 0 1 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 1 0 0 0 0 0 1 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 79 6 0 79 79 0 2 0
> >>>>>>> 25 0 0 25 26 0 6 0
> >>>>>>> 43 18 0 39 46 0 23 0
> >>>>>>> 36 0 0 36 36 0 31 0
> >>>>>>> 68 1 0 66 68 0 0 0
> >>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> >>>>>>> 36 0 0 36 36 0 0 0
> >>>>>>> 48 0 0 48 49 0 0 0
> >>>>>>> 20 0 0 20 20 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 3 14 0 1 0 0 11 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 4 0 0 0 0 4 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 4 22 0 0 0 0 16 0
> >>>>>>> 2 0 0 0 0 0 23 0
> >>>>>>>=20
> >>>>>>> Regards,
> >>>>>>>=20
> >>>>>>> Lo=C3=AFc Blot,
> >>>>>>> UNIX Systems, Network and Security Engineer
> >>>>>>> http://www.unix-experience.fr
> >>>>>>>=20
> >>>>>>> 8 d=C3=A9cembre 2014 09:36 "Lo=C3=AFc Blot"
> >>>>>>> <loic.blot@unix-experience.fr> a
> >>>>>>> =C3=A9crit:
> >>>>>>>> Hi Rick,
> >>>>>>>> I stopped the jails this week-end and started it this
> >>>>>>>> morning,
> >>>>>>>> i'll
> >>>>>>>> give you some stats this week.
> >>>>>>>>=20
> >>>>>>>> Here is my nfsstat -m output (with your rsize/wsize
> >>>>>>>> tweaks)
> >>>>>>=20
> >>>>>>=20
> >>>>>=20
> >>>>=20
> >>>=20
> >>=20
> > nfsv4,tcp,resvport,hard,cto,sec=3Dsys,acdirmin=3D3,acdirmax=3D60,acregm=
in=3D5,acregmax=3D60,nametimeo=3D60,negna
> >>>>>>=20
> >>>>>>>=20
> >>>>>>=20
> >>>>>>=20
> >>>>>=20
> >>>>=20
> >>>=20
> >>=20
> > etimeo=3D60,rsize=3D32768,wsize=3D32768,readdirsize=3D32768,readahead=
=3D1,wcommitsize=3D773136,timeout=3D120,retra
> >>>>>>=20
> >>>>>>> s=3D2147483647
> >>>>>>>=20
> >>>>>>> On server side my disks are on a raid controller which show a
> >>>>>>> 512b
> >>>>>>> volume and write performances
> >>>>>>> are very honest (dd if=3D/dev/zero of=3D/jails/test.dd bs=3D4096
> >>>>>>> count=3D100000000 =3D> 450MBps)
> >>>>>>>=20
> >>>>>>> Regards,
> >>>>>>>=20
> >>>>>>> Lo=C3=AFc Blot,
> >>>>>>> UNIX Systems, Network and Security Engineer
> >>>>>>> http://www.unix-experience.fr
> >>>>>>>=20
> >>>>>>> 5 d=C3=A9cembre 2014 15:14 "Rick Macklem" <rmacklem@uoguelph.ca> =
a
> >>>>>>> =C3=A9crit:
> >>>>>>>=20
> >>>>>>> Loic Blot wrote:
> >>>>>>>=20
> >>>>>>> Hi,
> >>>>>>> i'm trying to create a virtualisation environment based on
> >>>>>>> jails.
> >>>>>>> Those jails are stored under a big ZFS pool on a FreeBSD
> >>>>>>> 9.3
> >>>>>>> which
> >>>>>>> export a NFSv4 volume. This NFSv4 volume was mounted on a
> >>>>>>> big
> >>>>>>> hypervisor (2 Xeon E5v3 + 128GB memory and 8 ports (but
> >>>>>>> only 1
> >>>>>>> was
> >>>>>>> used at this time).
> >>>>>>>=20
> >>>>>>> The problem is simple, my hypervisors runs 6 jails (used 1%
> >>>>>>> cpu
> >>>>>>> and
> >>>>>>> 10GB RAM approximatively and less than 1MB bandwidth) and
> >>>>>>> works
> >>>>>>> fine at start but the system slows down and after 2-3 days
> >>>>>>> become
> >>>>>>> unusable. When i look at top command i see 80-100% on
> >>>>>>> system
> >>>>>>> and
> >>>>>>> commands are very very slow. Many process are tagged with
> >>>>>>> nfs_cl*.
> >>>>>>>=20
> >>>>>>> To be honest, I would expect the slowness to be because of
> >>>>>>> slow
> >>>>>>> response
> >>>>>>> from the NFSv4 server, but if you do:
> >>>>>>> # ps axHl
> >>>>>>> on a client when it is slow and post that, it would give us
> >>>>>>> some
> >>>>>>> more
> >>>>>>> information on where the client side processes are sitting.
> >>>>>>> If you also do something like:
> >>>>>>> # nfsstat -c -w 1
> >>>>>>> and let it run for a while, that should show you how many
> >>>>>>> RPCs
> >>>>>>> are
> >>>>>>> being done and which ones.
> >>>>>>>=20
> >>>>>>> # nfsstat -m
> >>>>>>> will show you what your mount is actually using.
> >>>>>>> The only mount option I can suggest trying is
> >>>>>>> "rsize=3D32768,wsize=3D32768",
> >>>>>>> since some network environments have difficulties with 64K.
> >>>>>>>=20
> >>>>>>> There are a few things you can try on the NFSv4 server side,
> >>>>>>> if
> >>>>>>> it
> >>>>>>> appears
> >>>>>>> that the clients are generating a large RPC load.
> >>>>>>> - disabling the DRC cache for TCP by setting
> >>>>>>> vfs.nfsd.cachetcp=3D0
> >>>>>>> - If the server is seeing a large write RPC load, then
> >>>>>>> "sync=3Ddisabled"
> >>>>>>> might help, although it does run a risk of data loss when
> >>>>>>> the
> >>>>>>> server
> >>>>>>> crashes.
> >>>>>>> Then there are a couple of other ZFS related things (I'm not
> >>>>>>> a
> >>>>>>> ZFS
> >>>>>>> guy,
> >>>>>>> but these have shown up on the mailing lists).
> >>>>>>> - make sure your volumes are 4K aligned and ashift=3D12 (in
> >>>>>>> case a
> >>>>>>> drive
> >>>>>>> that uses 4K sectors is pretending to be 512byte sectored)
> >>>>>>> - never run over 70-80% full if write performance is an
> >>>>>>> issue
> >>>>>>> - use a zil on an SSD with good write performance
> >>>>>>>=20
> >>>>>>> The only NFSv4 thing I can tell you is that it is known that
> >>>>>>> ZFS's
> >>>>>>> algorithm for determining sequential vs random I/O fails for
> >>>>>>> NFSv4
> >>>>>>> during writing and this can be a performance hit. The only
> >>>>>>> workaround
> >>>>>>> is to use NFSv3 mounts, since file handle affinity
> >>>>>>> apparently
> >>>>>>> fixes
> >>>>>>> the problem and this is only done for NFSv3.
> >>>>>>>=20
> >>>>>>> rick
> >>>>>>>=20
> >>>>>>> I saw that there are TSO issues with igb then i'm trying to
> >>>>>>> disable
> >>>>>>> it with sysctl but the situation wasn't solved.
> >>>>>>>=20
> >>>>>>> Someone has got ideas ? I can give you more informations if
> >>>>>>> you
> >>>>>>> need.
> >>>>>>>=20
> >>>>>>> Thanks in advance.
> >>>>>>> Regards,
> >>>>>>>=20
> >>>>>>> Lo=C3=AFc Blot,
> >>>>>>> UNIX Systems, Network and Security Engineer
> >>>>>>> http://www.unix-experience.fr
> >>>>>>> _______________________________________________
> >>>>>>> freebsd-fs@freebsd.org mailing list
> >>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >>>>>>> To unsubscribe, send any mail to
> >>>>>>> "freebsd-fs-unsubscribe@freebsd.org"
> >>>>>>>=20
> >>>>>>> _______________________________________________
> >>>>>>> freebsd-fs@freebsd.org mailing list
> >>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >>>>>>> To unsubscribe, send any mail to
> >>>>>>> "freebsd-fs-unsubscribe@freebsd.org"
> >>>>>>>=20
> >>>>>>> _______________________________________________
> >>>>>>> freebsd-fs@freebsd.org mailing list
> >>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >>>>>>> To unsubscribe, send any mail to
> >>>>>>> "freebsd-fs-unsubscribe@freebsd.org"
> >>>>>>>=20
> >>>>>>> _______________________________________________
> >>>>>>> freebsd-fs@freebsd.org mailing list
> >>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >>>>>>> To unsubscribe, send any mail to
> >>>>>>> "freebsd-fs-unsubscribe@freebsd.org"
> >>>>>>> _______________________________________________
> >>>>>>> freebsd-fs@freebsd.org mailing list
> >>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >>>>>>> To unsubscribe, send any mail to
> >>>>>>> "freebsd-fs-unsubscribe@freebsd.org"
> >>>>>>>=20
> >>>>>>> _______________________________________________
> >>>>>>> freebsd-fs@freebsd.org mailing list
> >>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >>>>>>> To unsubscribe, send any mail to
> >>>>>>> "freebsd-fs-unsubscribe@freebsd.org"
> >>>>>=20
> >>>>> _______________________________________________
> >>>>> freebsd-fs@freebsd.org mailing list
> >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >>>>> To unsubscribe, send any mail to
> >>>>> "freebsd-fs-unsubscribe@freebsd.org"
>=20
>=20
>=20
>=20