Date: Mon, 05 Jan 2015 16:34:45 +0000 From: "=?utf-8?B?TG/Dr2MgQmxvdA==?=" <loic.blot@unix-experience.fr> To: "Rick Macklem" <rmacklem@uoguelph.ca> Cc: freebsd-fs@freebsd.org Subject: Re: High Kernel Load with nfsv4 Message-ID: <f3106e9c5d2b715a8dac4e6b6de494a3@mail.unix-experience.fr> In-Reply-To: <956766012.5685731.1420464894657.JavaMail.root@uoguelph.ca> References: <956766012.5685731.1420464894657.JavaMail.root@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Rick,=0Anfsstat -e -s don't show usefull datas on server.=0A=0AServer =
Info:=0A Getattr Setattr Lookup Readlink Read Write Cr=
eate Remove=0A 26935254 16911 5755728 302 2334920 3673=
866 0 328332=0A Rename Link Symlink Mkdir Rmd=
ir Readdir RdirPlus Access=0A 77980 28 0 =
0 3 8900 3 1806052=0A Mknod Fsstat Fsinfo=
PathConf Commit LookupP SetClId SetClIdCf=0A 1 1095 =
0 0 614377 8172 8 8=0A Open =
OpenAttr OpenDwnGr OpenCfrm DelePurge DeleRet GetFH Lock=0A =
1595299 0 44145 1495 0 0 5197490 63=
5015=0A LockT LockU Close Verify NVerify PutFH PutPu=
bFH PutRootFH=0A 0 614919 1270938 0 0 226886=
76 0 5=0A Renew RestoreFH SaveFH Secinfo RelLckOw=
n V4Create=0A 42104 197606 275820 0 143 4578=
=0AServer:=0ARetfailed Faults Clients=0A 0 0 =
6=0AOpenOwner Opens LockOwner Locks Delegs =0A 32335 145=
448 204 181 0 =0AServer Cache Stats:=0A Inprog =
Idem Non-idem Misses CacheSize TCPPeak=0A 0 0 =
1 15082947 60 16522=0A=0AOnly GetAttr and Lookup increase =
and it's only every 4-5 seconds and only +2 to +5 into theses values.=0A=
=0ANow on client, if i take four processes stack i got=0A=0A PID TID =
COMM TDNAME KSTACK =0A63170 1=
02547 mv - mi_switch+0xe1 turnstile_wait+0x4=
2a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65 nfs_lookup+0x3d0 VOP_LOOKUP_=
APV+0xa1 lookup+0x59c namei+0x4d4 vn_open_cred+0x21d kern_openat+0x26f am=
d64_syscall+0x351 Xfast_syscall+0xfb=0A=0AAnother mv:=0A63140 101738 mv =
- mi_switch+0xe1 turnstile_wait+0x42a __mtx_l=
ock_sleep+0x253 nfscl_nodeleg+0x65 nfs_lookup+0x3d0 VOP_LOOKUP_APV+0xa1 l=
ookup+0x59c namei+0x4d4 kern_statat_vnhook+0xae sys_lstat+0x30 amd64_sysc=
all+0x351 Xfast_syscall+0xfb =0A=0A62070 102170 sendmail - =
mi_switch+0xe1 sleepq_timedwait+0x3a _sleep+0x26e clnt_vc_call+0=
x666 clnt_reconnect_call+0x4fa newnfs_request+0xa8c nfscl_request+0x72 nf=
srpc_lookup+0x1fb nfs_lookup+0x508 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei=
+0x4d4 kern_statat_vnhook+0xae sys_lstat+0x30 amd64_syscall+0x351 Xfast_s=
yscall+0xfb=0A=0A63200 100930 mv - mi_switch=
+0xe1 turnstile_wait+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65 nfs_=
lookup+0x3d0 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4 kern_statat_vnh=
ook+0xae sys_lstat+0x30 amd64_syscall+0x351 Xfast_syscall+0xfb=0A=0AWhen =
client is in this state, server was doing nothing special (procstat -kk)=
=0A=0APID TID COMM TDNAME KSTACK =
=0A 895 100538 nfsd nfsd: master mi_switch+0xe1 s=
leepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+=
0x18b svc_run_internal+0x4a1 svc_run+0x1de nfsrvd_nfsd+0x1ca nfssvc_nfsd+=
0x107 sys_nfssvc+0x9c amd64_syscall+0x351 Xfast_syscall+0xfb =0A 895 100=
568 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals=
+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_t=
hread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100569 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab slee=
pq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_star=
t+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100570 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_si=
g+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork=
_exit+0x9a fork_trampoline+0xe =0A 895 100571 nfsd nfsd: ser=
vice mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_=
wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
fork_trampoline+0xe =0A 895 100572 nfsd nfsd: service mi=
_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0=
x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_tram=
poline+0xe =0A 895 100573 nfsd nfsd: service mi_switch+0x=
e1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_r=
un_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=
=0A 895 100575 nfsd nfsd: service mi_switch+0xe1 sleepq_=
catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_interna=
l+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 =
100576 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_sign=
als+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e sv=
c_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100577 nfs=
d nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab s=
leepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_s=
tart+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100578 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait=
_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb f=
ork_exit+0x9a fork_trampoline+0xe =0A 895 100579 nfsd nfsd: =
service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _=
cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0=
x9a fork_trampoline+0xe =0A 895 100580 nfsd nfsd: service =
mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_si=
g+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_t=
rampoline+0xe =0A 895 100581 nfsd nfsd: service mi_switch=
+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a sv=
c_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+=
0xe =0A 895 100582 nfsd nfsd: service mi_switch+0xe1 slee=
pq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_inte=
rnal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 8=
95 100583 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_s=
ignals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e=
svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100584 =
nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xa=
b sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_threa=
d_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100585 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_w=
ait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0x=
b fork_exit+0x9a fork_trampoline+0xe =0A 895 100586 nfsd nfs=
d: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0x=
f _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exi=
t+0x9a fork_trampoline+0xe =0A 895 100587 nfsd nfsd: service=
mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait=
_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a for=
k_trampoline+0xe =0A 895 100588 nfsd nfsd: service mi_swi=
tch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoli=
ne+0xe =0A 895 100589 nfsd nfsd: service mi_switch+0xe1 s=
leepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_i=
nternal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A=
895 100590 nfsd nfsd: service mi_switch+0xe1 sleepq_catc=
h_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x=
87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 1005=
92 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+=
0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_th=
read_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100593 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleep=
q_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start=
+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100594 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig=
+0xf=20_cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb for=
k_exit+0x9a fork_trampoline+0xe =0A 895 100595 nfsd nfsd: se=
rvice mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv=
_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9=
a fork_trampoline+0xe =0A 895 100596 nfsd nfsd: service m=
i_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_tra=
mpoline+0xe =0A 895 100597 nfsd nfsd: service mi_switch+0=
xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_=
run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0x=
e =0A 895 100598 nfsd nfsd: service mi_switch+0xe1 sleepq=
_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_intern=
al+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895=
100599 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_sig=
nals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e s=
vc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100600 nf=
sd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab =
sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_=
start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100602 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wai=
t_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb =
fork_exit+0x9a fork_trampoline+0xe =0A 895 100603 nfsd nfsd:=
service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf =
_cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+=
0x9a fork_trampoline+0xe =0A 895 100604 nfsd nfsd: service =
mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_s=
ig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_=
trampoline+0xe =0A 895 100605 nfsd nfsd: service mi_switc=
h+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a s=
vc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline=
+0xe =0A 895 100606 nfsd nfsd: service mi_switch+0xe1 sle=
epq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_int=
ernal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A =
895 100607 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_=
signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87=
e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100608=
nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0x=
ab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thre=
ad_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100609 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_=
wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0=
xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100610 nfsd nf=
sd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0=
xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_ex=
it+0x9a fork_trampoline+0xe =0A 895 100611 nfsd nfsd: servic=
e mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wai=
t_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fo=
rk_trampoline+0xe =0A 895 100612 nfsd nfsd: service mi_sw=
itch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16=
a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampol=
ine+0xe =0A 895 100613 nfsd nfsd: service mi_switch+0xe1 =
sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_=
internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =
=0A 895 100614 nfsd nfsd: service mi_switch+0xe1 sleepq_c=
atch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal=
+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 1=
00615 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signa=
ls+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc=
_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100617 nfsd=
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sl=
eepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_st=
art+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100618 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_=
sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fo=
rk_exit+0x9a fork_trampoline+0xe =0A 895 100619 nfsd nfsd: s=
ervice mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _c=
v_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x=
9a fork_trampoline+0xe =0A 895 100621 nfsd nfsd: service =
mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig=
+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_tr=
ampoline+0xe =0A 895 100622 nfsd nfsd: service mi_switch+=
0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0=
xe =0A 895 100623 nfsd nfsd: service mi_switch+0xe1 sleep=
q_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_inter=
nal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 89=
5 100624 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_si=
gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e =
svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100625 n=
fsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab=
sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread=
_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100626 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wa=
it_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb=
fork_exit+0x9a fork_trampoline+0xe =0A 895 100627 nfsd nfsd=
: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf=
_cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit=
+0x9a fork_trampoline+0xe =0A 895 100628 nfsd nfsd: service =
mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_=
sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork=
_trampoline+0xe =0A 895 100629 nfsd nfsd: service mi_swit=
ch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a =
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampolin=
e+0xe =0A 895 100630 nfsd nfsd: service mi_switch+0xe1 sl=
eepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_in=
ternal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A =
895 100631 nfsd nfsd: service mi_switch+0xe1 sleepq_catch=
_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x8=
7e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 10063=
2 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thr=
ead_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100633 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq=
_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+=
0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100634 nfsd n=
fsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+=
0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_e=
xit+0x9a fork_trampoline+0xe =0A 895 100635 nfsd nfsd: servi=
ce mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wa=
it_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a f=
ork_trampoline+0xe =0A 895 100636 nfsd nfsd: service mi_s=
witch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1=
6a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampo=
line+0xe =0A 895 100638 nfsd=20=20=20=20 nfsd: service mi_swi=
tch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoli=
ne+0xe =0A 895 100639 nfsd nfsd: service mi_switch+0xe1 s=
leepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_i=
nternal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A=
895 100640 nfsd nfsd: service mi_switch+0xe1 sleepq_catc=
h_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x=
87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 1006=
41 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+=
0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_th=
read_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100642 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleep=
q_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start=
+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100643 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig=
+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_=
exit+0x9a fork_trampoline+0xe =0A 895 100644 nfsd nfsd: serv=
ice mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_w=
ait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a =
fork_trampoline+0xe =0A 895 100645 nfsd nfsd: service mi_=
switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x=
16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_tramp=
oline+0xe =0A 895 100646 nfsd nfsd: service mi_switch+0xe=
1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_ru=
n_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =
=0A 895 100647 nfsd nfsd: service mi_switch+0xe1 sleepq_c=
atch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal=
+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 1=
00648 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signa=
ls+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc=
_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100649 nfsd=
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sl=
eepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_st=
art+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100651 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_=
sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fo=
rk_exit+0x9a fork_trampoline+0xe =0A 895 100652 nfsd nfsd: s=
ervice mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _c=
v_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x=
9a fork_trampoline+0xe =0A 895 100653 nfsd nfsd: service =
mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig=
+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_tr=
ampoline+0xe =0A 895 100654 nfsd nfsd: service mi_switch+=
0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0=
xe =0A 895 100655 nfsd nfsd: service mi_switch+0xe1 sleep=
q_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_inter=
nal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 89=
5 100656 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_si=
gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e =
svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100657 n=
fsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab=
sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread=
_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100658 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wa=
it_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb=
fork_exit+0x9a fork_trampoline+0xe =0A 895 100659 nfsd nfsd=
: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf=
_cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit=
+0x9a fork_trampoline+0xe =0A 895 100661 nfsd nfsd: service =
mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_=
sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork=
_trampoline+0xe =0A 895 100662 nfsd nfsd: service mi_swit=
ch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a =
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampolin=
e+0xe =0A 895 100684 nfsd nfsd: service mi_switch+0xe1 sl=
eepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_in=
ternal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A =
895 100685 nfsd nfsd: service mi_switch+0xe1 sleepq_catch=
_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x8=
7e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 10068=
6 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thr=
ead_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100797 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq=
_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+=
0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100798 nfsd n=
fsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+=
0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_e=
xit+0x9a fork_trampoline+0xe =0A 895 100799 nfsd nfsd: servi=
ce mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wa=
it_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a f=
ork_trampoline+0xe =0A 895 100800 nfsd nfsd: service mi_s=
witch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1=
6a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampo=
line+0xe =0A 895 100801 nfsd nfsd: service mi_switch+0xe1=
sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run=
_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=
=0A=0AI really think it's a client side problem, maybe a lookup problem.=
=0A=0ARegards,=0A=0ALo=C3=AFc Blot,=0AUNIX Systems, Network and Security =
Engineer=0Ahttp://www.unix-experience.fr=0A=0A5 janvier 2015 14:35 "Rick =
Macklem" <rmacklem@uoguelph.ca> a =C3=A9crit: =0A> Loic Blot wrote:=0A> =
=0A>> Hi,=0A>> happy new year Rick and @freebsd-fs.=0A>> =0A>> After some=
days, i looked my NFSv4.1 mount. At server start it was=0A>> calm, but a=
fter 4 days, here is the top stat...=0A>> =0A>> CPU: 0.0% user, 0.0% nice=
, 100% system, 0.0% interrupt, 0.0%=0A>> idle=0A>> =0A>> Definitively i t=
hink it's a problem on client side. What can i look=0A>> into running ker=
nel to resolve this issue ?=0A> =0A> Well, I'd start with:=0A> # nfsstat =
-e -s=0A> - run repeatedly on the server (once every N seconds in a loop)=
.=0A> Then look at the output, comparing the counts and see which RPCs=0A=
> are being performed by the client(s). You are looking for which=0A> RPC=
s are being done a lot. (If one RPC is almost 100% of the load,=0A> then =
it might be a client/caching issue for whatever that RPC is=0A> doing.)=
=0A> =0A> Also look at the Open/Lock counts near the end of the output.=
=0A> If the # of Opens/Locks is large, it may be possible to reduce the=
=0A> CPU overheads by using larger hash tables.=0A> =0A> Then you need to=
profile the server kernel to see where the CPU=0A> is being used.=0A> Ho=
pefully someone else can fill you in on how to do that, because=0A> I'll =
admit I don't know how to.=0A> Basically you are looking to see if the CP=
U is being used in=0A> the NFS server code or ZFS.=0A> =0A> Good luck wit=
h it, rick=0A> =0A>> Regards,=0A>> =0A>> Lo=C3=AFc Blot,=0A>> UNIX System=
s, Network and Security Engineer=0A>> http://www.unix-experience.fr=0A>>; =
=0A>> 30 d=C3=A9cembre 2014 16:16 "Lo=C3=AFc Blot" <loic.blot@unix-experi=
ence.fr> a=0A>> =C3=A9crit:=0A>>> Hi Rick,=0A>>> i upgraded my jail host =
from FreeBSD 9.3 to 10.1 to use NFS v4.1=0A>>> (mountoptions:=0A>>> rw,rs=
ize=3D32768,wsize=3D32768,tcp,nfsv4,minorversion=3D1)=0A>>> =0A>>> Perfor=
mance is quite stable but it's slow. Not as slow as before=0A>>> but slow=
... services was launched=0A>>> but no client are using them and system C=
PU % was 10-50%.=0A>>> =0A>>> I don't see anything on NFSv4.1 server, it'=
s perfectly stable and=0A>>> functionnal.=0A>>> =0A>>> Regards,=0A>>> =0A=
>>> Lo=C3=AFc Blot,=0A>>> UNIX Systems, Network and Security Engineer=0A>=
>> http://www.unix-experience.fr=0A>>>; =0A>>> 23 d=C3=A9cembre 2014 00:20=
"Rick Macklem" <rmacklem@uoguelph.ca> a=0A>>> =C3=A9crit:=0A>>> =0A>>>> =
Loic Blot wrote:=0A>>>> =0A>>>>> Hi,=0A>>>>> =0A>>>>> To clarify because =
of our exchanges. Here are the current sysctl=0A>>>>> options for server:=
=0A>>>>> =0A>>>>> vfs.nfsd.enable_nobodycheck=3D0=0A>>>>> vfs.nfsd.enable=
_nogroupcheck=3D0=0A>>>>> =0A>>>>> vfs.nfsd.maxthreads=3D200=0A>>>>> vfs.=
nfsd.tcphighwater=3D10000=0A>>>>> vfs.nfsd.tcpcachetimeo=3D300=0A>>>>> vf=
s.nfsd.server_min_nfsvers=3D4=0A>>>>> =0A>>>>> kern.maxvnodes=3D10000000=
=0A>>>>> kern.ipc.maxsockbuf=3D4194304=0A>>>>> net.inet.tcp.sendbuf_max=
=3D4194304=0A>>>>> net.inet.tcp.recvbuf_max=3D4194304=0A>>>>> =0A>>>>> vf=
s.lookup_shared=3D0=0A>>>>> =0A>>>>> Regards,=0A>>>>> =0A>>>>> Lo=C3=AFc =
Blot,=0A>>>>> UNIX Systems, Network and Security Engineer=0A>>>>> http://=
www.unix-experience.fr=0A>>>>> =0A>>>>> 22 d=C3=A9cembre 2014 09:42 "Lo=
=C3=AFc Blot" <loic.blot@unix-experience.fr>=0A>>>>> a=0A>>>>> =C3=A9crit=
:=0A>>>>> =0A>>>>> Hi Rick,=0A>>>>> my 5 jails runs this weekend and now =
i have some stats on this=0A>>>>> monday.=0A>>>>> =0A>>>>> Hopefully dead=
lock was fixed, yeah, but everything isn't good :(=0A>>>>> =0A>>>>> On NF=
Sv4 server (FreeBSD 10.1) system uses 35% CPU=0A>>>>> =0A>>>>> As i can s=
ee this is because of nfsd:=0A>>>>> =0A>>>>> 918 root 96 20 0 12352K 3372=
K rpcsvc 6 51.4H=0A>>>>> 273.68% nfsd: server (nfsd)=0A>>>>> =0A>>>>> If =
i look at dmesg i see:=0A>>>>> nfsd server cache flooded, try increasing =
vfs.nfsd.tcphighwater=0A>>>> =0A>>>> Well, you have a couple of choices:=
=0A>>>> 1 - Use NFSv4.1 (add "minorversion=3D1" to your mount options).=
=0A>>>> (NFSv4.1 avoids use of the DRC and instead uses something=0A>>>> =
called sessions. See below.)=0A>>>> OR=0A>>>> =0A>>>>> vfs.nfsd.tcphighwa=
ter was set to 10000, i increase it to 15000=0A>>>> =0A>>>> 2 - Bump vfs.=
nfsd.tcphighwater way up, until you no longer see=0A>>>> "nfs server cach=
e flooded" messages. (I think Garrett Wollman uses=0A>>>> 100000. (You ma=
y still see quite a bit of CPU overheads.)=0A>>>> =0A>>>> OR=0A>>>> =0A>>=
>> 3 - Set vfs.nfsd.cachetcp=3D0 (which disables the DRC and gets rid=0A>=
>>> of the CPU overheads). However, there is a risk of data corruption=0A=
>>>> if you have a client->server network partitioning of a moderate=0A>>=
>> duration, because a non-idempotent RPC may get redone, becasue=0A>>>> =
the client times out waiting for a reply. If a non-idempotent=0A>>>> RPC =
gets done twice on the server, data corruption can happen.=0A>>>> (The DR=
C provides improved correctness, but does add overhead.)=0A>>>> =0A>>>> I=
f #1 works for you, it is the preferred solution, since Sessions=0A>>>> i=
n NFSv4.1 solves the correctness problem in a good, space bound=0A>>>> wa=
y. A session basically has N (usually 32 or 64) slots and only=0A>>>> all=
ows one outstanding RPC/slot. As such, it can cache the=0A>>>> previous=
=0A>>>> reply for each slot (32 or 64 of them) and guarantee "exactly=0A>=
>>> once"=0A>>>> RPC semantics.=0A>>>> =0A>>>> rick=0A>>>> =0A>>>>> Here =
is 'nfsstat -s' output:=0A>>>>> =0A>>>>> Server Info:=0A>>>>> Getattr Set=
attr Lookup Readlink Read Write Create=0A>>>>> Remove=0A>>>>> 12600652 18=
12 2501097 156 1386423 1983729 123=0A>>>>> 162067=0A>>>>> Rename Link Sym=
link Mkdir Rmdir Readdir RdirPlus=0A>>>>> Access=0A>>>>> 36762 9 0 0 0 31=
47 0=0A>>>>> 623524=0A>>>>> Mknod Fsstat Fsinfo PathConf Commit=0A>>>>> 0=
0 0 0 328117=0A>>>>> Server Ret-Failed=0A>>>>> 0=0A>>>>> Server Faults=
=0A>>>>> 0=0A>>>>> Server Cache Stats:=0A>>>>> Inprog Idem Non-idem Misse=
s=0A>>>>> 0 0 0 12635512=0A>>>>> Server Write Gathering:=0A>>>>> WriteOps=
WriteRPC Opsaved=0A>>>>> 1983729 1983729 0=0A>>>>> =0A>>>>> And here is =
'procstat -kk' for nfsd (server)=0A>>>>> =0A>>>>> 918 100528 nfsd nfsd: m=
aster mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_timedwait_s=
ig+0x10=0A>>>>> _cv_timedwait_sig_sbt+0x18b svc_run_internal+0x4a1 svc_ru=
n+0x1de=0A>>>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c=0A>>>=
>> amd64_syscall+0x351 Xfast_syscall+0xfb=0A>>>>> 918 100568 nfsd nfsd: s=
ervice mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0=
xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb=
fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100569 nfsd nfsd:=
service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig=
+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0=
xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100570 nfsd nfs=
d: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_s=
ig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start=
+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100571 nfsd n=
fsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait=
_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_sta=
rt+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100572 nfsd=
nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wa=
it_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_s=
tart+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100573 nf=
sd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_=
wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread=
_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100574 =
nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleep=
q_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thre=
ad_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 10057=
5 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sle=
epq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_th=
read_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100=
576 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab s=
leepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_=
thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 1=
00577 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab=
sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e sv=
c_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918=
100578 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0x=
ab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e =
svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 9=
18 100579 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+=
0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87=
e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>>=
918 100580 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signal=
s+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x=
87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>=
>> 918 100581 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_sign=
als+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+=
0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>=
>>>> 918 100582 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_si=
gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_interna=
l+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=
=0A>>>>> 918 100583 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc=
h_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_int=
ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0=
xe=0A>>>>> 918 100584 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_ca=
tch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_i=
nternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline=
+0xe=0A>>>>> 918 100585 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_=
catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run=
_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoli=
ne+0xe=0A>>>>> 918 100586 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleep=
q_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_r=
un_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampo=
line+0xe=0A>>>>> 918 100587 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sle=
epq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram=
poline+0xe=0A>>>>> 918 100588 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s=
leepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> s=
vc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tr=
ampoline+0xe=0A>>>>> 918 100589 nfsd nfsd: service mi_switch+0xe1=0A>>>>>=
sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>>=
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_=
trampoline+0xe=0A>>>>> 918 100590 nfsd nfsd: service mi_switch+0xe1=0A>>>=
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>=
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> for=
k_trampoline+0xe=0A>>>>> 918 100591 nfsd nfsd: service mi_switch+0xe1=0A>=
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>=
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> f=
ork_trampoline+0xe=0A>>>>> 918 100592 nfsd nfsd: service mi_switch+0xe1=
=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=
=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>=
>> fork_trampoline+0xe=0A>>>>> 918 100593 nfsd nfsd: service mi_switch+0x=
e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1=
6a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>=
>>>> fork_trampoline+0xe=0A>>>>> 918 100594 nfsd nfsd: service mi_switch+=
0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0=
x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100595 nfsd nfsd: service mi_swi=
tch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_s=
ig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x=
9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100596 nfsd nfsd: service mi_s=
witch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait=
_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+=
0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100597 nfsd nfsd: service mi=
_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wa=
it_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exi=
t+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100598 nfsd nfsd: service =
mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_=
wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e=
xit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100599 nfsd nfsd: servic=
e mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _c=
v_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork=
_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100600 nfsd nfsd: serv=
ice mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf =
_cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fo=
rk_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100601 nfsd nfsd: se=
rvice mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0x=
f _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb =
fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100602 nfsd nfsd: =
service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+=
0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0x=
b fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100603 nfsd nfsd=
: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_si=
g+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+=
0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100604 nfsd nf=
sd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_=
sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_star=
t+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100605 nfsd =
nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wai=
t_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_st=
art+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100606 nfs=
d nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_w=
ait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_=
start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100607 n=
fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq=
_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_threa=
d_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100608=
nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab slee=
pq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thr=
ead_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 1006=
09 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sl=
eepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_t=
hread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 10=
0610 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab =
sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc=
_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 =
100611 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xa=
b sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s=
vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 91=
8 100612 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e=
svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> =
918 100613 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals=
+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x8=
7e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>=
> 918 100614 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signa=
ls+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0=
x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>=
>>> 918 100615 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_sig=
nals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal=
+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A=
>>>>> 918 100616 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_s=
ignals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_intern=
al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=
=0A>>>>> 918 100617 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc=
h_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_int=
ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0=
xe=0A>>>>> 918 100618 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_ca=
tch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_i=
nternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline=
+0xe=0A>>>>> 918 100619 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_=
catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run=
_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoli=
ne+0xe=0A>>>>> 918 100620 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleep=
q_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_r=
un_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampo=
line+0xe=0A>>>>> 918 100621 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sle=
epq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram=
poline+0xe=0A>>>>> 918 100622 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s=
leepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> s=
vc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tr=
ampoline+0xe=0A>>>>> 918 100623 nfsd nfsd: service mi_switch+0xe1=0A>>>>>=
sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>>=
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_=
trampoline+0xe=0A>>>>> 918 100624 nfsd nfsd: service mi_switch+0xe1=0A>>>=
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>=
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> for=
k_trampoline+0xe=0A>>>>> 918 100625 nfsd nfsd: service mi_switch+0xe1=0A>=
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>=
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> f=
ork_trampoline+0xe=0A>>>>> 918 100626 nfsd nfsd: service mi_switch+0xe1=
=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=
=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>=
>> fork_trampoline+0xe=0A>>>>> 918 100627 nfsd nfsd: service mi_switch+0x=
e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1=
6a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>=
>>>> fork_trampoline+0xe=0A>>>>> 918 100628 nfsd nfsd: service mi_switch+=
0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0=
x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100629 nfsd nfsd: service mi_swi=
tch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_s=
ig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x=
9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100630 nfsd nfsd: service mi_s=
witch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait=
_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+=
0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100631 nfsd nfsd: service mi=
_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wa=
it_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exi=
t+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100632 nfsd nfsd: service =
mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_=
wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e=
xit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100633 nfsd nfsd: servic=
e mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _c=
v_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork=
_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100634 nfsd nfsd: serv=
ice mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf =
_cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fo=
rk_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100635 nfsd nfsd: se=
rvice mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0x=
f _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb =
fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100636 nfsd nfsd: =
service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+=
0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0x=
b fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100637 nfsd nfsd=
: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_si=
g+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+=
0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100638 nfsd nf=
sd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_=
sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_star=
t+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100639 nfsd =
nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wai=
t_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_st=
art+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100640 nfs=
d nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_w=
ait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_=
start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100641 n=
fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq=
_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_threa=
d_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100642=
nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab slee=
pq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thr=
ead_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 1006=
43 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sl=
eepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_t=
hread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 10=
0644 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab =
sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc=
_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 =
100645 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xa=
b sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s=
vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 91=
8 100646 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e=
svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> =
918 100647 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals=
+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x8=
7e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>=
> 918 100648 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signa=
ls+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0=
x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>=
>>> 918 100649 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_sig=
nals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal=
+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A=
>>>>> 918 100650 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_s=
ignals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_intern=
al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=
=0A>>>>> 918 100651 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc=
h_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_int=
ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0=
xe=0A>>>>> 918 100652 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_ca=
tch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_i=
nternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline=
+0xe=0A>>>>> 918 100653 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_=
catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run=
_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoli=
ne+0xe=0A>>>>> 918 100654 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleep=
q_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_r=
un_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampo=
line+0xe=0A>>>>> 918 100655 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sle=
epq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram=
poline+0xe=0A>>>>> 918 100656 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s=
leepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> s=
vc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tr=
ampoline+0xe=0A>>>>> 918 100657 nfsd nfsd: service mi_switch+0xe1=0A>>>>>=
sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>>=
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_=
trampoline+0xe=0A>>>>> 918 100658 nfsd nfsd: service mi_switch+0xe1=0A>>>=
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>=
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> for=
k_trampoline+0xe=0A>>>>> 918 100659 nfsd nfsd: service mi_switch+0xe1=0A>=
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>=
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> f=
ork_trampoline+0xe=0A>>>>> 918 100660 nfsd nfsd: service mi_switch+0xe1=
=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=
=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>=
>> fork_trampoline+0xe=0A>>>>> 918 100661 nfsd nfsd: service mi_switch+0x=
e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1=
6a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>=
>>>> fork_trampoline+0xe=0A>>>>> 918 100662 nfsd nfsd: service mi_switch+=
0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0=
x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>>> fork_trampoline+0xe=0A>>>>> ---=0A>>>>> =0A>>>>> Now if we look =
at client (FreeBSD 9.3)=0A>>>>> =0A>>>>> We see system was very busy and =
do many and many interrupts=0A>>>>> =0A>>>>> CPU: 0.0% user, 0.0% nice, 3=
7.8% system, 51.2% interrupt, 11.0%=0A>>>>> idle=0A>>>>> =0A>>>>> A look =
at process list shows that there are many sendmail process=0A>>>>> in=0A>=
>>>> state nfstry=0A>>>>> =0A>>>>> nfstry 18 32:27 0.88% sendmail: Queue =
runner@00:30:00 for=0A>>>>> /var/spool/clientm=0A>>>>> =0A>>>>> Here is '=
nfsstat -c' output:=0A>>>>> =0A>>>>> Client Info:=0A>>>>> Rpc Counts:=0A>=
>>>> Getattr Setattr Lookup Readlink Read Write Create=0A>>>>> Remove=0A>=
>>>> 1051347 1724 2494481 118 903902 1901285 162676=0A>>>>> 161899=0A>>>>=
> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus=0A>>>>> Access=0A>>>>>=
36744 2 0 114 40 3131 0=0A>>>>> 544136=0A>>>>> Mknod Fsstat Fsinfo PathC=
onf Commit=0A>>>>> 9 0 0 0 245821=0A>>>>> Rpc Info:=0A>>>>> TimedOut Inva=
lid X Replies Retries Requests=0A>>>>> 0 0 0 0 8356557=0A>>>>> Cache Info=
:=0A>>>>> Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits=0A=
>>>>> Misses=0A>>>>> 108754455 491475 54229224 2437229 46814561 821723 51=
32123=0A>>>>> 1871871=0A>>>>> BioRLHits Misses BioD Hits Misses DirE Hits=
Misses Accs Hits=0A>>>>> Misses=0A>>>>> 144035 118 53736 2753 27813 1 57=
238839=0A>>>>> 544205=0A>>>>> =0A>>>>> If you need more things, tell me, =
i let the PoC in this state.=0A>>>>> =0A>>>>> Thanks=0A>>>>> =0A>>>>> Reg=
ards,=0A>>>>> =0A>>>>> Lo=C3=AFc Blot,=0A>>>>> UNIX Systems, Network and =
Security Engineer=0A>>>>> http://www.unix-experience.fr=0A>>>>>; =0A>>>>> =
21 d=C3=A9cembre 2014 01:33 "Rick Macklem" <rmacklem@uoguelph.ca> a=0A>>>=
>> =C3=A9crit:=0A>>>>> =0A>>>>> Loic Blot wrote:=0A>>>>> =0A>>>>>> Hi Ric=
k,=0A>>>>>> ok, i don't need locallocks, i haven't understand option was =
for=0A>>>>>> that=0A>>>>>> usage, i removed it.=0A>>>>>> I do more tests =
on monday.=0A>>>>>> Thanks for the deadlock fix, for other people :)=0A>>=
>>> =0A>>>>> Good. Please let us know if running with=0A>>>>> vfs.nfsd.en=
able_locallocks=3D0=0A>>>>> gets rid of the deadlocks? (I think it fixes =
the one you saw.)=0A>>>>> =0A>>>>> On the performance side, you might als=
o want to try different=0A>>>>> values=0A>>>>> of=0A>>>>> readahead, if t=
he Linux client has such a mount option. (With the=0A>>>>> NFSv4-ZFS sequ=
ential vs random I/O heuristic, I have no idea what=0A>>>>> the=0A>>>>> o=
ptimal readahead value would be.)=0A>>>>> =0A>>>>> Good luck with it and =
please let us know how it goes, rick=0A>>>>> ps: I now have a patch to fi=
x the deadlock when=0A>>>>> vfs.nfsd.enable_locallocks=3D1=0A>>>>> is set=
. I'll post it for anyone who is interested after I put it=0A>>>>> throug=
h some testing.=0A>>>>> =0A>>>>> --=0A>>>>> Best regards,=0A>>>>> Lo=C3=
=AFc BLOT,=0A>>>>> UNIX systems, security and network engineer=0A>>>>> ht=
tp://www.unix-experience.fr=0A>>>>> =0A>>>>> Le jeudi 18 d=C3=A9cembre 20=
14 =C3=A0 19:46 -0500, Rick Macklem a =C3=A9crit :=0A>>>>> =0A>>>>> Loic =
Blot wrote:=0A>>>>>> Hi rick,=0A>>>>>> i tried to start a LXC container o=
n Debian Squeeze from my=0A>>>>>> freebsd=0A>>>>>> ZFS+NFSv4 server and i=
also have a deadlock on nfsd=0A>>>>>> (vfs.lookup_shared=3D0). Deadlock =
procs each time i launch a=0A>>>>>> squeeze=0A>>>>>> container, it seems =
(3 tries, 3 fails).=0A>>>>> =0A>>>>> Well, I`ll take a look at this `proc=
stat -kk`, but the only thing=0A>>>>> I`ve seen posted w.r.t. avoiding de=
adlocks in ZFS is to not use=0A>>>>> nullfs. (I have no idea if you are u=
sing any nullfs mounts, but=0A>>>>> if so, try getting rid of them.)=0A>>=
>>> =0A>>>>> Here`s a high level post about the ZFS and vnode locking pro=
blem,=0A>>>>> but there is no patch available, as far as I know.=0A>>>>> =
=0A>>>>> http://docs.FreeBSD.org/cgi/mid.cgi?54739F41.8030407=0A>>>>>; =0A=
>>>>> rick=0A>>>>> =0A>>>>> 921 - D 0:00.02 nfsd: server (nfsd)=0A>>>>> =
=0A>>>>> Here is the procstat -kk=0A>>>>> =0A>>>>> PID TID COMM TDNAME KS=
TACK=0A>>>>> 921 100538 nfsd nfsd: master mi_switch+0xe1=0A>>>>> sleepq_w=
ait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e=0A>>>>> vop_stdlock+0x3c VOP_=
LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> nfsvno_advlock+0x119 nfsrv_dolocal+0=
x84 nfsrv_lockctrl+0x14ad=0A>>>>> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 n=
fssvc_program+0x554=0A>>>>> svc_run_internal+0xc77 svc_run+0x1de nfsrvd_n=
fsd+0x1ca=0A>>>>> nfssvc_nfsd+0x107 sys_nfssvc+0x9c=0A>>>>> 921 100572 nf=
sd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_=
wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e sv=
c_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921=
100573 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0x=
ab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_interna=
l+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=
=0A>>>>> 921 100574 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc=
h_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram=
poline+0xe=0A>>>>> 921 100575 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s=
leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=
=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>=
>> fork_trampoline+0xe=0A>>>>> 921 100576 nfsd nfsd: service mi_switch+0x=
e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait=
_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+=
0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100577 nfsd nfsd: service mi=
_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>=
> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb =
fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100578 nfsd nfsd: =
service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+=
0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_=
start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100579 n=
fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq=
_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s=
vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 92=
1 100580 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_intern=
al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=
=0A>>>>> 921 100581 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc=
h_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram=
poline+0xe=0A>>>>> 921 100582 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s=
leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=
=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>=
>> fork_trampoline+0xe=0A>>>>> 921 100583 nfsd nfsd: service mi_switch+0x=
e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait=
_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+=
0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100584 nfsd nfsd: service mi=
_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>=
> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb =
fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100585 nfsd nfsd: =
service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+=
0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_=
start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100586 n=
fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq=
_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s=
vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 92=
1 100587 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_intern=
al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=
=0A>>>>> 921 100588 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc=
h_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram=
poline+0xe=0A>>>>> 921 100589 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s=
leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=
=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>=
>> fork_trampoline+0xe=0A>>>>> 921 100590 nfsd nfsd: service mi_switch+0x=
e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait=
_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+=
0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100591 nfsd nfsd: service mi=
_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>=
> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb =
fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100592 nfsd nfsd: =
service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+=
0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_=
start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100593 n=
fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq=
_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s=
vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 92=
1 100594 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_intern=
al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=
=0A>>>>> 921 100595 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc=
h_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram=
poline+0xe=0A>>>>> 921 100596 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s=
leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=
=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>=
>> fork_trampoline+0xe=0A>>>>> 921 100597 nfsd nfsd: service mi_switch+0x=
e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait=
_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+=
0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100598 nfsd nfsd: service mi=
_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>=
> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb =
fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100599 nfsd nfsd: =
service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+=
0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_=
start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100600 n=
fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq=
_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s=
vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 92=
1 100601 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_intern=
al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=
=0A>>>>> 921 100602 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc=
h_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram=
poline+0xe=0A>>>>> 921 100603 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s=
leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=
=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>=
>> fork_trampoline+0xe=0A>>>>> 921 100604 nfsd nfsd: service mi_switch+0x=
e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait=
_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+=
0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100605 nfsd nfsd: service mi=
_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>=
> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb =
fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100606 nfsd nfsd: =
service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+=
0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_=
start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100607 n=
fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq=
_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s=
vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 92=
1 100608 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_intern=
al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=
=0A>>>>> 921 100609 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc=
h_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram=
poline+0xe=0A>>>>> 921 100610 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s=
leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=
=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>=
>> fork_trampoline+0xe=0A>>>>> 921 100611 nfsd nfsd: service mi_switch+0x=
e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait=
_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+=
0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100612 nfsd nfsd: service mi=
_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>=
> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb =
fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100613 nfsd nfsd: =
service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+=
0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_=
start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100614 n=
fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq=
_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s=
vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 92=
1 100615 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_intern=
al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=
=0A>>>>> 921 100616 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait=
+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>> nfsrv_getlockfi=
le+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1=0A>>>>> nfsrvd_dorpc+0xec=
6 nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0x=
b fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 921 100617 nfsd nfsd: servic=
e mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A=
>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+=
0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100618 nfsd nf=
sd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmslee=
p+0x66 nfsv4_lock+0x9b=0A>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 sv=
c_run_internal+0xc77=0A>>>>> svc_thread_start+0xb fork_exit+0x9a fork_tra=
mpoline+0xe=0A>>>>> 921 100619 nfsd nfsd: service mi_switch+0xe1=0A>>>>> =
sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=
=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>=
>> fork_trampoline+0xe=0A>>>>> 921 100620 nfsd nfsd: service mi_switch+0x=
e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait=
_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+=
0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100621 nfsd nfsd: service mi=
_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>=
> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb =
fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100622 nfsd nfsd: =
service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+=
0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_=
start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100623 n=
fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq=
_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s=
vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 92=
1 100624 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_intern=
al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=
=0A>>>>> 921 100625 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc=
h_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram=
poline+0xe=0A>>>>> 921 100626 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s=
leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=
=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>=
>> fork_trampoline+0xe=0A>>>>> 921 100627 nfsd nfsd: service mi_switch+0x=
e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait=
_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+=
0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100628 nfsd nfsd: service mi=
_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>=
> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb =
fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100629 nfsd nfsd: =
service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+=
0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_=
start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100630 n=
fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq=
_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s=
vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 92=
1 100631 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_intern=
al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=
=0A>>>>> 921 100632 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc=
h_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram=
poline+0xe=0A>>>>> 921 100633 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s=
leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=
=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>=
>> fork_trampoline+0xe=0A>>>>> 921 100634 nfsd nfsd: service mi_switch+0x=
e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait=
_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+=
0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100635 nfsd nfsd: service mi=
_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>=
> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb =
fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100636 nfsd nfsd: =
service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+=
0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_=
start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100637 n=
fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq=
_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s=
vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 92=
1 100638 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_intern=
al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=
=0A>>>>> 921 100639 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc=
h_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram=
poline+0xe=0A>>>>> 921 100640 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s=
leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=
=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>=
>> fork_trampoline+0xe=0A>>>>> 921 100641 nfsd nfsd: service mi_switch+0x=
e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait=
_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+=
0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100642 nfsd nfsd: service mi=
_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>=
> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb =
fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100643 nfsd nfsd: =
service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+=
0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_=
start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100644 n=
fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq=
_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s=
vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 92=
1 100645 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_intern=
al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=
=0A>>>>> 921 100646 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc=
h_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram=
poline+0xe=0A>>>>> 921 100647 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s=
leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=
=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>=
>> fork_trampoline+0xe=0A>>>>> 921 100648 nfsd nfsd: service mi_switch+0x=
e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait=
_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+=
0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100649 nfsd nfsd: service mi=
_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>=
> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb =
fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100650 nfsd nfsd: =
service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+=
0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_=
start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100651 n=
fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq=
_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s=
vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 92=
1 100652 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_intern=
al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=
=0A>>>>> 921 100653 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc=
h_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram=
poline+0xe=0A>>>>> 921 100654 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s=
leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=
=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>=
>> fork_trampoline+0xe=0A>>>>> 921 100655 nfsd nfsd: service mi_switch+0x=
e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait=
_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+=
0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100656 nfsd nfsd: service mi=
_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>=
> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb =
fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100657 nfsd nfsd: =
service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+=
0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_=
start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100658 n=
fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq=
_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s=
vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 92=
1 100659 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_intern=
al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=
=0A>>>>> 921 100660 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc=
h_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram=
poline+0xe=0A>>>>> 921 100661 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s=
leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=
=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>=
>> fork_trampoline+0xe=0A>>>>> 921 100662 nfsd nfsd: service mi_switch+0x=
e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait=
_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+=
0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100663 nfsd nfsd: service mi=
_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>=
> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb =
fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100664 nfsd nfsd: =
service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+=
0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_=
start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100665 n=
fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq=
_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s=
vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 92=
1 100666 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a _slee=
p+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>> nfsrv_setclient+0xbd nfsrv=
d_setclientid+0x3c8=0A>>>>> nfsrvd_dorpc+0xc76=0A>>>>> nfssvc_program+0x5=
54 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+=
0x9a fork_trampoline+0xe=0A>>>>> =0A>>>>> Regards,=0A>>>>> =0A>>>>> Lo=C3=
=AFc Blot,=0A>>>>> UNIX Systems, Network and Security Engineer=0A>>>>> ht=
tp://www.unix-experience.fr=0A>>>>> =0A>>>>> 15 d=C3=A9cembre 2014 15:18 =
"Rick Macklem" <rmacklem@uoguelph.ca> a=0A>>>>> =C3=A9crit:=0A>>>>> =0A>>=
>>> Loic Blot wrote:=0A>>>>> =0A>>>>>> For more informations, here is pro=
cstat -kk on nfsd, if you=0A>>>>>> need=0A>>>>>> more=0A>>>>>> hot datas,=
tell me.=0A>>>>>> =0A>>>>>> Regards, PID TID COMM TDNAME KSTACK=0A>>>>>>=
918 100529 nfsd nfsd: master mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sl=
eeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+=
0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run=
_internal+0xc77 svc_run+0x1de=0A>>>>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x10=
7 sys_nfssvc+0x9c=0A>>>>>> amd64_syscall+0x351=0A>>>>> =0A>>>>> Well, mos=
t of the threads are stuck like this one, waiting for=0A>>>>> a=0A>>>>> v=
node=0A>>>>> lock in ZFS. All of them appear to be in zfs_fhtovp().=0A>>>=
>> I`m not a ZFS guy, so I can`t help much. I`ll try changing the=0A>>>>>=
subject line=0A>>>>> to include ZFS vnode lock, so maybe the ZFS guys wi=
ll take a=0A>>>>> look.=0A>>>>> =0A>>>>> The only thing I`ve seen suggest=
ed is trying:=0A>>>>> sysctl vfs.lookup_shared=3D0=0A>>>>> to disable sha=
red vop_lookup()s. Apparently zfs_lookup()=0A>>>>> doesn`t=0A>>>>> obey t=
he vnode locking rules for lookup and rename, according=0A>>>>> to=0A>>>>=
> the posting I saw.=0A>>>>> =0A>>>>> I`ve added a couple of comments abo=
ut the other threads below,=0A>>>>> but=0A>>>>> they are all either waiti=
ng for an RPC request or waiting for=0A>>>>> the=0A>>>>> threads stuck on=
the ZFS vnode lock to complete.=0A>>>>> =0A>>>>> rick=0A>>>>> =0A>>>>>> =
918 100564 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_catch_signal=
s+0xab sleepq_wait_sig+0xf=0A>>>>>> _cv_wait_sig+0x16a=0A>>>>>> svc_run_i=
nternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>> fork_trampolin=
e+0xe=0A>>>>> =0A>>>>> Fyi, this thread is just waiting for an RPC to arr=
ive. (Normal)=0A>>>>> =0A>>>>>> 918 100565 nfsd nfsd: service mi_switch+0=
xe1=0A>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>> _cv_w=
ait_sig+0x16a=0A>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e=
xit+0x9a=0A>>>>>> fork_trampoline+0xe=0A>>>>>> 918 100566 nfsd nfsd: serv=
ice mi_switch+0xe1=0A>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=
=0A>>>>>> _cv_wait_sig+0x16a=0A>>>>>> svc_run_internal+0x87e svc_thread_s=
tart+0xb fork_exit+0x9a=0A>>>>>> fork_trampoline+0xe=0A>>>>>> 918 100567 =
nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_catch_signals+0xab slee=
pq_wait_sig+0xf=0A>>>>>> _cv_wait_sig+0x16a=0A>>>>>> svc_run_internal+0x8=
7e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>> fork_trampoline+0xe=0A>>>=
>>> 918 100568 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_catch_si=
gnals+0xab sleepq_wait_sig+0xf=0A>>>>>> _cv_wait_sig+0x16a=0A>>>>>> svc_r=
un_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>> fork_tramp=
oline+0xe=0A>>>>>> 918 100569 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> =
sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>> _cv_wait_sig+0x16a=
=0A>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>=
>>>> fork_trampoline+0xe=0A>>>>>> 918 100570 nfsd nfsd: service mi_switch=
+0xe1=0A>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>> _cv=
_wait_sig+0x16a=0A>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork=
_exit+0x9a=0A>>>>>> fork_trampoline+0xe=0A>>>>>> 918 100571 nfsd nfsd: se=
rvice mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x6=
6 nfsv4_lock+0x9b=0A>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_ru=
n_internal+0xc77=0A>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampo=
line+0xe=0A>>>>>> 918 100572 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> s=
leepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>> nfsr=
v_setclient+0xbd nfsrvd_setclientid+0x3c8=0A>>>>>> nfsrvd_dorpc+0xc76=0A>=
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_sta=
rt+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> =0A>>>>> This =
one (and a few others) are waiting for the nfsv4_lock.=0A>>>>> This=0A>>>=
>> happens=0A>>>>> because other threads are stuck with RPCs in progress.=
(ie. The=0A>>>>> ones=0A>>>>> waiting on the vnode lock in zfs_fhtovp().=
)=0A>>>>> For these, the RPC needs to lock out other threads to do the=0A=
>>>>> operation,=0A>>>>> so it waits for the nfsv4_lock() which can exclu=
sively lock the=0A>>>>> NFSv4=0A>>>>> data structures once all other nfsd=
threads complete their RPCs=0A>>>>> in=0A>>>>> progress.=0A>>>>> =0A>>>>=
>> 918 100573 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a=
_sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>> nfsrvd_dorpc+0x316 =
nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=
fork_exit+0x9a fork_trampoline+0xe=0A>>>>> =0A>>>>> Same as above.=0A>>>=
>> =0A>>>>>> 918 100574 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq=
_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno=
_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+=
0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_=
exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100575 nfsd nfsd: service mi_s=
witch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=
=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_=
fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x=
917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thr=
ead_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 10=
0576 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+=
0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _=
vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_f=
htovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_inter=
nal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_tram=
poline+0xe=0A>>>>>> 918 100577 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>=
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock=
+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>>=
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_p=
rogram+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>=
> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100578 nfsd nfsd: servi=
ce mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+=
0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>=
zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dor=
pc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> sv=
c_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 9=
18 100579 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sle=
eplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0=
xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c n=
fsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_=
internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork=
_trampoline+0xe=0A>>>>>> 918 100580 nfsd nfsd: service mi_switch+0xe1=0A>=
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_st=
dlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>=
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfs=
svc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A=
>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100581 nfsd nfsd: =
service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_=
args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>=
>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrv=
d_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>=
>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>=
>>> 918 100582 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3=
a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_=
APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0=
x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc=
_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a=
fork_trampoline+0xe=0A>>>>>> 918 100583 nfsd nfsd: service mi_switch+0xe=
1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> v=
op_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38=
d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>=
> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0=
xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100584 nfsd n=
fsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __loc=
kmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x4=
3=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 =
nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=
=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0x=
e=0A>>>>>> 918 100585 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_w=
ait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP=
_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_f=
htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x=
554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_ex=
it+0x9a fork_trampoline+0xe=0A>>>>>> 918 100586 nfsd nfsd: service mi_swi=
tch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>=
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhto=
vp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=
=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread=
_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 10058=
7 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x1=
5d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_=
lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhto=
vp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal=
+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampol=
ine+0xe=0A>>>>>> 918 100588 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x=
3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nf=
svno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_prog=
ram+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> f=
ork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100589 nfsd nfsd: service =
mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x9=
02=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zf=
s_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+=
0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_t=
hread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 =
100590 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleepl=
k+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab=
_vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd=
_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_tr=
ampoline+0xe=0A>>>>>> 918 100591 nfsd nfsd: service mi_switch+0xe1=0A>>>>=
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlo=
ck+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>=
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc=
_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>=
>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100592 nfsd nfsd: ser=
vice mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_arg=
s+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>=
>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_d=
orpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> =
svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>=
918 100593 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a s=
leeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV=
+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c=
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_ru=
n_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fo=
rk_trampoline+0xe=0A>>>>>> 918 100594 nfsd nfsd: service mi_switch+0xe1=
=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vo=
p_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=
=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>=
nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0x=
b=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100595 nfsd nf=
sd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lock=
mgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=
=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 n=
fsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A=
>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=
=0A>>>>>> 918 100596 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wa=
it+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_=
LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fh=
tovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x5=
54 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exi=
t+0x9a fork_trampoline+0xe=0A>>>>>> 918 100597 nfsd nfsd: service mi_swit=
ch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>=
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtov=
p+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=
=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread=
_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 10059=
8 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x1=
5d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_=
lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhto=
vp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal=
+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampol=
ine+0xe=0A>>>>>> 918 100599 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x=
3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nf=
svno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_prog=
ram+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> f=
ork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100600 nfsd nfsd: service =
mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x9=
02=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zf=
s_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+=
0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_t=
hread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 =
100601 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleepl=
k+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab=
_vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd=
_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_tr=
ampoline+0xe=0A>>>>>> 918 100602 nfsd nfsd: service mi_switch+0xe1=0A>>>>=
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlo=
ck+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>=
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc=
_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>=
>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100603 nfsd nfsd: ser=
vice mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_arg=
s+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>=
>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_d=
orpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> =
svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>=
918 100604 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a s=
leeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV=
+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c=
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_ru=
n_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fo=
rk_trampoline+0xe=0A>>>>>> 918 100605 nfsd nfsd: service mi_switch+0xe1=
=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vo=
p_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=
=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>=
nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0x=
b=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100606 nfsd nf=
sd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lock=
mgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=
=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 n=
fsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A=
>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=
=0A>>>>>> 918 100607 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wa=
it+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_=
LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fh=
tovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x5=
54 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exi=
t+0x9a fork_trampoline+0xe=0A>>>>> =0A>>>>> Lots more waiting for the ZFS=
vnode lock in zfs_fhtovp().=0A>>>>> =0A>>>>> 918 100608 nfsd nfsd: servi=
ce mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nf=
sv4_lock+0x9b=0A>>>>> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd=
_lock+0x5b1=0A>>>>> nfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_inter=
nal+0xc77=0A>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=
=0A>>>>> 918 100609 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait=
+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOC=
K1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+=
0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x554 svc=
_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+0x9a f=
ork_trampoline+0xe=0A>>>>> 918 100610 nfsd nfsd: service mi_switch+0xe1=
=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e=0A>>>>> vop_=
stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> nfsvno_advlock+0x11=
9 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad=0A>>>>> nfsrvd_locku+0x283 nfs=
rvd_dorpc+0xec6 nfssvc_program+0x554=0A>>>>> svc_run_internal+0xc77 svc_t=
hread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 10=
0611 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a _sleep+0x=
287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>> nfsrvd_dorpc+0x316 nfssvc_prog=
ram+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb fork_exit+0=
x9a fork_trampoline+0xe=0A>>>>> 918 100612 nfsd nfsd: service mi_switch+0=
xe1=0A>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=
=0A>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77=
=0A>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>>> =
918 100613 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a _sl=
eep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>> nfsrvd_dorpc+0x316 nfssv=
c_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb fork_=
exit+0x9a fork_trampoline+0xe=0A>>>>> 918 100614 nfsd nfsd: service mi_sw=
itch+0xe1=0A>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock=
+0x9b=0A>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0x=
c77=0A>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>=
>> 918 100615 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a =
_sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>> nfsrvd_dorpc+0x316 nf=
ssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb fo=
rk_exit+0x9a fork_trampoline+0xe=0A>>>>> 918 100616 nfsd nfsd: service mi=
_switch+0xe1=0A>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_l=
ock+0x9b=0A>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal=
+0xc77=0A>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A=
>>>>> 918 100617 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x=
3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>> nfsrvd_dorpc+0x316=
nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=
fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918 100618 nfsd nfsd: service=
mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv=
4_lock+0x9b=0A>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_inter=
nal+0xc77=0A>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=
=0A>>>>> 918 100619 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait=
+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOC=
K1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+=
0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x554 svc=
_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+0x9a f=
ork_trampoline+0xe=0A>>>>> 918 100620 nfsd nfsd: service mi_switch+0xe1=
=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vop_=
stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A=
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfss=
vc_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>=
>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918 100621 nfsd nfsd: serv=
ice mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 n=
fsv4_lock+0x9b=0A>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_in=
ternal+0xc77=0A>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+=
0xe=0A>>>>> 918 100622 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_w=
ait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c VOP_=
LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhto=
vp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x554 =
svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+0x9=
a fork_trampoline+0xe=0A>>>>> 918 100623 nfsd nfsd: service mi_switch+0xe=
1=0A>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A=
>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77=0A>>=
>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918 1=
00624 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+=
0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _v=
n_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhto=
vp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_internal+=
0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoline=
+0xe=0A>>>>> 918 100625 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_=
wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c VOP=
_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fht=
ovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x554=
svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+0x=
9a fork_trampoline+0xe=0A>>>>> 918 100626 nfsd nfsd: service mi_switch+0x=
e1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vo=
p_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=
=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> n=
fssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=
=0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918 100627 nfsd nfsd:=
service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_=
args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>=
>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_d=
orpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> sv=
c_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918=
100628 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleepl=
k+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab =
_vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fh=
tovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_interna=
l+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoli=
ne+0xe=0A>>>>> 918 100629 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleep=
q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_f=
htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x5=
54 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+=
0x9a fork_trampoline+0xe=0A>>>>> 918 100630 nfsd nfsd: service mi_switch+=
0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> =
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38=
d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> =
nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=
=0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918 100631 nfsd nfsd:=
service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_=
args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>=
>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_d=
orpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> sv=
c_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918=
100632 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleepl=
k+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab =
_vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fh=
tovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_interna=
l+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoli=
ne+0xe=0A>>>>> 918 100633 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleep=
q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_f=
htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x5=
54 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+=
0x9a fork_trampoline+0xe=0A>>>>> 918 100634 nfsd nfsd: service mi_switch+=
0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> =
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38=
d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> =
nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=
=0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918 100635 nfsd nfsd:=
service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_=
args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>=
>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_d=
orpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> sv=
c_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918=
100636 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleepl=
k+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab =
_vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fh=
tovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_interna=
l+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoli=
ne+0xe=0A>>>>> 918 100637 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleep=
q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_f=
htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x5=
54 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+=
0x9a fork_trampoline+0xe=0A>>>>> 918 100638 nfsd nfsd: service mi_switch+=
0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> =
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38=
d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> =
nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=
=0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918 100639 nfsd nfsd:=
service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_=
args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>=
>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_d=
orpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> sv=
c_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918=
100640 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleepl=
k+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab =
_vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fh=
tovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_interna=
l+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoli=
ne+0xe=0A>>>>> 918 100641 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleep=
q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_f=
htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x5=
54 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+=
0x9a fork_trampoline+0xe=0A>>>>> 918 100642 nfsd nfsd: service mi_switch+=
0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> =
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38=
d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> =
nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=
=0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918 100643 nfsd nfsd:=
service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_=
args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>=
>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_d=
orpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> sv=
c_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918=
100644 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleepl=
k+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab =
_vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fh=
tovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_interna=
l+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoli=
ne+0xe=0A>>>>> 918 100645 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleep=
q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_f=
htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x5=
54 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+=
0x9a fork_trampoline+0xe=0A>>>>> 918 100646 nfsd nfsd: service mi_switch+=
0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> =
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38=
d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> =
nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=
=0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918 100647 nfsd nfsd:=
service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_=
args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>=
>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_d=
orpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> sv=
c_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918=
100648 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleepl=
k+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab =
_vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fh=
tovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_interna=
l+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoli=
ne+0xe=0A>>>>> 918 100649 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleep=
q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_f=
htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x5=
54 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+=
0x9a fork_trampoline+0xe=0A>>>>> 918 100650 nfsd nfsd: service mi_switch+=
0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> =
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38=
d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> =
nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=
=0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918 100651 nfsd nfsd:=
service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_=
args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>=
>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_d=
orpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> sv=
c_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918=
100652 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleepl=
k+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab =
_vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fh=
tovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_interna=
l+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoli=
ne+0xe=0A>>>>> 918 100653 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleep=
q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_f=
htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x5=
54 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+=
0x9a fork_trampoline+0xe=0A>>>>> 918 100654 nfsd nfsd: service mi_switch+=
0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> =
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38=
d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> =
nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=
=0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918 100655 nfsd nfsd:=
service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_=
args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>=
>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_d=
orpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> sv=
c_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918=
100656 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleepl=
k+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab =
_vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fh=
tovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_interna=
l+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoli=
ne+0xe=0A>>>>> 918 100657 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleep=
q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_f=
htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x5=
54 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+=
0x9a fork_trampoline+0xe=0A>>>>> 918 100658 nfsd nfsd: service mi_switch+=
0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> =
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38=
d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> =
nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=
=0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> =0A>>>>> Lo=C3=AFc Bl=
ot,=0A>>>>> UNIX Systems, Network and Security Engineer=0A>>>>> http://ww=
w.unix-experience.fr=0A>>>>> =0A>>>>> 15 d=C3=A9cembre 2014 13:29 "Lo=C3=
=AFc Blot"=0A>>>>> <loic.blot@unix-experience.fr>=0A>>>>> a=0A>>>>> =C3=
=A9crit:=0A>>>>> =0A>>>>> Hmmm...=0A>>>>> now i'm experiencing a deadlock=
.=0A>>>>> =0A>>>>> 0 918 915 0 21 0 12352 3372 zfs D - 1:48.64 nfsd: serv=
er=0A>>>>> (nfsd)=0A>>>>> =0A>>>>> the only issue was to reboot the serve=
r, but after rebooting=0A>>>>> deadlock arrives a second time when i=0A>>=
>>> start my jails over NFS.=0A>>>>> =0A>>>>> Regards,=0A>>>>> =0A>>>>> L=
o=C3=AFc Blot,=0A>>>>> UNIX Systems, Network and Security Engineer=0A>>>>=
> http://www.unix-experience.fr=0A>>>>>; =0A>>>>> 15 d=C3=A9cembre 2014 10=
:07 "Lo=C3=AFc Blot"=0A>>>>> <loic.blot@unix-experience.fr>=0A>>>>> a=0A>=
>>>> =C3=A9crit:=0A>>>>> =0A>>>>> Hi Rick,=0A>>>>> after talking with my =
N+1, NFSv4 is required on our=0A>>>>> infrastructure.=0A>>>>> I tried to =
upgrade NFSv4+ZFS=0A>>>>> server from 9.3 to 10.1, i hope this will resol=
ve some=0A>>>>> issues...=0A>>>>> =0A>>>>> Regards,=0A>>>>> =0A>>>>> Lo=
=C3=AFc Blot,=0A>>>>> UNIX Systems, Network and Security Engineer=0A>>>>>=
http://www.unix-experience.fr=0A>>>>>; =0A>>>>> 10 d=C3=A9cembre 2014 15:=
36 "Lo=C3=AFc Blot"=0A>>>>> <loic.blot@unix-experience.fr>=0A>>>>> a=0A>>=
>>> =C3=A9crit:=0A>>>>> =0A>>>>> Hi Rick,=0A>>>>> thanks for your suggest=
ion.=0A>>>>> For my locking bug, rpc.lockd is stucked in rpcrecv state on=
=0A>>>>> the=0A>>>>> server. kill -9 doesn't affect the=0A>>>>> process, =
it's blocked.... (State: Ds)=0A>>>>> =0A>>>>> for the performances=0A>>>>=
> =0A>>>>> NFSv3: 60Mbps=0A>>>>> NFSv4: 45Mbps=0A>>>>> Regards,=0A>>>>> =
=0A>>>>> Lo=C3=AFc Blot,=0A>>>>> UNIX Systems, Network and Security Engin=
eer=0A>>>>> http://www.unix-experience.fr=0A>>>>>; =0A>>>>> 10 d=C3=A9cemb=
re 2014 13:56 "Rick Macklem" <rmacklem@uoguelph.ca>=0A>>>>> a=0A>>>>> =C3=
=A9crit:=0A>>>>> =0A>>>>> Loic Blot wrote:=0A>>>>> =0A>>>>>> Hi Rick,=0A>=
>>>>> I'm trying NFSv3.=0A>>>>>> Some jails are starting very well but no=
w i have an issue=0A>>>>>> with=0A>>>>>> lockd=0A>>>>>> after some minute=
s:=0A>>>>>> =0A>>>>>> nfs server 10.10.X.8:/jails: lockd not responding=
=0A>>>>>> nfs server 10.10.X.8:/jails lockd is alive again=0A>>>>>> =0A>>=
>>>> I look at mbuf, but i seems there is no problem.=0A>>>>> =0A>>>>> We=
ll, if you need locks to be visible across multiple=0A>>>>> clients,=0A>>=
>>> then=0A>>>>> I'm afraid you are stuck with using NFSv4 and the=0A>>>>=
> performance=0A>>>>> you=0A>>>>> get=0A>>>>> from it. (There is no way t=
o do file handle affinity for=0A>>>>> NFSv4=0A>>>>> because=0A>>>>> the r=
ead and write ops are buried in the compound RPC and=0A>>>>> not=0A>>>>> =
easily=0A>>>>> recognized.)=0A>>>>> =0A>>>>> If the locks don't need to b=
e visible across multiple=0A>>>>> clients,=0A>>>>> I'd=0A>>>>> suggest tr=
ying the "nolockd" option with nfsv3.=0A>>>>> =0A>>>>>> Here is my rc.con=
f on server:=0A>>>>>> =0A>>>>>> nfs_server_enable=3D"YES"=0A>>>>>> nfsv4_=
server_enable=3D"YES"=0A>>>>>> nfsuserd_enable=3D"YES"=0A>>>>>> nfsd_serv=
er_flags=3D"-u -t -n 256"=0A>>>>>> mountd_enable=3D"YES"=0A>>>>>> mountd_=
flags=3D"-r"=0A>>>>>> nfsuserd_flags=3D"-usertimeout 0 -force 20"=0A>>>>>=
> rpcbind_enable=3D"YES"=0A>>>>>> rpc_lockd_enable=3D"YES"=0A>>>>>> rpc_s=
tatd_enable=3D"YES"=0A>>>>>> =0A>>>>>> Here is the client:=0A>>>>>> =0A>>=
>>>> nfsuserd_enable=3D"YES"=0A>>>>>> nfsuserd_flags=3D"-usertimeout 0 -f=
orce 20"=0A>>>>>> nfscbd_enable=3D"YES"=0A>>>>>> rpc_lockd_enable=3D"YES"=
=0A>>>>>> rpc_statd_enable=3D"YES"=0A>>>>>> =0A>>>>>> Have you got an ide=
a ?=0A>>>>>> =0A>>>>>> Regards,=0A>>>>>> =0A>>>>>> Lo=C3=AFc Blot,=0A>>>>=
>> UNIX Systems, Network and Security Engineer=0A>>>>>> http://www.unix-e=
xperience.fr=0A>>>>>> =0A>>>>>> 9 d=C3=A9cembre 2014 04:31 "Rick Macklem"=
<rmacklem@uoguelph.ca>=0A>>>>>> a=0A>>>>>> =C3=A9crit:=0A>>>>>>> Loic Bl=
ot wrote:=0A>>>>>>> =0A>>>>>>>> Hi rick,=0A>>>>>>>> =0A>>>>>>>> I waited =
3 hours (no lag at jail launch) and now I do:=0A>>>>>>>> sysrc=0A>>>>>>>>=
memcached_flags=3D"-v -m 512"=0A>>>>>>>> Command was very very slow...=
=0A>>>>>>>> =0A>>>>>>>> Here is a dd over NFS:=0A>>>>>>>> =0A>>>>>>>> 601=
062912 bytes transferred in 21.060679 secs (28539579=0A>>>>>>>> bytes/sec=
)=0A>>>>>>> =0A>>>>>>> Can you try the same read using an NFSv3 mount?=0A=
>>>>>>> (If it runs much faster, you have probably been bitten by=0A>>>>>=
>> the=0A>>>>>>> ZFS=0A>>>>>>> "sequential vs random" read heuristic whic=
h I've been told=0A>>>>>>> things=0A>>>>>>> NFS is doing "random" reads w=
ithout file handle affinity.=0A>>>>>>> File=0A>>>>>>> handle affinity is =
very hard to do for NFSv4, so it isn't=0A>>>>>>> done.)=0A>>>>> =0A>>>>> =
I was actually suggesting that you try the "dd" over nfsv3=0A>>>>> to=0A>=
>>>> see=0A>>>>> how=0A>>>>> the performance compared with nfsv4. If you =
do that, please=0A>>>>> post=0A>>>>> the=0A>>>>> comparable results.=0A>>=
>>> =0A>>>>> Someday I would like to try and get ZFS's sequential vs=0A>>=
>>> random=0A>>>>> read=0A>>>>> heuristic modified and any info on what d=
ifference in=0A>>>>> performance=0A>>>>> that=0A>>>>> might make for NFS =
would be useful.=0A>>>>> =0A>>>>> rick=0A>>>>> =0A>>>>> rick=0A>>>>> =0A>=
>>>> This is quite slow...=0A>>>>> =0A>>>>> You can found some nfsstat be=
low (command isn't finished=0A>>>>> yet)=0A>>>>> =0A>>>>> nfsstat -c -w 1=
=0A>>>>> =0A>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir=0A>=
>>>> 0 0 0 0 0 0 0 0=0A>>>>> 4 0 0 0 0 0 16 0=0A>>>>> 2 0 0 0 0 0 17 0=0A=
>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>=
>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 4 0 0 0 0 4 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>=
>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>=
>> 4 0 0 0 0 0 3 0=0A>>>>> 0 0 0 0 0 0 3 0=0A>>>>> 37 10 0 8 0 0 14 1=0A>=
>>>> 18 16 0 4 1 2 4 0=0A>>>>> 78 91 0 82 6 12 30 0=0A>>>>> 19 18 0 2 2 4=
2 0=0A>>>>> 0 0 0 0 2 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> GtAttr Looku=
p Rdlink Read Write Rename Access Rddir=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0=
0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 1 0 0 0 0 1 0=0A>>>>> 4 =
6 0 0 6 0 3 0=0A>>>>> 2 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0=200 0 0 0=0A>>>>> 1=
0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 1 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 =
0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0=
0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 =
0 0 0 0 0 0=0A>>>>> 6 108 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0=
0 0 0 0 0 0=0A>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir=
=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=
=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=
=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 98 54 0 86 11 0 25 0=0A>>>>> 36 24 0 39 =
25 0 10 1=0A>>>>> 67 8 0 63 63 0 41 0=0A>>>>> 34 0 0 35 34 0 0 0=0A>>>>> =
75 0 0 75 77 0 0 0=0A>>>>> 34 0 0 35 35 0 0 0=0A>>>>> 75 0 0 74 76 0 0 0=
=0A>>>>> 33 0 0 34 33 0 0 0=0A>>>>> 0 0 0 0 5 0 0 0=0A>>>>> 0 0 0 0 0 0 6=
0=0A>>>>> 11 0 0 0 0 0 11 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 17 0 0 0 0=
1 0=0A>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir=0A>>>>> =
4 5 0 0 0 0 12 0=0A>>>>> 2 0 0 0 0 0 26 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>>=
0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> =
0 0 0 0 0 0 0 0=0A>>>>> 0 4 0 0 0 0 4 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0=
0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 4 0 0 0 0 0 2 0=0A>>>>> 2 =
0 0 0 0 0 24 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 =
0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0=
0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> GtAttr Lookup Rdlink Read Wr=
ite Rename Access Rddir=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=
=0A>>>>> 4 0 0 0 0 0 7 0=0A>>>>> 2 1 0 0 0 0 1 0=0A>>>>> 0 0 0 0 2 0 0 0=
=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 6 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=
=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=
=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 4 6 0 0 0 0 3 0=
=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 2 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=
=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=
=0A>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir=0A>>>>> 0 0 =
0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0=
0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 4 71 0 0 0 0 0 0=0A>>>>> 0 1 0=
0 0 0 0 0=0A>>>>> 2 36 0 0 0 0 1 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0=
0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 1 0 0 =
0 0 0 1 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 79 6 0 =
79 79 0 2 0=0A>>>>> 25 0 0 25 26 0 6 0=0A>>>>> 43 18 0 39 46 0 23 0=0A>>>=
>> 36 0 0 36 36 0 31 0=0A>>>>> 68 1 0 66 68 0 0 0=0A>>>>> GtAttr Lookup R=
dlink Read Write Rename Access Rddir=0A>>>>> 36 0 0 36 36 0 0 0=0A>>>>> 4=
8 0 0 48 49 0 0 0=0A>>>>> 20 0 0 20 20 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>=
>>> 3 14 0 1 0 0 11 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>=
>>>> 0 4 0 0 0 0 4 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 4 22 0 0 0 0 16 0=0A=
>>>>> 2 0 0 0 0 0 23 0=0A>>>>> =0A>>>>> Regards,=0A>>>>> =0A>>>>> Lo=C3=
=AFc Blot,=0A>>>>> UNIX Systems, Network and Security Engineer=0A>>>>> ht=
tp://www.unix-experience.fr=0A>>>>> =0A>>>>> 8 d=C3=A9cembre 2014 09:36 "=
Lo=C3=AFc Blot"=0A>>>>> <loic.blot@unix-experience.fr> a=0A>>>>> =C3=A9cr=
it:=0A>>>>>> Hi Rick,=0A>>>>>> I stopped the jails this week-end and star=
ted it this=0A>>>>>> morning,=0A>>>>>> i'll=0A>>>>>> give you some stats =
this week.=0A>>>>>> =0A>>>>>> Here is my nfsstat -m output (with your rsi=
ze/wsize=0A>>>>>> tweaks)=0A>>>> =0A>>>> =0A>>> =0A>> =0A> nfsv4,tcp,resv=
port,hard,cto,sec=3Dsys,acdirmin=3D3,acdirmax=3D60,acregmin=3D5,acregmax=
=3D60,nametimeo=3D60,negna=0A>>>> =0A>>>>> =0A>>>> =0A>>>> =0A>>> =0A>> =
=0A> etimeo=3D60,rsize=3D32768,wsize=3D32768,readdirsize=3D32768,readahea=
d=3D1,wcommitsize=3D773136,timeout=3D120,retra=0A>>>> =0A>>>>> s=3D214748=
3647=0A>>>>> =0A>>>>> On server side my disks are on a raid controller wh=
ich show a=0A>>>>> 512b=0A>>>>> volume and write performances=0A>>>>> are=
very honest (dd if=3D/dev/zero of=3D/jails/test.dd bs=3D4096=0A>>>>> cou=
nt=3D100000000 =3D> 450MBps)=0A>>>>> =0A>>>>> Regards,=0A>>>>> =0A>>>>> L=
o=C3=AFc Blot,=0A>>>>> UNIX Systems, Network and Security Engineer=0A>>>>=
> http://www.unix-experience.fr=0A>>>>>; =0A>>>>> 5 d=C3=A9cembre 2014 15:=
14 "Rick Macklem" <rmacklem@uoguelph.ca> a=0A>>>>> =C3=A9crit:=0A>>>>> =
=0A>>>>> Loic Blot wrote:=0A>>>>> =0A>>>>> Hi,=0A>>>>> i'm trying to crea=
te a virtualisation environment based on=0A>>>>> jails.=0A>>>>> Those jai=
ls are stored under a big ZFS pool on a FreeBSD=0A>>>>> 9.3=0A>>>>> which=
=0A>>>>> export a NFSv4 volume. This NFSv4 volume was mounted on a=0A>>>>=
> big=0A>>>>> hypervisor (2 Xeon E5v3 + 128GB memory and 8 ports (but=0A>=
>>>> only 1=0A>>>>> was=0A>>>>> used at this time).=0A>>>>> =0A>>>>> The =
problem is simple, my hypervisors runs 6 jails (used 1%=0A>>>>> cpu=0A>>>=
>> and=0A>>>>> 10GB RAM approximatively and less than 1MB bandwidth) and=
=0A>>>>> works=0A>>>>> fine at start but the system slows down and after =
2-3 days=0A>>>>> become=0A>>>>> unusable. When i look at top command i se=
e 80-100% on=0A>>>>> system=0A>>>>> and=0A>>>>> commands are very very sl=
ow. Many process are tagged with=0A>>>>> nfs_cl*.=0A>>>>> =0A>>>>> To be =
honest, I would expect the slowness to be because of=0A>>>>> slow=0A>>>>>=
response=0A>>>>> from the NFSv4 server, but if you do:=0A>>>>> # ps axHl=
=0A>>>>> on a client when it is slow and post that, it would give us=0A>>=
>>> some=0A>>>>> more=0A>>>>> information on where the client side proces=
ses are sitting.=0A>>>>> If you also do something like:=0A>>>>> # nfsstat=
-c -w 1=0A>>>>> and let it run for a while, that should show you how man=
y=0A>>>>> RPCs=0A>>>>> are=0A>>>>> being done and which ones.=0A>>>>> =0A=
>>>>> # nfsstat -m=0A>>>>> will show you what your mount is actually usin=
g.=0A>>>>> The only mount option I can suggest trying is=0A>>>>> "rsize=
=3D32768,wsize=3D32768",=0A>>>>> since some network environments have dif=
ficulties with 64K.=0A>>>>> =0A>>>>> There are a few things you can try o=
n the NFSv4 server side,=0A>>>>> if=0A>>>>> it=0A>>>>> appears=0A>>>>> th=
at the clients are generating a large RPC load.=0A>>>>> - disabling the D=
RC cache for TCP by setting=0A>>>>> vfs.nfsd.cachetcp=3D0=0A>>>>> - If th=
e server is seeing a large write RPC load, then=0A>>>>> "sync=3Ddisabled"=
=0A>>>>> might help, although it does run a risk of data loss when=0A>>>>=
> the=0A>>>>> server=0A>>>>> crashes.=0A>>>>> Then there are a couple of =
other ZFS related things (I'm not=0A>>>>> a=0A>>>>> ZFS=0A>>>>> guy,=0A>>=
>>> but these have shown up on the mailing lists).=0A>>>>> - make sure yo=
ur volumes are 4K aligned and ashift=3D12 (in=0A>>>>> case a=0A>>>>> driv=
e=0A>>>>> that uses 4K sectors is pretending to be 512byte sectored)=0A>>=
>>> - never run over 70-80% full if write performance is an=0A>>>>> issue=
=0A>>>>> - use a zil on an SSD with good write performance=0A>>>>> =0A>>>=
>> The only NFSv4 thing I can tell you is that it is known that=0A>>>>> Z=
FS's=0A>>>>> algorithm for determining sequential vs random I/O fails for=
=0A>>>>> NFSv4=0A>>>>> during writing and this can be a performance hit. =
The only=0A>>>>> workaround=0A>>>>> is to use NFSv3 mounts, since file ha=
ndle affinity=0A>>>>> apparently=0A>>>>> fixes=0A>>>>> the problem and th=
is is only done for NFSv3.=0A>>>>> =0A>>>>> rick=0A>>>>> =0A>>>>> I saw t=
hat there are TSO issues with igb then i'm trying to=0A>>>>> disable=0A>>=
>>> it with sysctl but the situation wasn't solved.=0A>>>>> =0A>>>>> Some=
one has got ideas ? I can give you more informations if=0A>>>>> you=0A>>>=
>> need.=0A>>>>> =0A>>>>> Thanks in advance.=0A>>>>> Regards,=0A>>>>> =0A=
>>>>> Lo=C3=AFc Blot,=0A>>>>> UNIX Systems, Network and Security Engineer=
=0A>>>>> http://www.unix-experience.fr=0A>>>>>; __________________________=
_____________________=0A>>>>> freebsd-fs@freebsd.org mailing list=0A>>>>>=
http://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A>>>>>; To unsubscr=
ibe, send any mail to=0A>>>>> "freebsd-fs-unsubscribe@freebsd.org"=0A>>>>=
> =0A>>>>> _______________________________________________=0A>>>>> freebs=
d-fs@freebsd.org mailing list=0A>>>>> http://lists.freebsd.org/mailman/li=
stinfo/freebsd-fs=0A>>>>> To unsubscribe, send any mail to=0A>>>>> "freeb=
sd-fs-unsubscribe@freebsd.org"=0A>>>>> =0A>>>>> _________________________=
______________________=0A>>>>> freebsd-fs@freebsd.org mailing list=0A>>>>=
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A>>>>>; To unsubsc=
ribe, send any mail to=0A>>>>> "freebsd-fs-unsubscribe@freebsd.org"=0A>>>=
>> =0A>>>>> _______________________________________________=0A>>>>> freeb=
sd-fs@freebsd.org mailing list=0A>>>>> http://lists.freebsd.org/mailman/l=
istinfo/freebsd-fs=0A>>>>> To unsubscribe, send any mail to=0A>>>>> "free=
bsd-fs-unsubscribe@freebsd.org"=0A>>>>> _________________________________=
______________=0A>>>>> freebsd-fs@freebsd.org mailing list=0A>>>>> http:/=
/lists.freebsd.org/mailman/listinfo/freebsd-fs=0A>>>>> To unsubscribe, se=
nd any mail to=0A>>>>> "freebsd-fs-unsubscribe@freebsd.org"=0A>>>>> =0A>>=
>>> _______________________________________________=0A>>>>> freebsd-fs@fr=
eebsd.org mailing list=0A>>>>> http://lists.freebsd.org/mailman/listinfo/=
freebsd-fs=0A>>>>> To unsubscribe, send any mail to=0A>>>>> "freebsd-fs-u=
nsubscribe@freebsd.org"=0A>>> =0A>>> ____________________________________=
___________=0A>>> freebsd-fs@freebsd.org mailing list=0A>>> http://lists.=
freebsd.org/mailman/listinfo/freebsd-fs=0A>>> To unsubscribe, send any ma=
il to=0A>>> "freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?f3106e9c5d2b715a8dac4e6b6de494a3>