From owner-freebsd-fs@FreeBSD.ORG Wed Jan 14 09:16:37 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id ED9D5780 for ; Wed, 14 Jan 2015 09:16:36 +0000 (UTC) Received: from smtp.unix-experience.fr (195-154-176-227.rev.poneytelecom.eu [195.154.176.227]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5027F16E for ; Wed, 14 Jan 2015 09:16:35 +0000 (UTC) Received: from smtp.unix-experience.fr (unknown [192.168.200.21]) by smtp.unix-experience.fr (Postfix) with ESMTP id 60CE43AA7; Wed, 14 Jan 2015 09:08:33 +0000 (UTC) X-Virus-Scanned: scanned by unix-experience.fr Received: from smtp.unix-experience.fr ([192.168.200.21]) by smtp.unix-experience.fr (smtp.unix-experience.fr [192.168.200.21]) (amavisd-new, port 10024) with ESMTP id lertYh7C7L3P; Wed, 14 Jan 2015 09:08:24 +0000 (UTC) Received: from mail.unix-experience.fr (repo.unix-experience.fr [192.168.200.30]) by smtp.unix-experience.fr (Postfix) with ESMTPSA id 08FB83A9B; Wed, 14 Jan 2015 09:08:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=unix-experience.fr; s=uxselect; t=1421226504; bh=HxcZm1xwE1wrfxcR5tHIMcmLYopK3EtrAKFASbWfVNM=; h=Date:From:Subject:To:Cc:In-Reply-To:References; b=lLA6D3RVV3n1t+/95AXAx1PHr3DAwQrXEVXLmVL/hX6URcZu12qnlvjEYpiC5Z6gJ aMxv0hXdLrhAJDKLxyS3MB/uxF+UHHa/gFmStMdETT6Xk3kaXPazX207Dbe14+fT3Z A/wWZrRSYrC2usT5L1RCQar8T2tWbtneBPawzcxg= Mime-Version: 1.0 Date: Wed, 14 Jan 2015 09:08:23 +0000 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-ID: <4426a915c891b0faf962f1975c4dee1e@mail.unix-experience.fr> X-Mailer: RainLoop/1.7.1.215 From: "=?utf-8?B?TG/Dr2MgQmxvdA==?=" Subject: Re: High Kernel Load with nfsv4 To: "Rick Macklem" In-Reply-To: <1566336890.7368425.1420586936307.JavaMail.root@uoguelph.ca> References: <1566336890.7368425.1420586936307.JavaMail.root@uoguelph.ca> Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2015 09:16:37 -0000 Hi Rick,=0Ayou fix doesn't help, i'm sorry. It improves the NFS server sp= eed, but don't remove the problem.=0A=0AI think have loader.conf variable= s for there options could be great.=0A=0AI have wrote a cpp bench for tes= ting my NFS server configuration but i can't reproduce the problem wherea= s the bench let many file descriptors opened (randomly).=0A=0ARegards,=0A= =0ALo=C3=AFc Blot,=0AUNIX Systems, Network and Security Engineer=0Ahttp:/= /www.unix-experience.fr=0A=0A7 janvier 2015 00:29 "Rick Macklem" a =C3=A9crit: =0A> Loic Blot wrote:=0A> =0A>> Hi Rick,=0A>= > =0A>> i saw that some people has issues with igb cards with NFS=0A>> Fo= r example:=0A>> http://freebsd.1045724.n5.nabble.com/NFS-over-LAGG-lacp-p= oor-performance-td5906349.html=0A>> =0A>> can my problem be related ? I u= se igb with default queue number. Here=0A>> are my vmstat -i outputs=0A> = =0A> I have no idea. Maybe someone familiar with this will respond?=0A> = =0A> I do think that the large # of NFSv4 Opens (which are actually a for= m of lock)=0A> could be a factor. The client and server has to search tho= se lists for a match=0A> for many NFSv4 operations, including all reads/w= rites.=0A> =0A> On the server side, the default hash table sizes are very= small. This is in=0A> part that I did testing on 256Mbyte i386 systems, = so that the values were=0A> safe for such a machine.=0A> I'd suggest you = increase the following on the server's kernel.=0A> In sys/fs/nfs/nfs.h:= =0A> NFSSTATEHASHSIZE - This one is in every client header, so if you hav= e a large#=0A> of clients, you don't want to increase it too much. Howeve= r=0A> for a fairly large server handling not too many clients, I'd=0A> tr= y something like 1000 instead of 10.=0A> (I just tried 100 on the small i= 386 laptop I have handy and it=0A> seemed ok for a small test.)=0A> NFSLO= CKHASHSIZE - This one is a single global table, so I'd bump it way up,=0A= > 20000 maybe?=0A> In sys/fs/nfsport.h:=0A> NFSRV_V4STATELIMIT - The comm= ent notes that the default of 500000 seems safe for=0A> a 256Mbyte i386, = so I'd bump it to something like 2000000=0A> for your case.=0A> You will = have to rebuild a kernel from sources after editing these values and=0A> = boot it on the server. Maybe these should become tunables so building a k= ernel=0A> isn't necessary?=0A> =0A> I looked and there isn't much that ca= n be done in the client. At this point,=0A> the open_owners and opens are= single lists for a client (a mount point on a=0A> client machine for Fre= eBSD). If you post what you get for "nfsstat -e -c"=0A> on a typical clie= nt in your setup, that would tell me if it is the open_owners=0A> (which = I suspect) or opens that will be a long list. (I would have to code=0A> a= patch to make either of these a hash table instead of a single linked=0A= > list. I should do this. It was on my to-do list, but got forgotten.;-)= =0A> =0A> rick=0A> =0A>> Server side:=0A>> =0A>> interrupt total rate=0A>= > irq1: atkbd0 18 0=0A>> irq20: ehci1 2790134 2=0A>> irq21: ehci0 2547642= 2=0A>> cpu0:timer 36299188 35=0A>> irq264: ciss0 6352476 6=0A>> irq265: = igb0:que 0 2716692 2=0A>> irq266: igb0:que 1 32205278 31=0A>> irq267: igb= 0:que 2 38395109 37=0A>> irq268: igb0:que 3 1413468 1=0A>> irq269: igb0:q= ue 4 39207930 38=0A>> irq270: igb0:que 5 1622715 1=0A>> irq271: igb0:que = 6 1634676 1=0A>> irq272: igb0:que 7 1190123 1=0A>> irq273: igb0:link 2 0= =0A>> cpu1:timer 14074423 13=0A>> cpu8:timer 12204739 11=0A>> cpu9:timer = 11384192 11=0A>> cpu3:timer 10461566 10=0A>> cpu4:timer 12785103 12=0A>> = cpu6:timer 10739344 10=0A>> cpu5:timer 10978294 10=0A>> cpu7:timer 105997= 05 10=0A>> cpu2:timer 13998891 13=0A>> cpu10:timer 11602361 11=0A>> cpu11= :timer 11568523 11=0A>> Total 296772592 290=0A>> =0A>> And client side:= =0A>> interrupt total rate=0A>> irq9: acpi0 4 0=0A>> irq22: ehci1 950519 = 2=0A>> irq23: ehci0 1865060 4=0A>> cpu0:timer 248128035 546=0A>> irq268: = mfi0 406896 0=0A>> irq269: igb0:que 0 2510556 5=0A>> irq270: igb0:que 1 2= 825336 6=0A>> irq271: igb0:que 2 2092958 4=0A>> irq272: igb0:que 3 196084= 9 4=0A>> irq273: igb0:que 4 2645369 5=0A>> irq274: igb0:que 5 2735187 6= =0A>> irq275: igb0:que 6 2290531 5=0A>> irq276: igb0:que 7 2384370 5=0A>>= irq277: igb0:link 2 0=0A>> irq287: igb2:que 0 1465051 3=0A>> irq288: igb= 2:que 1 856381 1=0A>> irq289: igb2:que 2 809318 1=0A>> irq290: igb2:que 3= 897154 1=0A>> irq291: igb2:que 4 875755 1=0A>> irq292: igb2:que 5 358661= 17 78=0A>> irq293: igb2:que 6 846517 1=0A>> irq294: igb2:que 7 857979 1= =0A>> irq295: igb2:link 2 0=0A>> irq296: igb3:que 0 535212 1=0A>> irq297:= igb3:que 1 454359 1=0A>> irq298: igb3:que 2 454142 1=0A>> irq299: igb3:q= ue 3 454623 1=0A>> irq300: igb3:que 4 456297 1=0A>> irq301: igb3:que 5 45= 5482 1=0A>> irq302: igb3:que 6 456128 1=0A>> irq303: igb3:que 7 454680 1= =0A>> irq304: igb3:link 3 0=0A>> irq305: ahci0 75 0=0A>> cpu1:timer 25723= 3702 566=0A>> cpu13:timer 255603184 562=0A>> cpu7:timer 258492826 569=0A>= > cpu12:timer 255819351 563=0A>> cpu6:timer 258493465 569=0A>> cpu15:time= r 254694003 560=0A>> cpu3:timer 258171320 568=0A>> cpu22:timer 256506877 = 564=0A>> cpu5:timer 253401435 558=0A>> cpu16:timer 255412360 562=0A>> cpu= 11:timer 257318013 566=0A>> cpu20:timer 253648060 558=0A>> cpu2:timer 257= 864543 567=0A>> cpu17:timer 261828899 576=0A>> cpu9:timer 257497326 567= =0A>> cpu18:timer 258451190 569=0A>> cpu8:timer 257784504 567=0A>> cpu14:= timer 254923723 561=0A>> cpu10:timer 257265498 566=0A>> cpu19:timer 25877= 5946 569=0A>> cpu4:timer 256368658 564=0A>> cpu23:timer 255050534 561=0A>= > cpu21:timer 257663842 567=0A>> Total 6225260206 13710=0A>> =0A>> Please= note igb2 on client side is the dedicated link for NFSv4=0A>> =0A>> Rega= rds,=0A>> =0A>> Lo=C3=AFc Blot,=0A>> UNIX Systems, Network and Security E= ngineer=0A>> http://www.unix-experience.fr=0A>> =0A>> 6 janvier 2015 04:1= 7 "Rick Macklem" a =C3=A9crit:=0A>>> Loic Blot wro= te:=0A>>> =0A>>>> Hi Rick,=0A>>>> nfsstat -e -s don't show usefull datas = on server.=0A>>> =0A>>> Well, as far as I know, it returns valid informat= ion.=0A>>> (See below.)=0A>>> =0A>>>> Server Info:=0A>>>> Getattr Setattr= Lookup Readlink Read Write Create=0A>>>> Remove=0A>>>> 26935254 16911 57= 55728 302 2334920 3673866 0=0A>>>> 328332=0A>>>> Rename Link Symlink Mkdi= r Rmdir Readdir RdirPlus=0A>>>> Access=0A>>>> 77980 28 0 0 3 8900 3=0A>>>= > 1806052=0A>>>> Mknod Fsstat Fsinfo PathConf Commit LookupP SetClId=0A>>= >> SetClIdCf=0A>>>> 1 1095 0 0 614377 8172 8=0A>>>> 8=0A>>>> Open OpenAtt= r OpenDwnGr OpenCfrm DelePurge DeleRet GetFH=0A>>>> Lock=0A>>>> 1595299 0= 44145 1495 0 0 5197490=0A>>>> 635015=0A>>>> LockT LockU Close Verify NVe= rify PutFH PutPubFH=0A>>>> PutRootFH=0A>>>> 0 614919 1270938 0 0 22688676= 0=0A>>>> 5=0A>>>> Renew RestoreFH SaveFH Secinfo RelLckOwn V4Create=0A>>= >> 42104 197606 275820 0 143 4578=0A>>>> Server:=0A>>>> Retfailed Faults = Clients=0A>>>> 0 0 6=0A>>>> OpenOwner Opens LockOwner Locks Delegs=0A>>>>= 32335 145448 204 181 0=0A>>> =0A>>> Well, 145448 Opens are a lot of Open= files. Each of these uses=0A>>> a kernel malloc'd data structure that is= linked into multiple=0A>>> linked lists.=0A>>> =0A>>> The question is..w= hy aren't these Opens being closed?=0A>>> Since FreeBSD does I/O on an mm= ap'd file after closing it,=0A>>> the FreeBSD NFSv4 client is forced to d= elay doing Close RPCs=0A>>> until the vnode is VOP_INACTIVE()/VOP_RECLAIM= ()'d. (The=0A>>> VOP_RECLAIM() case is needed, since VOP_INACTIVE() isn't= =0A>>> guaranteed to be called.)=0A>>> =0A>>> Since there were about 1.5 = million Opens and 1.27 million=0A>>> Closes, it does appear that Opens ar= e being Closed.=0A>>> Now, I'm not sure I would have imagined 1.5million = file Opens=0A>>> in a few days. My guess is this is the bottleneck.=0A>>>= =0A>>> I'd suggest that you do:=0A>>> # nfsstat -e -c=0A>>> on each of t= he NFSv4 clients and see how many Opens/client=0A>>> there are. I vaguely= remember an upper limit in the client,=0A>>> but can't remember what it = is set to.=0A>>> --> I suspect the client Open/Lock limit needs to be inc= reased.=0A>>> (I can't remember if the server also has a limit, but I=0A>= >> think it does.)=0A>>> Then the size of the hash tables used to search = the Opens=0A>>> may also need to be increased a lot.=0A>>> =0A>>> Also, I= 'd suggest you take a look at whatever apps. are=0A>>> running on the cli= ent(s) and try to figure out why they=0A>>> are Opening so many files?=0A= >>> =0A>>> My guess is that the client(s) are gettig bogged down by all= =0A>>> these Opens.=0A>>> =0A>>>> Server Cache Stats:=0A>>>> Inprog Idem = Non-idem Misses CacheSize TCPPeak=0A>>>> 0 0 1 15082947 60 16522=0A>>>> = =0A>>>> Only GetAttr and Lookup increase and it's only every 4-5 seconds= =0A>>>> and=0A>>>> only +2 to +5 into theses values.=0A>>>> =0A>>>> Now o= n client, if i take four processes stack i got=0A>>>> =0A>>>> PID TID COM= M TDNAME KSTACK=0A>>>> 63170 102547 mv - mi_switch+0xe1=0A>>>> turnstile_= wait+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65=0A>>>> nfs_lookup+0x= 3d0 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4=0A>>>> vn_open_cred+0x21= d kern_openat+0x26f amd64_syscall+0x351=0A>>>> Xfast_syscall+0xfb=0A>>>> = =0A>>>> Another mv:=0A>>>> 63140 101738 mv - mi_switch+0xe1=0A>>>> turnst= ile_wait+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65=0A>>>> nfs_looku= p+0x3d0 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4=0A>>>> kern_statat_v= nhook+0xae sys_lstat+0x30 amd64_syscall+0x351=0A>>>> Xfast_syscall+0xfb= =0A>>>> =0A>>>> 62070 102170 sendmail - mi_switch+0xe1=0A>>>> sleepq_time= dwait+0x3a _sleep+0x26e clnt_vc_call+0x666=0A>>>> clnt_reconnect_call+0x4= fa newnfs_request+0xa8c nfscl_request+0x72=0A>>>> nfsrpc_lookup+0x1fb nfs= _lookup+0x508 VOP_LOOKUP_APV+0xa1=0A>>>> lookup+0x59c namei+0x4d4 kern_st= atat_vnhook+0xae sys_lstat+0x30=0A>>>> amd64_syscall+0x351 Xfast_syscall+= 0xfb=0A>>>> =0A>>>> 63200 100930 mv - mi_switch+0xe1=0A>>>> turnstile_wai= t+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65=0A>>>> nfs_lookup+0x3d0= VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4=0A>>>> kern_statat_vnhook+0= xae sys_lstat+0x30 amd64_syscall+0x351=0A>>>> Xfast_syscall+0xfb=0A>>> = =0A>>> The above simply says that thread 102710 is waiting for a Lookup= =0A>>> reply from the server and the other 3 are waiting for the mutex=0A= >>> that protects the state structures in the client. (I suspect=0A>>> so= me other thread in the client is wading through the Open list,=0A>>> if a= single client has a lot of these 145K Opens.)=0A>>> =0A>>>> When client = is in this state, server was doing nothing special=0A>>>> (procstat -kk)= =0A>>>> =0A>>>> PID TID COMM TDNAME KSTACK=0A>>>> 895 100538 nfsd nfsd: m= aster mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_timedwait_si= g+0x10=0A>>>> _cv_timedwait_sig_sbt+0x18b svc_run_internal+0x4a1 svc_run+= 0x1de=0A>>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c=0A>>>> a= md64_syscall+0x351 Xfast_syscall+0xfb=0A>>>> 895 100568 nfsd nfsd: servic= e mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv= _wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 895 100569 nfsd nfsd: service = mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_w= ait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exi= t+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 895 100570 nfsd nfsd: service mi= _switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wai= t_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 895 100571 nfsd nfsd: service mi_s= witch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_= sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x= 9a=0A>>>> fork_trampoline+0xe=0A>>>> 895 100572 nfsd nfsd: service mi_swi= tch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_si= g+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100573 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100575 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100576 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100577 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100578 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100579 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100580 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100581 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100582 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100583 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100584 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100585 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100586 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100587 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100588 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100589 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100590 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100592 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100593 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100594 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100595 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100596 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100597 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100598 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100599 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100600 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100602 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100603 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100604 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100605 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100606 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100607 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100608 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100609 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100610 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100611 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100612 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100613 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100614 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100615 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100617 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100618 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100619 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100621 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100622 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100623 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100624 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100625 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100626 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100627 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100628 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100629 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100630 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100631 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100632 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100633 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100634 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100635 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100636 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100638 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100639 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100640 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100641 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100642 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100643 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100644 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100645 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100646 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100647 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100648 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100649 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100651 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100652 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100653 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100654 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100655 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100656 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100657 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100658 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100659 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100661 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100662 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100684 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100685 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100686 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100797 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100798 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100799 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100800 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> 895 100801 nfsd nfsd: service mi_switc= h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>> fork_trampoline+0xe=0A>>>> =0A>>>> I really think it's a client s= ide problem, maybe a lookup problem.=0A>>>> =0A>>>> Regards,=0A>>>> =0A>>= >> Lo=C3=AFc Blot,=0A>>>> UNIX Systems, Network and Security Engineer=0A>= >>> http://www.unix-experience.fr=0A>>>> =0A>>>> 5 janvier 2015 14:35 "Ri= ck Macklem" a=0A>>>> =C3=A9crit:=0A>>>>> Loic Blot= wrote:=0A>>>>> =0A>>>>>> Hi,=0A>>>>>> happy new year Rick and @freebsd-f= s.=0A>>>>>> =0A>>>>>> After some days, i looked my NFSv4.1 mount. At serv= er start it=0A>>>>>> was=0A>>>>>> calm, but after 4 days, here is the top= stat...=0A>>>>>> =0A>>>>>> CPU: 0.0% user, 0.0% nice, 100% system, 0.0% = interrupt, 0.0%=0A>>>>>> idle=0A>>>>>> =0A>>>>>> Definitively i think it'= s a problem on client side. What can i=0A>>>>>> look=0A>>>>>> into runnin= g kernel to resolve this issue ?=0A>>>>> =0A>>>>> Well, I'd start with:= =0A>>>>> # nfsstat -e -s=0A>>>>> - run repeatedly on the server (once eve= ry N seconds in a loop).=0A>>>>> Then look at the output, comparing the c= ounts and see which RPCs=0A>>>>> are being performed by the client(s). Yo= u are looking for which=0A>>>>> RPCs are being done a lot. (If one RPC is= almost 100% of the=0A>>>>> load,=0A>>>>> then it might be a client/cachi= ng issue for whatever that RPC is=0A>>>>> doing.)=0A>>>>> =0A>>>>> Also l= ook at the Open/Lock counts near the end of the output.=0A>>>>> If the # = of Opens/Locks is large, it may be possible to reduce=0A>>>>> the=0A>>>>>= CPU overheads by using larger hash tables.=0A>>>>> =0A>>>>> Then you nee= d to profile the server kernel to see where the CPU=0A>>>>> is being used= .=0A>>>>> Hopefully someone else can fill you in on how to do that, becau= se=0A>>>>> I'll admit I don't know how to.=0A>>>>> Basically you are look= ing to see if the CPU is being used in=0A>>>>> the NFS server code or ZFS= .=0A>>>>> =0A>>>>> Good luck with it, rick=0A>>>>> =0A>>>>>> Regards,=0A>= >>>>> =0A>>>>>> Lo=C3=AFc Blot,=0A>>>>>> UNIX Systems, Network and Securi= ty Engineer=0A>>>>>> http://www.unix-experience.fr=0A>>>>>> =0A>>>>>> 30 = d=C3=A9cembre 2014 16:16 "Lo=C3=AFc Blot"=0A>>>>>> =0A>>>>>> a=0A>>>>>> =C3=A9crit:=0A>>>>>>> Hi Rick,=0A>>>>>>> i u= pgraded my jail host from FreeBSD 9.3 to 10.1 to use NFS=0A>>>>>>> v4.1= =0A>>>>>>> (mountoptions:=0A>>>>>>> rw,rsize=3D32768,wsize=3D32768,tcp,nf= sv4,minorversion=3D1)=0A>>>>>>> =0A>>>>>>> Performance is quite stable bu= t it's slow. Not as slow as=0A>>>>>>> before=0A>>>>>>> but slow... servic= es was launched=0A>>>>>>> but no client are using them and system CPU % w= as 10-50%.=0A>>>>>>> =0A>>>>>>> I don't see anything on NFSv4.1 server, i= t's perfectly stable=0A>>>>>>> and=0A>>>>>>> functionnal.=0A>>>>>>> =0A>>= >>>>> Regards,=0A>>>>>>> =0A>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>> UNIX System= s, Network and Security Engineer=0A>>>>>>> http://www.unix-experience.fr= =0A>>>>>>> =0A>>>>>>> 23 d=C3=A9cembre 2014 00:20 "Rick Macklem" a=0A>>>>>>> =C3=A9crit:=0A>>>>>>> =0A>>>>>>>> Loic Blot wr= ote:=0A>>>>>>>> =0A>>>>>>>>> Hi,=0A>>>>>>>>> =0A>>>>>>>>> To clarify beca= use of our exchanges. Here are the current=0A>>>>>>>>> sysctl=0A>>>>>>>>>= options for server:=0A>>>>>>>>> =0A>>>>>>>>> vfs.nfsd.enable_nobodycheck= =3D0=0A>>>>>>>>> vfs.nfsd.enable_nogroupcheck=3D0=0A>>>>>>>>> =0A>>>>>>>>= > vfs.nfsd.maxthreads=3D200=0A>>>>>>>>> vfs.nfsd.tcphighwater=3D10000=0A>= >>>>>>>> vfs.nfsd.tcpcachetimeo=3D300=0A>>>>>>>>> vfs.nfsd.server_min_nfs= vers=3D4=0A>>>>>>>>> =0A>>>>>>>>> kern.maxvnodes=3D10000000=0A>>>>>>>>> k= ern.ipc.maxsockbuf=3D4194304=0A>>>>>>>>> net.inet.tcp.sendbuf_max=3D41943= 04=0A>>>>>>>>> net.inet.tcp.recvbuf_max=3D4194304=0A>>>>>>>>> =0A>>>>>>>>= > vfs.lookup_shared=3D0=0A>>>>>>>>> =0A>>>>>>>>> Regards,=0A>>>>>>>>> =0A= >>>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>>>> UNIX Systems, Network and Security = Engineer=0A>>>>>>>>> http://www.unix-experience.fr=0A>>>>>>>>> =0A>>>>>>>= >> 22 d=C3=A9cembre 2014 09:42 "Lo=C3=AFc Blot"=0A>>>>>>>>> =0A>>>>>>>>> a=0A>>>>>>>>> =C3=A9crit:=0A>>>>>>>>> =0A>>= >>>>>>> Hi Rick,=0A>>>>>>>>> my 5 jails runs this weekend and now i have = some stats on=0A>>>>>>>>> this=0A>>>>>>>>> monday.=0A>>>>>>>>> =0A>>>>>>>= >> Hopefully deadlock was fixed, yeah, but everything isn't good=0A>>>>>>= >>> :(=0A>>>>>>>>> =0A>>>>>>>>> On NFSv4 server (FreeBSD 10.1) system use= s 35% CPU=0A>>>>>>>>> =0A>>>>>>>>> As i can see this is because of nfsd:= =0A>>>>>>>>> =0A>>>>>>>>> 918 root 96 20 0 12352K 3372K rpcsvc 6 51.4H=0A= >>>>>>>>> 273.68% nfsd: server (nfsd)=0A>>>>>>>>> =0A>>>>>>>>> If i look = at dmesg i see:=0A>>>>>>>>> nfsd server cache flooded, try increasing=0A>= >>>>>>>> vfs.nfsd.tcphighwater=0A>>>>>>>> =0A>>>>>>>> Well, you have a co= uple of choices:=0A>>>>>>>> 1 - Use NFSv4.1 (add "minorversion=3D1" to yo= ur mount options).=0A>>>>>>>> (NFSv4.1 avoids use of the DRC and instead = uses something=0A>>>>>>>> called sessions. See below.)=0A>>>>>>>> OR=0A>>= >>>>>> =0A>>>>>>>>> vfs.nfsd.tcphighwater was set to 10000, i increase it= to=0A>>>>>>>>> 15000=0A>>>>>>>> =0A>>>>>>>> 2 - Bump vfs.nfsd.tcphighwat= er way up, until you no longer see=0A>>>>>>>> "nfs server cache flooded" = messages. (I think Garrett Wollman=0A>>>>>>>> uses=0A>>>>>>>> 100000. (Yo= u may still see quite a bit of CPU overheads.)=0A>>>>>>>> =0A>>>>>>>> OR= =0A>>>>>>>> =0A>>>>>>>> 3 - Set vfs.nfsd.cachetcp=3D0 (which disables the= DRC and gets=0A>>>>>>>> rid=0A>>>>>>>> of the CPU overheads). However, t= here is a risk of data=0A>>>>>>>> corruption=0A>>>>>>>> if you have a cli= ent->server network partitioning of a=0A>>>>>>>> moderate=0A>>>>>>>> dura= tion, because a non-idempotent RPC may get redone, becasue=0A>>>>>>>> the= client times out waiting for a reply. If a non-idempotent=0A>>>>>>>> RPC= gets done twice on the server, data corruption can happen.=0A>>>>>>>> (T= he DRC provides improved correctness, but does add=0A>>>>>>>> overhead.)= =0A>>>>>>>> =0A>>>>>>>> If #1 works for you, it is the preferred solution= , since=0A>>>>>>>> Sessions=0A>>>>>>>> in NFSv4.1 solves the correctness = problem in a good, space=0A>>>>>>>> bound=0A>>>>>>>> way. A session basic= ally has N (usually 32 or 64) slots and=0A>>>>>>>> only=0A>>>>>>>> allows= one outstanding RPC/slot. As such, it can cache the=0A>>>>>>>> previous= =0A>>>>>>>> reply for each slot (32 or 64 of them) and guarantee "exactly= =0A>>>>>>>> once"=0A>>>>>>>> RPC semantics.=0A>>>>>>>> =0A>>>>>>>> rick= =0A>>>>>>>> =0A>>>>>>>>> Here is 'nfsstat -s' output:=0A>>>>>>>>> =0A>>>>= >>>>> Server Info:=0A>>>>>>>>> Getattr Setattr Lookup Readlink Read Write= Create=0A>>>>>>>>> Remove=0A>>>>>>>>> 12600652 1812 2501097 156 1386423 = 1983729 123=0A>>>>>>>>> 162067=0A>>>>>>>>> Rename Link Symlink Mkdir Rmdi= r Readdir RdirPlus=0A>>>>>>>>> Access=0A>>>>>>>>> 36762 9 0 0 0 3147 0=0A= >>>>>>>>> 623524=0A>>>>>>>>> Mknod Fsstat Fsinfo PathConf Commit=0A>>>>>>= >>> 0 0 0 0 328117=0A>>>>>>>>> Server Ret-Failed=0A>>>>>>>>> 0=0A>>>>>>>>= > Server Faults=0A>>>>>>>>> 0=0A>>>>>>>>> Server Cache Stats:=0A>>>>>>>>>= Inprog Idem Non-idem Misses=0A>>>>>>>>> 0 0 0 12635512=0A>>>>>>>>> Serve= r Write Gathering:=0A>>>>>>>>> WriteOps WriteRPC Opsaved=0A>>>>>>>>> 1983= 729 1983729 0=0A>>>>>>>>> =0A>>>>>>>>> And here is 'procstat -kk' for nfs= d (server)=0A>>>>>>>>> =0A>>>>>>>>> 918 100528 nfsd nfsd: master mi_switc= h+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10=0A= >>>>>>>>> _cv_timedwait_sig_sbt+0x18b svc_run_internal+0x4a1=0A>>>>>>>>> = svc_run+0x1de=0A>>>>>>>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+= 0x9c=0A>>>>>>>>> amd64_syscall+0x351 Xfast_syscall+0xfb=0A>>>>>>>>> 918 1= 00568 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+= 0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_r= un_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_tr= ampoline+0xe=0A>>>>>>>>> 918 100569 nfsd nfsd: service mi_switch+0xe1=0A>= >>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wa= it_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork= _exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100570 nfsd nf= sd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_w= ait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0= x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe= =0A>>>>>>>>> 918 100571 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sle= epq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a= =0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100572 nfsd nfsd: servic= e mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0x= f=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_t= hread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>= > 918 100573 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_s= ignals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>= > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> = fork_trampoline+0xe=0A>>>>>>>>> 918 100574 nfsd nfsd: service mi_switch+0= xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>>= _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0= xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100575 = nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab s= leepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoli= ne+0xe=0A>>>>>>>>> 918 100576 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>= >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig= +0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100577 nfsd nfsd: se= rvice mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_si= g+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>= >>>>> 918 100578 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_cat= ch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>= >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>= >>> fork_trampoline+0xe=0A>>>>>>>>> 918 100579 nfsd nfsd: service mi_swit= ch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>= >>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_sta= rt+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100= 580 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0x= ab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run= _internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_tram= poline+0xe=0A>>>>>>>>> 918 100581 nfsd nfsd: service mi_switch+0xe1=0A>>>= >>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait= _sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100582 nfsd nfsd= : service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A= >>>>>>>>> 918 100583 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq= _catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A= >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>= >>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100584 nfsd nfsd: service mi_= switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>= >>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918= 100585 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signal= s+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_= trampoline+0xe=0A>>>>>>>>> 918 100586 nfsd nfsd: service mi_switch+0xe1= =0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _c= v_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100587 nfs= d nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab slee= pq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+= 0xe=0A>>>>>>>>> 918 100588 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> = sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x= 16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9= a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100589 nfsd nfsd: servi= ce mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0= xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_= thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>= >> 918 100590 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_= signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>= >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>>= fork_trampoline+0xe=0A>>>>>>>>> 918 100591 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>= > _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+= 0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100592= nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab = sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_in= ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampol= ine+0xe=0A>>>>>>>>> 918 100593 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>= >>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_si= g+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100594 nfsd nfsd: s= ervice mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_s= ig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>= >>>>>> 918 100595 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_ca= tch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>= >>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>= >>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100596 nfsd nfsd: service mi_swi= tch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>= >>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_st= art+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 10= 0597 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_ru= n_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_tra= mpoline+0xe=0A>>>>>>>>> 918 100598 nfsd nfsd: service mi_switch+0xe1=0A>>= >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wai= t_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_= exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100599 nfsd nfs= d: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wa= it_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x= 87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe= =0A>>>>>>>>> 918 100600 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sle= epq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a= =0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100601 nfsd nfsd: servic= e mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0x= f=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_t= hread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>= > 918 100602 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_s= ignals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>= > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> = fork_trampoline+0xe=0A>>>>>>>>> 918 100603 nfsd nfsd: service mi_switch+0= xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>>= _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0= xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100604 = nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab s= leepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoli= ne+0xe=0A>>>>>>>>> 918 100605 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>= >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig= +0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100606 nfsd nfsd: se= rvice mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_si= g+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>= >>>>> 918 100607 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_cat= ch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>= >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>= >>> fork_trampoline+0xe=0A>>>>>>>>> 918 100608 nfsd nfsd: service mi_swit= ch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>= >>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_sta= rt+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100= 609 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0x= ab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run= _internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_tram= poline+0xe=0A>>>>>>>>> 918 100610 nfsd nfsd: service mi_switch+0xe1=0A>>>= >>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait= _sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100611 nfsd nfsd= : service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A= >>>>>>>>> 918 100612 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq= _catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A= >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>= >>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100613 nfsd nfsd: service mi_= switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>= >>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918= 100614 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signal= s+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_= trampoline+0xe=0A>>>>>>>>> 918 100615 nfsd nfsd: service mi_switch+0xe1= =0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _c= v_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100616 nfs= d nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab slee= pq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+= 0xe=0A>>>>>>>>> 918 100617 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> = sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x= 16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9= a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100618 nfsd nfsd: servi= ce mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0= xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_= thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>= >> 918 100619 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_= signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>= >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>>= fork_trampoline+0xe=0A>>>>>>>>> 918 100620 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>= > _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+= 0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100621= nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab = sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_in= ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampol= ine+0xe=0A>>>>>>>>> 918 100622 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>= >>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_si= g+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100623 nfsd nfsd: s= ervice mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_s= ig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>= >>>>>> 918 100624 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_ca= tch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>= >>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>= >>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100625 nfsd nfsd: service mi_swi= tch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>= >>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_st= art+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 10= 0626 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_ru= n_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_tra= mpoline+0xe=0A>>>>>>>>> 918 100627 nfsd nfsd: service mi_switch+0xe1=0A>>= >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wai= t_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_= exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100628 nfsd nfs= d: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wa= it_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x= 87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe= =0A>>>>>>>>> 918 100629 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sle= epq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a= =0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100630 nfsd nfsd: servic= e mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0x= f=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_t= hread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>= > 918 100631 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_s= ignals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>= > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> = fork_trampoline+0xe=0A>>>>>>>>> 918 100632 nfsd nfsd: service mi_switch+0= xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>>= _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0= xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100633 = nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab s= leepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoli= ne+0xe=0A>>>>>>>>> 918 100634 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>= >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig= +0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100635 nfsd nfsd: se= rvice mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_si= g+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>= >>>>> 918 100636 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_cat= ch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>= >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>= >>> fork_trampoline+0xe=0A>>>>>>>>> 918 100637 nfsd nfsd: service mi_swit= ch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>= >>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_sta= rt+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100= 638 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0x= ab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run= _internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_tram= poline+0xe=0A>>>>>>>>> 918 100639 nfsd nfsd: service mi_switch+0xe1=0A>>>= >>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait= _sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100640 nfsd nfsd= : service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A= >>>>>>>>> 918 100641 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq= _catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A= >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>= >>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100642 nfsd nfsd: service mi_= switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>= >>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918= 100643 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signal= s+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_= trampoline+0xe=0A>>>>>>>>> 918 100644 nfsd nfsd: service mi_switch+0xe1= =0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _c= v_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100645 nfs= d nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab slee= pq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+= 0xe=0A>>>>>>>>> 918 100646 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> = sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x= 16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9= a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100647 nfsd nfsd: servi= ce mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0= xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_= thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>= >> 918 100648 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_= signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>= >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>>= fork_trampoline+0xe=0A>>>>>>>>> 918 100649 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>= > _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+= 0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100650= nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab = sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_in= ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampol= ine+0xe=0A>>>>>>>>> 918 100651 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>= >>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_si= g+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100652 nfsd nfsd: s= ervice mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_s= ig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>= >>>>>> 918 100653 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_ca= tch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>= >>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>= >>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100654 nfsd nfsd: service mi_swi= tch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>= >>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_st= art+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 10= 0655 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_ru= n_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_tra= mpoline+0xe=0A>>>>>>>>> 918 100656 nfsd nfsd: service mi_switch+0xe1=0A>>= >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wai= t_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_= exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100657 nfsd nfs= d: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wa= it_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x= 87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe= =0A>>>>>>>>> 918 100658 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sle= epq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a= =0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100659 nfsd nfsd: servic= e mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0x= f=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_t= hread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>= > 918 100660 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_s= ignals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>= > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> = fork_trampoline+0xe=0A>>>>>>>>> 918 100661 nfsd nfsd: service mi_switch+0= xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>>= _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0= xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100662 = nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab s= leepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoli= ne+0xe=0A>>>>>>>>> ---=0A>>>>>>>>> =0A>>>>>>>>> Now if we look at client = (FreeBSD 9.3)=0A>>>>>>>>> =0A>>>>>>>>> We see system was very busy and do= many and many interrupts=0A>>>>>>>>> =0A>>>>>>>>> CPU: 0.0% user, 0.0% n= ice, 37.8% system, 51.2% interrupt,=0A>>>>>>>>> 11.0%=0A>>>>>>>>> idle=0A= >>>>>>>>> =0A>>>>>>>>> A look at process list shows that there are many s= endmail=0A>>>>>>>>> process=0A>>>>>>>>> in=0A>>>>>>>>> state nfstry=0A>>>= >>>>>> =0A>>>>>>>>> nfstry 18 32:27 0.88% sendmail: Queue runner@00:30:00= for=0A>>>>>>>>> /var/spool/clientm=0A>>>>>>>>> =0A>>>>>>>>> Here is 'nfs= stat -c' output:=0A>>>>>>>>> =0A>>>>>>>>> Client Info:=0A>>>>>>>>> Rpc Co= unts:=0A>>>>>>>>> Getattr Setattr Lookup Readlink Read Write Create=0A>>>= >>>>>> Remove=0A>>>>>>>>> 1051347 1724 2494481 118 903902 1901285 162676= =0A>>>>>>>>> 161899=0A>>>>>>>>> Rename Link Symlink Mkdir Rmdir Readdir R= dirPlus=0A>>>>>>>>> Access=0A>>>>>>>>> 36744 2 0 114 40 3131 0=0A>>>>>>>>= > 544136=0A>>>>>>>>> Mknod Fsstat Fsinfo PathConf Commit=0A>>>>>>>>> 9 0 = 0 0 245821=0A>>>>>>>>> Rpc Info:=0A>>>>>>>>> TimedOut Invalid X Replies R= etries Requests=0A>>>>>>>>> 0 0 0 0 8356557=0A>>>>>>>>> Cache Info:=0A>>>= >>>>>> Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits=0A>>>= >>>>>> Misses=0A>>>>>>>>> 108754455 491475 54229224 2437229 46814561 8217= 23 5132123=0A>>>>>>>>> 1871871=0A>>>>>>>>> BioRLHits Misses BioD Hits Mis= ses DirE Hits Misses Accs Hits=0A>>>>>>>>> Misses=0A>>>>>>>>> 144035 118 = 53736 2753 27813 1 57238839=0A>>>>>>>>> 544205=0A>>>>>>>>> =0A>>>>>>>>> I= f you need more things, tell me, i let the PoC in this=0A>>>>>>>>> state.= =0A>>>>>>>>> =0A>>>>>>>>> Thanks=0A>>>>>>>>> =0A>>>>>>>>> Regards,=0A>>>>= >>>>> =0A>>>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>>>> UNIX Systems, Network and = Security Engineer=0A>>>>>>>>> http://www.unix-experience.fr=0A>>>>>>>>> = =0A>>>>>>>>> 21 d=C3=A9cembre 2014 01:33 "Rick Macklem" =0A>>>>>>>>> a=0A>>>>>>>>> =C3=A9crit:=0A>>>>>>>>> =0A>>>>>>>>> Loic= Blot wrote:=0A>>>>>>>>> =0A>>>>>>>>>> Hi Rick,=0A>>>>>>>>>> ok, i don't = need locallocks, i haven't understand option was=0A>>>>>>>>>> for=0A>>>>>= >>>>> that=0A>>>>>>>>>> usage, i removed it.=0A>>>>>>>>>> I do more tests= on monday.=0A>>>>>>>>>> Thanks for the deadlock fix, for other people :)= =0A>>>>>>>>> =0A>>>>>>>>> Good. Please let us know if running with=0A>>>>= >>>>> vfs.nfsd.enable_locallocks=3D0=0A>>>>>>>>> gets rid of the deadlock= s? (I think it fixes the one you=0A>>>>>>>>> saw.)=0A>>>>>>>>> =0A>>>>>>>= >> On the performance side, you might also want to try different=0A>>>>>>= >>> values=0A>>>>>>>>> of=0A>>>>>>>>> readahead, if the Linux client has = such a mount option. (With=0A>>>>>>>>> the=0A>>>>>>>>> NFSv4-ZFS sequenti= al vs random I/O heuristic, I have no idea=0A>>>>>>>>> what=0A>>>>>>>>> t= he=0A>>>>>>>>> optimal readahead value would be.)=0A>>>>>>>>> =0A>>>>>>>>= > Good luck with it and please let us know how it goes, rick=0A>>>>>>>>> = ps: I now have a patch to fix the deadlock when=0A>>>>>>>>> vfs.nfsd.enab= le_locallocks=3D1=0A>>>>>>>>> is set. I'll post it for anyone who is inte= rested after I put=0A>>>>>>>>> it=0A>>>>>>>>> through some testing.=0A>>>= >>>>>> =0A>>>>>>>>> --=0A>>>>>>>>> Best regards,=0A>>>>>>>>> Lo=C3=AFc BL= OT,=0A>>>>>>>>> UNIX systems, security and network engineer=0A>>>>>>>>> h= ttp://www.unix-experience.fr=0A>>>>>>>>> =0A>>>>>>>>> Le jeudi 18 d=C3=A9= cembre 2014 =C3=A0 19:46 -0500, Rick Macklem a =C3=A9crit=0A>>>>>>>>> := =0A>>>>>>>>> =0A>>>>>>>>> Loic Blot wrote:=0A>>>>>>>>>> Hi rick,=0A>>>>>>= >>>> i tried to start a LXC container on Debian Squeeze from my=0A>>>>>>>= >>> freebsd=0A>>>>>>>>>> ZFS+NFSv4 server and i also have a deadlock on n= fsd=0A>>>>>>>>>> (vfs.lookup_shared=3D0). Deadlock procs each time i laun= ch a=0A>>>>>>>>>> squeeze=0A>>>>>>>>>> container, it seems (3 tries, 3 fa= ils).=0A>>>>>>>>> =0A>>>>>>>>> Well, I`ll take a look at this `procstat -= kk`, but the only=0A>>>>>>>>> thing=0A>>>>>>>>> I`ve seen posted w.r.t. a= voiding deadlocks in ZFS is to not=0A>>>>>>>>> use=0A>>>>>>>>> nullfs. (I= have no idea if you are using any nullfs mounts,=0A>>>>>>>>> but=0A>>>>>= >>>> if so, try getting rid of them.)=0A>>>>>>>>> =0A>>>>>>>>> Here`s a h= igh level post about the ZFS and vnode locking=0A>>>>>>>>> problem,=0A>>>= >>>>>> but there is no patch available, as far as I know.=0A>>>>>>>>> =0A= >>>>>>>>> http://docs.FreeBSD.org/cgi/mid.cgi?54739F41.8030407=0A>>>>>>>>= > =0A>>>>>>>>> rick=0A>>>>>>>>> =0A>>>>>>>>> 921 - D 0:00.02 nfsd: server= (nfsd)=0A>>>>>>>>> =0A>>>>>>>>> Here is the procstat -kk=0A>>>>>>>>> =0A= >>>>>>>>> PID TID COMM TDNAME KSTACK=0A>>>>>>>>> 921 100538 nfsd nfsd: ma= ster mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_= args+0xc9e=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43= =0A>>>>>>>>> nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14a= d=0A>>>>>>>>> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554= =0A>>>>>>>>> svc_run_internal+0xc77 svc_run+0x1de nfsrvd_nfsd+0x1ca=0A>>>= >>>>>> nfssvc_nfsd+0x107 sys_nfssvc+0x9c=0A>>>>>>>>> 921 100572 nfsd nfsd= : service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A= >>>>>>>>> 921 100573 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq= _catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A= >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>= >>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100574 nfsd nfsd: service mi_= switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>= >>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921= 100575 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signal= s+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_= trampoline+0xe=0A>>>>>>>>> 921 100576 nfsd nfsd: service mi_switch+0xe1= =0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _c= v_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100577 nfs= d nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab slee= pq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+= 0xe=0A>>>>>>>>> 921 100578 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> = sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x= 16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9= a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100579 nfsd nfsd: servi= ce mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0= xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_= thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>= >> 921 100580 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_= signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>= >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>>= fork_trampoline+0xe=0A>>>>>>>>> 921 100581 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>= > _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+= 0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100582= nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab = sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_in= ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampol= ine+0xe=0A>>>>>>>>> 921 100583 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>= >>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_si= g+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100584 nfsd nfsd: s= ervice mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_s= ig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>= >>>>>> 921 100585 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_ca= tch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>= >>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>= >>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100586 nfsd nfsd: service mi_swi= tch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>= >>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_st= art+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 10= 0587 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_ru= n_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_tra= mpoline+0xe=0A>>>>>>>>> 921 100588 nfsd nfsd: service mi_switch+0xe1=0A>>= >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wai= t_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_= exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100589 nfsd nfs= d: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wa= it_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x= 87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe= =0A>>>>>>>>> 921 100590 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sle= epq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a= =0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100591 nfsd nfsd: servic= e mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0x= f=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_t= hread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>= > 921 100592 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_s= ignals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>= > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> = fork_trampoline+0xe=0A>>>>>>>>> 921 100593 nfsd nfsd: service mi_switch+0= xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>>= _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0= xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100594 = nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab s= leepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoli= ne+0xe=0A>>>>>>>>> 921 100595 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>= >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig= +0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100596 nfsd nfsd: se= rvice mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_si= g+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>= >>>>> 921 100597 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_cat= ch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>= >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>= >>> fork_trampoline+0xe=0A>>>>>>>>> 921 100598 nfsd nfsd: service mi_swit= ch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>= >>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_sta= rt+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100= 599 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0x= ab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run= _internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_tram= poline+0xe=0A>>>>>>>>> 921 100600 nfsd nfsd: service mi_switch+0xe1=0A>>>= >>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait= _sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100601 nfsd nfsd= : service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A= >>>>>>>>> 921 100602 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq= _catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A= >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>= >>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100603 nfsd nfsd: service mi_= switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>= >>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921= 100604 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signal= s+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_= trampoline+0xe=0A>>>>>>>>> 921 100605 nfsd nfsd: service mi_switch+0xe1= =0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _c= v_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100606 nfs= d nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab slee= pq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+= 0xe=0A>>>>>>>>> 921 100607 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> = sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x= 16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9= a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100608 nfsd nfsd: servi= ce mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0= xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_= thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>= >> 921 100609 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_= signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>= >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>>= fork_trampoline+0xe=0A>>>>>>>>> 921 100610 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>= > _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+= 0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100611= nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab = sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_in= ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampol= ine+0xe=0A>>>>>>>>> 921 100612 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>= >>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_si= g+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100613 nfsd nfsd: s= ervice mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_s= ig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>= >>>>>> 921 100614 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_ca= tch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>= >>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>= >>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100615 nfsd nfsd: service mi_swi= tch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>= >>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_st= art+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 10= 0616 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a _slee= p+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>>>>> nfsrv_getlockfile+0x17= 9 nfsrv_lockctrl+0x21f=0A>>>>>>>>> nfsrvd_lock+0x5b1=0A>>>>>>>>> nfsrvd_d= orpc+0xec6 nfssvc_program+0x554=0A>>>>>>>>> svc_run_internal+0xc77=0A>>>>= >>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>= 921 100617 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>>= svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> f= ork_trampoline+0xe=0A>>>>>>>>> 921 100618 nfsd nfsd: service mi_switch+0x= e1=0A>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x= 9b=0A>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554=0A>>>>>>>>> svc_ru= n_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_tra= mpoline+0xe=0A>>>>>>>>> 921 100619 nfsd nfsd: service mi_switch+0xe1=0A>>= >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wai= t_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_= exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100620 nfsd nfs= d: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wa= it_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x= 87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe= =0A>>>>>>>>> 921 100621 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sle= epq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a= =0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100622 nfsd nfsd: servic= e mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0x= f=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_t= hread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>= > 921 100623 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_s= ignals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>= > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> = fork_trampoline+0xe=0A>>>>>>>>> 921 100624 nfsd nfsd: service mi_switch+0= xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>>= _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0= xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100625 = nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab s= leepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoli= ne+0xe=0A>>>>>>>>> 921 100626 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>= >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig= +0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100627 nfsd nfsd: se= rvice mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_si= g+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>= >>>>> 921 100628 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_cat= ch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>= >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>= >>> fork_trampoline+0xe=0A>>>>>>>>> 921 100629 nfsd nfsd: service mi_swit= ch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>= >>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_sta= rt+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100= 630 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0x= ab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run= _internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_tram= poline+0xe=0A>>>>>>>>> 921 100631 nfsd nfsd: service mi_switch+0xe1=0A>>>= >>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait= _sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100632 nfsd nfsd= : service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A= >>>>>>>>> 921 100633 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq= _catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A= >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>= >>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100634 nfsd nfsd: service mi_= switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>= >>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921= 100635 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signal= s+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_= trampoline+0xe=0A>>>>>>>>> 921 100636 nfsd nfsd: service mi_switch+0xe1= =0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _c= v_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100637 nfs= d nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab slee= pq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+= 0xe=0A>>>>>>>>> 921 100638 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> = sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x= 16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9= a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100639 nfsd nfsd: servi= ce mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0= xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_= thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>= >> 921 100640 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_= signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>= >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>>= fork_trampoline+0xe=0A>>>>>>>>> 921 100641 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>= > _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+= 0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100642= nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab = sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_in= ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampol= ine+0xe=0A>>>>>>>>> 921 100643 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>= >>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_si= g+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100644 nfsd nfsd: s= ervice mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_s= ig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>= >>>>>> 921 100645 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_ca= tch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>= >>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>= >>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100646 nfsd nfsd: service mi_swi= tch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>= >>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_st= art+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 10= 0647 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_ru= n_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_tra= mpoline+0xe=0A>>>>>>>>> 921 100648 nfsd nfsd: service mi_switch+0xe1=0A>>= >>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wai= t_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_= exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100649 nfsd nfs= d: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wa= it_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x= 87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe= =0A>>>>>>>>> 921 100650 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sle= epq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a= =0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100651 nfsd nfsd: servic= e mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0x= f=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_t= hread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>= > 921 100652 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_s= ignals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>= > svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> = fork_trampoline+0xe=0A>>>>>>>>> 921 100653 nfsd nfsd: service mi_switch+0= xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>>= _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0= xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100654 = nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab s= leepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoli= ne+0xe=0A>>>>>>>>> 921 100655 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>= >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig= +0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100656 nfsd nfsd: se= rvice mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_si= g+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>= >>>>> 921 100657 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_cat= ch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>= >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>= >>> fork_trampoline+0xe=0A>>>>>>>>> 921 100658 nfsd nfsd: service mi_swit= ch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>= >>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_sta= rt+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100= 659 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0x= ab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run= _internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_tram= poline+0xe=0A>>>>>>>>> 921 100660 nfsd nfsd: service mi_switch+0xe1=0A>>>= >>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait= _sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100661 nfsd nfsd= : service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A= >>>>>>>>> 921 100662 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq= _catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A= >>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>= >>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100663 nfsd nfsd: service mi_= switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>= >>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921= 100664 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signal= s+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_= trampoline+0xe=0A>>>>>>>>> 921 100665 nfsd nfsd: service mi_switch+0xe1= =0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _c= v_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100666 nfs= d nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 = nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>>>>> nfsrv_setclient+0xbd nfsrvd_se= tclientid+0x3c8=0A>>>>>>>>> nfsrvd_dorpc+0xc76=0A>>>>>>>>> nfssvc_program= +0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>= > fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> =0A>>>>>>>>> Regards,=0A= >>>>>>>>> =0A>>>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>>>> UNIX Systems, Network = and Security Engineer=0A>>>>>>>>> http://www.unix-experience.fr=0A>>>>>>>= >> =0A>>>>>>>>> 15 d=C3=A9cembre 2014 15:18 "Rick Macklem" =0A>>>>>>>>> a=0A>>>>>>>>> =C3=A9crit:=0A>>>>>>>>> =0A>>>>>>>>> L= oic Blot wrote:=0A>>>>>>>>> =0A>>>>>>>>>> For more informations, here is = procstat -kk on nfsd, if you=0A>>>>>>>>>> need=0A>>>>>>>>>> more=0A>>>>>>= >>>> hot datas, tell me.=0A>>>>>>>>>> =0A>>>>>>>>>> Regards, PID TID COMM= TDNAME KSTACK=0A>>>>>>>>>> 918 100529 nfsd nfsd: master mi_switch+0xe1= =0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>= >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_= fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorp= c+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_run+= 0x1de=0A>>>>>>>>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c=0A= >>>>>>>>>> amd64_syscall+0x351=0A>>>>>>>>> =0A>>>>>>>>> Well, most of the= threads are stuck like this one, waiting=0A>>>>>>>>> for=0A>>>>>>>>> a= =0A>>>>>>>>> vnode=0A>>>>>>>>> lock in ZFS. All of them appear to be in z= fs_fhtovp().=0A>>>>>>>>> I`m not a ZFS guy, so I can`t help much. I`ll tr= y changing=0A>>>>>>>>> the=0A>>>>>>>>> subject line=0A>>>>>>>>> to includ= e ZFS vnode lock, so maybe the ZFS guys will take a=0A>>>>>>>>> look.=0A>= >>>>>>>> =0A>>>>>>>>> The only thing I`ve seen suggested is trying:=0A>>>= >>>>>> sysctl vfs.lookup_shared=3D0=0A>>>>>>>>> to disable shared vop_loo= kup()s. Apparently zfs_lookup()=0A>>>>>>>>> doesn`t=0A>>>>>>>>> obey the = vnode locking rules for lookup and rename, according=0A>>>>>>>>> to=0A>>>= >>>>>> the posting I saw.=0A>>>>>>>>> =0A>>>>>>>>> I`ve added a couple of= comments about the other threads=0A>>>>>>>>> below,=0A>>>>>>>>> but=0A>>= >>>>>>> they are all either waiting for an RPC request or waiting for=0A>= >>>>>>>> the=0A>>>>>>>>> threads stuck on the ZFS vnode lock to complete.= =0A>>>>>>>>> =0A>>>>>>>>> rick=0A>>>>>>>>> =0A>>>>>>>>>> 918 100564 nfsd = nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_catch_signals+0xab sleep= q_wait_sig+0xf=0A>>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>>> svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>>> fork_trampolin= e+0xe=0A>>>>>>>>> =0A>>>>>>>>> Fyi, this thread is just waiting for an RP= C to arrive.=0A>>>>>>>>> (Normal)=0A>>>>>>>>> =0A>>>>>>>>>> 918 100565 nf= sd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_catch_signals+0xab sl= eepq_wait_sig+0xf=0A>>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>>> svc_run_in= ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>>> fork_trampo= line+0xe=0A>>>>>>>>>> 918 100566 nfsd nfsd: service mi_switch+0xe1=0A>>>>= >>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>>> _cv_wai= t_sig+0x16a=0A>>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork= _exit+0x9a=0A>>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>>> 918 100567 nfsd = nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_catch_signals+0xab sleep= q_wait_sig+0xf=0A>>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>>> svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>>> fork_trampolin= e+0xe=0A>>>>>>>>>> 918 100568 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>= >>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>>> _cv_wait_s= ig+0x16a=0A>>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_ex= it+0x9a=0A>>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>>> 918 100569 nfsd nfs= d: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_catch_signals+0xab sleepq_w= ait_sig+0xf=0A>>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>>> fork_trampoline+0= xe=0A>>>>>>>>>> 918 100570 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>>= sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>>> _cv_wait_sig+= 0x16a=0A>>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>>> 918 100571 nfsd nfsd: = service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsle= ep+0x66 nfsv4_lock+0x9b=0A>>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x= 554=0A>>>>>>>>>> svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb= fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100572 nfsd nfsd: se= rvice mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep= +0x66 nfsv4_lock+0x9b=0A>>>>>>>>>> nfsrv_setclient+0xbd nfsrvd_setclienti= d+0x3c8=0A>>>>>>>>>> nfsrvd_dorpc+0xc76=0A>>>>>>>>>> nfssvc_program+0x554= svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fo= rk_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> =0A>>>>>>>>> This one (and a= few others) are waiting for the nfsv4_lock.=0A>>>>>>>>> This=0A>>>>>>>>>= happens=0A>>>>>>>>> because other threads are stuck with RPCs in progres= s. (ie.=0A>>>>>>>>> The=0A>>>>>>>>> ones=0A>>>>>>>>> waiting on the vnode= lock in zfs_fhtovp().)=0A>>>>>>>>> For these, the RPC needs to lock out = other threads to do the=0A>>>>>>>>> operation,=0A>>>>>>>>> so it waits fo= r the nfsv4_lock() which can exclusively lock=0A>>>>>>>>> the=0A>>>>>>>>>= NFSv4=0A>>>>>>>>> data structures once all other nfsd threads complete t= heir=0A>>>>>>>>> RPCs=0A>>>>>>>>> in=0A>>>>>>>>> progress.=0A>>>>>>>>> = =0A>>>>>>>>>> 918 100573 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> s= leepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>>>>>> = nfsrvd_dorpc+0x316 nfssvc_program+0x554=0A>>>>>>>>>> svc_run_internal+0xc= 77=0A>>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe= =0A>>>>>>>>> =0A>>>>>>>>> Same as above.=0A>>>>>>>>> =0A>>>>>>>>>> 918 10= 0574 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a slee= plk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_AP= V+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fh= tovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program= +0x554 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>= >>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100575 nfsd nfsd:= service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __loc= kmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock= +0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_= fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_= internal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0= x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100576 nfsd nfsd: service mi_swi= tch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902= =0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>= >>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfs= rvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77= =0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_tramp= oline+0xe=0A>>>>>>>>>> 918 100577 nfsd nfsd: service mi_switch+0xe1=0A>>>= >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> = vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp= +0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x91= 7=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> s= vc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>= >>>>>>> 918 100578 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_= wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3= c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>= >>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> = nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start= +0xb=0A>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100= 579 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleep= lk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV= +0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fht= ovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+= 0x554 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>= >> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100580 nfsd nfsd: = service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lock= mgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+= 0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_f= htovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_i= nternal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x= 9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100581 nfsd nfsd: service mi_swit= ch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902= =0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>= >>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfs= rvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77= =0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_tramp= oline+0xe=0A>>>>>>>>>> 918 100582 nfsd nfsd: service mi_switch+0xe1=0A>>>= >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> = vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp= +0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x91= 7=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> s= vc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>= >>>>>>> 918 100583 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_= wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3= c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>= >>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> = nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start= +0xb=0A>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100= 584 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleep= lk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV= +0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fht= ovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+= 0x554 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>= >> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100585 nfsd nfsd: = service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lock= mgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+= 0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_f= htovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_i= nternal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x= 9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100586 nfsd nfsd: service mi_swit= ch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902= =0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>= >>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfs= rvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77= =0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_tramp= oline+0xe=0A>>>>>>>>>> 918 100587 nfsd nfsd: service mi_switch+0xe1=0A>>>= >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> = vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp= +0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x91= 7=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> s= vc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>= >>>>>>> 918 100588 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_= wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3= c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>= >>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> = nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start= +0xb=0A>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100= 589 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleep= lk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV= +0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fht= ovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+= 0x554 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>= >> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100590 nfsd nfsd: = service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lock= mgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+= 0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_f= htovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_i= nternal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x= 9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100591 nfsd nfsd: service mi_swit= ch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902= =0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>= >>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfs= rvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77= =0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_tramp= oline+0xe=0A>>>>>>>>>> 918 100592 nfsd nfsd: service mi_switch+0xe1=0A>>>= >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> = vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp= +0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x91= 7=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> s= vc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>= >>>>>>> 918 100593 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_= wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3= c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>= >>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> = nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start= +0xb=0A>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100= 594 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleep= lk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV= +0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fht= ovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+= 0x554 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>= >> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100595 nfsd nfsd: = service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lock= mgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+= 0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_f= htovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_i= nternal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x= 9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100596 nfsd nfsd: service mi_swit= ch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902= =0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>= >>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfs= rvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77= =0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_tramp= oline+0xe=0A>>>>>>>>>> 918 100597 nfsd nfsd: service mi_switch+0xe1=0A>>>= >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> = vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp= +0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x91= 7=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> s= vc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>= >>>>>>> 918 100598 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_= wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3= c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>= >>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> = nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start= +0xb=0A>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100= 599 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleep= lk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV= +0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fht= ovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+= 0x554 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>= >> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100600 nfsd nfsd: = service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lock= mgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+= 0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_f= htovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_i= nternal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x= 9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100601 nfsd nfsd: service mi_swit= ch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902= =0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>= >>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfs= rvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77= =0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_tramp= oline+0xe=0A>>>>>>>>>> 918 100602 nfsd nfsd: service mi_switch+0xe1=0A>>>= >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> = vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp= +0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x91= 7=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> s= vc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>= >>>>>>> 918 100603 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_= wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3= c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>= >>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> = nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start= +0xb=0A>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100= 604 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleep= lk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV= +0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fht= ovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+= 0x554 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>= >> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100605 nfsd nfsd: = service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lock= mgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+= 0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_f= htovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_i= nternal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x= 9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100606 nfsd nfsd: service mi_swit= ch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902= =0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>= >>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfs= rvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77= =0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_tramp= oline+0xe=0A>>>>>>>>>> 918 100607 nfsd nfsd: service mi_switch+0xe1=0A>>>= >>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> = vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp= +0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x91= 7=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> s= vc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>= >>>>>> =0A>>>>>>>>> Lots more waiting for the ZFS vnode lock in zfs_fhtov= p().=0A>>>>>>>>> =0A>>>>>>>>> 918 100608 nfsd nfsd: service mi_switch+0xe= 1=0A>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9= b=0A>>>>>>>>> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f=0A>>>>>>>>> nf= srvd_lock+0x5b1=0A>>>>>>>>> nfsrvd_dorpc+0xec6 nfssvc_program+0x554=0A>>>= >>>>>> svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb fork_exit+= 0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100609 nfsd nfsd: service mi_swi= tch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902= =0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>= > zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd= _dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>= >>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0= xe=0A>>>>>>>>> 918 100610 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> s= leepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e=0A>>>>>>>>> vop_stdloc= k+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> nfsvno_advlock+0x119 = nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad=0A>>>>>>>>> nfsrvd_locku+0x283 n= fsrvd_dorpc+0xec6 nfssvc_program+0x554=0A>>>>>>>>> svc_run_internal+0xc77= svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>= >>>>>>> 918 100611 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_w= ait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>>>>> nfsrvd_d= orpc+0x316 nfssvc_program+0x554=0A>>>>>>>>> svc_run_internal+0xc77=0A>>>>= >>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>= 918 100612 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3= a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>>>>> nfsrvd_dorpc+0x= 316 nfssvc_program+0x554=0A>>>>>>>>> svc_run_internal+0xc77=0A>>>>>>>>> s= vc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 10= 0613 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a _slee= p+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>>>>> nfsrvd_dorpc+0x316 nfs= svc_program+0x554=0A>>>>>>>>> svc_run_internal+0xc77=0A>>>>>>>>> svc_thre= ad_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100614 nf= sd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a _sleep+0x287= nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_pro= gram+0x554=0A>>>>>>>>> svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_star= t+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100615 nfsd nfsd= : service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsl= eep+0x66 nfsv4_lock+0x9b=0A>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x= 554=0A>>>>>>>>> svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb f= ork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100616 nfsd nfsd: servi= ce mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x6= 6 nfsv4_lock+0x9b=0A>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554=0A>= >>>>>>>> svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb fork_exi= t+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100617 nfsd nfsd: service mi_s= witch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4= _lock+0x9b=0A>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554=0A>>>>>>>>= > svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a = fork_trampoline+0xe=0A>>>>>>>>> 918 100618 nfsd nfsd: service mi_switch+0= xe1=0A>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0= x9b=0A>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554=0A>>>>>>>>> svc_r= un_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_tr= ampoline+0xe=0A>>>>>>>>> 918 100619 nfsd nfsd: service mi_switch+0xe1=0A>= >>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> = vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+= 0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917= =0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_= thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>= >> 918 100620 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0= x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_L= OCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsv= no_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_pro= gram+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>= >>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100621 nfsd nfsd= : service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsl= eep+0x66 nfsv4_lock+0x9b=0A>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x= 554=0A>>>>>>>>> svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb f= ork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100622 nfsd nfsd: servi= ce mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_ar= gs+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A= >>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc= 8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0x= c77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_tram= poline+0xe=0A>>>>>>>>> 918 100623 nfsd nfsd: service mi_switch+0xe1=0A>>>= >>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>= >>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554=0A>>>>>>>>> svc_run_intern= al+0xc77=0A>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+= 0xe=0A>>>>>>>>> 918 100624 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> = sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlo= ck+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>= >>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>= >> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_sta= rt+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100= 625 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleepl= k+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0= xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+= 0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554= svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork= _exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100626 nfsd nfsd: service = mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+= 0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>= >>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 n= fsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77= =0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampol= ine+0xe=0A>>>>>>>>> 918 100627 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>= >>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_s= tdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d= =0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>= >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread= _start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918= 100628 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sl= eeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_A= PV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fht= ovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0= x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> = fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100629 nfsd nfsd: serv= ice mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_a= rgs+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43= =0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+= 0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal= +0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_t= rampoline+0xe=0A>>>>>>>>> 918 100630 nfsd nfsd: service mi_switch+0xe1=0A= >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>= vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp= +0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917= =0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_= thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>= >> 918 100631 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0= x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_L= OCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsv= no_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_pro= gram+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>= >>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100632 nfsd nfsd= : service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __loc= kmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+= 0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fht= ovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_inte= rnal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fo= rk_trampoline+0xe=0A>>>>>>>>> 918 100633 nfsd nfsd: service mi_switch+0xe= 1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>= >>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fh= tovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0= x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> = svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>= >>>>>> 918 100634 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wa= it+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c V= OP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> = nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc= _program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A= >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100635 nfsd = nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d _= _lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_l= ock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd= _fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_= internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9= a fork_trampoline+0xe=0A>>>>>>>>> 918 100636 nfsd nfsd: service mi_switch= +0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>= >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zf= s_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dor= pc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>= >>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe= =0A>>>>>>>>> 918 100637 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sle= epq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+= 0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>= >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> = nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+= 0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100638= nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0= x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab= _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7= c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 sv= c_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_ex= it+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100639 nfsd nfsd: service mi_= switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x9= 02=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>= >>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsr= vd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A= >>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline= +0xe=0A>>>>>>>>> 918 100640 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>= sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdl= ock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A= >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>= >>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_st= art+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 10= 0641 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleep= lk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+= 0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp= +0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x55= 4 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> for= k_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100642 nfsd nfsd: service= mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args= +0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>= >>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 = nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc7= 7=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampo= line+0xe=0A>>>>>>>>> 918 100643 nfsd nfsd: service mi_switch+0xe1=0A>>>>>= >>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_= stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38= d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>= >>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_threa= d_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 91= 8 100644 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a s= leeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_= APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fh= tovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+= 0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>= fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100645 nfsd nfsd: ser= vice mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_= args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43= =0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+= 0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal= +0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_t= rampoline+0xe=0A>>>>>>>>> 918 100646 nfsd nfsd: service mi_switch+0xe1=0A= >>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>= vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp= +0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917= =0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_= thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>= >> 918 100647 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0= x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_L= OCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsv= no_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_pro= gram+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>= >>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100648 nfsd nfsd= : service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __loc= kmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+= 0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fht= ovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_inte= rnal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fo= rk_trampoline+0xe=0A>>>>>>>>> 918 100649 nfsd nfsd: service mi_switch+0xe= 1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>= >>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fh= tovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0= x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> = svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>= >>>>>> 918 100650 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wa= it+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c V= OP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> = nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc= _program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A= >>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100651 nfsd = nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d _= _lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_l= ock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd= _fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_= internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9= a fork_trampoline+0xe=0A>>>>>>>>> 918 100652 nfsd nfsd: service mi_switch= +0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>= >>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zf= s_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dor= pc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>= >>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe= =0A>>>>>>>>> 918 100653 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sle= epq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+= 0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>= >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> = nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+= 0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100654= nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0= x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab= _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7= c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 sv= c_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_ex= it+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100655 nfsd nfsd: service mi_= switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x9= 02=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>= >>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsr= vd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A= >>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline= +0xe=0A>>>>>>>>> 918 100656 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>= sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdl= ock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A= >>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>= >>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_st= art+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 10= 0657 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleep= lk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+= 0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp= +0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x55= 4 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> for= k_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100658 nfsd nfsd: service= mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args= +0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>= >>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 = nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc7= 7=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampo= line+0xe=0A>>>>>>>>> =0A>>>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>>>> UNIX System= s, Network and Security Engineer=0A>>>>>>>>> http://www.unix-experience.f= r=0A>>>>>>>>> =0A>>>>>>>>> 15 d=C3=A9cembre 2014 13:29 "Lo=C3=AFc Blot"= =0A>>>>>>>>> =0A>>>>>>>>> a=0A>>>>>>>>> =C3= =A9crit:=0A>>>>>>>>> =0A>>>>>>>>> Hmmm...=0A>>>>>>>>> now i'm experiencin= g a deadlock.=0A>>>>>>>>> =0A>>>>>>>>> 0 918 915 0 21 0 12352 3372 zfs D = - 1:48.64 nfsd: server=0A>>>>>>>>> (nfsd)=0A>>>>>>>>> =0A>>>>>>>>> the on= ly issue was to reboot the server, but after rebooting=0A>>>>>>>>> deadlo= ck arrives a second time when i=0A>>>>>>>>> start my jails over NFS.=0A>>= >>>>>>> =0A>>>>>>>>> Regards,=0A>>>>>>>>> =0A>>>>>>>>> Lo=C3=AFc Blot,=0A= >>>>>>>>> UNIX Systems, Network and Security Engineer=0A>>>>>>>>> http://= www.unix-experience.fr=0A>>>>>>>>> =0A>>>>>>>>> 15 d=C3=A9cembre 2014 10:= 07 "Lo=C3=AFc Blot"=0A>>>>>>>>> =0A>>>>>>>>= > a=0A>>>>>>>>> =C3=A9crit:=0A>>>>>>>>> =0A>>>>>>>>> Hi Rick,=0A>>>>>>>>>= after talking with my N+1, NFSv4 is required on our=0A>>>>>>>>> infrastr= ucture.=0A>>>>>>>>> I tried to upgrade NFSv4+ZFS=0A>>>>>>>>> server from = 9.3 to 10.1, i hope this will resolve some=0A>>>>>>>>> issues...=0A>>>>>>= >>> =0A>>>>>>>>> Regards,=0A>>>>>>>>> =0A>>>>>>>>> Lo=C3=AFc Blot,=0A>>>>= >>>>> UNIX Systems, Network and Security Engineer=0A>>>>>>>>> http://www.= unix-experience.fr=0A>>>>>>>>> =0A>>>>>>>>> 10 d=C3=A9cembre 2014 15:36 "= Lo=C3=AFc Blot"=0A>>>>>>>>> =0A>>>>>>>>> a= =0A>>>>>>>>> =C3=A9crit:=0A>>>>>>>>> =0A>>>>>>>>> Hi Rick,=0A>>>>>>>>> th= anks for your suggestion.=0A>>>>>>>>> For my locking bug, rpc.lockd is st= ucked in rpcrecv state on=0A>>>>>>>>> the=0A>>>>>>>>> server. kill -9 doe= sn't affect the=0A>>>>>>>>> process, it's blocked.... (State: Ds)=0A>>>>>= >>>> =0A>>>>>>>>> for the performances=0A>>>>>>>>> =0A>>>>>>>>> NFSv3: 60= Mbps=0A>>>>>>>>> NFSv4: 45Mbps=0A>>>>>>>>> Regards,=0A>>>>>>>>> =0A>>>>>>= >>> Lo=C3=AFc Blot,=0A>>>>>>>>> UNIX Systems, Network and Security Engine= er=0A>>>>>>>>> http://www.unix-experience.fr=0A>>>>>>>>> =0A>>>>>>>>> 10 = d=C3=A9cembre 2014 13:56 "Rick Macklem" =0A>>>>>>>>= > a=0A>>>>>>>>> =C3=A9crit:=0A>>>>>>>>> =0A>>>>>>>>> Loic Blot wrote:=0A>= >>>>>>>> =0A>>>>>>>>>> Hi Rick,=0A>>>>>>>>>> I'm trying NFSv3.=0A>>>>>>>>= >> Some jails are starting very well but now i have an issue=0A>>>>>>>>>>= with=0A>>>>>>>>>> lockd=0A>>>>>>>>>> after some minutes:=0A>>>>>>>>>> = =0A>>>>>>>>>> nfs server 10.10.X.8:/jails: lockd not responding=0A>>>>>>>= >>> nfs server 10.10.X.8:/jails lockd is alive again=0A>>>>>>>>>> =0A>>>>= >>>>>> I look at mbuf, but i seems there is no problem.=0A>>>>>>>>> =0A>>= >>>>>>> Well, if you need locks to be visible across multiple=0A>>>>>>>>>= clients,=0A>>>>>>>>> then=0A>>>>>>>>> I'm afraid you are stuck with usin= g NFSv4 and the=0A>>>>>>>>> performance=0A>>>>>>>>> you=0A>>>>>>>>> get= =0A>>>>>>>>> from it. (There is no way to do file handle affinity for=0A>= >>>>>>>> NFSv4=0A>>>>>>>>> because=0A>>>>>>>>> the read and write ops are= buried in the compound RPC and=0A>>>>>>>>> not=0A>>>>>>>>> easily=0A>>>>= >>>>> recognized.)=0A>>>>>>>>> =0A>>>>>>>>> If the locks don't need to be= visible across multiple=0A>>>>>>>>> clients,=0A>>>>>>>>> I'd=0A>>>>>>>>>= suggest trying the "nolockd" option with nfsv3.=0A>>>>>>>>> =0A>>>>>>>>>= > Here is my rc.conf on server:=0A>>>>>>>>>> =0A>>>>>>>>>> nfs_server_ena= ble=3D"YES"=0A>>>>>>>>>> nfsv4_server_enable=3D"YES"=0A>>>>>>>>>> nfsuser= d_enable=3D"YES"=0A>>>>>>>>>> nfsd_server_flags=3D"-u -t -n 256"=0A>>>>>>= >>>> mountd_enable=3D"YES"=0A>>>>>>>>>> mountd_flags=3D"-r"=0A>>>>>>>>>> = nfsuserd_flags=3D"-usertimeout 0 -force 20"=0A>>>>>>>>>> rpcbind_enable= =3D"YES"=0A>>>>>>>>>> rpc_lockd_enable=3D"YES"=0A>>>>>>>>>> rpc_statd_ena= ble=3D"YES"=0A>>>>>>>>>> =0A>>>>>>>>>> Here is the client:=0A>>>>>>>>>> = =0A>>>>>>>>>> nfsuserd_enable=3D"YES"=0A>>>>>>>>>> nfsuserd_flags=3D"-use= rtimeout 0 -force 20"=0A>>>>>>>>>> nfscbd_enable=3D"YES"=0A>>>>>>>>>> rpc= _lockd_enable=3D"YES"=0A>>>>>>>>>> rpc_statd_enable=3D"YES"=0A>>>>>>>>>> = =0A>>>>>>>>>> Have you got an idea ?=0A>>>>>>>>>> =0A>>>>>>>>>> Regards,= =0A>>>>>>>>>> =0A>>>>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>>>>> UNIX Systems, Ne= twork and Security Engineer=0A>>>>>>>>>> http://www.unix-experience.fr=0A= >>>>>>>>>> =0A>>>>>>>>>> 9 d=C3=A9cembre 2014 04:31 "Rick Macklem" =0A>>>>>>>>>> a=0A>>>>>>>>>> =C3=A9crit:=0A>>>>>>>>>>> Lo= ic Blot wrote:=0A>>>>>>>>>>> =0A>>>>>>>>>>>> Hi rick,=0A>>>>>>>>>>>> =0A>= >>>>>>>>>>> I waited 3 hours (no lag at jail launch) and now I do:=0A>>>>= >>>>>>>> sysrc=0A>>>>>>>>>>>> memcached_flags=3D"-v -m 512"=0A>>>>>>>>>>>= > Command was very very slow...=0A>>>>>>>>>>>> =0A>>>>>>>>>>>> Here is a = dd over NFS:=0A>>>>>>>>>>>> =0A>>>>>>>>>>>> 601062912 bytes transferred i= n 21.060679 secs (28539579=0A>>>>>>>>>>>> bytes/sec)=0A>>>>>>>>>>> =0A>>>= >>>>>>>> Can you try the same read using an NFSv3 mount?=0A>>>>>>>>>>> (I= f it runs much faster, you have probably been bitten by=0A>>>>>>>>>>> the= =0A>>>>>>>>>>> ZFS=0A>>>>>>>>>>> "sequential vs random" read heuristic wh= ich I've been told=0A>>>>>>>>>>> things=0A>>>>>>>>>>> NFS is doing "rando= m" reads without file handle affinity.=0A>>>>>>>>>>> File=0A>>>>>>>>>>> h= andle affinity is very hard to do for NFSv4, so it isn't=0A>>>>>>>>>>> do= ne.)=0A>>>>>>>>> =0A>>>>>>>>> I was actually suggesting that you try the = "dd" over nfsv3=0A>>>>>>>>> to=0A>>>>>>>>> see=0A>>>>>>>>> how=0A>>>>>>>>= > the performance compared with nfsv4. If you do that, please=0A>>>>>>>>>= post=0A>>>>>>>>> the=0A>>>>>>>>> comparable results.=0A>>>>>>>>> =0A>>>>= >>>>> Someday I would like to try and get ZFS's sequential vs=0A>>>>>>>>>= random=0A>>>>>>>>> read=0A>>>>>>>>> heuristic modified and any info on w= hat difference in=0A>>>>>>>>> performance=0A>>>>>>>>> that=0A>>>>>>>>> mi= ght make for NFS would be useful.=0A>>>>>>>>> =0A>>>>>>>>> rick=0A>>>>>>>= >> =0A>>>>>>>>> rick=0A>>>>>>>>> =0A>>>>>>>>> This is quite slow...=0A>>>= >>>>>> =0A>>>>>>>>> You can found some nfsstat below (command isn't finis= hed=0A>>>>>>>>> yet)=0A>>>>>>>>> =0A>>>>>>>>> nfsstat -c -w 1=0A>>>>>>>>>= =0A>>>>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir=0A>>>>>= >>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 4 0 0 0 0 0 16 0=0A>>>>>>>>> 2 0 0 0 0 0= 17 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>= 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 4 0 0 0 0 4 0= =0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 = 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 4 0 0 0 0 0 3 0=0A>>>= >>>>>> 0 0 0 0 0 0 3 0=0A>>>>>>>>> 37 10 0 8 0 0 14 1=0A>>>>>>>>> 18 16 0= 4 1 2 4 0=0A>>>>>>>>> 78 91 0 82 6 12 30 0=0A>>>>>>>>> 19 18 0 2 2 4 2 0= =0A>>>>>>>>> 0 0 0 0 2 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> GtAt= tr Lookup Rdlink Read Write Rename Access Rddir=0A>>>>>>>>> 0 0 0 0 0 0 0= 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 = 1 0 0 0 0 1 0=0A>>>>>>>>> 4 6 0 0 6 0 3 0=0A>>>>>>>>> 2 0 0 0 0 0 0 0=0A>= >>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 1 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 = 1 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>= >> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 = 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0= 0 0 0 0 0 0=0A>>>>>>>>> 6 108 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A= >>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> GtAttr Lookup Rdlink Read Write Ren= ame Access Rddir=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0= =0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 = 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>= >>>>>> 98 54 0 86 11 0 25 0=0A>>>>>>>>> 36 24 0 39 25 0 10 1=0A>>>>>>>>> = 67 8 0 63 63 0 41 0=0A>>>>>>>>> 34 0 0 35 34 0 0 0=0A>>>>>>>>> 75 0 0 75 = 77 0 0 0=0A>>>>>>>>> 34 0 0 35 35 0 0 0=0A>>>>>>>>> 75 0 0 74 76 0 0 0=0A= >>>>>>>>> 33 0 0 34 33 0 0 0=0A>>>>>>>>> 0 0 0 0 5 0 0 0=0A>>>>>>>>> 0 0 = 0 0 0 0 6 0=0A>>>>>>>>> 11 0 0 0 0 0 11 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>= >>>>>>>> 0 17 0 0 0 0 1 0=0A>>>>>>>>> GtAttr Lookup Rdlink Read Write Ren= ame Access Rddir=0A>>>>>>>>> 4 5 0 0 0 0 12 0=0A>>>>>>>>> 2 0 0 0 0 0 26 = 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0= 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>= >>>>>>> 0 4 0 0 0 0 4 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0= 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 4 0 0 0 0 0 2 0=0A>>>>>>>>= > 2 0 0 0 0 0 24 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 = 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0= 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>= >>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir=0A>>>>>>>>> 0= 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 4 0 0 0 0 0 7 0=0A= >>>>>>>>> 2 1 0 0 0 0 1 0=0A>>>>>>>>> 0 0 0 0 2 0 0 0=0A>>>>>>>>> 0 0 0 0= 0 0 0 0=0A>>>>>>>>> 0 0 0 0 6 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>= >>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0= 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 4 = 6 0 0 0 0 3 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 2 0 0 0 0 0 0 0=0A>= >>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 = 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> GtAttr Lookup Rdlink Read= Write Rename Access Rddir=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 = 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>= >>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 4 71 0 0 0 0 0 0=0A>>>>>>>>> 0 1 0 0 0 0= 0 0=0A>>>>>>>>> 2 36 0 0 0 0 1 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>= 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0= =0A>>>>>>>>> 1 0 0 0 0 0 1 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 = 0 0 0 0 0 0=0A>>>>>>>>> 79 6 0 79 79 0 2 0=0A>>>>>>>>> 25 0 0 25 26 0 6 0= =0A>>>>>>>>> 43 18 0 39 46 0 23 0=0A>>>>>>>>> 36 0 0 36 36 0 31 0=0A>>>>>= >>>> 68 1 0 66 68 0 0 0=0A>>>>>>>>> GtAttr Lookup Rdlink Read Write Renam= e Access Rddir=0A>>>>>>>>> 36 0 0 36 36 0 0 0=0A>>>>>>>>> 48 0 0 48 49 0 = 0 0=0A>>>>>>>>> 20 0 0 20 20 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>= > 3 14 0 1 0 0 11 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0= 0=0A>>>>>>>>> 0 4 0 0 0 0 4 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 4 = 22 0 0 0 0 16 0=0A>>>>>>>>> 2 0 0 0 0 0 23 0=0A>>>>>>>>> =0A>>>>>>>>> Reg= ards,=0A>>>>>>>>> =0A>>>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>>>> UNIX Systems, = Network and Security Engineer=0A>>>>>>>>> http://www.unix-experience.fr= =0A>>>>>>>>> =0A>>>>>>>>> 8 d=C3=A9cembre 2014 09:36 "Lo=C3=AFc Blot"=0A>= >>>>>>>> a=0A>>>>>>>>> =C3=A9crit:=0A>>>>>= >>>>> Hi Rick,=0A>>>>>>>>>> I stopped the jails this week-end and started= it this=0A>>>>>>>>>> morning,=0A>>>>>>>>>> i'll=0A>>>>>>>>>> give you so= me stats this week.=0A>>>>>>>>>> =0A>>>>>>>>>> Here is my nfsstat -m outp= ut (with your rsize/wsize=0A>>>>>>>>>> tweaks)=0A>>>>>>>> =0A>>>>>>>> =0A= >>>>>>> =0A>>>>>> =0A>>>>> =0A>>>> =0A>>> =0A>> =0A> nfsv4,tcp,resvport,h= ard,cto,sec=3Dsys,acdirmin=3D3,acdirmax=3D60,acregmin=3D5,acregmax=3D60,n= ametimeo=3D60,negna=0A>>>>>>>> =0A>>>>>>>>> =0A>>>>>>>> =0A>>>>>>>> =0A>>= >>>>> =0A>>>>>> =0A>>>>> =0A>>>> =0A>>> =0A>> =0A> etimeo=3D60,rsize=3D32= 768,wsize=3D32768,readdirsize=3D32768,readahead=3D1,wcommitsize=3D773136,= timeout=3D120,retra=0A>>>>>>>> =0A>>>>>>>>> s=3D2147483647=0A>>>>>>>>> = =0A>>>>>>>>> On server side my disks are on a raid controller which show = a=0A>>>>>>>>> 512b=0A>>>>>>>>> volume and write performances=0A>>>>>>>>> = are very honest (dd if=3D/dev/zero of=3D/jails/test.dd bs=3D4096=0A>>>>>>= >>> count=3D100000000 =3D> 450MBps)=0A>>>>>>>>> =0A>>>>>>>>> Regards,=0A>= >>>>>>>> =0A>>>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>>>> UNIX Systems, Network a= nd Security Engineer=0A>>>>>>>>> http://www.unix-experience.fr=0A>>>>>>>>= > =0A>>>>>>>>> 5 d=C3=A9cembre 2014 15:14 "Rick Macklem" a=0A>>>>>>>>> =C3=A9crit:=0A>>>>>>>>> =0A>>>>>>>>> Loic Blot wrote= :=0A>>>>>>>>> =0A>>>>>>>>> Hi,=0A>>>>>>>>> i'm trying to create a virtual= isation environment based on=0A>>>>>>>>> jails.=0A>>>>>>>>> Those jails a= re stored under a big ZFS pool on a FreeBSD=0A>>>>>>>>> 9.3=0A>>>>>>>>> w= hich=0A>>>>>>>>> export a NFSv4 volume. This NFSv4 volume was mounted on = a=0A>>>>>>>>> big=0A>>>>>>>>> hypervisor (2 Xeon E5v3 + 128GB memory and = 8 ports (but=0A>>>>>>>>> only 1=0A>>>>>>>>> was=0A>>>>>>>>> used at this = time).=0A>>>>>>>>> =0A>>>>>>>>> The problem is simple, my hypervisors run= s 6 jails (used 1%=0A>>>>>>>>> cpu=0A>>>>>>>>> and=0A>>>>>>>>> 10GB RAM a= pproximatively and less than 1MB bandwidth) and=0A>>>>>>>>> works=0A>>>>>= >>>> fine at start but the system slows down and after 2-3 days=0A>>>>>>>= >> become=0A>>>>>>>>> unusable. When i look at top command i see 80-100% = on=0A>>>>>>>>> system=0A>>>>>>>>> and=0A>>>>>>>>> commands are very very = slow. Many process are tagged with=0A>>>>>>>>> nfs_cl*.=0A>>>>>>>>> =0A>>= >>>>>>> To be honest, I would expect the slowness to be because of=0A>>>>= >>>>> slow=0A>>>>>>>>> response=0A>>>>>>>>> from the NFSv4 server, but if= you do:=0A>>>>>>>>> # ps axHl=0A>>>>>>>>> on a client when it is slow an= d post that, it would give us=0A>>>>>>>>> some=0A>>>>>>>>> more=0A>>>>>>>= >> information on where the client side processes are sitting.=0A>>>>>>>>= > If you also do something like:=0A>>>>>>>>> # nfsstat -c -w 1=0A>>>>>>>>= > and let it run for a while, that should show you how many=0A>>>>>>>>> R= PCs=0A>>>>>>>>> are=0A>>>>>>>>> being done and which ones.=0A>>>>>>>>> = =0A>>>>>>>>> # nfsstat -m=0A>>>>>>>>> will show you what your mount is ac= tually using.=0A>>>>>>>>> The only mount option I can suggest trying is= =0A>>>>>>>>> "rsize=3D32768,wsize=3D32768",=0A>>>>>>>>> since some networ= k environments have difficulties with 64K.=0A>>>>>>>>> =0A>>>>>>>>> There= are a few things you can try on the NFSv4 server side,=0A>>>>>>>>> if=0A= >>>>>>>>> it=0A>>>>>>>>> appears=0A>>>>>>>>> that the clients are generat= ing a large RPC load.=0A>>>>>>>>> - disabling the DRC cache for TCP by se= tting=0A>>>>>>>>> vfs.nfsd.cachetcp=3D0=0A>>>>>>>>> - If the server is se= eing a large write RPC load, then=0A>>>>>>>>> "sync=3Ddisabled"=0A>>>>>>>= >> might help, although it does run a risk of data loss when=0A>>>>>>>>> = the=0A>>>>>>>>> server=0A>>>>>>>>> crashes.=0A>>>>>>>>> Then there are a = couple of other ZFS related things (I'm not=0A>>>>>>>>> a=0A>>>>>>>>> ZFS= =0A>>>>>>>>> guy,=0A>>>>>>>>> but these have shown up on the mailing list= s).=0A>>>>>>>>> - make sure your volumes are 4K aligned and ashift=3D12 (= in=0A>>>>>>>>> case a=0A>>>>>>>>> drive=0A>>>>>>>>> that uses 4K sectors = is pretending to be 512byte sectored)=0A>>>>>>>>> - never run over 70-80%= full if write performance is an=0A>>>>>>>>> issue=0A>>>>>>>>> - use a zi= l on an SSD with good write performance=0A>>>>>>>>> =0A>>>>>>>>> The only= NFSv4 thing I can tell you is that it is known that=0A>>>>>>>>> ZFS's=0A= >>>>>>>>> algorithm for determining sequential vs random I/O fails for=0A= >>>>>>>>> NFSv4=0A>>>>>>>>> during writing and this can be a performance = hit. The only=0A>>>>>>>>> workaround=0A>>>>>>>>> is to use NFSv3 mounts, = since file handle affinity=0A>>>>>>>>> apparently=0A>>>>>>>>> fixes=0A>>>= >>>>>> the problem and this is only done for NFSv3.=0A>>>>>>>>> =0A>>>>>>= >>> rick=0A>>>>>>>>> =0A>>>>>>>>> I saw that there are TSO issues with ig= b then i'm trying to=0A>>>>>>>>> disable=0A>>>>>>>>> it with sysctl but t= he situation wasn't solved.=0A>>>>>>>>> =0A>>>>>>>>> Someone has got idea= s ? I can give you more informations if=0A>>>>>>>>> you=0A>>>>>>>>> need.= =0A>>>>>>>>> =0A>>>>>>>>> Thanks in advance.=0A>>>>>>>>> Regards,=0A>>>>>= >>>> =0A>>>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>>>> UNIX Systems, Network and S= ecurity Engineer=0A>>>>>>>>> http://www.unix-experience.fr=0A>>>>>>>>> __= _____________________________________________=0A>>>>>>>>> freebsd-fs@free= bsd.org mailing list=0A>>>>>>>>> http://lists.freebsd.org/mailman/listinf= o/freebsd-fs=0A>>>>>>>>> To unsubscribe, send any mail to=0A>>>>>>>>> "fr= eebsd-fs-unsubscribe@freebsd.org"=0A>>>>>>>>> =0A>>>>>>>>> ______________= _________________________________=0A>>>>>>>>> freebsd-fs@freebsd.org mail= ing list=0A>>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs= =0A>>>>>>>>> To unsubscribe, send any mail to=0A>>>>>>>>> "freebsd-fs-uns= ubscribe@freebsd.org"=0A>>>>>>>>> =0A>>>>>>>>> __________________________= _____________________=0A>>>>>>>>> freebsd-fs@freebsd.org mailing list=0A>= >>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A>>>>>>>>>= To unsubscribe, send any mail to=0A>>>>>>>>> "freebsd-fs-unsubscribe@fre= ebsd.org"=0A>>>>>>>>> =0A>>>>>>>>> ______________________________________= _________=0A>>>>>>>>> freebsd-fs@freebsd.org mailing list=0A>>>>>>>>> htt= p://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A>>>>>>>>> To unsubscr= ibe, send any mail to=0A>>>>>>>>> "freebsd-fs-unsubscribe@freebsd.org"=0A= >>>>>>>>> _______________________________________________=0A>>>>>>>>> fre= ebsd-fs@freebsd.org mailing list=0A>>>>>>>>> http://lists.freebsd.org/mai= lman/listinfo/freebsd-fs=0A>>>>>>>>> To unsubscribe, send any mail to=0A>= >>>>>>>> "freebsd-fs-unsubscribe@freebsd.org"=0A>>>>>>>>> =0A>>>>>>>>> __= _____________________________________________=0A>>>>>>>>> freebsd-fs@free= bsd.org mailing list=0A>>>>>>>>> http://lists.freebsd.org/mailman/listinf= o/freebsd-fs=0A>>>>>>>>> To unsubscribe, send any mail to=0A>>>>>>>>> "fr= eebsd-fs-unsubscribe@freebsd.org"=0A>>>>>>> =0A>>>>>>> __________________= _____________________________=0A>>>>>>> freebsd-fs@freebsd.org mailing li= st=0A>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A>>>>>= >> To unsubscribe, send any mail to=0A>>>>>>> "freebsd-fs-unsubscribe@fre= ebsd.org"