From owner-freebsd-fs@FreeBSD.ORG Mon Jan 5 16:34:59 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E3564CAE for ; Mon, 5 Jan 2015 16:34:59 +0000 (UTC) Received: from smtp.unix-experience.fr (195-154-176-227.rev.poneytelecom.eu [195.154.176.227]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 421A32C5E for ; Mon, 5 Jan 2015 16:34:58 +0000 (UTC) Received: from smtp.unix-experience.fr (unknown [192.168.200.21]) by smtp.unix-experience.fr (Postfix) with ESMTP id 8EC5F2820E; Mon, 5 Jan 2015 16:34:54 +0000 (UTC) X-Virus-Scanned: scanned by unix-experience.fr Received: from smtp.unix-experience.fr ([192.168.200.21]) by smtp.unix-experience.fr (smtp.unix-experience.fr [192.168.200.21]) (amavisd-new, port 10024) with ESMTP id y00zgvTS4pns; Mon, 5 Jan 2015 16:34:46 +0000 (UTC) Received: from mail.unix-experience.fr (repo.unix-experience.fr [192.168.200.30]) by smtp.unix-experience.fr (Postfix) with ESMTPSA id B9183281FB; Mon, 5 Jan 2015 16:34:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=unix-experience.fr; s=uxselect; t=1420475685; bh=BO5NX1GV94nxvoMVxysjXPULK+gObAjTwZebc1DF8jQ=; h=Date:From:Subject:To:Cc:In-Reply-To:References; b=MylQ0Eced1RKNPKlnR7oWf5XaXi7vj+QFUwPGWBQUuT0cTpfR4h4LJOsLy07yaEb/ 4XWGlTab/Ku/yWwdh8z0x21VI+PsVX68kiZchhqgIWAyQ/qRAxlKTBKSNNPFei+f1h 5Xhkx6kWvQvXKN6VMFDD0oWpcdBNRK2zYmvAlHvI= Mime-Version: 1.0 Date: Mon, 05 Jan 2015 16:34:45 +0000 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-ID: X-Mailer: RainLoop/1.7.1.215 From: "=?utf-8?B?TG/Dr2MgQmxvdA==?=" Subject: Re: High Kernel Load with nfsv4 To: "Rick Macklem" In-Reply-To: <956766012.5685731.1420464894657.JavaMail.root@uoguelph.ca> References: <956766012.5685731.1420464894657.JavaMail.root@uoguelph.ca> Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Jan 2015 16:35:00 -0000 Hi Rick,=0Anfsstat -e -s don't show usefull datas on server.=0A=0AServer = Info:=0A Getattr Setattr Lookup Readlink Read Write Cr= eate Remove=0A 26935254 16911 5755728 302 2334920 3673= 866 0 328332=0A Rename Link Symlink Mkdir Rmd= ir Readdir RdirPlus Access=0A 77980 28 0 = 0 3 8900 3 1806052=0A Mknod Fsstat Fsinfo= PathConf Commit LookupP SetClId SetClIdCf=0A 1 1095 = 0 0 614377 8172 8 8=0A Open = OpenAttr OpenDwnGr OpenCfrm DelePurge DeleRet GetFH Lock=0A = 1595299 0 44145 1495 0 0 5197490 63= 5015=0A LockT LockU Close Verify NVerify PutFH PutPu= bFH PutRootFH=0A 0 614919 1270938 0 0 226886= 76 0 5=0A Renew RestoreFH SaveFH Secinfo RelLckOw= n V4Create=0A 42104 197606 275820 0 143 4578= =0AServer:=0ARetfailed Faults Clients=0A 0 0 = 6=0AOpenOwner Opens LockOwner Locks Delegs =0A 32335 145= 448 204 181 0 =0AServer Cache Stats:=0A Inprog = Idem Non-idem Misses CacheSize TCPPeak=0A 0 0 = 1 15082947 60 16522=0A=0AOnly GetAttr and Lookup increase = and it's only every 4-5 seconds and only +2 to +5 into theses values.=0A= =0ANow on client, if i take four processes stack i got=0A=0A PID TID = COMM TDNAME KSTACK =0A63170 1= 02547 mv - mi_switch+0xe1 turnstile_wait+0x4= 2a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65 nfs_lookup+0x3d0 VOP_LOOKUP_= APV+0xa1 lookup+0x59c namei+0x4d4 vn_open_cred+0x21d kern_openat+0x26f am= d64_syscall+0x351 Xfast_syscall+0xfb=0A=0AAnother mv:=0A63140 101738 mv = - mi_switch+0xe1 turnstile_wait+0x42a __mtx_l= ock_sleep+0x253 nfscl_nodeleg+0x65 nfs_lookup+0x3d0 VOP_LOOKUP_APV+0xa1 l= ookup+0x59c namei+0x4d4 kern_statat_vnhook+0xae sys_lstat+0x30 amd64_sysc= all+0x351 Xfast_syscall+0xfb =0A=0A62070 102170 sendmail - = mi_switch+0xe1 sleepq_timedwait+0x3a _sleep+0x26e clnt_vc_call+0= x666 clnt_reconnect_call+0x4fa newnfs_request+0xa8c nfscl_request+0x72 nf= srpc_lookup+0x1fb nfs_lookup+0x508 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei= +0x4d4 kern_statat_vnhook+0xae sys_lstat+0x30 amd64_syscall+0x351 Xfast_s= yscall+0xfb=0A=0A63200 100930 mv - mi_switch= +0xe1 turnstile_wait+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65 nfs_= lookup+0x3d0 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4 kern_statat_vnh= ook+0xae sys_lstat+0x30 amd64_syscall+0x351 Xfast_syscall+0xfb=0A=0AWhen = client is in this state, server was doing nothing special (procstat -kk)= =0A=0APID TID COMM TDNAME KSTACK = =0A 895 100538 nfsd nfsd: master mi_switch+0xe1 s= leepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+= 0x18b svc_run_internal+0x4a1 svc_run+0x1de nfsrvd_nfsd+0x1ca nfssvc_nfsd+= 0x107 sys_nfssvc+0x9c amd64_syscall+0x351 Xfast_syscall+0xfb =0A 895 100= 568 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals= +0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_t= hread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100569 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab slee= pq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_star= t+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100570 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_si= g+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork= _exit+0x9a fork_trampoline+0xe =0A 895 100571 nfsd nfsd: ser= vice mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_= wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= fork_trampoline+0xe =0A 895 100572 nfsd nfsd: service mi= _switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0= x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_tram= poline+0xe =0A 895 100573 nfsd nfsd: service mi_switch+0x= e1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_r= un_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe= =0A 895 100575 nfsd nfsd: service mi_switch+0xe1 sleepq_= catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_interna= l+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 = 100576 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_sign= als+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e sv= c_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100577 nfs= d nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab s= leepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_s= tart+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100578 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait= _sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb f= ork_exit+0x9a fork_trampoline+0xe =0A 895 100579 nfsd nfsd: = service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _= cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0= x9a fork_trampoline+0xe =0A 895 100580 nfsd nfsd: service = mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_si= g+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_t= rampoline+0xe =0A 895 100581 nfsd nfsd: service mi_switch= +0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a sv= c_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+= 0xe =0A 895 100582 nfsd nfsd: service mi_switch+0xe1 slee= pq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_inte= rnal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 8= 95 100583 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_s= ignals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e= svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100584 = nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xa= b sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100585 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_w= ait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0x= b fork_exit+0x9a fork_trampoline+0xe =0A 895 100586 nfsd nfs= d: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0x= f _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exi= t+0x9a fork_trampoline+0xe =0A 895 100587 nfsd nfsd: service= mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait= _sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a for= k_trampoline+0xe =0A 895 100588 nfsd nfsd: service mi_swi= tch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a= svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoli= ne+0xe =0A 895 100589 nfsd nfsd: service mi_switch+0xe1 s= leepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_i= nternal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A= 895 100590 nfsd nfsd: service mi_switch+0xe1 sleepq_catc= h_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x= 87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 1005= 92 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+= 0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_th= read_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100593 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleep= q_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start= +0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100594 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig= +0xf=20_cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb for= k_exit+0x9a fork_trampoline+0xe =0A 895 100595 nfsd nfsd: se= rvice mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv= _wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9= a fork_trampoline+0xe =0A 895 100596 nfsd nfsd: service m= i_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+= 0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_tra= mpoline+0xe =0A 895 100597 nfsd nfsd: service mi_switch+0= xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_= run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0x= e =0A 895 100598 nfsd nfsd: service mi_switch+0xe1 sleepq= _catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895= 100599 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_sig= nals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100600 nf= sd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab = sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_= start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100602 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a fork_trampoline+0xe =0A 895 100603 nfsd nfsd:= service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf = _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a fork_trampoline+0xe =0A 895 100604 nfsd nfsd: service = mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_s= ig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_= trampoline+0xe =0A 895 100605 nfsd nfsd: service mi_switc= h+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a s= vc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline= +0xe =0A 895 100606 nfsd nfsd: service mi_switch+0xe1 sle= epq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A = 895 100607 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_= signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87= e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100608= nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0x= ab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thre= ad_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100609 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_= wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0= xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100610 nfsd nf= sd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0= xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_ex= it+0x9a fork_trampoline+0xe =0A 895 100611 nfsd nfsd: servic= e mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wai= t_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fo= rk_trampoline+0xe =0A 895 100612 nfsd nfsd: service mi_sw= itch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16= a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampol= ine+0xe =0A 895 100613 nfsd nfsd: service mi_switch+0xe1 = sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_= internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe = =0A 895 100614 nfsd nfsd: service mi_switch+0xe1 sleepq_c= atch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 1= 00615 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signa= ls+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc= _thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100617 nfsd= nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sl= eepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_st= art+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100618 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_= sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fo= rk_exit+0x9a fork_trampoline+0xe =0A 895 100619 nfsd nfsd: s= ervice mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _c= v_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x= 9a fork_trampoline+0xe =0A 895 100621 nfsd nfsd: service = mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig= +0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_tr= ampoline+0xe =0A 895 100622 nfsd nfsd: service mi_switch+= 0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0= xe =0A 895 100623 nfsd nfsd: service mi_switch+0xe1 sleep= q_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 89= 5 100624 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100625 n= fsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab= sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100626 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wa= it_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb= fork_exit+0x9a fork_trampoline+0xe =0A 895 100627 nfsd nfsd= : service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf= _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a fork_trampoline+0xe =0A 895 100628 nfsd nfsd: service = mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_= sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork= _trampoline+0xe =0A 895 100629 nfsd nfsd: service mi_swit= ch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a = svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampolin= e+0xe =0A 895 100630 nfsd nfsd: service mi_switch+0xe1 sl= eepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_in= ternal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A = 895 100631 nfsd nfsd: service mi_switch+0xe1 sleepq_catch= _signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 10063= 2 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0= xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100633 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq= _wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+= 0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100634 nfsd n= fsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a fork_trampoline+0xe =0A 895 100635 nfsd nfsd: servi= ce mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wa= it_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a f= ork_trampoline+0xe =0A 895 100636 nfsd nfsd: service mi_s= witch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1= 6a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampo= line+0xe =0A 895 100638 nfsd=20=20=20=20 nfsd: service mi_swi= tch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a= svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoli= ne+0xe =0A 895 100639 nfsd nfsd: service mi_switch+0xe1 s= leepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_i= nternal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A= 895 100640 nfsd nfsd: service mi_switch+0xe1 sleepq_catc= h_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x= 87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 1006= 41 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+= 0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_th= read_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100642 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleep= q_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start= +0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100643 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig= +0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_= exit+0x9a fork_trampoline+0xe =0A 895 100644 nfsd nfsd: serv= ice mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_w= ait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a = fork_trampoline+0xe =0A 895 100645 nfsd nfsd: service mi_= switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x= 16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_tramp= oline+0xe =0A 895 100646 nfsd nfsd: service mi_switch+0xe= 1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_ru= n_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe = =0A 895 100647 nfsd nfsd: service mi_switch+0xe1 sleepq_c= atch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 1= 00648 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signa= ls+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc= _thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100649 nfsd= nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sl= eepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_st= art+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100651 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_= sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fo= rk_exit+0x9a fork_trampoline+0xe =0A 895 100652 nfsd nfsd: s= ervice mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _c= v_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x= 9a fork_trampoline+0xe =0A 895 100653 nfsd nfsd: service = mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig= +0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_tr= ampoline+0xe =0A 895 100654 nfsd nfsd: service mi_switch+= 0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0= xe =0A 895 100655 nfsd nfsd: service mi_switch+0xe1 sleep= q_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 89= 5 100656 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100657 n= fsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab= sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100658 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wa= it_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb= fork_exit+0x9a fork_trampoline+0xe =0A 895 100659 nfsd nfsd= : service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf= _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a fork_trampoline+0xe =0A 895 100661 nfsd nfsd: service = mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_= sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork= _trampoline+0xe =0A 895 100662 nfsd nfsd: service mi_swit= ch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a = svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampolin= e+0xe =0A 895 100684 nfsd nfsd: service mi_switch+0xe1 sl= eepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_in= ternal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A = 895 100685 nfsd nfsd: service mi_switch+0xe1 sleepq_catch= _signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 10068= 6 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0= xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100797 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq= _wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+= 0xb fork_exit+0x9a fork_trampoline+0xe =0A 895 100798 nfsd n= fsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a fork_trampoline+0xe =0A 895 100799 nfsd nfsd: servi= ce mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wa= it_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a f= ork_trampoline+0xe =0A 895 100800 nfsd nfsd: service mi_s= witch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1= 6a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampo= line+0xe =0A 895 100801 nfsd nfsd: service mi_switch+0xe1= sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run= _internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe= =0A=0AI really think it's a client side problem, maybe a lookup problem.= =0A=0ARegards,=0A=0ALo=C3=AFc Blot,=0AUNIX Systems, Network and Security = Engineer=0Ahttp://www.unix-experience.fr=0A=0A5 janvier 2015 14:35 "Rick = Macklem" a =C3=A9crit: =0A> Loic Blot wrote:=0A> = =0A>> Hi,=0A>> happy new year Rick and @freebsd-fs.=0A>> =0A>> After some= days, i looked my NFSv4.1 mount. At server start it was=0A>> calm, but a= fter 4 days, here is the top stat...=0A>> =0A>> CPU: 0.0% user, 0.0% nice= , 100% system, 0.0% interrupt, 0.0%=0A>> idle=0A>> =0A>> Definitively i t= hink it's a problem on client side. What can i look=0A>> into running ker= nel to resolve this issue ?=0A> =0A> Well, I'd start with:=0A> # nfsstat = -e -s=0A> - run repeatedly on the server (once every N seconds in a loop)= .=0A> Then look at the output, comparing the counts and see which RPCs=0A= > are being performed by the client(s). You are looking for which=0A> RPC= s are being done a lot. (If one RPC is almost 100% of the load,=0A> then = it might be a client/caching issue for whatever that RPC is=0A> doing.)= =0A> =0A> Also look at the Open/Lock counts near the end of the output.= =0A> If the # of Opens/Locks is large, it may be possible to reduce the= =0A> CPU overheads by using larger hash tables.=0A> =0A> Then you need to= profile the server kernel to see where the CPU=0A> is being used.=0A> Ho= pefully someone else can fill you in on how to do that, because=0A> I'll = admit I don't know how to.=0A> Basically you are looking to see if the CP= U is being used in=0A> the NFS server code or ZFS.=0A> =0A> Good luck wit= h it, rick=0A> =0A>> Regards,=0A>> =0A>> Lo=C3=AFc Blot,=0A>> UNIX System= s, Network and Security Engineer=0A>> http://www.unix-experience.fr=0A>> = =0A>> 30 d=C3=A9cembre 2014 16:16 "Lo=C3=AFc Blot" a=0A>> =C3=A9crit:=0A>>> Hi Rick,=0A>>> i upgraded my jail host = from FreeBSD 9.3 to 10.1 to use NFS v4.1=0A>>> (mountoptions:=0A>>> rw,rs= ize=3D32768,wsize=3D32768,tcp,nfsv4,minorversion=3D1)=0A>>> =0A>>> Perfor= mance is quite stable but it's slow. Not as slow as before=0A>>> but slow= ... services was launched=0A>>> but no client are using them and system C= PU % was 10-50%.=0A>>> =0A>>> I don't see anything on NFSv4.1 server, it'= s perfectly stable and=0A>>> functionnal.=0A>>> =0A>>> Regards,=0A>>> =0A= >>> Lo=C3=AFc Blot,=0A>>> UNIX Systems, Network and Security Engineer=0A>= >> http://www.unix-experience.fr=0A>>> =0A>>> 23 d=C3=A9cembre 2014 00:20= "Rick Macklem" a=0A>>> =C3=A9crit:=0A>>> =0A>>>> = Loic Blot wrote:=0A>>>> =0A>>>>> Hi,=0A>>>>> =0A>>>>> To clarify because = of our exchanges. Here are the current sysctl=0A>>>>> options for server:= =0A>>>>> =0A>>>>> vfs.nfsd.enable_nobodycheck=3D0=0A>>>>> vfs.nfsd.enable= _nogroupcheck=3D0=0A>>>>> =0A>>>>> vfs.nfsd.maxthreads=3D200=0A>>>>> vfs.= nfsd.tcphighwater=3D10000=0A>>>>> vfs.nfsd.tcpcachetimeo=3D300=0A>>>>> vf= s.nfsd.server_min_nfsvers=3D4=0A>>>>> =0A>>>>> kern.maxvnodes=3D10000000= =0A>>>>> kern.ipc.maxsockbuf=3D4194304=0A>>>>> net.inet.tcp.sendbuf_max= =3D4194304=0A>>>>> net.inet.tcp.recvbuf_max=3D4194304=0A>>>>> =0A>>>>> vf= s.lookup_shared=3D0=0A>>>>> =0A>>>>> Regards,=0A>>>>> =0A>>>>> Lo=C3=AFc = Blot,=0A>>>>> UNIX Systems, Network and Security Engineer=0A>>>>> http://= www.unix-experience.fr=0A>>>>> =0A>>>>> 22 d=C3=A9cembre 2014 09:42 "Lo= =C3=AFc Blot" =0A>>>>> a=0A>>>>> =C3=A9crit= :=0A>>>>> =0A>>>>> Hi Rick,=0A>>>>> my 5 jails runs this weekend and now = i have some stats on this=0A>>>>> monday.=0A>>>>> =0A>>>>> Hopefully dead= lock was fixed, yeah, but everything isn't good :(=0A>>>>> =0A>>>>> On NF= Sv4 server (FreeBSD 10.1) system uses 35% CPU=0A>>>>> =0A>>>>> As i can s= ee this is because of nfsd:=0A>>>>> =0A>>>>> 918 root 96 20 0 12352K 3372= K rpcsvc 6 51.4H=0A>>>>> 273.68% nfsd: server (nfsd)=0A>>>>> =0A>>>>> If = i look at dmesg i see:=0A>>>>> nfsd server cache flooded, try increasing = vfs.nfsd.tcphighwater=0A>>>> =0A>>>> Well, you have a couple of choices:= =0A>>>> 1 - Use NFSv4.1 (add "minorversion=3D1" to your mount options).= =0A>>>> (NFSv4.1 avoids use of the DRC and instead uses something=0A>>>> = called sessions. See below.)=0A>>>> OR=0A>>>> =0A>>>>> vfs.nfsd.tcphighwa= ter was set to 10000, i increase it to 15000=0A>>>> =0A>>>> 2 - Bump vfs.= nfsd.tcphighwater way up, until you no longer see=0A>>>> "nfs server cach= e flooded" messages. (I think Garrett Wollman uses=0A>>>> 100000. (You ma= y still see quite a bit of CPU overheads.)=0A>>>> =0A>>>> OR=0A>>>> =0A>>= >> 3 - Set vfs.nfsd.cachetcp=3D0 (which disables the DRC and gets rid=0A>= >>> of the CPU overheads). However, there is a risk of data corruption=0A= >>>> if you have a client->server network partitioning of a moderate=0A>>= >> duration, because a non-idempotent RPC may get redone, becasue=0A>>>> = the client times out waiting for a reply. If a non-idempotent=0A>>>> RPC = gets done twice on the server, data corruption can happen.=0A>>>> (The DR= C provides improved correctness, but does add overhead.)=0A>>>> =0A>>>> I= f #1 works for you, it is the preferred solution, since Sessions=0A>>>> i= n NFSv4.1 solves the correctness problem in a good, space bound=0A>>>> wa= y. A session basically has N (usually 32 or 64) slots and only=0A>>>> all= ows one outstanding RPC/slot. As such, it can cache the=0A>>>> previous= =0A>>>> reply for each slot (32 or 64 of them) and guarantee "exactly=0A>= >>> once"=0A>>>> RPC semantics.=0A>>>> =0A>>>> rick=0A>>>> =0A>>>>> Here = is 'nfsstat -s' output:=0A>>>>> =0A>>>>> Server Info:=0A>>>>> Getattr Set= attr Lookup Readlink Read Write Create=0A>>>>> Remove=0A>>>>> 12600652 18= 12 2501097 156 1386423 1983729 123=0A>>>>> 162067=0A>>>>> Rename Link Sym= link Mkdir Rmdir Readdir RdirPlus=0A>>>>> Access=0A>>>>> 36762 9 0 0 0 31= 47 0=0A>>>>> 623524=0A>>>>> Mknod Fsstat Fsinfo PathConf Commit=0A>>>>> 0= 0 0 0 328117=0A>>>>> Server Ret-Failed=0A>>>>> 0=0A>>>>> Server Faults= =0A>>>>> 0=0A>>>>> Server Cache Stats:=0A>>>>> Inprog Idem Non-idem Misse= s=0A>>>>> 0 0 0 12635512=0A>>>>> Server Write Gathering:=0A>>>>> WriteOps= WriteRPC Opsaved=0A>>>>> 1983729 1983729 0=0A>>>>> =0A>>>>> And here is = 'procstat -kk' for nfsd (server)=0A>>>>> =0A>>>>> 918 100528 nfsd nfsd: m= aster mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_timedwait_s= ig+0x10=0A>>>>> _cv_timedwait_sig_sbt+0x18b svc_run_internal+0x4a1 svc_ru= n+0x1de=0A>>>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c=0A>>>= >> amd64_syscall+0x351 Xfast_syscall+0xfb=0A>>>>> 918 100568 nfsd nfsd: s= ervice mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0= xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb= fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100569 nfsd nfsd:= service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig= +0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0= xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100570 nfsd nfs= d: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_s= ig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start= +0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100571 nfsd n= fsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait= _sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_sta= rt+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100572 nfsd= nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wa= it_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_s= tart+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100573 nf= sd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_= wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100574 = nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleep= q_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thre= ad_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 10057= 5 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sle= epq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_th= read_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100= 576 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab s= leepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_= thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 1= 00577 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e sv= c_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918= 100578 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0x= ab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 9= 18 100579 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+= 0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87= e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>>= 918 100580 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signal= s+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x= 87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>= >> 918 100581 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_sign= als+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+= 0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>= >>>> 918 100582 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_interna= l+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe= =0A>>>>> 918 100583 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc= h_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0= xe=0A>>>>> 918 100584 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_ca= tch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_i= nternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline= +0xe=0A>>>>> 918 100585 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_= catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run= _internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoli= ne+0xe=0A>>>>> 918 100586 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_r= un_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampo= line+0xe=0A>>>>> 918 100587 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sle= epq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram= poline+0xe=0A>>>>> 918 100588 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s= leepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> s= vc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tr= ampoline+0xe=0A>>>>> 918 100589 nfsd nfsd: service mi_switch+0xe1=0A>>>>>= sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>>= svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_= trampoline+0xe=0A>>>>> 918 100590 nfsd nfsd: service mi_switch+0xe1=0A>>>= >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>= >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> for= k_trampoline+0xe=0A>>>>> 918 100591 nfsd nfsd: service mi_switch+0xe1=0A>= >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>= >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> f= ork_trampoline+0xe=0A>>>>> 918 100592 nfsd nfsd: service mi_switch+0xe1= =0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a= =0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>= >> fork_trampoline+0xe=0A>>>>> 918 100593 nfsd nfsd: service mi_switch+0x= e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1= 6a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>= >>>> fork_trampoline+0xe=0A>>>>> 918 100594 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0= x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100595 nfsd nfsd: service mi_swi= tch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_s= ig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x= 9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100596 nfsd nfsd: service mi_s= witch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait= _sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100597 nfsd nfsd: service mi= _switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wa= it_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exi= t+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100598 nfsd nfsd: service = mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_= wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100599 nfsd nfsd: servic= e mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _c= v_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork= _exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100600 nfsd nfsd: serv= ice mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf = _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fo= rk_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100601 nfsd nfsd: se= rvice mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0x= f _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100602 nfsd nfsd: = service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0x= b fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100603 nfsd nfsd= : service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_si= g+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+= 0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100604 nfsd nf= sd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_= sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_star= t+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100605 nfsd = nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_st= art+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100606 nfs= d nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_w= ait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_= start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100607 n= fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq= _wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100608= nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab slee= pq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 1006= 09 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sl= eepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_t= hread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 10= 0610 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab = sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc= _thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 = 100611 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xa= b sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 91= 8 100612 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e= svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> = 918 100613 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>= > 918 100614 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signa= ls+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0= x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>= >>> 918 100615 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_sig= nals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A= >>>>> 918 100616 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_s= ignals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe= =0A>>>>> 918 100617 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc= h_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0= xe=0A>>>>> 918 100618 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_ca= tch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_i= nternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline= +0xe=0A>>>>> 918 100619 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_= catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run= _internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoli= ne+0xe=0A>>>>> 918 100620 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_r= un_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampo= line+0xe=0A>>>>> 918 100621 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sle= epq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram= poline+0xe=0A>>>>> 918 100622 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s= leepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> s= vc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tr= ampoline+0xe=0A>>>>> 918 100623 nfsd nfsd: service mi_switch+0xe1=0A>>>>>= sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>>= svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_= trampoline+0xe=0A>>>>> 918 100624 nfsd nfsd: service mi_switch+0xe1=0A>>>= >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>= >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> for= k_trampoline+0xe=0A>>>>> 918 100625 nfsd nfsd: service mi_switch+0xe1=0A>= >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>= >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> f= ork_trampoline+0xe=0A>>>>> 918 100626 nfsd nfsd: service mi_switch+0xe1= =0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a= =0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>= >> fork_trampoline+0xe=0A>>>>> 918 100627 nfsd nfsd: service mi_switch+0x= e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1= 6a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>= >>>> fork_trampoline+0xe=0A>>>>> 918 100628 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0= x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100629 nfsd nfsd: service mi_swi= tch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_s= ig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x= 9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100630 nfsd nfsd: service mi_s= witch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait= _sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100631 nfsd nfsd: service mi= _switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wa= it_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exi= t+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100632 nfsd nfsd: service = mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_= wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100633 nfsd nfsd: servic= e mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _c= v_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork= _exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100634 nfsd nfsd: serv= ice mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf = _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fo= rk_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100635 nfsd nfsd: se= rvice mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0x= f _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100636 nfsd nfsd: = service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0x= b fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100637 nfsd nfsd= : service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_si= g+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+= 0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100638 nfsd nf= sd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_= sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_star= t+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100639 nfsd = nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wai= t_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_st= art+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100640 nfs= d nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_w= ait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_= start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100641 n= fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq= _wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_threa= d_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 100642= nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab slee= pq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 1006= 43 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sl= eepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_t= hread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 10= 0644 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab = sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc= _thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 = 100645 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xa= b sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 91= 8 100646 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e= svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> = 918 100647 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>= > 918 100648 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signa= ls+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0= x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>= >>> 918 100649 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_sig= nals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A= >>>>> 918 100650 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_s= ignals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe= =0A>>>>> 918 100651 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc= h_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0= xe=0A>>>>> 918 100652 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_ca= tch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run_i= nternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline= +0xe=0A>>>>> 918 100653 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_= catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_run= _internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoli= ne+0xe=0A>>>>> 918 100654 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleep= q_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc_r= un_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampo= line+0xe=0A>>>>> 918 100655 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sle= epq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram= poline+0xe=0A>>>>> 918 100656 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s= leepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>> s= vc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tr= ampoline+0xe=0A>>>>> 918 100657 nfsd nfsd: service mi_switch+0xe1=0A>>>>>= sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>>>= svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_= trampoline+0xe=0A>>>>> 918 100658 nfsd nfsd: service mi_switch+0xe1=0A>>>= >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>>>= >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> for= k_trampoline+0xe=0A>>>>> 918 100659 nfsd nfsd: service mi_switch+0xe1=0A>= >>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A>= >>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> f= ork_trampoline+0xe=0A>>>>> 918 100660 nfsd nfsd: service mi_switch+0xe1= =0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a= =0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>= >> fork_trampoline+0xe=0A>>>>> 918 100661 nfsd nfsd: service mi_switch+0x= e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1= 6a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>= >>>> fork_trampoline+0xe=0A>>>>> 918 100662 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0= x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0A>>>>> fork_trampoline+0xe=0A>>>>> ---=0A>>>>> =0A>>>>> Now if we look = at client (FreeBSD 9.3)=0A>>>>> =0A>>>>> We see system was very busy and = do many and many interrupts=0A>>>>> =0A>>>>> CPU: 0.0% user, 0.0% nice, 3= 7.8% system, 51.2% interrupt, 11.0%=0A>>>>> idle=0A>>>>> =0A>>>>> A look = at process list shows that there are many sendmail process=0A>>>>> in=0A>= >>>> state nfstry=0A>>>>> =0A>>>>> nfstry 18 32:27 0.88% sendmail: Queue = runner@00:30:00 for=0A>>>>> /var/spool/clientm=0A>>>>> =0A>>>>> Here is '= nfsstat -c' output:=0A>>>>> =0A>>>>> Client Info:=0A>>>>> Rpc Counts:=0A>= >>>> Getattr Setattr Lookup Readlink Read Write Create=0A>>>>> Remove=0A>= >>>> 1051347 1724 2494481 118 903902 1901285 162676=0A>>>>> 161899=0A>>>>= > Rename Link Symlink Mkdir Rmdir Readdir RdirPlus=0A>>>>> Access=0A>>>>>= 36744 2 0 114 40 3131 0=0A>>>>> 544136=0A>>>>> Mknod Fsstat Fsinfo PathC= onf Commit=0A>>>>> 9 0 0 0 245821=0A>>>>> Rpc Info:=0A>>>>> TimedOut Inva= lid X Replies Retries Requests=0A>>>>> 0 0 0 0 8356557=0A>>>>> Cache Info= :=0A>>>>> Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits=0A= >>>>> Misses=0A>>>>> 108754455 491475 54229224 2437229 46814561 821723 51= 32123=0A>>>>> 1871871=0A>>>>> BioRLHits Misses BioD Hits Misses DirE Hits= Misses Accs Hits=0A>>>>> Misses=0A>>>>> 144035 118 53736 2753 27813 1 57= 238839=0A>>>>> 544205=0A>>>>> =0A>>>>> If you need more things, tell me, = i let the PoC in this state.=0A>>>>> =0A>>>>> Thanks=0A>>>>> =0A>>>>> Reg= ards,=0A>>>>> =0A>>>>> Lo=C3=AFc Blot,=0A>>>>> UNIX Systems, Network and = Security Engineer=0A>>>>> http://www.unix-experience.fr=0A>>>>> =0A>>>>> = 21 d=C3=A9cembre 2014 01:33 "Rick Macklem" a=0A>>>= >> =C3=A9crit:=0A>>>>> =0A>>>>> Loic Blot wrote:=0A>>>>> =0A>>>>>> Hi Ric= k,=0A>>>>>> ok, i don't need locallocks, i haven't understand option was = for=0A>>>>>> that=0A>>>>>> usage, i removed it.=0A>>>>>> I do more tests = on monday.=0A>>>>>> Thanks for the deadlock fix, for other people :)=0A>>= >>> =0A>>>>> Good. Please let us know if running with=0A>>>>> vfs.nfsd.en= able_locallocks=3D0=0A>>>>> gets rid of the deadlocks? (I think it fixes = the one you saw.)=0A>>>>> =0A>>>>> On the performance side, you might als= o want to try different=0A>>>>> values=0A>>>>> of=0A>>>>> readahead, if t= he Linux client has such a mount option. (With the=0A>>>>> NFSv4-ZFS sequ= ential vs random I/O heuristic, I have no idea what=0A>>>>> the=0A>>>>> o= ptimal readahead value would be.)=0A>>>>> =0A>>>>> Good luck with it and = please let us know how it goes, rick=0A>>>>> ps: I now have a patch to fi= x the deadlock when=0A>>>>> vfs.nfsd.enable_locallocks=3D1=0A>>>>> is set= . I'll post it for anyone who is interested after I put it=0A>>>>> throug= h some testing.=0A>>>>> =0A>>>>> --=0A>>>>> Best regards,=0A>>>>> Lo=C3= =AFc BLOT,=0A>>>>> UNIX systems, security and network engineer=0A>>>>> ht= tp://www.unix-experience.fr=0A>>>>> =0A>>>>> Le jeudi 18 d=C3=A9cembre 20= 14 =C3=A0 19:46 -0500, Rick Macklem a =C3=A9crit :=0A>>>>> =0A>>>>> Loic = Blot wrote:=0A>>>>>> Hi rick,=0A>>>>>> i tried to start a LXC container o= n Debian Squeeze from my=0A>>>>>> freebsd=0A>>>>>> ZFS+NFSv4 server and i= also have a deadlock on nfsd=0A>>>>>> (vfs.lookup_shared=3D0). Deadlock = procs each time i launch a=0A>>>>>> squeeze=0A>>>>>> container, it seems = (3 tries, 3 fails).=0A>>>>> =0A>>>>> Well, I`ll take a look at this `proc= stat -kk`, but the only thing=0A>>>>> I`ve seen posted w.r.t. avoiding de= adlocks in ZFS is to not use=0A>>>>> nullfs. (I have no idea if you are u= sing any nullfs mounts, but=0A>>>>> if so, try getting rid of them.)=0A>>= >>> =0A>>>>> Here`s a high level post about the ZFS and vnode locking pro= blem,=0A>>>>> but there is no patch available, as far as I know.=0A>>>>> = =0A>>>>> http://docs.FreeBSD.org/cgi/mid.cgi?54739F41.8030407=0A>>>>> =0A= >>>>> rick=0A>>>>> =0A>>>>> 921 - D 0:00.02 nfsd: server (nfsd)=0A>>>>> = =0A>>>>> Here is the procstat -kk=0A>>>>> =0A>>>>> PID TID COMM TDNAME KS= TACK=0A>>>>> 921 100538 nfsd nfsd: master mi_switch+0xe1=0A>>>>> sleepq_w= ait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e=0A>>>>> vop_stdlock+0x3c VOP_= LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> nfsvno_advlock+0x119 nfsrv_dolocal+0= x84 nfsrv_lockctrl+0x14ad=0A>>>>> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 n= fssvc_program+0x554=0A>>>>> svc_run_internal+0xc77 svc_run+0x1de nfsrvd_n= fsd+0x1ca=0A>>>>> nfssvc_nfsd+0x107 sys_nfssvc+0x9c=0A>>>>> 921 100572 nf= sd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_= wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e sv= c_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921= 100573 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0x= ab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_interna= l+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe= =0A>>>>> 921 100574 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc= h_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram= poline+0xe=0A>>>>> 921 100575 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s= leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a= =0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>= >> fork_trampoline+0xe=0A>>>>> 921 100576 nfsd nfsd: service mi_switch+0x= e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait= _sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100577 nfsd nfsd: service mi= _switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>= > _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100578 nfsd nfsd: = service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_= start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100579 n= fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq= _wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 92= 1 100580 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe= =0A>>>>> 921 100581 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc= h_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram= poline+0xe=0A>>>>> 921 100582 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s= leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a= =0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>= >> fork_trampoline+0xe=0A>>>>> 921 100583 nfsd nfsd: service mi_switch+0x= e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait= _sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100584 nfsd nfsd: service mi= _switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>= > _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100585 nfsd nfsd: = service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_= start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100586 n= fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq= _wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 92= 1 100587 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe= =0A>>>>> 921 100588 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc= h_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram= poline+0xe=0A>>>>> 921 100589 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s= leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a= =0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>= >> fork_trampoline+0xe=0A>>>>> 921 100590 nfsd nfsd: service mi_switch+0x= e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait= _sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100591 nfsd nfsd: service mi= _switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>= > _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100592 nfsd nfsd: = service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_= start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100593 n= fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq= _wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 92= 1 100594 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe= =0A>>>>> 921 100595 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc= h_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram= poline+0xe=0A>>>>> 921 100596 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s= leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a= =0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>= >> fork_trampoline+0xe=0A>>>>> 921 100597 nfsd nfsd: service mi_switch+0x= e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait= _sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100598 nfsd nfsd: service mi= _switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>= > _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100599 nfsd nfsd: = service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_= start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100600 n= fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq= _wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 92= 1 100601 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe= =0A>>>>> 921 100602 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc= h_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram= poline+0xe=0A>>>>> 921 100603 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s= leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a= =0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>= >> fork_trampoline+0xe=0A>>>>> 921 100604 nfsd nfsd: service mi_switch+0x= e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait= _sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100605 nfsd nfsd: service mi= _switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>= > _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100606 nfsd nfsd: = service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_= start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100607 n= fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq= _wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 92= 1 100608 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe= =0A>>>>> 921 100609 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc= h_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram= poline+0xe=0A>>>>> 921 100610 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s= leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a= =0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>= >> fork_trampoline+0xe=0A>>>>> 921 100611 nfsd nfsd: service mi_switch+0x= e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait= _sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100612 nfsd nfsd: service mi= _switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>= > _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100613 nfsd nfsd: = service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_= start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100614 n= fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq= _wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 92= 1 100615 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe= =0A>>>>> 921 100616 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait= +0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>> nfsrv_getlockfi= le+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1=0A>>>>> nfsrvd_dorpc+0xec= 6 nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0x= b fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 921 100617 nfsd nfsd: servic= e mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A= >>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+= 0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100618 nfsd nf= sd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmslee= p+0x66 nfsv4_lock+0x9b=0A>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 sv= c_run_internal+0xc77=0A>>>>> svc_thread_start+0xb fork_exit+0x9a fork_tra= mpoline+0xe=0A>>>>> 921 100619 nfsd nfsd: service mi_switch+0xe1=0A>>>>> = sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a= =0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>= >> fork_trampoline+0xe=0A>>>>> 921 100620 nfsd nfsd: service mi_switch+0x= e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait= _sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100621 nfsd nfsd: service mi= _switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>= > _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100622 nfsd nfsd: = service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_= start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100623 n= fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq= _wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 92= 1 100624 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe= =0A>>>>> 921 100625 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc= h_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram= poline+0xe=0A>>>>> 921 100626 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s= leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a= =0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>= >> fork_trampoline+0xe=0A>>>>> 921 100627 nfsd nfsd: service mi_switch+0x= e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait= _sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100628 nfsd nfsd: service mi= _switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>= > _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100629 nfsd nfsd: = service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_= start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100630 n= fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq= _wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 92= 1 100631 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe= =0A>>>>> 921 100632 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc= h_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram= poline+0xe=0A>>>>> 921 100633 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s= leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a= =0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>= >> fork_trampoline+0xe=0A>>>>> 921 100634 nfsd nfsd: service mi_switch+0x= e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait= _sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100635 nfsd nfsd: service mi= _switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>= > _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100636 nfsd nfsd: = service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_= start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100637 n= fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq= _wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 92= 1 100638 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe= =0A>>>>> 921 100639 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc= h_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram= poline+0xe=0A>>>>> 921 100640 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s= leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a= =0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>= >> fork_trampoline+0xe=0A>>>>> 921 100641 nfsd nfsd: service mi_switch+0x= e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait= _sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100642 nfsd nfsd: service mi= _switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>= > _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100643 nfsd nfsd: = service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_= start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100644 n= fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq= _wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 92= 1 100645 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe= =0A>>>>> 921 100646 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc= h_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram= poline+0xe=0A>>>>> 921 100647 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s= leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a= =0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>= >> fork_trampoline+0xe=0A>>>>> 921 100648 nfsd nfsd: service mi_switch+0x= e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait= _sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100649 nfsd nfsd: service mi= _switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>= > _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100650 nfsd nfsd: = service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_= start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100651 n= fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq= _wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 92= 1 100652 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe= =0A>>>>> 921 100653 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc= h_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram= poline+0xe=0A>>>>> 921 100654 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s= leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a= =0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>= >> fork_trampoline+0xe=0A>>>>> 921 100655 nfsd nfsd: service mi_switch+0x= e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait= _sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100656 nfsd nfsd: service mi= _switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>= > _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100657 nfsd nfsd: = service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_= start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100658 n= fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq= _wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 92= 1 100659 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe= =0A>>>>> 921 100660 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catc= h_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_tram= poline+0xe=0A>>>>> 921 100661 nfsd nfsd: service mi_switch+0xe1=0A>>>>> s= leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a= =0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>= >> fork_trampoline+0xe=0A>>>>> 921 100662 nfsd nfsd: service mi_switch+0x= e1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>> _cv_wait= _sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100663 nfsd nfsd: service mi= _switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>= > _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100664 nfsd nfsd: = service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e svc_thread_= start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 921 100665 n= fsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_catch_signals+0xab sleepq= _wait_sig+0xf=0A>>>>> _cv_wait_sig+0x16a=0A>>>>> svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 92= 1 100666 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a _slee= p+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>> nfsrv_setclient+0xbd nfsrv= d_setclientid+0x3c8=0A>>>>> nfsrvd_dorpc+0xc76=0A>>>>> nfssvc_program+0x5= 54 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+= 0x9a fork_trampoline+0xe=0A>>>>> =0A>>>>> Regards,=0A>>>>> =0A>>>>> Lo=C3= =AFc Blot,=0A>>>>> UNIX Systems, Network and Security Engineer=0A>>>>> ht= tp://www.unix-experience.fr=0A>>>>> =0A>>>>> 15 d=C3=A9cembre 2014 15:18 = "Rick Macklem" a=0A>>>>> =C3=A9crit:=0A>>>>> =0A>>= >>> Loic Blot wrote:=0A>>>>> =0A>>>>>> For more informations, here is pro= cstat -kk on nfsd, if you=0A>>>>>> need=0A>>>>>> more=0A>>>>>> hot datas,= tell me.=0A>>>>>> =0A>>>>>> Regards, PID TID COMM TDNAME KSTACK=0A>>>>>>= 918 100529 nfsd nfsd: master mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sl= eeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+= 0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c = nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run= _internal+0xc77 svc_run+0x1de=0A>>>>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x10= 7 sys_nfssvc+0x9c=0A>>>>>> amd64_syscall+0x351=0A>>>>> =0A>>>>> Well, mos= t of the threads are stuck like this one, waiting for=0A>>>>> a=0A>>>>> v= node=0A>>>>> lock in ZFS. All of them appear to be in zfs_fhtovp().=0A>>>= >> I`m not a ZFS guy, so I can`t help much. I`ll try changing the=0A>>>>>= subject line=0A>>>>> to include ZFS vnode lock, so maybe the ZFS guys wi= ll take a=0A>>>>> look.=0A>>>>> =0A>>>>> The only thing I`ve seen suggest= ed is trying:=0A>>>>> sysctl vfs.lookup_shared=3D0=0A>>>>> to disable sha= red vop_lookup()s. Apparently zfs_lookup()=0A>>>>> doesn`t=0A>>>>> obey t= he vnode locking rules for lookup and rename, according=0A>>>>> to=0A>>>>= > the posting I saw.=0A>>>>> =0A>>>>> I`ve added a couple of comments abo= ut the other threads below,=0A>>>>> but=0A>>>>> they are all either waiti= ng for an RPC request or waiting for=0A>>>>> the=0A>>>>> threads stuck on= the ZFS vnode lock to complete.=0A>>>>> =0A>>>>> rick=0A>>>>> =0A>>>>>> = 918 100564 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_catch_signal= s+0xab sleepq_wait_sig+0xf=0A>>>>>> _cv_wait_sig+0x16a=0A>>>>>> svc_run_i= nternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>> fork_trampolin= e+0xe=0A>>>>> =0A>>>>> Fyi, this thread is just waiting for an RPC to arr= ive. (Normal)=0A>>>>> =0A>>>>>> 918 100565 nfsd nfsd: service mi_switch+0= xe1=0A>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>> _cv_w= ait_sig+0x16a=0A>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a=0A>>>>>> fork_trampoline+0xe=0A>>>>>> 918 100566 nfsd nfsd: serv= ice mi_switch+0xe1=0A>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A>>>>>> _cv_wait_sig+0x16a=0A>>>>>> svc_run_internal+0x87e svc_thread_s= tart+0xb fork_exit+0x9a=0A>>>>>> fork_trampoline+0xe=0A>>>>>> 918 100567 = nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_catch_signals+0xab slee= pq_wait_sig+0xf=0A>>>>>> _cv_wait_sig+0x16a=0A>>>>>> svc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>> fork_trampoline+0xe=0A>>>= >>> 918 100568 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf=0A>>>>>> _cv_wait_sig+0x16a=0A>>>>>> svc_r= un_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>> fork_tramp= oline+0xe=0A>>>>>> 918 100569 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> = sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>> _cv_wait_sig+0x16a= =0A>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>= >>>> fork_trampoline+0xe=0A>>>>>> 918 100570 nfsd nfsd: service mi_switch= +0xe1=0A>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>> _cv= _wait_sig+0x16a=0A>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork= _exit+0x9a=0A>>>>>> fork_trampoline+0xe=0A>>>>>> 918 100571 nfsd nfsd: se= rvice mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x6= 6 nfsv4_lock+0x9b=0A>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_ru= n_internal+0xc77=0A>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampo= line+0xe=0A>>>>>> 918 100572 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> s= leepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>> nfsr= v_setclient+0xbd nfsrvd_setclientid+0x3c8=0A>>>>>> nfsrvd_dorpc+0xc76=0A>= >>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_sta= rt+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> =0A>>>>> This = one (and a few others) are waiting for the nfsv4_lock.=0A>>>>> This=0A>>>= >> happens=0A>>>>> because other threads are stuck with RPCs in progress.= (ie. The=0A>>>>> ones=0A>>>>> waiting on the vnode lock in zfs_fhtovp().= )=0A>>>>> For these, the RPC needs to lock out other threads to do the=0A= >>>>> operation,=0A>>>>> so it waits for the nfsv4_lock() which can exclu= sively lock the=0A>>>>> NFSv4=0A>>>>> data structures once all other nfsd= threads complete their RPCs=0A>>>>> in=0A>>>>> progress.=0A>>>>> =0A>>>>= >> 918 100573 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a= _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>> nfsrvd_dorpc+0x316 = nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb= fork_exit+0x9a fork_trampoline+0xe=0A>>>>> =0A>>>>> Same as above.=0A>>>= >> =0A>>>>>> 918 100574 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq= _wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c V= OP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno= _fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+= 0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_= exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100575 nfsd nfsd: service mi_s= witch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902= =0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_= fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x= 917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thr= ead_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 10= 0576 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+= 0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _= vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_f= htovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_inter= nal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_tram= poline+0xe=0A>>>>>> 918 100577 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>= sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock= +0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>>= nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_p= rogram+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>= > fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100578 nfsd nfsd: servi= ce mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+= 0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>= zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dor= pc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> sv= c_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 9= 18 100579 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sle= eplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0= xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c n= fsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_= internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork= _trampoline+0xe=0A>>>>>> 918 100580 nfsd nfsd: service mi_switch+0xe1=0A>= >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_st= dlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>= >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfs= svc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A= >>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100581 nfsd nfsd: = service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_= args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>= >>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrv= d_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>= >> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>= >>> 918 100582 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3= a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_= APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0= x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc= _run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a= fork_trampoline+0xe=0A>>>>>> 918 100583 nfsd nfsd: service mi_switch+0xe= 1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> v= op_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38= d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>= > nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0= xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100584 nfsd n= fsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __loc= kmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x4= 3=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 = nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77= =0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0x= e=0A>>>>>> 918 100585 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_w= ait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP= _LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_f= htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x= 554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_ex= it+0x9a fork_trampoline+0xe=0A>>>>>> 918 100586 nfsd nfsd: service mi_swi= tch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>= >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhto= vp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917= =0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread= _start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 10058= 7 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x1= 5d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_= lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhto= vp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal= +0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampol= ine+0xe=0A>>>>>> 918 100588 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sl= eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x= 3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nf= svno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_prog= ram+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> f= ork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100589 nfsd nfsd: service = mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x9= 02=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zf= s_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+= 0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_t= hread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 = 100590 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleepl= k+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab= _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd= _fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_int= ernal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_tr= ampoline+0xe=0A>>>>>> 918 100591 nfsd nfsd: service mi_switch+0xe1=0A>>>>= >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlo= ck+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>= >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc= _program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>= >>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100592 nfsd nfsd: ser= vice mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_arg= s+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>= >> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_d= orpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> = svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>= 918 100593 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a s= leeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV= +0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c= nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_ru= n_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fo= rk_trampoline+0xe=0A>>>>>> 918 100594 nfsd nfsd: service mi_switch+0xe1= =0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vo= p_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d= =0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>= nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0x= b=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100595 nfsd nf= sd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lock= mgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43= =0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 n= fsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A= >>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe= =0A>>>>>> 918 100596 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wa= it+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_= LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fh= tovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x5= 54 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exi= t+0x9a fork_trampoline+0xe=0A>>>>>> 918 100597 nfsd nfsd: service mi_swit= ch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>= >>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtov= p+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917= =0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread= _start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 10059= 8 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x1= 5d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_= lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhto= vp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal= +0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampol= ine+0xe=0A>>>>>> 918 100599 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sl= eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x= 3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nf= svno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_prog= ram+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> f= ork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100600 nfsd nfsd: service = mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x9= 02=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zf= s_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+= 0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_t= hread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 = 100601 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleepl= k+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab= _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd= _fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_int= ernal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_tr= ampoline+0xe=0A>>>>>> 918 100602 nfsd nfsd: service mi_switch+0xe1=0A>>>>= >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlo= ck+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>= >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc= _program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>= >>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100603 nfsd nfsd: ser= vice mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_arg= s+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>= >> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_d= orpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> = svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>= 918 100604 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a s= leeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV= +0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c= nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_ru= n_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fo= rk_trampoline+0xe=0A>>>>>> 918 100605 nfsd nfsd: service mi_switch+0xe1= =0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vo= p_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d= =0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>= nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0x= b=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100606 nfsd nf= sd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lock= mgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43= =0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 n= fsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A= >>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe= =0A>>>>>> 918 100607 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wa= it+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_= LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fh= tovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x5= 54 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exi= t+0x9a fork_trampoline+0xe=0A>>>>> =0A>>>>> Lots more waiting for the ZFS= vnode lock in zfs_fhtovp().=0A>>>>> =0A>>>>> 918 100608 nfsd nfsd: servi= ce mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nf= sv4_lock+0x9b=0A>>>>> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd= _lock+0x5b1=0A>>>>> nfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_inter= nal+0xc77=0A>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe= =0A>>>>> 918 100609 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait= +0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOC= K1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+= 0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x554 svc= _run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+0x9a f= ork_trampoline+0xe=0A>>>>> 918 100610 nfsd nfsd: service mi_switch+0xe1= =0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e=0A>>>>> vop_= stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> nfsvno_advlock+0x11= 9 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad=0A>>>>> nfsrvd_locku+0x283 nfs= rvd_dorpc+0xec6 nfssvc_program+0x554=0A>>>>> svc_run_internal+0xc77 svc_t= hread_start+0xb fork_exit+0x9a=0A>>>>> fork_trampoline+0xe=0A>>>>> 918 10= 0611 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a _sleep+0x= 287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>> nfsrvd_dorpc+0x316 nfssvc_prog= ram+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb fork_exit+0= x9a fork_trampoline+0xe=0A>>>>> 918 100612 nfsd nfsd: service mi_switch+0= xe1=0A>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b= =0A>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77= =0A>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>>> = 918 100613 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a _sl= eep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>> nfsrvd_dorpc+0x316 nfssv= c_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb fork_= exit+0x9a fork_trampoline+0xe=0A>>>>> 918 100614 nfsd nfsd: service mi_sw= itch+0xe1=0A>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock= +0x9b=0A>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0x= c77=0A>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>= >> 918 100615 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a = _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>> nfsrvd_dorpc+0x316 nf= ssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb fo= rk_exit+0x9a fork_trampoline+0xe=0A>>>>> 918 100616 nfsd nfsd: service mi= _switch+0xe1=0A>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_l= ock+0x9b=0A>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal= +0xc77=0A>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A= >>>>> 918 100617 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x= 3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>> nfsrvd_dorpc+0x316= nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb= fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918 100618 nfsd nfsd: service= mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv= 4_lock+0x9b=0A>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_inter= nal+0xc77=0A>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe= =0A>>>>> 918 100619 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait= +0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOC= K1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+= 0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x554 svc= _run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+0x9a f= ork_trampoline+0xe=0A>>>>> 918 100620 nfsd nfsd: service mi_switch+0xe1= =0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vop_= stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A= >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfss= vc_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>= >>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918 100621 nfsd nfsd: serv= ice mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 n= fsv4_lock+0x9b=0A>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_in= ternal+0xc77=0A>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+= 0xe=0A>>>>> 918 100622 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_w= ait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c VOP_= LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhto= vp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x554 = svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+0x9= a fork_trampoline+0xe=0A>>>>> 918 100623 nfsd nfsd: service mi_switch+0xe= 1=0A>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A= >>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77=0A>>= >>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918 1= 00624 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+= 0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _v= n_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhto= vp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_internal+= 0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoline= +0xe=0A>>>>> 918 100625 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_= wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c VOP= _LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fht= ovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x554= svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+0x= 9a fork_trampoline+0xe=0A>>>>> 918 100626 nfsd nfsd: service mi_switch+0x= e1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vo= p_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d= =0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> n= fssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb= =0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918 100627 nfsd nfsd:= service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_= args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>= >>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_d= orpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> sv= c_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918= 100628 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleepl= k+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab = _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fh= tovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_interna= l+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoli= ne+0xe=0A>>>>> 918 100629 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleep= q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c V= OP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_f= htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x5= 54 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+= 0x9a fork_trampoline+0xe=0A>>>>> 918 100630 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> = vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38= d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> = nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb= =0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918 100631 nfsd nfsd:= service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_= args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>= >>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_d= orpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> sv= c_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918= 100632 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleepl= k+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab = _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fh= tovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_interna= l+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoli= ne+0xe=0A>>>>> 918 100633 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleep= q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c V= OP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_f= htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x5= 54 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+= 0x9a fork_trampoline+0xe=0A>>>>> 918 100634 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> = vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38= d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> = nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb= =0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918 100635 nfsd nfsd:= service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_= args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>= >>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_d= orpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> sv= c_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918= 100636 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleepl= k+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab = _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fh= tovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_interna= l+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoli= ne+0xe=0A>>>>> 918 100637 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleep= q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c V= OP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_f= htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x5= 54 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+= 0x9a fork_trampoline+0xe=0A>>>>> 918 100638 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> = vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38= d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> = nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb= =0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918 100639 nfsd nfsd:= service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_= args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>= >>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_d= orpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> sv= c_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918= 100640 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleepl= k+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab = _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fh= tovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_interna= l+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoli= ne+0xe=0A>>>>> 918 100641 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleep= q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c V= OP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_f= htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x5= 54 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+= 0x9a fork_trampoline+0xe=0A>>>>> 918 100642 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> = vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38= d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> = nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb= =0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918 100643 nfsd nfsd:= service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_= args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>= >>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_d= orpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> sv= c_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918= 100644 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleepl= k+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab = _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fh= tovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_interna= l+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoli= ne+0xe=0A>>>>> 918 100645 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleep= q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c V= OP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_f= htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x5= 54 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+= 0x9a fork_trampoline+0xe=0A>>>>> 918 100646 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> = vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38= d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> = nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb= =0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918 100647 nfsd nfsd:= service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_= args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>= >>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_d= orpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> sv= c_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918= 100648 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleepl= k+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab = _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fh= tovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_interna= l+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoli= ne+0xe=0A>>>>> 918 100649 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleep= q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c V= OP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_f= htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x5= 54 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+= 0x9a fork_trampoline+0xe=0A>>>>> 918 100650 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> = vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38= d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> = nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb= =0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918 100651 nfsd nfsd:= service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_= args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>= >>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_d= orpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> sv= c_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918= 100652 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleepl= k+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab = _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fh= tovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_interna= l+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoli= ne+0xe=0A>>>>> 918 100653 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleep= q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c V= OP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_f= htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x5= 54 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+= 0x9a fork_trampoline+0xe=0A>>>>> 918 100654 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> = vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38= d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> = nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb= =0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918 100655 nfsd nfsd:= service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_= args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>= >>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_d= orpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> sv= c_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> 918= 100656 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleepq_wait+0x3a sleepl= k+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab = _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fh= tovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x554 svc_run_interna= l+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+0x9a fork_trampoli= ne+0xe=0A>>>>> 918 100657 nfsd nfsd: service mi_switch+0xe1=0A>>>>> sleep= q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> vop_stdlock+0x3c V= OP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38d=0A>>>>> nfsvno_f= htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> nfssvc_program+0x5= 54 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb=0A>>>>> fork_exit+= 0x9a fork_trampoline+0xe=0A>>>>> 918 100658 nfsd nfsd: service mi_switch+= 0xe1=0A>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>> = vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>> zfs_fhtovp+0x38= d=0A>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>> = nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>> svc_thread_start+0xb= =0A>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> =0A>>>>> Lo=C3=AFc Bl= ot,=0A>>>>> UNIX Systems, Network and Security Engineer=0A>>>>> http://ww= w.unix-experience.fr=0A>>>>> =0A>>>>> 15 d=C3=A9cembre 2014 13:29 "Lo=C3= =AFc Blot"=0A>>>>> =0A>>>>> a=0A>>>>> =C3= =A9crit:=0A>>>>> =0A>>>>> Hmmm...=0A>>>>> now i'm experiencing a deadlock= .=0A>>>>> =0A>>>>> 0 918 915 0 21 0 12352 3372 zfs D - 1:48.64 nfsd: serv= er=0A>>>>> (nfsd)=0A>>>>> =0A>>>>> the only issue was to reboot the serve= r, but after rebooting=0A>>>>> deadlock arrives a second time when i=0A>>= >>> start my jails over NFS.=0A>>>>> =0A>>>>> Regards,=0A>>>>> =0A>>>>> L= o=C3=AFc Blot,=0A>>>>> UNIX Systems, Network and Security Engineer=0A>>>>= > http://www.unix-experience.fr=0A>>>>> =0A>>>>> 15 d=C3=A9cembre 2014 10= :07 "Lo=C3=AFc Blot"=0A>>>>> =0A>>>>> a=0A>= >>>> =C3=A9crit:=0A>>>>> =0A>>>>> Hi Rick,=0A>>>>> after talking with my = N+1, NFSv4 is required on our=0A>>>>> infrastructure.=0A>>>>> I tried to = upgrade NFSv4+ZFS=0A>>>>> server from 9.3 to 10.1, i hope this will resol= ve some=0A>>>>> issues...=0A>>>>> =0A>>>>> Regards,=0A>>>>> =0A>>>>> Lo= =C3=AFc Blot,=0A>>>>> UNIX Systems, Network and Security Engineer=0A>>>>>= http://www.unix-experience.fr=0A>>>>> =0A>>>>> 10 d=C3=A9cembre 2014 15:= 36 "Lo=C3=AFc Blot"=0A>>>>> =0A>>>>> a=0A>>= >>> =C3=A9crit:=0A>>>>> =0A>>>>> Hi Rick,=0A>>>>> thanks for your suggest= ion.=0A>>>>> For my locking bug, rpc.lockd is stucked in rpcrecv state on= =0A>>>>> the=0A>>>>> server. kill -9 doesn't affect the=0A>>>>> process, = it's blocked.... (State: Ds)=0A>>>>> =0A>>>>> for the performances=0A>>>>= > =0A>>>>> NFSv3: 60Mbps=0A>>>>> NFSv4: 45Mbps=0A>>>>> Regards,=0A>>>>> = =0A>>>>> Lo=C3=AFc Blot,=0A>>>>> UNIX Systems, Network and Security Engin= eer=0A>>>>> http://www.unix-experience.fr=0A>>>>> =0A>>>>> 10 d=C3=A9cemb= re 2014 13:56 "Rick Macklem" =0A>>>>> a=0A>>>>> =C3= =A9crit:=0A>>>>> =0A>>>>> Loic Blot wrote:=0A>>>>> =0A>>>>>> Hi Rick,=0A>= >>>>> I'm trying NFSv3.=0A>>>>>> Some jails are starting very well but no= w i have an issue=0A>>>>>> with=0A>>>>>> lockd=0A>>>>>> after some minute= s:=0A>>>>>> =0A>>>>>> nfs server 10.10.X.8:/jails: lockd not responding= =0A>>>>>> nfs server 10.10.X.8:/jails lockd is alive again=0A>>>>>> =0A>>= >>>> I look at mbuf, but i seems there is no problem.=0A>>>>> =0A>>>>> We= ll, if you need locks to be visible across multiple=0A>>>>> clients,=0A>>= >>> then=0A>>>>> I'm afraid you are stuck with using NFSv4 and the=0A>>>>= > performance=0A>>>>> you=0A>>>>> get=0A>>>>> from it. (There is no way t= o do file handle affinity for=0A>>>>> NFSv4=0A>>>>> because=0A>>>>> the r= ead and write ops are buried in the compound RPC and=0A>>>>> not=0A>>>>> = easily=0A>>>>> recognized.)=0A>>>>> =0A>>>>> If the locks don't need to b= e visible across multiple=0A>>>>> clients,=0A>>>>> I'd=0A>>>>> suggest tr= ying the "nolockd" option with nfsv3.=0A>>>>> =0A>>>>>> Here is my rc.con= f on server:=0A>>>>>> =0A>>>>>> nfs_server_enable=3D"YES"=0A>>>>>> nfsv4_= server_enable=3D"YES"=0A>>>>>> nfsuserd_enable=3D"YES"=0A>>>>>> nfsd_serv= er_flags=3D"-u -t -n 256"=0A>>>>>> mountd_enable=3D"YES"=0A>>>>>> mountd_= flags=3D"-r"=0A>>>>>> nfsuserd_flags=3D"-usertimeout 0 -force 20"=0A>>>>>= > rpcbind_enable=3D"YES"=0A>>>>>> rpc_lockd_enable=3D"YES"=0A>>>>>> rpc_s= tatd_enable=3D"YES"=0A>>>>>> =0A>>>>>> Here is the client:=0A>>>>>> =0A>>= >>>> nfsuserd_enable=3D"YES"=0A>>>>>> nfsuserd_flags=3D"-usertimeout 0 -f= orce 20"=0A>>>>>> nfscbd_enable=3D"YES"=0A>>>>>> rpc_lockd_enable=3D"YES"= =0A>>>>>> rpc_statd_enable=3D"YES"=0A>>>>>> =0A>>>>>> Have you got an ide= a ?=0A>>>>>> =0A>>>>>> Regards,=0A>>>>>> =0A>>>>>> Lo=C3=AFc Blot,=0A>>>>= >> UNIX Systems, Network and Security Engineer=0A>>>>>> http://www.unix-e= xperience.fr=0A>>>>>> =0A>>>>>> 9 d=C3=A9cembre 2014 04:31 "Rick Macklem"= =0A>>>>>> a=0A>>>>>> =C3=A9crit:=0A>>>>>>> Loic Bl= ot wrote:=0A>>>>>>> =0A>>>>>>>> Hi rick,=0A>>>>>>>> =0A>>>>>>>> I waited = 3 hours (no lag at jail launch) and now I do:=0A>>>>>>>> sysrc=0A>>>>>>>>= memcached_flags=3D"-v -m 512"=0A>>>>>>>> Command was very very slow...= =0A>>>>>>>> =0A>>>>>>>> Here is a dd over NFS:=0A>>>>>>>> =0A>>>>>>>> 601= 062912 bytes transferred in 21.060679 secs (28539579=0A>>>>>>>> bytes/sec= )=0A>>>>>>> =0A>>>>>>> Can you try the same read using an NFSv3 mount?=0A= >>>>>>> (If it runs much faster, you have probably been bitten by=0A>>>>>= >> the=0A>>>>>>> ZFS=0A>>>>>>> "sequential vs random" read heuristic whic= h I've been told=0A>>>>>>> things=0A>>>>>>> NFS is doing "random" reads w= ithout file handle affinity.=0A>>>>>>> File=0A>>>>>>> handle affinity is = very hard to do for NFSv4, so it isn't=0A>>>>>>> done.)=0A>>>>> =0A>>>>> = I was actually suggesting that you try the "dd" over nfsv3=0A>>>>> to=0A>= >>>> see=0A>>>>> how=0A>>>>> the performance compared with nfsv4. If you = do that, please=0A>>>>> post=0A>>>>> the=0A>>>>> comparable results.=0A>>= >>> =0A>>>>> Someday I would like to try and get ZFS's sequential vs=0A>>= >>> random=0A>>>>> read=0A>>>>> heuristic modified and any info on what d= ifference in=0A>>>>> performance=0A>>>>> that=0A>>>>> might make for NFS = would be useful.=0A>>>>> =0A>>>>> rick=0A>>>>> =0A>>>>> rick=0A>>>>> =0A>= >>>> This is quite slow...=0A>>>>> =0A>>>>> You can found some nfsstat be= low (command isn't finished=0A>>>>> yet)=0A>>>>> =0A>>>>> nfsstat -c -w 1= =0A>>>>> =0A>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir=0A>= >>>> 0 0 0 0 0 0 0 0=0A>>>>> 4 0 0 0 0 0 16 0=0A>>>>> 2 0 0 0 0 0 17 0=0A= >>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>= >>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 4 0 0 0 0 4 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>= >>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>= >> 4 0 0 0 0 0 3 0=0A>>>>> 0 0 0 0 0 0 3 0=0A>>>>> 37 10 0 8 0 0 14 1=0A>= >>>> 18 16 0 4 1 2 4 0=0A>>>>> 78 91 0 82 6 12 30 0=0A>>>>> 19 18 0 2 2 4= 2 0=0A>>>>> 0 0 0 0 2 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> GtAttr Looku= p Rdlink Read Write Rename Access Rddir=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0= 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 1 0 0 0 0 1 0=0A>>>>> 4 = 6 0 0 6 0 3 0=0A>>>>> 2 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0=200 0 0 0=0A>>>>> 1= 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 1 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 = 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0= 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 = 0 0 0 0 0 0=0A>>>>> 6 108 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0= 0 0 0 0 0 0=0A>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir= =0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0= =0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0= =0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 98 54 0 86 11 0 25 0=0A>>>>> 36 24 0 39 = 25 0 10 1=0A>>>>> 67 8 0 63 63 0 41 0=0A>>>>> 34 0 0 35 34 0 0 0=0A>>>>> = 75 0 0 75 77 0 0 0=0A>>>>> 34 0 0 35 35 0 0 0=0A>>>>> 75 0 0 74 76 0 0 0= =0A>>>>> 33 0 0 34 33 0 0 0=0A>>>>> 0 0 0 0 5 0 0 0=0A>>>>> 0 0 0 0 0 0 6= 0=0A>>>>> 11 0 0 0 0 0 11 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 17 0 0 0 0= 1 0=0A>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir=0A>>>>> = 4 5 0 0 0 0 12 0=0A>>>>> 2 0 0 0 0 0 26 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>>= 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> = 0 0 0 0 0 0 0 0=0A>>>>> 0 4 0 0 0 0 4 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0= 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 4 0 0 0 0 0 2 0=0A>>>>> 2 = 0 0 0 0 0 24 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 = 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0= 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> GtAttr Lookup Rdlink Read Wr= ite Rename Access Rddir=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0= =0A>>>>> 4 0 0 0 0 0 7 0=0A>>>>> 2 1 0 0 0 0 1 0=0A>>>>> 0 0 0 0 2 0 0 0= =0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 6 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0= =0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0= =0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 4 6 0 0 0 0 3 0= =0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 2 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0= =0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0= =0A>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir=0A>>>>> 0 0 = 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0= 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 4 71 0 0 0 0 0 0=0A>>>>> 0 1 0= 0 0 0 0 0=0A>>>>> 2 36 0 0 0 0 1 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0= 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 1 0 0 = 0 0 0 1 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 79 6 0 = 79 79 0 2 0=0A>>>>> 25 0 0 25 26 0 6 0=0A>>>>> 43 18 0 39 46 0 23 0=0A>>>= >> 36 0 0 36 36 0 31 0=0A>>>>> 68 1 0 66 68 0 0 0=0A>>>>> GtAttr Lookup R= dlink Read Write Rename Access Rddir=0A>>>>> 36 0 0 36 36 0 0 0=0A>>>>> 4= 8 0 0 48 49 0 0 0=0A>>>>> 20 0 0 20 20 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>= >>> 3 14 0 1 0 0 11 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>= >>>> 0 4 0 0 0 0 4 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 4 22 0 0 0 0 16 0=0A= >>>>> 2 0 0 0 0 0 23 0=0A>>>>> =0A>>>>> Regards,=0A>>>>> =0A>>>>> Lo=C3= =AFc Blot,=0A>>>>> UNIX Systems, Network and Security Engineer=0A>>>>> ht= tp://www.unix-experience.fr=0A>>>>> =0A>>>>> 8 d=C3=A9cembre 2014 09:36 "= Lo=C3=AFc Blot"=0A>>>>> a=0A>>>>> =C3=A9cr= it:=0A>>>>>> Hi Rick,=0A>>>>>> I stopped the jails this week-end and star= ted it this=0A>>>>>> morning,=0A>>>>>> i'll=0A>>>>>> give you some stats = this week.=0A>>>>>> =0A>>>>>> Here is my nfsstat -m output (with your rsi= ze/wsize=0A>>>>>> tweaks)=0A>>>> =0A>>>> =0A>>> =0A>> =0A> nfsv4,tcp,resv= port,hard,cto,sec=3Dsys,acdirmin=3D3,acdirmax=3D60,acregmin=3D5,acregmax= =3D60,nametimeo=3D60,negna=0A>>>> =0A>>>>> =0A>>>> =0A>>>> =0A>>> =0A>> = =0A> etimeo=3D60,rsize=3D32768,wsize=3D32768,readdirsize=3D32768,readahea= d=3D1,wcommitsize=3D773136,timeout=3D120,retra=0A>>>> =0A>>>>> s=3D214748= 3647=0A>>>>> =0A>>>>> On server side my disks are on a raid controller wh= ich show a=0A>>>>> 512b=0A>>>>> volume and write performances=0A>>>>> are= very honest (dd if=3D/dev/zero of=3D/jails/test.dd bs=3D4096=0A>>>>> cou= nt=3D100000000 =3D> 450MBps)=0A>>>>> =0A>>>>> Regards,=0A>>>>> =0A>>>>> L= o=C3=AFc Blot,=0A>>>>> UNIX Systems, Network and Security Engineer=0A>>>>= > http://www.unix-experience.fr=0A>>>>> =0A>>>>> 5 d=C3=A9cembre 2014 15:= 14 "Rick Macklem" a=0A>>>>> =C3=A9crit:=0A>>>>> = =0A>>>>> Loic Blot wrote:=0A>>>>> =0A>>>>> Hi,=0A>>>>> i'm trying to crea= te a virtualisation environment based on=0A>>>>> jails.=0A>>>>> Those jai= ls are stored under a big ZFS pool on a FreeBSD=0A>>>>> 9.3=0A>>>>> which= =0A>>>>> export a NFSv4 volume. This NFSv4 volume was mounted on a=0A>>>>= > big=0A>>>>> hypervisor (2 Xeon E5v3 + 128GB memory and 8 ports (but=0A>= >>>> only 1=0A>>>>> was=0A>>>>> used at this time).=0A>>>>> =0A>>>>> The = problem is simple, my hypervisors runs 6 jails (used 1%=0A>>>>> cpu=0A>>>= >> and=0A>>>>> 10GB RAM approximatively and less than 1MB bandwidth) and= =0A>>>>> works=0A>>>>> fine at start but the system slows down and after = 2-3 days=0A>>>>> become=0A>>>>> unusable. When i look at top command i se= e 80-100% on=0A>>>>> system=0A>>>>> and=0A>>>>> commands are very very sl= ow. Many process are tagged with=0A>>>>> nfs_cl*.=0A>>>>> =0A>>>>> To be = honest, I would expect the slowness to be because of=0A>>>>> slow=0A>>>>>= response=0A>>>>> from the NFSv4 server, but if you do:=0A>>>>> # ps axHl= =0A>>>>> on a client when it is slow and post that, it would give us=0A>>= >>> some=0A>>>>> more=0A>>>>> information on where the client side proces= ses are sitting.=0A>>>>> If you also do something like:=0A>>>>> # nfsstat= -c -w 1=0A>>>>> and let it run for a while, that should show you how man= y=0A>>>>> RPCs=0A>>>>> are=0A>>>>> being done and which ones.=0A>>>>> =0A= >>>>> # nfsstat -m=0A>>>>> will show you what your mount is actually usin= g.=0A>>>>> The only mount option I can suggest trying is=0A>>>>> "rsize= =3D32768,wsize=3D32768",=0A>>>>> since some network environments have dif= ficulties with 64K.=0A>>>>> =0A>>>>> There are a few things you can try o= n the NFSv4 server side,=0A>>>>> if=0A>>>>> it=0A>>>>> appears=0A>>>>> th= at the clients are generating a large RPC load.=0A>>>>> - disabling the D= RC cache for TCP by setting=0A>>>>> vfs.nfsd.cachetcp=3D0=0A>>>>> - If th= e server is seeing a large write RPC load, then=0A>>>>> "sync=3Ddisabled"= =0A>>>>> might help, although it does run a risk of data loss when=0A>>>>= > the=0A>>>>> server=0A>>>>> crashes.=0A>>>>> Then there are a couple of = other ZFS related things (I'm not=0A>>>>> a=0A>>>>> ZFS=0A>>>>> guy,=0A>>= >>> but these have shown up on the mailing lists).=0A>>>>> - make sure yo= ur volumes are 4K aligned and ashift=3D12 (in=0A>>>>> case a=0A>>>>> driv= e=0A>>>>> that uses 4K sectors is pretending to be 512byte sectored)=0A>>= >>> - never run over 70-80% full if write performance is an=0A>>>>> issue= =0A>>>>> - use a zil on an SSD with good write performance=0A>>>>> =0A>>>= >> The only NFSv4 thing I can tell you is that it is known that=0A>>>>> Z= FS's=0A>>>>> algorithm for determining sequential vs random I/O fails for= =0A>>>>> NFSv4=0A>>>>> during writing and this can be a performance hit. = The only=0A>>>>> workaround=0A>>>>> is to use NFSv3 mounts, since file ha= ndle affinity=0A>>>>> apparently=0A>>>>> fixes=0A>>>>> the problem and th= is is only done for NFSv3.=0A>>>>> =0A>>>>> rick=0A>>>>> =0A>>>>> I saw t= hat there are TSO issues with igb then i'm trying to=0A>>>>> disable=0A>>= >>> it with sysctl but the situation wasn't solved.=0A>>>>> =0A>>>>> Some= one has got ideas ? I can give you more informations if=0A>>>>> you=0A>>>= >> need.=0A>>>>> =0A>>>>> Thanks in advance.=0A>>>>> Regards,=0A>>>>> =0A= >>>>> Lo=C3=AFc Blot,=0A>>>>> UNIX Systems, Network and Security Engineer= =0A>>>>> http://www.unix-experience.fr=0A>>>>> __________________________= _____________________=0A>>>>> freebsd-fs@freebsd.org mailing list=0A>>>>>= http://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A>>>>> To unsubscr= ibe, send any mail to=0A>>>>> "freebsd-fs-unsubscribe@freebsd.org"=0A>>>>= > =0A>>>>> _______________________________________________=0A>>>>> freebs= d-fs@freebsd.org mailing list=0A>>>>> http://lists.freebsd.org/mailman/li= stinfo/freebsd-fs=0A>>>>> To unsubscribe, send any mail to=0A>>>>> "freeb= sd-fs-unsubscribe@freebsd.org"=0A>>>>> =0A>>>>> _________________________= ______________________=0A>>>>> freebsd-fs@freebsd.org mailing list=0A>>>>= > http://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A>>>>> To unsubsc= ribe, send any mail to=0A>>>>> "freebsd-fs-unsubscribe@freebsd.org"=0A>>>= >> =0A>>>>> _______________________________________________=0A>>>>> freeb= sd-fs@freebsd.org mailing list=0A>>>>> http://lists.freebsd.org/mailman/l= istinfo/freebsd-fs=0A>>>>> To unsubscribe, send any mail to=0A>>>>> "free= bsd-fs-unsubscribe@freebsd.org"=0A>>>>> _________________________________= ______________=0A>>>>> freebsd-fs@freebsd.org mailing list=0A>>>>> http:/= /lists.freebsd.org/mailman/listinfo/freebsd-fs=0A>>>>> To unsubscribe, se= nd any mail to=0A>>>>> "freebsd-fs-unsubscribe@freebsd.org"=0A>>>>> =0A>>= >>> _______________________________________________=0A>>>>> freebsd-fs@fr= eebsd.org mailing list=0A>>>>> http://lists.freebsd.org/mailman/listinfo/= freebsd-fs=0A>>>>> To unsubscribe, send any mail to=0A>>>>> "freebsd-fs-u= nsubscribe@freebsd.org"=0A>>> =0A>>> ____________________________________= ___________=0A>>> freebsd-fs@freebsd.org mailing list=0A>>> http://lists.= freebsd.org/mailman/listinfo/freebsd-fs=0A>>> To unsubscribe, send any ma= il to=0A>>> "freebsd-fs-unsubscribe@freebsd.org"