Date: Tue, 27 Sep 2011 05:41:57 -0700 (PDT) From: Kirill Yelizarov <ykirill@yahoo.com> To: "freebsd-stable@freebsd.org" <freebsd-stable@freebsd.org> Subject: Re: NFSD hang Message-ID: <1317127317.76570.YahooMailNeo@web120526.mail.ne1.yahoo.com> In-Reply-To: <20110927115935.GA29196@icarus.home.lan> References: <1317017670.4307.YahooMailNeo@web120527.mail.ne1.yahoo.com> <20110926063210.GA54741@icarus.home.lan> <1317034584.14989.YahooMailNeo@web120530.mail.ne1.yahoo.com> <1317121450.5432.YahooMailNeo@web120529.mail.ne1.yahoo.com> <20110927115935.GA29196@icarus.home.lan>
next in thread | previous in thread | raw e-mail | index | archive | help
=0A=0A=0A=0A________________________________=0AFrom: Jeremy Chadwick <freeb= sd@jdc.parodius.com>=0ATo: Kirill Yelizarov <ykirill@yahoo.com>=0ACc: rmack= lem@uoguelph.ca; freebsd-stable@freebsd.org=0ASent: Tuesday, September 27, = 2011 3:59 PM=0ASubject: Re: NFSD hang=0A=0AOn Tue, Sep 27, 2011 at 04:04:10= AM -0700, Kirill Yelizarov wrote:=0A> I found a had sync enabled on my serv= er so I set? zfs?set?sync=3Ddisabled data=0A> and will look for failures. A= re there any other setting for nfs over zfs i can check or set? =0A> =0A> _= _______________________________=0A> =0A> # uname -a=0A> FreeBSD brat.faberl= ic.com 8.2-STABLE FreeBSD 8.2-STABLE #0: Thu Jun? 9 11:22:38 MSD 2011???? r= oot@**:/usr/obj/usr/src/sys/BRAT? amd64 Sources were taken at that time=0A>= =0A> There are a lot of this. Should i paste them all here or part is enou= gh?=0A> =0A> brat# procstat -k -k 1666=0A> ? PID??? TID COMM???????????? TD= NAME?????????? KSTACK?????????????????????? =0A> ?1666 100323 nfsd?????????= ??? nfsd: master???? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_time= dwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_run+0x8b = nfssvc_nfsd+0x97 nfssvc_nfsserver+0x53 nfssvc+0x44 syscallenter+0x186 sysca= ll+0x40 Xfast_syscall+0xe2 =0A> ?1666 100391 nfsd???????????? nfsd: service= ??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12 _c= v_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_start+0xb fork_exit= +0x114 fork_trampoline+0xe =0A> ?1666 100392 nfsd???????????? nfsd: service= ??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12 _c= v_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_start+0xb fork_exit= +0x114 fork_trampoline+0xe =0A> ?1666 100393 nfsd???????????? nfsd: service= ??? <running>??????????????????? =0A> ?1666 100394 nfsd???????????? nfsd: s= ervice??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_timedwait_sig+0= x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_start+0xb for= k_exit+0x114 fork_trampoline+0xe =0A> ?1666 100395 nfsd???????????? nfsd: s= ervice??? <running>??????????????????? =0A> ?1666 100396 nfsd???????????? n= fsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_timedwait= _sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_start+0= xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100397 nfsd???????????? n= fsd: service??? <running>??????????????????? =0A> ?1666 100398 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100399 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100400 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100401 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100402 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100403 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100404 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100405 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100406 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100407 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100408 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100409 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100410 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100411 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100412 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100413 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100414 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100415 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100416 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100417 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100418 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100419 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100420 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100421 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100422 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100423 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100424 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100425 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100426 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100427 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100428 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100429 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100430 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100431 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100432 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100433 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100434 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100435 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100436 nfsd????????= ???? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x309 sleepq_tim= edwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_thread_s= tart+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100437 nfsd????????= ???? nfsd: service??? <running>??????????????????? =0A> ?1666 100438 nfsd??= ?????????? nfsd: service??? <running>??????????????????? =0A> ?1666 100439 = nfsd???????????? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x30= 9 sleepq_timedwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 = svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100440 = nfsd???????????? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x30= 9 sleepq_timedwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 = svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100441 = nfsd???????????? nfsd: service??? mi_switch+0x176 sleepq_catch_signals+0x30= 9 sleepq_timedwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+0x939 = svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 100442 = nfsd???????????? nfsd: service??? <running>??????????????????? =0A> ?1666 1= 00443 nfsd???????????? nfsd: service??? mi_switch+0x176 sleepq_catch_signal= s+0x309 sleepq_timedwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+= 0x939 svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 1= 00444 nfsd???????????? nfsd: service??? mi_switch+0x176 sleepq_catch_signal= s+0x309 sleepq_timedwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+= 0x939 svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 1= 00445 nfsd???????????? nfsd: service??? mi_switch+0x176 sleepq_catch_signal= s+0x309 sleepq_timedwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+= 0x939 svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 1= 00446 nfsd???????????? nfsd: service??? mi_switch+0x176 sleepq_catch_signal= s+0x309 sleepq_timedwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+= 0x939 svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe =0A> ?1666 1= 00447 nfsd???????????? nfsd: service??? mi_switch+0x176 sleepq_catch_signal= s+0x309 sleepq_timedwait_sig+0x12 _cv_timedwait_sig+0x11d svc_run_internal+= 0x939 svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe=0A> =0A> =0A= > =0A> ________________________________=0A> From: Jeremy Chadwick <freebsd@= jdc.parodius.com>=0A> To: Kirill Yelizarov <ykirill@yahoo.com>=0A> Cc: "fre= ebsd-stable@freebsd.org" <freebsd-stable@freebsd.org>=0A> Sent: Monday, Sep= tember 26, 2011 10:32 AM=0A> Subject: Re: NFSD hang=0A> =0A> On Sun, Sep 25= , 2011 at 11:14:30PM -0700, Kirill Yelizarov wrote:=0A> > Good Day!=0A> > I= 'v got a problem with nfs share on zfs volume. Everything worked fine for a= few month and now it hang. This share stores logs from 9 servers at night,= about 1-2Gb from each server. ZFS is filled to 26% and it is v28=0A> > =0A= > > last pid: 46573;? load averages: 195.82, 199.86, 200.12????????????????= ?????????????????????????????????????????????????????????????? up 108+21:56= :50 10:05:06=0A> > 432 processes: 208 running, 224 sleeping=0A> > CPU:? 0.0= % user,? 0.0% nice,? 100% system,? 0.0% interrupt,? 0.0% idle=0A> > Mem: 28= 0M Active, 1469M Inact, 9584M Wired, 161M Cache, 1232M Buf, 311M Free=0A> >= Swap: 16G Total, 16G Free=0A> > =0A> > ? PID USERNAME????? THR PRI NICE?? = SIZE??? RES STATE?? C?? TIME?? WCPU COMMAND=0A> > ?1666 root????????? 256? = 76??? 0? 5788K? 5120K RUN??? 14 476.8H 1508.64% nfsd=0A> > =0A> > # zpool l= ist=0A> > NAME?? SIZE? ALLOC?? FREE??? CAP? DEDUP? HEALTH? ALTROOT=0A> > da= ta? 3.62T?? 954G? 2.69T??? 25%? 1.00x? ONLINE? -=0A> > =0A> > # zfs list=0A= > > NAME?? USED? AVAIL? REFER? MOUNTPOINT=0A> > data?? 954G? 2.64T?? 954G? = /data=0A> > =0A> > # zfs mount=0A> > data??????????????????????????? /data= =0A> > =0A> > What should i look for to resolve it?=0A> =0A> What version o= f FreeBSD exactly, and what build date?=0A> =0A> Please provide output from= "procstat -k -k 1666" (yes, two -k's).=0A=0ACan you explain the correlatio= n between the "sync" parameter (which I=0Ahave to assume was set to "standa= rd" -- the default -- on all of your=0Afilesystems) and your nfsd issue?=A0= I do not see the correlation.=0A=0AMy intention of asking for procstat -k = -k output (which you did provide;=0Athank you) was for Rick Macklem (who's = currently working on NFS on=0AFreeBSD) to chime in with some insights.=A0 H= e may be busy, but I've CC'd=0Ahim here.=0A=0AI found it in the wiki http:/= /wiki.freebsd.org/ZFSTuningGuide. So i gave it a try. I thought it is someh= ow related with zfs because i couldn't even run ls on zfs volume. I had to = reset this server because it didn't respond to init commands.=0A=0AKirill= =0A
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1317127317.76570.YahooMailNeo>