Date: Thu, 1 Jul 2010 11:51:28 -0700 (PDT) From: alan bryan <alan.bryan@yahoo.com> To: Garrett Cooper <yanefbsd@gmail.com> Cc: freebsd-stable@freebsd.org Subject: Re: NFS 75 second stall Message-ID: <538823.39365.qm@web50508.mail.re2.yahoo.com> In-Reply-To: <AANLkTilNvy3FYUNjjiJ85eWrF7jTAvJJ9E7Q2eqhhQj6@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
=0A=0A--- On Thu, 7/1/10, Garrett Cooper <yanefbsd@gmail.com> wrote:=0A=0A>= From: Garrett Cooper <yanefbsd@gmail.com>=0A> Subject: Re: NFS 75 second s= tall=0A> To: "alan bryan" <alan.bryan@yahoo.com>=0A> Cc: freebsd-stable@fre= ebsd.org=0A> Date: Thursday, July 1, 2010, 11:13 AM=0A> On Thu, Jul 1, 2010= at 11:01 AM, alan=0A> bryan <alan.bryan@yahoo.com>=0A> wrote:=0A> > Setup:= =0A> >=0A> > server - FreeBSD 8-stable from today.=A0 2 UFS dirs=0A> export= ed via NFS.=0A> > client - FreeBSD 8.0-Release. =A0Running a test php=0A> s= cript that copies around various files to/from 2 separate=0A> NFS mounts.= =0A> >=0A> > Situation:=0A> >=0A> > script is started (forked to do 20 simu= ltaneous runs)=0A> and 20 1GB files are copied to the NFS dir which works= =0A> fine.=A0 When it then switches to reading those files back=0A> and sim= ultaneously writing to the other NFS mount I see a=0A> hang of 75 seconds.= =A0 If I do an "ls -l" on the NFS mount it=0A> hangs too.=A0 After 75 secon= ds the client has reported:=0A> >=0A> > nfs server 192.168.10.133:/usr/loca= l/export1: not=0A> responding=0A> > nfs server 192.168.10.133:/usr/local/ex= port1: is alive=0A> again=0A> > nfs server 192.168.10.133:/usr/local/export= 1: not=0A> responding=0A> > nfs server 192.168.10.133:/usr/local/export1: i= s alive=0A> again=0A> >=0A> > and then things start working again.=A0 The s= erver was=0A> originally FreeBSD 8.0-Release also but was upgraded to the= =0A> latest stable to see if this issue could be avoided.=0A> >=0A> > # nfs= stat -s -W -w 1=0A> > =A0GtAttr Lookup Rdlink=A0=A0=A0Read=A0 Write Rename= =0A> Access=A0 Rddir=0A> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 2= 22=A0 =A0 257=A0 =A0=0A> =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=0A> > =A0 =A0 =A0 = 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 178=A0 =A0 135=A0 =A0=0A> =A0 0=A0 =A0 = =A0 0=A0 =A0 =A0 0=0A> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0=A0= =A085=A0 =A0 127=A0=0A> =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=0A> > =A0 =A0 = =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0=0A> =A0 =A0 0= =A0 =A0 =A0 0=A0 =A0 =A0 0=0A> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0= =A0 =A0 0=A0 =A0 =A0 0=A0=0A> =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=0A> > = =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0=0A> = =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=0A> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0= =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0=0A> =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0= =0A> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0= =0A> =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=0A> >=0A> > ... for 75 rows of all= zeros=0A> >=0A> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 272=A0 = =A0 266=A0 =A0=0A> =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=0A> > =A0 =A0 =A0 0=A0 = =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 167=A0 =A0 165=A0 =A0=0A> =A0 0=A0 =A0 =A0 0= =A0 =A0 =A0 0=0A> >=0A> > I also tried runs with 15 simultaneous processes = and=0A> 25. =A015 processes gave only about a 5 second stall but 25=0A> gav= e again the same 75 second stall.=0A> >=0A> > Further, I tested with 2 moun= ts to the same server but=0A> from ZFS filesytems with the exact same stall= /timeout=0A> periods. =A0So, it doesn't appear to matter what the=0A> under= lying filesystem is - it's something in NFS or=0A> networking code.=0A> >= =0A> > Any ideas on what's going on here? =A0What's causing=0A> the complet= e stall period of zero NFS activity? =A0 Any flaws=0A> with my testing meth= ods?=0A> >=0A> > Thanks for any and all help/ideas.=0A> =0A> What network d= river are you using? Have you tried=0A> tcpdumping the packets?=0A> -Garret= t=0A> =0A=0AI'm using igb currently but have also used em. I have not trie= d tcpdumping the packets yet on this test. Any suggestions on things to lo= ok out for (I'm not that familiar with that whole process).=0A=0AWhich brin= gs up another point - I'm using TCP connections for NFS, not UDP. =0A=0A--= Alan=0A=0A=0A=0A
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?538823.39365.qm>