From owner-freebsd-stable@FreeBSD.ORG Thu Jul 1 18:51:34 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 63F8E1065677 for ; Thu, 1 Jul 2010 18:51:34 +0000 (UTC) (envelope-from alan.bryan@yahoo.com) Received: from web50508.mail.re2.yahoo.com (web50508.mail.re2.yahoo.com [206.190.38.84]) by mx1.freebsd.org (Postfix) with SMTP id 0D5DF8FC19 for ; Thu, 1 Jul 2010 18:51:33 +0000 (UTC) Received: (qmail 39739 invoked by uid 60001); 1 Jul 2010 18:51:28 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1278010288; bh=fDGwQjDDjK+2jb49OrsyAcNwZH734YhkskEk+gy31Hw=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=vIv+QkopnkWaUiEgIWeYr41wJeI/b+00IuKrKkpOdrgBM8N80lWN/s4D9aGUJZfs5hwTsWv+4We7NrnzxHPKOQGjZ4Di3dU0cgDa4dZVxB0w03cZlXw8yadr8gEkyS/35agajVmUN3NOlohxt+I539rZMkhsj+M+M+PiauxwzUI= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=mIDERZWNl5rurs0q3YobSvrRASD9sW+jY+dvj9tMgcTPFh+ZhkaGbSkoPSrTME+EwGI0ayP29l01bgNpv5x2UUmUi2x1VqOwz6NS49PavgAuWSWMBzNdCljQQDQRjDovKXy6A8CAaHtWJKBU2+5MyI+SOlNvG/6CkedgiEL6uxA=; Message-ID: <538823.39365.qm@web50508.mail.re2.yahoo.com> X-YMail-OSG: Sp13VBwVM1m8adgJ1HQiiPDsLdYmRHWgLEFhjpKzfSrVmji NEiXbWyBJaPX0o4El7XNxokqekhluLg5q37Geyyj36sBSChcPmciFs7q_4kB Qw8pDZ1vU9wzuHo62vMJAx_63qBTnjEJinvrPobROh6DDCTUBo_qrqQ3UgQf b8ZPp2TB0VGDPS9joTVH4_lejuY.0V9pKrykQ5aVA0cMOa6IIDT2gMS3Vin5 O5gBnuc9_iVEm8KRcVJId0J7A8Q9Ta2D7DjmZSsIS223HPrwx5DfNhjJxOSz aysv0DRhoItxSjTfI4JxLXJtCQVSx..KKPL4xDyFjH4OSXevi8cfxPff_Yj8 KmA-- Received: from [99.24.6.121] by web50508.mail.re2.yahoo.com via HTTP; Thu, 01 Jul 2010 11:51:28 PDT X-Mailer: YahooMailClassic/11.1.4 YahooMailWebService/0.8.104.274457 Date: Thu, 1 Jul 2010 11:51:28 -0700 (PDT) From: alan bryan To: Garrett Cooper In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org Subject: Re: NFS 75 second stall X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Jul 2010 18:51:34 -0000 =0A=0A--- On Thu, 7/1/10, Garrett Cooper wrote:=0A=0A>= From: Garrett Cooper =0A> Subject: Re: NFS 75 second s= tall=0A> To: "alan bryan" =0A> Cc: freebsd-stable@fre= ebsd.org=0A> Date: Thursday, July 1, 2010, 11:13 AM=0A> On Thu, Jul 1, 2010= at 11:01 AM, alan=0A> bryan =0A> wrote:=0A> > Setup:= =0A> >=0A> > server - FreeBSD 8-stable from today.=A0 2 UFS dirs=0A> export= ed via NFS.=0A> > client - FreeBSD 8.0-Release. =A0Running a test php=0A> s= cript that copies around various files to/from 2 separate=0A> NFS mounts.= =0A> >=0A> > Situation:=0A> >=0A> > script is started (forked to do 20 simu= ltaneous runs)=0A> and 20 1GB files are copied to the NFS dir which works= =0A> fine.=A0 When it then switches to reading those files back=0A> and sim= ultaneously writing to the other NFS mount I see a=0A> hang of 75 seconds.= =A0 If I do an "ls -l" on the NFS mount it=0A> hangs too.=A0 After 75 secon= ds the client has reported:=0A> >=0A> > nfs server 192.168.10.133:/usr/loca= l/export1: not=0A> responding=0A> > nfs server 192.168.10.133:/usr/local/ex= port1: is alive=0A> again=0A> > nfs server 192.168.10.133:/usr/local/export= 1: not=0A> responding=0A> > nfs server 192.168.10.133:/usr/local/export1: i= s alive=0A> again=0A> >=0A> > and then things start working again.=A0 The s= erver was=0A> originally FreeBSD 8.0-Release also but was upgraded to the= =0A> latest stable to see if this issue could be avoided.=0A> >=0A> > # nfs= stat -s -W -w 1=0A> > =A0GtAttr Lookup Rdlink=A0=A0=A0Read=A0 Write Rename= =0A> Access=A0 Rddir=0A> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 2= 22=A0 =A0 257=A0 =A0=0A> =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=0A> > =A0 =A0 =A0 = 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 178=A0 =A0 135=A0 =A0=0A> =A0 0=A0 =A0 = =A0 0=A0 =A0 =A0 0=0A> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0=A0= =A085=A0 =A0 127=A0=0A> =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=0A> > =A0 =A0 = =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0=0A> =A0 =A0 0= =A0 =A0 =A0 0=A0 =A0 =A0 0=0A> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0= =A0 =A0 0=A0 =A0 =A0 0=A0=0A> =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=0A> > = =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0=0A> = =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=0A> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0= =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0=0A> =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0= =0A> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0= =0A> =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=0A> >=0A> > ... for 75 rows of all= zeros=0A> >=0A> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 272=A0 = =A0 266=A0 =A0=0A> =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=0A> > =A0 =A0 =A0 0=A0 = =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 167=A0 =A0 165=A0 =A0=0A> =A0 0=A0 =A0 =A0 0= =A0 =A0 =A0 0=0A> >=0A> > I also tried runs with 15 simultaneous processes = and=0A> 25. =A015 processes gave only about a 5 second stall but 25=0A> gav= e again the same 75 second stall.=0A> >=0A> > Further, I tested with 2 moun= ts to the same server but=0A> from ZFS filesytems with the exact same stall= /timeout=0A> periods. =A0So, it doesn't appear to matter what the=0A> under= lying filesystem is - it's something in NFS or=0A> networking code.=0A> >= =0A> > Any ideas on what's going on here? =A0What's causing=0A> the complet= e stall period of zero NFS activity? =A0 Any flaws=0A> with my testing meth= ods?=0A> >=0A> > Thanks for any and all help/ideas.=0A> =0A> What network d= river are you using? Have you tried=0A> tcpdumping the packets?=0A> -Garret= t=0A> =0A=0AI'm using igb currently but have also used em. I have not trie= d tcpdumping the packets yet on this test. Any suggestions on things to lo= ok out for (I'm not that familiar with that whole process).=0A=0AWhich brin= gs up another point - I'm using TCP connections for NFS, not UDP. =0A=0A--= Alan=0A=0A=0A=0A