From owner-freebsd-stable@FreeBSD.ORG Thu Jul 1 21:12:57 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E3CEE106566C for ; Thu, 1 Jul 2010 21:12:57 +0000 (UTC) (envelope-from yanefbsd@gmail.com) Received: from mail-gw0-f54.google.com (mail-gw0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 94F518FC13 for ; Thu, 1 Jul 2010 21:12:57 +0000 (UTC) Received: by gwb1 with SMTP id 1so738000gwb.13 for ; Thu, 01 Jul 2010 14:12:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=xGI/owDW1VsVZSD5P3ND09gRhXQsqYjObGVeoyLDb8U=; b=Zj3XlNMYGSH19zojv0VovJi7xSaY3Z5W8E2eqLHThmcSguLS3lsBiLdE4v5geUGBr1 cA7OqcFblsv5NFm+a7z66C40aZ6mXd9HINbG7zhV68EcKCHR8LJGez56ZpsxOY3Ib6fz tkeIUAf5xsiaMn5LbHZaQVQl9sec0XHwlUJJk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=VAXA71bEUSj9+zYns1N46MX4ndZFx2aDqsZoAEI6Kp2jxtvf1W9vAfhOUZm1jZnkTx ty8ELddknVUne93LP059kGMcMaTYFk7Bd2UiNwkUX6kHE6CGdUbHg4l74Ap/quZ+NEx8 fzFlo7gjAoXhF3FRgyYxBzUIQnxFeoGZDNqO0= MIME-Version: 1.0 Received: by 10.229.185.2 with SMTP id cm2mr82880qcb.47.1278018773031; Thu, 01 Jul 2010 14:12:53 -0700 (PDT) Received: by 10.229.221.83 with HTTP; Thu, 1 Jul 2010 14:12:52 -0700 (PDT) In-Reply-To: <119072.59868.qm@web50504.mail.re2.yahoo.com> References: <119072.59868.qm@web50504.mail.re2.yahoo.com> Date: Thu, 1 Jul 2010 14:12:52 -0700 Message-ID: From: Garrett Cooper To: alan bryan Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org Subject: Re: NFS 75 second stall X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Jul 2010 21:12:58 -0000 On Thu, Jul 1, 2010 at 1:36 PM, alan bryan wrote: > > > --- On Thu, 7/1/10, Garrett Cooper wrote: > >> From: Garrett Cooper >> Subject: Re: NFS 75 second stall >> To: "alan bryan" >> Cc: freebsd-stable@freebsd.org >> Date: Thursday, July 1, 2010, 1:28 PM >> On Thu, Jul 1, 2010 at 1:18 PM, alan >> bryan >> wrote: >> > >> > >> > --- On Thu, 7/1/10, Garrett Cooper >> wrote: >> > >> >> From: Garrett Cooper >> >> Subject: Re: NFS 75 second stall >> >> To: "alan bryan" >> >> Cc: freebsd-stable@freebsd.org >> >> Date: Thursday, July 1, 2010, 12:23 PM >> >> On Thu, Jul 1, 2010 at 11:51 AM, alan >> >> bryan >> >> wrote: >> >> > >> >> > >> >> > --- On Thu, 7/1/10, Garrett Cooper >> >> wrote: >> >> > >> >> >> From: Garrett Cooper >> >> >> Subject: Re: NFS 75 second stall >> >> >> To: "alan bryan" >> >> >> Cc: freebsd-stable@freebsd.org >> >> >> Date: Thursday, July 1, 2010, 11:13 AM >> >> >> On Thu, Jul 1, 2010 at 11:01 AM, alan >> >> >> bryan >> >> >> wrote: >> >> >> > Setup: >> >> >> > >> >> >> > server - FreeBSD 8-stable from >> today.=A0 2 UFS >> >> dirs >> >> >> exported via NFS. >> >> >> > client - FreeBSD 8.0-Release. >> =A0Running a >> >> test php >> >> >> script that copies around various files >> to/from 2 >> >> separate >> >> >> NFS mounts. >> >> >> > >> >> >> > Situation: >> >> >> > >> >> >> > script is started (forked to do 20 >> >> simultaneous runs) >> >> >> and 20 1GB files are copied to the NFS >> dir which >> >> works >> >> >> fine.=A0 When it then switches to reading >> those >> >> files back >> >> >> and simultaneously writing to the other >> NFS mount >> >> I see a >> >> >> hang of 75 seconds.=A0 If I do an "ls -l" >> on the >> >> NFS mount it >> >> >> hangs too.=A0 After 75 seconds the client >> has >> >> reported: >> >> >> > >> >> >> > nfs server >> 192.168.10.133:/usr/local/export1: >> >> not >> >> >> responding >> >> >> > nfs server >> 192.168.10.133:/usr/local/export1: >> >> is alive >> >> >> again >> >> >> > nfs server >> 192.168.10.133:/usr/local/export1: >> >> not >> >> >> responding >> >> >> > nfs server >> 192.168.10.133:/usr/local/export1: >> >> is alive >> >> >> again >> >> >> > >> >> >> > and then things start working >> again.=A0 The >> >> server was >> >> >> originally FreeBSD 8.0-Release also but >> was >> >> upgraded to the >> >> >> latest stable to see if this issue could >> be >> >> avoided. >> >> >> > >> >> >> > # nfsstat -s -W -w 1 >> >> >> > =A0GtAttr Lookup Rdlink=A0=A0=A0Read >> Write >> >> Rename >> >> >> Access=A0 Rddir >> >> >> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0 >> 222 >> >> 257 >> >> >> =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0 >> >> >> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0 >> 178 >> >> 135 >> >> >> =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0 >> >> >> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0 >> =A0=A0=A085 >> >> =A0 127 >> >> >> =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0 >> >> >> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0 >> =A0 0 >> >> =A0 0 >> >> >> =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0 >> >> >> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0 >> =A0 0 >> >> =A0 0 >> >> >> =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0 >> >> >> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0 >> =A0 0 >> >> =A0 0 >> >> >> =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0 >> >> >> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0 >> =A0 0 >> >> =A0 0 >> >> >> =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0 >> >> >> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0 >> =A0 0 >> >> =A0 0 >> >> >> =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0 >> >> >> > >> >> >> > ... for 75 rows of all zeros >> >> >> > >> >> >> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0 >> 272 >> >> 266 >> >> >> =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0 >> >> >> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0 >> 167 >> >> 165 >> >> >> =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0 >> >> >> > >> >> >> > I also tried runs with 15 >> simultaneous >> >> processes and >> >> >> 25. =A015 processes gave only about a 5 >> second >> >> stall but 25 >> >> >> gave again the same 75 second stall. >> >> >> > >> >> >> > Further, I tested with 2 mounts to >> the same >> >> server but >> >> >> from ZFS filesytems with the exact same >> >> stall/timeout >> >> >> periods. =A0So, it doesn't appear to >> matter what >> >> the >> >> >> underlying filesystem is - it's something >> in NFS >> >> or >> >> >> networking code. >> >> >> > >> >> >> > Any ideas on what's going on here? >> =A0What's >> >> causing >> >> >> the complete stall period of zero NFS >> activity? >> >> Any flaws >> >> >> with my testing methods? >> >> >> > >> >> >> > Thanks for any and all help/ideas. >> >> >> >> >> >> What network driver are you using? Have >> you tried >> >> >> tcpdumping the packets? >> >> >> -Garrett >> >> >> >> >> > >> >> > I'm using igb currently but have also used >> em. =A0I >> >> have not tried tcpdumping the packets yet on this >> test. >> >> =A0Any suggestions on things to look out for (I'm >> not that >> >> familiar with that whole process). >> >> > >> >> > Which brings up another point - I'm using >> TCP >> >> connections for NFS, not UDP. >> >> >> >> =A0 =A0 Is the net.inet.tcp.tso sysctl enabled or >> >> not? What about rxcsum and txcsum? >> >> Thanks, >> >> -Garrett >> >> >> > >> > I haven't intentionally/explicitly set any of this so >> it's "default": >> > >> > # sysctl net.inet.tcp.tso >> > net.inet.tcp.tso: 1 >> > >> > >> > igb0: >> flags=3D8843 >> metric 0 mtu 1500 >> > >> =A0options=3D13b >> > =A0 =A0 =A0 =A0ether 00:30:48:c3:26:94 >> > =A0 =A0 =A0 =A0inet 192.168.10.133 netmask 0xffffff00 >> broadcast 192.168.10.255 >> > =A0 =A0 =A0 =A0media: Ethernet autoselect (1000baseT >> ) >> > =A0 =A0 =A0 =A0status: active >> >> Devise all of the available permutations that you need to >> use to test >> this out; there are a total of 3 variables, so 9 >> permutations, but >> you've already `tested one', so that makes the permutation >> count 8. >> Example: >> >> TXCSUM=3Doff, RXCSUM=3Don, TSO=3Don >> TXCSUM=3Don, RXCSUM=3Doff, TSO=3Don >> TXCSUM=3Don, RXCSUM=3Doff, TSO=3Doff >> >> ... >> >> Try executing the permutations on the client first, keeping >> the server >> constant, then make the client constant and make the server >> variable, >> and finally do both to the server and client. >> >> Be sure to take measurements for each permutation to ensure >> that >> things make functional sense. >> >> The reason why I'm suggesting this is that there were >> issues with >> em(4) [and igb(4) too I think since it uses common code], >> with various >> hardware offload bits on 8.0-RELEASE (IIRC disabling txcsum >> did the >> trick, but you may have to do more than that in order to >> get things to >> work). >> >> Here's a similar thread with a different driver: >> http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008264.html >> (just to illustrate the thought process used to determine >> the source >> of failure). > > Thanks for the detailed test plan! Np ;).. > Is it also fair to then assume that if I update the NFS client machine to= the latest 8-Stable that should also fix this issue? =A0(Both will then be= running the latest 8-stable code). =A0These are not in production so I can= test or upgrade with no issues. You can try that, but it would be better to determine (with minimal effort) that disabling one of the features is the root cause; after that you can upgrade to a later version if you want and see whether or not there's an issue with the stack (there's been some churn with em(4) cards though so I'd avoid doing that unless you're willing to live with soaking things in a bit more on the 8.0-RELEASE machine side). FWIW I've seen whacky ass problems with TSO on and ipfw on my Dell 2950 (uses bce(4)) so I've disabled it for the time being... Cheers, -Garrett