From owner-freebsd-stable@FreeBSD.ORG Mon Aug 16 06:35:54 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 05B2F1065672 for ; Mon, 16 Aug 2010 06:35:54 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta01.westchester.pa.mail.comcast.net (qmta01.westchester.pa.mail.comcast.net [76.96.62.16]) by mx1.freebsd.org (Postfix) with ESMTP id 9A6798FC12 for ; Mon, 16 Aug 2010 06:35:52 +0000 (UTC) Received: from omta13.westchester.pa.mail.comcast.net ([76.96.62.52]) by qmta01.westchester.pa.mail.comcast.net with comcast id uub21e00117dt5G51ubtNH; Mon, 16 Aug 2010 06:35:53 +0000 Received: from koitsu.dyndns.org ([98.248.41.155]) by omta13.westchester.pa.mail.comcast.net with comcast id uubr1e0093LrwQ23ZubsvX; Mon, 16 Aug 2010 06:35:53 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 350CA9B425; Sun, 15 Aug 2010 23:35:50 -0700 (PDT) Date: Sun, 15 Aug 2010 23:35:50 -0700 From: Jeremy Chadwick To: Mark Morley Message-ID: <20100816063550.GA35083@icarus.home.lan> References: <20100812175029.76D811065696@hub.freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100812175029.76D811065696@hub.freebsd.org> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: FreeBSD Stable Subject: Re: NFS stalling on 8.1-STABLE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Aug 2010 06:35:54 -0000 On Thu, Aug 12, 2010 at 10:35:49AM -0700, Mark Morley wrote: > I have five front end web servers that all mount their content from > the same server via NFS. If I stress the link on any one of the > machines (eg: copy a large directory with a lot of files to/from the > mounted file system) the client will pause. That is, all processes > trying to access that mount will freeze. The log files with hundreds > or thousands of nfs server not responding / is alive again messages. > After 60 seconds it returns to normal, unless the load is still there > in which case it continues to pause. > > This has only started happening since I upgraded the client machines > to 8.1-STABLE (previously four of them were 8.0 and one was 7.3). The > server is 7.1-RELEASE-p11. No other changes have taken place in terms > of hardware or software or mount options, etc. > > All nics involved are gigabit em cards, and they are on a private > network (web access to the boxes is via an external interface). Are there any indications in dmesg that the NIC is responsible, e.g. interface down/up, etc.? Does switching to UDP-based NFS solve the problem for you? What OS version (uname -a) and NIC are used on the NFS server? Can you please provide the following output from one of the client machines running 8.1-STABLE with gigE em(4)? You can X-out machine names, MAC addresses, and IP addresses/netblocks if need be. * uname -a * ifconfig emX (where X is the interface number which would be used for NFS) * netstat -idn -I emX * pciconf -lvc (provide only the data for emX please) * vmstat -i * sysctl hw.pci * As root, run "sysctl dev.em.X.stats=1" then do "dmesg" and provide the output for NIC statistics (will start with "emX:") Thanks. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |