From owner-freebsd-stable@FreeBSD.ORG Tue Feb 2 20:00:04 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6DA1E1065679 for ; Tue, 2 Feb 2010 20:00:04 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 09B328FC14 for ; Tue, 2 Feb 2010 20:00:03 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApsEAIcTaEuDaFvJ/2dsb2JhbACPJMpShEUE X-IronPort-AV: E=Sophos;i="4.49,393,1262581200"; d="scan'208";a="63999152" Received: from ganges.cs.uoguelph.ca ([131.104.91.201]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 02 Feb 2010 15:00:03 -0500 Received: from localhost (localhost.localdomain [127.0.0.1]) by ganges.cs.uoguelph.ca (Postfix) with ESMTP id 430F3FB8066; Tue, 2 Feb 2010 15:00:03 -0500 (EST) X-Virus-Scanned: amavisd-new at ganges.cs.uoguelph.ca Received: from ganges.cs.uoguelph.ca ([127.0.0.1]) by localhost (ganges.cs.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oT6KW3j8jZM7; Tue, 2 Feb 2010 15:00:02 -0500 (EST) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by ganges.cs.uoguelph.ca (Postfix) with ESMTP id D5D87FB805D; Tue, 2 Feb 2010 15:00:02 -0500 (EST) Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id o12KB5s12049; Tue, 2 Feb 2010 15:11:05 -0500 (EST) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Tue, 2 Feb 2010 15:11:05 -0500 (EST) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: alan bryan In-Reply-To: <878205.72256.qm@web50503.mail.re2.yahoo.com> Message-ID: References: <878205.72256.qm@web50503.mail.re2.yahoo.com> MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="-559023410-684387517-1265141465=:9141" Cc: freebsd-stable@freebsd.org Subject: Re: Zombie NFS writing from FreeBSD clients to FreeBSD 8.0 server with ZFS X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Feb 2010 20:00:04 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---559023410-684387517-1265141465=:9141 Content-Type: TEXT/PLAIN; charset=X-UNKNOWN; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE On Tue, 2 Feb 2010, alan bryan wrote: > --- On Wed, 1/6/10, Rick Macklem wrote: > From: Rick Macklem > Subject: Re: Zombie NFS writing from FreeBSD clients to FreeBSD 8.0 serve= r with ZFS > To: "alan bryan" > Cc: freebsd-stable@freebsd.org > Date: Wednesday, January 6, 2010, 12:19 PM >=20 >=20 > On Wed, 6 Jan 2010, alan bryan wrote: >=20 > > I have a AMD64 FreeBSD 8.0 server with ZFS filesystem > being shared via NFS.=A0 These are being accessed by the > clients.=A0 The clients are a mix of FreeBSD 6.2 32bit > and FreeBSD 7.0 64bit.=A0 I have seen similar behavior > from both versions of FreeBSD as clients. > > > [stuff related to lotsa writes being done snipped] > > > > Any ideas on what to try, where to look for more > insight, etc...?? > > > - Are they the same write being retried over and over? yes - it appears so > - Is the server replying to them with an error? yes >=20 > The above might give some insight into what's happening, Didn't help much. The server is replying with NFS3ERR_IO (EIO) for some reason. Unfortunately EIO is the catch all for just about everything. (There is only one case in the NFS server code that generates EIO and that would be caused by a bogus data mbuf list.) As such, I suspect the EIO is coming from ZFS for some reason, but can't be sure. Btw, a binary capture using tcpdump can be read into wireshark, which is much better at decoding NFS etc than tcpdump. rick ---559023410-684387517-1265141465=:9141--