From owner-freebsd-stable@FreeBSD.ORG Wed Jan 6 19:30:41 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 635C61065672 for ; Wed, 6 Jan 2010 19:30:41 +0000 (UTC) (envelope-from alan.bryan@yahoo.com) Received: from web50501.mail.re2.yahoo.com (web50501.mail.re2.yahoo.com [206.190.38.77]) by mx1.freebsd.org (Postfix) with SMTP id 065CC8FC0C for ; Wed, 6 Jan 2010 19:30:40 +0000 (UTC) Received: (qmail 76293 invoked by uid 60001); 6 Jan 2010 19:03:56 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1262804636; bh=+KOph0cJSqF6BwE5w1hfXUY4Dtxnlfd/4EStxHohf04=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=zSxCgD5yZmtvPfCQTPEks83tllmB2AAvoDZcIvC1FccKNBeFLuszuHMJe6003LI8TIahPYZcWwcFy5CLEjCqdU0gIuIOiU7Vfqd55kHD7zGkIU1fs23UMDr+lSuqrHZXmtrLbBIQK6Ka6l6m1AnHPgzulFmXP1mR63X7KbF8DCo= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=sKiKrWxchlVLSjjCxjXrEJzKokTTz76d0i2vfFnAaTWdnlX1/DP5Q0/Clz9XHLjfO/ZNMZY3m4ldBmN8i2OFPoCOj2F1RR7s0BtBIe8po2f6FkjhtSu2jI9ikhdVr6zPNwOE+rbwffUnzfCjtVGSIE5qeuufPR4io+XiaqNsOFw=; Message-ID: <700718.76026.qm@web50501.mail.re2.yahoo.com> X-YMail-OSG: bzhFqV4VM1ljj5FU4EIqor9zZOTNdOrq470BONgND3CQO9UhZ4vRV5QelKcx6wXVcX1SK0j3hePw9obFZQpsBXcUEhPsJ3fh9k.qrlOjejYIVhmqLp6ozk7uo6SJzCO5TV_7b0KNbN2mRSS4ZTHJojn0jJsWoTGoi8aOSRxD6mtKSuKR5i18ZO0MdxEiadoE3deU66fOYyFVsmSsWhYI_.QpfeB.HWUxLmH3V0O.R_TOyiBiSl0xaj_FNnpjYUDi_D2mEM63jMOamI03Ua.TBw6idAv4NUwSJ25enWAD2NFDZEn6GDk0qUGiULCqBGlSHdPqcts5WGUqZCWixK2a1JXkE.xDA5zHDvEPDEzyrsXH9g1q28JIzhg6VZ2X4gA5MjoFzZNkAKJZXia_kgDgwA_h9sqrG9H1yfFpC8mnLLEdzYeFOcYJSzEBz602 Received: from [99.24.6.121] by web50501.mail.re2.yahoo.com via HTTP; Wed, 06 Jan 2010 11:03:56 PST X-Mailer: YahooMailClassic/9.0.20 YahooMailWebService/0.8.100.260964 Date: Wed, 6 Jan 2010 11:03:56 -0800 (PST) From: alan bryan To: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Subject: Zombie NFS writing from FreeBSD clients to FreeBSD 8.0 server with ZFS X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jan 2010 19:30:41 -0000 I have a AMD64 FreeBSD 8.0 server with ZFS filesystem being shared via NFS. These are being accessed by the clients. The clients are a mix of FreeBSD 6.2 32bit and FreeBSD 7.0 64bit. I have seen similar behavior from both versions of FreeBSD as clients. The behavior that I'm seeing is that everything is fine for a period of time and then the client starts "writing" large amounts of data to the NFS server. "Writing" is in quotes as nothing is actually being written to disk - this can go on for 12+ hrs or more - until the client is rebooted. It appears to cap out at around the 10-20Mbps rate and just sit there. Other clients are fine during this time. On the server: # nfsstat -s -w 1 GtAttr Lookup Rdlink Read Write Rename Access Rddir 0 25 2 38 7661 0 48 0 0 16 0 6 7601 0 14 0 0 21 0 13 7541 0 19 0 (client apache is stopped - nfsstat on client shows no activity) (server writes still continue) 0 13 0 1 7331 0 5 0 0 19 0 25 7479 0 59 0 0 9 0 9 7777 0 23 0 0 14 0 51 7640 0 33 0 0 8 0 40 4476 0 25 0 (Right here is when the bad client is rebooted) (everything is good again) 4 26 1 66 21 0 31 0 0 6 0 50 19 0 4 0 0 15 0 86 23 0 32 0 On the clients (webservers) I killed apache and it appears that nothing is then writing any longer. The same nfsstat -c -w 1 then shows zero activity. However, the high write volume continues at the server until this broken client machine is rebooted - and then the large drop in writes as you see above. At that point everything is now normal again. Looking in /var/log/messages on the client and server in these time periods hasn't shown any errors. Client mounts in /etc/fstab look like: 192.168.1.33:/tank/share /usr/local/www/share nfs rw,-b,-i,-U,-3 0 0 Any ideas on what to try, where to look for more insight, etc...?? Thanks, Alan