From owner-freebsd-hackers@FreeBSD.ORG Wed Dec 3 13:57:13 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B2FDE1065670 for ; Wed, 3 Dec 2008 13:57:13 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.terabit.net.ua (mail.terabit.net.ua [195.137.202.147]) by mx1.freebsd.org (Postfix) with ESMTP id 4A0388FC0C for ; Wed, 3 Dec 2008 13:57:13 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from skuns.zoral.com.ua ([91.193.166.194] helo=mail.zoral.com.ua) by mail.terabit.net.ua with esmtps (TLSv1:AES256-SHA:256) (Exim 4.63 (FreeBSD)) (envelope-from ) id 1L7rdC-00030f-9y; Wed, 03 Dec 2008 15:19:06 +0200 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id mB3DJ3Tu081988 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 3 Dec 2008 15:19:03 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.3/8.14.3) with ESMTP id mB3DJ2Fe006114; Wed, 3 Dec 2008 15:19:02 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.3/8.14.3/Submit) id mB3DJ2t5006113; Wed, 3 Dec 2008 15:19:02 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 3 Dec 2008 15:19:02 +0200 From: Kostik Belousov To: David Wolfskill Message-ID: <20081203131902.GK3045@deviant.kiev.zoral.com.ua> References: <20081203001538.GC96383@bunrab.catwhisker.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Nwn6wlxaqfRJxJFc" Content-Disposition: inline In-Reply-To: <20081203001538.GC96383@bunrab.catwhisker.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: ClamAV version 0.93.3, clamav-milter version 0.93.3 on skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua X-Virus-Scanned: mail.terabit.net.ua 1L7rdC-00030f-9y 0f434ac2832ec0d21d72c7418f5c217e X-Terabit: YES Cc: hackers@freebsd.org Subject: Re: NFS (& amd?) dysfunction descending a hierarchy X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Dec 2008 13:57:13 -0000 --Nwn6wlxaqfRJxJFc Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Dec 02, 2008 at 04:15:38PM -0800, David Wolfskill wrote: > I seem to have a fairly- (though not deterministly so) reproducible > mode of failure with an NFS-mounted directory hierarchy: An attempt to > traverse a "sufficiently large" hierarchy (e.g., via "tar zcpf" or "rm > -fr") will fail to "visit" some subdirectories, typically apparently > acting as if the subdirectories in question do not actually exist > (despite the names having been returned in the output of a previous > readdir()). >=20 > The file system is mounted read-write, courtesy of amd(8); none of > the files has any non-default flags; there are no ACLs involved; > and I owned the lot (that is, as "owning user" of the files). >=20 > An example of "sufficiently large" has been demonstrated to be a recent > copy of a FreeBSD ports tree. (The problem was discovered using a > hierarchy that had some proprietary content; I tried a copy of the ports > tree to see if I could replicate the issue with something a FreeBSD > hacker would more likely have handy. And avoid NDA issues. :-}) >=20 > Now, before I go further: I'm not pointing the finger at FreeBSD, > here (yet). At minimum, there could be fault with FreeBSD (as the NFS > client); with amd(8); with the NetApp Filer (as the NFS server); > or the network -- or the configuration(s) of any of them. >=20 > But I just tried this, using the same NFS server, but a machine running > Solaris 8 as an NFS client, and was unable to re-create the problem. >=20 > And I found a way to avoid having the problem occur using a FreeBSD NFS > client: whack amd(8)'s config so that the dismount_interval is 12 hours > instead of the default 2 minutes, thus effectivly preventing amd(8) from > its normal attempts to unmount file systems. Please note that I don't > consider this a fix -- or even an acceptable circumvention, in the long > term. Rather, it's a diagnostic change, in an attempt to better > understand the nature of the problem. >=20 > Here are step-by-step instructions to recreate the problem; > unfortunately, I believe I don't have the resources to test this > anywhere but at work, though I will try it at home, to the extent > that I can: >=20 > * Set up the environment. > * The failing environment uses NetApp filers as NFS servers. I don't > know what kind or how recent the software is on them, but can > find out. (I exepct they're fairly well-maintained.) > * Ensure that the NFS space available is at least 10 GB or more. > I will refer to this as "~/NFS/", as I tend to create such symlinks > to keep track of things. > * I used a dual, quad-core machine running FreeBSD RELENG_7_1 as of > yesterday morning as an NFS client. It also had a recently-updated > /usr/ports tree, which was a CVS working directory (so each "real" > subdirectory also had a CVS subdirectory within it). > * Set up amd(8) so that ~/NFS is mounted on demand when it's > referenced, and only via amd(8). Ensure that the dismount_interval > has the default value of 120 seconds. > * Create a reference tarball. > * cd /usr && tar zcpf ~/NFS/ports.tgz ports/ > * Create the test directory hierarchy. > * cd ~/NFS && tar zxpf ports.tgz > * Clear any cache. > * Unmount ~/NFS, then re-mount it. Or just reboot the NFS client > machine. Or arrange to have done all of the above set-up stuff > from a differnet NFS client. > * Set up for information capture (optional). > * Use ps(1) or your favorite alternative tool to determine the PID for > amd(8). Note that `cat /var/run/amd.pid` won't do the trick. :-{ > * Run ktrace(1) to capture activity from amd(8) and its descendants, > e.g.: >=20 > sudo ktrace -dip ${amd_pid} -f ktrace_amd.out >=20 > * Start a packet-capture for NFS traffic, e.g.: >=20 > sudo tcpdump -s 0 -n -w nfs.bpf host ${nfs_server} >=20 > * Start the test. > * Do this under ktrace(1), if you did the above optional step: >=20 > rm -fr ~/NFS/ports; echo $? >=20 > As soon as rm(1) issues a whine, you might as well interrupt it > (^C). >=20 > * Stop the information capture, if you started it. > * ^C for the tcpdump(1) process. > * sudo ktrace -C >=20 >=20 > If the packet capture file is too big for the analysis program you > prefer to digest as a unit, see the net/tcpslice port for a bit of > relief. (Wireshark seems to want to read an entire packet capture file > into main memory.) >=20 > I have performed the above, with the "information-gathering" step; I can > *probably* make that information available, but I'll need to check -- > some organizations get paranoid about things like host names. I don't > expect that my current employer is, but I don't know yet, so I won't > promise. >=20 > In the mean time, I should be able to extract somewhat-relevant > information from what I've collected, if that would be useful. While I > wouldn't mind sharing the results, I strongly suspect that blow-by-blow > analysis wouldn't be ideal for this (or any other) mailing list; I would > be very happy to work with others to figure out what's gone wrong (or is > misconfigured) and get things working properly. >=20 > If someone(s) would be willing to help, I'd appreciate it very much. If > (enough) folks would actually prefer that the details stay in the list > (or some other list), I'm willing to do that, too. >=20 I highly suspect that I know what happen. What branch are you using ? I committed a possible fix yesterday, r185557. The patch shall be directly applicable to 7. --Nwn6wlxaqfRJxJFc Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEYEARECAAYFAkk2h0YACgkQC3+MBN1Mb4iyDQCfUeI7WAmfV9tEaxL3R5y0V40z 4IAAoLeHIp1rETAo2URZx0BgpAArrrdR =SWaV -----END PGP SIGNATURE----- --Nwn6wlxaqfRJxJFc--