From owner-freebsd-current@FreeBSD.ORG Thu Jan 12 01:42:26 2012 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B8030106566C for ; Thu, 12 Jan 2012 01:42:26 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 625F48FC1D for ; Thu, 12 Jan 2012 01:42:26 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ap8EAFc5Dk+DaFvO/2dsb2JhbABDhQ6pAoFyAQEBAwEBAQEgBCcgCwUWGAICDRkCKQEJJgYIBwQBCBQEh1kIpiyRVoEvgk6HCoEWBIg6ii2CJYp5h2E X-IronPort-AV: E=Sophos;i="4.71,495,1320642000"; d="scan'208";a="151711730" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 11 Jan 2012 20:42:25 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 47D50B3F54; Wed, 11 Jan 2012 20:42:25 -0500 (EST) Date: Wed, 11 Jan 2012 20:42:25 -0500 (EST) From: Rick Macklem To: Martin Cracauer Message-ID: <2072420569.94661.1326332545279.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <20120111182110.GA75991@cons.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: freebsd-current@freebsd.org, Stefan Bethke Subject: Re: Data corruption over NFS in -current X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jan 2012 01:42:26 -0000 Martin Cracauer wrote: > Stefan Bethke wrote on Wed, Jan 11, 2012 at 07:14:44PM +0100: > > Am 11.01.2012 um 17:57 schrieb Martin Cracauer: > > > > > I'm sorry for the unspecific bug report but I thought a heads-up > > > is > > > better than none. > > > > > > $ uname -a > > > FreeBSD wings.cons.org 10.0-CURRENT FreeBSD 10.0-CURRENT #2: Wed > > > Dec > > > 28 12:19:21 EST 2011 > > > cracauer@wings.cons.org:/usr/src/sys/amd64/compile/WINGS amd64 > > > > I'm sure Rick will want to know which NFS version, which client code > > (default new code I'm assuming) and which mount options... > > It's all default both in fstab and as reported by mount(8). > I assume that by the above statement, you mean that you don't specify any mount options in your /etc/fstab entry except "rw"? (If this isn't correct, please post your /etc/fstab entries for the NFS mounts.) - If I am correct, in that you just specify "rw", the main difference between the old and new NFS client will be the rsize/wsize used. The new NFS client will use MAX_BSIZE (64Kb) decreased to whatever the server says is the largest it can handle. This should be fine, unless the server says it can handle >= 64Kb, but actually only works correctly for 32Kb (which is what the old NFS client will default to, I think?). A few things to try/check: - Look locally on the server to see if the file is corrupted there. - Try the old NFS client. (Set the fs type to "oldnfs" instead of "nfs" on the lines in your /etc/fstab.) - If switching to the old client helps, it might be a bug in the way the new client generates the create verifier. I just looked at the code and I'm not certain the code in the new client would work correctly for a amd64. (I only have i386 to test with.) - I can easily generate a patch that changes the new client to do this the same way as the old client, but there is no point, unless the old client doesn't have the problem. --> Exclusive create problems might explain the incorrect ownership, since it first does a create that will fill in user/group in whatever default way the Linux server chooses to and then does a Setattr RPC to change them to the correct values. If the Setattr RPC fails, then the file exists owned by whatever the server chooses. (I don't know if Linux servers use the gid of the directory or the gid of the requestor or ???) - If you have a non-Linux NFS server, try running against that to see if it is a Linux server specific problem. (Since I haven't seen any other reports like this, I suspect it might be an interoperability problem related to the Linux server.) Also, if you can reproduce the problem fairly easily, capture a packet trace via # tcpdump -s 0 -w xxx host running on the client (or similar). Then email me "xxx" as an attachment and I can look at it in wireshark. (If you choose to look at it in wireshark, I would suggest you look for Create RPCs to see if they are Exclusive Creates, plus try and see where the data for the corrupt file is written.) Even if the capture is pretty large, it should be easy to find the interesting part, so long as you know the name of the corrupt file and search for that. > This is a diskless PXE boot but the mount affected (usr) is not the > root filesystem, so this should come in via fstab. > > BTW, my /usr/ports is another mount so the corruption is cross-mount > (garbage from /usr/ports entering /usr). > > Appending nfsstat output. > nfsstat output is pretty useless for this kind of situation. I did find it interesting that you do so many Fsstat RPCs, but that shouldn't be a problem, it's just weird to see that. rick > I am re-running things contiguously to see how reproducible this is. > This machine was recently updated from a -current almost a year old, > so it's its first time with the new NFS client code. > > Martin > > > > I see filesystem corruption on NFS filesystems here. I am running > > > a > > > heavy shellscript that is noodling around with ascii files > > > assembling > > > them with awk and whatnot. Some actions are concurrent with up to > > > 21 > > > forks doing full-CPU load scripting. This machine is a K8 with a > > > total of 8 cores, diskless NFS and memory filesystem for /tmp. > > > > > > I observe two problems: > > > - for no reason whatsoever, some files change from my > > > (user/group) cracauer/wheel to root/cracauer > > > - the same files will later be corrupted. The beginning of the > > > file > > > is normal but then it has what looks like parts of /usr/ports, > > > including our CVS files and binary junk, mostly zeros > > > > > > I did do some ports building lately but not at the same time that > > > this > > > problem manifested itself. I speculate some ports blocks were > > > still > > > resident in the filesystem buffer cache. > > > > > > Server is Linux. > > > > > > Martin > > > -- > > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > > Martin Cracauer http://www.cons.org/cracauer/ > > > _______________________________________________ > > > freebsd-current@freebsd.org mailing list > > > http://lists.freebsd.org/mailman/listinfo/freebsd-current > > > To unsubscribe, send any mail to > > > "freebsd-current-unsubscribe@freebsd.org" > > > > -- > > Stefan Bethke Fon +49 151 14070811 > > > > > > > > -- > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > Martin Cracauer http://www.cons.org/cracauer/ > > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to > "freebsd-current-unsubscribe@freebsd.org"