From owner-freebsd-net@FreeBSD.ORG Mon Dec 15 20:59:37 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2E5DB386 for ; Mon, 15 Dec 2014 20:59:37 +0000 (UTC) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id BDFCA96D for ; Mon, 15 Dec 2014 20:59:36 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AggFAKVKj1SDaFve/2dsb2JhbABag1hYBIMCwk0KhShKAoE7AQEBAQF9hAwBAQEDAQEBASArIAYFBRYYAgINGQIpAQkmBggHBAEaAgSIAwgNvjWWPQEBAQEBAQQBAQEBAQEBAQEZgSGNbxACAQYVNAeCLTsRgTAFiT6IAoMcgyAwhF+FEoJTgzgigX4egW4gMAd9QX4BAQE X-IronPort-AV: E=Sophos;i="5.07,581,1413259200"; d="scan'208";a="176737449" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 15 Dec 2014 15:59:29 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 21EB9B4091; Mon, 15 Dec 2014 15:59:29 -0500 (EST) Date: Mon, 15 Dec 2014 15:59:29 -0500 (EST) From: Rick Macklem To: Mehmet Erol Sanliturk Message-ID: <2048229686.13136235.1418677169130.JavaMail.root@uoguelph.ca> In-Reply-To: Subject: Re: compiling on nfs directories MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.91.201] X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926) Cc: freebsd-net@freebsd.org, Gerrit =?utf-8?B?S8O8aG4=?= X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2014 20:59:37 -0000 Mehmet Erol Sanliturk wrote: > On Mon, Dec 15, 2014 at 1:24 AM, Gerrit K=C3=BChn > > wrote: > > > > Hi all, > > > > I ran into some weird issue here last week: > > I have an NFS-Server for storage and diskless booting (pxe / nfs > > root) > > running under FreeBSD. The clients are running Gentoo Linux. Some > > time > > ago, I replaced the server, going from a HDD-based storage array > > (ZFS) > > under FreeBSD 8.3 to an SSD-based array under FreeBSD 10-stable (as > > of > > February this year - I know this needs updates). > > > > Only now I recognized that this somehow appears to have broken some > > of my > > Gentoo ebuilds that do not install cleanly anymore. They complain > > about > > "soiled libtool library files found" and "insecure RUNPATHs" in the > > installation stage of shared libs. > > > > I was not able to find any useful solution for this in the Net so > > far. > > However, I was able to verify that this is somehow an issue with > > the nfs > > server by plugging in a USB-drive into the diskless clients and > > mounting > > this as /var/tmp/portage (the directory structure where Gentoo's > > ebuilds > > are compiled). This makes the error messages go away, and > > everything works > > again (like it did before the server update). > > > > Are there any suggestions what might be causing this and how to fix > > it? > > > > > > cu > > Gerrit > > > > >=20 >=20 > With respect to information given in your message , may pure guess is > the > following : >=20 >=20 > When a client generates a file in NFS server , it assumes that > everything > is written into the file . > The next step ( reading the generated file ) starts , BUT the file is > NOT > completely written into disk yet , > therefore it reads an incomplete file which causes errors in the > client . >=20 Well, not exactly. The NFS client chooses whether or not the written data must be committed to stable storage (disk) right away via a flag argument on the write. It is up to the client to keep track of what has been written and if the FILE_STABLE flag wasn't set, must do a separate Commit RPC to force the data to stable storage on the server. It is also up to the NFS client to keep track of the file's size while it is being grown, since the NFS server's size may be smaller until the data gets written to the server. Also, note that he didn't see the problem with FreeBSD8.3, which would have been following the same rules on the server as 10.1. What I suspect might cause this is one of two things: 1 - The modify time of the file is now changing at a time the Linux client doesn't expect, due to changes in ZFS or maybe TOD clock resolution. (At one time, the TOD clock was only at a resolution of 1sec, so the client wouldn't see the modify time change often. I think it is now at a much higher resolution, but would have to look at the code/test to be sure.) 2 - I think you mention this one later in your message, in that the build might be depending on file locking. If this is the case, trying NFSv4, which does better file locking, might fix the problem. Gerrit, I would suggest that you do "nfsstat -m" on the Linux client, to see what the mount options are. The Linux client might be using NFSv4 already. Also, avoid "soft, intr" especially if you are using NFSv4, since these can cause slow server response to result in a failure of a read/write when it shouldn't fail, due to timeout or interruption by a signal. If you could find out more about what causes the specific build failure on the Linux side, that might help. If you can reproduce a build failure quickly/easily, you can capture packets via "tcpdump -s 0 -w host " on the server and then look at it in wireshark to see what the server is replying when the build failure occurs. (I don't mind looking at a packet trace if it is relatively small, if you email it to me as an attachment.) Good luck with it, rick ps: I am not familiar with the Linux mount options, but if it has stuff like "nocto", you could try those. > In FreeBSD NFS server , there is NOT ( or I could NOT be able to find > ) a > facility to store written data immediately into disk . >=20 > NFS server is collecting data up to a point ( number of bytes ) and > then > writing it to disk , during this phase ( whether the NFS server is > busy or > not ) is not important ) . With this structure , > the tasks which a program writes a small number of bytes to be read > by > another program can not be > processed by a NFS server only . >=20 > I did not try "locking in NFS server" : If this route is taken , then > it is > necessary to adjust the clients for such periods to wait that NFS > server > has removed the lock which themselves can continue ( Each such read > requires a waiting loop without generating an error message about > unavailable data and termination . ) . >=20 > In Linux NFS server , there is an option to immediately write the > received > data into disk . This is improving the above situation considerable > but not > completely solving the problem ( because during reads of data , data > in > cache is NOT concatenated to the data in disk ) . >=20 >=20 > Another MAJOR problem is that , the NFS server is NOT concatenating > data in > cache to data in disk during reads : This defect is making NFS server > useless for , let's say "real time" , applications used concurrently > or as > a single one by the clients without using another "Server" within NFS > server . >=20 >=20 >=20 > In your case , during software builds , a step is using the > previously > generated files : In local disk , writing and reading are sequential > , in > the sense that written data is found during reading . In NFS server > this is > not the case . >=20 >=20 > With respect to my knowledge obtained from messages in FreeBSD > mailing > lists about making a possibility to read data immediately after it is > written into NFS server is NOT available . >=20 >=20 >=20 > Thank you very much . >=20 > Mehmet Erol Sanliturk > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org"