From owner-freebsd-net@FreeBSD.ORG Mon Dec 15 23:06:35 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 61C915E7 for ; Mon, 15 Dec 2014 23:06:35 +0000 (UTC) Received: from mail-yk0-x232.google.com (mail-yk0-x232.google.com [IPv6:2607:f8b0:4002:c07::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 18ACBA1A for ; Mon, 15 Dec 2014 23:06:35 +0000 (UTC) Received: by mail-yk0-f178.google.com with SMTP id 20so5391443yks.37 for ; Mon, 15 Dec 2014 15:06:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=mYTzQSPMCtHUyjn/3J34WuNuoAtRWVUgv6rXNE50Ids=; b=YR6RWYJzt4HidCWgaZI6h8v7lnIxXJJyv7iJTwav1NCOf3xdVWS9zH4EWmaTICPOC9 KQrBVnBxGYRXpcOxUubwRCuwSH6YZIIXGyHrhr4krBWH/8eVA22xnfUCgF57t1fWyNRA cT6ZyJyp3yRw8/fYrqMUqJBpN+80txO+do1JwGVOhmVcORC4hWPgXQ9QqiPxKrsaBv7e SWVbq6ZOua04gohgvimM/AfxXJXpAFU30kENNFnbfhq6SrbTUrMHVUL13zDc/3Wfpiga J/6lOF4syzVyDktgUWcB+ZXLBms68aHMWzfjGYvayEbJGsBlA+LbjqtrXuCz6qOROEBO ekUg== MIME-Version: 1.0 X-Received: by 10.236.221.168 with SMTP id r38mr24188840yhp.137.1418684794237; Mon, 15 Dec 2014 15:06:34 -0800 (PST) Received: by 10.170.90.131 with HTTP; Mon, 15 Dec 2014 15:06:34 -0800 (PST) In-Reply-To: <2048229686.13136235.1418677169130.JavaMail.root@uoguelph.ca> References: <2048229686.13136235.1418677169130.JavaMail.root@uoguelph.ca> Date: Mon, 15 Dec 2014 15:06:34 -0800 Message-ID: Subject: Re: compiling on nfs directories From: Mehmet Erol Sanliturk To: Rick Macklem Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-net@freebsd.org, =?UTF-8?B?R2Vycml0IEvDvGhu?= X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2014 23:06:35 -0000 On Mon, Dec 15, 2014 at 12:59 PM, Rick Macklem wrote= : > > Mehmet Erol Sanliturk wrote: > > On Mon, Dec 15, 2014 at 1:24 AM, Gerrit K=C3=BChn > > > > wrote: > > > > > > Hi all, > > > > > > I ran into some weird issue here last week: > > > I have an NFS-Server for storage and diskless booting (pxe / nfs > > > root) > > > running under FreeBSD. The clients are running Gentoo Linux. Some > > > time > > > ago, I replaced the server, going from a HDD-based storage array > > > (ZFS) > > > under FreeBSD 8.3 to an SSD-based array under FreeBSD 10-stable (as > > > of > > > February this year - I know this needs updates). > > > > > > Only now I recognized that this somehow appears to have broken some > > > of my > > > Gentoo ebuilds that do not install cleanly anymore. They complain > > > about > > > "soiled libtool library files found" and "insecure RUNPATHs" in the > > > installation stage of shared libs. > > > > > > I was not able to find any useful solution for this in the Net so > > > far. > > > However, I was able to verify that this is somehow an issue with > > > the nfs > > > server by plugging in a USB-drive into the diskless clients and > > > mounting > > > this as /var/tmp/portage (the directory structure where Gentoo's > > > ebuilds > > > are compiled). This makes the error messages go away, and > > > everything works > > > again (like it did before the server update). > > > > > > Are there any suggestions what might be causing this and how to fix > > > it? > > > > > > > > > cu > > > Gerrit > > > > > > > > > > > > With respect to information given in your message , may pure guess is > > the > > following : > > > > > > When a client generates a file in NFS server , it assumes that > > everything > > is written into the file . > > The next step ( reading the generated file ) starts , BUT the file is > > NOT > > completely written into disk yet , > > therefore it reads an incomplete file which causes errors in the > > client . > > > Well, not exactly. The NFS client chooses whether or not the written > data must be committed to stable storage (disk) right away via a flag > argument on the write. It is up to the client to keep track of what has > been written and if the FILE_STABLE flag wasn't set, must do a separate > Commit RPC to force the data to stable storage on the server. > It is also up to the NFS client to keep track of the file's size while > it is being grown, since the NFS server's size may be smaller until > the data gets written to the server. > Also, note that he didn't see the problem with FreeBSD8.3, which would > have been following the same rules on the server as 10.1. > > What I suspect might cause this is one of two things: > 1 - The modify time of the file is now changing at a time the Linux > client doesn't expect, due to changes in ZFS or maybe TOD clock > resolution. (At one time, the TOD clock was only at a resolution > of 1sec, so the client wouldn't see the modify time change often. > I think it is now at a much higher resolution, but would have to > look at the code/test to be sure.) > 2 - I think you mention this one later in your message, in that the > build might be depending on file locking. If this is the case, > trying NFSv4, which does better file locking, might fix the > problem. > > Gerrit, I would suggest that you do "nfsstat -m" on the Linux client, > to see what the mount options are. The Linux client might be using NFSv4 > already. > Also, avoid "soft, intr" especially if you are using NFSv4, since these > can cause slow server response to result in a failure of a read/write > when it shouldn't fail, due to timeout or interruption by a signal. > > If you could find out more about what causes the specific build failure > on the Linux side, that might help. > If you can reproduce a build failure quickly/easily, you can capture > packets via "tcpdump -s 0 -w host " on the > server and then look at it in wireshark to see what the server is replyin= g > when the build failure occurs. (I don't mind looking at a packet trace if > it is relatively small, if you email it to me as an attachment.) > > Good luck with it, rick > ps: I am not familiar with the Linux mount options, but if it has > stuff like "nocto", you could try those. > > > In FreeBSD NFS server , there is NOT ( or I could NOT be able to find > > ) a > > facility to store written data immediately into disk . > > > > NFS server is collecting data up to a point ( number of bytes ) and > > then > > writing it to disk , during this phase ( whether the NFS server is > > busy or > > not ) is not important ) . With this structure , > > the tasks which a program writes a small number of bytes to be read > > by > > another program can not be > > processed by a NFS server only . > > > > I did not try "locking in NFS server" : If this route is taken , then > > it is > > necessary to adjust the clients for such periods to wait that NFS > > server > > has removed the lock which themselves can continue ( Each such read > > requires a waiting loop without generating an error message about > > unavailable data and termination . ) . > > > > In Linux NFS server , there is an option to immediately write the > > received > > data into disk . This is improving the above situation considerable > > but not > > completely solving the problem ( because during reads of data , data > > in > > cache is NOT concatenated to the data in disk ) . > > > > > > Another MAJOR problem is that , the NFS server is NOT concatenating > > data in > > cache to data in disk during reads : This defect is making NFS server > > useless for , let's say "real time" , applications used concurrently > > or as > > a single one by the clients without using another "Server" within NFS > > server . > > > > > > > > In your case , during software builds , a step is using the > > previously > > generated files : In local disk , writing and reading are sequential > > , in > > the sense that written data is found during reading . In NFS server > > this is > > not the case . > > > > > > With respect to my knowledge obtained from messages in FreeBSD > > mailing > > lists about making a possibility to read data immediately after it is > > written into NFS server is NOT available . > > > > > > > > Thank you very much . > > > > Mehmet Erol Sanliturk > > _______________________________________________ > > When a C program is written to be used in an NFS environment , some possibilities may be used to synchronize write and reads from the programs with the unsolved "cached data" problem . When ready programs are used , such as "make" , "ld" , there is no choice . I am using Pascal programs , then there is no such facilities . The solution may be to improve the NFS Server and Client modules to use cached data during reads : If end of file is reached : Before sending EOF signal , check whether there is data in cache or not . If there is data in cache : continue reading from the cache up to end , else send an EOF signal . ( For random access files , also there is a need to look at the cached values . ) Since the above modification requires knowledge of internal structure of NFS Server , and perhaps NFS Client ,I am not able to supply any patch . Also I am not able to understand its implementation difficulty . My opinion is that the above modification would be a wonderful improvement for NFS system in FreeBSD , because it will behave just like a local data store usable as "real time" data processing tasks . In the present structure , this is NOT possible with NFS Client and Server only . Thank you very much . Mehmet Erol Sanliturk