Date: Mon, 15 Dec 2014 15:59:29 -0500 (EST) From: Rick Macklem <rmacklem@uoguelph.ca> To: Mehmet Erol Sanliturk <m.e.sanliturk@gmail.com> Cc: freebsd-net@freebsd.org, Gerrit =?utf-8?B?S8O8aG4=?= <gerrit.kuehn@aei.mpg.de> Subject: Re: compiling on nfs directories Message-ID: <2048229686.13136235.1418677169130.JavaMail.root@uoguelph.ca> In-Reply-To: <CAOgwaMs%2BYLUoLSHDsu6BOYgwr_oi09xNk9yOnSNYjjXqaiDCQQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Mehmet Erol Sanliturk wrote:
> On Mon, Dec 15, 2014 at 1:24 AM, Gerrit K=C3=BChn
> <gerrit.kuehn@aei.mpg.de>
> wrote:
> >
> > Hi all,
> >
> > I ran into some weird issue here last week:
> > I have an NFS-Server for storage and diskless booting (pxe / nfs
> > root)
> > running under FreeBSD. The clients are running Gentoo Linux. Some
> > time
> > ago, I replaced the server, going from a HDD-based storage array
> > (ZFS)
> > under FreeBSD 8.3 to an SSD-based array under FreeBSD 10-stable (as
> > of
> > February this year - I know this needs updates).
> >
> > Only now I recognized that this somehow appears to have broken some
> > of my
> > Gentoo ebuilds that do not install cleanly anymore. They complain
> > about
> > "soiled libtool library files found" and "insecure RUNPATHs" in the
> > installation stage of shared libs.
> >
> > I was not able to find any useful solution for this in the Net so
> > far.
> > However, I was able to verify that this is somehow an issue with
> > the nfs
> > server by plugging in a USB-drive into the diskless clients and
> > mounting
> > this as /var/tmp/portage (the directory structure where Gentoo's
> > ebuilds
> > are compiled). This makes the error messages go away, and
> > everything works
> > again (like it did before the server update).
> >
> > Are there any suggestions what might be causing this and how to fix
> > it?
> >
> >
> > cu
> > Gerrit
> >
> >
>=20
>=20
> With respect to information given in your message , may pure guess is
> the
> following :
>=20
>=20
> When a client generates a file in NFS server , it assumes that
> everything
> is written into the file .
> The next step ( reading the generated file ) starts , BUT the file is
> NOT
> completely written into disk yet ,
> therefore it reads an incomplete file which causes errors in the
> client .
>=20
Well, not exactly. The NFS client chooses whether or not the written
data must be committed to stable storage (disk) right away via a flag
argument on the write. It is up to the client to keep track of what has
been written and if the FILE_STABLE flag wasn't set, must do a separate
Commit RPC to force the data to stable storage on the server.
It is also up to the NFS client to keep track of the file's size while
it is being grown, since the NFS server's size may be smaller until
the data gets written to the server.
Also, note that he didn't see the problem with FreeBSD8.3, which would
have been following the same rules on the server as 10.1.
What I suspect might cause this is one of two things:
1 - The modify time of the file is now changing at a time the Linux
client doesn't expect, due to changes in ZFS or maybe TOD clock
resolution. (At one time, the TOD clock was only at a resolution
of 1sec, so the client wouldn't see the modify time change often.
I think it is now at a much higher resolution, but would have to
look at the code/test to be sure.)
2 - I think you mention this one later in your message, in that the
build might be depending on file locking. If this is the case,
trying NFSv4, which does better file locking, might fix the
problem.
Gerrit, I would suggest that you do "nfsstat -m" on the Linux client,
to see what the mount options are. The Linux client might be using NFSv4
already.
Also, avoid "soft, intr" especially if you are using NFSv4, since these
can cause slow server response to result in a failure of a read/write
when it shouldn't fail, due to timeout or interruption by a signal.
If you could find out more about what causes the specific build failure
on the Linux side, that might help.
If you can reproduce a build failure quickly/easily, you can capture
packets via "tcpdump -s 0 -w <file> host <client-hostname>" on the
server and then look at it in wireshark to see what the server is replying
when the build failure occurs. (I don't mind looking at a packet trace if
it is relatively small, if you email it to me as an attachment.)
Good luck with it, rick
ps: I am not familiar with the Linux mount options, but if it has
stuff like "nocto", you could try those.
> In FreeBSD NFS server , there is NOT ( or I could NOT be able to find
> ) a
> facility to store written data immediately into disk .
>=20
> NFS server is collecting data up to a point ( number of bytes ) and
> then
> writing it to disk , during this phase ( whether the NFS server is
> busy or
> not ) is not important ) . With this structure ,
> the tasks which a program writes a small number of bytes to be read
> by
> another program can not be
> processed by a NFS server only .
>=20
> I did not try "locking in NFS server" : If this route is taken , then
> it is
> necessary to adjust the clients for such periods to wait that NFS
> server
> has removed the lock which themselves can continue ( Each such read
> requires a waiting loop without generating an error message about
> unavailable data and termination . ) .
>=20
> In Linux NFS server , there is an option to immediately write the
> received
> data into disk . This is improving the above situation considerable
> but not
> completely solving the problem ( because during reads of data , data
> in
> cache is NOT concatenated to the data in disk ) .
>=20
>=20
> Another MAJOR problem is that , the NFS server is NOT concatenating
> data in
> cache to data in disk during reads : This defect is making NFS server
> useless for , let's say "real time" , applications used concurrently
> or as
> a single one by the clients without using another "Server" within NFS
> server .
>=20
>=20
>=20
> In your case , during software builds , a step is using the
> previously
> generated files : In local disk , writing and reading are sequential
> , in
> the sense that written data is found during reading . In NFS server
> this is
> not the case .
>=20
>=20
> With respect to my knowledge obtained from messages in FreeBSD
> mailing
> lists about making a possibility to read data immediately after it is
> written into NFS server is NOT available .
>=20
>=20
>=20
> Thank you very much .
>=20
> Mehmet Erol Sanliturk
> _______________________________________________
> freebsd-net@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to
> "freebsd-net-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2048229686.13136235.1418677169130.JavaMail.root>