Date: Thu, 30 Oct 2014 17:07:31 -0700 From: "Russell L. Carter" <rcarter@pinyon.org> To: Garrett Wollman <wollman@csail.mit.edu>, freebsd-fs@freebsd.org Subject: Re: Definite NFS bug Message-ID: <5452D2C3.9040902@pinyon.org> In-Reply-To: <21586.48982.64913.250497@khavrinen.csail.mit.edu> References: <21586.48982.64913.250497@khavrinen.csail.mit.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 10/30/14 15:44, Garrett Wollman wrote:
> Like many other users, I upgrade my FreeBSD servers by
> NFS-mounting /usr/src and /usr/obj from a shared build server.[1]
> Since I upgraded the build server to 9.3, clients running 9.3
> kernels have been randomly erroring out during installkernel and
> installworld. Today I had some time to look more closely into this
> and found that the error is definitely coming from the server: at
> some point, it just randomly starts returning errors to client
> ACCESS and GETATTR operations. The errors are a mix of NFS3ERR_IO
> and NFS3ERR_ACCES, but there is nothing on the server to indicate
> any kind of error, and restarting the operation on the client
> causes it to fail in a different place. With enough patients and
> restarts, it's possible to complete the installation in just four
> or five passes.
>
> Needless to say this is a bit worrying. Strangely, 9.1 and 9.2
> clients don't see this issue at all; it's only 9.3 clients that
> break.
>
> It's easy to reproduce, just 'cd /usr/sc && find . -type f
> >/dev/null'. It does not seem to depend on the client NFS version
> (3 or 4) or implementation ("old" or "new"). I haven't tried the
> "old" server yet -- I'll need to figure out how to do that first.
>
> If anyone is willing to help debug this, I can share a packet
> trace, but I don't think it's very informative. Also, if anyone
> has a good dtrace script that I could run on the server that would
> report what's going on when that first NFS3ERR_IO is returned, that
> would be great.
This sounds sort of like what I have been complaining about.
I of course have no competency here but if I build the world
- -j1, I have a much better chance of successful remote installs.
The problems I'm seeing on -current for the last few months
seem to me to be out-of-date targets, so that the failure is a
desire by the remote client to try to rebuild the out-of-date target
on the RO file system. My new plan is to dump all of the
st_atim and st_mtim for every .depend list on both systems
when I see the problem again, to see if something jumps out.
I just reinstalled everybody with -j1 builds of r273808M, no problems.
Last week however, a fast box failed. Kind of concerning for
an install to fail say 2/3 through. I have to admit when
soon after I had a crash on that 2/3 system (on NFS unmount),
I had to step out of the room for the reboot. Exciting.
I am traveling on Sunday for a week, but I've got a few days to
run things on several big fast 8cpu boxes (my old laptop is
much less afflicted with this problem, though it occasionally
fails too).
Russell
> -GAWollman
>
> [1] I'd run my own freebsd-update server but unfortunately it is
> too tied to building things that look like official FreeBSD
> security updates, and isn't really designed for (e.g.) updating
> kernels when we change a configuration option. It also doesn't
> have any obvious knobs for building with anything other than a
> default {make,src}.conf. And with a pkg-able base just around the
> corner I don't really want to put much effort into making
> freebsd-update do what I want. NFS, on the other hand, is a big
> deal and so I need to track down and fix these bugs.
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs To
> unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAEBAgAGBQJUUtLDAAoJEFnLrGVSDFaEu0MQAJOlPWcsduuiS75LUe42uj+E
SRnxSvm5JgUdJojatx7cL5TQjEvXbYov8CE8OLZUqGxIi0D0IdpKlr6WJes8KOUC
wfix7doQZQe3IPqgYAJZz0y6j89q6+QABPTS2oy+cPpYmop9568TvuJJZCCixBOF
Zv3XYa4I7uIl1pYF2zl2nJHtOwLi2wjT+851heqXo8GvIo8SAhBouTN5biPh2JGl
Yabbb4e5xePvigMLEwxbPNslv3nhT1JOcsH9GoFLo5zph2+Txw6ZPy1Sccyv88AQ
w5ID129VMzZChX6zYT7+LtJYLmZME3bVrA2R6YeEdnr/Is8qm5eKtpkMrUz+5Qn4
ULf3fJSCjYdlfatfBIFfi2jFJWBkBY7qVu9S5nqfG9yn4DCLY2UYl4skP71Eo4hz
DPDKQwpuij/Tf8y459Vj60AsOt87Sh0eYBnW+nWJdgIPWptYLNmjv/VHvC8ZFbnn
HsrvUw9DovnTfd7rn+GR4F4+nlnjXqOKdPJtLroId3tSxZzy9L08n7Y6AvAWFFWM
oQ4q/B4LxpOmjXqIBTCrC5ux7GdtKGN2gkAYvY4zh3ngPJJ9ts0BRHbq2zRMo9OA
eUT8Cf+D/wQcFcd+27eI1RJu8IbyycStwGMXbA57UkvJkfSA5CVpcey+T5z9uyPa
7xlgxCpHOIHSJ6l2BeSQ
=4Q5V
-----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5452D2C3.9040902>