Date: Wed, 1 Jul 2015 20:10:31 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: Graham Allan <allan@physics.umn.edu> Cc: freebsd-fs@freebsd.org Subject: Re: Strange NFS problem implicating nfsuserd? Message-ID: <972685551.2776991.1435795831472.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <55946FFE.8070402@physics.umn.edu> References: <55946FFE.8070402@physics.umn.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
Graham Allan wrote:
> I spent a few days digging into a strange NFSv4 problem at our site,
> which I think I may have finally resolved but don't really understand why.
>
> We have a bunch of large-ish NFS servers running FreeBSD 9.3 exporting
> ZFS filesystems to mostly "RHEL-clone" linux clients. Over the last few
> weeks I started getting reports that peoples' jobs would fail
> erratically with i/o errors, and it became apparent that they pointed in
> general to all our FreeBSD NFS servers rather than just one.
>
> Ultimately I could trivially reproduce the problem running
> "find . -type f -exec cat {} > /dev/null \;"
> on one of the NFS-mounted filesystems.
>
> Linux clients would eventually error with "Input/output error"
> FreeBSD clients would eventually error with "Permission denied" on files
> or directories which should be readable.
>
> Reverting to earlier patch releases didn't make any difference, though
> it seemed like the problem started roughly when I updated p8->p13.
>
> Finally I seem to have pinpointed it to one change made in rc.conf for
> nfsuserd, which I committed at around the right date:
>
> nfsuserd_flags="-usermax 500 -usertimeout 600 16"
>
> became:
>
> nfsuserd_flags="-domain xxx.yyy.zzz -usermax 500 -usertimeout 600 16"
>
> probably because I saw a user mapping failure somewhere previously, and
> decided to make the domain explicit.
>
> Undoing this change appears to eliminate the problem - but this makes no
> sense to me. Starting nfsuserd with either set of options (adding
> -verbose) prints the same output:
>
> Starting nfsuserd.
> nfsuserd: domain=xxx.yyy.zzz usermax=500 usertimeout=36000
>
> So the domain chosen by default is the same as the one explicitly
> specified (as I would expect).
>
> I've reproduced this across 4-5 different servers and a similar number
> of different client systems. I'm wondering if any plausible explanation
> suggests itself?
>
As far as I know, the domain is only set when
the nfsuserd is started and it just uses the domain part of the machine's
host name if not explicitly defined by "-domain". Maybe there is some bug
in nfsuserd.c that gets tickled by the option, although I just looked and
the argument parsing looks ok.
If your xxx.yyy.zzz is identical, then I can't see how this would affect
anything.
What will cause intermittent mapping problems is having more than one
username that maps to the same uid. (One of them will be cached at random.)
(There was a common case of both "root" and "toor" in the password database
for uid == 0.)
rick
> Graham
> --
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?972685551.2776991.1435795831472.JavaMail.zimbra>