Date: Tue, 2 Feb 2016 07:10:58 +0100 From: Ben Woods <woodsb02@gmail.com> To: Vick Khera <vivek@khera.org> Cc: "freebsd-questions@freebsd.org" <freebsd-questions@freebsd.org>, freebsd-fs@freebsd.org Subject: Re: NFS unstable with high load on server Message-ID: <CAOc73CCHS4r-proJ_jT4T%2BBfcQB9pND8Ld8QZqYJOCkuq2LqiA@mail.gmail.com> In-Reply-To: <CALd%2BdcfzPU=nMGo41BBZzt3jQnsQJaANVyA222TDM_is2Ueo0A@mail.gmail.com> References: <CALd%2BdcfzPU=nMGo41BBZzt3jQnsQJaANVyA222TDM_is2Ueo0A@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Monday, 1 February 2016, Vick Khera <vivek@khera.org> wrote: > I have a handful of servers at my data center all running FreeBSD 10.2. On > one of them I have a copy of the FreeBSD sources shared via NFS. When this > server is running a large poudriere run re-building all the ports I need, > the clients' NFS mounts become unstable. That is, the clients keep getting > read failures. The interactive performance of the NFS server is just fine, > however. The local file system is a ZFS mirror. > > What could be causing NFS to be unstable in this situation? > > Specifics: > > Server "lorax" FreeBSD 10.2-RELEASE-p7 kernel locally compiled, with NFS > server and ZFS as dynamic kernel modules. 16GB RAM, Xeon 3.1GHz quad > processor. > > The directory /u/lorax1 a ZFS dataset on a mirrored pool, and is NFS > exported via the ZFS exports file. I put the FreeBSD sources on this > dataset and symlink to /usr/src. > > > Client "bluefish" FreeBSD 10.2-RELEASE-p5 kernel locally compiled, NFS > client built in to kernel. 32GB RAM, Xeon 3.1GHz quad processor (basically > same hardware but more RAM). > > The directory /n/lorax1 is NFS mounted from lorax via autofs. The NFS > options are "intr,nolockd". /usr/src is symlinked to the sources in that > NFS mount. > > > What I observe: > > [lorax]~% cd /usr/src > [lorax]src% svn status > [lorax]src% w > 9:12AM up 12 days, 19:19, 4 users, load averages: 4.43, 4.45, 3.61 > USER TTY FROM LOGIN@ IDLE WHAT > vivek pts/0 vick.int.kcilink.com 8:44AM - tmux: client > (/tmp/ > vivek pts/1 tmux(19747).%0 8:44AM 19 sed > y%*+%pp%;s%[^_a > vivek pts/2 tmux(19747).%1 8:56AM - w > vivek pts/3 tmux(19747).%2 8:56AM - slogin > bluefish-prv > [lorax]src% pwd > /u/lorax1/usr10/src > > So right now the load average is more than 1 per processor on lorax. I can > quite easily run "svn status" on the source directory, and the interactive > performance is pretty snappy for editing local files and navigating around > the file system. > > > On the client: > > [bluefish]~% cd /usr/src > [bluefish]src% pwd > /n/lorax1/usr10/src > [bluefish]src% svn status > svn: E070008: Can't read directory '/n/lorax1/usr10/src/contrib/sqlite3': > Partial results are valid but processing is incomplete > [bluefish]src% svn status > svn: E070008: Can't read directory '/n/lorax1/usr10/src/lib/libfetch': > Partial results are valid but processing is incomplete > [bluefish]src% svn status > svn: E070008: Can't read directory > '/n/lorax1/usr10/src/release/picobsd/tinyware/msg': Partial results are > valid but processing is incomplete > [bluefish]src% w > 9:14AM up 93 days, 23:55, 1 user, load averages: 0.10, 0.15, 0.15 > USER TTY FROM LOGIN@ IDLE WHAT > vivek pts/0 lorax-prv.kcilink.com 8:56AM - w > [bluefish]src% df . > Filesystem 1K-blocks Used Avail Capacity Mounted on > lorax-prv:/u/lorax1 932845181 6090910 926754271 1% /n/lorax1 > > > What I see is more or less random failures to read the NFS volume. When the > server is not so busy running poudriere builds, the client never has any > failures. > > I also observe this kind of failure doing buildworld or installworld on > the client when the server is busy -- I get strange random failures reading > the files causing the build or install to fail. > > My workaround is to not do build/installs on client machines when the NFS > server is busy doing large jobs like building all packages, but there is > definitely something wrong here I'd like to fix. I observe this on all the > local NFS clients. I rebooted the server before to try to clear this up but > it did not fix it. > > Any help would be appreciated. > I just wanted to point out that I am experiencing this exact same issue in my home setup. Performing an installworld from an NFS mount works perfectly, until I start running poudriere on the NFS server. Then I start getting NFS timeouts and the installworld fails. The NFS server is also using ZFS, but the NFS export in my case is being done via the ZFS property "sharenfs" (I am not using the /etc/exports file). I suspect this will boil down to a ZFS tuning issue, where poudriere and installworld are both stress testing the server. Both of these would obviously cause significant memory and CPU usage, and the "recently used" portion of the ARC to be constantly flushed as they access a large number of different files. It might be interesting if you could report the output of the heading lines (including memory and ARC details) from the "top" command before/after running poudriere and attempting the installworld. Regards, Ben -- -- From: Benjamin Woods woodsb02@gmail.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOc73CCHS4r-proJ_jT4T%2BBfcQB9pND8Ld8QZqYJOCkuq2LqiA>