Date: Wed, 4 Apr 2018 13:20:33 -0700 From: "Russell L. Carter" <rcarter@pinyon.org> To: Mike Tancsa <mike@sentex.net> Cc: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Re: Linux NFS client and FreeBSD server strangeness Message-ID: <6ca61caa-f4a9-68e4-32d9-6142e175e09b@pinyon.org> In-Reply-To: <369fab06-6213-ba87-cc66-c9829e8a76a0@sentex.net> References: <369fab06-6213-ba87-cc66-c9829e8a76a0@sentex.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On 04/04/18 11:27, Mike Tancsa wrote: > Not sure where the tweaking needs to happen, but I am getting strange > behaviour between a Linux nfs client and FreeBSD RELENG_11 NFS server. I initially had terrible performance with a Linux client writing to a FreeBSD server several years back, and after a lot of false starts this snippet in /etc/sysctl.conf solved it: # NFS client sux0rs w/o: net.inet.tcp.tso=0 A caveat is I use NFS V4.1, though. Good luck, Russell > The FreeBSD server starts with > > > nfs_client_enable="YES" > nfs_server_enable="YES" > > > rpcbind_enable="YES" > rpc_lockd_enable="YES" > rpc_statd_enable="YES" > nfs_server_flags="-u -t -n 16" > > and on the Linux client I have been trying various options to no avail. > The mount works, but on a straight up write to the FreeBSD server, > everything is very bursty. I noticed this (I think) a few months ago > where Linux dumps across an nfs mount seemed to take a lot longer and > were getting very bursty. > > It seems if there are a mixture of reads and writes, everything is > pretty fast. But if a client is just writing to the server, something, > somewhere is blocking. Doing something simple like > ls -l /nfsmount > from the client "wakes" up the server/client so that write stream can > keep going. Otherwise, it will do a big blast of writes and then several > seconds of pausing on the dump. > > Linux Dump is a simple > > /sbin/dump u -0 -f - / | /bin/bzip2 >/backup/dump-root-0.bz2 > > Mount is > > mount.nfs -o tcp,intr,noatime,vers=3 192.168.yy.xx:/path > > If I run ifstat on the FreeBSD nfs server, the traffic pattern looks like > > # ifstat -b -i cxl0 > cxl0 > Kbps in Kbps out > 0.00 0.00 > 0.00 0.00 > 0.00 0.00 > 0.00 0.00 > 0.00 0.00 > 8.12e+06 45127.03 > 0.00 0.00 > 0.00 0.00 > 0.00 0.00 > 0.00 0.00 > 6.04e+06 33525.76 > 901122.1 4983.72 > 0.00 0.00 > > if I do a bunch of ls -l /nfsmount on the client > > eg > > while true > do > ls -l /backup/ > /dev/null > done > > traffic pattern is > > > cxl0 > Kbps in Kbps out > 0.00 0.00 > 3.31e+06 18520.03 > 5.89e+06 32571.52 > 4.84e+06 28325.71 > 2.12e+06 19466.56 > 614727.0 12246.10 > 874927.6 13557.18 > 1.06e+06 14386.78 > 917865.4 13696.87 > 1.09e+06 14608.64 > 1.06e+06 14376.12 > 164077.3 5286.64 > > > Leading up to the stall, pcap snippet attached. > > Note, doing something like > > dd if=/dev/zero of=/backup/test.bin bs=4096 count=5000000 > > I can saturate the 10G link and max out the disk on the server > > # dd if=/dev/zero of=/backup/test.bin bs=4096 count=5000000 > 5000000+0 records in > 5000000+0 records out > 20480000000 bytes (20 GB, 19 GiB) copied, 36.6238 s, 559 MB/s > > and its a pretty steady stream unlike the dump. Any ideas whats going > on and how I might be able to work around this ? > > > 192.168.xx.yy:/zbackup1/virtbox4b/backup on /backup type nfs > (rw,noatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.242.254,mountvers=3,mountport=774,mountproto=tcp,local_lock=none,addr=192.168.yy.xx) > > > > ---Mike > > > ------------------- > Mike Tancsa, tel +1 519 651 3400 x203 > Sentex Communications, mike@sentex.net > Providing Internet services since 1994 www.sentex.net > Cambridge, Ontario Canada > > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6ca61caa-f4a9-68e4-32d9-6142e175e09b>