Date: Sat, 25 Jan 2014 21:36:26 -0500 (EST) From: Rick Macklem <rmacklem@uoguelph.ca> To: wollman@freebsd.org Cc: freebsd-fs@freebsd.org, freebsd-net@freebsd.org Subject: Re: Terrible NFS performance under 9.2-RELEASE? Message-ID: <188195924.16327973.1390703786000.JavaMail.root@uoguelph.ca> In-Reply-To: <201401260225.s0Q2PUp1045129@hergotha.csail.mit.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
Garrett Wollman wrote:
> In article
> <278396201.16318356.1390701347722.JavaMail.root@uoguelph.ca>, Rick
> Macklem writes:
>
> >Well, when I get home in April, I'll try the fairly recent Linux
> >client
> >I have at home and see what it does. Not sure what trick they could
> >use
> >to avoid the read before write for partial pages. (I suppose I can
> >look at their sources, but that could be pretty scary;-)
>
> For what it's worth, our performance for large-block 100%-read
> workloads is also not what it could (or ought to) be. Between two
> 20G-attached servers, I can get about 12 Gbit/s with three parallel
> TCP connections. (Multiple connections are required to trick the
> lagg
> hash into balancing the load across both 10G links, because the hash
> function used for load-balancing uses the source and destination
> ports.) On the same pair of servers, "dd if=/mnt/test bs=1024k" runs
> at about 3 Gbit/s, whereas reading from the local filesystem goes
> anywhere from 1.5 to 3 G*byte*/s (i.e., eight times faster) with much
> higher CPU utilization. Luckily, most of our users are only
> connected
> at 1G so they don't notice.
>
Have you tried increasing readahead by any chance? I think the default
is 1, which means the client will make 2 read requests and then wait for
those replies before doing any more reads. Since you have fast links,
maybe the 2 * 64K reads isn't enough to keep the pipe filled? (This
depends on latency, which you didn't mention.)
Might be worth trying, rick
ps: If/when you have a test server, you could also try compiling a
kernel with MAXBSIZE set to 128Kbytes instead of 64Kbytes. You'll
need to boot this kernel on both the server and client (assuming
a FreeBSD client) before the default rsize will increase to 128Kbytes.
I'm no ZFS guy, but I understand 128Kbytes is the blocksize it likes.
> I'm going to lose my test server soon (it has to go into production
> shortly), so I'm not really able to work on this. I'll have another
> test server soon (old hardware being replaced by the new server) and
> hope to be able to try out the new code that's going to be in 10.1,
> with the expectation of upgrading to 10.x over summer break.
>
> -GAWollman
>
> _______________________________________________
> freebsd-net@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to
> "freebsd-net-unsubscribe@freebsd.org"
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?188195924.16327973.1390703786000.JavaMail.root>