Date: Mon, 27 Jul 2009 02:05:11 +0100 (BST) From: Chris Hedley <freebsd-current@chrishedley.com> To: Matthew Dillon <dillon@apollo.backplane.com> Cc: freebsd-current@freebsd.org Subject: Re: Linux NFS ate my bge Message-ID: <alpine.BSF.2.00.0907270158150.3583@teapot.cbhnet> In-Reply-To: <200907222340.n6MNe3K5013221@apollo.backplane.com> References: <alpine.BSF.2.00.0907222232310.3132@teapot.cbhnet> <200907222307.n6MN7YhU012788@apollo.backplane.com> <alpine.BSF.2.00.0907230010250.3132@teapot.cbhnet> <200907222340.n6MNe3K5013221@apollo.backplane.com>
index | next in thread | previous in thread | raw e-mail
On Wed, 22 Jul 2009, Matthew Dillon wrote: > TCP will likely work better, for several reasons, not the least of > which being that the NFS client does not have to estimate a retransmit > timeout on a rpc-by-rpc basis. Such estimations fail utterly in the > face of a large number of concurrent RPCs because latency winds up being > governed by the disk backlog on the server. A UDP mount will wind up > retransmitting even under completely lossless conditions. > > Another reason TCP tends to work better is that UDP uses IP fragmentation > and IP fragmentation reassembly is not typically in the critical path. > The desired NFS filesystem block size is 16K (smaller will typically > reduce performance), so even a 9000 MTU won't help. It's interesting how this flies in the face of the assumptions I'd made: I'd just guessed that UDP would somehow be the better option, I think I'd had some vague idea it might somehow be more suited to fragmented file chunks being squirted over the network and TCP being a compromise. Well my somewhat wonky assumptions aside, changing over to TCP seems to have fixed it: I haven't seen the problem rematerialise even with a much more protracted network loading than before (essentially emerging [I use Gentoo for Linux] an update of pretty much everything). The performance is better; it could still do with some serious improvement as it's a lot more sluggish than is ideal, though I suspect that the fault lies at the Linux end. Though it may be my configuration options. Of course it took me two attempts to get TCP configured: I'd completely forgotten that I can't simply change it in fstab (I wasn't having a good day when it came to being insightful!) and had to change the entry in the pxelinux config to tell it to use TCP. But I got there in the end, so thank you. :) > Also use netstat ... not sure what option, I think -x, to determine the > actual size of the socket buffer being employed for the connection > (TCP or UDP). There are multiple internal caps in the kernel and it > is often not as big as you might have thought it should be. You want > a 256KB socket buffer at a minimum for a GigE network. Smaller works > (at least for linear transfers), but you lose a lot of RPC concurrency > from the client. Again, something that matters more for a linux client > vs a FreeBSD client. I think this will be my next port of call in order to hopefully get the performance up to a better standard, but there's time for experimenting with that. For now, I'm just happy that my FreeBSD system no longer locks up when being bombarded with requests! Cheers, Chris.home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.BSF.2.00.0907270158150.3583>