Date: Fri, 12 Oct 2007 22:45:58 +0200 From: Kris Kennaway <kris@FreeBSD.org> To: Mohan Srinivasan <mohans@FreeBSD.org> Cc: freebsd-net@FreeBSD.org, src-committers@FreeBSD.org, cvs-all@FreeBSD.org, cvs-src@FreeBSD.org Subject: Re: cvs commit: src/sys/nfsclient nfs.h nfs_socket.c nfs_subs.c nfsmount.h Message-ID: <470FDD06.4090904@FreeBSD.org> In-Reply-To: <200710121912.l9CJCLeI085992@repoman.freebsd.org> References: <200710121912.l9CJCLeI085992@repoman.freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Mohan Srinivasan wrote: > mohans 2007-10-12 19:12:21 UTC > > FreeBSD src repository > > Modified files: > sys/nfsclient nfs.h nfs_socket.c nfs_subs.c nfsmount.h > Log: > NFS MP scaling changes. > - Eliminate the hideous nfs_sndlock that serialized NFS/TCP request senders > thru the sndlock. > - Institute a new nfs_connectlock that serializes NFS/TCP reconnects. Add > logic to wait for pending request senders to finish sending before > reconnecting. Dial down the sb_timeo for NFS/TCP sockets to 1 sec. > - Break out the nfs xid manipulation under a new nfs xid lock, rather than > over loading the nfs request lock for this purpose. > - Fix some of the locking in nfs_request. > Many thanks to Kris Kennaway for his help with this and for initiating the > MP scaling analysis and work. Kris also tested this patch thorougly. > Approved by: re@ (Ken Smith) For the benefit of others: this change improved throughput by about 10% at high I/O loads with a dual core client, and by a factor of 10 on an 8 core client (this was mostly the home-brew nfs_sndlock, which mohan correctly describes :-). Mohan's previous commit that increases the nfs server socket buffer size is also very important for NFS performance. Without it I was only getting 1-2MB/sec throughput over 10Gb ethernet with UDP mounts, because the minuscule 32kb socket buffer was constantly filling up and forcing retransmits. With the new default of 256KB I still get full buffers with 10ge, so you may need to increase this further to eliminate this problem. It might be OK for gige speeds, although I was still seeing some buffer full events, so maybe we should consider increasing the default sockbuf size to 512KB or so if this is widespread. As a side comment there is a bug in either the nfs client or server that corrupts I/O when there is packet loss with UDP mounts (the default). TCP mounts handle this at the TCP layer. Kris
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?470FDD06.4090904>