Date: Wed, 11 Jun 2003 02:40:40 -0700 From: Terry Lambert <tlambert2@mindspring.com> To: Sean Chittenden <sean@chittenden.org> Cc: Eric Anderson <anderson@centtech.com> Subject: Re: Slow disk write speeds over network Message-ID: <3EE6F918.1C1FF28E@mindspring.com> References: <20030609211526.58641.qmail@web14908.mail.yahoo.com> <3EE4FAED.6090603@centtech.com> <3EE595D2.B223CA19@mindspring.com> <3EE5F8DE.30001@centtech.com> <20030610195632.GQ65470@perrin.int.nxad.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Sean Chittenden wrote: > > >...and yet more sysctl's for this: > > > > > > kern.polling.enable=1 > > > kern.polling.user_frac=50 # 0..100; whatever works best > > > > > >If you've got a really terrible Gigabit Ethernet card, then > > >you may be copying all your packets over again (e.g. m_pullup()), > > >and that could be eating your bus, too. > > > > Ok, so the end result is that after playing around with sysctl's, > > I've found that the tcp transfers are doing 20MB/s over FTP, but my > > NFS is around 1-2MB/s - still slow.. So we've cleared up some tcp > > issues, but 2yet still NFS is stinky.. > > > > Any more ideas? > > I'm using UDP NFS over a 100Mbit/FD link with the following settings > and get about 12-14Mbps: Numbers taken in context of original poster... YMMV: > net.inet.tcp.recvspace=65536 This is most important for writes. The sendspace is pretty well not going to help you out, unless you are starvation deadlocked; it didn't look like you were from your previous posting. BTW: I believe this is the default. > net.inet.tcp.sendspace=65536 Double the default. Might not be a good idea, unless you have a ton of memory. You will potentially use 64K send + 64K receive times number of sockets. Assuming 4G and near-perfect tuning, you will be limited to 16384 simultaneous connections fully packed before memory pressure causes your machine to crash. I tend to like smaller buffers and more connections. If you only have 512M, drop this number to 2048 simultaneous connections if all buffers are full. > kern.maxfiles=65536 Seems kind of overkill for the number of connections you can support without overcommit, and the number of client machines you say you have. > kern.ipc.maxsockbuf=2097152 > kern.ipc.somaxconn=8192 IPC numbers; not relevent. > net.inet.tcp.delayed_ack=0 This will make it more responsive, at some cost in overhead. > net.inet.udp.recvspace=65536 > net.inet.udp.maxdgram=57344 These are important for UDP NFS. I do not reccomend it. > net.local.stream.sendspace=65536 > net.local.stream.recvspace=65536 IPC numbers; not relelvent. > vfs.nfs.async=1 This is very dangerous, if you care about your data. It permits NFS to ACK writes before they have been committed to stable storage. With a large enough window size, this should not be necessary. > net.inet.udp.log_in_vain=1 This is just overhead; I reccomend turning it off. > net.inet.icmp.icmplim=20000 This is only useful for TCP; but it can be very useful. Basically, this is "connection rate limiting". If you have a ton of clients, or trying to "netbench" the system, then set this number up. For 100 NFS clients, it likely does not matter. > I'm not taking into account jumbo frames or anything like that, so you > may want to increase the size of some of these values where > appropriate, but some of these may be a start. -sc In my experience, Intel GigE cards do not play nice with others when it comes to jumbo frames or negotiation. I much prefer the Tigon/Alteon/Broadcom/whoever-they-are-this-week-still-no-firmware, though I would obviously like the same firmware access to the Tigon III's as they used to give us to the Tigon II's. -- Terry
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3EE6F918.1C1FF28E>