Date: Tue, 18 Mar 2003 13:28:31 -0800 From: Ed Mooring <emooring@Lnxw.com> To: Borje Josefsson <bj@dc.luth.se>, freebsd-hackers@FreeBSD.ORG Cc: freebsd-hackers@FreeBSD.ORG Subject: Re: High CPU usage on high-bandwidth long distance connections. Message-ID: <20030318132831.P6615@newbast.lynuxworks.com> In-Reply-To: <200303181951.h2IJpTKl001940@dc.luth.se>; from bj@dc.luth.se on Tue, Mar 18, 2003 at 08:51:29PM %2B0100 References: <200303181951.h2IJpTKl001940@dc.luth.se>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Mar 18, 2003 at 08:51:29PM +0100, Borje Josefsson wrote: [snip scenario] > > The hosts are connected directly (no LAN equipment inbetween) to high > capacity backbone routers (10 Gbit/sec backbone), and are approx 1000 > km/625 miles(!) apart. Measuring RTT gives: > RTTmax = 20.64 ms. Buffer size needed = 3.69 Mbytes, so I add 25% and set: > > sysctl net.inet.tcp.sendspace=4836562 > sysctl net.inet.tcp.recvspace=4836562 > > MTU=4470 all the way. > > OS = FreeBSD 4-STABLE (as of today). > > **** Now the problem: > > The receiver works fine, but on the *sender* I run out if CPU (doesn't > matter if host a or host b is sender). Measuring bandwidth with ttcp gives: > > ttcp-t: buflen=61440, nbuf=30517, align=16384/0, port=5001 tcp > ttcp-t: 1874964480 bytes in 22.39 real seconds = 638.82 Mbit/sec +++ > ttcp-t: 30517 I/O calls, msec/call = 0.75, calls/sec = 1362.82 > ttcp-t: 0.0user 20.8sys 0:22real 93% 16i+382d 326maxrss 0+15pf 9+280csw > > This is very repeatable (within a few %), and is the same regardless of > which direction I use. > > During that period, the sender shows: > > 0.0% user, 0.0% nice, 94.6% system, 5.4% interrupt, 0.0% idle I had something vaguely similar happen while I was porting the FreeBSD 4.2 networking stack to LynxOS. It turned out the culprit was sbappend(). It does a linear pointer chase down the mbuf chain each time you do a write() or send(). With a high bandwidth-delay product, that chain can get very long. This topic came up on freebsd-net last July, and Luigi Rizzo provided the following URL for a patch to cache the end of the mbuf chain, so sbappend() stays O(1) instead of O(n). http://docs.freebsd.org/cgi/getmsg.cgi?fetch=366972+0+archive/2001/freebsd-net/20010211.freebsd-net The subject of the July thread was 'the incredible shrinking socket', if you want to hunt through the archives. Hope this helps. -- Ed Mooring (mooring@lynuxworks.com) To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20030318132831.P6615>