From owner-freebsd-hackers Tue Mar 18 13:28:34 2003 Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5654437B401 for ; Tue, 18 Mar 2003 13:28:32 -0800 (PST) Received: from smtp.lynuxworks.com (smtp.Lynuxworks.com [207.21.185.24]) by mx1.FreeBSD.org (Postfix) with ESMTP id AF2B243F3F for ; Tue, 18 Mar 2003 13:28:31 -0800 (PST) (envelope-from mooring@Lnxw.com) Received: from newbast.lynuxworks.com (newbast.lynxworks.com [207.21.185.6]) by smtp.lynuxworks.com (8.11.6/8.9.3) with ESMTP id h2ILSVQ30523; Tue, 18 Mar 2003 13:28:31 -0800 Received: (from mooring@localhost) by newbast.lynuxworks.com (8.11.2/8.11.2) id h2ILSVd23153; Tue, 18 Mar 2003 13:28:31 -0800 Date: Tue, 18 Mar 2003 13:28:31 -0800 From: Ed Mooring To: Borje Josefsson , freebsd-hackers@FreeBSD.ORG Cc: freebsd-hackers@FreeBSD.ORG Subject: Re: High CPU usage on high-bandwidth long distance connections. Message-ID: <20030318132831.P6615@newbast.lynuxworks.com> Reply-To: mooring@Lnxw.com References: <200303181951.h2IJpTKl001940@dc.luth.se> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <200303181951.h2IJpTKl001940@dc.luth.se>; from bj@dc.luth.se on Tue, Mar 18, 2003 at 08:51:29PM +0100 Sender: owner-freebsd-hackers@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, Mar 18, 2003 at 08:51:29PM +0100, Borje Josefsson wrote: [snip scenario] > > The hosts are connected directly (no LAN equipment inbetween) to high > capacity backbone routers (10 Gbit/sec backbone), and are approx 1000 > km/625 miles(!) apart. Measuring RTT gives: > RTTmax = 20.64 ms. Buffer size needed = 3.69 Mbytes, so I add 25% and set: > > sysctl net.inet.tcp.sendspace=4836562 > sysctl net.inet.tcp.recvspace=4836562 > > MTU=4470 all the way. > > OS = FreeBSD 4-STABLE (as of today). > > **** Now the problem: > > The receiver works fine, but on the *sender* I run out if CPU (doesn't > matter if host a or host b is sender). Measuring bandwidth with ttcp gives: > > ttcp-t: buflen=61440, nbuf=30517, align=16384/0, port=5001 tcp > ttcp-t: 1874964480 bytes in 22.39 real seconds = 638.82 Mbit/sec +++ > ttcp-t: 30517 I/O calls, msec/call = 0.75, calls/sec = 1362.82 > ttcp-t: 0.0user 20.8sys 0:22real 93% 16i+382d 326maxrss 0+15pf 9+280csw > > This is very repeatable (within a few %), and is the same regardless of > which direction I use. > > During that period, the sender shows: > > 0.0% user, 0.0% nice, 94.6% system, 5.4% interrupt, 0.0% idle I had something vaguely similar happen while I was porting the FreeBSD 4.2 networking stack to LynxOS. It turned out the culprit was sbappend(). It does a linear pointer chase down the mbuf chain each time you do a write() or send(). With a high bandwidth-delay product, that chain can get very long. This topic came up on freebsd-net last July, and Luigi Rizzo provided the following URL for a patch to cache the end of the mbuf chain, so sbappend() stays O(1) instead of O(n). http://docs.freebsd.org/cgi/getmsg.cgi?fetch=366972+0+archive/2001/freebsd-net/20010211.freebsd-net The subject of the July thread was 'the incredible shrinking socket', if you want to hunt through the archives. Hope this helps. -- Ed Mooring (mooring@lynuxworks.com) To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message