From owner-freebsd-hackers@FreeBSD.ORG Fri Apr 11 09:35:06 2003 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6524637B401; Fri, 11 Apr 2003 09:35:06 -0700 (PDT) Received: from samson.dc.luth.se (samson.dc.luth.se [130.240.112.30]) by mx1.FreeBSD.org (Postfix) with ESMTP id F0FD143F3F; Fri, 11 Apr 2003 09:35:04 -0700 (PDT) (envelope-from bj@dc.luth.se) Received: from dc.luth.se (root@bompe.dc.luth.se [130.240.60.42]) by samson.dc.luth.se (8.12.5/8.12.5) with ESMTP id h3BGZ0jY022899; Fri, 11 Apr 2003 18:35:00 +0200 (MET DST) Received: from bompe.dc.luth.se (bj@localhost.dc.luth.se [127.0.0.1]) by dc.luth.se (8.12.6/8.11.3) with ESMTP id h3BGZ0Kl087202; Fri, 11 Apr 2003 18:35:00 +0200 (CEST) (envelope-from bj@bompe.dc.luth.se) Message-Id: <200304111635.h3BGZ0Kl087202@dc.luth.se> X-Mailer: exmh version 2.5 07/13/2001 with nmh-1.0.4 To: Terry Lambert In-reply-to: Your message of Fri, 11 Apr 2003 09:22:47 PDT. <3E96EBD7.5CA4C171@mindspring.com> Dcc: X-Disposition-notification-to: Borje.Josefsson@dc.luth.se X-uri: http://www.dc.luth.se/~bj/index.html Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Date: Fri, 11 Apr 2003 18:35:00 +0200 From: Borje Josefsson cc: freebsd-hackers@freebsd.org cc: freebsd-performance@freebsd.org cc: "Jin Guojun \[DSD\]" cc: Eric Anderson cc: David Gilbert Subject: Re: tcp_output starving -- is due to mbuf get delay? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: bj@dc.luth.se List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Apr 2003 16:35:06 -0000 On Fri, 11 Apr 2003 09:22:47 PDT Terry Lambert wrote: > Borje Josefsson wrote: > > I should add that I have tried with MTU 1500 also. Using NetBSD as se= nder > > works fine (just a little bit higher CPU load). When we tried MTU1500= with > > FreeBSD as sender, we got even lower performance. > > = > > Somebody else in this thread said that he had got full GE speed betwe= en > > two FreeBSD boxes connected back-to-back. I don't question that, but = that > > doesn't prove anything. The problem arises when You are trying to do = this > > long-distance and have to handle a large mbuf queue. > = > The boxes were not connected "back to back", they were connected > through three Gigabit switches and a VLAN trunk. But they were in > a lab, yes. > = > I'd be happy to try long distance for you, and even go so far as > to fix the problem for you, if you are willing to drop 10GBit fiber > to my house. 8-) 8-). = > = > As far as a large mbuf queue, one thing that's an obvious difference > is SACK support; however, this can not be the problem, since the > NetBSD->FreeBSD speed is unafected (supposedly). > = > What is the FreeBSD->NetBSD speed? > = > Some knobs to try on FreeBSD: ttcp-t: buflen=3D61440, nbuf=3D20345, align=3D16384/0, port=3D5001 ttcp-t: socket ttcp-t: connect ttcp-t: 1249996800 bytes in 16.82 real seconds =3D 567.09 Mbit/sec +++ ttcp-t: 20345 I/O calls, msec/call =3D 0.85, calls/sec =3D 1209.79 ttcp-t: 0.0user 15.5sys 0:16real 92% 16i+380d 326maxrss 0+15pf 16+232csw During that time "top" shows (on the sender): CPU states: 0.4% user, 0.0% nice, 93.0% system, 6.6% interrupt, 0.0% = idle Just for comparation, running in the other direction (with NetBSD as = sender): CPU states: 0.0% user, 0.0% nice, 19.9% system, 13.9% interrupt, 66.2% = idle Netstat of that test: bge0 in bge0 out total in total out = packets errs packets errs colls packets errs packets errs colls 14709 0 22742 0 0 14709 0 22742 0 0 18602 0 28002 0 0 18602 0 28002 0 0 18603 0 28006 0 0 18603 0 28006 0 0 18599 0 28009 0 0 18600 0 28009 0 0 18605 0 28006 0 0 18605 0 28006 0 0 18607 0 28006 0 0 18608 0 28006 0 0 18608 0 28006 0 0 18608 0 28006 0 0 bge0 in bge0 out total in total out = packets errs packets errs colls packets errs packets errs colls 14389511 908 14772089 0 0 18404167 908 18231823 0 0 18607 0 28003 0 0 18607 0 28003 0 0 18598 0 28006 0 0 18599 0 28006 0 0 5823 0 8161 0 0 5823 0 8161 0 0 Note how stable it is! > net.inet.ip.intr_queue_maxlen -> 300 > net.inet.ip.check_interface -> 0 > net.inet.tcp.rfc1323 -> 0 > net.inet.tcp.inflight_enable -> 1 > net.inet.tcp.inflight_debug -> 0 > net.inet.tcp.delayed_ack -> 0 > net.inet.tcp.newreno -> 0 > net.inet.tcp.slowstart_flightsize -> 4 > net.inet.tcp.msl -> 1000 > net.inet.tcp.always_keepalive -> 0 > net.inet.tcp.sendspace -> 65536 (on sender) Hmm. Isn't 65536 an order of magnitude to low? I used = tcp.sendspace=3D3223281 for the test above. RTTmax =3D 20.63 ms. Buffer s= ize = needed =3D 2578625 bytes, and then add a few %. > Don't try them all at once and expect magic; you will probably need > some combination. > = > Also, try recompiling your kernel *without* IPSEC support. I'll do that. --B=F6rje