From owner-freebsd-performance@FreeBSD.ORG Mon Apr 21 02:24:32 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0C0AB37B401; Mon, 21 Apr 2003 02:24:32 -0700 (PDT) Received: from samson.dc.luth.se (samson.dc.luth.se [130.240.112.30]) by mx1.FreeBSD.org (Postfix) with ESMTP id ABCD343FD7; Mon, 21 Apr 2003 02:24:30 -0700 (PDT) (envelope-from bj@dc.luth.se) Received: from dc.luth.se (root@bompe.dc.luth.se [130.240.60.42]) by samson.dc.luth.se (8.12.5/8.12.5) with ESMTP id h3L9OTLG010830; Mon, 21 Apr 2003 11:24:29 +0200 (MET DST) Received: from bompe.dc.luth.se (bj@localhost.dc.luth.se [127.0.0.1]) by dc.luth.se (8.12.6/8.11.3) with ESMTP id h3L9OT2F032404; Mon, 21 Apr 2003 11:24:29 +0200 (CEST) (envelope-from bj@bompe.dc.luth.se) Message-Id: <200304210924.h3L9OT2F032404@dc.luth.se> X-Mailer: exmh version 2.5 07/13/2001 with nmh-1.0.4 To: "Jin Guojun [NCS]" In-reply-to: Your message of Mon, 21 Apr 2003 10:27:59 +0200. <200304210827.h3L8Rx2F032265@dc.luth.se> Dcc: From: Borje Josefsson X-Disposition-notification-to: Borje.Josefsson@dc.luth.se X-uri: http://www.dc.luth.se/~bj/index.html Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Date: Mon, 21 Apr 2003 11:24:29 +0200 Sender: bj@dc.luth.se cc: freebsd-hackers@freebsd.org cc: freebsd-performance@freebsd.org Subject: Re: patch for test (Was: tcp_output starving -- is due to mbuf get delay?) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: bj@dc.luth.se List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Apr 2003 09:24:32 -0000 On Mon, 21 Apr 2003 10:27:59 +0200 Borje Josefsson wrote: > This patch definitively works, and gives much higher PPS (32000 > instead of 19000). This is on a low-end system (PIII 900MHz with > 33MHz bus), I'll test one of my larger systems later today. = OK. I have now tested on a larger system. Result is better than without the patch, but *not* as good as (for = example) NetBSD or Linux. Value Before patch After patch NetBSD Mbit/sec 617 838 921 PPS (MTU=3D4470) 20000 27500 28000 The problem is (still) that I run out of CPU on the FreeBSD *sender*. Thi= s = doesn't happen on NetBSD (same hardware). The hardware is Xeon 2,8GHz, = PCI-X bus, connected directly to the core routers of a 10 Gbps network. = RTT=3D21 ms, MTU=3D4470. OS=3DFreeBSD 4.8RC with Your patch applied. wilma % vmstat 1 (edited to shorten lines) memory page faults cpu avm fre flt re pi po fr sr in sy cs us sy id 8608 977836 4 0 0 0 0 0 233 20 7 0 2 98 12192 977836 4 0 0 0 0 0 237 59 16 0 1 99 12192 977836 4 0 0 0 0 0 233 20 8 0 2 98 12636 977608 78 0 0 0 7 0 2377 870 241 0 28 72 12636 977608 4 0 0 0 0 0 6522 1834 19 0 100 0 12636 977608 4 0 0 0 0 0 6531 1816 19 0 100 0 12636 977608 4 0 0 0 0 0 6499 1827 19 0 100 0 12636 977608 4 0 0 0 0 0 6575 1821 21 0 100 0 13044 977608 6 0 0 0 0 0 6611 1825 21 0 100 0 top(1) shows: CPU states: 0.0% user, 0.0% nice, 93.4% system, 6.6% interrupt, 0.0% = idle Mem: 6136K Active, 8920K Inact, 34M Wired, 64K Cache, 9600K Buf, 954M Fre= e Swap: 2048M Total, 2048M Free PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 215 root 43 0 1024K 652K RUN 0:11 92.37% 39.11% ttcp Compare that to when I use NetBSD as sender: CPU states: 0.0% user, 0.0% nice, 6.5% system, 5.5% interrupt, 88.1% = idle Memory: 39M Act, 12K Inact, 628K Wired, 2688K Exec, 5488K File, 399M Free= Swap: 1025M Total, 1025M Free PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 17938 root 2 0 204K 688K netio 0:00 7.80% 1.42% ttcp = The "slow ramping" effect that I described in my earlier letter is not at= = all as visible here, so that might be something else (my small test syste= m = has some switches between itself and the core). bge0 in bge0 out total in total out = packets errs packets errs colls packets errs packets errs colls 6 0 4 0 0 7 0 4 0 0 18364 0 12525 0 0 18364 0 12525 0 0 27664 0 18861 0 0 27665 0 18861 0 0 27511 0 18749 0 0 27511 0 18749 0 0 27281 0 18572 0 0 27282 0 18572 0 0 Net result: Much better, but not as good as the "competitors"... --B=F6rje