From owner-freebsd-hackers@FreeBSD.ORG Fri Apr 11 10:09:29 2003 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E97B737B401; Fri, 11 Apr 2003 10:09:28 -0700 (PDT) Received: from samson.dc.luth.se (samson.dc.luth.se [130.240.112.30]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4E81443FA3; Fri, 11 Apr 2003 10:09:27 -0700 (PDT) (envelope-from bj@dc.luth.se) Received: from dc.luth.se (root@bompe.dc.luth.se [130.240.60.42]) by samson.dc.luth.se (8.12.5/8.12.5) with ESMTP id h3BH9MjY028181; Fri, 11 Apr 2003 19:09:22 +0200 (MET DST) Received: from bompe.dc.luth.se (bj@localhost.dc.luth.se [127.0.0.1]) by dc.luth.se (8.12.6/8.11.3) with ESMTP id h3BH9LKl087299; Fri, 11 Apr 2003 19:09:21 +0200 (CEST) (envelope-from bj@bompe.dc.luth.se) Message-Id: <200304111709.h3BH9LKl087299@dc.luth.se> X-Mailer: exmh version 2.5 07/13/2001 with nmh-1.0.4 To: Terry Lambert In-reply-to: Your message of Fri, 11 Apr 2003 09:32:51 PDT. <3E96EE33.FAF4FABB@mindspring.com> Dcc: X-Disposition-notification-to: Borje.Josefsson@dc.luth.se X-uri: http://www.dc.luth.se/~bj/index.html Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Fri, 11 Apr 2003 19:09:21 +0200 From: Borje Josefsson cc: freebsd-hackers@freebsd.org cc: freebsd-performance@freebsd.org cc: "Jin Guojun \[DSD\]" cc: Eric Anderson cc: David Gilbert Subject: Re: tcp_output starving -- is due to mbuf get delay? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: bj@dc.luth.se List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Apr 2003 17:09:29 -0000 On Fri, 11 Apr 2003 09:32:51 PDT Terry Lambert wrote: > See other posting; I listed a bunch of OIDs to play with. > = > One other, if you are running -CURRENT, would be: > = > net.isr.netisr_enable -> 1 I'm running 4.8RC on the sender right now. I did a quick test with some combination of the OID:s You sent, except I = didn't reboot between each test: > Some knobs to try on FreeBSD: > > net.inet.ip.intr_queue_maxlen -> 300 > net.inet.ip.check_interface -> 0 > net.inet.tcp.rfc1323 -> 0 > net.inet.tcp.inflight_enable -> 1 > net.inet.tcp.inflight_debug -> 0 > net.inet.tcp.delayed_ack -> 0 > net.inet.tcp.newreno -> 0 > net.inet.tcp.slowstart_flightsize -> 4 > net.inet.tcp.msl -> 1000 > net.inet.tcp.always_keepalive -> 0 > net.inet.tcp.sendspace -> 65536 (on sender) I didn't test all combinations, but all of them give the same result (~57= 0 = Mbit/sec, and I run out of CPU), except inflight_enable, which = *considerably* lowers the performance.I get around 570 Mbit/sec, and run = out of CPU. Netstat -m (tcp and ip portion) when I started and after the trials: =3D=3D=3D before =3D=3D=3D tcp: 12103350 packets sent 2692690 data packets (1962127658 bytes) 10203 data packets (14829100 bytes) retransmitted 0 resends initiated by MTU discovery 6446084 ack-only packets (199 delayed) 0 URG only packets 0 window probe packets 2955524 window update packets 216 control packets 16306846 packets received 847040 acks (for 1962119767 bytes) 14399 duplicate acks 0 acks for unsent data 15281399 packets (2057886406 bytes) received in-sequence 4425 completely duplicate packets (4551322 bytes) 0 old duplicate packets 65 packets with some dup. data (14957 bytes duped) 38818 out-of-order packets (59514384 bytes) 0 packets (0 bytes) of data after window 0 window probes 120161 window update packets 3 packets received after close 4 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 160 connection requests 49 connection accepts 0 bad connection attempts 0 listen queue overflows 63 connections established (including accepts) 383 connections closed (including 5 drops) 10 connections updated cached RTT on close 10 connections updated cached RTT variance on close 2 connections updated cached ssthresh on close 146 embryonic connections dropped 846952 segments updated rtt (of 521492 attempts) 36 retransmit timeouts 2 connections dropped by rexmit timeout 0 persist timeouts 0 connections dropped by persist timeout 438 keepalive timeouts 438 keepalive probes sent 0 connections dropped by keepalive 26449 correct ACK header predictions 15280149 correct data packet header predictions 49 syncache entries added 0 retransmitted 0 dupsyn 0 dropped 49 completed 0 bucket overflow 0 cache overflow 0 reset 0 stale 0 aborted 0 badack 0 unreach 0 zone failures 0 cookies sent 0 cookies received ip: 16393480 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 1 fragment received 0 fragments dropped (dup or out of space) 1 fragment dropped after timeout 0 packets reassembled ok 16393029 packets for this host 450 packets for unknown/unsupported protocol 0 packets forwarded (0 packets fast forwarded) 0 packets not forwardable 0 packets received for unknown multicast group 0 redirects sent 12201936 packets sent from this host 115 packets sent with fabricated ip header 1367 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 tunneling packets that can't find gif 0 datagrams with bad address in header =3D=3D=3D=3D after =3D=3D=3D=3D tcp: 13331442 packets sent 3920701 data packets (2694929130 bytes) 10203 data packets (14829100 bytes) retransmitted 0 resends initiated by MTU discovery 6446155 ack-only packets (207 delayed) 0 URG only packets 0 window probe packets 2955524 window update packets 226 control packets 17131926 packets received 1526734 acks (for 2694921244 bytes) 14404 duplicate acks 0 acks for unsent data 15281806 packets (2057905910 bytes) received in-sequence 4425 completely duplicate packets (4551322 bytes) 0 old duplicate packets 65 packets with some dup. data (14957 bytes duped) 38818 out-of-order packets (59514384 bytes) 0 packets (0 bytes) of data after window 0 window probes 265251 window update packets 3 packets received after close 4 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 165 connection requests 49 connection accepts 0 bad connection attempts 0 listen queue overflows 68 connections established (including accepts) 388 connections closed (including 5 drops) 15 connections updated cached RTT on close 15 connections updated cached RTT variance on close 2 connections updated cached ssthresh on close 146 embryonic connections dropped 1526646 segments updated rtt (of 793534 attempts) 36 retransmit timeouts 2 connections dropped by rexmit timeout 0 persist timeouts 0 connections dropped by persist timeout 438 keepalive timeouts 438 keepalive probes sent 0 connections dropped by keepalive 79056 correct ACK header predictions 15280431 correct data packet header predictions 49 syncache entries added 0 retransmitted 0 dupsyn 0 dropped 49 completed 0 bucket overflow 0 cache overflow 0 reset 0 stale 0 aborted 0 badack 0 unreach 0 zone failures 0 cookies sent 0 cookies received ip: 17218567 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 1 fragment received 0 fragments dropped (dup or out of space) 1 fragment dropped after timeout 0 packets reassembled ok 17218116 packets for this host 450 packets for unknown/unsupported protocol 0 packets forwarded (0 packets fast forwarded) 0 packets not forwardable 0 packets received for unknown multicast group 0 redirects sent 13430039 packets sent from this host 115 packets sent with fabricated ip header 1367 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 tunneling packets that can't find gif 0 datagrams with bad address in header