From owner-freebsd-hackers@FreeBSD.ORG Sat Apr 12 12:58:16 2003 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0CBE037B401; Sat, 12 Apr 2003 12:58:16 -0700 (PDT) Received: from angelica.unixdaemons.com (angelica.unixdaemons.com [209.148.64.135]) by mx1.FreeBSD.org (Postfix) with ESMTP id 39C5E43F85; Sat, 12 Apr 2003 12:58:14 -0700 (PDT) (envelope-from hiten@angelica.unixdaemons.com) Received: from angelica.unixdaemons.com (hiten@localhost.unixdaemons.com [127.0.0.1])h3CJvWtK031254; Sat, 12 Apr 2003 15:57:32 -0400 (EDT) Received: (from hiten@localhost) by angelica.unixdaemons.com (8.12.9/8.12.1/Submit) id h3CJvB5f031172; Sat, 12 Apr 2003 15:57:11 -0400 (EDT) (envelope-from hiten) Date: Sat, 12 Apr 2003 15:57:11 -0400 From: Hiten Pandya To: Mike Silbersack Message-ID: <20030412195711.GA30459@unixdaemons.com> References: <200304101311.h3ADBgjY022790@samson.dc.luth.se> <20030410114227.A472@odysseus.silby.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20030410114227.A472@odysseus.silby.com> User-Agent: Mutt/1.4.1i X-Operating-System: FreeBSD i386 X-Public-Key: http://www.pittgoth.com/~hiten/pubkey.asc X-URL: http://www.unixdaemons.com/~hiten X-PGP: http://pgp.mit.edu:11371/pks/lookup?search=Hiten+Pandya&op=index cc: freebsd-performance@freebsd.org cc: Eric Anderson cc: David Gilbert cc: freebsd-hackers@freebsd.org Subject: Re: tcp_output starving -- is due to mbuf get delay? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Apr 2003 19:58:16 -0000 [ cross post to hackers removed, thanks ] Mike Silbersack (Thu, Apr 10, 2003 at 11:44:42AM -0500) wrote: > > On Thu, 10 Apr 2003, Borje Josefsson wrote: > > > What we did in NetBSD (-current) was to increase IFQ_MAXLEN in (their) > > sys/net/if.h, apart from that it's only "traditional" TCP tuning. > > > > My hosts are connected directly to core routers in a 10Gbps nationwide > > network, so if anybody is interested in some testing I am more than > > willing to participate. If anybody produces a patch, I have a third system > > that I can use for piloting of that too. > > > > --B?rje > > This brings up something I've been wondering about, which you might want > to investigate: > > >From tcp_output: > > if (error == ENOBUFS) { > if (!callout_active(tp->tt_rexmt) && > !callout_active(tp->tt_persist)) > callout_reset(tp->tt_rexmt, tp->t_rxtcur, > tcp_timer_rexmt, tp); > tcp_quench(tp->t_inpcb, 0); > return (0); > } > > That tcp_quench knocks the window size back to one packet, if I'm not > mistaken. You might want to put a counter there and see if that's > happening frequently to you; if so, it might explain some loss of > performance. Maybe something like this: %%% Index: sys/netinet/tcp_output.c =================================================================== RCS file: /home/ncvs/src/sys/netinet/tcp_output.c,v retrieving revision 1.78 diff -u -r1.78 tcp_output.c --- sys/netinet/tcp_output.c 19 Feb 2003 22:18:05 -0000 1.78 +++ sys/netinet/tcp_output.c 12 Apr 2003 19:52:31 -0000 @@ -930,6 +930,7 @@ !callout_active(tp->tt_persist)) callout_reset(tp->tt_rexmt, tp->t_rxtcur, tcp_timer_rexmt, tp); + tcpstat.tcps_selfquench++; tcp_quench(tp->t_inpcb, 0); return (0); } Index: sys/netinet/tcp_var.h =================================================================== RCS file: /home/ncvs/src/sys/netinet/tcp_var.h,v retrieving revision 1.88 diff -u -r1.88 tcp_var.h --- sys/netinet/tcp_var.h 1 Apr 2003 21:16:46 -0000 1.88 +++ sys/netinet/tcp_var.h 12 Apr 2003 19:52:31 -0000 @@ -394,6 +394,8 @@ u_long tcps_sc_zonefail; /* zalloc() failed */ u_long tcps_sc_sendcookie; /* SYN cookie sent */ u_long tcps_sc_recvcookie; /* SYN cookie received */ + + u_long tcps_selfquench; /* self-quench count */ }; /* Index: usr.bin/netstat/inet.c =================================================================== RCS file: /home/ncvs/src/usr.bin/netstat/inet.c,v retrieving revision 1.58 diff -u -r1.58 inet.c --- usr.bin/netstat/inet.c 2 Apr 2003 20:14:44 -0000 1.58 +++ usr.bin/netstat/inet.c 12 Apr 2003 19:52:32 -0000 @@ -389,6 +389,7 @@ p(tcps_sndprobe, "\t\t%lu window probe packet%s\n"); p(tcps_sndwinup, "\t\t%lu window update packet%s\n"); p(tcps_sndctrl, "\t\t%lu control packet%s\n"); + p(tcps_selfquench, "\t\t%lu send%s resulting in self-quench\n"); p(tcps_rcvtotal, "\t%lu packet%s received\n"); p2(tcps_rcvackpack, tcps_rcvackbyte, "\t\t%lu ack%s (for %lu byte%s)\n"); p(tcps_rcvdupack, "\t\t%lu duplicate ack%s\n"); %%% Cheers. -- Hiten