From owner-freebsd-hackers@FreeBSD.ORG  Thu Apr 10 08:48:22 2003
Return-Path: <owner-freebsd-hackers@FreeBSD.ORG>
Delivered-To: freebsd-hackers@freebsd.org
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 7B3A937B401
	for <freebsd-hackers@freebsd.org>;
	Thu, 10 Apr 2003 08:48:22 -0700 (PDT)
Received: from relay.pair.com (relay.pair.com [209.68.1.20])
	by mx1.FreeBSD.org (Postfix) with SMTP id 06D5543FBF
	for <freebsd-hackers@freebsd.org>;
	Thu, 10 Apr 2003 08:48:21 -0700 (PDT)	(envelope-from silby@silby.com)
Received: (qmail 21143 invoked from network); 10 Apr 2003 15:48:20 -0000
Received: from niwun.pair.com (HELO localhost) (209.68.2.70)
  by relay.pair.com with SMTP; 10 Apr 2003 15:48:20 -0000
X-pair-Authenticated: 209.68.2.70
Date: Thu, 10 Apr 2003 11:44:42 -0500 (CDT)
From: Mike Silbersack <silby@silby.com>
To: Borje Josefsson <bj@dc.luth.se>
In-Reply-To: <200304101311.h3ADBgjY022790@samson.dc.luth.se>
Message-ID: <20030410114227.A472@odysseus.silby.com>
References: <200304101311.h3ADBgjY022790@samson.dc.luth.se>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=iso-8859-1
Content-Transfer-Encoding: QUOTED-PRINTABLE
cc: freebsd-hackers@freebsd.org
cc: freebsd-performance@freebsd.org
cc: "Jin Guojun \[DSD\]" <j_guojun@lbl.gov>
cc: Eric Anderson <anderson@centtech.com>
cc: David Gilbert <dgilbert@velocet.ca>
Subject: Re: tcp_output starving -- is due to mbuf get delay? 
X-BeenThere: freebsd-hackers@freebsd.org
X-Mailman-Version: 2.1.1
Precedence: list
List-Id: Technical Discussions relating to FreeBSD
	<freebsd-hackers.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>,
	<mailto:freebsd-hackers-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-hackers>
List-Post: <mailto:freebsd-hackers@freebsd.org>
List-Help: <mailto:freebsd-hackers-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>,
	<mailto:freebsd-hackers-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 10 Apr 2003 15:48:22 -0000


On Thu, 10 Apr 2003, Borje Josefsson wrote:

> What we did in NetBSD (-current) was to increase IFQ_MAXLEN in (their)
> sys/net/if.h, apart from that it's only "traditional" TCP tuning.
>
> My hosts are connected directly to core routers in a 10Gbps nationwide
> network, so if anybody is interested in some testing I am more than
> willing to participate. If anybody produces a patch, I have a third syste=
m
> that I can use for piloting of that too.
>
> --B=F6rje

This brings up something I've been wondering about, which you might want
to investigate:

From=20tcp_output:

=09=09if (error =3D=3D ENOBUFS) {
=09                if (!callout_active(tp->tt_rexmt) &&
                            !callout_active(tp->tt_persist))
=09                        callout_reset(tp->tt_rexmt, tp->t_rxtcur,
                                      tcp_timer_rexmt, tp);
=09=09=09tcp_quench(tp->t_inpcb, 0);
=09=09=09return (0);
=09=09}

That tcp_quench knocks the window size back to one packet, if I'm not
mistaken.  You might want to put a counter there and see if that's
happening frequently to you; if so, it might explain some loss of
performance.

Have you tried running kernel profiling yet?  It would be interesting to
see which functions are using up the largest amount of time.

Mike "Silby" Silbersack