From owner-freebsd-net@FreeBSD.ORG  Sun Jan 21 12:25:44 2007
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
X-Original-To: freebsd-net@freebsd.org
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 522B616A402
	for <freebsd-net@freebsd.org>; Sun, 21 Jan 2007 12:25:44 +0000 (UTC)
	(envelope-from bde@zeta.org.au)
Received: from mailout2.pacific.net.au (mailout2-3.pacific.net.au [61.8.2.226])
	by mx1.freebsd.org (Postfix) with ESMTP id 1C77B13C45A
	for <freebsd-net@freebsd.org>; Sun, 21 Jan 2007 12:25:44 +0000 (UTC)
	(envelope-from bde@zeta.org.au)
Received: from mailproxy1.pacific.net.au (mailproxy1.pacific.net.au
	[61.8.2.162])
	by mailout2.pacific.net.au (Postfix) with ESMTP id 826E66E2BF;
	Sun, 21 Jan 2007 23:25:40 +1100 (EST)
Received: from katana.zip.com.au (katana.zip.com.au [61.8.7.246])
	by mailproxy1.pacific.net.au (Postfix) with ESMTP id 002D58C06;
	Sun, 21 Jan 2007 23:25:41 +1100 (EST)
Date: Sun, 21 Jan 2007 23:25:40 +1100 (EST)
From: Bruce Evans <bde@zeta.org.au>
X-X-Sender: bde@delplex.bde.org
To: Max Laier <max@love2party.net>
In-Reply-To: <200701210809.27770.max@love2party.net>
Message-ID: <20070121215054.C24780@delplex.bde.org>
References: <20070121155510.C23922@delplex.bde.org>
	<200701210809.27770.max@love2party.net>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
Cc: freebsd-net@freebsd.org
Subject: Re: slow writes on nfs with bge devices
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sun, 21 Jan 2007 12:25:44 -0000

On Sun, 21 Jan 2007, Max Laier wrote:

> On Sunday 21 January 2007 07:25, Bruce Evans wrote:
>> nfs writes much less well with bge NICs than with other NICs (sk, fxp,
>
> Do you use hardware checksumming on the bge?  There is an XXX in
> bge_start_locked() that looks a bit suspicious to me.

I use the default for that.  Wouldn't checksum problems show up as errors
somwhere?

>>> [excessive quoting deleted]

tcpdump shows a lot of intervals between nfs write requests of almost
exactly 10 mS (even with HZ = 1000, but more obvious with HZ = 100)
for the broken case, so the problem is apparently related to timeouts,
or timeouts are masking a larger problem.  (The average interval between
write requests needs to be about 683 uS to avoid wasting any bandwith.
fxp <-> fxp on the same network averages 708 uS.  fxp <-> bge 5701
averages 3981.)

Bruce