From owner-freebsd-net@FreeBSD.ORG Mon Jan 22 03:23:30 2007 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6F11316A400 for ; Mon, 22 Jan 2007 03:23:30 +0000 (UTC) (envelope-from bde@zeta.org.au) Received: from mailout2.pacific.net.au (mailout2-3.pacific.net.au [61.8.2.226]) by mx1.freebsd.org (Postfix) with ESMTP id 3727E13C442 for ; Mon, 22 Jan 2007 03:23:30 +0000 (UTC) (envelope-from bde@zeta.org.au) Received: from mailproxy2.pacific.net.au (mailproxy2.pacific.net.au [61.8.2.163]) by mailout2.pacific.net.au (Postfix) with ESMTP id 8E5C46E2E9; Mon, 22 Jan 2007 14:23:26 +1100 (EST) Received: from besplex.bde.org (katana.zip.com.au [61.8.7.246]) by mailproxy2.pacific.net.au (Postfix) with ESMTP id D135227436; Mon, 22 Jan 2007 14:23:27 +1100 (EST) Date: Mon, 22 Jan 2007 14:23:26 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Max Laier In-Reply-To: <200701212033.34914.max@love2party.net> Message-ID: <20070122140144.R7272@besplex.bde.org> References: <20070121155510.C23922@delplex.bde.org> <200701210809.27770.max@love2party.net> <20070121215054.C24780@delplex.bde.org> <200701212033.34914.max@love2party.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@freebsd.org Subject: Re: slow writes on nfs with bge devices X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jan 2007 03:23:30 -0000 On Sun, 21 Jan 2007, Max Laier wrote: > On Sunday 21 January 2007 13:25, Bruce Evans wrote: >> On Sun, 21 Jan 2007, Max Laier wrote: >>> On Sunday 21 January 2007 07:25, Bruce Evans wrote: >>>> nfs writes much less well with bge NICs than with other NICs (sk, >>>> fxp, >>> >>> Do you use hardware checksumming on the bge? There is an XXX in >>> bge_start_locked() that looks a bit suspicious to me. >> >> I use the default for that. Wouldn't checksum problems show up as >> errors somwhere? > > Did you look at the code in question? It is concerned with fragmented > packet chains (which NFS over UDP usually generated) and only commits to > sending them, if there are enough descriptors available at once. This > can easily explain burstyness. > > Can you just try to disable the delayed checksums via "ifconfig -txcsum"? > Should be an easy enough test. I haven't looked closely at the bge checksum code, but tried turning off checksums (various combinations) yesterday. THis made no difference. >> tcpdump shows a lot of intervals between nfs write requests of almost >> exactly 10 mS (even with HZ = 1000, but more obvious with HZ = 100) >> for the broken case, so the problem is apparently related to timeouts, > > See above, I really think that there is something about that if_start loop > that might be causing this. I have some local changes in this area (much larger ifq length and watermark stuff to reduce tx interrupts). I think they make no difference, but will have to test an unchanged driver more carefully. I assume that the if_start loop always calls the interface if_start if there is something in the ifq and the interface is not marked as active. Bruce