From owner-freebsd-net@FreeBSD.ORG Fri Aug 11 10:07:48 2006 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DD35016A4DA for ; Fri, 11 Aug 2006 10:07:48 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8D5DD43D46 for ; Fri, 11 Aug 2006 10:07:48 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 113C146C1E; Fri, 11 Aug 2006 06:07:48 -0400 (EDT) Date: Fri, 11 Aug 2006 11:07:47 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Dima Roshin In-Reply-To: Message-ID: <20060811105923.S45647@fledge.watson.org> References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@freebsd.org Subject: Re: Gigabit ethernet questions? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Aug 2006 10:07:48 -0000 On Wed, 9 Aug 2006, Dima Roshin wrote: > Greeting colleagues. I've got two DL-360(pciX bus) servers, with BCM5704 > NetXtreme Dual Gigabit Adapters(bge). The Uname is 6.1-RELEASE-p3. The bge > interfaces of the both servers are connected with each other with a cat6 > patchcord. On any recent box, or even somewhat older ones, achieving gigabit speeds with decent frame sizes (~1500 or greater) should be trivial. Using two dual-Xeons in the netperf cluster (of similar configuration), without any optimization/configuration at all (not even default TCP buffer size changes), and likely with at least some debugging compiled in, I got 930mbps on the netperf TCP stream test. So if you're not getting that speed, you need to look at the configuration closely. In particular, I would... (0) Make sure expensive debugging features, such as WITNESS and INVARIANTS, are disabled. Make sure netperf is not compiled with -DHISTOGRAM, which is not the default (anymore). (1) Confirm that your cabling is all good, and probably replace the cable to be sure. Remove the switch from the loop to make sure it's not a switch problem. (2) Disable polling. One problem I've observed with polling is that you must poll at a very high rate on high speed links, or the polling rate is too small for the buffer on the ethernet card, so packets are dropped because it's drained too infrequently. With interrupt moderation on modern cards, you get significantly polling-like effects anyway, and to be honest, syncing or sourcing a gigabit on a decent box should work fine without special stack optimizations. I don't remember the numbers, but you may find that to make polling reliable, you need to further increase HZ. (3) Check for interrupt problems. Make sure that the receive interrupt rate isn't firing significantly faster than desired, that there's no interrupt shadowing to other interrupt handlers, etc. Also confirm that all PCI segments used for gigabit networking are 64-bit, not 32-bit. I believe PCI-X should be fine. (4) Use top -S and vmstat -systat 1 to characerize the system load during a test run (ideally start it running when the test starts, and capture the output 60 seconds in or so), as this is valuable debugging information that will help us decide what the source of the problem is. Where gigabit gets tricky is with small frame sizes, where the per-packet cost dominates -- i.e., 0-byte UDP payloads. At large frame sizes, raw TCP shouldn't be a problem. Robert N M Watson Computer Laboratory University of Cambridge