From owner-freebsd-current@FreeBSD.ORG Thu Nov 18 12:29:17 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 71CBF16A4CE; Thu, 18 Nov 2004 12:29:17 +0000 (GMT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id F30E043D54; Thu, 18 Nov 2004 12:29:16 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.13.1/8.13.1) with ESMTP id iAICRi9L069152; Thu, 18 Nov 2004 07:27:44 -0500 (EST) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)iAICRiVU069149; Thu, 18 Nov 2004 12:27:44 GMT (envelope-from robert@fledge.watson.org) Date: Thu, 18 Nov 2004 12:27:44 +0000 (GMT) From: Robert Watson X-Sender: robert@fledge.watson.org To: Emanuel Strobl In-Reply-To: <200411172357.47735.Emanuel.Strobl@gmx.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-current@freebsd.org cc: freebsd-stable@freebsd.org Subject: Re: serious networking (em) performance (ggate and NFS) problem X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Nov 2004 12:29:17 -0000 On Wed, 17 Nov 2004, Emanuel Strobl wrote: > I really love 5.3 in many ways but here're some unbelievable transfer > rates, after I went out and bought a pair of Intel GigaBit Ethernet > Cards to solve my performance problem (*laugh*): I think the first thing you want to do is to try and determine whether the problem is a link layer problem, network layer problem, or application (file sharing) layer problem. Here's where I'd start looking: (1) I'd first off check that there wasn't a serious interrupt problem on the box, which is often triggered by ACPI problems. Get the box to be as idle as possible, and then use vmstat -i or stat -vmstat to see if anything is spewing interrupts. (2) Confirm that your hardware is capable of the desired rates: typically this involves looking at whether you have a decent card (most if_em cards are decent), whether it's 32-bit or 64-bit PCI, and so on. For unidirectional send on 32-bit PCI, be aware that it is not possible to achieve gigabit performance because the PCI bus isn't fast enough, for example. (3) Next, I'd use a tool like netperf (see ports collection) to establish three characteristics: round trip latency from user space to user space (UDP_RR), TCP throughput (TCP_STREAM), and large packet throughput (UDP_STREAM). With decent boxes on 5.3, you should have no trouble at all maxing out a single gig-e with if_em, assuming all is working well hardware wise and there's no software problem specific to your configuration. (4) Note that router latency (and even switch latency) can have a substantial impact on gigabit performance, even with no packet loss, in part due to stuff like ethernet flow control. You may want to put the two boxes back-to-back for testing purposes. (5) Next, I'd measure CPU consumption on the end box -- in particular, use top -S and systat -vmstat 1 to compare the idle condition of the system and the system under load. If you determine there is a link layer or IP layer problem, we can start digging into things like the error statistics in the card, negotiation issues, etc. If not, you want to move up the stack to try and characterize where it is you're hitting the performance issue. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Principal Research Scientist, McAfee Research