From owner-freebsd-net@FreeBSD.ORG Wed Oct 5 06:26:51 2005 Return-Path: X-Original-To: net@freebsd.org Delivered-To: freebsd-net@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3CF1616A41F for ; Wed, 5 Oct 2005 06:26:51 +0000 (GMT) (envelope-from toasty@dragondata.com) Received: from tokyo01.jp.mail.your.org (tokyo01.jp.mail.your.org [204.9.54.5]) by mx1.FreeBSD.org (Postfix) with ESMTP id C507A43D45 for ; Wed, 5 Oct 2005 06:26:50 +0000 (GMT) (envelope-from toasty@dragondata.com) Received: from mail.dragondata.com (server3-b.your.org [64.202.113.67]) by tokyo01.jp.mail.your.org (Postfix) with ESMTP id 80DC02AD5591; Wed, 5 Oct 2005 06:49:16 +0000 (UTC) Received: from [69.31.99.45] (pool045.dhcp.your.org [69.31.99.45]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mail.dragondata.com (Postfix) with ESMTP id 660AE3D183D; Wed, 5 Oct 2005 01:26:48 -0500 (CDT) In-Reply-To: <1128470191.75484.TMDA@seddon.ca> References: <4341089F.7010504@jku.at> <20051003104548.GB70355@cell.sick.ru> <4341242F.9060602@jku.at> <20051003123210.GF70355@cell.sick.ru> <43426EF3.3020404@jku.at> <9CD8C672-1EF2-42FE-A61E-83DC684C893D@dragondata.com> <43429157.90606@jku.at> <4342987D.7000200@benswebs.com> <20051004161217.GB43195@obiwan.tataz.chchile.org> <1128470191.75484.TMDA@seddon.ca> Mime-Version: 1.0 (Apple Message framework v734) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <979B163D-7078-4558-9095-DC329707A5B4@dragondata.com> Content-Transfer-Encoding: 7bit From: Kevin Day Date: Wed, 5 Oct 2005 01:27:09 -0500 To: Dave+Seddon X-Mailer: Apple Mail (2.734) Cc: net@freebsd.org Subject: Re: dummynet, em driver, device polling issues :-(( X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Oct 2005 06:26:51 -0000 On Oct 4, 2005, at 6:56 PM, Dave+Seddon wrote: > You mention your running at "near" line rate. What are you pushing > or pulling? Whats the rough spec of these machines pushing out > this much data? What setting do you have for the polling? I've > been trying to do near line rate and can't even get close with new > HP-DL380s (Single 3.4 Ghz Xeon). I think the PCI bus might be the > problem. The em Intel NICs I found to be very slow and stop after > about 3 hours. - The Intel NICs I have are dual port, although > they end up on seperate IRQs. > In one case, we had a system acting as a router. It was a Dell PowerEdge 2650, with two dual "server" adapters. each were on separate PCI busses. 3 were "lan" links, and one was a "wan" link. The lan links were receiving about 300mbps each, all going out the "wan" link at near 900mbps at peak. We were never able to get above 944mbps, but I never cared enough to figure out where the bottleneck was there. This was with PCI-X, and a pretty stripped config on the server side. Nothing fancy on polling, i think we set HZ to 10000, turned on idle_poll, and set user_frac to 10 because we had some cpu hungry tasks that were not a high priority. For anyone watching, the config we had there that we were successful with was: em0@pci2:6:0: class=0x020000 card=0x10118086 chip=0x10108086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = '82546EB Dual Port Gigabit Ethernet Controller (Copper)' class = network subclass = ethernet em1@pci2:6:1: class=0x020000 card=0x10118086 chip=0x10108086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = '82546EB Dual Port Gigabit Ethernet Controller (Copper)' class = network subclass = ethernet em2@pci1:8:0: class=0x020000 card=0x10128086 chip=0x10128086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = '82546EB Dual Port Gigabit Ethernet Controller (Fiber)' class = network subclass = ethernet em3@pci1:8:1: class=0x020000 card=0x10128086 chip=0x10128086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = '82546EB Dual Port Gigabit Ethernet Controller (Fiber)' class = network subclass = ethernet We also have some web servers that are sending 300-400mbps each at peak using thttpd or lighttpd, with the built in Dell 2850 em parts. They also are connected via PCI-X speed buses internally.