From owner-freebsd-net@FreeBSD.ORG Fri Sep 4 17:30:45 2009 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CC378106566B for ; Fri, 4 Sep 2009 17:30:45 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mail.digiware.nl (unknown [IPv6:2001:4cb8:1:106::2]) by mx1.freebsd.org (Postfix) with ESMTP id 780FB8FC1B for ; Fri, 4 Sep 2009 17:30:45 +0000 (UTC) Received: from localhost (localhost.digiware.nl [127.0.0.1]) by mail.digiware.nl (Postfix) with ESMTP id 7F33B153434; Fri, 4 Sep 2009 19:30:44 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.nl Received: from mail.digiware.nl ([127.0.0.1]) by localhost (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nfKi-irX-Yh2; Fri, 4 Sep 2009 19:30:42 +0200 (CEST) Received: from [212.61.27.67] (opteron [212.61.27.67]) by mail.digiware.nl (Postfix) with ESMTP id 3C940153435; Fri, 4 Sep 2009 19:30:42 +0200 (CEST) Message-ID: <4AA14EC1.6010402@digiware.nl> Date: Fri, 04 Sep 2009 19:30:41 +0200 From: Willem Jan Withagen Organization: Digiware User-Agent: Thunderbird 2.0.0.23 (Windows/20090812) MIME-Version: 1.0 To: Manish Vachharajani References: <4AA14018.3010102@digiware.nl> <5bc218350909041002x670460c8nf202a714182d1bf6@mail.gmail.com> In-Reply-To: <5bc218350909041002x670460c8nf202a714182d1bf6@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: net@freebsd.org Subject: Re: UDP output performance X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Sep 2009 17:30:46 -0000 Manish Vachharajani wrote: > Hmm, what version of FreeBSD are you using? I don't know the solution > but I wonder if it is related to a similar problem we are having with > TCP connection scaling, both under 7.2 and 8.0 over a 10 Gb link. > We've been trying to track it down, and if you see it for UDP as well > that may give some clues. Well, asfar as I remember the majority of the code path for UDP and TCP is rather diffent. And our caseis evenmore special, since we do not do controlflow and thus do not have packets coming back in over that same interface/ip-nr. There is almost no state at all. > If you do a netstat -idh what is the output? Does the recieving > interface show any Ierrs or drops? If so, you should be able to do a > sysctl dev.em..stats=1 and then see some output via > dmesg. Does this show any missed packets? No, errors at all. But I would also be very reluctant to add more logging since that IS going to impact throughput. > Oh, also, what kind of machine are you running on? FreeBSD 7.2-STABLE #0: Fri Sep 4 18:01:30 CEST 2009 CPU: Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz (2133.42-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x6f2 Stepping = 2 real memory = 1072623616 (1022 MB) avail memory = 1040248832 (992 MB) 7 * em device over PCI-E em6@pci0:7:0:0: class=0x020000 card=0x00008086 chip=0x109a8086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82573L Intel PRO/1000 PL Network Adaptor' class = network subclass = ethernet And Yes the idea is to do something similar over a 10Gb interface. --WjW > Manish > > On Fri, Sep 4, 2009 at 10:28 AM, Willem Jan Withagen wrote: >> First of: I've been googleing for about a day, but I'll take any suggestions >> for more info. >> >> What I'm trying to do is get as much 1440 byte UDP packets out of an em >> device. And when tat works, get as much out of the 7 em devices that this >> board has. :) >> >> Currently I run into trouble at 250*174 = 43500 packets/sec. >> >> How is the setup: >> em0 gets 1 stream of 174 p/s which is ~ 2Mbit >> this gets repeated to 250 streams >> currently to 2 other servers, 125 streams each. >> each on their own 1 Gbit port >> >> This works uptil 123 streams each, going high gives packet loss. >> So this is at about 500Mbit/sec on a 1Gbit port >> >> And why do I know that the packetloss is not in the network? >> Well there are no errors on >> the output interface >> the ports on the switch >> the input ports on the receivers >> the mib of the switch does not show any signs of dropped packets, or >> likes. >> >> Also I can change the order of the queing in my repeater, and then the >> packetloss moves to the host which is last the outputlist. >> I tried raising the socketbuffer " sysctl -w net.inet.udp.maxdgram=184320" >> in a few steps. But that did not bring anything. >> >> So my guess is that I'm dropping packets somewhere from the output socket to >> the wire. >> >> BTW al stats in systat -vm are close to 0%. >> >> What tunables are there to turn? >> >> And if not tuneable, what parts of the code would be target for closer >> inspection. >> >> Any help is more than welcome >> >> --WjW >> >> >> >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> >