From owner-freebsd-net@FreeBSD.ORG Sun Apr 15 02:11:58 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EE713106564A for ; Sun, 15 Apr 2012 02:11:58 +0000 (UTC) (envelope-from juli@clockworksquid.com) Received: from mail-wi0-f178.google.com (mail-wi0-f178.google.com [209.85.212.178]) by mx1.freebsd.org (Postfix) with ESMTP id 770988FC08 for ; Sun, 15 Apr 2012 02:11:58 +0000 (UTC) Received: by wibhq7 with SMTP id hq7so3450015wib.13 for ; Sat, 14 Apr 2012 19:11:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type :content-transfer-encoding:x-gm-message-state; bh=8ZS2ZKq1nc/Awl2rnUsA7zC/+3l6PzjYEo2HSxYrxeo=; b=gLwbaQ0/4fCpwImSYUbWI6ad3QqmMyFUjHMDtpW0lVpky1crb0EZolAL8+Pf0eD9OQ E10M6gV95BBd/UOS78BK283CfUFdCbvm8neO9AqwhnhOTMFXsoWGV0N1rIJq8x7wlyW/ 2l91n/JKBlurqoCE44gNrI+kAEkhmTNaB4B5kyCO9SrfAgsjn7SsVvOezJWhb9ikHLkc +DIW3SPNpU6w/ppIP4OiuqnrMvymSysYoE8+/TX/TK0t73rG4PkADfScJ4bZsAFntLNs yRFynou7AUztlMNGV1NKvsYRowtrH0NUTkX6S2++5xRRf08dZcakRzbCkNvFi/ySafwd A/hg== Received: by 10.180.81.135 with SMTP id a7mr7788903wiy.16.1334455911607; Sat, 14 Apr 2012 19:11:51 -0700 (PDT) MIME-Version: 1.0 Sender: juli@clockworksquid.com Received: by 10.216.212.157 with HTTP; Sat, 14 Apr 2012 19:11:31 -0700 (PDT) In-Reply-To: References: <4F89C005.2020304@rawbw.com> <4F89CE37.1070706@rawbw.com> <4F89CFC1.80306@rawbw.com> <4F89D1F2.3080009@rawbw.com> From: Juli Mallett Date: Sat, 14 Apr 2012 19:11:31 -0700 X-Google-Sender-Auth: WzUTfA3Oep4yVvQtH0wyfabEYvs Message-ID: To: Jason Leschnik Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Gm-Message-State: ALoCoQmk+jErjxHb9+urrTon8Ghsep3/v9+y+m6eME/fKcdTQtqpkh9s1SbetvbfTF1MszvqChAo Cc: Yuri , freebsd-net@freebsd.org Subject: Re: Why host transmit rate on 1Gb ethernet is only ~750Mbps ? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 15 Apr 2012 02:11:59 -0000 On Sat, Apr 14, 2012 at 18:14, Jason Leschnik wrote: > What about the results on the end node, during the testing?. This is not a very quick way to isolate the problem. Some NICs and some configurations just can't do line rate at any packet size, or at any plausible packet size, and if that's the case then looking at load isn't very instructive unless interrupt load is the problem. iperf with UDP is already pretty basic and not very resource-hungry. I've pushed a gigabit with iperf and UDP on pretty trivial hardware, and had problems with high-end hardware and a low-end NIC. 2.5GHz CPU is not enough information. How is the re device attached? What kind of 2.5GHz CPU? If you have netgraph in your kernel and ng_source, you can use it pretty easily. Dump a real packet or dd some random data into a file called "packet-source". Make it the size of the frame size you want to test. Put in a real packet header and then use truncate(1) to adjust its size for a variety of tests. These examples below assume your NIC is called e0 =E2=80=94 replace that wi= th whatever your NIC is really called. In one window, run netstat -w1 -I e0. In another, run this shell script: %%% #! /bin/sh sudo ngctl mkpeer e0: source orphans output sudo nghook e0:orphans input < packet-source sudo ngctl msg e0:orphans start 1000000000 %%% When you want to stop it, just do "sudo ngctl destroy e0:orphans" (I may have that wrong.) Now, watch your packet output rate and bandwidth in netstat. Is it managing 1gbps, or hovering at 750mbps? If the latter, you're not going to get better performance without driver tweaks, and maybe not even then. If you want to put netmap (not netgraph) into your kernel and try its pkt-gen program (in tools/tools/netmap/pkt-gen.c IIRC) you may be able to see if that's better, in which case interrupt overhead may be your issue, in which case you're probably still out-of-luck unless you want to hack the driver or make your application use netmap. Now, on your other host, the one receiving this traffic, put the interface in promiscuous mode. Run netstat -w1 -I {whatever} to watch its receive rate. Is it receiving all of the packets that are getting sent? If the source is managing to send 1gbps and the target is receiving less, there's your problem. If they can both do it, then your bottleneck is somewhere between the Ethernet stack and iperf, but it's likely not CPU time unless you're trying to send very small frames, and even then 2.5GHz can probably swing it. I have a 3GHz (IIRC) NetBurst Xeon that can to 980mbps with infinitesimal frames, and it's rubbish. Builds world in about a light year* and has the disk performance of a muddy lake full of boating enthusiasts. It just doesn't take much CPU time to blithely spew traffic forth, even with iperf. Good luck! *: Not an actual unit of measurement of wall, system, user or real time.