Date: Sat, 14 Apr 2012 19:11:31 -0700 From: Juli Mallett <jmallett@FreeBSD.org> To: Jason Leschnik <leschnik@gmail.com> Cc: Yuri <yuri@rawbw.com>, freebsd-net@freebsd.org Subject: Re: Why host transmit rate on 1Gb ethernet is only ~750Mbps ? Message-ID: <CACVs6=_NRM-=wcWLwMjoVBx=mf=bY5BkWCVWO8OY=-jMj0SO7A@mail.gmail.com> In-Reply-To: <CAN1XHtcftT=3Z-89W9-tRVWsR7QrG-aBX5iBsc%2BNK_Z0DVWirg@mail.gmail.com> References: <4F89C005.2020304@rawbw.com> <CAN1XHtdfj%2BfTXKEngvT1BwRpMW29TEfQqy2XbSrdCvV_v_HHpw@mail.gmail.com> <4F89CE37.1070706@rawbw.com> <CAN1XHtfEK-yeXRp1fSZjDf%2BTgxc25borMOZ%2B6K1e1nZBpcr8cQ@mail.gmail.com> <4F89CFC1.80306@rawbw.com> <CAN1XHtf97wNMJJnJtCsCbDp9GU4gEaj1=RqRF3KVGCYfkyO7OQ@mail.gmail.com> <4F89D1F2.3080009@rawbw.com> <CAN1XHtcftT=3Z-89W9-tRVWsR7QrG-aBX5iBsc%2BNK_Z0DVWirg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Apr 14, 2012 at 18:14, Jason Leschnik <leschnik@gmail.com> wrote:
> What about the results on the end node, during the testing?.
This is not a very quick way to isolate the problem. Some NICs and
some configurations just can't do line rate at any packet size, or at
any plausible packet size, and if that's the case then looking at load
isn't very instructive unless interrupt load is the problem. iperf
with UDP is already pretty basic and not very resource-hungry. I've
pushed a gigabit with iperf and UDP on pretty trivial hardware, and
had problems with high-end hardware and a low-end NIC.
2.5GHz CPU is not enough information. How is the re device attached?
What kind of 2.5GHz CPU?
If you have netgraph in your kernel and ng_source, you can use it
pretty easily. Dump a real packet or dd some random data into a file
called "packet-source". Make it the size of the frame size you want
to test. Put in a real packet header and then use truncate(1) to
adjust its size for a variety of tests.
These examples below assume your NIC is called e0 =E2=80=94 replace that wi=
th
whatever your NIC is really called.
In one window, run netstat -w1 -I e0.
In another, run this shell script:
%%%
#! /bin/sh
sudo ngctl mkpeer e0: source orphans output
sudo nghook e0:orphans input < packet-source
sudo ngctl msg e0:orphans start 1000000000
%%%
When you want to stop it, just do "sudo ngctl destroy e0:orphans" (I
may have that wrong.)
Now, watch your packet output rate and bandwidth in netstat. Is it
managing 1gbps, or hovering at 750mbps? If the latter, you're not
going to get better performance without driver tweaks, and maybe not
even then. If you want to put netmap (not netgraph) into your kernel
and try its pkt-gen program (in tools/tools/netmap/pkt-gen.c IIRC) you
may be able to see if that's better, in which case interrupt overhead
may be your issue, in which case you're probably still out-of-luck
unless you want to hack the driver or make your application use
netmap.
Now, on your other host, the one receiving this traffic, put the
interface in promiscuous mode. Run netstat -w1 -I {whatever} to watch
its receive rate. Is it receiving all of the packets that are getting
sent?
If the source is managing to send 1gbps and the target is receiving
less, there's your problem. If they can both do it, then your
bottleneck is somewhere between the Ethernet stack and iperf, but it's
likely not CPU time unless you're trying to send very small frames,
and even then 2.5GHz can probably swing it. I have a 3GHz (IIRC)
NetBurst Xeon that can to 980mbps with infinitesimal frames, and it's
rubbish. Builds world in about a light year* and has the disk
performance of a muddy lake full of boating enthusiasts. It just
doesn't take much CPU time to blithely spew traffic forth, even with
iperf.
Good luck!
*: Not an actual unit of measurement of wall, system, user or real time.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CACVs6=_NRM-=wcWLwMjoVBx=mf=bY5BkWCVWO8OY=-jMj0SO7A>