Date: Thu, 05 Jul 2012 17:40:37 +0400 From: "Alexander V. Chernikov" <melifaro@FreeBSD.org> To: Luigi Rizzo <rizzo@iet.unipi.it> Cc: Doug Barton <dougb@freebsd.org>, net@freebsd.org Subject: Re: FreeBSD 10G forwarding performance @Intel Message-ID: <4FF59955.5090406@FreeBSD.org> In-Reply-To: <20120704154856.GC3680@onelab2.iet.unipi.it> References: <4FF319A2.6070905@FreeBSD.org> <4FF361CA.4000506@FreeBSD.org> <20120703214419.GC92445@onelab2.iet.unipi.it> <4FF36438.2030902@FreeBSD.org> <4FF3E2C4.7050701@FreeBSD.org> <4FF3FB14.8020006@FreeBSD.org> <4FF402D1.4000505@FreeBSD.org> <20120704091241.GA99164@onelab2.iet.unipi.it> <4FF412B9.3000406@FreeBSD.org> <20120704154856.GC3680@onelab2.iet.unipi.it>
next in thread | previous in thread | raw e-mail | index | archive | help
On 04.07.2012 19:48, Luigi Rizzo wrote:
> On Wed, Jul 04, 2012 at 01:54:01PM +0400, Alexander V. Chernikov wrote:
>> On 04.07.2012 13:12, Luigi Rizzo wrote:
>>> Alex,
>>> i am sure you are aware that in FreeBSD we have netmap too
>> Yes, I'm aware of that :)
>>
>>> which is probably a lot more usable than packetshader
>>> (hw independent, included in the OS, also works on linux...)
>> I'm actually not talking about usability and comparison here :). Thay
>> have nice idea and nice performance graphs. And packetshader is actually
>> _platform_ with fast packet delivery being one (and the only open) part
>> of platform.
>
> i am not sure if i should read the above as a feature or a limitation :)
I'm not trying to compare their i/o code with netmap implementation :)
>
>>
>> Their graphs shows 40MPPS (27G/64byte) CPU-only IPv4 packet forwarding
>> on "two four-core Intel Nehalem CPUs (2.66GHz)" which illustrates
>> software routing possibilities quite clearly.
>
> i suggest to be cautious about graphs in papers (including mine) and
> rely on numbers you can reproduce yourself.
Yup. Of course. However, even it if we divide their number by 4, there
is still a huge gap.
> As your nice experiments showed (i especially liked when you moved
> from one /24 to four /28 routes), at these speeds a factor
> of 2 or more in throughput can easily arise from tiny changes
> in configurations, bus, memory and CPU speeds, and so on.
Traffic stats with most possible counters eliminated:
(there is a possibility in ixgbe code to update rx/tx packets once per
rx_process_limit (which is 100 by default)):
input (ix0) output
packets errs idrops bytes packets errs bytes colls
2.8M 0 0 186M 2.8M 0 186M 0
2.8M 0 0 187M 2.8M 0 186M 0
And it seems that netstat uses 1024 as divisor (no HN_DIVISOR_1000
passed in if.c to show_stat), so real frame count from Ixia side is much
closer to 3MPPS (~ 2.961600 ).
This is wrong from my point of view and we should change it, at least
for packets count.
Here is the patch itself:
http://static.ipfw.ru/files/fbsd10g/no_ifcounters.diff
IPFW contention:
Same setup as shown upper, same traffic level
17:48 [0] test15# ipfw show
00100 0 0 allow ip from any to any
65535 0 0 deny ip from any to any
net.inet.ip.fw.enable: 0 -> 1
input (ix0) output
packets errs idrops bytes packets errs bytes colls
2.1M 734k 0 187M 2.1M 0 139M 0
2.1M 736k 0 187M 2.1M 0 139M 0
2.1M 737k 0 187M 2.1M 0 89M 0
2.1M 735k 0 187M 2.1M 0 189M 0
net.inet.ip.fw.update_counters: 1 -> 0
2.3M 636k 0 187M 2.3M 0 148M 0
2.5M 343k 0 187M 2.5M 0 164M 0
2.5M 351k 0 187M 2.5M 0 164M 0
2.5M 345k 0 187M 2.5M 0 164M 0
Patch here: http://static.ipfw.ru/files/fbsd10g/no_ipfw_counters.diff
It seems that ipfw counters are suffering from this problem, too.
Unfortunately, there is no DPCPU allocator in our kernel.
I'm planning to make a very simple per-cpu counters patch:
(
allocate 65k*(u64_bytes+u64_packets) memory for each CPU per vnet
instance init and make ipfw use it as counter backend.
There is a problem with several rules residing in single entry. This can
(probably) be worked-around by using fast counters for the first such
rule (or not using fast counters for such rules at all)
)
What do you think about this?
>
> cheers
> luigi
>
--
WBR, Alexander
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4FF59955.5090406>