Date: Tue, 28 May 2013 14:57:11 +0200 From: Axel Fischer <afischer@marvell.com> To: Igor Mozolevsky <igor@hybrid-lab.co.uk> Cc: Hackers freeBSD <freebsd-hackers@freebsd.org>, Ralf Assmann <rassmann@marvell.com>, Lino Sanfilippo <lsanfil@marvell.com>, Markus Althoff <malthoff@marvell.com> Subject: Re: Low Tx-Rx performance with 10Gb NICs Message-ID: <1369745831.14405.28.camel@EL-DT095.site> In-Reply-To: <CADWvR2hkjUMUpa-_R_RbkdJthERTeZNevAxe%2B6rZVA-sMy_qaQ@mail.gmail.com> References: <175CCF5F49938B4D99B2E3EF7F558EBE381FA6E5AA@SC-VEXCH4.marvell.com> <CADWvR2g_VfZcCrXH1TcOVtPKT8GLbSKaqcH46piFBUjoOzXWUA@mail.gmail.com> <1369406798.20748.30.camel@EL-DT095.site> <CADWvR2hkjUMUpa-_R_RbkdJthERTeZNevAxe%2B6rZVA-sMy_qaQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Igor, all,
thank you for your quick response regarding
the 10GBit NIC performance.
We noticed the following when using an Intel
NIC as a reference NIC for our performance
measurements:
- We got the expected performance on FreeBSD
9.0 (32bit) and 9.1 (64bit) with:
1) LRO enabled (SW in Kernel,not on NIC)
and
2) Processor affinity set for the receive interrupts of the NIC to one
CPU (e.g.7).
and
3) Processor affinity of the netperf process
set to CPU 0 (rx) and CPU 1 (tx).
But if we do not enable LRO or do not
set processor affinity of the rx interrupts of
the NIC we got a very bad performance of about
11 Gbit/s. Especially tx performance descreased
dramatically durning a tx/rx netperf test using
4 tx streams and 4 rx streams ...
=> So my questions are:
1) Are "processor affinity" and "LRO" an essential
requirement for appropriate duplex 10GBit
performance ?
2) Why does tx performance decrease dramatically
if we do not use LRO or proc.-affinity on the
receive side ... ?
3) Is the behaviour probably related to the Intel
platform (CPU related) (Intel(R) Core(TM) i7-3770
CPU @ 3.40GHz) ...?
Thanks you very much in advance,
Axel Fischer
Machine info:
============
hw.machine: amd64
hw.model: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
hw.ncpu: 4
hw.byteorder: 1234
hw.physmem: 17052545024
hw.usermem: 16792178688
hw.pagesize: 4096
hw.floatingpoint: 1
hw.machine_arch: amd64
hw.realmem: 17706254336
=========================================================
-------- Weitergeleitete Nachricht --------
Von: Igor Mozolevsky <igor@hybrid-lab.co.uk>
An: Axel Fischer <afischer@marvell.com>
Kopie: Igor Mozolevsky <igor@hybrid-lab.co.uk>, Lino Sanfilippo
<lsanfil@marvell.com>, Hackers freeBSD <freebsd-hackers@freebsd.org>,
Ralf Assmann <rassmann@marvell.com>, Markus Althoff
<malthoff@marvell.com>
Betreff: Low Tx-Rx performance with 10Gb NICs
Datum: Fri, 24 May 2013 08:41:25 -0700
On Friday, 24 May 2013, Axel Fischer wrote:
Additionally I noticed the following TCP errors
with netstat -s ...:
1186 data packets (1717328 bytes) retransmitted
6847875 window update packets
2319 duplicate acks
25831 out-of-order packets (37403288 bytes)
3733 discarded due to memory problems (drops)
1186 segment rexmits in SACK recovery episodes
1717328 byte rexmits in SACK recovery episodes
Looks like your data is flooding your memory buffers, have a look
through https://calomel.org/freebsd_network_tuning.html
--
Igor M.
--
Axel Fischer | R&D Software / SW Engineer | Marvell Semiconductor
Germany GmbH
Office +49 (7243) 502 370 | Fax +49 (7243) 502 982
afischer@marvell.com
M A R V E L L |www.marvell.com
This communication, together with any attachments hereto or links
contained herein, is for the sole use of the intended recipient(s) and
may contain information that is confidential or legally protected. If
you are not the intended recipient, you are hereby notified that any
review, disclosure, copying, dissemination, distribution or use of this
communication is STRICTLY PROHIBITED. If you have received this
communication in error, please notify the sender immediately by return
e-mail message and delete the original and all copies of the
communication, along with any attachments hereto or links herein, from
your system.
Marvell Semiconductor Germany GmbH, Siemensstr. 23, 76275 Ettlingen,
Amtsgericht Mannheim HRB 361620
Geschäftsführer: Dipl.-Volksw. Mathias Horak
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1369745831.14405.28.camel>