Date: Tue, 19 May 2015 22:04:10 +0000 From: "Pokala, Ravi" <rpokala@panasas.com> To: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>, "jfv@freebsd.org" <jfv@freebsd.org>, "erj@freebsd.org" <erj@freebsd.org> Cc: "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>, "Lewis, Fred" <flewis@panasas.com>, "Sundararajan, Lakshmi" <lakshmi.n@msystechnologies.com> Subject: Performance issues with Intel Fortville (XL710/ixl(4)) Message-ID: <D1810169.136A5C%rpokala@panasas.com>
next in thread | raw e-mail | index | archive | help
Hi folks,
At Panasas, we are working with the Intel XL710 40G NIC (aka Fortville),
and we're seeing some performance issues w/ 11-CURRENT (r282653).
Motherboard: Intel S2600KP (aka Kennedy Pass)
CPU: E5-2660 v3 @ 2.6GHz (aka Haswell Xeon)
(1 socket x 10 physical cores x 2 SMT threads) = 20 logical cores
NIC: Intel XL710, 2x40Gbps QSFP, configured in 4x10Gbps mode
RAM: 4x 16GB DDR4 DIMMs
What we've seen so far:
- TX performance is pretty consistently lower than RX performance. All
numbers below are for unidrectional tests using `iperf':
10Gbps links threads/link TX Gbps RX Gbps TX/RX
1 1 9.02 9.85 91.57%
1 8 8.49 9.91 85.67%
1 16 7.00 9.91 70.63%
1 32 6.68 9.92 67.40%
- With multiple active links, both TX and RX performance suffer greatly;
the aggregate bandwidth tops out at about a third of the theoretical
40Gbps implied by 4x 10Gbps.
10Gbps links threads/link TX Gbps RX Gbps % of 40Gbps
4 1 13.39 13.38 33.4%
- Multi-link bidirectional throughput is absolutely terrible; the
aggregate is less than a tenth of the theoretical 40Gbps.
10Gbps links threads/link TX Gbps RX Gbps % of 40Gbps
4 1 3.83 2.96 9.6% / 7.4%
- Occasional interrupt storm messages are seen from the IRQs associated
with the NICs. Since that can impact performance, those runs were not
included in the data listed above.
Our questions:
- How stable is ixl(4) in -CURRENT? By that, we mean both how quickly is
the driver changing, and does the driver cause any system instability?
- What type of performance have others been getting w/ Fortville? In
40Gbps mode? In 4x10Gbps mode?
- Does anyone have any tuning parameters they can recommend for this
card?
- We did our testing w/ 11-CURRENT, but we will initially ship Fortville
running on 10.1-RELEASE or 10.2-RELEASE. The presence of RSS - even though
it is disabled by default - makes the driver back-port non-trivial. Is
there an estimate on when the 11-CURRENT version of the driver (1.4.1)
will get MFCed to 10-STABLE?
My colleagues Lakshmi and Fred (CCed) are working on this; please make
sure to include them if you have any comments.
Thanks,
Ravi
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?D1810169.136A5C%rpokala>