Date: Wed, 09 Feb 2011 09:29:59 +0400 From: rihad <rihad@mail.ru> To: freebsd-net@freebsd.org Subject: Slow Intel 10GbE CX4 adapter behaviour Message-ID: <4D522657.10500@mail.ru>
next in thread | raw e-mail | index | archive | help
Hi, we're a medium sized ISP that need to pass all incoming user traffic
through a Intel Server Systems FreeBSD PC and its dummynet pipes. Up
until yesterday it had two 1 gb em cards, one for input, one for output.
As we were approaching the bandwidth limitation we switched the cards
for a two-port Intel 10GbE CX4 PCI-E adapter. With the then used FreeBSD
7.2 and the built-in FreeBSD ixgbe driver 1.7.3 (IIRC) it was very slow,
and at only about 300-400 mbps load (~30-50 IP kpps) the internet access
was very slow. Also, there were many "IP fragmentation failed" errors
(1-30 kpps in "systat -ip"). So I decided to source-upgrade the world to
8.3-RC3 (ixgbe 2.3.8). Late in the night yesterday I didn't have enough
opportunity to test the newer FreeBSD under load, but from the time we
did and I know, the same slowness started happening at about 300-400
mbps load. There are no more fragmentation failed errors. No evident
drops as per "netstat -s | fgrep drop". Only the speed is slooow. Even
the ssh console lags a bit. Both ix0 and ix1 are configured at their
default settings.
Then I read something about the number of ixgbe device descriptors
(hw.ixgbe.txd & hw.ixgbe.rxd) being set low at 256 by default, with up
to 4096 permittable. But after some grepping on the source tree I saw
that contrary to what the old docs say they are both set to an optimal
value:
/sys/dev/ixgbe/ixgbe.c:
/*
** Number of TX descriptors per ring,
** setting higher than RX as this seems
** the better performing choice.
*/
static int ixgbe_txd = PERFORM_TXD;
TUNABLE_INT("hw.ixgbe.txd", &ixgbe_txd);
/* Number of RX descriptors per ring */
static int ixgbe_rxd = PERFORM_RXD;
TUNABLE_INT("hw.ixgbe.rxd", &ixgbe_rxd)
/sys/dev/ixgbe/ixgbe.h:
/*
* TxDescriptors Valid Range: 64-4096 Default Value: 256 This value is the
* number of transmit descriptors allocated by the driver. Increasing this
* value allows the driver to queue more transmits. Each descriptor is 16
* bytes. Performance tests have show the 2K value to be optimal for top
* performance.
*/
#define DEFAULT_TXD 1024
#define PERFORM_TXD 2048
#define MAX_TXD 4096
#define MIN_TXD 64
So, here's my kernel config for your viewing pleasure:
include GENERIC
ident SHAPER
nomakeoptions DEBUG
nooptions COMPAT_FREEBSD4 # Compatible with FreeBSD4
nooptions COMPAT_FREEBSD5 # Compatible with FreeBSD5
nooptions COMPAT_FREEBSD6 # Compatible with FreeBSD6
options COMPAT_FREEBSD7 # Compatible with FreeBSD7
nooptions COMPAT_FREEBSD32 # Compatible with i386 binaries
nooptions INET6 # IPv6 communications protocols
options ZERO_COPY_SOCKETS
# XXX 20091227: em(4) wants DEVICE_POLLING off for its fast-interrupts
to work
#options DEVICE_POLLING
options IPFIREWALL #firewall
options IPFIREWALL_DEFAULT_TO_ACCEPT #allow everything by default
Here's /etc/sysctl.conf:
net.inet.ip.fw.verbose=0
kern.ipc.shmall=65536
kern.ipc.shmmax=268435456
kern.ipc.semmap=1024
kern.ipc.nmbclusters=111111
net.inet.ip.fastforwarding=1
net.inet.ip.dummynet.io_fast=1 #XXX no longer used in 8.3??
net.isr.direct=0
net.inet.ip.intr_queue_maxlen=5000
hw.intr_storm_threshold=9000
#dev.em.0.rx_processing_limit=-1 # device not used any more
Any tips? I'll be happy to try and add some more info upon request.
Thanks.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4D522657.10500>