Date: Wed, 9 Feb 2011 08:43:04 +0200 From: Nikolay Denev <ndenev@gmail.com> To: rihad <rihad@mail.ru> Cc: freebsd-net@freebsd.org Subject: Re: Slow Intel 10GbE CX4 adapter behaviour Message-ID: <A6E9AE49-56FB-4217-BDA8-0FD74C901115@gmail.com> In-Reply-To: <4D522657.10500@mail.ru> References: <4D522657.10500@mail.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
On 9 Feb, 2011, at 07:29 , rihad wrote:
> Hi, we're a medium sized ISP that need to pass all incoming user =
traffic through a Intel Server Systems FreeBSD PC and its dummynet =
pipes. Up until yesterday it had two 1 gb em cards, one for input, one =
for output. As we were approaching the bandwidth limitation we switched =
the cards for a two-port Intel 10GbE CX4 PCI-E adapter. With the then =
used FreeBSD 7.2 and the built-in FreeBSD ixgbe driver 1.7.3 (IIRC) it =
was very slow, and at only about 300-400 mbps load (~30-50 IP kpps) the =
internet access was very slow. Also, there were many "IP fragmentation =
failed" errors (1-30 kpps in "systat -ip"). So I decided to =
source-upgrade the world to 8.3-RC3 (ixgbe 2.3.8). Late in the night =
yesterday I didn't have enough opportunity to test the newer FreeBSD =
under load, but from the time we did and I know, the same slowness =
started happening at about 300-400 mbps load. There are no more =
fragmentation failed errors. No evident drops as per "netstat -s | fgrep =
drop". Only the speed is slooow. Even the ssh console lags a bit. Both =
ix0 and ix1 are configured at their default settings.
>=20
> Then I read something about the number of ixgbe device descriptors =
(hw.ixgbe.txd & hw.ixgbe.rxd) being set low at 256 by default, with up =
to 4096 permittable. But after some grepping on the source tree I saw =
that contrary to what the old docs say they are both set to an optimal =
value:
>=20
> /sys/dev/ixgbe/ixgbe.c:
> /*
> ** Number of TX descriptors per ring,
> ** setting higher than RX as this seems
> ** the better performing choice.
> */
> static int ixgbe_txd =3D PERFORM_TXD;
> TUNABLE_INT("hw.ixgbe.txd", &ixgbe_txd);
>=20
> /* Number of RX descriptors per ring */
> static int ixgbe_rxd =3D PERFORM_RXD;
> TUNABLE_INT("hw.ixgbe.rxd", &ixgbe_rxd)
>=20
>=20
> /sys/dev/ixgbe/ixgbe.h:
> /*
> * TxDescriptors Valid Range: 64-4096 Default Value: 256 This value is =
the
> * number of transmit descriptors allocated by the driver. Increasing =
this
> * value allows the driver to queue more transmits. Each descriptor is =
16
> * bytes. Performance tests have show the 2K value to be optimal for =
top
> * performance.
> */
> #define DEFAULT_TXD 1024
> #define PERFORM_TXD 2048
> #define MAX_TXD 4096
> #define MIN_TXD 64
>=20
>=20
>=20
> So, here's my kernel config for your viewing pleasure:
> include GENERIC
>=20
> ident SHAPER
>=20
> nomakeoptions DEBUG
>=20
> nooptions COMPAT_FREEBSD4 # Compatible with FreeBSD4
> nooptions COMPAT_FREEBSD5 # Compatible with FreeBSD5
> nooptions COMPAT_FREEBSD6 # Compatible with FreeBSD6
> options COMPAT_FREEBSD7 # Compatible with FreeBSD7
> nooptions COMPAT_FREEBSD32 # Compatible with i386 =
binaries
>=20
> nooptions INET6 # IPv6 communications =
protocols
> options ZERO_COPY_SOCKETS
> # XXX 20091227: em(4) wants DEVICE_POLLING off for its fast-interrupts =
to work
> #options DEVICE_POLLING
> options IPFIREWALL #firewall
> options IPFIREWALL_DEFAULT_TO_ACCEPT #allow everything by =
default
>=20
>=20
> Here's /etc/sysctl.conf:
>=20
> net.inet.ip.fw.verbose=3D0
>=20
> kern.ipc.shmall=3D65536
> kern.ipc.shmmax=3D268435456
> kern.ipc.semmap=3D1024
> kern.ipc.nmbclusters=3D111111
>=20
> net.inet.ip.fastforwarding=3D1
> net.inet.ip.dummynet.io_fast=3D1 #XXX no longer used in 8.3??
> net.isr.direct=3D0
> net.inet.ip.intr_queue_maxlen=3D5000
>=20
> hw.intr_storm_threshold=3D9000
> #dev.em.0.rx_processing_limit=3D-1 # device not used any more
>=20
>=20
>=20
>=20
> Any tips? I'll be happy to try and add some more info upon request.
>=20
>=20
> Thanks.
> _______________________________________________
> freebsd-net@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
I don't know if it's the same issue, but I had severe performance issues=20=
with ixgbe cards until I disable LRO (ifconfig ix0 -lro). That was on =
7.2 too.
Regards,
Nikolay=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?A6E9AE49-56FB-4217-BDA8-0FD74C901115>