From owner-freebsd-net@FreeBSD.ORG Wed Feb 9 07:11:19 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D57FB1065696 for ; Wed, 9 Feb 2011 07:11:19 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id 644738FC19 for ; Wed, 9 Feb 2011 07:11:19 +0000 (UTC) Received: by eyf6 with SMTP id 6so3413711eyf.13 for ; Tue, 08 Feb 2011 23:11:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:subject:mime-version:content-type:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to:x-mailer; bh=vijqIIsYgSgjCGTBDhh3EVC+1szZYA8GfcV15ilhccM=; b=Rd/AtOsizinNZa+jr8KeBXWCPw2U23kU9V3B7pH+M1vn+KV2L4eeaSU+3GlhfYuBcW jD3sxDR+09nCd0qYIEt/ekdQWMw70EsmHyDOrd3uuPoP/xoMyfJDBpvGMkdG1DxvnrHq JCVWW+23n+ib4AYbQwBR2L5l5KuxTj4jZW2jU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; b=USvaWdj0z4rCHZp/X9NmmGBiG+xlHU+0AWm8kRlEhDTL0chJwSb84AbVjpDLpDsW5n ybR9JpB3AfUdrGF7WQX3BCLwJCGn184rPI8ACqEIlTwAoZ8b5HeyMNvyAAbJWXHiSjXR lNMSqDaZaoJm3pVtV7spMy2ANAglncrydWK/E= Received: by 10.213.3.20 with SMTP id 20mr1343062ebl.5.1297233788537; Tue, 08 Feb 2011 22:43:08 -0800 (PST) Received: from [10.0.0.49] (93-152-151-19.ddns.onlinedirect.bg [93.152.151.19]) by mx.google.com with ESMTPS id u1sm223066eeh.16.2011.02.08.22.43.06 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 08 Feb 2011 22:43:07 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: text/plain; charset=us-ascii From: Nikolay Denev In-Reply-To: <4D522657.10500@mail.ru> Date: Wed, 9 Feb 2011 08:43:04 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <4D522657.10500@mail.ru> To: rihad X-Mailer: Apple Mail (2.1082) Cc: freebsd-net@freebsd.org Subject: Re: Slow Intel 10GbE CX4 adapter behaviour X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 07:11:19 -0000 On 9 Feb, 2011, at 07:29 , rihad wrote: > Hi, we're a medium sized ISP that need to pass all incoming user = traffic through a Intel Server Systems FreeBSD PC and its dummynet = pipes. Up until yesterday it had two 1 gb em cards, one for input, one = for output. As we were approaching the bandwidth limitation we switched = the cards for a two-port Intel 10GbE CX4 PCI-E adapter. With the then = used FreeBSD 7.2 and the built-in FreeBSD ixgbe driver 1.7.3 (IIRC) it = was very slow, and at only about 300-400 mbps load (~30-50 IP kpps) the = internet access was very slow. Also, there were many "IP fragmentation = failed" errors (1-30 kpps in "systat -ip"). So I decided to = source-upgrade the world to 8.3-RC3 (ixgbe 2.3.8). Late in the night = yesterday I didn't have enough opportunity to test the newer FreeBSD = under load, but from the time we did and I know, the same slowness = started happening at about 300-400 mbps load. There are no more = fragmentation failed errors. No evident drops as per "netstat -s | fgrep = drop". Only the speed is slooow. Even the ssh console lags a bit. Both = ix0 and ix1 are configured at their default settings. >=20 > Then I read something about the number of ixgbe device descriptors = (hw.ixgbe.txd & hw.ixgbe.rxd) being set low at 256 by default, with up = to 4096 permittable. But after some grepping on the source tree I saw = that contrary to what the old docs say they are both set to an optimal = value: >=20 > /sys/dev/ixgbe/ixgbe.c: > /* > ** Number of TX descriptors per ring, > ** setting higher than RX as this seems > ** the better performing choice. > */ > static int ixgbe_txd =3D PERFORM_TXD; > TUNABLE_INT("hw.ixgbe.txd", &ixgbe_txd); >=20 > /* Number of RX descriptors per ring */ > static int ixgbe_rxd =3D PERFORM_RXD; > TUNABLE_INT("hw.ixgbe.rxd", &ixgbe_rxd) >=20 >=20 > /sys/dev/ixgbe/ixgbe.h: > /* > * TxDescriptors Valid Range: 64-4096 Default Value: 256 This value is = the > * number of transmit descriptors allocated by the driver. Increasing = this > * value allows the driver to queue more transmits. Each descriptor is = 16 > * bytes. Performance tests have show the 2K value to be optimal for = top > * performance. > */ > #define DEFAULT_TXD 1024 > #define PERFORM_TXD 2048 > #define MAX_TXD 4096 > #define MIN_TXD 64 >=20 >=20 >=20 > So, here's my kernel config for your viewing pleasure: > include GENERIC >=20 > ident SHAPER >=20 > nomakeoptions DEBUG >=20 > nooptions COMPAT_FREEBSD4 # Compatible with FreeBSD4 > nooptions COMPAT_FREEBSD5 # Compatible with FreeBSD5 > nooptions COMPAT_FREEBSD6 # Compatible with FreeBSD6 > options COMPAT_FREEBSD7 # Compatible with FreeBSD7 > nooptions COMPAT_FREEBSD32 # Compatible with i386 = binaries >=20 > nooptions INET6 # IPv6 communications = protocols > options ZERO_COPY_SOCKETS > # XXX 20091227: em(4) wants DEVICE_POLLING off for its fast-interrupts = to work > #options DEVICE_POLLING > options IPFIREWALL #firewall > options IPFIREWALL_DEFAULT_TO_ACCEPT #allow everything by = default >=20 >=20 > Here's /etc/sysctl.conf: >=20 > net.inet.ip.fw.verbose=3D0 >=20 > kern.ipc.shmall=3D65536 > kern.ipc.shmmax=3D268435456 > kern.ipc.semmap=3D1024 > kern.ipc.nmbclusters=3D111111 >=20 > net.inet.ip.fastforwarding=3D1 > net.inet.ip.dummynet.io_fast=3D1 #XXX no longer used in 8.3?? > net.isr.direct=3D0 > net.inet.ip.intr_queue_maxlen=3D5000 >=20 > hw.intr_storm_threshold=3D9000 > #dev.em.0.rx_processing_limit=3D-1 # device not used any more >=20 >=20 >=20 >=20 > Any tips? I'll be happy to try and add some more info upon request. >=20 >=20 > Thanks. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" I don't know if it's the same issue, but I had severe performance issues=20= with ixgbe cards until I disable LRO (ifconfig ix0 -lro). That was on = 7.2 too. Regards, Nikolay=