From owner-freebsd-net@FreeBSD.ORG Wed Feb 9 05:49:15 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5E3921065694 for ; Wed, 9 Feb 2011 05:49:15 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from fallback5.mail.ru (fallback5.mail.ru [94.100.176.59]) by mx1.freebsd.org (Postfix) with ESMTP id D50628FC1B for ; Wed, 9 Feb 2011 05:49:14 +0000 (UTC) Received: from smtp2.mail.ru (smtp2.mail.ru [94.100.176.130]) by fallback5.mail.ru (mPOP.Fallback_MX) with ESMTP id 60E1A262554B for ; Wed, 9 Feb 2011 08:30:03 +0300 (MSK) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail; h=Content-Transfer-Encoding:Content-Type:Subject:To:MIME-Version:Reply-To:From:Date:Message-ID; bh=6wiyVsDKdVkanID5HPURTd2EW5w5kNkVB8Sde7Qyo58=; b=ll5w7koifyUn/mHRIrqM33rivpMebVbEeBdcFv/06yEtxg9ZV70GoOuSzDSNTP60K82x/Y/bg1MTwNPvDpqktH9LT7b6q8CfaN8pqzCy2YuN2ZaAYTT5V8NaLwHaAfx6; Received: from [217.25.27.27] (port=64028 helo=[217.25.27.27]) by smtp2.mail.ru with asmtp id 1Pn2cr-0002nn-00 for freebsd-net@freebsd.org; Wed, 09 Feb 2011 08:30:01 +0300 Message-ID: <4D522657.10500@mail.ru> Date: Wed, 09 Feb 2011 09:29:59 +0400 From: rihad User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7 MIME-Version: 1.0 To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mras: Ok X-MR-Warn: 1 Subject: Slow Intel 10GbE CX4 adapter behaviour X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: rihad List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 05:49:15 -0000 Hi, we're a medium sized ISP that need to pass all incoming user traffic through a Intel Server Systems FreeBSD PC and its dummynet pipes. Up until yesterday it had two 1 gb em cards, one for input, one for output. As we were approaching the bandwidth limitation we switched the cards for a two-port Intel 10GbE CX4 PCI-E adapter. With the then used FreeBSD 7.2 and the built-in FreeBSD ixgbe driver 1.7.3 (IIRC) it was very slow, and at only about 300-400 mbps load (~30-50 IP kpps) the internet access was very slow. Also, there were many "IP fragmentation failed" errors (1-30 kpps in "systat -ip"). So I decided to source-upgrade the world to 8.3-RC3 (ixgbe 2.3.8). Late in the night yesterday I didn't have enough opportunity to test the newer FreeBSD under load, but from the time we did and I know, the same slowness started happening at about 300-400 mbps load. There are no more fragmentation failed errors. No evident drops as per "netstat -s | fgrep drop". Only the speed is slooow. Even the ssh console lags a bit. Both ix0 and ix1 are configured at their default settings. Then I read something about the number of ixgbe device descriptors (hw.ixgbe.txd & hw.ixgbe.rxd) being set low at 256 by default, with up to 4096 permittable. But after some grepping on the source tree I saw that contrary to what the old docs say they are both set to an optimal value: /sys/dev/ixgbe/ixgbe.c: /* ** Number of TX descriptors per ring, ** setting higher than RX as this seems ** the better performing choice. */ static int ixgbe_txd = PERFORM_TXD; TUNABLE_INT("hw.ixgbe.txd", &ixgbe_txd); /* Number of RX descriptors per ring */ static int ixgbe_rxd = PERFORM_RXD; TUNABLE_INT("hw.ixgbe.rxd", &ixgbe_rxd) /sys/dev/ixgbe/ixgbe.h: /* * TxDescriptors Valid Range: 64-4096 Default Value: 256 This value is the * number of transmit descriptors allocated by the driver. Increasing this * value allows the driver to queue more transmits. Each descriptor is 16 * bytes. Performance tests have show the 2K value to be optimal for top * performance. */ #define DEFAULT_TXD 1024 #define PERFORM_TXD 2048 #define MAX_TXD 4096 #define MIN_TXD 64 So, here's my kernel config for your viewing pleasure: include GENERIC ident SHAPER nomakeoptions DEBUG nooptions COMPAT_FREEBSD4 # Compatible with FreeBSD4 nooptions COMPAT_FREEBSD5 # Compatible with FreeBSD5 nooptions COMPAT_FREEBSD6 # Compatible with FreeBSD6 options COMPAT_FREEBSD7 # Compatible with FreeBSD7 nooptions COMPAT_FREEBSD32 # Compatible with i386 binaries nooptions INET6 # IPv6 communications protocols options ZERO_COPY_SOCKETS # XXX 20091227: em(4) wants DEVICE_POLLING off for its fast-interrupts to work #options DEVICE_POLLING options IPFIREWALL #firewall options IPFIREWALL_DEFAULT_TO_ACCEPT #allow everything by default Here's /etc/sysctl.conf: net.inet.ip.fw.verbose=0 kern.ipc.shmall=65536 kern.ipc.shmmax=268435456 kern.ipc.semmap=1024 kern.ipc.nmbclusters=111111 net.inet.ip.fastforwarding=1 net.inet.ip.dummynet.io_fast=1 #XXX no longer used in 8.3?? net.isr.direct=0 net.inet.ip.intr_queue_maxlen=5000 hw.intr_storm_threshold=9000 #dev.em.0.rx_processing_limit=-1 # device not used any more Any tips? I'll be happy to try and add some more info upon request. Thanks.