From owner-freebsd-net@FreeBSD.ORG Mon Sep 30 09:51:36 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 604382B9 for ; Mon, 30 Sep 2013 09:51:36 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-we0-x230.google.com (mail-we0-x230.google.com [IPv6:2a00:1450:400c:c03::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E0BAD2793 for ; Mon, 30 Sep 2013 09:51:35 +0000 (UTC) Received: by mail-we0-f176.google.com with SMTP id u56so5235746wes.21 for ; Mon, 30 Sep 2013 02:51:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=3c1GpcIYaONS7ptP6CsFDKT1BkQMxVMAS97G8ZMIQBc=; b=FZIlwAWzwL2WBiXPrrCTyLajcvOQOBsKv0qYXbB2Aw/ucGHWmcJijyY64CTKjpnkDa DuTjAjgtnbDAaociC/y9gxtnxJE7+3kzba2tgk9hV24NeCYslb1LE8vmvNhSfpMGkoKs uT7VA2GSBo/+m9XGDcQ5A5wzYAVbtCyG007/1q90RRs0ro7xTO2Q127tMZ/1Yqmyk8fB 9xHPiyCUItH4Sn79FNXtwQBFw2WQaO7ngH3ZhH4Tvzq+kkmZYdpRCyqStPwJqQO+kIlo SRkUt6+6Rh5aerJ8hJHXmDN70KGTuMIaR9mJQb9XX70Sz5JoTEf/GOPzHaqRfLX86sIl RDwA== MIME-Version: 1.0 X-Received: by 10.180.109.132 with SMTP id hs4mr13045119wib.46.1380534694400; Mon, 30 Sep 2013 02:51:34 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.216.73.133 with HTTP; Mon, 30 Sep 2013 02:51:34 -0700 (PDT) In-Reply-To: <20130930121027.Horde.taTEbLTlKvRSSUADQ3_q1_A@webmail.polynet.lviv.ua> References: <20130930121027.Horde.taTEbLTlKvRSSUADQ3_q1_A@webmail.polynet.lviv.ua> Date: Mon, 30 Sep 2013 02:51:34 -0700 X-Google-Sender-Auth: n5PYEZXwetYHk6xNGKdGCFEcvjY Message-ID: Subject: Re: Intel 82580 lagg(4) problem From: Adrian Chadd To: Andriy Kopystyansky Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Sep 2013 09:51:36 -0000 Hi, There's a lagg/lacp option to not hash based on the incoming assigned flowid on the incoming mbuf when transmitting but instead calculate its own hash for outbound transmission. It's net.link.lagg.0.use_flowid . Set it to 0 and retry. Thanks, -adrian On 30 September 2013 02:10, Andriy Kopystyansky wrote: > Greetings! > We've got a strange problem with lagg(4) interfaces built on Intel's 82580 > chipset igb (dual port 1Gb card). > We cannot send over lagg interface more than ~1Gb/s, yet receiving ~2GB > over it. > Looks like something indicating the problem can be seen here: > > anri@host:[8:13]~#ifstat -i lagg0 -i igb1 -i igb3 1 > lagg0 igb1 igb3 > KB/s in KB/s out KB/s in KB/s out KB/s in KB/s out > 9116.50 26515.16 4147.70 28871.43 5004.86 23683.31 > 8423.08 26544.62 3853.22 28980.60 4594.21 23520.97 > 8796.48 26395.28 4248.46 28344.00 4567.42 23978.17 > > Note (IN) traffic - everything's ok, about 50% on each igb interfaces and > total summary on lagg0. > But (OUT) traffic looks weird - there is more traffic on the single igb1 > than on lagg0! > Tried with default driver came with the system, also the new one > igb-2.3.10 from Intel site - no luck. > > Testing lagg interface built on other cards (em, bce) on the same machine > shows expected normal behavior for both directions. > ------------------------------**--------- > anri@host:[9:02]~#uname -a > FreeBSD host 9.1-RELEASE-p4 FreeBSD 9.1-RELEASE-p4 #5: Fri Sep 20 14:52:18 > EEST 2013 anri@host:/usr/obj/usr/src/**sys/NODEONE amd64 > ------------------------------**-------- > Interfaces: > lagg0: flags=8843 metric 0 mtu > 1500 > options=401bb MTU,VLAN_HWCSUM,TSO4,VLAN_**HWTSO> > ether 90:e2:ba:36:95:1d > inet 172.16.0.1 netmask 0xffffffc0 broadcast 172.16.0.63 > media: Ethernet autoselect > status: active > laggproto lacp lagghash l2,l3,l4 > laggport: igb3 flags=1c > laggport: igb1 flags=1c > > igb1: flags=8843 metric 0 mtu > 1500 > options=401bb MTU,VLAN_HWCSUM,TSO4,VLAN_**HWTSO> > ether 90:e2:ba:36:95:1d > media: Ethernet autoselect (1000baseT ) > status: active > > igb3: flags=8843 metric 0 mtu > 1500 > options=401bb MTU,VLAN_HWCSUM,TSO4,VLAN_**HWTSO> > ether 90:e2:ba:36:95:1d > media: Ethernet autoselect (1000baseT ) > status: active > > ------------------------------**--------- > Hardware: IBM 3550 with 2x Intel(R) Xeon(R) CPU E5430 @ 2.66GHz (amd64) > pciconf -lv: > igb1@pci0:19:0:1: class=0x020000 card=0x12b28086 chip=0x15168086 > rev=0x01 hdr=0x00 > vendor = 'Intel Corporation' > device = '82580 Gigabit Network Connection' > class = network > subclass = ethernet > > ------------------------------**--------- > /boot/loader.conf: > hw.igb.rxd=4096 > hw.igb.txd=4096 > hw.igb.max_interrupt_rate=**32000 > hw.igb.num_queues=2 > net.isr.maxthreads=8 > net.isr.dispatch=direct > net.isr.bindthreads=1 > net.link.ifqmaxlen=10240 > > /etc/sysctl.conf: (relevant part) > > dev.igb.1.rx_processing_limit=**-1 > dev.igb.3.rx_processing_limit=**-1 > net.inet.ip.intr_queue_maxlen=**512 > net.route.netisr_maxqlen=4096 > net.inet.ip.redirect=0 > net.inet.ip.fastforwarding=1 > kern.maxfiles=204800 > kern.maxfilesperproc=200000 > kern.ipc.nmbclusters=204800 > kern.ipc.maxsockets=204800 > > Any tips, advices? > > thanks in advance, with best regards, > Andriy Kopystyansky > > > ______________________________**_________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/**mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@**freebsd.org > " >