From owner-freebsd-stable@FreeBSD.ORG Sun May 18 10:31:58 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 093F7368 for ; Sun, 18 May 2014 10:31:58 +0000 (UTC) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 892282E6F for ; Sun, 18 May 2014 10:31:57 +0000 (UTC) Received: from mh0.gentlemail.de (mh0.gentlemail.de [78.138.80.135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id s4IAVm9P058785; Sun, 18 May 2014 12:31:53 +0200 (CEST) (envelope-from h.schmalzbauer@omnilan.de) Received: from titan.inop.mo1.omnilan.net (titan.inop.mo1.omnilan.net [IPv6:2001:a60:f0bb:1::3:1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id 266D53B9C; Sun, 18 May 2014 12:31:48 +0200 (CEST) Message-ID: <53788C0E.6090500@omnilan.de> Date: Sun, 18 May 2014 12:31:42 +0200 From: Harald Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: Damian Danielecki Subject: Re: [9.3 PRE] Intel i210AT ethernet adapter MSI-X problems (igb driver) References: In-Reply-To: X-Enigmail-Version: 1.1.2 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig9F6D8790CD62E58C851A3041" X-Greylist: ACL 119 matched, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [78.138.80.130]); Sun, 18 May 2014 12:31:53 +0200 (CEST) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: 78.138.80.135; Sender-helo: mh0.gentlemail.de; ) Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 May 2014 10:31:58 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig9F6D8790CD62E58C851A3041 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Bez=FCglich Damian Danielecki's Nachricht vom 17.05.2014 14:40 (localtime= ): > I am receiving as many as 30.000-40.000 interrupts per second under no > big nfs traffic. > Avg traffic is 20Mb/s, avg interrupts number on igb device is 13.000/s.= > IMHO. I could compare this to similar igb adapters on other FreeBSD 9 > servers and in this case should be less interrupts (lightweight > traffic conditions). > Additionally all of these interrupts occupies only one IRQ. > The interested thing is CPU usage caused by %interrupts is very small > (under 1%) so this is reason I am still able to use this ethernet > card. >=20 > # uname -a > FreeBSD nfsd.xxx.pl 9.3-PRERELEASE FreeBSD 9.3-PRERELEASE #5: Fri May > 16 15:41:36 CEST 2014 > root@nfsd.xxx.pl:/usr/obj/usr/src/sys/FREEBSD9 amd64 > This is custom minimalist kernel. >=20 >=20 > I see in the sources of igb driver that i210 is generally supported: > /usr/src/sys/dev/e1000 # grep 'I210' * |wc -l > 64 >=20 > I guess mine adapter (onboard quad gigabit I210T) is not correctly > handled in the sources of the driver but system recognizes it as igb. >=20 > # dmidecode > (...) > Manufacturer: Supermicro > Product Name: X10SLM+-LN4F > (...) >=20 > # dmesg |grep igb0 > igb0: port > 0xc000-0xc01f mem 0xf7400000-0xf747ffff,0xf7480000-0xf7483fff irq 18 > at device 0.0 on pci4 > igb0: Using MSIX interrupts with 5 vectors > igb0: Ethernet address: 0c:c4:7a:01:e3:50 > igb0: Bound queue 0 to cpu 0 > igb0: Bound queue 1 to cpu 1 > igb0: Bound queue 2 to cpu 2 > igb0: Bound queue 3 to cpu 3 > igb0: link state changed to UP >=20 >=20 > No device description: > #pciconf -vl > igb0@pci0:4:0:0: class=3D0x020000 card=3D0x153315d9 chip=3D0x153= 38086 > rev=3D0x03 hdr=3D0x00 > vendor =3D 'Intel Corporation' > class =3D network > subclass =3D ethernet >=20 >=20 > Number of interrupts taken by device since system startup: > # vmstat -i > irq269: igb0:que 0 501886805 13364 > irq270: igb0:que 1 40477 1 > irq271: igb0:que 2 40417 1 > irq272: igb0:que 3 7526720 200 > irq273: igb0:link 12 0 >=20 > Sample current number of interrupts with not big NFS traffic: > # systat -vm1 > Interrupts > 34352 total (!!!) > (...) > 29937 igb0:que 0 > 1 igb0:que 1 > 1 igb0:que 2 > 1 igb0:que 3 > 1 igb0:link AFAIK i210 doesn't support multiple queues. I had to add 'hw.igb.num_queues=3D1' to my loader.conf when I did some tests with i210. In my case it was PCIe-passthrough, so probably different story, but in case you might want to see what results you get with limiting queues to 1. -Harry --------------enig9F6D8790CD62E58C851A3041 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAlN4jBMACgkQLDqVQ9VXb8hUmACdEL1rwcUuUbRJeG9U8SVLUf+A ph8AoJG5N4act9WoTAxWOm2twKVwo35g =Fhxk -----END PGP SIGNATURE----- --------------enig9F6D8790CD62E58C851A3041--