From owner-freebsd-stable@FreeBSD.ORG Sat May 17 12:40:09 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 592AE4E2 for ; Sat, 17 May 2014 12:40:09 +0000 (UTC) Received: from mail-lb0-x243.google.com (mail-lb0-x243.google.com [IPv6:2a00:1450:4010:c04::243]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DC0DA2D67 for ; Sat, 17 May 2014 12:40:08 +0000 (UTC) Received: by mail-lb0-f195.google.com with SMTP id l4so950438lbv.6 for ; Sat, 17 May 2014 05:40:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=Dyek6aeIM1ZBLBc566RMutbjPWrKintsOKiuffHF628=; b=0SItmZB970HEldvsPvhmymJHLVcjfgqzVcncVpiW2CAKvcwsPbcc2pefBBSbfjIuR6 6v4r8t9eAYxUQhNY8gq96UVT5emnGZr9e4N8dK5IuDDIbtllJ7W8jXfaH4VUGDhWNNI6 bcfcdd0M2mzh1dDYqUpaZm6cQa3lzz6ys7s6mCBJOlqqJljs3YdDxNaW4vg0ajhOHZxl xf6aZwq3E5ZBSgYfaj7llSfYVGneNf8WI/Tpy49GeOaiOFBUtdVYxKo1TL51fFHgYpqz CTAecuN9PyY3EKcHqErfo3XrSSOiacWiCO8WYnY1woRlGxE/NJCAxgo3pcVYU2JGOQhw xkdg== MIME-Version: 1.0 X-Received: by 10.152.28.225 with SMTP id e1mr7986101lah.1.1400330406524; Sat, 17 May 2014 05:40:06 -0700 (PDT) Received: by 10.112.161.230 with HTTP; Sat, 17 May 2014 05:40:06 -0700 (PDT) Date: Sat, 17 May 2014 14:40:06 +0200 Message-ID: Subject: [9.3 PRE] Intel i210AT ethernet adapter MSI-X problems (igb driver) From: Damian Danielecki To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 May 2014 12:40:09 -0000 I am receiving as many as 30.000-40.000 interrupts per second under no big nfs traffic. Avg traffic is 20Mb/s, avg interrupts number on igb device is 13.000/s. IMHO. I could compare this to similar igb adapters on other FreeBSD 9 servers and in this case should be less interrupts (lightweight traffic conditions). Additionally all of these interrupts occupies only one IRQ. The interested thing is CPU usage caused by %interrupts is very small (under 1%) so this is reason I am still able to use this ethernet card. # uname -a FreeBSD nfsd.xxx.pl 9.3-PRERELEASE FreeBSD 9.3-PRERELEASE #5: Fri May 16 15:41:36 CEST 2014 root@nfsd.xxx.pl:/usr/obj/usr/src/sys/FREEBSD9 amd64 This is custom minimalist kernel. I see in the sources of igb driver that i210 is generally supported: /usr/src/sys/dev/e1000 # grep 'I210' * |wc -l 64 I guess mine adapter (onboard quad gigabit I210T) is not correctly handled in the sources of the driver but system recognizes it as igb. # dmidecode (...) Manufacturer: Supermicro Product Name: X10SLM+-LN4F (...) # dmesg |grep igb0 igb0: port 0xc000-0xc01f mem 0xf7400000-0xf747ffff,0xf7480000-0xf7483fff irq 18 at device 0.0 on pci4 igb0: Using MSIX interrupts with 5 vectors igb0: Ethernet address: 0c:c4:7a:01:e3:50 igb0: Bound queue 0 to cpu 0 igb0: Bound queue 1 to cpu 1 igb0: Bound queue 2 to cpu 2 igb0: Bound queue 3 to cpu 3 igb0: link state changed to UP No device description: #pciconf -vl igb0@pci0:4:0:0: class=0x020000 card=0x153315d9 chip=0x15338086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' class = network subclass = ethernet Number of interrupts taken by device since system startup: # vmstat -i irq269: igb0:que 0 501886805 13364 irq270: igb0:que 1 40477 1 irq271: igb0:que 2 40417 1 irq272: igb0:que 3 7526720 200 irq273: igb0:link 12 0 Sample current number of interrupts with not big NFS traffic: # systat -vm1 Interrupts 34352 total (!!!) (...) 29937 igb0:que 0 1 igb0:que 1 1 igb0:que 2 1 igb0:que 3 1 igb0:link As you can see above there are abnormal high numbers of interrupts and all of them sits on que0. sysctls untouched by me: hw.igb.rx_process_limit: 100 hw.igb.num_queues: 4 hw.igb.header_split: 0 hw.igb.buf_ring_size: 4096 hw.igb.max_interrupt_rate: 8000 hw.igb.enable_msix: 1 hw.igb.enable_aim: 1 hw.igb.txd: 1024 hw.igb.rxd: 1024 For example this is properly working IRQ balancing on I350 Gigabit ethernet adaptor with FreeBSD 9.2-RELEASE: otherserver# vmstat -i irq264: igb0:que 0 14586306636 962 irq265: igb0:que 1 12313136472 812 irq266: igb0:que 2 12400518935 818 irq267: igb0:que 3 12230694398 807 irq268: igb0:que 4 12624900681 833 irq269: igb0:que 5 12311037080 812 irq270: igb0:que 6 31682657476 2090 irq271: igb0:que 7 12203814868 805 irq272: igb0:link 77 0 Regards, DD