From owner-freebsd-stable@FreeBSD.ORG Thu Oct 14 23:24:18 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 05468106566B for ; Thu, 14 Oct 2010 23:24:18 +0000 (UTC) (envelope-from TERRY@tmk.com) Received: from server.tmk.com (server.tmk.com [204.141.35.63]) by mx1.freebsd.org (Postfix) with ESMTP id C57B48FC1B for ; Thu, 14 Oct 2010 23:24:17 +0000 (UTC) Received: from tmk.com by tmk.com (PMDF V6.4 #37010) id <01NT1XCU8FOG008KN2@tmk.com> for freebsd-stable@freebsd.org; Thu, 14 Oct 2010 18:53:40 -0400 (EDT) Date: Thu, 14 Oct 2010 18:26:11 -0400 (EDT) From: Terry Kennedy To: freebsd-stable@freebsd.org Message-id: <01NT1YE1I98Q008KN2@tmk.com> MIME-version: 1.0 Content-type: TEXT/PLAIN; CHARSET=us-ascii Subject: Bogus "igb1: Could not setup receive structures" in 8-STABLE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Oct 2010 23:24:18 -0000 I've run across a strange problem with the igb driver in 8-STABLE - when I try to do anything with the second igb interface, I get one or more "igb1: Could not setup receive structures" error messages. This can be reproduced as simply as booting in single-user mode with an empty /boot/loader.conf and doing: ifconfig igb0 up ifconfig igb1 up I've tried to track this down, and as far as I can see, this is from some change introduced between 8.1-RELEASE (igb 1.9.5) and the current 8-STABLE (igb 2.0.1). When I try it when booting from the 8.1-RELEASE amd64 DVD, I can bring up both interfaces. When I try it with an 8-STABLE kernel, I get the error and igb1 is missing the "RUNNING" flag: igb0: flags=8843 metric 0 mtu 9000 options=1bb ether 00:25:90:xx:xx:bc inet6 fe80::225:90ff:fe02:xxbc%igb0 prefixlen 64 scopeid 0x1 nd6 options=3 media: Ethernet 1000baseT status: active igb1: flags=8803 metric 0 mtu 9000 options=1bb ether 00:25:90:xx:xx:bd inet6 fe80::225:90ff:fe02:xxbd%igb1 prefixlen 64 tentative scopeid 0x2 nd6 options=3 media: Ethernet 1000baseT status: active I see 3 mbuf_jumbo_page allocation failures from: (1:20) rz1m:/sys/dev/e1000# vmstat -z | grep -v 0\$ ITEM SIZE LIMIT USED FREE REQUESTS FAILURES 64 Bucket: 536, 0, 263, 3, 263, 106 128 Bucket: 1048, 0, 523, 2, 525, 139 mbuf_jumbo_page: 4096, 12800, 12307, 493, 30343, 3 which correspond to the 3 "igb1: Could not setup receive structures" mess- ages. If I try another "ifconfig igb1 up", I get another console message and the counter goes to 4. If I bump the kern.ipc.nmbjumbop sysctl to a larger value, like 15000, I get the same error message when trying to work with the igb1 device, so I don't think it is a "real" error but indicates a problem in the driver. This is on a Supermicro X8DTH-iF, BIOS 2.0a (latest) with a dual on-board 82576. The dev.igb sysctl's for the two ports (excluding 0 values) are at- tached. Note that most of the igb1 values are zero: (0:34) rz1m:/sys/dev/e1000# sysctl -a | grep igb.1 | grep -v ": 0" dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection version - 2.0.1 dev.igb.1.%driver: igb dev.igb.1.%location: slot=0 function=1 dev.igb.1.%pnpinfo: vendor=0x8086 device=0x10c9 subvendor=0x15d9 subdevice=0x0400 class=0x020000 dev.igb.1.%parent: pci1 dev.igb.1.nvm: -1 dev.igb.1.flow_control: 3 dev.igb.1.enable_aim: 1 dev.igb.1.rx_processing_limit: 100 dev.igb.1.link_irq: 1 dev.igb.1.device_control: 13632065 dev.igb.1.extended_int_mask: 2147483648 dev.igb.1.fc_high_water: 47488 dev.igb.1.fc_low_water: 47472 (0:35) rz1m:/sys/dev/e1000# sysctl -a | grep igb.0 | grep -v ": 0" dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 2.0.1 dev.igb.0.%driver: igb dev.igb.0.%location: slot=0 function=0 dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10c9 subvendor=0x15d9 subdevice=0x0400 class=0x020000 dev.igb.0.%parent: pci1 dev.igb.0.nvm: -1 dev.igb.0.flow_control: 3 dev.igb.0.enable_aim: 1 dev.igb.0.rx_processing_limit: 100 dev.igb.0.link_irq: 4 dev.igb.0.device_control: 1087373889 dev.igb.0.rx_control: 67338274 dev.igb.0.interrupt_mask: 4 dev.igb.0.extended_int_mask: 2147484671 dev.igb.0.fc_high_water: 47488 dev.igb.0.fc_low_water: 47472 dev.igb.0.queue0.txd_head: 823 dev.igb.0.queue0.txd_tail: 823 dev.igb.0.queue0.tx_packets: 1402 dev.igb.0.queue0.rxd_head: 319 dev.igb.0.queue0.rxd_tail: 318 dev.igb.0.queue0.rx_packets: 319 dev.igb.0.queue0.rx_bytes: 69075 dev.igb.0.queue1.txd_head: 2 dev.igb.0.queue1.txd_tail: 2 dev.igb.0.queue1.tx_packets: 1 dev.igb.0.queue1.rxd_head: 52 dev.igb.0.queue1.rxd_tail: 51 dev.igb.0.queue1.rx_packets: 52 dev.igb.0.queue1.rx_bytes: 6369 dev.igb.0.queue2.txd_head: 27 dev.igb.0.queue2.txd_tail: 27 dev.igb.0.queue2.tx_packets: 13 dev.igb.0.queue2.rxd_head: 72 dev.igb.0.queue2.rxd_tail: 71 dev.igb.0.queue2.rx_packets: 72 dev.igb.0.queue2.rx_bytes: 8789 dev.igb.0.queue3.txd_head: 177 dev.igb.0.queue3.txd_tail: 177 dev.igb.0.queue3.tx_packets: 64 dev.igb.0.queue3.rxd_head: 88 dev.igb.0.queue3.rxd_tail: 87 dev.igb.0.queue3.rx_packets: 88 dev.igb.0.queue3.rx_bytes: 16454 dev.igb.0.queue4.rxd_head: 21 dev.igb.0.queue4.rxd_tail: 20 dev.igb.0.queue4.rx_packets: 21 dev.igb.0.queue4.rx_bytes: 3547 dev.igb.0.queue5.txd_head: 19 dev.igb.0.queue5.txd_tail: 19 dev.igb.0.queue5.tx_packets: 9 dev.igb.0.queue5.rxd_head: 120 dev.igb.0.queue5.rxd_tail: 119 dev.igb.0.queue5.rx_packets: 120 dev.igb.0.queue5.rx_bytes: 35518 dev.igb.0.queue6.txd_head: 21 dev.igb.0.queue6.txd_tail: 21 dev.igb.0.queue6.tx_packets: 10 dev.igb.0.queue6.rxd_head: 21 dev.igb.0.queue6.rxd_tail: 20 dev.igb.0.queue6.rx_packets: 21 dev.igb.0.queue6.rx_bytes: 2876 dev.igb.0.queue7.txd_head: 100 dev.igb.0.queue7.txd_tail: 100 dev.igb.0.queue7.tx_packets: 50 dev.igb.0.queue7.rxd_head: 74 dev.igb.0.queue7.rxd_tail: 73 dev.igb.0.queue7.rx_packets: 1098 dev.igb.0.queue7.rx_bytes: 116918 dev.igb.0.queue8.txd_head: 11 dev.igb.0.queue8.txd_tail: 11 dev.igb.0.queue8.tx_packets: 6 dev.igb.0.queue8.rxd_head: 25 dev.igb.0.queue8.rxd_tail: 24 dev.igb.0.queue8.rx_packets: 25 dev.igb.0.queue8.rx_bytes: 3698 dev.igb.0.mac_stats.total_pkts_recvd: 3138 dev.igb.0.mac_stats.good_pkts_recvd: 1815 dev.igb.0.mac_stats.bcast_pkts_recvd: 383 dev.igb.0.mac_stats.mcast_pkts_recvd: 114 dev.igb.0.mac_stats.rx_frames_64: 217 dev.igb.0.mac_stats.rx_frames_65_127: 673 dev.igb.0.mac_stats.rx_frames_128_255: 779 dev.igb.0.mac_stats.rx_frames_256_511: 61 dev.igb.0.mac_stats.rx_frames_512_1023: 46 dev.igb.0.mac_stats.rx_frames_1024_1522: 39 dev.igb.0.mac_stats.good_octets_recvd: 270378 dev.igb.0.mac_stats.good_octets_txd: 252216 dev.igb.0.mac_stats.total_pkts_txd: 1554 dev.igb.0.mac_stats.good_pkts_txd: 1554 dev.igb.0.mac_stats.bcast_pkts_txd: 36 dev.igb.0.mac_stats.mcast_pkts_txd: 65 dev.igb.0.mac_stats.tx_frames_64: 32 dev.igb.0.mac_stats.tx_frames_65_127: 219 dev.igb.0.mac_stats.tx_frames_128_255: 1222 dev.igb.0.mac_stats.tx_frames_256_511: 55 dev.igb.0.mac_stats.tx_frames_512_1023: 19 dev.igb.0.mac_stats.tx_frames_1024_1522: 7 dev.igb.0.interrupts.asserts: 3350 dev.igb.0.interrupts.rx_pkt_timer: 1815 dev.igb.0.interrupts.tx_abs_timer: 1815 dev.igb.0.interrupts.tx_queue_empty: 1554 dev.igb.0.host.rx_good_bytes: 270378 dev.igb.0.host.tx_good_bytes: 252216 Both ports are cabled to the same Cisco switch. If I swap the cables at the FreeBSD end between igb0 and igb1, the problem stays with igb1 so I don't think it is the switch. I have 3 identical systems, all of which exhibit this same issue. Un- fortunately, I don't have any other hardware with dual igb's. I can give a developer root access as well as a web-based remote console if needed to track this down. Terry Kennedy http://www.tmk.com terry@tmk.com New York, NY USA