From owner-freebsd-stable@FreeBSD.ORG Thu Oct 14 23:36:04 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2C32D106566C for ; Thu, 14 Oct 2010 23:36:04 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 83C3A8FC15 for ; Thu, 14 Oct 2010 23:36:03 +0000 (UTC) Received: by wwb39 with SMTP id 39so288221wwb.31 for ; Thu, 14 Oct 2010 16:36:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type; bh=kx/9oRj53ctl4Zz76Ox1SGbQfps8S03VH2VNMed0yxE=; b=OdbZAZ9Hje3jeTfcSD032hGoowYokmdEZWnaYx+Hn+ww8J5xT1l54hCNAvWQNdDBtC 5zVZVKiTM3AZA3IeGmof6zAZLd+FETLfWggtAktWjXlfikGTSHZXZuIA1KWFtGLne1VD 4oJz+iCY0RQVxQk0V2kKTYkfDp1f7a3V1uAxo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=PRwU1006IztikZ430uGGhqabrVQb7HxBACovkRVTVlNLUkafnCmD9t2NUPnP2dslqg au1qis/fhsElYeqDRQKpP7t/CrsLAGtoWbohqsnlIQ5/QOiD2kcWRei08JPubqCrpKlH a+jFFL/0Nx6XeTPzc/JGGgHJ1MfbP3/Aqpwfw= MIME-Version: 1.0 Received: by 10.216.47.196 with SMTP id t46mr739948web.13.1287099362181; Thu, 14 Oct 2010 16:36:02 -0700 (PDT) Received: by 10.216.48.20 with HTTP; Thu, 14 Oct 2010 16:36:02 -0700 (PDT) In-Reply-To: <01NT1YE1I98Q008KN2@tmk.com> References: <01NT1YE1I98Q008KN2@tmk.com> Date: Thu, 14 Oct 2010 16:36:02 -0700 Message-ID: From: Jack Vogel To: Terry Kennedy Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-stable@freebsd.org Subject: Re: Bogus "igb1: Could not setup receive structures" in 8-STABLE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Oct 2010 23:36:04 -0000 The problem is mbuf resources, the driver is autoconfiguring the number of queues based on the number of cores, on newer systems with lots of them this is outstripping the mbuf resource pool. I have decided to hard limit the queues to 8, you can fix the number manually by searching for num_queues in if_igb.c and setting it to something other than 0 for now. I am at work on a number of issues with igb and em right now which is why there has not been an MFC yet. Questions to me, Jack On Thu, Oct 14, 2010 at 3:26 PM, Terry Kennedy wrote: > I've run across a strange problem with the igb driver in 8-STABLE - when > I try to do anything with the second igb interface, I get one or more > "igb1: > Could not setup receive structures" error messages. > > This can be reproduced as simply as booting in single-user mode with an > empty /boot/loader.conf and doing: > > ifconfig igb0 up > ifconfig igb1 up > > I've tried to track this down, and as far as I can see, this is from some > change introduced between 8.1-RELEASE (igb 1.9.5) and the current 8-STABLE > (igb 2.0.1). When I try it when booting from the 8.1-RELEASE amd64 DVD, I > can bring up both interfaces. When I try it with an 8-STABLE kernel, I get > the error and igb1 is missing the "RUNNING" flag: > > igb0: flags=8843 metric 0 mtu 9000 > > options=1bb > ether 00:25:90:xx:xx:bc > inet6 fe80::225:90ff:fe02:xxbc%igb0 prefixlen 64 scopeid 0x1 > nd6 options=3 > media: Ethernet 1000baseT > status: active > igb1: flags=8803 metric 0 mtu 9000 > > options=1bb > ether 00:25:90:xx:xx:bd > inet6 fe80::225:90ff:fe02:xxbd%igb1 prefixlen 64 tentative scopeid > 0x2 > nd6 options=3 > media: Ethernet 1000baseT > status: active > > I see 3 mbuf_jumbo_page allocation failures from: > > (1:20) rz1m:/sys/dev/e1000# vmstat -z | grep -v 0\$ > ITEM SIZE LIMIT USED FREE REQUESTS > FAILURES > > 64 Bucket: 536, 0, 263, 3, 263, > 106 > 128 Bucket: 1048, 0, 523, 2, 525, > 139 > mbuf_jumbo_page: 4096, 12800, 12307, 493, 30343, > 3 > > which correspond to the 3 "igb1: Could not setup receive structures" mess- > ages. If I try another "ifconfig igb1 up", I get another console message > and > the counter goes to 4. If I bump the kern.ipc.nmbjumbop sysctl to a larger > value, like 15000, I get the same error message when trying to work with > the > igb1 device, so I don't think it is a "real" error but indicates a problem > in the driver. > > This is on a Supermicro X8DTH-iF, BIOS 2.0a (latest) with a dual on-board > 82576. The dev.igb sysctl's for the two ports (excluding 0 values) are at- > tached. Note that most of the igb1 values are zero: > > (0:34) rz1m:/sys/dev/e1000# sysctl -a | grep igb.1 | grep -v ": 0" > dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection version - 2.0.1 > dev.igb.1.%driver: igb > dev.igb.1.%location: slot=0 function=1 > dev.igb.1.%pnpinfo: vendor=0x8086 device=0x10c9 subvendor=0x15d9 > subdevice=0x0400 class=0x020000 > dev.igb.1.%parent: pci1 > dev.igb.1.nvm: -1 > dev.igb.1.flow_control: 3 > dev.igb.1.enable_aim: 1 > dev.igb.1.rx_processing_limit: 100 > dev.igb.1.link_irq: 1 > dev.igb.1.device_control: 13632065 > dev.igb.1.extended_int_mask: 2147483648 > dev.igb.1.fc_high_water: 47488 > dev.igb.1.fc_low_water: 47472 > > (0:35) rz1m:/sys/dev/e1000# sysctl -a | grep igb.0 | grep -v ": 0" > dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 2.0.1 > dev.igb.0.%driver: igb > dev.igb.0.%location: slot=0 function=0 > dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10c9 subvendor=0x15d9 > subdevice=0x0400 class=0x020000 > dev.igb.0.%parent: pci1 > dev.igb.0.nvm: -1 > dev.igb.0.flow_control: 3 > dev.igb.0.enable_aim: 1 > dev.igb.0.rx_processing_limit: 100 > dev.igb.0.link_irq: 4 > dev.igb.0.device_control: 1087373889 > dev.igb.0.rx_control: 67338274 > dev.igb.0.interrupt_mask: 4 > dev.igb.0.extended_int_mask: 2147484671 > dev.igb.0.fc_high_water: 47488 > dev.igb.0.fc_low_water: 47472 > dev.igb.0.queue0.txd_head: 823 > dev.igb.0.queue0.txd_tail: 823 > dev.igb.0.queue0.tx_packets: 1402 > dev.igb.0.queue0.rxd_head: 319 > dev.igb.0.queue0.rxd_tail: 318 > dev.igb.0.queue0.rx_packets: 319 > dev.igb.0.queue0.rx_bytes: 69075 > dev.igb.0.queue1.txd_head: 2 > dev.igb.0.queue1.txd_tail: 2 > dev.igb.0.queue1.tx_packets: 1 > dev.igb.0.queue1.rxd_head: 52 > dev.igb.0.queue1.rxd_tail: 51 > dev.igb.0.queue1.rx_packets: 52 > dev.igb.0.queue1.rx_bytes: 6369 > dev.igb.0.queue2.txd_head: 27 > dev.igb.0.queue2.txd_tail: 27 > dev.igb.0.queue2.tx_packets: 13 > dev.igb.0.queue2.rxd_head: 72 > dev.igb.0.queue2.rxd_tail: 71 > dev.igb.0.queue2.rx_packets: 72 > dev.igb.0.queue2.rx_bytes: 8789 > dev.igb.0.queue3.txd_head: 177 > dev.igb.0.queue3.txd_tail: 177 > dev.igb.0.queue3.tx_packets: 64 > dev.igb.0.queue3.rxd_head: 88 > dev.igb.0.queue3.rxd_tail: 87 > dev.igb.0.queue3.rx_packets: 88 > dev.igb.0.queue3.rx_bytes: 16454 > dev.igb.0.queue4.rxd_head: 21 > dev.igb.0.queue4.rxd_tail: 20 > dev.igb.0.queue4.rx_packets: 21 > dev.igb.0.queue4.rx_bytes: 3547 > dev.igb.0.queue5.txd_head: 19 > dev.igb.0.queue5.txd_tail: 19 > dev.igb.0.queue5.tx_packets: 9 > dev.igb.0.queue5.rxd_head: 120 > dev.igb.0.queue5.rxd_tail: 119 > dev.igb.0.queue5.rx_packets: 120 > dev.igb.0.queue5.rx_bytes: 35518 > dev.igb.0.queue6.txd_head: 21 > dev.igb.0.queue6.txd_tail: 21 > dev.igb.0.queue6.tx_packets: 10 > dev.igb.0.queue6.rxd_head: 21 > dev.igb.0.queue6.rxd_tail: 20 > dev.igb.0.queue6.rx_packets: 21 > dev.igb.0.queue6.rx_bytes: 2876 > dev.igb.0.queue7.txd_head: 100 > dev.igb.0.queue7.txd_tail: 100 > dev.igb.0.queue7.tx_packets: 50 > dev.igb.0.queue7.rxd_head: 74 > dev.igb.0.queue7.rxd_tail: 73 > dev.igb.0.queue7.rx_packets: 1098 > dev.igb.0.queue7.rx_bytes: 116918 > dev.igb.0.queue8.txd_head: 11 > dev.igb.0.queue8.txd_tail: 11 > dev.igb.0.queue8.tx_packets: 6 > dev.igb.0.queue8.rxd_head: 25 > dev.igb.0.queue8.rxd_tail: 24 > dev.igb.0.queue8.rx_packets: 25 > dev.igb.0.queue8.rx_bytes: 3698 > dev.igb.0.mac_stats.total_pkts_recvd: 3138 > dev.igb.0.mac_stats.good_pkts_recvd: 1815 > dev.igb.0.mac_stats.bcast_pkts_recvd: 383 > dev.igb.0.mac_stats.mcast_pkts_recvd: 114 > dev.igb.0.mac_stats.rx_frames_64: 217 > dev.igb.0.mac_stats.rx_frames_65_127: 673 > dev.igb.0.mac_stats.rx_frames_128_255: 779 > dev.igb.0.mac_stats.rx_frames_256_511: 61 > dev.igb.0.mac_stats.rx_frames_512_1023: 46 > dev.igb.0.mac_stats.rx_frames_1024_1522: 39 > dev.igb.0.mac_stats.good_octets_recvd: 270378 > dev.igb.0.mac_stats.good_octets_txd: 252216 > dev.igb.0.mac_stats.total_pkts_txd: 1554 > dev.igb.0.mac_stats.good_pkts_txd: 1554 > dev.igb.0.mac_stats.bcast_pkts_txd: 36 > dev.igb.0.mac_stats.mcast_pkts_txd: 65 > dev.igb.0.mac_stats.tx_frames_64: 32 > dev.igb.0.mac_stats.tx_frames_65_127: 219 > dev.igb.0.mac_stats.tx_frames_128_255: 1222 > dev.igb.0.mac_stats.tx_frames_256_511: 55 > dev.igb.0.mac_stats.tx_frames_512_1023: 19 > dev.igb.0.mac_stats.tx_frames_1024_1522: 7 > dev.igb.0.interrupts.asserts: 3350 > dev.igb.0.interrupts.rx_pkt_timer: 1815 > dev.igb.0.interrupts.tx_abs_timer: 1815 > dev.igb.0.interrupts.tx_queue_empty: 1554 > dev.igb.0.host.rx_good_bytes: 270378 > dev.igb.0.host.tx_good_bytes: 252216 > > Both ports are cabled to the same Cisco switch. If I swap the cables at > the FreeBSD end between igb0 and igb1, the problem stays with igb1 so I > don't think it is the switch. > > I have 3 identical systems, all of which exhibit this same issue. Un- > fortunately, I don't have any other hardware with dual igb's. > > I can give a developer root access as well as a web-based remote console > if needed to track this down. > > Terry Kennedy http://www.tmk.com > terry@tmk.com New York, NY USA >