From owner-freebsd-stable@FreeBSD.ORG Tue Oct 23 21:12:13 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A96EE865 for ; Tue, 23 Oct 2012 21:12:13 +0000 (UTC) (envelope-from marius@alchemy.franken.de) Received: from alchemy.franken.de (alchemy.franken.de [194.94.249.214]) by mx1.freebsd.org (Postfix) with ESMTP id 366C28FC0A for ; Tue, 23 Oct 2012 21:12:12 +0000 (UTC) Received: from alchemy.franken.de (localhost [127.0.0.1]) by alchemy.franken.de (8.14.4/8.14.4/ALCHEMY.FRANKEN.DE) with ESMTP id q9NLC57s021345; Tue, 23 Oct 2012 23:12:05 +0200 (CEST) (envelope-from marius@alchemy.franken.de) Received: (from marius@localhost) by alchemy.franken.de (8.14.4/8.14.4/Submit) id q9NLC5Q9021344; Tue, 23 Oct 2012 23:12:05 +0200 (CEST) (envelope-from marius) Date: Tue, 23 Oct 2012 23:12:05 +0200 From: Marius Strobl To: Harald Schmalzbauer Subject: Re: msi-x enabled igb works only if module loaded twice [Was: Re: kldload if_igb twice needed to bring nic into operation] Message-ID: <20121023211205.GA21019@alchemy.franken.de> References: <50859F7F.2080308@omnilan.de> <5085A323.8030501@omnilan.de> <50866839.5090204@omnilan.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <50866839.5090204@omnilan.de> User-Agent: Mutt/1.4.2.3i Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2012 21:12:13 -0000 On Tue, Oct 23, 2012 at 11:49:45AM +0200, Harald Schmalzbauer wrote: > schrieb Harald Schmalzbauer am 22.10.2012 21:48 (localtime): > > schrieb Harald Schmalzbauer am 22.10.2012 21:33 (localtime): > >> Hello, > >> > >> when using igb as module, no packet is received. > >> If I send out anything, I see the packet with tcpdump, also the switch > >> learns the MAC address, but nothing comes back in - total silenc, no > >> boradcasts, nothing. > >> If I unload the module and load it again, everything works as expected! > >> No matter if I load it by 4th loader, or later, I always have tio unload > >> first then load it again. > >> I'ts late here, I'll see tomorrow if things change when compieled into > >> kernel. > > It doesn't matter if igb is loaded as module or compiled into kernel. > > >> Maby somebody has an idea what the source of the problem could be. > >> Please find atteched some info, the OS is 9-RC2-amd64 on ESXi5.1 and > >> nics are pci-passthrough. > > I found one possibly relevant difference: > > > > Non-Working state: dev.igb.0.link_irq: 0 > > Working state: dev.igb.0.link_irq: 2 > > This is only true with msi-x!!! > If I disable mis-x, the problem itself vanishes. igb just works fine > from the initial loading (with dev.igb.0.link_irq=0!). > So dev.igb.0.link_irq is only relevant with msi-x. > But what makes me curious is why it also works mith mis-x enabled after > the second kldload!?! Hrm, this is strange; in r231621 (MFC'ed to stable/9 in r232092, so it's part of 9.1-RC2) I've blacklisted the fictitious PCI-PCI bridge VMware uses in case of PCI pass-through for MSI/MSI-X (as in the cases observed it didn't even work with a single MSI-X message). So unless they've changed the ID of the fictitious bridge, igb(4) should fail to allocate a MSI/MSI-X in the first place. Could you please provide the output of `pciconf -lv` on that "system"? Marius