Date: Tue, 23 Oct 2012 14:25:58 -0700 From: Jack Vogel <jfvogel@gmail.com> To: Marius Strobl <marius@alchemy.franken.de> Cc: Harald Schmalzbauer <h.schmalzbauer@omnilan.de>, freebsd-stable@freebsd.org Subject: Re: msi-x enabled igb works only if module loaded twice [Was: Re: kldload if_igb twice needed to bring nic into operation] Message-ID: <CAFOYbcmz90bHTfD7Qqgkb_aHRHHu43Cm4ZUKpYiizooRSp%2Bs4A@mail.gmail.com> In-Reply-To: <20121023211205.GA21019@alchemy.franken.de> References: <50859F7F.2080308@omnilan.de> <5085A323.8030501@omnilan.de> <50866839.5090204@omnilan.de> <20121023211205.GA21019@alchemy.franken.de>
next in thread | previous in thread | raw e-mail | index | archive | help
LOL, wow this is interesting. When I first was developing the VF support, and using Linux KVM as the host I had exactly this problem, turned out it was because of the way in which MSIX was handled in the Linux side. When my driver would first attempt to get some vectors the Linux code would go look at the vector table, but the way the PCI code in FreeBSD works that table is not set up yet, so Linux would see no legitimate vectors in the table, and decide the guest was ineligible for MSIX :(, but by this time the FreeBSD code actually DID write some vectors into the table, and thus when you load the driver the SECOND time Linux would see the table populated and TADA!! would enable the guest to use MSIX. Now, I went thru a bunch of efforts via our Linux team here to have the KVM code fixed, its design was bogus, and I believe it has been, but it sounds like maybe VMWare has the same broken design?? There is nothing you can do about this because the issue is in the host, not the guest, well getting the host code fixed is the solution :) Hope this helps, Jack On Tue, Oct 23, 2012 at 2:12 PM, Marius Strobl <marius@alchemy.franken.de>wrote: > On Tue, Oct 23, 2012 at 11:49:45AM +0200, Harald Schmalzbauer wrote: > > schrieb Harald Schmalzbauer am 22.10.2012 21:48 (localtime): > > > schrieb Harald Schmalzbauer am 22.10.2012 21:33 (localtime): > > >> Hello, > > >> > > >> when using igb as module, no packet is received. > > >> If I send out anything, I see the packet with tcpdump, also the > switch > > >> learns the MAC address, but nothing comes back in - total silenc, no > > >> boradcasts, nothing. > > >> If I unload the module and load it again, everything works as > expected! > > >> No matter if I load it by 4th loader, or later, I always have tio > unload > > >> first then load it again. > > >> I'ts late here, I'll see tomorrow if things change when compieled into > > >> kernel. > > > > It doesn't matter if igb is loaded as module or compiled into kernel. > > > > >> Maby somebody has an idea what the source of the problem could be. > > >> Please find atteched some info, the OS is 9-RC2-amd64 on ESXi5.1 and > > >> nics are pci-passthrough. > > > I found one possibly relevant difference: > > > > > > Non-Working state: dev.igb.0.link_irq: 0 > > > Working state: dev.igb.0.link_irq: 2 > > > > This is only true with msi-x!!! > > If I disable mis-x, the problem itself vanishes. igb just works fine > > from the initial loading (with dev.igb.0.link_irq=0!). > > So dev.igb.0.link_irq is only relevant with msi-x. > > But what makes me curious is why it also works mith mis-x enabled after > > the second kldload!?! > > Hrm, this is strange; in r231621 (MFC'ed to stable/9 in r232092, so > it's part of 9.1-RC2) I've blacklisted the fictitious PCI-PCI bridge > VMware uses in case of PCI pass-through for MSI/MSI-X (as in the cases > observed it didn't even work with a single MSI-X message). So unless > they've changed the ID of the fictitious bridge, igb(4) should fail > to allocate a MSI/MSI-X in the first place. Could you please provide > the output of `pciconf -lv` on that "system"? > > Marius > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFOYbcmz90bHTfD7Qqgkb_aHRHHu43Cm4ZUKpYiizooRSp%2Bs4A>