From owner-freebsd-stable@FreeBSD.ORG Tue Oct 23 21:26:00 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 53282189 for ; Tue, 23 Oct 2012 21:26:00 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-vb0-f54.google.com (mail-vb0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id F36BD8FC12 for ; Tue, 23 Oct 2012 21:25:59 +0000 (UTC) Received: by mail-vb0-f54.google.com with SMTP id v11so6029784vbm.13 for ; Tue, 23 Oct 2012 14:25:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=TkZ0UHgDMH6a8YtO7+J+w4/jHVikZ63+sf1mVT1qcqM=; b=D7QzV0/ILmx139V5d6oiEv1gHYJqruWldGGrH30aNW0FnyyVO6x7+/5il+7bkCgyx+ 8LH7pJLgEFIgB/twEe6mqRpix9ih/DrTmRkTrcyAPES0hn+7PoBQsGGp1xwYZgBEef6C GnYc/BLYvplwB7gIWvukKxvHG53vaeYKyUsBQIPAuRj8OhcEGIwEz8Dza5rHItVFm5LO yUj0mi+9PTypjqN5LI04CLVTpdd6PfNNenwgesuOilScyvIsTCjq4e5mXbH8GTsDpYjK jL+N6+Fo875WuEPUXFkPczzZPvyLN/ZIvAAAY7BO1o7LxC008Gk7H8mON1AaznlWs0S2 e3Sg== MIME-Version: 1.0 Received: by 10.220.119.196 with SMTP id a4mr4654566vcr.19.1351027558976; Tue, 23 Oct 2012 14:25:58 -0700 (PDT) Received: by 10.58.68.8 with HTTP; Tue, 23 Oct 2012 14:25:58 -0700 (PDT) In-Reply-To: <20121023211205.GA21019@alchemy.franken.de> References: <50859F7F.2080308@omnilan.de> <5085A323.8030501@omnilan.de> <50866839.5090204@omnilan.de> <20121023211205.GA21019@alchemy.franken.de> Date: Tue, 23 Oct 2012 14:25:58 -0700 Message-ID: Subject: Re: msi-x enabled igb works only if module loaded twice [Was: Re: kldload if_igb twice needed to bring nic into operation] From: Jack Vogel To: Marius Strobl Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Harald Schmalzbauer , freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2012 21:26:00 -0000 LOL, wow this is interesting. When I first was developing the VF support, and using Linux KVM as the host I had exactly this problem, turned out it was because of the way in which MSIX was handled in the Linux side. When my driver would first attempt to get some vectors the Linux code would go look at the vector table, but the way the PCI code in FreeBSD works that table is not set up yet, so Linux would see no legitimate vectors in the table, and decide the guest was ineligible for MSIX :(, but by this time the FreeBSD code actually DID write some vectors into the table, and thus when you load the driver the SECOND time Linux would see the table populated and TADA!! would enable the guest to use MSIX. Now, I went thru a bunch of efforts via our Linux team here to have the KVM code fixed, its design was bogus, and I believe it has been, but it sounds like maybe VMWare has the same broken design?? There is nothing you can do about this because the issue is in the host, not the guest, well getting the host code fixed is the solution :) Hope this helps, Jack On Tue, Oct 23, 2012 at 2:12 PM, Marius Strobl wrote: > On Tue, Oct 23, 2012 at 11:49:45AM +0200, Harald Schmalzbauer wrote: > > schrieb Harald Schmalzbauer am 22.10.2012 21:48 (localtime): > > > schrieb Harald Schmalzbauer am 22.10.2012 21:33 (localtime): > > >> Hello, > > >> > > >> when using igb as module, no packet is received. > > >> If I send out anything, I see the packet with tcpdump, also the > switch > > >> learns the MAC address, but nothing comes back in - total silenc, no > > >> boradcasts, nothing. > > >> If I unload the module and load it again, everything works as > expected! > > >> No matter if I load it by 4th loader, or later, I always have tio > unload > > >> first then load it again. > > >> I'ts late here, I'll see tomorrow if things change when compieled into > > >> kernel. > > > > It doesn't matter if igb is loaded as module or compiled into kernel. > > > > >> Maby somebody has an idea what the source of the problem could be. > > >> Please find atteched some info, the OS is 9-RC2-amd64 on ESXi5.1 and > > >> nics are pci-passthrough. > > > I found one possibly relevant difference: > > > > > > Non-Working state: dev.igb.0.link_irq: 0 > > > Working state: dev.igb.0.link_irq: 2 > > > > This is only true with msi-x!!! > > If I disable mis-x, the problem itself vanishes. igb just works fine > > from the initial loading (with dev.igb.0.link_irq=0!). > > So dev.igb.0.link_irq is only relevant with msi-x. > > But what makes me curious is why it also works mith mis-x enabled after > > the second kldload!?! > > Hrm, this is strange; in r231621 (MFC'ed to stable/9 in r232092, so > it's part of 9.1-RC2) I've blacklisted the fictitious PCI-PCI bridge > VMware uses in case of PCI pass-through for MSI/MSI-X (as in the cases > observed it didn't even work with a single MSI-X message). So unless > they've changed the ID of the fictitious bridge, igb(4) should fail > to allocate a MSI/MSI-X in the first place. Could you please provide > the output of `pciconf -lv` on that "system"? > > Marius > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >