From owner-freebsd-stable@FreeBSD.ORG Tue Oct 23 14:18:52 2012 Return-Path: <owner-freebsd-stable@FreeBSD.ORG> Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6E416D78 for <freebsd-stable@freebsd.org>; Tue, 23 Oct 2012 14:18:52 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id 3F7A58FC0A for <freebsd-stable@freebsd.org>; Tue, 23 Oct 2012 14:18:52 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 93054B98F; Tue, 23 Oct 2012 10:18:51 -0400 (EDT) From: John Baldwin <jhb@freebsd.org> To: freebsd-stable@freebsd.org Subject: Re: Possible to reset PCI device at boot? [Was: Re: msi-x enabled igb works only if module loaded twice] Date: Tue, 23 Oct 2012 10:12:06 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p20; KDE/4.5.5; amd64; ; ) References: <50859F7F.2080308@omnilan.de> <50866839.5090204@omnilan.de> <5086904C.3020409@omnilan.de> In-Reply-To: <5086904C.3020409@omnilan.de> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201210231012.06641.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 23 Oct 2012 10:18:51 -0400 (EDT) Cc: Harald Schmalzbauer <h.schmalzbauer@omnilan.de> X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org> List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-stable>, <mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable> List-Post: <mailto:freebsd-stable@freebsd.org> List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help> List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>, <mailto:freebsd-stable-request@freebsd.org?subject=subscribe> X-List-Received-Date: Tue, 23 Oct 2012 14:18:52 -0000 On Tuesday, October 23, 2012 8:40:44 am Harald Schmalzbauer wrote: > schrieb Harald Schmalzbauer am 23.10.2012 11:49 (localtime): > > schrieb Harald Schmalzbauer am 22.10.2012 21:48 (localtime): > >> schrieb Harald Schmalzbauer am 22.10.2012 21:33 (localtime): > >>> Hello, > >>> > >>> when using igb as module, no packet is received. > >>> If I send out anything, I see the packet with tcpdump, also the switch > >>> learns the MAC address, but nothing comes back in - total silenc, no > >>> boradcasts, nothing. > >>> If I unload the module and load it again, everything works as expected! > >>> No matter if I load it by 4th loader, or later, I always have tio unload > >>> first then load it again. > >>> I'ts late here, I'll see tomorrow if things change when compieled into > >>> kernel. > > It doesn't matter if igb is loaded as module or compiled into kernel. > > > >>> Maby somebody has an idea what the source of the problem could be. > >>> Please find atteched some info, the OS is 9-RC2-amd64 on ESXi5.1 and > >>> nics are pci-passthrough. > >> I found one possibly relevant difference: > >> > >> Non-Working state: dev.igb.0.link_irq: 0 > >> Working state: dev.igb.0.link_irq: 2 > > This is only true with msi-x!!! > > If I disable mis-x, the problem itself vanishes. igb just works fine > > from the initial loading (with dev.igb.0.link_irq=0!). > > So dev.igb.0.link_irq is only relevant with msi-x. > > But what makes me curious is why it also works mith mis-x enabled after > > the second kldload!?! > > I think I found the root cause: > When ESXi powers up the guest, the passthru-devices are intialized with: > VMKPCIPassthru: 2565: BDF = 02:00.1 intrType = 2 numVectors: 1 > intrType=2 seems to mean MSI. > I guess, IOMMUIntel is instructed to remap one irq-vector for the > device. But igb uses MSI-X and wants 3 vectors. > If I unload if_igb and reload again, the ESXi-log shows the following: > VMKPCIPassthru: 2565: BDF = 02:00.1 intrType = 4 numVectors: 3 > intrType=4 seems to mean MSI-X. > After that initialization, if_igb works fine and saves 25kIRQ/s! > > I haven't found a way to change the power-up behaviour for the guest > with ESXi. > > Is it possible to re-init a pci device from userland? The problem is you want the igb driver to retry MSI-X even after a re-init and that basically requires a full detach/attach, so your existing workaround is actually the "best" way to do this. :( Alternatively, you could try forcing igb to not use MSI, only use either MSI-X or INTx. -- John Baldwin