Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 23 Oct 2012 10:12:06 -0400
From:      John Baldwin <jhb@freebsd.org>
To:        freebsd-stable@freebsd.org
Cc:        Harald Schmalzbauer <h.schmalzbauer@omnilan.de>
Subject:   Re: Possible to reset PCI device at boot? [Was: Re: msi-x enabled igb works only if module loaded twice]
Message-ID:  <201210231012.06641.jhb@freebsd.org>
In-Reply-To: <5086904C.3020409@omnilan.de>
References:  <50859F7F.2080308@omnilan.de> <50866839.5090204@omnilan.de> <5086904C.3020409@omnilan.de>

next in thread | previous in thread | raw e-mail | index | archive | help
On Tuesday, October 23, 2012 8:40:44 am Harald Schmalzbauer wrote:
>  schrieb Harald Schmalzbauer am 23.10.2012 11:49 (localtime):
> >  schrieb Harald Schmalzbauer am 22.10.2012 21:48 (localtime):
> >>  schrieb Harald Schmalzbauer am 22.10.2012 21:33 (localtime):
> >>>  Hello,
> >>>
> >>> when using igb as module, no packet is received.
> >>> If I send out anything, I see the packet with tcpdump,  also the switch
> >>> learns the MAC address, but nothing comes back in - total silenc, no
> >>> boradcasts, nothing.
> >>> If I unload the module and load it again, everything works as expected!
> >>> No matter if I load it by 4th loader, or later, I always have tio unload
> >>> first then load it again.
> >>> I'ts late here, I'll see tomorrow if things change when compieled into
> >>> kernel.
> > It doesn't matter if igb is loaded as module or compiled into kernel.
> >
> >>> Maby somebody has an idea what the source of the problem could be.
> >>> Please find atteched some info, the OS is 9-RC2-amd64 on ESXi5.1 and
> >>> nics are pci-passthrough.
> >> I found one possibly relevant difference:
> >>
> >> Non-Working state:    dev.igb.0.link_irq: 0
> >> Working state:           dev.igb.0.link_irq: 2
> > This is only true with msi-x!!!
> > If I disable mis-x, the problem itself vanishes. igb just works fine
> > from the initial loading (with dev.igb.0.link_irq=0!).
> > So dev.igb.0.link_irq is only relevant with msi-x.
> > But what makes me curious is why it also works mith mis-x enabled after
> > the second kldload!?!
>  
> I think I found the root cause:
> When ESXi powers up the guest, the passthru-devices are intialized with:
> VMKPCIPassthru: 2565: BDF = 02:00.1 intrType = 2 numVectors: 1
> intrType=2 seems to mean MSI.
> I guess, IOMMUIntel is instructed to remap one irq-vector for the
> device. But igb uses MSI-X and wants 3 vectors.
> If I unload if_igb and reload again, the ESXi-log shows the following:
> VMKPCIPassthru: 2565: BDF = 02:00.1 intrType = 4 numVectors: 3
> intrType=4 seems to mean MSI-X.
> After that initialization, if_igb works fine and saves 25kIRQ/s!
> 
> I haven't found a way to change the power-up behaviour for the guest
> with ESXi.
> 
> Is it possible to re-init a pci device from userland?

The problem is you want the igb driver to retry MSI-X even after a re-init
and that basically requires a full detach/attach, so your existing workaround
is actually the "best" way to do this. :(  Alternatively, you could try
forcing igb to not use MSI, only use either MSI-X or INTx.

-- 
John Baldwin



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201210231012.06641.jhb>