From owner-freebsd-stable@FreeBSD.ORG Tue Oct 23 12:40:47 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 43E5CB86 for ; Tue, 23 Oct 2012 12:40:47 +0000 (UTC) (envelope-from h.schmalzbauer@omnilan.de) Received: from host.omnilan.net (s1.omnilan.net [62.245.232.135]) by mx1.freebsd.org (Postfix) with ESMTP id BA1D08FC0C for ; Tue, 23 Oct 2012 12:40:46 +0000 (UTC) Received: from titan.inop.wdn.omnilan.net (titan.inop.wdn.omnilan.net [172.21.3.1]) (authenticated bits=0) by host.omnilan.net (8.13.8/8.13.8) with ESMTP id q9NCg6is044318 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 23 Oct 2012 14:42:06 +0200 (CEST) (envelope-from h.schmalzbauer@omnilan.de) Message-ID: <5086904C.3020409@omnilan.de> Date: Tue, 23 Oct 2012 14:40:44 +0200 From: Harald Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: Possible to reset PCI device at boot? [Was: Re: msi-x enabled igb works only if module loaded twice] References: <50859F7F.2080308@omnilan.de> <5085A323.8030501@omnilan.de> <50866839.5090204@omnilan.de> In-Reply-To: <50866839.5090204@omnilan.de> X-Enigmail-Version: 1.1.2 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigEE634B028EB53CF4C392AB55" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2012 12:40:47 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigEE634B028EB53CF4C392AB55 Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: quoted-printable schrieb Harald Schmalzbauer am 23.10.2012 11:49 (localtime): > schrieb Harald Schmalzbauer am 22.10.2012 21:48 (localtime): >> schrieb Harald Schmalzbauer am 22.10.2012 21:33 (localtime): >>> Hello, >>> >>> when using igb as module, no packet is received. >>> If I send out anything, I see the packet with tcpdump, also the swit= ch >>> learns the MAC address, but nothing comes back in - total silenc, no >>> boradcasts, nothing. >>> If I unload the module and load it again, everything works as expecte= d! >>> No matter if I load it by 4th loader, or later, I always have tio unl= oad >>> first then load it again. >>> I'ts late here, I'll see tomorrow if things change when compieled int= o >>> kernel. > It doesn't matter if igb is loaded as module or compiled into kernel. > >>> Maby somebody has an idea what the source of the problem could be. >>> Please find atteched some info, the OS is 9-RC2-amd64 on ESXi5.1 and >>> nics are pci-passthrough. >> I found one possibly relevant difference: >> >> Non-Working state: dev.igb.0.link_irq: 0 >> Working state: dev.igb.0.link_irq: 2 > This is only true with msi-x!!! > If I disable mis-x, the problem itself vanishes. igb just works fine > from the initial loading (with dev.igb.0.link_irq=3D0!). > So dev.igb.0.link_irq is only relevant with msi-x. > But what makes me curious is why it also works mith mis-x enabled after= > the second kldload!?! =20 I think I found the root cause: When ESXi powers up the guest, the passthru-devices are intialized with: VMKPCIPassthru: 2565: BDF =3D 02:00.1 intrType =3D 2 numVectors: 1 intrType=3D2 seems to mean MSI. I guess, IOMMUIntel is instructed to remap one irq-vector for the device. But igb uses MSI-X and wants 3 vectors. If I unload if_igb and reload again, the ESXi-log shows the following: VMKPCIPassthru: 2565: BDF =3D 02:00.1 intrType =3D 4 numVectors: 3 intrType=3D4 seems to mean MSI-X. After that initialization, if_igb works fine and saves 25kIRQ/s! I haven't found a way to change the power-up behaviour for the guest with ESXi. Is it possible to re-init a pci device from userland? Thanks, -Harry --------------enigEE634B028EB53CF4C392AB55 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAlCGkEwACgkQLDqVQ9VXb8hw1QCaAwgB2JrUv1VbvfkC+Wk1HpIc YDQAoMuhBJqde3gHrF8prBxmSK3o2C31 =8FRZ -----END PGP SIGNATURE----- --------------enigEE634B028EB53CF4C392AB55--