From owner-freebsd-stable@FreeBSD.ORG Tue Oct 23 09:49:55 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D2C89DC2 for ; Tue, 23 Oct 2012 09:49:55 +0000 (UTC) (envelope-from h.schmalzbauer@omnilan.de) Received: from host.omnilan.net (s1.omnilan.net [62.245.232.135]) by mx1.freebsd.org (Postfix) with ESMTP id 237A38FC18 for ; Tue, 23 Oct 2012 09:49:54 +0000 (UTC) Received: from titan.inop.wdn.omnilan.net (titan.inop.wdn.omnilan.net [172.21.3.1]) (authenticated bits=0) by host.omnilan.net (8.13.8/8.13.8) with ESMTP id q9N9pCdd041413 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 23 Oct 2012 11:51:14 +0200 (CEST) (envelope-from h.schmalzbauer@omnilan.de) Message-ID: <50866839.5090204@omnilan.de> Date: Tue, 23 Oct 2012 11:49:45 +0200 From: Harald Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: msi-x enabled igb works only if module loaded twice [Was: Re: kldload if_igb twice needed to bring nic into operation] References: <50859F7F.2080308@omnilan.de> <5085A323.8030501@omnilan.de> In-Reply-To: <5085A323.8030501@omnilan.de> X-Enigmail-Version: 1.1.2 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig6E93DD9F60145D847A8DB415" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2012 09:49:55 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig6E93DD9F60145D847A8DB415 Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: quoted-printable schrieb Harald Schmalzbauer am 22.10.2012 21:48 (localtime): > schrieb Harald Schmalzbauer am 22.10.2012 21:33 (localtime): >> Hello, >> >> when using igb as module, no packet is received. >> If I send out anything, I see the packet with tcpdump, also the switc= h >> learns the MAC address, but nothing comes back in - total silenc, no >> boradcasts, nothing. >> If I unload the module and load it again, everything works as expected= ! >> No matter if I load it by 4th loader, or later, I always have tio unlo= ad >> first then load it again. >> I'ts late here, I'll see tomorrow if things change when compieled into= >> kernel. It doesn't matter if igb is loaded as module or compiled into kernel. >> Maby somebody has an idea what the source of the problem could be. >> Please find atteched some info, the OS is 9-RC2-amd64 on ESXi5.1 and >> nics are pci-passthrough. > I found one possibly relevant difference: > > Non-Working state: dev.igb.0.link_irq: 0 > Working state: dev.igb.0.link_irq: 2 This is only true with msi-x!!! If I disable mis-x, the problem itself vanishes. igb just works fine from the initial loading (with dev.igb.0.link_irq=3D0!). So dev.igb.0.link_irq is only relevant with msi-x. But what makes me curious is why it also works mith mis-x enabled after the second kldload!?! Here's the verbose comparison: 1st attaching with msi-x enabled (non-operational): Oct 23 09:27:20 flint kernel: pci5: driver added Oct 23 09:27:20 flint kernel: found-> vendor=3D0x8086, dev=3D0x10c9, revid=3D0x01 Oct 23 09:27:20 flint kernel: domain=3D0, bus=3D5, slot=3D0, func=3D0 Oct 23 09:27:20 flint kernel: class=3D02-00-00, hdrtype=3D0x00, mfdev=3D0= Oct 23 09:27:20 flint kernel: cmdreg=3D0x0003, statreg=3D0x0010, cachelnsz=3D16 (dwords) Oct 23 09:27:20 flint kernel: lattimer=3D0x40 (1920 ns), mingnt=3D0x00 (0= ns), maxlat=3D0x00 (0 ns) Oct 23 09:27:20 flint kernel: intpin=3Da, irq=3D16 Oct 23 09:27:20 flint kernel: powerspec 3 supports D0 D3 current D0 Oct 23 09:27:20 flint kernel: MSI supports 1 message, 64 bit, vector mask= s Oct 23 09:27:20 flint kernel: MSI-X supports 10 messages in map 0x1c Oct 23 09:27:20 flint kernel: pci0:5:0:0: reprobing on driver added 00-0x601f mem 0xd6620000-0xd663ffff,0xd6800000-0xd6bfffff,0xd6600000-0xd6603fff irq 16 at device 0.0 on pci5 Oct 23 09:27:20 flint kernel: igb0: attempting to allocate 3 MSI-X vectors (10 supported) Oct 23 09:27:20 flint kernel: msi: routing MSI-X IRQ 257 to local APIC 1 vector 50 Oct 23 09:27:20 flint kernel: msi: routing MSI-X IRQ 258 to local APIC 2 vector 50 Oct 23 09:27:20 flint kernel: msi: routing MSI-X IRQ 259 to local APIC 3 vector 50 Oct 23 09:27:20 flint kernel: igb0: using IRQs 257-259 for MSI-X Oct 23 09:27:20 flint kernel: igb0: Using MSIX interrupts with 3 vectors Oct 23 09:27:20 flint kernel: igb0: bpf attached Oct 23 09:27:20 flint kernel: igb0: Ethernet address: 90:e2:ba:18:f8:85 Oct 23 09:27:20 flint kernel: msi: Assigning MSI-X IRQ 257 to local APIC 0 vector 51 Oct 23 09:27:20 flint kernel: igb0: Bound queue 0 to cpu 0 Oct 23 09:27:20 flint kernel: msi: Assigning MSI-X IRQ 258 to local APIC 1 vector 50 Oct 23 09:27:20 flint kernel: igb0: Bound queue 1 to cpu 2nd attaching with mis-x enabled (working): Oct 23 09:28:12 flint kernel: pci5: driver added Oct 23 09:28:12 flint kernel: found-> vendor=3D0x8086, dev=3D0x10c9, revid=3D0x01 Oct 23 09:28:12 flint kernel: domain=3D0, bus=3D5, slot=3D0, func=3D0 Oct 23 09:28:12 flint kernel: class=3D02-00-00, hdrtype=3D0x00, mfdev=3D0= Oct 23 09:28:12 flint kernel: cmdreg=3D0x0407, statreg=3D0x0010, cachelnsz=3D16 (dwords) Oct 23 09:28:12 flint kernel: lattimer=3D0x40 (1920 ns), mingnt=3D0x00 (0= ns), maxlat=3D0x00 (0 ns) Oct 23 09:28:12 flint kernel: intpin=3Da, irq=3D16 Oct 23 09:28:12 flint kernel: powerspec 3 supports D0 D3 current D0 Oct 23 09:28:12 flint kernel: MSI supports 1 message, 64 bit, vector mask= s Oct 23 09:28:12 flint kernel: MSI-X supports 10 messages in map 0x1c Oct 23 09:28:12 flint kernel: pci0:5:0:0: reprobing on driver added Oct 23 09:28:12 flint kernel: igb0: port 0x6000-0x601f mem 0xd6620000-0xd663ffff,0xd6800000-0xd6bfffff,0xd6600000-0xd6603fff irq 16 at device 0.0 on pci5 Oct 23 09:28:12 flint kernel: igb0: attempting to allocate 3 MSI-X vectors (10 supported) Oct 23 09:28:12 flint kernel: msi: routing MSI-X IRQ 257 to local APIC 3 vector 50 Oct 23 09:28:12 flint kernel: msi: routing MSI-X IRQ 258 to local APIC 0 vector 51 Oct 23 09:28:12 flint kernel: msi: routing MSI-X IRQ 259 to local APIC 1 vector 50 Oct 23 09:28:12 flint kernel: igb0: using IRQs 257-259 for MSI-X Oct 23 09:28:12 flint kernel: igb0: Using MSIX interrupts with 3 vectors Oct 23 09:28:12 flint kernel: igb0: bpf attached Oct 23 09:28:12 flint kernel: igb0: Ethernet address: 90:e2:ba:18:f8:85 Oct 23 09:28:12 flint kernel: msi: Assigning MSI-X IRQ 257 to local APIC 0 vector 52 Oct 23 09:28:12 flint kernel: igb0: Bound queue 0 to cpu 0 Oct 23 09:28:12 flint kernel: msi: Assigning MSI-X IRQ 258 to local APIC 1 vector 51 Oct 23 09:28:12 flint kernel: igb0: Bound queue 1 to cpu 1 Since I have no idea how drivers use MSI(X), I have no idea what the problem for this strange behaviour is. As workarround I can disable MSI-X for the driver, or better for the whole system, since also mps doesn't work with MSI-X enabled in my setup.= But I'd highly appreciate any help understanding what's going wrong. It likely has something todo with my (VT-d) passthrough setup. Thaks in advance, -Harry --------------enig6E93DD9F60145D847A8DB415 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAlCGaD4ACgkQLDqVQ9VXb8iq0wCgoHxrnXQZi6m8A6AYBC5pttXu DxcAoKUncJU03dIBdUGWlu0pDVB65gkq =J0Ra -----END PGP SIGNATURE----- --------------enig6E93DD9F60145D847A8DB415--