From owner-freebsd-stable@FreeBSD.ORG Thu Nov 19 02:23:13 2009 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B7AD3106566B for ; Thu, 19 Nov 2009 02:23:13 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by mx1.freebsd.org (Postfix) with ESMTP id 7F5918FC14 for ; Thu, 19 Nov 2009 02:23:13 +0000 (UTC) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.14.3/8.14.3) with ESMTP id nAJ2NCFL091290; Wed, 18 Nov 2009 21:23:12 -0500 (EST) (envelope-from mike@sentex.net) Message-Id: <200911190223.nAJ2NCFL091290@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Wed, 18 Nov 2009 21:23:20 -0500 To: Jack Vogel From: Mike Tancsa In-Reply-To: <2a41acea0911181629r4eb1a199we28a94a82ec67cf@mail.gmail.com > References: <200911182130.nAILUJCR089766@lava.sentex.ca> <2a41acea0911181629r4eb1a199we28a94a82ec67cf@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Cc: freebsd-stable@freebsd.org Subject: Re: bug with some em nics on RELENG_7 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Nov 2009 02:23:13 -0000 At 07:29 PM 11/18/2009, Jack Vogel wrote: >Hey Mike, > >Can you check if you see the same behavior on RELENG 8? Hi Jack, Yes, I will reboot the hardware with a RELENG_8 image tomorrow to test >Not sure why this happens on Hartwell (82574) and not on 82571, that's >an interesting bit, the 82574 is the ONLY interface in the em driver that >has MSIX support, unfortunately its kinda hacked in, but it did not really >fit into the igb driver either for various technical reasons. Is this the "FILTER" vs "ITHREAD" ? Is there a way to force this chipset to use the same logic as 82571s ? # dmesg |grep ^em em0: port 0xbc00-0xbc1f mem 0xface0000-0xfacfffff,0xfacc0000-0xfacdffff irq 16 at device 0.0 on pci1 em0: Using MSI interrupt em0: [FILTER] em0: Ethernet address: 00:15:17:78:e6:e0 em1: port 0xb880-0xb89f mem 0xfac80000-0xfac9ffff,0xfac60000-0xfac7ffff irq 17 at device 0.1 on pci1 em1: Using MSI interrupt em1: [FILTER] em1: Ethernet address: 00:15:17:78:e6:e1 em2: port 0xcc00-0xcc1f mem 0xfade0000-0xfadfffff,0xfadc0000-0xfaddffff irq 16 at device 0.0 on pci3 em2: Using MSI interrupt em2: [FILTER] em2: Ethernet address: 00:15:17:cf:26:de em3: port 0xc880-0xc89f mem 0xfad80000-0xfad9ffff,0xfad60000-0xfad7ffff irq 17 at device 0.1 on pci3 em3: Using MSI interrupt em3: [FILTER] em3: Ethernet address: 00:15:17:cf:26:df em4: port 0xdc00-0xdc1f mem 0xfaee0000-0xfaefffff,0xfaedc000-0xfaedffff irq 16 at device 0.0 on pci6 em4: Using MSIX interrupts em4: [ITHREAD] em4: [ITHREAD] em4: [ITHREAD] em4: Ethernet address: 00:30:48:d6:ef:12 em5: port 0xec00-0xec1f mem 0xfafe0000-0xfaffffff,0xfafdc000-0xfafdffff irq 17 at device 0.0 on pci7 em5: Using MSIX interrupts em5: [ITHREAD] em5: [ITHREAD] em5: [ITHREAD] em5: Ethernet address: 00:30:48:d6:ef:13 >What if you boot up, then do NOT ping or anything until the interface is >assigned an address (and so init is run), and the cable is plugged in. If >that happens first does it work? yes. If I have the cables plugged in and reboot the box, its ok. I am pretty sure all is ok if I boot it up, with no address assigned, plug the cables in, and then assign addr. I havent tested it out yet, but not sure how things play out when the ports are connected to a switch that is not in portfast mode, so the carrier does not always come up right away. The other thing I saw was that the NIC was getting stuck with the carrier showing up, even though cable was unplugged. However, I was not able to find the exact conditions this happened. >Do let me know if you can check on 8, if not I can have my validation >engineer try this. I will report back tomorrow. ---Mike >Best regards, > >Jack > > >On Wed, Nov 18, 2009 at 1:30 PM, Mike Tancsa ><mike@sentex.net> wrote: > >On two Intel chipset Supermicro boards (X8STi and X8STE-0) using the >onboard em nics (dmesg info below), I seem to have run into an issue >where if I boot the box up with the cables unplugged, I cannot get >the NICS to properly work post boot up. This is quite repeatable >for me. So at boot time, I have > ># ifconfig em5 >em5: flags=8843 metric 0 mtu 1500 > options=19b > ether 00:30:48:d6:ef:13 > inet 3.3.3.3 netmask 0xffffff00 broadcast 3.3.3.255 > media: Ethernet autoselect > status: no carrier > > >I then ping something that would be across the wire while the nic is >down. e.g. ping 3.3.3.1 > >I then plug in the cable so that the other side has 3.3.3.1 > >ifconfig shows all looks good > ># ifconfig em5 >em5: flags=8843 metric 0 mtu 1500 > options=19b > ether 00:30:48:d6:ef:13 > inet 3.3.3.3 netmask 0xffffff00 broadcast 3.3.3.255 > media: Ethernet autoselect (1000baseTX ) > status: active > >I try and ping 3.3.3.1 which is on xover (via a switch shows the >same behaviour), and no response to the pings.... BUT, I do see the >MAC addr show up ># ping -c 2 -S 3.3.3.3 3.3.3.1 >PING 3.3.3.1 (3.3.3.1) from 3.3.3.3: 56 data bytes > >--- 3.3.3.1 ping statistics --- >2 packets transmitted, 0 packets received, 100.0% packet loss ># arp -na >? (3.3.3.1) at 00:30:48:94:88:20 on em5 [ethernet] >? (3.3.3.3) at 00:30:48:d6:ef:13 on em5 permanent [ethernet] > >I can see its mac addr ?!? > >Furthermore, if I do > ># ifconfig em5 3.3.3.55/32 alias > >On the other side, I see > >0(ich10)# tcpdump -nei igb0 >tcpdump: verbose output suppressed, use -v or -vv for full protocol decode >listening on igb0, link-type EN10MB (Ethernet), capture size 96 bytes >16:16:03.380886 00:30:48:d6:ef:13 > ff:ff:ff:ff:ff:ff, ethertype ARP >(0x0806), length 60: Request who-has 3.3.3.55 tell 3.3.3.55, length 46 > > >and I can ping if I specify the alias as the source IP > ># ping -S 3.3.3.55 3.3.3.1 >PING 3.3.3.1 (3.3.3.1) from 3.3.3.55: 56 data bytes >64 bytes from 3.3.3.1: icmp_seq=0 ttl=64 time=0.184 ms >64 bytes from 3.3.3.1: icmp_seq=1 ttl=64 time=0.051 ms >64 bytes from 3.3.3.1: icmp_seq=2 ttl=64 time=0.055 ms > > > >16:17:01.603345 00:30:48:d6:ef:13 > 00:30:48:94:88:20, ethertype ARP >(0x0806), length 60: Reply 3.3.3.55 is-at 00:30:48:d6:ef:13, length 46 >16:17:01.603349 00:30:48:94:88:20 > 00:30:48:d6:ef:13, ethertype >IPv4 (0x0800), length 98: 3.3.3.1 > 3.3.3.55: ICMP >echo reply, id 7946, seq 0, length 64 >16:17:02.603497 00:30:48:d6:ef:13 > 00:30:48:94:88:20, ethertype >IPv4 (0x0800), length 98: 3.3.3.55 > 3.3.3.1: ICMP >echo request, id 7946, seq 1, length 64 >16:17:02.603502 00:30:48:94:88:20 > 00:30:48:d6:ef:13, ethertype >IPv4 (0x0800), length 98: 3.3.3.1 > 3.3.3.55: ICMP >echo reply, id 7946, seq 1, length 64 >16:17:03.604510 00:30:48:d6:ef:13 > 00:30:48:94:88:20, ethertype >IPv4 (0x0800), length 98: 3.3.3.55 > 3.3.3.1: ICMP >echo request, id 7946, seq 2, length 64 >16:17:03.604516 00:30:48:94:88:20 > 00:30:48:d6:ef:13, ethertype >IPv4 (0x0800), length 98: 3.3.3.1 > 3.3.3.55: ICMP >echo reply, id 7946, seq 2, length 64 > > > >but not using the initial IP addr > >0[iolite3A]# ping -S 3.3.3.3 3.3.3.1 >PING 3.3.3.1 (3.3.3.1) from 3.3.3.3: 56 data bytes >^C >--- 3.3.3.1 ping statistics --- >2 packets transmitted, 0 packets received, 100.0% packet loss ># > >Yet, > ># ping -S 3.3.3.3 3.3.3.1 >PING 3.3.3.1 (3.3.3.1) from 3.3.3.3: 56 data bytes >^C >--- 3.3.3.1 ping statistics --- >2 packets transmitted, 0 packets received, 100.0% packet loss ># ping -S 3.3.3.4 3.3.3.1 >PING 3.3.3.1 (3.3.3.1) from 3.3.3.4: 56 data bytes >64 bytes from 3.3.3.1: icmp_seq=0 ttl=64 time=0.176 ms >64 bytes from 3.3.3.1: icmp_seq=1 ttl=64 time=0.050 ms >^C >--- 3.3.3.1 ping statistics --- >2 packets transmitted, 2 packets received, 0.0% packet loss >round-trip min/avg/max/stddev = 0.050/0.113/0.176/0.063 ms > > >Strange, eh ? > > >em4@pci0:6:0:0: class=0x020000 card=0x10d315d9 chip=0x10d38086 >rev=0x00 hdr=0x00 > vendor = 'Intel Corporation' > class = network > subclass = ethernet > cap 01[c8] = powerspec 2 supports D0 D3 current D0 > cap 05[d0] = MSI supports 1 message, 64 bit > cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) > cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled >em5@pci0:7:0:0: class=0x020000 card=0x10d315d9 chip=0x10d38086 >rev=0x00 hdr=0x00 > vendor = 'Intel Corporation' > class = network > subclass = ethernet > cap 01[c8] = powerspec 2 supports D0 D3 current D0 > cap 05[d0] = MSI supports 1 message, 64 bit > cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) > cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled > > >em4: port 0xdc00-0xdc1f >mem 0xfaee0000-0xfaefffff,0xfaedc000-0xfaedffff irq 16 at device 0.0 on pci6 >em4: Using MSIX interrupts >em4: [ITHREAD] >em4: [ITHREAD] >em4: [ITHREAD] >em4: Ethernet address: 00:30:48:d6:ef:12 >pcib7: irq 16 at device 28.1 on pci0 >pci7: on pcib7 >em5: port 0xec00-0xec1f >mem 0xfafe0000-0xfaffffff,0xfafdc000-0xfafdffff irq 17 at device 0.0 on pci7 >em5: Using MSIX interrupts >em5: [ITHREAD] >em5: [ITHREAD] >em5: [ITHREAD] >em5: Ethernet address: 00:30:48:d6:ef:13 > >The same does NOT happen with my external 2 port pcie nics (it says >HP, but they are intel branded) >eg >em0@pci0:1:0:0: class=0x020000 card=0x115e8086 chip=0x105e8086 >rev=0x06 hdr=0x00 > vendor = 'Intel Corporation' > device = 'HP NC360T PCIe DP Gigabit Server Adapter (n1e5132)' > class = network > subclass = ethernet > cap 01[c8] = powerspec 2 supports D0 D3 current D0 > cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message > cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x4(x4) >em1@pci0:1:0:1: class=0x020000 card=0x115e8086 chip=0x105e8086 >rev=0x06 hdr=0x00 > vendor = 'Intel Corporation' > device = 'HP NC360T PCIe DP Gigabit Server Adapter (n1e5132)' > class = network > subclass = ethernet > cap 01[c8] = powerspec 2 supports D0 D3 current D0 > cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message > cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x4(x4) > > ---Mike > >-------------------------------------------------------------------- >Mike Tancsa, tel +1 519 651 3400 >Sentex >Communications, >mike@sentex.net >Providing Internet since >1994 www.sentex.net >Cambridge, Ontario >Canada www.sentex.net/mike > -------------------------------------------------------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet since 1994 www.sentex.net Cambridge, Ontario Canada www.sentex.net/mike