Date: Thu, 20 Jul 2006 11:49:23 -0400 (EDT) From: "Brian A. Seklecki" <lavalamp@spiritual-machines.org> To: Peter Ross <Peter.Ross@alumni.tu-berlin.de> Cc: Geoffrey Giesemann <geoffwa@idkfa.ath.cx>, freebsd-current@freebsd.org, Gleb Smirnoff <glebius@FreeBSD.org>, wmoran@collaborativefusion.com, David Barbero <sico@loquefaltaba.com> Subject: Dell PowerEdge 850 bge(4) RELENG_6 (WAS: Re: bge(4) problem) Message-ID: <20060720104238.L8726@arbitor.digitalfreaks.org> In-Reply-To: <43767.150.101.159.26.1140420612.squirrel@mailbox.TU-Berlin.DE> References: <43767.150.101.159.26.1140420612.squirrel@mailbox.TU-Berlin.DE>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, 20 Feb 2006, Peter Ross wrote:
> I installed a IBM x336, a 1U server.
>
> it came with two bge interfaces built-in:
> (from dmesg) <Broadcom BCM5721 Gigabit Ethernet, ASIC rev. 0x4101>
>
[...]
> which are working fine.
>
> But I have a problem with two dual port Broadcom cards plugged in into
> this box:
> <Broadcom BCM5704C Dual Gigabit Ethernet, ASIC rev. 0x2100>
>
> I cannot connect them to the 1000MBit switch (a Dell Powerswitch,
I have a Dell PowerEdge 850 (Generation 3 or 4, hard to tell the way Dell
rolls things out). The NICs onboard are:
bge0: <Broadcom BCM5721 Gigabit Ethernet, ASIC rev. 0x4101> mem
0xdf9f0000-0xdf9fffff irq 16 at device 0.0 on pci5
bge1: <Broadcom BCM5721 Gigabit Ethernet, ASIC rev. 0x4101> mem
0xdf7f0000-0xdf7fffff irq 17 at device 0.0 on pci6
With the MFC pull up of if_bge.c from 1.24 -> 1.91.2.13 (RELENG_6,
RELENG_6_1) compiled in, the link speed negotation / interface link state
change problems you describe on this platform persist.
Unforunately, I have a RELENG_5_3 box i haven't tried to backport the
changes to (I had to hack PCIIDs from -rHEAD just to get 5.3 to recognize
this chipset), but it doesn't exhibit this problem.
BTW, the problem persists on Dell PowerConnect switches and Cisco
Catalyst 3550-48T SMIs.
I think this is a driver problem, because, at least on my system, I'm able
to run "tcpdump -n -i bge0 -ttt -e" and see incoming packets making it
into the NIC cards input buffer. The destination MAC address is the same,
and the packets have the SYN flag set, but no TCP socket is ever opened,
so it's possible the data is being dropped in the handoff from the layer2
driver to the upper-layer tcp(4) or ip(4) layers:
This is:
# uname -a
FreeBSD tantrum.pitbpa1.priv.collaborativefusion.com 6.1-RELEASE-p3
FreeBSD 6.1-RELEASE-p3 #0: Thu Jul 20 10:31:01 EDT 2006
i386
# ident /boot/kernel/kernel |grep -i bge
$FreeBSD: src/sys/dev/bge/if_bge.c,v 1.91.2.13 2006/03/04 09:34:48 oleg Exp $
# ifconfig bge1
bge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=1b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING>
inet6 fe80::213:72ff:fe3b:d8bb%bge1 prefixlen 64 scopeid 0x2
inet 206.210.72.75 netmask 0xffffffc0 broadcast 206.210.72.127
ether 00:13:72:3b:d8:bb
media: Ethernet 100baseTX <full-duplex>
status: active
# tcpdump -n -i bge1 -ttt -e "port 22"
listening on bge1, link-type EN10MB (Ethernet), capture size 96 bytes
000000 00:05:dd:c0:9f:00 > 00:13:72:3b:d8:bb, ethertype IPv4 (0x0800),
length 62: 206.210.89.202.55952 > 206.210.72.75.22: S
4262123753:4262123753(0) win 65535 <mss 1460,sackOK,eol>
*** But: ****
# netstat -tan|grep -i 22
tcp4 0 32 192.168.126.138.22 192.168.2.50.49175
ESTABLISHED
tcp4 0 0 *.22 *.* LISTEN
tcp6 0 0 *.22 *.* LISTEN
There are no errors on the interfaces:
[root@tantrum ~]# netstat -i
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs
Coll
bge0 1500 <Link#1> 00:13:72:3b:d8:ba 1060 0 268 0
0
bge0 1500 fe80:1::213:7 fe80:1::213:72ff: 0 - 4 -
-
bge0 1500 192.168.126.1 tantrum-v14.corp. 469 - 449 -
-
bge1 1500 <Link#2> 00:13:72:3b:d8:bb 175 0 0 0
0
bge1 1500 fe80:2::213:7 fe80:2::213:72ff: 0 - 4 -
-
bge1 1500 206.210.72.64 unallocated.pitbp 24 - 38 -
-
Interesting things in "netstat -s":
5 connection accepts
This number does not increment on these syn packets.
However the following stats do update on connection attempts inbound:
11 syncache entries added
14 retransmitted
12 dupsyn
69 dropped
6 completed
0 bucket overflow
0 cache overflow
0 reset
4 stale
0 aborted
0 badack
0 unreach
0 zone failures
The following also increment when a TCP socket does not open as expected:
# netstat -s|egrep -i "packets for this host|total packets"
ipv4:
2078 total packets received
1800 packets for this host
0 total packets received
0 packets for this host
So TCP seems to be affected, ipv4 does not. That may explain part of the
problem.
Cisco switch interface:
as0#sh int fa0/10
FastEthernet0/10 is down, line protocol is down (notconnect)
Hardware is Fast Ethernet, address is 0013.7fd2.b88a (bia
0013.7fd2.b88a)
Description: Tantrum Mgmnt
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:25, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
250653 packets input, 35479549 bytes, 0 no buffer
Received 36 broadcasts (0 multicast)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 10 multicast, 0 pause input
0 input packets with dribble condition detected
1753610 packets output, 260154228 bytes, 0 underruns
0 output errors, 0 collisions, 3 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
----
So in conclusion:
1) Manually ifconfig down, ifconfig up'ing the interface post-boot
resolves the problem
2) The Cisco Catalyst switch shows no errors on it's interface
3) The FBSD Interface counters show no errors
4) The kernel seems to see IPV4 packets coming in from the L2 driver
bge(4)
5) The TCP/UDP code never seems to recieve the packets
6) Disabling IPMI in the Dell onboard BMC does not resolve the issue
7) Auto negotiate instead of forced full-duplex on both the switch/system
does not solve the problem. It causes the system to negotiate up at
100mbs/half-duplex (which is technically impossible), plus Cisco TAC
does not support autonegotate configurations.
8) There are no packet filters installed or enabled
9) This is the latest RELENG_6_1 with the bge(4) patch
The solution seems to be to simply not use bge(4) hardware. Use Intel
instead. Unfortunately, Dell seems to have therr heart set on replacing
OEM Intel integrated chipsets with BGE in the new 1950 and 2950s.
Not good.
~BAS
PS. Also, I just wanted to say that this kind of dry humor wasn't lost on
anyone here and much appreciated. >:}
>> Are the NICs a copper ones? Do they work properly against each other,
>> or against any other equipment? Since you are connecting them to
>> a switch that is 1000 km away, I suppose you are using some media
>> converters; what equipment is the NIC connected to?
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20060720104238.L8726>