Date: Tue, 4 Mar 2014 16:53:33 GMT From: Ray Abadie <rabadie@instawares.com> To: freebsd-gnats-submit@FreeBSD.org Subject: misc/187258: BCM57810 bxe(4) unstable/fails to initialize Message-ID: <201403041653.s24GrXpQ068732@cgiserv.freebsd.org> Resent-Message-ID: <201403041700.s24H00cH094549@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 187258
>Category: misc
>Synopsis: BCM57810 bxe(4) unstable/fails to initialize
>Confidential: no
>Severity: non-critical
>Priority: low
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Tue Mar 04 17:00:00 UTC 2014
>Closed-Date:
>Last-Modified:
>Originator: Ray Abadie
>Release: 10.0-RELEASE
>Organization:
TAG, Inc.
>Environment:
FreeBSD xxxxxx 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC 2014 root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
>Description:
Broadcom BCM957810A1008G dual-NIC card (BCM54810 chipset) fails to initialize consistently. On reboot both, either or none of the ports becomes active. The inactive ports show "no carrier" despite being connected to operational switch ports. Hardware is Dell R710. Have disabled TSO, RXCSUM and TXCSUM as per recommendations on other bxe(4) related bug reports to no avail. When ports do initialize they often fail and go to "no carrier" state inside of 1 hour. Traffic on link is minimal. Same card running fine in VMWare ESXi 4.1.
bxe1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=504b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,VLAN_HWFILTER,VLAN_HWTSO>
ether 00:10:18:e9:bd:82
inet 10.1.225.1 netmask 0xffffff00 broadcast 10.1.225.255
inet6 fe80::210:18ff:fee9:bd82%bxe1 prefixlen 64 scopeid 0x6
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (none)
status: no carrier
Enabled "Flying Monkeys" debug in bxe driver (via loader.conf hw.bxe.debug=0xffffffff and got this:
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=71)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=71)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4957) Received SIOCSIFMEDIA/SIOCGIFMEDI
A ioctl (cmd=56)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4957) Received SIOCSIFMEDIA/SIOCGIFMEDI
A ioctl (cmd=56)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=58)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=107)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=150)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4957) Received SIOCSIFMEDIA/SIOCGIFMEDI
A ioctl (cmd=56)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=235)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=248)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=123)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=140)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=145)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=143)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=59)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=71)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=71)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4957) Received SIOCSIFMEDIA/SIOCGIFMEDI
A ioctl (cmd=56)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4957) Received SIOCSIFMEDIA/SIOCGIFMEDI
A ioctl (cmd=56)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=58)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=107)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=150)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4957) Received SIOCSIFMEDIA/SIOCGIFMEDI
A ioctl (cmd=56)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=235)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=248)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=123)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=140)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=145)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=143)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=59)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4957) Received SIOCSIFMEDIA/SIOCGIFMEDI
A ioctl (cmd=56)
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_intr_sp(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:9193) ---> SP INTR <---
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_igu_ack_sb_gen(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.h:2374) write 0x02200000 to IGU
addr 0x442000
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_handle_sp_tq(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:8968) ---> SP TASK <---
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_handle_sp_tq(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:8973) dsb status 0x0001
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_handle_sp_tq(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:8977) ---> ATTN INTR <---
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_attn_int(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:8609) attn_bits 0x00000100 attn_ack
0x00000000 asserted 0x00000100 deasserted 0x00000000
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_attn_int_asserted(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:7413) aeu_mask 0x00000017 n
ewly asserted 0x00000100
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_attn_int_asserted(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:7417) new mask 0x00000017
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_attn_int_asserted(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:7423) attn_state 0x00000000
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_attn_int_asserted(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:7425) new state 0x00000100
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_stats_handle(/usr/src/sys/modules/bxe/../../dev/bxe/bxe_stats.c:1463) state 0 -> event 3 -
> state 0
Mar 4 09:21:02 asnas04 kernel: bxe0: ELINK: port 0, XGXS?0, int_status 0x0
Mar 4 09:21:02 asnas04 kernel: bxe0: ELINK: int_mask 0x0 MI_INT 1, SERDES_LINK 0
Mar 4 09:21:02 asnas04 kernel: bxe0: ELINK: 10G 0, XGXS_LINK 0
Mar 4 09:21:02 asnas04 kernel: bxe0: ELINK: BCM848xx: PMD_SIGNAL 1.a811 = 0x3eca
Mar 4 09:21:02 asnas04 kernel: bxe0: ELINK: BCM848x3: link speed is 10000
Mar 4 09:21:02 asnas04 kernel: bxe0: ELINK: Ext PHY pause result 0xf
Mar 4 09:21:02 asnas04 kernel: bxe0: ELINK: phy in index 1 link is up
Mar 4 09:21:02 asnas04 kernel: bxe0: ELINK: 0x81d1 = 0x0
Mar 4 09:21:02 asnas04 kernel: bxe0: ELINK: lane 2 gp_speed 0x202
Mar 4 09:21:02 asnas04 kernel: bxe0: ELINK: phy link down
Mar 4 09:21:02 asnas04 kernel: bxe0: ELINK: phy_link_up 0 line_speed 0
Mar 4 09:21:02 asnas04 kernel: bxe0: ELINK: duplex 1 flow_ctrl 0x400 link_status 0x20
Mar 4 09:21:02 asnas04 kernel: bxe0: ELINK: Active external phy selected: 1
Mar 4 09:21:02 asnas04 kernel: bxe0: ELINK: latch_status = 0x3c0001
Mar 4 09:21:02 asnas04 kernel: bxe0: ELINK: vars->flow_ctrl = 0x300, vars->link_status = 0x8cf660, ext_phy_line_speed = 10000
Mar 4 09:21:02 asnas04 kernel: bxe0: ELINK: ext_phy_link_up = 1, int_link_up = 0, init_preceding = 0
Mar 4 09:21:02 asnas04 kernel: bxe0: ELINK: Begin Warpcore init, link_speed 10000, serdes_net_if = 0x1000000
Mar 4 09:21:02 asnas04 kernel: bxe0: ELINK: Setting 10G XFI
Mar 4 09:21:02 asnas04 kernel: bxe0: ELINK: Configure WC for LPI pass through
Mar 4 09:21:02 asnas04 kernel: bxe0: ELINK: Exit config init
Mar 4 09:21:02 asnas04 kernel: bxe0: ELINK: Port 0: Link is down
Mar 4 09:21:02 asnas04 kernel: bxe0: ELINK: elink_set_led: port 0, mode 0
Mar 4 09:21:02 asnas04 kernel: bxe0: ELINK: speed 0x0, hw_led_mode 0x1
Mar 4 09:21:02 asnas04 kernel: bxe0: ELINK: Port 0x0: LED MODE OFF
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_read_mf_cfg(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:7262) mf_cfg function enabled
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_link_report_locked(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:12300) NIC Link is Down
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_attn_int_asserted(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:7498) about to mask 0x00000
100 at IGU addr 0x00442d08
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_igu_ack_sb_gen(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.h:2374) write 0x01400001 to IGU
addr 0x442000
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_intr_sp(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:9193) ---> SP INTR <---
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_igu_ack_sb_gen(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.h:2374) write 0x02200000 to IGU
addr 0x442000
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_handle_sp_tq(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:8968) ---> SP TASK <---
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_handle_sp_tq(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:8973) dsb status 0x0001
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_handle_sp_tq(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:8977) ---> ATTN INTR <---
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_attn_int(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:8609) attn_bits 0x00000000 attn_ack
0x00000100 asserted 0x00000000 deasserted 0x00000100
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_attn_int_deasserted(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:8536) attn: 0x00000000 0x
00000000 0x02180000 0x00000000 0x00000000
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_attn_int_deasserted(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:8568) about to mask 0xfff
ffeff at IGU addr 0x00442d10
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_attn_int_deasserted(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:8583) aeu_mask 0x00000017
newly deasserted 0x00000100
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_attn_int_deasserted(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:8585) new mask 0x00000117
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_attn_int_deasserted(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:8590) attn_state 0x000001
00
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_attn_int_deasserted(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:8592) new state 0x0000000
0
Mar 4 09:21:02 asnas04 kernel: bxe0: bxe_igu_ack_sb_gen(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.h:2374) write 0x01400002 to IGU
addr 0x442000
Mar 4 09:21:53 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=71)
Mar 4 09:21:53 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=71)
Mar 4 09:21:57 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4957) Received SIOCSIFMEDIA/SIOCGIFMEDI
A ioctl (cmd=56)
Mar 4 09:21:57 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4957) Received SIOCSIFMEDIA/SIOCGIFMEDI
A ioctl (cmd=56)
Mar 4 09:21:57 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=58)
Mar 4 09:21:57 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=107)
Mar 4 09:21:57 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=150)
Mar 4 09:21:57 asnas04 kernel: bxe1: bxe_rxeof(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:3332) fp[00] Rx hw_cq_cons=94 hw_sw_con
s=93 BD prod=91 cons=91 CQE typebxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4957) Received SIOCSIFMEDIA/SIOCGIFMEDI
A ioctl (cmd=56)
Mar 4 09:21:57 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=235)
Mar 4 09:21:57 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=248)
Mar 4 09:21:57 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=123)
Mar 4 09:21:57 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=140)
Mar 4 09:21:57 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=145)
Mar 4 09:21:57 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=143)
Mar 4 09:21:57 asnas04 kernel: bxe0: bxe_ioctl(/usr/src/sys/modules/bxe/../../dev/bxe/bxe.c:4993) Received Unknown Ioctl (cmd=59)
>How-To-Repeat:
Occurs every reboot, zero, one or both ports fail to initialize and when they successfully do initialize they fail inside of 1 hour.
>Fix:
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201403041653.s24GrXpQ068732>