Date: Tue, 28 Aug 2018 11:35:12 +0200 From: Marius Halden <marius.h@lden.org> To: Navdeep Parhar <np@FreeBSD.org> Cc: freebsd-net@freebsd.org Subject: Re: cxl nic not working after reboot Message-ID: <1535448912.3229649.1488521184.1EE83C33@webmail.messagingengine.com> In-Reply-To: <1535395530.234212.1487804896.64257978@webmail.messagingengine.com> References: <1535379569.2132387.1487489784.45227861@webmail.messagingengine.com> <20180827151302.GA10167@ox> <1535395530.234212.1487804896.64257978@webmail.messagingengine.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Aug 27, 2018, at 20:45, Marius Halden wrote:
> On Mon, Aug 27, 2018, at 17:13, Navdeep Parhar wrote:
> > On Mon, Aug 27, 2018 at 04:19:29PM +0200, Marius Halden wrote:
> > > Hi,
> > >=20
> > > We have some routers with Chelsio T540-CR NICs using 1Gbps SFPs (1000
> > > Base-LX IIRC) to connect to our ISPs. After upgrading them to FreeBSD
> > > 11.2-p2 (BSDRP v1.91) we have run into some issues. When the machines
> > > are rebooted networking does not function at all through the ports
> > > connected with SFPs, ports connected with DAC cables work properly.
> > > According to ifconfig the SFPs are detected properly and they both
> >=20
> > What exact ifconfig command do you use to bring the interfaces up?
>=20
> It's just the normal rc.conf lines with `ifconfig_cxl0=3D"inet =E2=80=A6"=
` and=20
> `ifconfig_cxl0_ipv6=3D"inet6 =E2=80=A6"`, nothing special beyond that.
>=20
> > Provide the output of these commands when the link isn't up:
> > # ifconfig -mvvv <interface>
> > # sysctl -n dev.t5nex.0.misc.devlog
Output from these commands follows. (Ip addresses has been changed.)
--=20
Marius Halden
# ifconfig -mvvv cxl0
cxl0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=3De800bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_H=
WCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
capabilities=3Decc7bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,V=
LAN_HWCSUM,TSO4,TSO6,LRO,TOE4,TOE6,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_=
IPV6>
ether 00:07:43:42:cb:30
hwaddr 00:07:43:42:cb:30
inet 1.2.3.4 netmask 0xfffffffc broadcast 1.2.3.4
inet6 fe80::207:43ff:fe42:cb30%cxl0 prefixlen 64 scopeid 0x5
inet6 a:b:c:d::e prefixlen 126
nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet 1000baseSX <full-duplex,rxpause,txpause>
status: active
supported media:
media 1000baseSX mediaopt full-duplex,rxpause,txpause
media 1000baseSX mediaopt full-duplex,rxpause
media 1000baseSX mediaopt full-duplex,txpause
media 1000baseSX mediaopt full-duplex
plugged: SFP/SFP+/SFP28 1000BASE-LX (LC)
vendor: Fiberworks PN: SFP-L80D-C49-U SN: FBSW491434 DATE: 2017-09-27
module temperature: 40.41 C Voltage: 3.26 Volts
RX: 0.28 mW (-5.50 dBm) TX: 1.62 mW (2.10 dBm)
SFF8472 DUMP (0xA0 0..127 range):
03 04 07 00 00 00 02 10 10 10 01 01 0D 00 50 FF
00 00 00 00 46 69 62 65 72 77 6F 72 6B 73 20 20
20 20 20 20 00 8A 34 BC 53 46 50 2D 4C 38 30 44
2D 43 34 39 2D 55 20 20 31 2E 30 20 05 D2 00 29
00 1A 00 00 46 42 53 57 34 39 31 34 33 34 20 20
20 20 20 20 31 37 30 39 32 37 20 20 68 90 01 B8
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
# sysctl -n dev.t5nex.0.misc.devlog
Seq# Tstamp Level Facility Message
15 893704 INFO PORT module[0]: gpio 11 vendor =
id 8a34bc, identifier 0x03, SFP28(byte 36/192) 0xff, SFP(byte 3/131) 0x00, =
1G (byte 6) 0x02
16 893706 INFO PORT optical length(byte 15/142=
) 255, copper cable(byte 8/147) 0xff, length(byte 18/146) 0, module_type 0x=
02
17 893708 INFO PORT hw_mac_init_port[0], ptype=
0x9, speed 0x2, lanes 0x1, fec 0x0
18 893748 INFO TM pktsched channel 0 sets sp=
eed (from 0) to 1000000 kbps
19 893763 INFO PORT hw_mac_init_port[1], ptype=
0x9, speed 0x4, lanes 0x2, fec 0x0
20 893795 INFO TM pktsched channel 1 sets sp=
eed (from 0) to 10000000 kbps
21 893807 INFO PORT hw_mac_init_port[2], ptype=
0x9, speed 0x4, lanes 0x4, fec 0x0
22 893840 INFO TM pktsched channel 2 sets sp=
eed (from 0) to 10000000 kbps
23 897131 INFO PORT module[3]: gpio 7 vendor i=
d 78a714, identifier 0x03, SFP28(byte 36/192) 0xff, SFP(byte 3/131) 0x00, 1=
G (byte 6) 0x00
24 897132 INFO PORT optical length(byte 15/142=
) 0, copper cable(byte 8/147) 0x04, length(byte 18/146) 3, module_type 0x04
25 897133 INFO PORT hw_mac_init_port[3], ptype=
0x9, speed 0x4, lanes 0x8, fec 0x0
26 897165 INFO TM pktsched channel 3 sets sp=
eed (from 0) to 10000000 kbps
27 921140 INFO TM pktsched_cl_rl[0:0]: mode =
| unit | rate 0x010001 min 0 max 100000 pktsize 1500
28 922146 INFO TM pktsched_cl_rl[0:1]: mode =
| unit | rate 0x010001 min 0 max 200000 pktsize 1500
29 923161 INFO TM pktsched_cl_rl[0:2]: mode =
| unit | rate 0x010001 min 0 max 400000 pktsize 1500
30 924167 INFO TM pktsched_cl_rl[0:3]: mode =
| unit | rate 0x010001 min 0 max 500000 pktsize 1500
31 925176 INFO TM pktsched_cl_rl[0:4]: mode =
| unit | rate 0x010001 min 0 max 800000 pktsize 1500
32 926192 INFO TM pktsched_cl_rl[0:5]: mode =
| unit | rate 0x010001 min 0 max 1000000 pktsize 1500
33 927208 INFO TM pktsched_cl_rl[0:6]: mode =
| unit | rate 0x010001 min 0 max 1200000 pktsize 1500
34 927209 NOTICE TM ch_cl_rate[0/6]: capped de=
ficit_incr from required 12000 to 10000; rate 1200000 (eff 1000000) deficit=
_max 6144
35 928230 INFO TM pktsched_cl_rl[0:7]: mode =
| unit | rate 0x010001 min 0 max 1500000 pktsize 1500
36 928231 NOTICE TM ch_cl_rate[0/7]: capped de=
ficit_incr from required 15000 to 10000; rate 1500000 (eff 1000000) deficit=
_max 6144
37 929236 INFO TM pktsched_cl_rl[0:8]: mode =
| unit | rate 0x010001 min 0 max 1800000 pktsize 1500
38 929237 NOTICE TM ch_cl_rate[0/8]: capped de=
ficit_incr from required 18000 to 10000; rate 1800000 (eff 1000000) deficit=
_max 6144
39 930246 INFO TM pktsched_cl_rl[0:9]: mode =
| unit | rate 0x010001 min 0 max 2000000 pktsize 1500
40 930246 NOTICE TM ch_cl_rate[0/9]: capped de=
ficit_incr from required 20000 to 10000; rate 2000000 (eff 1000000) deficit=
_max 6144
41 931254 INFO TM pktsched_cl_rl[0:10]: mode=
| unit | rate 0x010001 min 0 max 2500000 pktsize 1500
42 931255 NOTICE TM ch_cl_rate[0/10]: capped d=
eficit_incr from required 25000 to 10000; rate 2500000 (eff 1000000) defici=
t_max 6144
43 932261 INFO TM pktsched_cl_rl[0:11]: mode=
| unit | rate 0x010001 min 0 max 3000000 pktsize 1500
44 932261 NOTICE TM ch_cl_rate[0/11]: capped d=
eficit_incr from required 30000 to 10000; rate 3000000 (eff 1000000) defici=
t_max 6144
45 933268 INFO TM pktsched_cl_rl[0:12]: mode=
| unit | rate 0x010001 min 0 max 3500000 pktsize 1500
46 933269 NOTICE TM ch_cl_rate[0/12]: capped d=
eficit_incr from required 35000 to 10000; rate 3500000 (eff 1000000) defici=
t_max 6144
47 934275 INFO TM pktsched_cl_rl[0:13]: mode=
| unit | rate 0x010001 min 0 max 4000000 pktsize 1500
48 934276 NOTICE TM ch_cl_rate[0/13]: capped d=
eficit_incr from required 40000 to 10000; rate 4000000 (eff 1000000) defici=
t_max 6144
49 935284 INFO TM pktsched_cl_rl[0:14]: mode=
| unit | rate 0x010001 min 0 max 5000000 pktsize 1500
50 935284 NOTICE TM ch_cl_rate[0/14]: capped d=
eficit_incr from required 50000 to 10000; rate 5000000 (eff 1000000) defici=
t_max 6144
51 936290 INFO TM pktsched_cl_rl[0:15]: mode=
| unit | rate 0x010001 min 0 max 10000000 pktsize 1500
52 936291 NOTICE TM ch_cl_rate[0/15]: capped d=
eficit_incr from required 100000 to 10000; rate 10000000 (eff 1000000) defi=
cit_max 6144
53 937311 INFO TM pktsched_cl_rl[1:0]: mode =
| unit | rate 0x010001 min 0 max 100000 pktsize 1500
54 938318 INFO TM pktsched_cl_rl[1:1]: mode =
| unit | rate 0x010001 min 0 max 200000 pktsize 1500
55 939334 INFO TM pktsched_cl_rl[1:2]: mode =
| unit | rate 0x010001 min 0 max 400000 pktsize 1500
56 940349 INFO TM pktsched_cl_rl[1:3]: mode =
| unit | rate 0x010001 min 0 max 500000 pktsize 1500
57 941367 INFO TM pktsched_cl_rl[1:4]: mode =
| unit | rate 0x010001 min 0 max 800000 pktsize 1500
58 942379 INFO TM pktsched_cl_rl[1:5]: mode =
| unit | rate 0x010001 min 0 max 1000000 pktsize 1500
59 943386 INFO TM pktsched_cl_rl[1:6]: mode =
| unit | rate 0x010001 min 0 max 1200000 pktsize 1500
60 943387 NOTICE TM ch_cl_rate[1/6]: capped de=
ficit_incr from required 12000 to 10000; rate 1200000 (eff 1000000) deficit=
_max 6144
61 944393 INFO TM pktsched_cl_rl[1:7]: mode =
| unit | rate 0x010001 min 0 max 1500000 pktsize 1500
62 944394 NOTICE TM ch_cl_rate[1/7]: capped de=
ficit_incr from required 15000 to 10000; rate 1500000 (eff 1000000) deficit=
_max 6144
63 945419 INFO TM pktsched_cl_rl[1:8]: mode =
| unit | rate 0x010001 min 0 max 1800000 pktsize 1500
64 945420 NOTICE TM ch_cl_rate[1/8]: capped de=
ficit_incr from required 18000 to 10000; rate 1800000 (eff 1000000) deficit=
_max 6144
65 946430 INFO TM pktsched_cl_rl[1:9]: mode =
| unit | rate 0x010001 min 0 max 2000000 pktsize 1500
66 946431 NOTICE TM ch_cl_rate[1/9]: capped de=
ficit_incr from required 20000 to 10000; rate 2000000 (eff 1000000) deficit=
_max 6144
67 947458 INFO TM pktsched_cl_rl[1:10]: mode=
| unit | rate 0x010001 min 0 max 2500000 pktsize 1500
68 947459 NOTICE TM ch_cl_rate[1/10]: capped d=
eficit_incr from required 25000 to 10000; rate 2500000 (eff 1000000) defici=
t_max 6144
69 948470 INFO TM pktsched_cl_rl[1:11]: mode=
| unit | rate 0x010001 min 0 max 3000000 pktsize 1500
70 948470 NOTICE TM ch_cl_rate[1/11]: capped d=
eficit_incr from required 30000 to 10000; rate 3000000 (eff 1000000) defici=
t_max 6144
71 949492 INFO TM pktsched_cl_rl[1:12]: mode=
| unit | rate 0x010001 min 0 max 3500000 pktsize 1500
72 949493 NOTICE TM ch_cl_rate[1/12]: capped d=
eficit_incr from required 35000 to 10000; rate 3500000 (eff 1000000) defici=
t_max 6144
73 950499 INFO TM pktsched_cl_rl[1:13]: mode=
| unit | rate 0x010001 min 0 max 4000000 pktsize 1500
74 950500 NOTICE TM ch_cl_rate[1/13]: capped d=
eficit_incr from required 40000 to 10000; rate 4000000 (eff 1000000) defici=
t_max 6144
75 951503 INFO TM pktsched_cl_rl[1:14]: mode=
| unit | rate 0x010001 min 0 max 5000000 pktsize 1500
76 951504 NOTICE TM ch_cl_rate[1/14]: capped d=
eficit_incr from required 50000 to 10000; rate 5000000 (eff 1000000) defici=
t_max 6144
77 952510 INFO TM pktsched_cl_rl[1:15]: mode=
| unit | rate 0x010001 min 0 max 10000000 pktsize 1500
78 952511 NOTICE TM ch_cl_rate[1/15]: capped d=
eficit_incr from required 100000 to 10000; rate 10000000 (eff 1000000) defi=
cit_max 6144
79 953519 INFO TM pktsched_cl_rl[2:0]: mode =
| unit | rate 0x010001 min 0 max 100000 pktsize 1500
80 954526 INFO TM pktsched_cl_rl[2:1]: mode =
| unit | rate 0x010001 min 0 max 200000 pktsize 1500
81 955534 INFO TM pktsched_cl_rl[2:2]: mode =
| unit | rate 0x010001 min 0 max 400000 pktsize 1500
82 956541 INFO TM pktsched_cl_rl[2:3]: mode =
| unit | rate 0x010001 min 0 max 500000 pktsize 1500
83 957545 INFO TM pktsched_cl_rl[2:4]: mode =
| unit | rate 0x010001 min 0 max 800000 pktsize 1500
84 958553 INFO TM pktsched_cl_rl[2:5]: mode =
| unit | rate 0x010001 min 0 max 1000000 pktsize 1500
85 959560 INFO TM pktsched_cl_rl[2:6]: mode =
| unit | rate 0x010001 min 0 max 1200000 pktsize 1500
86 959561 NOTICE TM ch_cl_rate[2/6]: capped de=
ficit_incr from required 12000 to 10000; rate 1200000 (eff 1000000) deficit=
_max 6144
87 960567 INFO TM pktsched_cl_rl[2:7]: mode =
| unit | rate 0x010001 min 0 max 1500000 pktsize 1500
88 960568 NOTICE TM ch_cl_rate[2/7]: capped de=
ficit_incr from required 15000 to 10000; rate 1500000 (eff 1000000) deficit=
_max 6144
89 961581 INFO TM pktsched_cl_rl[2:8]: mode =
| unit | rate 0x010001 min 0 max 1800000 pktsize 1500
90 961582 NOTICE TM ch_cl_rate[2/8]: capped de=
ficit_incr from required 18000 to 10000; rate 1800000 (eff 1000000) deficit=
_max 6144
91 962591 INFO TM pktsched_cl_rl[2:9]: mode =
| unit | rate 0x010001 min 0 max 2000000 pktsize 1500
92 962592 NOTICE TM ch_cl_rate[2/9]: capped de=
ficit_incr from required 20000 to 10000; rate 2000000 (eff 1000000) deficit=
_max 6144
93 963613 INFO TM pktsched_cl_rl[2:10]: mode=
| unit | rate 0x010001 min 0 max 2500000 pktsize 1500
94 963613 NOTICE TM ch_cl_rate[2/10]: capped d=
eficit_incr from required 25000 to 10000; rate 2500000 (eff 1000000) defici=
t_max 6144
95 964618 INFO TM pktsched_cl_rl[2:11]: mode=
| unit | rate 0x010001 min 0 max 3000000 pktsize 1500
96 964619 NOTICE TM ch_cl_rate[2/11]: capped d=
eficit_incr from required 30000 to 10000; rate 3000000 (eff 1000000) defici=
t_max 6144
97 965626 INFO TM pktsched_cl_rl[2:12]: mode=
| unit | rate 0x010001 min 0 max 3500000 pktsize 1500
98 965627 NOTICE TM ch_cl_rate[2/12]: capped d=
eficit_incr from required 35000 to 10000; rate 3500000 (eff 1000000) defici=
t_max 6144
99 966632 INFO TM pktsched_cl_rl[2:13]: mode=
| unit | rate 0x010001 min 0 max 4000000 pktsize 1500
100 966633 NOTICE TM ch_cl_rate[2/13]: capped d=
eficit_incr from required 40000 to 10000; rate 4000000 (eff 1000000) defici=
t_max 6144
101 967637 INFO TM pktsched_cl_rl[2:14]: mode=
| unit | rate 0x010001 min 0 max 5000000 pktsize 1500
102 967638 NOTICE TM ch_cl_rate[2/14]: capped d=
eficit_incr from required 50000 to 10000; rate 5000000 (eff 1000000) defici=
t_max 6144
103 968643 INFO TM pktsched_cl_rl[2:15]: mode=
| unit | rate 0x010001 min 0 max 10000000 pktsize 1500
104 968644 NOTICE TM ch_cl_rate[2/15]: capped d=
eficit_incr from required 100000 to 10000; rate 10000000 (eff 1000000) defi=
cit_max 6144
105 969651 INFO TM pktsched_cl_rl[3:0]: mode =
| unit | rate 0x010001 min 0 max 100000 pktsize 1500
106 970656 INFO TM pktsched_cl_rl[3:1]: mode =
| unit | rate 0x010001 min 0 max 200000 pktsize 1500
107 971662 INFO TM pktsched_cl_rl[3:2]: mode =
| unit | rate 0x010001 min 0 max 400000 pktsize 1500
108 972676 INFO TM pktsched_cl_rl[3:3]: mode =
| unit | rate 0x010001 min 0 max 500000 pktsize 1500
109 973683 INFO TM pktsched_cl_rl[3:4]: mode =
| unit | rate 0x010001 min 0 max 800000 pktsize 1500
110 974696 INFO TM pktsched_cl_rl[3:5]: mode =
| unit | rate 0x010001 min 0 max 1000000 pktsize 1500
111 975701 INFO TM pktsched_cl_rl[3:6]: mode =
| unit | rate 0x010001 min 0 max 1200000 pktsize 1500
112 975702 NOTICE TM ch_cl_rate[3/6]: capped de=
ficit_incr from required 12000 to 10000; rate 1200000 (eff 1000000) deficit=
_max 6144
113 976718 INFO TM pktsched_cl_rl[3:7]: mode =
| unit | rate 0x010001 min 0 max 1500000 pktsize 1500
114 976719 NOTICE TM ch_cl_rate[3/7]: capped de=
ficit_incr from required 15000 to 10000; rate 1500000 (eff 1000000) deficit=
_max 6144
115 977724 INFO TM pktsched_cl_rl[3:8]: mode =
| unit | rate 0x010001 min 0 max 1800000 pktsize 1500
116 977724 NOTICE TM ch_cl_rate[3/8]: capped de=
ficit_incr from required 18000 to 10000; rate 1800000 (eff 1000000) deficit=
_max 6144
117 978729 INFO TM pktsched_cl_rl[3:9]: mode =
| unit | rate 0x010001 min 0 max 2000000 pktsize 1500
118 978730 NOTICE TM ch_cl_rate[3/9]: capped de=
ficit_incr from required 20000 to 10000; rate 2000000 (eff 1000000) deficit=
_max 6144
119 979741 INFO TM pktsched_cl_rl[3:10]: mode=
| unit | rate 0x010001 min 0 max 2500000 pktsize 1500
120 979742 NOTICE TM ch_cl_rate[3/10]: capped d=
eficit_incr from required 25000 to 10000; rate 2500000 (eff 1000000) defici=
t_max 6144
121 980748 INFO TM pktsched_cl_rl[3:11]: mode=
| unit | rate 0x010001 min 0 max 3000000 pktsize 1500
122 980749 NOTICE TM ch_cl_rate[3/11]: capped d=
eficit_incr from required 30000 to 10000; rate 3000000 (eff 1000000) defici=
t_max 6144
123 981752 INFO TM pktsched_cl_rl[3:12]: mode=
| unit | rate 0x010001 min 0 max 3500000 pktsize 1500
124 981753 NOTICE TM ch_cl_rate[3/12]: capped d=
eficit_incr from required 35000 to 10000; rate 3500000 (eff 1000000) defici=
t_max 6144
125 982757 INFO TM pktsched_cl_rl[3:13]: mode=
| unit | rate 0x010001 min 0 max 4000000 pktsize 1500
126 982758 NOTICE TM ch_cl_rate[3/13]: capped d=
eficit_incr from required 40000 to 10000; rate 4000000 (eff 1000000) defici=
t_max 6144
127 983764 INFO TM pktsched_cl_rl[3:14]: mode=
| unit | rate 0x010001 min 0 max 5000000 pktsize 1500
128 983765 NOTICE TM ch_cl_rate[3/14]: capped d=
eficit_incr from required 50000 to 10000; rate 5000000 (eff 1000000) defici=
t_max 6144
129 984777 INFO TM pktsched_cl_rl[3:15]: mode=
| unit | rate 0x010001 min 0 max 10000000 pktsize 1500
130 984778 NOTICE TM ch_cl_rate[3/15]: capped d=
eficit_incr from required 100000 to 10000; rate 10000000 (eff 1000000) defi=
cit_max 6144
131 4187455 INFO PORT port_link_state_handler[0]=
powering up
132 4187465 INFO PORT port[0] update (flowcid 37=
976 rc 0)
133 4387456 INFO PORT port_link_state_handler[3]=
powering up
134 4387461 INFO PORT port[3] update (flowcid 37=
976 rc 0)
135 4687456 INFO PORT port_hss_sigdet[3]: hss_si=
gdet changed to 0x8
136 4787455 INFO PORT hw_mac_link_status[3] int_=
cause 0x1001c70, link_status 0x12
137 4787459 INFO PORT port[0] link up (1) (speed=
0x2 acaps 0x70002 lpcaps 0x70000)
138 4787461 INFO PORT port[0] set PAUSE PARAMS: =
pppen 0 txpe 0x1 rxpe 0x1
139 4787466 INFO PORT port[0] update (flowcid 37=
976 rc 0)
140 4987456 INFO PORT port[3] link up (1) (speed=
0x4 acaps 0x70004 lpcaps 0x70000)
141 4987458 INFO PORT port[3] set PAUSE PARAMS: =
pppen 0 txpe 0x1 rxpe 0x1
142 4987461 INFO PORT port[3] update (flowcid 37=
976 rc 0)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1535448912.3229649.1488521184.1EE83C33>