Date: Thu, 30 Aug 2018 13:21:50 +0200 From: Marius Halden <marius.h@lden.org> To: freebsd-net@freebsd.org Subject: Re: cxl nic not working after reboot Message-ID: <1535628110.3034153.1491191792.12397BCF@webmail.messagingengine.com> In-Reply-To: <1535538153.459716.1489861176.29777B66@webmail.messagingengine.com> References: <1535379569.2132387.1487489784.45227861@webmail.messagingengine.com> <20180827151302.GA10167@ox> <1535395530.234212.1487804896.64257978@webmail.messagingengine.com> <1535448912.3229649.1488521184.1EE83C33@webmail.messagingengine.com> <20180829062843.GA25476@ox> <1535538153.459716.1489861176.29777B66@webmail.messagingengine.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Aug 29, 2018, at 12:22, Marius Halden wrote:
> On Wed, Aug 29, 2018, at 08:28, Navdeep Parhar wrote:
> > > > > Provide the output of these commands when the link isn't up:
> > > > > # ifconfig -mvvv <interface>
> > > > > # sysctl -n dev.t5nex.0.misc.devlog
> >=20
> > Can you provide the output from when the port is working?
>=20
> Sure, the output from when it works follows.
I tried to downgrade to a the previous bsdrp version we were running based =
on 11.1-RELEASE-p10, but it did not start working again. ifconfig output an=
d t5nex0 devlog follows (non-working state), but it all looks fine to me (m=
edia has always been reported wrong). I did notice though that "hw_mac_init=
_port[0], =E2=80=A6" will always be logged when it works but not when when =
it doesn't work. I confirmed this on a box still running the old version wh=
ich functions as intended.
The ones not working has a newer firmware version than the ones working.
Non-working:
# sysctl dev.t5nex.0.firmware_version
dev.t5nex.0.firmware_version: 1.19.1.0
Working:
# sysctl dev.t5nex.0.firmware_version
dev.t5nex.0.firmware_version: 1.16.45.0
Any other ideas?
--=20
Marius Halden
# ifconfig -mvvv cxl0
cxl0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=3De800bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCS=
UM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
capabilities=3Decc7bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN=
_HWCSUM,TSO4,TSO6,LRO,TOE4,TOE6,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV=
6>
ether 00:07:43:42:cb:30
hwaddr 00:07:43:42:cb:30
inet 1.2.3.4 netmask 0xfffffffc broadcast 1.2.3.4
inet6 fe80::207:43ff:fe42:cb30%cxl0 prefixlen 64 scopeid 0x5
inet6 a:b:c:d::d prefixlen 126
nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-SR <full-duplex>
status: active
supported media:
media 10Gbase-SR mediaopt full-duplex
plugged: SFP/SFP+/SFP28 1000BASE-LX (LC)
vendor: Fiberworks PN: SFP-L80D-C49-U SN: FBSW491434 DATE: 2017-09-27
module temperature: 41.41 C Voltage: 3.26 Volts
RX: 0.27 mW (-5.53 dBm) TX: 1.62 mW (2.10 dBm)
SFF8472 DUMP (0xA0 0..127 range):
03 04 07 00 00 00 02 10 10 10 01 01 0D 00 50 FF
00 00 00 00 46 69 62 65 72 77 6F 72 6B 73 20 20
20 20 20 20 00 8A 34 BC 53 46 50 2D 4C 38 30 44
2D 43 34 39 2D 55 20 20 31 2E 30 20 05 D2 00 29
00 1A 00 00 46 42 53 57 34 39 31 34 33 34 20 20
20 20 20 20 31 37 30 39 32 37 20 20 68 90 01 B8
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
# sysctl dev.t5nex.0.misc.devlog
dev.t5nex.0.misc.devlog: Seq# Tstamp Level Facility M=
essage
18 893740 INFO TM pktsched channel 0 sets sp=
eed (from 0) to 1000000 kbps
19 893755 INFO PORT hw_mac_init_port[1], ptype=
0x9, speed 0x4, lanes 0x2, fec 0x0
20 893787 INFO TM pktsched channel 1 sets sp=
eed (from 0) to 10000000 kbps
21 893800 INFO PORT hw_mac_init_port[2], ptype=
0x9, speed 0x4, lanes 0x4, fec 0x0
22 893832 INFO TM pktsched channel 2 sets sp=
eed (from 0) to 10000000 kbps
23 897121 INFO PORT module[3]: gpio 7 vendor i=
d 78a714, identifier 0x03, SFP28(byte 36/192) 0xff, SFP(byte 3/131) 0x00, 1=
G (byte 6) 0x00
24 897122 INFO PORT optical length(byte 15/142=
) 0, copper cable(byte 8/147) 0x04, length(byte 18/146) 3, module_type 0x04
25 897123 INFO PORT hw_mac_init_port[3], ptype=
0x9, speed 0x4, lanes 0x8, fec 0x0
26 897156 INFO TM pktsched channel 3 sets sp=
eed (from 0) to 10000000 kbps
27 922432 NOTICE PORT port[3:0x09:0x0a]: l1cfg, =
cannot force no/multiple speed(s), mcaps 0x10070006 acaps 0x70004 rcaps 0x3=
0006
28 922435 INFO PORT hw_mac_init_port[3], ptype=
0x9, speed 0x4, lanes 0x8, fec 0x0
29 923895 INFO TM pktsched_cl_rl[0:0]: mode =
| unit | rate 0x010001 min 0 max 100000 pktsize 1500
30 924899 INFO TM pktsched_cl_rl[0:1]: mode =
| unit | rate 0x010001 min 0 max 200000 pktsize 1500
31 925904 INFO TM pktsched_cl_rl[0:2]: mode =
| unit | rate 0x010001 min 0 max 400000 pktsize 1500
32 926909 INFO TM pktsched_cl_rl[0:3]: mode =
| unit | rate 0x010001 min 0 max 500000 pktsize 1500
33 927915 INFO TM pktsched_cl_rl[0:4]: mode =
| unit | rate 0x010001 min 0 max 800000 pktsize 1500
34 928920 INFO TM pktsched_cl_rl[0:5]: mode =
| unit | rate 0x010001 min 0 max 1000000 pktsize 1500
35 929931 INFO TM pktsched_cl_rl[0:6]: mode =
| unit | rate 0x010001 min 0 max 1200000 pktsize 1500
36 929932 NOTICE TM ch_cl_rate[0/6]: capped de=
ficit_incr from required 12000 to 10000; rate 1200000 (eff 1000000) deficit=
_max 6144
37 930937 INFO TM pktsched_cl_rl[0:7]: mode =
| unit | rate 0x010001 min 0 max 1500000 pktsize 1500
38 930938 NOTICE TM ch_cl_rate[0/7]: capped de=
ficit_incr from required 15000 to 10000; rate 1500000 (eff 1000000) deficit=
_max 6144
39 931947 INFO TM pktsched_cl_rl[0:8]: mode =
| unit | rate 0x010001 min 0 max 1800000 pktsize 1500
40 931948 NOTICE TM ch_cl_rate[0/8]: capped de=
ficit_incr from required 18000 to 10000; rate 1800000 (eff 1000000) deficit=
_max 6144
41 932952 INFO TM pktsched_cl_rl[0:9]: mode =
| unit | rate 0x010001 min 0 max 2000000 pktsize 1500
42 932953 NOTICE TM ch_cl_rate[0/9]: capped de=
ficit_incr from required 20000 to 10000; rate 2000000 (eff 1000000) deficit=
_max 6144
43 933959 INFO TM pktsched_cl_rl[0:10]: mode=
| unit | rate 0x010001 min 0 max 2500000 pktsize 1500
44 933960 NOTICE TM ch_cl_rate[0/10]: capped d=
eficit_incr from required 25000 to 10000; rate 2500000 (eff 1000000) defici=
t_max 6144
45 934964 INFO TM pktsched_cl_rl[0:11]: mode=
| unit | rate 0x010001 min 0 max 3000000 pktsize 1500
46 934965 NOTICE TM ch_cl_rate[0/11]: capped d=
eficit_incr from required 30000 to 10000; rate 3000000 (eff 1000000) defici=
t_max 6144
47 935969 INFO TM pktsched_cl_rl[0:12]: mode=
| unit | rate 0x010001 min 0 max 3500000 pktsize 1500
48 935970 NOTICE TM ch_cl_rate[0/12]: capped d=
eficit_incr from required 35000 to 10000; rate 3500000 (eff 1000000) defici=
t_max 6144
49 936974 INFO TM pktsched_cl_rl[0:13]: mode=
| unit | rate 0x010001 min 0 max 4000000 pktsize 1500
50 936975 NOTICE TM ch_cl_rate[0/13]: capped d=
eficit_incr from required 40000 to 10000; rate 4000000 (eff 1000000) defici=
t_max 6144
51 937980 INFO TM pktsched_cl_rl[0:14]: mode=
| unit | rate 0x010001 min 0 max 5000000 pktsize 1500
52 937981 NOTICE TM ch_cl_rate[0/14]: capped d=
eficit_incr from required 50000 to 10000; rate 5000000 (eff 1000000) defici=
t_max 6144
53 938986 INFO TM pktsched_cl_rl[0:15]: mode=
| unit | rate 0x010001 min 0 max 10000000 pktsize 1500
54 938987 NOTICE TM ch_cl_rate[0/15]: capped d=
eficit_incr from required 100000 to 10000; rate 10000000 (eff 1000000) defi=
cit_max 6144
55 939994 INFO TM pktsched_cl_rl[1:0]: mode =
| unit | rate 0x010001 min 0 max 100000 pktsize 1500
56 940999 INFO TM pktsched_cl_rl[1:1]: mode =
| unit | rate 0x010001 min 0 max 200000 pktsize 1500
57 942004 INFO TM pktsched_cl_rl[1:2]: mode =
| unit | rate 0x010001 min 0 max 400000 pktsize 1500
58 943009 INFO TM pktsched_cl_rl[1:3]: mode =
| unit | rate 0x010001 min 0 max 500000 pktsize 1500
59 944014 INFO TM pktsched_cl_rl[1:4]: mode =
| unit | rate 0x010001 min 0 max 800000 pktsize 1500
60 945021 INFO TM pktsched_cl_rl[1:5]: mode =
| unit | rate 0x010001 min 0 max 1000000 pktsize 1500
61 946032 INFO TM pktsched_cl_rl[1:6]: mode =
| unit | rate 0x010001 min 0 max 1200000 pktsize 1500
62 946033 NOTICE TM ch_cl_rate[1/6]: capped de=
ficit_incr from required 12000 to 10000; rate 1200000 (eff 1000000) deficit=
_max 6144
63 947037 INFO TM pktsched_cl_rl[1:7]: mode =
| unit | rate 0x010001 min 0 max 1500000 pktsize 1500
64 947038 NOTICE TM ch_cl_rate[1/7]: capped de=
ficit_incr from required 15000 to 10000; rate 1500000 (eff 1000000) deficit=
_max 6144
65 948043 INFO TM pktsched_cl_rl[1:8]: mode =
| unit | rate 0x010001 min 0 max 1800000 pktsize 1500
66 948043 NOTICE TM ch_cl_rate[1/8]: capped de=
ficit_incr from required 18000 to 10000; rate 1800000 (eff 1000000) deficit=
_max 6144
67 949049 INFO TM pktsched_cl_rl[1:9]: mode =
| unit | rate 0x010001 min 0 max 2000000 pktsize 1500
68 949050 NOTICE TM ch_cl_rate[1/9]: capped de=
ficit_incr from required 20000 to 10000; rate 2000000 (eff 1000000) deficit=
_max 6144
69 950054 INFO TM pktsched_cl_rl[1:10]: mode=
| unit | rate 0x010001 min 0 max 2500000 pktsize 1500
70 950055 NOTICE TM ch_cl_rate[1/10]: capped d=
eficit_incr from required 25000 to 10000; rate 2500000 (eff 1000000) defici=
t_max 6144
71 951066 INFO TM pktsched_cl_rl[1:11]: mode=
| unit | rate 0x010001 min 0 max 3000000 pktsize 1500
72 951067 NOTICE TM ch_cl_rate[1/11]: capped d=
eficit_incr from required 30000 to 10000; rate 3000000 (eff 1000000) defici=
t_max 6144
73 952072 INFO TM pktsched_cl_rl[1:12]: mode=
| unit | rate 0x010001 min 0 max 3500000 pktsize 1500
74 952073 NOTICE TM ch_cl_rate[1/12]: capped d=
eficit_incr from required 35000 to 10000; rate 3500000 (eff 1000000) defici=
t_max 6144
75 953078 INFO TM pktsched_cl_rl[1:13]: mode=
| unit | rate 0x010001 min 0 max 4000000 pktsize 1500
76 953079 NOTICE TM ch_cl_rate[1/13]: capped d=
eficit_incr from required 40000 to 10000; rate 4000000 (eff 1000000) defici=
t_max 6144
77 954083 INFO TM pktsched_cl_rl[1:14]: mode=
| unit | rate 0x010001 min 0 max 5000000 pktsize 1500
78 954084 NOTICE TM ch_cl_rate[1/14]: capped d=
eficit_incr from required 50000 to 10000; rate 5000000 (eff 1000000) defici=
t_max 6144
79 955095 INFO TM pktsched_cl_rl[1:15]: mode=
| unit | rate 0x010001 min 0 max 10000000 pktsize 1500
80 955096 NOTICE TM ch_cl_rate[1/15]: capped d=
eficit_incr from required 100000 to 10000; rate 10000000 (eff 1000000) defi=
cit_max 6144
81 956100 INFO TM pktsched_cl_rl[2:0]: mode =
| unit | rate 0x010001 min 0 max 100000 pktsize 1500
82 957107 INFO TM pktsched_cl_rl[2:1]: mode =
| unit | rate 0x010001 min 0 max 200000 pktsize 1500
83 958112 INFO TM pktsched_cl_rl[2:2]: mode =
| unit | rate 0x010001 min 0 max 400000 pktsize 1500
84 959122 INFO TM pktsched_cl_rl[2:3]: mode =
| unit | rate 0x010001 min 0 max 500000 pktsize 1500
85 960127 INFO TM pktsched_cl_rl[2:4]: mode =
| unit | rate 0x010001 min 0 max 800000 pktsize 1500
86 961137 INFO TM pktsched_cl_rl[2:5]: mode =
| unit | rate 0x010001 min 0 max 1000000 pktsize 1500
87 962143 INFO TM pktsched_cl_rl[2:6]: mode =
| unit | rate 0x010001 min 0 max 1200000 pktsize 1500
88 962143 NOTICE TM ch_cl_rate[2/6]: capped de=
ficit_incr from required 12000 to 10000; rate 1200000 (eff 1000000) deficit=
_max 6144
89 963148 INFO TM pktsched_cl_rl[2:7]: mode =
| unit | rate 0x010001 min 0 max 1500000 pktsize 1500
90 963149 NOTICE TM ch_cl_rate[2/7]: capped de=
ficit_incr from required 15000 to 10000; rate 1500000 (eff 1000000) deficit=
_max 6144
91 964153 INFO TM pktsched_cl_rl[2:8]: mode =
| unit | rate 0x010001 min 0 max 1800000 pktsize 1500
92 964154 NOTICE TM ch_cl_rate[2/8]: capped de=
ficit_incr from required 18000 to 10000; rate 1800000 (eff 1000000) deficit=
_max 6144
93 965159 INFO TM pktsched_cl_rl[2:9]: mode =
| unit | rate 0x010001 min 0 max 2000000 pktsize 1500
94 965160 NOTICE TM ch_cl_rate[2/9]: capped de=
ficit_incr from required 20000 to 10000; rate 2000000 (eff 1000000) deficit=
_max 6144
95 966164 INFO TM pktsched_cl_rl[2:10]: mode=
| unit | rate 0x010001 min 0 max 2500000 pktsize 1500
96 966165 NOTICE TM ch_cl_rate[2/10]: capped d=
eficit_incr from required 25000 to 10000; rate 2500000 (eff 1000000) defici=
t_max 6144
97 967174 INFO TM pktsched_cl_rl[2:11]: mode=
| unit | rate 0x010001 min 0 max 3000000 pktsize 1500
98 967175 NOTICE TM ch_cl_rate[2/11]: capped d=
eficit_incr from required 30000 to 10000; rate 3000000 (eff 1000000) defici=
t_max 6144
99 968186 INFO TM pktsched_cl_rl[2:12]: mode=
| unit | rate 0x010001 min 0 max 3500000 pktsize 1500
100 968187 NOTICE TM ch_cl_rate[2/12]: capped d=
eficit_incr from required 35000 to 10000; rate 3500000 (eff 1000000) defici=
t_max 6144
101 969191 INFO TM pktsched_cl_rl[2:13]: mode=
| unit | rate 0x010001 min 0 max 4000000 pktsize 1500
102 969192 NOTICE TM ch_cl_rate[2/13]: capped d=
eficit_incr from required 40000 to 10000; rate 4000000 (eff 1000000) defici=
t_max 6144
103 970198 INFO TM pktsched_cl_rl[2:14]: mode=
| unit | rate 0x010001 min 0 max 5000000 pktsize 1500
104 970199 NOTICE TM ch_cl_rate[2/14]: capped d=
eficit_incr from required 50000 to 10000; rate 5000000 (eff 1000000) defici=
t_max 6144
105 971206 INFO TM pktsched_cl_rl[2:15]: mode=
| unit | rate 0x010001 min 0 max 10000000 pktsize 1500
106 971206 NOTICE TM ch_cl_rate[2/15]: capped d=
eficit_incr from required 100000 to 10000; rate 10000000 (eff 1000000) defi=
cit_max 6144
107 972212 INFO TM pktsched_cl_rl[3:0]: mode =
| unit | rate 0x010001 min 0 max 100000 pktsize 1500
108 973218 INFO TM pktsched_cl_rl[3:1]: mode =
| unit | rate 0x010001 min 0 max 200000 pktsize 1500
109 974225 INFO TM pktsched_cl_rl[3:2]: mode =
| unit | rate 0x010001 min 0 max 400000 pktsize 1500
110 975236 INFO TM pktsched_cl_rl[3:3]: mode =
| unit | rate 0x010001 min 0 max 500000 pktsize 1500
111 976241 INFO TM pktsched_cl_rl[3:4]: mode =
| unit | rate 0x010001 min 0 max 800000 pktsize 1500
112 977246 INFO TM pktsched_cl_rl[3:5]: mode =
| unit | rate 0x010001 min 0 max 1000000 pktsize 1500
113 978251 INFO TM pktsched_cl_rl[3:6]: mode =
| unit | rate 0x010001 min 0 max 1200000 pktsize 1500
114 978252 NOTICE TM ch_cl_rate[3/6]: capped de=
ficit_incr from required 12000 to 10000; rate 1200000 (eff 1000000) deficit=
_max 6144
115 979257 INFO TM pktsched_cl_rl[3:7]: mode =
| unit | rate 0x010001 min 0 max 1500000 pktsize 1500
116 979257 NOTICE TM ch_cl_rate[3/7]: capped de=
ficit_incr from required 15000 to 10000; rate 1500000 (eff 1000000) deficit=
_max 6144
117 980263 INFO TM pktsched_cl_rl[3:8]: mode =
| unit | rate 0x010001 min 0 max 1800000 pktsize 1500
118 980264 NOTICE TM ch_cl_rate[3/8]: capped de=
ficit_incr from required 18000 to 10000; rate 1800000 (eff 1000000) deficit=
_max 6144
119 981270 INFO TM pktsched_cl_rl[3:9]: mode =
| unit | rate 0x010001 min 0 max 2000000 pktsize 1500
120 981271 NOTICE TM ch_cl_rate[3/9]: capped de=
ficit_incr from required 20000 to 10000; rate 2000000 (eff 1000000) deficit=
_max 6144
121 982276 INFO TM pktsched_cl_rl[3:10]: mode=
| unit | rate 0x010001 min 0 max 2500000 pktsize 1500
122 982277 NOTICE TM ch_cl_rate[3/10]: capped d=
eficit_incr from required 25000 to 10000; rate 2500000 (eff 1000000) defici=
t_max 6144
123 983281 INFO TM pktsched_cl_rl[3:11]: mode=
| unit | rate 0x010001 min 0 max 3000000 pktsize 1500
124 983282 NOTICE TM ch_cl_rate[3/11]: capped d=
eficit_incr from required 30000 to 10000; rate 3000000 (eff 1000000) defici=
t_max 6144
125 984286 INFO TM pktsched_cl_rl[3:12]: mode=
| unit | rate 0x010001 min 0 max 3500000 pktsize 1500
126 984287 NOTICE TM ch_cl_rate[3/12]: capped d=
eficit_incr from required 35000 to 10000; rate 3500000 (eff 1000000) defici=
t_max 6144
127 985292 INFO TM pktsched_cl_rl[3:13]: mode=
| unit | rate 0x010001 min 0 max 4000000 pktsize 1500
128 985292 NOTICE TM ch_cl_rate[3/13]: capped d=
eficit_incr from required 40000 to 10000; rate 4000000 (eff 1000000) defici=
t_max 6144
129 986297 INFO TM pktsched_cl_rl[3:14]: mode=
| unit | rate 0x010001 min 0 max 5000000 pktsize 1500
130 986298 NOTICE TM ch_cl_rate[3/14]: capped d=
eficit_incr from required 50000 to 10000; rate 5000000 (eff 1000000) defici=
t_max 6144
131 987302 INFO TM pktsched_cl_rl[3:15]: mode=
| unit | rate 0x010001 min 0 max 10000000 pktsize 1500
132 987303 NOTICE TM ch_cl_rate[3/15]: capped d=
eficit_incr from required 100000 to 10000; rate 10000000 (eff 1000000) defi=
cit_max 6144
133 987415 INFO PORT port_link_state_handler[3]=
powering down
134 6187430 INFO PORT port_link_state_handler[0]=
powering up
135 6187441 INFO PORT port[0] update (flowcid 37=
976 rc 0)
136 6387432 INFO PORT port_link_state_handler[3]=
powering up
137 6387437 INFO PORT port[3] update (flowcid 37=
976 rc 0)
138 6687431 INFO PORT port_hss_sigdet[3]: hss_si=
gdet changed to 0x8
139 6787431 INFO PORT hw_mac_link_status[3] int_=
cause 0x1001c70, link_status 0x12
140 6787434 INFO PORT port[0] link up (1) (speed=
0x2 acaps 0x70002 lpcaps 0x70000)
141 6787436 INFO PORT port[0] set PAUSE PARAMS: =
pppen 0 txpe 0x1 rxpe 0x1
142 6787442 INFO PORT port[0] update (flowcid 37=
976 rc 0)
143 6987432 INFO PORT port[3] link up (1) (speed=
0x4 acaps 0x70006 lpcaps 0x70000)
144 6987433 INFO PORT port[3] set PAUSE PARAMS: =
pppen 0 txpe 0x1 rxpe 0x1
145 6987436 INFO PORT port[3] update (flowcid 37=
976 rc 0)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1535628110.3034153.1491191792.12397BCF>