Date: Mon, 26 Feb 2024 18:25:26 +0000 From: Alex Shalima <alex@hotelwifi.com> To: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: X710 stalled TX Queue and loss of networking Message-ID: <BY3PR13MB4929243A0160A0B6206530C3CE5A2@BY3PR13MB4929.namprd13.prod.outlook.com>
next in thread | raw e-mail | index | archive | help
--_000_BY3PR13MB4929243A0160A0B6206530C3CE5A2BY3PR13MB4929namp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hello, DATA We are running FreeBSD 13.2-RELEASE-p9 #25 on top of several Dell R650 (Exa= mple Service Tag: 8FKQRY3). The system is running bhyve for other FreeBSD V= irtual Machines. All these servers have X710-DA4 Fiber Network cards (4 port of SFP+). dev.ixl.0.%desc: Intel(R) Ethernet Controller X710 for 10GbE SFP+ - 2.3.3-k dev.ixl.0.fw_version: fw 9.840.76614 api 1.15 nvm 9.40 etid 8000e9b5 oem 22= .5632.7 Some servers have an additional X710-DA2 (same card but with 2 ports) for e= xtra fiber ports. ISSUE Periodically, the networking will stop working on individual interfaces. Du= ring packet capture we can see that the networking card is receiving traffi= c, but no traffic is being set out. During further investigation we found t= hat ixl interface TX queue is getting into STALLED mode. [user@server ~]$ sysctl dev.ixl | grep ring_state dev.ixl.5.iflib.txq0.ring_state: pidx_head: 0751 pidx_tail: 0751 cidx: 0751= state: IDLE dev.ixl.4.iflib.txq0.ring_state: pidx_head: 1254 pidx_tail: 1254 cidx: 1254= state: IDLE dev.ixl.3.iflib.txq0.ring_state: pidx_head: 1193 pidx_tail: 1193 cidx: 1195= state: STALLED dev.ixl.2.iflib.txq0.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000= state: IDLE dev.ixl.1.iflib.txq0.ring_state: pidx_head: 1393 pidx_tail: 1393 cidx: 1395= state: STALLED dev.ixl.0.iflib.txq0.ring_state: pidx_head: 0181 pidx_tail: 0181 cidx: 0183= state: STALLE RESOLUTIONS TRIED * Factory resetting the system (not a permanent fix, issue comes back) * Recreating Netowrking interfaces invluding VLANs (not a permanent fix= , issue comes back) * Updating the driver with Dell iDRAC to the latest official QUESTION Is there anything else we can try to get this permanently resolved? Best Regards, Alex --_000_BY3PR13MB4929243A0160A0B6206530C3CE5A2BY3PR13MB4929namp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr= osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:= //www.w3.org/TR/REC-html40"> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"= > <meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:Wingdings; panose-1:5 0 0 0 0 0 0 0 0 0;} @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} @font-face {font-family:Aptos;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; font-size:11.0pt; font-family:"Aptos",sans-serif; mso-ligatures:standardcontextual;} p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph {mso-style-priority:34; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; font-size:11.0pt; font-family:"Aptos",sans-serif; mso-ligatures:standardcontextual;} span.EmailStyle17 {mso-style-type:personal-compose; font-family:"Aptos",sans-serif; color:windowtext;} .MsoChpDefault {mso-style-type:export-only; font-size:11.0pt; font-family:"Aptos",sans-serif;} @page WordSection1 {size:8.5in 11.0in; margin:56.7pt 42.5pt 56.7pt 85.05pt;} div.WordSection1 {page:WordSection1;} /* List Definitions */ @list l0 {mso-list-id:1854220148; mso-list-type:hybrid; mso-list-template-ids:183414800 1981729794 67698691 67698693 67698689 6769= 8691 67698693 67698689 67698691 67698693;} @list l0:level1 {mso-level-start-at:0; mso-level-number-format:bullet; mso-level-text:-; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"Aptos",sans-serif; mso-fareast-font-family:"Times New Roman"; mso-bidi-font-family:"Times New Roman";} @list l0:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"Courier New";} @list l0:level3 {mso-level-number-format:bullet; mso-level-text:\F0A7; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:Wingdings;} @list l0:level4 {mso-level-number-format:bullet; mso-level-text:\F0B7; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:Symbol;} @list l0:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"Courier New";} @list l0:level6 {mso-level-number-format:bullet; mso-level-text:\F0A7; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:Wingdings;} @list l0:level7 {mso-level-number-format:bullet; mso-level-text:\F0B7; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:Symbol;} @list l0:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"Courier New";} @list l0:level9 {mso-level-number-format:bullet; mso-level-text:\F0A7; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:Wingdings;} ol {margin-bottom:0in;} ul {margin-bottom:0in;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--> </head> <body lang=3D"EN-US" link=3D"#467886" vlink=3D"#96607D" style=3D"word-wrap:= break-word"> <div class=3D"WordSection1"> <p class=3D"MsoNormal">Hello,<o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">DATA<o:p></o:p></p> <p class=3D"MsoNormal">We are running FreeBSD 13.2-RELEASE-p9 #25 on top of= several Dell R650 (Example Service Tag: 8FKQRY3). The system is running bh= yve for other FreeBSD Virtual Machines.<o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">All these servers have X710-DA4 Fiber Network cards = (4 port of SFP+).<o:p></o:p></p> <p class=3D"MsoNormal">dev.ixl.0.%desc: Intel(R) Ethernet Controller X710 f= or 10GbE SFP+ - 2.3.3-k<o:p></o:p></p> <p class=3D"MsoNormal">dev.ixl.0.fw_version: fw 9.840.76614 api 1.15 nvm 9.= 40 etid 8000e9b5 oem 22.5632.7<o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">Some servers have an additional X710-DA2 (same card = but with 2 ports) for extra fiber ports.<o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">ISSUE<o:p></o:p></p> <p class=3D"MsoNormal">Periodically, the networking will stop working on in= dividual interfaces. During packet capture we can see that the networking c= ard is receiving traffic, but no traffic is being set out. During further i= nvestigation we found that ixl interface TX queue is getting into STALLED mode.<o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">[user@server ~]$ sysctl dev.ixl | grep ring_state<br= > dev.ixl.5.iflib.txq0.ring_state: pidx_head: 0751 pidx_tail: 0751 cidx: 0751= state: IDLE<br> dev.ixl.4.iflib.txq0.ring_state: pidx_head: 1254 pidx_tail: 1254 cidx: 1254= state: IDLE<br> dev.ixl.3.iflib.txq0.ring_state: pidx_head: 1193 pidx_tail: 1193 cidx: 1195= state: STALLED<br> dev.ixl.2.iflib.txq0.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000= state: IDLE<br> dev.ixl.1.iflib.txq0.ring_state: pidx_head: 1393 pidx_tail: 1393 cidx: 1395= state: STALLED<br> dev.ixl.0.iflib.txq0.ring_state: pidx_head: 0181 pidx_tail: 0181 cidx: 0183= state: STALLE<o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">RESOLUTIONS TRIED<o:p></o:p></p> <ul style=3D"margin-top:0in" type=3D"disc"> <li class=3D"MsoListParagraph" style=3D"margin-left:0in;mso-list:l0 level1 = lfo1">Factory resetting the system (not a permanent fix, issue comes back)<= o:p></o:p></li><li class=3D"MsoListParagraph" style=3D"margin-left:0in;mso-= list:l0 level1 lfo1">Recreating Netowrking interfaces invluding VLANs (not = a permanent fix, issue comes back)<o:p></o:p></li><li class=3D"MsoListParag= raph" style=3D"margin-left:0in;mso-list:l0 level1 lfo1">Updating the driver= with Dell iDRAC to the latest official <o:p></o:p></li></ul> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">QUESTION<o:p></o:p></p> <p class=3D"MsoNormal">Is there anything else we can try to get this perman= ently resolved? <o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal"><span style=3D"color:#212121">Best Regards,<o:p></o:= p></span></p> <p class=3D"MsoNormal">Alex<o:p></o:p></p> </div> </body> </html> --_000_BY3PR13MB4929243A0160A0B6206530C3CE5A2BY3PR13MB4929namp_--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?BY3PR13MB4929243A0160A0B6206530C3CE5A2>