Date: Mon, 26 Feb 2024 18:25:26 +0000 From: Alex Shalima <alex@hotelwifi.com> To: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: X710 stalled TX Queue and loss of networking Message-ID: <BY3PR13MB4929243A0160A0B6206530C3CE5A2@BY3PR13MB4929.namprd13.prod.outlook.com>
next in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] Hello, DATA We are running FreeBSD 13.2-RELEASE-p9 #25 on top of several Dell R650 (Example Service Tag: 8FKQRY3). The system is running bhyve for other FreeBSD Virtual Machines. All these servers have X710-DA4 Fiber Network cards (4 port of SFP+). dev.ixl.0.%desc: Intel(R) Ethernet Controller X710 for 10GbE SFP+ - 2.3.3-k dev.ixl.0.fw_version: fw 9.840.76614 api 1.15 nvm 9.40 etid 8000e9b5 oem 22.5632.7 Some servers have an additional X710-DA2 (same card but with 2 ports) for extra fiber ports. ISSUE Periodically, the networking will stop working on individual interfaces. During packet capture we can see that the networking card is receiving traffic, but no traffic is being set out. During further investigation we found that ixl interface TX queue is getting into STALLED mode. [user@server ~]$ sysctl dev.ixl | grep ring_state dev.ixl.5.iflib.txq0.ring_state: pidx_head: 0751 pidx_tail: 0751 cidx: 0751 state: IDLE dev.ixl.4.iflib.txq0.ring_state: pidx_head: 1254 pidx_tail: 1254 cidx: 1254 state: IDLE dev.ixl.3.iflib.txq0.ring_state: pidx_head: 1193 pidx_tail: 1193 cidx: 1195 state: STALLED dev.ixl.2.iflib.txq0.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE dev.ixl.1.iflib.txq0.ring_state: pidx_head: 1393 pidx_tail: 1393 cidx: 1395 state: STALLED dev.ixl.0.iflib.txq0.ring_state: pidx_head: 0181 pidx_tail: 0181 cidx: 0183 state: STALLE RESOLUTIONS TRIED * Factory resetting the system (not a permanent fix, issue comes back) * Recreating Netowrking interfaces invluding VLANs (not a permanent fix, issue comes back) * Updating the driver with Dell iDRAC to the latest official QUESTION Is there anything else we can try to get this permanently resolved? Best Regards, Alex [-- Attachment #2 --] <html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">; <head> <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:Wingdings; panose-1:5 0 0 0 0 0 0 0 0 0;} @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} @font-face {font-family:Aptos;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; font-size:11.0pt; font-family:"Aptos",sans-serif; mso-ligatures:standardcontextual;} p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph {mso-style-priority:34; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; font-size:11.0pt; font-family:"Aptos",sans-serif; mso-ligatures:standardcontextual;} span.EmailStyle17 {mso-style-type:personal-compose; font-family:"Aptos",sans-serif; color:windowtext;} .MsoChpDefault {mso-style-type:export-only; font-size:11.0pt; font-family:"Aptos",sans-serif;} @page WordSection1 {size:8.5in 11.0in; margin:56.7pt 42.5pt 56.7pt 85.05pt;} div.WordSection1 {page:WordSection1;} /* List Definitions */ @list l0 {mso-list-id:1854220148; mso-list-type:hybrid; mso-list-template-ids:183414800 1981729794 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l0:level1 {mso-level-start-at:0; mso-level-number-format:bullet; mso-level-text:-; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"Aptos",sans-serif; mso-fareast-font-family:"Times New Roman"; mso-bidi-font-family:"Times New Roman";} @list l0:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"Courier New";} @list l0:level3 {mso-level-number-format:bullet; mso-level-text:\F0A7; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:Wingdings;} @list l0:level4 {mso-level-number-format:bullet; mso-level-text:\F0B7; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:Symbol;} @list l0:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"Courier New";} @list l0:level6 {mso-level-number-format:bullet; mso-level-text:\F0A7; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:Wingdings;} @list l0:level7 {mso-level-number-format:bullet; mso-level-text:\F0B7; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:Symbol;} @list l0:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"Courier New";} @list l0:level9 {mso-level-number-format:bullet; mso-level-text:\F0A7; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:Wingdings;} ol {margin-bottom:0in;} ul {margin-bottom:0in;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext="edit" spidmax="1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext="edit"> <o:idmap v:ext="edit" data="1" /> </o:shapelayout></xml><![endif]--> </head> <body lang="EN-US" link="#467886" vlink="#96607D" style="word-wrap:break-word"> <div class="WordSection1"> <p class="MsoNormal">Hello,<o:p></o:p></p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">DATA<o:p></o:p></p> <p class="MsoNormal">We are running FreeBSD 13.2-RELEASE-p9 #25 on top of several Dell R650 (Example Service Tag: 8FKQRY3). The system is running bhyve for other FreeBSD Virtual Machines.<o:p></o:p></p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">All these servers have X710-DA4 Fiber Network cards (4 port of SFP+).<o:p></o:p></p> <p class="MsoNormal">dev.ixl.0.%desc: Intel(R) Ethernet Controller X710 for 10GbE SFP+ - 2.3.3-k<o:p></o:p></p> <p class="MsoNormal">dev.ixl.0.fw_version: fw 9.840.76614 api 1.15 nvm 9.40 etid 8000e9b5 oem 22.5632.7<o:p></o:p></p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">Some servers have an additional X710-DA2 (same card but with 2 ports) for extra fiber ports.<o:p></o:p></p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">ISSUE<o:p></o:p></p> <p class="MsoNormal">Periodically, the networking will stop working on individual interfaces. During packet capture we can see that the networking card is receiving traffic, but no traffic is being set out. During further investigation we found that ixl interface TX queue is getting into STALLED mode.<o:p></o:p></p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">[user@server ~]$ sysctl dev.ixl | grep ring_state<br> dev.ixl.5.iflib.txq0.ring_state: pidx_head: 0751 pidx_tail: 0751 cidx: 0751 state: IDLE<br> dev.ixl.4.iflib.txq0.ring_state: pidx_head: 1254 pidx_tail: 1254 cidx: 1254 state: IDLE<br> dev.ixl.3.iflib.txq0.ring_state: pidx_head: 1193 pidx_tail: 1193 cidx: 1195 state: STALLED<br> dev.ixl.2.iflib.txq0.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE<br> dev.ixl.1.iflib.txq0.ring_state: pidx_head: 1393 pidx_tail: 1393 cidx: 1395 state: STALLED<br> dev.ixl.0.iflib.txq0.ring_state: pidx_head: 0181 pidx_tail: 0181 cidx: 0183 state: STALLE<o:p></o:p></p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">RESOLUTIONS TRIED<o:p></o:p></p> <ul style="margin-top:0in" type="disc"> <li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Factory resetting the system (not a permanent fix, issue comes back)<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Recreating Netowrking interfaces invluding VLANs (not a permanent fix, issue comes back)<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Updating the driver with Dell iDRAC to the latest official <o:p></o:p></li></ul> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">QUESTION<o:p></o:p></p> <p class="MsoNormal">Is there anything else we can try to get this permanently resolved? <o:p></o:p></p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal"><span style="color:#212121">Best Regards,<o:p></o:p></span></p> <p class="MsoNormal">Alex<o:p></o:p></p> </div> </body> </html>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?BY3PR13MB4929243A0160A0B6206530C3CE5A2>