Date: Wed, 27 Aug 2025 23:08:00 +0800 From: Zhenlei Huang <zlei@FreeBSD.org> To: John Baldwin <jhb@freebsd.org> Cc: ShengYi Hung <aokblast@FreeBSD.org>, hackers@freebsd.org Subject: Re: downgraded pcie link width Message-ID: <46F3153E-6106-4A94-A4CF-9090B6ABA8B6@FreeBSD.org> In-Reply-To: <10fb2a55-0d92-4d83-8bce-45775efb6259@FreeBSD.org> References: <062CDEA5-69A0-49BA-A294-75DBBC3F5829@FreeBSD.org> <m2ms7lx9gz.fsf@FreeBSD.org> <m2ecsxx76y.fsf@FreeBSD.org> <10fb2a55-0d92-4d83-8bce-45775efb6259@FreeBSD.org>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --] > On Aug 27, 2025, at 9:12 PM, John Baldwin <jhb@freebsd.org> wrote: > > On 8/27/25 08:47, ShengYi Hung wrote: >> Sorry, for HADW bit. You have to set this bit instead of clear it to >> disable it then issue retraining. It will disable upconfiguration in TS2 and let device runs on >> the maximum width. >> This can be doen by using setpci. > > pciconf in the base system can also write to configuration registers. That said, > if this should ideally have cooperation from the device driver, then it may make > sense to instead add a new devctl operation to trigger a retrain so that we can > suspend the device driver while retraining. Hi John and ShengYi, I'll explore tomorrow. Thanks for your suggestion. > >> ShengYi Hung <aokblast@FreeBSD.org> writes: >>> --text follows this line-- >>> >>> >>> In my understanding, the >>> >>> Zhenlei Huang <zlei@FreeBSD.org> writes: >>> >>>> Hi, >>>> >>>> I'm recently hacking on the QLogic FastLinQ QL41212HLCU 25GbE adapter, and found something weird. >>>> >>>> It is a two SFP28 port card with PCIe 3.0 x8 link [1]. I connected the two ports with DAC cable directly to do benchmark. >>>> The weirdness is that no matter how much load I try to put into the card, it can only reach to about 13Gbps. >>>> I used iperf3 to do the benchmark. Also tried disabling TSO and LRO, enabling Jumbo MTU, but no luck. >>>> >>>> I checked the SFP module ( SFP28 DAC cable ) and ifconfig shows the link is 25000G, >>>> >>>> ``` >>>> # ifconfig -j1 -mv ql0 >>>> ql0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 >>>> options=8d00bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,HWSTATS> >>>> capabilities=8d07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,HWSTATS> >>>> ether xx:xx:xx:xx:xx:xx >>>> inet 172.16.1.1 netmask 0xffffff00 broadcast 172.16.1.255 >>>> media: Ethernet autoselect (25GBase-CR <full-duplex>) >>>> status: active >>>> supported media: >>>> media autoselect >>>> media autoselect mediaopt full-duplex >>>> media 25GBase-CR >>>> media 25GBase-SR >>>> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> >>>> drivername: ql0 >>>> plugged: SFP/SFP+/SFP28 25GBASE-CR CA-25G-S (Copper pigtail) >>>> vendor: OEM PN: CAB-ZSP/ZSP-P2M SN: XXXXXXXXXXXXX DATE: 2025-07-04 >>>> ``` >>>> >>>> and finally I observed something unusual from pciconf, >>>> >>>> ``` >>>> # pciconf -lcv ql0 >>>> ... >>>> cap 10[70] = PCI-Express 2 endpoint max data 256(512) FLR NS >>>> max read 4096 >>>> link x2(x8) speed 8.0(8.0) ClockPM disabled >>>> ``` >>>> >>>> That can also be verified by lspci from pciutils ports. >>>> ``` >>>> # lspci -s 08:00.0 -vv >>>> ... >>>> LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported >>>> ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+ >>>> LnkSta: Speed 8GT/s, Width x2 (downgraded) >>>> TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- >>>> ``` >>>> >>>> What I have tried, >>>> >>>> 1. Plugged the card into different mother board ( 3 different vendors, Dell, HP, and Gigabyte ), and different PCIe slot ( x16 and x4 ). >>>> 2. Upgraded the BIOS of mother board. >>>> 3. Disabled ASPM in BIOS. >>>> 4. Upgraded the firmware of card. >>>> 5. Booted with Debian 13 live CD. >>>> >>>> Nothing has changed. The PCIe link width can only be negotiated to maximum of x2, with or without driver loaded, with / without load on the card. >>>> It is also interesting that it can only be negotiated to x1 on Gigabyte motherboard, which has only one PCIe 2.0 x16 slot. >>>> >>>> After Googling I found some articles say that the PCIe link width is negotiated at the training stage, which is at POST before the driver loads. >>>> They hint that downgraded link width is mostly caused by wrong BIOS configure, or hardware issues such as scratched gold fingers. >>>> I would almost give up and found the product brief [2], in which it declares `Supports PCIe upconfigure to reduce link width to conserve power`. >>>> So interesting, maybe it is the firmware's fault that the firmware does not **upconfigure** ( retraining ) on sufficient load ? >>>> >>>> Are your FastLinQ 41000 ethernet cards been rightly negotiated to x8 ? >>>> >>>> What can I do next ? >>>> >>>> CC John, I guess he is familiar with PCIe spec :) >>>> >>>> >>>> [1] https://www.marvell.com/products/ethernet-adapters-and-controllers/41000-ethernet-adapters.html >>>> [2] https://www.marvell.com/content/dam/marvell/en/public-collateral/ethernet-adaptersandcontrollers/marvell-ethernet-adapters-fastlinq-41000-series-product-brief.pdf >>>> >>>> Best regards, >>>> Zhenlei > > -- > John Baldwin [-- Attachment #2 --] <html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Aug 27, 2025, at 9:12 PM, John Baldwin <<a href="mailto:jhb@freebsd.org" class="">jhb@freebsd.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta charset="UTF-8" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">On 8/27/25 08:47, ShengYi Hung wrote:</span><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote type="cite" style="font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Sorry, for HADW bit. You have to set this bit instead of clear it to<br class="">disable it then issue retraining. It will disable upconfiguration in TS2 and let device runs on<br class="">the maximum width.<br class="">This can be doen by using setpci.<br class=""></blockquote><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">pciconf in the base system can also write to configuration registers. That said,</span><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">if this should ideally have cooperation from the device driver, then it may make</span><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">sense to instead add a new devctl operation to trigger a retrain so that we can</span><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">suspend the device driver while retraining.</span><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""></div></blockquote><div><br class=""></div><div>Hi John and ShengYi,</div><div><br class=""></div><div><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">I'll explore tomorrow.</span></div><div><br class=""></div><div>Thanks for your suggestion.</div><br class=""><blockquote type="cite" class=""><div class=""><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote type="cite" style="font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">ShengYi Hung <<a href="mailto:aokblast@FreeBSD.org" class="">aokblast@FreeBSD.org</a>> writes:<br class=""><blockquote type="cite" class="">--text follows this line--<br class=""><br class=""><br class="">In my understanding, the<br class=""><br class="">Zhenlei Huang <<a href="mailto:zlei@FreeBSD.org" class="">zlei@FreeBSD.org</a>> writes:<br class=""><br class=""><blockquote type="cite" class="">Hi,<br class=""><br class="">I'm recently hacking on the QLogic FastLinQ QL41212HLCU 25GbE adapter, and found something weird.<br class=""><br class="">It is a two SFP28 port card with PCIe 3.0 x8 link [1]. I connected the two ports with DAC cable directly to do benchmark.<br class="">The weirdness is that no matter how much load I try to put into the card, it can only reach to about 13Gbps.<br class="">I used iperf3 to do the benchmark. Also tried disabling TSO and LRO, enabling Jumbo MTU, but no luck.<br class=""><br class="">I checked the SFP module ( SFP28 DAC cable ) and ifconfig shows the link is 25000G,<br class=""><br class="">```<br class=""># ifconfig -j1 -mv ql0<br class="">ql0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>options=8d00bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,HWSTATS><br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>capabilities=8d07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,HWSTATS><br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>ether xx:xx:xx:xx:xx:xx<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>inet 172.16.1.1 netmask 0xffffff00 broadcast 172.16.1.255<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>media: Ethernet autoselect (25GBase-CR <full-duplex>)<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>status: active<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>supported media:<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>media autoselect<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>media autoselect mediaopt full-duplex<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>media 25GBase-CR<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>media 25GBase-SR<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL><br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>drivername: ql0<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>plugged: SFP/SFP+/SFP28 25GBASE-CR CA-25G-S (Copper pigtail)<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>vendor: OEM PN: CAB-ZSP/ZSP-P2M SN: XXXXXXXXXXXXX DATE: 2025-07-04<br class="">```<br class=""><br class=""> and finally I observed something unusual from pciconf,<br class=""><br class="">```<br class=""># pciconf -lcv ql0<br class="">...<br class=""> cap 10[70] = PCI-Express 2 endpoint max data 256(512) FLR NS<br class=""> max read 4096<br class=""> link x2(x8) speed 8.0(8.0) ClockPM disabled<br class="">```<br class=""><br class="">That can also be verified by lspci from pciutils ports.<br class="">```<br class=""># lspci -s 08:00.0 -vv<br class="">...<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>LnkCap:<span class="Apple-tab-span" style="white-space: pre;"> </span>Port #0, Speed 8GT/s, Width x8, ASPM not supported<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>LnkSta:<span class="Apple-tab-span" style="white-space: pre;"> </span>Speed 8GT/s, Width x2 (downgraded)<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-<br class="">```<br class=""><br class="">What I have tried,<br class=""><br class=""> 1. Plugged the card into different mother board ( 3 different vendors, Dell, HP, and Gigabyte ), and different PCIe slot ( x16 and x4 ).<br class=""> 2. Upgraded the BIOS of mother board.<br class=""> 3. Disabled ASPM in BIOS.<br class=""> 4. Upgraded the firmware of card.<br class=""> 5. Booted with Debian 13 live CD.<br class=""><br class="">Nothing has changed. The PCIe link width can only be negotiated to maximum of x2, with or without driver loaded, with / without load on the card.<br class="">It is also interesting that it can only be negotiated to x1 on Gigabyte motherboard, which has only one PCIe 2.0 x16 slot.<br class=""><br class="">After Googling I found some articles say that the PCIe link width is negotiated at the training stage, which is at POST before the driver loads.<br class="">They hint that downgraded link width is mostly caused by wrong BIOS configure, or hardware issues such as scratched gold fingers.<br class="">I would almost give up and found the product brief [2], in which it declares `Supports PCIe upconfigure to reduce link width to conserve power`.<br class="">So interesting, maybe it is the firmware's fault that the firmware does not **upconfigure** ( retraining ) on sufficient load ?<br class=""><br class="">Are your FastLinQ 41000 ethernet cards been rightly negotiated to x8 ?<br class=""><br class="">What can I do next ?<br class=""><br class="">CC John, I guess he is familiar with PCIe spec :)<br class=""><br class=""><br class="">[1] <a href="https://www.marvell.com/products/ethernet-adapters-and-controllers/41000-ethernet-adapters.html" class="">https://www.marvell.com/products/ethernet-adapters-and-controllers/41000-ethernet-adapters.html</a><br class="">[2] <a href="https://www.marvell.com/content/dam/marvell/en/public-collateral/ethernet-adaptersandcontrollers/marvell-ethernet-adapters-fastlinq-41000-series-product-brief.pdf" class="">https://www.marvell.com/content/dam/marvell/en/public-collateral/ethernet-adaptersandcontrollers/marvell-ethernet-adapters-fastlinq-41000-series-product-brief.pdf</a><br class=""><br class="">Best regards,<br class="">Zhenlei<br class=""></blockquote></blockquote></blockquote><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">--<span class="Apple-converted-space"> </span></span><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">John Baldwin</span></div></blockquote></div><br class=""><div class=""> <div><br class=""></div> </div> <br class=""></body></html>help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?46F3153E-6106-4A94-A4CF-9090B6ABA8B6>