Date: Sun, 30 May 2021 13:58:05 +0000 From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 256264] Devices behind PEX 8664 PCIe Switch not detected since 11.0-RELEASE Message-ID: <bug-256264-227-UtdO3XMXAu@https.bugs.freebsd.org/bugzilla/> In-Reply-To: <bug-256264-227@https.bugs.freebsd.org/bugzilla/> References: <bug-256264-227@https.bugs.freebsd.org/bugzilla/>
next in thread | previous in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D256264 --- Comment #1 from Christopher Forgeron <chris@acsi.ca> --- Some lspci commands I used to try and narrow this down: 1) This is the 10.2 System. You can see both the Intel 82599ES and the LSI = 2008 cards. Ignore the MegaRAID, that's a built in PERC card that's always detec= ted (different bus): lspci -vvnnt -[0000:00]-+-00.0 Intel Corporation 5500 I/O Hub to ESI Port [8086:3403] +-01.0-[01]--+-00.0 Broadcom Corporation NetXtreme II BCM5709S Gigabit Ethernet [14e4:163a] | \-00.1 Broadcom Corporation NetXtreme II BCM5709S Gigabit Ethernet [14e4:163a] +-07.0-[03-09]----00.0-[04-09]--+-00.0-[05]-- | +-01.0-[06]-- | +-02.0-[07]-- | +-04.0-[08]--+-00.0 Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] | | \-00.1 Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] | \-10.0-[09]----00.0 LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] +-14.0 Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers [8086:342e] +-14.1 Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers [8086:3422] +-14.2 Intel Corporation 7500/5520/5500/X58 I/O Hub Control Sta= tus and RAS Registers [8086:3423] +-16.0 Intel Corporation 5520/5500/X58 Chipset QuickData Techno= logy Device [8086:3430] +-16.1 Intel Corporation 5520/5500/X58 Chipset QuickData Techno= logy Device [8086:3431] +-16.2 Intel Corporation 5520/5500/X58 Chipset QuickData Techno= logy Device [8086:3432] +-16.3 Intel Corporation 5520/5500/X58 Chipset QuickData Techno= logy Device [8086:3433] +-16.4 Intel Corporation 5520/5500/X58 Chipset QuickData Techno= logy Device [8086:3429] +-16.5 Intel Corporation 5520/5500/X58 Chipset QuickData Techno= logy Device [8086:342a] +-16.6 Intel Corporation 5520/5500/X58 Chipset QuickData Techno= logy Device [8086:342b] +-16.7 Intel Corporation 5520/5500/X58 Chipset QuickData Techno= logy Device [8086:342c] +-1a.0 Intel Corporation 82801I (ICH9 Family) USB UHCI Controll= er #4 [8086:2937] +-1a.1 Intel Corporation 82801I (ICH9 Family) USB UHCI Controll= er #5 [8086:2938] +-1a.7 Intel Corporation 82801I (ICH9 Family) USB2 EHCI Control= ler #2 [8086:293c] +-1c.0-[02]----00.0 LSI Logic / Symbios Logic MegaRAID SAS 1078 [1000:0060] +-1d.0 Intel Corporation 82801I (ICH9 Family) USB UHCI Controll= er #1 [8086:2934] +-1d.1 Intel Corporation 82801I (ICH9 Family) USB UHCI Controll= er #2 [8086:2935] +-1d.2 Intel Corporation 82801I (ICH9 Family) USB UHCI Controll= er #3 [8086:2936] +-1d.7 Intel Corporation 82801I (ICH9 Family) USB2 EHCI Control= ler #1 [8086:293a] +-1e.0-[0a]----03.0 Matrox Electronics Systems Ltd. MGA G200eW WPCM450 [102b:0532] +-1f.0 Intel Corporation 82801IB (ICH9) LPC Interface Controller [8086:2918] \-1f.5 Intel Corporation 82801I (ICH9 Family) 2 port SATA Controller [IDE mode] [8086:2926] 2) This is the same command in 13.0, same hardware: lspci -vvnnt -[0000:00]-+-00.0 Intel Corporation 5500 I/O Hub to ESI Port [8086:3403] +-01.0-[01]--+-00.0 Broadcom Inc. and subsidiaries NetXtreme II BCM5709S Gigabit Ethernet [14e4:163a] | \-00.1 Broadcom Inc. and subsidiaries NetXtreme II BCM5709S Gigabit Ethernet [14e4:163a] +-07.0-[03-09]----00.0-[04-09]--+-00.0-[05]-- | +-01.0-[06]-- | +-02.0-[07]-- | +-04.0-[08]-- | \-10.0-[09]-- +-14.0 Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers [8086:342e] +-14.1 Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers [8086:3422] +-14.2 Intel Corporation 7500/5520/5500/X58 I/O Hub Control Sta= tus and RAS Registers [8086:3423] +-16.0 Intel Corporation 5520/5500/X58 Chipset QuickData Techno= logy Device [8086:3430] +-16.1 Intel Corporation 5520/5500/X58 Chipset QuickData Techno= logy Device [8086:3431] +-16.2 Intel Corporation 5520/5500/X58 Chipset QuickData Techno= logy Device [8086:3432] +-16.3 Intel Corporation 5520/5500/X58 Chipset QuickData Techno= logy Device [8086:3433] +-16.4 Intel Corporation 5520/5500/X58 Chipset QuickData Techno= logy Device [8086:3429] +-16.5 Intel Corporation 5520/5500/X58 Chipset QuickData Techno= logy Device [8086:342a] +-16.6 Intel Corporation 5520/5500/X58 Chipset QuickData Techno= logy Device [8086:342b] +-16.7 Intel Corporation 5520/5500/X58 Chipset QuickData Techno= logy Device [8086:342c] +-1a.0 Intel Corporation 82801I (ICH9 Family) USB UHCI Controll= er #4 [8086:2937] +-1a.1 Intel Corporation 82801I (ICH9 Family) USB UHCI Controll= er #5 [8086:2938] +-1a.7 Intel Corporation 82801I (ICH9 Family) USB2 EHCI Control= ler #2 [8086:293c] +-1c.0-[02]----00.0 Broadcom / LSI MegaRAID SAS 1078 [1000:0060] +-1d.0 Intel Corporation 82801I (ICH9 Family) USB UHCI Controll= er #1 [8086:2934] +-1d.1 Intel Corporation 82801I (ICH9 Family) USB UHCI Controll= er #2 [8086:2935] +-1d.2 Intel Corporation 82801I (ICH9 Family) USB UHCI Controll= er #3 [8086:2936] +-1d.7 Intel Corporation 82801I (ICH9 Family) USB2 EHCI Control= ler #1 [8086:293a] +-1e.0-[0a]----03.0 Matrox Electronics Systems Ltd. MGA G200eW WPCM450 [102b:0532] +-1f.0 Intel Corporation 82801IB (ICH9) LPC Interface Controller [8086:2918] \-1f.5 Intel Corporation 82801I (ICH9 Family) 2 port SATA Controller [IDE mode] [8086:2926] 3) on 13.0 a quick summary: lspci -M 00:00.0 Host bridge: Intel Corporation 5500 I/O Hub to ESI Port (rev 13) 00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13) ## 00:01.0 is a bridge from 00 to 01-01 00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13) ## 00:07.0 is a bridge from 00 to 03-09 00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 13) 00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch = Pad Registers (rev 13) 00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13) 00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13) 00:16.1 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13) 00:16.2 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13) 00:16.3 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13) 00:16.4 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13) 00:16.5 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13) 00:16.6 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13) 00:16.7 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13) 00:1a.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 02) 00:1a.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 02) 00:1a.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 02) 00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port= 1 (rev 02) ## 00:1c.0 is a bridge from 00 to 02-02 00:1d.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 02) 00:1d.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 02) 00:1d.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 02) 00:1d.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 02) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) ## 00:1e.0 is a bridge from 00 to 0a-0a 00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controll= er (rev 02) 00:1f.5 IDE interface: Intel Corporation 82801I (ICH9 Family) 2 port SATA Controller [IDE mode] (rev 02) 01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709S Gigabit Ethernet (rev 20) 01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709S Gigabit Ethernet (rev 20) 02:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS 1078 (rev 04) 03:00.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Expr= ess Gen 2 (5.0 GT/s) Switch (rev aa) ## 03:00.0 is a bridge from 03 to 04-09 04:00.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Expr= ess Gen 2 (5.0 GT/s) Switch (rev aa) ## 04:00.0 is a bridge from 04 to 05-05 04:01.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Expr= ess Gen 2 (5.0 GT/s) Switch (rev aa) ## 04:01.0 is a bridge from 04 to 06-06 04:02.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Expr= ess Gen 2 (5.0 GT/s) Switch (rev aa) ## 04:02.0 is a bridge from 04 to 07-07 04:04.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Expr= ess Gen 2 (5.0 GT/s) Switch (rev aa) ## 04:04.0 is a bridge from 04 to 08-08 04:10.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Expr= ess Gen 2 (5.0 GT/s) Switch (rev aa) ## 04:10.0 is a bridge from 04 to 09-09 0a:03.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200= eW WPCM450 (rev 0a) Summary of buses: 00: Primary host bus 1e.0 Bridge to 0a-0a 1c.0 Bridge to 02-02 07.0 Bridge to 03-09 01.0 Bridge to 01-01 01: Entered via 00:01.0 02: Entered via 00:1c.0 03: Entered via 00:07.0 00.0 Bridge to 04-09 04: Entered via 03:00.0 10.0 Bridge to 09-09 04.0 Bridge to 08-08 02.0 Bridge to 07-07 01.0 Bridge to 06-06 00.0 Bridge to 05-05 0a: Entered via 00:1e.0 4) Now a dump of the switch I think is in front of my devices in question: lspci -vv -s :04:10.0 04:10.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Expr= ess Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=3Dfast >TAbort- <T= Abort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 255 Bus: primary=3D04, secondary=3D09, subordinate=3D09, sec-latency=3D0 I/O behind bridge: 0000e000-0000efff [size=3D4K] Memory behind bridge: df000000-df1fffff [size=3D2M] Prefetchable memory behind bridge: [disabled] Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=3Dfast >TAbort- <T= Abort- <MAbort- <SERR- <PERR- BridgeCtl: Parity+ SERR+ NoISA+ VGA- VGA16- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=3D0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=3D0 DScale=3D0 PME- Capabilities: [48] MSI: Enable+ Count=3D1/8 Maskable+ 64bit+ Address: 00000000fee22000 Data: 0030 Masking: 00000000 Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00 DevCap: MaxPayload 2048 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq+ RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 256 bytes, MaxReadReq 128 bytes DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- LnkCap: Port #16, Speed 5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <2us, L1 <2us ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp- LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s (ok), Width x8 (downgraded) TrErr- Train- SlotClk- DLActive+ BWMgmt- ABWMgmt- SltCap: AttnBtn+ PwrCtrl+ MRL+ AttnInd+ PwrInd+ HotPlug+ Surprise- Slot #240, PowerLimit 25.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn+ PwrFlt+ MRL+ PresDet+ CmdCplt+ HPI= rq+ LinkChg+ Control: AttnInd Off, PwrInd Off, Power+ Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock- Changed: MRL- PresDet- LinkState- DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR- 10BitTagComp- 10BitTagReq- OBFF Not Supported, Ext= Fmt- EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- ARIFwd+ AtomicOpsCap: Routing- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled, ARIFwd- AtomicOpsCtl: EgressBlck- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDi= s-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationCompl= ete- EqualizationPhase1- EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch Capabilities: [100 v1] Device Serial Number aa-86-00-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol+ UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ AdvNonFatalErr+ AERCap: First Error Pointer: 1f, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn- MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 00000000 00000000 00000000 00000000 Capabilities: [138 v1] Power Budgeting <?> Capabilities: [148 v1] Virtual Channel Caps: LPEVC=3D0 RefClk=3D100ns PATEntryBits=3D8 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=3DFixed Status: InProgress- VC0: Caps: PATOffset=3D03 MaxTimeSlots=3D1 RejSnoopTra= ns- Arb: Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR25= 6- Ctrl: Enable+ ID=3D0 ArbSelect=3DWRR64 TC/VC=3Dff Status: NegoPending- InProgress- Port Arbitration Table <?> Capabilities: [e00 v1] Multicast McastCap: MaxGroups 64, ECRCRegen+ McastCtl: NumGroups 1, Enable- McastBAR: IndexPos 0, BaseAddr 0000000000000000 McastReceiveVec: 0000000000000000 McastBlockAllVec: 0000000000000000 McastBlockUntransVec: 0000000000000000 McastOverlayBAR: OverlaySize 0 (disabled), BaseAddr 0000000000000000 Capabilities: [f24 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamF= wd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamF= wd- EgressCtrl- DirectTrans- Capabilities: [b70 v1] Vendor Specific Information: ID=3D0001 Rev= =3D0 Len=3D010 <?> 5) Back to 10.2 to dump all the bridges: root@pool31:~ # lspci -vv | grep bridge 00:00.0 Host bridge: Intel Corporation 5500 I/O Hub to ESI Port (rev 13) 00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13) (prog-if 00 [Normal decode]) Memory behind bridge: da000000-ddffffff 00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13) (prog-if 00 [Normal decode]) I/O behind bridge: 0000e000-0000efff Memory behind bridge: df000000-df2fffff 00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port= 1 (rev 02) (prog-if 00 [Normal decode]) I/O behind bridge: 0000f000-0000ffff Memory behind bridge: df300000-df3fffff 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) (prog-if 01 [Subtractive decode]) Memory behind bridge: de000000-deffffff Prefetchable memory behind bridge: 00000000d9800000-00000000d9ffffff 00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controll= er (rev 02) 03:00.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Expr= ess Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode]) I/O behind bridge: 0000e000-0000efff Memory behind bridge: df000000-df1fffff 04:00.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Expr= ess Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode]) 04:01.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Expr= ess Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode]) 04:02.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Expr= ess Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode]) 04:04.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Expr= ess Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode]) 04:10.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Expr= ess Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode]) I/O behind bridge: 0000e000-0000efff Memory behind bridge: df000000-df1fffff=20=20 After this I think I'm just generating noise, as I'm not sure what info is needed. I await instruction.=20 Thank you for reading this far. --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-256264-227-UtdO3XMXAu>