Date: Thu, 1 Sep 2011 23:44:18 +0400 From: Sergey Kandaurov <pluknet@gmail.com> To: John Baldwin <jhb@freebsd.org> Cc: stable@freebsd.org Subject: Re: mfi(4) patch to add MSI-X support, possibly address command timeouts Message-ID: <CAE-mSOKbRKxN2vUXFxA87xhT7C8m235dTC6yWzd2wZ%2B1_pCyXg@mail.gmail.com> In-Reply-To: <201109010707.11442.jhb@freebsd.org> References: <201108311334.10804.jhb@freebsd.org> <201108311717.03127.jhb@freebsd.org> <CAE-mSOK8Cx=PKz65Wh52qcauPEitqNmjvZhhgxH_fsaoKzhvZg@mail.gmail.com> <201109010707.11442.jhb@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 1 September 2011 15:07, John Baldwin <jhb@freebsd.org> wrote: > On Wednesday, August 31, 2011 5:27:09 pm Sergey Kandaurov wrote: >> On 1 September 2011 01:17, John Baldwin <jhb@freebsd.org> wrote: >> > On Wednesday, August 31, 2011 3:24:12 pm Sergey Kandaurov wrote: >> >> On 31 August 2011 21:34, John Baldwin <jhb@freebsd.org> wrote: >> >> > I'd like some folks to test a patch to the mfi(4) driver that may h= elp to >> >> > address issues several folks have reported. =A0The patch does two t= hings, first >> >> > it adds some dummy reads of PCI registers when checking device stat= us in the >> >> > interrupt handler to "flush" the writes to ACK interrupts. =A0The L= inux >> >> > megaraid-sas driver uses this approach and some folks have tested a= patch from >> >> > Scott Long which had a somewhat similar effect. =A0Second, it enabl= es the use of >> >> > MSI-X interrupts for many newer devices. >> >> > >> >> > The patch is available below and at www.freebsd.org/~jhb/patches/mf= i.patch >> >> >> >> mfi0: <LSI MegaSAS Gen2> port 0x3000-0x30ff mem >> >> 0x9dd40000-0x9dd43fff,0x9dd00000-0x9dd3ffff irq 26 at device 0.0 on >> >> pci26 >> >> mfi0: Using MSI-X >> >> mfi0: Megaraid SAS driver Ver 3.00 >> >> >> >> However, booting never finishes ending up with: >> >> mfi0: COMMAND 0xffffff8000b3a550 TIMEOUT AFTER 58 SECONDS >> >> mfi0: COMMAND 0xffffff8000b3a550 TIMEOUT AFTER 88 SECONDS >> >> mfi0: COMMAND 0xffffff8000b3a550 TIMEOUT AFTER 118 SECONDS >> >> mfi0: COMMAND 0xffffff8000b3a550 TIMEOUT AFTER 148 SECONDS >> >> mfi0: COMMAND 0xffffff8000b3a550 TIMEOUT AFTER 179 SECONDS >> >> mfi0: COMMAND 0xffffff8000b3a550 TIMEOUT AFTER 209 SECONDS >> > >> > Did this work fine without the patch? >> >> Yes, like a charm. > > Hmm, the Linux driver definitely uses MSI-X for your board. > > What chipset do you have, That is gen2 IBM ServeRAID M5015 SAS/SATA Controller. So the chipset should be LSI SAS2108, as per its datasheet. > and does your system use MSI IRQs for any other > devices? Yep, 2-port i82576 and 4-port i82576, each port uses 9 vectors. Also 4 bce (BCM5709) embedded ports register their irq above 255, one irq per interface, so I suspect they should use some kind of MSI/MSIX, too. Indeed, verbose dmesg confirms that. bce0: <Broadcom NetXtreme II BCM5709 1000Base-T (C0)> mem 0x96000000-0x97ffffff irq 28 at device 0.0 on pci11 bce0: Reserved 0x2000000 bytes for rid 0x10 type 3 at 0x96000000 bce0: attempting to allocate 1 MSI vectors (16 supported) msi: routing MSI IRQ 256 to local APIC 0 vector 64 bce0: using IRQ 256 for MSI miibus0: <MII bus> on bce0 brgphy0: <BCM5709C 10/100/1000baseTX PHY> PHY 1 on miibus0 brgphy0: OUI 0x0050ef, model 0x003c, rev. 8 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bce0: [MPSAFE] bce0: [ITHREAD] bce0: ASIC (0x57092003); Rev (C0); Bus (PCIe x2, 5Gbps); B/C (5.2.2); Flags (MSI|MFW); MFW (NCSI 2.0.10) [...] igb0: <Intel(R) PRO/1000 Network Connection version - 2.0.7> port 0x4020-0x403f mem 0x9dc20000-0x9dc3ffff,0x9d800000-0x9dbfffff,0x9dc44000-0x9dc47fff irq 24 at device 0.0 on pci21 igb0: Reserved 0x20000 bytes for rid 0x10 type 3 at 0x9dc20000 igb0: Reserved 0x4000 bytes for rid 0x1c type 3 at 0x9dc44000 igb0: attempting to allocate 9 MSI-X vectors (10 supported) msi: routing MSI-X IRQ 260 to local APIC 0 vector 49 msi: routing MSI-X IRQ 261 to local APIC 0 vector 50 msi: routing MSI-X IRQ 262 to local APIC 0 vector 51 msi: routing MSI-X IRQ 263 to local APIC 0 vector 52 msi: routing MSI-X IRQ 264 to local APIC 0 vector 53 msi: routing MSI-X IRQ 265 to local APIC 0 vector 54 msi: routing MSI-X IRQ 266 to local APIC 0 vector 55 msi: routing MSI-X IRQ 267 to local APIC 0 vector 56 msi: routing MSI-X IRQ 268 to local APIC 0 vector 57 igb0: using IRQs 260-268 for MSI-X igb0: Using MSIX interrupts with 9 vectors igb0: [MPSAFE] igb0: [ITHREAD] igb0: [MPSAFE] igb0: [ITHREAD] igb0: [MPSAFE] igb0: [ITHREAD] igb0: [MPSAFE] igb0: [ITHREAD] igb0: [MPSAFE] igb0: [ITHREAD] igb0: [MPSAFE] igb0: [ITHREAD] igb0: [MPSAFE] igb0: [ITHREAD] igb0: [MPSAFE] igb0: [ITHREAD] igb0: [MPSAFE] igb0: [ITHREAD] > > Also, can you break into ddb and do 'show intrcnt' to see if you are gett= ing > interrupts for the mfi device? Captured shortly after boot hangs. db> show intrcnt irq4: uart0 1 irq14: ata0 35 irq17: uhci0 uhci2+ 16 cpu0: timer 148278 irq278: mfi0 2 And a bit later. irq4: uart0 3 irq14: ata0 35 irq17: uhci0 uhci2+ 16 cpu0: timer 187327 irq278: mfi0 2 --=20 wbr, pluknet
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAE-mSOKbRKxN2vUXFxA87xhT7C8m235dTC6yWzd2wZ%2B1_pCyXg>