From owner-freebsd-stable@FreeBSD.ORG Thu Sep 1 19:44:20 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 48637106564A; Thu, 1 Sep 2011 19:44:20 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from mail-yw0-f54.google.com (mail-yw0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id EA9DE8FC12; Thu, 1 Sep 2011 19:44:19 +0000 (UTC) Received: by ywo32 with SMTP id 32so2121488ywo.13 for ; Thu, 01 Sep 2011 12:44:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=dokeaIwdFwubzvoPidwVg1/beWEdRcm631S1pvpb6c4=; b=cD0QxR8fHpmAFMTPrBMNRyOOU8xDZGbI1eZUzPuHV7TdvNO74HLZt625Q0ENZ2hVc7 njyvcVQBNdBUXTu0NhrbOohPAeXr0OXywQOv0tc3PX1xEp5yoU60/0cPU3CXgLf86QC0 OLkBCUu6/MAkMJt4FFB18ShNmKh7ums5Ulch8= MIME-Version: 1.0 Received: by 10.150.134.20 with SMTP id h20mr438652ybd.285.1314906258995; Thu, 01 Sep 2011 12:44:18 -0700 (PDT) Received: by 10.150.53.2 with HTTP; Thu, 1 Sep 2011 12:44:18 -0700 (PDT) In-Reply-To: <201109010707.11442.jhb@freebsd.org> References: <201108311334.10804.jhb@freebsd.org> <201108311717.03127.jhb@freebsd.org> <201109010707.11442.jhb@freebsd.org> Date: Thu, 1 Sep 2011 23:44:18 +0400 Message-ID: From: Sergey Kandaurov To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: stable@freebsd.org Subject: Re: mfi(4) patch to add MSI-X support, possibly address command timeouts X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Sep 2011 19:44:20 -0000 On 1 September 2011 15:07, John Baldwin wrote: > On Wednesday, August 31, 2011 5:27:09 pm Sergey Kandaurov wrote: >> On 1 September 2011 01:17, John Baldwin wrote: >> > On Wednesday, August 31, 2011 3:24:12 pm Sergey Kandaurov wrote: >> >> On 31 August 2011 21:34, John Baldwin wrote: >> >> > I'd like some folks to test a patch to the mfi(4) driver that may h= elp to >> >> > address issues several folks have reported. =A0The patch does two t= hings, first >> >> > it adds some dummy reads of PCI registers when checking device stat= us in the >> >> > interrupt handler to "flush" the writes to ACK interrupts. =A0The L= inux >> >> > megaraid-sas driver uses this approach and some folks have tested a= patch from >> >> > Scott Long which had a somewhat similar effect. =A0Second, it enabl= es the use of >> >> > MSI-X interrupts for many newer devices. >> >> > >> >> > The patch is available below and at www.freebsd.org/~jhb/patches/mf= i.patch >> >> >> >> mfi0: port 0x3000-0x30ff mem >> >> 0x9dd40000-0x9dd43fff,0x9dd00000-0x9dd3ffff irq 26 at device 0.0 on >> >> pci26 >> >> mfi0: Using MSI-X >> >> mfi0: Megaraid SAS driver Ver 3.00 >> >> >> >> However, booting never finishes ending up with: >> >> mfi0: COMMAND 0xffffff8000b3a550 TIMEOUT AFTER 58 SECONDS >> >> mfi0: COMMAND 0xffffff8000b3a550 TIMEOUT AFTER 88 SECONDS >> >> mfi0: COMMAND 0xffffff8000b3a550 TIMEOUT AFTER 118 SECONDS >> >> mfi0: COMMAND 0xffffff8000b3a550 TIMEOUT AFTER 148 SECONDS >> >> mfi0: COMMAND 0xffffff8000b3a550 TIMEOUT AFTER 179 SECONDS >> >> mfi0: COMMAND 0xffffff8000b3a550 TIMEOUT AFTER 209 SECONDS >> > >> > Did this work fine without the patch? >> >> Yes, like a charm. > > Hmm, the Linux driver definitely uses MSI-X for your board. > > What chipset do you have, That is gen2 IBM ServeRAID M5015 SAS/SATA Controller. So the chipset should be LSI SAS2108, as per its datasheet. > and does your system use MSI IRQs for any other > devices? Yep, 2-port i82576 and 4-port i82576, each port uses 9 vectors. Also 4 bce (BCM5709) embedded ports register their irq above 255, one irq per interface, so I suspect they should use some kind of MSI/MSIX, too. Indeed, verbose dmesg confirms that. bce0: mem 0x96000000-0x97ffffff irq 28 at device 0.0 on pci11 bce0: Reserved 0x2000000 bytes for rid 0x10 type 3 at 0x96000000 bce0: attempting to allocate 1 MSI vectors (16 supported) msi: routing MSI IRQ 256 to local APIC 0 vector 64 bce0: using IRQ 256 for MSI miibus0: on bce0 brgphy0: PHY 1 on miibus0 brgphy0: OUI 0x0050ef, model 0x003c, rev. 8 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bce0: [MPSAFE] bce0: [ITHREAD] bce0: ASIC (0x57092003); Rev (C0); Bus (PCIe x2, 5Gbps); B/C (5.2.2); Flags (MSI|MFW); MFW (NCSI 2.0.10) [...] igb0: port 0x4020-0x403f mem 0x9dc20000-0x9dc3ffff,0x9d800000-0x9dbfffff,0x9dc44000-0x9dc47fff irq 24 at device 0.0 on pci21 igb0: Reserved 0x20000 bytes for rid 0x10 type 3 at 0x9dc20000 igb0: Reserved 0x4000 bytes for rid 0x1c type 3 at 0x9dc44000 igb0: attempting to allocate 9 MSI-X vectors (10 supported) msi: routing MSI-X IRQ 260 to local APIC 0 vector 49 msi: routing MSI-X IRQ 261 to local APIC 0 vector 50 msi: routing MSI-X IRQ 262 to local APIC 0 vector 51 msi: routing MSI-X IRQ 263 to local APIC 0 vector 52 msi: routing MSI-X IRQ 264 to local APIC 0 vector 53 msi: routing MSI-X IRQ 265 to local APIC 0 vector 54 msi: routing MSI-X IRQ 266 to local APIC 0 vector 55 msi: routing MSI-X IRQ 267 to local APIC 0 vector 56 msi: routing MSI-X IRQ 268 to local APIC 0 vector 57 igb0: using IRQs 260-268 for MSI-X igb0: Using MSIX interrupts with 9 vectors igb0: [MPSAFE] igb0: [ITHREAD] igb0: [MPSAFE] igb0: [ITHREAD] igb0: [MPSAFE] igb0: [ITHREAD] igb0: [MPSAFE] igb0: [ITHREAD] igb0: [MPSAFE] igb0: [ITHREAD] igb0: [MPSAFE] igb0: [ITHREAD] igb0: [MPSAFE] igb0: [ITHREAD] igb0: [MPSAFE] igb0: [ITHREAD] igb0: [MPSAFE] igb0: [ITHREAD] > > Also, can you break into ddb and do 'show intrcnt' to see if you are gett= ing > interrupts for the mfi device? Captured shortly after boot hangs. db> show intrcnt irq4: uart0 1 irq14: ata0 35 irq17: uhci0 uhci2+ 16 cpu0: timer 148278 irq278: mfi0 2 And a bit later. irq4: uart0 3 irq14: ata0 35 irq17: uhci0 uhci2+ 16 cpu0: timer 187327 irq278: mfi0 2 --=20 wbr, pluknet