From owner-freebsd-drivers@FreeBSD.ORG Wed Apr 20 19:43:34 2011 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8032C106567D; Wed, 20 Apr 2011 19:43:34 +0000 (UTC) (envelope-from bcketchum@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id A11B08FC08; Wed, 20 Apr 2011 19:43:33 +0000 (UTC) Received: by bwz12 with SMTP id 12so1223036bwz.13 for ; Wed, 20 Apr 2011 12:43:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=ZRVxhfGU3G2wz71f9T2SngxojP8PO5OR5bieb+35fto=; b=i3dj1WZl+YEg9iZsVHaKA4jmxJcXnUpUEFdwhkHTxkZ/pomlm9P7483W/WQ1lMMj7G fHat6jdyf2gc+o+mBqI4H9sXI7sPB7BoHADckCz26s3OiXzt4SUZ6QIPdUQeHwZbVEO+ fop2YkfiIhK9dSS2/nhPGOJI43oR0njc5TSvs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=nm2xxB5XgR3cBVfUemsb4MZ40ME80Vn5mxh4u5altUG8G1pgeoiVD0H2VVt/UN6GZ1 u9Kor5/3MAf7fnFXjlwJvfwrr7U8YNxSDQ+DKlInd9lnPqvUqPygpbakGAJvcKGlN8uw dXlBLzrJnRYa4gfKbrtSLfP4+LQepYlsUMRTE= MIME-Version: 1.0 Received: by 10.204.8.141 with SMTP id h13mr379409bkh.64.1303328612368; Wed, 20 Apr 2011 12:43:32 -0700 (PDT) Received: by 10.204.36.77 with HTTP; Wed, 20 Apr 2011 12:43:32 -0700 (PDT) In-Reply-To: <201104151211.49148.jhb@freebsd.org> References: <201104151211.49148.jhb@freebsd.org> Date: Wed, 20 Apr 2011 14:43:32 -0500 Message-ID: From: Bret Ketchum To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-drivers@freebsd.org Subject: Re: MSI interrupts. X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Apr 2011 19:43:34 -0000 My response, which included dmesg output along with verbose boot messages was rejected by the mailing list moderator as too large so I'll reply with some of the interesting bits. Here is the MSI related mesages grepped out of the verbose boot messages: ioapic0: routing intpin 9 (ISA IRQ 9) to lapic 0 vector 48 MSI supports 2 messages, vector masks MSI supports 2 messages, vector masks MSI supports 2 messages, vector masks MSI supports 2 messages, vector masks MSI supports 2 messages, vector masks MSI supports 2 messages, vector masks MSI supports 1 message, 64 bit, vector masks MSI supports 1 message, 64 bit, vector masks igb0: attempting to allocate 5 MSI-X vectors (10 supported) msi: routing MSI-X IRQ 256 to local APIC 0 vector 49 msi: routing MSI-X IRQ 257 to local APIC 0 vector 50 msi: routing MSI-X IRQ 258 to local APIC 0 vector 51 msi: routing MSI-X IRQ 259 to local APIC 0 vector 52 msi: routing MSI-X IRQ 260 to local APIC 0 vector 53 igb0: Using MSIX interrupts with 5 vectors igb1: attempting to allocate 5 MSI-X vectors (10 supported) msi: routing MSI-X IRQ 261 to local APIC 0 vector 54 msi: routing MSI-X IRQ 262 to local APIC 0 vector 55 msi: routing MSI-X IRQ 263 to local APIC 0 vector 56 msi: routing MSI-X IRQ 264 to local APIC 0 vector 57 msi: routing MSI-X IRQ 265 to local APIC 0 vector 58 igb1: Using MSIX interrupts with 5 vectors cmlpci0: attempting to allocate 1 MSI vectors (32 supported) msi: routing MSI IRQ 266 to local APIC 0 vector 80 cmlpci1: attempting to allocate 1 MSI vectors (32 supported) msi: routing MSI IRQ 267 to local APIC 0 vector 112 cmlpci2: attempting to allocate 1 MSI vectors (32 supported) msi: routing MSI IRQ 268 to local APIC 0 vector 144 cmlpci3: attempting to allocate 1 MSI vectors (32 supported) msi: routing MSI IRQ 269 to local APIC 0 vector 176 cmlpci4: attempting to allocate 1 MSI vectors (16 supported) msi: routing MSI IRQ 270 to local APIC 0 vector 64 cmlpci5: attempting to allocate 1 MSI vectors (16 supported) msi: routing MSI IRQ 271 to local APIC 0 vector 96 cmlpci6: attempting to allocate 1 MSI vectors (16 supported) msi: routing MSI IRQ 272 to local APIC 0 vector 160 cmlpci7: attempting to allocate 1 MSI vectors (16 supported) msi: routing MSI IRQ 273 to local APIC 0 vector 192 ioapic0: routing intpin 16 (PCI IRQ 16) to lapic 0 vector 59 ioapic0: routing intpin 21 (PCI IRQ 21) to lapic 0 vector 60 ioapic0: routing intpin 18 (PCI IRQ 18) to lapic 0 vector 61 ioapic0: routing intpin 23 (PCI IRQ 23) to lapic 0 vector 62 ioapic0: routing intpin 19 (PCI IRQ 19) to lapic 0 vector 63 ioapic0: routing intpin 14 (ISA IRQ 14) to lapic 0 vector 65 ioapic0: routing intpin 15 (ISA IRQ 15) to lapic 0 vector 66 MSI supports 2 messages, vector masks MSI supports 2 messages, vector masks MSI supports 2 messages, vector masks MSI supports 2 messages, vector masks MSI supports 2 messages, vector masks MSI supports 2 messages, vector masks cmlpci8: attempting to allocate 1 MSI vectors (2 supported) msi: routing MSI IRQ 274 to local APIC 0 vector 68 MSI supports 4 messages, 64 bit, vector masks MSI supports 4 messages, 64 bit, vector masks MSI supports 4 messages, 64 bit, vector masks MSI supports 4 messages, 64 bit, vector masks MSI supports 16 messages, 64 bit, vector masks MSI supports 16 messages, 64 bit, vector masks cmlpci9: attempting to allocate 1 MSI vectors (16 supported) msi: routing MSI IRQ 275 to local APIC 0 vector 208 cmlpci10: attempting to allocate 1 MSI vectors (16 supported) msi: routing MSI IRQ 276 to local APIC 0 vector 224 MSI supports 16 messages, 64 bit, vector masks MSI supports 16 messages, 64 bit, vector masks cmlpci11: attempting to allocate 1 MSI vectors (16 supported) APIC: Couldn't find APIC vectors for 1 IRQs ioapic2: routing intpin 3 (PCI IRQ 51) to lapic 0 vector 67 cmlpci12: attempting to allocate 1 MSI vectors (16 supported) APIC: Couldn't find APIC vectors for 1 IRQs ioapic2: routing intpin 5 (PCI IRQ 53) to lapic 0 vector 69 ioapic0: routing intpin 4 (ISA IRQ 4) to lapic 0 vector 70 ioapic0: routing intpin 3 (ISA IRQ 3) to lapic 0 vector 71 ioapic0: routing intpin 1 (ISA IRQ 1) to lapic 0 vector 72 ioapic0: routing intpin 3 (ISA IRQ 3) to lapic 2 vector 48 ioapic0: routing intpin 4 (ISA IRQ 4) to lapic 4 vector 48 ioapic0: routing intpin 9 (ISA IRQ 9) to lapic 6 vector 48 ioapic0: routing intpin 15 (ISA IRQ 15) to lapic 2 vector 49 ioapic0: routing intpin 16 (PCI IRQ 16) to lapic 4 vector 49 ioapic0: routing intpin 18 (PCI IRQ 18) to lapic 6 vector 49 ioapic0: routing intpin 21 (PCI IRQ 21) to lapic 2 vector 50 ioapic0: routing intpin 23 (PCI IRQ 23) to lapic 4 vector 50 ioapic2: routing intpin 3 (PCI IRQ 51) to lapic 6 vector 50 msi: Assigning MSI-X IRQ 258 to local APIC 2 vector 51 msi: Assigning MSI-X IRQ 260 to local APIC 2 vector 52 msi: Assigning MSI-X IRQ 263 to local APIC 2 vector 53 msi: Assigning MSI-X IRQ 265 to local APIC 4 vector 51 msi: Assigning MSI IRQ 266 to local APIC 6 vector 51 msi: Assigning MSI IRQ 268 to local APIC 2 vector 54 msi: Assigning MSI IRQ 269 to local APIC 4 vector 52 msi: Assigning MSI IRQ 270 to local APIC 6 vector 52 msi: Assigning MSI IRQ 272 to local APIC 2 vector 55 msi: Assigning MSI IRQ 273 to local APIC 4 vector 53 msi: Assigning MSI IRQ 274 to local APIC 6 vector 53 msi: Assigning MSI IRQ 276 to local APIC 2 vector 56 vectors are allocated starting at 48. The first 48 (beginning at 0) are reserved IDT I/O interrupts. This is an amd64 build. The maximum that will be allocated is 192. Note that when requesting a single MSI interrupts apic_alloc_vectors actually looks for a range of free vectors based upon the number of vectors the PCI HBA supports. Yet subsequent calls to apic_alloc_vectors returns vectors within the already allocated range: msi: routing MSI IRQ 270 to local APIC 0 vector 64 cmlpci5: attempting to allocate 1 MSI vectors (16 supported) .... ioapic0: routing intpin 14 (ISA IRQ 14) to lapic 0 vector 65 ioapic0: routing intpin 15 (ISA IRQ 15) to lapic 0 vector 66 Question, why limit the allocation of vectors to a range supported by a PCI device when vectors within that range may be allocated in subsequent requests? In this case enough vectors are allocated that a supported range cannot be found: cmlpci11: attempting to allocate 1 MSI vectors (16 supported) APIC: Couldn't find APIC vectors for 1 IRQs On Fri, Apr 15, 2011 at 11:11 AM, John Baldwin wrote: > On Tuesday, April 12, 2011 9:16:16 am Bret Ketchum wrote: > > I've a roll-your-own driver for FreeBSD 8.x that uses MSI interrupts > for > > PCI-E HBAs where one or more will be installed in a off-the-shelf amd64 > > pizza box. The driver is using bus_setup_intr() and depending upon the > slots > > the HBAs are install I see log messages from apic_alloc_vectors(), for > > example: > > > > Apr 12 06:44:15 mfsbsd kernel: xxxpci10: attempting to allocate 1 MSI > > vectors (16 supported) > > Apr 12 06:44:15 mfsbsd kernel: APIC: Couldn't find APIC vectors for 1 > IRQs > > Apr 12 06:44:15 mfsbsd kernel: ioapic1: routing intpin 13 (PCI IRQ 37) to > > lapic 0 vector 59 > > > > Using vmstat -ia: > > > > interrupt total rate > > irq37: xxxpci10 74 0 > > > > The problem appears to be that HBA interrupts are not being delivered > to > > the driver. If I swap cards around in slots I can eliminate the message > and: > > > > Apr 12 06:44:15 mfsbsd kernel: msi: routing MSI IRQ 266 to local APIC 0 > > vector 80 > > > > And interrupts appear to be delivered properly. Before I dive in, can > > anyone explain this behavior? > > Hmm, can you capture messages with bootverbose enabled? A full boot -v > dmesg > might be useful as well. > > -- > John Baldwin >