From owner-freebsd-drivers@FreeBSD.ORG  Fri Apr 15 16:42:41 2011
Return-Path: <owner-freebsd-drivers@FreeBSD.ORG>
Delivered-To: freebsd-drivers@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 7109C106566B
	for <freebsd-drivers@freebsd.org>; Fri, 15 Apr 2011 16:42:41 +0000 (UTC)
	(envelope-from jhb@freebsd.org)
Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42])
	by mx1.freebsd.org (Postfix) with ESMTP id 48ED08FC1D
	for <freebsd-drivers@freebsd.org>; Fri, 15 Apr 2011 16:42:41 +0000 (UTC)
Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net
	[66.111.2.69])
	by cyrus.watson.org (Postfix) with ESMTPSA id DFDBE46B09;
	Fri, 15 Apr 2011 12:42:40 -0400 (EDT)
Received: from jhbbsd.localnet (unknown [209.249.190.124])
	by bigwig.baldwin.cx (Postfix) with ESMTPSA id 709EE8A01B;
	Fri, 15 Apr 2011 12:42:40 -0400 (EDT)
From: John Baldwin <jhb@freebsd.org>
To: freebsd-drivers@freebsd.org
Date: Fri, 15 Apr 2011 12:11:49 -0400
User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110325; KDE/4.5.5; amd64; ; )
References: <BANLkTim9WWQDjAJsHNNDRTN+nTXbxAonFA@mail.gmail.com>
In-Reply-To: <BANLkTim9WWQDjAJsHNNDRTN+nTXbxAonFA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: Text/Plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
Message-Id: <201104151211.49148.jhb@freebsd.org>
X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6
	(bigwig.baldwin.cx); Fri, 15 Apr 2011 12:42:40 -0400 (EDT)
Cc: Bret Ketchum <bcketchum@gmail.com>
Subject: Re: MSI interrupts.
X-BeenThere: freebsd-drivers@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Writing device drivers for FreeBSD <freebsd-drivers.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-drivers>, 
	<mailto:freebsd-drivers-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-drivers>
List-Post: <mailto:freebsd-drivers@freebsd.org>
List-Help: <mailto:freebsd-drivers-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-drivers>,
	<mailto:freebsd-drivers-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 15 Apr 2011 16:42:41 -0000

On Tuesday, April 12, 2011 9:16:16 am Bret Ketchum wrote:
>     I've a roll-your-own driver for FreeBSD 8.x that uses MSI interrupts for
> PCI-E HBAs where one or more will be installed in a off-the-shelf amd64
> pizza box. The driver is using bus_setup_intr() and depending upon the slots
> the HBAs are install I see log messages from apic_alloc_vectors(), for
> example:
> 
> Apr 12 06:44:15 mfsbsd kernel: xxxpci10: attempting to allocate 1 MSI
> vectors (16 supported)
> Apr 12 06:44:15 mfsbsd kernel: APIC: Couldn't find APIC vectors for 1 IRQs
> Apr 12 06:44:15 mfsbsd kernel: ioapic1: routing intpin 13 (PCI IRQ 37) to
> lapic 0 vector 59
> 
>     Using vmstat -ia:
> 
> interrupt                          total       rate
> irq37: xxxpci10                       74          0
> 
>     The problem appears to be that HBA interrupts are not being delivered to
> the driver. If I swap cards around in slots I can eliminate the message and:
> 
> Apr 12 06:44:15 mfsbsd kernel: msi: routing MSI IRQ 266 to local APIC 0
> vector 80
> 
>     And interrupts appear to be delivered properly. Before I dive in, can
> anyone explain this behavior?

Hmm, can you capture messages with bootverbose enabled?  A full boot -v dmesg 
might be useful as well.

-- 
John Baldwin