From owner-svn-src-all@FreeBSD.ORG Wed Jun 17 15:47:32 2009 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EE8A61065675; Wed, 17 Jun 2009 15:47:32 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id C03528FC2C; Wed, 17 Jun 2009 15:47:32 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 74A9C46C1F; Wed, 17 Jun 2009 11:47:32 -0400 (EDT) Received: from jhbbsd.hudson-trading.com (unknown [209.249.190.8]) by bigwig.baldwin.cx (Postfix) with ESMTPA id 52BD68A073; Wed, 17 Jun 2009 11:47:31 -0400 (EDT) From: John Baldwin To: Jeff Roberson Date: Wed, 17 Jun 2009 11:47:26 -0400 User-Agent: KMail/1.9.7 References: <200901290922.n0T9MuJ4047351@svn.freebsd.org> In-Reply-To: <200901290922.n0T9MuJ4047351@svn.freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200906171147.26968.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0.1 (bigwig.baldwin.cx); Wed, 17 Jun 2009 11:47:31 -0400 (EDT) X-Virus-Scanned: clamav-milter 0.95.1 at bigwig.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-2.5 required=4.2 tests=AWL,BAYES_00,RDNS_NONE autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on bigwig.baldwin.cx Cc: svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org Subject: Re: svn commit: r187880 - in head/sys: amd64/amd64 amd64/include i386/i386 i386/include X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Jun 2009 15:47:34 -0000 On Thursday 29 January 2009 4:22:56 am Jeff Roberson wrote: > Author: jeff > Date: Thu Jan 29 09:22:56 2009 > New Revision: 187880 > URL: http://svn.freebsd.org/changeset/base/187880 > > Log: > - Allocate apic vectors on a per-cpu basis. This allows us to allocate > more irqs as we have more cpus. This is principally useful on systems > with msi devices which may want many irqs per-cpu. > > Discussed with: jhb > Sponsored by: Nokia It turns out that this completely breaks devices using multiple MSI (not MSI-X) messages. In MSI-X each interrupt has a separate address and data register, so each one can have an independent IDT vector and CPU. This is not the case with MSI. With MSI you only have a single address and data register set. If you have multiple messages, the OS is required to leave the bottom N (log_2 (count)) bits set to zero and the individual message is encoded by via those bottom N bits. This requires that multiple MSI messages be treated as a single group that all go to the same CPU. Furthermore, the block of IDT vectors for the group has to be aligned on an N bit boundary. This was the purpose of the apic_alloc_vectors() function you removed. I think that part of the change will need to be reverted. msi_alloc() will probably need to preallocate vectors again rather than deferring them to msi_enable_intr(). Also, we can move MSI groups around. I had changes in a p4 tree to basically fail to bind messages that weren't the first message in such a group. Those can be revived for this purpose. I think it is currently accidentally working for mav@'s AHCI tests because he does a bus_setup_intr() of all the interrupts in sequence during attach. That causes the IDT vectors to happen to all be contiguous on CPU 0 during boot. It would fail if the driver was kldloaded. Also, he must currently just be very lucky that the first IDT vector he allocates is properly aligned. -- John Baldwin