From owner-freebsd-arch@FreeBSD.ORG Mon Jun 8 15:33:59 2009 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3FE151065675; Mon, 8 Jun 2009 15:33:59 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 11D628FC14; Mon, 8 Jun 2009 15:33:59 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id BB26446B4C; Mon, 8 Jun 2009 11:33:58 -0400 (EDT) Received: from jhbbsd.hudson-trading.com (unknown [209.249.190.8]) by bigwig.baldwin.cx (Postfix) with ESMTPA id 4D0A28A049; Mon, 8 Jun 2009 11:33:57 -0400 (EDT) From: John Baldwin To: freebsd-current@freebsd.org Date: Mon, 8 Jun 2009 11:16:40 -0400 User-Agent: KMail/1.9.7 References: <4A2CD6AC.80407@FreeBSD.org> In-Reply-To: <4A2CD6AC.80407@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200906081116.40462.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0.1 (bigwig.baldwin.cx); Mon, 08 Jun 2009 11:33:57 -0400 (EDT) X-Virus-Scanned: clamav-milter 0.95.1 at bigwig.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-2.5 required=4.2 tests=AWL,BAYES_00,RDNS_NONE autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on bigwig.baldwin.cx Cc: Alexander Motin , freebsd-arch@freebsd.org Subject: Re: Multiple MSI on SMP, misrouting or misunderstanding? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Jun 2009 15:33:59 -0000 On Monday 08 June 2009 5:15:24 am Alexander Motin wrote: > Hi. > > While experimenting with using multiple MSIs support on AHCI controller > I have got the problem. When system boots as UP - everything is fine, > driver allocates all available 16 MSIs and works. But when system booted > as SMP, interrupts begin to behave strange: I didn't receive expected > AHCI IRQs, but instead receive IRQ1 interrupts of atkbd0, while I have > no PS/2 keyboard/mouse attached. > > As I have found, problem appears due to IRQ rebalancing between CPUs. As > I have got, MSI requires that all vectors from the same group to be > allocated sequentially, but IRQ rebalancing breaks correct order, that > happed during initial allocation. > > I was quite surprised by this issue. If multiple MSI vectors of the same > device have to be allocated sequentially and bound to the same CPU, then > they will be unable to give any SMP scalability benefits. Am I right, or > there is some special technique expected to be used to somehow > distribute grouped MSI vectors between CPUs which we don't have? > > I have made small patch that denies rebalancing for grouped MSIs, to > make them work at least somehow. It works fine for me, but I am not sure > that it is the best solution. It is a limitation of MSI. With MSI, you have a single address register for the entire group of messages (the individual messages are just distinguished by toggling the lower N bits in the message data register). On x86 the address register includes the APIC ID. That means that all of the messages get sent to the same CPU. With MSI-X, there is a table with separate address and data registers for each message. This allows a driver to distribute interrupts across CPUs. I had old patches prior to the per-CPU IDT stuff to handle this quirk of MSI groups. The approach I used there was that I would only allow reassigning of the entire group by assigning to the first interrupt in the group. With per-CPU IDTs that gets trickier though as you need to allocate a whole block of aligned, consecutive IDT vectors in the new CPU. -- John Baldwin