From owner-freebsd-current@FreeBSD.ORG Mon Jun 8 09:15:38 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B2418106566C; Mon, 8 Jun 2009 09:15:38 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from cmail.optima.ua (cmail.optima.ua [195.248.191.121]) by mx1.freebsd.org (Postfix) with ESMTP id 062C68FC1D; Mon, 8 Jun 2009 09:15:37 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from [212.86.226.226] (account mav@alkar.net HELO mavbook.mavhome.dp.ua) by cmail.optima.ua (CommuniGate Pro SMTP 5.2.9) with ESMTPSA id 245158151; Mon, 08 Jun 2009 12:15:34 +0300 Message-ID: <4A2CD6AC.80407@FreeBSD.org> Date: Mon, 08 Jun 2009 12:15:24 +0300 From: Alexander Motin User-Agent: Thunderbird 2.0.0.21 (X11/20090405) MIME-Version: 1.0 To: FreeBSD-Current , freebsd-arch@freebsd.org Content-Type: multipart/mixed; boundary="------------010407070706090009080901" Cc: Subject: Multiple MSI on SMP, misrouting or misunderstanding? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Jun 2009 09:15:39 -0000 This is a multi-part message in MIME format. --------------010407070706090009080901 Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: 7bit Hi. While experimenting with using multiple MSIs support on AHCI controller I have got the problem. When system boots as UP - everything is fine, driver allocates all available 16 MSIs and works. But when system booted as SMP, interrupts begin to behave strange: I didn't receive expected AHCI IRQs, but instead receive IRQ1 interrupts of atkbd0, while I have no PS/2 keyboard/mouse attached. As I have found, problem appears due to IRQ rebalancing between CPUs. As I have got, MSI requires that all vectors from the same group to be allocated sequentially, but IRQ rebalancing breaks correct order, that happed during initial allocation. I was quite surprised by this issue. If multiple MSI vectors of the same device have to be allocated sequentially and bound to the same CPU, then they will be unable to give any SMP scalability benefits. Am I right, or there is some special technique expected to be used to somehow distribute grouped MSI vectors between CPUs which we don't have? I have made small patch that denies rebalancing for grouped MSIs, to make them work at least somehow. It works fine for me, but I am not sure that it is the best solution. -- Alexander Motin --------------010407070706090009080901 Content-Type: text/plain; name="msi.c.reassign.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="msi.c.reassign.patch" --- msi.c.prev 2009-06-08 11:30:13.000000000 +0300 +++ msi.c 2009-06-08 11:30:06.000000000 +0300 @@ -210,6 +210,8 @@ msi_assign_cpu(struct intsrc *isrc, u_in old_id = msi->msi_cpu; if (old_vector && old_id == apic_id) return; + if (old_vector && !msi->msi_msix && msi->msi_first->msi_count > 1) + return; /* Allocate IDT vector on this cpu. */ vector = apic_alloc_vector(apic_id, msi->msi_irq); if (vector == 0) --------------010407070706090009080901--