From owner-cvs-all@FreeBSD.ORG Tue Feb 28 22:49:45 2006 Return-Path: X-Original-To: cvs-all@FreeBSD.org Delivered-To: cvs-all@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5BA6C16A420; Tue, 28 Feb 2006 22:49:45 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id D6C0E43D46; Tue, 28 Feb 2006 22:49:44 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from [10.10.3.185] ([69.15.205.254]) (authenticated bits=0) by pooker.samsco.org (8.13.4/8.13.4) with ESMTP id k1SMngVr057827; Tue, 28 Feb 2006 15:49:42 -0700 (MST) (envelope-from scottl@samsco.org) Message-ID: <4404D37E.9040502@samsco.org> Date: Tue, 28 Feb 2006 15:49:34 -0700 From: Scott Long User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.12) Gecko/20060206 X-Accept-Language: en-us, en MIME-Version: 1.0 To: John Baldwin References: <200602282224.k1SMOtJt070241@repoman.freebsd.org> <200602281735.12240.jhb@freebsd.org> In-Reply-To: <200602281735.12240.jhb@freebsd.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.0 required=3.8 tests=none autolearn=failed version=3.1.0 X-Spam-Checker-Version: SpamAssassin 3.1.0 (2005-09-13) on pooker.samsco.org Cc: cvs-src@FreeBSD.org, src-committers@FreeBSD.org, cvs-all@FreeBSD.org Subject: Re: cvs commit: src/sys/amd64/amd64 intr_machdep.c io_apic.c local_apic.c mp_machdep.c src/sys/amd64/include apicvar.h intr_machdep.h src/sys/amd64/isa atpic.c src/sys/i386/i386 intr_machdep.c io_apic.c local_apic.c mp_machdep.c ... X-BeenThere: cvs-all@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CVS commit messages for the entire tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Feb 2006 22:49:46 -0000 John Baldwin wrote: > On Tuesday 28 February 2006 17:24, John Baldwin wrote: > >>jhb 2006-02-28 22:24:55 UTC >> >> FreeBSD src repository >> >> Modified files: >> sys/amd64/amd64 intr_machdep.c io_apic.c local_apic.c >> mp_machdep.c >> sys/amd64/include apicvar.h intr_machdep.h >> sys/amd64/isa atpic.c >> sys/i386/i386 intr_machdep.c io_apic.c local_apic.c >> mp_machdep.c >> sys/i386/include apicvar.h intr_machdep.h >> sys/i386/isa atpic.c >> Log: >> Rework how we wire up interrupt sources to CPUs: >> - Throw out all of the logical APIC ID stuff. The Intel docs are somewhat >> ambiguous, but it seems that the "flat" cluster model we are currently >> using is only supported on Pentium and P6 family CPUs. The other >> "hierarchy" cluster model that is supported on all Intel CPUs with >> local APICs is severely underdocumented. For example, it's not clear >> if the OS needs to glean the topology of the APIC hierarchy from >> somewhere (neither ACPI nor MP Table include it) and setup the logical >> clusters based on the physical hierarchy or not. Not only that, but on >> certain Intel chipsets, even though there were 4 CPUs in a logical >> cluster, all the interrupts were only sent to one CPU anyway. >> - We now bind interrupts to individual CPUs using physical addressing via >> the local APIC IDs. This code has also moved out of the ioapic PIC >> driver and into the common interrupt source code so that it can be >> shared with MSI interrupt sources since MSI is addressed to APICs the >> same way that I/O APIC pins are. > > - Use fixed delivery mode rather than low priority, as apparently low > priority mode only works with logical APIC IDs (though this is not > clearly documented that I've seen). Also, I've observed behavior where > low priority mode will deliver interrupts to a different CPU than the > one you've specifically routed the IRQ to using physical addressing > on certain Intel chipsets. FYI, we use low priority delivery method > with a wildcard physical address on all released versions of FreeBSD, > back to revision 1.1 of sys/i386/i386/mpapic.c. > >> - Interrupt source classes grow a new method pic_assign_cpu() to bind an >> interrupt source to a specific local APIC ID. >> - The SMP code now tells the interrupt code which CPUs are avaiable to >> handle interrupts in a simpler and more intuitive manner. For one thing, >> it means we could now choose to not route interrupts to HT cores if we >> wanted to (this code is currently in place in fact, but under an #if 0 >> for now). >> - For now we simply do static round-robin of IRQs to CPUs when the first >> interrupt handler just as before, with the change that IRQs are now >> bound to individual CPUs rather than groups of up to 4 CPUs. >> - Because the IRQ to CPU mapping has now been moved up a layer, it would >> be easier to manage this mapping from higher levels. For example, we >> could allow drivers to specify a CPU affinity map for their interrupts, >> or we could allow a userland tool to bind IRQs to specific CPUs. > > > FYI, I think I would prefer the latter, as a sys admin knows that he's > routing packets between two different network interfaces and thus wants > those devices on separate CPUs (for example), whereas the driver writer > isn't really in a position to know that sort of thing. > Where this is really useful is for people developing FreeBSD-based appliances that have very specific and fixed needs. Also, it's not so much important which CPU gets the interrupt as it is which CPU runs the ithread for that interrupt. I guess that you can get a little better latency by preempting directly from the low-level interrupt handler into the ithread, but I don't know if that is noticable noise above the cost of the context switch and inevitable lock operations and contention involved. Anyways, thanks for this work, it looks very promising. Scott