Date: Wed, 1 Nov 2000 10:52:30 +0000 (GMT) From: Terry Lambert <tlambert@primenet.com> To: jhb@FreeBSD.ORG (John Baldwin) Cc: tlambert@primenet.com (Terry Lambert), freebsd-smp@FreeBSD.ORG, drony@spray.se Subject: Re: HLT Message-ID: <200011011052.DAA00688@usr02.primenet.com> In-Reply-To: <200011010509.VAA44996@john.baldwin.cx> from "John Baldwin" at Oct 31, 2000 09:09:04 PM
next in thread | previous in thread | raw e-mail | index | archive | help
> Err, no. This has nothing to do with Giant, zero, zilch, nada. Please do go > read the code before making such claims. It has everything to do with making > sure that you don't put a CPU to sleep when there is work for it to do, and > to make sure you don't increase interrupt latency by having to unnecessarily > use an IPI to wake up a sleeping CPU so it can run a thread that suddenly > appears. One can try hacking the code to enable hlt if one really wants it > (just change the #ifndef SMP to #ifndef SMP_XXX) but it will have a negative > effect on interrupt responsiveness when at least one CPU is idle. I thought the issue was a missing IPI in the scheduler code, when something became ready to run, where you could have a halted CPU waiting for work, and an active CPU that originated the work via handling an interrupt, and failed to IPI the halted CPU like it should to tell it that there were ready-to-run processes? It seems to me that we are not talking about increased interrupt latency, unless the halted CPY holds the BGL. I was under the impression that the halted CPU would wake up from the I/O APIC IPI, should an interrupt occur, since that's sent to all CPUs, and the system currently operates in virtual wire mode. Please help me understand why there is interrupt latency, if not for the BGL being held during HLT on idle? Are you perhaps talking about a missed opportunity latency, where the I/O APIC signals all CPUs, one takes the interrupt, and the process which will become ready to run as a result of completed interrupt processing is not yet marked ready to run (presumably by wakeup/wakeone from the interrupt handler), so the idle processor goes back to sleep, just when it's about to have work it _could_ do? I think the problem is that there is no IPI when the task is put on the scheduler read-to-run; I guess I just always sort of assumed that the reason for that was the BGL, not lack of an IPI instruction. I still think the fix for this is to make the idle task a real task, with a HLT. Then you differentiate the APIC IPI (virtual wire, interrupts to handle) from the scheduler IPI (wake up, you have work to do). This really gets down to wanting per CPU scheduler queues, so that the scheduler IPI can be targetted, instead of broadcast. If the CPU never mixes interrupt processing with scheduler checking (which it wouldn't have to, without a global scheduler queue), then you could always know that another CPU was idle, without having to lock, since it would have zero runnable tasks on its task list (idle would always be runnable, so if it isn't there, it must be running, and therefore the CPU must be halted, so doing a read is safe; even if you have a miss, it's not fatal, the CPU merely gets [non-broadcast] IPI'ed twice). Since interrupts can be farmed out to multiple CPUs, you would still have to lock the CPU out while manipulating its task list out from under it, but you could do that pretty elegantly by having unscheduled global read to run queues, to use for migration, and kicking the CPU to go get its own tasks, so it wouldn't have to self-lock, it'd only acquire the global migration queue lock until it was done migrating. If you had 32 of these, you'd have exhausted the APIC ID space, and could have per CPU ones, with little or no contention. So I guess I need you to clarify where the latency is coming from: the laws of physics causing a latency to be required, or the design being defective. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200011011052.DAA00688>