Date: Fri, 06 Sep 2002 04:32:42 -0700 From: Terry Lambert <tlambert2@mindspring.com> To: ticso@cicely.de Cc: John Baldwin <jhb@FreeBSD.ORG>, Andrew Gallatin <gallatin@cs.duke.edu>, freebsd-alpha@FreeBSD.ORG Subject: Re: ithread preemption Message-ID: <3D78925A.DAA13463@mindspring.com> References: <XFMail.20020905163105.jhb@FreeBSD.org> <3D78098B.CEBF13EC@mindspring.com> <20020906090517.GI13050@cicely9.cicely.de>
next in thread | previous in thread | raw e-mail | index | archive | help
Bernd Walter wrote: > Interrupts are disabled globaly on alpha too. > The only platform where we disable on the CPU is the PC164 as > a workaround, but this system is UP. This misses the point. The point is whether or not ithreads run to completion on the processor to which the interrupt is delivered. The problem in this case on the Alpha is that interrupts are routed through the PAL code on a particular processor, and so the return has to be to the PAL code on the same processor, because there is a context cons'ed up for it that have to be destructed on the same CPU where it was cons'ed. > > If x86 NUMA systems are ever supported, or in some cases the > > AMD Alpha-derived arbitration is used, there could be a problem > > (but not for the same reasons as on the Alpha). > > I expect ithreads to be one of the less critical points on NUMA. The problem in this case is that you have to do completion counting on the ISR's for any given interrupt. For example, on an SMP box running two CPUs where IRQ A is a shared interrupt for two devices, then if you want to dispatch for one device to the first CPU, and the second device to the second, then when you reenable interrupts depends on all ISRs having run to completion. So you have to set a global count to "2", for the number of ISRs that have to run, and then run each one on a CPU, decrement the count, and when the count goes from 1->0, reenable the interrupt. For NUMA, it really depends on the cluster architecture. If you have devices associated based on CPU clusters, that's one thing; it's an easy call. If you have it on the basis of adjacency, it is not so easy a call, because the adjacency det for two different devices can be only partially intersecting (i.e. dev 1 is associated with CPU's 1,2,3,4, dev 2 with CPU's 5,6,7,8, and dev 3 with CPU's 3,4,5,6). This gets into the same issue as the Alpha, where the CPU to take the interrupt has to complete the interrupt, only in this case, you are talking abbout the associativity set. Yeah, most likely this won't be a problem, but then that's likely the same thing that was thought when the current Alpha problem was introduced. > > Using all the same primitives for ithreads as for threads will > > lead to things like a shared IRQ which comes true for multiple > > devices will need to count down to enable, if the ithreads for > > servicing are run simultaneously on seperate CPUs. This also > > implies a global lock on enabling a newly arrived device that's > > sharing an already allocated IRQ, and inter-CPU contnetion on > > any IRQ whose use cont is != 1. > > Currently shared interrupts also share an ithread. Yeah; this isn't very efficient with only 4 interrupts and a lot of PCI cards. > > Interrupt threads make some very messy things elegant, but they > > do so at the cost of moving the mess elsewhere. If you have to > > characterize it, call it "The law of conservation of inelegance". > > 8-). The only thing that ever saves you from it is a top-down > > design. > > Possible, at least that's our current design. I'm not sure if this is true. As you said, currently, shared interrupts also share an ithread. Thread affinity as an explicit hard0coded attribute is probably not the correct fix for the current Alpha probems. It will make it harder to do it right later (just like it's harder to fix a foundation after you've built a house on it). -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-alpha" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3D78925A.DAA13463>