Date: Fri, 6 Sep 2002 14:00:12 +0200 From: Bernd Walter <ticso@cicely9.cicely.de> To: Terry Lambert <tlambert2@mindspring.com> Cc: ticso@cicely.de, John Baldwin <jhb@FreeBSD.ORG>, Andrew Gallatin <gallatin@cs.duke.edu>, freebsd-alpha@FreeBSD.ORG Subject: Re: ithread preemption Message-ID: <20020906120011.GO13050@cicely9.cicely.de> In-Reply-To: <3D78925A.DAA13463@mindspring.com> References: <XFMail.20020905163105.jhb@FreeBSD.org> <3D78098B.CEBF13EC@mindspring.com> <20020906090517.GI13050@cicely9.cicely.de> <3D78925A.DAA13463@mindspring.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Sep 06, 2002 at 04:32:42AM -0700, Terry Lambert wrote: > Bernd Walter wrote: > > Interrupts are disabled globaly on alpha too. > > The only platform where we disable on the CPU is the PC164 as > > a workaround, but this system is UP. > > This misses the point. The point is whether or not ithreads > run to completion on the processor to which the interrupt is > delivered. What I missed is why you think this is different with APICs on i386. Well I have to say that I don't know much about APICs. > The problem in this case on the Alpha is that interrupts are > routed through the PAL code on a particular processor, and > so the return has to be to the PAL code on the same processor, > because there is a context cons'ed up for it that have to be > destructed on the same CPU where it was cons'ed. That's the most logical (if not the only) theorie so far. > > > If x86 NUMA systems are ever supported, or in some cases the > > > AMD Alpha-derived arbitration is used, there could be a problem > > > (but not for the same reasons as on the Alpha). > > > > I expect ithreads to be one of the less critical points on NUMA. > > The problem in this case is that you have to do completion > counting on the ISR's for any given interrupt. For example, > on an SMP box running two CPUs where IRQ A is a shared interrupt > for two devices, then if you want to dispatch for one device to > the first CPU, and the second device to the second, then when > you reenable interrupts depends on all ISRs having run to > completion. So you have to set a global count to "2", for the > number of ISRs that have to run, and then run each one on a CPU, > decrement the count, and when the count goes from 1->0, reenable > the interrupt. If you want more than one ithread per intline I agree. > For NUMA, it really depends on the cluster architecture. If you > have devices associated based on CPU clusters, that's one thing; > it's an easy call. If you have it on the basis of adjacency, it > is not so easy a call, because the adjacency det for two different > devices can be only partially intersecting (i.e. dev 1 is associated > with CPU's 1,2,3,4, dev 2 with CPU's 5,6,7,8, and dev 3 with CPU's > 3,4,5,6). This gets into the same issue as the Alpha, where the > CPU to take the interrupt has to complete the interrupt, only in > this case, you are talking abbout the associativity set. I see. > Yeah, most likely this won't be a problem, but then that's likely > the same thing that was thought when the current Alpha problem > was introduced. Ack. > > > Using all the same primitives for ithreads as for threads will > > > lead to things like a shared IRQ which comes true for multiple > > > devices will need to count down to enable, if the ithreads for > > > servicing are run simultaneously on seperate CPUs. This also > > > implies a global lock on enabling a newly arrived device that's > > > sharing an already allocated IRQ, and inter-CPU contnetion on > > > any IRQ whose use cont is != 1. > > > > Currently shared interrupts also share an ithread. > > Yeah; this isn't very efficient with only 4 interrupts and a lot > of PCI cards. After all you are writing to the alpha list :) On alphas we typically have 4 intlines per slot on the primary busses. Only small machines like LCA share 4 lines for all slots. As typical chips take only one intline we even don't share intlines over PCI-PCI bridges with up to 4 chips. But generally I see the point that when the handling for one device is blocked the service for devices sharing the same intline are also blocked - there is a good reason that blocking in device drivers has to be short timed. > > > Interrupt threads make some very messy things elegant, but they > > > do so at the cost of moving the mess elsewhere. If you have to > > > characterize it, call it "The law of conservation of inelegance". > > > 8-). The only thing that ever saves you from it is a top-down > > > design. > > > > Possible, at least that's our current design. > > I'm not sure if this is true. As you said, currently, shared > interrupts also share an ithread. > > Thread affinity as an explicit hard0coded attribute is probably > not the correct fix for the current Alpha probems. It will make > it harder to do it right later (just like it's harder to fix a > foundation after you've built a house on it). I can't ague on that yet, because I understand the reason why returning back to PAL could be on a different CPU only since a few hours. -- B.Walter COSMO-Project http://www.cosmo-project.de ticso@cicely.de Usergroup info@cosmo-project.de To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-alpha" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20020906120011.GO13050>