From owner-freebsd-alpha Fri Sep 6 5: 1: 4 2002 Delivered-To: freebsd-alpha@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DC4E637B400; Fri, 6 Sep 2002 05:00:58 -0700 (PDT) Received: from srv1.cosmo-project.de (srv1.cosmo-project.de [213.83.6.106]) by mx1.FreeBSD.org (Postfix) with ESMTP id CA2F443E75; Fri, 6 Sep 2002 05:00:57 -0700 (PDT) (envelope-from ticso@cicely9.cicely.de) Received: from cicely5.cicely.de (cicely5.cicely.de [IPv6:3ffe:400:8d0:301:200:92ff:fe9b:20e7]) by srv1.cosmo-project.de (8.12.5/8.12.5) with ESMTP id g86C0i6K082274 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=OK); Fri, 6 Sep 2002 14:00:48 +0200 (CEST) (envelope-from ticso@cicely9.cicely.de) Received: from cicely9.cicely.de (cicely9.cicely.de [IPv6:3ffe:400:8d0:301:210:5aff:fe30:1c1a]) by cicely5.cicely.de (8.12.6/8.12.6) with ESMTP id g86C0gNk006366 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Fri, 6 Sep 2002 14:00:43 +0200 (CEST) (envelope-from ticso@cicely9.cicely.de) Received: from cicely9.cicely.de (localhost [127.0.0.1]) by cicely9.cicely.de (8.12.6/8.12.5) with ESMTP id g86C0ED3017145; Fri, 6 Sep 2002 14:00:14 +0200 (CEST) (envelope-from ticso@cicely9.cicely.de) Received: (from ticso@localhost) by cicely9.cicely.de (8.12.6/8.12.6/Submit) id g86C0CH4017144; Fri, 6 Sep 2002 14:00:12 +0200 (CEST) Date: Fri, 6 Sep 2002 14:00:12 +0200 From: Bernd Walter To: Terry Lambert Cc: ticso@cicely.de, John Baldwin , Andrew Gallatin , freebsd-alpha@FreeBSD.ORG Subject: Re: ithread preemption Message-ID: <20020906120011.GO13050@cicely9.cicely.de> Reply-To: ticso@cicely.de References: <3D78098B.CEBF13EC@mindspring.com> <20020906090517.GI13050@cicely9.cicely.de> <3D78925A.DAA13463@mindspring.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3D78925A.DAA13463@mindspring.com> X-Operating-System: FreeBSD cicely9.cicely.de 5.0-CURRENT alpha User-Agent: Mutt/1.5.1i Sender: owner-freebsd-alpha@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Fri, Sep 06, 2002 at 04:32:42AM -0700, Terry Lambert wrote: > Bernd Walter wrote: > > Interrupts are disabled globaly on alpha too. > > The only platform where we disable on the CPU is the PC164 as > > a workaround, but this system is UP. > > This misses the point. The point is whether or not ithreads > run to completion on the processor to which the interrupt is > delivered. What I missed is why you think this is different with APICs on i386. Well I have to say that I don't know much about APICs. > The problem in this case on the Alpha is that interrupts are > routed through the PAL code on a particular processor, and > so the return has to be to the PAL code on the same processor, > because there is a context cons'ed up for it that have to be > destructed on the same CPU where it was cons'ed. That's the most logical (if not the only) theorie so far. > > > If x86 NUMA systems are ever supported, or in some cases the > > > AMD Alpha-derived arbitration is used, there could be a problem > > > (but not for the same reasons as on the Alpha). > > > > I expect ithreads to be one of the less critical points on NUMA. > > The problem in this case is that you have to do completion > counting on the ISR's for any given interrupt. For example, > on an SMP box running two CPUs where IRQ A is a shared interrupt > for two devices, then if you want to dispatch for one device to > the first CPU, and the second device to the second, then when > you reenable interrupts depends on all ISRs having run to > completion. So you have to set a global count to "2", for the > number of ISRs that have to run, and then run each one on a CPU, > decrement the count, and when the count goes from 1->0, reenable > the interrupt. If you want more than one ithread per intline I agree. > For NUMA, it really depends on the cluster architecture. If you > have devices associated based on CPU clusters, that's one thing; > it's an easy call. If you have it on the basis of adjacency, it > is not so easy a call, because the adjacency det for two different > devices can be only partially intersecting (i.e. dev 1 is associated > with CPU's 1,2,3,4, dev 2 with CPU's 5,6,7,8, and dev 3 with CPU's > 3,4,5,6). This gets into the same issue as the Alpha, where the > CPU to take the interrupt has to complete the interrupt, only in > this case, you are talking abbout the associativity set. I see. > Yeah, most likely this won't be a problem, but then that's likely > the same thing that was thought when the current Alpha problem > was introduced. Ack. > > > Using all the same primitives for ithreads as for threads will > > > lead to things like a shared IRQ which comes true for multiple > > > devices will need to count down to enable, if the ithreads for > > > servicing are run simultaneously on seperate CPUs. This also > > > implies a global lock on enabling a newly arrived device that's > > > sharing an already allocated IRQ, and inter-CPU contnetion on > > > any IRQ whose use cont is != 1. > > > > Currently shared interrupts also share an ithread. > > Yeah; this isn't very efficient with only 4 interrupts and a lot > of PCI cards. After all you are writing to the alpha list :) On alphas we typically have 4 intlines per slot on the primary busses. Only small machines like LCA share 4 lines for all slots. As typical chips take only one intline we even don't share intlines over PCI-PCI bridges with up to 4 chips. But generally I see the point that when the handling for one device is blocked the service for devices sharing the same intline are also blocked - there is a good reason that blocking in device drivers has to be short timed. > > > Interrupt threads make some very messy things elegant, but they > > > do so at the cost of moving the mess elsewhere. If you have to > > > characterize it, call it "The law of conservation of inelegance". > > > 8-). The only thing that ever saves you from it is a top-down > > > design. > > > > Possible, at least that's our current design. > > I'm not sure if this is true. As you said, currently, shared > interrupts also share an ithread. > > Thread affinity as an explicit hard0coded attribute is probably > not the correct fix for the current Alpha probems. It will make > it harder to do it right later (just like it's harder to fix a > foundation after you've built a house on it). I can't ague on that yet, because I understand the reason why returning back to PAL could be on a different CPU only since a few hours. -- B.Walter COSMO-Project http://www.cosmo-project.de ticso@cicely.de Usergroup info@cosmo-project.de To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-alpha" in the body of the message