Date: Mon, 19 Mar 2001 09:29:18 -0800 (PST) From: John Baldwin <jhb@FreeBSD.org> To: Bruce Evans <bde@zeta.org.au> Cc: Matthew Jacob <mjacob@feral.com>, arch@FreeBSD.org Subject: Re: 'final' man pages Message-ID: <XFMail.010319092918.jhb@FreeBSD.org> In-Reply-To: <Pine.BSF.4.21.0103191531450.33784-100000@besplex.bde.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 19-Mar-01 Bruce Evans wrote: >> >> If a machine supports an external mechanism for disabling interrupts such >> >> as >> >> a >> >> PIC mask, then we can simply disable individual interrupts that way until >> >> the >> >> ithread associated with an interrupt source completes. Note that we only >> > >> > And this is what we do on i386's. >> >> And alpha's. > > I couldn't find where this is done on alphas. I asked you in private mail, > but you didn't reply on this point (although you replied on many others, > thanks :). Woops. The way this is done is with the it_enable() and it_disable() function pointers in struct ithd. They are set to platform-specific functions to talk to the ICU or PCI controller for that platform to enable and disable an individual interrupt source. > Anyway, there must be an external mechanism for the SMP case to work > right. An internal (per-CPU) mechanism just can't be used to control > other CPUs, except grossly and inefficiently by propagating the internal > mechanism to all CPUs, or by halting or spinning all CPUs except the one > which took the interrupt. So the rest of this discussion only applies > to single-CPU systems with no external mechanism. The internal mechanism > would probably look much like an spl mechanism, so it is inherently > mismatched with our SMP-friendly mechanisms. Yes, SMP w/o an external mechanism gets ugly. Well, you actually don't need to propagate the mask. If another CPU gets an interrupt, then it will see that the ithread is already scheduled and/or running, just mark it_need and leave the interrupt masked on the current CPU. Hmm, but then it won't be unmasked except on the CPU that finishes the ithread, so you would need to propagate unmasks at least if we ever do SMP on an arch with no external mechanism. >> >> really need to do this for level-triggered interrupts. If we don't have >> >> this >> >> available and are forced to (ab)use the CPU interrupt level to mask >> >> interrupts, >> >> then we instead disable all other device interrupts on this CPU when an >> >> interrupt comes in until the associated ithread finishes. >> > >> > We should disable precisely all the interrupts with the same or lower >> > priority while running the ithread. >> >> Errrm, what about a MP system that wires all interrupts to one processor (I >> think some alphas may be this brain-damaged) we would want lower priority >> interrupts to fire so that their threads would be scheduled and then picked >> up >> by another CPU and run. Ithread priorities will ensure that ithreads are >> processed in the right order. No hardware masking is needed. > > See above. I can't see how to do this without an external mechanism > (perhaps one transparent to the O/S). The interrupt that we're > servicing must be stopped from repeating somehow. For MP systems, it > must also be stopped from affecting other CPUs. Hmm. An internal > mechanism could be made to work at the cost of propagating it to all > CPUs provided it is mask-based and not level-based. We would just use > it to mask active interrupts. For this to work right, there must be at > most one interrupt source per mask bit. When this is not possible > (e.g., for i386's with ICUs and lots of interrupt sources), ithread > priorities can't work right unless the ithreads for all the interrupt > sources that share a mask bit have the same priority. I was mainly pointing out that if we can mask the current level, we don't need to mask all lower levels as well, as we may want a lower level ithread to run on a given CPU. However, the current CPU does always need to block the current interrupt if it is level triggered. Using an external mask to do this is much more efficient. Without it you would degenerate to blocking all other interrupts each time an interrupt came in until its thread had finished. >> >> Also, we lower the IPL back to 0 before >> >> running an ithread, so that only the actual interrupt stack that >> >> schedules >> >> an >> >> ithread to run runs at a raised IPL. >> > >> > This is a bug. In general, we can't lower the IPL below the level >> > needed to mask the interrupt that we're handling, because level-triggered >> > interrupts will repeat endlessly. I'm not sure why this isn't fatal >> > for alphas. It may be handled by "lazy interrupt masking" -- let the >> > interrupt repeat and mask it properly on the first repetition so that >> > it won't repeat again. >> >> We mask the interrupts in the PIC just like on x86, otherwise a PCI >> interrupt >> can cause problems because it is level triggered. > > Oops. For some reason I couldn't find this masking using grep. It is even > spelled the same as on i386's (ICU, not PIC :). > > My main point is that the "MI" code in ithread_schedule() isn't actually MI. > It assumes that there is an external mechanism. Well, it uses a MD hook, mtx_intr_enable(), and if you want to leave the implicit intr_disable() from the interrupt code in effect you simply don't have it do anything. This would basically leave interrupts disabled until the MD enable interrupt function called from ithread_loop() reenabled interrupts. > Does the restriction to one device IPL on alphas have much effect given > that the ICU does the actual masking? How normal are ICUs on alphas? It actually ends up being very similar to the i386. The IPL is raised on the interrupt stack (just like PSL_I is cleared on i386 on the interrupt stack) and it is lowered otherwise. We depend on the ICU masking, which is more important on the alpha as all PCI interrupts (except maybe for one of the archs (pc164?) which just uses a ICU that is broken) are level-triggered. > Getting back to the original subject of changing the API for > disable_intr(), etc., all these layers of interrupt mechanisms are > confusing. I think most of them should be kept out of MI code. The > disable_intr() level is perhaps the only layer simple enough to have > an MI API. Yes, and so far it is the only one used in MI code, as the mutexes use the foo_intr() API, and mtx_intr_enable() is actually a slight extension to that API. (It changes the saved state returned from disable_intr() so that interrupts will always be enabled when it is passed to restore_intr()). >> In theory we don't even need >> to mask edge triggered interrupts at all as all that would happen is that >> the >> ithread's it_need bit would get set again if it is already running. > > Almost in practice too. I implemented this for the spl mechanism and > still rotted bits for it in my tree. SMPog implemented it too. It > should work for the same reasons as it used to. At least my version > of it masks any interrupts if they occur again, to handle level-triggered > interrupts transparently. But it just wastes time to let interrupts > repeat when you know that they will. Yep, might as well keep them masked until you are ready for another one. > Bruce -- John Baldwin <jhb@FreeBSD.org> -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?XFMail.010319092918.jhb>