Date: Mon, 19 Mar 2001 16:50:59 +1100 (EST) From: Bruce Evans <bde@zeta.org.au> To: John Baldwin <jhb@FreeBSD.org> Cc: arch@FreeBSD.org, Matthew Jacob <mjacob@feral.com> Subject: Re: 'final' man pages Message-ID: <Pine.BSF.4.21.0103191531450.33784-100000@besplex.bde.org> In-Reply-To: <XFMail.010318160847.jhb@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, 18 Mar 2001, John Baldwin wrote: > On 18-Mar-01 Bruce Evans wrote: > > On Sun, 18 Mar 2001, John Baldwin wrote: > > > >> On 18-Mar-01 Matthew Jacob wrote: > >> > Bruce- the following paragrahph is really hard to parse. I *think* you're > >> > saying that we cannot guarantee nesting. That's a fair addition. Anything > >> > else? > >> > >> Hmm, as long as you pair state changes with restores you can handle nesting > >> in > >> the same way that spl's did, i.e.: > >> > >> x = intr_disable() > >> ... > >> y = intr_disable() > >> .... > >> restore_intr(y) > >> .... > >> restore_intr(x) > >> > >> will DTRT and not enable interrupts until after the second restore_intr(). > >> You do have to pair them up much like lock acquire/releases. > > > > The problem is if interrupts are controlled by masks. A lower priority > > interrupt may occur while the above code is running (unless you dumb down > > the masks a little to give levels). The handler for this interrupt must > > keep it disabled and not proceed far before continuing with the above > > code (since the above code has higher priority). This invalidates x and > > y if x and y contain the mask. > > They won't be the same masks. Well, even if they are, you would just have > intr_disable() disable all interrupts, then no further interrupts can trigger > that might affect the masks. The point of intr_disable() is that the masks > can't change because we can't be preempted. Having an interrupt come in is > preemption, and this API explicitly prevents that. Having > critcal_enter()/critical_exit() might be better names for this. Oops. I was thinking of spls for some reason. > >> If a machine supports an external mechanism for disabling interrupts such as > >> a > >> PIC mask, then we can simply disable individual interrupts that way until > >> the > >> ithread associated with an interrupt source completes. Note that we only > > > > And this is what we do on i386's. > > And alpha's. I couldn't find where this is done on alphas. I asked you in private mail, but you didn't reply on this point (although you replied on many others, thanks :). Anyway, there must be an external mechanism for the SMP case to work right. An internal (per-CPU) mechanism just can't be used to control other CPUs, except grossly and inefficiently by propagating the internal mechanism to all CPUs, or by halting or spinning all CPUs except the one which took the interrupt. So the rest of this discussion only applies to single-CPU systems with no external mechanism. The internal mechanism would probably look much like an spl mechanism, so it is inherently mismatched with our SMP-friendly mechanisms. > >> really need to do this for level-triggered interrupts. If we don't have > >> this > >> available and are forced to (ab)use the CPU interrupt level to mask > >> interrupts, > >> then we instead disable all other device interrupts on this CPU when an > >> interrupt comes in until the associated ithread finishes. > > > > We should disable precisely all the interrupts with the same or lower > > priority while running the ithread. > > Errrm, what about a MP system that wires all interrupts to one processor (I > think some alphas may be this brain-damaged) we would want lower priority > interrupts to fire so that their threads would be scheduled and then picked up > by another CPU and run. Ithread priorities will ensure that ithreads are > processed in the right order. No hardware masking is needed. See above. I can't see how to do this without an external mechanism (perhaps one transparent to the O/S). The interrupt that we're servicing must be stopped from repeating somehow. For MP systems, it must also be stopped from affecting other CPUs. Hmm. An internal mechanism could be made to work at the cost of propagating it to all CPUs provided it is mask-based and not level-based. We would just use it to mask active interrupts. For this to work right, there must be at most one interrupt source per mask bit. When this is not possible (e.g., for i386's with ICUs and lots of interrupt sources), ithread priorities can't work right unless the ithreads for all the interrupt sources that share a mask bit have the same priority. > >> Also, we lower the IPL back to 0 before > >> running an ithread, so that only the actual interrupt stack that schedules > >> an > >> ithread to run runs at a raised IPL. > > > > This is a bug. In general, we can't lower the IPL below the level > > needed to mask the interrupt that we're handling, because level-triggered > > interrupts will repeat endlessly. I'm not sure why this isn't fatal > > for alphas. It may be handled by "lazy interrupt masking" -- let the > > interrupt repeat and mask it properly on the first repetition so that > > it won't repeat again. > > We mask the interrupts in the PIC just like on x86, otherwise a PCI interrupt > can cause problems because it is level triggered. Oops. For some reason I couldn't find this masking using grep. It is even spelled the same as on i386's (ICU, not PIC :). My main point is that the "MI" code in ithread_schedule() isn't actually MI. It assumes that there is an external mechanism. Does the restriction to one device IPL on alphas have much effect given that the ICU does the actual masking? How normal are ICUs on alphas? Getting back to the original subject of changing the API for disable_intr(), etc., all these layers of interrupt mechanisms are confusing. I think most of them should be kept out of MI code. The disable_intr() level is perhaps the only layer simple enough to have an MI API. > In theory we don't even need > to mask edge triggered interrupts at all as all that would happen is that the > ithread's it_need bit would get set again if it is already running. Almost in practice too. I implemented this for the spl mechanism and still rotted bits for it in my tree. SMPog implemented it too. It should work for the same reasons as it used to. At least my version of it masks any interrupts if they occur again, to handle level-triggered interrupts transparently. But it just wastes time to let interrupts repeat when you know that they will. Bruce To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.BSF.4.21.0103191531450.33784-100000>