Date: Mon, 24 Jul 2000 18:09:18 +1000 (EST) From: Bruce Evans <bde@zeta.org.au> To: Greg Lehey <grog@lemis.com> Cc: Matthew Dillon <dillon@apollo.backplane.com>, Chuck Paterson <cp@bsdi.com>, David Greenman <dg@root.com>, freebsd-smp@FreeBSD.ORG Subject: Re: : ipending (was: SMP progress (was: Stepping on Toes)) Message-ID: <Pine.BSF.4.21.0007241652480.785-100000@besplex.bde.org> In-Reply-To: <20000724124520.F82241@wantadilla.lemis.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, 24 Jul 2000, Greg Lehey wrote:
> On Sunday, 23 July 2000 at 3:36:39 +1000, Bruce Evans wrote:
> > On Sat, 22 Jul 2000, Matthew Dillon wrote:
> >> I think you still have to mask level interrupts, otherwise you won't
> >> be able to sti. Some subsystems may generate a phenominal number
> >> of interrupts while the interrupt routine is running -- for example,
> > ^^^ another
> >> the serial ports.
> >
> > E.g., one serial port interrupting every 87 usec gives about 50
> > interrupts while the keyboard interrupt handler is busy-waiting to
> > program the keyboard LEDs.
>
> Ugh. Are keyboards that slow? But is that such a big problem? The
Yes, keyboards are slow, and keyboard LED programming is sloppy. All
PIO devices doing large i/o's or busy waiting have the same problem
(perhaps for only hundreds of usec instead of several msec). Other
examples: IDE drives in PIO mode and the printer driver.
> interrupts occur at the same rate when no keyboard interrupt is
> running. The interrupt only schedules the handler thread. If it's
> already scheduled, it's very fast.
The point is that we must do lots of fiddling with the PIC to allow
the keyboard interrupt handler/thread to be interrupted at all.
> On Saturday, 22 July 2000 at 10:57:40 -0600, Chuck Paterson wrote:
> >
> > When I read you mail I didn't answer because what you said
> > sounded right. But, what was I thinking. The short answer is Matt
> > is correct.
> > ...
> > Matt's comment about bunches of extra level triggered
> > interrupts being a problem is something that is going to have to be
> > looked into with BSD/OS. The reason we don't mask the interrupts now
> > is that doing the actual masking operation is soooo expensive. I
> > suspect we will want to mark which edge triggered interrupts we
level?
> > which to mask.
FreeBSD does some lazy masking of interrupts in the SMP case. I think
this amounts to detecting which interrupts should be masked, masking
them after they repeat, and then forgetting that they should have been
masked before they repeat.
> My understanding is that the threaded interrupt stuff splits
> interrupts (potentially) into two separate processing steps:
>
> 1. Receiving the interrupt, handling the PIC/APIC, and scheduling an
> interrupt thread.
>
> 2. Processing the interrupt thread.
>
> I understand that we would do (1) immediately and (2) when we have
> time for it, but as quickly as possible. Sure, while we're handling a
> slow operation like the keyboard LEDs, a lot of other interrupts can
> come in. But the interrupt rate won't change, the serial interrupts
> still come in at 11.5 kHz. All that happens there is that step (1)
> will be executed each time, just like it would be at any other time.
Except for fast interrupts, up to 99% of the overhead for step (1) is
currently avoided by not touching the PIC. This shouldn't change
(on i386's). This is not very portable. I'm not sure about the APIC
case, and i386-PIC style fast interrupts are not implementable in
general. E.g., suppose interrupts masks are actually levels and the
clock interrupt is hardwired to a level higher than that of serial
interrupts. Then fast (serial) interrupts can't interrupt slow
(clock) interrupts. This is essentially the default state for the
PIC, with the hardwiring especially braindamaged, and we fiddle with
the PIC masks to get away from it.
> The other issue is that we should potentially be able to handle one
> interrupt in one processor and another in a different processor. I
> know that there are problems in this attitude, and that we have
> deferred it, but I can't see how this can work at all if we mask the
> interrupts.
Yes, concurrency is limited by the number of hardware interrupt priority
levels or bits in the hardware interrupt masks.
> Looking at the code we have at the moment, the "slow" interrupts write
> two values to the PIC, one to set the mask and one to EOI, before
More precisely:
- one to set the mask in the slave PIC if irq >= 8
- one to set the mask in the master PIC
- one to send EOI to the slave PIC if irq >= 8 and not AUTO_EOI_2
- one to send EOI to the master PIC if not AUTO_EOI_1
> processing the interrupt, and another after the interrupt to unmask
> the interrupt:
More precisely:
- one to clear the mask in the slave PIC if irq >= 8
- one to clear the mask in the master PIC
> This happens on every interrupt, and as Bruce points out, the writes
> take a long time. By comparison, the code I have now is much shorter
> and contains only the minimum 1 outb instruction:
>
> Xintr3:
> pushl $0
> pushl $0
> pushal
> pushl %ds
> pushl %es
> pushl %fs
> mov $0x10 ,%ax
> mov %ax,%ds
> mov %ax,%es
> mov %ax,%fs
> movb $0x20 ,%al
> outb %al,$0x020
> incb intr_nesting_level
> Xresume3:
> pushl 3 IRQ number
> sti
> call sched_ithd
> jmp doreti
But this doesn't actually work. It fails in most cases for level triggered
interrupts and works in most cases for edge triggered interrupts. With
level triggered interrupts that stay on, it gives endless recursion after
the "sti". With edge triggered interrupts, the recursion may be limited
by the hardware not toggling the irq too often. The lazy masking that I
mentioned above avoids the setting the (A)PIC masks initially by detecting
the recursion and recovering from it by setting the masks later, etc. It
can almost be used to simplify the code as above. Unfortunately, this ends
up being more complicated, of course.
Bruce
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-smp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.BSF.4.21.0007241652480.785-100000>