From owner-freebsd-smp Mon Jul 17 0: 0:53 2000 Delivered-To: freebsd-smp@freebsd.org Received: from wantadilla.lemis.com (wantadilla.lemis.com [192.109.197.80]) by hub.freebsd.org (Postfix) with ESMTP id 65F7337B7B4; Mon, 17 Jul 2000 00:00:43 -0700 (PDT) (envelope-from grog@wantadilla.lemis.com) Received: (from grog@localhost) by wantadilla.lemis.com (8.9.3/8.9.3) id QAA22205; Mon, 17 Jul 2000 16:30:38 +0930 (CST) (envelope-from grog) Date: Mon, 17 Jul 2000 16:30:38 +0930 From: Greg Lehey To: arch@FreeBSD.org, smp@FreeBSD.org Subject: Tidying up the interrupt registration process Message-ID: <20000717163038.J26231@wantadilla.lemis.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0pre2i Organization: LEMIS, PO Box 460, Echunga SA 5153, Australia Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 WWW-Home-Page: http://www.lemis.com/~grog X-PGP-Fingerprint: 6B 7B C3 8C 61 CD 54 AF 13 24 52 F8 6D A4 95 EF Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org I think I now understand how drivers register interrupts with newbus, and it seems that some tidying up is in order. Currently the handlers call BUS_SETUP_INTR with flags intr_type (defined in sys/bus.h): enum intr_type { INTR_TYPE_TTY = 1, INTR_TYPE_BIO = 2, INTR_TYPE_NET = 4, INTR_TYPE_CAM = 8, INTR_TYPE_MISC = 16, INTR_TYPE_FAST = 128 }; At least in nexus_setup_intr (sys/i386/i386/nexus.c), and possibly elsewhere as well, these flags get converted into an interrupt mask and flags for struct intrec, defined in sys/i386/isa/intr_machdep.h: #define INTR_FAST 0x00000001 /* fast interrupt handler */ #define INTR_EXCL 0x00010000 /* exclusive interrupt */ INTR_EXCL comes from the r_flags field in struct resource (sys/rman.h), where it is the complement of RF_SHAREABLE: #define RF_ALLOCATED 0x0001 /* resource has been reserved */ #define RF_ACTIVE 0x0002 /* resource allocation has been activated */ #define RF_SHAREABLE 0x0004 /* resource permits contemporaneous sharing */ #define RF_TIMESHARE 0x0008 /* resource permits time-division sharing */ #define RF_WANTED 0x0010 /* somebody is waiting for this resource */ #define RF_FIRSTSHARE 0x0020 /* first in sharing list */ These flag bits are all lightly used, and there's confusing duplication of functionality. For example, in nexus_setup_intr we have: case (INTR_TYPE_TTY | INTR_TYPE_FAST): mask = &tty_imask; icflags |= INTR_FAST; In sys/pci/pci_compat.c we have if (intflags & INTR_FAST) flags |= INTR_TYPE_FAST; It seems to me that we should define the flags to the *setup_intr functions to match those in struct intrec. Probably the RF_* flags are different enough in purpose that we shouldn't do the same thing there. In addition, these flags will have to change a lot in the change to threaded interrupts. First, I intend to add two additional flags to struct intrec: #define INTR_HEAVY 0x00000002 /* heavyweight interrupt process */ #define INTR_LIGHT 0x00000004 /* light weight interrupt thread */ #define INTR_THREADED (INTR_LIGHT | INTR_HEAVY) /* any kind of interrupt thread */ This will allow us to let traditional interrupts, lightweight threads and heavyweight processes to coexist; when the change is complete, we may be able to let them go away. The INTR_TYPE_TTY and friends are a different issue; at the moment I don't know if they're enough, but probably we should replace them with a process priority. I'd be grateful for any thoughts on these subjects. About fast interrupts, which currently seem to be used only by the sio driver: they perform their entire work before reenabling interrupts, and it's possible that we can keep them like that, though I haven't looked at the code yet. Does anybody know any reason why we should convert them to threaded interrupts? Greg -- Finger grog@lemis.com for PGP public key See complete headers for address and phone numbers To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Jul 17 8:55:45 2000 Delivered-To: freebsd-smp@freebsd.org Received: from Ilsa.StevesCafe.com (Ilsa.StevesCafe.com [206.168.13.65]) by hub.freebsd.org (Postfix) with ESMTP id E19BE37BC9A; Mon, 17 Jul 2000 08:54:00 -0700 (PDT) (envelope-from fbsd@Ilsa.StevesCafe.com) Received: from Ilsa.StevesCafe.com (localhost [127.0.0.1]) by Ilsa.StevesCafe.com (8.9.3/8.9.3) with ESMTP id JAA24715; Mon, 17 Jul 2000 09:53:50 -0600 (MDT) (envelope-from fbsd@Ilsa.StevesCafe.com) Message-Id: <200007171553.JAA24715@Ilsa.StevesCafe.com> X-Mailer: exmh version 2.0.2 2/24/98 From: Steve Passe To: Greg Lehey Cc: arch@FreeBSD.ORG, smp@FreeBSD.ORG Subject: Re: Tidying up the interrupt registration process In-reply-to: Your message of "Mon, 17 Jul 2000 16:30:38 +0930." <20000717163038.J26231@wantadilla.lemis.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Mon, 17 Jul 2000 09:53:49 -0600 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Hi, > I think I now understand how drivers register interrupts with newbus, > and it seems that some tidying up is in order. > ... > About fast interrupts, which currently seem to be used only by the sio > driver: they perform their entire work before reenabling interrupts, > and it's possible that we can keep them like that, though I haven't > looked at the code yet. Does anybody know any reason why we should > convert them to threaded interrupts? Its been quite awhile since I looked at this code, so forgive any inaccuracies on my part... I believe the sio FAST_INTR gets away with "they perform their entire work before reenabling interrupts" because it defers much of the work to the tty soft INTR level. In the ISR itself, it drains the hardware ASAP, then lets the soft INTR process the queues, etc. This method is used to prevent overflowing the sio fifos, which would otherwise occur if the INT were defered till spl() said it could proceed. If the new kernel model allows INTR threads to preempt, I would favor eliminating this sio 'special case', and use high priority for the sio ISR to get the same effect. I am a great fan of ISRs that look something like: sio_intr( struct threadStuff * s ) { ssignal( &(s->intrSem) ); } -- Steve Passe | powered by smp@csn.net | Symmetric MultiProcessor FreeBSD To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Jul 17 10: 5:19 2000 Delivered-To: freebsd-smp@freebsd.org Received: from mailhost.iprg.nokia.com (mailhost.iprg.nokia.com [205.226.5.12]) by hub.freebsd.org (Postfix) with ESMTP id A547837BBAD; Mon, 17 Jul 2000 10:05:05 -0700 (PDT) (envelope-from jre@iprg.nokia.com) Received: from darkstar.iprg.nokia.com (darkstar.iprg.nokia.com [205.226.5.69]) by mailhost.iprg.nokia.com (8.9.3/8.9.3-GLGS) with ESMTP id KAA21292; Mon, 17 Jul 2000 10:05:04 -0700 (PDT) Received: (from root@localhost) by darkstar.iprg.nokia.com (8.9.3/8.9.3-VIRSCAN) id KAA17809; Mon, 17 Jul 2000 10:05:03 -0700 X-Virus-Scanned: Mon, 17 Jul 2000 10:05:03 -0700 Nokia Silicon Valley Email Exploit Scanner Received: from radio.iprg.nokia.com (205.226.1.150, claiming to be "iprg.nokia.com") by darkstar.iprg.nokia.com(WTS.12.69) smtpdH6iIWK; Mon, 17 Jul 2000 10:04:58 PDT Message-ID: <39733CBB.7147BA74@iprg.nokia.com> Date: Mon, 17 Jul 2000 10:04:59 -0700 From: Joe Eykholt Organization: Nokia IPRG X-Mailer: Mozilla 4.7 [en] (X11; I; FreeBSD 2.2.6-RELEASE i386) X-Accept-Language: en MIME-Version: 1.0 To: Steve Passe Cc: Greg Lehey , arch@FreeBSD.ORG, smp@FreeBSD.ORG Subject: Re: Tidying up the interrupt registration process References: <200007171553.JAA24715@Ilsa.StevesCafe.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Steve's correct about the fast interrupt being needed to prevent overflow of the serial FIFOs. So, FAST interrupts haven't been blocked by spl (even splhigh) and the lower half of the driver has to disable interrupts in order to blcok the FAST interrupt. Just one more thing to consider: its pretty nice when using a serial console to be able to break into the debugger even when at splhigh(), and the FAST_INTR setup allows this. If serial interrupts were threads, then this probably wouldn't be possible. Solaris 2.5 (and probably the more recent versions also) uses the equivalent of fast interrupts for serial also. Joe Steve Passe wrote: > > Hi, > > > I think I now understand how drivers register interrupts with newbus, > > and it seems that some tidying up is in order. > > ... > > About fast interrupts, which currently seem to be used only by the sio > > driver: they perform their entire work before reenabling interrupts, > > and it's possible that we can keep them like that, though I haven't > > looked at the code yet. Does anybody know any reason why we should > > convert them to threaded interrupts? > > Its been quite awhile since I looked at this code, so forgive any inaccuracies > on my part... I believe the sio FAST_INTR gets away with "they perform their > entire work before reenabling interrupts" because it defers much of the work > to the tty soft INTR level. In the ISR itself, it drains the hardware ASAP, > then lets the soft INTR process the queues, etc. This method is used to > prevent overflowing the sio fifos, which would otherwise occur if the INT were > defered till spl() said it could proceed. If the new kernel model allows > INTR threads to preempt, I would favor eliminating this sio 'special case', > and use high priority for the sio ISR to get the same effect. I am a great > fan of ISRs that look something like: > > sio_intr( struct threadStuff * s ) > { > ssignal( &(s->intrSem) ); > } > > -- > Steve Passe | powered by > smp@csn.net | Symmetric MultiProcessor FreeBSD > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-smp" in the body of the message -- Joe Eykholt jre@iprg.nokia.com +1 650 625 2041 Nokia Internet Communications http://www.iprg.nokia.com 313 Fairchild Drive, Mountain View, CA 94043 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Jul 17 11: 8:27 2000 Delivered-To: freebsd-smp@freebsd.org Received: from mail.integratus.com (miami.integratus.com [63.209.2.83]) by hub.freebsd.org (Postfix) with SMTP id C83A537BBFB for ; Mon, 17 Jul 2000 11:08:21 -0700 (PDT) (envelope-from jar@integratus.com) Received: (qmail 25452 invoked from network); 17 Jul 2000 18:08:18 -0000 Received: from kungfu.integratus.com (HELO integratus.com) (172.20.5.168) by tortuga1.integratus.com with SMTP; 17 Jul 2000 18:08:18 -0000 Message-ID: <39734B92.E2B3B6CE@integratus.com> Date: Mon, 17 Jul 2000 11:08:18 -0700 From: Jack Rusher Organization: Integratus X-Mailer: Mozilla 4.73 [en] (X11; I; Linux 2.2.12 i386) X-Accept-Language: en MIME-Version: 1.0 To: Joe Eykholt Cc: Steve Passe , Greg Lehey , arch@FreeBSD.ORG, smp@FreeBSD.ORG Subject: Re: Tidying up the interrupt registration process References: <200007171553.JAA24715@Ilsa.StevesCafe.com> <39733CBB.7147BA74@iprg.nokia.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Joe Eykholt wrote: > > Just one more thing to consider: its pretty nice when using a serial > console to be able to break into the debugger even when at splhigh(), > and the FAST_INTR setup allows this. If serial interrupts were > threads, then this probably wouldn't be possible. > > Solaris 2.5 (and probably the more recent versions also) uses the > equivalent of fast interrupts for serial also. Solaris (up to 2.8) has fifteen interrupt levels, laid out like this: 1-9: "low" interrupts that are handled by kernel threads 10: "clock" interrupt, which (until 2.8) was used for the clock 11-15: "high" interrupts ...the low interrupts are general device interrupts (SCSI, frame buffer, etc), while the high set are for the serial devices. The low set are all handled by a 1/1 thread/interrupt scheme. Each CPU has a set of 9 interrupt threads waiting around to do the dirty work. This allows multiple CPUs to share the processing burden associated with device I/O. There is one system wide clock thread (until 2.8). The high set are handled in the context of the currently running thread, using the shortest code path they can manage. They did this because of the same overflow/console debugging issues that are currently being discussed here. -- Jack Rusher, Senior Engineer | mailto:jar@integratus.com Integratus, Inc. | http://www.integratus.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Jul 18 20:23:44 2000 Delivered-To: freebsd-smp@freebsd.org Received: from rover.village.org (rover.village.org [204.144.255.49]) by hub.freebsd.org (Postfix) with ESMTP id F121337BAB5; Tue, 18 Jul 2000 20:23:33 -0700 (PDT) (envelope-from imp@harmony.village.org) Received: from harmony.village.org (harmony.village.org [10.0.0.6]) by rover.village.org (8.9.3/8.9.3) with ESMTP id VAA89215; Tue, 18 Jul 2000 21:23:32 -0600 (MDT) (envelope-from imp@harmony.village.org) Received: from harmony.village.org (localhost.village.org [127.0.0.1]) by harmony.village.org (8.9.3/8.8.3) with ESMTP id VAA76910; Tue, 18 Jul 2000 21:23:23 -0600 (MDT) Message-Id: <200007190323.VAA76910@harmony.village.org> To: Greg Lehey Subject: Re: Tidying up the interrupt registration process Cc: arch@FreeBSD.ORG, smp@FreeBSD.ORG In-reply-to: Your message of "Mon, 17 Jul 2000 16:30:38 +0930." <20000717163038.J26231@wantadilla.lemis.com> References: <20000717163038.J26231@wantadilla.lemis.com> Date: Tue, 18 Jul 2000 21:23:23 -0600 From: Warner Losh Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org In message <20000717163038.J26231@wantadilla.lemis.com> Greg Lehey writes: : I think I now understand how drivers register interrupts with newbus, : and it seems that some tidying up is in order. Yes. : These flag bits are all lightly used, and there's confusing : duplication of functionality. For example, in nexus_setup_intr we : have: : : case (INTR_TYPE_TTY | INTR_TYPE_FAST): : mask = &tty_imask; : icflags |= INTR_FAST; This code is wrong. It should be moved out of the case statement. BDE posted a fix that I thought I'd committed, but it appears that I haven't done so yet. : It seems to me that we should define the flags to the *setup_intr : functions to match those in struct intrec. Probably the RF_* flags : are different enough in purpose that we shouldn't do the same thing : there. I think that INTR_FAST is special enough to be the only one that needs help. : In addition, these flags will have to change a lot in the change to : threaded interrupts. First, I intend to add two additional flags to : struct intrec: : : #define INTR_HEAVY 0x00000002 /* heavyweight interrupt process */ : #define INTR_LIGHT 0x00000004 /* light weight interrupt thread */ : #define INTR_THREADED (INTR_LIGHT | INTR_HEAVY) /* any kind of interrupt thread */ : : This will allow us to let traditional interrupts, lightweight threads : and heavyweight processes to coexist; when the change is complete, we : may be able to let them go away. The INTR_TYPE_TTY and friends are a : different issue; at the moment I don't know if they're enough, but : probably we should replace them with a process priority. I'd be : grateful for any thoughts on these subjects. The INTR_TYPE_TTY interrupts, except fast ones, are masked as a unit. Likewise with the INTR_TYPE_NET, etc. The intent here is to provide assistance in serialization of access to crititical structures. When you do a splnet, all the net devices who have an interrupt have their interrupt blocked. : About fast interrupts, which currently seem to be used only by the sio : driver: they perform their entire work before reenabling interrupts, : and it's possible that we can keep them like that, though I haven't : looked at the code yet. Does anybody know any reason why we should : convert them to threaded interrupts? Yes. They are used by more than the sio driver. I use them in at least two pci drivers that I have written for hardware that has extreme latency requirements that normal interrupts cannot accomplish. This is the same sort of thing as the serial fifos. These have extremely hard requirements to be serviced in (on the order of micro seconds). The "upper half" of the serial driver harvests the characters from the serial line when it gets interrupts and uses splsofttty() to activate the lower half of the driver. These devices simply cannot tolerate scheduling delays at all, even very fast ones. pci modems are an interesting problem. You want to share an interrupt, but fast interrupts can't be shared. On sufficiently fast machines, it isn't an issue, but on slow machines you can easily lose characters because interrupts are masked for too long. Heavy VM load (eg swapping) seems to make this problem happen more often than even just simple heavy disk I/O, but I've not done extensive tests to draw boundaries around this problem. Warner To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Jul 18 20:39:34 2000 Delivered-To: freebsd-smp@freebsd.org Received: from wantadilla.lemis.com (wantadilla.lemis.com [192.109.197.80]) by hub.freebsd.org (Postfix) with ESMTP id 5401437B5EA; Tue, 18 Jul 2000 20:39:18 -0700 (PDT) (envelope-from grog@wantadilla.lemis.com) Received: (from grog@localhost) by wantadilla.lemis.com (8.9.3/8.9.3) id NAA09314; Wed, 19 Jul 2000 13:09:07 +0930 (CST) (envelope-from grog) Date: Wed, 19 Jul 2000 13:09:07 +0930 From: Greg Lehey To: Warner Losh Cc: arch@FreeBSD.ORG, smp@FreeBSD.ORG Subject: Re: Tidying up the interrupt registration process Message-ID: <20000719130907.H12072@wantadilla.lemis.com> References: <20000717163038.J26231@wantadilla.lemis.com> <200007190323.VAA76910@harmony.village.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0pre2i In-Reply-To: <200007190323.VAA76910@harmony.village.org> Organization: LEMIS, PO Box 460, Echunga SA 5153, Australia Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 WWW-Home-Page: http://www.lemis.com/~grog X-PGP-Fingerprint: 6B 7B C3 8C 61 CD 54 AF 13 24 52 F8 6D A4 95 EF Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Tuesday, 18 July 2000 at 21:23:23 -0600, Warner Losh wrote: > In message <20000717163038.J26231@wantadilla.lemis.com> Greg Lehey writes: >> These flag bits are all lightly used, and there's confusing >> duplication of functionality. For example, in nexus_setup_intr we >> have: >> >> case (INTR_TYPE_TTY | INTR_TYPE_FAST): >> mask = &tty_imask; >> icflags |= INTR_FAST; > > This code is wrong. It should be moved out of the case statement. > BDE posted a fix that I thought I'd committed, but it appears that I > haven't done so yet. I think this function needs to be completely rewritten in the course of the implementation of interrupt threads. It sort of works now, so I don't think we should touch it further until the threads are up and hobbling. >> In addition, these flags will have to change a lot in the change to >> threaded interrupts. First, I intend to add two additional flags to >> struct intrec: >> >> #define INTR_HEAVY 0x00000002 /* heavyweight interrupt process */ >> #define INTR_LIGHT 0x00000004 /* light weight interrupt thread */ >> #define INTR_THREADED (INTR_LIGHT | INTR_HEAVY) /* any kind of interrupt thread */ >> >> This will allow us to let traditional interrupts, lightweight threads >> and heavyweight processes to coexist; when the change is complete, we >> may be able to let them go away. The INTR_TYPE_TTY and friends are a >> different issue; at the moment I don't know if they're enough, but >> probably we should replace them with a process priority. I'd be >> grateful for any thoughts on these subjects. > > The INTR_TYPE_TTY interrupts, except fast ones, are masked as a unit. > Likewise with the INTR_TYPE_NET, etc. The intent here is to provide > assistance in serialization of access to crititical structures. When > you do a splnet, all the net devices who have an interrupt have their > interrupt blocked. Right, that's the current way things are done. This will change too, to be replaced by relative scheduling priorities. That's the more interesting question: how should they relate to each other? >> About fast interrupts, which currently seem to be used only by the sio >> driver: they perform their entire work before reenabling interrupts, >> and it's possible that we can keep them like that, though I haven't >> looked at the code yet. Does anybody know any reason why we should >> convert them to threaded interrupts? > > Yes. They are used by more than the sio driver. I use them in at > least two pci drivers that I have written for hardware that has > extreme latency requirements that normal interrupts cannot > accomplish. This is the same sort of thing as the serial fifos. > These have extremely hard requirements to be serviced in (on the order > of micro seconds). The "upper half" of the serial driver harvests the > characters from the serial line when it gets interrupts and uses > splsofttty() to activate the lower half of the driver. These devices > simply cannot tolerate scheduling delays at all, even very fast ones. That's what I thought. Does anybody else see a reason to convert fast interrupts into threads? > pci modems are an interesting problem. You want to share an > interrupt, but fast interrupts can't be shared. On sufficiently > fast machines, it isn't an issue, but on slow machines you can > easily lose characters because interrupts are masked for too long. > Heavy VM load (eg swapping) seems to make this problem happen more > often than even just simple heavy disk I/O, but I've not done > extensive tests to draw boundaries around this problem. I can't see anything inherent in the treatment of fast interrupts which says they can't be shared. I'll take another look at the code; maybe we can get rid of this restriction. Greg -- Finger grog@lemis.com for PGP public key See complete headers for address and phone numbers To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Jul 18 20:55:28 2000 Delivered-To: freebsd-smp@freebsd.org Received: from mass.osd.bsdi.com (adsl-63-202-177-51.dsl.snfc21.pacbell.net [63.202.177.51]) by hub.freebsd.org (Postfix) with ESMTP id DC2BD37B7FE; Tue, 18 Jul 2000 20:55:22 -0700 (PDT) (envelope-from msmith@mass.osd.bsdi.com) Received: from mass.osd.bsdi.com (localhost [127.0.0.1]) by mass.osd.bsdi.com (8.9.3/8.9.3) with ESMTP id VAA21389; Tue, 18 Jul 2000 21:03:59 -0700 (PDT) (envelope-from msmith@mass.osd.bsdi.com) Message-Id: <200007190403.VAA21389@mass.osd.bsdi.com> X-Mailer: exmh version 2.1.1 10/15/1999 To: Greg Lehey Cc: arch@FreeBSD.ORG, smp@FreeBSD.ORG Subject: Re: Tidying up the interrupt registration process In-reply-to: Your message of "Wed, 19 Jul 2000 13:09:07 +0930." <20000719130907.H12072@wantadilla.lemis.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Tue, 18 Jul 2000 21:03:59 -0700 From: Mike Smith Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > The INTR_TYPE_TTY interrupts, except fast ones, are masked as a unit. > > Likewise with the INTR_TYPE_NET, etc. The intent here is to provide > > assistance in serialization of access to crititical structures. When > > you do a splnet, all the net devices who have an interrupt have their > > interrupt blocked. > > Right, that's the current way things are done. This will change too, > to be replaced by relative scheduling priorities. That's the more > interesting question: how should they relate to each other? As a general rule, they shouldn't. Don't get carried away with something relatively trivial like which interrupts should have "more" scheduling priority - just get them being scheduled at all for now. 8) > That's what I thought. Does anybody else see a reason to convert fast > interrupts into threads? No. Or more specifically, we need a mechanism for "classic-style" low-latency interrupt handlers. These will typically be coupled with a software-interrupt thread, more or less as they are now. > I can't see anything inherent in the treatment of fast interrupts > which says they can't be shared. I'll take another look at the code; > maybe we can get rid of this restriction. Sharing a 'fast' interrupt completely defeats the point of making it 'fast'. You should not be able to register a 'fast' handler on any source with anything else attached, nor anything else on a source that has a 'fast' handler already registered. Yes, this does impose some configuration constraints on the system, but there are few viable alternatives. -- ... every activity meets with opposition, everyone who acts has his rivals and unfortunately opponents also. But not because people want to be opponents, rather because the tasks and relationships force people to take different points of view. [Dr. Fritz Todt] To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Jul 18 20:58: 3 2000 Delivered-To: freebsd-smp@freebsd.org Received: from fw.wintelcom.net (ns1.wintelcom.net [209.1.153.20]) by hub.freebsd.org (Postfix) with ESMTP id 9FDB937B7FE; Tue, 18 Jul 2000 20:57:58 -0700 (PDT) (envelope-from bright@fw.wintelcom.net) Received: (from bright@localhost) by fw.wintelcom.net (8.10.0/8.10.0) id e6J3vuP14070; Tue, 18 Jul 2000 20:57:56 -0700 (PDT) Date: Tue, 18 Jul 2000 20:57:56 -0700 From: Alfred Perlstein To: Mike Smith Cc: Greg Lehey , arch@FreeBSD.ORG, smp@FreeBSD.ORG Subject: Re: Tidying up the interrupt registration process Message-ID: <20000718205756.Q13979@fw.wintelcom.net> References: <20000719130907.H12072@wantadilla.lemis.com> <200007190403.VAA21389@mass.osd.bsdi.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.4i In-Reply-To: <200007190403.VAA21389@mass.osd.bsdi.com>; from msmith@FreeBSD.ORG on Tue, Jul 18, 2000 at 09:03:59PM -0700 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org * Mike Smith [000718 20:55] wrote: > > Sharing a 'fast' interrupt completely defeats the point of making it > 'fast'. You should not be able to register a 'fast' handler on any > source with anything else attached, nor anything else on a source that > has a 'fast' handler already registered. Yes, this does impose some > configuration constraints on the system, but there are few viable > alternatives. Just wondering, could a device fall back to non-fast mode if the hardware forced this sort of situation but still complain about it? -- -Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org] "I have the heart of a child; I keep it in a jar on my desk." To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Jul 18 20:59:34 2000 Delivered-To: freebsd-smp@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id E39E337B7FE; Tue, 18 Jul 2000 20:59:28 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@LOCALHOST [127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id VAA09445; Tue, 18 Jul 2000 21:58:57 -0600 (MDT) Message-Id: <200007190358.VAA09445@berserker.bsdi.com> To: Greg Lehey Cc: Warner Losh , arch@freebsd.org, smp@freebsd.org Subject: Re: Tidying up the interrupt registration process From: Chuck Paterson Date: Tue, 18 Jul 2000 21:58:57 -0600 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org }That's what I thought. Does anybody else see a reason to convert fast }interrupts into threads? The short answer is no, you absolutely don't want to convert them to fully instantiated threads, especially when you only have a heavy wait solution. There is another middle ground that is less clear, and it depends partially on what you deem a thread. If you just switch the stack pointer and curproc, but use spin locks and don't allow for a context switch are you converting it to a thread. At this point the statistical stuff will charge time properly to interrupts rather than user processes, or other kernel processes, you also don't have to worry about pathological cases blowing out the stack. Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Jul 18 21: 7: 7 2000 Delivered-To: freebsd-smp@freebsd.org Received: from mass.osd.bsdi.com (adsl-63-202-177-51.dsl.snfc21.pacbell.net [63.202.177.51]) by hub.freebsd.org (Postfix) with ESMTP id D2BAA37BC7B; Tue, 18 Jul 2000 21:07:02 -0700 (PDT) (envelope-from msmith@mass.osd.bsdi.com) Received: from mass.osd.bsdi.com (localhost [127.0.0.1]) by mass.osd.bsdi.com (8.9.3/8.9.3) with ESMTP id VAA21470; Tue, 18 Jul 2000 21:15:53 -0700 (PDT) (envelope-from msmith@mass.osd.bsdi.com) Message-Id: <200007190415.VAA21470@mass.osd.bsdi.com> X-Mailer: exmh version 2.1.1 10/15/1999 To: Alfred Perlstein Cc: arch@FreeBSD.ORG, smp@FreeBSD.ORG Subject: Re: Tidying up the interrupt registration process In-reply-to: Your message of "Tue, 18 Jul 2000 20:57:56 PDT." <20000718205756.Q13979@fw.wintelcom.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Tue, 18 Jul 2000 21:15:53 -0700 From: Mike Smith Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > * Mike Smith [000718 20:55] wrote: > > > > Sharing a 'fast' interrupt completely defeats the point of making it > > 'fast'. You should not be able to register a 'fast' handler on any > > source with anything else attached, nor anything else on a source that > > has a 'fast' handler already registered. Yes, this does impose some > > configuration constraints on the system, but there are few viable > > alternatives. > > Just wondering, could a device fall back to non-fast mode if the > hardware forced this sort of situation but still complain about it? You don't typically bother requesting a 'fast' interrupt unless you really need it. This decision would have to be left up to the device driver - some might be OK accepting the tradeoff (eg. sio), wheras for others this might constitute a fatal error. -- ... every activity meets with opposition, everyone who acts has his rivals and unfortunately opponents also. But not because people want to be opponents, rather because the tasks and relationships force people to take different points of view. [Dr. Fritz Todt] To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Jul 18 21:22:12 2000 Delivered-To: freebsd-smp@freebsd.org Received: from wantadilla.lemis.com (wantadilla.lemis.com [192.109.197.80]) by hub.freebsd.org (Postfix) with ESMTP id 8194337B99B; Tue, 18 Jul 2000 21:22:06 -0700 (PDT) (envelope-from grog@wantadilla.lemis.com) Received: (from grog@localhost) by wantadilla.lemis.com (8.9.3/8.9.3) id NAA27189; Wed, 19 Jul 2000 13:51:50 +0930 (CST) (envelope-from grog) Date: Wed, 19 Jul 2000 13:51:49 +0930 From: Greg Lehey To: Chuck Paterson Cc: Warner Losh , arch@freebsd.org, smp@freebsd.org Subject: Re: Tidying up the interrupt registration process Message-ID: <20000719135149.I12072@wantadilla.lemis.com> References: <200007190358.VAA09445@berserker.bsdi.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0pre2i In-Reply-To: <200007190358.VAA09445@berserker.bsdi.com> Organization: LEMIS, PO Box 460, Echunga SA 5153, Australia Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 WWW-Home-Page: http://www.lemis.com/~grog X-PGP-Fingerprint: 6B 7B C3 8C 61 CD 54 AF 13 24 52 F8 6D A4 95 EF Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Tuesday, 18 July 2000 at 21:58:57 -0600, Chuck Paterson wrote: > > }That's what I thought. Does anybody else see a reason to convert fast > }interrupts into threads? > > The short answer is no, you absolutely don't want to > convert them to fully instantiated threads, especially when you > only have a heavy wait solution. There is another middle ground > that is less clear, and it depends partially on what you deem a > thread. If you just switch the stack pointer and curproc, but use > spin locks and don't allow for a context switch are you converting > it to a thread. At this point the statistical stuff will charge > time properly to interrupts rather than user processes, or other > kernel processes, you also don't have to worry about pathological > cases blowing out the stack. Does BSD/OS have fast interrupts? I haven't seen any evidence. In FreeBSD, a fast interrupt runs before EOI, so we can't convert it to a thread. Greg -- Finger grog@lemis.com for PGP public key See complete headers for address and phone numbers To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Jul 18 21:27:39 2000 Delivered-To: freebsd-smp@freebsd.org Received: from wantadilla.lemis.com (wantadilla.lemis.com [192.109.197.80]) by hub.freebsd.org (Postfix) with ESMTP id B2B6437BB4C; Tue, 18 Jul 2000 21:27:31 -0700 (PDT) (envelope-from grog@wantadilla.lemis.com) Received: (from grog@localhost) by wantadilla.lemis.com (8.9.3/8.9.3) id NAA29575; Wed, 19 Jul 2000 13:57:30 +0930 (CST) (envelope-from grog) Date: Wed, 19 Jul 2000 13:57:30 +0930 From: Greg Lehey To: Mike Smith Cc: arch@FreeBSD.ORG, smp@FreeBSD.ORG Subject: Re: Tidying up the interrupt registration process Message-ID: <20000719135730.J12072@wantadilla.lemis.com> References: <20000719130907.H12072@wantadilla.lemis.com> <200007190403.VAA21389@mass.osd.bsdi.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0pre2i In-Reply-To: <200007190403.VAA21389@mass.osd.bsdi.com> Organization: LEMIS, PO Box 460, Echunga SA 5153, Australia Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 WWW-Home-Page: http://www.lemis.com/~grog X-PGP-Fingerprint: 6B 7B C3 8C 61 CD 54 AF 13 24 52 F8 6D A4 95 EF Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Tuesday, 18 July 2000 at 21:03:59 -0700, Mike Smith wrote: >>> The INTR_TYPE_TTY interrupts, except fast ones, are masked as a unit. >>> Likewise with the INTR_TYPE_NET, etc. The intent here is to provide >>> assistance in serialization of access to crititical structures. When >>> you do a splnet, all the net devices who have an interrupt have their >>> interrupt blocked. >> >> Right, that's the current way things are done. This will change too, >> to be replaced by relative scheduling priorities. That's the more >> interesting question: how should they relate to each other? > > As a general rule, they shouldn't. Don't get carried away with something > relatively trivial like which interrupts should have "more" scheduling > priority - just get them being scheduled at all for now. 8) That doesn't mean we shouldn't be thinking about the next step. >> I can't see anything inherent in the treatment of fast interrupts >> which says they can't be shared. I'll take another look at the code; >> maybe we can get rid of this restriction. > > Sharing a 'fast' interrupt completely defeats the point of making it > 'fast'. No, it partially defeats it. It's a tradeoff. > You should not be able to register a 'fast' handler on any source > with anything else attached, nor anything else on a source that has > a 'fast' handler already registered. Yes, this does impose some > configuration constraints on the system, but there are few viable > alternatives. You haven't really explained that. Sure, if you have more than one interrupt on an IRQ, it will take longer, but it will still ensure that nothing else interrupts in the meantime. You can't compare this to the current fast interrupt scheme which limits you to one interrupt handler, because the hardware reality isn't like that. You need to compare it to the alternative of a shared slow interrupt handler: clearly a fast interrupt handler will still be faster. And Warner produced a valid example of where it would make a difference. As regards sharing fast and slow interrupts on the same IRQ: I can see reasons for wanting to do this as well. I don't want to have to write the code, though :-) Greg -- Finger grog@lemis.com for PGP public key See complete headers for address and phone numbers To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Jul 18 21:37:16 2000 Delivered-To: freebsd-smp@freebsd.org Received: from wantadilla.lemis.com (wantadilla.lemis.com [192.109.197.80]) by hub.freebsd.org (Postfix) with ESMTP id A437F37B99B; Tue, 18 Jul 2000 21:37:09 -0700 (PDT) (envelope-from grog@wantadilla.lemis.com) Received: (from grog@localhost) by wantadilla.lemis.com (8.9.3/8.9.3) id OAA33609; Wed, 19 Jul 2000 14:07:07 +0930 (CST) (envelope-from grog) Date: Wed, 19 Jul 2000 14:07:07 +0930 From: Greg Lehey To: Mike Smith Cc: Alfred Perlstein , arch@FreeBSD.ORG, smp@FreeBSD.ORG Subject: Re: Tidying up the interrupt registration process Message-ID: <20000719140707.K12072@wantadilla.lemis.com> References: <20000718205756.Q13979@fw.wintelcom.net> <200007190415.VAA21470@mass.osd.bsdi.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0pre2i In-Reply-To: <200007190415.VAA21470@mass.osd.bsdi.com> Organization: LEMIS, PO Box 460, Echunga SA 5153, Australia Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 WWW-Home-Page: http://www.lemis.com/~grog X-PGP-Fingerprint: 6B 7B C3 8C 61 CD 54 AF 13 24 52 F8 6D A4 95 EF Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Tuesday, 18 July 2000 at 21:15:53 -0700, Mike Smith wrote: >> * Mike Smith [000718 20:55] wrote: >>> >>> Sharing a 'fast' interrupt completely defeats the point of making it >>> 'fast'. You should not be able to register a 'fast' handler on any >>> source with anything else attached, nor anything else on a source that >>> has a 'fast' handler already registered. Yes, this does impose some >>> configuration constraints on the system, but there are few viable >>> alternatives. >> >> Just wondering, could a device fall back to non-fast mode if the >> hardware forced this sort of situation but still complain about it? > > You don't typically bother requesting a 'fast' interrupt unless you > really need it. This decision would have to be left up to the > device driver - some might be OK accepting the tradeoff (eg. sio), > wheras for others this might constitute a fatal error. In fact, unless I'm missing something, it looks as if there's no code there to stop you from sharing fast interrupts. I could have sworn there was, but the code in intr_machdep.c doesn't check. The cy driver also sets INTR_EXCL, which is checked for, but nexus_setup_intr doesn't. If you share a fast interrupt and a slow interrupt, the first-level handler becomes the mux, and so it appears that the "fast" attribute would just go away, *without* any warning. I don't know if there's something else that would stop it from working at all. Greg -- Finger grog@lemis.com for PGP public key See complete headers for address and phone numbers To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Jul 18 21:41:42 2000 Delivered-To: freebsd-smp@freebsd.org Received: from mass.osd.bsdi.com (adsl-63-202-177-51.dsl.snfc21.pacbell.net [63.202.177.51]) by hub.freebsd.org (Postfix) with ESMTP id 531AF37BC88; Tue, 18 Jul 2000 21:41:38 -0700 (PDT) (envelope-from msmith@mass.osd.bsdi.com) Received: from mass.osd.bsdi.com (localhost [127.0.0.1]) by mass.osd.bsdi.com (8.9.3/8.9.3) with ESMTP id VAA21651; Tue, 18 Jul 2000 21:49:32 -0700 (PDT) (envelope-from msmith@mass.osd.bsdi.com) Message-Id: <200007190449.VAA21651@mass.osd.bsdi.com> X-Mailer: exmh version 2.1.1 10/15/1999 To: Greg Lehey Cc: arch@FreeBSD.ORG, smp@FreeBSD.ORG Subject: Re: Tidying up the interrupt registration process In-reply-to: Your message of "Wed, 19 Jul 2000 13:57:30 +0930." <20000719135730.J12072@wantadilla.lemis.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Tue, 18 Jul 2000 21:49:32 -0700 From: Mike Smith Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > You should not be able to register a 'fast' handler on any source > > with anything else attached, nor anything else on a source that has > > a 'fast' handler already registered. Yes, this does impose some > > configuration constraints on the system, but there are few viable > > alternatives. > > You haven't really explained that. Sure, if you have more than one > interrupt on an IRQ, it will take longer, but it will still ensure > that nothing else interrupts in the meantime. You can't compare this > to the current fast interrupt scheme which limits you to one interrupt > handler, because the hardware reality isn't like that. You need to > compare it to the alternative of a shared slow interrupt handler: > clearly a fast interrupt handler will still be faster. And Warner > produced a valid example of where it would make a difference. This is not correct. There are two separate things that a 'fast' interrupt handler seeks to achieve: reduced and constrained interrupt latency. Allowing 'fast' interrupts to be shared removes any hope of constraining latency, as well as increasing it. Note also that a 'fast' handler may be computationally expensive - the 'fast' criterion has to do with how it is invoked, not how it runs. > As regards sharing fast and slow interrupts on the same IRQ: I can see > reasons for wanting to do this as well. I don't want to have to write > the code, though :-) It would be relatively trivial, actually, since the dispatcher that invokes the actual interrupt thread in the 'not-fast' case is really a 'fast' interrupt handler. -- ... every activity meets with opposition, everyone who acts has his rivals and unfortunately opponents also. But not because people want to be opponents, rather because the tasks and relationships force people to take different points of view. [Dr. Fritz Todt] To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Jul 18 21:53:51 2000 Delivered-To: freebsd-smp@freebsd.org Received: from wantadilla.lemis.com (wantadilla.lemis.com [192.109.197.80]) by hub.freebsd.org (Postfix) with ESMTP id 682A937B99B; Tue, 18 Jul 2000 21:53:40 -0700 (PDT) (envelope-from grog@wantadilla.lemis.com) Received: (from grog@localhost) by wantadilla.lemis.com (8.9.3/8.9.3) id OAA40543; Wed, 19 Jul 2000 14:23:37 +0930 (CST) (envelope-from grog) Date: Wed, 19 Jul 2000 14:23:37 +0930 From: Greg Lehey To: Mike Smith Cc: arch@FreeBSD.ORG, smp@FreeBSD.ORG Subject: Re: Tidying up the interrupt registration process Message-ID: <20000719142337.L12072@wantadilla.lemis.com> References: <20000719135730.J12072@wantadilla.lemis.com> <200007190449.VAA21651@mass.osd.bsdi.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0pre2i In-Reply-To: <200007190449.VAA21651@mass.osd.bsdi.com> Organization: LEMIS, PO Box 460, Echunga SA 5153, Australia Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 WWW-Home-Page: http://www.lemis.com/~grog X-PGP-Fingerprint: 6B 7B C3 8C 61 CD 54 AF 13 24 52 F8 6D A4 95 EF Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Tuesday, 18 July 2000 at 21:49:32 -0700, Mike Smith wrote: >>> You should not be able to register a 'fast' handler on any source >>> with anything else attached, nor anything else on a source that has >>> a 'fast' handler already registered. Yes, this does impose some >>> configuration constraints on the system, but there are few viable >>> alternatives. >> >> You haven't really explained that. Sure, if you have more than one >> interrupt on an IRQ, it will take longer, but it will still ensure >> that nothing else interrupts in the meantime. You can't compare this >> to the current fast interrupt scheme which limits you to one interrupt >> handler, because the hardware reality isn't like that. You need to >> compare it to the alternative of a shared slow interrupt handler: >> clearly a fast interrupt handler will still be faster. And Warner >> produced a valid example of where it would make a difference. > > This is not correct. You need to be more specific in "this". > There are two separate things that a 'fast' interrupt handler seeks > to achieve: reduced and constrained interrupt latency. Right. > Allowing 'fast' interrupts to be shared removes any hope of > constraining latency, as well as increasing it. You're being rather categorical there. As I said, it's a tradeoff. > Note also that a 'fast' handler may be computationally expensive - > the 'fast' criterion has to do with how it is invoked, not how it > runs. Indeed. Much of your hope lies in the wise use of the resource. In fact, I'm astounded how much code there is in siointerrupt. >> As regards sharing fast and slow interrupts on the same IRQ: I can see >> reasons for wanting to do this as well. I don't want to have to write >> the code, though :-) > > It would be relatively trivial, actually, since the dispatcher that > invokes the actual interrupt thread in the 'not-fast' case is really a > 'fast' interrupt handler. No, that's not correct. You set a fast interrupt in icu_setup() by passing the INTR_FAST flag. Slightly simplified: int icu_setup(int intr, inthand2_t *handler, void *arg, u_int *maskptr, int flags) { ... if (flags & INTR_FAST) { vector = TPR_FAST_INTS + intr; setidt(vector, fastintr[intr], SDT_SYS386IGT, SEL_KPL, GSEL(GCODE_SEL, SEL_KPL)); } else { vector = TPR_SLOW_INTS + intr; setidt(vector, slowintr[intr], SDT_SYS386IGT, SEL_KPL, GSEL(GCODE_SEL, SEL_KPL)); The call to set the mux (in add_intrdesc) is: if (icu_setup(irq, intr_mux, head, 0, 0) != 0) return (-1); So it's a slow interrupt. But you might be right that it would be easier to do than I thought. Greg -- Finger grog@lemis.com for PGP public key See complete headers for address and phone numbers To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Jul 18 22:59: 6 2000 Delivered-To: freebsd-smp@freebsd.org Received: from rover.village.org (rover.village.org [204.144.255.49]) by hub.freebsd.org (Postfix) with ESMTP id CA68A37BCE2; Tue, 18 Jul 2000 22:58:55 -0700 (PDT) (envelope-from imp@harmony.village.org) Received: from harmony.village.org (harmony.village.org [10.0.0.6]) by rover.village.org (8.9.3/8.9.3) with ESMTP id XAA89830; Tue, 18 Jul 2000 23:58:53 -0600 (MDT) (envelope-from imp@harmony.village.org) Received: from harmony.village.org (localhost.village.org [127.0.0.1]) by harmony.village.org (8.9.3/8.8.3) with ESMTP id XAA77880; Tue, 18 Jul 2000 23:58:43 -0600 (MDT) Message-Id: <200007190558.XAA77880@harmony.village.org> To: Greg Lehey Subject: Re: Tidying up the interrupt registration process Cc: arch@FreeBSD.ORG, smp@FreeBSD.ORG In-reply-to: Your message of "Wed, 19 Jul 2000 13:09:07 +0930." <20000719130907.H12072@wantadilla.lemis.com> References: <20000719130907.H12072@wantadilla.lemis.com> <20000717163038.J26231@wantadilla.lemis.com> <200007190323.VAA76910@harmony.village.org> Date: Tue, 18 Jul 2000 23:58:43 -0600 From: Warner Losh Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org In message <20000719130907.H12072@wantadilla.lemis.com> Greg Lehey writes: : I think this function needs to be completely rewritten in the course : of the implementation of interrupt threads. It sort of works now, so : I don't think we should touch it further until the threads are up and : hobbling. I want to fix the fast interrupt part of it. : That's what I thought. Does anybody else see a reason to convert fast : interrupts into threads? So long as they run with the lowest possible latency, that would be OK. : > pci modems are an interesting problem. You want to share an : > interrupt, but fast interrupts can't be shared. On sufficiently : > fast machines, it isn't an issue, but on slow machines you can : > easily lose characters because interrupts are masked for too long. : > Heavy VM load (eg swapping) seems to make this problem happen more : > often than even just simple heavy disk I/O, but I've not done : > extensive tests to draw boundaries around this problem. : : I can't see anything inherent in the treatment of fast interrupts : which says they can't be shared. I'll take another look at the code; : maybe we can get rid of this restriction. I think it is an implementation thing, but bde will know better. Warner To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Jul 18 23:13:31 2000 Delivered-To: freebsd-smp@freebsd.org Received: from wantadilla.lemis.com (wantadilla.lemis.com [192.109.197.80]) by hub.freebsd.org (Postfix) with ESMTP id 8624B37BCD6; Tue, 18 Jul 2000 23:13:22 -0700 (PDT) (envelope-from grog@wantadilla.lemis.com) Received: (from grog@localhost) by wantadilla.lemis.com (8.9.3/8.9.3) id PAA73758; Wed, 19 Jul 2000 15:43:13 +0930 (CST) (envelope-from grog) Date: Wed, 19 Jul 2000 15:43:13 +0930 From: Greg Lehey To: Warner Losh Cc: arch@FreeBSD.ORG, smp@FreeBSD.ORG Subject: Re: Tidying up the interrupt registration process Message-ID: <20000719154313.M12072@wantadilla.lemis.com> References: <20000719130907.H12072@wantadilla.lemis.com> <20000717163038.J26231@wantadilla.lemis.com> <200007190323.VAA76910@harmony.village.org> <20000719130907.H12072@wantadilla.lemis.com> <200007190558.XAA77880@harmony.village.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0pre2i In-Reply-To: <200007190558.XAA77880@harmony.village.org> Organization: LEMIS, PO Box 460, Echunga SA 5153, Australia Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 WWW-Home-Page: http://www.lemis.com/~grog X-PGP-Fingerprint: 6B 7B C3 8C 61 CD 54 AF 13 24 52 F8 6D A4 95 EF Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Tuesday, 18 July 2000 at 23:58:43 -0600, Warner Losh wrote: > In message <20000719130907.H12072@wantadilla.lemis.com> Greg Lehey writes: >> I think this function needs to be completely rewritten in the course >> of the implementation of interrupt threads. It sort of works now, so >> I don't think we should touch it further until the threads are up and >> hobbling. > > I want to fix the fast interrupt part of it. Well, I was lying when I said we wouldn't touch it. In the meantime, I've seen that the BSDi implementation has already addressed the question of priorities, and now we have no more interrupt masks, the mask parameter to inthand_add changes into a pri parameter. So the code you're talking about becomes: switch (flags) { case INTR_TYPE_TTY: pri = PI_TTYLOW; break; case (INTR_TYPE_TTY | INTR_FAST): pri = PI_TTYHIGH; break; The flags parameter also gets passed directly to inthand_add. >> That's what I thought. Does anybody else see a reason to convert fast >> interrupts into threads? > > So long as they run with the lowest possible latency, that would be > OK. See the discussion. It won't work like that. >>> pci modems are an interesting problem. You want to share an >>> interrupt, but fast interrupts can't be shared. On sufficiently >>> fast machines, it isn't an issue, but on slow machines you can >>> easily lose characters because interrupts are masked for too long. >>> Heavy VM load (eg swapping) seems to make this problem happen more >>> often than even just simple heavy disk I/O, but I've not done >>> extensive tests to draw boundaries around this problem. >> >> I can't see anything inherent in the treatment of fast interrupts >> which says they can't be shared. I'll take another look at the code; >> maybe we can get rid of this restriction. > > I think it is an implementation thing, but bde will know better. He's been very quiet lately. Greg -- Finger grog@lemis.com for PGP public key See complete headers for address and phone numbers To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Jul 19 8:15:48 2000 Delivered-To: freebsd-smp@freebsd.org Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by hub.freebsd.org (Postfix) with ESMTP id D494737BDDC; Wed, 19 Jul 2000 08:15:36 -0700 (PDT) (envelope-from bde@zeta.org.au) Received: from bde.zeta.org.au (bde.zeta.org.au [203.2.228.102]) by mailman.zeta.org.au (8.8.7/8.8.7) with ESMTP id BAA28186; Thu, 20 Jul 2000 01:15:15 +1000 Date: Thu, 20 Jul 2000 01:15:08 +1000 (EST) From: Bruce Evans X-Sender: bde@besplex.bde.org To: Greg Lehey Cc: Mike Smith , Alfred Perlstein , arch@FreeBSD.ORG, smp@FreeBSD.ORG Subject: Re: Tidying up the interrupt registration process In-Reply-To: <20000719140707.K12072@wantadilla.lemis.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Wed, 19 Jul 2000, Greg Lehey wrote: > On Tuesday, 18 July 2000 at 21:15:53 -0700, Mike Smith wrote: > >> * Mike Smith [000718 20:55] wrote: > >>> > >>> Sharing a 'fast' interrupt completely defeats the point of making it > >>> 'fast'. You should not be able to register a 'fast' handler on any > >>> source with anything else attached, nor anything else on a source that > >>> has a 'fast' handler already registered. Yes, this does impose some > >>> configuration constraints on the system, but there are few viable > >>> alternatives. Not quite. Attaching multiple fast interrupt handlers to the same irq is not much different from attaching them to different irqs. The handlers should be executed serially in the !SMP case and on different CPUs (serially per CPU) in the SMP case. This is currently not implemented. Sharing the irq is only undesirable (in the all-fast case) because it adds overheads for polling all the devices. Attaching a mixture of fast and normal handlers to the same irq seems to be impossible to implement. If interrupts are edge triggered, then we don't need to touch the PIC masks currently do in the i386 !SMP case (I'm not sure about other cases), but we won't see new interrupts for fast devices on the same irq while we are handling normal interrupts. This defeats the point of using fast interrupts. If interrupts are level triggered, then we must set the PIC masks to prevent endless interrupts, and again we won't see fast interrupts whule handling normal ones. > >> Just wondering, could a device fall back to non-fast mode if the > >> hardware forced this sort of situation but still complain about it? > > > > You don't typically bother requesting a 'fast' interrupt unless you > > really need it. This decision would have to be left up to the > > device driver - some might be OK accepting the tradeoff (eg. sio), > > wheras for others this might constitute a fatal error. The sio and cy drivers strongly prefer to use fast interrupts. The sio driver just falls back to using no interrupts if it can't attach a fast interrupt. The cy driver falls back to normal interrupt and gives up if it can't attach either type. Support for falling back is still deficient. The fallback may need to be done later when another device that requires the non-preferred type is attached. > In fact, unless I'm missing something, it looks as if there's no code > there to stop you from sharing fast interrupts. I could have sworn > there was, but the code in intr_machdep.c doesn't check. The cy This is a bug. Sharing of fast interrupts is possible but is not implemented. Drivers shouldn't need to know this. > driver also sets INTR_EXCL, which is checked for, but nexus_setup_intr > doesn't. If you share a fast interrupt and a slow interrupt, the > first-level handler becomes the mux, and so it appears that the "fast" > attribute would just go away, *without* any warning. I don't know if > there's something else that would stop it from working at all. I think implementation of sharing of fast interrupt handlers only requires a different mux that doesn't call spl*() and is attached as a fast handler. Bruce To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Jul 19 20: 8:35 2000 Delivered-To: freebsd-smp@freebsd.org Received: from wantadilla.lemis.com (wantadilla.lemis.com [192.109.197.80]) by hub.freebsd.org (Postfix) with ESMTP id AB69937BA56 for ; Wed, 19 Jul 2000 20:08:28 -0700 (PDT) (envelope-from grog@wantadilla.lemis.com) Received: (from grog@localhost) by wantadilla.lemis.com (8.9.3/8.9.3) id MAA30400 for FreeBSD-smp@FreeBSD.org; Thu, 20 Jul 2000 12:38:25 +0930 (CST) (envelope-from grog) Date: Thu, 20 Jul 2000 12:38:25 +0930 From: Greg Lehey To: FreeBSD-smp@FreeBSD.org Subject: Interrupt threads: implementation questions Message-ID: <20000720123824.M50320@wantadilla.lemis.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0pre2i Organization: LEMIS, PO Box 460, Echunga SA 5153, Australia Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 WWW-Home-Page: http://www.lemis.com/~grog X-PGP-Fingerprint: 6B 7B C3 8C 61 CD 54 AF 13 24 52 F8 6D A4 95 EF Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org I'm currently working my way through the creation of interrupt processes or threads, and I find that it's a lot more complicated than I thought. The BSD/OS implementation of processes is now so different from ours that porting is quite complicated. At the moment I'm looking at how the process context is allocated. BSD/OS just does /* Allocate new proc. */ psz = sizeof(struct proc); if ((SCARG(uap, runflags) & SF_KTHREAD)) { if (suser(p1->p_cred->pc_ucred, NULL) == EPERM) SCARG(uap, runflags) &= ~SF_KTHREAD; else { /* kernel threads define their proc size */ psz = SCARG(uap, runflags) >> SF_SZSHIFT; } } MALLOC(newproc, struct proc *, psz, M_PROC, M_WAITOK); The reason for this code is that the "struct proc" for kernel threads (in this case, interrupt threads) is really a struct ithd, which contains a struct proc and a few additional fields. In FreeBSD, we allocate the struct proc in fork1() with /* Allocate new proc. */ newproc = zalloc(proc_zone); proc_zone is of type vm_zone_t, and contains additional information. If I read the code correctly, I should be able to define another zone variable, ithread_zone, and then change the code above to /* Allocate new proc. */ if (flags & RFITHREAD) newproc = zalloc(ithread_zone); else newproc = zalloc(proc_zone); Am I missing anything? One thing might be that other kernel threads (non-interrupt) might use a different structure again, though it would be nice to be able to keep them consistent. Or should I just add fields to struct proc? There aren't many additional fields in struct ithd: here's the entire definition from BSD/OS: /* * Interrupt thread (based on a proc) */ struct ithd { struct proc it_proc; int it_need; /* Needs service */ int it_runstat; /* Run lock */ struct intrhand *it_ih; /* head of handler queue */ /* struct ithd *it_interrupted; /* Who we interrupted */ /* stats -- (none yet) */ }; My feeling is that this is enough bloat not to add it to every process, especially when we decide to add the statistics. A better alternative might be to have a pointer to these data in the struct proc, possibly reusing a field which has no meaning in kernel thread context. We could then leave fork1() effectively as it is. BTW, don't get too upset about the term "interrupt thread". As agreed, I'm currently implementing the heavyweight interrupt processes, but the real difference isn't in the allocation, it's in the scheduling. Greg -- Finger grog@lemis.com for PGP public key See complete headers for address and phone numbers To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Jul 22 2:28:27 2000 Delivered-To: freebsd-smp@freebsd.org Received: from wantadilla.lemis.com (wantadilla.lemis.com [192.109.197.80]) by hub.freebsd.org (Postfix) with ESMTP id 7D16137B736 for ; Sat, 22 Jul 2000 02:28:22 -0700 (PDT) (envelope-from grog@wantadilla.lemis.com) Received: (from grog@localhost) by wantadilla.lemis.com (8.9.3/8.9.3) id SAA13805; Sat, 22 Jul 2000 18:57:05 +0930 (CST) (envelope-from grog) Date: Sat, 22 Jul 2000 18:57:05 +0930 From: Greg Lehey To: Chuck Paterson Cc: Matthew Dillon , David Greenman , freebsd-smp@freebsd.org Subject: ipending (was: SMP progress (was: Stepping on Toes)) Message-ID: <20000722185705.A10221@wantadilla.lemis.com> References: <200007051652.KAA14768@berserker.bsdi.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0.1i In-Reply-To: <200007051652.KAA14768@berserker.bsdi.com>; from cp@bsdi.com on Wed, Jul 05, 2000 at 10:52:23AM -0600 Organization: LEMIS, PO Box 460, Echunga SA 5153, Australia Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 WWW-Home-Page: http://www.lemis.com/~grog X-PGP-Fingerprint: 6B 7B C3 8C 61 CD 54 AF 13 24 52 F8 6D A4 95 EF Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Wednesday, 5 July 2000 at 10:52:23 -0600, Chuck Paterson wrote: >> >> Jake Burkholder is porting the BSD/OS mutexes. I don't expect there >> to be much of a difference in regards to your heavy-weight interrupt >> work. I'm going to take a look at Jake's patchset tonight. I think >> the only operational item we need to research is the sti/cli stuff in >> the BSDI mutexes... we should be able to remove them at some point >> (my interrupt code is already using the ipending mechanism to deal >> with the scheduler mutex being active on the current cpu). >> >> If Jake's removed that, then we'll want to put it back in at some point >> since it saves a significant amount of overhead ('sti' and 'cli' are >> expensive instructions). > > I believe ipending wants to go away totally. It really > isn't meaningful in the thread environment and the locked operations > needed to support it once multiple processor are running > in the kernel are more expensive the sti, cli. After working through the code, I agree. It's complicated to maintain, and it's not needed. Removing it also has helped me find a number of places where I need to change things. In a similar way, I'm removing interrupt mask copies in memory. We still mask interrupts which aren't in use, but no others. If anybody has any reason not to want to do this, we should talk about it. Greg -- Finger grog@lemis.com for PGP public key See complete headers for address and phone numbers To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Jul 22 9:20:35 2000 Delivered-To: freebsd-smp@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id DCCB337B809 for ; Sat, 22 Jul 2000 09:20:31 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.9.3/8.9.1) id JAA29862; Sat, 22 Jul 2000 09:20:23 -0700 (PDT) (envelope-from dillon) Date: Sat, 22 Jul 2000 09:20:23 -0700 (PDT) From: Matthew Dillon Message-Id: <200007221620.JAA29862@apollo.backplane.com> To: Greg Lehey Cc: Chuck Paterson , David Greenman , freebsd-smp@freebsd.org Subject: Re: ipending (was: SMP progress (was: Stepping on Toes)) References: <200007051652.KAA14768@berserker.bsdi.com> <20000722185705.A10221@wantadilla.lemis.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org :In a similar way, I'm removing interrupt mask copies in memory. We :still mask interrupts which aren't in use, but no others. If anybody :has any reason not to want to do this, we should talk about it. : :Greg I think you still have to mask level interrupts, otherwise you won't be able to sti. Some subsystems may generate a phenominal number of interrupts while the interrupt routine is running -- for example, the serial ports. I think the masking was put in there as an optmiization not only for that, but also so the interrupt could be EOI'd early so as to allow a new interrupt to become pending while the interrupt routine was running (thus closing a potential window of opportunity where an interrupt might otherwise be missed). If you remove the masking you have to delay the EOI and that is probably a huge mistake because it may lead to lost interrupts. Another example: if a keyboard interrupt is lost you can lose the keyboard entierly. If the EOI is delayed there is a much greater chance of losing the keyboard interrupt. -Matt Matthew Dillon To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Jul 22 9:58:22 2000 Delivered-To: freebsd-smp@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id D646A37BA08 for ; Sat, 22 Jul 2000 09:58:17 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@LOCALHOST [127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id KAA20309; Sat, 22 Jul 2000 10:57:40 -0600 (MDT) Message-Id: <200007221657.KAA20309@berserker.bsdi.com> To: Matthew Dillon Cc: Greg Lehey , David Greenman , freebsd-smp@freebsd.org Subject: Re: ipending (was: SMP progress (was: Stepping on Toes)) From: Chuck Paterson Date: Sat, 22 Jul 2000 10:57:40 -0600 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Greg, When I read you mail I didn't answer because what you said sounded right. But, what was I thinking. The short answer is Matt is correct. When running in APIC mode BSD/OS uses auto EIO for the io APIC and counts on the level which gets set in the local APIC to mask interrupts while the handler is running. The local APIC gets and EIO when the handler is finished. If the thread blocks then the interrupt get masked in hardware if it is a level triggered and in all cases it gets EIO'd. Given that you are just doing heavy weight interrupts this is the equivalent of blocking on an interrupt and you will always need to mask at least level triggered interrupts and EIO them. Matt's comment about bunches of extra level triggered interrupts being a problem is something that is going to have to be looked into with BSD/OS. The reason we don't mask the interrupts now is that doing the actual masking operation is soooo expensive. I suspect we will want to mark which edge triggered interrupts we which to mask. Chuck } } I think you still have to mask level interrupts, otherwise you won't } be able to sti. Some subsystems may generate a phenominal number } of interrupts while the interrupt routine is running -- for example, } the serial ports. I think the masking was put in there as an } optimization not only for that, but also so the interrupt could be } EOI'd early so as to allow a new interrupt to become pending while } the interrupt routine was running (thus closing a potential window of } opportunity where an interrupt might otherwise be missed). } } If you remove the masking you have to delay the EOI and that is probably } a huge mistake because it may lead to lost interrupts. Another example: } if a keyboard interrupt is lost you can lose the keyboard entirely. If } the EOI is delayed there is a much greater chance of losing the keyboard } interrupt. } } -Matt } Matthew Dillon } } } To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Jul 22 10:36:50 2000 Delivered-To: freebsd-smp@freebsd.org Received: from gidora.zeta.org.au (gidora.zeta.org.au [203.26.10.25]) by hub.freebsd.org (Postfix) with SMTP id 28A3937BB32 for ; Sat, 22 Jul 2000 10:36:34 -0700 (PDT) (envelope-from bde@zeta.org.au) Received: (qmail 31055 invoked from network); 22 Jul 2000 17:36:27 -0000 Received: from unknown (HELO bde.zeta.org.au) (203.2.228.102) by gidora.zeta.org.au with SMTP; 22 Jul 2000 17:36:27 -0000 Date: Sun, 23 Jul 2000 03:36:39 +1000 (EST) From: Bruce Evans X-Sender: bde@besplex.bde.org To: Matthew Dillon Cc: Greg Lehey , Chuck Paterson , David Greenman , freebsd-smp@FreeBSD.ORG Subject: Re: ipending (was: SMP progress (was: Stepping on Toes)) In-Reply-To: <200007221620.JAA29862@apollo.backplane.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Sat, 22 Jul 2000, Matthew Dillon wrote: > :In a similar way, I'm removing interrupt mask copies in memory. We > :still mask interrupts which aren't in use, but no others. If anybody > :has any reason not to want to do this, we should talk about it. > : > :Greg That would be wrong. The interrupt masks are kept in memory because it is much faster to access them there. Several hundred times faster than PIC accesses on current machines. When I implemented this on 1992's machines, the memory copies were less than ten times faster. The APIC case is not as bad. I don't know the details of it. > I think you still have to mask level interrupts, otherwise you won't > be able to sti. Some subsystems may generate a phenominal number > of interrupts while the interrupt routine is running -- for example, ^^^ another > the serial ports. I think the masking was put in there as an E.g., one serial port interrupting every 87 usec gives about 50 interrupts while the keyboard interrupt handler is busy-waiting to program the keyboard LEDs. > optmiization not only for that, but also so the interrupt could be > EOI'd early so as to allow a new interrupt to become pending while > the interrupt routine was running (thus closing a potential window of > opportunity where an interrupt might otherwise be missed). This only works right for interrupts other than the one being handled (we guarantee not to miss other interrupts provided they are live for at least the few usec needed for interrupt processing before the EOI). Masking the current interrupt prevents ipending getting set for it, and there is race to exit from the interrupt handler and clear the masks so that a new transient interrupt can be seen before it goes away. We certainly lose this race in some cases, e.g., when the exit is interrupted by another interrupt handler than takes too long. I suspect that most "stray" interrupts are caused by losing this race. > If you remove the masking you have to delay the EOI and that is probably > a huge mistake because it may lead to lost interrupts. Another example: > if a keyboard interrupt is lost you can lose the keyboard entierly. If This doesn't seem to be a problem in practice. I think the keyboard controller times out and sends a new interrupt. > the EOI is delayed there is a much greater chance of losing the keyboard > interrupt. Bruce To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message