From owner-freebsd-arch Sun May 21 10:32: 1 2000 Delivered-To: freebsd-arch@freebsd.org Received: from dustdevil.waterspout.com (standpipe.waterspout.com [208.13.60.152]) by hub.freebsd.org (Postfix) with ESMTP id 2640537B90A for ; Sun, 21 May 2000 10:31:58 -0700 (PDT) (envelope-from csg@dustdevil.waterspout.com) Received: (from csg@localhost) by dustdevil.waterspout.com (8.9.3/8.9.3) id MAA07682; Sun, 21 May 2000 12:35:46 -0500 (EST) (envelope-from csg) Date: Sun, 21 May 2000 12:35:46 -0500 From: "C. Stephen Gunn" To: Doug Rabson Cc: arch@FreeBSD.ORG Subject: Re: A new api for asynchronous task execution (3) Message-ID: <20000521123546.A7522@waterspout.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0.1i In-Reply-To: ; from dfr@nlsystems.com on Thu, May 18, 2000 at 09:24:13PM +0100 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Doug & -arch, First let me say that your taskqueue stuff looks great. In looking over the API, and your implementation, I have some questions. First, as someone else already pointed out, you were calling the tasks at splhigh(). While you fixed this in taskqueue_run(), the call to taskqueue_run() inside taskqueue_free() runs at splhigh(). This is probably intended to avoid new tasks being queued by interrupt handlers while disposing of the queue. What about an intelligent (or stupid) task that tries to queue itself onto the queue again? This isn't an uncommon mechanism for periodic tasks. Having taskqueue_enqueue() return an error if we're in the process of deleting this queue could address this problem. It's a rare case, since I don't foresee the creation/deletion of queues happening with much frequency. I was also wondering about the ta_context mechanism. I initially thought you were trying to hide the task structure behind the API so that binary-only modules (and the like) would be tolerant of changes in the structure. This is probably not the case, since task_enqueue() takes a struct *task. Would there be any benefit to simply passing the task structure to the task_fn()? Tasks could then define a struct with a struct task at the top. This mechanism is certainly used lots of placed in the kernel. I know there's little real difference. I guess my questions is whether the rest of the kernel sets any precident. Next, I was wondering if it made any sense to put a pointer to the taskqueue in the task structure. This would make it easier for a task_fn() to re-queue itself. Or should we doucment that practice as inappropriate. Finally, are you planning to call taskqueue_free() on all the queues at system shutdown? This is obviously outside the scope of the code you'd provided, I was just wondering. - Steve To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun May 21 12:39:54 2000 Delivered-To: freebsd-arch@freebsd.org Received: from obie.softweyr.com (obie.softweyr.com [204.68.178.33]) by hub.freebsd.org (Postfix) with ESMTP id CD5BA37B7E4; Sun, 21 May 2000 12:39:49 -0700 (PDT) (envelope-from wes@softweyr.com) Received: from softweyr.com (homer.softweyr.com [204.68.178.39]) by obie.softweyr.com (8.8.8/8.8.8) with ESMTP id NAA24821; Sun, 21 May 2000 13:39:42 -0600 (MDT) (envelope-from wes@softweyr.com) Message-ID: <39283C21.9A5E6055@softweyr.com> Date: Sun, 21 May 2000 13:42:25 -0600 From: Wes Peters Organization: Softweyr LLC X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 4.0-STABLE i386) X-Accept-Language: en MIME-Version: 1.0 To: Michael Schuster - TSC SunOS Germany Cc: Mike Smith , Chuck Paterson , Doug Rabson , arch@FreeBSD.ORG Subject: Re: A new api for asynchronous task execution References: <200005190529.WAA06634@mass.cdrom.com> <3924E16A.B1123749@softweyr.com> <3924E25D.DEB9F987@germany.sun.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Michael Schuster - TSC SunOS Germany wrote: > > Wes Peters wrote: > > > > Mike Smith wrote: > > > > > > You could call this "priority lending" so that the rest of us understand > > > what you're talking about. 8) > > > > The only system I've ever worked on that implements them refer to > > them as inversion-safe or inversion-proof semaphores. I've never seen > > another name, including "priority lending", in any literature or > > article on the subject. > > 'scuse me for barging in - this sounds like what I know as "priority > inheritance" (as one solution to the priority inversion problem). Are we > talking of the same thing here? If no, what _is_ the difference? If yes, I > could probably dig up one or two papers using these terms (we do at Sun, > but that's not necessarily the place you're looking ... :-) It's not really inheritance, since the two processes don't have to be related to each other in any way other than both are attempting to or have acquired the same lock. -- "Where am I, and what am I doing in this handbasket?" Wes Peters Softweyr LLC wes@softweyr.com http://softweyr.com/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun May 21 14:18: 6 2000 Delivered-To: freebsd-arch@freebsd.org Received: from obie.softweyr.com (obie.softweyr.com [204.68.178.33]) by hub.freebsd.org (Postfix) with ESMTP id 9ABB937B5B8 for ; Sun, 21 May 2000 14:18:03 -0700 (PDT) (envelope-from wes@softweyr.com) Received: from softweyr.com (homer.softweyr.com [204.68.178.39]) by obie.softweyr.com (8.8.8/8.8.8) with ESMTP id PAA24979; Sun, 21 May 2000 15:17:50 -0600 (MDT) (envelope-from wes@softweyr.com) Message-ID: <39285320.B1DD9837@softweyr.com> Date: Sun, 21 May 2000 15:20:32 -0600 From: Wes Peters Organization: Softweyr LLC X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 4.0-STABLE i386) X-Accept-Language: en MIME-Version: 1.0 To: Terry Lambert Cc: Chuck Paterson , Doug Rabson , arch@FreeBSD.ORG Subject: Re: Sparc & api for asynchronous task execution (2) References: <200005191744.KAA00232@usr02.primenet.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Terry Lambert wrote: > > > Terry Lambert wrote: > > > > > > > }I didn't realize that every function call involved a fault on the sparc > > > > }architecture - that sounds pretty nasty. > > > > > > > > Actually every function call doesn't cause a fault, every > > > > time you overflow/underflow the current set of register windows causes a > > > > fault. (Perhaps what you meant). This means that calling a function > > > > from the bottom most function will cause two faults, one for going down > > > > and one eventually as you go up. > > > > > > > > This makes going up and down when you don't overflow > > > > very fast at the expense of when you add to the total depth. > > > > The register window sizes weren't picked willy-nilly. The SPARC default > > size is 7 windows, chosen after months of analyzing every M68K SunOS > > program they could get their hands, including compiled C, Pascal, LISP, > > and Fortran programs. > > > > I suspect C++ and Java probably skew these stats a little. This might > > account for the 8 windows in more modern SPARC processors. > > Please watch your attributions; I said none of this. Precisely where did you get the idea I was accusing you of anything? I was merely adding to your assertion that the register window system was carefully designed as an optimization, not as the foolishness the original questioner seemed to think. Calling into question those who are supporting your arguments isn't a great way to enforce your point. -- "Where am I, and what am I doing in this handbasket?" Wes Peters Softweyr LLC wes@softweyr.com http://softweyr.com/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun May 21 15:42:28 2000 Delivered-To: freebsd-arch@freebsd.org Received: from obie.softweyr.com (obie.softweyr.com [204.68.178.33]) by hub.freebsd.org (Postfix) with ESMTP id 6DC9637B8A6 for ; Sun, 21 May 2000 15:42:24 -0700 (PDT) (envelope-from wes@softweyr.com) Received: from softweyr.com (homer.softweyr.com [204.68.178.39]) by obie.softweyr.com (8.8.8/8.8.8) with ESMTP id QAA25124; Sun, 21 May 2000 16:42:07 -0600 (MDT) (envelope-from wes@softweyr.com) Message-ID: <392866E2.389A6935@softweyr.com> Date: Sun, 21 May 2000 16:44:50 -0600 From: Wes Peters Organization: Softweyr LLC X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 4.0-STABLE i386) X-Accept-Language: en MIME-Version: 1.0 To: Arun Sharma Cc: arch@FreeBSD.ORG Subject: Re: Sparc & api for asynchronous task execution (2) References: <200005182045.NAA21595@usr08.primenet.com> <3924DA18.392990EA@softweyr.com> <20000520104248.A2866@sharmas.dhs.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Arun Sharma wrote: > > On Fri, May 19, 2000 at 12:07:20AM -0600, Wes Peters wrote: > > > > This makes going up and down when you don't overflow > > > > very fast at the expense of when you add to the total depth. > > > > The register window sizes weren't picked willy-nilly. The SPARC default > > size is 7 windows, chosen after months of analyzing every M68K SunOS > > program they could get their hands, including compiled C, Pascal, LISP, > > and Fortran programs. > > Times have changed. If you have looked at stack traces in Mozilla or > Konqueror, they easily go 40 calls deep. Even old SPARC software typically went 20 to 25 frames deep, that's not the issue. The issue is how many frames are active during the normal useful 75% - 85% of a program lifecycle, when it is actively accomplishing work, as opposed to starting up or shutting down. I suspect this is higher for C++/Java code than apocryphal C/Pascal/Fortran code, but I doubt it is more than 1 or 2 higher than the original 7. > The real problem with sparc register windows is flushing of the stack > on every system call. Clever software tricks can avoid this on IA-64. On every context switch, you mean. The fastest way to do this is to put a LOT of registers on the chip, obviating the need to spill registers unless you have a LOT of threads. -- "Where am I, and what am I doing in this handbasket?" Wes Peters Softweyr LLC wes@softweyr.com http://softweyr.com/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun May 21 17: 6:18 2000 Delivered-To: freebsd-arch@freebsd.org Received: from urban.iinet.net.au (urban.iinet.net.au [203.59.24.231]) by hub.freebsd.org (Postfix) with ESMTP id 11F7D37B98F; Sun, 21 May 2000 17:06:11 -0700 (PDT) (envelope-from julian@elischer.org) Received: from muzak.iinet.net.au (muzak.iinet.net.au [203.59.24.237]) by urban.iinet.net.au (8.8.7/8.8.7) with ESMTP id IAA10049; Mon, 22 May 2000 08:06:04 +0800 Received: from jules.elischer.org (reggae-09-5.nv.iinet.net.au [203.59.67.5]) by muzak.iinet.net.au (8.8.5/8.8.5) with SMTP id IAA24655; Mon, 22 May 2000 08:05:53 +0800 Message-ID: <39287950.167EB0E7@elischer.org> Date: Sun, 21 May 2000 17:03:28 -0700 From: Julian Elischer X-Mailer: Mozilla 3.04Gold (X11; I; FreeBSD 5.0-CURRENT i386) MIME-Version: 1.0 To: Mike Smith Cc: Chuck Paterson , arch@freebsd.org Subject: Re: BSD* mutex summary References: <200005191923.MAA09426@mass.cdrom.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Mike Smith wrote: > > Given the recent exercise over the proposed kernel thread architecture, > where after much discussion and the invention of a great deal of new > terminology we basically reinvented scheduler activations (described in a > 10 year old paper), I think that cleaving to existing terminology is > probably a good idea. > I didn't use the existing terminology deliberatly so that we wouldn't get preconcieved notions of how various components would behave. The aim was that when we had narrowed down what we wanted, we'd figure out what 'normal' nomenclature would best fit. This in fact happenned. If you ask jason, we are basically now using the terms from the SA paper. -- __--_|\ Julian Elischer / \ julian@elischer.org ( OZ ) World tour 2000 ---> X_.---._/ presently in: Perth v To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun May 21 21:11:21 2000 Delivered-To: freebsd-arch@freebsd.org Received: from obie.softweyr.com (obie.softweyr.com [204.68.178.33]) by hub.freebsd.org (Postfix) with ESMTP id C746D37B538 for ; Sun, 21 May 2000 21:11:17 -0700 (PDT) (envelope-from wes@softweyr.com) Received: from softweyr.com (homer.softweyr.com [204.68.178.39]) by obie.softweyr.com (8.8.8/8.8.8) with ESMTP id WAA25670; Sun, 21 May 2000 22:10:54 -0600 (MDT) (envelope-from wes@softweyr.com) Message-ID: <3928B3EF.B0BFFB1@softweyr.com> Date: Sun, 21 May 2000 22:13:35 -0600 From: Wes Peters Organization: Softweyr LLC X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 4.0-STABLE i386) X-Accept-Language: en MIME-Version: 1.0 To: Hajimu UMEMOTO Cc: des@flood.ping.uio.no, arch@FreeBSD.ORG Subject: Re: fetch(1) References: <200005122016.e4CKGtF38185@peace.mahoroba.org> <200005201832.e4KIWYc06469@peace.mahoroba.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG "Hajimu UMEMOTO (梅本 肇)" wrote: > > >>>>> On 15 May 2000 10:58:40 +0200 > >>>>> Dag-Erling Smorgrav said: > > > http://www.imasy.or.jp/~ume/ipv6/FreeBSD/libfetch-ipv6.diff.gz > > http://www.imasy.or.jp/~ume/ipv6/FreeBSD/fetch-20000508-ipv6.diff > > des> The patches look fine - I don't have enough experience with IPv6 to > des> judge their technical merits, but they're relatively low-impact and I > des> assume that you know what you're doing. If you commit the libfetch > des> part of the patches, I'll merge the fetch part and put up a new > des> tarball. > > Thank you. > I just committed my IPv6 support code. > Please merge fetch part. Speaking of which, if you don't have an implementation of fetchListFTP, I do. Let me know if any of you would like to review it. I don't have it with me at home right now, or I'd just attach it. -- "Where am I, and what am I doing in this handbasket?" Wes Peters Softweyr LLC wes@softweyr.com http://softweyr.com/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun May 21 23:39:36 2000 Delivered-To: freebsd-arch@freebsd.org Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by hub.freebsd.org (Postfix) with ESMTP id EB75137C039 for ; Sun, 21 May 2000 23:39:33 -0700 (PDT) (envelope-from michael.schuster@germany.sun.com) Received: from emuc05-home.Germany.Sun.COM ([129.157.51.10]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id XAA12483; Sun, 21 May 2000 23:39:31 -0700 (PDT) Received: from germany.sun.com (hacker [129.157.167.97]) by emuc05-home.Germany.Sun.COM (8.8.8+Sun/8.8.8/ENSMAIL,v1.7) with ESMTP id IAA11189; Mon, 22 May 2000 08:39:30 +0200 (MET DST) Message-ID: <3928D5D9.75E06D94@germany.sun.com> Date: Mon, 22 May 2000 08:38:17 +0200 From: Michael Schuster - TSC SunOS Germany Organization: Sun Microsystems, Inc. X-Mailer: Mozilla 4.73 [en] (X11; I; SunOS 5.8 sun4u) X-Accept-Language: en MIME-Version: 1.0 To: Chuck Paterson Cc: arch@FreeBSD.ORG Subject: Re: A new api for asynchronous task execution References: <200005191912.NAA11320@berserker.bsdi.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Chuck Paterson wrote: > Solaris's default mutexs are adaptive mutexs. Adaptive mutexs > may spin for a bit before giving up and task switching. Let me clarify this: An adaptive mutex spins only when the holder of the mutex is _running_ on another CPU (under the assumption that it will release the lock "soon"). Otherwise, it will block. cheers Michael -- Michael Schuster / Michael.Schuster@germany.sun.com Sun Microsystems GmbH / Richard-Reitzner Allee 8, D-85540 Haar (+49 89) 46008 974 / x12974 Recursion, n.: see 'Recursion' To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon May 22 0:31:43 2000 Delivered-To: freebsd-arch@freebsd.org Received: from flood.ping.uio.no (flood.ping.uio.no [129.240.78.31]) by hub.freebsd.org (Postfix) with ESMTP id 4C71837B54A; Mon, 22 May 2000 00:31:40 -0700 (PDT) (envelope-from des@flood.ping.uio.no) Received: (from des@localhost) by flood.ping.uio.no (8.9.3/8.9.3) id JAA75680; Mon, 22 May 2000 09:31:38 +0200 (CEST) (envelope-from des@flood.ping.uio.no) To: Brian Fundakowski Feldman Cc: Hajimu UMEMOTO , arch@FreeBSD.org Subject: Re: fetch(1) References: From: Dag-Erling Smorgrav Date: 22 May 2000 09:31:37 +0200 In-Reply-To: Brian Fundakowski Feldman's message of "Fri, 19 May 2000 21:39:30 -0400 (EDT)" Message-ID: Lines: 11 User-Agent: Gnus/5.0802 (Gnus v5.8.2) Emacs/20.4 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Brian Fundakowski Feldman writes: > Right now with that change, the code will merrilly skip along addr[], > setting the members to 0 and incrementing i to 6. While this is not > especially harmful, you should probably fix the "*p, i < 6" which is > precisely equivalent to "i < 6" to be "*p != '\0' && i < 6". Uh, yeah. Too much blood in my cokestream :) DES -- Dag-Erling Smorgrav - des@flood.ping.uio.no To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon May 22 0:37: 3 2000 Delivered-To: freebsd-arch@freebsd.org Received: from flood.ping.uio.no (flood.ping.uio.no [129.240.78.31]) by hub.freebsd.org (Postfix) with ESMTP id 6F62337BBA3 for ; Mon, 22 May 2000 00:36:54 -0700 (PDT) (envelope-from des@flood.ping.uio.no) Received: (from des@localhost) by flood.ping.uio.no (8.9.3/8.9.3) id JAA75704; Mon, 22 May 2000 09:36:46 +0200 (CEST) (envelope-from des@flood.ping.uio.no) To: Wes Peters Cc: Hajimu UMEMOTO , arch@FreeBSD.ORG Subject: Re: fetch(1) References: <200005122016.e4CKGtF38185@peace.mahoroba.org> <200005201832.e4KIWYc06469@peace.mahoroba.org> <3928B3EF.B0BFFB1@softweyr.com> From: Dag-Erling Smorgrav Date: 22 May 2000 09:36:45 +0200 In-Reply-To: Wes Peters's message of "Sun, 21 May 2000 22:13:35 -0600" Message-ID: Lines: 10 User-Agent: Gnus/5.0802 (Gnus v5.8.2) Emacs/20.4 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Wes Peters writes: > Speaking of which, if you don't have an implementation of fetchListFTP, > I do. Let me know if any of you would like to review it. I don't have > it with me at home right now, or I'd just attach it. Sure, fire away, I'll review it. DES -- Dag-Erling Smorgrav - des@flood.ping.uio.no To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon May 22 1:20:39 2000 Delivered-To: freebsd-arch@freebsd.org Received: from flood.ping.uio.no (flood.ping.uio.no [129.240.78.31]) by hub.freebsd.org (Postfix) with ESMTP id 60A1C37B6BC for ; Mon, 22 May 2000 01:20:35 -0700 (PDT) (envelope-from des@flood.ping.uio.no) Received: (from des@localhost) by flood.ping.uio.no (8.9.3/8.9.3) id KAA75895; Mon, 22 May 2000 10:20:26 +0200 (CEST) (envelope-from des@flood.ping.uio.no) To: Hajimu UMEMOTO (=?iso-2022-jp?b?GyRCR19LXBsoQiAbJEJIJRsoQg==?=) Cc: arch@FreeBSD.ORG Subject: Re: fetch(1) References: <200005122016.e4CKGtF38185@peace.mahoroba.org> <200005201832.e4KIWYc06469@peace.mahoroba.org> From: Dag-Erling Smorgrav Date: 22 May 2000 10:20:26 +0200 In-Reply-To: Hajimu UMEMOTO's message of "Sun, 21 May 2000 03:32:34 +0900 (JST)" Message-ID: Lines: 10 User-Agent: Gnus/5.0802 (Gnus v5.8.2) Emacs/20.4 MIME-Version: 1.0 Content-Type: text/plain; charset=iso-2022-jp Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Hajimu UMEMOTO (梅本 肇) writes: > Thank you. > I just committed my IPv6 support code. > Please merge fetch part. Does it need -DINET6 in CFLAGS? DES -- Dag-Erling Smorgrav - des@flood.ping.uio.no To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon May 22 1:26: 3 2000 Delivered-To: freebsd-arch@freebsd.org Received: from anchor-post-30.mail.demon.net (anchor-post-30.mail.demon.net [194.217.242.88]) by hub.freebsd.org (Postfix) with ESMTP id D1D2D37C13D for ; Mon, 22 May 2000 01:25:48 -0700 (PDT) (envelope-from dfr@nlsystems.com) Received: from nlsys.demon.co.uk ([158.152.125.33] helo=herring.nlsystems.com) by anchor-post-30.mail.demon.net with esmtp (Exim 2.12 #1) id 12tnWz-000FEZ-0U; Mon, 22 May 2000 09:25:46 +0100 Received: from salmon.nlsystems.com (salmon.nlsystems.com [10.0.0.3]) by herring.nlsystems.com (8.9.3/8.8.8) with ESMTP id JAA96525; Mon, 22 May 2000 09:28:55 +0100 (BST) (envelope-from dfr@nlsystems.com) Date: Mon, 22 May 2000 09:28:41 +0100 (BST) From: Doug Rabson To: "C. Stephen Gunn" Cc: arch@FreeBSD.ORG Subject: Re: A new api for asynchronous task execution (3) In-Reply-To: <20000521123546.A7522@waterspout.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Sun, 21 May 2000, C. Stephen Gunn wrote: > > Doug & -arch, > > First let me say that your taskqueue stuff looks great. In looking > over the API, and your implementation, I have some questions. > > First, as someone else already pointed out, you were calling the > tasks at splhigh(). While you fixed this in taskqueue_run(), the > call to taskqueue_run() inside taskqueue_free() runs at splhigh(). > This is probably intended to avoid new tasks being queued by > interrupt handlers while disposing of the queue. > > What about an intelligent (or stupid) task that tries to queue > itself onto the queue again? This isn't an uncommon mechanism for > periodic tasks. > > Having taskqueue_enqueue() return an error if we're in the process > of deleting this queue could address this problem. It's a rare > case, since I don't foresee the creation/deletion of queues happening > with much frequency. The tasks called via taskqueue_free() should probably not be called at splhigh(). I think the right solution is to put the queue into a 'closing' state and return errors from taskqueue_enqueue as you suggest. > > I was also wondering about the ta_context mechanism. I initially > thought you were trying to hide the task structure behind the API > so that binary-only modules (and the like) would be tolerant of > changes in the structure. This is probably not the case, since > task_enqueue() takes a struct *task. > > Would there be any benefit to simply passing the task structure to > the task_fn()? > > Tasks could then define a struct with a struct task at the top. > This mechanism is certainly used lots of placed in the kernel. > I know there's little real difference. I guess my questions is > whether the rest of the kernel sets any precident. I'm not particularly trying to hide the task structure. The caller is reponsible for allocating the task anyway. The intention of having a context value to pass to the function is flexibility. If you just pass the task pointer, then you force users to lay their structures out in a certain way which may not be possible in general. Adding a pointer to the structure is a pretty low-cost way of allowing the user to do anything they need. > > Next, I was wondering if it made any sense to put a pointer to the > taskqueue in the task structure. This would make it easier for > a task_fn() to re-queue itself. Or should we doucment that > practice as inappropriate. I thought about this and decided against it. I have some vague plans to extend the system to allow several queues with the same name. This would change taskqueue_find() to cycle through the queues round-robin fashion and would be used for SMP load balancing. > > Finally, are you planning to call taskqueue_free() on all the queues > at system shutdown? This is obviously outside the scope of the > code you'd provided, I was just wondering. Probably not. -- Doug Rabson Mail: dfr@nlsystems.com Nonlinear Systems Ltd. Phone: +44 20 8442 9037 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon May 22 2:19:35 2000 Delivered-To: freebsd-arch@freebsd.org Received: from hitpro.hitachi.co.jp (hitpro.hitachi.co.jp [133.145.224.7]) by hub.freebsd.org (Postfix) with ESMTP id 992D237B796 for ; Mon, 22 May 2000 02:19:31 -0700 (PDT) (envelope-from ume@bisd.hitachi.co.jp) Received: from bisdgw.bisd.hitachi.co.jp by hitpro.hitachi.co.jp (8.9.3/3.7W-hitpro) id SAA16441; Mon, 22 May 2000 18:19:21 +0900 (JST) Received: from plum.ssr.bisd.hitachi.co.jp by bisdgw.bisd.hitachi.co.jp (8.9.3+3.2W/3.7W-bisdgw) with ESMTP id SAA18684; Mon, 22 May 2000 18:19:21 +0900 (JST) (envelope-from ume@bisd.hitachi.co.jp) Received: from localhost (IDENT:2/+h6wXcF0O8X6Eh2ahkf/zVk9GrDbun+5FFVdXnSjrjI3jiSMKu7zJAv+GHGVc9@localhost [::1]) by plum.ssr.bisd.hitachi.co.jp (8.10.1/3.7W-plum) with ESMTP id e4M9JLx14594; Mon, 22 May 2000 18:19:21 +0900 (JST) (envelope-from ume@bisd.hitachi.co.jp) Message-Id: <200005220919.e4M9JLx14594@plum.ssr.bisd.hitachi.co.jp> To: des@flood.ping.uio.no Cc: arch@FreeBSD.ORG Cc: ume@bisd.hitachi.co.jp Subject: Re: fetch(1) From: Hajimu UMEMOTO (=?iso-2022-jp?B?GyRCR19LXBsoQiAbJEJIJRsoQg==?=) In-Reply-To: References: <200005201832.e4KIWYc06469@peace.mahoroba.org> X-Mailer: xcite1.20> Mew version 1.94.2 on XEmacs 21.1 (Bryce Canyon) X-PGP-Fingerprint: D3 3D D3 54 88 13 DE 22 3F 31 C4 4D A1 08 84 7B X-PGP-Public-Key: http://www.imasy.org/~ume/ume@bisd.hitachi.co.jp.asc X-URL: http://www.imasy.org/~ume/ X-OS: FreeBSD 4.0-STABLE Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Date: Mon, 22 May 2000 18:19:20 +0900 X-Dispatcher: imput version 20000228(IM140) Lines: 11 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG >>>>> On 22 May 2000 10:20:26 +0200 >>>>> des@flood.ping.uio.no (Dag-Erling Smorgrav) said: des> Does it need -DINET6 in CFLAGS? Do you say about fetch(1)? If so, you don't need -DINET6. -- Hajimu UMEMOTO @ Business Solution System Development Div., Hitachi Ltd. E-Mail: ume@bisd.hitachi.co.jp ume@mahoroba.org ume@FreeBSD.org URL: http://www.imasy.org/~ume/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon May 22 5: 7:28 2000 Delivered-To: freebsd-arch@freebsd.org Received: from flood.ping.uio.no (flood.ping.uio.no [129.240.78.31]) by hub.freebsd.org (Postfix) with ESMTP id 80EC837B653 for ; Mon, 22 May 2000 05:07:24 -0700 (PDT) (envelope-from des@flood.ping.uio.no) Received: (from des@localhost) by flood.ping.uio.no (8.9.3/8.9.3) id OAA76772; Mon, 22 May 2000 14:07:17 +0200 (CEST) (envelope-from des@flood.ping.uio.no) To: Hajimu UMEMOTO (=?iso-2022-jp?b?GyRCR19LXBsoQiAbJEJIJRsoQg==?=) Cc: arch@FreeBSD.ORG Subject: Re: fetch(1) References: <200005201832.e4KIWYc06469@peace.mahoroba.org> <200005220919.e4M9JLx14594@plum.ssr.bisd.hitachi.co.jp> From: Dag-Erling Smorgrav Date: 22 May 2000 14:07:16 +0200 In-Reply-To: Hajimu UMEMOTO's message of "Mon, 22 May 2000 18:19:20 +0900" Message-ID: Lines: 12 User-Agent: Gnus/5.0802 (Gnus v5.8.2) Emacs/20.4 MIME-Version: 1.0 Content-Type: text/plain; charset=iso-2022-jp Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Hajimu UMEMOTO (梅本 肇) writes: > >>>>> On 22 May 2000 10:20:26 +0200 > >>>>> des@flood.ping.uio.no (Dag-Erling Smorgrav) said: > des> Does it need -DINET6 in CFLAGS? > Do you say about fetch(1)? If so, you don't need -DINET6. OK, the new (IPv6-enabled) version is now available from the usual place: DES -- Dag-Erling Smorgrav - des@flood.ping.uio.no To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon May 22 7:28:59 2000 Delivered-To: freebsd-arch@freebsd.org Received: from dustdevil.waterspout.com (fourier.physics.purdue.edu [128.210.146.43]) by hub.freebsd.org (Postfix) with ESMTP id 6550437B98D for ; Mon, 22 May 2000 07:28:51 -0700 (PDT) (envelope-from csg@dustdevil.waterspout.com) Received: (from csg@localhost) by dustdevil.waterspout.com (8.9.3/8.9.3) id JAA14163; Mon, 22 May 2000 09:32:18 -0500 (EST) (envelope-from csg) Date: Mon, 22 May 2000 09:32:18 -0500 From: "C. Stephen Gunn" To: Doug Rabson Cc: arch@FreeBSD.ORG Subject: Re: A new api for asynchronous task execution (3) Message-ID: <20000522093218.A14101@waterspout.com> References: <20000521123546.A7522@waterspout.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0.1i In-Reply-To: ; from dfr@nlsystems.com on Mon, May 22, 2000 at 09:28:41AM +0100 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Mon, May 22, 2000 at 09:28:41AM +0100, Doug Rabson wrote: > The tasks called via taskqueue_free() should probably not be called at > splhigh(). I think the right solution is to put the queue into a 'closing' > state and return errors from taskqueue_enqueue as you suggest. Perhaps we should not be running the tasks at all. If we don't care enough to clean them up at system shutdown, who cares when a queue is going away. But, it's possible I misunderstand the frequency of taskqueue creation and destruction in the average system. > I'm not particularly trying to hide the task structure. The caller is > responsible for allocating the task anyway. The intention of having a > context value to pass to the function is flexibility. > > If you just pass the task pointer, then you force users to lay their > structures out in a certain way which may not be possible in > general. Adding a pointer to the structure is a pretty low-cost way of > allowing the user to do anything they need. Couldn't you still pass the struct task so that the task_fn() could inspect anything it might be interested in without caching a pointer? You could still access application data just as easily from the pointer the struct task. (or stick it at the head of another structure, your choice). > > Next, I was wondering if it made any sense to put a pointer to the > > taskqueue in the task structure. This would make it easier for > > a task_fn() to re-queue itself. Or should we document that > > practice as inappropriate. > > I thought about this and decided against it. I have some vague plans to > extend the system to allow several queues with the same name. This > would change taskqueue_find() to cycle through the queues round-robin > fashion and would be used for SMP load balancing. That's an interesting idea. You could also multiplex these queues on taskqueue_run() and taskqueue_enqueue(). I'm not sure that calling taskqueue_find() for every enqueue() should be necessary. I'd expect to call taskqueue_find() and cache a pointer to the queue. Which could be bad, since the queue could go away with some areas of the kernel retaining a pointer. Hmm... - Steve To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon May 22 13:23:30 2000 Delivered-To: freebsd-arch@freebsd.org Received: from finch-post-11.mail.demon.net (finch-post-11.mail.demon.net [194.217.242.39]) by hub.freebsd.org (Postfix) with ESMTP id AED7237BC2B for ; Mon, 22 May 2000 13:23:20 -0700 (PDT) (envelope-from dfr@nlsystems.com) Received: from nlsys.demon.co.uk ([158.152.125.33] helo=herring.nlsystems.com) by finch-post-11.mail.demon.net with esmtp (Exim 2.12 #1) id 12tyjA-000I5i-0B; Mon, 22 May 2000 20:23:04 +0000 Received: from salmon.nlsystems.com (salmon.nlsystems.com [10.0.0.3]) by herring.nlsystems.com (8.9.3/8.8.8) with ESMTP id VAA29767; Mon, 22 May 2000 21:29:03 +0100 (BST) (envelope-from dfr@nlsystems.com) Date: Mon, 22 May 2000 21:28:45 +0100 (BST) From: Doug Rabson To: "C. Stephen Gunn" Cc: arch@FreeBSD.ORG Subject: Re: A new api for asynchronous task execution (3) In-Reply-To: <20000522093218.A14101@waterspout.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Mon, 22 May 2000, C. Stephen Gunn wrote: > On Mon, May 22, 2000 at 09:28:41AM +0100, Doug Rabson wrote: > > > The tasks called via taskqueue_free() should probably not be called at > > splhigh(). I think the right solution is to put the queue into a 'closing' > > state and return errors from taskqueue_enqueue as you suggest. > > Perhaps we should not be running the tasks at all. If we don't > care enough to clean them up at system shutdown, who cares when > a queue is going away. But, it's possible I misunderstand the > frequency of taskqueue creation and destruction in the average > system. Possibly. I was originally going to just drop the tasks on the floor but Nick Hibma suggested that I drain the queue one last time at taskqueue_free(). I don't think it really matters as long as its clearly documented what the behaviour is. > > > I'm not particularly trying to hide the task structure. The caller is > > responsible for allocating the task anyway. The intention of having a > > context value to pass to the function is flexibility. > > > > If you just pass the task pointer, then you force users to lay their > > structures out in a certain way which may not be possible in > > general. Adding a pointer to the structure is a pretty low-cost way of > > allowing the user to do anything they need. > > Couldn't you still pass the struct task so that the task_fn() could > inspect anything it might be interested in without caching a pointer? > You could still access application data just as easily from the > pointer the struct task. (or stick it at the head of another > structure, your choice). The point is that if the caller wants the task pointer, they can put that as the value of ta_context but if they want something else and we always pass the task pointer, then they are stuck. > > > > Next, I was wondering if it made any sense to put a pointer to the > > > taskqueue in the task structure. This would make it easier for > > > a task_fn() to re-queue itself. Or should we document that > > > practice as inappropriate. > > > > I thought about this and decided against it. I have some vague plans to > > extend the system to allow several queues with the same name. This > > would change taskqueue_find() to cycle through the queues round-robin > > fashion and would be used for SMP load balancing. > > That's an interesting idea. You could also multiplex these queues > on taskqueue_run() and taskqueue_enqueue(). I'm not sure that > calling taskqueue_find() for every enqueue() should be necessary. > > I'd expect to call taskqueue_find() and cache a pointer to the > queue. Which could be bad, since the queue could go away with some > areas of the kernel retaining a pointer. Hmm... There is definately some more work to do here to make something which is both useful and performs well. It would mean making taskqueue_find() scale better for lots of queues for a start. -- Doug Rabson Mail: dfr@nlsystems.com Nonlinear Systems Ltd. Phone: +44 20 8442 9037 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue May 23 16:30:53 2000 Delivered-To: freebsd-arch@freebsd.org Received: from smtp02.primenet.com (smtp02.primenet.com [206.165.6.132]) by hub.freebsd.org (Postfix) with ESMTP id AB83137B902 for ; Tue, 23 May 2000 16:30:48 -0700 (PDT) (envelope-from tlambert@usr05.primenet.com) Received: (from daemon@localhost) by smtp02.primenet.com (8.9.3/8.9.3) id QAA13383; Tue, 23 May 2000 16:30:41 -0700 (MST) Received: from usr05.primenet.com(206.165.6.205) via SMTP by smtp02.primenet.com, id smtpdAAAqGaW_z; Tue May 23 16:30:33 2000 Received: (from tlambert@localhost) by usr05.primenet.com (8.8.5/8.8.5) id QAA01056; Tue, 23 May 2000 16:30:34 -0700 (MST) From: Terry Lambert Message-Id: <200005232330.QAA01056@usr05.primenet.com> Subject: Re: Sparc & api for asynchronous task execution (2) To: wes@softweyr.com (Wes Peters) Date: Tue, 23 May 2000 23:30:34 +0000 (GMT) Cc: tlambert@primenet.com (Terry Lambert), cp@bsdi.com (Chuck Paterson), dfr@nlsystems.com (Doug Rabson), arch@FreeBSD.ORG In-Reply-To: <39285320.B1DD9837@softweyr.com> from "Wes Peters" at May 21, 2000 03:20:32 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > > Please watch your attributions; I said none of this. > > Precisely where did you get the idea I was accusing you of anything? I didn't think you wre accusing me of anything. > I was merely adding to your assertion that the register window system > was carefully designed as an optimization, not as the foolishness the > original questioner seemed to think. Yeah, but you didn't include any of my text, only "Terry Lambert wrote:". 8-p. > Calling into question those who are supporting your arguments isn't a > great way to enforce your point. I'm just trying to get out from under being held accountable for someone else's writings. I've been take to task before for things written by other people and then misattributes in these mailing lists, so I'm a bit sensitive... Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 7:46: 9 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id F068637BC6C for ; Wed, 24 May 2000 07:46:04 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@localhost [127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id IAA05589 for ; Wed, 24 May 2000 08:46:03 -0600 (MDT) Message-Id: <200005241446.IAA05589@berserker.bsdi.com> To: arch@freebsd.org Subject: Preemptive kernel on older X86 hardware From: Chuck Paterson Date: Wed, 24 May 2000 08:46:03 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG The 386 processors do not support the locked exchange instruction. For these systems the locked exchange can be replaced with roughly a "cli, tst, cmp, branch, store, sti". The following are the obvious choices 1. Make two separate builds. 2. Add both sets of code to all macros and do run time checks. The penalties are: a) doing the check on each mutex operation. b) bigger kernel c) since both sets of code will be spread through out the kernel next to each other it is reasonable to expect that running either set of code will be using more cache lines than they would if they existed alone. To add a little perspective there are currently around 1300 locking operation in the BSD/OS kernel that would be effected by this. This is really a guess, but when all is said and done I would say BSD/OS will end up with between 1700 and 2500. Kernel size seems like a non issue, the number of extra cache lines used also seems likely to be a non issue. So it seems like it really comes down to adding the extra test which will be significant compared to the actual mutex operation. 3) Drop support for 386. Neither the 386 nor the 486 have a processor priority register or cycle counters. Currently the BSD/OS SMPng kernel requires both of these. There already exists some left over code to deal with not having a cycle counter. Doing a casual inspection there really doesn't seem to be anything too ugly in making the system run without these when there is only a single processor. I really don't know what BSDI will finally decided. I am confident that we will not have run time checks in the locking operations. The argument, within BSDI, for supporting the older stuff is that new embedded systems are being built with these processors. Raw chip cost being the reason. While I believe the person telling me this, I haven't personally seen the evidence, I haven't looked either. I have talked to a couple of people who think that supporting this older stuff won't be important to FreeBSD by the time the kernel is preemptive. Some even thought supporting the original Pentium processors might not be required. I'll propose the following as it reduces the work required: Once FreeBSD has a preemptive kernel FreeBSD will only run on Pentium or better X86 processors. I'm not wed to the above at all, it is more to get discussion started than anything else. Then again, if no one complains, it is easier. Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 8:13:45 2000 Delivered-To: freebsd-arch@freebsd.org Received: from outmail.utsunomiya-u.ac.jp (outmail.utsunomiya-u.ac.jp [160.12.196.3]) by hub.freebsd.org (Postfix) with ESMTP id D62AB37BC74 for ; Wed, 24 May 2000 08:13:33 -0700 (PDT) (envelope-from yokota@zodiac.mech.utsunomiya-u.ac.jp) Received: from zodiac.mech.utsunomiya-u.ac.jp (IDENT:3H1romzcfWiPAHT4zBW6/lMAyahQm27O@zodiac.mech.utsunomiya-u.ac.jp [160.12.42.1]) by outmail.utsunomiya-u.ac.jp (8.9.3/3.7Wpl2) with ESMTP id AAA09895; Thu, 25 May 2000 00:13:17 +0900 (JST) Received: from zodiac.mech.utsunomiya-u.ac.jp (IDENT:8Gn3pliOSUYbClq0tzEU4osyo2qY3VhW@zodiac.mech.utsunomiya-u.ac.jp [160.12.42.1]) by zodiac.mech.utsunomiya-u.ac.jp (8.9.3+3.2W/3.7W/zodiac-May2000) with ESMTP id AAA20525; Thu, 25 May 2000 00:19:55 +0900 (JST) Message-Id: <200005241519.AAA20525@zodiac.mech.utsunomiya-u.ac.jp> To: Chuck Paterson Cc: arch@freebsd.org, yokota@zodiac.mech.utsunomiya-u.ac.jp Subject: Re: Preemptive kernel on older X86 hardware In-reply-to: Your message of "Wed, 24 May 2000 08:46:03 CST." <200005241446.IAA05589@berserker.bsdi.com> References: <200005241446.IAA05589@berserker.bsdi.com> Date: Thu, 25 May 2000 00:19:54 +0900 From: Kazutaka YOKOTA Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG [...] >Neither the 386 nor the 486 have a processor priority register or >cycle counters. Currently the BSD/OS SMPng kernel requires both of >these. There already exists some left over code to deal with not >having a cycle counter. Doing a casual inspection there really >doesn't seem to be anything too ugly in making the system run >without these when there is only a single processor. 386 and 486 do not support multi-processor configuration, don't they? Kazu To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 8:15:44 2000 Delivered-To: freebsd-arch@freebsd.org Received: from vuurwerk.nl (envy.vuurwerk.nl [194.178.232.112]) by hub.freebsd.org (Postfix) with SMTP id 2333537BC74 for ; Wed, 24 May 2000 08:15:41 -0700 (PDT) (envelope-from petervd@vuurwerk.nl) Received: (qmail 59891 invoked from network); 24 May 2000 15:15:38 -0000 Received: from kesteren.vuurwerk.nl (HELO vuurwerk.nl) (194.178.232.59) by envy.vuurwerk.nl with SMTP; 24 May 2000 15:15:38 -0000 Received: (qmail 10788 invoked by uid 11109); 24 May 2000 15:15:37 -0000 Mail-Followup-To: arch@FreeBSD.ORG Date: Wed, 24 May 2000 17:15:37 +0200 From: Peter van Dijk To: arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware Message-ID: <20000524171537.A10632@vuurwerk.nl> References: <200005241446.IAA05589@berserker.bsdi.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0.1i In-Reply-To: <200005241446.IAA05589@berserker.bsdi.com>; from cp@bsdi.com on Wed, May 24, 2000 at 08:46:03AM -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Wed, May 24, 2000 at 08:46:03AM -0600, Chuck Paterson wrote: [snip] > > > Once FreeBSD has a preemptive kernel FreeBSD will only run on > Pentium or better X86 processors. When would this be? Greetz, Peter. -- petervd@vuurwerk.nl - Peter van Dijk [student:developer:madly in love] To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 8:20:26 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id CAAF337BCCC for ; Wed, 24 May 2000 08:20:23 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@localhost [127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id JAA05951; Wed, 24 May 2000 09:19:06 -0600 (MDT) Message-Id: <200005241519.JAA05951@berserker.bsdi.com> To: Kazutaka YOKOTA Cc: arch@freebsd.org Subject: Re: Preemptive kernel on older X86 hardware From: Chuck Paterson Date: Wed, 24 May 2000 09:19:06 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG } }386 and 486 do not support multi-processor configuration, don't they? } }Kazu The 386 does not. The 486 does, but only with a bunch of external hardware. I don't believe doing this is on the table at all. Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 8:24:40 2000 Delivered-To: freebsd-arch@freebsd.org Received: from dan.emsphone.com (dan.emsphone.com [199.67.51.101]) by hub.freebsd.org (Postfix) with ESMTP id D52EE37BCDC for ; Wed, 24 May 2000 08:24:32 -0700 (PDT) (envelope-from dan@dan.emsphone.com) Received: (from dan@localhost) by dan.emsphone.com (8.9.3/8.9.3) id KAA14552; Wed, 24 May 2000 10:24:26 -0500 (CDT) (envelope-from dan) Date: Wed, 24 May 2000 10:24:26 -0500 From: Dan Nelson To: Chuck Paterson Cc: arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware Message-ID: <20000524102426.A8590@dan.emsphone.com> References: <200005241446.IAA05589@berserker.bsdi.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2i In-Reply-To: <200005241446.IAA05589@berserker.bsdi.com>; from "Chuck Paterson" on Wed May 24 08:46:03 GMT 2000 X-OS: FreeBSD 5.0-CURRENT Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG In the last episode (May 24), Chuck Paterson said: > The 386 processors do not support the locked exchange instruction. > For these systems the locked exchange can be replaced with roughly a > "cli, tst, cmp, branch, store, sti". > > The following are the obvious choices > > 1. Make two separate builds. Why not just make it conditional on the "cpu I386_CPU" line in the kernel Makefile? The comment already reads: # deleting the specification for CPUs you don't need to use may make # parts of the system run faster. This is especially true removing # I386_CPU. Distribute GENERIC just like usual, and let the user remove I386_CPU when he builds his own kerenl. > I have talked to a couple of people who think that supporting this > older stuff won't be important to FreeBSD by the time the kernel is > preemptive. Some even thought supporting the original Pentium > processors might not be required. I'll propose the following as it I wouldn't go that far, unless you're thinking 3-4 years in the future? There are a lot of overdrive and AMD CPUs out there that use the 486 instruction set. -- Dan Nelson dnelson@emsphone.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 8:28:27 2000 Delivered-To: freebsd-arch@freebsd.org Received: from Ilsa.StevesCafe.com (Ilsa.StevesCafe.com [206.168.13.65]) by hub.freebsd.org (Postfix) with ESMTP id C837237BCC3 for ; Wed, 24 May 2000 08:28:22 -0700 (PDT) (envelope-from fbsd@Ilsa.StevesCafe.com) Received: from Ilsa.StevesCafe.com (localhost [127.0.0.1]) by Ilsa.StevesCafe.com (8.9.3/8.9.3) with ESMTP id JAA23192; Wed, 24 May 2000 09:28:16 -0600 (MDT) (envelope-from fbsd@Ilsa.StevesCafe.com) Message-Id: <200005241528.JAA23192@Ilsa.StevesCafe.com> X-Mailer: exmh version 2.0.2 2/24/98 From: Steve Passe To: Chuck Paterson Cc: arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware In-reply-to: Your message of "Wed, 24 May 2000 08:46:03 MDT." <200005241446.IAA05589@berserker.bsdi.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Wed, 24 May 2000 09:28:16 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Hi, > The 386 processors do not support the locked exchange instruction. > For these systems the locked exchange can be replaced with roughly > a "cli, tst, cmp, branch, store, sti". > > The following are the obvious choices ... > 3) Drop support for 386. We could live with this, but I suspect there are others depending on 386EX type solutions. > Neither the 386 nor the 486 have a processor priority register or > cycle counters. Currently the BSD/OS SMPng kernel requires both of ... > The argument, within BSDI, for supporting the older stuff is that > new embedded systems are being built with these processors. Raw > chip cost being the reason. While I believe the person telling me > this, I haven't personally seen the evidence, I haven't looked > either. The majority of systems we ship today use 486 class CPUs. The real work gets done on pentium (ie 586) class CPUs, but for each of these there are up to 5-6 "distribution" systems that use a FreeBSD/486 system to monitor system health, etc. The 486 boards are plentiful (and cheaper) in the "embedded" world. > I have talked to a couple of people who think that supporting this > older stuff won't be important to FreeBSD by the time the kernel > is preemptive. Some even thought supporting the original Pentium > processors might not be required. I'll propose the following as it > reduces the work required: > > > Once FreeBSD has a preemptive kernel FreeBSD will only run on > Pentium or better X86 processors. > > > I'm not wed to the above at all, it is more to get discussion > started than anything else. Then again, if no one complains, it is > easier. We would have no need for SMP on 486 or 586, however losing UP on 486 would be a problem. -- Steve Passe | powered by smp@timing.com | Symmetric MultiProcessor FreeBSD To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 8:29: 8 2000 Delivered-To: freebsd-arch@freebsd.org Received: from peach.ocn.ne.jp (peach.ocn.ne.jp [210.145.254.87]) by hub.freebsd.org (Postfix) with ESMTP id 5006837BCB6 for ; Wed, 24 May 2000 08:29:06 -0700 (PDT) (envelope-from dcs@newsguy.com) Received: from newsguy.com (p28-dn01kiryunisiki.gunma.ocn.ne.jp [211.0.245.29]) by peach.ocn.ne.jp (8.9.1a/OCN/) with ESMTP id AAA06520; Thu, 25 May 2000 00:28:54 +0900 (JST) Message-ID: <392BF518.F8170D0E@newsguy.com> Date: Thu, 25 May 2000 00:28:24 +0900 From: "Daniel C. Sobral" X-Mailer: Mozilla 4.7 [en] (Win98; I) X-Accept-Language: en,pt-BR,ja MIME-Version: 1.0 To: Chuck Paterson Cc: arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware References: <200005241446.IAA05589@berserker.bsdi.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Chuck Paterson wrote: > > I have talked to a couple of people who think that supporting this > older stuff won't be important to FreeBSD by the time the kernel > is preemptive. Some even thought supporting the original Pentium > processors might not be required. I'll propose the following as it > reduces the work required: > > Once FreeBSD has a preemptive kernel FreeBSD will only run on > Pentium or better X86 processors. We had a lot of trouble already making people give up on their MFM/RLL controllers! Personally, I think this is the way to go, but not before 6.0. If 5.0 doesn't work on 486's, there'll be hell to pay. -- Daniel C. Sobral (8-DCS) dcs@newsguy.com dcs@freebsd.org capo@another.bsdconspiracy.org "Sentience hurts." To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 8:32:31 2000 Delivered-To: freebsd-arch@freebsd.org Received: from haldjas.folklore.ee (Haldjas.folklore.ee [193.40.6.121]) by hub.freebsd.org (Postfix) with ESMTP id 63E8137BCC3 for ; Wed, 24 May 2000 08:32:23 -0700 (PDT) (envelope-from narvi@haldjas.folklore.ee) Received: from localhost (narvi@localhost) by haldjas.folklore.ee (8.9.3/8.9.3) with SMTP id RAA33569; Wed, 24 May 2000 17:30:35 +0200 (EET) (envelope-from narvi@haldjas.folklore.ee) Date: Wed, 24 May 2000 17:30:35 +0200 (EET) From: Narvi To: Kazutaka YOKOTA Cc: Chuck Paterson , arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware In-Reply-To: <200005241519.AAA20525@zodiac.mech.utsunomiya-u.ac.jp> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Thu, 25 May 2000, Kazutaka YOKOTA wrote: > [...] > >Neither the 386 nor the 486 have a processor priority register or > >cycle counters. Currently the BSD/OS SMPng kernel requires both of > >these. There already exists some left over code to deal with not > >having a cycle counter. Doing a casual inspection there really > >doesn't seem to be anything too ugly in making the system run > >without these when there is only a single processor. > > 386 and 486 do not support multi-processor configuration, don't they? > They do in principle, but the fact can be overlooked. But loosing the ability to run on x86, x<5 is catastrophic. > Kazu > Sander sander@haldjas.folklore.ee To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 8:50:22 2000 Delivered-To: freebsd-arch@freebsd.org Received: from point.osg.gov.bc.ca (point.osg.gov.bc.ca [142.32.102.44]) by hub.freebsd.org (Postfix) with ESMTP id D3E8837BCB7 for ; Wed, 24 May 2000 08:50:14 -0700 (PDT) (envelope-from Cy.Schubert@uumail.gov.bc.ca) Received: (from daemon@localhost) by point.osg.gov.bc.ca (8.8.7/8.8.8) id IAA25771; Wed, 24 May 2000 08:50:00 -0700 Received: from passer.osg.gov.bc.ca(142.32.110.29) via SMTP by point.osg.gov.bc.ca, id smtpda25767; Wed May 24 08:49:47 2000 Received: (from uucp@localhost) by passer.osg.gov.bc.ca (8.9.3/8.9.1) id IAA09790; Wed, 24 May 2000 08:49:47 -0700 (PDT) Received: from cwsys9.cwsent.com(10.2.2.1), claiming to be "cwsys.cwsent.com" via SMTP by passer9.cwsent.com, id smtpdqG9788; Wed May 24 08:48:55 2000 Received: (from uucp@localhost) by cwsys.cwsent.com (8.10.1/8.9.1) id e4OFmtY22468; Wed, 24 May 2000 08:48:55 -0700 (PDT) Message-Id: <200005241548.e4OFmtY22468@cwsys.cwsent.com> Received: from localhost.cwsent.com(127.0.0.1), claiming to be "cwsys" via SMTP by localhost.cwsent.com, id smtpdl22464; Wed May 24 08:48:00 2000 X-Mailer: exmh version 2.1.1 10/15/1999 Reply-To: Cy Schubert - ITSD Open Systems Group From: Cy Schubert - ITSD Open Systems Group X-OS: FreeBSD 4.0-STABLE X-Sender: cy To: Dan Nelson Cc: Chuck Paterson , arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware In-reply-to: Your message of "Wed, 24 May 2000 10:24:26 CDT." <20000524102426.A8590@dan.emsphone.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Wed, 24 May 2000 08:48:00 -0700 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG In message <20000524102426.A8590@dan.emsphone.com>, Dan Nelson writes: > In the last episode (May 24), Chuck Paterson said: > > The 386 processors do not support the locked exchange instruction. > > For these systems the locked exchange can be replaced with roughly a > > "cli, tst, cmp, branch, store, sti". > > > > The following are the obvious choices > > > > 1. Make two separate builds. > > Why not just make it conditional on the "cpu I386_CPU" line in the > kernel Makefile? The comment already reads: > > # deleting the specification for CPUs you don't need to use may make > # parts of the system run faster. This is especially true removing > # I386_CPU. > > Distribute GENERIC just like usual, and let the user remove I386_CPU > when he builds his own kerenl. ... and have the user select options to build a custom kernel just after install, the same way Tru64-UNIX does. > > > I have talked to a couple of people who think that supporting this > > older stuff won't be important to FreeBSD by the time the kernel is > > preemptive. Some even thought supporting the original Pentium > > processors might not be required. I'll propose the following as it > > I wouldn't go that far, unless you're thinking 3-4 years in the future? > There are a lot of overdrive and AMD CPUs out there that use the 486 > instruction set. 486's make good cheap firewalls. You'll probably see them in use for that and other roles for a lot longer than you think. Regards, Phone: (250)387-8437 Cy Schubert Fax: (250)387-5766 Team Leader, Sun/DEC Team Internet: Cy.Schubert@osg.gov.bc.ca Open Systems Group, ITSD, ISTA Province of BC To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 9:52:15 2000 Delivered-To: freebsd-arch@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id E77B137BCF8 for ; Wed, 24 May 2000 09:52:12 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.9.3/8.9.1) id JAA75843; Wed, 24 May 2000 09:52:04 -0700 (PDT) (envelope-from dillon) Date: Wed, 24 May 2000 09:52:04 -0700 (PDT) From: Matthew Dillon Message-Id: <200005241652.JAA75843@apollo.backplane.com> To: "Daniel C. Sobral" Cc: Chuck Paterson , arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware References: <200005241446.IAA05589@berserker.bsdi.com> <392BF518.F8170D0E@newsguy.com> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG :> processors might not be required. I'll propose the following as it :> reduces the work required: :> :> Once FreeBSD has a preemptive kernel FreeBSD will only run on :> Pentium or better X86 processors. : :We had a lot of trouble already making people give up on their MFM/RLL :controllers! : :Personally, I think this is the way to go, but not before 6.0. If 5.0 :doesn't work on 486's, there'll be hell to pay. : :-- :Daniel C. Sobral (8-DCS) Woa everyone, before this blows up into an unnecessary flamefest... Nobody is advocating scrapping the 386 for UP kernels. Even when/if the kernel becomes threaded, it still probably will not be preemptive but even if it were we are not going to be throwing cmpexg instructions in mainline *UP* code. If some of the imported code happens to do that, it's trivial to fix with #ifdef's for 386/486 support. For SMP kernels I don't think it's worth supporting either the 386 or the 486. -Matt To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 10:28:50 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id D988137B743 for ; Wed, 24 May 2000 10:28:47 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@localhost [127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id LAA07094; Wed, 24 May 2000 11:28:39 -0600 (MDT) Message-Id: <200005241728.LAA07094@berserker.bsdi.com> To: Matthew Dillon Cc: "Daniel C. Sobral" , arch@freebsd.org Subject: Re: Preemptive kernel on older X86 hardware From: Chuck Paterson Date: Wed, 24 May 2000 11:28:39 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG } Nobody is advocating scrapping the 386 for UP kernels. Even when/if } the kernel becomes threaded, it still probably will not be preemptive } but even if it were we are not going to be throwing cmpexg instructions } in mainline *UP* code. If some of the imported code happens to do that, } it's trivial to fix with #ifdef's for 386/486 support. This is actually not true (the nobody part). I'm afraid I have no idea how to do a preemptive kernel where it doesn't diverge so greatly that it won't be effectively a totally separate kernel. I think if FreeBSD is not willing to bite the bullet on going to the preemptive kernel then it just shouldn't get done. Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 10:40:29 2000 Delivered-To: freebsd-arch@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id 3A00637B743 for ; Wed, 24 May 2000 10:40:26 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.9.3/8.9.1) id KAA76080; Wed, 24 May 2000 10:40:21 -0700 (PDT) (envelope-from dillon) Date: Wed, 24 May 2000 10:40:21 -0700 (PDT) From: Matthew Dillon Message-Id: <200005241740.KAA76080@apollo.backplane.com> To: Chuck Paterson Cc: "Daniel C. Sobral" , arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware References: <200005241728.LAA07094@berserker.bsdi.com> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG : : :} Nobody is advocating scrapping the 386 for UP kernels. Even when/if :} the kernel becomes threaded, it still probably will not be preemptive :} but even if it were we are not going to be throwing cmpexg instructions :} in mainline *UP* code. If some of the imported code happens to do that, :} it's trivial to fix with #ifdef's for 386/486 support. : :This is actually not true (the nobody part). I'm afraid I have no :idea how to do a preemptive kernel where it doesn't diverge so :greatly that it won't be effectively a totally separate kernel. I :think if FreeBSD is not willing to bite the bullet on going to the :preemptive kernel then it just shouldn't get done. : :Chuck I would not characterize this as 'biting the bullet'. Having a preemptive kernel is unlikely to improve performance. The only reason there might be preemption at all is to deal wth interrupts. Interrupts currently preempt supervisor code. If interrupts are moved to interrupt threads then interrupt threads would need to be able to preempt supervisor code. In this fashion the supervisor thread would be preempted, but that is very different from having supervisor threads preempt other supervisor threads (something we probably will not do). The issues involved with kernel thread preemption are very similar to the issues involved with parallel kernel operation on an SMP system (something which we currently do not do for the most part due to the big giant lock). As pieces of the kernel become MP safe it will be possible to run more and more supervisor code in parallel on multiple cpu's. When that occurs the same locking used to protect those threads from each other will also make it possible to preempt kernel threads. Even with the capability, nobody I know who actually designs this stuff would ever contemplate actually *preempting* a kernel thread with another one (for non-interrupt threads, that is) -- there isn't a single benefit that would be gained from having the capability. The assembly used to implement the MP locks is irrelevant. We are not going to be strewing cmpexg instructions all over the code. If they exist anywhere itl will be in one or two low level assembly routines (such as i386/i386/mplock.s), and thus relatively trivial to conditionally compile based on the cpu (for either the UP or the SMP model). -Matt Matthew Dillon To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 10:54:42 2000 Delivered-To: freebsd-arch@freebsd.org Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by hub.freebsd.org (Postfix) with ESMTP id 8D07337BA04 for ; Wed, 24 May 2000 10:54:38 -0700 (PDT) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.9.3/8.9.3) with ESMTP id NAA20309; Wed, 24 May 2000 13:54:34 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.9.3/8.9.1) id NAA22048; Wed, 24 May 2000 13:54:04 -0400 (EDT) (envelope-from gallatin@cs.duke.edu) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Date: Wed, 24 May 2000 13:54:03 -0400 (EDT) To: Chuck Paterson Cc: arch@freebsd.org Subject: Re: Preemptive kernel on older X86 hardware In-Reply-To: <200005241446.IAA05589@berserker.bsdi.com> References: <200005241446.IAA05589@berserker.bsdi.com> X-Mailer: VM 6.43 under 20.4 "Emerald" XEmacs Lucid Message-ID: <14636.5048.656520.901095@grasshopper.cs.duke.edu> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Chuck Paterson writes: > > The 386 processors do not support the locked exchange instruction. > For these systems the locked exchange can be replaced with roughly > a "cli, tst, cmp, branch, store, sti". > > The following are the obvious choices > > 1. Make two separate builds. > > 2. Add both sets of code to all macros and do > run time checks. The penalties are: > > a) doing the check on each mutex operation. > b) bigger kernel > c) since both sets of code will be spread through > out the kernel next to each other it is > reasonable to expect that running either > set of code will be using more cache lines than > they would if they existed alone. > > To add a little perspective there are currently around > 1300 locking operation in the BSD/OS kernel that would > be effected by this. This is really a guess, but when > all is said and done I would say BSD/OS will end up with > between 1700 and 2500. Kernel size seems like a non > issue, the number of extra cache lines used also seems > likely to be a non issue. So it seems like it really comes > down to adding the extra test which will be significant > compared to the actual mutex operation. > Maybe we can have our cake & eat it too: Its my understanding that Tru64 (nee Digital UNIX, nee OSF/1 alpha) removes SMP locks for UP operation by having the first lock call actually be a call to a function which checks the number of CPUs and then patches the caller to remove the call to itself, and possibly patch out the SMP lock calls on a single processor system. I believe that Peter Wemm has brought this up in the past. Would it be possible for the mutex macros to have a call like this which queried the CPU version & patched in either the locked exchange instructions or a call to a special routine for 386/486 cpus? If this isn't possible on an x86, please don't laugh too hard. I really don't know much about low-level x86 details. Cheers, Drew ------------------------------------------------------------------------------ Andrew Gallatin, Sr Systems Programmer http://www.cs.duke.edu/~gallatin Duke University Email: gallatin@cs.duke.edu Department of Computer Science Phone: (919) 660-6590 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 11: 1: 1 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id 6938437BD4B for ; Wed, 24 May 2000 11:00:56 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@localhost [127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id MAA07427; Wed, 24 May 2000 12:00:49 -0600 (MDT) Message-Id: <200005241800.MAA07427@berserker.bsdi.com> To: Andrew Gallatin Cc: arch@freebsd.org Subject: Re: Preemptive kernel on older X86 hardware From: Chuck Paterson Date: Wed, 24 May 2000 12:00:49 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG } }If this isn't possible on an x86, please don't laugh too hard. I }really don't know much about low-level x86 details. } }Cheers, } }Drew It can be done. In BSD/OS and Solaris however the locks must be gotten because they are what protect from interrupt service. Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 11:17:26 2000 Delivered-To: freebsd-arch@freebsd.org Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by hub.freebsd.org (Postfix) with ESMTP id 6889B37B87F for ; Wed, 24 May 2000 11:17:23 -0700 (PDT) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.9.3/8.9.3) with ESMTP id OAA20777; Wed, 24 May 2000 14:17:23 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.9.3/8.9.1) id OAA22104; Wed, 24 May 2000 14:16:52 -0400 (EDT) (envelope-from gallatin@cs.duke.edu) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Date: Wed, 24 May 2000 14:16:52 -0400 (EDT) To: Chuck Paterson Cc: arch@freebsd.org Subject: Re: Preemptive kernel on older X86 hardware In-Reply-To: <200005241800.MAA07427@berserker.bsdi.com> References: <200005241800.MAA07427@berserker.bsdi.com> X-Mailer: VM 6.43 under 20.4 "Emerald" XEmacs Lucid Message-ID: <14636.7064.775632.285834@grasshopper.cs.duke.edu> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Chuck Paterson writes: > } > }If this isn't possible on an x86, please don't laugh too hard. I > }really don't know much about low-level x86 details. > } > }Cheers, > } > }Drew > > It can be done. In BSD/OS and Solaris however the locks must > be gotten because they are what protect from interrupt service. > Err... But I wasn't advocating removing the locks. I was suggesting having the routine patch in the correct locking code (or calls thereto) based on the processor type at the first invocation of each mutex. This way we could have a GENERIC kernel with the correct locking code for both 3/486 and modern x86 processors compiled in without the overhead of doing the processor type check on every call. Sorry if I wasn't clear.. Drew ------------------------------------------------------------------------------ Andrew Gallatin, Sr Systems Programmer http://www.cs.duke.edu/~gallatin Duke University Email: gallatin@cs.duke.edu Department of Computer Science Phone: (919) 660-6590 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 11:29:43 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id 3158C37B6A2 for ; Wed, 24 May 2000 11:29:36 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@localhost [127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id MAA07732; Wed, 24 May 2000 12:29:29 -0600 (MDT) Message-Id: <200005241829.MAA07732@berserker.bsdi.com> To: Andrew Gallatin Cc: arch@freebsd.org Subject: Re: Preemptive kernel on older X86 hardware From: Chuck Paterson Date: Wed, 24 May 2000 12:29:28 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG }Err... But I wasn't advocating removing the locks. I was suggesting }having the routine patch in the correct locking code (or calls }thereto) based on the processor type at the first invocation of each }mutex. This way we could have a GENERIC kernel with the correct }locking code for both 3/486 and modern x86 processors compiled in }without the overhead of doing the processor type check on every call. } }Sorry if I wasn't clear.. } }Drew I am on a conference call and was also thinking about other mail. You were probably clear. Yes, this can be done. We did consider this for BSD/OS and I forgot about it. It is actually a very legitimate way to solve the problem. Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 11:46:19 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id C632C37BD3C for ; Wed, 24 May 2000 11:46:15 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@localhost [127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id MAA07977; Wed, 24 May 2000 12:46:04 -0600 (MDT) Message-Id: <200005241846.MAA07977@berserker.bsdi.com> To: Dan Nelson Cc: arch@freebsd.org Subject: Re: Preemptive kernel on older X86 hardware From: Chuck Paterson Date: Wed, 24 May 2000 12:46:04 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG } }Why not just make it conditional on the "cpu I386_CPU" line in the }kernel Makefile? The comment already reads: } }# deleting the specification for CPUs you don't need to use may make }# parts of the system run faster. This is especially true removing }# I386_CPU. Technically this works just fine. I personally don't understand the implications of saying that a kernel build is necessary. I'm not saying it is bad, I just saying I don't understand. If people say this is acceptable then this is fine. Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 12:23:38 2000 Delivered-To: freebsd-arch@freebsd.org Received: from dan.emsphone.com (dan.emsphone.com [199.67.51.101]) by hub.freebsd.org (Postfix) with ESMTP id 2111837B760 for ; Wed, 24 May 2000 12:23:35 -0700 (PDT) (envelope-from dan@dan.emsphone.com) Received: (from dan@localhost) by dan.emsphone.com (8.9.3/8.9.3) id OAA18933; Wed, 24 May 2000 14:23:30 -0500 (CDT) (envelope-from dan) Date: Wed, 24 May 2000 14:23:30 -0500 From: Dan Nelson To: Chuck Paterson Cc: arch@freebsd.org Subject: Re: Preemptive kernel on older X86 hardware Message-ID: <20000524142329.B15434@dan.emsphone.com> References: <200005241846.MAA07977@berserker.bsdi.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2i In-Reply-To: <200005241846.MAA07977@berserker.bsdi.com>; from "Chuck Paterson" on Wed May 24 12:46:04 GMT 2000 X-OS: FreeBSD 5.0-CURRENT Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG In the last episode (May 24), Chuck Paterson said: > Dan Nelson said: > }Why not just make it conditional on the "cpu I386_CPU" line in the > }kernel Makefile? The comment already reads: > } > } deleting the specification for CPUs you don't need to use may make > } parts of the system run faster. This is especially true removing > } I386_CPU. > > Technically this works just fine. > > I personally don't understand the implications of saying that a > kernel build is necessary. I'm not saying it is bad, I just saying I > don't understand. If people say this is acceptable then this is fine. It's only necessary if an optimized SMP-aware kernel can't run on a 386 (due to use pentium-specific instructions) *and* a kernel that runs on a 386 is too slow to be usable on an SMP box (due to lack of fast pentium-specific instructions). The current FreeBSD build process requires you to do a recompile to enable SMP, so given that we need to recompile anyhow, it's not that big a deal. When we get to the point where a single kernel runs optimally on both SMP and UP hardware, then it will be an issue. BTW, FreeBSD does a little bit of per-processor runtime optimization in the kernel already; bcopy, bzero, and copyin/out get replaced with 586-specific functions when the kernel is booting, if a pentium is detected (see sys/i386/isa/npx.c). -- Dan Nelson dnelson@emsphone.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 12:32:56 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id 2E3D337BD3C for ; Wed, 24 May 2000 12:32:52 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@[127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id NAA08860; Wed, 24 May 2000 13:32:47 -0600 (MDT) Message-Id: <200005241932.NAA08860@berserker.bsdi.com> To: Dan Nelson Cc: arch@freebsd.org Subject: Re: Preemptive kernel on older X86 hardware From: Chuck Paterson Date: Wed, 24 May 2000 13:32:47 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG } }It's only necessary if an optimized SMP-aware kernel can't run on a 386 }(due to use pentium-specific instructions) *and* a kernel that runs on }a 386 is too slow to be usable on an SMP box (due to lack of fast }pentium-specific instructions). Well, "too" is a very interesting word.) Given the amount of tracing code BSD/OS has been able to leave in with trivial penalty I suspect that it won't be "too" slow. Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 12:44:54 2000 Delivered-To: freebsd-arch@freebsd.org Received: from zippy.cdrom.com (zippy.cdrom.com [204.216.27.228]) by hub.freebsd.org (Postfix) with ESMTP id 8661E37BD6C for ; Wed, 24 May 2000 12:44:50 -0700 (PDT) (envelope-from jkh@zippy.cdrom.com) Received: from localhost (jkh@localhost [127.0.0.1]) by zippy.cdrom.com (8.9.3/8.9.3) with ESMTP id MAA76156; Wed, 24 May 2000 12:46:43 -0700 (PDT) (envelope-from jkh@zippy.cdrom.com) To: Chuck Paterson Cc: Matthew Dillon , "Daniel C. Sobral" , arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware In-reply-to: Your message of "Wed, 24 May 2000 11:28:39 MDT." <200005241728.LAA07094@berserker.bsdi.com> Date: Wed, 24 May 2000 12:46:43 -0700 Message-ID: <76153.959197603@localhost> From: "Jordan K. Hubbard" Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > This is actually not true (the nobody part). I'm afraid I have no > idea how to do a preemptive kernel where it doesn't diverge so > greatly that it won't be effectively a totally separate kernel. I > think if FreeBSD is not willing to bite the bullet on going to the > preemptive kernel then it just shouldn't get done. Well, I think we can still have our cake and eat it too. I agree that the 386/486 markets need to keep being supported since FreeBSD frequently gets used at 3am "on the sly" by many techs who grab an old box out of the junk room closet and deploy some service on it before the boss notices. A 486 box still makes a fine low-speed router or mail server for a small to midsize company (or small to midsize group within a big company). It's also the case that the 386/486 parts have just recently passed through the DoD's long acceptance trial process and are now available in radiation hardened packaging for use in satellites and other hostile environments, so we don't want to turn our backs on that market just yet. This simply means that we will need to start generating another kernel for such situations, something we currently need to do for SMP systems in any case and should probably just add as an option in the installer. There's also your option #2 to consider, that being to take the performance hit for now and get past the point where the 386/486 CPUs remain relevant at all. - Jordan To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 12:50:30 2000 Delivered-To: freebsd-arch@freebsd.org Received: from zippy.cdrom.com (zippy.cdrom.com [204.216.27.228]) by hub.freebsd.org (Postfix) with ESMTP id BB7ED37BC72 for ; Wed, 24 May 2000 12:50:28 -0700 (PDT) (envelope-from jkh@zippy.cdrom.com) Received: from localhost (jkh@localhost [127.0.0.1]) by zippy.cdrom.com (8.9.3/8.9.3) with ESMTP id MAA84529; Wed, 24 May 2000 12:52:26 -0700 (PDT) (envelope-from jkh@zippy.cdrom.com) To: Chuck Paterson Cc: Andrew Gallatin , arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware In-reply-to: Your message of "Wed, 24 May 2000 12:29:28 MDT." <200005241829.MAA07732@berserker.bsdi.com> Date: Wed, 24 May 2000 12:52:26 -0700 Message-ID: <84526.959197946@localhost> From: "Jordan K. Hubbard" Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > Yes, this can be done. We did consider this for > BSD/OS and I forgot about it. It is actually a very legitimate > way to solve the problem. I like this idea too. :) - Jordan To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 13:42:36 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id 8E7EC37BD39 for ; Wed, 24 May 2000 13:42:23 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@[127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id OAA09548; Wed, 24 May 2000 14:42:14 -0600 (MDT) Message-Id: <200005242042.OAA09548@berserker.bsdi.com> To: Matthew Dillon Cc: arch@freebsd.org Subject: Re: Preemptive kernel on older X86 hardware From: Chuck Paterson Date: Wed, 24 May 2000 14:42:14 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG At first I thought we fundamentally disagreed and now reading more, well I'm not sure. Sorry if I'm repeat that which is totally obvious. The first thing is that in a fine grained implementation threads really aren't protected from preemption by a lock. All the lock does is protect the thread from other processes trying to look at the same data. Doing mtx_enter() has no effect on blocking interrupts. This is only true for non spin locks, which really are the only thing used in "generic" kernel code. Not all of the mutex operations are added cost. They are at least partially offset by the cost of removing SPLS. Particularly if the SPLS turn into function calls. If a thread is running and is preempted by an interrupt thread such that thread may end up being put on a run queue. This typically does not happen, but can happen. If this happens it may not be the same thread that ends up running next. This results in a top half thread effectively being preempted by another top half thread. This all works just find in either the single or multi processor environment as long as all the locks stay in place. } I would not characterize this as 'biting the bullet'. Having a } preemptive kernel is unlikely to improve performance. Accept that better attention can be paid to scheduling priorities. But, yes in general this is true. } The only reason there might be preemption at all is to deal with } interrupts. Interrupts currently preempt supervisor code. If } interrupts are moved to interrupt threads then interrupt threads would } need to be able to preempt supervisor code. In this fashion the } supervisor thread would be preempted, but that is very different } from having supervisor threads preempt other supervisor threads } (something we probably will not do). If you don't change to the highest priority task on return from an interrupt this is true. Even if you don't normally do this, BSD/OS currently does not, it can happen in pathological cases where the running thread when the interrupt occurred get put a way on a run queue. the thread got suspended } The issues involved with kernel thread preemption are very similar } to the issues involved with parallel kernel operation on an SMP } system (something which we currently do not do for the most part } due to the big giant lock). As pieces of the kernel become MP safe } it will be possible to run more and more supervisor code in parallel } on multiple cpu's. } When that occurs the same locking used to protect those threads from } each other will also make it possible to preempt kernel threads. } Even with the capability, nobody I know who actually designs this stuff } would ever contemplate actually *preempting* a kernel thread with } another one (for non-interrupt threads, that is) -- there isn't a single } benefit that would be gained from having the capability. Yes the same locking used to protect threads from each other, or more correctly the data the thread may be accessing, protects from preemption. Actually it doesn't protect from preemption, it protects from it mattering. The point being is that the locking and not left out as I thought you where saying. Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 14:10:44 2000 Delivered-To: freebsd-arch@freebsd.org Received: from postal.linkfast.net (postal.linkfast.net [208.160.105.2]) by hub.freebsd.org (Postfix) with ESMTP id 706BD37BAAD for ; Wed, 24 May 2000 14:10:41 -0700 (PDT) (envelope-from fullermd@linkfast.net) Received: by postal.linkfast.net (Postfix, from userid 100) id 711B49B06; Wed, 24 May 2000 16:10:40 -0500 (CDT) Date: Wed, 24 May 2000 16:10:40 -0500 From: Matthew Fuller To: Steve Passe Cc: Chuck Paterson , arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware Message-ID: <20000524161040.Z660@linkfast.net> References: <200005241446.IAA05589@berserker.bsdi.com> <200005241528.JAA23192@Ilsa.StevesCafe.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0i In-Reply-To: <200005241528.JAA23192@Ilsa.StevesCafe.com>; from smp@timing.com on Wed, May 24, 2000 at 09:28:16AM -0600 X-OS: FreeBSD Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Wed, May 24, 2000 at 09:28:16AM -0600, a little birdie told me that Steve Passe remarked > > We would have no need for SMP on 486 or 586, however losing UP on 486 > would be a problem. Just as a side point (a fair bit of this discussion is over my head, so I'm not sure if this is really relevant or not, but...), I would dispute the second half of the first part above. While SMP on 586 is certainly not mainstream, I'm sure I'm far from the only one with a multi-proc 586 machine. It'd be quite a shame to abandon that market without at least more looking into the problems that would be faced in supporting it. -- Matthew Fuller (MF4839) | fullermd@over-yonder.net Unix Systems Administrator | fullermd@linkfast.net Specializing in FreeBSD | http://www.over-yonder.net/ "The only reason I'm burning my candle at both ends, is because I haven't figured out how to light the middle yet" To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 14:36:53 2000 Delivered-To: freebsd-arch@freebsd.org Received: from timingpdc.timing.com (timingpdc.timing.com [206.168.13.194]) by hub.freebsd.org (Postfix) with ESMTP id 5466237B88C for ; Wed, 24 May 2000 14:36:39 -0700 (PDT) (envelope-from smp@timing.com) Received: from RoadRunner.timing.com ([206.168.13.190]) by timingpdc.timing.com (Post.Office MTA v3.1.2 release (PO205-101c) ID# 103-49575U100L2S100) with ESMTP id AAA516; Wed, 24 May 2000 15:38:06 -0600 Received: from RoadRunner.timing.com (localhost [127.0.0.1]) by RoadRunner.timing.com (8.9.3/8.9.3) with ESMTP id PAA21433; Wed, 24 May 2000 15:36:37 -0600 (MDT) (envelope-from smp@RoadRunner.timing.com) Message-Id: <200005242136.PAA21433@RoadRunner.timing.com> X-Mailer: exmh version 2.0.2 2/24/98 From: Steve Passe To: Matthew Fuller Cc: Chuck Paterson , arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware In-reply-to: Your message of "Wed, 24 May 2000 16:10:40 CDT." <20000524161040.Z660@linkfast.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Wed, 24 May 2000 15:36:37 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Hi, > On Wed, May 24, 2000 at 09:28:16AM -0600, a little birdie told me > that Steve Passe remarked > > > > We would have no need for SMP on 486 or 586, however losing UP on 486 > > would be a problem. > > Just as a side point (a fair bit of this discussion is over my head, so > I'm not sure if this is really relevant or not, but...), I would dispute > the second half of the first part above. > > While SMP on 586 is certainly not mainstream, I'm sure I'm far from the > only one with a multi-proc 586 machine. It'd be quite a shame to abandon > that market without at least more looking into the problems that would be > faced in supporting it. I should have been clearer, when I said "we" in my last posting, I meant "Timing Solutions", NOT FreeBSD. That's the trouble with wearing multiple hats! -- Steve Passe | powered by smp@timing.com | Symmetric MultiProcessor FreeBSD To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 15:56: 7 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id DA3C637B636 for ; Wed, 24 May 2000 15:56:04 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@[127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id QAA14414 for ; Wed, 24 May 2000 16:56:04 -0600 (MDT) Message-Id: <200005242256.QAA14414@berserker.bsdi.com> To: arch@freebsd.org Subject: Re: Preemptive kernel on older X86 hardware From: Chuck Paterson Date: Wed, 24 May 2000 16:56:03 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG I should point out that what is being proposed here is a totally different model for kernel protection. SPLS go away totally and are replaced by locks, almost always mutexs. Mutexs protect data, not a thread of control. Saying that a SPL is replaced by a mutex is still kind of misleading because the model is so different. I would like to claim that this change is totally performance neutral, but I don't know that for sure, and never will until the work is done. It is however very close to neutral. I can say that a kernel build that takes approximately 30 minutes on an admittedly slow uniprocessor took about 10 seconds less with the SMPng kernel. A build is obviously heavy user time. Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 16: 4:52 2000 Delivered-To: freebsd-arch@freebsd.org Received: from mail.rz.fh-wilhelmshaven.de (mail.rz.fh-wilhelmshaven.de [139.13.25.134]) by hub.freebsd.org (Postfix) with ESMTP id 914A337B975 for ; Wed, 24 May 2000 16:04:48 -0700 (PDT) (envelope-from ohoyer@fbwi.fh-wilhelmshaven.de) Received: from fettesau.stuwo.fh-wilhelmshaven.de (stuwopc5.stuwo.fh-wilhelmshaven.de [139.13.209.5]) by mail.rz.fh-wilhelmshaven.de (8.9.3/8.9.3) with SMTP id BAA02007; Thu, 25 May 2000 01:04:37 +0200 (MET DST) Message-Id: <4.1.20000525004708.027035d0@mail.rz.fh-wilhelmshaven.de> X-Sender: ohoyer@mail.rz.fh-wilhelmshaven.de X-Mailer: QUALCOMM Windows Eudora Pro Version 4.1 Date: Thu, 25 May 2000 01:04:24 +0200 To: Chuck Paterson From: Olaf Hoyer Subject: Re: Preemptive kernel on older X86 hardware Cc: freebsd-arch@freebsd.org In-Reply-To: <200005241846.MAA07977@berserker.bsdi.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG At 12:46 24.05.00 -0600, you wrote: >} >}Why not just make it conditional on the "cpu I386_CPU" line in the >}kernel Makefile? The comment already reads: >} >}# deleting the specification for CPUs you don't need to use may make >}# parts of the system run faster. This is especially true removing >}# I386_CPU. > >Technically this works just fine. > >I personally don't understand the implications of saying that a >kernel build is necessary. I'm not saying it is bad, I just saying >I don't understand. If people say this is acceptable then this > Hi! Well, it is (as far I understood it in the past being a youngster in the UNIX world) the standard way/clasisc way of installin an UNIX system. It has the benefits of simply grabbing some installation media, dumping all the code onto the target system, and expect it to run. Very essential in all those situations when you need to "just need to check/try out something on the fly". A generic kernel supports at least enough devices to get the system into usable shape. In a more recent box, where you have some special/additional hardware, you always will compile your own kernel afterwards, especially if you have some SMP machine. (Yes, have my dual iP233MMX (socket7) running happily) This procedure has following advantages: a)It will run on all systems, so that you save time and hassle. b) It is the standard way of doing it, the people are used to it. So why fix it when it ain't broke... Point is, the more errors may occur during the install process, and be it the need to make something fit for older machines, some errors will occur, because you have to think a bit more or pay attention to other things than those you do daily/at least on every install. Its a possibility less to make faulty decisions. Also, for support of 386/486 : Small Unix -based computer systems like BSD, Linux and some derived works (like PicoBSD similar Linux projects) may be used as a router, little firewall, mail server etc. So it also goes about recycling/extending the lifespan of hardware that otherwise would be thrown into the junkyard. This also is some cost factor, especially in commercial environment. So an enterprise is able to lower its overall cost for new hardware by simply using it appropriately with some powerful OS behind it. The embedded market is also a very big factor. I heard some figures about numbers of CPUs going into the embedded market;those outsell the CPUs found in PCs easily. *BSD is one of the favoured OS used therein, also due to the BSD License. And, as previously stated, when you do some critical design, where whether money is a factor, or even size or power consumption, then a 386 or 486 is a favourite about a pentium class CPU. Also, when the computing power of a 386 will be enough for a given system with a given, not rising workload, why spend extra money for a 486 class CPU? (I'm also thinking of those PC104 systems...) Regards Olaf Hoyer -------- Olaf Hoyer www.nightfire.de mailto:Olaf.Hoyer@nightfire.de FreeBSD- Turning PC's into workstations ICQ:22838075 Liebe und Hass sind nicht blind, aber geblendet vom Feuer, dass sie selber mit sich tragen. (Nietzsche) To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 18: 3:44 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id 62BF837B9B1 for ; Wed, 24 May 2000 18:03:40 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@[127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id TAA15470 for ; Wed, 24 May 2000 19:03:38 -0600 (MDT) Message-Id: <200005250103.TAA15470@berserker.bsdi.com> To: arch@freebsd.org Subject: Short summary From: Chuck Paterson Date: Wed, 24 May 2000 19:03:38 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG What is being proposed here is a major kernel architectural change. The locking provided by SPLS in the traditional BSD kernel goes away and is replaced by mutexs. This model is very different. Interrupts do not in general get blocked. When interrupt service code needs access to some piece of data which is actively being modified by what is traditionally thought of as the "top half" the interrupt thread will block at that point. The finer grained the locking the less chance of a collision, but the number of locking operation goes up and a price has to be paid. BSDI has been shipping only a MP capable kernel since 98. This kernel has a very few number of locks, for the sake of discussion it is a Giant lock kernel. This kernel ran as fast as the previous non-MP capable kernel in a uniprocessor environment. Trying to take the locking any finer would have significantly impacted uniprocessor performance. The architecture being proposed here is the same as the BSD/OS SMPng kernel. An MP capable version of this kernel runs virtually the same speed as the Giant lock BSD/OS kernel in a uniprocessor environment. It occurred to me today that in a uniprocessor environment the lock prefix to the cmpxchg can be removed. I ran some experiments. The following data is from a very limited sample size. On a couple of different systems with different clock rates removing the lock prefix reduced execution time of mutex operations to one third of their original value. Running the same job with two kernels whose only difference was the lock prefix there was a reduction in system time of 2.5 percent. This suggested that the total system time used for locking with the SMP locks in place is 3.6 percent and with the the locks trimmed for uniprocessor only operation is 1.2 percent. (Please excuse rounding errors). It appears that there is a desire to retain both 386 and 486 processors, and perhaps a uniprocessor only version of Pentium or better. There are several acceptable ways to achieve this so it doesn't seem to be a big deal, except for the small matter of doing the work. Both the uniprocessor and multiprocessor versions run under the new architecture, the old SPL code will (is it will or would for a proposal?) be totally gone. Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 18:21:25 2000 Delivered-To: freebsd-arch@freebsd.org Received: from smtp04.primenet.com (smtp04.primenet.com [206.165.6.134]) by hub.freebsd.org (Postfix) with ESMTP id EF40137B6F8 for ; Wed, 24 May 2000 18:21:17 -0700 (PDT) (envelope-from tlambert@usr05.primenet.com) Received: (from daemon@localhost) by smtp04.primenet.com (8.9.3/8.9.3) id SAA25278; Wed, 24 May 2000 18:21:07 -0700 (MST) Received: from usr05.primenet.com(206.165.6.205) via SMTP by smtp04.primenet.com, id smtpdAAAHoaywX; Wed May 24 18:21:04 2000 Received: (from tlambert@localhost) by usr05.primenet.com (8.8.5/8.8.5) id SAA11655; Wed, 24 May 2000 18:21:10 -0700 (MST) From: Terry Lambert Message-Id: <200005250121.SAA11655@usr05.primenet.com> Subject: Re: Preemptive kernel on older X86 hardware To: cp@bsdi.com (Chuck Paterson) Date: Thu, 25 May 2000 01:21:10 +0000 (GMT) Cc: arch@FreeBSD.ORG In-Reply-To: <200005241446.IAA05589@berserker.bsdi.com> from "Chuck Paterson" at May 24, 2000 08:46:03 AM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > The 386 processors do not support the locked exchange instruction. > For these systems the locked exchange can be replaced with roughly > a "cli, tst, cmp, branch, store, sti". Jeffrey Hsu recently commented on this to me in regards to a paper by Michael Greenwald, in regard to having lock-free synchronized lists in the kernel. > The following are the obvious choices > > 1. Make two separate builds. This would be my choice: using a CPU type test in the loader in order to pick which kernel to load. Ideally, the kernel would be modular enough that this might degrade to the "load a different module as a replacement for a generic module" case. This would even work in the swap path, if it was adequately staged. Nice to get back to the "generic console driver" precepts from 1993... > Neither the 386 nor the 486 have a processor priority register or > cycle counters. Currently the BSD/OS SMPng kernel requires both of > these. There already exists some left over code to deal with not > having a cycle counter. Doing a casual inspection there really > doesn't seem to be anything too ugly in making the system run > without these when there is only a single processor. It would be interesting to know why the cycle counter was required, and how BSDI would deal with this on SMP Alpha and SPARC machines, or the new IBM 64 PPC processor NUMA machines... > I really don't know what BSDI will finally decided. I am confident > that we will not have run time checks in the locking operations. God forbid! > The argument, within BSDI, for supporting the older stuff is that > new embedded systems are being built with these processors. Raw > chip cost being the reason. While I believe the person telling me > this, I haven't personally seen the evidence, I haven't looked > either. You can buy external APICs for 486 based systems to achieve 486 SMP. This is where Intel SMP came from in the first place. The APICs went into the processor proper more to thwart third party processor manufacturers from creating SMP boxes using Intel APIC chips but not CPU chips, than anything else. My main rationale might be that there are a number of MEI coherency model SMP machines available, and likely to be built. Given the pricing on the PPC 603 chips, I think that a 603 PPC box based on the BeBox trick of replacing the L2 cache logic with discrete MEI logic, or even an external custom ASIC MMU that can use an L2 cache but still uses an MEI coherency modely would be likely. I can think of some pretty trivial embeded boxes that could be built on an ASIC, and would use multiple PPCs (maybe one for MPEG2 encoding?) based on the IBM "Blue Logic" PPC core component library. I think it is a bad idea to rule out PDAs and set-top boxes, entirely. > I have talked to a couple of people who think that supporting this > older stuff won't be important to FreeBSD by the time the kernel > is preemptive. This is a more important point. PEOPLE, THIS IS NOT STRICTLY AN SMP ISSUE! > Some even thought supporting the original Pentium processors might > not be required. This is just insane. As new processor migrate up Moore's law, the old processor cores will fall into embedded systems. IBM's recent announcement regarding SOI and CuSOI technologies should make people think twice about whether or not it's possible to upgrade old chips merely through a different set of fabrication technologies. > I'll propose the following as it reduces the work required: > > Once FreeBSD has a preemptive kernel FreeBSD will only run on > Pentium or better X86 processors. I think the work required to build two kernels instead of one, and then CPU-testing in the loader to pick one, is really trivial. I think there are better approaches to the problem than this, but this is enough to throw out that idea entirely. Supporting loading of more segments, with segment attributes, is an obvious approach; Microsoft does this with VxD's and other NT and Windows kernel components, today. The only thing you have to worry about is being able to serialize access to the section being replaced (only during the replacement process) and not be in it when you flip the page table entry. You could even do defragmentation of the kernel address space, if you thought is was more useful than running on the new code and copying the new code to the old location, and then running there instead. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 18:25:39 2000 Delivered-To: freebsd-arch@freebsd.org Received: from smtp05.primenet.com (smtp05.primenet.com [206.165.6.135]) by hub.freebsd.org (Postfix) with ESMTP id CF66D37B583 for ; Wed, 24 May 2000 18:25:34 -0700 (PDT) (envelope-from tlambert@usr05.primenet.com) Received: (from daemon@localhost) by smtp05.primenet.com (8.9.3/8.9.3) id SAA14660; Wed, 24 May 2000 18:25:33 -0700 (MST) Received: from usr05.primenet.com(206.165.6.205) via SMTP by smtp05.primenet.com, id smtpdAAAeHaGMC; Wed May 24 18:25:24 2000 Received: (from tlambert@localhost) by usr05.primenet.com (8.8.5/8.8.5) id SAA11774; Wed, 24 May 2000 18:25:21 -0700 (MST) From: Terry Lambert Message-Id: <200005250125.SAA11774@usr05.primenet.com> Subject: Re: Preemptive kernel on older X86 hardware To: cp@bsdi.com (Chuck Paterson) Date: Thu, 25 May 2000 01:25:20 +0000 (GMT) Cc: gallatin@cs.duke.edu (Andrew Gallatin), arch@FreeBSD.ORG In-Reply-To: <200005241800.MAA07427@berserker.bsdi.com> from "Chuck Paterson" at May 24, 2000 12:00:49 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > }If this isn't possible on an x86, please don't laugh too hard. I > }really don't know much about low-level x86 details. > } > }Cheers, > } > }Drew > > It can be done. In BSD/OS and Solaris however the locks must > be gotten because they are what protect from interrupt service. NT and Windows do this today for VxD jump tables, when one VxD uses the entry points of another on which it depends. Go out to the Microsoft developer site, and look for documentation on the function "peloader" in any Windows 9x or higher DDK. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 18:25:48 2000 Delivered-To: freebsd-arch@freebsd.org Received: from mass.cdrom.com (mass.cdrom.com [204.216.28.234]) by hub.freebsd.org (Postfix) with ESMTP id 92ED937B583 for ; Wed, 24 May 2000 18:25:46 -0700 (PDT) (envelope-from msmith@mass.cdrom.com) Received: from mass.cdrom.com (localhost [127.0.0.1]) by mass.cdrom.com (8.9.3/8.9.3) with ESMTP id SAA03314; Wed, 24 May 2000 18:27:30 -0700 (PDT) (envelope-from msmith@mass.cdrom.com) Message-Id: <200005250127.SAA03314@mass.cdrom.com> X-Mailer: exmh version 2.1.1 10/15/1999 To: Terry Lambert Cc: cp@bsdi.com (Chuck Paterson), arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware In-reply-to: Your message of "Thu, 25 May 2000 01:21:10 -0000." <200005250121.SAA11655@usr05.primenet.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Wed, 24 May 2000 18:27:30 -0700 From: Mike Smith Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > I think the work required to build two kernels instead of one, and > then CPU-testing in the loader to pick one, is really trivial. I > think there are better approaches to the problem than this, but > this is enough to throw out that idea entirely. Are the mutexes inlined, or are they all function calls? If the latter, loading lock_smp.ko vs. lock_std.ko would be pretty trivial... 8) -- \\ Give a man a fish, and you feed him for a day. \\ Mike Smith \\ Tell him he should learn how to fish himself, \\ msmith@freebsd.org \\ and he'll hate you for a lifetime. \\ msmith@cdrom.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 18:49:31 2000 Delivered-To: freebsd-arch@freebsd.org Received: from smtp02.primenet.com (smtp02.primenet.com [206.165.6.132]) by hub.freebsd.org (Postfix) with ESMTP id D28E637BBBB; Wed, 24 May 2000 18:49:22 -0700 (PDT) (envelope-from tlambert@usr05.primenet.com) Received: (from daemon@localhost) by smtp02.primenet.com (8.9.3/8.9.3) id SAA09486; Wed, 24 May 2000 18:49:13 -0700 (MST) Received: from usr05.primenet.com(206.165.6.205) via SMTP by smtp02.primenet.com, id smtpdAAABraass; Wed May 24 18:48:56 2000 Received: (from tlambert@localhost) by usr05.primenet.com (8.8.5/8.8.5) id SAA12472; Wed, 24 May 2000 18:48:57 -0700 (MST) From: Terry Lambert Message-Id: <200005250148.SAA12472@usr05.primenet.com> Subject: Re: Preemptive kernel on older X86 hardware To: msmith@FreeBSD.ORG (Mike Smith) Date: Thu, 25 May 2000 01:48:57 +0000 (GMT) Cc: tlambert@primenet.com (Terry Lambert), cp@bsdi.com (Chuck Paterson), arch@FreeBSD.ORG In-Reply-To: <200005250127.SAA03314@mass.cdrom.com> from "Mike Smith" at May 24, 2000 06:27:30 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > > I think the work required to build two kernels instead of one, and > > then CPU-testing in the loader to pick one, is really trivial. I > > think there are better approaches to the problem than this, but > > this is enough to throw out that idea entirely. > > Are the mutexes inlined, or are they all function calls? If the latter, > loading lock_smp.ko vs. lock_std.ko would be pretty trivial... 8) I expect that they are inlined. What would be the "NOP" overhead on a pipelined instruction cache? I suspect that it would be the overhead of loading the instructions themselves, and that most of these cases would be prefetched. You could also tag functions that used these things to go into a seperate section, and then load "586_critical_sections.ko" instead, which would also be pretty trivial. There are really a heck of a lot of ways to solve this problem, without going to "processor not supported". One I've been evilly considering is that the older processors lacking the instruction could simply take an instruction fault, and handle it in software, like the old uVAX II trying to run VAX 11/780 instructions that it didn't support in silicon (I think the CRC32 instruction was one such beast). A more salient FreeBSD example might be the FPU emulator... I will say, without direct comment on Chuck's other posts at this time, that his suggested method is the same as that used in SVR4.0.2 ES/MP and in Solaris until at least 2.4. Going to kernel preemption and making the FS and other code reentrant rin UnixWare (SVR4.2) resulted in the SMP kernel running 135% of the speed of the non-SMP prior kernel -- on _UP_ hardware. This was in Novell benchmarks that weren't published, to my knowledge. The one problem I have with data locking rather than critical sectioning is that it is seductive, and can not easily be made to scale well over 4 processors. Some of the new Intel stuff is supposed to have 4 processors per chip casing... so this is not totally satisfying to me. Sun avoids this on their hardware by using clustering software and (effectively) discrete hardware. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 18:52:57 2000 Delivered-To: freebsd-arch@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id 42CFB37B776 for ; Wed, 24 May 2000 18:52:51 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.9.3/8.9.1) id SAA78130; Wed, 24 May 2000 18:52:45 -0700 (PDT) (envelope-from dillon) Date: Wed, 24 May 2000 18:52:45 -0700 (PDT) From: Matthew Dillon Message-Id: <200005250152.SAA78130@apollo.backplane.com> To: Chuck Paterson Cc: arch@FreeBSD.ORG Subject: Re: Short summary Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG :virtually the same speed as the Giant lock BSD/OS kernel in a uniprocessor :environment. It occurred to me today that in a uniprocessor environment :the lock prefix to the cmpxchg can be removed. I ran some :experiments. The following data is from a very limited sample size. On :a couple of different systems with different clock rates removing :the lock prefix reduced execution time of mutex operations to one :third of their original value. Running the same job with two kernels :whose only difference was the lock prefix there was a reduction in :system time of 2.5 percent. This suggested that the total system :time used for locking with the SMP locks in place is 3.6 percent :and with the the locks trimmed for uniprocessor only operation is :1.2 percent. (Please excuse rounding errors). Chuck, there was extensive debate and testing on both Linux and FreeBSD with regards to locked instructions in an SMP environment. It was determined that there is an optimization one can make which improves lock performance on SMP systems. The jist of the optimization is that if you use a lock prefix when locking, you do *not* need a lock prefix when unlocking. Write ordering is guarenteed on Intel (586 or above). Also, for recursive locks for the case where you ALREADY hold the lock, you do not need a lock prefix when incrementing or decrementing the count. Take a look at the FreeBSD mp_unlock code in 4.x or 5.x (with a reasonably recent cvs update) for an example. /usr/src/sys/i386/i386/mplock.s, the MPrellock_edx subroutine. These changes saved over a microsecond in syscall overhead for FreeBSD SMP. This optimization radically improves the performance of an unlock at the cost of adding a slight delay before contending cpu's see the change. Since there is no lock contention 99.999% of the time, the delay is completely absorbed and you realize an increase in performance across the board. The recursion optimization makes recursive locks practical in an SMP setting. There is virtually *NO* overhead after you've obtained the initial lock. -Matt To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 19: 4: 2 2000 Delivered-To: freebsd-arch@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id 90DE037B6CC for ; Wed, 24 May 2000 19:03:59 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.9.3/8.9.1) id TAA78162; Wed, 24 May 2000 19:03:55 -0700 (PDT) (envelope-from dillon) Date: Wed, 24 May 2000 19:03:55 -0700 (PDT) From: Matthew Dillon Message-Id: <200005250203.TAA78162@apollo.backplane.com> To: Chuck Paterson Cc: arch@FreeBSD.ORG Subject: Re: Short summary References: <200005250103.TAA15470@berserker.bsdi.com> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG : What is being proposed here is a major kernel architectural :change. The locking provided by SPLS in the traditional BSD kernel goes away :and is replaced by mutexs. This model is very different. Interrupts :do not in general get blocked. When interrupt service code needs :access to some piece of data which is actively being modified by :what is traditionally thought of as the "top half" the interrupt :thread will block at that point. The finer grained the locking the :less chance of a collision, but the number of locking operation :goes up and a price has to be paid. :... :Chuck We've discussed SMP evolution on and off over the last year or two, and I think everyone will agree that the data-centric mid-range granularity mutex model is the best way to go. The one problem with this concept is that there is no easy migration path getting from SPL ops to Mutex ops, because an SPL op currently involves a bitmask of several interrupts rather then a single interrupt. Thus the SPL ops would translate into several mutexes rather then a single mutex and wind up being *very* expensive for the period of time it takes us to then move onto the next step (collapsing them into single mutexes again). We were looking for an easier migration path in order to try to keep the kernel runnable for the month or two (or longer) it would require to get everything straightened out. At this point in time I've thrown up my hands and am now willing to take this intermediate step just to get the ball rolling, under the assumption that we can collapse the mutexes together in fairly short order once we get them working in the first place. Another possibility is to simply collapse all the spl*()'s into a single mutex temporarily and then slowly move code sections into using their own data-mutexes, which is easier to do but has obvious interrupt performance issues. Do you have any thoughts on how to deal with this problem? -Matt Matthew Dillon To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 19: 5:36 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id F35D337B9AF for ; Wed, 24 May 2000 19:05:32 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@[127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id UAA16126; Wed, 24 May 2000 20:05:25 -0600 (MDT) Message-Id: <200005250205.UAA16126@berserker.bsdi.com> To: Terry Lambert Cc: arch@freebsd.org Subject: Re: Preemptive kernel on older X86 hardware From: Chuck Paterson Date: Wed, 24 May 2000 20:05:25 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG } }Jeffrey Hsu recently commented on this to me in regards to a paper }by Michael Greenwald, in regard to having lock-free synchronized }lists in the kernel. } If you do this in MI code you also have to have primitives to handle the relaxed memory model on machines like Sparc. } }It would be interesting to know why the cycle counter was required, }and how BSDI would deal with this on SMP Alpha and SPARC machines, }or the new IBM 64 PPC processor NUMA machines... } It is not actually required, it is just better, and therefore one more differnce. The cycle counter is used for a couple of things. One is a very fast tracing facility, the other is microtime. On sparc there is a cycle counter for the tracing but I believe BSD/OS falls back to the standard clock stuff for microtime. This stuff is all in MD code. } }You can buy external APICs for 486 based systems to achieve 486 }SMP. This is where Intel SMP came from in the first place. The }APICs went into the processor proper more to thwart third party }processor manufacturers from creating SMP boxes using Intel APIC }chips but not CPU chips, than anything else. } My understanding is that the external apics have a different programming model and are no fun to deal with. Seems like something someone could do at anytime if they had a few months to kill. } }I think the work required to build two kernels instead of one, and }then CPU-testing in the loader to pick one, is really trivial. I }think there are better approaches to the problem than this, but }this is enough to throw out that idea entirely. } }Supporting loading of more segments, with segment attributes, is an }obvious approach; Microsoft does this with VxD's and other NT and }Windows kernel components, today. The only thing you have to worry }about is being able to serialize access to the section being replaced }(only during the replacement process) and not be in it when you flip }the page table entry. You could even do defragmentation of the kernel }address space, if you thought is was more useful than running on the }new code and copying the new code to the old location, and then }running there instead. } } } Terry Lambert } terry@lambert.org }--- First let me say that I purposely picked the proposal I did because some people said it was a possibility and it needed to get accepted or put to rest. I think we know the answer to that The work doing two kernels is no in chosing the one to run, but getting the low level primitives and interrupt code working across all the desired platforms. I would assert this is not trivial, but not a big deal either. The problem I see with Windows thing you mention above is that every module will change. If you want to go fast the mutex operations are going to be inlined. This means that there isn't like one piece you can just replace. Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 19: 8: 6 2000 Delivered-To: freebsd-arch@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id 70E7237B6CC for ; Wed, 24 May 2000 19:08:04 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.9.3/8.9.1) id TAA78220; Wed, 24 May 2000 19:08:00 -0700 (PDT) (envelope-from dillon) Date: Wed, 24 May 2000 19:08:00 -0700 (PDT) From: Matthew Dillon Message-Id: <200005250208.TAA78220@apollo.backplane.com> To: Chuck Paterson Cc: Terry Lambert , arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware References: <200005250205.UAA16126@berserker.bsdi.com> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG We've had very good luck encapsulating our MP lock code in real honest-to-god subroutines rather then trying to make them inline macros. On intel anyway, subroutine calls are *cheap*, especially compared to the overhead of a locked instruction or even an L1 cache miss. It's a no-brainer. -Matt To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 19:11:19 2000 Delivered-To: freebsd-arch@freebsd.org Received: from mass.cdrom.com (mass.cdrom.com [204.216.28.234]) by hub.freebsd.org (Postfix) with ESMTP id 2BAEE37BA03 for ; Wed, 24 May 2000 19:11:17 -0700 (PDT) (envelope-from msmith@mass.cdrom.com) Received: from mass.cdrom.com (localhost [127.0.0.1]) by mass.cdrom.com (8.9.3/8.9.3) with ESMTP id TAA03797; Wed, 24 May 2000 19:13:10 -0700 (PDT) (envelope-from msmith@mass.cdrom.com) Message-Id: <200005250213.TAA03797@mass.cdrom.com> X-Mailer: exmh version 2.1.1 10/15/1999 To: Terry Lambert Cc: arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware In-reply-to: Your message of "Thu, 25 May 2000 01:48:57 -0000." <200005250148.SAA12472@usr05.primenet.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Wed, 24 May 2000 19:13:10 -0700 From: Mike Smith Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > > > I think the work required to build two kernels instead of one, and > > > then CPU-testing in the loader to pick one, is really trivial. I > > > think there are better approaches to the problem than this, but > > > this is enough to throw out that idea entirely. > > > > Are the mutexes inlined, or are they all function calls? If the latter, > > loading lock_smp.ko vs. lock_std.ko would be pretty trivial... 8) > > I expect that they are inlined. This would actually be a fairly Bad Thing on at least some platforms. Maybe we should consider whether they should be inlined on some and function calls on others (where there might be separate implementations). Function call overhead on i386 is pretty low... On eg. Sparc where there's only likely to be one implementation, we don't need the ability to switch them around, and thus they could be inlined. > You could also tag functions that used these things to go into a > seperate section, and then load "586_critical_sections.ko" instead, > which would also be pretty trivial. That'd make for some pretty damn ugly modules/drivers. > One I've been evilly considering is that the older processors > lacking the instruction could simply take an instruction fault, This would only make sense if the fault handler patched the offending code; it'd be too expensive otherwise. I don't think this'd work on i386, since the replacement code sequence is larger... -- \\ Give a man a fish, and you feed him for a day. \\ Mike Smith \\ Tell him he should learn how to fish himself, \\ msmith@freebsd.org \\ and he'll hate you for a lifetime. \\ msmith@cdrom.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 19:18:32 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id E7A4637B9AF for ; Wed, 24 May 2000 19:18:27 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@[127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id UAA16278; Wed, 24 May 2000 20:18:17 -0600 (MDT) Message-Id: <200005250218.UAA16278@berserker.bsdi.com> To: Matthew Dillon Cc: arch@freebsd.org Subject: Re: Short summary From: Chuck Paterson Date: Wed, 24 May 2000 20:18:17 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG } Chuck, there was extensive debate and testing on both Linux and } FreeBSD with regards to locked instructions in an SMP environment. } It was determined that there is an optimization one can make which } improves lock performance on SMP systems. } } The jist of the optimization is that if you use a lock prefix when } locking, you do *not* need a lock prefix when unlocking. Write } ordering is guarenteed on Intel (586 or above). This won't work with the BSD/OS locks. The reason is that we use the same word to detect that the someone is waiting for the lock to be released. This works with spins locks kind of (more just ahead) because you don't need to do anything if someone else want the lock you just go ahead and release it. With non spin locks when you release a contested lock you need to go put another process on the run queueu. The "more just head" is address by you ahead actually. } } Also, for recursive locks for the case where you ALREADY hold the lock, } you do not need a lock prefix when incrementing or decrementing the } count. } The BSD/OS mutexs generally use the locked operation and take a miss on the mutex if it is already held, even by the same process. There is a flag to on the mtx_enter/mtx_exit the recursion is likely and that the code should check this before doing the locked operation. By default BSD/OS mutexs are always optimized for the non contested, non-recursed cased. This means that everything is just a cmpxchg and if that wins your done. } the cost of adding a slight delay before contending cpu's see the } change. Since there is no lock contention 99.999% of the time, the } delay is completely absorbed and you realize an increase in performance } across the board. } } The recursion optimization makes recursive locks practical in an SMP } setting. There is virtually *NO* overhead after you've obtained the } initial lock. } } -Matt } To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 19:28: 8 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id 1735C37B83B for ; Wed, 24 May 2000 19:28:06 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@[127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id UAA16415; Wed, 24 May 2000 20:28:00 -0600 (MDT) Message-Id: <200005250228.UAA16415@berserker.bsdi.com> To: Matthew Dillon Cc: Terry Lambert , arch@freebsd.org Subject: Re: Preemptive kernel on older X86 hardware From: Chuck Paterson Date: Wed, 24 May 2000 20:28:00 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG We are seeing 30 clock ticks for a locked mutex operation. Only 10 if you take out the lock for UP systems. (hopefully I did the math right.) I just don't see how putting a function call is going to be cheap relatively. Is it possible that you are hitting so many spin locks that the function calltime gets burried. Chuck ----- Begin Included Message ----- Date: Wed, 24 May 2000 19:08:00 -0700 From: Matthew Dillon To: Chuck Paterson Subject: Re: Preemptive kernel on older X86 hardware cc: Terry Lambert , arch@FreeBSD.ORG We've had very good luck encapsulating our MP lock code in real honest-to-god subroutines rather then trying to make them inline macros. On intel anyway, subroutine calls are *cheap*, especially compared to the overhead of a locked instruction or even an L1 cache miss. It's a no-brainer. -Matt To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message ----- End Included Message ----- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 19:29:13 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id E8FC837B83B; Wed, 24 May 2000 19:29:10 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@[127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id UAA16470; Wed, 24 May 2000 20:29:09 -0600 (MDT) Message-Id: <200005250229.UAA16470@berserker.bsdi.com> To: Mike Smith Cc: Terry Lambert , arch@freebsd.org Subject: Re: Preemptive kernel on older X86 hardware From: Chuck Paterson Date: Wed, 24 May 2000 20:29:09 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG }> }> I expect that they are inlined. } }This would actually be a fairly Bad Thing on at least some platforms. }Maybe we should consider whether they should be inlined on some and }function calls on others (where there might be separate implementations). BSD/OS inlines for sparc on Intel, but this is MD code so you can do as you will. Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 19:31:31 2000 Delivered-To: freebsd-arch@freebsd.org Received: from urban.iinet.net.au (urban.iinet.net.au [203.59.24.231]) by hub.freebsd.org (Postfix) with ESMTP id 3DA3D37BCD3 for ; Wed, 24 May 2000 19:31:27 -0700 (PDT) (envelope-from julian@elischer.org) Received: from popserver-02.iinet.net.au (popserver-02.iinet.net.au [203.59.24.148]) by urban.iinet.net.au (8.8.7/8.8.7) with ESMTP id KAA12597; Thu, 25 May 2000 10:31:20 +0800 Received: from jules.elischer.org (reggae-05-120.nv.iinet.net.au [203.59.72.120]) by popserver-02.iinet.net.au (8.9.3/8.9.3) with SMTP id KAA14899; Thu, 25 May 2000 10:31:11 +0800 Message-ID: <392C901D.31DFF4F5@elischer.org> Date: Wed, 24 May 2000 19:29:49 -0700 From: Julian Elischer X-Mailer: Mozilla 3.04Gold (X11; I; FreeBSD 5.0-CURRENT i386) MIME-Version: 1.0 To: Matthew Dillon Cc: Chuck Paterson , Terry Lambert , arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware References: <200005250205.UAA16126@berserker.bsdi.com> <200005250208.TAA78220@apollo.backplane.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Matthew Dillon wrote: > > We've had very good luck encapsulating our MP lock code in real > honest-to-god subroutines rather then trying to make them inline > macros. > > On intel anyway, subroutine calls are *cheap*, especially compared > to the overhead of a locked instruction or even an L1 cache miss. > > It's a no-brainer. One paper I read recently (can't remember which) used the method of making all locks calls to very generic (slow) code to start with, and as each lock is run, it replaces the calling code with a call to the appropriate 'quick' call, depending on teh cpu type and whether there si more than one of them. In the UP case, it replaces them with very simple locks or NOPS depending on the type of lock etc. The kernel speeds up as it gradually replaces all the locks. (In fact not so gradual, as >90% of them were replace by the end of booting). Doesn't work for romable code, though you could even do that if you used a redirection table :-) > > -Matt > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-arch" in the body of the message -- __--_|\ Julian Elischer / \ julian@elischer.org ( OZ ) World tour 2000 ---> X_.---._/ presently in: Perth v To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 19:31:53 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id 7A0E137BCF7; Wed, 24 May 2000 19:31:51 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@[127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id UAA16563; Wed, 24 May 2000 20:31:50 -0600 (MDT) Message-Id: <200005250231.UAA16563@berserker.bsdi.com> Cc: Mike Smith , Terry Lambert , arch@freebsd.org Subject: Re: Preemptive kernel on older X86 hardware From: Chuck Paterson Date: Wed, 24 May 2000 20:31:49 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Try again to at least make so sense. BSD/OS inlines for sparc and Intel, but this is MD code so you can do as you will. } BSD/OS inlines for sparc on Intel, but this is MD code so }you can do as you will. } }Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 19:41: 7 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id 7FF1737B83B for ; Wed, 24 May 2000 19:41:04 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@[127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id UAA16669; Wed, 24 May 2000 20:41:01 -0600 (MDT) Message-Id: <200005250241.UAA16669@berserker.bsdi.com> To: Matthew Dillon Cc: arch@freebsd.org Subject: Re: Short summary From: Chuck Paterson Date: Wed, 24 May 2000 20:41:00 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG What BSD/OS did, and I the only way I understand to solve the problem is sort of the same as your collapse into a single mutex, except we collapsed even more. This is the short version: We created a single lock "Giant". All the paths into the kernel got Giant and as they left released it. We also made sleep and tsleep release Giant, and re-acquire it when a process starts to run. At that point we turned SPLS off. There are at this point no interrupts in the kernel, but who cares it was just used to make progress. The amazing thing was that the system didn't "feel" bad. I later learned that many/most/all Cray models run without interrupts in the kernel. Some place along here we added the extra argument to sleep/tsleep, maybe first maybe now. We were then able to start pulling code out from under Giant. We build facilities to do the "right" thing for safe and unsafe drivers through dev switch and interrupt service. Chuck ----- Begin Included Message ----- We've discussed SMP evolution on and off over the last year or two, and I think everyone will agree that the data-centric mid-range granularity mutex model is the best way to go. The one problem with this concept is that there is no easy migration path getting from SPL ops to Mutex ops, because an SPL op currently involves a bitmask of several interrupts rather then a single interrupt. Thus the SPL ops would translate into several mutexes rather then a single mutex and wind up being *very* expensive for the period of time it takes us to then move onto the next step (collapsing them into single mutexes again). We were looking for an easier migration path in order to try to keep the kernel runnable for the month or two (or longer) it would require to get everything straightened out. At this point in time I've thrown up my hands and am now willing to take this intermediate step just to get the ball rolling, under the assumption that we can collapse the mutexes together in fairly short order once we get them working in the first place. Another possibility is to simply collapse all the spl*()'s into a single mutex temporarily and then slowly move code sections into using their own data-mutexes, which is easier to do but has obvious interrupt performance issues. Do you have any thoughts on how to deal with this problem? -Matt Matthew Dillon To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message ----- End Included Message ----- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 20:22:50 2000 Delivered-To: freebsd-arch@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id 6BEF037B9DE for ; Wed, 24 May 2000 20:22:47 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.9.3/8.9.1) id UAA78579; Wed, 24 May 2000 20:22:43 -0700 (PDT) (envelope-from dillon) Date: Wed, 24 May 2000 20:22:43 -0700 (PDT) From: Matthew Dillon Message-Id: <200005250322.UAA78579@apollo.backplane.com> To: Chuck Paterson Cc: arch@freebsd.org Subject: Re: Short summary References: <200005250218.UAA16278@berserker.bsdi.com> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG :} The jist of the optimization is that if you use a lock prefix when :} locking, you do *not* need a lock prefix when unlocking. Write :} ordering is guarenteed on Intel (586 or above). : : This won't work with the BSD/OS locks. The reason is that : we use the same word to detect that the someone is waiting : for the lock to be released. This works with spins : locks kind of (more just ahead) because you don't : need to do anything if someone else want the lock : you just go ahead and release it. With non spin : locks when you release a contested lock you need : to go put another process on the run queueu. Ouch, having the contending cpu actually do a locked write to the lock (i.e. cache line) held by another cpu is really, really slow. Both processors will eat the full overhead of the hardware cache coherency protocol - It's about 3 times as expensive as a contended lock without the ping-pong writing and about twice as expensive as a non-contending lock, and recursive locks using this model will be about 5x as expensive even in the best case. If there is any way to avoid this, I would avoid this. : The "more just head" is address by you ahead actually. :} :} Also, for recursive locks for the case where you ALREADY hold the lock, :} you do not need a lock prefix when incrementing or decrementing the :} count. :} : : The BSD/OS mutexs generally use the locked operation and : take a miss on the mutex if it is already held, even : by the same process. There is a flag to on the : mtx_enter/mtx_exit the recursion is likely and : that the code should check this before doing the : locked operation. : : By default BSD/OS mutexs are always optimized for the : non contested, non-recursed cased. This means that : everything is just a cmpxchg and if that wins your : done. If you can get rid of the contending-cpu-writes-to-the-lock case, your best case recursion code will be about 5 times faster in the recursion case and your best case non-contending non-recursive lock case will be about twice as fast. -Matt Matthew Dillon To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 20:36:29 2000 Delivered-To: freebsd-arch@freebsd.org Received: from zippy.cdrom.com (zippy.cdrom.com [204.216.27.228]) by hub.freebsd.org (Postfix) with ESMTP id 1A19137BDCA for ; Wed, 24 May 2000 20:36:26 -0700 (PDT) (envelope-from jkh@zippy.cdrom.com) Received: from localhost (jkh@localhost [127.0.0.1]) by zippy.cdrom.com (8.9.3/8.9.3) with ESMTP id UAA75463; Wed, 24 May 2000 20:38:21 -0700 (PDT) (envelope-from jkh@zippy.cdrom.com) To: Matthew Dillon Cc: Chuck Paterson , arch@FreeBSD.ORG Subject: Re: Short summary In-reply-to: Your message of "Wed, 24 May 2000 19:03:55 PDT." <200005250203.TAA78162@apollo.backplane.com> Date: Wed, 24 May 2000 20:38:21 -0700 Message-ID: <75460.959225901@localhost> From: "Jordan K. Hubbard" Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > The one problem with this concept is that there is no easy migration > path getting from SPL ops to Mutex ops, because an SPL op currently > involves a bitmask of several interrupts rather then a single interrupt. > Thus the SPL ops would translate into several mutexes rather then a > single mutex and wind up being *very* expensive for the period of time > it takes us to then move onto the next step (collapsing them into single > mutexes again). We were looking for an easier migration path in > order to try to keep the kernel runnable for the month or two (or > longer) it would require to get everything straightened out. I think it's also fair to say that we'd be willing to accept a certain period of outright breakage in FreeBSD-current if it gets us to the right place, e.g. a painless transition is not necessarily the goal so much as getting things done right. We've gone through similar periods with -current in the past and nobody had too much difficultly with the concept provided that we ANNOUNCED it well in advance and told the cvsuppers to basically turn their cron jobs off or elect not to try and build the world until the tree was declared safe again. - Jordan To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 20:37:37 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id 7AD0237BDCA for ; Wed, 24 May 2000 20:37:34 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@[127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id VAA17273; Wed, 24 May 2000 21:37:31 -0600 (MDT) Message-Id: <200005250337.VAA17273@berserker.bsdi.com> To: Matthew Dillon Cc: arch@freebsd.org Subject: Re: Short summary From: Chuck Paterson Date: Wed, 24 May 2000 21:37:31 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG } } Ouch, having the contending cpu actually do a locked write } to the lock (i.e. cache line) held by another cpu is really, } really slow. Both processors will eat the full overhead of } the hardware cache coherency protocol - It's about 3 times } as expensive as a contended lock without the ping-pong writing } and about twice as expensive as a non-contending lock, } and recursive locks using this model will be about 5x as expensive } even in the best case. } } If there is any way to avoid this, I would avoid this. } I don't believe so. The mutex release has to be a locked operation so we can detect if the mutex is just going contended. Also remember that when this happens the other cpu isn't involved when we do the unlock. The other cpu is running some other process, or actually may be the cpu releasing the lock. We don't have the two processors banging on the cache line at the same time like occurs with spin locks. When we believe recursion is likely we can do the checking for recursion up front. We do mess with the recursion count without locks. Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 20:42:16 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id B687537BA56 for ; Wed, 24 May 2000 20:42:12 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@[127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id VAA17364; Wed, 24 May 2000 21:42:06 -0600 (MDT) Message-Id: <200005250342.VAA17364@berserker.bsdi.com> To: "Jordan K. Hubbard" Cc: Matthew Dillon , arch@freebsd.org Subject: Re: Short summary From: Chuck Paterson Date: Wed, 24 May 2000 21:42:05 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG While it will be true that the tree will be broken sometimes I personally believe that in general it has to be kept running. You might not want to use it for any real work. The reason being that these changes are way to big to debug all at once. Chuck } }I think it's also fair to say that we'd be willing to accept a certain }period of outright breakage in FreeBSD-current if it gets us to the }right place, e.g. a painless transition is not necessarily the goal so }much as getting things done right. We've gone through similar periods }with -current in the past and nobody had too much difficultly with the }concept provided that we ANNOUNCED it well in advance and told the }cvsuppers to basically turn their cron jobs off or elect not to try }and build the world until the tree was declared safe again. } To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 20:45:40 2000 Delivered-To: freebsd-arch@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id DD31C37BDCA for ; Wed, 24 May 2000 20:45:37 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.9.3/8.9.1) id UAA78676; Wed, 24 May 2000 20:45:34 -0700 (PDT) (envelope-from dillon) Date: Wed, 24 May 2000 20:45:34 -0700 (PDT) From: Matthew Dillon Message-Id: <200005250345.UAA78676@apollo.backplane.com> To: Chuck Paterson Cc: Terry Lambert , arch@freebsd.org Subject: Re: Preemptive kernel on older X86 hardware References: <200005250228.UAA16415@berserker.bsdi.com> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG : We are seeing 30 clock ticks for a locked mutex :operation. Only 10 if you take out the lock for UP :systems. (hopefully I did the math right.) I just don't see how :putting a function call is going to be cheap relatively. : : Is it possible that you are hitting so many :spin locks that the function calltime gets burried. : :Chuck Function call overhead is around 17 nS (On a PIII 450) in the cache case. A locked instruction of any sort, non-contending, eats 73 nS. A locked instruction with the second cpu doing a contending locked instruction costs 322 nS on EACH cpu. <<<<<<<<<<<<< VERY EXPENSIVE The non-locked unlock operation eats one clock cycle (less then 10 nS), so being able to use this for unlocking will double the best-case performance of an SMP lock in the non-contending case, and triple it (or even better) in the contending case. So using a procedure is perfectly acceptable. It really is a no-brainer. You also really, really want to use non-locked instructions for unlocking and for recursive locks if you can. -Matt Matthew Dillon :----- Begin Included Message ----- :Date: Wed, 24 May 2000 19:08:00 -0700 :From: Matthew Dillon :To: Chuck Paterson :Subject: Re: Preemptive kernel on older X86 hardware :cc: Terry Lambert , arch@FreeBSD.ORG : : We've had very good luck encapsulating our MP lock code in real : honest-to-god subroutines rather then trying to make them inline : macros. : : On intel anyway, subroutine calls are *cheap*, especially compared : to the overhead of a locked instruction or even an L1 cache miss. : : It's a no-brainer. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 21: 1:37 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id 0E28837BDCA for ; Wed, 24 May 2000 21:01:35 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@[127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id WAA17540; Wed, 24 May 2000 22:01:18 -0600 (MDT) Message-Id: <200005250401.WAA17540@berserker.bsdi.com> To: Matthew Dillon Cc: Terry Lambert , arch@freebsd.org Subject: Re: Preemptive kernel on older X86 hardware From: Chuck Paterson Date: Wed, 24 May 2000 22:01:18 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG } Function call overhead is around 17 nS (On a PIII 450) in } the cache case. } } A locked instruction of any sort, non-contending, eats } 73 nS. This means that you use 17ns and on a UP machine with and non locked cmpxchg the time is going to propably more than double. I still content that this is the single most important thing to make fast in the uncontended case and everybit counts. Actually this can be decided by experimentation when the thing is running. It's not like a fundamental design decision. } A locked instruction with the second cpu doing a contending locked } instruction costs 322 nS on EACH cpu. <<<<<<<<<<<<< VERY EXPENSIVE With a non spin mutex 322ns just doesn't matter because when this happens we are going to do a context switch. } The non-locked unlock operation eats one clock cycle (less then 10 nS), } so being able to use this for unlocking will double the best-case } performance of an SMP lock in the non-contending case, and triple it } (or even better) in the contending case. If we have contention on a unlock then once again speed doesn't matter much we are going to being mucking about putting something on the run queueu. If the task we put on the run queue is high priority then we will be switching to it. Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed May 24 22:54:25 2000 Delivered-To: freebsd-arch@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id AD8C737B64C for ; Wed, 24 May 2000 22:54:23 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.9.3/8.9.1) id WAA79002; Wed, 24 May 2000 22:54:19 -0700 (PDT) (envelope-from dillon) Date: Wed, 24 May 2000 22:54:19 -0700 (PDT) From: Matthew Dillon Message-Id: <200005250554.WAA79002@apollo.backplane.com> To: Chuck Paterson Cc: Terry Lambert , arch@freebsd.org Subject: Re: Preemptive kernel on older X86 hardware References: <200005250401.WAA17540@berserker.bsdi.com> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG : : :} Function call overhead is around 17 nS (On a PIII 450) in :} the cache case. :} :} A locked instruction of any sort, non-contending, eats :} 73 nS. : :This means that you use 17ns and on a UP machine with and non locked :cmpxchg the time is going to propably more than double. I :still content that this is the single most important thing :to make fast in the uncontended case and everybit counts. : :Actually this can be decided by experimentation when the :thing is running. It's not like a fundamental design decision. No, such small numbers aren't that important. Do you want me to tell you what the cost of an L2 cache miss is with a *normal* memory read? You might be surprised how bad it is. -Matt Matthew Dillon To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu May 25 1: 5:34 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id 9219137BC09 for ; Thu, 25 May 2000 01:05:28 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@[127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id CAA19058; Thu, 25 May 2000 02:05:21 -0600 (MDT) Message-Id: <200005250805.CAA19058@berserker.bsdi.com> To: Matthew Dillon Cc: Terry Lambert , arch@freebsd.org Subject: Re: Preemptive kernel on older X86 hardware From: Chuck Paterson Date: Thu, 25 May 2000 02:05:21 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG } } No, such small numbers aren't that important. Do you want me } to tell you what the cost of an L2 cache miss is with a *normal* } memory read? You might be surprised how bad it is. } } -Matt } Matthew Dillon } } Actually it wouldn't suprise me anymore. We had a case where locking wasn't quite right in some tlb shoot down code. The shooter managed to get out of pmap and back in again while the shootie was trying to recover from the page it had removed from the tlb and access one or two cache line. Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu May 25 1:29:36 2000 Delivered-To: freebsd-arch@freebsd.org Received: from zippy.cdrom.com (zippy.cdrom.com [204.216.27.228]) by hub.freebsd.org (Postfix) with ESMTP id AD6DA37BA6C for ; Thu, 25 May 2000 01:29:34 -0700 (PDT) (envelope-from jkh@zippy.cdrom.com) Received: from localhost (jkh@localhost [127.0.0.1]) by zippy.cdrom.com (8.9.3/8.9.3) with ESMTP id BAA82648; Thu, 25 May 2000 01:31:24 -0700 (PDT) (envelope-from jkh@zippy.cdrom.com) To: Matthew Dillon Cc: Chuck Paterson , Terry Lambert , arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware In-reply-to: Your message of "Wed, 24 May 2000 19:08:00 PDT." <200005250208.TAA78220@apollo.backplane.com> Date: Thu, 25 May 2000 01:31:23 -0700 Message-ID: <82645.959243483@localhost> From: "Jordan K. Hubbard" Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > On intel anyway, subroutine calls are *cheap*, especially compared > to the overhead of a locked instruction or even an L1 cache miss. I don't believe this is true on all the architectures FreeBSD is anticipated to run on in the "near future", however. - Jordan To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu May 25 1:44:37 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id 2DFA837BDF6 for ; Thu, 25 May 2000 01:44:29 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@[127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id CAA19436; Thu, 25 May 2000 02:44:20 -0600 (MDT) Message-Id: <200005250844.CAA19436@berserker.bsdi.com> To: "Jordan K. Hubbard" Cc: Matthew Dillon , Terry Lambert , arch@freebsd.org Subject: Re: Preemptive kernel on older X86 hardware From: Chuck Paterson Date: Thu, 25 May 2000 02:44:20 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG The good news is that the actual implementation of the mutexs is machine dependent and can change wildly over different architectures with no need for the callers to know. Chuck "Jordan K. Hubbard" wrote on: Thu, 25 May 2000 01:31:23 PDT }> On intel anyway, subroutine calls are *cheap*, especially compared }> to the overhead of a locked instruction or even an L1 cache miss. } }I don't believe this is true on all the architectures FreeBSD is }anticipated to run on in the "near future", however. } }- Jordan To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu May 25 3:39:21 2000 Delivered-To: freebsd-arch@freebsd.org Received: from overcee.netplex.com.au (peter1.yahoo.com [208.48.107.4]) by hub.freebsd.org (Postfix) with ESMTP id 9E59F37BE07 for ; Thu, 25 May 2000 03:39:11 -0700 (PDT) (envelope-from peter@netplex.com.au) Received: from netplex.com.au (localhost [127.0.0.1]) by overcee.netplex.com.au (Postfix) with ESMTP id D802F1CE1; Thu, 25 May 2000 03:39:10 -0700 (PDT) (envelope-from peter@netplex.com.au) X-Mailer: exmh version 2.1.1 10/15/1999 To: Chuck Paterson Cc: "Jordan K. Hubbard" , Matthew Dillon , Terry Lambert , arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware In-Reply-To: Message from Chuck Paterson of "Thu, 25 May 2000 02:44:20 MDT." <200005250844.CAA19436@berserker.bsdi.com> Date: Thu, 25 May 2000 03:39:10 -0700 From: Peter Wemm Message-Id: <20000525103910.D802F1CE1@overcee.netplex.com.au> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Chuck Paterson wrote: > > The good news is that the actual implementation of the > mutexs is machine dependent and can change wildly over different > architectures with no need for the callers to know. > > Chuck I haven't read the whole thread yet, but I'd like to remind folks that we have an in-kernel linker that can probably help a bit. We can arrange to have either the SMP or UP versions of the mutex modules linked in depending on which type of system it is. A bit of work would be required to make this fly, but we have the infrastructure there. Cheers, -Peter -- Peter Wemm - peter@FreeBSD.org; peter@yahoo-inc.com; peter@netplex.com.au "All of this is for nothing if we don't go to the stars" - JMS/B5 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu May 25 4:39:41 2000 Delivered-To: freebsd-arch@freebsd.org Received: from haldjas.folklore.ee (Haldjas.folklore.ee [193.40.6.121]) by hub.freebsd.org (Postfix) with ESMTP id B011837BE26; Thu, 25 May 2000 04:39:04 -0700 (PDT) (envelope-from narvi@haldjas.folklore.ee) Received: from localhost (narvi@localhost) by haldjas.folklore.ee (8.9.3/8.9.3) with SMTP id NAA49930; Thu, 25 May 2000 13:38:52 +0200 (EET) (envelope-from narvi@haldjas.folklore.ee) Date: Thu, 25 May 2000 13:38:52 +0200 (EET) From: Narvi To: Mike Smith Cc: Terry Lambert , Chuck Paterson , arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware In-Reply-To: <200005250127.SAA03314@mass.cdrom.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Wed, 24 May 2000, Mike Smith wrote: > > I think the work required to build two kernels instead of one, and > > then CPU-testing in the loader to pick one, is really trivial. I > > think there are better approaches to the problem than this, but > > this is enough to throw out that idea entirely. > > Are the mutexes inlined, or are they all function calls? If the latter, > loading lock_smp.ko vs. lock_std.ko would be pretty trivial... 8) > or even lock_i386.ko or lock_i486.ko if the differences in functionality provided by the different processors are that great. Of course, it's not that much help if we call cyrix MMX capable CPUs 486-class cpus. > -- > \\ Give a man a fish, and you feed him for a day. \\ Mike Smith > \\ Tell him he should learn how to fish himself, \\ msmith@freebsd.org > \\ and he'll hate you for a lifetime. \\ msmith@cdrom.com > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu May 25 6:49: 0 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id 0B8F337C189 for ; Thu, 25 May 2000 06:48:57 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@[127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id HAA22603; Thu, 25 May 2000 07:48:53 -0600 (MDT) Message-Id: <200005251348.HAA22603@berserker.bsdi.com> To: Matthew Dillon Cc: arch@freebsd.org Subject: Re: Preemptive kernel on older X86 hardware From: Chuck Paterson Date: Thu, 25 May 2000 07:48:53 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Matt, I looked a little more at the BSD/OS mutex macros. I see that we don't do a recusrion check first even when it has been flaged as likely. This does seem less than perfect. I'll flip this stuff around as well as undoing the lock prefix on releasing spin locks. I'll let you know what happens. Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu May 25 7:30:49 2000 Delivered-To: freebsd-arch@freebsd.org Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by hub.freebsd.org (Postfix) with ESMTP id 4E70137C2CF for ; Thu, 25 May 2000 07:30:46 -0700 (PDT) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.9.3/8.9.3) with ESMTP id KAA09993; Thu, 25 May 2000 10:30:43 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.9.3/8.9.1) id KAA27339; Thu, 25 May 2000 10:30:13 -0400 (EDT) (envelope-from gallatin@cs.duke.edu) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Date: Thu, 25 May 2000 10:30:13 -0400 (EDT) To: Chuck Paterson Cc: arch@freebsd.org Subject: Re: Short summary In-Reply-To: <200005250103.TAA15470@berserker.bsdi.com> References: <200005250103.TAA15470@berserker.bsdi.com> X-Mailer: VM 6.43 under 20.4 "Emerald" XEmacs Lucid Message-ID: <14637.13569.649341.206224@grasshopper.cs.duke.edu> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Chuck Paterson writes: > BSD/OS SMPng kernel. An MP capable version of this kernel runs > virtually the same speed as the Giant lock BSD/OS kernel in a uniprocessor > environment. It occurred to me today that in a uniprocessor environment Chuck, What sorts of workloads have you made comparisons between the SMPng & Giant-lock uniprocessor kernels on? Do you have any quantative results that state what the cost or benefit of SMPng on a uniprocessor in a heavily interrupt driven environment is? I'm curious about tortuous workloads like a firewall between multiple 100Mb or Gigabit network segments, or a saturated Web (or NFS) server w/multiple disk controllers and network adapters (Eg, workloads like the imfamous Mindcraft benchmark (in single CPU mode), or SPEC SFS97 or SPEC WEB99). I'm certainly not trying to stand in the way of progress, I'm just curious what the quantitative performance difference is like 'on the edge' Thanks, Drew ------------------------------------------------------------------------------ Andrew Gallatin, Sr Systems Programmer http://www.cs.duke.edu/~gallatin Duke University Email: gallatin@cs.duke.edu Department of Computer Science Phone: (919) 660-6590 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu May 25 8: 1:10 2000 Delivered-To: freebsd-arch@freebsd.org Received: from rover.village.org (rover.village.org [204.144.255.49]) by hub.freebsd.org (Postfix) with ESMTP id 717A537C412 for ; Thu, 25 May 2000 08:01:07 -0700 (PDT) (envelope-from imp@billy-club.village.org) Received: from billy-club.village.org (billy-club.village.org [10.0.0.3]) by rover.village.org (8.9.3/8.9.3) with ESMTP id JAA19056; Thu, 25 May 2000 09:01:05 -0600 (MDT) (envelope-from imp@billy-club.village.org) Received: from billy-club.village.org (localhost.village.org [127.0.0.1]) by billy-club.village.org (8.9.3/8.8.3) with ESMTP id JAA01183; Thu, 25 May 2000 09:01:26 -0600 (MDT) Message-Id: <200005251501.JAA01183@billy-club.village.org> To: Chuck Paterson Subject: Re: Preemptive kernel on older X86 hardware Cc: arch@FreeBSD.ORG In-reply-to: Your message of "Wed, 24 May 2000 08:46:03 MDT." <200005241446.IAA05589@berserker.bsdi.com> References: <200005241446.IAA05589@berserker.bsdi.com> Date: Thu, 25 May 2000 09:01:26 -0600 From: Warner Losh Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG In message <200005241446.IAA05589@berserker.bsdi.com> Chuck Paterson writes: : Once FreeBSD has a preemptive kernel FreeBSD will only run on : Pentium or better X86 processors. This is likely unacceptible in the embedded market. We're using boards with a UP 486 133's on them (actually AMDs, but you get the idea). We need some way to build kernels for these boxes. It sure would be nice, but isn't required, to have the same kernel for pentium UP machines. I would have no problem saying SMP is only supported on Pentiums or newer. Warner To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu May 25 8: 2:50 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id 0BA5E37C45C for ; Thu, 25 May 2000 08:02:47 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@[127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id JAA24170; Thu, 25 May 2000 09:02:39 -0600 (MDT) Message-Id: <200005251502.JAA24170@berserker.bsdi.com> To: Andrew Gallatin Cc: arch@freebsd.org Subject: Re: Short summary From: Chuck Paterson Date: Thu, 25 May 2000 09:02:39 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG }Chuck, } }What sorts of workloads have you made comparisons between the SMPng & }Giant-lock uniprocessor kernels on? Do you have any quantative }results that state what the cost or benefit of SMPng on a uniprocessor }in a heavily interrupt driven environment is? We have done virtually no performance testing, our kernel is just barely getting to the point where we can to this testing. There is every reason to expect it to do well, but it just isn't known. We have one customer looking at it for these sorts of things and so far the results are positive. }I'm curious about tortuous workloads like a firewall between multiple }100Mb or Gigabit network segments, or a saturated Web (or NFS) server }w/multiple disk controllers and network adapters (Eg, workloads like }the imfamous Mindcraft benchmark (in single CPU mode), or SPEC SFS97 }or SPEC WEB99). } We currently run NFS under a single lock. I have no clue when we BSDI will do anything about this. The NFS code which is in both FreeBSD and BSD/OS is going to have a major wacking to multi thread it. There are structural problems which are not present in the other file systems. Something is going to have to be done with the huge inline macros so they know what mutex to release. The sharing of macros between server and client makes it even harder. On a related subject talk about blowing cache lines for nothing. I have done just a very few tests and server and client appear to be just slightly faster. The client was (3.5% in one test) in the test I still have numbers for. As a firewall there is yet another issue. You would like to have multiple processors doing checksums at the same time. However, you also don't want to re-order packets. There is also the small issue of static struct sockaddr_in ipaddr = { sizeof(ipaddr), AF_INET }; static struct route ipforward_rt; not exactly MP ready. The above is from FreeBSD, BSD/OS has this problem also. Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu May 25 8: 3:56 2000 Delivered-To: freebsd-arch@freebsd.org Received: from rover.village.org (rover.village.org [204.144.255.49]) by hub.freebsd.org (Postfix) with ESMTP id 1446137C467 for ; Thu, 25 May 2000 08:03:52 -0700 (PDT) (envelope-from imp@billy-club.village.org) Received: from billy-club.village.org (billy-club.village.org [10.0.0.3]) by rover.village.org (8.9.3/8.9.3) with ESMTP id JAA19072; Thu, 25 May 2000 09:03:49 -0600 (MDT) (envelope-from imp@billy-club.village.org) Received: from billy-club.village.org (localhost.village.org [127.0.0.1]) by billy-club.village.org (8.9.3/8.8.3) with ESMTP id JAA01226; Thu, 25 May 2000 09:04:09 -0600 (MDT) Message-Id: <200005251504.JAA01226@billy-club.village.org> To: Matthew Dillon Subject: Re: Preemptive kernel on older X86 hardware Cc: "Daniel C. Sobral" , Chuck Paterson , arch@FreeBSD.ORG In-reply-to: Your message of "Wed, 24 May 2000 09:52:04 PDT." <200005241652.JAA75843@apollo.backplane.com> References: <200005241652.JAA75843@apollo.backplane.com> <200005241446.IAA05589@berserker.bsdi.com> <392BF518.F8170D0E@newsguy.com> Date: Thu, 25 May 2000 09:04:09 -0600 From: Warner Losh Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG In message <200005241652.JAA75843@apollo.backplane.com> Matthew Dillon writes: : For SMP kernels I don't think it's worth supporting either the 386 or : the 486. I agree 100%. I don't want to see that hold up the current work. If someone really wants to do a sequent port, then when they have code to merge we can talk. I'd rather put the burdon on the theoretical porters to an older SMP platform than on the real works porting to modern SMP platforms. Warner To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu May 25 8:27:56 2000 Delivered-To: freebsd-arch@freebsd.org Received: from overcee.netplex.com.au (peter1.yahoo.com [208.48.107.4]) by hub.freebsd.org (Postfix) with ESMTP id 13D4337C679 for ; Thu, 25 May 2000 08:27:49 -0700 (PDT) (envelope-from peter@netplex.com.au) Received: from netplex.com.au (localhost [127.0.0.1]) by overcee.netplex.com.au (Postfix) with ESMTP id AFBF21CE1; Thu, 25 May 2000 08:27:47 -0700 (PDT) (envelope-from peter@netplex.com.au) X-Mailer: exmh version 2.1.1 10/15/1999 To: Warner Losh Cc: Chuck Paterson , arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware In-Reply-To: Message from Warner Losh of "Thu, 25 May 2000 09:01:26 MDT." <200005251501.JAA01183@billy-club.village.org> Date: Thu, 25 May 2000 08:27:47 -0700 From: Peter Wemm Message-Id: <20000525152747.AFBF21CE1@overcee.netplex.com.au> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Warner Losh wrote: > In message <200005241446.IAA05589@berserker.bsdi.com> Chuck Paterson writes: > : Once FreeBSD has a preemptive kernel FreeBSD will only run on > : Pentium or better X86 processors. > > This is likely unacceptible in the embedded market. We're using > boards with a UP 486 133's on them (actually AMDs, but you get the > idea). We need some way to build kernels for these boxes. It sure > would be nice, but isn't required, to have the same kernel for pentium > UP machines. While I think we need to be able to support 386 and 486 still, I'm moderately sure that we could get away with de-supporting those in GENERIC for 5.0+, assuming that it buys us something. I would not have too much trouble with a proposal that a I386_CPU and maybe I486_CPU becoming mutually exclusive with the 586+ stuff. ie: if you will still be able to build a kernel specifically to run on a 486, but by default it would not fly. I think 586+ is a convenient boundary because I am not aware of many 586's that don't have PNPBIOS support, while 486's are mixed as they predate win95 by a fair way. Aiming for a default fresh-install target (remember, 5.0 is 6-12 months away) where we require minimum 586+ and PNPBIOS etc etc would simplify things a fair bit.. In such a scenario it should still be possible to build a kernel to specifically support an i486 on a non-PNP isa-only system without PCI etc. I have a 486 still running and would hate to loose it for sentimental reasons, but I do custom builds for it anyway. I strongly doubt that there will be many *fresh* 486 installs, if any at all. But that is changing the subject. :-) > I would have no problem saying SMP is only supported on Pentiums or > newer. I don't think we support 486 SMP right now, but I could be wrong. > Warner Cheers, -Peter -- Peter Wemm - peter@FreeBSD.org; peter@yahoo-inc.com; peter@netplex.com.au "All of this is for nothing if we don't go to the stars" - JMS/B5 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu May 25 9:14: 7 2000 Delivered-To: freebsd-arch@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id E58E337C79D for ; Thu, 25 May 2000 09:14:03 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.9.3/8.9.1) id JAA82872; Thu, 25 May 2000 09:13:59 -0700 (PDT) (envelope-from dillon) Date: Thu, 25 May 2000 09:13:59 -0700 (PDT) From: Matthew Dillon Message-Id: <200005251613.JAA82872@apollo.backplane.com> To: Chuck Paterson Cc: "Jordan K. Hubbard" , Terry Lambert , arch@freebsd.org Subject: Re: Preemptive kernel on older X86 hardware References: <200005250844.CAA19436@berserker.bsdi.com> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG : : The good news is that the actual implementation of the :mutexs is machine dependent and can change wildly over different :architectures with no need for the callers to know. : :Chuck : :"Jordan K. Hubbard" wrote on: Thu, 25 May 2000 01:31:23 PDT :}> On intel anyway, subroutine calls are *cheap*, especially compared :}> to the overhead of a locked instruction or even an L1 cache miss. :} :}I don't believe this is true on all the architectures FreeBSD is :}anticipated to run on in the "near future", however. :} :}- Jordan Lets use subroutines during development at least, it will make things easier. I don't think anyone can argue with that :-) -Matt Matthew Dillon To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu May 25 9:26:43 2000 Delivered-To: freebsd-arch@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id 554A637C812 for ; Thu, 25 May 2000 09:26:41 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.9.3/8.9.1) id JAA82954; Thu, 25 May 2000 09:26:37 -0700 (PDT) (envelope-from dillon) Date: Thu, 25 May 2000 09:26:37 -0700 (PDT) From: Matthew Dillon Message-Id: <200005251626.JAA82954@apollo.backplane.com> To: Chuck Paterson Cc: Andrew Gallatin , arch@FreeBSD.ORG Subject: Re: Short summary References: <200005251502.JAA24170@berserker.bsdi.com> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG :}I'm curious about tortuous workloads like a firewall between multiple :}100Mb or Gigabit network segments, or a saturated Web (or NFS) server :}w/multiple disk controllers and network adapters (Eg, workloads like :}the imfamous Mindcraft benchmark (in single CPU mode), or SPEC SFS97 :}or SPEC WEB99). :} : : We currently run NFS under a single lock. I have no :clue when we BSDI will do anything about this. The NFS code which :is in both FreeBSD and BSD/OS is going to have a major wacking to :multi thread it. There are structural problems which are not present :in the other file systems. Something is going to have to be done :with the huge inline macros so they know what mutex to release. :The sharing of macros between server and client makes it even :harder. On a related subject talk about blowing cache lines for nothing. :I have done just a very few tests and server and client appear to be just :slightly faster. The client was (3.5% in one test) in the test I :still have numbers for. :... :Chuck I'd argue that we should ignore NFS at least until after the NIC interrupts are made SMP safe. Just fixing the NIC itnerrupts alone will at least double NFS's performance. Not only do the NFS macros make things difficult (even with the cleanups I did last year!), but NFS also has a tendancy to setup I/O's in one process and complete them in another. For example, when you transmit an RPC over a tcp link, *ANY* process doing an rpc op, whether related or not, may wind up receiving the response. -Matt Matthew Dillon To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu May 25 9:28:52 2000 Delivered-To: freebsd-arch@freebsd.org Received: from mass.cdrom.com (adsl-63-206-88-224.dsl.snfc21.pacbell.net [63.206.88.224]) by hub.freebsd.org (Postfix) with ESMTP id 940C337C92A for ; Thu, 25 May 2000 09:28:45 -0700 (PDT) (envelope-from msmith@mass.cdrom.com) Received: from mass.cdrom.com (localhost [127.0.0.1]) by mass.cdrom.com (8.9.3/8.9.3) with ESMTP id JAA02002; Thu, 25 May 2000 09:30:06 -0700 (PDT) (envelope-from msmith@mass.cdrom.com) Message-Id: <200005251630.JAA02002@mass.cdrom.com> X-Mailer: exmh version 2.1.1 10/15/1999 To: "Jordan K. Hubbard" Cc: arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware In-reply-to: Your message of "Thu, 25 May 2000 01:31:23 PDT." <82645.959243483@localhost> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 25 May 2000 09:30:05 -0700 From: Mike Smith Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > > On intel anyway, subroutine calls are *cheap*, especially compared > > to the overhead of a locked instruction or even an L1 cache miss. > > I don't believe this is true on all the architectures FreeBSD is > anticipated to run on in the "near future", however. The point that you may be missing is that we only need care about this on platforms where we need to conditionalise the mutex implementation. -- \\ Give a man a fish, and you feed him for a day. \\ Mike Smith \\ Tell him he should learn how to fish himself, \\ msmith@freebsd.org \\ and he'll hate you for a lifetime. \\ msmith@cdrom.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu May 25 9:35:23 2000 Delivered-To: freebsd-arch@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id C90AC37C97F for ; Thu, 25 May 2000 09:35:15 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.9.3/8.9.1) id JAA83014; Thu, 25 May 2000 09:35:11 -0700 (PDT) (envelope-from dillon) Date: Thu, 25 May 2000 09:35:11 -0700 (PDT) From: Matthew Dillon Message-Id: <200005251635.JAA83014@apollo.backplane.com> To: Chuck Paterson Cc: arch@freebsd.org Subject: Re: Short summary References: <200005250241.UAA16669@berserker.bsdi.com> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG : : :What BSD/OS did, and I the only way I understand to solve the :problem is sort of the same as your collapse into a single :mutex, except we collapsed even more. : :This is the short version: :We created a single lock "Giant". All the paths into the kernel :got Giant and as they left released it. We also made sleep and :tsleep release Giant, and re-acquire it when a process starts to :run. At that point we turned SPLS off. There are at this point no :interrupts in the kernel, but who cares it was just used to make :progress. The amazing thing was that the system didn't "feel" bad. :I later learned that many/most/all Cray models run without interrupts :in the kernel. Some place along here we added the extra argument :to sleep/tsleep, maybe first maybe now. We were then able to start :pulling code out from under Giant. We build facilities to do the :"right" thing for safe and unsafe drivers through dev switch and :interrupt service. : :Chuck I think this will work to get the ball rolling. We can simply 'cli' in the MP lock code and 'sti' in the MP last-unlock code ( i386/i386/mplock.s ). Then we can turn the spl*() calls into NOPs and do the (relatively trivial) fixup to the scheduler. Actually, I don't think we would have to touch the scheduler at all for this step, it already releases the MP lock and it already supports scheduling supervisor contexts to multiple cpu's. (In fact, we already support lockless system calls even though only a few trivial calls do it at the moment!). The next step would be to implement interrupt threads and simply allow them to be scheduled by the scheduler holding the MP lock. After the interrupts are all threaded, we can start removing the MP lock and switching subsystems over to use mutexes. What do you think? -Matt Matthew Dillon To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu May 25 10: 0:20 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id A3FE737B52A for ; Thu, 25 May 2000 10:00:17 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@[127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id LAA25373; Thu, 25 May 2000 11:00:08 -0600 (MDT) Message-Id: <200005251700.LAA25373@berserker.bsdi.com> To: Matthew Dillon Cc: "Jordan K. Hubbard" , Terry Lambert , arch@freebsd.org Subject: Re: Preemptive kernel on older X86 hardware From: Chuck Paterson Date: Thu, 25 May 2000 11:00:07 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG } } Lets use subroutines during development at least, it will make } things easier. I don't think anyone can argue with that :-) } Almost.) I certainly think that the actually locking stuff can be in a function but we really want to wrap the function in a macro so we can put tracing in. Being able to look at a trace and see file and line numbers for mutex locks and unlocks is invaluable. Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu May 25 10: 2:32 2000 Delivered-To: freebsd-arch@freebsd.org Received: from obie.softweyr.com (obie.softweyr.com [204.68.178.33]) by hub.freebsd.org (Postfix) with ESMTP id 7CD7737B637 for ; Thu, 25 May 2000 10:02:29 -0700 (PDT) (envelope-from wes@softweyr.com) Received: from softweyr.com (homer.softweyr.com [204.68.178.39]) by obie.softweyr.com (8.8.8/8.8.8) with ESMTP id LAA06218; Thu, 25 May 2000 11:02:22 -0600 (MDT) (envelope-from wes@softweyr.com) Message-ID: <392D5D52.133E3F54@softweyr.com> Date: Thu, 25 May 2000 11:05:22 -0600 From: Wes Peters Organization: Softweyr LLC X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 4.0-STABLE i386) X-Accept-Language: en MIME-Version: 1.0 To: Chuck Paterson Cc: arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware References: <200005241446.IAA05589@berserker.bsdi.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Chuck Paterson wrote: > > The argument, within BSDI, for supporting the older stuff is that > new embedded systems are being built with these processors. Raw > chip cost being the reason. While I believe the person telling me > this, I haven't personally seen the evidence, I haven't looked > either. 386ex, 486gx, ns486, elon400. Yup, designers are still using these chips in embedded systems. -- "Where am I, and what am I doing in this handbasket?" Wes Peters Softweyr LLC wes@softweyr.com http://softweyr.com/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu May 25 11:15:43 2000 Delivered-To: freebsd-arch@freebsd.org Received: from rover.village.org (rover.village.org [204.144.255.49]) by hub.freebsd.org (Postfix) with ESMTP id F14F837B5A0 for ; Thu, 25 May 2000 11:15:35 -0700 (PDT) (envelope-from imp@harmony.village.org) Received: from harmony.village.org (harmony.village.org [10.0.0.6]) by rover.village.org (8.9.3/8.9.3) with ESMTP id MAA19845; Thu, 25 May 2000 12:15:34 -0600 (MDT) (envelope-from imp@harmony.village.org) Received: from harmony.village.org (localhost.village.org [127.0.0.1]) by harmony.village.org (8.9.3/8.8.3) with ESMTP id MAA85600; Thu, 25 May 2000 12:14:22 -0600 (MDT) Message-Id: <200005251814.MAA85600@harmony.village.org> To: Peter Wemm Subject: Re: Preemptive kernel on older X86 hardware Cc: Chuck Paterson , arch@FreeBSD.ORG In-reply-to: Your message of "Thu, 25 May 2000 08:27:47 PDT." <20000525152747.AFBF21CE1@overcee.netplex.com.au> References: <20000525152747.AFBF21CE1@overcee.netplex.com.au> Date: Thu, 25 May 2000 12:14:22 -0600 From: Warner Losh Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG In message <20000525152747.AFBF21CE1@overcee.netplex.com.au> Peter Wemm writes: : I would not have too much trouble with a proposal that a I386_CPU and : maybe I486_CPU becoming mutually exclusive with the 586+ stuff. ie: if you : will still be able to build a kernel specifically to run on a 486, but by : default it would not fly. I think this would be OK, so long as it buys us something. : I think 586+ is a convenient boundary because I am not aware of many 586's : that don't have PNPBIOS support, while 486's are mixed as they predate win95 : by a fair way. Yes. Lots of the SBCs in the embedded world still are 486 based due to their cheap cost, lower power consumption and low pin counts. I just took plastic off a new 486 133 (AMD 5x86 133) the other day which was made this year. It does have PNPBIOS support. : Aiming for a default fresh-install target (remember, 5.0 is 6-12 months : away) where we require minimum 586+ and PNPBIOS etc etc would simplify : things a fair bit.. In such a scenario it should still be possible to : build a kernel to specifically support an i486 on a non-PNP isa-only system : without PCI etc. I have a 486 still running and would hate to loose it for : sentimental reasons, but I do custom builds for it anyway. I strongly doubt : that there will be many *fresh* 486 installs, if any at all. But how does one upgrade then. These older machines typically do do freshinstalls to upgrade them. In the village we have a spare machine that we upgrade by doing a fresh install, then mutate it to be one of the other machines, then pull the trigger and take the old one down and put the new one in to do a rolling upgrade. Also, there will be many fresh installs onto the embedded hardware. I personally just do an installworld to make these machines, but other users might not be as sophisticated. : But that is changing the subject. :-) : : > I would have no problem saying SMP is only supported on Pentiums or : > newer. : : I don't think we support 486 SMP right now, but I could be wrong. I think that you are right... warner To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu May 25 11:24:13 2000 Delivered-To: freebsd-arch@freebsd.org Received: from rover.village.org (rover.village.org [204.144.255.49]) by hub.freebsd.org (Postfix) with ESMTP id 5266F37B5A0 for ; Thu, 25 May 2000 11:24:10 -0700 (PDT) (envelope-from imp@harmony.village.org) Received: from harmony.village.org (harmony.village.org [10.0.0.6]) by rover.village.org (8.9.3/8.9.3) with ESMTP id MAA19889; Thu, 25 May 2000 12:24:08 -0600 (MDT) (envelope-from imp@harmony.village.org) Received: from harmony.village.org (localhost.village.org [127.0.0.1]) by harmony.village.org (8.9.3/8.8.3) with ESMTP id MAA85688; Thu, 25 May 2000 12:22:58 -0600 (MDT) Message-Id: <200005251822.MAA85688@harmony.village.org> To: Julian Elischer Subject: Re: Preemptive kernel on older X86 hardware Cc: Matthew Dillon , Chuck Paterson , Terry Lambert , arch@FreeBSD.ORG In-reply-to: Your message of "Wed, 24 May 2000 19:29:49 PDT." <392C901D.31DFF4F5@elischer.org> References: <392C901D.31DFF4F5@elischer.org> <200005250205.UAA16126@berserker.bsdi.com> <200005250208.TAA78220@apollo.backplane.com> Date: Thu, 25 May 2000 12:22:58 -0600 From: Warner Losh Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG In message <392C901D.31DFF4F5@elischer.org> Julian Elischer writes: : One paper I read recently (can't remember which) : used the method of making all locks calls to very generic (slow) : code to start with, and as each lock is run, it replaces the : calling code with a call to the appropriate 'quick' call, : depending on teh cpu type and whether there si more than one : of them. Sounds like SunOS 4.x shared library code. It did exactly this for all functions in the shared library. Some weird early cache designs made this horribly expensive (I'm thinkiong of Solbourn's KAP processor here, which had some bad silicon bugs they never fixed). Warner To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu May 25 11:26: 6 2000 Delivered-To: freebsd-arch@freebsd.org Received: from rover.village.org (rover.village.org [204.144.255.49]) by hub.freebsd.org (Postfix) with ESMTP id 3521937B5A0 for ; Thu, 25 May 2000 11:26:04 -0700 (PDT) (envelope-from imp@harmony.village.org) Received: from harmony.village.org (harmony.village.org [10.0.0.6]) by rover.village.org (8.9.3/8.9.3) with ESMTP id MAA19902; Thu, 25 May 2000 12:26:02 -0600 (MDT) (envelope-from imp@harmony.village.org) Received: from harmony.village.org (localhost.village.org [127.0.0.1]) by harmony.village.org (8.9.3/8.8.3) with ESMTP id MAA85708; Thu, 25 May 2000 12:24:52 -0600 (MDT) Message-Id: <200005251824.MAA85708@harmony.village.org> To: "Jordan K. Hubbard" Subject: Re: Short summary Cc: Matthew Dillon , Chuck Paterson , arch@FreeBSD.ORG In-reply-to: Your message of "Wed, 24 May 2000 20:38:21 PDT." <75460.959225901@localhost> References: <75460.959225901@localhost> Date: Thu, 25 May 2000 12:24:52 -0600 From: Warner Losh Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG In message <75460.959225901@localhost> "Jordan K. Hubbard" writes: : with -current in the past and nobody had too much difficultly with the : concept provided that we ANNOUNCED it well in advance and told the : cvsuppers to basically turn their cron jobs off or elect not to try : and build the world until the tree was declared safe again. There has been precident in the past for doing this. Certain parts of the system have even been broken for periods of time. So long as this period of time is going to be on the order of one month or less (which in reality means it might stretch to two or three months), I think it is acceptible. Warner To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu May 25 11:32:17 2000 Delivered-To: freebsd-arch@freebsd.org Received: from overcee.netplex.com.au (peter1.yahoo.com [208.48.107.4]) by hub.freebsd.org (Postfix) with ESMTP id F04B237B5A0 for ; Thu, 25 May 2000 11:32:14 -0700 (PDT) (envelope-from peter@netplex.com.au) Received: from netplex.com.au (localhost [127.0.0.1]) by overcee.netplex.com.au (Postfix) with ESMTP id AE08A1CE1; Thu, 25 May 2000 11:32:13 -0700 (PDT) (envelope-from peter@netplex.com.au) X-Mailer: exmh version 2.1.1 10/15/1999 To: Matthew Dillon Cc: Chuck Paterson , "Jordan K. Hubbard" , Terry Lambert , arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware In-Reply-To: Message from Matthew Dillon of "Thu, 25 May 2000 09:13:59 PDT." <200005251613.JAA82872@apollo.backplane.com> Date: Thu, 25 May 2000 11:32:13 -0700 From: Peter Wemm Message-Id: <20000525183213.AE08A1CE1@overcee.netplex.com.au> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Matthew Dillon wrote: > > : > : The good news is that the actual implementation of the > :mutexs is machine dependent and can change wildly over different > :architectures with no need for the callers to know. > : > :Chuck > : > :"Jordan K. Hubbard" wrote on: Thu, 25 May 2000 01:31:23 PDT > :}> On intel anyway, subroutine calls are *cheap*, especially compared > :}> to the overhead of a locked instruction or even an L1 cache miss. > :} > :}I don't believe this is true on all the architectures FreeBSD is > :}anticipated to run on in the "near future", however. > :} > :}- Jordan > > Lets use subroutines during development at least, it will make > things easier. I don't think anyone can argue with that :-) Which is also required for binary kld's. Having two different versions of kld's - one for SMP and one for UP would be bad. Although I have strategy to take care of this when the revamped config/build process is finished. It will be possible to use kernel config-style build parameters for modules as well. For example, building a "GENERIC" module would cause it to make calls to the mutex subroutines, while some compile option would allow you to make a tuned SMP-specific kld for your local machine. Cheers, -Peter To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu May 25 11:37:16 2000 Delivered-To: freebsd-arch@freebsd.org Received: from overcee.netplex.com.au (peter1.yahoo.com [208.48.107.4]) by hub.freebsd.org (Postfix) with ESMTP id 2802037B6B2 for ; Thu, 25 May 2000 11:37:15 -0700 (PDT) (envelope-from peter@netplex.com.au) Received: from netplex.com.au (localhost [127.0.0.1]) by overcee.netplex.com.au (Postfix) with ESMTP id B28F31CE1; Thu, 25 May 2000 11:37:13 -0700 (PDT) (envelope-from peter@netplex.com.au) X-Mailer: exmh version 2.1.1 10/15/1999 To: Warner Losh Cc: Julian Elischer , Matthew Dillon , Chuck Paterson , Terry Lambert , arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware In-Reply-To: Message from Warner Losh of "Thu, 25 May 2000 12:22:58 MDT." <200005251822.MAA85688@harmony.village.org> Date: Thu, 25 May 2000 11:37:13 -0700 From: Peter Wemm Message-Id: <20000525183713.B28F31CE1@overcee.netplex.com.au> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Warner Losh wrote: > In message <392C901D.31DFF4F5@elischer.org> Julian Elischer writes: > : One paper I read recently (can't remember which) > : used the method of making all locks calls to very generic (slow) > : code to start with, and as each lock is run, it replaces the > : calling code with a call to the appropriate 'quick' call, > : depending on teh cpu type and whether there si more than one > : of them. > > Sounds like SunOS 4.x shared library code. It did exactly this for > all functions in the shared library. Some weird early cache designs > made this horribly expensive (I'm thinkiong of Solbourn's KAP > processor here, which had some bad silicon bugs they never fixed). Digital Unix (Tru64) does this too according to their white papers. Everything is compiled to use the generic lock calls. The generic routine modifies the caller binary code to instead call the specific routines for the cpu configuration or replaces the call with NOPs for uniprocessor systems. Cheers, -Peter -- Peter Wemm - peter@FreeBSD.org; peter@yahoo-inc.com; peter@netplex.com.au "All of this is for nothing if we don't go to the stars" - JMS/B5 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu May 25 11:49:48 2000 Delivered-To: freebsd-arch@freebsd.org Received: from rover.village.org (rover.village.org [204.144.255.49]) by hub.freebsd.org (Postfix) with ESMTP id 935A137B6DC for ; Thu, 25 May 2000 11:49:44 -0700 (PDT) (envelope-from imp@harmony.village.org) Received: from harmony.village.org (harmony.village.org [10.0.0.6]) by rover.village.org (8.9.3/8.9.3) with ESMTP id MAA20004; Thu, 25 May 2000 12:49:43 -0600 (MDT) (envelope-from imp@harmony.village.org) Received: from harmony.village.org (localhost.village.org [127.0.0.1]) by harmony.village.org (8.9.3/8.8.3) with ESMTP id MAA85904; Thu, 25 May 2000 12:48:33 -0600 (MDT) Message-Id: <200005251848.MAA85904@harmony.village.org> To: Peter Wemm Subject: Re: Preemptive kernel on older X86 hardware Cc: arch@FreeBSD.ORG In-reply-to: Your message of "Thu, 25 May 2000 11:37:13 PDT." <20000525183713.B28F31CE1@overcee.netplex.com.au> References: <20000525183713.B28F31CE1@overcee.netplex.com.au> Date: Thu, 25 May 2000 12:48:33 -0600 From: Warner Losh Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG In message <20000525183713.B28F31CE1@overcee.netplex.com.au> Peter Wemm writes: : Digital Unix (Tru64) does this too according to their white papers. : Everything is compiled to use the generic lock calls. The generic routine : modifies the caller binary code to instead call the specific routines for : the cpu configuration or replaces the call with NOPs for uniprocessor : systems. IRIX (and now Linux/mips) will bcopy the right interrupt routine to the fixed location in memory where these things live on the MIPS cpu based on what CPU you have, even to the point of having specific versions with work arounds for specific revs of the CPU (eg a R44000 v 2.2 will get the routine that works around bugs in that chip, while a R4400 v5.0 will get a faster version since that chip has no know bugs and a R3000A will get a completely different routine, etc). Warner To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu May 25 14: 0:35 2000 Delivered-To: freebsd-arch@freebsd.org Received: from alcanet.com.au (mail.alcanet.com.au [203.62.196.10]) by hub.freebsd.org (Postfix) with ESMTP id E245937B557; Thu, 25 May 2000 14:00:28 -0700 (PDT) (envelope-from jeremyp@pc0640.alcatel.com.au) Received: by border.alcanet.com.au id <115225>; Fri, 26 May 2000 07:00:36 +1000 From: Peter Jeremy Subject: Re: Preemptive kernel on older X86 hardware To: Mike Smith Cc: Terry Lambert , arch@FreeBSD.ORG Message-Id: <00May26.070036est.115225@border.alcanet.com.au> MIME-version: 1.0 X-Mailer: Mutt 1.0i Content-type: text/plain; charset=us-ascii Date: Fri, 26 May 2000 07:00:35 +1000 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Thu, May 25, 2000 at 11:50:12AM +1000, Terry Lambert wrote: >What would be the "NOP" overhead on a pipelined instruction cache? On a 486, a NOP takes 1 cycle to execute. I suspect it's zero on the more recent processors. As Terry points out, there's also the fetch cost - but I-fetches are (effectively) free unless the processor bus is saturated. BTW, the IA32 has NOP's of various lengths. On the 486, single cycle NOPs can be 1, 2, 3 or 6 bytes long. There are also 5-byte almost-NOP (trash a register or flags) sequences. This means that a NOP sequence is fairly cheap. On Thu, May 25, 2000 at 12:13:43PM +1000, Mike Smith wrote: >> One I've been evilly considering is that the older processors >> lacking the instruction could simply take an instruction fault, > >This would only make sense if the fault handler patched the offending >code; it'd be too expensive otherwise. Not to mention having to make sure that the invalid opcode handler didn't need to access any locks. > I don't think this'd work on >i386, since the replacement code sequence is larger... You just have the larger code as the default and patch the smaller sequence if appropriate. Peter To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu May 25 17:22:25 2000 Delivered-To: freebsd-arch@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id 42AE337B7BA; Thu, 25 May 2000 17:22:23 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.9.3/8.9.1) id RAA84878; Thu, 25 May 2000 17:22:12 -0700 (PDT) (envelope-from dillon) Date: Thu, 25 May 2000 17:22:12 -0700 (PDT) From: Matthew Dillon Message-Id: <200005260022.RAA84878@apollo.backplane.com> To: Peter Jeremy Cc: Mike Smith , Terry Lambert , arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware References: <00May26.070036est.115225@border.alcanet.com.au> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG NOP's are one cycle on a PIII. -Matt To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri May 26 0:18:40 2000 Delivered-To: freebsd-arch@freebsd.org Received: from point.osg.gov.bc.ca (point.osg.gov.bc.ca [142.32.102.44]) by hub.freebsd.org (Postfix) with ESMTP id 589B237B89B for ; Fri, 26 May 2000 00:18:36 -0700 (PDT) (envelope-from Cy.Schubert@uumail.gov.bc.ca) Received: (from daemon@localhost) by point.osg.gov.bc.ca (8.8.7/8.8.8) id AAA02742; Fri, 26 May 2000 00:17:44 -0700 Received: from passer.osg.gov.bc.ca(142.32.110.29) via SMTP by point.osg.gov.bc.ca, id smtpda02740; Fri May 26 00:17:44 2000 Received: (from uucp@localhost) by passer.osg.gov.bc.ca (8.9.3/8.9.1) id AAA00738; Fri, 26 May 2000 00:17:41 -0700 (PDT) Received: from cwsys9.cwsent.com(10.2.2.1), claiming to be "cwsys.cwsent.com" via SMTP by passer9.cwsent.com, id smtpdwQS736; Fri May 26 00:16:59 2000 Received: (from uucp@localhost) by cwsys.cwsent.com (8.10.1/8.9.1) id e4Q7GwR01336; Fri, 26 May 2000 00:16:58 -0700 (PDT) Message-Id: <200005260716.e4Q7GwR01336@cwsys.cwsent.com> Received: from localhost.cwsent.com(127.0.0.1), claiming to be "cwsys" via SMTP by localhost.cwsent.com, id smtpdsx1332; Fri May 26 00:15:59 2000 X-Mailer: exmh version 2.1.1 10/15/1999 Reply-To: Cy Schubert - ITSD Open Systems Group From: Cy Schubert - ITSD Open Systems Group X-OS: FreeBSD 4.0-STABLE X-Sender: cy To: Peter Wemm Cc: Warner Losh , Chuck Paterson , arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware In-reply-to: Your message of "Thu, 25 May 2000 08:27:47 PDT." <20000525152747.AFBF21CE1@overcee.netplex.com.au> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Fri, 26 May 2000 00:15:59 -0700 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG In message <20000525152747.AFBF21CE1@overcee.netplex.com.au>, Peter Wemm writes : > Warner Losh wrote: > > In message <200005241446.IAA05589@berserker.bsdi.com> Chuck Paterson writes > : > > : Once FreeBSD has a preemptive kernel FreeBSD will only run on > > : Pentium or better X86 processors. > > > > This is likely unacceptible in the embedded market. We're using > > boards with a UP 486 133's on them (actually AMDs, but you get the > > idea). We need some way to build kernels for these boxes. It sure > > would be nice, but isn't required, to have the same kernel for pentium > > UP machines. > > While I think we need to be able to support 386 and 486 still, I'm moderately > sure that we could get away with de-supporting those in GENERIC for 5.0+, > assuming that it buys us something. > > I would not have too much trouble with a proposal that a I386_CPU and > maybe I486_CPU becoming mutually exclusive with the 586+ stuff. ie: if you > will still be able to build a kernel specifically to run on a 486, but by > default it would not fly. > > I think 586+ is a convenient boundary because I am not aware of many 586's > that don't have PNPBIOS support, while 486's are mixed as they predate win95 > by a fair way. > > Aiming for a default fresh-install target (remember, 5.0 is 6-12 months > away) where we require minimum 586+ and PNPBIOS etc etc would simplify > things a fair bit.. In such a scenario it should still be possible to > build a kernel to specifically support an i486 on a non-PNP isa-only system > without PCI etc. I have a 486 still running and would hate to loose it for > sentimental reasons, but I do custom builds for it anyway. I strongly doubt > that there will be many *fresh* 486 installs, if any at all. 486's make good cheap firewalls and in one case I use a 486 as an Xserver/testbed. If a component breaks, no fuss, just replace the whole box. I'd hate to lose the ability to do a fresh install on a 486 or re-install for DRP reasons. A procedure for the user building a boot/install floppy for 486 systems from source (on a 586/686) or supplying a 386/486-only boot floppy on the CDROM would be a good compromise. With the lack of PNPBIOS in this case, since 386/486 machines are simpler, a manual configuration manually using visual configuration mode would also be acceptable -- if you're not sure what's inside a 486 machine, you probably won't have the skill to configure tune the kernel to run on it anyhow. > > But that is changing the subject. :-) > > > I would have no problem saying SMP is only supported on Pentiums or > > newer. > > I don't think we support 486 SMP right now, but I could be wrong. I think that not supporting 486 SMP is much of an issue. The only 486 SMP boxes I've worked on were NCR 3600's running NCR AT&T SYSV R4.2, about the size of a fridge. Regards, Phone: (250)387-8437 Cy Schubert Fax: (250)387-5766 Team Leader, Sun/DEC Team Internet: Cy.Schubert@osg.gov.bc.ca Open Systems Group, ITSD, ISTA Province of BC To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri May 26 5: 6:22 2000 Delivered-To: freebsd-arch@freebsd.org Received: from haldjas.folklore.ee (Haldjas.folklore.ee [193.40.6.121]) by hub.freebsd.org (Postfix) with ESMTP id 3FC0A37B548; Fri, 26 May 2000 05:06:14 -0700 (PDT) (envelope-from narvi@haldjas.folklore.ee) Received: from localhost (narvi@localhost) by haldjas.folklore.ee (8.9.3/8.9.3) with SMTP id OAA70460; Fri, 26 May 2000 14:05:49 +0200 (EET) (envelope-from narvi@haldjas.folklore.ee) Date: Fri, 26 May 2000 14:05:48 +0200 (EET) From: Narvi To: Peter Jeremy Cc: Mike Smith , Terry Lambert , arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware In-Reply-To: <00May26.070036est.115225@border.alcanet.com.au> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Fri, 26 May 2000, Peter Jeremy wrote: > On Thu, May 25, 2000 at 11:50:12AM +1000, Terry Lambert wrote: > >What would be the "NOP" overhead on a pipelined instruction cache? > > On a 486, a NOP takes 1 cycle to execute. I suspect it's zero on the > more recent processors. As Terry points out, there's also the fetch > cost - but I-fetches are (effectively) free unless the processor bus > is saturated. > NOP cost is not free on a lot of processors. The OOO processors have more resources than just 'execute' that are contended for. At the very least, it eats up issue slots. When and how they are removed from the instruction stream is a processor specific issue and varies widely. > > I don't think this'd work on > >i386, since the replacement code sequence is larger... > > You just have the larger code as the default and patch the smaller > sequence if appropriate. > > Peter > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri May 26 6: 5:35 2000 Delivered-To: freebsd-arch@freebsd.org Received: from flood.ping.uio.no (flood.ping.uio.no [129.240.78.31]) by hub.freebsd.org (Postfix) with ESMTP id 5830237B8FD; Fri, 26 May 2000 06:05:24 -0700 (PDT) (envelope-from des@flood.ping.uio.no) Received: (from des@localhost) by flood.ping.uio.no (8.9.3/8.9.3) id PAA95544; Fri, 26 May 2000 15:05:22 +0200 (CEST) (envelope-from des@flood.ping.uio.no) To: cvs-committers@FreeBSD.org Cc: arch@FreeBSD.org Subject: Re: cvs commit: src/lib/libfetch ftp.c References: <200005261256.FAA31080@freefall.freebsd.org> From: Dag-Erling Smorgrav Date: 26 May 2000 15:05:22 +0200 In-Reply-To: Dag-Erling Smorgrav's message of "Fri, 26 May 2000 05:56:53 -0700 (PDT)" Message-ID: Lines: 19 User-Agent: Gnus/5.0802 (Gnus v5.8.2) Emacs/20.4 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Dag-Erling Smorgrav writes: > Log: > Honor FTP_PASSIVE_MODE so caller doesn't need to. There's a catch here: traditionally, active mode was the default and passive mode was explicitly enabled by setting FTP_PASSIVE_MODE or using the -p switch on the command line. Now, passive mode is the default unless FTP_PASSIVE_MODE is set to NO, but there's no way to turn it off on the command line. One might argue that passive mode is quite likely to work while active mode is quite likely not to work, so we err on the side of caution, but I still dislike the fact that the only way to force active mode is to set an environment variable. DES -- Dag-Erling Smorgrav - des@flood.ping.uio.no To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri May 26 6:27: 9 2000 Delivered-To: freebsd-arch@freebsd.org Received: from flood.ping.uio.no (flood.ping.uio.no [129.240.78.31]) by hub.freebsd.org (Postfix) with ESMTP id 3010837B8D3 for ; Fri, 26 May 2000 06:26:49 -0700 (PDT) (envelope-from des@flood.ping.uio.no) Received: (from des@localhost) by flood.ping.uio.no (8.9.3/8.9.3) id PAA95627; Fri, 26 May 2000 15:26:01 +0200 (CEST) (envelope-from des@flood.ping.uio.no) To: arch@freebsd.org Subject: fetch and pkg_* From: Dag-Erling Smorgrav Date: 26 May 2000 15:26:01 +0200 Message-ID: Lines: 8 User-Agent: Gnus/5.0802 (Gnus v5.8.2) Emacs/20.4 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG There's a new snapshot of fetch out, as well as a patch for pkg_install to make it use libfetch instead of libftpio. DES -- Dag-Erling Smorgrav - des@flood.ping.uio.no To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri May 26 9:59:49 2000 Delivered-To: freebsd-arch@freebsd.org Received: from peach.ocn.ne.jp (peach.ocn.ne.jp [210.145.254.87]) by hub.freebsd.org (Postfix) with ESMTP id EA20A37BE51 for ; Fri, 26 May 2000 09:59:40 -0700 (PDT) (envelope-from dcs@newsguy.com) Received: from newsguy.com (p03-dn03kiryunisiki.gunma.ocn.ne.jp [210.232.224.132]) by peach.ocn.ne.jp (8.9.1a/OCN/) with ESMTP id BAA06759; Sat, 27 May 2000 01:59:18 +0900 (JST) Message-ID: <392E97B7.1A575FAC@newsguy.com> Date: Sat, 27 May 2000 00:26:47 +0900 From: "Daniel C. Sobral" X-Mailer: Mozilla 4.7 [en] (Win98; I) X-Accept-Language: en,pt-BR,ja MIME-Version: 1.0 To: Chuck Paterson Cc: Matthew Dillon , "Jordan K. Hubbard" , Terry Lambert , arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware References: <200005251700.LAA25373@berserker.bsdi.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Chuck Paterson wrote: > > Almost.) I certainly think that the actually locking > stuff can be in a function but we really want to wrap the > function in a macro so we can put tracing in. Being able > to look at a trace and see file and line numbers for mutex > locks and unlocks is invaluable. Won't declaring the function as inline work? -- Daniel C. Sobral (8-DCS) dcs@newsguy.com dcs@freebsd.org capo@another.bsdconspiracy.org "Sentience hurts." To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri May 26 10:38:10 2000 Delivered-To: freebsd-arch@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id 1D51537BE92 for ; Fri, 26 May 2000 10:38:08 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.9.3/8.9.1) id KAA90605; Fri, 26 May 2000 10:38:05 -0700 (PDT) (envelope-from dillon) Date: Fri, 26 May 2000 10:38:05 -0700 (PDT) From: Matthew Dillon Message-Id: <200005261738.KAA90605@apollo.backplane.com> To: "Daniel C. Sobral" Cc: Chuck Paterson , "Jordan K. Hubbard" , Terry Lambert , arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware References: <200005251700.LAA25373@berserker.bsdi.com> <392E97B7.1A575FAC@newsguy.com> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG :> Almost.) I certainly think that the actually locking :> stuff can be in a function but we really want to wrap the :> function in a macro so we can put tracing in. Being able :> to look at a trace and see file and line numbers for mutex :> locks and unlocks is invaluable. : :Won't declaring the function as inline work? : :-- :Daniel C. Sobral (8-DCS) Not if you want to use __FILE__, __LINE__, etc... for tracing purposes. Besides, using an inline there creates a lot of bloat. -Matt Matthew Dillon To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri May 26 11:23:27 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id 48C3137C66A for ; Fri, 26 May 2000 11:19:16 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@localhost [127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id MAA14981; Fri, 26 May 2000 12:19:10 -0600 (MDT) Message-Id: <200005261819.MAA14981@berserker.bsdi.com> To: Matthew Dillon Cc: arch@freebsd.org Subject: Re: Short summary From: Chuck Paterson Date: Fri, 26 May 2000 12:19:10 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Some confusion may have been introduced by when I said that we allowed no interrupts in the kernel at the transitional stage. This isn't true, we allowed all interrupts, its just that all they did is schedule a thread, which got put on a run queue which then go run on the way out. I believe that we want to get to the point where all top half code is running under one mutex with interrupts enabled before adding any mutexs. One advantage of having a single well known mutex for unconverted code is that sleep and tsleep can know to release it. Running cli'd will prevent IPIs from being delivered. Any of these that need to run synchronously will fail, such as tlb shoot downs. The code which implements the mutex operations knows that the processor should not be cli'd when acquiring or releasing a non spin mutex. Aside from making the non debug code simpler it has proved to be a valuable debugging tool. Once you have the kernel running with interrupts enabled and under a single lock work can begin in parallel. This without actually having to start another processor. a) Putting in the mechanism so you can have safe and unsafe drivers. You need this unless you want to try and convert all the drivers at once. b) The actual code to handle light weight interrupt threads. c) Malloc, which has to be addressed to a first degree before much or top half work can be done. d) Implementation/porting/debugging of mutex order checking code. e) Converting drivers and driver sub systems. There are multiple paths to getting to running under a single mutex. What Matt suggested below can be modified slight to achieve this. I'll try and lay out what was done with BSD/OS though it was a long time ago. I should also point out that it didn't take very long. One person for a couple maybe 3 weeks. The actual code took very little time, but debugging when the most basic stuff is suspect was painful. The trace stuff in the SMPng code can now trace to memory on the PCI bus. This would have made things much easier. The spontaneous reboots where a killer. This is what we did for BSD/OS, not the modification of Matt's suggestion to achieve the same thing. Its been a while so I am undoubtedly forgetting details. It was all straight forward and because apparent in the normal course of events. 1) Defined basic types for macros. 2) Added mutex argument to sleep and tsleep, modified all callers to pass in NULL. 3) Added code to create the process for an interrupt thread. Really just called fork I think. Got thread created and safely stopped, not running not on run queue, and other process running. 4) Did basic testing of mutexs, both in user code and then in kernel. Not used for anything but test to see that they operated as expected. In our case this didn't really get all the bugs out. 5) Put code in spls so they could be turned into NOP by setting a new_mode variable 6) Added code to top level interrupt handlers to put a thread on the run queue and return leaving interrupts for that level blocked rather than calling the actual handlers for that level if new mode was set. 6) Added the code to the interrupt threads to a) release sched lock (spin mutex with a bad name) b) acquire giant (non spin) c) call the handles associated with the thread associated with a given level d) release giant e) acquire sched f) re-enable interrupts g) call cpu_switch 8) Added code to the mutex routines so they would not do anything unless new mode was set. 9) Added code to cpu switch to fix up recursion count and owner on sched_lock 10) Added code around all calls to cpu_switch to acquire and release the sched lock. 10) Add acquiring and releasing of giant in trap an syscall. 11) Made sleep/tsleep give up giant if held on entry and acquire on exit. 12) Flipped the switch and debugged for a week or so. A couple of items above where actually done here, but I don't remember which ones. I only hacked up the non sio mode interrupt code so we never made the kernel in this configuration run with more than one processor. At this point someone else was working on the real interrupt handler and could have some confidence that when he started trying to run them the rest of the system would behave. The SMPng kernel went directly from hacked up code above to the light weight interrupt code. This took a long time, months. This code is very optimized, assembler code which is then patched together on the fly to match the level, type of interrupt source and the number of handlers which need to be called. The goal, which was achieved, was to make getting in an out of an interrupt thread as cheap or cheaper than what was present in the previous kernel. This was all part of the effort to make sure we would not require two kernels in the long run. I would have really liked to have a MP capable kernel in less time. This could have been achieved with a fairly simple code to always use heavy weight interrupt threads. The person who really knew how to deal with the APIC was the same person doing the light weight threads, so this didn't happen in BSD/OS. We, meaning FreeBSD this time, should certainly consider this. This is assuming an approach similar to that which I have outlined is adopted. While waiting for real interrupt code: Really made sleep and tsleep do the right thing. Actually this took several go arounds, but it got close here. I Made malloc use its own lock. Added the run priorities really needed by interrupt threads. Added the priority propagation code. Added the code for safe and unsafe drivers. And mainly worked on the non SMPng kernel. Not because I didn't have stuff to do on the SMPng kernel, but other stuff got pushed first. Chuck ----- Begin Included Message ----- I think this will work to get the ball rolling. We can simply 'cli' in the MP lock code and 'sti' in the MP last-unlock code ( i386/i386/mplock.s ). Then we can turn the spl*() calls into NOPs and do the (relatively trivial) fixup to the scheduler. Actually, I don't think we would have to touch the scheduler at all for this step, it already releases the MP lock and it already supports scheduling supervisor contexts to multiple cpu's. (In fact, we already support lockless system calls even though only a few trivial calls do it at the moment!). The next step would be to implement interrupt threads and simply allow them to be scheduled by the scheduler holding the MP lock. After the interrupts are all threaded, we can start removing the MP lock and switching subsystems over to use mutexes. What do you think? -Matt Matthew Dillon To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message ----- End Included Message ----- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri May 26 11:59:53 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id AAC6B37B51B for ; Fri, 26 May 2000 11:59:50 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@localhost [127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id MAA15375 for ; Fri, 26 May 2000 12:59:49 -0600 (MDT) Message-Id: <200005261859.MAA15375@berserker.bsdi.com> To: arch@freebsd.org Subject: Sorry, SMPng tracing From: Chuck Paterson Date: Fri, 26 May 2000 12:59:49 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Sorry, I didn't realize the trace was near as big as it was. Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri May 26 12:16:33 2000 Delivered-To: freebsd-arch@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id 62B4537B54F for ; Fri, 26 May 2000 12:16:31 -0700 (PDT) (envelope-from cp@berserker.bsdi.com) Received: from berserker.bsdi.com (cp@localhost [127.0.0.1]) by berserker.bsdi.com (8.9.3/8.9.3) with ESMTP id NAA15580 for ; Fri, 26 May 2000 13:16:31 -0600 (MDT) Message-Id: <200005261916.NAA15580@berserker.bsdi.com> Cc: arch@freebsd.org Subject: SMPng tracing From: Chuck Paterson Date: Fri, 26 May 2000 13:16:30 -0600 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Presuming that the list server is going refuse to deliver the way to large piece of mail I posted, I put its contents in freefall:~cp/trace_output. It is an example of the tracing in the BSD/OS SMPng kernel. Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri May 26 16:58:53 2000 Delivered-To: freebsd-arch@freebsd.org Received: from smtp09.phx.gblx.net (smtp09.phx.gblx.net [206.165.6.139]) by hub.freebsd.org (Postfix) with ESMTP id 742FF37B93D for ; Fri, 26 May 2000 16:58:49 -0700 (PDT) (envelope-from tlambert@usr05.primenet.com) Received: (from daemon@localhost) by smtp09.phx.gblx.net (8.9.3/8.9.3) id QAA43492; Fri, 26 May 2000 16:58:42 -0700 Received: from usr05.primenet.com(206.165.6.205) via SMTP by smtp09.phx.gblx.net, id smtpdvRHMMa; Fri May 26 16:58:35 2000 Received: (from tlambert@localhost) by usr05.primenet.com (8.8.5/8.8.5) id QAA23009; Fri, 26 May 2000 16:58:26 -0700 (MST) From: Terry Lambert Message-Id: <200005262358.QAA23009@usr05.primenet.com> Subject: Re: Preemptive kernel on older X86 hardware To: cp@bsdi.com (Chuck Paterson) Date: Fri, 26 May 2000 23:58:26 +0000 (GMT) Cc: dillon@apollo.backplane.com (Matthew Dillon), tlambert@primenet.com (Terry Lambert), arch@freebsd.org In-Reply-To: <200005250228.UAA16415@berserker.bsdi.com> from "Chuck Paterson" at May 24, 2000 08:28:00 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > We are seeing 30 clock ticks for a locked mutex > operation. Only 10 if you take out the lock for UP > systems. (hopefully I did the math right.) I just don't see how > putting a function call is going to be cheap relatively. > > Is it possible that you are hitting so many > spin locks that the function calltime gets burried. I've long been a fan of non-statistical profiling; I even implemented it for VxD profiling (using a modified gprof) under Windows. This uses "compiler magic" to put entry and exit functions in ever function; for it to be effective, you have to disable tail call optimization. The Microsoft compiler only generates an "entry" function, so you have to fudge the return code based on the caller data you get from the stack, so that if you know how to fudge it, then you push additional data on a seperate call return stack, and set the return code to point to your exit code instead. The return code then saves the return registers, records the (non-statistical) arc, puts the right stuff on the stack, and then returns again. Ugly, but it works. With source code for the compiler, we wouldn't have to screw around like this to get the same _real_, as opposed to statistical, data. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri May 26 17:16:27 2000 Delivered-To: freebsd-arch@freebsd.org Received: from smtp02.primenet.com (smtp02.primenet.com [206.165.6.132]) by hub.freebsd.org (Postfix) with ESMTP id B676A37BA00 for ; Fri, 26 May 2000 17:16:22 -0700 (PDT) (envelope-from tlambert@usr05.primenet.com) Received: (from daemon@localhost) by smtp02.primenet.com (8.9.3/8.9.3) id RAA03899; Fri, 26 May 2000 17:16:11 -0700 (MST) Received: from usr05.primenet.com(206.165.6.205) via SMTP by smtp02.primenet.com, id smtpdAAA5gaGKh; Fri May 26 17:16:07 2000 Received: (from tlambert@localhost) by usr05.primenet.com (8.8.5/8.8.5) id RAA23476; Fri, 26 May 2000 17:16:07 -0700 (MST) From: Terry Lambert Message-Id: <200005270016.RAA23476@usr05.primenet.com> Subject: Re: Preemptive kernel on older X86 hardware To: peter@netplex.com.au (Peter Wemm) Date: Sat, 27 May 2000 00:16:07 +0000 (GMT) Cc: dillon@apollo.backplane.com (Matthew Dillon), cp@bsdi.com (Chuck Paterson), jkh@zippy.cdrom.com (Jordan K. Hubbard), tlambert@primenet.com (Terry Lambert), arch@FreeBSD.ORG In-Reply-To: <20000525183213.AE08A1CE1@overcee.netplex.com.au> from "Peter Wemm" at May 25, 2000 11:32:13 AM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > > Lets use subroutines during development at least, it will make > > things easier. I don't think anyone can argue with that :-) > > Which is also required for binary kld's. Having two different versions > of kld's - one for SMP and one for UP would be bad. Although I have strategy > to take care of this when the revamped config/build process is finished. > It will be possible to use kernel config-style build parameters for > modules as well. > > For example, building a "GENERIC" module would cause it to make calls to > the mutex subroutines, while some compile option would allow you to make > a tuned SMP-specific kld for your local machine. I would suggest using inlines, but have subroutines available, and compile modules to use the subroutine version. The inlines don't have to be inlines during developement, as you say, they can be macro-wrapped functions. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sat May 27 1:11:35 2000 Delivered-To: freebsd-arch@freebsd.org Received: from peach.ocn.ne.jp (peach.ocn.ne.jp [210.145.254.87]) by hub.freebsd.org (Postfix) with ESMTP id D1D0037B5C7 for ; Sat, 27 May 2000 01:11:32 -0700 (PDT) (envelope-from dcs@newsguy.com) Received: from newsguy.com (p23-dn03kiryunisiki.gunma.ocn.ne.jp [210.232.224.152]) by peach.ocn.ne.jp (8.9.1a/OCN/) with ESMTP id RAA16335; Sat, 27 May 2000 17:11:18 +0900 (JST) Message-ID: <392F8374.C99B68AB@newsguy.com> Date: Sat, 27 May 2000 17:12:36 +0900 From: "Daniel C. Sobral" X-Mailer: Mozilla 4.7 [en] (Win98; I) X-Accept-Language: en,pt-BR,ja MIME-Version: 1.0 To: Matthew Dillon Cc: Chuck Paterson , "Jordan K. Hubbard" , Terry Lambert , arch@FreeBSD.ORG Subject: Re: Preemptive kernel on older X86 hardware References: <200005251700.LAA25373@berserker.bsdi.com> <392E97B7.1A575FAC@newsguy.com> <200005261738.KAA90605@apollo.backplane.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Matthew Dillon wrote: > > :Won't declaring the function as inline work? > : > :-- > :Daniel C. Sobral (8-DCS) > > Not if you want to use __FILE__, __LINE__, etc... for tracing purposes. > > Besides, using an inline there creates a lot of bloat. For debugging purposes only, but I didn't think of __FILE__ & cia, indeed. -- Daniel C. Sobral (8-DCS) dcs@newsguy.com dcs@freebsd.org capo@another.bsdconspiracy.org "Sentience hurts." To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sat May 27 3:44:57 2000 Delivered-To: freebsd-arch@freebsd.org Received: from anchor-post-34.mail.demon.net (anchor-post-34.mail.demon.net [194.217.242.92]) by hub.freebsd.org (Postfix) with ESMTP id 1AB6637B88D for ; Sat, 27 May 2000 03:44:55 -0700 (PDT) (envelope-from dfr@nlsystems.com) Received: from nlsys.demon.co.uk ([158.152.125.33] helo=herring.nlsystems.com) by anchor-post-34.mail.demon.net with esmtp (Exim 2.12 #1) id 12ve5N-000AwU-0Y; Sat, 27 May 2000 11:44:54 +0100 Received: from salmon.nlsystems.com (salmon.nlsystems.com [10.0.0.3]) by herring.nlsystems.com (8.9.3/8.8.8) with ESMTP id LAA69538; Sat, 27 May 2000 11:51:45 +0100 (BST) (envelope-from dfr@nlsystems.com) Date: Sat, 27 May 2000 11:50:13 +0100 (BST) From: Doug Rabson To: Chuck Paterson Cc: Matthew Dillon , "Jordan K. Hubbard" , Terry Lambert , arch@freebsd.org Subject: Re: Preemptive kernel on older X86 hardware In-Reply-To: <200005251700.LAA25373@berserker.bsdi.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Thu, 25 May 2000, Chuck Paterson wrote: > } > } Lets use subroutines during development at least, it will make > } things easier. I don't think anyone can argue with that :-) > } > > Almost.) I certainly think that the actually locking > stuff can be in a function but we really want to wrap the > function in a macro so we can put tracing in. Being able > to look at a trace and see file and line numbers for mutex > locks and unlocks is invaluable. Absolutely. If using functions, it might also be a good idea to wrap with an inline which checks for M_SPIN or M_DEF and calls a different implementation function for each. This might allow a slightly more efficient implementation. -- Doug Rabson Mail: dfr@nlsystems.com Nonlinear Systems Ltd. Phone: +44 20 8442 9037 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sat May 27 4: 2:22 2000 Delivered-To: freebsd-arch@freebsd.org Received: from finch-post-11.mail.demon.net (finch-post-11.mail.demon.net [194.217.242.39]) by hub.freebsd.org (Postfix) with ESMTP id A0BA337B8E3 for ; Sat, 27 May 2000 04:02:18 -0700 (PDT) (envelope-from dfr@nlsystems.com) Received: from nlsys.demon.co.uk ([158.152.125.33] helo=herring.nlsystems.com) by finch-post-11.mail.demon.net with esmtp (Exim 2.12 #1) id 12veMC-000GEr-0B; Sat, 27 May 2000 11:02:17 +0000 Received: from salmon.nlsystems.com (salmon.nlsystems.com [10.0.0.3]) by herring.nlsystems.com (8.9.3/8.8.8) with ESMTP id MAA69578; Sat, 27 May 2000 12:09:19 +0100 (BST) (envelope-from dfr@nlsystems.com) Date: Sat, 27 May 2000 12:07:47 +0100 (BST) From: Doug Rabson To: Matthew Dillon Cc: Chuck Paterson , arch@freebsd.org Subject: Re: Short summary In-Reply-To: <200005251635.JAA83014@apollo.backplane.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Thu, 25 May 2000, Matthew Dillon wrote: > > : > : > :What BSD/OS did, and I the only way I understand to solve the > :problem is sort of the same as your collapse into a single > :mutex, except we collapsed even more. > : > :This is the short version: > :We created a single lock "Giant". All the paths into the kernel > :got Giant and as they left released it. We also made sleep and > :tsleep release Giant, and re-acquire it when a process starts to > :run. At that point we turned SPLS off. There are at this point no > :interrupts in the kernel, but who cares it was just used to make > :progress. The amazing thing was that the system didn't "feel" bad. > :I later learned that many/most/all Cray models run without interrupts > :in the kernel. Some place along here we added the extra argument > :to sleep/tsleep, maybe first maybe now. We were then able to start > :pulling code out from under Giant. We build facilities to do the > :"right" thing for safe and unsafe drivers through dev switch and > :interrupt service. > : > :Chuck > > I think this will work to get the ball rolling. We can simply > 'cli' in the MP lock code and 'sti' in the MP last-unlock code > ( i386/i386/mplock.s ). Then we can turn the spl*() calls into > NOPs and do the (relatively trivial) fixup to the scheduler. > Actually, I don't think we would have to touch the scheduler at all > for this step, it already releases the MP lock and it already supports > scheduling supervisor contexts to multiple cpu's. (In fact, we > already support lockless system calls even though only a few trivial > calls do it at the moment!). > > The next step would be to implement interrupt threads and simply > allow them to be scheduled by the scheduler holding the MP lock. > > After the interrupts are all threaded, we can start removing the MP > lock and switching subsystems over to use mutexes. > > What do you think? I think this is a workable plan. If we can get to the stage of just being able to sleep instead of spin on the giant lock, we ought to see a performance improvement since cpus not inside the kernel can then be running user-space processes more often. -- Doug Rabson Mail: dfr@nlsystems.com Nonlinear Systems Ltd. Phone: +44 20 8442 9037 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message