From owner-freebsd-smp Sun Feb 11 12: 1: 4 2001 Delivered-To: freebsd-smp@freebsd.org Received: from thelab.hub.org (SHW39-29.accesscable.net [24.138.39.29]) by hub.freebsd.org (Postfix) with ESMTP id 287D237B401 for ; Sun, 11 Feb 2001 12:01:01 -0800 (PST) Received: from localhost (scrappy@localhost) by thelab.hub.org (8.11.2/8.11.1) with ESMTP id f1BJwet94732 for ; Sun, 11 Feb 2001 15:58:40 -0400 (AST) (envelope-from scrappy@hub.org) X-Authentication-Warning: thelab.hub.org: scrappy owned process doing -bs Date: Sun, 11 Feb 2001 15:58:40 -0400 (AST) From: The Hermit Hacker To: Subject: Quad motherboards ... Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Anyone using any right now that they are particularly happy with, both as far as stability and price are concerned? What should I be seeing as far as prices are concerned? Thanks ... Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy Systems Administrator @ hub.org primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Feb 12 7:33: 7 2001 Delivered-To: freebsd-smp@freebsd.org Received: from ida.interface-business.de (ida.interface-business.de [193.101.57.9]) by hub.freebsd.org (Postfix) with ESMTP id 5F0C837B491; Mon, 12 Feb 2001 07:32:59 -0800 (PST) Received: (from j@localhost) by ida.interface-business.de id f1CFWuY04246; Mon, 12 Feb 2001 16:32:56 +0100 (MET) Date: Mon, 12 Feb 2001 16:32:56 +0100 From: J Wunsch To: freebsd-scsi@FreeBSD.ORG Cc: freebsd-smp@FreeBSD.ORG Subject: Re: Problems with AIC7770-based controller on -current Message-ID: <20010212163256.A84752@ida.interface-business.de> Reply-To: Joerg Wunsch Mail-Followup-To: freebsd-scsi@FreeBSD.ORG, freebsd-smp@freebsd.org References: <20010209182116.P822@ida.interface-business.de> <200102121509.f1CF9wO21693@aslan.scsiguy.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <200102121509.f1CF9wO21693@aslan.scsiguy.com>; from gibbs@scsiguy.com on Mon, Feb 12, 2001 at 08:09:58AM -0700 X-Phone: +49-351-31809-14 X-PGP-Fingerprint: DC 47 E6 E4 FF A6 E9 8F 93 21 E0 7D F9 12 D6 4E Organization: interface systems GmbH, Dresden Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org As Justin T. Gibbs wrote: > >after upgrading this old EISA machine to -current (as of a couple of > >days ago), it no longer works. First there was a typo in eisaconf.c > >(i just committed a fix), but now that it at least scans the EISA bus > >again, the ahc driver no longer wants to talk to my disks. > > Are you still having problems with this? My last commit to -stable > should have corrected a fairly serious issue with twin eisa controllers. Problems vanished after CVS upgrading this morning. Do you want me to try those fixes on -current anyway? Well, almost. After reading a bunch of old commit mails, i'm now fairly positive the problem is not ahc(4)- but SMPNG-related. Whatever it has been, one of the commits made at last weekend must have fixed something like interrupt routing for this old Saturn-based dual Pentium machine. The kernel now at least boots again. (For the folks on -smp: before, it timed out at any access to the SCSI controller, so i couldn't even get a single-user shell to run.) Still, the SMPNG kernel is anything else but stable, but that doesn't seem to be unexpected. :) It sometimes panics right before even displaying the kernel's copyright notice, the crash then happens somehewhere in module_register(). At other occasions, it crashed later during fsck, or even later while running /etc/rc, so i resorted to build a kernel without SMP now in order to get a working environment at all. I'll see to cvs update again in order to not stumble across problems that have already been fixed since. [Reminder: i'm not subscribed to either list, so if you want me to see the replies, retain me in the Cc list, please.] -- J"org Wunsch Unix support engineer joerg_wunsch@interface-systems.de http://www.interface-systems.de/~j To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Feb 12 7:42: 1 2001 Delivered-To: freebsd-smp@freebsd.org Received: from aslan.scsiguy.com (aslan.scsiguy.com [63.229.232.106]) by hub.freebsd.org (Postfix) with ESMTP id C5AE437B491; Mon, 12 Feb 2001 07:41:57 -0800 (PST) Received: from scsiguy.com (localhost [127.0.0.1]) by aslan.scsiguy.com (8.11.2/8.9.3) with ESMTP id f1CFfrO22303; Mon, 12 Feb 2001 08:41:53 -0700 (MST) (envelope-from gibbs@scsiguy.com) Message-Id: <200102121541.f1CFfrO22303@aslan.scsiguy.com> X-Mailer: exmh version 2.2 06/23/2000 with nmh-1.0.4 To: Joerg Wunsch Cc: freebsd-scsi@FreeBSD.ORG, freebsd-smp@FreeBSD.ORG Subject: Re: Problems with AIC7770-based controller on -current In-Reply-To: Your message of "Mon, 12 Feb 2001 16:32:56 +0100." <20010212163256.A84752@ida.interface-business.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Mon, 12 Feb 2001 08:41:53 -0700 From: "Justin T. Gibbs" Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org >Problems vanished after CVS upgrading this morning. Do you want me to >try those fixes on -current anyway? The drivers are roughly identical between the two branches, so you are probably right to think that it is SMPNG releated. If the -stable driver works for you, then the bug is elsewhere. -- Justin To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Feb 12 7:47:36 2001 Delivered-To: freebsd-smp@freebsd.org Received: from ida.interface-business.de (ida.interface-business.de [193.101.57.9]) by hub.freebsd.org (Postfix) with ESMTP id ECD7C37B401; Mon, 12 Feb 2001 07:47:30 -0800 (PST) Received: (from j@localhost) by ida.interface-business.de id f1CFlO145468; Mon, 12 Feb 2001 16:47:24 +0100 (MET) Date: Mon, 12 Feb 2001 16:47:24 +0100 From: J Wunsch To: freebsd-scsi@FreeBSD.ORG, freebsd-smp@FreeBSD.ORG Subject: Re: Problems with AIC7770-based controller on -current Message-ID: <20010212164724.E84752@ida.interface-business.de> Reply-To: Joerg Wunsch Mail-Followup-To: freebsd-scsi@FreeBSD.ORG, freebsd-smp@FreeBSD.ORG References: <20010212163256.A84752@ida.interface-business.de> <200102121541.f1CFfrO22303@aslan.scsiguy.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <200102121541.f1CFfrO22303@aslan.scsiguy.com>; from gibbs@scsiguy.com on Mon, Feb 12, 2001 at 08:41:53AM -0700 X-Phone: +49-351-31809-14 X-PGP-Fingerprint: DC 47 E6 E4 FF A6 E9 8F 93 21 E0 7D F9 12 D6 4E Organization: interface systems GmbH, Dresden Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org As Justin T. Gibbs wrote: > The drivers are roughly identical between the two branches, so you are > probably right to think that it is SMPNG releated. If the -stable > driver works for you, then the bug is elsewhere. It's actually the -current driver that works for me. No matter, the problems i've encountered now are very certainly SMPNG ones. Thanks for your response! -- J"org Wunsch Unix support engineer joerg_wunsch@interface-systems.de http://www.interface-systems.de/~j To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Feb 14 11:31:35 2001 Delivered-To: freebsd-smp@freebsd.org Received: from p1.cs.ohiou.edu (p1.cs.ohiou.edu [132.235.1.11]) by hub.freebsd.org (Postfix) with ESMTP id 8BE9337B4EC for ; Wed, 14 Feb 2001 11:31:33 -0800 (PST) Received: from localhost (frussell@localhost) by p1.cs.ohiou.edu (8.9.3/8.9.3) with SMTP id OAA07072 for ; Wed, 14 Feb 2001 14:31:31 -0500 (EST) Date: Wed, 14 Feb 2001 14:31:30 -0500 (EST) From: Russell Francis X-Sender: frussell@p1 To: freebsd-smp@FreeBSD.org Subject: possible problem with SMP? Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Hello, I am not sure if this is the right place for this question so please feel free to redirect. I am running 4.0-STABLE and recently compiled an SMP kernel, rebooted and it works great. Something that puzzles me is that xosview only displays the stats for one processor and I am unable to get it above 50%. I have a threaded application that also runs but no faster that on a UP kernel. I am just wondering what is going on here any tips on how to find out would be appreciated thanks. -Russ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Feb 14 15:47:39 2001 Delivered-To: freebsd-smp@freebsd.org Received: from corey.datafast.net.au (corey.datafast.net.au [203.123.67.4]) by hub.freebsd.org (Postfix) with SMTP id 3C12137B401 for ; Wed, 14 Feb 2001 15:47:35 -0800 (PST) Received: (qmail 79508 invoked by uid 1000); 14 Feb 2001 23:47:35 -0000 From: "Corey Ralph" Date: Thu, 15 Feb 2001 10:47:35 +1100 To: freebsd-smp@freebsd.org Subject: SMP on HP LH3000R? Message-ID: <20010215104735.A71182@corey.datafast.net.au> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org I have recently added a second cpu into a HP netserver LH3000R. It worked fine before, but now halts when booting. The first time, it told me that I had the options wrong, that I needed to set NBUS to 8 and NAPIC to 2. I did that, now it boots most of the way and locks up after mounting the first hard drive, when it would normally mount the raid. I booted it again with a non-smp kernel and it is working fine again. uname -a: FreeBSD mail.datafast.net.au 4.0-RELEASE FreeBSD 4.0-RELEASE #2: Tue Feb 6 10:29:49 EST 2001 root@mail.datafast.net.au:/usr/src/sys/compile/MAILSERVER i386 Cheers, Corey To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Feb 14 15:54: 0 2001 Delivered-To: freebsd-smp@freebsd.org Received: from mass.dis.org (mass.dis.org [216.240.45.41]) by hub.freebsd.org (Postfix) with ESMTP id 1B64437B401 for ; Wed, 14 Feb 2001 15:53:57 -0800 (PST) Received: from mass.dis.org (localhost [127.0.0.1]) by mass.dis.org (8.11.1/8.11.1) with ESMTP id f1ENsQi01801; Wed, 14 Feb 2001 15:54:26 -0800 (PST) (envelope-from msmith@mass.dis.org) Message-Id: <200102142354.f1ENsQi01801@mass.dis.org> X-Mailer: exmh version 2.1.1 10/15/1999 To: "Corey Ralph" Cc: freebsd-smp@freebsd.org Subject: Re: SMP on HP LH3000R? In-reply-to: Your message of "Thu, 15 Feb 2001 10:47:35 +1100." <20010215104735.A71182@corey.datafast.net.au> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Wed, 14 Feb 2001 15:54:26 -0800 From: Mike Smith Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > I have recently added a second cpu into a HP netserver LH3000R. It > worked fine before, but now halts when booting. > > The first time, it told me that I had the options wrong, that I needed > to set NBUS to 8 and NAPIC to 2. I did that, now it boots most of the > way and locks up after mounting the first hard drive, when it would > normally mount the raid. > > I booted it again with a non-smp kernel and it is working fine again. > > uname -a: > > FreeBSD mail.datafast.net.au 4.0-RELEASE FreeBSD 4.0-RELEASE #2: Tue Feb Aigh! You want 4.2 on this puppy, at least. -- ... every activity meets with opposition, everyone who acts has his rivals and unfortunately opponents also. But not because people want to be opponents, rather because the tasks and relationships force people to take different points of view. [Dr. Fritz Todt] V I C T O R Y N O T V E N G E A N C E To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Feb 14 19:11:13 2001 Delivered-To: freebsd-smp@freebsd.org Received: from obsecurity.dyndns.org (adsl-63-207-60-67.dsl.lsan03.pacbell.net [63.207.60.67]) by hub.freebsd.org (Postfix) with ESMTP id 80EE837B491 for ; Wed, 14 Feb 2001 19:11:10 -0800 (PST) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 34E4066B26; Wed, 14 Feb 2001 19:11:10 -0800 (PST) Date: Wed, 14 Feb 2001 19:11:10 -0800 From: Kris Kennaway To: Russell Francis Cc: freebsd-smp@FreeBSD.org Subject: Re: possible problem with SMP? Message-ID: <20010214191110.D78224@mollari.cthul.hu> References: Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-md5; protocol="application/pgp-signature"; boundary="vni90+aGYgRvsTuO" Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: ; from frussell@p1.cs.ohiou.edu on Wed, Feb 14, 2001 at 02:31:30PM -0500 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org --vni90+aGYgRvsTuO Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Wed, Feb 14, 2001 at 02:31:30PM -0500, Russell Francis wrote: > 50%. I have a threaded application that also runs but no faster that on a > UP kernel. I am just wondering what is going on here any tips on how to > find out would be appreciated thanks. This is actually a FAQ. FreeBSD's userland threads library runs within a single process, so it's not scheduled over multiple CPUs. This is very efficient for UP systems, but not optimal for SMP machines. The LinuxThreads port is an example of rfork()-bases threads (it's a native port of the Linux code) which may be more efficient on SMP for your application. LinuxThreads is also far from optimal - a project is underway to Do Kernel-Supported Threading Right (see the list archives or www.freebsd.org/~jasone/kse) but won't be ready before 6.0-RELEASE, probably. Kris --vni90+aGYgRvsTuO Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.4 (FreeBSD) Comment: For info see http://www.gnupg.org iD8DBQE6i0jNWry0BWjoQKURApqHAJoC6MKM6GF0lWE2zRK9G1QxFbpYOgCgs9N3 ZuIVXGv24LzMuczl/vhH9Oc= =gKlJ -----END PGP SIGNATURE----- --vni90+aGYgRvsTuO-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Feb 14 19:38: 9 2001 Delivered-To: freebsd-smp@freebsd.org Received: from gtei2.bellatlantic.net (gtei2.bellatlantic.net [199.45.40.146]) by hub.freebsd.org (Postfix) with ESMTP id EF42337B491 for ; Wed, 14 Feb 2001 19:38:05 -0800 (PST) Received: from me-513q3sc0zun0.pengar.com (adsl-64-223-148-61.mannh.adsl.bellatlantic.net [64.223.148.61]) by gtei2.bellatlantic.net (8.9.1/8.9.1) with ESMTP id WAA29824 for ; Wed, 14 Feb 2001 22:38:14 -0500 (EST) Message-Id: <5.0.2.1.0.20010214222040.00aac5c0@hobbiton.shire.net> X-Sender: seth-pc@hobbiton.shire.net X-Mailer: QUALCOMM Windows Eudora Version 5.0.2 Date: Wed, 14 Feb 2001 22:34:57 -0500 To: freebsd-smp@FreeBSD.org From: Seth Leigh Subject: Re: possible problem with SMP? In-Reply-To: <20010214191110.D78224@mollari.cthul.hu> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Actually, The Many to Many threading model has been talked up a lot in the past as being better than the One to One model, and I know that I used to believe the talk. I am not so sure anymore. Check out in Solaris 8 the alternate threads library. Do a man on threads and it talks about it. It's in /usr/lib/lwp/libthread.so rather than the standard threads library /usr/lib/libthread.so. The alternate libthread is a One to One model, where all threads are bound automatically, and it isn't possible to create unbound threads, or pure user-level threads. Although it has been said in the past that user-level threads have advantages in terms of scheduling not requiring context switches and all, but there are advantages to making threads bound as well. Consider what many multi-threaded applications do with threads. Many threaded apps have threads which make blocking system calls. On Solaris, when a user-level thread blocks on the lwp it is running on, the kernel has to tell the threads library to create another lwp so that the remaining runnable user-level threads don't get starved for CPU time. Since many threaded apps use threads which all can block, the threads end up causing lwps to be created anyhow. Then, when the thread blocks itself in the threads library (eg: blocks on a mutex, or waits on a condition variable), the lwp is parked for 5 minutes, after which it is released. Well, the a few minutes later the thread is rescheduled onto an lwp and then blocks in the kernel, and we are back to square one. This can lead to lwp churning, whereas if you had just made the threads bound it wouldn't have been an issue. Additionally, a threads library that is strictly One to One is much simpler, and therefore can be more thoroughly debugged and perhaps more efficient. I hear that Sun is considering carefully the future of the threads library, doing very thorough testing on a variety of real-world multi-threaded applications, and that a switch from the current Many to Many model to a more simple One to One model is *not* out of the question. Indeed, they are supposed to be considering it seriously. To be honest, if FreeBSD can just get a threads library that uses a One to One ratio of threads to lwps (I am quite ignorant as to whether FreeBSD has the equivelant of the LWP structure, so just substitute process or whatever if I am off) I would be quite happy, and I am sure that performance would be just fine. Besides, Solaris 8 on modern UltraSparc machines runs in 64-bit mode by default. A large 64-bit Solaris 8 machine with enough RAM can support up to something like 81,000 lwps. I hardly think that "using too many kernel resources" is going to be an issue for such a system with a One to One threads model. Indeed, with a 1 MB default thread stack size, the application is more likely to be limited in how many threads it can create just do to limited physical RAM than it is to be limited by how many lwps the kernel can support (well, I must keep in mind that Solaris won't actually map any physical RAM to any given page in that 1 MB thread stack until and unless an access is made into an address in that page, but still, 81,000 threads would be an awful lot...). I would be very much interested to see just how FreeBSD intends to Do Threading Right. Seth At 07:11 PM 2/14/2001 -0800, you wrote: >On Wed, Feb 14, 2001 at 02:31:30PM -0500, Russell Francis wrote: > > > 50%. I have a threaded application that also runs but no faster that on a > > UP kernel. I am just wondering what is going on here any tips on how to > > find out would be appreciated thanks. > >This is actually a FAQ. FreeBSD's userland threads library runs within >a single process, so it's not scheduled over multiple CPUs. This is >very efficient for UP systems, but not optimal for SMP machines. > >The LinuxThreads port is an example of rfork()-bases threads (it's a >native port of the Linux code) which may be more efficient on SMP for >your application. LinuxThreads is also far from optimal - a project is >underway to Do Kernel-Supported Threading Right (see the list archives >or www.freebsd.org/~jasone/kse) but won't be ready before 6.0-RELEASE, >probably. > >Kris To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Feb 14 19:49:46 2001 Delivered-To: freebsd-smp@freebsd.org Received: from gtei1.bellatlantic.net (gtei1.bellatlantic.net [199.45.40.145]) by hub.freebsd.org (Postfix) with ESMTP id AD6C037B491 for ; Wed, 14 Feb 2001 19:49:43 -0800 (PST) Received: from me-513q3sc0zun0.pengar.com (adsl-64-223-148-61.mannh.adsl.bellatlantic.net [64.223.148.61]) by gtei1.bellatlantic.net (8.9.1/8.9.1) with ESMTP id WAA24523 for ; Wed, 14 Feb 2001 22:49:55 -0500 (EST) Message-Id: <5.0.2.1.0.20010214223500.01b83950@hobbiton.shire.net> X-Sender: seth-pc@hobbiton.shire.net X-Mailer: QUALCOMM Windows Eudora Version 5.0.2 Date: Wed, 14 Feb 2001 22:46:41 -0500 To: freebsd-smp@FreeBSD.org From: Seth Leigh Subject: What about Solaris? Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org I know I have been subscribed to this list for years, but as a non-contributer to FreeBSD (at least not yet) I don't always pay attention to it. I do know that in the past I have seen some posts where people dissed Solaris, calling it Slowaris, posting benchmarks showing Solaris systems being outperformed by the same machine running Linux, FreeBSD, or what have you. I am just curious what people think about Solaris. I wouldn't be surprised to find that a single-cpu Solaris system would be outperformed by the same level of hardware running FreeBSD or Linux. For heaven's sake, Solaris has been architected for extreme scalability, including very fine granularity of locking within the kernel, preemptible kernel, interupt threads, etc. With all that lock overhead, I don't doubt the uniprocessor Solaris machine sacrifices some performance for a uniform code base. Didn't I just read that FreeBSD-SMP is just about ready actually to have some kernel code no longer protected by the Big Giant Lock? My guess would be that Solaris could likely be outperformed on a single or possible even a dual-cpu machine running such a non-scalable OS. My guess is that if you slap FreeBSD-SMP, Linux, or what have you, on a 4-way box you will start to see Solaris overtake the other ones. And if you can find an 8-way box that can run FreeBSD or Linux and Solaris, I think I would have to be very surprised if the Solaris system didn't spank the other two. Consider that Solaris works fine on up to a 64-cpu machine right now, and machines with more cpus are probably right around the corner. People run Solaris on machines with 12, or 24, or whatever, cpus all the time, and it works great. I guess I have just been doing a lot of thinking about Solaris recently because I have been reading the most excellent book "Solaris Internals: Core Kernel Architecture" by Jim Mauro and Richard McDougall. I have been extremely impressed by what appears to my fairly-new-to-operating-system-theory mind to be a very well thought out, well implemented, highly scaleable architecture. And then I think back to the posts I have seen in the past where folks raked Solaris over the coals, and I have to wonder, what is it about Solaris that people don't like? What technically bothers someone about Solaris as opposed to something like Linux or FreeBSD? This isn't supposed to be a troll post, I am honestly interested in what people think. Seth Leigh To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Feb 14 19:54:21 2001 Delivered-To: freebsd-smp@freebsd.org Received: from obsecurity.dyndns.org (adsl-63-207-60-67.dsl.lsan03.pacbell.net [63.207.60.67]) by hub.freebsd.org (Postfix) with ESMTP id 8FE2C37B503 for ; Wed, 14 Feb 2001 19:54:19 -0800 (PST) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id EDE6F66B26; Wed, 14 Feb 2001 19:54:18 -0800 (PST) Date: Wed, 14 Feb 2001 19:54:18 -0800 From: Kris Kennaway To: Seth Leigh Cc: freebsd-smp@FreeBSD.org Subject: Re: possible problem with SMP? Message-ID: <20010214195418.A79489@mollari.cthul.hu> References: <20010214191110.D78224@mollari.cthul.hu> <5.0.2.1.0.20010214222040.00aac5c0@hobbiton.shire.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-md5; protocol="application/pgp-signature"; boundary="azLHFNyN32YCQGCU" Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <5.0.2.1.0.20010214222040.00aac5c0@hobbiton.shire.net>; from seth@pengar.com on Wed, Feb 14, 2001 at 10:34:57PM -0500 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org --azLHFNyN32YCQGCU Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Feb 14, 2001 at 10:34:57PM -0500, Seth Leigh wrote: > To be honest, if FreeBSD can just get a threads library that uses a One t= o=20 > One ratio of threads to lwps (I am quite ignorant as to whether FreeBSD h= as=20 > the equivelant of the LWP structure, so just substitute process or whatev= er=20 > if I am off) I would be quite happy, and I am sure that performance would= =20 > be just fine. See the LinuxThreads port already mentioned. > I would be very much interested to see just how FreeBSD intends to Do=20 > Threading Right. See the webpage already mentioned. Kris --azLHFNyN32YCQGCU Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.4 (FreeBSD) Comment: For info see http://www.gnupg.org iD8DBQE6i1LqWry0BWjoQKURApbrAJ9eo7J84x2ZTOOQyqm6mb0R1aSo3ACfdBd1 0m/HCwWC9jEd7xLqhnlxneQ= =7ukB -----END PGP SIGNATURE----- --azLHFNyN32YCQGCU-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Feb 14 21: 4:44 2001 Delivered-To: freebsd-smp@freebsd.org Received: from gtei1.bellatlantic.net (gtei1.bellatlantic.net [199.45.40.145]) by hub.freebsd.org (Postfix) with ESMTP id 3FE9737B401 for ; Wed, 14 Feb 2001 21:04:41 -0800 (PST) Received: from me-513q3sc0zun0.pengar.com (adsl-64-223-148-61.mannh.adsl.bellatlantic.net [64.223.148.61]) by gtei1.bellatlantic.net (8.9.1/8.9.1) with ESMTP id AAA13696 for ; Thu, 15 Feb 2001 00:04:47 -0500 (EST) Message-Id: <5.0.2.1.0.20010214234050.026959f8@hobbiton.shire.net> X-Sender: seth-pc@hobbiton.shire.net X-Mailer: QUALCOMM Windows Eudora Version 5.0.2 Date: Thu, 15 Feb 2001 00:01:30 -0500 To: freebsd-smp@FreeBSD.org From: Seth Leigh Subject: Re: possible problem with SMP? In-Reply-To: <20010214195418.A79489@mollari.cthul.hu> References: <5.0.2.1.0.20010214222040.00aac5c0@hobbiton.shire.net> <20010214191110.D78224@mollari.cthul.hu> <5.0.2.1.0.20010214222040.00aac5c0@hobbiton.shire.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org OK, having read this, I am still not sure of how this scheme is any better than the way it is done, for example, in Solaris, with the exception that this scheme will prevent a single process from competing unfairly with other processes simply by creating a large number of threads. To be honest, if assuring that individual processes compete with each other fairly for cpu time irrespective of the number of threads created is very important, then running processes under the Solaris Resource Manager scheduling class under different Lnodes sounds like a perfectly reasonable way to do it. In fact, I would imagine that it would be possible simply to implement a new scheduling class that eliminates the unfair competition between processes but avoids most of the complexity of the Solaris Resource Manager simply by calculating the priorities of all of the runnable lwps on the system such that no single process' lwps get more time on the cpus in general than is fair. Fair in this case could simply be that no single process gets more time than any other process which is able to use the same number of cpus. Ie: two processes with two threads would get equal time, a process with 4 threads would get around twice as much time as one with only 2 threads on a 4-way machine, or a process with 4 threads would get around four times as much time as one with 1 thread. For a 4-way machine, the scheme could simply make sure that no process got more than 4 times as much cpu time as any other process, even if one process had 20 threads and the other had only 1. Solaris allows you to create such a scheduling class as a loadable module that would do things just as fairly as your mind can dream up. In fact, it was a 3rd party that apparently wrote the Solaris Resource Manager scheduling class as a commercial product, and Sun liked it so much they bought it. Anyhow, perhaps I just haven't seen the light. I should probably go back and re-read the paper on the web site, since I just don't really see how this is "The Right Way" over how Solaris does this, for example. One question I have is this. Since I assume you aren't going to be messing with the system clock interupts, but you seem to be adding your own timers to the UTS to handle preemption at the user level, aren't you guys adding more wasted overhead dealing with timer interupts? Ie: you keep all the existing kernel-level timer interupts, but add a lot of timer signals and such going up to the UTS? At least with a One to One scheduling model, you only have as much timer-related overhead as the system currently has, and with per-process run queues, fine lock granularity throughout the system allowing for multiple cpus to be running in the scheduler at the same time, etc., letting the kernel scheduler do all the work of scheduling the lwps doesn't seem to be too bad. I don't see the logic in the argument "letting the kernel scheduler handle the scheduling of all the threads (via lwps) is bad, because it overloads the kernel scheduler, so we are going to do all of this thread scheduling using a user-level thread scheduler". If you can write a user-level thread scheduler that won't get "overloaded" by a large number of threads, why can't you just write the kernel-level scheduler the same way? I suppose if the answer is that the existing FreeBSD scheduler is just too clumsy to provide good scalability over multiple cpus and a large number of runnable lwps, then wouldn't the first job be to improve the scheduler, rather than simply try to shift all the load into a UTS? I look forward to being enlightened. Seth At 07:54 PM 2/14/2001 -0800, you wrote: >On Wed, Feb 14, 2001 at 10:34:57PM -0500, Seth Leigh wrote: > > > To be honest, if FreeBSD can just get a threads library that uses a One to > > One ratio of threads to lwps (I am quite ignorant as to whether FreeBSD > has > > the equivelant of the LWP structure, so just substitute process or > whatever > > if I am off) I would be quite happy, and I am sure that performance would > > be just fine. > >See the LinuxThreads port already mentioned. > > > I would be very much interested to see just how FreeBSD intends to Do > > Threading Right. > >See the webpage already mentioned. > >Kris To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Feb 14 21:29:34 2001 Delivered-To: freebsd-smp@freebsd.org Received: from mail.numachi.com (numachi.numachi.com [198.175.254.2]) by hub.freebsd.org (Postfix) with SMTP id 30B3637B491 for ; Wed, 14 Feb 2001 21:29:30 -0800 (PST) Received: (qmail 21926 invoked by uid 3001); 15 Feb 2001 05:29:29 -0000 Received: from natto.numachi.com (198.175.254.216) by numachi.numachi.com with SMTP; 15 Feb 2001 05:29:29 -0000 Received: (qmail 34280 invoked by uid 1001); 15 Feb 2001 05:29:29 -0000 Date: Thu, 15 Feb 2001 00:29:29 -0500 From: Brian Reichert To: Seth Leigh Cc: freebsd-smp@FreeBSD.org Subject: Re: What about Solaris? Message-ID: <20010215002929.X91352@numachi.com> References: <5.0.2.1.0.20010214223500.01b83950@hobbiton.shire.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <5.0.2.1.0.20010214223500.01b83950@hobbiton.shire.net>; from seth@pengar.com on Wed, Feb 14, 2001 at 10:46:41PM -0500 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Wed, Feb 14, 2001 at 10:46:41PM -0500, Seth Leigh wrote: > And then I think back to the posts I have seen in the past where folks > raked Solaris over the coals, and I have to wonder, what is it about > Solaris that people don't like? What technically bothers someone about > Solaris as opposed to something like Linux or FreeBSD? Other than a slick threading model (and I like truss a great deal), I dare say my biggest problems with Solaris from both a sysadmin point of view, and a software engineer point of view, is the sheer amount of _cruft_ that you are saddled with, out-of-the-box. (Hint: look at how many versions of Bourne shell a Solaris box has on it.) (I don't know if that counts as 'technical', though, as per your question.) Couple that with the classic issues of open-source vs. not, and how that affects your ability to solve esoteric problems, and you'll start to get a feel for how 'useful' Solaris can be. > This isn't supposed to be a troll post, I am honestly interested in what > people think. > > Seth Leigh > > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-smp" in the body of the message > -- Brian 'you Bastard' Reichert 37 Crystal Ave. #303 Daytime number: (603) 434-6842 Derry NH 03038-1713 USA Intel architecture: the left-hand path To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Feb 14 21:37:17 2001 Delivered-To: freebsd-smp@freebsd.org Received: from ns.yogotech.com (ns.yogotech.com [206.127.123.66]) by hub.freebsd.org (Postfix) with ESMTP id D110F37B4EC for ; Wed, 14 Feb 2001 21:37:14 -0800 (PST) Received: from nomad.yogotech.com (nomad.yogotech.com [206.127.123.131]) by ns.yogotech.com (8.9.3/8.9.3) with ESMTP id WAA12300; Wed, 14 Feb 2001 22:37:13 -0700 (MST) (envelope-from nate@nomad.yogotech.com) Received: (from nate@localhost) by nomad.yogotech.com (8.8.8/8.8.8) id WAA04286; Wed, 14 Feb 2001 22:37:12 -0700 (MST) (envelope-from nate) From: Nate Williams MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <14987.27399.662691.291906@nomad.yogotech.com> Date: Wed, 14 Feb 2001 22:37:11 -0700 (MST) To: Seth Leigh Cc: freebsd-smp@FreeBSD.ORG Subject: Re: possible problem with SMP? In-Reply-To: <5.0.2.1.0.20010214234050.026959f8@hobbiton.shire.net> References: <5.0.2.1.0.20010214222040.00aac5c0@hobbiton.shire.net> <20010214191110.D78224@mollari.cthul.hu> <5.0.2.1.0.20010214234050.026959f8@hobbiton.shire.net> X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid Reply-To: nate@yogotech.com (Nate Williams) Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > I don't see the logic in the argument "letting the kernel scheduler handle > the scheduling of all the threads (via lwps) is bad, because it overloads > the kernel scheduler, so we are going to do all of this thread scheduling > using a user-level thread scheduler". That's not the only thing. IMO, the single-biggest gain from a many-many mapping is that 'userland threads' are much lighter weight than any kernel thread doing context switching. The kernel has alot more context it must keep track of, and the fact of the matter is that in many cases, there is no need for multiple 'kernel' threads of execution. For a good example of this, I'll use Solaris itself. JDK1.1.8 using Green Threads and the stock JIT can often *blow the doors off* the same setup using kernel threads in many applications. From talks given at JavaOne, this is primarily because of the 'context switching' overhead of using kernel threads. (Sun's JDK uses a one-one mapping, because Java has no way of intelligently determining which threads should stay in the userland, vs. which need to own a 'kernel' context.) However, it turns out that it's *really easy* to blow yourself up and cause massive slowdowns if your Java application calls certain kind of I/O processes (in particular, read directly from stdin, or writing to files). Because there is no way to determine if a thread is going to do this kind of thing a-priori, Sun changed their default threading model in subsuquent versions of the JDK to use kernel threads. In most cases, the people won't see a measurable difference, but in certain cases they no longer see big 'pauses' due to syscalls that previously blocked all running threads. However, if performance is a goal, a properly written Java application can run faster using the 'many-one' model, where 'many threads' are mapped to a single kernel thread. Unfortunately, because Java doesn't support the 'many-many' model, you can't really choose the appropriate context for your thread to run in, and make Java run even better on your hardware. Kernel context switches are simply expensive, and even if they aren't as expensive, simply moving back-forth into the kernel is expensive because of the safety precautions that must be taken if you want the system to be robust. Nate To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Feb 14 21:44:19 2001 Delivered-To: freebsd-smp@freebsd.org Received: from obsecurity.dyndns.org (adsl-63-207-60-67.dsl.lsan03.pacbell.net [63.207.60.67]) by hub.freebsd.org (Postfix) with ESMTP id F3F9C37B401 for ; Wed, 14 Feb 2001 21:44:16 -0800 (PST) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 9BB1866B26; Wed, 14 Feb 2001 21:44:16 -0800 (PST) Date: Wed, 14 Feb 2001 21:44:16 -0800 From: Kris Kennaway To: Seth Leigh Cc: freebsd-smp@FreeBSD.org Subject: Re: possible problem with SMP? Message-ID: <20010214214416.A80965@mollari.cthul.hu> References: <5.0.2.1.0.20010214222040.00aac5c0@hobbiton.shire.net> <20010214191110.D78224@mollari.cthul.hu> <5.0.2.1.0.20010214222040.00aac5c0@hobbiton.shire.net> <20010214195418.A79489@mollari.cthul.hu> <5.0.2.1.0.20010214234050.026959f8@hobbiton.shire.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-md5; protocol="application/pgp-signature"; boundary="LQksG6bCIzRHxTLp" Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <5.0.2.1.0.20010214234050.026959f8@hobbiton.shire.net>; from seth@pengar.com on Thu, Feb 15, 2001 at 12:01:30AM -0500 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org --LQksG6bCIzRHxTLp Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Feb 15, 2001 at 12:01:30AM -0500, Seth Leigh wrote: > Anyhow, perhaps I just haven't seen the light. I should probably go back= =20 > and re-read the paper on the web site, since I just don't really see how= =20 > this is "The Right Way" over how Solaris does this, for example. Read the references too - they specifically compare the 1-1 to scheduler activation models. This has been well researched (both in the CS literature and by by FreeBSD in coming up with a design), and people who have done previous implementations found good performance benefits over the 1-1 model. Kris --LQksG6bCIzRHxTLp Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.4 (FreeBSD) Comment: For info see http://www.gnupg.org iD8DBQE6i2ywWry0BWjoQKURAtZUAJ9cfjJiOdInm1TKkVF4DaRUP/sCTQCfSXl7 qPpThUhQs+QYZCmkOrYj/ow= =aqZe -----END PGP SIGNATURE----- --LQksG6bCIzRHxTLp-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Feb 14 23: 9: 3 2001 Delivered-To: freebsd-smp@freebsd.org Received: from smtp04.primenet.com (smtp04.primenet.com [206.165.6.134]) by hub.freebsd.org (Postfix) with ESMTP id 5618D37B401 for ; Wed, 14 Feb 2001 23:08:54 -0800 (PST) Received: (from daemon@localhost) by smtp04.primenet.com (8.9.3/8.9.3) id AAA04377; Thu, 15 Feb 2001 00:03:25 -0700 (MST) Received: from usr08.primenet.com(206.165.6.208) via SMTP by smtp04.primenet.com, id smtpdAAAH4aGEi; Thu Feb 15 00:03:14 2001 Received: (from tlambert@localhost) by usr08.primenet.com (8.8.5/8.8.5) id AAA19435; Thu, 15 Feb 2001 00:08:32 -0700 (MST) From: Terry Lambert Message-Id: <200102150708.AAA19435@usr08.primenet.com> Subject: Re: possible problem with SMP? To: seth@pengar.com (Seth Leigh) Date: Thu, 15 Feb 2001 07:08:32 +0000 (GMT) Cc: freebsd-smp@FreeBSD.ORG In-Reply-To: <5.0.2.1.0.20010214234050.026959f8@hobbiton.shire.net> from "Seth Leigh" at Feb 15, 2001 12:01:30 AM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org I'm certainly not a spokesperson for this, but I'm very well versed on threads, and participated in the design discussions for the FreeBSD and (while at USL) SVR4 threading, as well as the Usenet discussions on the Solaris 2.5 threading and beyond. I guess that entitles me to express myself as much as the next guy... > OK, having read this, I am still not sure of how this scheme is any better > than the way it is done, for example, in Solaris, with the exception that > this scheme will prevent a single process from competing unfairly with > other processes simply by creating a large number of threads. You need to consider that threads were originally designed to reduce context switch overhead. SMP scaling is merely a side effect of some particular implementations. The benefit is that it in fact delivers on the context switch overhead promise. In a design where the threads are independently scheduled, the probability of getting another thread in the same process is proportional to the number of threads in the process (N) vs. the total number of threads (M) is (N-1):(M). that means that as a system becomes heavily loaded, (M) -> (M)>>(N-1), and the absolute probability value (N-1)/(M) approaches zero percent. On the other hand, where quantum is remaining, an activations (or async system call) implementation, the probablity remains 1:1 ...100%. The beauty of this increases, when you realize that the method of implementation has the same SMP scalability as that of the Solaris approach. Solaris 2.8 comes much closer to achieving the same ends, but still has the same overhead scaling issues for N:M (N threads on M CPUs, in this case, even if you optimally limit a program to M scheduling objects regardless of how big N gets). > To be honest, if assuring that individual processes compete with > each other fairly for cpu time irrespective of the number of > threads created is very important, then running processes under > the Solaris Resource Manager scheduling class under different > Lnodes sounds like a perfectly reasonable way to do it. In fact, > I would imagine that it would be possible simply to implement a > new scheduling class that eliminates the unfair competition > between processes but avoids most of the complexity of the Solaris > Resource Manager simply by calculating the priorities of all of > the runnable lwps on the system such that no single process' lwps > get more time on the cpus in general than is fair. Frankly, I think that SVR4 style scheduling classes are a very useful feature for an OS to have. However, the primary goals of the design are not to enforce fairness, though permitting it to be enforced was certainly a consideration, in the discussions I attended. Historically, SVR4 has had a tendency to use scheduling classes in the classical "if the only tool you have is a hammer..." way; in particular, the idea of a fixed scheduling class was implemented because a badly behaved process could thrash the buffer cache, forcing the X server pages out of core, to the point of the X server being unusable as an interface. This broke the cognitive tactile/visual association that made the interface work (or what Ed Lane, a friend of mine, and one of the people who complained loadly until it was at least worked around, called "move mouse, wiggle cursor". The fixed scheduling class addressed this by providing the X server with sufficient quanta in any set of K quanta to thrash its own pages back in: a workaround, since the OS wastes much time page thrashing, which it would not waste, if the problem were addressed directly (e.g. with a per vnode working set quota). Unfortunately, non-kernel methods of addressing the problem are unlikely to be successful; the linker was one of the main offending programs: it mapped object files, and seeked around in them as if they were memory. Changing the linker behaviour (and the behaviour of all other offendors) was not really a viable option (a working set quota on the vnode would have forced it to steal pages from itself, instead of from the LRU lists of swapped out processes, once it hit the working set size limit). As a side note to this tangent, it's pretty amusing to note that the most recent Solaris release "reinvents" the buffer cache, and deunifies it once again, to get around heavy usage starvation problems, which could just as easily have been fixed with those same working set quotas, or, as FreeBSD has recently started doing, by maintaining purpose-dedicated minimum reserves. In any case, fairness is a benefit, and permitting enforcement of fairness was a design goal, but I don't look at it as a primary design criteria, and you probably shouldn't, either, when trying to compare implementations. > Anyhow, perhaps I just haven't seen the light. I should probably go back > and re-read the paper on the web site, since I just don't really see how > this is "The Right Way" over how Solaris does this, for example. I really recommend that you do. Read it in the context of the idea that the CPU should be spending as much time as possible in user space, and that time spent in kernel space on context switches or I/O waits is nothing but overhead. At the same time, there are some subtleties that are easy to overlook that you should also keep in mind. I think the number one of these is that a threaded program that should be a threaded program, philosophically speaking, will not have a lot of inter-thread contention. In other words, seperate threads have seperate locality of reference for data, and most reasources are not contended. Again, the ability to communicate in a single address space among several threads is a side effect of implementation, just as SMP scalability turned out to be a side effect (even if a desirable one). If you look at the NT and 95/98 threads implementations, you'll see that they make heavy usage of thread local storage to instnace COM objects, and that objects instanced after the creation of a new thread do not exist in (are not mapped into) the address space of the creating thread, without explicitly marshalling the instance data (anyone who has tried to use the LDAP API in a multithreaded environment under the Microsoft OS and toolchain, can attest that the connection created in one thread is not automatically marshalled; the technical reason is that the DLL of the created library doesn't have thread_attach and thread_detach entry points which force the data marshalling to occur transparently). The address space sharing, when and if it exists at all, is an artifact of the implementation. To put this into perspective, note that most SMP scaling for MESI architecture shared memory multiprocessing is anecdotally held to "fall off" and become "below the point of diminishing returns" at only 4 processors. Some implementations will claim 8 processors, and many papers will claim that NUMA and similar compromise data-flow architectures, which can't support true symmetry, are required to exceed 4 processors. Yet in the early 90s, Sequent manufactured both the Symmetry and Balance series of servers (680x0 and 80[4|5]86), and were able to support MESI cache coherency, with continuing returns for 32 processors in a single box. What this says to me that the assumption that a certain amount of IPI overhead "is inevitable" is really based on assumptions that may be valid about the architecture of SVR4 ES/MP or SMP SVR4.2, but which were not valid for Dynix and other SMP and MP OSs of the time. In other words, contention for shared resourvces is not so much an artifact of SMP, as it is of a particular SMP implementation. Part of that has to be the threads library, which is intended, at least these days, to provide SMP scalability for applications, instead of merely increasing utilized quantum and I/O concurrency (the original reasons for threads, and "liblwp" on SunOs 4.x: a call conversion based user space threads library, which was not SMP scalable, the basis of the University of Washington "User Space Threads and SPARC Register Windows" paper). > One question I have is this. Since I assume you aren't going to be messing > with the system clock interupts, but you seem to be adding your own timers > to the UTS to handle preemption at the user level, aren't you guys adding > more wasted overhead dealing with timer interupts? Scheduler activations are not timer interrupts. And they are only created in the context of blocking calls. While my personal preference would be to put all blocking call POSIX semantics into a library, and make all system calls non-blocking, I do understand the compatability issue that lead to the use of the more complicated activations framework. Really, you should read the "Scheduler Activations" paper, which discusses these issues in detail. If you look under the covers of the cooperative kernel and user space scheduler in Solaris 2.8, I think you'll find a number of similarities. There is some additional protection domain crossing -- actually, two calls per activation -- that would not be there with a fully async system call interface, with call contexts dual-mapped in both user and kernel space, but when you compare that overhead to a high probability of a heavy weight context switch, the benefits should be obvious. Note: The only place this will not be true will be on a system with an intentionally homogeneous load, to ensure that all of the contention will be between threads in a single process. It is my experience, and, I think, the general consensus, that this does not represent a real world load that one could expect on any system capable of doing useful work. To be CPU bound enough to require multiple processors in the first place, in this I/O bound world we live in today, really means non-homogeneaity of the code competing for quanta. > I don't see the logic in the argument "letting the kernel scheduler handle > the scheduling of all the threads (via lwps) is bad, because it overloads > the kernel scheduler, so we are going to do all of this thread scheduling > using a user-level thread scheduler". The issue is one of who is responsible for non-default policy implementation. If you put this in the kernel, then you make the code arbitrarily complex. The problem you are then faced with is how to institute CPU affinity for threads in a process, and how to implement threads group affinity to prefer threads in one process, when a kernel sleep (voluntary context switch) takes place? Almost any way you cut it, it is nearly impossible to implement thread group affinity in a scheduler, once something is already on a scheduling queue, without leading to starvation of threads not in the threads group. The closest approach (as is described in "UNIX for Modern Architectures" is a two queue, two handed clock algorithm (effectively, a three handed clock, with preferential queue insertion of ready-to-run "LWPs", to use Sun terminology). This makes the scheduler incredibly bursty with regard to your threaded process, and at the same time turns it into a fair-share scheduler, meaning that it ignores priorities when trying to decide what to run next. > If you can write a user-level thread scheduler that won't get > "overloaded" by a large number of threads, why can't you just > write the kernel-level scheduler the same way? I suppose if > the answer is that the existing FreeBSD scheduler is just too > clumsy to provide good scalability over multiple cpus and a > large number of runnable lwps, then wouldn't the first job be to > improve the scheduler, rather than simply try to shift all the > load into a UTS? The answer is that it's not possible to build a scheduler that implements priority based fairness, when user processes are permitted to request an unbounded amount of quantum. The best you can do is implement quantum allocation fairness, and then implement a policy based decision on which thread in a group of threads will get a quantum, once it is assigned to a process. Technically, it's possible to implement this all in the kernel, but it significantly and unnecesssarily increases the complexity. In the case of calls which can be run to completion without blocking in the kernel, it also increases the amount of protection domain crossing. Again, you want to think about how you can spend the most time you can possibly spend, actually running user code, instead of operating system. In terms of "fairness", accounting scheduler overhead for implementation of policy decisions against the process which initiated the need for such overhead, is eminently fair. If you want to look at the simplest case, a multithreaded application running on a uniprocessor box, the lowest possible overhead is obtained by staying in user space -- not crossing the protection domain at all -- and context switching between threads within the same process, in user space, for as long as the quantum holds out. Obviously, aio calls beat non-blocking calls, since a non-blocking call that would block results in a fault or other kernel operation occurring, and returning to the caller with EWOULDBLK, completing sucessfully the next time it is attempted (an extra protection domain crossing, compared to AIO), but any approach beats context switching to a different competing process, and losing the remainder of your quantum. Ask yourself how to make this work with the SMP side effect of some implementations becoming a design goal instead of an unintended side effect. It's only when you try to make things complicated, by adding other processors, and then insisting that they be additive in a single multithreaded program, instead of two programs, that things start to get complicated. And then you have to remember your first principles: why you had threads in the first place, and not how you are now able to abuse them to achieve primitive and generally grossly inefficient SMP scaling. If you want to achieve SMP scalability, at least be honest; and design to achieve it efficiently as possible, not as a desirable side effect of the abuse of a technology that _happens_ to result in _some_ scalability. Another good reference in this regard, by the way, is the Linus response to more recent Microsoft vs. Linux benchmarks, in which he admits scalability problems in Linux (which uses the original SVR4 kernel threading model, pretty much unaltered), and states with regard to early claims of benchmark bias "...we were arrogant..." (referring to the fact that MS OSs can in fact beat Linux fairly, in some benchmarks). Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Feb 14 23:24:46 2001 Delivered-To: freebsd-smp@freebsd.org Received: from gtei2.bellatlantic.net (gtei2.bellatlantic.net [199.45.40.146]) by hub.freebsd.org (Postfix) with ESMTP id AA2B037B401 for ; Wed, 14 Feb 2001 23:24:40 -0800 (PST) Received: from me-513q3sc0zun0.pengar.com (adsl-64-223-148-61.mannh.adsl.bellatlantic.net [64.223.148.61]) by gtei2.bellatlantic.net (8.9.1/8.9.1) with ESMTP id CAA24280; Thu, 15 Feb 2001 02:24:29 -0500 (EST) Message-Id: <5.0.2.1.0.20010215020051.01b9fc70@hobbiton.shire.net> X-Sender: seth-pc@hobbiton.shire.net X-Mailer: QUALCOMM Windows Eudora Version 5.0.2 Date: Thu, 15 Feb 2001 02:21:14 -0500 To: nate@yogotech.com (Nate Williams) From: Seth Leigh Subject: Re: possible problem with SMP? Cc: freebsd-smp@FreeBSD.ORG In-Reply-To: <14987.27399.662691.291906@nomad.yogotech.com> References: <5.0.2.1.0.20010214234050.026959f8@hobbiton.shire.net> <5.0.2.1.0.20010214222040.00aac5c0@hobbiton.shire.net> <20010214191110.D78224@mollari.cthul.hu> <5.0.2.1.0.20010214234050.026959f8@hobbiton.shire.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org There are problems of cpu starvation with a Many to Many model in that usually (unless you manually step up the concurrency) the Many to Many ends up being Many to Number of CPUs. If Number of CPUs is equal to 1, then usually you will only get one LWP beyond the ones that are sleeping. Using the Solaris libpthread, a user-level thread will not give up the LWP it's using until it voluntarily makes a call into libthread which gives libthread an opportunity to deschedule that thread and schedule a different one. To illustrate this, I talked to a customer the other day whose application had around 200 threads. He had around 40 LWPs created, but 39 of them were blocked in the kernel, and 1 appeared to be running when looking at the threads in dbx. The application appeared to be hung, though. Finally he figured out that the one thread that was actually running on the one free LWP was in fact running a hugely inefficient code chunk that was simply taking forever to complete, and since it wasn't calling into libthread, it was hogging that cpu and the other 160 or so threads in the program simply were not getting *any* time whatsoever. The solution to this was either to attempt to set the concurrency manually to something like "a few" LWPs higher than the number he expected to block, or throw in some sched_yield()s into all of his threads to make sure others got some time, or else simply make them bound. I advised him to just make them bound, run some tests to see how this affected his performance, and then decide from there. In all likelihood the majority of his threads can and do block at some point, and I see very little reason if that is the case why he shouldn't just make his threads bound. Daniel Berg and Bil Lewis, in their threads book, seem to agree. I am starting to believe, based on the number of customers I have talked to writing threaded code, that most real-world applications that use threads use threads that sometimes block in the kernel. That being the case, I honestly see no real reason not to make them bound. Fact is, once they block, they cause a new LWP to be created anyways, and then basically if all your threads block at least once in a relatively short amount of time you end up with most of your threads having their "own" LWP anyhow, plus you have the additional overhead of the more complicated thread library. In short, for many if not most real-world applications, user-level threads don't really buy you anything. As for information going from the kernel to the threads library, I read about the callups in that paper at the FreeBSD web site. I really like the idea of using Solaris Doors for that sort of thing. Not only do you get the full benefit of direct callup into the application with the door server, but you have a generalized callup interface that isn't hard-wired to work just a few different ways in the kernel, as the threads-specific callups seem to be planned to do. This could pay off in a big way, as the FreeBSD team found other places where Doors could really improve things. Are the FreeBSD developers who are planning and designing this whole thing aware of what Solaris Doors are, and how the idea of doors might be used for FreeBSD? But basically, the question of Many to One, One to One, Many to Many, etc. really comes down to a question of how complex the library will be, and how threads will typically be used. I maintain, without proof, that the case of pure user-level threads that don't block at all, that really benefit from the fast user-level context switching, is not all that frequent in the real world. Something in my mind likes the idea of only having one set of scheduler code in a system. Not multiple sets, such as a UTS plus a kernel scheduler. Simplicity can be a virtue. I will have to write some more code at work and test it against both /usr/lib/libthread.so.1 and /usr/lib/lwp/libthread.so.1 and see what I can measure in terms of timing. Seth At 10:37 PM 2/14/2001 -0700, Nate Williams wrote: > > I don't see the logic in the argument "letting the kernel scheduler handle > > the scheduling of all the threads (via lwps) is bad, because it overloads > > the kernel scheduler, so we are going to do all of this thread scheduling > > using a user-level thread scheduler". > >That's not the only thing. IMO, the single-biggest gain from a >many-many mapping is that 'userland threads' are much lighter weight >than any kernel thread doing context switching. The kernel has alot >more context it must keep track of, and the fact of the matter is that >in many cases, there is no need for multiple 'kernel' threads of >execution. > >For a good example of this, I'll use Solaris itself. JDK1.1.8 using >Green Threads and the stock JIT can often *blow the doors off* the same >setup using kernel threads in many applications. From talks given at >JavaOne, this is primarily because of the 'context switching' overhead >of using kernel threads. (Sun's JDK uses a one-one mapping, because >Java has no way of intelligently determining which threads should stay >in the userland, vs. which need to own a 'kernel' context.) > >However, it turns out that it's *really easy* to blow yourself up and >cause massive slowdowns if your Java application calls certain kind of >I/O processes (in particular, read directly from stdin, or writing to >files). Because there is no way to determine if a thread is going to do >this kind of thing a-priori, Sun changed their default threading model >in subsuquent versions of the JDK to use kernel threads. In most cases, >the people won't see a measurable difference, but in certain cases they >no longer see big 'pauses' due to syscalls that previously blocked all >running threads. > >However, if performance is a goal, a properly written Java application >can run faster using the 'many-one' model, where 'many threads' are >mapped to a single kernel thread. Unfortunately, because Java doesn't >support the 'many-many' model, you can't really choose the appropriate >context for your thread to run in, and make Java run even better on your >hardware. > >Kernel context switches are simply expensive, and even if they aren't as >expensive, simply moving back-forth into the kernel is expensive because >of the safety precautions that must be taken if you want the system to >be robust. > > > >Nate > > >To Unsubscribe: send mail to majordomo@FreeBSD.org >with "unsubscribe freebsd-smp" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Feb 14 23:59:22 2001 Delivered-To: freebsd-smp@freebsd.org Received: from gtei2.bellatlantic.net (gtei2.bellatlantic.net [199.45.40.146]) by hub.freebsd.org (Postfix) with ESMTP id 0A91C37B491 for ; Wed, 14 Feb 2001 23:59:12 -0800 (PST) Received: from me-513q3sc0zun0.pengar.com (adsl-64-223-148-61.mannh.adsl.bellatlantic.net [64.223.148.61]) by gtei2.bellatlantic.net (8.9.1/8.9.1) with ESMTP id CAA26069; Thu, 15 Feb 2001 02:59:22 -0500 (EST) Message-Id: <5.0.2.1.0.20010215025043.027a3008@hobbiton.shire.net> X-Sender: seth-pc@hobbiton.shire.net X-Mailer: QUALCOMM Windows Eudora Version 5.0.2 Date: Thu, 15 Feb 2001 02:56:07 -0500 To: Terry Lambert From: Seth Leigh Subject: Re: possible problem with SMP? Cc: freebsd-smp@FreeBSD.ORG In-Reply-To: <200102150708.AAA19435@usr08.primenet.com> References: <5.0.2.1.0.20010214234050.026959f8@hobbiton.shire.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org I just read pages 407 to 410 of the Jim Mauro/Richard McDougall "Solaris Internals" book and it talks specifically about scheduler activations. They mention how they are used by the kernel scheduler to keep LWPs from a given process running throughout the rest of their quantum even when one of them blocks, and some other things. They specifically talk about Solaris Doors as being the mechanism by which the kernel calls up into the threads library. I will have to read a lot more in the next day or so about Solaris scheduler activations until I know exactly what they are and what they do. I am still only on page 150 of the book. It's not easy reading. ;-) But I will finish reading it, and then I will read it all the way through again, since I really do want to understand this stuff. Terry, have you looked into the Solaris Doors mechanism, and do you think that implementing a facility like that in FreeBSD would clean up and improve the way FreeBSD plans to implement the threads library upcalls, and other possible places for upcalls as well? Despite my as-yet limited understanding, it sounds promising to me. Seth At 07:08 AM 2/15/2001 +0000, Terry Lambert wrote: >I'm certainly not a spokesperson for this, but I'm very well >versed on threads, and participated in the design discussions >for the FreeBSD and (while at USL) SVR4 threading, as well as >the Usenet discussions on the Solaris 2.5 threading and beyond. >I guess that entitles me to express myself as much as the next >guy... > > > > OK, having read this, I am still not sure of how this scheme is any better > > than the way it is done, for example, in Solaris, with the exception that > > this scheme will prevent a single process from competing unfairly with > > other processes simply by creating a large number of threads. > >You need to consider that threads were originally designed to >reduce context switch overhead. SMP scaling is merely a side >effect of some particular implementations. > >The benefit is that it in fact delivers on the context switch >overhead promise. In a design where the threads are independently >scheduled, the probability of getting another thread in the same >process is proportional to the number of threads in the process (N) >vs. the total number of threads (M) is (N-1):(M). that means that >as a system becomes heavily loaded, (M) -> (M)>>(N-1), and the >absolute probability value (N-1)/(M) approaches zero percent. On >the other hand, where quantum is remaining, an activations (or async >system call) implementation, the probablity remains 1:1 ...100%. > >The beauty of this increases, when you realize that the method >of implementation has the same SMP scalability as that of the >Solaris approach. > >Solaris 2.8 comes much closer to achieving the same ends, but >still has the same overhead scaling issues for N:M (N threads on >M CPUs, in this case, even if you optimally limit a program to >M scheduling objects regardless of how big N gets). > > > > To be honest, if assuring that individual processes compete with > > each other fairly for cpu time irrespective of the number of > > threads created is very important, then running processes under > > the Solaris Resource Manager scheduling class under different > > Lnodes sounds like a perfectly reasonable way to do it. In fact, > > I would imagine that it would be possible simply to implement a > > new scheduling class that eliminates the unfair competition > > between processes but avoids most of the complexity of the Solaris > > Resource Manager simply by calculating the priorities of all of > > the runnable lwps on the system such that no single process' lwps > > get more time on the cpus in general than is fair. > >Frankly, I think that SVR4 style scheduling classes are a very >useful feature for an OS to have. However, the primary goals of >the design are not to enforce fairness, though permitting it to >be enforced was certainly a consideration, in the discussions I >attended. > >Historically, SVR4 has had a tendency to use scheduling classes >in the classical "if the only tool you have is a hammer..." way; >in particular, the idea of a fixed scheduling class was implemented >because a badly behaved process could thrash the buffer cache, >forcing the X server pages out of core, to the point of the X >server being unusable as an interface. This broke the cognitive >tactile/visual association that made the interface work (or what >Ed Lane, a friend of mine, and one of the people who complained >loadly until it was at least worked around, called "move mouse, >wiggle cursor". The fixed scheduling class addressed this by >providing the X server with sufficient quanta in any set of K >quanta to thrash its own pages back in: a workaround, since the >OS wastes much time page thrashing, which it would not waste, if >the problem were addressed directly (e.g. with a per vnode working >set quota). Unfortunately, non-kernel methods of addressing the >problem are unlikely to be successful; the linker was one of the >main offending programs: it mapped object files, and seeked around >in them as if they were memory. Changing the linker behaviour (and >the behaviour of all other offendors) was not really a viable >option (a working set quota on the vnode would have forced it to >steal pages from itself, instead of from the LRU lists of swapped >out processes, once it hit the working set size limit). > >As a side note to this tangent, it's pretty amusing to note that >the most recent Solaris release "reinvents" the buffer cache, >and deunifies it once again, to get around heavy usage starvation >problems, which could just as easily have been fixed with those >same working set quotas, or, as FreeBSD has recently started doing, >by maintaining purpose-dedicated minimum reserves. > >In any case, fairness is a benefit, and permitting enforcement of >fairness was a design goal, but I don't look at it as a primary >design criteria, and you probably shouldn't, either, when trying >to compare implementations. > > > > Anyhow, perhaps I just haven't seen the light. I should probably go back > > and re-read the paper on the web site, since I just don't really see how > > this is "The Right Way" over how Solaris does this, for example. > >I really recommend that you do. Read it in the context of the >idea that the CPU should be spending as much time as possible >in user space, and that time spent in kernel space on context >switches or I/O waits is nothing but overhead. > >At the same time, there are some subtleties that are easy to >overlook that you should also keep in mind. I think the number >one of these is that a threaded program that should be a threaded >program, philosophically speaking, will not have a lot of >inter-thread contention. In other words, seperate threads have >seperate locality of reference for data, and most reasources >are not contended. > >Again, the ability to communicate in a single address space among >several threads is a side effect of implementation, just as SMP >scalability turned out to be a side effect (even if a desirable >one). If you look at the NT and 95/98 threads implementations, >you'll see that they make heavy usage of thread local storage to >instnace COM objects, and that objects instanced after the >creation of a new thread do not exist in (are not mapped into) >the address space of the creating thread, without explicitly >marshalling the instance data (anyone who has tried to use the >LDAP API in a multithreaded environment under the Microsoft OS >and toolchain, can attest that the connection created in one >thread is not automatically marshalled; the technical reason is >that the DLL of the created library doesn't have thread_attach >and thread_detach entry points which force the data marshalling >to occur transparently). The address space sharing, when and if >it exists at all, is an artifact of the implementation. > >To put this into perspective, note that most SMP scaling for >MESI architecture shared memory multiprocessing is anecdotally >held to "fall off" and become "below the point of diminishing >returns" at only 4 processors. Some implementations will claim >8 processors, and many papers will claim that NUMA and similar >compromise data-flow architectures, which can't support true >symmetry, are required to exceed 4 processors. Yet in the early >90s, Sequent manufactured both the Symmetry and Balance series >of servers (680x0 and 80[4|5]86), and were able to support >MESI cache coherency, with continuing returns for 32 processors >in a single box. > >What this says to me that the assumption that a certain amount >of IPI overhead "is inevitable" is really based on assumptions >that may be valid about the architecture of SVR4 ES/MP or SMP >SVR4.2, but which were not valid for Dynix and other SMP and >MP OSs of the time. In other words, contention for shared >resourvces is not so much an artifact of SMP, as it is of a >particular SMP implementation. > >Part of that has to be the threads library, which is intended, >at least these days, to provide SMP scalability for applications, >instead of merely increasing utilized quantum and I/O concurrency >(the original reasons for threads, and "liblwp" on SunOs 4.x: a >call conversion based user space threads library, which was not >SMP scalable, the basis of the University of Washington "User >Space Threads and SPARC Register Windows" paper). > > > > One question I have is this. Since I assume you aren't going to be > messing > > with the system clock interupts, but you seem to be adding your own timers > > to the UTS to handle preemption at the user level, aren't you guys adding > > more wasted overhead dealing with timer interupts? > >Scheduler activations are not timer interrupts. And they are only >created in the context of blocking calls. While my personal >preference would be to put all blocking call POSIX semantics >into a library, and make all system calls non-blocking, I do >understand the compatability issue that lead to the use of the >more complicated activations framework. > >Really, you should read the "Scheduler Activations" paper, which >discusses these issues in detail. If you look under the covers >of the cooperative kernel and user space scheduler in Solaris 2.8, >I think you'll find a number of similarities. > >There is some additional protection domain crossing -- actually, >two calls per activation -- that would not be there with a fully >async system call interface, with call contexts dual-mapped in >both user and kernel space, but when you compare that overhead >to a high probability of a heavy weight context switch, the >benefits should be obvious. > >Note: The only place this will not be true will be on a system >with an intentionally homogeneous load, to ensure that all of the >contention will be between threads in a single process. It is my >experience, and, I think, the general consensus, that this does >not represent a real world load that one could expect on any >system capable of doing useful work. To be CPU bound enough to >require multiple processors in the first place, in this I/O bound >world we live in today, really means non-homogeneaity of the code >competing for quanta. > > > > I don't see the logic in the argument "letting the kernel scheduler handle > > the scheduling of all the threads (via lwps) is bad, because it overloads > > the kernel scheduler, so we are going to do all of this thread scheduling > > using a user-level thread scheduler". > >The issue is one of who is responsible for non-default policy >implementation. If you put this in the kernel, then you make >the code arbitrarily complex. The problem you are then faced >with is how to institute CPU affinity for threads in a process, >and how to implement threads group affinity to prefer threads >in one process, when a kernel sleep (voluntary context switch) >takes place? Almost any way you cut it, it is nearly impossible >to implement thread group affinity in a scheduler, once something >is already on a scheduling queue, without leading to starvation >of threads not in the threads group. The closest approach (as is >described in "UNIX for Modern Architectures" is a two queue, two >handed clock algorithm (effectively, a three handed clock, with >preferential queue insertion of ready-to-run "LWPs", to use Sun >terminology). This makes the scheduler incredibly bursty with >regard to your threaded process, and at the same time turns it >into a fair-share scheduler, meaning that it ignores priorities >when trying to decide what to run next. > > > > If you can write a user-level thread scheduler that won't get > > "overloaded" by a large number of threads, why can't you just > > write the kernel-level scheduler the same way? I suppose if > > the answer is that the existing FreeBSD scheduler is just too > > clumsy to provide good scalability over multiple cpus and a > > large number of runnable lwps, then wouldn't the first job be to > > improve the scheduler, rather than simply try to shift all the > > load into a UTS? > >The answer is that it's not possible to build a scheduler that >implements priority based fairness, when user processes are >permitted to request an unbounded amount of quantum. > >The best you can do is implement quantum allocation fairness, >and then implement a policy based decision on which thread in >a group of threads will get a quantum, once it is assigned to >a process. > >Technically, it's possible to implement this all in the kernel, >but it significantly and unnecesssarily increases the complexity. >In the case of calls which can be run to completion without >blocking in the kernel, it also increases the amount of protection >domain crossing. > >Again, you want to think about how you can spend the most time >you can possibly spend, actually running user code, instead of >operating system. In terms of "fairness", accounting scheduler >overhead for implementation of policy decisions against the >process which initiated the need for such overhead, is eminently >fair. > > >If you want to look at the simplest case, a multithreaded >application running on a uniprocessor box, the lowest possible >overhead is obtained by staying in user space -- not crossing >the protection domain at all -- and context switching between >threads within the same process, in user space, for as long as >the quantum holds out. Obviously, aio calls beat non-blocking >calls, since a non-blocking call that would block results in a >fault or other kernel operation occurring, and returning to the >caller with EWOULDBLK, completing sucessfully the next time it >is attempted (an extra protection domain crossing, compared to >AIO), but any approach beats context switching to a different >competing process, and losing the remainder of your quantum. > >Ask yourself how to make this work with the SMP side effect >of some implementations becoming a design goal instead of an >unintended side effect. > >It's only when you try to make things complicated, by adding >other processors, and then insisting that they be additive in >a single multithreaded program, instead of two programs, that >things start to get complicated. And then you have to remember >your first principles: why you had threads in the first place, >and not how you are now able to abuse them to achieve primitive >and generally grossly inefficient SMP scaling. > >If you want to achieve SMP scalability, at least be honest; and >design to achieve it efficiently as possible, not as a desirable >side effect of the abuse of a technology that _happens_ to result >in _some_ scalability. > >Another good reference in this regard, by the way, is the Linus >response to more recent Microsoft vs. Linux benchmarks, in which >he admits scalability problems in Linux (which uses the original >SVR4 kernel threading model, pretty much unaltered), and states >with regard to early claims of benchmark bias "...we were >arrogant..." (referring to the fact that MS OSs can in fact beat >Linux fairly, in some benchmarks). > > > > Terry Lambert > terry@lambert.org >--- >Any opinions in this posting are my own and not those of my present >or previous employers. > > >To Unsubscribe: send mail to majordomo@FreeBSD.org >with "unsubscribe freebsd-smp" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Feb 15 1:31:53 2001 Delivered-To: freebsd-smp@freebsd.org Received: from magnesium.net (toxic.magnesium.net [207.154.84.15]) by hub.freebsd.org (Postfix) with SMTP id 9510D37B4EC for ; Thu, 15 Feb 2001 01:31:51 -0800 (PST) Received: (qmail 94833 invoked by uid 1142); 15 Feb 2001 09:32:44 -0000 Date: 15 Feb 2001 01:32:44 -0800 Date: Thu, 15 Feb 2001 01:31:47 -0800 From: Jason Evans To: Seth Leigh Cc: Terry Lambert , freebsd-smp@FreeBSD.ORG Subject: Re: possible problem with SMP? Message-ID: <20010215013147.N56728@canonware.com> References: <5.0.2.1.0.20010214234050.026959f8@hobbiton.shire.net> <200102150708.AAA19435@usr08.primenet.com> <5.0.2.1.0.20010215025043.027a3008@hobbiton.shire.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <5.0.2.1.0.20010215025043.027a3008@hobbiton.shire.net>; from seth@pengar.com on Thu, Feb 15, 2001 at 02:56:07AM -0500 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Thu, Feb 15, 2001 at 02:56:07AM -0500, Seth Leigh wrote: > Terry, have you looked into the Solaris Doors mechanism, and do you think > that implementing a facility like that in FreeBSD would clean up and > improve the way FreeBSD plans to implement the threads library upcalls, and > other possible places for upcalls as well? Despite my as-yet limited > understanding, it sounds promising to me. I have looked at the Solaris doors mechanism, and am of the opinion that it is significantly more heavy-weight than what we need for KSEs. Implementing doors is a large task by itself. Jason To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Feb 15 9: 7: 0 2001 Delivered-To: freebsd-smp@freebsd.org Received: from ns.yogotech.com (ns.yogotech.com [206.127.123.66]) by hub.freebsd.org (Postfix) with ESMTP id E2ABF37B491 for ; Thu, 15 Feb 2001 09:06:54 -0800 (PST) Received: from nomad.yogotech.com (nomad.yogotech.com [206.127.123.131]) by ns.yogotech.com (8.9.3/8.9.3) with ESMTP id KAA23180; Thu, 15 Feb 2001 10:06:52 -0700 (MST) (envelope-from nate@nomad.yogotech.com) Received: (from nate@localhost) by nomad.yogotech.com (8.8.8/8.8.8) id KAA06278; Thu, 15 Feb 2001 10:06:52 -0700 (MST) (envelope-from nate) From: Nate Williams MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <14988.3243.652929.139953@nomad.yogotech.com> Date: Thu, 15 Feb 2001 10:06:51 -0700 (MST) To: Seth Leigh Cc: nate@yogotech.com (Nate Williams), freebsd-smp@FreeBSD.ORG Subject: Re: possible problem with SMP? In-Reply-To: <5.0.2.1.0.20010215020051.01b9fc70@hobbiton.shire.net> References: <5.0.2.1.0.20010214234050.026959f8@hobbiton.shire.net> <5.0.2.1.0.20010214222040.00aac5c0@hobbiton.shire.net> <20010214191110.D78224@mollari.cthul.hu> <5.0.2.1.0.20010215020051.01b9fc70@hobbiton.shire.net> X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid Reply-To: nate@yogotech.com (Nate Williams) Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > There are problems of cpu starvation with a Many to Many model in that > usually (unless you manually step up the concurrency) the Many to Many ends > up being Many to Number of CPUs. If Number of CPUs is equal to 1, then > usually you will only get one LWP beyond the ones that are sleeping. Using > the Solaris libpthread, a user-level thread will not give up the LWP it's > using until it voluntarily makes a call into libthread which gives > libthread an opportunity to deschedule that thread and schedule a different > one. This is the same 'model' that Green Threads use, which isn't very effecient. It is possible to have user-space threads be pre-emtive (setjmp/longjmp), so don't blame the implementation issues solely on user threads. > To illustrate this, I talked to a customer the other day whose > application had around 200 threads. He had around 40 LWPs created, but 39 > of them were blocked in the kernel, and 1 appeared to be running when > looking at the threads in dbx. The application appeared to be hung, > though. Finally he figured out that the one thread that was actually > running on the one free LWP was in fact running a hugely inefficient code > chunk that was simply taking forever to complete, and since it wasn't > calling into libthread, it was hogging that cpu and the other 160 or so > threads in the program simply were not getting *any* time whatsoever. This is a thread scheduler problem. > The solution to this was either to attempt to set the concurrency manually > to something like "a few" LWPs higher than the number he expected to block, > or throw in some sched_yield()s into all of his threads to make sure others > got some time, or else simply make them bound. I advised him to just make > them bound, run some tests to see how this affected his performance, and > then decide from there. In all likelihood the majority of his threads can > and do block at some point, and I see very little reason if that is the > case why he shouldn't just make his threads bound. Context switching is now *MUCH* higher. In properly written threaded programs (which are arguable hard to design, but not necessarily any more so than properly written high-performance multi-process software), context switching can be a big deal, especially in the case of *LOTS* of threads. At some point, the more threads you use, the higher the context switching becomes (this is fairly obvious). > I am starting to believe, based on the number of customers I have talked to > writing threaded code, that most real-world applications that use threads > use threads that sometimes block in the kernel. No argument there, see previous email re: I/O. > That being the case, I > honestly see no real reason not to make them bound. Only those threads that block need to be bound. And, in a good design, you can design the system so these threads have their own kernel context. However, with the current FreeBSD kernel-thread design, this can be some more 'automatically', since the kernel can make this decision for you. This is the essence of kernel/scheduler activations. > Fact is, once they > block, they cause a new LWP to be created anyways No, they don't. The number of LWP's is fixed at compile time. > and then basically if > all your threads block at least once in a relatively short amount of time > you end up with most of your threads having their "own" LWP anyhow, plus > you have the additional overhead of the more complicated thread > library. In short, for many if not most real-world applications, > user-level threads don't really buy you anything. Again, this is pure conjecture, and not based on any real-world benchmarks or experience. I can point to *lots* of both that show the opposite. Nate To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Feb 15 15:37: 3 2001 Delivered-To: freebsd-smp@freebsd.org Received: from smtp02.primenet.com (smtp02.primenet.com [206.165.6.132]) by hub.freebsd.org (Postfix) with ESMTP id 9385F37B401 for ; Thu, 15 Feb 2001 15:36:57 -0800 (PST) Received: (from daemon@localhost) by smtp02.primenet.com (8.9.3/8.9.3) id QAA26899; Thu, 15 Feb 2001 16:30:50 -0700 (MST) Received: from usr08.primenet.com(206.165.6.208) via SMTP by smtp02.primenet.com, id smtpdAAAW8aa3Z; Thu Feb 15 16:30:11 2001 Received: (from tlambert@localhost) by usr08.primenet.com (8.8.5/8.8.5) id QAA09980; Thu, 15 Feb 2001 16:36:10 -0700 (MST) From: Terry Lambert Message-Id: <200102152336.QAA09980@usr08.primenet.com> Subject: Re: possible problem with SMP? To: seth@pengar.com (Seth Leigh) Date: Thu, 15 Feb 2001 23:36:10 +0000 (GMT) Cc: nate@yogotech.com (Nate Williams), freebsd-smp@FreeBSD.ORG In-Reply-To: <5.0.2.1.0.20010215020051.01b9fc70@hobbiton.shire.net> from "Seth Leigh" at Feb 15, 2001 02:21:14 AM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > There are problems of cpu starvation with a Many to Many model in that > usually (unless you manually step up the concurrency) the Many to Many ends > up being Many to Number of CPUs. If Number of CPUs is equal to 1, then > usually you will only get one LWP beyond the ones that are sleeping. Using > the Solaris libpthread, a user-level thread will not give up the LWP it's > using until it voluntarily makes a call into libthread which gives > libthread an opportunity to deschedule that thread and schedule a different > one. To illustrate this, I talked to a customer the other day whose > application had around 200 threads. He had around 40 LWPs created, but 39 > of them were blocked in the kernel, and 1 appeared to be running when > looking at the threads in dbx. The application appeared to be hung, > though. Finally he figured out that the one thread that was actually > running on the one free LWP was in fact running a hugely inefficient code > chunk that was simply taking forever to complete, and since it wasn't > calling into libthread, it was hogging that cpu and the other 160 or so > threads in the program simply were not getting *any* time whatsoever. This application is poorly written; here's why. The application you are describing is using threads in order to implement multiple concurrent blocking calls. It's an attempt to increase concurrency of requests, not concurrency of code execution. The current user space threads code in BSD is very, very similar in effect: it makes non-blocking calls, and for those calls which will block, assumes that the OS will schedule the requested work to have completed by the nex time that a non-blocking call is made, so that the call will complete (e.g. an attempt to read data into a user buffer that can't be satisfied results in a fault and a return, and the next read attempt will find the fault satisfied). Writes are satisfied if there is buffer space available. In effect, both these models are a call-conversion model, where you exchange a blocking call for a non-blocking call (or a block on an LWP, and the creation of a new LWP to continue running), plus a thread context switch. Neither of these models improves concurrency on a UP system. On an SMP system, the Solaris model _may_ improce concurrency, _if_ the LWP is not a transient object. If it sticks around, then it would permit another process to be scheduled into user space. > I am starting to believe, based on the number of customers I have talked to > writing threaded code, that most real-world applications that use threads > use threads that sometimes block in the kernel. Threaded applications will block on resource unavailability; it's really irrelevent whether this happens in the kernel or in user space. If the blocking operation is due to a pthreads mutex, you are much better served by blocking in user space in a user space portion of the threads scheduler, so as to both avoid protection domain crossing (by way of a kernel call), and heavy weight kernel locking semantics to implement the mutex (indeed, if you have a program bug, you do not want your program deadlock to escalate to a kernel deadlock simply because you are using the same locking code). > That being the case, I honestly see no real reason not to make them > bound. If this were the only way they were being used, and if using them this way were correct threads programming, I'd agree. But the point of threads is to improve concurrency (virtual, on UP, real on SMP) of user space code, and to provide linear programmers a way to be able to think of state machine operations as if they were linear procedural operations (it turns out that anything you can do with threads can be described as a state machine, and can probably be significantly simplified using Carnot algebra, the same tool used in simplification of digital logic circuits). If the only reason it to permit concurrent blocking, well, then wok is not being done, for the most part, and only in the I/O case is concurrency occurring, and then only if it can make it all the way down to the driver, and your physical hadware support concurrent outstanding operations (e.g. SCSI tagged commands). > But basically, the question of Many to One, One to One, Many to Many, etc. > really comes down to a question of how complex the library will be, and how > threads will typically be used. I maintain, without proof, that the case > of pure user-level threads that don't block at all, that really benefit > from the fast user-level context switching, is not all that frequent in the > real world. I disagree. What if I want CPU affinity in an SMP system, to avoid cache-busting, and to let me use more than a tiny number (say 4) of processors? > Something in my mind likes the idea of only having one set of scheduler > code in a system. Not multiple sets, such as a UTS plus a kernel > scheduler. Simplicity can be a virtue. CPU affinity is most easily (and, I think, elegantly) implemented by having per CPU run queues. This is about the highest level of scheduler complexity that is acceptable in the kernel. Remeber the UNIX philosophy: adding kernel complexity to avoid user space complexity, unnecessarily, is a bad thing. If you could demonstrate that adding the code to the kernel would save a lot of user space processing, on average you might have a point. I maintain, however, that the majority of code written today is not threaded code, and that for all but a few classes of problem, threads are really the wrong tool for the job, and the programmers using them are copping out on having to do hard work. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Feb 15 15:41:55 2001 Delivered-To: freebsd-smp@freebsd.org Received: from ns.yogotech.com (ns.yogotech.com [206.127.123.66]) by hub.freebsd.org (Postfix) with ESMTP id 3F65237B699 for ; Thu, 15 Feb 2001 15:41:52 -0800 (PST) Received: from nomad.yogotech.com (nomad.yogotech.com [206.127.123.131]) by ns.yogotech.com (8.9.3/8.9.3) with ESMTP id QAA29748; Thu, 15 Feb 2001 16:41:48 -0700 (MST) (envelope-from nate@nomad.yogotech.com) Received: (from nate@localhost) by nomad.yogotech.com (8.8.8/8.8.8) id QAA10091; Thu, 15 Feb 2001 16:41:47 -0700 (MST) (envelope-from nate) From: Nate Williams MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <14988.26939.53659.399934@nomad.yogotech.com> Date: Thu, 15 Feb 2001 16:41:47 -0700 (MST) To: Terry Lambert Cc: seth@pengar.com (Seth Leigh), nate@yogotech.com (Nate Williams), freebsd-smp@FreeBSD.ORG Subject: Re: possible problem with SMP? In-Reply-To: <200102152336.QAA09980@usr08.primenet.com> References: <5.0.2.1.0.20010215020051.01b9fc70@hobbiton.shire.net> <200102152336.QAA09980@usr08.primenet.com> X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid Reply-To: nate@yogotech.com (Nate Williams) Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > In effect, both these models are a call-conversion model, where you > exchange a blocking call for a non-blocking call (or a block on an > LWP, and the creation of a new LWP to continue running), plus a > thread context switch. > > Neither of these models improves concurrency on a UP system. Sure they do. Not in all cases, but in most cases they do ipmrove concurrency since they're allowing other threads to execute while a non-threaded application would block on that system call. It does not scale as well on a UP system like SA's do, but in many case the threading model is more effecient 'most of the time' vs. a multi-process model. > Threaded applications will block on resource unavailability; it's > really irrelevent whether this happens in the kernel or in user > space. Except the cost of 'switching context' in userland is much lower than in the kernel. (As you said, this is the what 'Lightweight' implies with LWP's. :) Nate To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Feb 15 16: 3:57 2001 Delivered-To: freebsd-smp@freebsd.org Received: from smtp10.phx.gblx.net (smtp10.phx.gblx.net [206.165.6.140]) by hub.freebsd.org (Postfix) with ESMTP id 8B76937B4EC for ; Thu, 15 Feb 2001 16:03:47 -0800 (PST) Received: (from daemon@localhost) by smtp10.phx.gblx.net (8.9.3/8.9.3) id RAA69812; Thu, 15 Feb 2001 17:03:16 -0700 Received: from usr08.primenet.com(206.165.6.208) via SMTP by smtp10.phx.gblx.net, id smtpdc8elia; Thu Feb 15 17:03:12 2001 Received: (from tlambert@localhost) by usr08.primenet.com (8.8.5/8.8.5) id RAA10529; Thu, 15 Feb 2001 17:03:39 -0700 (MST) From: Terry Lambert Message-Id: <200102160003.RAA10529@usr08.primenet.com> Subject: Re: possible problem with SMP? To: seth@pengar.com (Seth Leigh) Date: Fri, 16 Feb 2001 00:03:39 +0000 (GMT) Cc: tlambert@primenet.com (Terry Lambert), freebsd-smp@FreeBSD.ORG In-Reply-To: <5.0.2.1.0.20010215025043.027a3008@hobbiton.shire.net> from "Seth Leigh" at Feb 15, 2001 02:56:07 AM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Please do not quote entire articles, particularly _after_ your comments, when responding in the future; thanks. 8-). > I just read pages 407 to 410 of the Jim Mauro/Richard McDougall "Solaris > Internals" book and it talks specifically about scheduler > activations. They mention how they are used by the kernel scheduler to > keep LWPs from a given process running throughout the rest of their quantum > even when one of them blocks, and some other things. They specifically > talk about Solaris Doors as being the mechanism by which the kernel calls > up into the threads library. I will have to read a lot more in the next > day or so about Solaris scheduler activations until I know exactly what > they are and what they do. I am still only on page 150 of the book. It's > not easy reading. ;-) But I will finish reading it, and then I will read > it all the way through again, since I really do want to understand this stuff. > > Terry, have you looked into the Solaris Doors mechanism, and do you think > that implementing a facility like that in FreeBSD would clean up and > improve the way FreeBSD plans to implement the threads library upcalls, and > other possible places for upcalls as well? Despite my as-yet limited > understanding, it sounds promising to me. I have looked at the mechanism. I think you will find that they are a Solaris 2.7 introduction, but am willing to be wrong. Doors are interesting. The thing they most resemble is a VMS asynchronous system trap, or an NT I/O completion routine. The DEC MTS system (Multi Threading Service), which was, as far as I am aware, an internal DEC product shared with only a few vendors, including my then employer, Novell, used ASTs in order to implement a simple call conversion threading library. It was superior to that in BSD, in that operations were not prevented and retried, which is an unnecessary protection domain crossing. The DEC code also was missing a number of obvious primitives, including timers and interlocks, for which VMS provided AST enabled routines; in the course of porting Mentat threads to VMS, I has to add these facilities to the threading system in order to support IPX, SPX, and the NPC streams mux. Other than typing in VMS sources from microfiche in order to fix bugs in the printer symbiont and the CTERM protocol handler, this was my first time out as a paid BLISS programmer. 8-). Where does Solaris' Doors mechanism have it over on other call conversion enabling technology? I'd have to say that it has it because it exists _simultaneously_ with the other mechanisms. The Doors paradigm is fundamentally similar to the signal code, in that it runs on a seperate stack, but can be triggered without having to go through a trampolie. Fundamentally, this means I cross U->K to set things in motion, then K->U, back to my program, once I have started rolling. Later, after the ball gets to the bottom of the hill, the kernel triggers a door, which runs K->U, causes some code to run, and then jumps bad U->K, when it's done. Why is it called a Door? All I can guess is that someone spent too much time writing PC BBS code in their youth. 8-). An astute observer will note that the number of protection domain crossings for this is 4; this is the same number we have for the other more advanced mechanisms. It's also the same number we have for the non-blocking I/O call conversion scheduler, if we assume that the first call always fails. If we don't make that assumption, then it's actually more than that approach (this doesn't make the approach better, since there is a loss of I/O concurrency; philosophically, it's the same thing as the difference between soft updates and DOW). In contrast, an async call gate mechanism has 2 domain crossings in all cases (asuming a doubly mapped call competion status structure). From a generic perspective, yes, this would simplify the FreeBSD activation upcall, and it could be useful in terms of the ability to write user space daemons that would handle processing of some kernel events (an external pager would be one example). Whether the code simplification would be worth it really depends on how the completion communication is handled for activations; it should be possible to drop it to 3 domain crossings per activation with the current BSD plans. It would certainly drop the complexity in the case that the user space code was in the user space scheduler at the time of the process being involuntarily preempted (instead of being in a "safe" region). This alone might justify the code, without having to appeal to the "think what you could do later!" crowd. ...Again, all IMO, of course. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Feb 15 16: 9:25 2001 Delivered-To: freebsd-smp@freebsd.org Received: from smtp10.phx.gblx.net (smtp10.phx.gblx.net [206.165.6.140]) by hub.freebsd.org (Postfix) with ESMTP id 300B837B491 for ; Thu, 15 Feb 2001 16:09:23 -0800 (PST) Received: (from daemon@localhost) by smtp10.phx.gblx.net (8.9.3/8.9.3) id RAA24114; Thu, 15 Feb 2001 17:08:51 -0700 Received: from usr08.primenet.com(206.165.6.208) via SMTP by smtp10.phx.gblx.net, id smtpdqUF5ia; Thu Feb 15 17:08:49 2001 Received: (from tlambert@localhost) by usr08.primenet.com (8.8.5/8.8.5) id RAA10700; Thu, 15 Feb 2001 17:09:13 -0700 (MST) From: Terry Lambert Message-Id: <200102160009.RAA10700@usr08.primenet.com> Subject: Re: possible problem with SMP? To: jasone@canonware.com (Jason Evans) Date: Fri, 16 Feb 2001 00:09:12 +0000 (GMT) Cc: seth@pengar.com (Seth Leigh), tlambert@primenet.com (Terry Lambert), freebsd-smp@FreeBSD.ORG In-Reply-To: <20010215013147.N56728@canonware.com> from "Jason Evans" at Feb 15, 2001 01:31:47 AM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > Terry, have you looked into the Solaris Doors mechanism, and do you think > > that implementing a facility like that in FreeBSD would clean up and > > improve the way FreeBSD plans to implement the threads library upcalls, and > > other possible places for upcalls as well? Despite my as-yet limited > > understanding, it sounds promising to me. > > I have looked at the Solaris doors mechanism, and am of the opinion that it > is significantly more heavy-weight than what we need for KSEs. > Implementing doors is a large task by itself. See my other posting; it may be worth the decrease in complexity in the perverse corner cases. Personally, I'm inclined to agree with you, though, since it would mean more domain crossings than are absolutely necessary (so does the activation approach, but at least it can be justified in terms of maintaining backwards binary compataibility, and it's less than a Doors approach, if completion is handled correctly, which the documents indicate is the plan). From a generic perspective, maybe we already have a competing technology for Doors in the kqueue code (ignoring threads using the code as an implementation detail). Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Feb 15 16:51:39 2001 Delivered-To: freebsd-smp@freebsd.org Received: from smtp04.primenet.com (smtp04.primenet.com [206.165.6.134]) by hub.freebsd.org (Postfix) with ESMTP id B51DA37B4EC for ; Thu, 15 Feb 2001 16:51:36 -0800 (PST) Received: (from daemon@localhost) by smtp04.primenet.com (8.9.3/8.9.3) id RAA11898; Thu, 15 Feb 2001 17:46:07 -0700 (MST) Received: from usr08.primenet.com(206.165.6.208) via SMTP by smtp04.primenet.com, id smtpdAAASEaqmx; Thu Feb 15 17:45:59 2001 Received: (from tlambert@localhost) by usr08.primenet.com (8.8.5/8.8.5) id RAA12056; Thu, 15 Feb 2001 17:51:25 -0700 (MST) From: Terry Lambert Message-Id: <200102160051.RAA12056@usr08.primenet.com> Subject: Re: possible problem with SMP? To: nate@yogotech.com Date: Fri, 16 Feb 2001 00:51:24 +0000 (GMT) Cc: tlambert@primenet.com (Terry Lambert), seth@pengar.com (Seth Leigh), freebsd-smp@FreeBSD.ORG In-Reply-To: <14988.26939.53659.399934@nomad.yogotech.com> from "Nate Williams" at Feb 15, 2001 04:41:47 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > In effect, both these models are a call-conversion model, where you > > exchange a blocking call for a non-blocking call (or a block on an > > LWP, and the creation of a new LWP to continue running), plus a > > thread context switch. > > > > Neither of these models improves concurrency on a UP system. > > Sure they do. Not in all cases, but in most cases they do ipmrove > concurrency since they're allowing other threads to execute while a > non-threaded application would block on that system call. > > It does not scale as well on a UP system like SA's do, but in many case > the threading model is more effecient 'most of the time' vs. a > multi-process model. I meant in the context of allowing the code to execute concurrently; they both eliminate all the stalls you can eliminate, but the processor can't run more than one set of code at a time, if all you have is one processor. This probably wasn't clear; I've been approaching this thread from the context of threads in UP vs. SMP, since it was posted to the -smp mailing list, instead of to -arch. I've assumed that the original poster is looking at threads as a means of SMP scalability, and not from the perspective of the best way to achieve SMP scalability (i.e. it started out as a "how do I build a better hammer" discussion instead of as a "what is the right tool for this job" discussion). > > Threaded applications will block on resource unavailability; it's > > really irrelevent whether this happens in the kernel or in user > > space. > > Except the cost of 'switching context' in userland is much lower than > in the kernel. (As you said, this is the what 'Lightweight' implies > with LWP's. :) Right, exactly; and it's not just the context switch, it's the protection domain crossing, as well. On SPARC machines, this is somewhat mitigated, since even the liblwp user space threads on SunOS 4.x had a helper system call to deal with the SPARC register window problem, and that had to be called on context switches in user space, in case there was a stack frame push, but on most machines, doing as much in user space will result in significant savings. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Feb 15 17:21:23 2001 Delivered-To: freebsd-smp@freebsd.org Received: from sharmas.dhs.org (c62443-a.frmt1.sfba.home.com [24.0.69.165]) by hub.freebsd.org (Postfix) with ESMTP id 44DDE37B491 for ; Thu, 15 Feb 2001 17:21:22 -0800 (PST) Received: (from adsharma@localhost) by sharmas.dhs.org (8.9.3/8.9.3) id RAA08029; Thu, 15 Feb 2001 17:20:15 -0800 Date: Thu, 15 Feb 2001 17:20:15 -0800 From: Arun Sharma Message-Id: <200102160120.RAA08029@sharmas.dhs.org> To: frussell@p1.cs.ohiou.edu Cc: freebsd-smp@freebsd.org Subject: Re: possible problem with SMP? In-Reply-To: References: Reply-To: adsharma@sharmas.dhs.org Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 14 Feb 2001 20:31:40 +0100, Russell Francis wrote: > Something that puzzles me is that xosview only displays the stats > for one processor I sent patches for this (both xosview and per CPU sysctls) last year, but no one responded. I'd agree that there is a value to keeping the commit privileges to a few "commiters". But I find the fact that patches posted to the list go unanswered/ignored discouraging for new contributors and when it happens repeatedly, a bit insulting too. Now, I'm not motivated to do anything for FreeBSD. -Arun To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Feb 15 17:24: 6 2001 Delivered-To: freebsd-smp@freebsd.org Received: from sharmas.dhs.org (c62443-a.frmt1.sfba.home.com [24.0.69.165]) by hub.freebsd.org (Postfix) with ESMTP id 3EB2837B65D for ; Thu, 15 Feb 2001 17:24:03 -0800 (PST) Received: (from adsharma@localhost) by sharmas.dhs.org (8.9.3/8.9.3) id RAA08035; Thu, 15 Feb 2001 17:22:49 -0800 Date: Thu, 15 Feb 2001 17:22:49 -0800 From: Arun Sharma Message-Id: <200102160122.RAA08035@sharmas.dhs.org> To: nate@yogotech.com Cc: smp@freebsd.org Subject: Re: possible problem with SMP? In-Reply-To: <14987.27399.662691.291906@nomad.yogotech.com> References: <5.0.2.1.0.20010214234050.026959f8@hobbiton.shire.net> <14987.27399.662691.291906@nomad.yogotech.com> Reply-To: adsharma@sharmas.dhs.org Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 15 Feb 2001 06:37:23 +0100, Nate Williams wrote: > Sun's JDK uses a one-one mapping, because Do you have any references for this ? libpthread on Solaris uses n:m mapping, AFAIK. -Arun To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Feb 15 17:31:20 2001 Delivered-To: freebsd-smp@freebsd.org Received: from mail12.svr.pol.co.uk (mail12.svr.pol.co.uk [195.92.193.215]) by hub.freebsd.org (Postfix) with ESMTP id BD3DE37B503 for ; Thu, 15 Feb 2001 17:31:16 -0800 (PST) Received: from [195.92.67.23] (helo=mail18.svr.pol.co.uk) by mail12.svr.pol.co.uk with esmtp (Exim 3.13 #0) id 14TZjv-0004Ga-00 for freebsd-smp@freebsd.org; Fri, 16 Feb 2001 01:31:15 +0000 Received: from modem-97.california.dialup.pol.co.uk ([62.137.56.97] helo=DEDICATION1) by mail18.svr.pol.co.uk with smtp (Exim 3.13 #0) id 14TZju-0003Lx-00 for freebsd-smp@freebsd.org; Fri, 16 Feb 2001 01:31:15 +0000 Message-ID: <00f501c097b8$18ed85d0$01010a0a@DEDICATIONINET.local> From: "Andy C" To: Subject: Hmm Date: Fri, 16 Feb 2001 01:30:50 -0000 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4133.2400 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org I get this error on boot, anyone know whats wrong with it?: -- APIC_IO: Testing 8254 interrupt delivery APIC_IO: Broken MP table detected: 8254 is not connected to IOAPIC #0 intpin 2 APIC_IO: routing 8254 via 8259 and IOAPIC #0 intpin 0 SMP: AP CPU #1 Launched! -- Cheers, Andy. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Feb 15 17:57: 2 2001 Delivered-To: freebsd-smp@freebsd.org Received: from gtei2.bellatlantic.net (gtei2.bellatlantic.net [199.45.40.146]) by hub.freebsd.org (Postfix) with ESMTP id DE5EC37B401 for ; Thu, 15 Feb 2001 17:56:58 -0800 (PST) Received: from me-513q3sc0zun0.pengar.com (adsl-64-223-148-61.mannh.adsl.bellatlantic.net [64.223.148.61]) by gtei2.bellatlantic.net (8.9.1/8.9.1) with ESMTP id UAA05207 for ; Thu, 15 Feb 2001 20:57:12 -0500 (EST) Message-Id: <5.0.2.1.0.20010215204946.027a3c88@hobbiton.shire.net> X-Sender: seth-pc@hobbiton.shire.net X-Mailer: QUALCOMM Windows Eudora Version 5.0.2 Date: Thu, 15 Feb 2001 20:53:59 -0500 To: freebsd-smp@freebsd.org From: Seth Leigh Subject: Re: possible problem with SMP? In-Reply-To: <200102160122.RAA08035@sharmas.dhs.org> References: <14987.27399.662691.291906@nomad.yogotech.com> <5.0.2.1.0.20010214234050.026959f8@hobbiton.shire.net> <14987.27399.662691.291906@nomad.yogotech.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org With the normal threads library under Solaris (/usr/lib/libthread.so.1 and /usr/lib/libpthread.so.1) the default mode of operation is indeed n:m. However, if you declare a pthread_attr_t and call pthread_attr_setscope() on it with PTHREAD_SCOPE_SYSTEM, then any threads created subsequently are bound to an lwp. Threads created with PTHREAD_SCOPE_SYSTEM are 1:1. Additionally, with Solaris 8 there is an alternate threads library (/usr/lib/lwp/libpthread.so.1) which is strictly 1:1. Ie: bound threads are all you get, and all scheduling and such is left to the kernel. Seth At 05:22 PM 2/15/2001 -0800, Arun Sharma wrote: >On 15 Feb 2001 06:37:23 +0100, Nate Williams wrote: > > Sun's JDK uses a one-one mapping, because > >Do you have any references for this ? libpthread on Solaris uses n:m >mapping, AFAIK. > > -Arun > > >To Unsubscribe: send mail to majordomo@FreeBSD.org >with "unsubscribe freebsd-smp" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Feb 15 19:17:32 2001 Delivered-To: freebsd-smp@freebsd.org Received: from ns.yogotech.com (ns.yogotech.com [206.127.123.66]) by hub.freebsd.org (Postfix) with ESMTP id C725537B491 for ; Thu, 15 Feb 2001 19:17:29 -0800 (PST) Received: from nomad.yogotech.com (nomad.yogotech.com [206.127.123.131]) by ns.yogotech.com (8.9.3/8.9.3) with ESMTP id UAA03256; Thu, 15 Feb 2001 20:17:27 -0700 (MST) (envelope-from nate@nomad.yogotech.com) Received: (from nate@localhost) by nomad.yogotech.com (8.8.8/8.8.8) id UAA10794; Thu, 15 Feb 2001 20:17:26 -0700 (MST) (envelope-from nate) From: Nate Williams MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <14988.39877.884249.535240@nomad.yogotech.com> Date: Thu, 15 Feb 2001 20:17:25 -0700 (MST) To: adsharma@sharmas.dhs.org Cc: nate@yogotech.com, smp@freebsd.org Subject: Re: possible problem with SMP? In-Reply-To: <200102160122.RAA08035@sharmas.dhs.org> References: <5.0.2.1.0.20010214234050.026959f8@hobbiton.shire.net> <14987.27399.662691.291906@nomad.yogotech.com> <200102160122.RAA08035@sharmas.dhs.org> X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid Reply-To: nate@yogotech.com (Nate Williams) Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > Sun's JDK uses a one-one mapping, because > > Do you have any references for this ? The source code is freely downloadable, and it's easy to see this from reading the source code. (One of my other hats is the FreeBSD/JDK effort, so I'm fairly familiar with the JDK internals...) Nate To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Feb 15 21: 8:20 2001 Delivered-To: freebsd-smp@freebsd.org Received: from dan.emsphone.com (dan.emsphone.com [199.67.51.101]) by hub.freebsd.org (Postfix) with ESMTP id 78B6337B65D for ; Thu, 15 Feb 2001 21:08:18 -0800 (PST) Received: (from dan@localhost) by dan.emsphone.com (8.11.1/8.11.1) id f1G58FT02748; Thu, 15 Feb 2001 23:08:15 -0600 (CST) (envelope-from dan) Date: Thu, 15 Feb 2001 23:08:15 -0600 From: Dan Nelson To: Andy C Cc: freebsd-smp@FreeBSD.ORG Subject: Re: Hmm Message-ID: <20010215230815.A12821@dan.emsphone.com> References: <00f501c097b8$18ed85d0$01010a0a@DEDICATIONINET.local> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.3.14i In-Reply-To: <00f501c097b8$18ed85d0$01010a0a@DEDICATIONINET.local>; from "Andy C" on Fri Feb 16 01:30:50 GMT 2001 X-OS: FreeBSD 5.0-CURRENT Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org In the last episode (Feb 16), Andy C said: > I get this error on boot, anyone know whats wrong with it?: > > -- > APIC_IO: Testing 8254 interrupt delivery > APIC_IO: Broken MP table detected: 8254 is not connected to IOAPIC #0 intpin 2 > APIC_IO: routing 8254 via 8259 and IOAPIC #0 intpin 0 > SMP: AP CPU #1 Launched! > -- I'd say exactly what the message says. The MP table doesn't match the actual setup. FreeBSD then compensates for it, and the kernel continues booting. My system prints the same message and it doesn't seem to hurt. -- Dan Nelson dnelson@emsphone.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Feb 15 21:17: 0 2001 Delivered-To: freebsd-smp@freebsd.org Received: from obsecurity.dyndns.org (adsl-64-165-226-49.dsl.lsan03.pacbell.net [64.165.226.49]) by hub.freebsd.org (Postfix) with ESMTP id A3EE737B4EC for ; Thu, 15 Feb 2001 21:16:55 -0800 (PST) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 2C90166E6A; Thu, 15 Feb 2001 21:16:55 -0800 (PST) Date: Thu, 15 Feb 2001 21:16:55 -0800 From: Kris Kennaway To: Arun Sharma Cc: frussell@p1.cs.ohiou.edu, freebsd-smp@freebsd.org Subject: Re: possible problem with SMP? Message-ID: <20010215211654.A29410@mollari.cthul.hu> References: <200102160120.RAA08029@sharmas.dhs.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-md5; protocol="application/pgp-signature"; boundary="17pEHd4RhPHOinZp" Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <200102160120.RAA08029@sharmas.dhs.org>; from adsharma@sharmas.dhs.org on Thu, Feb 15, 2001 at 05:20:15PM -0800 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org --17pEHd4RhPHOinZp Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Feb 15, 2001 at 05:20:15PM -0800, Arun Sharma wrote: > On 14 Feb 2001 20:31:40 +0100, Russell Francis = wrote: > > Something that puzzles me is that xosview only displays the stats > > for one processor >=20 > I sent patches for this (both xosview and per CPU sysctls) last year, but > no one responded. I'd agree that there is a value to keeping the commit > privileges to a few "commiters". But I find the fact that patches posted > to the list go unanswered/ignored discouraging for new contributors and > when it happens repeatedly, a bit insulting too. Which PR number are your patches in? I couldn't find them when I just checked. Sending patches to the mailing lists is a good way to have them lost or ignored -- I won't claim that turnaround time on the PR database is always stellar, but at least they're there for all time for committers to notice and get back to. > Now, I'm not motivated to do anything for FreeBSD. That's a shame :-( Kris --17pEHd4RhPHOinZp Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.4 (FreeBSD) Comment: For info see http://www.gnupg.org iD8DBQE6jLfGWry0BWjoQKURAj6JAKDEalA4oggTvurViCEm82i22AqZLwCgriF5 EEia6o543iaN2TuWrk9h1Bg= =Q3Cd -----END PGP SIGNATURE----- --17pEHd4RhPHOinZp-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Feb 15 21:31:31 2001 Delivered-To: freebsd-smp@freebsd.org Received: from p1.cs.ohiou.edu (p1.cs.ohiou.edu [132.235.1.11]) by hub.freebsd.org (Postfix) with ESMTP id E0FB037B401 for ; Thu, 15 Feb 2001 21:31:29 -0800 (PST) Received: from localhost (frussell@localhost) by p1.cs.ohiou.edu (8.9.3/8.9.3) with SMTP id AAA09658; Fri, 16 Feb 2001 00:31:24 -0500 (EST) Date: Fri, 16 Feb 2001 00:31:23 -0500 (EST) From: Russell Francis X-Sender: frussell@p1 To: Arun Sharma Cc: freebsd-smp@freebsd.org Subject: Re: possible problem with SMP? In-Reply-To: <200102160120.RAA08029@sharmas.dhs.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > I sent patches for this (both xosview and per CPU sysctls) last year, but > no one responded. I'd agree that there is a value to keeping the commit > privileges to a few "commiters". But I find the fact that patches posted > to the list go unanswered/ignored discouraging for new contributors and > when it happens repeatedly, a bit insulting too. I was in the process of doing exactly that, do you still have the patches? Perhaps you can save me a little time :-) > Now, I'm not motivated to do anything for FreeBSD. It can be frustrating at times but in my opinion working on projects like this are their own reward. Some patches get accepted others lost but they were all a hell of alot of fun to write! Am I wrong here? -Russ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Feb 15 21:34:23 2001 Delivered-To: freebsd-smp@freebsd.org Received: from web1704.mail.yahoo.com (web1704.mail.yahoo.com [128.11.23.215]) by hub.freebsd.org (Postfix) with SMTP id DEF4637B4EC for ; Thu, 15 Feb 2001 21:34:19 -0800 (PST) Received: (qmail 25952 invoked by uid 60001); 16 Feb 2001 05:34:19 -0000 Message-ID: <20010216053419.25951.qmail@web1704.mail.yahoo.com> Received: from [61.153.1.210] by web1704.mail.yahoo.com; Thu, 15 Feb 2001 21:34:19 PST Date: Thu, 15 Feb 2001 21:34:19 -0800 (PST) From: Yifeng Xu Subject: Re: possible problem with SMP? To: Seth Leigh , freebsd-smp@freebsd.org In-Reply-To: <5.0.2.1.0.20010215204946.027a3c88@hobbiton.shire.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org by reading jasone's KSE project web pages, I do think KSE will support PTHREAD_SCOPE_SYSTEM very well, FreeBSD will do support 1:1 and M:N thread model, it seems KSE will have bright future, I think it's perfect and I am waiting. David Xu --- Seth Leigh wrote: > With the normal threads library under Solaris (/usr/lib/libthread.so.1 and > /usr/lib/libpthread.so.1) the default mode of operation is indeed > n:m. However, if you declare a pthread_attr_t and call > pthread_attr_setscope() on it with PTHREAD_SCOPE_SYSTEM, then any threads > created subsequently are bound to an lwp. Threads created with > PTHREAD_SCOPE_SYSTEM are 1:1. > > Additionally, with Solaris 8 there is an alternate threads library > (/usr/lib/lwp/libpthread.so.1) which is strictly 1:1. Ie: bound threads > are all you get, and all scheduling and such is left to the kernel. > > Seth > > At 05:22 PM 2/15/2001 -0800, Arun Sharma wrote: > >On 15 Feb 2001 06:37:23 +0100, Nate Williams wrote: > > > Sun's JDK uses a one-one mapping, because > > > >Do you have any references for this ? libpthread on Solaris uses n:m > >mapping, AFAIK. > > > > -Arun > > > > > >To Unsubscribe: send mail to majordomo@FreeBSD.org > >with "unsubscribe freebsd-smp" in the body of the message > > > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-smp" in the body of the message __________________________________________________ Do You Yahoo!? Get personalized email addresses from Yahoo! Mail - only $35 a year! http://personal.mail.yahoo.com/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Feb 15 22:19:25 2001 Delivered-To: freebsd-smp@freebsd.org Received: from sharmas.dhs.org (c62443-a.frmt1.sfba.home.com [24.0.69.165]) by hub.freebsd.org (Postfix) with ESMTP id 5ECCA37B401 for ; Thu, 15 Feb 2001 22:19:20 -0800 (PST) Received: (from adsharma@localhost) by sharmas.dhs.org (8.9.3/8.9.3) id WAA08617; Thu, 15 Feb 2001 22:18:14 -0800 Date: Thu, 15 Feb 2001 22:18:14 -0800 From: Arun Sharma To: Kris Kennaway Cc: freebsd-smp@freebsd.org Subject: Re: possible problem with SMP? Message-ID: <20010215221814.A8608@sharmas.dhs.org> References: <200102160120.RAA08029@sharmas.dhs.org> <20010215211654.A29410@mollari.cthul.hu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2i In-Reply-To: <20010215211654.A29410@mollari.cthul.hu>; from kris@obsecurity.org on Thu, Feb 15, 2001 at 09:16:55PM -0800 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Thu, Feb 15, 2001 at 09:16:55PM -0800, Kris Kennaway wrote: > > Which PR number are your patches in? I couldn't find them when I just > checked. Sending patches to the mailing lists is a good way to have > them lost or ignored -- I won't claim that turnaround time on the PR > database is always stellar, but at least they're there for all time > for committers to notice and get back to. http://www.freebsd.org/cgi/query-pr.cgi?pr=18524 http://marc.theaimsgroup.com/?l=freebsd-smp&m=93979495907143&w=2 -Arun To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Feb 15 22:35:40 2001 Delivered-To: freebsd-smp@freebsd.org Received: from obsecurity.dyndns.org (adsl-64-165-226-49.dsl.lsan03.pacbell.net [64.165.226.49]) by hub.freebsd.org (Postfix) with ESMTP id E98BF37B491 for ; Thu, 15 Feb 2001 22:35:35 -0800 (PST) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 7A3D666E6A; Thu, 15 Feb 2001 22:35:35 -0800 (PST) Date: Thu, 15 Feb 2001 22:35:35 -0800 From: Kris Kennaway To: Arun Sharma Cc: freebsd-smp@freebsd.org Subject: Re: possible problem with SMP? Message-ID: <20010215223535.B30269@mollari.cthul.hu> References: <200102160120.RAA08029@sharmas.dhs.org> <20010215211654.A29410@mollari.cthul.hu> <20010215221814.A8608@sharmas.dhs.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-md5; protocol="application/pgp-signature"; boundary="4SFOXa2GPu3tIq4H" Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <20010215221814.A8608@sharmas.dhs.org>; from arun@sharmas.dhs.org on Thu, Feb 15, 2001 at 10:18:14PM -0800 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org --4SFOXa2GPu3tIq4H Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Feb 15, 2001 at 10:18:14PM -0800, Arun Sharma wrote: > On Thu, Feb 15, 2001 at 09:16:55PM -0800, Kris Kennaway wrote: > >=20 > > Which PR number are your patches in? I couldn't find them when I just > > checked. Sending patches to the mailing lists is a good way to have > > them lost or ignored -- I won't claim that turnaround time on the PR > > database is always stellar, but at least they're there for all time > > for committers to notice and get back to. >=20 > http://www.freebsd.org/cgi/query-pr.cgi?pr=3D18524 > http://marc.theaimsgroup.com/?l=3Dfreebsd-smp&m=3D93979495907143&w=3D2 Okay, I see the xosview patch is predicated on the SMP changes. At the time these patches were submitted there weren't many people working on SMP - you might have better luck with them now. Kris --4SFOXa2GPu3tIq4H Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.4 (FreeBSD) Comment: For info see http://www.gnupg.org iD8DBQE6jMo2Wry0BWjoQKURAiDFAJ9PAPtZalE3lSG6wO/1rGe7XQJwOACg++Oh 3KXa7RQu9RbkpW0vdM06zSk= =o46w -----END PGP SIGNATURE----- --4SFOXa2GPu3tIq4H-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Feb 15 23: 3: 8 2001 Delivered-To: freebsd-smp@freebsd.org Received: from sharmas.dhs.org (c62443-a.frmt1.sfba.home.com [24.0.69.165]) by hub.freebsd.org (Postfix) with ESMTP id EDB1837B67D for ; Thu, 15 Feb 2001 23:03:05 -0800 (PST) Received: (from adsharma@localhost) by sharmas.dhs.org (8.9.3/8.9.3) id XAA08817; Thu, 15 Feb 2001 23:02:00 -0800 Date: Thu, 15 Feb 2001 23:02:00 -0800 From: Arun Sharma Message-Id: <200102160702.XAA08817@sharmas.dhs.org> To: seth@pengar.com Cc: freebsd-smp@freebsd.org Subject: Re: possible problem with SMP? In-Reply-To: <5.0.2.1.0.20010215204946.027a3c88@hobbiton.shire.net> References: <200102160122.RAA08035@sharmas.dhs.org> <5.0.2.1.0.20010215204946.027a3c88@hobbiton.shire.net> Reply-To: adsharma@sharmas.dhs.org Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 16 Feb 2001 02:57:09 +0100, Seth Leigh wrote: > With the normal threads library under Solaris (/usr/lib/libthread.so.1 and > /usr/lib/libpthread.so.1) the default mode of operation is indeed > n:m. However, if you declare a pthread_attr_t and call > pthread_attr_setscope() on it with PTHREAD_SCOPE_SYSTEM, then any threads > created subsequently are bound to an lwp. Threads created with > PTHREAD_SCOPE_SYSTEM are 1:1. Sun JDKs do not seem to do PTHREAD_SCOPE_SYSTEM under normal operation, so I can't see how their implementation would be 1:1. > > Additionally, with Solaris 8 there is an alternate threads library > (/usr/lib/lwp/libpthread.so.1) which is strictly 1:1. Ie: bound threads > are all you get, and all scheduling and such is left to the kernel. > Since JDKs for pre Solaris 8 versions of Solaris exist, I assume that the JDK is not being linked against this lib. -Arun To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Feb 15 23:23:16 2001 Delivered-To: freebsd-smp@freebsd.org Received: from gtei1.bellatlantic.net (gtei1.bellatlantic.net [199.45.40.145]) by hub.freebsd.org (Postfix) with ESMTP id F236E37B503 for ; Thu, 15 Feb 2001 23:23:13 -0800 (PST) Received: from me-513q3sc0zun0.pengar.com (adsl-64-223-148-61.mannh.adsl.bellatlantic.net [64.223.148.61]) by gtei1.bellatlantic.net (8.9.1/8.9.1) with ESMTP id CAA25135; Fri, 16 Feb 2001 02:23:25 -0500 (EST) Message-Id: <5.0.2.1.0.20010216021732.00aae718@hobbiton.shire.net> X-Sender: seth-pc@hobbiton.shire.net X-Mailer: QUALCOMM Windows Eudora Version 5.0.2 Date: Fri, 16 Feb 2001 02:20:12 -0500 To: adsharma@sharmas.dhs.org From: Seth Leigh Subject: Re: possible problem with SMP? Cc: freebsd-smp@freebsd.org In-Reply-To: <200102160702.XAA08817@sharmas.dhs.org> References: <5.0.2.1.0.20010215204946.027a3c88@hobbiton.shire.net> <200102160122.RAA08035@sharmas.dhs.org> <5.0.2.1.0.20010215204946.027a3c88@hobbiton.shire.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org At 11:02 PM 2/15/2001 -0800, Arun Sharma wrote: > > Additionally, with Solaris 8 there is an alternate threads library > > (/usr/lib/lwp/libpthread.so.1) which is strictly 1:1. Ie: bound threads > > are all you get, and all scheduling and such is left to the kernel. > > > >Since JDKs for pre Solaris 8 versions of Solaris exist, I assume that the >JDK is not being linked against this lib. > > -Arun Actually, not that many people are using it yet as far as I know. It's there for people to experiment with, and for people to do actual performance testing against. As a purely bound thread implementation, the code for this library is much simpler than the standard libthread. I just mentioned its existence for completeness. ;-) Actually, if you have Solaris 8 and some multi-threaded libraries, you may well give the alternate threads library a whirl and just see how it performs. You can do this by setting LD_LIBRARY_PATH to /usr/lib/lwp/ and you should run with the alternate threads lib. Seth To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Feb 16 2: 7:24 2001 Delivered-To: freebsd-smp@freebsd.org Received: from winston.osd.bsdi.com (winston.osd.bsdi.com [204.216.27.229]) by hub.freebsd.org (Postfix) with ESMTP id EF26237B67D for ; Fri, 16 Feb 2001 02:07:20 -0800 (PST) Received: from winston.osd.bsdi.com (jkh@localhost [127.0.0.1]) by winston.osd.bsdi.com (8.11.2/8.11.1) with ESMTP id f1GA72H95915; Fri, 16 Feb 2001 02:07:02 -0800 (PST) (envelope-from jkh@winston.osd.bsdi.com) To: adsharma@sharmas.dhs.org Cc: frussell@p1.cs.ohiou.edu, freebsd-smp@FreeBSD.ORG Subject: Re: possible problem with SMP? In-Reply-To: Message from Arun Sharma of "Thu, 15 Feb 2001 17:20:15 PST." <200102160120.RAA08029@sharmas.dhs.org> Date: Fri, 16 Feb 2001 02:07:02 -0800 Message-ID: <95910.982318022@winston.osd.bsdi.com> From: Jordan Hubbard Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > I sent patches for this (both xosview and per CPU sysctls) last year, but > no one responded. I'd agree that there is a value to keeping the commit > privileges to a few "commiters". But I find the fact that patches posted > to the list go unanswered/ignored discouraging for new contributors and > when it happens repeatedly, a bit insulting too. I'm sorry that your PRs were ignored - could you perhaps cite the PR numbers so that I can unearth these again? Sometimes people and patches simply slip through the cracks and it's not through lack of people *wanting* to keep up with submissions. There are certain frailties in volunteer-driven structures like ours and this is one of them, as much as I hate to say it. What the PR system does at least do for us is allow us to pull up the PRs in quesiton now and try to deal with them in "better late than never" fashion. If it's also clear that a lot of your PRs are logjamming in the system, asking to for such commit "privileges" yourself is the time-honored way of going about doing it. All you need is for one of the almost 250 other committers to be willing to "mentor" you in your application and that's something many of them are very willing to do. - Jordan To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Feb 16 9: 6:51 2001 Delivered-To: freebsd-smp@freebsd.org Received: from gw.nectar.com (gw.nectar.com [208.42.49.153]) by hub.freebsd.org (Postfix) with ESMTP id 96E7437B491 for ; Fri, 16 Feb 2001 09:06:47 -0800 (PST) Received: from hamlet.nectar.com (hamlet.nectar.com [10.0.1.102]) by gw.nectar.com (Postfix) with ESMTP id E7BDB19380; Fri, 16 Feb 2001 11:06:46 -0600 (CST) Received: (from nectar@localhost) by hamlet.nectar.com (8.11.2/8.9.3) id f1GH6ki90361; Fri, 16 Feb 2001 11:06:46 -0600 (CST) (envelope-from nectar@spawn.nectar.com) Date: Fri, 16 Feb 2001 11:06:46 -0600 From: "Jacques A. Vidrine" To: Terry Lambert Cc: Seth Leigh , freebsd-smp@FreeBSD.ORG Subject: Re: possible problem with SMP? Message-ID: <20010216110646.B90210@hamlet.nectar.com> Mail-Followup-To: "Jacques A. Vidrine" , Terry Lambert , Seth Leigh , freebsd-smp@FreeBSD.ORG References: <5.0.2.1.0.20010215025043.027a3008@hobbiton.shire.net> <200102160003.RAA10529@usr08.primenet.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <200102160003.RAA10529@usr08.primenet.com>; from tlambert@primenet.com on Fri, Feb 16, 2001 at 12:03:39AM +0000 X-Url: http://www.nectar.com/ Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Fri, Feb 16, 2001 at 12:03:39AM +0000, Terry Lambert wrote: > I have looked at the [Doors] mechanism. I think you will find that > they are a Solaris 2.7 introduction, but am willing to be wrong. Stevens says Doors were introduced in Solaris 2.5, and first documented in Solaris 2.6. > Doors are interesting. The thing they most resemble is a VMS > asynchronous system trap, or an NT I/O completion routine. Funny, to me it just looks like an efficient RPC mechanism that happens to do some thread management for you. > Fundamentally, this means I cross U->K to set things in motion, > then K->U, back to my program, once I have started rolling. > Later, after the ball gets to the bottom of the hill, the kernel > triggers a door, which runs K->U, causes some code to run, and > then jumps bad U->K, when it's done. Oh, I guess I see what you mean. With the typical socket IPC, there are more transitions (select and then read arguments/write results). Cheers, -- Jacques Vidrine / n@nectar.com / jvidrine@verio.net / nectar@FreeBSD.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Feb 16 10:47:12 2001 Delivered-To: freebsd-smp@freebsd.org Received: from sharmas.dhs.org (c62443-a.frmt1.sfba.home.com [24.0.69.165]) by hub.freebsd.org (Postfix) with ESMTP id 6F4B037B65D for ; Fri, 16 Feb 2001 10:47:08 -0800 (PST) Received: (from adsharma@localhost) by sharmas.dhs.org (8.9.3/8.9.3) id KAA10771; Fri, 16 Feb 2001 10:46:02 -0800 Date: Fri, 16 Feb 2001 10:46:02 -0800 From: Arun Sharma To: Jordan Hubbard Cc: freebsd-smp@FreeBSD.ORG Subject: Re: possible problem with SMP? Message-ID: <20010216104602.A10715@sharmas.dhs.org> References: <95910.982318022@winston.osd.bsdi.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2i In-Reply-To: <95910.982318022@winston.osd.bsdi.com>; from jkh@winston.osd.bsdi.com on Fri, Feb 16, 2001 at 02:07:02AM -0800 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Fri, Feb 16, 2001 at 02:07:02AM -0800, Jordan Hubbard wrote: > > I sent patches for this (both xosview and per CPU sysctls) last year, but > > no one responded. I'd agree that there is a value to keeping the commit > > privileges to a few "commiters". But I find the fact that patches posted > > to the list go unanswered/ignored discouraging for new contributors and > > when it happens repeatedly, a bit insulting too. > > I'm sorry that your PRs were ignored - could you perhaps cite the PR > numbers so that I can unearth these again? I did in a subsequent follow up. > Sometimes people and > patches simply slip through the cracks and it's not through lack of > people *wanting* to keep up with submissions. There are certain > frailties in volunteer-driven structures like ours and this is one of > them, as much as I hate to say it. What the PR system does at least > do for us is allow us to pull up the PRs in quesiton now and try to > deal with them in "better late than never" fashion. I'm not alleging malicious intent nor am I saying that the FreeBSD repository should be open to everone to commit. If I look at the changes happening to the FreeBSD source base, most of them happen from the commiters, and I see hardly any patches being posted here from outsiders and occasional tinkerers (let me say it anyway - like it happens on the Linux mailing lists). > > If it's also clear that a lot of your PRs are logjamming in the > system, asking to for such commit "privileges" yourself is the > time-honored way of going about doing it. All you need is for one of > the almost 250 other committers to be willing to "mentor" you in your > application and that's something many of them are very willing to do. > That I think would require a level of commitment that I may not be able to provide, given my day time job. All I'm saying is, please do something about: (a) Making sure that patches posted here get tested and responded to. (b) Encourage people to post patches. Some of them will be good and some bad - but as long as there is a process to filter out the bad and misguided ones, it could do wonders to the source base, as well as to the general FUD that the "BSD camp is elitist and cliquish". -Arun To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Feb 16 21:24:43 2001 Delivered-To: freebsd-smp@freebsd.org Received: from obsecurity.dyndns.org (adsl-64-165-226-49.dsl.lsan03.pacbell.net [64.165.226.49]) by hub.freebsd.org (Postfix) with ESMTP id 2291237B401 for ; Fri, 16 Feb 2001 21:24:41 -0800 (PST) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 9D23266F7F; Fri, 16 Feb 2001 21:24:40 -0800 (PST) Date: Fri, 16 Feb 2001 21:24:40 -0800 From: Kris Kennaway To: Arun Sharma Cc: Jordan Hubbard , freebsd-smp@FreeBSD.ORG Subject: Re: possible problem with SMP? Message-ID: <20010216212440.A6920@mollari.cthul.hu> References: <95910.982318022@winston.osd.bsdi.com> <20010216104602.A10715@sharmas.dhs.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-md5; protocol="application/pgp-signature"; boundary="+QahgC5+KEYLbs62" Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <20010216104602.A10715@sharmas.dhs.org>; from arun@sharmas.dhs.org on Fri, Feb 16, 2001 at 10:46:02AM -0800 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org --+QahgC5+KEYLbs62 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Feb 16, 2001 at 10:46:02AM -0800, Arun Sharma wrote: > All I'm saying is, please do something about: >=20 > (a) Making sure that patches posted here get tested and responded to. > (b) Encourage people to post patches. >=20 > Some of them will be good and some bad - but as long as there is a process > to filter out the bad and misguided ones, it could do wonders to the sour= ce > base, as well as to the general FUD that the "BSD camp is elitist and cli= quish". Well, posting patches to the list is the wrong way to go about it..as I mentioned before things which don't get submitted via the PR DB (as well as announced on mailing lists so people know about them) fall through the cracks because they are not tracked. i.e. the tracking mechanism for patches is GNATS. Kris --+QahgC5+KEYLbs62 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.4 (FreeBSD) Comment: For info see http://www.gnupg.org iD8DBQE6jgsYWry0BWjoQKURAlknAJ9oi6rXz+a9n3woXmzaK2NJFQF2LwCgiXJV mVqpPKV3vJdzbmJNjzAlQvo= =Qz/+ -----END PGP SIGNATURE----- --+QahgC5+KEYLbs62-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Feb 17 23:48:12 2001 Delivered-To: freebsd-smp@freebsd.org Received: from gtei1.bellatlantic.net (gtei1.bellatlantic.net [199.45.40.145]) by hub.freebsd.org (Postfix) with ESMTP id 3AF5B37B4EC for ; Sat, 17 Feb 2001 23:48:06 -0800 (PST) Received: from me-513q3sc0zun0.pengar.com (adsl-64-223-147-101.mannh.adsl.bellatlantic.net [64.223.147.101]) by gtei1.bellatlantic.net (8.9.1/8.9.1) with ESMTP id CAA09167 for ; Sun, 18 Feb 2001 02:48:13 -0500 (EST) Message-Id: <5.0.2.1.0.20010218021929.00aaef98@hobbiton.shire.net> X-Sender: seth-pc@hobbiton.shire.net X-Mailer: QUALCOMM Windows Eudora Version 5.0.2 Date: Sun, 18 Feb 2001 02:42:51 -0500 To: freebsd-smp@FreeBSD.ORG From: Seth Leigh Subject: Well I read the stuff, and I get it now. Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org OK, so I have been reading the articles you guys recommended about scheduler activations, and I now understand what you are talking about. I have one concern, which I haven't see addressed yet. Granted, I am not done reading the four articles I printed off, but I figured I'd post a question here about it. How are you going to implement thread time-slicing at the user level? If you don't implement preemption, are threads in the user-level thread library simply going to run as long as they want until they make a thread library call, giving the threads library a chance to block them, or until they block in the kernel, resulting in a new activation and upcall into the threads library scheduler? The gist of what I am saying is like this. Solaris, for example, by default gets a timer interupt 100 times per second which causes (unless real-time threads are running) the kernel scheduler to run and re-prioritize all the runnable kernel threads, and decide which of them will run during the next tick. Now, I assume that FreeBSD is going to continue having its kernel get these timer interupts, so that the kernel can fairly divvy up the CPUs amongst the various processes running. Now, with a kernel-threads supported thread library, all thread scheduling was simply done in the kernel, and the user-level threads library doesn't have to worry about it. Threads get time-sliced, everyone gets some time, and we're all happy. I am talking about purely One to One model here. In a Many to Many model, at least on Solaris which is the only OS I am familiar with how stuff works, in user level threads run on a given LWP until they give the threads library a chance to block them. If they don't block in the threads library, they hold onto the LWP as long as they want. This can easily result in performance which is unpredictable and undesirable. Now, take this to the scheduler activations. One a 4-way machine the kernel would provide up to 4 (the Anderson paper left the decision of how many to give a process to the kernel in its processor allocation code) scheduler activations, which provide the threads library four execution contexts in which to run threads. All scheduling of user-level threads is done by the threads library onto the available scheduler activations. Now, how are you going to time-slice threads at user-level onto the available scheduler activations? If the answer is "we're not, we're gonna let threads run until they block in the kernel (thus resulting in a new activation being used to upcall into the threads library, giving us a new execution context in which to run one of the other runnable threads) or in the threads library (on, say, a mutex), giving the threads library code a chance to context switch to some other thread" then I personally don't think that's a good idea. It would provide for far too much processor starvation for some of the threads. Basically, it isn't "fair". Granted, nobody said life as a thread would be fair, but still. On the other hand, setting timers to have the threads library interupt into its scheduler 100 times per second or whatever for the purposes of time-slicing threads onto the available scheduler activations seems like a *horrible* waste of time. You can lower the number of times per second, but you still waste time, and the granularity of scheduling gets coarser and coarser. Inquiring minds (well, mind) want to know. Please educate me. How are you going to time-slice user-level threads? Actually, I just thought of how it could be done while re-reading my post prior to sending it off. Is this what you guys have thought of too? The idea is that since the kernel is already getting interupted 100 times per second (or however many times FreeBSD does it) anyhow, the running scheduler activation is *already* going to be preempted off the cpu for the duration of that tick processing. So, after the tick processing is done and the kernel dispatcher decides that this particular scheduling activation may continue running as it was doing before the timer interupt fired, rather than simply context switch back into that particular scheduler activation, the kernel would use a *second* scheduler activation and upcall into the threads library's scheduler. This would basically allow the threads library's scheduler to "piggyback" onto the kernel's scheduler, without requiring anymore crossings of the protection boundaries than were going to be had anyhow. Basically, this scheme would use twice as many scheduler activations as it wanted to really have be run, basically using half of them to call up into the threads library after each tick to decide whether to keep running the preempted thread or scheduler a different one. What do you all think? Or is this already the plan? Seth To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message