Date: Thu, 05 Nov 98 14:34:03 -0600 From: "Richard Seaman, Jr." <lists@tar.com> To: "Peter Wemm" <peter@netplex.com.au> Cc: "current@FreeBSD.ORG" <current@FreeBSD.ORG> Subject: Re: Kernel threading (was Re: Thread Scheduler bug) Message-ID: <199811052034.OAA14134@ns.tar.com>
index | next in thread | raw e-mail
[-- Attachment #1 --]
On Thu, 05 Nov 1998 22:28:46 +0800, Peter Wemm wrote:
>Also, while on the subject, something Julian said has got me thinking about
>creating kernel threads on the fly when and if a user thread happens to
>block. This sounds rather interesting.. - async IO is done this way too I
>think. It would require a fair amount of cooperation between the thread
>"engine" and the kernel, perhaps by having an executive thread of sorts
>that handled the kernel interation and thread activation.
I attached a message from Martin Cracauer (and the attachment to that
message) from about 2 1/2 months ago on this subject.
[-- Attachment #2 --]
Return-Path: owner-freebsd-current@FreeBSD.ORG
Return-Path: <owner-freebsd-current@FreeBSD.ORG>
Received: from sarip.sol.net (mail@sarip.sol.net [169.207.30.120])
by ns.tar.com (8.8.8/8.8.7) with ESMTP id HAA04550
for <lists@tar.com>; Wed, 26 Aug 1998 07:36:22 -0500 (CDT)
Received: from hub.freebsd.org (hub.FreeBSD.ORG [204.216.27.18])
by sarip.sol.net (8.8.8/8.8.8/SNNS-1.02) with ESMTP id HAA10199;
Wed, 26 Aug 1998 07:32:20 -0500 (CDT)
Received: from localhost (daemon@localhost)
by hub.freebsd.org (8.8.8/8.8.8) with SMTP id FAA19457;
Wed, 26 Aug 1998 05:09:51 -0700 (PDT)
(envelope-from owner-freebsd-current)
Received: by hub.freebsd.org (bulk_mailer v1.6); Wed, 26 Aug 1998 05:07:54 -0700
Received: (from majordom@localhost)
by hub.freebsd.org (8.8.8/8.8.8) id FAA19159
for freebsd-current-outgoing; Wed, 26 Aug 1998 05:07:53 -0700 (PDT)
(envelope-from owner-freebsd-current@FreeBSD.ORG)
Received: from gilgamesch.bik-gmbh.de (gilgamesch.bik-gmbh.de [194.233.237.91])
by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id FAA19150;
Wed, 26 Aug 1998 05:07:50 -0700 (PDT)
(envelope-from cracauer@gilgamesch.bik-gmbh.de)
Received: (from cracauer@localhost) by gilgamesch.bik-gmbh.de (8.8.8/8.7.3) id OAA20913; Wed, 26
Aug 1998 14:06:43 +0200 (MET DST)
Message-ID: <19980826140642.A20511@cons.org>
Date: Wed, 26 Aug 1998 14:06:42 +0200
From: Martin Cracauer <cracauer@cons.org>
To: Mike Smith <mike@smith.net.au>, Michael Hancock <michaelh@cet.co.jp>
Cc: Gary Palmer <gpalmer@FreeBSD.ORG>, Chuck Robey <chuckr@glue.umd.edu>,
freebsd-current@FreeBSD.ORG
Subject: Re: Threads across processors
References: <Pine.SV4.3.95.980826000204.19157C-100000@parkplace.cet.co.jp>
<199808251338.NAA02533@dingo.cdrom.com>
Mime-Version: 1.0
Content-Type: multipart/mixed; boundary=2oS5YaxWCcQjTEyO
X-Mailer: Mutt 0.93.1i
In-Reply-To: <199808251338.NAA02533@dingo.cdrom.com>; from Mike Smith on Tue, Aug 25, 1998 at
01:38:18PM +0000
Sender: owner-freebsd-current@FreeBSD.ORG
X-Loop: FreeBSD.ORG
X-UIDL: 0566b91027b7d922b61cf63f5acf6a23
In <199808251338.NAA02533@dingo.cdrom.com>, Mike Smith wrote:
> > On Tue, 25 Aug 1998, Gary Palmer wrote:
> >
> > > Heck, SMI wrote `doors' for the very reason that IPC *blows* in all cases, and
> > > that to pull off the speedups with NSCD that they wanted, they had to get the
> > > IPC overhead reduced a lot. I think I even have slides somewhere comparing
> > > pipes, SYSV SHM, etc times for message passing in terms of transit time.
> >
> > Our pipes are very fast. SYSV SHM's blunder is that it uses full blown
> > system calls for synchronization.
Aehm, and pipes don't require full-blown system calls to send/receive
notifications and require kernel resheduling before anything happens
after sending a message?
> Yes. Anyone that thinks in terms of a context switch per transaction
> between coprocesses is not designing properly.
For your amusement, I appended a message I once forwarded to -hackers,
regarding mapping of userlevel threads to kernel shedulable entities.
But in a way that is like sendfile and other combined system calls:
Did anyone actually gain any data how much slower a
one-process-per-thread model is? For any application?
> Using a shared mmap()
> region and datastructures that don't require locking is another
> cost-effective technique.
I'm afraid I have to count this as *very* cheap shot :-)
Martin
--
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Martin Cracauer <cracauer@cons.org> http://www.cons.org/cracauer
BSD User Group Hamburg, Germany http://www.bsdhh.org/
[-- Attachment #3 --]
I wrote:
> > I still like the simpliticity of a kernel-only thread solution. If
> > that way turns out to be too inefficient, the DEC way seems to be a
> > solution that doesn't need async system calls and has no efficiency
> > disadvantage I can see (compared to a sysyem with async syscalls
> > only).
> >
> > I hope to get further details on the DEC implementation.
Here's what Dave Butenhof <butenhof@zko.dec.com> told me about DEC's
interface. Dave implements the userlevel part of DEC's thread
interface. This answer is a bit out of context, if you need the
previous discussion, please let me know.
My original question was how blocking syscalls are treated. In Digital
Unix, the kernel reports a blocking syscall in one thread back to the
userlevel library, which reshedules another thread on that "kernel"
thread. I asked for further details how a userlevel library could get
rid of an already blocking syscall, and here's what I heared:
> ~Message-Id: <32A5797C.6231@zko.dec.com>
> ~Date: Wed, 04 Dec 1996 08:15:40 -0500
> ~From: Dave Butenhof <butenhof@zko.dec.com>
> ~Organization: Digital Equipment Corporation
> ~To: cracauer@wavehh.hanse.de
> ~Cc: "Butenhof, Dave" <butenhof@zko.dec.com>
> ~Subject: Re: Blocking syscall handling in one-to-many thread implementations
>
> [...]
>
> [This was my, Martins, question]
> > But how exactly is resheduling on your KECs done? If a KECs is waiting
> > in a blocking syscall, how can the userlevel sheduler reassign it? How
> > can the userlevel library free it from the syscall?
> >
> > And what happens to the syscall? Is it translated into a non-blocking
> > version and the kernel informs the userlevel sheduler when it arrives?
>
> I was trying to describe what happens from YOUR perspective, more than
> what actually happens in the thread library & kernel. The internals are,
> as always, a little more complicated.
>
> The thread library maintains a constant pool of (up to) <n> KECs, where
> <n> is normally set by the number of physical processors in the system.
> (It may be smaller, if your process is locked into a processor set.)
> These are the "virtual processors" (VPs). The thread library schedules
> user threads on the pool of VPs, trying to keep all <n> of them busy. If
> you don't have enough user threads to keep that many VPs busy, they may
> not all get started, or VPs already running may go into the "null
> thread" loop and idle -- which returns the KEC to the kernel's pool for
> reuse. We'll get it back if we need it later.
>
> When a thread blocks in the kernel, the KEC stays in the kernel, but it
> gives us an upcall in a *new* (or recycled) KEC to replace the VP. When
> the blocking operation finishes, the kernel gives us a completion upcall
> in the original KEC. It's no longer a VP, so we just save the context
> and dismiss it.
>
> The key is the distinction between "KEC" and "VP". There may be 100 KECs
> attached to a process, but, on a typical quad-CPU system, only (up to) 4
> of them at any time are "VPs". The rest are just holding kernel context
> across some blocking operation. Whereas, in a strict kernel-mode
> implementation, each user thread is a KEC, we have a KEC only for each
> running thread and each thread blocked in the kernel. The number of KECs
> will fluctuate -- and, if you hit your quota for KECs (Mach threads),
> any additional kernel blocking operations will start to lower
> concurrency. The most common blocking operations, mutexes & condition
> variables (and some others), however, are completely in user mode.
>
> We're going to continue streamlining the process as we go (like moving
> some kernel blocking states out into user mode, and reducing the context
> that the kernel needs to keep below a full KEC), but, in general, it
> works very well. The kernel developer and I (I work mostly in the
> library) have kicked around the idea of doing a paper on the design.
> Mostly, I've been too busy with a book for a ridiculously long time, and
> the development requirements never stop. Maybe some day. Possibly once
> we've gone through a full release cycle and have the architecture
> stabilized better.
>
> /---[ Dave Butenhof ]-----------------------[ butenhof@zko.dec.com ]---\
> | Digital Equipment Corporation 110 Spit Brook Rd ZKO2-3/Q18 |
> | 603.881.2218, FAX 603.881.0120 Nashua NH 03062-2698 |
> \-----------------[ Better Living Through Concurrency ]----------------/
>
>
help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199811052034.OAA14134>