Date: Thu, 26 Apr 2001 22:36:39 -0700 From: Julian Elischer <julian@elischer.org> To: Daniel Eischen <eischen@vigrid.com> Cc: Arch@freebsd.org, alfred@freebsd.org, Robert Watson <rwatson@freebsd.org> Subject: Re: KSE threading support (first parts) Message-ID: <3AE90567.CA50293E@elischer.org> References: <Pine.SUN.3.91.1010426145508.194A-100000@pcnet1.pcnet.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Daniel Eischen wrote:
>
> On Thu, 26 Apr 2001, Julian Elischer wrote:
> > Daniel Eischen wrote:
> > > I still don't see the need to have multiple KSEs within a KSEG ;-)
> >
> > KSEs in the same KSEG are using the same pool of quanta to
> > complete KSECs.
> > SOMETHING has to hold that information. A KSE is a virtual
> > processor where a KSEG is a virtual multiprocessor.
> >
> > You allocate quanta to the KSEG. KSEs use these quanta.
> > The KSEG is competing (almost) fairly with other processes
> > in the system. If you want "system" thread scheduling, you
> > can create more KSEGs. They compete against other processes and
> > against each other for slices of the 'real' machine.
>
> Right, like I've said before, it's easier just to combine the KSEG and KSE
> into one entity and forget about fair scheduling. Limit the number of
> "combined KSE/KSEGs" to the number of processors (scope system threads
> still get their own "combined KSE/KSEG" to satisfy POSIX). The common
> case is a single processor system anyways, and in that case it doesn't
> make sense to have more than 1 KSE in a KSEG.
By grouping KSEs ito an equivalence group, I give the system enough
information to allow it to schedule resuming KSECs on an 'equivalent'
but idle KSE. i.e. a syscall may be initiated on one kse and completed on
another as long as they are in the same KSEG.
The UTS doesn't really have to think to much about this except to
know that the paralelism in a KSEG is equal to the lesser of
the number of KSEs and the number of processors. (having more KSEs
than processors is completely pointless, and I happen to think it
should not be allowed.)
I tried to get away from having a KSEG but it ended up getting
more complicated again.
Most threaded applications would have one KSEG and N KSEs
(N==num-processors)
or ONE KSEG and one KSE (which is still useful as you have N KSECs).
Few apps would have more than one KSEG and hardly any would have
more than 2.
>
> In a multiprocessor system, if you have an extra quantum, who really
> cares? If someone really does care, then it can be a kernel tunable
> or resource limit.
>
> I just don't see any benefit for the added complexity. It certainly
> is easier for the UTS with a combined KSE/KSEG.
but I think it will be simpler WITH it..
Treat the KSEs in a KSEG as interchangable. threads that go into
the system on one may be reported to have completed their syscall
on another.
>
> > > Signals should be sent (via an upcall) to the first available
> > > KSE to return to userland (return from syscall, after preemption,
> > > etc.). The userland thread scheduler will pick a thread to
> > > receive the signal. If the thread is running or in one
> > > of the scheduling queues for the current KSEG, it will
> > > be able to handle it without any other assist from the kernel.
> > > If the thread is running or in one of the scheduling queues for
> > > another KSEG, it will mark the signal pending in the target
> > > thread and "signal" the appropriate KSEG with help from the
> > > kernel (one of the new user<->kernel interfaces or syscalls).
> >
> > OK so 'signals' and everything to do with them are "Per process".
> > I may edit the patch to indicate this. This does indicate a mutex
> > with SMP so that if two processors return their KSEs to userland
> > at the same time, they don't deliver the same signal twice.
> > Can two KSEs (KSEs are on different processors) deliver
> > DIFFERENT signals to userland at the same time?
>
> I suppose they could, as long as they are delivered via an
> upcall (on the special stack used for upcalls, and the running
> thread marked as preempted). The UTS will have to use some
> locking mechanisms, but it has to do that normally anyways.
>
> > >
> > > (We may have to replace "KSEG" in the above with "KSE")
> >
> > yes, you are correct.. it should read: (I think)
> > > Signals should be sent (via an upcall) to the first available
> > > KSE to return to userland (return from syscall, after preemption,
> > > etc.). The userland thread scheduler will pick a thread to
> > > receive the signal. If the thread is running or in one
> > > of the scheduling queues for the current KSEG, it will
> > > be able to handle it without any other assist from the kernel.
> >
> > Is this what you mean?
>
> For the most part. But if the target thread is running in one
> of the other KSEs for that KSEG, then it will still require an
> assist from the kernel.
Why? Surely it is to be considered to be processing another signal.
note teh signal, and it should pick it up when it's completed the one
its doing..
>
> > This is tricky... when a KSE returns to userland it is running
> > NO threads. All threaded syscalls return to userland in the 'suspended'
> > state, so that the UTS can decide what to run.
>
> This is only when syscalls or when you need to notify the KSE of
> special events (signals, interruptions from other KSEs, etc).
> Normally, a syscall that doesn't block just returns without
> any upcall.
well, I don't know that this is true.. the thread could starve other threads
by doing only non-blocking syscalls. Ihad thought that all syscalls would return
via upcalls to allow the UTS to decide whether to pre-empt them.
>
> > All syscalls return via
> > an upcall to the UTS (actually the original newkse() call returns
> > infinitly many times.. that is how the upcall is achieved). The return
> > values, error returns and data movements have been made to the appropriate
> > memory locations.. It's as if the thread did a 'yield()' immediatly
> > after returning from a normal syscall..
> > So we can be sure that THIS KSE isn't running the interrupt thread.
>
> Running threads can cause synchronous signals, so it's quite possible
> the running thread generated the signal. The KSE in which the thread
> was running would then get the notification.
the KSE would, but when the upcall is made, the previous running thread
is made to lookas if it had yielded.. the KSE is running but the thread
is 'suspended'.
>
> I don't see any problem with this as long as interrupted thread
> contexts are available to the UTS.
>
> > If the thread is however being run on a different KSE (regardless of
> > whether in this KSEG or not) then the signal must be noted so that
> > the thread can see it at some future time. If it's not running but in
> > another KSEG then it's treated as if running, (the signal noted) and the
> > UTS will make it runnable at the next opportunity that that KSEG
> > is runnable. (If we ran the thread on this KSE regardless of the fact
> > that it's from another KSEG, then it will be running with a priority
> > other than what the programmer assigned it. (maybe he wants lower
> > priority signal handling)).
>
> 1996 POSIX spec says that signals should be delivered "as soon as
> possible". This leaves some leeway (I'll have to see if Austin
> changes any of this), but my approach in the current threads library
> is to deliver the signal right away unless the thread is in a critical
> region (in which case the signal is delivered when it exits the
> critical region).
>
> > > If the thread is running or in one of the scheduling queues for
> > > another KSE, it will mark the signal pending in the target
> > > thread and "signal" the appropriate KSEG with help from the
> > > kernel (one of the new user<->kernel interfaces or syscalls).
> >
> > If a KSEG is not running because it had no work, then
> > yes, you need to wake up one of its KSEs to handle the signal.
>
> Yeah, but I was thinking more along the lines of interrupting a
> currently running KSE.
If you are returning from the kernel, all threads are effectively
suspeended and equivalent. The KSE is always 'idle' when returning
to the UTS. The UTS then decides what to run next.. (usually the
thread that was just active, but not always.)
>
> > > It might be nice to have a general way of sending messages
> > > between KSEGs (KSEs?).
> >
> > Userland-to-kernel? or userland-to-userland?
> > "kind of like a signal?" :-)
>
> Userland to userland with an assist from the kernel. KSE A wants
> to interrupt KSE B and send B an upcall message of some sort.
> The UTS knows what the message format is, but the kernel doesn't
> need to know other than possibly its message type and size.
fair enough.
>
> --
> Dan Eischen
--
__--_|\ Julian Elischer
/ \ julian@elischer.org
( OZ ) World tour 2000-2001
---> X_.---._/
v
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3AE90567.CA50293E>