Date: Wed, 24 Nov 1999 02:04:15 -0800 (PST) From: Julian Elischer <julian@whistle.com> To: "Daniel M. Eischen" <eischen@vigrid.com> Cc: freebsd-arch@freebsd.org Subject: Re: Threads Message-ID: <Pine.BSF.4.10.9911232348550.11412-100000@current1.whistle.com> In-Reply-To: <3837F679.BCEC1312@vigrid.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, 21 Nov 1999, Daniel M. Eischen wrote:
> Julian Elischer wrote:
> >
> > On Sun, 21 Nov 1999, Daniel M. Eischen wrote:
> > What better way to install them than to do a syscall down?
> > All the kernel needs to know is that it sets some stuff, pre-empts the
> > present KSE and runs another.
>
> I assumed that the UTS would have to perform a system call to
> install the UTS upcall entry point(s).
yes. There is a lot of information that can be supplied just in the
action of making that call. You have a return address and some stack.
I'd do it so that the context that the call saves to return to is the
majority of the context that an upcall would get.
If the upcalls all jumped in just as though they had made the syscall,
then they have access to a bunch of stuff that can be easily tailored.
e.g. local and static variables etc. To work with it you don;t need to be
an expert in stack frames etc. You just need to know that you call a
function that throws you onto a new stack, instead of returning. So the
code for your prototypical async handler might be:
/*
* This function is called with the name of the function you want the
* subproc to start its first thread on. This is how you start a new
* subproc off on it's first thread. This stack is reserved for
* returning async events FOR THIS SUBPROC. Remember each subproc can
* have only one running KSE at a time, so the contents that the kernel
* places in sc_state for the async handler are safe from other returning
* events for othe rsobprocs and hopefully from other events associated
* with this subproc (we hope)
*/
void
handler (void startlocation(void *), void *arg)
{
struct syscall_state Asc_state;
upcall_reason_t reason;
reason = register_upcall(&syscall_state);
switch (reason) {
case ORIGINAL:
newthread(startthread, startlocation, arg)
/* notreached */
case INTERRUPT:
newthread(signalhandler, sc_state, NULL);
/* notreached */
case IO_COMPLETION:
newthread(finish_syscall, sc_state, NULL);
/* notreached */
case IO_BLOCKED:
find_runnable_thread();
/* notreached */
}
}
> > >
> > > That's OK, but why not use a ucontext passed in with a new system
> > > call? Make the UTS supply ucontexts (made with makecontext) for event
> > > notifications also.
I think that the easiest context to create is one that you create by just
being there. It's self correcting in the face of compiler changes etc.
> >
> > I'm not that fussed about it but it just seems to me that the information
> > being passed back and forth is DIFFERENT than that being passed back and
> > forth in the non-threaded world.
>
> True, but it is passed back and forth in NEW ways. It doesn't affect
> non-MT system calls.
That's where I disagree.
I think nearly all syscalls can block given page faults etc. and having all
syscalls potentially return via the UTS is going to mean some change
in the kernel/process protocol.
If a blocked syscall returns, then when it returns the UTS needs to
be able to decide whether it is the most important thread to continue or not.
So it can't just 'return', but has to come back via the UTS. This requires
that things be considerably different. At least this is how I see it.
> > I don't want to get into the trap of deciding that we are doing
> > everything the "scheduler activations way", because we may be doing some
> > things quite different.
>
> Granted, but let's also not let NIH have any impact on what we decide ;-)
If we had that at the moment we wouldn't be discussing this the way we are..
>
> > My philosophy is that things should be precomputed early and when they are
> > required they are just 'used'.
>
> Agreed.
>
> > Here is a first half-thought out run-threough of how an IO syscall blocks
> > and the UTS gets notified.
> > ---------------------
> >
> > in a syscall,
> >
> > the thread loads some values on the user stack
> >
> > the thread calls the ULT aware syscall handler
> >
> > The ULT allocates an IO status block
> >
> > the ULT sycall handler calls the new callgate
> >
> > The callgate causes the user stack to be saved and registers to be saved,
> > etc. the kernel stack is activated.
> >
> > a flag (a) is set.
> >
> > a setjmp() is done and saved at (b) and the stack is marked at (c)
> >
> > syscall is processed......
>
> This seems overly complicated. What if the system call doesn't
> block? You've just done all the above for nothing.
the only extra work is the setjmp.
The rest already occurs to some extent in the preent system.
There is already some marshalling of arguments etc before calling the
kernel or doing the syscall. it's just slightly different.
The setjmp may not be needed, but some information needs to be done.
Maybe a dummy frame is pushed (see later)
> >
> > at some point a tsleep() is hit.
> >
> > it notices that flag (a) is set, and
> >
> > grabs a new KSE from the KSE cache.
> >
> > copies the kernel stack up to to (c) to that on the new KSE
> >
> > copies the Async KSE's stack to (c) onto the old KSE
>
> Why not just context switch to the cached context beginning
> at the UTS defined async notification entry points?
Because it has to get some information about the syscall that blocked, and
if it just took a copy of the stack as if it had just Made the call
itself, it has that all available.
The second time through (on the longjmp()) it goes to the 'it blocked' code
rather than processing the request. It can set the 'it blocked' status on the
IO status block allocated to that thread, and jump to the async
return.
The stack and a jetjmp contain all the information that might be required, so
why not make use of it?
Remember once this has happenned, the actual syscall is not going to
return via that path anyhow, as it should go back via the UTS.
>
> >
> > hacks the contents of (b) to fit the new KSE
> >
> > does a longjmp().. We are now effectively the thread returning
> >
> > Set 'incomplete' into the IO status block.
> >
> > * some stack munging thing done here I am not sure about *
> >
> > returns to the UTS indicating thread blocked
> >
> > * some user stack munging thing done here I am not sure about *
> >
> > UTS suspends thread
> >
> > UTS schedules new thread
> >
> > ***Time passes****
> >
> > down in the bowels of the system the original KSE unblocks..
> >
> > control passes back up to the top of the kernel.
> >
> > It sets the status of the IO as it would have, then returns,
> >
> > and finds itself in the ASYNC entrypoint of the UTS. (or maybe on the
> > original return path.. depends on state of stack)
> >
> > the UTS notices that the IO has completed, and schedules the original
> > thread to continue. The KSE then runs that thread until the UTS says
> > otherwise.
>
> I think the above approach is too complicated and can be simplified
> to eliminate the overhead in setting up the system call.
I don't think there's much overhead other than allocating an
IO status block in teh thread's last stack frame.
>
> Here's my thoughts.
>
> At thread library intialization time, the UTS makes contexts for
> async notification event handlers. Contexts are in the form of a
> return from a trap to the kernel. My idea is to use setcontext(2)
> and makecontext(2) to create the contexts. We may need to give
> the kernel additional user stacks, but that isn't clear yet. Flag
> (async_notify) is set in the calling process.
I think we are in violent agreement here, it's just implementation details.
>
> A thread makes a system call and it blocks; tsleep is hit.
> (Notice that nothing needs to be done differently in order
> to make a system call.)
>
> It notices that flag (async_notify) is set. It grabs a new
> KSE (kernel register save area and kernel stack) from the KSE
> cache, and places the current KSE in the sleep queue.
exactly.
>
> A unique ID is created to identify the blocked KSE (integer
> ID?). This unique ID, along with an ID that identifies the
> event and any other necessary information, is copied to the
> top of the user stack of the relevent async notification
> event handler. The parameters to the UTS event handler are
> setup to point to the top of the stack.
See my example earlier..
>
> The kernel returns directly to the UTS entry point on the
> predefined context/user stack, just as if a setcontext(2)
> was performed.
Basically that's what I wan texcept that the syscall that sets the
async handler returns multiple times.. Including when a blocked
syscall resumes.
>
> The UTS knows what thread was running and marks it as blocked
> in the kernel and saves the unique KSE ID.
>
How does it know what was running?
> The UTS schedules a new thread.
>
> ***Time passes***
>
> The original KSE unblocks. It is added to the current processes
> unblocked KSE list.
>
> When the process runs again (or the next subprocess, since you
> can have more than 1 cooperating processes), a new KSE is
> allocated and context is set to the relevent UTS event handler.
> Event information is copied out to the top of the predefined
> user stack, parameters set accordingly, and the kernel returns
> to the UTS event handler. If a thread was preempted for the
> notification, then its context (just like getcontext(2)) is
> also copied out to the UTS event handler user stack.
I think we agree here.
My only point is that a lot of this can be achieved using stack munging and
that we don't need to create a lot of special frame handling and
context saving if we use the saved state that si already existant in the stacks.
>
> The UTS event handler receives notification that the thread that
> _was_ running on this process was preempted for this notification,
> and that other KSEs have unblocked in the kernel. The UTS marks
> the unblocked threads has active. The context of the thread that
> was running can be resumed with setcontext(2) or hacked into a
> jmp_buf and resumed with longjmp.
>
Yes, sure.
> The UTS resumes the blocked thread in one of two ways. A new
> system call to resume the KSE identified by it's unique ID is
> one way. The other way is to have the kernel copy the context
> of the unblocked KSE out to the UTS event handler. It can be
> resumed with setcontext(2).
I think that I'd like to see this shown with simulated stacks and such
in pictures.. I think you and I are in agreement, but having
trouble saying that.
>
> Dan Eischen
> eischen@vigrid.com
>
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.BSF.4.10.9911232348550.11412-100000>