Date: Tue, 28 Nov 2000 19:15:19 -0500 (EST) From: Daniel Eischen <eischen@vigrid.com> To: Julian Elischer <julian@elischer.org> Cc: arch@FreeBSD.ORG Subject: Re: Thread-specific data and KSEs Message-ID: <Pine.SUN.3.91.1001128181622.9762A-100000@pcnet1.pcnet.com> In-Reply-To: <3A242FAF.313295F0@elischer.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 28 Nov 2000, Julian Elischer wrote:
> Daniel Eischen wrote:
> > I think we want to avoid saving and restoring FP state unless it's
> > necessary. That's probably only when a fault occurs or when the
> > KSE is preempted. I like the idea of having the kernel save the
> > FP state in the thread state storage area (ucontext_t?) in the
> > KSE mailbox thingy.
>
> The question is, what happens to the FPU context when you swap threads?
> should each thread have it's own FPU context?
Yes, it currently does now. It's hard to imagine every thread
not having FPU context. A thread that is preempted needs to
have it's context restored when it runs again. You seem to
be forgetting that the UTS can choose another thread to run
and _not_ resume the preempted thread. So the UTS chooses
another thread, runs it, and then you can have another fault
or preemption which needs to save FP state again.
> If there is one for the KSE, might that be not enough?
> especially if the KSE was pre-empted. If a thread is migrated to
> another KSE, having last been pre-empted, it becomes important
> that the FPU state go with it because it may have been part way
> through some calculations when that KSE was stopped. And what if
> the new KSE already has one that wsa stopped in thasame way?
> it looks to me like you need to have one per thread.
Ahh, exactly.
> It's not much different fromt eh point of view of the kernel.
> WHen you create a KSE you give it's mailbox.
> when you schedule a thread onto the KSE you set a pointer in that
> mailbox to the thread's context and state storage area.
> The kernel can easily follow that link when it pre-empts the KSE
> to store the General regs, the FPU regs etc.
> Theoretically it might only store the regs there in a syscall
> if it looks like the syscall will block. but the aim would be
> to make allthreads look the same when stopped so that the UTS
> can restart any one it chooses.
That would be my goal also.
> > Also, are we going to allow the kernel to follow links out of
> > the mailbox, or are we going to limit UTS<->kernel communication
> > to just this one page? I think it might be preferable to only
> > communicate via the mailbox and never have the kernel attempt
> > to read/write to other areas of KSE/thread storage.
>
> The kernel already has to follow links etc for (for example)
> the readv() syscall. it's not that big a step. If you allocate
> all the thread-context blocks together, the pages they are in will
> be pretty hot. There are great advantages to having the KSEs
> being able to follow links. For example it means that the kernel
> can ALWAYS deliver a linked list of ALL completed and 'ready-to-run'
> threads. It can set them up so that each one will look exactly
> as if it has just returned from the syscall. If you only deal
> with one structure, you have to consider what happens when you
> cannot fit all returning threads into the single structure.
You should only have to copy context back to the KSE for
one thread. When a thread blocks in the kernel, you don't
need to wait to copyout it's context. You can do it immediately.
All you need to pass out when the thread is resumed in the
kernel and ready to return to userland is the return value
from the system call. For faults and preemptions there are
no return values I'd guess.
> As the kernel takes control (as the syscall or trap is entered)
> it notes where the context block is and when and if it decide
> it needs to save context, it knows where to put it. The UTS is
> given everything on a plate, and it's almost easier to do it
> this way for the kernel too. It can store this address with
> the KSEC and use it without any fear of ever having a clash with
> some other returning syscall (for example). I can imagine where
> a syscall starts on one KSE and is completed on another.
> it makes sence for the context to travel with the thread/KSEC
> rather than the KSE, which may suddenly have 4 syscalls
> all coming back within the same upcall. (where do you save
> all that data?)
Here's the way I see it. A thread blocks in the kernel, or
is preempted, has a fault. "There can be only one" (name the
movie!) thread running in the KSE at a time. You copyout
the context to the KSE, _then_ make the upcall. The KSE
upcall handler then copies the context (along with FP
state if saved) to the threads context storage area. Another
thread is chosen and executed. The kernel need not follow
links, the KSE upcall handler can handle placing the context
in the threads storage area.
When a thread becomes unblocked in the kernel, the UTS
already has its context. All it needs now is enough information
for the return value of the system call. If the UTS has to
munge with the context a bit, it can.
This will let you set the 1 page used for the mailbox so that
it won't be paged out and not worry about whether memory anywhere
else in the UTS/thread is paged out.
> > For instance,
> > we could place the pointer to the thread state storage area
> > in the mailbox. But that would require a copyin, and then a
> > copyout to another page that might be paged out.
>
> Since the thread storage is part of the thread control block
> that the UTS has just used to schedule the thread, it's
> unlikely to be paged out. Even less so if it shares a page
> with other thread control blocks
> And you could always protect it with madvise().
>
> In any case what you suggest above is EXACTLY what jason and I
> were planning on doing. The kernel will define a structure
> in /sys/i386/include/kse.h (or somewhere) called something like
> struct user_process_context
> which you would include in your thread control block.
> it would include a link to other such blocks (so we can
> return a linked list of completed or pre-empted threads (KSECs)
> and a status word that says whether it is a completed syscall, or a
> preempted thread, or whatever, and a cookie that the kernel
> doesn't touch so you can extend it with whatever else you need.
Well, if you want to take the extra step of performing a copyin
and following a link, I can live with that. But, I wouldn't wait until
the thread becomes unblocked to copyout its context. Just
do it immediately and its much easier.
Actually, you have to copyout its context immediately. Let's
say a thread blocks in the kernel on a read(). Then let's say
the thread is sent a signal (and sa_flags is SA_RESTART). The
UTS needs the threads context (at least the stack pointer) so
it can create a signal frame on top of its stack. The signal
handler will run in the context of the thread while it is still
blocked in the kernel. The thread will also need to use its
context storage area because it may again be preempted or
blocked. I've spent considerable time trying to get signal
handling working correctly in our threads library, and this
is about the only way that really works.
Actually, there is even another problem. Suppose you have:
static pthread_t tid;
static jmp_buf jmpbuf;
static void sighandler(int signo)
{
if (signo == SIGALRM && pthread_equal(pthread_self(), tid))
_longjmp(jmpbuf, 1);
}
my_thread(void *arg)
{
char buf[128];
int fd = (int)arg;
int ret;
tid == pthread_self();
if (_setjmp(jmpbuf) == 0) {
ret = read(fd, buf, sizeof(buf));
}
else {
printf("Thread is exiting.\n");
pthread_exit(NULL);
}
}
pthread_kill(tid, SIGUSR1);
...
pthread_kill(tid, SIGALRM);
...
This is perfectly valid. And you can also have compilers that generate
builtin longjmps for exception handling. In that case, we can't even
wrap longjmp/_longjmp in order to do cleanup handling. So the kernel
still thinks the read() is active. All sorts of gnarly stuff can happen.
I think we need a way to tell the kernel to halt any pending activities
for the KSEC that was blocked before trying to deliver any signals.
If the thread returns normally from the signal handler, then the
KSEC can be resumed. In the case of an abnormal return/jump out
of the signal handler, I don't know how we'd inform the kernel that
the KSEC could be reused; the UTS doesn't know if the thread is
still operating in the signal handler or has jumped out of it.
It'd be nice if the UTS could retrieve the KSEC state/storage.
If it could, the UTS could copy it to the signal handling frame
so that a normal return from the signal handler could pass it
back to the kernel.
We need to work this out.
> > The drawback
> > of only using the mailbox is that it requires an additional copy
> > by the UTS every time an upcall is made (to copy the thread state
> > from the mailbox to the storage area in the thread).
>
> You forget that a single upcall may want to return 37 completed
> syscalls or pre-empted threads.
Already explained above.
> Using my scheme.. there is an upcall, and bingo the UTS has ALL
> the completed items at once.
> It sorts them onto the runnable queues, selects it's favourite
> and puts that address into it's mailbox, loads the context,
> and it's off an running the next thread.
--
Dan Eischen
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.SUN.3.91.1001128181622.9762A-100000>