Date: Thu, 24 Jul 2003 13:59:28 -0700 (PDT) From: Julian Elischer <julian@elischer.org> To: deischen@freebsd.org Cc: marcel@xcllnt.net Subject: Re: KSD/TSD take 2 (was: KSE critical regions) Message-ID: <Pine.BSF.4.21.0307241229170.69488-100000@InterJet.elischer.org> In-Reply-To: <Pine.GSO.4.10.10307241018120.17678-100000@pcnet5.pcnet.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 24 Jul 2003, Daniel Eischen wrote:
> >
> > /*
> > * Thread mailbox.
> > *
> > * This describes a user thread to the kernel scheduler.
> > */
> > struct kse_thr_mailbox {
> > uint32_t tm_flags; /* Thread flags */
> > struct kse_thr_mailbox *tm_next; /* Next thread in list */
> > void *tm_udata; /* For use by the UTS */
> > uint32_t tm_uticks;
> > uint32_t tm_sticks;
> > register_t tm_spare[4];
> > ucontext_t tm_context; /* User and machine context */
> > siginfo_t tm_syncsig;
> > };
>
> We need to allocate a few spare ptrs for TLS at the beginning.
> I think tm_context should be at the end since it is the thing
> most likely to change. tm_context will be aligned because
> it contains an alignment clause for the FPU state, at least
> for i386, and should for other archs if it is necessary.
> We also need some way to tell if kse_thr_mailbox is a fake
> mailbox or not. We could either use a version/type field
> or a second tm_udata.
No this is not quite right..
(CC'd to threads so that we have a record of this...)
The TLS spec specifies a few things..
on x86,
the memory segment defined by the LDT entry referenced by %gs
must have at offset 0, the address of the thread control block (TCB),
and at
some offset in the TCB is a pointer to the Dynamic
Thread Vector (dtv). The pointer is at offset 'dtpoff' (a symbol)
from the base of the Thread Control block. "Static" (as in defined in
modules linked statically with the main module) __thread variables
are defined to be contiguously allocated BEFORE (i.e. at -ve offsets)
the Thread control block.. The DTV contains pointers to subsections
of the tls storage (even to noncontiguous parts) needed by the linker
and to resolve addresses at runtime.
in pictures:
[LDT 1]
(%gs=X*8) [LDT 2]
...
[LDT X-1]
[%gs]----(index)--->[LDT X]------+
[LDT X+1] |
|
+---------------+
V
--------------------------------------------------------------
| | | | | | | | | | | | | | | Z
--------------------------------------------------------------
| | XXX THIS AREA NOT DEFINED | Z segment 'g'
-------------------------------------
|
|
+---------------+
|
V
--------------------------------------------------//------------
Z | | | | | | | | | | | | | | Z
--------------------------------------------------//------------
"statically linked" TLS storage # TCB contents Z TCB
--------------------------------------------------//------------
^ <--- dtpoff-->|
| |
| +---------------+
| |
+-------------------)------+
| |
V |
--------------------------------------------------//------------
Z | | | | | | | | | DTV entries | Z
--------------------------------------------------//------------
|
|
+---------------------------------------+
|
V
--------------------------------------------------//------------
Z | | | NON-Contiguous (dynamically linked TLS storage)|
--------------------------------------------------//------------
Under the KSE library, the area marked "XXX" is the KSE MAILBOX
Under libthr it is part of an array of thread pointers.
(i.e this works correctly for both libraries)
On ia64 it is slightly differnt.. Ther eis an actual tp register
(I believe)
TP
|
V
--------------------------------------------------//------------
Z | | | | | | | | | | | | | | Z
--------------------------------------------------//------------
| TCB contents (*) | |XXX|"statically linked" TLS storage
--------------------------------------------------//------------
| | ^
| | |
V +---------------+ |
[KSE mbox] | |
| +--+
| |
V |
--------------------------------------------------//------------
Z | | | | | | | | | DTV entries Z
--------------------------------------------------//------------
|
|
+---------------------------------------+
|
V
--------------------------------------------------//------------
Z | | | NON-Contiguous (dynamically linked TLS storage)|
--------------------------------------------------//------------
Under the KSE library, the area marked "XXX" is undefined and could be
either a pointer to the KSE mailbox or the TCB. (which would contain a
pointer to the KSE mailbox). In either case the pointer to the KSE
mailbox would be set by the UTS when sceduling that thread. it is
important to note that the kernel doesn't need or read this pointer as
it already knows where the KSE mailbox is, and the UTS upcall doesn't
need it as it gets the mailbox address as an argument when upcalled. It
is only required by the UTS in the case that it is enterred from the
thread directly.
(*) The TCB could be anywhere, and the XXX area could be used to point
to it, however it is probably more efficient to allocate the (static)
TLS immediatly following the TCB and do it in a single allocation,
and use -ve offsets from the TP to get to it. (i.e. the TP points
between the TCB and the (static) TLS.
ON non-i386, and non ia64 systems there is a hybrid.
as follows
TP
|
+--------------------+
|
V
--------------------------------------------------//------------
Z | | | | | | | | | | | | | | Z
--------------------------------------------------//------------
"statically linked" TLS storage # TCB contents Z TCB
--------------------------------------------------//------------
^ <--- dtpoff-->| |
| | |
| +---------------+ |
| | V
+-------------------)------+ [KSE mbox]
| |
V |
--------------------------------------------------//------------
Z | | | | | | | | | DTV entries Z
--------------------------------------------------//------------
|
|
+---------------------------------------+
|
V
--------------------------------------------------//------------
Z | | | NON-Contiguous (dynamically linked TLS storage)|
--------------------------------------------------//------------
>
> struct kse_thr_mailbox {
> void *tm_tls[4]; /* reserved for TLS */
> uint32_t tm_flags;
> uint32_t tm_version;
> struct kse_thr_mailbox *tm_next;
> void *tm_udata;
> uint32_t tm_uticks;
> uint32_t tm_sticks;
> register_t tm_spare[4];
> siginfo_t tm_syncsig;
> ucontext_t tm_context;
> };
looking at the above diagrams, we see:
struct kse_thr_mailbox need not have any TLS stuff. The offset to the
DTV pointer is relative to whatever the TP is pointing to, which MAY
be the mailbox or it may be the larger TCB (which might
contian the thread mailbox but may not) In anycase there is no reason
for the mailbox to contain a TLS entry becauee the TLS is not something
that we are communicating to the kernel.
In the i386 case, (and only the i386 case (unless the amd-64 case is the
same)) the KSE MAILBOX is what we are pointing %gs:0 at, and in that
case, the pointer to the TCB (not the thread mailbox) is stored there.
(and set there by the UTS when scheduling a thread).
Thus the struct kse_mailbox would have:
struct kse_mailbox {
#ifdef __i386__
void *TLS_tcb; /* current TCB for TLS */
#endif
uint32_t km_version; /* Mailbox version */
uint32_t km_flags; /* KSE flags */
struct kse_thr_mailbox *km_curthread; /* Currently running thread */
struct kse_thr_mailbox *km_completed; /* Threads back from kernel */
[...]
>
> > /*
> > * KSE mailbox.
> > *
> > * Communication path between the UTS and the kernel scheduler specific to
> > * a single KSE.
> > */
> > struct kse_mailbox {
> > uint32_t km_version; /* Mailbox version */
> > uint32_t km_flags; /* KSE flags */
> > struct kse_thr_mailbox *km_curthread; /* Currently running thread */
> > struct kse_thr_mailbox *km_completed; /* Threads back from kernel */
> > sigset_t km_sigscaught; /* Caught signals */
>
> Do we need to have all that. Isn't setting TMF_NOUPCALL enough?
yes, it should be enough...
Setting TMF_NOUPCALL ensures that any pre-emption will not result in a
change in KSE, which is all that is needed to ensure that you can
guarantee that finding your KSE is possible.
>
> tmbx = TP;
> ret = (kse_critical_t)tmbx->tm_flags;
> tmbx->tm_flags |= TMF_NOUPCALL;
> return (ret);
>
> --
> Dan Eischen
>
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.BSF.4.21.0307241229170.69488-100000>