Date: Thu, 12 Sep 2013 16:44:46 -0400 From: Dheeraj Kandula <dkandula@gmail.com> To: Alfred Perlstein <bright@mu.org> Cc: Svatopluk Kraus <onwahe@gmail.com>, "freebsd-arch@freebsd.org" <freebsd-arch@freebsd.org> Subject: Re: Why do we need to acquire the current thread's lock before context switching? Message-ID: <CA%2BqNgxS3Sm%2BTWfEXGhr=9KxAgGtx4pp3deO=Wm=PeZMbgf9piw@mail.gmail.com> In-Reply-To: <FAF0B30B-0F54-43F6-9239-AC0CC64AC955@mu.org> References: <CA%2BqNgxSVkSi88UC3gmfwigmP0UCO6dz%2B_Zxhf_=URK7p4c-Ghg@mail.gmail.com> <CAFHCsPXJkxvJrhfbZt5T=Bm=ZS8-%2BE9xL1cY7b6UENHJ74YR5Q@mail.gmail.com> <CA%2BqNgxT68eobU%2BG4AjKeU6wZb0xM_sktDdQ=jCcmYyzQR%2Basiw@mail.gmail.com> <201309120824.52916.jhb@freebsd.org> <FAF0B30B-0F54-43F6-9239-AC0CC64AC955@mu.org>
next in thread | previous in thread | raw e-mail | index | archive | help
# svn diff
Index: sys/sys/proc.h
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
--- sys/sys/proc.h (revision 255488)
+++ sys/sys/proc.h (working copy)
@@ -197,12 +197,44 @@
};
/*
+ * Comments by: Svatopluk Kraus & John Baldwin <jhb@freebsd.org>
+ *
+ * Svatopluk Kraus' comment:
+ * Think about td_lock like something what is lent by current thread
owner. If
+ * a thread is running, it's owned by scheduler and td_lock points
+ * to scheduler lock. If a thread is sleeping, it's owned by sleeping queu=
e
+ * and td_lock points to sleep queue lock. If a thread is contested, it's
+ * owned by turnstile queue and td_lock points to turnstile queue lock.
And so
+ * on. This way an owner can work with owned threads safely without giant
+ * lock. The td_lock pointer is changed atomically, so it's safe.
+ *
+ * John Baldwin's comment:
+ * For example: take a thread that is asleep on a sleep
+ * queue. td_lock points to the relevant SC_LOCK() for the sleep queue
chain
+ * in that case, so any other thread that wants to examine that thread's
+ * state ends up locking the sleep queue while it examines that thread. I=
n
+ * particular, the thread that is doing a wakeup() can resume all of the
+ * sleeping threads for a wait channel by holding the one SC_LOCK() for
that
+ * wait channel since that will be td_lock for all those threads.
+ *
+ * In general mutexes are only unlocked by the thread that locks them,
+ * and the td_lock of the old thread is unlocked during sched_switch().
+ * However, the old thread has to grab td_lock of the new thread during
+ * sched_switch() and then hand it off to the new thread when it resumes.
+ * This is why sched_throw() and sched_switch() in ULE directly assign
+ * 'mtx_lock' of the run queue lock before calling cpu_throw() or
+ * cpu_switch(). That gives the effect that the new thread resumes while
+ * holding the lock pinted to by its td_lock.
+ */
+/*
* Kernel runnable context (thread).
* This is what is put to sleep and reactivated.
* Thread context. Processes may have multiple threads.
*/
struct thread {
- struct mtx *volatile td_lock; /* replaces sched lock */
+ struct mtx *volatile td_lock; /* replaces sched lock. Look at the comment
+ * above for further details.
+ */
struct proc *td_proc; /* (*) Associated process. */
TAILQ_ENTRY(thread) td_plist; /* (*) All threads in this proc. */
TAILQ_ENTRY(thread) td_runq; /* (t) Run queue. */
On Thu, Sep 12, 2013 at 4:21 PM, Alfred Perlstein <bright@mu.org> wrote:
> Both these explanations are so great. Is there any way we can add this to
> proc.h or maybe document somewhere and then link to it from proc.h?
>
> Sent from my iPhone
>
> On Sep 12, 2013, at 5:24 AM, John Baldwin <jhb@freebsd.org> wrote:
>
> > On Thursday, September 12, 2013 7:16:20 am Dheeraj Kandula wrote:
> >> Thanks a lot Svatopluk for the clarification. Right after I replied to
> >> Alfred's mail, I realized that it can't be thread specific lock as it
> >> should also protect the scheduler variables. So if I understand it
> right,
> >> even though it is a mutex, it can be unlocked by another thread which =
is
> >> usually not the case with regular mutexes as the thread that locks it
> must
> >> unlock it unlike a binary semaphore. Isn't it?
> >
> > It's less complicated than that. :) It is a mutex, but to expand on wh=
at
> > Svatopluk said with an example: take a thread that is asleep on a sleep
> > queue. td_lock points to the relevant SC_LOCK() for the sleep queue
> chain
> > in that case, so any other thread that wants to examine that thread's
> > state ends up locking the sleep queue while it examines that thread. I=
n
> > particular, the thread that is doing a wakeup() can resume all of the
> > sleeping threads for a wait channel by holding the one SC_LOCK() for th=
at
> > wait channel since that will be td_lock for all those threads.
> >
> > In general mutexes are only unlocked by the thread that locks them,
> > and the td_lock of the old thread is unlocked during sched_switch().
> > However, the old thread has to grab td_lock of the new thread during
> > sched_switch() and then hand it off to the new thread when it resumes.
> > This is why sched_throw() and sched_switch() in ULE directly assign
> > 'mtx_lock' of the run queue lock before calling cpu_throw() or
> > cpu_switch(). That gives the effect that the new thread resumes while
> > holding the lock pinted to by its td_lock.
> >
> >> Dheeraj
> >>
> >>
> >> On Thu, Sep 12, 2013 at 7:04 AM, Svatopluk Kraus <onwahe@gmail.com>
> wrote:
> >>
> >>> Think about td_lock like something what is lent by current thread
> owner.
> >>> If a thread is running, it's owned by scheduler and td_lock points
> >>> to scheduler lock. If a thread is sleeping, it's owned by sleeping
> queue
> >>> and td_lock points to sleep queue lock. If a thread is contested, it'=
s
> >>> owned by turnstile queue and td_lock points to turnstile queue lock.
> And so
> >>> on. This way an owner can work with owned threads safely without gian=
t
> >>> lock. The td_lock pointer is changed atomically, so it's safe.
> >>>
> >>> Svatopluk Kraus
> >>>
> >>> On Thu, Sep 12, 2013 at 12:48 PM, Dheeraj Kandula <dkandula@gmail.com
> >wrote:
> >>>
> >>>> Thanks a lot Alfred for the clarification. So is the td_lock granula=
r
> i.e.
> >>>> one separate lock for each thread but also used for protecting the
> >>>> scheduler variables or is it just one lock used by all threads and t=
he
> >>>> scheduler as well. I will anyway go through the code that you
> suggested
> >>>> but
> >>>> just wanted to have a deeper understanding before I go about hunting
> in
> >>>> the
> >>>> code.
> >>>>
> >>>> Dheeraj
> >>>>
> >>>>
> >>>> On Thu, Sep 12, 2013 at 3:10 AM, Alfred Perlstein <bright@mu.org>
> wrote:
> >>>>
> >>>>> On 9/11/13 2:39 PM, Dheeraj Kandula wrote:
> >>>>>
> >>>>>> Hey All,
> >>>>>>
> >>>>>> When the current thread is being context switched with a newly
> selected
> >>>>>> thread, why is the current thread's lock acquired before context
> >>>> switch =96
> >>>>>> mi_switch() is invoked after thread_lock(td) is called. A thread a=
t
> any
> >>>>>> time runs only on one of the cores of a CPU. Hence when it is bein=
g
> >>>>>> context
> >>>>>> switched it is added either to the real time runq or the timeshare
> >>>> runq or
> >>>>>> the idle runq with the lock still held or it is added to the sleep
> >>>> queue
> >>>>>> or
> >>>>>> the blocked queue. So this happens atomically even without the loc=
k.
> >>>> Isn't
> >>>>>> it? Am I missing something here? I don't see any contention for th=
e
> >>>> thread
> >>>>>> in order to demand a lock for the thread which will basically
> protect
> >>>> the
> >>>>>> contents of the thread structure for the thread.
> >>>>>>
> >>>>>> Dheeraj
> >>>>> The thread lock also happens to protect various scheduler variables=
:
> >>>>>
> >>>>> struct mtx *volatile td_lock; /* replaces sched lock */
> >>>>>
> >>>>> see sys/kern/sched_ule.c on how the thread lock td_lock is changed
> >>>>> depending on what the thread is doing.
> >>>>>
> >>>>> --
> >>>>> Alfred Perlstein
> >>>> _______________________________________________
> >>>> freebsd-arch@freebsd.org mailing list
> >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-arch
> >>>> To unsubscribe, send any mail to "
> freebsd-arch-unsubscribe@freebsd.org"
> >> _______________________________________________
> >> freebsd-arch@freebsd.org mailing list
> >> http://lists.freebsd.org/mailman/listinfo/freebsd-arch
> >> To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org=
"
> >
> > --
> > John Baldwin
> > _______________________________________________
> > freebsd-arch@freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-arch
> > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org"
> >
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2BqNgxS3Sm%2BTWfEXGhr=9KxAgGtx4pp3deO=Wm=PeZMbgf9piw>