Date: Mon, 02 Mar 2015 14:02:53 -0800 From: Harrison Grundy <harrison.grundy@astrodoggroup.com> To: freebsd-arch@freebsd.org Subject: Re: Minor ULE changes and optimizations Message-ID: <54F4DE0D.7070606@astrodoggroup.com> In-Reply-To: <5490895.NN1ciTh6gZ@ralph.baldwin.cx> References: <54EF2C54.7030207@astrodoggroup.com> <1547642.s3cC06khRt@ralph.baldwin.cx> <54F1E25F.5040905@astrodoggroup.com> <5490895.NN1ciTh6gZ@ralph.baldwin.cx>
next in thread | previous in thread | raw e-mail | index | archive | help
On 03/02/15 10:53, John Baldwin wrote: > On Saturday, February 28, 2015 07:44:31 AM Harrison Grundy wrote: >> On 02/28/15 04:24, John Baldwin wrote: >>> On Friday, February 27, 2015 07:50:55 AM Harrison Grundy wrote: >>>> On 02/27/15 06:14, John Baldwin wrote: >>>>> On Thursday, February 26, 2015 06:23:16 AM Harrison Grundy >>>>> >>>>> wrote: >>>>>> https://reviews.freebsd.org/D1969 This allows a >>>>>> non-migratable thread to pin itself to a CPU if it is already >>>>>> running on that CPU. >>>>>> >>>>>> I've been running these patches for the past week or so >>>>>> without issue. Any additional testing or comments would be >>>>>> greatly appreciated. >>>>> >>>>> Can you explain the reason / use case for this? This seems to >>>>> be allowing an API violation. sched_pin() was designed to be >>>>> a lower-level API than sched_bind(), so you wouldn't call >>>>> sched_bind() if you were already pinned. In addition, >>>>> sched_pin() is sometimes used by code that assumes it won't >>>>> migrate until sched_unpin() (e.g. temporary mappings inside an >>>>> sfbuf). If you allow sched_bind() to move a thread that is >>>>> pinned you will allow someone to unintentionally break those >>>>> sort of things instead of getting an assertion failure panic. >>>> >>>> For a pinned thread, the underlying idea is that if you're >>>> already on the CPU you pinned to, calling sched_bind with that >>>> CPU specified allows you to set TSF_BOUND without calling >>>> sched_unpin first. >>>> >>>> If a pinned thread were to call sched_bind for a CPU it isn't >>>> pinned to, it would still hit the assert and fail. >>>> >>>> For any unpinned thread, if you're already running on the correct >>>> CPU, you can skip the THREAD_CAN_MIGRATE check and the call to >>>> mi_switch. >>> >>> Ah, ok, so you aren't allowing migration in theory. However, I'm >>> still curious as to why you want/need this. This makes the API >>> usage a bit more complex to reason about (sched_bind() can >>> sometimes be called while pinned but not always after this change), >>> so I think that extra complexity needs a reason to exist. >> >> Primarily, it allows those threads already on a CPU to skip the call >> to mi_switch and get out of sched_bind a bit faster. > > sched_bind() already does this. Internally it skips the call to mi_switch() > if the thread is already on the correct CPU: > > void > sched_bind(struct thread *td, int cpu) > { > ... > ts->ts_flags |= TSF_BOUND; > sched_pin(); > if (PCPU_GET(cpuid) == cpu) > return; > ... > } > > Calling sched_pin() before sched_bind() isn't going to really change that. > Once you do thread_lock(td) your thread is effectively pinned until you do a > thread_unlock() since the spin lock blocks preemption (and thus migration as > well), so in a sequence of: > > thread_lock(td); > sched_bind(td, cpu); > > The thread is effectively pinned once thread_lock() returns and will not need > to use mi_switch() if it is already on the correct CPU. > >> Additionally, it allows a driver to call sched_pin, then bind to that >> same cpu later without having to write something like >> "critical_enter(); sched_unpin(); sched_bind(foo, bar); >> critical_exit();", since otherwise it could be migrated/preempted >> between unpin and bind. > > But why would a driver want to do that? This code: > > sched_pin(td); > > /* do something */ > > thread_lock(td); > sched_unpin(td); > sched_bind(td, PCPU_GET(cpuid)); > thread_unlock(td); > > /* do something else */ > > thread_lock(td); > sched_unbind(td); > thread_unlock(td); > > Is equivalent to: > > sched_pin(td); > > /* do something */ > > /* do something else */ > > sched_unpin(td); > > But the latter form is lighter weight and easier to read / understand. > > Letting you sched_bind() to the current CPU while you are pinned doesn't > enable any new functionality than you can already achieve by just using > sched_pin() and sched_unpin(). > The difference between the two is that TSF_BOUND is set for "do something else" in the former case. As I understand the difference, sched_pin is designed for temporarily assigning to a CPU, while sched_bind is intended for longer-term affinity. The patch would allow you to set the bound flag without unpinning, basically. It seems easier to do this here, than add a "set bound flag" function that allows drivers to "promote" themselves from pinned to bound, though that would also be an option to get to the same place. --- Harrison
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?54F4DE0D.7070606>