Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 02 Mar 2015 14:02:53 -0800
From:      Harrison Grundy <harrison.grundy@astrodoggroup.com>
To:        freebsd-arch@freebsd.org
Subject:   Re: Minor ULE changes and optimizations
Message-ID:  <54F4DE0D.7070606@astrodoggroup.com>
In-Reply-To: <5490895.NN1ciTh6gZ@ralph.baldwin.cx>
References:  <54EF2C54.7030207@astrodoggroup.com> <1547642.s3cC06khRt@ralph.baldwin.cx> <54F1E25F.5040905@astrodoggroup.com> <5490895.NN1ciTh6gZ@ralph.baldwin.cx>

next in thread | previous in thread | raw e-mail | index | archive | help


On 03/02/15 10:53, John Baldwin wrote:
> On Saturday, February 28, 2015 07:44:31 AM Harrison Grundy wrote:
>> On 02/28/15 04:24, John Baldwin wrote:
>>> On Friday, February 27, 2015 07:50:55 AM Harrison Grundy wrote:
>>>> On 02/27/15 06:14, John Baldwin wrote:
>>>>> On Thursday, February 26, 2015 06:23:16 AM Harrison Grundy
>>>>>
>>>>> wrote:
>>>>>> https://reviews.freebsd.org/D1969 This allows a
>>>>>> non-migratable thread to pin itself to a CPU if it is already
>>>>>> running on that CPU.
>>>>>>
>>>>>> I've been running these patches for the past week or so
>>>>>> without issue. Any additional testing or comments would be
>>>>>> greatly appreciated.
>>>>>
>>>>> Can you explain the reason / use case for this?  This seems to
>>>>> be allowing an API violation.  sched_pin() was designed to be
>>>>> a lower-level API than sched_bind(), so you wouldn't call
>>>>> sched_bind() if you were already pinned. In addition,
>>>>> sched_pin() is sometimes used by code that assumes it won't
>>>>> migrate until sched_unpin() (e.g. temporary mappings inside an
>>>>> sfbuf).  If you allow sched_bind() to move a thread that is
>>>>> pinned you will allow someone to unintentionally break those
>>>>> sort of things instead of getting an assertion failure panic.
>>>>
>>>> For a pinned thread, the underlying idea is that if you're
>>>> already on the CPU you pinned to, calling sched_bind with that
>>>> CPU specified allows you to set TSF_BOUND without calling
>>>> sched_unpin first.
>>>>
>>>> If a pinned thread were to call sched_bind for a CPU it isn't
>>>> pinned to, it would still hit the assert and fail.
>>>>
>>>> For any unpinned thread, if you're already running on the correct
>>>> CPU, you can skip the THREAD_CAN_MIGRATE check and the call to
>>>> mi_switch.
>>>
>>> Ah, ok, so you aren't allowing migration in theory.  However, I'm
>>> still curious as to why you want/need this.  This makes the API
>>> usage a bit more complex to reason about (sched_bind() can
>>> sometimes be called while pinned but not always after this change),
>>> so I think that extra complexity needs a reason to exist.
>>
>> Primarily, it allows those threads already on a CPU to skip the call
>> to mi_switch and get out of sched_bind a bit faster.
> 
> sched_bind() already does this.  Internally it skips the call to mi_switch() 
> if the thread is already on the correct CPU:
> 
> void
> sched_bind(struct thread *td, int cpu)
> {
> 	...
> 	ts->ts_flags |= TSF_BOUND;
> 	sched_pin();
> 	if (PCPU_GET(cpuid) == cpu)
> 		return;
> 	...
> }
> 
> Calling sched_pin() before sched_bind() isn't going to really change that.  
> Once you do thread_lock(td) your thread is effectively pinned until you do a 
> thread_unlock() since the spin lock blocks preemption (and thus migration as 
> well), so in a sequence of:
> 
> 	thread_lock(td);
> 	sched_bind(td, cpu);
> 
> The thread is effectively pinned once thread_lock() returns and will not need
> to use mi_switch() if it is already on the correct CPU.
> 
>> Additionally, it allows a driver to call sched_pin, then bind to that
>> same cpu later without having to write something like
>> "critical_enter(); sched_unpin(); sched_bind(foo, bar);
>> critical_exit();", since otherwise it could be migrated/preempted
>> between unpin and bind.
> 
> But why would a driver want to do that?  This code:
> 
> 	sched_pin(td);
> 
> 	/* do something */
> 
> 	thread_lock(td);
> 	sched_unpin(td);
> 	sched_bind(td, PCPU_GET(cpuid));
> 	thread_unlock(td);
> 
> 	/* do something else */
> 
> 	thread_lock(td);
> 	sched_unbind(td);
> 	thread_unlock(td);
> 
> Is equivalent to:
> 
> 	sched_pin(td);
> 
> 	/* do something */
> 
> 	/* do something else */
> 
> 	sched_unpin(td);
> 
> But the latter form is lighter weight and easier to read / understand.
> 
> Letting you sched_bind() to the current CPU while you are pinned doesn't 
> enable any new functionality than you can already achieve by just using 
> sched_pin() and sched_unpin().
> 

The difference between the two is that TSF_BOUND is set for "do
something else" in the former case.

As I understand the difference, sched_pin is designed for temporarily
assigning to a CPU, while sched_bind is intended for longer-term affinity.

The patch would allow you to set the bound flag without unpinning,
basically. It seems easier to do this here, than add a "set bound flag"
function that allows drivers to "promote" themselves from pinned to
bound, though that would also be an option to get to the same place.


--- Harrison



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?54F4DE0D.7070606>