From owner-freebsd-arch@FreeBSD.ORG Mon Mar 2 22:06:27 2015 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 622D2ED8 for ; Mon, 2 Mar 2015 22:06:27 +0000 (UTC) Received: from mailout.easymail.ca (mailout.easymail.ca [64.68.201.169]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F1D42EA5 for ; Mon, 2 Mar 2015 22:06:26 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mailout.easymail.ca (Postfix) with ESMTP id 96CDEE3F1 for ; Mon, 2 Mar 2015 17:06:24 -0500 (EST) X-Virus-Scanned: Debian amavisd-new at mailout.easymail.ca X-Spam-Flag: NO X-Spam-Score: -3.849 X-Spam-Level: X-Spam-Status: No, score=-3.849 required=5 tests=[ALL_TRUSTED=-1.8, AWL=-0.142, BAYES_00=-2.599, DNS_FROM_AHBL_RHSBL=0.692] Received: from mailout.easymail.ca ([127.0.0.1]) by localhost (easymail-mailout.easydns.vpn [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1EQlRRS4806j for ; Mon, 2 Mar 2015 17:06:24 -0500 (EST) Received: from bsddt1241.lv01.astrodoggroup.com (unknown [40.141.24.126]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mailout.easymail.ca (Postfix) with ESMTPSA id 331D7E348 for ; Mon, 2 Mar 2015 17:06:24 -0500 (EST) Message-ID: <54F4DE0D.7070606@astrodoggroup.com> Date: Mon, 02 Mar 2015 14:02:53 -0800 From: Harrison Grundy User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: freebsd-arch@freebsd.org Subject: Re: Minor ULE changes and optimizations References: <54EF2C54.7030207@astrodoggroup.com> <1547642.s3cC06khRt@ralph.baldwin.cx> <54F1E25F.5040905@astrodoggroup.com> <5490895.NN1ciTh6gZ@ralph.baldwin.cx> In-Reply-To: <5490895.NN1ciTh6gZ@ralph.baldwin.cx> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Mar 2015 22:06:27 -0000 On 03/02/15 10:53, John Baldwin wrote: > On Saturday, February 28, 2015 07:44:31 AM Harrison Grundy wrote: >> On 02/28/15 04:24, John Baldwin wrote: >>> On Friday, February 27, 2015 07:50:55 AM Harrison Grundy wrote: >>>> On 02/27/15 06:14, John Baldwin wrote: >>>>> On Thursday, February 26, 2015 06:23:16 AM Harrison Grundy >>>>> >>>>> wrote: >>>>>> https://reviews.freebsd.org/D1969 This allows a >>>>>> non-migratable thread to pin itself to a CPU if it is already >>>>>> running on that CPU. >>>>>> >>>>>> I've been running these patches for the past week or so >>>>>> without issue. Any additional testing or comments would be >>>>>> greatly appreciated. >>>>> >>>>> Can you explain the reason / use case for this? This seems to >>>>> be allowing an API violation. sched_pin() was designed to be >>>>> a lower-level API than sched_bind(), so you wouldn't call >>>>> sched_bind() if you were already pinned. In addition, >>>>> sched_pin() is sometimes used by code that assumes it won't >>>>> migrate until sched_unpin() (e.g. temporary mappings inside an >>>>> sfbuf). If you allow sched_bind() to move a thread that is >>>>> pinned you will allow someone to unintentionally break those >>>>> sort of things instead of getting an assertion failure panic. >>>> >>>> For a pinned thread, the underlying idea is that if you're >>>> already on the CPU you pinned to, calling sched_bind with that >>>> CPU specified allows you to set TSF_BOUND without calling >>>> sched_unpin first. >>>> >>>> If a pinned thread were to call sched_bind for a CPU it isn't >>>> pinned to, it would still hit the assert and fail. >>>> >>>> For any unpinned thread, if you're already running on the correct >>>> CPU, you can skip the THREAD_CAN_MIGRATE check and the call to >>>> mi_switch. >>> >>> Ah, ok, so you aren't allowing migration in theory. However, I'm >>> still curious as to why you want/need this. This makes the API >>> usage a bit more complex to reason about (sched_bind() can >>> sometimes be called while pinned but not always after this change), >>> so I think that extra complexity needs a reason to exist. >> >> Primarily, it allows those threads already on a CPU to skip the call >> to mi_switch and get out of sched_bind a bit faster. > > sched_bind() already does this. Internally it skips the call to mi_switch() > if the thread is already on the correct CPU: > > void > sched_bind(struct thread *td, int cpu) > { > ... > ts->ts_flags |= TSF_BOUND; > sched_pin(); > if (PCPU_GET(cpuid) == cpu) > return; > ... > } > > Calling sched_pin() before sched_bind() isn't going to really change that. > Once you do thread_lock(td) your thread is effectively pinned until you do a > thread_unlock() since the spin lock blocks preemption (and thus migration as > well), so in a sequence of: > > thread_lock(td); > sched_bind(td, cpu); > > The thread is effectively pinned once thread_lock() returns and will not need > to use mi_switch() if it is already on the correct CPU. > >> Additionally, it allows a driver to call sched_pin, then bind to that >> same cpu later without having to write something like >> "critical_enter(); sched_unpin(); sched_bind(foo, bar); >> critical_exit();", since otherwise it could be migrated/preempted >> between unpin and bind. > > But why would a driver want to do that? This code: > > sched_pin(td); > > /* do something */ > > thread_lock(td); > sched_unpin(td); > sched_bind(td, PCPU_GET(cpuid)); > thread_unlock(td); > > /* do something else */ > > thread_lock(td); > sched_unbind(td); > thread_unlock(td); > > Is equivalent to: > > sched_pin(td); > > /* do something */ > > /* do something else */ > > sched_unpin(td); > > But the latter form is lighter weight and easier to read / understand. > > Letting you sched_bind() to the current CPU while you are pinned doesn't > enable any new functionality than you can already achieve by just using > sched_pin() and sched_unpin(). > The difference between the two is that TSF_BOUND is set for "do something else" in the former case. As I understand the difference, sched_pin is designed for temporarily assigning to a CPU, while sched_bind is intended for longer-term affinity. The patch would allow you to set the bound flag without unpinning, basically. It seems easier to do this here, than add a "set bound flag" function that allows drivers to "promote" themselves from pinned to bound, though that would also be an option to get to the same place. --- Harrison