From owner-freebsd-arch@FreeBSD.ORG Mon Mar 2 21:31:49 2015 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 025AE900 for ; Mon, 2 Mar 2015 21:31:49 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CEA5AA83 for ; Mon, 2 Mar 2015 21:31:48 +0000 (UTC) Received: from ralph.baldwin.cx (pool-173-54-116-245.nwrknj.fios.verizon.net [173.54.116.245]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id BE591B9B0; Mon, 2 Mar 2015 16:31:47 -0500 (EST) From: John Baldwin To: freebsd-arch@freebsd.org Subject: Re: Minor ULE changes and optimizations Date: Mon, 02 Mar 2015 13:53:13 -0500 Message-ID: <5490895.NN1ciTh6gZ@ralph.baldwin.cx> User-Agent: KMail/4.14.2 (FreeBSD/10.1-STABLE; KDE/4.14.2; amd64; ; ) In-Reply-To: <54F1E25F.5040905@astrodoggroup.com> References: <54EF2C54.7030207@astrodoggroup.com> <1547642.s3cC06khRt@ralph.baldwin.cx> <54F1E25F.5040905@astrodoggroup.com> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 02 Mar 2015 16:31:47 -0500 (EST) Cc: Harrison Grundy X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Mar 2015 21:31:49 -0000 On Saturday, February 28, 2015 07:44:31 AM Harrison Grundy wrote: > On 02/28/15 04:24, John Baldwin wrote: > > On Friday, February 27, 2015 07:50:55 AM Harrison Grundy wrote: > >> On 02/27/15 06:14, John Baldwin wrote: > >>> On Thursday, February 26, 2015 06:23:16 AM Harrison Grundy > >>> > >>> wrote: > >>>> https://reviews.freebsd.org/D1969 This allows a > >>>> non-migratable thread to pin itself to a CPU if it is already > >>>> running on that CPU. > >>>> > >>>> I've been running these patches for the past week or so > >>>> without issue. Any additional testing or comments would be > >>>> greatly appreciated. > >>> > >>> Can you explain the reason / use case for this? This seems to > >>> be allowing an API violation. sched_pin() was designed to be > >>> a lower-level API than sched_bind(), so you wouldn't call > >>> sched_bind() if you were already pinned. In addition, > >>> sched_pin() is sometimes used by code that assumes it won't > >>> migrate until sched_unpin() (e.g. temporary mappings inside an > >>> sfbuf). If you allow sched_bind() to move a thread that is > >>> pinned you will allow someone to unintentionally break those > >>> sort of things instead of getting an assertion failure panic. > >> > >> For a pinned thread, the underlying idea is that if you're > >> already on the CPU you pinned to, calling sched_bind with that > >> CPU specified allows you to set TSF_BOUND without calling > >> sched_unpin first. > >> > >> If a pinned thread were to call sched_bind for a CPU it isn't > >> pinned to, it would still hit the assert and fail. > >> > >> For any unpinned thread, if you're already running on the correct > >> CPU, you can skip the THREAD_CAN_MIGRATE check and the call to > >> mi_switch. > > > > Ah, ok, so you aren't allowing migration in theory. However, I'm > > still curious as to why you want/need this. This makes the API > > usage a bit more complex to reason about (sched_bind() can > > sometimes be called while pinned but not always after this change), > > so I think that extra complexity needs a reason to exist. > > Primarily, it allows those threads already on a CPU to skip the call > to mi_switch and get out of sched_bind a bit faster. sched_bind() already does this. Internally it skips the call to mi_switch() if the thread is already on the correct CPU: void sched_bind(struct thread *td, int cpu) { ... ts->ts_flags |= TSF_BOUND; sched_pin(); if (PCPU_GET(cpuid) == cpu) return; ... } Calling sched_pin() before sched_bind() isn't going to really change that. Once you do thread_lock(td) your thread is effectively pinned until you do a thread_unlock() since the spin lock blocks preemption (and thus migration as well), so in a sequence of: thread_lock(td); sched_bind(td, cpu); The thread is effectively pinned once thread_lock() returns and will not need to use mi_switch() if it is already on the correct CPU. > Additionally, it allows a driver to call sched_pin, then bind to that > same cpu later without having to write something like > "critical_enter(); sched_unpin(); sched_bind(foo, bar); > critical_exit();", since otherwise it could be migrated/preempted > between unpin and bind. But why would a driver want to do that? This code: sched_pin(td); /* do something */ thread_lock(td); sched_unpin(td); sched_bind(td, PCPU_GET(cpuid)); thread_unlock(td); /* do something else */ thread_lock(td); sched_unbind(td); thread_unlock(td); Is equivalent to: sched_pin(td); /* do something */ /* do something else */ sched_unpin(td); But the latter form is lighter weight and easier to read / understand. Letting you sched_bind() to the current CPU while you are pinned doesn't enable any new functionality than you can already achieve by just using sched_pin() and sched_unpin(). -- John Baldwin