From owner-freebsd-arch@FreeBSD.ORG  Fri Feb 14 18:18:54 2014
Return-Path: <owner-freebsd-arch@FreeBSD.ORG>
Delivered-To: freebsd-arch@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id C9364E11;
 Fri, 14 Feb 2014 18:18:54 +0000 (UTC)
Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1])
 (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits))
 (No client certificate requested)
 by mx1.freebsd.org (Postfix) with ESMTPS id 96C4416EE;
 Fri, 14 Feb 2014 18:18:54 +0000 (UTC)
Received: from jhbbsd.localnet (unknown [209.249.190.124])
 by bigwig.baldwin.cx (Postfix) with ESMTPSA id 92A71B9DF;
 Fri, 14 Feb 2014 13:18:53 -0500 (EST)
From: John Baldwin <jhb@freebsd.org>
To: Adrian Chadd <adrian@freebsd.org>
Subject: Re: can the scheduler decide to schedule an interrupted but runnable
 thread on another CPU core? What are the implications for code?
Date: Fri, 14 Feb 2014 13:18:44 -0500
User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20130906; KDE/4.5.5; amd64; ; )
References: <CAJ-Vmo=7Nz1jqXy+rTQ7u9_ZP7jeFOKUJxU1O51tYJjvTUmWTg@mail.gmail.com>
 <201402141139.49158.jhb@freebsd.org>
 <CAJ-Vmo=vQ+MX+r3z6_Y4aJiWUBxXgXE7APjTsUysVPN2aoghXQ@mail.gmail.com>
In-Reply-To: <CAJ-Vmo=vQ+MX+r3z6_Y4aJiWUBxXgXE7APjTsUysVPN2aoghXQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: Text/Plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
Message-Id: <201402141318.44743.jhb@freebsd.org>
X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7
 (bigwig.baldwin.cx); Fri, 14 Feb 2014 13:18:53 -0500 (EST)
Cc: "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>,
 Ryan Stone <rysto32@gmail.com>,
 "freebsd-arch@freebsd.org" <freebsd-arch@freebsd.org>
X-BeenThere: freebsd-arch@freebsd.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Discussion related to FreeBSD architecture <freebsd-arch.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-arch>,
 <mailto:freebsd-arch-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-arch/>
List-Post: <mailto:freebsd-arch@freebsd.org>
List-Help: <mailto:freebsd-arch-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-arch>,
 <mailto:freebsd-arch-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 14 Feb 2014 18:18:54 -0000

On Friday, February 14, 2014 12:55:03 pm Adrian Chadd wrote:
> On 14 February 2014 08:39, John Baldwin <jhb@freebsd.org> wrote:
> > On Friday, February 14, 2014 4:22:34 am Adrian Chadd wrote:
> >> Ok, so now I remember the other odd thing.
> >>
> >> I was seeing the sending context(s) jumping from one CPU to another
> >> during flowtable_insert_common(), around the locking bits.
> >>
> >> But I thread pinned all the sender user threads!
> >>
> >> So, why would the senders still be scheduled on other CPUs if I've
> >> pinned the userland threads?
> >>
> >> (and yes, I verified that the userland threads weren't moving around.)
> >
> > Can you clarify a bit?  It's not clear how sender thraeds differ from
> > userland threads differ from sender user threads.  (I.e. one reading
> > is that these are all the same thing and should thus all be pinned
> > (I assume you mean using cpuset to bind them to specific cores rather
> > than sched_pin))
> 
> Yup, I'm doing a manual, poor-mans RSS in lieu of merging in roberts stuff:
> 
> * the userland threads are using the cpuset call to map a thread into
> a cpuset, yes
> * the NIC TX/RX ring routines in cxgbe are pinned to the same CPU as
> the userland threads

If they are all cpuset to a single CPU, they should not migrate, though
I think sched_bind() can override that.  However, that requires code to
explicitly call sched_bind() which should be rare.

-- 
John Baldwin