From owner-freebsd-hackers@FreeBSD.ORG  Fri Nov 15 08:30:52 2013
Return-Path: <owner-freebsd-hackers@FreeBSD.ORG>
Delivered-To: freebsd-hackers@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id 0D782352
 for <freebsd-hackers@freebsd.org>; Fri, 15 Nov 2013 08:30:52 +0000 (UTC)
Received: from mail-ee0-x234.google.com (mail-ee0-x234.google.com
 [IPv6:2a00:1450:4013:c00::234])
 (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits))
 (No client certificate requested)
 by mx1.freebsd.org (Postfix) with ESMTPS id A135C271C
 for <freebsd-hackers@freebsd.org>; Fri, 15 Nov 2013 08:30:51 +0000 (UTC)
Received: by mail-ee0-f52.google.com with SMTP id l10so893519eei.39
 for <freebsd-hackers@freebsd.org>; Fri, 15 Nov 2013 00:30:50 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
 h=mime-version:date:message-id:subject:from:to:content-type;
 bh=qDNRU8klCXXe8j4ltY0PqMa84AxcFJUJ5earZ8qFgxs=;
 b=ylkiobPKpjQiNl8EUTvZxpmFjJbCAga9EGrqjqzwe8uRj+jLB/8UhWAf4b4Bb4nuni
 rDURaWAuMoJy+qBFDlw8UzmGciiUA7zgF1otcd3xBrtHIrJPc0jUZWjDwjr7sNO2EQzv
 V3WNijqKxo/M5q8cbnhLWpFN0f2ekL25ilIvYwh3fOI07vStJqieOXlxVrFUoHrB5enj
 4VJPwANnu7TyO8DCdvGdOah6MLT7UWyKJi7DPCk9Tdh0vOOz8+gp3KrOx4h7ygG8uFYv
 uvkID0uAQBB3Ebmas9loQnJYUmj7urX4UvV2vON5c4ED8Wwxpg7qFRm2VOMPzxmOzWIR
 OYDA==
MIME-Version: 1.0
X-Received: by 10.14.203.70 with SMTP id e46mr1656824eeo.33.1384504250037;
 Fri, 15 Nov 2013 00:30:50 -0800 (PST)
Received: by 10.14.127.195 with HTTP; Fri, 15 Nov 2013 00:30:49 -0800 (PST)
Date: Fri, 15 Nov 2013 00:30:49 -0800
Message-ID: <CALCpEUFpOjGGw1Ys4fTyB24gRQ-ab3X_qzrTqUNpjBBSZnP84g@mail.gmail.com>
Subject: ULE thread scheduling in -head
From: hiren panchasara <hiren.panchasara@gmail.com>
To: "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>
Content-Type: text/plain; charset=UTF-8
X-BeenThere: freebsd-hackers@freebsd.org
X-Mailman-Version: 2.1.16
Precedence: list
List-Id: Technical Discussions relating to FreeBSD
 <freebsd-hackers.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-hackers>, 
 <mailto:freebsd-hackers-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-hackers/>
List-Post: <mailto:freebsd-hackers@freebsd.org>
List-Help: <mailto:freebsd-hackers-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>,
 <mailto:freebsd-hackers-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 15 Nov 2013 08:30:52 -0000

I want to understand how thread affinity works and for that I need to
understand how scheduling works. I am reading things and trying to map
them to the code in FreeBSD-head (in and around sys/kern/sched_ule.c)

In terms of scheduler queues, I found 2 different things:
1)
"ULE: A Modern Scheduler For FreeBSD" paper from jeffr mentions
following in 3.1 Queue Mechanisms:
Fairness is implemented by keeping two queues; current and next. Each
thread that is granted a
slice is assigned to either the current queue or the next queue.

2)
I found another interesting struct in there: struct tdq - per
processor runqs and statistics which has 3 queues: realtime, timeshare
and idle.

I could not find 1) in source code.

Can someone throw some light on how a thread gets scheduled? Do 1) and
2) both exist and work together currently in -head FreeBSD?

Trying to follow the code, sched_setup() calls sched_setup_smp() where
for each cpu we call tdq_setup() where we initialize 3 queues
mentioned in 2)  -- But it seems this is all one time setup on boot.
What happens when a new thread is spawned?

Any good pointer/reference for further reading would be great.

cheers,
Hiren