From owner-freebsd-hackers@FreeBSD.ORG  Sat Jun  7 01:08:34 2014
Return-Path: <owner-freebsd-hackers@FreeBSD.ORG>
Delivered-To: freebsd-hackers@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id 3E793423
 for <freebsd-hackers@freebsd.org>; Sat,  7 Jun 2014 01:08:34 +0000 (UTC)
Received: from smtp1.bushwire.net (f5.bushwire.net [199.48.133.46])
 by mx1.freebsd.org (Postfix) with SMTP id 129B62467
 for <freebsd-hackers@freebsd.org>; Sat,  7 Jun 2014 01:08:33 +0000 (UTC)
Received: (qmail 51759 invoked by uid 1001); 7 Jun 2014 01:01:51 -0000
Delivered-To: qmda-intercept-freebsd-hackers@freebsd.org
DomainKey-Signature: a=rsa-sha1; q=dns; c=simple; s=s384; d=delta.emu.st; 
 b=h4iBe5mQlTpOHTAUq4eVg92++21r5TSj6VieYY89sNjDCqvsbpWNaXTxZimz1B0P;
Comments: DomainKeys? See http://en.wikipedia.org/wiki/DomainKeys
DomainKey-Trace-MD: h=13; b=46; l=C18R70D31M65F38T31S70R80?69M17C39C27I81;
Comments: QMDA 0.3
Received: (qmail 51752 invoked by uid 1001); 7 Jun 2014 01:01:51 -0000
Date: 7 Jun 2014 01:01:51 +0000
Message-ID: <20140607010151.51751.qmail@f5-external.bushwire.net>
From: "Mark Delany" <n7w@delta.emu.st>
To: freebsd-hackers@freebsd.org
Subject: Re: Best practice for accepting TCP connections on multicore?
References: <CAAGHsvDhaqQbwir5P+oaH_Qa8VZ0aj9A2SGrn+2shJMQ21B6Jw@mail.gmail.com>
 <CAJ-Vmo=eJ4o2j=3kUOjR=HOMDOR5pD2HcXvxm=dYjkaB9bj8EQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <CAJ-Vmo=eJ4o2j=3kUOjR=HOMDOR5pD2HcXvxm=dYjkaB9bj8EQ@mail.gmail.com>
X-BeenThere: freebsd-hackers@freebsd.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: Technical Discussions relating to FreeBSD
 <freebsd-hackers.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-hackers>, 
 <mailto:freebsd-hackers-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-hackers/>
List-Post: <mailto:freebsd-hackers@freebsd.org>
List-Help: <mailto:freebsd-hackers-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>,
 <mailto:freebsd-hackers-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sat, 07 Jun 2014 01:08:34 -0000

On 06Jun14, Adrian Chadd allegedly wrote:
> Hi,
> 
> I'm working on exactly this with the RSS support that's in the tree.
> 
> The vague design I have in mind is to have one bound thread per RSS
> allocated CPU, and then put a listen TCP (and later UDP) socket in
> each thread.

The most interesting aspect of this is how you notify a specific
thread that it should call accept().

What would be nice is if the listen socket only returns readable (via
select/poll/kqueue) on a CPUSET mask. This allows a thread to bind to
a cpu then only accept sockets for interfaces best suited to that CPU.

A simpler alternative might be to live with the listener-thread model
that accepts then farms out the fd, but have a syscall that asks the
kernel if there is a preferred CPU for a given socket. Then the
listener-thread could use that as a hint to farm out the fd to a
CPU-bound thread, something like:

for (int bindCPU=0; bindCPU < maxCPU; ++bindCPU) {
    pool[i] = createThreadPool(bindCPU);
}

while (select() == 1) {
   int newfd = accept();
   int preferredCPU = syscall_preferredCPU(newfd); /* New fancy syscall */

   if (preferredCPU == -1) {			/* If no preference, randomize */
      preferredCPU = randomInRange(0, maxCPU-1);
   }

   pool[preferredCPU]->queueAndNotify(newfd);
}

If it hasn't already, at the point of calling syscall_preferredCPU()
the kernel could make some decisions about binding CPU preferences to
the stack for that socket.

It would be up to user space to distribute work to the right CPU bound
thread - it can choose not to if its pool is unbalanced.


Mark.