Date: 7 Jun 2014 01:01:51 +0000 From: "Mark Delany" <n7w@delta.emu.st> To: freebsd-hackers@freebsd.org Subject: Re: Best practice for accepting TCP connections on multicore? Message-ID: <20140607010151.51751.qmail@f5-external.bushwire.net> In-Reply-To: <CAJ-Vmo=eJ4o2j=3kUOjR=HOMDOR5pD2HcXvxm=dYjkaB9bj8EQ@mail.gmail.com> References: <CAAGHsvDhaqQbwir5P%2BoaH_Qa8VZ0aj9A2SGrn%2B2shJMQ21B6Jw@mail.gmail.com> <CAJ-Vmo=eJ4o2j=3kUOjR=HOMDOR5pD2HcXvxm=dYjkaB9bj8EQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 06Jun14, Adrian Chadd allegedly wrote:
> Hi,
>
> I'm working on exactly this with the RSS support that's in the tree.
>
> The vague design I have in mind is to have one bound thread per RSS
> allocated CPU, and then put a listen TCP (and later UDP) socket in
> each thread.
The most interesting aspect of this is how you notify a specific
thread that it should call accept().
What would be nice is if the listen socket only returns readable (via
select/poll/kqueue) on a CPUSET mask. This allows a thread to bind to
a cpu then only accept sockets for interfaces best suited to that CPU.
A simpler alternative might be to live with the listener-thread model
that accepts then farms out the fd, but have a syscall that asks the
kernel if there is a preferred CPU for a given socket. Then the
listener-thread could use that as a hint to farm out the fd to a
CPU-bound thread, something like:
for (int bindCPU=0; bindCPU < maxCPU; ++bindCPU) {
pool[i] = createThreadPool(bindCPU);
}
while (select() == 1) {
int newfd = accept();
int preferredCPU = syscall_preferredCPU(newfd); /* New fancy syscall */
if (preferredCPU == -1) { /* If no preference, randomize */
preferredCPU = randomInRange(0, maxCPU-1);
}
pool[preferredCPU]->queueAndNotify(newfd);
}
If it hasn't already, at the point of calling syscall_preferredCPU()
the kernel could make some decisions about binding CPU preferences to
the stack for that socket.
It would be up to user space to distribute work to the right CPU bound
thread - it can choose not to if its pool is unbalanced.
Mark.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20140607010151.51751.qmail>