Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 09 Jun 2014 21:42:22 +0430
From:      Hooman Fazaeli <hoomanfazaeli@gmail.com>
To:        Adrian Chadd <adrian@freebsd.org>
Cc:        "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>, Mark Delany <n7w@delta.emu.st>
Subject:   Re: Best practice for accepting TCP connections on multicore?
Message-ID:  <5395EAF6.4020505@gmail.com>
In-Reply-To: <CAJ-Vmon9Q%2BvkRRw7y9wTg5Rvn=zU-9j%2Bv5vtmc_EY8oncacUdw@mail.gmail.com>
References:  <CAAGHsvDhaqQbwir5P%2BoaH_Qa8VZ0aj9A2SGrn%2B2shJMQ21B6Jw@mail.gmail.com> <CAJ-Vmo=eJ4o2j=3kUOjR=HOMDOR5pD2HcXvxm=dYjkaB9bj8EQ@mail.gmail.com> <20140607010151.51751.qmail@f5-external.bushwire.net> <CAJ-Vmon9Q%2BvkRRw7y9wTg5Rvn=zU-9j%2Bv5vtmc_EY8oncacUdw@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
On 6/7/2014 6:32 AM, Adrian Chadd wrote:
> Hi,
>
> In the RSS model I'm hacking on:
>
> * you query the kernel for the current RSS bucket -> CPU mapping.
> Right now it's one bucket -> one CPU but at some point it may be one
> bucket -> cpuset;
> * you spawn a thread for each RSS bucket;
> * you pin each thread to the RSS bucket current CPU id;
> * you create a listen socket in that thread, marked as BINDMULTI (ie,
> multiple things can listen on this and the kernel will load balance
> between them) and RSS_BUCKET (ie, please place this socket in the
> given RSS bucket, rather than the global/wildcard list);
> * then when a connection comes in, the kernel will first do a lookup
> for a matching wildcard socket in the per-RSS PCBGROUP, rather than
> the global wildcard table;
> * if it finds it, that socket gets the incoming connection.
>
> At some point I'll add some notification via kqueue or what not that
> the RSS buckets need rebalancing, and userland can then re-pin the
> per-bucket threads.
>
> At the moment the hacks I've done only support one listen socket per
> entry. My hope is that BINDMULTI will do some basic hash to load
> balance within a set of matching PCB entries - and it'll be combined,
> so if you do BINDMULTI without RSS, it'll just load balance between
> multiple sockets with no CPU affinity knowledge. If you do RSS, it'll
> distribute only CPU-local requests to a thread that's sitting in the
> right RSS bucket. if you enable both, you can use a thread pool for
> each RSS bucket CPU and (eventually, when I write it) it'll load
> balance among those.
>
> But for now I'm assuming one incoming thread per RSS bucket will be
> enough for people to experiment with.
>
> anyway, I guess I should email out the details:
>
> * http://github.com/erikarn/freebsd - the 'local/rss' branch has the
> RSS changes to dev/e1000/if_igb.c and netinet/
would you please point to the exact URL for the branch?
> * http://github.com/erikarn/freebsd-rss - has some RSS examples. Look
> at rss-http.
>
> I haven't yet tested this at > 1GE because all I have at home are igb
> and em NICs. If someone would like to donate ixgbe and T4 hardware,
> i'll gratefully take it and do up RSS patches for those drivers.
>
>

-- 

Best regards.
Hooman Fazaeli




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5395EAF6.4020505>