Date: Mon, 09 Jun 2014 21:42:22 +0430 From: Hooman Fazaeli <hoomanfazaeli@gmail.com> To: Adrian Chadd <adrian@freebsd.org> Cc: "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>, Mark Delany <n7w@delta.emu.st> Subject: Re: Best practice for accepting TCP connections on multicore? Message-ID: <5395EAF6.4020505@gmail.com> In-Reply-To: <CAJ-Vmon9Q%2BvkRRw7y9wTg5Rvn=zU-9j%2Bv5vtmc_EY8oncacUdw@mail.gmail.com> References: <CAAGHsvDhaqQbwir5P%2BoaH_Qa8VZ0aj9A2SGrn%2B2shJMQ21B6Jw@mail.gmail.com> <CAJ-Vmo=eJ4o2j=3kUOjR=HOMDOR5pD2HcXvxm=dYjkaB9bj8EQ@mail.gmail.com> <20140607010151.51751.qmail@f5-external.bushwire.net> <CAJ-Vmon9Q%2BvkRRw7y9wTg5Rvn=zU-9j%2Bv5vtmc_EY8oncacUdw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 6/7/2014 6:32 AM, Adrian Chadd wrote: > Hi, > > In the RSS model I'm hacking on: > > * you query the kernel for the current RSS bucket -> CPU mapping. > Right now it's one bucket -> one CPU but at some point it may be one > bucket -> cpuset; > * you spawn a thread for each RSS bucket; > * you pin each thread to the RSS bucket current CPU id; > * you create a listen socket in that thread, marked as BINDMULTI (ie, > multiple things can listen on this and the kernel will load balance > between them) and RSS_BUCKET (ie, please place this socket in the > given RSS bucket, rather than the global/wildcard list); > * then when a connection comes in, the kernel will first do a lookup > for a matching wildcard socket in the per-RSS PCBGROUP, rather than > the global wildcard table; > * if it finds it, that socket gets the incoming connection. > > At some point I'll add some notification via kqueue or what not that > the RSS buckets need rebalancing, and userland can then re-pin the > per-bucket threads. > > At the moment the hacks I've done only support one listen socket per > entry. My hope is that BINDMULTI will do some basic hash to load > balance within a set of matching PCB entries - and it'll be combined, > so if you do BINDMULTI without RSS, it'll just load balance between > multiple sockets with no CPU affinity knowledge. If you do RSS, it'll > distribute only CPU-local requests to a thread that's sitting in the > right RSS bucket. if you enable both, you can use a thread pool for > each RSS bucket CPU and (eventually, when I write it) it'll load > balance among those. > > But for now I'm assuming one incoming thread per RSS bucket will be > enough for people to experiment with. > > anyway, I guess I should email out the details: > > * http://github.com/erikarn/freebsd - the 'local/rss' branch has the > RSS changes to dev/e1000/if_igb.c and netinet/ would you please point to the exact URL for the branch? > * http://github.com/erikarn/freebsd-rss - has some RSS examples. Look > at rss-http. > > I haven't yet tested this at > 1GE because all I have at home are igb > and em NICs. If someone would like to donate ixgbe and T4 hardware, > i'll gratefully take it and do up RSS patches for those drivers. > > -- Best regards. Hooman Fazaeli
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5395EAF6.4020505>