Date: Mon, 17 Mar 2014 12:58:26 -0700 From: Adrian Chadd <adrian@freebsd.org> To: "Robert N. M. Watson" <rwatson@freebsd.org> Cc: David Malone <dwmalone@maths.tcd.ie>, "svn-src-head@freebsd.org" <svn-src-head@freebsd.org>, "svn-src-all@freebsd.org" <svn-src-all@freebsd.org>, "src-committers@freebsd.org" <src-committers@freebsd.org> Subject: Re: svn commit: r263198 - in head/sys: amd64/conf conf net netinet netinet6 sys Message-ID: <CAJ-VmokYQFyOsLbbDEKCB17HQetc=8ZmVAxsMSXskcaJeQ5ETQ@mail.gmail.com> In-Reply-To: <DF7D9B2B-75B4-4BB5-8EDC-715521EEB2B6@FreeBSD.org> References: <201403150057.s2F0vofg081606@svn.freebsd.org> <20140317075436.GA5864@walton.maths.tcd.ie> <DF7D9B2B-75B4-4BB5-8EDC-715521EEB2B6@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 17 March 2014 01:26, Robert N. M. Watson <rwatson@freebsd.org> wrote: > > On 17 Mar 2014, at 07:54, David Malone <dwmalone@maths.tcd.ie> wrote: > >>> (1) Merge a software implementation of the Toeplitz hash specified in >>> RSS implemented by David Malone. This is used to allow suitable >>> pcbgroup placement of connections before the first packet is >>> received from the NIC. Software hashing is generally avoided, >>> however, due to high cost of the hash on general-purpose CPUs. >> >> I could look at a faster software implementation, but I guess most of >> the value is when hashing is done on the NIC. > > In my benchmarking (a couple of years ago) the software path never really= turned up a lot. If you end up with a flow migrating from an RSS NIC to a = non-RSS NIC, you fall back on the 'reservation' hash table, using the conve= ntional hash and picking up contention, but you don't end up doing the soft= ware version of Toeplitz per-packet. The only reason we need a software imp= lementation (currently) is to do an initial placement of a new outbound flo= w into the hash table prior to receiving a packet with the hardware-generat= ed hash on it. For the inbound direction, we can pick it up from the first = packet. (Although, actually, we also need to do it for inbound flows that c= ome from non-RSS NICs -- or localhost or such). If it's done right, then yes, we should only see a per-packet RSS hash on: * NICs / paths that don't do it in hardware on the receive side; * Doing per-packet sanity checking - ie, doing a software RSS as well as the hardware RSS in order to verify that they both line up. I plan on adding the latter at some point soon so we can ensure that things end up on the right queue. For higher connection rate workloads, we want to ensure that the initial hash selection for outbound matches the inbound, as we don't want to be grabbing non-local-CPU locks to migrate connections to other pcbgroups after they've been established. We absolutely want to keep things on the local CPU as much as we can. Thanks! -a
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-VmokYQFyOsLbbDEKCB17HQetc=8ZmVAxsMSXskcaJeQ5ETQ>