Date: Fri, 14 Mar 2014 21:47:32 -0700 From: Adrian Chadd <adrian@freebsd.org> To: Robert Watson <rwatson@freebsd.org> Cc: svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org Subject: Re: svn commit: r263198 - in head/sys: amd64/conf conf net netinet netinet6 sys Message-ID: <CAJ-Vmo=5YdLTkD0Dv5qHQokTQxtjFB=t2NqXfFzuZRLqjOOu%2Bg@mail.gmail.com> In-Reply-To: <201403150057.s2F0vofg081606@svn.freebsd.org> References: <201403150057.s2F0vofg081606@svn.freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Woo! Thanks! Adrian On Mar 14, 2014 5:57 PM, "Robert Watson" <rwatson@freebsd.org> wrote: > Author: rwatson > Date: Sat Mar 15 00:57:50 2014 > New Revision: 263198 > URL: http://svnweb.freebsd.org/changeset/base/263198 > > Log: > Several years after initial development, merge prototype support for > linking NIC Receive Side Scaling (RSS) to the network stack's > connection-group implementation. This prototype (and derived patches) > are in use at Juniper and several other FreeBSD-using companies, so > despite some reservations about its maturity, merge the patch to the > base tree so that it can be iteratively refined in collaboration rather > than maintained as a set of gradually diverging patch sets. > > (1) Merge a software implementation of the Toeplitz hash specified in > RSS implemented by David Malone. This is used to allow suitable > pcbgroup placement of connections before the first packet is > received from the NIC. Software hashing is generally avoided, > however, due to high cost of the hash on general-purpose CPUs. > > (2) In in_rss.c, maintain authoritative versions of RSS state intended > to be pushed to each NIC, including keying material, hash > algorithm/ configuration, and buckets. Provide software-facing > interfaces to hash 2- and 4-tuples for IPv4 and IPv6 using both > the RSS standardised Toeplitz and a 'naive' variation with a hash > efficient in software but with poor distribution properties. > Implement rss_m2cpuid()to be used by netisr and other load > balancing code to look up the CPU on which an mbuf should be > processed. > > (3) In the Ethernet link layer, allow netisr distribution using RSS as > a source of policy as an alternative to source ordering; continue > to default to direct dispatch (i.e., don't try and requeue packets > for processing on the 'right' CPU if they arrive in a directly > dispatchable context). > > (4) Allow RSS to control tuning of connection groups in order to align > groups with RSS buckets. If a packet arrives on a protocol using > connection groups, and contains a suitable hardware-generated > hash, use that hash value to select the connection group for pcb > lookup for both IPv4 and IPv6. If no hardware-generated Toeplitz > hash is available, we fall back on regular PCB lookup risking > contention rather than pay the cost of Toeplitz in software -- > this is a less scalable but, at my last measurement, faster > approach. As core counts go up, we may want to revise this > strategy despite CPU overhead. > > Where device drivers suitably configure NICs, and connection groups / > RSS are enabled, this should avoid both lock and line contention during > connection lookup for TCP. This commit does not modify any device > drivers to tune device RSS configuration to the global RSS > configuration; patches are in circulation to do this for at least > Chelsio T3 and Intel 1G/10G drivers. Currently, the KPI for device > drivers is not particularly robust, nor aware of more advanced features > such as runtime reconfiguration/rebalancing. This will hopefully prove > a useful starting point for refinement. > > No MFC is scheduled as we will first want to nail down a more mature > and maintainable KPI/KBI for device drivers. > > Sponsored by: Juniper Networks (original work) > Sponsored by: EMC/Isilon (patch update and merge) > > Added: > head/sys/netinet/in_rss.c (contents, props changed) > head/sys/netinet/in_rss.h (contents, props changed) > head/sys/netinet/toeplitz.c (contents, props changed) > head/sys/netinet/toeplitz.h (contents, props changed) > Modified: > head/sys/amd64/conf/GENERIC > head/sys/conf/files > head/sys/conf/options > head/sys/net/if_ethersubr.c > head/sys/netinet/in_pcb.c > head/sys/netinet/in_pcbgroup.c > head/sys/netinet6/in6_pcb.c > head/sys/netinet6/in6_pcbgroup.c > head/sys/sys/priv.h > > Modified: head/sys/amd64/conf/GENERIC > > ============================================================================== > --- head/sys/amd64/conf/GENERIC Sat Mar 15 00:23:35 2014 (r263197) > +++ head/sys/amd64/conf/GENERIC Sat Mar 15 00:57:50 2014 (r263198) > @@ -28,6 +28,8 @@ options SCHED_ULE # ULE scheduler > options PREEMPTION # Enable kernel thread preemption > options INET # InterNETworking > options INET6 # IPv6 communications protocols > +options PCBGROUP # Protocol control-block groups > +options RSS # Receive-side scaling support > options TCP_OFFLOAD # TCP offload > options SCTP # Stream Control Transmission > Protocol > options FFS # Berkeley Fast Filesystem > > Modified: head/sys/conf/files > > ============================================================================== > --- head/sys/conf/files Sat Mar 15 00:23:35 2014 (r263197) > +++ head/sys/conf/files Sat Mar 15 00:57:50 2014 (r263198) > @@ -3267,6 +3267,7 @@ netinet/in_pcb.c optional inet | inet6 > netinet/in_pcbgroup.c optional inet pcbgroup | inet6 pcbgroup > netinet/in_proto.c optional inet | inet6 > netinet/in_rmx.c optional inet > +netinet/in_rss.c optional inet rss | inet6 rss > netinet/ip_divert.c optional inet ipdivert ipfirewall > netinet/ip_ecn.c optional inet | inet6 > netinet/ip_encap.c optional inet | inet6 > @@ -3308,6 +3309,7 @@ netinet/tcp_syncache.c optional inet | > netinet/tcp_timer.c optional inet | inet6 > netinet/tcp_timewait.c optional inet | inet6 > netinet/tcp_usrreq.c optional inet | inet6 > +netinet/toeplitz.c optional inet rss | inet6 rss > netinet/udp_usrreq.c optional inet | inet6 > netinet/libalias/alias.c optional libalias inet | netgraph_nat inet > netinet/libalias/alias_db.c optional libalias inet | netgraph_nat inet > > Modified: head/sys/conf/options > > ============================================================================== > --- head/sys/conf/options Sat Mar 15 00:23:35 2014 (r263197) > +++ head/sys/conf/options Sat Mar 15 00:57:50 2014 (r263198) > @@ -427,6 +427,7 @@ PCBGROUP opt_pcbgroup.h > PF_DEFAULT_TO_DROP opt_pf.h > RADIX_MPATH opt_mpath.h > ROUTETABLES opt_route.h > +RSS opt_rss.h > SLIP_IFF_OPTS opt_slip.h > TCPDEBUG > TCP_OFFLOAD opt_inet.h # Enable code to dispatch TCP offloading > > Modified: head/sys/net/if_ethersubr.c > > ============================================================================== > --- head/sys/net/if_ethersubr.c Sat Mar 15 00:23:35 2014 (r263197) > +++ head/sys/net/if_ethersubr.c Sat Mar 15 00:57:50 2014 (r263198) > @@ -34,6 +34,7 @@ > #include "opt_inet6.h" > #include "opt_netgraph.h" > #include "opt_mbuf_profiling.h" > +#include "opt_rss.h" > > #include <sys/param.h> > #include <sys/systm.h> > @@ -70,6 +71,7 @@ > #include <netinet/in.h> > #include <netinet/in_var.h> > #include <netinet/if_ether.h> > +#include <netinet/in_rss.h> > #include <netinet/ip_carp.h> > #include <netinet/ip_var.h> > #endif > @@ -583,7 +585,22 @@ ether_input_internal(struct ifnet *ifp, > > /* > * Ethernet input dispatch; by default, direct dispatch here regardless of > - * global configuration. > + * global configuration. However, if RSS is enabled, hook up RSS affinity > + * so that when deferred or hybrid dispatch is enabled, we can > redistribute > + * load based on RSS. > + * > + * XXXRW: Would be nice if the ifnet passed up a flag indicating whether > or > + * not it had already done work distribution via multi-queue. Then we > could > + * direct dispatch in the event load balancing was already complete and > + * handle the case of interfaces with different capabilities better. > + * > + * XXXRW: Sort of want an M_DISTRIBUTED flag to avoid multiple > distributions > + * at multiple layers? > + * > + * XXXRW: For now, enable all this only if RSS is compiled in, although it > + * works fine without RSS. Need to characterise the performance overhead > + * of the detour through the netisr code in the event the result is always > + * direct dispatch. > */ > static void > ether_nh_input(struct mbuf *m) > @@ -596,8 +613,14 @@ static struct netisr_handler ether_nh = > .nh_name = "ether", > .nh_handler = ether_nh_input, > .nh_proto = NETISR_ETHER, > +#ifdef RSS > + .nh_policy = NETISR_POLICY_CPU, > + .nh_dispatch = NETISR_DISPATCH_DIRECT, > + .nh_m2cpuid = rss_m2cpuid, > +#else > .nh_policy = NETISR_POLICY_SOURCE, > .nh_dispatch = NETISR_DISPATCH_DIRECT, > +#endif > }; > > static void > > Modified: head/sys/netinet/in_pcb.c > > ============================================================================== > --- head/sys/netinet/in_pcb.c Sat Mar 15 00:23:35 2014 (r263197) > +++ head/sys/netinet/in_pcb.c Sat Mar 15 00:57:50 2014 (r263198) > @@ -43,6 +43,7 @@ __FBSDID("$FreeBSD$"); > #include "opt_inet.h" > #include "opt_inet6.h" > #include "opt_pcbgroup.h" > +#include "opt_rss.h" > > #include <sys/param.h> > #include <sys/systm.h> > @@ -75,6 +76,7 @@ __FBSDID("$FreeBSD$"); > #if defined(INET) || defined(INET6) > #include <netinet/in.h> > #include <netinet/in_pcb.h> > +#include <netinet/in_rss.h> > #include <netinet/ip_var.h> > #include <netinet/tcp_var.h> > #include <netinet/udp.h> > @@ -1820,7 +1822,7 @@ struct inpcb * > in_pcblookup(struct inpcbinfo *pcbinfo, struct in_addr faddr, u_int fport, > struct in_addr laddr, u_int lport, int lookupflags, struct ifnet *ifp) > { > -#if defined(PCBGROUP) > +#if defined(PCBGROUP) && !defined(RSS) > struct inpcbgroup *pcbgroup; > #endif > > @@ -1829,7 +1831,17 @@ in_pcblookup(struct inpcbinfo *pcbinfo, > KASSERT((lookupflags & (INPLOOKUP_RLOCKPCB | INPLOOKUP_WLOCKPCB)) > != 0, > ("%s: LOCKPCB not set", __func__)); > > -#if defined(PCBGROUP) > + /* > + * When not using RSS, use connection groups in preference to the > + * reservation table when looking up 4-tuples. When using RSS, > just > + * use the reservation table, due to the cost of the Toeplitz hash > + * in software. > + * > + * XXXRW: This policy belongs in the pcbgroup code, as in principle > + * we could be doing RSS with a non-Toeplitz hash that is > affordable > + * in software. > + */ > +#if defined(PCBGROUP) && !defined(RSS) > if (in_pcbgroup_enabled(pcbinfo)) { > pcbgroup = in_pcbgroup_bytuple(pcbinfo, laddr, lport, > faddr, > fport); > @@ -1856,16 +1868,27 @@ in_pcblookup_mbuf(struct inpcbinfo *pcbi > ("%s: LOCKPCB not set", __func__)); > > #ifdef PCBGROUP > - if (in_pcbgroup_enabled(pcbinfo)) { > + /* > + * If we can use a hardware-generated hash to look up the > connection > + * group, use that connection group to find the inpcb. Otherwise > + * fall back on a software hash -- or the reservation table if > we're > + * using RSS. > + * > + * XXXRW: As above, that policy belongs in the pcbgroup code. > + */ > + if (in_pcbgroup_enabled(pcbinfo) && > + !(M_HASHTYPE_TEST(m, M_HASHTYPE_NONE))) { > pcbgroup = in_pcbgroup_byhash(pcbinfo, M_HASHTYPE_GET(m), > m->m_pkthdr.flowid); > if (pcbgroup != NULL) > return (in_pcblookup_group(pcbinfo, pcbgroup, > faddr, > fport, laddr, lport, lookupflags, ifp)); > +#ifndef RSS > pcbgroup = in_pcbgroup_bytuple(pcbinfo, laddr, lport, > faddr, > fport); > return (in_pcblookup_group(pcbinfo, pcbgroup, faddr, fport, > laddr, lport, lookupflags, ifp)); > +#endif > } > #endif > return (in_pcblookup_hash(pcbinfo, faddr, fport, laddr, lport, > > Modified: head/sys/netinet/in_pcbgroup.c > > ============================================================================== > --- head/sys/netinet/in_pcbgroup.c Sat Mar 15 00:23:35 2014 > (r263197) > +++ head/sys/netinet/in_pcbgroup.c Sat Mar 15 00:57:50 2014 > (r263198) > @@ -32,6 +32,7 @@ > __FBSDID("$FreeBSD$"); > > #include "opt_inet6.h" > +#include "opt_rss.h" > > #include <sys/param.h> > #include <sys/lock.h> > @@ -43,6 +44,7 @@ __FBSDID("$FreeBSD$"); > > #include <netinet/in.h> > #include <netinet/in_pcb.h> > +#include <netinet/in_rss.h> > #ifdef INET6 > #include <netinet6/in6_pcb.h> > #endif /* INET6 */ > @@ -60,6 +62,13 @@ __FBSDID("$FreeBSD$"); > * minimal cache line migration and lock contention during steady state > * operation. > * > + * Hardware-offloaded checksums are often inefficient in software -- for > + * example, Toeplitz, specified by RSS, introduced a significant overhead > if > + * performed during per-packge processing. It is therefore desirable to > fall > + * back on traditional reservation table lookups without affinity where > + * hardware-offloaded checksums aren't available, such as for traffic over > + * non-RSS interfaces. > + * > * Internet protocols, such as UDP and TCP, register to use connection > groups > * by providing an ipi_hashfields value other than IPI_HASHFIELDS_NONE; > this > * indicates to the connection group code whether a 2-tuple or 4-tuple is > @@ -72,6 +81,11 @@ __FBSDID("$FreeBSD$"); > * signficantly more expensive than without connection groups, but that > * steady-state processing can be significantly faster. > * > + * When RSS is used, certain connection group parameters, such as the > number > + * of groups, are provided by the RSS implementation, found in in_rss.c. > + * Otherwise, in_pcbgroup.c selects possible sensible parameters > + * corresponding to the degree of parallelism exposed by netisr. > + * > * Most of the implementation of connection groups is in this file; > however, > * connection group lookup is implemented in in_pcb.c alongside > reservation > * table lookups -- see in_pcblookup_group(). > @@ -126,11 +140,25 @@ in_pcbgroup_init(struct inpcbinfo *pcbin > if (mp_ncpus == 1) > return; > > +#ifdef RSS > /* > - * Use one group per CPU for now. If we decide to do dynamic > - * rebalancing a la RSS, we'll need to shift left by at least 1. > + * If we're using RSS, then RSS determines the number of connection > + * groups to use: one connection group per RSS bucket. If for some > + * reason RSS isn't able to provide a number of buckets, disable > + * connection groups entirely. > + * > + * XXXRW: Can this ever happen? > + */ > + numpcbgroups = rss_getnumbuckets(); > + if (numpcbgroups == 0) > + return; > +#else > + /* > + * Otherwise, we'll just use one per CPU for now. If we decide to > + * do dynamic rebalancing a la RSS, we'll need similar logic here. > */ > numpcbgroups = mp_ncpus; > +#endif > > pcbinfo->ipi_hashfields = hashfields; > pcbinfo->ipi_pcbgroups = malloc(numpcbgroups * > @@ -146,10 +174,19 @@ in_pcbgroup_init(struct inpcbinfo *pcbin > > /* > * Initialise notional affinity of the pcbgroup -- for RSS, > - * we want the same notion of affinity as NICs to be used. > - * Just round robin for the time being. > + * we want the same notion of affinity as NICs to be used. > In > + * the non-RSS case, just round robin for the time being. > + * > + * XXXRW: The notion of a bucket to CPU mapping is common > at > + * both pcbgroup and RSS layers -- does that mean that we > + * should migrate it all from RSS to here, and just leave > RSS > + * responsible only for providing hashing and mapping > funtions? > */ > +#ifdef RSS > + pcbgroup->ipg_cpu = rss_getcpu(pgn); > +#else > pcbgroup->ipg_cpu = (pgn % mp_ncpus); > +#endif > } > } > > @@ -179,23 +216,38 @@ in_pcbgroup_destroy(struct inpcbinfo *pc > > /* > * Given a hash of whatever the covered tuple might be, return a pcbgroup > - * index. > + * index. Where RSS is supported, try to align bucket selection with RSS > CPU > + * affinity strategy. > */ > static __inline u_int > in_pcbgroup_getbucket(struct inpcbinfo *pcbinfo, uint32_t hash) > { > > +#ifdef RSS > + return (rss_getbucket(hash)); > +#else > return (hash % pcbinfo->ipi_npcbgroups); > +#endif > } > > /* > * Map a (hashtype, hash) tuple into a connection group, or NULL if the > hash > - * information is insufficient to identify the pcbgroup. > + * information is insufficient to identify the pcbgroup. This might > occur if > + * a TCP packet turns up with a 2-tuple hash, or if an RSS hash is > present but > + * RSS is not compiled into the kernel. > */ > struct inpcbgroup * > in_pcbgroup_byhash(struct inpcbinfo *pcbinfo, u_int hashtype, uint32_t > hash) > { > > +#ifdef RSS > + if ((pcbinfo->ipi_hashfields == IPI_HASHFIELDS_4TUPLE && > + hashtype == M_HASHTYPE_RSS_TCP_IPV4) || > + (pcbinfo->ipi_hashfields == IPI_HASHFIELDS_2TUPLE && > + hashtype == M_HASHTYPE_RSS_IPV4)) > + return (&pcbinfo->ipi_pcbgroups[ > + in_pcbgroup_getbucket(pcbinfo, hash)]); > +#endif > return (NULL); > } > > @@ -213,13 +265,26 @@ in_pcbgroup_bytuple(struct inpcbinfo *pc > { > uint32_t hash; > > + /* > + * RSS note: we pass foreign addr/port as source, and local > addr/port > + * as destination, as we want to align with what the hardware is > + * doing. > + */ > switch (pcbinfo->ipi_hashfields) { > case IPI_HASHFIELDS_4TUPLE: > +#ifdef RSS > + hash = rss_hash_ip4_4tuple(faddr, fport, laddr, lport); > +#else > hash = faddr.s_addr ^ fport; > +#endif > break; > > case IPI_HASHFIELDS_2TUPLE: > +#ifdef RSS > + hash = rss_hash_ip4_2tuple(faddr, laddr); > +#else > hash = faddr.s_addr ^ laddr.s_addr; > +#endif > break; > > default: > > Added: head/sys/netinet/in_rss.c > > ============================================================================== > --- /dev/null 00:00:00 1970 (empty, because file is newly added) > +++ head/sys/netinet/in_rss.c Sat Mar 15 00:57:50 2014 (r263198) > @@ -0,0 +1,505 @@ > +/*- > + * Copyright (c) 2010-2011 Juniper Networks, Inc. > + * All rights reserved. > + * > + * This software was developed by Robert N. M. Watson under contract > + * to Juniper Networks, Inc. > + * > + * Redistribution and use in source and binary forms, with or without > + * modification, are permitted provided that the following conditions > + * are met: > + * 1. Redistributions of source code must retain the above copyright > + * notice, this list of conditions and the following disclaimer. > + * 2. Redistributions in binary form must reproduce the above copyright > + * notice, this list of conditions and the following disclaimer in the > + * documentation and/or other materials provided with the distribution. > + * > + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND > + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE > + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR > PURPOSE > + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE > + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR > CONSEQUENTIAL > + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS > + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) > + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, > STRICT > + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY > WAY > + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF > + * SUCH DAMAGE. > + */ > + > +#include <sys/cdefs.h> > + > +__FBSDID("$FreeBSD$"); > + > +#include "opt_inet6.h" > +#include "opt_pcbgroup.h" > + > +#ifndef PCBGROUP > +#error "options RSS depends on options PCBGROUP" > +#endif > + > +#include <sys/param.h> > +#include <sys/mbuf.h> > +#include <sys/socket.h> > +#include <sys/priv.h> > +#include <sys/kernel.h> > +#include <sys/smp.h> > +#include <sys/sysctl.h> > + > +#include <net/if.h> > +#include <net/if_var.h> > +#include <net/netisr.h> > + > +#include <netinet/in.h> > +#include <netinet/in_pcb.h> > +#include <netinet/in_rss.h> > +#include <netinet/in_var.h> > +#include <netinet/toeplitz.h> > + > +/*- > + * Operating system parts of receiver-side scaling (RSS), which allows > + * network cards to direct flows to particular receive queues based on > hashes > + * of header tuples. This implementation aligns RSS buckets with > connection > + * groups at the TCP/IP layer, so each bucket is associated with exactly > one > + * group. As a result, the group lookup structures (and lock) should > have an > + * effective affinity with exactly one CPU. > + * > + * Network device drivers needing to configure RSS will query this > framework > + * for parameters, such as the current RSS key, hashing policies, number > of > + * bits, and indirection table mapping hashes to buckets and CPUs. They > may > + * provide their own supplementary information, such as queue<->CPU > bindings. > + * It is the responsibility of the network device driver to inject packets > + * into the stack on as close to the right CPU as possible, if playing by > RSS > + * rules. > + * > + * TODO: > + * > + * - Synchronization for rss_key and other future-configurable parameters. > + * - Event handler drivers can register to pick up RSS configuration > changes. > + * - Should we allow rss_basecpu to be configured? > + * - Randomize key on boot. > + * - IPv6 support. > + * - Statistics on how often there's a misalignment between hardware > + * placement and pcbgroup expectations. > + */ > + > +SYSCTL_NODE(_net_inet, OID_AUTO, rss, CTLFLAG_RW, 0, "Receive-side > steering"); > + > +/* > + * Toeplitz is the only required hash function in the RSS spec, so use it > by > + * default. > + */ > +static u_int rss_hashalgo = RSS_HASH_TOEPLITZ; > +SYSCTL_INT(_net_inet_rss, OID_AUTO, hashalgo, CTLFLAG_RD, &rss_hashalgo, > 0, > + "RSS hash algorithm"); > +TUNABLE_INT("net.inet.rss.hashalgo", &rss_hashalgo); > + > +/* > + * Size of the indirection table; at most 128 entries per the RSS spec. > We > + * size it to at least 2 times the number of CPUs by default to allow > useful > + * rebalancing. If not set explicitly with a loader tunable, we tune > based > + * on the number of CPUs present. > + * > + * XXXRW: buckets might be better to use for the tunable than bits. > + */ > +static u_int rss_bits; > +SYSCTL_INT(_net_inet_rss, OID_AUTO, bits, CTLFLAG_RD, &rss_bits, 0, > + "RSS bits"); > +TUNABLE_INT("net.inet.rss.bits", &rss_bits); > + > +static u_int rss_mask; > +SYSCTL_INT(_net_inet_rss, OID_AUTO, mask, CTLFLAG_RD, &rss_mask, 0, > + "RSS mask"); > + > +static const u_int rss_maxbits = RSS_MAXBITS; > +SYSCTL_INT(_net_inet_rss, OID_AUTO, maxbits, CTLFLAG_RD, > + __DECONST(int *, &rss_maxbits), 0, "RSS maximum bits"); > + > +/* > + * RSS's own count of the number of CPUs it could be using for processing. > + * Bounded to 64 by RSS constants. > + */ > +static u_int rss_ncpus; > +SYSCTL_INT(_net_inet_rss, OID_AUTO, ncpus, CTLFLAG_RD, &rss_ncpus, 0, > + "Number of CPUs available to RSS"); > + > +#define RSS_MAXCPUS (1 << (RSS_MAXBITS - 1)) > +static const u_int rss_maxcpus = RSS_MAXCPUS; > +SYSCTL_INT(_net_inet_rss, OID_AUTO, maxcpus, CTLFLAG_RD, > + __DECONST(int *, &rss_maxcpus), 0, "RSS maximum CPUs that can be > used"); > + > +/* > + * Variable exists just for reporting rss_bits in a user-friendly way. > + */ > +static u_int rss_buckets; > +SYSCTL_INT(_net_inet_rss, OID_AUTO, buckets, CTLFLAG_RD, &rss_buckets, 0, > + "RSS buckets"); > + > +/* > + * Base CPU number; devices will add this to all CPU numbers returned by > the > + * RSS indirection table. Currently unmodifable in FreeBSD. > + */ > +static const u_int rss_basecpu; > +SYSCTL_INT(_net_inet_rss, OID_AUTO, basecpu, CTLFLAG_RD, > + __DECONST(int *, &rss_basecpu), 0, "RSS base CPU"); > + > +/* > + * RSS secret key, intended to prevent attacks on load-balancing. Its > + * effectiveness may be limited by algorithm choice and available entropy > + * during the boot. > + * > + * XXXRW: And that we don't randomize it yet! > + * > + * XXXRW: This default is actually the default key from Chelsio T3 cards, > as > + * it offers reasonable distribution, unlike all-0 keys which always > + * generate a hash of 0 (upsettingly). > + */ > +static uint8_t rss_key[RSS_KEYSIZE] = { > + 0x43, 0xa3, 0x8f, 0xb0, 0x41, 0x67, 0x25, 0x3d, > + 0x25, 0x5b, 0x0e, 0xc2, 0x6d, 0x5a, 0x56, 0xda, > + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, > + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, > + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, > +}; > + > +/* > + * RSS hash->CPU table, which maps hashed packet headers to particular > CPUs. > + * Drivers may supplement this table with a seperate CPU<->queue table > when > + * programming devices. > + */ > +struct rss_table_entry { > + uint8_t rte_cpu; /* CPU affinity of bucket. */ > +}; > +static struct rss_table_entry rss_table[RSS_TABLE_MAXLEN]; > + > +static void > +rss_init(__unused void *arg) > +{ > + u_int i; > + > + /* > + * Validate tunables, coerce to sensible values. > + */ > + switch (rss_hashalgo) { > + case RSS_HASH_TOEPLITZ: > + case RSS_HASH_NAIVE: > + break; > + > + default: > + printf("%s: invalid RSS hashalgo %u, coercing to %u", > + __func__, rss_hashalgo, RSS_HASH_TOEPLITZ); > + rss_hashalgo = RSS_HASH_TOEPLITZ; > + } > + > + /* > + * Count available CPUs. > + * > + * XXXRW: Note incorrect assumptions regarding contiguity of this > set > + * elsewhere. > + */ > + rss_ncpus = 0; > + for (i = 0; i <= mp_maxid; i++) { > + if (CPU_ABSENT(i)) > + continue; > + rss_ncpus++; > + } > + if (rss_ncpus > RSS_MAXCPUS) > + rss_ncpus = RSS_MAXCPUS; > + > + /* > + * Tune RSS table entries to be no less than 2x the number of CPUs > + * -- unless we're running uniprocessor, in which case there's not > + * much point in having buckets to rearrange for load-balancing! > + */ > + if (rss_ncpus > 1) { > + if (rss_bits == 0) > + rss_bits = fls(rss_ncpus - 1) + 1; > + > + /* > + * Microsoft limits RSS table entries to 128, so apply that > + * limit to both auto-detected CPU counts and > user-configured > + * ones. > + */ > + if (rss_bits == 0 || rss_bits > RSS_MAXBITS) { > + printf("%s: RSS bits %u not valid, coercing to > %u", > + __func__, rss_bits, RSS_MAXBITS); > + rss_bits = RSS_MAXBITS; > + } > + > + /* > + * Figure out how many buckets to use; warn if less than > the > + * number of configured CPUs, although this is not a fatal > + * problem. > + */ > + rss_buckets = (1 << rss_bits); > + if (rss_buckets < rss_ncpus) > + printf("%s: WARNING: rss_buckets (%u) less than " > + "rss_ncpus (%u)\n", __func__, rss_buckets, > + rss_ncpus); > + rss_mask = rss_buckets - 1; > + } else { > + rss_bits = 0; > + rss_buckets = 1; > + rss_mask = 0; > + } > + > + /* > + * Set up initial CPU assignments: round-robin by default. > + * > + * XXXRW: Need a mapping to non-contiguous IDs here. > + */ > + for (i = 0; i < rss_buckets; i++) > + rss_table[i].rte_cpu = i % rss_ncpus; > + > + /* > + * Randomize rrs_key. > + * > + * XXXRW: Not yet. If nothing else, will require an rss_isbadkey() > + * loop to check for "bad" RSS keys. > + */ > +} > +SYSINIT(rss_init, SI_SUB_SOFTINTR, SI_ORDER_SECOND, rss_init, NULL); > + > +static uint32_t > +rss_naive_hash(u_int keylen, const uint8_t *key, u_int datalen, > + const uint8_t *data) > +{ > + uint32_t v; > + u_int i; > + > + v = 0; > + for (i = 0; i < keylen; i++) > + v += key[i]; > + for (i = 0; i < datalen; i++) > + v += data[i]; > + return (v); > +} > + > +static uint32_t > +rss_hash(u_int datalen, const uint8_t *data) > +{ > + > + switch (rss_hashalgo) { > + case RSS_HASH_TOEPLITZ: > + return (toeplitz_hash(sizeof(rss_key), rss_key, datalen, > + data)); > + > + case RSS_HASH_NAIVE: > + return (rss_naive_hash(sizeof(rss_key), rss_key, datalen, > + data)); > + > + default: > + panic("%s: unsupported/unknown hashalgo %d", __func__, > + rss_hashalgo); > + } > +} > + > +/* > + * Hash an IPv4 2-tuple. > + */ > +uint32_t > +rss_hash_ip4_2tuple(struct in_addr src, struct in_addr dst) > +{ > + uint8_t data[sizeof(src) + sizeof(dst)]; > + u_int datalen; > + > + datalen = 0; > + bcopy(&src, &data[datalen], sizeof(src)); > + datalen += sizeof(src); > + bcopy(&dst, &data[datalen], sizeof(dst)); > + datalen += sizeof(dst); > + return (rss_hash(datalen, data)); > +} > + > +/* > + * Hash an IPv4 4-tuple. > + */ > +uint32_t > +rss_hash_ip4_4tuple(struct in_addr src, u_short srcport, struct in_addr > dst, > + u_short dstport) > +{ > + uint8_t data[sizeof(src) + sizeof(dst) + sizeof(srcport) + > + sizeof(dstport)]; > + u_int datalen; > + > + datalen = 0; > + bcopy(&src, &data[datalen], sizeof(src)); > + datalen += sizeof(src); > + bcopy(&dst, &data[datalen], sizeof(dst)); > + datalen += sizeof(dst); > + bcopy(&srcport, &data[datalen], sizeof(srcport)); > + datalen += sizeof(srcport); > + bcopy(&dstport, &data[datalen], sizeof(dstport)); > + datalen += sizeof(dstport); > + return (rss_hash(datalen, data)); > +} > + > +#ifdef INET6 > +/* > + * Hash an IPv6 2-tuple. > + */ > +uint32_t > +rss_hash_ip6_2tuple(struct in6_addr src, struct in6_addr dst) > +{ > + uint8_t data[sizeof(src) + sizeof(dst)]; > + u_int datalen; > + > + datalen = 0; > + bcopy(&src, &data[datalen], sizeof(src)); > + datalen += sizeof(src); > + bcopy(&dst, &data[datalen], sizeof(dst)); > + datalen += sizeof(dst); > + return (rss_hash(datalen, data)); > +} > + > +/* > + * Hash an IPv6 4-tuple. > + */ > +uint32_t > +rss_hash_ip6_4tuple(struct in6_addr src, u_short srcport, > + struct in6_addr dst, u_short dstport) > +{ > + uint8_t data[sizeof(src) + sizeof(dst) + sizeof(srcport) + > + sizeof(dstport)]; > + u_int datalen; > + > + datalen = 0; > + bcopy(&src, &data[datalen], sizeof(src)); > + datalen += sizeof(src); > + bcopy(&dst, &data[datalen], sizeof(dst)); > + datalen += sizeof(dst); > + bcopy(&srcport, &data[datalen], sizeof(srcport)); > + datalen += sizeof(srcport); > + bcopy(&dstport, &data[datalen], sizeof(dstport)); > + datalen += sizeof(dstport); > + return (rss_hash(datalen, data)); > +} > +#endif /* INET6 */ > + > +/* > + * Query the number of RSS bits in use. > + */ > +u_int > +rss_getbits(void) > +{ > + > + return (rss_bits); > +} > + > +/* > + * Query the RSS bucket associated with an RSS hash. > + */ > +u_int > +rss_getbucket(u_int hash) > +{ > + > + return (hash & rss_mask); > +} > + > +/* > + * Query the RSS CPU associated with an RSS bucket. > + */ > +u_int > +rss_getcpu(u_int bucket) > +{ > + > + return (rss_table[bucket].rte_cpu); > +} > + > +/* > + * netisr CPU affinity lookup routine for use by protocols. > + */ > +struct mbuf * > +rss_m2cpuid(struct mbuf *m, uintptr_t source, u_int *cpuid) > +{ > + > + M_ASSERTPKTHDR(m); > + > + switch (M_HASHTYPE_GET(m)) { > + case M_HASHTYPE_RSS_IPV4: > + case M_HASHTYPE_RSS_TCP_IPV4: > + *cpuid = rss_getcpu(rss_getbucket(m->m_pkthdr.flowid)); > + return (m); > + > + default: > + *cpuid = NETISR_CPUID_NONE; > + return (m); > + } > +} > + > +/* > + * Query the RSS hash algorithm. > + */ > +u_int > +rss_gethashalgo(void) > +{ > + > + return (rss_hashalgo); > +} > + > +/* > + * Query the current RSS key; likely to be used by device drivers when > + * configuring hardware RSS. Caller must pass an array of size > RSS_KEYSIZE. > + * > + * XXXRW: Perhaps we should do the accept-a-length-and-truncate thing? > + */ > +void > +rss_getkey(uint8_t *key) > +{ > + > + bcopy(rss_key, key, sizeof(rss_key)); > +} > + > +/* > + * Query the number of buckets; this may be used by both network device > + * drivers, which will need to populate hardware shadows of the software > + * indirection table, and the network stack itself (such as when deciding > how > + * many connection groups to allocate). > + */ > +u_int > +rss_getnumbuckets(void) > +{ > + > + return (rss_buckets); > +} > + > +/* > + * Query the number of CPUs in use by RSS; may be useful to device drivers > + * trying to figure out how to map a larger number of CPUs into a smaller > + * number of receive queues. > + */ > +u_int > +rss_getnumcpus(void) > +{ > + > + return (rss_ncpus); > +} > + > +/* > + * XXXRW: Confirm that sysctl -a won't dump this keying material, don't > want > + * it appearing in debugging output unnecessarily. > + */ > +static int > +sysctl_rss_key(SYSCTL_HANDLER_ARGS) > +{ > + uint8_t temp_rss_key[RSS_KEYSIZE]; > + int error; > + > + error = priv_check(req->td, PRIV_NETINET_HASHKEY); > + if (error) > + return (error); > + > + bcopy(rss_key, temp_rss_key, sizeof(temp_rss_key)); > + error = sysctl_handle_opaque(oidp, temp_rss_key, > + sizeof(temp_rss_key), req); > + if (error) > + return (error); > + if (req->newptr != NULL) { > + /* XXXRW: Not yet. */ > + return (EINVAL); > + } > + return (0); > +} > +SYSCTL_PROC(_net_inet_rss, OID_AUTO, key, > + CTLTYPE_OPAQUE | CTLFLAG_RD | CTLFLAG_MPSAFE, NULL, 0, sysctl_rss_key, > + "", "RSS keying material"); > > Added: head/sys/netinet/in_rss.h > > ============================================================================== > --- /dev/null 00:00:00 1970 (empty, because file is newly added) > +++ head/sys/netinet/in_rss.h Sat Mar 15 00:57:50 2014 (r263198) > @@ -0,0 +1,94 @@ > +/*- > + * Copyright (c) 2010-2011 Juniper Networks, Inc. > + * All rights reserved. > + * > + * This software was developed by Robert N. M. Watson under contract > + * to Juniper Networks, Inc. > + * > + * Redistribution and use in source and binary forms, with or without > + * modification, are permitted provided that the following conditions > + * are met: > + * 1. Redistributions of source code must retain the above copyright > + * notice, this list of conditions and the following disclaimer. > + * 2. Redistributions in binary form must reproduce the above copyright > + * notice, this list of conditions and the following disclaimer in the > + * documentation and/or other materials provided with the distribution. > + * > + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND > + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE > + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR > PURPOSE > + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE > + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR > CONSEQUENTIAL > + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS > + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) > + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, > STRICT > + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY > WAY > + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF > + * SUCH DAMAGE. > + * > + * $FreeBSD$ > + */ > + > +#ifndef _NETINET_IN_RSS_H_ > +#define _NETINET_IN_RSS_H_ > + > +#include <netinet/in.h> /* in_addr_t */ > + > +/* > + * Supported RSS hash functions. > + */ > +#define RSS_HASH_NAIVE 0x00000001 /* Poor but fast > hash. */ > +#define RSS_HASH_TOEPLITZ 0x00000002 /* Required by > RSS. */ > +#define RSS_HASH_CRC32 0x00000004 /* Future; some > NICs do it. */ > + > +#define RSS_HASH_MASK (RSS_HASH_NAIVE | > RSS_HASH_TOEPLITZ) > + > +/* > + * Instances of struct inpcbinfo declare an RSS hash type indicating what > + * header fields are covered. > + */ > +#define RSS_HASHFIELDS_NONE 0 > +#define RSS_HASHFIELDS_4TUPLE 1 > +#define RSS_HASHFIELDS_2TUPLE 2 > + > +/* > + * Compile-time limits on the size of the indirection table. > + */ > +#define RSS_MAXBITS 7 > +#define RSS_TABLE_MAXLEN (1 << RSS_MAXBITS) > + > +/* > + * Maximum key size used throughout. It's OK for hardware to use only the > + * first 16 bytes, which is all that's required for IPv4. > + */ > +#define RSS_KEYSIZE 40 > + > +/* > + * Device driver interfaces to query RSS properties that must be > programmed > + * into hardware. > + */ > +u_int rss_getbits(void); > +u_int rss_getbucket(u_int hash); > +u_int rss_getcpu(u_int bucket); > +void rss_getkey(uint8_t *key); > +u_int rss_gethashalgo(void); > +u_int rss_getnumbuckets(void); > +u_int rss_getnumcpus(void); > + > +/* > + * Network stack interface to generate a hash for a protocol tuple. > + */ > +uint32_t rss_hash_ip4_4tuple(struct in_addr src, u_short srcport, > + struct in_addr dst, u_short dstport); > +uint32_t rss_hash_ip4_2tuple(struct in_addr src, struct in_addr > dst); > +uint32_t rss_hash_ip6_4tuple(struct in6_addr src, u_short srcport, > + struct in6_addr dst, u_short dstport); > +uint32_t rss_hash_ip6_2tuple(struct in6_addr src, > + struct in6_addr dst); > + > +/* > + * Network stack interface to query desired CPU affinity of a packet. > + */ > +struct mbuf *rss_m2cpuid(struct mbuf *m, uintptr_t source, u_int > *cpuid); > + > +#endif /* !_NETINET_IN_RSS_H_ */ > > Added: head/sys/netinet/toeplitz.c > > ============================================================================== > --- /dev/null 00:00:00 1970 (empty, because file is newly added) > +++ head/sys/netinet/toeplitz.c Sat Mar 15 00:57:50 2014 (r263198) > @@ -0,0 +1,58 @@ > +/*- > + * Copyright (c) 2010 David Malone <dwmalone@FreeBSD.org> > + * All rights reserved. > + * > + * Redistribution and use in source and binary forms, with or without > + * modification, are permitted provided that the following conditions > + * are met: > + * 1. Redistributions of source code must retain the above copyright > + * notice, this list of conditions and the following disclaimer. > + * 2. Redistributions in binary form must reproduce the above copyright > + * notice, this list of conditions and the following disclaimer in the > + * documentation and/or other materials provided with the distribution. > + * > + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND > + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE > + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR > PURPOSE > + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE > + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR > CONSEQUENTIAL > + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS > + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) > + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, > STRICT > + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY > WAY > + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF > + * SUCH DAMAGE. > + */ > + > +#include <sys/cdefs.h> > +__FBSDID("$FreeBSD$"); > + > +#include <sys/types.h> > + > +#include <netinet/in_rss.h> > +#include <netinet/toeplitz.h> > + > +#include <sys/systm.h> > + > +uint32_t > +toeplitz_hash(u_int keylen, const uint8_t *key, u_int datalen, > + const uint8_t *data) > +{ > + uint32_t hash = 0, v; > + u_int i, b; > + > + /* XXXRW: Perhaps an assertion about key length vs. data length? */ > + > + v = (key[0]<<24) + (key[1]<<16) + (key[2] <<8) + key[3]; > + for (i = 0; i < datalen; i++) { > + for (b = 0; b < 8; b++) { > + if (data[i] & (1<<(7-b))) > + hash ^= v; > + v <<= 1; > + if ((i + 4) < RSS_KEYSIZE && > + (key[i+4] & (1<<(7-b)))) > + v |= 1; > + } > + } > + return (hash); > +} > > Added: head/sys/netinet/toeplitz.h > > ============================================================================== > --- /dev/null 00:00:00 1970 (empty, because file is newly added) > > *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-Vmo=5YdLTkD0Dv5qHQokTQxtjFB=t2NqXfFzuZRLqjOOu%2Bg>