From owner-freebsd-current@FreeBSD.ORG Fri Nov 28 07:54:03 2014 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 64EFD63C; Fri, 28 Nov 2014 07:54:03 +0000 (UTC) Received: from mail.turbocat.net (heidi.turbocat.net [88.198.202.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E414BF84; Fri, 28 Nov 2014 07:54:01 +0000 (UTC) Received: from laptop015.home.selasky.org (cm-176.74.213.204.customer.telag.net [176.74.213.204]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id BAD091FE022; Fri, 28 Nov 2014 08:53:57 +0100 (CET) Message-ID: <54782A2F.3070103@selasky.org> Date: Fri, 28 Nov 2014 08:54:23 +0100 From: Hans Petter Selasky User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: Adrian Chadd Subject: Re: [RFC] Removal of M_FLOWID flag from m_flags [WAS: Add support for hardware transmit rate limiting queues] References: <546CE948.2070105@selasky.org> <546D0CE3.602@selasky.org> <54775B0E.3000004@selasky.org> <54775E81.2070505@selasky.org> In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Cc: "K. Macy" , FreeBSD Current , Michael Tuexen , Navdeep Parhar , Lawrence Stewart , Luigi Rizzo X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Nov 2014 07:54:03 -0000 On 11/27/14 18:33, Adrian Chadd wrote: > On 27 November 2014 at 09:25, Hans Petter Selasky wrote: >> On 11/27/14 18:20, Adrian Chadd wrote: >>> >>> On 27 November 2014 at 09:18, Adrian Chadd wrote: >>>> >>>> Erk - SCTP folk - are you using the mbuf flowid field for something >>>> SCTP specific? >>> >>> >>> erk - yes, you are. >>> >>> It seems we're going to run into "what exactly should flowid be used >>> for" problems. >>> >> >> Hi, >> >> If the flowid has special meaning inside the SCTP, please define an own >> "rsstype" for it. As far as I could see, it is only used to spread the >> traffic on the network adapter and on the CPU cores. > > I'm more worried about at what layers we're going to be using the > flowid values and that various pieces may stomp over other pieces. > > With the RSS stuff enabled, the IPv4 (and soon IPv6) input paths will > re-hash an input frame if it doesn't have an RSS hash; it'll then > overwrite whatever flowid value is in the mbuf. This is so no matter > what the NIC hands us or what de-encapsulation hands us, we will > always put that flow on into the right RSS bucket and a consistent > CPU. This should mostly alleviate any out-of-order issues seen with > internet-facing machines where things handle fragments and such. Hi, I'm not changing anything in the receive direction regarding the flowid. It should be exactly the same as before, except M_HASHTYPE_NONE which now indicates that flowid is not set. > > Then for the output side of things, it'll have to do a software RSS > hash on frames that don't have an RSS hash. Right now I assume in the > TCP path that the inp has an RSS flow setup in the inp and that the > TCP timers and other assorted stuff uses the inp details. > > For the UDP output side of things I currently always re-calculate the > RSS hash and stamp the mbuf with it before we send it out, again so it > ends up on the same output RSS bucket and thus CPU / NIC queue. > > If the inp ends up with a different flowid (eg a hardware ring flowid) then: > > * it won't match up on the receive side, so the whole RSS lock > contention avoidance thing can't happen; > * there's a known mapping for RSS bucket -> CPU IDs (since we're not > doing RSS bucket -> CPU ID reassignment yet) - but not for other > flowid types to CPU IDs. Not necessarily. Would could make a standard, that the lower X-bits of the flowid, always indicate RX/TX ring pair and the CPU core, and then the upper 32-X bits are free to use for other purposes. > > In general I think the tidyup patch looks fine and I'll do some more > RSS testing once it's committed. (But you did sneak in your new hash > type in the diff :-) I'll put the new hash type in a separate patch. It doesn't belong there - you're right. > > I think we then need to actually plan out how this stuff should hold > together before we all rush into it or we'll end up with a mess of > components that can't actually interact together. I don't want to have > to explain to people that they can't use SCTP, RSS and hardware ring / > flow assignment at the same time. It's just going to be painful in the > long run. Do you have code not committed which plan to use the flowid in new areas of the FreeBSD kernel? I would like to have a list of usages for the flowid field? --HPS