From owner-freebsd-current@FreeBSD.ORG Thu Nov 27 17:34:00 2014 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1F8BAF6E; Thu, 27 Nov 2014 17:34:00 +0000 (UTC) Received: from mail-wg0-x235.google.com (mail-wg0-x235.google.com [IPv6:2a00:1450:400c:c00::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B422938F; Thu, 27 Nov 2014 17:33:59 +0000 (UTC) Received: by mail-wg0-f53.google.com with SMTP id l18so6999430wgh.40 for ; Thu, 27 Nov 2014 09:33:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=OV7T+YANLVUQ/KA3MduDKH2vbFXgP99X0uOISA70lgc=; b=EkDR5kJTxaaQ+AB9Lv5rKZNe0d2qku2T97fDm8jSHjCuqQQ/YOCsz19AczhR9BKdJc 3v8cmqqJrPd5g5oV9RxnupgEbEa/+Y1POoKDvvk2T6mqEA9cdhVQpErAPs2VEusUQ8Jx UBYVyznKnSmK4ejx0qUVmnCl4PRcptHM2b/QefF/wemF4WVri0YvFGv9Zyat+Nr1ELzk BC/F75dbCj0GN1wPrv+qdqVvnBrJqU9uEzLtS6tGHxoUlpWXyTyFUddQ34+nWGFl2hk9 VLLagpCKfRgI+2pi/etdBftTVhULYpKBnuPCwSTVGqdSWs2SRRGBDZQDDroDvZcV/CEg 9pfQ== MIME-Version: 1.0 X-Received: by 10.194.24.103 with SMTP id t7mr41241177wjf.15.1417109638192; Thu, 27 Nov 2014 09:33:58 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.216.106.136 with HTTP; Thu, 27 Nov 2014 09:33:58 -0800 (PST) In-Reply-To: <54775E81.2070505@selasky.org> References: <546CE948.2070105@selasky.org> <546D0CE3.602@selasky.org> <54775B0E.3000004@selasky.org> <54775E81.2070505@selasky.org> Date: Thu, 27 Nov 2014 09:33:58 -0800 X-Google-Sender-Auth: HH29ScQ1EPnJIYZrvRDtjQMnvl0 Message-ID: Subject: Re: [RFC] Removal of M_FLOWID flag from m_flags [WAS: Add support for hardware transmit rate limiting queues] From: Adrian Chadd To: Hans Petter Selasky Content-Type: text/plain; charset=UTF-8 Cc: "K. Macy" , FreeBSD Current , Michael Tuexen , Navdeep Parhar , Lawrence Stewart , Luigi Rizzo X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Nov 2014 17:34:00 -0000 On 27 November 2014 at 09:25, Hans Petter Selasky wrote: > On 11/27/14 18:20, Adrian Chadd wrote: >> >> On 27 November 2014 at 09:18, Adrian Chadd wrote: >>> >>> Erk - SCTP folk - are you using the mbuf flowid field for something >>> SCTP specific? >> >> >> erk - yes, you are. >> >> It seems we're going to run into "what exactly should flowid be used >> for" problems. >> > > Hi, > > If the flowid has special meaning inside the SCTP, please define an own > "rsstype" for it. As far as I could see, it is only used to spread the > traffic on the network adapter and on the CPU cores. I'm more worried about at what layers we're going to be using the flowid values and that various pieces may stomp over other pieces. With the RSS stuff enabled, the IPv4 (and soon IPv6) input paths will re-hash an input frame if it doesn't have an RSS hash; it'll then overwrite whatever flowid value is in the mbuf. This is so no matter what the NIC hands us or what de-encapsulation hands us, we will always put that flow on into the right RSS bucket and a consistent CPU. This should mostly alleviate any out-of-order issues seen with internet-facing machines where things handle fragments and such. Then for the output side of things, it'll have to do a software RSS hash on frames that don't have an RSS hash. Right now I assume in the TCP path that the inp has an RSS flow setup in the inp and that the TCP timers and other assorted stuff uses the inp details. For the UDP output side of things I currently always re-calculate the RSS hash and stamp the mbuf with it before we send it out, again so it ends up on the same output RSS bucket and thus CPU / NIC queue. If the inp ends up with a different flowid (eg a hardware ring flowid) then: * it won't match up on the receive side, so the whole RSS lock contention avoidance thing can't happen; * there's a known mapping for RSS bucket -> CPU IDs (since we're not doing RSS bucket -> CPU ID reassignment yet) - but not for other flowid types to CPU IDs. In general I think the tidyup patch looks fine and I'll do some more RSS testing once it's committed. (But you did sneak in your new hash type in the diff :-) I think we then need to actually plan out how this stuff should hold together before we all rush into it or we'll end up with a mess of components that can't actually interact together. I don't want to have to explain to people that they can't use SCTP, RSS and hardware ring / flow assignment at the same time. It's just going to be painful in the long run. -adrian