From owner-freebsd-net@freebsd.org Tue Mar 7 05:48:31 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8E110D01A72 for ; Tue, 7 Mar 2017 05:48:31 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-pg0-x22c.google.com (mail-pg0-x22c.google.com [IPv6:2607:f8b0:400e:c05::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5C18A1277 for ; Tue, 7 Mar 2017 05:48:31 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: by mail-pg0-x22c.google.com with SMTP id b129so74895469pgc.2 for ; Mon, 06 Mar 2017 21:48:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:mail-followup-to :references:mime-version:content-disposition :content-transfer-encoding:in-reply-to:user-agent; bh=wksC5G//VuG0edyP+ofLqx18DrhqHj88CW1YGy1LoIU=; b=sfK+nfif5t6gZmqh7bqqhN9wJDbBhQ4kjD/mm7WoKUZEuWpgLjqs9Zc78SIXTZ9WWB m9geZL9MebBAf7ScPX0EG/+XakOiO9mNpAwb99JrZNU6nW3aUAVPLzwbGGF5Zz93k64G GlDI+POJ0Bom6uJ6/aGopfeYiu2pjemEqm9fXAwjqWYvc2CMHzih5p5RYCZ5jn8pbTQn vDDwT1YiAW7hjMG++kZdvBTTQa/TYmYV3HJ2BqXzxC6rFBVJxJ5l7hHSA9anUwgEsPbJ tkoYiRwy2rr2inAtVjG2ygNX+0u4dfNYK8/BKmTxNseW0mwlsfMxubFtqAhgmo3fZ65E ho1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :mail-followup-to:references:mime-version:content-disposition :content-transfer-encoding:in-reply-to:user-agent; bh=wksC5G//VuG0edyP+ofLqx18DrhqHj88CW1YGy1LoIU=; b=GVETi/0p63qRpsDK/wpaxriPB2/dmSmEhdMDbmqmC192HbQFpE9pQaeUuw6QGXIUVA GFqnmkW6q63wdsxGyCNXGJ3CDPscQEvsoGkqvwIAZqhWtuzanlbXY4ezFZXkQHAG13lJ 1sTUyqVY53Pu2obObB84DOULrLq6nj9rwBDdAu4KjdVXSFPKXwypBOsKqEARybI6dwff LoQAkRSnYkP9dWzlzAhPltda3eZvSaq7t2Lrk6YDW6vq4Ko6e9yaKF2Cj1yEeAcVYyjo U/529uuENvbmdFTNSuRjrv7eIt5f3A/zBTbsWoPtC9Y18mkFlA6SnRuN24Ze+iCgrpJH Lkzw== X-Gm-Message-State: AMke39nWQq0u4PbvacTJIWLmSVCXXRKvVmILabh8KIJ8EmNWaVoLdBIBw68WpteBd1XZ+w== X-Received: by 10.84.197.3 with SMTP id m3mr31990128pld.89.1488865710683; Mon, 06 Mar 2017 21:48:30 -0800 (PST) Received: from ox ([2601:641:c000:b800:184b:905b:e95a:39fb]) by smtp.gmail.com with ESMTPSA id v186sm43073137pgv.44.2017.03.06.21.48.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 06 Mar 2017 21:48:29 -0800 (PST) Sender: Navdeep Parhar Date: Mon, 6 Mar 2017 21:48:27 -0800 From: Navdeep Parhar To: =?utf-8?Q?Mi=C5=82osz?= Kaniewski Cc: freebsd-net@freebsd.org Subject: Re: RSS, cxgbe, netmap and non-TCP traffic Message-ID: <20170307054827.GB12859@ox> Mail-Followup-To: =?utf-8?Q?Mi=C5=82osz?= Kaniewski , freebsd-net@freebsd.org References: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Mar 2017 05:48:31 -0000 On Mon, Mar 06, 2017 at 11:10:21AM +0100, Miłosz Kaniewski wrote: > Hello, > > I am trying to split my traffic into two flows: > 1. TCP traffic > 2. Other traffic (non-TCP traffic) > > On my NIC (Chelsio T540-CR) I have configured 9 queues (these are > Netmap queues configured with hw.cxgbe.nnmtx/rx sysctl) . Now I would > like to direct TCP traffic to queues 0-7 and non-TCP traffic to queue > number 8. To achieve this I have configured two filters using cxgbetool: > > cxgbetool t5nex0 filter 0 proto 6 hitcnts 1 > cxgbetool t5nex0 filter 1 hitcnts 1 queue 8 tcbhash 1 hitcnts 1 is the default, tcbhash is unnecessary, and queue# should be the cntxt_id of the netmap rxq that you want to steer traffic to. Use something like this to get the cntxt_id: # sysctl dev.cxl | grep rxq | grep cntxt_id | grep -vw fl # cxgbetool t5nex0 filter 0 proto 6 action pass # cxgbetool t5nex0 filter 1 action pass queue I'd also recommend that you explicitly provide the physical port# for the traffic you're trying to steer to avoid surprises. For example, with the above rules you'll have _all_ non-TCP traffic on all ports sent to queue . This is likely not what you want. You can refine your rules with "iport port#". # cxgbetool t5nex0 filter 1 iport 0 action pass queue (this will steer traffic that shows up on port 0 only) > > And it seems ok because with such configuration non-TCP traffic is > placed only at queue number 8 and TCP traffic is processed by RSS. But > there is a problem because RSS uses all 9 queues and in result some > TCP packets are also distibuted to queue number 8. > > My question is how to limit the number of queues that are used by RSS > to 8 (queues 0-7)? I tried to set net.inet.rss.bits to "3" but it doesn't > seems to changes anything. You'll need to modify the driver to do this. Do not put the queue for non-TCP traffic in the indirection table and traffic subject to RSS will never arrive on that queue. In this sample diff, the last queue is not put in the indirection table and you should use the cntxt_id of this queue when trying to steer non-TCP traffic to it. --- a/sys/dev/cxgbe/t4_netmap.c Tue Feb 28 12:58:56 2017 -0800 +++ b/sys/dev/cxgbe/t4_netmap.c Mon Mar 06 21:32:03 2017 -0800 @@ -384,6 +384,8 @@ cxgbe_netmap_on(struct adapter *sc, stru } for (i = 0; i < vi->rss_size;) { for_each_nm_rxq(vi, j, nm_rxq) { + if (j == vi->nnmrxq - 1) + break; vi->nm_rss[i++] = nm_rxq->iq_abs_id; if (i == vi->rss_size) break; Regards, Navdeep