Date: Mon, 6 Mar 2017 21:48:27 -0800 From: Navdeep Parhar <np@FreeBSD.org> To: =?utf-8?Q?Mi=C5=82osz?= Kaniewski <milosz.kaniewski@gmail.com> Cc: freebsd-net@freebsd.org Subject: Re: RSS, cxgbe, netmap and non-TCP traffic Message-ID: <20170307054827.GB12859@ox> In-Reply-To: <CAC4mxp7Js9rQ6NCn%2BW9WNUrCKUO21ZK%2BP9Ui8M0NeQL%2BL4bAaQ@mail.gmail.com> References: <CAC4mxp7Js9rQ6NCn%2BW9WNUrCKUO21ZK%2BP9Ui8M0NeQL%2BL4bAaQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Mar 06, 2017 at 11:10:21AM +0100, Miłosz Kaniewski wrote:
> Hello,
>
> I am trying to split my traffic into two flows:
> 1. TCP traffic
> 2. Other traffic (non-TCP traffic)
>
> On my NIC (Chelsio T540-CR) I have configured 9 queues (these are
> Netmap queues configured with hw.cxgbe.nnmtx/rx sysctl) . Now I would
> like to direct TCP traffic to queues 0-7 and non-TCP traffic to queue
> number 8. To achieve this I have configured two filters using cxgbetool:
>
> cxgbetool t5nex0 filter 0 proto 6 hitcnts 1
> cxgbetool t5nex0 filter 1 hitcnts 1 queue 8 tcbhash 1
hitcnts 1 is the default, tcbhash is unnecessary, and queue# should be
the cntxt_id of the netmap rxq that you want to steer traffic to. Use
something like this to get the cntxt_id:
# sysctl dev.cxl | grep rxq | grep cntxt_id | grep -vw fl
# cxgbetool t5nex0 filter 0 proto 6 action pass
# cxgbetool t5nex0 filter 1 action pass queue <cntxt_id>
I'd also recommend that you explicitly provide the physical port# for
the traffic you're trying to steer to avoid surprises. For example,
with the above rules you'll have _all_ non-TCP traffic on all ports sent
to queue <cntxt_id>. This is likely not what you want. You can refine
your rules with "iport port#".
# cxgbetool t5nex0 filter 1 iport 0 action pass queue <cntxt_id>
(this will steer traffic that shows up on port 0 only)
>
> And it seems ok because with such configuration non-TCP traffic is
> placed only at queue number 8 and TCP traffic is processed by RSS. But
> there is a problem because RSS uses all 9 queues and in result some
> TCP packets are also distibuted to queue number 8.
>
> My question is how to limit the number of queues that are used by RSS
> to 8 (queues 0-7)? I tried to set net.inet.rss.bits to "3" but it doesn't
> seems to changes anything.
You'll need to modify the driver to do this. Do not put the queue for
non-TCP traffic in the indirection table and traffic subject to RSS will
never arrive on that queue. In this sample diff, the last queue is not
put in the indirection table and you should use the cntxt_id of this
queue when trying to steer non-TCP traffic to it.
--- a/sys/dev/cxgbe/t4_netmap.c Tue Feb 28 12:58:56 2017 -0800
+++ b/sys/dev/cxgbe/t4_netmap.c Mon Mar 06 21:32:03 2017 -0800
@@ -384,6 +384,8 @@ cxgbe_netmap_on(struct adapter *sc, stru
}
for (i = 0; i < vi->rss_size;) {
for_each_nm_rxq(vi, j, nm_rxq) {
+ if (j == vi->nnmrxq - 1)
+ break;
vi->nm_rss[i++] = nm_rxq->iq_abs_id;
if (i == vi->rss_size)
break;
Regards,
Navdeep
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20170307054827.GB12859>