Date: Thu, 9 Mar 2017 10:01:55 -0800 From: Navdeep Parhar <nparhar@gmail.com> To: Mike Tancsa <mike@sentex.net> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: Chelsio netmap support ? (RELENG_11) Message-ID: <CAPFoGT_FSQrHv89AAfy=coRJJDwZs2RGjo3vLaP15Lh=8XgAUQ@mail.gmail.com> In-Reply-To: <d8d41dfa-1b45-1d50-aeb3-bf921f2cfe9d@sentex.net> References: <93e10699-ecc7-c062-bf85-a26de0de9633@sentex.net> <CAPFoGT9u4Ctw0cMtXeEC=6xP--sSW8rEbdSwGwcXZOpRtYp=EQ@mail.gmail.com> <CAPFoGT8iQg9on9xi%2ByNkZJF-knmHE7b58G1u18SvJyVAJa7MPA@mail.gmail.com> <CAPFoGT_CYSPLrB6h742U%2BTS5qqKeLdrF2tnPGGw%2BiyBwPjmDWw@mail.gmail.com> <c7f36da2-ce4d-4377-c3d5-509c9ba692a5@sentex.net> <CAPFoGT-oxsVCNAm05porR0cJi7o_4GTrw-O8V9zZTpRu9_q82Q@mail.gmail.com> <901224a7-f3ed-84a4-5402-3542cd0e865b@sentex.net> <CAPFoGT8pfNE2vNbSuytv0K_4GpYyO6UYTqPp4R-yLVFOfavbug@mail.gmail.com> <d8d41dfa-1b45-1d50-aeb3-bf921f2cfe9d@sentex.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Mar 8, 2017 at 6:28 AM, Mike Tancsa <mike@sentex.net> wrote: > On 3/7/2017 9:08 PM, Navdeep Parhar wrote: >> On Tue, Mar 7, 2017 at 5:46 PM, Mike Tancsa <mike@sentex.net> wrote: >> >>> >>> # dmesg | grep netm >>> netmap: loaded module >>> vcxl0: netmap queues/slots: TX 2/1023, RX 2/1024 >>> vcxl0: 1 txq, 1 rxq (NIC); 1 txq, 1 rxq (TOE); 2 txq, 2 rxq (netmap) >>> vcxl1: netmap queues/slots: TX 2/1023, RX 2/1024 >>> vcxl1: 1 txq, 1 rxq (NIC); 1 txq, 1 rxq (TOE); 2 txq, 2 rxq (netmap) >>> igb0: netmap queues/slots: TX 4/1024, RX 4/1024 >>> igb1: netmap queues/slots: TX 4/1024, RX 4/1024 >>> >>> It maxes out at about 800Kpps with and without netmap. Is there a way >> >> Are you actually using a netmap based application that acts as a >> packet router or is this just the vcxl interface running as a normal >> ifnet? > > the later, vcxl running normal ifnet. I thought there would be a benefit > to utilizing netmap ? Sorry, this is not clear to me. The kernel's routing code does not utilize netmap even if it's available. You'll need something like netmap-fwd for netmap based routing. If you're not using netmap there is no need to create the extra vcxl interfaces. > >> >>> to increase the queues for the Chelsio nic, like the onboard igb ? >> >> If you're not running a netmap based router get rid of the num_vis=2 >> and simply try with the cxl0/cxl1 interfaces. They should each have 4 >> rxq/4 txq on your system. In case you want to increase the number of >> queues, use this: > > The tests with the regular cxl also show the box topping out at 0.8Mpps > for forwarding. I would have expected multiple streams to do better. There is a lot of information about forwarding on the bsdrp.net website. Have you tried the tips there? The numbers there are significantly better than what you observe. I suspect your router is CPU-bound. https://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_hp_proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr https://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_superserver_5018a-ftn4_with_10-gigabit_chelsio_t540-cr There's a projects/routing branch that does much better than the stock kernel. I'm not sure what work remains to be done before it can be merged into head. https://github.com/ocochard/netbenches/blob/master/Xeon_E5-2650-8Cores-Chelsio_T540-CR/forwarding-pf-ipfw/results/fbsd11-routing.r287531/README.md Regards, Navdeep > >> >> The "NIC" queues are the normal tx/rx queues, the "netmap" queues are >> active when the interface is in netmap mode. >> >> Does netsend generate a single flow or multiple flows? If it's a >> single flow it will use a single queue only. > > I think its as a single flow. However, I was using a separate box to > generate a second flow as well. It still topped out at about 800Kpps > before dropping packets. > > ---Mike > > > > -- > ------------------- > Mike Tancsa, tel +1 519 651 3400 > Sentex Communications, mike@sentex.net > Providing Internet services since 1994 www.sentex.net > Cambridge, Ontario Canada http://www.tancsa.com/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAPFoGT_FSQrHv89AAfy=coRJJDwZs2RGjo3vLaP15Lh=8XgAUQ>