Date: Tue, 7 Mar 2017 16:53:34 -0500 From: Mike Tancsa <mike@sentex.net> To: Navdeep Parhar <nparhar@gmail.com>, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: Chelsio netmap support ? (RELENG_11) Message-ID: <c7f36da2-ce4d-4377-c3d5-509c9ba692a5@sentex.net> In-Reply-To: <CAPFoGT_CYSPLrB6h742U%2BTS5qqKeLdrF2tnPGGw%2BiyBwPjmDWw@mail.gmail.com> References: <93e10699-ecc7-c062-bf85-a26de0de9633@sentex.net> <CAPFoGT9u4Ctw0cMtXeEC=6xP--sSW8rEbdSwGwcXZOpRtYp=EQ@mail.gmail.com> <CAPFoGT8iQg9on9xi%2ByNkZJF-knmHE7b58G1u18SvJyVAJa7MPA@mail.gmail.com> <CAPFoGT_CYSPLrB6h742U%2BTS5qqKeLdrF2tnPGGw%2BiyBwPjmDWw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 3/7/2017 3:34 PM, Navdeep Parhar wrote: > cxgbe's netmap support was overhauled in r302110 so it should already > be in 11.0. Try hw.cxgbe.num_vis=2 and use the vcxl interfaces for > netmap. > > https://lists.freebsd.org/pipermail/svn-src-all/2016-June/126504.html Thanks very much Navdeep! Yes indeed, adding that in boot/loader.conf did the trick # ifconfig vcxl1 vcxl1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> ether 00:07:43:39:9d:ba inet 10.151.10.1 netmask 0xffffff00 broadcast 10.151.10.255 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet 10Gbase-Twinax <full-duplex> status: active plugged: SFP/SFP+/SFP28 Unknown (Copper pigtail) vendor: CISCO-TYCO PN: 1-2053783-2 SN: TED1605BGWC DATE: 2016-02-01 # dmesg | grep -i netma netmap: loaded module vcxl0: netmap queues/slots: TX 2/1023, RX 2/1024 vcxl0: 1 txq, 1 rxq (NIC); 1 txq, 1 rxq (TOE); 2 txq, 2 rxq (netmap) vcxl1: netmap queues/slots: TX 2/1023, RX 2/1024 vcxl1: 1 txq, 1 rxq (NIC); 1 txq, 1 rxq (TOE); 2 txq, 2 rxq (netmap) igb0: netmap queues/slots: TX 4/1024, RX 4/1024 igb1: netmap queues/slots: TX 4/1024, RX 4/1024 Using netsend, I cant seem to blast through a single flow of packets greater than 800Kpps without packet loss. Can you point me to any performance tweaks for forwarding / routing ? I have 3 boxes, with one in the middle (netsend box) <----> (router with chelsio nics) <---> (netreceive box) #./netsend 10.151.10.3 500 100 800000 20 Sending packet of payload size 100 every 0.000001250s for 20 seconds calling time every 100 cycles start: 1488952041.000000000 finish: 1488952061.000000837 send calls: 16000000 send errors: 0 approx send rate: 800000 pps time/packet: 1250 ns approx error rate: 0 waited: 333086341 approx waits/sec: 16654317 approx wait rate: 20 dev.cxl.0.stats.rx_frames_1024_1518: 4444000 dev.cxl.0.stats.rx_frames_512_1023: 96620637 dev.cxl.0.stats.rx_frames_256_511: 5876251 dev.cxl.0.stats.rx_frames_128_255: 100581998 dev.cxl.0.stats.rx_frames_65_127: 1203 dev.cxl.0.stats.rx_frames_64: 263 dev.cxl.0.stats.rx_ucast_frames: 207523216 dev.cxl.0.stats.rx_mcast_frames: 929 dev.cxl.0.stats.rx_bcast_frames: 207 dev.cxl.0.stats.rx_frames: 207524352 dev.cxl.0.stats.rx_octets: 85804034610 dev.cxl.0.stats.tx_frames_256_511: 59 dev.cxl.0.stats.tx_frames_65_127: 406 dev.cxl.0.stats.tx_frames_64: 12 dev.cxl.0.stats.tx_ucast_frames: 414 dev.cxl.0.stats.tx_mcast_frames: 59 dev.cxl.0.stats.tx_bcast_frames: 4 dev.cxl.0.stats.tx_frames: 477 dev.cxl.0.stats.tx_octets: 59172 dev.cxl.0.max_speed: 10 dev.cxl.0.autoneg: 1 dev.cxl.0.linkdnrc: n/a dev.cxl.0.qsize_txq: 1024 dev.cxl.0.qsize_rxq: 1024 dev.cxl.0.holdoff_pktc_idx: -1 dev.cxl.0.holdoff_tmr_idx: 1 dev.cxl.0.nofldtxq: 4 dev.cxl.0.nofldrxq: 2 dev.cxl.0.rss_size: 64 dev.cxl.0.ntxq: 4 dev.cxl.0.nrxq: 4 dev.cxl.0.viid: 1224 dev.cxl.0.%parent: t5nex0 dev.cxl.0.%pnpinfo: dev.cxl.0.%location: dev.cxl.0.%driver: cxl dev.cxl.0.%desc: port 0 dev.vcxl.0.ofld_txq.0.sidx: 1023 dev.vcxl.0.ofld_txq.0.cntxt_id: 36 dev.vcxl.0.ofld_txq.0.dmalen: 65536 dev.vcxl.0.ofld_txq.0.ba: 1376874496 dev.vcxl.0.txq.0.r_starts: 477 dev.vcxl.0.txq.0.r_enqueues: 477 dev.vcxl.0.txq.0.txpkt_wrs: 477 dev.vcxl.0.txq.0.sgl_wrs: 465 dev.vcxl.0.txq.0.imm_wrs: 12 dev.vcxl.0.txq.0.txcsum: 7 dev.vcxl.0.txq.0.tc: -1 dev.vcxl.0.txq.0.sidx: 1023 dev.vcxl.0.txq.0.pidx: 489 dev.vcxl.0.txq.0.cidx: 489 dev.vcxl.0.txq.0.cntxt_id: 35 dev.vcxl.0.txq.0.abs_id: 35 dev.vcxl.0.txq.0.dmalen: 65536 dev.vcxl.0.txq.0.ba: 1376808960 dev.vcxl.0.ofld_rxq.0.fl.cluster_allocated: 32 dev.vcxl.0.ofld_rxq.0.fl.pidx: 32 dev.vcxl.0.ofld_rxq.0.fl.packing: 1 dev.vcxl.0.ofld_rxq.0.fl.padding: 1 dev.vcxl.0.ofld_rxq.0.fl.cntxt_id: 34 dev.vcxl.0.ofld_rxq.0.fl.dmalen: 8192 dev.vcxl.0.ofld_rxq.0.fl.ba: 249552896 dev.vcxl.0.ofld_rxq.0.cntxt_id: 18 dev.vcxl.0.ofld_rxq.0.abs_id: 18 dev.vcxl.0.ofld_rxq.0.dmalen: 65536 dev.vcxl.0.ofld_rxq.0.ba: 1376219136 dev.vcxl.0.rxq.0.fl.cluster_fast_recycled: 1827953 dev.vcxl.0.rxq.0.fl.cluster_recycled: 30549354 dev.vcxl.0.rxq.0.fl.cluster_allocated: 148525 dev.vcxl.0.rxq.0.fl.mbuf_inlined: 61393725 dev.vcxl.0.rxq.0.fl.mbuf_allocated: 122009392 dev.vcxl.0.rxq.0.fl.pidx: 624 dev.vcxl.0.rxq.0.fl.rx_offset: 2304 dev.vcxl.0.rxq.0.fl.cidx: 639 dev.vcxl.0.rxq.0.fl.packing: 1 dev.vcxl.0.rxq.0.fl.padding: 1 dev.vcxl.0.rxq.0.fl.cntxt_id: 33 dev.vcxl.0.rxq.0.fl.dmalen: 8192 dev.vcxl.0.rxq.0.fl.ba: 249544704 dev.vcxl.0.rxq.0.rxcsum: 183402000 dev.vcxl.0.rxq.0.cidx: 151 dev.vcxl.0.rxq.0.cntxt_id: 17 dev.vcxl.0.rxq.0.abs_id: 17 dev.vcxl.0.rxq.0.dmalen: 65536 dev.vcxl.0.rxq.0.ba: 1376153600 dev.vcxl.0.qsize_txq: 1024 dev.vcxl.0.qsize_rxq: 1024 dev.vcxl.0.holdoff_pktc_idx: -1 dev.vcxl.0.holdoff_tmr_idx: 1 dev.vcxl.0.nnmtxq: 2 dev.vcxl.0.nnmrxq: 2 dev.vcxl.0.first_ofld_txq: 4 dev.vcxl.0.first_ofld_rxq: 2 dev.vcxl.0.nofldtxq: 1 dev.vcxl.0.nofldrxq: 1 dev.vcxl.0.rss_size: 64 dev.vcxl.0.first_txq: 4 dev.vcxl.0.first_rxq: 4 dev.vcxl.0.ntxq: 1 dev.vcxl.0.nrxq: 1 dev.vcxl.0.viid: 1226 dev.vcxl.0.%parent: cxl0 dev.vcxl.0.%pnpinfo: dev.vcxl.0.%location: dev.vcxl.0.%driver: vcxl dev.vcxl.0.%desc: port 0 vi 1 ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?c7f36da2-ce4d-4377-c3d5-509c9ba692a5>