From owner-freebsd-net@freebsd.org Tue Mar 7 21:53:37 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5E39ED01E37 for ; Tue, 7 Mar 2017 21:53:37 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost2.sentex.ca (smarthost2.sentex.ca [IPv6:2607:f3e0:80:80::2]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "smarthost.sentex.ca", Issuer "smarthost.sentex.ca" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 3B9771F11 for ; Tue, 7 Mar 2017 21:53:37 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (lava.sentex.ca [IPv6:2607:f3e0:0:5::11]) by smarthost2.sentex.ca (8.15.2/8.15.2) with ESMTPS id v27LraHa095492 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Tue, 7 Mar 2017 16:53:36 -0500 (EST) (envelope-from mike@sentex.net) Received: from [IPv6:2607:f3e0:0:4:e95c:7d13:e5c8:9187] ([IPv6:2607:f3e0:0:4:e95c:7d13:e5c8:9187]) by lava.sentex.ca (8.15.2/8.15.2) with ESMTP id v27LrXT6022148; Tue, 7 Mar 2017 16:53:33 -0500 (EST) (envelope-from mike@sentex.net) Subject: Re: Chelsio netmap support ? (RELENG_11) To: Navdeep Parhar , "freebsd-net@freebsd.org" References: <93e10699-ecc7-c062-bf85-a26de0de9633@sentex.net> From: Mike Tancsa Organization: Sentex Communications Message-ID: Date: Tue, 7 Mar 2017 16:53:34 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.7.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.78 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Mar 2017 21:53:37 -0000 On 3/7/2017 3:34 PM, Navdeep Parhar wrote: > cxgbe's netmap support was overhauled in r302110 so it should already > be in 11.0. Try hw.cxgbe.num_vis=2 and use the vcxl interfaces for > netmap. > > https://lists.freebsd.org/pipermail/svn-src-all/2016-June/126504.html Thanks very much Navdeep! Yes indeed, adding that in boot/loader.conf did the trick # ifconfig vcxl1 vcxl1: flags=8843 metric 0 mtu 1500 options=ec07bb ether 00:07:43:39:9d:ba inet 10.151.10.1 netmask 0xffffff00 broadcast 10.151.10.255 nd6 options=29 media: Ethernet 10Gbase-Twinax status: active plugged: SFP/SFP+/SFP28 Unknown (Copper pigtail) vendor: CISCO-TYCO PN: 1-2053783-2 SN: TED1605BGWC DATE: 2016-02-01 # dmesg | grep -i netma netmap: loaded module vcxl0: netmap queues/slots: TX 2/1023, RX 2/1024 vcxl0: 1 txq, 1 rxq (NIC); 1 txq, 1 rxq (TOE); 2 txq, 2 rxq (netmap) vcxl1: netmap queues/slots: TX 2/1023, RX 2/1024 vcxl1: 1 txq, 1 rxq (NIC); 1 txq, 1 rxq (TOE); 2 txq, 2 rxq (netmap) igb0: netmap queues/slots: TX 4/1024, RX 4/1024 igb1: netmap queues/slots: TX 4/1024, RX 4/1024 Using netsend, I cant seem to blast through a single flow of packets greater than 800Kpps without packet loss. Can you point me to any performance tweaks for forwarding / routing ? I have 3 boxes, with one in the middle (netsend box) <----> (router with chelsio nics) <---> (netreceive box) #./netsend 10.151.10.3 500 100 800000 20 Sending packet of payload size 100 every 0.000001250s for 20 seconds calling time every 100 cycles start: 1488952041.000000000 finish: 1488952061.000000837 send calls: 16000000 send errors: 0 approx send rate: 800000 pps time/packet: 1250 ns approx error rate: 0 waited: 333086341 approx waits/sec: 16654317 approx wait rate: 20 dev.cxl.0.stats.rx_frames_1024_1518: 4444000 dev.cxl.0.stats.rx_frames_512_1023: 96620637 dev.cxl.0.stats.rx_frames_256_511: 5876251 dev.cxl.0.stats.rx_frames_128_255: 100581998 dev.cxl.0.stats.rx_frames_65_127: 1203 dev.cxl.0.stats.rx_frames_64: 263 dev.cxl.0.stats.rx_ucast_frames: 207523216 dev.cxl.0.stats.rx_mcast_frames: 929 dev.cxl.0.stats.rx_bcast_frames: 207 dev.cxl.0.stats.rx_frames: 207524352 dev.cxl.0.stats.rx_octets: 85804034610 dev.cxl.0.stats.tx_frames_256_511: 59 dev.cxl.0.stats.tx_frames_65_127: 406 dev.cxl.0.stats.tx_frames_64: 12 dev.cxl.0.stats.tx_ucast_frames: 414 dev.cxl.0.stats.tx_mcast_frames: 59 dev.cxl.0.stats.tx_bcast_frames: 4 dev.cxl.0.stats.tx_frames: 477 dev.cxl.0.stats.tx_octets: 59172 dev.cxl.0.max_speed: 10 dev.cxl.0.autoneg: 1 dev.cxl.0.linkdnrc: n/a dev.cxl.0.qsize_txq: 1024 dev.cxl.0.qsize_rxq: 1024 dev.cxl.0.holdoff_pktc_idx: -1 dev.cxl.0.holdoff_tmr_idx: 1 dev.cxl.0.nofldtxq: 4 dev.cxl.0.nofldrxq: 2 dev.cxl.0.rss_size: 64 dev.cxl.0.ntxq: 4 dev.cxl.0.nrxq: 4 dev.cxl.0.viid: 1224 dev.cxl.0.%parent: t5nex0 dev.cxl.0.%pnpinfo: dev.cxl.0.%location: dev.cxl.0.%driver: cxl dev.cxl.0.%desc: port 0 dev.vcxl.0.ofld_txq.0.sidx: 1023 dev.vcxl.0.ofld_txq.0.cntxt_id: 36 dev.vcxl.0.ofld_txq.0.dmalen: 65536 dev.vcxl.0.ofld_txq.0.ba: 1376874496 dev.vcxl.0.txq.0.r_starts: 477 dev.vcxl.0.txq.0.r_enqueues: 477 dev.vcxl.0.txq.0.txpkt_wrs: 477 dev.vcxl.0.txq.0.sgl_wrs: 465 dev.vcxl.0.txq.0.imm_wrs: 12 dev.vcxl.0.txq.0.txcsum: 7 dev.vcxl.0.txq.0.tc: -1 dev.vcxl.0.txq.0.sidx: 1023 dev.vcxl.0.txq.0.pidx: 489 dev.vcxl.0.txq.0.cidx: 489 dev.vcxl.0.txq.0.cntxt_id: 35 dev.vcxl.0.txq.0.abs_id: 35 dev.vcxl.0.txq.0.dmalen: 65536 dev.vcxl.0.txq.0.ba: 1376808960 dev.vcxl.0.ofld_rxq.0.fl.cluster_allocated: 32 dev.vcxl.0.ofld_rxq.0.fl.pidx: 32 dev.vcxl.0.ofld_rxq.0.fl.packing: 1 dev.vcxl.0.ofld_rxq.0.fl.padding: 1 dev.vcxl.0.ofld_rxq.0.fl.cntxt_id: 34 dev.vcxl.0.ofld_rxq.0.fl.dmalen: 8192 dev.vcxl.0.ofld_rxq.0.fl.ba: 249552896 dev.vcxl.0.ofld_rxq.0.cntxt_id: 18 dev.vcxl.0.ofld_rxq.0.abs_id: 18 dev.vcxl.0.ofld_rxq.0.dmalen: 65536 dev.vcxl.0.ofld_rxq.0.ba: 1376219136 dev.vcxl.0.rxq.0.fl.cluster_fast_recycled: 1827953 dev.vcxl.0.rxq.0.fl.cluster_recycled: 30549354 dev.vcxl.0.rxq.0.fl.cluster_allocated: 148525 dev.vcxl.0.rxq.0.fl.mbuf_inlined: 61393725 dev.vcxl.0.rxq.0.fl.mbuf_allocated: 122009392 dev.vcxl.0.rxq.0.fl.pidx: 624 dev.vcxl.0.rxq.0.fl.rx_offset: 2304 dev.vcxl.0.rxq.0.fl.cidx: 639 dev.vcxl.0.rxq.0.fl.packing: 1 dev.vcxl.0.rxq.0.fl.padding: 1 dev.vcxl.0.rxq.0.fl.cntxt_id: 33 dev.vcxl.0.rxq.0.fl.dmalen: 8192 dev.vcxl.0.rxq.0.fl.ba: 249544704 dev.vcxl.0.rxq.0.rxcsum: 183402000 dev.vcxl.0.rxq.0.cidx: 151 dev.vcxl.0.rxq.0.cntxt_id: 17 dev.vcxl.0.rxq.0.abs_id: 17 dev.vcxl.0.rxq.0.dmalen: 65536 dev.vcxl.0.rxq.0.ba: 1376153600 dev.vcxl.0.qsize_txq: 1024 dev.vcxl.0.qsize_rxq: 1024 dev.vcxl.0.holdoff_pktc_idx: -1 dev.vcxl.0.holdoff_tmr_idx: 1 dev.vcxl.0.nnmtxq: 2 dev.vcxl.0.nnmrxq: 2 dev.vcxl.0.first_ofld_txq: 4 dev.vcxl.0.first_ofld_rxq: 2 dev.vcxl.0.nofldtxq: 1 dev.vcxl.0.nofldrxq: 1 dev.vcxl.0.rss_size: 64 dev.vcxl.0.first_txq: 4 dev.vcxl.0.first_rxq: 4 dev.vcxl.0.ntxq: 1 dev.vcxl.0.nrxq: 1 dev.vcxl.0.viid: 1226 dev.vcxl.0.%parent: cxl0 dev.vcxl.0.%pnpinfo: dev.vcxl.0.%location: dev.vcxl.0.%driver: vcxl dev.vcxl.0.%desc: port 0 vi 1 ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/