From owner-freebsd-net@freebsd.org Sat Jan 23 17:13:02 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6FDC8A8EAAE for ; Sat, 23 Jan 2016 17:13:02 +0000 (UTC) (envelope-from cenzatti@hush.com) Received: from smtp3.hushmail.com (smtp3a.hushmail.com [65.39.178.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.hushmail.com", Issuer "smtp.hushmail.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 568BF1293 for ; Sat, 23 Jan 2016 17:13:01 +0000 (UTC) (envelope-from cenzatti@hush.com) Received: from smtp3.hushmail.com (smtp3a.hushmail.com [65.39.178.201]) by smtp3.hushmail.com (Postfix) with SMTP id 37272E0264 for ; Sat, 23 Jan 2016 17:13:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=hush.com; h=date:to:subject:from; s=hush; bh=hVKRM2Erm5dv9jzviRkO4K6UzdLVheBEupEpBAPByUc=; b=gNIeGVsMQLQbbnoFFTt28F/9Z/T9wg5gOoocpDW73vlHqq5+6DEmfcL3tk3KDFzjVwlMumPjT0Ysea3/Y+p8YoquUedXG2jdlykzvUI+1nFew8hq2QxeriEJ6C2UPPdwCglxmJK9QoOfbMVWYCAOv/HMH3/9LBwacwsZamE8311Turm6tw0mkhCq12B6H1Y2UbTXYEvpQdSDoPadRP2yKdPfxmYCCmALZ4pkGwQpL4vz01odMwmDMq9u6Mxcip2aCon7gkCZT00xLSFbhn5H2Lx9/FAftstV7L5SzRQZ2SO+eUGpuXDfLjbykT6UM0vrbvaSvujF0I38w9d4SneNaQ== Received: from smtp.hushmail.com (w5.hushmail.com [65.39.178.80]) by smtp3.hushmail.com (Postfix) with ESMTP; Sat, 23 Jan 2016 17:13:00 +0000 (UTC) Received: by smtp.hushmail.com (Postfix, from userid 99) id 0F448A0121; Sat, 23 Jan 2016 17:13:00 +0000 (UTC) MIME-Version: 1.0 Date: Sat, 23 Jan 2016 15:12:59 -0200 To: "Navdeep Parhar" Cc: freebsd-net@freebsd.org Subject: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX From: "Marcus Cenzatti" In-Reply-To: <20160123154052.GA4574@ox> References: <20160123053428.2091EA0121@smtp.hushmail.com> <20160123154052.GA4574@ox> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="UTF-8" Message-Id: <20160123171300.0F448A0121@smtp.hushmail.com> X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 17:13:02 -0000 On 1/23/2016 at 1:40 PM, "Navdeep Parhar" wrote: > >On Sat, Jan 23, 2016 at 03:34:27AM -0200, Marcus Cenzatti wrote: >> hello, >> >> I am testing a chelsio t520-so-cr connected to a Intel card with >ix(4) >> driver, I can get the ncxl0 interface to transmit at 14Mpps to >another >> chelsio or to a Intel card. However I can only get 800Kpps-1Mpps >for >> RX tests from both chelsio or Intel. >> >> I have test with both FreeBSD 11 and FreeBSD 10.3-PRERELEASE. >> >> I tested it untuned first and later I have applied tuning >> recommendations I found on BSDRP[1] website. Results still >ranging >> from 800Kpps to 1Mpps for RX. >> >> Tests are done w/ with pkt-gen in netmap mode on ncxl interface >with >> both IP address and MAC address source/dest. > >The ncxl interfaces have their own MAC addresses. Make sure the >sender >uses the MAC of the receiver's ncxl interface as the destination >MAC. >(netmap's pkt-gen -f tx transmits L2 broadcasts by default). > >Check for PAUSE frames coming out of the receiver (sysctl dev.cxl >| grep >tx_pause). If it's receiving frames on netmap interface the >tx_pause >counter should not move. > >Regards, >Navdeep > hello, yes, MAC addresses are correct, I did the tests again and tx_pause won't move, here is the full transcript for the tests: ===> BOX #1 CHELSIO chelsio# ifconfig -v ncxl0 ncxl0: flags=8843 metric 0 mtu 1500 ether 00:07:43:33:8d:c1 inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255 nd6 options=29 media: Ethernet 10Gbase-SR status: active chelsio# ifconfig -v cxl0 cxl0: flags=8843 metric 0 mtu 1500 options=ec00bb ether 00:07:43:33:8d:c0 nd6 options=29 media: Ethernet 10Gbase-SR status: active plugged: SFP/SFP+/SFP28 10G Base-SR (LC) vendor: FINISAR CORP. PN: FTLX8571D3BCL-FC SN: AL1073K DATE: 2011-06-28 module temperature: 42.79 C Voltage: 3.23 Volts RX: 0.53 mW (-2.74 dBm) TX: 0.48 mW (-3.12 dBm) chelsio# ./pkt-gen -i ncxl0 -f rx -d 00:07:43:33:8d:c1 -s 00:07:e9:44:d2:ba 311.132189 main [1715] interface is ncxl0 311.132447 extract_ip_range [291] range is 0.0.0.0:90 to 0.0.0.0:90 311.132472 extract_ip_range [291] range is 0.0.0.0:7 to 0.0.0.0:7 311.191286 main [1910] mapped 334980KB at 0x801c00000 Receiving from netmap:ncxl0: 2 queues, 1 threads and 1 cpus. 311.191371 main [1996] Wait 2 secs for phy reset 313.230423 main [1998] Ready... 313.230570 nm_open [456] overriding ifname ncxl0 ringid 0x0 flags 0x1 313.247998 receiver_body [1247] reading from netmap:ncxl0 fd 4 main_fd 3 314.249172 receiver_body [1254] waiting for initial packets, poll returns 0 0 (...) when the other box starts transmitting: (...) 380.492236 main_thread [1512] 0 pps (0 pkts in 1006897 usec) 381.326159 receiver_body [1254] waiting for initial packets, poll returns 0 0 381.493141 main_thread [1512] 80110 pps (80137 pkts in 1000906 usec) 382.494145 main_thread [1512] 801585 pps (801888 pkts in 1001004 usec) 383.494979 main_thread [1512] 801381 pps (801632 pkts in 1000833 usec) 384.497148 main_thread [1512] 801490 pps (802144 pkts in 1002169 usec) 385.497650 main_thread [1512] 801416 pps (801568 pkts in 1000503 usec) 386.502147 main_thread [1512] 801110 pps (802464 pkts in 1004497 usec) 387.502853 main_thread [1512] 800844 pps (801056 pkts in 1000705 usec) 388.503651 main_thread [1512] 801519 pps (801760 pkts in 1000799 usec) 389.504654 main_thread [1512] 801394 pps (801696 pkts in 1001002 usec) 184.339250 sender_body [1179] pending tx tail 1433 head 51 on ring 0 ===> BOX #2 Intel: on the other box: intel# ifconfig ix0 ix0: flags=28943 metric 0 mtu 1500 options=8407b8 ether 00:07:e9:44:d2:ba inet 10.1.1.1 netmask 0xffffff00 broadcast 10.1.1.255 nd6 options=29 media: Ethernet Other instance 8 (10Gbase-SR ) status: active intel# ./pkt-gen -i ix0 -f tx -d 00:07:43:33:8d:c1 -s 00:07:e9:44:d2:ba 267.767848 main [1715] interface is ix0 267.767990 extract_ip_range [291] range is 0.0.0.0:90 to 0.0.0.0:90 267.768006 extract_ip_range [291] range is 0.0.0.0:7 to 0.0.0.0:7 267.872796 main [1910] mapped 334980KB at 0x801c00000 Sending on netmap:ix0: 8 queues, 1 threads and 1 cpus. 00 -> 00 (00:00:00:00:00:00 -> ff:ff:ff:ff:ff:ff) 267.872867 main [1994] Sending 512 packets every 0.000000000 s 267.872872 main [1996] Wait 2 secs for phy reset 269.878594 main [1998] Ready... 269.878716 nm_open [456] overriding ifname ix0 ringid 0x0 flags 0x1 269.896132 sender_body [1073] start, fd 4 main_fd 3 269.930589 sender_body [1142] drop copy 270.907474 main_thread [1512] 14483889 pps (14648527 pkts in 1011367 usec) 271.910479 main_thread [1512] 14882743 pps (14927466 pkts in 1003005 usec) 272.910975 main_thread [1512] 14879303 pps (14886683 pkts in 1000496 usec) 273.912467 main_thread [1512] 14879189 pps (14901389 pkts in 1001492 usec) 274.912970 main_thread [1512] 14881027 pps (14888527 pkts in 1000504 usec) ===> BOX #1 Chelsio, Other Terminal chelsio# while true ;do sysctl dev.cxl | grep tx_pause ; sleep 1 ; done dev.cxl.1.stats.tx_pause: 0 dev.cxl.0.stats.tx_pause: 0 dev.cxl.1.stats.tx_pause: 0 dev.cxl.0.stats.tx_pause: 0 dev.cxl.1.stats.tx_pause: 0 dev.cxl.0.stats.tx_pause: 0 dev.cxl.1.stats.tx_pause: 0 dev.cxl.0.stats.tx_pause: 0 (...) counters for tx_pause keep unchanged. tried with and without rxcsum/txcsu/tso4 what else should i try?