From owner-freebsd-net@freebsd.org Sat Jan 23 17:13:02 2016
Return-Path: <owner-freebsd-net@freebsd.org>
Delivered-To: freebsd-net@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
[IPv6:2001:1900:2254:206a::19:1])
by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6FDC8A8EAAE
for <freebsd-net@mailman.ysv.freebsd.org>;
Sat, 23 Jan 2016 17:13:02 +0000 (UTC)
(envelope-from cenzatti@hush.com)
Received: from smtp3.hushmail.com (smtp3a.hushmail.com [65.39.178.201])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "smtp.hushmail.com", Issuer "smtp.hushmail.com" (not verified))
by mx1.freebsd.org (Postfix) with ESMTPS id 568BF1293
for <freebsd-net@freebsd.org>; Sat, 23 Jan 2016 17:13:01 +0000 (UTC)
(envelope-from cenzatti@hush.com)
Received: from smtp3.hushmail.com (smtp3a.hushmail.com [65.39.178.201])
by smtp3.hushmail.com (Postfix) with SMTP id 37272E0264
for <freebsd-net@freebsd.org>; Sat, 23 Jan 2016 17:13:00 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=hush.com;
h=date:to:subject:from; s=hush;
bh=hVKRM2Erm5dv9jzviRkO4K6UzdLVheBEupEpBAPByUc=;
b=gNIeGVsMQLQbbnoFFTt28F/9Z/T9wg5gOoocpDW73vlHqq5+6DEmfcL3tk3KDFzjVwlMumPjT0Ysea3/Y+p8YoquUedXG2jdlykzvUI+1nFew8hq2QxeriEJ6C2UPPdwCglxmJK9QoOfbMVWYCAOv/HMH3/9LBwacwsZamE8311Turm6tw0mkhCq12B6H1Y2UbTXYEvpQdSDoPadRP2yKdPfxmYCCmALZ4pkGwQpL4vz01odMwmDMq9u6Mxcip2aCon7gkCZT00xLSFbhn5H2Lx9/FAftstV7L5SzRQZ2SO+eUGpuXDfLjbykT6UM0vrbvaSvujF0I38w9d4SneNaQ==
Received: from smtp.hushmail.com (w5.hushmail.com [65.39.178.80])
by smtp3.hushmail.com (Postfix) with ESMTP;
Sat, 23 Jan 2016 17:13:00 +0000 (UTC)
Received: by smtp.hushmail.com (Postfix, from userid 99)
id 0F448A0121; Sat, 23 Jan 2016 17:13:00 +0000 (UTC)
MIME-Version: 1.0
Date: Sat, 23 Jan 2016 15:12:59 -0200
To: "Navdeep Parhar" <nparhar@gmail.com>
Cc: freebsd-net@freebsd.org
Subject: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX
From: "Marcus Cenzatti" <cenzatti@hush.com>
In-Reply-To: <20160123154052.GA4574@ox>
References: <20160123053428.2091EA0121@smtp.hushmail.com>
<20160123154052.GA4574@ox>
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="UTF-8"
Message-Id: <20160123171300.0F448A0121@smtp.hushmail.com>
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.20
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-net>,
<mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-net>,
<mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sat, 23 Jan 2016 17:13:02 -0000
On 1/23/2016 at 1:40 PM, "Navdeep Parhar" <nparhar@gmail.com> wrote:
>
>On Sat, Jan 23, 2016 at 03:34:27AM -0200, Marcus Cenzatti wrote:
>> hello,
>>
>> I am testing a chelsio t520-so-cr connected to a Intel card with
>ix(4)
>> driver, I can get the ncxl0 interface to transmit at 14Mpps to
>another
>> chelsio or to a Intel card. However I can only get 800Kpps-1Mpps
>for
>> RX tests from both chelsio or Intel.
>>
>> I have test with both FreeBSD 11 and FreeBSD 10.3-PRERELEASE.
>>
>> I tested it untuned first and later I have applied tuning
>> recommendations I found on BSDRP[1] website. Results still
>ranging
>> from 800Kpps to 1Mpps for RX.
>>
>> Tests are done w/ with pkt-gen in netmap mode on ncxl interface
>with
>> both IP address and MAC address source/dest.
>
>The ncxl interfaces have their own MAC addresses. Make sure the
>sender
>uses the MAC of the receiver's ncxl interface as the destination
>MAC.
>(netmap's pkt-gen -f tx transmits L2 broadcasts by default).
>
>Check for PAUSE frames coming out of the receiver (sysctl dev.cxl
>| grep
>tx_pause). If it's receiving frames on netmap interface the
>tx_pause
>counter should not move.
>
>Regards,
>Navdeep
>
hello,
yes, MAC addresses are correct, I did the tests again and tx_pause won't move, here is the full transcript for the tests:
===> BOX #1 CHELSIO
chelsio# ifconfig -v ncxl0
ncxl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 00:07:43:33:8d:c1
inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-SR <full-duplex>
status: active
chelsio# ifconfig -v cxl0
cxl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=ec00bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:07:43:33:8d:c0
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-SR <full-duplex>
status: active
plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
vendor: FINISAR CORP. PN: FTLX8571D3BCL-FC SN: AL1073K DATE: 2011-06-28
module temperature: 42.79 C Voltage: 3.23 Volts
RX: 0.53 mW (-2.74 dBm) TX: 0.48 mW (-3.12 dBm)
chelsio# ./pkt-gen -i ncxl0 -f rx -d 00:07:43:33:8d:c1 -s 00:07:e9:44:d2:ba
311.132189 main [1715] interface is ncxl0
311.132447 extract_ip_range [291] range is 0.0.0.0:90 to 0.0.0.0:90
311.132472 extract_ip_range [291] range is 0.0.0.0:7 to 0.0.0.0:7
311.191286 main [1910] mapped 334980KB at 0x801c00000
Receiving from netmap:ncxl0: 2 queues, 1 threads and 1 cpus.
311.191371 main [1996] Wait 2 secs for phy reset
313.230423 main [1998] Ready...
313.230570 nm_open [456] overriding ifname ncxl0 ringid 0x0 flags 0x1
313.247998 receiver_body [1247] reading from netmap:ncxl0 fd 4 main_fd 3
314.249172 receiver_body [1254] waiting for initial packets, poll returns 0 0
(...)
when the other box starts transmitting:
(...)
380.492236 main_thread [1512] 0 pps (0 pkts in 1006897 usec)
381.326159 receiver_body [1254] waiting for initial packets, poll returns 0 0
381.493141 main_thread [1512] 80110 pps (80137 pkts in 1000906 usec)
382.494145 main_thread [1512] 801585 pps (801888 pkts in 1001004 usec)
383.494979 main_thread [1512] 801381 pps (801632 pkts in 1000833 usec)
384.497148 main_thread [1512] 801490 pps (802144 pkts in 1002169 usec)
385.497650 main_thread [1512] 801416 pps (801568 pkts in 1000503 usec)
386.502147 main_thread [1512] 801110 pps (802464 pkts in 1004497 usec)
387.502853 main_thread [1512] 800844 pps (801056 pkts in 1000705 usec)
388.503651 main_thread [1512] 801519 pps (801760 pkts in 1000799 usec)
389.504654 main_thread [1512] 801394 pps (801696 pkts in 1001002 usec)
184.339250 sender_body [1179] pending tx tail 1433 head 51 on ring 0
===> BOX #2 Intel:
on the other box:
intel# ifconfig ix0
ix0: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8407b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO>
ether 00:07:e9:44:d2:ba
inet 10.1.1.1 netmask 0xffffff00 broadcast 10.1.1.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet Other <full-duplex,master,rxpause,txpause> instance 8 (10Gbase-SR
<full-duplex,rxpause,txpause>)
status: active
intel# ./pkt-gen -i ix0 -f tx -d 00:07:43:33:8d:c1 -s 00:07:e9:44:d2:ba
267.767848 main [1715] interface is ix0
267.767990 extract_ip_range [291] range is 0.0.0.0:90 to 0.0.0.0:90
267.768006 extract_ip_range [291] range is 0.0.0.0:7 to 0.0.0.0:7
267.872796 main [1910] mapped 334980KB at 0x801c00000
Sending on netmap:ix0: 8 queues, 1 threads and 1 cpus.
00 -> 00 (00:00:00:00:00:00 -> ff:ff:ff:ff:ff:ff)
267.872867 main [1994] Sending 512 packets every 0.000000000 s
267.872872 main [1996] Wait 2 secs for phy reset
269.878594 main [1998] Ready...
269.878716 nm_open [456] overriding ifname ix0 ringid 0x0 flags 0x1
269.896132 sender_body [1073] start, fd 4 main_fd 3
269.930589 sender_body [1142] drop copy
270.907474 main_thread [1512] 14483889 pps (14648527 pkts in 1011367 usec)
271.910479 main_thread [1512] 14882743 pps (14927466 pkts in 1003005 usec)
272.910975 main_thread [1512] 14879303 pps (14886683 pkts in 1000496 usec)
273.912467 main_thread [1512] 14879189 pps (14901389 pkts in 1001492 usec)
274.912970 main_thread [1512] 14881027 pps (14888527 pkts in 1000504 usec)
===> BOX #1 Chelsio, Other Terminal
chelsio# while true ;do sysctl dev.cxl | grep tx_pause ; sleep 1 ; done
dev.cxl.1.stats.tx_pause: 0
dev.cxl.0.stats.tx_pause: 0
dev.cxl.1.stats.tx_pause: 0
dev.cxl.0.stats.tx_pause: 0
dev.cxl.1.stats.tx_pause: 0
dev.cxl.0.stats.tx_pause: 0
dev.cxl.1.stats.tx_pause: 0
dev.cxl.0.stats.tx_pause: 0
(...)
counters for tx_pause keep unchanged.
tried with and without rxcsum/txcsu/tso4
what else should i try?