Date: Sat, 23 Jan 2016 15:12:59 -0200 From: "Marcus Cenzatti" <cenzatti@hush.com> To: "Navdeep Parhar" <nparhar@gmail.com> Cc: freebsd-net@freebsd.org Subject: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX Message-ID: <20160123171300.0F448A0121@smtp.hushmail.com> In-Reply-To: <20160123154052.GA4574@ox> References: <20160123053428.2091EA0121@smtp.hushmail.com> <20160123154052.GA4574@ox>
next in thread | previous in thread | raw e-mail | index | archive | help
On 1/23/2016 at 1:40 PM, "Navdeep Parhar" <nparhar@gmail.com> wrote:
>
>On Sat, Jan 23, 2016 at 03:34:27AM -0200, Marcus Cenzatti wrote:
>> hello,
>>
>> I am testing a chelsio t520-so-cr connected to a Intel card with
>ix(4)
>> driver, I can get the ncxl0 interface to transmit at 14Mpps to
>another
>> chelsio or to a Intel card. However I can only get 800Kpps-1Mpps
>for
>> RX tests from both chelsio or Intel.
>>
>> I have test with both FreeBSD 11 and FreeBSD 10.3-PRERELEASE.
>>
>> I tested it untuned first and later I have applied tuning
>> recommendations I found on BSDRP[1] website. Results still
>ranging
>> from 800Kpps to 1Mpps for RX.
>>
>> Tests are done w/ with pkt-gen in netmap mode on ncxl interface
>with
>> both IP address and MAC address source/dest.
>
>The ncxl interfaces have their own MAC addresses. Make sure the
>sender
>uses the MAC of the receiver's ncxl interface as the destination
>MAC.
>(netmap's pkt-gen -f tx transmits L2 broadcasts by default).
>
>Check for PAUSE frames coming out of the receiver (sysctl dev.cxl
>| grep
>tx_pause). If it's receiving frames on netmap interface the
>tx_pause
>counter should not move.
>
>Regards,
>Navdeep
>
hello,
yes, MAC addresses are correct, I did the tests again and tx_pause won't move, here is the full transcript for the tests:
===> BOX #1 CHELSIO
chelsio# ifconfig -v ncxl0
ncxl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 00:07:43:33:8d:c1
inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-SR <full-duplex>
status: active
chelsio# ifconfig -v cxl0
cxl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=ec00bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:07:43:33:8d:c0
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-SR <full-duplex>
status: active
plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
vendor: FINISAR CORP. PN: FTLX8571D3BCL-FC SN: AL1073K DATE: 2011-06-28
module temperature: 42.79 C Voltage: 3.23 Volts
RX: 0.53 mW (-2.74 dBm) TX: 0.48 mW (-3.12 dBm)
chelsio# ./pkt-gen -i ncxl0 -f rx -d 00:07:43:33:8d:c1 -s 00:07:e9:44:d2:ba
311.132189 main [1715] interface is ncxl0
311.132447 extract_ip_range [291] range is 0.0.0.0:90 to 0.0.0.0:90
311.132472 extract_ip_range [291] range is 0.0.0.0:7 to 0.0.0.0:7
311.191286 main [1910] mapped 334980KB at 0x801c00000
Receiving from netmap:ncxl0: 2 queues, 1 threads and 1 cpus.
311.191371 main [1996] Wait 2 secs for phy reset
313.230423 main [1998] Ready...
313.230570 nm_open [456] overriding ifname ncxl0 ringid 0x0 flags 0x1
313.247998 receiver_body [1247] reading from netmap:ncxl0 fd 4 main_fd 3
314.249172 receiver_body [1254] waiting for initial packets, poll returns 0 0
(...)
when the other box starts transmitting:
(...)
380.492236 main_thread [1512] 0 pps (0 pkts in 1006897 usec)
381.326159 receiver_body [1254] waiting for initial packets, poll returns 0 0
381.493141 main_thread [1512] 80110 pps (80137 pkts in 1000906 usec)
382.494145 main_thread [1512] 801585 pps (801888 pkts in 1001004 usec)
383.494979 main_thread [1512] 801381 pps (801632 pkts in 1000833 usec)
384.497148 main_thread [1512] 801490 pps (802144 pkts in 1002169 usec)
385.497650 main_thread [1512] 801416 pps (801568 pkts in 1000503 usec)
386.502147 main_thread [1512] 801110 pps (802464 pkts in 1004497 usec)
387.502853 main_thread [1512] 800844 pps (801056 pkts in 1000705 usec)
388.503651 main_thread [1512] 801519 pps (801760 pkts in 1000799 usec)
389.504654 main_thread [1512] 801394 pps (801696 pkts in 1001002 usec)
184.339250 sender_body [1179] pending tx tail 1433 head 51 on ring 0
===> BOX #2 Intel:
on the other box:
intel# ifconfig ix0
ix0: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8407b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO>
ether 00:07:e9:44:d2:ba
inet 10.1.1.1 netmask 0xffffff00 broadcast 10.1.1.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet Other <full-duplex,master,rxpause,txpause> instance 8 (10Gbase-SR
<full-duplex,rxpause,txpause>)
status: active
intel# ./pkt-gen -i ix0 -f tx -d 00:07:43:33:8d:c1 -s 00:07:e9:44:d2:ba
267.767848 main [1715] interface is ix0
267.767990 extract_ip_range [291] range is 0.0.0.0:90 to 0.0.0.0:90
267.768006 extract_ip_range [291] range is 0.0.0.0:7 to 0.0.0.0:7
267.872796 main [1910] mapped 334980KB at 0x801c00000
Sending on netmap:ix0: 8 queues, 1 threads and 1 cpus.
00 -> 00 (00:00:00:00:00:00 -> ff:ff:ff:ff:ff:ff)
267.872867 main [1994] Sending 512 packets every 0.000000000 s
267.872872 main [1996] Wait 2 secs for phy reset
269.878594 main [1998] Ready...
269.878716 nm_open [456] overriding ifname ix0 ringid 0x0 flags 0x1
269.896132 sender_body [1073] start, fd 4 main_fd 3
269.930589 sender_body [1142] drop copy
270.907474 main_thread [1512] 14483889 pps (14648527 pkts in 1011367 usec)
271.910479 main_thread [1512] 14882743 pps (14927466 pkts in 1003005 usec)
272.910975 main_thread [1512] 14879303 pps (14886683 pkts in 1000496 usec)
273.912467 main_thread [1512] 14879189 pps (14901389 pkts in 1001492 usec)
274.912970 main_thread [1512] 14881027 pps (14888527 pkts in 1000504 usec)
===> BOX #1 Chelsio, Other Terminal
chelsio# while true ;do sysctl dev.cxl | grep tx_pause ; sleep 1 ; done
dev.cxl.1.stats.tx_pause: 0
dev.cxl.0.stats.tx_pause: 0
dev.cxl.1.stats.tx_pause: 0
dev.cxl.0.stats.tx_pause: 0
dev.cxl.1.stats.tx_pause: 0
dev.cxl.0.stats.tx_pause: 0
dev.cxl.1.stats.tx_pause: 0
dev.cxl.0.stats.tx_pause: 0
(...)
counters for tx_pause keep unchanged.
tried with and without rxcsum/txcsu/tso4
what else should i try?
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20160123171300.0F448A0121>