Date: Sun, 24 Jan 2016 00:40:39 -0200 From: "Marcus Cenzatti" <cenzatti@hush.com> To: "Luigi Rizzo" <rizzo@iet.unipi.it>, freebsd-net@freebsd.org, "Navdeep Parhar" <nparhar@gmail.com> Subject: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX Message-ID: <20160124024040.128A2A0128@smtp.hushmail.com> In-Reply-To: <CA%2BhQ2%2BgBiBdf1QtSCmR=ikDCo1LKYcs6NaTkEnDS9LZGkHn7rQ@mail.gmail.com> References: <20160123053428.2091EA0121@smtp.hushmail.com> <20160123154052.GA4574@ox> <20160123171300.0F448A0121@smtp.hushmail.com> <CA%2BhQ2%2Bg4kU4LA4PexRPBv7z49ZWh-mDqdpw18SeoYaBueHyjZg@mail.gmail.com> <20160123174840.32B1DA0121@smtp.hushmail.com> <20160123183836.GB4574@ox> <CA%2BhQ2%2BiWKRhhFntjkkYyfwSmbCrKrw8BkEhUXFbbf0hG%2BQs0yA@mail.gmail.com> <20160123211816.GE4574@ox> <CA%2BhQ2%2BgBiBdf1QtSCmR=ikDCo1LKYcs6NaTkEnDS9LZGkHn7rQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 1/23/2016 at 10:11 PM, "Luigi Rizzo" <rizzo@iet.unipi.it> wrote:
>
>On Sat, Jan 23, 2016 at 1:18 PM, Navdeep Parhar
><nparhar@gmail.com> wrote:
>> On Sat, Jan 23, 2016 at 11:12:28AM -0800, Luigi Rizzo wrote:
>>> On Sat, Jan 23, 2016 at 10:38 AM, Navdeep Parhar
><nparhar@gmail.com> wrote:
>>> > On Sat, Jan 23, 2016 at 03:48:39PM -0200, Marcus Cenzatti
>wrote:
>>> > ...
>>> >>
>>> >> woops, my bad, yes probably we had some drop, with -S and -D
>now I get 1.2Mpps.
>>> >
>>> > Run "netstat -hdw1 -i cxl<n>" on the receiver during your
>test.
>>>
>>> Navdeep, does this give any info on the ncxl port rather
>>> than the cxl port connected to the host stack ?
>>
>> You're right, it should have been "netstat -hdw 1 -I ncxl<n>".
>In these
>> kinds of experiments it might even be best to run two netstats in
>> parallel on cxl and ncxl.
>>
>>>
>>> ...
>>> > Do you know if the transmitter will pad up so as not to put
>runts on the
>>> > wire? If not then you might want to bump up the size of the
>frame
>>> > explicitly (there's some pkt-gen knob for this).
>>> >
>>>
>>> ix/ixl do automatic padding, and in any case pkt-gen
>>> by default generates valid packet sizes (and so it does
>>> with the variable-size tests I suggested).
>>>
>>> Is there any parameter that controls interrupt moderation ?
>>>
>>> In any case we need to know the numbers when sending to the
>>> ncxl MAC address as opposed to broadcast.
>>>
>>> I suspects one of these problems:
>>>
>>> - the interrupt moderation interval is too long thus limiting
>>> the rate to one ring/interval. Unlikely though, even
>>> with 1k slots, the measured 1.2 Mpps corresponds to almost
>>> 1ms which is too long
>>>
>>> - the receiver cannot cope with the input load and somehow
>>> takes a long time to recover from congestion. If this is
>>> the case, running the sender at a lower rate might reach
>>> a peak throughput > 1.2 Mpps when the receiver can still
>>> keep up, and then drop to the low rate under congestion.
>>>
>>> - and of course bus errors, when the device is connected on
>>> a PCIe slot with only 1-2 data lanes.
>>> This actually happens a lot, physical connector sizes
>>> do not reflect the number of active PCIe lanes.
>>
>> There are no drops or PAUSE or any sign of backpressure. The
>netstat
>> counters show 900K incoming and 0 drops/errors, which means 900K
>packets
>> on the wire for the port and all were delivered to the driver
>> successfully.
>
>I am not 100% convinced by the above explanation, both because
>of the way the experiment was conducted (broadcast
>destination, duplication in the hw to two queues,
>stats only from cxl0 and not ncxl0), and because sending
>to the unicast DMAC already showed a higher throughput.
>
>So I suspect the 900k counter does not actually reflect
>packets on the wire. Where and how the drops are counted
>I have no idea (presumably some internal counter in the NIC,
>but for broadcast traffic replicated to multiple queues
>I don't know which counter tracks the drop - one per queue,
>a global one, etc.
>
>
>> The mismatch in the transmitter's counter and the incoming
>counter can
>> only be explained by
>> a) Frames whose DMAC address didn't match the local interface's
>MAC.
>
>the DMAC was the same for all frames.
>
>> This can be tested by switching cxl0 and ncxl0 to promisc mode
>to see if
>> that opens the flood gates.
>> b) Frames mangled badly enough to be discarded. But these
>should show
>> as an error or drop in at least one of these:
>>
>> sysctl dev.cxl.<n>.stats
>> sysctl -n dev.t5nex.0.misc.tp_err_stats
>
>one question -- do the sysctl for cxl.<n> and t5nex.0
>also report the ncxl* stats, or you have a separate entry
>for those ?
>
>cheers
>luigi
dear gentlemen,
i started over in the end of the day, rebooted everything and removed all tweeks (had no sysctl tweeks on the previous output but had removed tx|rx csum and tso from both intel and chelsio).
mr Adrian Chadd suggested to shut cxl0 down but if I ifconfig cxl0 down both cxl0 and ncxl0 will be down, anyway I removed IP address from cxl0 and let only ncxl0 addressed.
I have new numbers:
- MAC address based tests: 14Mpps TX => 11Mpps RX (much better, finally, but still lower than Intel-Intel, not a problem at all)
- IP address based tests: 14Mpps TX => 300Kpps RX (kernel rate possibly, but on netmap mode, why? a problem for sure)
on both experiments we actually had no error pointed on netstat or sysctls, and actual RX speed observed on netstat on the second test was 15Mpps.
here is the full transcript for the two session tests with stats outputs.
things i could not understand are:
1) on mac address tests the sender (tx) pkt-gen shows:
Sending on netmap:ix0: 8 queues, 1 threads and 1 cpus.
10.0.0.1 -> 10.1.0.1 (00:07:e9:44:d2:ba -> 00:07:43:33:8d:c1)
where those ip addresses come from? same is observed on rx:
529.477028 extract_ip_range [291] range is 10.0.0.1:0 to 10.0.0.1:0
529.477042 extract_ip_range [291] range is 10.1.0.1:0 to 10.1.0.1:0
2) on ip address tests, the sender (tx/intel) pkt-gen shows:
172.16.0.2 -> 172.16.0.1 (00:00:00:00:00:00 -> ff:ff:ff:ff:ff:ff)
does this mean we are transmitting to broadcast?
the receiver (chelsio/tx) shows this, that makes much more sense:
500.757057 main [1715] interface is ncxl0
500.757321 extract_ip_range [291] range is 172.16.0.2:0 to 172.16.0.2:0
500.757348 extract_ip_range [291] range is 172.16.0.1:0 to 172.16.0.1:0
501.007425 main [1910] mapped 334980KB at 0x801c00000
so I could not explain why I had 800Kpps rates on chelsio when I had that tweeks and after rebooting i have a much more decent 11Mpps rate but i could confirm, if I remove the tweeks I get 800K-1.2M pps again on MAC address tests.
since cxl0 is not addressed and ncxl0 will only work in netmap mode i believe it will never arp reply, therefore the netmap application should arp reply when someone arp requests the actual mac address for a given ip address, will pkt-gen arp reply? because this could, maybe, explain the destination part for 00:00:00:00:00:00 -> ff:ff:ff:ff:ff:ff (but source mac address should be correct)
just to remember, when I test intel-intel I get the same 14Mpps RX and TX rates on both IP address test as well as my previously wrong (using -s|-d instead of -S|-D pkt-gen options) mac address tests, but i don't remember if mac address were shown like that
luigi, the sysctl you asked navdeep about, sysctl dev.ncxl.0.stats, does not exist, only sysctl dev.cxl.0.stats exists and I could not find other stats similar to dev.t5nex.0.misc.tp_err_stats so probably it's not different betweek cxl and ncxl
==== FULL TESTS TRANSCRIPT ====
intel# ifconfig -v ix0
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8406b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO6,LRO,VLAN_HWTSO>
ether 00:07:e9:44:d2:ba
inet 172.16.0.2 netmask 0xffffff00 broadcast 172.16.0.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-SR (10Gbase-SR <full-duplex,rxpause,txpause>)
status: active
plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
vendor: FINISAR CORP. PN: FTLX8571D3BCL-FC SN: AM31TRU DATE: 2012-01-21
module temperature: 34.98 C Voltage: 3.28 Volts
RX: 0.50 mW (-2.98 dBm) TX: 0.58 mW (-2.33 dBm)
intel# ./pkt-gen -i ix0 -f tx -S 00:07:e9:44:d2:ba -D 00:07:43:33:8d:c1
268.505741 main [1715] interface is ix0
268.505795 extract_ip_range [291] range is 10.0.0.1:0 to 10.0.0.1:0
268.505806 extract_ip_range [291] range is 10.1.0.1:0 to 10.1.0.1:0
268.611395 main [1910] mapped 334980KB at 0x801c00000
Sending on netmap:ix0: 8 queues, 1 threads and 1 cpus.
10.0.0.1 -> 10.1.0.1 (00:07:e9:44:d2:ba -> 00:07:43:33:8d:c1)
268.611467 main [1994] Sending 512 packets every 0.000000000 s
268.611473 main [1996] Wait 2 secs for phy reset
270.617971 main [1998] Ready...
270.618120 nm_open [456] overriding ifname ix0 ringid 0x0 flags 0x1
270.635480 sender_body [1073] start, fd 4 main_fd 3
270.670823 sender_body [1142] drop copy
271.637012 main_thread [1512] 14461469 pps (14484000 pkts in 1001558 usec)
272.647011 main_thread [1512] 14880527 pps (15029317 pkts in 1009999 usec)
273.648511 main_thread [1512] 14884638 pps (14906965 pkts in 1001500 usec)
274.649506 main_thread [1512] 14883795 pps (14898619 pkts in 1000996 usec)
275.650509 main_thread [1512] 14879377 pps (14894301 pkts in 1001003 usec)
276.651511 main_thread [1512] 14883276 pps (14898174 pkts in 1001001 usec)
277.652512 main_thread [1512] 14880941 pps (14895852 pkts in 1001002 usec)
278.653517 main_thread [1512] 14879125 pps (14894079 pkts in 1001005 usec)
279.654525 main_thread [1512] 14887529 pps (14902431 pkts in 1001001 usec)
280.656518 main_thread [1512] 14864554 pps (14894268 pkts in 1001999 usec)
281.657012 main_thread [1512] 14882656 pps (14890008 pkts in 1000494 usec)
282.658506 main_thread [1512] 14881346 pps (14903594 pkts in 1001495 usec)
283.664016 main_thread [1512] 14882829 pps (14964819 pkts in 1005509 usec)
284.690513 main_thread [1512] 14880361 pps (15274661 pkts in 1026498 usec)
285.691504 main_thread [1512] 14877683 pps (14892412 pkts in 1000990 usec)
286.692143 main_thread [1512] 14885234 pps (14894761 pkts in 1000640 usec)
287.693510 main_thread [1512] 14885267 pps (14905615 pkts in 1001367 usec)
288.694010 main_thread [1512] 14879692 pps (14887132 pkts in 1000500 usec)
^C289.189696 sigint_h [328] received control-C on thread 0x801806800
289.189750 sender_body [1171] flush tail 19 head 19 on thread 0x801806800
289.189790 sender_body [1179] pending tx tail 72 head 1105 on ring 0
289.189823 sender_body [1179] pending tx tail 154 head 1105 on ring 0
289.189848 sender_body [1179] pending tx tail 222 head 1105 on ring 0
289.189871 sender_body [1179] pending tx tail 274 head 1105 on ring 0
289.189901 sender_body [1179] pending tx tail 338 head 1105 on ring 0
289.700237 main_thread [1512] 7332017 pps (7377666 pkts in 1006226 usec)
Sent 275688674 packets, 60 bytes each, in 18.55 seconds.
Speed: 14.86 Mpps Bandwidth: 7.13 Gbps (raw 9.98 Gbps)
intel#
chelsio# ./pkt-gen -i ncxl0 -f rx -S 00:07:e9:44:d2:ba -D 00:07:43:33:8d:c1
529.476963 main [1715] interface is ncxl0
529.477028 extract_ip_range [291] range is 10.0.0.1:0 to 10.0.0.1:0
529.477042 extract_ip_range [291] range is 10.1.0.1:0 to 10.1.0.1:0
529.535750 main [1910] mapped 334980KB at 0x801c00000
Receiving from netmap:ncxl0: 2 queues, 1 threads and 1 cpus.
529.535829 main [1996] Wait 2 secs for phy reset
531.537267 main [1998] Ready...
531.537399 nm_open [456] overriding ifname ncxl0 ringid 0x0 flags 0x1
531.555050 receiver_body [1247] reading from netmap:ncxl0 fd 4 main_fd 3
532.555628 main_thread [1512] 11173691 pps (11180373 pkts in 1000598 usec)
533.556129 main_thread [1512] 11175384 pps (11180983 pkts in 1000501 usec)
534.567273 main_thread [1512] 11176345 pps (11300905 pkts in 1011145 usec)
535.568126 main_thread [1512] 11175813 pps (11185335 pkts in 1000852 usec)
536.569623 main_thread [1512] 11174668 pps (11191408 pkts in 1001498 usec)
537.580127 main_thread [1512] 11176569 pps (11293968 pkts in 1010504 usec)
(..)
739.193065 main_thread [1512] 11173118 pps (11177744 pkts in 1000414 usec)
740.194062 main_thread [1512] 11175690 pps (11186832 pkts in 1000997 usec)
741.202063 main_thread [1512] 11176354 pps (11265776 pkts in 1008001 usec)
742.203564 main_thread [1512] 11176509 pps (11193296 pkts in 1001502 usec)
743.204559 main_thread [1512] 11175825 pps (11186945 pkts in 1000995 usec)
^C743.392524 sigint_h [328] received control-C on thread 0x801806800
744.212517 main_thread [1512] 2080802 pps (2097359 pkts in 1007957 usec)
Received 2367310284 packets, in 211.84 seconds.
Speed: 11.18 Mpps
chelsio# netstat -hd -I ncxl0
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop
ncxl0 1.5K <Link#4> 00:07:43:33:8d:c1 0 0 0 0 0 0 0
ncxl0 - 172.16.0.0 172.16.0.1 158M - - 0 - - -
chelsio# netstat -hd -I cxl0
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop
cxl0 1.5K <Link#3> 00:07:43:33:8d:c0 7.1G 0 669M 0 0 0 0
chelsio# netstat -hdwi 1 -l ncxl0
input (Total) output
packets errs idrops bytes packets errs bytes colls drops
15M 0 3.7M 908M 4 0 660 0 0
15M 0 3.7M 908M 4 0 612 0 0
15M 0 3.7M 908M 4 0 612 0 0
15M 0 3.7M 908M 4 0 612 0 0
15M 0 3.7M 908M 4 0 612 0 0
15M 0 3.7M 908M 9 0 1.2K 0 0
15M 0 3.7M 908M 6 0 1.1K 0 0
15M 0 3.7M 908M 4 0 612 0 0
^C
chelsio# netstat -hdwi 1 -l cxl0
input (Total) output
packets errs idrops bytes packets errs bytes colls drops
15M 0 3.7M 908M 4 0 660 0 0
15M 0 3.7M 908M 4 0 612 0 0
15M 0 3.7M 908M 6 0 763 0 0
15M 0 3.7M 908M 4 0 612 0 0
15M 0 3.7M 908M 4 0 612 0 0
15M 0 3.7M 908M 9 0 1.0K 0 0
^C
chelsio# netstat -hd -I ncxl0
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop
ncxl0 1.5K <Link#4> 00:07:43:33:8d:c1 0 0 0 0 0 0 0
ncxl0 - 172.16.0.0 172.16.0.1 39M - - 0 - - -
chelsio#
chelsio# netstat -hd -I cxl0
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop
cxl0 1.5K <Link#3> 00:07:43:33:8d:c0 6.0G 1 612k 0 0 0 0
chelsio# netstat -hdwi 1 -l ncxl0
input (Total) output
packets errs idrops bytes packets errs bytes colls drops
15M 0 0 908M 8 0 1.1K 0 0
15M 0 0 908M 4 0 604 0 0
15M 0 0 908M 4 0 604 0 0
15M 0 0 908M 15 0 1.8K 0 0
15M 0 0 908M 10 0 1.2K 0 0
15M 0 0 908M 4 0 604 0 0
15M 0 0 908M 6 0 822 0 0
15M 0 0 908M 6 0 826 0 0
^C
chelsio# netstat -hdwi 1 -l cxl0
input (Total) output
packets errs idrops bytes packets errs bytes colls drops
15M 0 0 908M 8 0 1.0K 0 0
15M 0 0 908M 4 0 604 0 0
15M 0 0 908M 5 0 756 0 0
15M 0 0 908M 10 0 1.2K 0 0
15M 0 0 908M 4 0 604 0 0
15M 0 0 908M 4 0 604 0 0
^C
chelsio# sysctl -n dev.t5nex.0.misc.tp_err_stats
channel 0 channel 1 channel 2 channel 3
macInErrs: 0 0 0 0
hdrInErrs: 0 0 0 0
tcpInErrs: 0 0 0 0
tcp6InErrs: 0 0 0 0
tnlCongDrops: 42862617 0 0 0
tnlTxDrops: 0 0 0 0
ofldVlanDrops: 0 0 0 0
ofldChanDrops: 0 0 0 0
ofldNoNeigh: 0
ofldCongDefer: 0
chelsio# sysctl dev.cxl.0.stats
dev.cxl.0.stats.rx_trunc3: 0
dev.cxl.0.stats.rx_trunc2: 0
dev.cxl.0.stats.rx_trunc1: 0
dev.cxl.0.stats.rx_trunc0: 0
dev.cxl.0.stats.rx_ovflow3: 0
dev.cxl.0.stats.rx_ovflow2: 0
dev.cxl.0.stats.rx_ovflow1: 0
dev.cxl.0.stats.rx_ovflow0: 853083783
dev.cxl.0.stats.rx_ppp7: 0
dev.cxl.0.stats.rx_ppp6: 0
dev.cxl.0.stats.rx_ppp5: 0
dev.cxl.0.stats.rx_ppp4: 0
dev.cxl.0.stats.rx_ppp3: 0
dev.cxl.0.stats.rx_ppp2: 0
dev.cxl.0.stats.rx_ppp1: 0
dev.cxl.0.stats.rx_ppp0: 0
dev.cxl.0.stats.rx_pause: 0
dev.cxl.0.stats.rx_frames_1519_max: 0
dev.cxl.0.stats.rx_frames_1024_1518: 0
dev.cxl.0.stats.rx_frames_512_1023: 0
dev.cxl.0.stats.rx_frames_256_511: 0
dev.cxl.0.stats.rx_frames_128_255: 0
dev.cxl.0.stats.rx_frames_65_127: 0
dev.cxl.0.stats.rx_frames_64: 9778077198
dev.cxl.0.stats.rx_runt: 0
dev.cxl.0.stats.rx_symbol_err: 0
dev.cxl.0.stats.rx_len_err: 0
dev.cxl.0.stats.rx_fcs_err: 0
dev.cxl.0.stats.rx_jabber: 0
dev.cxl.0.stats.rx_too_long: 0
dev.cxl.0.stats.rx_ucast_frames: 5365842946
dev.cxl.0.stats.rx_mcast_frames: 0
dev.cxl.0.stats.rx_bcast_frames: 4412237958
dev.cxl.0.stats.rx_frames: 9778082382
dev.cxl.0.stats.rx_octets: 625797304704
dev.cxl.0.stats.tx_ppp7: 0
dev.cxl.0.stats.tx_ppp6: 0
dev.cxl.0.stats.tx_ppp5: 0
dev.cxl.0.stats.tx_ppp4: 0
dev.cxl.0.stats.tx_ppp3: 0
dev.cxl.0.stats.tx_ppp2: 0
dev.cxl.0.stats.tx_ppp1: 0
dev.cxl.0.stats.tx_ppp0: 0
dev.cxl.0.stats.tx_pause: 0
dev.cxl.0.stats.tx_drop: 0
dev.cxl.0.stats.tx_frames_1519_max: 0
dev.cxl.0.stats.tx_frames_1024_1518: 0
dev.cxl.0.stats.tx_frames_512_1023: 0
dev.cxl.0.stats.tx_frames_256_511: 0
dev.cxl.0.stats.tx_frames_128_255: 0
dev.cxl.0.stats.tx_frames_65_127: 0
dev.cxl.0.stats.tx_frames_64: 0
dev.cxl.0.stats.tx_error_frames: 0
dev.cxl.0.stats.tx_ucast_frames: 0
dev.cxl.0.stats.tx_mcast_frames: 0
dev.cxl.0.stats.tx_bcast_frames: 0
dev.cxl.0.stats.tx_frames: 0
dev.cxl.0.stats.tx_octets: 0
dev.cxl.0.stats.tx_parse_error: 0
chelsio# sysctl dev.ncxl.0.stats
===> IP based tests:
intel# ./pkt-gen -i ix0 -f tx -s 172.16.0.2 -d 172.16.0.1
150.673411 main [1715] interface is ix0
150.673545 extract_ip_range [291] range is 172.16.0.2:0 to 172.16.0.2:0
150.673560 extract_ip_range [291] range is 172.16.0.1:0 to 172.16.0.1:0
150.779091 main [1910] mapped 334980KB at 0x801c00000
Sending on netmap:ix0: 8 queues, 1 threads and 1 cpus.
172.16.0.2 -> 172.16.0.1 (00:00:00:00:00:00 -> ff:ff:ff:ff:ff:ff)
150.779164 main [1994] Sending 512 packets every 0.000000000 s
150.779169 main [1996] Wait 2 secs for phy reset
152.846385 main [1998] Ready...
152.846515 nm_open [456] overriding ifname ix0 ringid 0x0 flags 0x1
152.863946 sender_body [1073] start, fd 4 main_fd 3
152.898386 sender_body [1142] drop copy
153.864749 main_thread [1512] 14477663 pps (14489636 pkts in 1000827 usec)
154.868750 main_thread [1512] 14879784 pps (14939318 pkts in 1004001 usec)
155.873246 main_thread [1512] 14883091 pps (14950005 pkts in 1004496 usec)
156.874753 main_thread [1512] 14883859 pps (14906289 pkts in 1001507 usec)
(...)
chelsio# ./pkt-gen -i ncxl0 -f rx -s 172.16.0.2 -d 172.16.0.1
500.757057 main [1715] interface is ncxl0
500.757321 extract_ip_range [291] range is 172.16.0.2:0 to 172.16.0.2:0
500.757348 extract_ip_range [291] range is 172.16.0.1:0 to 172.16.0.1:0
501.007425 main [1910] mapped 334980KB at 0x801c00000
Receiving from netmap:ncxl0: 2 queues, 1 threads and 1 cpus.
501.007511 main [1996] Wait 2 secs for phy reset
503.015342 main [1998] Ready...
503.015487 nm_open [456] overriding ifname ncxl0 ringid 0x0 flags 0x1
503.033543 receiver_body [1247] reading from netmap:ncxl0 fd 4 main_fd 3
504.034782 main_thread [1512] 303341 pps (303727 pkts in 1001273 usec)
505.035783 main_thread [1512] 301714 pps (302016 pkts in 1001000 usec)
506.036779 main_thread [1512] 302355 pps (302656 pkts in 1000996 usec)
507.037780 main_thread [1512] 301651 pps (301953 pkts in 1001002 usec)
508.038783 main_thread [1512] 301922 pps (302225 pkts in 1001002 usec)
509.039780 main_thread [1512] 301633 pps (301934 pkts in 1000997 usec)
510.040776 main_thread [1512] 304081 pps (304384 pkts in 1000996 usec)
511.043280 main_thread [1512] 305602 pps (306367 pkts in 1002504 usec)
512.044780 main_thread [1512] 305207 pps (305665 pkts in 1001500 usec)
513.045780 main_thread [1512] 305103 pps (305408 pkts in 1001001 usec)
514.056282 main_thread [1512] 304894 pps (308096 pkts in 1010502 usec)
515.057781 main_thread [1512] 305015 pps (305472 pkts in 1001499 usec)
516.058780 main_thread [1512] 305935 pps (306240 pkts in 1000998 usec)
517.066781 main_thread [1512] 305333 pps (307776 pkts in 1008001 usec)
518.073779 main_thread [1512] 304556 pps (306688 pkts in 1006999 usec)
519.081781 main_thread [1512] 304190 pps (306624 pkts in 1008002 usec)
520.087780 main_thread [1512] 304351 pps (306177 pkts in 1005998 usec)
521.088778 main_thread [1512] 305102 pps (305407 pkts in 1000999 usec)
522.089775 main_thread [1512] 305232 pps (305536 pkts in 1000997 usec)
523.090281 main_thread [1512] 305606 pps (305760 pkts in 1000505 usec)
524.091280 main_thread [1512] 305391 pps (305696 pkts in 1000999 usec)
525.093787 main_thread [1512] 305986 pps (306753 pkts in 1002507 usec)
526.101276 main_thread [1512] 305424 pps (307711 pkts in 1007489 usec)
527.102772 main_thread [1512] 305272 pps (305729 pkts in 1001496 usec)
528.106277 main_thread [1512] 305122 pps (306192 pkts in 1003506 usec)
529.107778 main_thread [1512] 305253 pps (305711 pkts in 1001501 usec)
523.090281 main_thread [1512] 305606 pps (305760 pkts in 1000505 usec)
524.091280 main_thread [1512] 305391 pps (305696 pkts in 1000999 usec)
525.093787 main_thread [1512] 305986 pps (306753 pkts in 1002507 usec)
526.101276 main_thread [1512] 305424 pps (307711 pkts in 1007489 usec)
527.102772 main_thread [1512] 305272 pps (305729 pkts in 1001496 usec)
528.106277 main_thread [1512] 305122 pps (306192 pkts in 1003506 usec)
529.107778 main_thread [1512] 305253 pps (305711 pkts in 1001501 usec)
530.110774 main_thread [1512] 304432 pps (305344 pkts in 1002995 usec)
531.111774 main_thread [1512] 304847 pps (305152 pkts in 1001000 usec)
532.115782 main_thread [1512] 305209 pps (306432 pkts in 1004008 usec)
533.122273 main_thread [1512] 305092 pps (307072 pkts in 1006491 usec)
534.129776 main_thread [1512] 304277 pps (306560 pkts in 1007504 usec)
535.130272 main_thread [1512] 305193 pps (305344 pkts in 1000496 usec)
536.135271 main_thread [1512] 304414 pps (305936 pkts in 1004999 usec)
537.136272 main_thread [1512] 305279 pps (305584 pkts in 1001000 usec)
538.137772 main_thread [1512] 305334 pps (305792 pkts in 1001500 usec)
539.138606 main_thread [1512] 304897 pps (305152 pkts in 1000835 usec)
540.140772 main_thread [1512] 305195 pps (305856 pkts in 1002165 usec)
541.148272 main_thread [1512] 304786 pps (307072 pkts in 1007501 usec)
542.149770 main_thread [1512] 304773 pps (305230 pkts in 1001498 usec)
543.153270 main_thread [1512] 305859 pps (306930 pkts in 1003500 usec)
544.154357 main_thread [1512] 305014 pps (305345 pkts in 1001086 usec)
545.156774 main_thread [1512] 304990 pps (305727 pkts in 1002418 usec)
^C546.157076 sigint_h [328] received control-C on thread 0x801806800
546.157273 main_thread [1512] 305512 pps (305664 pkts in 1000499 usec)
Received 13138095 packets, in 43.12 seconds.
Speed: 304.66 Kpps
chelsio# netstat -hd -I ncxl0
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop
ncxl0 1.5K <Link#4> 00:07:43:33:8d:c1 0 0 0 0 0 0 0
ncxl0 - 172.16.0.0 172.16.0.1 39M - - 0 - - -
chelsio#
chelsio# netstat -hd -I cxl0
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop
cxl0 1.5K <Link#3> 00:07:43:33:8d:c0 6.0G 1 612k 0 0 0 0
chelsio# netstat -hdwi 1 -l ncxl0
input (Total) output
packets errs idrops bytes packets errs bytes colls drops
15M 0 0 908M 8 0 1.1K 0 0
15M 0 0 908M 4 0 604 0 0
15M 0 0 908M 4 0 604 0 0
15M 0 0 908M 15 0 1.8K 0 0
15M 0 0 908M 10 0 1.2K 0 0
15M 0 0 908M 4 0 604 0 0
15M 0 0 908M 6 0 822 0 0
15M 0 0 908M 6 0 826 0 0
^C
chelsio# netstat -hdwi 1 -l cxl0
input (Total) output
packets errs idrops bytes packets errs bytes colls drops
15M 0 0 908M 8 0 1.0K 0 0
15M 0 0 908M 4 0 604 0 0
15M 0 0 908M 5 0 756 0 0
15M 0 0 908M 10 0 1.2K 0 0
15M 0 0 908M 4 0 604 0 0
15M 0 0 908M 4 0 604 0 0
^C
chelsio# sysctl -n dev.t5nex.0.misc.tp_err_stats
channel 0 channel 1 channel 2 channel 3
macInErrs: 0 0 0 0
hdrInErrs: 0 0 0 0
tcpInErrs: 0 0 0 0
tcp6InErrs: 0 0 0 0
tnlCongDrops: 612276 0 0 0
tnlTxDrops: 0 0 0 0
ofldVlanDrops: 0 0 0 0
ofldChanDrops: 0 0 0 0
ofldNoNeigh: 0
ofldCongDefer: 0
chelsio# sysctl dev.ncxl.0.stats
sysctl: unknown oid 'dev.ncxl.0.stats'
Exit 1
chelsio# sysctl dev.cxl.0.stats
dev.cxl.0.stats.rx_trunc3: 0
dev.cxl.0.stats.rx_trunc2: 0
dev.cxl.0.stats.rx_trunc1: 0
dev.cxl.0.stats.rx_trunc0: 0
dev.cxl.0.stats.rx_ovflow3: 0
dev.cxl.0.stats.rx_ovflow2: 0
dev.cxl.0.stats.rx_ovflow1: 0
dev.cxl.0.stats.rx_ovflow0: 0
dev.cxl.0.stats.rx_ppp7: 0
dev.cxl.0.stats.rx_ppp6: 0
dev.cxl.0.stats.rx_ppp5: 0
dev.cxl.0.stats.rx_ppp4: 0
dev.cxl.0.stats.rx_ppp3: 0
dev.cxl.0.stats.rx_ppp2: 0
dev.cxl.0.stats.rx_ppp1: 0
dev.cxl.0.stats.rx_ppp0: 0
dev.cxl.0.stats.rx_pause: 0
dev.cxl.0.stats.rx_frames_1519_max: 0
dev.cxl.0.stats.rx_frames_1024_1518: 0
dev.cxl.0.stats.rx_frames_512_1023: 0
dev.cxl.0.stats.rx_frames_256_511: 0
dev.cxl.0.stats.rx_frames_128_255: 0
dev.cxl.0.stats.rx_frames_65_127: 0
dev.cxl.0.stats.rx_frames_64: 8216274634
dev.cxl.0.stats.rx_runt: 0
dev.cxl.0.stats.rx_symbol_err: 0
dev.cxl.0.stats.rx_len_err: 0
dev.cxl.0.stats.rx_fcs_err: 1
dev.cxl.0.stats.rx_jabber: 0
dev.cxl.0.stats.rx_too_long: 0
dev.cxl.0.stats.rx_ucast_frames: 0
dev.cxl.0.stats.rx_mcast_frames: 0
dev.cxl.0.stats.rx_bcast_frames: 8216279461
dev.cxl.0.stats.rx_frames: 8216280197
dev.cxl.0.stats.rx_octets: 525841964992
dev.cxl.0.stats.tx_ppp7: 0
dev.cxl.0.stats.tx_ppp6: 0
dev.cxl.0.stats.tx_ppp5: 0
dev.cxl.0.stats.tx_ppp4: 0
dev.cxl.0.stats.tx_ppp3: 0
dev.cxl.0.stats.tx_ppp2: 0
dev.cxl.0.stats.tx_ppp1: 0
dev.cxl.0.stats.tx_ppp0: 0
dev.cxl.0.stats.tx_pause: 0
dev.cxl.0.stats.tx_drop: 0
dev.cxl.0.stats.tx_frames_1519_max: 0
dev.cxl.0.stats.tx_frames_1024_1518: 0
dev.cxl.0.stats.tx_frames_512_1023: 0
dev.cxl.0.stats.tx_frames_256_511: 0
dev.cxl.0.stats.tx_frames_128_255: 0
dev.cxl.0.stats.tx_frames_65_127: 0
dev.cxl.0.stats.tx_frames_64: 0
dev.cxl.0.stats.tx_error_frames: 0
dev.cxl.0.stats.tx_ucast_frames: 0
dev.cxl.0.stats.tx_mcast_frames: 0
dev.cxl.0.stats.tx_bcast_frames: 0
dev.cxl.0.stats.tx_frames: 0
dev.cxl.0.stats.tx_octets: 0
dev.cxl.0.stats.tx_parse_error: 0
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20160124024040.128A2A0128>