Date: Tue, 7 Mar 2017 20:46:48 -0500 From: Mike Tancsa <mike@sentex.net> To: Navdeep Parhar <nparhar@gmail.com> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: Chelsio netmap support ? (RELENG_11) Message-ID: <901224a7-f3ed-84a4-5402-3542cd0e865b@sentex.net> In-Reply-To: <CAPFoGT-oxsVCNAm05porR0cJi7o_4GTrw-O8V9zZTpRu9_q82Q@mail.gmail.com> References: <93e10699-ecc7-c062-bf85-a26de0de9633@sentex.net> <CAPFoGT9u4Ctw0cMtXeEC=6xP--sSW8rEbdSwGwcXZOpRtYp=EQ@mail.gmail.com> <CAPFoGT8iQg9on9xi%2ByNkZJF-knmHE7b58G1u18SvJyVAJa7MPA@mail.gmail.com> <CAPFoGT_CYSPLrB6h742U%2BTS5qqKeLdrF2tnPGGw%2BiyBwPjmDWw@mail.gmail.com> <c7f36da2-ce4d-4377-c3d5-509c9ba692a5@sentex.net> <CAPFoGT-oxsVCNAm05porR0cJi7o_4GTrw-O8V9zZTpRu9_q82Q@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 3/7/2017 5:07 PM, Navdeep Parhar wrote:
> On Tue, Mar 7, 2017 at 1:53 PM, Mike Tancsa <mike@sentex.net> wrote:
> ...
>>
>> Using netsend, I cant seem to blast through a single flow of packets
>> greater than 800Kpps without packet loss. Can you point me to any
>> performance tweaks for forwarding / routing ?
>>
>> I have 3 boxes, with one in the middle
>>
>> (netsend box) <----> (router with chelsio nics) <---> (netreceive box)
>
> How is the router configured -- is this something netmap based?
> Please provide more details of the configuration. What kind of
> CPU/chipset is it?
Hi,
Xeon E3-1225 v5 @ 3.30GHz
Supermicro X11SSL-F Intel Skylake-S/Skylake-H/Greenlow
PCI-e 3.0 slots
t5iov0@pci0:2:0:0: class=0x020000 card=0x00001425 chip=0x50071425
rev=0x00 hdr=0x00
vendor = 'Chelsio Communications Inc'
device = 'T520-SO Unified Wire Ethernet Controller'
class = network
subclass = ethernet
bar [10] = type Memory, range 64, base 0xdd800000, size 524288,
enabled
bar [18] = type Memory, range 64, base 0xdd780000, size 524288,
enabled
bar [20] = type Memory, range 64, base 0xdd88c000, size 8192, enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 8 messages, 64 bit, vector masks
cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR NS
link x8(x8) speed 8.0(8.0)
cap 11[b0] = MSI-X supports 34 messages
Table in map 0x20[0x0], PBA in map 0x20[0x1000]
cap 03[d0] = VPD
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0002[140] = VC 1 max VC1
ecap 0003[170] = Serial 1 0000000000000000
ecap 000e[190] = ARI 1
ecap 0019[1a0] = PCIe Sec 1 lane errors 0
ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI
disabled
0 VFs configured out of 16 supported
First VF RID Offset 0x0008, VF RID Stride 0x0004
VF Device ID 0x5807
Page Sizes: 4096 (enabled), 8192, 65536, 262144,
1048576, 4194304
Two 10G interfaces on a switch vcxl0 (192.168.1.1) vcxl1 (10.151.10.1)
192.168.1.2 does the netsend to 10.151.10.3 so it goes through the
router in question. Router in question is RELENG11
dev.cxl.0.pause_settings=0
dev.cxl.1.pause_settings=0
loader.conf
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
zfs_load="YES"
#t4fw_cfg_load="YES"
t5fw_cfg_load="YES"
#t6fw_cfg_load="YES"
if_cxgbe_load="YES"
hw.cxgbe.num_vis=2
comconsole_speed="115200" # Set the current serial console speed
console="comconsole,vidconsole" # A comma separated list of
console(s)
comconsole_port="0x2F8"
> Do you see any PAUSE frames out of the T5? "sysctl dev.cxl dev.vcxl |
> grep _pause" Is any CPU core pegged at 100% during the test?
dev.cxl.1.stats.rx_pause: 0
dev.cxl.1.stats.tx_pause: 0
dev.cxl.1.pause_settings: 0
dev.cxl.0.stats.rx_pause: 0
dev.cxl.0.stats.tx_pause: 0
dev.cxl.0.pause_settings: 0
while forwarding, top shows the system 25% in interrupt
# dmesg | grep netm
netmap: loaded module
vcxl0: netmap queues/slots: TX 2/1023, RX 2/1024
vcxl0: 1 txq, 1 rxq (NIC); 1 txq, 1 rxq (TOE); 2 txq, 2 rxq (netmap)
vcxl1: netmap queues/slots: TX 2/1023, RX 2/1024
vcxl1: 1 txq, 1 rxq (NIC); 1 txq, 1 rxq (TOE); 2 txq, 2 rxq (netmap)
igb0: netmap queues/slots: TX 4/1024, RX 4/1024
igb1: netmap queues/slots: TX 4/1024, RX 4/1024
It maxes out at about 800Kpps with and without netmap. Is there a way
to increase the queues for the Chelsio nic, like the onboard igb ?
top -nCHSIzs1
last pid: 2517; load averages: 1.15, 0.45, 0.27 up 0+04:26:13
20:44:09
393 processes: 8 running, 346 sleeping, 39 waiting
Mem: 10M Active, 37M Inact, 360M Wired, 15G Free
ARC: 67M Total, 691K MFU, 64M MRU, 16K Anon, 430K Header, 1758K Other
Swap: 32G Total, 32G Free
PID USERNAME PRI NICE SIZE RES STATE C TIME CPU COMMAND
11 root -92 - 0K 672K CPU0 0 5:45 80.08%
intr{irq272: t5nex0:0b0}
---Mike
--
-------------------
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, mike@sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada http://www.tancsa.com/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?901224a7-f3ed-84a4-5402-3542cd0e865b>