Date: Mon, 13 Jun 2022 22:42:59 +0200 From: Hans Petter Selasky <hps@selasky.org> To: mike.jakubik@swiftsmsgateway.com, freebsd-net <freebsd-net@FreeBSD.org> Subject: Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5) Message-ID: <63396d47-3d0b-fd83-7b2e-ae5c02eeae2e@selasky.org> In-Reply-To: <1815e506878.cf301a5a1195924.6506017618978817828@swiftsmsgateway.com> References: <1815e506878.cf301a5a1195924.6506017618978817828@swiftsmsgateway.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 6/13/22 20:25, Mike Jakubik wrote: > Hello, > > I have two new servers with a Mellnox ConnectX-6 card linked at 25Gb/s, however, I am unable to get much more than 6Gb/s when testing with iperf3. > > > > The servers are Lenovo SR665 (2 x AMD EPYC 7443 24-Core Processor, 256 GB RAM, Mellanox ConnectX-6 Lx 10/25GbE SFP28 2-port OCP Ethernet Adapter) > > > > They are connected to a Dell N3224PX-ON switch. Both servers are idle and not in use, with a fresh install of stable/13-ebea872f8, nothing running on them except ssh, sendmail, etc. > > > > When i test with iperf3 I am unable to get a higher avg than about 6Gb/s. I have tried just about every knob listed in https://calomel.org/freebsd_network_tuning.html with little impact on the performance. The network cards have HW LRO enabled as per the driver documentation (though this only seems to lower IRQ usage with no impact on actual throughput). > > > > The same exact servers tested on Linux (fedora 34) produced nearly 3x the performance (see attached screenshots), i was able to get a steady 14.6Gb/s rate with nearly 0 retries shown in iperf, the performance on FreeBSD seems to avg at around 6Gbs but it is very sporadic during the iperf run. > > > > I have run out of ideas, any suggestions are welcome. Considering Netflix uses very similar HW and they push 400 Gb/s tells me there is something really wrong here or Netflix isnt sharing all their secret sauce. > > Some ideas: Try to disable "rxpause,txpause" when setting the media. Keep HW LRO off for now, it doesn't work for large number of connections. What is the CPU usage during test? Is iperf3 running on a CPU-core which has direct access to the NIC's numa domain? Is the NIC installed in the "correct" PCI high-performance slot? There are some sysctl knobs which may tell where the problem is, if it's PCI backpressure or something else. sysctl -a | grep diag_pci_enable sysctl -a | grep diag_general_enable Set these two to 1, then run some traffic and dump all mce sysctls: sysctl -a | grep mce > dump.txt --HPS
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?63396d47-3d0b-fd83-7b2e-ae5c02eeae2e>