Date: Tue, 2 Aug 2016 20:43:01 +0200 From: Ben RUBSON <ben.rubson@gmail.com> To: freebsd-net@freebsd.org Subject: Unstable local network throughput Message-ID: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com>
next in thread | raw e-mail | index | archive | help
Hello, I'm trying to reach the 40Gb/s max throughtput between 2 hosts running a = ConnectX-3 Mellanox network adapter. FreeBSD 10.3 just installed, last updates performed. Network adapters running last firmwares / last drivers. No workload at all, just iPerf as the benchmark tool. ### Step 1 : I never achieved to go beyond around 30Gb/s. I did the usual tuning (MTU, kern.ipc.maxsockbuf, = net.inet.tcp.sendbuf_max, net.inet.tcp.recvbuf_max...). I played with adapter interrupt moderation. I played with iPerf options (window / buffer size, number of = threads...). But it did not help. Results fluctuate, throughput is not sustained, and using 2 or more = iPerf threads did not help but degraded the results "quality". ### Step 2 : Let's start Linux on these 2 physical hosts. I only had to use jumbo frames in order to achieve the 40Gb/s max = throughtput... OK, network between the 2 hosts is not the root cause, and my hardware = can run these adapters up to their max throughput. Good point. ### Step 3 : Go back to FreeBSD on these physical hosts. Let's run this simple command to test FreeBSD itself : # iperf -c 127.0.0.1 -i 1 -t 60 Strangely enough, higher results are around 35GB/s. Even more strange, from one run to another, I do not get identical = results : sometimes 17Gb/s, sometimes 20, sometimes 30... Throughput can also suddenly drop down, then increase again... Power management in BIOS is totally disabled, as well as FreeBSD powerd, = so CPU frequency is not throttled. Another strange thing, increasing the number of iPerf threads (-P 2 for = example), does not improve the results at all. iPerf3 gave the same random results. ### Step 4 : Let's start Linux again on these 2 hosts. Let's run the same simple command : # iperf -c 127.0.0.1 -i 1 -t 60 Result : 45Gb/s. With 2 threads : 90Gb/s. With 4 threads : 180Gb/s. So here we have expected results, and they stay identical over the time. ### Step 5 : Does FreeBSD suffers when sending or when receiving ? Let's start one host with Linux, the other one with FreeBSD. Results : Linux --> FreeBSD : around 30GB/s. FreeBSD --> Linux : 40Gb/s. So sounds like FreeBSD suffers when receiving. ### Step 6 : FreeBSD 11-BETA3 gave the same random results. ### Questions : I think my tests show that there is something wrong with FreeBSD (tuning = ? something else ?). Do you have the same kind of random results on your hosts ? Could you help me trying to have sustained througput @step3, as we have = @step4 (I think this is what we should expect) ? There would then be no reason not to achieve max throughput through = Mellanox adapters themselves. Thank you very much ! Best regards, Ben
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3C0D892F-2BE8-4650-B9FC-93C8EE0443E1>