From owner-freebsd-net@freebsd.org Tue Aug 2 18:43:05 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 61B66BACA04 for ; Tue, 2 Aug 2016 18:43:05 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x242.google.com (mail-wm0-x242.google.com [IPv6:2a00:1450:400c:c09::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EC2E6117C for ; Tue, 2 Aug 2016 18:43:04 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x242.google.com with SMTP id x83so32328128wma.3 for ; Tue, 02 Aug 2016 11:43:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-transfer-encoding:subject:message-id:date:to :mime-version; bh=tKUYGJqy+sEEV+BFGH+3WzdR2s5vrMd3D7d/YVqWfHA=; b=m5uW/DE/WNMc9i/aVGfrP67bF/ci1/HeJ1DrywpAoW7Fted7i6veR5b/1JpvJ1394u ltnoHYjlMgtkBlY2afwF5f5c6dcGhTLpl4ELGueDVA6MLjyEvnFDmobGTn4L1kLTGdMM eQRfue36CgwvF/zQz8QSw1Ii/LQtBQ2HMvEuN6LTFLmRdgkfxDSj+24uPf4qrFh+anwW FEc2v0YmMcW9Z9y5xGagm3lbXdpklOCa1F5ViIhYEN3KlW8QpWSvfz8Cb2RPT/6Us3ji YO8oKEmzH6hMfY8r1Hse2nI5ezpbpTwOdFCLq21EefRhbUSbJWTHGRFELvuAq3VgSsrq /JzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-transfer-encoding:subject :message-id:date:to:mime-version; bh=tKUYGJqy+sEEV+BFGH+3WzdR2s5vrMd3D7d/YVqWfHA=; b=OPwfHGoWrSqqkSDMrE/ddYk2o24sQ/hgbqLgAXnih5TKTzVrc0olHKhhR9GyCGTo1z ONmymgGt+0B/4k3g+Cvx7G8H3aWEDWXfqcWwyE8zBHkZJ95dK1Azx+8wAU2w9yf2SnAB WsFA6nryInlBCGJOZM2YP6I6X6DBPoYsm0gSZ3X9v1i6aDfZ/ggs4KQO9NMq0bvoVntg 7BIAhQw1aWRiaQ4BWNFOq+8QU8WU1EqF3hIvpZA5rFb0yoc/EszEGc8eKzNMGT4l2snd vrhVWj5G2Qvta5NdLIk9cdCQQfVOSEaHWWr3iUNuGmGi/IJKBPiO1b1vjpghy76Fs0jL YlHw== X-Gm-Message-State: AEkoouu+H+DPHfEDxxy3RTHoTVx6sMyp6ZJ1mGPyAy9nU+0aAs7UL9WJsirPpGtGEnjTyw== X-Received: by 10.28.229.1 with SMTP id c1mr67186225wmh.0.1470163382815; Tue, 02 Aug 2016 11:43:02 -0700 (PDT) Received: from macbook-air-de-benjamin-1.home (LFbn-1-7077-85.w90-116.abo.wanadoo.fr. [90.116.246.85]) by smtp.gmail.com with ESMTPSA id x203sm23274966wmg.0.2016.08.02.11.43.01 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 02 Aug 2016 11:43:02 -0700 (PDT) From: Ben RUBSON Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: Unstable local network throughput Message-Id: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> Date: Tue, 2 Aug 2016 20:43:01 +0200 To: freebsd-net@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 18:43:05 -0000 Hello, I'm trying to reach the 40Gb/s max throughtput between 2 hosts running a = ConnectX-3 Mellanox network adapter. FreeBSD 10.3 just installed, last updates performed. Network adapters running last firmwares / last drivers. No workload at all, just iPerf as the benchmark tool. ### Step 1 : I never achieved to go beyond around 30Gb/s. I did the usual tuning (MTU, kern.ipc.maxsockbuf, = net.inet.tcp.sendbuf_max, net.inet.tcp.recvbuf_max...). I played with adapter interrupt moderation. I played with iPerf options (window / buffer size, number of = threads...). But it did not help. Results fluctuate, throughput is not sustained, and using 2 or more = iPerf threads did not help but degraded the results "quality". ### Step 2 : Let's start Linux on these 2 physical hosts. I only had to use jumbo frames in order to achieve the 40Gb/s max = throughtput... OK, network between the 2 hosts is not the root cause, and my hardware = can run these adapters up to their max throughput. Good point. ### Step 3 : Go back to FreeBSD on these physical hosts. Let's run this simple command to test FreeBSD itself : # iperf -c 127.0.0.1 -i 1 -t 60 Strangely enough, higher results are around 35GB/s. Even more strange, from one run to another, I do not get identical = results : sometimes 17Gb/s, sometimes 20, sometimes 30... Throughput can also suddenly drop down, then increase again... Power management in BIOS is totally disabled, as well as FreeBSD powerd, = so CPU frequency is not throttled. Another strange thing, increasing the number of iPerf threads (-P 2 for = example), does not improve the results at all. iPerf3 gave the same random results. ### Step 4 : Let's start Linux again on these 2 hosts. Let's run the same simple command : # iperf -c 127.0.0.1 -i 1 -t 60 Result : 45Gb/s. With 2 threads : 90Gb/s. With 4 threads : 180Gb/s. So here we have expected results, and they stay identical over the time. ### Step 5 : Does FreeBSD suffers when sending or when receiving ? Let's start one host with Linux, the other one with FreeBSD. Results : Linux --> FreeBSD : around 30GB/s. FreeBSD --> Linux : 40Gb/s. So sounds like FreeBSD suffers when receiving. ### Step 6 : FreeBSD 11-BETA3 gave the same random results. ### Questions : I think my tests show that there is something wrong with FreeBSD (tuning = ? something else ?). Do you have the same kind of random results on your hosts ? Could you help me trying to have sustained througput @step3, as we have = @step4 (I think this is what we should expect) ? There would then be no reason not to achieve max throughput through = Mellanox adapters themselves. Thank you very much ! Best regards, Ben