From owner-freebsd-net@freebsd.org  Tue Aug  2 19:31:40 2016
Return-Path: <owner-freebsd-net@freebsd.org>
Delivered-To: freebsd-net@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1C041BACB20
 for <freebsd-net@mailman.ysv.freebsd.org>;
 Tue,  2 Aug 2016 19:31:40 +0000 (UTC) (envelope-from hps@selasky.org)
Received: from mail.turbocat.net (heidi.turbocat.net [88.198.202.214])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id DAFC11D66
 for <freebsd-net@freebsd.org>; Tue,  2 Aug 2016 19:31:39 +0000 (UTC)
 (envelope-from hps@selasky.org)
Received: from laptop015.home.selasky.org (unknown [62.141.129.119])
 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
 (No client certificate requested)
 by mail.turbocat.net (Postfix) with ESMTPSA id DA6B31FE022;
 Tue,  2 Aug 2016 21:31:30 +0200 (CEST)
Subject: Re: Unstable local network throughput
To: Ben RUBSON <ben.rubson@gmail.com>, freebsd-net@freebsd.org
References: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com>
From: Hans Petter Selasky <hps@selasky.org>
Message-ID: <bed13ae3-0b8f-b1af-7418-7bf1b9fc74bc@selasky.org>
Date: Tue, 2 Aug 2016 21:35:48 +0200
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101
 Thunderbird/45.0
MIME-Version: 1.0
In-Reply-To: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 02 Aug 2016 19:31:40 -0000

On 08/02/16 20:43, Ben RUBSON wrote:
> Hello,
>
> I'm trying to reach the 40Gb/s max throughtput between 2 hosts running a ConnectX-3 Mellanox network adapter.
>
> FreeBSD 10.3 just installed, last updates performed.
> Network adapters running last firmwares / last drivers.
> No workload at all, just iPerf as the benchmark tool.
>

Hi,

The CX-3 driver doesn't bind the worker threads to specific CPU cores by 
default, so if your CPU has more than one so-called numa, you'll end up 
that the bottle-neck is the high-speed link between the CPU cores and 
not the card. A quick and dirty workaround is to "cpuset" iperf and the 
interrupt and taskqueue threads to specific CPU cores.

Are you using "options RSS" and "options PCBGROUP" in your kernel config?

Are you also testing CX-4 cards from Mellanox?

--HPS