From owner-freebsd-net@freebsd.org  Wed Aug  3 02:33:18 2016
Return-Path: <owner-freebsd-net@freebsd.org>
Delivered-To: freebsd-net@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id D9FA2BACF22
 for <freebsd-net@mailman.ysv.freebsd.org>;
 Wed,  3 Aug 2016 02:33:18 +0000 (UTC)
 (envelope-from eugen@grosbein.net)
Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247])
 (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
 (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified))
 by mx1.freebsd.org (Postfix) with ESMTPS id 740AB14C3
 for <freebsd-net@freebsd.org>; Wed,  3 Aug 2016 02:33:17 +0000 (UTC)
 (envelope-from eugen@grosbein.net)
Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221])
 by hz.grosbein.net (8.14.9/8.14.9) with ESMTP id u732X6hD025487
 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT);
 Wed, 3 Aug 2016 04:33:07 +0200 (CEST)
 (envelope-from eugen@grosbein.net)
X-Envelope-From: eugen@grosbein.net
X-Envelope-To: ben.rubson@gmail.com
Received: from [10.58.0.10] (dadvw [10.58.0.10])
 by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id u732X3kT063800
 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT);
 Wed, 3 Aug 2016 09:33:03 +0700 (KRAT)
 (envelope-from eugen@grosbein.net)
Subject: Re: Unstable local network throughput
To: Ben RUBSON <ben.rubson@gmail.com>, freebsd-net@freebsd.org
References: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com>
From: Eugene Grosbein <eugen@grosbein.net>
Message-ID: <57A157D9.3040707@grosbein.net>
Date: Wed, 3 Aug 2016 09:32:57 +0700
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101
 Thunderbird/38.7.2
MIME-Version: 1.0
In-Reply-To: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM
 autolearn=no version=3.3.2
X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1%
 *      [score: 0.0000] *  2.6 LOCAL_FROM From my domains
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on hz.grosbein.net
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 03 Aug 2016 02:33:18 -0000

03.08.2016 1:43, Ben RUBSON пишет:
> Hello,
>
> I'm trying to reach the 40Gb/s max throughtput between 2 hosts running a ConnectX-3 Mellanox network adapter.

If you have gateway_enable="YES" (sysctl net.inet.ip.forwarding=1)
then try to disable this forwarding setting and rerun your tests to compare results.