From owner-freebsd-infiniband@FreeBSD.ORG  Thu Apr 17 09:10:43 2014
Return-Path: <owner-freebsd-infiniband@FreeBSD.ORG>
Delivered-To: freebsd-infiniband@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id A3773FE7
 for <freebsd-infiniband@freebsd.org>; Thu, 17 Apr 2014 09:10:43 +0000 (UTC)
Received: from mail.tdx.com (mail.tdx.com [62.13.128.18])
 by mx1.freebsd.org (Postfix) with ESMTP id 4A0A7105C
 for <freebsd-infiniband@freebsd.org>; Thu, 17 Apr 2014 09:10:42 +0000 (UTC)
Received: from study64.tdx.co.uk (study64.tdx.co.uk [62.13.130.231])
 (authenticated bits=0)
 by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id s3H99l3Q072150
 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
 Thu, 17 Apr 2014 10:09:47 +0100 (BST)
Date: Thu, 17 Apr 2014 10:09:46 +0100
From: Karl Pielorz <kpielorz_lst@tdx.co.uk>
To: =?UTF-8?Q?Stig_Inge_Lea_Bj=C3=B8rnsen?= <stiginge@pvv.org>,
 freebsd-infiniband@freebsd.org
Subject: Re: Mellanox MT25418 performance IPOIB?
Message-ID: <830453C43410411669F120CC@study64.tdx.co.uk>
In-Reply-To: <lilt1l$15v$1@ger.gmane.org>
References: <DF61C9D7E8748BF2824DDFED@study64.tdx.co.uk>
 <lilt1l$15v$1@ger.gmane.org>
X-Mailer: Mulberry/4.0.8 (Mac OS X)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
X-BeenThere: freebsd-infiniband@freebsd.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Infiniband on FreeBSD <freebsd-infiniband.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-infiniband>, 
 <mailto:freebsd-infiniband-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-infiniband/>
List-Post: <mailto:freebsd-infiniband@freebsd.org>
List-Help: <mailto:freebsd-infiniband-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband>, 
 <mailto:freebsd-infiniband-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 17 Apr 2014 09:10:43 -0000


--On 16 April 2014 14:28:37 +0200 Stig Inge Lea Bj=C3=B8rnsen=20
<stiginge@pvv.org> wrote:

> You could try to use Infiniband in datagram mode, recompile without the
> IPOIB_CM option.

Ok, same boxes - I just removed IPOIB_CM out of the kernel config and=20
recompiled / installed it.

iperf now consistently returns:

[  3] local 10.10.10.2 port 37715 connected with 10.10.10.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  8.71 GBytes  7.46 Gbits/sec

> The maximum MTU will then be 4092, but it might default to 2044. I run
> OpenSM on both machines and set the MTU explicitly:

Yes, mtu defaulted to 2044 - raising it to 4092 actually causes a loss of=20
.5-1Gbit with iperf (obviously not 'real world') - but at leas I can now=20
try and use the stuff with real world (now it's returning more than a few=20
hundred kilobytes a second!).

> The bandwidth reported by iperf is ~7.5 Gbits/sec in both directions.

Same as here then.

Thanks for your email!

Regards,

-Karl