From owner-freebsd-infiniband@FreeBSD.ORG  Sat Jan 10 18:00:07 2015
Return-Path: <owner-freebsd-infiniband@FreeBSD.ORG>
Delivered-To: freebsd-infiniband@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id 0BF08A94
 for <freebsd-infiniband@freebsd.org>; Sat, 10 Jan 2015 18:00:07 +0000 (UTC)
Received: from mail.tdx.com (mail.tdx.com [62.13.128.18])
 by mx1.freebsd.org (Postfix) with ESMTP id CC115EDA
 for <freebsd-infiniband@freebsd.org>; Sat, 10 Jan 2015 18:00:06 +0000 (UTC)
Received: from [10.12.30.100] (vpn01-01.tdx.co.uk [62.13.130.213])
 (authenticated bits=0)
 by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id t0AHxvH6022182
 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
 for <freebsd-infiniband@freebsd.org>; Sat, 10 Jan 2015 17:59:59 GMT
Date: Sat, 10 Jan 2015 17:59:57 +0000
From: Karl Pielorz <kpielorz_lst@tdx.co.uk>
To: freebsd-infiniband@freebsd.org
Subject: FreeBSD 11 Infiniband performance / usable?
Message-ID: <E9EFD3AA27C919F9B8ADA6FB@A54336F45AB8EC4FED46DD7E>
X-Mailer: Mulberry/4.0.8 (Mac OS X)
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
X-BeenThere: freebsd-infiniband@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: Infiniband on FreeBSD <freebsd-infiniband.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-infiniband>, 
 <mailto:freebsd-infiniband-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-infiniband/>
List-Post: <mailto:freebsd-infiniband@freebsd.org>
List-Help: <mailto:freebsd-infiniband-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband>, 
 <mailto:freebsd-infiniband-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sat, 10 Jan 2015 18:00:07 -0000


Hi,

I've now got my MT25418's running 'back to back' on a couple of FreeBSD 11 
boxes [amd64 Haswell based] - but the performance seems rather poor using 
them for IPoIB? (iperf sites around 2Mbyte/sec on defaults - rising to 
98Mbyte/sec if I set an MTU of 4k) - default on the card was 64k.

Disabling IBOIB_CM in the kernel doesn't seem to make any difference (under 
10.0 this made a huge difference).

Also - the two boxes have lost site of each other now [I had this problem 
previously with 10.0 - one of the reasons why I was looking at 11].


What's the current state of IB support in FreeBSD 11? - I know a lot of 
work has been done on this - but is it usable yet?

The two machines both have MT25418's fitted running 'back to back' with one 
CX4 cable between them, and opensm running on them (after running the 
kernel sysctl's to switch the ports from 'eth' mode to 'ib').

Cheers,

-Karl