From owner-freebsd-infiniband@FreeBSD.ORG  Thu Apr 17 17:22:13 2014
Return-Path: <owner-freebsd-infiniband@FreeBSD.ORG>
Delivered-To: freebsd-infiniband@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id 0D93F9B8
 for <freebsd-infiniband@freebsd.org>; Thu, 17 Apr 2014 17:22:13 +0000 (UTC)
Received: from mail.tdx.com (mail.tdx.com [62.13.128.18])
 by mx1.freebsd.org (Postfix) with ESMTP id A771919F2
 for <freebsd-infiniband@freebsd.org>; Thu, 17 Apr 2014 17:22:12 +0000 (UTC)
Received: from study64.tdx.co.uk (study64.tdx.co.uk [62.13.130.231])
 (authenticated bits=0)
 by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id s3HHM9Yf017862
 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
 Thu, 17 Apr 2014 18:22:10 +0100 (BST)
Date: Thu, 17 Apr 2014 18:22:08 +0100
From: Karl Pielorz <kpielorz_lst@tdx.co.uk>
To: Jason Bacon <jwbacon@tds.net>,
 =?UTF-8?Q?Stig_Inge_Lea_Bj=C3=B8rnsen?= <stiginge@pvv.org>,
 freebsd-infiniband@freebsd.org
Subject: Re: Mellanox MT25418 performance IPOIB?
Message-ID: <2E4C44F3E209188F46C55C4F@study64.tdx.co.uk>
In-Reply-To: <534FD0BB.1070308@tds.net>
References: <DF61C9D7E8748BF2824DDFED@study64.tdx.co.uk>
 <lilt1l$15v$1@ger.gmane.org> <830453C43410411669F120CC@study64.tdx.co.uk>
 <534FD0BB.1070308@tds.net>
X-Mailer: Mulberry/4.0.8 (Mac OS X)
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
X-BeenThere: freebsd-infiniband@freebsd.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Infiniband on FreeBSD <freebsd-infiniband.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-infiniband>, 
 <mailto:freebsd-infiniband-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-infiniband/>
List-Post: <mailto:freebsd-infiniband@freebsd.org>
List-Help: <mailto:freebsd-infiniband-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband>, 
 <mailto:freebsd-infiniband-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 17 Apr 2014 17:22:13 -0000


--On 17 April 2014 08:01:47 -0500 Jason Bacon <jwbacon@tds.net> wrote:

> Hi Karl,
>
> What type of network are you running?  DDR?  QDR?  FDR?  Switch and HCA
> models?

Ok, still 'new' to IB - I've basically just got two MT25418 cards running 
back to back in two machines.

'ibportstate' seems to tell me I'm running LinkWidthActive 4X, 
LinkSpeedActive 5.0Gbps.

> FYI, with some tuning effort, I was able to get up to 600 megabytes/sec
> both ways over DDR IB from a CentOS NFS server with 12 SATA disks on a
> PERC H710, ext4, RAID 6.  Might be able to do a little better with
> FreeBSD and ZFS where IB is providing 7.5gb/sec.

I'm planning on using these cards for HAST between two boxes running ZFS 
(so they're just doing TCP/IP duties). Though at the moment I've not been 
overly impressed with HAST's performance (even running with 'none' as the 
secondary) - it seems to dull the IO performance of any disks it's using 
even when just reading.

I'm going to give it the benefit of doubt though and just build the boxes 
up, sling a bunch of drives in and see what the performance comes out as.

-Karl