From owner-freebsd-infiniband@FreeBSD.ORG Wed Jan 7 14:24:06 2015 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0A31C60C for ; Wed, 7 Jan 2015 14:24:06 +0000 (UTC) Received: from mail.tdx.com (mail.tdx.com [62.13.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id C5FE164B0B for ; Wed, 7 Jan 2015 14:24:05 +0000 (UTC) Received: from [10.12.30.106] (vpn01-01.tdx.co.uk [62.13.130.213]) (authenticated bits=0) by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id t07EO3jP085699 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 7 Jan 2015 14:24:04 GMT Date: Wed, 07 Jan 2015 14:24:04 +0000 From: Karl Pielorz To: Oded Shanoon , freebsd-infiniband@freebsd.org Subject: RE: Infiniband under FreeBSD 11.0-CURRENT... Message-ID: <922BDD3D8725118FA5C28939@[10.12.30.106]> In-Reply-To: <640D4900BB006C2D80760288@[10.12.30.106]> References: <54AB853C.7070300@selasky.org> <9406DDD429A698CCFE8EA77B@Karls-Mac-mini.local> <54ACE6B1.1060705@selasky.org> <62BC1791A40C62A1836B5368@[10.12.30.106]> <640D4900BB006C2D80760288@[10.12.30.106]> X-Mailer: Mulberry/4.0.8 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2015 14:24:06 -0000 --On 07 January 2015 13:45 +0000 Karl Pielorz wrote: > Ok, I'll set them to ib - and check the relevant modules are loaded, Ok - sysctl'ing the ports to 'ib' - and opensm / ibstat now see them. >> Also, please verify the relevant modules are loaded: >> Kldstat -v | grep 'mlx4\|mlx4ib\|ibcore\|ipoib' This gives: 421 ipoib 420 ibcore 423 mlx4 422 mlx4ib Having configured the interfaces, and run the obligatory 'iperf' on them - the results aren't outstanding: " [ 4] local 10.90.90.2 port 5001 connected with 10.90.90.1 port 64249 [ ID] Interval Transfer Bandwidth [ 4] 0.0-12.2 sec 1.12 MBytes 771 Kbits/sec ... [ 3] local 10.90.90.1 port 64249 connected with 10.90.90.2 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.7 sec 1.12 MBytes 879 Kbits/sec " Under 10.x I know it was better to disable connected mode - is there a way this can be done under 11.x - or should I see better performance than that using CM? 'ibstat' on the system returns: " CA 'mlx4_0' CA type: MT25418 Number of ports: 2 Firmware version: 2.6.0 Hardware version: a0 Node GUID: 0x001a4bffff0c328c System image GUID: 0x001a4bffff0c328f Port 1: State: Active Physical state: LinkUp Rate: 20 Base lid: 1 LMC: 0 SM lid: 1 Capability mask: 0x0251086a Port GUID: 0x001a4bffff0c328d Port 2: State: Initializing Physical state: LinkUp Rate: 20 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x02510868 Port GUID: 0x001a4bffff0c328e " (Second port is 'Initialising' as I only have one instance of opensm running bound to the first port). Thanks, -Karl