From owner-freebsd-infiniband@FreeBSD.ORG Thu May 15 15:00:25 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 38676C5F for ; Thu, 15 May 2014 15:00:25 +0000 (UTC) Received: from morbid.purplehat.org (morbid.purplehat.org [206.225.82.173]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1AA00210E for ; Thu, 15 May 2014 15:00:24 +0000 (UTC) Received: from localhost (morbid.purplehat.org [206.225.82.173]) by morbid.purplehat.org (Postfix) with ESMTP id 3A355D8CCE1 for ; Thu, 15 May 2014 07:50:50 -0700 (MST) Received: from morbid.purplehat.org ([206.225.82.173]) by localhost (morbid.purplehat.org [206.225.82.173]) (maiad, port 10024) with ESMTP id 82615-08 for ; Thu, 15 May 2014 07:50:49 -0700 (MST) Received: from [172.17.0.60] (fw1.aspsys.com [173.14.5.129]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: jankyj@unfs.us) by morbid.purplehat.org (Postfix) with ESMTPSA id DAE97D8CCE0 for ; Thu, 15 May 2014 07:50:48 -0700 (MST) Message-ID: <5374D431.5020501@unfs.us> Date: Thu, 15 May 2014 08:50:25 -0600 From: "Janky Jay, III" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: freebsd-infiniband@freebsd.org Subject: FBSD to FBSD NFS Mounts over IB. Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 May 2014 15:00:25 -0000 Hello All, We're recently been poking around with IB support in FBSD because we would really like to use ZFS for cluster storage. We're currently testing between two FBSD (10.0-R-p2) systems and two Linux (CentOS 6.5 2.6.32) systems and we're running into something a little strange... I have set up one of the FBSD systems to run OpenSM and also be an NFS server which all the systems seem to be able to mount over the IB devices without any issue at all. Small reads and writes to and from the NFS server to all the other nodes also seems to work without any issue. However, if I try to dump large amounts of data using "dd" (in order to test speeds and stability), the FBSD NFS client craps out immediately. I just get the following message(s) over and over: newnfs server 10.11.1.1:/data: not responding newnfs server 10.11.1.1:/data: not responding newnfs server 10.11.1.1:/data: not responding newnfs server 10.11.1.1:/data: not responding newnfs server 10.11.1.1:/data: not responding newnfs server 10.11.1.1:/data: not responding Strangely enough, though, the Linux nodes doesn't seem to have a problem at all. I'm unsure if anyone else has seen this problem or if it's even an IB issue at all (could be an NFS issue). There just doesn't seem to be much (if any) documentation about NFS over IB on FBSD any so I'm just curious if someone here may be able to point me in the right direction. The output of "ibstat" for both FBSD nodes is as follows: node1 (NFS Server) CA 'mlx4_0' CA type: MT26428 Number of ports: 1 Firmware version: 2.9.1000 Hardware version: b0 Node GUID: 0x002590ffff16a3cc System image GUID: 0x002590ffff16a3cf Port 1: State: Active Physical state: LinkUp Rate: 40 Base lid: 1 LMC: 0 SM lid: 1 Capability mask: 0x0251086a Port GUID: 0x002590ffff16a3cd node2 (NFS Client) CA 'mlx4_0' CA type: MT26428 Number of ports: 1 Firmware version: 2.9.1000 Hardware version: b0 Node GUID: 0x002590ffff16b3a0 System image GUID: 0x002590ffff16b3a3 Port 1: State: Active Physical state: LinkUp Rate: 40 Base lid: 3 LMC: 0 SM lid: 1 Capability mask: 0x02510868 Port GUID: 0x002590ffff16b3a1 If anyone has any ideas as to what I can try, it would be greatly appreciated! Regards, Janky Jay, III