From owner-freebsd-infiniband@FreeBSD.ORG Thu May 15 20:34:10 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6785DE7 for ; Thu, 15 May 2014 20:34:10 +0000 (UTC) Received: from morbid.purplehat.org (morbid.purplehat.org [206.225.82.173]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47D0E20B5 for ; Thu, 15 May 2014 20:34:09 +0000 (UTC) Received: from localhost (morbid.purplehat.org [206.225.82.173]) by morbid.purplehat.org (Postfix) with ESMTP id 230DBD8CCB4 for ; Thu, 15 May 2014 13:34:08 -0700 (MST) Received: from morbid.purplehat.org ([206.225.82.173]) by localhost (morbid.purplehat.org [206.225.82.173]) (maiad, port 10024) with ESMTP id 86642-06 for ; Thu, 15 May 2014 13:34:03 -0700 (MST) Received: from [172.17.0.60] (fw1.aspsys.com [173.14.5.129]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: jankyj@unfs.us) by morbid.purplehat.org (Postfix) with ESMTPSA id 05F86D8CCB3 for ; Thu, 15 May 2014 13:34:02 -0700 (MST) Message-ID: <537524A2.1010901@unfs.us> Date: Thu, 15 May 2014 14:33:38 -0600 From: "Janky Jay, III" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: freebsd-infiniband@freebsd.org Subject: Re: FBSD to FBSD NFS Mounts over IB. References: <5374D431.5020501@unfs.us> <55BC554716A7EA5C54F5DD02@study64.tdx.co.uk> <5374F7EA.6060505@unfs.us> In-Reply-To: <5374F7EA.6060505@unfs.us> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 May 2014 20:34:10 -0000 On 05/15/2014 11:22 AM, Janky Jay, III wrote: > Hello Karl, > > On 05/15/2014 10:50 AM, Karl Pielorz wrote: >> >> >> --On 15 May 2014 08:50:25 -0600 "Janky Jay, III" wrote: >> >>> I have set up one of the FBSD systems to run OpenSM and also be an >>> NFS >>> server which all the systems seem to be able to mount over the IB >>> devices >>> without any issue at all. Small reads and writes to and from the NFS >>> server to all the other nodes also seems to work without any issue. >>> However, if I try to dump large amounts of data using "dd" (in order to >>> test speeds and stability), the FBSD NFS client craps out immediately. I >>> just get the following message(s) over and over: >>> >>> newnfs server 10.11.1.1:/data: not responding >>> newnfs server 10.11.1.1:/data: not responding >> >> Can both sides 'ping' each other when this happens? >> > > I just tested this while node2 was hanging with another NFS > transfer (just a "cp /home/file /data/file") and both nodes (1 and 2) > can ping each other without any issues. > Quick update: I rebuild the kernel without the IPOIB_CM to get rid of connected mode and everything seems to be working fine now. Hopefully this will turn out to be as stable as I need and we can use ZFS for cluster storage. Thanks again for the replies and help, Karl! Regards, Janky Jay, III