Date: Thu, 15 May 2014 10:45:56 -0500 From: Jason Bacon <jwbacon@tds.net> To: "Janky Jay, III" <jankyj@unfs.us>, freebsd-infiniband@freebsd.org Subject: Re: FBSD to FBSD NFS Mounts over IB. Message-ID: <5374E134.5080809@tds.net> In-Reply-To: <5374D431.5020501@unfs.us> References: <5374D431.5020501@unfs.us>
next in thread | previous in thread | raw e-mail | index | archive | help
Are you using connected mode? I've seen severe performance issues with IPOIB in connected mode. In one direction, it outperformed CentOS 6.5, but in the other it was abysmal. Another list member recently reported that disabling connected mode made things much more stable, although you won't get the throughput that you would from a stable connected mode. A Mellanox engineer also told me that they're in the process of revamping the IPOIB layer, due out sometime later this year. As for documentation, the archives of this list are probably the best source right now. I keep my progress and tools posted here: http://acadix.biz/infiniband.php Regards, Jason On 5/15/14 9:50 AM, Janky Jay, III wrote: > Hello All, > > We're recently been poking around with IB support in FBSD because > we would really like to use ZFS for cluster storage. We're currently > testing between two FBSD (10.0-R-p2) systems and two Linux (CentOS 6.5 > 2.6.32) systems and we're running into something a little strange... > > I have set up one of the FBSD systems to run OpenSM and also be an > NFS server which all the systems seem to be able to mount over the IB > devices without any issue at all. Small reads and writes to and from > the NFS server to all the other nodes also seems to work without any > issue. However, if I try to dump large amounts of data using "dd" (in > order to test speeds and stability), the FBSD NFS client craps out > immediately. I just get the following message(s) over and over: > > newnfs server 10.11.1.1:/data: not responding > newnfs server 10.11.1.1:/data: not responding > newnfs server 10.11.1.1:/data: not responding > newnfs server 10.11.1.1:/data: not responding > newnfs server 10.11.1.1:/data: not responding > newnfs server 10.11.1.1:/data: not responding > > Strangely enough, though, the Linux nodes doesn't seem to have a > problem at all. I'm unsure if anyone else has seen this problem or if > it's even an IB issue at all (could be an NFS issue). There just > doesn't seem to be much (if any) documentation about NFS over IB on > FBSD any so I'm just curious if someone here may be able to point me > in the right direction. The output of "ibstat" for both FBSD nodes is > as follows: > > node1 (NFS Server) > CA 'mlx4_0' > CA type: MT26428 > Number of ports: 1 > Firmware version: 2.9.1000 > Hardware version: b0 > Node GUID: 0x002590ffff16a3cc > System image GUID: 0x002590ffff16a3cf > Port 1: > State: Active > Physical state: LinkUp > Rate: 40 > Base lid: 1 > LMC: 0 > SM lid: 1 > Capability mask: 0x0251086a > Port GUID: 0x002590ffff16a3cd > > node2 (NFS Client) > CA 'mlx4_0' > CA type: MT26428 > Number of ports: 1 > Firmware version: 2.9.1000 > Hardware version: b0 > Node GUID: 0x002590ffff16b3a0 > System image GUID: 0x002590ffff16b3a3 > Port 1: > State: Active > Physical state: LinkUp > Rate: 40 > Base lid: 3 > LMC: 0 > SM lid: 1 > Capability mask: 0x02510868 > Port GUID: 0x002590ffff16b3a1 > > If anyone has any ideas as to what I can try, it would be greatly > appreciated! > > Regards, > Janky Jay, III > > _______________________________________________ > freebsd-infiniband@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to > "freebsd-infiniband-unsubscribe@freebsd.org" -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Jason W. Bacon jwbacon@tds.net Circumstances don't make a man: They reveal him. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5374E134.5080809>