Date: Wed, 30 Nov 2016 22:58:41 +0100 From: Hans Petter Selasky <hps@selasky.org> To: =?UTF-8?B?Vmluw61jaXVzIEZlcnLDo28=?= <ferrao@vmio.com.br>, "freebsd-infiniband@freebsd.org" <freebsd-infiniband@freebsd.org> Subject: Re: Slow speed with Mellanox IS5023 Switch and Connect-X3 cards Message-ID: <fd59996f-1d35-c5bb-f7e9-36e52390105a@selasky.org> In-Reply-To: <AFE58C41-4206-491E-95BD-E9C2A6B60A81@vmio.com.br> References: <AFE58C41-4206-491E-95BD-E9C2A6B60A81@vmio.com.br>
next in thread | previous in thread | raw e-mail | index | archive | help
On 11/30/16 20:18, Vinícius Ferrão wrote: > Hello guys, > > I’m testing a server running FreeBSD 10.3-RELEASE and a Connect-X3 card. When communicating with other hosts in a point-to-point connections everything works as expected. But when I put the FreeBSD server on tem switching fabric the performances drops to a unusable state. In this setup we are using IPoIB with NFS on top of it. > > Initial benchmarks with dd returned this numbers: > > [root@node005 scratch]# dd if=/dev/zero of=./dd1.dd bs=128k count=100 > 100+0 records in > 100+0 records out > 13107200 bytes (13 MB) copied, 3.96714 s, 3.3 MB/s > > This happens only on the FreeBSD machine. If we test Linux to Linux speeds through the same switch we got a very reasonable performance: > > [root@node005 home]# dd if=/dev/zero of=./dd1.dd bs=128k count=100 > 100+0 records in > 100+0 records out > 13107200 bytes (13 MB) copied, 0.0463456 s, 283 MB/s > > And finally if we remove the switch from the setup, and link the FreeBSD machine directly with a CentOS 7.2 client the throughput is exactly what we expect: > > [root@node010 scratch]# dd if=/dev/zero of=./dd1.dd bs=128k count=1000 > 1000+0 records in > 1000+0 records out > 131072000 bytes (131 MB) copied, 0.640333 s, 205 MB/s > > This Mellanox Switch is an unmanaged box and isn’t really anything to consider. This message is a shot on the dark to see if someone have an ideia of what to, or at least start a debugging process of the issue. > > Thanks, > V. Hi, What are the link speeds negoitiated towards the switch? --HPS
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?fd59996f-1d35-c5bb-f7e9-36e52390105a>