Date: Wed, 30 Nov 2016 19:18:16 +0000 From: =?utf-8?B?Vmluw61jaXVzIEZlcnLDo28=?= <ferrao@vmio.com.br> To: "freebsd-infiniband@freebsd.org" <freebsd-infiniband@freebsd.org> Subject: Slow speed with Mellanox IS5023 Switch and Connect-X3 cards Message-ID: <AFE58C41-4206-491E-95BD-E9C2A6B60A81@vmio.com.br>
next in thread | raw e-mail | index | archive | help
Hello guys, I’m testing a server running FreeBSD 10.3-RELEASE and a Connect-X3 card. When communicating with other hosts in a point-to-point connections everything works as expected. But when I put the FreeBSD server on tem switching fabric the performances drops to a unusable state. In this setup we are using IPoIB with NFS on top of it. Initial benchmarks with dd returned this numbers: [root@node005 scratch]# dd if=/dev/zero of=./dd1.dd bs=128k count=100 100+0 records in 100+0 records out 13107200 bytes (13 MB) copied, 3.96714 s, 3.3 MB/s This happens only on the FreeBSD machine. If we test Linux to Linux speeds through the same switch we got a very reasonable performance: [root@node005 home]# dd if=/dev/zero of=./dd1.dd bs=128k count=100 100+0 records in 100+0 records out 13107200 bytes (13 MB) copied, 0.0463456 s, 283 MB/s And finally if we remove the switch from the setup, and link the FreeBSD machine directly with a CentOS 7.2 client the throughput is exactly what we expect: [root@node010 scratch]# dd if=/dev/zero of=./dd1.dd bs=128k count=1000 1000+0 records in 1000+0 records out 131072000 bytes (131 MB) copied, 0.640333 s, 205 MB/s This Mellanox Switch is an unmanaged box and isn’t really anything to consider. This message is a shot on the dark to see if someone have an ideia of what to, or at least start a debugging process of the issue. Thanks, V.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?AFE58C41-4206-491E-95BD-E9C2A6B60A81>