Date: Mon, 29 Oct 2018 06:13:55 +0300 From: Andrew Vylegzhanin <avv314@gmail.com> To: freebsd-infiniband@freebsd.org, freebsd-fs@freebsd.org Subject: NFS + Infiniband problem Message-ID: <CA%2BBi_YiHoxFc3wsEPnMeBHWgW-nh6sXQCEgBTb=-nD6-XcjZ%2Bg@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hello everyone, I have a several FreeBSD machines connected via Infiniband netwok ( FDR switch Mellanox SW3036 + ConnectX-3 VPI cards ). One of them is a NAS-server with multiply ZFS pools. All kernels (11.2-RELEASE on clients and 12.0-BETA1 (11.2 also tried) on server) are with infiniband connected mode (option IPOIB_CM, option SDM) and world with OFED stack support. (WITH_OFED='yes'). File transfers via FTP or SSH between server and clients works almost flawless ( ~ 12 Gbit/s ). But when I try to copy in/out some significant data via NFS share mounted on clients, NFS i/o hangs at all or got extremely slow (couple kB/s) transfer speed after uncertain amount of copied data. For example, on the one node I can copy 1GB file, and after NFS hang on file with size 30 kb. Some details: [root@node4 ~]# mount_nfs -o wsize=30000 -o proto=tcp 10.0.2.1:/zdata2 /mnt [root@node4 ~]# dd if=/dev/zero of=/mnt/N1 bs=1m count=1024 Ctrl-T for "hang" dd load: 0.01 cmd: dd 1061 [bo_wwait] 70.95r 0.00u 0.00s 0% 2112k load: 0.01 cmd: dd 1061 [bo_wwait] 72.89r 0.00u 0.00s 0% 2112k for "slow" dd load: 0.00 cmd: dd 2254 [nfsaio] 224.18r 0.00u 0.13s 0% 3132k load: 0.00 cmd: dd 2254 [nfsaio] 225.94r 0.00u 0.13s 0% 3132k I've tried mount with different wsize option with same result. Any help would be greatly appreciated. -- Andrew
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2BBi_YiHoxFc3wsEPnMeBHWgW-nh6sXQCEgBTb=-nD6-XcjZ%2Bg>