Date: Thu, 6 Dec 2018 05:21:04 +0300 From: Andrew Vylegzhanin <avv314@gmail.com> To: rmacklem@uoguelph.ca, freebsd-net <freebsd-net@freebsd.org> Cc: "Rodney W. Grimes" <freebsd-rwg@pdx.rh.cn85.dnsmgr.net>, freebsd-fs@freebsd.org, freebsd-infiniband@freebsd.org Subject: Re: NFS + Infiniband problem Message-ID: <CA%2BBi_Yi5f8aRDhc7%2BgzmnaTmFf7-zox0ge37Cv9yU5mdD7Af-w@mail.gmail.com> In-Reply-To: <CA%2BBi_YjyGYQpvuEiNVgaudD0PVMtRacZo7RVFCxG4ruP8VM23g@mail.gmail.com> References: <CA%2BBi_YiHoxFc3wsEPnMeBHWgW-nh6sXQCEgBTb=-nD6-XcjZ%2Bg@mail.gmail.com> <201810291506.w9TF6YAP057202@pdx.rh.CN85.dnsmgr.net> <YTOPR0101MB11622A9797376128D2FA182ADDF30@YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM> <CA%2BBi_YhHvpqkmP7WVd4wJj0u9rnbMZpum7FsKxZsu=%2B=8ze8-w@mail.gmail.com> <YTOPR0101MB116206BEB6259AA2836B16BCDDCD0@YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM> <CA%2BBi_YgpzBYyho_ex4p9nx0HHvvRb5kncKapUBEJQk88Et3BCQ@mail.gmail.com> <YTOPR0101MB11622A98B2358A0A83CD6927DDCE0@YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM> <CA%2BBi_YjyGYQpvuEiNVgaudD0PVMtRacZo7RVFCxG4ruP8VM23g@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi,
Back to the thread after month.
Some prehistory of this case:
>>>> I have a several FreeBSD machines connected via Infiniband netwok (
FDR switch Mellanox SW3036 + ConnectX-3 VPI cards ).
>>>> One of them is a NAS-server with multiply ZFS pools.
>>>>
>>>>All kernels (11.2-RELEASE on clients and 12.0-BETA1 (11.2 also tried)
on server) are with infiniband connected mode (option IPOIB_CM, option SDM)
and world with >>>> OFED stack support. (WITH_OFED=3D'yes').
>>>>
>>>>File transfers via FTP or SSH between server and clients works almost
flawless ( ~ 12 Gbit/s ).
>>>>But when I try to copy in/out some significant data via NFS share
mounted on clients, NFS i/o hangs at all or got extremely slow (couple
kB/s) transfer speed after uncertain amount of copied data. For example, on
the one node I can copy 1GB file, and after NFS hang on file with size 30 k
>=D1=87=D1=82, 1 =D0=BD=D0=BE=D1=8F=D0=B1. 2018 =D0=B3. =D0=B2 04:46, Andre=
w Vylegzhanin <avv314@gmail.com>:
>> =D1=87=D1=82, 1 =D0=BD=D0=BE=D1=8F=D0=B1. 2018 =D0=B3. =D0=B2 3:27, Rick=
Macklem <rmacklem@uoguelph.ca>:
>>
>>Also need to test setup with infniband set from connected mode to
datagram mode.
Tests for datagram mode IPoIB give disappointing results for NFS: just 20 -
40 MB/s for in average for NFS seq reads and even less for writes.
And inifniband interface report significant errors on:
# netstat -nI ib0 1
input ib0 output
packets errs idrops bytes packets errs bytes colls
11267 0 0 1172852 20471 652 42046744 0
15628 0 0 1626860 28387 994 58257440 0
16920 0 0 1761896 30832 1065 63196256 0
13566 0 0 1410424 24882 722 51205964 0
17942 0 0 1867312 32652 1114 67164484 0
9443 0 0 982104 17340 525 35610908 0
Similar transfer speed and errors were got for other protocols: ftp, scp .
I've tried to mount NFS share via UDP also, but no success.
With UDP, NFS operations are hangs with MTU errors on IB interface: kernel:
ib0: packet len 4096 (> 2044) too long to send, dropping.
So, in short summary, IPoIB actually inoperable in datagram mode.
Any ideas?
Regards,
Andrew
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2BBi_Yi5f8aRDhc7%2BgzmnaTmFf7-zox0ge37Cv9yU5mdD7Af-w>