Date: Tue, 27 Aug 2019 08:26:15 -0500 From: Jason Bacon <bacon4000@gmail.com> To: Justin Clift <justin@postgresql.org> Cc: Hans Petter Selasky <hps@selasky.org>, freebsd-infiniband@freebsd.org, owner-freebsd-infiniband@freebsd.org Subject: Re: Kernel modules Message-ID: <c7f8d0fd-1db5-a383-cd8e-724bab77afd0@gmail.com> In-Reply-To: <141d6ade245d08a4b29a0b49310c16bd@postgresql.org> References: <0eba9ec9-692f-7677-2b10-4e67a232821c@gmail.com> <f3f94452-155f-79f4-72d8-bf65760ae5b0@selasky.org> <598a58f0-89b8-d00d-5ed7-74dd7005950f@gmail.com> <73ce0738-4d63-2f25-2ff6-00f0092de136@selasky.org> <2090dd24-db43-b689-4289-f50bd70090ea@gmail.com> <6673df26-8bba-ebd3-b2c5-d7e9c97db557@gmail.com> <d82f3a60-6ad4-dba8-a15b-355a536a9a83@gmail.com> <bd42597e-2981-4667-468e-b008b9be290b@selasky.org> <2f4d9a14-4ff6-0d34-06f0-bbb4ac76c6bd@gmail.com> <5166ec29-876b-0bd3-8a84-8a222647e87a@gmail.com> <b6e6f8931f59fb2ecf985478ea4d77b7@postgresql.org> <236a3839-e880-ab17-146a-4521d1894813@gmail.com> <ea3b8d21-3ce0-17ea-1e04-a84ef7c81baa@gmail.com> <691d5884-6947-8044-ecf6-05ab97e9faca@gmail.com> <bb497012-81c9-f05a-6d1a-6061fa731348@selasky.org> <d4185139-f16e-51e4-222e-d2f6215b81c5@gmail.com> <141d6ade245d08a4b29a0b49310c16bd@postgresql.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2019-08-26 23:41, Justin Clift wrote: > On 2019-08-26 23:39, Jason Bacon wrote: >> On 2019-08-26 08:13, Hans Petter Selasky wrote: > <snip> >>> Mellanox found a bug in ipoib which can lead to similar sympthoms=20 >>> that you see. Can you try the attached patch? >>> >>> Thank you! > <snip> >> That's great to hear... >> >> Unfortunately, I no longer have admin access to a large cluster with >> Mellanox HCAs, as I just changed jobs. >> >> I did my best to thoroughly test FreeBSD IB before I left my old >> position.=C2=A0 This was the only outstanding issue with the FreeBSD f= ile >> server I was testing, so if this fix resolves it, I would say that >> FreeBSD is production-ready for IB clusters. > > That would be welcome news.=C2=A0 People still turn up in the FreeNAS F= orums > from time to time, looking for IB support.=C2=A0 It'd be nice to have s= table > IB drivers, suitable for adding to the FreeNAS image. :) > > + Justin I'll add that iperf throughput fell short of identical CentOS nodes by=20 something like 15%, but NFS nevertheless outperformed CentOS in some=20 aspects and averaged out about the same.=C2=A0 PowerEdge R720xd, RAID 6 o= n a=20 PERC controller with 12 2T SAS disks, using mrsas driver on FreeBSD.=C2=A0= I=20 ran ZFS on top of a hardware RAID, did not try RAIDZ*. Also note: The loads that triggered the issue (e.g. a canu de novo=20 assembly using very poor quality sequence data) also caused problems=20 with the CentOS file servers - the server hanging temporarily and=20 clients hanging indefinitely, even after load subsided.=C2=A0 I did not s= ee=20 the same client issues with the FreeBSD file server, just the buffer=20 space issue.=C2=A0 I got around the issue with the CentOS servers by=20 switching to NFS over RDMA. Intended to try a parallel filesystem at some point (e.g. gluster,=20 ceph), but wasn't able to find time before I left. --=20 Earth is a beta site.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?c7f8d0fd-1db5-a383-cd8e-724bab77afd0>