Date: Tue, 30 Oct 2018 13:17:07 -0500 From: Jason Bacon <bacon4000@gmail.com> To: freebsd-infiniband@freebsd.org Subject: Re: NFS + Infiniband problem Message-ID: <7448666e-6028-12e7-aa74-4fd162e13dc7@gmail.com> In-Reply-To: <CA%2BBi_YhHvpqkmP7WVd4wJj0u9rnbMZpum7FsKxZsu=%2B=8ze8-w@mail.gmail.com> References: <CA%2BBi_YiHoxFc3wsEPnMeBHWgW-nh6sXQCEgBTb=-nD6-XcjZ%2Bg@mail.gmail.com> <201810291506.w9TF6YAP057202@pdx.rh.CN85.dnsmgr.net> <YTOPR0101MB11622A9797376128D2FA182ADDF30@YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM> <CA%2BBi_YhHvpqkmP7WVd4wJj0u9rnbMZpum7FsKxZsu=%2B=8ze8-w@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 10/29/18 11:57 PM, Andrew Vylegzhanin wrote: >>>> Some details: >>>> [root@node4 ~]# mount_nfs -o wsize=30000 -o proto=tcp 10.0.2.1:/zdata2 > /mnt >>> ^^^^^^^^^^^^ >>> I am not sure what the interaction between page sizes, TSO needs, >>> buffer needs and all that are but I always use a power of 2 wsize >>> and rsize. > Again after some tests. > I've tried 4096,8192,16384 wsize/rsize. Only 4096 value give some > measurable result and it's extremely slow ~ 10-16MB/s for writing (depend > on number of threads), 10-12 MB/s for reading. With other values NFS hangs > (or almost hangs - couple kB/s in average) > > Changing sysctl net.inet.tcp.tso=0 (w/o reboot) on both sides had no > effect. > > AFAIK, infiniband interface has no option for TSO,LRO: > ib0: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 65520 > options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE> > lladdr 80.0.2.8.fe.80.0.0.0.0.0.0.e4.1d.2d.3.0.50.df.51 > inet 10.0.2.1 netmask 0xffffff00 broadcast 10.0.2.255 > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > > > BTW, hers is my sysctl.conf file with optimisation for congestion control > and tcp buffers on 10/40 Gbit/s links (the server had 40 Gbit/s Intel ixl > ethernet also): > > kern.ipc.maxsockbuf=16777216 > > net.inet.tcp.sendbuf_max=16777216 > > net.inet.tcp.recvbuf_max=16777216 > > net.inet.tcp.sendbuf_auto=1 > > net.inet.tcp.recvbuf_auto=1 > > net.inet.tcp.sendbuf_inc=16384 > > net.inet.tcp.recvbuf_inc=524288 > > net.inet.tcp.cc.algorithm=htcp > > >>> You might try that. And as Rick suggested, turn of >>> TSO, if you can. Is infiniband using RDMA to do this, if so then >>> the page size stuff is probably very important, use multiples of >>> 4096 only. >> RDMA is not supported by the FreeBSD NFS client. There is a way to use > RDMA >> on a separate connection with NFSv4.1 or later, but I've never written > code >> for that. (Not practical to try to implement without access to hardware > that >> does it.) >> > Hope I could help with testing. > > -- > Andrew > >> rick >> [performance stuff snipped] > _______________________________________________ > freebsd-infiniband@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.org" Did you try NFS over a standard gigabit interface just to rule out any issues that are independent of IB? It's odd that other protocols seem to work well. Jason -- Earth is a beta site.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7448666e-6028-12e7-aa74-4fd162e13dc7>