Date: Wed, 31 Oct 2018 02:53:49 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: Andrew Vylegzhanin <avv314@gmail.com> Cc: "Rodney W. Grimes" <freebsd-rwg@pdx.rh.CN85.dnsmgr.net>, "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org>, "freebsd-infiniband@freebsd.org" <freebsd-infiniband@freebsd.org> Subject: Re: NFS + Infiniband problem Message-ID: <YTOPR0101MB116206BEB6259AA2836B16BCDDCD0@YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <CA%2BBi_YhHvpqkmP7WVd4wJj0u9rnbMZpum7FsKxZsu=%2B=8ze8-w@mail.gmail.com> References: <CA%2BBi_YiHoxFc3wsEPnMeBHWgW-nh6sXQCEgBTb=-nD6-XcjZ%2Bg@mail.gmail.com> <201810291506.w9TF6YAP057202@pdx.rh.CN85.dnsmgr.net> <YTOPR0101MB11622A9797376128D2FA182ADDF30@YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM>, <CA%2BBi_YhHvpqkmP7WVd4wJj0u9rnbMZpum7FsKxZsu=%2B=8ze8-w@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Andrew Vylegzhanin wrote: >> >> Some details: >> >> [root@node4 ~]# mount_nfs -o wsize=3D30000 -o proto=3Dtcp 10.0.2.1:/z= data2 /mnt >> > ^^^^^^^^^^^^ > > >Again after some tests. >I've tried 4096,8192,16384 wsize/rsize. Only 4096 value give some measurab= le >result and it's extremely slow ~ 10-16MB/s for writing (depend on numbe= r of >threads), 10-12 MB/s for reading. With other values NFS hangs (or alm= ost hangs - >couple kB/s in average) > >Changing sysctl net.inet.tcp.tso=3D0 (w/o reboot) on both sides had no ef= fect. > >AFAIK, infiniband interface has no option for TSO,LRO: >ib0: flags=3D8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 65520 > options=3D80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE> > lladdr 80.0.2.8.fe.80.0.0.0.0.0.0.e4.1d.2d.3.0.50.df.51 > inet 10.0.2.1 netmask 0xffffff00 broadcast 10.0.2.255 > nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > > >BTW, hers is my sysctl.conf file with optimisation for congestion control = and tcp >buffers on 10/40 Gbit/s links (the server had 40 Gbit/s Intel ixl = ethernet also): > >kern.ipc.maxsockbuf=3D16777216 > >net.inet.tcp.sendbuf_max=3D16777216 > >net.inet.tcp.recvbuf_max=3D16777216 > >net.inet.tcp.sendbuf_auto=3D1 > >net.inet.tcp.recvbuf_auto=3D1 > >net.inet.tcp.sendbuf_inc=3D16384 > >net.inet.tcp.recvbuf_inc=3D524288 > >net.inet.tcp.cc.algorithm=3Dhtcp Well, I'm not familiar with the current TCP stack (and, as noted before, I = know nothing about InfiniBand). All I can suggest is testing with the default co= ngestion control algorithm. (I always test with the default, which appears to be new= reno.) NFS traffic looks very different than a typical use of TCP. Lots of small T= CP segments in both directions interspersed with some larger ones (the write requests or read replies). rick
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YTOPR0101MB116206BEB6259AA2836B16BCDDCD0>