Date: Wed, 31 Oct 2018 02:53:49 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: Andrew Vylegzhanin <avv314@gmail.com> Cc: "Rodney W. Grimes" <freebsd-rwg@pdx.rh.CN85.dnsmgr.net>, "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org>, "freebsd-infiniband@freebsd.org" <freebsd-infiniband@freebsd.org> Subject: Re: NFS + Infiniband problem Message-ID: <YTOPR0101MB116206BEB6259AA2836B16BCDDCD0@YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <CA%2BBi_YhHvpqkmP7WVd4wJj0u9rnbMZpum7FsKxZsu=%2B=8ze8-w@mail.gmail.com> References: <CA%2BBi_YiHoxFc3wsEPnMeBHWgW-nh6sXQCEgBTb=-nD6-XcjZ%2Bg@mail.gmail.com> <201810291506.w9TF6YAP057202@pdx.rh.CN85.dnsmgr.net> <YTOPR0101MB11622A9797376128D2FA182ADDF30@YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM>, <CA%2BBi_YhHvpqkmP7WVd4wJj0u9rnbMZpum7FsKxZsu=%2B=8ze8-w@mail.gmail.com>
index | next in thread | previous in thread | raw e-mail
Andrew Vylegzhanin wrote: >> >> Some details: >> >> [root@node4 ~]# mount_nfs -o wsize=30000 -o proto=tcp 10.0.2.1:/zdata2 /mnt >> > ^^^^^^^^^^^^ > > >Again after some tests. >I've tried 4096,8192,16384 wsize/rsize. Only 4096 value give some measurable >result and it's extremely slow ~ 10-16MB/s for writing (depend on number of >threads), 10-12 MB/s for reading. With other values NFS hangs (or almost hangs - >couple kB/s in average) > >Changing sysctl net.inet.tcp.tso=0 (w/o reboot) on both sides had no effect. > >AFAIK, infiniband interface has no option for TSO,LRO: >ib0: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 65520 > options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE> > lladdr 80.0.2.8.fe.80.0.0.0.0.0.0.e4.1d.2d.3.0.50.df.51 > inet 10.0.2.1 netmask 0xffffff00 broadcast 10.0.2.255 > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > > >BTW, hers is my sysctl.conf file with optimisation for congestion control and tcp >buffers on 10/40 Gbit/s links (the server had 40 Gbit/s Intel ixl ethernet also): > >kern.ipc.maxsockbuf=16777216 > >net.inet.tcp.sendbuf_max=16777216 > >net.inet.tcp.recvbuf_max=16777216 > >net.inet.tcp.sendbuf_auto=1 > >net.inet.tcp.recvbuf_auto=1 > >net.inet.tcp.sendbuf_inc=16384 > >net.inet.tcp.recvbuf_inc=524288 > >net.inet.tcp.cc.algorithm=htcp Well, I'm not familiar with the current TCP stack (and, as noted before, I know nothing about InfiniBand). All I can suggest is testing with the default congestion control algorithm. (I always test with the default, which appears to be newreno.) NFS traffic looks very different than a typical use of TCP. Lots of small TCP segments in both directions interspersed with some larger ones (the write requests or read replies). rickhelp
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YTOPR0101MB116206BEB6259AA2836B16BCDDCD0>