Date: Tue, 30 Oct 2018 07:57:21 +0300 From: Andrew Vylegzhanin <avv314@gmail.com> To: rmacklem@uoguelph.ca Cc: "Rodney W. Grimes" <freebsd-rwg@pdx.rh.cn85.dnsmgr.net>, freebsd-fs@freebsd.org, freebsd-infiniband@freebsd.org Subject: Re: NFS + Infiniband problem Message-ID: <CA%2BBi_YhHvpqkmP7WVd4wJj0u9rnbMZpum7FsKxZsu=%2B=8ze8-w@mail.gmail.com> In-Reply-To: <YTOPR0101MB11622A9797376128D2FA182ADDF30@YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM> References: <CA%2BBi_YiHoxFc3wsEPnMeBHWgW-nh6sXQCEgBTb=-nD6-XcjZ%2Bg@mail.gmail.com> <201810291506.w9TF6YAP057202@pdx.rh.CN85.dnsmgr.net> <YTOPR0101MB11622A9797376128D2FA182ADDF30@YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM>
next in thread | previous in thread | raw e-mail | index | archive | help
> >> Some details:
> >> [root@node4 ~]# mount_nfs -o wsize=30000 -o proto=tcp 10.0.2.1:/zdata2
/mnt
> > ^^^^^^^^^^^^
> >I am not sure what the interaction between page sizes, TSO needs,
> >buffer needs and all that are but I always use a power of 2 wsize
> >and rsize.
Again after some tests.
I've tried 4096,8192,16384 wsize/rsize. Only 4096 value give some
measurable result and it's extremely slow ~ 10-16MB/s for writing (depend
on number of threads), 10-12 MB/s for reading. With other values NFS hangs
(or almost hangs - couple kB/s in average)
Changing sysctl net.inet.tcp.tso=0 (w/o reboot) on both sides had no
effect.
AFAIK, infiniband interface has no option for TSO,LRO:
ib0: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 65520
options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
lladdr 80.0.2.8.fe.80.0.0.0.0.0.0.e4.1d.2d.3.0.50.df.51
inet 10.0.2.1 netmask 0xffffff00 broadcast 10.0.2.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
BTW, hers is my sysctl.conf file with optimisation for congestion control
and tcp buffers on 10/40 Gbit/s links (the server had 40 Gbit/s Intel ixl
ethernet also):
kern.ipc.maxsockbuf=16777216
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.recvbuf_auto=1
net.inet.tcp.sendbuf_inc=16384
net.inet.tcp.recvbuf_inc=524288
net.inet.tcp.cc.algorithm=htcp
> > You might try that. And as Rick suggested, turn of
> >TSO, if you can. Is infiniband using RDMA to do this, if so then
> >the page size stuff is probably very important, use multiples of
> >4096 only.
> RDMA is not supported by the FreeBSD NFS client. There is a way to use
RDMA
> on a separate connection with NFSv4.1 or later, but I've never written
code
> for that. (Not practical to try to implement without access to hardware
that
> does it.)
>
Hope I could help with testing.
--
Andrew
> rick
> [performance stuff snipped]
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2BBi_YhHvpqkmP7WVd4wJj0u9rnbMZpum7FsKxZsu=%2B=8ze8-w>