Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 30 Oct 2018 13:17:07 -0500
From:      Jason Bacon <bacon4000@gmail.com>
To:        freebsd-infiniband@freebsd.org
Subject:   Re: NFS + Infiniband problem
Message-ID:  <7448666e-6028-12e7-aa74-4fd162e13dc7@gmail.com>
In-Reply-To: <CA%2BBi_YhHvpqkmP7WVd4wJj0u9rnbMZpum7FsKxZsu=%2B=8ze8-w@mail.gmail.com>
References:  <CA%2BBi_YiHoxFc3wsEPnMeBHWgW-nh6sXQCEgBTb=-nD6-XcjZ%2Bg@mail.gmail.com> <201810291506.w9TF6YAP057202@pdx.rh.CN85.dnsmgr.net> <YTOPR0101MB11622A9797376128D2FA182ADDF30@YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM> <CA%2BBi_YhHvpqkmP7WVd4wJj0u9rnbMZpum7FsKxZsu=%2B=8ze8-w@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
On 10/29/18 11:57 PM, Andrew Vylegzhanin wrote:
>>>> Some details:
>>>> [root@node4 ~]# mount_nfs -o wsize=30000 -o proto=tcp 10.0.2.1:/zdata2
> /mnt
>>>                                ^^^^^^^^^^^^
>>> I am not sure what the interaction between page sizes, TSO needs,
>>> buffer needs and all that are but I always use a power of 2 wsize
>>> and rsize.
> Again after some tests.
> I've tried 4096,8192,16384 wsize/rsize. Only 4096 value give some
> measurable result and it's extremely slow ~ 10-16MB/s for writing (depend
> on number of threads), 10-12 MB/s for reading. With other values NFS hangs
> (or almost hangs - couple kB/s in average)
>
> Changing sysctl net.inet.tcp.tso=0 (w/o reboot) on both sides  had no
> effect.
>
> AFAIK, infiniband interface has no option for TSO,LRO:
> ib0: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 65520
>           options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
>           lladdr 80.0.2.8.fe.80.0.0.0.0.0.0.e4.1d.2d.3.0.50.df.51
>           inet 10.0.2.1 netmask 0xffffff00 broadcast 10.0.2.255
>           nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>
>
> BTW, hers is my sysctl.conf file with optimisation for congestion control
> and tcp buffers on 10/40 Gbit/s links (the server had 40 Gbit/s Intel ixl
> ethernet also):
>
> kern.ipc.maxsockbuf=16777216
>
> net.inet.tcp.sendbuf_max=16777216
>
> net.inet.tcp.recvbuf_max=16777216
>
> net.inet.tcp.sendbuf_auto=1
>
> net.inet.tcp.recvbuf_auto=1
>
> net.inet.tcp.sendbuf_inc=16384
>
> net.inet.tcp.recvbuf_inc=524288
>
> net.inet.tcp.cc.algorithm=htcp
>
>
>>>    You might try that.  And as Rick suggested, turn of
>>> TSO, if you can.  Is infiniband using RDMA to do this, if so then
>>> the page size stuff is probably very important, use multiples of
>>> 4096 only.
>> RDMA is not supported by the FreeBSD NFS client. There is a way to use
> RDMA
>> on a separate connection with NFSv4.1 or later, but I've never written
> code
>> for that. (Not practical to try to implement without access to hardware
> that
>> does it.)
>>
> Hope I could help with testing.
>
> --
> Andrew
>
>> rick
>> [performance stuff snipped]
> _______________________________________________
> freebsd-infiniband@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-infiniband
> To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.org"

Did you try NFS over a standard gigabit interface just to rule out any 
issues that are independent of IB?  It's odd that other protocols seem 
to work well.

     Jason

-- 
Earth is a beta site.




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7448666e-6028-12e7-aa74-4fd162e13dc7>