Date: Wed, 18 May 2022 20:04:14 -0400 From: Adam Stylinski <kungfujesus06@gmail.com> To: freebsd-fs@freebsd.org Subject: zfs/nfsd performance limiter Message-ID: <CAJwHY9WMOOLy=rb9FNjExQtYej21Zv=Po9Cbg=19gkw1SLFSww@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hello, I have two systems connected via ConnectX-3 mellanox cards in ethernet mode. They have their MTU's maxed at 9000, their ring buffers maxed at 8192, and I can hit around 36 gbps with iperf. When using an NFS client (client = linux, server = freebsd), I see a maximum rate of around 20gbps. The test file is fully in ARC. The test is performed with an NFS mount nconnect=4 and an rsize/wsize of 1MB. Here's the flame graph of the kernel of the system in question, with idle stacks removed: https://gist.github.com/KungFuJesus/918c6dcf40ae07767d5382deafab3a52#file-nfs_fg-svg The longest functions seems like maybe it's the ERMS aware memcpy happening from the ARC? Is there maybe a missing fast path that could take fewer copies into the socket buffer?
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJwHY9WMOOLy=rb9FNjExQtYej21Zv=Po9Cbg=19gkw1SLFSww>