Date: Sun, 25 Feb 2024 19:16:52 -0800 From: Rick Macklem <rick.macklem@gmail.com> To: Mark Saad <nonesuch@longcount.org> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: NFS performance with 10GBase-T Message-ID: <CAM5tNy6-hJVBtQs8BVfZjZ4mgcsQu7cQCaaf4qDRvO4rGbed1Q@mail.gmail.com> In-Reply-To: <CAMXt9NZYKFnaXLnS10PuARvj2B7oYzd98Y5aExCkhz3cY%2BHwpQ@mail.gmail.com> References: <034cc6ea-26d8-4520-879a-672459832407@fsfe.org> <9066A50F-26DC-4314-B79E-66120A2B5A2F@freebsd.org> <CAM5tNy5guh99YZ7DLxgVsaE2LcXAqSSjs2FocMYpGJ9tspkYEQ@mail.gmail.com> <CAMXt9NZYKFnaXLnS10PuARvj2B7oYzd98Y5aExCkhz3cY%2BHwpQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Feb 25, 2024 at 4:57=E2=80=AFPM Mark Saad <nonesuch@longcount.org> = wrote: > > H > > On Sun, Feb 25, 2024 at 6:51=E2=80=AFPM Rick Macklem <rick.macklem@gmail.= com> wrote: >> >> On Sun, Feb 25, 2024 at 1:21=E2=80=AFAM <tuexen@freebsd.org> wrote: >> > >> > CAUTION: This email originated from outside of the University of Guelp= h. Do not click links or open attachments unless you recognize the sender a= nd know the content is safe. If in doubt, forward suspicious emails to IThe= lp@uoguelph.ca. >> > >> > >> > > On Feb 25, 2024, at 01:18, Hannes Hauswedell <h2+lists2024@fsfe.org>= wrote: >> > > >> > > Hi everyone, >> > > >> > > I am coming here from >> > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D2771971160 >> > I guess this should read: >> > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277197 >> Btw, what Hannes reported in the PR was that performance was >> about the same for Linux and FreeBSD NFS clients when the link >> was using a 1500byte ethernet frames. >> However, Linux performs much better with 9K jumbo frames >> whereas FreeBSD performance does not improve for 9K jumbo >> frames. (Some mount options I suggested did help somewhat >> for FreeBSD. Basically increasing rsize/wsize did help, but he >> still sees performance below what Linux gets when 9K jumbo frames >> are used. (I did note the potential problem of mbuf cluster pool >> fragmentation when 9K jumbo frames are used, although I did not >> intent to imply that this issue is involved, just that it is a known >> deficiency that "might" be a factor.) >> >> rick >> > >> > Best regards >> > Michael >> > > >> > > TL;DR: >> > > >> > > * I have a FreeBSD14 Server and Client with an Intel X540 (ix) adapt= or each. >> > > * I am trying to improve the NFS throughput. >> > > * I get 1160 MiB/s via nc, but only ~200 MiB/s via NFS. >> > > * Increasing rsize and wsize to 1 MiB increases throughput to 395 Mi= B/s >> > > * But a Linux client achieves 560-600 MiB/s with any rsize. >> > > * The mtu is set to 9000 but this doesn't make a difference for the = FreeBSD client (it does make a difference for Linux). >> > > >> > > I assume < 400 MiB/s is not the expected performance? Do you have an= y advice on debugging this? >> > > >> > > Thank you for your help, >> > > Hannes >> > > >> > >> > >> > >> > Hannes > In the dmesg posted I see that you have a epair loaded . Are you tryin= g to do NFS inside of a Jail ? > > Rick, Didn't someone from Isilon or Dell/EMC post about the 9K frames a l= ong time ago ? I know in isilon land > their FreeBSD can do this, but I can't say I have any idea how it's being= done. They do have some kernel auto-tune magic as well > to find optimal send and receive buffers. Maybe what we are seeing is Lin= ux having better ergonomics on buffers out of the box ? > Oops, my bad. Yes, I just took a quick look and it appears that jumbo mbuf clusters are now allocated from separate uma_zones, so the fragmentation problem sho= uld not exist. (I hadn't heard about this for quite a while, but hadn't noticed a commit fixing it I'll wade through the commit log to see when it got changed tomor= row.) So, I think the above comment should be ignored, unless others need to corr= ect it further. Sorry about this, rick ps: I have no idea how Linux does network buffers. > Hannes > Can you post your boot.conf and sysctl.conf settings. > -- > mark saad | nonesuch@longcount.org
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAM5tNy6-hJVBtQs8BVfZjZ4mgcsQu7cQCaaf4qDRvO4rGbed1Q>