Date: Sun, 25 Feb 2024 19:56:37 -0500 From: Mark Saad <nonesuch@longcount.org> To: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: NFS performance with 10GBase-T Message-ID: <CAMXt9NZYKFnaXLnS10PuARvj2B7oYzd98Y5aExCkhz3cY%2BHwpQ@mail.gmail.com> In-Reply-To: <CAM5tNy5guh99YZ7DLxgVsaE2LcXAqSSjs2FocMYpGJ9tspkYEQ@mail.gmail.com> References: <034cc6ea-26d8-4520-879a-672459832407@fsfe.org> <9066A50F-26DC-4314-B79E-66120A2B5A2F@freebsd.org> <CAM5tNy5guh99YZ7DLxgVsaE2LcXAqSSjs2FocMYpGJ9tspkYEQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] H On Sun, Feb 25, 2024 at 6:51 PM Rick Macklem <rick.macklem@gmail.com> wrote: > On Sun, Feb 25, 2024 at 1:21 AM <tuexen@freebsd.org> wrote: > > > > CAUTION: This email originated from outside of the University of Guelph. > Do not click links or open attachments unless you recognize the sender and > know the content is safe. If in doubt, forward suspicious emails to > IThelp@uoguelph.ca. > > > > > > > On Feb 25, 2024, at 01:18, Hannes Hauswedell <h2+lists2024@fsfe.org> > wrote: > > > > > > Hi everyone, > > > > > > I am coming here from > > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=2771971160 > > I guess this should read: > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277197 > Btw, what Hannes reported in the PR was that performance was > about the same for Linux and FreeBSD NFS clients when the link > was using a 1500byte ethernet frames. > However, Linux performs much better with 9K jumbo frames > whereas FreeBSD performance does not improve for 9K jumbo > frames. (Some mount options I suggested did help somewhat > for FreeBSD. Basically increasing rsize/wsize did help, but he > still sees performance below what Linux gets when 9K jumbo frames > are used. (I did note the potential problem of mbuf cluster pool > fragmentation when 9K jumbo frames are used, although I did not > intent to imply that this issue is involved, just that it is a known > deficiency that "might" be a factor.) > > rick > > > > Best regards > > Michael > > > > > > TL;DR: > > > > > > * I have a FreeBSD14 Server and Client with an Intel X540 (ix) adaptor > each. > > > * I am trying to improve the NFS throughput. > > > * I get 1160 MiB/s via nc, but only ~200 MiB/s via NFS. > > > * Increasing rsize and wsize to 1 MiB increases throughput to 395 MiB/s > > > * But a Linux client achieves 560-600 MiB/s with any rsize. > > > * The mtu is set to 9000 but this doesn't make a difference for the > FreeBSD client (it does make a difference for Linux). > > > > > > I assume < 400 MiB/s is not the expected performance? Do you have any > advice on debugging this? > > > > > > Thank you for your help, > > > Hannes > > > > > > > > > > > Hannes In the dmesg posted I see that you have a epair loaded . Are you trying to do NFS inside of a Jail ? Rick, Didn't someone from Isilon or Dell/EMC post about the 9K frames a long time ago ? I know in isilon land their FreeBSD can do this, but I can't say I have any idea how it's being done. They do have some kernel auto-tune magic as well to find optimal send and receive buffers. Maybe what we are seeing is Linux having better ergonomics on buffers out of the box ? Hannes Can you post your boot.conf and sysctl.conf settings. -- mark saad | nonesuch@longcount.org [-- Attachment #2 --] <div dir="ltr"><div>H<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Feb 25, 2024 at 6:51 PM Rick Macklem <<a href="mailto:rick.macklem@gmail.com">rick.macklem@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Sun, Feb 25, 2024 at 1:21 AM <<a href="mailto:tuexen@freebsd.org" target="_blank">tuexen@freebsd.org</a>> wrote:<br> ><br> > CAUTION: This email originated from outside of the University of Guelph. Do not click links or open attachments unless you recognize the sender and know the content is safe. If in doubt, forward suspicious emails to <a href="mailto:IThelp@uoguelph.ca" target="_blank">IThelp@uoguelph.ca</a>.<br> ><br> ><br> > > On Feb 25, 2024, at 01:18, Hannes Hauswedell <<a href="mailto:h2%2Blists2024@fsfe.org" target="_blank">h2+lists2024@fsfe.org</a>> wrote:<br> > ><br> > > Hi everyone,<br> > ><br> > > I am coming here from<br> > > <a href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=2771971160" rel="noreferrer" target="_blank">https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=2771971160</a><br>; > I guess this should read:<br> > <a href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277197" rel="noreferrer" target="_blank">https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277197</a><br>; Btw, what Hannes reported in the PR was that performance was<br> about the same for Linux and FreeBSD NFS clients when the link<br> was using a 1500byte ethernet frames.<br> However, Linux performs much better with 9K jumbo frames<br> whereas FreeBSD performance does not improve for 9K jumbo<br> frames. (Some mount options I suggested did help somewhat<br> for FreeBSD. Basically increasing rsize/wsize did help, but he<br> still sees performance below what Linux gets when 9K jumbo frames<br> are used. (I did note the potential problem of mbuf cluster pool<br> fragmentation when 9K jumbo frames are used, although I did not<br> intent to imply that this issue is involved, just that it is a known<br> deficiency that "might" be a factor.)<br> <br> rick<br> ><br> > Best regards<br> > Michael<br> > ><br> > > TL;DR:<br> > ><br> > > * I have a FreeBSD14 Server and Client with an Intel X540 (ix) adaptor each.<br> > > * I am trying to improve the NFS throughput.<br> > > * I get 1160 MiB/s via nc, but only ~200 MiB/s via NFS.<br> > > * Increasing rsize and wsize to 1 MiB increases throughput to 395 MiB/s<br> > > * But a Linux client achieves 560-600 MiB/s with any rsize.<br> > > * The mtu is set to 9000 but this doesn't make a difference for the FreeBSD client (it does make a difference for Linux).<br> > ><br> > > I assume < 400 MiB/s is not the expected performance? Do you have any advice on debugging this?<br> > ><br> > > Thank you for your help,<br> > > Hannes<br> > ><br> ><br> ><br> ><br> <br> </blockquote></div><div> Hannes</div><div> In the dmesg posted I see that you have a epair loaded . Are you trying to do NFS inside of a Jail ? <br></div><div><br></div><div>Rick, Didn't someone from Isilon or Dell/EMC post about the 9K frames a long time ago ? I know in isilon land</div><div>their FreeBSD can do this, but I can't say I have any idea how it's being done. They do have some kernel auto-tune magic as well</div><div>to find optimal send and receive buffers. Maybe what we are seeing is Linux having better ergonomics on buffers out of the box ?</div><div><br></div><div>Hannes</div><div> Can you post your boot.conf and sysctl.conf settings. <br></div><div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature">mark saad | <a href="mailto:nonesuch@longcount.org" target="_blank">nonesuch@longcount.org</a><br></div></div></div>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAMXt9NZYKFnaXLnS10PuARvj2B7oYzd98Y5aExCkhz3cY%2BHwpQ>