Date: Sun, 25 Feb 2024 19:56:37 -0500 From: Mark Saad <nonesuch@longcount.org> To: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: NFS performance with 10GBase-T Message-ID: <CAMXt9NZYKFnaXLnS10PuARvj2B7oYzd98Y5aExCkhz3cY%2BHwpQ@mail.gmail.com> In-Reply-To: <CAM5tNy5guh99YZ7DLxgVsaE2LcXAqSSjs2FocMYpGJ9tspkYEQ@mail.gmail.com> References: <034cc6ea-26d8-4520-879a-672459832407@fsfe.org> <9066A50F-26DC-4314-B79E-66120A2B5A2F@freebsd.org> <CAM5tNy5guh99YZ7DLxgVsaE2LcXAqSSjs2FocMYpGJ9tspkYEQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--0000000000007ac81606123e6250 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable H On Sun, Feb 25, 2024 at 6:51=E2=80=AFPM Rick Macklem <rick.macklem@gmail.co= m> wrote: > On Sun, Feb 25, 2024 at 1:21=E2=80=AFAM <tuexen@freebsd.org> wrote: > > > > CAUTION: This email originated from outside of the University of Guelph= . > Do not click links or open attachments unless you recognize the sender an= d > know the content is safe. If in doubt, forward suspicious emails to > IThelp@uoguelph.ca. > > > > > > > On Feb 25, 2024, at 01:18, Hannes Hauswedell <h2+lists2024@fsfe.org> > wrote: > > > > > > Hi everyone, > > > > > > I am coming here from > > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D2771971160 > > I guess this should read: > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277197 > Btw, what Hannes reported in the PR was that performance was > about the same for Linux and FreeBSD NFS clients when the link > was using a 1500byte ethernet frames. > However, Linux performs much better with 9K jumbo frames > whereas FreeBSD performance does not improve for 9K jumbo > frames. (Some mount options I suggested did help somewhat > for FreeBSD. Basically increasing rsize/wsize did help, but he > still sees performance below what Linux gets when 9K jumbo frames > are used. (I did note the potential problem of mbuf cluster pool > fragmentation when 9K jumbo frames are used, although I did not > intent to imply that this issue is involved, just that it is a known > deficiency that "might" be a factor.) > > rick > > > > Best regards > > Michael > > > > > > TL;DR: > > > > > > * I have a FreeBSD14 Server and Client with an Intel X540 (ix) adapto= r > each. > > > * I am trying to improve the NFS throughput. > > > * I get 1160 MiB/s via nc, but only ~200 MiB/s via NFS. > > > * Increasing rsize and wsize to 1 MiB increases throughput to 395 MiB= /s > > > * But a Linux client achieves 560-600 MiB/s with any rsize. > > > * The mtu is set to 9000 but this doesn't make a difference for the > FreeBSD client (it does make a difference for Linux). > > > > > > I assume < 400 MiB/s is not the expected performance? Do you have any > advice on debugging this? > > > > > > Thank you for your help, > > > Hannes > > > > > > > > > > > Hannes In the dmesg posted I see that you have a epair loaded . Are you trying to do NFS inside of a Jail ? Rick, Didn't someone from Isilon or Dell/EMC post about the 9K frames a long time ago ? I know in isilon land their FreeBSD can do this, but I can't say I have any idea how it's being done. They do have some kernel auto-tune magic as well to find optimal send and receive buffers. Maybe what we are seeing is Linux having better ergonomics on buffers out of the box ? Hannes Can you post your boot.conf and sysctl.conf settings. --=20 mark saad | nonesuch@longcount.org --0000000000007ac81606123e6250 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>H<br></div><br><div class=3D"gmail_quote"><div dir=3D= "ltr" class=3D"gmail_attr">On Sun, Feb 25, 2024 at 6:51=E2=80=AFPM Rick Mac= klem <<a href=3D"mailto:rick.macklem@gmail.com">rick.macklem@gmail.com</= a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0p= x 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On= Sun, Feb 25, 2024 at 1:21=E2=80=AFAM <<a href=3D"mailto:tuexen@freebsd.= org" target=3D"_blank">tuexen@freebsd.org</a>> wrote:<br> ><br> > CAUTION: This email originated from outside of the University of Guelp= h. Do not click links or open attachments unless you recognize the sender a= nd know the content is safe. If in doubt, forward suspicious emails to <a h= ref=3D"mailto:IThelp@uoguelph.ca" target=3D"_blank">IThelp@uoguelph.ca</a>.= <br> ><br> ><br> > > On Feb 25, 2024, at 01:18, Hannes Hauswedell <<a href=3D"mailt= o:h2%2Blists2024@fsfe.org" target=3D"_blank">h2+lists2024@fsfe.org</a>> = wrote:<br> > ><br> > > Hi everyone,<br> > ><br> > > I am coming here from<br> > > <a href=3D"https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D27= 71971160" rel=3D"noreferrer" target=3D"_blank">https://bugs.freebsd.org/bug= zilla/show_bug.cgi?id=3D2771971160</a><br> > I guess this should read:<br> > <a href=3D"https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277197"= rel=3D"noreferrer" target=3D"_blank">https://bugs.freebsd.org/bugzilla/sho= w_bug.cgi?id=3D277197</a><br> Btw, what Hannes reported in the PR was that performance was<br> about the same for Linux and FreeBSD NFS clients when the link<br> was using a 1500byte ethernet frames.<br> However, Linux performs much better with 9K jumbo frames<br> whereas FreeBSD performance does not improve for 9K jumbo<br> frames. (Some mount options I suggested did help somewhat<br> for FreeBSD. Basically increasing rsize/wsize did help, but he<br> still sees performance below what Linux gets when 9K jumbo frames<br> are used. (I did note the potential problem of mbuf cluster pool<br> fragmentation when 9K jumbo frames are used, although I did not<br> intent to imply that this issue is involved, just that it is a known<br> deficiency that "might" be a factor.)<br> <br> rick<br> ><br> > Best regards<br> > Michael<br> > ><br> > > TL;DR:<br> > ><br> > > * I have a FreeBSD14 Server and Client with an Intel X540 (ix) ad= aptor each.<br> > > * I am trying to improve the NFS throughput.<br> > > * I get 1160 MiB/s via nc, but only ~200 MiB/s via NFS.<br> > > * Increasing rsize and wsize to 1 MiB increases throughput to 395= MiB/s<br> > > * But a Linux client achieves 560-600 MiB/s with any rsize.<br> > > * The mtu is set to 9000 but this doesn't make a difference f= or the FreeBSD client (it does make a difference for Linux).<br> > ><br> > > I assume < 400 MiB/s is not the expected performance? Do you h= ave any advice on debugging this?<br> > ><br> > > Thank you for your help,<br> > > Hannes<br> > ><br> ><br> ><br> ><br> <br> </blockquote></div><div>=C2=A0Hannes</div><div>=C2=A0=C2=A0 In the dmesg po= sted I see that you have a epair loaded . Are you trying to do NFS inside o= f a Jail ? <br></div><div><br></div><div>Rick, Didn't someone from Isil= on or Dell/EMC post about the 9K frames a long time ago ?=C2=A0 I know in i= silon land</div><div>their FreeBSD can do this, but I can't say I have = any idea how it's being done. They do have some kernel auto-tune magic = as well</div><div>to find optimal send and receive buffers. Maybe what we a= re seeing is Linux having better ergonomics on buffers out of the box ?</di= v><div><br></div><div>Hannes</div><div>=C2=A0 Can you post your boot.conf a= nd sysctl.conf settings. <br></div><div><span class=3D"gmail_signature_pref= ix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature">mark saad | <a= href=3D"mailto:nonesuch@longcount.org" target=3D"_blank">nonesuch@longcoun= t.org</a><br></div></div></div> --0000000000007ac81606123e6250--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAMXt9NZYKFnaXLnS10PuARvj2B7oYzd98Y5aExCkhz3cY%2BHwpQ>