Date: Fri, 3 Dec 2021 23:59:38 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: Ryan Stone <rysto32@gmail.com>, John Jasen <jjasen@gmail.com> Cc: FreeBSD Net <freebsd-net@freebsd.org>, Navdeep Parhar <navdeep@chelsio.com> Subject: Re: Chelsio cards, jumbo frames, memory fragmentation and performance in FreeBSD 13.x? Message-ID: <YQXPR0101MB096804818D76CC3A7E694B57DD6A9@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <CAFMmRNwTJsZLieX2UXgUJoBrP-MREoDFjfcF7_wWz29QUtykcQ@mail.gmail.com> References: <CAACLuR2cRb4JW5KYjaTjK1dVWHuuWK03HVtNQH1XWeJ__KuGrw@mail.gmail.com> <CAFMmRNwTJsZLieX2UXgUJoBrP-MREoDFjfcF7_wWz29QUtykcQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Ryan Stone <rysto32@gmail.com> wrote:=0A= >=0A= > Set it to 4k and move on. I'm not aware of any efforts in the VM=0A= > layer or network stack to make >PAGE_SIZE clusters usable in practical=0A= > situations. Honestly, at this point I wish that we'd just kill them=0A= > entirely.=0A= I feel about the same. However, I'm guessing that there are some net=0A= drivers that can't work without them (at least for jumbo frames) and=0A= no one available to fix those drivers.=0A= =0A= At the least, we should strongly encourage net device driver authors=0A= to avoid their use wherever practicable.=0A= =0A= I also think that "don't use them" should be the default for device=0A= drivers that can optionally not use them.=0A= =0A= I'll admit I cannot think of a good way to fix jumbo clusters, so avoiding= =0A= them is about all we can do, I think?=0A= - Separate allocation pools would avoid the fragmentation problem,=0A= but waste a lot of space unless the mix of cluster sizes needed for the= =0A= network load on the machine was somehow known.=0A= =0A= rick=0A= =0A= On Fri, Dec 3, 2021 at 12:52 PM John Jasen <jjasen@gmail.com> wrote:=0A= >=0A= > Pretty close to two years ago, we tripped across conditions where heavily= =0A= > used FreeBSD 11.x packet-filter firewalls would slow to a crawl and load= =0A= > would go crazy.=0A= >=0A= > In a fit of hopefulness with our upgrade to FreeBSD 13.0, I removed the= =0A= > hw.cxgbe.largest_rx_cluster settings we put in place -- only to have our= =0A= > hardest hit firewall experience high load and plummeting bandwidth within= a=0A= > few hours.=0A= >=0A= > So, I have two questions -- is there a better fix for this? Or, is the=0A= > approach to set largest_rx_cluster to 4k and move on?=0A= >=0A= > Thanks in advance!=0A= =0A=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YQXPR0101MB096804818D76CC3A7E694B57DD6A9>