Date: Fri, 21 Oct 2022 10:31:36 -0700 From: Navdeep Parhar <nparhar@gmail.com> To: mike tancsa <mike@sentex.net>, Freebsd performance <freebsd-performance@freebsd.org> Subject: Re: Chelsio Forwarding performance and RELENG_13 vs RELENG_12 Message-ID: <92cdf4b8-2209-ec44-8151-a59b9e8f1504@gmail.com> In-Reply-To: <7b86e3fe-62e4-7b3e-f4bf-30e4894db9db@sentex.net> References: <7b86e3fe-62e4-7b3e-f4bf-30e4894db9db@sentex.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On 10/18/22 12:16 PM, mike tancsa wrote: > I updated a RELENG_12 router along with the hardware to RELENG_13 (oct > 14th kernel) and was surprised to see an increase in > dev.cxl.0.stats.rx_ovflow0 at a somewhat faster rate than I was seeing > on the older slightly slower hardware under about the same load. > (Xeon(R) E-2226G CPU @ 3.40GHz) vs a 4 core Xeon same freq, same memory > speed. About 150Kpps in and out and a 1Gb/s throughput > > loader.conf is the same > > > hw.cxgbe.toecaps_allowed="0" > hw.cxgbe.rdmacaps_allowed="0" > hw.cxgbe.iscsicaps_allowed="0" > hw.cxgbe.fcoecaps_allowed="0" > hw.cxgbe.pause_settings="0" > hw.cxgbe.attack_filter="1" > hw.cxgbe.drop_pkts_with_l3_errors="1" > > As there is a large routing table, I do have > > [fib_algo] inet.0 (radix4_lockless#46) rebuild_fd_flm: switching algo to > radix4 > [fib_algo] inet6.0 (radix6_lockless#58) rebuild_fd_flm: switching algo > to radix6 > > kicking in. > > and sysctl.conf > > net.route.multipath=0 > > net.inet.ip.redirect=0 > net.inet6.ip6.redirect=0 > kern.ipc.maxsockbuf=16777216 > net.inet.tcp.blackhole=1 > > Are there any other tweaks that can be done in order to better > forwarding performance ? I do see at bootup time > > cxl0: nrxq (6), hw RSS table size (128); expect uneven traffic > distribution. > cxl1: nrxq (6), hw RSS table size (128); expect uneven traffic > distribution. > cxl3: nrxq (6), hw RSS table size (128); expect uneven traffic > distribution. > > The cpu is 6 core. No HT enabled The old system was 4-core so it must have used 4 queues. Can you please try that on the new system and see how it does? hw.cxgbe.ntxq=4 hw.cxgbe.nrxq=4 Regards, Navdeep
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?92cdf4b8-2209-ec44-8151-a59b9e8f1504>