Date: Fri, 21 Oct 2022 11:13:14 -0700 From: Navdeep Parhar <nparhar@gmail.com> To: mike tancsa <mike@sentex.net>, Freebsd performance <freebsd-performance@freebsd.org> Subject: Re: Chelsio Forwarding performance and RELENG_13 vs RELENG_12 Message-ID: <fa98a400-75b4-6453-0b40-ae098ac3ab59@gmail.com> In-Reply-To: <b741dd17-f6eb-66c3-82f6-52b87475704f@sentex.net> References: <7b86e3fe-62e4-7b3e-f4bf-30e4894db9db@sentex.net> <92cdf4b8-2209-ec44-8151-a59b9e8f1504@gmail.com> <b741dd17-f6eb-66c3-82f6-52b87475704f@sentex.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On 10/21/22 10:57 AM, mike tancsa wrote: > On 10/21/2022 1:31 PM, Navdeep Parhar wrote: >> On 10/18/22 12:16 PM, mike tancsa wrote: >>> I updated a RELENG_12 router along with the hardware to RELENG_13 >>> (oct 14th kernel) and was surprised to see an increase in >>> dev.cxl.0.stats.rx_ovflow0 at a somewhat faster rate than I was >>> seeing on the older slightly slower hardware under about the same >>> load. (Xeon(R) E-2226G CPU @ 3.40GHz) vs a 4 core Xeon same freq, >>> same memory speed. About 150Kpps in and out and a 1Gb/s throughput >>> >>> loader.conf is the same >>> >>> >>> hw.cxgbe.toecaps_allowed="0" >>> hw.cxgbe.rdmacaps_allowed="0" >>> hw.cxgbe.iscsicaps_allowed="0" >>> hw.cxgbe.fcoecaps_allowed="0" >>> hw.cxgbe.pause_settings="0" >>> hw.cxgbe.attack_filter="1" >>> hw.cxgbe.drop_pkts_with_l3_errors="1" >>> >>> As there is a large routing table, I do have >>> >>> [fib_algo] inet.0 (radix4_lockless#46) rebuild_fd_flm: switching algo >>> to radix4 >>> [fib_algo] inet6.0 (radix6_lockless#58) rebuild_fd_flm: switching >>> algo to radix6 >>> >>> kicking in. >>> >>> and sysctl.conf >>> >>> net.route.multipath=0 >>> >>> net.inet.ip.redirect=0 >>> net.inet6.ip6.redirect=0 >>> kern.ipc.maxsockbuf=16777216 >>> net.inet.tcp.blackhole=1 >>> >>> Are there any other tweaks that can be done in order to better >>> forwarding performance ? I do see at bootup time >>> >>> cxl0: nrxq (6), hw RSS table size (128); expect uneven traffic >>> distribution. >>> cxl1: nrxq (6), hw RSS table size (128); expect uneven traffic >>> distribution. >>> cxl3: nrxq (6), hw RSS table size (128); expect uneven traffic >>> distribution. >>> >>> The cpu is 6 core. No HT enabled >> >> The old system was 4-core so it must have used 4 queues. Can you >> please try that on the new system and see how it does? >> >> hw.cxgbe.ntxq=4 >> hw.cxgbe.nrxq=4 >> > Thanks Navdeep! > > Unfortunately, still the odd dropped packet :( Can you try increasing the size of the queues? hw.cxgbe.qsize_txq=2048 hw.cxgbe.qsize_rxq=2048 The stats show that you are using MTU 1500. If you were using MTU 9000 I'd also have suggested setting largest_rx_cluster to 4K. hw.cxgbe.largest_rx_cluster=4096 Regards, Navdeep
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?fa98a400-75b4-6453-0b40-ae098ac3ab59>