From nobody Fri Oct 21 17:57:22 2022 X-Original-To: freebsd-performance@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MvBy25SVzz4gLyM for ; Fri, 21 Oct 2022 17:57:30 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MvBy151yxz3CLn for ; Fri, 21 Oct 2022 17:57:29 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.16.1/8.16.1) with ESMTPS id 29LHvL9w095168 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Fri, 21 Oct 2022 13:57:22 -0400 (EDT) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:f808:cefc:42f3:2221] ([IPv6:2607:f3e0:0:4:f808:cefc:42f3:2221]) by pyroxene2a.sentex.ca (8.16.1/8.15.2) with ESMTPS id 29LHvLhx012064 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Fri, 21 Oct 2022 13:57:21 -0400 (EDT) (envelope-from mike@sentex.net) Message-ID: Date: Fri, 21 Oct 2022 13:57:22 -0400 List-Id: Performance/tuning List-Archive: https://lists.freebsd.org/archives/freebsd-performance List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-performance@freebsd.org X-BeenThere: freebsd-performance@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.3.3 Subject: Re: Chelsio Forwarding performance and RELENG_13 vs RELENG_12 Content-Language: en-US To: Navdeep Parhar , Freebsd performance References: <7b86e3fe-62e4-7b3e-f4bf-30e4894db9db@sentex.net> <92cdf4b8-2209-ec44-8151-a59b9e8f1504@gmail.com> From: mike tancsa In-Reply-To: <92cdf4b8-2209-ec44-8151-a59b9e8f1504@gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.84 on 64.7.153.18 X-Rspamd-Queue-Id: 4MvBy151yxz3CLn X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:1::12 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-3.40 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; RCVD_IN_DNSWL_LOW(-0.10)[199.212.134.19:received]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_ALL(0.00)[]; FREEMAIL_TO(0.00)[gmail.com,freebsd.org]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-performance@freebsd.org]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[mike]; RCPT_COUNT_TWO(0.00)[2]; DMARC_NA(0.00)[sentex.net]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; ARC_NA(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On 10/21/2022 1:31 PM, Navdeep Parhar wrote: > On 10/18/22 12:16 PM, mike tancsa wrote: >> I updated a RELENG_12 router along with the hardware to RELENG_13 >> (oct 14th kernel) and was surprised to see an increase in >> dev.cxl.0.stats.rx_ovflow0 at a somewhat faster rate than I was >> seeing on the older slightly slower hardware under about the same >> load. (Xeon(R) E-2226G CPU @ 3.40GHz) vs a 4 core Xeon same freq, >> same memory speed. About 150Kpps in and out and a 1Gb/s throughput >> >> loader.conf is the same >> >> >> hw.cxgbe.toecaps_allowed="0" >> hw.cxgbe.rdmacaps_allowed="0" >> hw.cxgbe.iscsicaps_allowed="0" >> hw.cxgbe.fcoecaps_allowed="0" >> hw.cxgbe.pause_settings="0" >> hw.cxgbe.attack_filter="1" >> hw.cxgbe.drop_pkts_with_l3_errors="1" >> >> As there is a large routing table, I do have >> >> [fib_algo] inet.0 (radix4_lockless#46) rebuild_fd_flm: switching algo >> to radix4 >> [fib_algo] inet6.0 (radix6_lockless#58) rebuild_fd_flm: switching >> algo to radix6 >> >> kicking in. >> >> and sysctl.conf >> >> net.route.multipath=0 >> >> net.inet.ip.redirect=0 >> net.inet6.ip6.redirect=0 >> kern.ipc.maxsockbuf=16777216 >> net.inet.tcp.blackhole=1 >> >> Are there any other tweaks that can be done in order to better >> forwarding performance ? I do see at bootup time >> >> cxl0: nrxq (6), hw RSS table size (128); expect uneven traffic >> distribution. >> cxl1: nrxq (6), hw RSS table size (128); expect uneven traffic >> distribution. >> cxl3: nrxq (6), hw RSS table size (128); expect uneven traffic >> distribution. >> >> The cpu is 6 core. No HT enabled > > The old system was 4-core so it must have used 4 queues.  Can you > please try that on the new system and see how it does? > > hw.cxgbe.ntxq=4 > hw.cxgbe.nrxq=4 > Thanks Navdeep! Unfortunately, still the odd dropped packet :( dev.cxl.0.stats.rx_ovflow0: 78 Since my initial post, I did try with 16, and it did not seem to impact the rate of overflows either. dev.cxl.0.stats.rx_trunc3: 0 dev.cxl.0.stats.rx_trunc2: 0 dev.cxl.0.stats.rx_trunc1: 0 dev.cxl.0.stats.rx_trunc0: 1 dev.cxl.0.stats.rx_ovflow3: 0 dev.cxl.0.stats.rx_ovflow2: 0 dev.cxl.0.stats.rx_ovflow1: 0 dev.cxl.0.stats.rx_ovflow0: 78 dev.cxl.0.stats.rx_ppp7: 0 dev.cxl.0.stats.rx_ppp6: 0 dev.cxl.0.stats.rx_ppp5: 0 dev.cxl.0.stats.rx_ppp4: 0 dev.cxl.0.stats.rx_ppp3: 0 dev.cxl.0.stats.rx_ppp2: 0 dev.cxl.0.stats.rx_ppp1: 0 dev.cxl.0.stats.rx_ppp0: 0 dev.cxl.0.stats.rx_pause: 0 dev.cxl.0.stats.rx_frames_1519_max: 0 dev.cxl.0.stats.rx_frames_1024_1518: 20724413 dev.cxl.0.stats.rx_frames_512_1023: 1371427 dev.cxl.0.stats.rx_frames_256_511: 1515522 dev.cxl.0.stats.rx_frames_128_255: 2371419 dev.cxl.0.stats.rx_frames_65_127: 3386302 dev.cxl.0.stats.rx_frames_64: 2882681 dev.cxl.0.stats.rx_runt: 0 dev.cxl.0.stats.rx_symbol_err: 0 dev.cxl.0.stats.rx_len_err: 0 dev.cxl.0.stats.rx_fcs_err: 0 dev.cxl.0.stats.rx_jabber: 0 dev.cxl.0.stats.rx_too_long: 0 dev.cxl.0.stats.rx_ucast_frames: 32180697 dev.cxl.0.stats.rx_mcast_frames: 30513 dev.cxl.0.stats.rx_bcast_frames: 40594 dev.cxl.0.stats.rx_frames: 32251814 dev.cxl.0.stats.rx_octets: 31681379196 dev.cxl.0.stats.tx_ppp7: 0 dev.cxl.0.stats.tx_ppp6: 0 dev.cxl.0.stats.tx_ppp5: 0 dev.cxl.0.stats.tx_ppp4: 0 dev.cxl.0.stats.tx_ppp3: 0 dev.cxl.0.stats.tx_ppp2: 0 dev.cxl.0.stats.tx_ppp1: 0 dev.cxl.0.stats.tx_ppp0: 0 dev.cxl.0.stats.tx_pause: 0 dev.cxl.0.stats.tx_drop: 0 dev.cxl.0.stats.tx_frames_1519_max: 0 dev.cxl.0.stats.tx_frames_1024_1518: 5369922 dev.cxl.0.stats.tx_frames_512_1023: 736985 dev.cxl.0.stats.tx_frames_256_511: 842554 dev.cxl.0.stats.tx_frames_128_255: 2708331 dev.cxl.0.stats.tx_frames_65_127: 6254876 dev.cxl.0.stats.tx_frames_64: 2076101 dev.cxl.0.stats.tx_error_frames: 0 dev.cxl.0.stats.tx_ucast_frames: 17988628 dev.cxl.0.stats.tx_mcast_frames: 17 dev.cxl.0.stats.tx_bcast_frames: 130 dev.cxl.0.stats.tx_frames: 17988780 dev.cxl.0.stats.tx_octets: 9797845532 dev.cxl.0.stats.tx_parse_error: 0 dev.cxl.0.tc.14.params: uninitialized dev.cxl.0.tc.14.refcount: 0 dev.cxl.0.tc.14.flags: 0 dev.cxl.0.tc.14.state: 0 dev.cxl.0.tc.13.params: uninitialized dev.cxl.0.tc.13.refcount: 0 dev.cxl.0.tc.13.flags: 0 dev.cxl.0.tc.13.state: 0 dev.cxl.0.tc.12.params: uninitialized dev.cxl.0.tc.12.refcount: 0 dev.cxl.0.tc.12.flags: 0 dev.cxl.0.tc.12.state: 0 dev.cxl.0.tc.11.params: uninitialized dev.cxl.0.tc.11.refcount: 0 dev.cxl.0.tc.11.flags: 0 dev.cxl.0.tc.11.state: 0 dev.cxl.0.tc.10.params: uninitialized dev.cxl.0.tc.10.refcount: 0 dev.cxl.0.tc.10.flags: 0 dev.cxl.0.tc.10.state: 0 dev.cxl.0.tc.9.params: uninitialized dev.cxl.0.tc.9.refcount: 0 dev.cxl.0.tc.9.flags: 0 dev.cxl.0.tc.9.state: 0 dev.cxl.0.tc.8.params: uninitialized dev.cxl.0.tc.8.refcount: 0 dev.cxl.0.tc.8.flags: 0 dev.cxl.0.tc.8.state: 0 dev.cxl.0.tc.7.params: uninitialized dev.cxl.0.tc.7.refcount: 0 dev.cxl.0.tc.7.flags: 0 dev.cxl.0.tc.7.state: 0 dev.cxl.0.tc.6.params: uninitialized dev.cxl.0.tc.6.refcount: 0 dev.cxl.0.tc.6.flags: 0 dev.cxl.0.tc.6.state: 0 dev.cxl.0.tc.5.params: uninitialized dev.cxl.0.tc.5.refcount: 0 dev.cxl.0.tc.5.flags: 0 dev.cxl.0.tc.5.state: 0 dev.cxl.0.tc.4.params: uninitialized dev.cxl.0.tc.4.refcount: 0 dev.cxl.0.tc.4.flags: 0 dev.cxl.0.tc.4.state: 0 dev.cxl.0.tc.3.params: uninitialized dev.cxl.0.tc.3.refcount: 0 dev.cxl.0.tc.3.flags: 0 dev.cxl.0.tc.3.state: 0 dev.cxl.0.tc.2.params: uninitialized dev.cxl.0.tc.2.refcount: 0 dev.cxl.0.tc.2.flags: 0 dev.cxl.0.tc.2.state: 0 dev.cxl.0.tc.1.params: uninitialized dev.cxl.0.tc.1.refcount: 0 dev.cxl.0.tc.1.flags: 0 dev.cxl.0.tc.1.state: 0 dev.cxl.0.tc.0.params: uninitialized dev.cxl.0.tc.0.refcount: 0 dev.cxl.0.tc.0.flags: 0 dev.cxl.0.tc.0.state: 0 dev.cxl.0.tc.burstsize: 0 dev.cxl.0.tc.pktsize: 0 dev.cxl.0.rx_c_chan: 0 dev.cxl.0.rx_e_chan_map: 1 dev.cxl.0.mps_bg_map: 1 dev.cxl.0.max_speed: 10 dev.cxl.0.lpacaps: 458752 dev.cxl.0.acaps: 268435460 dev.cxl.0.pcaps: 269418502 dev.cxl.0.rcaps: 270532612 dev.cxl.0.force_fec: -1 dev.cxl.0.autoneg: -1 dev.cxl.0.module_fec: n/a dev.cxl.0.requested_fec: 20 dev.cxl.0.link_fec: 4 dev.cxl.0.pause_settings: 0 dev.cxl.0.linkdnrc: n/a dev.cxl.0.qsize_txq: 1024 dev.cxl.0.qsize_rxq: 1024 dev.cxl.0.holdoff_pktc_idx: -1 dev.cxl.0.holdoff_tmr_idx: 1 dev.cxl.0.tx_vm_wr: 0 dev.cxl.0.rsrv_noflowq: 0 dev.cxl.0.rss_size: 128 dev.cxl.0.rss_base: 0 dev.cxl.0.first_txq: 0 dev.cxl.0.first_rxq: 0 dev.cxl.0.ntxq: 4 dev.cxl.0.nrxq: 4 dev.cxl.0.viid: 2372 dev.cxl.0.txq.3.vxlan_txcsum: 0 dev.cxl.0.txq.3.vxlan_tso_wrs: 0 dev.cxl.0.txq.3.raw_wrs: 0 dev.cxl.0.txq.3.txpkts_flush: 24 dev.cxl.0.txq.3.txpkts1_pkts: 919 dev.cxl.0.txq.3.txpkts0_pkts: 1881 dev.cxl.0.txq.3.txpkts1_wrs: 154 dev.cxl.0.txq.3.txpkts0_wrs: 246 dev.cxl.0.txq.3.txpkt_wrs: 4638655 dev.cxl.0.txq.3.sgl_wrs: 2407744 dev.cxl.0.txq.3.imm_wrs: 2231313 dev.cxl.0.txq.3.tso_wrs: 0 dev.cxl.0.txq.3.vlan_insertion: 0 dev.cxl.0.txq.3.txcsum: 662 dev.cxl.0.txq.3.tc: -1 dev.cxl.0.txq.3.mp_ring.cons_idle2: 378 dev.cxl.0.txq.3.mp_ring.cons_idle: 4631431 dev.cxl.0.txq.3.mp_ring.stalls: 0 dev.cxl.0.txq.3.mp_ring.abdications: 0 dev.cxl.0.txq.3.mp_ring.not_consumer: 9659 dev.cxl.0.txq.3.mp_ring.takeovers: 0 dev.cxl.0.txq.3.mp_ring.consumer3: 376 dev.cxl.0.txq.3.mp_ring.consumer2: 2 dev.cxl.0.txq.3.mp_ring.fast_consumer: 4631421 dev.cxl.0.txq.3.mp_ring.consumed: 4641458 dev.cxl.0.txq.3.mp_ring.dropped: 0 dev.cxl.0.txq.3.mp_ring.state: 463863545964 dev.cxl.0.txq.3.sidx: 1023 dev.cxl.0.txq.3.pidx: 973 dev.cxl.0.txq.3.cidx: 951 dev.cxl.0.txq.3.cntxt_id: 27 dev.cxl.0.txq.3.abs_id: 27 dev.cxl.0.txq.3.dmalen: 65536 dev.cxl.0.txq.3.ba: 304218112 dev.cxl.0.txq.2.vxlan_txcsum: 0 dev.cxl.0.txq.2.vxlan_tso_wrs: 0 dev.cxl.0.txq.2.raw_wrs: 0 dev.cxl.0.txq.2.txpkts_flush: 24 dev.cxl.0.txq.2.txpkts1_pkts: 926 dev.cxl.0.txq.2.txpkts0_pkts: 1761 dev.cxl.0.txq.2.txpkts1_wrs: 129 dev.cxl.0.txq.2.txpkts0_wrs: 235 dev.cxl.0.txq.2.txpkt_wrs: 4646429 dev.cxl.0.txq.2.sgl_wrs: 3128918 dev.cxl.0.txq.2.imm_wrs: 1517875 dev.cxl.0.txq.2.tso_wrs: 0 dev.cxl.0.txq.2.vlan_insertion: 0 dev.cxl.0.txq.2.txcsum: 5925 dev.cxl.0.txq.2.tc: -1 dev.cxl.0.txq.2.mp_ring.cons_idle2: 458 dev.cxl.0.txq.2.mp_ring.cons_idle: 4639888 dev.cxl.0.txq.2.mp_ring.stalls: 0 dev.cxl.0.txq.2.mp_ring.abdications: 0 dev.cxl.0.txq.2.mp_ring.not_consumer: 8775 dev.cxl.0.txq.2.mp_ring.takeovers: 0 dev.cxl.0.txq.2.mp_ring.consumer3: 459 dev.cxl.0.txq.2.mp_ring.consumer2: 0 dev.cxl.0.txq.2.mp_ring.fast_consumer: 4639882 dev.cxl.0.txq.2.mp_ring.consumed: 4649116 dev.cxl.0.txq.2.mp_ring.dropped: 0 dev.cxl.0.txq.2.mp_ring.state: 2594199831132 dev.cxl.0.txq.2.sidx: 1023 dev.cxl.0.txq.2.pidx: 31 dev.cxl.0.txq.2.cidx: 26 dev.cxl.0.txq.2.cntxt_id: 26 dev.cxl.0.txq.2.abs_id: 26 dev.cxl.0.txq.2.dmalen: 65536 dev.cxl.0.txq.2.ba: 376111104 dev.cxl.0.txq.1.vxlan_txcsum: 0 dev.cxl.0.txq.1.vxlan_tso_wrs: 0 dev.cxl.0.txq.1.raw_wrs: 0 dev.cxl.0.txq.1.txpkts_flush: 14 dev.cxl.0.txq.1.txpkts1_pkts: 1260 dev.cxl.0.txq.1.txpkts0_pkts: 2118 dev.cxl.0.txq.1.txpkts1_wrs: 195 dev.cxl.0.txq.1.txpkts0_wrs: 266 dev.cxl.0.txq.1.txpkt_wrs: 4664613 dev.cxl.0.txq.1.sgl_wrs: 2593073 dev.cxl.0.txq.1.imm_wrs: 2072003 dev.cxl.0.txq.1.tso_wrs: 0 dev.cxl.0.txq.1.vlan_insertion: 0 dev.cxl.0.txq.1.txcsum: 598 dev.cxl.0.txq.1.tc: -1 dev.cxl.0.txq.1.mp_ring.cons_idle2: 519 dev.cxl.0.txq.1.mp_ring.cons_idle: 4654682 dev.cxl.0.txq.1.mp_ring.stalls: 0 dev.cxl.0.txq.1.mp_ring.abdications: 0 dev.cxl.0.txq.1.mp_ring.not_consumer: 12800 dev.cxl.0.txq.1.mp_ring.takeovers: 0 dev.cxl.0.txq.1.mp_ring.consumer3: 516 dev.cxl.0.txq.1.mp_ring.consumer2: 3 dev.cxl.0.txq.1.mp_ring.fast_consumer: 4654681 dev.cxl.0.txq.1.mp_ring.consumed: 4668000 dev.cxl.0.txq.1.mp_ring.dropped: 0 dev.cxl.0.txq.1.mp_ring.state: 219046674483 dev.cxl.0.txq.1.sidx: 1023 dev.cxl.0.txq.1.pidx: 1001 dev.cxl.0.txq.1.cidx: 1000 dev.cxl.0.txq.1.cntxt_id: 25 dev.cxl.0.txq.1.abs_id: 25 dev.cxl.0.txq.1.dmalen: 65536 dev.cxl.0.txq.1.ba: 304152576 dev.cxl.0.txq.0.vxlan_txcsum: 0 dev.cxl.0.txq.0.vxlan_tso_wrs: 0 dev.cxl.0.txq.0.raw_wrs: 0 dev.cxl.0.txq.0.txpkts_flush: 21 dev.cxl.0.txq.0.txpkts1_pkts: 717 dev.cxl.0.txq.0.txpkts0_pkts: 1328 dev.cxl.0.txq.0.txpkts1_wrs: 121 dev.cxl.0.txq.0.txpkts0_wrs: 172 dev.cxl.0.txq.0.txpkt_wrs: 4028347 dev.cxl.0.txq.0.sgl_wrs: 2240846 dev.cxl.0.txq.0.imm_wrs: 1787797 dev.cxl.0.txq.0.tso_wrs: 0 dev.cxl.0.txq.0.vlan_insertion: 0 dev.cxl.0.txq.0.txcsum: 2122 dev.cxl.0.txq.0.tc: -1 dev.cxl.0.txq.0.mp_ring.cons_idle2: 401 dev.cxl.0.txq.0.mp_ring.cons_idle: 4023203 dev.cxl.0.txq.0.mp_ring.stalls: 0 dev.cxl.0.txq.0.mp_ring.abdications: 0 dev.cxl.0.txq.0.mp_ring.not_consumer: 6799 dev.cxl.0.txq.0.mp_ring.takeovers: 0 dev.cxl.0.txq.0.mp_ring.consumer3: 400 dev.cxl.0.txq.0.mp_ring.consumer2: 1 dev.cxl.0.txq.0.mp_ring.fast_consumer: 4023202 dev.cxl.0.txq.0.mp_ring.consumed: 4030402 dev.cxl.0.txq.0.mp_ring.dropped: 0 dev.cxl.0.txq.0.mp_ring.state: 3457501430565 dev.cxl.0.txq.0.sidx: 1023 dev.cxl.0.txq.0.pidx: 126 dev.cxl.0.txq.0.cidx: 118 dev.cxl.0.txq.0.cntxt_id: 24 dev.cxl.0.txq.0.abs_id: 24 dev.cxl.0.txq.0.dmalen: 65536 dev.cxl.0.txq.0.ba: 358457344 dev.cxl.0.rxq.3.vxlan_rxcsum: 0 dev.cxl.0.rxq.3.vlan_extraction: 0 dev.cxl.0.rxq.3.rxcsum: 8468401 dev.cxl.0.rxq.3.lro_flushed: 0 dev.cxl.0.rxq.3.lro_queued: 0 dev.cxl.0.rxq.3.cidx: 757 dev.cxl.0.rxq.3.cntxt_id: 12 dev.cxl.0.rxq.3.abs_id: 12 dev.cxl.0.rxq.3.dmalen: 65536 dev.cxl.0.rxq.3.ba: 358350848 dev.cxl.0.rxq.3.fl.cluster_fast_recycled: 831 dev.cxl.0.rxq.3.fl.cluster_recycled: 2780573 dev.cxl.0.rxq.3.fl.cluster_allocated: 219644 dev.cxl.0.rxq.3.fl.pidx: 800 dev.cxl.0.rxq.3.fl.rx_offset: 1792 dev.cxl.0.rxq.3.fl.cidx: 808 dev.cxl.0.rxq.3.fl.packing: 1 dev.cxl.0.rxq.3.fl.padding: 1 dev.cxl.0.rxq.3.fl.cntxt_id: 23 dev.cxl.0.rxq.3.fl.dmalen: 8192 dev.cxl.0.rxq.3.fl.ba: 358416384 dev.cxl.0.rxq.2.vxlan_rxcsum: 0 dev.cxl.0.rxq.2.vlan_extraction: 0 dev.cxl.0.rxq.2.rxcsum: 7525378 dev.cxl.0.rxq.2.lro_flushed: 0 dev.cxl.0.rxq.2.lro_queued: 0 dev.cxl.0.rxq.2.cidx: 956 dev.cxl.0.rxq.2.cntxt_id: 11 dev.cxl.0.rxq.2.abs_id: 11 dev.cxl.0.rxq.2.dmalen: 65536 dev.cxl.0.rxq.2.ba: 358260736 dev.cxl.0.rxq.2.fl.cluster_fast_recycled: 1606 dev.cxl.0.rxq.2.fl.cluster_recycled: 2356578 dev.cxl.0.rxq.2.fl.cluster_allocated: 203736 dev.cxl.0.rxq.2.fl.pidx: 584 dev.cxl.0.rxq.2.fl.rx_offset: 1536 dev.cxl.0.rxq.2.fl.cidx: 596 dev.cxl.0.rxq.2.fl.packing: 1 dev.cxl.0.rxq.2.fl.padding: 1 dev.cxl.0.rxq.2.fl.cntxt_id: 22 dev.cxl.0.rxq.2.fl.dmalen: 8192 dev.cxl.0.rxq.2.fl.ba: 358326272 dev.cxl.0.rxq.1.vxlan_rxcsum: 0 dev.cxl.0.rxq.1.vlan_extraction: 0 dev.cxl.0.rxq.1.rxcsum: 8143582 dev.cxl.0.rxq.1.lro_flushed: 0 dev.cxl.0.rxq.1.lro_queued: 0 dev.cxl.0.rxq.1.cidx: 318 dev.cxl.0.rxq.1.cntxt_id: 10 dev.cxl.0.rxq.1.abs_id: 10 dev.cxl.0.rxq.1.dmalen: 65536 dev.cxl.0.rxq.1.ba: 358170624 dev.cxl.0.rxq.1.fl.cluster_fast_recycled: 951 dev.cxl.0.rxq.1.fl.cluster_recycled: 2598029 dev.cxl.0.rxq.1.fl.cluster_allocated: 200964 dev.cxl.0.rxq.1.fl.pidx: 864 dev.cxl.0.rxq.1.fl.rx_offset: 1536 dev.cxl.0.rxq.1.fl.cidx: 879 dev.cxl.0.rxq.1.fl.packing: 1 dev.cxl.0.rxq.1.fl.padding: 1 dev.cxl.0.rxq.1.fl.cntxt_id: 21 dev.cxl.0.rxq.1.fl.dmalen: 8192 dev.cxl.0.rxq.1.fl.ba: 358236160 dev.cxl.0.rxq.0.vxlan_rxcsum: 0 dev.cxl.0.rxq.0.vlan_extraction: 0 dev.cxl.0.rxq.0.rxcsum: 6572711 dev.cxl.0.rxq.0.lro_flushed: 0 dev.cxl.0.rxq.0.lro_queued: 0 dev.cxl.0.rxq.0.cidx: 26 dev.cxl.0.rxq.0.cntxt_id: 9 dev.cxl.0.rxq.0.abs_id: 9 dev.cxl.0.rxq.0.dmalen: 65536 dev.cxl.0.rxq.0.ba: 358096896 dev.cxl.0.rxq.0.fl.cluster_fast_recycled: 2183 dev.cxl.0.rxq.0.fl.cluster_recycled: 2054949 dev.cxl.0.rxq.0.fl.cluster_allocated: 154708 dev.cxl.0.rxq.0.fl.pidx: 8 dev.cxl.0.rxq.0.fl.rx_offset: 1536 dev.cxl.0.rxq.0.fl.cidx: 21 dev.cxl.0.rxq.0.fl.packing: 1 dev.cxl.0.rxq.0.fl.padding: 1 dev.cxl.0.rxq.0.fl.cntxt_id: 20 dev.cxl.0.rxq.0.fl.dmalen: 8192 dev.cxl.0.rxq.0.fl.ba: 358162432 dev.cxl.0.%parent: t5nex0 dev.cxl.0.%pnpinfo: dev.cxl.0.%location: port=0 dev.cxl.0.%driver: cxl dev.cxl.0.%desc: port 0 hw.cxgbe.tx_coalesce_gap: 5 hw.cxgbe.tx_coalesce_pkts: 32 hw.cxgbe.tx_coalesce: 1 hw.cxgbe.defrags: 0 hw.cxgbe.pullups: 11 hw.cxgbe.lro_mbufs: 0 hw.cxgbe.lro_entries: 8 hw.cxgbe.tscale: 1 hw.cxgbe.safest_rx_cluster: 4096 hw.cxgbe.largest_rx_cluster: 16384 hw.cxgbe.fl_pack: -1 hw.cxgbe.buffer_packing: -1 hw.cxgbe.cong_drop: 0 hw.cxgbe.spg_len: 64 hw.cxgbe.fl_pad: -1 hw.cxgbe.fl_pktshift: 0 hw.cxgbe.nm_txcsum: 0 hw.cxgbe.nm_split_rss: 0 hw.cxgbe.lazy_tx_credit_flush: 1 hw.cxgbe.starve_fl: 0 hw.cxgbe.nm_cong_drop: 1 hw.cxgbe.nm_holdoff_tmr_idx: 2 hw.cxgbe.nm_rx_nframes: 64 hw.cxgbe.nm_rx_ndesc: 256 hw.cxgbe.nm_black_hole: 0 hw.cxgbe.tls.combo_wrs: 0 hw.cxgbe.tls.inline_keys: 0 hw.cxgbe.kern_tls: 0 hw.cxgbe.cop_managed_offloading: 0 hw.cxgbe.drop_pkts_with_l4_errors: 0 hw.cxgbe.drop_pkts_with_l3_errors: 1 hw.cxgbe.drop_pkts_with_l2_errors: 1 hw.cxgbe.drop_ip_fragments: 0 hw.cxgbe.attack_filter: 1 hw.cxgbe.tx_vm_wr: 0 hw.cxgbe.reset_on_fatal_err: 0 hw.cxgbe.panic_on_fatal_err: 0 hw.cxgbe.pcie_relaxed_ordering: 0 hw.cxgbe.num_vis: 1 hw.cxgbe.fcoecaps_allowed: 0 hw.cxgbe.iscsicaps_allowed: 0 hw.cxgbe.cryptocaps_allowed: -1 hw.cxgbe.rdmacaps_allowed: 0 hw.cxgbe.toecaps_allowed: 0 hw.cxgbe.niccaps_allowed: 33 hw.cxgbe.switchcaps_allowed: 3 hw.cxgbe.linkcaps_allowed: 0 hw.cxgbe.nbmcaps_allowed: 0 hw.cxgbe.fw_install: 1 hw.cxgbe.autoneg: -1 hw.cxgbe.force_fec: -1 hw.cxgbe.fec: -1 hw.cxgbe.pause_settings: 0 hw.cxgbe.config_file: default hw.cxgbe.interrupt_types: 7 hw.cxgbe.qsize_rxq: 1024 hw.cxgbe.qsize_txq: 1024 hw.cxgbe.holdoff_pktc_idx: -1 hw.cxgbe.holdoff_timer_idx: 1 hw.cxgbe.nnmrxq_vi: 2 hw.cxgbe.nnmtxq_vi: 2 hw.cxgbe.nnmrxq: 8 hw.cxgbe.nnmtxq: 8 hw.cxgbe.native_netmap: 2 hw.cxgbe.holdoff_pktc_idx_ofld: -1 hw.cxgbe.holdoff_timer_idx_ofld: 1 hw.cxgbe.nofldrxq_vi: 1 hw.cxgbe.nofldtxq_vi: 1 hw.cxgbe.nofldrxq: 2 hw.cxgbe.nofldtxq: 8 hw.cxgbe.rsrv_noflowq: 0 hw.cxgbe.nrxq_vi: 1 hw.cxgbe.ntxq_vi: 1 hw.cxgbe.nrxq: 4 hw.cxgbe.ntxq: 4 hw.cxgbe.toe.tls_rx_timeout: 5 hw.cxgbe.toe.rexmt_backoff.15: -1 hw.cxgbe.toe.rexmt_backoff.14: -1 hw.cxgbe.toe.rexmt_backoff.13: -1 hw.cxgbe.toe.rexmt_backoff.12: -1 hw.cxgbe.toe.rexmt_backoff.11: -1 hw.cxgbe.toe.rexmt_backoff.10: -1 hw.cxgbe.toe.rexmt_backoff.9: -1 hw.cxgbe.toe.rexmt_backoff.8: -1 hw.cxgbe.toe.rexmt_backoff.7: -1 hw.cxgbe.toe.rexmt_backoff.6: -1 hw.cxgbe.toe.rexmt_backoff.5: -1 hw.cxgbe.toe.rexmt_backoff.4: -1 hw.cxgbe.toe.rexmt_backoff.3: -1 hw.cxgbe.toe.rexmt_backoff.2: -1 hw.cxgbe.toe.rexmt_backoff.1: -1 hw.cxgbe.toe.rexmt_backoff.0: -1 hw.cxgbe.toe.rexmt_count: 0 hw.cxgbe.toe.rexmt_max: 0 hw.cxgbe.toe.rexmt_min: 0 hw.cxgbe.toe.keepalive_count: 0 hw.cxgbe.toe.keepalive_interval: 0 hw.cxgbe.toe.keepalive_idle: 0 hw.cxgbe.clip_db_auto: 1