From nobody Tue Oct 18 19:16:00 2022 X-Original-To: freebsd-performance@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MsNrM2kwCz4fGsc for ; Tue, 18 Oct 2022 19:16:19 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MsNrL0yvDz3nl4; Tue, 18 Oct 2022 19:16:18 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.16.1/8.16.1) with ESMTPS id 29IJG0cx083114 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Tue, 18 Oct 2022 15:16:01 -0400 (EDT) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:5132:67e3:2e60:df9c] ([IPv6:2607:f3e0:0:4:5132:67e3:2e60:df9c]) by pyroxene2a.sentex.ca (8.16.1/8.15.2) with ESMTPS id 29IJG0aT074463 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Tue, 18 Oct 2022 15:16:00 -0400 (EDT) (envelope-from mike@sentex.net) Message-ID: <7b86e3fe-62e4-7b3e-f4bf-30e4894db9db@sentex.net> Date: Tue, 18 Oct 2022 15:16:00 -0400 List-Id: Performance/tuning List-Archive: https://lists.freebsd.org/archives/freebsd-performance List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-performance@freebsd.org X-BeenThere: freebsd-performance@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.3.3 Content-Language: en-US To: Freebsd performance From: mike tancsa Subject: Chelsio Forwarding performance and RELENG_13 vs RELENG_12 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.84 on 64.7.153.18 X-Rspamd-Queue-Id: 4MsNrL0yvDz3nl4 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:1::12 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-3.40 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; RCVD_IN_DNSWL_LOW(-0.10)[199.212.134.19:received]; MIME_GOOD(-0.10)[text/plain]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-performance@freebsd.org]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_THREE(0.00)[3]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[mike]; TO_DN_ALL(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[sentex.net]; ARC_NA(0.00)[] X-ThisMailContainsUnwantedMimeParts: N I updated a RELENG_12 router along with the hardware to RELENG_13 (oct 14th kernel) and was surprised to see an increase in dev.cxl.0.stats.rx_ovflow0 at a somewhat faster rate than I was seeing on the older slightly slower hardware under about the same load.  (Xeon(R) E-2226G CPU @ 3.40GHz) vs a 4 core Xeon same freq, same memory speed. About 150Kpps in and out and a 1Gb/s throughput loader.conf is the same hw.cxgbe.toecaps_allowed="0" hw.cxgbe.rdmacaps_allowed="0" hw.cxgbe.iscsicaps_allowed="0" hw.cxgbe.fcoecaps_allowed="0" hw.cxgbe.pause_settings="0" hw.cxgbe.attack_filter="1" hw.cxgbe.drop_pkts_with_l3_errors="1" As there is a large routing table, I do have [fib_algo] inet.0 (radix4_lockless#46) rebuild_fd_flm: switching algo to radix4 [fib_algo] inet6.0 (radix6_lockless#58) rebuild_fd_flm: switching algo to radix6 kicking in. and sysctl.conf net.route.multipath=0 net.inet.ip.redirect=0 net.inet6.ip6.redirect=0 kern.ipc.maxsockbuf=16777216 net.inet.tcp.blackhole=1 Are there any other tweaks that can be done in order to better forwarding performance ? I do see at bootup time cxl0: nrxq (6), hw RSS table size (128); expect uneven traffic distribution. cxl1: nrxq (6), hw RSS table size (128); expect uneven traffic distribution. cxl3: nrxq (6), hw RSS table size (128); expect uneven traffic distribution. The cpu is 6 core. No HT enabled real memory  = 34359738368 (32768 MB) avail memory = 33238708224 (31698 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: < > FreeBSD/SMP: Multiprocessor System Detected: 6 CPUs FreeBSD/SMP: 1 package(s) x 6 core(s) random: registering fast source Intel Secure Key RNG just a handful of ipfw rules (no states) that were the same as before and a dozen or so cxgbe firewall rules in the NIC Anything I can try / look at that might be causing the odd overflow on cxl0 ? Its a T540-CR with 3 ports in use. t5nex0@pci0:2:0:4:      class=0x020000 rev=0x00 hdr=0x00 vendor=0x1425 device=0x5403 subvendor=0x1425 subdevice=0x0000     vendor     = 'Chelsio Communications Inc'     device     = 'T540-CR Unified Wire Ethernet Controller'     class      = network     subclass   = ethernet     bar   [10] = type Memory, range 64, base 0x91300000, size 524288, enabled     bar   [18] = type Memory, range 64, base 0x90000000, size 16777216, enabled     bar   [20] = type Memory, range 64, base 0x91984000, size 8192, enabled     cap 01[40] = powerspec 3  supports D0 D3  current D0     cap 05[50] = MSI supports 32 messages, 64 bit, vector masks     cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR                  max read 4096                  link x8(x8) speed 8.0(8.0) ASPM L0s/L1(L0s/L1)     cap 11[b0] = MSI-X supports 128 messages, enabled                  Table in map 0x20[0x0], PBA in map 0x20[0x1000]     cap 03[d0] = VPD     ecap 0001[100] = AER 2 0 fatal 0 non-fatal 5 corrected     ecap 0003[170] = Serial 1 0000000000000000     ecap 000e[190] = ARI 1     ecap 0019[1a0] = PCIe Sec 1 lane errors 0     ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled                      0 VFs configured out of 0 supported                      First VF RID Offset 0x0008, VF RID Stride 0x0004                      VF Device ID 0x5803                      Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304     ecap 0017[200] = TPH Requester 1 Thanks for any suggestions     ---Mike