From nobody Fri Oct 21 17:31:36 2022 X-Original-To: freebsd-performance@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MvBNF3G49z4gGyS for ; Fri, 21 Oct 2022 17:31:41 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MvBND0DlPz47T5 for ; Fri, 21 Oct 2022 17:31:40 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: by mail-pj1-x1029.google.com with SMTP id ez6so3031567pjb.1 for ; Fri, 21 Oct 2022 10:31:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=Jyc73eCl8GweAqmQtRLzwAEUayVcC35JhOpQ6yuzTVg=; b=j2RCn7/YNwdxAYWffq4TfR74AZr/3TzMGPhK84MvQR4yQ/a5IR+PwPQnZKOHub5NRu WR118DEMlnwOXTLy1EJyvV7E7Mrz9we4pK6drL3X1jA4BcdRbICAaMmbGfhMoHHZEX9F VL6x0QYs0pZJRXNay20mPp7i75YFW80iXIUCFMuPKueHPknGWOvsVQG+lsx6phdlamR/ PTuqBIj0cWFTyyNKPFCNzgdI/GKwf0ctGWOPYXX11vuqTMsuHbgH3KfZRJLdk0lOJE57 ge1nmNwDqYYKqLIQ4Z/1yKtI5j7B4uoqUmUsMGnk1XM+qh7YdJPiECxOCwHoaNp1BdrZ MF7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Jyc73eCl8GweAqmQtRLzwAEUayVcC35JhOpQ6yuzTVg=; b=LdW8ojnmFstT2cEPKGGxKBEc4nTeuViGN4Hq2gGNFhYMf2mDeR5IVCGMUZi+BSxGW2 M2i2uxBhvcibfn5jWMxCf61qUOASZuTRiJb+7nsjQnXTeVl+cTPN+Pcg7YrTwMji7sKj jadrvNgNQeL36pO4JSbOok52JaooHn9MYyIJVbZcwQOgzd9VMCohn0TxU69NpsPHUVEG Nk9/3D6PvszA1C/XFMi8VgxQcWHAgrlgbwPr6/gnkX+LzrV12+NiY7ZVPv+X2GX9AKq9 YC4nmS/TUINUVOfCdGPl28ANT4MBkEZIKmTTLWfeAmKdCKg3tF6djhgJVueDSAx5KViS 7NEw== X-Gm-Message-State: ACrzQf2h10M5rdV4px0WUlStEa5B8DJ+v8BCnDKvJYUQnhmWxyLN35+b TZn1jmaw4zpoy3yxgoM9LfNesrakdiw= X-Google-Smtp-Source: AMsMyM4Okgr2yCYff7aX8BtXjAEae7p1QrZu2eUiRPsNd0RtaWg8BjKKc+nBKiGuh+7ZhGCMpEXrQA== X-Received: by 2002:a17:90b:1c8c:b0:203:89fb:ba79 with SMTP id oo12-20020a17090b1c8c00b0020389fbba79mr59114145pjb.92.1666373498504; Fri, 21 Oct 2022 10:31:38 -0700 (PDT) Received: from [10.192.161.10] (stargate.chelsio.com. [12.32.117.8]) by smtp.googlemail.com with ESMTPSA id a188-20020a6366c5000000b00460ea630c1bsm13661433pgc.46.2022.10.21.10.31.37 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Oct 2022 10:31:38 -0700 (PDT) Message-ID: <92cdf4b8-2209-ec44-8151-a59b9e8f1504@gmail.com> Date: Fri, 21 Oct 2022 10:31:36 -0700 List-Id: Performance/tuning List-Archive: https://lists.freebsd.org/archives/freebsd-performance List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-performance@freebsd.org X-BeenThere: freebsd-performance@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.3.2 Subject: Re: Chelsio Forwarding performance and RELENG_13 vs RELENG_12 Content-Language: en-US To: mike tancsa , Freebsd performance References: <7b86e3fe-62e4-7b3e-f4bf-30e4894db9db@sentex.net> From: Navdeep Parhar In-Reply-To: <7b86e3fe-62e4-7b3e-f4bf-30e4894db9db@sentex.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4MvBND0DlPz47T5 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b="j2RCn7/Y"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of nparhar@gmail.com designates 2607:f8b0:4864:20::1029 as permitted sender) smtp.mailfrom=nparhar@gmail.com X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-performance@freebsd.org]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1029:from]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-performance@freebsd.org] X-ThisMailContainsUnwantedMimeParts: N On 10/18/22 12:16 PM, mike tancsa wrote: > I updated a RELENG_12 router along with the hardware to RELENG_13 (oct > 14th kernel) and was surprised to see an increase in > dev.cxl.0.stats.rx_ovflow0 at a somewhat faster rate than I was seeing > on the older slightly slower hardware under about the same load. > (Xeon(R) E-2226G CPU @ 3.40GHz) vs a 4 core Xeon same freq, same memory > speed. About 150Kpps in and out and a 1Gb/s throughput > > loader.conf is the same > > > hw.cxgbe.toecaps_allowed="0" > hw.cxgbe.rdmacaps_allowed="0" > hw.cxgbe.iscsicaps_allowed="0" > hw.cxgbe.fcoecaps_allowed="0" > hw.cxgbe.pause_settings="0" > hw.cxgbe.attack_filter="1" > hw.cxgbe.drop_pkts_with_l3_errors="1" > > As there is a large routing table, I do have > > [fib_algo] inet.0 (radix4_lockless#46) rebuild_fd_flm: switching algo to > radix4 > [fib_algo] inet6.0 (radix6_lockless#58) rebuild_fd_flm: switching algo > to radix6 > > kicking in. > > and sysctl.conf > > net.route.multipath=0 > > net.inet.ip.redirect=0 > net.inet6.ip6.redirect=0 > kern.ipc.maxsockbuf=16777216 > net.inet.tcp.blackhole=1 > > Are there any other tweaks that can be done in order to better > forwarding performance ? I do see at bootup time > > cxl0: nrxq (6), hw RSS table size (128); expect uneven traffic > distribution. > cxl1: nrxq (6), hw RSS table size (128); expect uneven traffic > distribution. > cxl3: nrxq (6), hw RSS table size (128); expect uneven traffic > distribution. > > The cpu is 6 core. No HT enabled The old system was 4-core so it must have used 4 queues. Can you please try that on the new system and see how it does? hw.cxgbe.ntxq=4 hw.cxgbe.nrxq=4 Regards, Navdeep