From nobody Mon Jul 3 15:48:01 2023 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qvr160FFjz4lYTM for ; Mon, 3 Jul 2023 15:48:10 +0000 (UTC) (envelope-from Josef.Zahner1@swisscom.com) Received: from mail.swisscom.com (mailout120.swisscom.com [138.188.166.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.swisscom.com", Issuer "SwissSign RSA TLS OV ICA 2021 - 1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Qvr146hXnz4HLq; Mon, 3 Jul 2023 15:48:08 +0000 (UTC) (envelope-from Josef.Zahner1@swisscom.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of Josef.Zahner1@swisscom.com designates 138.188.166.120 as permitted sender) smtp.mailfrom=Josef.Zahner1@swisscom.com; dmarc=none; arc=reject ("signature check failed: fail, {[1] = sig:microsoft.com:reject}") Received: by mail.swisscom.com; Mon, 3 Jul 2023 17:48:06 +0200 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-256; boundary="----=_Part_89646_1792331142.1688399286006" X-Mailer: Totemo_TrustMail_(Notification) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Orjl27Zb7x+cZ7cydTsCKBe+9j8zZ4Znn/9oYYS/ESLHrpUk27x+vt8vYjofQTTp+x4u9vYN/qk2T6l846qmraorl00vCFi2rO4EW94h8s9vnFqI1AX+HmIho7Y3ZuVN7385eLWem8fLY/PG1spw6SS2xz9V2keK0IjdVWnanHSxV8x9r9r6kkanngkYCrm39rgH+wcU5OYtNmCDz2FE+5FwRU9v675Aidf1qS2ChWs1SNivmjSXJRk2lsmTygjsneXDu1qdEZTTkFIfVxVltDT7G4JeNCMi/bU4knNc4mmg4po2lKMAAjx29X/JGZLSTGzMLwxTmdxYf//iJgQOZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=NDKUThVlfbStmWMlHmIeeAFw3f+OmIl7V6XXCHn938I=; b=jANa2tIyxN5P4bt/t/7uxTfmpYu9HuJL8OmzNyA7EnT0+w6CRV/Oujix4ZvA3tZN5hEZ3PQMRN6ujD1QBQCKXAXq/BuD1DT1eRE4+B15htFbjtDLzERYZak1Oy7uKzx/tLzWFV1fqiyzWbZtd6wBzfq8m3Yz3/23rXCtUDirzXX5lEihlxf8UkmmeQ+Scf8Jfh+rDgP6Q97OE2rN4eICPqltH979vq5C0fCogI5uK4KZdXEtODJL83NIeWUcpIOt7BZm4zSWfjCTOrhX6NHUn9HsJWW6wjRkvrHkdo4vE9BunroY/si4djDUXGUkfg9wSa9V5DThrzjyXOdA/68eww== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=swisscom.com; dmarc=pass action=none header.from=swisscom.com; dkim=pass header.d=swisscom.com; arc=none From: To: CC: Subject: Re: Chelsio NIC with RSS - Traffic distribution to different Queues Thread-Topic: Chelsio NIC with RSS - Traffic distribution to different Queues Thread-Index: AQHZqBPGF/8DZJWBME+vE7NtdY0QYq+eOzWAgAAIwuOAAN8nAIACmRGPgAP6hgCAAl4aG4AAFJ8DgAAEr8g= Date: Mon, 3 Jul 2023 15:48:01 +0000 Message-ID: References: <8fe00cbc-f218-a587-48d8-1612223ccd49@FreeBSD.org> In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_2e1fccfb-80ca-4fe1-a574-1516544edb53_Enabled=True;MSIP_Label_2e1fccfb-80ca-4fe1-a574-1516544edb53_SiteId=364e5b87-c1c7-420d-9bee-c35d19b557a1;MSIP_Label_2e1fccfb-80ca-4fe1-a574-1516544edb53_SetDate=2023-07-03T13:30:07.1239524Z;MSIP_Label_2e1fccfb-80ca-4fe1-a574-1516544edb53_ContentBits=0;MSIP_Label_2e1fccfb-80ca-4fe1-a574-1516544edb53_Method=Standard x-ms-publictraffictype: Email x-ms-traffictypediagnostic: ZR0P278MB0757:EE_|ZR2P278MB1050:EE_ x-ms-office365-filtering-correlation-id: b1c63c21-7e4f-4714-1faa-08db7bdce218 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 68Up+FiBAq02jAvZ676jkBKJE5X4MiK7b9zAQUSviCRlbQyC8oDW+Fgn4oo4+7MAVJWZ5xiTl5CvQll958n+6gU92Ee+3k5L4oFp/9lqlTOuILHX/9Ef7dgJoONMGM7Z77T51tShv/k6nZjRrOyuqrKdnVA1GuHLOQx+Ys/YcIlALMPiFS1YREoPimn5UJqsia/jl2JeC3fC1jySVUL4LvxKhIDEXWs7nUl3vHslvaS2W5JzgHKkxa/BZ026gHIhI/0sl8sTOalB6gMwm3z0LsfrHH8AIiuZ6yml9wQvvZ36z2VRwtpuaZEDJ0LIPzi1rfU4JzOrvCJHZyhb/iaKnyXha7/Qp6+WVpM1E7dvxHDpg7tME7H7+GF+RylIu9YmVPzcBJsWw4k0JZhh+0eQYazutmNE8QvHpQy9MrNZgupZvWYL8bdqIUZ7Ph69XgHOATp2fthVkzRhO9mYz8xfAepAo5r3UTkJeKGvc9LlHhw2BS5mGV9AM4v4ZS5mfJvJAuxFNi8KwoSSpc8wWmrD1YtbZdFe02ExLNu/PkFtreivR7BfGXVMLozZWU8iJGG3KFH4929hXjhqyjUsQLC/YYJ8tnKQPp4CGsUH2ShKKnuNMEWhdxZSov9eKtucFYzO x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:ZR0P278MB0757.CHEP278.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230028)(4636009)(376002)(346002)(136003)(39860400002)(366004)(396003)(451199021)(26005)(82960400001)(478600001)(9686003)(71200400001)(6506007)(86362001)(10290500003)(186003)(2940100002)(38100700002)(66476007)(64756008)(6916009)(4326008)(66556008)(66946007)(66446008)(83380400001)(450100002)(7696005)(53546011)(76116006)(316002)(122000001)(5660300002)(8676002)(8936002)(52536014)(38070700005)(41300700001)(2906002)(10300500001)(55016003)(33656002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?Windows-1252?Q?2EKIIk36OWUuKx8sYwR9PgyfusyTOZn1yIgVfvaU1evskEmIMLrdOyOI?= =?Windows-1252?Q?fjjE7UHv8WFBH8lK7jPj+R3slSmWHfJwNIjJGAtGwX986ay7yD6jnREH?= =?Windows-1252?Q?INKeY/VlgewpVCXYl2I6BdgTTeqeKwif4EC6Gg4gLlolI7/R2YoLtLEa?= =?Windows-1252?Q?fHNbQzxkSF7XGqaR5qcPNdyqGiATh8pCDUga4PW0gxHe+CdNxmcEmiS4?= =?Windows-1252?Q?efD8BGcQmiFJZPogqYMIuxI7KLYr4+3q4qQBcBgBb80IOqO4vcD6NL4B?= =?Windows-1252?Q?D4oFXm1cSWC4yZBd5YWyXWcn/TMP8dMYdjtf5lupG5iLUIlvEP3g7enO?= =?Windows-1252?Q?VUMAXjqAeO0wKJCP9SsoWGWQMWGlOneCiHOnG0PPI7Y/OD/b81tfZTMu?= =?Windows-1252?Q?58NswmoDgTK1IHqaAEqzCsKqNA57GRdN8z8iCDNEoUp49vmpnx9yvfu3?= =?Windows-1252?Q?3yzAIuzYz1m7otTMzE/L/nH9kGm1WUtmOBcOsfx39+gwuOzXqadnXnM4?= =?Windows-1252?Q?/a88MQdu8ASnOCl2TbcIBU5NRRg0q12VxoRxsZBPpqo46ae9JmNiigpU?= =?Windows-1252?Q?O7hmo3wV/OzgbeIUAnpKghWgh+EEzHDXkGBW+0En1eoImyJNYf2BsO07?= =?Windows-1252?Q?D6+10eZA9sCyEQgHvWPkCM44bZhGyM4H+ahNXJ8ZI0S94ErVLcnOjveU?= =?Windows-1252?Q?W09P/AwbGZ+E6NKFSFTcyb4Gx2SYSHf1/m7hVKxqyWHJT56casK+6Pb+?= =?Windows-1252?Q?dNbuhf6a00+0rVk93kRtgyMC4KTXpR4SeBONZKSJZyYfH6ebgmkBNaOc?= =?Windows-1252?Q?0nUUnJVa4/pASlNR9Dc0Fmg0OoKNAUXRvguJyjqayAMspzNunLy4OSUh?= =?Windows-1252?Q?gVmhEhlw1FVKB0cEZ8iTUSfx1/wDTMBb2sFS8hVJNSiUlRTpAJq8oOF2?= =?Windows-1252?Q?DlCsbGO3Dnt0Xzeo2Ode57DNZf+tHdrkMUHzytsH6Y0/xihw3gfSM4to?= =?Windows-1252?Q?tBJSDyAwf96zxMs0HaOpKLhlEGHASLtf6ju5O2A3q3ZOavziHFl1OWA4?= =?Windows-1252?Q?XF7y0l5webZf+fEOUyU/vUUpMOQjULrSuzqtJqWvxvsnvwCHS4llrRmw?= =?Windows-1252?Q?jHRO1ZUspFvoHqzXH1gP/8sgF6+KUsmJRH4CWP5PrSrcfFyjqHZUNZw7?= =?Windows-1252?Q?FWEOOxfd6rNolMajcE+InR3K9A5CG0lMryEam7T8x/9G2o4WWBfur819?= =?Windows-1252?Q?YkloZLEC983Ohh1T23mT7tZHrV2Mekz21TZ8AU2LDUBDqIlp0l7AyxRb?= =?Windows-1252?Q?w1FXSXgY8uBplYsrOUlrMxd+YgtDGwRbsNo1UmWxEAq0g48QVsahyjRG?= =?Windows-1252?Q?unYUwZe0HYVc9VzV1c7WfQwCulnWWPGvpAtRgkCykc0Z0MlBU8FqVP+2?= =?Windows-1252?Q?57P/bliLpLAZtPWW1uy+EpV8j6QL2uqQYS1ZqYbWoz6oODil0vNtPpe9?= =?Windows-1252?Q?MzbIzjoTgkVaTqwrQ3OqutaEfnjZJWCZmC4wyZs9XtObncqcR/Y1KSFs?= =?Windows-1252?Q?n/psiMneDncWy/0ai7sAnJVyA2WgsZWOnODSIwAubu+hG4oZXDs2vPTS?= =?Windows-1252?Q?V5awfg+yLiiLQTGTSi8OeV/BJ8Ab4w++ypW0GinafPHTNqavgJElp85x?= =?Windows-1252?Q?SMenDUN4I006uHeSSAmXx1SRczRY+dRNmB4Kpiya8g3+pUx/HS5K6Wab?= =?Windows-1252?Q?vs6rP7NVvgYP1KsxqS4=3D?= X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: ZR0P278MB0757.CHEP278.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: b1c63c21-7e4f-4714-1faa-08db7bdce218 X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Jul 2023 15:48:01.8922 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 364e5b87-c1c7-420d-9bee-c35d19b557a1 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: FZX5L6v7ixXRlP82OfobRcNOsChN+dY+B9cu93rlcrBioOZ7uBo0WRNE+OMej9Zd7YNARc7M5UxI/VrVtO+wImUcY/9rB49ZnILXM6mz5dU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: ZR2P278MB1050 X-OriginatorOrg: swisscom.com X-CFilter-Loop: Reflected X-Trustmail: processed X-Spamd-Result: default: False [-4.48 / 15.00]; SIGNED_SMIME(-2.00)[]; ARC_REJECT(1.00)[signature check failed: fail, {[1] = sig:microsoft.com:reject}]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; NEURAL_HAM_SHORT(-0.95)[-0.955]; NEURAL_HAM_LONG(-0.94)[-0.938]; R_SPF_ALLOW(-0.20)[+ip4:138.188.166.96/27:c]; RCVD_IN_DNSWL_MED(-0.20)[138.188.166.120:from]; MIME_GOOD(-0.20)[multipart/signed,multipart/alternative,text/plain]; XM_UA_NO_VERSION(0.01)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~,4:~]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_NO_DN(0.00)[]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[swisscom.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; HAS_ATTACHMENT(0.00)[]; ASN(0.00)[asn:3303, ipnet:138.188.0.0/16, country:CH]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4Qvr146hXnz4HLq X-Spamd-Bar: ---- X-ThisMailContainsUnwantedMimeParts: N ------=_Part_89646_1792331142.1688399286006 Content-Type: multipart/alternative; boundary="_000_ZR0P278MB0757E403874283B0311A5242D929AZR0P278MB0757CHEP_" Content-Language: en-GB --_000_ZR0P278MB0757E403874283B0311A5242D929AZR0P278MB0757CHEP_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Sorry for the spam, I do see the values with sysctl now. It seems that Free= BSD always loads the if_cxgbe.ko from /boot/kernel/if_cxgbe.ko. So what I= =92ve done is, I renamed the original file and copied the newly compiled if= _cxgbe.ko from /boot/module to /boot/kernel. Is there a cleaner way to get = it work? Btw. Do I need t5fw_cfg.ko as well, I haven=92t found any document= ation what exactly it does=85 Quiet difficult for someone who isn=92t familiar with FreeBSD at all :-P. So I=92ve retested again. Sadly it does still share the load over all confi= gured CPUs (0-3) for RSS (there are 8 cores plus 8 HT cores per physical CP= U). As I already mentioned, the new sysctl values are now visible, so I thi= nk the driver should be fine. My expectation with that configuration was, t= hat CPU0 shouldn=92t be used for RSS (as all values are smaller than the av= ailable CPUs), but it has been used, so the network traffic does flapping a= s hell. hw.cxgbe.cong_drop=3D"1" hw.cxgbe.nrxq=3D"3" hw.cxgbe.pause_settings=3D"0" hw.cxgbe.rsrv_noflowq=3D"1" hw.cxgbe.rsrv_norssq=3D"1" hw.cxgbe.rx_budget=3D"128" if_cxgbe_load=3D"yes" net.inet.rss.bits=3D"2" net.inet.rss.enabled=3D"1" net.isr.bindthreads=3D"1" net.isr.maxthreads=3D"-1" t5fw_cfg_load=3D"yes" Do I miss anything? Cheers Josef From: Zahner Josef, GSA-REP-LI Date: Monday, 3 July 2023 at 16:50 To: Navdeep Parhar Cc: freebsd-net@freebsd.org Subject: Re: Chelsio NIC with RSS - Traffic distribution to different Queue= s Again me, I tried to build the chelsio driver (if_cxgbe) on my machine. root@fw-94:/usr/src/sys/modules/cxgbe # make ; make install -> no errors so= far and I see the new kernel module in /boot/modules: root@fw-94:/usr/src/sys/modules/cxgbe # ls -alh /boot/modules/ total 4584 drwxr-xr-x 2 root wheel 16B Jul 3 16:37 . drwxr-xr-x 15 root wheel 68B Jul 3 16:37 .. -r-xr-xr-x 1 root wheel 46K Jul 3 16:37 ccr.ko -r-xr-xr-x 1 root wheel 47K Jul 3 16:37 cxgbei.ko -r-xr-xr-x 1 root wheel 11K Jul 3 16:37 if_cc.ko -r-xr-xr-x 1 root wheel 11K Jul 3 16:37 if_ccv.ko -r-xr-xr-x 1 root wheel 746K Jul 3 16:37 if_cxgbe.ko -r-xr-xr-x 1 root wheel 41K Jul 3 16:37 if_cxgbev.ko -r-xr-xr-x 1 root wheel 11K Jul 3 16:37 if_cxl.ko -r-xr-xr-x 1 root wheel 11K Jul 3 16:37 if_cxlv.ko -r-xr-xr-x 1 root wheel 154K Jul 3 16:37 iw_cxgbe.ko -rw-r--r-- 1 root wheel 1.3K Jul 3 16:37 linker.hints -r-xr-xr-x 1 root wheel 131K Jul 3 16:37 t4_tom.ko -r-xr-xr-x 1 root wheel 596K Jul 3 16:37 t4fw_cfg.ko -r-xr-xr-x 1 root wheel 725K Jul 3 16:37 t5fw_cfg.ko -r-xr-xr-x 1 root wheel 777K Jul 3 16:37 t6fw_cfg.ko However still the same result, the new sysctl values are not visible. Any h= elp would be appreciated. Btw.: reboot has been done. Cheers Josef From: Zahner Josef, GSA-REP-LI Date: Monday, 3 July 2023 at 15:45 To: Navdeep Parhar Cc: freebsd-net@freebsd.org Subject: Re: Chelsio NIC with RSS - Traffic distribution to different Queue= s Hi Navdeep Nevermind, I found the following command to apply your patch diff: root@fw-94:/sys/dev/cxgbe # patch < /root/chelsio_rss_patch.diff There was an inconsistency with =93adapter.h=94, your change on line 219 wa= s in my case a few lines further below, so I decided to manually edit those= two changes in the file. However I tried to load the new sysctl values like =93hw.cxgbe.rsrv_norssq= =BB in loader.conf, how can I verify that it works as it should? It seems t= hat they don=92t work as the don=92t show up when I try to print out the ne= w sysctl values. You mentioned that I=92ve to patch the kernel, do you mean= execute the patch diff? Or is there anything else I=92ve to do except to e= xecute your patch diff? What about my driver, do I need to change anything = there? Thank you, cheers Josef --_000_ZR0P278MB0757E403874283B0311A5242D929AZR0P278MB0757CHEP_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable

Sorry for the spam, I do see the values with sysctl = now. It seems that FreeBSD always loads the if_cxgbe.ko from /boot/kernel/i= f_cxgbe.ko. So what I=92ve done is, I renamed the original file and copied the newly compiled if_cxgbe.ko from /boot/mod= ule to /boot/kernel. Is there a cleaner way to get it work? Btw. Do I need t5fw_cfg.ko as well, I haven=92t found any documentation what exactl= y it does=85

 

Quiet difficult for someone who isn=92t familiar wit= h FreeBSD at all :-P.

 

So I=92ve retested again. Sadly it does still share = the load over all configured CPUs (0-3) for RSS (there are 8 cores plus 8 H= T cores per physical CPU). As I already mentioned, the new sysctl values are now visible, so I think the driver sh= ould be fine. My expectation with that configuration was, that CPU0 shouldn= =92t be used for RSS (as all values are smaller than the available CPUs), b= ut it has been used, so the network traffic does flapping as hell.

 

hw.cxgbe.cong_drop=3D"1"

hw.cxgbe.nrxq=3D"3"

hw.cxgbe.pause_settings=3D"0"

hw.cxgbe.rsrv_noflowq=3D"1"

hw.cxgbe.rsrv_norssq=3D"1"

hw.cxgbe.rx_budget=3D"128"

if_cxgbe_load=3D"yes"

net.inet.rss.bits=3D"2"

net.inet.rss.enabled=3D"1"

net.isr.bindthreads=3D"1"

net.isr.maxthreads=3D"-1"

t5fw_cfg_load=3D"yes"

 

Do I miss anything?

 

Cheers Josef

 

 

From: Zahner Josef, GSA-R= EP-LI <Josef.Zahner1@swisscom.com>
Date: Monday, 3 July 2023 at 16:50
To: Navdeep Parhar <np@freebsd.org>
Cc: freebsd-net@freebsd.org <freebsd-net@freebsd.org>
Subject: Re: Chelsio NIC with RSS - Traffic distribution to differen= t Queues

Again me, I tried to build the chelsio driver (if_cx= gbe) on my machine.

 

root@fw-94:/usr/src/sys/modules/cxgbe # make ; make = install -> no errors so far and I see the new kernel module in /boot/mod= ules:

 

root@fw-94:/usr/src/sys/modules/cxgbe # ls -alh /boo= t/modules/

total 4584

drwxr-xr-x   2 root  wheel  = ;  16B Jul  3 16:37 .

drwxr-xr-x  15 root  wheel  &nbs= p; 68B Jul  3 16:37 ..

-r-xr-xr-x   1 root  wheel  = ;  46K Jul  3 16:37 ccr.ko

-r-xr-xr-x   1 root  wheel  = ;  47K Jul  3 16:37 cxgbei.ko

-r-xr-xr-x   1 root  wheel  = ;  11K Jul  3 16:37 if_cc.ko

-r-xr-xr-x   1 root  wheel  = ;  11K Jul  3 16:37 if_ccv.ko

-r-xr-xr-x   1 root  wheel  = ; 746K Jul  3 16:37 if_cxgbe.ko

-r-xr-xr-x   1 root  wheel  = ;  41K Jul  3 16:37 if_cxgbev.ko

-r-xr-xr-x   1 root  wheel  = ;  11K Jul  3 16:37 if_cxl.ko

-r-xr-xr-x   1 root  wheel  = ;  11K Jul  3 16:37 if_cxlv.ko

-r-xr-xr-x   1 root  wheel  = ; 154K Jul  3 16:37 iw_cxgbe.ko

-rw-r--r--   1 root  wheel  = ; 1.3K Jul  3 16:37 linker.hints

-r-xr-xr-x   1 root  wheel  = ; 131K Jul  3 16:37 t4_tom.ko

-r-xr-xr-x   1 root  wheel  = ; 596K Jul  3 16:37 t4fw_cfg.ko

-r-xr-xr-x   1 root  wheel  = ; 725K Jul  3 16:37 t5fw_cfg.ko

-r-xr-xr-x   1 root  wheel  = ; 777K Jul  3 16:37 t6fw_cfg.ko

 

However still the same result, the new sysctl values= are not visible. Any help would be appreciated.

 

Btw.: reboot has been done.

 

Cheers Josef

 

From: Zahner Josef, GSA-R= EP-LI <Josef.Zahner1@swisscom.com>
Date: Monday, 3 July 2023 at 15:45
To: Navdeep Parhar <np@freebsd.org>
Cc: freebsd-net@freebsd.org <freebsd-net@freebsd.org>
Subject: Re: Chelsio NIC with RSS - Traffic distribution to differen= t Queues

Hi N= avdeep

&nbs= p;

Neve= rmind, I found the following command to apply your patch diff:<= /o:p>

root@fw-94:/sys/dev/cxgbe # patch < /roo= t/chelsio_rss_patch.diff

&nbs= p;

Ther= e was an inconsistency with =93adapter.h=94, your change on line 219 was in= my case a few lines further below, so I decided to manually edit those two= changes in the file.

&nbs= p;

Howe= ver I tried to load the new sysctl values like =93hw.cxgbe.rsrv_norssq=BB in loader.conf, how can I verify that it works as it should? It seems that th= ey don=92t work as the don=92t show up when I try to print out the new sysc= tl values. You mentioned that I=92ve to patch the kernel, do you mean execu= te the patch diff? Or is there anything else I=92ve to do except to execute your patch diff? What about my driver,= do I need to change anything there?

&nbs= p;

Than= k you, cheers

Jose= f

&nbs= p;

&nbs= p;

--_000_ZR0P278MB0757E403874283B0311A5242D929AZR0P278MB0757CHEP_-- ------=_Part_89646_1792331142.1688399286006 Content-Type: application/pkcs7-signature; name=smime.p7s; smime-type=signed-data Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCAMIIH KzCCBROgAwIBAgIUTx1IzA+W/yTzA5lqYJox9cE9fx0wDQYJKoZIhvcNAQELBQAwUzELMAkGA1UE BhMCQ0gxFTATBgNVBAoTDFN3aXNzU2lnbiBBRzEtMCsGA1UEAxMkU3dpc3NTaWduIFJTQSBTTUlN RSBMQ1AgSUNBIDIwMjEgLSAyMB4XDTIyMDQyOTA5MDMxMloXDTI1MDQyOTA5MDMxMlowgYMxCzAJ BgNVBAYTAkNIMR4wHAYDVQQKExVTd2lzc2NvbSAoU2Nod2VpeikgQUcxKTAnBgkqhkiG9w0BCQEW Gmpvc2VmLnphaG5lcjFAc3dpc3Njb20uY29tMSkwJwYDVQQDEyBTZWN1cmUgTWFpbDogR2F0ZXdh eSBDZXJ0aWZpY2F0ZTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALWL60L8MKsvXVrM C2QjifRYKIolHpx2KvKSyk75F6eEFVnz19XTfK+kKbkZzc9BMYL9xK78nYEoC8wPyEPwcxrw4i8l SPmhkZ674JEndHrX6qHJOku5+OyIhbLa550XtSZRJSd7l0NnYQWCFtWUAPL/j/ButDdDgp7sScWN D+BmOZw0/FyveEgqeJBAAcxZ5em8/2Ub/xew8LBdwOXYnzG6VIck+jTHv6RhNZ1+RfOvip+3sOwm TsvRL0oC8QGWC86OAq81+tCeYQPoywrybSjqYCwE+ZRb5djg76DvwgjfrVF8grddF6OSwkdUxSlJ USZ50RB4WO23r6YWjYX272ECAwEAAaOCAsQwggLAMCUGA1UdEQQeMByBGmpvc2VmLnphaG5lcjFA c3dpc3Njb20uY29tMA4GA1UdDwEB/wQEAwIEsDATBgNVHSUEDDAKBggrBgEFBQcDBDAdBgNVHQ4E FgQUNxi5osjEbpfWsyrkPqFhIrG5I9gwHwYDVR0jBBgwFoAU+lTAgqb+lr0Ex1+fX4IMPcOVT0cw gf8GA1UdHwSB9zCB9DBHoEWgQ4ZBaHR0cDovL2NybC5zd2lzc3NpZ24ubmV0L0ZBNTRDMDgyQTZG RTk2QkQwNEM3NUY5RjVGODIwQzNEQzM5NTRGNDcwgaiggaWggaKGgZ9sZGFwOi8vZGlyZWN0b3J5 LnN3aXNzc2lnbi5uZXQvQ049RkE1NEMwODJBNkZFOTZCRDA0Qzc1RjlGNUY4MjBDM0RDMzk1NEY0 NyUyQ089U3dpc3NTaWduJTJDQz1DSD9jZXJ0aWZpY2F0ZVJldm9jYXRpb25MaXN0P2Jhc2U/b2Jq ZWN0Q2xhc3M9Y1JMRGlzdHJpYnV0aW9uUG9pbnQwZwYDVR0gBGAwXjBSBghghXQBWQIBCzBGMEQG CCsGAQUFBwIBFjhodHRwczovL3JlcG9zaXRvcnkuc3dpc3NzaWduLmNvbS9Td2lzc1NpZ25fQ1BT X1NNSU1FLnBkZjAIBgYEAI96AQMwgcYGCCsGAQUFBwEBBIG5MIG2MGQGCCsGAQUFBzAChlhodHRw Oi8vc3dpc3NzaWduLm5ldC9jZ2ktYmluL2F1dGhvcml0eS9kb3dubG9hZC9GQTU0QzA4MkE2RkU5 NkJEMDRDNzVGOUY1RjgyMEMzREMzOTU0RjQ3ME4GCCsGAQUFBzABhkJodHRwOi8vb2NzcC5zd2lz c3NpZ24ubmV0L0ZBNTRDMDgyQTZGRTk2QkQwNEM3NUY5RjVGODIwQzNEQzM5NTRGNDcwDQYJKoZI hvcNAQELBQADggIBAKC/bdS77QzqssisgA9nHbH+E0q2i4Ge2o3BxSlFxp/rzqXW0jpOb4ntgqQ5 /JECExksUdzjwqARiT/grBfZedbpQK40nNe3fKoZDQpmF8P8T34Ngf13EtpIm7+IJOXjhBQ6PYeM 1vLTn/g0EUJK3k9fdw3acOUHVQ4dstKP1q9IB5FGtiAIfUtk0m9RT0aJjMqQG1l33ignJrfGZ+R2 X121LjxJ+6gFzWLKklJIfZtqxDGeLIjm5ixHeOrkj7zYily3W16I3iFJnvYL0t1x9UHFp+fWuNUh HiqMK48mLHDzG/ffw3fSgDrnLunlxd4uHz9Io4IFwOKjiOJNxPzuQ6kfibasoKhv/MTFXarfXR7R fAWpOgl9EhxKBJqQkDLZAfCjYQwCU+mThs1p9Sc/JG1IZ2jI3XD4fCHnejMA/RlbINc5uOrPvRys GGPYo5UmPZ4yj4vJRa+U9udZRO5YZrpIto5J8MxZ3Y+cizKUkovzqIVdg2gLj7+Ij4YSRxK5HnoE dN7EPmbfX3hLnAmsG0cft84/o+r2xMQiMXQOmeTI2ZnGk23MF3vyUExkEhcFtAkVup/XiAjcs9St /kD47K7tVNkZGxN0XnaILJt1R6pAGo6Xj9WfGnRX6Ph3PJ+XguEhMhLVEfuuBUuAGFUWX9hovJuM X8C83YJTkvTzbmNAMIIHRzCCBS+gAwIBAgIPLFqp2VT9squWrRe2X4z0MA0GCSqGSIb3DQEBCwUA MFMxCzAJBgNVBAYTAkNIMRUwEwYDVQQKEwxTd2lzc1NpZ24gQUcxLTArBgNVBAMTJFN3aXNzU2ln biBSU0EgU01JTUUgUm9vdCBDQSAyMDIxIC0gMTAeFw0yMTA4MDQxMjEyNTlaFw0zNjA3MzExMjEy NTlaMFMxCzAJBgNVBAYTAkNIMRUwEwYDVQQKEwxTd2lzc1NpZ24gQUcxLTArBgNVBAMTJFN3aXNz U2lnbiBSU0EgU01JTUUgTENQIElDQSAyMDIxIC0gMjCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCC AgoCggIBAL0FGcjUDXE+AwMQtJMWAOvPI9rAByO7UTw4EtNAdcNyY3jrCyd5X41WiKnZfVlD0YgX duPHRour3+KIqj0SwJ0I7UEQ19zvSeC/MVLWg52Z/PKaIoBmVawQlJChzIuV0NZIXTco3xrtGCXf Q8MhREG8FaWkg4LGIIBSAqiAZgG+btlFlnXQFyB8ZT+BrX2OWEYj3FIn2kI+O9xY8kqmJJsb6zQJ H8zhxdpA871kQVjEifuDis0hgFhnCGwiNw+fk60L3/un/cX/aooD2B+8r8LJDXh89U49Ulo8QK58 yzj61GGHZ52kbHiKtBNpVXzmJVTfCM+mLzhMiBSVHCpRaWiPXnwDmvukZUxwj75KXKreu527fX1a OILnuQCKRUevIL6DALhJQBifQKpf/9wCnH732eYtYCZacZ15pGYcwkKYl9fLcOOLxKdW6CBxdsSu eSvmyazWTKnnCINHthUiR1O6IkekXZ3tZn93kQ5PsoF1L+Trv9eBOtTCRbApHZWb/TRvRmnPc3ge D0VOsE/ncDa5yLZ7st/e9HyEUmKOG7TzMa4KVjsqGf7QEFTnRhI4TYGdhawE57fp13DbjvRwz+Ia 0wxpNecDSuDgyxf1M/S9fiJClZkzCo3CpxiTR7AyrLhEMBjvEfgZc2REZPacmeBRFtBLXCBJwXU0 zNFb6tGtAgMBAAGjggIWMIICEjAOBgNVHQ8BAf8EBAMCAQYwEwYDVR0lBAwwCgYIKwYBBQUHAwQw EgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQU+lTAgqb+lr0Ex1+fX4IMPcOVT0cwHwYDVR0j BBgwFoAUCQy/KqIdBCQMsvlACkHCz1pyqoAwgf8GA1UdHwSB9zCB9DBHoEWgQ4ZBaHR0cDovL2Ny bC5zd2lzc3NpZ24ubmV0LzA5MENCRjJBQTIxRDA0MjQwQ0IyRjk0MDBBNDFDMkNGNUE3MkFBODAw gaiggaWggaKGgZ9sZGFwOi8vZGlyZWN0b3J5LnN3aXNzc2lnbi5uZXQvQ049MDkwQ0JGMkFBMjFE MDQyNDBDQjJGOTQwMEE0MUMyQ0Y1QTcyQUE4MCUyQ089U3dpc3NTaWduJTJDQz1DSD9jZXJ0aWZp Y2F0ZVJldm9jYXRpb25MaXN0P2Jhc2U/b2JqZWN0Q2xhc3M9Y1JMRGlzdHJpYnV0aW9uUG9pbnQw HwYDVR0gBBgwFjAKBghghXQBWQIBCzAIBgYEAI96AQMwdAYIKwYBBQUHAQEEaDBmMGQGCCsGAQUF BzAChlhodHRwOi8vc3dpc3NzaWduLm5ldC9jZ2ktYmluL2F1dGhvcml0eS9kb3dubG9hZC8wOTBD QkYyQUEyMUQwNDI0MENCMkY5NDAwQTQxQzJDRjVBNzJBQTgwMA0GCSqGSIb3DQEBCwUAA4ICAQAH Dx6XBIV6CZbsgakZM4/C04La3NPAJCJEDZUL8RYYpySTwDcX1D4zYDyVRhWVEom3fWkB7Nxdo/o4 VbTcg+KJdTvRADn3ycapTHPOOfyukWFmAKPpuLwwl3kwfkLZFFgWuGJF5/HqmnjYstzQAUgXVmqR phrW/PQSBM4v1cTeKdd7f23A8uh7etz9/yB+l6b+zU0gmTZrPDLFmfbbphKtJ52/GF7suI3qA6q9 TuHGkLxtdqIhhXLsMhJfAS33TYZz4zhBuFR/zJD4+IrdB7EYhaK6w47jpyfrvd0b/UoLOZr/UuWn 0PFp5QeXGON556v/Za1ZZk0WGgievGlwrnyh1ollExlAnA2WMxvrqAZvk6l5qc3Sy73f9L0atmAm o3qymnXvVrN8y5DNWqP2DEXLrrIh2WuFlxtr9CLELZ4AnGbl8z89mD1ZdnvfsbyMBuy6hEQfVvGy CUvPEAjBuhju5XAiKbplyJ4pvpjLifpPOl6s2aFTlQeJRlInMhD+q70HXhx+HQIAmjX2nt9G/J4X +nsg0MfRx5NVOrmwXyQW0wzVvP7T+xY5M2p2LyQY7sjuKi4Lf5XgMPHgXWwZxqKAVe1SkuaiXkuy z6L01caBYVw8Wt57GgtoF1yQMmAGTrM4cpVjBl2euDYKi8h6/+66YrRbIJruLHX5INyx3Y9fEjCC BoswggRzoAMCAQICEADeTFUg9tz0AhsPEVT3jRAwDQYJKoZIhvcNAQELBQAwRTELMAkGA1UEBhMC Q0gxFTATBgNVBAoTDFN3aXNzU2lnbiBBRzEfMB0GA1UEAxMWU3dpc3NTaWduIEdvbGQgQ0EgLSBH MjAeFw0yMTA4MDMxMzE0NTVaFw0zNjEwMjMxMzE0NTVaMFMxCzAJBgNVBAYTAkNIMRUwEwYDVQQK EwxTd2lzc1NpZ24gQUcxLTArBgNVBAMTJFN3aXNzU2lnbiBSU0EgU01JTUUgUm9vdCBDQSAyMDIx IC0gMTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBANayuLQ4jya6N8gBI0UWfr5kOIZy ZmFYjSSKWbMMoSqrfruFfGVmcKfpItvuuzL6q7GGP6tIgbir8yrdN8cuC/ar31WtVCAUOreJRG3n 6D+uiCEYjkdWlQDJ7GVuVkcUTa0uJtLUi2zK8zMt+fCbjreGJoHnC56LDHwFpzx6+fCFkJyJZzl1 EbFjsNQjLH3cHyt27QStuhHJB0kN4ygPLhEU0ray/3i4/lpTgCSsC0i6TjIxUeyq/rtELAvX+X2r jdpsqwjd80E9j/VBQVzGzFHKDkQft2qAdlVpUeZM/ReA+7NU7rBKHTOTBnm5YRGs5A5bs93gsSVc t9TTzfR7ngFUK4KQoeHKQ43wQaQcB8DWMxajRUaPhExp/ZNXndPlb8skDDEtA5jCADlEeSKBbeTq /AtkJm78yp4aA0Ttf01N6RGydr2GfXu7VD9RkEfHi/j/TizyCDCMGcEsRzWevatTpCKunwwhGSm9 npvPhNyO0TVLIhCBG2LtwEvTK5AiSR8tIa6Rxd/x1kFUcg7eyjQQ9cmandVcFuTNJbHHqHFGrPhT hReJqyQaOBgyJHPpVa74gGMDb4Sw36CUtalT8Itq9VR55f9bnKJvIuH/QCllbG+OSGkxPoEbO4tY +lsvO2t9ayTwvPKN5ZrmrHjL2IIrABcdeWoJLtZuds8w+9tZAgMBAAGjggFnMIIBYzAPBgNVHRMB Af8EBTADAQH/MB0GA1UdDgQWBBQJDL8qoh0EJAyy+UAKQcLPWnKqgDAfBgNVHSMEGDAWgBRbJXuW pGVRfrg588B4Zl7oOufw7jAOBgNVHQ8BAf8EBAMCAQYwgf8GA1UdHwSB9zCB9DBHoEWgQ4ZBaHR0 cDovL2NybC5zd2lzc3NpZ24ubmV0LzVCMjU3Qjk2QTQ2NTUxN0VCODM5RjNDMDc4NjY1RUU4M0FF N0YwRUUwgaiggaWggaKGgZ9sZGFwOi8vZGlyZWN0b3J5LnN3aXNzc2lnbi5uZXQvQ049NUIyNTdC OTZBNDY1NTE3RUI4MzlGM0MwNzg2NjVFRTgzQUU3RjBFRSUyQ089U3dpc3NTaWduJTJDQz1DSD9j ZXJ0aWZpY2F0ZVJldm9jYXRpb25MaXN0P2Jhc2U/b2JqZWN0Q2xhc3M9Y1JMRGlzdHJpYnV0aW9u UG9pbnQwDQYJKoZIhvcNAQELBQADggIBAAL46l3QisysCAM2VGb/CimG4VSDLDPox2yuEKlUgX8q LYgmraaoNCgPGQapneIClQQyZEwLyvnjfcMTW+iXQI534+OCgYetAWAHw8XhIP23MJc+uhxxdItf Taex/k58CXh6h1/xrKElEZfHLBGckOpyFj0CNam75SPYH1JAt9COXdojvDPpMv24dZ9Dvj1XS4dh 3u2WyHB8frcT4QlAuxkCBw9r3R/SzA5aEhjkwbGcvr4rER7lmsXgoTWx5OGyYq7A6Gx8lof6YN4t iRwUQUA5onfvsBVbAT8ezuYUqZy+gp+xYhffIkO1Mm+3BfwYytp6Q11ltSb+WkGhaXSX8UNRjdx/ 2VeEpx1R8oJtqw5806Pl4MmVBG3yx5134qX4yMW5ZwZvbf3Gf26+xWrbBDbbMG9dvciZ/sRylsy5 y3SLJKkTC3i1Bsr1iyYWc5gdcZWd8/BS6WxVfgUiF9CJPGXtV4B3/NisvbNTjwd7WBN6sefJsPjj yaGR4nTOymgbshvElmCUkNvlCLc+zIh9Z8BV/Chz3hw72s8PHLYI0jM++TySSKBacIgeEBeYenbd YEg+ckU+cGuM60h8WbVWBRIUCkZNAjYJ0WxzIVIn2GvE0nKnTH7bNs7TPctc4u4b3fk6/U0T/w7O UrYWcTOgl8Vf9oIYF7U6m5u2eKsb6/1aMIIFujCCA6KgAwIBAgIJALtAHEP1Xk+wMA0GCSqGSIb3 DQEBBQUAMEUxCzAJBgNVBAYTAkNIMRUwEwYDVQQKEwxTd2lzc1NpZ24gQUcxHzAdBgNVBAMTFlN3 aXNzU2lnbiBHb2xkIENBIC0gRzIwHhcNMDYxMDI1MDgzMDM1WhcNMzYxMDI1MDgzMDM1WjBFMQsw CQYDVQQGEwJDSDEVMBMGA1UEChMMU3dpc3NTaWduIEFHMR8wHQYDVQQDExZTd2lzc1NpZ24gR29s ZCBDQSAtIEcyMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAr+TufoskDhJuqVAtFkQ7 kpJcyrhdhJJCEyq8ZVeCQD5XJM1QiyUqt2/876LQwB8CJEoTlo8jE+YoWACjR8cGp4QjK7u9lit/ VcyLwVcfDmJlD909Vopz2q5+bbqBHH5CjCA12UNNhPqE21Is8w4ndwtrvxEvcnifLtg+5hg3Wipy +dpikJKVyh+c6bM8K8vzARO/Ws/BtQpgvd21mWRTuKCWs2/iJneRjOBiEAKfNA+k1ZIzUd6+jbqE emA8atufK+ze3gE/bk3lUIbLtK/tREDFylqM2tIrfKjuvqblCqoOpd8FUrdVxyJdMmqXl2MT28nb eTZ7hTpKxVKJ+STnnXepgv9VHKVxaSvRAiTysybUa9oEVeXBCsdtMDeQKuSeFDNeFhdVxVu1yzSJ kvGdJo+hB9TGsnhQ2wwMC3wLjEHXuendjIj3o02yMszYF9rNt85mndT9Xv+9lz4pded+p2JYryU0 pUHHPbwNUMoDAw8IWh+Vc3hiv69yFGkOpeUDDniOJihC8AcLYiAQZzlG+qkDzAQ4embvIIO1jEpW jpEA/I5cgt6IoMPiaG59je883WX0XaxR7ySArqpWl2/5rX3aYT+YdzylkbYcjCbaZaIJbcHiVOO5 ykxMgI93e2CaHt+28kgeDrpOVG2Y4OGiGqJ3UM/EY5LsRxmd6+ZrzsECAwEAAaOBrDCBqTAOBgNV HQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUWyV7lqRlUX64OfPAeGZe6Drn 8O4wHwYDVR0jBBgwFoAUWyV7lqRlUX64OfPAeGZe6Drn8O4wRgYDVR0gBD8wPTA7BglghXQBWQEC AQEwLjAsBggrBgEFBQcCARYgaHR0cDovL3JlcG9zaXRvcnkuc3dpc3NzaWduLmNvbS8wDQYJKoZI hvcNAQEFBQADggIBACe645R88a7A3hfm5djV9VSwg/S7zV4Fe0+fdWavPOhWfvxyeDgD2StiGwC5 +OlgzczOUYrHUDFu4Up+GC9pWbY9ZIEr44OE5iKHjn3g7gKZYbge9LgriBIWhMIxkziWMaa5O1M/ wySTVltpkuzFwbs4AOPsF6m43Md8AYOfMke6UiI0HTJ6CVanfCU2qT1L2sCCbwq7EsiHSycR+R4t x5M/nttfJmtS2S6K8RTGRI0Vqbe/vd6mGu6uLftIdxf+u+yvGPUqUfA5hJeVbG4bwyvEdGB5JbAK J9/fXtI5z0V9QkvfsywexcZdylU6oJxpmo/a77KwPJ+HbBIrZXAVUjEaJM9vMSNQH4xPjyPDdEFj HFWoFN0+4FFQz/EbMFYOkrCChdiDyyJkvC24JdVUorgG6q2SpCSgwYa1ShNqR88uC1aVVMvOmttq tKay20EIhid392qgQmwLOM7XdVAyksLfKzAiSNDVQTglXaTpXZ/GlHXQRf0wl0OPkKsKx4ZzYEpp Ld6leNcG2mqeSz53OiATIgHQv2ieY2BrNU0LbbqhPcCT4H8js1WtciVORvnSFu+wZMEBnunKoGqY Ds/YYPIvSbjkQuE4NRb0yG5P94FW6LqjviOvrv1vA+ACOzB2+httQc8Bsem4yWb02ybzOqR08kkk W8mw0FfB+j564ZfJAAAxggOyMIIDrgIBATBrMFMxCzAJBgNVBAYTAkNIMRUwEwYDVQQKEwxTd2lz c1NpZ24gQUcxLTArBgNVBAMTJFN3aXNzU2lnbiBSU0EgU01JTUUgTENQIElDQSAyMDIxIC0gMgIU Tx1IzA+W/yTzA5lqYJox9cE9fx0wDQYJYIZIAWUDBAIBBQCgggIYMBgGCSqGSIb3DQEJAzELBgkq hkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTIzMDcwMzE1NDgwNlowLQYJKoZIhvcNAQk0MSAwHjAN BglghkgBZQMEAgEFAKENBgkqhkiG9w0BAQsFADAvBgkqhkiG9w0BCQQxIgQghBSmf1R+NA1p5LWi BhUeVImPmqf9AOYy4XYZIBS/DX4wegYJKwYBBAGCNxAEMW0wazBTMQswCQYDVQQGEwJDSDEVMBMG A1UEChMMU3dpc3NTaWduIEFHMS0wKwYDVQQDEyRTd2lzc1NpZ24gUlNBIFNNSU1FIExDUCBJQ0Eg MjAyMSAtIDICFE8dSMwPlv8k8wOZamCaMfXBPX8dMHwGCyqGSIb3DQEJEAILMW2gazBTMQswCQYD VQQGEwJDSDEVMBMGA1UEChMMU3dpc3NTaWduIEFHMS0wKwYDVQQDEyRTd2lzc1NpZ24gUlNBIFNN SU1FIExDUCBJQ0EgMjAyMSAtIDICFE8dSMwPlv8k8wOZamCaMfXBPX8dMIGDBgkqhkiG9w0BCQ8x djB0MAsGCWCGSAFlAwQBKjALBglghkgBZQMEARYwCwYJYIZIAWUDBAECMAoGCCqGSIb3DQMHMAsG CWCGSAFlAwQCAzALBglghkgBZQMEAgIwCwYJYIZIAWUDBAIBMAsGCWCGSAFlAwQCBDALBglghkgB ZQMEAgcwDQYJKoZIhvcNAQELBQAEggEAltlpNAw5qH+M6cIvqsWHsIM3AVs+nQw3FTm8NPIgxX1H EfBjOP727aYOeRTydZR6Lgi4rr3YwilW67WaLSrXoceBZzgrCdp8+UcHJuYkBexpoQKHDmjnZ3UZ nnz9lEr+N9PNHkq5SNLgE33r/QsEZA4HCTkn27BC2xEoVSSo379iGAr/oUYo+OKSLK60mHymKNkX G3Z8XL5J5LsL3otZqLDcaEM4B+WCdW712OA8Dgsal7nTQHmTDcmLXfeKXF6CRZq6XnRO7V/zyYns U42YuUYmX3q5C18KHdQl3zbB6wgEQax+wIYxAOhxCFDeME8UV1EbZpu7jXVcNawOpncljQAAAAAA AA== ------=_Part_89646_1792331142.1688399286006--