Date: Mon, 10 Jul 2023 09:36:24 +0000 From: <Josef.Zahner1@swisscom.com> To: <np@freebsd.org> Cc: <freebsd-net@freebsd.org> Subject: Re: Chelsio NIC with RSS - Traffic distribution to different Queues Message-ID: <ZR0P278MB075798BC3BA3A925AC7F5E54D930A@ZR0P278MB0757.CHEP278.PROD.OUTLOOK.COM> In-Reply-To: <CAPFoGT8iqdu1-Y3SMHfZXSL0n3oMdTdESQHE56HyqhaWsW39Kw@mail.gmail.com> References: <ZR0P278MB0757DCEE4448080AA4E20987D926A@ZR0P278MB0757.CHEP278.PROD.OUTLOOK.COM> <CAPFoGT83PZ7dGbzbBi8FO1ChnGBCHQmtocXAAaS5Y18e6DE-NQ@mail.gmail.com> <ZR0P278MB0757DC6EDBA73A38D84D31ACD927A@ZR0P278MB0757.CHEP278.PROD.OUTLOOK.COM> <8fe00cbc-f218-a587-48d8-1612223ccd49@FreeBSD.org> <ZR0P278MB0757B1F52CAA215EE1C53CF0D925A@ZR0P278MB0757.CHEP278.PROD.OUTLOOK.COM> <CAPFoGT_OVv0AaqxUUwn9yaVY26yCRu2Y3egmrWG14w2wcdCsog@mail.gmail.com> <ZR0P278MB0757A3D9DD60F6DFA047E70CD929A@ZR0P278MB0757.CHEP278.PROD.OUTLOOK.COM> <ZR0P278MB0757F40DA0CDC7A041D6AAF9D929A@ZR0P278MB0757.CHEP278.PROD.OUTLOOK.COM> <ZR0P278MB0757E403874283B0311A5242D929A@ZR0P278MB0757.CHEP278.PROD.OUTLOOK.COM> <CAPFoGT8iqdu1-Y3SMHfZXSL0n3oMdTdESQHE56HyqhaWsW39Kw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
------=_Part_95457_1099939871.1688981788704 Content-Type: multipart/alternative; boundary="_000_ZR0P278MB075798BC3BA3A925AC7F5E54D930AZR0P278MB0757CHEP_" Content-Language: en-GB --_000_ZR0P278MB075798BC3BA3A925AC7F5E54D930AZR0P278MB0757CHEP_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Ok we are getting closer to a solution. But I=92m a little bit confused, I= =92ll try to explain why. I=92ve removed the following sysctl values, as you stated that they are not= official FreeBSD values: - net.inet.rss.bits=3D"2" - net.inet.rss.enabled=3D"1" - net.isr.bindthreads=3D"1" - net.isr.maxthreads=3D"-1" Without the setting above and without your proposed sysctl values, my syste= m was stable and the traffic distributed over all CPUs (even HT cores). So = no more drops on CARP if CPU0 has had high load. It looks like my problem i= s clearly related to the =93Custom=94 OPNsense RSS values above. As the RSS= sysctl values are noted in the OPNsense guide (https://docs.opnsense.org/t= roubleshooting/performance.html), my intention was, that those values needs= to be set anyway for RSS. But it seems that those 4 values are making the = thing worse! Back to your topic. You are right, queue-0 does only control traffic if I= =92m enabling your sysctl values. Where can I see which protocols are falli= ng into control traffic (queue 0)? So your change works as expected, thanks= a lot! As you stated, queue 0 is never CPU 0, which is important to know. I=92m pretty sure that I needed the RSS values above to get RSS work on int= el NICs and with =93netstat -Q=94 I could see the RSS queues. With Chelsio = it seems that the FreeBSD process =93netisr=94 doesn=92t do the RSS stuff a= nd =93netstat -Q=94 shows nothing about RSS. For me this means that your dr= iver directly does the RSS magic, is this correct? Please help to shed ligh= t on this topic. Cheers Josef --_000_ZR0P278MB075798BC3BA3A925AC7F5E54D930AZR0P278MB0757CHEP_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr= osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:x=3D"urn:schemas-microsoft-com:office:excel" xmlns:m=3D"http://schema= s.microsoft.com/office/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html= 40"> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1= 252"> <meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0cm; font-size:10.0pt; font-family:"Calibri",sans-serif;} a:link, span.MsoHyperlink {mso-style-priority:99; color:#0563C1; text-decoration:underline;} span.EmailStyle19 {mso-style-type:personal-reply; font-family:"Calibri",sans-serif; color:windowtext;} .MsoChpDefault {mso-style-type:export-only; font-size:10.0pt; mso-ligatures:none;} @page WordSection1 {size:612.0pt 792.0pt; margin:72.0pt 72.0pt 72.0pt 72.0pt;} div.WordSection1 {page:WordSection1;} --></style> </head> <body lang=3D"en-CH" link=3D"#0563C1" vlink=3D"#954F72" style=3D"word-wrap:= break-word"> <div class=3D"WordSection1"> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-size:11.0pt;mso-f= areast-language:EN-US">Ok we are getting closer to a solution. But I=92m a = little bit confused, I=92ll try to explain why.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-size:11.0pt;mso-f= areast-language:EN-US"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-size:11.0pt;mso-f= areast-language:EN-US">I=92ve removed the following sysctl values, as you s= tated that they are not official FreeBSD values:<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-size:11.0pt">-</s= pan><span style=3D"font-size:11.0pt"> net.inet.rss.bits=3D"2"<br> </span><span lang=3D"EN-US" style=3D"font-size:11.0pt">-</span><span style= =3D"font-size:11.0pt"> net.inet.rss.enabled=3D"1"<br> </span><span lang=3D"EN-US" style=3D"font-size:11.0pt">-</span><span style= =3D"font-size:11.0pt"> net.isr.bindthreads=3D"1"<br> </span><span lang=3D"EN-US" style=3D"font-size:11.0pt">-</span><span style= =3D"font-size:11.0pt"> net.isr.maxthreads=3D"-1"</span><span lang= =3D"EN-US" style=3D"font-size:11.0pt;mso-fareast-language:EN-US"><o:p></o:p= ></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language= :EN-US"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-size:11.0pt;mso-f= areast-language:EN-US">Without the setting above and without your proposed = sysctl values, my system was stable and the traffic distributed over all CP= Us (even HT cores). So no more drops on CARP if CPU0 has had high load. It looks like my problem is clearly relate= d to the =93Custom=94 OPNsense RSS values above. As the RSS sysctl values a= re noted in the OPNsense guide (<a href=3D"https://docs.opnsense.org/troubl= eshooting/performance.html">https://docs.opnsense.org/troubleshooting/perfo= rmance.html</a>), my intention was, that those values needs to be set anyway for RSS. But it= seems that those 4 values are making the thing worse! <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-size:11.0pt;mso-f= areast-language:EN-US"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-size:11.0pt;mso-f= areast-language:EN-US">Back to your topic. You are right, queue-0 does only= control traffic if I=92m enabling your sysctl values. Where can I see whic= h protocols are falling into control traffic (queue 0)? So your change works as expected, thanks a lot!<o:p></o:p></spa= n></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-size:11.0pt;mso-f= areast-language:EN-US"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-size:11.0pt;mso-f= areast-language:EN-US">As </span><span lang=3D"EN-US" style=3D"font-size:11.0pt">you stated, queue 0 = is never CPU 0, which is important to know.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-size:11.0pt"><o:p= > </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-size:11.0pt">I=92= m pretty sure that I needed the RSS values above to get RSS work on intel N= ICs and with =93</span><span lang=3D"EN-US" style=3D"font-size:11.0pt;font-= family:"Courier New";mso-fareast-language:EN-US">netstat -Q</span><span lang=3D"EN-US" style=3D"font-size:11.0pt">=94 I could see t= he RSS queues. With Chelsio it seems that the FreeBSD process =93netisr=94 = doesn=92t do the RSS stuff and =93netstat -Q=94 shows nothing about RSS. Fo= r me this means that your driver directly does the RSS magic, is this correct? Please help to shed light on this topic.<o:p><= /o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-size:11.0pt"><o:p= > </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-size:11.0pt">Chee= rs Josef<o:p></o:p></span></p> </div> </body> </html> --_000_ZR0P278MB075798BC3BA3A925AC7F5E54D930AZR0P278MB0757CHEP_-- ------=_Part_95457_1099939871.1688981788704 Content-Type: application/pkcs7-signature; name=smime.p7s; smime-type=signed-data Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCAMIIH KzCCBROgAwIBAgIUTx1IzA+W/yTzA5lqYJox9cE9fx0wDQYJKoZIhvcNAQELBQAwUzELMAkGA1UE BhMCQ0gxFTATBgNVBAoTDFN3aXNzU2lnbiBBRzEtMCsGA1UEAxMkU3dpc3NTaWduIFJTQSBTTUlN RSBMQ1AgSUNBIDIwMjEgLSAyMB4XDTIyMDQyOTA5MDMxMloXDTI1MDQyOTA5MDMxMlowgYMxCzAJ BgNVBAYTAkNIMR4wHAYDVQQKExVTd2lzc2NvbSAoU2Nod2VpeikgQUcxKTAnBgkqhkiG9w0BCQEW Gmpvc2VmLnphaG5lcjFAc3dpc3Njb20uY29tMSkwJwYDVQQDEyBTZWN1cmUgTWFpbDogR2F0ZXdh eSBDZXJ0aWZpY2F0ZTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALWL60L8MKsvXVrM C2QjifRYKIolHpx2KvKSyk75F6eEFVnz19XTfK+kKbkZzc9BMYL9xK78nYEoC8wPyEPwcxrw4i8l SPmhkZ674JEndHrX6qHJOku5+OyIhbLa550XtSZRJSd7l0NnYQWCFtWUAPL/j/ButDdDgp7sScWN D+BmOZw0/FyveEgqeJBAAcxZ5em8/2Ub/xew8LBdwOXYnzG6VIck+jTHv6RhNZ1+RfOvip+3sOwm TsvRL0oC8QGWC86OAq81+tCeYQPoywrybSjqYCwE+ZRb5djg76DvwgjfrVF8grddF6OSwkdUxSlJ USZ50RB4WO23r6YWjYX272ECAwEAAaOCAsQwggLAMCUGA1UdEQQeMByBGmpvc2VmLnphaG5lcjFA c3dpc3Njb20uY29tMA4GA1UdDwEB/wQEAwIEsDATBgNVHSUEDDAKBggrBgEFBQcDBDAdBgNVHQ4E FgQUNxi5osjEbpfWsyrkPqFhIrG5I9gwHwYDVR0jBBgwFoAU+lTAgqb+lr0Ex1+fX4IMPcOVT0cw gf8GA1UdHwSB9zCB9DBHoEWgQ4ZBaHR0cDovL2NybC5zd2lzc3NpZ24ubmV0L0ZBNTRDMDgyQTZG RTk2QkQwNEM3NUY5RjVGODIwQzNEQzM5NTRGNDcwgaiggaWggaKGgZ9sZGFwOi8vZGlyZWN0b3J5 LnN3aXNzc2lnbi5uZXQvQ049RkE1NEMwODJBNkZFOTZCRDA0Qzc1RjlGNUY4MjBDM0RDMzk1NEY0 NyUyQ089U3dpc3NTaWduJTJDQz1DSD9jZXJ0aWZpY2F0ZVJldm9jYXRpb25MaXN0P2Jhc2U/b2Jq ZWN0Q2xhc3M9Y1JMRGlzdHJpYnV0aW9uUG9pbnQwZwYDVR0gBGAwXjBSBghghXQBWQIBCzBGMEQG CCsGAQUFBwIBFjhodHRwczovL3JlcG9zaXRvcnkuc3dpc3NzaWduLmNvbS9Td2lzc1NpZ25fQ1BT X1NNSU1FLnBkZjAIBgYEAI96AQMwgcYGCCsGAQUFBwEBBIG5MIG2MGQGCCsGAQUFBzAChlhodHRw Oi8vc3dpc3NzaWduLm5ldC9jZ2ktYmluL2F1dGhvcml0eS9kb3dubG9hZC9GQTU0QzA4MkE2RkU5 NkJEMDRDNzVGOUY1RjgyMEMzREMzOTU0RjQ3ME4GCCsGAQUFBzABhkJodHRwOi8vb2NzcC5zd2lz c3NpZ24ubmV0L0ZBNTRDMDgyQTZGRTk2QkQwNEM3NUY5RjVGODIwQzNEQzM5NTRGNDcwDQYJKoZI hvcNAQELBQADggIBAKC/bdS77QzqssisgA9nHbH+E0q2i4Ge2o3BxSlFxp/rzqXW0jpOb4ntgqQ5 /JECExksUdzjwqARiT/grBfZedbpQK40nNe3fKoZDQpmF8P8T34Ngf13EtpIm7+IJOXjhBQ6PYeM 1vLTn/g0EUJK3k9fdw3acOUHVQ4dstKP1q9IB5FGtiAIfUtk0m9RT0aJjMqQG1l33ignJrfGZ+R2 X121LjxJ+6gFzWLKklJIfZtqxDGeLIjm5ixHeOrkj7zYily3W16I3iFJnvYL0t1x9UHFp+fWuNUh HiqMK48mLHDzG/ffw3fSgDrnLunlxd4uHz9Io4IFwOKjiOJNxPzuQ6kfibasoKhv/MTFXarfXR7R fAWpOgl9EhxKBJqQkDLZAfCjYQwCU+mThs1p9Sc/JG1IZ2jI3XD4fCHnejMA/RlbINc5uOrPvRys GGPYo5UmPZ4yj4vJRa+U9udZRO5YZrpIto5J8MxZ3Y+cizKUkovzqIVdg2gLj7+Ij4YSRxK5HnoE dN7EPmbfX3hLnAmsG0cft84/o+r2xMQiMXQOmeTI2ZnGk23MF3vyUExkEhcFtAkVup/XiAjcs9St /kD47K7tVNkZGxN0XnaILJt1R6pAGo6Xj9WfGnRX6Ph3PJ+XguEhMhLVEfuuBUuAGFUWX9hovJuM X8C83YJTkvTzbmNAMIIHRzCCBS+gAwIBAgIPLFqp2VT9squWrRe2X4z0MA0GCSqGSIb3DQEBCwUA MFMxCzAJBgNVBAYTAkNIMRUwEwYDVQQKEwxTd2lzc1NpZ24gQUcxLTArBgNVBAMTJFN3aXNzU2ln biBSU0EgU01JTUUgUm9vdCBDQSAyMDIxIC0gMTAeFw0yMTA4MDQxMjEyNTlaFw0zNjA3MzExMjEy NTlaMFMxCzAJBgNVBAYTAkNIMRUwEwYDVQQKEwxTd2lzc1NpZ24gQUcxLTArBgNVBAMTJFN3aXNz U2lnbiBSU0EgU01JTUUgTENQIElDQSAyMDIxIC0gMjCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCC AgoCggIBAL0FGcjUDXE+AwMQtJMWAOvPI9rAByO7UTw4EtNAdcNyY3jrCyd5X41WiKnZfVlD0YgX duPHRour3+KIqj0SwJ0I7UEQ19zvSeC/MVLWg52Z/PKaIoBmVawQlJChzIuV0NZIXTco3xrtGCXf Q8MhREG8FaWkg4LGIIBSAqiAZgG+btlFlnXQFyB8ZT+BrX2OWEYj3FIn2kI+O9xY8kqmJJsb6zQJ H8zhxdpA871kQVjEifuDis0hgFhnCGwiNw+fk60L3/un/cX/aooD2B+8r8LJDXh89U49Ulo8QK58 yzj61GGHZ52kbHiKtBNpVXzmJVTfCM+mLzhMiBSVHCpRaWiPXnwDmvukZUxwj75KXKreu527fX1a OILnuQCKRUevIL6DALhJQBifQKpf/9wCnH732eYtYCZacZ15pGYcwkKYl9fLcOOLxKdW6CBxdsSu eSvmyazWTKnnCINHthUiR1O6IkekXZ3tZn93kQ5PsoF1L+Trv9eBOtTCRbApHZWb/TRvRmnPc3ge D0VOsE/ncDa5yLZ7st/e9HyEUmKOG7TzMa4KVjsqGf7QEFTnRhI4TYGdhawE57fp13DbjvRwz+Ia 0wxpNecDSuDgyxf1M/S9fiJClZkzCo3CpxiTR7AyrLhEMBjvEfgZc2REZPacmeBRFtBLXCBJwXU0 zNFb6tGtAgMBAAGjggIWMIICEjAOBgNVHQ8BAf8EBAMCAQYwEwYDVR0lBAwwCgYIKwYBBQUHAwQw EgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQU+lTAgqb+lr0Ex1+fX4IMPcOVT0cwHwYDVR0j BBgwFoAUCQy/KqIdBCQMsvlACkHCz1pyqoAwgf8GA1UdHwSB9zCB9DBHoEWgQ4ZBaHR0cDovL2Ny bC5zd2lzc3NpZ24ubmV0LzA5MENCRjJBQTIxRDA0MjQwQ0IyRjk0MDBBNDFDMkNGNUE3MkFBODAw gaiggaWggaKGgZ9sZGFwOi8vZGlyZWN0b3J5LnN3aXNzc2lnbi5uZXQvQ049MDkwQ0JGMkFBMjFE MDQyNDBDQjJGOTQwMEE0MUMyQ0Y1QTcyQUE4MCUyQ089U3dpc3NTaWduJTJDQz1DSD9jZXJ0aWZp Y2F0ZVJldm9jYXRpb25MaXN0P2Jhc2U/b2JqZWN0Q2xhc3M9Y1JMRGlzdHJpYnV0aW9uUG9pbnQw HwYDVR0gBBgwFjAKBghghXQBWQIBCzAIBgYEAI96AQMwdAYIKwYBBQUHAQEEaDBmMGQGCCsGAQUF BzAChlhodHRwOi8vc3dpc3NzaWduLm5ldC9jZ2ktYmluL2F1dGhvcml0eS9kb3dubG9hZC8wOTBD QkYyQUEyMUQwNDI0MENCMkY5NDAwQTQxQzJDRjVBNzJBQTgwMA0GCSqGSIb3DQEBCwUAA4ICAQAH Dx6XBIV6CZbsgakZM4/C04La3NPAJCJEDZUL8RYYpySTwDcX1D4zYDyVRhWVEom3fWkB7Nxdo/o4 VbTcg+KJdTvRADn3ycapTHPOOfyukWFmAKPpuLwwl3kwfkLZFFgWuGJF5/HqmnjYstzQAUgXVmqR phrW/PQSBM4v1cTeKdd7f23A8uh7etz9/yB+l6b+zU0gmTZrPDLFmfbbphKtJ52/GF7suI3qA6q9 TuHGkLxtdqIhhXLsMhJfAS33TYZz4zhBuFR/zJD4+IrdB7EYhaK6w47jpyfrvd0b/UoLOZr/UuWn 0PFp5QeXGON556v/Za1ZZk0WGgievGlwrnyh1ollExlAnA2WMxvrqAZvk6l5qc3Sy73f9L0atmAm o3qymnXvVrN8y5DNWqP2DEXLrrIh2WuFlxtr9CLELZ4AnGbl8z89mD1ZdnvfsbyMBuy6hEQfVvGy CUvPEAjBuhju5XAiKbplyJ4pvpjLifpPOl6s2aFTlQeJRlInMhD+q70HXhx+HQIAmjX2nt9G/J4X +nsg0MfRx5NVOrmwXyQW0wzVvP7T+xY5M2p2LyQY7sjuKi4Lf5XgMPHgXWwZxqKAVe1SkuaiXkuy z6L01caBYVw8Wt57GgtoF1yQMmAGTrM4cpVjBl2euDYKi8h6/+66YrRbIJruLHX5INyx3Y9fEjCC BoswggRzoAMCAQICEADeTFUg9tz0AhsPEVT3jRAwDQYJKoZIhvcNAQELBQAwRTELMAkGA1UEBhMC Q0gxFTATBgNVBAoTDFN3aXNzU2lnbiBBRzEfMB0GA1UEAxMWU3dpc3NTaWduIEdvbGQgQ0EgLSBH MjAeFw0yMTA4MDMxMzE0NTVaFw0zNjEwMjMxMzE0NTVaMFMxCzAJBgNVBAYTAkNIMRUwEwYDVQQK EwxTd2lzc1NpZ24gQUcxLTArBgNVBAMTJFN3aXNzU2lnbiBSU0EgU01JTUUgUm9vdCBDQSAyMDIx IC0gMTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBANayuLQ4jya6N8gBI0UWfr5kOIZy ZmFYjSSKWbMMoSqrfruFfGVmcKfpItvuuzL6q7GGP6tIgbir8yrdN8cuC/ar31WtVCAUOreJRG3n 6D+uiCEYjkdWlQDJ7GVuVkcUTa0uJtLUi2zK8zMt+fCbjreGJoHnC56LDHwFpzx6+fCFkJyJZzl1 EbFjsNQjLH3cHyt27QStuhHJB0kN4ygPLhEU0ray/3i4/lpTgCSsC0i6TjIxUeyq/rtELAvX+X2r jdpsqwjd80E9j/VBQVzGzFHKDkQft2qAdlVpUeZM/ReA+7NU7rBKHTOTBnm5YRGs5A5bs93gsSVc t9TTzfR7ngFUK4KQoeHKQ43wQaQcB8DWMxajRUaPhExp/ZNXndPlb8skDDEtA5jCADlEeSKBbeTq /AtkJm78yp4aA0Ttf01N6RGydr2GfXu7VD9RkEfHi/j/TizyCDCMGcEsRzWevatTpCKunwwhGSm9 npvPhNyO0TVLIhCBG2LtwEvTK5AiSR8tIa6Rxd/x1kFUcg7eyjQQ9cmandVcFuTNJbHHqHFGrPhT hReJqyQaOBgyJHPpVa74gGMDb4Sw36CUtalT8Itq9VR55f9bnKJvIuH/QCllbG+OSGkxPoEbO4tY +lsvO2t9ayTwvPKN5ZrmrHjL2IIrABcdeWoJLtZuds8w+9tZAgMBAAGjggFnMIIBYzAPBgNVHRMB Af8EBTADAQH/MB0GA1UdDgQWBBQJDL8qoh0EJAyy+UAKQcLPWnKqgDAfBgNVHSMEGDAWgBRbJXuW pGVRfrg588B4Zl7oOufw7jAOBgNVHQ8BAf8EBAMCAQYwgf8GA1UdHwSB9zCB9DBHoEWgQ4ZBaHR0 cDovL2NybC5zd2lzc3NpZ24ubmV0LzVCMjU3Qjk2QTQ2NTUxN0VCODM5RjNDMDc4NjY1RUU4M0FF N0YwRUUwgaiggaWggaKGgZ9sZGFwOi8vZGlyZWN0b3J5LnN3aXNzc2lnbi5uZXQvQ049NUIyNTdC OTZBNDY1NTE3RUI4MzlGM0MwNzg2NjVFRTgzQUU3RjBFRSUyQ089U3dpc3NTaWduJTJDQz1DSD9j ZXJ0aWZpY2F0ZVJldm9jYXRpb25MaXN0P2Jhc2U/b2JqZWN0Q2xhc3M9Y1JMRGlzdHJpYnV0aW9u UG9pbnQwDQYJKoZIhvcNAQELBQADggIBAAL46l3QisysCAM2VGb/CimG4VSDLDPox2yuEKlUgX8q LYgmraaoNCgPGQapneIClQQyZEwLyvnjfcMTW+iXQI534+OCgYetAWAHw8XhIP23MJc+uhxxdItf Taex/k58CXh6h1/xrKElEZfHLBGckOpyFj0CNam75SPYH1JAt9COXdojvDPpMv24dZ9Dvj1XS4dh 3u2WyHB8frcT4QlAuxkCBw9r3R/SzA5aEhjkwbGcvr4rER7lmsXgoTWx5OGyYq7A6Gx8lof6YN4t iRwUQUA5onfvsBVbAT8ezuYUqZy+gp+xYhffIkO1Mm+3BfwYytp6Q11ltSb+WkGhaXSX8UNRjdx/ 2VeEpx1R8oJtqw5806Pl4MmVBG3yx5134qX4yMW5ZwZvbf3Gf26+xWrbBDbbMG9dvciZ/sRylsy5 y3SLJKkTC3i1Bsr1iyYWc5gdcZWd8/BS6WxVfgUiF9CJPGXtV4B3/NisvbNTjwd7WBN6sefJsPjj yaGR4nTOymgbshvElmCUkNvlCLc+zIh9Z8BV/Chz3hw72s8PHLYI0jM++TySSKBacIgeEBeYenbd YEg+ckU+cGuM60h8WbVWBRIUCkZNAjYJ0WxzIVIn2GvE0nKnTH7bNs7TPctc4u4b3fk6/U0T/w7O UrYWcTOgl8Vf9oIYF7U6m5u2eKsb6/1aMIIFujCCA6KgAwIBAgIJALtAHEP1Xk+wMA0GCSqGSIb3 DQEBBQUAMEUxCzAJBgNVBAYTAkNIMRUwEwYDVQQKEwxTd2lzc1NpZ24gQUcxHzAdBgNVBAMTFlN3 aXNzU2lnbiBHb2xkIENBIC0gRzIwHhcNMDYxMDI1MDgzMDM1WhcNMzYxMDI1MDgzMDM1WjBFMQsw CQYDVQQGEwJDSDEVMBMGA1UEChMMU3dpc3NTaWduIEFHMR8wHQYDVQQDExZTd2lzc1NpZ24gR29s ZCBDQSAtIEcyMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAr+TufoskDhJuqVAtFkQ7 kpJcyrhdhJJCEyq8ZVeCQD5XJM1QiyUqt2/876LQwB8CJEoTlo8jE+YoWACjR8cGp4QjK7u9lit/ VcyLwVcfDmJlD909Vopz2q5+bbqBHH5CjCA12UNNhPqE21Is8w4ndwtrvxEvcnifLtg+5hg3Wipy +dpikJKVyh+c6bM8K8vzARO/Ws/BtQpgvd21mWRTuKCWs2/iJneRjOBiEAKfNA+k1ZIzUd6+jbqE emA8atufK+ze3gE/bk3lUIbLtK/tREDFylqM2tIrfKjuvqblCqoOpd8FUrdVxyJdMmqXl2MT28nb eTZ7hTpKxVKJ+STnnXepgv9VHKVxaSvRAiTysybUa9oEVeXBCsdtMDeQKuSeFDNeFhdVxVu1yzSJ kvGdJo+hB9TGsnhQ2wwMC3wLjEHXuendjIj3o02yMszYF9rNt85mndT9Xv+9lz4pded+p2JYryU0 pUHHPbwNUMoDAw8IWh+Vc3hiv69yFGkOpeUDDniOJihC8AcLYiAQZzlG+qkDzAQ4embvIIO1jEpW jpEA/I5cgt6IoMPiaG59je883WX0XaxR7ySArqpWl2/5rX3aYT+YdzylkbYcjCbaZaIJbcHiVOO5 ykxMgI93e2CaHt+28kgeDrpOVG2Y4OGiGqJ3UM/EY5LsRxmd6+ZrzsECAwEAAaOBrDCBqTAOBgNV HQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUWyV7lqRlUX64OfPAeGZe6Drn 8O4wHwYDVR0jBBgwFoAUWyV7lqRlUX64OfPAeGZe6Drn8O4wRgYDVR0gBD8wPTA7BglghXQBWQEC AQEwLjAsBggrBgEFBQcCARYgaHR0cDovL3JlcG9zaXRvcnkuc3dpc3NzaWduLmNvbS8wDQYJKoZI hvcNAQEFBQADggIBACe645R88a7A3hfm5djV9VSwg/S7zV4Fe0+fdWavPOhWfvxyeDgD2StiGwC5 +OlgzczOUYrHUDFu4Up+GC9pWbY9ZIEr44OE5iKHjn3g7gKZYbge9LgriBIWhMIxkziWMaa5O1M/ wySTVltpkuzFwbs4AOPsF6m43Md8AYOfMke6UiI0HTJ6CVanfCU2qT1L2sCCbwq7EsiHSycR+R4t x5M/nttfJmtS2S6K8RTGRI0Vqbe/vd6mGu6uLftIdxf+u+yvGPUqUfA5hJeVbG4bwyvEdGB5JbAK J9/fXtI5z0V9QkvfsywexcZdylU6oJxpmo/a77KwPJ+HbBIrZXAVUjEaJM9vMSNQH4xPjyPDdEFj HFWoFN0+4FFQz/EbMFYOkrCChdiDyyJkvC24JdVUorgG6q2SpCSgwYa1ShNqR88uC1aVVMvOmttq tKay20EIhid392qgQmwLOM7XdVAyksLfKzAiSNDVQTglXaTpXZ/GlHXQRf0wl0OPkKsKx4ZzYEpp Ld6leNcG2mqeSz53OiATIgHQv2ieY2BrNU0LbbqhPcCT4H8js1WtciVORvnSFu+wZMEBnunKoGqY Ds/YYPIvSbjkQuE4NRb0yG5P94FW6LqjviOvrv1vA+ACOzB2+httQc8Bsem4yWb02ybzOqR08kkk W8mw0FfB+j564ZfJAAAxggOyMIIDrgIBATBrMFMxCzAJBgNVBAYTAkNIMRUwEwYDVQQKEwxTd2lz c1NpZ24gQUcxLTArBgNVBAMTJFN3aXNzU2lnbiBSU0EgU01JTUUgTENQIElDQSAyMDIxIC0gMgIU Tx1IzA+W/yTzA5lqYJox9cE9fx0wDQYJYIZIAWUDBAIBBQCgggIYMBgGCSqGSIb3DQEJAzELBgkq hkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTIzMDcxMDA5MzYyOFowLQYJKoZIhvcNAQk0MSAwHjAN BglghkgBZQMEAgEFAKENBgkqhkiG9w0BAQsFADAvBgkqhkiG9w0BCQQxIgQgUNHzFWYtZO8Dsqjg 7QfsQwIzNNHQwAQZ9ZuF+B5oaqgwegYJKwYBBAGCNxAEMW0wazBTMQswCQYDVQQGEwJDSDEVMBMG A1UEChMMU3dpc3NTaWduIEFHMS0wKwYDVQQDEyRTd2lzc1NpZ24gUlNBIFNNSU1FIExDUCBJQ0Eg MjAyMSAtIDICFE8dSMwPlv8k8wOZamCaMfXBPX8dMHwGCyqGSIb3DQEJEAILMW2gazBTMQswCQYD VQQGEwJDSDEVMBMGA1UEChMMU3dpc3NTaWduIEFHMS0wKwYDVQQDEyRTd2lzc1NpZ24gUlNBIFNN SU1FIExDUCBJQ0EgMjAyMSAtIDICFE8dSMwPlv8k8wOZamCaMfXBPX8dMIGDBgkqhkiG9w0BCQ8x djB0MAsGCWCGSAFlAwQBKjALBglghkgBZQMEARYwCwYJYIZIAWUDBAECMAoGCCqGSIb3DQMHMAsG CWCGSAFlAwQCAzALBglghkgBZQMEAgIwCwYJYIZIAWUDBAIBMAsGCWCGSAFlAwQCBDALBglghkgB ZQMEAgcwDQYJKoZIhvcNAQELBQAEggEAHcw+UMsmoaa2BV2KQk5q/Rb1CGMMB90BwB6or/6m9vxO ILxsTY6NKiyZQXIK09DEGS57KIlMtAY2q+g9IKnfMpTFz+yjRYUrhG7L2Eq91iyJUZSk9W34hDz5 quAAHdPtjA7u9g4kfBaPZ4LnAJSRrV38weNYxOFFoqd4F1CbQY0ucZcK+m+jvUrxkK3D/c8rQW/T nM0uYsp4UOl6L1Yu0htWNYEQBQKeUjbiC0f6dRpmnE0umC0HtP0UHP/mAEjNFVUwiycV3frfZYlk DUoAekGT5jj3Rbrw0jmf/AFFwuy6dB9p8eqRmT4oi638geEPS5FNC9QAJdMXrY4907oD/wAAAAAA AA== ------=_Part_95457_1099939871.1688981788704--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?ZR0P278MB075798BC3BA3A925AC7F5E54D930A>