Date: Thu, 6 Jul 2023 00:58:07 -0700 From: Navdeep Parhar <np@freebsd.org> To: Josef.Zahner1@swisscom.com Cc: freebsd-net@freebsd.org Subject: Re: Chelsio NIC with RSS - Traffic distribution to different Queues Message-ID: <CAPFoGT8iqdu1-Y3SMHfZXSL0n3oMdTdESQHE56HyqhaWsW39Kw@mail.gmail.com> In-Reply-To: <ZR0P278MB0757E403874283B0311A5242D929A@ZR0P278MB0757.CHEP278.PROD.OUTLOOK.COM> References: <ZR0P278MB0757DCEE4448080AA4E20987D926A@ZR0P278MB0757.CHEP278.PROD.OUTLOOK.COM> <CAPFoGT83PZ7dGbzbBi8FO1ChnGBCHQmtocXAAaS5Y18e6DE-NQ@mail.gmail.com> <ZR0P278MB0757DC6EDBA73A38D84D31ACD927A@ZR0P278MB0757.CHEP278.PROD.OUTLOOK.COM> <8fe00cbc-f218-a587-48d8-1612223ccd49@FreeBSD.org> <ZR0P278MB0757B1F52CAA215EE1C53CF0D925A@ZR0P278MB0757.CHEP278.PROD.OUTLOOK.COM> <CAPFoGT_OVv0AaqxUUwn9yaVY26yCRu2Y3egmrWG14w2wcdCsog@mail.gmail.com> <ZR0P278MB0757A3D9DD60F6DFA047E70CD929A@ZR0P278MB0757.CHEP278.PROD.OUTLOOK.COM> <ZR0P278MB0757F40DA0CDC7A041D6AAF9D929A@ZR0P278MB0757.CHEP278.PROD.OUTLOOK.COM> <ZR0P278MB0757E403874283B0311A5242D929A@ZR0P278MB0757.CHEP278.PROD.OUTLOOK.COM>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Jul 3, 2023 at 8:48=E2=80=AFAM <Josef.Zahner1@swisscom.com> wrote: > > Sorry for the spam, I do see the values with sysctl now. It seems that Fr= eeBSD always loads the if_cxgbe.ko from /boot/kernel/if_cxgbe.ko. So what I= =E2=80=99ve done is, I renamed the original file and copied the newly compi= led if_cxgbe.ko from /boot/module to /boot/kernel. Is there a cleaner way t= o get it work? Btw. Do I need t5fw_cfg.ko as well, I haven=E2=80=99t found = any documentation what exactly it does=E2=80=A6 > > > > Quiet difficult for someone who isn=E2=80=99t familiar with FreeBSD at al= l :-P. > > > > So I=E2=80=99ve retested again. Sadly it does still share the load over a= ll configured CPUs (0-3) for RSS (there are 8 cores plus 8 HT cores per phy= sical CPU). As I already mentioned, the new sysctl values are now visible, = so I think the driver should be fine. My expectation with that configuratio= n was, that CPU0 shouldn=E2=80=99t be used for RSS (as all values are small= er than the available CPUs), but it has been used, so the network traffic d= oes flapping as hell. Each rx queue gets its own interrupt and I verified with "vmstat -i" that only non-RSS traffic (ARP) resulted in activity on queue 0, TCP and UDP traffic were seen on other queues, as expected. Note that rx queue 0 does not necessarily mean CPU0. > hw.cxgbe.cong_drop=3D"1" > hw.cxgbe.nrxq=3D"3" > hw.cxgbe.pause_settings=3D"0" > hw.cxgbe.rsrv_noflowq=3D"1" > hw.cxgbe.rsrv_norssq=3D"1" > hw.cxgbe.rx_budget=3D"128" > if_cxgbe_load=3D"yes" okay. > net.inet.rss.bits=3D"2" > net.inet.rss.enabled=3D"1" > net.isr.bindthreads=3D"1" > net.isr.maxthreads=3D"-1" What are all these for? Please go back to defaults and retry with only the driver tunables above. It looks like OPNsense ships with "options RSS" in the kernel. That complicates things a bit because I tested my patch on FreeBSD GENERIC kernel which does not have this option. Run "top -SHIPzt" during the test to see exactly what is hogging CPU0. Regards, Navdeep
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAPFoGT8iqdu1-Y3SMHfZXSL0n3oMdTdESQHE56HyqhaWsW39Kw>