Date: Sat, 1 Jul 2023 18:20:48 -0700 From: Navdeep Parhar <np@freebsd.org> To: Josef.Zahner1@swisscom.com Cc: freebsd-net@freebsd.org Subject: Re: Chelsio NIC with RSS - Traffic distribution to different Queues Message-ID: <CAPFoGT_OVv0AaqxUUwn9yaVY26yCRu2Y3egmrWG14w2wcdCsog@mail.gmail.com> In-Reply-To: <ZR0P278MB0757B1F52CAA215EE1C53CF0D925A@ZR0P278MB0757.CHEP278.PROD.OUTLOOK.COM> References: <ZR0P278MB0757DCEE4448080AA4E20987D926A@ZR0P278MB0757.CHEP278.PROD.OUTLOOK.COM> <CAPFoGT83PZ7dGbzbBi8FO1ChnGBCHQmtocXAAaS5Y18e6DE-NQ@mail.gmail.com> <ZR0P278MB0757DC6EDBA73A38D84D31ACD927A@ZR0P278MB0757.CHEP278.PROD.OUTLOOK.COM> <8fe00cbc-f218-a587-48d8-1612223ccd49@FreeBSD.org> <ZR0P278MB0757B1F52CAA215EE1C53CF0D925A@ZR0P278MB0757.CHEP278.PROD.OUTLOOK.COM>
next in thread | previous in thread | raw e-mail | index | archive | help
Hello, Please try this patch: https://people.freebsd.org/~np/norssq.diff It adds these sysctls to the driver. 1) hw.cxgbe.rsrv_norssq. This is what you originally asked for. 2) hw.cxgbe.rx_budget. This can be used to force the driver's RX to yield periodically. What kind of system (cores, memory, etc.) is this? Control packets are either getting dropped or the threads/timers responsible for sending or processing these packets are starved of CPU. It would be useful to monitor interface activity with "netstat -d -I <ifnet>" during the test. # sysctl hw.model hw.ncpu hw.physmem # netstat -dw1 -I cxl0 Try the settings listed below. nrxq=3DX might help in case the driver RX threads are hogging all the cores because all rx queues are heavily loaded. Set nrxq to something less than the number of cores in the system. rx_budget can be changed any time (try 64, 128, 256) and might improve the responsiveness of the rest of the system during load. (in loader.conf) hw.cxgbe.nrxq=3D2 (3 if you've patched the kernel and set n= orssq) hw.cxgbe.rsrv_noflowq=3D1 hw.cxgbe.pause_settings=3D0 hw.cxgbe.cong_drop=3D1 (2 would be better but needs a recent dri= ver) hw.cxgbe.rsrv_norssq=3D1 (needs patch) hw.cxgbe.rx_budget=3D128 (needs patch) Let us know how it goes. Regards, Navdeep On Thu, Jun 29, 2023 at 5:53=E2=80=AFAM <Josef.Zahner1@swisscom.com> wrote: > > Can you tell me which netstat command you have in mind? I tried =E2=80=9C= netstat -Q=E2=80=9D, it shows a few drops but not that much that it would e= xplain the CARP drops. What I can tell you is, that especially CARP on the = corresponding server is just sending out packets as long as it is the maste= r box and CPU0 load is below 100%. It doesn=E2=80=99t receive any CARP traf= fic at all, just normal network traffic. What I see is, that those CARP pac= kets are not sent anymore if CPU0 has 100% load -> if that happens the serv= er switches to standby and the traffic is gone on the machine. So because o= f this behavior we would like to have an option, which allows us to have Co= ntrol Plane Traffic (LACP, CARP,=E2=80=A6?) in RSS RX queue 0 and nothing e= lse. Question is, what would Control Plane traffic be. Hopefully as well CA= RP/VRRP,=E2=80=A6 > > We tried hw.cxgbe.cong_drops=3D1, but it doesn=E2=80=99t help in our case= . > > Can you explain a bit what your patch will do? Am I right that you will p= ost the link later on here? > > Cheers Josef
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAPFoGT_OVv0AaqxUUwn9yaVY26yCRu2Y3egmrWG14w2wcdCsog>