From nobody Sun Jul 2 01:20:48 2023 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qtrq2596Dz4hRj6 for ; Sun, 2 Jul 2023 01:21:02 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-ed1-f50.google.com (mail-ed1-f50.google.com [209.85.208.50]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Qtrq20ZyWz4D1n for ; Sun, 2 Jul 2023 01:21:02 +0000 (UTC) (envelope-from nparhar@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of nparhar@gmail.com designates 209.85.208.50 as permitted sender) smtp.mailfrom=nparhar@gmail.com; dmarc=none Received: by mail-ed1-f50.google.com with SMTP id 4fb4d7f45d1cf-51d9850ef09so3626162a12.0 for ; Sat, 01 Jul 2023 18:21:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688260860; x=1690852860; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FBOhlIR9H3QN8f4pff9iv31nOiq0S2Y0uzQvJHAJWkM=; b=Nx8NRzgqvpj0Jk4H9hOCVqMT7R1vmX6BD8sdQZTE/AaTrY8wb/eWB/+7TSugvAUr2B bg76RlAxrYul+/4eNdgOdjQGaX7mySGAop16fQgPhANDIl5zwupF/97fZpQLsAtYVk8K BIgngbNQt7U6P8OgF6xxhpq/ixOrIECBfiLekz377hUx4v1VqFJf2d3TOuUjPeXo5+Re ZYEapGcIy7LcwS24F+uNnYri2NfKh0f+IZHbhkQmmep5ZFlQ5+Qxy5cWEoh0wKwVWtZY pG541SOs6KSj+4f5ESM1NNQThwoSvxJy0+yvrcFVdsjAGPBiyTXHi8lQTmJuLOpL20eM YUFQ== X-Gm-Message-State: ABy/qLYVRQ4rQa2LxsjP/bjO5RAa7ZEFThatBv2xD/RuV/S8vrzt/44I uX5bkbL4D3ErhaN0WyES0ij4TZwG5mcCYUBZwl270U5cUfQ= X-Google-Smtp-Source: APBJJlESJzlKea/kVuSTvsLVBzYHRH2DEz5/3oL9fx2IgZQJsumWEKOQPGPX5PmOdn4/gXz8URJEy5VoUbaPqL9m548= X-Received: by 2002:aa7:ccc7:0:b0:51d:f589:9c7a with SMTP id y7-20020aa7ccc7000000b0051df5899c7amr2968730edt.17.1688260860072; Sat, 01 Jul 2023 18:21:00 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 References: <8fe00cbc-f218-a587-48d8-1612223ccd49@FreeBSD.org> In-Reply-To: From: Navdeep Parhar Date: Sat, 1 Jul 2023 18:20:48 -0700 Message-ID: Subject: Re: Chelsio NIC with RSS - Traffic distribution to different Queues To: Josef.Zahner1@swisscom.com Cc: freebsd-net@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; FORGED_SENDER(0.30)[np@freebsd.org,nparhar@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.208.50:from]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_IN_DNSWL_NONE(0.00)[209.85.208.50:from]; RCVD_TLS_LAST(0.00)[]; FROM_NEQ_ENVFROM(0.00)[np@freebsd.org,nparhar@gmail.com]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[freebsd.org]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4Qtrq20ZyWz4D1n X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N Hello, Please try this patch: https://people.freebsd.org/~np/norssq.diff It adds these sysctls to the driver. 1) hw.cxgbe.rsrv_norssq. This is what you originally asked for. 2) hw.cxgbe.rx_budget. This can be used to force the driver's RX to yield periodically. What kind of system (cores, memory, etc.) is this? Control packets are either getting dropped or the threads/timers responsible for sending or processing these packets are starved of CPU. It would be useful to monitor interface activity with "netstat -d -I " during the test. # sysctl hw.model hw.ncpu hw.physmem # netstat -dw1 -I cxl0 Try the settings listed below. nrxq=3DX might help in case the driver RX threads are hogging all the cores because all rx queues are heavily loaded. Set nrxq to something less than the number of cores in the system. rx_budget can be changed any time (try 64, 128, 256) and might improve the responsiveness of the rest of the system during load. (in loader.conf) hw.cxgbe.nrxq=3D2 (3 if you've patched the kernel and set n= orssq) hw.cxgbe.rsrv_noflowq=3D1 hw.cxgbe.pause_settings=3D0 hw.cxgbe.cong_drop=3D1 (2 would be better but needs a recent dri= ver) hw.cxgbe.rsrv_norssq=3D1 (needs patch) hw.cxgbe.rx_budget=3D128 (needs patch) Let us know how it goes. Regards, Navdeep On Thu, Jun 29, 2023 at 5:53=E2=80=AFAM wrote: > > Can you tell me which netstat command you have in mind? I tried =E2=80=9C= netstat -Q=E2=80=9D, it shows a few drops but not that much that it would e= xplain the CARP drops. What I can tell you is, that especially CARP on the = corresponding server is just sending out packets as long as it is the maste= r box and CPU0 load is below 100%. It doesn=E2=80=99t receive any CARP traf= fic at all, just normal network traffic. What I see is, that those CARP pac= kets are not sent anymore if CPU0 has 100% load -> if that happens the serv= er switches to standby and the traffic is gone on the machine. So because o= f this behavior we would like to have an option, which allows us to have Co= ntrol Plane Traffic (LACP, CARP,=E2=80=A6?) in RSS RX queue 0 and nothing e= lse. Question is, what would Control Plane traffic be. Hopefully as well CA= RP/VRRP,=E2=80=A6 > > We tried hw.cxgbe.cong_drops=3D1, but it doesn=E2=80=99t help in our case= . > > Can you explain a bit what your patch will do? Am I right that you will p= ost the link later on here? > > Cheers Josef