From nobody Mon Dec 8 16:33:25 2025 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4dQ6xG2vT3z6K3hV for ; Mon, 08 Dec 2025 16:33:38 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-qt1-f173.google.com (mail-qt1-f173.google.com [209.85.160.173]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4dQ6xF69zxz3LNl for ; Mon, 08 Dec 2025 16:33:37 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-qt1-f173.google.com with SMTP id d75a77b69052e-4eddfb8c7f5so46624561cf.1 for ; Mon, 08 Dec 2025 08:33:37 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1765211617; x=1765816417; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=1PTPb+iqCX11rayrYer5IhVx8PfqXMOk/bORCRFoFEg=; b=X1l2avJ9nef28ZiiVNjRQe9PXyFFnei9VarZjj1pp4tKw8DxteFPVQ3geHPeZ6k/bp atTxzJ0sLbBKFI8bM2hypca3PpOA2aWFqaYYj3iXJT4tvVVdc+cTCwcJabBhpchUhq6h KV5gWC7UbF0VbdOhcn7aqjD5euOLyUlEUTdTmz0GbjSTdar0iS6tMFifi5r168PxjKyp yL+fPqRCym9dGtQct66HdM88S+VyKrLwqUFgGzlzmHWOi7MQi7wIUxuaJE+TFZV37x81 eEPlU3dYRTz4Y0kWGiraDKjQczdvvvFCBNYFrX2RhLE4eRDhnmPXBGWcuZzGOnXnwbPi KxmA== X-Forwarded-Encrypted: i=1; AJvYcCXQN+31cbwkfk+J/5q34C7ADIzJBs2SmZ3XZQtlfYdeRm5wlswr9b4sD9hWv7R1/vMlJZE=@freebsd.org X-Gm-Message-State: AOJu0YwbIYrB28773j/fAU4sLx/LsYsnB6upjNLV5Kx7KdkMbwmNqrig SZxSxXQxqU8WlBuYYWf7L171NL6qFeuqrzOUTaji5eMlOmYzwbcL4wvCQOyqY9CrEe1kkYozNnc 66+QUXjXw+QLVb6suBZwq+pxmTh2jbHj9/qtc X-Gm-Gg: ASbGnctFXgIbCki1inDLvpMMe0Zd4mPbMh0CdHj+Bb7ALWMwALq+kdHpYkexs0qIhhg 9qVLl0DIL9J7PLJwXEfWUZ1+njRJTAk1XD5Qpi7unOSuwipqZhwbwFYhHbT5Xcx1ya18bh74/kf 69pDocHPU84A6MTCatUIfu5/p8uGroJHPF3macvNWPfdSL7IWU9T9qi+GEe6gE7SEY83hvfUC+B HwwyCOen9qK8hu4bVyoNmoewpT5R2ui7mQi1bSDX+C69ZRKCoIePa/+zLR59zXHKDlpspSqEub6 sQ1BZ1aaWCEo8mQmO4+3HqAGWjWkyw== X-Google-Smtp-Source: AGHT+IHlXi4zwfecTFyeXRKdo0KJmIr8FuJfZzI5fm64buppBaCzwT7P8Yp/xbbilfUKVCCmihzRDxZ4ixkRE76YgSw= X-Received: by 2002:a05:622a:4a18:b0:4ee:2868:4e4a with SMTP id d75a77b69052e-4f03fd4fe1emr132461581cf.14.1765211616771; Mon, 08 Dec 2025 08:33:36 -0800 (PST) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Adrian Chadd Date: Mon, 8 Dec 2025 08:33:25 -0800 X-Gm-Features: AQt7F2pJAB0TVsD9lTrKJ5Z2jactmAeKbq6BbBowW9wAqZn6eKOdkPdm2g1_AwU Message-ID: Subject: Re: RSS causing bad forwarding performance? To: Kajetan Staszkiewicz Cc: Konstantin Belousov , net@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 4dQ6xF69zxz3LNl On Mon, 8 Dec 2025 at 06:24, Kajetan Staszkiewicz wrote: > > On 2025-12-08 00:55, Konstantin Belousov wrote: > > > It is somewhat strange that with/without RSS results differ for UDP. > > mlx5en driver always enable hashing the packet into rx queue. And, > > with single UDP stream I would expect all packets to hit the same queue. > With a single UDP stream and RSS disabled the DUT gets 2 CPU cores > loaded. One at 100%, I understand this is where the interrupts for > incoming packets land and it handles receiving, forwarding and sending > the packet (with direct ISR dispatch) and another around 15-20%, my best > guess that it's handling interrupts for confirmations of packets sent > out through the other NIC. > > With a single UDP stream and RSS enabled the DUT gets only 1 CPU core > loaded. I understand that thanks to RSS the outbound queue on mce1 is > the same as inbound queue on mce0 and thus the same CPU core handles irq > for both queues. > > > As consequence, with/without RSS should be same (low). > > It is low for no RSS, but with RSS it's not just low, it's terrible. > > > Would it be UDP which encapsulates some other traffic, e.g. tunnel that > > can be further classified by the internal headers, like inner headers > > of the vxlan, then more that one receive queue could be used. > > The script stl/udp_1pkt_simple.py (provided with TRex) creates UDP > packets from port 1025 to port 12, filled with 0x78, length 10 B. My > goal is to test packets per second performance, so I've choosen this > test as it creates very short packets. > > > BTW, mce cards have huge numbers of supported offloads, but all of them are > > host-oriented, they would not help for the forwarding. > > > Again, since iperf stream would hit single send/receive queue. > > Parallel iperfs between same machines scale. > > It seems that parallel streams forwarded through the machine scale too. > It's a single stream that kills it, and only with option RSS enabled. RSS was never really designed for optimising a single flow by having it consume two CPU cores. It was designed for optimising a /whole lot of flows/ by directing them to a consistent CPU mapping and if used in conjunction with CPU selection for the transmit side, to avoid cross-CPU locking/synchronisation entirely. It doesn't help that the RSS defaults (ie only one netisr, not hybrid mode IIRC, etc) are not the best for lots of flows. So in short, I think you're testing the wrong thing. -adrian