From nobody Mon Dec 8 14:23:42 2025 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4dQ43Y2zD0z6Jsv0 for ; Mon, 08 Dec 2025 14:23:53 +0000 (UTC) (envelope-from vegeta@tuxpowered.net) Received: from mail-ej1-x62f.google.com (mail-ej1-x62f.google.com [IPv6:2a00:1450:4864:20::62f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4dQ43X47NFz3vCp for ; Mon, 08 Dec 2025 14:23:52 +0000 (UTC) (envelope-from vegeta@tuxpowered.net) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x62f.google.com with SMTP id a640c23a62f3a-b73161849e1so1069436766b.2 for ; Mon, 08 Dec 2025 06:23:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tuxpowered-net.20230601.gappssmtp.com; s=20230601; t=1765203825; x=1765808625; darn=freebsd.org; h=in-reply-to:autocrypt:content-language:references:cc:to:subject :from:user-agent:mime-version:date:message-id:from:to:cc:subject :date:message-id:reply-to; bh=1TikoKEgAWbQn43dMdFEBDzvJ8fXrPyuWp6YZ0KbeFQ=; b=GBMM8RVaLSaNfqlozT4wGgm99KdKcJM+joLExf4/y85blqz5UP2Bwaa5rFAz24+hB+ LJTDkpoZW2HtgCO1qP8SCQ6gEsAaJLZq4N+8kxtp+b9IN8FYpqvGx68Y4ztWD9KnBI5g zqRzmhrYyBMGYvy8M1kcBdiv0T+akZ78rqA3vO+w0m/p/SFudrWDGDBMowWWxjJTbXHC vgvNd+bfxLFPkmU7HbeOslcBmWW5O1yBu9JToiQareA4AoyHGGQIJ9KTHLreanlzEquZ GzarZEiD++n4HfqA17GEmTnk+pPN+ebDQqPFKQQRbV6xP2rUjxJBT7hAbo6Yhhp7L52N vysQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1765203825; x=1765808625; h=in-reply-to:autocrypt:content-language:references:cc:to:subject :from:user-agent:mime-version:date:message-id:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1TikoKEgAWbQn43dMdFEBDzvJ8fXrPyuWp6YZ0KbeFQ=; b=DIeE411aNvZyMplSdUuUaRZwWExM0GE5/VkKGdy5ZloKwLZtudYuNSxjUYdBPvewUZ vFxxbSvhEfE8yRfYXHpwrqrmVC1G1osmNmOR5UgPZYBp5q7QA0g04qjdzX7NPxlyaqSN KpzZocQiA8fNcvXLaszm/5wYY0cBuNwjVyaJTpIP8Xx2pnwPL6KDZkGRNUouwhK3Z6SE 7QejECV8VOy5u5c8Z8aqIcXlZRmKvyokpSHRWJf6y78uEbx+581Mw79JpwHaycj7lrc7 airvMBQsDp1Ck0f6vi0rLspuafgJB1PIiqKzI8vMzFvmdWTqTmaalujHz77+c8LHpc8+ rkbA== X-Gm-Message-State: AOJu0Yz/Q2kPAUCuBuOLS1QpHMv9FeIziJbmHpyNXMIwcdhgNlz/JdKF Wckqph88aXoCqeF4Gnedr+cPegY/+H9L+WE62XXnYDuRgHQw1HzH4HPw2VvJHJSS3c0= X-Gm-Gg: ASbGnct4NYjkVj3fOBnjw4CLUa9Gi52QBoLAI4zCQwjqcfBOucgi8WuEqcSRs85GGVQ s7jdI/7NibZtyj0YOba6t3V30Cm90Th3irIwbG0uNqBIKVzfrQ1RUQPvTFZW32g51/QeJydynld SJsQsJFwNd18N7y7dOKUNOxpiHzs/qMyVSMAT7gM/y1dt4qdGY3qTvE4lITJubvlc8puK68OGm0 wLwVTMKhRLjwaPb1OE1mF56D18d5fkW99XBT24PwVXHmjz5WEPmV57ttNZfpB/2k9pb9rWfh8iT 8CNAOOy8DV7sP7TvH+llKjqDXWeGerX11Y0ceD63pEcsQh2E/6RYKA2FU7gQV0r/7ghkGZ9CJud Su6zApqnzHkVGloWxI3mp6PfC26TWdD4n/9tp38d6kV2Yj/3NwBt44PetHO1PDEq+kZqZJN9qiD KB/hJIwHJmu+/l18njTyh04GwFx906jDAk0r5lv6U1HDnKfAl6e+iJhlXXSYvupp5Ll0V0kKjOi 4AgxBT2sMM= X-Google-Smtp-Source: AGHT+IGxxXMefjvSvNWJGDgEOTsFIkYu4chzTmORnS5A+WLLcqoOhBQ5SwFDfjSuUoent62J3dVhJg== X-Received: by 2002:a17:906:c104:b0:b73:80de:e6b2 with SMTP id a640c23a62f3a-b7a2477ab1amr845116966b.31.1765203824693; Mon, 08 Dec 2025 06:23:44 -0800 (PST) Received: from [192.168.178.122] (149.224.164.71.dynamic-pppoe.dt.ipv4.wtnet.de. [149.224.164.71]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-b79f4457f9dsm1129811466b.8.2025.12.08.06.23.43 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 08 Dec 2025 06:23:44 -0800 (PST) Message-ID: Date: Mon, 8 Dec 2025 15:23:42 +0100 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: Kajetan Staszkiewicz Subject: Re: RSS causing bad forwarding performance? To: Konstantin Belousov Cc: net@freebsd.org References: Content-Language: en-GB Autocrypt: addr=vegeta@tuxpowered.net; keydata= xsFNBGSvtwgBEADIx3vgFBaDsFD4MOGIsWSmHag9q8x3J5OrqBR9aIdeeDW4ghnPM1NgD8EQ HQvaAufffQ/vYXSWWJyDdquVARWprEXXQIMQZcDhw0pHtSrNK6NFF5UWfBkxYxAr2hTlRp0b R7QZk3ezGUElBpf+SJq5cCOy//32hnzJiKb+5hlL0QOheWKwKignhLckW8Yat+kjhsxw7pR8 vn/XSCwyejx3I8v2DZsTuXVOvbKr6kNwDryjl6JJwKFoQ/aNUeD7dmLP2ieB9HCHBBBIi16Z JcUCyJw8LI6GPrfr5zPEP38Up/psDQWoldbO2Kf5DyCN2HGFKLrK9StyjiMs4dgaA0ZXxIdn JTzdAP6+d1qIfvv5mGhbqTvHgX6ReR7l93eE3Q6WJqGiuFGUtKdU5qaRHd4IdbFnhNK/rWjg ZoKAlZwhnZ9BWZC8Vb9DznURYQUubt2Gr7Sutt0043d/WoWyGS2p7dEfXaeE1WE7n/6KqbBU zG/rF/20eeT0lmrNAy9pgFD5WmTtzHnljBzQSBDMTxZP3iEmFa0pXP+Ch/H26AxV99MXs7Tz Xj6VF5NKcIJ67m1pwJSW2vO9UhL2OVBJI3571C+9qn52QJjZdm4R4gHpgjbr4EoCUdlchCa1 iUQ1gV6SJI70WqgwmVprYwvaN1Rdld2iQFX+W6aOq6be1VzrwQARAQABzSxLYWpldGFuIFN0 YXN6a2lld2ljeiA8dmVnZXRhQHR1eHBvd2VyZWQubmV0PsLBlwQTAQgAQQIbAwUJB4YelwUL CQgHAgYVCgkICwIEFgIDAQIeAQIXgBYhBH0hCHMWPZA3mb0mbICq13+m8wBTBQJkr7gjAhkB AAoJEICq13+m8wBT14sQAKj1sG3yLeRfoKmmMgdbCErSrEg0uCChvWhRz/PCNfJB4SrUfSBj unM56CeCVUf1SBI7cq6tJDujMor433OpiuZvtlgJezfoeyTvgjiGshVnstNAik77+B6lnhvJ VwA7O3aT19kW/wUiVx9bATAleE4SQjyXq1z0onzh+FLeYZucfzISzUgOx2Ggb/eseDZ+v2re ja47WmMl/iU6ERvG3+GbmgZuYGRDIhzsa0l1YFzMrCmqrZ6ysW2JMwH+wkbw94yyLmF1k7uU KkRrejDiQjDk8Db2Smf7MaLGOCQGrz0Q2vSuETIavw2zQYs0bsQLuhV2/TlXegdfbe4wNhsD t4Zs2KEr8lHrXfIckxDn/vwlh2TWnPLQqlN13dctesfK/HFWqReIhfYu2B9WQCugLR2NAlO+ hw9wuOzBu8SfOX+CIcqHfX2Q+c7KrHFSsscENu2QnE27my5vqjkig4cpjZDLitKTyqKm8UNI f2O1xF137zA5byn/4rQFlfn8LbhuPdLBexvasjIQzuSgTZZ7cjUqbXFXssYsU0CFUHCoH5yF VrW8RGvx+W1l2nZQr03cZEoQEL+La4+LIRiuwFfohpz5xCsP0GdBDVIinC9vAkW7I6Y6ssCv ykMhaOGXZzs8mR47KCt6aFPX3vir9WmHQvHvSXaSxLNzfzmwl1e1hXD1zsFNBGSvtwgBEACw 4wl+FEyUehwSjs6/jhECE9r4fzwG+nUg1Q2ct8BneAjjUV/0UcMPQtphIGKqlJTxnxIEiz8D R3kb3Y535qkAeAU4RV8ONCUrJLyXoLei/Ymk7161Gui9x3AB1Z2Yi3x76MuRAFH7QIAxhXYo MN97IpgFDrv/ALwCD/eROFWEm5vNP8fvvpKBxtNaolebXWMfSFo3GJ8C73x+L8vW3D0uOp43 9MKUVAm6SMZXvYQA2P5+q15gxVUs0uhT69gHTrUMPHqPvARxZK5vpY+n3Phys9CZw84WaXcz qLjvmpKqqs/ody3r7caXZcN7eg3sihI0ud6R3UufM4WJ1UV8YLdwIi8dRMx0ozzjw+3E5ji0 gatXhhdZ9N7MsEOfy2o4IxukxJSvsDO9WRqIY2PgyXHlpiM026hhXiJRyCeV0TN1MAwId8YM 2+Ujce9n+Cu78d8+1lLVx82kvArm5zEL/Dj9b4SAZbyzQd5JzkiEWcYtZvTBG+NiAXgm9DR9 i4IC0TuEXfxT+vuriDKYhlyXzPhvaCngIkQ574YwGOrbjfCsSvZCrrSHtb+Mw1uC7kNvegfW 9ZUegD7knKXCt+4AX1xP27JB+ERdFoi7Ri7ROZLJB3Ne8oDS/aN40roKHj8mkm15lAMwrYB1 7ct/J8UCfQH4eagW8SwS2M6Tut6B4VWG+wARAQABwsF8BBgBCAAmFiEEfSEIcxY9kDeZvSZs gKrXf6bzAFMFAmSvtwgCGwwFCQeGHpcACgkQgKrXf6bzAFNbXRAAgFwTrMTEZDO79izcm/uU uGBoa3SKkxIwfhjYWwkHoLyr9P8fqRX6NjW///e8YWdLmf1jBESRnQYEbuSvociDpc7CJ57f 3GaKlHZs2ci5u0tqM0H/VKI/cAuPBGXli/unnbozlsU1fU4uZfY/4Tl2P8FNApJh2vbNalIt Mc9l0Iz3d5URPWAe7Pnb29tEVu5TNR/bJg7ihLsTY18XcePkHRRrnPF9ui9egB5FbCAQ/VSg Pl7/kD/PkOT/3kc+C4RhddRMUmPxH0G4hvBPLRuvLgwtaj8vnurN1NxbllzK33ZWkvbUhIrw Qcv23jfhQmg/cpzsQKeyu7L53bgUc4+zYoq/wd3n9SMCO5vTMUrswNqmD4wyopjblCGSeKNy kiiFA70umn5tB+Ra8H5k+n/e2QluKns+DoIg1Hm5chk/emBG42JYTdrMhLFQGZnzJU3WJf1j cVzOFLcqDuq+IwRPJrCZXsLft2O79uU+zbla+RdWd0uIzjbM4R8jch470h5cK61kcaQ7UwSb OpdPyq9PqFl26x2g4jDNmFLAuQDEBJoxmZA2bNfQk+DwYYTuoTItN3F674nb8Fk+tQyXL6fl 5CXOJOTJArKmokrPLcr3HXGUQpfBzXRDuK0UKgn3m9UXq2laaODgswuoZqm0vqWuWIRMm84J Wbiwhrslf0hn78s= In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------npnlp1J6uaFFyvI8poIH2Io8" X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 4dQ43X47NFz3vCp This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------npnlp1J6uaFFyvI8poIH2Io8 Content-Type: multipart/mixed; boundary="------------ecD3I21BcKXgGHCd6W8Eqn09"; protected-headers="v1" From: Kajetan Staszkiewicz To: Konstantin Belousov Cc: net@freebsd.org Message-ID: Subject: Re: RSS causing bad forwarding performance? References: In-Reply-To: --------------ecD3I21BcKXgGHCd6W8Eqn09 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 2025-12-08 00:55, Konstantin Belousov wrote: > It is somewhat strange that with/without RSS results differ for UDP. > mlx5en driver always enable hashing the packet into rx queue. And, > with single UDP stream I would expect all packets to hit the same queue= =2E With a single UDP stream and RSS disabled the DUT gets 2 CPU cores loaded. One at 100%, I understand this is where the interrupts for incoming packets land and it handles receiving, forwarding and sending the packet (with direct ISR dispatch) and another around 15-20%, my best guess that it's handling interrupts for confirmations of packets sent out through the other NIC. With a single UDP stream and RSS enabled the DUT gets only 1 CPU core loaded. I understand that thanks to RSS the outbound queue on mce1 is the same as inbound queue on mce0 and thus the same CPU core handles irq for both queues. > As consequence, with/without RSS should be same (low). It is low for no RSS, but with RSS it's not just low, it's terrible. > Would it be UDP which encapsulates some other traffic, e.g. tunnel that= > can be further classified by the internal headers, like inner headers > of the vxlan, then more that one receive queue could be used. The script stl/udp_1pkt_simple.py (provided with TRex) creates UDP packets from port 1025 to port 12, filled with 0x78, length 10 B. My goal is to test packets per second performance, so I've choosen this test as it creates very short packets. > BTW, mce cards have huge numbers of supported offloads, but all of them= are > host-oriented, they would not help for the forwarding. > Again, since iperf stream would hit single send/receive queue. > Parallel iperfs between same machines scale. It seems that parallel streams forwarded through the machine scale too. It's a single stream that kills it, and only with option RSS enabled. --=20 | pozdrawiam / regards | Powered by Debian and FreeBSD | | Kajetan Staszkiewicz | www: http://tuxpowered.net | | | matrix: @vegeta:tuxpowered.net | `----------------------^--------------------------------' --------------ecD3I21BcKXgGHCd6W8Eqn09-- --------------npnlp1J6uaFFyvI8poIH2Io8 Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature.asc" -----BEGIN PGP SIGNATURE----- wsF5BAABCAAjFiEEfSEIcxY9kDeZvSZsgKrXf6bzAFMFAmk2328FAwAAAAAACgkQgKrXf6bzAFO2 vA/9E3GMFqWceT4kTNuBVbzs/30JJXrtjf4nro9TeUrRLHywuGPaEL9ozc14BDtjiZE63BUX7W9y cKbd9Zd0ndQjLUW4IldBGyCUW9B5edfls6+rqFOJ6kbpt/SfFYucYU7rS9Erj/0js1VQj6m03/T/ 5u9yedHveavK7e69afQXGkczLmphTpXUy/AHrJhWVX/aSzlpVkuJju+4hjzhyyR7A0XJ6IH4DXzQ ubzD6P9pUB3CmiwIW5lfBsQlPSUEI1wndycfH1juvl/6wLKfqV4o+GN3AF20Tkiu6eTTf0CB4yoI yOSd7AOdV78MNWKJEMyk+UsXStX1EkXyYMmm06OHC3YrzOtexYOtijDE49eIh1tJddq2CeNWpkzl d+AQ7+4c8eHk3ZKyuo3FAcnCqvH7khFyAbCSP/AgvMfveCEkHSAVDpCBKkwHG7Zh7qR/gJI5lWzZ smyF/CSLY0Q9LLL3yN9dAlTAzHpW0X/Ily3nUK+TBs0mEmb+ZVqN/Yr0QgPjNIXOVKxT2vbNRvfR jcElCsNoiJ3kDnyMSZbPfkKX3fGa3hNFibZAtChIqADX40wkLmJPtiCSkZkt55fjrWWmC3hwwinV 4UotmV+IGVgXuPj6rz1N3G5bVQWrijmKoyhSeGrjiw7XJQn5QdkStkSc2l+LSOfp162kxLLZ6mZz 3dY= =YY0X -----END PGP SIGNATURE----- --------------npnlp1J6uaFFyvI8poIH2Io8--