From nobody Mon Dec 8 21:44:08 2025 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4dQFqk75Npz6KRBv for ; Mon, 08 Dec 2025 21:44:18 +0000 (UTC) (envelope-from vegeta@tuxpowered.net) Received: from mail-ej1-x62a.google.com (mail-ej1-x62a.google.com [IPv6:2a00:1450:4864:20::62a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4dQFqj3fMJz3C8T for ; Mon, 08 Dec 2025 21:44:17 +0000 (UTC) (envelope-from vegeta@tuxpowered.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=tuxpowered-net.20230601.gappssmtp.com header.s=20230601 header.b=22fiq2zM; dmarc=none; spf=pass (mx1.freebsd.org: domain of vegeta@tuxpowered.net designates 2a00:1450:4864:20::62a as permitted sender) smtp.mailfrom=vegeta@tuxpowered.net Received: by mail-ej1-x62a.google.com with SMTP id a640c23a62f3a-b73161849e1so78825966b.2 for ; Mon, 08 Dec 2025 13:44:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tuxpowered-net.20230601.gappssmtp.com; s=20230601; t=1765230250; x=1765835050; darn=freebsd.org; h=in-reply-to:autocrypt:references:cc:to:content-language:subject :from:user-agent:mime-version:date:message-id:from:to:cc:subject :date:message-id:reply-to; bh=odlaVl98Q11fBDq/TUOvW4gfRTCF/0kZUJ/1iSRK2z0=; b=22fiq2zMUsJIfW8ZvVI1yWtII8xUZf22dRJnix8z6TkbKAQm7c3f1LCDMnnjD1C+8o 58iVRBD0v6cvucCm8vq9fys10yqImFYdLDKwHLcAFY6lBKUIKp4I0YYLCCT81uVu/zdd Z4UhaRXWJIDmsuKdsx7U1WKAjFC0EFg+7+0U+RPWwSmWlH1tkW05l6DpZ54w4xfmEn6W Q3WKgUFVzddEmkfPYq1Eizu5YiJCB9c4i4BzY+VcC+2twvd8vMKpYrYCcyQ3G3wStR4E 3MrENXeEDSwYh63FxJpQgkw6eM7hPpkPqUg2mSxH9lwkevIdmiZ7rOplfGAhoRWGPCCE 96gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1765230250; x=1765835050; h=in-reply-to:autocrypt:references:cc:to:content-language:subject :from:user-agent:mime-version:date:message-id:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=odlaVl98Q11fBDq/TUOvW4gfRTCF/0kZUJ/1iSRK2z0=; b=WF3Sftb/KiLMxp8K4ArPE08W6lqb6geiEr2P4apAB0z7nAgzFPt75wF8xwl4YG0+wj JsCmFucXinhQSUm+DBPyjT+hcUkG6TPrjJhyOotgJFVAo3+LpgJe/U6gDXI9xDuLOVuu ivd+q2yqKYl8z5biwmYWQ1nBcNAiPLxqJERZL60wZ6x4tPeYkZXEJrgUF1xkR4EK6WiM pJYTXRNpTdhRHgtKibDxW1aMsWjl533YmwYun+89rrJZZsLYywdkDi1QGB5wlSia3erx FLUtEbpk2Rna361ulfxkXEod3USXvH5S0hJs8XY/hF8tCTtsOZqK9exLR6anePBOBia/ yq2Q== X-Gm-Message-State: AOJu0YzWIMtpohcgXeH2gA4deHwcqK7+5bAKhz68AyMVveTzNx8AyShU PwvJmys3hxaFx1Bj9Hk6J7k7b3HXdplcgEMtBv3jL1vQ7ADWDdoQupzeFii3+RctSdN7ELX6tZ9 wu/dx X-Gm-Gg: ASbGncueenvRs+HduE+lafJUc9RFjq9r/OBu/rVdyjJA1b1vRtCAehUrN+BKi1a65X5 2x7GXrLNY4Lkzh07ugQfqO8xhbNZB0WrbDFindQPzdfDLEv9DOHTR9Ye32ip/NXNNoeeizDnXtF epQKDr5MHCWZAP8md3rtJr0tNlW+ATrdsqVGlPv3E/5I68O2CEwICbxhuYOmy0VkdlIRh1DeiL0 zH9PO8Q9q9V+tK7uUMV49XSWo0E9l1yeyT8k6VaTYkFZfZfcY6tI8GoqPQlkjSw/ZjJsar7wAa/ 4kIO2HF/JaNMKVEOAKd2i3FY9zyACUcaf3Muinao/CWJr355QMALbY7PIbjadyviI6vaHtignf8 X6s3WCvnK+1zsRMEWUwemIUeisy2oVyNr2bPjpa8m1Ggxwnn3gvt0cXDBFOILKz/v3lrzmxSGLw TBH5BB4KvcqoDppIxdNpmZrXMbvrM86KfpaJhTp4YTrXgPA71N1n/OVOQJrOv12hhJMT8ViKKT7 a8lSajuUdE= X-Google-Smtp-Source: AGHT+IH68qVK2W+X3AjtpcdA0DG/WH1l7AftFspRA7GmHNa6uz1xmYCyFSaULsyBe3H8nm7iKuKFxQ== X-Received: by 2002:a17:907:3f1c:b0:b79:fe3a:6d3 with SMTP id a640c23a62f3a-b7a24777343mr865179566b.37.1765230250068; Mon, 08 Dec 2025 13:44:10 -0800 (PST) Received: from [192.168.178.122] (149.224.164.71.dynamic-pppoe.dt.ipv4.wtnet.de. [149.224.164.71]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-b79f4a2f401sm1188964666b.65.2025.12.08.13.44.09 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 08 Dec 2025 13:44:09 -0800 (PST) Message-ID: Date: Mon, 8 Dec 2025 22:44:08 +0100 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: Kajetan Staszkiewicz Subject: Re: RSS causing bad forwarding performance? Content-Language: en-GB To: Konstantin Belousov Cc: net@freebsd.org References: Autocrypt: addr=vegeta@tuxpowered.net; keydata= xsFNBGSvtwgBEADIx3vgFBaDsFD4MOGIsWSmHag9q8x3J5OrqBR9aIdeeDW4ghnPM1NgD8EQ HQvaAufffQ/vYXSWWJyDdquVARWprEXXQIMQZcDhw0pHtSrNK6NFF5UWfBkxYxAr2hTlRp0b R7QZk3ezGUElBpf+SJq5cCOy//32hnzJiKb+5hlL0QOheWKwKignhLckW8Yat+kjhsxw7pR8 vn/XSCwyejx3I8v2DZsTuXVOvbKr6kNwDryjl6JJwKFoQ/aNUeD7dmLP2ieB9HCHBBBIi16Z JcUCyJw8LI6GPrfr5zPEP38Up/psDQWoldbO2Kf5DyCN2HGFKLrK9StyjiMs4dgaA0ZXxIdn JTzdAP6+d1qIfvv5mGhbqTvHgX6ReR7l93eE3Q6WJqGiuFGUtKdU5qaRHd4IdbFnhNK/rWjg ZoKAlZwhnZ9BWZC8Vb9DznURYQUubt2Gr7Sutt0043d/WoWyGS2p7dEfXaeE1WE7n/6KqbBU zG/rF/20eeT0lmrNAy9pgFD5WmTtzHnljBzQSBDMTxZP3iEmFa0pXP+Ch/H26AxV99MXs7Tz Xj6VF5NKcIJ67m1pwJSW2vO9UhL2OVBJI3571C+9qn52QJjZdm4R4gHpgjbr4EoCUdlchCa1 iUQ1gV6SJI70WqgwmVprYwvaN1Rdld2iQFX+W6aOq6be1VzrwQARAQABzSxLYWpldGFuIFN0 YXN6a2lld2ljeiA8dmVnZXRhQHR1eHBvd2VyZWQubmV0PsLBlwQTAQgAQQIbAwUJB4YelwUL CQgHAgYVCgkICwIEFgIDAQIeAQIXgBYhBH0hCHMWPZA3mb0mbICq13+m8wBTBQJkr7gjAhkB AAoJEICq13+m8wBT14sQAKj1sG3yLeRfoKmmMgdbCErSrEg0uCChvWhRz/PCNfJB4SrUfSBj unM56CeCVUf1SBI7cq6tJDujMor433OpiuZvtlgJezfoeyTvgjiGshVnstNAik77+B6lnhvJ VwA7O3aT19kW/wUiVx9bATAleE4SQjyXq1z0onzh+FLeYZucfzISzUgOx2Ggb/eseDZ+v2re ja47WmMl/iU6ERvG3+GbmgZuYGRDIhzsa0l1YFzMrCmqrZ6ysW2JMwH+wkbw94yyLmF1k7uU KkRrejDiQjDk8Db2Smf7MaLGOCQGrz0Q2vSuETIavw2zQYs0bsQLuhV2/TlXegdfbe4wNhsD t4Zs2KEr8lHrXfIckxDn/vwlh2TWnPLQqlN13dctesfK/HFWqReIhfYu2B9WQCugLR2NAlO+ hw9wuOzBu8SfOX+CIcqHfX2Q+c7KrHFSsscENu2QnE27my5vqjkig4cpjZDLitKTyqKm8UNI f2O1xF137zA5byn/4rQFlfn8LbhuPdLBexvasjIQzuSgTZZ7cjUqbXFXssYsU0CFUHCoH5yF VrW8RGvx+W1l2nZQr03cZEoQEL+La4+LIRiuwFfohpz5xCsP0GdBDVIinC9vAkW7I6Y6ssCv ykMhaOGXZzs8mR47KCt6aFPX3vir9WmHQvHvSXaSxLNzfzmwl1e1hXD1zsFNBGSvtwgBEACw 4wl+FEyUehwSjs6/jhECE9r4fzwG+nUg1Q2ct8BneAjjUV/0UcMPQtphIGKqlJTxnxIEiz8D R3kb3Y535qkAeAU4RV8ONCUrJLyXoLei/Ymk7161Gui9x3AB1Z2Yi3x76MuRAFH7QIAxhXYo MN97IpgFDrv/ALwCD/eROFWEm5vNP8fvvpKBxtNaolebXWMfSFo3GJ8C73x+L8vW3D0uOp43 9MKUVAm6SMZXvYQA2P5+q15gxVUs0uhT69gHTrUMPHqPvARxZK5vpY+n3Phys9CZw84WaXcz qLjvmpKqqs/ody3r7caXZcN7eg3sihI0ud6R3UufM4WJ1UV8YLdwIi8dRMx0ozzjw+3E5ji0 gatXhhdZ9N7MsEOfy2o4IxukxJSvsDO9WRqIY2PgyXHlpiM026hhXiJRyCeV0TN1MAwId8YM 2+Ujce9n+Cu78d8+1lLVx82kvArm5zEL/Dj9b4SAZbyzQd5JzkiEWcYtZvTBG+NiAXgm9DR9 i4IC0TuEXfxT+vuriDKYhlyXzPhvaCngIkQ574YwGOrbjfCsSvZCrrSHtb+Mw1uC7kNvegfW 9ZUegD7knKXCt+4AX1xP27JB+ERdFoi7Ri7ROZLJB3Ne8oDS/aN40roKHj8mkm15lAMwrYB1 7ct/J8UCfQH4eagW8SwS2M6Tut6B4VWG+wARAQABwsF8BBgBCAAmFiEEfSEIcxY9kDeZvSZs gKrXf6bzAFMFAmSvtwgCGwwFCQeGHpcACgkQgKrXf6bzAFNbXRAAgFwTrMTEZDO79izcm/uU uGBoa3SKkxIwfhjYWwkHoLyr9P8fqRX6NjW///e8YWdLmf1jBESRnQYEbuSvociDpc7CJ57f 3GaKlHZs2ci5u0tqM0H/VKI/cAuPBGXli/unnbozlsU1fU4uZfY/4Tl2P8FNApJh2vbNalIt Mc9l0Iz3d5URPWAe7Pnb29tEVu5TNR/bJg7ihLsTY18XcePkHRRrnPF9ui9egB5FbCAQ/VSg Pl7/kD/PkOT/3kc+C4RhddRMUmPxH0G4hvBPLRuvLgwtaj8vnurN1NxbllzK33ZWkvbUhIrw Qcv23jfhQmg/cpzsQKeyu7L53bgUc4+zYoq/wd3n9SMCO5vTMUrswNqmD4wyopjblCGSeKNy kiiFA70umn5tB+Ra8H5k+n/e2QluKns+DoIg1Hm5chk/emBG42JYTdrMhLFQGZnzJU3WJf1j cVzOFLcqDuq+IwRPJrCZXsLft2O79uU+zbla+RdWd0uIzjbM4R8jch470h5cK61kcaQ7UwSb OpdPyq9PqFl26x2g4jDNmFLAuQDEBJoxmZA2bNfQk+DwYYTuoTItN3F674nb8Fk+tQyXL6fl 5CXOJOTJArKmokrPLcr3HXGUQpfBzXRDuK0UKgn3m9UXq2laaODgswuoZqm0vqWuWIRMm84J Wbiwhrslf0hn78s= In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------9hYYcHnQELeKB44TQwnP7RDJ" X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.59 / 15.00]; SIGNED_PGP(-2.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_SHORT(-0.99)[-0.988]; MIME_GOOD(-0.20)[multipart/signed,multipart/mixed,text/plain]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[tuxpowered-net.20230601.gappssmtp.com:s=20230601]; RCVD_TLS_LAST(0.00)[]; HAS_ATTACHMENT(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~]; DMARC_NA(0.00)[tuxpowered.net]; ARC_NA(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; DKIM_TRACE(0.00)[tuxpowered-net.20230601.gappssmtp.com:+]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[net@freebsd.org]; PREVIOUSLY_DELIVERED(0.00)[net@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::62a:from] X-Rspamd-Queue-Id: 4dQFqj3fMJz3C8T This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------9hYYcHnQELeKB44TQwnP7RDJ Content-Type: multipart/mixed; boundary="------------OvOSabp5Sn5BRx3B3bD0M0xD"; protected-headers="v1" From: Kajetan Staszkiewicz To: Konstantin Belousov Cc: net@freebsd.org Message-ID: Subject: Re: RSS causing bad forwarding performance? References: In-Reply-To: --------------OvOSabp5Sn5BRx3B3bD0M0xD Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I think I've found the culprit. When using RSS the mlx5 driver binds queue interrupts to CPUs. To simulate what it would do, but without RSS, I've configured both NICs to have only one queue and then bound the RX NIC to CPU 1 and the TX NIC to CPU 2. I got 3.3Mpps forwarded this way. Then I've bound the TX NIC to CPU 1. The forwarding rate immediately dropped to 0.5Mpps. Both NIC queues on CPU 1: RX IRQ 546/s TX IRQ 546/s dev.mce.1.txstat0tc0.enobuf grows at 3.9M/s Forwarding at 0.5Mp/s mce0 queue 0 on CPU 1, mce1 queue 0 on CPU 2: RX IRQ: 403/s TX IRQ: 85000/s dev.mce.1.txstat0tc0.enobuf does not grow Forwarding at 3.3Mp/s Let's play with the TX NICs sysctls dev.mce.1.conf.tx_coalesce_mode=3D1 dev.mce.1.conf.tx_completion_fact=3D32 dev.mce.1.conf.tx_coalesce_pkts=3D128 dev.mce.1.conf.tx_coalesce_usecs=3D128 TX IRQ: 4300/s dev.mce.1.txstat0tc0.enobuf grows at 1.6M/s Forwarding rate 2.19Mp/s dev.mce.1.conf.tx_queue_size=3D16384 dev.mce.1.conf.tx_coalesce_mode=3D0 dev.mce.1.conf.tx_completion_fact=3D512 dev.mce.1.conf.tx_coalesce_pkts=3D128 dev.mce.1.conf.tx_coalesce_usecs=3D128 TX IRQ: 7700/s dev.mce.1.txstat0tc0.enobuf does not grow Forwarding at 3.39Mp/s Now the TX IRQ can be set back to CPU 1 and forwarding rate is at 3.27Mp/= s. Lesson learned: don't forget about interrupt coalescing on the TX side to= o. --=20 | pozdrawiam / regards | Powered by Debian and FreeBSD | | Kajetan Staszkiewicz | www: http://tuxpowered.net | | | matrix: @vegeta:tuxpowered.net | `----------------------^--------------------------------' --------------OvOSabp5Sn5BRx3B3bD0M0xD-- --------------9hYYcHnQELeKB44TQwnP7RDJ Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature.asc" -----BEGIN PGP SIGNATURE----- wsF5BAABCAAjFiEEfSEIcxY9kDeZvSZsgKrXf6bzAFMFAmk3RqgFAwAAAAAACgkQgKrXf6bzAFOx 9w/9EyJzbSWJDES0UFYUxkxlZZ9k0PixkbM7AVcHgrN3idBXjxrB45lpiTpWwbDDi+wzR08qycC7 Tb/KrZ5DaaDuu1t9PN/RWihPSVUTtseYtSng9MRxr0rZieB/t+gfNQcXnR7wjHEXkZC84v1+oVFB Atr44upFxk0618GVd+JH+ReZx49TUb5nmYJoCWGaJF7+R9mjLJeGsCdRavY0OZqZqXY4jhDHser4 dytmJS1651KKHNgjqEVtcvQOOstNjrGCCkN2lO5Ts35aNrfnqYwQhiMWpjkY1+D4mTO+lum2bufp jmFlGr0Ycv2aeJvgFGaeMARmo2ryR43MSXRec5bbTKdVcN6BWzmGB3oHvzvpPj0fab/Ok8jJS/w6 UcyxV2Yspqx1JDRGC8Lb1aTjivN6UhWLpin+Gaep1NcYMvimagL0+2LVHcc7DrRhYUeI28BRcUYb ZjE1V2MyeiNY0MIPj+w/BjZcszWK1t+4YLMz1w0MdFnbbxbNbzdZpCTgsB2cT0Hx5W+ml+m758ng fxEbDf4RF3k8I0meGXhQAvpTsAAdpvmJ3pysAzMm1LO6U3L0E4KJ3J0X3WeFZhRRGrNEEw+kf0X/ eZBWjOs/QSWY2C+5kF7BZnoy15AcyWTXw8tk2sgQwCwRA4ntM1o+golI8NW8ExmvKHv/GwaSlaw7 f+I= =TCfb -----END PGP SIGNATURE----- --------------9hYYcHnQELeKB44TQwnP7RDJ--