From nobody Mon Sep 9 16:52:13 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X2Xtm5ln8z5TNvQ for ; Mon, 09 Sep 2024 16:52:16 +0000 (UTC) (envelope-from omegadraconis@gmail.com) Received: from mail-io1-xd36.google.com (mail-io1-xd36.google.com [IPv6:2607:f8b0:4864:20::d36]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X2Xtm1D7vz418H for ; Mon, 9 Sep 2024 16:52:16 +0000 (UTC) (envelope-from omegadraconis@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=McM5b7CO; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of omegadraconis@gmail.com designates 2607:f8b0:4864:20::d36 as permitted sender) smtp.mailfrom=omegadraconis@gmail.com Received: by mail-io1-xd36.google.com with SMTP id ca18e2360f4ac-82cdada0f21so75901939f.3 for ; Mon, 09 Sep 2024 09:52:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725900735; x=1726505535; darn=freebsd.org; h=in-reply-to:autocrypt:from:content-language:references:to:subject :user-agent:mime-version:date:message-id:from:to:cc:subject:date :message-id:reply-to; bh=WaxFWXSgPKZWBinsyRJbA5fWFRDVZ2jJPOnvR4foMnk=; b=McM5b7COVyt+iAQ4IHGDBrNCr45gkDnk5Xt5HCzVbJ6RwJxp0foQFl3cxAL7W5tlhZ S2kjYSc07GevKPkR+E3e2rs46dXK4EaeBf77cx8BgralTqTe5FK3MP7FxHhX7DHWoskL OZiBWh0/DpR00kQUIlBg5dIba5sNmeZqAD8K/2KWj81OiWkBPZwdddSA9zHs7vVZUFVS FO66Z0+reF4JlGTXKTdF9x0zGlZovPNG4F0AODoCRUmAeY8xgfdQ6ZkH9rxe5wS3/hFn 6Ahw1+4/XeZ0/YqjwlyEpXRWsBd7tIK6N7ES/EmwzmVtESAyPCjduB8+aUFDhoNbvPgV D7Mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725900735; x=1726505535; h=in-reply-to:autocrypt:from:content-language:references:to:subject :user-agent:mime-version:date:message-id:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=WaxFWXSgPKZWBinsyRJbA5fWFRDVZ2jJPOnvR4foMnk=; b=r9z3uWnmpXRTM0Zpn+8u4CQJRAHIdCZzANkdXG5+jI5Zrw9ihswCVhtgQSxJXoXOAf 58DkB01PJB73OtzhtTbjv18+YdJC6Epydvm2KszPd9SgChXH3vgHQdaVHiIXTfsuSaNA QinRFw/EZcxtRy7Sy89dT4OjcNhL4P05Qtov9VJTNLuOG89i5UEpPF3toohUzWQYMoiT WcG6YZ0M7LIUwoXMtWBdp1JXwoBd3ng0wpW3SHmPHHrAcaMSBY946gkveqShmdhVC4DX bl/ENJPi2MuTgV8PBWmGSAnEgCah2X2bFg1cP1FQeu4Bmwp/q2NBHUorPrhvJonMCQya ZNIA== X-Gm-Message-State: AOJu0YyHKpchizsCBE6gS7rmJqmGJbPdO7qIjOyuL1W5NwrQxPZk8Z/j J06XTh3dpdVC19NLZnU19Z8w988kxLzmP/eXNxyG4mzJCGaNut0zMBXtIwYC X-Google-Smtp-Source: AGHT+IGsRo8p34QFSVHL5O9hbeNfoBVJ0yFbcYThkaW+01uTbAOY6z/SEJ4UCKjxBJIttEiiW5zoSQ== X-Received: by 2002:a05:6e02:1a61:b0:39f:d10e:55e0 with SMTP id e9e14a558f8ab-3a04f0ee539mr131802855ab.19.1725900734530; Mon, 09 Sep 2024 09:52:14 -0700 (PDT) Received: from [192.168.10.217] (99-110-175-208.lightspeed.clmboh.sbcglobal.net. [99.110.175.208]) by smtp.gmail.com with ESMTPSA id e9e14a558f8ab-3a05900e2c8sm15093785ab.53.2024.09.09.09.52.13 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 09 Sep 2024 09:52:14 -0700 (PDT) Content-Type: multipart/alternative; boundary="------------AO23IMBYeIKUL9oSX56jt4qw" Message-ID: <2854f8ea-0827-4f3a-8269-1cca0cd8a201@gmail.com> Date: Mon, 9 Sep 2024 12:52:13 -0400 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Support for Firebox M270 and the Intel X553 MDIO To: freebsd-net@freebsd.org References: <9111F7D4-7362-4C84-B019-6B3E700DABF6@gmail.com> Content-Language: en-US From: Jason Hensler Autocrypt: addr=Jason@colddarkness.com; keydata= xjMEYh5OtxYJKwYBBAHaRw8BAQdAYRlCJw/Y2KWmAVzAXGH8fTmvcfgYWpqeHR5P6qDiJ9HN Jkphc29uIEhlbnNsZXIgPEphc29uQGNvbGRkYXJrbmVzcy5jb20+wosEExYIADMWIQRlCor0 cQmwcdlj1EgwixcsJVXnHwUCYh5OtwIbAwULCQgHAgYVCAkKCwIFFgIDAQAACgkQMIsXLCVV 5x/UlQD+LRZ/eUbknDJPhna0yW6yaOhAHf6ByiHgc+ATy1yQSokBAN6S7n6HXxIJ6aVwO+si sYXd6RghM7GQMT9QYOaP9ikHzjgEYh5OuBIKKwYBBAGXVQEFAQEHQG+iDmMbKujVJNcaZz6J 4WCeRXVBHli/BJhQX7ROZ8MbAwEIB8J4BBgWCAAgFiEEZQqK9HEJsHHZY9RIMIsXLCVV5x8F AmIeTrgCGwwACgkQMIsXLCVV5x87kAEA0Fv7GgP7VtklqYFIVMIghHcFXs6iZVx3HaG6nozP /DQBAK+NEu5LlePyqbT8h9LrfKJT3Z45/WwBPfQuwCsj0mII In-Reply-To: X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.94 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.95)[-0.950]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; XM_UA_NO_VERSION(0.01)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::d36:from] X-Rspamd-Queue-Id: 4X2Xtm1D7vz418H This is a multi-part message in MIME format. --------------AO23IMBYeIKUL9oSX56jt4qw Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Hello Peter, Sorry to dig up the old thread, not sure about the mailing list rules/etiquette on old threads. I'm looking to get a silicom i3000 up and running with FreeBSD (Opnsense). It has similar specs to the M270 with two X553 cpu ports and a Marvell 88E6190X switch. I found this thread and tried the patches in that you've linked to and also added the patch for the ixgbe_[write|read]_phy_reg_mdi_22, however I'm still not getting a /dev/mdio device. Did you make any progress on this one? Do you have any suggestions? At-least one user was able to get pf+ on it and /dev/mdio did appear. Thanks, Jason Hensler On 12/16/23 4:57 AM, Peter A Barlow wrote: > Hi Eric, > > Thank you. I’ve spent a few days researching this quite extensively. > I’m aware of this. It seems that some mods have been incorporated into > pf+ (closed source) probably in the ixgbe drivers to facilitate the > MDIO bus and detection of the Marvell switch. However, like some other > M270 owners, I don’t wish to invest in pf+ and would like to figure > out just how much work is involved in modifying the drivers to work > with the M270 unit under FreeBSD, or OPNsense. > > Yesterday I stumbled upon Intel’s DPDK project. If you look at their > mail archive you’ll find a number of mods to the X550 driver to > address this issue. I’m currently looking at this for clues. > [dpdk-dev] [PATCH v3 2/2] net/ixgbe : backplane port MDIO support > > mail-archive.com > apple-touch-icon-114x114.png > > > > > Peter > > >> On 15 Dec 2023, at 18:56, Eric Joyner wrote: >> >> On Fri, Dec 15, 2023 at 12:51 AM Peter A Barlow >> wrote: >> >> I’m looking at running FreeBSD on an old Firebox M270. >> It has a C3558 CPU with integrated X553 LAN controller which >> connects over MDIO to a Marvell 88E6190 switch. >> Out of the box the X553 backplane is detected but there doesn’t >> seem to be any attempt at probing the MDIO for connected devices. >> >> I’ve played around with the Intel ixgbe drivers, compiling the >> kernel etc to see if I can figure it out but I’m really >> struggling to understand what needs to be done. >> >> At this stage I’m reaching out to the community to see if anyone >> can clarify something for me….are there some fundamental changes >> or additions required to the drivers to make this work, or is it >> something that should work already but needs some options >> enabling or configurations tweaking. I’m reluctant to put more >> time into trawling through the code if it’s a ‘simple’ >> configuration issue. >> >> Any pointers would be very welcome. >> Thank you. >> >> >> I found this thread: >> https://forum.netgate.com/topic/154974/pfsense-on-watchguard-m270/112 >> >> I think the TL;DR is that you need pfSense Plus since the required >> software to get it to work isn't publicly available. >> >> - Eric > --------------AO23IMBYeIKUL9oSX56jt4qw Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit

Hello Peter,

Sorry to dig up the old thread, not sure about the mailing list rules/etiquette on old threads.

I'm looking to get a silicom i3000 up and running with FreeBSD (Opnsense). It has similar specs to the M270 with two X553 cpu ports and a Marvell 88E6190X switch. I found this thread and tried the patches in that you've linked to and also added the patch for the ixgbe_[write|read]_phy_reg_mdi_22, however I'm still not getting a /dev/mdio device. Did you make any progress on this one? Do you have any suggestions?

At-least one user was able to get pf+ on it and /dev/mdio did appear.

Thanks,

Jason Hensler

On 12/16/23 4:57 AM, Peter A Barlow wrote:
Hi Eric,

Thank you. I’ve spent a few days researching this quite extensively. I’m aware of this. It seems that some mods have been incorporated into pf+ (closed source) probably in the ixgbe drivers to facilitate the MDIO bus and detection of the Marvell switch. However, like some other M270 owners, I don’t wish to invest in pf+ and would like to figure out just how much work is involved in modifying the drivers to work with the M270 unit under FreeBSD, or OPNsense.

Yesterday I stumbled upon Intel’s DPDK project. If you look at their mail archive you’ll find a number of mods to the X550 driver to address this issue. I’m currently looking at this for clues.

Peter


On 15 Dec 2023, at 18:56, Eric Joyner <erj@freebsd.org> wrote:

On Fri, Dec 15, 2023 at 12:51 AM Peter A Barlow <peterbarlow2000@gmail.com> wrote:
I’m looking at running FreeBSD on an old Firebox M270.
It has a C3558 CPU with integrated X553 LAN controller which connects over MDIO to a Marvell 88E6190 switch.
Out of the box the X553 backplane is detected but there doesn’t seem to be any attempt at probing the MDIO for connected devices.

I’ve played around with the Intel ixgbe drivers, compiling the kernel etc to see if I can figure it out but I’m really struggling to understand what needs to be done.

At this stage I’m reaching out to the community to see if anyone can clarify something for me….are there some fundamental changes or additions required to the drivers to make this work, or is it something that should work already but needs some options enabling or configurations tweaking. I’m reluctant to put more time into trawling through the code if it’s a ‘simple’ configuration issue.

Any pointers would be very welcome.
Thank you.

I found this thread: https://forum.netgate.com/topic/154974/pfsense-on-watchguard-m270/112

I think the TL;DR is that you need pfSense Plus since the required software to get it to work isn't publicly available.

- Eric 

--------------AO23IMBYeIKUL9oSX56jt4qw-- From nobody Mon Sep 9 17:38:34 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X2YwJ4HcVz5TW77 for ; Mon, 09 Sep 2024 17:38:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X2YwJ2yBKz47GS for ; Mon, 9 Sep 2024 17:38:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1725903520; a=rsa-sha256; cv=none; b=n46e2/PqersaoatRGpv4+XfBUWevHUkBx0uP3uUnwrUauKNZwsV8eVmrvjs0NmDwnhAhue YuvE97iwwrRtbhKN31x745MUZ2XShYr5DiGu020aR3bGWUjFx8mqWQmiJqmZdCu2cY/zOX 8HJzgiHZt1ypjDqtjSsbS7SshU5Iaf8M25ju//JyC8WV0F5Ut1CMzAhHw/1PF0CoyLR9nB u3lujfvdK0S4zQ4GzCqjr9lpDz5eDC233Odo+VfxYZPkTBacqkskyrgdtSwu6IjVzzoqfp g8NqY2CBZC9hXXtORytEigSIr5s0jCe6p+2rz0Xkr5jcz5/RhFk5OThozjKdoQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1725903520; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GAKFZZglMN8uKiMD+rVIp1zsJ5YvGF8VaNnlSPA1JzI=; b=J0T4oq31tol8q4VxnFZPNjZpF3gxHsZG45H+gvcUjeCnxO50FWDeka3YIjh9229K42Swdm JEqDLu9g/mhuhMqhIWaMjKIDXYbVF02yjkwhOkWZmZOXxy4gKV4/Ds7nu2c0qVMRO16tQT ySTmnmXj6RC8i2z7xU6B4qpKPlf3XXRZGS1hSpY+DYDX9inhkODB8xHmnsP2ovTeINxwkO KUM3H7Zcj/OiOSjrLs//alMlTqGu7jmMfbNU9NXKWhFpuHqkLrxyTmX9tQvDijTgUN1FwT VJVBbCvjHp2yeapc5bKCxvi84kwSFeJSaQ6kETpDC+ksDut5dd/VI9+vI5lc7A== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4X2YwJ2ZSbz1BvF for ; Mon, 9 Sep 2024 17:38:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 489HcegR032334 for ; Mon, 9 Sep 2024 17:38:40 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 489Hcet4032332 for net@FreeBSD.org; Mon, 9 Sep 2024 17:38:40 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 280701] FreeBSD-SA-24:05 fix breaks ICMP/ICMP6 states handling in pf firewall (ping, traceroute) Date: Mon, 09 Sep 2024 17:38:34 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.1-RELEASE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: glebius@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: resolution bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D280701 Gleb Smirnoff changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED Status|Open |Closed --- Comment #81 from Gleb Smirnoff --- The first and original regression reported with this bug has been resolved = and fixed. The original bug report > replies to ping initiated from machines on networks behind pf firewall/NAT > to anything outside the local networks get blocked (state violation), > resulting in 'Request timed out'. was fixed by 2da98eef1f35, 46c4fc50d301. However, there are reports of more regressions remaining in the OPNsense fo= rk, that were not confirmed on FreeBSD, yet. With the official policy of one b= ug per bug report the FreeBSD core team is forcing this bug into closed state. For any other bugs related or not to SA-24:05 we request submitters to crea= te separate bug reports. We also recommend to provide at least reproduce reci= pes or at most atf(7) automated test cases. That would speed up bug resolving immensely. Finally, I'd like to remind that the project code of conduct applies not on= ly to the developers with a commit access, but to all participants in any discussion in the project space: https://www.freebsd.org/internal/code-of-conduct/ --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Mon Sep 9 17:53:25 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X2ZFN0Prtz5TYK3 for ; Mon, 09 Sep 2024 17:53:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X2ZFM6WsRz4Ch9 for ; Mon, 9 Sep 2024 17:53:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1725904407; a=rsa-sha256; cv=none; b=QD7kb1SvopqzDChHABS85NwxW0bH9WDa3Wn38KRq3KR765J+NO2HgoHBDJHxedS5HXa27t AA0T9iVa1e33YEle8P3Za/xSWZzS4LkfYRfqMqN3R0EMy+AUmG41SHusuiegNXCao7hUEo 3heve8LtgI4DuEtQj2n7IvpEwIHSvK9KmKZbPW3xtTSv09YI3pJVOhhoG62eZyFxngweRv ZTF9G2KkhZsJWNIdBwv+D7vMuiX/jwz2AI8VInQV2Nitmh9/LJhDDkqVgv6m1nxuZr/5Pc /C3LIpwQXCMcs4Nvy95/X19IKJPg6PnX9be6DfBqAUVgCF3qIelQryIng5pb9A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1725904407; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=swi6H1sYHeDgDtEAOCovd3yUuYpzshPgaa3taD45RjU=; b=osenU5VMHy9G1r2nqe1ovvyz22GT9BcWzL0K8zpVODnh1W8jH8w2c4Wac5diByk1v7D59f 0hsVMIKEqTQ9Va4qn1HZvkyIXmtQBrX8Fu9qoo2sGsyNE7iVRmQ9yA0FHN9IQJXYCZA1D4 6IRTf3+ZpIJdGVTGpWEiGnzN4j+5Gi7QhXn79Zh7P3A5u5nTwYH0dUx5BRrSnDhin15n2L O2hE2OmgqiD4ns0T1F7i1j80vOvJC5Z8l9yXy66L6S2GCCQ2EQ81rF8MtyD3hXWcfN6uLY EvhlNYSFJfcuWIc1aWfaMcU/Xxp8+fsz0qeLB0mtqBYjaoL8kIV2aKqL3RC+pQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4X2ZFM67Hnz1BhG for ; Mon, 9 Sep 2024 17:53:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 489HrR7W002899 for ; Mon, 9 Sep 2024 17:53:27 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 489HrRXj002898 for net@FreeBSD.org; Mon, 9 Sep 2024 17:53:27 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 280701] FreeBSD-SA-24:05 fix breaks ICMP/ICMP6 states handling in pf firewall (ping, traceroute) Date: Mon, 09 Sep 2024 17:53:25 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.1-RELEASE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: franco@opnsense.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D280701 --- Comment #82 from Franco Fichtner --- Thanks for the response! Based on this particular resolution in Comment 81, OPNsense will back out t= he SA in full effective with the next stable release 24.7.4 and we wish all involved parties the best on their way forward. Respectfully, Franco --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Mon Sep 9 18:04:54 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X2ZVb1k75z5TZL0 for ; Mon, 09 Sep 2024 18:04:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X2ZVb0jBKz4Fmn for ; Mon, 9 Sep 2024 18:04:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1725905095; a=rsa-sha256; cv=none; b=Fqt3EcBSJoNoq0mWoocSJ6J4tRH3GyW3ECPDjh2yR+oFcEYHwoWVsvcrhr+gObB7Ul/0+W ppb+lYcuqV26RKooIFqMzOcz4TIqXUA7ItNSdKov3GjA0gon0rZuBeATUcUYyjKSlOIytk Y/ACOZL4tX21bJUM2Kynw9DuQKlyhW7nACWXPRmdV27gQC6G332E/F1Y4MUzZXt4oYhf1X Kop8L8cOu99ZJwuN6apaKe1JJWz210swwBu38+l4bmAXyGRB7XthkbTebaS1sYhEoBxeYX 1MHj8vxyaOehkhpfcyxleWYFhbQo4uR4aOg8ccrKN2dioQJMGax8Ad0+Vcp/9w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1725905095; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QsiGiaXIftbojI3qBAXT6qCjId7Bea+0fNcFiajYmWU=; b=BWoIUxc7FKx4KIzbpaN2e5MvHcRvysIYTNXliTsDr3rE1UWyxuJM8hu3Yp1GKs4642npgR Qt4lFYheSxT4i4sxq/mZHUpr23co+itO2p1XfajnWdEXAVBz0Ju/uLAHneG0Fxdshzyqr/ wI5fSrqcTTCZvwvM8AcqYgwBwECzy0taW/nXB+lvjNZd3WXBIB0nc2QtKWdXcNUcGSNScS a2gArpOvXiF4zvDHw1UEt1KpUln3c8QHBfmLd0WEU5L2zaNL8f/ua2IR3LpgpjaeIz+lKi MKSyegb7F/41laae8OOS2MbKCWEEczNSzfXr4s7M/dUwYklDvKYJhDAb1VTyUA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4X2ZVb0Dbwz1Cn1 for ; Mon, 9 Sep 2024 18:04:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 489I4sjU050594 for ; Mon, 9 Sep 2024 18:04:54 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 489I4sP4050593 for net@FreeBSD.org; Mon, 9 Sep 2024 18:04:54 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 280701] FreeBSD-SA-24:05 fix breaks ICMP/ICMP6 states handling in pf firewall (ping, traceroute) Date: Mon, 09 Sep 2024 18:04:54 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.1-RELEASE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: freebsd_email@congenio.de X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D280701 --- Comment #83 from Dr. Uwe Meyer-Gruhl --- Just for reference: I just created bugs https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D281= 395 and https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D281397. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Mon Sep 9 18:41:06 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X2bJN2bMDz5Tg0X for ; Mon, 09 Sep 2024 18:41:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X2bJN1Xfvz4LHh for ; Mon, 9 Sep 2024 18:41:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1725907268; a=rsa-sha256; cv=none; b=P+cHeDuFoG/bWA7lM7I0+PiJFc5P5Tap4+T0tx8jjHz0kLkPHEARN9bjWF49awN1jllw67 GAHf0LSBkNO7kpxofbeVBk4XQRFVO1XZu1trjxOcFijKimGqp/8+28H136wUTqaE6+RhWK 4mLMfGIZVAdY9AG43LDRAaS6pmTypgghmHBmiJs1TrCK3nLCdagL1Y1wN0xIOobUf9B3Xf M+99/4u5LCFBIHCZ6qXM+cau2m3ASwlHeecv5byZvvc0EPthNBi52bS872EyeU9hzZXaBk 73oVoaDXRPQzfuuHPujubkf+yBt/iXf43vfAEYhbeEt3blvc2ZwuUD0VMNW5iQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1725907268; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zlTdvL1fF6oCXOjAX74LsnqNI+lIqhve2C1QH+hEJA8=; b=GiY54IhVuJFw3cPNd0qbwLlgzwo0kUJPvYMmTQ7w5aGcuX3qao6FTpedgpRODtkUzELQUG 2RekWIWjgr10r7TxDKVGBlSIzMdiCwo4XDNpmr2z7pY8L7niRvl9gE2Ub7komQvcnRME0m 4MFpk2Q+AVfBw3sswmRRlCTNztZXrQs4vaR8qyYmbbSaeq38B8ddxXzDeeI6PiNhBxCJsb FSMoOsQDfgBcXmKCwda+2fZVx9VjYOdsCGdfDXp5nJqFxZERdYHiS2UBcGSTXAWRz8IT08 fGEJI9S9z23HG0VQu5FhKQIrxD8KfoF3vB9MOoWBHil65cn7gdk7oLvZV+cfEQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4X2bJN17lLzFN9 for ; Mon, 9 Sep 2024 18:41:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 489If8tb010412 for ; Mon, 9 Sep 2024 18:41:08 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 489If8YT010406 for net@FreeBSD.org; Mon, 9 Sep 2024 18:41:08 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 280701] FreeBSD-SA-24:05 fix breaks ICMP/ICMP6 states handling in pf firewall (ping, traceroute) Date: Mon, 09 Sep 2024 18:41:06 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.1-RELEASE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: doktornotor@mailinator.com X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D280701 --- Comment #84 from doktornotor --- (In reply to Gleb Smirnoff from comment #81) > Finally, I'd like to remind that the project code of conduct applies not = only > to the developers with a commit access, but to all participants in any > discussion in the project space: > https://www.freebsd.org/internal/code-of-conduct/ That you for the reference and reminder. To avoid future clashes with your = CoC, I will do my best to refrain from reporting bugs in any code committed / sponsored by Rubicon Communications, LLC ("Netgate").=20 Hopefully, that will make FreeBSD even better. Have a nice day. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Sep 10 01:05:44 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X2lrJ6VMJz5W5yZ; Tue, 10 Sep 2024 01:05:52 +0000 (UTC) (envelope-from vadimnuclight@gmail.com) Received: from mail-lf1-x134.google.com (mail-lf1-x134.google.com [IPv6:2a00:1450:4864:20::134]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X2lrH62PYz4Xrg; Tue, 10 Sep 2024 01:05:51 +0000 (UTC) (envelope-from vadimnuclight@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=daMs+lUl; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of vadimnuclight@gmail.com designates 2a00:1450:4864:20::134 as permitted sender) smtp.mailfrom=vadimnuclight@gmail.com Received: by mail-lf1-x134.google.com with SMTP id 2adb3069b0e04-5365cc68efaso2934780e87.1; Mon, 09 Sep 2024 18:05:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725930349; x=1726535149; darn=freebsd.org; h=content-transfer-encoding:mime-version:message-id:subject:cc:to :from:date:from:to:cc:subject:date:message-id:reply-to; bh=9Nr+PMWJ0cvFk9krbCuHyFG2/xuUijibH8ER+7a1Gek=; b=daMs+lUl2JvLETVKX9X9xRrOgIwHzoaiXx/s6ROKvZ4Blwka+wA45pwh2xg6FWnI4z fjdaxHGelNWDkmTtjBH/ZY3TCIUvqv+i9GHBLTisMJtKarHZecKeJH2b3NWxTFlN+HD1 kJYYFXJ/Coj4uA9XeNS5H/k7fL0/sWuEROIu1lqXkhYGtp6H7i7v+Y44YZNvbw0NU5Of 9QMgwZIFydv1NONCxoqqlb2/hT7xrGUNtMZEcyXJGhDok8Sbfo6toTxUnUNVdq2t0SMR 0o/x2X3alIyD5vpaOKtRsJ4Ujn5XpbY0xGRmdqxBbDAaCbehy/9f8sVk0/XpIscpglQD Ny9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725930349; x=1726535149; h=content-transfer-encoding:mime-version:message-id:subject:cc:to :from:date:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=9Nr+PMWJ0cvFk9krbCuHyFG2/xuUijibH8ER+7a1Gek=; b=UhPyebZPMzaUOMKmF/DJnkVmw2nrMiePgVfveP/l6oXcZOsFccjoId7anVs0RLAzcm N9xM0Bw6dyg50MGv590355VXT5+TyzfWmzKT3T9CoQ34w4iTmw7IBFai5d2S7POALcH0 EYCKZRDtjotKXUcayoKxn7Mkem/M5cyzYHfxVJBuYxFk+67GU7foRvXl4bHb/Yw/KhiS DqUq5a7G2KCiICiXtZadwpyqanDSSg1A9h+ilAuFI/g7WEOSV1nD7YJA55HBu55EMLdH XBQ3BHEucb+fiwL1g3hCLrXjMYkcggXA8YduzxTd1S5dnEsHDisOOODqgJx3thyFX/an PMBw== X-Forwarded-Encrypted: i=1; AJvYcCV84kkRkeWISeXlDgn2jfzvfw6a5m75CCF5GX+RbRF2F9Nf3p7sU7LeuEztrN1L/J8Z/kobLM4px7/4quM=@freebsd.org, AJvYcCXylIcOAr5hAKXI3Xn9MeVZfBXI76kRHMVQl9s26VpAXN0ZMtOA4pBpHTMsmGm0x97ag5J0S6xRYsBiVH3UqWw=@freebsd.org X-Gm-Message-State: AOJu0Yz0PpwA33HLZkmemvzZ1NHCfLPV5lwCmDPHFifgDfyIrtFGDVYv Jb+fOtzCtJgrcsa7ZUyW1OwhD4m0A/ODcSL4pHTWQ57O+jCBXVFLCwVuFG2O X-Google-Smtp-Source: AGHT+IFgmynDElvygkAMoUaHF9Nx9iG/peV6DgXkFUjzvvnKTA+tvW8l0ViqB0ZX33ODbvZM9xiTKw== X-Received: by 2002:a05:6512:1110:b0:52d:b150:b9b3 with SMTP id 2adb3069b0e04-536587b545dmr8620481e87.32.1725930348584; Mon, 09 Sep 2024 18:05:48 -0700 (PDT) Received: from nuclight.lan ([37.204.254.214]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5365f8cb6fbsm922220e87.134.2024.09.09.18.05.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Sep 2024 18:05:48 -0700 (PDT) Date: Tue, 10 Sep 2024 04:05:44 +0300 From: Vadim Goncharov To: freebsd-arch@FreeBSD.org, freebsd-hackers@FreeBSD.org, freebsd-net@FreeBSD.org, tcpdump-workers@lists.tcpdump.org, tech-net@NetBSD.org Cc: Alexander Nasonov Subject: BPF64: proposal of platform-independent hardware-friendly backwards-compatible eBPF alternative Message-ID: <20240910040544.125245ad@nuclight.lan> X-Mailer: Claws Mail 3.19.1 (GTK+ 2.24.33; amd64-portbld-freebsd12.4) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.17 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; NEURAL_HAM_SHORT(-0.17)[-0.169]; MIME_GOOD(-0.10)[text/plain]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; TO_DN_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FREEMAIL_FROM(0.00)[gmail.com]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::134:from]; RCPT_COUNT_FIVE(0.00)[6]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-arch@freebsd.org,freebsd-hackers@freebsd.org,freebsd-net@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+] X-Rspamd-Queue-Id: 4X2lrH62PYz4Xrg Hello! We don't need ELF relocations! We want better loop control! No so little parameters, Verifier! Leave our code alone! -- Ping Floyd I've recently had some experience with Linux's ePBF in it's XDP, and this l= eft quite negative impression. I was following via https://github.com/xdp-proje= ct/xdp-tutorial and after 3rd lesson was trying to create a simple program for searching TCP timestamp option and incrementing it by one. As you know, eBPF tool stack consists of at least clang and eBPF verifier in the kernel, and after two d= ozen tries eBPF verifier still didn't accept my code. I was digging into verifier sources, and the abysses opened in front of me! Carefully and boringly going via disassembler and verifier output, I've found that clang optimizer ignor= es just checked register - patching one byte in assembler sources (and target = .o) did help. I've filed https://github.com/iovisor/bcc/issues/5062 with details if one curious. So, looking at eBPF ecosystem, I must say it's a Frankenstein. Sewn from go= od, sometimes brilliant parts, it's a monster in outcome. Verifier is in it's o= wn right, compiler/optimizer is in it's own right... But at the end you even don't have a high-level programming language! You must write in C, relative= ly low-level C, and restricted subset of C. This requires very skilled professionals - it's far from something not even user-friendly, but at least sysadmin-friendly, like `ipfw` or `iptables` firewall rules. Thus I looked at the foundation of eBPF architecture, with which presupposi= tions in mind it was created with. In fact, it tries to be just usual programming after checks - that is, with all that pointers. It's too x86-centric and Linux-centric - number of registers was added just ten. So if you look at t= he GitHub ticket above, when I tried to add debug to program - you know, just specific `printf()`s - it failed verifier checks again because compiler now had to move some variables between registers and memory, as there is limit = on just 5 arguments to call due to limit of 5 registers! And verifier, despite being more than 20,000 lines of code, still was not smart enough to track i= nfo between registers and stack. So, if we'd started from beginning, what should we do? Remember classic BPF: it has very simple validator due to it's Virtual Machine design - only forw= ard jumps, checks for packet boundaries at runtime, etc. You'd say eBPF tries f= or performance if verifier's checks were passed? But in practice you have to t= oss in as much packet boundary checks as near to actual access as possible, or verifier may "forget" it, because of compiler optimizer. So this is not of much difference for checking if access is after packet in classic BPF - the same CMP/JUMP in JIT if buffer is linear, and if your OS has put packet in several buffers, like *BSD or DPDK `mbuf`'s, the runtime check overhead is negligible in comparison. Ensuring kernel stability? Just don't allow arbitrary pointers, like origin= al BPF. Guaranteed termination time? It's possible if you place some restrictions. = For example, don't allow backward jumps but allow function calls - in case of stack overflow, terminate program. Really need backward jumps? Let's analyze for what purpose. You'll find these are needed for loops on packet contents. Solve it but supporting loops in "hardware"-controlled loops, which can't be infinite. Finally, platforms. It's beginning of sunset of x86 era now - RISC is comin= g. ARM is now not only on mobiles, but on desktops and servers. Moreover, it's era of specialized hardware accelerators - e.g. GPU, neural processors. Even general purpose ARM64 has 31 register, and specialized hardware can implement much more. Then, don't tie to Linux kernel - BPF helpers are very rigid interface, from ancient era, like syscalls. So, let's continue *Berkeley* Packet Filter with Berkeley RISC design - hav= ing register window idea, updated by SPARC and then by Itanium (to not waste registers). Take NetBSD's coprocessor functions which set is passed with a context, instead of hardcoded enums of functions - for example, BPF maps = is not something universal, both NetBSD and FreeBSD have their own tables in firewall. Add more features actually needed for *network* processor - e.g. 128-bit registers for IPv6 (eBPF axed out even BPF_MSH!). And do all of this in ful= ly backwards-compatible way - new language should allow to run older programs from e.g. `tcpdump` to run without any modifications, binary-compatible (again, eBPF does not do this - it's incompatible with classiv BPF and uses a translator from it). Next, eBPF took "we are masquerading usual x86 programming" way not only ju= st in assembly language. They have very complex ELF infrastructure around it w= hich may be not suitable for every network card - having pc-addressed literals, = as in RISC processors allows for much simpler format: just BLOB of instruction= s. BPF64 adds BPF_LITERAL "instruction" of varying length (it's interpreted by just skipping over contents as if it was jump), which, if have special signatures and format, allow for this BLOB of instructions to contain some metadata about itself for loading, much simpler than ELF (esp. with DWARF). Then, ecosystem. eBPF defines functions callable from user code like: > enum bpf_func_id___x { BPF_FUNC_snprintf___x =3D 42 /* avoid zero */ }; That is, ancient syscall-like way of global constant, instead of context. A "context" here is the structure passed with code to execution which contains function pointers of what is available to this user code, in spirit of NetBSD's `bpf_ctx_t` for their BPF_COP/BPF_COPX extensions. This is not only provides better way than "set in stone" syscall-like number, but BPF64 goes further and defines an "packages" in running kernel with namespaces to allow e.g. Foo::Bar::baz() function to call Foo::quux() from another BPF program, populating ("linking") it's context with needed function without relocation= s. These "packages" expected to be available to admin in e.g. sysctl tree, with descriptions, versioning and other attributes. Some other quotes about how restricted eBPF is: > First, a BPF program using bpf_trace_printk() has to have a GPL-compatibl= e license. > Another hard limitation is that bpf_trace_printk() can accept only up to = 3 input arguments (in addition to fmt and fmt_size). This is quite often pr= etty limiting and you might need to use multiple bpf_trace_printk() invocat= ions to log all the data. This limitation stems from the BPF helpers abilit= y to accept only up to 5 input arguments in total. > Previously, bpf_trace_printk() allowed the use of only one string (%s) ar= gument, which was quite limiting. Linux 5.13 release lifts this restriction= and allows multiple string arguments, as long as total formatted output do= esn't exceed 512 bytes. Another annoying restriction was the lack of suppor= t for width specifiers, like %10d or %-20s. This restriction is gone now as= well > Helper function bpf_snprintf > Outputs a string into the str buffer of size str_size based on a format s= tring stored in a read-only map pointed by fmt. > > Each format specifier in fmt corresponds to one u64 element in the data a= rray. For strings and pointers where pointees are accessed, only the pointe= r values are stored in the data array. The data_len is the size of data in = bytes - must be a multiple of 8. > > Formats %s and %p{i,I}{4,6} require to read kernel memory. Reading kernel= memory may fail due to either invalid address or valid address but requiri= ng a major memory fault. If reading kernel memory fails, the string for %s = will be an empty string, and the ip address for %p{i,I}{4,6} will be 0. Not= returning error to bpf program is consistent with what bpf_trace_printk() = does for now. > > Returns > > The strictly positive length of the formatted string, including the trail= ing zero character. If the return value is greater than str_size, str conta= ins a truncated string, guaranteed to be zero-terminated except when str_si= ze is 0. > > Or -EBUSY if the per-CPU memory copy buffer is busy. > > static long (* const bpf_snprintf)(char *str, __u32 str_size, const char = *fmt, __u64 *data, __u32 data_len) =3D (void *) 165; So. In summary: BPF64 has not only traditional ("firewall on network packet= s") area of use but also suitable - and having goals to design in mind - for: * non-network. e.g. syscall arguments checking * network protocols passing untrusted code which can be run by other side in restricted sandbox (e.g. muSCTP custom (de)compression rules) * an alternative to https://capnproto.org/rpc.html for multi-method calls on high-latency links (e.g. MQTT5-like and/or for environments when Cap=E2=80=99n Proto itself is not applicable, e.g. no direct connect= ions) I've put a sketch of design to https://github.com/nuclight/bpf64 with files: * The `nuc_ts_prog_kern.c` (and it's include `nuc_ts_common_kern_user.h`) is XDP/eBPF program (for Linux 6.5) for parsing TCP packet and incrementing it's Timestamp option, if any, recording statisitics intop eBPF map. * The `nuc_ts_incr.baw` (and it's include `nuc_ts_incr.bah`) is the equival= ent program doing the same thing, but in a new BPF64 Assembler Wrapper langua= ge, not yet written and subject to change. Note this is a lower-level language than C, viewed as intermediate solution until BPF64 becomes stable, after which more higher-level language (higher than C) should be written, at le= ast as expressible as `tcpdump` (`libpcap`) one. * `bpf64spec.md` for draft of Specification, but currently it's more in a "a draft of draft" state :-) I am requesting comments about this architecture before implementing, especially from people knowledgeable in JIT compilers, because, though while I see BPF64 much more simpler than eBPF (probably just several man-months to implement), my sabbatical has ended - and design mistakes are much harder to fix when implementation already exists. --=20 WBR, @nuclight From nobody Tue Sep 10 11:45:57 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X323032y3z5WK1c; Tue, 10 Sep 2024 11:46:04 +0000 (UTC) (envelope-from vadimnuclight@gmail.com) Received: from mail-lj1-x231.google.com (mail-lj1-x231.google.com [IPv6:2a00:1450:4864:20::231]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X32300fVhz4t3M; Tue, 10 Sep 2024 11:46:04 +0000 (UTC) (envelope-from vadimnuclight@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lj1-x231.google.com with SMTP id 38308e7fff4ca-2f75c0b78fbso6881101fa.1; Tue, 10 Sep 2024 04:46:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725968760; x=1726573560; darn=freebsd.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=WBkYJiNc2Nbz4EKv3F00lPQUtex0LWZeDlyUvZRnzLo=; b=eLmSAbnTTmxkV1BYRPvdL8ihc6tJk78BhA7fTqW43ezwpZsIQF8yfzUR5q57j1m8Hq PmxxvQ9gnR6I520lqhsrN/HSU7toU+Zrkr09eVhW3sckiEanD3a1G8lVWkEUG5LaPLwZ wK1imPmbTOizLqPsE9QlI8S0hIRosQxxfP9YksLVp0VdJxH209h/fTY9ExMaPIfXCBov eS+Gw18luw4/s7PvnFdRiHZgXRz5nBwYLsHkn+cF64Lil9lNdQl6SElKTmEejMRY5Up5 cYANsZPrZ33iOgqJJW5I0E2HFST3rkuuuGbnyovpEq9+T3A4WYJ4YMeBiFqrci9eHXRU CG9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725968760; x=1726573560; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WBkYJiNc2Nbz4EKv3F00lPQUtex0LWZeDlyUvZRnzLo=; b=ITJmRrJAdnu++v3tPrkUoF9eSjrR8477wAgf7CZy5FV/U8xju4VpffAXrUBWSGX+yT VzZfc183gY8ZAD95hgLRPkAOQBNymtS65AWbscY4aMakUqu/PS6zZG6fzZ6vuPxDQkeS /n6r2yOM2YXLnXQWAOOTJFcNYBmNMvwkhiJ9+bgg5p0h5apd8VD8greF+3omjn4AHLUP lT4jjbSxeNNiLu1Yiw+NkDiLETRMkCVF29yIyyLA7Z3JdXH/EGLBMtGH24KLqp0RMTfY 21I+iqlRe96t9GeP5HBzb/tILoBZiulOkcHijdSRSlsWNYJ6wP0SEDnclIJZLrceXYpZ Mt2w== X-Forwarded-Encrypted: i=1; AJvYcCXABsXwFqB9228+lHx1x/NRaTNdGIhiAtvPEsG+ym0Qe7PTj6I1BmoO7P5N26nDz3+bC9sqLFi+imd0r48=@freebsd.org, AJvYcCXN/5rVKkRrdRC8xfQ4EvjUmuH9gjlcBFLDaK+BC2oxZ1Xyq8PU1B5rJ+LiV2I/2nMA7y5jLuSxU3oIYoTgpkI=@freebsd.org X-Gm-Message-State: AOJu0YyG7smioDa/4fInc26m+OFrnDfxv8+P0t+VgXwsk5SopzV5x55O zy2qQI/hEhuqgYkObWZ5LKpAV2NILwU3IPybXGSvkDtSoJ69xTFm X-Google-Smtp-Source: AGHT+IGT6G/PcabgIwPjGt4XifTh3HHZgQq37Ig9miee4qJLIrWgIe6f17r20iP9+oq+FYhtHbvv5A== X-Received: by 2002:a05:651c:1a0c:b0:2f7:6653:8046 with SMTP id 38308e7fff4ca-2f766538106mr45293161fa.25.1725968759775; Tue, 10 Sep 2024 04:45:59 -0700 (PDT) Received: from nuclight.lan ([37.204.254.214]) by smtp.gmail.com with ESMTPSA id 38308e7fff4ca-2f75c07c539sm11600861fa.88.2024.09.10.04.45.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Sep 2024 04:45:59 -0700 (PDT) Date: Tue, 10 Sep 2024 14:45:57 +0300 From: Vadim Goncharov To: "Poul-Henning Kamp" , tcpdump-workers@lists.tcpdump.org Cc: freebsd-arch@FreeBSD.org, freebsd-hackers@FreeBSD.org, freebsd-net@FreeBSD.org, tech-net@NetBSD.org, Alexander Nasonov Subject: Re: BPF64: proposal of platform-independent hardware-friendly backwards-compatible eBPF alternative Message-ID: <20240910144557.4d95052a@nuclight.lan> In-Reply-To: <202409100638.48A6cor2090591@critter.freebsd.dk> References: <20240910040544.125245ad@nuclight.lan> <202409100638.48A6cor2090591@critter.freebsd.dk> X-Mailer: Claws Mail 3.19.1 (GTK+ 2.24.33; amd64-portbld-freebsd12.4) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4X32300fVhz4t3M On Tue, 10 Sep 2024 06:38:50 +0000 "Poul-Henning Kamp" wrote: > -------- > Vadim Goncharov writes: > > > I've put a sketch of design to https://github.com/nuclight/bpf64 > > with files: > > Counter proposal: > > 1. Define the Lua execution environment in the kernel. > > 2. Add syscall to submit a precompiled Lua program (as bytecode) Anyone who thinks "any generic bytecode" misses the main point, see below. > 3. Add syscall to execute submitted Lua program > > And yes: I'm being 100% serious. Well, preparing spec/letter in a rush I probably forgot the main reason for BPF (and successors) to exist thinking it's obviuos: safety. Let's restate: *BPF* allows UNTRUSTED user code to be executed SAFELY in kernel. It's easy for your Lua code (or whatever) code to hang kernel by infinite loop. Or crash it by access on arbitrary pointer. That's why original BPF has no backward jumps and memory access, and eBPF's nightmare verifier walks all code paths and check pointers. And that's why DTrace also has it's own VM and bytecode in kernel (see https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-924.pdf Chapter 7) Your "counter proposal" was essentially available for all these decades in form "oh, just write KLD in C instead of that limited tcpdump". > If we are going to reinvent "Channel Programs" 67 years after IBM > came up with them for their 709 vacuum tube computer, at the very > least we should use a sensible language syntax. Don't know what that is, quick googling shows something modern on AMQP. But Lua at least doesn't have *sensible* syntax, Perl or Tcl much better. And I'm surprised why Fort, being available in loader, wasn't ported for all these years. -- WBR, @nuclight From nobody Tue Sep 10 12:59:02 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X33gR2X2Yz5WT26; Tue, 10 Sep 2024 12:59:15 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X33gR1bPhz487P; Tue, 10 Sep 2024 12:59:15 +0000 (UTC) (envelope-from theraven@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1725973155; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=52WhLO+OOHHelT2Ulj2UpeO8k0p2xywwOt9PKUFdeUE=; b=Z9leBlhmFM1wChnwg8J+UYDNYI79vTXSLsfTgdJJEXM21aXDvdIedF4jMih1Sc8fZNkLAO jhV7+IsLEg0wk00SVmpRYo4hj5EjlJt8MXXRzg7G4fzPq73nCIeqs5YYYKkh+igF1vZi/5 L804Mq/G8taMah23Et0VIZ8OH9ZiIJT4EkD2enMxWmakB8Yl60PcQHeG5O0KoM6lHVnS3L U36iIZ578yTmEDfaK2R3+FvYTRsHN7NyvqH1Uo7pJ5NRKtlu+2jBWsSiHE4UB/Xrv5JGuH c/zGstZ5ZbIly10xanfW2yAN0X3jA47l/ilTzoA0xMqMxh1uhA9p3VyLkt9BQA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1725973155; a=rsa-sha256; cv=none; b=EPHvqxYtQCoIHtuF6ELe6d98c6hcy6ZTAffa6LfPF4ZEMaKjVZMM7FS/yaDtPF0U/9Wj/T zEnYrlAs0rsdrRQE4oxgxcknTcqFUe5V893KkgDF2Iid22HDVvC6nn/41n9u6qcr/gix1V 7C5uEzfDERYUr8ymKHvFFsvCqBKdsd6nRJftshmxD5eFcn25dBEW3EJPey7ZxnU9KZm5us 1j1z3rGA2LsBNm8OZtraxZ+/nOd0LeGbp30vj/UM2i/W8mOd/6lEBh8Y/V7UvNh+MxdmLT PsMGCSg8rcMkl5a9uxP+xXZCEomHbYR871vc/onLG8rGWUOz10WyaE5/U8+/mw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1725973155; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=52WhLO+OOHHelT2Ulj2UpeO8k0p2xywwOt9PKUFdeUE=; b=XPXxoJsZW7hScfl6faNRdizk8iNIMDcmwRx20lDNX2XGUlHp7BNFeCsT5OyHfcG+EGsbs3 9LUlhAqs+VDtS85Akd351XwffW8FkkfXjezE+XD9L+PhNGKOjjwldUvvlUMpK4Omz3kRWP PRl9pCTfsKazovqSwBGfUHzjTJaKQonVzi9G9bG5/hK8J1ezTaCzYdXHcWdLEXI/KvpwNy SAe0fogKCY8Q6ZUzFQ0Thkx8I4LJ7q81fFqJcO9mNW90aerrwz57g2K7MtrGRSQQ5be2Uk 9/ntKjaddeVAB1YIJNtDQRrAdtUHdQyBqDQCnCMNQEw9FeNO63xxLUOCJfOC4A== Received: from smtp.theravensnest.org (smtp.theravensnest.org [45.77.103.195]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: theraven) by smtp.freebsd.org (Postfix) with ESMTPSA id 4X33gR10JPzRMq; Tue, 10 Sep 2024 12:59:15 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from smtpclient.apple (host109-155-136-107.range109-155.btcentralplus.com [109.155.136.107]) by smtp.theravensnest.org (Postfix) with ESMTPSA id A6A0065B5; Tue, 10 Sep 2024 13:59:13 +0100 (BST) From: David Chisnall Message-Id: <4D84AF55-51C7-4C2B-94F7-D486A29E8821@FreeBSD.org> Content-Type: multipart/alternative; boundary="Apple-Mail=_568E13D8-1F5C-410F-B911-1402B36B059B" List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3776.700.51\)) Subject: Re: BPF64: proposal of platform-independent hardware-friendly backwards-compatible eBPF alternative Date: Tue, 10 Sep 2024 13:59:02 +0100 In-Reply-To: <20240910144557.4d95052a@nuclight.lan> Cc: Poul-Henning Kamp , tcpdump-workers@lists.tcpdump.org, "freebsd-arch@freebsd.org" , "freebsd-hackers@freebsd.org" , "freebsd-net@freebsd.org" , "tech-net@netbsd.org" , Alexander Nasonov To: Vadim Goncharov References: <20240910040544.125245ad@nuclight.lan> <202409100638.48A6cor2090591@critter.freebsd.dk> <20240910144557.4d95052a@nuclight.lan> X-Mailer: Apple Mail (2.3776.700.51) --Apple-Mail=_568E13D8-1F5C-410F-B911-1402B36B059B Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 On 10 Sep 2024, at 12:45, Vadim Goncharov = wrote: >=20 > It's easy for your Lua code (or whatever) code to hang kernel by > infinite loop. Or crash it by access on arbitrary pointer. That's why > original BPF has no backward jumps and memory access, and eBPF's > nightmare verifier walks all code paths and check pointers. I=E2=80=99m not convinced by the second: Lua has a GC=E2=80=99d heap, = you=E2=80=99d need to expose FFI things to it that did unsafe things, = and that=E2=80=99s equally a problem for eBPF. The first is not a problem. The Lua interpreter has a bytecode limit. = You can define a bounded number of bytecodes that it will execute. The = problem comes from the standard library. Things like string.gmatch can = have high-order polynomial complexity and so it=E2=80=99s possible for a = Lua program that executes a small number of bytecodes to create a string = that takes a vast amount of time to match on. Again, this is also a = problem for eBPF if you expose a similar function, the solution is to = not expose functions with large data-dependent runtimes to untrusted = script. More generally, there are a lot of problems with interpreting or JITing = untrusted code in the kernel in *any* runtime. Speculative execution = makes it easy to use these as primitives to leak kernel secrets, either = via timing of the programs themselves, using the JIT to generate = gadgets, or by leaking data via cache priming. Both eBPF and Lua have these problems. The thing I would like to see for our current use of semi-trusted Lua in = the kernel (ZFS channel programs) is a way of exposing them (under = /dev/something) as file descriptors and modifying the ioctls that run = them to take a file descriptor argument. I would like to separate the = two operations: - Load a channel program. - Run a channel program. In the post-Spectre world, the former remains a privileged operation. = Even though Linux pretends it isn=E2=80=99t, allowing arbitrary (even = arbitrary constrained) code to run in the kernel=E2=80=99s address space = is a problem. Invoking such code; however, should follow the same rules = as everything else. A trusted entity should be able to load a pile of = Lua / eBPF / BPF64 / whatever programs into the kernel and then set up = permissions so that sandboxed programs (and jails) can use a defined = subset of them. David --Apple-Mail=_568E13D8-1F5C-410F-B911-1402B36B059B Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 On 10 Sep = 2024, at 12:45, Vadim Goncharov <vadimnuclight@gmail.com> = wrote:

It's easy for your Lua code (or whatever) = code to hang kernel by
infinite loop. Or crash it by access on arbitrary pointer. = That's why
original BPF has no backward jumps and memory access, and = eBPF's
nightmare verifier walks all code paths and check = pointers.

I=E2=80=99m not = convinced by the second: Lua has a GC=E2=80=99d heap, you=E2=80=99d need = to expose FFI things to it that did unsafe things, and that=E2=80=99s = equally a problem for eBPF.

The first is not a = problem.  The Lua interpreter has a bytecode limit.  You can = define a bounded number of bytecodes that it will execute.  The = problem comes from the standard library.  Things like string.gmatch = can have high-order polynomial complexity and so it=E2=80=99s possible = for a Lua program that executes a small number of bytecodes to create a = string that takes a vast amount of time to match on.  Again, this = is also a problem for eBPF if you expose a similar function, the = solution is to not expose functions with large data-dependent runtimes = to untrusted script.

More generally, there are = a lot of problems with interpreting or JITing untrusted code in the = kernel in *any* runtime.  Speculative execution makes it easy to = use these as primitives to leak kernel secrets, either via timing of the = programs themselves, using the JIT to generate gadgets, or by leaking = data via cache priming.

Both eBPF and Lua have = these problems.

The thing I would like to see = for our current use of semi-trusted Lua in the kernel (ZFS channel = programs) is a way of exposing them (under /dev/something) as file = descriptors and modifying the ioctls that run them to take a file = descriptor argument.  I would like to separate the two = operations:

 - Load a channel = program.
 - Run a channel = program.

In the post-Spectre world, the former = remains a privileged operation.  Even though Linux pretends it = isn=E2=80=99t, allowing arbitrary (even arbitrary constrained) code to = run in the kernel=E2=80=99s address space is a problem.  Invoking = such code; however, should follow the same rules as everything else. =  A trusted entity should be able to load a pile of Lua / eBPF / = BPF64 / whatever programs into the kernel and then set up permissions so = that sandboxed programs (and jails) can use a defined subset of = them.

David

= --Apple-Mail=_568E13D8-1F5C-410F-B911-1402B36B059B-- From nobody Tue Sep 10 13:09:15 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X33v465Hpz5WTtq; Tue, 10 Sep 2024 13:09:20 +0000 (UTC) (envelope-from vadimnuclight@gmail.com) Received: from mail-lf1-x12d.google.com (mail-lf1-x12d.google.com [IPv6:2a00:1450:4864:20::12d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X33v44LG2z4Dl6; Tue, 10 Sep 2024 13:09:20 +0000 (UTC) (envelope-from vadimnuclight@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x12d.google.com with SMTP id 2adb3069b0e04-5365cc68efaso3536809e87.1; Tue, 10 Sep 2024 06:09:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725973759; x=1726578559; darn=freebsd.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=RkKU7OzzhXolcRcGkGudy213qu3KY/KqHoqUnGVQTPQ=; b=TjIKSiuTIbl6+zsrcY5WHuTne3sayhEsaq05ctDTCdwWOYxKF6lAy0HYjJjFX2tnpK NOluUSyHTxtCEGgxuaf9rIuaUSlkgaRmV7KHAVuYiziMa3eO8Gb4SDPAu2S55CTrF0kH uKVqfwSuHVNzK8Q99JfhPa7dao4cMpzcj2J6lCITJ3h4u7TDASUsyeDmZUDF7tghluZr B0IICSZd0FeYH5ECePlPRK4dMpuCA/AT7nHOzXsBUVryealF17IlnaJK+XmHxCice0aQ RUK+5BjVLR0WDYerZMLnj0d4vjb1ojcfEdL38KFldCgyVHzh7l0JmafqYGCkFvFFmrJL wx4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725973759; x=1726578559; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RkKU7OzzhXolcRcGkGudy213qu3KY/KqHoqUnGVQTPQ=; b=R8jy9qfeNQDxpDKAx52YW0XlvljEN4VblyOX3P32nv2sHS8j2gClezwdpryv3L6y5I KZuBH2vniCjn0R5c6ZO9fRwVJAJyhogJxwE0KnjK3u9UNmDeuU8y2pFoBjmLOuJ/e2G6 Lz82S4Mba9TgMRweggBOGN6lZu4hQrjxm3MdUTyQkJG/XqypOsdqKDddBAZyiZiwk3mj XN8XysnFkwPTd/DUAByG1KcNUvm+EYUYxEEHdhqhcNzvud8FPm8iNUm7zzT5mbnjsYtC HKhT4fdtuZwjPN0ML98Id9m+9l1IhL0KytgeJcH2QA9ZSJOl5QfaePFnTjwsqS86joGy b/7g== X-Forwarded-Encrypted: i=1; AJvYcCVezgU/L3uEP2R0J3jz+xMcVCeVg0GWQv1GO6nTm2h5lc7SUMB0cP7DGQK8Oh6A9Z/1glVwcGPCf4reNUmcszQ=@freebsd.org, AJvYcCXrCqD/Cda2K/UO+KmO4HKjVu/0UM0o93RVDlvwND7ww9aoYsIxZwVRemcJYkC3SdISd6uWfvwWb3qnv0Q=@freebsd.org X-Gm-Message-State: AOJu0YzhHOXgk5vkBqIvh3Jq8jKMhIe5mozxg4Im/YoL6dgLtC8kEM05 3bF+6KL3zMyxsV1Nl9MycWzFMJT/YNC2j5gVib4AENlNxKRiydzE X-Google-Smtp-Source: AGHT+IHCZSlqVTBTlUDxjG0SinNhtLNa+Ng/lDo/GxAOb0zVvoslx8qbRLU0qLFsfO+55WpvVTXp2Q== X-Received: by 2002:a05:6512:15a7:b0:535:6992:f2c3 with SMTP id 2adb3069b0e04-536587f5ce0mr10817602e87.41.1725973758289; Tue, 10 Sep 2024 06:09:18 -0700 (PDT) Received: from nuclight.lan ([37.204.254.214]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5365f90c306sm1153853e87.245.2024.09.10.06.09.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Sep 2024 06:09:18 -0700 (PDT) Date: Tue, 10 Sep 2024 16:09:15 +0300 From: Vadim Goncharov To: "Poul-Henning Kamp" Cc: freebsd-arch@FreeBSD.org, freebsd-hackers@FreeBSD.org, freebsd-net@FreeBSD.org, tech-net@NetBSD.org, Alexander Nasonov Subject: Re: BPF64: proposal of platform-independent hardware-friendly backwards-compatible eBPF alternative Message-ID: <20240910160915.55ff579b@nuclight.lan> In-Reply-To: <202409101224.48ACO7oj094058@critter.freebsd.dk> References: <20240910040544.125245ad@nuclight.lan> <202409100638.48A6cor2090591@critter.freebsd.dk> <20240910144557.4d95052a@nuclight.lan> <202409101224.48ACO7oj094058@critter.freebsd.dk> X-Mailer: Claws Mail 3.19.1 (GTK+ 2.24.33; amd64-portbld-freebsd12.4) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4X33v44LG2z4Dl6 On Tue, 10 Sep 2024 12:24:07 +0000 "Poul-Henning Kamp" wrote: > -------- > Vadim Goncharov writes: >=20 > > It's easy for your Lua code (or whatever) code to hang kernel by > > infinite loop. Or crash it by access on arbitrary pointer. =20 >=20 > Lua has pointers now ? It's implementation has. Do you have mathematical verifier of such loaded bytecode proving it's C interpreter will have no side effects during it's running? > > Your "counter proposal" was essentially available for all these > > decades in form "oh, just write KLD in C instead of that limited > > tcpdump". =20 >=20 > You're yelling at the guy who implemented a (very fast!) firewall > where the rules were compiled to C code in a KLD. That's exactly the way which must be avoided. See 5.2 of https://www.usenix.org/legacy/events/bsdcon02/full_papers/lidl/lidl.pdf > > > If we are going to reinvent "Channel Programs" 67 years after IBM > > > came up with them for their 709 vacuum tube computer, at the very > > > least we should use a sensible language syntax. =20 > > > > Don't know what that is, quick googling [=E2=80=A6] =20 >=20 > Well, you probably should do some more research then, because > unawareness of history is /the/ major cause of pointlessly repeating > mistakes. You're either trolling or completely misunderstand the problem domain.=20 <> (c) https://www.ece.ucdavis.edu/~vojin/CLASSES/EEC272/S2005/Papers/IBM-A= rchitecture-Bashe_sep81.pdf This has nothing to do with BPF at all. Go and read original papers on kernel filters and why they're *such* restricted, e.g. Van Jacobson's paper on BPF/tcpdump, aforementitioned paper on BSD/OS's IPFW (esp. section 5.7 on loops), etc. --=20 WBR, @nuclight From nobody Tue Sep 10 13:44:47 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X34h56SKpz5WYnp; Tue, 10 Sep 2024 13:44:53 +0000 (UTC) (envelope-from vadimnuclight@gmail.com) Received: from mail-lf1-x136.google.com (mail-lf1-x136.google.com [IPv6:2a00:1450:4864:20::136]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X34h54cG5z4Lj6; Tue, 10 Sep 2024 13:44:53 +0000 (UTC) (envelope-from vadimnuclight@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x136.google.com with SMTP id 2adb3069b0e04-5344ab30508so6086811e87.0; Tue, 10 Sep 2024 06:44:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725975892; x=1726580692; darn=freebsd.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=K+m/jXapYzzXBOdJT5EyQSuIHmGF1QSU2be+dgGMMww=; b=T6GVpJhbQSx6/URyN6PPzxi7ypP0zKtbikTBd0H5C9buGHazTYPowuBPGKO/hFk4XP RmK/tIa7ObKa8Ohx4JGDYXaa+Jz1Y5aoIRWI0rl6eFtvhsQYVVpxYIoA3Iwe98Ut0cgL /5QYuXkpwOJwWPr9WeedzPTRZyY06sISA0YgUrKv4jvxdYHXxkldsD7nv6HKARCGn5Zn s6L5/t8AwBJZRgrJMW+xVatWop+XA9RqEE3VSa1hJN1+3Hj1YmFttByuqAD2w8ZEbzhK aBdqAzDR0msKUEkRr3jGk/JXcJ9rV0Fr2ak77Ze2Lzxz68vg5VGO4fJMGOA8iMBMBu8n jIPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725975892; x=1726580692; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=K+m/jXapYzzXBOdJT5EyQSuIHmGF1QSU2be+dgGMMww=; b=eRnu3sc67XvxB61xInDithI9dlO0j2b92NPLWxkctx8zAPd6xp4RJxDLRVip3RxfP6 uQTKK/PSTJmL2XGULIRODsPb+sI48Ts+O/vwG9VtzwdELuEX81LDBbECeaJUTQ52fABo uq+KdjibZh8/FbVGQMBo87Xqh9cGNf/7bjvnphJPYRCmc8+FH19j9bwM2oyQtNm6Hfff GxtFRAB8nG291wyN/KTUG0EaesxBaK9QLZB8eUbfOKyyjgQhWuCpTjQvqDiDGvm8drP9 GjnC+SL8Cx1lE2ifxIPuTJh9OxkhjXMhWLFcxw86nHIC0RqdFXIPErRd+zuKVNx1Orda 47cg== X-Forwarded-Encrypted: i=1; AJvYcCUh5bIP+W0j+lThDKRqnVFX9x8FIb07Panuy+C3XdokX/T8X2q/mJq7DKNphg2I9rIqiEIPGYGEDPhOnEU=@freebsd.org, AJvYcCUy8S3ov+JBxkFjigWHd305wMfVfsuPznmYyOkLh9+PdLwyTSH2OU/BeYpsyoPCYKs3b1w5JnexmYI+TuE5TMDK@freebsd.org, AJvYcCVvzCBXJs5H8FmmzA8hVsotBtIkuFrRJ1aAIyL3COpv40nNt18rEFLXz+gk7C25VunqVD2KncbxTXpy2AA=@freebsd.org X-Gm-Message-State: AOJu0YzZlyIwa40T9zfgx94clsP5S/v0mkdPfW2ce3f2u95Rf6xMdVxO p3s/0aKZVSJMnEx21v4p+YmUiLYYYJ1iheTUU9Fkmx8t6gX+DdTM4C522jRD X-Google-Smtp-Source: AGHT+IF1I+H9IpV3UVoCM8UL/JKt5Oi6jnQ8OUUoZY6/7NCvz13NY+Q89FJXXtqaMIAOsUoRddtaMw== X-Received: by 2002:a05:6512:3e0e:b0:536:628d:20e with SMTP id 2adb3069b0e04-5366bb48633mr1099505e87.29.1725975891203; Tue, 10 Sep 2024 06:44:51 -0700 (PDT) Received: from nuclight.lan ([37.204.254.214]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5365f912ee5sm1181543e87.301.2024.09.10.06.44.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Sep 2024 06:44:50 -0700 (PDT) Date: Tue, 10 Sep 2024 16:44:47 +0300 From: Vadim Goncharov To: David Chisnall Cc: Poul-Henning Kamp , tcpdump-workers@lists.tcpdump.org, "freebsd-arch@freebsd.org" , "freebsd-hackers@freebsd.org" , "freebsd-net@freebsd.org" , "tech-net@netbsd.org" , Alexander Nasonov Subject: Re: BPF64: proposal of platform-independent hardware-friendly backwards-compatible eBPF alternative Message-ID: <20240910164447.30039291@nuclight.lan> In-Reply-To: <4D84AF55-51C7-4C2B-94F7-D486A29E8821@FreeBSD.org> References: <20240910040544.125245ad@nuclight.lan> <202409100638.48A6cor2090591@critter.freebsd.dk> <20240910144557.4d95052a@nuclight.lan> <4D84AF55-51C7-4C2B-94F7-D486A29E8821@FreeBSD.org> X-Mailer: Claws Mail 3.19.1 (GTK+ 2.24.33; amd64-portbld-freebsd12.4) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4X34h54cG5z4Lj6 On Tue, 10 Sep 2024 13:59:02 +0100 David Chisnall wrote: > On 10 Sep 2024, at 12:45, Vadim Goncharov > wrote: > >=20 > > It's easy for your Lua code (or whatever) code to hang kernel by > > infinite loop. Or crash it by access on arbitrary pointer. That's > > why original BPF has no backward jumps and memory access, and eBPF's > > nightmare verifier walks all code paths and check pointers. =20 >=20 > I=E2=80=99m not convinced by the second: Lua has a GC=E2=80=99d heap, you= =E2=80=99d need to > expose FFI things to it that did unsafe things, and that=E2=80=99s equall= y a > problem for eBPF. Not quite. For eBPF (and BPF64) there must be not just FFI but special wrappers or even written from scratch functions keeping in mind they work for restricted environment. Lua, of course, does not have such thing - it will be needed to reimplement standard library. > The first is not a problem. The Lua interpreter has a bytecode > limit. You can define a bounded number of bytecodes that it will > execute. The problem comes from the standard library. Things like > string.gmatch can have high-order polynomial complexity and so it=E2=80= =99s > possible for a Lua program that executes a small number of bytecodes > to create a string that takes a vast amount of time to match on. > Again, this is also a problem for eBPF if you expose a similar > function, the solution is to not expose functions with large > data-dependent runtimes to untrusted script. In BPF64 some safety belts are supposed - e.g. on CALL/RET time is checked, and if exceeded, program is marked unsafe and disabled. > More generally, there are a lot of problems with interpreting or > JITing untrusted code in the kernel in *any* runtime. Speculative > execution makes it easy to use these as primitives to leak kernel > secrets, either via timing of the programs themselves, using the JIT > to generate gadgets, or by leaking data via cache priming. >=20 > Both eBPF and Lua have these problems. > [...] > - Run a channel program. >=20 > In the post-Spectre world, the former remains a privileged operation. > Even though Linux pretends it isn=E2=80=99t, allowing arbitrary (even > arbitrary constrained) code to run in the kernel=E2=80=99s address space = is a > problem. Invoking such code; however, should follow the same rules > as everything else. A trusted entity should be able to load a pile > of Lua / eBPF / BPF64 / whatever programs into the kernel and then > set up permissions so that sandboxed programs (and jails) can use a > defined subset of them. I am not an experience assembler user and don't understand how Spectre works - that's why I've written RFC letter even before spec finished - but isn't that (Spectre) an x86-specific thing? BPF64 has more registers and primarily target RISC architectures if we're speaking of JIT. For BPF64 I've did separate stack as register window exactly to mitigate ROP and it's gadgets. And BPF64 is meant as backwards-compatible extension of existing BPF, that is, it has bytecode interpreter (for(;;) switch/case) as primary form and JIT only then - thus e.g. JIT can be disabled for non-root users in case of doubt. eBPF can't do this - it always exists in native machine code form at execution, bytecode is only for verifier stage. ^^ that's fallback if you say "safe JIT is impossible", but may be you have advices on how to do architecture to still do it safe? As BPF64 looks doable improvement for us in much lower resource investment than even to *porting* eBPF to *BSD. > The thing I would like to see for our current use of semi-trusted Lua > in the kernel (ZFS channel programs) is a way of exposing them (under > /dev/something) as file descriptors and modifying the ioctls that run > them to take a file descriptor argument. I would like to separate > the two operations: >=20 > - Load a channel program. Didn't hear about, looked at the zfs-program(8) and see no reason why these are called "channel" programs (just to please some old farts?) and even reason for them to run in kernel, for same userland-utilities-achi= evable things, seems doubtful. --=20 WBR, @nuclight From nobody Tue Sep 10 14:29:19 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X35gT2FPTz5VtlY; Tue, 10 Sep 2024 14:29:25 +0000 (UTC) (envelope-from jhibbits@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X35gT1Kzyz4S4b; Tue, 10 Sep 2024 14:29:25 +0000 (UTC) (envelope-from jhibbits@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1725978565; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=s3GEIp6Pm27U5X0WQLnTgbvAeltQZEvg1PejEPUlfp4=; b=cvzKdpS4y/XM6wOWOoHbMXAfIppBsl9X8BwqQ7AAqXW9pzvE+Mh62dMSLS4zmRQFjqS7TC rJeGxdzDesn8wxJ9WriGDftWQ3/eScH7FuaOtfSyvY1CstweHpu+twpgaBSK+C2z+STb9j PYhDr9+p6ZfHxcw1iCHoGrUkXddxcBjtL62/ZBH7R8ATRpSyB7gHuTcWFYy1IW7JnDM/4d fidKR9umuA/kYDU126QBHKYfHg/n+JXg/G+FFasoY9dHwcGFlcIDmdUH5hJCcFl/ZAMMUM U4OhAeef162Db9/sXDHWiPK9DS7thAOdoDc1VzdvwOKa7U9/OYJ7bHka09AbvQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1725978565; a=rsa-sha256; cv=none; b=Fq/+YRBojozG5185gbYrB4r+d2lR0/MQva7scfwLKpaTniOKBoEEozKQosyfIe3k9/fiOg iUfKTPTiC2gc4XKgS6rZcl8777Or5eSd2ogz8WOQWDz/mW466GvmI9LRqJ9v8AQK3VYAgL JameTunSlRMuBT9YYhBa1Mc2Pkp4M3JYtDWJgC05N3hqiPdgwy1wsx8AidE3a8vxjuaM0g 00dUPQmsCdmYI6rrJeRNEFOjFP58tIayCoAu4H/T7nqk+LZVLhj3X0Lq8UcGzJBZC0agCT wLRRp7/KieJOK+CnoTX0pkuGsn2cIv+WlLFbinc+SISv/zfMczKicT1QL28CDw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1725978565; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=s3GEIp6Pm27U5X0WQLnTgbvAeltQZEvg1PejEPUlfp4=; b=Ra7qVGw7JRrPb3O11lG8B9ZCMgsWRLRnu+hxI9LzTpoMvXWrXHoVaNHjaoKCogDwhCeeox EkKSXQXELolYnR7SnAAhu1hA0ynYmsNTaxFTiDUHlJpmcPsr5LJZiFVloRfW1IOzlRaBqE 82PXpn0ggAO8bPlZF8AY9f+p3qHYIE/W173ykWFViTdNQu8Ayh+QIQHWW20y+ds80B1i/P UsOkNKeHV4pyrsxNNh9U7T0IjzD7otF8lke3h6Hoqt1QDJLr5XQNXCYuC57hig7zUcDwH8 LqzrUhQbcfzifoP/XyKQOD6VLnLplPUz9DBuYk8klPkqbBafjnlfGogNDPE5hg== Received: from ralga.knownspace (ip-163-182-7-56.dynamic.fuse.net [163.182.7.56]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: jhibbits) by smtp.freebsd.org (Postfix) with ESMTPSA id 4X35gS5DtpzSJp; Tue, 10 Sep 2024 14:29:24 +0000 (UTC) (envelope-from jhibbits@FreeBSD.org) Date: Tue, 10 Sep 2024 10:29:19 -0400 From: Justin Hibbits To: David Chisnall Cc: Vadim Goncharov , Poul-Henning Kamp , tcpdump-workers@lists.tcpdump.org, "freebsd-arch@freebsd.org" , "freebsd-hackers@freebsd.org" , "freebsd-net@freebsd.org" , "tech-net@netbsd.org" , Alexander Nasonov Subject: Re: BPF64: proposal of platform-independent hardware-friendly backwards-compatible eBPF alternative Message-ID: <20240910102919.7d8927d0@ralga.knownspace> In-Reply-To: <4D84AF55-51C7-4C2B-94F7-D486A29E8821@FreeBSD.org> References: <20240910040544.125245ad@nuclight.lan> <202409100638.48A6cor2090591@critter.freebsd.dk> <20240910144557.4d95052a@nuclight.lan> <4D84AF55-51C7-4C2B-94F7-D486A29E8821@FreeBSD.org> X-Mailer: Claws Mail 4.3.0 (GTK 3.24.43; powerpc64le-unknown-linux-gnu) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Tue, 10 Sep 2024 13:59:02 +0100 David Chisnall wrote: > On 10 Sep 2024, at 12:45, Vadim Goncharov > wrote: > >=20 > > It's easy for your Lua code (or whatever) code to hang kernel by > > infinite loop. Or crash it by access on arbitrary pointer. That's > > why original BPF has no backward jumps and memory access, and eBPF's > > nightmare verifier walks all code paths and check pointers. =20 >=20 > I=E2=80=99m not convinced by the second: Lua has a GC=E2=80=99d heap, you= =E2=80=99d need to > expose FFI things to it that did unsafe things, and that=E2=80=99s equall= y a > problem for eBPF. >=20 > The first is not a problem. The Lua interpreter has a bytecode > limit. You can define a bounded number of bytecodes that it will > execute. The problem comes from the standard library. Things like > string.gmatch can have high-order polynomial complexity and so it=E2=80= =99s > possible for a Lua program that executes a small number of bytecodes > to create a string that takes a vast amount of time to match on. > Again, this is also a problem for eBPF if you expose a similar > function, the solution is to not expose functions with large > data-dependent runtimes to untrusted script. >=20 > More generally, there are a lot of problems with interpreting or > JITing untrusted code in the kernel in *any* runtime. Speculative > execution makes it easy to use these as primitives to leak kernel > secrets, either via timing of the programs themselves, using the JIT > to generate gadgets, or by leaking data via cache priming. >=20 > Both eBPF and Lua have these problems. >=20 > The thing I would like to see for our current use of semi-trusted Lua > in the kernel (ZFS channel programs) is a way of exposing them (under > /dev/something) as file descriptors and modifying the ioctls that run > them to take a file descriptor argument. I would like to separate > the two operations: >=20 > - Load a channel program. > - Run a channel program. >=20 > In the post-Spectre world, the former remains a privileged operation. > Even though Linux pretends it isn=E2=80=99t, allowing arbitrary (even > arbitrary constrained) code to run in the kernel=E2=80=99s address space = is a > problem. Invoking such code; however, should follow the same rules > as everything else. A trusted entity should be able to load a pile > of Lua / eBPF / BPF64 / whatever programs into the kernel and then > set up permissions so that sandboxed programs (and jails) can use a > defined subset of them. >=20 > David >=20 This sounds a lot like IBM i / OS400 / System/360, or even Singularity from Microsoft, which uses a trusted JIT or AOT compiler to convert arbitrary bytecode to constrained machine code, so the machine code translator becomes the only trusted party, to accept or reject the arbitrary source (byte)code from the user. - Justin From nobody Tue Sep 10 14:58:25 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X36KB534yz5Vxnp; Tue, 10 Sep 2024 14:58:38 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X36KB4M8tz4bZ8; Tue, 10 Sep 2024 14:58:38 +0000 (UTC) (envelope-from theraven@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1725980318; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=RgfWg2bulIbwtlV/4Gjmiz25NxsSSsRKimR8IwQMrQE=; b=Yqjp1onLc6T66JnXpukvkLGCGByhHTqXQ/326gFljqWs22byt1+e31gu+GCi/qbx1Qe6is 19z0z9mVS66YgQnLUcS8Fz5A2RX2KYusseSh5luBCoLs5aUugArFyS9I70IO7MGUA5Kz6d lWjLGexiEHflu+Vbtgb4qeqaKpr5cLMisHU7Nt47w8Kmuudk3TS1OHJwwx85Ru4oJ156U3 l8R7uKdnkjq0HlcgFFQlrd8u+/TqjRFjvd30cVT72QopUl2IpcAEUxhWyC0YYCJIvhnkP5 +zeQyqQpg+Q9/lmtGm4wXowi5s2CZYfEBp0XtFOFICzaDFaX6K+LnUTLU+uS+A== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1725980318; a=rsa-sha256; cv=none; b=r7J1AAmkrkm88Vv2xWg8U5zAVGa+oX4cpBbaBVKACfFD7x3NRSIVu05KyQwMQL5yG2WZyt Q3mPi9W6FkJr+qTpzJlpVZ6BiGdlc9ekOH8MGH46zWjkNV+CNRQlHBabZvH/r3O4sUHqPs 6c5dUmvpMZEEEBSJvn23ioPLwYjA7g0f7v+EuPSR3yDVoDFZpMFonBC5rg7cI7ceP/yn8s 1NtdgdOcRPVVpI6xEKbTSMOWiFonw7QgqQcRTBkUe3K7Ez/rW+p2WYCZ4BpkwfiQ1bjFmS j/LAwD8cbfSJUr7cvByrmKoDrbTBfDjDrLRVzVeYYxnVZO7f2gk2DsaP+GPMqw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1725980318; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=RgfWg2bulIbwtlV/4Gjmiz25NxsSSsRKimR8IwQMrQE=; b=q7NHQXHrOSUosQwO+DzcksYTdwz4obEKP7ay6HIibVp7dwPFksuNRXj5sP48Tj396u9aAp +nWPswQuT7dFtKGmNEIWLzx/QaAzKaNUQzQO1Q5d+vPgTD7sJgK4u5kTgexTYzKf7h8dus 3DspqJvgkEq+rSS/7k55rXxyOGUICY3pRc0cAuvGsfhelj0rqRi3CTGFuO9+eqT1T9xBJz RSASC+aARtnuWwLKcKyhPz3StD71XHDdzb8aEOzcFcQkv3JnUsQ/7PY/Ya2SZ9yA4fr+6i 7JB2X+kMw0q+nvd1HxmuuP55rhSdwyT28vfUxIZjLQS5ihaUonNv4WbU0fZbiA== Received: from smtp.theravensnest.org (smtp.theravensnest.org [45.77.103.195]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: theraven) by smtp.freebsd.org (Postfix) with ESMTPSA id 4X36KB3VKczSKG; Tue, 10 Sep 2024 14:58:38 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from smtpclient.apple (host109-155-136-107.range109-155.btcentralplus.com [109.155.136.107]) by smtp.theravensnest.org (Postfix) with ESMTPSA id DA1E365B7; Tue, 10 Sep 2024 15:58:36 +0100 (BST) From: David Chisnall Message-Id: <3F3533E4-6059-4B4F-825F-6995745FDE35@FreeBSD.org> Content-Type: multipart/alternative; boundary="Apple-Mail=_C79A00F0-FADC-4B5C-84B5-8912A75C117E" List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3776.700.51\)) Subject: Re: BPF64: proposal of platform-independent hardware-friendly backwards-compatible eBPF alternative Date: Tue, 10 Sep 2024 15:58:25 +0100 In-Reply-To: <20240910164447.30039291@nuclight.lan> Cc: Poul-Henning Kamp , tcpdump-workers@lists.tcpdump.org, "freebsd-arch@freebsd.org" , "freebsd-hackers@freebsd.org" , "freebsd-net@freebsd.org" , "tech-net@netbsd.org" , Alexander Nasonov To: Vadim Goncharov References: <20240910040544.125245ad@nuclight.lan> <202409100638.48A6cor2090591@critter.freebsd.dk> <20240910144557.4d95052a@nuclight.lan> <4D84AF55-51C7-4C2B-94F7-D486A29E8821@FreeBSD.org> <20240910164447.30039291@nuclight.lan> X-Mailer: Apple Mail (2.3776.700.51) --Apple-Mail=_C79A00F0-FADC-4B5C-84B5-8912A75C117E Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 On 10 Sep 2024, at 14:44, Vadim Goncharov = wrote: >=20 > I am not an experience assembler user and don't understand how Spectre > works - that's why I've written RFC letter even before spec finished - = but > isn't that (Spectre) an x86-specific thing? BPF64 has more registers > and primarily target RISC architectures if we're speaking of JIT. No, speculative execution vulnerabilities are present in any CPUs that = do speculative execution that does not have explicit mitigations against = them (i.e. all that have shipped now). Cache side channels are present = in any system with caches and do not have explicit mitigations (i.e. all = that have shipped so far). Mitigations around these things are an active research area, but so far = everything that=E2=80=99s been proposed has a performance hit and = several of them were broken before anyone even implemented them outside = a simulator. > And BPF64 is meant as backwards-compatible extension of existing BPF, > that is, it has bytecode interpreter (for(;;) switch/case) as primary > form and JIT only then - thus e.g. JIT can be disabled for non-root > users in case of doubt. eBPF can't do this - it always exists in = native > machine code form at execution, bytecode is only for verifier stage. This has absolutely no impact on cache side channels. The JIT makes = some attacks harder but prime-and-probe attacks are still possible. BPF can be loaded only by root, who can also load kernel modules and map = /dev/[k]mem, and FreeBSD does not protect the root <-> kernel boundary. Please read some of the (many) attacks on eBPF to better understand the = security landscape here. It=E2=80=99s a *very* hard problem to solve. David --Apple-Mail=_C79A00F0-FADC-4B5C-84B5-8912A75C117E Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 On 10 Sep = 2024, at 14:44, Vadim Goncharov <vadimnuclight@gmail.com> = wrote:

I am not an experience assembler user and = don't understand how Spectre
works - = that's why I've written RFC letter even before spec finished - = but
isn't = that (Spectre) an x86-specific thing? BPF64 has more registers
and = primarily target RISC architectures if we're speaking of JIT.

No, speculative execution = vulnerabilities are present in any CPUs that do speculative execution = that does not have explicit mitigations against them (i.e. all that have = shipped now).  Cache side channels are present in any system with = caches and do not have explicit mitigations (i.e. all that have shipped = so far).

Mitigations around these things are an = active research area, but so far everything that=E2=80=99s been proposed = has a performance hit and several of them were broken before anyone even = implemented them outside a simulator.

And BPF64 is meant as = backwards-compatible extension of existing BPF,
that = is, it has bytecode interpreter (for(;;) switch/case) as = primary
form = and JIT only then - thus e.g. JIT can be disabled for non-root
users = in case of doubt. eBPF can't do this - it always exists in = native
machine = code form at execution, bytecode is only for verifier stage.

This has absolutely no impact = on cache side channels.  The JIT makes some attacks harder but = prime-and-probe attacks are still possible.

BPF = can be loaded only by root, who can also load kernel modules and map = /dev/[k]mem, and FreeBSD does not protect the root <-> kernel = boundary.

Please read some of the (many) = attacks on eBPF to better understand the security landscape here. =  It=E2=80=99s a *very* hard problem to = solve.

David

= --Apple-Mail=_C79A00F0-FADC-4B5C-84B5-8912A75C117E-- From nobody Tue Sep 10 15:17:11 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X36kj54Gbz5W0jd; Tue, 10 Sep 2024 15:17:17 +0000 (UTC) (envelope-from vadimnuclight@gmail.com) Received: from mail-lf1-x136.google.com (mail-lf1-x136.google.com [IPv6:2a00:1450:4864:20::136]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X36kh6wg7z4h7w; Tue, 10 Sep 2024 15:17:16 +0000 (UTC) (envelope-from vadimnuclight@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x136.google.com with SMTP id 2adb3069b0e04-5365b71a6bdso5150035e87.2; Tue, 10 Sep 2024 08:17:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725981434; x=1726586234; darn=freebsd.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=W+EtQAWIcFr+qyUilH/Xt2d7xpL9Fl6f8mgiQAu4olk=; b=O55e2XG+xisoVf3uMiBEdhWfYniaXzAQYrUJHVg1sXj+MGRDanOdQtICsuv2TplTUd 361DPWCVnIWoTon7NGA5ZFKuCwah9rllLz2PgBDZ5irWsyjMNvBGlV4zL1RSEG9AGc1K cqJEb2v3amcci6zXKiGeSsGF7J1tttOMVTCXGB7OTZrHsFKlNamq49nFO52z2aJoVCau qYS/8MbEzTkiVBWd6eN/zj5EMlRnU4OUZQ6wA/LcAIZ2MK3TGfFBx0dy7VSGo9RfDXnu 71K/kQN2Naz+TxI3d0kqtx2jK5+kHCZVZB9UmVSjSVft0Htg1CsizUjN1s2Pis5IiEDa c8ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725981434; x=1726586234; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=W+EtQAWIcFr+qyUilH/Xt2d7xpL9Fl6f8mgiQAu4olk=; b=nhnhVMjkVhbtgyGerv/WRmZ6ndHTbdBeB1YerOkPrdYsbUp+9pfggG+d5ebzXDupzX SNVNtyxo5DB6XjZ9iaHz441XZchTvtILLfzQqO0ibs3D6iY+Vky+2J7x0gLCqob7xWCA j0V0SwqY42VwSkwQKoTHv6EdCpq0415eqcv++FfBK5F3mUXwmZQBa4VrZU2JbrM9p/Hl pQt/z312SqP3iMMPpIGwj1axb1gYujizwFW2fUWxIAoLRhDXT7uxmoZoHicoHLZzl6tx 9IHWEe8vZl4LaUM8E3CNODNks45tenshhhkLuZeQrVIB4PiK/JyW5Y0MxxHooag5Atnt IsjQ== X-Forwarded-Encrypted: i=1; AJvYcCUBrZXLR5iL++2H5jTPRqk24AzFLTFq+nhBgOtGpW6la2Jo5125JwEVlVHTVydGUunpyTsumveEuwYKSL7ezUE=@freebsd.org, AJvYcCV/6pcIHgaoxKppqu7x19+k1dC7jaMNwXUoVjAyP6Rcwnj44SECl2i57W1dI1As7Q1IuvjU6a2DxYWdVEg=@freebsd.org X-Gm-Message-State: AOJu0YxorJ8tM1QPfCYVH3paHt3Gxxx7OdtWyIfkil/ZctV41tMyvebk B4eWRJ1FE0MGH4j+RKYUW1frAQgw0q8FWJu9Y/Vi+/eN8xnuRPOT X-Google-Smtp-Source: AGHT+IHB+HtNqR5Mhpl+gR9Q78EPAz5C5X6JRviKpNPloEM6ML1p7/WA+anUT0/yPZnqvpaJWeXDyw== X-Received: by 2002:a05:6512:2215:b0:534:5453:ecda with SMTP id 2adb3069b0e04-536587ae6c8mr9702590e87.23.1725981433978; Tue, 10 Sep 2024 08:17:13 -0700 (PDT) Received: from nuclight.lan ([37.204.254.214]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5365f86925asm1198726e87.31.2024.09.10.08.17.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Sep 2024 08:17:13 -0700 (PDT) Date: Tue, 10 Sep 2024 18:17:11 +0300 From: Vadim Goncharov To: "Gavin D. Howard" Cc: freebsd-arch@FreeBSD.org, freebsd-hackers@FreeBSD.org, freebsd-net@FreeBSD.org, tcpdump-workers@lists.tcpdump.org, tech-net@NetBSD.org, Alexander Nasonov Subject: Re: BPF64: proposal of platform-independent hardware-friendly backwards-compatible eBPF alternative Message-ID: <20240910181711.5d324ac5@nuclight.lan> In-Reply-To: References: <20240910040544.125245ad@nuclight.lan> X-Mailer: Claws Mail 3.19.1 (GTK+ 2.24.33; amd64-portbld-freebsd12.4) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4X36kh6wg7z4h7w On Tue, 10 Sep 2024 14:41:20 +0000 "Gavin D. Howard" wrote: > Hello, > > New user here, not a contributor. > > > Ensuring kernel stability? Just don't allow arbitrary pointers, > > like original BPF. Guaranteed termination time? It's possible if > > you place some restrictions. For example, don't allow backward > > jumps but allow function calls - in case of stack overflow, > > terminate program. Really need backward jumps? Let's analyze for > > what purpose. You'll find these are needed for loops on packet > > contents. Solve it but supporting loops in "hardware"-controlled > > loops, which can't be infinite. > > If I understand Turing-completeness correctly, the "no backward jumps > but allow recursion and quit on stack overflow" is exactly equivalent > to having non-infinite loops. Sure, but then look at practical usefulness: suppose you have stack of 32 frames (current limit of eBPF and BPF64). Then you can have only 31 iterations on a loop, loosing ability to call more functions from loop body. > I'm not entirely sure, but I think the lack of backwards jumps would > be "simple" to check on LLVM IR: just make sure that a basic block > does not jump, directly or indirectly, to a basic block that > dominates it. [1] > > [1]: https://en.wikipedia.org/wiki/Dominator_(graph_theory) > > And then the stack overflow mechanic would be an entirely runtime > check, which is safer than pure static checking. > > But the good thing about this is that FreeBSD could use LLVM IR as the > BPF64 language, which means any language that compiles to LLVM is a > possible target. > > As for restricting access, I think it would be possible to check the > instructions in LLVM IR for any unsafe instructions or calls to > restricted functions. > > The downsides: > > * Someone would need to write an LLVM analyze pass or whatever they're > called. Maybe more than one. > * The kernel would need the ability to compile LLVM IR, making LLVM > part of the Ring 0 domain. > * Either that, or someone builds an LLVM-to-bytecode > translator. Well, using LLVM were supposed for higher-level languages when bytecode is no longer experimental, utilizing full power of optimizer, but for direct using... let's see how they use it in Linux currently: 1) you write .c code, relatively low-level/restricted, with eBPF .h-s 2) clang turns .c into LLVM IR file (yes, this step is separate, may be things has changed since then, but at least it was so) 3) file with LLVM IR turned to eBPF bytecode in ELF file 4) ELF .o loaded by ip(8) or specialized loader into kernel 5) verifier checks it and most probably will return error to you :) 6) bytecode is JIT-compiled in kernel and then may be run Linux people are in mood "let's throw more man-months instead of thinking of better design", eBPF infrastructure consists of hundreds of thousandls lines of code, but seems that utilizing LLVM IR directly required too much resources even from them so was rejected. > * But the analysis pass(es) must still live in the kernel. > * There would need to be tutorials or some docs on how to write > whatever language so that backwards jumps are avoided. So BPF64 took simplicity pass: while you have tutorials etc. it's still very hard to write (non-toy) code that passes verifier. I think a language where you do not need backward jumps but have usable constructs (so you just can't write bad code), even BAW, is a better way to go than try to train people to fight with unnecessary Cerber. > Please take my words with a full dose of salt; I'm not an expert on > this or on FreeBSD goals. BPF64 is not FreeBSD-only, you can see several non-FreeBSD mailing lists here. It can be cross-platform and independent enough to be implemented in e.g. network card or switch, for performance - having more registers allows to achieve better results then eBPF for same goal. -- WBR, @nuclight From nobody Tue Sep 10 15:24:40 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X36vF5SGlz5W1P0 for ; Tue, 10 Sep 2024 15:24:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X36vF3QZlz4mxR for ; Tue, 10 Sep 2024 15:24:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1725981881; a=rsa-sha256; cv=none; b=klmfItRPFgAIewy8+D4FN80Skcxhrogc7Xq18lVOSGwDjsBMrT7ZIDPoXYINJ1Zzgk+1Y7 RpH8QU6Q5WLvH14P9abaaS82Qm4ndAdRxYktDXRHOLbZ+dyRiW2OpDFQ7NNjsQNd1hqcSv 5Y3zu/VY2A7yZKBIj87fU6b3/YaNSLesIafCzDoBkKJxQnBjiQvJFBfvxKZxrnS1Zm+w6t +cKd8HlgrHxD0qt5+GNVAisFknFl01Mn730tSVdDzKTKM7WhMwNf+XoVMqkAsi79j7UqiH 89240hYlkekn+IZzDJNgysw/ffj0/RGjIh6UcsWAFJHhhZ/uCf3tGwJhPodXyw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1725981881; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BBb/UPZ2HHE6JLiHeH4jVlBDRDcih9Nm/OjbXw6F9Ss=; b=FnknsJPaN0wHHRpt4pfFEKBgCnhRY1LKi7be4vbQptzTS0Ornt8IfLWnsHRcLYqyy4CR6z 0Jmdo53qD1BW/QaI2S6mmIWNCIB8P6v97Kv/OQs6auzfJUOHCgmT9lA8c3EdJINcyqpDrg qnkjNQd5lTA+/OozxZeNUZf7muEhHTT3h39M2gqwXBX6hLrYzxBbIIKPgLEE9ZMiQ6N4+v YphGbMFI0r8oskRAALXtggLmtoj3dsSi5rlU4+tn1u6HXvU2LG15WaWyirpk350FR6uhl5 sURBZgdKs8G5/vQek7snECUKGmVTw4I6dZNGXP/Rmr1c1aZ3RiOoqc1UZlVgAA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4X36vF30C2zsd1 for ; Tue, 10 Sep 2024 15:24:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 48AFOfnj065219 for ; Tue, 10 Sep 2024 15:24:41 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 48AFOfQI065218 for net@FreeBSD.org; Tue, 10 Sep 2024 15:24:41 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 281391] IPv6 multicast sent to wrong MAC address Date: Tue, 10 Sep 2024 15:24:40 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D281391 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |net@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Sep 10 19:04:40 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X3CnC4Lk6z5WT67; Tue, 10 Sep 2024 19:04:47 +0000 (UTC) (envelope-from alnsn@yandex.ru) Received: from forward501d.mail.yandex.net (forward501d.mail.yandex.net [178.154.239.209]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4X3CnC1c43z4SDZ; Tue, 10 Sep 2024 19:04:47 +0000 (UTC) (envelope-from alnsn@yandex.ru) Authentication-Results: mx1.freebsd.org; none Received: from mail-nwsmtp-smtp-production-main-24.klg.yp-c.yandex.net (mail-nwsmtp-smtp-production-main-24.klg.yp-c.yandex.net [IPv6:2a02:6b8:c42:2d4c:0:640:de18:0]) by forward501d.mail.yandex.net (Yandex) with ESMTPS id 5D0AF608E9; Tue, 10 Sep 2024 22:04:43 +0300 (MSK) Received: by mail-nwsmtp-smtp-production-main-24.klg.yp-c.yandex.net (smtp/Yandex) with ESMTPSA id e4oBfACoCKo0-nXAlgKWZ; Tue, 10 Sep 2024 22:04:42 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1725995082; bh=3E8FQYsPiOVQqyKRmki79OX4khwQPIv/cFy4p19RpQs=; h=Message-ID:Subject:References:To:From:In-Reply-To:Cc:Date; b=AstHlPfEJDQ59hdwaIZQTCj4D1L3wuaW+vTKo8Y3AZ4vgvq0nOYkvxFbFqOrDzo3V KxGtNabuZWlviT0cvlnfeGzE/S7nIasiIE7yMOfNdfVmzc1I1mTYL+FzC7xbfadNJF /e5akAtRl++SrhgB3XJcVtzTVcoEVOgdeBWXFKJY= Date: Tue, 10 Sep 2024 20:04:40 +0100 From: Alexander Nasonov To: Poul-Henning Kamp Cc: Vadim Goncharov , freebsd-arch@FreeBSD.org, freebsd-hackers@FreeBSD.org, freebsd-net@FreeBSD.org, tech-net@NetBSD.org, Alexander Nasonov Subject: Re: BPF64: proposal of platform-independent hardware-friendly backwards-compatible eBPF alternative Message-ID: References: <20240910040544.125245ad@nuclight.lan> <202409100638.48A6cor2090591@critter.freebsd.dk> <20240910144557.4d95052a@nuclight.lan> <202409101224.48ACO7oj094058@critter.freebsd.dk> <20240910160915.55ff579b@nuclight.lan> <202409101432.48AEWu1q094702@critter.freebsd.dk> List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <202409101432.48AEWu1q094702@critter.freebsd.dk> X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:200350, ipnet:178.154.224.0/19, country:RU] X-Rspamd-Queue-Id: 4X3CnC1c43z4SDZ Poul-Henning Kamp wrote: > A) How powerful do you want the downloaded bytecode to be ? ... > There are basically two possible answers to A. > > Either the downloaded code is "real", which means it can include > loops, function calls etc. or it is a "toy" which relies on > Brinch-Hansen's "all arrows point to the right" argument to prove > that it will always terminate. BPF can be easily modified to have functions and calls but it will increase worst case execution complexity to quadratic and the stack growth will have to be watched closely: - always jump forward (within the current function), - always call backwards (outside of the current function). Backward jumps (within the current function) can be implemented with some kind of slowly closing door. For instance, in packet filtering, the first backward jump disables access to the first byte, the second backward jump disables access to the second byte, etc. In general, it will require a special counter but BPFJIT compiler should be able to spot simple loops that process at least one byte on each iteration and minimise runtime overhead. > ... > The answer to that is that you can write them in /any/ programming > language you want, as long as the interpreter for the resulting > (byte-)code an spot attempts to jump backwards, and refuse to > do so, if so configured. I guess it will work for a simple single-pass interpreter that generates bytecode as it reads source code but anything more complicated may reshuffle bytecode and some forward jumps may become backward jumps etc. > And that is why I say: If we're going to do this, let us do > it with Lua: It's already in our tree and it will do the > job nicely. Lua is a good choice, IMO but we likely need a subset of Lua (e.g. no metatables, no C or paused GC with a periodic lua_State rotations to collect garbage). Alex From nobody Tue Sep 10 22:12:28 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X3Hxt0Zhpz5TfHl; Tue, 10 Sep 2024 22:12:34 +0000 (UTC) (envelope-from vadimnuclight@gmail.com) Received: from mail-lj1-x22e.google.com (mail-lj1-x22e.google.com [IPv6:2a00:1450:4864:20::22e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X3Hxs4cWVz4sWw; Tue, 10 Sep 2024 22:12:33 +0000 (UTC) (envelope-from vadimnuclight@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lj1-x22e.google.com with SMTP id 38308e7fff4ca-2f025b94e07so50790821fa.0; Tue, 10 Sep 2024 15:12:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726006351; x=1726611151; darn=freebsd.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=VQklzI5f+UhE/4HXXB6RHKU+J3Qo/ASreqA6+yuuQMA=; b=Gt0Mxk0EpZN59+KWga6Y85o6QJxPxRtDzMbNCHMxOpiO95o/UbDHjCCMYrjGqeIu7I 6EVZDtyxrXexv/aKlQBVKWCfvz+YmoJMuTk20Iuz80tjI7KS3diMT+yDBZghH4SlK6+0 7vNouIrF8WU3PrIOuDzHKfZyudU2vMPdiEyRFBPVtm8NJPkCuww9BZKwlnm3zsL6eTiQ lJHMWVFumb9aqhaHr5oxWd/UKf79Tk4pu+kDoLVi5p2SRacCxoGBztvLqGmCwqVUpQ0f 41p8A6rARrO8fy/7iFLPQ2EX57k+xDRpUh8msj8o9zwdCSa5JvamrHhJDzCM8yAeevZE aG4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726006351; x=1726611151; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VQklzI5f+UhE/4HXXB6RHKU+J3Qo/ASreqA6+yuuQMA=; b=FrWGWSlI4V3C/Y5L7RHRSZ88lRJqCqQ44c8jPsOjgh/JlTg8/cjjy2vTpr2xwqWqLW 1xqlqDpWV8357RJycWFzS5yTsQEwXDi2pVcXZv3LHcoUeEz4lX/VmTsvaqZFysW9gSTV s5ID/WErJj0fR6yBT6+ZsCoNBG1Rl6/osSwKfp06Ov0alR/GWqNXEKGSeHE+FkipS3pn NMVpl68rsT1wuu+GeEbHVukVud5Ri6Fie5o5U3/97UaoqieywVsiK44zx4jeh4eAVsH1 rNUplXNXShVR7mYYN4qwixMgkpRuBPx2FwxKmLaxHdMsURLY4dH1MIiD3/zjiSj/azNP vkEA== X-Forwarded-Encrypted: i=1; AJvYcCUXyhl85uihZ4xjmNCvxgw17NL/6on20C7C+DJyGBs/HvOzIvL5HVYPlh3vj35o3OzEEpxXp4Omsr9pQeeCSSwe@freebsd.org, AJvYcCX+g+SUU4VBC6hqQRhaoGw/K6YNNGZyKCWdY1y2RTiuZJsKefVZ9xAuO2gIoh7r1lk6tT17vBviGrqpC+c=@freebsd.org, AJvYcCXfLs0EFHjn8KXuPPelq99ALzOXy1TOAcVLhxFuESmpvVFWMxBuEj1iqCkteUYCV+liXHKVGslHkagEidc=@freebsd.org X-Gm-Message-State: AOJu0YzSy9AGJhXKHBVk7pjrgB96buI3IoxqhmPrxrzqoyTyXXWMTQvZ o/O1jYKXV2n9QZHMqWoObZ31hPPniboiPuYefbUctmApXTQPewTDnWqsHZjW X-Google-Smtp-Source: AGHT+IHV0riUgCN7WGyLkuUPPbvyKnE92FLj4+LbmsITqsigNsHw13jPjtefeDp//sUQyk2pW6hqNA== X-Received: by 2002:a05:651c:1a0b:b0:2f7:6653:8044 with SMTP id 38308e7fff4ca-2f76653812dmr52539331fa.20.1726006350849; Tue, 10 Sep 2024 15:12:30 -0700 (PDT) Received: from nuclight.lan ([37.204.254.214]) by smtp.gmail.com with ESMTPSA id 38308e7fff4ca-2f75c07cf5esm13208111fa.86.2024.09.10.15.12.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Sep 2024 15:12:30 -0700 (PDT) Date: Wed, 11 Sep 2024 01:12:28 +0300 From: Vadim Goncharov To: David Chisnall Cc: Poul-Henning Kamp , tcpdump-workers@lists.tcpdump.org, "freebsd-arch@freebsd.org" , "freebsd-hackers@freebsd.org" , "freebsd-net@freebsd.org" , "tech-net@netbsd.org" , Alexander Nasonov Subject: Re: BPF64: proposal of platform-independent hardware-friendly backwards-compatible eBPF alternative Message-ID: <20240911011228.161f94db@nuclight.lan> In-Reply-To: <3F3533E4-6059-4B4F-825F-6995745FDE35@FreeBSD.org> References: <20240910040544.125245ad@nuclight.lan> <202409100638.48A6cor2090591@critter.freebsd.dk> <20240910144557.4d95052a@nuclight.lan> <4D84AF55-51C7-4C2B-94F7-D486A29E8821@FreeBSD.org> <20240910164447.30039291@nuclight.lan> <3F3533E4-6059-4B4F-825F-6995745FDE35@FreeBSD.org> X-Mailer: Claws Mail 3.19.1 (GTK+ 2.24.33; amd64-portbld-freebsd12.4) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4X3Hxs4cWVz4sWw On Tue, 10 Sep 2024 15:58:25 +0100 David Chisnall wrote: > On 10 Sep 2024, at 14:44, Vadim Goncharov > wrote: > >=20 > > I am not an experience assembler user and don't understand how > > Spectre works - that's why I've written RFC letter even before spec > > finished - but isn't that (Spectre) an x86-specific thing? BPF64 > > has more registers and primarily target RISC architectures if we're > > speaking of JIT. =20 >=20 > No, speculative execution vulnerabilities are present in any CPUs > that do speculative execution that does not have explicit mitigations > against them (i.e. all that have shipped now). Cache side channels > are present in any system with caches and do not have explicit > mitigations (i.e. all that have shipped so far). >=20 > Mitigations around these things are an active research area, but so > far everything that=E2=80=99s been proposed has a performance hit and sev= eral > of them were broken before anyone even implemented them outside a > simulator. >=20 > > And BPF64 is meant as backwards-compatible extension of existing > > BPF, that is, it has bytecode interpreter (for(;;) switch/case) as > > primary form and JIT only then - thus e.g. JIT can be disabled for > > non-root users in case of doubt. eBPF can't do this - it always > > exists in native machine code form at execution, bytecode is only > > for verifier stage. =20 >=20 > This has absolutely no impact on cache side channels. The JIT makes > some attacks harder but prime-and-probe attacks are still possible. Wait, do you want to say that problem is not in JIT, that is, that current BPF (e.g. tcpdump) present in the kernel - are also vulnerable? Also, let's classify vulnerabilities. Is speculative execution vulnerability the same as cache side channels? In any case, what impact is? E.g. attacker could leak secrets, but *where* would them leak? BPF typically returns one 32-bit number as a verdict (often as just boolean), is it really attack vector? That is, may be solution is just "don't give read access to BPF-writable memory segments to untrusteds". Next, if problem is with timing, then isn't that enough to just restrict BPF code on having access to timers with resolution high enough? > BPF can be loaded only by root, who can also load kernel modules and > map /dev/[k]mem, and FreeBSD does not protect the root <-> kernel > boundary. Wrong. It is possible for decades to do `chmod a+r /dev/bpf*` and run tcpdump as non-root, which will load BPF code into kernel. Is *that* also a vulnerability, and if so, why it was never reported? > Please read some of the (many) attacks on eBPF to better understand > the security landscape here. It=E2=80=99s a *very* hard problem to solve. =20 Finally, the most big (in effort) question: suppose we limited to trusted root user etc. so it's of no concern. Are there now any objections/suggestions/comments on (rest of) BPF64 ? --=20 WBR, @nuclight From nobody Tue Sep 10 23:47:35 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X3L3g4wZ9z5TtpP; Tue, 10 Sep 2024 23:47:43 +0000 (UTC) (envelope-from vadimnuclight@gmail.com) Received: from mail-lj1-x22c.google.com (mail-lj1-x22c.google.com [IPv6:2a00:1450:4864:20::22c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X3L3f3QfJz476k; Tue, 10 Sep 2024 23:47:42 +0000 (UTC) (envelope-from vadimnuclight@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=JQpQoezA; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of vadimnuclight@gmail.com designates 2a00:1450:4864:20::22c as permitted sender) smtp.mailfrom=vadimnuclight@gmail.com Received: by mail-lj1-x22c.google.com with SMTP id 38308e7fff4ca-2f6580c2bbfso2953571fa.1; Tue, 10 Sep 2024 16:47:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726012060; x=1726616860; darn=freebsd.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=pLn0PIT2lwV+Djl805vfqvUiLnwCki2KPcUPimdE1mU=; b=JQpQoezARMrUdMigjLsU/NfW1PkDPD6Aa9gI/PQhObE02NfRyrPdaSQ6vRkXU6R1zh T4Hc2rNqRXGal/sCKKcfjsi+5bEtq/CH5VxpE1dxlIrbVNRCVlTy+sKUQlNGCYDIyHKt DSAbTF2ygoqINrpt0Q7nQjtKIrwbT74stMHfRuMq5bzd8LQc/uyL9TCyxIkXC9b7kBUq odIfyHokp6V8hKB8PA4pVxI8OyBrSO2oDPMx9qdvji/4kMGrYyJgsoesh22JqwDg8pvG VhkJoS0xSKqGEoHzKpQz6ThNEtoX0LniyRwVEOBDVsFNNzD3lnfmSWKjd4k6Mu3YWX8i 2VyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726012060; x=1726616860; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pLn0PIT2lwV+Djl805vfqvUiLnwCki2KPcUPimdE1mU=; b=QPN1tM80O+C4c0E87P9QyQURC4x8kaRGbxu9upCwovByDg6whRVKwoE9CiLvJRzfaq /UNIAbT/2JpQPlmH5hMrvkgbSPoK2H2+49f/Zyx+BK6TAponmTMK2oxsLQgO0Uf1cPTp wHqvoPFieJb3E9EvDsumbsS4Yt0lFlnfK2ltE64jjluE85jZkBmb6popiHyST5ZJjpT0 EzP1K+O/EcQCtdodk74OfVm5jSXvge/jg+fZznB8YxVgPpOJsiWgrSqJDVJkvanCnvoK nGNXwCen6XZzrr9QKyfIs6DKwNIobXfAjqiizk0MWMrgExAPfz021udRvJ363j81jBbg mNrw== X-Forwarded-Encrypted: i=1; AJvYcCUIhjJ1GZRlt1kOYsVRvTaHCzQjlfvbcTtK8BKhqFe3kZkHEiJhKqxtcZCLTJS+aBRfmTIIWkqzTNbgXPw=@freebsd.org, AJvYcCVGNoZBKdqC15RKc8pIdSRYdRHvNZEwXWp/DYG/M7jHLEJ9WAfooxVtC96a7itePRGsqdXi4RZRslml18s=@freebsd.org, AJvYcCWi2SotehQSxWaJ+u/mW/Uy3+8KQqFrcZwNTgqAXdIMErrwUVs/6G1MPBqbA9KD+ggZoDpepqnrCjMQN2jviU1B@freebsd.org X-Gm-Message-State: AOJu0Yxn37B/OcQfcfcOqHSvFDY8bWdY/BnkM1Um1u5lrUdTof1O4iK8 1m7P/Z4gp1qvWrCip2T/qb0aU8xtwa5eBPbpxDP7EwywWd0lrORHEGjD4OjM X-Google-Smtp-Source: AGHT+IFS3vPrdngBZPNcSrE198Tqj5S5aP4CHm4pOYWmA1awgaVQ++udPn+7VFedXQGZ7dqVjpIOBQ== X-Received: by 2002:a05:6512:1387:b0:536:552e:5d36 with SMTP id 2adb3069b0e04-5366b91f158mr1474569e87.12.1726012059924; Tue, 10 Sep 2024 16:47:39 -0700 (PDT) Received: from nuclight.lan ([37.204.254.214]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5365f90d482sm1362675e87.262.2024.09.10.16.47.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Sep 2024 16:47:39 -0700 (PDT) Date: Wed, 11 Sep 2024 02:47:35 +0300 From: Vadim Goncharov To: Rob Wing Cc: David Chisnall , Poul-Henning Kamp , "freebsd-arch@freebsd.org" , "freebsd-hackers@freebsd.org" , "freebsd-net@freebsd.org" , "tech-net@netbsd.org" Subject: That's how (why) BSD is dying (Was: BPF64) Message-ID: <20240911024735.37522532@nuclight.lan> In-Reply-To: References: <20240910040544.125245ad@nuclight.lan> <202409100638.48A6cor2090591@critter.freebsd.dk> <20240910144557.4d95052a@nuclight.lan> <4D84AF55-51C7-4C2B-94F7-D486A29E8821@FreeBSD.org> <20240910164447.30039291@nuclight.lan> <3F3533E4-6059-4B4F-825F-6995745FDE35@FreeBSD.org> <20240911011228.161f94db@nuclight.lan> X-Mailer: Claws Mail 3.19.1 (GTK+ 2.24.33; amd64-portbld-freebsd12.4) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.49 / 15.00]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.989]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; FREEMAIL_TO(0.00)[gmail.com]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MLMMJ_DEST(0.00)[freebsd-arch@freebsd.org,freebsd-hackers@freebsd.org,freebsd-net@freebsd.org]; TAGGED_RCPT(0.00)[]; RCPT_COUNT_SEVEN(0.00)[7]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::22c:from] X-Rspamd-Queue-Id: 4X3L3f3QfJz476k On Tue, 10 Sep 2024 15:05:21 -0800 Rob Wing wrote: > Doesn't NetBSD have Lua in the kernel...anyone have any experience > with it? Lua is not suitable for the discussed problem domains, don't listen to those who did not read material and did not understand short descriptions. > re BPF64: >=20 > looks like hand waving, proposing two unwritten languages that > extends an ISA that is still being drafted by IETF...hmm What's the point to waste resources on writing thing that is known to be not accepted to base system from the very beginning? FreeBSD already had precedent when a *ready* code framework was rejected be some FreeBSD users' enemy, leaving FreeBSD users to suck with absolutely *no* alternative (yep, sensors) - as ridiculous as if it was to say "current BSD firewalls are inferior to Linux so we'll better sit without that bad code (and firewall) AT ALL". Yes, the situation in networking for us was for many years - and still is - exactly so. This is exactly the way how to loose market competition and die - just don't try to innovate and do something better than competitor; then you find yourself porting compatibility layers after decades of rot, like Netlink or Linuxulator ABI, in a try to at least hobble with a cane instead of lying in a coma. > you can make anything look good on paper but the devil is in the > details...which there are plenty of So do you have to say something constructive on the real subject? Or even bikesheds are in short supply now? > On Tuesday, September 10, 2024, Vadim Goncharov > wrote: >=20 > > On Tue, 10 Sep 2024 15:58:25 +0100 > > David Chisnall wrote: > > =20 > > > On 10 Sep 2024, at 14:44, Vadim Goncharov > > > wrote: =20 > > > > > > > > I am not an experience assembler user and don't understand how > > > > Spectre works - that's why I've written RFC letter even before > > > > spec finished - but isn't that (Spectre) an x86-specific thing? > > > > BPF64 has more registers and primarily target RISC > > > > architectures if we're speaking of JIT. =20 > > > > > > No, speculative execution vulnerabilities are present in any CPUs > > > that do speculative execution that does not have explicit > > > mitigations against them (i.e. all that have shipped now). Cache > > > side channels are present in any system with caches and do not > > > have explicit mitigations (i.e. all that have shipped so far). > > > > > > Mitigations around these things are an active research area, but > > > so far everything that=E2=80=99s been proposed has a performance hit = and > > > several of them were broken before anyone even implemented them > > > outside a simulator. > > > =20 > > > > And BPF64 is meant as backwards-compatible extension of existing > > > > BPF, that is, it has bytecode interpreter (for(;;) switch/case) > > > > as primary form and JIT only then - thus e.g. JIT can be > > > > disabled for non-root users in case of doubt. eBPF can't do > > > > this - it always exists in native machine code form at > > > > execution, bytecode is only for verifier stage. =20 > > > > > > This has absolutely no impact on cache side channels. The JIT > > > makes some attacks harder but prime-and-probe attacks are still > > > possible. =20 > > > > Wait, do you want to say that problem is not in JIT, that is, that > > current BPF (e.g. tcpdump) present in the kernel - are also > > vulnerable? Also, let's classify vulnerabilities. Is speculative > > execution vulnerability the same as cache side channels? In any > > case, what impact is? E.g. attacker could leak secrets, but *where* > > would them leak? BPF typically returns one 32-bit number as a > > verdict (often as just boolean), is it really attack vector? That > > is, may be solution is just "don't give read access to BPF-writable > > memory segments to untrusteds". > > > > Next, if problem is with timing, then isn't that enough to just > > restrict BPF code on having access to timers with resolution high > > enough? > > =20 > > > BPF can be loaded only by root, who can also load kernel modules > > > and map /dev/[k]mem, and FreeBSD does not protect the root <-> > > > kernel boundary. =20 > > > > Wrong. It is possible for decades to do `chmod a+r /dev/bpf*` and > > run tcpdump as non-root, which will load BPF code into kernel. Is > > *that* also a vulnerability, and if so, why it was never reported? > > =20 > > > Please read some of the (many) attacks on eBPF to better > > > understand the security landscape here. It=E2=80=99s a *very* hard > > > problem to solve. =20 > > > > Finally, the most big (in effort) question: suppose we limited to > > trusted root user etc. so it's of no concern. Are there now any > > objections/suggestions/comments on (rest of) BPF64 ? > > > > -- > > WBR, @nuclight > > > > =20 --=20 WBR, @nuclight From nobody Wed Sep 11 09:05:18 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X3ZRB6BRCz5WB2X; Wed, 11 Sep 2024 09:05:26 +0000 (UTC) (envelope-from vadimnuclight@gmail.com) Received: from mail-lf1-x12a.google.com (mail-lf1-x12a.google.com [IPv6:2a00:1450:4864:20::12a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X3ZRB04qPz4BHx; Wed, 11 Sep 2024 09:05:26 +0000 (UTC) (envelope-from vadimnuclight@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=XZDy8rHe; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of vadimnuclight@gmail.com designates 2a00:1450:4864:20::12a as permitted sender) smtp.mailfrom=vadimnuclight@gmail.com Received: by mail-lf1-x12a.google.com with SMTP id 2adb3069b0e04-5366fd6fdf1so2272410e87.0; Wed, 11 Sep 2024 02:05:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726045524; x=1726650324; darn=freebsd.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=aygyrhQ+/cZDvRSNjzEqcDkc5n4UHMSS0q3kDo8ROik=; b=XZDy8rHeoSRNIob10aLzEwgev209wHvm0C9hqPI8yrGq5+Pqownx9/gjdI0pXltJdR ZnhHKHivvTvARAXuGkuh7otMDmTlXMXISO1yl+UaVwIN72GzK65qKm0NP8gZILBui1Zp dsGJurnd8HlHtq2jcl1AA7TGY+X6t2dc3hYgh8PYjfcsgFIRuYMjtHJW4AKLkVaz2O4O ddnUPVN5XZXT7efl2ITc7zR7NtjbleU0hyY44HCAJ53JrKjzg/J2F+RSmM8CiCAfydVY vaw1P3AyVTeeLdjdxncU4R9XE51xlKPgACeD8+YB6x7csXSgteH3UUQO6bNkOEo7r/vk CgjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726045524; x=1726650324; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aygyrhQ+/cZDvRSNjzEqcDkc5n4UHMSS0q3kDo8ROik=; b=a8h6jY1OEQYvExg+gxcBH+tiAsVIyV9C3CL87QjwRPxZBVwT3oCV7Sac9tlH6pfz99 AzFv1KNzYv2acMDBJT2MX7vJAo7QHhfbdMfmTICBQmHYsJo1Ousg6vT4n1HBUnHYKLwV g5IxDCVxe65G/qcLu2r8TTJl7BDTZZaxUjmgVyGJjEzZ4rZI+9WXJI3KKtmV+TWtbDBB u0+e2ltH+y75SU4H09d/3E6SSg8c1/ipyyLqrbSgAdCY1vlGZVfupGrxneC+kOnTAQbc sLviSzLz2xEKMws0wMeddpzkzQxtHRY4e7W2qiueo1OPw5ExPBenHhEHnxWR2bi3I4dK Ty6Q== X-Forwarded-Encrypted: i=1; AJvYcCU9O2gsRL2NL9BKC3i0VaJhTuMnlu0/XwRydalh3F1obUngb/MLHEDQ41xs6/Mq1mR8frUawRoMPgOtW87nAID3@freebsd.org, AJvYcCUfnW5Ccji0temXzlyfHjObcGTWBmJjmu49Bm4UYaKAezK2xEKTj9rBci/KhnddgdoxCfySysqTjCCe5qs=@freebsd.org, AJvYcCWpzp9DwEFJzU0jpe7tlTucwb8MFjStk/1avOuTdn4vPjcrimbUVbAD38Y5kkxJMLeUy8aK7PgqtC/Xwsg=@freebsd.org X-Gm-Message-State: AOJu0YwmCX2pMGd5uuieeQq6NHed7hx4j1Rl1HGBMULQ0FRGOscNfT3W LpTGY17PVs7KLhz8nw40PSz1o9SLgPGsMF0xbano+X/B6wEo0wrV X-Google-Smtp-Source: AGHT+IHFLBaXxhDXONnQW29ANP0GW0U6/BDTTyLArumdV+Kh1qEDoKIpQr+cVHC495aTj9UXrGCLEQ== X-Received: by 2002:a05:6512:3989:b0:535:3ce5:6173 with SMTP id 2adb3069b0e04-536587f6087mr11930442e87.37.1726045523447; Wed, 11 Sep 2024 02:05:23 -0700 (PDT) Received: from nuclight.lan ([37.204.254.214]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5365f912ee5sm1513806e87.301.2024.09.11.02.05.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Sep 2024 02:05:23 -0700 (PDT) Date: Wed, 11 Sep 2024 12:05:18 +0300 From: Vadim Goncharov To: Philip Paeps Cc: David Chisnall , Poul-Henning Kamp , freebsd-arch@FreeBSD.org, freebsd-hackers@FreeBSD.org, freebsd-net@FreeBSD.org, tech-net@NetBSD.org Subject: Re: BPF64: proposal of platform-independent hardware-friendly backwards-compatible eBPF alternative Message-ID: <20240911120518.1ba191b5@nuclight.lan> In-Reply-To: References: <20240910040544.125245ad@nuclight.lan> <202409100638.48A6cor2090591@critter.freebsd.dk> <20240910144557.4d95052a@nuclight.lan> <4D84AF55-51C7-4C2B-94F7-D486A29E8821@FreeBSD.org> <20240910164447.30039291@nuclight.lan> <3F3533E4-6059-4B4F-825F-6995745FDE35@FreeBSD.org> <20240911011228.161f94db@nuclight.lan> X-Mailer: Claws Mail 3.19.1 (GTK+ 2.24.33; amd64-portbld-freebsd12.4) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.55 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.55)[-0.554]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[text/plain]; RCVD_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-arch@freebsd.org,freebsd-hackers@freebsd.org,freebsd-net@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::12a:from]; RCPT_COUNT_SEVEN(0.00)[7]; DKIM_TRACE(0.00)[gmail.com:+] X-Rspamd-Queue-Id: 4X3ZRB04qPz4BHx On Wed, 11 Sep 2024 10:14:44 +0800 Philip Paeps wrote: > On 2024-09-11 06:12:28 (+0800), Vadim Goncharov wrote: > > David Chisnall wrote: > >> BPF can be loaded only by root, who can also load kernel modules > >> and map /dev/[k]mem, and FreeBSD does not protect the root <-> > >> kernel boundary. > > > > Wrong. It is possible for decades to do `chmod a+r /dev/bpf*` and > > run tcpdump as non-root, which will load BPF code into kernel. Is > > *that* also a vulnerability, and if so, why it was never reported? > > This is equivalent to chmod a+w /dev/mem. > > Unwise configuration decisions are not vulnerabilities. But then a possibility to give this to non-root is. And many things are considered vulnerabilitites even if they are only available to root - for example, when root can be tricked into running malicious code etc. (unconscious) actions without direct intention. Equivalency of classic BPF to writable /dev/mem is too loud and controversial statement. Demonstrate how it can be done on stock FreeBSD 13 with /dev/bpf available to attacker (e.g. `sudo tcpdump` allowed). -- WBR, @nuclight From nobody Wed Sep 11 11:21:09 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X3dS230wdz5WS1f; Wed, 11 Sep 2024 11:21:22 +0000 (UTC) (envelope-from theraven@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X3dS20pX2z4ZLd; Wed, 11 Sep 2024 11:21:22 +0000 (UTC) (envelope-from theraven@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1726053682; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=f/2x/6NihxJiRrA1BaJGxpPygChNh4vd66EqRwfJEzQ=; b=maL8AuBoi5mVYQcLONv0iPiXg1fht+yMMOqm+CvjlIZ2TPrHMi9ZGyCYl5/f41yk6lW7Ay 4mjJu9HugSIlQ1+4tOkAibwKGMlrQ+aidcxmwVTsF1FZypo8zwcL2E5WGyEJT9GSQNfEvC ni0nvX7vJ5/yCuP12pxQMzj0+ALztRT4/cOkyoyCwY1/DNddFKiB3hTQaNYh2VD2IBCobf iKEHH1G7R4z+CIklTRRDHpXu45QqIFDseS8HhawIAdMxu2GqQXiMntMKiSKkBNkFjdogjD x+9oJ9iH+tQk1UKkcUiSY655Jbz5QchC2Zi66/Gqzt1txr3BGFNIpyE5FEjePw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1726053682; a=rsa-sha256; cv=none; b=nNimEszrCpoKTZ6eJJWhwAeIkHo2A24DEDAvdgJiivd1FIqwZNiub/Z33EjNyXsaSfcuzE dWVBYhhJWEX9/1AHuHv1SaKHBABu083XiZ01F0xJeAvjCALDdUXxhft3lGGr31/gTvucwk wDQPhpMr6wDBhw0cmw2w7T9f/Finaagu948QHlNKzRuFyn5TNGTtknKMcXhHG0UkI9gPMr VjgPnE6pvjG++rumuCEv61882Bjp0slSS3X3lWOuKkRSs0dh7laeX/i1FK+Gl9SmUbunDf YbgQ3NhMijzjqnw4CDXWmRs9TPiZU+7bZ5tkgal32K4YyI4ITRIdAqvaGnaPbw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1726053682; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=f/2x/6NihxJiRrA1BaJGxpPygChNh4vd66EqRwfJEzQ=; b=Jx69LwcELMGUEZggBBlBAssiSMYuDvjr1egy2h9mJd5tLnEClD0COIoplTIy+c5um90GK2 LgvMFdJAlTs7hEWr+K51Cv3bcv1yeYOvGe6sqQJH+1iFjLKSmCzF3DAGDcX2v8C7GbHkfO oJotKy9VmdeIT0K5hUUaKSom344zvg7gtnUcg8P9Sacu1gS9+9dS+K8Fvd9I8px6LO/OXi vGtbjK5016UmhsxxHEoPPJeGUdRF4TrLgmAUNtSUqedLuY7/wwZneASy9wRiAqIL8Nd6Pp DdXKeIWcov6be5miBqLrA7901hcnqcWWPFGMoppQwFB1cdx3Jm3wVHYSFWS2Gw== Received: from smtp.theravensnest.org (smtp.theravensnest.org [45.77.103.195]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: theraven) by smtp.freebsd.org (Postfix) with ESMTPSA id 4X3dS20FYrz18Zn; Wed, 11 Sep 2024 11:21:21 +0000 (UTC) (envelope-from theraven@freebsd.org) Received: from smtpclient.apple (host109-155-136-107.range109-155.btcentralplus.com [109.155.136.107]) by smtp.theravensnest.org (Postfix) with ESMTPSA id F245D65DC; Wed, 11 Sep 2024 12:21:20 +0100 (BST) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable From: David Chisnall List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 (1.0) Subject: Re: BPF64: proposal of platform-independent hardware-friendly backwards-compatible eBPF alternative Date: Wed, 11 Sep 2024 12:21:09 +0100 Message-Id: References: <20240911120518.1ba191b5@nuclight.lan> Cc: Philip Paeps , Poul-Henning Kamp , freebsd-arch@freebsd.org, freebsd-hackers@freebsd.org, freebsd-net@freebsd.org, tech-net@netbsd.org In-Reply-To: <20240911120518.1ba191b5@nuclight.lan> To: Vadim Goncharov X-Mailer: iPad Mail (21G93) On 11 Sep 2024, at 10:06, Vadim Goncharov wrote: >=20 > But then a possibility to give this to non-root is. And many things are > considered vulnerabilitites even if they are only available to root - > for example, when root can be tricked into running malicious code etc. > (unconscious) actions without direct intention. When the root user intentionality changes some thin from a secure default to= an insecure setting, it is not a security vulnerability in the system that s= hipped the safe defaults.=20 > Equivalency of classic BPF to writable /dev/mem is too loud and > controversial statement. Demonstrate how it can be done on stock > FreeBSD 13 with /dev/bpf available to attacker (e.g. `sudo tcpdump` > allowed). Two things to unpick here. First: Demanding a proof of concept before you accept that something is a vulnerabi= lity is how you build insecure systems. Ask the Matrix team how well that at= titude has worked for them in the last few weeks. You build secure systems b= y defining a threat model and then evaluating primitives against that threat= model, not by throwing together a bunch of primitives and saying =E2=80=98w= ell, *I* can=E2=80=99t assemble them into an exploit and so no one can=E2=80= =99. Second, there are documented attacks on eBPF that give the equivalent of *re= ad* access to /dev/mem. This is why BPF is restricted to root. We have a thr= eat model. The threat model says that we do not need to ensure that BPF cann= ot leak kernel data indirectly because only the user who has the ability to l= eak kernel data directly can use it and this user has a simpler way of achie= ving the same result. If you allow non-root users to run code (native or aga= inst any virtual machine) then you are changing the threat model. You *must*= prevent users from leaking kernel data that they could not leak via existin= g mechanisms. The two most common attacks using eBPF are generally in the following two ca= tegories: - Use eBPF to mount a speculative execution attack on the kernel. Please re= ad up on what these can do. No one should be building a thing that runs code= in the kernel without understanding speculative, cache, and timing side cha= nnels. - Use eBPF to build a set of gadgets that you can then use to go from one m= emory-safety bug in the kernel to arbitrary-code execution. This is the threat landscape in which something in the same space as eBPF mu= st exist. A proposed design should *start* with an explanation of how it mit= igates both of these categories of attack. David From nobody Wed Sep 11 23:47:54 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X3y1f4yDYz5Vsfm for ; Wed, 11 Sep 2024 23:48:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X3y1f2V5Jz4CQR for ; Wed, 11 Sep 2024 23:48:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1726098486; a=rsa-sha256; cv=none; b=xJxVM/ERUrkAU2B101tnqGYA0zIHmId4oBDFn6/NgSwTnPfgV2RkNUwsYqhBzTHm49LbR/ 3lM9XH5JuZ1ET//6O/xZi0zbHX7t7LN40nVXpMWjIvFBS11VCD/ZYXqG8eV5sqHeZ7UeWL +nQgzLjUmMW1q7p9pUiguhgEvlmSw9l628qFXtw4lxZCxVcuyIEVIbFJtxMd59GtYq9lIE 2/GwfRstnoie1vn0g/I5uyrtluKSU3a5DJvL1A7sB0nv7exkw5jpkv5McXjv0id6DqBEWO 5FLlzONsqE5bTDfSPbwftFZm9myF8VK7yvIf9ZlnRF4xbMrV8Ywd/QYL1jZ+fw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1726098486; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=B64kpsWAjJvfaWDh0HQojd12XRPdvlXujOMlW4gsMaY=; b=xujNHKsgPvVRzWhNr0G1QjIQx4xB4u2i2W39lBdRa5ZCBLQpcOvYZ1t+qeJ3fGGKCRkmUX d3MDYYYz1iCMGOPZ1Vw/4PzqZVZCW5TXBBsz42zIDtaOhKrQoJAFYtjGr0/UrWA6cOzdxl rENhTK4qAMwy+36uQQ7QNhVRWF5J8s0enDsaT2MaThRdC9Z5MvSD/Z/Qr2G6mWl/QVZfLr 4deA/HZr4VOjOldMQo/wM3//9bSwCfLAmdXd/6dX3qRA/GXuzCaTeTkytYPPoAmyVgyqm8 b+csYlWJ2phogZzESJA8zVKbouY5X8tr12s5hpheW4IVdprQeMRiyvXf3wOLBg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4X3y1f25HpzrFy for ; Wed, 11 Sep 2024 23:48:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 48BNm67H026478 for ; Wed, 11 Sep 2024 23:48:06 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 48BNm6jc026468 for net@FreeBSD.org; Wed, 11 Sep 2024 23:48:06 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 280701] FreeBSD-SA-24:05 fix breaks ICMP/ICMP6 states handling in pf firewall (ping, traceroute) Date: Wed, 11 Sep 2024 23:47:54 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.1-RELEASE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: emaste@freebsd.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: see_also Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D280701 Ed Maste changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.freebsd.org/bu | |gzilla/show_bug.cgi?id=3D2= 813 | |97 See Also| |https://bugs.freebsd.org/bu | |gzilla/show_bug.cgi?id=3D2= 813 | |95 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Sep 11 23:47:54 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X3y1h3X6Sz5Vsmg for ; Wed, 11 Sep 2024 23:48:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X3y1h1gL0z4CRS for ; Wed, 11 Sep 2024 23:48:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1726098488; a=rsa-sha256; cv=none; b=evEymiPqttDDcZ3WXG91vfWiKG5MvykxKvd6pVJ1VI8Xtcg2s7C9kJsPmQY/4EA2QGJavR qLWT9qsw8FIsECvkgJYkmM+iEPfIpUBLU1u+NLzA1sdn5f8py0TPSXRb+GPtxEJ+8UBN79 l7UV53Ta/8INDOmUxQuzvOK4p2n60VMpFUy1gSN32+abMDnpvVAxYgHji5dhgeMlHes2yy UTyOC3cyigB5Zm4IzM1d+Dhjbh2xSLAuGNcK3e0XnonB3W4/P1/LoGDQv8TnmcZa+6oJq9 NrQfsRXVPmDbKilSrJrW4RoLUZnSKJZ2UXXLSBMNhvuLhtfvC6uOV2KfSx/mSg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1726098488; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=B64kpsWAjJvfaWDh0HQojd12XRPdvlXujOMlW4gsMaY=; b=FXSbDJ4TZ93go0WvknahhYs3Eh5Lhsxj3EAmu//SfzW/a6Y2ePFjNMRFj4SA2pBEjoleWn o0SzMe1czMWL/NqpugxDLVwl/CVK9gmsM/d+V2xOQAwkOhKf2MoqRPwzQK1T5HEfnvvdq1 O5g8EXl3F8gaktsLpVYFx4llFf7iYwBoM+PABWdwwxkdk2ribim54a8eXACvPyNssYYcXt vFtzXrvt8AIVJKRUcya/zWPZwBdnWp1++qHAswpXv82sDYuTnjza0UYqzTCT0FETnDGYj7 MwHRlE2MD9CsmvH2vkW/ChXBV1oSwtqYOPTyIl0maShCqRHduUtIDPHjY+Jx7A== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4X3y1h1G8zzrxp for ; Wed, 11 Sep 2024 23:48:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 48BNm7Wr026527 for ; Wed, 11 Sep 2024 23:48:07 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 48BNm7n5026524 for net@FreeBSD.org; Wed, 11 Sep 2024 23:48:07 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 280701] FreeBSD-SA-24:05 fix breaks ICMP/ICMP6 states handling in pf firewall (ping, traceroute) Date: Wed, 11 Sep 2024 23:47:54 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.1-RELEASE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: emaste@freebsd.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: see_also Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D280701 Ed Maste changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.freebsd.org/bu | |gzilla/show_bug.cgi?id=3D2= 813 | |97 See Also| |https://bugs.freebsd.org/bu | |gzilla/show_bug.cgi?id=3D2= 813 | |95 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Sep 11 23:57:17 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X3yDM44CTz5VtfM; Wed, 11 Sep 2024 23:57:23 +0000 (UTC) (envelope-from vadimnuclight@gmail.com) Received: from mail-lj1-x22b.google.com (mail-lj1-x22b.google.com [IPv6:2a00:1450:4864:20::22b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X3yDM25B7z4FPd; Wed, 11 Sep 2024 23:57:23 +0000 (UTC) (envelope-from vadimnuclight@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lj1-x22b.google.com with SMTP id 38308e7fff4ca-2f7528f4658so3540351fa.3; Wed, 11 Sep 2024 16:57:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726099041; x=1726703841; darn=freebsd.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=BTPaR5wX0V190g/8+LMON57KsrpCZmF7qQI/FvvC2Eo=; b=PzmCM5AwJk1+Q33RogTWyofj6rh8QWyOOeG2ic+sqIhpV+4MUS336FCYh9UMVgM2ij 6/YOIHwh8IvPdJ34RqSDqHWspqPesiUwKw19rhDOEo/N51SvBI226a5v1PjAq9a7LTlP g8OBsBbcMFr8+qgcDf29puSLcQUQ8FQ0Z3pEAnVO+t5W7qGtYIHMc2g3yPuvlpTDjHbu 0Uan+7ktxA+qncYUpCdI5TE6nJEIT/uB6GdKUV1k9wNPilc+XuZ7eKSMN32FbbY3bzPt x8/etQq60KBVO1K8NTrkIb71ZzzZkkH61yjanLojZ9QBz5upmbjSCTk50aXkNAyYBRRX 11Lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726099041; x=1726703841; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BTPaR5wX0V190g/8+LMON57KsrpCZmF7qQI/FvvC2Eo=; b=Ty+wEIgCCgpYQFMwkygd6eOe+7ImdCuVr+JKgJl/xHBM7G7DgCRlEO5TQtUvj5PjcW DR9E6iTnI8aOtrbBueg2+FNeQ8pSf/vV5meZgchEwPr4VOlJcbwoBD+9+H47ViqNCiLH vO+kwwaPQEq/sUdw7/cZbIUspobfXFd2wvuXNocsBiLwDRcSvhXzDRermzvLPQqoAQK/ JqUUeewsRp2rTUod60VqgLxJBFXbGdhrdHHwHLGxiopGW8fRSmNi8mnjjFlfOTCQ7wZ9 3ASt7IM2uCbqKojS9N6e0T220fLECh21O5mDafeSAa7AmAcoSz0e0NmYToREz+HhxNYj c2vQ== X-Forwarded-Encrypted: i=1; AJvYcCVL8OZtSsiUnabVOX1pjaihQjzwv0PTnukQzOXxLEw6/JB51GNwMKm1ed3Xga1TsSnBsL9TshTjm3zEraM=@freebsd.org, AJvYcCVQ7NSSj1FXVG/MxG4zFSvIWQZMK2IeXke47w0ORiSHbW28ieGhtOcHMmkqbrk1bQ8KoJ1fD5JjgOKKWiA2c0mX@freebsd.org, AJvYcCWNMiTIVfl3EJD1J+o9PFxAQe4hgMzoyW5EIHFYuTGgENrUTLa7slAr2vQ2VQvbjNY77gUQ7VGcgKiqGnk=@freebsd.org X-Gm-Message-State: AOJu0Yyar0aj7zuPTn/Kgrk4Qnrv9VAbfOy5KD/QOwuUJcDx2GteMKCC XE404ntEafb2c5pI9a8Bupxet00EjyH4NelqF9GDnCoF8cGbpDTosPc93YEh X-Google-Smtp-Source: AGHT+IEB8ksEThG5HzDJjvPwwBoIofOrfOch9Ucn/tK7deg+YJt5CHaIIOQyMT9bCCvQTwHNfo1qfA== X-Received: by 2002:a05:651c:2105:b0:2f3:ac52:416b with SMTP id 38308e7fff4ca-2f787f33a2cmr3863201fa.35.1726099040895; Wed, 11 Sep 2024 16:57:20 -0700 (PDT) Received: from nuclight.lan ([37.204.254.214]) by smtp.gmail.com with ESMTPSA id 38308e7fff4ca-2f75bfe6aecsm17094201fa.5.2024.09.11.16.57.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Sep 2024 16:57:20 -0700 (PDT) Date: Thu, 12 Sep 2024 02:57:17 +0300 From: Vadim Goncharov To: David Chisnall Cc: Philip Paeps , freebsd-arch@freebsd.org, freebsd-hackers@freebsd.org, freebsd-net@freebsd.org, tech-net@netbsd.org Subject: Re: BPF64: proposal of platform-independent hardware-friendly backwards-compatible eBPF alternative Message-ID: <20240912025717.455295f1@nuclight.lan> In-Reply-To: References: <20240911120518.1ba191b5@nuclight.lan> X-Mailer: Claws Mail 3.19.1 (GTK+ 2.24.33; amd64-portbld-freebsd12.4) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4X3yDM25B7z4FPd On Wed, 11 Sep 2024 12:21:09 +0100 David Chisnall wrote: > On 11 Sep 2024, at 10:06, Vadim Goncharov > wrote: > >=20 > > But then a possibility to give this to non-root is. And many things > > are considered vulnerabilitites even if they are only available to > > root - for example, when root can be tricked into running malicious > > code etc. (unconscious) actions without direct intention. =20 >=20 > When the root user intentionality changes some thin from a secure > default to an insecure setting, it is not a security vulnerability in > the system that shipped the safe defaults.=20 This is just not true. See, for example, FreeBSD-SA-17:06.openssh for vulnerability disabled by default, and workaround proposed to return to default (disabled) state. > > Equivalency of classic BPF to writable /dev/mem is too loud and > > controversial statement. Demonstrate how it can be done on stock > > FreeBSD 13 with /dev/bpf available to attacker (e.g. `sudo tcpdump` > > allowed). =20 >=20 > Two things to unpick here. First: >=20 > Demanding a proof of concept before you accept that something is a > vulnerability is how you build insecure systems. Ask the Matrix team > how well that attitude has worked for them in the last few weeks. You > build secure systems by defining a threat model and then evaluating > primitives against that threat model, not by throwing together a > bunch of primitives and saying =E2=80=98well, *I* can=E2=80=99t assemble = them into an > exploit and so no one can=E2=80=99. >=20 > Second, there are documented attacks on eBPF that give the equivalent > of *read* access to /dev/mem. This is why BPF is restricted to root. Stop. Just stop, and re-read carefully. You (and perhaps Philip) confusing two things: BPF and eBPF (and BPF64 third), all completely different beasts. Last two letters in this thread, I was talking about classic BPF existing in *BSD for decades (on FreeBSD allowed to have permissions on dev/bpf*). So you assert that THIS classic BPF also vulnerable to aforementioned attacks, and thus SA must be issued, just like that FreeBSD-SA-17:06.openssh, with a fix (at least preventing changing default permissions) of hole existing for *decades*. This is too strong assertion to be accepted without proofs, and as I can deduce from your other words and readings about Spectre (see below), this statement is not true (classic /dev/bpf is not vulnerable). > We have a threat model. The threat model says that we do not need to > ensure that BPF cannot leak kernel data indirectly because only the > user who has the ability to leak kernel data directly can use it and > this user has a simpler way of achieving the same result. If you > allow non-root users to run code (native or against any virtual > machine) then you are changing the threat model. You *must* prevent > users from leaking kernel data that they could not leak via existing > mechanisms. >=20 > The two most common attacks using eBPF are generally in the following > two categories: >=20 > - Use eBPF to mount a speculative execution attack on the kernel. > Please read up on what these can do. No one should be building a > thing that runs code in the kernel without understanding speculative, > cache, and timing side channels. >=20 > This is the threat landscape in which something in the same space as > eBPF must exist. A proposed design should *start* with an explanation > of how it mitigates both of these categories of attack. Again, you are talking about eBPF here, not classic BPF. So far Spectre was mentioned as example of those speculative, cache, and timing side channels: https://en.wikipedia.org/wiki/Spectre_(security_vulnerability) refers to mitigations e.g. in Firefox - https://www.mozilla.org/en-US/security/advisories/mfsa2018-01/ with key phrase "The precision of performance.now() has been reduced from 5=CE=BCs to 20=CE= =BCs," So to prevent this class of attacks you need to deprive untrusted code from (precise) timers. And if we then go eBPF sources, we see at https://elixir.bootlin.com/linux/v6.10/source/include/uapi/linux/bpf.h#L1884 * u64 bpf_ktime_get_ns(void) * Description * Return the time elapsed since system boot, in nanoseconds. * Does not include time the system was suspended. * See: **clock_gettime**\ (**CLOCK_MONOTONIC**) This is because eBPF is used in Linux as one-catch-all for tracing and profiling - they do not have DTrace. And we have. And BPF don't need to be DTrace and don't need timers. Thus, we can conclude, your eBPF assertion is simply not applicable to *BSD classic BPF, and consequently, to current state of BPF64 which don't include any timers or time sources available to user code at all. > - Use eBPF to build a set of gadgets that you can then use to go > from one memory-safety bug in the kernel to arbitrary-code execution. This requires further explanations (including to ensure we're not mixing things again). As far as I can see currently, this is also not applicable to BPF64 due to lack of pointers. --=20 WBR, @nuclight From nobody Thu Sep 12 00:26:14 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X3ysn4Gflz5Vym1; Thu, 12 Sep 2024 00:26:21 +0000 (UTC) (envelope-from vadimnuclight@gmail.com) Received: from mail-lj1-x233.google.com (mail-lj1-x233.google.com [IPv6:2a00:1450:4864:20::233]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X3ysm1GqYz4M7X; Thu, 12 Sep 2024 00:26:20 +0000 (UTC) (envelope-from vadimnuclight@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=jwlbnWxx; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of vadimnuclight@gmail.com designates 2a00:1450:4864:20::233 as permitted sender) smtp.mailfrom=vadimnuclight@gmail.com Received: by mail-lj1-x233.google.com with SMTP id 38308e7fff4ca-2f75b13c2a8so5057741fa.3; Wed, 11 Sep 2024 17:26:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726100778; x=1726705578; darn=freebsd.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=cYoWHowlaxUIxtahmcQwJ2VDHGzi1OZKXzI/rYfVDfY=; b=jwlbnWxxnS3GL55mBLX7gZVXBFiLGIH6ljRENO0LaB7+MvHFU6HyLoxk2xjm9r1MVG nzOupoNR1juC06bUH7ztouZ8OmeDJvm3ANvXie/aJT2eAkM4oHbKxa+KA35IStImsB80 VkbAHPu2Z1ac6pcJp0QQcE3oQ8tpKuaJlpg2kYTMn4Nx9Q6sQP2YCidDnKVEFZysF08X cw0sBym2ZtBI3luefdW57cQ+CZBBxzhB787cmzTOR8pOTnoBKh0Dcy6AYgF7QT5G4LaD OBhNy9/nGF0UXanf595IigXVJNpZT9MhASvXk+VqVFJ/AB0YypWP+29jbiSOsvxMVfgl 859g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726100778; x=1726705578; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cYoWHowlaxUIxtahmcQwJ2VDHGzi1OZKXzI/rYfVDfY=; b=QHuYbXeqzgI6Usji1e+EsSYrqy/hSedlFg8knMbInCtcLzYGAtkGWkKZQubNHHgUpd KdI25OwI+gIlDsxlwm/nqpGxGu6prGxEmJ6Q4fCU11HawmiDJzm7SkW9Rx2xFQwoQwZu d0dXCfD1tAaY9lfMJKSSIT0MJzR7OJRdBX7UUuAHDuC1eFBXslqJG9aQDSY0aTj6m+D7 kfAa1P28kD3Q0e6wJ10/iBh6kL71AMXl3z625mUFmsqTQ2g5VtIAe1/oAEnaibFQhElE 2fr/mhd/p1FPEDS1DW46WS0RYdOay8ViQSnjC6RqSsVz2jmYlMgg8TJrZuO8Rac7gQJJ mt5w== X-Forwarded-Encrypted: i=1; AJvYcCXA9etlJIVR002cenHH3bT6WDke6MlfLSnoxs9gerNtSntBDhCXik4xncyF7FaU/vd4pS4NnyALtnZ81LhS/fQ=@freebsd.org, AJvYcCXLT6KCfkdn3b141C114OgYlBzyBQda5gqEGQJUDpAtQdKGm/du4106tMPPzp31W6EpPHBtYIn2G5lQHwU=@freebsd.org X-Gm-Message-State: AOJu0YykR4+1ZRuVWKuiyv71VpVMllbdl9PvhLUu0F5MV+DyVLQfHYmH AHkR7HYAdTzpsuYC6FMdLqaO+xsnv6efYvqhKRlbmB/swlz7JdIU X-Google-Smtp-Source: AGHT+IFG03hcxcakfjwOaJGlKdhk0XrccinW5TqrwyPuUhlOiQiPNISLM95vPsD6s52zTxb4QtHeBw== X-Received: by 2002:a2e:a553:0:b0:2f7:631a:6e21 with SMTP id 38308e7fff4ca-2f787ed5c6amr4932251fa.24.1726100777473; Wed, 11 Sep 2024 17:26:17 -0700 (PDT) Received: from nuclight.lan ([37.204.254.214]) by smtp.gmail.com with ESMTPSA id 38308e7fff4ca-2f75c008f6csm17091391fa.59.2024.09.11.17.26.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Sep 2024 17:26:17 -0700 (PDT) Date: Thu, 12 Sep 2024 03:26:14 +0300 From: Vadim Goncharov To: Rob Wing Cc: "freebsd-arch@freebsd.org" , "freebsd-hackers@freebsd.org" , "freebsd-net@freebsd.org" , "tech-net@netbsd.org" Subject: Re: That's how (why) BSD is dying (Was: BPF64) Message-ID: <20240912032614.36796b03@nuclight.lan> In-Reply-To: References: <20240910040544.125245ad@nuclight.lan> <202409100638.48A6cor2090591@critter.freebsd.dk> <20240910144557.4d95052a@nuclight.lan> <4D84AF55-51C7-4C2B-94F7-D486A29E8821@FreeBSD.org> <20240910164447.30039291@nuclight.lan> <3F3533E4-6059-4B4F-825F-6995745FDE35@FreeBSD.org> <20240911011228.161f94db@nuclight.lan> <20240911024735.37522532@nuclight.lan> X-Mailer: Claws Mail 3.19.1 (GTK+ 2.24.33; amd64-portbld-freebsd12.4) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Bar: --- X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FREEMAIL_TO(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FREEMAIL_FROM(0.00)[gmail.com]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::233:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MLMMJ_DEST(0.00)[freebsd-arch@freebsd.org,freebsd-hackers@freebsd.org,freebsd-net@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; TAGGED_RCPT(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCPT_COUNT_FIVE(0.00)[5] X-Rspamd-Queue-Id: 4X3ysm1GqYz4M7X On Tue, 10 Sep 2024 16:55:15 -0800 Rob Wing wrote: > On Tuesday, September 10, 2024, Vadim Goncharov > wrote: > > > > > > What's the point to waste resources on writing thing that is known > > to be not accepted to base system from the very beginning? > > > what's the point of talking about thing you're never going to > write/finish even if it was accepted? Because - and I've written that from the beginning - wrong architecture will cost too much if your code already exists, than if fixed before that. So I've received worthy feedback describing problems; not that were good solutions yet, though... > Or even bikesheds are in short supply now? > > have you considered calling it eBPF++ or BPF128? Well, first it was called fBPF as next letter from eBPF, and for 128 it lacks 128 bit arithmetics. Then I realized that for some things there will be different implementations in different kernels, e.g. for atomic(9) operations, so better to call generic common thing neutral BPF64 and e.g. fBPF be FreeBSD dialect, nBPF for NetBSD dialect... > at any rate, don't claim the sky is falling on the grounds that I > think your proposal is far fetched Not you. FreeBSD already had precedent when already written and committed code framework for device sensors was removed: https://lists.freebsd.org/pipermail/cvs-src/2007-October/082398.html and that happened by exactly the same person with similar tone (not you). ...well, ok, there were some really technical arguments in that case e.g. about sysctl_add_oid() but that was the most technical of them, all other being same arrogant rant. Anyway, such cases are very demotivating from contributing. > if you think you've got a great idea and have a need for it, then > write it and use it - if other people adopt it, even better > > all the best to your endeavors and don't let the naysayers hold you > back Well, what I really need is a technical help for areas I don't know, as obviously I won't be able to write entire ecosystem alone. For example, I don't know how many registers are available to a function in a kernel on an ARM, MIPS and RISC-V. If I allocate too much, then somebody will have trouble implementing JIT on such machine, as I don't have such hardware. And e.g. https://en.wikipedia.org/wiki/MIPS_architecture says 2 registers are for kernel - OK, and what if we are kernel itself? :-) Do we need Thread Pointer of Global Pointer, or we can stash them to stack? Etc. -- WBR, @nuclight From nobody Thu Sep 12 06:33:52 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X48DV2HfSz5V7D1 for ; Thu, 12 Sep 2024 07:28:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X48DT6QVnz4TXF for ; Thu, 12 Sep 2024 07:28:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1726126089; a=rsa-sha256; cv=none; b=tEnCC7l6TbN3rKvkxrdjetdieOIikEblOKFIdD+QkAu7lAmiOjcAqoiWz8qbndZx0OYPhm AAqZ+EUoYNX8fwqpQgw4b8BhT1wzfS6hRozIK8SWVaLFdQrK3DCyYSrRW/jhsg8DIrdL+p sz2Tq0rsZunYPjNTO103N/ILX7FNDM052Uf7dNUjNjM1Ubsbsp0LK9LE1koEnJz9zYNokv SikvfxkDc6pAKsGwUih+D/ylU806Lh5BFw9qGJ0u1JDsZkVHdEoaUn4D3x6p8vd6N+rj8S 6gMDv8B6Zv+JrhLhs7osKvP4XOzbaiUsTYyBSqU1lzTOSCfMNW63aU2PVNH4wQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1726126089; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+1URROnKdCAelIQyhNK7xizyqdlzwVeJ3hzZFy8AF10=; b=K5RIA4HO49QqgXTR/alRtN6xE06D6evtqT3m7tQ1vqxorEbTasV9BLmTHTpVSamysBg/6K z8WsV1vp1ScDRfUrvhxs2zKQ/RWQlOtzeHaqxPJP20bJLNArbIIuI4cLVa+CDnm3lW3wN6 xIIGM3EkzBqFnPPDR96ZT19+PEAvGLjGku7M8Rr1oD0KkEg463HeUbi+SmEQFWloqMHwAe FCvL2UEGgX6S1A7OpwT73b4Q0z1JozStC7MMehWWB3yVmGg9ToP23eaZrurWXx0D9EoYco IvPEHfGT7T6vmwlqTJau9xyAu4JCxT76yIr5z8oluR0TimOiby71uFSzDFY2yg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4X48DT5Hkfz15Fs for ; Thu, 12 Sep 2024 07:28:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 48C7S8TH082319 for ; Thu, 12 Sep 2024 07:28:09 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 48C6XwW2069809 for net@FreeBSD.org; Thu, 12 Sep 2024 06:33:58 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 272416] Seldom crash happening with RTL8125 Date: Thu, 12 Sep 2024 06:33:52 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.2-RELEASE X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: kbmithunchakravarthy@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D272416 Mithun changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kbmithunchakravarthy@gmail. | |com --- Comment #10 from Mithun --- (In reply to Jonathan Vasquez from comment #9) Hi Jonathan Vasquez, I=E2=80=99m experiencing a similar problem with the if_re module crashing o= n my FreeBSD system when I load the module. I saw your comment and bug report about the Realtek onboard NIC issue. Would it be possible for you to share the patch = you mentioned? I would greatly appreciate any guidance on how to apply and test= it. Thanks for your help! Best regards, Mithun --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Sep 12 11:04:35 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X4F2S2Z0Jz5WBpH; Thu, 12 Sep 2024 11:04:48 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X4F2S22NSz4vpr; Thu, 12 Sep 2024 11:04:48 +0000 (UTC) (envelope-from theraven@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1726139088; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=5XCP+OtTCjv0BrnUPkhgNamjxRyHSjEizsK/4AyPSBw=; b=ULGygp69dhp5iepNCNIxJwvU8JhuN2XttxgIcksnENWn/EkpWlZMBN1/bsv2rLJTwHnLPf 5MImRTxEq5If68X23Td4Ou+Tft+r3y4Xx28CqvaZmYQQ9FMGDRKMjOdDUlKflh1PAobTle cLViEG004vf50NwrD1E7BA5cIEE9z5YDp9Ht/y9b+fjc1RC2Ncwi9rcRR6toD3wkJZarwL 3grzszg0kMJs2HNJycX5ftumlBwT+KeTRlfdOgYaC0lhp/b3/X86iETKGnSuKzFO9oJvxg IBgHq9vkJ3IHOROxGGQQmhe6rKU3Yv9pDuZ/N4r5FTT9O32j6/XMYR6uujvqQA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1726139088; a=rsa-sha256; cv=none; b=wxPxwDj8dWbkf2i7is2g5+KNMe+VnjkUC1Zuy28Y8p1vZH+eUoM43beUdzU9wAGTavdA3a LaSn0+8q9rYBoHa/6uoFfZJyZW3B0f0Wh7fr1/hXWuAUpM5+YyXlE8Z8NqNS8Xaekp/9LS An2P7ZA2UUnhcZCS0sAHLVmcLuYwcuutD3/Xa1p3ynRBMvYoBntkjqWmOESESk/sKhRjLy LMZYeOeGo//hqCcldzRfTYZBF+O3bll+mpfhVlIRY8s+u/9Jm6l8p6jczAhgXPOA3P4W5k Cz5ykBolBFp6rE/rZuXKZ+4YvkjcuTmdOuUjwcKB/Y34W2hxhs0f4GRN7FKmkA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1726139088; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=5XCP+OtTCjv0BrnUPkhgNamjxRyHSjEizsK/4AyPSBw=; b=Q5F34EXluziTC0xEWXA/U7vv5rQteg2SuSGd/bAu9eC0RWEQImbmQIvw8YSk3YMA7bV8uG 4/1W3Rqxjj4+7edt26NTqWFJlThUXjjNL+Bi9hBp1gJvQIdOyP23F9x9B0AOxHp3gusPkh iZPdXa+dIvaoOJyHF0PTCvvTbEE2cwsQ0Fw37Lp5ZVi40my7cvEIYrnaLm1IVHv4nfM3gk ULPUhz9GQBFaMxn+fSWGdzbIagta91PkI3NRZoiZF0Vkq/wODgbj3IOrc6uS7nc0hY915P Mh7LF7d/YeKlYvsxlC2woNOIKEiqqUn03nHgNnC6vjvUBytj0GSi7ebwg/uo1g== Received: from smtp.theravensnest.org (smtp.theravensnest.org [45.77.103.195]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: theraven) by smtp.freebsd.org (Postfix) with ESMTPSA id 4X4F2S1RrKzPkW; Thu, 12 Sep 2024 11:04:48 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from smtpclient.apple (host109-155-136-107.range109-155.btcentralplus.com [109.155.136.107]) by smtp.theravensnest.org (Postfix) with ESMTPSA id E3E7A6603; Thu, 12 Sep 2024 12:04:46 +0100 (BST) From: David Chisnall Message-Id: <746547C3-DF15-435D-AECE-9B2D195703B5@FreeBSD.org> Content-Type: multipart/alternative; boundary="Apple-Mail=_10FE2401-9F6E-4915-9D75-86D8B0F37D7C" List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3776.700.51\)) Subject: Re: BPF64: proposal of platform-independent hardware-friendly backwards-compatible eBPF alternative Date: Thu, 12 Sep 2024 12:04:35 +0100 In-Reply-To: <20240912025717.455295f1@nuclight.lan> Cc: Philip Paeps , freebsd-arch@freebsd.org, freebsd-hackers@freebsd.org, freebsd-net@freebsd.org, tech-net@netbsd.org To: Vadim Goncharov References: <20240911120518.1ba191b5@nuclight.lan> <20240912025717.455295f1@nuclight.lan> X-Mailer: Apple Mail (2.3776.700.51) --Apple-Mail=_10FE2401-9F6E-4915-9D75-86D8B0F37D7C Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 On 12 Sep 2024, at 00:57, Vadim Goncharov = wrote: >=20 > This is just not true. See, for example, FreeBSD-SA-17:06.openssh for > vulnerability disabled by default, and workaround proposed to return > to default (disabled) state. No, this was changing from one supported expected-secure setting to = another. If you make a device directly usable by non-root users, you = are expected to understand the security implications. > Stop. Just stop, and re-read carefully. I have read carefully. I am not sure at this point whether you are = intentionally failing to engage in good faith or if you simply do not = understand the security landscape that you are operating in and have no = interest in learning. Either way, it is not productive to keep having = this conversation so this will be my last comment in this thread. > You (and perhaps Philip) > confusing two things: BPF and eBPF (and BPF64 third), all completely > different beasts. They are all mechanisms for running semi-trusted / untrusted code in the = kernel. > Last two letters in this thread, I was talking about > classic BPF existing in *BSD for decades (on FreeBSD allowed to have > permissions on dev/bpf*). FreeBSD allows you to change the permissions on anything in /dev/. It = is up to you to understand the security implications of doing this. = Allowing non-root access to /dev/bpf has security implications. I do it = for my user on a couple of single-user FreeBSD dev boxes, because I also = have root on these systems and so anything WireShark can do on my = behalf, I can also do via su. There is no security issue because my = threat model *for this system* is such that I can accept a weaker = posture than the default. There are legitimate reasons for broadening = the permissions on these systems. > So you assert that THIS classic BPF also > vulnerable to aforementioned attacks, and thus SA must be issued, just > like that FreeBSD-SA-17:06.openssh, with a fix (at least preventing > changing default permissions) of hole existing for *decades*. No, for the same reason that there=E2=80=99s no security advisory if you = `chmod +r /dev/mem`. There=E2=80=99s a reason that both /dev/mem and = /dev/bpf are restricted to root by default. Anyone who relaxes these = permissions must understand what they=E2=80=99re doing and have a threat = model that can justify why it=E2=80=99s acceptable. By analogy with FreeBSD-SA-17:06.openssh: this SA applied where password = login was enabled. Enabling password authentication weakens the = security of OpenSSH and so is not done by default, but that was not the = problem that merited the SA. The SA was issued because it had the = additional effect of allowing remote attackers to mount a denial of = service attack. We would not issue an OpenSSH SA saying =E2=80=98enabling= password authentication weakens security because people can log in with = passwords, which are less secure than SSH keys=E2=80=99. The = expectation is that anyone changing this setting knows what they=E2=80=99r= e doing. If it is not a problem for their use case, they can do it. = Precisely the same logic applies to allowing non-root access to /dev/bpf = or /dev/mem. > This is > too strong assertion to be accepted without proofs, and as I can = deduce > from your other words and readings about Spectre (see below), this > statement is not true (classic /dev/bpf is not vulnerable). I have not built a PoC, but I would fully expect that it=E2=80=99s = possible to build an attack that first primes the cache and trains the = branch predictor and then runs a crafted BPF program that has an = out-of-bounds read (which executes only in speculation) to leak kernel = memory (including contents of the direct map, so anything owned by = another process on the same system), and then inspects the contents of = the cache to see the value that was observed in speculation. All of the = necessary primitives are there. If you are designing a system that expects non-root users to be able to = run code in the kernel, the onus is on you to explain why it is safe. = The default assumption must be that it is unsafe. David --Apple-Mail=_10FE2401-9F6E-4915-9D75-86D8B0F37D7C Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 On 12 Sep = 2024, at 00:57, Vadim Goncharov <vadimnuclight@gmail.com> = wrote:

This is just not true. = See, for example, FreeBSD-SA-17:06.openssh for
vulnerability disabled by default, and workaround proposed = to return
to = default (disabled) state.

No, this = was changing from one supported expected-secure setting to another. =  If you make a device directly usable by non-root users, you are = expected to understand the security implications.

Stop. = Just stop, and re-read carefully. =

I have read carefully. =  I am not sure at this point whether you are intentionally failing = to engage in good faith or if you simply do not understand the security = landscape that you are operating in and have no interest in learning. =  Either way, it is not productive to keep having this conversation = so this will be my last comment in this thread.

You = (and perhaps Philip)
confusing two things: BPF and eBPF (and BPF64 third), all = completely
different = beasts.

They are all = mechanisms for running semi-trusted / untrusted code in the = kernel.

Last two letters in = this thread, I was talking about
classic = BPF existing in *BSD for decades (on FreeBSD allowed to have
permissions on dev/bpf*). =

FreeBSD allows you to = change the permissions on anything in /dev/.  It is up to you to = understand the security implications of doing this.  Allowing = non-root access to /dev/bpf has security implications.  I do it for = my user on a couple of single-user FreeBSD dev boxes, because I also = have root on these systems and so anything WireShark can do on my = behalf, I can also do via su.  There is no security issue because = my threat model *for this system* is such that I can accept a weaker = posture than the default.  There are legitimate reasons for = broadening the permissions on these = systems.

So you assert that THIS = classic BPF also
vulnerable to aforementioned attacks, and thus SA must be = issued, just
like = that FreeBSD-SA-17:06.openssh, with a fix (at least preventing
changing default permissions) of hole existing for = *decades*.

No, for the same = reason that there=E2=80=99s no security advisory if you `chmod +r = /dev/mem`.  There=E2=80=99s a reason that both /dev/mem and = /dev/bpf are restricted to root by default.  Anyone who relaxes = these permissions must understand what they=E2=80=99re doing and have a = threat model that can justify why it=E2=80=99s = acceptable.

By analogy with = FreeBSD-SA-17:06.openssh: this SA applied where password login was = enabled.  Enabling password authentication weakens the security of = OpenSSH and so is not done by default, but that was not the problem that = merited the SA.  The SA was issued because it had the additional = effect of allowing remote attackers to mount a denial of service attack. =  We would not issue an OpenSSH SA saying =E2=80=98enabling password = authentication weakens security because people can log in with = passwords, which are less secure than SSH keys=E2=80=99.  The = expectation is that anyone changing this setting knows what they=E2=80=99r= e doing.  If it is not a problem for their use case, they can do = it.  Precisely the same logic applies to allowing non-root access = to /dev/bpf or /dev/mem.

This is
too = strong assertion to be accepted without proofs, and as I can = deduce
from = your other words and readings about Spectre (see below), this
statement is not true (classic /dev/bpf is not = vulnerable).

I = have not built a PoC, but I would fully expect that it=E2=80=99s = possible to build an attack that first primes the cache and trains the = branch predictor and then runs a crafted BPF program that has an = out-of-bounds read (which executes only in speculation) to leak kernel = memory (including contents of the direct map, so anything owned by = another process on the same system), and then inspects the contents of = the cache to see the value that was observed in speculation.  All = of the necessary primitives are there.

If you = are designing a system that expects non-root users to be able to run = code in the kernel, the onus is on you to explain why it is safe. =  The default assumption must be that it is = unsafe.

David


=


= --Apple-Mail=_10FE2401-9F6E-4915-9D75-86D8B0F37D7C-- From nobody Thu Sep 12 11:53:33 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X4G6k2szcz5WKtt for ; Thu, 12 Sep 2024 11:53:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X4G6k0tj3z53nv for ; Thu, 12 Sep 2024 11:53:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1726142014; a=rsa-sha256; cv=none; b=esa1bjEkljahGeEiUel4qAGabVVsrBdkEZMKnuaoyWdlholYY3jRjjSGFWou48Svxu0IaV awwA9V0DmHOWXJKn2HRVs15cCyysFWw3z4rr8lFTz9MO1HwOzZU67/yQdaIpZi3Yqzws8/ R2LgN0k6Q48JJeoVYK/6asw+Yn62y+VaB69XdebEuF/xtReZpQhoxUrdB6oJnmNvzm4ITZ zHNeRGnebHdthbJjjH9pxoxG1/OqAN9LEewyeo/xOzpc8YE2mcbNNbDSZbfTCJbXy6VWxF aIVp6Qj3Lnsdu4al1NiyskpDL3CRVGWkQVhpD/za7wQYXxtXvfM0Er4fLEGoRA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1726142014; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DPXt4rmmuC4jvzKS9y7y/cOX8apeLi0gUiPI4Fca2KI=; b=vZFQDarP/KvokqgJcSMY19AtF0F8YT/bRfOle7zR7QmTp88+UNHbD/IzwxsrMteul0Z4O2 fmvURcmHt/inJNxLTg6/c91qN90laOlwyeWGSKkYiiu/3+iV8Mkoq+sr60sZb+eXDL+OED tSP68UAG076cJKxeXS7EsGvqKspqv9g2ondSr3NQFcaCQ+ixpxIZEJpNxbaNRJfCP1btT1 GAa9dxvl9Z8pb3uImfGCeKV/ie0z3J/FhIKyM2GvoTSMB15484udMK6k6OhT4hnaTJLZky ybJ3AiMDhP3oobuzC/WBMvUoR7A6enwpLXjRqYai52Q2qiSJ0QCm60vxKK4Y4g== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4X4G6k0NFJz1DKG for ; Thu, 12 Sep 2024 11:53:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 48CBrX7f065555 for ; Thu, 12 Sep 2024 11:53:33 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 48CBrXdo065554 for net@FreeBSD.org; Thu, 12 Sep 2024 11:53:33 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 280036] Data corruption over if_ovpn (OpenVPN DCO) observed Date: Thu, 12 Sep 2024 11:53:33 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: commit-hook@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D280036 --- Comment #17 from commit-hook@FreeBSD.org --- A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3D8f251937850142748cdf67a4663034293= 4ff9f91 commit 8f251937850142748cdf67a46630342934ff9f91 Author: Kristof Provost AuthorDate: 2024-09-04 12:54:23 +0000 Commit: Kristof Provost CommitDate: 2024-09-12 07:57:04 +0000 if_ovpn: ensure it's safe to modify the mbuf PR: 280036 Reviewed by: ae MFC after: 1 week Sponsored by: Rubicon Communications, LLC ("Netgate") Differential Revision: https://reviews.freebsd.org/D46529 (cherry picked from commit 5644e2c6d47c6113a61ab7fc0776b7227677656a) sys/net/if_ovpn.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Sep 12 12:03:50 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X4GLb4M5Nz5WMGJ for ; Thu, 12 Sep 2024 12:03:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X4GLb1fhNz54fx for ; Thu, 12 Sep 2024 12:03:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1726142631; a=rsa-sha256; cv=none; b=VPbb+oM4q8lLxzyhEv/4wIUmbmBTJ6YwpekPWbD0VzexK4UZenGAfcosSCmy1kpBWx3dzO QJyEkITP7VzTbIY4bb0clRmtZmA9+Kuh4OLXxbuMmdbwPM0vZ91E1UvtS29EqOL/1+Lrp9 FgGdi/JLGFiluSZUtDeizB1J/9/+9G0eHKl1VtYmCEJgYlhzwuZBHJ4/PP+upTA6d9ErDs xVfWLhDIba9yTeJHOvG0k62a/dRPsNHyKk4Jd2waQLiAhDRo7qJxFHCCk0aqxkArKAfyxZ 0IeIGNn35ayxNbC9cV49Q5LaZ9PXmJhCTTFrNhxb/xCMUWGfOdoC7VAaDnnyYA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1726142631; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OIiJ83jOwf9kwmucl1dWExtGLHW1CsFNMojXfH9xDNs=; b=wRe5BlH34ZikaHG0NO9eataDB9BiLHC6Km1q+Zl0/ZdZE5b+N97PLDLJ+ZhOThF8PLQukH lOIVRAiCExwJ/PD0fmQIp2PdsbPJDpuK/+/leH9dW8vK+y2kt8PVVbAMe+Xse6Q9hsjIN3 M/2gX2jbkFZP+Zb5A1FnuozHh64JtvNc7VXeNXpYbJCmyW99HxNuVlK2MIh5NTy//4+iym PAzNxbBUBf8YtJoZEJcFKdr99a6bHIt2EJzWaLg0j2KFGDNaRcFMy6F4YLMUlNZ4W3ELRr vfavK0+Embmn9INCKzDI70E0bQRsWkQLR6PNap4Wi2VFUb91B4Z39V1rbwiwQw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4X4GLb18v9z1DYy for ; Thu, 12 Sep 2024 12:03:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 48CC3pM3082237 for ; Thu, 12 Sep 2024 12:03:51 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 48CC3pOX082236 for net@FreeBSD.org; Thu, 12 Sep 2024 12:03:51 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 280036] Data corruption over if_ovpn (OpenVPN DCO) observed Date: Thu, 12 Sep 2024 12:03:50 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: kp@freebsd.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status resolution Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D280036 Kristof Provost changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |Closed Resolution|--- |FIXED --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Sep 12 13:20:05 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X4J2Z1Zzqz5WWvr for ; Thu, 12 Sep 2024 13:20:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X4J2Z0YWzz40rC for ; Thu, 12 Sep 2024 13:20:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1726147206; a=rsa-sha256; cv=none; b=njfUn5xpipK33ddO480DxVphkSflhQGx9yRbtcpAMmnZSImG2oAZvKKuZolowMHu3muo6J ZIyaXoWe90arPLCPRSLE6+pM8XuBumWiJJQfAQtUm3NiI0r5X2W+NPUuyk+ovXdyNp3msl l54yDGeKoMkcAhfFou2XEKAdmoSWMOlpEaMditB8MExhEwg4bMJhvkXjuIESk4LAKyEKZC YQzp/pmvAYhDQLcSC3iETtn6H+MncO43LnocNih93gYWFbUWyMTqAoqXx9vmaibImiRPT1 EQJzME8+M2wRie9gDuUuymK+bQKQw+/MgqOM56eP/ZWnfpfFvS7TgyBRgoiIBg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1726147206; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6M0OA0N4nOeXTCD4kAnLnPA60CW7ogbiZfHWV/Tv7Hw=; b=FCJu7BKTocxf9LqA19NW8y0gKrdM6KlHnRmcB4gowDfhUrfVQxoSV00IW3C6nn0Fp/hdlx oLUq4xneCgFL4yQXA2/BT68ebCtrl8LNv0dJMBP8Gdyie8LA3kLaf3d8VhAXWaXp50kUKG YMyBUT74JxAyTPC8glEDoCX09icOzUgf6cEoDwV4qXaPi9yiQaI0lHXtHn8PAMe8jUdjpG X+nSXm0wfatMCpErkHif37dvMglw9/xx1H0rGy2pOQHtLZChbqb8+nhew6gfKvLvhiQJSn ffkMelLBpKbTl7IWK8PiD/teEBC5qG0X0jbnhe5GOeCXQZl+TlMrZuRFPXGceQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4X4J2Z09CCz1FBq for ; Thu, 12 Sep 2024 13:20:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 48CDK5eM050187 for ; Thu, 12 Sep 2024 13:20:05 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 48CDK5Nl050186 for net@FreeBSD.org; Thu, 12 Sep 2024 13:20:05 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 272416] Seldom crash happening with RTL8125 Date: Thu, 12 Sep 2024 13:20:05 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.2-RELEASE X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: jon@xyinn.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D272416 --- Comment #11 from Jonathan Vasquez --- Hey Mithun, I don't have a patch for this. I installed an Intel NIC on a spare PCIe slot that I have and it's been rock solid and fast now. So I'm no longer using Realtek for my networking. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Sep 12 17:16:18 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X4PHB2sZjz5WHgY for ; Thu, 12 Sep 2024 17:16:22 +0000 (UTC) (envelope-from cryintothebluesky@gmail.com) Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X4PH90cl7z4TVy for ; Thu, 12 Sep 2024 17:16:21 +0000 (UTC) (envelope-from cryintothebluesky@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=H3Q5mTu4; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of cryintothebluesky@gmail.com designates 2a00:1450:4864:20::32b as permitted sender) smtp.mailfrom=cryintothebluesky@gmail.com Received: by mail-wm1-x32b.google.com with SMTP id 5b1f17b1804b1-42cb57f8b41so93205e9.0 for ; Thu, 12 Sep 2024 10:16:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726161380; x=1726766180; darn=freebsd.org; h=content-transfer-encoding:mime-version:message-id:subject:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=dfh9wOSY+X9YfzrGq3lFAuWQ3d+Jm2KgHRWMKkW6QmA=; b=H3Q5mTu4IrqoRR9eOpowyoJ/5r+0gxnHVKdqS2YlbkGmLRdde+k+FGJe0hmtIYCvHT gs5qNVlGd7E4VzgnexYw7fdDIzj8Vj9tdXDAnbtzAVtw1Xp/8RM+8xvukyU7SgW1OZBy B5CpV0NOZdBQx1/l1UIurthcFa0VoobaqzedkBLz0qnqo/cKmf6xNYe6oMaS1rDss9Vw pRxlgfHDwBDXN/C8x1T8sqNKwjVOOdWa4wwlxPSw2j0blTSWdLCRCFQvAprGIZtooBcE ygbp0O7K4jcdpuabemswPd+yawIH3RAV0DsREt3H+if39GIYpPaCwaZgW2nRoa48BEBj GtUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726161380; x=1726766180; h=content-transfer-encoding:mime-version:message-id:subject:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=dfh9wOSY+X9YfzrGq3lFAuWQ3d+Jm2KgHRWMKkW6QmA=; b=vJkEdk7Fyb0oMP2miidETYiFcXX4X1BFFDUW6MnPAEn9eKq3vm419RpdUq4Wy0R6XY aCWoF0z/OOToSPYd7UcIAfnA16ivkt0y3XZCnEdUBw1Yjj0r8SvmrJvrhWB1T24ZLL9M pU/mkEIwLpSPiwmr8s8fvxOheGqdk0o5HmPTtPy32N9kK13VgG1ORvJm6tWBxZMtWRVO vCAErpzG5o/MJn4wUJrfndFmjlsIuNEYdNAFuZXWNCEJYzPj4FALU5OKS+458Q2Jf+M4 +tBUjGmoXqGneplb9eVV2ytfzSWiaDUvgJr9+HhhX6hrs3uFR7+Bbegcm8Unr/shzyCS rgTQ== X-Gm-Message-State: AOJu0YwnJEV1KwFHw2GC5w/JLwnfSaHC0dfEIIrNW+vw6H+Vvf9AJcsz W3QNBBJ3jR+DcEIZpT+TN5bFTBrvV0mn79POSk3+znrFIF+SIOk0aOkftA== X-Google-Smtp-Source: AGHT+IEGnE47OefgnNNqiwRHNOngcVOap7AeXy0XpYGqeZYI2XaumdfAmAZpvRMkKydH4VtqE/Jrgw== X-Received: by 2002:a05:600c:1c12:b0:42c:b5a6:69bd with SMTP id 5b1f17b1804b1-42d964e0027mr1227745e9.30.1726161379489; Thu, 12 Sep 2024 10:16:19 -0700 (PDT) Received: from z600.home.lan (118.129.159.143.dyn.plus.net. [143.159.129.118]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-42caeb8afc9sm181077415e9.44.2024.09.12.10.16.18 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Sep 2024 10:16:18 -0700 (PDT) Date: Thu, 12 Sep 2024 18:16:18 +0100 From: Sad Clouds To: freebsd-net@FreeBSD.org Subject: Performance issues with vnet jails + epair + bridge Message-Id: <20240912181618.7895d10ad5ff2ebae9883192@gmail.com> X-Mailer: Sylpheed 3.8.0beta1 (GTK+ 2.24.33; x86_64-pc-linux-gnu) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.49 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.995]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_FROM(0.00)[gmail.com]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::32b:from] X-Rspamd-Queue-Id: 4X4PH90cl7z4TVy Hi, I'm using FreeBSD-14.1 and on this particular system I only have a single physical network interface, so I followed instructions for networking vnet jails via epair and bridge, e.g. devel { vnet; vnet.interface = "e0b_devel"; exec.prestart += "/jails/jib addm devel genet0"; exec.poststop += "/jails/jib destroy devel"; } The issue is bulk TCP performance throughput between this jail and the host is quite poor, with one CPU spinning 100% in kernel and others sitting mostly idle. It seems there is some lock contention somewhere, but I'm not sure if this is around vnet, epair or bridge subsystems. Are there other alternatives for vnet jails? Can anyone recommend specific deployment scenarios? I've seen references to netgraph which could be used with jails. Does it have better performance and scalability and could replace epair and bridge combination? Thanks. From nobody Thu Sep 12 17:25:32 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X4PV14Lw1z5WKKY for ; Thu, 12 Sep 2024 17:25:45 +0000 (UTC) (envelope-from pprocacci@gmail.com) Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X4PV12k4Rz4VYF for ; Thu, 12 Sep 2024 17:25:45 +0000 (UTC) (envelope-from pprocacci@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x636.google.com with SMTP id a640c23a62f3a-a7a843bef98so155338966b.2 for ; Thu, 12 Sep 2024 10:25:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726161944; x=1726766744; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=JFG+z53YCIxCIvkf/zwz+x9EQipLaVqBcDVBO1vi2rM=; b=ZL+qyTTMKaIf0/gVX4eaqAWI6X+x9hBLdeFXn3ejkbtliIuSK7EZHieAaCQUcPTP7B /ldDhDYKvqZKw4XwdG2r9scutYZYorhncLEN8Ff+H/la7TQjvNplqwg7DBmQFuVQniaL qpBe1UoYuFBntXgDPpTNxoo5kaF5UjFeOiYGR1mIIamOvM+DRaD1uieqVA/4DNPtF/ma bpCoPJyJEwmytTqNVPW/oopkfZVvGBxQ1qhQSdMaYFwhkxXj4/ZxLW8x7uiT3ubnVcsx QNIPa1mlZ69R3vblJlxI3m0tDKBQaHMq6i/zATrUHOD5GsBsgtG4dOlUuX8f/q3T8DZA J8yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726161944; x=1726766744; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=JFG+z53YCIxCIvkf/zwz+x9EQipLaVqBcDVBO1vi2rM=; b=jq+Vvd73/V3rX73vfduVFes7QajWrpMO9kIZzDsNAi9AfvTQrfOozUv8OObVo9Vx8B UYtkb3a4qvZTSHXwjpPRstNU7lxuDHbHJTM+gKlTe5tsUh25dGfF6WkMAKqdYP79QGor 9RfUs7D76F4tttfj7fbit1/7VZ+o6mzwu84RKCUKrZ/jzGMzj9h2ygWgVfD088bS/QhW bdvO7AX2mJRq5OM6DJc9OJ71CSFOHe4d06Xf9b1GedI3LdWpwRMP6qkSrHt+DPAAFIoQ r6RbirlLPUgeA0DXY/4bvTp/XnPBIHO0P5Qp0a3JwPVJfZL+gZNr//E5unfmGYYNI/FS 1NDw== X-Gm-Message-State: AOJu0Yy673/EbuN+DH4tOsTA2Fyu6in+B3GKAQaV1tWkrT+7fVW25tD1 hrRzGtKdXR5HdfQOVJiq+peTWvzdxJBuuaI89AkhRukEO13xw7Z+zg0KiMjruz3qpgrVTNWG2rh Z9v2CE3Aqj5f+/gaKUqxlmGNUTw== X-Google-Smtp-Source: AGHT+IFMis4d78birh5oc8xNAGocDSKqOAu2o3wC+sibPggjsoMDbblW+X6Xj3N4/+81kiqVg47KgSFXabgG2ylf22E= X-Received: by 2002:a17:906:c14a:b0:a86:7514:e649 with SMTP id a640c23a62f3a-a902960cff6mr358745966b.52.1726161943109; Thu, 12 Sep 2024 10:25:43 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 References: <20240912181618.7895d10ad5ff2ebae9883192@gmail.com> In-Reply-To: <20240912181618.7895d10ad5ff2ebae9883192@gmail.com> From: Paul Procacci Date: Thu, 12 Sep 2024 13:25:32 -0400 Message-ID: Subject: Re: Performance issues with vnet jails + epair + bridge To: Sad Clouds Cc: freebsd-net@freebsd.org Content-Type: multipart/alternative; boundary="00000000000071abcf0621ef652e" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4X4PV12k4Rz4VYF --00000000000071abcf0621ef652e Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Sep 12, 2024 at 1:16=E2=80=AFPM Sad Clouds wrote: > Hi, I'm using FreeBSD-14.1 and on this particular system I only have a > single physical network interface, so I followed instructions for > networking vnet jails via epair and bridge, e.g. > > devel > { > vnet; > vnet.interface =3D "e0b_devel"; > exec.prestart +=3D "/jails/jib addm devel genet0"; > exec.poststop +=3D "/jails/jib destroy devel"; > } > > The issue is bulk TCP performance throughput between this jail and the > host is quite poor, with one CPU spinning 100% in kernel and others > sitting mostly idle. > > It seems there is some lock contention somewhere, but I'm not sure if > this is around vnet, epair or bridge subsystems. Are there > other alternatives for vnet jails? Can anyone recommend specific > deployment scenarios? I've seen references to netgraph which could be > used with jails. Does it have better performance and scalability and > could replace epair and bridge combination? > > Thanks. > > You need to define `poor'. You need to show `top -SH` while the `problem' occurs. My guess is packets are getting shuttled between a global taskqueue thread. This is the default, or at least I'm not aware of this default being changed. You can try enabling `options RSS' in your kernel as this would introduce a taskqueue worker thread per cpu. ~Paul --=20 __________________ :(){ :|:& };: --00000000000071abcf0621ef652e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Thu, Sep 12, 2024 at 1:1= 6=E2=80=AFPM Sad Clouds <= cryintothebluesky@gmail.com> wrote:
Hi, I'm using FreeBSD-14.1 and on this parti= cular system I only have a
single physical network interface, so I followed instructions for
networking vnet jails via epair and bridge, e.g.

devel
{
=C2=A0 =C2=A0 =C2=A0 =C2=A0 vnet;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 vnet.interface =3D "e0b_devel";
=C2=A0 =C2=A0 =C2=A0 =C2=A0 exec.prestart +=3D "/jails/jib addm devel = genet0";
=C2=A0 =C2=A0 =C2=A0 =C2=A0 exec.poststop +=3D "/jails/jib destroy dev= el";
}

The issue is bulk TCP performance throughput between this jail and the
host is quite poor, with one CPU spinning 100% in kernel and others
sitting mostly idle.

It seems there is some lock contention somewhere, but I'm not sure if this is around vnet, epair or bridge subsystems. Are there
other alternatives for vnet jails? Can anyone recommend specific
deployment scenarios? I've seen references to netgraph which could be used with jails. Does it have better performance and scalability and
could replace epair and bridge combination?

Thanks.



You need to define `poor'.
= You need to show `top -SH` while the `problem' occurs.

My guess is packets are getting shuttled between a global tas= kqueue thread.
This is the default, or at least I'm not aware = of this default being changed.
You can try enabling `options R= SS' in your kernel as this would introduce a taskqueue worker thread pe= r cpu.

~Paul

--
__________________

:(){ :|:& };:
<= /div>
--00000000000071abcf0621ef652e-- From nobody Thu Sep 12 18:00:14 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X4QG04xnPz5WPHW for ; Thu, 12 Sep 2024 18:00:24 +0000 (UTC) (envelope-from SRS0=L5lp=QK=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4X4QG02lL9z4Z4y for ; Thu, 12 Sep 2024 18:00:24 +0000 (UTC) (envelope-from SRS0=L5lp=QK=quip.cz=000.fbsd@elsa.codelab.cz) Authentication-Results: mx1.freebsd.org; none Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id ED39BD78A8; Thu, 12 Sep 2024 20:00:15 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quip.cz; s=private; t=1726164015; bh=jwsugpb7Xg9kP3xIdNIdSoPUYlzkgWYROxYMuaUdtXQ=; h=Date:Subject:To:References:From:In-Reply-To; b=HelzxkWprzbahL06Vg/me3l/0SAS8/36FF5CGKlh9ZBy0NW0BpYqWmNjz1n6Cwxws nZ+RrqhsF3dB6thwPIKDdReyxYKeHHayKrJ+2/EX6a6bq7Vc4dLHShYLP49e7reG+z LjNR7kBP+NVjlWW9ns3lBIW8vUeUdA4KvVf/NqvU= Received: from [192.168.145.49] (ip-89-177-27-225.bb.vodafone.cz [89.177.27.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 802FBD78AA; Thu, 12 Sep 2024 20:00:14 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quip.cz; s=private; t=1726164014; bh=jwsugpb7Xg9kP3xIdNIdSoPUYlzkgWYROxYMuaUdtXQ=; h=Date:Subject:To:References:From:In-Reply-To; b=i80cLI44cWNhK1GgsDZz7TD8uPL5B17VLKKhP1K3vZ/e/JZL9tOHxWZBAnQ40Z2di Oxp5KD09NK3iSH8Dln/N40BOnW52XPqtgEBTiRzD3CofDtfwl0zgqi8kc+RqEz+MB5 Nvm1/NJ0hW7FVyXRrVIevigpm8luRl8+efASf6sU= Message-ID: Date: Thu, 12 Sep 2024 20:00:14 +0200 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Performance issues with vnet jails + epair + bridge To: Sad Clouds , freebsd-net@FreeBSD.org References: <20240912181618.7895d10ad5ff2ebae9883192@gmail.com> Content-Language: en-US From: Miroslav Lachman <000.fbsd@quip.cz> In-Reply-To: <20240912181618.7895d10ad5ff2ebae9883192@gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ] X-Rspamd-Queue-Id: 4X4QG02lL9z4Z4y On 12/09/2024 19:16, Sad Clouds wrote: > Hi, I'm using FreeBSD-14.1 and on this particular system I only have a > single physical network interface, so I followed instructions for > networking vnet jails via epair and bridge, e.g. > > devel > { > vnet; > vnet.interface = "e0b_devel"; > exec.prestart += "/jails/jib addm devel genet0"; > exec.poststop += "/jails/jib destroy devel"; > } > > The issue is bulk TCP performance throughput between this jail and the > host is quite poor, with one CPU spinning 100% in kernel and others > sitting mostly idle. > > It seems there is some lock contention somewhere, but I'm not sure if > this is around vnet, epair or bridge subsystems. Are there > other alternatives for vnet jails? Can anyone recommend specific > deployment scenarios? I've seen references to netgraph which could be > used with jails. Does it have better performance and scalability and > could replace epair and bridge combination? You can try to disable one of (or all of) the following: LRO, TSO, RXCSUM, TXCSUM by ifconfig on you NIC. ifconfig em0 -rxcsum -txcsum -tso -lro" to disable The same thing without dashes "-" to enable Use your NIC name instead of em0. If some part disabled fixes you problem, put it into your rc.conf ifconfig line. Or you can try netgraph buddy https://github.com/bellhyve/ngbuddy Kind regards Miroslav Lachman From nobody Thu Sep 12 18:50:20 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X4RMh1nprz5WW6s for ; Thu, 12 Sep 2024 18:50:24 +0000 (UTC) (envelope-from cryintothebluesky@gmail.com) Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X4RMg2plDz4j5N for ; Thu, 12 Sep 2024 18:50:23 +0000 (UTC) (envelope-from cryintothebluesky@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-wm1-x32f.google.com with SMTP id 5b1f17b1804b1-42cafda818aso12236945e9.2 for ; Thu, 12 Sep 2024 11:50:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726167022; x=1726771822; darn=freebsd.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=FuwxJ8qEQQDa/4S2t8feeNNxf7SWqs2c8D4i7VlN5bk=; b=EA8PfWCY9igcuZoHut3/Et0IRgGkbi/RGjGAE6nHP6SDUAqEBCJ/rv67exBwV8xLrt YoZFDKEqhsKAg6H+7GvAEP26PuUNmCTL4SyucJhpHTUzHZUV+/9Qvw+SPyQRKiigpwLS 6YsulEvyUfex64a0QnEtZFUIwL61rWggYWFZlprg6tY99Is5tvLjTZNiLXc09+UqjxYx df7YSzP5GeaojHAUkFfs3HXxHDxSeJu/xWNsenmjQGUq5OnDfalO6SUOreOv7agR+S8s skHJ3v9xLIaFX3tmOmRSYYHulj/RMQH5eWGD2t9Ylb0ednOealjQM+xI/aDedUbwQAiD +Ucw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726167022; x=1726771822; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FuwxJ8qEQQDa/4S2t8feeNNxf7SWqs2c8D4i7VlN5bk=; b=qTaZtrQx6GWhoBeR8feNyO7160uV1f0nb1M7sG9KfJSoE+DdDw1Cj5rDG3V7SZPFuV M1bNCPAOUTj+c4Yqev9YPEYTaLZlG2CSwLvDYvvAicXEVy4XXO1xWWAF0Tl/r40XuEwH e+3DcYfpLGhUaXtYFcg+AhOkqTLjtloGysklPsqkVo9Sv9P5Zzcmw93R8OlTbDtx3HA8 TrcFatwC4SryYQpRvHUtw/MeD6mvCtLWEiVa0OmUDhyDujG5TpjcFUOFcUCHj4PEtRb3 QVFmkYV5b2qWd9p/m0DEZkyFuUynS3JDZ3ZMhmvIYbzd7oNoPCucylvPz/ZR+60nI2XN qenA== X-Gm-Message-State: AOJu0YyqT1kLLkmcvu/+ebIBDfsdZ0PHboULflDy2H1IG8FLsotfxSEA J0lTcfDdDJpIu+vwyJsP489Xe0qV38qwhalK2Chd2GdkSNJ94mJk X-Google-Smtp-Source: AGHT+IF/K3tPFVFJvOMOuWm+C7vvYSoCQJZrAF2kOQ6om3VhAHx3SkcOpw68+0rBER21xhtGikv4AQ== X-Received: by 2002:a05:600c:4f03:b0:42c:b97a:5f7d with SMTP id 5b1f17b1804b1-42cdb509fc7mr34791985e9.7.1726167021265; Thu, 12 Sep 2024 11:50:21 -0700 (PDT) Received: from z600.home.lan (118.129.159.143.dyn.plus.net. [143.159.129.118]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-42d9b16c9d7sm644245e9.30.2024.09.12.11.50.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Sep 2024 11:50:20 -0700 (PDT) Date: Thu, 12 Sep 2024 19:50:20 +0100 From: Sad Clouds To: Paul Procacci Cc: freebsd-net@freebsd.org Subject: Re: Performance issues with vnet jails + epair + bridge Message-Id: <20240912195020.e1bac64a72cc68dee2302238@gmail.com> In-Reply-To: References: <20240912181618.7895d10ad5ff2ebae9883192@gmail.com> X-Mailer: Sylpheed 3.8.0beta1 (GTK+ 2.24.33; x86_64-pc-linux-gnu) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4X4RMg2plDz4j5N On Thu, 12 Sep 2024 13:25:32 -0400 Paul Procacci wrote: > You need to define `poor'. > You need to show `top -SH` while the `problem' occurs. > > My guess is packets are getting shuttled between a global taskqueue thread. > This is the default, or at least I'm not aware of this default being > changed. > You can try enabling `options RSS' in your kernel as this would introduce a > taskqueue worker thread per cpu. Hello, a bit more info. I have a tcp benchmark which performs simultaneous send and receive between sever and client processes. This is running on a 4-core Raspberry Pi 4 and using a total of 8 threads: 4 threads for server process and 4 threads for client process. Each thread opens a single TCP connection. # ifconfig genet0 genet0: flags=1008943 metric 0 mtu 1500 options=280009 ether dc:a6:32:31:71:32 inet 192.168.1.18 netmask 0xffffff00 broadcast 192.168.1.255 media: Ethernet autoselect (1000baseT ) status: active nd6 options=29 Running TCP test via localhost keeps all CPUs busy and gives combined throughput of around 900 MiB/sec. Running TCP test between the host and jail gives combined throughput of around 128 MiB/sec. Disabling RXCSUM and re-running the test gives the same 128 MiB/sec. Running "top -SH" shows this: last pid: 4542; load averages: 0.68, 0.42, 0.27 up 0+09:39:19 19:19:03 154 threads: 7 running, 126 sleeping, 21 waiting CPU 0: 0.0% user, 0.0% nice, 100% system, 0.0% interrupt, 0.0% idle CPU 1: 0.0% user, 0.0% nice, 15.6% system, 0.0% interrupt, 84.4% idle CPU 2: 0.8% user, 0.0% nice, 42.2% system, 0.0% interrupt, 57.0% idle CPU 3: 0.8% user, 0.0% nice, 23.4% system, 0.0% interrupt, 75.8% idle Mem: 12M Active, 46M Inact, 274M Wired, 162M Buf, 3500M Free Swap: 4096M Total, 4096M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 0 root -64 - 0B 608K CPU0 0 4:11 100.00% kernel{epair_task} 11 root 187 ki31 0B 64K RUN 1 575:35 81.58% idle{idle: cpu1} 11 root 187 ki31 0B 64K CPU3 3 574:58 74.14% idle{idle: cpu3} 11 root 187 ki31 0B 64K RUN 2 576:04 60.94% idle{idle: cpu2} 4542 root 4 0 29M 3408K select 2 0:01 11.95% tcp_bench{tcp_bench} 4542 root 28 0 29M 3408K select 2 0:00 11.79% tcp_bench{tcp_bench} 4542 root 32 0 29M 3408K select 3 0:01 10.85% tcp_bench{tcp_bench} 4538 root 26 0 29M 3624K select 3 0:01 10.62% tcp_bench{tcp_bench} 4538 root 21 0 29M 3624K CPU1 1 0:00 10.07% tcp_bench{tcp_bench} 4538 root 26 0 29M 3624K select 1 0:01 10.05% tcp_bench{tcp_bench} 4538 root 24 0 29M 3624K select 1 0:01 8.91% tcp_bench{tcp_bench} 4542 root 31 0 29M 3408K select 1 0:01 8.78% tcp_bench{tcp_bench} 4537 root 20 0 14M 3748K CPU2 2 0:00 0.24% top I will try adding "options RSS" to GENERIC config file and then build new kernel. From nobody Thu Sep 12 19:20:23 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X4S2N1LZ7z5WZxS for ; Thu, 12 Sep 2024 19:20:28 +0000 (UTC) (envelope-from cryintothebluesky@gmail.com) Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X4S2M3gSfz4qKn for ; Thu, 12 Sep 2024 19:20:27 +0000 (UTC) (envelope-from cryintothebluesky@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=Vlls807q; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of cryintothebluesky@gmail.com designates 2a00:1450:4864:20::32a as permitted sender) smtp.mailfrom=cryintothebluesky@gmail.com Received: by mail-wm1-x32a.google.com with SMTP id 5b1f17b1804b1-42cae102702so1069535e9.0 for ; Thu, 12 Sep 2024 12:20:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726168825; x=1726773625; darn=freebsd.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:to:from:date:from:to:cc:subject:date:message-id :reply-to; bh=w8EL6XNxAgIf1W/3o0Gijc8Tgs0Trqxij76rDseIgSU=; b=Vlls807qKzqDCa1KGK0bWm085JEIInjTSmu6RhXrPbyCgjydrpu5bsuwPEOEYh0HHb qH8p12YJjZKI4Q6gaj5JhVbYSuIT93dOt5lX9SpEern24CX2145e/w6mJyrWjBN3o3Ll xw9IhN5vFm5uBoDv6Uq0DiVvXxMsFeYgJN1uAWm24hZF4IqMj/oluZQAM5bjd87isl0H dTxiNY9LIOoF3m0O4YFb8cwDQp8OP/2S00nJW6tg7GC6oV1Qhoyv7muXSMjlk6qKcgmv UILXEmOzT8VR1UKbLkgk6BUEpP3rtp2dp55iS8OL4LLBIW7i7pjQkrd5pWACuAmV9uga HEbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726168825; x=1726773625; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=w8EL6XNxAgIf1W/3o0Gijc8Tgs0Trqxij76rDseIgSU=; b=e6zE6fBVhrfYZZQh6CA9n9zI+KTViC5Kmp8VAYJ/abJaTyjDE9vgBFbSzFoUvCvjKE QXA9WBEzABFJYy2d5M1rAF716SBYnbz5ctRV7mrt8uC32pA23jTuP/rC6Ig+nkTYOD0N fP2MWP7Pd0p1WOrcFVeUu6GzZlEh0HqYbHj9cdNpJWyVVaszQCHRV6MXRO9CJGAkQL1q Ql2g9uVd2amJGR3jaONQWquTBgRqA+XzLa3XDo//UZzDtcHbjnH72xoIag8J4NAUL/j1 XRM3nvXd6k4oT7xIygZ0wtfhWVFjxrumNHG+yZ5l+2C1OyI9puW/7Wd1Gj3mE1F0tcuI p8QA== X-Gm-Message-State: AOJu0YzLWtk5sEdL03EkHepx70VtUxwKEfEQhfHDStLIA5mck/pkdW5C g0XHOroTGtZMd3RsLkXCZqyZcW8BIgtt+PGUUsKUhUYSfpMMl5rsGDRrRg== X-Google-Smtp-Source: AGHT+IHyIFfyoJnK9WTthJecMxiK1cDjXh0YuaU69YEM/mSj5jBuQtqbXJsYQSqtmLDVg//PtHETtQ== X-Received: by 2002:a05:600c:3ba4:b0:426:63b4:73b0 with SMTP id 5b1f17b1804b1-42d964e1875mr3092545e9.34.1726168824698; Thu, 12 Sep 2024 12:20:24 -0700 (PDT) Received: from z600.home.lan (118.129.159.143.dyn.plus.net. [143.159.129.118]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-42d9b1947e2sm1098245e9.44.2024.09.12.12.20.24 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Sep 2024 12:20:24 -0700 (PDT) Date: Thu, 12 Sep 2024 20:20:23 +0100 From: Sad Clouds To: freebsd-net@FreeBSD.org Subject: Re: Performance issues with vnet jails + epair + bridge Message-Id: <20240912202023.0a9ddaf7a69dddcc54e17439@gmail.com> In-Reply-To: <20240912181618.7895d10ad5ff2ebae9883192@gmail.com> References: <20240912181618.7895d10ad5ff2ebae9883192@gmail.com> X-Mailer: Sylpheed 3.8.0beta1 (GTK+ 2.24.33; x86_64-pc-linux-gnu) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.45 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.95)[-0.946]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_FROM(0.00)[gmail.com]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::32a:from] X-Rspamd-Queue-Id: 4X4S2M3gSfz4qKn There seems to be an open bug describing similar issue: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=272944 From nobody Fri Sep 13 06:18:49 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X4kf53NKpz5W27h for ; Fri, 13 Sep 2024 06:18:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X4kf52Dp2z4HrL for ; Fri, 13 Sep 2024 06:18:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1726208333; a=rsa-sha256; cv=none; b=kF1m7yXURNkoEQHY59eNEFkfjlkuJ1PPd0OAxfVY+bFor0vFxiAntCNAiX1imNvyXvF58b fJCvHcdcrk16RUrdeDaV+1GWklZ7S6Tv4+3YWDU5CoHJ6mQ4BxEA/yvObMIcXmuZlaPQ4a R6GnWtCPY9LthvsEB/rYAZalLzVQpTp/Oit3azAHRSBCE4RgGWVJaAjstPPSqAtJKSU4wS xOQsAygj18TxxI/cHvoydPZwAmvw1bTUnYuiwi6n/Hjwm8k8r4BwJDE3NN/1GS0rruMLgq v6bDLJmiXcKCTEGGoW4lwcC8xIJxa7Dbi7Y8TAOeitI9Zw60QdK3Dvl2YquA2g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1726208333; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=62ncuM5XG+ttTIpD5tOYR/vSSi5Kohd/HfHQOo4r+uM=; b=VCKywknnIGRJCz0Z8Jz5I+dHHIQd4Af189cT5fi/V63WiLAPxpS0JxT04DhqZT1JC/1J1/ TER4mvnvrma+PfolN7BaSVIM+bqPxfiROoIcLGywUPiWJ6+fKnZNKw9My8Jl0EwjJiDfsH D96FVfCdtqkYke1ibqrJXfoOokNYaoPFV1xzM4d7SHh3a8Sv77cRzlnv6fe4QMNMaLTQjL /21K/ydtDJ7tXbDNwEKrik6pNH9kC+N77pbLcYrjziTy2EsCGav64BrsZLih+99ZBXKiSb Tt574J7tdOPpsLzaZKOuq1+fp0jeuqE/Ytjb3saU9Qy3U2mNR63rGny4U84idw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4X4kf51sFJzXsX for ; Fri, 13 Sep 2024 06:18:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 48D6IrxY018470 for ; Fri, 13 Sep 2024 06:18:53 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 48D6IrEa018463 for net@FreeBSD.org; Fri, 13 Sep 2024 06:18:53 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 272416] Seldom crash happening with RTL8125 Date: Fri, 13 Sep 2024 06:18:49 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.2-RELEASE X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: kbmithunchakravarthy@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D272416 --- Comment #12 from Mithun --- (In reply to Jonathan Vasquez from comment #11) Hi Jonathan, Thanks for your response!. Unfortunately, I don't have a spare slot to swit= ch NICs, so I'll continue looking for a solution with the if_re module. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Sep 13 07:03:56 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X4lf91Y74z5W8cx for ; Fri, 13 Sep 2024 07:04:01 +0000 (UTC) (envelope-from cryintothebluesky@gmail.com) Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X4lf86gN2z4P9F for ; Fri, 13 Sep 2024 07:04:00 +0000 (UTC) (envelope-from cryintothebluesky@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-wm1-x331.google.com with SMTP id 5b1f17b1804b1-42cc43454d5so4852695e9.3 for ; Fri, 13 Sep 2024 00:04:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726211039; x=1726815839; darn=freebsd.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=BIyGYmYiFRoIHYNRYqJeC/DHp1kUJ3rlr7O+EU9Hv0g=; b=EmCKHzg4bEGE9xDcVOhphCfTMFwuF0/AZDgD8mhy2pwj6uZY8YPOx+tyVuyeJBCXvw DF6qwhE1itO6uFiauptZy+UunbPEHptSI2EtB+7G6HtvuNzBsNU9ep6fxXRUSAFfhg8O k7sscWJJ1cSsPm99+1wfdNUdOLCijjIYtg9aLJJrGZhxgZ0+oJsfwwC3rK6vuZtmO8/o DK89uFLa9NXHz4SuhNZOjCWIx2Aku3jhDqoqrczOKCqgvinFXoOxgqrtUyOdrT26oIVn JMtTkc02dFFCrRlNepmMv00GPigSA0pe4IEu+9/Q2sSnn94jIEpAHqnCG1i1WQy97hTa mqFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726211039; x=1726815839; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BIyGYmYiFRoIHYNRYqJeC/DHp1kUJ3rlr7O+EU9Hv0g=; b=e/i3XXtKFYvlqu5eXoDH4A6mq2m06Da4HnSHOydsr6WBhqhaYfkQngjySlB24XcAuU 2qEPiC/FdGN+JtO3kgaLcYq9eC7LrwIMBdsrby/uIvlcbmYirHjHm8lXIC1QGEA1atu1 NFQf/13UO+aQRdJAmef5VM3CHRiXN2GuR3c4iiC1gkTsES723bn5n+LwsvFLzBmq4O9X yYZr/i9xkoyhIst6AU0k3+HSXO0m4E5C/PzHO+DPduxKUKIcsbrd5jIqSrAxkEWNDS1o 1bL8vOhb5BgzN3bdBiDhEczfZZzce28KSUWHG665zZpJvSIKx9hcfztdldeUDej4BxNC 5qvQ== X-Gm-Message-State: AOJu0Yy01CSzskkxUsOTnlYXEhqYQ5vEtEMFDebb65xAybp84YC0erRR T8/g9Dm1a9a9vydKgeN5ekIRzqFk08j17wgesJRIBe0HgBKM+bZ/HDawKw== X-Google-Smtp-Source: AGHT+IGRPNcGv7thhqTa8mOOK6FMIZm/nBhUMJEf9qN7ytczfJqnh2I5Vh+7s0YdSjWAThfVvkVWJg== X-Received: by 2002:a05:600c:654e:b0:42c:e0da:f132 with SMTP id 5b1f17b1804b1-42d9a9109fdmr11888355e9.32.1726211038392; Fri, 13 Sep 2024 00:03:58 -0700 (PDT) Received: from z600.home.lan (29.129.159.143.dyn.plus.net. [143.159.129.29]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-42d9b05a76csm13760295e9.8.2024.09.13.00.03.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Sep 2024 00:03:57 -0700 (PDT) Date: Fri, 13 Sep 2024 08:03:56 +0100 From: Sad Clouds To: Paul Procacci Cc: freebsd-net@freebsd.org Subject: Re: Performance issues with vnet jails + epair + bridge Message-Id: <20240913080356.98ea2c352595ae0bbd9f0ce8@gmail.com> In-Reply-To: References: <20240912181618.7895d10ad5ff2ebae9883192@gmail.com> X-Mailer: Sylpheed 3.8.0beta1 (GTK+ 2.24.33; x86_64-pc-linux-gnu) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4X4lf86gN2z4P9F I built new kernel with "options RSS" however TCP throughput performance now decreased from 128 MiB/sec to 106 MiB/sec. Looks like the problem has shifted from epair to netisr PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 12 root -56 - 0B 272K CPU3 3 3:45 100.00% intr{swi1: netisr 0} 11 root 187 ki31 0B 64K RUN 0 9:00 62.41% idle{idle: cpu0} 11 root 187 ki31 0B 64K CPU2 2 9:36 61.23% idle{idle: cpu2} 11 root 187 ki31 0B 64K RUN 1 8:24 55.03% idle{idle: cpu1} 0 root -64 - 0B 656K - 2 0:50 21.50% kernel{epair_task_2} Are there any other tricks I can try to distribute the load over multiple CPU cores? I found this document which describes various vnet examples: https://freebsdfoundation.org/wp-content/uploads/2020/03/Jail-vnet-by-Examples.pdf The problem is, with Raspberry Pi I cannot add any pcie network cards with multiple ports or SR-IOV features. I was hoping to use some software based virtualization device, but if the scalability is this bad, then I may have to discard vnet completely and configure jails to use the host network stack. Thanks. From nobody Fri Sep 13 07:07:47 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X4lkl0w6dz5W9GC for ; Fri, 13 Sep 2024 07:07:59 +0000 (UTC) (envelope-from freebsd-net@dino.sk) Received: from cm0.netlabit.sk (mailhost.netlabit.sk [84.245.65.72]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4X4lkj5jN1z4Q6q for ; Fri, 13 Sep 2024 07:07:57 +0000 (UTC) (envelope-from freebsd-net@dino.sk) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of freebsd-net@dino.sk designates 84.245.65.72 as permitted sender) smtp.mailfrom=freebsd-net@dino.sk Received: from zeta.dino.sk ([84.245.95.254]) (AUTH: LOGIN milan, TLS: TLSv1.3,256bits,TLS_AES_256_GCM_SHA384) by cm0.netlabit.sk with ESMTPSA id 0000000001496D47.0000000066E3E4C5.0000902C; Fri, 13 Sep 2024 09:07:49 +0200 Date: Fri, 13 Sep 2024 09:07:47 +0200 From: Milan Obuch To: freebsd-net@freebsd.org Subject: Re: Ethernet device with shared mdio Message-ID: <20240913090747.48062204@zeta.dino.sk> In-Reply-To: References: X-Mailer: Claws Mail 3.21.0 (GTK+ 2.24.33; amd64-portbld-freebsd13.3) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.33 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_SHORT(-0.03)[-0.025]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:5578, ipnet:84.245.64.0/18, country:SK]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; MID_RHS_MATCH_FROMTLD(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; R_DKIM_NA(0.00)[]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[dino.sk]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 4X4lkj5jN1z4Q6q On Fri, 6 Sep 2024 18:03:39 +0000 Mike Belanger wrote: > The following device tree specifies a shared mdio. > The ffec driver uses miibus. > When there is a shared mdio, one of the device instances will not be > able to properly configure the PHY, as it needs to use the other > devices resource to read/write the PHY. > > &fec1 {pinctrl-names = "default"; > pinctrl-0 = <&pinctrl_fec1>; > phy-mode = "rgmii-id"; > phy-handle = <ðphy0>; > fsl,magic-packet; > status = "okay"; > > mdio { > #address-cells = <1>; > #size-cells = <0>; > > ethphy0: ethernet-phy@0 { > compatible = "ethernet-phy-ieee802.3-c22"; reg = <0>; > }; > > ethphy1: ethernet-phy@1 { > compatible = "ethernet-phy-ieee802.3-c22"; reg = <1>; > }; > }; > }; > > &fec2 { > pinctrl-names = "default"; > pinctrl-0 = <&pinctrl_fec2>; > phy-mode = "rgmii-txid"; > phy-handle = <ðphy1>; > phy-supply = <®_fec2_supply>; > nvmem-cells = <&fec_mac1>; > nvmem-cell-names = "mac-address"; > rx-internal-delay-ps = <2000>; > fsl,magic-packet; > status = "okay"; > }; > > Does FreeBSD have any plans for supporting hardware that specifies a > shared mdio in the dtb? Just knowing the general approach being > considered would be helpful. > I can't speak for FreeBSD project, I just can share my experience with similar case. It is described in my post to hackers mailing list (see https://lists.freebsd.org/archives/freebsd-hackers/2021-December/000649.html for details), unfortunately, no response received. Another attempt to get some attention a week later on net mailing list was done, see https://lists.freebsd.org/archives/freebsd-net/2021-December/001114.html for the post, with no response either. As you see, my case was similar, just the mdio block was attached to second controller. This makes it a bit more problematic - you can't use mdio controller before being initialized, naturally. I was not able to use miiproxy approach as noted in my post to hackers mailing list, additionally, miiproxy was removed from the tree with MIPS arch some time later. I resolved the issue by modifying cgem driver and mii layer. This was just a proof of concept with some hacks, but I was able to use both ports with proper link state change detection. I did not continue the work because vendor changed hardware design and there was no shared mdio anymore. If you are interested I can dig for the sources, big part of my changes would not be necessary, just the idea of decoupling MDIO and MII interfaces still applies, I think. By the way, which board are you working on? Is it accessible for general audience? Regards, Milan From nobody Fri Sep 13 07:36:59 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X4mNH339Yz5WF1P for ; Fri, 13 Sep 2024 07:37:03 +0000 (UTC) (envelope-from cryintothebluesky@gmail.com) Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X4mNG2bl4z4TwQ for ; Fri, 13 Sep 2024 07:37:02 +0000 (UTC) (envelope-from cryintothebluesky@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=MbAbyEY7; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of cryintothebluesky@gmail.com designates 2a00:1450:4864:20::333 as permitted sender) smtp.mailfrom=cryintothebluesky@gmail.com Received: by mail-wm1-x333.google.com with SMTP id 5b1f17b1804b1-42bbffe38e6so14703305e9.0 for ; Fri, 13 Sep 2024 00:37:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726213021; x=1726817821; darn=freebsd.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:from:date:from:to:cc:subject:date:message-id :reply-to; bh=ei3eRXsR0qj1xn/aQka2w/a24Pl/29nD++RvXItFSVE=; b=MbAbyEY7svsHzEJFisVbxzUOxcYEMaPpBcuhmP8hLvXDwn4h+Ve3Lhk4Q7QfewPqbG TJoi2ym0Iv0lm2DG3YHwUdFmQpU8mwcokTDnyCEBheoPbM7qaNqOLSYkOHMtf+gMDC7l +XdbAcvpoETRgyjbMPVcDboYZ4FMeQAz9q3FapTR2C8kq2Cg7oDfNTtFbme+kLamjmaY 0aWMmGcx9Evt9jb04CxcE1Kt02Rt4MvpvjyfC4QyU2nEeY2kbMA3A2oK7Zzaw3vEVjf2 zsnlmpCTpmprpOtZiUIiRMm/hzcMAsuLpzmeOvj+O7ltaxlJFmK/Ro+jk6SWPDpaur+P tAhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726213021; x=1726817821; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ei3eRXsR0qj1xn/aQka2w/a24Pl/29nD++RvXItFSVE=; b=c3XOdeHMLXDKRQKEdHhLyRB7ewYeik0UnAy/MtOzNjWVdUf+oxYwTdOo0X+0Yts97I +6JFHWga6aDWQsfGiFb/IOS2BDhIOHomGQBnWzWuBsmWGVrxV7w1Cp00jPDL2eAfFJ6g przZQk601P2B6cEZBwBzi2jwG/ngqikQTwk56zG675Rw9F6m0HxG8pEh4kQ4srRn8Wjj +2EhOHp2fpekYE6KbV4TT8j6F7NIFrXW24NDvhwClmqBonvCOMX9JAwsfmBtWFGi78gg 4fjKRAIfqjo3cNt9sRe/UYEIxjJ8JTLvB+iwSB4+8Nm30ET10bLsxSW1XKzgnFGfPHtr e6Uw== X-Forwarded-Encrypted: i=1; AJvYcCXxU6IDLHlu50egpHkwA9GRMfhr2XWoX2ygga9beia2RDR6PP4BEcSdVkwYdm5+oNjQl0wuUsp1g0zdRA==@freebsd.org X-Gm-Message-State: AOJu0YxRZBA4bxEgH0C9buhJbfn2zcOM3RasT/z3EocaKV9bOdQDKEjk VPmIAOrLsL7NrcdN7MJxyMcUiAzqUwxO0gGRPVt4Qsluf5D3RVtoPS3b+w== X-Google-Smtp-Source: AGHT+IEDCrnd+YItga+5SAWqUu/RhfEEY30fC+n31KZX18xV1TzLhN3GJ1YHkdBMM1bCRUsI5tKNxQ== X-Received: by 2002:a05:600c:474c:b0:42b:ac80:52ea with SMTP id 5b1f17b1804b1-42cdb5097fcmr45022545e9.6.1726213020479; Fri, 13 Sep 2024 00:37:00 -0700 (PDT) Received: from z600.home.lan (29.129.159.143.dyn.plus.net. [143.159.129.29]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-378956ddcdcsm15994327f8f.100.2024.09.13.00.37.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Sep 2024 00:37:00 -0700 (PDT) Date: Fri, 13 Sep 2024 08:36:59 +0100 From: Sad Clouds Cc: Paul Procacci , freebsd-net@freebsd.org Subject: Re: Performance issues with vnet jails + epair + bridge Message-Id: <20240913083659.443548a87559c3cbaba4e9d8@gmail.com> In-Reply-To: <20240913080356.98ea2c352595ae0bbd9f0ce8@gmail.com> References: <20240912181618.7895d10ad5ff2ebae9883192@gmail.com> <20240913080356.98ea2c352595ae0bbd9f0ce8@gmail.com> X-Mailer: Sylpheed 3.8.0beta1 (GTK+ 2.24.33; x86_64-pc-linux-gnu) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Bar: - X-Spamd-Result: default: False [-1.17 / 15.00]; MISSING_TO(2.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.67)[-0.674]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::333:from] X-Rspamd-Queue-Id: 4X4mNG2bl4z4TwQ On Fri, 13 Sep 2024 08:03:56 +0100 Sad Clouds wrote: > I built new kernel with "options RSS" however TCP throughput performance > now decreased from 128 MiB/sec to 106 MiB/sec. > > Looks like the problem has shifted from epair to netisr > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 12 root -56 - 0B 272K CPU3 3 3:45 100.00% intr{swi1: netisr 0} > 11 root 187 ki31 0B 64K RUN 0 9:00 62.41% idle{idle: cpu0} > 11 root 187 ki31 0B 64K CPU2 2 9:36 61.23% idle{idle: cpu2} > 11 root 187 ki31 0B 64K RUN 1 8:24 55.03% idle{idle: cpu1} > 0 root -64 - 0B 656K - 2 0:50 21.50% kernel{epair_task_2} I think the issue may be to do with the genet driver itself. I think the hardware is limited to one CPU per send or receive interrupt. On Linux the best I can do is set SMP affinity for send on CPU0 and for receive on CPU1, but that still leaves 2 other CPUs idle. $ cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 ... 37: 74141 0 0 0 GICv2 189 Level eth0 38: 43174 0 0 0 GICv2 190 Level eth0 From nobody Fri Sep 13 09:09:38 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X4pRB4Tzpz5WT8P for ; Fri, 13 Sep 2024 09:09:42 +0000 (UTC) (envelope-from cryintothebluesky@gmail.com) Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X4pR92bYWz4mDr for ; Fri, 13 Sep 2024 09:09:41 +0000 (UTC) (envelope-from cryintothebluesky@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=Mbhoa5ha; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of cryintothebluesky@gmail.com designates 2a00:1450:4864:20::42e as permitted sender) smtp.mailfrom=cryintothebluesky@gmail.com Received: by mail-wr1-x42e.google.com with SMTP id ffacd0b85a97d-374ca65cafdso448990f8f.2 for ; Fri, 13 Sep 2024 02:09:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726218579; x=1726823379; darn=freebsd.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:from:date:from:to:cc:subject:date:message-id :reply-to; bh=pjngj03fUoPNLnA7mGzIbOXC4s/ceLLTinF5SINwafo=; b=Mbhoa5hacNlkRDU9UKk+yYQF47IW+v3XLtHbjWK/MzZVOHErUIKZRWStOduet+K23k IgSyGrvdsHIwUoehPFJ8TxPNSHDQzihUkQu02tZSyEvIXo2xzdsGa2jxh8rO17BEYD5p yq0hrZ37qRpYaYYqRezXUeyOzvcLtyKj2lBUlCCgJ9AuwuesXU4RvXZNKcQVYN2HPydg eylfctvw3e5IRRm32s9e8OyvTbPfPRVOrkt53ky/mdbem43SKrU5Bn3poIrIhaxop/8o umChzUO+nX6JRzNai+fAB/uxjYlk4pwXrph1b6FNs7NTHJz+XbM5o4x3hukYUyT89ZTH g0QA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726218579; x=1726823379; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pjngj03fUoPNLnA7mGzIbOXC4s/ceLLTinF5SINwafo=; b=IW2+3OOM/ZvvEgb1cdsxIeKo4XImATs/VGQyIXqyCgi6l/93S+5Ss+6c5+CKOItdtm +SdYK6t/yON+f8O1S9AvsTyj/kkPVtDa/hgP/ojyyBktNzzq6NQqn1eDJaDtf/w+D6ne BDa85W5z50n+6A//dPyNEma/aEJx0qE1gLArP1+ajZoZaCbeFtd6v9XIlEnkWxxCqf2V ZvTwC/DfpGFYEPim9R75NqRTr0Vb/cllzNaDTY4F8pD4+5QZw1PZE9Z+GjpQhGpgYcvp BZQPgUva6PNd72/TN5TvPu7cYkDU6Pvp1H0vopeWlkb2IWYI4yF18cHxHOkDZ+Jiia5q jOvw== X-Gm-Message-State: AOJu0Yz+urkkdqS4cdaiDRBo1RQvewmn+/vgwru2otaDVQrOJOSalBRH Trr4u/IUhQLFtkDYclcY/FnOsJm59N1Havz9gQldnFCjyb+rRLNOkURQLw== X-Google-Smtp-Source: AGHT+IGyR1BLMGM+zTyS0S9rak+ZYHSWq/ennqcrK4YhO1n4IZgga11z/9sWij16KXxfk4lQIwMc5w== X-Received: by 2002:adf:ea46:0:b0:374:c712:507a with SMTP id ffacd0b85a97d-378d61f125fmr1131088f8f.32.1726218579347; Fri, 13 Sep 2024 02:09:39 -0700 (PDT) Received: from z600.home.lan (29.129.159.143.dyn.plus.net. [143.159.129.29]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-37895665632sm16171803f8f.37.2024.09.13.02.09.38 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Sep 2024 02:09:38 -0700 (PDT) Date: Fri, 13 Sep 2024 10:09:38 +0100 From: Sad Clouds Cc: freebsd-net@FreeBSD.org Subject: Re: Performance issues with vnet jails + epair + bridge Message-Id: <20240913100938.3eac55c9fbd976fa72d58bb5@gmail.com> In-Reply-To: <20240912181618.7895d10ad5ff2ebae9883192@gmail.com> References: <20240912181618.7895d10ad5ff2ebae9883192@gmail.com> X-Mailer: Sylpheed 3.8.0beta1 (GTK+ 2.24.33; x86_64-pc-linux-gnu) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Bar: - X-Spamd-Result: default: False [-1.43 / 15.00]; MISSING_TO(2.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.93)[-0.926]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FREEMAIL_FROM(0.00)[gmail.com]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::42e:from] X-Rspamd-Queue-Id: 4X4pR92bYWz4mDr Tried both, kernel built with "options RSS" and the following in /boot/loader.conf net.isr.maxthreads=-1 net.isr.bindthreads=1 net.isr.dispatch=deferred Not sure if there are race conditions with these implementations, but after a few short tests, the epair_task_0 got stuck 100% on CPU and stayed there in this state indefinitely, until I rebooted. PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 0 root -64 - 0B 672K CPU0 0 6:24 100.00% kernel{epair_task_0} Is RSS considered too experimental, hence not enabled by default? From nobody Fri Sep 13 12:08:02 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X4tPD02PNz5WrDL for ; Fri, 13 Sep 2024 12:08:16 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: from mail-qt1-x82f.google.com (mail-qt1-x82f.google.com [IPv6:2607:f8b0:4864:20::82f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X4tPC1MGbz4D5l for ; Fri, 13 Sep 2024 12:08:15 +0000 (UTC) (envelope-from nonesuch@longcount.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=longcount.org header.s=google header.b=KrEA2MEy; dmarc=none; spf=pass (mx1.freebsd.org: domain of nonesuch@longcount.org designates 2607:f8b0:4864:20::82f as permitted sender) smtp.mailfrom=nonesuch@longcount.org Received: by mail-qt1-x82f.google.com with SMTP id d75a77b69052e-4581d2b0fbaso6228231cf.1 for ; Fri, 13 Sep 2024 05:08:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=longcount.org; s=google; t=1726229294; x=1726834094; darn=freebsd.org; h=to:in-reply-to:references:message-id:date:subject:mime-version:from :content-transfer-encoding:from:to:cc:subject:date:message-id :reply-to; bh=2LuxSWi2eKVAZcSMN37vYz+JPnq4HUfOQeZyR7whpGA=; b=KrEA2MEy5mWSpI7omnqYWJW53zK99gI240BAr15/Ph4oNjnK+820sWHD1y1OO/MB3R ZMUSgw5kNG5kV9E5Pu6rPz5ObLdclpy/N+zLC12MW5HCk/AhYDauuDs+ygmJXP4GhZgo /NC4LVRAJQDBU4NmyvKmxkN8sqnX7/moXvYGqAmGzCukVc3C/mtPwgAUFq6yY/4wm5lo ImDA1sdWNnB+xjFcxgJUCfKpgEkpJo60gGDweiDgcDS6Kq+QTL/kP+J8YeOOvEt/xwgh 7J5KyAGiv4XotbGZ8YamBtrDH9u+Ur9+d56HE02TSXLxdSTEZ/HGx7L+FBdynXkhbhNC gY8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726229294; x=1726834094; h=to:in-reply-to:references:message-id:date:subject:mime-version:from :content-transfer-encoding:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=2LuxSWi2eKVAZcSMN37vYz+JPnq4HUfOQeZyR7whpGA=; b=wevYnjDxCVrC4QqYMP+2OVzfQXZj76J3HLdtTTgaIBvzWIe+E5mNSuQfJFGe9UXJbW G55ImGKtfPe6a9qSMVyXfBtwc435NKRHqbzO2Ym5H4J1rZ4gdSYvyQiQH5Pw6vtv2vCt jFkmgv0zhQ/ftFK9Qzc1RvxBB8/N4sxqUl1pL0xxE72rqE0sIVSJTroTPcj6v6yXF+ch 72U6QS2gD4zsS6dbslzysepYonJxxRnqibHbEOwb8J9MbQbxphNgADccyLklGsFWO1Os B62Lq9GoFMgMTV3/GxeDgkuwDJYokq8wbU560uYb/A1dYNBYFqh19Qn+AASmnLLka1ll qUQg== X-Gm-Message-State: AOJu0YzvoxrYnsGF/bzVxsdrSuQj3rFqUbwF17QhsG/tKhJeDQrC3D9E fKGQvxqo0+vhumcIdZGkUmYqHStouEsoDNGS71fN6jXmZ89Cgu86WKsP1lTt69afi/NiRu6PPnm M3nk= X-Google-Smtp-Source: AGHT+IF51dBG2Zx0ZAojvSCmvFeNicXYzYC8c8PQ/8CvlqdBfb9IcY8ifvM1k2Sze/KzRHgRCaiEKQ== X-Received: by 2002:a05:6214:3912:b0:6c3:5f65:f9d7 with SMTP id 6a1803df08f44-6c57e11c553mr39685686d6.36.1726229293513; Fri, 13 Sep 2024 05:08:13 -0700 (PDT) Received: from smtpclient.apple (pool-68-161-195-36.nycmny.fios.verizon.net. [68.161.195.36]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6c53474d904sm65252676d6.76.2024.09.13.05.08.13 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 13 Sep 2024 05:08:13 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable From: Mark Saad List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 (1.0) Subject: Re: Performance issues with vnet jails + epair + bridge Date: Fri, 13 Sep 2024 08:08:02 -0400 Message-Id: <39B2C95D-1E4F-4133-8923-AD305DFA9435@longcount.org> References: <20240913100938.3eac55c9fbd976fa72d58bb5@gmail.com> In-Reply-To: <20240913100938.3eac55c9fbd976fa72d58bb5@gmail.com> To: freebsd-net@freebsd.org X-Mailer: iPhone Mail (20H350) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; R_DKIM_ALLOW(-0.20)[longcount.org:s=google]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; DKIM_TRACE(0.00)[longcount.org:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[longcount.org]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; APPLE_IOS_MAILER_COMMON(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TO_DN_NONE(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::82f:from] X-Rspamd-Queue-Id: 4X4tPC1MGbz4D5l >=20 > On Sep 13, 2024, at 5:09 AM, Sad Clouds wrot= e: >=20 > =EF=BB=BFTried both, kernel built with "options RSS" and the following > in /boot/loader.conf >=20 > net.isr.maxthreads=3D-1 > net.isr.bindthreads=3D1 > net.isr.dispatch=3Ddeferred >=20 > Not sure if there are race conditions with these implementations, but > after a few short tests, the epair_task_0 got stuck 100% on CPU and > stayed there in this state indefinitely, until I rebooted. >=20 > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 0 root -64 - 0B 672K CPU0 0 6:24 100.00% kernel{e= pair_task_0} >=20 > Is RSS considered too experimental, hence not enabled by default? Sad Can you go back a bit you mentioned there is a RPi in the mix ? Some of t= he raspberries have their nic usb attached under the covers . Which will kil= l the total speed of things.=20 =20 Can you cobble together a diagram of what you have on either end ? --- Mark Saad | nonesuch@longcount.org= From nobody Fri Sep 13 13:40:55 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X4wSM6WtDz5X2jN for ; Fri, 13 Sep 2024 13:41:07 +0000 (UTC) (envelope-from mibelanger@blackberry.com) Received: from smtp-p01.blackberry.com (smtp-p01.blackberry.com [208.65.78.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4X4wSM4cSdz4SDW for ; Fri, 13 Sep 2024 13:41:07 +0000 (UTC) (envelope-from mibelanger@blackberry.com) Authentication-Results: mx1.freebsd.org; none Received: from pps.filterd (mhs402ykf.rim.net [127.0.0.1]) by mhs402ykf.rim.net (8.18.1.2/8.18.1.2) with ESMTP id 48D17vsO007518; Fri, 13 Sep 2024 09:40:58 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=blackberry.com; h=content-type : date : from : in-reply-to : message-id : mime-version : references : subject : to; s=corp19; bh=ZtDbhCfULbwuZmc/VPKGNMTZbu7pal8aJFnC41w7xWw=; b=JvdgnRsdLtwoJbTdaODjm4sAR0qJfPfh/RkN2tv4NRZd7oZlxmRyFk7dTlYkSnbOBgVv XRTw39RRQWUVq8kAPj7kKnPUV/ycQcOhLZaHV7Fa9n77CNw5R8wfH8/k/XYzfpBDRa9g G7CQvUXOv7hni/ukg3VCyLGaQE1Tuyfxm4lL267MEoiAxbJ9tV9Kc1QQx2jCP0g0CFAm PgrcqfKDN1zT/KDrIo51D63BUDQw6wx0VZo3WL7J2t9oxpDBzm1/Wdvx1AXdrSyb0K+G kjyuMMauQYpVBfAWL6/gyeThHIrCMSWIKX+joLO5mmQwu3o35SE/tKjeCN8f07WEb6sH Mg== Received: from xch212ykf.rim.net (xch212ykf.rim.net [10.12.114.212]) by mhs402ykf.rim.net (PPS) with ESMTPS id 41gh1wy0ba-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Fri, 13 Sep 2024 09:40:58 -0400 Received: from XCH212YKF.rim.net (10.12.114.212) by XCH212YKF.rim.net (10.12.114.212) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.34; Fri, 13 Sep 2024 09:40:57 -0400 Received: from xce210cnc.rim.net (10.4.225.60) by XCH212YKF.rim.net (10.12.114.212) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.34 via Frontend Transport; Fri, 13 Sep 2024 09:40:57 -0400 Received: from YT6PR01CU002.outbound.protection.outlook.com (40.93.18.30) by hybrid-smtp.blackberry.com (10.4.225.60) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.34; Fri, 13 Sep 2024 09:40:57 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Ssi0IDeMyzz5Z3HUTOE/sn3fjC5RUKEXRcPOHdHM0jaE1F0RL/ax8uRDCDZ72+KoWrWG5OsZZuYo8vRS5iys4w0axOkdbFt3nUzLn+AC/j1+szGrbnPuVKITYRJv7PZzrB9d2XJdyPmnPPb5QavaxsOZ/MOl57b6lMYsj0Y9KZwiOCAS0ostpsbFt7AFkPpLguclK47CrgE0h58ZCeHkrjmBshZLt9CgaB39FGi9S3xDPvh+xTdFFMg5Xkwsey5Yv3SWAxP86CWXyeWA05pHxSiIbun+QILIZDf9SwJfqA3++KzTRfrHWQho5aA/g2Ku/PBrWlh0PWlkbPzQjDY3IA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=iKwMjN4mopEL7kRrSUALarQkx3Xi+HJ/2oOFYfdUxHY=; b=ZbWrPaovWhrLOptwEWrqdzY3t9KN32OGf9kyCbfG8ygh3DvubTMsAJqcDkZR0MyusrLXRUadCPkbdnL4T6b9FD2faE3p3E1Ypd2kC7YhJ/HV70lDULFilbFrUa+K6JNLK4LssOz5goJIEhxtg+5oyBzwcKNSzNAd1c597w7JXZP12/AOzcKifQy8edrtrPckM0PPC4BgdIJ9mCet5ZeC/iqqDiWccfnDv9ERJXQIKMZYjs5OT3GujdMnFi3aVZH+v1yJWjuBd+87DvO7IrDBSzdaZ/kmgvE0yY47EyAWgaWzNlYfOOwCQk8CGS/zzKjumwv+3cSrpy+9JsYtxiFkkA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=blackberry.com; dmarc=pass action=none header.from=blackberry.com; dkim=pass header.d=blackberry.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=blackberryltd.onmicrosoft.com; s=selector2-blackberryltd-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iKwMjN4mopEL7kRrSUALarQkx3Xi+HJ/2oOFYfdUxHY=; b=UAjiLESexRMb/JdXQrdId1vQFT/0oq1Xxb4BsBk0NZQaWFLFLw1Fs7BFE9fce0q8N87/ztCsnk4tOBKFiMuuB83ljy70c0yo15KRy+sNbP4Gj1U+/Sr2chSjGS6kskmlAQidBIgoJLn4T9YM6Kb6o+3l01dT8mHJFfzH8fah9tA= Received: from YQXPR01MB4198.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c01:11::8) by YT2PR01MB6388.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:6f::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.20; Fri, 13 Sep 2024 13:40:55 +0000 Received: from YQXPR01MB4198.CANPRD01.PROD.OUTLOOK.COM ([fe80::3789:7f3:7258:6f2a]) by YQXPR01MB4198.CANPRD01.PROD.OUTLOOK.COM ([fe80::3789:7f3:7258:6f2a%7]) with mapi id 15.20.7962.018; Fri, 13 Sep 2024 13:40:55 +0000 From: Mike Belanger To: Milan Obuch , "freebsd-net@freebsd.org" Subject: Re: Ethernet device with shared mdio Thread-Topic: Ethernet device with shared mdio Thread-Index: AQHbAIbfN21ZAF2/J0iYsO7DV1IqvrJVVjiAgABnZ2c= Date: Fri, 13 Sep 2024 13:40:55 +0000 Message-ID: References: <20240913090747.48062204@zeta.dino.sk> In-Reply-To: <20240913090747.48062204@zeta.dino.sk> Accept-Language: en-US Content-Language: en-CA X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: YQXPR01MB4198:EE_|YT2PR01MB6388:EE_ x-ms-office365-filtering-correlation-id: df91125d-b9ae-4acf-6f86-08dcd3f9b171 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0;ARA:13230040|1800799024|366016|376014|38070700018; x-microsoft-antispam-message-info: =?utf-8?B?OWkwc2hWSHpvUWh6cno3YmwrVkRjWU16TE4ybHMzaDdjRVVIZ0xyTUdJMzdN?= =?utf-8?B?eUtGUE4zZ0k3Wk5yanZNdTRWcFIvQVkyWHBJUC94ZW4remJ6UzV1WkF2R29U?= =?utf-8?B?clhaYS9JSmZLZGtyNzloWE90aEFqMno3dmIvOGgrSHdyNzgrYWV6NjdqcUdD?= =?utf-8?B?eFFNclp6NEdtc05jN2FwV25qQXlwbTNPSWxQYkVMYnYrL3FGK2RaNDJBUS90?= =?utf-8?B?RXdZd0pOaFVBVENKMndBQXNCNEJ3NDF2SERINmFVM0tHYlNBTnFyTTZVU3NP?= =?utf-8?B?OGUyWnhYRHVYNFZXeC9JZnRndXh6NmlOMVhOMk5NQnFMYkt0ZkwwWlI1ZFBu?= =?utf-8?B?ZnM1UldXdWF5ellRSENQUXZrL09hTjAyS2NZanpmVHBTNVVFWCs3M0xvcGkz?= =?utf-8?B?RTlORllONVpFTzZmOGlla1EraFJsa3RKUmIwNkttQmRRWUEyZGFiWUFGQ3A0?= =?utf-8?B?RGcvcWhqV1ZwcncyYVlRYzlYb2tFelFvdUFrYk5MZnd3eWs5YzlQNFpWK3k5?= =?utf-8?B?TktWb3Q2cFMvZHlKVHdSSFZWQ0Z5L1hFVWVRRkVYTFczSmVUYmk3UTlRWTRU?= =?utf-8?B?aWk5RDFsSjNaTXFjYVdEOGpaemtrbEFwbU9WcnUxaS8zWGN2YVVJbERjSlZD?= =?utf-8?B?dXcwRkM2VklqdUpTMEhPNHJRZmdhQ3dOa25zV3VaUFVPaG05Y0xHNUpJYnpB?= =?utf-8?B?cHozV2UzYzVlcUdLRjAxcDVuYzhuTm96R1o0RndtMlZ0RmtnSjFWNkJQaWNN?= =?utf-8?B?cEg1eGUwUnR1TnNPWC9zWnFGSGtWa1RpbFVsbXdoVXEyN2FqZzVyV1hQYkc0?= =?utf-8?B?Y0JLQ1I4NjFxSFFwbUJkR1J6SE1FdzN4OE82RytlaVpMZmxhbzdxdVI5Q3R1?= =?utf-8?B?MkcyUHp4dXRZRTVZckdvbkdacCtralo3MkNyNW0xNDZUQkdsTUM0d2tBNzBw?= =?utf-8?B?S2FMbFRSc2ZuNDdaUzNZdGE1MjBFQVRVeTNUTWdrdldHT3ZmMkNMb21MTHE1?= =?utf-8?B?eW9OeGhRdHMyT05KOEpPV0tQaERtMXpkQlNWU2dOeVByKzhCMER5c2wrdXlr?= =?utf-8?B?Mk9lQmRJbWg0WG5yK0lsQ05OQ2tGcFdHdFc5b2NHNytnUzZuZVpYK0dqR2F6?= =?utf-8?B?YnBvaWI2VFFSMzBZTDNBZTRxS2VBaFZ0cVNqQnhTV2NydkdpMVRmcU1YZ2Nw?= =?utf-8?B?d3BZNk1SSStXay8vOEM2Zys3NW85UFk0RnlsQVhMbDdlS1lWZ1JiSFZDaSt6?= =?utf-8?B?cldVOXNjL3hNRzAyN2NEaEZuanhpeW40UituMUNsZ2JyRDVlYmQxT21kNStD?= =?utf-8?B?SWlJcGtnc2Q5Y0g3VlBiR3ZRbkJ6cVNxRlE4QUdpNURXNEljcGREUGxzZDl3?= =?utf-8?B?d1lvYUFUK3RHYnV2Y0ExR2Q2cGs1R1k3T2t4ZmlpcnYzc3Y3ckQyNlNZZm81?= =?utf-8?B?OTk2cCtIYmkvTEllY1A4RzFvOXEyc2FlLzh4V0Rjdmp4VlNFQ0xYZ1k5aEw3?= =?utf-8?B?N0tEdUJuaGNaOFYzMzFHUjRqRG9WdE9kY0E2K0J5QnhvL0VGbk90bmZJMFhx?= =?utf-8?B?T3NaTktSM3FCNDF1cjl3ZGFabEJYWDJCeHRjU1lXMHFkNHZkcXVqQjFNeis3?= =?utf-8?B?VWczN2p6SHFnRzI5S2dLOCtiYmxFazNkSUVMd1VNa1ZPQmlxQTFsUldoUUxs?= =?utf-8?B?WnAxYmpFTTNBTEMzQzZiUWpCMHU3dFFhQ2JIY09WTTFzNTdaMFVNZllDUW9X?= =?utf-8?B?cTFNL1F1T2k5cWJYVUkwcjJuOEtDNVhiYk93ODhZWkJrZjkrT3hxWllHNTMy?= =?utf-8?Q?+KEoWlPeyZCqgjFMLXs4XIc0UcfxIQ8Kw/+C4=3D?= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:YQXPR01MB4198.CANPRD01.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(366016)(376014)(38070700018);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?utf-8?B?NUhjdXM1VXpDQndyYUR4KzZ0Vk9ZeDJvMWFZZjA0VDIzNmtmenc3QjlGdGJv?= =?utf-8?B?ZXNlVlpWRHhVM01oNjFYb2tqRHhqQlQxUVVLZkNIdDlXSzk4blVob01XSjV1?= =?utf-8?B?b1QrSkRuZVRuY1ZpR1Mvb1dzSDdOTkdlVGNUVDZYclJvbEZjcmd3ZWR3YnJJ?= =?utf-8?B?Zjc2UzgvNGdTR3dOakJJWXMzWFljbzl3TFV6TFFlZkxucExSZXNrNzg0a0Nq?= =?utf-8?B?QzRTd2M5QkhENTRvMlB2cmJtZnBFNDRyelo3NWFocTg2WEFPald1Z3RxOXVI?= =?utf-8?B?bXdMc2lpVzNYczlocDJkS3RTeTg0R3JSZnU5blk3clA1bVAzU2FqbzRUNXNH?= =?utf-8?B?VlZOVkhuTGYyeFpJRGJJdTY3NzlHL0ZLSExTVE9EOURHT3NqeHZUbWFsV2tT?= =?utf-8?B?ZG5aRU9QUkhBN1N3eGJ0cmZZQjZTZjEyakUzQXVCVDhtNE5FOFZPb3RaYldL?= =?utf-8?B?K2FGNkwxZzZ0VnZ5UlVHVDRjSzFrZXNNZ3I2TnJScXpoWW9jOXQrUUVqWjJP?= =?utf-8?B?UDlMajd2RTB2UlRQOFl2TVZOdTF6aTFnMkhwN1RoaVdHcnNYWTVBVWtPV0Iw?= =?utf-8?B?UWdDbnpUYWNYa3cwUVZYSWRpODNoNWpvRUVnS2lIV1NUV1NDcXg4UWExbVRl?= =?utf-8?B?VkxTYlFmZURNNEdxL3JMQlFWSWVzTVhiN0paVmNvRGEvVTBRZXZEclhpWk1T?= =?utf-8?B?dng1dUlhZzV6Ny92VFhFNjZBa0l2OW1DejVQclZqY09qRFhZTldwaFZTdVdR?= =?utf-8?B?OEJXZUI2b2MraWZCZmxyMUdkdUU3ODFJMlRHNGVNaFZhUzErT09aUFM4ZmFU?= =?utf-8?B?VFRUTHVEVS9IWCtqWURPOFJ1WS9NOEZFc2d5N1hhaTF4VXhBYThvM0EzRGR3?= =?utf-8?B?SHZBQkh2VFUxMmxDcWRzRksrYjl2a1JHSlYyajdhQ1JOMmY2eUs1dDMxTXA3?= =?utf-8?B?bldOWEEzMGIwWkhGQmdacEZBVEkxeTVEdDNReEhBNXUzVHZURytEWVMwVmJ2?= =?utf-8?B?a3JCemhTeWpjZ1lhNUdjZWE3dHRyTmdTcHU4WGhEVCtNVFFKSUFXdURDYzY5?= =?utf-8?B?UXZjT08wSWYxTjZxN1NpUDdTc0huTngvMDdGekh2VEQ4N2xOck9rWUV0Nmc5?= =?utf-8?B?bFhldmdXSFJ4Y3VPZjJ2cDFTR3hQVGZUUkVwcnFRdEJNTE56aHBSbWVoQWtU?= =?utf-8?B?T055N2ZNS1B2dVJNM0JYamJZeWtRY09nbUdZL3dweVdCbHFDQm52Y21xZWxC?= =?utf-8?B?MGY0NFNRNU5LbkN1K293eThuTUFHUU9wMndwOTJTaDN3RHBIWnRrdlIxUm5K?= =?utf-8?B?MlFPMElJTDgrU29VaTNTdGdZbVJHNC9VSnhRMWVZd2o1eTIyc1JUY1k0Y3RD?= =?utf-8?B?THFrUEJoTGtmak5CRGVMK1F2NTArajhaOHh0QnRDcHV2Q0hYQit1dHBtdkpK?= =?utf-8?B?Umg1SWdFcks1U3UxSkpkWDczSEt5NUJSWSt4U3R0bnFvUzk3aU9pVi84UWND?= =?utf-8?B?Z095RFlFYU92aW1OMXBlWm56WmNBSmN2STlvV2hzS0U2YXRpVE0rSWdjN3h1?= =?utf-8?B?UVN4aG0rYWR2SVJacUk4Rm5oS0h3Tk5rZ3FaZW9XMHY3enVOdWp1d1hLVFVC?= =?utf-8?B?TDNMb3RIODVYaXR4emNuMVVqd2VLT2l1UEdRRTR3WUVtU3BSbjRKbzJ6ek1a?= =?utf-8?B?OWpaS0dQQ1FZeS9sSno4S04rQ3pRMlRWUVpyVkRSYzM2dEh2ZzhzdUZMeVFU?= =?utf-8?B?T054QWZmNFFrbml5SzRjeVpkNmRWK3A5OFZzb2UvWWJtaGsvY1k1N3krSisy?= =?utf-8?B?eGMvYWxyeGZ6T3NVajlHSzhXZkhmakdtZXVabFRGT2dCV1RlY0Y1azJGV293?= =?utf-8?B?ZE4rNDNNaWFaK2ZwZzBMVnBoSzFzZFNvVFdxc2F4cndOSzJyaHZ6YzIrRko5?= =?utf-8?B?MlJEVmFHdUk1QzZOQVNKT0w0MEZKYngzaEM5a3JrZHRxaW9rZklsRk1SaXdE?= =?utf-8?B?dUIrMHFOR2hlVFJIbnZCUkEvQnEzeC9hMnNwR0NCQUZmL3RnQmd1dVp6WWxJ?= =?utf-8?B?SW51Z3czd0dvSFcvSUlubmNRY2pPOHVhK21Qd0pnTEpzbzA3Tm9NbGpqWDlh?= =?utf-8?B?R0NEMEtmMmlJTXF6QVBSRkhZOXJMR1BzWXNEVnphZjZMWStjakMrV2lDc1pJ?= =?utf-8?Q?fYR24AwKSjoFy/RZ43usOyI=3D?= Content-Type: multipart/alternative; boundary="_000_YQXPR01MB4198334C48683A60E53E5D14BC652YQXPR01MB4198CANP_" List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR01MB4198.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: df91125d-b9ae-4acf-6f86-08dcd3f9b171 X-MS-Exchange-CrossTenant-originalarrivaltime: 13 Sep 2024 13:40:55.6800 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 7fe064f1-1f82-4006-b05f-62ea659f38b0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: FXNiDtQ9WDAmCKG3EQKr0MPaEFkBYns0JsJpm6vjO/cLig6/OJygt+IbNhBpBwWyLdzLjkFUklURMIz3WWWuPgR2POOxpMKjWOX/+Oiygz8= X-MS-Exchange-Transport-CrossTenantHeadersStamped: YT2PR01MB6388 X-OriginatorOrg: blackberry.com X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.60.29 definitions=2024-09-13_10,2024-09-13_02,2024-09-02_01 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:18705, ipnet:208.65.76.0/22, country:CA] X-Rspamd-Queue-Id: 4X4wSM4cSdz4SDW --_000_YQXPR01MB4198334C48683A60E53E5D14BC652YQXPR01MB4198CANP_ Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8" VGhhbmsgeW91IGZvciB0aGUgcmVzcG9uc2UgYW5kIGZvciBzaGFyaW5nIHlvdXIgc2NlbmFyaW8u DQoNCldl4oCZdmUgYWxzbyBoYWNrZWQgdXAgdGhlIGNnZW0gYW5kIHRoZSBmZmVjIGRyaXZlciB0 byBzdXBwb3J0IGEgc2hhcmVkIG1kaW8uDQpUaGF0IHdhcyBub3QgdG9vIGRpZmZpY3VsdCwgYnV0 IHdlIGhhdmUgYSBuZXcgc2NlbmFyaW8gd2hlcmUgdGhlIG1kaW8gaXMgbm93IGJlaW5nIHNoYXJl ZCBiZXR3ZWVuIHR3byBkaWZmZXJlbnQgZGV2aWNlcyB0aGF0IHVzZSBkaWZmZXJlbnQgZHJpdmVy cyAoZmZlYyBhbmQgZXFvcykuDQpUaGlzIHByZXNlbnRzIGEgZmV3IGV4dHJhIGNoYWxsZW5nZXMu DQoNCkkgd2FzIGhvcGluZyB0aGF0IEZyZWVCU0QgbWF5IGhhdmUgY29uc2lkZXJlZCBzdXBwb3J0 aW5nIGEgc2hhcmVkIG1kaW8uICBXZSBjYW4gY29tZSB1cCB3aXRoIHNvbWV0aGluZywgYnV0IGlm IHRoZXJlIGlzIGFuIGV4aXN0aW5nIGFyY2hpdGVjdHVyZS9hcHByb2FjaCBpbiB0aGUgd29ya3Pi gKZ3ZSB3b3VsZCBsaWtlIHRvIHVzZSBhIGNvbnNpc3RlbnQgYXBwcm9hY2guICBBdCBmaXJzdCBn bGFuY2UsIG1paXByb3h5IGRpZCBub3Qgc2VlbSBsaWtlIGEgZml0Lg0KDQpJIGRvIG5vdCBoYXZl IHRoZSBoYXJkd2FyZS4gIEkgYW0gdHJ5aW5nIHRvIGhlbHAgc29tZWJvZHkgZWxzZSB3aXRoIHRo aXMuICBJIGhhdmUgc2VlbiB0aGUgZHRiLg0KSXTigJlzIGEgVmFyaXNjaXRlIERBUi1NWDhNLVBM VVMuDQoNClJlZ2FyZHMsDQpNaWtlLg0KDQpGcm9tOiBvd25lci1mcmVlYnNkLW5ldEBGcmVlQlNE Lm9yZyA8b3duZXItZnJlZWJzZC1uZXRARnJlZUJTRC5vcmc+IG9uIGJlaGFsZiBvZiBNaWxhbiBP YnVjaCA8ZnJlZWJzZC1uZXRAZGluby5zaz4NCkRhdGU6IEZyaWRheSwgU2VwdGVtYmVyIDEzLCAy MDI0IGF0IDM6MDjigK9BTQ0KVG86IGZyZWVic2QtbmV0QGZyZWVic2Qub3JnIDxmcmVlYnNkLW5l dEBmcmVlYnNkLm9yZz4NClN1YmplY3Q6IFJlOiBFdGhlcm5ldCBkZXZpY2Ugd2l0aCBzaGFyZWQg bWRpbw0KQ0FVVElPTiAtIFRoaXMgZW1haWwgaXMgZnJvbSBhbiBleHRlcm5hbCBzb3VyY2UuIFBs ZWFzZSBiZSBjYXV0aW91cyB3aXRoIGxpbmtzIGFuZCBhdHRhY2htZW50cy4gKGdvL3RhZ2luZm8p DQoNCk9uIEZyaSwgNiBTZXAgMjAyNCAxODowMzozOSArMDAwMA0KTWlrZSBCZWxhbmdlciA8bWli ZWxhbmdlckBibGFja2JlcnJ5LmNvbT4gd3JvdGU6DQoNCj4gVGhlIGZvbGxvd2luZyBkZXZpY2Ug dHJlZSBzcGVjaWZpZXMgYSBzaGFyZWQgbWRpby4NCj4gVGhlIGZmZWMgZHJpdmVyIHVzZXMgbWlp YnVzLg0KPiBXaGVuIHRoZXJlIGlzIGEgc2hhcmVkIG1kaW8sIG9uZSBvZiB0aGUgZGV2aWNlIGlu c3RhbmNlcyB3aWxsIG5vdCBiZQ0KPiBhYmxlIHRvIHByb3Blcmx5IGNvbmZpZ3VyZSB0aGUgUEhZ LCBhcyBpdCBuZWVkcyB0byB1c2UgdGhlIG90aGVyDQo+IGRldmljZXMgcmVzb3VyY2UgdG8gcmVh ZC93cml0ZSB0aGUgUEhZLg0KPg0KPiAmZmVjMSB7cGluY3RybC1uYW1lcyA9ICJkZWZhdWx0IjsN Cj4gICAgICAgIHBpbmN0cmwtMCA9IDwmcGluY3RybF9mZWMxPjsNCj4gICAgICAgIHBoeS1tb2Rl ID0gInJnbWlpLWlkIjsNCj4gICAgICAgIHBoeS1oYW5kbGUgPSA8JmV0aHBoeTA+Ow0KPiAgICAg ICAgZnNsLG1hZ2ljLXBhY2tldDsNCj4gICAgICAgIHN0YXR1cyA9ICJva2F5IjsNCj4NCj4gICAg ICAgIG1kaW8gew0KPiAgICAgICAgICAgICAgI2FkZHJlc3MtY2VsbHMgPSA8MT47DQo+ICAgICAg ICAgICAgICAjc2l6ZS1jZWxscyA9IDwwPjsNCj4NCj4gICAgICAgICAgICAgIGV0aHBoeTA6IGV0 aGVybmV0LXBoeUAwIHsNCj4gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBj b21wYXRpYmxlID0gImV0aGVybmV0LXBoeS1pZWVlODAyLjMtYzIyIjsgcmVnID0gPDA+Ow0KPiAg ICAgICAgICAgICAgfTsNCj4NCj4gICAgICAgICAgICAgIGV0aHBoeTE6IGV0aGVybmV0LXBoeUAx IHsNCj4gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjb21wYXRpYmxlID0g ImV0aGVybmV0LXBoeS1pZWVlODAyLjMtYzIyIjsgcmVnID0gPDE+Ow0KPiAgICAgICAgICAgICAg fTsNCj4gICAgICAgIH07DQo+IH07DQo+DQo+ICZmZWMyIHsNCj4gICAgICAgIHBpbmN0cmwtbmFt ZXMgPSAiZGVmYXVsdCI7DQo+ICAgICAgICBwaW5jdHJsLTAgPSA8JnBpbmN0cmxfZmVjMj47DQo+ ICAgICAgICBwaHktbW9kZSA9ICJyZ21paS10eGlkIjsNCj4gICAgICAgIHBoeS1oYW5kbGUgPSA8 JmV0aHBoeTE+Ow0KPiAgICAgICAgcGh5LXN1cHBseSA9IDwmcmVnX2ZlYzJfc3VwcGx5PjsNCj4g ICAgICAgIG52bWVtLWNlbGxzID0gPCZmZWNfbWFjMT47DQo+ICAgICAgICBudm1lbS1jZWxsLW5h bWVzID0gIm1hYy1hZGRyZXNzIjsNCj4gICAgICAgIHJ4LWludGVybmFsLWRlbGF5LXBzID0gPDIw MDA+Ow0KPiAgICAgICAgZnNsLG1hZ2ljLXBhY2tldDsNCj4gICAgICAgIHN0YXR1cyA9ICJva2F5 IjsNCj4gfTsNCj4NCj4gRG9lcyBGcmVlQlNEIGhhdmUgYW55IHBsYW5zIGZvciBzdXBwb3J0aW5n IGhhcmR3YXJlIHRoYXQgc3BlY2lmaWVzIGENCj4gc2hhcmVkIG1kaW8gaW4gdGhlIGR0Yj8gSnVz dCBrbm93aW5nIHRoZSBnZW5lcmFsIGFwcHJvYWNoIGJlaW5nDQo+IGNvbnNpZGVyZWQgd291bGQg YmUgaGVscGZ1bC4NCj4NCg0KSSBjYW4ndCBzcGVhayBmb3IgRnJlZUJTRCBwcm9qZWN0LCBJIGp1 c3QgY2FuIHNoYXJlIG15IGV4cGVyaWVuY2Ugd2l0aA0Kc2ltaWxhciBjYXNlLiBJdCBpcyBkZXNj cmliZWQgaW4gbXkgcG9zdCB0byBoYWNrZXJzIG1haWxpbmcgbGlzdCAoc2VlDQpodHRwczovL3Vy bGRlZmVuc2UuY29tL3YzL19faHR0cHM6Ly9saXN0cy5mcmVlYnNkLm9yZy9hcmNoaXZlcy9mcmVl YnNkLWhhY2tlcnMvMjAyMS1EZWNlbWJlci8wMDA2NDkuaHRtbF9fOyEhSm9lVy1JaENVa1MwSmch ZnYwREhGTjVYYjRGYkt3cmUxSDRVRENVdmJtaEFvTzF5NUhnUWlEa042d3V2MnQzQjRweVMxYWt1 S3VDbjZacU8xQWZickNhRnNWc0ppYmRmdWk0S2ZKUUd3JDxodHRwczovL3VybGRlZmVuc2UuY29t L3YzL19faHR0cHM6L2xpc3RzLmZyZWVic2Qub3JnL2FyY2hpdmVzL2ZyZWVic2QtaGFja2Vycy8y MDIxLURlY2VtYmVyLzAwMDY0OS5odG1sX187ISFKb2VXLUloQ1VrUzBKZyFmdjBESEZONVhiNEZi S3dyZTFINFVEQ1V2Ym1oQW9PMXk1SGdRaURrTjZ3dXYydDNCNHB5UzFha3VLdUNuNlpxTzFBZmJy Q2FGc1ZzSmliZGZ1aTRLZkpRR3ckPg0KZm9yIGRldGFpbHMpLCB1bmZvcnR1bmF0ZWx5LCBubyBy ZXNwb25zZSByZWNlaXZlZC4gQW5vdGhlciBhdHRlbXB0IHRvDQpnZXQgc29tZSBhdHRlbnRpb24g YSB3ZWVrIGxhdGVyIG9uIG5ldCBtYWlsaW5nIGxpc3Qgd2FzIGRvbmUsIHNlZQ0KaHR0cHM6Ly91 cmxkZWZlbnNlLmNvbS92My9fX2h0dHBzOi8vbGlzdHMuZnJlZWJzZC5vcmcvYXJjaGl2ZXMvZnJl ZWJzZC1uZXQvMjAyMS1EZWNlbWJlci8wMDExMTQuaHRtbF9fOyEhSm9lVy1JaENVa1MwSmchZnYw REhGTjVYYjRGYkt3cmUxSDRVRENVdmJtaEFvTzF5NUhnUWlEa042d3V2MnQzQjRweVMxYWt1S3VD bjZacU8xQWZickNhRnNWc0ppYmRmdWpLTjNfeENBJDxodHRwczovL3VybGRlZmVuc2UuY29tL3Yz L19faHR0cHM6L2xpc3RzLmZyZWVic2Qub3JnL2FyY2hpdmVzL2ZyZWVic2QtbmV0LzIwMjEtRGVj ZW1iZXIvMDAxMTE0Lmh0bWxfXzshIUpvZVctSWhDVWtTMEpnIWZ2MERIRk41WGI0RmJLd3JlMUg0 VURDVXZibWhBb08xeTVIZ1FpRGtONnd1djJ0M0I0cHlTMWFrdUt1Q242WnFPMUFmYnJDYUZzVnNK aWJkZnVqS04zX3hDQSQ+DQpmb3IgdGhlIHBvc3QsIHdpdGggbm8gcmVzcG9uc2UgZWl0aGVyLg0K DQpBcyB5b3Ugc2VlLCBteSBjYXNlIHdhcyBzaW1pbGFyLCBqdXN0IHRoZSBtZGlvIGJsb2NrIHdh cyBhdHRhY2hlZCB0bw0Kc2Vjb25kIGNvbnRyb2xsZXIuIFRoaXMgbWFrZXMgaXQgYSBiaXQgbW9y ZSBwcm9ibGVtYXRpYyAtIHlvdSBjYW4ndCB1c2UNCm1kaW8gY29udHJvbGxlciBiZWZvcmUgYmVp bmcgaW5pdGlhbGl6ZWQsIG5hdHVyYWxseS4NCg0KSSB3YXMgbm90IGFibGUgdG8gdXNlIG1paXBy b3h5IGFwcHJvYWNoIGFzIG5vdGVkIGluIG15IHBvc3QgdG8gaGFja2Vycw0KbWFpbGluZyBsaXN0 LCBhZGRpdGlvbmFsbHksIG1paXByb3h5IHdhcyByZW1vdmVkIGZyb20gdGhlIHRyZWUgd2l0aA0K TUlQUyBhcmNoIHNvbWUgdGltZSBsYXRlci4gSSByZXNvbHZlZCB0aGUgaXNzdWUgYnkgbW9kaWZ5 aW5nIGNnZW0gZHJpdmVyDQphbmQgbWlpIGxheWVyLiBUaGlzIHdhcyBqdXN0IGEgcHJvb2Ygb2Yg Y29uY2VwdCB3aXRoIHNvbWUgaGFja3MsIGJ1dCBJDQp3YXMgYWJsZSB0byB1c2UgYm90aCBwb3J0 cyB3aXRoIHByb3BlciBsaW5rIHN0YXRlIGNoYW5nZSBkZXRlY3Rpb24uIEkNCmRpZCBub3QgY29u dGludWUgdGhlIHdvcmsgYmVjYXVzZSB2ZW5kb3IgY2hhbmdlZCBoYXJkd2FyZSBkZXNpZ24gYW5k DQp0aGVyZSB3YXMgbm8gc2hhcmVkIG1kaW8gYW55bW9yZS4NCg0KSWYgeW91IGFyZSBpbnRlcmVz dGVkIEkgY2FuIGRpZyBmb3IgdGhlIHNvdXJjZXMsIGJpZyBwYXJ0IG9mIG15IGNoYW5nZXMNCndv dWxkIG5vdCBiZSBuZWNlc3NhcnksIGp1c3QgdGhlIGlkZWEgb2YgZGVjb3VwbGluZyBNRElPIGFu ZCBNSUkNCmludGVyZmFjZXMgc3RpbGwgYXBwbGllcywgSSB0aGluay4gQnkgdGhlIHdheSwgd2hp Y2ggYm9hcmQgYXJlIHlvdQ0Kd29ya2luZyBvbj8gSXMgaXQgYWNjZXNzaWJsZSBmb3IgZ2VuZXJh bCBhdWRpZW5jZT8NCg0KUmVnYXJkcywNCk1pbGFuDQoKLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQpUaGlzIHRyYW5z bWlzc2lvbiAoaW5jbHVkaW5nIGFueSBhdHRhY2htZW50cykgbWF5IGNvbnRhaW4gY29uZmlkZW50 aWFsIGluZm9ybWF0aW9uLCBwcml2aWxlZ2VkIG1hdGVyaWFsIChpbmNsdWRpbmcgbWF0ZXJpYWwg cHJvdGVjdGVkIGJ5IHRoZSBzb2xpY2l0b3ItY2xpZW50IG9yIG90aGVyIGFwcGxpY2FibGUgcHJp dmlsZWdlcyksIG9yIGNvbnN0aXR1dGUgbm9uLXB1YmxpYyBpbmZvcm1hdGlvbi4gQW55IHVzZSBv ZiB0aGlzIGluZm9ybWF0aW9uIGJ5IGFueW9uZSBvdGhlciB0aGFuIHRoZSBpbnRlbmRlZCByZWNp cGllbnQgaXMgcHJvaGliaXRlZC4gSWYgeW91IGhhdmUgcmVjZWl2ZWQgdGhpcyB0cmFuc21pc3Np b24gaW4gZXJyb3IsIHBsZWFzZSBpbW1lZGlhdGVseSByZXBseSB0byB0aGUgc2VuZGVyIGFuZCBk ZWxldGUgdGhpcyBpbmZvcm1hdGlvbiBmcm9tIHlvdXIgc3lzdGVtLiBVc2UsIGRpc3NlbWluYXRp b24sIGRpc3RyaWJ1dGlvbiwgb3IgcmVwcm9kdWN0aW9uIG9mIHRoaXMgdHJhbnNtaXNzaW9uIGJ5 IHVuaW50ZW5kZWQgcmVjaXBpZW50cyBpcyBub3QgYXV0aG9yaXplZCBhbmQgbWF5IGJlIHVubGF3 ZnVsLgo= --_000_YQXPR01MB4198334C48683A60E53E5D14BC652YQXPR01MB4198CANP_ Content-Transfer-Encoding: base64 Content-Type: text/html; charset="utf-8" PGh0bWwgeG1sbnM6bz0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6b2ZmaWNlIiB4 bWxuczp3PSJ1cm46c2NoZW1hcy1taWNyb3NvZnQtY29tOm9mZmljZTp3b3JkIiB4bWxuczptPSJo dHRwOi8vc2NoZW1hcy5taWNyb3NvZnQuY29tL29mZmljZS8yMDA0LzEyL29tbWwiIHhtbG5zPSJo dHRwOi8vd3d3LnczLm9yZy9UUi9SRUMtaHRtbDQwIj4NCjxoZWFkPg0KPG1ldGEgaHR0cC1lcXVp dj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPg0KPG1l dGEgbmFtZT0iR2VuZXJhdG9yIiBjb250ZW50PSJNaWNyb3NvZnQgV29yZCAxNSAoZmlsdGVyZWQg bWVkaXVtKSI+DQo8c3R5bGU+PCEtLQ0KLyogRm9udCBEZWZpbml0aW9ucyAqLw0KQGZvbnQtZmFj ZQ0KCXtmb250LWZhbWlseToiQ2FtYnJpYSBNYXRoIjsNCglwYW5vc2UtMToyIDQgNSAzIDUgNCA2 IDMgMiA0O30NCkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6QXB0b3M7DQoJcGFub3NlLTE6MiAx MSAwIDQgMiAyIDIgMiAyIDQ7fQ0KLyogU3R5bGUgRGVmaW5pdGlvbnMgKi8NCnAuTXNvTm9ybWFs LCBsaS5Nc29Ob3JtYWwsIGRpdi5Nc29Ob3JtYWwNCgl7bWFyZ2luOjBjbTsNCglmb250LXNpemU6 MTAuMHB0Ow0KCWZvbnQtZmFtaWx5OiJBcHRvcyIsc2Fucy1zZXJpZjt9DQphOmxpbmssIHNwYW4u TXNvSHlwZXJsaW5rDQoJe21zby1zdHlsZS1wcmlvcml0eTo5OTsNCgljb2xvcjpibHVlOw0KCXRl eHQtZGVjb3JhdGlvbjp1bmRlcmxpbmU7fQ0Kc3Bhbi5FbWFpbFN0eWxlMTkNCgl7bXNvLXN0eWxl LXR5cGU6cGVyc29uYWwtcmVwbHk7DQoJZm9udC1mYW1pbHk6IkFwdG9zIixzYW5zLXNlcmlmOw0K CWNvbG9yOndpbmRvd3RleHQ7fQ0KLk1zb0NocERlZmF1bHQNCgl7bXNvLXN0eWxlLXR5cGU6ZXhw b3J0LW9ubHk7DQoJZm9udC1zaXplOjEwLjBwdDsNCgltc28tbGlnYXR1cmVzOm5vbmU7fQ0KQHBh Z2UgV29yZFNlY3Rpb24xDQoJe3NpemU6NjEyLjBwdCA3OTIuMHB0Ow0KCW1hcmdpbjo3Mi4wcHQg NzIuMHB0IDcyLjBwdCA3Mi4wcHQ7fQ0KZGl2LldvcmRTZWN0aW9uMQ0KCXtwYWdlOldvcmRTZWN0 aW9uMTt9DQotLT48L3N0eWxlPg0KPC9oZWFkPg0KPGJvZHkgbGFuZz0iRU4tQ0EiIGxpbms9ImJs dWUiIHZsaW5rPSJwdXJwbGUiIHN0eWxlPSJ3b3JkLXdyYXA6YnJlYWstd29yZCI+DQo8ZGl2IGNs YXNzPSJXb3JkU2VjdGlvbjEiPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZv bnQtc2l6ZToxMS4wcHQiPlRoYW5rIHlvdSBmb3IgdGhlIHJlc3BvbnNlIGFuZCBmb3Igc2hhcmlu ZyB5b3VyIHNjZW5hcmlvLjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3Jt YWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0Ij48bzpwPiZuYnNwOzwvbzpwPjwvc3Bh bj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBw dCI+V2XigJl2ZSBhbHNvIGhhY2tlZCB1cCB0aGUgY2dlbSBhbmQgdGhlIGZmZWMgZHJpdmVyIHRv IHN1cHBvcnQgYSBzaGFyZWQgbWRpby48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0i TXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdCI+VGhhdCB3YXMgbm90IHRv byBkaWZmaWN1bHQsIGJ1dCB3ZSBoYXZlIGEgbmV3IHNjZW5hcmlvIHdoZXJlIHRoZSBtZGlvIGlz IG5vdyBiZWluZyBzaGFyZWQgYmV0d2VlbiB0d28gZGlmZmVyZW50IGRldmljZXMgdGhhdCB1c2Ug ZGlmZmVyZW50IGRyaXZlcnMgKGZmZWMgYW5kIGVxb3MpLjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4N CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0Ij5UaGlz IHByZXNlbnRzIGEgZmV3IGV4dHJhIGNoYWxsZW5nZXMuPG86cD48L286cD48L3NwYW4+PC9wPg0K PHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQiPjxvOnA+ Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxl PSJmb250LXNpemU6MTEuMHB0Ij5JIHdhcyBob3BpbmcgdGhhdCBGcmVlQlNEIG1heSBoYXZlIGNv bnNpZGVyZWQgc3VwcG9ydGluZyBhIHNoYXJlZCBtZGlvLiZuYnNwOyBXZSBjYW4gY29tZSB1cCB3 aXRoIHNvbWV0aGluZywgYnV0IGlmIHRoZXJlIGlzIGFuIGV4aXN0aW5nIGFyY2hpdGVjdHVyZS9h cHByb2FjaCBpbiB0aGUgd29ya3PigKZ3ZSB3b3VsZCBsaWtlIHRvIHVzZSBhIGNvbnNpc3RlbnQg YXBwcm9hY2guJm5ic3A7DQogQXQgZmlyc3QgZ2xhbmNlLCBtaWlwcm94eSBkaWQgbm90IHNlZW0g bGlrZSBhIGZpdC48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48 c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdCI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9w Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQiPkkg ZG8gbm90IGhhdmUgdGhlIGhhcmR3YXJlLiZuYnNwOyBJIGFtIHRyeWluZyB0byBoZWxwIHNvbWVi b2R5IGVsc2Ugd2l0aCB0aGlzLiZuYnNwOyBJIGhhdmUgc2VlbiB0aGUgZHRiLjxvOnA+PC9vOnA+ PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6 MTEuMHB0Ij5JdOKAmXMgYSBWYXJpc2NpdGUgREFSLU1YOE0tUExVUy48bzpwPjwvbzpwPjwvc3Bh bj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBw dCI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNw YW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQiPlJlZ2FyZHMsPG86cD48L286cD48L3NwYW4+PC9w Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQiPk1p a2UuIDxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0 eWxlPSJmb250LXNpemU6MTEuMHB0Ij48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8ZGl2 IGlkPSJtYWlsLWVkaXRvci1yZWZlcmVuY2UtbWVzc2FnZS1jb250YWluZXIiPg0KPGRpdj4NCjxk aXY+DQo8ZGl2IHN0eWxlPSJib3JkZXI6bm9uZTtib3JkZXItdG9wOnNvbGlkICNCNUM0REYgMS4w cHQ7cGFkZGluZzozLjBwdCAwY20gMGNtIDBjbSI+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHls ZT0ibWFyZ2luLWJvdHRvbToxMi4wcHQiPjxiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTIuMHB0 O2NvbG9yOmJsYWNrIj5Gcm9tOg0KPC9zcGFuPjwvYj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEy LjBwdDtjb2xvcjpibGFjayI+b3duZXItZnJlZWJzZC1uZXRARnJlZUJTRC5vcmcgJmx0O293bmVy LWZyZWVic2QtbmV0QEZyZWVCU0Qub3JnJmd0OyBvbiBiZWhhbGYgb2YgTWlsYW4gT2J1Y2ggJmx0 O2ZyZWVic2QtbmV0QGRpbm8uc2smZ3Q7PGJyPg0KPGI+RGF0ZTogPC9iPkZyaWRheSwgU2VwdGVt YmVyIDEzLCAyMDI0IGF0IDM6MDg8L3NwYW4+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMi4wcHQ7 Zm9udC1mYW1pbHk6JnF1b3Q7QXJpYWwmcXVvdDssc2Fucy1zZXJpZjtjb2xvcjpibGFjayI+4oCv PC9zcGFuPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTIuMHB0O2NvbG9yOmJsYWNrIj5BTTxicj4N CjxiPlRvOiA8L2I+ZnJlZWJzZC1uZXRAZnJlZWJzZC5vcmcgJmx0O2ZyZWVic2QtbmV0QGZyZWVi c2Qub3JnJmd0Ozxicj4NCjxiPlN1YmplY3Q6IDwvYj5SZTogRXRoZXJuZXQgZGV2aWNlIHdpdGgg c2hhcmVkIG1kaW88bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFz cz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWJvdHRvbToxMi4wcHQiPjxzcGFuIHN0eWxlPSJm b250LXNpemU6MTEuMHB0Ij5DQVVUSU9OIC0gVGhpcyBlbWFpbCBpcyBmcm9tIGFuIGV4dGVybmFs IHNvdXJjZS4gUGxlYXNlIGJlIGNhdXRpb3VzIHdpdGggbGlua3MgYW5kIGF0dGFjaG1lbnRzLiAo Z28vdGFnaW5mbyk8YnI+DQo8YnI+DQpPbiBGcmksIDYgU2VwIDIwMjQgMTg6MDM6MzkgKzAwMDA8 YnI+DQpNaWtlIEJlbGFuZ2VyICZsdDttaWJlbGFuZ2VyQGJsYWNrYmVycnkuY29tJmd0OyB3cm90 ZTo8YnI+DQo8YnI+DQomZ3Q7IFRoZSBmb2xsb3dpbmcgZGV2aWNlIHRyZWUgc3BlY2lmaWVzIGEg c2hhcmVkIG1kaW8uPGJyPg0KJmd0OyBUaGUgZmZlYyBkcml2ZXIgdXNlcyBtaWlidXMuPGJyPg0K Jmd0OyBXaGVuIHRoZXJlIGlzIGEgc2hhcmVkIG1kaW8sIG9uZSBvZiB0aGUgZGV2aWNlIGluc3Rh bmNlcyB3aWxsIG5vdCBiZTxicj4NCiZndDsgYWJsZSB0byBwcm9wZXJseSBjb25maWd1cmUgdGhl IFBIWSwgYXMgaXQgbmVlZHMgdG8gdXNlIHRoZSBvdGhlcjxicj4NCiZndDsgZGV2aWNlcyByZXNv dXJjZSB0byByZWFkL3dyaXRlIHRoZSBQSFkuPGJyPg0KJmd0OyA8YnI+DQomZ3Q7ICZhbXA7ZmVj MSB7cGluY3RybC1uYW1lcyA9ICZxdW90O2RlZmF1bHQmcXVvdDs7PGJyPg0KJmd0OyZuYnNwOyZu YnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyBwaW5jdHJsLTAgPSAmbHQ7JmFtcDtw aW5jdHJsX2ZlYzEmZ3Q7Ozxicj4NCiZndDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm bmJzcDsmbmJzcDsgcGh5LW1vZGUgPSAmcXVvdDtyZ21paS1pZCZxdW90Ozs8YnI+DQomZ3Q7Jm5i c3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7IHBoeS1oYW5kbGUgPSAmbHQ7 JmFtcDtldGhwaHkwJmd0Ozs8YnI+DQomZ3Q7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7 Jm5ic3A7Jm5ic3A7IGZzbCxtYWdpYy1wYWNrZXQ7PGJyPg0KJmd0OyZuYnNwOyZuYnNwOyZuYnNw OyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyBzdGF0dXMgPSAmcXVvdDtva2F5JnF1b3Q7Ozxicj4N CiZndDsgPGJyPg0KJmd0OyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNw OyBtZGlvIHs8YnI+DQomZ3Q7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i c3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7ICNhZGRyZXNzLWNlbGxzID0g Jmx0OzEmZ3Q7Ozxicj4NCiZndDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsgI3NpemUtY2VsbHMgPSAm bHQ7MCZndDs7PGJyPg0KJmd0OyA8YnI+DQomZ3Q7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i c3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7IGV0aHBo eTA6IGV0aGVybmV0LXBoeUAwIHs8YnI+DQomZ3Q7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i c3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7 Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i c3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7 Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7IGNvbXBhdGlibGUgPSAmcXVvdDtldGhlcm5l dC1waHktaWVlZTgwMi4zLWMyMiZxdW90OzsgcmVnID0gJmx0OzAmZ3Q7Ozxicj4NCiZndDsmbmJz cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm bmJzcDsmbmJzcDsmbmJzcDsgfTs8YnI+DQomZ3Q7IDxicj4NCiZndDsmbmJzcDsmbmJzcDsmbmJz cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm bmJzcDsgZXRocGh5MTogZXRoZXJuZXQtcGh5QDEgezxicj4NCiZndDsmbmJzcDsmbmJzcDsmbmJz cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsgY29tcGF0aWJsZSA9ICZx dW90O2V0aGVybmV0LXBoeS1pZWVlODAyLjMtYzIyJnF1b3Q7OyByZWcgPSAmbHQ7MSZndDs7PGJy Pg0KJmd0OyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZu YnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyB9Ozxicj4NCiZndDsmbmJzcDsmbmJzcDsmbmJz cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsgfTs8YnI+DQomZ3Q7IH07PGJyPg0KJmd0OyA8YnI+ DQomZ3Q7ICZhbXA7ZmVjMiB7PGJyPg0KJmd0OyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNw OyZuYnNwOyZuYnNwOyBwaW5jdHJsLW5hbWVzID0gJnF1b3Q7ZGVmYXVsdCZxdW90Ozs8YnI+DQom Z3Q7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7IHBpbmN0cmwtMCA9 ICZsdDsmYW1wO3BpbmN0cmxfZmVjMiZndDs7PGJyPg0KJmd0OyZuYnNwOyZuYnNwOyZuYnNwOyZu YnNwOyZuYnNwOyZuYnNwOyZuYnNwOyBwaHktbW9kZSA9ICZxdW90O3JnbWlpLXR4aWQmcXVvdDs7 PGJyPg0KJmd0OyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyBwaHkt aGFuZGxlID0gJmx0OyZhbXA7ZXRocGh5MSZndDs7PGJyPg0KJmd0OyZuYnNwOyZuYnNwOyZuYnNw OyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyBwaHktc3VwcGx5ID0gJmx0OyZhbXA7cmVnX2ZlYzJf c3VwcGx5Jmd0Ozs8YnI+DQomZ3Q7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7 Jm5ic3A7IG52bWVtLWNlbGxzID0gJmx0OyZhbXA7ZmVjX21hYzEmZ3Q7Ozxicj4NCiZndDsmbmJz cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsgbnZtZW0tY2VsbC1uYW1lcyA9 ICZxdW90O21hYy1hZGRyZXNzJnF1b3Q7Ozxicj4NCiZndDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz cDsmbmJzcDsmbmJzcDsmbmJzcDsgcngtaW50ZXJuYWwtZGVsYXktcHMgPSAmbHQ7MjAwMCZndDs7 PGJyPg0KJmd0OyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyBmc2ws bWFnaWMtcGFja2V0Ozxicj4NCiZndDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz cDsmbmJzcDsgc3RhdHVzID0gJnF1b3Q7b2theSZxdW90Ozs8YnI+DQomZ3Q7IH07PGJyPg0KJmd0 OyA8YnI+DQomZ3Q7IERvZXMgRnJlZUJTRCBoYXZlIGFueSBwbGFucyBmb3Igc3VwcG9ydGluZyBo YXJkd2FyZSB0aGF0IHNwZWNpZmllcyBhPGJyPg0KJmd0OyBzaGFyZWQgbWRpbyBpbiB0aGUgZHRi PyBKdXN0IGtub3dpbmcgdGhlIGdlbmVyYWwgYXBwcm9hY2ggYmVpbmc8YnI+DQomZ3Q7IGNvbnNp ZGVyZWQgd291bGQgYmUgaGVscGZ1bC48YnI+DQomZ3Q7IDxicj4NCjxicj4NCkkgY2FuJ3Qgc3Bl YWsgZm9yIEZyZWVCU0QgcHJvamVjdCwgSSBqdXN0IGNhbiBzaGFyZSBteSBleHBlcmllbmNlIHdp dGg8YnI+DQpzaW1pbGFyIGNhc2UuIEl0IGlzIGRlc2NyaWJlZCBpbiBteSBwb3N0IHRvIGhhY2tl cnMgbWFpbGluZyBsaXN0IChzZWU8YnI+DQo8YSBocmVmPSJodHRwczovL3VybGRlZmVuc2UuY29t L3YzL19faHR0cHM6L2xpc3RzLmZyZWVic2Qub3JnL2FyY2hpdmVzL2ZyZWVic2QtaGFja2Vycy8y MDIxLURlY2VtYmVyLzAwMDY0OS5odG1sX187ISFKb2VXLUloQ1VrUzBKZyFmdjBESEZONVhiNEZi S3dyZTFINFVEQ1V2Ym1oQW9PMXk1SGdRaURrTjZ3dXYydDNCNHB5UzFha3VLdUNuNlpxTzFBZmJy Q2FGc1ZzSmliZGZ1aTRLZkpRR3ckIj5odHRwczovL3VybGRlZmVuc2UuY29tL3YzL19faHR0cHM6 Ly9saXN0cy5mcmVlYnNkLm9yZy9hcmNoaXZlcy9mcmVlYnNkLWhhY2tlcnMvMjAyMS1EZWNlbWJl ci8wMDA2NDkuaHRtbF9fOyEhSm9lVy1JaENVa1MwSmchZnYwREhGTjVYYjRGYkt3cmUxSDRVRENV dmJtaEFvTzF5NUhnUWlEa042d3V2MnQzQjRweVMxYWt1S3VDbjZacU8xQWZickNhRnNWc0ppYmRm dWk0S2ZKUUd3JDwvYT4NCjxicj4NCmZvciBkZXRhaWxzKSwgdW5mb3J0dW5hdGVseSwgbm8gcmVz cG9uc2UgcmVjZWl2ZWQuIEFub3RoZXIgYXR0ZW1wdCB0bzxicj4NCmdldCBzb21lIGF0dGVudGlv biBhIHdlZWsgbGF0ZXIgb24gbmV0IG1haWxpbmcgbGlzdCB3YXMgZG9uZSwgc2VlPGJyPg0KPGEg aHJlZj0iaHR0cHM6Ly91cmxkZWZlbnNlLmNvbS92My9fX2h0dHBzOi9saXN0cy5mcmVlYnNkLm9y Zy9hcmNoaXZlcy9mcmVlYnNkLW5ldC8yMDIxLURlY2VtYmVyLzAwMTExNC5odG1sX187ISFKb2VX LUloQ1VrUzBKZyFmdjBESEZONVhiNEZiS3dyZTFINFVEQ1V2Ym1oQW9PMXk1SGdRaURrTjZ3dXYy dDNCNHB5UzFha3VLdUNuNlpxTzFBZmJyQ2FGc1ZzSmliZGZ1aktOM194Q0EkIj5odHRwczovL3Vy bGRlZmVuc2UuY29tL3YzL19faHR0cHM6Ly9saXN0cy5mcmVlYnNkLm9yZy9hcmNoaXZlcy9mcmVl YnNkLW5ldC8yMDIxLURlY2VtYmVyLzAwMTExNC5odG1sX187ISFKb2VXLUloQ1VrUzBKZyFmdjBE SEZONVhiNEZiS3dyZTFINFVEQ1V2Ym1oQW9PMXk1SGdRaURrTjZ3dXYydDNCNHB5UzFha3VLdUNu NlpxTzFBZmJyQ2FGc1ZzSmliZGZ1aktOM194Q0EkPC9hPg0KPGJyPg0KZm9yIHRoZSBwb3N0LCB3 aXRoIG5vIHJlc3BvbnNlIGVpdGhlci48YnI+DQo8YnI+DQpBcyB5b3Ugc2VlLCBteSBjYXNlIHdh cyBzaW1pbGFyLCBqdXN0IHRoZSBtZGlvIGJsb2NrIHdhcyBhdHRhY2hlZCB0bzxicj4NCnNlY29u ZCBjb250cm9sbGVyLiBUaGlzIG1ha2VzIGl0IGEgYml0IG1vcmUgcHJvYmxlbWF0aWMgLSB5b3Ug Y2FuJ3QgdXNlPGJyPg0KbWRpbyBjb250cm9sbGVyIGJlZm9yZSBiZWluZyBpbml0aWFsaXplZCwg bmF0dXJhbGx5Ljxicj4NCjxicj4NCkkgd2FzIG5vdCBhYmxlIHRvIHVzZSBtaWlwcm94eSBhcHBy b2FjaCBhcyBub3RlZCBpbiBteSBwb3N0IHRvIGhhY2tlcnM8YnI+DQptYWlsaW5nIGxpc3QsIGFk ZGl0aW9uYWxseSwgbWlpcHJveHkgd2FzIHJlbW92ZWQgZnJvbSB0aGUgdHJlZSB3aXRoPGJyPg0K TUlQUyBhcmNoIHNvbWUgdGltZSBsYXRlci4gSSByZXNvbHZlZCB0aGUgaXNzdWUgYnkgbW9kaWZ5 aW5nIGNnZW0gZHJpdmVyPGJyPg0KYW5kIG1paSBsYXllci4gVGhpcyB3YXMganVzdCBhIHByb29m IG9mIGNvbmNlcHQgd2l0aCBzb21lIGhhY2tzLCBidXQgSTxicj4NCndhcyBhYmxlIHRvIHVzZSBi b3RoIHBvcnRzIHdpdGggcHJvcGVyIGxpbmsgc3RhdGUgY2hhbmdlIGRldGVjdGlvbi4gSTxicj4N CmRpZCBub3QgY29udGludWUgdGhlIHdvcmsgYmVjYXVzZSB2ZW5kb3IgY2hhbmdlZCBoYXJkd2Fy ZSBkZXNpZ24gYW5kPGJyPg0KdGhlcmUgd2FzIG5vIHNoYXJlZCBtZGlvIGFueW1vcmUuPGJyPg0K PGJyPg0KSWYgeW91IGFyZSBpbnRlcmVzdGVkIEkgY2FuIGRpZyBmb3IgdGhlIHNvdXJjZXMsIGJp ZyBwYXJ0IG9mIG15IGNoYW5nZXM8YnI+DQp3b3VsZCBub3QgYmUgbmVjZXNzYXJ5LCBqdXN0IHRo ZSBpZGVhIG9mIGRlY291cGxpbmcgTURJTyBhbmQgTUlJPGJyPg0KaW50ZXJmYWNlcyBzdGlsbCBh cHBsaWVzLCBJIHRoaW5rLiBCeSB0aGUgd2F5LCB3aGljaCBib2FyZCBhcmUgeW91PGJyPg0Kd29y a2luZyBvbj8gSXMgaXQgYWNjZXNzaWJsZSBmb3IgZ2VuZXJhbCBhdWRpZW5jZT88YnI+DQo8YnI+ DQpSZWdhcmRzLDxicj4NCk1pbGFuPG86cD48L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8L2Rp dj4NCjwvZGl2Pg0KPC9kaXY+DQo8L2Rpdj4NCgo8SFI+VGhpcyB0cmFuc21pc3Npb24gKGluY2x1 ZGluZyBhbnkgYXR0YWNobWVudHMpIG1heSBjb250YWluIGNvbmZpZGVudGlhbCBpbmZvcm1hdGlv biwgcHJpdmlsZWdlZCBtYXRlcmlhbCAoaW5jbHVkaW5nIG1hdGVyaWFsIHByb3RlY3RlZCBieSB0 aGUgc29saWNpdG9yLWNsaWVudCBvciBvdGhlciBhcHBsaWNhYmxlIHByaXZpbGVnZXMpLCBvciBj b25zdGl0dXRlIG5vbi1wdWJsaWMgaW5mb3JtYXRpb24uIEFueSB1c2Ugb2YgdGhpcyBpbmZvcm1h dGlvbiBieSBhbnlvbmUgb3RoZXIgdGhhbiB0aGUgaW50ZW5kZWQgcmVjaXBpZW50IGlzIHByb2hp Yml0ZWQuIElmIHlvdSBoYXZlIHJlY2VpdmVkIHRoaXMgdHJhbnNtaXNzaW9uIGluIGVycm9yLCBw bGVhc2UgaW1tZWRpYXRlbHkgcmVwbHkgdG8gdGhlIHNlbmRlciBhbmQgZGVsZXRlIHRoaXMgaW5m b3JtYXRpb24gZnJvbSB5b3VyIHN5c3RlbS4gVXNlLCBkaXNzZW1pbmF0aW9uLCBkaXN0cmlidXRp b24sIG9yIHJlcHJvZHVjdGlvbiBvZiB0aGlzIHRyYW5zbWlzc2lvbiBieSB1bmludGVuZGVkIHJl Y2lwaWVudHMgaXMgbm90IGF1dGhvcml6ZWQgYW5kIG1heSBiZSB1bmxhd2Z1bC48QlI+CjwvYm9k eT4NCjwvaHRtbD4NCg== --_000_YQXPR01MB4198334C48683A60E53E5D14BC652YQXPR01MB4198CANP_-- From nobody Fri Sep 13 14:13:32 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X4x9t006nz5Tsh8 for ; Fri, 13 Sep 2024 14:13:38 +0000 (UTC) (envelope-from freebsd-net@dino.sk) Received: from cm0.netlabit.sk (mailhost.netlabit.sk [84.245.65.72]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4X4x9r1vBRz4Xqx for ; Fri, 13 Sep 2024 14:13:35 +0000 (UTC) (envelope-from freebsd-net@dino.sk) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of freebsd-net@dino.sk designates 84.245.65.72 as permitted sender) smtp.mailfrom=freebsd-net@dino.sk Received: from zeta.dino.sk ([84.245.95.254]) (AUTH: LOGIN milan, TLS: TLSv1.3,256bits,TLS_AES_256_GCM_SHA384) by cm0.netlabit.sk with ESMTPSA id 0000000001496E75.0000000066E4488D.0001010D; Fri, 13 Sep 2024 16:13:33 +0200 Date: Fri, 13 Sep 2024 16:13:32 +0200 From: Milan Obuch To: freebsd-net@freebsd.org Subject: Re: Ethernet device with shared mdio Message-ID: <20240913161332.4749ce87@zeta.dino.sk> In-Reply-To: References: <20240913090747.48062204@zeta.dino.sk> X-Mailer: Claws Mail 3.21.0 (GTK+ 2.24.33; amd64-portbld-freebsd13.3) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.47 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_SHORT(-0.17)[-0.168]; MIME_GOOD(-0.10)[text/plain]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:5578, ipnet:84.245.64.0/18, country:SK]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; MID_RHS_MATCH_FROMTLD(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; R_DKIM_NA(0.00)[]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[dino.sk]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 4X4x9r1vBRz4Xqx On Fri, 13 Sep 2024 13:40:55 +0000 Mike Belanger wrote: > Thank you for the response and for sharing your scenario. >=20 > We=E2=80=99ve also hacked up the cgem and the ffec driver to support a sh= ared > mdio. That was not too difficult, but we have a new scenario where > the mdio is now being shared between two different devices that use > different drivers (ffec and eqos). This presents a few extra > challenges. Could you elaborate a bit more? Are your hacks published somewhere or could you share? > I was hoping that FreeBSD may have considered supporting a shared > mdio. We can come up with something, but if there is an existing > architecture/approach in the works=E2=80=A6we would like to use a consist= ent > approach. At first glance, miiproxy did not seem like a fit. I don't know anything specific. I think I saw some DTBs with shared MDIO, but did not analysed the details. And miiproxy was just possible solution for the case, where MDIO controller is being initialised after MII controller (which was my case), but I did use some hacks for proof of concept. > I do not have the hardware. I am trying to help somebody else with > this. I have seen the dtb. It=E2=80=99s a Variscite DAR-MX8M-PLUS. OK, this makes the development a bit slower :) Regards, Milan From nobody Fri Sep 13 14:54:39 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X4y5J3JY3z5V0YS for ; Fri, 13 Sep 2024 14:54:44 +0000 (UTC) (envelope-from cryintothebluesky@gmail.com) Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X4y5J1NkZz4fnv for ; Fri, 13 Sep 2024 14:54:44 +0000 (UTC) (envelope-from cryintothebluesky@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-wr1-x430.google.com with SMTP id ffacd0b85a97d-374b25263a3so769367f8f.0 for ; Fri, 13 Sep 2024 07:54:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726239282; x=1726844082; darn=freebsd.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=66VqBt6WHHM6i8VQZMZmoxGW2+WZBcBvDn8DSAYPBAg=; b=EzidUWTygPRbpmzkn8wRkqiSmdB4YoVw1jpw15lKMFvBzxuZSb1c1LpdNPs/hJc8Q5 DEbSzBFctHVkcZG9mRlCa/jT8VxxRD8KLrfZf2iKWNcdOl4aiCe+w631+3gRBUgPk7Ri gkn4iNv5z8UZ5uWEboG2C2zjGAmX4prrLddAGseYB81Qs0k20o7WBsAwOE+8tn1eY119 g2ekzNWBtuDdYxpARFKqxxH1lovcA/vA6FSHAarWvl6F8AVRADBjY7cvfK3fDy+GgrJ4 Dsv319YSMJIlrjXApmmi+RCrGo/rDdRh2V4T1yomd09TGKlDsZNkvCiRL6JzB1tr8Hem LECQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726239282; x=1726844082; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=66VqBt6WHHM6i8VQZMZmoxGW2+WZBcBvDn8DSAYPBAg=; b=F/NF5dZrcHcyVxwoh7xrdbYTdhaP2tF7/G3ruHTVTg7CEak2GJS25Qr/jj22tYWMm7 XEKP+PQ+vKXm62d1Vp4oWAStcRmUDi15vjGG8LvGmhwpyiDffOQbPExlzUT7c+RW+atZ rjowaJCh890+SZNK7sfBNRJn28b3V9W74pCHyWUSnsR1+DAKB9Jv3s2YNIx/H8S6SD2K MLpEO++F3paXZVcUJ8tlzRsBdrBlgpBCTe/7R14IZgFewCj6bOgZOd6a9X7FRJ7yjn84 /cHLwYvfrdo9gQrKkq8quVT77s3W50z/i3QfFg9T2LQG5BcB0LdTwhuxWI+d2MNJjgpa nj+g== X-Gm-Message-State: AOJu0Yz4Bbdze6jSv6QJtjmOVaHh6V7SU9N8rbz3unVMD6PPLtmnskUz F5OFOU0ofNA5c2q11eImAltIO2vSY3PozpMsyOmurH5r9iHonOJF/WZ4zw== X-Google-Smtp-Source: AGHT+IEm9vpaacu4DVGv2FY97gFDCrnpWjFMZH1p1onh2NTEd8pVSfF/w0pL5SfJayJRADuzA36pFQ== X-Received: by 2002:a5d:4cc6:0:b0:374:c3f7:6af1 with SMTP id ffacd0b85a97d-378d61e28e7mr1899209f8f.15.1726239281737; Fri, 13 Sep 2024 07:54:41 -0700 (PDT) Received: from z600.home.lan (29.129.159.143.dyn.plus.net. [143.159.129.29]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-42d9b15abc5sm28229525e9.17.2024.09.13.07.54.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Sep 2024 07:54:40 -0700 (PDT) Date: Fri, 13 Sep 2024 15:54:39 +0100 From: Sad Clouds To: Mark Saad Cc: freebsd-net@freebsd.org Subject: Re: Performance issues with vnet jails + epair + bridge Message-Id: <20240913155439.1e171a88bd01ce9b97558a90@gmail.com> In-Reply-To: <39B2C95D-1E4F-4133-8923-AD305DFA9435@longcount.org> References: <20240913100938.3eac55c9fbd976fa72d58bb5@gmail.com> <39B2C95D-1E4F-4133-8923-AD305DFA9435@longcount.org> X-Mailer: Sylpheed 3.8.0beta1 (GTK+ 2.24.33; x86_64-pc-linux-gnu) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4X4y5J1NkZz4fnv On Fri, 13 Sep 2024 08:08:02 -0400 Mark Saad wrote: > Sad > Can you go back a bit you mentioned there is a RPi in the mix ? Some of the raspberries have their nic usb attached under the covers . Which will kill the total speed of things. > > Can you cobble together a diagram of what you have on either end ? Hello, I'm not sending data across the network, only between the host and the jails. I'm trying to evaluate how FreeBSD handles TCP data locally within a single host. I understand that vnet jails will have more overhead, compared to a shared TCP/IP stack via localhost. So I'm trying to measure it and see where the bottlenecks are. The Raspberry Pi 4 host has a single vnet jail, exchanging data with the host via epair(4) and if_bridge(4) interfaces. I don't really know what topology FreeBSD is using to represent all this so can't draw any diagrams, but I think all data flows through the kernel internally and never leaves the physical network interface. From nobody Sat Sep 14 00:34:33 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X5ByK6fL0z5WRqK for ; Sat, 14 Sep 2024 00:34:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X5ByK4v3Zz4kdv for ; Sat, 14 Sep 2024 00:34:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1726274073; a=rsa-sha256; cv=none; b=fckSX9oqV87pCPXcEmNhD2wmFf3UDgRrhpWQrXzGk01yLFta6YhR29AcXkk4mJwmBuO1Q0 zvaVhrQW2jdvv1+4bsgZ18y3+2Sqkcf0vfvzDEWj0vsZqxsmIFxSgdoqDiYR++i+ZrbRJs xEq31FXB2yWpFZupqinyQDneXOoe/65XOoDIMK6nMrD/CD+j63rzq51MglgN8r1XDLBmHh RAuCYE2V4wo7j+hIX3GMIuWRlU2P5Ok1yEbSfhUP9BWgkXI9IgwRCQ9NhIjJfohmp9YyCP Idy+YoDg6BuGF5vvHVLinBEIAyCWp4U8ctEUVq1xp5paAZtqKWSy9dXlpApeoQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1726274073; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7cw+UrUau0djjDqhjvgzFWIseH/GT+U+++wyLgmbpbM=; b=xt00WXdAqhJ6nQIAXBM6sQYPhajwz6Q8hDNH77ZdKboaraxkrUTDpDYWP8faSZXLAHdM1r 99BS/sIQjyiUM7UuwOjHNjwcX/0XidBqymj9PTiL5mPjhjbaAsY5I8hyujRewKKOQ62gon jnwhhNfhO5guisvb95kt0XW2IkRZE0pV6MY7jc50Pp2jm5iUGjWDj/obeWciqFxLxYDxGB 6dvA8FRKT3Fs21U7lr1mQDRAVcu3LLCdeJWv0vnQLK+E9FTmo8np9SEm+XVvWi8h6LaiRF FHcdQo8s7VEyHLR63ag4YUTHcT4GPuNMNQqhR9HBAe16Aj0dgE5r7s6K0dlKOg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4X5ByK4Vq6z151R for ; Sat, 14 Sep 2024 00:34:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 48E0YXrq060183 for ; Sat, 14 Sep 2024 00:34:33 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 48E0YXMa060182 for net@FreeBSD.org; Sat, 14 Sep 2024 00:34:33 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 281460] if_ovpn doesn't work with crypto.ko Date: Sat, 14 Sep 2024 00:34:33 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D281460 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |net@FreeBSD.org --- Comment #2 from Mark Linimon --- ^Triage: appears to be a problem with loadable modules. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Sep 14 00:48:15 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X5CG809p9z5WTfJ for ; Sat, 14 Sep 2024 00:48:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X5CG76FxHz4mBl for ; Sat, 14 Sep 2024 00:48:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1726274895; a=rsa-sha256; cv=none; b=EaaeFIZ/OV01sFviNHOnslKlnUyVTRft0CXdebHFTz00PXfiFy2gs3ahzluRVB0yEYLcFp 9TxnUywbSF/t4TztUuB7/EaX8UpEtJcucrLa+6uZV4W7foIJBbT4EzBbjQMAU+f5a1mT7j r4VfVPoNpPXJSGmeLceGvAt5iueVQrZu2bVVYRPDR6H4chx1cBEqM0FrPmYS1NKZFk8HWF z2zjO+VZh1fr1GasNaAmSzAfChEBvztYT1Jfc1YHTwBkbRKL2g4uAUa20UvRI8XbjZUeic Ct9S54Af23ZWvFAKaH6Fk8uF8M2zk7UvqDXLFe8wkdolwdKJ5ySZ9bZxyfNsgA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1726274895; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rMlpUsPWvna95KmDy9vtEDz102m2rLymPn+dAxJOB+o=; b=qvnPDNgOS4PvKiwyJTiJlDLORe4XiJu5vhDmUatxwrrW8jHgefO5+fVjgdDwIsrmSUW8Oj cW/d3A+p6xwl4+I4q5nfeYQeDeIDAcwqswide+PQWoTinpE5s46oLIEEINa1WC0JqMqlF1 aQ++6Xqx5I+7ryto0mVSB0x0S5ns9WtzL/yMRNYoGoHpDKCuC2zwFymuEBNWkaUVqH6JrA Md+QV5dsrnOKBB+OJK6mKymsP6k40Ib5UfKHC3hm/xT/jJeqlnjnOva/VkUmxu1y+DQXH0 qLABCNMf4UNdgnV/MhjAlJO58zSMl/oh1oFRstyfyEeB7IAsXG/gJUrPM74LqQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4X5CG75sw8z152R for ; Sat, 14 Sep 2024 00:48:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 48E0mFQl090433 for ; Sat, 14 Sep 2024 00:48:15 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 48E0mFxx090430 for net@FreeBSD.org; Sat, 14 Sep 2024 00:48:15 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 281452] Realtek 'if_re' module crashes on module load Date: Sat, 14 Sep 2024 00:48:15 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D281452 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |net@FreeBSD.org --- Comment #3 from Mark Linimon --- ^Triage: appears to be a problem only on kldload. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Sep 14 02:45:03 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X5Fs76DcBz5WnR3 for ; Sat, 14 Sep 2024 02:45:15 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X5Fs75mf8z47nm; Sat, 14 Sep 2024 02:45:15 +0000 (UTC) (envelope-from zlei@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1726281915; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7dR1TVlJo1EFEbPoJ/VGdnJDKmwogwK3eP/zhEOSDPY=; b=ar1OYxMJUBr2EKJtTLv8roa9Cr7k1V9fKsvzj+EbrjDkJYJ4FOxWtD+7K5+U7a8Z5j4Fwa QcucJd6PhzLVy9qhmJpOa3AdfYTYp01mUIABKBWPHpBiZaL20+xsYEgHz/hHMqrPuTQGgi x46tND/sqhBALXd3ru4PibwCXlff3T4CA7s2uEUwZS7GRg5DFl4ujDbgXaIy1qSbzn0X+c 6N3UzehmondtaiFZoUXWFYZF5xuGHrK/vD2Uwzcbf1o6A7l0gSyyLFWpBS0aan6uIr0WZ4 KZSK2LN+hnt+tRZrRaGd6tl/9C2teT9IK3Ln9wiPZQnewzoTbRsHbgDUGAApNg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1726281915; a=rsa-sha256; cv=none; b=ZoJpJVCWptsAz2RFVUOcWTdhgf6ba+P1hrZpaZRgYZHYbJ/2/aldUV9vMnDXyHJaDo6C+g 0VSlmLBzdOm23V0ZHQA4BPjApla++/SjR0Y9OEWtChWm0GPIwTqtvudhuUV20wuTgm264E zdsbLwKinbKCpTygM+hmVylFsFg8SfqlKiMwQbj0X+MbZASm0iY3FX+a0VUzbSBh/hIcKZ PXWF6Qewwf69Kt+5TkagDzv5LKxOddMxTFivhqXjKCVgPhtQZ5twSpRtS1RUaoHkX/qK/V 9VFLTFF2pJAc5S9kBvdsOsw6NJKNAfEuy0DlouuHLAfKb+/7CAsQOgot5t3rzw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1726281915; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7dR1TVlJo1EFEbPoJ/VGdnJDKmwogwK3eP/zhEOSDPY=; b=BYsizEG99bIkMIDoKWROcjbPRyHRnLTLqiv4TgTS5zolmh93iXk5jdIzqII0lBFSFcVNEW SdlBdPFO+Unhtd5p8GbbMX/zFpeBX0nrlkLNLDlcG0MEIyD1EpTV1xq2lCCWJssGEOin5T iLTBULKymE6Z3Ud+iSv29Uc2Ah1NuNh/CGAx0mCYSdh85waIV4cyKJA0A5UuNzpAuyL5wB KObwOBxq0dRAVb/dt0zrV+IuBC+zmkUXjRK8RA6CmUPqo1Y3eHpF/mxJV76H0tEhAqf3iU xFDgnnngjs6Jn1U+1f0NI59YgDshmcdKUnrEi4gUH9OwbiSEtr7A7Ife66DXrQ== Received: from smtpclient.apple (unknown [IPv6:2001:19f0:6001:9db:98f0:9fe0:3545:10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: zlei/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4X5Fs55gMzzFjS; Sat, 14 Sep 2024 02:45:12 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Content-Type: text/plain; charset=us-ascii List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.10\)) Subject: Re: Performance issues with vnet jails + epair + bridge From: Zhenlei Huang In-Reply-To: <20240913155439.1e171a88bd01ce9b97558a90@gmail.com> Date: Sat, 14 Sep 2024 10:45:03 +0800 Cc: Mark Saad , FreeBSD Net Content-Transfer-Encoding: quoted-printable Message-Id: References: <20240913100938.3eac55c9fbd976fa72d58bb5@gmail.com> <39B2C95D-1E4F-4133-8923-AD305DFA9435@longcount.org> <20240913155439.1e171a88bd01ce9b97558a90@gmail.com> To: Sad Clouds X-Mailer: Apple Mail (2.3696.120.41.1.10) > On Sep 13, 2024, at 10:54 PM, Sad Clouds = wrote: >=20 > On Fri, 13 Sep 2024 08:08:02 -0400 > Mark Saad wrote: >=20 >> Sad >> Can you go back a bit you mentioned there is a RPi in the mix ? = Some of the raspberries have their nic usb attached under the covers . = Which will kill the total speed of things.=20 >>=20 >> Can you cobble together a diagram of what you have on either end ? >=20 > Hello, I'm not sending data across the network, only between the host > and the jails. I'm trying to evaluate how FreeBSD handles TCP data > locally within a single host. When you take vnet into account, the **locally** traffic should within on single vnet jail. If you want traffic across vnet jails, if_epair or = netgraph hooks should be employed, and it of course will introduce some overhead. >=20 > I understand that vnet jails will have more overhead, compared to a > shared TCP/IP stack via localhost. So I'm trying to measure it and see > where the bottlenecks are. The overhead of vnet jail should be neglectable, compared to legacy jail or no-jail. Bare in mind when VIMAGE option is enabled, there is a = default vnet 0. It is not visible via jls and can not be destroyed. So when you = see bottlenecks, for example this case, it is mostly caused by other = components such as if_epair, but not the vnet jail itself. >=20 > The Raspberry Pi 4 host has a single vnet jail, exchanging data with > the host via epair(4) and if_bridge(4) interfaces. I don't really know > what topology FreeBSD is using to represent all this so can't draw any > diagrams, but I think all data flows through the kernel internally and > never leaves the physical network interface. For vnet jails, when you try to describe the network topology, you can treat them as VM / physical boxes. I have one box with dozens of vnet jails. Each of them has very single responsibility, e.g. DHCP, LADP, pf firewall, OOB access. The topology = looks quite clear and it is easy to maintenance. The only overhead is too much hops between the vnet jail instances. For my use case the performance is not critical and it works great for years. >=20 Best regards, Zhenlei From nobody Sat Sep 14 03:56:49 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X5HRj6ZYHz5Wy0P for ; Sat, 14 Sep 2024 03:56:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X5HRj4kMQz4JKx for ; Sat, 14 Sep 2024 03:56:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1726286209; a=rsa-sha256; cv=none; b=xfX/Z02eLcolKMqCw62XHSROtC5XtWpxDgkJ/FY+iw5i/uqn0/y20nGEGUoHvHwZ5ABgm7 n0O3K3HVgSM7zDdcdZnub/NvfQeM1ybYA9ckuVdoYlmvEkekg3r85uWd4McHdl0Juv8cM4 oZTYaNYLx30K46YDDWrEHr8oMDcDMJmjM1PVQLiIItrkJhXyWcLVYPY8bDNSD+r4VQS9rZ iSU834iMoxX8nzjROYw5WYzsv6JmA35X1xjn4NTa5TFzL1jkM6FwfWNGKEk49RaWTSBJU8 2q19M8VhDuPj3m0baVIo1S3REjxqEWun8GxnuQ6JquRFt8qzDHW+jr3sbevbGw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1726286209; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vgDo1rmw6FNDZ/VKeftJfNfo/p6TXQoPdgPvX5ax1a8=; b=lpWZr3a/OgcrKvgHJO2mnwLu+85UvsV+KiE8IdTpdXEhYAsHSQ8WM6EOfzgzotAEFVDrCd Yb5N3Dmq8M/QAJjpMQSGnAR5HbuRosMuQEOqEz8Sdu1i+yjJeQIjI5lMY+QxtxskvehQwG PV9urzongKuis01/pJ8RPyU8L8w2msduilcO5W/xREgoTtj2qgeUqC9upg23K4GZB052eB eCT9vsrpfYUIAObKZGndYj4JbMDQmUVVOgVGB6WY85OO555Q4Q53Q6H0IXH+2qMXBe25Yk khh8FkoXG+UkRbVnt8GZNQLGxr2NsIwSrUjSFVi3gaxBEMWyjKIAbjEyvwx3tw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4X5HRj4J9fz1B78 for ; Sat, 14 Sep 2024 03:56:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 48E3unr5040111 for ; Sat, 14 Sep 2024 03:56:49 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 48E3unhr040110 for net@FreeBSD.org; Sat, 14 Sep 2024 03:56:49 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 281452] Realtek 'if_re' module crashes on module load Date: Sat, 14 Sep 2024 03:56:49 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: zlei@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D281452 --- Comment #4 from Zhenlei Huang --- @Mithun It seems this has been fixed by Kristof via commit 43387b4e5740 (if: guard against if_ioctl being NULL). The commit has been MFCed to stable/14 with commit 9a8a26aefb36. May you please have a try with 9a8a26aefb36 ? See also https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275920 . --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Sep 14 10:01:56 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X5RY05RFbz5WVyD for ; Sat, 14 Sep 2024 10:01:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X5RY037kXz41Sm for ; Sat, 14 Sep 2024 10:01:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1726308116; a=rsa-sha256; cv=none; b=ka9bfg+ZwfSPswozn8a4fvPoXlbhkqlTOawzJX9+K/xRwVKAtCPPSj/0BjWyRzGeEOCtCO NrJl6FKYFp56bLVHzaTDmkjKaq65VGPFd/Th2kq2udzcUOnkqCZMJoM56JAAiUZFEEpk5K 0WSuHZ2nPUm8iN2fEry6YODQoYZOjAlwKviqKyius8KTCSwck1XrSe798D3YwYOBHv050s X9Vcv7aDoTkF6FH2seRXLAaTkUq42D/907DVPE6ATP80gfea9m+mzl3/3IiAqHN45Ws9Ik qVUy9bE1guljeLcSWaQAYFj0gpOK6JjAj2/6BKanI/Ywz+5aibOvqhhNigp82Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1726308116; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=80DRe+lvIu4DQwFexWleTk71qK1iD89P6Fc7yZa/t8Q=; b=pbYxYgP9midJn5WQidNmU4k52rLhSzF7sq60fhe9rpa631JaAsojApqvUHseoeWJlQClIB BVzFI0gnHbtbIpnsQkIdnGagq22qvHr61NszV6DvYykVVjs7SODWik+Pl7BVw2iwYTZJmV BnkaYhvyWFCx7zRG9MDZE7q7pZ+C6+cA4uZSKq/y+oLg9OPaaU0x0uCwc4Y4Kp/M5U/7Ax s+sxW6Fjp0Meow86eFFTsIhHDGH44piyA82GaFI/kHCi+ymS4UPUtJwCBOpuUn58iiEB/a XrrBhvYQbgiWukxB49ErzufpsTqalYHHzFRGl7q5T/tXBuWA4u79KuMldzCn7w== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4X5RY02ldvzM6w for ; Sat, 14 Sep 2024 10:01:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 48EA1um3097238 for ; Sat, 14 Sep 2024 10:01:56 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 48EA1uJv097237 for net@FreeBSD.org; Sat, 14 Sep 2024 10:01:56 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 272573] Lock order reversal with OpenVPN over UDP (udpinp) Date: Sat, 14 Sep 2024 10:01:56 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: zlei@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: kp@freebsd.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D272573 Zhenlei Huang changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |In Progress Assignee|net@FreeBSD.org |kp@freebsd.org --- Comment #1 from Zhenlei Huang --- Should be fixed by Kristof via commit 3acf7e0da487 (if_ovpn: avoid LOR betw= een ovpn and UDP locks) . --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Sep 14 10:25:16 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X5S431S6Sz5WZ3S for ; Sat, 14 Sep 2024 10:25:23 +0000 (UTC) (envelope-from cryintothebluesky@gmail.com) Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X5S415SxVz451C; Sat, 14 Sep 2024 10:25:21 +0000 (UTC) (envelope-from cryintothebluesky@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-wm1-x32f.google.com with SMTP id 5b1f17b1804b1-42cb58d810eso24821915e9.0; Sat, 14 Sep 2024 03:25:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726309519; x=1726914319; darn=freebsd.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=Jp8gLxTf/zPs0W0X7TOUzgJkz422vPHSLThv0ZljW70=; b=Zu4eIfAQcXOnj1hnS5AutcmkbDKx0VdB9cWqRjS94nvPhDk4WSA8z7KXo92hv6hoMV 9chE4cw9QC/a7JWT9rv4h9RTo1k0CXegLo0ZgC96/X9/G7TtTDHI9zg7U6WysU0uIrUq PYSdLEUfBqWUDH04gxj0NeHP2IsDrSJ0KQ9hH3Q/Bd94zrSFO8Arz4vERludJXXL8bAz AirOahhUiRX0r45t+ZXVK/yHpGxD6S6LvSe0M+C2B5TqimLCXybyXMYxQNjgUlvJ/0ig BhyUMqg8Sf9BnZZBL+KkiL7lb+jR0Xf5gEDAQ7IW02N8BhnHGyVa8lSMqm6cFXOYJqFH qGOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726309519; x=1726914319; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Jp8gLxTf/zPs0W0X7TOUzgJkz422vPHSLThv0ZljW70=; b=WQZdvFuMIk1e3bCQ6P9Uc9dsD2NDvQP0CJQkadgdNYknLckJYE356J9rXb4W7JGjN2 OsnAoOsq3jzf5m5tgJDUUSi4eUQDWhz9PeH+UkeTimsBwh/3yXIZH1JuJmonJ0UjfkYl KMQ7i+DRvW9IHJnYPBy2PQKnh8IQZARqwEPmr769M3gaQZ3T1bzi0E5On928PpGjUgzO Rt531uJ75IiaC6zTGikS+PiI+c1QIojGeqd2sgErvhPxKJKqr5po/nRGNpsgwloYtWa7 Z7O4aEhKFWa4N26qzkHGoBC7VkNSHKCGz7kIKUNRZqxbdrzprpCjLVXdqYdr9ZcMtHtI roVQ== X-Forwarded-Encrypted: i=1; AJvYcCX7Fpab3EYAQie4pJb3J6f+FvQAhXhcnuDhXWbY4h7RT3EfSv5cMjZtwQ0D7bL7pyJvt24jjpt7LSHiZw==@freebsd.org X-Gm-Message-State: AOJu0Yye63+NP59w40b9UYIlwiTq4L/3G0N0Y28jndrfMko0ouxIrGfW nJfHeLNhZMRqNUAV+ZPxpLDvqTGOweNfF1E29BqsD8Z/E4Ib86mQtYpiCw== X-Google-Smtp-Source: AGHT+IGX4WysTFOt285jGG3+swlw6KEaW9LKyU8Y3faxQQwQwnwdO8OufX5tq2/ltxKEqsP1AwXkWw== X-Received: by 2002:a05:600c:1ca2:b0:42c:ae4e:a96c with SMTP id 5b1f17b1804b1-42cdcbb8630mr56429505e9.16.1726309518329; Sat, 14 Sep 2024 03:25:18 -0700 (PDT) Received: from z600.home.lan (113.129.159.143.dyn.plus.net. [143.159.129.113]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-42d9b054d2dsm50466505e9.3.2024.09.14.03.25.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 14 Sep 2024 03:25:17 -0700 (PDT) Date: Sat, 14 Sep 2024 11:25:16 +0100 From: Sad Clouds To: Zhenlei Huang Cc: Mark Saad , FreeBSD Net Subject: Re: Performance issues with vnet jails + epair + bridge Message-Id: <20240914112516.cfb31bae68ab90b83ca7ad4b@gmail.com> In-Reply-To: References: <20240913100938.3eac55c9fbd976fa72d58bb5@gmail.com> <39B2C95D-1E4F-4133-8923-AD305DFA9435@longcount.org> <20240913155439.1e171a88bd01ce9b97558a90@gmail.com> X-Mailer: Sylpheed 3.8.0beta1 (GTK+ 2.24.33; x86_64-pc-linux-gnu) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4X5S415SxVz451C On Sat, 14 Sep 2024 10:45:03 +0800 Zhenlei Huang wrote: > The overhead of vnet jail should be neglectable, compared to legacy jail > or no-jail. Bare in mind when VIMAGE option is enabled, there is a default > vnet 0. It is not visible via jls and can not be destroyed. So when you see > bottlenecks, for example this case, it is mostly caused by other components > such as if_epair, but not the vnet jail itself. Perhaps this needs a correction - the vnet itself may be OK, but due to a single physical NIC on this appliance, I cannot use vnet jails without virtualised devices like if_epair(4) and if_bridge(4). I think there may be other scalability bottlenecks. I have a similar setup on Solaris Here devel is a Solaris zone with exclusive IP configuration, which I think may be similar to FreeBSD vnet. It has a virtual NIC devel/net0 which operates over the physical NIC also called net0 in the global zone: $ dladm LINK CLASS MTU STATE OVER net0 phys 1500 up -- net1 phys 1500 up -- net2 phys 1500 up -- net3 phys 1500 up -- pkgsrc/net0 vnic 1500 up net0 devel/net0 vnic 1500 up net0 If I run TCP bulk data benchmark with 64 concurrent threads, 32 threads with server process in the global zone and 32 threads with client process in the devel zone, then the system evenly spreads the load across all CPU cores and none of them are sitting idle: $ mpstat -A core 1 COR minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys st idl sze 0 0 0 2262 2561 4 4744 2085 209 7271 0 747842 272 528 0 0 8 1 0 0 3187 4209 2 9102 3768 514 10605 0 597012 221 579 0 0 8 2 0 0 2091 3251 7 6768 2884 307 9557 0 658124 244 556 0 0 8 3 0 0 1745 1786 16 3494 1520 176 8847 0 746373 273 527 0 0 8 4 0 0 2797 2767 3 5908 2414 371 7849 0 692873 253 547 0 0 8 5 0 0 2782 2359 5 4857 2012 324 9431 0 684840 251 549 0 0 8 6 0 0 4324 4133 0 9138 3592 538 12525 0 516342 191 609 0 0 8 7 0 0 2180 3249 0 6960 2926 321 8825 0 697861 257 543 0 0 8 With FreeBSD I tried "options RSS" and increasing "net.isr.maxthreads" however this resulted in some really flaky kernel behavior. So I'm thinking that if_epair(4) may be OK for some low-bandwidth use cases, i.e. testing firewall rules, etc, but not suitable for things like file/object storage servers, etc. From nobody Sun Sep 15 17:01:07 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X6DpS1Cw3z5WNGl for ; Sun, 15 Sep 2024 17:01:20 +0000 (UTC) (envelope-from dfr@rabson.org) Received: from mail-yw1-x1130.google.com (mail-yw1-x1130.google.com [IPv6:2607:f8b0:4864:20::1130]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X6DpR0vffz4dP6 for ; Sun, 15 Sep 2024 17:01:19 +0000 (UTC) (envelope-from dfr@rabson.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=rabson-org.20230601.gappssmtp.com header.s=20230601 header.b=3KdWIFdW; dmarc=none; spf=pass (mx1.freebsd.org: domain of dfr@rabson.org designates 2607:f8b0:4864:20::1130 as permitted sender) smtp.mailfrom=dfr@rabson.org Received: by mail-yw1-x1130.google.com with SMTP id 00721157ae682-6d3f017f80eso18954437b3.1 for ; Sun, 15 Sep 2024 10:01:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rabson-org.20230601.gappssmtp.com; s=20230601; t=1726419678; x=1727024478; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=MSp8NonbLHfPzCLPIrtYx1bYtlq6lwU/+aVm9yMeJOE=; b=3KdWIFdWBgrg6zUUPaFpdcsO2PFgOCQON7tfOz3gvkPwf9qefRqrpTFQcLcSOyFeqe fvL3ZLHl6QuyuCmfVFCtU5lLsP9e4xfWxFido89jFhxead9dxj8hVpHGU79uMGX2w0i8 fj78FM8ORr6ysnZYYjx3yaKF4vazoqPkT/6cCNgcQtaq+ps72cpJOFQXbbvmHabf1+VS oqm4M7mf/wbac3XqCB7sIh/pdABi9JTzAZM+m2TQzeKbWg3Fj5cd/ENEHsQz4v27R0tc blE0+lstBFsMI9M1J3J4+D060YZHXvLa9lkXGYaycmoB//JRCheltpLkwq9edCNOobav Eq3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726419678; x=1727024478; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=MSp8NonbLHfPzCLPIrtYx1bYtlq6lwU/+aVm9yMeJOE=; b=j+wWPpRfD/D2U5qT3NyuUUWYItlOJoqaH9FkTrMBe9CvBsAb+KctZz84Mhc/14yvLG HOAiYEljS2W47U+LYK/wTbc0ScL2C3gFJ6oi1cQGZxqLhItfwr+ZSENvc/ydcHAEIv2D Nm1TBGwxD0zVG/icXsksbJwpp+2aNqWFTwwaAF5Lde/EhBqtSK9eRmGDTF/HRwL6pagi gOOa47QYIlGKUJ5ShoS0QP6bV3xdJ0QiE26vvh96gNOS0IZ2+yAVKtDZdyhhkVF58van t+StabTOAhc5jtVu7JA1sJTiGAanV+Q0Ff46cfTouI8wmgdJ7V4ckE+rMz04lw4jg5lC YMFg== X-Forwarded-Encrypted: i=1; AJvYcCUZpVJJFAP/CH+EX28LDQSEJ2dlOPSIaiRnHpKzEPpSJs1Gw5NZqdjxlk72xf7d6U265h23X7+Bq5Yu8Q==@freebsd.org X-Gm-Message-State: AOJu0Ywfn1NhgYMAM41O3XnofPWQB//HspBEJ6vwioRuT3k3DU604GIi 9tYC/Tiknx93oU+zZFI/FBfhDL0enDpd16eURHJmOdMrSjaujkFEmHWJe7LsvW385YzOrnRVNq9 D9c7dFap2LGfOMI7MQgjDtlnp/kRrCBC+zQzb9/LRt25UpaLk X-Google-Smtp-Source: AGHT+IGuBtArHT0qVGTFBB7VZaWXOYisMn6skrcFYVCQe50t3n8BeiGYAHco2MszCtuQnqBMa0QQZf8HLXRD+t0AWbs= X-Received: by 2002:a05:690c:6c09:b0:6dd:c6a8:5775 with SMTP id 00721157ae682-6ddc6a8593emr9918987b3.39.1726419678178; Sun, 15 Sep 2024 10:01:18 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 References: <20240913100938.3eac55c9fbd976fa72d58bb5@gmail.com> <39B2C95D-1E4F-4133-8923-AD305DFA9435@longcount.org> <20240913155439.1e171a88bd01ce9b97558a90@gmail.com> <20240914112516.cfb31bae68ab90b83ca7ad4b@gmail.com> In-Reply-To: <20240914112516.cfb31bae68ab90b83ca7ad4b@gmail.com> From: Doug Rabson Date: Sun, 15 Sep 2024 18:01:07 +0100 Message-ID: Subject: Re: Performance issues with vnet jails + epair + bridge To: Sad Clouds Cc: Zhenlei Huang , Mark Saad , FreeBSD Net Content-Type: multipart/alternative; boundary="000000000000a6cf0c06222b6702" X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; R_DKIM_ALLOW(-0.20)[rabson-org.20230601.gappssmtp.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FREEFALL_USER(0.00)[dfr]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1130:from]; MIME_TRACE(0.00)[0:+,1:+,2:~]; MISSING_XM_UA(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DMARC_NA(0.00)[rabson.org]; TO_DN_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; FREEMAIL_TO(0.00)[gmail.com]; DKIM_TRACE(0.00)[rabson-org.20230601.gappssmtp.com:+] X-Rspamd-Queue-Id: 4X6DpR0vffz4dP6 --000000000000a6cf0c06222b6702 Content-Type: text/plain; charset="UTF-8" I just did a throughput test with iperf3 client on a FreeBSD 14.1 host with an intel 10GB nic connecting to an iperf3 server running in a vnet jail on a truenas host (13.something) also with an intel 10GB nic and I get full 10GB throughput in this setup. In the past, I had to disable LRO on the truenas host for this to work properly. Doug. On Sat, 14 Sept 2024 at 11:25, Sad Clouds wrote: > On Sat, 14 Sep 2024 10:45:03 +0800 > Zhenlei Huang wrote: > > > The overhead of vnet jail should be neglectable, compared to legacy jail > > or no-jail. Bare in mind when VIMAGE option is enabled, there is a > default > > vnet 0. It is not visible via jls and can not be destroyed. So when you > see > > bottlenecks, for example this case, it is mostly caused by other > components > > such as if_epair, but not the vnet jail itself. > > Perhaps this needs a correction - the vnet itself may be OK, but due to > a single physical NIC on this appliance, I cannot use vnet jails > without virtualised devices like if_epair(4) and if_bridge(4). I think > there may be other scalability bottlenecks. > > I have a similar setup on Solaris > > Here devel is a Solaris zone with exclusive IP configuration, which I > think may be similar to FreeBSD vnet. It has a virtual NIC devel/net0 > which operates over the physical NIC also called net0 in the global > zone: > > $ dladm > LINK CLASS MTU STATE OVER > net0 phys 1500 up -- > net1 phys 1500 up -- > net2 phys 1500 up -- > net3 phys 1500 up -- > pkgsrc/net0 vnic 1500 up net0 > devel/net0 vnic 1500 up net0 > > If I run TCP bulk data benchmark with 64 concurrent threads, 32 > threads with server process in the global zone and 32 threads with > client process in the devel zone, then the system evenly spreads the > load across all CPU cores and none of them are sitting idle: > > $ mpstat -A core 1 > COR minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys st > idl sze > 0 0 0 2262 2561 4 4744 2085 209 7271 0 747842 272 528 > 0 0 8 > 1 0 0 3187 4209 2 9102 3768 514 10605 0 597012 221 579 > 0 0 8 > 2 0 0 2091 3251 7 6768 2884 307 9557 0 658124 244 556 > 0 0 8 > 3 0 0 1745 1786 16 3494 1520 176 8847 0 746373 273 527 > 0 0 8 > 4 0 0 2797 2767 3 5908 2414 371 7849 0 692873 253 547 > 0 0 8 > 5 0 0 2782 2359 5 4857 2012 324 9431 0 684840 251 549 > 0 0 8 > 6 0 0 4324 4133 0 9138 3592 538 12525 0 516342 191 609 > 0 0 8 > 7 0 0 2180 3249 0 6960 2926 321 8825 0 697861 257 543 > 0 0 8 > > With FreeBSD I tried "options RSS" and increasing "net.isr.maxthreads" > however this resulted in some really flaky kernel behavior. So I'm > thinking that if_epair(4) may be OK for some low-bandwidth use cases, > i.e. testing firewall rules, etc, but not suitable for things like > file/object storage servers, etc. > > --000000000000a6cf0c06222b6702 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I just did a throughput test with iperf3 client on a FreeB= SD 14.1 host with an intel 10GB nic connecting to an iperf3 server running = in a vnet jail on a truenas host (13.something) also with an intel 10GB nic= and I get full 10GB throughput in this setup. In the past, I had to disabl= e LRO on the truenas host for this to work properly.

Dou= g.



=
On Sat, 14 Sept 2024 at 11:25, Sad Cl= ouds <cryintothebluesky@g= mail.com> wrote:
On Sat, 14 Sep 2024 10:45= :03 +0800
Zhenlei Huang <zlei@FreeBSD.org> wrote:

> The overhead of vnet jail should be neglectable, compared to legacy ja= il
> or no-jail. Bare in mind when VIMAGE option is enabled, there is a def= ault
> vnet 0. It is not visible via jls and can not be destroyed. So when yo= u see
> bottlenecks, for example this case, it is mostly caused by other compo= nents
> such as if_epair, but not the vnet jail itself.

Perhaps this needs a correction - the vnet itself may be OK, but due to
a single physical NIC on this appliance, I cannot use vnet jails
without virtualised devices like if_epair(4) and if_bridge(4). I think
there may be other scalability bottlenecks.

I have a similar setup on Solaris

Here devel is a Solaris zone with exclusive IP configuration, which I
think may be similar to FreeBSD vnet. It has a virtual NIC devel/net0
which operates over the physical NIC also called net0 in the global
zone:

$ dladm
LINK=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 CLASS=C2=A0 =C2= =A0 =C2=A0MTU=C2=A0 =C2=A0 STATE=C2=A0 =C2=A0 OVER
net0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 phys=C2=A0 =C2= =A0 =C2=A0 1500=C2=A0 =C2=A0up=C2=A0 =C2=A0 =C2=A0 =C2=A0--
net1=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 phys=C2=A0 =C2= =A0 =C2=A0 1500=C2=A0 =C2=A0up=C2=A0 =C2=A0 =C2=A0 =C2=A0--
net2=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 phys=C2=A0 =C2= =A0 =C2=A0 1500=C2=A0 =C2=A0up=C2=A0 =C2=A0 =C2=A0 =C2=A0--
net3=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 phys=C2=A0 =C2= =A0 =C2=A0 1500=C2=A0 =C2=A0up=C2=A0 =C2=A0 =C2=A0 =C2=A0--
pkgsrc/net0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0vnic=C2=A0 =C2=A0 =C2=A0 1500= =C2=A0 =C2=A0up=C2=A0 =C2=A0 =C2=A0 =C2=A0net0
devel/net0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 vnic=C2=A0 =C2=A0 =C2=A0 1500= =C2=A0 =C2=A0up=C2=A0 =C2=A0 =C2=A0 =C2=A0net0

If I run TCP bulk data benchmark with 64 concurrent threads, 32
threads with server process in the global zone and 32 threads with
client process in the devel zone, then the system evenly spreads the
load across all CPU cores and none of them are sitting idle:

$ mpstat -A core 1
=C2=A0COR minf mjf xcal=C2=A0 intr ithr=C2=A0 csw icsw migr smtx=C2=A0 srw= =C2=A0 syscl=C2=A0 usr sys=C2=A0 st idl sze
=C2=A0 =C2=A00=C2=A0 =C2=A0 0=C2=A0 =C2=A00 2262=C2=A0 2561=C2=A0 =C2=A0 4 = 4744 2085=C2=A0 209 7271=C2=A0 =C2=A0 0 747842=C2=A0 272 528=C2=A0 =C2=A00= =C2=A0 =C2=A00=C2=A0 =C2=A08
=C2=A0 =C2=A01=C2=A0 =C2=A0 0=C2=A0 =C2=A00 3187=C2=A0 4209=C2=A0 =C2=A0 2 = 9102 3768=C2=A0 514 10605=C2=A0 =C2=A00 597012=C2=A0 221 579=C2=A0 =C2=A00= =C2=A0 =C2=A00=C2=A0 =C2=A08
=C2=A0 =C2=A02=C2=A0 =C2=A0 0=C2=A0 =C2=A00 2091=C2=A0 3251=C2=A0 =C2=A0 7 = 6768 2884=C2=A0 307 9557=C2=A0 =C2=A0 0 658124=C2=A0 244 556=C2=A0 =C2=A00= =C2=A0 =C2=A00=C2=A0 =C2=A08
=C2=A0 =C2=A03=C2=A0 =C2=A0 0=C2=A0 =C2=A00 1745=C2=A0 1786=C2=A0 =C2=A016 = 3494 1520=C2=A0 176 8847=C2=A0 =C2=A0 0 746373=C2=A0 273 527=C2=A0 =C2=A00= =C2=A0 =C2=A00=C2=A0 =C2=A08
=C2=A0 =C2=A04=C2=A0 =C2=A0 0=C2=A0 =C2=A00 2797=C2=A0 2767=C2=A0 =C2=A0 3 = 5908 2414=C2=A0 371 7849=C2=A0 =C2=A0 0 692873=C2=A0 253 547=C2=A0 =C2=A00= =C2=A0 =C2=A00=C2=A0 =C2=A08
=C2=A0 =C2=A05=C2=A0 =C2=A0 0=C2=A0 =C2=A00 2782=C2=A0 2359=C2=A0 =C2=A0 5 = 4857 2012=C2=A0 324 9431=C2=A0 =C2=A0 0 684840=C2=A0 251 549=C2=A0 =C2=A00= =C2=A0 =C2=A00=C2=A0 =C2=A08
=C2=A0 =C2=A06=C2=A0 =C2=A0 0=C2=A0 =C2=A00 4324=C2=A0 4133=C2=A0 =C2=A0 0 = 9138 3592=C2=A0 538 12525=C2=A0 =C2=A00 516342=C2=A0 191 609=C2=A0 =C2=A00= =C2=A0 =C2=A00=C2=A0 =C2=A08
=C2=A0 =C2=A07=C2=A0 =C2=A0 0=C2=A0 =C2=A00 2180=C2=A0 3249=C2=A0 =C2=A0 0 = 6960 2926=C2=A0 321 8825=C2=A0 =C2=A0 0 697861=C2=A0 257 543=C2=A0 =C2=A00= =C2=A0 =C2=A00=C2=A0 =C2=A08

With FreeBSD I tried "options RSS" and increasing "net.isr.m= axthreads"
however this resulted in some really flaky kernel behavior. So I'm
thinking that if_epair(4) may be OK for some low-bandwidth use cases,
i.e. testing firewall rules, etc, but not suitable for things like
file/object storage servers, etc.

--000000000000a6cf0c06222b6702-- From nobody Sun Sep 15 17:44:51 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X6Fmy35wzz5WV8H for ; Sun, 15 Sep 2024 17:45:06 +0000 (UTC) (envelope-from zarychtam@plan-b.pwste.edu.pl) Received: from plan-b.pwste.edu.pl (plan-b.pwste.edu.pl [IPv6:2001:678:618::40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "plan-b.pwste.edu.pl", Issuer "GEANT OV RSA CA 4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X6Fmw4YyVz4jZJ for ; Sun, 15 Sep 2024 17:45:04 +0000 (UTC) (envelope-from zarychtam@plan-b.pwste.edu.pl) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=plan-b.pwste.edu.pl header.s=plan-b-mailer header.b="m1Q3I/xV"; dmarc=pass (policy=quarantine) header.from=plan-b.pwste.edu.pl; spf=pass (mx1.freebsd.org: domain of zarychtam@plan-b.pwste.edu.pl designates 2001:678:618::40 as permitted sender) smtp.mailfrom=zarychtam@plan-b.pwste.edu.pl Received: from plan-b.pwste.edu.pl (zarychtam@localhost [127.0.0.1]) by plan-b.pwste.edu.pl (8.18.1/8.17.2) with ESMTPS id 48FHiqna044492 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sun, 15 Sep 2024 19:44:53 +0200 (CEST) (envelope-from zarychtam@plan-b.pwste.edu.pl) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=plan-b.pwste.edu.pl; s=plan-b-mailer; t=1726422293; bh=iE7UYqWN8JbFKmNmmk7X5RpZp4ntYnmKtxTa0xlNmog=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=m1Q3I/xVigS9tupIL3SYqtIIpE8/jzRYPwoZQ3uRPhPhqi4fhFSgOf8Avvw9NejGf JLnhk5fNJgstgBn0wj/LSYVdY0qILu2HJudaZe0mY9DCgBHpwZrpXYxXD47t7YDBZU E6F/S4PvgdSx8pgmoMzVt8tPQiv3R8N1iLCxUJUr9uIUQl4NiE9isYJW+uycO5MZTK cVtC7gbAXy/EUG7C3S7cZBjUCSefuyc/V5y4Dw3FRT1lSrkKbQL7ICt/s3eTEDk4N9 bi3iCnjsT9+AmI4yyma8kMPeV170+uNvKq5ZRSG4FKig5LFGfZRxORZFMKDde9HEf1 jaATdJLcE2tcg== Received: (from zarychtam@localhost) by plan-b.pwste.edu.pl (8.18.1/8.18.1/Submit) id 48FHiqtM044491; Sun, 15 Sep 2024 19:44:52 +0200 (CEST) (envelope-from zarychtam) Date: Sun, 15 Sep 2024 19:44:51 +0200 From: Marek Zarychta To: Sad Clouds Cc: freebsd-net@freebsd.org Subject: Re: Performance issues with vnet jails + epair + bridge Message-ID: References: <20240912181618.7895d10ad5ff2ebae9883192@gmail.com> List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20240912181618.7895d10ad5ff2ebae9883192@gmail.com> X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.988]; DMARC_POLICY_ALLOW(-0.50)[plan-b.pwste.edu.pl,quarantine]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[plan-b.pwste.edu.pl:s=plan-b-mailer]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; MISSING_XM_UA(0.00)[]; ASN(0.00)[asn:206006, ipnet:2001:678:618::/48, country:PL]; MIME_TRACE(0.00)[0:+]; TO_DN_SOME(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; RCPT_COUNT_TWO(0.00)[2]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[plan-b.pwste.edu.pl:+] X-Rspamd-Queue-Id: 4X6Fmw4YyVz4jZJ Dnia Thu, Sep 12, 2024 at 06:16:18PM +0100, Sad Clouds napisał(a): > Hi, I'm using FreeBSD-14.1 and on this particular system I only have a > single physical network interface, so I followed instructions for > networking vnet jails via epair and bridge, e.g. > > devel > { > vnet; > vnet.interface = "e0b_devel"; > exec.prestart += "/jails/jib addm devel genet0"; > exec.poststop += "/jails/jib destroy devel"; > } > > The issue is bulk TCP performance throughput between this jail and the > host is quite poor, with one CPU spinning 100% in kernel and others > sitting mostly idle. > > It seems there is some lock contention somewhere, but I'm not sure if > this is around vnet, epair or bridge subsystems. Are there > other alternatives for vnet jails? Can anyone recommend specific > deployment scenarios? I've seen references to netgraph which could be > used with jails. Does it have better performance and scalability and > could replace epair and bridge combination? > > Thanks. Have you tried to use kernel built with "options RSS" ? >From my experience it could help in some specific scenarios. -- Marek Zarychta From nobody Sun Sep 15 17:56:54 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X6G2d5mDcz5WWpD for ; Sun, 15 Sep 2024 17:56:57 +0000 (UTC) (envelope-from cryintothebluesky@gmail.com) Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X6G2d3z25z4m4c; Sun, 15 Sep 2024 17:56:57 +0000 (UTC) (envelope-from cryintothebluesky@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-wm1-x32b.google.com with SMTP id 5b1f17b1804b1-42cba8340beso29394435e9.1; Sun, 15 Sep 2024 10:56:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726423016; x=1727027816; darn=freebsd.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=dA6LzmCxBV/8elNsi//MgmiyvV9RKKTFmerxKxOoMvI=; b=UCQto/R+uruu8JCexGBLjEX6+KM3LoG3oEA4crIkbWgXoV4UHfDZL5tmqKOqj3bf2+ BJxUWrSghKNWxfwmW4VP5G8PpbShUzfjqhutsk03g10zPO1qLgCZk1a8gPPYB9bVkGXK BlSkVbhJlE5ICjwVPmVi2mnTALMJZtBa4gu8HMeEC4H3xQLhhVxD+TDVTgcJWUgj7sVL +eIfDKYqbtdHNkUY1C3AMGveffpneMUo/GXLR0YRrhFtuXFOK9G50Sv/oVRr8asB8IdC NnmXRsTEf+l6XNdx/ahM6C4WpNiYP3YWNdaDECJd+Gz2rnpmpe37Yak+l5Dsx3QwYfbd LqSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726423016; x=1727027816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dA6LzmCxBV/8elNsi//MgmiyvV9RKKTFmerxKxOoMvI=; b=iXHgmnIx6ECCEyFfcIiibbFwmKJmQf3m3LhyPbv1tosXH+9FvQ8aE/Z/4FBaE0H67l dr7ZGKHNBLiPp5knxKpileey0zAqtZ0IW5yhUk2hVIQeJtmP2qEFHSTs5AkTj+1X+6BY In4kKxfuGXDfcPTpW5dJtP9PNIAu40ISHNYupdQA1bZujLcgLtAiFvrtCA5qRd9sMqV8 SxOW/VgMUi+reoIXsRvk3xNvcoaGiF48ntq/TPxwenc8TeodXFW/5vZozgE7d9m/tYRi 6zsvnXHjSEeFPZIDG2FiXT4NV8L1S0zA5oT/tg7D7CsoTiNpRs4Xu6kukO4PfVE8V167 Qs6w== X-Forwarded-Encrypted: i=1; AJvYcCXug7ieGEMVFSe7qpPIL5c8KL7waGKf/GHKiZGPIKYDnFe0wnrobQX4huZSriVHgMnCRZvuIpRZy2h9wg==@freebsd.org X-Gm-Message-State: AOJu0YxFwbpgjyqEKc+GOdDGbLzgKqIumuqQEhM72Sorq/BD5pg7mgZo 4erhu8f4Yq38a/Y720GStP8q+so+xKa6nOyQXhk9xCx/DQ7O2MdzLqckOw== X-Google-Smtp-Source: AGHT+IFj3q1mslPmunZIfzULqM8ox8w2G1G5uzTtRkJKQVxxaTunSD/mxzWwP+nhI7vyn5+74gPC2w== X-Received: by 2002:a05:6000:20c3:b0:378:c6d5:e2b3 with SMTP id ffacd0b85a97d-378c6d5e608mr7844277f8f.23.1726423015782; Sun, 15 Sep 2024 10:56:55 -0700 (PDT) Received: from z600.home.lan (28.129.159.143.dyn.plus.net. [143.159.129.28]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-42d9b189d0dsm88389275e9.33.2024.09.15.10.56.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 15 Sep 2024 10:56:55 -0700 (PDT) Date: Sun, 15 Sep 2024 18:56:54 +0100 From: Sad Clouds To: Doug Rabson Cc: Zhenlei Huang , Mark Saad , FreeBSD Net Subject: Re: Performance issues with vnet jails + epair + bridge Message-Id: <20240915185654.b51cfec5aa2520e5b801cc87@gmail.com> In-Reply-To: References: <20240913100938.3eac55c9fbd976fa72d58bb5@gmail.com> <39B2C95D-1E4F-4133-8923-AD305DFA9435@longcount.org> <20240913155439.1e171a88bd01ce9b97558a90@gmail.com> <20240914112516.cfb31bae68ab90b83ca7ad4b@gmail.com> X-Mailer: Sylpheed 3.8.0beta1 (GTK+ 2.24.33; x86_64-pc-linux-gnu) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4X6G2d3z25z4m4c On Sun, 15 Sep 2024 18:01:07 +0100 Doug Rabson wrote: > I just did a throughput test with iperf3 client on a FreeBSD 14.1 host with > an intel 10GB nic connecting to an iperf3 server running in a vnet jail on > a truenas host (13.something) also with an intel 10GB nic and I get full > 10GB throughput in this setup. In the past, I had to disable LRO on the > truenas host for this to work properly. > > Doug. Hello Doug, can you please confirm that you are NOT using if_epair(4)? I imagine you dedicate one of the Intel 10Gb ports to a jail. This is not an option for some of us, so a virtual NIC of some sort is the only option with vnet jails. Other people also mentioned that vnet by itself is not an issue and your test confirms this, however I'm observing poor scalability specifically with the epair virtual NIC. I will be trying netgraph when I have some more time. If there are other alternatives to if_epair then I would be interested to learn about them. From nobody Sun Sep 15 21:00:37 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X6L6Z1bkvz5X353 for ; Sun, 15 Sep 2024 21:00:38 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X6L6Y58RYz415B for ; Sun, 15 Sep 2024 21:00:37 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1726434037; a=rsa-sha256; cv=none; b=CZJRRrtBZklteRPBw9W6A3ngaEZRCUNKYnoVngLheMfu+ky4XS72BLCxhO4V41i4IZ/BcZ 0VmbS4zmizkYH/DU1hX8MlW+3GUmSNtjhplYGoNPI7CpV+NE0gfeneImgzJOvbcVlyVb9Y CcQ984fP26+4tAVwm9SOrWjF/pQ7p2kzmK/APMMT8OYC3R7ZcCKdglN11ifn50O1FpiSbV 70A24nbZYvryhtQqeC1suAg8G/YlKWSGJkVlKh2yWsZQb5OiBsopFRKrAk1HgB+Jkh6up4 Ss/NQl79DHBKTbcJXKDUWCQ+KfOBvoraH94UR/XeEtDWCj8F8MJokecgoDmTYw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1726434037; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=NcPpNKMD0hOGpMuLWHzkqLovhsS+bHrNh8Z4nSWk6wY=; b=T+qb4kD0taLBAbmfyZoo2FiXZ9qI4W+mDCjIYb3lAl7g9diVGz/XaA95Z6oE7iM+RNANdY 1b+fCpGPO5VhFu4JGsOsEKHrXYxtBL2nS9PJT4bbBtcVgVu/tgeVqnofr+ic/kC7rEX9gu F9XEXvbigb2pAHKwirTseBessdHU6PHkUSyDtpuTRc2YqnyqoTmJqWYqlAlmufiW2ewI4X yC+H/vPIk4BEE3GZVebxF/Dpc61+nDLdMxSs/XSf0s8k0UUp9NQkQ5sg4sw9/UFstP3uB0 vMoFsKu5AT9mMDoasL2AqZ4nAD+1odvYi4fooT7wAB7E+mifmCstwy86rYeX6Q== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4X6L6Y4mdFzQfy for ; Sun, 15 Sep 2024 21:00:37 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 48FL0bPD055369 for ; Sun, 15 Sep 2024 21:00:37 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 48FL0b4w055365 for net@FreeBSD.org; Sun, 15 Sep 2024 21:00:37 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202409152100.48FL0b4w055365@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: net@FreeBSD.org Subject: Problem reports for net@FreeBSD.org that need special attention Date: Sun, 15 Sep 2024 21:00:37 +0000 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="17264340370.9be255f.53281" Content-Transfer-Encoding: 7bit --17264340370.9be255f.53281 Date: Sun, 15 Sep 2024 21:00:37 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 254445 | cloned_interfaces="bridge0" does not respect net. Open | 166724 | if_re(4): watchdog timeout Open | 200836 | iovctl(8): Return descriptions in the returned sc Open | 223824 | Panic in ng_base.c (netgraph) Open | 232472 | ixgbe(4): SR-IOV passthru not working on Hyper-V Open | 234073 | ixl(4): Host X710-DA2 drops connect starting bhyv Open | 241106 | tun/ppp: panic: vm_fault: fault on nofault entry Open | 245981 | bnxt(4): BCM57414 / BCM57416 not initializing: bn Open | 257038 | em(4): Panic on HTTP traffic to or from jail thro Open | 257286 | gateway with `ping -6 -e` is ignored Open | 258623 | cxgbe(4): Slow routing performance: 2 numa domain Open | 258850 | lagg(4): interface vanishes when both member inte Open | 261866 | ixgbe(4): Resets media type -> autoselect after s Open | 262024 | em(4): iflib handles bad packets incorrectly Open | 262093 | ixl(4): RX packet errors on Intel X710 after 12.2 Open | 263568 | ix(4): SR-IOV connection lost after loading VM wi In Progress | 118111 | rc: network.subr Add MAC address based interface 17 problems total for which you should take action. --17264340370.9be255f.53281 Date: Sun, 15 Sep 2024 21:00:37 +0000 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8" 
The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.

Status      |    Bug Id | Description
------------+-----------+---------------------------------------------------
New         |    254445 | cloned_interfaces="bridge0" does not respect net.
Open        |    166724 | if_re(4): watchdog timeout
Open        |    200836 | iovctl(8): Return descriptions in the returned sc
Open        |    223824 | Panic in ng_base.c (netgraph)
Open        |    232472 | ixgbe(4): SR-IOV passthru not working on Hyper-V 
Open        |    234073 | ixl(4): Host X710-DA2 drops connect starting bhyv
Open        |    241106 | tun/ppp: panic: vm_fault: fault on nofault entry 
Open        |    245981 | bnxt(4): BCM57414 / BCM57416 not initializing: bn
Open        |    257038 | em(4): Panic on HTTP traffic to or from jail thro
Open        |    257286 | gateway with `ping -6 -e` is ignored
Open        |    258623 | cxgbe(4): Slow routing performance: 2 numa domain
Open        |    258850 | lagg(4): interface vanishes when both member inte
Open        |    261866 | ixgbe(4): Resets media type -> autoselect after s
Open        |    262024 | em(4): iflib handles bad packets incorrectly
Open        |    262093 | ixl(4): RX packet errors on Intel X710 after 12.2
Open        |    263568 | ix(4): SR-IOV connection lost after loading VM wi
In Progress |    118111 | rc: network.subr Add MAC address based interface 

17 problems total for which you should take action.
--17264340370.9be255f.53281-- From nobody Sun Sep 15 23:34:08 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X6PWl6kNZz5VDhC for ; Sun, 15 Sep 2024 23:34:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X6PWl5hNdz4Sfw for ; Sun, 15 Sep 2024 23:34:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1726443251; a=rsa-sha256; cv=none; b=B7QwkXm1BdQJBGNMb0uKTvqPDxiy1vaEpUcFFRd5hndCtyfnoQkvgzwlwKHUp7XhDc18h4 wBR2njYuGu9PBZXCDctmfrROYWdLkURasCBzE9RGI0vbrE19N5+dMvKfn+7Jw+5zUtHdwp D/fwd2o0/+7YHwer3WbK42TM3yoez+/FgsU4cprD6W8FACY97Eclwe9xuG6djF3LojQoNW DXYuSQBLLOCk9jcjG1rIG11Q3+rO8ypT1mjmWBY1cCh0NzjOEfF8cfMfkyME9xb97sfiE4 gFe1fSjBRBxNEGrJCnQDt8ycHrFbIW4JuU2tnNBFRD0y02Gawrr3OUATuyighA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1726443251; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PclqX/SouOwCuT6DmUmvOPe9gXPASPJaIGeW67+iWZs=; b=kujOoxNN+PuqX2IwvjAucIKLYx09HLqb39v/EmkHEWlAsG9UC7hK/uK4VI762WC/+6WFJB VoN3qwybE504OQMIsUfqJn8j54UqRIZOV3w9siNJfve4+CJOI0JxqfHl1Pl5ZtpMkGCQwf qFsZknLWDepafx5H/58UQ3ZptsSCZUH+8PVGQZajtb9o9ipvDv82U5SO3urB+XGWY3VHtx J6g9DsNSeHmI5qOvDqqGcVH47wBnlb81pGL95YvAdZaXeLstN+QhWslCTZRmWKFxIwEcWt IKNVdYDmew16bpRHK2R2XZRHAFYFjoN4m9XIuKzDpNBxMpwJYIOfpxuPo6nsbw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4X6PWl5HxDzX1Y for ; Sun, 15 Sep 2024 23:34:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 48FNYBCZ095120 for ; Sun, 15 Sep 2024 23:34:11 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 48FNYBFV095113 for net@FreeBSD.org; Sun, 15 Sep 2024 23:34:11 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 281391] IPv6 multicast sent to wrong MAC address Date: Sun, 15 Sep 2024 23:34:08 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: antonfb@hesiod.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D281391 antonfb@hesiod.org changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |antonfb@hesiod.org --- Comment #1 from antonfb@hesiod.org --- I have a test program which shows 13.3 multicast reaching 14.1 but the 14.1 packets do not reach 13.3 The program tests ipv4 or ipv6. It was working fine when written for 13.1 years ago. So, anyhow, I think I've hit this problem. --=20 You are receiving this mail because: You are the assignee for the bug.=