From nobody Sun Aug 27 15:43:30 2023 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RYdJN2302z4rLnN for ; Sun, 27 Aug 2023 15:43:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RYdJN11ZGz4HBq for ; Sun, 27 Aug 2023 15:43:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693151012; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=elmuvqolyKU+aebo8zgzYUNCz4LmtBRuPZYuYJ19+lA=; b=PHBymTIAnMJPCPLeBnooAm48LAb4fUEdmpv5HBNkxa55dWIaDxfUcSeQpeubub/O5Weuu1 /oS852U2BOoq8QCs76TSTo9MU2yVkFaRuQtZZHTCUdZfEet9hGsGl6dKqnw1yQk4MXM+IH Z7iniG/LsJffWkHMLZ06xC7Q8Q8qy5LRailCljF6WzoEGgOPhBEkaLLzqaibTwTMZgmtoi QLvm37LTOMLRiCZpx1rOyAb/qmCt7wpkDlZSNPjANYlcJryUAxRl299PniZA+XvruV7+LH +rhMnQw4yNAR9N62iyQTyeETBb0okLIOIM27PvKGgHyooyFZ8P4+lSAxSfgHAQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1693151012; a=rsa-sha256; cv=none; b=iw/uD/5P1IxPKPwJ8ywsEcueFDKoHugjAHlK/Q3gIjJtMMw9wlZTx1EAaiSJ96Npgzdzu2 zHBP+LdSj4/qj9PWyVwkTmDgY8vh9bloBFr9vi12kc6PycaQMI7fS9xcZ35y3M4T/x35fu S8qYdGUEHs0g7APVdgfsTOF0eBWMliTjgC5x8tmoFJjCe36jvrGmvURW89gcmnwK/3sPGu +E3Vwvb9pG1y0D6CzFXvkkvlo9LsSYufDRjL43rxTdkncAmUvk8SRH1T/5wLVP1ibHrJOS kncx4FYXxyxR3tme+v2jfyRDPZRtcjA7IfZloGP0PBjilu356DZrjFI37DPHnQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4RYdJN00cdzXtp for ; Sun, 27 Aug 2023 15:43:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 37RFhVjL065418 for ; Sun, 27 Aug 2023 15:43:31 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 37RFhViS065417 for net@FreeBSD.org; Sun, 27 Aug 2023 15:43:31 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 271366] Invoking IPv6 network device address event may sleep with the following non-sleepable locks held Date: Sun, 27 Aug 2023 15:43:30 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: freebsd@nerdbynature.de X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D271366 Christian Kujau changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |freebsd@nerdbynature.de --- Comment #2 from Christian Kujau --- Same here on amd64, running as a Xen DomU virtual machine: $ uname -rv 15.0-CURRENT FreeBSD 15.0-CURRENT amd64 1500000 #0 main-n265049-8ed0ecf8024= d: Sat Aug 26 14:44:23 UTC 2023=20=20=20=20 dummy@freebsd:/usr/obj/usr/src/amd64.amd64/sys/GENERIC $ grep if /etc/rc.conf ifconfig_DEFAULT=3D"DHCP inet6 accept_rtadv" $ dmesg [...] lo0: link state changed to UP xn0: 2 link states coalesced xn0: link state changed to UP Invoking IPv6 network device address event may sleep with the following non-sleepable locks held: exclusive sleep mutex xnrx_2 (netfront receive lock) r =3D 0 (0xfffffe006a3= 04a78) locked @ /usr/src/sys/dev/xen/netfront/netfront.c:677 stack backtrace: #0 0xffffffff80bcd5c5 at witness_debugger+0x65 #1 0xffffffff80bce75a at witness_warn+0x3fa #2 0xffffffff80d7b39c at in6_update_ifa+0xc0c #3 0xffffffff80da7109 at in6_ifadd+0x1d9 #4 0xffffffff80da3854 at nd6_ra_input+0x1034 #5 0xffffffff80d75c14 at icmp6_input+0x724 #6 0xffffffff80d8e5df at ip6_input+0xc9f #7 0xffffffff80cb14cf at netisr_dispatch_src+0xaf #8 0xffffffff80c93caa at ether_demux+0x17a #9 0xffffffff80c95313 at ether_nh_input+0x393 #10 0xffffffff80cb14cf at netisr_dispatch_src+0xaf #11 0xffffffff80c940e9 at ether_input+0xd9 #12 0xffffffff809cb78c at xn_rxeof+0x5ec #13 0xffffffff809ccad8 at xn_intr+0x48 #14 0xffffffff80b126f9 at ithread_loop+0x279 #15 0xffffffff80b0eab2 at fork_exit+0x82 #16 0xffffffff8102fede at fork_trampoline+0xe lock order reversal: (sleepable after non-sleepable) 1st 0xfffffe006a304a78 xnrx_2 (netfront receive lock, sleep mutex) @ /usr/src/sys/dev/xen/netfront/netfront.c:677 2nd 0xffffffff81aaaba0 in6_multi_sx (in6_multi_sx, sx) @ /usr/src/sys/netinet6/in6_mcast.c:1217 lock order netfront receive lock -> in6_multi_sx attempted at: #0 0xffffffff80bcd18d at witness_checkorder+0xbfd #1 0xffffffff80b645c2 at _sx_xlock+0x62 #2 0xffffffff80d83b61 at in6_joingroup+0x31 #3 0xffffffff80d7b765 at in6_update_ifa+0xfd5 #4 0xffffffff80da7109 at in6_ifadd+0x1d9 #5 0xffffffff80da3854 at nd6_ra_input+0x1034 #6 0xffffffff80d75c14 at icmp6_input+0x724 #7 0xffffffff80d8e5df at ip6_input+0xc9f #8 0xffffffff80cb14cf at netisr_dispatch_src+0xaf #9 0xffffffff80c93caa at ether_demux+0x17a #10 0xffffffff80c95313 at ether_nh_input+0x393 #11 0xffffffff80cb14cf at netisr_dispatch_src+0xaf #12 0xffffffff80c940e9 at ether_input+0xd9 #13 0xffffffff809cb78c at xn_rxeof+0x5ec #14 0xffffffff809ccad8 at xn_intr+0x48 #15 0xffffffff80b126f9 at ithread_loop+0x279 #16 0xffffffff80b0eab2 at fork_exit+0x82 #17 0xffffffff8102fede at fork_trampoline+0xe --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Aug 27 21:00:50 2023 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RYmLV480Dz4rdhk for ; Sun, 27 Aug 2023 21:00:50 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RYmLV1r5Bz3GNj for ; Sun, 27 Aug 2023 21:00:50 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693170050; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=FD15jBAsWEbAmpxXuPG2bGmzL3ujj14/tI1PrirvDyk=; b=cWaSkHZXndzojpz1pagcVijQnXaH5G7joS2Cevg3B17xVcPv6vtYPeU5l4HVUtFgBW431y P+UOi3iaUPhHynuDkJJENsHfS748iVYbgvxhY3srzsK9c5zXJcjITRYH0AX5OG/wxO8lzj Ej3RVNYRP8iYMVUCOSQeXIZ4/9h5Zqi79oCZwAhnTxjykTEZSx/BmMYjdbhmFzFtW+eJXT NEwx8xBBJm9Z8J3Itr5tqCO+AwvRuf0cu+pbNfQ7iqn7jXqCnUYCCqPXZ+ykZLlYwlNoCF FBdmniwuF61Stps6vfKDEU4KWKxAsH76wTsEBqBxZe5C8Xas64tEyEjEyV+j1w== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1693170050; a=rsa-sha256; cv=none; b=pi/e3regun3l/O8m1u/ZpUTy+D5cgjgcweZsbL5owu448xF1aw02s/GPowh15IdzNe4jOI E+x90p9Y7Vjz7T4H2YNgb5TVOVwbxz3QNFAh2iKZDFPlsQhFFrwpcCTPNJ7KGruUyDHzuF 0yl//bdaZJLp7TlFkA6VzMWTAxVXV76/hOG8JBtjuxfsONnRTthyXwhl4bIcZOG6W/vT5y fsQXq/owHh0C9uZBXrASIiBdvrMRiudoIXuCcnL5GKD/g6A5gswfqb4ETKhmc3SsW8/Hox 7QMNxPGvBXrr+9DAfMXuE0gDCpoDyxFT6maxI4WAM3DJsndth/s8nrBhfewScQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4RYmLV0y14zhjy for ; Sun, 27 Aug 2023 21:00:50 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 37RL0oJi022860 for ; Sun, 27 Aug 2023 21:00:50 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 37RL0otT022859 for net@FreeBSD.org; Sun, 27 Aug 2023 21:00:50 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202308272100.37RL0otT022859@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: net@FreeBSD.org Subject: Problem reports for net@FreeBSD.org that need special attention Date: Sun, 27 Aug 2023 21:00:50 +0000 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="16931700500.E1D59F.20175" Content-Transfer-Encoding: 7bit --16931700500.E1D59F.20175 Date: Sun, 27 Aug 2023 21:00:50 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 204438 | setsockopt() handling of kern.ipc.maxsockbuf limi New | 213410 | [carp] service netif restart causes hang only whe Open | 7556 | ppp: sl_compress_init() will fail if called anyth Open | 193452 | Dell PowerEdge 210 II -- Kernel panic bce (broadc Open | 202510 | [CARP] advertisements sourced from CARP IP cause Open | 207261 | netmap: Doesn't do TX sync with kqueue Open | 225438 | panic in6_unlink_ifa() due to race Open | 236888 | ppp daemon: Allow MTU to be overridden for PPPoE Open | 237072 | netgraph(4): performance issue [on HardenedBSD]? Open | 237973 | pf: implement egress keyword to simplify rules ac Open | 238324 | Add XG-C100C/AQtion AQC107 10GbE NIC driver Open | 240944 | em(4): Crash with Intel 82571EB NIC with AMD Pile In Progress | 118111 | rc: network.subr Add MAC address based interface Open | 270912 | dns/unbound: issues with ASLR 14 problems total for which you should take action. --16931700500.E1D59F.20175 Date: Sun, 27 Aug 2023 21:00:50 +0000 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8" 
The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.

Status      |    Bug Id | Description
------------+-----------+---------------------------------------------------
New         |    204438 | setsockopt() handling of kern.ipc.maxsockbuf limi
New         |    213410 | [carp] service netif restart causes hang only whe
Open        |      7556 | ppp: sl_compress_init() will fail if called anyth
Open        |    193452 | Dell PowerEdge 210 II -- Kernel panic bce (broadc
Open        |    202510 | [CARP] advertisements sourced from CARP IP cause 
Open        |    207261 | netmap: Doesn't do TX sync with kqueue
Open        |    225438 | panic in6_unlink_ifa() due to race
Open        |    236888 | ppp daemon: Allow MTU to be overridden for PPPoE 
Open        |    237072 | netgraph(4): performance issue [on HardenedBSD]?
Open        |    237973 | pf: implement egress keyword to simplify rules ac
Open        |    238324 | Add XG-C100C/AQtion AQC107 10GbE NIC driver
Open        |    240944 | em(4): Crash with Intel 82571EB NIC with AMD Pile
In Progress |    118111 | rc: network.subr Add MAC address based interface 
Open        |    270912 | dns/unbound: issues with ASLR

14 problems total for which you should take action.
--16931700500.E1D59F.20175-- From nobody Mon Aug 28 01:51:38 2023 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RYtpD3KGYz4rt72 for ; Mon, 28 Aug 2023 01:51:48 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RYtpD2lwbz3dQM; Mon, 28 Aug 2023 01:51:48 +0000 (UTC) (envelope-from zlei@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693187508; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=AlU6kgzc+vIyA8yu8WeFkvXj9Elxgjh2RjMPdoPwyNQ=; b=T8rfgsN6wXTdajnMqXB0txfvQa645E2PtsisMez8E2XYpql5F7gAoQhdi2QUR22inaLjyY GJd9gA978i+TKWVocLGc9Cq2CYkaZjpKlbtsRvQYmGCxZxk680R/VQw/CExC6Kzotu8SnN E4UPCYruPf6OC+sHJgFJpfLhtoj8cIFPwaI2ophVtsP1HUUPgHn1G2xW/VKxJwDUhN+Yf1 w/q1qjBPJEsPVWM65Y3uVdvSERR80NqtX+oiuzRkYX/299ULfQbN2/E9SsqlcRiG9ImFMm w/73UWZjshuCuqyYePBRv3oLrQMWIiEJlxqaisS5ZEgVo5fnUT10KkKrNq4Pgg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693187508; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=AlU6kgzc+vIyA8yu8WeFkvXj9Elxgjh2RjMPdoPwyNQ=; b=gw4uBz8MDKPbs7IJUYM9+DhfBo4s4pZSkU3wlKsZAkEkEronbGiIDuwZ2pPa+QcgDkT/0v p0a0XbrHjNTKJj95zLIP+6HAqRDgHDiO/cVMDBoLqN+1w/Mg35kvE30GvKYiOMZ35XijGh mAO0vPhOTrLeGBkL2Y9n8tncJ59Kf6dPWlUAPuq1zI3SsYZjE6SPX409x+V71y/IMnyfoq o05G/eU1devKBnILHc0oAqNEa+Ki+qHo7DPbqsMUXC8eFWt+OVc1fiySAszz8uItO2tYqT YZdGO0GOAdBuhqHtXLKQvW4WvAJRFcCre0vK6NGJNq6OeIt95/wj3H5yoUWJjw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1693187508; a=rsa-sha256; cv=none; b=CCJvvkwM+tWnXyWoAb9xkfWZJQSET2Lq0BnZ/u+hxlp/OgRM2p8JzNXXbrnkUyHvh1Ylyh E15ac5QnRhVL8U66+f4TsxCIWivrI205U1fbf8h9tIBDdnEehwI+zUPUI3cH41bsedJal9 CpzrYz29B1b9eZg9OEq6OKQRlUtkEx4+nrmA+jA8PW2CasogRJeJn5EeUtCOWs73/eZck6 vwmIg9XOqHzX71DrK6qfQgHTOjvBITdP208IRwgVIx9Hx9OrF2nB1dsPET/Z+DWml3Ssro CP1d6GRhQk58lgq0VX1h8Oimm0AQCta9v6lvXqS6RCX9eUgUQtIVxZ8niMNPLw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from smtpclient.apple (unknown [IPv6:2001:19f0:6001:9db:98f0:9fe0:3545:10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: zlei/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4RYtpC2b0gz1bYn; Mon, 28 Aug 2023 01:51:46 +0000 (UTC) (envelope-from zlei@FreeBSD.org) From: Zhenlei Huang Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.4\)) Subject: About IFNET_PCP_NONE Message-Id: Date: Mon, 28 Aug 2023 09:51:38 +0800 Cc: freebsd-net To: Konstantin Belousov X-Mailer: Apple Mail (2.3696.120.41.1.4) Hi Konstantin, I was just about going to open a PR for = https://reviews.freebsd.org/D39536 and realized I might made wrong assumption. I thought IFNET_PCP_NONE is something like IEEE8021Q_PCP_BE but I second = why not use IEEE8021Q_PCP_BE but a new const IFNET_PCP_NONE. So despite its naming IFNET_PCP_NONE, is it actually a flag to let = specific interface completely bypass (disable) PCP processing? The const IFNET_PCP_NONE is defined in sys/net/if.h with=20 ``` #define IFNET_PCP_NONE 0xff /* PCP disabled */ ``` Best regards, Zhenlei From nobody Mon Aug 28 07:32:35 2023 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZ2Mc5YM4z4r01V; Mon, 28 Aug 2023 07:32:44 +0000 (UTC) (envelope-from weh@microsoft.com) Received: from APC01-SG2-obe.outbound.protection.outlook.com (mail-sgaapc01on2105.outbound.protection.outlook.com [40.107.215.105]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZ2Mb35C3z3CWY; Mon, 28 Aug 2023 07:32:43 +0000 (UTC) (envelope-from weh@microsoft.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=microsoft.com header.s=selector2 header.b=PLxewIZe; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (mx1.freebsd.org: domain of weh@microsoft.com designates 40.107.215.105 as permitted sender) smtp.mailfrom=weh@microsoft.com; dmarc=pass (policy=reject) header.from=microsoft.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SJ9KOURO2/nY1FBfPCOV3FtpHkDkAjbehnOxQMgyK/6VWVPrHigeY0kDDtwak89x6zT5STBRyUApwKvQnjDtLRVAT1RJspBPU4ROdkrEfKwLtG1SfrVQombxa/5F1qRpEh1DtNux7PjmtAHnrBM6seOmewD5IylOtxp3mx4uRr15+Fm9jjUZgJ73QLjEpduVA81PCS1muUaBeJLW/O13JlJ/t/E/mdH4wFVOgsIPbb6EVMMHxXG+z72OQeOndlYioRpRNQUzKnYceNDPW14HIlwyWOD0Hbb+eWOVSlVrT1u580HsXB4IR717dNpVzPvvVGkSoM+n+IwyJQWCRtHTjg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=P2U6L2iDPUzMhus1Cndi4Bym8K6J4WEs8tq9xnXEo80=; b=Nr8R5e0vx60QxJuLYGQlgM6mZxFF+AF5KR+2n1x/S3/96e1S+QRqK1zEbuJv6cAlikHY48K1gVeJdzxqjNGF1QQXp9ky1M4lDyrsczFlhzBCY2+oVdEP8cAvHSvjdRJV93tXXmvZqdSRG8kj5wjRfayyIRBvQDTk8wPJinNePSRjVp/6QFsE2sbfgTU1ata1NNx5sbqsAAyjDQu/1Gn3oOvAukwZ9RiB00M9W2EyOgiCGVpBMgHAUY5yC/jxiedVh1f+MYkELhWKwGFfjWYMOXtrNSguHGYLaTPHEwAOGwsuki0kVyWurke/rOHrci/jA1mwNUDC+1/ebHz0Z4yKgg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=P2U6L2iDPUzMhus1Cndi4Bym8K6J4WEs8tq9xnXEo80=; b=PLxewIZeRWegUK7zyfhm12yOJ8i6i+C6GMUfC96JZwhHTAr6lwI18Yi3shqLwrzS9VNuCGP1ABFpzgg5tjr7NPXJVOG8h4PO7BTOAmTdGtcgmppWNaSQegApO79hunnPdqDMERVio3270F78+zRqy53No/F2Ohgcq2Ketxo9f3c= Received: from SI2P153MB0441.APCP153.PROD.OUTLOOK.COM (2603:1096:4:fc::7) by SEZP153MB0791.APCP153.PROD.OUTLOOK.COM (2603:1096:101:a6::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.6; Mon, 28 Aug 2023 07:32:36 +0000 Received: from SI2P153MB0441.APCP153.PROD.OUTLOOK.COM ([fe80::7bae:a915:919b:6f83]) by SI2P153MB0441.APCP153.PROD.OUTLOOK.COM ([fe80::7bae:a915:919b:6f83%7]) with mapi id 15.20.6768.005; Mon, 28 Aug 2023 07:32:36 +0000 From: Wei Hu To: "freebsd-hackers@FreeBSD.org" CC: "freebsd-net@FreeBSD.org" Subject: Very slow scp performance comparing to Linux Thread-Topic: Very slow scp performance comparing to Linux Thread-Index: AdnZfvJbaS4mlPy2QcSWspi9tSFg8Q== Date: Mon, 28 Aug 2023 07:32:35 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=c0bd6cc2-296c-4e45-ab32-f5c2dce362b4;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2023-08-28T07:11:28Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SI2P153MB0441:EE_|SEZP153MB0791:EE_ x-ms-office365-filtering-correlation-id: f3c5db47-01c9-41fa-5405-08dba798f2df x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: e+NCVxVbVI5oa5VonIW3B1F3HF47QEtE7ZsxQlo+j6CN5Et2d/nn41wWUA0qBYF0a1A5WNMWch+XID8VtDAIexn2zTG7gMyg84JxYqDFYyrhrgQD4IEFGYeVBfZF9D7M0QrwIVCS59Djs/QyNtX1mmaVrOhqflclyWIGgWW00hkyA3poQKSQA0m5gy2pppieCdw1TBwSjawXapTfMoQIlQoCx3YZ4m8HqqIj9/ln5fQO/e+VOJi08ps0txwWz85Mf65/OB8UMxCpt71znOxtrQHjCVCVFI2yRZ0jKab4fbenFOiIq2U01BC2iaFXqqPiyJEh+76r9P2G/TXcqEjc0sYOFl0vwT3j8AVrdjmju6YpJjROXQchVitdQW0C9vriHPI0i4p7gbrA3s9E5FiJugBCy8MYxpts40vuwKspCJTyZm2LCUCROUEkZC6ghSF7QggM0aFbNKfKUjYCErLCqW7kT3m7rP78Bnu5DqhkNwJbbB4VyWwUu1nztGP4d/ArMD0CjunEVA26zk4yMrV1gh/nJypgCLxRWyuctl4yIs6Nl/U9nXC1m7oRqh4nPohXw9RbP3eid9igSsTCrvCCWH6wgyIlk0rj0lTV6QHKqDKAKGMFAxACmR6vOC1YelAIRFndZBiKw7nrqNLiYj6vhYgoyX5eEjTzgXcd8FfngjhCmNqdQaexMzTtFp7EUSAbuKs5uvwH2g0ACg1S8bhtuQ== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SI2P153MB0441.APCP153.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230031)(376002)(396003)(366004)(39860400002)(136003)(346002)(186009)(451199024)(1800799009)(4326008)(8676002)(8936002)(5660300002)(316002)(6916009)(2906002)(66556008)(66446008)(64756008)(33656002)(66476007)(66946007)(76116006)(450100002)(52536014)(12101799020)(41300700001)(6506007)(7696005)(26005)(9686003)(55016003)(38100700002)(38070700005)(122000001)(82950400001)(478600001)(82960400001)(71200400001)(8990500004)(86362001)(10290500003);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?I5/G9n3AA2iOceDiVGp6Ue2awkO55hUw5RJUoY50bWLif9W8git5f3JuIgsx?= =?us-ascii?Q?BHkFh9nxFcqNBaxDPprUPLfYcuhG6ZUJtmbItR6ITsIIaK4j09ZY26C2HTRj?= =?us-ascii?Q?Kh2ijgsNSuHsEsmLuSF5VX3EbC/ss7JMd4LCzolTf8Xy9z109Sueb6B7Ff1k?= =?us-ascii?Q?ijpb3e4Hx6H+paD5pRX1UKD/z3370Nxu8fj8X3yy1DZWGdPIEG25OoXpLd6U?= =?us-ascii?Q?bNdillJCy9VYlx891yCeWFuu3p1H68xcBX465jlH48rdr8bn1eGjU/pq1h/c?= =?us-ascii?Q?jpm7oYwgRBvYbFuUDmfY6G151jwrXhct3RFOjPHckr94vikwT6uhglbLp5eM?= =?us-ascii?Q?OiNCiWYZbS7hVUtHzPrByer/0r1fre/tc1CX2MgbnCb6MEvGQ0oYUhkf31Kd?= =?us-ascii?Q?Fj/+9PLZ8RMK67UsSe9Ges1mHMbuSriQb9hSMCHLwhrS0xnuznaIdXQ2GWGA?= =?us-ascii?Q?GvrihEviwvOPiOvVa+nQInpQhGHwMZpDFLEjpGq+zwkGmG8LnL2Vb+3kXs4n?= =?us-ascii?Q?mkI1pp7q10zNTlhO76XWNvRf8G2s9ahdbpdXZjPfJxxWo8Weux6vAyvBkSkd?= =?us-ascii?Q?7c/wmkBHY7wMAljTynq0BUiBdZGM5Il1rGeX0DP5x+wPdCYV+Q6rpAmbOHOR?= =?us-ascii?Q?uHSq5J+ZugEii6Rza7H8mP2hm6HwciBVTG2BFxriBZ90Pn1T4axohC/zbCG/?= =?us-ascii?Q?vsPk7pXGLeXHf0aQqvabTubvLCWpaA8Os94Z3Y8E8MzMA+wIlqYQG9q+bDTf?= =?us-ascii?Q?uhqhLjwKk24/Da07BClgbP1gNENbFAQLHjNo2ndHSwNrsk2u90STlB0fWlOL?= =?us-ascii?Q?MG/qK5+xccTwRBWnTjGJWbcReYHkuZBes5QARfgAHXctoVAS1WYXzDZWxdPv?= =?us-ascii?Q?9oKHpi1PQWvXn6C7fZx6IQBen0Ls1yd868Csmnzr+aEldqRCeuI3cLQNNgpC?= =?us-ascii?Q?PaFVCnZX7UWm/NkGa7p0bTSs7NBYA1tSbo0QCZnUojXbE/sOwaFJ3ff9G5+L?= =?us-ascii?Q?dOG4D7pq5f/bPAQ++zHZwxgLoQrbtKg7BwV90wAyBkCnjTZE4NbJHpf1tLwL?= =?us-ascii?Q?7vj7RvKH0xvHZ7r02NCtZiirHvOx8qCbH2Tkmh12yuDrxravS1bTNYWMKsNs?= =?us-ascii?Q?BjZqL24yv5thefvTTHfq2s6ZRfC0OEEOrp4/OelFB/zAvFOPJ9l4A+oQrU2i?= =?us-ascii?Q?EptOcU1J9rrrkywf3/mBpf+PhH4NeDYfSInfQSsHSYuAarJlpouifjQcl61e?= =?us-ascii?Q?TVY6LnFGWk2YPICfI5GSBZoCzommke8kW2/vHC3+Sn9uKWzhtIld8iLbKV8N?= =?us-ascii?Q?Eg/AMMqtJl1o3SEZy5IN43ciYHD751898Nq9nSl4R829IG1TqmdjOMh7/BY3?= =?us-ascii?Q?R3xshKZu+lrUsNI7WUKQButO4AyUUlO+QS0HDx53SJcO0nYlfDspQXgc8GYo?= =?us-ascii?Q?/WNIUcR4faMqQ+1gBx0q4SuZgyTQye9C2RGUkL5sc9/p/29yjx1FnqvYNzuo?= =?us-ascii?Q?fZAs6Kf2E2L1k0glReDgwb/IRgkChh8pWx4Q5PgPthz4jncosg++t9sG5aew?= =?us-ascii?Q?TRJFd+Je6ybZ+vbTYGs=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SI2P153MB0441.APCP153.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: f3c5db47-01c9-41fa-5405-08dba798f2df X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Aug 2023 07:32:35.4400 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 4yHkZoylDoawJoZRxZqVC+dRmVDZNOIuwoyCBrtkvXO29a39erZY8TEy1VHPU6/JZZQU7tiqm2IqooTj9GJ4cg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SEZP153MB0791 X-Spamd-Result: default: False [-9.00 / 15.00]; WHITELIST_SPF_DKIM(-3.00)[microsoft.com:d:+,microsoft.com:s:+]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DWL_DNSWL_LOW(-1.00)[microsoft.com:dkim]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[microsoft.com,reject]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; R_DKIM_ALLOW(-0.20)[microsoft.com:s=selector2]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org,freebsd-net@freebsd.org]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[40.107.215.105:from]; FROM_HAS_DN(0.00)[]; TO_DN_EQ_ADDR_ALL(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; DKIM_TRACE(0.00)[microsoft.com:+]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.215.105:from] X-Spamd-Bar: -------- X-Rspamd-Queue-Id: 4RZ2Mb35C3z3CWY Hi, When I was testing a new NIC, I found the single stream scp performance was= almost 8 time slower than Linux on the RX side. Initially I thought it mig= ht be something with the NIC. But when I switched to sending the file on lo= calhost, the numbers stay the same.=20 Here I was sending a 2GB file from sender to receiver using scp. FreeBSD i= s a recent NON-DEBUG build from CURRENT. The Ubuntu Linux kernel is 6.2.0. = Both run in HyperV VMs on the same type of hardware. The FreeBSD VM has 16 = vcpus, while Ubuntu VM has 4 vcpu. Sender Receiver throughput Linux FreeBSD 70 MB/s Linux Linux 550 MB/s FreeBSD FreeBSD 70 MB/s FreeBSD Linux 350 MB/s FreeBSD localhost 70 MB/s Linux localhost 550 MB/s >From theses test, it seems I can rule out the issue on NIC and its driver. = Looks the FreeBSD kernel network stack is much slower than Linux on single = stream TCP, or there are some problem with scp? I also tried turning on following kernel parameters on FreeBSD kernel. But = it makes no difference, neither do the other tcp cc algorithms such as htcp= and newreno. net.inet.tcp.soreceive_stream=3D"1" net.isr.maxthreads=3D"-1" net.isr.bindthreads=3D"1" net.inet.ip.intr_queue_maxlen=3D2048 net.inet.tcp.recvbuf_max=3D16777216 net.inet.tcp.recvspace=3D419430 net.inet.tcp.sendbuf_max=3D16777216 net.inet.tcp.sendspace=3D209715 kern.ipc.maxsockbuf=3D16777216 Any ideas? Thanks, Wei From nobody Mon Aug 28 07:54:11 2023 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZ2rW42Rhz4rG7c for ; Mon, 28 Aug 2023 07:54:19 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZ2rW2LkNz3GHf; Mon, 28 Aug 2023 07:54:19 +0000 (UTC) (envelope-from kostikbel@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.17.1/8.17.1) with ESMTPS id 37S7sBm6024820 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Mon, 28 Aug 2023 10:54:14 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 37S7sBm6024820 Received: (from kostik@localhost) by tom.home (8.17.1/8.17.1/Submit) id 37S7sBmi024818; Mon, 28 Aug 2023 10:54:11 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 28 Aug 2023 10:54:11 +0300 From: Konstantin Belousov To: Zhenlei Huang Cc: freebsd-net Subject: Re: About IFNET_PCP_NONE Message-ID: References: List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on tom.home X-Rspamd-Queue-Id: 4RZ2rW2LkNz3GHf X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] On Mon, Aug 28, 2023 at 09:51:38AM +0800, Zhenlei Huang wrote: > Hi Konstantin, > > > I was just about going to open a PR for https://reviews.freebsd.org/D39536 and > realized I might made wrong assumption. > > I thought IFNET_PCP_NONE is something like IEEE8021Q_PCP_BE but I second why not > use IEEE8021Q_PCP_BE but a new const IFNET_PCP_NONE. > > So despite its naming IFNET_PCP_NONE, is it actually a flag to let specific interface > completely bypass (disable) PCP processing? > > The const IFNET_PCP_NONE is defined in sys/net/if.h with > ``` > #define IFNET_PCP_NONE 0xff /* PCP disabled */ > ``` I fail to understand your question. IFNET_PCP_NONE is a value that means that no 802.1q prio is inserted into the packet. Otherwise, non-vlan traffic is tagged with the priority. IEEE8021Q_PCP_BE is a name of one of the priorities, it seems from my code reading. From nobody Mon Aug 28 14:01:56 2023 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZC0r2GWwz4rZnq; Mon, 28 Aug 2023 14:02:04 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZC0r0Cq3z4MqG; Mon, 28 Aug 2023 14:02:03 +0000 (UTC) (envelope-from mike@sentex.net) Authentication-Results: mx1.freebsd.org; none Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.17.1/8.16.1) with ESMTPS id 37SE1udR081946 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL); Mon, 28 Aug 2023 10:01:56 -0400 (EDT) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:5558:f64a:f719:8ead] ([IPv6:2607:f3e0:0:4:5558:f64a:f719:8ead]) by pyroxene2a.sentex.ca (8.16.1/8.15.2) with ESMTPS id 37SE1uP0028691 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Mon, 28 Aug 2023 10:01:56 -0400 (EDT) (envelope-from mike@sentex.net) Message-ID: <73af5253-55ba-13d1-6d31-9bc96233e7d5@sentex.net> Date: Mon, 28 Aug 2023 10:01:56 -0400 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.14.0 Subject: Re: Very slow scp performance comparing to Linux Content-Language: en-US To: Wei Hu , "freebsd-hackers@FreeBSD.org" Cc: "freebsd-net@FreeBSD.org" References: From: mike tancsa In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.84 on 64.7.153.18 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA] X-Rspamd-Queue-Id: 4RZC0r0Cq3z4MqG On 8/28/2023 3:32 AM, Wei Hu wrote: > Hi, > > When I was testing a new NIC, I found the single stream scp performance was almost 8 time slower than Linux on the RX side. Initially I thought it might be something with the NIC. But when I switched to sending the file on localhost, the numbers stay the same. > Just curious, how does iperf3 perform in comparison ?     ---Mike From nobody Mon Aug 28 14:25:01 2023 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZCWV58rFz4rbfK for ; Mon, 28 Aug 2023 14:25:10 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZCWV4fBPz4RBX; Mon, 28 Aug 2023 14:25:10 +0000 (UTC) (envelope-from zlei@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693232710; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=WJyBDK4GnzXe2QPyWi0YhqrIf8Q6BnjasdK4N9shaBc=; b=GuaRMKuipSvUhEhzxbm7/VKZsKz+MOq6RO7zPDrTWQW5NmwZunSUX/QWeelFs+BZQHz8YK vwjG9JbH4AiTq/WptvWuo62cruIkT1NlsB5Znjoq/ug7ch2bLz4D8HlCTelARe3J17qCRs l3INtD8CJY+vXOWg3B3vVDAFjCwVpRHlEfBV8fN2sLKeIDuMU6ea0nhlRMSVyRHN6FG5hI YluHyN/QNeK6RWm4Q/bvGZFjj5wdscAznIO4THYpH4izehbSwBFnBTV8/kwi/8j/DK1uK/ WJsiaucFh53Qgcyepw2s5w+rntjfYstFKrYv0KN57oR35AzkzLjaknanXYxM3Q== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1693232710; a=rsa-sha256; cv=none; b=FR0rmOngT32b44STqYxSRoe/5gs1zmMWf2gkr/BjcZoPZBzJ8Kh1XzOiMsaEQhnSkPfoZt Gwa6WfuJdnKf2F6WWJfWtdy19uCkLpzL53xC57QcUVIv0kKcBaElkRCU0qclXz6f8TDB1c ECGz/7d5kEJnMzTB66mtF8TudrUQKNwjGoISfQfEfQISBiolb+xDPyzMYwtfzgifytt0Gg IBM+ZLpf6XfNx6uV8T6YqnqhzGC+yDfgfFb7tGCU6lDGqlj+NEs3//aq9ah7XS5ndAiC7E +fo9XjInlCc/zepYHplca3goNslObKSQA0UCxBowvaqM2LnrlCFmJzcxbGlCCw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693232710; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=WJyBDK4GnzXe2QPyWi0YhqrIf8Q6BnjasdK4N9shaBc=; b=LPwuw2PdQTX2fa+HfXKa9Kt4ldSYElhfQnzZ+MNvxkvl7W+40jIQgIReET9eTnKCTzHOfz B+8ZeRy8sFs8jrRlhffGQ4+FoTtRtz6ParCcT00g3zer0Pk2JPvx2iBIXXQQCr4pqHKmM0 SyZqfd5hZErWtxeUUXy5xeQ4OgB044eI9nbaRPe3SE3iD3o7nmykeNUZE0GsAUH/4/uleG 89luEyZaue9fU3l+sQ2UZW+MNvOYqp3JzU84wX1+AnkiE6GJeicW8QM3spZIwENikh7hIC etu8ZZK9457Ug7LXJ8ywQ/XRiY/dlxYf7i+Z6wBtqFszPiD9mF6sbsWbNNOUXg== Received: from smtpclient.apple (ns1.oxydns.net [45.32.91.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: zlei/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4RZCWT4j9Hz6ch; Mon, 28 Aug 2023 14:25:09 +0000 (UTC) (envelope-from zlei@FreeBSD.org) From: Zhenlei Huang Message-Id: <20ECDA33-2638-4CFB-8740-5E8BF1F8072B@FreeBSD.org> Content-Type: multipart/alternative; boundary="Apple-Mail=_01138789-003C-4029-A9FC-14A405EA3F7E" List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.4\)) Subject: Re: About IFNET_PCP_NONE Date: Mon, 28 Aug 2023 22:25:01 +0800 In-Reply-To: Cc: freebsd-net To: Konstantin Belousov References: X-Mailer: Apple Mail (2.3696.120.41.1.4) --Apple-Mail=_01138789-003C-4029-A9FC-14A405EA3F7E Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On Aug 28, 2023, at 3:54 PM, Konstantin Belousov = wrote: >=20 > On Mon, Aug 28, 2023 at 09:51:38AM +0800, Zhenlei Huang wrote: >> Hi Konstantin, >>=20 >>=20 >> I was just about going to open a PR for = https://reviews.freebsd.org/D39536 and >> realized I might made wrong assumption. >>=20 >> I thought IFNET_PCP_NONE is something like IEEE8021Q_PCP_BE but I = second why not >> use IEEE8021Q_PCP_BE but a new const IFNET_PCP_NONE. I meant ``` int ether_output_frame(struct ifnet *ifp, struct mbuf *m) { uint8_t pcp; pcp =3D ifp->if_pcp; if (pcp !=3D 0 /* IEEE8021Q_PCP_BE */ && ifp->if_type !=3D = IFT_L2VLAN && !ether_set_pcp(&m, ifp, pcp)) return (0); ... } ``` >>=20 >> So despite its naming IFNET_PCP_NONE, is it actually a flag to let = specific interface >> completely bypass (disable) PCP processing? >>=20 >> The const IFNET_PCP_NONE is defined in sys/net/if.h with=20 >> ``` >> #define IFNET_PCP_NONE 0xff /* PCP disabled */ >> ``` > I fail to understand your question. >=20 > IFNET_PCP_NONE is a value that means that no 802.1q prio is inserted = into > the packet. Otherwise, non-vlan traffic is tagged with the priority. Think about the following case: 1. Set interface's PCP to IFNET_PCP_NONE, application / firewall provide = per-flow PCP, should the traffic be tagged with the priority ? >=20 > IEEE8021Q_PCP_BE is a name of one of the priorities, it seems from my > code reading. Best regards, Zhenlei --Apple-Mail=_01138789-003C-4029-A9FC-14A405EA3F7E Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii

On Aug 28, 2023, at 3:54 PM, Konstantin Belousov <kostikbel@gmail.com>= wrote:

On Mon, Aug 28, 2023 at 09:51:38AM +0800, Zhenlei Huang = wrote:
Hi = Konstantin,


I was just about = going to open a PR for https://reviews.freebsd.org/D39536 and
realized I might made wrong assumption.

I thought IFNET_PCP_NONE is something like IEEE8021Q_PCP_BE = but I second why not
use IEEE8021Q_PCP_BE but a new const = IFNET_PCP_NONE.

I meant

```
int
ether_output_f= rame(struct ifnet *ifp, struct mbuf *m)
{
  =   uint8_t pcp;

    = pcp =3D ifp->if_pcp;
    if (pcp !=3D 0 /* = IEEE8021Q_PCP_BE */ && ifp->if_type !=3D IFT_L2VLAN = &&
        !ether_set_pcp(&m, = ifp, pcp))
        return = (0);

...

}
```


So despite its naming IFNET_PCP_NONE, is it actually a flag = to let specific interface
completely bypass (disable) PCP = processing?

The const IFNET_PCP_NONE is = defined in sys/net/if.h with 
```
#define IFNET_PCP_NONE 0xff   /* PCP disabled */
```
I fail to understand your question.

IFNET_PCP_NONE = is a value that means that no 802.1q prio is inserted into
the packet. =  Otherwise, non-vlan traffic is tagged with the priority.

Think = about the following case:

1. Set = interface's PCP to IFNET_PCP_NONE, application / firewall provide = per-flow PCP, should
the traffic be tagged with = the priority ?



IEEE8021Q_PCP_BE is a name of one of the priorities, it seems = from my
code = reading.


Best regards,
Zhenlei

= --Apple-Mail=_01138789-003C-4029-A9FC-14A405EA3F7E-- From nobody Tue Aug 29 01:23:33 2023 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZV7L3Wkkz4s9KV for ; Tue, 29 Aug 2023 01:23:42 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZV7K6vJqz3JWR; Tue, 29 Aug 2023 01:23:41 +0000 (UTC) (envelope-from kostikbel@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.17.1/8.17.1) with ESMTPS id 37T1NXcY084215 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Tue, 29 Aug 2023 04:23:36 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 37T1NXcY084215 Received: (from kostik@localhost) by tom.home (8.17.1/8.17.1/Submit) id 37T1NXlC084214; Tue, 29 Aug 2023 04:23:33 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 29 Aug 2023 04:23:33 +0300 From: Konstantin Belousov To: Zhenlei Huang Cc: freebsd-net Subject: Re: About IFNET_PCP_NONE Message-ID: References: <20ECDA33-2638-4CFB-8740-5E8BF1F8072B@FreeBSD.org> List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20ECDA33-2638-4CFB-8740-5E8BF1F8072B@FreeBSD.org> X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on tom.home X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] X-Rspamd-Queue-Id: 4RZV7K6vJqz3JWR On Mon, Aug 28, 2023 at 10:25:01PM +0800, Zhenlei Huang wrote: > > > > On Aug 28, 2023, at 3:54 PM, Konstantin Belousov wrote: > > > > On Mon, Aug 28, 2023 at 09:51:38AM +0800, Zhenlei Huang wrote: > >> Hi Konstantin, > >> > >> > >> I was just about going to open a PR for https://reviews.freebsd.org/D39536 and > >> realized I might made wrong assumption. > >> > >> I thought IFNET_PCP_NONE is something like IEEE8021Q_PCP_BE but I second why not > >> use IEEE8021Q_PCP_BE but a new const IFNET_PCP_NONE. > > I meant > > ``` > int > ether_output_frame(struct ifnet *ifp, struct mbuf *m) > { > uint8_t pcp; > > pcp = ifp->if_pcp; > if (pcp != 0 /* IEEE8021Q_PCP_BE */ && ifp->if_type != IFT_L2VLAN && > !ether_set_pcp(&m, ifp, pcp)) > return (0); > > ... This is wrong. PCP_BE is just one of the priorities, that should allowed to be specified in the 802.1q pseudo-vlan header. > > } > ``` > > >> > >> So despite its naming IFNET_PCP_NONE, is it actually a flag to let specific interface > >> completely bypass (disable) PCP processing? > >> > >> The const IFNET_PCP_NONE is defined in sys/net/if.h with > >> ``` > >> #define IFNET_PCP_NONE 0xff /* PCP disabled */ > >> ``` > > I fail to understand your question. > > > > IFNET_PCP_NONE is a value that means that no 802.1q prio is inserted into > > the packet. Otherwise, non-vlan traffic is tagged with the priority. > > Think about the following case: > > 1. Set interface's PCP to IFNET_PCP_NONE, application / firewall provide per-flow PCP, should > the traffic be tagged with the priority ? Yes, it should, but only for packets from the specified flows. > > > > > > IEEE8021Q_PCP_BE is a name of one of the priorities, it seems from my > > code reading. > > > Best regards, > Zhenlei > From nobody Tue Aug 29 16:26:05 2023 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZt8k4YnBz4rxbP for ; Tue, 29 Aug 2023 16:26:14 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZt8k3pWTz3bbK; Tue, 29 Aug 2023 16:26:14 +0000 (UTC) (envelope-from zlei@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693326374; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=DZHQ+BrG8BKxbwxMTjG2fdPoYtbsbYYAv1hYIk1WvcE=; b=YE58iKl5UaVGMBjNY4duwALm2x//P+eStKALVrPh7pUKBZMZ1A0YqjQCyW3ki7RQ6BUEuK 6zti7BmxWpRjLyRrar8laAm3Va0i0cZWJgMjnfwcsoThQ0iLNEWom5Xsn7Yi/G97Z7MYCq q6Avuo3Atpf/4uzOor3osh96yl5WUqr0E1EPlnecDaKtamDaTj6zggnOJgeilIYnJNso6i s3QzuZ8lCCYtg4Q4FIC2chB92cP/1KPAwSTTLjTp9elYf5LMvzPZ+E600UaCD0sU2amuWT b3FBAu76smZm5ASLjtKAOYXRADuQZ4YeZJbD8EE/NSgrHvZupUFGbgr85JIVkg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1693326374; a=rsa-sha256; cv=none; b=BmIKl5fUYxRSdztIbRaqLDKvF25w+Is6Icq00xbQLuAm6QqvutWtPsPN5x6d4qALUmf4JH 455O+wS05Qw9SQ2jDvOpY7f25mVSP3efCdAKU9guXRK4INHv+UoPX3fCxFNrnlvQQYPBXS KnfaBlLaf7B3M/pHRDjuYaAh9E3gQNLNy2BloxHAWTzvA4iUfgEOri3vzwdYviqUfHRewa OoZeNpsx+kC1e1oZyUDVIsapN8IdhT7lOK1X4rStxJNdAJTWw3aJsmcISIbo5VRTL5FFzm Qb0b9l2tZdkUpmIWssglHt7Ak89n4+g45VZB6Yi64Ts0pBk7Lhh5nfCV2Oqddw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693326374; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=DZHQ+BrG8BKxbwxMTjG2fdPoYtbsbYYAv1hYIk1WvcE=; b=Ml/Z/167863x+APvyEdsslPPFk6iltp/cF9b3oH0hyoJ0dAl1G+x2jCJ1A2U2wPHpi5ZeI uJpUk1cHt9bWMxWfb9JOWnYF9yMTuxCApvcdMk7aeiOT/K/X/py/67NT+drcXlTOTcsbk7 WgEHKjg9L8vOOeMmidi0AR7MwtAYqyPCLTQ3YHIfDqGzi6dYwup1SgCaWs1KbZtcfzqoRX 5aN8dE9tyj1/t2chPgI3TvIjYQzctLMi9dfts4m8OBDEkhoG5mKHAat+ABc5mZyV5qePjd AMsqJEX2HIhKpaGUmkWOgg3kAbbyCcEz7vrZ+qHgIHtJdXCH5LVGANcSU9c7Cg== Received: from smtpclient.apple (ns1.oxydns.net [45.32.91.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: zlei/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4RZt8j3z3yz16C1; Tue, 29 Aug 2023 16:26:13 +0000 (UTC) (envelope-from zlei@FreeBSD.org) From: Zhenlei Huang Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_23D9BC82-82BB-4A5E-BAEE-39A65F10568E" List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.4\)) Subject: Re: About IFNET_PCP_NONE Date: Wed, 30 Aug 2023 00:26:05 +0800 In-Reply-To: Cc: freebsd-net To: Konstantin Belousov References: <20ECDA33-2638-4CFB-8740-5E8BF1F8072B@FreeBSD.org> X-Mailer: Apple Mail (2.3696.120.41.1.4) --Apple-Mail=_23D9BC82-82BB-4A5E-BAEE-39A65F10568E Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On Aug 29, 2023, at 9:23 AM, Konstantin Belousov = wrote: >=20 > On Mon, Aug 28, 2023 at 10:25:01PM +0800, Zhenlei Huang wrote: >>=20 >>=20 >>> On Aug 28, 2023, at 3:54 PM, Konstantin Belousov = wrote: >>>=20 >>> On Mon, Aug 28, 2023 at 09:51:38AM +0800, Zhenlei Huang wrote: >>>> Hi Konstantin, >>>>=20 >>>>=20 >>>> I was just about going to open a PR for = https://reviews.freebsd.org/D39536 and >>>> realized I might made wrong assumption. >>>>=20 >>>> I thought IFNET_PCP_NONE is something like IEEE8021Q_PCP_BE but I = second why not >>>> use IEEE8021Q_PCP_BE but a new const IFNET_PCP_NONE. >>=20 >> I meant >>=20 >> ``` >> int >> ether_output_frame(struct ifnet *ifp, struct mbuf *m) >> { >> uint8_t pcp; >>=20 >> pcp =3D ifp->if_pcp; >> if (pcp !=3D 0 /* IEEE8021Q_PCP_BE */ && ifp->if_type !=3D = IFT_L2VLAN && >> !ether_set_pcp(&m, ifp, pcp)) >> return (0); >>=20 >> ... > This is wrong. PCP_BE is just one of the priorities, that should = allowed to > be specified in the 802.1q pseudo-vlan header. >=20 >>=20 >> } >> ``` >>=20 >>>>=20 >>>> So despite its naming IFNET_PCP_NONE, is it actually a flag to let = specific interface >>>> completely bypass (disable) PCP processing? >>>>=20 >>>> The const IFNET_PCP_NONE is defined in sys/net/if.h with=20 >>>> ``` >>>> #define IFNET_PCP_NONE 0xff /* PCP disabled */ >>>> ``` >>> I fail to understand your question. >>>=20 >>> IFNET_PCP_NONE is a value that means that no 802.1q prio is inserted = into >>> the packet. Otherwise, non-vlan traffic is tagged with the = priority. >>=20 >> Think about the following case: >>=20 >> 1. Set interface's PCP to IFNET_PCP_NONE, application / firewall = provide per-flow PCP, should >> the traffic be tagged with the priority ? > Yes, it should, but only for packets from the specified flows. Thanks for the clarification ! I updated https://reviews.freebsd.org/D39536 = as per discussion. >=20 >>=20 >>=20 >>>=20 >>> IEEE8021Q_PCP_BE is a name of one of the priorities, it seems from = my >>> code reading. >>=20 >>=20 >> Best regards, >> Zhenlei >>=20 --Apple-Mail=_23D9BC82-82BB-4A5E-BAEE-39A65F10568E Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii

On Aug 29, 2023, at 9:23 AM, Konstantin Belousov <kostikbel@gmail.com>= wrote:

On Mon, Aug 28, 2023 at 10:25:01PM +0800, Zhenlei Huang = wrote:


On Aug = 28, 2023, at 3:54 PM, Konstantin Belousov <kostikbel@gmail.com>= wrote:

On Mon, Aug 28, 2023 at 09:51:38AM = +0800, Zhenlei Huang wrote:
Hi Konstantin,


I = was just about going to open a PR for https://reviews.freebsd.org/D39536 and
realized I might made wrong assumption.

I thought IFNET_PCP_NONE is something like IEEE8021Q_PCP_BE = but I second why not
use IEEE8021Q_PCP_BE but a new const = IFNET_PCP_NONE.

I = meant

```
int
ether_output_frame(struct ifnet *ifp, struct mbuf *m)
{
   uint8_t pcp;

   pcp =3D ifp->if_pcp;
   if (pcp !=3D 0 /* IEEE8021Q_PCP_BE */ = && ifp->if_type !=3D IFT_L2VLAN &&
=        !ether_set_pcp(&m, ifp, = pcp))
       return = (0);

...
This is = wrong.  PCP_BE is just one of the priorities, that should allowed = to
be specified in the 802.1q pseudo-vlan header.


}
```


So despite its naming IFNET_PCP_NONE, is it actually a flag = to let specific interface
completely bypass (disable) PCP = processing?

The const IFNET_PCP_NONE is = defined in sys/net/if.h with
```
#define = IFNET_PCP_NONE 0xff   /* PCP disabled */
```
I fail to understand your question.

IFNET_PCP_NONE is a value that means that no = 802.1q prio is inserted into
the packet.  Otherwise, = non-vlan traffic is tagged with the priority.

Think about the following = case:

1. Set interface's PCP to = IFNET_PCP_NONE, application / firewall provide per-flow PCP, should
the traffic be tagged with the priority ?
Yes, it should, but only for packets from the = specified flows.

Thanks for the clarification !

I updated https://reviews.freebsd.org/D39536 as per = discussion.





IEEE8021Q_PCP_BE is a name of one of the = priorities, it seems from my
code reading.


Best regards,
Zhenlei



= --Apple-Mail=_23D9BC82-82BB-4A5E-BAEE-39A65F10568E-- From nobody Wed Aug 30 13:58:00 2023 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RbQqT4b6Pz4s843 for ; Wed, 30 Aug 2023 13:58:13 +0000 (UTC) (envelope-from pavel.popa@edu.unife.it) Received: from mail-pj1-x1030.google.com (mail-pj1-x1030.google.com [IPv6:2607:f8b0:4864:20::1030]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RbQqS0srsz4cHr for ; Wed, 30 Aug 2023 13:58:12 +0000 (UTC) (envelope-from pavel.popa@edu.unife.it) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=edu.unife.it header.s=google header.b=kaMthHbe; spf=pass (mx1.freebsd.org: domain of pavel.popa@edu.unife.it designates 2607:f8b0:4864:20::1030 as permitted sender) smtp.mailfrom=pavel.popa@edu.unife.it; dmarc=pass (policy=quarantine) header.from=edu.unife.it Received: by mail-pj1-x1030.google.com with SMTP id 98e67ed59e1d1-271914b8aa4so1909859a91.1 for ; Wed, 30 Aug 2023 06:58:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=edu.unife.it; s=google; t=1693403890; x=1694008690; darn=freebsd.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=g0ChMsAr+vujS0gJnfsdE5AZVCItgmHyd3qOyB7cUDk=; b=kaMthHbeHmHRee/wl1kx1gcHAGx2OAE4tK/RP8yrE7XntEYVPZjR5A1m9LyMxHEV3O 3o0MYLU1RSSXqSFTHWoagbdyexblI6Az7xPa9Ag3lGGO14ViHk6cW6L7bkoJNJwPHCi6 L5U+qWLRu8mcQjXtbiC9RofhjtHpdh2B1jrbw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693403890; x=1694008690; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=g0ChMsAr+vujS0gJnfsdE5AZVCItgmHyd3qOyB7cUDk=; b=IHdN47SV+8uAMO2oiw94dQ9VHE2gAXknGSTeP8B+Zmhal3lcQ6yYxR9sCEckBHnEy4 XLuM3ONBOBrGw4tnVqB2yE+bhE2Yk/K/BnnSvR2nii3jZEw478SchrheiHZugAN4CnJd MaaaQWdbePppaNVRSbrWwXRNbnkC4MyTXIsxiPBMq8Z6v7Xg8yIiN9LBGtN8Xy/5TlyT 6/m6zQ1p9v64EVbrKeAAObSt3v4L0+Uzcjuo8JNTFE4epsq8Q9uOv7uHnawFwLeXrjSd U+/fQUXuUB3qqfnhJJ8EIq+a1CRJFqSvogy70JLxULKA6KSNSmpIP6uUF/xOqMzXxd+G 6Trw== X-Gm-Message-State: AOJu0YwkB/oq1cZsK8mserzlA8o5Kyb6wbanmYZpP/j/S3xd300wOpmi fO8VGvca5FsIOSDKvKpFHMDzdJRTlICsjKOG7tArXs+jT7QdXGUTUqTtbA== X-Google-Smtp-Source: AGHT+IGjvR2/QE8MnJgV4k+yRhWMRyt+L3QWF+p5UTYKb89BmIajRSwEhCsJYvJToeafrvlXFVpGSVSHBF/5EVENwCo= X-Received: by 2002:a17:90b:ec9:b0:267:fe44:86c3 with SMTP id gz9-20020a17090b0ec900b00267fe4486c3mr2111998pjb.31.1693403890493; Wed, 30 Aug 2023 06:58:10 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 From: PAVEL POPA Date: Wed, 30 Aug 2023 15:58:00 +0200 Message-ID: Subject: About IFLIB compliant network device driver development To: freebsd-net@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Spamd-Bar: ---- X-Spamd-Result: default: False [-5.00 / 15.00]; DWL_DNSWL_LOW(-1.00)[unife.it:dkim]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[edu.unife.it,quarantine]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[edu.unife.it:s=google]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1030:from]; RCVD_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[edu.unife.it:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4RbQqS0srsz4cHr I have a NIC (a SmartNIC actually) for which I have to implement a driver, which in addition to RX and TX rings exposes also completion CMPT rings. Due to this additional complication (the CMPT rings), I'm not sure how appropriate it is to implement such a driver via the IFLIB framework. Does anyone have any similar experience that can share? Any suggestions at all, ideas, feedbacks? Thanks in advance, Pavel From nobody Thu Aug 31 03:50:49 2023 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RbnJD2mqwz4rfCr for ; Thu, 31 Aug 2023 03:50:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RbnJD1kDvz3bDD for ; Thu, 31 Aug 2023 03:50:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1693453852; a=rsa-sha256; cv=none; b=LQIqOr+YxTZtxab2M1xjGVRCB6asbb4xjo5ueDZOa7Lnt6G/bcV191dC7ro7ClYMxdlJ7D 2yti8pBp6mwi2SRTSHvcurdo0hDv3/+4qkbsIOn7uRLTrq87MfIDvBJ+7/fcsoBxT14oBa AJaYYQo7dvGwhU7qcnKg6FHpdnBZipwAVFrr4Kaqdx2cWu7SBwyYFZQ/IxCeqiF1EpFbJH Zz2yycz6N3PDEJ765Z5F64OUC3HE5c4xkh+NgCFGnUl5WeZP/hP+MbHub0UuJJ8+/fx6Nq EjWTX7Ch3xadHCPhcHE9OCfBh6qn6WcTiRQr8Bo2jRNrKmeKvSVfEQL2AH/uGg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693453852; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AAB6GndVpeNwojzNu1gu7ijwOAPwLP9mqfisdr233Us=; b=Y+T+O5oOUNQTbqeuNIS6iYioU2ZfII0JS/vBUbA6120wg1zoRy3h9raxaWE6FKsu6ADgBk TV4ou/YC401IDvYuoLviAceQR7TMhcE518tdljkizbDZ4lkyM5AiKNV1PToQCcaTPAnuEV wwFS0Q2HJniRs/suFmDhbe+LVIhyz87bilLLnTwZf4TNpetKYxB17EiWIz+x/jTYFhxJMB E+GNtoPPRDte3pVkEYgErrVIRGWaWqslriv1zfBU0u3CpIAorNu6xOzpwbfAIdwyETBBs7 T58Err5VdmQNnNOVzGrTyu0hLLa5OMR43a1fl3kFyCs3FTMaUJXthfwziDtOaA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4RbnJD0pmZz12FK for ; Thu, 31 Aug 2023 03:50:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 37V3oqBs091120 for ; Thu, 31 Aug 2023 03:50:52 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 37V3oqoP091119 for net@FreeBSD.org; Thu, 31 Aug 2023 03:50:52 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 221122] Attaching interface to a bridge stops all traffic on uplink NIC for few seconds Date: Thu, 31 Aug 2023 03:50:49 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: spork@bway.net X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221122 spork@bway.net changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |spork@bway.net --- Comment #31 from spork@bway.net --- I burned a few hours on this last night, first thinking something was amiss with iocage (fair assumption, as it seems to be another abandoned project). Then while troubleshooting, I started running the bridge creation and inter= face additions by hand and noticed my prompt was hanging for a few seconds. Then= I found the link flaps in the logs: Aug 29 20:42:56 clweb5 kernel: ext0: link state changed to DOWN Aug 29 20:43:01 clweb5 kernel: ext0: Link is up, 1 Gbps Full Duplex, Reques= ted FEC: None, Negotiated FEC: None, Autoneg: True, Flow Control: None Aug 29 20:43:01 clweb5 kernel: ext0: link state changed to UP Aug 29 20:45:53 clweb5 kernel: ext0: link state changed to DOWN Aug 29 20:45:57 clweb5 kernel: ext0: Link is up, 1 Gbps Full Duplex, Reques= ted FEC: None, Negotiated FEC: None, Autoneg: True, Flow Control: None Aug 29 20:45:57 clweb5 kernel: ext0: link state changed to UP Aug 29 20:48:10 clweb5 kernel: ext0: link state changed to DOWN Aug 29 20:48:15 clweb5 kernel: ext0: Link is up, 1 Gbps Full Duplex, Reques= ted FEC: None, Negotiated FEC: None, Autoneg: True, Flow Control: None Aug 29 20:48:15 clweb5 kernel: ext0: link state changed to UP Seems to take about 5 seconds for it to recover, which is kind of rough on a box that will be hosting multiple jails. I understand there were workarounds posted, but I'm curious about the fix mentioned here and under what conditions this should not happen? NICs are ixl(4) OS is: 13.2-RELEASE-p2 FreeBSD 13.2-RELEASE-p2 GENERIC amd64 I did dig through the manpage for if_bridge(4), and I'm sure I saw the note about matching capabilities, but it didn't really jump out as a cause. Mayb= e a note that specifically calls out the most common use case (bridging with epair(4) for jails, bhyve or other virtualization methods) would be a good idea? Or even something in epair(4)'s manpage? --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Aug 31 04:00:16 2023 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RbnW61N3Tz4rfXf for ; Thu, 31 Aug 2023 04:00:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RbnW55NmSz3cqB for ; Thu, 31 Aug 2023 04:00:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1693454417; a=rsa-sha256; cv=none; b=YGw0bPT21Ub6WJayFB0JN9PPK8aDVfBs5SMcHUGRuroJWWtvCAyoCgzdU07qIWp5HeNV8/ N8wvGbNtNBiDcQc4FoCbntwLGQ7wElbA3nomMtUAWc88nXXICmI16P1z+vWqhr/8V2cwlY oZ7q8b6VpNPJM4uHrPnGnOvtfe8zI6R06HJyk9hhwGKmZOHaURvoAjabZ0wqchlrP31rOD Eas/rI13vuIZOnlLM5/tYmHC24cWjpaTcMqmjD7i3wZXURy8Xgm3e7O92eBaD2/amI+MAF 3hakOxQntfg9KUvS/DxrrAnlikcGOe7PqV1O3HjbpQwIjDod8Se2zvg50vRrSg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693454417; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0PzWx9fOOCkzD7iJ0Id8A1iY/y3dH8Azf55LNQv8pug=; b=oweVmXPr6UnzShoXK5VRj8lQH8w22ZSx2R7jJ4b3z5af0NYrsLAcC+M3o/RBVo2/SYTpuW kvPa2Qw+nXPjWKsCkZ6zyWzJLK4FtnU76OHz4xRxwRpPRAj2bictLeoqE5B92RbMI401+J GLE6b3pFks2fxp8fQGk4r7tkl/d1NwSWzpWlmkha7SIQvsDKNGNgqEd5SUsb06tk6YN+A+ nBIxfPi2jy0V+qTd6wIIUswvd1D1I7DYWwGRbvjyH0S+t2utEj78LmwuDn/7kNuxE0q8rw 8TPmqQkH6Ryp1a1ByevfFcHwbfbGrr6QtnYnS+QvKWieMfg2d0MCwvA/SgPRfg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4RbnW52Qkyz12lC for ; Thu, 31 Aug 2023 04:00:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 37V40HNr003501 for ; Thu, 31 Aug 2023 04:00:17 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 37V40Hia003500 for net@FreeBSD.org; Thu, 31 Aug 2023 04:00:17 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 221122] Attaching interface to a bridge stops all traffic on uplink NIC for few seconds Date: Thu, 31 Aug 2023 04:00:16 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: spork@bway.net X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221122 --- Comment #32 from spork@bway.net --- (In reply to spork from comment #31) Sorry forgot to show my diffs for the interface options between bridged and not-bridged: [root@clweb5 /home/spork]# diff /tmp/options-ixl-nobridge /tmp/options-ixl-bridge 1c1 < options=3D4e503bb --- > options=3D4a500b9 3c3 < options=3D4e503bb --- > options=3D4a500b9 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Aug 31 19:51:42 2023 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RcBd80q1Bz4rXmh for ; Thu, 31 Aug 2023 19:51:56 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: from mail-pj1-x102b.google.com (mail-pj1-x102b.google.com [IPv6:2607:f8b0:4864:20::102b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RcBd71dg4z3C1Z for ; Thu, 31 Aug 2023 19:51:55 +0000 (UTC) (envelope-from v.maffione@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=dYMIWotV; spf=pass (mx1.freebsd.org: domain of v.maffione@gmail.com designates 2607:f8b0:4864:20::102b as permitted sender) smtp.mailfrom=v.maffione@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pj1-x102b.google.com with SMTP id 98e67ed59e1d1-27178b6417fso933641a91.0 for ; Thu, 31 Aug 2023 12:51:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1693511513; x=1694116313; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=3M9kWrT1XppFW3AOSuBUt40MqEniDCWAccugeFZMV7I=; b=dYMIWotV67mEwWvrnpG1Ta8SlyVBX6nePOuPSrLaW4VnM5NbWAE3pyednSVNn+XKtC B3szE+aUh/Rh2DqLkh0tKGxWHko7BHajR4uUcq0cFhbjHSUwvkVFGAZ6lpiZ00D45Opd uaktgxdWWlKrmBhUatpYi1gpNToiAcq1Ct55AF4uFuynMySxuF8JheTKly7LiEh74C0w w0Mdpj9dHIhOHbYOtj+oT/ZlQvTWem1218GKZa0JrQwou4bsZoU0llEggSp3ffgx3rW4 9G9TplMzphSJdbGvQvkHVxwIEnAtTTVvlTeMuGr1JPWtm81CwiVtbcnVVKTPIkaNdPQS sGZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693511513; x=1694116313; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=3M9kWrT1XppFW3AOSuBUt40MqEniDCWAccugeFZMV7I=; b=TYG6pAZgzVlP1rJEANF2eX3KgJttGy3gC/KlyEOhQPraR4ungauyenyoezraUhcGQl 57ANFfKeTE8IaUaACt7/aCVHA6Z/d0FQ+iIQ59VOjlvy0rs/GguhuW8bg/yVHiSKD4xS yULxWNENQksSXL1aUvV9JiTRvLJgK6+DSj+dYCDgsz83pFFIm9UNzrYLOurJr4idCZPk 3JCX7rtqUs6x2aDuQ4Po7X84xiYKnpMQS4v0pWuz8HKAQuhuQAOjDiB+ya2A3PiaB7OE TbSDvYtxd/oMO+UPqmmXu0qWYSn6wWOXNyrPPzHFstwotici+hsL+xEFOd2L+FgqdG0V hQFQ== X-Gm-Message-State: AOJu0Yyb+8z+9u/wda6hopsGyjLXukG6TITLEBnE7zCWIHEGj5CW7206 QgF5Xo6hgGjvY6/JLG3MmoKOcpI03BFxxwWgYN1XaUgm X-Google-Smtp-Source: AGHT+IFETjWbaiC/TbP1EWUssUd3nxmpAsMQE+GORKepsaLKbWs9xtaScPX2GtkEmgXCdGZrt2DZzmPisR/aAt/WMWQ= X-Received: by 2002:a17:90a:c58e:b0:268:14d7:bc34 with SMTP id l14-20020a17090ac58e00b0026814d7bc34mr360287pjt.20.1693511513512; Thu, 31 Aug 2023 12:51:53 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Vincenzo Maffione Date: Thu, 31 Aug 2023 21:51:42 +0200 Message-ID: Subject: Re: About IFLIB compliant network device driver development To: PAVEL POPA Cc: freebsd-net@freebsd.org Content-Type: multipart/alternative; boundary="0000000000002fa14106043d60aa" X-Spamd-Bar: --- X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FREEFALL_USER(0.00)[vmaffione]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::102b:from]; BLOCKLISTDE_FAIL(0.00)[2607:f8b0:4864:20::102b:server fail]; ARC_NA(0.00)[]; TAGGED_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROMTLD(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; RCVD_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4RcBd71dg4z3C1Z --0000000000002fa14106043d60aa Content-Type: text/plain; charset="UTF-8" Hi, I think it's pretty common nowadays to have NICs with completion rings/queues. It is definitely appropriate to implement such drivers with iflib, and indeed iflib provides separate callbacks for updating the TX/RX "submission" and "completion" queues: - ift_txd_encap: submit a packet to a TX ring (or submission queue) - ift_txd_flush: flush submitted TX descriptors to the hardware. - ift_txd_credits_update: check the completion queue (or reclaim completed descriptor from the TX ring). - ift_rxd_refill: submit RX descriptors to a RX ring (or submission queue) - ift_rxd_flush: flush submitted RX descriptors to the NIC hardware. - ift_rxd_pkt_get: get a received packet from the completion queue (or reclaim completed descriptor from the RX ring) - ... If you want to look at an example of iflib driver using completion queues, you can check out sys/dev/vmware/vmxnet3/. It's actually a pretty complex example because vmxnet3 is a complex paravirtualized device with multiple versions etc.. However, the complexity does not come from the completion rings... Cheers, Vincenzo Il giorno mer 30 ago 2023 alle ore 15:58 PAVEL POPA ha scritto: > I have a NIC (a SmartNIC actually) for which I have to implement a > driver, which in addition to RX and TX rings exposes also completion > CMPT rings. Due to this additional complication (the CMPT rings), I'm > not sure how appropriate it is to implement such a driver via the > IFLIB framework. Does anyone have any similar experience that can > share? Any suggestions at all, ideas, feedbacks? > > Thanks in advance, > Pavel > > -- Vincenzo --0000000000002fa14106043d60aa Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,
=C2=A0 I think it's pretty common n= owadays to have NICs with completion rings/queues. It is definitely appropr= iate to implement such drivers with iflib, and indeed iflib provides separa= te callbacks for updating the TX/RX "submission" and "comple= tion" queues:
  • ift_txd_encap: submit a packet to a TX= ring (or submission queue)
  • ift_txd_flush: flush submitted TX d= escriptors to the hardware.
  • ift_txd_credits_update: check the compl= etion queue (or reclaim completed descriptor from the TX ring).
  • ift= _rxd_refill: submit RX descriptors to a RX ring (or submission queue)
    • <= li>ift_rxd_flush: flush submitted RX descriptors to the NIC hardware.
  • ift_rxd_pkt_get: get a received packet from the completion queue (or= reclaim completed descriptor from the RX ring)
  • ...
I= f you want to look at an example of iflib driver using completion queues, y= ou can check out sys/dev/vmware/vmxnet3/.

It's= actually a pretty complex example because vmxnet3 is a complex paravirtual= ized device with multiple versions etc..
However, the complexity = does not come from the completion rings...

Cheers,=
=C2=A0 Vincenzo



Il = giorno mer 30 ago 2023 alle ore 15:58 PAVEL POPA <pavel.popa@edu.unife.it> ha scritto:
<= blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-l= eft:1px solid rgb(204,204,204);padding-left:1ex">I have a NIC (a SmartNIC a= ctually) for which I have to implement a
driver, which in addition to RX and TX rings exposes also completion
CMPT rings. Due to this additional complication (the CMPT rings), I'm not sure how appropriate it is to implement such a driver via the
IFLIB framework. Does anyone have any similar experience that can
share? Any suggestions at all, ideas, feedbacks?

Thanks in advance,
Pavel



--
Vincenzo
--0000000000002fa14106043d60aa-- From nobody Thu Aug 31 20:18:15 2023 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RcCCX13Bhz4rZBx for ; Thu, 31 Aug 2023 20:18:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RcCCW73HSz3FB7 for ; Thu, 31 Aug 2023 20:18:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1693513096; a=rsa-sha256; cv=none; b=q657dcbrskv3LN0uh9JRteLjoBZTdpUdnDys3Owyb51Ep/b/mmPcA2EL3DCxEjTDUYzTPw 9PyXciuC/PX9gtRhuzwjL34gBr8rnKL5XxuuoHQJ4IhctAc77XsGZtu0mzqE/1TQkLrCJ4 O94QWzsaB6FwFqTyNYxojM2NZP/e0IMtgSCj/9omzbkMI3/sMbrDUSp23Of3X12mRjpi3l xIglQLxKpqard01Gd+gU2HPnLXFOrqoGs2MaN6nLIv+Zk8el3HiaSOU0ZzqaPNUJggaXDO yvWZf4k0RAwEO9rsPdXGjr7t+8tx/dCPlBZxx18hULGhQMpKl1tg+jaMsxKSww== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693513096; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=w8rgAyHzVo8MACz9fjF+Hs0uPdXa/CZW8rwG3eGZjOA=; b=beu7T7IFiYI36hRZbao6B9cS0noHzvY296cdWX0l8TqsdPfT5rc9YHlKkEz5wbwIyuWHjF 6FJS/ZMKACZOWX6X7rQUxLH/BkH9/a5ODyJkd5uNF2S1mHQ1gCV9TbgYXH00884mYAerQS Syu1WF8GoCoZNk66Vvz6pIF32CfNWKIEC4Mksq0RAqVs1NxWfIOvx4AaWpnVi3qUUG+vI5 S7NN7QKNLX+SdNTw0o5+Q+ZWn37XQga0eFs56B7JQSuMbUVWzMfqIFttOOpsVpEUA+Qwxv qVj4/4GbNUs1Fy02IKCaNk4YzjRSS5Ls4UjREaWoPUeiPs8htLRaHkNzgxKdBQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4RcCCW6838zXd4 for ; Thu, 31 Aug 2023 20:18:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 37VKIFY0078478 for ; Thu, 31 Aug 2023 20:18:15 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 37VKIFSt078476 for net@FreeBSD.org; Thu, 31 Aug 2023 20:18:15 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 221122] Attaching interface to a bridge stops all traffic on uplink NIC for few seconds Date: Thu, 31 Aug 2023 20:18:15 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: spork@bway.net X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221122 --- Comment #33 from spork@bway.net --- Some additional testing here... There are two workarounds presented in this thread: - Add "-txcsum -tso4 -tso6 -txcsum6" (or whatever your NIC requires) to the ifconfig statement for your interface(s) in rc.conf. This requires knowing = what you need to disable to make sure your NIC and epair have equal capabilities= so that when the epair interface is added to the bridge, there's no need to re= init the NIC to make the capabilities match, and therefore, no connectivity loss. - Pre-plumb the bridge and epair interfaces by adding them to rc.conf's cloned_interfaces and add the epair to the "addm" ifconfig line. On boot, t= he "addm" runs and we don't care about the reinit of the NIC because it's duri= ng boot. This method does not require knowing what capabilities need to be disabled on the NIC. I'm finding neither of these actually work as workarounds, because in 13.2 = with my ixl NICs I can see both with iocage (a jail shutdown or restart) and with manual ifconfig commands (removing a vtnet interface from a bridge) cause t= he NIC to reinit. In other words, removing an epair/vtnet interface from a bri= dge seems to put the offloading capabilities back in place, rendering either workaround useless. Again, I'm not clear on what the fix was that was mentioned in comment #28,= so if I'm way off base here, let me know! Example follows... We have a bridge containing my external ixl interface and an epair/vtnet interface from a jail: [root@clweb5 /home/spork]# ifconfig bridge0 bridge0: flags=3D8843 metric 0 mtu = 1500 ether 58:9c:fc:10:ff:d9 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: vnet0.10 flags=3D143 ifmaxaddr 0 port 7 priority 128 path cost 2000 member: ext0 flags=3D143 ifmaxaddr 0 port 1 priority 128 path cost 55 groups: bridge nd6 options=3D9 The ext0 (ixl) interface was already a member of the bridge when the jail started to there was NO NIC reinit/loss of connectivity when the jail start= ed (good!). ext0 options look like this while a member of bridge0 (ie: txcsum and two f= or v4 and v6 are disabled): ext0: flags=3D8963 metric 0= mtu 1500 =20=20=20=20=20=20=20 options=3D4a500b9 Now I manually pull vtnet0.10 from the above bridge: [root@clweb5 /home/spork]# ifconfig bridge0 deletem vnet0.10 And we see connectivity drop for 5 seconds: Aug 31 15:32:57 clweb5 kernel: vnet0.10: promiscuous mode disabled Aug 31 15:32:57 clweb5 kernel: ext0: link state changed to DOWN Aug 31 15:33:02 clweb5 kernel: ext0: Link is up, 1 Gbps Full Duplex, Reques= ted FEC: None, Negotiated FEC: None, Autoneg: True, Flow Control: None Aug 31 15:33:02 clweb5 kernel: ext0: link state changed to UP And we see why - removing the vtnet bridge member causes something(?) to put all the flags I'd removed from ext0 back in place (txcsum, txcsum6, tso4, tso6): [root@clweb5 /home/spork]# ifconfig ext0 ext0: flags=3D8963 metric 0= mtu 1500 =20=20=20=20=20=20=20 options=3D4e503bb Again, this is me manually removing the interface from the bridge, not ioca= ge. Standard jails and iocage jails both call a "destroy" on the vtnet/epair interface, so this isn't just an iocage issue. Sorry this is so long... anyhow the questions again: - Did the prior workarounds "work" and then stop working later? - Did the behavior of bringing explicitly-removed flags back to an interface when members are removed from a bridge change at some point? - What was the fix in comment #28? --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Sep 1 02:40:09 2023 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RcMh96hnFz4rw99 for ; Fri, 1 Sep 2023 02:40:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RcMh94T1rz4KhS for ; Fri, 1 Sep 2023 02:40:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1693536009; a=rsa-sha256; cv=none; b=s5bjOf5fnKVWDCYBTC+1sm5h4zdFpPHQEmyahn4RzHtSVeN+drL7rJf3kgIv+/rJfYk9Sg HhdVGwzzoDnZK14lAN8WACwmUbE8F7ggFtCOnjay+fUbq2/9ECUnEHdN5T91CE/X+/0d9N JNKphMMuIPDiG9fOT8gFg1xnU8S6DHJv4MNzUrINRZtFCvwAWIAe9uVxTq4+WcEejmBhDI Fhc/W15NpsJn3NVIDbznH/5KgCuJhIKcXUhrp8rVEmTafHQ0v49k2AlZ08wJnOZ52R+krP fvXQHOutEURBBWjfgmWH+WciQGOEuYRLYPX4ZPJAJpfINj4dvS9RLFqNMCyvcg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693536009; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=atFhh+Y/OZjI6TMrTkf+1Cg2GoxT9RcLl9m6WxqPkdU=; b=rxGZI+UlQzTdiOw4wfR5uzsZwGnDimRJj/+xauQO/Qr7yLMspwssrOpvOW+QemuRcgy/TS 5TNA6Ug8KvUBP+hTmOSexsbzDWaqo82JRl1H+LM6c3NLHXUUJmrOXarPcOIrficY5/LHwE IkQ2pQyblZ0SRmaDpoXcvAJ1A2FeB4r+SWzKDyriY9pLbmierRrAn8ujR2bHxDxQtwCqF5 tcE3KRihQhsOS8Oy5bfN/NXx3CGkfealX/3QjudE1WdF5LeQgzuwGIREQUcRSm1JMWEd1s CeTjMRyHHfsXTTcxeeGd6Yqhx8kOc7JbODygtDgO6RlVrK2Gpu+osNjrUE2n/A== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4RcMh93RFVzjs2 for ; Fri, 1 Sep 2023 02:40:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3812e9lR040648 for ; Fri, 1 Sep 2023 02:40:09 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3812e9Vx040647 for net@FreeBSD.org; Fri, 1 Sep 2023 02:40:09 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 221122] Attaching interface to a bridge stops all traffic on uplink NIC for few seconds Date: Fri, 01 Sep 2023 02:40:09 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: spork@bway.net X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221122 --- Comment #34 from spork@bway.net --- The answer to "when did interface capabilities get restored when a member is removed" is "back in 2008". This commit altered how interface flags were dealt with:=20 https://cgit.freebsd.org/src/commit/sys/net/if_bridge.c?id=3Dec29c623005ca6= a32d44fb59bc2a759a96dc75e4 You can see a variable "bif_savedcaps" was added so that the bridge now tra= cks what the original interface flags were. Then when a member is removed, it looks like all of a bridge's interfaces a= re looped through and the original flags are restored (in bridge_delete_member= ()): + /* reneable any interface capabilities */ + bridge_set_ifcap(sc, bif, bif->bif_savedcaps); Not sure where, but this kind of feels like it could be a tunable, like "net.link.bridge.restore_caps" or similar, given a) jails will trigger this with lots of NICs b) these days 5 seconds of downtime is actually not a min= or issue in many environments and c) it need not change any defaults, but rc.d/jail and 3rd party jail scripts could opt to set it d) jails are kind = of a big reason people come to FreeBSD. I'm not much of a coder, but I could get that sysctl like 80% there I think after looking at the other "net.link.bridge" tunables... any takers on help= ing? Any thoughts on whether this makes sense? --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Sep 1 23:07:01 2023 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Rctvr1Nhpz4rsM3 for ; Fri, 1 Sep 2023 23:07:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Rctvr0N9kz3CwJ for ; Fri, 1 Sep 2023 23:07:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1693609624; a=rsa-sha256; cv=none; b=f5NlFzDLKDuXiBRtLd2FPpc25dHVX5LfdYTqlFmna1YvIs6AT/xZyAfhdaqUyQBVmDLoms bblj0/AdIgLXznYlz9EqjZze+09ues8xL14uE5PZCF7WLi7jBqeOuQOd7DR9qQSjW4p4Ho l/f/HRaeLLGprk6xMbUzEmi+6dcz6P0D50DB3H6KgooSB4OFsSc+g6tYEX60rTw1IKQrHp RGqWPC9BmE7Ni4uiR8BVIPCsGOowkXvul03CpuPTe0G9JnBJKpVhA/s447n6w7oK3oyRR/ SvjQTZM4RqpbKrLoZLQ8dcQzakqfcFnGzBR+Y8tO+j1TzZSudaE13OUznAuXYA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693609624; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=e+HRKLNN9JT4DFp1UzuSCFCbtlfSwyqdwKaM9XR7TBI=; b=cUhXUq10e/9uhnl6jBSnuRV4AHKzglzzRAGKdWq9liRhXDeoQOa5u2G78n5whuSAm+Ljen X5JV91R3Gvqj0W3rVfUT+j012yblJ3Ht3fYfGc7Z4olScJfzMuSDTFIxY19c8CB7qc1rjq HVz7lWRj0IaKZ8O0Ab5zvvn0lrX4sbzMRYRxsg2/icV115cVm/EvbZp51uNFf9JdgDOaRr 1xt3Q89P7qv9uNfbRWLBKAdGbkonm5NX0mV1wksuJJpkhci6YF5EGI5Sgo26f9op4FeYPA nrj+5HE9ShVzUxESjvyDe2Zct7L55eM75rhn2OsHp9W5ru5oHnfYAK6KJlOadA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Rctvq6YyCz4rl for ; Fri, 1 Sep 2023 23:07:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 381N73IO090069 for ; Fri, 1 Sep 2023 23:07:03 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 381N73kP090068 for net@FreeBSD.org; Fri, 1 Sep 2023 23:07:03 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 221122] Attaching interface to a bridge stops all traffic on uplink NIC for few seconds Date: Fri, 01 Sep 2023 23:07:01 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: spork@bway.net X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221122 --- Comment #35 from spork@bway.net --- OK, really done for now... :) I'm trying this out for a bit. [root@clweb5 /usr/src/sys/net]# diff -u if_bridge.c.dist if_bridge.c.caps --- if_bridge.c.dist 2023-08-31 22:47:16.758453000 -0400 +++ if_bridge.c.caps 2023-09-01 19:05:41.724323000 -0400 @@ -452,6 +452,13 @@ CTLFLAG_RWTUN | CTLFLAG_VNET, &VNET_NAME(log_stp), 0, "Log STP state changes"); +/* restore member if capabilites */ +VNET_DEFINE_STATIC(int, restore_caps) =3D 1; +#define V_restore_caps VNET(restore_caps) +SYSCTL_INT(_net_link_bridge, OID_AUTO, restore_caps, + CTLFLAG_RWTUN | CTLFLAG_VNET, &VNET_NAME(restore_caps), 0, + "Restore member interface flags on reinit"); + /* share MAC with first bridge member */ VNET_DEFINE_STATIC(int, bridge_inherit_mac); #define V_bridge_inherit_mac VNET(bridge_inherit_mac) @@ -1151,7 +1158,8 @@ #endif break; } - /* reneable any interface capabilities */ + /* reneable any interface capabilities if restore_caps is s= et */ + if (V_restore_caps) bridge_set_ifcap(sc, bif, bif->bif_savedcaps); } bstp_destroy(&bif->bif_stp); /* prepare to free */ --=20 You are receiving this mail because: You are the assignee for the bug.=