From nobody Mon Dec 6 02:49:48 2021 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4D4941891D8C for ; Mon, 6 Dec 2021 02:49:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4J6nvy0k8Vz3ByF for ; Mon, 6 Dec 2021 02:49:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id E7F5023750 for ; Mon, 6 Dec 2021 02:49:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 1B62nnao006004 for ; Mon, 6 Dec 2021 02:49:49 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 1B62nnQL006003 for jail@FreeBSD.org; Mon, 6 Dec 2021 02:49:49 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 233310] jails: Modularize configuration system (conf.d) Date: Mon, 06 Dec 2021 02:49:48 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: conf X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: feature, needs-patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: v_bachvarov@mail.bg X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: mfc-stable13? mfc-stable12? X-Bugzilla-Changed-Fields: blocked Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1638758990; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=C+h9lsDX83/EXODfm3mXraucIArjzHtU2WNv4g55oR0=; b=qbDlYTN4N60lcJQHztzjG+C+IhAZU2F+HASxxHY1lrRPbPEqdpPFop3kHVh00KgeSyPjMk 80a3xjYNZ7C9q5+ab9Tw7K12HEq+KO3xi1p3eWJbpyh2+18FQObp0GQefs4yW3sPvZb1bI dUb98FA0X6HADSSi2ekn1xC1Q3iRddpL/J7iqouDzaf9hD0/4oRugVEtCEVepqTOmBzExY 4Vo3kp3gPqRMt9P8OiteVCDHKpre2IJeAmHiP/iQSMZjL8VeIaneWHdXBK0Umv+0kUlc11 TKvZaZn3ORDi/6cmDMUl/l+uyNUJa8gjgIb0IsoSF6Omh0uFNdY4z1h1FyHxdg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1638758990; a=rsa-sha256; cv=none; b=L6tgSob9JtUaY/ppApie4znhz3dT6xwGyUw11vPgaxEVsIfOfr6FgaWLuqpYeTKSV6H0aJ MBY0WieX8xEmRSxinQ+VM+u2XViMaG8JazuB9kFvLoIQoBFjh9trKhS65V1tupnfDw6YVx NPcRvCuLsxxHRqUe/f4HIpKnqhYoI0YoBO/1R3X9jTDt7gRqPStn0LBjwqdvChewsQ/v5F wJeloI3Ini2ZX3DAJHNe5a+BzggVtRNWpHDy2VSWMWnM1loawDd8oXk092a0DS1eue9bAx Y5aXEoMd+78dFBHMk0euoBQ9QA0g75q2Ewb1JpaZfXcagHB0aGvvxHNMsFK9WQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D233310 Rocco changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |260248 Referenced Bugs: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D260248 [Bug 260248] jails: depend parameter does not work in modular jail files (conf.d) --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Feb 1 15:27:46 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 944C0198D0B2 for ; Tue, 1 Feb 2022 15:27:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Jp82D1mgVz3JQl for ; Tue, 1 Feb 2022 15:27:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 5DC13135A1 for ; Tue, 1 Feb 2022 15:27:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 211FRlZQ005965 for ; Tue, 1 Feb 2022 15:27:47 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 211FRlSU005964 for jail@FreeBSD.org; Tue, 1 Feb 2022 15:27:47 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 255685] PF: JAIL: fail to connect from jail to jail service when pf enabled Date: Tue, 01 Feb 2022 15:27:46 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: freebsdbugzilla@agneau.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1643729268; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tDAfKbQgUhHi5wQMCvnx6E+h6IIyfOVTwzWEozbrPKs=; b=Eg9zm9qwIy9hWUdZXND9SzQiG/41Q3n+8jLa8aTcC+vuRFGARTk9yYnke2OvcbFy+4JA3K kaR08cVzW13JgLSeb8KmvZbX0oQfr494FGe9+4ptB7vhCkQa+U5m6y08ZI1731rLgxMZB+ 74O32c5lChxfT6sFrBKeq6jrvHYmnltsNGPOrBlMF32nXiVautsfY38cmzDjG+aDSnXSU1 JrBGxZFxjYcqdd/Dc646YVMnfMCxFmHrnFluyzX4Av6Eri5USElcBjv6c/Ut7074D7bs4k aNtddUCc2lT2/BEG44Tn6H77RE6mbYZFVoLhsxrWrAExfDOkV6qByVj+F8nQgw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1643729268; a=rsa-sha256; cv=none; b=o2w3G7ycrxy6bfebV9VdXOUZrTRvmfFSt9o0S7GTTMzS+5Bfc8TO3lK8j48lIaCMlWS9DF 6U6FEYIqIlZIcxjjAZq6MNs7uRFziNLTmBGWcyLZifSdugh0mnnF8oHpn2hZ8zOyPG1ieR 5jt0h6h420avV7jMmt5acPiAsHaBsRLKUlWNbMHIpodQZiew1pVOCTjOcShZkf5ummIQqU 0JYaIFLquDuGGBGzIDjrbatqkFtfQs8/YEW9wA2yijpNLVAyrXVgZYsUBoG7ILbPXcH1+I eUzKJ+2FY+8MHOkjOv8DY6Evyobl9+GOMv8wGbnBW4drUvGyTo6SXFqcgs1VJw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D255685 --- Comment #5 from Laurent Frigault --- (In reply to Laurent Frigault from comment #4) I'd like to add a precision. On 13.0 : When adding an ipv4 with a non /32 mask to a jail 2 routes are added : 10.10.10.0/24 link#1 U bge0 10.10.10.10 link#1 UHS lo0 When adding an ipv4 with a /32 mask only one route is added to the physical interface and the second route to lo0 is missing: 192.168.249.247 link#1 UH bge0 Before 13.0, the 2 routes are always added even with /32 mask. 192.168.249.195 link#1 UHS 0 5 lo0 =3D> 192.168.249.195/32 link#1 U 0 0 bce0 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Feb 1 17:53:17 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 18D1F19B9B62 for ; Tue, 1 Feb 2022 17:53:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JpCG54txRz3hT1 for ; Tue, 1 Feb 2022 17:53:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 54B0D153B2 for ; Tue, 1 Feb 2022 17:53:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 211HrHL3082270 for ; Tue, 1 Feb 2022 17:53:17 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 211HrHME082269 for jail@FreeBSD.org; Tue, 1 Feb 2022 17:53:17 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 255685] PF: JAIL: fail to connect from jail to jail service when pf enabled Date: Tue, 01 Feb 2022 17:53:17 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: freebsdbugzilla@agneau.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1643737997; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OevjCyH1oVR6vtzxgWX1eQj72tlSjfsxlYF8qx55YRE=; b=kbM9ta3YzzgOsV0w1551yEQtySd2GDYW2HC3VMhHiC378CHt5siJ4QrV4fI784vQ+ETT7n rua9g0na7gq3U1ZybvMJrGWmf17NiFbdi2MVGEoqT8tdhD1VL1K7KTrYeftnkJdK1HUCEk FxFj5IMlI27MgNLXqLBcFl6+AaEFuc7mNT8L4WZVE9q8UCM9WX5LMlMqdtBnVd4C7GYFne FpabtpyDRt4TYftaKsPMQOCyNF04ROKCZ4mH2yIdCeh8YdJsI/vcfWf/0/p8VyAuYUanWW 3axvMbazB9JHPB/K+veFzs8JRcErznHgOq6QZ/xaTtk7XG2VapmHWAkRuZHGYA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1643737997; a=rsa-sha256; cv=none; b=SiCuk8biQKBPsugDfzR262hGqrGyhrPsHRuwG6anYcid87rzU7WhKffJwOIL4WugD1utLj ZSCZpKjW60Ff3TVaHnfeLuuu0vMYQk/rJI1Cf73iEHeP9F1B6hPUk+kJq7NLmA457gtZK7 sn/VDbPdBOf+LVEmWulIcoYvOj9AwL9QZllGKH1HvJjdM0m4jbW1wBbPvNts57z0nMI73c a+0bxXc3P402D/3ibuSlEd/gq5nGdIngt1BiBnXiKy9sQF/LQD2UXVBftNM2660xSPjUip GU+v2SkliVQL0j4zVkuol8ttdXilZ6BI4lbWLDJcAJvmN8WwTL8Tz38LYFRo8w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D255685 --- Comment #6 from Laurent Frigault --- (In reply to Laurent Frigault from comment #5) man ifconfig still says: .. alias Establish an additional network address for this interface. T= his is sometimes useful when changing network numbers, and one wis= hes to accept packets addressed to the old interface. If the addr= ess is on the same subnet as the first network address for this interface, a non-conflicting netmask must be given. Usually 0xffffffff is most appropriate. but it looks like since 13.0 we can now add aliases with non /32 mask even= if there is already an ip with the same non /32 subnet and this works with jail ips too. example: host configuration: ifconfig_bge0_alias0=3D"inet 192.168.249.240 netmask 255.255.255.128" jail configuration: ip4.addr +=3D "192.168.249.247/25"; # netstat -rn |fgrep 192.168.=20=20=20=20 192.168.249.128/25 link#1 U bge0 192.168.249.240 link#1 UHS lo0 192.168.249.247 link#1 UHS lo0 lo0 host routes are back and the 2 ips can talk to each other via lo0 This change may ne related to https://www.freebsd.org/releases/13.0R/relnot= es/ ... Duplicate routes installation issue for /32 or /128 interface aliases has b= een fixed. 81728a538d24 ... maybe the ifconfig manual page should be updated to remove=20 "Usually 0xffffffff is most appropriate" from the alias item --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Feb 5 04:28:41 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id EF16E19AAF5D for ; Sat, 5 Feb 2022 04:28:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JrKCs2WJGz4ctx for ; Sat, 5 Feb 2022 04:28:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 24ADD18AB8 for ; Sat, 5 Feb 2022 04:28:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 2154Sfst040755 for ; Sat, 5 Feb 2022 04:28:41 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 2154SfFN040754 for jail@FreeBSD.org; Sat, 5 Feb 2022 04:28:41 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 240442] Upgrade from 11Stable to 12Stable - dying jail requires host reboot Date: Sat, 05 Feb 2022 04:28:41 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 12.0-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: beebeetles@posteo.de X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1644035321; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jOtjHvIT1y9Rox/vR+Wkw+vdPXnHvpFXnL1k6s0It/E=; b=XA6/xVeshgcuBXJF9NzkPGbLguy7TqK2BgrcKkM0zPYt1fcfOgdJ1VJBzwcf1ydV4tEk1t Hb2cfxrADyqWj6vHWvASwkmGszEVKNpkgCZMdiB7VxDS2WNKXfVoJWdFzNIZ8xcqTcY05+ xy7/71IMbeVkfz5CCbuAjStx24QKOgr9t+akt3vMWdsEQ9roRrTfgPLEfmm/LpQyFH0fNb 3RSAEx1Do9rnK2eIxkFJf73kzkdj6xyCbzLBre0pkaxq6W3RK/HIVgH6M3z4w0UU0sEa8z 6QO4GkZ/x0Sh3p0rtTzXo9vxHdGPLb2Szet//3telejoKpTTGj7ZyDqmcoLOgA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1644035321; a=rsa-sha256; cv=none; b=PlayxBYXFUEJ7B6eoaMbeYUP7YwggM5mgjj7JkRYPZCEE8+C4CUFCzynmqlJ+onl4jtIp9 CMZxhFakxPoH5N7pi8qFJQRT/uBwpSHSLQ/34cV/gewzkr7NTisogqpxxQPOMaRmDertMQ HnDqX4u9BHGMEUJTJ+fNUmKMeSyxRf1NAxbpk5fRWcumKux2/6t7CQAuM+lcpzn1Y2TRkn e81LddngAb0ZvDUDOkWkefZUoOQiDhchDF2rrt7gmL6SoYhkoEWgQ0KNEg+kJC4EHg02Nv IbZY512F4vzA0LzEEec64fJ4ZXSltF9myKpkv3RmbBAlwO7mqifkNS2pHL5+Bg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D240442 beebeetles@posteo.de changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |beebeetles@posteo.de --- Comment #1 from beebeetles@posteo.de --- I'm experiencing something similar on 13.0-RELEASE: My jails are stuck at t= he dying stage, and will always show up in `jls -d`. Does not require rebootin= g in my case though. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Feb 22 21:30:21 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 0FFFA19D21F9; Tue, 22 Feb 2022 21:30:35 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: from mail-qv1-xf34.google.com (mail-qv1-xf34.google.com [IPv6:2607:f8b0:4864:20::f34]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K3C5635nGz4qHs; Tue, 22 Feb 2022 21:30:34 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: by mail-qv1-xf34.google.com with SMTP id o5so3076025qvm.3; Tue, 22 Feb 2022 13:30:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:from:date:message-id:subject:to; bh=n3LscQcDRA+v4tjDyFVMynDA7lNyzUO1Ng9mlnCudmU=; b=CiKXEW0S5U4z57w7syhGgTmS+hsbs+EDHtlAdJ/dW0qImqfFX8412kBIjui/+ZDf6C pdqoWTWIWdQJAU7Q+KG0+BZHW8IoALZlpg6/t8aYwbbTmMBFX5BEUgoqUjqWTqMWxEgy mGo0McI8teDnZ0szWqAxW8iGOLX7V4pHmXbpILfpjkVrU9FJG2517A+zxVUkn/w7mEzf N7MT3r5bVkBNV3ha30leiD9LohWFVz6GsTpxmWihtsc06NSKILUJuu6nCqtooyW7B1Ox 7ZFNNJeLYjLX6ZXAvblprP0zy5ZGDBG9jerp8ViL7CIRHrFqFlN9CBlCumxjENjqW/Ji 3ynQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=n3LscQcDRA+v4tjDyFVMynDA7lNyzUO1Ng9mlnCudmU=; b=VjyrflB9VWbz115nnD8vw/rOX1cKo0P8+l97A53lF0LgeI1dG+r6HP1/kysPtmonjo Bws5f4kLz7Iqz1hUqjeEXzBgF12Y7DHgQwSPvMjUubvXozdfxdqy30+okZoJ5g0R1xdN SjbX+NY8lYEtDVFTiIl8/wxtbN9SScLHzk2jI+vCEOBKCYpmS1TKblUPsOaLlpmhLt80 h8ttxvttkJ6HIQ10G28UnL5vrtTX+YRIg2VGaIgfN+dZeUDqGvl0QNXKrpP8iDV7YX0N Dm18clcvBiUMAiXTT48E0vrlrPhroaUAI9AgJKl7SwROKk3B9m5S1fO7vEPWOqAzAG54 RMeA== X-Gm-Message-State: AOAM532OfNybMYnR0aOgnpuu3W2XqTh8OflaYztTwf14aKhe21gz+0Mi 6qm97ZxIeRrEh7Q6q+MiLEpfUB2vJfJcPZ8SR9a/VLyvupA= X-Google-Smtp-Source: ABdhPJyhyE513GaclIIRmhIn9lxmNsWltqXB2D/egd8Up0+bTB75kchZJoatfJDLWpwH4EvtR/PyitGNCsZHdi4OYM4= X-Received: by 2002:ac8:13c1:0:b0:2d5:3c22:8e22 with SMTP id i1-20020ac813c1000000b002d53c228e22mr23715493qtj.306.1645565432126; Tue, 22 Feb 2022 13:30:32 -0800 (PST) List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 From: Sami Halabi Date: Tue, 22 Feb 2022 23:30:21 +0200 Message-ID: Subject: linux debian jail - network problems To: freebsd-jail@freebsd.org, freebsd-net@freebsd.org, freebsd-emulation@freebsd.org Content-Type: multipart/alternative; boundary="0000000000000960bf05d8a20fa9" X-Rspamd-Queue-Id: 4K3C5635nGz4qHs X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=CiKXEW0S; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of sodynet1@gmail.com designates 2607:f8b0:4864:20::f34 as permitted sender) smtp.mailfrom=sodynet1@gmail.com X-Spamd-Result: default: False [-2.00 / 15.00]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; TO_DN_NONE(0.00)[]; URI_COUNT_ODD(1.00)[5]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::f34:from]; HTTP_TO_IP(1.00)[]; MLMMJ_DEST(0.00)[freebsd-jail,freebsd-net,freebsd-emulation]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N --0000000000000960bf05d8a20fa9 Content-Type: text/plain; charset="UTF-8" Hi all, sorry for the cross post but I need help and I'm not sure where it hangs. I create linux jail (debian bullseye) via cbsd. the jail is being populated with the debian userland.. so far so good... services running (sshd) and I can login to the jail, I also can update packages and I can install apache httpd and all works fine (apt install or make from src). I also manage to install packages even if their scripts depend on "ip" command that fails: cbsd@j2> ip Cannot open netlink socket: Address family not supported by protocol ifconfig show empty interfaces: cbsd@j2> ifconfig eth0: flags=4163 mtu 1500 ether 00:50:56:0a:b3:a0 (Ethernet) RX packets 139798314 bytes 12029597009 (11.2 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 26879143 bytes 34400160833 (32.0 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo0: flags=4169 mtu 16384 loop (Local Loopback) RX packets 28548 bytes 160312960 (152.8 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 28548 bytes 160312960 (152.8 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 I know linux emulation doesn't implement netlink.. so what I do is fake the response by replacing /bin/ip by a bash script that prints the correct IP and fakes some other (needed by packages i Installed): #!/bin/bash if [ "$1" = "-o" ]; then echo "1: eth0 inet 192.168.1.2/24 brd 192.168.1.255 scope global eth0" elif [ "$1" = "route" ]; then if [ "$2" = "get" ]; then echo "8.8.8.8 via 192.168.1.2 dev eth0 src 192.168.1.2 " else echo "default via 192.168.1.2 dev eth0" fi else echo "1: eth0: mtu 1500 qdisc mq state UP qlen 1000" echo " inet 192.168.1.2 /24 brd 192.168.1.255 scope global eth0" still ifconfig shows no IP... its time to say it a regular jail and *NOT* VNET. *however* package that pull ips via libraries fail.. eg: installed bind916 (name) in the logs I see these errors (relevant only): cbsd@j2> service named start Starting domain name service...: namednamed: prctl(PR_SET_DUMPABLE) failed: Invalid argument cbsd@j2> log file shows: 22-Feb-2022 23:11:58.705 general: notice: BIND 9 is maintained by Internet Systems Consortium, 22-Feb-2022 23:11:58.705 general: notice: Inc. (ISC), a non-profit 501(c)(3) public-benefit 22-Feb-2022 23:11:58.705 general: notice: corporation. Support and training for BIND 9 are 22-Feb-2022 23:11:58.705 general: notice: available at https://www.isc.org/support 22-Feb-2022 23:11:58.705 general: notice: ---------------------------------------------------- 22-Feb-2022 23:11:58.705 general: info: found 6 CPUs, using 6 worker threads 22-Feb-2022 23:11:58.705 general: info: using 6 UDP listeners per interface 22-Feb-2022 23:11:58.705 general: info: using up to 21000 sockets 22-Feb-2022 23:11:58.715 general: info: loading configuration from '/etc/bind/named.conf' 22-Feb-2022 23:11:58.715 general: info: reading built-in trust anchors from file '/etc/bind/bind.keys' 22-Feb-2022 23:11:58.715 general: info: looking for GeoIP2 databases in '/usr/share/GeoIP' 22-Feb-2022 23:11:58.715 general: info: using default UDP/IPv4 port range: [1024, 65535] 22-Feb-2022 23:11:58.715 general: info: using default UDP/IPv6 port range: [1024, 65535] 22-Feb-2022 23:11:58.715 network: info: no IPv6 interfaces found 22-Feb-2022 23:11:58.715 general: error: ifiter_getifaddrs.c:79: unexpected error: 22-Feb-2022 23:11:58.715 general: error: getting interface addresses: getifaddrs: Address family not supported by protocol 22-Feb-2022 23:11:58.715 network: warning: not listening on any interfaces *snip* *snip* 22-Feb-2022 23:11:58.735 general: error: socket.c:2405: unexpected error: 22-Feb-2022 23:11:58.735 general: error: setsockopt(50, IP_RECVTOS) failed: Protocol not available 22-Feb-2022 23:11:58.735 general: notice: couldn't add command channel 127.0.0.1#953: permission denied 22-Feb-2022 23:11:58.735 general: error: socket.c:2405: unexpected error: 22-Feb-2022 23:11:58.735 general: error: setsockopt(50, IP_RECVTOS) failed: Protocol not available 22-Feb-2022 23:11:58.735 general: notice: couldn't add command channel 127.0.0.1#953: permission denied 22-Feb-2022 23:11:58.735 zoneload: info: managed-keys-zone: loaded serial 24 22-Feb-2022 23:11:58.735 zoneload: info: zone 0.in-addr.arpa/IN: loaded serial 1 22-Feb-2022 23:11:58.735 general: error: socket.c:2405: unexpected error: 22-Feb-2022 23:11:58.735 general: error: setsockopt(512, IP_RECVTOS) failed: Protocol not available 22-Feb-2022 23:11:58.735 general: error: socket.c:2405: unexpected error: 22-Feb-2022 23:11:58.735 general: error: setsockopt(513, IP_RECVTOS) failed: Protocol not available 22-Feb-2022 23:11:58.745 zoneload: info: zone 255.in-addr.arpa/IN: loaded serial 1 22-Feb-2022 23:11:58.745 zoneload: info: zone j1.royalshells.com/IN: loaded serial 2022022106 22-Feb-2022 23:11:58.745 notify: info: zone j1.royalshells.com/IN: sending notifies (serial 2022022106) 22-Feb-2022 23:11:58.745 general: error: socket.c:2405: unexpected error: 22-Feb-2022 23:11:58.745 general: error: setsockopt(514, IP_RECVTOS) failed: Protocol not available 22-Feb-2022 23:11:58.745 zoneload: info: zone localhost/IN: loaded serial 2 22-Feb-2022 23:11:58.745 general: error: socket.c:2405: unexpected error: 22-Feb-2022 23:11:58.745 general: error: setsockopt(515, IP_RECVTOS) failed: Protocol not available 22-Feb-2022 23:11:58.745 zoneload: info: zone 127.in-addr.arpa/IN: loaded serial 1 22-Feb-2022 23:11:58.745 general: notice: all zones loaded 22-Feb-2022 23:11:58.745 general: notice: running 22-Feb-2022 23:11:58.795 general: error: socket.c:2405: unexpected error: 22-Feb-2022 23:11:58.795 general: error: setsockopt(50, IP_RECVTOS) failed: Protocol not available 22-Feb-2022 23:12:58.811 general: error: ifiter_getifaddrs.c:79: unexpected error: 22-Feb-2022 23:12:58.811 general: error: getting interface addresses: getifaddrs: Address family not supported by protocol 22-Feb-2022 23:12:58.811 network: warning: not listening on any interfaces Any Idea how to fix this?? cbsd@j2> named -V BIND 9.16.22-Debian (Extended Support Version) running on Linux x86_64 3.2.0 FreeBSD 12.3-RELEASE-p1 GENERIC installing newer versions I have also problems with dovecot mail package.. but will leave it for now Thanks in advance, Sami --0000000000000960bf05d8a20fa9 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi all,
sorry for the cross post but I need help and I= 'm not sure where it hangs.

I create linux jai= l (debian bullseye) via cbsd.
the jail is being populated with th= e debian userland..
so far so good... services running (sshd) and= I can login to the jail, I also can update packages=C2=A0and I can install= apache httpd and all works fine (apt install or make from src).
= I also manage to install packages even if their scripts depend on "ip&= quot; command that fails:
cbsd@j2> ip
Cannot open netlink s= ocket: Address family not supported by protocol

ifconfig show empty interfaces:
cbsd@j2> ifconfig
eth0: f= lags=3D4163<UP,BROADCAST,RUNNING,MULTICAST> =C2=A0mtu 1500
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 ether 00:50:56:0a:b3:a0 =C2=A0(Ethernet)
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 RX packets 139798314 =C2=A0bytes 12029597009 (11.2 GiB)=C2=A0 =C2=A0 =C2=A0 =C2=A0 RX errors 0 =C2=A0dropped 0 =C2=A0overruns 0 = =C2=A0frame 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 TX packets 26879143 =C2=A0byte= s 34400160833 (32.0 GiB)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 TX errors 0 =C2=A0d= ropped 0 overruns 0 =C2=A0carrier 0 =C2=A0collisions 0

lo0: flags=3D= 4169<UP,LOOPBACK,RUNNING,MULTICAST> =C2=A0mtu 16384
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 loop =C2=A0(Local Loopback)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 RX= packets 28548 =C2=A0bytes 160312960 (152.8 MiB)
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 RX errors 0 =C2=A0dropped 0 =C2=A0overruns 0 =C2=A0frame 0
=C2=A0= =C2=A0 =C2=A0 =C2=A0 TX packets 28548 =C2=A0bytes 160312960 (152.8 MiB)=C2=A0 =C2=A0 =C2=A0 =C2=A0 TX errors 0 =C2=A0dropped 0 overruns 0 =C2=A0c= arrier 0 =C2=A0collisions 0

I know linux emula= tion doesn't implement netlink.. so what I do is fake the response by r= eplacing /bin/ip by a bash script that prints the correct IP and fakes some= other (needed by packages i Installed):
#!/bin/bash
if [ = "$1" =3D "-o" ]; then
echo "1: eth0 inet 192.168.1.2/24 brd 192.168.1.255 scope glob= al eth0"
elif [ "$1" =3D "route" ]; then
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 if [ "$2" =3D "get" ]; then=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "8.8.8.8= via=C2=A0 192.168.1.2=C2=A0=C2=A0=C2=A0dev eth0 =C2=A0src=C2=A0 192.168.1.2=C2=A0 "
=C2=A0 =C2=A0 =C2=A0 =C2=A0 else
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "default via=C2=A0 192.168.1.2=C2=A0=C2=A0=C2=A0dev eth0"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = fi
else
echo "1: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> m= tu 1500 qdisc mq state UP qlen 1000"
echo " =C2=A0inet=C2=A0 192.168.1.2=C2=A0 /24 brd=C2=A0 192.168.1.255 scope global eth0"

still ifconfig shows no IP... its time to say it a regular jail= and *NOT* VNET.

*however* package that pull ips v= ia libraries fail..
eg: installed bind916 (name) in the logs I se= e these errors (relevant only):
cbsd@j2> service named startStarting domain name service...: namednamed: prctl(PR_SET_DUMPABLE) faile= d: Invalid argument
cbsd@j2>


<= div>log file shows:
22-Feb-2022 23:11:58.705 general: notice: BIN= D 9 is maintained by Internet Systems Consortium,
22-Feb-2022 23:11:58.7= 05 general: notice: Inc. (ISC), a non-profit 501(c)(3) public-benefit
22= -Feb-2022 23:11:58.705 general: notice: corporation.=C2=A0 Support and trai= ning for BIND 9 are
22-Feb-2022 23:11:58.705 general: notice: available = at https://www.isc.org/support<= br>22-Feb-2022 23:11:58.705 general: notice: ------------------------------= ----------------------
22-Feb-2022 23:11:58.705 general: info: found 6 C= PUs, using 6 worker threads
22-Feb-2022 23:11:58.705 general: info: usin= g 6 UDP listeners per interface
22-Feb-2022 23:11:58.705 general: info: = using up to 21000 sockets
22-Feb-2022 23:11:58.715 general: info: loadin= g configuration from '/etc/bind/named.conf'
22-Feb-2022 23:11:58= .715 general: info: reading built-in trust anchors from file '/etc/bind= /bind.keys'
22-Feb-2022 23:11:58.715 general: info: looking for GeoI= P2 databases in '/usr/share/GeoIP'
22-Feb-2022 23:11:58.715 gene= ral: info: using default UDP/IPv4 port range: [1024, 65535]
22-Feb-2022 = 23:11:58.715 general: info: using default UDP/IPv6 port range: [1024, 65535= ]
22-Feb-2022 23:11:58.715 network: info: no IPv6 interfaces found
22= -Feb-2022 23:11:58.715 general: error: ifiter_getifaddrs.c:79: unexpected e= rror:
22-Feb-2022 23:11:58.715 general: error: getting interface address= es: getifaddrs: Address family not supported by protocol
22-Feb-2022 23:= 11:58.715 network: warning: not listening on any interfaces
*= snip*
*snip*
22-Feb-2022 23:11:58.735 general: error: s= ocket.c:2405: unexpected error:
22-Feb-2022 23:11:58.735 general: error:= setsockopt(50, IP_RECVTOS) failed: Protocol not available
22-Feb-2022 2= 3:11:58.735 general: notice: couldn't add command channel 127.0.0.1#953= : permission denied
22-Feb-2022 23:11:58.735 general: error: = socket.c:2405: unexpected error:
22-Feb-2022 23:11:58.735 general: error= : setsockopt(50, IP_RECVTOS) failed: Protocol not available
22-Feb-2022 = 23:11:58.735 general: notice: couldn't add command channel 127.0.0.1#95= 3: permission denied
22-Feb-2022 23:11:58.735 zoneload: info: managed-ke= ys-zone: loaded serial 24
22-Feb-2022 23:11:58.735 zoneload: info: zone = 0.in-addr.arpa/IN: loaded serial 1
22-Feb-2022 23:11:58.735 general: err= or: socket.c:2405: unexpected error:
22-Feb-2022 23:11:58.735 general: e= rror: setsockopt(512, IP_RECVTOS) failed: Protocol not available
22-Feb-= 2022 23:11:58.735 general: error: socket.c:2405: unexpected error:
22-Fe= b-2022 23:11:58.735 general: error: setsockopt(513, IP_RECVTOS) failed: Pro= tocol not available
22-Feb-2022 23:11:58.745 zoneload: info: zone 255.in= -addr.arpa/IN: loaded serial 1
22-Feb-2022 23:11:58.745 zoneload: info: = zone j1.royalshells.com/IN: lo= aded serial 2022022106
22-Feb-2022 23:11:58.745 notify: info: zone j1.royalshells.com/IN: sending noti= fies (serial 2022022106)
22-Feb-2022 23:11:58.745 general: error: socket= .c:2405: unexpected error:
22-Feb-2022 23:11:58.745 general: error: sets= ockopt(514, IP_RECVTOS) failed: Protocol not available
22-Feb-2022 23:11= :58.745 zoneload: info: zone localhost/IN: loaded serial 2
22-Feb-2022 2= 3:11:58.745 general: error: socket.c:2405: unexpected error:
22-Feb-2022= 23:11:58.745 general: error: setsockopt(515, IP_RECVTOS) failed: Protocol = not available
22-Feb-2022 23:11:58.745 zoneload: info: zone 127.in-addr.= arpa/IN: loaded serial 1
22-Feb-2022 23:11:58.745 general: notice: all z= ones loaded
22-Feb-2022 23:11:58.745 general: notice: running
22-Feb-= 2022 23:11:58.795 general: error: socket.c:2405: unexpected error:
22-Fe= b-2022 23:11:58.795 general: error: setsockopt(50, IP_RECVTOS) failed: Prot= ocol not available
22-Feb-2022 23:12:58.811 general: error: i= fiter_getifaddrs.c:79: unexpected error:
22-Feb-2022 23:12:58.811 genera= l: error: getting interface addresses: getifaddrs: Address family not suppo= rted by protocol
22-Feb-2022 23:12:58.811 network: warning: n= ot listening on any interfaces

Any Idea how to= fix this??

cbsd@j2> named -V
BIND 9.16.22-D= ebian (Extended Support Version) <id:59bfaba>
running on Linux x86= _64 3.2.0 FreeBSD 12.3-RELEASE-p1 GENERIC

inst= alling newer=C2=A0versions=C2=A0

I have also probl= ems with dovecot mail package.. but will leave it for now

Thanks in advance,
Sami

--0000000000000960bf05d8a20fa9-- From nobody Thu Feb 24 20:05:27 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 0D8C619E2C45; Thu, 24 Feb 2022 20:05:44 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: from mail-qt1-x82f.google.com (mail-qt1-x82f.google.com [IPv6:2607:f8b0:4864:20::82f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K4P6H2Qd1z4c0R; Thu, 24 Feb 2022 20:05:43 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: by mail-qt1-x82f.google.com with SMTP id v3so526625qta.11; Thu, 24 Feb 2022 12:05:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=KFGKRLMqHwXAGyq5rtZxeGU4x4Lsz1Xzh3y18G9xLjQ=; b=Zm6+f5PuLizA5fUfVuRMg+o8XEsn76CnR3+Fqkt0FHjA13sp8YKZ1lWYuWsPjeOXGd sF5WxGP91JGmruwLTg8UFuXoyi0n/Ob02Z4EqaqjGnVVVAoZ3DAWDG4JEpGYX2fBJWu3 2urXjxYTJWNb8XLCY9f4EKK1Ijw8UMiIVD64joNzGgYJE5i1WP7kEwS0kekN0uPPTSax MU9rBtooEWM9MPRieC04C8h24MEf7B4I7lisYqupnPwt0pkQl84wDif14umJYsDsV2Fe w51NB8Ce1wbuqpAdzSXZQAI+S7n2TvuK/361ZP0sBGsK1Wv4/BOwkxVu5NcWKMnfKfqH FFSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=KFGKRLMqHwXAGyq5rtZxeGU4x4Lsz1Xzh3y18G9xLjQ=; b=dJqFC0uXPrfEKdLBTJJtUd8uzTJyiF3caT8EGk92k2cu4NDsTHe2XiVjJfnAEOMpLl euFvLl1k+dCnAdTHwwYjpB4LFStxfaJBRo/2OJbTSqVXSrN8ltQYzatNSd7NnNF9hVeA Tk/6gK8b19VKplWjghIcqnSSwhdDZ9U1PKKoPqRih9RsvlgjiId3i2gvKEEs95OP6fbd uuxMsYAP7QLAmBaiiZlf6ovplJdYXjVFnkUUiXGz06qUT1qS7iY93Hk9CQD+7uyyU5Bq UoXTIs9cvUY7N7JncgMDFUEtKtEY8p6dExzZGthBOH2560k8DS9C2M/Oc6khuYirT6Ox CTiA== X-Gm-Message-State: AOAM530O485yExKLcuq9A1o+7cbdDZQf9KangjPJkTWoSIyx1+dvgKoT JPilwbBrXn4KkUhqYJcQKLEhmDr303X+Ld1sjKd2U7Z1 X-Google-Smtp-Source: ABdhPJyxKEOqEWcc/4kg2okwMBBbZh84cxlppJKPnI25hjAz4Y9PACUIaNBqLwNVEjbtboHY0SvkQmQO4xc9ATnjfkc= X-Received: by 2002:ac8:13c1:0:b0:2d5:3c22:8e22 with SMTP id i1-20020ac813c1000000b002d53c228e22mr3916881qtj.306.1645733136539; Thu, 24 Feb 2022 12:05:36 -0800 (PST) List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Sami Halabi Date: Thu, 24 Feb 2022 22:05:27 +0200 Message-ID: Subject: Re: linux debian jail - network problems To: freebsd-jail@freebsd.org, freebsd-net@freebsd.org, freebsd-emulation@freebsd.org, FreeBSD Current Content-Type: multipart/alternative; boundary="000000000000ffa55705d8c91ab9" X-Rspamd-Queue-Id: 4K4P6H2Qd1z4c0R X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=Zm6+f5Pu; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of sodynet1@gmail.com designates 2607:f8b0:4864:20::82f as permitted sender) smtp.mailfrom=sodynet1@gmail.com X-Spamd-Result: default: False [-1.99 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; URI_COUNT_ODD(1.00)[5]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.99)[-0.987]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::82f:from]; HTTP_TO_IP(1.00)[]; MLMMJ_DEST(0.00)[freebsd-jail,freebsd-net,freebsd-emulation,freebsd-current]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N --000000000000ffa55705d8c91ab9 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi, Added Current, maybe will be lucky ;) Anyone have idea how approach and fix this? Sami =D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A =D7=99=D7=95=D7=9D =D7=92=D7=B3, 22 = =D7=91=D7=A4=D7=91=D7=A8=D7=B3 2022, 23:30, =D7=9E=D7=90=D7=AA Sami Halabi = =E2=80=8F: > Hi all, > sorry for the cross post but I need help and I'm not sure where it hangs. > > I create linux jail (debian bullseye) via cbsd. > the jail is being populated with the debian userland.. > so far so good... services running (sshd) and I can login to the jail, I > also can update packages and I can install apache httpd and all works fin= e > (apt install or make from src). > I also manage to install packages even if their scripts depend on "ip" > command that fails: > cbsd@j2> ip > Cannot open netlink socket: Address family not supported by protocol > > ifconfig show empty interfaces: > cbsd@j2> ifconfig > eth0: flags=3D4163 mtu 1500 > ether 00:50:56:0a:b3:a0 (Ethernet) > RX packets 139798314 bytes 12029597009 (11.2 GiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 26879143 bytes 34400160833 (32.0 GiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > lo0: flags=3D4169 mtu 16384 > loop (Local Loopback) > RX packets 28548 bytes 160312960 (152.8 MiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 28548 bytes 160312960 (152.8 MiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > I know linux emulation doesn't implement netlink.. so what I do is fake > the response by replacing /bin/ip by a bash script that prints the correc= t > IP and fakes some other (needed by packages i Installed): > #!/bin/bash > if [ "$1" =3D "-o" ]; then > echo "1: eth0 inet 192.168.1.2/24 brd 192.168.1.255 scope global eth0" > elif [ "$1" =3D "route" ]; then > if [ "$2" =3D "get" ]; then > echo "8.8.8.8 via 192.168.1.2 dev eth0 src > 192.168.1.2 " > else > echo "default via 192.168.1.2 dev eth0" > fi > else > echo "1: eth0: mtu 1500 qdisc mq state > UP qlen 1000" > echo " inet 192.168.1.2 /24 brd 192.168.1.255 scope global eth0" > > > still ifconfig shows no IP... its time to say it a regular jail and *NOT* > VNET. > > *however* package that pull ips via libraries fail.. > eg: installed bind916 (name) in the logs I see these errors (relevant > only): > cbsd@j2> service named start > Starting domain name service...: namednamed: prctl(PR_SET_DUMPABLE) > failed: Invalid argument > cbsd@j2> > > > log file shows: > 22-Feb-2022 23:11:58.705 general: notice: BIND 9 is maintained by Interne= t > Systems Consortium, > 22-Feb-2022 23:11:58.705 general: notice: Inc. (ISC), a non-profit > 501(c)(3) public-benefit > 22-Feb-2022 23:11:58.705 general: notice: corporation. Support and > training for BIND 9 are > 22-Feb-2022 23:11:58.705 general: notice: available at > https://www.isc.org/support > 22-Feb-2022 23:11:58.705 general: notice: > ---------------------------------------------------- > 22-Feb-2022 23:11:58.705 general: info: found 6 CPUs, using 6 worker > threads > 22-Feb-2022 23:11:58.705 general: info: using 6 UDP listeners per interfa= ce > 22-Feb-2022 23:11:58.705 general: info: using up to 21000 sockets > 22-Feb-2022 23:11:58.715 general: info: loading configuration from > '/etc/bind/named.conf' > 22-Feb-2022 23:11:58.715 general: info: reading built-in trust anchors > from file '/etc/bind/bind.keys' > 22-Feb-2022 23:11:58.715 general: info: looking for GeoIP2 databases in > '/usr/share/GeoIP' > 22-Feb-2022 23:11:58.715 general: info: using default UDP/IPv4 port range= : > [1024, 65535] > 22-Feb-2022 23:11:58.715 general: info: using default UDP/IPv6 port range= : > [1024, 65535] > 22-Feb-2022 23:11:58.715 network: info: no IPv6 interfaces found > 22-Feb-2022 23:11:58.715 general: error: ifiter_getifaddrs.c:79: > unexpected error: > 22-Feb-2022 23:11:58.715 general: error: getting interface addresses: > getifaddrs: Address family not supported by protocol > 22-Feb-2022 23:11:58.715 network: warning: not listening on any interface= s > *snip* > *snip* > 22-Feb-2022 23:11:58.735 general: error: socket.c:2405: unexpected error: > 22-Feb-2022 23:11:58.735 general: error: setsockopt(50, IP_RECVTOS) > failed: Protocol not available > 22-Feb-2022 23:11:58.735 general: notice: couldn't add command channel > 127.0.0.1#953: permission denied > 22-Feb-2022 23:11:58.735 general: error: socket.c:2405: unexpected error: > 22-Feb-2022 23:11:58.735 general: error: setsockopt(50, IP_RECVTOS) > failed: Protocol not available > 22-Feb-2022 23:11:58.735 general: notice: couldn't add command channel > 127.0.0.1#953: permission denied > 22-Feb-2022 23:11:58.735 zoneload: info: managed-keys-zone: loaded serial > 24 > 22-Feb-2022 23:11:58.735 zoneload: info: zone 0.in-addr.arpa/IN: loaded > serial 1 > 22-Feb-2022 23:11:58.735 general: error: socket.c:2405: unexpected error: > 22-Feb-2022 23:11:58.735 general: error: setsockopt(512, IP_RECVTOS) > failed: Protocol not available > 22-Feb-2022 23:11:58.735 general: error: socket.c:2405: unexpected error: > 22-Feb-2022 23:11:58.735 general: error: setsockopt(513, IP_RECVTOS) > failed: Protocol not available > 22-Feb-2022 23:11:58.745 zoneload: info: zone 255.in-addr.arpa/IN: loaded > serial 1 > 22-Feb-2022 23:11:58.745 zoneload: info: zone j1.royalshells.com/IN: > loaded serial 2022022106 > 22-Feb-2022 23:11:58.745 notify: info: zone j1.royalshells.com/IN: > sending notifies (serial 2022022106) > 22-Feb-2022 23:11:58.745 general: error: socket.c:2405: unexpected error: > 22-Feb-2022 23:11:58.745 general: error: setsockopt(514, IP_RECVTOS) > failed: Protocol not available > 22-Feb-2022 23:11:58.745 zoneload: info: zone localhost/IN: loaded serial= 2 > 22-Feb-2022 23:11:58.745 general: error: socket.c:2405: unexpected error: > 22-Feb-2022 23:11:58.745 general: error: setsockopt(515, IP_RECVTOS) > failed: Protocol not available > 22-Feb-2022 23:11:58.745 zoneload: info: zone 127.in-addr.arpa/IN: loaded > serial 1 > 22-Feb-2022 23:11:58.745 general: notice: all zones loaded > 22-Feb-2022 23:11:58.745 general: notice: running > 22-Feb-2022 23:11:58.795 general: error: socket.c:2405: unexpected error: > 22-Feb-2022 23:11:58.795 general: error: setsockopt(50, IP_RECVTOS) > failed: Protocol not available > 22-Feb-2022 23:12:58.811 general: error: ifiter_getifaddrs.c:79: > unexpected error: > 22-Feb-2022 23:12:58.811 general: error: getting interface addresses: > getifaddrs: Address family not supported by protocol > 22-Feb-2022 23:12:58.811 network: warning: not listening on any interface= s > > Any Idea how to fix this?? > > cbsd@j2> named -V > BIND 9.16.22-Debian (Extended Support Version) > running on Linux x86_64 3.2.0 FreeBSD 12.3-RELEASE-p1 GENERIC > > installing newer versions > > I have also problems with dovecot mail package.. but will leave it for no= w > > Thanks in advance, > Sami > > --000000000000ffa55705d8c91ab9 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,
Added Current, maybe will be lucky ;= )

Anyone have idea how a= pproach and fix this?

Sa= mi

=D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A =D7=99=D7=95=D7=9D =D7=92=D7= =B3, 22 =D7=91=D7=A4=D7=91=D7=A8=D7=B3 2022, 23:30, =D7=9E=D7=90=D7=AA Sami= Halabi =E2=80=8F<sodynet1@gmail.c= om>:
Hi all= ,
sorry for the cross post but I need help and I'm not sure where i= t hangs.

I create linux jail (debian bullseye) via= cbsd.
the jail is being populated with the debian userland..
so far so good... services running (sshd) and I can login to the jai= l, I also can update packages=C2=A0and I can install apache httpd and all w= orks fine (apt install or make from src).
I also manage to instal= l packages even if their scripts depend on "ip" command that fail= s:
cbsd@j2> ip
Cannot open netlink socket: Address family n= ot supported by protocol

ifconfig show empty i= nterfaces:
cbsd@j2> ifconfig
eth0: flags=3D4163<UP,BROAD= CAST,RUNNING,MULTICAST> =C2=A0mtu 1500
=C2=A0 =C2=A0 =C2=A0 =C2=A0 et= her 00:50:56:0a:b3:a0 =C2=A0(Ethernet)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 RX pa= ckets 139798314 =C2=A0bytes 12029597009 (11.2 GiB)
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 RX errors 0 =C2=A0dropped 0 =C2=A0overruns 0 =C2=A0frame 0
=C2=A0= =C2=A0 =C2=A0 =C2=A0 TX packets 26879143 =C2=A0bytes 34400160833 (32.0 GiB= )
=C2=A0 =C2=A0 =C2=A0 =C2=A0 TX errors 0 =C2=A0dropped 0 overruns 0 =C2= =A0carrier 0 =C2=A0collisions 0

lo0: flags=3D4169<UP,LOOPBACK,RUN= NING,MULTICAST> =C2=A0mtu 16384
=C2=A0 =C2=A0 =C2=A0 =C2=A0 loop =C2= =A0(Local Loopback)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 RX packets 28548 =C2=A0b= ytes 160312960 (152.8 MiB)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 RX errors 0 =C2= =A0dropped 0 =C2=A0overruns 0 =C2=A0frame 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = TX packets 28548 =C2=A0bytes 160312960 (152.8 MiB)
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 TX errors 0 =C2=A0dropped 0 overruns 0 =C2=A0carrier 0 =C2=A0collisi= ons 0

I know linux emulation doesn't imple= ment netlink.. so what I do is fake the response by replacing /bin/ip by a = bash script that prints the correct IP and fakes some other (needed by pack= ages i Installed):
#!/bin/bash
if [ "$1" =3D &qu= ot;-o" ]; then
echo "1: eth0 inet 192.168.1.2/24 brd 192.168.1= .255 scope global eth0"
elif [ "$1" =3D "route"= ]; then
=C2=A0 =C2=A0 =C2=A0 =C2=A0 if [ "$2" =3D "get&q= uot; ]; then
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ech= o "8.8.8.8 via=C2=A0 192.168.1.2=C2=A0=C2=A0=C2=A0dev eth0 =C2=A0src=C2=A0 192.168.1.2=C2=A0 "
=C2=A0 =C2=A0 =C2=A0 =C2=A0 else
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "default via=C2=A0 192.168.1.2=C2=A0=C2=A0=C2=A0dev eth0"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = fi
else
echo "1: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> m= tu 1500 qdisc mq state UP qlen 1000"
echo " =C2=A0inet=C2=A0 192.168.1.2=C2=A0 /24 brd=C2=A0 192.168.1.255 scope global eth0"

still ifconfig shows no IP... its time to say it a regular jail= and *NOT* VNET.

*however* package that pull ips v= ia libraries fail..
eg: installed bind916 (name) in the logs I se= e these errors (relevant only):
cbsd@j2> service named startStarting domain name service...: namednamed: prctl(PR_SET_DUMPABLE) faile= d: Invalid argument
cbsd@j2>


<= div>log file shows:
22-Feb-2022 23:11:58.705 general: notice: BIN= D 9 is maintained by Internet Systems Consortium,
22-Feb-2022 23:11:58.7= 05 general: notice: Inc. (ISC), a non-profit 501(c)(3) public-benefit
22= -Feb-2022 23:11:58.705 general: notice: corporation.=C2=A0 Support and trai= ning for BIND 9 are
22-Feb-2022 23:11:58.705 general: notice: available = at https://www.isc.org/support
22-Feb-2022 23:11:58.705 general: n= otice: ----------------------------------------------------
22-Feb-2022 = 23:11:58.705 general: info: found 6 CPUs, using 6 worker threads
22-Feb-= 2022 23:11:58.705 general: info: using 6 UDP listeners per interface
22-= Feb-2022 23:11:58.705 general: info: using up to 21000 sockets
22-Feb-20= 22 23:11:58.715 general: info: loading configuration from '/etc/bind/na= med.conf'
22-Feb-2022 23:11:58.715 general: info: reading built-in t= rust anchors from file '/etc/bind/bind.keys'
22-Feb-2022 23:11:5= 8.715 general: info: looking for GeoIP2 databases in '/usr/share/GeoIP&= #39;
22-Feb-2022 23:11:58.715 general: info: using default UDP/IPv4 port= range: [1024, 65535]
22-Feb-2022 23:11:58.715 general: info: using defa= ult UDP/IPv6 port range: [1024, 65535]
22-Feb-2022 23:11:58.715 network:= info: no IPv6 interfaces found
22-Feb-2022 23:11:58.715 general: error:= ifiter_getifaddrs.c:79: unexpected error:
22-Feb-2022 23:11:58.715 gene= ral: error: getting interface addresses: getifaddrs: Address family not sup= ported by protocol
22-Feb-2022 23:11:58.715 network: warning: not listen= ing on any interfaces
*snip*
*snip*
22-Fe= b-2022 23:11:58.735 general: error: socket.c:2405: unexpected error:
22-= Feb-2022 23:11:58.735 general: error: setsockopt(50, IP_RECVTOS) failed: Pr= otocol not available
22-Feb-2022 23:11:58.735 general: notice: couldn= 9;t add command channel 127.0.0.1#953: permission denied
22-F= eb-2022 23:11:58.735 general: error: socket.c:2405: unexpected error:
22= -Feb-2022 23:11:58.735 general: error: setsockopt(50, IP_RECVTOS) failed: P= rotocol not available
22-Feb-2022 23:11:58.735 general: notice: couldn&#= 39;t add command channel 127.0.0.1#953: permission denied
22-Feb-2022 23= :11:58.735 zoneload: info: managed-keys-zone: loaded serial 24
22-Feb-20= 22 23:11:58.735 zoneload: info: zone 0.in-addr.arpa/IN: loaded serial 1
= 22-Feb-2022 23:11:58.735 general: error: socket.c:2405: unexpected error:22-Feb-2022 23:11:58.735 general: error: setsockopt(512, IP_RECVTOS) fail= ed: Protocol not available
22-Feb-2022 23:11:58.735 general: error: sock= et.c:2405: unexpected error:
22-Feb-2022 23:11:58.735 general: error: se= tsockopt(513, IP_RECVTOS) failed: Protocol not available
22-Feb-2022 23:= 11:58.745 zoneload: info: zone 255.in-addr.arpa/IN: loaded serial 1
22-F= eb-2022 23:11:58.745 zoneload: info: zone j1.royalshells.com/IN: loa= ded serial 2022022106
22-Feb-2022 23:11:58.745 notify: info: zone j1.= royalshells.com/IN: sending notifies (serial 2022022106)
22-Feb-2022= 23:11:58.745 general: error: socket.c:2405: unexpected error:
22-Feb-20= 22 23:11:58.745 general: error: setsockopt(514, IP_RECVTOS) failed: Protoco= l not available
22-Feb-2022 23:11:58.745 zoneload: info: zone localhost/= IN: loaded serial 2
22-Feb-2022 23:11:58.745 general: error: socket.c:24= 05: unexpected error:
22-Feb-2022 23:11:58.745 general: error: setsockop= t(515, IP_RECVTOS) failed: Protocol not available
22-Feb-2022 23:11:58.7= 45 zoneload: info: zone 127.in-addr.arpa/IN: loaded serial 1
22-Feb-2022= 23:11:58.745 general: notice: all zones loaded
22-Feb-2022 23:11:58.745= general: notice: running
22-Feb-2022 23:11:58.795 general: error: socke= t.c:2405: unexpected error:
22-Feb-2022 23:11:58.795 general: error: set= sockopt(50, IP_RECVTOS) failed: Protocol not available
22-Feb= -2022 23:12:58.811 general: error: ifiter_getifaddrs.c:79: unexpected error= :
22-Feb-2022 23:12:58.811 general: error: getting interface addresses: = getifaddrs: Address family not supported by protocol
22-Feb-2= 022 23:12:58.811 network: warning: not listening on any interfaces

Any Idea how to fix this??

cb= sd@j2> named -V
BIND 9.16.22-Debian (Extended Support Version) <id= :59bfaba>
running on Linux x86_64 3.2.0 FreeBSD 12.3-RELEASE-p1 GENER= IC

installing newer=C2=A0versions=C2=A0
<= div>
I have also problems with dovecot mail package.. but wil= l leave it for now

Thanks in advance,
Sa= mi

--000000000000ffa55705d8c91ab9-- From nobody Fri Feb 25 06:34:57 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 950E319DD040; Fri, 25 Feb 2022 06:35:17 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: from mail-qt1-x832.google.com (mail-qt1-x832.google.com [IPv6:2607:f8b0:4864:20::832]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K4g4h4Xzcz4RQk; Fri, 25 Feb 2022 06:35:16 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: by mail-qt1-x832.google.com with SMTP id bc10so1645834qtb.5; Thu, 24 Feb 2022 22:35:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=fWBo/URZoi7GG03o7vt9qXqiXnYq5muufK8Jg8Xiu00=; b=j87/lFRWmTvftRlyexHI3vUGMB6QWsFMryMTSVvqhyyWXZ6a1ECyUpWQ5EmBNdt4pK AQa8Q5mWGCnlkuP6N3yOCgucXcSKOZb5YwM+cMe5SHJ6leHPZu/edn7rzXVPjiHThX7s WXBYrGdJ9kbkEGq5dMH8qqDBBdIxn+qr2mz9djZ+zDqNZvnRPLUZ3DOSrxTa/b/thQ+y +6Wje0urS0TFid5BajQ+zsFuykJym2HBeG7zc43V059Y5qG8aKiYwGGt16i7mONQBEfm jXQvGjMpQqyjf8MRsGSpmk5pl7KAQdZ8Iddo9wfnx9wRr349ZKxhxQH+1tGbtg0RLrh3 MgSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=fWBo/URZoi7GG03o7vt9qXqiXnYq5muufK8Jg8Xiu00=; b=Vt1XmhGAyaYBT60++KgDz9TUZANutBVAVKXYZkMKF1XQrqn8DqGWD8Y7i47O5nqVCn +2G1dVXEIS4iG9McHH1XoOgRWJw7mfKZv4C6o9EpitM/1xCMI51PcYX/mCbaDFpkEefm ijQbhTbhMiXx8E9TtYKWwBEL5SvSAfnIr1lkN7W1LBEdluRW/02AjyunwQPtAj8/5Yda Vh9+N6oc4oE5Z0IIAxQSsUZ+VHL01vi3Df5t2kZZuTontRRKfIXVwGi38pySBSMYlYCB WKrpNe4BovBsju0Refvs4wj+VrnhPTsndwSYdLdFUosQUoal53V3oX6JuZWJsW2V5Eb2 oHow== X-Gm-Message-State: AOAM531US9E169+7pBVU1DXnIP1nK2Imm4nu7HQMje23Amcvceg+oIPn MRKyj14Zol1u8mIhevb+LqAjmAVAL5eheQWEOzp11+eJ X-Google-Smtp-Source: ABdhPJxjM+w/VPzaFpdLyU0Lp32W+EWL2gU3bKR4kOTWpeXg03n5BtrCKcQnz0kjhH6uopcBuPqpsD8aQu5cEwcn48o= X-Received: by 2002:ac8:5a49:0:b0:2de:6fa3:f928 with SMTP id o9-20020ac85a49000000b002de6fa3f928mr5581228qta.677.1645770909898; Thu, 24 Feb 2022 22:35:09 -0800 (PST) List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 References: <8020452A-63EA-4424-8D20-CC9B9397B603@gmail.com> In-Reply-To: <8020452A-63EA-4424-8D20-CC9B9397B603@gmail.com> From: Sami Halabi Date: Fri, 25 Feb 2022 08:34:57 +0200 Message-ID: Subject: Re: linux debian jail - network problems To: Zhenlei Huang Cc: freebsd-jail@freebsd.org, freebsd-net@freebsd.org, freebsd-emulation@freebsd.org, FreeBSD Current Content-Type: multipart/alternative; boundary="000000000000775b5f05d8d1e65b" X-Rspamd-Queue-Id: 4K4g4h4Xzcz4RQk X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b="j87/lFRW"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of sodynet1@gmail.com designates 2607:f8b0:4864:20::832 as permitted sender) smtp.mailfrom=sodynet1@gmail.com X-Spamd-Result: default: False [0.00 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; URI_COUNT_ODD(1.00)[5]; RCPT_COUNT_FIVE(0.00)[5]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; NEURAL_SPAM_SHORT(1.00)[1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::832:from]; HTTP_TO_IP(1.00)[]; MLMMJ_DEST(0.00)[freebsd-jail,freebsd-net,freebsd-emulation,freebsd-current]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N --000000000000775b5f05d8d1e65b Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi, Thank you for your response.. I wonder if Is it really only netlink problem= ? Their are fee problems in the logs.. I dont kbow if they all related only to netlink (prctl immutable for example).. I also saw oncompatibilities in socket.c .... Btw: I tried to enter the link you sent and it asked for username and password.. its not public review? Sami =D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A =D7=99=D7=95=D7=9D =D7=95=D7=B3, 25 = =D7=91=D7=A4=D7=91=D7=A8=D7=B3 2022, 04:18, =D7=9E=D7=90=D7=AA Zhenlei Huan= g =E2=80=8F< zlei.huang@gmail.com>: > Hi, > You can also track the WIP netlink feature, > https://reviews.freebsd.org/D33975 > > On Feb 25, 2022, at 4:05 AM, Sami Halabi wrote: > > Hi, > Added Current, maybe will be lucky ;) > > Anyone have idea how approach and fix this? > > Sami > > =D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A =D7=99=D7=95=D7=9D =D7=92=D7=B3, 22 = =D7=91=D7=A4=D7=91=D7=A8=D7=B3 2022, 23:30, =D7=9E=D7=90=D7=AA Sami Halabi = =E2=80=8F >: > >> Hi all, >> sorry for the cross post but I need help and I'm not sure where it hangs= . >> >> I create linux jail (debian bullseye) via cbsd. >> the jail is being populated with the debian userland.. >> so far so good... services running (sshd) and I can login to the jail, I >> also can update packages and I can install apache httpd and all works fi= ne >> (apt install or make from src). >> I also manage to install packages even if their scripts depend on "ip" >> command that fails: >> cbsd@j2> ip >> Cannot open netlink socket: Address family not supported by protocol >> >> ifconfig show empty interfaces: >> cbsd@j2> ifconfig >> eth0: flags=3D4163 mtu 1500 >> ether 00:50:56:0a:b3:a0 (Ethernet) >> RX packets 139798314 bytes 12029597009 (11.2 GiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 26879143 bytes 34400160833 (32.0 GiB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> lo0: flags=3D4169 mtu 16384 >> loop (Local Loopback) >> RX packets 28548 bytes 160312960 (152.8 MiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 28548 bytes 160312960 (152.8 MiB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> I know linux emulation doesn't implement netlink.. so what I do is fake >> the response by replacing /bin/ip by a bash script that prints the corre= ct >> IP and fakes some other (needed by packages i Installed): >> #!/bin/bash >> if [ "$1" =3D "-o" ]; then >> echo "1: eth0 inet 192.168.1.2/24 brd 192.168.1.255 scope global eth0" >> elif [ "$1" =3D "route" ]; then >> if [ "$2" =3D "get" ]; then >> echo "8.8.8.8 via 192.168.1.2 dev eth0 src >> 192.168.1.2 " >> else >> echo "default via 192.168.1.2 dev eth0" >> fi >> else >> echo "1: eth0: mtu 1500 qdisc mq state >> UP qlen 1000" >> echo " inet 192.168.1.2 /24 brd 192.168.1.255 scope global eth0" >> >> >> still ifconfig shows no IP... its time to say it a regular jail and *NOT= * >> VNET. >> >> *however* package that pull ips via libraries fail.. >> eg: installed bind916 (name) in the logs I see these errors (relevant >> only): >> cbsd@j2> service named start >> Starting domain name service...: namednamed: prctl(PR_SET_DUMPABLE) >> failed: Invalid argument >> cbsd@j2> >> >> >> log file shows: >> 22-Feb-2022 23:11:58.705 general: notice: BIND 9 is maintained by >> Internet Systems Consortium, >> 22-Feb-2022 23:11:58.705 general: notice: Inc. (ISC), a non-profit >> 501(c)(3) public-benefit >> 22-Feb-2022 23:11:58.705 general: notice: corporation. Support and >> training for BIND 9 are >> 22-Feb-2022 23:11:58.705 general: notice: available at >> https://www.isc.org/support >> 22-Feb-2022 23:11:58.705 general: notice: >> ---------------------------------------------------- >> 22-Feb-2022 23:11:58.705 general: info: found 6 CPUs, using 6 worker >> threads >> 22-Feb-2022 23:11:58.705 general: info: using 6 UDP listeners per >> interface >> 22-Feb-2022 23:11:58.705 general: info: using up to 21000 sockets >> 22-Feb-2022 23:11:58.715 general: info: loading configuration from >> '/etc/bind/named.conf' >> 22-Feb-2022 23:11:58.715 general: info: reading built-in trust anchors >> from file '/etc/bind/bind.keys' >> 22-Feb-2022 23:11:58.715 general: info: looking for GeoIP2 databases in >> '/usr/share/GeoIP' >> 22-Feb-2022 23:11:58.715 general: info: using default UDP/IPv4 port >> range: [1024, 65535] >> 22-Feb-2022 23:11:58.715 general: info: using default UDP/IPv6 port >> range: [1024, 65535] >> 22-Feb-2022 23:11:58.715 network: info: no IPv6 interfaces found >> 22-Feb-2022 23:11:58.715 general: error: ifiter_getifaddrs.c:79: >> unexpected error: >> 22-Feb-2022 23:11:58.715 general: error: getting interface addresses: >> getifaddrs: Address family not supported by protocol >> 22-Feb-2022 23:11:58.715 network: warning: not listening on any interfac= es >> *snip* >> *snip* >> 22-Feb-2022 23:11:58.735 general: error: socket.c:2405: unexpected error= : >> 22-Feb-2022 23:11:58.735 general: error: setsockopt(50, IP_RECVTOS) >> failed: Protocol not available >> 22-Feb-2022 23:11:58.735 general: notice: couldn't add command channel >> 127.0.0.1#953: permission denied >> 22-Feb-2022 23:11:58.735 general: error: socket.c:2405: unexpected error= : >> 22-Feb-2022 23:11:58.735 general: error: setsockopt(50, IP_RECVTOS) >> failed: Protocol not available >> 22-Feb-2022 23:11:58.735 general: notice: couldn't add command channel >> 127.0.0.1#953: permission denied >> 22-Feb-2022 23:11:58.735 zoneload: info: managed-keys-zone: loaded seria= l >> 24 >> 22-Feb-2022 23:11:58.735 zoneload: info: zone 0.in-addr.arpa/IN: loaded >> serial 1 >> 22-Feb-2022 23:11:58.735 general: error: socket.c:2405: unexpected error= : >> 22-Feb-2022 23:11:58.735 general: error: setsockopt(512, IP_RECVTOS) >> failed: Protocol not available >> 22-Feb-2022 23:11:58.735 general: error: socket.c:2405: unexpected error= : >> 22-Feb-2022 23:11:58.735 general: error: setsockopt(513, IP_RECVTOS) >> failed: Protocol not available >> 22-Feb-2022 23:11:58.745 zoneload: info: zone 255.in-addr.arpa/IN: loade= d >> serial 1 >> 22-Feb-2022 23:11:58.745 zoneload: info: zone j1.royalshells.com/IN: >> loaded serial 2022022106 >> 22-Feb-2022 23:11:58.745 notify: info: zone j1.royalshells.com/IN: >> sending notifies (serial 2022022106) >> 22-Feb-2022 23:11:58.745 general: error: socket.c:2405: unexpected error= : >> 22-Feb-2022 23:11:58.745 general: error: setsockopt(514, IP_RECVTOS) >> failed: Protocol not available >> 22-Feb-2022 23:11:58.745 zoneload: info: zone localhost/IN: loaded seria= l >> 2 >> 22-Feb-2022 23:11:58.745 general: error: socket.c:2405: unexpected error= : >> 22-Feb-2022 23:11:58.745 general: error: setsockopt(515, IP_RECVTOS) >> failed: Protocol not available >> 22-Feb-2022 23:11:58.745 zoneload: info: zone 127.in-addr.arpa/IN: loade= d >> serial 1 >> 22-Feb-2022 23:11:58.745 general: notice: all zones loaded >> 22-Feb-2022 23:11:58.745 general: notice: running >> 22-Feb-2022 23:11:58.795 general: error: socket.c:2405: unexpected error= : >> 22-Feb-2022 23:11:58.795 general: error: setsockopt(50, IP_RECVTOS) >> failed: Protocol not available >> 22-Feb-2022 23:12:58.811 general: error: ifiter_getifaddrs.c:79: >> unexpected error: >> 22-Feb-2022 23:12:58.811 general: error: getting interface addresses: >> getifaddrs: Address family not supported by protocol >> 22-Feb-2022 23:12:58.811 network: warning: not listening on any interfac= es >> >> Any Idea how to fix this?? >> >> cbsd@j2> named -V >> BIND 9.16.22-Debian (Extended Support Version) >> running on Linux x86_64 3.2.0 FreeBSD 12.3-RELEASE-p1 GENERIC >> >> installing newer versions >> >> I have also problems with dovecot mail package.. but will leave it for n= ow >> >> Thanks in advance, >> Sami >> >> > --000000000000775b5f05d8d1e65b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,
Thank you for your response.. I wond= er if Is it really only netlink problem?
Their are f= ee problems in the logs.. I dont kbow if they all related only to netlink (= prctl immutable for example).. I also saw oncompatibilities in socket.c ...= .

Btw: I tried to enter = the link you sent and it asked for username and password.. its not public r= eview?

Sami
<= br>
=D7=91= =D7=AA=D7=90=D7=A8=D7=99=D7=9A =D7=99=D7=95=D7=9D =D7=95=D7=B3, 25 =D7=91= =D7=A4=D7=91=D7=A8=D7=B3 2022, 04:18, =D7=9E=D7=90=D7=AA Zhenlei Huang =E2= =80=8F<zlei.huang@gmail.com&= gt;:
Hi,
You can also track the WIP netli= nk feature,=C2=A0https://reviews.freebsd.org/D33975
On Feb 25, 2022, at 4:05 AM, Sami Halabi &= lt;sodynet1@gmail.com> wrote:

Hi,Added Current, maybe will be lucky ;)

Anyone have idea how approach and fix this?

Sami

=D7=91=D7=AA=D7= =90=D7=A8=D7=99=D7=9A =D7=99=D7=95=D7=9D =D7=92=D7=B3, 22 =D7=91=D7=A4=D7= =91=D7=A8=D7=B3 2022, 23:30, =D7=9E=D7=90=D7=AA Sami Halabi =E2=80=8F<so= dynet1@gmail.com>:
Hi all,
sorry for the cross post but I need help and I'm n= ot sure where it hangs.

I create linux jail (debia= n bullseye) via cbsd.
the jail is being populated with the debian= userland..
so far so good... services running (sshd) and I can l= ogin to the jail, I also can update packages=C2=A0and I can install apache = httpd and all works fine (apt install or make from src).
I also m= anage to install packages even if their scripts depend on "ip" co= mmand that fails:
cbsd@j2> ip
Cannot open netlink socket: A= ddress family not supported by protocol

ifconf= ig show empty interfaces:
cbsd@j2> ifconfig
eth0: flags=3D4= 163<UP,BROADCAST,RUNNING,MULTICAST> =C2=A0mtu 1500
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 ether 00:50:56:0a:b3:a0 =C2=A0(Ethernet)
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 RX packets 139798314 =C2=A0bytes 12029597009 (11.2 GiB)
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 RX errors 0 =C2=A0dropped 0 =C2=A0overruns 0 =C2= =A0frame 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 TX packets 26879143 =C2=A0bytes 3= 4400160833 (32.0 GiB)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 TX errors 0 =C2=A0drop= ped 0 overruns 0 =C2=A0carrier 0 =C2=A0collisions 0

lo0: flags=3D416= 9<UP,LOOPBACK,RUNNING,MULTICAST> =C2=A0mtu 16384
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 loop =C2=A0(Local Loopback)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 RX pa= ckets 28548 =C2=A0bytes 160312960 (152.8 MiB)
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 RX errors 0 =C2=A0dropped 0 =C2=A0overruns 0 =C2=A0frame 0
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 TX packets 28548 =C2=A0bytes 160312960 (152.8 MiB)
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 TX errors 0 =C2=A0dropped 0 overruns 0 =C2=A0ca= rrier 0 =C2=A0collisions 0

I know linux emulat= ion doesn't implement netlink.. so what I do is fake the response by re= placing /bin/ip by a bash script that prints the correct IP and fakes some = other (needed by packages i Installed):
#!/bin/bash
if [ &= quot;$1" =3D "-o" ]; then
echo "1: eth0 inet = 192.168.1.2/24 brd 192.168.1.255 scope global eth0"
elif [ &quo= t;$1" =3D "route" ]; then
=C2=A0 =C2=A0 =C2=A0 =C2=A0 if = [ "$2" =3D "get" ]; then
=C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "8.8.8.8 via=C2=A0 192.168.1.2=C2=A0=C2=A0=C2=A0dev eth0 =C2=A0src=C2=A0 192.168.1.2=C2=A0 "
=C2=A0 =C2=A0 =C2=A0 =C2=A0 else
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "default via=C2=A0 192.168.1.2=C2=A0=C2=A0=C2=A0dev eth0"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = fi
else
echo "1: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> m= tu 1500 qdisc mq state UP qlen 1000"
echo " =C2=A0inet=C2=A0 192.168.1.2=C2=A0 /24 brd=C2=A0 192.168.1.255 scope global eth0"

still ifconfig shows no IP... its time to say it a regular jail= and *NOT* VNET.

*however* package that pull ips v= ia libraries fail..
eg: installed bind916 (name) in the logs I se= e these errors (relevant only):
cbsd@j2> service named startStarting domain name service...: namednamed: prctl(PR_SET_DUMPABLE) faile= d: Invalid argument
cbsd@j2>


<= div>log file shows:
22-Feb-2022 23:11:58.705 general: notice: BIN= D 9 is maintained by Internet Systems Consortium,
22-Feb-2022 23:11:58.7= 05 general: notice: Inc. (ISC), a non-profit 501(c)(3) public-benefit
22= -Feb-2022 23:11:58.705 general: notice: corporation.=C2=A0 Support and trai= ning for BIND 9 are
22-Feb-2022 23:11:58.705 general: notice: available = at https://www.isc.org/support
22-Feb-2022 23:11:58.705= general: notice: ----------------------------------------------------
2= 2-Feb-2022 23:11:58.705 general: info: found 6 CPUs, using 6 worker threads=
22-Feb-2022 23:11:58.705 general: info: using 6 UDP listeners per inter= face
22-Feb-2022 23:11:58.705 general: info: using up to 21000 sockets22-Feb-2022 23:11:58.715 general: info: loading configuration from '/= etc/bind/named.conf'
22-Feb-2022 23:11:58.715 general: info: reading= built-in trust anchors from file '/etc/bind/bind.keys'
22-Feb-2= 022 23:11:58.715 general: info: looking for GeoIP2 databases in '/usr/s= hare/GeoIP'
22-Feb-2022 23:11:58.715 general: info: using default UD= P/IPv4 port range: [1024, 65535]
22-Feb-2022 23:11:58.715 general: info:= using default UDP/IPv6 port range: [1024, 65535]
22-Feb-2022 23:11:58.7= 15 network: info: no IPv6 interfaces found
22-Feb-2022 23:11:58.715 gene= ral: error: ifiter_getifaddrs.c:79: unexpected error:
22-Feb-2022 23:11:= 58.715 general: error: getting interface addresses: getifaddrs: Address fam= ily not supported by protocol
22-Feb-2022 23:11:58.715 network: warning:= not listening on any interfaces
*snip*
*snip*
22-Feb-2022 23:11:58.735 general: error: socket.c:2405: unexpected er= ror:
22-Feb-2022 23:11:58.735 general: error: setsockopt(50, IP_RECVTOS)= failed: Protocol not available
22-Feb-2022 23:11:58.735 general: notice= : couldn't add command channel 127.0.0.1#953: permission denied
22-Feb-2022 23:11:58.735 general: error: socket.c:2405: unexpected e= rror:
22-Feb-2022 23:11:58.735 general: error: setsockopt(50, IP_RECVTOS= ) failed: Protocol not available
22-Feb-2022 23:11:58.735 general: notic= e: couldn't add command channel 127.0.0.1#953: permission denied
22-= Feb-2022 23:11:58.735 zoneload: info: managed-keys-zone: loaded serial 2422-Feb-2022 23:11:58.735 zoneload: info: zone 0.in-addr.arpa/IN: loaded s= erial 1
22-Feb-2022 23:11:58.735 general: error: socket.c:2405: unexpect= ed error:
22-Feb-2022 23:11:58.735 general: error: setsockopt(512, IP_RE= CVTOS) failed: Protocol not available
22-Feb-2022 23:11:58.735 general: = error: socket.c:2405: unexpected error:
22-Feb-2022 23:11:58.735 general= : error: setsockopt(513, IP_RECVTOS) failed: Protocol not available
22-F= eb-2022 23:11:58.745 zoneload: info: zone 255.in-addr.arpa/IN: loaded seria= l 1
22-Feb-2022 23:11:58.745 zoneload: info: zone j1.royal= shells.com/IN: loaded serial 2022022106
22-Feb-2022 23:11:58.745 not= ify: info: zone j1.royalshells.com/IN: sending notifies (= serial 2022022106)
22-Feb-2022 23:11:58.745 general: error: socket.c:240= 5: unexpected error:
22-Feb-2022 23:11:58.745 general: error: setsockopt= (514, IP_RECVTOS) failed: Protocol not available
22-Feb-2022 23:11:58.74= 5 zoneload: info: zone localhost/IN: loaded serial 2
22-Feb-2022 23:11:5= 8.745 general: error: socket.c:2405: unexpected error:
22-Feb-2022 23:11= :58.745 general: error: setsockopt(515, IP_RECVTOS) failed: Protocol not av= ailable
22-Feb-2022 23:11:58.745 zoneload: info: zone 127.in-addr.arpa/I= N: loaded serial 1
22-Feb-2022 23:11:58.745 general: notice: all zones l= oaded
22-Feb-2022 23:11:58.745 general: notice: running
22-Feb-2022 2= 3:11:58.795 general: error: socket.c:2405: unexpected error:
22-Feb-2022= 23:11:58.795 general: error: setsockopt(50, IP_RECVTOS) failed: Protocol n= ot available
22-Feb-2022 23:12:58.811 general: error: ifiter_= getifaddrs.c:79: unexpected error:
22-Feb-2022 23:12:58.811 general: err= or: getting interface addresses: getifaddrs: Address family not supported b= y protocol
22-Feb-2022 23:12:58.811 network: warning: not lis= tening on any interfaces

Any Idea how to fix t= his??

cbsd@j2> named -V
BIND 9.16.22-Debian = (Extended Support Version) <id:59bfaba>
running on Linux x86_64 3.= 2.0 FreeBSD 12.3-RELEASE-p1 GENERIC

installing= newer=C2=A0versions=C2=A0

I have also problems wi= th dovecot mail package.. but will leave it for now

Thanks in advance,
Sami


--000000000000775b5f05d8d1e65b-- From nobody Wed Mar 2 02:22:33 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4DC5B19DA3EB; Wed, 2 Mar 2022 02:22:48 +0000 (UTC) (envelope-from zlei.huang@gmail.com) Received: from mail-pj1-x102d.google.com (mail-pj1-x102d.google.com [IPv6:2607:f8b0:4864:20::102d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K7dF31Ftjz4l4H; Wed, 2 Mar 2022 02:22:47 +0000 (UTC) (envelope-from zlei.huang@gmail.com) Received: by mail-pj1-x102d.google.com with SMTP id bx5so508264pjb.3; Tue, 01 Mar 2022 18:22:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=dF8GjNyhrJygx7QdROuqlz/OYPY28u9QW81FNQZNIH0=; b=dEon5KNtA6m8Vm+ot8JDvUYr7mn4XBsFEeUHc+1B5bQkB4xezaF+lNHp3eZ+raQoS0 duLeacQ8N/TtczVwLNa34XBluBM47PhBmFtoTcguC7t/xFA+F4zvdVLs+mRiBKLamO9/ x2t+HRmvvS5BWgLo1m8UQBdp9ZBQS46fNiSFrN/A/uyHviwWu5nD0YceRG6yyIA5wdwX Esd2Oi2XRb5hwW03aEUj4WYk6lU6/TQGq52GO7haU1fvN/TjfHlkB38Z5g0oTK7KoSEe qfbjudFrDEqYIfzChd6HHYaS1pEMRSiHblwYfqrkMO1mm9qqlVHsbr8Nej4A3HhPyzJO 9RbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=dF8GjNyhrJygx7QdROuqlz/OYPY28u9QW81FNQZNIH0=; b=QWEF825u145sE/LrqC0XvrtqRPZyQ7Layj5CpvyHqjwiSk2CxP1JZlV6RHV6v1SGwJ tBwQbYkAUXCCFOZ9v7R1m75nYiToF6BlENrTlAdA5cv09ifB/6h9BvIsyPbo2eyLiIQ7 pwTxXieTCtOf7mFhqIRq94SojYyXCwMQGSOGhRTlEN5BsFpiXrgGT1rbvvBH5lg8tQ4x 4scLpLwFSjfJdSXpCUhIxtrJOHXmvS5ce4ObKkImnR/AzoAUZVxad75AwRgq3X2LbLt9 x+WORIKxZRRuIu+YWSgXSaxbPu9SFb6xNTIEjgUvOQ3ai6gUhlhZB4E9p3wlOTiXeb1D WfkA== X-Gm-Message-State: AOAM531Z3gg2HVgo93N00qHBxt3EMyVRtZ9P/Er2o+UKkTa/VthLo8mq KU9lA1xvfC69HYqsLGMkbhlEpT4SrCEoqAJT X-Google-Smtp-Source: ABdhPJyyJD/1PTDvBB0G4A7iiKl9d1RqIpN8fZ/agq6bFzMtr1V11vNZDtASVQu/R7Xin1cN1Aa3IQ== X-Received: by 2002:a17:902:e88e:b0:150:25e3:739c with SMTP id w14-20020a170902e88e00b0015025e3739cmr24569044plg.13.1646187765902; Tue, 01 Mar 2022 18:22:45 -0800 (PST) Received: from [172.17.252.129] (ns1.oxydns.net. [45.32.91.63]) by smtp.gmail.com with ESMTPSA id j9-20020a056a00234900b004f3b1c23497sm18700478pfj.101.2022.03.01.18.22.43 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 01 Mar 2022 18:22:45 -0800 (PST) From: Zhenlei Huang Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_783D5F85-39C4-4639-BA1C-DCB831F843AE" List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.7\)) Subject: Re: linux debian jail - network problems Date: Wed, 2 Mar 2022 10:22:33 +0800 In-Reply-To: Cc: freebsd-jail@freebsd.org, freebsd-net@freebsd.org, FreeBSD Current , "Alexander V. Chernikov" To: Sami Halabi References: <8020452A-63EA-4424-8D20-CC9B9397B603@gmail.com> X-Mailer: Apple Mail (2.3608.120.23.2.7) X-Rspamd-Queue-Id: 4K7dF31Ftjz4l4H X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=dEon5KNt; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of zleihuang@gmail.com designates 2607:f8b0:4864:20::102d as permitted sender) smtp.mailfrom=zleihuang@gmail.com X-Spamd-Result: default: False [-2.35 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MV_CASE(0.50)[]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-0.85)[-0.846]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::102d:from]; HTTP_TO_IP(1.00)[]; MLMMJ_DEST(0.00)[freebsd-jail,freebsd-net,freebsd-current]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_783D5F85-39C4-4639-BA1C-DCB831F843AE Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Mar 1, 2022, at 6:42 PM, Sami Halabi wrote: >=20 > How can I see the netlink wip status ? Sorry it is not currently public visible. FreeBSD's Phabricator is a = tool that is development focused. If you're interested in it, please CC the author Alexander V. Chernikov = . >=20 > =D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A =D7=99=D7=95=D7=9D =D7=95=D7=B3, = 25 =D7=91=D7=A4=D7=91=D7=A8=D7=B3 2022, 08:34, =D7=9E=D7=90=D7=AA Sami = Halabi =E2=80=8F>: > Hi, > Thank you for your response.. I wonder if Is it really only netlink = problem? Maybe is or not. I'm not familiar with Linux emulation. You can refer to=20= 1. https://docs.freebsd.org/en/articles/linux-emulation/ = 2. https://wiki.freebsd.org/Linuxulator = > Their are fee problems in the logs.. I dont kbow if they all related = only to netlink (prctl immutable for example).. I also saw = oncompatibilities in socket.c .... >=20 > Btw: I tried to enter the link you sent and it asked for username and = password.. its not public review? >=20 > Sami >=20 > =D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A =D7=99=D7=95=D7=9D =D7=95=D7=B3, = 25 =D7=91=D7=A4=D7=91=D7=A8=D7=B3 2022, 04:18, =D7=9E=D7=90=D7=AA = Zhenlei Huang =E2=80=8F>: > Hi, > You can also track the WIP netlink feature, = https://reviews.freebsd.org/D33975 >=20 >> On Feb 25, 2022, at 4:05 AM, Sami Halabi > wrote: >>=20 >> Hi, >> Added Current, maybe will be lucky ;) >>=20 >> Anyone have idea how approach and fix this? >>=20 >> Sami >>=20 >> =D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A =D7=99=D7=95=D7=9D =D7=92=D7=B3, = 22 =D7=91=D7=A4=D7=91=D7=A8=D7=B3 2022, 23:30, =D7=9E=D7=90=D7=AA Sami = Halabi =E2=80=8F>: >> Hi all, >> sorry for the cross post but I need help and I'm not sure where it = hangs. >>=20 >> I create linux jail (debian bullseye) via cbsd. >> the jail is being populated with the debian userland.. >> so far so good... services running (sshd) and I can login to the = jail, I also can update packages and I can install apache httpd and all = works fine (apt install or make from src). >> I also manage to install packages even if their scripts depend on = "ip" command that fails: >> cbsd@j2> ip >> Cannot open netlink socket: Address family not supported by protocol >>=20 >> ifconfig show empty interfaces: >> cbsd@j2> ifconfig >> eth0: flags=3D4163 mtu 1500 >> ether 00:50:56:0a:b3:a0 (Ethernet) >> RX packets 139798314 bytes 12029597009 (11.2 GiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 26879143 bytes 34400160833 (32.0 GiB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >>=20 >> lo0: flags=3D4169 mtu 16384 >> loop (Local Loopback) >> RX packets 28548 bytes 160312960 (152.8 MiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 28548 bytes 160312960 (152.8 MiB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >>=20 >> I know linux emulation doesn't implement netlink.. so what I do is = fake the response by replacing /bin/ip by a bash script that prints the = correct IP and fakes some other (needed by packages i Installed): >> #!/bin/bash >> if [ "$1" =3D "-o" ]; then >> echo "1: eth0 inet 192.168.1.2/24 brd = 192.168.1.255 scope global eth0" >> elif [ "$1" =3D "route" ]; then >> if [ "$2" =3D "get" ]; then >> echo "8.8.8.8 via 192.168.1.2 dev eth0 src = 192.168.1.2 " >> else >> echo "default via 192.168.1.2 dev eth0" >> fi >> else >> echo "1: eth0: mtu 1500 qdisc mq = state UP qlen 1000" >> echo " inet 192.168.1.2 /24 brd 192.168.1.255 scope global eth0" >>=20 >>=20 >> still ifconfig shows no IP... its time to say it a regular jail and = *NOT* VNET. >>=20 >> *however* package that pull ips via libraries fail.. >> eg: installed bind916 (name) in the logs I see these errors (relevant = only): >> cbsd@j2> service named start >> Starting domain name service...: namednamed: prctl(PR_SET_DUMPABLE) = failed: Invalid argument >> cbsd@j2> >>=20 >>=20 >> log file shows: >> 22-Feb-2022 23:11:58.705 general: notice: BIND 9 is maintained by = Internet Systems Consortium, >> 22-Feb-2022 23:11:58.705 general: notice: Inc. (ISC), a non-profit = 501(c)(3) public-benefit >> 22-Feb-2022 23:11:58.705 general: notice: corporation. Support and = training for BIND 9 are >> 22-Feb-2022 23:11:58.705 general: notice: available at = https://www.isc.org/support >> 22-Feb-2022 23:11:58.705 general: notice: = ---------------------------------------------------- >> 22-Feb-2022 23:11:58.705 general: info: found 6 CPUs, using 6 worker = threads >> 22-Feb-2022 23:11:58.705 general: info: using 6 UDP listeners per = interface >> 22-Feb-2022 23:11:58.705 general: info: using up to 21000 sockets >> 22-Feb-2022 23:11:58.715 general: info: loading configuration from = '/etc/bind/named.conf' >> 22-Feb-2022 23:11:58.715 general: info: reading built-in trust = anchors from file '/etc/bind/bind.keys' >> 22-Feb-2022 23:11:58.715 general: info: looking for GeoIP2 databases = in '/usr/share/GeoIP' >> 22-Feb-2022 23:11:58.715 general: info: using default UDP/IPv4 port = range: [1024, 65535] >> 22-Feb-2022 23:11:58.715 general: info: using default UDP/IPv6 port = range: [1024, 65535] >> 22-Feb-2022 23:11:58.715 network: info: no IPv6 interfaces found >> 22-Feb-2022 23:11:58.715 general: error: ifiter_getifaddrs.c:79: = unexpected error: >> 22-Feb-2022 23:11:58.715 general: error: getting interface addresses: = getifaddrs: Address family not supported by protocol >> 22-Feb-2022 23:11:58.715 network: warning: not listening on any = interfaces >> *snip* >> *snip* >> 22-Feb-2022 23:11:58.735 general: error: socket.c:2405: unexpected = error: >> 22-Feb-2022 23:11:58.735 general: error: setsockopt(50, IP_RECVTOS) = failed: Protocol not available >> 22-Feb-2022 23:11:58.735 general: notice: couldn't add command = channel 127.0.0.1#953: permission denied >> 22-Feb-2022 23:11:58.735 general: error: socket.c:2405: unexpected = error: >> 22-Feb-2022 23:11:58.735 general: error: setsockopt(50, IP_RECVTOS) = failed: Protocol not available >> 22-Feb-2022 23:11:58.735 general: notice: couldn't add command = channel 127.0.0.1#953: permission denied >> 22-Feb-2022 23:11:58.735 zoneload: info: managed-keys-zone: loaded = serial 24 >> 22-Feb-2022 23:11:58.735 zoneload: info: zone 0.in-addr.arpa/IN: = loaded serial 1 >> 22-Feb-2022 23:11:58.735 general: error: socket.c:2405: unexpected = error: >> 22-Feb-2022 23:11:58.735 general: error: setsockopt(512, IP_RECVTOS) = failed: Protocol not available >> 22-Feb-2022 23:11:58.735 general: error: socket.c:2405: unexpected = error: >> 22-Feb-2022 23:11:58.735 general: error: setsockopt(513, IP_RECVTOS) = failed: Protocol not available >> 22-Feb-2022 23:11:58.745 zoneload: info: zone 255.in-addr.arpa/IN: = loaded serial 1 >> 22-Feb-2022 23:11:58.745 zoneload: info: zone j1.royalshells.com/IN = : loaded serial 2022022106 >> 22-Feb-2022 23:11:58.745 notify: info: zone j1.royalshells.com/IN = : sending notifies (serial 2022022106) >> 22-Feb-2022 23:11:58.745 general: error: socket.c:2405: unexpected = error: >> 22-Feb-2022 23:11:58.745 general: error: setsockopt(514, IP_RECVTOS) = failed: Protocol not available >> 22-Feb-2022 23:11:58.745 zoneload: info: zone localhost/IN: loaded = serial 2 >> 22-Feb-2022 23:11:58.745 general: error: socket.c:2405: unexpected = error: >> 22-Feb-2022 23:11:58.745 general: error: setsockopt(515, IP_RECVTOS) = failed: Protocol not available >> 22-Feb-2022 23:11:58.745 zoneload: info: zone 127.in-addr.arpa/IN: = loaded serial 1 >> 22-Feb-2022 23:11:58.745 general: notice: all zones loaded >> 22-Feb-2022 23:11:58.745 general: notice: running >> 22-Feb-2022 23:11:58.795 general: error: socket.c:2405: unexpected = error: >> 22-Feb-2022 23:11:58.795 general: error: setsockopt(50, IP_RECVTOS) = failed: Protocol not available >> 22-Feb-2022 23:12:58.811 general: error: ifiter_getifaddrs.c:79: = unexpected error: >> 22-Feb-2022 23:12:58.811 general: error: getting interface addresses: = getifaddrs: Address family not supported by protocol >> 22-Feb-2022 23:12:58.811 network: warning: not listening on any = interfaces >>=20 >> Any Idea how to fix this?? >>=20 >> cbsd@j2> named -V >> BIND 9.16.22-Debian (Extended Support Version) >> running on Linux x86_64 3.2.0 FreeBSD 12.3-RELEASE-p1 GENERIC >>=20 >> installing newer versions=20 >>=20 >> I have also problems with dovecot mail package.. but will leave it = for now >>=20 >> Thanks in advance, >> Sami >>=20 >=20 --Apple-Mail=_783D5F85-39C4-4639-BA1C-DCB831F843AE Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On Mar 1, 2022, at 6:42 PM, Sami Halabi <sodynet1@gmail.com> = wrote:

How can I see the netlink wip status = ?

Sorry it is not = currently public visible. FreeBSD's Phabricator is a tool that is = development focused.
If you're interested in it, please CC the = author Alexander V. Chernikov .


=D7=91=D7=AA=D7= =90=D7=A8=D7=99=D7=9A =D7=99=D7=95=D7=9D =D7=95=D7=B3, 25 =D7=91=D7=A4=D7=91= =D7=A8=D7=B3 2022, 08:34, =D7=9E=D7=90=D7=AA Sami Halabi =E2=80=8F<sodynet1@gmail.com>:
Hi,
Thank you for your response.. I wonder if Is it really only = netlink = problem?

Maybe is or not. I'm not familiar with Linux = emulation. You can refer to 
=
Their are fee problems in the = logs.. I dont kbow if they all related only to netlink (prctl immutable = for example).. I also saw oncompatibilities in socket.c ....

Btw: I tried to enter the link you sent and it asked for = username and password.. its not public review?

Sami

=D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A = =D7=99=D7=95=D7=9D =D7=95=D7=B3, 25 =D7=91=D7=A4=D7=91=D7=A8=D7=B3 2022, = 04:18, =D7=9E=D7=90=D7=AA Zhenlei Huang =E2=80=8F<zlei.huang@gmail.com>:
Hi,
You can also track the WIP netlink = feature, https://reviews.freebsd.org/D33975

On Feb 25, 2022, at 4:05 AM, Sami Halabi <sodynet1@gmail.com> wrote:

Hi,
Added Current, maybe will be lucky ;)

Anyone have idea how approach and fix this?

Sami

=D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A = =D7=99=D7=95=D7=9D =D7=92=D7=B3, 22 =D7=91=D7=A4=D7=91=D7=A8=D7=B3 2022, = 23:30, =D7=9E=D7=90=D7=AA Sami Halabi =E2=80=8F<sodynet1@gmail.com>:
Hi all,
sorry for the cross post but I need = help and I'm not sure where it hangs.

I create linux jail (debian bullseye) = via cbsd.
the jail is being populated with the = debian userland..
so far so good... services = running (sshd) and I can login to the jail, I also can update = packages and I can install apache httpd and all works fine (apt = install or make from src).
I also manage to install = packages even if their scripts depend on "ip" command that = fails:
cbsd@j2> ip
Cannot open = netlink socket: Address family not supported by protocol

ifconfig show empty interfaces:
cbsd@j2> ifconfig
eth0: = flags=3D4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 00:50:56:0a:b3:a0 =  (Ethernet)
        RX packets = 139798314  bytes 12029597009 (11.2 GiB)
    =     RX errors 0  dropped 0  overruns 0  frame = 0
        TX packets 26879143 =  bytes 34400160833 (32.0 GiB)
      =   TX errors 0  dropped 0 overruns 0  carrier 0 =  collisions 0

lo0: = flags=3D4169<UP,LOOPBACK,RUNNING,MULTICAST>  mtu 16384
        loop  (Local Loopback)
        RX packets 28548  bytes = 160312960 (152.8 MiB)
        RX = errors 0  dropped 0  overruns 0  frame 0
        TX packets 28548  bytes = 160312960 (152.8 MiB)
        TX = errors 0  dropped 0 overruns 0  carrier 0  collisions = 0

I know linux emulation doesn't implement netlink.. so what I = do is fake the response by replacing /bin/ip by a bash script that = prints the correct IP and fakes some other (needed by packages i = Installed):
#!/bin/bash
if [ "$1" =3D "-o" ]; then
echo "1: eth0 inet = 192.168.1.2/24 brd = 192.168.1.255 scope global eth0"
elif [ "$1" =3D "route" = ]; then
        if [ "$2" =3D "get" ]; = then
              =   echo "8.8.8.8 via  192.168.1.2   dev eth0  src  192.168.1.2  "
        else
                echo = "default via  192.168.1.2   dev eth0"
    =     fi
else
echo "1: eth0: = <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen = 1000"
echo "  inet  192.168.1.2  /24 brd  192.168.1.255 scope global eth0"


still ifconfig shows no IP... its time to say it a regular = jail and *NOT* VNET.

*however* package that pull ips via libraries = fail..
eg: installed bind916 (name) in the logs I = see these errors (relevant only):
cbsd@j2> = service named start
Starting domain name service...: = namednamed: prctl(PR_SET_DUMPABLE) failed: Invalid argument
cbsd@j2>


log = file shows:
22-Feb-2022 23:11:58.705 general: = notice: BIND 9 is maintained by Internet Systems Consortium,
22-Feb-2022 23:11:58.705 general: notice: Inc. (ISC), a = non-profit 501(c)(3) public-benefit
22-Feb-2022 = 23:11:58.705 general: notice: corporation.  Support and training = for BIND 9 are
22-Feb-2022 23:11:58.705 general: notice: = available at https://www.isc.org/support
22-Feb-2022 = 23:11:58.705 general: notice: = ----------------------------------------------------
22-Feb-2022 23:11:58.705 general: info: found 6 CPUs, using 6 = worker threads
22-Feb-2022 23:11:58.705 general: info: = using 6 UDP listeners per interface
22-Feb-2022 = 23:11:58.705 general: info: using up to 21000 sockets
22-Feb-2022 23:11:58.715 general: info: loading configuration = from '/etc/bind/named.conf'
22-Feb-2022 23:11:58.715 = general: info: reading built-in trust anchors from file = '/etc/bind/bind.keys'
22-Feb-2022 23:11:58.715 general: = info: looking for GeoIP2 databases in '/usr/share/GeoIP'
22-Feb-2022 23:11:58.715 general: info: using default = UDP/IPv4 port range: [1024, 65535]
22-Feb-2022 = 23:11:58.715 general: info: using default UDP/IPv6 port range: [1024, = 65535]
22-Feb-2022 23:11:58.715 network: info: no IPv6 = interfaces found
22-Feb-2022 23:11:58.715 general: error: = ifiter_getifaddrs.c:79: unexpected error:
22-Feb-2022 = 23:11:58.715 general: error: getting interface addresses: getifaddrs: = Address family not supported by protocol
22-Feb-2022 = 23:11:58.715 network: warning: not listening on any interfaces
*snip*
*snip*
22-Feb-2022 23:11:58.735 general: = error: socket.c:2405: unexpected error:
22-Feb-2022 = 23:11:58.735 general: error: setsockopt(50, IP_RECVTOS) failed: Protocol = not available
22-Feb-2022 23:11:58.735 general: notice: = couldn't add command channel 127.0.0.1#953: permission denied
22-Feb-2022 23:11:58.735 general: = error: socket.c:2405: unexpected error:
22-Feb-2022 = 23:11:58.735 general: error: setsockopt(50, IP_RECVTOS) failed: Protocol = not available
22-Feb-2022 23:11:58.735 general: notice: = couldn't add command channel 127.0.0.1#953: permission denied
22-Feb-2022 23:11:58.735 zoneload: info: managed-keys-zone: = loaded serial 24
22-Feb-2022 23:11:58.735 zoneload: info: = zone 0.in-addr.arpa/IN: loaded serial 1
22-Feb-2022 = 23:11:58.735 general: error: socket.c:2405: unexpected error:
22-Feb-2022 23:11:58.735 general: error: setsockopt(512, = IP_RECVTOS) failed: Protocol not available
22-Feb-2022 = 23:11:58.735 general: error: socket.c:2405: unexpected error:
22-Feb-2022 23:11:58.735 general: error: setsockopt(513, = IP_RECVTOS) failed: Protocol not available
22-Feb-2022 = 23:11:58.745 zoneload: info: zone 255.in-addr.arpa/IN: loaded serial = 1
22-Feb-2022 23:11:58.745 zoneload: info: zone j1.royalshells.com/IN: = loaded serial 2022022106
22-Feb-2022 23:11:58.745 notify: = info: zone j1.royalshells.com/IN: sending notifies (serial = 2022022106)
22-Feb-2022 23:11:58.745 general: error: = socket.c:2405: unexpected error:
22-Feb-2022 23:11:58.745 = general: error: setsockopt(514, IP_RECVTOS) failed: Protocol not = available
22-Feb-2022 23:11:58.745 zoneload: info: zone = localhost/IN: loaded serial 2
22-Feb-2022 23:11:58.745 = general: error: socket.c:2405: unexpected error:
22-Feb-2022= 23:11:58.745 general: error: setsockopt(515, IP_RECVTOS) failed: = Protocol not available
22-Feb-2022 23:11:58.745 zoneload: = info: zone 127.in-addr.arpa/IN: loaded serial 1
22-Feb-2022 = 23:11:58.745 general: notice: all zones loaded
22-Feb-2022 = 23:11:58.745 general: notice: running
22-Feb-2022 = 23:11:58.795 general: error: socket.c:2405: unexpected error:
22-Feb-2022 23:11:58.795 general: error: setsockopt(50, = IP_RECVTOS) failed: Protocol not available
22-Feb-2022 23:12:58.811 general: error: = ifiter_getifaddrs.c:79: unexpected error:
22-Feb-2022 = 23:12:58.811 general: error: getting interface addresses: getifaddrs: = Address family not supported by protocol
22-Feb-2022 23:12:58.811 network: warning: not listening on = any interfaces

Any Idea how to fix this??

cbsd@j2> named -V
BIND 9.16.22-Debian (Extended Support Version) = <id:59bfaba>
running on Linux x86_64 3.2.0 FreeBSD = 12.3-RELEASE-p1 GENERIC

installing = newer versions 

I have also problems with dovecot mail package.. but will = leave it for now

Thanks in advance,
Sami



= --Apple-Mail=_783D5F85-39C4-4639-BA1C-DCB831F843AE-- From nobody Tue Mar 8 21:00:29 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id E71F81A01C11; Tue, 8 Mar 2022 21:00:47 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: from mail-qv1-xf2f.google.com (mail-qv1-xf2f.google.com [IPv6:2607:f8b0:4864:20::f2f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KCnmH2KDnz3nqT; Tue, 8 Mar 2022 21:00:47 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: by mail-qv1-xf2f.google.com with SMTP id eq14so453830qvb.3; Tue, 08 Mar 2022 13:00:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:from:date:message-id:subject:to; bh=vHrVESQpZYjZq79y8iE93xFejNfeYSxoh8AhQAukmkg=; b=CUn71zvZeWPajTqHGFMoFBHvwXMs2dwCVVu7csz8Uv0fgNTDq3Mzi4tXyNKUk2LNKy 1HIjDpPw2TDa5Sz9Hcg+60J5oXshMe7f0imltF6e0RuYcKgcTx6iHBvKR3QRTxMNeD6e oTCDcqntLCB3eClTcxmDmYhOqO2Cnc9hc2o8CO5F+Q29dOcaAJhCd2dAZquMdb2Q/0hz kmEhfVJ7v4wSQNnydYJgr7c/qPjJPuQawDq/5H3krvo1Q5xaY4pqkhnrxVYRWQAUZLlv l1zbTCj/P/xM36oon1dbx5aYlEc2UjwJp2QIUgBamC0X/uh7hkHVZYWLC7TnBSkzp2+r K5Mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=vHrVESQpZYjZq79y8iE93xFejNfeYSxoh8AhQAukmkg=; b=B3JtwHoMubwIDMeX332mXu394Q7Z6EXjvZ/AFWV9uotdNSZ2Cm7XhZrTmsmwhfWnc1 0QFt+XFep3Vjxa9gbuy+N9eMDRGNr8MOmq+pvpMXiUQTBmf4VqQqcl8P6BeXWfeaZ1Gr qZCn2CuGPOKPSE1gxJqHxk7hVCEi1Ucr7YSOXr329/CUMsUukdQ7RR7CSpmosuOIDmMp /py7Cw4TqAhjuCFpfhXksrcg4tj84elA6TqNLBP5frUHf03ChO0SdvDYhjo6zauex69J vXneQyfpL/jGp3CTpQH1Zm14i6TTNiiQGqjUiZ7DeyktV9SNCNYED5I7mYNNQv4RqWP5 QnzQ== X-Gm-Message-State: AOAM531yQ6IeKs3E0RNQjO+lz5hCz+d3FXI9Z1/NJd012kvKtJCDYu1r 8qdfep1cI3XJz+utxqzq+1cxrSRIDTEBC33BeRuoqx0LPa4= X-Google-Smtp-Source: ABdhPJxfMXt3ApE+nJMvDARBbNTmATVsr8zVHmqtGTUv/WbPILtKgy29E09KFY2uaaCb+LGrPP/+aEjGnyn+K9xRQ+E= X-Received: by 2002:a05:6214:4103:b0:42d:7ad0:44ff with SMTP id kc3-20020a056214410300b0042d7ad044ffmr13850533qvb.42.1646773240780; Tue, 08 Mar 2022 13:00:40 -0800 (PST) List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 From: Sami Halabi Date: Tue, 8 Mar 2022 23:00:29 +0200 Message-ID: Subject: running cron jobs setpriority permission denied To: freebsd-stable@freebsd.org, FreeBSD Current , freebsd-jail@freebsd.org, freebsd-net@freebsd.org, Oleg Ginzburg Content-Type: multipart/alternative; boundary="0000000000000add3605d9bb4698" X-Rspamd-Queue-Id: 4KCnmH2KDnz3nqT X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=CUn71zvZ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of sodynet1@gmail.com designates 2607:f8b0:4864:20::f2f as permitted sender) smtp.mailfrom=sodynet1@gmail.com X-Spamd-Result: default: False [-4.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCPT_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::f2f:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-stable,freebsd-current,freebsd-jail,freebsd-net]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-ThisMailContainsUnwantedMimeParts: N --0000000000000add3605d9bb4698 Content-Type: text/plain; charset="UTF-8" Hi, I have a jail ran by cbsd which has a cronjob like this: * * * * * root /usr/local/directadmin/dataskq I see every minute this error logged in /var/log/messages: cron[71002]: setpriority 'root' (daemon): Permission denied I see in ps xau that it runs but at nobody user even when loggin to the jail I have: cron[68825]: setpriority 'root' (daemon): Permission denied login[68900]: setpriority 'root' (root): Permission denied jexec[69404]: setpriority 'root' (root): Permission denied # uname -a FreeBSD j5.sody.com 12.3-RELEASE-p1 FreeBSD 12.3-RELEASE-p1 GENERIC amd64 what am I missing? Sami -- Sami Halabi Information Systems Engineer NMS Projects Expert, FreeBSD SysAdmin Expert Asterisk Expert --0000000000000add3605d9bb4698 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

I have a jail ran by cbsd which has= a cronjob=C2=A0like this:
* * * * * root /usr/local/directadmin/= dataskq

I see every minute this error logged i= n /var/log/messages:
cron[71002]: setpriority 'root' (dae= mon): Permission denied

I see in ps xau that i= t runs but at nobody user

even when loggin=C2=A0to= the jail I have:
cron[68825]: setpriority 'root' (daemon= ): Permission denied
login[68900]: setpriority 'root'= (root): Permission denied
jexec[69404]: setpriority 'roo= t' (root): Permission denied

# uname -aFreeBSD j5.sody.com 12.3-RELEASE-p1 Fre= eBSD 12.3-RELEASE-p1 GENERIC =C2=A0amd64

what = am I missing?

Sami

--
Sami Halabi
Informati= on Systems Engineer
NMS Projects Expert,=C2=A0FreeBSD SysAdmin Expert
Asterisk Expert
--0000000000000add3605d9bb4698-- From nobody Wed Mar 9 11:24:15 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 434321A150E8; Wed, 9 Mar 2022 11:24:29 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: from mail-qv1-xf2d.google.com (mail-qv1-xf2d.google.com [IPv6:2607:f8b0:4864:20::f2d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KD8wr4FVgz3Fwl; Wed, 9 Mar 2022 11:24:28 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: by mail-qv1-xf2d.google.com with SMTP id e22so1674929qvf.9; Wed, 09 Mar 2022 03:24:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=hOErKVoem/0p/g6CJngpwAoj0n1R8dEb84EvsQQKKUc=; b=J/xE09S5XGd7oQgSB3C1B2qwL4c22PGEGRiMPlizBoshKpcr/o8tzSjBORTfQ9H5pU DST4U/A1cM9k+p3pxJQWiiNPxN9aj+VnlmVGjC6FcW1hpqiWdrxZrapgWs6MzVWUzMTy ePOQ/hFc7RzK9yqJjkXGrpneYeNICKhBceTtcG6PBpH6fpfjjWh8jPct0zr/mWedYKAj ZDA6PdkgVQTu09Hx+WvKEIGSbAiRuD4Hz2zUuew8ZNi2hw0A6Y6V7V70wfqATTF9AZOM fhUxVID6HIjqlWVxUN1hbAASGDR0xzfVVZ98Im86MQqvOg80mYPNpybkVNQRlInXLmWR MvdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=hOErKVoem/0p/g6CJngpwAoj0n1R8dEb84EvsQQKKUc=; b=BnI4+MO33QddXhOORzpJJ1BVs2ZVSe+H25SsIJG6QKekKIU26Dwe4jUT1tW1sXasD2 /ENmj3ihFfY99+NaACxXlr44LUNeLXg1QsHS+ewVv0YLJ8AWdvz3rTgzLXkr2tLlmyLR dkTC6hsAHMVDMTAEtgdH65g+58ByF9G0lBWnb2qa4+qEchF7PMvQy8xkqx3hqTf7Gl1t qvi+NnLHx1rxaf/UmqTN/xB4jXtmn0VIckw+rokIdAz7EF4lyOhMi1lYnbp4v2DhC+PR KJ7T1mni7Ey9vHfYE+Hi1RDO1p0zNMkliBjYIFzJhxuHGhOEQvjyAsLZgqewaY4ZnEW5 r1mg== X-Gm-Message-State: AOAM531emqfCRi9U8sskY5+Am5xLS0zh0S67uOkYQgfek+kkbLPRCdmg 4UsdlBSaQdMr1xEYPGMQouzw+PmPelqe849aveJVwCkz7OY= X-Google-Smtp-Source: ABdhPJyodp7KXZC9fixfZt+p522di5346EIZBUk988I8VMToku2LNVRUomV9ujV9BCyAinIItbGMnHqVvW5Z64vUopY= X-Received: by 2002:a05:6214:4103:b0:42d:7ad0:44ff with SMTP id kc3-20020a056214410300b0042d7ad044ffmr15785822qvb.42.1646825062333; Wed, 09 Mar 2022 03:24:22 -0800 (PST) List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 References: <465361599.31.1646816636649@mailrelay> In-Reply-To: <465361599.31.1646816636649@mailrelay> From: Sami Halabi Date: Wed, 9 Mar 2022 13:24:15 +0200 Message-ID: Subject: Re: running cron jobs setpriority permission denied To: Ronald Klop Cc: freebsd-net@freebsd.org, freebsd-stable@freebsd.org, freebsd-jail@freebsd.org, FreeBSD Current , Oleg Ginzburg Content-Type: multipart/alternative; boundary="000000000000d9004c05d9c7565a" X-Rspamd-Queue-Id: 4KD8wr4FVgz3Fwl X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b="J/xE09S5"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of sodynet1@gmail.com designates 2607:f8b0:4864:20::f2d as permitted sender) smtp.mailfrom=sodynet1@gmail.com X-Spamd-Result: default: False [-3.29 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.29)[-0.294]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCPT_COUNT_FIVE(0.00)[6]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::f2d:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-net,freebsd-stable,freebsd-jail,freebsd-current]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-ThisMailContainsUnwantedMimeParts: N --000000000000d9004c05d9c7565a Content-Type: text/plain; charset="UTF-8" Hi, Thank You!! indeed that helped! Sami On Wed, Mar 9, 2022 at 11:03 AM Ronald Klop wrote: > It sounds similar to this issue. > > https://github.com/cbsd/cbsd/issues/437 "default nice 1 prevents cron in > jail #437" > > Does that help? > > Regards, > Ronald. > > > > *Van:* Sami Halabi > *Datum:* dinsdag, 8 maart 2022 22:00 > *Aan:* freebsd-stable@freebsd.org, FreeBSD Current < > freebsd-current@freebsd.org>, freebsd-jail@freebsd.org, > freebsd-net@freebsd.org, Oleg Ginzburg > *Onderwerp:* running cron jobs setpriority permission denied > > Hi, > > I have a jail ran by cbsd which has a cronjob like this: > * * * * * root /usr/local/directadmin/dataskq > > I see every minute this error logged in /var/log/messages: > cron[71002]: setpriority 'root' (daemon): Permission denied > > I see in ps xau that it runs but at nobody user > > even when loggin to the jail I have: > cron[68825]: setpriority 'root' (daemon): Permission denied > login[68900]: setpriority 'root' (root): Permission denied > jexec[69404]: setpriority 'root' (root): Permission denied > > # uname -a > FreeBSD j5.sody.com 12.3-RELEASE-p1 FreeBSD 12.3-RELEASE-p1 GENERIC amd64 > > what am I missing? > > Sami > > -- > Sami Halabi > Information Systems Engineer > NMS Projects Expert, FreeBSD SysAdmin Expert > Asterisk Expert > > -- Sami Halabi Information Systems Engineer NMS Projects Expert, FreeBSD SysAdmin Expert Asterisk Expert --000000000000d9004c05d9c7565a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,
Thank You!! indeed that helped!

Sami

On Wed, Mar 9, 2022 at 11:03 AM Ronald Klop <ronald-lists@klop.ws> wrote:
=
It sounds similar to= this issue.

https= ://github.com/cbsd/cbsd/issues/437 "default nice 1 prevents cron i= n jail #437"

Does that help?

Regards,
Ronald.

=C2=A0

Van: Sami Halabi <sodynet1@gmail.com>
Datum: dinsdag, 8 maart 2022 22:00
Aan: freebsd-stable@freebsd.org, FreeBSD Current <freebsd-current@freeb= sd.org>, freebsd-jail@freebsd.org, freebsd-net@freebsd.org, Oleg Ginzburg <olevole@olevole.ru>=
Onderwerp: running cron jobs setpriority permission denied=

Hi,
=C2=A0
I have a jail ran by cbsd which has a cronjob=C2=A0like this:
* * * * * root /usr/local/directadmin/dataskq
=C2=A0
I see every minute this error logged in /var/log/messages:
cron[71002]: setpriority 'root' (daemon): Permission denied
=C2=A0
I see in ps xau that it runs but at nobody user
=C2=A0
even when loggin=C2=A0to the jail I have:
cron[68825]: setpriority 'root' (daemon): Permission denied
login[68900]: setpriority 'root' (root): Permission denied
jexec[69404]: setpriority 'root' (root): Permission denied
=C2=A0
# uname -a
FreeBSD j5.sody.com 12= .3-RELEASE-p1 FreeBSD 12.3-RELEASE-p1 GENERIC =C2=A0amd64
=C2=A0
what am I missing?
=C2=A0
Sami
=C2=A0
--
Sami Halabi
Information Systems Engineer
NMS Projects Expert,=C2=A0FreeBSD Sys= Admin Expert
Asterisk Expert


-- <= br>
Sami Halabi
Information Systems Engineer
NMS Projec= ts Expert,=C2=A0FreeBSD SysAdmin Expert
Asterisk Expert
--000000000000d9004c05d9c7565a-- From nobody Mon Mar 14 07:05:32 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id E1CB01A0B1E2 for ; Mon, 14 Mar 2022 07:05:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KH6xm1WYKz3jwm for ; Mon, 14 Mar 2022 07:05:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 152335DC for ; Mon, 14 Mar 2022 07:05:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 22E75Vl2056618 for ; Mon, 14 Mar 2022 07:05:31 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 22E75VpZ056616 for jail@FreeBSD.org; Mon, 14 Mar 2022 07:05:31 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 228351] cannot unhide log or tty with devfs.rules Date: Mon, 14 Mar 2022 07:05:32 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: conf X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: zlei.huang@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1647241532; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KHfcDU9y2sfMRVSNEJiYPvyRHtHZUYkXAdEH3hCH9bM=; b=wlRdOKhX4p0ynR/UixUhzmNdrrGdzqouROMS0fmxbB3zeAHp8kZ2ECrvB4xY1m2qqbroCQ 3yFk/gUbJV7NXnXM/LBCSq7vDdXUeyyHCUB5lX63rco3cEiETGAwj1ymfCG7Dj9eAH5LX7 1SpkAQ7jKA5wEpaprA6NylXjcfnRWTNbJWz3ysRGYq2zCk8ye50dWJqnxNtz5GME+VogQb RReSp8LbmNxLAN2fWr4H6uRmav3oGT7k8+LOsj6MlDxjMec6yZ38XXWf4JguMxxZWQeeJN k+H4jg/xILfkUNb/rOX+b/gRPGVSrAzuXTpNGNZv1w+CBS7ifbjb6M5ZID3bKA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1647241532; a=rsa-sha256; cv=none; b=QWQXsK2nDGCtoJCQwz1Gi2ZwH8HFlncKfGBOV1/ORSDtWMizjD61N+1MvrliiWVybZjq1/ 8O4CIfD3WEDJ5FXwu8PODiCypS1aQL1hAyxS2fA9nrqBGCxvQ7UXaMf2pLtsrmfTLGz1WX qM26d+F7z7nXL9d+jzxfiZHVEZfN1Dr+vMozDF374/bmARARtETcc1B8uWmOfyPp1V4swc tKf7XvMyufRhftZVOtWG45YFK9jy1uH3qPW4U8kA0T+aJVUulbrexyXxpCJGsQROAbIY2u r6G4Y6hr1Lpc+tVH9SSUDcWVIl+Jlt/iiDBb9UTgXk3V4MgxkY4TjUfAF8UmfQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D228351 --- Comment #2 from Zhenlei Huang --- (In reply to Jason Mader from comment #0) The `/dev/log` is just a symbolic link to `/var/run/log` created by /etc/rc.d/syslogd . It is not created by devfs(5). For jails, the symbolic link is not created due to PR https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D179828 . Do you have any cases requiring `/dev/log` then ? See also: https://cgit.freebsd.org/src/commit?id=3D3a361e0c330fe9d5ab506cc02ee3bcc3d2= d474ce --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Mon Mar 14 13:56:52 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 00A951A1C2F3 for ; Mon, 14 Mar 2022 13:56:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KHJ4N5RwTz3Hjt for ; Mon, 14 Mar 2022 13:56:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 9C3206267 for ; Mon, 14 Mar 2022 13:56:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 22EDuqA1076932 for ; Mon, 14 Mar 2022 13:56:52 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 22EDuqXX076931 for jail@FreeBSD.org; Mon, 14 Mar 2022 13:56:52 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 228351] cannot unhide log or tty with devfs.rules Date: Mon, 14 Mar 2022 13:56:52 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: conf X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: jasonmader@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1647266212; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Df7coz83cmdC7XRBZ990GLwrzD1yjwxgNeaYv0y2NSo=; b=n/eSXlc6MynPCZyEXXs9VlCJpyDDJ2N3XlgwjNXsDiKWZa38px1pYGNA4BvquZYHudt9zH UEziPYYn6qcj1DaDTK67GOITbqVRIO+VLezuBMp5CdOdC3RiBjtMMdIzM3Ihbdo/PQwL8O EHbJ+ITinhojj9Pss7zodHhiBfK7mWg3f4HLxNq2Vj0P6WWkIMMylY8XH6ul3PwPlHAuNm O3ZYgmDYKB8wverlrDi/yAKCd+XJrDSv9iGUeyG3Fu6azngLEI8zyRiwbnqn5LtynOioQG Rs8pBh4njGsIx/KBfKtvixghBnSwjYUFrkQPgdCTxTx4PLl5ioVCDVW2Pd8fWA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1647266212; a=rsa-sha256; cv=none; b=ThQkdR0k2fAZIwexNeao2Ac4vO83lAgeCwGPVWFg41XVsULI+EPanXLHTZ2EBhqUXasFUe SVt/7BoKtbNOMCOFEuvTyX47ZRI3cXnUxWQXBsgTzJa1RR90czHIrivAJ9oot6CZvM0JD6 1sf3iAK8wEFC+yfv1Sinu5hhtNsP8DVTYil+3etwylRvh0GnIL4V86xxhDf5nhrgfcmHef ms1zIUAjbHC4VmYPJe9uyP2tf6W6moQoJHd5RcmiZcwBOsgd1UtSD4T8CCSr/XhvG+evnB pUF5Rg7W9iNw5dqlIdKkEbeysMSs5Ohm+Wnz9CxrMRJlwimTCU6cUVytgiE1vQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D228351 --- Comment #3 from Jason Mader --- (In reply to Zhenlei Huang from comment #2) I probably don't. I guess it was just unexpected since, /etc/defaults/devfs.rules has: [devfsrules_unhide_basic=3D2] add path log unhide --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Mar 15 09:09:54 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 112C11A12606 for ; Tue, 15 Mar 2022 09:09:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KHnfp4cgNz4jQL for ; Tue, 15 Mar 2022 09:09:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 7AC4F1E7EA for ; Tue, 15 Mar 2022 09:09:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 22F99s99002927 for ; Tue, 15 Mar 2022 09:09:54 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 22F99sCZ002926 for jail@FreeBSD.org; Tue, 15 Mar 2022 09:09:54 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 228351] cannot unhide log or tty with devfs.rules Date: Tue, 15 Mar 2022 09:09:54 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: conf X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: zlei.huang@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1647335394; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aDPrbiUq0ullgJzqkv0hhW6JhLAwLvfVw24Hnd0TLug=; b=MyLkJPa/xnfuHPaTn1VFz0uxGoU4bZl/M5+tq2KVhXR3jvmKDbO7UOsHKQ1xdxihrqmbDO R6/xDg4y/JEltmZ3Y1+J0LRoq90fzpzLrNKsA0MMjzsWrMX5lANCux2UI9rI5UNEiWPEqu j9QJ8Qd9cUNkMUnIWytM/eZu/BC529xSW/X5URYoUDaa54sOxDZSqnKohFlIUMpFVHZlHX X03r79wqBhchV5qsMDLUQEE72b7Ix/sAZYbZLD3NdCsl+jSvWMneRq8kaSul/w1lDkiGl5 oCzamOU4JpsSG1cqW1a+P3DLidXLimqc3IydNdxuU1O7fS7AuEJZNv8ZVbysVQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1647335394; a=rsa-sha256; cv=none; b=audpRBc//c1mz3gyVYpO9HkaEi6U+umrxryXnhMTGaLrXSWPGtMfLjYywrwfnlYlsIS67T gNq6pnn6Ax+HQVjBRl3SodegCImGnQqII+QG9o0fWNCr879cdLZPL73z4xyvWem9+8JAby tCINshtgKlYBNaPP/tINIyCPP+BAUSDkAtKV5FZz2Wzu2EuB8cCLnna7FG2QoghdxSwO0Z gaiTeYKAcnfT6sP8NDDbiUQ7tkEdbMvp65gNyN343kwd6epTWxbLA1feF4Tyo8Cidgr/sS 86d2PQqOCPgV2IU1SzTV9X6vsaAF8qSgVD2jsTptypKvmTG9ivRTIO9g0rWfaA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D228351 --- Comment #4 from Zhenlei Huang --- (In reply to Zhenlei Huang from comment #2) > For jails, the symbolic link is not created due to PR https://bugs.freebs= d.org/bugzilla/show_bug.cgi?id=3D179828 . Sorry it is misleading. The unhide for log is introduce by https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D160711 , but after https://cgit.freebsd.org/src/commit/etc/rc.d/jail?id=3D84b354cb9ab61224713c= 159b1484e8f070fd37be the logic of creating /dev/log is removed, as consequence there's no /dev/l= og anymore for created jails. The symptoms is devfs can not unhide log but that is not true. You can repe= at on FreeBSD 13.0 as following: # mkdir -p /tmp/dev # mount -t devfs devfs /tmp/dev # ln -sf /var/run/log /tmp/dev/log # devfs -m /tmp/dev rule -s 4 applyset # ls -ll /tmp/dev (In reply to Jason Mader from comment #3) > I probably don't. I guess it was just unexpected since, /etc/defaults/dev= fs.rules has: The /dev/log is transitional symlink for old binaries, it is about ten years since it was there. We can bravely remove it as it seems nobody complains a= bout it, at least for jail users. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Mar 18 18:39:26 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id C4E871A16DA0 for ; Fri, 18 Mar 2022 18:39:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KKt8Z4YLTz4vd5 for ; Fri, 18 Mar 2022 18:39:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 798B123E53 for ; Fri, 18 Mar 2022 18:39:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 22IIdQlF070762 for ; Fri, 18 Mar 2022 18:39:26 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 22IIdQIN070761 for jail@FreeBSD.org; Fri, 18 Mar 2022 18:39:26 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 258364] [jail]Two issues that can easily exhaust the host kernel's numvnodes or dp_dirty_total in jail, and may cause DoS attack Date: Fri, 18 Mar 2022 18:39:26 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: emaste@freebsd.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc bug_group product version component Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1647628766; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kS3qWGI7buP2BPu1ta/caHjGQT4yROFeRcyzqAa4/Gk=; b=Eh6IXkM5poOOr/6ggkZeU1BbBbGIHK8COvxek8FwiLme8eZdpnShQ3vQKIebIYPVyyNkGD 56S44SoyYSowwAIJVg6seQgyVOYZ6HsEhOSn2N3a+SKuQbJfDbXfl0tltsYxyDb55RBrC4 dFT9ukd0T/ZEwQWdePJ8/AkTOaXRSMa0nczwYGS4WbiESNL37Yf6V95IYVQeHc7mdjLbpU NtMQRAB9TPNDhTKqWyFvfKwfY7nHHlSauIYZtU9GPAUUiadOCRr1Kk/KrnFGSVSrDj1i7D tRU7i0uO1QIi8eFzq+6jlHwgOKrHifcHUvATqXuYf/qjbK+PJ0PgaTTjTCODyQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1647628766; a=rsa-sha256; cv=none; b=wqDjmpMe68VMl1fYie2BfQJA8fpfbbeC11DBakidlyTlkcDX0BcUwyyH6ziYGRA3605Uu5 cncvkKZJ3fzjQRI12lbpFKCgKKwyTDPr88VLtA5Bgof8MWozkT7odRf33uOrTQV9NOdmHD wHNsjL7dd03Vja1Q+eqN0RnS46ChXLUrP1NjgWHUsAe4Iyh9Ph40sciUVy29ptDe4S8V/H NdamjkaViNN4bbX7cSY5fSg4X5AnKwJQH3YJhR4WG6v966CtMCAcL+9k0G0JOQnbrg+zNs I8btBNxAMtQENub80j7yE6GrmR3+MjMJ5X1vlK3OAQth6Y5puBdbIG0ddhVWxA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D258364 Ed Maste changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |emaste@freebsd.org Group|freebsd_committer | Product|Security |Base System Version|unspecified |CURRENT Component|Base |kern --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Mar 24 16:54:39 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 42BD51A2B673 for ; Thu, 24 Mar 2022 16:54:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KPWXv0Fhrz4s8C for ; Thu, 24 Mar 2022 16:54:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id DD29D1BBAD for ; Thu, 24 Mar 2022 16:54:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 22OGscGu023681 for ; Thu, 24 Mar 2022 16:54:38 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 22OGsc62023680 for jail@FreeBSD.org; Thu, 24 Mar 2022 16:54:38 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 262471] jail utility crashes when supplied -m and vnet parameter Date: Thu, 24 Mar 2022 16:54:39 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 13.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: markj@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc assigned_to bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1648140879; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=onycK77BGO9H0n/4yirnpHGpUVqIU/wl9qof5Esu4KI=; b=cqP0xALE2gtuR5p/Eo6Y3678PEa9v2Cb6JMK/YpAZPe1kZAxokrOeSwy2KPAqLbEyI3oxc gFchzGIT1d3m8lp2XstUboZW91qaKoNKx+ptTqq94/bm3y2nkPfSCYsozNE2/Arb8mtWA2 3qNrNYBvho3+l87/PkHHX+VUtDAMo1Vg3j0EA1BP8gFweC/ZOaAzz06KSg0UKbv7+nKjLx HdOdKBfTOaN6CEMD3y1MAuU+pt9dDaa9yA/gEwn49dQ49sM3cNHxocR9VX0mgc2uHP1yCF aPk+pnuzQ9EkLmE5dtPNcITdfZ/v4k64lDURw0jutpcAD9GjJAlz49r6UBqCcQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1648140879; a=rsa-sha256; cv=none; b=cpS41uTLHiNyBwWztPpkjmTUpzMmhiD/kdDizzGprydgXYw5d2DZiuzKiG52p+TIz1KZuO 7MezPE/posAhQmvIdh/V/VrYOTSyha1ul9qbEWztqbMKJGfphOlO3NvJyYFK1HdMrCgeEf K0d9VpqXIarzCIS80cL5H1RBiVPXWGyDse0r2VUuALjxxXDdlooLDkxSSVHTQWC8oQavTM 8NkROrp41vNcDmxxvCr6/OKfPFH4+8dcrTGPYUeEKiVTwJTLih15yy+HNg9eL74Muo42i3 no0ibwZMnQYTzm8q4SsQmeWRd03nvdoaMwoLgUEIhxld/eRNQSPhL0s5pZFMnw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D262471 Mark Johnston changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jamie@FreeBSD.org, | |markj@FreeBSD.org Assignee|bugs@FreeBSD.org |jail@FreeBSD.org Status|New |Open --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Mar 24 17:53:32 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 589131A3CB66 for ; Thu, 24 Mar 2022 17:53:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KPXrs0PR1z3PMs for ; Thu, 24 Mar 2022 17:53:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id E2BCE1C847 for ; Thu, 24 Mar 2022 17:53:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 22OHrWmX054877 for ; Thu, 24 Mar 2022 17:53:32 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 22OHrW7A054876 for jail@FreeBSD.org; Thu, 24 Mar 2022 17:53:32 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 262471] jail utility crashes when supplied -m and vnet parameter Date: Thu, 24 Mar 2022 17:53:32 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 13.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: jamie@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jamie@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1648144413; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/mXOagxpv+s1VFGX+tc29PV+WCorQlVIR0JgjiYyikY=; b=kh47pmmUB1v37sww6J0idEvaSdvRkYQHC9sKL454MuYCSE0n15COHZv2z/nBbMQMQQsN1E kgUTeSj3r+tYtI9oZOWYsE5pbcuWfBLWhVS5P4R/TNpIfjoCQq9Q2r4VhXQTl/cnF6LH0B XzHlCdpb86pcCvREfylXHJiiJiJSOtZGBvbC4RoCmNqPijfqYmw+F+WnQRzqdeEXM0uA0j cpmxhJwh03H5Rsfg1WYBOeGLGXm+r04GVbYAAqmehJn8WXSd9TTv4LbPw6346kLgBTv1qF l0+AZ10zMcFnfO+Pw1PUd482zOdvJxf1jiWCv7QG7P7/OOppyZoHNUK8WH8sMw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1648144413; a=rsa-sha256; cv=none; b=Q787h29FXX1foBtRGnOXcKNhpD5i9HoE5vs1vBJwK/hq4ccohj8foaNGYAFVyj4Hbd9zUy Pk6+T82EMYFdHWv0nPz49l9zi5meRD1C+NTX1+DqRLbRQawIifmTTD098XMU1RkV/oW4HP 1vkLm+dcU65JA2gSM9BcWJQCTphmpmllGEl4FbJD9mWDKVCnd/CAJ9q7vuWZmHPnVhVwfP 0VzHodWJYAkCTbzTKBP814uhtrubh0X9O/kc/stnCpSqT9RxNO0N652AVp6Q5VELrLXMkP aZG7bSqglqURPaemSkYkUB8wWCW1cLXVmFpEv+eoYTeLThvbeE8TUlDIs/zJUA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D262471 Jamie Gritton changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|jail@FreeBSD.org |jamie@FreeBSD.org --- Comment #1 from Jamie Gritton --- I see the problem. In the check for non-changeable parameters, it's trying= to dereference the value of vnet. There's code to catch boolean parameters wi= th implicit values, but it misses vnet which isn't a true boolean. Working on making a hideous-looking if statement even worse... --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Mar 29 18:52:35 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id C639D1A43E0A for ; Tue, 29 Mar 2022 18:52:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KSdwj4Z29z4wSf for ; Tue, 29 Mar 2022 18:52:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 7ABD924196 for ; Tue, 29 Mar 2022 18:52:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 22TIqbBD083332 for ; Tue, 29 Mar 2022 18:52:37 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 22TIqbQC083331 for jail@FreeBSD.org; Tue, 29 Mar 2022 18:52:37 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 233310] jails: Modularize configuration system (conf.d) Date: Tue, 29 Mar 2022 18:52:35 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: conf X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: feature, needs-patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: debdrup@freebsd.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: mfc-stable13? mfc-stable12? X-Bugzilla-Changed-Fields: resolution cc bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1648579957; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jX1b4ARjX1h2Lv4rvvwy0FGD9wIVYpIYfbH63LZxyd0=; b=q78sZo9nYbwa1/SP+Q/1xf2mPUf4t2Dj90T/5oji+0zSEXDhpU6UibckvONTpZviCpdaPe PGf9qkfkCGtUnCENiBFPKMTXtKxHU4bIzanLR/NZBvmCYP/vVFMMed84CBpXe4oAANlkft ylTyX9i5aWtQsk9su2JE5u84qTBRlPGYJzhPlqnipseouJ8mb4uA9awnsBt1T4go++YHRW KGaEx42otPLbRfaAjJ1wNYNUV5jlbeJqIzmhBi6HQZGpXs0RJhE898vbikSSQrZzMTodKm 1Y1M/At6vERsV2qcjF9xx2E+605iyXpqFzqNRrkMw5iPRnzKNdpjEU06YaeeKA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1648579957; a=rsa-sha256; cv=none; b=b6bRQPhZ4llOPEZPW1lgZqh5HNksYxQHYTC5rTNEPN+erDTdbasOe9TKuvWXsImwtDPBWP OCzCvtUNs8BuU01Zpa44Z94Cl0HJFyVxODt55+BD6Py9DBZc8WaBIJj3tBJdG0UhMtsGJm Ywqz/rR4Kl5JULVdfY8iS2MB9sDngyZJ3rPH2FvcppBcJEMXnnxWgzDKyHFz3T2RX6xC/m YeSTFMqLcExGHLZl2poplhsJQQSogF95mx8TxMgbAQg8Z7+GVWugm2MIDZeQ7NvZUZURl7 onPGMx0ue0fYSxgrsJ6WJFpvsVqrvxMXWe3Gy2R4lZJAXZksXh7XXZgAE5cQAA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D233310 Daniel Ebdrup Jensen changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED CC| |debdrup@freebsd.org Status|Open |Closed --- Comment #13 from Daniel Ebdrup Jensen --- This was landed as commit 7955efd574b9 [1] back in September, 2021 - so I t= hink it's safe to close this bug report now. :) 1: https://cgit.freebsd.org/src/commit/?id=3D7955efd574b9 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Mar 30 12:38:48 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 5AAF21A3B9E0 for ; Wed, 30 Mar 2022 12:38:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KT5Zv6lYrz4V38 for ; Wed, 30 Mar 2022 12:38:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id C47D9130CA for ; Wed, 30 Mar 2022 12:38:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 22UCclOi051242 for ; Wed, 30 Mar 2022 12:38:47 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 22UCclNg051241 for jail@FreeBSD.org; Wed, 30 Mar 2022 12:38:47 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 240106] VNET issue with ARP and routing sockets in jails Date: Wed, 30 Mar 2022 12:38:48 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: bugs.freebsd.org@gabor.adorjani.net X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1648643928; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ycVU4A8h6GcDxGhAN02SbA+Dm7AxGJ+jXxe1iaXRhD8=; b=bPzmAl8xpNsQKWFuT2DhBRdFL7b1yPegUY0a4bOoHBWOSAaJuQwqOvcbg4OMvv5+b4DWC3 Gng9q/ORq+2reYM8CoO0uSRz52jvRyxX1VidZ8KOzkNgluLIYbAh20vrDpDq74WOStzF6b jvEt/NogZe7emPn7YkOuSRW3EXuf9Jj8SDzUKFdfpsC/n2wvnmyK1eG+njf9HQkMJe/yI1 kXL0KjGJnaMPmZN8MGPQJHRTZh7pWEOM+lxb1bXY33Zv+Acg3ZxPmJJ2Lo2jk45UGvXXMh qe2s9aQ6Jnhw6o2EHKEepjCPpfBHSVrALDMm9WCJaelcawV52BCIXMaVMoeXhA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1648643928; a=rsa-sha256; cv=none; b=akvR6Xmosts/IKr4YlnMhwEqRkCypXkX04n58D35oJIWgnpjScFgJ2Inf3EqSYIyE8C5Zn oRjnhu1j2dy3jWBHAy2H5wiJ+9mKTwApRa3m4jyuL29M8fLMTlTdmw9/bjfnx6hFIPVMdq 0YUMWePqli2ax3QDEpISvZAHHFmH2R3g/FfEH3QCr9VSU5K2MJ9Cxu3ziBuMwxQqexzlU3 Sc7HXZtEtQnBRY/iqYvJxO1R5+DIRMu6TJtPYBgq1glE+ikADR13PAvVSxbLGY3IfHAOVv I2sUxn5FGE9Vrmi6W9OfaQaXJMCzu5ggtpx24b+eSbWOFp1Y1+QVEQg6GjTl4Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D240106 Gabor ADORJANI changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bugs.freebsd.org@gabor.ador | |jani.net --- Comment #11 from Gabor ADORJANI -= -- I believe I ran into the same issue today on 13.1-BETA3. Setup: I use a NUC for virtualisation host with a single NIC: em0. It has v= Pro (poor man's service processor), which shares the NIC with the OS and communicates on the native VLAN (VLAN1). Because of this I put the OS to a tagged one. I set up several tagged VLANs: 2, 4, 6, 8. The host OS uses em0.2 on VLAN2. I set up a bridge for each VLAN interface, as well as for the physical: em0 -> vm-sw1 em0.2 -> vm-sw2 em0.4 -> vm-sw4 em0.6 -> vm-sw6 em0.8 -> vm-sw8 Then I created a jail with Bastille, assigning it to VLAN2/vm-sw2 using VNE= T, with an IP from the subnet also used on the host. I could ping the host from the jail and vice versa, but could not reach the external world from the jail, nor could ping the jail from the router in the same subnet. After 'ifconfig vm-sw1 destroy' it suddenly started working and the jail now has full IP4/6 connectivity. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Mar 30 15:17:50 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id EAE821A4BCBB for ; Wed, 30 Mar 2022 15:17:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KT96S5cTtz3NVY for ; Wed, 30 Mar 2022 15:17:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id A150115A59 for ; Wed, 30 Mar 2022 15:17:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 22UFHqFE030331 for ; Wed, 30 Mar 2022 15:17:52 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 22UFHq0S030330 for jail@FreeBSD.org; Wed, 30 Mar 2022 15:17:52 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 240106] VNET issue with ARP and routing sockets in jails Date: Wed, 30 Mar 2022 15:17:50 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: bz@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1648653472; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/FrtnEHeK8ClUmg7GvKYN/HYFnciSoa9FmM4RobDKUg=; b=iJM0QbEa1LeflGsIp35Wq3HnRJbUXajSDQXnHzOvG2+xchBJl6cHwyDfe3DFNszdBQvylj d1X7nMQ+WC4LuZpVIgCIUm5Y1FSNGnJMmGzl0kNhBuzMS0rP7tSwcQ+z3BfUZajGsm6KuF MuXUWfL31mU2vro0Jnnt4k8XHvDMW4enIYqdgrRiRACfq6QpckOnwCNR21RakJwbuiN+bU ZQvqbeY2IrdhXIcBdeps8gMnCycCUvsAAz8mIR7hsqk5RzUG3vodt5V2DUOzi40UmpTFrv +OFzETr/6AiTKEwwUzjkkq7thPFhQQreICu2jMYf2xE24Bp1fgIvk4V2r/VLyQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1648653472; a=rsa-sha256; cv=none; b=QtUn9BOIyAzvribGTfUYsLvDArAbGVKqbBk97LiOFDqIYSTIYFONgLV0CL8UTKtUQ8OOTB /rZnWICzAyfpuor2Q/GwgVAbNmEuTeu2PoOhxindgAnnfLf1yA8saNCG3tnRv6u9vZUIhP rtfqf6vHfKaoXEhGmrta7lh0fgNUAXziiIX8EIoLRdqXq+G5XBozRtBgHvrqpKRZxmG2TK IkQSbUxULBu4Q8hgvWFBdPryOwH4ilsMjFc5od1378YYd+t+k00Xz48lXqtFRBuZCUmj/r yJVCiQ4uTrX9GR/21XbAtbqq1QHGoqWhsfoV510miYUc8TUBk3LFlg5DTIC05A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D240106 Bjoern A. Zeeb changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bz@FreeBSD.org, | |kp@freebsd.org --- Comment #12 from Bjoern A. Zeeb --- I beleive I ran into similar problems with main last year and I am still se= eing occasional "blackouts"; I believe back then I could trigger traffic by send= ing packets from one specific part of jail / host / remote to another and that would hold until the entry expired but I have no more notes on this. I am also adding kp@ given bridge ... --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Mar 30 15:32:39 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 941C21A4F8A3 for ; Wed, 30 Mar 2022 15:32:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KT9RX31tHz3Qh2 for ; Wed, 30 Mar 2022 15:32:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 435B715C55 for ; Wed, 30 Mar 2022 15:32:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 22UFWeBs040074 for ; Wed, 30 Mar 2022 15:32:40 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 22UFWeO9040073 for jail@FreeBSD.org; Wed, 30 Mar 2022 15:32:40 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 240106] VNET issue with ARP and routing sockets in jails Date: Wed, 30 Mar 2022 15:32:39 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: kp@freebsd.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1648654360; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NAMIW16rDz0w3cnlAs+QHYX44MfQG8wwageaveE/3MY=; b=XNrj81sytTWC0fZlaoKTzWXa4yhSurSfBflejEF4MOPfM84DWlV7vAgPvIvdhqExXUC7m8 8aJP9aQHm9mr6pAGuqlNhE6G1TDF1ZiL5ihFNYXl3UR96oKLwzFTOsk71yjQAaSAQ2zv2U JmAPqGY6bB/fxVl+mpUD+VuY4uyNOrxUWbyAeVqyxnsaWzektxV8RxknLgIOyViXjZlCeM xBvyMzEhyhdVmjUmEdjeMNhw4ZWiQH4q+HM1Hc73eBWZBdgueaAotAzq9optRwhc5h3gjV ru52CwcNWuUDbkZ8pLMK1oD4JAKdspo/2Jj8Tu2svWZa+9GEuRHwZeQQHaEYmw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1648654360; a=rsa-sha256; cv=none; b=RXEzbrIq/UiJeLNbbiQ6T+o8ttlMtGD+1OEYdawbRm+JGHain8z7IAiN45918wZGDqZ+ap refOxjTfjYk9KjyywKolujScQZ3aH5M4+BkWKUkFxI5yDt9SvPX6ET5RXEh1c96tTdorYm Fqp7+bfQuxjIIgwanxrRzfMV5zmaxp1wE4wwpYlQq8QUESGmYHSzj7VXTKtxP9PnUdrfbn /+Qc3GpzXbCqKhV2S7ZlI1SGR3JfiPypVuutq7I819nRZfpXuQNQsD7u8J/oFVEqdvYggw 9tzk+qV0WoJaNsIS4WXTM7jvqj+8kCk3zNLsMmiaNqt6DIUBOdvbyXCNZ3bsPA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D240106 --- Comment #13 from Kristof Provost --- (In reply to Bjoern A. Zeeb from comment #12) Note that the issue described in #10 is a configuration problem more than a bug. In this configuration the bridge will grab all packets, including those wit= h a vlan tag and nothing will be passed to the vlan interfaces. That's expected. After all, the system has been configured to bridge all packets arriving on= em0 to the members of vm-sw1, and that includes those with ETHERTYPE_VLAN. This patch should make it do what the user wants, but I'm not convinced tha= t's actually appropriate: diff --git a/sys/net/if_bridge.c b/sys/net/if_bridge.c index 12c807fe2009..98c79764bc69 100644 --- a/sys/net/if_bridge.c +++ b/sys/net/if_bridge.c @@ -2467,6 +2467,11 @@ bridge_input(struct ifnet *ifp, struct mbuf *m) eh =3D mtod(m, struct ether_header *); + if (ntohs(eh->ether_type) =3D=3D ETHERTYPE_VLAN || + ntohs(eh->ether_type) =3D=3D ETHERTYPE_QINQ) { + return (m); + } + bridge_span(sc, m); if (m->m_flags & (M_BCAST|M_MCAST)) { --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Mar 30 16:07:37 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 2AB9E1A32A5E for ; Wed, 30 Mar 2022 16:07:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KTBCs6h8Pz3rcv for ; Wed, 30 Mar 2022 16:07:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id BE97C165AE for ; Wed, 30 Mar 2022 16:07:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 22UG7buZ056889 for ; Wed, 30 Mar 2022 16:07:37 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 22UG7b4q056888 for jail@FreeBSD.org; Wed, 30 Mar 2022 16:07:37 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 240106] VNET issue with ARP and routing sockets in jails Date: Wed, 30 Mar 2022 16:07:37 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: bugs.freebsd.org@gabor.adorjani.net X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1648656458; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NNvu8zR+Yz+CwRJadAPeEvNzM9uf4gtm0BPoV4nURVc=; b=YhKkHDVFdQ8kbqcY3ZUBWXcJijm5Ge8B5GkAdqaxCg9a9dw3miuYH8xii5ac+X4qpdx5zC pcIFSAwQZTE5rBb/1z28izhdZY1L25soFarGGW1fysqE7lo3d33Ltv7G5HdDjPjrDmlibR wQTkNcLThkXimCCUrPzkGCLAB0bs4habHtSRXJe33KqW8E+HDUPeMmOVrzWvaQTjiqj42v zI74QiI37eanYGW4g1q/EBXzwCk09Cz1zMZlL1yHGmHcKo/FimsBNSQbubSqWfa/DJuPuM /YcvgnzT6tHXScz4udcU+BJkTUDKXbC/g/WRZXHNym0B4iWdrBweDxfFBQVggg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1648656458; a=rsa-sha256; cv=none; b=PyTB+Q5cp6S+dliuS0oshQoJnj+Jf0ngp0QHbVaXOXzTSpukFNpeqaRUGTUN3eDAnasndB pLqiJO0imorKqyapdYq9L49XuePBlO8QPy3NXGponI1/fI4pQg02s43fsZnbfS7zGrxuJI 9M087sREvjDKUmAGDYtS69m8rgbWRC+OX/HDYZcyVmyJPKexjn9QUO/vW/7lSg1aTrGnj8 v3UjtW7odm83+WL3yxP78L7TTm23ZOnlABmMnUgyMSwP26v8TFtMhzuw9MRdA1ReRFQE+b sp4pcBQtPp3AoZk3dQP1oH0iqs5Cb8ancDGUq7Vrqugjah8JOKN+jJCJ9Vc4gQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D240106 --- Comment #14 from Gabor ADORJANI -= -- (In reply to Kristof Provost from comment #13) Thanks Kristof, you are right, I didn't see the forest for the trees. It's = not a bug, but a feature. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Mar 30 17:22:42 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id C3AE71A4AA48 for ; Wed, 30 Mar 2022 17:22:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KTCtW4RQYz4jL9 for ; Wed, 30 Mar 2022 17:22:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 78B3717906 for ; Wed, 30 Mar 2022 17:22:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 22UHMhnq098695 for ; Wed, 30 Mar 2022 17:22:43 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 22UHMh3F098694 for jail@FreeBSD.org; Wed, 30 Mar 2022 17:22:43 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 240106] VNET issue with ARP and routing sockets in jails Date: Wed, 30 Mar 2022 17:22:42 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: crest@rlwinm.de X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1648660963; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=obesNZ1ZmYhpL7eRL0bpOK8E4WBy3Pd8lWblQ1IOfho=; b=jLrbuYW1CiJ9c96HISd8mA0luoODEAWJUKfz75XjyLtaa5lAb2i9m9/+MItAh/PL0nGxVr n0WodY7P3xbjQ2/tBNmI6FkQOy8H/jJ0bnGZ9qluRg7SbVHO8uYRh50TyjsZAp/YxoKSEg cz2EzNnzaKdgXdoKVmZgp3atbEBYy2LEHyNkSj5eOSpggL9wpzQE+IYppo0W/JZJhon5GI M3xnFczLYQogV2HuziWRsEXE40pg+1v7lzdZfs9XphwYDz/W7OKCmdhhXqfbI+Zg6+0GAv VWh03uRlLY3/ink+yEqYDGG/F9521z4u1f+fjTrqkOfm/FJzRcjmG0Bg2xK+dw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1648660963; a=rsa-sha256; cv=none; b=ApqKzRyz16wgmA/nS5nvWOqXF9cqZvYnXwRVrvVRkmtfdQt3plqxaIqFqjeWvXUKvax7dD Z+kA2mpJP/UJdf65Xq+vb7s124AI1/XNxwSILLSq3PxiUlEKRxtS4wbabXTjbIHE+LVdcg FWT+G0m80X6puhOVi0E+YGO6Wi7P2/eLTZeQCXHXzI5/9B8wp2k04xfdctlU1DeqbqO9Cy CjkcPhPsM6/llYDVrseLzUw5GLycjvaRCd+GjmKuQAHl6MtrLSumSqTFjy7nZol/5xZgsL S6rc6LS+KiGLiM8ckWTatUsha0+vaNt3v5SlmxyVmYazOWY/m5pdqJAKOFOsCQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D240106 crest@rlwinm.de changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |crest@rlwinm.de --- Comment #15 from crest@rlwinm.de --- You probably want to create the vlan interfaces on the physical interface a= nd add them as member interfaces to the bridges and all IP interfaces belong on the bridge interface. Don't put IP addresses on bridge member interfaces. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Mar 30 18:03:05 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id E90C01A3E67E for ; Wed, 30 Mar 2022 18:03:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KTDn55Plsz4rLP for ; Wed, 30 Mar 2022 18:03:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 77EFE17F67 for ; Wed, 30 Mar 2022 18:03:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 22UI35Gc018071 for ; Wed, 30 Mar 2022 18:03:05 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 22UI35sq018070 for jail@FreeBSD.org; Wed, 30 Mar 2022 18:03:05 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 240106] VNET issue with ARP and routing sockets in jails Date: Wed, 30 Mar 2022 18:03:05 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: bugs.freebsd.org@gabor.adorjani.net X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1648663385; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Nu0FIPhlTRZXZ/YRR/PIy8jhCc/1bfZZjrVnZ0NVoiw=; b=i/QTnIoaK9NYvByuy3G6OL6mkjcxjrtSWahU5UO0J7auRnmZb4ajicGHNBq+leUQH8vLnI XoXbEx0karU1xbw6CshuJKwAVxiNNjdBuT945L2PjuXANwXW2tMxxe5Jfnzq/KL3RH39EY KAN4PXFPGTVVTS9wG57dvx3qrmwLFKv4iAorMlWja5d0e8BIvxKceEFW9KSvZ3QDHLyDMw fTkXGhLjjBoevDb+6ZNnZuv2Kiiz46fsn+C0q+j+h0uKzTDcQ6zHyYgW5XFDTLoAT+iFVU T3cgK97621ox4YO1ZpV1yEKSAp+PaWnHjT3befPmSyLPQW5WoO/NG6DtyXxtGQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1648663385; a=rsa-sha256; cv=none; b=j589KBbUxiI/7MfokCyELsCdOKG7JMs09lkmWbzQK2C8hdg48kAj75He4wZuEItPjlXAAW Fy0c86K5Bt3ETXRm2OYPjPJ+HoopTpOyLQlXNA+E9goNPSqZvKATelX5YGePh6xFYtriQK NreSzTR6w69FXi4MPbeUoxbxjaix2cIksNdcj8F+iXV4g6+RkeXhbLWqaiWUeyikD2IDr2 Z3sqyopKRT++FG5Vih9ym9CnBUR8mlgi0cot1k5UcAiIlyKrWFuYdDjYaKHWAzgCCU9O8v 0JjUIJ54ai4LcjwJXseGMdah82+toTgpRsn6hcCb+08cbiQQHhwJ1R9aju2NlQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D240106 --- Comment #16 from Gabor ADORJANI -= -- Thanks, I'll try - I've been doing this on Linux for many years. Ironically, some years ago I read the opposite in some FreeBSD-related doc: assigning t= he addresses to the parent interface was the recommended way. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Apr 1 12:00:35 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 0527919F9256; Fri, 1 Apr 2022 12:01:08 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from mailgate.Leidinger.net (mailgate.leidinger.net [IPv6:2a00:1828:2000:313::1:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "mailgate.leidinger.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KVJfV5zRfz3vkj; Fri, 1 Apr 2022 12:01:06 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from outgoing.leidinger.net (p508d4362.dip0.t-ipconnect.de [80.141.67.98]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-256) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "outgoing.leidinger.net", Issuer "R3" (verified OK)) by mailgate.Leidinger.net (Postfix) with ESMTPSA id D4A4523E57; Fri, 1 Apr 2022 14:00:56 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1648814457; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=vrZoto60ohjowkXyPGD+Cwv2G6ow+lqUe5Mb+PWLwak=; b=OV3++RhrAAje37ojtcN2EcWzLCD2ncl+biSG59/y8clnW9sbckTqeP+SGDRM8P0+6A9Plx dy4qlcNc3WrterVXOGqD/TaAfmmFh+oxx2bxc7OHzjKcAz2KSN2ydqNH3pWl7B0g6no349 Cydnc2ohr3wDPniTGg4FU4OhHTHX1YFgkO5blukcMrLss08bW+8L1YuG58Cppm+AAsNHe1 hKO/4XFanUJgl2hkbO6APoSZxsDUBFkb5bAsH76RfZdwT99MLa9QwJe3QN9McRmWown3s3 SgIejZc5cgZ/zygiZkIJN50a4/HcaVzbLo3NmxkVPBliOED1N5P59yJ6omrgZA== Received: from webmail.leidinger.net (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-256) server-digest SHA256) (Client did not present a certificate) by outgoing.leidinger.net (Postfix) with ESMTPS id 729E6C680; Fri, 1 Apr 2022 14:00:38 +0200 (CEST) Date: Fri, 01 Apr 2022 14:00:35 +0200 Message-ID: <20220401140035.Horde.LYLAkhpBnQPotxJtLawHfO8@webmail.leidinger.net> From: Alexander Leidinger To: current@freebsd.org, jail@freebsd.org Subject: injecting vars into rc-service-scripts at jail-start? Accept-Language: de,en Content-Type: multipart/signed; boundary="=_OFj-_eOqdGFBDWtOYkHfhOt"; protocol="application/pgp-signature"; micalg=pgp-sha256 List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 X-Rspamd-Queue-Id: 4KVJfV5zRfz3vkj X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=leidinger.net header.s=outgoing-alex header.b=OV3++Rhr; dmarc=pass (policy=quarantine) header.from=leidinger.net; spf=pass (mx1.freebsd.org: domain of Alexander@leidinger.net designates 2a00:1828:2000:313::1:5 as permitted sender) smtp.mailfrom=Alexander@leidinger.net X-Spamd-Result: default: False [-5.10 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[leidinger.net:s=outgoing-alex]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCVD_COUNT_THREE(0.00)[3]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[leidinger.net:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[leidinger.net,quarantine]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MLMMJ_DEST(0.00)[current,jail]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:34240, ipnet:2a00:1828::/32, country:DE]; RCVD_TLS_ALL(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[80.141.67.98:received] X-ThisMailContainsUnwantedMimeParts: N This message is in MIME format and has been PGP signed. --=_OFj-_eOqdGFBDWtOYkHfhOt Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, I'm overlooking something fundamental it seems... Context: I'm working on my auto-jailing of services idea: if the auto-jail is=20=20 enabled,=20a service like syslog is started inside a jail (which=20=20 inherits=20the FS and depending on some settings also inherits network=20= =20 and=20other stuff or not). My previous implementation was using _rc_prefix (jailstart) to denote=20=20 the=20start of a service inside a jail so that "service XXX start" on a=20= =20 host=20would "service XXX jailstart" inside a jail. This had off course=20= =20 issues=20as there is no infrastructure for multiple prefix like=20=20 onejailstart=20or jailonestart... Problem: Now I try to find a way to do it without a prefix, and the first thing=20= =20 which=20comes to my mind is to do "jail xxx 'exec.start=3D/usr/bin/env=20= =20 _rc_svcs=3Djailing /usr/bin/service XXX CMD ARGS'". My expectation is, that this would set _rc_svcs=3Djailing for the=20=20 command=20service XXX CMND args. Having a "set -x" in rc.subr shows=20=20 clearly=20in the jail-console log, that inside that jail, the variable=20= =20 _rc_svcj=20is not set. Using "-v" for the env command shows in the log=20= =20 that=20it is called and it sets the var and executes the service command=20= =20 with=20syslog start as arguments. I tried to find some env-cleanup part in rc.subr, which would discard=20=20 all=20_rc* variables, but if there is something like that I overlooked it. For a stop, I call "jexec /usr/bin/env _rc_svcj=3Djailing=20=20 /usr/sbin/service=20XXX stop args", and it works, so I rather tend to=20=20 believe=20there is no env-cleanup. What am I doing wrong so that _rc_svcj is not picked up inside the jail? So here is my diff between "prefix driven" (=3D working) and "var=20=20 driven"=20(var not picked up inside the jail): ---snip--- case "$rc_arg" in start) - if [ "${_rc_prefix}" !=3D jail ]; t= hen + if [ "${_rc_svcj}" !=3D jailing ]; = then _return=3D1 $JAIL_CMD -c=20=20 $_svcj_generic_params=20$_svcj_cmd_options \ -=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20 exec.start=3D"/usr/sbin/service ${name} jailstart $rc_extra_args" \ -=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20 exec.stop=3D"/usr/sbin/service ${name} jailstop $rc_extra_args" \ +=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20 exec.start=3D"/usr/bin/env _rc_svcj=3Djailing /usr/sbin/service ${name}=20= =20 ${rc_arg}=20$rc_extra_args" \ +=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20 exec.stop=3D"/usr/bin/env _rc_svcj=3Djailing /usr/sbin/service ${name}=20= =20 ${rc_arg}=20$rc_extra_args" \ =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20 exec.consolelog=3D"/var/log/svcj_${name}_console.log" \ name=3Dsvcj-${name}=20= =20 &&=20_return=3D0 else # normal start of=20= =20 _cmd=20via _run_rc_doit ---snip--- What set -x tells what it calls: ---snip--- + /usr/sbin/jail -c 'path=3D/' mount.nodevfs 'host=3Dinherit'=20=20 'ip4=3Dinherit' 'ip6=3Dinherit' allow.reserved_ports=20=20 'exec.start=3D/usr/bin/env -v _rc_svcj=3Djailing /usr/sbin/service -v=20=20 syslogd=20start ' 'exec.stop=3D/usr/bin/env _rc_svcj=3Djailing=20=20 /usr/sbin/service=20syslogd start '=20=20 'exec.consolelog=3D/var/log/svcj_syslogd_console.log' 'name=3Dsvcj-syslogd' ---snip--- Bye, Alexander. --=20 http://www.Leidinger.net=20Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF --=_OFj-_eOqdGFBDWtOYkHfhOt Content-Type: application/pgp-signature Content-Description: Digitale PGP-Signatur Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIzBAABCAAdFiEER9UlYXp1PSd08nWXEg2wmwP42IYFAmJG6WIACgkQEg2wmwP4 2IbsvRAAqMPhMXAr4CVQULA0/Xt61LneIcNV2PkkPVXoUL/aJA2hC2OVFZXq0Wwf wnz6CuA3Dw50AJjUCfNySSkl4nCP4tGSiHaDu2iBZFiE3b4EehtPoqgpDghNJJ7M bj9+ZkSTCZs+RrUyS3A+tD5Ib7ugWN1absvJ35srEvLfviR05YDD7j2aIinWC/8D 7wWlYXYZLN2CAubuueuHZ1g+1HAJtPWlGA1vLJGgAtdT6Bp07SbCtTJB2GMYLvq4 XzAnDLXhWB31GPMQrscPzBqyTBqQ/xz4YCBSECP7l88Zv7oshu4R23jJEY1jhKe0 8elCzOKCK53pScT3iS6jMVQrsLfuv8sSLgVEVxa48oaqyNcBfKK74ircaCAHakKY NlG6main9TcO49lQVYidWcvZc1UHkgRsHy3yJucgUHoa+GH7KWvXEJoAmG4KlLHv DKl6jHPoS6amH79NITwFWQiXz34LjpxT/KqpKqnoIJ+sEq/GmcAmL2kuPoErG1NO hLlQMB3+3mB/nLHJW38Z8UBt8OQIl+8pXowr4b44lLKznom6zFIIARPpgd0VQ+z5 DEXDtqLgOM7ILwT3iiaeAMUwYNjQT31NnIWstAvdiK6iyoV1Xdga9N6062x7iX80 4NSQqXs9+VjUH3v8qegAhzcQl4UYxb4fTtgn5LS5O2BP+uL8Kf0= =fEbf -----END PGP SIGNATURE----- --=_OFj-_eOqdGFBDWtOYkHfhOt-- From nobody Fri Apr 1 13:10:25 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 330031A464B3; Fri, 1 Apr 2022 13:10:49 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from mailgate.Leidinger.net (mailgate.leidinger.net [IPv6:2a00:1828:2000:313::1:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "mailgate.leidinger.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KVLBw2Vqtz4j42; Fri, 1 Apr 2022 13:10:48 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from outgoing.leidinger.net (p508d4362.dip0.t-ipconnect.de [80.141.67.98]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-256) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "outgoing.leidinger.net", Issuer "R3" (verified OK)) by mailgate.Leidinger.net (Postfix) with ESMTPSA id 9EB6C23FD4; Fri, 1 Apr 2022 15:10:44 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1648818644; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=31kyP7b/PGIhQZ92TPyDRRkzV3TSEBc0KT8wNxnPGP4=; b=Fz3PshCxjveHYKpHV0Zh2Rpuk3qBLWdntSjliyPDgLDENws0alGttfP4aWAL0Xegbo7NRx bmSByd58ccpsLzpyYNA8HthZJoAj5Ht7m4QQBwyYuA4tMLIhs5TVnacLzx3EC0lYcYIR9d WjyAE6fLO2YUzhgQYR/2/faPSqBY/87YDZMXHbVcZgB8uETK1ElH4M31byc62fal+eXtJS jV5EYoaB8ZL469NDjKyM/VjPJOFLf/VaHfVvoWJNGBJUCUUrbe1PkOQDbPTsMGwXfZgds6 +8/h95kQ3P4NcqVlGW5ykl/t3ojhqF4Ymuxpph1uU0griZgOoASIpsRtzigPCQ== Received: from webmail.leidinger.net (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-256) server-digest SHA256) (Client did not present a certificate) by outgoing.leidinger.net (Postfix) with ESMTPS id 5896AC282; Fri, 1 Apr 2022 15:10:26 +0200 (CEST) Date: Fri, 01 Apr 2022 15:10:25 +0200 Message-ID: <20220401151025.Horde.1Q8TGYwGtnEqeSARyDAkxSB@webmail.leidinger.net> From: Alexander Leidinger To: Jens Schweikhardt Cc: current@freebsd.org, jail@freebsd.org Subject: Re: injecting vars into rc-service-scripts at jail-start? References: <20220401140035.Horde.LYLAkhpBnQPotxJtLawHfO8@webmail.leidinger.net> <2063721701.9653898.1648815987577.JavaMail.zimbra@schweikhardt.net> In-Reply-To: <2063721701.9653898.1648815987577.JavaMail.zimbra@schweikhardt.net> Accept-Language: de,en Content-Type: multipart/signed; boundary="=_y2mTEICHMU0zESw1Ibqzhqd"; protocol="application/pgp-signature"; micalg=pgp-sha256 List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 X-Rspamd-Queue-Id: 4KVLBw2Vqtz4j42 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=leidinger.net header.s=outgoing-alex header.b=Fz3PshCx; dmarc=pass (policy=quarantine) header.from=leidinger.net; spf=pass (mx1.freebsd.org: domain of Alexander@leidinger.net designates 2a00:1828:2000:313::1:5 as permitted sender) smtp.mailfrom=Alexander@leidinger.net X-Spamd-Result: default: False [-5.10 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[leidinger.net:s=outgoing-alex]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+mx:c]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[leidinger.net:+]; DMARC_POLICY_ALLOW(-0.50)[leidinger.net,quarantine]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MLMMJ_DEST(0.00)[current,jail]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:34240, ipnet:2a00:1828::/32, country:DE]; RCVD_TLS_ALL(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[80.141.67.98:received] X-ThisMailContainsUnwantedMimeParts: N This message is in MIME format and has been PGP signed. --=_y2mTEICHMU0zESw1Ibqzhqd Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Quoting Jens Schweikhardt (from Fri, 1 Apr=20= =20 2022=2014:26:27 +0200 (CEST)): > Identifier confusion? You use _rc_svcs and _rc_svcj in your description. Typo.... s/svcs/svcj/ in the explanation. The diff/code has the vars correct (svcj) and the conditional and the=20=20 setting=20are close to each other and are "_rc_svcj". Bye, Alexander. --=20 http://www.Leidinger.net=20Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF --=_y2mTEICHMU0zESw1Ibqzhqd Content-Type: application/pgp-signature Content-Description: Digitale PGP-Signatur Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIzBAABCAAdFiEER9UlYXp1PSd08nWXEg2wmwP42IYFAmJG+cEACgkQEg2wmwP4 2IZrvQ//R383L/qHZ0C+FcZ08WlafMyLGqSy1Qn4IeeWdmmXqeudgNPzpIkkevFh wpEMz10dMltpcd1yF3a9ayqXvXlbfZhqrpOzoiHpdSkzSZMQTFP/7MNWw7Or0izi m/OaRj/GW/Wsog5V4AMCEujZQ0LMPcWjV9quLW9Ct1JFizIXb54ydjFx+Ix9Pk2c 6lDsAtkh27RMqMp2TuJbWN8VG+1YSn71mEKyE2y7rdGiAx9h2zfcL9OMDr/r1fXR YAUPyguDz06mcQe5yYwP8t0h+YmP86qqeR1bIBoSKtjysOHaaSRRBtl4NkMFu/vQ 3dz6gWHwTKb7rF7LmM9roVM1PJZ8bjkYYSgptEaAhxHN37jX4AGCuhMKObNYaV/j uhd6XwnrIBfNsf2zSlLX5hkF8S+BLBxQjJS0ivuzRAH0icXFN5LAPf4bgaR1Qsgj MZscy9lPJgDIwcoDFMvrF/1IVh/12dfPENFC528kKW+dYnM+NcR+rKJ01prAdieN SAJvTkFwyaNF22VcFkY40MwuQiVA+m6zwZK22z9qLG1quYtzNR2Lfj3RRkSsZ53C Tg/1+FcbMhlWabXE17YeATMM+BSpZ35QKk9Np9tu8UDzR9R/05DvliqX61zwIFrs RDmHWAR3BL//sAPbV0toM4lhInE80DiMMb2gY2yjKd4WkInc2dw= =vw5a -----END PGP SIGNATURE----- --=_y2mTEICHMU0zESw1Ibqzhqd-- From nobody Sun Apr 3 19:48:42 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 1004B1A5101C for ; Sun, 3 Apr 2022 19:48:59 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from mailgate.Leidinger.net (mailgate.leidinger.net [IPv6:2a00:1828:2000:313::1:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "mailgate.leidinger.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KWkxQ1N04z3JPt; Sun, 3 Apr 2022 19:48:58 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from outgoing.leidinger.net (p508d4362.dip0.t-ipconnect.de [80.141.67.98]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-256) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "outgoing.leidinger.net", Issuer "R3" (verified OK)) by mailgate.Leidinger.net (Postfix) with ESMTPSA id 7F01A264C1; Sun, 3 Apr 2022 21:48:47 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1649015327; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=iV32RqFPNg5oh/9x17EPnh1lAglxLljd11X9QyvoEgM=; b=X605mG0pJa2fzZt8gFluqF6Fj3nhOP+bW0AY7xIRCSzyiGAOFPoAmdBP1s0MWs4fPfzGHt qRBq4zkixH1N+346mZn7RqHwyJJIW3LETJolRh5rMliVhNnS9nfKW8vztvPo3Q7qPxNeV6 TbI8+t6GCtb5VOr0j5xvfBEHbRmCPBcKg1ER0BEFhpmTawDPRUY71LXPSDDdR8wrPYTGMJ 23DyVbt6BEWpIMhcYGaJPveLmz6hxn26SXzZ79QEYKa/gFvYTlKi/Ff0zSQQAFFAHnVJqy MHbwd7RTCqgUoezuzvwB4kZAN1MI2fA+AQeWknMFncN9t47e41/2iUG6M82ZDA== Received: from webmail.leidinger.net (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-256) server-digest SHA256) (Client did not present a certificate) by outgoing.leidinger.net (Postfix) with ESMTPS id EBAC0C380; Sun, 3 Apr 2022 21:48:42 +0200 (CEST) Date: Sun, 03 Apr 2022 21:48:42 +0200 Message-ID: <20220403214842.Horde.vlwSVh0KOZ6sL7aDfgA9KKL@webmail.leidinger.net> From: Alexander Leidinger To: security@freebsd.org, jail@freebsd.org Subject: Auto-jailing of services - 2nd implementation Accept-Language: de,en Content-Type: multipart/signed; boundary="=_YVfRcdE_UVp6rdkJjuoW1Qz"; protocol="application/pgp-signature"; micalg=pgp-sha256 List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 X-Rspamd-Queue-Id: 4KWkxQ1N04z3JPt X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=leidinger.net header.s=outgoing-alex header.b=X605mG0p; dmarc=pass (policy=quarantine) header.from=leidinger.net; spf=pass (mx1.freebsd.org: domain of Alexander@leidinger.net designates 2a00:1828:2000:313::1:5 as permitted sender) smtp.mailfrom=Alexander@leidinger.net X-Spamd-Result: default: False [-6.10 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; HAS_ATTACHMENT(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[leidinger.net:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[leidinger.net,quarantine]; NEURAL_HAM_SHORT(-1.00)[-1.000]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:+,4:~]; ASN(0.00)[asn:34240, ipnet:2a00:1828::/32, country:DE]; RECEIVED_SPAMHAUS_PBL(0.00)[80.141.67.98:received]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[leidinger.net:s=outgoing-alex]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.20)[multipart/signed,multipart/mixed,text/plain,text/diff]; MLMMJ_DEST(0.00)[jail]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N This message is in MIME format and has been PGP signed. --=_YVfRcdE_UVp6rdkJjuoW1Qz Content-Type: multipart/mixed; boundary="=_w3-e5ke9RnnaZj37w0YmaYo" This message is in MIME format. --=_w3-e5ke9RnnaZj37w0YmaYo Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, attached is a new implementation of service jails (auto-jailing of=20=20 services).=20This one now supports rc command prefixes (e.g. onestart)=20= =20 and=20I tested it in nested jails. The benefit of auto-jailing services=20= =20 is,=20that you can apply some restrictions to services (and what other=20= =20 processes=20it may see). If your service requires access to network but=20= =20 not=20sysvipc, and it doesn't run as root, it can be limited to network=20= =20 access=20with or without raw sockets, filesystem-permitted files, and=20=20 doesn't=20see other processes on the system. For a few services I have added the required "svcj-config" in the=20=20 start=20scripts (e.g. network access for syslog by setting=20=20 syslogd_svj_options=3Dnet_basic). Possible svcj config options for service jails: + netv4) + _svcj_cmd_options=3D"ip4=3Dinherit allow.reserved_ports=20=20 ${_svcj_cmd_options}" +=09 ;; + netv6) + _svcj_cmd_options=3D"ip6=3Dinherit allow.reserved_ports=20=20 ${_svcj_cmd_options}" +=09 ;; + net_basic) + _svcj_cmd_options=3D"ip4=3Dinherit ip6=3Dinherit allow.reserved_ports= =20=20 ${_svcj_cmd_options}" +=09 ;; + net_raw) + _svcj_cmd_options=3D"allow.raw_sockets ${_svcj_cmd_options}" + ;; + net_all) + _svcj_cmd_options=3D"allow.socket_af allow.raw_sockets=20=20 allow.reserved_ports=20ip4=3Dinherit ip6=3Dinherit ${_svcj_cmd_options}" + ;; + sysvipc) + _svcj_cmd_options=3D"sysvmsg=3Dinherit sysvsem=3Dinherit=20=20 sysvshm=3Dinherit ${_svcj_cmd_options}" + ;; + mlock) + _svcj_cmd_options=3D"allow.mlock ${_svcj_cmd_options}" + ;; + vmm) + _svcj_cmd_options=3D"allow.vmm ${_svcj_cmd_options}" By setting syslogd_svcj=3D"YES" in rc.conf your syslogd will be started=20= =20 in=20a jail which inherits the full filesystem and the ipv4 and ipv6=20=20 addresses=20of the parent. It would be nice if interested people could experiment a little bit=20=20 with=20this, e.g. adding name_svcj_options=3D"X Y" from above and=20=20 name_svcj=3D"YES" into rc.conf and see if it works. Note, doing that for=20= =20 sshd=20doesn't make sense in the generic case, it wouldn't see your=20=20 jails.=20It may make sense for services. Any kind of feedback and tested name_svcj_options submissions welcome... Bye, Alexander. --=20 http://www.Leidinger.net=20Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF --=_w3-e5ke9RnnaZj37w0YmaYo Content-Type: text/diff; charset=utf-8; name=svcj.diff Content-Disposition: attachment; size=9845; filename=svcj.diff Content-Transfer-Encoding: quoted-printable diff --git a/libexec/rc/rc.d/auditdistd b/libexec/rc/rc.d/auditdistd index 13cb5d5b69d..3218bd35755 100755 --- a/libexec/rc/rc.d/auditdistd +++ b/libexec/rc/rc.d/auditdistd @@ -19,4 +19,7 @@ required_files=3D"/etc/security/${name}.conf" extra_commands=3D"reload" =20 =20load_rc_config $name + +: ${auditdistd_svcj_options:=3D"net_basic"} + run_rc_command "$1" diff --git a/libexec/rc/rc.d/ftpd b/libexec/rc/rc.d/ftpd index dc623ea5943..a04c7ce5ee2 100755 --- a/libexec/rc/rc.d/ftpd +++ b/libexec/rc/rc.d/ftpd @@ -23,4 +23,7 @@ ftpd_prestart() } =20 =20load_rc_config $name + +: ${ftpd_svcj_options:=3D"net_all"} + run_rc_command "$1" diff --git a/libexec/rc/rc.d/inetd b/libexec/rc/rc.d/inetd index aa8ac20aeae..8cf7be5d91e 100755 --- a/libexec/rc/rc.d/inetd +++ b/libexec/rc/rc.d/inetd @@ -18,4 +18,7 @@ required_files=3D"/etc/${name}.conf" extra_commands=3D"reload" =20 =20load_rc_config $name + +: ${inetd_svcj_options:=3D"net_basic"} + run_rc_command "$1" diff --git a/libexec/rc/rc.d/kadmind b/libexec/rc/rc.d/kadmind index 773b2d0e499..1bdd420e415 100755 --- a/libexec/rc/rc.d/kadmind +++ b/libexec/rc/rc.d/kadmind @@ -26,4 +26,7 @@ kadmind_start_precmd() } =20 =20load_rc_config $name + +: ${kadmind_svcj_options:=3D"net_basic"} + run_rc_command "$1" diff --git a/libexec/rc/rc.d/kdc b/libexec/rc/rc.d/kdc index c2747ae08ca..11205d6e092 100755 --- a/libexec/rc/rc.d/kdc +++ b/libexec/rc/rc.d/kdc @@ -26,4 +26,7 @@ kdc_start_precmd() } =20 =20load_rc_config $name + +: ${kdc_svcj_options:=3D"net_basic"} + run_rc_command "$1" diff --git a/libexec/rc/rc.d/kpasswdd b/libexec/rc/rc.d/kpasswdd index a2875bf1515..af7b7a6b9aa 100755 --- a/libexec/rc/rc.d/kpasswdd +++ b/libexec/rc/rc.d/kpasswdd @@ -26,4 +26,7 @@ kpasswdd_start_precmd() } =20 =20load_rc_config $name + +: ${kapsswd_svcj_options:=3D"net_basic"} + run_rc_command "$1" diff --git a/libexec/rc/rc.d/local_unbound b/libexec/rc/rc.d/local_unbound index 19cb9a6c5c0..7436034495f 100755 --- a/libexec/rc/rc.d/local_unbound +++ b/libexec/rc/rc.d/local_unbound @@ -34,6 +34,7 @@ load_rc_config $name : ${local_unbound_anchor:=3D${local_unbound_workdir}/root.key} : ${local_unbound_forwarders:=3D} : ${local_unbound_tls:=3D} +: ${local_unbound_svcj_options:=3D"net_basic"} =20 =20do_as_unbound() { diff --git a/libexec/rc/rc.d/lpd b/libexec/rc/rc.d/lpd index fc8180cb221..725adda9072 100755 --- a/libexec/rc/rc.d/lpd +++ b/libexec/rc/rc.d/lpd @@ -25,4 +25,7 @@ chkprintcap() } =20 =20load_rc_config $name + +: ${lpd_svcj_options:=3D"net_basic"} + run_rc_command "$1" diff --git a/libexec/rc/rc.d/syslogd b/libexec/rc/rc.d/syslogd index 2351c086212..95d2b156b88 100755 --- a/libexec/rc/rc.d/syslogd +++ b/libexec/rc/rc.d/syslogd @@ -71,4 +71,7 @@ set_socketlist() echo $_socketargs } load_rc_config $name + +: ${syslogd_svcj_options:=3D"net_basic"} + run_rc_command "$1" diff --git a/libexec/rc/rc.subr b/libexec/rc/rc.subr index dc4f49612c2..f339738c0a3 100644 --- a/libexec/rc/rc.subr +++ b/libexec/rc/rc.subr @@ -51,6 +51,9 @@ PROTECT=3D"/usr/bin/protect" ID=3D"/usr/bin/id" IDCMD=3D"if [ -x $ID ]; then $ID -un; fi" PS=3D"/bin/ps -ww" +SERVICE=3D/usr/sbin/service +JAIL_CMD=3D/usr/sbin/jail +_svcj_generic_params=3D"path=3D/ mount.nodevfs host=3Dinherit" JID=3D0 # rc_service provides the path to the service script that we are executing= . # This is not being set here in an execution context, necessarily, so it's @@ -368,6 +371,16 @@ _find_processes() $_procname|$_procnamebn|${_procnamebn}:|"(${_procnamebn})"|"[${_proc= namebn}]")' fi =20 +=09if checkyesno ${name}_svcj; then + JID=3D$(/usr/sbin/jls -j svcj-${name} jid) + + case ${JID} in + ''|*[!0-9]*) + # svj-jail doesn't exist, fallback to host-check + JID=3D0 + ;; + esac + fi _proccheck=3D"\ $PS 2>/dev/null -o pid=3D -o jid=3D -o command=3D $_psargs"' | while read _npid _jid '"$_fp_args"'; do @@ -959,6 +972,18 @@ run_rc_command() _pidcmd=3D _procname=3D${procname:-${command}} =20 +=09# If a specifc jail has a specific svcj request, honor it (YES/NO). + # If not (variable empty), evaluate the global svcj catch-call. + # A global YES can be overriden by a specific NO, and a global NO is over= riden + # by a specific YES. + eval _svcj=3D\$${name}_svcj + if [ -z "$_svcj" ]; then + _svcj=3D${svcj_all_enable} + if [ -z "$_svcj" ]; then + eval ${name}_svcj=3DNO + fi + fi + # setup pid check command if [ -n "$_procname" ]; then if [ -n "$pidfile" ]; then @@ -994,7 +1019,7 @@ run_rc_command() _fib=3D\$${name}_fib _env=3D\$${name}_env \ _prepend=3D\$${name}_prepend _login_class=3D\${${name}_login_class:-d= aemon} \ _limits=3D\$${name}_limits _oomprotect=3D\$${name}_oomprotect \ - _env_file=3D\$${name}_env_file + _env_file=3D\$${name}_env_file _svcj_options=3D\$${name}_svcj_options =20 =20 if [ -n "$_env_file" ] && [ -r "${_env_file}" ]; then # load env from f= ile set -a @@ -1008,6 +1033,42 @@ run_rc_command() fi fi =20 +=09if [ -n "$_svcj_options" ]; then # translate service jail options + _svcj_cmd_options=3D"" + + for _svcj_option in $_svcj_options; do + case "$_svcj_option" in + netv4) + _svcj_cmd_options=3D"ip4=3Dinherit allow.reserved_ports ${_svcj_cmd_o= ptions}" + ;; + netv6) + _svcj_cmd_options=3D"ip6=3Dinherit allow.reserved_ports ${_svcj_cmd_o= ptions}" + ;; + net_basic) + _svcj_cmd_options=3D"ip4=3Dinherit ip6=3Dinherit allow.reserved_ports= ${_svcj_cmd_options}" + ;; + net_raw) + _svcj_cmd_options=3D"allow.raw_sockets ${_svcj_cmd_options}" + ;; + net_all) + _svcj_cmd_options=3D"allow.socket_af allow.raw_sockets allow.reserved= _ports ip4=3Dinherit ip6=3Dinherit ${_svcj_cmd_options}" + ;; + sysvipc) + _svcj_cmd_options=3D"sysvmsg=3Dinherit sysvsem=3Dinherit sysvshm=3Din= herit ${_svcj_cmd_options}" + ;; + mlock) + _svcj_cmd_options=3D"allow.mlock ${_svcj_cmd_options}" + ;; + vmm) + _svcj_cmd_options=3D"allow.vmm ${_svcj_cmd_options}" + ;; + *) + echo ${name}: unknown service jail option: $_svcj_option + ;; + esac + done + fi + [ -z "$autoboot" ] && eval $_pidcmd # determine the pid if necessary =20 =20 for _elem in $_keywords; do @@ -1053,9 +1114,50 @@ run_rc_command() if [ -n "$_env" ]; then eval "export -- $_env" fi - _run_rc_precmd || return 1 - _run_rc_doit "$_cmd $rc_extra_args" || return 1 - _run_rc_postcmd + + if [ "${_rc_svcj}" !=3D jailing ]; then + _run_rc_precmd || return 1 + fi + if ! checkyesno ${name}_svcj; then + _run_rc_doit "$_cmd $rc_extra_args" || return 1 + else + case "$rc_arg" in + start) + if [ "${_rc_svcj}" !=3D jailing ]; then + _return=3D1 + $JAIL_CMD -c $_svcj_generic_params $_svcj_cmd_options \ + exec.start=3D"export _rc_svcj=3Djailing; for d in /etc/rc.d $loc= al_startup; do [ -x \$d/${name} ] && \$d/${name} ${_rc_prefix}start $rc_ext= ra_args && break; done" \ + exec.stop=3D"export _rc_svcj=3Djailing; for d in /etc/rc.d $loca= l_startup; do [ -x \$d/${name} ] && \$d/${name} ${_rc_prefix}stop $rc_extra= _args && break; done" \ + exec.consolelog=3D"/var/log/svcj_${name}_console.log" \ + name=3Dsvcj-${name} && _return=3D0 + else + _run_rc_doit "$_cmd $rc_extra_args" || _return=3D1 + fi + ;; + stop) + if [ "${_rc_svcj}" !=3D jailing ]; then + $SERVICE -j svcj-${name} ${name} ${_rc_prefix}stop $rc_extra_args ||= _return=3D1 + $JAIL_CMD -r svcj-${name} 2>/dev/null + else + _run_rc_doit "$_cmd $rc_extra_args" || _return=3D1 + fi + ;; + restart|status) ;; # no special case needed for svcj or handled somewh= ere else + *) +if checkyesno ${name}_svcj; then +echo XXX: check if \"$rc_arg\" needs to be executed in the jail or outside +fi +# if [ "${_rc_svcj}" !=3D jailing ]; then +# $SERVICE -j svcj-${name} ${name} ${_rc_prefix}${rc_arg} $rc_extra_a= rgs || _return=3D1 +# else + _run_rc_doit "$_cmd $rc_extra_args" || _return=3D1 +# fi + ;; + esac + fi + if [ "${_rc_svcj}" !=3D jailing ]; then + _run_rc_postcmd + fi return $_return fi =20 @@=20-1113,9 +1215,21 @@ run_rc_command() return 1 fi =20 -=09 if ! _run_rc_precmd; then - warn "failed precmd routine for ${name}" - return 1 + if [ "${_rc_svcj}" !=3D jailing ]; then + if ! _run_rc_precmd; then + warn "failed precmd routine for ${name}" + return 1 + fi + fi + + if checkyesno ${name}_svcj; then + if [ "${_rc_svcj}" !=3D jailing ]; then + $JAIL_CMD -c $_svcj_generic_params $_svcj_cmd_options\ + exec.start=3D"export _rc_svcj=3Djailing; for d in /etc/rc.d $loca= l_startup; do [ -x \$d/${name} ] && \$d/${name} ${_rc_prefix}start $rc_extr= a_args && break; done" \ + exec.stop=3D"export _rc_svcj=3Djailing; for d in /etc/rc.d $local= _startup; do [ -x \$d/${name} ] && \$d/${name} ${_rc_prefix}stop $rc_extra_= args && break; done" \ + exec.consolelog=3D"/var/log/svcj_${name}_console.log" \ + name=3Dsvcj-${name} || return 1 + fi fi =20 =20 # setup the full command to run @@ -1152,16 +1266,28 @@ $command $rc_flags $command_args" # Prepend default limits _doit=3D"$_cd limits -C $_login_class $_limits $_doit" =20 + +=09 local _really_run_it=3Dtrue + if checkyesno ${name}_svcj; then + if [ "${_rc_svcj}" !=3D jailing ]; then + _really_run_it=3Dfalse + fi + fi + + if [ "$_really_run_it" =3D true ]; then # run the full command # - if ! _run_rc_doit "$_doit"; then - warn "failed to start ${name}" - return 1 + if ! _run_rc_doit "$_doit"; then + warn "failed to start ${name}" + return 1 + fi fi =20 +=09 if [ "${_rc_svcj}" !=3D jailing ]; then # finally, run postcmd # - _run_rc_postcmd + _run_rc_postcmd + fi ;; =20 =20 stop) @@ -1183,6 +1309,11 @@ $command $rc_flags $command_args" # and run postcmd. wait_for_pids $rc_pid =20 +=09 if checkyesno ${name}_svcj; then + # remove service jail + $JAIL_CMD -r svcj-${name} 2>/dev/null + fi + _run_rc_postcmd ;; =20 --=_w3-e5ke9RnnaZj37w0YmaYo-- --=_YVfRcdE_UVp6rdkJjuoW1Qz Content-Type: application/pgp-signature Content-Description: Digitale PGP-Signatur Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIzBAABCAAdFiEER9UlYXp1PSd08nWXEg2wmwP42IYFAmJJ+hkACgkQEg2wmwP4 2IYpnA//WhZ/G/sLzATWcHOtHntm3S1hVQNJCUy9dHKbtkpyXSWHkfenSTDlLqeb 2k3VlnH5qmHvcg0bT67Za53o9uqm3TwdX4/L7SOAwL4EgsgzHMBIuuG+A8UIkUiP KOk0XFTFKdkRe1VSH09Xf8ol2EeLZ/fpIsKGTLq0L32CUhbcl+P616RVrxrz6GJW Pze2YlTK885T0jWGqCFkJ3MtycOiId9HJP1K+4eCuQoWjfvzXQVpnueKt+jEQJEM xzcp9VMlOOrLxEdpjyMxCoyfu4czQG9aeEznoAww6npSUmjcQanXnVi/fkOjeXbM /Y4tnSK0mEHa4w1n/YIdcUTElbzUOF2qy4JdTfaRrgdumWF/2wV+GQ9YFEmR6MSZ I0ORHT/r684dAxbRLzk2g4yL9wrvRPUSrYru4n+bxaqDH79sCodzpjOM3wgN2fGy o7yRNuAexv7X+ROcDan/hbWs2YdXBz1qJ/ofepf4K/HxUtElsruL8tBATZDzXg0y Prk1TSUSIQXEauSG/FoQxMnXidVFCvIbUHxZs2UzH8yBVKPk1zsWmWZwOYzsnU5R uqyOZfxhzvgZO3g0dQ4uf7aBGvS+riTUFiBAVe4yt21AP6WvY3NFTLB4fH7C11z5 wYz0BAWpAHY8BNG+gPqK1631yrelC070bPFw9S3LoPwhUbzke44= =2g/S -----END PGP SIGNATURE----- --=_YVfRcdE_UVp6rdkJjuoW1Qz-- From nobody Mon Apr 4 18:50:59 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id E537F1A9C5C9 for ; Mon, 4 Apr 2022 18:51:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KXKc45RlPz3pGq for ; Mon, 4 Apr 2022 18:51:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 9C36817557 for ; Mon, 4 Apr 2022 18:51:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 234Ip0jw000476 for ; Mon, 4 Apr 2022 18:51:00 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 234Ip09n000475 for jail@FreeBSD.org; Mon, 4 Apr 2022 18:51:00 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 255830] dummynet(4) queues/pipes do not work inside of a VNET jail Date: Mon, 04 Apr 2022 18:50:59 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: kumba@gentoo.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1649098260; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3+7msy843lP9018iKWJwM8vn/HQc77O3FhJdb5Lejt0=; b=fTi1Dh42an6yvlLGWf3YjLoHtjqCCNgQDsSNeiQKlOLJ8qeeAsTApCuYtJOGMPCa1QLssz FCO726wbOPdusNjqQ+t+KHOLrW6abM8RG34A0Tg9WwQu92XgYgWF1JKo9hE832n05bq5cy 2equsZipuGihvRpbqRdyBJYMw/Wqs05azq2QOqP0Vkd3p4loOMox7OJXh1qDJNjtYG0XAQ eZ8CfRgjsInUdYhsTYyJyH+gC7mKfmyuGcu4GHacyPILmOvvufZ3c1TguRt0mEB62o4y8Z HbnIkQMZAwfQMjlKWvsxHYsuu+Sd3pW6cj2tdrYez7gwoJM4Cx1A0npI4fd7xQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1649098260; a=rsa-sha256; cv=none; b=fAnUWGBpSx7tc2mwCTYy5HEt6qPP1bBY+W+I/m+T4+yYUguobHkgjQ/nriQUFGjyfm/xKv oC9y5GzitimScry1r0yvOnkSYvwex5pR4C6xBbTmVEVzgzqkuq6nTwYQK0l9lRtAmyfOxk 6GXuxTWPo/rLmap1WBD4DRYeXQ4x1yLt7LdcqSNFZnExUGrq1HjY0QwGDC8in+oCjACklY +AljX8n3TloN7Y7+W6pimycJwBLus+zG9CSlW7vs4m6vN1iiVwaI8iBBoXaOifbG1owUog Deh4LjGHw+lgCK1OgEnIV5Y+n8y6nhUtqlVBpbyGoRE8Q51mS5i63w0+LkHOng== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D255830 --- Comment #8 from Joshua Kinard --- (In reply to Kristof Provost from comment #7) FWIW, I've revisited this in 13.1-RC1 on my router and was able to successf= ully put my router configuration into a jail by allowing the jail to have direct access to the WAN and LAN ports (the jail host has six GbE ports total, so = it runs management services on a third port). I was also able to get the separate wireless subnet into the same jail after putting wlan0 into the jail and tweaking hostapd's rc.d script to allow run= ning in a jail. So far, the only issue noticed, which appears to be benign, are these messa= ges in dmesg when the jail is shutdown or restarted: [nhop_ctl] inet.0 nhop_free: failed to unlink nh#3/inet/em0/resolve [nhop_ctl] inet.0 nhop_free: failed to unlink nh#2/inet/em0/ [nhop_ctl] inet.0 nhop_free: failed to unlink nh#5/inet/em1/resolve [nhop_ctl] inet6.0 nhop_free: failed to unlink nh#5/inet6/em1/resolve If those messages are not an issue, I'd say this bug can be closed. Thanks! --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Mon Apr 11 02:45:09 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 9CCA81A9F4A8 for ; Mon, 11 Apr 2022 02:45:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KcCrP3PKfz3m34 for ; Mon, 11 Apr 2022 02:45:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 51E6D1AC97 for ; Mon, 11 Apr 2022 02:45:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 23B2j9KY004430 for ; Mon, 11 Apr 2022 02:45:09 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 23B2j9Fk004429 for jail@FreeBSD.org; Mon, 11 Apr 2022 02:45:09 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 228351] cannot unhide log or tty with devfs.rules Date: Mon, 11 Apr 2022 02:45:09 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: conf X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: zlei.huang@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1649645109; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZQmwEaSN3Kg3Bi1S+OyrESK5HOCoboR/QSNEnpxb7ZA=; b=koq3syrDUn8ul8AgXhzOv9nQ28SM9sPpvzC4pq8nDAtOZ4HUZZAH1wPkRqMy9eF2tyN23S sUlejWmUmdl4/DMR029Z94vYGCHKM0bGSXtJdCoNMVGPBaTXaNlP0/2AY/YM/fkVzn9BJz vhCl0OWGkW6+I0J22rITPni1hlpRrnXKeKi7ep/FVP3LIzlc+qSarkxx5qr7vXkWzcmB6Y u8uCzIFbUgSJ7xeU+GkXucCbm7syqxJ1xVGcfgI+EKtGaR+ymDzDGeBmvevCEhMSYdIZLD +mTEKX2BuzaMTtSIn6TpmClbtSsmBZiornh2yvkwO3Mh4ckw/MQtiHNvA/h6KQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1649645109; a=rsa-sha256; cv=none; b=J1ClfQ7WZrq5d4ThD12Grx2roIRvlt9jEoXbyB4z3kkbYYR4igkBn3+M8nLhLN1vzQgAqm N1M7RzAd2en0Ul9HGGwRSogJVYNSzrr/MDDiEqcP+xA7UrAiIOof8RlJoywuR7CxE8ygG2 2psS45i1kqP4dEgRaCqcbpxtPzjNzdM67gnDIxqG02vCTAtuqjjixjMR68nZ6FLWaVv/2X CBlIFOBc8JDWMQbAPX01Iq1I29n97S2tqDM1+RIYv3919HZvspgVru+n4SgvnBL3mtbEAx UEJGn5gMhjHxjnMNsSRyDJN/kHZYHTzmT/FYrYZRXlQHSjF5qa37DYyL2T049w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D228351 --- Comment #5 from Zhenlei Huang --- Proposed change to prevent the ambiguity for end users https://reviews.freebsd.org/D34563 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Apr 21 12:42:21 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 267B91995767 for ; Thu, 21 Apr 2022 12:42:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Kkccs5XlLz3wJH for ; Thu, 21 Apr 2022 12:42:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 9FE6F74F for ; Thu, 21 Apr 2022 12:42:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 23LCgLa2007234 for ; Thu, 21 Apr 2022 12:42:21 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 23LCgLOk007233 for jail@FreeBSD.org; Thu, 21 Apr 2022 12:42:21 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 240106] VNET issue with ARP and routing sockets in jails Date: Thu, 21 Apr 2022 12:42:21 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: pen@lysator.liu.se X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1650544941; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HVx4yLhPbr+ZFyPMHz5aFSm6Femk4R794KShgEcWLfI=; b=tHu0U9wHshB6ukK/9lZmlZtEAvbIgyo/dM+5McCX1hGTfCXr3Vxjr9YIIY1j0yF48cDzfH RI6UBukHB64Rvy8klpnVHAXyPwLtRMYb300DBpjjQaUS/A9L7UpVr7+ttGU04oflaTe4nf PgFSW5iSUg3XvCFpqJ1WsNFiiZR9iTC04Rv5qd+rCr/0VWas17FO3G8A5mbgcLE0aZEU5D HcpZijlH4306d0/3dZac4log+0mW1B/DINckYUea32lODx5A7W8yaQ7lAsrgZ9BhzaVuRG pqG/v4RxouKy1t45X4QdtzUoP4xjZyM52wM3MGcjF/tMtY2c2A52lu8hZ7wj6g== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1650544941; a=rsa-sha256; cv=none; b=KDiZ0+amCywrQBoI9jiXKM2Jd4r9lugkjDlBA4NbjqYptm509cYC+gf5A8Z1M9vmZwX2Rl RJR9QduLongNAxv3a4UjED4I6lq6sYi/itP+2r+rmrGBb/QHE3GU0u6xGmw70SqPdfSQdb YAl/3fZjhD5Ho7Sgs79Vn3hk4Xz8e1I96MHFFK0xC8rYGyfxKRuhXvE0QO8BI8PSh85d/5 1suC5T/OU+RMyBdLSPBLJXaFW3Vcc7BmO9i2g2+Nw5fF0xh4vaFfygiTWK6uRzqhpglYIA 0UB/d302QVe/ZVZrliByxFAcrJHmCIW32rj5lwb9lhOOUni4SGgaaYEAAFDOWQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D240106 Peter Eriksson changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |pen@lysator.liu.se --- Comment #17 from Peter Eriksson --- I'm seeing the same (or similar issue) on 12.3-RELEASE-p5 when trying to br= idge a vlan interface into a jail: # egrep 'ifconfig|cloned' rc.conf ifconfig_ixl0=3D"up" ifconfig_ixl2=3D"up" cloned_interfaces=3D"lagg0 vlan1601 bridge0" ifconfig_lagg0=3D"laggproto lacp laggport ixl0 laggport ixl2 130.236.8.40 n= etmask 255.255.255.224 lacp_fast_timeout" ifconfig_lagg0_ipv6=3D"inet6 2001:6b0:17:2400::8:40/64 lacp_fast_timeout" ifconfig_vlan1601=3D"vlandev lagg0 vlan 1601 up" ifconfig_bridge0=3D"addm vlan1601 up" # ifconfig bridge0 bridge0: flags=3D8843 metric 0 mtu = 1500 ether 02:90:7b:7b:f5:00 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: vnet0.1 flags=3D143 ifmaxaddr 0 port 14 priority 128 path cost 2000 member: lagg0 flags=3D143 ifmaxaddr 0 port 10 priority 128 path cost 1000 member: vlan1601 flags=3D143 ifmaxaddr 0 port 11 priority 128 path cost 2000000 groups: bridge nd6 options=3D9 root@filur00:/etc # iocage console test Last login: Thu Apr 21 14:27:18 on pts/0 FreeBSD 12.3-RELEASE-p5 GENERIC -- ... root@test:~ # ping 130.236.8.65 PING 130.236.8.65 (130.236.8.65): 56 data bytes ^C --- 130.236.8.65 ping statistics --- 2 packets transmitted, 0 packets received, 100.0% packet loss If I now manually remove the "lagg0" member from the bridge0 interface then things start to work fine. It would be nice if it didn't add it automatical= ly :-) root@filur00:/etc # ifconfig bridge0 deletem lagg0 root@filur00:/etc # iocage console test Last login: Thu Apr 21 14:38:34 on pts/0 FreeBSD 12.3-RELEASE-p5 GENERIC=20 .... root@test:~ # ping 130.236.8.65 PING 130.236.8.65 (130.236.8.65): 56 data bytes 64 bytes from 130.236.8.65: icmp_seq=3D0 ttl=3D255 time=3D0.249 ms ^C --- 130.236.8.65 ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/stddev =3D 0.249/0.249/0.249/0.000 ms --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Apr 21 13:07:25 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 3DE22199C0A9 for ; Thu, 21 Apr 2022 13:07:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Kkd9q0Z5Kz4WBN for ; Thu, 21 Apr 2022 13:07:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id E7B8711C2 for ; Thu, 21 Apr 2022 13:07:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 23LD7QYK018254 for ; Thu, 21 Apr 2022 13:07:26 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 23LD7Q8A018253 for jail@FreeBSD.org; Thu, 21 Apr 2022 13:07:26 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 240106] VNET issue with ARP and routing sockets in jails Date: Thu, 21 Apr 2022 13:07:25 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: pen@lysator.liu.se X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1650546447; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AdXMVG77GTw2+w/2SO0sqCdJOnjdDvVxZIN8OXgMBI0=; b=PzyQL72NbYiMbkfoF3NP7Baa2J2Bj4eJDSRWw384AVobh+Aj7sqdaD71K6KgQmni5dGhWn YRcdZTfCMIrJqr01PDQbOYhosJM2eKCfQHEhokMH/fDEz4PsDcf2wq8KuTsa1cVJdv+f8Q g5OOI7Ps2gReY33AtCSAgXOp/XWqvB2eJTUHg6GUuiq9MjCBb+6cUAKoNe0lz0B21qugQ8 CehryiSA1XUZL5PK6/8o79r2Tgyj49KbULTfuQb+wW3vLZataYIuhaFARolGlxMt6iXNwy KROZRSZUe48Omr+bd5q4qRMZgTkIcmTAk7dny6qPK3xBhbCXa5agrSfQ8ynIAw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1650546447; a=rsa-sha256; cv=none; b=ty2yHrzM3BeNXdnv3gqXOPAwaelzYv/NYidkvLnoLuZTaO0D0s9gNmXsmXeXk9lMUfmqaL fNjOJ1BCmjkx1VWfebxAVRoJPDYunWvkw45A3MyrnqhWZP3nveVIjdC8mC/47qJqHI1TAL 0nYZOyzsW/NFWuO6GAqHkcgEph3rdc4PoO7R6LF+j22Hi15LyQgKW3cFN53C3k0gYbjqJ0 u9pSAvLKTT6sbzjmCVlfHOyxw+8XhB2SSrx0W/4cFOWjwIMQxquqbNA0qlh5VcLirJ61O5 5UksNlBrWr6cXSkT6hVfmSRJ6X13xWhju5+BOliZ77eG1tzkyCRipV4vsCuTzQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D240106 --- Comment #18 from Peter Eriksson --- > root@filur00:/etc # ifconfig bridge0 deletem lagg0 Easy solution... Remove from rc.conf: ifconfig_bridge0=3D"addm vlan1601 up" and then tell iocage to not add lagg0 automatically to the jail's bridge: # iocage set vnet_default_interface=3Dvlan1601 test --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat May 7 08:13:44 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 401AA11CBAAB for ; Sat, 7 May 2022 08:13:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KwKvW6GpVz4XLP for ; Sat, 7 May 2022 08:13:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id B7B9350A8 for ; Sat, 7 May 2022 08:13:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 2478DhAY056290 for ; Sat, 7 May 2022 08:13:43 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 2478DhKE056289 for jail@FreeBSD.org; Sat, 7 May 2022 08:13:43 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 240106] VNET issue with ARP and routing sockets in jails Date: Sat, 07 May 2022 08:13:44 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ohartmann@walstatt.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1651911223; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Wv+ZbXw39jhsP2p8htWtXtBe/p3gv6yMXRSPhCtsRcE=; b=sDhDaGmrKBRna9AdiwgW1VYh4irDJY8s4rV75I/e37Xn3+sZSwPT70hGgSlsk8hBjm5jHS luKIQhTU6mZIfLy99eFakZOzDwLbf9tRQcBr66bzxKZAY3OkEmgOlAa1blc4rPUwzDA/L4 1kpFDVbGZfgEFwmSDgkHAlxX+SsvMWjruuKZO8BMj8EPIZY/lIzT3gI8nAs9EqnxVeSwbA j/ZcELQKDEVIAzezDM3PL0qIsNqW581K8FDfvOeoqnxcLJcv9tImStlnQjNx2jqBEjSTUG vwiC348MZBCjxlmF9MYKEk3ra3GE577TXjiShjsyCFd87g7uUcFLvdNW9TWymg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1651911223; a=rsa-sha256; cv=none; b=meIvQ2dgyM4Yga4NqLNIZme5AW7rp41vKSpdOklXJ9SJJhsNAX+g0SU+cPYS2EHb7JkotK SM7JsGNd6ZXQU477fTI3kOqzVBagftIxqwfoz/iC2uA5jeipunKvzo8hYa/kbpLCJFX+kf nrwVubLpEceOKLx614j4fAS0By5gcuun9XravRVI2Omz4YQk+wp7V2xIB+vLllGmB0M9mT Tow72q/0RjV3glBi5YwfSaJQ+CDYqz1jwbbDbybIW1BLWsRx7IAOrqZka4DjPgRB+84YUn pKIYdPFV3Yy+lk5G2WtiPtzQVSLYquxyQoUa00VwB8NzAqetVDwsi+LmbXVVaQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D240106 O. Hartmann changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ohartmann@walstatt.org --- Comment #19 from O. Hartmann --- Hello. We also have an similar issue on FreeBSD 12.3-RELEASE-p2 (XigmaNAS, stuck at -p2 for the moment) as described. The boxes in question do have two NICs, o= ne is supposed for the management (em0) access and the other one is supposed t= o be bound to offered services. Additionally, the second NIC (igb0) is accessible via an IP AND serves as the physical NIC as member of a bridge for vnet jai= ls, which do have epair interfaces (in Xigmanas created via the FreeBSD in-tree tool "jib"). Binding provided services as SAMBA and NFS to the second NIC (igb0) works as expected, also ping and ssh is no problem. Base host's IP (both NICs) and those of the jails are within the same netwo= rk. When it comes to the vnet jails on the bridge, of which the igb0 NIC is mem= ber of, trouble begins. We use several jails on those boxes. Pinging those jails from outside the campus network does work sporadically with some IPs, it takes a long time u= ntil the jail starts repsonding. Same behaviour is within the LAN.=20 We also already disabled pfil on the bridges as suggested: device if_bridge net.link.bridge.ipfw: 0 net.link.bridge.allow_llz_overlap: 0 net.link.bridge.inherit_mac: 0 net.link.bridge.log_stp: 0 net.link.bridge.pfil_local_phys: 0 net.link.bridge.pfil_member: 0 net.link.bridge.ipfw_arp: 0 net.link.bridge.pfil_bridge: 0 net.link.bridge.pfil_onlyip: 0 A curiosity is that if one can ping one or two out of the five jails on the host, in another attempt to do so one, at most two different hosts would an= swer the ping then and the former working pinged hosts do not anymore. It is like gambling. We also run another host with the very same XigmaNAS version, in that case,= he second NIC is configured to be part of another network and attached to anot= her switch - not problem there! In the problematic cases described above, we do not have direct access to t= he switches of the backend of the department, so I can't see whether I'm the culprit (misconfiguration, misunderstanding et cetera of network technology= ). Hope the problem could be solved anyway within FreeBSD 12.3. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue May 10 19:21:29 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 288791ACAC0C for ; Tue, 10 May 2022 19:21:48 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from smtp5.goneo.de (smtp5.goneo.de [85.220.129.30]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4KySZy6RqTz3QPJ for ; Tue, 10 May 2022 19:21:46 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from hub2.goneo.de (hub2.goneo.de [85.220.129.53]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by smtp5.goneo.de (Postfix) with ESMTPS id E0E1310A0160 for ; Tue, 10 May 2022 21:21:38 +0200 (CEST) Received: from hub2.goneo.de (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by hub2.goneo.de (Postfix) with ESMTPS id 4795210A3317 for ; Tue, 10 May 2022 21:21:37 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=walstatt-de.de; s=DKIM001; t=1652210497; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=mnC4rOg/BfKDawy04gfdbwcl7GDs3Vj2q5sccHBlXxQ=; b=KAlp7tv7+ORfEVUaF9W9kILuiUa35+p4Oav3TosvofJ1Mq1q7IZ0U8q4Orwn5a2Gr7bBq2 owRBlsvJWKB9BsU8dxVyWWJSxhofLuVXaVHFTC8S0NNyAz1TroHKAEp9Mh6MCUmhf+zKL+ qABIjTU2vf+GaZQELnf4kVLPWmCGhHtXwBEZiVCJk+V6GNirHztHQBVBWhfSECvr1tQLDr b6kVJrRY44iJJPAzMvI1Cucb0EKP+Bwxb3y1eSd7iRYwY5p/A5o+EAicAAZUQ9hfXLIkQv zaOD+c9A4dybjzOQhxG2hUBONMjru6/2PPbFStxlzricNQGQz5BG1itCE8E59g== Received: from hermann (dynamic-2a01-0c22-ad27-3c00-f407-9de1-b468-ac89.c22.pool.telefonica.de [IPv6:2a01:c22:ad27:3c00:f407:9de1:b468:ac89]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by hub2.goneo.de (Postfix) with ESMTPSA id 170AF10A3308 for ; Tue, 10 May 2022 21:21:37 +0200 (CEST) Date: Tue, 10 May 2022 21:21:29 +0200 From: FreeBSD User To: freebsd-jail@freebsd.org Subject: FreeBSD 12.3-p5: problems vnet on if_bridge Message-ID: <20220510212129.35041f02@hermann> List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-UID: a0f323 X-Rspamd-UID: fceb88 X-Rspamd-Queue-Id: 4KySZy6RqTz3QPJ X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=walstatt-de.de header.s=DKIM001 header.b=KAlp7tv7; dmarc=none; spf=none (mx1.freebsd.org: domain of freebsd@walstatt-de.de has no SPF policy when checking 85.220.129.30) smtp.mailfrom=freebsd@walstatt-de.de X-Spamd-Result: default: False [-2.89 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[walstatt-de.de:s=DKIM001]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-jail@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[walstatt-de.de]; DKIM_TRACE(0.00)[walstatt-de.de:+]; NEURAL_HAM_SHORT(-0.99)[-0.987]; MLMMJ_DEST(0.00)[freebsd-jail]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; MID_RHS_NOT_FQDN(0.50)[]; ASN(0.00)[asn:25394, ipnet:85.220.128.0/17, country:DE]; RCVD_TLS_ALL(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[85.220.129.30:from] X-ThisMailContainsUnwantedMimeParts: N Hello, I ran into serious trouble setting up a FreeBSD 12.3-RELEASE-p5 host having a second NIC and vnt jails attached to that second NIC (basically, the host is a recent Xigmanas with Bastille jails, but the issue also occurs on a vanilla FreeBSD 12.3). The host is compromised of two NICs, em0 (management only) and igb0 (service/jails). Both, the server and the jails as well as the igb0 interface are residing on the same network, but both NICs are connected to two different ports on a switch, to which we do not have access (part of the campus infrastructure). Both NICs are attached with a IPv4 of the same network, the host is listening on both NICs for services, i.e. port 22 for ssh. No problem to connect to both(!) addresses via ssh. igb0 is member of an if_bridge. The box also hosts a bunch of vnet jails, each jail does have an if_epair created via "jib" and these vnet epairs are members of the bridge, to which ifb0 is also member. Problem: while any service bound to NIC igb0/IPv4 residing on igb0 is accessible flawlessly, accessing an jail is almost impossible. Pinging a jail does work after a while the ping initiating host has been waiting, in ery rare situations someone can access the sshd of the jail, but any access of that kind is highly erratic. From 5 jails, at most two are responding to pings, the other don't and it is non-deterministic which host will respond. Following some advices found on the web, the following sysctl settings are provided to if_bridge: device if_bridge net.link.bridge.ipfw: 0 net.link.bridge.allow_llz_overlap: 0 net.link.bridge.inherit_mac: 0 net.link.bridge.log_stp: 0 net.link.bridge.pfil_local_phys: 0 net.link.bridge.pfil_member: 0 net.link.bridge.ipfw_arp: 0 net.link.bridge.pfil_bridge: 0 net.link.bridge.pfil_onlyip: 0 We do not have access to the switch the box is connected to, so I don't have access to any logs revealing a problem either to a conceptual misunderstanding of networking of mine and so a misconfiguration or a probelm with Layer 2 or the switches themselfes. I'd like to ask whether someone has a similar setup up and running and could report this - or give a hint of the problem I possibly made (igb0 is attached to an IPv4 AND is member of an if_brige on which IPv4 attached vnet jails are residing). We have also already setup another "similar" scenarion with the same FreeBSD 12.3-p5 version and also two NICs, but our "service/jail" NIC is part of a different IPv4 network and the NIC is attached to a different switch (to which we have full access). Thanks in advance, O. Hartmann From nobody Wed May 11 18:47:55 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 640CA1ACD1E1; Wed, 11 May 2022 18:48:09 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from smtp6.goneo.de (smtp6.goneo.de [85.220.129.31]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Kz3nh142tz4cjB; Wed, 11 May 2022 18:48:07 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from hub2.goneo.de (hub2.goneo.de [85.220.129.53]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by smtp6.goneo.de (Postfix) with ESMTPS id AEEA410A332B; Wed, 11 May 2022 20:48:00 +0200 (CEST) Received: from hub2.goneo.de (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by hub2.goneo.de (Postfix) with ESMTPS id 130DE10A1E8B; Wed, 11 May 2022 20:47:59 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=walstatt-de.de; s=DKIM001; t=1652294879; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=P/oRS0MTKb2PkQ/83CFeucuXOajJ7ouPhbdJ5iLoSno=; b=HtHzP10bvReq/JbaPmciY7y+9EhdmcxnDwedr0D2LDxUwvWUmc7YaOPdc9o1GxvdLnZC+L OsDBXbcM0KSXSdwPQcVWc+7D6+VEC0E4IwD0WhfpoFsmamDRqv7P3N1quyzqUkL6f8VrmI cWyqi9ejui3t/Yv7e0DCOP0gkrmy57BHxdi66YquooUkc/PqHA/TLVOejSzTlgl0SO7qlY mhLpZ7DbRhsPoJmp9l1Bo87BnfnPdS8TIe4SH1tjV/1bpi7exGtixZIoSILbGIMAtu+Pdq mxnU6mHbbgs18l5I1xU1bskctvOG2e5ul4ZcRr5GhQFisICpAcF+gVsgHR3O0Q== Received: from hermann (dynamic-2a01-0c23-5d0c-3f00-0006-38a3-27c0-2d03.c23.pool.telefonica.de [IPv6:2a01:c23:5d0c:3f00:6:38a3:27c0:2d03]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by hub2.goneo.de (Postfix) with ESMTPSA id 77A2010A1E89; Wed, 11 May 2022 20:47:58 +0200 (CEST) Date: Wed, 11 May 2022 20:47:55 +0200 From: FreeBSD User To: freebsd-jail@freebsd.org, freebsd-net@freebsd.org Subject: Re: FreeBSD 12.3-p5: problems vnet on if_bridge Message-ID: <20220511204755.2028dce9@hermann> In-Reply-To: <20220510212129.35041f02@hermann> References: <20220510212129.35041f02@hermann> List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-UID: 7c8d87 X-Rspamd-UID: d44854 X-Rspamd-Queue-Id: 4Kz3nh142tz4cjB X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=walstatt-de.de header.s=DKIM001 header.b=HtHzP10b; dmarc=none; spf=none (mx1.freebsd.org: domain of freebsd@walstatt-de.de has no SPF policy when checking 85.220.129.31) smtp.mailfrom=freebsd@walstatt-de.de X-Spamd-Result: default: False [-1.52 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[walstatt-de.de:s=DKIM001]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[walstatt-de.de]; NEURAL_SPAM_SHORT(0.38)[0.376]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[walstatt-de.de:+]; RCPT_COUNT_TWO(0.00)[2]; MLMMJ_DEST(0.00)[freebsd-jail,freebsd-net]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; MID_RHS_NOT_FQDN(0.50)[]; ASN(0.00)[asn:25394, ipnet:85.220.128.0/17, country:DE]; RCVD_TLS_ALL(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[85.220.129.31:from] X-ThisMailContainsUnwantedMimeParts: N On Tue, 10 May 2022 21:21:29 +0200 FreeBSD User wrote: > Hello, > > I ran into serious trouble setting up a FreeBSD 12.3-RELEASE-p5 host having a second NIC > and vnt jails attached to that second NIC (basically, the host is a recent Xigmanas with > Bastille jails, but the issue also occurs on a vanilla FreeBSD 12.3). > > The host is compromised of two NICs, em0 (management only) and igb0 (service/jails). > Both, the server and the jails as well as the igb0 interface are residing on the same > network, but both NICs are connected to two different ports on a switch, to which we do > not have access (part of the campus infrastructure). > > Both NICs are attached with a IPv4 of the same network, the host is listening on both > NICs for services, i.e. port 22 for ssh. No problem to connect to both(!) addresses via > ssh. igb0 is member of an if_bridge. The box also hosts a bunch of vnet jails, each jail > does have an if_epair created via "jib" and these vnet epairs are members of the bridge, > to which ifb0 is also member. > > Problem: while any service bound to NIC igb0/IPv4 residing on igb0 is accessible > flawlessly, accessing an jail is almost impossible. Pinging a jail does work after a > while the ping initiating host has been waiting, in ery rare situations someone can > access the sshd of the jail, but any access of that kind is highly erratic. From 5 > jails, at most two are responding to pings, the other don't and it is non-deterministic > which host will respond. > > Following some advices found on the web, the following sysctl settings are provided to > if_bridge: > > device if_bridge > net.link.bridge.ipfw: 0 > net.link.bridge.allow_llz_overlap: 0 > net.link.bridge.inherit_mac: 0 > net.link.bridge.log_stp: 0 > net.link.bridge.pfil_local_phys: 0 > net.link.bridge.pfil_member: 0 > net.link.bridge.ipfw_arp: 0 > net.link.bridge.pfil_bridge: 0 > net.link.bridge.pfil_onlyip: 0 > > We do not have access to the switch the box is connected to, so I don't have access to > any logs revealing a problem either to a conceptual misunderstanding of networking of > mine and so a misconfiguration or a probelm with Layer 2 or the switches themselfes. > > I'd like to ask whether someone has a similar setup up and running and could report this > - or give a hint of the problem I possibly made (igb0 is attached to an IPv4 AND is > member of an if_brige on which IPv4 attached vnet jails are residing). > > We have also already setup another "similar" scenarion with the same FreeBSD 12.3-p5 > version and also two NICs, but our "service/jail" NIC is part of a different IPv4 > network and the NIC is attached to a different switch (to which we have full access). > > Thanks in advance, > > O. Hartmann > On FreeBSD 12.3-p5, em0 seems to suffer from a bug regarding hardware chesum support, I see a lot of : [...] Flags [.], cksum 0xe826 (incorrect -> 0x606b), seq 101269476:101270000, ack 5077, win 257, options [nop,nop,TS val 2618589801 ecr 3610923914], length 524 Disabling TXCSUM via "ifconfig em0 -txcsum" renders incorrect -> correct. em0 is: em0@pci0:0:25:0: class=0x020000 card=0x20528086 chip=0x153b8086 rev=0x04 hdr=0x00 vendor = 'Intel Corporation' device = 'Ethernet Connection I217-V' class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xf7d00000, size 131072, enabled bar [14] = type Memory, range 32, base 0xf7d35000, size 4096, enabled bar [18] = type I/O Port, range 32, base 0xf080, size 32, enabled cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message cap 13[e0] = PCI Advanced Features: FLR TP I remember faintly that there was an issue when I used to use FBSD 12 From nobody Sat May 14 13:03:59 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6C75B1ADC8DA for ; Sat, 14 May 2022 13:04:19 +0000 (UTC) (envelope-from dfr@rabson.org) Received: from mail-lj1-x22c.google.com (mail-lj1-x22c.google.com [IPv6:2a00:1450:4864:20::22c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4L0m1Z3V4gz4Zp3 for ; Sat, 14 May 2022 13:04:18 +0000 (UTC) (envelope-from dfr@rabson.org) Received: by mail-lj1-x22c.google.com with SMTP id q130so13198501ljb.5 for ; Sat, 14 May 2022 06:04:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rabson-org.20210112.gappssmtp.com; s=20210112; h=mime-version:from:date:message-id:subject:to; bh=LBpzV9BVO7VnSFee4xLOnR9vPwxYaENk5qRSkNUAq00=; b=LKzAq7nzFajakWGk8MZU5dF59AlHmNNSOIPBm5G85MukCoppypCoGALLLS4aSaEELs B6E9bokfvp7wLnFtkNLauY4ViqosEm/4ZCjOlj7rx0PbVcvNHIBwoHc5bsPLmnVbEgh+ qntRbNZSOP0TAo+Ve+0NRo+YkS1BheREFSb3mKQFmxyLd1tCo86bfjknKc6+zlh6IuMN NLqEwxxnnTwiUk3zH0GGCyHvAzpDTVNOMKzVJJsd8hUCxvIMWOnQ1gINwNo2g8gcQG5e 71y0u7TLldfRrMgxE3uZC8Uw4lIs1Q6ZF1q7b3y92o+cPGscT7v9Dyj2jjN4q9He6l1o o15Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=LBpzV9BVO7VnSFee4xLOnR9vPwxYaENk5qRSkNUAq00=; b=UnYcIPjfqZBR2B2BsOB7dcDd9DfedeHTu/5FR0PJc/1u876XYrkOCHNhjKv/jLpTWD 2ryaItW3mvTxoGojPuc7if/oVs870NOmq+0Lnj6qjm49y424tHXPQOllpgyC8FhQfBPW H+TroTss9lJHf+w7usqHJr7nI/68UgVqxdTKy3oYnZ7cMAUfq4s2eY3YsQAtEhoIj7/Y pmtcEjxyZmaxTQHggmWb2jc+W5Ws5ts7KZNKN6guaSaTUXQA+zhoGJF3+7v09uv4wHoQ BgxgFXh/D15dx+KjRwDICv4VvgNoA901trM/8p3U712u6CkrXw5IukettHi2lKjMR+Jp h17g== X-Gm-Message-State: AOAM532kTIj6EYLFz1yoehmSW9O8ncAWKSd1F3Xztm/DZZTs1jJUolVe lj/nGRZoe+S/NkkwR6Gm+BxFWav8NrcuQufxtsRlFgPFhesF2NjQ X-Google-Smtp-Source: ABdhPJxK3VHwlvU6ec69U++k5eFNRXbpW+cmXseDR4uVOj1QhJQi5SSa05sA/NmBJ80PC44T0XCSXV8mIQTFx8s/af4= X-Received: by 2002:a2e:aa27:0:b0:250:9109:2e7f with SMTP id bf39-20020a2eaa27000000b0025091092e7fmr5714816ljb.134.1652533450904; Sat, 14 May 2022 06:04:10 -0700 (PDT) List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 From: Doug Rabson Date: Sat, 14 May 2022 14:03:59 +0100 Message-ID: Subject: FreeBSD containers with podman and buildah To: freebsd-jail@freebsd.org Content-Type: multipart/alternative; boundary="00000000000052279d05def86d88" X-Rspamd-Queue-Id: 4L0m1Z3V4gz4Zp3 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=rabson-org.20210112.gappssmtp.com header.s=20210112 header.b=LKzAq7nz; dmarc=none; spf=pass (mx1.freebsd.org: domain of dfr@rabson.org designates 2a00:1450:4864:20::22c as permitted sender) smtp.mailfrom=dfr@rabson.org X-Spamd-Result: default: False [-3.50 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[rabson-org.20210112.gappssmtp.com:s=20210112]; FREEFALL_USER(0.00)[dfr]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-jail@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000]; DMARC_NA(0.00)[rabson.org]; DKIM_TRACE(0.00)[rabson-org.20210112.gappssmtp.com:+]; NEURAL_HAM_SHORT(-1.00)[-0.999]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::22c:from]; MLMMJ_DEST(0.00)[freebsd-jail]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N --00000000000052279d05def86d88 Content-Type: text/plain; charset="UTF-8" Recently I've been working on porting the buildah and podman container tools to FreeBSD. Podman is a drop-in replacement for docker and buildah focuses on the narrower problem of building container images. At this point, there is enough functionality to show that these tools are viable on FreeBSD so I thought I would write a note here about how to install and try out my proof-of-concept. This will pull in source code for buildah and related modules, build everything and install to /usr/local. Be aware that if you have sysutils/runj installed, it will be overwritten with a modified version. This all happens in a directory named 'build' which can be deleted to clean up or to force a clean build: mkdir -p build fetch https://gist.github.com/dfr/ac4dc043ee3780b690c5887a61f53494/raw/11474779a16bdff1ca31c94437ddb25a8f1f364b/buildah-install.sh chmod +x buildah-install.sh (cd build && ../buildah-install.sh) Make a container and run things inside it: c=$(sudo buildah from docker.io/kwiat/freebsd:13.0-RELEASE) sudo buildah run $c freebsd-version sudo buildah run $c ifconfig sudo buildah rm $c Download and run images in podman: sudo podman run --rm docker.io/dougrabson/hello The containers will use the default 'podman' network which is defined in /usr/local/etc/cni/net.d/87-podman-bridge.conflist. This relies on NAT to allow the container traffic out to the internet and I use pf with the following simple pf.conf: nat on egress inet from to any -> (egress) nat on egress inet6 from to !ff00::/8 -> (egress) rdr-anchor "cni-rdr/*" table Note: I'm using the OpenBSD convention to identify the host's main interface by putting it into the 'egress' group using ifconfig, e.g.: sudo ifconfig vtnet0 group egress There is a lot of room for improvement in this area - NAT works fairly well for ipv4 but can get confused with ipv6 if the egress interface has non-routable addresses assigned to it. Port mapping is very limited and does not work for connections from localhost. Perhaps someone with better pf skills can help figure out how to get this working (probably needs to NAT from localhost back to the container network). --00000000000052279d05def86d88 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Recently I've been working on porting the buildah= and podman container tools to FreeBSD. Podman is a drop-in replacement for= docker and buildah=C2=A0focuses on the narrower problem of building contai= ner=C2=A0images. At this point, there is enough functionality=C2=A0to show = that these tools are viable on=C2=A0FreeBSD so I thought I would write a no= te here about=C2=A0how to install and try out my proof-of-concept.

This will pull in source code for buildah and related modules, = build everything and install to /usr/local. Be aware that if you have sysut= ils/runj installed, it will be overwritten with a modified version. This al= l happens in a directory named 'build' which can be deleted to clea= n up or to force a clean build:

mkdir -p build
fetc= h https://gis= t.github.com/dfr/ac4dc043ee3780b690c5887a61f53494/raw/11474779a16bdff1ca31c= 94437ddb25a8f1f364b/buildah-install.sh
chmod +x buildah-install.sh(cd build && ../buildah-install.sh)

Make a= container and run things inside it:

c=3D$(sudo builda= h from docker.io/kw= iat/freebsd:13.0-RELEASE)
sudo buildah run $c freebsd-version
sud= o buildah run $c ifconfig
sudo buildah rm $c

Down= load and run images in podman:

sudo podman run --rm docker.io/dougrabson/hello<= /font>

The containers will use the default 'podman'= network which is defined in /usr/local/etc/cni/ne= t.d/87-podman-bridge.conflist. This relies on NAT to allow the conta= iner traffic out to the internet and I use pf with the following simple pf.conf:

nat on egress in= et from <cni-nat> to any -> (egress)
nat on egress inet6 from &= lt;cni-nat> to !ff00::/8 -> (egress)
rdr-anchor "cni-rdr/*&qu= ot;
table <cni-nat>

Note: I'm using the= OpenBSD convention to identify the host's main interface by putting it= into the 'egress' group using ifconfig, e.g.:

sudo ifconfig vtnet0 group egress

There is a lot of= room for improvement in this area - NAT works fairly well for ipv4 but can= get confused with ipv6 if the egress interface has non-routable addresses = assigned to it. Port mapping is very limited and does not work for connecti= ons from localhost. Perhaps someone with better pf skills can help figure o= ut how to get this working (probably needs to NAT from localhost back to th= e container network).
--00000000000052279d05def86d88-- From nobody Sat May 14 15:11:12 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id CCCEA1ACD165 for ; Sat, 14 May 2022 15:11:31 +0000 (UTC) (envelope-from cneirabustos@gmail.com) Received: from mail-lf1-x12f.google.com (mail-lf1-x12f.google.com [IPv6:2a00:1450:4864:20::12f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4L0prM11QJz4stb for ; Sat, 14 May 2022 15:11:31 +0000 (UTC) (envelope-from cneirabustos@gmail.com) Received: by mail-lf1-x12f.google.com with SMTP id c24so9603508lfv.11 for ; Sat, 14 May 2022 08:11:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=qT5dA0Al0O7ZdaFERemYImo4CBpV6TV61SYCRJ3AlVg=; b=efqemH2mB15cHI/Ak0g61hNUGA38idf9sigKUjsjGhznKeZvjTYEDB943wIFzV9mO4 8GneSNJ6SNUkj8t9CE+/eCQmVDxoBh5Otx9tT7hmy4rRAv+8kCFx3uPccdDW5ez7yZIu lhk99zpQtoIRTj7nbG0CK+Rt1lgwC2Nd4STDlLYXqvgqOiHKtmG7l350w1h0yyFG/JOn Dz576HTxBvfdp43LMIHf7zCR2cHedwTjmbRkiMMDFtJ0eK/oecfDfARDJbT4iR5jo12U BAjBd9zBtslxcSnrYJdW2KR/BFP/QhhCjN2NgGtC06vPFGzuKboj7uHwpZoxaqi3Oh1E CHKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=qT5dA0Al0O7ZdaFERemYImo4CBpV6TV61SYCRJ3AlVg=; b=AuREkZqx3pabRR/WPR1f0r1Attbn1ey9Cxe6IaFpi9GlWD3TANLdIp6y29LYTjbLaq zUpUvBZAXkx8y3vK6/BREa63kbCyF1w4eo3sYBpK9iiyrnwSY5wZB/VrU8v49bdYEj2h U2+NhqrP2UuUBGzcVqpLX8TYYQji3JYqoEid8rjVDjgx0siPsE3g2K5Wl5wPEp2OYqLo aZvhZ4r12IzJucIgqCY6p8WlmP12oZU3T3Cs0jx1EHcWJ9XYEZTJ5nLFG9aidxVcxmxH QY7jY82Njezlpl1JnUzkmhTeinXeQI4VKBjMaKV3bd5s4MxWhN7Ysf/A7m1eLaclkA49 bwrA== X-Gm-Message-State: AOAM532xTHP3eC5Ho9rgdAPpXG7AVhkChK3L6HXnveej+TxMn2YMT08l 3qpXtenKzdcBQbvud0BXFOQUoBg/BanwcUTg5ZGPgBggb7E= X-Google-Smtp-Source: ABdhPJyXVP/ceHY+rFj2e7Lg5OxHvSpj0qEAMY0kD10PsKnosAkcc1ziQEMyBwYdg3hKHB0p3bssjFU1wELhcKlyUrI= X-Received: by 2002:a19:fc1d:0:b0:473:dc7c:d012 with SMTP id a29-20020a19fc1d000000b00473dc7cd012mr6676263lfi.92.1652541083546; Sat, 14 May 2022 08:11:23 -0700 (PDT) List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: carlos antonio neira bustos Date: Sat, 14 May 2022 11:11:12 -0400 Message-ID: Subject: Re: FreeBSD containers with podman and buildah To: Doug Rabson Cc: freebsd-jail@freebsd.org Content-Type: multipart/alternative; boundary="00000000000042f8e605defa3403" X-Rspamd-Queue-Id: 4L0prM11QJz4stb X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=efqemH2m; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of cneirabustos@gmail.com designates 2a00:1450:4864:20::12f as permitted sender) smtp.mailfrom=cneirabustos@gmail.com X-Spamd-Result: default: False [-3.98 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-jail@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::12f:from]; MLMMJ_DEST(0.00)[freebsd-jail]; NEURAL_HAM_SHORT(-0.98)[-0.981]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-ThisMailContainsUnwantedMimeParts: N --00000000000042f8e605defa3403 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable This is great!. El s=C3=A1b, 14 may 2022 a la(s) 09:04, Doug Rabson (dfr@rabson.org) escrib= i=C3=B3: > Recently I've been working on porting the buildah and podman container > tools to FreeBSD. Podman is a drop-in replacement for docker and > buildah focuses on the narrower problem of building container images. At > this point, there is enough functionality to show that these tools are > viable on FreeBSD so I thought I would write a note here about how to > install and try out my proof-of-concept. > > This will pull in source code for buildah and related modules, build > everything and install to /usr/local. Be aware that if you have > sysutils/runj installed, it will be overwritten with a modified version. > This all happens in a directory named 'build' which can be deleted to cle= an > up or to force a clean build: > > mkdir -p build > fetch > https://gist.github.com/dfr/ac4dc043ee3780b690c5887a61f53494/raw/11474779= a16bdff1ca31c94437ddb25a8f1f364b/buildah-install.sh > chmod +x buildah-install.sh > (cd build && ../buildah-install.sh) > > > Make a container and run things inside it: > > c=3D$(sudo buildah from docker.io/kwiat/freebsd:13.0-RELEASE) > sudo buildah run $c freebsd-version > sudo buildah run $c ifconfig > sudo buildah rm $c > > > Download and run images in podman: > > sudo podman run --rm docker.io/dougrabson/hello > > > The containers will use the default 'podman' network which is defined in > /usr/local/etc/cni/net.d/87-podman-bridge.conflist. This relies on NAT to > allow the container traffic out to the internet and I use pf with the > following simple pf.conf: > > nat on egress inet from to any -> (egress) > nat on egress inet6 from to !ff00::/8 -> (egress) > rdr-anchor "cni-rdr/*" > table > > > Note: I'm using the OpenBSD convention to identify the host's main > interface by putting it into the 'egress' group using ifconfig, e.g.: > > sudo ifconfig vtnet0 group egress > > > There is a lot of room for improvement in this area - NAT works fairly > well for ipv4 but can get confused with ipv6 if the egress interface has > non-routable addresses assigned to it. Port mapping is very limited and > does not work for connections from localhost. Perhaps someone with better > pf skills can help figure out how to get this working (probably needs to > NAT from localhost back to the container network). > --00000000000042f8e605defa3403 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
This is great!.


=
El s=C3=A1b, 14 may 2022 a la(s) 09:0= 4, Doug Rabson (dfr@rabson.org) escri= bi=C3=B3:
Recently I've= been working on porting the buildah and podman container tools to FreeBSD.= Podman is a drop-in replacement for docker and buildah=C2=A0focuses on the= narrower problem of building container=C2=A0images. At this point, there i= s enough functionality=C2=A0to show that these tools are viable on=C2=A0Fre= eBSD so I thought I would write a note here about=C2=A0how to install and t= ry out my proof-of-concept.

This will pull in source co= de for buildah and related modules, build everything and install to /usr/lo= cal. Be aware that if you have sysutils/runj installed, it will be overwrit= ten with a modified version. This all happens in a directory named 'bui= ld' which can be deleted to clean up or to force a clean build:

=
mkdir -p build
fetch https://gist.github.com/df= r/ac4dc043ee3780b690c5887a61f53494/raw/11474779a16bdff1ca31c94437ddb25a8f1f= 364b/buildah-install.sh
chmod +x buildah-install.sh
(cd build &am= p;& ../buildah-install.sh)

Make a container and = run things inside it:

c=3D$(sudo buildah from doc= ker.io/kwiat/freebsd:13.0-RELEASE)
sudo buildah run $c freebsd-versi= on
sudo buildah run $c ifconfig
sudo buildah rm $c
Download and run images in podman:

sudo pod= man run --rm docker.io/dougrabson/hello

The containers will= use the default 'podman' network which is defined in /usr/local/etc/cni/net.d/87-podman-bridge.conflist. This= relies on NAT to allow the container traffic out to the internet and I use= pf with the following simple pf.conf:
<= br>
nat on egress inet from <cni-nat> to any ->= (egress)
nat on egress inet6 from <cni-nat> to !ff00::/8 -> (e= gress)
rdr-anchor "cni-rdr/*"
table <cni-nat><= /blockquote>
Note: I'm using the OpenBSD convention to identify the = host's main interface by putting it into the 'egress' group usi= ng ifconfig, e.g.:

sudo ifconfig vtnet0 group eg= ress

There is a lot of room for improvement in this = area - NAT works fairly well for ipv4 but can get confused with ipv6 if the= egress interface has non-routable addresses assigned to it. Port mapping i= s very limited and does not work for connections from localhost. Perhaps so= meone with better pf skills can help figure out how to get this working (pr= obably needs to NAT from localhost back to the container network).
--00000000000042f8e605defa3403-- From nobody Sun May 15 10:49:06 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id E71081ADD56D for ; Sun, 15 May 2022 10:49:19 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from smtp5.goneo.de (smtp5.goneo.de [85.220.129.30]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4L1JzL5KTnz3qxg; Sun, 15 May 2022 10:49:18 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from hub2.goneo.de (hub2.goneo.de [IPv6:2001:1640:5::8:53]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by smtp5.goneo.de (Postfix) with ESMTPS id F251E10A1E88; Sun, 15 May 2022 12:49:10 +0200 (CEST) Received: from hub2.goneo.de (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by hub2.goneo.de (Postfix) with ESMTPS id 55CC910A32FA; Sun, 15 May 2022 12:49:09 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=walstatt-de.de; s=DKIM001; t=1652611749; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=l9W1k3fmyi+i2clen4SmR5yTkbjh6qx+QJV4x78niT8=; b=rwAJZRKj2Qt6SHg3SuOrKaMdg4l9NcgfF4pbTMkQNM/6uZNUEigm+7phKqKeLSBGozQMsx rAQWld9AlK+IYuPXtl+HkiDWGYt9hq48isaNujm/dp9HyfZZR9gTds5/ZauXA32NprhBLm MSDo1Usjoep8cyfC8Y2QfaEL2nvIaOTUssmdFSmL+e7KV974s49U8a/qsgAU/gejSUefpN BPJt628ZAzgi79ybTUidhXoFJoUNdVmdvmLVJqjDKklO7HvnzexhvloVpGZA8veziv+gbk 6tDla0PuKfZi7DCEtwH4uvQlnj4mKjBtW42/qk9Pl7oE5NvOKn7jFFKaMvA4Ug== Received: from hermann (dynamic-2a01-0c23-64b5-f700-711a-9937-bc6c-d4d8.c23.pool.telefonica.de [IPv6:2a01:c23:64b5:f700:711a:9937:bc6c:d4d8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by hub2.goneo.de (Postfix) with ESMTPSA id 1C0A710A3317; Sun, 15 May 2022 12:49:09 +0200 (CEST) Date: Sun, 15 May 2022 12:49:06 +0200 From: FreeBSD User To: Alexander Leidinger Cc: security@freebsd.org, jail@freebsd.org Subject: Re: Auto-jailing of services - 2nd implementation Message-ID: <20220515124900.44aac19b@hermann> In-Reply-To: <20220403214842.Horde.vlwSVh0KOZ6sL7aDfgA9KKL@webmail.leidinger.net> References: <20220403214842.Horde.vlwSVh0KOZ6sL7aDfgA9KKL@webmail.leidinger.net> List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-UID: 1c959e X-Rspamd-UID: c95153 X-Rspamd-Queue-Id: 4L1JzL5KTnz3qxg X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=walstatt-de.de header.s=DKIM001 header.b=rwAJZRKj; dmarc=none; spf=none (mx1.freebsd.org: domain of freebsd@walstatt-de.de has no SPF policy when checking 85.220.129.30) smtp.mailfrom=freebsd@walstatt-de.de X-Spamd-Result: default: False [0.07 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[walstatt-de.de:s=DKIM001]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[walstatt-de.de]; NEURAL_SPAM_MEDIUM(0.04)[0.040]; NEURAL_SPAM_SHORT(0.93)[0.928]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[walstatt-de.de:+]; MLMMJ_DEST(0.00)[jail]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; MID_RHS_NOT_FQDN(0.50)[]; ASN(0.00)[asn:25394, ipnet:85.220.128.0/17, country:DE]; RCVD_TLS_ALL(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[85.220.129.30:from] X-ThisMailContainsUnwantedMimeParts: N On Sun, 03 Apr 2022 21:48:42 +0200 Alexander Leidinger wrote: > Hi, > > attached is a new implementation of service jails (auto-jailing of > services). This one now supports rc command prefixes (e.g. onestart) > and I tested it in nested jails. The benefit of auto-jailing services > is, that you can apply some restrictions to services (and what other > processes it may see). If your service requires access to network but > not sysvipc, and it doesn't run as root, it can be limited to network > access with or without raw sockets, filesystem-permitted files, and > doesn't see other processes on the system. > > For a few services I have added the required "svcj-config" in the > start scripts (e.g. network access for syslog by setting > syslogd_svj_options=net_basic). > > Possible svcj config options for service jails: > + netv4) > + _svcj_cmd_options="ip4=inherit > allow.reserved_ports ${_svcj_cmd_options}" > + ;; > + netv6) > + _svcj_cmd_options="ip6=inherit > allow.reserved_ports ${_svcj_cmd_options}" > + ;; > + net_basic) > + _svcj_cmd_options="ip4=inherit ip6=inherit > allow.reserved_ports ${_svcj_cmd_options}" > + ;; > + net_raw) > + _svcj_cmd_options="allow.raw_sockets > ${_svcj_cmd_options}" > + ;; > + net_all) > + _svcj_cmd_options="allow.socket_af > allow.raw_sockets allow.reserved_ports ip4=inherit ip6=inherit ${_svcj_cmd_options}" > + ;; > + sysvipc) > + _svcj_cmd_options="sysvmsg=inherit > sysvsem=inherit sysvshm=inherit ${_svcj_cmd_options}" > + ;; > + mlock) > + _svcj_cmd_options="allow.mlock > ${_svcj_cmd_options}" > + ;; > + vmm) > + _svcj_cmd_options="allow.vmm > ${_svcj_cmd_options}" > > By setting syslogd_svcj="YES" in rc.conf your syslogd will be started > in a jail which inherits the full filesystem and the ipv4 and ipv6 > addresses of the parent. > > It would be nice if interested people could experiment a little bit > with this, e.g. adding name_svcj_options="X Y" from above and > name_svcj="YES" into rc.conf and see if it works. Note, doing that for > sshd doesn't make sense in the generic case, it wouldn't see your > jails. It may make sense for services. > > Any kind of feedback and tested name_svcj_options submissions welcome... > > Bye, > Alexander. > Hello Alexander Leidinger, is this really interesting feature already part of recent CURRENT rc subsystem or do I have to "patch" CURRENT with the rc script provided by some place first to obtain the functionality you are talking here about? Thanks in advance and kind regards O. Hartmann p.s. would it be possible toput as service with a dedicated network interfacing (say, jailed vnet/vlan, forinstance an asterisk service running on a small router appliance, as we do in our projects?). From nobody Mon May 16 08:25:00 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id D97E01AD406D for ; Mon, 16 May 2022 08:25:33 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from mailgate.Leidinger.net (bastille.leidinger.net [89.238.82.207]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "mailgate.leidinger.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4L1sl13jtHz3hhw; Mon, 16 May 2022 08:25:33 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from outgoing.leidinger.net (p5b165562.dip0.t-ipconnect.de [91.22.85.98]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-256) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "outgoing.leidinger.net", Issuer "R3" (verified OK)) by mailgate.Leidinger.net (Postfix) with ESMTPSA id 1ED2B226B2; Mon, 16 May 2022 10:25:22 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1652689522; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=BVlxF4Gn/96WAC2RWmhaeI4Fk3MuzxY+sRB+73wpByY=; b=uQ+Pfb20Ptd8A79/EfoQyTe1A/vGQAckquQNxXnE6Yf7tZiGQSyZhaislafLV7jbYj0fga SqV+yDhaKMT9oeTU0Fd/Xv3LNZj+xlP3kFy0nCXGQqSseN7XsMlVxl18uwiLjI4I9nCgJD zcj+VCKl3xdwgATg4qi1fLvtJ0jIF8bT4QvIvLul/iUXbjEyYZ9Vu9CKiN4cNsqIP+QT7B HHdDFSKpNtI4q7thYr6KluKBaRGiTb1OhtAhtuRjNBBzEZayzheyA1NekxBg30mht219p0 tYAcU/26UADZVqsvFLs9Du8uIu+49yG1scGEVEqiDSEnUqx4JnfL/6KPK4CeVA== Received: from webmail.leidinger.net (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-256) server-digest SHA256) (Client did not present a certificate) by outgoing.leidinger.net (Postfix) with ESMTPS id 15DA04430; Mon, 16 May 2022 10:25:04 +0200 (CEST) Date: Mon, 16 May 2022 10:25:00 +0200 Message-ID: <20220516102500.Horde.Jmefw9B2HNSietK_UGUuNbn@webmail.leidinger.net> From: Alexander Leidinger To: FreeBSD User Cc: security@freebsd.org, jail@freebsd.org Subject: Re: Auto-jailing of services - 2nd implementation References: <20220403214842.Horde.vlwSVh0KOZ6sL7aDfgA9KKL@webmail.leidinger.net> <20220515124900.44aac19b@hermann> In-Reply-To: <20220515124900.44aac19b@hermann> Accept-Language: de,en Content-Type: multipart/signed; boundary="=_r46YExCcqoKewmYnE1X1ETS"; protocol="application/pgp-signature"; micalg=pgp-sha256 List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 X-Rspamd-Queue-Id: 4L1sl13jtHz3hhw X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N This message is in MIME format and has been PGP signed. --=_r46YExCcqoKewmYnE1X1ETS Content-Type: multipart/mixed; boundary="=_aaHXkfii9da2Qz_EfZwdseY" This message is in MIME format. --=_aaHXkfii9da2Qz_EfZwdseY Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Quoting FreeBSD User (from Sun, 15 May 2022=20=20 12:49:06=20+0200): > On Sun, 03 Apr 2022 21:48:42 +0200 > Alexander Leidinger wrote: > >> Hi, >> >> attached is a new implementation of service jails (auto-jailing of >> services). This one now supports rc command prefixes (e.g. onestart) >> and I tested it in nested jails. The benefit of auto-jailing services >> is, that you can apply some restrictions to services (and what other >> processes it may see). If your service requires access to network but >> not sysvipc, and it doesn't run as root, it can be limited to network >> access with or without raw sockets, filesystem-permitted files, and >> doesn't see other processes on the system. >> >> For a few services I have added the required "svcj-config" in the >> start scripts (e.g. network access for syslog by setting >> syslogd_svj_options=3Dnet_basic). >> >> Possible svcj config options for service jails: >> + netv4) >> + _svcj_cmd_options=3D"ip4=3Dinherit >> allow.reserved_ports ${_svcj_cmd_options}" >> + ;; >> + netv6) >> + _svcj_cmd_options=3D"ip6=3Dinherit >> allow.reserved_ports ${_svcj_cmd_options}" >> + ;; >> + net_basic) >> + _svcj_cmd_options=3D"ip4=3Dinherit ip6=3Dinherit >> allow.reserved_ports ${_svcj_cmd_options}" >> + ;; >> + net_raw) >> + _svcj_cmd_options=3D"allow.raw_sockets >> ${_svcj_cmd_options}" >> + ;; >> + net_all) >> + _svcj_cmd_options=3D"allow.socket_af >> allow.raw_sockets allow.reserved_ports ip4=3Dinherit ip6=3Dinherit=20=20 >>=20${_svcj_cmd_options}" >> + ;; >> + sysvipc) >> + _svcj_cmd_options=3D"sysvmsg=3Dinherit >> sysvsem=3Dinherit sysvshm=3Dinherit ${_svcj_cmd_options}" >> + ;; >> + mlock) >> + _svcj_cmd_options=3D"allow.mlock >> ${_svcj_cmd_options}" >> + ;; >> + vmm) >> + _svcj_cmd_options=3D"allow.vmm >> ${_svcj_cmd_options}" >> >> By setting syslogd_svcj=3D"YES" in rc.conf your syslogd will be started >> in a jail which inherits the full filesystem and the ipv4 and ipv6 >> addresses of the parent. >> >> It would be nice if interested people could experiment a little bit >> with this, e.g. adding name_svcj_options=3D"X Y" from above and >> name_svcj=3D"YES" into rc.conf and see if it works. Note, doing that for >> sshd doesn't make sense in the generic case, it wouldn't see your >> jails. It may make sense for services. >> >> Any kind of feedback and tested name_svcj_options submissions welcome... >> >> Bye, >> Alexander. >> > > Hello Alexander Leidinger, > > is this really interesting feature already part of recent CURRENT rc=20= =20 >=20subsystem or do I No. > have to "patch" CURRENT with the rc script provided by some place=20=20 >=20first to obtain the > functionality you are talking here about? The patch was supposed to be attached to the mail you quoted. A more=20=20 recent=20patch (now with docu in the rc.conf man page) is attached to=20=20 this=20email (rc.subr + service command + man pages + a few services ->=20= =20 "grep=20diff svcjails.diff" for the list of OS services which can enable=20= =20 without=20research about the svcj_options). At least /etc/rc.subr and=20=20 /usr/sbin/service=20need to be patched (respectively a=20=20 buildworld+installworld). >=20Thanks in advance and kind regards > > O. Hartmann > > p.s. would it be possible toput as service with a dedicated network=20=20 >=20interfacing (say, > jailed vnet/vlan, forinstance an asterisk service running on a small=20= =20 >=20router appliance, as > we do in our projects?). This will use the networking of the host. This is really automatic=20=20 stuff,=20no additional network interface (all what the hosts sees is=20=20 also=20available in the service-jail), no dedicated directory/filesystem=20= =20 area=20(as if it runs unjailed). All is used from the host. The=20=20 additional=20security this provides is the limit of what the process is=20= =20 allowed=20to do in the kernel and the namespace isolation. So you could=20= =20 prevent=20sysvipc access. You could restrict it to ipv6 even if ipv4 is=20= =20 configured.=20You wouldn't see processes outside of the service jail=20=20 even=20if running as root. If you want more advanced things, you need to=20= =20 create=20a jail on your own. Parts of what service jails do, could be=20=20 done=20with capabilities (sometimes even with more restrictions), but=20=20 this=20needs support inside the application for capabilities, whereas=20=20 service=20jails do not need changes to the application itself. If you want to put asterisk into one of my service jails, you would=20=20 have=20to set the following in rc.conf: asterisk_enable=3D"YES" asterisk_svcj_options=3D"" asterisk_svcj=3D"YES" The asterisk_svcj_options part is supposed to be included in the rc=20=20 script=20of asterisk (if/once this is committed to FreeBSD), but can be=20= =20 specified=20in rc.conf to override it if needed (or to test things). For=20= =20 asterisk=20I would assume at least asterisk_svcj_options=3D"net_basic". There's also a svcj_all_enable variable, but as long as not all=20=20 services have a correct setting of their svcj_options, this will not=20=20 do=20what you mean. Bye, Alexander. --=20 http://www.Leidinger.net=20Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF --=_aaHXkfii9da2Qz_EfZwdseY Content-Type: text/diff; charset=utf-8; name=svcjails.diff Content-Disposition: attachment; size=16780; filename=svcjails.diff Content-Transfer-Encoding: quoted-printable diff --git a/libexec/rc/rc.d/auditdistd b/libexec/rc/rc.d/auditdistd index 13cb5d5b69d..3218bd35755 100755 --- a/libexec/rc/rc.d/auditdistd +++ b/libexec/rc/rc.d/auditdistd @@ -19,4 +19,7 @@ required_files=3D"/etc/security/${name}.conf" extra_commands=3D"reload" =20 =20load_rc_config $name + +: ${auditdistd_svcj_options:=3D"net_basic"} + run_rc_command "$1" diff --git a/libexec/rc/rc.d/ftpd b/libexec/rc/rc.d/ftpd index dc623ea5943..a04c7ce5ee2 100755 --- a/libexec/rc/rc.d/ftpd +++ b/libexec/rc/rc.d/ftpd @@ -23,4 +23,7 @@ ftpd_prestart() } =20 =20load_rc_config $name + +: ${ftpd_svcj_options:=3D"net_all"} + run_rc_command "$1" diff --git a/libexec/rc/rc.d/inetd b/libexec/rc/rc.d/inetd index aa8ac20aeae..8cf7be5d91e 100755 --- a/libexec/rc/rc.d/inetd +++ b/libexec/rc/rc.d/inetd @@ -18,4 +18,7 @@ required_files=3D"/etc/${name}.conf" extra_commands=3D"reload" =20 =20load_rc_config $name + +: ${inetd_svcj_options:=3D"net_basic"} + run_rc_command "$1" diff --git a/libexec/rc/rc.d/kadmind b/libexec/rc/rc.d/kadmind index 773b2d0e499..1bdd420e415 100755 --- a/libexec/rc/rc.d/kadmind +++ b/libexec/rc/rc.d/kadmind @@ -26,4 +26,7 @@ kadmind_start_precmd() } =20 =20load_rc_config $name + +: ${kadmind_svcj_options:=3D"net_basic"} + run_rc_command "$1" diff --git a/libexec/rc/rc.d/kdc b/libexec/rc/rc.d/kdc index c2747ae08ca..11205d6e092 100755 --- a/libexec/rc/rc.d/kdc +++ b/libexec/rc/rc.d/kdc @@ -26,4 +26,7 @@ kdc_start_precmd() } =20 =20load_rc_config $name + +: ${kdc_svcj_options:=3D"net_basic"} + run_rc_command "$1" diff --git a/libexec/rc/rc.d/kpasswdd b/libexec/rc/rc.d/kpasswdd index a2875bf1515..af7b7a6b9aa 100755 --- a/libexec/rc/rc.d/kpasswdd +++ b/libexec/rc/rc.d/kpasswdd @@ -26,4 +26,7 @@ kpasswdd_start_precmd() } =20 =20load_rc_config $name + +: ${kapsswd_svcj_options:=3D"net_basic"} + run_rc_command "$1" diff --git a/libexec/rc/rc.d/local_unbound b/libexec/rc/rc.d/local_unbound index 19cb9a6c5c0..7436034495f 100755 --- a/libexec/rc/rc.d/local_unbound +++ b/libexec/rc/rc.d/local_unbound @@ -34,6 +34,7 @@ load_rc_config $name : ${local_unbound_anchor:=3D${local_unbound_workdir}/root.key} : ${local_unbound_forwarders:=3D} : ${local_unbound_tls:=3D} +: ${local_unbound_svcj_options:=3D"net_basic"} =20 =20do_as_unbound() { diff --git a/libexec/rc/rc.d/lpd b/libexec/rc/rc.d/lpd index fc8180cb221..725adda9072 100755 --- a/libexec/rc/rc.d/lpd +++ b/libexec/rc/rc.d/lpd @@ -25,4 +25,7 @@ chkprintcap() } =20 =20load_rc_config $name + +: ${lpd_svcj_options:=3D"net_basic"} + run_rc_command "$1" diff --git a/libexec/rc/rc.d/sshd b/libexec/rc/rc.d/sshd index 282f69f8e4c..9c3b5762579 100755 --- a/libexec/rc/rc.d/sshd +++ b/libexec/rc/rc.d/sshd @@ -81,4 +81,11 @@ sshd_precmd() } =20 =20load_rc_config $name + +# sshd in a jail would not see other jails. As such exclude it from +# svcj_all_enable=3D"YES" by setting sshd_svcj to NO. This allows to +# enable it in rc.conf. +: ${sshd_svcj:=3D"NO"} +: ${sshd_svcj_options:=3D"net_basic"} + run_rc_command "$1" diff --git a/libexec/rc/rc.d/syslogd b/libexec/rc/rc.d/syslogd index 2351c086212..95d2b156b88 100755 --- a/libexec/rc/rc.d/syslogd +++ b/libexec/rc/rc.d/syslogd @@ -71,4 +71,7 @@ set_socketlist() echo $_socketargs } load_rc_config $name + +: ${syslogd_svcj_options:=3D"net_basic"} + run_rc_command "$1" diff --git a/libexec/rc/rc.subr b/libexec/rc/rc.subr index dc4f49612c2..356fb0fea61 100644 --- a/libexec/rc/rc.subr +++ b/libexec/rc/rc.subr @@ -51,6 +51,9 @@ PROTECT=3D"/usr/bin/protect" ID=3D"/usr/bin/id" IDCMD=3D"if [ -x $ID ]; then $ID -un; fi" PS=3D"/bin/ps -ww" +SERVICE=3D/usr/sbin/service +JAIL_CMD=3D/usr/sbin/jail +_svcj_generic_params=3D"path=3D/ mount.nodevfs host=3Dinherit" JID=3D0 # rc_service provides the path to the service script that we are executing= . # This is not being set here in an execution context, necessarily, so it's @@ -368,6 +371,16 @@ _find_processes() $_procname|$_procnamebn|${_procnamebn}:|"(${_procnamebn})"|"[${_proc= namebn}]")' fi =20 +=09if checkyesno ${name}_svcj; then + JID=3D$(/usr/sbin/jls -j svcj-${name} jid) + + case ${JID} in + ''|*[!0-9]*) + # svj-jail doesn't exist, fallback to host-check + JID=3D0 + ;; + esac + fi _proccheck=3D"\ $PS 2>/dev/null -o pid=3D -o jid=3D -o command=3D $_psargs"' | while read _npid _jid '"$_fp_args"'; do @@ -959,6 +972,18 @@ run_rc_command() _pidcmd=3D _procname=3D${procname:-${command}} =20 +=09# If a specifc jail has a specific svcj request, honor it (YES/NO). + # If not (variable empty), evaluate the global svcj catch-call. + # A global YES can be overriden by a specific NO, and a global NO is over= riden + # by a specific YES. + eval _svcj=3D\$${name}_svcj + if [ -z "$_svcj" ]; then + _svcj=3D${svcj_all_enable} + if [ -z "$_svcj" ]; then + eval ${name}_svcj=3DNO + fi + fi + # setup pid check command if [ -n "$_procname" ]; then if [ -n "$pidfile" ]; then @@ -994,7 +1019,7 @@ run_rc_command() _fib=3D\$${name}_fib _env=3D\$${name}_env \ _prepend=3D\$${name}_prepend _login_class=3D\${${name}_login_class:-d= aemon} \ _limits=3D\$${name}_limits _oomprotect=3D\$${name}_oomprotect \ - _env_file=3D\$${name}_env_file + _env_file=3D\$${name}_env_file _svcj_options=3D\$${name}_svcj_options =20 =20 if [ -n "$_env_file" ] && [ -r "${_env_file}" ]; then # load env from f= ile set -a @@ -1008,6 +1033,42 @@ run_rc_command() fi fi =20 +=09if [ -n "$_svcj_options" ]; then # translate service jail options + _svcj_cmd_options=3D"" + + for _svcj_option in $_svcj_options; do + case "$_svcj_option" in + netv4) + _svcj_cmd_options=3D"ip4=3Dinherit allow.reserved_ports ${_svcj_cmd_o= ptions}" + ;; + netv6) + _svcj_cmd_options=3D"ip6=3Dinherit allow.reserved_ports ${_svcj_cmd_o= ptions}" + ;; + net_basic) + _svcj_cmd_options=3D"ip4=3Dinherit ip6=3Dinherit allow.reserved_ports= ${_svcj_cmd_options}" + ;; + net_raw) + _svcj_cmd_options=3D"allow.raw_sockets ${_svcj_cmd_options}" + ;; + net_all) + _svcj_cmd_options=3D"allow.socket_af allow.raw_sockets allow.reserved= _ports ip4=3Dinherit ip6=3Dinherit ${_svcj_cmd_options}" + ;; + sysvipc) + _svcj_cmd_options=3D"sysvmsg=3Dinherit sysvsem=3Dinherit sysvshm=3Din= herit ${_svcj_cmd_options}" + ;; + mlock) + _svcj_cmd_options=3D"allow.mlock ${_svcj_cmd_options}" + ;; + vmm) + _svcj_cmd_options=3D"allow.vmm ${_svcj_cmd_options}" + ;; + *) + echo ${name}: unknown service jail option: $_svcj_option + ;; + esac + done + fi + [ -z "$autoboot" ] && eval $_pidcmd # determine the pid if necessary =20 =20 for _elem in $_keywords; do @@ -1053,9 +1114,50 @@ run_rc_command() if [ -n "$_env" ]; then eval "export -- $_env" fi - _run_rc_precmd || return 1 - _run_rc_doit "$_cmd $rc_extra_args" || return 1 - _run_rc_postcmd + + if [ "${_rc_svcj}" !=3D jailing ]; then + _run_rc_precmd || return 1 + fi + if ! checkyesno ${name}_svcj; then + _run_rc_doit "$_cmd $rc_extra_args" || return 1 + else + case "$rc_arg" in + start) + if [ "${_rc_svcj}" !=3D jailing ]; then + _return=3D1 + $JAIL_CMD -c $_svcj_generic_params $_svcj_cmd_options \ + exec.start=3D"${SERVICE} -E _rc_svcj=3Djailing ${name} ${_rc_pre= fix}start $rc_extra_args" \ + exec.stop=3D"${SERVICE} -E _rc_svcj=3Djailing ${name} ${_rc_pref= ix}stop $rc_extra_args" \ + exec.consolelog=3D"/var/log/svcj_${name}_console.log" \ + name=3Dsvcj-${name} && _return=3D0 + else + _run_rc_doit "$_cmd $rc_extra_args" || _return=3D1 + fi + ;; + stop) + if [ "${_rc_svcj}" !=3D jailing ]; then + $SERVICE -E _rc_svcj=3Djailing -j svcj-${name} ${name} ${_rc_prefix}= stop $rc_extra_args || _return=3D1 + $JAIL_CMD -r svcj-${name} 2>/dev/null + else + _run_rc_doit "$_cmd $rc_extra_args" || _return=3D1 + fi + ;; + restart|status) ;; # no special case needed for svcj or handled somewh= ere else + *) +if checkyesno ${name}_svcj; then +echo XXX: check if \"$rc_arg\" needs to be executed in the jail or outside +fi +# if [ "${_rc_svcj}" !=3D jailing ]; then +# $SERVICE -j svcj-${name} ${name} ${_rc_prefix}${rc_arg} $rc_extra_a= rgs || _return=3D1 +# else + _run_rc_doit "$_cmd $rc_extra_args" || _return=3D1 +# fi + ;; + esac + fi + if [ "${_rc_svcj}" !=3D jailing ];=20then + _run_rc_postcmd + fi return $_return fi =20 @@=20-1113,9 +1215,21 @@ run_rc_command() return 1 fi =20 -=09 if ! _run_rc_precmd; then - warn "failed precmd routine for ${name}" - return 1 + if [ "${_rc_svcj}" !=3D jailing ]; then + if ! _run_rc_precmd; then + warn "failed precmd routine for ${name}" + return 1 + fi + fi + + if checkyesno ${name}_svcj; then + if [ "${_rc_svcj}" !=3D jailing ]; then + $JAIL_CMD -c $_svcj_generic_params $_svcj_cmd_options\ + exec.start=3D"${SERVICE} -E _rc_svcj=3Djailing ${name} ${_rc_pref= ix}start $rc_extra_args" \ + exec.stop=3D"${SERVICE} -E _rc_svcj=3Djailing ${name} ${_rc_prefi= x}stop $rc_extra_args" \ + exec.consolelog=3D"/var/log/svcj_${name}_console.log" \ + name=3Dsvcj-${name} || return 1 + fi fi =20 =20 # setup the full command to run @@ -1152,16 +1266,28 @@ $command $rc_flags $command_args" # Prepend default limits _doit=3D"$_cd limits -C $_login_class $_limits $_doit" =20 + +=09 local _really_run_it=3Dtrue + if checkyesno ${name}_svcj; then + if [ "${_rc_svcj}" !=3D jailing ]; then + _really_run_it=3Dfalse + fi + fi + + if [ "$_really_run_it" =3D true ]; then # run the full command # - if ! _run_rc_doit "$_doit"; then - warn "failed to start ${name}" - return 1 + if ! _run_rc_doit "$_doit"; then + warn "failed to start ${name}" + return 1 + fi fi =20 +=09 if [ "${_rc_svcj}" !=3D jailing ]; then # finally, run postcmd # - _run_rc_postcmd + _run_rc_postcmd + fi ;; =20 =20 stop) @@ -1183,6 +1309,11 @@ $command $rc_flags $command_args" # and run postcmd. wait_for_pids $rc_pid =20 +=09 if checkyesno ${name}_svcj; then + # remove service jail + $JAIL_CMD -r svcj-${name} 2>/dev/null + fi + _run_rc_postcmd ;; =20 @@=20-1211,6 +1342,7 @@ $command $rc_flags $command_args" =20 =20 _run_rc_precmd || return 1 =20 + =20 # run those in a subshell to keep global variables ( run_rc_command ${_rc_prefix}stop $rc_extra_args ) ( run_rc_command ${_rc_prefix}start $rc_extra_args ) diff --git a/share/man/man5/rc.conf.5 b/share/man/man5/rc.conf.5 index 01b09b1a59b..320e0c40765 100644 --- a/share/man/man5/rc.conf.5 +++ b/share/man/man5/rc.conf.5 @@ -239,6 +239,19 @@ such as PostgreSQL will not inherit the OOM killer pro= tection. .It Ao Ar name Ac Ns Va _user .Pq Vt str Run the service under this user account. +.It Ao Ar name Ac Ns Va _svcj +.Pq Vt bool +If set to +.Dq Li YES , +auto-jail the service with inherited filesystem and other +jail properties depending on +.Ao Ar name Ac Ns Va _svcj_options . +.It Ao Ar name Ac Ns Va _svcj_options +.Pq Vt str +A list of jail properties for the service. +See +.Sx SERVICE JAILS +for a list of valid properties. .It Va apm_enable .Pq Vt bool If set to @@ -372,6 +385,12 @@ is set to these are the flags to pass to the .Xr powerd 8 daemon. +.It Va svcj_all_enable +Enable auto-jailing of all services which are not explicitely +excluded. +See +.Sx SERVICE JAILS +for more info. .It Va tmpmfs Controls the creation of a .Pa /tmp @@ -4666,6 +4685,94 @@ Define the total number of seconds to wait for the z= fskeys script to unlock an encrypted dataset. The default is 10. .El +.Sh SERVICE JAILS +The service jails part of the rc system automatically puts a service +into a jail. +This jail inherits the filesystem and various other parts of the +parent (if you allow child-jails in your jails, service jails +can be used in jails) depending on the content of the +.Ao Ar name Ac Ns Va _svcj_options +variable. +Typically this variable is set inside rc scripts, but it can be +overriden in the rc config. +Valid options for +.Ao Ar name Ac Ns Va _svcj_options +are: +.Bl -tag -width indent-two +.It netv4 +Inherit the IPv4 address and allows to open reserved ports. +This can not be combined with +.Pa netv6 . +.It netv6 +Inherit the IPv6 address and allows to open reserved ports. +This can not be combined with +.Pa netv4 . +.It net_basic +Inherits the IPv4 and IPv6 addresses and allows to open +reserved ports. +.It net_raw +Allow to open raw sockets. This option can be combined with +.Pa netv4 , +.Pa netv6 , +.Pa net_basic . +.It net_all +Inherits the IPv4 and IPv6 addresses, allows to open reserved +ports, allows to open raw sockets, and allows to open sockets +of protocol stacks that have not had jail functionality added +to them. +.It sysvipc +Allows access to SysV semaphores, SysV shared memory and +SysV messages. +.It mlock +Allows to lock memory pages into the physical memory. +.It vmm +Allows access to +.Xr vmm 4 . +This option is only available when +.Xr vmm 4 +is enabled in the kernel. +.El + +All non-network options can be combined with all other options. + +If the +.Ao Ar name Ac Ns Va _svcj +variable is set to +.Dq Li YES , +this particular service is started in a +service jail named +.Va svcj- Ns Ar name Ac . + +The +.Va svcj_all_enable +variable allows to enable service jails for all services of the +system at once. +Services which have +.Ao Ar name Ac Ns Va _svcj +set to +.Dq Li NO +are excluded. +Some services may set +.Ao Ar name Ac Ns Va _svcj +to +.Dq Li NO +in the script to either prevent service jails for this +service at all, or may set it to +.Dq Li NO +if it is not set in the +rc config, to exclude it from +.Va svcj_all_enable +but allow to explicitely enable it. +The sshd service for example would not see other jails, if +it would run as a service jail. +This may or may not be what is needed, and as such it is +excluded from +.Va svcj_all_enable +but can be enabled via setting +.Va sshd_svcj +to +.Dq Li YES . +.El .Sh FILES .Bl -tag -width ".Pa /etc/defaults/rc.conf" -compact .It Pa /etc/defaults/rc.conf diff --git a/usr.sbin/service/service.8 b/usr.sbin/service/service.8 index 9902ae3c857..c2be0e0af03 100644 --- a/usr.sbin/service/service.8 +++ b/usr.sbin/service/service.8 @@ -48,6 +48,7 @@ .Nm .Op Fl j Ar jail .Op Fl v +.Op Fl E Ar var=3Dvalue .Ar script .Ar command .Sh DESCRIPTION @@ -67,6 +68,13 @@ the scripts using various criteria. .Pp The options are as follows: .Bl -tag -width F1 +.It Fl E Ar var=3Dvalue +Set the environment variable +.Ar var +to the specified +.Ar value +before starting the script. +This option can be used multiple times. .It Fl e List services that are enabled. The list of scripts to check is compiled using @@ -117,6 +125,9 @@ to which is how they are set in .Pa /etc/rc at boot time. +If the +.Fl E +option is used, the corresponding variable is set accordingly. .Sh EXIT STATUS .Ex -std .Sh EXAMPLES @@ -126,6 +137,7 @@ command: .Bd -literal -offset -ident service named status service -j dns named status +service -E LC_ALL=3DC.UTF-8 named start service -rv .Ed .Pp diff --git a/usr.sbin/service/service.sh b/usr.sbin/service/service.sh index 76cce580c5b..2f86d117fd1 100755 --- a/usr.sbin/service/service.sh +++ b/usr.sbin/service/service.sh @@ -37,10 +37,11 @@ usage () { echo "${0##*/} [-j ] -e" echo "${0##*/} [-j ] -R" echo "${0##*/} [-j ] [-v] -l | -r" - echo "${0##*/} [-j ] [-v] start|stop|etc." + echo "${0##*/} [-j ] [-v] [-E var=3Dvalue] = start|stop|etc." echo "${0##*/} -h" echo '' echo "-j Perform actions within the named jail" + echo "-E n=3Dval Set variable n to val before executing the rc.d script" echo '-e Show services that are enabled' echo "-R Stop and start enabled $local_startup services" echo "-l List all scripts in /etc/rc.d and $local_startup" @@ -49,9 +50,10 @@ usage () { echo '' } =20 -while=20getopts 'j:ehlrRv' COMMAND_LINE_ARGUMENT ; do +while getopts 'j:E:ehlrRv' COMMAND_LINE_ARGUMENT ; do case "${COMMAND_LINE_ARGUMENT}" in j) JAIL=3D"${OPTARG}" ;; + E) VARS=3D"${VARS} ${OPTARG}" ;; e) ENABLED=3Deopt ;; h) usage ; exit 0 ;; l) LIST=3Dlopt ;; @@ -72,6 +74,9 @@ if [ -n "${JAIL}" ]; then [ -n "${RCORDER}" ] && args=3D"${args} -r" [ -n "${RESTART}" ] && args=3D"${args} -R" [ -n "${VERBOSE}" ] && args=3D"${args} -v" + for var in ${VARS}; do + args=3D"${args} -E ${var}" + done =20 =20 # Call jexec(8) with the rebuild args and any=20positional args that # were left in $@ @@ -171,7 +176,7 @@ cd / for dir in /etc/rc.d $local_startup; do if [ -x "$dir/$script" ]; then [ -n "$VERBOSE" ] && echo "$script is located in $dir" - exec env -i -L -/daemon HOME=3D/ PATH=3D/sbin:/bin:/usr/sbin:/usr/bin "$= dir/$script" "$@" + exec env -i -L -/daemon HOME=3D/ PATH=3D/sbin:/bin:/usr/sbin:/usr/bin ${= VARS} "$dir/$script" "$@" fi done =20 --=_aaHXkfii9da2Qz_EfZwdseY-- --=_r46YExCcqoKewmYnE1X1ETS Content-Type: application/pgp-signature Content-Description: Digitale PGP-Signatur Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIzBAABCAAdFiEER9UlYXp1PSd08nWXEg2wmwP42IYFAmKCClsACgkQEg2wmwP4 2IY38Q/9GWrCAw2Sju5yu1BWZCpBUdX5TncddsY1WngYIq9Pv7C3Nx7D6ZbYGQmP 9V3dbtKfodJWSzOW7xYpEPnuaZofIWesQu+vqqneIVdu5usl9CcwelmYYSXqA6cy JDTnhBnbwlJuPlYFGwR5Tiwqva9psVZ7juwkMmztPVP6dpz+wUPUR6UdaGHippd6 fhprPL1+T9/i1ckir1AAcpsaHpypEJVX9uXTHbYWWmWWm9QPkqeAwmQR30yp6rs3 6MYhlGFXudH4l3p7InT5gyBpWArF7AetpKktFh+DKui/Iwqedn1R3raTGdT/2X+7 yt5bzXgA3VMlPgz+TiZfK4PAZz5DpRu4ZOoXmSAJKw7JrWpVWMrgdQX+av3z68He vDlG4pHVKTJ5C5ruiR7fR/6hg9P4V7Zm/f3YIYsdwn/puObPhcEcJLJh7j6u2vx0 4AHDZxsUVkWcXHuaowoo+MYfun86drHVuTJTCAB8osN/68QEJL+Y6eb99/AL1K7V h+vnn18BZENLaUd0zLUuEG1mbeEnZVttU3oDVBJZdWFHA+ASJg4iLy9xJeQJAvNu 1hWC/PaV6IdzMIrYtfJtQijB78AlfkRRi9dMlIaEXRDHKzSR8YWHHInjH1LQCVRZ lL8crilftqR70+q+WxkKlF+SrgJrw8rm04w1rjkP7h+7S58Lk6E= =L6fd -----END PGP SIGNATURE----- --=_r46YExCcqoKewmYnE1X1ETS-- From nobody Wed May 18 14:14:05 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id DD4731AED60C for ; Wed, 18 May 2022 14:14:16 +0000 (UTC) (envelope-from marko.cupac@mimar.rs) Received: from mail.mimar.rs (mail1.mimar.rs [193.53.106.132]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4L3FNR45P2z4gvF for ; Wed, 18 May 2022 14:14:15 +0000 (UTC) (envelope-from marko.cupac@mimar.rs) Received: from mail1.mimar.rs (localhost [127.0.2.132]) by mail.mimar.rs (Postfix) with ESMTP id 7AAAC376A597 for ; Wed, 18 May 2022 16:14:08 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=mimar.rs; h= content-transfer-encoding:content-type:content-type:mime-version :x-mailer:organization:message-id:subject:subject:from:from:date :date:received:received; s=20211119; t=1652883246; x=1654697647; bh=Div/SczvW8/wbvCJuN0fVovkEo9fCjff/22Nv56Qp5M=; b=buHljfLRmmjK F9VciPu1/IVpyvN7ToMrsfZ1xskZGyd43jXpNwi5ydgkAOZtZUr9JCkG8zYm6xFz Job566evh05gaBSY5iQOhfLatD3aSOT3P4aU9dAFL5ZYtPlV98Ibsa0zZh+ZsDU/ NOfw8w6+TTMeI8KAvl6Ysw4vpc8Hcks= X-Virus-Scanned: amavisd-new at mimar.rs Received: from mail.mimar.rs ([127.0.2.132]) by mail1.mimar.rs (mail.mimar.rs [127.0.2.132]) (amavisd-new, port 10026) with LMTP id xmENiG6kK3lj for ; Wed, 18 May 2022 16:14:06 +0200 (CEST) Received: from mephala.kappastar.com (nat-nat.kappastar.com [193.53.106.34]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: marko.cupac) by mail.mimar.rs (Postfix) with ESMTPSA id 20BAA376A596 for ; Wed, 18 May 2022 16:14:06 +0200 (CEST) Date: Wed, 18 May 2022 16:14:05 +0200 From: Marko =?UTF-8?B?Q3VwYcSH?= To: freebsd-jail@freebsd.org Subject: services not starting with jail Message-ID: <20220518161405.1a72b73d@mephala.kappastar.com> Organization: Mimar X-Mailer: Claws Mail 3.18.0 (GTK+ 2.24.33; amd64-portbld-freebsd13.0) List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4L3FNR45P2z4gvF X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=mimar.rs header.s=20211119 header.b=buHljfLR; dmarc=pass (policy=reject) header.from=mimar.rs; spf=pass (mx1.freebsd.org: domain of marko.cupac@mimar.rs designates 193.53.106.132 as permitted sender) smtp.mailfrom=marko.cupac@mimar.rs X-Spamd-Result: default: False [-3.29 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[mimar.rs:s=20211119]; NEURAL_HAM_LONG(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-jail@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; HAS_ORG_HEADER(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DKIM_TRACE(0.00)[mimar.rs:+]; DMARC_POLICY_ALLOW(-0.50)[mimar.rs,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-jail]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; R_MIXED_CHARSET(0.71)[subject]; ASN(0.00)[asn:12823, ipnet:193.53.106.0/24, country:RS]; RCVD_TLS_LAST(0.00)[] X-ThisMailContainsUnwantedMimeParts: N Hi, After upgrade of one of my jails to 13.1-RELEASE, another service won't start with jail by means of record in rc.conf. Services start fine when `service $service start' is executed in ssh session. Services also start fine when `@reboot /usr/sbin/service $service restart' is added to root's crontab. So far I had problem with ejabberd and amavisd-new, which started with 13.0-RELEASE, now grafana joined the club with 13.1-RELEASE. Here are relevant bug reports: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D256453 (ejabberd) https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D256974 (amavisd-new) Port maintainers are mostly puzzled as they can't reproduce the problem, but more people have chimed in with same symptoms. Could it be that the problem is related to jails? Thank you in advance, --=20 Before enlightenment - chop wood, draw water. After enlightenment - chop wood, draw water. Marko Cupa=C4=87 https://www.mimar.rs/ From nobody Tue May 24 10:00:37 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id DF77C1AE8F29; Tue, 24 May 2022 10:00:51 +0000 (UTC) (envelope-from ol@dbconn.net) Received: from mout-b-110.mailbox.org (mout-b-110.mailbox.org [IPv6:2001:67c:2050:102:465::110]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4L6qTH0hgqz3KNc; Tue, 24 May 2022 10:00:51 +0000 (UTC) (envelope-from ol@dbconn.net) Received: from smtp202.mailbox.org (smtp202.mailbox.org [IPv6:2001:67c:2050:b231:465::202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-b-110.mailbox.org (Postfix) with ESMTPS id 4L6qT64Zndz9sRB; Tue, 24 May 2022 12:00:42 +0200 (CEST) Date: Tue, 24 May 2022 12:00:37 +0200 From: Ole Lemke To: FreeBSD User Cc: freebsd-jail@freebsd.org, freebsd-net@freebsd.org Subject: Re: FreeBSD 12.3-p5: problems vnet on if_bridge Message-ID: <20220524120037.46b49baa@lenp43s> In-Reply-To: <20220511204755.2028dce9@hermann> References: <20220510212129.35041f02@hermann> <20220511204755.2028dce9@hermann> List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_//J.G76yaS6QF16AnR+droUK"; protocol="application/pgp-signature"; micalg=pgp-sha512 X-Rspamd-Queue-Id: 4L6qTH0hgqz3KNc X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of ol@dbconn.net designates 2001:67c:2050:102:465::110 as permitted sender) smtp.mailfrom=ol@dbconn.net X-Spamd-Result: default: False [-4.89 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.997]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2001:67c:2050::/48:c]; NEURAL_HAM_LONG(-0.99)[-0.993]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_NA(0.00)[dbconn.net]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-jail,freebsd-net]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:199118, ipnet:2001:67c:2050::/48, country:DE]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N --Sig_//J.G76yaS6QF16AnR+droUK Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Hello, could you solve the problem? I think I ran into the same problem. I opened a Ticket. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D264198 I seems to be related to IPFW and effects vnet-Jails and also bhyve VMs. regards Ole Wed, 11 May 2022 20:47:55 +0200 - FreeBSD User : > On Tue, 10 May 2022 21:21:29 +0200 > FreeBSD User wrote: >=20 > > Hello, > >=20 > > I ran into serious trouble setting up a FreeBSD 12.3-RELEASE-p5 > > host having a second NIC and vnt jails attached to that second NIC > > (basically, the host is a recent Xigmanas with Bastille jails, but > > the issue also occurs on a vanilla FreeBSD 12.3). > >=20 > > The host is compromised of two NICs, em0 (management only) and igb0 > > (service/jails). Both, the server and the jails as well as the igb0 > > interface are residing on the same network, but both NICs are > > connected to two different ports on a switch, to which we do not > > have access (part of the campus infrastructure). > >=20 > > Both NICs are attached with a IPv4 of the same network, the host is > > listening on both NICs for services, i.e. port 22 for ssh. No > > problem to connect to both(!) addresses via ssh. igb0 is member of > > an if_bridge. The box also hosts a bunch of vnet jails, each jail > > does have an if_epair created via "jib" and these vnet epairs are > > members of the bridge, to which ifb0 is also member. > >=20 > > Problem: while any service bound to NIC igb0/IPv4 residing on igb0 > > is accessible flawlessly, accessing an jail is almost impossible. > > Pinging a jail does work after a while the ping initiating host has > > been waiting, in ery rare situations someone can access the sshd of > > the jail, but any access of that kind is highly erratic. From 5 > > jails, at most two are responding to pings, the other don't and it > > is non-deterministic which host will respond.=20 > >=20 > > Following some advices found on the web, the following sysctl > > settings are provided to if_bridge:=20 > >=20 > > device if_bridge > > net.link.bridge.ipfw: 0 > > net.link.bridge.allow_llz_overlap: 0 > > net.link.bridge.inherit_mac: 0 > > net.link.bridge.log_stp: 0 > > net.link.bridge.pfil_local_phys: 0 > > net.link.bridge.pfil_member: 0 > > net.link.bridge.ipfw_arp: 0 > > net.link.bridge.pfil_bridge: 0 > > net.link.bridge.pfil_onlyip: 0 > >=20 > > We do not have access to the switch the box is connected to, so I > > don't have access to any logs revealing a problem either to a > > conceptual misunderstanding of networking of mine and so a > > misconfiguration or a probelm with Layer 2 or the switches > > themselfes. > >=20 > > I'd like to ask whether someone has a similar setup up and running > > and could report this > > - or give a hint of the problem I possibly made (igb0 is attached > > to an IPv4 AND is member of an if_brige on which IPv4 attached vnet > > jails are residing). > >=20 > > We have also already setup another "similar" scenarion with the > > same FreeBSD 12.3-p5 version and also two NICs, but our > > "service/jail" NIC is part of a different IPv4 network and the NIC > > is attached to a different switch (to which we have full access). > >=20 > > Thanks in advance, > >=20 > > O. Hartmann > >=20 >=20 > On FreeBSD 12.3-p5, em0 seems to suffer from a bug regarding hardware > chesum support, I see a lot of : > [...] > Flags [.], cksum 0xe826 (incorrect -> 0x606b), seq > 101269476:101270000, ack 5077, win 257, options [nop,nop,TS val > 2618589801 ecr 3610923914], length 524 >=20 > Disabling TXCSUM via "ifconfig em0 -txcsum" renders incorrect -> > correct. >=20 > em0 is: >=20 > em0@pci0:0:25:0: class=3D0x020000 card=3D0x20528086 > chip=3D0x153b8086 rev=3D0x04 hdr=3D0x00 vendor =3D 'Intel Corporation' > device =3D 'Ethernet Connection I217-V' > class =3D network > subclass =3D ethernet > bar [10] =3D type Memory, range 32, base 0xf7d00000, size 131072, > enabled bar [14] =3D type Memory, range 32, base 0xf7d35000, size > 4096, enabled bar [18] =3D type I/O Port, range 32, base 0xf080, size > 32, enabled cap 01[c8] =3D powerspec 2 supports D0 D3 current D0 > cap 05[d0] =3D MSI supports 1 message, 64 bit enabled with 1 message > cap 13[e0] =3D PCI Advanced Features: FLR TP >=20 >=20 > I remember faintly that there was an issue when I used to use FBSD 12 >=20 >=20 >=20 >=20 --Sig_//J.G76yaS6QF16AnR+droUK Content-Type: application/pgp-signature Content-Description: Digitale Signatur von OpenPGP -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEcfiBcfJXFWtoGXjffPhD3vCz2EwFAmKMrMUACgkQfPhD3vCz 2EzuvQ//R896neYnAL0V0sAXHkXspYDulmKBD1vS4POImoc10ATP9nOfE55jIiOr Uvdwq9cImshcH8FHC3Ozf8QYfrewmXHMX6eZTjSt99qwJggDVDXJXtLOQxMZbL7w 5+suO4gxbmFCcXjeqHen8oxKNgu9wcoFxC2e2p2E/z41oN69iviF8tV+WE9qwWlI 4OsyN0KpttgA5L12dk81pOMnu46nkpZ7fH0QSW77Q/qR58bpjMFCPx5OBvMySdN/ 25en208W/TAOwvTl1W7qFDX6RmEUbSp16CrijlgeRU8kSny9ttIe/sMzxDMo3dCZ 9FD8DpsBkRQnQYKg98aNmC/gzzklQvkoAXi+O/KRUrNkCzJGObfvF6cykusjLYgH vYD7yjUpMqtiSY7OmyWWrNJJvPLX9zQ3qhhWinZw9RetGhe6eWfGJkLJYwRkBaIv eNjaqL+2ru/o7aHZNlioMfLPko5xCoAQr4NfT4eF0GXR433851ngxbsNMsRbzx7g TDEpOYtlUxX/jJEQeurjQc5Ymay8/rFDMIeOM9nuBlBeZkmxxy/wqhhF0qUHPYfH L5Y8SZeaw9QZHZ7zsuMT4bJj/TbokyeHnQ2S4aQjCFCCq56CKFhgyHV4WSxJvwsV 7Ft6UAUOQnhvBB+K7DqHHXXiRWBKo7afkGWtwoHYDf4XJbkpvUo= =EyM7 -----END PGP SIGNATURE----- --Sig_//J.G76yaS6QF16AnR+droUK-- From nobody Tue May 24 13:58:17 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 527221B39238 for ; Tue, 24 May 2022 13:58:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4L6wlF5cZtz4bL4 for ; Tue, 24 May 2022 13:58:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 9003E25AD2 for ; Tue, 24 May 2022 13:58:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 24ODwH9a017287 for ; Tue, 24 May 2022 13:58:17 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 24ODwHdC017286 for jail@FreeBSD.org; Tue, 24 May 2022 13:58:17 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 228351] cannot unhide log or tty with devfs.rules Date: Tue, 24 May 2022 13:58:17 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: conf X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1653400698; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zG+zx1TwYTvD/05P60Jl9ZGLB9E9MQsXPSMC+YVSj+w=; b=gP81iMfCfCEqzJ4A473ns7b3N3OkfZEn8FVR5qjRNIHJSxKpZWebgHuh5STFtaUQdEHdXS M6HPngOBzt8DKwYPl8DFaog4Ds2qr7BIh7Tog/wcGvV92/7+5Qo+izPGK7UgVzNSKmmiZG i7xOzIuczWbRNooOrd73fcZBcl+NjRkxD27pWZZw8FX4DJqfcYXUsxzT7wHBQR/thQezfL z2Z9YK1dmFLQOv+FVDQo6sv4nAbd2MNKjP04fIQdq/9vjzg94wHmAkeNKn9O+fTBw68cww ySyLNePGMtRGUGk4RZ+XRyXnUzYTjBvR2kVgX2YAmcajbVweRBZBSBgJvMAgvA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1653400698; a=rsa-sha256; cv=none; b=Crz3lFOZWtetWFVCKvPKrk0NeK9/XOmEyxbHfMKwtbgReRxwRLXKVTfjCbJ6aElL9ZYJ0n 2B8eQcFBOvFNJ5ca7GBCKGFPEqN8scwcSek3JcevVu4aceqIbIw18eH2WIMzmmes+bckHO M2zKjrFOVzbG5htuP9W9dFZ8wOd9TiPeH/QAGOQtkuUp636gaUkF/tm2p2cCcA7H6qQiiH zXec/K0uw6w6xrS6nbjQ9cU+Hwt/coA2jC1nHndGOH/eOHrLRipHUoTBNoXBJQyQnz1fYQ d3l6k1XvVJqaR5LXeT/W74HYTtTnjU2Ms1WOJ7d/tHGR8tItlBlom5iYX8fN1g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D228351 --- Comment #6 from commit-hook@FreeBSD.org --- A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3D2670ea8a075ea29a0eee9d227c4cdf585= a7b3b55 commit 2670ea8a075ea29a0eee9d227c4cdf585a7b3b55 Author: Zhenlei Huang AuthorDate: 2022-05-24 13:54:38 +0000 Commit: Mark Johnston CommitDate: 2022-05-24 13:54:38 +0000 devfs.rules: Do not expose "log" in the default devfs rules. /etc/rc.d/jail no longer creates /dev/log as a symbolic link since commit 84b354cb9ab61224713c159b1484e8f070fd37be. PR: 228351 Reviewed by: jamie, mark MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D34563 sbin/devfs/devfs.rules | 1 - 1 file changed, 1 deletion(-) --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue May 24 15:17:19 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6ED001B4A09B; Tue, 24 May 2022 15:17:31 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from smtp5.goneo.de (smtp5.goneo.de [IPv6:2001:1640:5::8:30]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4L6yVf2g8Cz4nLH; Tue, 24 May 2022 15:17:30 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from hub2.goneo.de (hub2.goneo.de [85.220.129.53]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by smtp5.goneo.de (Postfix) with ESMTPS id B76C110A1E89; Tue, 24 May 2022 17:17:22 +0200 (CEST) Received: from hub2.goneo.de (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by hub2.goneo.de (Postfix) with ESMTPS id D1C2810A1E87; Tue, 24 May 2022 17:17:20 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=walstatt-de.de; s=DKIM001; t=1653405440; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7yzMHfvFX2VKwEGAWC06tkWV6C/L4V2XntPFhoOGVeg=; b=RlOhl2oSJLaZyFazdRD4EgAxfkYCOw/zQ33x4MWjqsXFkj3LYB1OJADO9041JjdpTILdis g/JgwIvDs+rpwhry1yInZY0cR8DXaeGH/17rTP5+QT15Jm6xrBSoAXMP+U0CYPQg8sdibF KfRU/wzQXCjmB4Mamje5GUjEKqT1XXhFT7Nc108Z0TK3joTb3KqA2iRgeaWm/4LXVCYq65 Dpm3wg+A42W7+XjarYklerJn3qyem3Dm+9fX8hH+FMTDZsSCoWZ7DQVGR1ax1lez8FkHRB 2/2+e3VKdP5QiRlIzIw/AGNxi29ShHnmj1gCutJz42aVR9Z07EcjPGWjEIojGg== Received: from hermann (dynamic-2a01-0c22-ad87-2200-a846-e7ef-d0c1-3159.c22.pool.telefonica.de [IPv6:2a01:c22:ad87:2200:a846:e7ef:d0c1:3159]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by hub2.goneo.de (Postfix) with ESMTPSA id 83A1C10A1E91; Tue, 24 May 2022 17:17:20 +0200 (CEST) Date: Tue, 24 May 2022 17:17:19 +0200 From: FreeBSD User To: Ole Cc: freebsd-jail@freebsd.org, freebsd-net@freebsd.org Subject: Re: FreeBSD 12.3-p5: problems vnet on if_bridge Message-ID: <20220524171654.705f67b2@hermann> In-Reply-To: <20220524115246.0ef1ff59@lenp43s> References: <20220510212129.35041f02@hermann> <20220511204755.2028dce9@hermann> <20220524115246.0ef1ff59@lenp43s> List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-UID: b8a757 X-Rspamd-UID: aca6c1 X-Rspamd-Queue-Id: 4L6yVf2g8Cz4nLH X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=walstatt-de.de header.s=DKIM001 header.b=RlOhl2oS; dmarc=none; spf=none (mx1.freebsd.org: domain of freebsd@walstatt-de.de has no SPF policy when checking 2001:1640:5::8:30) smtp.mailfrom=freebsd@walstatt-de.de X-Spamd-Result: default: False [-2.84 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[walstatt-de.de:s=DKIM001]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.999]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[walstatt-de.de]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[walstatt-de.de:+]; NEURAL_HAM_SHORT(-0.94)[-0.939]; MLMMJ_DEST(0.00)[freebsd-jail,freebsd-net]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; MID_RHS_NOT_FQDN(0.50)[]; ASN(0.00)[asn:25394, ipnet:2001:1640::/32, country:DE]; RCVD_TLS_ALL(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[2001:1640:5::8:30:from] X-ThisMailContainsUnwantedMimeParts: N On Tue, 24 May 2022 09:52:46 +0000 Ole wrote: Hello, > Hello, > > could you solve the problem? I think I ran into the same problem. > I opened a Ticket. I couldn't solve the problem. > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=264198 > > I seems to be related to IPFW and effects vnet-Jails and also bhyve VMs. There is also a PR regarding vnet/if_bridge/routing issues, at https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=240106 but I can not guarantee this PR is in any way similaror adjacent to the problem of mine (and probably yours). For reasons not to be discussed here I'm bound to FreeBSD 12.3-RELENG (XigmaNAS base) and running several boxes of different hardware grades with the very same XigmaNAS version gives different results. The host in question in my case has Intel NICs, the box with a very similar setup and vnet jails also on the same NIC the LAN resides on, has a Realtek NIC (it is a very small home NAS). Another has a dedicated NIC into another network (diffrent IP of the NIC and the vnet jails bound to that NIC compared to the LAN/host itself - Intel i350 NICs all over the place, no problems. Another case, our build platform, is running CURRENT and hosts about almost 15 jails on a NIC, which is part of the same network as the main host's NIC - no problem. All of our FreeBSD boxes uses IPFW as their IP filtering facility. Have you checked the MAC addresses of you vnet jails / vnet hosts? I have had on 12-CURRENT a serious issue with the very same MAC address on all jail-belonging parts of the epair vnet interface. I haven't checked this on our 12.3 (XigmaNAS) installations, I remember this issue as I writet this email, it could be a hint to look after ... > > regards > Ole Kind regarads, O. Hartmann > > > Wed, 11 May 2022 20:47:55 +0200 - FreeBSD User : > > > On Tue, 10 May 2022 21:21:29 +0200 > > FreeBSD User wrote: > > > > > Hello, > > > > > > I ran into serious trouble setting up a FreeBSD 12.3-RELEASE-p5 > > > host having a second NIC and vnt jails attached to that second NIC > > > (basically, the host is a recent Xigmanas with Bastille jails, but > > > the issue also occurs on a vanilla FreeBSD 12.3). > > > > > > The host is compromised of two NICs, em0 (management only) and igb0 > > > (service/jails). Both, the server and the jails as well as the igb0 > > > interface are residing on the same network, but both NICs are > > > connected to two different ports on a switch, to which we do not > > > have access (part of the campus infrastructure). > > > > > > Both NICs are attached with a IPv4 of the same network, the host is > > > listening on both NICs for services, i.e. port 22 for ssh. No > > > problem to connect to both(!) addresses via ssh. igb0 is member of > > > an if_bridge. The box also hosts a bunch of vnet jails, each jail > > > does have an if_epair created via "jib" and these vnet epairs are > > > members of the bridge, to which ifb0 is also member. > > > > > > Problem: while any service bound to NIC igb0/IPv4 residing on igb0 > > > is accessible flawlessly, accessing an jail is almost impossible. > > > Pinging a jail does work after a while the ping initiating host has > > > been waiting, in ery rare situations someone can access the sshd of > > > the jail, but any access of that kind is highly erratic. From 5 > > > jails, at most two are responding to pings, the other don't and it > > > is non-deterministic which host will respond. > > > > > > Following some advices found on the web, the following sysctl > > > settings are provided to if_bridge: > > > > > > device if_bridge > > > net.link.bridge.ipfw: 0 > > > net.link.bridge.allow_llz_overlap: 0 > > > net.link.bridge.inherit_mac: 0 > > > net.link.bridge.log_stp: 0 > > > net.link.bridge.pfil_local_phys: 0 > > > net.link.bridge.pfil_member: 0 > > > net.link.bridge.ipfw_arp: 0 > > > net.link.bridge.pfil_bridge: 0 > > > net.link.bridge.pfil_onlyip: 0 > > > > > > We do not have access to the switch the box is connected to, so I > > > don't have access to any logs revealing a problem either to a > > > conceptual misunderstanding of networking of mine and so a > > > misconfiguration or a probelm with Layer 2 or the switches > > > themselfes. > > > > > > I'd like to ask whether someone has a similar setup up and running > > > and could report this > > > - or give a hint of the problem I possibly made (igb0 is attached > > > to an IPv4 AND is member of an if_brige on which IPv4 attached vnet > > > jails are residing). > > > > > > We have also already setup another "similar" scenarion with the > > > same FreeBSD 12.3-p5 version and also two NICs, but our > > > "service/jail" NIC is part of a different IPv4 network and the NIC > > > is attached to a different switch (to which we have full access). > > > > > > Thanks in advance, > > > > > > O. Hartmann > > > > > > > On FreeBSD 12.3-p5, em0 seems to suffer from a bug regarding hardware > > chesum support, I see a lot of : > > [...] > > Flags [.], cksum 0xe826 (incorrect -> 0x606b), seq > > 101269476:101270000, ack 5077, win 257, options [nop,nop,TS val > > 2618589801 ecr 3610923914], length 524 > > > > Disabling TXCSUM via "ifconfig em0 -txcsum" renders incorrect -> > > correct. > > > > em0 is: > > > > em0@pci0:0:25:0: class=0x020000 card=0x20528086 > > chip=0x153b8086 rev=0x04 hdr=0x00 vendor = 'Intel Corporation' > > device = 'Ethernet Connection I217-V' > > class = network > > subclass = ethernet > > bar [10] = type Memory, range 32, base 0xf7d00000, size 131072, > > enabled bar [14] = type Memory, range 32, base 0xf7d35000, size > > 4096, enabled bar [18] = type I/O Port, range 32, base 0xf080, size > > 32, enabled cap 01[c8] = powerspec 2 supports D0 D3 current D0 > > cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message > > cap 13[e0] = PCI Advanced Features: FLR TP > > > > > > I remember faintly that there was an issue when I used to use FBSD 12 > > > > > > > > > > From nobody Tue Jun 7 00:30:46 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 448F81BDFB33 for ; Tue, 7 Jun 2022 00:30:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LHB926HhDz4cT0 for ; Tue, 7 Jun 2022 00:30:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id B08C210998 for ; Tue, 7 Jun 2022 00:30:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 2570Uktm033765 for ; Tue, 7 Jun 2022 00:30:46 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 2570UkeX033764 for jail@FreeBSD.org; Tue, 7 Jun 2022 00:30:46 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 228351] cannot unhide log or tty with devfs.rules Date: Tue, 07 Jun 2022 00:30:46 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: conf X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1654561846; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AKk45Xnq+6DKAPjSh0yIFaPdctWvGio/5aXc0lxDJBk=; b=j93qjlY7BmPgDRv3hds9E2BlGVFE9R3+F+JpAe3z+zwN7Dacz5r2++lpvSD6UWX7KblUWi wyDSuhRmRcBC7ot51E/8dgbFQUpl7jF7cTAFKlwOcFM6gxnyOZPWaOCnEiAYDaGWByhSri hEoeG5Si4HIEx97ZBJ4dwwFtNXhTbfQdEvl6HX4V1OHAnNo2Xyzo/B25YnWRYqiSaIfl/d snIT3AfjHqPYcZ4pVB/yE/KgjkR7QX2ZMINH60hfGhCsaD3RjGkyP2O2bwZy03b7zNficI 6cKDKBCKDDndGBV5lUWHLnYZOHAeFSccpWXgz0dYcsohUT1oP7Cf0liGuTaoQw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1654561847; a=rsa-sha256; cv=none; b=sX2gmdHWOzFIl7PP66rPn8bHvLrHLAlMGjyKNer4XB0wSWfvHnvpMLjgHU9DZ4hVgRUvlN RFD+h4nN/wqVQ/KSjbMA77Q9RpNc3zcJaHory+f9FF5BC9bhG4+EPGHtGVUFqsGgo51vy7 A6gF4L+5mgLDfgX5+3+5P3BbtdvvOmxwmXY7MnSTbylcAGXS1Cvz2PJegd3HdRUSdRvBDt fXSrbASEwny6qtAcEa+bTSMdlEgZluGKtkdt7ihZNnWsqNYCiVNTZIFdPSmBQj58nic2pS b58V/FOtV/3LJICay1PrccObWlG6EoSYJwnBAWv98K7mIBQOKf6NW830rrFCqw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D228351 --- Comment #7 from commit-hook@FreeBSD.org --- A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3D02fe4484379c1e67c22ad6abbeea10c8a= 20d10eb commit 02fe4484379c1e67c22ad6abbeea10c8a20d10eb Author: Zhenlei Huang AuthorDate: 2022-05-24 13:54:38 +0000 Commit: Mark Johnston CommitDate: 2022-06-07 00:29:58 +0000 devfs.rules: Do not expose "log" in the default devfs rules. /etc/rc.d/jail no longer creates /dev/log as a symbolic link since commit 84b354cb9ab61224713c159b1484e8f070fd37be. PR: 228351 Reviewed by: jamie, markj (cherry picked from commit 2670ea8a075ea29a0eee9d227c4cdf585a7b3b55) sbin/devfs/devfs.rules | 1 - 1 file changed, 1 deletion(-) --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Jun 28 23:38:15 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 94EE88626B9 for ; Tue, 28 Jun 2022 23:38:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LXgyK314Sz4rBX for ; Tue, 28 Jun 2022 23:38:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 448C126679 for ; Tue, 28 Jun 2022 23:38:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 25SNcHvw062047 for ; Tue, 28 Jun 2022 23:38:17 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 25SNcHee062046 for jail@FreeBSD.org; Tue, 28 Jun 2022 23:38:17 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 255830] dummynet(4) queues/pipes do not work inside of a VNET jail Date: Tue, 28 Jun 2022 23:38:15 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: pmc@citylink.dinoex.sub.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1656459497; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FJ4EMzkWH+5rRYttOMpXq7x60cyGWR9GQfJvyumVIQM=; b=N3gSDTEgvSlv371lEDloBn+Jiw3Jny131qza5MkhJRaWGh1/I3Qn2bcVqmLyQ21Tuj21T8 8NuMPVhYMCgFCtJHxmze13dYMu0ueeulqOG4grc11ZKz7AfQBEWjbprfdRba3nJ1lKWBnl 6NTMeXAy78r+JM7jmmM8LLEfvwyXMb5JUkJkm7CtxABuaPH/dFWfr/KWe73K3DaCjjUhm4 iDG7C2Y3+eRxGQSIXobsSTasJN2gAkLlCf+IqBUYfnu8lOuqhp0BNR0/gIo0gBuD/MvrTK 7M2nHdcq66J+s15pfjyrrxbeX3h6Ues4m7SbaI8dTM4AMs+Owi3vwwF1VBwFwQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1656459497; a=rsa-sha256; cv=none; b=waNsIgYLiXRgjhPQsUX4h52BZfJgjTWcgAjddl1kFENZIFvbKersoKsUlz+of21ItkkDWW y7oKnZE8HRGEcqaMslE2+TK3zANaLrtHEeOGW0yQbQMz2bCQ3WtHkQVy3xnfw6DCHGN+b/ zNnGWrkyjBOZBaYAaA44NuSsIrEywpMiEGcyC7gK3Ds+qSRnyKnS5fwp4Km9FqaWmyz3Ft m03/hP60FUhTugG3LATFGN8AF5UxKLeI/yo3n1irUHgkFawQcRqSa1y4qIvzefncm7dgAt THulTfDTD1Tx01jH+T42DMlrhwnJ6q+PnPPujelA8IpO7g7j33yzk7/o7aOZAg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D255830 Peter Much changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |pmc@citylink.dinoex.sub.org --- Comment #9 from Peter Much --- I am seeing these messages too, and no, Joshua, You're not the only one try= ing such things with dummynet. A few comments: * dummynet did never malfunction, it did simply reinsert the shaped packets into the host's ipfw instead of the jail's one - because there was only one dummynet on the system. The issue was more serios: every jail could manipul= ate the hosts pipes parameters, so it was necessary to run jails in securelevel= =3D3. * This appears to be fixed in 13.1. Kudos to the author! * What has sadly not been fixed alongside is the same issue with ng_ipfw, (Should be simpler, because netgraph itself is already VIMAGE-able) I once tried the bufferbloat thing, ran into the very same issue, considere= d it too difficult to fix on my own, and then moved the router into a bhyve inst= ead. There timing issues were more challenging, but the shaping apparently did w= ork. Currently it doesn't anymore. Thanks for Your configuration data, that might help me when looking into that issue occasionally - maybe the bhyve has to leave again. Now I tried to do the ng_ipfw fix myself, but with horror then noticed the beforementioned error-messages, and another one that looks worse: Apr 22 15:34:06 edge kernel: Freed UMA keg (IPFW counters) was = not empty (1 items). Lost 20 pages of memory. Jun 15 20:41:45 edge kernel: Freed UMA keg (IPFW counters) was = not empty (2 items). Lost 40 pages of memory. Jun 26 17:06:23 edge kernel: [55226] Freed UMA keg (IPFW counte= rs) was not empty (3 items). Lost 60 pages of memory. Jun 28 22:42:20 edge kernel: [248183] Freed UMA keg (IPFW count= ers) was not empty (3 items). Lost 40 pages of memory. But this one also obviousely exists since upgrade to 13.1 (RC2), and seems = to happen at shutdown after some uptime; it cannot be related to my recent hacking.=20 (I do massive and frequent ruleset updates during runtime.) --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Jun 29 01:03:28 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 986E186DD3E for ; Wed, 29 Jun 2022 01:03:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LXjrc2tBnz50Dn for ; Wed, 29 Jun 2022 01:03:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 43E4527BBC for ; Wed, 29 Jun 2022 01:03:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 25T13Sjw009201 for ; Wed, 29 Jun 2022 01:03:28 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 25T13SGU009200 for jail@FreeBSD.org; Wed, 29 Jun 2022 01:03:28 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 255830] dummynet(4) queues/pipes do not work inside of a VNET jail Date: Wed, 29 Jun 2022 01:03:28 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: freebsd@kumba.dev X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1656464608; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VQNjh0kFa5sAv+NhvDJsMvcto01PO5+XKI4bJZ0sB4c=; b=UydNnWAK22lVqqPLnBr8EjI6EU/Ra8OXIFu8Cx11zo7zGFFlWNmvbqp71L4VgougfNGO6C iGn7/8+ub3RZR71Py7Wno7ykpR6iHRlTeYZwytP0ojY1z//DkBNaziAx8VMdRNEK1fohms sNZYgrHZsiqIqD0dNQ+Kyzxtwodiq5MoEUi2PdQxtsZ9bz1mS76422mU3u1tWcRl60d6u4 KYETkx5vBQrV82Kpd8xT7liIp/LM6hKgIoQYkrcbXSQpH9y7b6AsctDQvJOguNI/05Fzo7 Y+gHzB7PuDJ+QmvFY81jZ+U1TI5PBoGozrYIEIyxxyHvCEIKl9L8gS8XUIpqvQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1656464608; a=rsa-sha256; cv=none; b=PuPkAOE9BCckxY5qjO81mUq0Y6e81bTjskL0IDL0lVRNAyg/cViAWxH98QU4W/A8vSeeG0 TSXOvqM8jpXUwUohXovQudW4yTK8lP14tFQhDVCWu0vCFy1R2zsAyZx+GaKmqO9W1AdkRw 47v4GPNfYzO4qrrBp44hLjh6dVMPnTM9Utk4hbp7uWqzd5yi+jNfkzK7xy7Ek5slRqaNsx qocNqAUz6H+qup5BO+cgXj9xSORMqgUAvlxSQnfHoiSUW0q06GCqoUdYEUReMibFkMvRug I1696ETvOuiIZNtfr2S++RidTz63RMICyj2kFpiivVRSvMbSVaE14HSbH4WJxQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D255830 --- Comment #10 from Joshua Kinard --- (In reply to Peter Much from comment #9) I'd give the jailed VNET router another try. It has been working fine on my end for a few months now (since ~13.1-BETA2). Just give the jail the exter= nal WAN interface and internal LAN interface and configure as normal. The host should listen on a third interface with a different address than the jail (= my router has six gigabit interfaces to play with). Only bug I've still got to hunt down is when the jail starts at boot, it sometimes fails to pull an address from my ISP even though I have SYNCDHCP = set. Easy fix to just restart netif em0 in the jail when it happens. I should debug it, though and figure out how to really fix it. I even have my wlan0 interface assigned to the jail and run hostapd out of it to run the wifi on= a separate subnet. Needs a small tweak to hostapd's rc script to let it run = into a jail (see #263359). --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Jun 30 07:04:24 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6496D8799F4 for ; Thu, 30 Jun 2022 07:04:37 +0000 (UTC) (envelope-from dfr@rabson.org) Received: from mail-lf1-x134.google.com (mail-lf1-x134.google.com [IPv6:2a00:1450:4864:20::134]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LYTpr4wYbz3Pgq for ; Thu, 30 Jun 2022 07:04:36 +0000 (UTC) (envelope-from dfr@rabson.org) Received: by mail-lf1-x134.google.com with SMTP id y16so8203239lfb.9 for ; Thu, 30 Jun 2022 00:04:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rabson-org.20210112.gappssmtp.com; s=20210112; h=mime-version:from:date:message-id:subject:to:cc; bh=p4VvHA3BYktBTM3ydmQ2UGT9diFJQW4CucBjOT2Yogo=; b=VZgO5Ul21RGSoNsvmPx20ZSSHGpbG3BWFReiJc1G+b+GhI8J7JlX8wPkbMs+RB3jSL VGJZnN43+0V9E8BNAGP65QJGvA3QNQXoiupPc2iyLQOR65GkHdvUwSmoqCKEYU/3D7Ox DwVPjBztGQozs3aTAM7gFAjg8ulnS5oZNZgkXW1+mwQ22n3Z2v3+ky0RzKe/GU0f0eDK KJYwln8PwzXGOiqchyV6RxZ6cRGbcqIB0bg1Buui8MZTBTytnZpk5uNlrJQDTlqRqAJg +LGUyNsaTcKblXTib3kPaCNYZ1BJES3IJ3XTfxyjMSOhNuqPesGoe2BPmr8L2GBrDgGY etHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=p4VvHA3BYktBTM3ydmQ2UGT9diFJQW4CucBjOT2Yogo=; b=6uKMFinXbuWQjEvR50OwxM8u36YZOsSlEezLNeUzTEZ4Kqe3fXqDz6cDdE/Pg21SIA guSw8VBcc2HGy/2+96xAURnRkHqoDyTdlOdeeqbjjU4vR7UddkY8WLlnXJc0Ynlq+4go 3VDd6DzLrLn1hiYBlG+DsHynXpQ68rw5q+oKbxK/4ZHeGroiu3GHqDNduqWtcsHN/CtF VxtNLrSmwdnX+j+LUIZ65X0QlsVtc6bCyHf8LbCtRiAHW17XtvoOMwtIuwQVytwFfkyt l4PkeO7CEJWdDV3UxNmYvV8baT4n9wkxmVJijZ+JJdcb00uEXdH0UX7kLoWrQrCTjgui GPJw== X-Gm-Message-State: AJIora8ktJi9GWCQFGUwfbo15a+KCNuk8e3szMU4Y9mzDZIgbQc6VC7X 4lTit8/oYV3nHvgSIEmfvpO8DNz0auBqZdOEUJglgBbDSzdT0AgK X-Google-Smtp-Source: AGRyM1seoLjlAbr7jxMFehu27NzvZGEFgZ0U0evnDjfyKOPfR8PmD/kYxHj0BOC8M7vr3WHLbmiOhYUd9YOMKekS1oY= X-Received: by 2002:a05:6512:25a3:b0:481:25b8:51b5 with SMTP id bf35-20020a05651225a300b0048125b851b5mr4785965lfb.472.1656572674896; Thu, 30 Jun 2022 00:04:34 -0700 (PDT) List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 From: Doug Rabson Date: Thu, 30 Jun 2022 08:04:24 +0100 Message-ID: Subject: Container Networking for jails To: freebsd-jail@freebsd.org, freebsd-net@freebsd.org Cc: Samuel Karp Content-Type: multipart/alternative; boundary="000000000000d509bf05e2a4e17e" X-Rspamd-Queue-Id: 4LYTpr4wYbz3Pgq X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=rabson-org.20210112.gappssmtp.com header.s=20210112 header.b=VZgO5Ul2; dmarc=none; spf=pass (mx1.freebsd.org: domain of dfr@rabson.org designates 2a00:1450:4864:20::134 as permitted sender) smtp.mailfrom=dfr@rabson.org X-Spamd-Result: default: False [-3.50 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[rabson-org.20210112.gappssmtp.com:s=20210112]; FREEFALL_USER(0.00)[dfr]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-jail@freebsd.org]; DMARC_NA(0.00)[rabson.org]; NEURAL_HAM_LONG(-1.00)[-1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[rabson-org.20210112.gappssmtp.com:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::134:from]; MLMMJ_DEST(0.00)[freebsd-jail]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N --000000000000d509bf05e2a4e17e Content-Type: text/plain; charset="UTF-8" I wanted to get a quick sanity check for my current approach to container networking with buildah and podman. These systems use CNI ( https://www.cni.dev) to set up the network. This uses a sequence of 'plugins' which are executables that perform successive steps in the process - a very common setup uses a 'bridge' plugin to add one half of an epair to a bridge and put the other half into the container's vnet. IP addresses are managed by an 'ipam' plugin and an optional 'portmap' plugin can be used to advertise container service ports on the host. All of these plugins run on the host with root privileges. In kubernetes and podman, it is possible for more than one container to share a network namespace in a 'pod'. Each container in the pod can communicate with its peers directly via localhost and they all share a single IP address. Mapping this over to jails, I am using one vnet jail to manage the network namespace and child jails of this to isolate the containers. The vnet jail uses '/' as its root path and the only things which run inside this jail are the CNI plugins. Using the host root means that a plugin can safely call host utilities such as ifconfig and route without having to trust the container's version of them. An important factor here is that the CNI plugins will only be run strictly before the container (to set up) or strictly after (to tear down) - at no point will CNI plugins be executed at the same time as container executables. The child jails use ip4/6=inherit to share the vnet and each will use a root path to the container's contents in the same way as a normal non-hierarchical jail. Can anyone see any potential security problems here, particularly around the use of nested jails? I believe that the only difference between this setup and a regular non-nested jail is that the vnet outlives the container briefly before it is torn down. --000000000000d509bf05e2a4e17e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

<= div>In kubernetes and podman, it is possible for more than one container to= share a network namespace in a 'pod'. Each container in the pod ca= n communicate with its peers directly via localhost and they all share a si= ngle IP address.



Can anyone see any potential security problems here, particularly around = the use of nested jails? I believe that the only difference between this se= tup and a regular non-nested jail is that the vnet outlives the container b= riefly before it is torn down.
--000000000000d509bf05e2a4e17e-- From nobody Thu Jun 30 07:18:11 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 5A35F87A6FC for ; Sun, 3 Jul 2022 16:29:56 +0000 (UTC) (envelope-from gijs@peskens.net) Received: from smtp.heteigenwijsje.nl (smtp.heteigenwijsje.nl [185.216.161.180]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4LbZCk2MfNz4gp9 for ; Sun, 3 Jul 2022 16:29:54 +0000 (UTC) (envelope-from gijs@peskens.net) Received: from mail.heteigenwijsje.nl (localhost [127.0.0.1]) by smtp.heteigenwijsje.nl (Postfix) with ESMTP id 6B27E8031 for ; Sun, 3 Jul 2022 18:20:16 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d= heteigenwijsje.nl; h=message-id:from:from:to:subject:subject :content-transfer-encoding:content-type:content-type :mime-version:references:in-reply-to:user-agent:date:date; s= dkim; t=1656865212; x=1659457213; bh=jwZ+s48UjfzoFws1XB+mmnw8YE1 NYPrgrXEsmQYjJco=; b=uOd3qB5zgFFkTO4oz2lW7eDRUqu4VNCpKKoXU/2wG1h l3BdI9k6OIKCQM0lxZlV0WtMjNbtejShARLdLr0FVaFshCYNkudlAYqO/xBie6d4 R0zWz4KOGpALtHSGAwooaiRyf4hlPwIEcZZ8YGrgnRKVGpw+AOy2KMIhla6C2cGg = X-Virus-Scanned: amavisd-new at mail.heteigenwijsje.nl Received: from smtp.heteigenwijsje.nl ([127.0.0.1]) by mail.heteigenwijsje.nl (mail.heteigenwijsje.nl [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 1NsBmG5jsahJ for ; Sun, 3 Jul 2022 18:20:12 +0200 (CEST) Received: from smtp.heteigenwijsje.nl (localhost [127.0.0.1]) by smtp.heteigenwijsje.nl (Postfix) with ESMTP id E4ACB8334; Sun, 3 Jul 2022 18:20:11 +0200 (CEST) Received: from [10.0.1.174] ([10.0.1.174]) by smtp.heteigenwijsje.nl with ESMTPSA id kOmgNbvBwWIzrAAAc3PRCQ (envelope-from ); Sun, 03 Jul 2022 18:20:11 +0200 Date: Thu, 30 Jun 2022 09:18:11 +0200 User-Agent: K-9 Mail for Android In-Reply-To: References: List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----WBQ1AR96SFCU65D8I93TK5TO9I9EQ3" Content-Transfer-Encoding: 7bit Subject: Re: Container Networking for jails To: freebsd-jail@freebsd.org,Doug Rabson ,freebsd-net@freebsd.org CC: Samuel Karp From: Gijs Peskens Message-ID: X-Rspamd-Queue-Id: 4LbZCk2MfNz4gp9 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=heteigenwijsje.nl header.s=dkim header.b=uOd3qB5z; dmarc=pass (policy=none) header.from=peskens.net; spf=pass (mx1.freebsd.org: domain of gijs@peskens.net designates 185.216.161.180 as permitted sender) smtp.mailfrom=gijs@peskens.net X-Spamd-Result: default: False [-3.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; R_DKIM_ALLOW(-0.20)[heteigenwijsje.nl:s=dkim]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-jail@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[heteigenwijsje.nl:+]; DMARC_POLICY_ALLOW(-0.50)[peskens.net,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-jail]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; DATE_IN_PAST(1.00)[81]; ASN(0.00)[asn:45040, ipnet:185.216.160.0/22, country:NL]; RCVD_TLS_LAST(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-ThisMailContainsUnwantedMimeParts: N ------WBQ1AR96SFCU65D8I93TK5TO9I9EQ3 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable I went with exactly the same design for the Docker port I started a while a= go=2E The reason I went with that design is that there weren't any facilities to= modify a jails vent network configuration from outside of the jail=2E So i= t's needed to enter the jail, run ifconfig et all=2E Linux jails will lack a compatible ifconfig=2E=20 So having a parent FreeBSD based vnet jail ensures that networking can be = configured for Linux children=2E=20 There is a risk to using the / filesystem: users that might be allowed to = setup and configure containers run standard system tools as root on the roo= t filesystem, even if they might not have root permission themselves=2E=2E = If an exploit was to be ever found in any of those tools to modify files th= at could be used as a step in a privilege escalation=2E=20 Imho, that risk is acceptable in a first port, but should be documented=2E= And ideally an option should be provided to use an alternative root if the= user deems the risk unacceptable=2E On 30 June 2022 09:04:24 CEST, Doug Rabson wrote: >I wanted to get a quick sanity check for my current approach to >container >networking with buildah and podman=2E These systems use CNI ( >https://www=2Ecni=2Edev) to set up the network=2E This uses a sequence of >'plugins' which are executables that perform successive steps in the >process - a very common setup uses a 'bridge' plugin to add one half of >an >epair to a bridge and put the other half into the container's vnet=2E IP >addresses are managed by an 'ipam' plugin and an optional 'portmap' >plugin >can be used to advertise container service ports on the host=2E All of >these >plugins run on the host with root privileges=2E > >In kubernetes and podman, it is possible for more than one container to >share a network namespace in a 'pod'=2E Each container in the pod can >communicate with its peers directly via localhost and they all share a >single IP address=2E > >Mapping this over to jails, I am using one vnet jail to manage the >network >namespace and child jails of this to isolate the containers=2E The vnet >jail >uses '/' as its root path and the only things which run inside this >jail >are the CNI plugins=2E Using the host root means that a plugin can safely >call host utilities such as ifconfig and route without having to trust >the >container's version of them=2E An important factor here is that the CNI >plugins will only be run strictly before the container (to set up) or >strictly after (to tear down) - at no point will CNI plugins be >executed at >the same time as container executables=2E > >The child jails use ip4/6=3Dinherit to share the vnet and each will use a >root path to the container's contents in the same way as a normal >non-hierarchical jail=2E > >Can anyone see any potential security problems here, particularly >around >the use of nested jails? I believe that the only difference between >this >setup and a regular non-nested jail is that the vnet outlives the >container >briefly before it is torn down=2E --=20 Verstuurd vanaf mijn Android apparaat met K-9 Mail=2E Excuseer mijn beknop= theid=2E ------WBQ1AR96SFCU65D8I93TK5TO9I9EQ3 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable I went with exactly the same design for the Docker= port I started a while ago=2E
The reason I went with that design is tha= t there weren't any facilities to modify a jails vent network configuration= from outside of the jail=2E So it's needed to enter the jail, run ifconfig= et all=2E
Linux jails will lack a compatible ifconfig=2E
So having = a parent FreeBSD based vnet jail ensures that networking can be configured = for Linux children=2E

There is a risk to using the / filesystem: us= ers that might be allowed to setup and configure containers run standard sy= stem tools as root on the root filesystem, even if they might not have root= permission themselves=2E=2E If an exploit was to be ever found in any of t= hose tools to modify files that could be used as a step in a privilege esca= lation=2E

Imho, that risk is acceptable in a first port, but should= be documented=2E And ideally an option should be provided to use an altern= ative root if the user deems the risk unacceptable=2E




On 30 June 2022 09:04:24 CEST, Doug Rabson <dfr= @rabson=2Eorg> wrote:
I wanted to get a quick sanity check for my current = approach to container networking with buildah and podman=2E These systems u= se CNI (https://www= =2Ecni=2Edev) to set up the network=2E This uses a sequence of 'plugins= ' which are executables that perform successive steps in the process - a ve= ry common setup uses a 'bridge' plugin to add one half of an epair to = a bridge and put the other half into the container's vnet=2E IP addresses a= re managed by an 'ipam' plugin and an optional 'portmap' plugin can be used= to advertise container service ports on the host=2E All of these plugins r= un on the host with root privileges=2E

In kubernet= es and podman, it is possible for more than one container to share a networ= k namespace in a 'pod'=2E Each container in the pod can communicate with it= s peers directly via localhost and they all share a single IP address=2E

Mapping this over to jails, I am using one vnet jail= to manage the network namespace and child jails of this to isolate the con= tainers=2E The vnet jail uses '/' as its root path and the only things whic= h run inside this jail are the CNI plugins=2E Using the host root means tha= t a plugin can safely call host utilities such as ifconfig and route w= ithout having to trust the container's version of them=2E An important fact= or here is that the CNI plugins will only be run strictly before the contai= ner (to set up) or strictly after (to tear down) - at no point will CNI plu= gins be executed at the same time as container executables=2E
The child jails use ip4/6=3Dinherit to share the vnet and each = will use a root path to the container's contents in the same way as a norma= l non-hierarchical jail=2E

Can anyone see any pote= ntial security problems here, particularly around the use of nested jails? = I believe that the only difference between this setup and a regular non-nes= ted jail is that the vnet outlives the container briefly before it is torn = down=2E

--
Verstuurd vanaf mijn Android apparaat met K-= 9 Mail=2E Excuseer mijn beknoptheid=2E ------WBQ1AR96SFCU65D8I93TK5TO9I9EQ3-- From nobody Mon Jul 4 11:14:08 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6692D8BAA66 for ; Mon, 4 Jul 2022 11:14:28 +0000 (UTC) (envelope-from dfr@rabson.org) Received: from mail-lj1-x22c.google.com (mail-lj1-x22c.google.com [IPv6:2a00:1450:4864:20::22c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Lc39H2fK8z4th0 for ; Mon, 4 Jul 2022 11:14:27 +0000 (UTC) (envelope-from dfr@rabson.org) Received: by mail-lj1-x22c.google.com with SMTP id l7so9919507ljj.4 for ; Mon, 04 Jul 2022 04:14:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rabson-org.20210112.gappssmtp.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=+VvGwgC6usEDsxdLDR7S9PP5pW6ke9jjSgXNB2tOJFo=; b=c7/20qhfpBnRNNyb5tHHtVVBlzwA1NPa0ZcnX/c4u06ofTHH3zpppCPE6TKTvnipvB cpim5vzUy/6Ut4YmR66dq8aXW9J90/Y8lkQT/+fg+ALweSDgtQ5uZH2IpyjInDjYLvIx bNAmBWXHIdIo09ItnY5MuvcFH+tqxGOTRR1UAAK3v3oMrEbW+5XmrS4UsvyZPBYKjk/f Qk6uUHMRErfeBRwhzUzGdTiI1O8+PnswyHROpcFt+pPgn/C/PNX7y82a67c1p/M5aotN RlNFsIg3/f7UuQJ+Vi6KfDhdEUNomFzgVfwGXtzXMtqQ/Ip3dX6ktD/zV6rTFsftQwvN vXGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=+VvGwgC6usEDsxdLDR7S9PP5pW6ke9jjSgXNB2tOJFo=; b=KyDYDy6s1yABSPbxUFIZfKt4H/NMzgz0cJtPoeWRyMvDEEeHcX4nQPxaRxWBv6CL5Q L7tCccb0r9QIGPH1Yp50GmnTi8T0Jiu2rOKVWQi436tbsretyb+mDVRxWWCAS1857BqY XnLdLcCkxsXNfzy5oicjFU2U3LmgBqFR4obQW/az8WelS/p9tjua/AG5f9fu9DDDWtHO mnd/sn23PL9/rwyVmGie4yxwFbJ3tuJcU6n09vJmJ0455Rb0s66xhjXlcu1hGn9Lw87j F12THz0kyOT50+8EkbgoYQxNy1Y+vFDWGgyamm1WQ2vrOTvntaV2FBwGa8JVwk7bYHP5 xDBA== X-Gm-Message-State: AJIora+KPuHlmgNhp/nT4+yTF8MCxe5vaenp8Cp6LbEVQvs8Ivw6u6ob 7wrAsZYTY9ejwU5DQoa9sjv9ucYcW6QEHrBIATfmOA== X-Google-Smtp-Source: AGRyM1vKmFq2nmRVS2QBazxAxTApzVgeJUIpdQMeDVWhgsgdO/Zp5avYFUkYxri2xlNWj1jQsQ0R3HytKqsSM1XhJKw= X-Received: by 2002:a05:651c:1544:b0:25a:8e6f:a1b6 with SMTP id y4-20020a05651c154400b0025a8e6fa1b6mr16491856ljp.314.1656933259917; Mon, 04 Jul 2022 04:14:19 -0700 (PDT) List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Doug Rabson Date: Mon, 4 Jul 2022 12:14:08 +0100 Message-ID: Subject: Re: Container Networking for jails To: Gijs Peskens Cc: freebsd-jail@freebsd.org, freebsd-net@freebsd.org, Samuel Karp Content-Type: multipart/alternative; boundary="0000000000005fdc7205e2f8d6e0" X-Rspamd-Queue-Id: 4Lc39H2fK8z4th0 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=rabson-org.20210112.gappssmtp.com header.s=20210112 header.b="c7/20qhf"; dmarc=none; spf=pass (mx1.freebsd.org: domain of dfr@rabson.org designates 2a00:1450:4864:20::22c as permitted sender) smtp.mailfrom=dfr@rabson.org X-Spamd-Result: default: False [-3.50 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[rabson-org.20210112.gappssmtp.com:s=20210112]; FREEFALL_USER(0.00)[dfr]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-jail@freebsd.org]; DMARC_NA(0.00)[rabson.org]; NEURAL_HAM_LONG(-1.00)[-1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[rabson-org.20210112.gappssmtp.com:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::22c:from]; MLMMJ_DEST(0.00)[freebsd-jail]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N --0000000000005fdc7205e2f8d6e0 Content-Type: text/plain; charset="UTF-8" I think it's important that configuring the container network does not rely on any utilities from inside the container - for one thing, there are no guarantees that these utilities even exist inside the container and as you note, local versions may be incompatible. On the subject of risk, with the current jail infrastructure, the only user which can create and modify containers is root. Certain users may have delegated authority, e.g. by using setuid on a daemon-less setup like podman or by adjusting permissions on a unix domain socket but this is clearly a huge risk and should be strongly discouraged (IMO). Rootless containers using something similar to linux user namespaces would be nice but it is probably a higher priority to get containers working well for root first. My concern for supporting an alternative 'tooling' image for network utilities is that it adds complexity to the infrastructure for very little gain. You could even make a weak argument that it adds a threat vector, e.g. if the network utilities image is fetched from a compromised repository (pretty far fetched IMO but possible). On Sun, 3 Jul 2022 at 17:29, Gijs Peskens wrote: > I went with exactly the same design for the Docker port I started a while > ago. > The reason I went with that design is that there weren't any facilities to > modify a jails vent network configuration from outside of the jail. So it's > needed to enter the jail, run ifconfig et all. > Linux jails will lack a compatible ifconfig. > So having a parent FreeBSD based vnet jail ensures that networking can be > configured for Linux children. > > There is a risk to using the / filesystem: users that might be allowed to > setup and configure containers run standard system tools as root on the > root filesystem, even if they might not have root permission themselves.. > If an exploit was to be ever found in any of those tools to modify files > that could be used as a step in a privilege escalation. > > Imho, that risk is acceptable in a first port, but should be documented. > And ideally an option should be provided to use an alternative root if the > user deems the risk unacceptable. > > > > > On 30 June 2022 09:04:24 CEST, Doug Rabson wrote: >> >> I wanted to get a quick sanity check for my current approach to container >> networking with buildah and podman. These systems use CNI ( >> https://www.cni.dev) to set up the network. This uses a sequence of >> 'plugins' which are executables that perform successive steps in the >> process - a very common setup uses a 'bridge' plugin to add one half of an >> epair to a bridge and put the other half into the container's vnet. IP >> addresses are managed by an 'ipam' plugin and an optional 'portmap' plugin >> can be used to advertise container service ports on the host. All of these >> plugins run on the host with root privileges. >> >> In kubernetes and podman, it is possible for more than one container to >> share a network namespace in a 'pod'. Each container in the pod can >> communicate with its peers directly via localhost and they all share a >> single IP address. >> >> Mapping this over to jails, I am using one vnet jail to manage the >> network namespace and child jails of this to isolate the containers. The >> vnet jail uses '/' as its root path and the only things which run inside >> this jail are the CNI plugins. Using the host root means that a plugin can >> safely call host utilities such as ifconfig and route without having to >> trust the container's version of them. An important factor here is that the >> CNI plugins will only be run strictly before the container (to set up) or >> strictly after (to tear down) - at no point will CNI plugins be executed at >> the same time as container executables. >> >> The child jails use ip4/6=inherit to share the vnet and each will use a >> root path to the container's contents in the same way as a normal >> non-hierarchical jail. >> >> Can anyone see any potential security problems here, particularly around >> the use of nested jails? I believe that the only difference between this >> setup and a regular non-nested jail is that the vnet outlives the container >> briefly before it is torn down. >> > > -- > Verstuurd vanaf mijn Android apparaat met K-9 Mail. Excuseer mijn > beknoptheid. > --0000000000005fdc7205e2f8d6e0 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I think it's=C2=A0important that configuring the conta= iner network does not rely on any utilities from inside the container - for= one thing, there are no guarantees that these utilities even exist inside = the container and as you note, local versions may be incompatible.

=
On the subject of risk, with the current jail infrastructure, th= e only user which can create and modify containers is root. Certain users m= ay have delegated authority, e.g. by using setuid on a daemon-less setup li= ke podman or by adjusting permissions on a unix domain socket but this is c= learly a huge risk and should be strongly discouraged (IMO). Rootless conta= iners using something similar to linux user namespaces would be nice but it= is probably a higher priority to get containers working well for root firs= t.

My concern for supporting an alternative 't= ooling' image for network utilities is that it adds complexity to the i= nfrastructure for very little gain. You could even make a weak argument tha= t it adds a threat vector, e.g. if the network utilities image is fetched f= rom a compromised repository (pretty far fetched IMO but possible).



On Sun, 3 Jul 2022 at 17:29, Gijs Peskens <= ;gijs@peskens.net> wrote:
I went with exactly the same design for the Docker= port I started a while ago.
The reason I went with that design is that = there weren't any facilities to modify a jails vent network configurati= on from outside of the jail. So it's needed to enter the jail, run ifco= nfig et all.
Linux jails will lack a compatible ifconfig.
So having = a parent FreeBSD based vnet jail ensures that networking can be configured = for Linux children.

There is a risk to using the / filesystem: user= s that might be allowed to setup and configure containers run standard syst= em tools as root on the root filesystem, even if they might not have root p= ermission themselves.. If an exploit was to be ever found in any of those t= ools to modify files that could be used as a step in a privilege escalation= .

Imho, that risk is acceptable in a first port, but should be docu= mented. And ideally an option should be provided to use an alternative root= if the user deems the risk unacceptable.




On 30 June 2022 09:04:24 CEST, Doug Rabson <dfr@rabson.org> wrote:

<= div>In kubernetes and podman, it is possible for more than one container to= share a network namespace in a 'pod'. Each container in the pod ca= n communicate with its peers directly via localhost and they all share a si= ngle IP address.



Can anyone see any potential security problems here, particularly around = the use of nested jails? I believe that the only difference between this se= tup and a regular non-nested jail is that the vnet outlives the container b= riefly before it is torn down.

--
Verstuurd vanaf mijn Android apparaat met K-9= Mail. Excuseer mijn beknoptheid.
--0000000000005fdc7205e2f8d6e0-- From nobody Sat Jul 23 11:56:20 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4LqlC02z0Wz4WV3B for ; Sat, 23 Jul 2022 11:56:28 +0000 (UTC) (envelope-from pi@freebsd.org) Received: from fc.opsec.eu (fc.opsec.eu [IPv6:2001:14f8:200:4::4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4LqlBz5n2tz41TK for ; Sat, 23 Jul 2022 11:56:27 +0000 (UTC) (envelope-from pi@freebsd.org) Received: from pi by fc.opsec.eu with local (Exim 4.95 (FreeBSD)) (envelope-from ) id 1oFDk4-000Bim-BD for freebsd-jail@freebsd.org; Sat, 23 Jul 2022 13:56:20 +0200 Date: Sat, 23 Jul 2022 13:56:20 +0200 From: Kurt Jaeger To: freebsd-jail@freebsd.org Subject: jail created with ip4=new and ipv4.addr shows ip4=disable on jail -s Message-ID: List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Rspamd-Queue-Id: 4LqlBz5n2tz41TK X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=softfail (mx1.freebsd.org: 2001:14f8:200:4::4 is neither permitted nor denied by domain of pi@freebsd.org) smtp.mailfrom=pi@freebsd.org X-Spamd-Result: default: False [-3.10 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-jail@freebsd.org]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:12502, ipnet:2001:14f8::/32, country:DE]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_ONE(0.00)[1]; R_SPF_SOFTFAIL(0.00)[~all]; DMARC_NA(0.00)[freebsd.org]; FREEFALL_USER(0.00)[pi]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; TO_DN_NONE(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DOM_EQ_FROM_DOM(0.00)[] X-ThisMailContainsUnwantedMimeParts: N Hello, On a 13.1 box: The jail is created with: /usr/sbin/jail -c allow.raw_sockets allow.sysvipc devfs_ruleset=4 host.hostname=somehostname path=/somepath ip4=new ip4.addr= ip6=new ip6.addr= command=/bin/sh /etc/rc But: jail -s displays: [...] ip4=disable ip6=disable Is that a bug and if not, why does it behave like that ? -- pi@FreeBSD.org +49 171 3101372 Now what ? From nobody Sat Jul 23 17:06:15 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Lqt4Z7497z4XDv5 for ; Sat, 23 Jul 2022 17:06:22 +0000 (UTC) (envelope-from jamie@gritton.org) Received: from gritton.org (gritton.org [162.220.209.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "gritton.org", Issuer "gritton.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Lqt4Z19Cwz4Pmh; Sat, 23 Jul 2022 17:06:22 +0000 (UTC) (envelope-from jamie@gritton.org) Received: from gritton.org ([127.0.0.3]) (authenticated bits=0) by gritton.org (8.16.1/8.16.1) with ESMTPA id 26NH6F3i025353; Sat, 23 Jul 2022 10:06:15 -0700 (PDT) (envelope-from jamie@gritton.org) List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Date: Sat, 23 Jul 2022 10:06:15 -0700 From: James Gritton To: freebsd-jail@freebsd.org Cc: Kurt Jaeger Subject: Re: jail created with ip4=new and ipv4.addr shows ip4=disable on jail -s In-Reply-To: References: User-Agent: Roundcube Webmail/1.4.11 Message-ID: <8e1bf678efc18f9d3c4d5ee16df3caa1@gritton.org> X-Sender: jamie@gritton.org Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4Lqt4Z19Cwz4Pmh X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of jamie@gritton.org designates 162.220.209.3 as permitted sender) smtp.mailfrom=jamie@gritton.org X-Spamd-Result: default: False [-3.30 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+ip4:162.220.209.0/28]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:30247, ipnet:162.220.208.0/22, country:US]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-jail@freebsd.org]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; FREEFALL_USER(0.00)[jamie]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_SOME(0.00)[]; DMARC_NA(0.00)[gritton.org]; MID_RHS_MATCH_FROM(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On 2022-07-23 04:56, Kurt Jaeger wrote: > Hello, > > On a 13.1 box: > > The jail is created with: > > /usr/sbin/jail -c allow.raw_sockets allow.sysvipc devfs_ruleset=4 > host.hostname=somehostname path=/somepath ip4=new ip4.addr= > ip6=new ip6.addr= command=/bin/sh /etc/rc > > But: > > jail -s > > displays: > > [...] ip4=disable ip6=disable > > Is that a bug and if not, why does it behave like that ? It's a bug in the reporting. ip4 is presented as a jailsys parameter with its values of disable, inherit, and new. jail_get(2) reports such values based on flags in the prison structure, but ip4 and ip6 are only stored as a single bit with disable indistinguishable from new. jail_get should be looking at the number of IP addresses, which is what tells the difference. - Jamie From nobody Sun Jul 24 15:31:58 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4LrRxM2F4Tz4Wqh4 for ; Sun, 24 Jul 2022 15:32:07 +0000 (UTC) (envelope-from pi@freebsd.org) Received: from fc.opsec.eu (fc.opsec.eu [IPv6:2001:14f8:200:4::4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4LrRxL3SmCz46DB for ; Sun, 24 Jul 2022 15:32:06 +0000 (UTC) (envelope-from pi@freebsd.org) Received: from pi by fc.opsec.eu with local (Exim 4.95 (FreeBSD)) (envelope-from ) id 1oFdaI-000JfC-D2; Sun, 24 Jul 2022 17:31:58 +0200 Date: Sun, 24 Jul 2022 17:31:58 +0200 From: Kurt Jaeger To: James Gritton Cc: freebsd-jail@freebsd.org Subject: Re: jail created with ip4=new and ipv4.addr shows ip4=disable on jail -s Message-ID: References: <8e1bf678efc18f9d3c4d5ee16df3caa1@gritton.org> List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <8e1bf678efc18f9d3c4d5ee16df3caa1@gritton.org> X-Rspamd-Queue-Id: 4LrRxL3SmCz46DB X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=softfail (mx1.freebsd.org: 2001:14f8:200:4::4 is neither permitted nor denied by domain of pi@freebsd.org) smtp.mailfrom=pi@freebsd.org X-Spamd-Result: default: False [-3.10 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-jail@freebsd.org]; RCPT_COUNT_TWO(0.00)[2]; ARC_NA(0.00)[]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all:c]; DMARC_NA(0.00)[freebsd.org]; FREEFALL_USER(0.00)[pi]; ASN(0.00)[asn:12502, ipnet:2001:14f8::/32, country:DE]; FROM_HAS_DN(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N Hi! > > On a 13.1 box: > > > > The jail is created with: > > > > /usr/sbin/jail -c allow.raw_sockets allow.sysvipc devfs_ruleset=4 > > host.hostname=somehostname path=/somepath ip4=new ip4.addr= > > ip6=new ip6.addr= command=/bin/sh /etc/rc > > But: > > jail -s > > displays: > > [...] ip4=disable ip6=disable > > Is that a bug and if not, why does it behave like that ? > > It's a bug in the reporting. Thanks very much for the explaination! > ip4 is presented as a jailsys parameter with > its values of disable, inherit, and new. jail_get(2) reports such values > based on flags in the prison structure, but ip4 and ip6 are only stored as > a single bit with disable indistinguishable from new. jail_get should be > looking at the number of IP addresses, which is what tells the difference. Interesting! -- pi@FreeBSD.org +49 171 3101372 Now what ? From nobody Tue Sep 20 13:34:48 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MX2bD3m7xz4cDS2 for ; Tue, 20 Sep 2022 13:34:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MX2bD2THSz3R3M for ; Tue, 20 Sep 2022 13:34:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4MX2bD1YhxzkPG for ; Tue, 20 Sep 2022 13:34:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 28KDYmqV049469 for ; Tue, 20 Sep 2022 13:34:48 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 28KDYmko049468 for jail@FreeBSD.org; Tue, 20 Sep 2022 13:34:48 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 258481] Jail information leak via zfs kstats Date: Tue, 20 Sep 2022 13:34:48 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: allanjude@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: allanjude@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1663680888; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2cB4VwTGPahHAhwZXavSCrg/4Cr6hVCo/UnkFVL2ZW8=; b=QJGxEovIja/AFkFxQS/CEibhUUMQZUN/WKNr2p5m12b8+9pKP9mO4WVSsysW0DngG7rTP+ oCHxjB+AYFgU/YDNzSsT+msmpSOouxDFK/CTvQfgzyt09iR5KGAp4HI51ZI9+C3jH9Ontf XRBBLv7f1oL3V2/8FdNrMnWZdncmmS/rUqEg8qqvJivEe7NgZL4sbtdldRyhc6BHDM/8Bc J0CSJYJPHb0BXuQ6qFiURUWMqZ8oJOZvTo9MnDdy3UBqLHpnEInXnQ9hL6E9L+4ZKoYGqo SUC0oCCOXhh0/12DbXYsoymC20NM6/RglUyiikWN71LVMktHwL1kO3Vf1p8YdA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1663680888; a=rsa-sha256; cv=none; b=LVhXqma+RS03F1ff5KINBwRXdHJ2TJSERqROYFUGElz2qDZhqvvqzpqq7Oi6PDpGkf91/D 43XSwTryeNccl2lNxI3aq+kopLVdhkR3ED/5+vkrrVI/yNeI3803Za7KVHQmojxzCtVbwM GjtWKSG7umIxwsPF9MN4LULB172xc6k7cAlnHzaaAD5QjCReCCHHG1Zvb9v5br/6Z7sXoP K9ECE5GxUVbXgVEaB4tMBnO0V9lAEsc25L7X1GNpoSlPk1tSQ3TI/9g92DHhbNuHlxlVjj ms9vzdBhBYcBzTJ88x89w/BbvVvyB+/rdMptLZ04H5SM3ICPmAsekNiplhT+Gg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D258481 Allan Jude changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|jail@FreeBSD.org |allanjude@FreeBSD.org CC| |allanjude@FreeBSD.org --- Comment #1 from Allan Jude --- I am looking into this --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Oct 12 00:50:38 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MnDbL2Xtqz4dsJt for ; Wed, 12 Oct 2022 00:50:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MnDbL0B52z3tLM for ; Wed, 12 Oct 2022 00:50:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4MnDbK6NjGz18FT for ; Wed, 12 Oct 2022 00:50:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 29C0obI0072278 for ; Wed, 12 Oct 2022 00:50:37 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 29C0obRo072277 for jail@FreeBSD.org; Wed, 12 Oct 2022 00:50:37 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 245633] Panic when shutting down jail in vnet_if_return->if_vmove->if_delgroups Date: Wed, 12 Oct 2022 00:50:38 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.1-RELEASE X-Bugzilla-Keywords: crash, panic X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: koobs@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: keywords Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1665535838; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/K9clhizrwWmwisHqyOcKxUOYglMhRr/cbfZWeiVPBQ=; b=oYHjcraNsA6g5zVHySgN37RAhVHrJavkj3JU070Si6HiJ+PfcWk8smyA09C5CZntP3hGHJ 46e8rX/Wy6tXvIGSAC5nq8xHq/qNGs59xTOZvMk8WKc4TUYrfGaFYFcPdqOpVhp/s97d2A 0DklY9gD9mnVXxFMijvaJD+vDD4jOInG8ItPlP1+Jz6Sr1zuAQ9ClkyGGs6vd0D6hIpqyj w1PVu5RXY/+eehSnS4wXVcJdmJd6N2cKAPfO1YH2Bl2BNhJwcLzQuCl5BSpJO6p+N9N/vV fEXvBu3jv56rtqTN8DGwphmoIQTVV0lRurwn21Bx7lMd3HqT5OC3chaSwQEDDg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1665535838; a=rsa-sha256; cv=none; b=I9aOYgkfDFiJbd2IDcYkK1ziMaDHIXGN2U6SykzT4Id6r4WuxAzdu7ycmRlvhy+Fs+q5DE t8mjxlGEVpp6LZf8NKodE/WffrdMdIqLvHddhG/x6D23e1x+WbfduBJKqvZT6mJEwSaeir RmSUCTjz8rMpkio6y8pCyRmybK6g6b14nwZYqxyUYhLoTfm1jIUyKai8yF+CKaU+3TvVMv wczaDv6GnRaS8/oY7dxPIGSNoBRMFNL5V/Ev1WmoCIcd9wIlU07KQ6ffSwq8HZ61zkNPM8 Q+d6OZZydGRpTATafbSt250f9F3NEFTkSEV4ex35Gtz+Kn8EuHHkcHUmoKsm5g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D245633 Kubilay Kocak changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |crash --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Oct 12 06:19:00 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MnMtD62JDz4fW5h for ; Wed, 12 Oct 2022 06:19:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MnMtD39MLz3Ggv for ; Wed, 12 Oct 2022 06:19:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4MnMtD2CQtz1DBj for ; Wed, 12 Oct 2022 06:19:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 29C6J0Wr079721 for ; Wed, 12 Oct 2022 06:19:00 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 29C6J0Z1079720 for jail@FreeBSD.org; Wed, 12 Oct 2022 06:19:00 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 245633] Panic when shutting down jail in vnet_if_return->if_vmove->if_delgroups Date: Wed, 12 Oct 2022 06:19:00 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.1-RELEASE X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: freebsd-bugs@virtualtec.ch X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1665555540; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jpF2mHUmmP1jgJnVc/0xeWT3kUl5dtHOACRbqHIlgLk=; b=gXZac3YQjjP8oVJir/0f1gjNptSakwh0dhMGQ//6TaLgJPV4DvjRNcWaStbIZOCSU//uOk ylctY0Et0QJ2SUJ4zwXmjaduvDwR4+vsFA7GwVXsOPNOAym07DhCF7cI+LDLPdgNeTdAEz SSdqC79010+LVXGIqMgSbkJG/diPxKL7WM5m7aAIgeBVpisu2ux56ylc/DfETjxQWBDaid 3OlcRa//e4W2wyI5ZOwReZSpNXfCEe/mWFZfywUjS4b3+OCPoOO2304acJhMiZ0wU8jpEc ORfsczaCJA+B+zxYe0zu7VSZ++Zmp12lmZUVCUWlcYdJJnrTwhEDSwma1TNsIg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1665555540; a=rsa-sha256; cv=none; b=S/3qclG0bPSwIcVArqdXHeOWXXpW8EDvorJibqoZBF4SUu0yaCF0zRk48I6HSh/RHxyVPE RO4VEonKkVRxWXAnGDpxXL9i4IJeRVJ/uK6A71YVmBD/nuk732DY6PIT50tU0cfkquKlcL 1qAa3wfYzpOD1/08eXNBPzFpxyc9VXMuFcRZ6BhQjGj1vl10oky0zEbv8cbNwv13V6toRd 94G8xFvljs/beMt/zjYCztxVy58lthqufVqY7nxGYkmP7vVe/zCfwkQt4aSklfrVhk9w6S qrA82P9ZBN/TKcDxadXADl1UlvwsiX5pAo888jYUyxR+zo742zUQwtKa1x8Wfw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D245633 --- Comment #6 from Markus Wild --- didn't realize this was still open. we fixed the issue for us with by adding=20 exec.prestop =3D "ifconfig epair0b -vnet JAILNAME"; to our jail.conf configs. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Mon Oct 17 12:35:27 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Mrc0H71JDz4fnjJ for ; Mon, 17 Oct 2022 12:35:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Mrc0H4N2kz3HJp for ; Mon, 17 Oct 2022 12:35:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Mrc0H3Nl5zpxN for ; Mon, 17 Oct 2022 12:35:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 29HCZRvB029842 for ; Mon, 17 Oct 2022 12:35:27 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 29HCZRLI029841 for jail@FreeBSD.org; Mon, 17 Oct 2022 12:35:27 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 180067] [jail] [patch] fix multicast support within jails Date: Mon, 17 Oct 2022 12:35:27 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.1-STABLE X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: grahamperrin@freebsd.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: keywords Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1666010127; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZQvqO7tDPPRGaNfPIrkrlD4Tu/5ZIlorzcNN33WlG7M=; b=cCMgZOwzk2vqopjHJIRbDvq2mKfEsxpOikS2SZylUVNuAvTk5vmFy4fp4oaHlDx+SpyfQf udw+JFYCpiMVDnmC/M5Fw0GJ86NrEprVMhSnueruMQAb8XzGJN1R0GcQWUHy0LmujqZJru lw+hR72u2BvSR5I4FaCNoL49ImJyXLU8CkQ0LwaD2H6sQ/IqtbXTQLSnnDuZ8TzkwPvWCE Ci5QwhWMF6VLnkdaQNDKDuPT4CZbRziqR+zKPeB+oYwrdN6qq76sfDi7fWVkRN8KkvTOl6 ha+NPzvbHLvHSOwgIes58B4+aEOOJrWdj3eijekC8z7QoLkXJuz9r51sv484Ww== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1666010127; a=rsa-sha256; cv=none; b=SuTyVdNDLuFcbJSPQRq5iwBYsUyVDZNhLlJvNt3gaLa040U/reIktnijXW6LJDuC2Q460W sSRqBl1QPGjFqxpenuPBxjKcMZlFLch57MgoyZG53wrsVloj1ucoizTA5PA2aL/0kXvWl4 mROlyNuMDap1J8CFB8/jyIPl0VAH+6HbW8tIRPDlTN5buRRsZy7G4h4QFzl4RkmD5mSdzm OXw2NGGFGUAo6Rl5X9SNQ8HGbwHjL3w92HNWEWcVELz13nv/pa36Kc31qvs5AFZtP5lPVu lFeLIRqDYGRgcjAiCMqTb9HOITBgj13jXvgkoYG1hvaySqqhgT2nJClYvfQbfA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D180067 Graham Perrin changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |patch --- Comment #3 from Graham Perrin --- Keyword:=20 patch or patch-ready =E2=80=93 in lieu of summary line prefix:=20 [patch] * bulk change for the keyword * summary lines may be edited manually (not in bulk).=20 Keyword descriptions and search interface:=20 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Mon Oct 17 12:40:48 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Mrc6S5C4cz4frv5 for ; Mon, 17 Oct 2022 12:40:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Mrc6S3ppYz3kfH for ; Mon, 17 Oct 2022 12:40:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Mrc6S2VKBzqWJ for ; Mon, 17 Oct 2022 12:40:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 29HCem4B039707 for ; Mon, 17 Oct 2022 12:40:48 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 29HCemcs039706 for jail@FreeBSD.org; Mon, 17 Oct 2022 12:40:48 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 155765] [patch] 'buildworld' does not honors WITHOUT_JAIL Date: Mon, 17 Oct 2022 12:40:48 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: misc X-Bugzilla-Version: Unspecified X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: grahamperrin@freebsd.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: kevans@freebsd.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: keywords Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1666010448; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hFSgirUsLnbGLcua93izmKUJ/zdZp5Dq4ajjx7NXeW4=; b=uTcGclHR70GH0I+Li6XHUHqN893iSjAuULK99CAzteiHi+STA87v1zcl9Uz2CyuIv76JnO Uyf5TuB+hZnHxqkNmDbxumfx/rnCaA3k+yB5Pxg3We6u+osvI6tNfYD4/F8ygco2WMIeQP Bra2VN/+xpdeDz5tgSMsYaaijvnoPN9zsjwWHELCJUSGEiAc7pRMNYot3yypExBThtq6aZ plIKiBAEtVZfJKpElu4VAHExDI87nGmDjqXDzIt7WyL62TopyY6uH3c8hW+PE30+RB0cZk ab1F9bJUrNHBcIsIatjJ1oYJij9yCEBVfsnv0uSjm7CnRz5W8xbPC6zlyCwlkA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1666010448; a=rsa-sha256; cv=none; b=BZXzgDuxrKY1fo3G79VlIAjMSPWzKiWCKrC8EnJE3Zm8NMVkdJfYc4p/mmCak8aYgkRA+b 6C7D7JsHuWV75AD7ksRsn9ijtrYrRPYZhbFILoc6/FoB+MepT3wQJUgrDfH/pzL/LtuH49 +OGx8tgKrFWlPkiTky+Xixh4TqTHT78fHDpb4Q3/yZQtyQJ8o6ycyW29OhNFF53+ZV3nCt hLnHwwLwCk6BwckqtK3dSaqy29cKCJFEFPOJipT+x8Ulpel3mNp2V89HcaBcGqWhbBnVao uPT1gm8qFDp54JU96GjZSOI+DtEzOt5gMSz9H6B+6bVHUTlsRpMJH/NjEbA6cQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D155765 Graham Perrin changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |patch --- Comment #4 from Graham Perrin --- Keyword:=20 patch or patch-ready =E2=80=93 in lieu of summary line prefix:=20 [patch] * bulk change for the keyword * summary lines may be edited manually (not in bulk).=20 Keyword descriptions and search interface:=20 --=20 You are receiving this mail because: You are on the CC list for the bug.= From nobody Mon Oct 17 19:47:55 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MrnbH3RTQz4fJFW for ; Mon, 17 Oct 2022 19:47:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MrnbH2G1zz41Kk for ; Mon, 17 Oct 2022 19:47:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4MrnbH14X3zwRP for ; Mon, 17 Oct 2022 19:47:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 29HJltLt060940 for ; Mon, 17 Oct 2022 19:47:55 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 29HJltZu060939 for jail@FreeBSD.org; Mon, 17 Oct 2022 19:47:55 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 248029] Allow ability to use socket option SO_REUSEPORT_LB in jail Date: Mon, 17 Oct 2022 19:47:55 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-STABLE X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: markj@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: markj@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc bug_status assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1666036075; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9u5pLLkGnlzuC3dS8zZQ7e9aXRNigGlkfuLQDhipdQc=; b=UNWO3xp+lnVFauVIWN7JbJsgwzv3X0TPx7YEZtqwYvGHoxjWfaK1SxxW2wEktzqJ6gILhI 39QFFvWAOV1WvBjHbLhH/zg0BFiW57Aq/CAdWGXmKyur3rnfklilGJJcVMb285mOqCyAc+ qzyTN/LU0I1eOWc6A2vTRBwghurKo0oLNyvoDzB2RYbPFlhKVrDKmVdHckXIccZl4UEL8l bascwL8Sz6Ko87P0FQn4Ch6pC523y+IRqgsw7me8cHiAJiswlfE5iHL/WfF3qJa2g6zQOR qI6ECnpjk3r079HCTESc/9yq7RTclu+UZ0YavVVGM6c7R3mOwNFkHI/Fd7TFWA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1666036075; a=rsa-sha256; cv=none; b=qm6ANGqJtYxl3a6J2Fqx9abqGBAwVWRpBCpo/b30vE3AelAzPApaQiSVz5DX8IdAXA7KIX 7doDj/6uBvhueHOZ9N+ac/y0CY3A+bdlJTNoS/gWKXYoKqc7dCtsfsWOyYS3d9EtsCNUEW nIKuRDkCX/KQj6/GTN8nzGXB6yqY5ySJ/e8uvyPXMkjJDPcfcdF6UCBFdtgg2wOfW/IKzd 1UZUHafcojeQHbJpPSeMsA+rLP0lIYM75OouzCe7E7ImJz+yWtC1aT/hQwOmPjmRWaMIXO edbNLimD3bhY2I5DaKOfX+K3PQbBEZ4uBRAGNjmLOCU3LjOHLF27Hn9mzs7ZUg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D248029 Mark Johnston changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |markj@FreeBSD.org Status|New |Open Assignee|jail@FreeBSD.org |markj@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Dec 13 18:00:24 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NWmW620sMz4kP66 for ; Tue, 13 Dec 2022 18:00:34 +0000 (UTC) (envelope-from jamie@freebsd.org) Received: from gritton.org (gritton.org [162.220.209.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "gritton.org", Issuer "gritton.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NWmW43QYRz4BHq; Tue, 13 Dec 2022 18:00:32 +0000 (UTC) (envelope-from jamie@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=softfail (mx1.freebsd.org: 162.220.209.3 is neither permitted nor denied by domain of jamie@freebsd.org) smtp.mailfrom=jamie@freebsd.org; dmarc=none Received: from gritton.org ([127.0.0.3]) (authenticated bits=0) by gritton.org (8.16.1/8.16.1) with ESMTPA id 2BDI0Ote014877; Tue, 13 Dec 2022 10:00:24 -0800 (PST) (envelope-from jamie@freebsd.org) List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Date: Tue, 13 Dec 2022 10:00:24 -0800 From: James Gritton To: jail@freebsd.org Cc: bz@freebsd.org, "glebius@FreeBSD.org" , Andrew Gallatin Subject: Re: prison_flag() check in hot path of in_pcblookup() In-Reply-To: References: User-Agent: Roundcube Webmail/1.4.11 Message-ID: X-Sender: jamie@freebsd.org Content-Type: multipart/alternative; boundary="=_3054962f98fc689e6f81a2c8ac68acda" X-Spamd-Result: default: False [-3.10 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MLMMJ_DEST(0.00)[jail@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; R_DKIM_NA(0.00)[]; FREEMAIL_CC(0.00)[freebsd.org,gmail.com]; ASN(0.00)[asn:30247, ipnet:162.220.208.0/22, country:US]; MID_RHS_MATCH_FROM(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; DMARC_NA(0.00)[freebsd.org]; RCPT_COUNT_THREE(0.00)[4]; FREEFALL_USER(0.00)[jamie]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; TO_DN_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4NWmW43QYRz4BHq X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N --=_3054962f98fc689e6f81a2c8ac68acda Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII; format=flowed On 2022-12-13 09:18, Andrew Gallatin wrote: > I was trying to improve the performance of in_pcblookup(), as it is a > very hot path for us (Netflix). One thing I noticed was the > prison_flag() check in in_pcblookup_hash_locked() can cause a cache > miss just by deref'ing the cred pointer, and it can also cause multiple > misses in tables with collisions by causing us to walk the entire chain > even after finding a perfect match. > > I'm curious why this check is needed. Can you explain it to me? It > originated in this commit: > > commit 413628a7e3d23a897cd959638d325395e4c9691b > Author: Bjoern A. Zeeb > Date: Sat Nov 29 14:32:14 2008 +0000 > > MFp4: > Bring in updated jail support from bz_jail branch. > > This enhances the current jail implementation to permit multiple > addresses per jail. In addtion to IPv4, IPv6 is supported as well. > > My thinking is that a jail will either use the host IP, and share its > port space, or it will have its own IP entirely (but I know nothing > about jails). In either case, a perfect 4-tuple match should be enough > to uniquely identify the connection. > > Even if this somehow is not the case and we have multiple connections > somehow sharing the same 4-tuple, how does checking the prison flag > help us? It would prefer the jailed connection over the non jailed, > but that would shadow a host connection. And if we had 2 jails sharing > the same 4-tuple, the first jail would win. > > I can't see how this check is doing anything useful, so I'd very much > like to remove this check if possible. Untested patch attached. For a complete 4-tuple, it should indeed be the case that a match would only ever identify a single prison. The later part of the function that examines wildcards definitely needs the check. I don't get the XXX comment about both being bound with SO_REUSEPORT, because I would only expect that to apply to listening, not to full connections. But I also expect Bjoern to know more than I do here... - Jamie --=_3054962f98fc689e6f81a2c8ac68acda Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=UTF-8

On 2022-12-13 09:18, Andrew Gallatin wrote:

I was trying to improve the performance of in_pcblookup(), as it is a = very hot path for us (Netflix). One thing I noticed was the prison_flag() c= heck in in_pcblookup_hash_locked() can cause a cache miss just by deref'ing the cred pointer, and it can = also cause multiple misses in tables with collisions by causing us to walk = the entire chain even after finding a perfect match.
 
I'm curious why this check is needed.  Can you explain it to me?&= nbsp; It originated in this commit:
 
commit 413628a7e3d23a897cd959638d32539=
5e4c9691b
Author: Bjoern A. Zeeb <bz@FreeBSD.org>
Date:   Sat Nov 29 14:32:14 2008 +0000

    MFp4:
      Bring in updated jail support from bz_jail branch.
   =20
    This enhances the current jail implementation to permit multiple
    addresses per jail. In addtion to IPv4, IPv6 is supported as well.
 
My thinking is that a jail will either use the host IP, and share its = port space, or it will have its own IP entirely (but I know nothing about j= ails).  In either case, a perfect 4-tuple match should be enough to un= iquely identify the connection.    
 
Even if this somehow is not the case and we have multiple connections = somehow sharing the same 4-tuple, how does checking the prison flag help us= ?  It would prefer the jailed connection over the non jailed, but that= would shadow a host connection.  And if we had 2 jails sharing the sa= me 4-tuple, the first jail would win.
 
I can't see how this check is doing anything useful, so I'd very much = like to remove this check if possible.   Untested patch attached= =2E
 
For a complete 4-tuple, it should indeed be the case that a match woul= d only ever identify a single prison.  The later part of the function = that examines wildcards definitely needs the check.  I don't get the X= XX comment about both being bound with SO_REUSEPORT, because I would only e= xpect that to apply to listening, not to full connections. But I also expec= t Bjoern to know more than I do here...
 
- Jamie
--=_3054962f98fc689e6f81a2c8ac68acda-- From nobody Tue Dec 13 19:03:54 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NWnwJ3sRqz4kX2T for ; Tue, 13 Dec 2022 19:04:00 +0000 (UTC) (envelope-from bz@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NWnwJ3MTRz4L4P; Tue, 13 Dec 2022 19:04:00 +0000 (UTC) (envelope-from bz@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1670958240; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=6axhjV1lfdZJTvgvxK6oaS4V8FvbdhFI8oSErcJdZ0w=; b=o3Q+ObL0apbc+viEiGn7XK1vMgBygywFgCfM4rcxnzlA3ZnGfpE6Axg/KZEC865mJP24JI dlDw7bh+svR1vGdMXnUIWNONlJPpDXOzaJMipq8zcy4utW6U63pGZ9BIsdKDVr3jfF5Ny0 TSc7RcOMtGX+j+hTQdArDgO6433eDXlnyybCBkCLt43dZRaH9e1qepi95oiSfoUWcBI7xU yvwHrawSvxXvodIDY5wmgG5+W8yh0SxzAIvuaLJzNMOfyKibHQc5qQtydPbRWOw6m6qw0f f93dcD552qZoT2NJJgx7AYzmd+Zu2zqtkv0AqJOVB9Gs7Kqt61EFxrrhB7Vr4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1670958240; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=6axhjV1lfdZJTvgvxK6oaS4V8FvbdhFI8oSErcJdZ0w=; b=Twu3jIB21VXaKH4C5+b1vldadR250BOiNJUCpa++dAdkZcKTD05lXI/m9tE/RDRG6h9i9Y zRtEe3Ho3CwXczhQq3HVlgI1OTJbZZQT8D3u24j5ApXqdnXxn7JuFkZ7d9PJnkuVhJGP0I Dj83N+LCnGl/T+8q3lGZ9zSQ5v/TwMXtxN93b538yzAhumJith2kCVggkhJleFG5nywurM PLccB7iXMh+yllXRV7mpMzD8/35UO15DjFbIZgiqPGJJX0m54ql3IsriiyKkHgBIY845BB WfsKt4KtMLpg5HK+X2mwYSeHaxRfb11bCQjUVQzrKOXN7ec1VWQ1MbDQhdN0Sg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1670958240; a=rsa-sha256; cv=none; b=vAEvbxcyoQBG1LwSvR1tksEYntwj6xKMZ7r7Ti5+RjD7unzWJgVekGwPJixweHAZlikmbI MY9GNl4mEF8dhmCbRYDOJejQJpOrM6aApV7G/eDIRVOgZBHKY2XK9le6dJevmhWvIPzqiA GBzNen9egYVYrVJvec+knWv74MKsHWxSkcmmN/MAMnofpSINxrYD1b6qx1SpEBOreH8E+B sBXWLr1bp8m62NavGtZoaTH110ljm/uoHS8LROG3j46YhUcFrg8Rknn4EVTiHpZsUUnqGJ WT8vlR9F/2CXOtpfO3UmBj6wgsq13KBPrigdHjGSOxXr8J6hvP3e9lqzzfiEbw== Received: from mx1.sbone.de (cross.sbone.de [195.201.62.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) (Authenticated sender: bz/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4NWnwJ1hCFzNK5; Tue, 13 Dec 2022 19:04:00 +0000 (UTC) (envelope-from bz@freebsd.org) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id C1BE38D4A162; Tue, 13 Dec 2022 19:03:58 +0000 (UTC) Received: from content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id DF6275C3A833; Tue, 13 Dec 2022 19:03:57 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) by content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (amavisd-new, port 10024) with ESMTP id uoSbzvEqXc0B; Tue, 13 Dec 2022 19:03:55 +0000 (UTC) Received: from strong-iwl0.sbone.de (strong-iwl0.sbone.de [IPv6:fde9:577b:c1a9:4902:b66b:fcff:fef3:e3d2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id 5E39B5C3A830; Tue, 13 Dec 2022 19:03:55 +0000 (UTC) Date: Tue, 13 Dec 2022 19:03:54 +0000 (UTC) From: "Bjoern A. Zeeb" To: James Gritton cc: jail@freebsd.org, "glebius@FreeBSD.org" , Andrew Gallatin Subject: Re: prison_flag() check in hot path of in_pcblookup() In-Reply-To: Message-ID: <89pn26q0-pps9-q8n7-1334-q15o5896p6p@serrofq.bet> References: X-OpenPGP-Key-Id: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-ThisMailContainsUnwantedMimeParts: N On Tue, 13 Dec 2022, James Gritton wrote: Hi, Argh, sorry Drew, I looked at the wrong of the two checks in the function earlier. Sorry, that's what happens if trying to be helpful when firefighting elsewhere. > On 2022-12-13 09:18, Andrew Gallatin wrote: > >> I was trying to improve the performance of in_pcblookup(), as it is a very >> hot path for us (Netflix). One thing I noticed was the prison_flag() check >> in in_pcblookup_hash_locked() can cause a cache miss just by deref'ing the >> cred pointer, and it can also cause multiple misses in tables with >> collisions by causing us to walk the entire chain even after finding a >> perfect match. >> >> I'm curious why this check is needed. Can you explain it to me? It >> originated in this commit: >> >> commit 413628a7e3d23a897cd959638d325395e4c9691b >> Author: Bjoern A. Zeeb >> Date: Sat Nov 29 14:32:14 2008 +0000 >> >> MFp4: >> Bring in updated jail support from bz_jail branch. >> >> This enhances the current jail implementation to permit multiple >> addresses per jail. In addtion to IPv4, IPv6 is supported as well. >> >> My thinking is that a jail will either use the host IP, and share its port >> space, or it will have its own IP entirely (but I know nothing about >> jails). Well having its own IP address entirely doesn't work with classic jails as there is only one network stack where they can be configured on an interface, and that is the base system. So de-facto all jail address/port space is always shared with the host (ignoring vnet jails with their own entire virtual network stack). Whether the host would bind/listen is a different story. I know 15 years ago people would bind the sshd of the base to a single-IP address which was not assigned to jails and then the sshd inside a jail would bind to a different (single) IP address. If one doesn't do that and the sshd inside the jail isn't runnig one would end up trying to connect to the sshd in the base system (sshd being a popular example). Not sure if people still do but that's still the case. There are special cases with classic multi-IP jails in that they then cannot have overlapping IP-ranges as otherwise it would not be deterministic which jail would get an inbound connection on inaddr_any:port_n if two jails were to listen on the same port_n. Hope that helps for basic understanding. >> In either case, a perfect 4-tuple match should be enough to >> uniquely identify the connection. >> >> Even if this somehow is not the case and we have multiple connections >> somehow sharing the same 4-tuple, how does checking the prison flag help >> us? That logic predates me and came from [1]. The jail_jailed_sockets_first sysctl got removed in the review process with rwatson. I am still trying to see where the SO_REUSEPORT comment (back then) came from. I know I only had the first lines initially, so must have been sometime during review with rwatson as well. Sadly p4 emails where truncated to 1000 lines so I cannot simply grep for the change (if it is in my mail archives) or had a useful commit message (but at least would give a date to check further private email). My current guess is that if we have the 4-tuple in both the base and a jail (hence the SO_REUSEPORT comment) we want the jail not getting a socket of the base system returned as that would mean one could "break out of prison". But if the inp belongs to a jail we know we can simply return. So if you find the one of the base system first you'll have to go and look through the others. XXX-jamie: is that all still true in hierarchical jails? Now whether fabricating the case of having both inps on the hash is still theorectically possible I cannot say. I haven't followed all the changes to that code lately close enough. >> It would prefer the jailed connection over the non jailed, but that >> would shadow a host connection. And if we had 2 jails sharing the same >> 4-tuple, the first jail would win. >> >> I can't see how this check is doing anything useful, so I'd very much like >> to remove this check if possible. Untested patch attached. > > For a complete 4-tuple, it should indeed be the case that a match would only > ever identify a single prison. The later part of the function that examines > wildcards definitely needs the check. I don't get the XXX comment about both > being bound with SO_REUSEPORT, because I would only expect that to apply to > listening, not to full connections. But I also expect Bjoern to know more > than I do here... /bz [1] https://people.freebsd.org/~pjd/patches/jail_2004120901.patch -- Bjoern A. Zeeb r15:7 From nobody Tue Dec 13 19:05:32 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NWny73NCxz4kX0R for ; Tue, 13 Dec 2022 19:05:35 +0000 (UTC) (envelope-from bz@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NWny72vFNz4MTC; Tue, 13 Dec 2022 19:05:35 +0000 (UTC) (envelope-from bz@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1670958335; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=/vBgwDf4Hzhq5IiRGfOrvugYCUWzVQP7fSn8ijOlcOk=; b=qjulhhZSN43evRFqRRyF6/mn2+QWzAWV/qwfBMliJUL+P0o3gz/BFQH6f1X7B8qGc+IrVM 131r7v0IbgqjxCKHTJrRkIotQElioSIsnXzNOlSElOCJsrQfr0gytbMPwFm0eWQhraa1rZ 6uXFa/UEsskEtmCt/EKZm0dD0yQ2149wrbjT0k44yzqSscaGCX1btsGYcCNXUeGhGdTfQx Y0IXPTatIbOGT1ucAxX0MCHml+2OftzgsXtUodE8pGTducLRbrepZ+3rxoWKH5SBp8nbqK Hiovbew9YyLM1IWMY1zxJEqR5PG6c9gzjm2M5fCCBl/rPGS9eZ5Ilp1l/EDgdA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1670958335; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=/vBgwDf4Hzhq5IiRGfOrvugYCUWzVQP7fSn8ijOlcOk=; b=OYo9h8QSBpklUEvdQfgW0WIB2xCVbfVYtYaI2FBKnwqoB1JlVFroqY0GoCpb/ARtxEXOKE 2yB7t1g+QlCPbtnA/1pOZ+K58kEE0Bn9I67qVfd6DlxgjNkQi1nK1t6mSIkNqthx9ioUe6 HCz/LVWwjLnTpTG7ONAIq4I3l6aEEQPPaYOPTAhT51rxYj8WoTjD6+HMmhG3bmMVLdWwlt Wc+fzfvxKTjaueJTabOlfimaCxwvJmqskd4SLlR7RgKUeKuemG4jRG5lGnAIEs6PYVVRc6 EwLUM47A/kDFCOJa5woLyjMMSxG2Wy2RHpZVLFSvsSSFTZzCZBjhKhhwJr0g3A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1670958335; a=rsa-sha256; cv=none; b=MPDsTcIkGg3jS2jAkR59hx6Adr7Pkd9iye21qJLL1kaBTb8a7h3fk4vHske52wYzLNtmbf MaRJbGlkNSJGd6hZhYZNY5faHs9Su9ePVRwKnj++oKeeGahD6f4+bHpCmA0wmCwNKaxDep Y0EGdqYeS2aGBrbbPaz/SrnWX3mf6gumJUI99X/US+x23ZFH9jPX4O2r/bzC/70u27qXn4 m3glaEwzdVARQE/5R+l2K7AFzL+wJhCbCUojxZTbV9x1zcob+Z5PDDG8/q56rQlQ1gk05R b04djGJuuF7vbkGnu/yC1NKoVtMwwDyeGgf9gPB9RIKT5iYYDvA+Q3RtFKLpIA== Received: from mx1.sbone.de (cross.sbone.de [195.201.62.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) (Authenticated sender: bz/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4NWny71L1XzMHx; Tue, 13 Dec 2022 19:05:35 +0000 (UTC) (envelope-from bz@freebsd.org) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id E9BE68D4A129; Tue, 13 Dec 2022 19:05:33 +0000 (UTC) Received: from content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id 72EDB5C3A833; Tue, 13 Dec 2022 19:05:33 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) by content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (amavisd-new, port 10024) with ESMTP id 7sL3MnGxJFSm; Tue, 13 Dec 2022 19:05:32 +0000 (UTC) Received: from strong-iwl0.sbone.de (strong-iwl0.sbone.de [IPv6:fde9:577b:c1a9:4902:b66b:fcff:fef3:e3d2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id 78F365C3A830; Tue, 13 Dec 2022 19:05:32 +0000 (UTC) Date: Tue, 13 Dec 2022 19:05:32 +0000 (UTC) From: "Bjoern A. Zeeb" To: Andrew Gallatin cc: James Gritton , jail@freebsd.org, "glebius@FreeBSD.org" Subject: Re: prison_flag() check in hot path of in_pcblookup() In-Reply-To: Message-ID: <6on81os3-501-s5n2-8nos-p85n8op23232@serrofq.bet> References: X-OpenPGP-Key-Id: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-ThisMailContainsUnwantedMimeParts: N On Tue, 13 Dec 2022, Andrew Gallatin wrote: > Are there regression tests for jails where this patch could be run to > ensure it is safe? Not that I now of but it would certainly good to have one for that case. But it's likely not going to be deterministic so the question will be more of the case "theoretically possible or not"? -- Bjoern A. Zeeb r15:7 From nobody Tue Dec 13 19:43:22 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NWpnm3j54z4kc0G for ; Tue, 13 Dec 2022 19:43:24 +0000 (UTC) (envelope-from jamie@freebsd.org) Received: from gritton.org (gritton.org [162.220.209.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "gritton.org", Issuer "gritton.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NWpnm2jhQz3CT0; Tue, 13 Dec 2022 19:43:24 +0000 (UTC) (envelope-from jamie@freebsd.org) Authentication-Results: mx1.freebsd.org; none Received: from gritton.org ([127.0.0.3]) (authenticated bits=0) by gritton.org (8.16.1/8.16.1) with ESMTPA id 2BDJhM9R026133; Tue, 13 Dec 2022 11:43:22 -0800 (PST) (envelope-from jamie@freebsd.org) List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Date: Tue, 13 Dec 2022 11:43:22 -0800 From: James Gritton To: jail@freebsd.org Cc: "glebius@FreeBSD.org" , Andrew Gallatin , "Bjoern A. Zeeb" Subject: Re: prison_flag() check in hot path of in_pcblookup() In-Reply-To: <89pn26q0-pps9-q8n7-1334-q15o5896p6p@serrofq.bet> References: <89pn26q0-pps9-q8n7-1334-q15o5896p6p@serrofq.bet> User-Agent: Roundcube Webmail/1.4.11 Message-ID: <2a96726f498df08c57bf54eff2afc960@freebsd.org> X-Sender: jamie@freebsd.org Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4NWpnm2jhQz3CT0 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:30247, ipnet:162.220.208.0/22, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On 2022-12-13 11:03, Bjoern A. Zeeb wrote: >>> In either case, a perfect 4-tuple match should be enough to uniquely >>> identify the connection. >>> >>> Even if this somehow is not the case and we have multiple connections >>> somehow sharing the same 4-tuple, how does checking the prison flag >>> help us? > > That logic predates me and came from [1]. The > jail_jailed_sockets_first > sysctl got removed in the review process with rwatson. I am still > trying > to see where the SO_REUSEPORT comment (back then) came from. I know I > only had the first lines initially, so must have been sometime during > review with rwatson as well. Sadly p4 emails where truncated to 1000 > lines so I cannot simply grep for the change (if it is in my mail > archives) or had a useful commit message (but at least would give a > date to check further private email). > > My current guess is that if we have the 4-tuple in both the base > and a jail (hence the SO_REUSEPORT comment) we want the jail not > getting > a socket of the base system returned as that would mean one could > "break > out of prison". But if the inp belongs to a jail we know we can simply > return. So if you find the one of the base system first you'll have to > go and look through the others. > > XXX-jamie: is that all still true in hierarchical jails? I believe so... Multiple jails in a hierarchy can share the same single IP address, but then you also could always have multiple non-hierarchical jails sharing the same single IP address. So in the single-address case, hierarchy doesn't matter. prison_ip_conflict_check() notably doesn't distinguish parent jails from the broader class of "other jails," which means that only the first-level jails in a hierarchy can have multiple IP addresses. So the multi-address case doesn't apply to hierarchical jails. - Jamie From nobody Tue Dec 13 23:03:42 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NWvF02KBzz4jnnQ for ; Tue, 13 Dec 2022 23:03:48 +0000 (UTC) (envelope-from bz@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NWvF01ZS9z3tkq for ; Tue, 13 Dec 2022 23:03:48 +0000 (UTC) (envelope-from bz@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1670972628; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=T9mvYqF2fYGdBfTOLloM25RVIXK3aUjtKbeZy+G1YOo=; b=asnguyhnSzX0QJOV9pVtrLR3hrS/GyCZ/CZBraFYyKPtk1m9cyuINEV7JCXt7g6+fO4OhW UH0mo/zu5j3eBi5Pen5fz6hOBW5Z6GK8SW2hdTb7uCzQSSr9URxSwFNEZzbDwg/7/bU7DS 25yATmXeeiC5R+bxmo+bamGYtA8KKQpdDkLqF3Pj9vTyeGO3VJKLuy03YpdPXASSZ89W/B dYuiuMrt+/k51Uq6xGoNwZxbNgUKAECRSeftzQwxEVFdopBj9ntqsZ7aSzHcZaP6XbSJcL 5RNI4Av2EK/e8tfT8bcC+2CfHEWjR4uawBPs/eGYjsNPlArvQM8XHVSrq28XwA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1670972628; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=T9mvYqF2fYGdBfTOLloM25RVIXK3aUjtKbeZy+G1YOo=; b=G00zl57zhCztCvI5+1Tlz/sN1LJHhh5FMfWWD3aOYou2F5N36UYqb+wqoOrhvu2387ryMm zmMinZ2pn0DO5WqfkqOvfIqVVmrcFJOmRs2aCafvN8AEcFmjD9e2/u8p3A8eWSjIkNWywP /W0UvnA98RDy6PS4+yOsi9lylggOcV/RTAZGSNbquNDu+ly137ovRcJEafdBEj5U17ciZg XOcsDYeYk2Hb5WW5/weLA8P/4S9Cs4RYpRxqKQS/OkSemh2Nw1RXGPr8UlvVyO3ivsRTwP iAQ2Wcb5YGIefeEEXZR53HpurtqxHVKQNolKKLOPadxj0Sbw7Zi99g9RBT4V4g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1670972628; a=rsa-sha256; cv=none; b=OKpW/7SQWzpaaOodnDreV+0qX1zLwRfiotLhVEL/bOAXVz4XN8hlNad6eLBVyoSRmIotlH zvyX7sCvOwGrZkmrQcubReWNp6uHVQA3qg44c0ed0Opci9amtquW3HuW21kKh2/p6rbK2h xB6Deq+kdHw4XVB3+HTdifWndnuHmgGT1GSsaZaqbo13XTYuNj8NMeg9uKKrKodptAltLN /Ytb70IB4rIkasCp1dwCLZR9cPxIPaxJlqpj0MLAbMQEw/daW9boPc/8QeC/jFgFe8WFx8 S6UywsRS3y25ezOJrzAynqcJ0vuEpOKS7ddZt7FdlddHxpk17pLKNt7psZ+AJQ== Received: from mx1.sbone.de (mx1.sbone.de [IPv6:2a01:4f8:13b:39f::9f:25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) (Authenticated sender: bz/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4NWvDz6ZCJzRKQ for ; Tue, 13 Dec 2022 23:03:47 +0000 (UTC) (envelope-from bz@FreeBSD.org) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id D026B8D4A129 for ; Tue, 13 Dec 2022 23:03:45 +0000 (UTC) Received: from content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id 429655C3A833 for ; Tue, 13 Dec 2022 23:03:45 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) by content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (amavisd-new, port 10024) with ESMTP id GFoyLQsKt5LT for ; Tue, 13 Dec 2022 23:03:43 +0000 (UTC) Received: from strong-iwl0.sbone.de (strong-iwl0.sbone.de [IPv6:fde9:577b:c1a9:4902:b66b:fcff:fef3:e3d2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id 5993A5C3A830 for ; Tue, 13 Dec 2022 23:03:43 +0000 (UTC) Date: Tue, 13 Dec 2022 23:03:42 +0000 (UTC) From: "Bjoern A. Zeeb" To: freebsd-jail@FreeBSD.org Subject: What's going on with vnets and epairs w/ addresses? Message-ID: <5r22os7n-ro15-27q-r356-rps331o06so5@mnoonqbm.arg> X-OpenPGP-Key-Id: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset=US-ASCII X-ThisMailContainsUnwantedMimeParts: N Hi, I have used scripts like the below for almost a decade and a half (obviously doing more than that in the middle). I haven't used them much lately but given other questions I just wanted to fire up a test. I have an end-November kernel doing the below my eapirs do not come back to be destroyed (immediately). I have to start polling for the jid to be no longer alive and not in dying state (hence added the jls/ifconfig -l lines and removed the error checking from ifconfig destroy). That seems sometimes rather unreasonably long (to the point I give up). If I don't configure the addresses below this isn't a problem. Sorry I am confused by too many incarnations of the code; I know I once had a version with an async shutdown path but I believe that never made it into mainline, so why are we holding onto the epairs now and not nuking the addresses and returning them and are clean? It's a bit more funny; I added a twiddle loop at the end and nothing happened. So I stop the script and start it again and suddenly another jail or two have cleaned up and their epairs are back. Something feels very very wonky. Play around with this and see ... and let me know if you can reproduce this... I quite wonder why some test cases haven't gone crazy ... /bz ------------------------------------------------------------------------ #!/bin/sh set -e set -x js=`jail -i -c -n jl host.hostname=left.example.net vnet persist` jb=`jail -i -c -n jr host.hostname=right.example.net vnet persist` # Create an epair connecting the two machines (vnet jails). ep=`ifconfig epair create | sed -e 's/a$//'` # Add one end to each vnet jail. ifconfig ${ep}a vnet ${js} ifconfig ${ep}b vnet ${jb} # Add an IP address on the epairs in each vnet jail. # XXX Leave these out and the cleanup seems to work fine. jexec ${js} ifconfig ${ep}a inet 192.0.2.1/24 jexec ${jb} ifconfig ${ep}b inet 192.0.2.2/24 # Clean up. jail -r ${jb} jail -r ${js} # You want to be able to remove this line ... set +e # No epairs to destroy with addresses configured; fine otherwise. ifconfig ${ep}a destroy # echo $? # Add this is here only as things are funny ... # jls -av jid dying # ifconfig -l # end ------------------------------------------------------------------------ -- Bjoern A. Zeeb r15:7 From nobody Tue Dec 13 23:54:17 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NWwML07tZz4k956 for ; Tue, 13 Dec 2022 23:54:22 +0000 (UTC) (envelope-from bz@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NWwMK6lFRz40qm; Tue, 13 Dec 2022 23:54:21 +0000 (UTC) (envelope-from bz@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1670975661; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=qJbTLqqkjA8jg9py/M2ZKo8RlIVfJG9uDiOf60vh8Qc=; b=Tv1TFKeVq4HKq6Lu8U8rzj000O64ziZ7aZW1EZCDsfzWNDTo9XsBiYkkJRF6x+x0swHROw x8QRHwVL1Qjk4KTCdaS11Z0VgPeu5w6t5xc/pqekIXS9gzBz2h7K6ISlLwMr61GXZPT02q chkTPCkqEwl5ZbPxyLXLh/cyE5h45FCRm07spedrD+4SQAbWLqWy1SXgvNiliq11ZFVlBy QdkbE8TqsbCuA2NUiGjbIlm4rwx6dZ2RoD0QNCMTJ7srM6O36KS67cbHJN2X7e2958KgYf wEGI1ZJCtk5VeTNNJZw+4pLVTGW8JL3VCWZ14hmnukvUO4bprkduufmbrnhK3w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1670975661; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=qJbTLqqkjA8jg9py/M2ZKo8RlIVfJG9uDiOf60vh8Qc=; b=dNIDfw3RnHrW0CCvFN+QPxvTV4bz6WTXNP6/1eFP+bW7utXonFdo8ENBea0t8b3nASl3OJ ZLoduMkBWq/+ejM48yLWYRBs9DqpSaqK13FlEvYyeZkpNyyC33DRcW+30gF9aPd+aUKiFh SomcMCIO9dnSqFMg/AiexvWNAnap8jPbUjX4pEK7lSSctp7viIqu15wKQqtRs1RBRQTXml gdvI4claUKHYMWzPPJcSZaBCazTIFW4uxhPfan4P6WTOnBOtv61wKEEHrUs6HZ1qJWTcxn J7PF8MSXMLSdFDhoyvV0CVA9R4LGm5OgjVAc2BjjLlTFvTIS+bX75WoSYoN5pA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1670975661; a=rsa-sha256; cv=none; b=egzQD3/Gb5Wb9OjCq6H5zDVgmNv0OpkInl1s+aPghd7CxS5+tsl9u+Cn6u8t/M3krl9y9z teG0D4Z6tUDf5CLkCH5MjAE9WAuQO05XE2BIMJtoZWlHDH2BWk90xnnoZEXY4E2DBKTXfD DKEwG9e1dUSK667Gxr95JKCGaeFqP/OoUIcF0TyLJPQ38lDuJGsjocGUFHpAIsmYFuiNCN p20W39ixx3MKTtiRUbocJ38/E3j1xfjlXvBIqkuplfx1gfep6y1ioE7/aNur0vAdkcpGCE ShdxE91Wdd4R9wYvj3NolZs8qjeovcVbcchp4eyxH/X28sXSM3jchlSdjmAGiA== Received: from mx1.sbone.de (cross.sbone.de [195.201.62.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) (Authenticated sender: bz/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4NWwMK55HDzSbH; Tue, 13 Dec 2022 23:54:21 +0000 (UTC) (envelope-from bz@freebsd.org) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id 76DD08D4A162; Tue, 13 Dec 2022 23:54:20 +0000 (UTC) Received: from content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id 2301C5C3A876; Tue, 13 Dec 2022 23:54:20 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) by content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (amavisd-new, port 10024) with ESMTP id UbHAl3xdLpNU; Tue, 13 Dec 2022 23:54:18 +0000 (UTC) Received: from strong-iwl0.sbone.de (strong-iwl0.sbone.de [IPv6:fde9:577b:c1a9:4902:b66b:fcff:fef3:e3d2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id F1DB65C3A830; Tue, 13 Dec 2022 23:54:17 +0000 (UTC) Date: Tue, 13 Dec 2022 23:54:17 +0000 (UTC) From: "Bjoern A. Zeeb" To: Andrew Gallatin cc: "pjd@FreeBSD.org" , James Gritton , jail@freebsd.org, "glebius@FreeBSD.org" Subject: Re: prison_flag() check in hot path of in_pcblookup() In-Reply-To: Message-ID: <6r10qop4-7p83-qs6s-q3r0-64756n243rp5@serrofq.bet> References: <6on81os3-501-s5n2-8nos-p85n8op23232@serrofq.bet> X-OpenPGP-Key-Id: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset=US-ASCII X-ThisMailContainsUnwantedMimeParts: N On Tue, 13 Dec 2022, Andrew Gallatin wrote: > [ I added pjd, since the original patch came from him ] > > Just to make sure I understand, I have a simple yes/no question: > > Can jails and the host ever share the same (local) port and the same IP? Can they currently (I tested only for TCP)? - local binds can overlap like they can with just the base system. so bind(... {AF_INET, laddr, lport} ... ) works fine (REUSEPORT). - tcp connect of a 2nd socket to the same {faddr, fport} from the above bind will fail with 'Address already in use' [currently] [I believe that would mean your patch could go in? Where does the error come from [%]?] [*] - tcp listen will work on {laddr, lport} if run in paralllel (REUSEPORT) or in base and jail at the same time. [%] likely in_pcbconnect_setup() ? Also one should check the other order (jail first then base); also we assume no other race conditions in this rather simple testing... [*] Now someone should run this on a FreeBSD 7.3 / 8.x or later and see how it behaves as the stack might have behaved differently. Also if you have two physical machines or two VMs connected remove the VNET layer and just (manually) test the two parts firing up one extra jail on each base system. I just used vnets for simplicity of my testing. (sorry vnet cleanup currently screwed as it seems 15 years of working also have changed due to other changes; you cal always run jail -r jl jr manually). I haven't done user space socket programming in a while so this was fun (and hopefully does what it should for this test case). I put the simple sources (shell script and C file) up at: https://people.freebsd.org/~bz/tmp/jail-in_pcblookup/ HTH /bz + pwd + STESTBIN=/home/test/socket + PORT=7 + jail -i -c -n jl 'host.hostname=server.example.net' vnet persist 'children.max=1' + js=211 + jail -i -c -n jr 'host.hostname=base.example.net' vnet persist 'children.max=1' + jb=212 + jexec 211 ifconfig lo0 inet 127.0.0.1/8 alias up + jexec 211 ifconfig lo0 inet6 ::1/128 alias + jexec 212 ifconfig lo0 inet 127.0.0.1/8 alias up + jexec 212 ifconfig lo0 inet6 ::1/128 alias + ifconfig epair create + sed -e 's/a$//' + ep=epair102 + ifconfig epair102a vnet 211 + ifconfig epair102b vnet 212 + jexec 211 ifconfig epair102a inet 192.0.2.1/24 + jexec 212 ifconfig epair102b inet 192.0.2.2/24 + jexec 211 jail -i -c -n jsj 'host.hostname=jails.example.net' 'ip4.addr=192.0.2.1' persist + jexec 211 /home/test/socket 192.0.2.1 7 /home/test/socket pid 23254 listening on [192.0.2.1 7] + jsj=213 + echo 'Listing listening connections from the server (base) system' + jexec 213 /home/test/socket 192.0.2.1 7 Listing listening connections from the server (base) system + jexec 211 netstat -an /home/test/socket pid 23257 listening on [192.0.2.1 7] Active Internet connections (including servers) Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp4 0 0 192.0.2.1.7 *.* LISTEN tcp4 0 0 192.0.2.1.7 *.* LISTEN + jexec 212 jail -i -c -n jbj 'host.hostname=jailb.example.net' 'ip4.addr=192.0.2.2' persist + jbj=214 + sleep 1 + echo 'Starting connection from base jail' Starting connection from base jail + sleep 1 + jexec 212 /home/test/socket 192.0.2.2 12345 192.0.2.1 7 /home/test/socket pid 23257 accepted [192.0.2.2 12345] /home/test/socket pid 23261 [192.0.2.2 12345] sleeping 60. + echo 'Starting connection from plain-old IP jail' Starting connection from plain-old IP jail + sleep 1 + jexec 214 /home/test/socket 192.0.2.2 12345 192.0.2.1 7 socket: connect : Address already in use + echo 'Listing server connections from the server (base) system' Listing server connections from the server (base) system + jexec 211 netstat -an Active Internet connections (including servers) Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp4 0 0 192.0.2.1.7 192.0.2.2.12345 ESTABLISHED tcp4 0 0 192.0.2.1.7 *.* LISTEN tcp4 0 0 192.0.2.1.7 *.* LISTEN + echo 'Listing client connections from the base system' Listing client connections from the base system + jexec 212 netstat -an Active Internet connections (including servers) Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp4 0 0 192.0.2.2.12345 192.0.2.1.7 ESTABLISHED + sleep 60 ^C -- Bjoern A. Zeeb r15:7 From nobody Wed Dec 14 01:56:45 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NWz4j12T8z4kQjf for ; Wed, 14 Dec 2022 01:56:53 +0000 (UTC) (envelope-from zlei.huang@gmail.com) Received: from mail-pf1-x429.google.com (mail-pf1-x429.google.com [IPv6:2607:f8b0:4864:20::429]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NWz4h69q4z4CjN; Wed, 14 Dec 2022 01:56:52 +0000 (UTC) (envelope-from zlei.huang@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pf1-x429.google.com with SMTP id 124so3518336pfy.0; Tue, 13 Dec 2022 17:56:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=EO5G41A1XTPGxdZR6Xvcs5IcgfDKITXVnhg/b6kWVKE=; b=duxw+4qaCVzooIx4vLqw/Gi7hEMqwa3UazQYAVOtDJQJygZg6B5HuYM7SJ/SGYfXL5 yXCPf9sQAKfo9lthp++10s+N9Ho9hQ0uHkD3lA0FFu3x4xCB67kL1MaVDjFyT2XvC+M/ 9v0GaSPP9Khe6wo8QMFuX3dkU3FKUAYwTBiRb7OiiK1WCF6KGaaMOdrVH0umOc9wYnzd ypV8oTsWTcNMioJ8FG+2fUD39dC50HvekacZ78uz6AbWCZGkXgbEMpyubszHRQoxj7hK 8J3VVfLdy+Ufph03oXgpisazhQo2To+g3t0S2OgTgrAvy1d8SPRgDAGjc51sQRlYQ7Ze w/1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=EO5G41A1XTPGxdZR6Xvcs5IcgfDKITXVnhg/b6kWVKE=; b=axIz9PypTRANpK0qNGeKHkw9wNB+zIIM7BKUVIBH39A+BMPQbSLzUvyhK3cFxl1NEP VHsMkHrIDPMCazqNn4ECwujewMlxsOmNYo+/15C5KzdEBIWiLGFSDRO2npr8Oi7LDqZ8 FCRyFzes/rD/R4ZJ+P+YS4AHUN7yptDkf7D8wmZDuVqS0gsSfBEMnJMhiercQH8dG8it lKCMuyUQ9pt4LfsJavpxyfQzKIOLc3nf6MnMN0C0Iob7Dlq09NhpPgQJEih4mxVke4p5 RQ1wIcHVDje6BO00l9MRyMbkbf8BslbnFFgL93ZyaC2UKZeUBc+2naM0EErfi1HsvOG7 1pow== X-Gm-Message-State: ANoB5pmnoI1x6khw/4dcZiinWTR0dL3x4dbtoq9QkZIxQByAqlyvTjN7 BhFv/oWWBYY0ABabu4OsEGv3sZH8pvw= X-Google-Smtp-Source: AA0mqf4PM8WspQM/BfiUqLJVNXvZIhcMXc4jI8PcaQ4ym1SW8+iLIXoYCvC8UxZb9Rkk5WlW/ZxWZQ== X-Received: by 2002:a62:ee14:0:b0:566:900d:6073 with SMTP id e20-20020a62ee14000000b00566900d6073mr21838670pfi.24.1670983011131; Tue, 13 Dec 2022 17:56:51 -0800 (PST) Received: from ?IPv6:fd03:a1b3:1819::8001? ([2001:19f0:6001:9db:98f0:9fe0:3545:10]) by smtp.gmail.com with ESMTPSA id p16-20020aa79e90000000b0057691fb0d37sm8188991pfq.193.2022.12.13.17.56.49 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 13 Dec 2022 17:56:50 -0800 (PST) From: Zhenlei Huang X-Google-Original-From: Zhenlei Huang Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_974C8467-6D6D-4A08-9719-957ECFA39962" List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.7\)) Subject: Re: What's going on with vnets and epairs w/ addresses? Date: Wed, 14 Dec 2022 09:56:45 +0800 In-Reply-To: <5r22os7n-ro15-27q-r356-rps331o06so5@mnoonqbm.arg> Cc: "freebsd-jail@freebsd.org" To: "Bjoern A. Zeeb" References: <5r22os7n-ro15-27q-r356-rps331o06so5@mnoonqbm.arg> X-Mailer: Apple Mail (2.3608.120.23.2.7) X-Rspamd-Queue-Id: 4NWz4h69q4z4CjN X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_974C8467-6D6D-4A08-9719-957ECFA39962 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi, I also encounter this problem while testing gif tunnel between jails. My script is similar but with additional gif tunnels. There are reports in mailing list [1], [2], and another one in forum [3] = . Seem to be a long standing issue. [1] = https://lists.freebsd.org/pipermail/freebsd-stable/2016-October/086126.htm= l = [2] = https://lists.freebsd.org/pipermail/freebsd-jail/2017-March/003357.html = [3] = https://forums.freebsd.org/threads/jails-stopping-prolonged-deaths-startin= g-networking-et-cetera.84200/ = Best regards, Zhenlei > On Dec 14, 2022, at 7:03 AM, Bjoern A. Zeeb wrote: >=20 > Hi, >=20 > I have used scripts like the below for almost a decade and a half > (obviously doing more than that in the middle). I haven't used them > much lately but given other questions I just wanted to fire up a test. >=20 > I have an end-November kernel doing the below my eapirs do not come = back > to be destroyed (immediately). > I have to start polling for the jid to be no longer alive and not in > dying state (hence added the jls/ifconfig -l lines and removed the > error checking from ifconfig destroy). That seems sometimes rather > unreasonably long (to the point I give up). >=20 > If I don't configure the addresses below this isn't a problem. >=20 > Sorry I am confused by too many incarnations of the code; I know I = once > had a version with an async shutdown path but I believe that never = made > it into mainline, so why are we holding onto the epairs now and not > nuking the addresses and returning them and are clean? >=20 > It's a bit more funny; I added a twiddle loop at the end and nothing > happened. So I stop the script and start it again and suddenly = another > jail or two have cleaned up and their epairs are back. Something = feels > very very wonky. Play around with this and see ... and let me know if > you can reproduce this... I quite wonder why some test cases haven't > gone crazy ... >=20 > /bz >=20 > = ------------------------------------------------------------------------ > #!/bin/sh >=20 > set -e > set -x >=20 > js=3D`jail -i -c -n jl host.hostname=3Dleft.example.net vnet persist` > jb=3D`jail -i -c -n jr host.hostname=3Dright.example.net vnet persist` >=20 > # Create an epair connecting the two machines (vnet jails). > ep=3D`ifconfig epair create | sed -e 's/a$//'` >=20 > # Add one end to each vnet jail. > ifconfig ${ep}a vnet ${js} > ifconfig ${ep}b vnet ${jb} >=20 > # Add an IP address on the epairs in each vnet jail. > # XXX Leave these out and the cleanup seems to work fine. > jexec ${js} ifconfig ${ep}a inet 192.0.2.1/24 > jexec ${jb} ifconfig ${ep}b inet 192.0.2.2/24 >=20 > # Clean up. > jail -r ${jb} > jail -r ${js} >=20 > # You want to be able to remove this line ... > set +e >=20 > # No epairs to destroy with addresses configured; fine otherwise. > ifconfig ${ep}a destroy > # echo $? >=20 > # Add this is here only as things are funny ... > # jls -av jid dying > # ifconfig -l >=20 > # end > = ------------------------------------------------------------------------ >=20 > --=20 > Bjoern A. Zeeb = r15:7 >=20 --Apple-Mail=_974C8467-6D6D-4A08-9719-957ECFA39962 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
Hi,

I also encounter this = problem while testing gif tunnel between jails.

My script is similar but with additional gif = tunnels.


There are reports in mailing list [1], [2], and = another one in forum [3] .

Seem to = be a long standing issue.



Best regards,
Zhenlei

On Dec 14, 2022, at 7:03 AM, Bjoern A. Zeeb <bz@FreeBSD.org> = wrote:

Hi,

I have used scripts like the = below for almost a decade and a half
(obviously doing more = than that in the middle).  I haven't used them
much = lately but given other questions I just wanted to fire up a test.

I have an end-November kernel doing the below = my eapirs do not come back
to be destroyed = (immediately).
I have to start polling for the jid to be = no longer alive and not in
dying state (hence added the = jls/ifconfig -l lines and removed the
error checking from = ifconfig destroy).  That seems sometimes rather
unreasonably long (to the point I give up).

If I don't configure the addresses below this isn't a = problem.

Sorry I am confused by too many = incarnations of the code; I know I once
had a version with = an async shutdown path but I believe that never made
it = into mainline, so why are we holding onto the epairs now and not
nuking the addresses and returning them and are clean?

It's a bit more funny; I added a twiddle loop = at the end and nothing
happened.  So I stop the = script and start it again and suddenly another
jail or two = have cleaned up and their epairs are back.  Something feels
very very wonky.  Play around with this and see ... and = let me know if
you can reproduce this...  I quite = wonder why some test cases haven't
gone crazy ...

/bz

---------------------------------------------------------------= ---------
#!/bin/sh

set -e
set -x

js=3D`jail -i -c -n jl = host.hostname=3Dleft.example.net vnet persist`
jb=3D`jail = -i -c -n jr host.hostname=3Dright.example.net vnet persist`

# Create an epair connecting the two machines (vnet = jails).
ep=3D`ifconfig epair create | sed -e 's/a$//'`

# Add one end to each vnet jail.
ifconfig ${ep}a vnet ${js}
ifconfig ${ep}b vnet = ${jb}

# Add an IP address on the epairs in = each vnet jail.
# XXX Leave these out and the cleanup = seems to work fine.
jexec ${js}  ifconfig ${ep}a inet =  192.0.2.1/24
jexec ${jb}  ifconfig ${ep}b inet =  192.0.2.2/24

# Clean up.
jail -r ${jb}
jail -r ${js}

# You want to be able to remove this line ...
set= +e

# No epairs to destroy with addresses = configured; fine otherwise.
ifconfig ${ep}a destroy
# echo $?

# Add this is here = only as things are funny ...
# jls -av jid dying
# ifconfig -l

# end
---------------------------------------------------------------= ---------

--
Bjoern A. Zeeb =             &n= bsp;           &nbs= p;            =             &n= bsp;  r15:7


= --Apple-Mail=_974C8467-6D6D-4A08-9719-957ECFA39962-- From nobody Wed Dec 14 07:28:06 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NX6R9702pz4jdPL for ; Wed, 14 Dec 2022 07:28:21 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from mailgate.Leidinger.net (bastille.leidinger.net [89.238.82.207]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "mailgate.leidinger.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NX6R93YVsz3nwy for ; Wed, 14 Dec 2022 07:28:21 +0000 (UTC) (envelope-from Alexander@leidinger.net) Authentication-Results: mx1.freebsd.org; none Received: from outgoing.leidinger.net (p5b165da3.dip0.t-ipconnect.de [91.22.93.163]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-256) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "outgoing.leidinger.net", Issuer "R3" (verified OK)) by mailgate.Leidinger.net (Postfix) with ESMTPSA id 3E2FC2212E for ; Wed, 14 Dec 2022 08:28:09 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1671002889; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to; bh=IDGZEKeIx/qcOjmadZM8QFRDK+ctArpQCEz+A833ddg=; b=KM7BBiz6O+g54uvwp0tB1nBFhpB7ghRgNB5BkrXWKBYqVvUQTLUd4NmnjlPZNfeDW5KZB2 Hu4JLNbxM9WvTPca46Qz/k5guRx5JOfdAgDNX1GIgl8jCVejFTnkWGB9v3DI9ONwOdK7OB +Ca+77j+hCMpFiEaPO+tIt/anI9oJ71NVQZShcWJv0sJs24/MldaB1j2+dq7mkFC/RZBsQ FFdr4XTpTIzIF2AEYMVN5xTvLsNF0f8CYotySN1jCmh0V2wjNvHg8DOQ1RlzJRrcXOUAdd vi7cgqUk21MbZPV6vOFc+1hOjyuyhFJ0u/AIj0kxpXQDyfBLUZzkLfy+ycY8VA== Received: from webmail.leidinger.net (localhost [127.0.0.1]) by outgoing.leidinger.net (Postfix) with ESMTP id C090B70F0 for ; Wed, 14 Dec 2022 08:28:06 +0100 (CET) Received: from www (uid 80) (envelope-from Alexander@leidinger.net) id 7b3cb by webmail.leidinger.net (DragonFly Mail Agent v0.13+ on webmail.leidinger.net); Wed, 14 Dec 2022 08:28:06 +0100 Date: Wed, 14 Dec 2022 08:28:06 +0100 Message-ID: <20221214082806.Horde.fnrSehaAFQsAtgLgj_MkKpA@webmail.leidinger.net> From: Alexander Leidinger To: "Bjoern A. Zeeb" , Kristof Provost Cc: freebsd-jail@freebsd.org Subject: Re: What's going on with vnets and epairs w/ addresses? In-Reply-To: <5r22os7n-ro15-27q-r356-rps331o06so5@mnoonqbm.arg> Accept-Language: de,en Content-Type: multipart/signed; boundary="=_u8b1FlGjBkWKpOnADRhAnyV"; protocol="application/pgp-signature"; micalg=pgp-sha256 List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 X-Rspamd-Queue-Id: 4NX6R93YVsz3nwy X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:34240, ipnet:89.238.64.0/18, country:DE] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N This message is in MIME format and has been PGP signed. --=_u8b1FlGjBkWKpOnADRhAnyV Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Quoting "Bjoern A. Zeeb" (from Tue, 13 Dec 2022=20=20 23:03:42=20+0000 (UTC)): > Hi, > > I have used scripts like the below for almost a decade and a half > (obviously doing more than that in the middle). I haven't used them > much lately but given other questions I just wanted to fire up a test. > > I have an end-November kernel doing the below my eapirs do not come back > to be destroyed (immediately). > I have to start polling for the jid to be no longer alive and not in > dying state (hence added the jls/ifconfig -l lines and removed the > error checking from ifconfig destroy). That seems sometimes rather > unreasonably long (to the point I give up). > > If I don't configure the addresses below this isn't a problem. > > Sorry I am confused by too many incarnations of the code; I know I once > had a version with an async shutdown path but I believe that never made > it into mainline, so why are we holding onto the epairs now and not > nuking the addresses and returning them and are clean? Kristof, isn't this (epair destruction in jails) one of the issues you=20= =20 looked=20at? Sorry if I remember incorrectly. What I have in my jails-shutdown is to do an "ifconfig $epair_in_jail=20=20 -vnet=20$jail; sleep 2; ifconfig $epair destroy". With this I don't see=20= =20 any=20issues, Everything is cleaned up when the stop finishes. Bye, Alexander. > It's a bit more funny; I added a twiddle loop at the end and nothing > happened. So I stop the script and start it again and suddenly another > jail or two have cleaned up and their epairs are back. Something feels > very very wonky. Play around with this and see ... and let me know if > you can reproduce this... I quite wonder why some test cases haven't > gone crazy ... > > /bz > > ------------------------------------------------------------------------ > #!/bin/sh > > set -e > set -x > > js=3D`jail -i -c -n jl host.hostname=3Dleft.example.net vnet persist` > jb=3D`jail -i -c -n jr host.hostname=3Dright.example.net vnet persist` > > # Create an epair connecting the two machines (vnet jails). > ep=3D`ifconfig epair create | sed -e 's/a$//'` > > # Add one end to each vnet jail. > ifconfig ${ep}a vnet ${js} > ifconfig ${ep}b vnet ${jb} > > # Add an IP address on the epairs in each vnet jail. > # XXX Leave these out and the cleanup seems to work fine. > jexec ${js} ifconfig ${ep}a inet 192.0.2.1/24 > jexec ${jb} ifconfig ${ep}b inet 192.0.2.2/24 > > # Clean up. > jail -r ${jb} > jail -r ${js} > > # You want to be able to remove this line ... > set +e > > # No epairs to destroy with addresses configured; fine otherwise. > ifconfig ${ep}a destroy > # echo $? > > # Add this is here only as things are funny ... > # jls -av jid dying > # ifconfig -l > > # end > ------------------------------------------------------------------------ > > --=20 >=20Bjoern A. Zeeb r15:= 7 --=20 http://www.Leidinger.net=20Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF --=_u8b1FlGjBkWKpOnADRhAnyV Content-Type: application/pgp-signature Content-Description: Digitale PGP-Signatur Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIzBAABCAAdFiEER9UlYXp1PSd08nWXEg2wmwP42IYFAmOZewUACgkQEg2wmwP4 2Iaeww/5AQdUYL0r/txzh4ftRY9NNDRMeD9U3Dzkmn59JOg+1yRGAFtVRd5E2dq2 NMYHgweNDDkuDxfOt0fHpdlGRkxA2x6OeguKB0F3kJjuG0Le9wcuTdCm9meMoyt2 QBjATXHQjLHfSM2ZHWr4lZXdAb5cMRUExQjRwneXnxLoRo86yhMHabiul2ufc8KC AnlCbZjgrRE7hc56s6Is0FeUkVL/yW2T26jus9YGU/JjUoJ3IaPlnWnEWwqP5seg 1Kdv1JD7Q0Zf5ABMZ/pdDiWIQhU4PIFwECYDyKNDMoRAAr0S2sNdNwIRH/5h0tdV ihomEvCZB+N/GSPuQGtHH5n4eaC8M2FO2khwJxCoRBiOclzESOOaQ5ZUtusR42j7 xw9ceILQs2cF4dh2lnDnEI0wFZ6YqsnFTAG24yd2JUZEzWxAyWR+v4qB+7EhtB3f 7t/A5tbxZ3yshfofGVATwCJsWydX0B7KGnQjmE1EcbyEX2bsRl5sx00p0X1Kqw67 JqbKEcNxzQX3R6deV0XvT2anG+II1+OTRZq15fKOf2ftxneeuBSWk39VeFHij6M/ HKBLmKh5P0s/4CHZZtcynnM9XnUK/ktvMSqgmzJ+F0yRnBOpXFIeuCZj32FYIb3r eTldv+DvZaFmZ7/6cTadhkWBJAoFcfPiRMrOVUREacyRFWuVJUQ= =D9Bc -----END PGP SIGNATURE----- --=_u8b1FlGjBkWKpOnADRhAnyV-- From nobody Wed Dec 14 18:42:55 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NXPQ136ycz4kHVG for ; Wed, 14 Dec 2022 18:43:21 +0000 (UTC) (envelope-from kp@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NXPQ12Nrlz3s3t; Wed, 14 Dec 2022 18:43:21 +0000 (UTC) (envelope-from kp@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671043401; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ILxOixkMDpnGywJffKKQX5qABVU+mhBqD0IfPaJ2T9Q=; b=LCA9OsNavtOt7j1ld5ZCtCvZzEoW7nWkbj+ZpdxZFfU1CqhkR6J9+4IHdGksYH6RQkAFvw t3hc3qAQQUcR97O/KPa9EGCldncOe1NQMQ/psSWR6elxFomcptvUOSAh9c1B6Sz1aaZe2D /MjGVSzi/779NgtIqDlZTcLD2rpaWEhfViD5dzuOOTJbdcI01PXCol+yI6mYmPh7N6HjV/ DFvt+IiPz/q+SSgLqHF3EapTTZtKb8fCurkeoEZ4hnjYYUX4pl/kVBnQ6xTTiuBhOV94l+ 8kIe+HqwXcyHtiPgRqTknoTTEgnysVV0BIR7hObeNJdpY+eX1X1RHlLle/58wQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671043401; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ILxOixkMDpnGywJffKKQX5qABVU+mhBqD0IfPaJ2T9Q=; b=s/fFxcIYaz8PGGlU6/C/Jd8uUztRsGcyaWjZpRwwilB7zZD/DrTG1ybo0mWGqpurWIK015 vy7/8x4gHGgA66P2IxL99H+PhlGYvC0nVgIN6MDRa+rTiuqZ4HfHf2BlUmU/jqREuWtAFi UbvshoxjCh8WSYiK9l+NNMSAfDK4+NqSguzS99CFuABh1d1yyuO0xAkw/c638mwEiwKPdZ 8hKAfyW+6xsND0IUntu7byodn72J72YMguo2VrK9SQwJPou358IRI4C64n19dNanKx/vMs AKHh+NfHqxsFHC5JeXk6biSPdihb7ftKWuFhdHScjtFaX3Kfs8yWWdzWfmXH4g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1671043401; a=rsa-sha256; cv=none; b=vkKdecRiAheaw8uV2UZB46YeFdf66oReLmQTVKzZQhacw+1vbuONVuOlQTtF/Zh7g0WswL 5el7DCFlXf6PjfrjGghGqC5T1vNXlaskhAvSTiRmS5sW7qKnc5Z+iz/GCyR0+dLEq0i0HJ mEJpYjyhifkIoOLatNkpK341FqsrwUKBpDoACqbu/dbTulWN5dymyYNDt+tPCAU+DObVIU Vrkx1mZXgAu3m55E0aRTmnFqc0/IFWWbmf7+8EqmBH41r5f7XIgCxHRzCKzB2wcT/MxGgP WmQn2z3u5tQ5Z75oddWuveJHqNHYVQgCgiKQvRYvcraIM4Hgfy9XB71opq8cDQ== Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mx1.codepro.be", Issuer "R3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 4NXPQ10qxHznX4; Wed, 14 Dec 2022 18:43:21 +0000 (UTC) (envelope-from kp@freebsd.org) Received: by venus.codepro.be (Postfix, authenticated sender kp) id AA46E46C35; Wed, 14 Dec 2022 19:43:17 +0100 (CET) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable From: Kristof Provost List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org Mime-Version: 1.0 (1.0) Subject: Re: What's going on with vnets and epairs w/ addresses? Date: Thu, 15 Dec 2022 07:42:55 +1300 Message-Id: <307E29A2-5B77-4394-B1DF-A9BC5F759475@freebsd.org> References: <20221214082806.Horde.fnrSehaAFQsAtgLgj_MkKpA@webmail.leidinger.net> Cc: "Bjoern A. Zeeb" , freebsd-jail@freebsd.org In-Reply-To: <20221214082806.Horde.fnrSehaAFQsAtgLgj_MkKpA@webmail.leidinger.net> To: Alexander Leidinger X-Mailer: iPhone Mail (20B110) X-ThisMailContainsUnwantedMimeParts: N > On 14 Dec 2022, at 20:28, Alexander Leidinger wr= ote: >=20 > =EF=BB=BF > Quoting "Bjoern A. Zeeb" (from Tue, 13 Dec 2022 23:03:42 += 0000 (UTC)): >=20 >> Hi, >>=20 >> I have used scripts like the below for almost a decade and a half >> (obviously doing more than that in the middle). I haven't used them >> much lately but given other questions I just wanted to fire up a test. >>=20 >> I have an end-November kernel doing the below my eapirs do not come back >> to be destroyed (immediately). >> I have to start polling for the jid to be no longer alive and not in >> dying state (hence added the jls/ifconfig -l lines and removed the >> error checking from ifconfig destroy). That seems sometimes rather >> unreasonably long (to the point I give up). >>=20 >> If I don't configure the addresses below this isn't a problem. >>=20 >> Sorry I am confused by too many incarnations of the code; I know I once >> had a version with an async shutdown path but I believe that never made >> it into mainline, so why are we holding onto the epairs now and not >> nuking the addresses and returning them and are clean? >=20 > Kristof, isn't this (epair destruction in jails) one of the issues you loo= ked at? Sorry if I remember incorrectly. >=20 I looked at panics around destroying interfaces and vnets.=20 My speculative guess here is that the jail is hanging around for some reason= , and that=E2=80=99s causing the epair and address to stick around too.=20 jls -na might confirm or deny that.=20 Br, Kristof= From nobody Fri Dec 16 10:30:57 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NYQPP0jNyzQfmw for ; Fri, 16 Dec 2022 10:31:21 +0000 (UTC) (envelope-from zlei.huang@gmail.com) Received: from mail-pf1-x431.google.com (mail-pf1-x431.google.com [IPv6:2607:f8b0:4864:20::431]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NYQPN30TKz46Ck; Fri, 16 Dec 2022 10:31:20 +0000 (UTC) (envelope-from zlei.huang@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=W94UzvRq; spf=pass (mx1.freebsd.org: domain of zlei.huang@gmail.com designates 2607:f8b0:4864:20::431 as permitted sender) smtp.mailfrom=zlei.huang@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pf1-x431.google.com with SMTP id w26so1483688pfj.6; Fri, 16 Dec 2022 02:31:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=mo+bdKqgDddGxwtVWA22dL0PkNS7K62O3bVawMx8Z+w=; b=W94UzvRqP3r5zTFGFMqNC79F5zYJWqYJXixyBZsGMyhC7dw+8MCWtdYt3LwreDMWZM c8qw7YhQoGLimVO8NT0msH4zoS7TnB3W2LtplK3m7yXeyeGdB4OklbheDOC4X3Svk39T fMl4bon9JsIUvbYdUUIPbeoLlCI7WJuAtM/HFa+DouyF+KGRNNzkkhrGXBcwqSoN1F4e PjqE76a9qcPDCdAMS3zWcZOJWt6ttvzSK8ablTJB41ZmtDegm+fcfvrYWHICX2bqhtQ2 C/0Q9VUREzlxH9KGTIVUviHO4OmUjdyPnAkfd/XFZfiqi6vF/Lq55GE+yxj5Tbz8HS1Z yArg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mo+bdKqgDddGxwtVWA22dL0PkNS7K62O3bVawMx8Z+w=; b=zDrOKOh/v+Adgw7CxBUo1JT4WvNaeoFpCkdWa+Eld6pSmsL+JzgnXlX7+df6/aQLad O9zFx8qfYTcUtak151VrU+s29QzZgUr3xAW65Peey+qpz1RFJhyoV3px9dVHTk1uZ2e1 jwBJa/vjn5moQTuQNdYUEmYqpdp0zp/Zh40q/ga4JHdtetb925Hc/nJmhd6Fq89OjbJk zNGPS5ZM1lL6tc+KisocVmjFqL3S29WIrqb3y7OwlW968P6fnXeAHaVel7lzIWUwAF+r jbJOWxKB3X0bHwQz/JBV9hkliAQKmDuXzryE9T+0JcjVf8UjaLbIsorttf8x01s77lCK zJPg== X-Gm-Message-State: ANoB5pnAO9rtGKUqX7ogNP1dfa+s5oZ8JGZ4b3+rA8gzk6qemQ3ugN+r dgky570l6Lil4As+kqMcdf819bsw3c527A== X-Google-Smtp-Source: AA0mqf5EBbSB6a1a7k64pG7U+/HzXHA19LcT0WMaFQx2ZWBLAYg5izEWhVt9/9dwAFTbHDMf7KWE1w== X-Received: by 2002:aa7:928f:0:b0:56d:465d:9fbc with SMTP id j15-20020aa7928f000000b0056d465d9fbcmr27538683pfa.25.1671186678858; Fri, 16 Dec 2022 02:31:18 -0800 (PST) Received: from [192.168.10.254] ([112.66.186.114]) by smtp.gmail.com with ESMTPSA id p62-20020a622941000000b0056b9ec7e2desm1144544pfp.125.2022.12.16.02.31.17 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 16 Dec 2022 02:31:18 -0800 (PST) From: Zhenlei Huang X-Google-Original-From: Zhenlei Huang Message-Id: <150A60D6-6757-46DD-988F-05A9FFA36821@FreeBSD.org> Content-Type: multipart/alternative; boundary="Apple-Mail=_2396E16C-866A-4D36-A1B5-6C4992A3A64A" List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.7\)) Subject: Re: What's going on with vnets and epairs w/ addresses? Date: Fri, 16 Dec 2022 18:30:57 +0800 In-Reply-To: Cc: "freebsd-jail@freebsd.org" , Gleb Smirnoff To: "Bjoern A. Zeeb" References: <5r22os7n-ro15-27q-r356-rps331o06so5@mnoonqbm.arg> X-Mailer: Apple Mail (2.3608.120.23.2.7) X-Spamd-Result: default: False [-0.41 / 15.00]; URI_COUNT_ODD(1.00)[29]; SUBJECT_ENDS_QUESTION(1.00)[]; MID_RHS_MATCH_TO(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.91)[-0.910]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TAGGED_FROM(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::431:from]; MIME_TRACE(0.00)[0:+,1:+,2:~]; MLMMJ_DEST(0.00)[freebsd-jail@freebsd.org] X-Rspamd-Queue-Id: 4NYQPN30TKz46Ck X-Spamd-Bar: / X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_2396E16C-866A-4D36-A1B5-6C4992A3A64A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi, I managed to repeat this issue on CURRENT/14 with this small snip: ------------------------------------------- #!/bin/sh # test jail name n=3D"test_ref_leak" jail -c name=3D$n path=3D/ vnet persist # The following line trigger jail pr_ref leak jexec $n ifconfig lo0 inet 127.0.0.1/8 jail -R $n # wait a moment sleep 1 jls -j $n ------------------------------------------- After DDB debugging and tracing , it seems that is triggered by a = combine of [1] and [2] [1] = https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915 = [2] = https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b = In [1] the per-VNET uma zone is shared with the global one. `pcbinfo->ipi_zone =3D pcbstor->ips_zone;` In [2] unref `inp->inp_cred` is deferred called in inpcb_dtor() by = uma_zfree_smr() . Unfortunately inps freed by uma_zfree_smr() are cached and inpcb_dtor() = is not called immediately , thus leaking `inp->inp_cred` ref and hence `prison->pr_ref`. And it is also not possible to free up the cache by per-VNET SYSUNINIT = tcp_destroy / udp_destroy / rip_destroy. Best regards, Zhenlei > On Dec 14, 2022, at 9:56 AM, Zhenlei Huang wrote: >=20 >=20 > Hi, >=20 > I also encounter this problem while testing gif tunnel between jails. >=20 > My script is similar but with additional gif tunnels. >=20 >=20 > There are reports in mailing list [1], [2], and another one in forum = [3] . >=20 > Seem to be a long standing issue. >=20 > [1] = https://lists.freebsd.org/pipermail/freebsd-stable/2016-October/086126.htm= l = > [2] = https://lists.freebsd.org/pipermail/freebsd-jail/2017-March/003357.html = > [3] = https://forums.freebsd.org/threads/jails-stopping-prolonged-deaths-startin= g-networking-et-cetera.84200/ = >=20 >=20 > Best regards, > Zhenlei >=20 >> On Dec 14, 2022, at 7:03 AM, Bjoern A. Zeeb > wrote: >>=20 >> Hi, >>=20 >> I have used scripts like the below for almost a decade and a half >> (obviously doing more than that in the middle). I haven't used them >> much lately but given other questions I just wanted to fire up a = test. >>=20 >> I have an end-November kernel doing the below my eapirs do not come = back >> to be destroyed (immediately). >> I have to start polling for the jid to be no longer alive and not in >> dying state (hence added the jls/ifconfig -l lines and removed the >> error checking from ifconfig destroy). That seems sometimes rather >> unreasonably long (to the point I give up). >>=20 >> If I don't configure the addresses below this isn't a problem. >>=20 >> Sorry I am confused by too many incarnations of the code; I know I = once >> had a version with an async shutdown path but I believe that never = made >> it into mainline, so why are we holding onto the epairs now and not >> nuking the addresses and returning them and are clean? >>=20 >> It's a bit more funny; I added a twiddle loop at the end and nothing >> happened. So I stop the script and start it again and suddenly = another >> jail or two have cleaned up and their epairs are back. Something = feels >> very very wonky. Play around with this and see ... and let me know = if >> you can reproduce this... I quite wonder why some test cases haven't >> gone crazy ... >>=20 >> /bz >>=20 >> = ------------------------------------------------------------------------ >> #!/bin/sh >>=20 >> set -e >> set -x >>=20 >> js=3D`jail -i -c -n jl host.hostname=3Dleft.example.net = vnet persist` >> jb=3D`jail -i -c -n jr host.hostname=3Dright.example.net = vnet persist` >>=20 >> # Create an epair connecting the two machines (vnet jails). >> ep=3D`ifconfig epair create | sed -e 's/a$//'` >>=20 >> # Add one end to each vnet jail. >> ifconfig ${ep}a vnet ${js} >> ifconfig ${ep}b vnet ${jb} >>=20 >> # Add an IP address on the epairs in each vnet jail. >> # XXX Leave these out and the cleanup seems to work fine. >> jexec ${js} ifconfig ${ep}a inet 192.0.2.1/24 >> jexec ${jb} ifconfig ${ep}b inet 192.0.2.2/24 >>=20 >> # Clean up. >> jail -r ${jb} >> jail -r ${js} >>=20 >> # You want to be able to remove this line ... >> set +e >>=20 >> # No epairs to destroy with addresses configured; fine otherwise. >> ifconfig ${ep}a destroy >> # echo $? >>=20 >> # Add this is here only as things are funny ... >> # jls -av jid dying >> # ifconfig -l >>=20 >> # end >> = ------------------------------------------------------------------------ >>=20 >> --=20 >> Bjoern A. Zeeb = r15:7 >>=20 >=20 --Apple-Mail=_2396E16C-866A-4D36-A1B5-6C4992A3A64A Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
Hi,

I= managed to repeat this issue on CURRENT/14 with this small = snip:

-------------------------------------------
#!/bin/sh

# test jail name
n=3D"test_ref_leak"

jail -c name=3D$n path=3D/ vnet = persist
# The following line trigger jail pr_ref = leak
jexec $n ifconfig lo0 inet = 127.0.0.1/8

jail= -R $n

# wait = a moment
sleep 1

jls -j $n


-------------------------------------------


After DDB debugging and tracing , it seems that is triggered = by a combine of [1] and [2]

[1] https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5a= c895915
[2] https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b1= 75b8c5b


In [1] the per-VNET uma zone is shared = with the global one.
`pcbinfo->ipi_zone =3D = pcbstor->ips_zone;`

In [2] unref `inp->inp_cred` is deferred called in = inpcb_dtor() by uma_zfree_smr() .

Unfortunately inps freed by uma_zfree_smr() are cached and inpcb_dtor() is not called immediately ,
thus leaking `inp->inp_cred` ref and hence = `prison->pr_ref`.

And it is also not = possible to free up the cache by per-VNET SYSUNINIT tcp_destroy / udp_destroy = / rip_destroy.



Best regards,
Zhenlei

On Dec 14, 2022, at 9:56 AM, Zhenlei Huang <zlei@FreeBSD.org> = wrote:


Hi,

I also encounter this problem while testing gif tunnel = between jails.

My script is similar but with additional gif = tunnels.


There are reports in mailing list [1], = [2], and another one in forum [3] .

Seem to be a long standing = issue.

[1] https://lists.freebsd.org/pipermail/freebsd-stable/2016-October= /086126.html
[2] https://lists.freebsd.org/pipermail/freebsd-jail/2017-March/003= 357.html
[3] https://forums.freebsd.org/threads/jails-stopping-prolonged-dea= ths-starting-networking-et-cetera.84200/


Best regards,
Zhenlei

On Dec 14, 2022, at 7:03 AM, Bjoern A. Zeeb <bz@FreeBSD.org> = wrote:

Hi,

I have used scripts like the = below for almost a decade and a half
(obviously doing more = than that in the middle).  I haven't used them
much = lately but given other questions I just wanted to fire up a test.

I have an end-November kernel doing the below = my eapirs do not come back
to be destroyed = (immediately).
I have to start polling for the jid to be = no longer alive and not in
dying state (hence added the = jls/ifconfig -l lines and removed the
error checking from = ifconfig destroy).  That seems sometimes rather
unreasonably long (to the point I give up).

If I don't configure the addresses below this isn't a = problem.

Sorry I am confused by too many = incarnations of the code; I know I once
had a version with = an async shutdown path but I believe that never made
it = into mainline, so why are we holding onto the epairs now and not
nuking the addresses and returning them and are clean?

It's a bit more funny; I added a twiddle loop = at the end and nothing
happened.  So I stop the = script and start it again and suddenly another
jail or two = have cleaned up and their epairs are back.  Something feels
very very wonky.  Play around with this and see ... and = let me know if
you can reproduce this...  I quite = wonder why some test cases haven't
gone crazy ...

/bz

---------------------------------------------------------------= ---------
#!/bin/sh

set -e
set -x

js=3D`jail -i -c -n jl = host.hostname=3Dleft.example.net vnet persist`
jb=3D`jail = -i -c -n jr host.hostname=3Dright.example.net vnet persist`

# Create an epair connecting the two machines (vnet = jails).
ep=3D`ifconfig epair create | sed -e 's/a$//'`

# Add one end to each vnet jail.
ifconfig ${ep}a vnet ${js}
ifconfig ${ep}b vnet = ${jb}

# Add an IP address on the epairs in = each vnet jail.
# XXX Leave these out and the cleanup = seems to work fine.
jexec ${js}  ifconfig ${ep}a inet =  192.0.2.1/24
jexec ${jb}  ifconfig ${ep}b inet =  192.0.2.2/24

# Clean up.
jail -r ${jb}
jail -r ${js}

# You want to be able to remove this line ...
set= +e

# No epairs to destroy with addresses = configured; fine otherwise.
ifconfig ${ep}a destroy
# echo $?

# Add this is here = only as things are funny ...
# jls -av jid dying
# ifconfig -l

# end
---------------------------------------------------------------= ---------

--
Bjoern A. Zeeb =             &n= bsp;           &nbs= p;            =             &n= bsp;  r15:7



= --Apple-Mail=_2396E16C-866A-4D36-A1B5-6C4992A3A64A-- From nobody Fri Dec 16 14:41:39 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NYWyM3bmlzt8KN for ; Fri, 16 Dec 2022 14:41:47 +0000 (UTC) (envelope-from zlei.huang@gmail.com) Received: from mail-pl1-x62e.google.com (mail-pl1-x62e.google.com [IPv6:2607:f8b0:4864:20::62e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NYWyL5nT6z4PXD for ; Fri, 16 Dec 2022 14:41:46 +0000 (UTC) (envelope-from zlei.huang@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=OEsN8Xp9; spf=pass (mx1.freebsd.org: domain of zlei.huang@gmail.com designates 2607:f8b0:4864:20::62e as permitted sender) smtp.mailfrom=zlei.huang@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-x62e.google.com with SMTP id x2so2470136plb.13 for ; Fri, 16 Dec 2022 06:41:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:date:message-id:subject:mime-version:content-transfer-encoding :from:from:to:cc:subject:date:message-id:reply-to; bh=97NNWxcpJ5oBsera/s8l1XNM0Cg0GZ6kFFWoULI6cN8=; b=OEsN8Xp961/q1ubdfvLSH8VOg/0ctTjeMTogNdj+/sLFMFY6Yv+LTXpqcadYO65r6Z 6nhqxbmTArDZEH3TP+o6b3GERU7WwpCdbrRq4pcGR9tmvQ4xuOaobzDOmzWBxwXhv+qJ qRrsb2O57sY6s4/ywHvdLyeBsa2t0KyTrc6ZhIfev1NAlUNZReEzG81UR1KUIRee6gBr 560elFDYocl8eY8DCOAoyGAjcACNUUVA1pOlix+QpPVZ8DD13cGJdyjnhG2zTMdRjIMQ yC2Vz9B5e8Pz9KaeRgMdjYOSYvc6te19r3csvoBHD8Xlm6xSgYSW4Z5DwBCQ9Hq6rBaA X3dA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:date:message-id:subject:mime-version:content-transfer-encoding :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=97NNWxcpJ5oBsera/s8l1XNM0Cg0GZ6kFFWoULI6cN8=; b=BNo4NlNsvr8BTMU/WOl6jz0HAKBreuYNaqUM/aHZpteI1qyKYSnY/C9gUfpw4B3hzb pugsZWx0kiJeXB2fyJ/ZBlxhzYAHblLiNStNsyGOiBY6dYEE9b0hAyl87L05vBS5+tMl UXcQ6miwrJSRBlJA++CSl5PSdii3GX6smncuK2GHbpeLrpsvfBbH6LidEhAPPidi9zn1 LroJWITphKyMkMMYvMu8YkZHhbY0oZsLWs2hffuAGpMIhP/ZYm+O4olwf1K7Z7+MNFPW medr6yOn4dJMcRNY5OOymyqCijK1ZSem+mLsXsX4wK0lAOztcnW+1FGBgl0Mq8mnqd8e dukw== X-Gm-Message-State: AFqh2kos5M49q/VXLhXpSypDCAbqSM9sp545EvyLfcw2keRjj9EBC0Ym dWcfS2ZSKDlkYN5OjYs3jqM+HEi1RlQTBQ== X-Google-Smtp-Source: AMrXdXvrme+H2zHtuEENFxZvOtg+0OldZ2RrNpkJ7vVCZJI1/x34Su/ow4Kz+TliMASa8hg8GxIUfg== X-Received: by 2002:a17:90b:3655:b0:223:430d:8dfc with SMTP id nh21-20020a17090b365500b00223430d8dfcmr8522218pjb.36.1671201705481; Fri, 16 Dec 2022 06:41:45 -0800 (PST) Received: from [172.17.252.129] (ns1.oxydns.net. [45.32.91.63]) by smtp.gmail.com with ESMTPSA id m1-20020a17090a7f8100b0021885b05660sm1473980pjl.24.2022.12.16.06.41.44 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 16 Dec 2022 06:41:45 -0800 (PST) From: Zhenlei Huang X-Google-Original-From: Zhenlei Huang Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.7\)) Subject: Is it possible to employ epoch to simplify managing prison lifecycle Message-Id: <9BD54A54-A809-4D3E-BCBA-639E6C61FE37@FreeBSD.org> Date: Fri, 16 Dec 2022 22:41:39 +0800 To: freebsd-jail@freebsd.org X-Mailer: Apple Mail (2.3608.120.23.2.7) X-Spamd-Result: default: False [-2.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; MID_RHS_MATCH_TO(1.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-jail@freebsd.org]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::62e:from]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; TAGGED_FROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-jail@freebsd.org] X-Rspamd-Queue-Id: 4NYWyL5nT6z4PXD X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N Hi, While hacking `sys/kern/kern_jail.c` I got lost. There're lots of ref / unref and flags to prevent visit invalid prison while concurrent modification is possible and some refs looks weird. Is it possible to employ epoch(9) to simplify managing of prison lifecycle ? Best regards, Zhenlei From nobody Fri Dec 16 16:29:33 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NYZLl3CxQz1Cgl4 for ; Fri, 16 Dec 2022 16:29:35 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oa1-x2c.google.com (mail-oa1-x2c.google.com [IPv6:2001:4860:4864:20::2c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NYZLk5zRDz3MB5 for ; Fri, 16 Dec 2022 16:29:34 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=cVOoACL2; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2001:4860:4864:20::2c as permitted sender) smtp.mailfrom=mjguzik@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oa1-x2c.google.com with SMTP id 586e51a60fabf-1445ca00781so3918150fac.1 for ; Fri, 16 Dec 2022 08:29:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=CWcP76QbEKObpJKnRw0F7XH5sU24e2zpbcwGWKwW3Qc=; b=cVOoACL20dTQGBtQ0CMLNcGJG5OzuHH3Akhol1YoTzI8hj11RzlDuNtHTrOhSBIM+U 6ll3g9xN4wmZqCgidZr1AN9wrDEBFOSp8lwZAzQ8fLx4KwsaFx8u2JiqGFYxF+qBRK/j 6iOS/Fi2SW0baSQjaDvz2WvdtHvIeyY37KbLiDLZVO+u8lOdYzVYB7czcpWbJk4pKvhc 43U7WecT8/2WtyRxeb5rLiKfdpRE8DUGaXRM4GC5vJAIZJP0ubAAvCdM+arN3lXTIf3L Qz7n+Zfgw7nnzH7b9Z9XGudIbpNsWBu+Fc9eZ/M8e7i/Joiv0+xQ5Mw3gzs+wSgxqTTT 6u1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=CWcP76QbEKObpJKnRw0F7XH5sU24e2zpbcwGWKwW3Qc=; b=wijdxnFL6urc5KQvUnkeZKlGBQHoWu+z5JAoZruXScbQ6P7gGDOW7iNDgZFJIOCBkw wkRb/cus1IxLRGp9X9JMdv9NQUEcj+4NYMoaGQhPmceNsbNQS2jQXvMkqf7G+qUztAOJ EHyHcSGi6xeiNvc8kY86CxXZBd39Q6VyeXblA+dQcD+leAQAkvS83KudGoyVq2JHSVca kP6cKiUsNPXr9aMonVYK3qzNi4VkAU30k2mgAi3H1ih1IHcNaZhUxjMq/b6msB22+o1q IyZg4/bJuls78g0pw9TmoQTl/levqS+VZTqx58rMq9ULUi8CZuzVX9+VQCSpVbVhzMqD sl6g== X-Gm-Message-State: AFqh2krYLcy4uZEf0igctseatKwLXdpA+FubM8tQ8kN2xeRGxTrXZeYx OUJOqCObkwd22NQr/W49/wiN0FQ/vNhv8QprjqI= X-Google-Smtp-Source: AA0mqf6uJ5yhXccF21s7z96qe0Psbfw45kX5laRaVjyfVNyZRVsnyVsGuc0T34MwZL+pJtLgdWlg2qAgxnZN2FGahgw= X-Received: by 2002:a05:6870:7985:b0:144:68b8:88eb with SMTP id he5-20020a056870798500b0014468b888ebmr919030oab.159.1671208174004; Fri, 16 Dec 2022 08:29:34 -0800 (PST) List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Received: by 2002:ac9:77d5:0:b0:491:8368:9bdd with HTTP; Fri, 16 Dec 2022 08:29:33 -0800 (PST) In-Reply-To: <9BD54A54-A809-4D3E-BCBA-639E6C61FE37@FreeBSD.org> References: <9BD54A54-A809-4D3E-BCBA-639E6C61FE37@FreeBSD.org> From: Mateusz Guzik Date: Fri, 16 Dec 2022 17:29:33 +0100 Message-ID: Subject: Re: Is it possible to employ epoch to simplify managing prison lifecycle To: Zhenlei Huang Cc: freebsd-jail@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-3.98 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.983]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2001:4860:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; TAGGED_RCPT(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-jail@freebsd.org]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2001:4860:4864::/48, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_IN_DNSWL_NONE(0.00)[2001:4860:4864:20::2c:from]; DKIM_TRACE(0.00)[gmail.com:+]; FREEMAIL_TO(0.00)[gmail.com]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-jail@freebsd.org] X-Rspamd-Queue-Id: 4NYZLk5zRDz3MB5 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On 12/16/22, Zhenlei Huang wrote: > Hi, > > While hacking `sys/kern/kern_jail.c` I got lost. > > There're lots of ref / unref and flags to prevent visit invalid prison > while > concurrent modification is possible and some refs looks weird. > > Is it possible to employ epoch(9) to simplify managing of prison lifecycle > ? > Some of the ref/unref cycles are probably avoidable to begin with, but ultimately the thing to do here is to employ per-cpu reference counting, if at all needed. I have a wip patch to provide such a mechanism, it may or may not land this month. -- Mateusz Guzik From nobody Fri Dec 16 18:00:46 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NYcN73ls3z1G5L2 for ; Fri, 16 Dec 2022 18:00:55 +0000 (UTC) (envelope-from jamie@gritton.org) Received: from gritton.org (gritton.org [162.220.209.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "gritton.org", Issuer "gritton.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NYcN64Kb5z3hyB for ; Fri, 16 Dec 2022 18:00:54 +0000 (UTC) (envelope-from jamie@gritton.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of jamie@gritton.org designates 162.220.209.3 as permitted sender) smtp.mailfrom=jamie@gritton.org; dmarc=none Received: from gritton.org ([127.0.0.3]) (authenticated bits=0) by gritton.org (8.16.1/8.16.1) with ESMTPA id 2BGI0kYq098223; Fri, 16 Dec 2022 10:00:46 -0800 (PST) (envelope-from jamie@gritton.org) List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Date: Fri, 16 Dec 2022 10:00:46 -0800 From: James Gritton To: freebsd-jail@freebsd.org Cc: Zhenlei Huang Subject: Re: Is it possible to employ epoch to simplify managing prison lifecycle In-Reply-To: <9BD54A54-A809-4D3E-BCBA-639E6C61FE37@FreeBSD.org> References: <9BD54A54-A809-4D3E-BCBA-639E6C61FE37@FreeBSD.org> User-Agent: Roundcube Webmail/1.4.11 Message-ID: <4e87ce0b5ea89835d0fa05a91d6e4774@gritton.org> X-Sender: jamie@gritton.org Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-3.30 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+ip4:162.220.209.0/28]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-jail@freebsd.org]; FREEMAIL_CC(0.00)[gmail.com]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:30247, ipnet:162.220.208.0/22, country:US]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[gritton.org]; FREEFALL_USER(0.00)[jamie]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4NYcN64Kb5z3hyB X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On 2022-12-16 06:41, Zhenlei Huang wrote: > While hacking `sys/kern/kern_jail.c` I got lost. > > There're lots of ref / unref and flags to prevent visit invalid prison > while > concurrent modification is possible and some refs looks weird. > > Is it possible to employ epoch(9) to simplify managing of prison > lifecycle ? I imagine it could be used, though I'm not sure offhand if it would make things more or less complicated. There are two issues with the prison_deref flags (which I assume you're talking about): Much of it is tracking whether/how the allprison_lock is held, and I don't see that changing. I want to make sure it remains locked as long as a half-formed prison is in the list, but I also want at least the exclusive holds to be as short as possible. I probably don't want to wait for some epoch timeout to remove something that's not in a usable state, and it doesn't seem right to keep hold of an exclusive lock until then. The other complexity is the two different kinds of reference counts, pr_uref for user-level visibility, and pr_ref for existing at all (and each with its prison_deref flag). I don't think that will change with epochs. I'm not very familiar with epoch(9), having only run into it by watching (but not really participating in) the changes with IP address lists falling under the network epoch. So there's probably more to the underlying concept that what little I got from that, and from a perusal of the man page. - Jamie From nobody Fri Dec 16 22:55:41 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NYkwM6DDRzsxP9 for ; Fri, 16 Dec 2022 22:55:47 +0000 (UTC) (envelope-from bz@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NYkwM5CTrz4GS8; Fri, 16 Dec 2022 22:55:47 +0000 (UTC) (envelope-from bz@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671231347; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=1PFo9vDfTBLn6qxwOf4dc5+A+0gWHaR8ub5jUTEWdFc=; b=kFWClTyajvr5oGWu+Zp6Nk1haqPDJsKs+j1EHIzEpem+ypsGxUKZiJ1VYwZ7T7OH4udX7R ey4kIAk1OXhflMtnfpyoVh9hJyE5/4yoRXX8jxsyPuH/r+bNXSi/RNHWhtk0jzWS2se/Hq uzJWvqsFFOLrwfS5XMhMu3B3gJrRRaAYvTEMw1Sr7LNm9C8WO35CB5kLYNZJKM7wZXYKyK 1zZYr+CRAYNlTVmxIRG7l9Gtf0E5hwKub1gmqdjulX9RvxpC94FRSkej+XSKOejbGj+maa FLD9Pe0clO14OI3KDJj+juX1NEiXXNPLsshUrbivgGoEfFaFdRm3ERJ/o0eonQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671231347; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=1PFo9vDfTBLn6qxwOf4dc5+A+0gWHaR8ub5jUTEWdFc=; b=EQg8noYDRn7y7Mr1/Rh3f70aLRa13gh5suhBC4FLR6FQkkxbH1QtrI3/CG/qmhkeafR4jK 44vVpIljYgPLLBhLigQmgXWQCqL7jwQYEHoHFSc+OXTxnr06vmYfgUgquBexaAihnEt4Vd zS2dx9NrpK5+5VN91g96Z88OjuIWcwTDBonbu01GcvMoG6fV3NUeZrWTCAdCN5hyZo3z8e RO4KOrJSjDQ3gF6GJmw3J1nofvGs5SNIJi9/zR9Foq2Fo1VrJuvV2GI0vRQvEjpI0i4Fcq dFp+pBujvkNAImrd9PKc+fCtDcJjutDYOkUNyjSaJgyUdGJeK2RscJG2kdSWUA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1671231347; a=rsa-sha256; cv=none; b=aESj0latZ9QRXsdb9JZvyA/ud06iXafHNjiV+ie55aPrFirm2o1q8JxgWMqbqpc2G6l5p9 N5jPwbrt6LzQDlAFvXtTABcPBvzf9wHO298tLRL7ylYJlatJBPbjTdB9ARoxDj+8GQza/G MpidleITXxhrcyP1y0Tiea1z8hYhyl83o3rC2zuXo27e75/arC/GxpMyTCFWVX/otx7rUl A6smYKAVXkYa0Yc4r2cG+Ld2HykY3DkKuqwKI0srJPZXIHlooCnkdjJ/P9wOC/Sw5Y4FAc CJPpv83Mqcgq2tSLcREnAAvDhzXJLH5P3VNez2LzNeVFjOm+4t62OyW/5zBOxg== Received: from mx1.sbone.de (cross.sbone.de [195.201.62.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) (Authenticated sender: bz/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4NYkwM1lKDzlQs; Fri, 16 Dec 2022 22:55:47 +0000 (UTC) (envelope-from bz@FreeBSD.org) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id D61CF8D4A228; Fri, 16 Dec 2022 22:55:45 +0000 (UTC) Received: from content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id D134A5C3A833; Fri, 16 Dec 2022 22:55:44 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) by content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (amavisd-new, port 10024) with ESMTP id ACn2vO6Lrzio; Fri, 16 Dec 2022 22:55:42 +0000 (UTC) Received: from strong-iwl0.sbone.de (strong-iwl0.sbone.de [IPv6:fde9:577b:c1a9:4902:b66b:fcff:fef3:e3d2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id 58AA15C3A830; Fri, 16 Dec 2022 22:55:42 +0000 (UTC) Date: Fri, 16 Dec 2022 22:55:41 +0000 (UTC) From: "Bjoern A. Zeeb" To: Zhenlei Huang cc: "freebsd-jail@freebsd.org" , Gleb Smirnoff Subject: Re: What's going on with vnets and epairs w/ addresses? In-Reply-To: <150A60D6-6757-46DD-988F-05A9FFA36821@FreeBSD.org> Message-ID: <1348s3p2-783s-sno2-pp6-rs9oq0s921n@SerrOFQ.bet> References: <5r22os7n-ro15-27q-r356-rps331o06so5@mnoonqbm.arg> <150A60D6-6757-46DD-988F-05A9FFA36821@FreeBSD.org> X-OpenPGP-Key-Id: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-ThisMailContainsUnwantedMimeParts: N On Fri, 16 Dec 2022, Zhenlei Huang wrote: Hi, > I managed to repeat this issue on CURRENT/14 with this small snip: > > ------------------------------------------- > #!/bin/sh > > # test jail name > n="test_ref_leak" > > jail -c name=$n path=/ vnet persist > # The following line trigger jail pr_ref leak > jexec $n ifconfig lo0 inet 127.0.0.1/8 > > jail -R $n > > # wait a moment > sleep 1 > > jls -j $n > > > ------------------------------------------- > > > After DDB debugging and tracing , it seems that is triggered by a combine of [1] and [2] > > [1] https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915 > [2] https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b > > > In [1] the per-VNET uma zone is shared with the global one. > `pcbinfo->ipi_zone = pcbstor->ips_zone;` > > In [2] unref `inp->inp_cred` is deferred called in inpcb_dtor() by uma_zfree_smr() . > > Unfortunately inps freed by uma_zfree_smr() are cached and inpcb_dtor() is not called immediately , > thus leaking `inp->inp_cred` ref and hence `prison->pr_ref`. > > And it is also not possible to free up the cache by per-VNET SYSUNINIT tcp_destroy / udp_destroy / rip_destroy. Thanks a lot for tracking it down. That seems to be a regression then that needs to be fixed before 14.0-RELEASE will happen as it'll break management utilities of people. Could you open a bug report and flag it as such? /bz > > > Best regards, > Zhenlei > >> On Dec 14, 2022, at 9:56 AM, Zhenlei Huang wrote: >> >> >> Hi, >> >> I also encounter this problem while testing gif tunnel between jails. >> >> My script is similar but with additional gif tunnels. >> >> >> There are reports in mailing list [1], [2], and another one in forum [3] . >> >> Seem to be a long standing issue. >> >> [1] https://lists.freebsd.org/pipermail/freebsd-stable/2016-October/086126.html >> [2] https://lists.freebsd.org/pipermail/freebsd-jail/2017-March/003357.html >> [3] https://forums.freebsd.org/threads/jails-stopping-prolonged-deaths-starting-networking-et-cetera.84200/ >> >> >> Best regards, >> Zhenlei >> >>> On Dec 14, 2022, at 7:03 AM, Bjoern A. Zeeb > wrote: >>> >>> Hi, >>> >>> I have used scripts like the below for almost a decade and a half >>> (obviously doing more than that in the middle). I haven't used them >>> much lately but given other questions I just wanted to fire up a test. >>> >>> I have an end-November kernel doing the below my eapirs do not come back >>> to be destroyed (immediately). >>> I have to start polling for the jid to be no longer alive and not in >>> dying state (hence added the jls/ifconfig -l lines and removed the >>> error checking from ifconfig destroy). That seems sometimes rather >>> unreasonably long (to the point I give up). >>> >>> If I don't configure the addresses below this isn't a problem. >>> >>> Sorry I am confused by too many incarnations of the code; I know I once >>> had a version with an async shutdown path but I believe that never made >>> it into mainline, so why are we holding onto the epairs now and not >>> nuking the addresses and returning them and are clean? >>> >>> It's a bit more funny; I added a twiddle loop at the end and nothing >>> happened. So I stop the script and start it again and suddenly another >>> jail or two have cleaned up and their epairs are back. Something feels >>> very very wonky. Play around with this and see ... and let me know if >>> you can reproduce this... I quite wonder why some test cases haven't >>> gone crazy ... >>> >>> /bz >>> >>> ------------------------------------------------------------------------ >>> #!/bin/sh >>> >>> set -e >>> set -x >>> >>> js=`jail -i -c -n jl host.hostname=left.example.net vnet persist` >>> jb=`jail -i -c -n jr host.hostname=right.example.net vnet persist` >>> >>> # Create an epair connecting the two machines (vnet jails). >>> ep=`ifconfig epair create | sed -e 's/a$//'` >>> >>> # Add one end to each vnet jail. >>> ifconfig ${ep}a vnet ${js} >>> ifconfig ${ep}b vnet ${jb} >>> >>> # Add an IP address on the epairs in each vnet jail. >>> # XXX Leave these out and the cleanup seems to work fine. >>> jexec ${js} ifconfig ${ep}a inet 192.0.2.1/24 >>> jexec ${jb} ifconfig ${ep}b inet 192.0.2.2/24 >>> >>> # Clean up. >>> jail -r ${jb} >>> jail -r ${js} >>> >>> # You want to be able to remove this line ... >>> set +e >>> >>> # No epairs to destroy with addresses configured; fine otherwise. >>> ifconfig ${ep}a destroy >>> # echo $? >>> >>> # Add this is here only as things are funny ... >>> # jls -av jid dying >>> # ifconfig -l >>> >>> # end >>> ------------------------------------------------------------------------ >>> >>> -- >>> Bjoern A. Zeeb r15:7 >>> >> > > -- Bjoern A. Zeeb r15:7 From nobody Sat Dec 17 13:36:10 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NZ6SZ3RN2z1ChSh for ; Sat, 17 Dec 2022 13:36:30 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NZ6SZ29FHz3FLL; Sat, 17 Dec 2022 13:36:30 +0000 (UTC) (envelope-from zlei@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671284190; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=1O+TSWc81MpqO/+CeGpBQK+6egGKFOcpGepOT5GAMUo=; b=fKzGxyqOn8icYn0puanjFlW6/t+SnB664umgNbPLdfsZJpdPmowBZ+sjP75mnNM1VeN/n2 sm1RhbJXyHo/SMmt5Z0a7svY3v3w7NNIW+Z81zdScg5wICKqOP6iA30bBRE0yTp7cUWRo3 HcEboJdDt2W+zEdkNH+qJUzaVw1c/eY2kegd/OcCQGoVwLnfS3oYd9Ohta+/BMuFhGg/tY KzEpWth2/Et3CjNr8/ilVQAw57HPJXs/dafdSfy6DlNB7ucnAt8mkHQ3dK0ujMkgLkr5vu u4PTnS4ECh+/AG97ncJGZ+rE4KVe6ARxfeKlm/HGnxvUFQ6+Q48AUdM9mTEr3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671284190; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=1O+TSWc81MpqO/+CeGpBQK+6egGKFOcpGepOT5GAMUo=; b=eqHE675QGkV4P9x0DPWZATiTTRTWj+mCX02B8Z0xAHX9Kr/vhOgoYvAaQwvr6dEdzJHF5I An9BTOPxTZkYJIVjEek6AjMSp4zmNW56qbbZhQWmVQ7+8pU2pHBbkyBs/RRxAck0RgOls4 PDwvmZameFBtHzbkF3LHEkMCuB+bYQLzrwLCJ5rydBFtF//M/6cycPxwm+bp8H2tLtX616 OkrgJhEAQfYVMIMIauP++MeuA7ZBmq53/uaFL0+CcVaoHQSB6D0ljmfQthxEMjlWCIlX4u 9LYmv2ZkXPjDejbVUGWRNNsWEHHufR0DGyD9iruORJW699QS+eQoJqp7abfkDA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1671284190; a=rsa-sha256; cv=none; b=uwUtAQ57f9dGrK4gOrudZw3Iu/71t3gO9pCkY/PPOKdyvAmyEzLeMSNvyL4wdZ+TMUaQYV MtdJjHXjSI0bPb7OZPhwiHeyfPawvUrBWUsnsLp4kJcHMlTyE9IgFz3gz6fFVzK0K+8D2t ZEDogUobeLndCLxwdi+tgmJN8HmbT/Oywxos5UgT2LgXL8z3f4CeUDX7+Lk1YwXvyJue4/ PMilM6UdUBiChSEy6icuqQSrqytqjAkCwuhkmiI2ochOCXv/cRmW9GbqXVqCdwrTWFytln MO/tUP67NM9uW6fcQDuATc5Aa3uurNr7QcGpHuGN+7RYolDnvk7+hGU9niz/3Q== Received: from [IPv6:2409:8a5e:aa71:20f8:40a:ad36:f9f7:a514] (unknown [IPv6:2409:8a5e:aa71:20f8:40a:ad36:f9f7:a514]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: zlei/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4NZ6SW481dz13xg; Sat, 17 Dec 2022 13:36:26 +0000 (UTC) (envelope-from zlei@FreeBSD.org) From: Zhenlei Huang Message-Id: <5F5B4BAD-156F-43E9-80FC-01CB5C9F8741@FreeBSD.org> Content-Type: multipart/alternative; boundary="Apple-Mail=_7038B4AC-00A7-4FE5-8E97-A1D0C96818EB" List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.7\)) Subject: Re: What's going on with vnets and epairs w/ addresses? Date: Sat, 17 Dec 2022 21:36:10 +0800 In-Reply-To: <1348s3p2-783s-sno2-pp6-rs9oq0s921n@SerrOFQ.bet> Cc: "freebsd-jail@freebsd.org" , Gleb Smirnoff To: "Bjoern A. Zeeb" References: <5r22os7n-ro15-27q-r356-rps331o06so5@mnoonqbm.arg> <150A60D6-6757-46DD-988F-05A9FFA36821@FreeBSD.org> <1348s3p2-783s-sno2-pp6-rs9oq0s921n@SerrOFQ.bet> X-Mailer: Apple Mail (2.3608.120.23.2.7) X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_7038B4AC-00A7-4FE5-8E97-A1D0C96818EB Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On Dec 17, 2022, at 6:55 AM, Bjoern A. Zeeb > wrote: >=20 > On Fri, 16 Dec 2022, Zhenlei Huang wrote: >=20 > Hi, >=20 >> I managed to repeat this issue on CURRENT/14 with this small snip: >>=20 >> ------------------------------------------- >> #!/bin/sh >>=20 >> # test jail name >> n=3D"test_ref_leak" >>=20 >> jail -c name=3D$n path=3D/ vnet persist >> # The following line trigger jail pr_ref leak >> jexec $n ifconfig lo0 inet 127.0.0.1/8 >>=20 >> jail -R $n >>=20 >> # wait a moment >> sleep 1 >>=20 >> jls -j $n >>=20 >>=20 >> ------------------------------------------- >>=20 >>=20 >> After DDB debugging and tracing , it seems that is triggered by a = combine of [1] and [2] >>=20 >> [1] = https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915 = > >> [2] = https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b = > >>=20 >>=20 >> In [1] the per-VNET uma zone is shared with the global one. >> `pcbinfo->ipi_zone =3D pcbstor->ips_zone;` >>=20 >> In [2] unref `inp->inp_cred` is deferred called in inpcb_dtor() by = uma_zfree_smr() . >>=20 >> Unfortunately inps freed by uma_zfree_smr() are cached and = inpcb_dtor() is not called immediately , >> thus leaking `inp->inp_cred` ref and hence `prison->pr_ref`. >>=20 >> And it is also not possible to free up the cache by per-VNET = SYSUNINIT tcp_destroy / udp_destroy / rip_destroy. >=20 > Thanks a lot for tracking it down. >=20 > That seems to be a regression then that needs to be fixed before > 14.0-RELEASE will happen as it'll break management utilities of = people. >=20 > Could you open a bug report and flag it as such? While I was trying to open a new bug report Bugzilla prompt an existing = PR https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D264981 = =20 opened by olivier. I think this issue is same with that one. >=20 > /bz >=20 >=20 >>=20 >>=20 >> Best regards, >> Zhenlei >>=20 >>> On Dec 14, 2022, at 9:56 AM, Zhenlei Huang > wrote: >>>=20 >>>=20 >>> Hi, >>>=20 >>> I also encounter this problem while testing gif tunnel between = jails. >>>=20 >>> My script is similar but with additional gif tunnels. >>>=20 >>>=20 >>> There are reports in mailing list [1], [2], and another one in forum = [3] . >>>=20 >>> Seem to be a long standing issue. >>>=20 >>> [1] = https://lists.freebsd.org/pipermail/freebsd-stable/2016-October/086126.htm= l = > >>> [2] = https://lists.freebsd.org/pipermail/freebsd-jail/2017-March/003357.html = = >= >>> [3] = https://forums.freebsd.org/threads/jails-stopping-prolonged-deaths-startin= g-networking-et-cetera.84200/ = > >>>=20 >>>=20 >>> Best regards, >>> Zhenlei >>>=20 >>>> On Dec 14, 2022, at 7:03 AM, Bjoern A. Zeeb >> = wrote: >>>>=20 >>>> Hi, >>>>=20 >>>> I have used scripts like the below for almost a decade and a half >>>> (obviously doing more than that in the middle). I haven't used = them >>>> much lately but given other questions I just wanted to fire up a = test. >>>>=20 >>>> I have an end-November kernel doing the below my eapirs do not come = back >>>> to be destroyed (immediately). >>>> I have to start polling for the jid to be no longer alive and not = in >>>> dying state (hence added the jls/ifconfig -l lines and removed the >>>> error checking from ifconfig destroy). That seems sometimes rather >>>> unreasonably long (to the point I give up). >>>>=20 >>>> If I don't configure the addresses below this isn't a problem. >>>>=20 >>>> Sorry I am confused by too many incarnations of the code; I know I = once >>>> had a version with an async shutdown path but I believe that never = made >>>> it into mainline, so why are we holding onto the epairs now and not >>>> nuking the addresses and returning them and are clean? >>>>=20 >>>> It's a bit more funny; I added a twiddle loop at the end and = nothing >>>> happened. So I stop the script and start it again and suddenly = another >>>> jail or two have cleaned up and their epairs are back. Something = feels >>>> very very wonky. Play around with this and see ... and let me know = if >>>> you can reproduce this... I quite wonder why some test cases = haven't >>>> gone crazy ... >>>>=20 >>>> /bz >>>>=20 >>>> = ------------------------------------------------------------------------ >>>> #!/bin/sh >>>>=20 >>>> set -e >>>> set -x >>>>=20 >>>> js=3D`jail -i -c -n jl host.hostname=3Dleft.example.net = > vnet persist` >>>> jb=3D`jail -i -c -n jr host.hostname=3Dright.example.net = > vnet persist` >>>>=20 >>>> # Create an epair connecting the two machines (vnet jails). >>>> ep=3D`ifconfig epair create | sed -e 's/a$//'` >>>>=20 >>>> # Add one end to each vnet jail. >>>> ifconfig ${ep}a vnet ${js} >>>> ifconfig ${ep}b vnet ${jb} >>>>=20 >>>> # Add an IP address on the epairs in each vnet jail. >>>> # XXX Leave these out and the cleanup seems to work fine. >>>> jexec ${js} ifconfig ${ep}a inet 192.0.2.1/24 >>>> jexec ${jb} ifconfig ${ep}b inet 192.0.2.2/24 >>>>=20 >>>> # Clean up. >>>> jail -r ${jb} >>>> jail -r ${js} >>>>=20 >>>> # You want to be able to remove this line ... >>>> set +e >>>>=20 >>>> # No epairs to destroy with addresses configured; fine otherwise. >>>> ifconfig ${ep}a destroy >>>> # echo $? >>>>=20 >>>> # Add this is here only as things are funny ... >>>> # jls -av jid dying >>>> # ifconfig -l >>>>=20 >>>> # end >>>> = ------------------------------------------------------------------------ >>>>=20 >>>> -- >>>> Bjoern A. Zeeb = r15:7 >>>>=20 >>>=20 >>=20 >>=20 >=20 > --=20 > Bjoern A. Zeeb = r15:7 --Apple-Mail=_7038B4AC-00A7-4FE5-8E97-A1D0C96818EB Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii

On = Dec 17, 2022, at 6:55 AM, Bjoern A. Zeeb <bz@FreeBSD.org> = wrote:

On Fri, 16 Dec 2022, Zhenlei = Huang wrote:

Hi,

I = managed to repeat this issue on CURRENT/14 with this small snip:

-------------------------------------------
#!/bin/sh

# test jail name
n=3D"test_ref_leak"

jail -c = name=3D$n path=3D/ vnet persist
# The following line = trigger jail pr_ref leak
jexec $n ifconfig lo0 inet = 127.0.0.1/8

jail -R $n

# wait a moment
sleep 1

jls -j $n


-------------------------------------------


After DDB debugging and tracing , it seems = that is triggered by a combine of [1] and [2]

[1] https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5a= c895915<https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5a= c895915>
[2] https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b1= 75b8c5b<https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b1= 75b8c5b>


In [1] the = per-VNET uma zone is shared with the global one.
`pcbinfo->ipi_zone =3D pcbstor->ips_zone;`

In [2] unref `inp->inp_cred` is deferred = called in inpcb_dtor() by uma_zfree_smr() .

Unfortunately inps freed by uma_zfree_smr() are cached and = inpcb_dtor() is not called immediately ,
thus leaking = `inp->inp_cred` ref and hence `prison->pr_ref`.

And it is also not possible to free up the cache by per-VNET = SYSUNINIT tcp_destroy / udp_destroy / rip_destroy.

Thanks a lot for tracking it down.

That seems to be a regression = then that needs to be fixed before
14.0-RELEASE will happen as it'll break management utilities = of people.

Could you = open a bug report and flag it as such?

While I was trying to open a new bug = report Bugzilla prompt an existing PR https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D264981&= nbsp;
opened by  olivier. I think this issue = is same with that one.


/bz




Best = regards,
Zhenlei

On Dec 14, 2022, at 9:56 AM, Zhenlei Huang = <zlei@FreeBSD.org> wrote:


Hi,

I also = encounter this problem while testing gif tunnel between jails.

My script is similar but with additional gif = tunnels.


There are reports = in mailing list [1], [2], and another one in forum [3] .
Seem to be a long standing issue.

[1] https://lists.freebsd.org/pipermail/freebsd-stable/2016-October= /086126.html<https://lists.freebsd.org/pipermail/freebsd-stable/2016-October= /086126.html>
[2] https://lists.freebsd.org/pipermail/freebsd-jail/2017-March/003= 357.html <https://lists.freebsd.org/pipermail/freebsd-jail/2017-March/003= 357.html>
[3] https://forums.freebsd.org/threads/jails-stopping-prolonged-dea= ths-starting-networking-et-cetera.84200/<https://forums.freebsd.org/threads/jails-stopping-prolonged-dea= ths-starting-networking-et-cetera.84200/>


Best regards,
Zhenlei

On Dec = 14, 2022, at 7:03 AM, Bjoern A. Zeeb <bz@FreeBSD.org <mailto:bz@FreeBSD.org>> wrote:

Hi,

I have used scripts like the = below for almost a decade and a half
(obviously doing more = than that in the middle).  I haven't used them
much = lately but given other questions I just wanted to fire up a test.

I have an end-November kernel doing the below = my eapirs do not come back
to be destroyed = (immediately).
I have to start polling for the jid to be = no longer alive and not in
dying state (hence added the = jls/ifconfig -l lines and removed the
error checking from = ifconfig destroy).  That seems sometimes rather
unreasonably long (to the point I give up).

If I don't configure the addresses below this isn't a = problem.

Sorry I am confused by too many = incarnations of the code; I know I once
had a version with = an async shutdown path but I believe that never made
it = into mainline, so why are we holding onto the epairs now and not
nuking the addresses and returning them and are clean?

It's a bit more funny; I added a twiddle loop = at the end and nothing
happened.  So I stop the = script and start it again and suddenly another
jail or two = have cleaned up and their epairs are back.  Something feels
very very wonky.  Play around with this and see ... and = let me know if
you can reproduce this...  I quite = wonder why some test cases haven't
gone crazy ...

/bz

---------------------------------------------------------------= ---------
#!/bin/sh

set -e
set -x

js=3D`jail -i -c -n jl = host.hostname=3Dleft.example.net <http://left.example.net/> vnet persist`
jb=3D`jail -i -c -n jr host.hostname=3Dright.example.net <http://right.example.net/> vnet persist`

# Create an epair connecting the two machines = (vnet jails).
ep=3D`ifconfig epair create | sed -e = 's/a$//'`

# Add one end to each vnet = jail.
ifconfig ${ep}a vnet ${js}
ifconfig = ${ep}b vnet ${jb}

# Add an IP address on = the epairs in each vnet jail.
# XXX Leave these out and = the cleanup seems to work fine.
jexec ${js}  ifconfig = ${ep}a inet  192.0.2.1/24
jexec ${jb}  ifconfig = ${ep}b inet  192.0.2.2/24

# Clean = up.
jail -r ${jb}
jail -r ${js}

# You want to be able to remove this line = ...
set +e

# No epairs to = destroy with addresses configured; fine otherwise.
ifconfig = ${ep}a destroy
# echo $?

# = Add this is here only as things are funny ...
# jls -av = jid dying
# ifconfig -l

# = end
---------------------------------------------------------------= ---------

--
Bjoern A. Zeeb =             &n= bsp;           &nbs= p;            =             &n= bsp;  r15:7





-- 
Bjoern A. Zeeb =             &n= bsp;           &nbs= p;            =             &n= bsp;  r15:7

= --Apple-Mail=_7038B4AC-00A7-4FE5-8E97-A1D0C96818EB-- From nobody Sat Dec 17 17:13:14 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NZCTP4Scrzt682 for ; Sat, 17 Dec 2022 17:22:33 +0000 (UTC) (envelope-from glebius@freebsd.org) Received: from glebi.us (glebi.us [162.251.186.162]) by mx1.freebsd.org (Postfix) with ESMTP id 4NZCTN5HKRz3y1N; Sat, 17 Dec 2022 17:22:32 +0000 (UTC) (envelope-from glebius@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=softfail (mx1.freebsd.org: 162.251.186.162 is neither permitted nor denied by domain of glebius@freebsd.org) smtp.mailfrom=glebius@freebsd.org; dmarc=none Received: by glebi.us (Postfix, from userid 1000) id C7C2843545; Sat, 17 Dec 2022 09:13:14 -0800 (PST) Date: Sat, 17 Dec 2022 09:13:14 -0800 From: Gleb Smirnoff To: Zhenlei Huang Cc: "Bjoern A. Zeeb" , "freebsd-jail@freebsd.org" Subject: Re: What's going on with vnets and epairs w/ addresses? Message-ID: References: <5r22os7n-ro15-27q-r356-rps331o06so5@mnoonqbm.arg> <150A60D6-6757-46DD-988F-05A9FFA36821@FreeBSD.org> List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <150A60D6-6757-46DD-988F-05A9FFA36821@FreeBSD.org> X-Spamd-Result: default: False [1.50 / 15.00]; VIOLATED_DIRECT_SPF(3.50)[]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.995]; MIME_GOOD(-0.10)[text/plain]; RCVD_NO_TLS_LAST(0.10)[]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[freebsd.org]; MLMMJ_DEST(0.00)[freebsd-jail@freebsd.org]; FREEFALL_USER(0.00)[glebius]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; R_SPF_SOFTFAIL(0.00)[~all:c]; TO_DN_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:27348, ipnet:162.251.186.0/24, country:US]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; GREYLIST(0.00)[pass,body] X-Rspamd-Queue-Id: 4NZCTN5HKRz3y1N X-Spamd-Bar: + X-ThisMailContainsUnwantedMimeParts: N Zhenlei, On Fri, Dec 16, 2022 at 06:30:57PM +0800, Zhenlei Huang wrote: Z> I managed to repeat this issue on CURRENT/14 with this small snip: Z> Z> ------------------------------------------- Z> #!/bin/sh Z> Z> # test jail name Z> n="test_ref_leak" Z> Z> jail -c name=$n path=/ vnet persist Z> # The following line trigger jail pr_ref leak Z> jexec $n ifconfig lo0 inet 127.0.0.1/8 Z> Z> jail -R $n Z> Z> # wait a moment Z> sleep 1 Z> Z> jls -j $n Z> Z> After DDB debugging and tracing , it seems that is triggered by a combine of [1] and [2] Z> Z> [1] https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915 Z> [2] https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b Z> Z> Z> In [1] the per-VNET uma zone is shared with the global one. Z> `pcbinfo->ipi_zone = pcbstor->ips_zone;` Z> Z> In [2] unref `inp->inp_cred` is deferred called in inpcb_dtor() by uma_zfree_smr() . Z> Z> Unfortunately inps freed by uma_zfree_smr() are cached and inpcb_dtor() is not called immediately , Z> thus leaking `inp->inp_cred` ref and hence `prison->pr_ref`. Z> Z> And it is also not possible to free up the cache by per-VNET SYSUNINIT tcp_destroy / udp_destroy / rip_destroy. This is known issue and I'd prefer not to call it a problem. The "leak" of a jail happens only if machine is idle wrt the networking activity. Getting back to the problem that started this thread - the epair(4)s not immediately popping back to prison0. IMHO, the problem again lies in the design of if_vmove and epair(4) in particular. The if_vmove shall not exist, instead we should do a full if_attach() and if_detach(). The state of an ifnet when it undergoes if_vmove doesn't carry any useful information. With Alexander melifaro@ we discussed better options for creating or attaching interfaces to jails that if_vmove. Until they are ready the most easy workaround to deal with annoying epair(4) come back problem is to remove it manually before destroying a jail, like I did in 80fc25025ff. -- Gleb Smirnoff From nobody Sat Dec 17 19:23:15 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NZG8m3v4Nz1Cgbs for ; Sat, 17 Dec 2022 19:23:20 +0000 (UTC) (envelope-from bz@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NZG8m3LHZz4Flh; Sat, 17 Dec 2022 19:23:20 +0000 (UTC) (envelope-from bz@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671305000; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=I76dFbrStqbMe81XkNxRp1pLwKcqskjKj/dACTxwY/0=; b=QMQKaLEOG0Qzp3cO1cIrS83MMfrp7Tc9S6cj6gAs4CtBn3IQ1KvNkqS8hkKJSBKdTywKOv 5qN+a+khqO0s0kACXeJPNVx8TNxFFfyEMEXStQhLZWESiwWkX+3+2g7nB6BXadmkPYHkE5 QVbOMteUmf/umgvO5DCqjNls/or4ceDaof4BCoMlllYdGX3o3dhiuptS91DhLoRjJ3Gcnc E954Y/m3COYXckyuc2Q7HCzZwCEOcSO6pOOYAleml+pQU9c0R6gyb7kjkE3mj86ADDToJw RDKwV+UpIskURXJXikb7K8LuvqGu3VIS/y3F9Tr8tKZTbnjwj5kKr2+RJmCdNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671305000; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=I76dFbrStqbMe81XkNxRp1pLwKcqskjKj/dACTxwY/0=; b=UJTGHS34nOszTrdWyRB6tncjMhARBc1DIeo903AQHPIUY81gQA/CuPfoPGJAXMQqygBSQ3 MJpxOuizkGbf9/s4CTW5kyNVd8FVOiTORAyXcDL6EGyc9K5R/K8a8QeU+EjnVb8ska6eMz UzwPZa7JIlA0F6csOhcJGVHsqDYMMiWqq6AII/uZ6ccJjM41h/x6qNQ02kJtzX48O3elv0 KhlYaShTU4TjqoACLkxy0yZSnb5fu9uIBE0NtlawJHRi5pI338hg8CCt1sAczzfd9dKQ+K YHFKuc2SQaLz6k3Ylv/h4A0HoSDCxV8xflapKIv6UUrAZJ6IztoKQPwPalm9Ww== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1671305000; a=rsa-sha256; cv=none; b=h1Qp+9Nglk3NNWcOq2SxMsmxDlP2z4HAUIUoxvDI1mhgrCICMfvs8BNHKe0lPPZ6jxBB9B TX2funXij+F/CtcJeTwZFDtrYWX6R7qSBcSyzGYQJ1GSG+wZoW1WlyAMXsYaxnRf24T95H 0yT+M0HR6Mzyoui5hwsETVMWRzokWVM7h8fHwnwxCm13NhVqrGI7LsMAvOfm51HlJSg1Os MKrNWmzfHxud/Pw8KaOffW9exrhEnc7YLFqNy8l9WC7xdAO/SdhOkHFl5N82SvcewEcnff VbeL+SXWMlZlrfeViHspiTLM7WBoBl7YMgy2QZBtOWLl+cr/rU5WLZXWM5nFmw== Received: from mx1.sbone.de (cross.sbone.de [195.201.62.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) (Authenticated sender: bz/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4NZG8m11jRz18JL; Sat, 17 Dec 2022 19:23:20 +0000 (UTC) (envelope-from bz@freebsd.org) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id 3D9478D4A129; Sat, 17 Dec 2022 19:23:18 +0000 (UTC) Received: from content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id 9B4F25C3A833; Sat, 17 Dec 2022 19:23:17 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) by content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (amavisd-new, port 10024) with ESMTP id eHZVb-2OWWyW; Sat, 17 Dec 2022 19:23:16 +0000 (UTC) Received: from strong-iwl0.sbone.de (strong-iwl0.sbone.de [IPv6:fde9:577b:c1a9:4902:b66b:fcff:fef3:e3d2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id D44C55C3A830; Sat, 17 Dec 2022 19:23:15 +0000 (UTC) Date: Sat, 17 Dec 2022 19:23:15 +0000 (UTC) From: "Bjoern A. Zeeb" To: Gleb Smirnoff cc: Zhenlei Huang , "freebsd-jail@freebsd.org" Subject: Re: What's going on with vnets and epairs w/ addresses? In-Reply-To: Message-ID: <9p9919q1-n639-p581-6q1o-so48o5ns6717@serrofq.bet> References: <5r22os7n-ro15-27q-r356-rps331o06so5@mnoonqbm.arg> <150A60D6-6757-46DD-988F-05A9FFA36821@FreeBSD.org> X-OpenPGP-Key-Id: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-ThisMailContainsUnwantedMimeParts: N On Sat, 17 Dec 2022, Gleb Smirnoff wrote: > Zhenlei, > > On Fri, Dec 16, 2022 at 06:30:57PM +0800, Zhenlei Huang wrote: > Z> I managed to repeat this issue on CURRENT/14 with this small snip: > Z> > Z> ------------------------------------------- > Z> #!/bin/sh > Z> > Z> # test jail name > Z> n="test_ref_leak" > Z> > Z> jail -c name=$n path=/ vnet persist > Z> # The following line trigger jail pr_ref leak > Z> jexec $n ifconfig lo0 inet 127.0.0.1/8 > Z> > Z> jail -R $n > Z> > Z> # wait a moment > Z> sleep 1 > Z> > Z> jls -j $n > Z> > Z> After DDB debugging and tracing , it seems that is triggered by a combine of [1] and [2] > Z> > Z> [1] https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915 > Z> [2] https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b > Z> > Z> > Z> In [1] the per-VNET uma zone is shared with the global one. > Z> `pcbinfo->ipi_zone = pcbstor->ips_zone;` > Z> > Z> In [2] unref `inp->inp_cred` is deferred called in inpcb_dtor() by uma_zfree_smr() . > Z> > Z> Unfortunately inps freed by uma_zfree_smr() are cached and inpcb_dtor() is not called immediately , > Z> thus leaking `inp->inp_cred` ref and hence `prison->pr_ref`. > Z> > Z> And it is also not possible to free up the cache by per-VNET SYSUNINIT tcp_destroy / udp_destroy / rip_destroy. > > This is known issue and I'd prefer not to call it a problem. The "leak" of a jail > happens only if machine is idle wrt the networking activity. > > Getting back to the problem that started this thread - the epair(4)s not immediately > popping back to prison0. IMHO, the problem again lies in the design of if_vmove and > epair(4) in particular. The if_vmove shall not exist, instead we should do a full > if_attach() and if_detach(). The state of an ifnet when it undergoes if_vmove doesn't > carry any useful information. With Alexander melifaro@ we discussed better options > for creating or attaching interfaces to jails that if_vmove. Until they are ready > the most easy workaround to deal with annoying epair(4) come back problem is to > remove it manually before destroying a jail, like I did in 80fc25025ff. Ok, move an em0 or cxl0 into the jail; the problem will be the same I bet and you need the physical interface to not disappear as then you cannot re-create a new jail with it. /bz -- Bjoern A. Zeeb r15:7 From nobody Sun Dec 18 06:10:03 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NZXX8281Cz1GKk1 for ; Sun, 18 Dec 2022 06:11:04 +0000 (UTC) (envelope-from zlei.huang@gmail.com) Received: from mail-pj1-x1034.google.com (mail-pj1-x1034.google.com [IPv6:2607:f8b0:4864:20::1034]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NZXX8037Yz4G8X; Sun, 18 Dec 2022 06:11:04 +0000 (UTC) (envelope-from zlei.huang@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pj1-x1034.google.com with SMTP id fy4so6242946pjb.0; Sat, 17 Dec 2022 22:11:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:subject:mime-version:from:from:to:cc:subject:date :message-id:reply-to; bh=YSCClxagI+cuEbmy9D2fCm7BvTqHt8IxALefIsLUmWQ=; b=SVw2jshqAdq71NlnN6894zo2kdY3m8tHuSCg20Op7iBqaIiCp0iqlxd2G2K9HDLtUA pDhICGNl6EU3PUdg5jfexejbCaNJzWT1rLnllhA33RSwW3wi3xQUY16UoUYsAK7bbWrt OKEnYeyD9wWDyVrnhmws2raFOuasiSYqYrclqsomCs6LVD7DeVCziJFiOx1jbKmx66Q1 E0YX2DtSZLvflNbcglqv6XOskz2BKJjdd6HbAJqObMJjMUVBbEAZGmKCltHrjJQnfilV 9OvLGVfLQAPaYFJiRwgt71tyRq1+SQpTlxpK1UhPYV2pL1NJZu9I1RXSv3P6FWn0Poj2 46Sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:subject:mime-version:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YSCClxagI+cuEbmy9D2fCm7BvTqHt8IxALefIsLUmWQ=; b=cfoHfbqcYHiXjKgJzN+4EWjV++Xrt0kFZj1QotHPZaiCIpy6K2+iFYKe1DILGYgWOV DrlgYvIhcT0casbjKQ7cVxa2KFxWs4dpbZXk7cttCLSQP8pKtjyaMRfSKrgHBasF/bhU wPwTXXmw1ttfnLSLSe7gHzR/L0XOnoMo8YOPu5se4kSvBUlpknkNyG1almVIxwdI5tm9 TnZ7RR/Jvaociab/2reEoMoowzfzyLsrv7Mneu/bMQQA2/CcbzmA29TnIz1nsYqktYNv xznnp200x1gRA2rZoah9MG2J3/P4pU3VGzsNM4We07cQ/XyDPHB4hRCS4UU1c3Hq/W7n /Byw== X-Gm-Message-State: ANoB5pnM20n/BsqkocZwLJnH6ir2iUJgBMH6XN0+ZqfPvy9GH/0y+Swk 4677uqZBW8QQDEaoSK6T/noYSSZck9FxW+0R X-Google-Smtp-Source: AA0mqf70Wr/IoJ2FEPwetzF6rwZk9zB4dSheklyMoiNelbOgHpiqdq0qvqOjG+YRfvVonUdTPJJG0Q== X-Received: by 2002:a17:902:a711:b0:189:747e:97cc with SMTP id w17-20020a170902a71100b00189747e97ccmr36006668plq.26.1671343862402; Sat, 17 Dec 2022 22:11:02 -0800 (PST) Received: from [172.17.252.129] (ns1.oxydns.net. [45.32.91.63]) by smtp.gmail.com with ESMTPSA id m10-20020a170902db0a00b0017f72a430adsm4458917plx.71.2022.12.17.22.11.00 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 17 Dec 2022 22:11:01 -0800 (PST) From: Zhenlei Huang X-Google-Original-From: Zhenlei Huang Content-Type: text/plain; charset=us-ascii List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.7\)) Subject: Re: What's going on with vnets and epairs w/ addresses? In-Reply-To: <9p9919q1-n639-p581-6q1o-so48o5ns6717@serrofq.bet> Date: Sun, 18 Dec 2022 14:10:03 +0800 Cc: Gleb Smirnoff , "freebsd-jail@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: <6B201617-68BC-4CC8-A2AE-908E96D69B67@FreeBSD.org> References: <5r22os7n-ro15-27q-r356-rps331o06so5@mnoonqbm.arg> <150A60D6-6757-46DD-988F-05A9FFA36821@FreeBSD.org> <9p9919q1-n639-p581-6q1o-so48o5ns6717@serrofq.bet> To: "Bjoern A. Zeeb" X-Mailer: Apple Mail (2.3608.120.23.2.7) X-Rspamd-Queue-Id: 4NZXX8037Yz4G8X X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N > On Dec 18, 2022, at 3:23 AM, Bjoern A. Zeeb wrote: >=20 > On Sat, 17 Dec 2022, Gleb Smirnoff wrote: >=20 >> Zhenlei, >>=20 >> On Fri, Dec 16, 2022 at 06:30:57PM +0800, Zhenlei Huang wrote: >> Z> I managed to repeat this issue on CURRENT/14 with this small snip: >> Z> >> Z> ------------------------------------------- >> Z> #!/bin/sh >> Z> >> Z> # test jail name >> Z> n=3D"test_ref_leak" >> Z> >> Z> jail -c name=3D$n path=3D/ vnet persist >> Z> # The following line trigger jail pr_ref leak >> Z> jexec $n ifconfig lo0 inet 127.0.0.1/8 >> Z> >> Z> jail -R $n >> Z> >> Z> # wait a moment >> Z> sleep 1 >> Z> >> Z> jls -j $n >> Z> >> Z> After DDB debugging and tracing , it seems that is triggered by a = combine of [1] and [2] >> Z> >> Z> [1] = https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915 = >> Z> [2] = https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b = I can confirm [2] also affects Non-VNET jails. Prison pr_ref leak cause jail stuck in dying state. >> Z> >> Z> >> Z> In [1] the per-VNET uma zone is shared with the global one. >> Z> `pcbinfo->ipi_zone =3D pcbstor->ips_zone;` >> Z> >> Z> In [2] unref `inp->inp_cred` is deferred called in inpcb_dtor() by = uma_zfree_smr() . >> Z> >> Z> Unfortunately inps freed by uma_zfree_smr() are cached and = inpcb_dtor() is not called immediately , >> Z> thus leaking `inp->inp_cred` ref and hence `prison->pr_ref`. >> Z> >> Z> And it is also not possible to free up the cache by per-VNET = SYSUNINIT tcp_destroy / udp_destroy / rip_destroy. >>=20 >> This is known issue and I'd prefer not to call it a problem. The = "leak" of a jail >> happens only if machine is idle wrt the networking activity. >>=20 >> Getting back to the problem that started this thread - the epair(4)s = not immediately >> popping back to prison0. IMHO, the problem again lies in the design = of if_vmove and >> epair(4) in particular. The if_vmove shall not exist, instead we = should do a full >> if_attach() and if_detach(). The state of an ifnet when it undergoes = if_vmove doesn't >> carry any useful information. With Alexander melifaro@ we discussed = better options >> for creating or attaching interfaces to jails that if_vmove. Until = they are ready >> the most easy workaround to deal with annoying epair(4) come back = problem is to >> remove it manually before destroying a jail, like I did in = 80fc25025ff. >=20 > Ok, move an em0 or cxl0 into the jail; the problem will be the same I > bet and you need the physical interface to not disappear as then you > cannot re-create a new jail with it. Re-read sys/kern/kern_jail.c, if pr_ref leaks, vnet_destroy() has no = chance to be called, thus if_vmove is not called and epair(4)s or em0, exl0 are not returned to = home vnet. That can be confirmed by setting debug point on vnet_destroy by DDB, and = then create and destroy vnet jails. So before the problem prison pr_ref count leaks is resolved, it will = cover other potential problems such as @glebius pointed out. I think the problem that prison ref count leaks should be resolved = first. I'm also reviewing the life cycles of prison / vnet and it seems they = could still be improved. >=20 > /bz >=20 > --=20 > Bjoern A. Zeeb = r15:7 From nobody Sun Dec 18 08:01:20 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NZZzW2RRfz1GprZ; Sun, 18 Dec 2022 08:01:27 +0000 (UTC) (envelope-from zlei.huang@gmail.com) Received: from mail-pg1-x529.google.com (mail-pg1-x529.google.com [IPv6:2607:f8b0:4864:20::529]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NZZzV4S6Sz4PJl; Sun, 18 Dec 2022 08:01:26 +0000 (UTC) (envelope-from zlei.huang@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=F6roBBiO; spf=pass (mx1.freebsd.org: domain of zlei.huang@gmail.com designates 2607:f8b0:4864:20::529 as permitted sender) smtp.mailfrom=zlei.huang@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pg1-x529.google.com with SMTP id 36so4320246pgp.10; Sun, 18 Dec 2022 00:01:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:date:message-id:subject:mime-version:content-transfer-encoding :from:from:to:cc:subject:date:message-id:reply-to; bh=DDhAhmQegCLVp1KDXNxcfrOo7um/S/b+5jj5cIu/r0E=; b=F6roBBiOnbCW6bf2KffLiGeV3YbwqYt+d3aaRuTN9mHk7g+iAuUEZE2I2KvsrdW7vj zpqk/MBf/ztjsNwLotmV+f42ZVWtw42lt6d2XQdZUQ/Wpi+vCon6sNbvub/C9MzvfnYN eEoql8cnjxI10aSicYhijN4pAKM4cDwQNFpY12siz/y+gOPVRrPb18w3lL4k2pLkWnfu qra0N3RsUUzEMsRO65ppZfTeleaTFcGLMH+qY3c/NkJnIpOfWpmP8jlhTD08tn9Pu2WS /FYRlQd68VpfkkFk5vp56zgFyp0OSRtw7zUBA9cERPtjoFGAfNL2csNWMwSvrqwWV676 EReg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:date:message-id:subject:mime-version:content-transfer-encoding :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=DDhAhmQegCLVp1KDXNxcfrOo7um/S/b+5jj5cIu/r0E=; b=iuoQYkpwgbqkXUeJLJCVUkzemgQlFIFP5R47crVlz4FWbimPco+ewpO4YZS6ZesJMQ 4ji4U5Ycf9fN/7TtilTmMnghG45c/GGiy10IlMQUi0q5LI+oWWsInGxF5VNQQRum7ab3 /WhIAh7GwyHx1AsafLXUBOL++AKs8wHIzCkLNfffue/xi6XpiuJPaou0xBMGHGuo7uvo aD9ANsWR20JxdwV8BEAiryFvQj2EYRX+OSQX6Q0iNhGMfHqR6BoN9ym76kB93lAwmNAP vVOh2u5NVUW21n5zdVO7Hq8NxsLokR6rn7SzYQO9EeU5EHLCNQdbiUkX+kOwZFtzk+ow 4Ryw== X-Gm-Message-State: ANoB5pkRKIijaDU0RkcScO1zU1WpdiyjJFcyCF1iqWmbhobBoQdMsGQQ pwh8GFiEa4sEhZxvHtV8KeR/G47KNpuBm/qS X-Google-Smtp-Source: AA0mqf7m9bM02iK+LxbpEyW+DNPiI8XWpJwdqbXQ0GZ+I5NwGuLT0GmGzOwCHNzgDBGY2Z/uE8yYUQ== X-Received: by 2002:a62:3644:0:b0:576:663b:6614 with SMTP id d65-20020a623644000000b00576663b6614mr37324328pfa.2.1671350485257; Sun, 18 Dec 2022 00:01:25 -0800 (PST) Received: from [172.17.252.129] (ns1.oxydns.net. [45.32.91.63]) by smtp.gmail.com with ESMTPSA id k26-20020aa7999a000000b00574679561b4sm4162084pfh.134.2022.12.18.00.01.23 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 18 Dec 2022 00:01:24 -0800 (PST) From: Zhenlei Huang X-Google-Original-From: Zhenlei Huang Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.7\)) Subject: Propose a new stage `vnet_shutdown` before `vnet_destroy` Message-Id: Date: Sun, 18 Dec 2022 16:01:20 +0800 To: freebsd-jail@freebsd.org, freebsd-net X-Mailer: Apple Mail (2.3608.120.23.2.7) X-Spamd-Result: default: False [-2.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; MID_RHS_MATCH_TO(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TAGGED_FROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::529:from]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DKIM_TRACE(0.00)[gmail.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; MLMMJ_DEST(0.00)[freebsd-jail@freebsd.org,freebsd-net@freebsd.org] X-Rspamd-Queue-Id: 4NZZzV4S6Sz4PJl X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N Hi, I'm currently working on route nexthop caching feature for tunneling = interfaces such as if_gif, if_gre, if_vxlan, and potentially if_wg. I encounter a nasty bug = related to VNET lifecycle. More preciously I'd like to call `rib_unsubscribe()` to unsubscribe = route event when the interface tunnel is deleted (gif_delete_tunnel). While on VNET shutting down, VNET SYSUNINIT was called and the routing = vnet subsystem is destroyed before the interface going down and hence cause pagefault. = I do not want to check `vnet.vnet_shutdown` state as it looks messed up. I'm recently reviewing the life cycles of prison and get some = inspirations. When the jail / prison is submitted to destroy ( by jail_remove syscall = ) then SIGKILL is sent to the prison's processes. I think it is correct order to destroy jail / = prison. To summarize, the life cycle=20 of jail / prison is: on jail create: PRISON_STATE_INVALID -> create VNET -> = PRISON_STATE_ALIVE -> setup network resources, ifnet, if addresses, = routing, etc. -> create / attach (network) processes=20 on jail destroy: jexec kill processes (1) by user -> mark it as = PRISON_STATE_DYING -> send SIGKILL to processes by kernel (2) -> = destroy VNET (if prison pr_ref go to the last one) -> DYED The (2) is a cleanup by kernel as (1) is possible not done by user. So it comes the idea about the life cycle of VNET. While on jail destroy, the network resources are cleaned up by = vnet_destroy ( SYSUNINIT ). Then the order of SYSUNINIT of network components is hacking as circular network = resource dependency is possible. For example the routing table entries (nhop) have reference of ifnet, = and ifnet have reference to route nhop (cache), as=20 I encountered. Just like the cleanup processes by kernel, we can introduce a new stage = `vnet_shutdown` that clean up network resources. When jail / prison is going to dye, after kernel has cleaned up = processes it call `vnet_shutdown` to cleanup network resources, then vnet_destroy will go smoothly as there's no circular network = resource dependency right now. The life cycle of prison becomes: on jail create: PRISON_STATE_INVALID -> create VNET -> = PRISON_STATE_ALIVE -> setup network resources, ifnet, if addresses, = routing, etc. -> create / attach (network) processes=20 on jail destroy: jexec kill processes (1) by user -> mark it as = PRISON_STATE_DYING -> send SIGKILL to processes by kernel (2) -> = vnet_shutdown cleanup network resources -> destroy VNET (if prison = pr_ref go to the last one) -> DYED This idea is still unmature and I hope to hear more voices about it. Thanks! Best regards, Zhenlei From nobody Sun Dec 18 16:20:45 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NZp3l0TMHz1GLwx for ; Sun, 18 Dec 2022 16:20:51 +0000 (UTC) (envelope-from bz@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NZp3l000Lz46rV; Sun, 18 Dec 2022 16:20:50 +0000 (UTC) (envelope-from bz@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671380451; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=1bXBignYNKw8DAFiXeUPG8NPWkfvGbBOpDeHmcvv8c8=; b=LJhe5sjKdo8foU//L5us/2R7j3BNIcbcb2nDj9rUH6yinBc9NJwSPckRlRD4bvmgDYjxCN rbEkGkMAoa6YgJc6z6XifwwgVXyWNXPm2h6fpvuJcQad3rqsRzzEa/tRW7iOPwSlATxtsN ZuJlnLzHK7FT6u/KTh2KfC1gimljtSKxmcp+WBPJoxU3y4sy0J6CKJmtDbfQH2p+ATfE95 mRCGk/7kGMT3KvR6+Czcx1EuX12LGRI/riuztcGH6BK+ORYYLuusY/IEcVbzuvd19Ej0Sw l3EgshG9+Wa3n/8GitO+RLB26JRdYC4/Wosmthr8p952xXLgsB0x9pHn3wrDpw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671380451; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=1bXBignYNKw8DAFiXeUPG8NPWkfvGbBOpDeHmcvv8c8=; b=rQhuOQ8+dH86ylwTw7v5VQ25dkpqFwDVWCgL2/r0DMTHmej5HkgNbfAdVDDIx5TXFANN9M vRBJgV/pQ5K1QZpe+xvDxeB06a/w2FdcOsKoIwTOKlZV7wG2a+kWKaHbtEPbmEI/Of7m01 3ced7pQeVk2CqV7IqQ6McoFa3ajTvQ5GVL7g9knc7KQfobs9v5wLRzaQAH8lK0edI1mPLA 7JKhHT0PfMrM6lNNvWN6IFLp1M/3JhMey7OjcYQKMRx79lTYjVfFmHMQKa0CfapKeW3g78 w988wjnkpi8I6nGsnHrI7EIgLY3M3PIX3foWOBtHJXEwVDbT922Jw27IyiMEiA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1671380451; a=rsa-sha256; cv=none; b=KhseE7kq/Rkjkw2+cdof8hR4GDDuJmbTkIWTqcwZjsXPgyMrE76+dGE7xL2KQoQuUD4EeR Zb8fiJrGuCH+Q/MnDymn+xNXS9yvBposAtYjSkIBZ7B4WrAhjVCVS4rUGZ4+2rlBhrqbwB wIxwtjPBbEj0ZgH0u49Y6VC0FJM7tY1+GX/Wb3++TjIbfwxqzL0JNDtqN4SIdgcKZEGEoC Q4hBk9NhpBxIniV3BwZ0CR2iwFZZoTIWigpqxpLEAK6CThJm/cBx+nq72iCZZopM8rxzKq wsQDIaL7HHYJflJ4H/gy5sAAUrv1RaNLiWhV9zrRsVmRvuCw93BIG7UwcHhKpw== Received: from mx1.sbone.de (mx1.sbone.de [IPv6:2a01:4f8:13b:39f::9f:25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) (Authenticated sender: bz/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4NZp3k4p2gzHs2; Sun, 18 Dec 2022 16:20:50 +0000 (UTC) (envelope-from bz@freebsd.org) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id 431888D4A228; Sun, 18 Dec 2022 16:20:49 +0000 (UTC) Received: from content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id CEC655C3A833; Sun, 18 Dec 2022 16:20:48 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) by content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (amavisd-new, port 10024) with ESMTP id sDqeDe9saDb2; Sun, 18 Dec 2022 16:20:46 +0000 (UTC) Received: from strong-iwl0.sbone.de (strong-iwl0.sbone.de [IPv6:fde9:577b:c1a9:4902:b66b:fcff:fef3:e3d2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id 390CD5C3A830; Sun, 18 Dec 2022 16:20:46 +0000 (UTC) Date: Sun, 18 Dec 2022 16:20:45 +0000 (UTC) From: "Bjoern A. Zeeb" To: Zhenlei Huang cc: Gleb Smirnoff , "freebsd-jail@freebsd.org" Subject: Re: What's going on with vnets and epairs w/ addresses? In-Reply-To: <6B201617-68BC-4CC8-A2AE-908E96D69B67@FreeBSD.org> Message-ID: <4r8p3sn4-7no8-n2p2-9r16-n8sq3qs4p528@serrofq.bet> References: <5r22os7n-ro15-27q-r356-rps331o06so5@mnoonqbm.arg> <150A60D6-6757-46DD-988F-05A9FFA36821@FreeBSD.org> <9p9919q1-n639-p581-6q1o-so48o5ns6717@serrofq.bet> <6B201617-68BC-4CC8-A2AE-908E96D69B67@FreeBSD.org> X-OpenPGP-Key-Id: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-ThisMailContainsUnwantedMimeParts: N On Sun, 18 Dec 2022, Zhenlei Huang wrote: > >> On Dec 18, 2022, at 3:23 AM, Bjoern A. Zeeb wrote: >> >> On Sat, 17 Dec 2022, Gleb Smirnoff wrote: >> >>> Zhenlei, >>> >>> On Fri, Dec 16, 2022 at 06:30:57PM +0800, Zhenlei Huang wrote: >>> Z> I managed to repeat this issue on CURRENT/14 with this small snip: >>> Z> >>> Z> ------------------------------------------- >>> Z> #!/bin/sh >>> Z> >>> Z> # test jail name >>> Z> n="test_ref_leak" >>> Z> >>> Z> jail -c name=$n path=/ vnet persist >>> Z> # The following line trigger jail pr_ref leak >>> Z> jexec $n ifconfig lo0 inet 127.0.0.1/8 >>> Z> >>> Z> jail -R $n >>> Z> >>> Z> # wait a moment >>> Z> sleep 1 >>> Z> >>> Z> jls -j $n >>> Z> >>> Z> After DDB debugging and tracing , it seems that is triggered by a combine of [1] and [2] >>> Z> >>> Z> [1] https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915 >>> Z> [2] https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b > > I can confirm [2] also affects Non-VNET jails. > Prison pr_ref leak cause jail stuck in dying state. Usually a TCP connection in TW would do this in the old days and things would solve themselves after a while. This was always the case even long before vnet or multi-IP jails. >>> Z> >>> Z> >>> Z> In [1] the per-VNET uma zone is shared with the global one. >>> Z> `pcbinfo->ipi_zone = pcbstor->ips_zone;` >>> Z> >>> Z> In [2] unref `inp->inp_cred` is deferred called in inpcb_dtor() by uma_zfree_smr() . >>> Z> >>> Z> Unfortunately inps freed by uma_zfree_smr() are cached and inpcb_dtor() is not called immediately , >>> Z> thus leaking `inp->inp_cred` ref and hence `prison->pr_ref`. >>> Z> >>> Z> And it is also not possible to free up the cache by per-VNET SYSUNINIT tcp_destroy / udp_destroy / rip_destroy. >>> >>> This is known issue and I'd prefer not to call it a problem. The "leak" of a jail >>> happens only if machine is idle wrt the networking activity. >>> >>> Getting back to the problem that started this thread - the epair(4)s not immediately >>> popping back to prison0. IMHO, the problem again lies in the design of if_vmove and >>> epair(4) in particular. The if_vmove shall not exist, instead we should do a full >>> if_attach() and if_detach(). The state of an ifnet when it undergoes if_vmove doesn't >>> carry any useful information. With Alexander melifaro@ we discussed better options >>> for creating or attaching interfaces to jails that if_vmove. Until they are ready >>> the most easy workaround to deal with annoying epair(4) come back problem is to >>> remove it manually before destroying a jail, like I did in 80fc25025ff. >> >> Ok, move an em0 or cxl0 into the jail; the problem will be the same I >> bet and you need the physical interface to not disappear as then you >> cannot re-create a new jail with it. > > Re-read sys/kern/kern_jail.c, if pr_ref leaks, vnet_destroy() has no chance to be called, thus > if_vmove is not called and epair(4)s or em0, exl0 are not returned to home vnet. > > That can be confirmed by setting debug point on vnet_destroy by DDB, and then create and destroy vnet jails. > > So before the problem prison pr_ref count leaks is resolved, it will cover other potential problems such as @glebius > pointed out. > > I think the problem that prison ref count leaks should be resolved first. > > I'm also reviewing the life cycles of prison / vnet and it seems they could still be improved. But that's the not the problem here as your own test case pointed out. The point is that if you start a plain vnet jail put an interface in and destroy the jail that works instantly. The moment you put an address on any interface (incl. loopback as your test showed, which will not do ARP/NDP things compared to an ethernet interface) the jail will no longer die immediately. Simply putting an address on an interface should not defer things. So indeed something holds onto things there and is not cleaned up anymore. Finding that "something" is the important bit and being able to clean it up. I always say, if you have a machine in shutdown -r you don't want it hanging for hours either (now if you toggle the power switch you can do a lot more without panicing the rest of the system but with jails we cannot do that). And we did have vnet jails shutting down preoperly and clearing up for years. People had spent a lot of time on that. So it is possible and we need to get back to that state. /bz -- Bjoern A. Zeeb r15:7 From nobody Sun Dec 18 16:52:58 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NZpn24ZP0z1GQLl for ; Sun, 18 Dec 2022 16:53:10 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NZpn22FBDz4CMs; Sun, 18 Dec 2022 16:53:10 +0000 (UTC) (envelope-from kevans@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671382390; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=+cn0thUASzpf2IzuuHtGIFqMyI8S51i7W6WMfVayoiI=; b=K0nHrm1YtEUtGLTpAXzc5TVkxrVNvoqf6B3idAE9LH00TlajNTMIf5OhJZN1P7L4yZHxVm 3A4YSvDTiRl+ffkEZU7l5INbs74hNvx0s+TPK7QI2k0ZWtTeHb+zFJzXy+svqkQIjaiA1+ U0tYupia+H90wRA6OrdP4P/TMiMWMJk76c9XaZTnSkF4jz8tBMNPcREEAnFvNY1QblDNFc ADMLg19i23Y2cNg3o5Aku4yxrspFN3dMTH7bMtpLoxMXiKw60ikUQq9D+f6TcOb3lj+QiB 10Be8OsIIv48UOiG+aKiSpW1KuamTP9+E/60AEY5/4QLrRmwlmUYpyoKwBiqAw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671382390; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=+cn0thUASzpf2IzuuHtGIFqMyI8S51i7W6WMfVayoiI=; b=nfUuMVZY/TiZMxy3Cv8Vd/96BV7sh41FSa1656bHrIDoTi060NWC9o38wmmn0zDtRhnQYW Lm8DbTJQC+waiAblYm1/R1zGw+HfzkCXvAfTf/BI+YFwJYhHUsT7OvLsrWicDLOtOwRE+l CqPJYifPpM9pzzA7VhdUdJ7ftnuxQVIiuGwBt24f7mdDOi6pUdj34Nr9Cwl54MGwR96QJj L5cvgeH0BFpMGaVVMIq549rQe0quQdu4itNiy84P4ZrBFaTgVxwVuF0xEO5Gddj4T7Cora uI2jRSV4lHq/q5i2YhKQFaZyJ30o8sDKzFVYWvVuJLuNO1kAScpUUO0T+zTR4Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1671382390; a=rsa-sha256; cv=none; b=Di4c4IQfohirFgYzQvWNfjKmFziKZe+D7a+5YPZocrsX9jhjMHfMtEFMlFKUxw3MmHGLX+ V3ZbP1bY3ICFRozumEnhpQyqrBgXDgyVbDU9WsNhDSPxc5Qf+iJteDzUQ/Ex1QZ/dbnU5w bHsD8/gm0TNh6WfFzgC/UY6uXO9MdT+sKPk8URTao6fPrlZNGwGW31jmB/2Ri3gXyZerts qcvpHefaDMHgGVwzpjMGB3pATm+NrzcKejydo9l78tiM9Tv00diN/fdf9YqwLVr1RwTTTH x8wrjOueWFohG2vnSwoz3i8I1wS892D2/GYdlbrSrrUEo4RJl3lK6Bl9Ra1q/g== Received: from mail-qk1-f181.google.com (mail-qk1-f181.google.com [209.85.222.181]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) (Authenticated sender: kevans) by smtp.freebsd.org (Postfix) with ESMTPSA id 4NZpn21BybzJtM; Sun, 18 Dec 2022 16:53:10 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: by mail-qk1-f181.google.com with SMTP id h8so2841487qkk.8; Sun, 18 Dec 2022 08:53:10 -0800 (PST) X-Gm-Message-State: ANoB5pmXCPIFvwAwPNocyLJ/2WIYEb1/RND3/0ORa0ZWjkjP87JghrWc KPc0C2G1+O3teD6jq6jZD9kD90SsKCR4SHHYqLI= X-Google-Smtp-Source: AA0mqf4KJ9MLDkSudKNx1hP6ANkk2uqPqB4dq0IuwFVysDNpxpeMWrrbYpd8noBS9X0tq62W0dx0eUIZ4SclAxHcdSA= X-Received: by 2002:a05:620a:2f8:b0:6ff:8af3:c97f with SMTP id a24-20020a05620a02f800b006ff8af3c97fmr1540351qko.425.1671382389650; Sun, 18 Dec 2022 08:53:09 -0800 (PST) List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 References: <5r22os7n-ro15-27q-r356-rps331o06so5@mnoonqbm.arg> <150A60D6-6757-46DD-988F-05A9FFA36821@FreeBSD.org> In-Reply-To: From: Kyle Evans Date: Sun, 18 Dec 2022 10:52:58 -0600 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: What's going on with vnets and epairs w/ addresses? To: Gleb Smirnoff Cc: Zhenlei Huang , "Bjoern A. Zeeb" , "freebsd-jail@freebsd.org" , Mark Johnston Content-Type: text/plain; charset="UTF-8" X-ThisMailContainsUnwantedMimeParts: N On Sat, Dec 17, 2022 at 11:22 AM Gleb Smirnoff wrote: > > Zhenlei, > > On Fri, Dec 16, 2022 at 06:30:57PM +0800, Zhenlei Huang wrote: > Z> I managed to repeat this issue on CURRENT/14 with this small snip: > Z> > Z> ------------------------------------------- > Z> #!/bin/sh > Z> > Z> # test jail name > Z> n="test_ref_leak" > Z> > Z> jail -c name=$n path=/ vnet persist > Z> # The following line trigger jail pr_ref leak > Z> jexec $n ifconfig lo0 inet 127.0.0.1/8 > Z> > Z> jail -R $n > Z> > Z> # wait a moment > Z> sleep 1 > Z> > Z> jls -j $n > Z> > Z> After DDB debugging and tracing , it seems that is triggered by a combine of [1] and [2] > Z> > Z> [1] https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915 > Z> [2] https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b > Z> > Z> > Z> In [1] the per-VNET uma zone is shared with the global one. > Z> `pcbinfo->ipi_zone = pcbstor->ips_zone;` > Z> > Z> In [2] unref `inp->inp_cred` is deferred called in inpcb_dtor() by uma_zfree_smr() . > Z> > Z> Unfortunately inps freed by uma_zfree_smr() are cached and inpcb_dtor() is not called immediately , > Z> thus leaking `inp->inp_cred` ref and hence `prison->pr_ref`. > Z> > Z> And it is also not possible to free up the cache by per-VNET SYSUNINIT tcp_destroy / udp_destroy / rip_destroy. > > This is known issue and I'd prefer not to call it a problem. The "leak" of a jail > happens only if machine is idle wrt the networking activity. > > Getting back to the problem that started this thread - the epair(4)s not immediately > popping back to prison0. IMHO, the problem again lies in the design of if_vmove and > epair(4) in particular. The if_vmove shall not exist, instead we should do a full > if_attach() and if_detach(). The state of an ifnet when it undergoes if_vmove doesn't > carry any useful information. With Alexander melifaro@ we discussed better options > for creating or attaching interfaces to jails that if_vmove. Until they are ready > the most easy workaround to deal with annoying epair(4) come back problem is to > remove it manually before destroying a jail, like I did in 80fc25025ff. > It still behaved much better prior to eb93b99d6986, which you and Mark were going to work on a solution for to allow the cred "leak" to close up much more quickly. CC markj@, since I think it's been six months since the last time I inquired about it, making this a good time to do it again... Thanks, Kyle Evans From nobody Sun Dec 18 17:44:14 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NZqw70dTBz1GmWT; Sun, 18 Dec 2022 17:44:23 +0000 (UTC) (envelope-from jamie@freebsd.org) Received: from gritton.org (gritton.org [162.220.209.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "gritton.org", Issuer "gritton.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NZqw60mhmz4Jqr; Sun, 18 Dec 2022 17:44:22 +0000 (UTC) (envelope-from jamie@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=softfail (mx1.freebsd.org: 162.220.209.3 is neither permitted nor denied by domain of jamie@freebsd.org) smtp.mailfrom=jamie@freebsd.org; dmarc=none Received: from gritton.org ([127.0.0.3]) (authenticated bits=0) by gritton.org (8.16.1/8.16.1) with ESMTPA id 2BIHiEVV000024; Sun, 18 Dec 2022 09:44:14 -0800 (PST) (envelope-from jamie@freebsd.org) List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Date: Sun, 18 Dec 2022 09:44:14 -0800 From: James Gritton To: freebsd-jail@freebsd.org, freebsd-net Cc: Zhenlei Huang Subject: Re: Propose a new stage `vnet_shutdown` before `vnet_destroy` In-Reply-To: References: User-Agent: Roundcube Webmail/1.4.11 Message-ID: <1c9dbf6d26b9525243dd6b3ffafa23cb@freebsd.org> X-Sender: jamie@freebsd.org Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-3.10 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org,freebsd-jail@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_CC(0.00)[gmail.com]; ASN(0.00)[asn:30247, ipnet:162.220.208.0/22, country:US]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; DMARC_NA(0.00)[freebsd.org]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[jamie]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all:c]; RCPT_COUNT_THREE(0.00)[3]; MID_RHS_MATCH_FROM(0.00)[]; TO_DN_SOME(0.00)[]; TAGGED_RCPT(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4NZqw60mhmz4Jqr X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On 2022-12-18 00:01, Zhenlei Huang wrote: > I'm currently working on route nexthop caching feature for tunneling > interfaces such as > if_gif, if_gre, if_vxlan, and potentially if_wg. I encounter a nasty > bug related to VNET lifecycle. > More preciously I'd like to call `rib_unsubscribe()` to unsubscribe > route event when the interface > tunnel is deleted (gif_delete_tunnel). > > While on VNET shutting down, VNET SYSUNINIT was called and the routing > vnet subsystem > is destroyed before the interface going down and hence cause > pagefault. I do not want to check > `vnet.vnet_shutdown` state as it looks messed up. > > I'm recently reviewing the life cycles of prison and get some > inspirations. > > When the jail / prison is submitted to destroy ( by jail_remove > syscall ) then SIGKILL is sent to > the prison's processes. I think it is correct order to destroy jail / > prison. To summarize, the life cycle > of jail / prison is: > > on jail create: PRISON_STATE_INVALID -> create VNET -> > PRISON_STATE_ALIVE -> setup network resources, ifnet, if addresses, > routing, etc. -> create / attach (network) processes > on jail destroy: jexec kill processes (1) by user -> mark it as > PRISON_STATE_DYING -> send SIGKILL to processes by kernel (2) -> > destroy VNET (if prison pr_ref go to the last one) -> DYED > > The (2) is a cleanup by kernel as (1) is possible not done by user. > > > So it comes the idea about the life cycle of VNET. > > While on jail destroy, the network resources are cleaned up by > vnet_destroy ( SYSUNINIT ). Then the > order of SYSUNINIT of network components is hacking as circular > network resource dependency is possible. > For example the routing table entries (nhop) have reference of ifnet, > and ifnet have reference to route nhop (cache), as > I encountered. > > Just like the cleanup processes by kernel, we can introduce a new > stage `vnet_shutdown` that clean up network resources. > When jail / prison is going to dye, after kernel has cleaned up > processes it call `vnet_shutdown` to cleanup network resources, > then vnet_destroy will go smoothly as there's no circular network > resource dependency right now. > > The life cycle of prison becomes: > > on jail create: PRISON_STATE_INVALID -> create VNET -> > PRISON_STATE_ALIVE -> setup network resources, ifnet, if addresses, > routing, etc. -> create / attach (network) processes > on jail destroy: jexec kill processes (1) by user -> mark it as > PRISON_STATE_DYING -> send SIGKILL to processes by kernel (2) -> > vnet_shutdown cleanup network resources -> destroy VNET (if prison > pr_ref go to the last one) -> DYED > > This idea is still unmature and I hope to hear more voices about it. This is absolutely the direction things need to go. Vnet isn't the only thing that can have these problems, though it's been the biggest offender. There could also be cycles that involve more than one subsystem, which could be helped by broad application of this idea. There's a function in kern_jail.c ready for this: prison_cleanup. It's called in "mark PRISON_STATE_DYING" stage of things. That's before the "send SIGKILL" part of your sequence, but otherwise fits. - Jamie From nobody Tue Dec 20 16:12:24 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Nc1n92ZLPz1G2DY for ; Tue, 20 Dec 2022 16:12:29 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qt1-x832.google.com (mail-qt1-x832.google.com [IPv6:2607:f8b0:4864:20::832]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Nc1n85wXYz3Jc4; Tue, 20 Dec 2022 16:12:28 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=puvB0DZU; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::832 as permitted sender) smtp.mailfrom=markjdb@gmail.com; dmarc=none Received: by mail-qt1-x832.google.com with SMTP id fu10so11346369qtb.0; Tue, 20 Dec 2022 08:12:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:from:to:cc:subject:date:message-id :reply-to; bh=cE/1SQjIO35aNrWMhZ6Kvh2hFmPYyUVj0GpkvFjNCOE=; b=puvB0DZUAxaDLxSEYuw++9XATVrzQ4IAVZYEQwYXRYBnkj5lspiYhaQ2+chZyqtj6q QgwUWMpsSIOC72SCzEDIzqmzGeWtD//8bZqhfmGHlNctPCGzIq0OfuiGqmkAuB4wx1pR llF94tTM1p+V1QTvmeCGJonpat8MbK0ilXzOAuY1rTGZFCqIHkedT0UDWzkva+GXKg+q McrlMUax02vmnLseIW8aRvpJscyMkux74xpJ7hctTxOdvbDUfCqLvUxK4jX1WsXDkmNw 0SmF6QcWv80W+w3lX0GT/ZpwTmNNrTlevNb2f5ER1zkEh4+92WWujTs3lv/j8DLKhwT6 GYTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cE/1SQjIO35aNrWMhZ6Kvh2hFmPYyUVj0GpkvFjNCOE=; b=zH49tPBbGVdC7flD2ILfRV3YGENz06kwuUf0peCs9H/0+TrxgVOmZ66nFewDLpX3Yb cr4DIDnkLJ0JBfPYOM2poNI4J5yCGhf0OtAYiEQUxqVV291w4DWXoNN87xPrOyZNGRir uOriVitnrY4BHFVzIJqC/WvQPSVz4CBOgpPdC+awbHuT3I39N2AKY7PvQZc6lc9b94WU OjmJkgwTz+C7s1G2FT+s0Z+oEg8zMm5DPBI0MUVh4/lMizitAWzgybcP81Srg3yCs2NW bWWrHXn9aim0IgBZ9DKX2EKmz1WPP8nKeJa4a0PHIXmFFMmXw+BsTLdPtI+Py5Vummcu GBGw== X-Gm-Message-State: ANoB5pnnmVQwOeJ6BwXmGt4MpifKObnjv37yBMfxB02tHImyAk8w534l 5RtReIW65dozc3QaIlWrdnCE2DlW77w= X-Google-Smtp-Source: AA0mqf7BKxJeU2+fQt8tvKw234Wuw0zARcAPOigNAPH8+2Td64C6/2b02EhD1EQDdXdomivskHYuKg== X-Received: by 2002:ac8:647:0:b0:3a5:f9cb:8852 with SMTP id e7-20020ac80647000000b003a5f9cb8852mr52714387qth.28.1671552747803; Tue, 20 Dec 2022 08:12:27 -0800 (PST) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id x10-20020ac8730a000000b00398a7c860c2sm7843150qto.4.2022.12.20.08.12.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Dec 2022 08:12:26 -0800 (PST) Date: Tue, 20 Dec 2022 11:12:24 -0500 From: Mark Johnston To: Kyle Evans Cc: Gleb Smirnoff , Zhenlei Huang , "Bjoern A. Zeeb" , "freebsd-jail@freebsd.org" Subject: Re: What's going on with vnets and epairs w/ addresses? Message-ID: References: <5r22os7n-ro15-27q-r356-rps331o06so5@mnoonqbm.arg> <150A60D6-6757-46DD-988F-05A9FFA36821@FreeBSD.org> List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spamd-Result: default: False [-1.42 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.72)[-0.725]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; MLMMJ_DEST(0.00)[freebsd-jail@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::832:from]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; DMARC_NA(0.00)[freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCPT_COUNT_FIVE(0.00)[5]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MIME_TRACE(0.00)[0:+]; FREEMAIL_CC(0.00)[freebsd.org,gmail.com] X-Rspamd-Queue-Id: 4Nc1n85wXYz3Jc4 X-Spamd-Bar: - X-ThisMailContainsUnwantedMimeParts: N On Sun, Dec 18, 2022 at 10:52:58AM -0600, Kyle Evans wrote: > On Sat, Dec 17, 2022 at 11:22 AM Gleb Smirnoff wrote: > > > > Zhenlei, > > > > On Fri, Dec 16, 2022 at 06:30:57PM +0800, Zhenlei Huang wrote: > > Z> I managed to repeat this issue on CURRENT/14 with this small snip: > > Z> > > Z> ------------------------------------------- > > Z> #!/bin/sh > > Z> > > Z> # test jail name > > Z> n="test_ref_leak" > > Z> > > Z> jail -c name=$n path=/ vnet persist > > Z> # The following line trigger jail pr_ref leak > > Z> jexec $n ifconfig lo0 inet 127.0.0.1/8 > > Z> > > Z> jail -R $n > > Z> > > Z> # wait a moment > > Z> sleep 1 > > Z> > > Z> jls -j $n > > Z> > > Z> After DDB debugging and tracing , it seems that is triggered by a combine of [1] and [2] > > Z> > > Z> [1] https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915 > > Z> [2] https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b > > Z> > > Z> > > Z> In [1] the per-VNET uma zone is shared with the global one. > > Z> `pcbinfo->ipi_zone = pcbstor->ips_zone;` > > Z> > > Z> In [2] unref `inp->inp_cred` is deferred called in inpcb_dtor() by uma_zfree_smr() . > > Z> > > Z> Unfortunately inps freed by uma_zfree_smr() are cached and inpcb_dtor() is not called immediately , > > Z> thus leaking `inp->inp_cred` ref and hence `prison->pr_ref`. > > Z> > > Z> And it is also not possible to free up the cache by per-VNET SYSUNINIT tcp_destroy / udp_destroy / rip_destroy. > > > > This is known issue and I'd prefer not to call it a problem. The "leak" of a jail > > happens only if machine is idle wrt the networking activity. > > > > Getting back to the problem that started this thread - the epair(4)s not immediately > > popping back to prison0. IMHO, the problem again lies in the design of if_vmove and > > epair(4) in particular. The if_vmove shall not exist, instead we should do a full > > if_attach() and if_detach(). The state of an ifnet when it undergoes if_vmove doesn't > > carry any useful information. With Alexander melifaro@ we discussed better options > > for creating or attaching interfaces to jails that if_vmove. Until they are ready > > the most easy workaround to deal with annoying epair(4) come back problem is to > > remove it manually before destroying a jail, like I did in 80fc25025ff. > > > > It still behaved much better prior to eb93b99d6986, which you and Mark > were going to work on a solution for to allow the cred "leak" to close > up much more quickly. CC markj@, since I think it's been six months > since the last time I inquired about it, making this a good time to do > it again... I spent some time trying to see if we could fix this in UMA/SMR and talked to Jeff about it a bit. At this point I don't think it's the right approach, at least for now. Really we have a composability problem where different layers are using different techniques to signal that they're done with a particular piece of memory, and they just aren't compatible. One thing I tried is to implement a UMA function which walks over all SMR zones and synchronizes all cached items (so that their destructors are called). This is really expensive, at minimum it has to bind to all CPUs in the system so that it can flush per-CPU buckets. If jail_deref() calls that function, the bug goes away at least in my limited testing, but its use is really a layering violation. We could, say, periodically scan cached UMA/SMR items and invoke their destructors, but for most SMR consumers this is unnecessary, and again there's a layering problem: the inpcb layer shouldn't "know" that it has to do that for its zones, since it's the jail layer that actually cares. It also seems kind of strange that dying jails still occupy a slot in the jail namespace. I don't really understand why the existence of a dying jail prevents creation of a new jail with the same name, but presumably there's a good reason for it? Now my inclination is to try and fix this in the inpcb layer, by not accessing the inp_cred at all in the lookup path until we hold the inpcb lock, and then releasing the cred ref before freeing a PCB to its zone. I think this is doable based on a few observations: - When doing an SMR-protected lookup, we always lock the returned inpcb before handing it to the caller. So we could in principle perform inp_cred checking after acquiring the lock but before returning. - If there are no jailed PCBs in a hash chain in_pcblookup_hash_locked() always scans the whole chain. - If we match only one PCB in a lookup, we can probably(?) return that PCB without dereferencing the cred pointer at all. If not, then the scan only has to keep track of a fixed number of PCBs before picking which one to return. So it looks like we can perform a lockless scan and keep track of matches on the stack, then lock the matched PCBs and perform prison checks if necessary, without making the common case more expensive. In fact there is a parallel thread on freebsd-jail which reports that this inp_cred access is a source of frequent cache misses. I was surprised to see that the scan calls prison_flag() before even checking the PCB's local address. So if the hash chain is large then we're potentially performing a lot of unnecessary memory accesses (though presumably it's common for most of the PCBs to be sharing a single cred?). In particular we can perhaps solve two problems at once. Any thoughts? Are there some fundamental reasons this can't work? From nobody Tue Dec 20 17:37:56 2022 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Nc3gs4dWTz1GDcv for ; Tue, 20 Dec 2022 17:38:01 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qt1-x831.google.com (mail-qt1-x831.google.com [IPv6:2607:f8b0:4864:20::831]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Nc3gr5dWnz3p5s; Tue, 20 Dec 2022 17:38:00 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=KdKJVeGl; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::831 as permitted sender) smtp.mailfrom=markjdb@gmail.com; dmarc=none Received: by mail-qt1-x831.google.com with SMTP id fu10so11571343qtb.0; Tue, 20 Dec 2022 09:38:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:from:to:cc:subject:date:message-id :reply-to; bh=sIgTR/IGraVNv51YA0ZZnXx/U6c/uXybsm3pdc6zCQo=; b=KdKJVeGl0kULFu04MqglcLkeJ0GG0Qs0P43uGM4PNVQs3EDjt9YMbbuj4pulxJ4TYj itwnotKYvlgmfu+27X/K/urJIaAenbqzFhc+I8Z372XWL7PsDWp+bxOPJZ2dOPWFknIB QEy+BXQ8k9hv72Ij9GgeEvRBYaWqVQRrMdiyU5nqHo+GlxMwRhHx7UxdivtCbVy8Rgcf hZ7IZKWsy3Prkj/u6ufdsQVevG4VJXJSjlBi680jjHLPge2g7N71GbvAObgaO71Vxr3N RvwTl8OCk92/vHKEm/tPqQVQ/UFniNaTnFFUsJl/I6a0lBLGLGBJHAOyV8324UTopkoG axQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sIgTR/IGraVNv51YA0ZZnXx/U6c/uXybsm3pdc6zCQo=; b=Sp7/INaqf1PIdMvJw6hIgLtbs3ca9NiN5BQtUfNtCWPoG2e/5tufYUoPzwi6FZqYkc CJ2DoYPV1PORBH75wMzbHzk0VBihqNsA/cmFbIRzHOQTSdmiWOvtXnL74sYXA7m3XgoL H2kmVGb9RjnWbNSVJO0XHNkW48ylndbhftMRx2AmJeYDFnEpEqxXCfMfC6K8cUmuum16 JpZfRttyl3oo/EqpwjLHv4ipgM10Cym9uq7L6MJR/fOZjNqu3SK1MSkgnKSb6BRzNsdn VQN/DYOlVm3nOcJSJjlBQvR505ez56wcYScbO+7vha8AxV96A477untNqH9yb1UGUQku 5XHw== X-Gm-Message-State: AFqh2kpQheRIhcwMQRM2lRR9Gc1hV6PszfBT1dtHWqWGeoImkuDmRhAE qQJTVDTpuTEIQWHIe0tXJH3fKZP0wSQ= X-Google-Smtp-Source: AMrXdXu70PvFkJLI4hTgUXNcRHUirzrjFEJN6JKG6WgoR92TJYsOoxMJt+TTDftoCqDkOw6dcVTJnQ== X-Received: by 2002:ac8:668d:0:b0:3a9:1ada:930f with SMTP id d13-20020ac8668d000000b003a91ada930fmr28752951qtp.23.1671557879641; Tue, 20 Dec 2022 09:37:59 -0800 (PST) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id r18-20020a05620a299200b006eeb3165565sm9411823qkp.80.2022.12.20.09.37.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Dec 2022 09:37:58 -0800 (PST) Date: Tue, 20 Dec 2022 12:37:56 -0500 From: Mark Johnston To: "Bjoern A. Zeeb" Cc: Andrew Gallatin , "pjd@FreeBSD.org" , James Gritton , jail@freebsd.org, "glebius@FreeBSD.org" Subject: Re: prison_flag() check in hot path of in_pcblookup() Message-ID: References: <6on81os3-501-s5n2-8nos-p85n8op23232@serrofq.bet> <6r10qop4-7p83-qs6s-q3r0-64756n243rp5@serrofq.bet> List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <6r10qop4-7p83-qs6s-q3r0-64756n243rp5@serrofq.bet> X-Spamd-Result: default: False [-2.66 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.96)[-0.961]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[freebsd.org]; MLMMJ_DEST(0.00)[jail@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::831:from]; ARC_NA(0.00)[]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCPT_COUNT_FIVE(0.00)[6]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org] X-Rspamd-Queue-Id: 4Nc3gr5dWnz3p5s X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N On Tue, Dec 13, 2022 at 11:54:17PM +0000, Bjoern A. Zeeb wrote: > On Tue, 13 Dec 2022, Andrew Gallatin wrote: > > > [ I added pjd, since the original patch came from him ] > > > > Just to make sure I understand, I have a simple yes/no question: > > > > Can jails and the host ever share the same (local) port and the same IP? > > Can they currently (I tested only for TCP)? > > - local binds can overlap like they can with just the base system. > so bind(... {AF_INET, laddr, lport} ... ) works fine (REUSEPORT). > > - tcp connect of a 2nd socket to the same {faddr, fport} from the above > bind will fail with 'Address already in use' [currently] > [I believe that would mean your patch could go in? Where does the error come from [%]?] [*] I presume that the patch just causes the first loop in in_pcblookup_hash_locked() to return immediately upon an exact match? I think this is only valid if in_pcblookup_hash_locked() can assume that faddr != INADDR_ANY. I'm pretty sure this is true but it's not entirely obvious to me. > - tcp listen will work on {laddr, lport} if run in paralllel (REUSEPORT) > or in base and jail at the same time. I wonder if we can improve wildcard matching by enforcing an invariant that jailed sockets always appear first in a hash chain? That would make hash insertion more expensive but that might be a reasonable tradeoff. > [%] likely in_pcbconnect_setup() ? Also one should check the other > order (jail first then base); also we assume no other race > conditions in this rather simple testing... From nobody Tue Dec 20 20:50:09 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Nc7xf6DxYz1GyGF for ; Tue, 20 Dec 2022 20:50:14 +0000 (UTC) (envelope-from bz@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Nc7xf4Swjz4Lbt; Tue, 20 Dec 2022 20:50:14 +0000 (UTC) (envelope-from bz@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671569414; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=q/Q/Vp7ojc0jFeICZS5HMz+J9jG6pn78B9j0KYAmZIo=; b=wqVyt5R2cXReagpVrZQb+gZ0iDu/TLssli0qWlZ+bdv0oJAePUjLkK0A8vR7oVl36kAGj/ 4aRmAJ9G9rT7FubG3x7GrAxZioxjZd3nStzsgbOBMXy0CY5VIu222kZJ9S+3L7GkEKtDlJ v1e2l+okNbNkZaleY1AQtTSHfqTTY4A8ID1KlYqN7ur5YO+LGC0WsaYsZFptXXutchLvdn Bl8XM0MxmTIW8VosebYhyVbeHwdKsDEUZHmzLF1ytw4QtN9Yi/UA/det5+fGiItRTDsZbN x43YtLLWqlJyNgrg7KdKNDWXKkdsEFpuvZS/hioNmr/d+uiaspXGZa03ZZ11+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671569414; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=q/Q/Vp7ojc0jFeICZS5HMz+J9jG6pn78B9j0KYAmZIo=; b=VFRZEHvSIVd0f8djJ2JsW0UoVEmBSQYxd7V0+JTqMZt30+1lYvDgIDXzhPr1Qn9pxks5kz 2NLBD6DNcp7CU5p16FYLNlZyTVBdBy6qASiJTgLQO9PtCbtZ7eaUY3e3XUcwES7Dg1vhs2 u/PFKXhcRrQ7UhsIl13HAz4Oi95ZPxjy8ZEW3crfq3eX6O8AlkLhNrVl2yiKiBVVeqC6N9 WHs7eS255MQk710DW8ExTHpxuIvRckoSTvDCk9HFWqOEtuXgmEUJDsEdXapSCarkaueHAr GhNXC5LkF6i0qZa8c0DnqwgJlhfD4FvesixfmVAJ2YnzbWKoIFE/a4+auJZQUw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1671569414; a=rsa-sha256; cv=none; b=txdgrRhEuSeudVrKrmoCsunqKi1N7Bj4Tdna9x2HhLqSxn/wN/NIgdnL9lIvPJvQtmCIBQ 1JOZvDeBKHbAOjdFSUJzYj3MVTgmq3uQSDLxq+suCsdD9ouxx1lKrU/RNMq1ncaqx/odLT 4m+lZ6C64LnWNMBBItI8n6k91G4f6J0QloWxJgrQUdqyFzijzkolSymDtlfJ0qRa4YBHNw cPoiVdAAFfYKj51En0Aix88Bg5nu1fpdURdfNCHRYIiY9GtHSfBlxqfrcT0jaQ1mBkx+pr APVV+KPjqN3oR/Uzxw7PwYjHUg2XP0rJ+0kISB18xy5aLwf0zWAfUUCwxFVwRw== Received: from mx1.sbone.de (cross.sbone.de [195.201.62.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) (Authenticated sender: bz/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Nc7xf1tpgzG79; Tue, 20 Dec 2022 20:50:14 +0000 (UTC) (envelope-from bz@freebsd.org) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id B90D98D4A228; Tue, 20 Dec 2022 20:50:12 +0000 (UTC) Received: from content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id E6E885C3A833; Tue, 20 Dec 2022 20:50:11 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) by content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (amavisd-new, port 10024) with ESMTP id euS0YaqQHCtP; Tue, 20 Dec 2022 20:50:10 +0000 (UTC) Received: from strong-iwl0.sbone.de (strong-iwl0.sbone.de [IPv6:fde9:577b:c1a9:4902:b66b:fcff:fef3:e3d2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id E98E75C3A830; Tue, 20 Dec 2022 20:50:09 +0000 (UTC) Date: Tue, 20 Dec 2022 20:50:09 +0000 (UTC) From: "Bjoern A. Zeeb" To: Mark Johnston cc: Kyle Evans , Gleb Smirnoff , Zhenlei Huang , "freebsd-jail@freebsd.org" Subject: Re: What's going on with vnets and epairs w/ addresses? In-Reply-To: Message-ID: References: <5r22os7n-ro15-27q-r356-rps331o06so5@mnoonqbm.arg> <150A60D6-6757-46DD-988F-05A9FFA36821@FreeBSD.org> X-OpenPGP-Key-Id: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-ThisMailContainsUnwantedMimeParts: N On Tue, 20 Dec 2022, Mark Johnston wrote: > On Sun, Dec 18, 2022 at 10:52:58AM -0600, Kyle Evans wrote: >> On Sat, Dec 17, 2022 at 11:22 AM Gleb Smirnoff wrote: >>> >>> Zhenlei, >>> >>> On Fri, Dec 16, 2022 at 06:30:57PM +0800, Zhenlei Huang wrote: >>> Z> I managed to repeat this issue on CURRENT/14 with this small snip: >>> Z> >>> Z> ------------------------------------------- >>> Z> #!/bin/sh >>> Z> >>> Z> # test jail name >>> Z> n="test_ref_leak" >>> Z> >>> Z> jail -c name=$n path=/ vnet persist >>> Z> # The following line trigger jail pr_ref leak >>> Z> jexec $n ifconfig lo0 inet 127.0.0.1/8 >>> Z> >>> Z> jail -R $n >>> Z> >>> Z> # wait a moment >>> Z> sleep 1 >>> Z> >>> Z> jls -j $n >>> Z> >>> Z> After DDB debugging and tracing , it seems that is triggered by a combine of [1] and [2] >>> Z> >>> Z> [1] https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915 >>> Z> [2] https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b >>> Z> >>> Z> >>> Z> In [1] the per-VNET uma zone is shared with the global one. >>> Z> `pcbinfo->ipi_zone = pcbstor->ips_zone;` >>> Z> >>> Z> In [2] unref `inp->inp_cred` is deferred called in inpcb_dtor() by uma_zfree_smr() . >>> Z> >>> Z> Unfortunately inps freed by uma_zfree_smr() are cached and inpcb_dtor() is not called immediately , >>> Z> thus leaking `inp->inp_cred` ref and hence `prison->pr_ref`. >>> Z> >>> Z> And it is also not possible to free up the cache by per-VNET SYSUNINIT tcp_destroy / udp_destroy / rip_destroy. >>> >>> This is known issue and I'd prefer not to call it a problem. The "leak" of a jail >>> happens only if machine is idle wrt the networking activity. >>> >>> Getting back to the problem that started this thread - the epair(4)s not immediately >>> popping back to prison0. IMHO, the problem again lies in the design of if_vmove and >>> epair(4) in particular. The if_vmove shall not exist, instead we should do a full >>> if_attach() and if_detach(). The state of an ifnet when it undergoes if_vmove doesn't >>> carry any useful information. With Alexander melifaro@ we discussed better options >>> for creating or attaching interfaces to jails that if_vmove. Until they are ready >>> the most easy workaround to deal with annoying epair(4) come back problem is to >>> remove it manually before destroying a jail, like I did in 80fc25025ff. >>> >> >> It still behaved much better prior to eb93b99d6986, which you and Mark >> were going to work on a solution for to allow the cred "leak" to close >> up much more quickly. CC markj@, since I think it's been six months >> since the last time I inquired about it, making this a good time to do >> it again... > > I spent some time trying to see if we could fix this in UMA/SMR and > talked to Jeff about it a bit. At this point I don't think it's the > right approach, at least for now. Really we have a composability > problem where different layers are using different techniques to signal > that they're done with a particular piece of memory, and they just > aren't compatible. > > One thing I tried is to implement a UMA function which walks over all > SMR zones and synchronizes all cached items (so that their destructors > are called). This is really expensive, at minimum it has to bind to all A semi-unrelated question -- do we have any documentation around SMR in the tree which is not in subr_smr.c? (I have to admit I find it highly confusing that the acronym is more easily found as "Shingled Magnetic Recording (SMR)" in a different header file). > CPUs in the system so that it can flush per-CPU buckets. If > jail_deref() calls that function, the bug goes away at least in my > limited testing, but its use is really a layering violation. > > We could, say, periodically scan cached UMA/SMR items and invoke their > destructors, but for most SMR consumers this is unnecessary, and again > there's a layering problem: the inpcb layer shouldn't "know" that it has > to do that for its zones, since it's the jail layer that actually cares. > > It also seems kind of strange that dying jails still occupy a slot in > the jail namespace. I don't really understand why the existence of a > dying jail prevents creation of a new jail with the same name, but > presumably there's a good reason for it? You can create a new jail but if you have (physical) resources tied to the old one which are not released, then you are stuck (physical network interfaces for example). > Now my inclination is to try and fix this in the inpcb layer, by not > accessing the inp_cred at all in the lookup path until we hold the inpcb > lock, and then releasing the cred ref before freeing a PCB to its zone. > I think this is doable based on a few observations: > - When doing an SMR-protected lookup, we always lock the returned inpcb > before handing it to the caller. So we could in principle perform > inp_cred checking after acquiring the lock but before returning. > - If there are no jailed PCBs in a hash chain in_pcblookup_hash_locked() > always scans the whole chain. > - If we match only one PCB in a lookup, we can probably(?) return that > PCB without dereferencing the cred pointer at all. If not, then the > scan only has to keep track of a fixed number of PCBs before picking > which one to return. So it looks like we can perform a lockless scan > and keep track of matches on the stack, then lock the matched PCBs and > perform prison checks if necessary, without making the common case > more expensive. > > In fact there is a parallel thread on freebsd-jail which reports that > this inp_cred access is a source of frequent cache misses. I was > surprised to see that the scan calls prison_flag() before even checking > the PCB's local address. So if the hash chain is large then we're > potentially performing a lot of unnecessary memory accesses (though > presumably it's common for most of the PCBs to be sharing a single > cred?). In particular we can perhaps solve two problems at once. I haven't heard back after I sent the test program there; I hope that can be solved independently first and any optimisations can then come. > Any thoughts? Are there some fundamental reasons this can't work? > -- Bjoern A. Zeeb r15:7 From nobody Thu Dec 22 15:38:07 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NdDzS0MBWz1G1jV for ; Thu, 22 Dec 2022 15:40:36 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NdDzS05ntz3QD0; Thu, 22 Dec 2022 15:40:36 +0000 (UTC) (envelope-from zlei@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671723636; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=j7P/IrGhElMTd4EWRaFuhm2MrMJHHcIYT8NCIc3T7ms=; b=D2SUmM9sf6R46Stgfi8CeoYdUVS+9+HV1Jf43aYZJRmfy8M6+MnD7GZDtdyXsctLnRZbqi 25tYSoDAGhYdybIhMUNRJsHItmnumF60WIdfQLu9CCsqC0yx/0lJBC/ZQiwHW4qAFfLNvS WK0eUVKtYJa/rlkTQcRt6V3CnofUeUFtLFWxZer4ZqJjT19hPjnbegDGy63Qb2v0VjFoSn C52umt5H5tUjgOor++eVMkD4vYhRLpblqOBBVR/XHddYU1WuYeeZcgp4D3aTXMsbwaTvAf 7sEZUCpR4YeisQZ4t/mmBEQqTDUfWt/UpsmaQHl6h4jcZJ3XMtvzv3uwhajWPA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671723636; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=j7P/IrGhElMTd4EWRaFuhm2MrMJHHcIYT8NCIc3T7ms=; b=SZAnRNHwRG+TxL5y8R6m9g8VQhUGjjwd4Hm5oM9UHPU4NgrOxVUes1cc8wGjUELUgrcjO7 FtKnxaQLFFF97nGWE055JnqFlK3pIGUavPQXFbxqhkDIikhZZM6xgleExqA5x9YR6IqedP 3i/jvJKycOiDsoat/RX68I3Pu0GW3dG9+o7zmy2QPuSExw6sDrYNkjcXfgYWmgNCMpV0nn Jbw9DiCV1Qc4enqWVsq2P7lq8mPk/LCww+zz+Xhx/BuWytt1DtzSRnmIAbBo0aLF9+4t++ +rsKrkiLZcBfva3hgLfxJOVMYYm3Zu6b4Mlw4/8hvPArRDvmr/SKOFHczfLvRg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1671723636; a=rsa-sha256; cv=none; b=l4vTmsatMCkhkJTUScGR1guOxYSWTE9ykblRVsdGQMiSskz2DHYXukFXExbp1Fj98p2d1e RnJ0J5A1RoQrvLWT8UOUedBsd7w2VOqjp0DqcBkxkUguZh8jcvcC2RgqXNTyossNzDucZH FA68HVAVJUmwJf4kqhDsqYmKvRPisSuLC60HSbe51UrRptyG+bNfKNFb1FQc0EEVLtZnIa jsAj6aQZxwmnm7+a2OQVAY3ZV6nJq86ucZu75adDygIDiXrZeEK+ENIjkv7lwofJywpt60 h/TeYUyuoIHEofsphzyYEaCiHTRfqyZJHeDdJ1misvIRILcC0/TzboMoBMCGeA== Received: from [IPv6:2409:8a5e:aa71:3294:6920:cc8d:9f5b:e8a4] (unknown [IPv6:2409:8a5e:aa71:3294:6920:cc8d:9f5b:e8a4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: zlei/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4NdDzN0ZX5z145c; Thu, 22 Dec 2022 15:40:31 +0000 (UTC) (envelope-from zlei@FreeBSD.org) From: Zhenlei Huang Message-Id: <069FE014-1D62-4479-B830-9FBDF36F2109@FreeBSD.org> Content-Type: multipart/alternative; boundary="Apple-Mail=_05999826-0A4B-4FD5-A20A-0ED8E067BC1C" List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.7\)) Subject: Re: What's going on with vnets and epairs w/ addresses? Date: Thu, 22 Dec 2022 23:38:07 +0800 In-Reply-To: Cc: Kyle Evans , Gleb Smirnoff , "Bjoern A. Zeeb" , "freebsd-jail@freebsd.org" To: Mark Johnston References: <5r22os7n-ro15-27q-r356-rps331o06so5@mnoonqbm.arg> <150A60D6-6757-46DD-988F-05A9FFA36821@FreeBSD.org> X-Mailer: Apple Mail (2.3608.120.23.2.7) X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_05999826-0A4B-4FD5-A20A-0ED8E067BC1C Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii >=20 > On Dec 21, 2022, at 12:12 AM, Mark Johnston > wrote: >=20 > On Sun, Dec 18, 2022 at 10:52:58AM -0600, Kyle Evans wrote: >> On Sat, Dec 17, 2022 at 11:22 AM Gleb Smirnoff > wrote: >>>=20 >>> Zhenlei, >>>=20 >>> On Fri, Dec 16, 2022 at 06:30:57PM +0800, Zhenlei Huang wrote: >>> Z> I managed to repeat this issue on CURRENT/14 with this small = snip: >>> Z> >>> Z> ------------------------------------------- >>> Z> #!/bin/sh >>> Z> >>> Z> # test jail name >>> Z> n=3D"test_ref_leak" >>> Z> >>> Z> jail -c name=3D$n path=3D/ vnet persist >>> Z> # The following line trigger jail pr_ref leak >>> Z> jexec $n ifconfig lo0 inet 127.0.0.1/8 >>> Z> >>> Z> jail -R $n >>> Z> >>> Z> # wait a moment >>> Z> sleep 1 >>> Z> >>> Z> jls -j $n >>> Z> >>> Z> After DDB debugging and tracing , it seems that is triggered by a = combine of [1] and [2] >>> Z> >>> Z> [1] = https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915 = = > >>> Z> [2] = https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b = = > >>> Z> >>> Z> >>> Z> In [1] the per-VNET uma zone is shared with the global one. >>> Z> `pcbinfo->ipi_zone =3D pcbstor->ips_zone;` >>> Z> >>> Z> In [2] unref `inp->inp_cred` is deferred called in inpcb_dtor() = by uma_zfree_smr() . >>> Z> >>> Z> Unfortunately inps freed by uma_zfree_smr() are cached and = inpcb_dtor() is not called immediately , >>> Z> thus leaking `inp->inp_cred` ref and hence `prison->pr_ref`. >>> Z> >>> Z> And it is also not possible to free up the cache by per-VNET = SYSUNINIT tcp_destroy / udp_destroy / rip_destroy. >>>=20 >>> This is known issue and I'd prefer not to call it a problem. The = "leak" of a jail >>> happens only if machine is idle wrt the networking activity. >>>=20 >>> Getting back to the problem that started this thread - the epair(4)s = not immediately >>> popping back to prison0. IMHO, the problem again lies in the design = of if_vmove and >>> epair(4) in particular. The if_vmove shall not exist, instead we = should do a full >>> if_attach() and if_detach(). The state of an ifnet when it undergoes = if_vmove doesn't >>> carry any useful information. With Alexander melifaro@ we discussed = better options >>> for creating or attaching interfaces to jails that if_vmove. Until = they are ready >>> the most easy workaround to deal with annoying epair(4) come back = problem is to >>> remove it manually before destroying a jail, like I did in = 80fc25025ff. >>>=20 >>=20 >> It still behaved much better prior to eb93b99d6986, which you and = Mark >> were going to work on a solution for to allow the cred "leak" to = close >> up much more quickly. CC markj@, since I think it's been six months >> since the last time I inquired about it, making this a good time to = do >> it again... >=20 > I spent some time trying to see if we could fix this in UMA/SMR and > talked to Jeff about it a bit. At this point I don't think it's the > right approach, at least for now. Really we have a composability > problem where different layers are using different techniques to = signal > that they're done with a particular piece of memory, and they just > aren't compatible. I originally thought that `uma_free_smr()` is somewhat like = `epoch_call()` with an internal `epoch_callback_t`, but after digging into the source code = it is not true. `uma_free_smr()` put the item into cache and until next allocation from = the cache the destructor get a chance to run. Can SMR provide some mean just like `epoch_callback_t` , so that the = destructors eventually get been invoked ? >=20 > One thing I tried is to implement a UMA function which walks over all > SMR zones and synchronizes all cached items (so that their destructors > are called). This is really expensive, at minimum it has to bind to = all > CPUs in the system so that it can flush per-CPU buckets. If > jail_deref() calls that function, the bug goes away at least in my > limited testing, but its use is really a layering violation. I've proposed a `vnet_shutdown()` stage in another mail. Maybe we can = introduce a `vnet_cleanup()` and INPCB layer register to listen a `cleanup` event = and the function which synchronizing cached items get been invoked. Is that still a layering violation? >=20 > We could, say, periodically scan cached UMA/SMR items and invoke their > destructors, but for most SMR consumers this is unnecessary, and again > there's a layering problem: the inpcb layer shouldn't "know" that it = has > to do that for its zones, since it's the jail layer that actually = cares. >=20 > It also seems kind of strange that dying jails still occupy a slot in > the jail namespace. I don't really understand why the existence of a > dying jail prevents creation of a new jail with the same name, but > presumably there's a good reason for it? >=20 > Now my inclination is to try and fix this in the inpcb layer, by not > accessing the inp_cred at all in the lookup path until we hold the = inpcb > lock, and then releasing the cred ref before freeing a PCB to its = zone. > I think this is doable based on a few observations: > - When doing an SMR-protected lookup, we always lock the returned = inpcb > before handing it to the caller. So we could in principle perform > inp_cred checking after acquiring the lock but before returning. > - If there are no jailed PCBs in a hash chain = in_pcblookup_hash_locked() > always scans the whole chain. > - If we match only one PCB in a lookup, we can probably(?) return that > PCB without dereferencing the cred pointer at all. If not, then the > scan only has to keep track of a fixed number of PCBs before picking > which one to return. So it looks like we can perform a lockless scan > and keep track of matches on the stack, then lock the matched PCBs = and > perform prison checks if necessary, without making the common case > more expensive. >=20 > In fact there is a parallel thread on freebsd-jail which reports that > this inp_cred access is a source of frequent cache misses. I was > surprised to see that the scan calls prison_flag() before even = checking > the PCB's local address. So if the hash chain is large then we're > potentially performing a lot of unnecessary memory accesses (though > presumably it's common for most of the PCBs to be sharing a single > cred?). In particular we can perhaps solve two problems at once. >=20 > Any thoughts? Are there some fundamental reasons this can't work? --Apple-Mail=_05999826-0A4B-4FD5-A20A-0ED8E067BC1C Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii

On Dec 21, 2022, at = 12:12 AM, Mark Johnston <markj@freebsd.org> wrote:

On Sun, Dec 18, 2022 at 10:52:58AM -0600, = Kyle Evans wrote:
On Sat, Dec 17, 2022 at 11:22 AM Gleb Smirnoff = <glebius@freebsd.org> wrote:

 Zhenlei,

On Fri, Dec 16, 2022 at 06:30:57PM +0800, Zhenlei Huang = wrote:
Z> I managed to repeat this issue on CURRENT/14 = with this small snip:
Z>
Z> = -------------------------------------------
Z> = #!/bin/sh
Z>
Z> # test jail name
Z> n=3D"test_ref_leak"
Z>
Z> jail -c name=3D$n path=3D/ vnet persist
Z> # The following line trigger jail pr_ref leak
Z> jexec $n ifconfig lo0 inet 127.0.0.1/8
Z>
Z> jail -R $n
Z>
Z> # wait a moment
Z> sleep 1
Z>
Z> jls -j $n
Z>
Z> After DDB debugging and tracing , it seems that is = triggered by a combine of [1] and [2]
Z>
Z> [1] https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5a= c895915 <https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5a= c895915>
Z> [2] https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b1= 75b8c5b <https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b1= 75b8c5b>
Z>
Z>
Z> In [1] the per-VNET uma zone is shared with the global = one.
Z> `pcbinfo->ipi_zone =3D = pcbstor->ips_zone;`
Z>
Z> In [2] = unref `inp->inp_cred` is deferred called in inpcb_dtor() by = uma_zfree_smr() .
Z>
Z> Unfortunately = inps freed by uma_zfree_smr() are cached and inpcb_dtor() is not called = immediately ,
Z> thus leaking `inp->inp_cred` ref = and hence `prison->pr_ref`.
Z>
Z> = And it is also not possible to free up the cache by per-VNET SYSUNINIT = tcp_destroy / udp_destroy / rip_destroy.

This= is known issue and I'd prefer not to call it a problem. The "leak" of a = jail
happens only if machine is idle wrt the networking = activity.

Getting back to the problem that = started this thread - the epair(4)s not immediately
popping = back to prison0. IMHO, the problem again lies in the design of if_vmove = and
epair(4) in particular. The if_vmove shall not exist, = instead we should do a full
if_attach() and if_detach(). = The state of an ifnet when it undergoes if_vmove doesn't
carry any useful information. With Alexander melifaro@ we = discussed better options
for creating or attaching = interfaces to jails that if_vmove. Until they are ready
the = most easy workaround to deal with annoying epair(4) come back problem is = to
remove it manually before destroying a jail, like I did = in 80fc25025ff.


It still behaved much better prior to eb93b99d6986, which you = and Mark
were going to work on a solution for to allow the = cred "leak" to close
up much more quickly. CC markj@, = since I think it's been six months
since the last time I = inquired about it, making this a good time to do
it = again...

I spent = some time trying to see if we could fix this in UMA/SMR and
talked to Jeff about it = a bit.  At this point I don't think it's the
right approach, at least for now. =  Really we have a composability
problem where different layers are using = different techniques to signal
that = they're done with a particular piece of memory, and they just
aren't = compatible.

I originally thought that = `uma_free_smr()` is somewhat like `epoch_call()` with
an internal `epoch_callback_t`, but after digging into the source = code it is not true.
`uma_free_smr()` put the item into cache and until next allocation = from the
cache the destructor get a chance = to run.

Can SMR provide some mean just like `epoch_callback_t` , so that the destructors
eventually get been invoked ?


One = thing I tried is to implement a UMA function which walks over = all
SMR = zones and synchronizes all cached items (so that their = destructors
are = called).  This is really expensive, at minimum it has to bind to = all
CPUs in = the system so that it can flush per-CPU buckets.  If
jail_deref() calls that = function, the bug goes away at least in my
limited testing, but its use is really a = layering violation.

I've proposed a `vnet_shutdown()` = stage in another mail. Maybe we can introduce
a `vnet_cleanup()`  and INPCB layer register to listen a = `cleanup` event and
the function which synchronizing = cached items get been invoked.
Is that still a layering = violation?


We could, say, periodically scan cached = UMA/SMR items and invoke their
destructors, but for most SMR consumers this is = unnecessary, and again
there's = a layering problem: the inpcb layer shouldn't "know" that it = has
to do = that for its zones, since it's the jail layer that actually = cares.

It also = seems kind of strange that dying jails still occupy a slot in
the jail namespace. =  I don't really understand why the existence of a
dying jail prevents = creation of a new jail with the same name, but
presumably there's a good reason for = it?

Now my = inclination is to try and fix this in the inpcb layer, by not
accessing the inp_cred = at all in the lookup path until we hold the inpcb
lock, and then releasing the cred ref = before freeing a PCB to its zone.
I think this is doable based on a few = observations:
- When = doing an SMR-protected lookup, we always lock the returned = inpcb
 before handing it to the caller.  So we could in = principle perform
 inp_cred checking after acquiring the lock but before = returning.
- If = there are no jailed PCBs in a hash chain = in_pcblookup_hash_locked()
 always scans the whole chain.
- If we match only one PCB in a lookup, we = can probably(?) return that
 PCB without dereferencing the cred pointer at all. =  If not, then the
 scan only has to keep track of a fixed number of PCBs = before picking
 which one to return.  So it looks like we can = perform a lockless scan
 and keep track of matches on the stack, then lock the = matched PCBs and
 perform prison checks if necessary, without making = the common case
 more expensive.

In fact there is a parallel thread on = freebsd-jail which reports that
this = inp_cred access is a source of frequent cache misses.  I = was
surprised to see that the scan calls prison_flag() before = even checking
the = PCB's local address.  So if the hash chain is large then = we're
potentially performing a lot of unnecessary memory accesses = (though
presumably it's common for most of the PCBs to be sharing a = single
cred?). =  In particular we can perhaps solve two problems at once.

Any thoughts?  Are = there some fundamental reasons this can't = work?
= --Apple-Mail=_05999826-0A4B-4FD5-A20A-0ED8E067BC1C-- From nobody Thu Dec 22 16:04:22 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NdFjW4d2Dz1G64x for ; Thu, 22 Dec 2022 16:13:35 +0000 (UTC) (envelope-from glebius@freebsd.org) Received: from glebi.us (glebi.us [162.251.186.162]) by mx1.freebsd.org (Postfix) with ESMTP id 4NdFjV590Xz3mNW; Thu, 22 Dec 2022 16:13:34 +0000 (UTC) (envelope-from glebius@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=softfail (mx1.freebsd.org: 162.251.186.162 is neither permitted nor denied by domain of glebius@freebsd.org) smtp.mailfrom=glebius@freebsd.org; dmarc=none Received: by glebi.us (Postfix, from userid 1000) id 061944EE96; Thu, 22 Dec 2022 08:04:22 -0800 (PST) Date: Thu, 22 Dec 2022 08:04:22 -0800 From: Gleb Smirnoff To: Kyle Evans Cc: Zhenlei Huang , "Bjoern A. Zeeb" , "freebsd-jail@freebsd.org" , Mark Johnston Subject: Re: What's going on with vnets and epairs w/ addresses? Message-ID: References: <5r22os7n-ro15-27q-r356-rps331o06so5@mnoonqbm.arg> <150A60D6-6757-46DD-988F-05A9FFA36821@FreeBSD.org> List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spamd-Result: default: False [1.59 / 15.00]; VIOLATED_DIRECT_SPF(3.50)[]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.91)[-0.910]; RCVD_NO_TLS_LAST(0.10)[]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; DMARC_NA(0.00)[freebsd.org]; MLMMJ_DEST(0.00)[freebsd-jail@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; FREEFALL_USER(0.00)[glebius]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_FIVE(0.00)[5]; R_SPF_SOFTFAIL(0.00)[~all:c]; TO_DN_SOME(0.00)[]; GREYLIST(0.00)[pass,body]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:27348, ipnet:162.251.186.0/24, country:US]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4NdFjV590Xz3mNW X-Spamd-Bar: + X-ThisMailContainsUnwantedMimeParts: N On Sun, Dec 18, 2022 at 10:52:58AM -0600, Kyle Evans wrote: K> It still behaved much better prior to eb93b99d6986, which you and Mark K> were going to work on a solution for to allow the cred "leak" to close K> up much more quickly. CC markj@, since I think it's been six months K> since the last time I inquired about it, making this a good time to do K> it again... I had a branch with a KPI to force purge SMR zone. That could be triggered by a sysctl or automatically from the jail code. I will look for it and share. Sorry for 4 day delay on this (and other topics), I've been mostly offline. I will reply to all mails on this thread before Monday. -- Gleb Smirnoff From nobody Fri Dec 23 11:15:01 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Ndl2v0LkTz1HWs3 for ; Fri, 23 Dec 2022 11:15:19 +0000 (UTC) (envelope-from melifaro@ipfw.ru) Received: from forward502a.mail.yandex.net (forward502a.mail.yandex.net [IPv6:2a02:6b8:c0e:500:1:45:d181:d502]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Ndl2s1Cfmz4XTn for ; Fri, 23 Dec 2022 11:15:16 +0000 (UTC) (envelope-from melifaro@ipfw.ru) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=ipfw.ru header.s=mail header.b=UfBM6UX+; spf=pass (mx1.freebsd.org: domain of melifaro@ipfw.ru designates 2a02:6b8:c0e:500:1:45:d181:d502 as permitted sender) smtp.mailfrom=melifaro@ipfw.ru; dmarc=none Received: from vla1-27b4fc0f1fa3.qloud-c.yandex.net (vla1-27b4fc0f1fa3.qloud-c.yandex.net [IPv6:2a02:6b8:c0d:4201:0:640:27b4:fc0f]) by forward502a.mail.yandex.net (Yandex) with ESMTP id 2E2575EB5F; Fri, 23 Dec 2022 14:15:13 +0300 (MSK) Received: by vla1-27b4fc0f1fa3.qloud-c.yandex.net (smtp/Yandex) with ESMTPSA id BFmn9L9YjeA1-GzVKILLN; Fri, 23 Dec 2022 14:15:12 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfw.ru; s=mail; t=1671794112; bh=QgE5v5PhacSC+MN3E4frWwA7qevtOphb3inPgVLv5CU=; h=Message-Id:To:Date:References:Cc:In-Reply-To:From:Subject; b=UfBM6UX+dexEHax/4y6V5QZD/EimZ8W4TK+rZO9JkGgWiTK5EPIo9OsQhtQzH49bW qlOwaKA3Fg3pumKNHYjZWcQfiXUQJ/arSpIsdZWzI0X5mL8JbQUsdEdaq5OeFB1nEN JwMNCpoJuN3H67xnxCWEce6jPWzlVSoQxar3vjIk= Content-Type: text/plain; charset=utf-8 List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.200.110.1.12\)) Subject: Re: Is it possible to employ epoch to simplify managing prison lifecycle From: "Alexander V. Chernikov" In-Reply-To: Date: Fri, 23 Dec 2022 11:15:01 +0000 Cc: Zhenlei Huang , freebsd-jail@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <4E70D6D2-4E80-4AAD-BB3C-9295F586D1FF@ipfw.ru> References: <9BD54A54-A809-4D3E-BCBA-639E6C61FE37@FreeBSD.org> To: Mateusz Guzik X-Mailer: Apple Mail (2.3731.200.110.1.12) X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip6:2a02:6b8:c00::/40]; R_DKIM_ALLOW(-0.20)[ipfw.ru:s=mail]; MIME_GOOD(-0.10)[text/plain]; FREEFALL_USER(0.00)[melifaro]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[ipfw.ru]; TAGGED_RCPT(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-jail@freebsd.org]; DKIM_TRACE(0.00)[ipfw.ru:+]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:208722, ipnet:2a02:6b8::/32, country:FI]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org] X-Rspamd-Queue-Id: 4Ndl2s1Cfmz4XTn X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N > On 16 Dec 2022, at 16:29, Mateusz Guzik wrote: >=20 > On 12/16/22, Zhenlei Huang wrote: >> Hi, >>=20 >> While hacking `sys/kern/kern_jail.c` I got lost. >>=20 >> There're lots of ref / unref and flags to prevent visit invalid = prison >> while >> concurrent modification is possible and some refs looks weird. >>=20 >> Is it possible to employ epoch(9) to simplify managing of prison = lifecycle >> ? >>=20 >=20 > Some of the ref/unref cycles are probably avoidable to begin with, but > ultimately the thing to do here is to employ per-cpu reference > counting, if at all needed. >=20 > I have a wip patch to provide such a mechanism, it may or may not land > this month. That would be nice. I=E2=80=99d love to convert nextops refcounting to = that one. Do you envision similar semantics as Linux percpu_ref? I mean, does one = need to explicitly mark =E2=80=9Cnot in active use=E2=80=9D stage? >=20 > --=20 > Mateusz Guzik >=20 From nobody Fri Dec 23 15:27:53 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NdrfN0pkPz1LqMs for ; Fri, 23 Dec 2022 15:27:56 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oa1-x32.google.com (mail-oa1-x32.google.com [IPv6:2001:4860:4864:20::32]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NdrfM5Pvyz3w38 for ; Fri, 23 Dec 2022 15:27:55 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oa1-x32.google.com with SMTP id 586e51a60fabf-12c8312131fso6260810fac.4 for ; Fri, 23 Dec 2022 07:27:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :references:in-reply-to:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=eGQlFsUOUKtaq95xtCJBqVL0R2iH0hrGzT+QWjGosRs=; b=A7Qog7cA7wyGeD1Rye1taA/T2aCmKMYh45+WtIw0Ysyednd/D2jcw0ST0/WO9EvHGo rivLHGxITOYTTUTOGNKF/hW1xV3OLtEp66XFAY13npyPYKPW3Mt5G0ZW1oSligzGJZFT TSmb4zkn+rRS++HiPrvo7pCOQaJYyoGK+X8yrRHlX1ASMtBNhg2pVYRgll5DAuFu3mfm NfIff04xUM+1CW0N1J44RfkWqS3T+5kBEPKJk1d7ll8vKsSpc0Ez/VY/XkrMpwYewCta E/ZRoVUhD1iaW55J+m5ihzYoVofhdaqntSh+spPW6wXyFkRnBUdg6cbIMeGahpw0Q1dV Gozw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :references:in-reply-to:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eGQlFsUOUKtaq95xtCJBqVL0R2iH0hrGzT+QWjGosRs=; b=5eWIL5I2ogHdvv+6JznGit+wPTQaESWnhMtweKTMzhsL65oeQBta9SJhYmKtP/BfPd aMflUPAlTy/fMu5dMfKRGYnZT8tiLByS2fbY10j/hA8F1c1kDjcrzVORVeZMxnT6b+71 9m3ErZUUKdJbQJ7/mkdXGCRNbHNXFINaOFEbxNhOV3nMmm9nlrcVNwycf2YxOjHmQ8YG vYVSxMLBBgf9dpBOBp4jcdabf1TIa/NAgyrqwFO30NQ2fD875CGmF0+Z9EeRO0DKqG+V VytbMJhCAgEOrzYFSyPBLAdRKFZf9B8lT6xx9Fva/o7cq6g3Znruuv06QCL4Hc43l1zZ ucgg== X-Gm-Message-State: AFqh2krIStY6fpMcfhtvM1Twr1e3iKIUvovd33ttik+QXc/aOqsIMiLC nggJHdI5inpGPYBHUC1iaawxPaMjiiZwuxm47skoIB+S X-Google-Smtp-Source: AMrXdXsTyIJsmK51Dc+kfLd+GJLtSJHJFVyl4FVy9amwH6cEUXoDKRRA6rFeoVtv7Stsb6NXmNoKHy9fUL88PAd4SFk= X-Received: by 2002:a05:6870:e9a:b0:144:68b8:88eb with SMTP id mm26-20020a0568700e9a00b0014468b888ebmr658188oab.159.1671809273886; Fri, 23 Dec 2022 07:27:53 -0800 (PST) List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Received: by 2002:ac9:77d5:0:b0:491:8368:9bdd with HTTP; Fri, 23 Dec 2022 07:27:53 -0800 (PST) In-Reply-To: <4E70D6D2-4E80-4AAD-BB3C-9295F586D1FF@ipfw.ru> References: <9BD54A54-A809-4D3E-BCBA-639E6C61FE37@FreeBSD.org> <4E70D6D2-4E80-4AAD-BB3C-9295F586D1FF@ipfw.ru> From: Mateusz Guzik Date: Fri, 23 Dec 2022 16:27:53 +0100 Message-ID: Subject: Re: Is it possible to employ epoch to simplify managing prison lifecycle To: "Alexander V. Chernikov" Cc: Zhenlei Huang , freebsd-jail@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4NdrfM5Pvyz3w38 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_RCPT(0.00)[]; ASN(0.00)[asn:15169, ipnet:2001:4860:4864::/48, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On 12/23/22, Alexander V. Chernikov wrote: > > >> On 16 Dec 2022, at 16:29, Mateusz Guzik wrote: >> >> On 12/16/22, Zhenlei Huang wrote: >>> Hi, >>> >>> While hacking `sys/kern/kern_jail.c` I got lost. >>> >>> There're lots of ref / unref and flags to prevent visit invalid prison >>> while >>> concurrent modification is possible and some refs looks weird. >>> >>> Is it possible to employ epoch(9) to simplify managing of prison >>> lifecycle >>> ? >>> >> >> Some of the ref/unref cycles are probably avoidable to begin with, but >> ultimately the thing to do here is to employ per-cpu reference >> counting, if at all needed. >> >> I have a wip patch to provide such a mechanism, it may or may not land >> this month. > That would be nice. I=E2=80=99d love to convert nextops refcounting to th= at one. > Do you envision similar semantics as Linux percpu_ref? I mean, does one n= eed > to explicitly mark =E2=80=9Cnot in active use=E2=80=9D stage? There *something* needed to disable per-cpu operation, otherwise how can you ever know if the count is 0, apart from going over all cpus every time, which defeats the point. More specifically, I have a on/off switch for said per-cpu op. This is modeled after what I did for counters in vfs, see vfs_ref et al. --=20 Mateusz Guzik From nobody Sat Dec 24 11:09:54 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NfLtW1sxRz1J544 for ; Sat, 24 Dec 2022 11:10:11 +0000 (UTC) (envelope-from melifaro@ipfw.ru) Received: from forward500b.mail.yandex.net (forward500b.mail.yandex.net [IPv6:2a02:6b8:c02:900:1:45:d181:d500]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4NfLtV6Zdhz45KM for ; Sat, 24 Dec 2022 11:10:10 +0000 (UTC) (envelope-from melifaro@ipfw.ru) Authentication-Results: mx1.freebsd.org; none Received: from sas2-cc22fd2335f8.qloud-c.yandex.net (sas2-cc22fd2335f8.qloud-c.yandex.net [IPv6:2a02:6b8:c08:6c82:0:640:cc22:fd23]) by forward500b.mail.yandex.net (Yandex) with ESMTP id B9F725E7B5; Sat, 24 Dec 2022 14:10:06 +0300 (MSK) Received: by sas2-cc22fd2335f8.qloud-c.yandex.net (smtp/Yandex) with ESMTPSA id 4AKHpT8YOqM1-RuYq2oCL; Sat, 24 Dec 2022 14:10:06 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfw.ru; s=mail; t=1671880206; bh=XjsRFPa9nPDZKpe61+ezOmKGeNL9+Y6rC6FWnBnKctI=; h=Message-Id:To:Date:References:Cc:In-Reply-To:From:Subject; b=rxBr/bPI4A6C4i63/gojbK7yrpea7aGUM0MVnD45M51p4LysBPcBopFIpNwI3/XUv 6GXZ6Zrq3BRANNV8WMv24ljMpcutQ5BWdfCZ03DJj1JfuA00CuA+RAPcgchABQ8fTE 7+95fsFU8uNi7urCCNSO47XX4Hb0Z1h/epHfWlwU= Content-Type: text/plain; charset=utf-8 List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.200.110.1.12\)) Subject: Re: Is it possible to employ epoch to simplify managing prison lifecycle From: "Alexander V. Chernikov" In-Reply-To: Date: Sat, 24 Dec 2022 11:09:54 +0000 Cc: Zhenlei Huang , freebsd-jail@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <9BD54A54-A809-4D3E-BCBA-639E6C61FE37@FreeBSD.org> <4E70D6D2-4E80-4AAD-BB3C-9295F586D1FF@ipfw.ru> To: Mateusz Guzik X-Mailer: Apple Mail (2.3731.200.110.1.12) X-Rspamd-Queue-Id: 4NfLtV6Zdhz45KM X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_RCPT(0.00)[]; ASN(0.00)[asn:208722, ipnet:2a02:6b8::/32, country:FI] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N > On 23 Dec 2022, at 15:27, Mateusz Guzik wrote: >=20 > On 12/23/22, Alexander V. Chernikov wrote: >>=20 >>=20 >>> On 16 Dec 2022, at 16:29, Mateusz Guzik wrote: >>>=20 >>> On 12/16/22, Zhenlei Huang wrote: >>>> Hi, >>>>=20 >>>> While hacking `sys/kern/kern_jail.c` I got lost. >>>>=20 >>>> There're lots of ref / unref and flags to prevent visit invalid = prison >>>> while >>>> concurrent modification is possible and some refs looks weird. >>>>=20 >>>> Is it possible to employ epoch(9) to simplify managing of prison >>>> lifecycle >>>> ? >>>>=20 >>>=20 >>> Some of the ref/unref cycles are probably avoidable to begin with, = but >>> ultimately the thing to do here is to employ per-cpu reference >>> counting, if at all needed. >>>=20 >>> I have a wip patch to provide such a mechanism, it may or may not = land >>> this month. >> That would be nice. I=E2=80=99d love to convert nextops refcounting = to that one. >> Do you envision similar semantics as Linux percpu_ref? I mean, does = one need >> to explicitly mark =E2=80=9Cnot in active use=E2=80=9D stage? >=20 > There *something* needed to disable per-cpu operation, otherwise how > can you ever know if the count is 0, apart from going over all cpus > every time, which defeats the point. Ack, sounds reasonable. Happy to test the KPI once it=E2=80=99s = available. >=20 > More specifically, I have a on/off switch for said per-cpu op. This is > modeled after what I did for counters in vfs, see vfs_ref et al. >=20 > --=20 > Mateusz Guzik >=20 From nobody Mon Dec 26 20:42:26 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Ngqgk1gQsz1HwHB for ; Mon, 26 Dec 2022 20:50:58 +0000 (UTC) (envelope-from glebius@freebsd.org) Received: from glebi.us (glebi.us [162.251.186.162]) by mx1.freebsd.org (Postfix) with ESMTP id 4Ngqgj4l1Gz43RB; Mon, 26 Dec 2022 20:50:57 +0000 (UTC) (envelope-from glebius@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=softfail (mx1.freebsd.org: 162.251.186.162 is neither permitted nor denied by domain of glebius@freebsd.org) smtp.mailfrom=glebius@freebsd.org; dmarc=none Received: by glebi.us (Postfix, from userid 1000) id D877429F68; Mon, 26 Dec 2022 12:42:26 -0800 (PST) Date: Mon, 26 Dec 2022 12:42:26 -0800 From: Gleb Smirnoff To: Zhenlei Huang Cc: Mark Johnston , "Bjoern A. Zeeb" , "freebsd-jail@freebsd.org" Subject: Re: What's going on with vnets and epairs w/ addresses? Message-ID: References: <5r22os7n-ro15-27q-r356-rps331o06so5@mnoonqbm.arg> <150A60D6-6757-46DD-988F-05A9FFA36821@FreeBSD.org> List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <150A60D6-6757-46DD-988F-05A9FFA36821@FreeBSD.org> X-Spamd-Result: default: False [1.52 / 15.00]; VIOLATED_DIRECT_SPF(3.50)[]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.977]; MIME_GOOD(-0.10)[text/plain]; RCVD_NO_TLS_LAST(0.10)[]; RCPT_COUNT_THREE(0.00)[4]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[freebsd.org]; MLMMJ_DEST(0.00)[freebsd-jail@freebsd.org]; FREEFALL_USER(0.00)[glebius]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; R_SPF_SOFTFAIL(0.00)[~all:c]; TO_DN_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:27348, ipnet:162.251.186.0/24, country:US]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; GREYLIST(0.00)[pass,body] X-Rspamd-Queue-Id: 4Ngqgj4l1Gz43RB X-Spamd-Bar: + X-ThisMailContainsUnwantedMimeParts: N Zhenlei, Bjoern, Mark, sorry for delayed response on this thread. Back when the problem was first introduced, I made a code that forces purge of SMR zones. However, I didn't push it in, hence the change on the test suite side to remove interfaces from inside the jail before destroying it was sufficient to close all leaks associated with the test suite. I just rebased the code to fresh main and put it here: https://github.com/glebius/FreeBSD/tree/smr-purge The proof of concept based on the test from Zhenlei looks like this: #!/bin/sh n="test_ref_leak" jail -c name=$n path=/ vnet persist # The following line trigger jail pr_ref leak jexec $n ifconfig lo0 inet 127.0.0.1/8 jail -R $n for zone in tcp_inpcb udp_inpcb; do sysctl vm.uma_zone_reclaim=${zone} done jls -j $n At the point of the call to jls(8) the jail no longer exists. My opinion on the whole problem matches Mark's opinion, that he expressed in his email on December 20. I like the idea of doing the prison checks at a later stage of inpcb lookup, especially given new discoveries on the performance impact by Drew. The proper fix may take a while. In addition to that I have strong opinion against the way we move interfaces between the jails. I claim that if did it right (tm), the problem we are talking about won't exist even with all the existing layering violations between inpcb+smr and jails+epoch. I will write a longer email on what I believe is the right (tm) way to manage interfaces/devices within jails. We already have had discussions on that with Alexander melifaro@ and Warner imp@. However, proper implementation will take a while. We may use code from my smr-purge branch as a temporary solution. Any thoughts on that? -- Gleb Smirnoff From nobody Tue Dec 27 19:39:47 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NhQJy4G3Qz1HWF2 for ; Tue, 27 Dec 2022 19:51:46 +0000 (UTC) (envelope-from Mathias.Picker@virtual-earth.de) Received: from www94.your-server.de (www94.your-server.de [213.133.104.94]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4NhQJx2sN9z4L3g for ; Tue, 27 Dec 2022 19:51:45 +0000 (UTC) (envelope-from Mathias.Picker@virtual-earth.de) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=virtual-earth.de header.s=default_1811 header.b=HDK+GKcN; spf=pass (mx1.freebsd.org: domain of Mathias.Picker@virtual-earth.de designates 213.133.104.94 as permitted sender) smtp.mailfrom=Mathias.Picker@virtual-earth.de; dmarc=pass (policy=none) header.from=virtual-earth.de DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=virtual-earth.de; s=default_1811; h=Content-Transfer-Encoding:Content-Type: MIME-Version:Message-ID:Date:Subject:To:From:Sender:Reply-To:Cc:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:In-Reply-To:References; bh=o9TPBW3ymV/AkZ7hd1+A0eHnqJeQZ/wzjY2T9Aq4osg=; b=HDK+GKcNeb6JwCNKW+YrUoPHOW YVfLoCg0KuCRGNgyo0RDapZN9xVza+qacjlT++qfm4lkHqSybCjmJskBvzGKykn5/cWv+aX51cBN4 dH+Wt4QBR0j1T4dRfg+/YPpLHDEXaNKZSKicAim+/pEtmqjMpq4zICt8L47uc6+ohTNSe/Tc4iCC5 E32V6I8nNuZOlQSDO9BfuXfZnh0Mo24b2x/txXqB1wOkUm6+WBgg6H+OVEVMJyYa7UT4cQ2/epMZ8 IPtnKqwy0IkpbDTMzeXvFMOO1reoS/6ZFMhieBKAUFbbqEz/iYlCt/wUCx1GS61t2mU0LkFNuTHY1 XExJuohg==; Received: from sslproxy02.your-server.de ([78.47.166.47]) by www94.your-server.de with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1pAFzE-0003my-Ah for freebsd-jail@FreeBSD.org; Tue, 27 Dec 2022 20:51:44 +0100 Received: from [77.21.4.8] (helo=danton.virtual-earth.de) by sslproxy02.your-server.de with esmtpsa (TLSv1.3:TLS_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pAFzE-000Ip4-2P for freebsd-jail@FreeBSD.org; Tue, 27 Dec 2022 20:51:44 +0100 User-agent: mu4e 1.8.13; emacs 28.2 From: Mathias Picker To: freebsd-jail@FreeBSD.org Subject: debian jail, setting max open files soft limit does not work Date: Tue, 27 Dec 2022 20:39:47 +0100 Message-ID: <86sfh0pquo.fsf@virtual-earth.de> List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable X-Authenticated-Sender: Mathias.Picker@virtual-earth.de X-Virus-Scanned: Clear (ClamAV 0.103.7/26763/Tue Dec 27 09:21:06 2022) X-Spamd-Result: default: False [-3.98 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.980]; DMARC_POLICY_ALLOW(-0.50)[virtual-earth.de,none]; R_DKIM_ALLOW(-0.20)[virtual-earth.de:s=default_1811]; R_SPF_ALLOW(-0.20)[+mx:c]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_NONE(0.00)[213.133.104.94:from]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-jail@FreeBSD.org]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:24940, ipnet:213.133.96.0/19, country:DE]; DKIM_TRACE(0.00)[virtual-earth.de:+]; RCVD_COUNT_THREE(0.00)[3]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_NONE(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; HAS_X_AS(0.00)[] X-Rspamd-Queue-Id: 4NhQJx2sN9z4L3g X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N Hi all, I=E2=80=99ve set up a jail on 13.1 running debian stretch, and now a=20 triplestore needing many openfiles for a data import. Since on Linux the soft limit is pretty hard :) I need to set the=20 soft limit. I=E2=80=99ve edited /etc/security/limits.conf to set soft and hard limit=20 to 20000 (just to check), but after login the soft limit stays at=20 1024. Using prlimit I can change the limits of a running process, but=20 that is not passed on to subprocesses, which the app creates=20 constantly for import, there, the soft limit returns to 1024 :( Running prlimit --nofile 20000 or prlimit --nofile 20000:20000=20 does not work, either, the soft limit stays at 1024, and the=20 import fails. Does anyone know a way to change the soft limit permenantly? Thanks, Mathias --=20 Mathias Picker=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20 Gesch=C3=A4ftsf=C3=BChrer Mathias.Picker@virtual-earth.de virtual earth Gesellschaft f=C3=BCr Wissens re/pr=C3=A4 sentation mbH http://www.virtual-earth.de/ HRB126870 support@virtual-earth.de Westendstr. 142 089 / 1250 3943=20=20=20=20=20=20=20=20=20=20=20=20 From nobody Tue Dec 27 20:34:25 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NhRGC2J7Pz1HcSb for ; Tue, 27 Dec 2022 20:34:27 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oi1-x22c.google.com (mail-oi1-x22c.google.com [IPv6:2607:f8b0:4864:20::22c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NhRGC0LHTz4Q7t for ; Tue, 27 Dec 2022 20:34:27 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oi1-x22c.google.com with SMTP id d127so11939387oif.12 for ; Tue, 27 Dec 2022 12:34:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :references:in-reply-to:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=A5NZ/6fJakOXPgNBER5OOkVUKnU3FIVR6vY31AWGtl8=; b=g945Oayb0IQjWFp6d6soqs3p9GpiAiOSovmSCeeD7vdlwpF8Hy0glp4exKkqDHEar1 GKK8mV6l2flnmE/f3rdnHN+Tzm3y8+RSkSmZ1YmU9h2t5HwWlRPtWfyCwPhv2zkaSd4o XW0//EEF3oUi8nsFmuY67pRV9XvnH5KL9num9Uy4snxcDbibSOIsvzgTuiI9HYbi85uC 8SQ6bSv/S2rFrx1Vjg79wYnu5nqwUzjoz07ry3sY0rgZobcvmwsGP8/wAklQYPN/pcAD nxnXxnCIF576IHNbQGcIQjS3GkSARfmfITZZcWLlB4WfNzQKa5wyLWqZ1qRPUD7O5RpG b9LA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :references:in-reply-to:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=A5NZ/6fJakOXPgNBER5OOkVUKnU3FIVR6vY31AWGtl8=; b=QtZpWPHlu5og0NlVu+qcB/ElEWCaWuJWqMACeAy6r3UadM1e+65D1drtkntEvENv0t f161PJmUDn7Cjx8AWhS/N9N5tKioyYA8VnvMKp9Mnqb2e7UtFdkgRX03gvIFXGadPKSI kkSX8VjOiDk2pmBox2FqrsLlfEXbijcx3vVJ3bVphZyLvXrNL4G4xN4sUyEHwsd15im1 oTMUA3wpV4DIPtOghRSdJejL9r14mX7CRIFwyoxIku6ambXbvzWv0zmSLynm6Sl24sAC G4jtW57kxXU9A8k4R6OIcNwqI3H6GoM3gtJlz4ElYNDIYB4iUBi8IBPELoKzu8H9Dqdj E2Bg== X-Gm-Message-State: AFqh2krWtnr8NATadxenbp0r/yglqhuJrezGBaRCR07djsN925qeuALy /rBUGs2Ghrvl0tQEUO4BUt5R0I3hyX1WWgJ1ukZq90el X-Google-Smtp-Source: AMrXdXuKwvlWDDgttSRhelEnGh5Pv5M7dFOAvowvT6pzxZlK+/Hu6hBSQo9cdPWdef3hXqmK1RqH6Z2fLMsmCqOf3a8= X-Received: by 2002:a05:6808:2:b0:35a:c014:9ca0 with SMTP id u2-20020a056808000200b0035ac0149ca0mr1175913oic.159.1672173265647; Tue, 27 Dec 2022 12:34:25 -0800 (PST) List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Received: by 2002:ac9:77d5:0:b0:491:8368:9bdd with HTTP; Tue, 27 Dec 2022 12:34:25 -0800 (PST) In-Reply-To: <86sfh0pquo.fsf@virtual-earth.de> References: <86sfh0pquo.fsf@virtual-earth.de> From: Mateusz Guzik Date: Tue, 27 Dec 2022 21:34:25 +0100 Message-ID: Subject: Re: debian jail, setting max open files soft limit does not work To: Mathias Picker Cc: freebsd-jail@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4NhRGC0LHTz4Q7t X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On 12/27/22, Mathias Picker wrote: > Hi all, > > > I=E2=80=99ve set up a jail on 13.1 running debian stretch, and now a > triplestore needing many openfiles for a data import. > > Since on Linux the soft limit is pretty hard :) I need to set the > soft limit. > > I=E2=80=99ve edited /etc/security/limits.conf to set soft and hard limit > to 20000 (just to check), but after login the soft limit stays at > 1024. > > Using prlimit I can change the limits of a running process, but > that is not passed on to subprocesses, which the app creates > constantly for import, there, the soft limit returns to 1024 :( > > Running prlimit --nofile 20000 or prlimit --nofile 20000:20000 > does not work, either, the soft limit stays at 1024, and the > import fails. > > Does anyone know a way to change the soft limit permenantly? > kernel code is buggy here, from a quick read you should be able to work around it by: sysctl compat.linux.default_openfiles=3D-1 --=20 Mateusz Guzik From nobody Wed Dec 28 08:22:04 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Nhl1M4JpZz2cBcs for ; Wed, 28 Dec 2022 08:24:23 +0000 (UTC) (envelope-from Mathias.Picker@virtual-earth.de) Received: from www94.your-server.de (www94.your-server.de [213.133.104.94]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Nhl1M17P9z44Qq for ; Wed, 28 Dec 2022 08:24:22 +0000 (UTC) (envelope-from Mathias.Picker@virtual-earth.de) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=virtual-earth.de; s=default_1811; h=Content-Transfer-Encoding:Content-Type: MIME-Version:Message-ID:In-reply-to:Date:Subject:Cc:To:From:References:Sender :Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID; bh=8C/QFi+i3mxduGMXVHrHyMypBtIXwL+gQyMfmTBpQSw=; b=V0wX1G8Fom5r9s1cAvKMg2nt8O nNVVFxyjqfnRFQuHLtiUqDqVneblA57dOZq/mYjKNhm0IzqZu9RU0zstzmIPMLSMIlIvETjnGsTaO 5feG0qG27QpmXpQAoKIP3capNumHb34Gx8HtkzRl2+KFsbNgODKrk5EgqoqfiZaj24Zc8CE9//g4E dMYZPYUbzEiWFYgH/+bTjy+30Sz/VMQwf6qRMALkgHjGGP/UBP/THsKy23Vhb/EOhiTCZvsqNEc0U JUM3s/NaM08EY62jH60yaEYSB/Ulfez3xQg2ZWKnlUFbSyXNKZgRs3f4wZxvFF/6w0s0jJlCzAm6N 4xB2wqGw==; Received: from sslproxy05.your-server.de ([78.46.172.2]) by www94.your-server.de with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1pARjY-000J3l-9B; Wed, 28 Dec 2022 09:24:20 +0100 Received: from [77.21.4.8] (helo=danton.virtual-earth.de) by sslproxy05.your-server.de with esmtpsa (TLSv1.3:TLS_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pARjY-0002D4-1X; Wed, 28 Dec 2022 09:24:20 +0100 References: <86sfh0pquo.fsf@virtual-earth.de> User-agent: mu4e 1.8.13; emacs 28.2 From: Mathias Picker To: Mateusz Guzik Cc: freebsd-jail@freebsd.org Subject: Re: debian jail, setting max open files soft limit does not work Date: Wed, 28 Dec 2022 09:22:04 +0100 In-reply-to: Message-ID: <86o7roos0c.fsf@virtual-earth.de> List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable X-Authenticated-Sender: Mathias.Picker@virtual-earth.de X-Virus-Scanned: Clear (ClamAV 0.103.7/26763/Tue Dec 27 09:21:06 2022) X-Rspamd-Queue-Id: 4Nhl1M17P9z44Qq X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:24940, ipnet:213.133.96.0/19, country:DE] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N Hi Mateusz, Mateusz Guzik writes: > On 12/27/22, Mathias Picker =20 > wrote: >> Hi all, >> >> I=E2=80=99ve set up a jail on 13.1 running debian stretch, and now a >> triplestore needing many openfiles for a data import. [snip] >> Running prlimit --nofile 20000 or prlimit --nofile 20000:20000 >> does not work, either, the soft limit stays at 1024, and the >> import fails. >> >> Does anyone know a way to change the soft limit permenantly? > > kernel code is buggy here, from a quick read you should be able=20 > to > work around it by: > > sysctl compat.linux.default_openfiles=3D-1 this worked just fine!!! :) Thanks a lot, Mathias --=20 Mathias Picker=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20 Gesch=C3=A4ftsf=C3=BChrer Mathias.Picker@virtual-earth.de virtual earth Gesellschaft f=C3=BCr Wissens re/pr=C3=A4 sentation mbH http://www.virtual-earth.de/ HRB126870 support@virtual-earth.de Westendstr. 142 089 / 1250 3943=20=20=20=20=20=20=20=20=20=20=20=20 From nobody Tue Jan 3 00:59:33 2023 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NmDsT2Hlrz2p0m9 for ; Tue, 3 Jan 2023 00:59:41 +0000 (UTC) (envelope-from zlei.huang@gmail.com) Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NmDsS1zRvz3DqW; Tue, 3 Jan 2023 00:59:40 +0000 (UTC) (envelope-from zlei.huang@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=JxbTnxvQ; spf=pass (mx1.freebsd.org: domain of zlei.huang@gmail.com designates 2607:f8b0:4864:20::1029 as permitted sender) smtp.mailfrom=zlei.huang@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pj1-x1029.google.com with SMTP id gv5-20020a17090b11c500b00223f01c73c3so26426857pjb.0; Mon, 02 Jan 2023 16:59:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=n9S8jeME4/Z/JqAIylWpSEK/498Z+t0BjGsklh/fTw4=; b=JxbTnxvQ0/L5BKAGusRKHfzsedkjLzzvLKXU+r3xKf8ype7KQ2vphhNqUGK2B5w0tG /uRUT3UkewJrXumHGruEOgDWBWIVIsvu+H3pfgsBQBSiiEfn4Rjwr1t1/NB7yrklCDG6 QLMm80g45L65zM3LSsmEKWKBAa/mPXcnA4fCoQK4mn6+L/YA9aYsWe9shcHz4QfWGqpJ W0vf7llbnouSvOn6xNgqqmdxvSDlbgs30snaS/GF4qJnHPW4sD42FDjyLUIbF+cAy3dz PfXzVtmU80V+MYqvZ7bkC1Gxd1Va2wQAMrDAyC8xU9jaq9pE13QJBU1eeVqO4LzKIKtx HsYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=n9S8jeME4/Z/JqAIylWpSEK/498Z+t0BjGsklh/fTw4=; b=0iNM4qUJpR1407/j2WdtwEHruqJi/F/SY0d8Cyt7qQRKAyKrWw4CedzPLai5zSqZxB vUSVaJkjLsKKgBx8rdt6PDSbjn8Yren7UNE4+kr/Hs15eJ8/Tk55vJbrn33U6pOnWMA9 RXqkmsA7TJN64GnezNDsHF5RhSFOflYio8/woC1d+f2qRV2RNso1WJ3bdL1qRRp8Jn1Z UaNMhIWaMKm6sul1uITEhegIvJ6jzUiGrn52W6ZEnRkIuaHh0rk1V4wPT836eAusODdT jpfNfgd07GrDuQJXHzh0EPOHZ4lyc7W+Z7jFr0GcR3lJA1PWUxQUC6m8IviIp3w7Khjp /k7A== X-Gm-Message-State: AFqh2kqpiVlqikxtLRvu3ap1U+tYfnmrMEZRBvSPaptdroJMb4SLh/0V wknDTpuIkLXgFhQpMKpzAgPhf0n2K28= X-Google-Smtp-Source: AMrXdXvah1UaNaMp6u5xFly3U0kB7IXjWTI+cw6WaS6FfWc3nb2RBOPZNKprqiJicjTdky6H2+1AAA== X-Received: by 2002:a17:90a:778c:b0:213:1d5:8acf with SMTP id v12-20020a17090a778c00b0021301d58acfmr44629300pjk.18.1672707578936; Mon, 02 Jan 2023 16:59:38 -0800 (PST) Received: from [172.17.252.129] (ns1.oxydns.net. [45.32.91.63]) by smtp.gmail.com with ESMTPSA id n63-20020a17090a2cc500b00213a9e1ec44sm19879293pjd.52.2023.01.02.16.59.37 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Jan 2023 16:59:38 -0800 (PST) Content-Type: text/plain; charset=us-ascii List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.7\)) Subject: Re: What's going on with vnets and epairs w/ addresses? From: Zhenlei Huang In-Reply-To: Date: Tue, 3 Jan 2023 08:59:33 +0800 Cc: Mark Johnston , "Bjoern A. Zeeb" , "freebsd-jail@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: References: <5r22os7n-ro15-27q-r356-rps331o06so5@mnoonqbm.arg> <150A60D6-6757-46DD-988F-05A9FFA36821@FreeBSD.org> To: Gleb Smirnoff X-Mailer: Apple Mail (2.3608.120.23.2.7) X-Spamd-Result: default: False [-2.48 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.984]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1029:from]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TAGGED_FROM(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-jail@freebsd.org] X-Rspamd-Queue-Id: 4NmDsS1zRvz3DqW X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N Hi, Happy New Year 2023! > On Dec 27, 2022, at 4:42 AM, Gleb Smirnoff = wrote: >=20 > Zhenlei, Bjoern, Mark, >=20 > sorry for delayed response on this thread. Back when the problem > was first introduced, I made a code that forces purge of SMR zones. > However, I didn't push it in, hence the change on the test suite side > to remove interfaces from inside the jail before destroying it was > sufficient to close all leaks associated with the test suite. >=20 > I just rebased the code to fresh main and put it here: >=20 > https://github.com/glebius/FreeBSD/tree/smr-purge >=20 > The proof of concept based on the test from Zhenlei looks like this: >=20 > #!/bin/sh > n=3D"test_ref_leak" >=20 > jail -c name=3D$n path=3D/ vnet persist > # The following line trigger jail pr_ref leak > jexec $n ifconfig lo0 inet 127.0.0.1/8 >=20 > jail -R $n >=20 > for zone in tcp_inpcb udp_inpcb; do > sysctl vm.uma_zone_reclaim=3D${zone} > done >=20 > jls -j $n >=20 > At the point of the call to jls(8) the jail no longer exists. >=20 > My opinion on the whole problem matches Mark's opinion, that he = expressed > in his email on December 20. I like the idea of doing the prison > checks at a later stage of inpcb lookup, especially given new = discoveries > on the performance impact by Drew. The proper fix may take a while. >=20 > In addition to that I have strong opinion against the way we move = interfaces > between the jails. I claim that if did it right (tm), the problem we = are > talking about won't exist even with all the existing layering = violations > between inpcb+smr and jails+epoch. I will write a longer email on what = I > believe is the right (tm) way to manage interfaces/devices within = jails. > We already have had discussions on that with Alexander melifaro@ and = Warner > imp@. However, proper implementation will take a while. >=20 > We may use code from my smr-purge branch as a temporary solution. Any > thoughts on that? The code in smr-purge branch should also apply to non-vnet jails. I think it is OK as a temporary solution. >=20 > --=20 > Gleb Smirnoff From nobody Thu Jan 5 01:21:58 2023 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NnTGH0g0tz2qy1J for ; Thu, 5 Jan 2023 01:21:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NnTGG5jVDz3tS9 for ; Thu, 5 Jan 2023 01:21:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1672881718; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=P6WTFYDsXxOfPfNafX48l5/rfqc6/ZYCg1823aJwqBk=; b=tUzr118kqrA2W7xltRXvsZxMK5Wr3Lk8zdwjWCkVKnXf78y+5bzz+vYOS+nSz/Mj+daobM tm6ShAyd2B5+DjlKGAudtmQFJXXWFdRUgi4tDZx2c8xhQsCcMfJDC/U84/Z4zkyySjS+o2 SACvVMiEc1loGT8q5p7iLf3ZdA9cZf4Jz2OF392TMGLefUBjYcEiwIoePufNu3a/0xAp5J 8wLL2QaylevF7AZUULRn4VV1YfIs6VXfDytMmWQbkI/DEKVc1IYhqP+yepMGh52eL97PAn uwR4jjy66ppiZyycbIvFIDMKSdDuzdyxBAGdDo7xpBmeM1KNCZnPqRrOVH0w+A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1672881718; a=rsa-sha256; cv=none; b=aLnhCrA5uBjLCp+K+IqRLpYWl0ff1kO1whLj7N+mvhY1ExqgY11ANxjv0QdxLns3KIY7m8 t5tXshPkmwd1tOmObe4c7TzqS++fsUMNaq/Yogw5Ow5QwtsF/XJc8K/p3WIsmA1Dmi4out VoN4McUZtZ9zRlbIh2+LNmO4LYdR0Ry3a60VT18qBlE5HuKV7z+QKovazmRAJmcMDp11vr 9X6Ki0SJYas3Ww/MIqmC6lHweTlLjMzctwIznTmQzWcX69o8MUQibnjr2r2oodkuVGmWmw 9ZHunbypd9QZBB1+0q5JOV11iaA1AL+DW6Py+tDhWFp4mhZeSu8S1Lijh9oC0g== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4NnTGG4nw0zX9t for ; Thu, 5 Jan 2023 01:21:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3051Lw4u015284 for ; Thu, 5 Jan 2023 01:21:58 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3051Lwmr015283 for jail@FreeBSD.org; Thu, 5 Jan 2023 01:21:58 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 208001] After turning off the jail does not remove network routes Date: Thu, 05 Jan 2023 01:21:58 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 10.3-BETA2 X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: admin@support.od.ua X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Feedback Timeout X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: resolution bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D208001 Vladislav V. Prodan changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |Feedback Timeout Status|Open |Closed --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Jan 6 10:14:06 2023 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NpK1v0Yv2z2pHFc; Fri, 6 Jan 2023 10:14:11 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NpK1v05fMz41ZY; Fri, 6 Jan 2023 10:14:11 +0000 (UTC) (envelope-from zlei@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1673000051; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=a4SKPPU3J5HnhI1Ab4inSkxLcICUlb1KdZSLjs276HQ=; b=ERKRiKJCzX/TYm3246pD427H4N8AOosfItpTZh2qCOut7vAwyGqJEkwn7dZtGQ15t2W+JT tx/tUUMY+l8e6hQ5+W0sOIEXi5vv9AcDVh7ce3NcRtvey1cCZEjdW4SuPbaTTQW1k3ieFq mVXy500s1gLhVgUArullzdkMRFF1QUiDitExRnA8XY1zuq3S5RuxKwiY7Lx8L3H6+nYZih xwFO0VuXM9YvisGw6c7gcCJz+Go1sseR/YN7OYhp0wZoORuIIKiQKsTIJ+AMIej4q+Wxq2 Ok1KCxFqWZaGHc/jd1Kge0N6G/1zfhuOfD9UlOBTrV+ykTB0nEVhXYSGFkNH6A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1673000051; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=a4SKPPU3J5HnhI1Ab4inSkxLcICUlb1KdZSLjs276HQ=; b=Qibx3oCrdtMH2qq5ZSK91EXuCiNxw++y2PxfxBAcaauFxO6jZ+KxNnE5nTd1NNWtdQsi2O 3g9m2XAhABusCfDbAupV0YF870j+ghJn03MLKSW3rI8xx7PTK2uchWwpRlSHEb9fqmAlRG pm0rYeukLdy0MA3fTuaOlRkqpoNEB/CKbVZgFX/Lr6gyvZNOTh6mI111K0ITduifTt/ipT yLR25OLkwPmdSpixx3bs5sFzU8XUj3z3StKdnN2U5uUwoZHYnRslWQjBjTTKFoIXHGfmPm gnlcv+2eSnY+CmZSs+2lQdJImzOcFArTHt8ZkTJUFERaOlNgdDpSI6eSFoBKyw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1673000051; a=rsa-sha256; cv=none; b=xnePYWrZA/9Posu9BHNeFCQ6/PmVW5rHTvyb/1zLUC0m+axbUjYuoiuC9pGVxspHPyfft5 1t1cUXIeoikRTQLZKCZiuloeaLIZlO3Y9wacH9AHVQSil9K0nBWZSvnFhsQw81tJ5vO8ry TlSmJI4UBO5uTQjrzmZixUwC5XHQqGXCjK6GoDE6BniGKxX3cWLRFAtJh042S4TgvoAmG7 LQR2riz1fVKvm7NzzTj2xik5UeUVjv8YMs9le4gm1zFK9u4gR4WvvRWsu2rrzTPfTLBE6l cilJRkZpW1A+A93+J8UsGBrUEbraJJrB0pTNRJhAnimIjJAZgtPK3SJFJHF1mg== Received: from smtpclient.apple (unknown [112.66.185.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: zlei/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4NpK1s6KpNz173v; Fri, 6 Jan 2023 10:14:09 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Content-Type: text/plain; charset=us-ascii List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) Subject: Re: Propose a new stage `vnet_shutdown` before `vnet_destroy` From: Zhenlei Huang In-Reply-To: <1c9dbf6d26b9525243dd6b3ffafa23cb@freebsd.org> Date: Fri, 6 Jan 2023 18:14:06 +0800 Cc: freebsd-jail@freebsd.org, freebsd-net Content-Transfer-Encoding: quoted-printable Message-Id: References: <1c9dbf6d26b9525243dd6b3ffafa23cb@freebsd.org> To: James Gritton X-Mailer: Apple Mail (2.3696.120.41.1.1) X-ThisMailContainsUnwantedMimeParts: N > On Dec 19, 2022, at 1:44 AM, James Gritton wrote: >=20 > On 2022-12-18 00:01, Zhenlei Huang wrote: >> I'm currently working on route nexthop caching feature for tunneling >> interfaces such as >> if_gif, if_gre, if_vxlan, and potentially if_wg. I encounter a nasty >> bug related to VNET lifecycle. >> More preciously I'd like to call `rib_unsubscribe()` to unsubscribe >> route event when the interface >> tunnel is deleted (gif_delete_tunnel). >> While on VNET shutting down, VNET SYSUNINIT was called and the = routing >> vnet subsystem >> is destroyed before the interface going down and hence cause >> pagefault. I do not want to check >> `vnet.vnet_shutdown` state as it looks messed up. >> I'm recently reviewing the life cycles of prison and get some = inspirations. >> When the jail / prison is submitted to destroy ( by jail_remove >> syscall ) then SIGKILL is sent to >> the prison's processes. I think it is correct order to destroy jail / >> prison. To summarize, the life cycle >> of jail / prison is: >> on jail create: PRISON_STATE_INVALID -> create VNET -> >> PRISON_STATE_ALIVE -> setup network resources, ifnet, if addresses, >> routing, etc. -> create / attach (network) processes >> on jail destroy: jexec kill processes (1) by user -> mark it as >> PRISON_STATE_DYING -> send SIGKILL to processes by kernel (2) -> >> destroy VNET (if prison pr_ref go to the last one) -> DYED >> The (2) is a cleanup by kernel as (1) is possible not done by user. >> So it comes the idea about the life cycle of VNET. >> While on jail destroy, the network resources are cleaned up by >> vnet_destroy ( SYSUNINIT ). Then the >> order of SYSUNINIT of network components is hacking as circular >> network resource dependency is possible. >> For example the routing table entries (nhop) have reference of ifnet, >> and ifnet have reference to route nhop (cache), as >> I encountered. >> Just like the cleanup processes by kernel, we can introduce a new >> stage `vnet_shutdown` that clean up network resources. >> When jail / prison is going to dye, after kernel has cleaned up >> processes it call `vnet_shutdown` to cleanup network resources, >> then vnet_destroy will go smoothly as there's no circular network >> resource dependency right now. >> The life cycle of prison becomes: >> on jail create: PRISON_STATE_INVALID -> create VNET -> >> PRISON_STATE_ALIVE -> setup network resources, ifnet, if addresses, >> routing, etc. -> create / attach (network) processes >> on jail destroy: jexec kill processes (1) by user -> mark it as >> PRISON_STATE_DYING -> send SIGKILL to processes by kernel (2) -> >> vnet_shutdown cleanup network resources -> destroy VNET (if prison >> pr_ref go to the last one) -> DYED >> This idea is still unmature and I hope to hear more voices about it. >=20 > This is absolutely the direction things need to go. Vnet isn't the > only thing that can have these problems, though it's been the biggest > offender. There could also be cycles that involve more than one > subsystem, which could be helped by broad application of this idea. >=20 > There's a function in kern_jail.c ready for this: prison_cleanup. > It's called in "mark PRISON_STATE_DYING" stage of things. That's > before the "send SIGKILL" part of your sequence, but otherwise fits. >=20 Submitted to Phabricator for review: https://reviews.freebsd.org/D37956 https://reviews.freebsd.org/D37957 > - Jamie Best regards, Zhenlei From nobody Fri Jan 6 10:14:06 2023 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NpKDT29qkz2pJXv; Fri, 6 Jan 2023 10:23:21 +0000 (UTC) (envelope-from owner-freebsd-net@freebsd.org) Received: from delivery.e-purifier.com (delivery.e-purifier.com [41.168.131.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4NpKDR6Czxz4410; Fri, 6 Jan 2023 10:23:19 +0000 (UTC) (envelope-from owner-freebsd-net@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=freebsd.org header.s=dkim header.b=ERKRiKJC; spf=softfail (mx1.freebsd.org: 41.168.131.21 is neither permitted nor denied by domain of owner-freebsd-net@freebsd.org) smtp.mailfrom=owner-freebsd-net@freebsd.org; dmarc=none; arc=pass ("freebsd.org:s=dkim:i=1") Received: from [192.168.212.33] (helo=sec-nCPT-ag03) by delivery.e-purifier.com with smtp (Exim 4.95) (envelope-from ) id 1pDjsU-0007PO-4r; Fri, 06 Jan 2023 12:23:10 +0200 Received: from mail pickup service by sec-nCPT-ag03.neotel.e-purifier.co.za with Microsoft SMTPSVC; Fri, 6 Jan 2023 12:23:08 +0200 Received: from sec-ncpt-spt04.e-purifier.com ([192.168.211.1]) by sec-nCPT-ag03.neotel.e-purifier.co.za with Microsoft SMTPSVC(7.5.7601.17514); Fri, 6 Jan 2023 12:14:41 +0200 Received: from localhost (localhost [127.0.0.1]) by sec-ncpt-spt04.e-purifier.com (Postfix) with ESMTP id C63E89DFD75 for ; Fri, 6 Jan 2023 12:14:41 +0200 (SAST) X-Virus-Scanned: by SpamTitan at e-purifier.com X-Spam-Flag: NO X-Spam-Score: 0.697 X-Spam-Level: X-Spam-Status: No, score=0.697 tagged_above=-999 required=4.5 tests=[BAYES_50=0.8, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, DMARC_MISSING=0.1, MAILING_LIST_MULTI=-0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, ST_RCVD_IN_HOSTKARMA_W=-0.001] autolearn=ham autolearn_force=no Received: from sec-ncpt-spt04.e-purifier.com (localhost [127.0.0.1]) by sec-ncpt-spt04.e-purifier.com (Postfix) with ESMTP id 27A4B9DFD7E for ; Fri, 6 Jan 2023 12:14:26 +0200 (SAST) Received-SPF: pass (freebsd.org ... _spf.freebsd.org: 96.47.72.81 is authorized to use 'freebsd-net+bounces-2840-ho=nanoteq.com@FreeBSD.org' in 'mfrom' identity (mechanism 'ip4:96.47.72.81' matched)) receiver=sec-ncpt-spt04.e-purifier.com; identity=mailfrom; envelope-from="freebsd-net+bounces-2840-ho=nanoteq.com@FreeBSD.org"; helo=mx2.freebsd.org; client-ip=96.47.72.81 Received: from mx2.freebsd.org (mx2.freebsd.org [96.47.72.81]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by sec-ncpt-spt04.e-purifier.com (Postfix) with ESMTPS id 928AD9DFD81 for ; Fri, 6 Jan 2023 12:14:20 +0200 (SAST) Received: from mx1.freebsd.org (mx1.freebsd.org [96.47.72.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits)) (Client CN "mx1.freebsd.org", Issuer "R3" (verified OK)) by mx2.freebsd.org (Postfix) with ESMTPS id 4NpK214vFQz41Q4 for ; Fri, 6 Jan 2023 10:14:17 +0000 (UTC) (envelope-from freebsd-net+bounces-2840-ho=nanoteq.com@FreeBSD.org) Received: from mlmmj.nyi.freebsd.org (mlmmj.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:24]) by mx1.freebsd.org (Postfix) with ESMTP id 4NpK213sK3z42BW for ; Fri, 6 Jan 2023 10:14:17 +0000 (UTC) (envelope-from freebsd-net+bounces-2840-ho=nanoteq.com@FreeBSD.org) Received: from mlmmj.nyi.freebsd.org (mlmmj.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:24]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NpK1v4mGyz2pHK8; Fri, 6 Jan 2023 10:14:11 +0000 (UTC) (envelope-from freebsd-net+bounces-2840@FreeBSD.org) X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NpK1v0Yv2z2pHFc; Fri, 6 Jan 2023 10:14:11 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NpK1v05fMz41ZY; Fri, 6 Jan 2023 10:14:11 +0000 (UTC) (envelope-from zlei@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1673000051; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=a4SKPPU3J5HnhI1Ab4inSkxLcICUlb1KdZSLjs276HQ=; b=ERKRiKJCzX/TYm3246pD427H4N8AOosfItpTZh2qCOut7vAwyGqJEkwn7dZtGQ15t2W+JT tx/tUUMY+l8e6hQ5+W0sOIEXi5vv9AcDVh7ce3NcRtvey1cCZEjdW4SuPbaTTQW1k3ieFq mVXy500s1gLhVgUArullzdkMRFF1QUiDitExRnA8XY1zuq3S5RuxKwiY7Lx8L3H6+nYZih xwFO0VuXM9YvisGw6c7gcCJz+Go1sseR/YN7OYhp0wZoORuIIKiQKsTIJ+AMIej4q+Wxq2 Ok1KCxFqWZaGHc/jd1Kge0N6G/1zfhuOfD9UlOBTrV+ykTB0nEVhXYSGFkNH6A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1673000051; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=a4SKPPU3J5HnhI1Ab4inSkxLcICUlb1KdZSLjs276HQ=; b=Qibx3oCrdtMH2qq5ZSK91EXuCiNxw++y2PxfxBAcaauFxO6jZ+KxNnE5nTd1NNWtdQsi2O 3g9m2XAhABusCfDbAupV0YF870j+ghJn03MLKSW3rI8xx7PTK2uchWwpRlSHEb9fqmAlRG pm0rYeukLdy0MA3fTuaOlRkqpoNEB/CKbVZgFX/Lr6gyvZNOTh6mI111K0ITduifTt/ipT yLR25OLkwPmdSpixx3bs5sFzU8XUj3z3StKdnN2U5uUwoZHYnRslWQjBjTTKFoIXHGfmPm gnlcv+2eSnY+CmZSs+2lQdJImzOcFArTHt8ZkTJUFERaOlNgdDpSI6eSFoBKyw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1673000051; a=rsa-sha256; cv=none; b=xnePYWrZA/9Posu9BHNeFCQ6/PmVW5rHTvyb/1zLUC0m+axbUjYuoiuC9pGVxspHPyfft5 1t1cUXIeoikRTQLZKCZiuloeaLIZlO3Y9wacH9AHVQSil9K0nBWZSvnFhsQw81tJ5vO8ry TlSmJI4UBO5uTQjrzmZixUwC5XHQqGXCjK6GoDE6BniGKxX3cWLRFAtJh042S4TgvoAmG7 LQR2riz1fVKvm7NzzTj2xik5UeUVjv8YMs9le4gm1zFK9u4gR4WvvRWsu2rrzTPfTLBE6l cilJRkZpW1A+A93+J8UsGBrUEbraJJrB0pTNRJhAnimIjJAZgtPK3SJFJHF1mg== Received: from smtpclient.apple (unknown [112.66.185.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: zlei/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4NpK1s6KpNz173v; Fri, 6 Jan 2023 10:14:09 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Content-Type: text/plain; charset=us-ascii List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) Subject: Re: Propose a new stage `vnet_shutdown` before `vnet_destroy` From: Zhenlei Huang In-Reply-To: <1c9dbf6d26b9525243dd6b3ffafa23cb@freebsd.org> Date: Fri, 6 Jan 2023 18:14:06 +0800 Cc: freebsd-jail@freebsd.org, freebsd-net Content-Transfer-Encoding: quoted-printable Message-Id: References: <1c9dbf6d26b9525243dd6b3ffafa23cb@freebsd.org> To: James Gritton X-Mailer: Apple Mail (2.3696.120.41.1.1) X-ThisMailContainsUnwantedMimeParts: N X-OriginalArrivalTime: 06 Jan 2023 10:14:41.0927 (UTC) FILETIME=[B12B3170:01D921B7] x-archived: yes x-dbused: RGF0YSBTb3VyY2U9MTkyLjE2OC4yMTEuMjc= X-Spamd-Result: default: False [-5.97 / 15.00]; DWL_DNSWL_MED(-2.00)[freebsd.org:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; ARC_ALLOW(-1.00)[freebsd.org:s=dkim:i=1]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[freebsd.org:s=dkim]; MAILLIST(-0.16)[generic]; MIME_GOOD(-0.10)[text/plain]; HAS_LIST_UNSUB(-0.01)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[freebsd.org]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MLMMJ_DEST(0.00)[freebsd-jail@freebsd.org,freebsd-net@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all:c]; RCVD_COUNT_TWELVE(0.00)[13]; TO_DN_SOME(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[41.168.131.21:from]; DKIM_TRACE(0.00)[freebsd.org:+]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_NEQ_ENVFROM(0.00)[zlei@FreeBSD.org,owner-freebsd-net@freebsd.org]; ASN(0.00)[asn:36937, ipnet:41.168.128.0/17, country:ZA]; FORGED_SENDER_MAILLIST(0.00)[] X-Rspamd-Queue-Id: 4NpKDR6Czxz4410 X-Spamd-Bar: ----- X-ThisMailContainsUnwantedMimeParts: N > On Dec 19, 2022, at 1:44 AM, James Gritton wrote: >=20 > On 2022-12-18 00:01, Zhenlei Huang wrote: >> I'm currently working on route nexthop caching feature for tunneling >> interfaces such as >> if_gif, if_gre, if_vxlan, and potentially if_wg. I encounter a nasty >> bug related to VNET lifecycle. >> More preciously I'd like to call `rib_unsubscribe()` to unsubscribe >> route event when the interface >> tunnel is deleted (gif_delete_tunnel). >> While on VNET shutting down, VNET SYSUNINIT was called and the = routing >> vnet subsystem >> is destroyed before the interface going down and hence cause >> pagefault. I do not want to check >> `vnet.vnet_shutdown` state as it looks messed up. >> I'm recently reviewing the life cycles of prison and get some = inspirations. >> When the jail / prison is submitted to destroy ( by jail_remove >> syscall ) then SIGKILL is sent to >> the prison's processes. I think it is correct order to destroy jail / >> prison. To summarize, the life cycle >> of jail / prison is: >> on jail create: PRISON_STATE_INVALID -> create VNET -> >> PRISON_STATE_ALIVE -> setup network resources, ifnet, if addresses, >> routing, etc. -> create / attach (network) processes >> on jail destroy: jexec kill processes (1) by user -> mark it as >> PRISON_STATE_DYING -> send SIGKILL to processes by kernel (2) -> >> destroy VNET (if prison pr_ref go to the last one) -> DYED >> The (2) is a cleanup by kernel as (1) is possible not done by user. >> So it comes the idea about the life cycle of VNET. >> While on jail destroy, the network resources are cleaned up by >> vnet_destroy ( SYSUNINIT ). Then the >> order of SYSUNINIT of network components is hacking as circular >> network resource dependency is possible. >> For example the routing table entries (nhop) have reference of ifnet, >> and ifnet have reference to route nhop (cache), as >> I encountered. >> Just like the cleanup processes by kernel, we can introduce a new >> stage `vnet_shutdown` that clean up network resources. >> When jail / prison is going to dye, after kernel has cleaned up >> processes it call `vnet_shutdown` to cleanup network resources, >> then vnet_destroy will go smoothly as there's no circular network >> resource dependency right now. >> The life cycle of prison becomes: >> on jail create: PRISON_STATE_INVALID -> create VNET -> >> PRISON_STATE_ALIVE -> setup network resources, ifnet, if addresses, >> routing, etc. -> create / attach (network) processes >> on jail destroy: jexec kill processes (1) by user -> mark it as >> PRISON_STATE_DYING -> send SIGKILL to processes by kernel (2) -> >> vnet_shutdown cleanup network resources -> destroy VNET (if prison >> pr_ref go to the last one) -> DYED >> This idea is still unmature and I hope to hear more voices about it. >=20 > This is absolutely the direction things need to go. Vnet isn't the > only thing that can have these problems, though it's been the biggest > offender. There could also be cycles that involve more than one > subsystem, which could be helped by broad application of this idea. >=20 > There's a function in kern_jail.c ready for this: prison_cleanup. > It's called in "mark PRISON_STATE_DYING" stage of things. That's > before the "send SIGKILL" part of your sequence, but otherwise fits. >=20 Submitted to Phabricator for review: https://reviews.freebsd.org/D37956 https://reviews.freebsd.org/D37957 > - Jamie Best regards, Zhenlei