From nobody Wed Dec 14 01:56:45 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NWz4j12T8z4kQjf for ; Wed, 14 Dec 2022 01:56:53 +0000 (UTC) (envelope-from zlei.huang@gmail.com) Received: from mail-pf1-x429.google.com (mail-pf1-x429.google.com [IPv6:2607:f8b0:4864:20::429]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NWz4h69q4z4CjN; Wed, 14 Dec 2022 01:56:52 +0000 (UTC) (envelope-from zlei.huang@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pf1-x429.google.com with SMTP id 124so3518336pfy.0; Tue, 13 Dec 2022 17:56:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=EO5G41A1XTPGxdZR6Xvcs5IcgfDKITXVnhg/b6kWVKE=; b=duxw+4qaCVzooIx4vLqw/Gi7hEMqwa3UazQYAVOtDJQJygZg6B5HuYM7SJ/SGYfXL5 yXCPf9sQAKfo9lthp++10s+N9Ho9hQ0uHkD3lA0FFu3x4xCB67kL1MaVDjFyT2XvC+M/ 9v0GaSPP9Khe6wo8QMFuX3dkU3FKUAYwTBiRb7OiiK1WCF6KGaaMOdrVH0umOc9wYnzd ypV8oTsWTcNMioJ8FG+2fUD39dC50HvekacZ78uz6AbWCZGkXgbEMpyubszHRQoxj7hK 8J3VVfLdy+Ufph03oXgpisazhQo2To+g3t0S2OgTgrAvy1d8SPRgDAGjc51sQRlYQ7Ze w/1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=EO5G41A1XTPGxdZR6Xvcs5IcgfDKITXVnhg/b6kWVKE=; b=axIz9PypTRANpK0qNGeKHkw9wNB+zIIM7BKUVIBH39A+BMPQbSLzUvyhK3cFxl1NEP VHsMkHrIDPMCazqNn4ECwujewMlxsOmNYo+/15C5KzdEBIWiLGFSDRO2npr8Oi7LDqZ8 FCRyFzes/rD/R4ZJ+P+YS4AHUN7yptDkf7D8wmZDuVqS0gsSfBEMnJMhiercQH8dG8it lKCMuyUQ9pt4LfsJavpxyfQzKIOLc3nf6MnMN0C0Iob7Dlq09NhpPgQJEih4mxVke4p5 RQ1wIcHVDje6BO00l9MRyMbkbf8BslbnFFgL93ZyaC2UKZeUBc+2naM0EErfi1HsvOG7 1pow== X-Gm-Message-State: ANoB5pmnoI1x6khw/4dcZiinWTR0dL3x4dbtoq9QkZIxQByAqlyvTjN7 BhFv/oWWBYY0ABabu4OsEGv3sZH8pvw= X-Google-Smtp-Source: AA0mqf4PM8WspQM/BfiUqLJVNXvZIhcMXc4jI8PcaQ4ym1SW8+iLIXoYCvC8UxZb9Rkk5WlW/ZxWZQ== X-Received: by 2002:a62:ee14:0:b0:566:900d:6073 with SMTP id e20-20020a62ee14000000b00566900d6073mr21838670pfi.24.1670983011131; Tue, 13 Dec 2022 17:56:51 -0800 (PST) Received: from ?IPv6:fd03:a1b3:1819::8001? ([2001:19f0:6001:9db:98f0:9fe0:3545:10]) by smtp.gmail.com with ESMTPSA id p16-20020aa79e90000000b0057691fb0d37sm8188991pfq.193.2022.12.13.17.56.49 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 13 Dec 2022 17:56:50 -0800 (PST) From: Zhenlei Huang X-Google-Original-From: Zhenlei Huang Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_974C8467-6D6D-4A08-9719-957ECFA39962" List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.7\)) Subject: Re: What's going on with vnets and epairs w/ addresses? Date: Wed, 14 Dec 2022 09:56:45 +0800 In-Reply-To: <5r22os7n-ro15-27q-r356-rps331o06so5@mnoonqbm.arg> Cc: "freebsd-jail@freebsd.org" To: "Bjoern A. Zeeb" References: <5r22os7n-ro15-27q-r356-rps331o06so5@mnoonqbm.arg> X-Mailer: Apple Mail (2.3608.120.23.2.7) X-Rspamd-Queue-Id: 4NWz4h69q4z4CjN X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_974C8467-6D6D-4A08-9719-957ECFA39962 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi, I also encounter this problem while testing gif tunnel between jails. My script is similar but with additional gif tunnels. There are reports in mailing list [1], [2], and another one in forum [3] = . Seem to be a long standing issue. [1] = https://lists.freebsd.org/pipermail/freebsd-stable/2016-October/086126.htm= l = [2] = https://lists.freebsd.org/pipermail/freebsd-jail/2017-March/003357.html = [3] = https://forums.freebsd.org/threads/jails-stopping-prolonged-deaths-startin= g-networking-et-cetera.84200/ = Best regards, Zhenlei > On Dec 14, 2022, at 7:03 AM, Bjoern A. Zeeb wrote: >=20 > Hi, >=20 > I have used scripts like the below for almost a decade and a half > (obviously doing more than that in the middle). I haven't used them > much lately but given other questions I just wanted to fire up a test. >=20 > I have an end-November kernel doing the below my eapirs do not come = back > to be destroyed (immediately). > I have to start polling for the jid to be no longer alive and not in > dying state (hence added the jls/ifconfig -l lines and removed the > error checking from ifconfig destroy). That seems sometimes rather > unreasonably long (to the point I give up). >=20 > If I don't configure the addresses below this isn't a problem. >=20 > Sorry I am confused by too many incarnations of the code; I know I = once > had a version with an async shutdown path but I believe that never = made > it into mainline, so why are we holding onto the epairs now and not > nuking the addresses and returning them and are clean? >=20 > It's a bit more funny; I added a twiddle loop at the end and nothing > happened. So I stop the script and start it again and suddenly = another > jail or two have cleaned up and their epairs are back. Something = feels > very very wonky. Play around with this and see ... and let me know if > you can reproduce this... I quite wonder why some test cases haven't > gone crazy ... >=20 > /bz >=20 > = ------------------------------------------------------------------------ > #!/bin/sh >=20 > set -e > set -x >=20 > js=3D`jail -i -c -n jl host.hostname=3Dleft.example.net vnet persist` > jb=3D`jail -i -c -n jr host.hostname=3Dright.example.net vnet persist` >=20 > # Create an epair connecting the two machines (vnet jails). > ep=3D`ifconfig epair create | sed -e 's/a$//'` >=20 > # Add one end to each vnet jail. > ifconfig ${ep}a vnet ${js} > ifconfig ${ep}b vnet ${jb} >=20 > # Add an IP address on the epairs in each vnet jail. > # XXX Leave these out and the cleanup seems to work fine. > jexec ${js} ifconfig ${ep}a inet 192.0.2.1/24 > jexec ${jb} ifconfig ${ep}b inet 192.0.2.2/24 >=20 > # Clean up. > jail -r ${jb} > jail -r ${js} >=20 > # You want to be able to remove this line ... > set +e >=20 > # No epairs to destroy with addresses configured; fine otherwise. > ifconfig ${ep}a destroy > # echo $? >=20 > # Add this is here only as things are funny ... > # jls -av jid dying > # ifconfig -l >=20 > # end > = ------------------------------------------------------------------------ >=20 > --=20 > Bjoern A. Zeeb = r15:7 >=20 --Apple-Mail=_974C8467-6D6D-4A08-9719-957ECFA39962 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
Hi,

I also encounter this = problem while testing gif tunnel between jails.

My script is similar but with additional gif = tunnels.


There are reports in mailing list [1], [2], and = another one in forum [3] .

Seem to = be a long standing issue.



Best regards,
Zhenlei

On Dec 14, 2022, at 7:03 AM, Bjoern A. Zeeb <bz@FreeBSD.org> = wrote:

Hi,

I have used scripts like the = below for almost a decade and a half
(obviously doing more = than that in the middle).  I haven't used them
much = lately but given other questions I just wanted to fire up a test.

I have an end-November kernel doing the below = my eapirs do not come back
to be destroyed = (immediately).
I have to start polling for the jid to be = no longer alive and not in
dying state (hence added the = jls/ifconfig -l lines and removed the
error checking from = ifconfig destroy).  That seems sometimes rather
unreasonably long (to the point I give up).

If I don't configure the addresses below this isn't a = problem.

Sorry I am confused by too many = incarnations of the code; I know I once
had a version with = an async shutdown path but I believe that never made
it = into mainline, so why are we holding onto the epairs now and not
nuking the addresses and returning them and are clean?

It's a bit more funny; I added a twiddle loop = at the end and nothing
happened.  So I stop the = script and start it again and suddenly another
jail or two = have cleaned up and their epairs are back.  Something feels
very very wonky.  Play around with this and see ... and = let me know if
you can reproduce this...  I quite = wonder why some test cases haven't
gone crazy ...

/bz

---------------------------------------------------------------= ---------
#!/bin/sh

set -e
set -x

js=3D`jail -i -c -n jl = host.hostname=3Dleft.example.net vnet persist`
jb=3D`jail = -i -c -n jr host.hostname=3Dright.example.net vnet persist`

# Create an epair connecting the two machines (vnet = jails).
ep=3D`ifconfig epair create | sed -e 's/a$//'`

# Add one end to each vnet jail.
ifconfig ${ep}a vnet ${js}
ifconfig ${ep}b vnet = ${jb}

# Add an IP address on the epairs in = each vnet jail.
# XXX Leave these out and the cleanup = seems to work fine.
jexec ${js}  ifconfig ${ep}a inet =  192.0.2.1/24
jexec ${jb}  ifconfig ${ep}b inet =  192.0.2.2/24

# Clean up.
jail -r ${jb}
jail -r ${js}

# You want to be able to remove this line ...
set= +e

# No epairs to destroy with addresses = configured; fine otherwise.
ifconfig ${ep}a destroy
# echo $?

# Add this is here = only as things are funny ...
# jls -av jid dying
# ifconfig -l

# end
---------------------------------------------------------------= ---------

--
Bjoern A. Zeeb =             &n= bsp;           &nbs= p;            =             &n= bsp;  r15:7


= --Apple-Mail=_974C8467-6D6D-4A08-9719-957ECFA39962--