From nobody Sat Dec 17 13:36:10 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NZ6SZ3RN2z1ChSh for ; Sat, 17 Dec 2022 13:36:30 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NZ6SZ29FHz3FLL; Sat, 17 Dec 2022 13:36:30 +0000 (UTC) (envelope-from zlei@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671284190; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=1O+TSWc81MpqO/+CeGpBQK+6egGKFOcpGepOT5GAMUo=; b=fKzGxyqOn8icYn0puanjFlW6/t+SnB664umgNbPLdfsZJpdPmowBZ+sjP75mnNM1VeN/n2 sm1RhbJXyHo/SMmt5Z0a7svY3v3w7NNIW+Z81zdScg5wICKqOP6iA30bBRE0yTp7cUWRo3 HcEboJdDt2W+zEdkNH+qJUzaVw1c/eY2kegd/OcCQGoVwLnfS3oYd9Ohta+/BMuFhGg/tY KzEpWth2/Et3CjNr8/ilVQAw57HPJXs/dafdSfy6DlNB7ucnAt8mkHQ3dK0ujMkgLkr5vu u4PTnS4ECh+/AG97ncJGZ+rE4KVe6ARxfeKlm/HGnxvUFQ6+Q48AUdM9mTEr3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671284190; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=1O+TSWc81MpqO/+CeGpBQK+6egGKFOcpGepOT5GAMUo=; b=eqHE675QGkV4P9x0DPWZATiTTRTWj+mCX02B8Z0xAHX9Kr/vhOgoYvAaQwvr6dEdzJHF5I An9BTOPxTZkYJIVjEek6AjMSp4zmNW56qbbZhQWmVQ7+8pU2pHBbkyBs/RRxAck0RgOls4 PDwvmZameFBtHzbkF3LHEkMCuB+bYQLzrwLCJ5rydBFtF//M/6cycPxwm+bp8H2tLtX616 OkrgJhEAQfYVMIMIauP++MeuA7ZBmq53/uaFL0+CcVaoHQSB6D0ljmfQthxEMjlWCIlX4u 9LYmv2ZkXPjDejbVUGWRNNsWEHHufR0DGyD9iruORJW699QS+eQoJqp7abfkDA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1671284190; a=rsa-sha256; cv=none; b=uwUtAQ57f9dGrK4gOrudZw3Iu/71t3gO9pCkY/PPOKdyvAmyEzLeMSNvyL4wdZ+TMUaQYV MtdJjHXjSI0bPb7OZPhwiHeyfPawvUrBWUsnsLp4kJcHMlTyE9IgFz3gz6fFVzK0K+8D2t ZEDogUobeLndCLxwdi+tgmJN8HmbT/Oywxos5UgT2LgXL8z3f4CeUDX7+Lk1YwXvyJue4/ PMilM6UdUBiChSEy6icuqQSrqytqjAkCwuhkmiI2ochOCXv/cRmW9GbqXVqCdwrTWFytln MO/tUP67NM9uW6fcQDuATc5Aa3uurNr7QcGpHuGN+7RYolDnvk7+hGU9niz/3Q== Received: from [IPv6:2409:8a5e:aa71:20f8:40a:ad36:f9f7:a514] (unknown [IPv6:2409:8a5e:aa71:20f8:40a:ad36:f9f7:a514]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: zlei/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4NZ6SW481dz13xg; Sat, 17 Dec 2022 13:36:26 +0000 (UTC) (envelope-from zlei@FreeBSD.org) From: Zhenlei Huang Message-Id: <5F5B4BAD-156F-43E9-80FC-01CB5C9F8741@FreeBSD.org> Content-Type: multipart/alternative; boundary="Apple-Mail=_7038B4AC-00A7-4FE5-8E97-A1D0C96818EB" List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.7\)) Subject: Re: What's going on with vnets and epairs w/ addresses? Date: Sat, 17 Dec 2022 21:36:10 +0800 In-Reply-To: <1348s3p2-783s-sno2-pp6-rs9oq0s921n@SerrOFQ.bet> Cc: "freebsd-jail@freebsd.org" , Gleb Smirnoff To: "Bjoern A. Zeeb" References: <5r22os7n-ro15-27q-r356-rps331o06so5@mnoonqbm.arg> <150A60D6-6757-46DD-988F-05A9FFA36821@FreeBSD.org> <1348s3p2-783s-sno2-pp6-rs9oq0s921n@SerrOFQ.bet> X-Mailer: Apple Mail (2.3608.120.23.2.7) X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_7038B4AC-00A7-4FE5-8E97-A1D0C96818EB Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On Dec 17, 2022, at 6:55 AM, Bjoern A. Zeeb > wrote: >=20 > On Fri, 16 Dec 2022, Zhenlei Huang wrote: >=20 > Hi, >=20 >> I managed to repeat this issue on CURRENT/14 with this small snip: >>=20 >> ------------------------------------------- >> #!/bin/sh >>=20 >> # test jail name >> n=3D"test_ref_leak" >>=20 >> jail -c name=3D$n path=3D/ vnet persist >> # The following line trigger jail pr_ref leak >> jexec $n ifconfig lo0 inet 127.0.0.1/8 >>=20 >> jail -R $n >>=20 >> # wait a moment >> sleep 1 >>=20 >> jls -j $n >>=20 >>=20 >> ------------------------------------------- >>=20 >>=20 >> After DDB debugging and tracing , it seems that is triggered by a = combine of [1] and [2] >>=20 >> [1] = https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915 = > >> [2] = https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b = > >>=20 >>=20 >> In [1] the per-VNET uma zone is shared with the global one. >> `pcbinfo->ipi_zone =3D pcbstor->ips_zone;` >>=20 >> In [2] unref `inp->inp_cred` is deferred called in inpcb_dtor() by = uma_zfree_smr() . >>=20 >> Unfortunately inps freed by uma_zfree_smr() are cached and = inpcb_dtor() is not called immediately , >> thus leaking `inp->inp_cred` ref and hence `prison->pr_ref`. >>=20 >> And it is also not possible to free up the cache by per-VNET = SYSUNINIT tcp_destroy / udp_destroy / rip_destroy. >=20 > Thanks a lot for tracking it down. >=20 > That seems to be a regression then that needs to be fixed before > 14.0-RELEASE will happen as it'll break management utilities of = people. >=20 > Could you open a bug report and flag it as such? While I was trying to open a new bug report Bugzilla prompt an existing = PR https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D264981 = =20 opened by olivier. I think this issue is same with that one. >=20 > /bz >=20 >=20 >>=20 >>=20 >> Best regards, >> Zhenlei >>=20 >>> On Dec 14, 2022, at 9:56 AM, Zhenlei Huang > wrote: >>>=20 >>>=20 >>> Hi, >>>=20 >>> I also encounter this problem while testing gif tunnel between = jails. >>>=20 >>> My script is similar but with additional gif tunnels. >>>=20 >>>=20 >>> There are reports in mailing list [1], [2], and another one in forum = [3] . >>>=20 >>> Seem to be a long standing issue. >>>=20 >>> [1] = https://lists.freebsd.org/pipermail/freebsd-stable/2016-October/086126.htm= l = > >>> [2] = https://lists.freebsd.org/pipermail/freebsd-jail/2017-March/003357.html = = >= >>> [3] = https://forums.freebsd.org/threads/jails-stopping-prolonged-deaths-startin= g-networking-et-cetera.84200/ = > >>>=20 >>>=20 >>> Best regards, >>> Zhenlei >>>=20 >>>> On Dec 14, 2022, at 7:03 AM, Bjoern A. Zeeb >> = wrote: >>>>=20 >>>> Hi, >>>>=20 >>>> I have used scripts like the below for almost a decade and a half >>>> (obviously doing more than that in the middle). I haven't used = them >>>> much lately but given other questions I just wanted to fire up a = test. >>>>=20 >>>> I have an end-November kernel doing the below my eapirs do not come = back >>>> to be destroyed (immediately). >>>> I have to start polling for the jid to be no longer alive and not = in >>>> dying state (hence added the jls/ifconfig -l lines and removed the >>>> error checking from ifconfig destroy). That seems sometimes rather >>>> unreasonably long (to the point I give up). >>>>=20 >>>> If I don't configure the addresses below this isn't a problem. >>>>=20 >>>> Sorry I am confused by too many incarnations of the code; I know I = once >>>> had a version with an async shutdown path but I believe that never = made >>>> it into mainline, so why are we holding onto the epairs now and not >>>> nuking the addresses and returning them and are clean? >>>>=20 >>>> It's a bit more funny; I added a twiddle loop at the end and = nothing >>>> happened. So I stop the script and start it again and suddenly = another >>>> jail or two have cleaned up and their epairs are back. Something = feels >>>> very very wonky. Play around with this and see ... and let me know = if >>>> you can reproduce this... I quite wonder why some test cases = haven't >>>> gone crazy ... >>>>=20 >>>> /bz >>>>=20 >>>> = ------------------------------------------------------------------------ >>>> #!/bin/sh >>>>=20 >>>> set -e >>>> set -x >>>>=20 >>>> js=3D`jail -i -c -n jl host.hostname=3Dleft.example.net = > vnet persist` >>>> jb=3D`jail -i -c -n jr host.hostname=3Dright.example.net = > vnet persist` >>>>=20 >>>> # Create an epair connecting the two machines (vnet jails). >>>> ep=3D`ifconfig epair create | sed -e 's/a$//'` >>>>=20 >>>> # Add one end to each vnet jail. >>>> ifconfig ${ep}a vnet ${js} >>>> ifconfig ${ep}b vnet ${jb} >>>>=20 >>>> # Add an IP address on the epairs in each vnet jail. >>>> # XXX Leave these out and the cleanup seems to work fine. >>>> jexec ${js} ifconfig ${ep}a inet 192.0.2.1/24 >>>> jexec ${jb} ifconfig ${ep}b inet 192.0.2.2/24 >>>>=20 >>>> # Clean up. >>>> jail -r ${jb} >>>> jail -r ${js} >>>>=20 >>>> # You want to be able to remove this line ... >>>> set +e >>>>=20 >>>> # No epairs to destroy with addresses configured; fine otherwise. >>>> ifconfig ${ep}a destroy >>>> # echo $? >>>>=20 >>>> # Add this is here only as things are funny ... >>>> # jls -av jid dying >>>> # ifconfig -l >>>>=20 >>>> # end >>>> = ------------------------------------------------------------------------ >>>>=20 >>>> -- >>>> Bjoern A. Zeeb = r15:7 >>>>=20 >>>=20 >>=20 >>=20 >=20 > --=20 > Bjoern A. Zeeb = r15:7 --Apple-Mail=_7038B4AC-00A7-4FE5-8E97-A1D0C96818EB Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii

On = Dec 17, 2022, at 6:55 AM, Bjoern A. Zeeb <bz@FreeBSD.org> = wrote:

On Fri, 16 Dec 2022, Zhenlei = Huang wrote:

Hi,

I = managed to repeat this issue on CURRENT/14 with this small snip:

-------------------------------------------
#!/bin/sh

# test jail name
n=3D"test_ref_leak"

jail -c = name=3D$n path=3D/ vnet persist
# The following line = trigger jail pr_ref leak
jexec $n ifconfig lo0 inet = 127.0.0.1/8

jail -R $n

# wait a moment
sleep 1

jls -j $n


-------------------------------------------


After DDB debugging and tracing , it seems = that is triggered by a combine of [1] and [2]

[1] https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5a= c895915<https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5a= c895915>
[2] https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b1= 75b8c5b<https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b1= 75b8c5b>


In [1] the = per-VNET uma zone is shared with the global one.
`pcbinfo->ipi_zone =3D pcbstor->ips_zone;`

In [2] unref `inp->inp_cred` is deferred = called in inpcb_dtor() by uma_zfree_smr() .

Unfortunately inps freed by uma_zfree_smr() are cached and = inpcb_dtor() is not called immediately ,
thus leaking = `inp->inp_cred` ref and hence `prison->pr_ref`.

And it is also not possible to free up the cache by per-VNET = SYSUNINIT tcp_destroy / udp_destroy / rip_destroy.

Thanks a lot for tracking it down.

That seems to be a regression = then that needs to be fixed before
14.0-RELEASE will happen as it'll break management utilities = of people.

Could you = open a bug report and flag it as such?

While I was trying to open a new bug = report Bugzilla prompt an existing PR https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D264981&= nbsp;
opened by  olivier. I think this issue = is same with that one.


/bz




Best = regards,
Zhenlei

On Dec 14, 2022, at 9:56 AM, Zhenlei Huang = <zlei@FreeBSD.org> wrote:


Hi,

I also = encounter this problem while testing gif tunnel between jails.

My script is similar but with additional gif = tunnels.


There are reports = in mailing list [1], [2], and another one in forum [3] .
Seem to be a long standing issue.

[1] https://lists.freebsd.org/pipermail/freebsd-stable/2016-October= /086126.html<https://lists.freebsd.org/pipermail/freebsd-stable/2016-October= /086126.html>
[2] https://lists.freebsd.org/pipermail/freebsd-jail/2017-March/003= 357.html <https://lists.freebsd.org/pipermail/freebsd-jail/2017-March/003= 357.html>
[3] https://forums.freebsd.org/threads/jails-stopping-prolonged-dea= ths-starting-networking-et-cetera.84200/<https://forums.freebsd.org/threads/jails-stopping-prolonged-dea= ths-starting-networking-et-cetera.84200/>


Best regards,
Zhenlei

On Dec = 14, 2022, at 7:03 AM, Bjoern A. Zeeb <bz@FreeBSD.org <mailto:bz@FreeBSD.org>> wrote:

Hi,

I have used scripts like the = below for almost a decade and a half
(obviously doing more = than that in the middle).  I haven't used them
much = lately but given other questions I just wanted to fire up a test.

I have an end-November kernel doing the below = my eapirs do not come back
to be destroyed = (immediately).
I have to start polling for the jid to be = no longer alive and not in
dying state (hence added the = jls/ifconfig -l lines and removed the
error checking from = ifconfig destroy).  That seems sometimes rather
unreasonably long (to the point I give up).

If I don't configure the addresses below this isn't a = problem.

Sorry I am confused by too many = incarnations of the code; I know I once
had a version with = an async shutdown path but I believe that never made
it = into mainline, so why are we holding onto the epairs now and not
nuking the addresses and returning them and are clean?

It's a bit more funny; I added a twiddle loop = at the end and nothing
happened.  So I stop the = script and start it again and suddenly another
jail or two = have cleaned up and their epairs are back.  Something feels
very very wonky.  Play around with this and see ... and = let me know if
you can reproduce this...  I quite = wonder why some test cases haven't
gone crazy ...

/bz

---------------------------------------------------------------= ---------
#!/bin/sh

set -e
set -x

js=3D`jail -i -c -n jl = host.hostname=3Dleft.example.net <http://left.example.net/> vnet persist`
jb=3D`jail -i -c -n jr host.hostname=3Dright.example.net <http://right.example.net/> vnet persist`

# Create an epair connecting the two machines = (vnet jails).
ep=3D`ifconfig epair create | sed -e = 's/a$//'`

# Add one end to each vnet = jail.
ifconfig ${ep}a vnet ${js}
ifconfig = ${ep}b vnet ${jb}

# Add an IP address on = the epairs in each vnet jail.
# XXX Leave these out and = the cleanup seems to work fine.
jexec ${js}  ifconfig = ${ep}a inet  192.0.2.1/24
jexec ${jb}  ifconfig = ${ep}b inet  192.0.2.2/24

# Clean = up.
jail -r ${jb}
jail -r ${js}

# You want to be able to remove this line = ...
set +e

# No epairs to = destroy with addresses configured; fine otherwise.
ifconfig = ${ep}a destroy
# echo $?

# = Add this is here only as things are funny ...
# jls -av = jid dying
# ifconfig -l

# = end
---------------------------------------------------------------= ---------

--
Bjoern A. Zeeb =             &n= bsp;           &nbs= p;            =             &n= bsp;  r15:7





-- 
Bjoern A. Zeeb =             &n= bsp;           &nbs= p;            =             &n= bsp;  r15:7

= --Apple-Mail=_7038B4AC-00A7-4FE5-8E97-A1D0C96818EB--