From nobody Fri Dec 16 22:55:41 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NYkwM6DDRzsxP9 for ; Fri, 16 Dec 2022 22:55:47 +0000 (UTC) (envelope-from bz@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NYkwM5CTrz4GS8; Fri, 16 Dec 2022 22:55:47 +0000 (UTC) (envelope-from bz@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671231347; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=1PFo9vDfTBLn6qxwOf4dc5+A+0gWHaR8ub5jUTEWdFc=; b=kFWClTyajvr5oGWu+Zp6Nk1haqPDJsKs+j1EHIzEpem+ypsGxUKZiJ1VYwZ7T7OH4udX7R ey4kIAk1OXhflMtnfpyoVh9hJyE5/4yoRXX8jxsyPuH/r+bNXSi/RNHWhtk0jzWS2se/Hq uzJWvqsFFOLrwfS5XMhMu3B3gJrRRaAYvTEMw1Sr7LNm9C8WO35CB5kLYNZJKM7wZXYKyK 1zZYr+CRAYNlTVmxIRG7l9Gtf0E5hwKub1gmqdjulX9RvxpC94FRSkej+XSKOejbGj+maa FLD9Pe0clO14OI3KDJj+juX1NEiXXNPLsshUrbivgGoEfFaFdRm3ERJ/o0eonQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671231347; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=1PFo9vDfTBLn6qxwOf4dc5+A+0gWHaR8ub5jUTEWdFc=; b=EQg8noYDRn7y7Mr1/Rh3f70aLRa13gh5suhBC4FLR6FQkkxbH1QtrI3/CG/qmhkeafR4jK 44vVpIljYgPLLBhLigQmgXWQCqL7jwQYEHoHFSc+OXTxnr06vmYfgUgquBexaAihnEt4Vd zS2dx9NrpK5+5VN91g96Z88OjuIWcwTDBonbu01GcvMoG6fV3NUeZrWTCAdCN5hyZo3z8e RO4KOrJSjDQ3gF6GJmw3J1nofvGs5SNIJi9/zR9Foq2Fo1VrJuvV2GI0vRQvEjpI0i4Fcq dFp+pBujvkNAImrd9PKc+fCtDcJjutDYOkUNyjSaJgyUdGJeK2RscJG2kdSWUA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1671231347; a=rsa-sha256; cv=none; b=aESj0latZ9QRXsdb9JZvyA/ud06iXafHNjiV+ie55aPrFirm2o1q8JxgWMqbqpc2G6l5p9 N5jPwbrt6LzQDlAFvXtTABcPBvzf9wHO298tLRL7ylYJlatJBPbjTdB9ARoxDj+8GQza/G MpidleITXxhrcyP1y0Tiea1z8hYhyl83o3rC2zuXo27e75/arC/GxpMyTCFWVX/otx7rUl A6smYKAVXkYa0Yc4r2cG+Ld2HykY3DkKuqwKI0srJPZXIHlooCnkdjJ/P9wOC/Sw5Y4FAc CJPpv83Mqcgq2tSLcREnAAvDhzXJLH5P3VNez2LzNeVFjOm+4t62OyW/5zBOxg== Received: from mx1.sbone.de (cross.sbone.de [195.201.62.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) (Authenticated sender: bz/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4NYkwM1lKDzlQs; Fri, 16 Dec 2022 22:55:47 +0000 (UTC) (envelope-from bz@FreeBSD.org) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id D61CF8D4A228; Fri, 16 Dec 2022 22:55:45 +0000 (UTC) Received: from content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id D134A5C3A833; Fri, 16 Dec 2022 22:55:44 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) by content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (amavisd-new, port 10024) with ESMTP id ACn2vO6Lrzio; Fri, 16 Dec 2022 22:55:42 +0000 (UTC) Received: from strong-iwl0.sbone.de (strong-iwl0.sbone.de [IPv6:fde9:577b:c1a9:4902:b66b:fcff:fef3:e3d2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id 58AA15C3A830; Fri, 16 Dec 2022 22:55:42 +0000 (UTC) Date: Fri, 16 Dec 2022 22:55:41 +0000 (UTC) From: "Bjoern A. Zeeb" To: Zhenlei Huang cc: "freebsd-jail@freebsd.org" , Gleb Smirnoff Subject: Re: What's going on with vnets and epairs w/ addresses? In-Reply-To: <150A60D6-6757-46DD-988F-05A9FFA36821@FreeBSD.org> Message-ID: <1348s3p2-783s-sno2-pp6-rs9oq0s921n@SerrOFQ.bet> References: <5r22os7n-ro15-27q-r356-rps331o06so5@mnoonqbm.arg> <150A60D6-6757-46DD-988F-05A9FFA36821@FreeBSD.org> X-OpenPGP-Key-Id: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-ThisMailContainsUnwantedMimeParts: N On Fri, 16 Dec 2022, Zhenlei Huang wrote: Hi, > I managed to repeat this issue on CURRENT/14 with this small snip: > > ------------------------------------------- > #!/bin/sh > > # test jail name > n="test_ref_leak" > > jail -c name=$n path=/ vnet persist > # The following line trigger jail pr_ref leak > jexec $n ifconfig lo0 inet 127.0.0.1/8 > > jail -R $n > > # wait a moment > sleep 1 > > jls -j $n > > > ------------------------------------------- > > > After DDB debugging and tracing , it seems that is triggered by a combine of [1] and [2] > > [1] https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915 > [2] https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b > > > In [1] the per-VNET uma zone is shared with the global one. > `pcbinfo->ipi_zone = pcbstor->ips_zone;` > > In [2] unref `inp->inp_cred` is deferred called in inpcb_dtor() by uma_zfree_smr() . > > Unfortunately inps freed by uma_zfree_smr() are cached and inpcb_dtor() is not called immediately , > thus leaking `inp->inp_cred` ref and hence `prison->pr_ref`. > > And it is also not possible to free up the cache by per-VNET SYSUNINIT tcp_destroy / udp_destroy / rip_destroy. Thanks a lot for tracking it down. That seems to be a regression then that needs to be fixed before 14.0-RELEASE will happen as it'll break management utilities of people. Could you open a bug report and flag it as such? /bz > > > Best regards, > Zhenlei > >> On Dec 14, 2022, at 9:56 AM, Zhenlei Huang wrote: >> >> >> Hi, >> >> I also encounter this problem while testing gif tunnel between jails. >> >> My script is similar but with additional gif tunnels. >> >> >> There are reports in mailing list [1], [2], and another one in forum [3] . >> >> Seem to be a long standing issue. >> >> [1] https://lists.freebsd.org/pipermail/freebsd-stable/2016-October/086126.html >> [2] https://lists.freebsd.org/pipermail/freebsd-jail/2017-March/003357.html >> [3] https://forums.freebsd.org/threads/jails-stopping-prolonged-deaths-starting-networking-et-cetera.84200/ >> >> >> Best regards, >> Zhenlei >> >>> On Dec 14, 2022, at 7:03 AM, Bjoern A. Zeeb > wrote: >>> >>> Hi, >>> >>> I have used scripts like the below for almost a decade and a half >>> (obviously doing more than that in the middle). I haven't used them >>> much lately but given other questions I just wanted to fire up a test. >>> >>> I have an end-November kernel doing the below my eapirs do not come back >>> to be destroyed (immediately). >>> I have to start polling for the jid to be no longer alive and not in >>> dying state (hence added the jls/ifconfig -l lines and removed the >>> error checking from ifconfig destroy). That seems sometimes rather >>> unreasonably long (to the point I give up). >>> >>> If I don't configure the addresses below this isn't a problem. >>> >>> Sorry I am confused by too many incarnations of the code; I know I once >>> had a version with an async shutdown path but I believe that never made >>> it into mainline, so why are we holding onto the epairs now and not >>> nuking the addresses and returning them and are clean? >>> >>> It's a bit more funny; I added a twiddle loop at the end and nothing >>> happened. So I stop the script and start it again and suddenly another >>> jail or two have cleaned up and their epairs are back. Something feels >>> very very wonky. Play around with this and see ... and let me know if >>> you can reproduce this... I quite wonder why some test cases haven't >>> gone crazy ... >>> >>> /bz >>> >>> ------------------------------------------------------------------------ >>> #!/bin/sh >>> >>> set -e >>> set -x >>> >>> js=`jail -i -c -n jl host.hostname=left.example.net vnet persist` >>> jb=`jail -i -c -n jr host.hostname=right.example.net vnet persist` >>> >>> # Create an epair connecting the two machines (vnet jails). >>> ep=`ifconfig epair create | sed -e 's/a$//'` >>> >>> # Add one end to each vnet jail. >>> ifconfig ${ep}a vnet ${js} >>> ifconfig ${ep}b vnet ${jb} >>> >>> # Add an IP address on the epairs in each vnet jail. >>> # XXX Leave these out and the cleanup seems to work fine. >>> jexec ${js} ifconfig ${ep}a inet 192.0.2.1/24 >>> jexec ${jb} ifconfig ${ep}b inet 192.0.2.2/24 >>> >>> # Clean up. >>> jail -r ${jb} >>> jail -r ${js} >>> >>> # You want to be able to remove this line ... >>> set +e >>> >>> # No epairs to destroy with addresses configured; fine otherwise. >>> ifconfig ${ep}a destroy >>> # echo $? >>> >>> # Add this is here only as things are funny ... >>> # jls -av jid dying >>> # ifconfig -l >>> >>> # end >>> ------------------------------------------------------------------------ >>> >>> -- >>> Bjoern A. Zeeb r15:7 >>> >> > > -- Bjoern A. Zeeb r15:7