Date: Thu, 22 Dec 2022 23:38:07 +0800 From: Zhenlei Huang <zlei@FreeBSD.org> To: Mark Johnston <markj@freebsd.org> Cc: Kyle Evans <kevans@freebsd.org>, Gleb Smirnoff <glebius@freebsd.org>, "Bjoern A. Zeeb" <bz@freebsd.org>, "freebsd-jail@freebsd.org" <freebsd-jail@freebsd.org> Subject: Re: What's going on with vnets and epairs w/ addresses? Message-ID: <069FE014-1D62-4479-B830-9FBDF36F2109@FreeBSD.org> In-Reply-To: <Y6He6OD6PA0ntoK9@nuc> References: <5r22os7n-ro15-27q-r356-rps331o06so5@mnoonqbm.arg> <B6C70A88-11F8-40D7-85E4-63BBA0F7931D@FreeBSD.org> <150A60D6-6757-46DD-988F-05A9FFA36821@FreeBSD.org> <Y534qgEG1nX5i1iB@FreeBSD.org> <CACNAnaE6UQB=zNBjVNrF%2Bpd%2Bmh=6H0%2BROYf1%2BD=HKBTp8aX27g@mail.gmail.com> <Y6He6OD6PA0ntoK9@nuc>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_05999826-0A4B-4FD5-A20A-0ED8E067BC1C Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii >=20 > On Dec 21, 2022, at 12:12 AM, Mark Johnston <markj@freebsd.org = <mailto:markj@freebsd.org>> wrote: >=20 > On Sun, Dec 18, 2022 at 10:52:58AM -0600, Kyle Evans wrote: >> On Sat, Dec 17, 2022 at 11:22 AM Gleb Smirnoff <glebius@freebsd.org = <mailto:glebius@freebsd.org>> wrote: >>>=20 >>> Zhenlei, >>>=20 >>> On Fri, Dec 16, 2022 at 06:30:57PM +0800, Zhenlei Huang wrote: >>> Z> I managed to repeat this issue on CURRENT/14 with this small = snip: >>> Z> >>> Z> ------------------------------------------- >>> Z> #!/bin/sh >>> Z> >>> Z> # test jail name >>> Z> n=3D"test_ref_leak" >>> Z> >>> Z> jail -c name=3D$n path=3D/ vnet persist >>> Z> # The following line trigger jail pr_ref leak >>> Z> jexec $n ifconfig lo0 inet 127.0.0.1/8 >>> Z> >>> Z> jail -R $n >>> Z> >>> Z> # wait a moment >>> Z> sleep 1 >>> Z> >>> Z> jls -j $n >>> Z> >>> Z> After DDB debugging and tracing , it seems that is triggered by a = combine of [1] and [2] >>> Z> >>> Z> [1] = https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915 = <https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915>; = <https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915 = <https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915>>; >>> Z> [2] = https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b = <https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b>; = <https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b = <https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b>>; >>> Z> >>> Z> >>> Z> In [1] the per-VNET uma zone is shared with the global one. >>> Z> `pcbinfo->ipi_zone =3D pcbstor->ips_zone;` >>> Z> >>> Z> In [2] unref `inp->inp_cred` is deferred called in inpcb_dtor() = by uma_zfree_smr() . >>> Z> >>> Z> Unfortunately inps freed by uma_zfree_smr() are cached and = inpcb_dtor() is not called immediately , >>> Z> thus leaking `inp->inp_cred` ref and hence `prison->pr_ref`. >>> Z> >>> Z> And it is also not possible to free up the cache by per-VNET = SYSUNINIT tcp_destroy / udp_destroy / rip_destroy. >>>=20 >>> This is known issue and I'd prefer not to call it a problem. The = "leak" of a jail >>> happens only if machine is idle wrt the networking activity. >>>=20 >>> Getting back to the problem that started this thread - the epair(4)s = not immediately >>> popping back to prison0. IMHO, the problem again lies in the design = of if_vmove and >>> epair(4) in particular. The if_vmove shall not exist, instead we = should do a full >>> if_attach() and if_detach(). The state of an ifnet when it undergoes = if_vmove doesn't >>> carry any useful information. With Alexander melifaro@ we discussed = better options >>> for creating or attaching interfaces to jails that if_vmove. Until = they are ready >>> the most easy workaround to deal with annoying epair(4) come back = problem is to >>> remove it manually before destroying a jail, like I did in = 80fc25025ff. >>>=20 >>=20 >> It still behaved much better prior to eb93b99d6986, which you and = Mark >> were going to work on a solution for to allow the cred "leak" to = close >> up much more quickly. CC markj@, since I think it's been six months >> since the last time I inquired about it, making this a good time to = do >> it again... >=20 > I spent some time trying to see if we could fix this in UMA/SMR and > talked to Jeff about it a bit. At this point I don't think it's the > right approach, at least for now. Really we have a composability > problem where different layers are using different techniques to = signal > that they're done with a particular piece of memory, and they just > aren't compatible. I originally thought that `uma_free_smr()` is somewhat like = `epoch_call()` with an internal `epoch_callback_t`, but after digging into the source code = it is not true. `uma_free_smr()` put the item into cache and until next allocation from = the cache the destructor get a chance to run. Can SMR provide some mean just like `epoch_callback_t` , so that the = destructors eventually get been invoked ? >=20 > One thing I tried is to implement a UMA function which walks over all > SMR zones and synchronizes all cached items (so that their destructors > are called). This is really expensive, at minimum it has to bind to = all > CPUs in the system so that it can flush per-CPU buckets. If > jail_deref() calls that function, the bug goes away at least in my > limited testing, but its use is really a layering violation. I've proposed a `vnet_shutdown()` stage in another mail. Maybe we can = introduce a `vnet_cleanup()` and INPCB layer register to listen a `cleanup` event = and the function which synchronizing cached items get been invoked. Is that still a layering violation? >=20 > We could, say, periodically scan cached UMA/SMR items and invoke their > destructors, but for most SMR consumers this is unnecessary, and again > there's a layering problem: the inpcb layer shouldn't "know" that it = has > to do that for its zones, since it's the jail layer that actually = cares. >=20 > It also seems kind of strange that dying jails still occupy a slot in > the jail namespace. I don't really understand why the existence of a > dying jail prevents creation of a new jail with the same name, but > presumably there's a good reason for it? >=20 > Now my inclination is to try and fix this in the inpcb layer, by not > accessing the inp_cred at all in the lookup path until we hold the = inpcb > lock, and then releasing the cred ref before freeing a PCB to its = zone. > I think this is doable based on a few observations: > - When doing an SMR-protected lookup, we always lock the returned = inpcb > before handing it to the caller. So we could in principle perform > inp_cred checking after acquiring the lock but before returning. > - If there are no jailed PCBs in a hash chain = in_pcblookup_hash_locked() > always scans the whole chain. > - If we match only one PCB in a lookup, we can probably(?) return that > PCB without dereferencing the cred pointer at all. If not, then the > scan only has to keep track of a fixed number of PCBs before picking > which one to return. So it looks like we can perform a lockless scan > and keep track of matches on the stack, then lock the matched PCBs = and > perform prison checks if necessary, without making the common case > more expensive. >=20 > In fact there is a parallel thread on freebsd-jail which reports that > this inp_cred access is a source of frequent cache misses. I was > surprised to see that the scan calls prison_flag() before even = checking > the PCB's local address. So if the hash chain is large then we're > potentially performing a lot of unnecessary memory accesses (though > presumably it's common for most of the PCBs to be sharing a single > cred?). In particular we can perhaps solve two problems at once. >=20 > Any thoughts? Are there some fundamental reasons this can't work? --Apple-Mail=_05999826-0A4B-4FD5-A20A-0ED8E067BC1C Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" = class=3D""><blockquote type=3D"cite" class=3D"" style=3D"font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; = -webkit-text-stroke-width: 0px; text-decoration: none;"><div = class=3D""><br class=3D"Apple-interchange-newline">On Dec 21, 2022, at = 12:12 AM, Mark Johnston <<a href=3D"mailto:markj@freebsd.org" = class=3D"">markj@freebsd.org</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""><span class=3D"" = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: = normal; letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;">On Sun, Dec 18, 2022 at 10:52:58AM -0600, = Kyle Evans wrote:</span><br class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><blockquote type=3D"cite" class=3D"" = style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;">On Sat, Dec 17, 2022 at 11:22 AM Gleb Smirnoff = <<a href=3D"mailto:glebius@freebsd.org" = class=3D"">glebius@freebsd.org</a>> wrote:<br class=3D""><blockquote = type=3D"cite" class=3D""><br class=3D""> Zhenlei,<br class=3D""><br = class=3D"">On Fri, Dec 16, 2022 at 06:30:57PM +0800, Zhenlei Huang = wrote:<br class=3D"">Z> I managed to repeat this issue on CURRENT/14 = with this small snip:<br class=3D"">Z><br class=3D"">Z> = -------------------------------------------<br class=3D"">Z> = #!/bin/sh<br class=3D"">Z><br class=3D"">Z> # test jail name<br = class=3D"">Z> n=3D"test_ref_leak"<br class=3D"">Z><br = class=3D"">Z> jail -c name=3D$n path=3D/ vnet persist<br = class=3D"">Z> # The following line trigger jail pr_ref leak<br = class=3D"">Z> jexec $n ifconfig lo0 inet 127.0.0.1/8<br = class=3D"">Z><br class=3D"">Z> jail -R $n<br class=3D"">Z><br = class=3D"">Z> # wait a moment<br class=3D"">Z> sleep 1<br = class=3D"">Z><br class=3D"">Z> jls -j $n<br class=3D"">Z><br = class=3D"">Z> After DDB debugging and tracing , it seems that is = triggered by a combine of [1] and [2]<br class=3D"">Z><br = class=3D"">Z> [1]<span class=3D"Apple-converted-space"> </span><a = href=3D"https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac89= 5915" = class=3D"">https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5a= c895915</a><span class=3D"Apple-converted-space"> </span><<a = href=3D"https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac89= 5915" = class=3D"">https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5a= c895915</a>><br class=3D"">Z> [2]<span = class=3D"Apple-converted-space"> </span><a = href=3D"https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b= 8c5b" = class=3D"">https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b1= 75b8c5b</a><span class=3D"Apple-converted-space"> </span><<a = href=3D"https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b= 8c5b" = class=3D"">https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b1= 75b8c5b</a>><br class=3D"">Z><br class=3D"">Z><br = class=3D"">Z> In [1] the per-VNET uma zone is shared with the global = one.<br class=3D"">Z> `pcbinfo->ipi_zone =3D = pcbstor->ips_zone;`<br class=3D"">Z><br class=3D"">Z> In [2] = unref `inp->inp_cred` is deferred called in inpcb_dtor() by = uma_zfree_smr() .<br class=3D"">Z><br class=3D"">Z> Unfortunately = inps freed by uma_zfree_smr() are cached and inpcb_dtor() is not called = immediately ,<br class=3D"">Z> thus leaking `inp->inp_cred` ref = and hence `prison->pr_ref`.<br class=3D"">Z><br class=3D"">Z> = And it is also not possible to free up the cache by per-VNET SYSUNINIT = tcp_destroy / udp_destroy / rip_destroy.<br class=3D""><br class=3D"">This= is known issue and I'd prefer not to call it a problem. The "leak" of a = jail<br class=3D"">happens only if machine is idle wrt the networking = activity.<br class=3D""><br class=3D"">Getting back to the problem that = started this thread - the epair(4)s not immediately<br class=3D"">popping = back to prison0. IMHO, the problem again lies in the design of if_vmove = and<br class=3D"">epair(4) in particular. The if_vmove shall not exist, = instead we should do a full<br class=3D"">if_attach() and if_detach(). = The state of an ifnet when it undergoes if_vmove doesn't<br = class=3D"">carry any useful information. With Alexander melifaro@ we = discussed better options<br class=3D"">for creating or attaching = interfaces to jails that if_vmove. Until they are ready<br class=3D"">the = most easy workaround to deal with annoying epair(4) come back problem is = to<br class=3D"">remove it manually before destroying a jail, like I did = in 80fc25025ff.<br class=3D""><br class=3D""></blockquote><br = class=3D"">It still behaved much better prior to eb93b99d6986, which you = and Mark<br class=3D"">were going to work on a solution for to allow the = cred "leak" to close<br class=3D"">up much more quickly. CC markj@, = since I think it's been six months<br class=3D"">since the last time I = inquired about it, making this a good time to do<br class=3D"">it = again...<br class=3D""></blockquote><br class=3D"" style=3D"caret-color: = rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: = normal; font-variant-caps: normal; font-weight: normal; letter-spacing: = normal; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">I spent = some time trying to see if we could fix this in UMA/SMR and</span><br = class=3D"" style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: normal; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, 0); font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;">talked to Jeff about it = a bit. At this point I don't think it's the</span><br class=3D"" = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: = normal; letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><span class=3D"" = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: = normal; letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;">right approach, at least for now. = Really we have a composability</span><br class=3D"" = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: = normal; letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><span class=3D"" = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: = normal; letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;">problem where different layers are using = different techniques to signal</span><br class=3D"" style=3D"caret-color: = rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: = normal; font-variant-caps: normal; font-weight: normal; letter-spacing: = normal; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">that = they're done with a particular piece of memory, and they just</span><br = class=3D"" style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: normal; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, 0); font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;">aren't = compatible.</span><br class=3D"" style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"></div></blockquote><div class=3D"" = style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; = text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, = 0);"><br class=3D""></div><div class=3D"" style=3D"font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; = -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: = rgb(0, 0, 0); color: rgb(0, 0, 0);">I originally thought that = `uma_free_smr()` is somewhat like `epoch_call()` with</div><div class=3D""= style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; = text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, = 0);">an internal `epoch_callback_t`, but after digging into the source = code it is not true.</div><div class=3D"" style=3D"font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; = -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: = rgb(0, 0, 0); color: rgb(0, 0, 0);">`<span class=3D"" = style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, = 0);">uma_free_smr()` put the item into cache and until next allocation = from the</span></div><div class=3D"" style=3D"font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: normal; letter-spacing: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; = -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: = rgb(0, 0, 0); color: rgb(0, 0, 0);"><span class=3D"" style=3D"caret-color:= rgb(0, 0, 0); color: rgb(0, 0, 0);">cache the destructor get a chance = to run.</span></div><div class=3D"" style=3D"font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: normal; letter-spacing: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; = -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: = rgb(0, 0, 0); color: rgb(0, 0, 0);"><span class=3D"" style=3D"caret-color:= rgb(0, 0, 0); color: rgb(0, 0, 0);"><br class=3D""></span></div><div = class=3D"" style=3D"font-family: Helvetica; font-size: 12px; font-style: = normal; font-variant-caps: normal; font-weight: normal; letter-spacing: = normal; orphans: auto; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; widows: auto; word-spacing: = 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; = text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, = 0);"><span class=3D"" style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, = 0, 0);">Can SMR provide some mean just like `</span><span class=3D"" = style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, = 0);">epoch_callback_t` , so that the destructors</span></div><div = class=3D"" style=3D"font-family: Helvetica; font-size: 12px; font-style: = normal; font-variant-caps: normal; font-weight: normal; letter-spacing: = normal; orphans: auto; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; widows: auto; word-spacing: = 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; = text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, = 0);"><span class=3D"" style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, = 0, 0);">eventually get been invoked ?</span></div><br class=3D"" = style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; = text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, = 0);"><blockquote type=3D"cite" class=3D"" style=3D"font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; = -webkit-text-stroke-width: 0px; text-decoration: none;"><div = class=3D""><br class=3D"" style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">One = thing I tried is to implement a UMA function which walks over = all</span><br class=3D"" style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">SMR = zones and synchronizes all cached items (so that their = destructors</span><br class=3D"" style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">are = called). This is really expensive, at minimum it has to bind to = all</span><br class=3D"" style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">CPUs in = the system so that it can flush per-CPU buckets. If</span><br = class=3D"" style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: normal; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, 0); font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;">jail_deref() calls that = function, the bug goes away at least in my</span><br class=3D"" = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: = normal; letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><span class=3D"" = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: = normal; letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;">limited testing, but its use is really a = layering violation.</span><br class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"></div></blockquote><div class=3D"" = style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; = text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, = 0);"><br class=3D""></div><div class=3D"" style=3D"font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; = -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: = rgb(0, 0, 0); color: rgb(0, 0, 0);">I've proposed a `vnet_shutdown()` = stage in another mail. Maybe we can introduce</div><div class=3D"" = style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; = text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, = 0);">a `vnet_cleanup()` and INPCB layer register to listen a = `cleanup` event and</div><div class=3D"" style=3D"font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; = -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: = rgb(0, 0, 0); color: rgb(0, 0, 0);">the function which synchronizing = cached items get been invoked.</div><div class=3D"" style=3D"font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; = -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: = rgb(0, 0, 0); color: rgb(0, 0, 0);">Is that still a layering = violation?</div><br class=3D"" style=3D"font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: normal; letter-spacing: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; = -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: = rgb(0, 0, 0); color: rgb(0, 0, 0);"><blockquote type=3D"cite" class=3D"" = style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; = text-decoration: none;"><div class=3D""><br class=3D"" = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: = normal; letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><span class=3D"" = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: = normal; letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;">We could, say, periodically scan cached = UMA/SMR items and invoke their</span><br class=3D"" style=3D"caret-color: = rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: = normal; font-variant-caps: normal; font-weight: normal; letter-spacing: = normal; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline = !important;">destructors, but for most SMR consumers this is = unnecessary, and again</span><br class=3D"" style=3D"caret-color: rgb(0, = 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">there's = a layering problem: the inpcb layer shouldn't "know" that it = has</span><br class=3D"" style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">to do = that for its zones, since it's the jail layer that actually = cares.</span><br class=3D"" style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><br class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">It also = seems kind of strange that dying jails still occupy a slot in</span><br = class=3D"" style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: normal; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, 0); font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;">the jail namespace. = I don't really understand why the existence of a</span><br = class=3D"" style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: normal; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, 0); font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;">dying jail prevents = creation of a new jail with the same name, but</span><br class=3D"" = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: = normal; letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><span class=3D"" = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: = normal; letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;">presumably there's a good reason for = it?</span><br class=3D"" style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><br class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">Now my = inclination is to try and fix this in the inpcb layer, by not</span><br = class=3D"" style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: normal; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, 0); font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;">accessing the inp_cred = at all in the lookup path until we hold the inpcb</span><br class=3D"" = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: = normal; letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><span class=3D"" = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: = normal; letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;">lock, and then releasing the cred ref = before freeing a PCB to its zone.</span><br class=3D"" = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: = normal; letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><span class=3D"" = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: = normal; letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;">I think this is doable based on a few = observations:</span><br class=3D"" style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">- When = doing an SMR-protected lookup, we always lock the returned = inpcb</span><br class=3D"" style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline = !important;"> before handing it to the caller. So we could in = principle perform</span><br class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline = !important;"> inp_cred checking after acquiring the lock but before = returning.</span><br class=3D"" style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">- If = there are no jailed PCBs in a hash chain = in_pcblookup_hash_locked()</span><br class=3D"" style=3D"caret-color: = rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: = normal; font-variant-caps: normal; font-weight: normal; letter-spacing: = normal; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline = !important;"> always scans the whole chain.</span><br class=3D"" = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: = normal; letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><span class=3D"" = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: = normal; letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;">- If we match only one PCB in a lookup, we = can probably(?) return that</span><br class=3D"" style=3D"caret-color: = rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: = normal; font-variant-caps: normal; font-weight: normal; letter-spacing: = normal; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline = !important;"> PCB without dereferencing the cred pointer at all. = If not, then the</span><br class=3D"" style=3D"caret-color: rgb(0, = 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline = !important;"> scan only has to keep track of a fixed number of PCBs = before picking</span><br class=3D"" style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline = !important;"> which one to return. So it looks like we can = perform a lockless scan</span><br class=3D"" style=3D"caret-color: = rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: = normal; font-variant-caps: normal; font-weight: normal; letter-spacing: = normal; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline = !important;"> and keep track of matches on the stack, then lock the = matched PCBs and</span><br class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline = !important;"> perform prison checks if necessary, without making = the common case</span><br class=3D"" style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline = !important;"> more expensive.</span><br class=3D"" = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: = normal; letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><br class=3D"" = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: = normal; letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><span class=3D"" = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: = normal; letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;">In fact there is a parallel thread on = freebsd-jail which reports that</span><br class=3D"" style=3D"caret-color:= rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: = normal; font-variant-caps: normal; font-weight: normal; letter-spacing: = normal; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">this = inp_cred access is a source of frequent cache misses. I = was</span><br class=3D"" style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline = !important;">surprised to see that the scan calls prison_flag() before = even checking</span><br class=3D"" style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">the = PCB's local address. So if the hash chain is large then = we're</span><br class=3D"" style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline = !important;">potentially performing a lot of unnecessary memory accesses = (though</span><br class=3D"" style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline = !important;">presumably it's common for most of the PCBs to be sharing a = single</span><br class=3D"" style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">cred?). = In particular we can perhaps solve two problems at once.</span><br = class=3D"" style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: normal; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><br class=3D"" style=3D"caret-color: rgb(0, 0, 0); font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><span class=3D"" style=3D"caret-color: rgb(0, 0, 0); font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;">Any thoughts? Are = there some fundamental reasons this can't = work?</span></div></blockquote></body></html>= --Apple-Mail=_05999826-0A4B-4FD5-A20A-0ED8E067BC1C--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?069FE014-1D62-4479-B830-9FBDF36F2109>