Date: Thu, 19 Mar 2026 23:47:07 +0800 From: Zhenlei Huang <zlei@FreeBSD.org> To: Konstantin Belousov <kostikbel@gmail.com> Cc: FreeBSD Current <current@freebsd.org> Subject: Re: userret: returning with the following locks held Message-ID: <F976AFFB-8A44-40BD-990A-DBA1B389421C@FreeBSD.org> In-Reply-To: <abroibATpGiFzk9e@kib.kiev.ua> References: <A0EEC016-7FFA-4036-99DF-AC0B7A874B3F@FreeBSD.org> <abroibATpGiFzk9e@kib.kiev.ua>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --]
> On Mar 19, 2026, at 2:01 AM, Konstantin Belousov <kostikbel@gmail.com> wrote:
>
> On Wed, Mar 18, 2026 at 05:36:00PM +0800, Zhenlei Huang wrote:
>> Hi,
>>
>> While I'm working on the if_detach() / if_vmove() racing issue and I wrote
>> a test and try to repeat the race condition and got this assert panic,
>>
>> ```
>> <6>epair1b: link state changed to UP
>> userret: returning with the following locks held:
>> exclusive sx ifnet_detach_sx (ifnet_detach_sx) r = 1 (0xffffffff83ab75e0) locked @ /home/zlei/freebsd-src/sys/net/if.c:1286
>> panic: witness_warn
>> cpuid = 2
>> time = 1773824038
>> KDB: stack backtrace:
>> db_trace_self_wrapper() at db_trace_self_wrapper+0xa5/frame 0xfffffe00ec001710
>> kdb_backtrace() at kdb_backtrace+0xc6/frame 0xfffffe00ec001870
>> vpanic() at vpanic+0x214/frame 0xfffffe00ec001a10
>> panic() at panic+0xb5/frame 0xfffffe00ec001ad0
>> witness_warn() at witness_warn+0x7f7/frame 0xfffffe00ec001c30
>> userret() at userret+0x11c/frame 0xfffffe00ec001d10
>> amd64_syscall() at amd64_syscall+0x694/frame 0xfffffe00ec001f30
>> fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00ec001f30
>> --- syscall (19, FreeBSD ELF64, compat.lseek), rip = 0xfd288de65ba, rsp = 0xfd284aad4b8, rbp = 0xfd284aad500 ---
>> Uptime: 29m8s
>> Dumping 1355 out of 16137 MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91%
>> ```
>>
>> The line 1286 of sys/net/if.c is ( my WIP revision ),
>>
>> ```
>> 1250 static int
>> 1251 if_vmove_reclaim(struct thread *td, char *ifname, int jid)
>> 1252 {
>> ...
>> 1282 /* Get interface back from child jail/vnet. */
>> 1283 found = if_unlink_ifnet(ifp, true);
>> 1284 MPASS(found);
>> 1285 sx_xlock(&ifnet_detach_sxlock);
>> 1286 if_vmove(ifp, vnet_dst);
>> 1287 sx_xunlock(&ifnet_detach_sxlock);
>> 1288
>> 1289 /* Report the new if_xname back to the userland. */
>> 1290 sprintf(ifname, "%s", ifp->if_xname);
>> 1291
>> 1292 prison_free(pr);
>> 1293 CURVNET_RESTORE();
>> 1294 return (0);
>> 1295 }
>> ```
>>
>> That puzzled me a lot. The current thread only reaches to `if_vmove()` , how does it happen to userret() ?
>>
>> Is that witness warn is false report ?
>
> I doubt that this is false report.
>
> userret() is the common code that prepares thread for return from kernel
> to usermode. As part of the preparation, it checks that no locks is
> left owned by the thread. What you see is the result of the failing
> check.
>
> In other words, you have some syscall that locked a lock, did not released
> it, then returned. The layer that manages return to userspace caught the
> issue.
Thanks ! I'll take care of the locking issue ( syscall returned without lock released ).
I'll report back if I re-encounter this issue.
Best regards,
Zhenlei
[-- Attachment #2 --]
<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Mar 19, 2026, at 2:01 AM, Konstantin Belousov <<a href="mailto:kostikbel@gmail.com" class="">kostikbel@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta charset="UTF-8" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">On Wed, Mar 18, 2026 at 05:36:00PM +0800, Zhenlei Huang wrote:</span><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote type="cite" style="font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Hi,<br class=""><br class="">While I'm working on the if_detach() / if_vmove() racing issue and I wrote<br class="">a test and try to repeat the race condition and got this assert panic,<br class=""><br class="">```<br class=""><6>epair1b: link state changed to UP<br class="">userret: returning with the following locks held:<br class="">exclusive sx ifnet_detach_sx (ifnet_detach_sx) r = 1 (0xffffffff83ab75e0) locked @ /home/zlei/freebsd-src/sys/net/if.c:1286<br class="">panic: witness_warn<br class="">cpuid = 2<br class="">time = 1773824038<br class="">KDB: stack backtrace:<br class="">db_trace_self_wrapper() at db_trace_self_wrapper+0xa5/frame 0xfffffe00ec001710<br class="">kdb_backtrace() at kdb_backtrace+0xc6/frame 0xfffffe00ec001870<br class="">vpanic() at vpanic+0x214/frame 0xfffffe00ec001a10<br class="">panic() at panic+0xb5/frame 0xfffffe00ec001ad0<br class="">witness_warn() at witness_warn+0x7f7/frame 0xfffffe00ec001c30<br class="">userret() at userret+0x11c/frame 0xfffffe00ec001d10<br class="">amd64_syscall() at amd64_syscall+0x694/frame 0xfffffe00ec001f30<br class="">fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00ec001f30<br class="">--- syscall (19, FreeBSD ELF64, compat.lseek), rip = 0xfd288de65ba, rsp = 0xfd284aad4b8, rbp = 0xfd284aad500 ---<br class="">Uptime: 29m8s<br class="">Dumping 1355 out of 16137 MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91%<br class="">```<br class=""><br class="">The line 1286 of sys/net/if.c is ( my WIP revision ),<br class=""><br class="">```<br class="">1250 static int<br class="">1251 if_vmove_reclaim(struct thread *td, char *ifname, int jid)<br class="">1252 {<br class="">...<br class="">1282 /* Get interface back from child jail/vnet. */<br class="">1283 found = if_unlink_ifnet(ifp, true);<br class="">1284 MPASS(found);<br class="">1285 sx_xlock(&ifnet_detach_sxlock);<br class="">1286 if_vmove(ifp, vnet_dst);<br class="">1287 sx_xunlock(&ifnet_detach_sxlock);<br class="">1288<span class="Apple-converted-space"> </span><br class="">1289 /* Report the new if_xname back to the userland. */<br class="">1290 sprintf(ifname, "%s", ifp->if_xname);<br class="">1291<span class="Apple-converted-space"> </span><br class="">1292 prison_free(pr);<br class="">1293 CURVNET_RESTORE();<br class="">1294 return (0);<br class="">1295 }<br class="">```<br class=""><br class="">That puzzled me a lot. The current thread only reaches to `if_vmove()` , how does it happen to userret() ?<br class=""><br class="">Is that witness warn is false report ?<br class=""></blockquote><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">I doubt that this is false report.</span><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">userret() is the common code that prepares thread for return from kernel</span><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">to usermode. As part of the preparation, it checks that no locks is</span><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">left owned by the thread. What you see is the result of the failing</span><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">check.</span><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">In other words, you have some syscall that locked a lock, did not released</span><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">it, then returned. The layer that manages return to userspace caught the</span><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">issue.</span></div></blockquote><br class=""></div><div>Thanks ! I'll take care of the locking issue ( <span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""> </span><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">syscall returned without </span>lock released ).</div><div><br class=""></div><div>I'll report back if I re-encounter this issue.</div><br class=""><div class="">
<div>Best regards,</div><div>Zhenlei</div>
</div>
<br class=""></body></html>
home |
help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?F976AFFB-8A44-40BD-990A-DBA1B389421C>