Date: Mon, 28 Dec 2020 12:44:18 -0800 From: John Baldwin <jhb@FreeBSD.org> To: Mateusz Guzik <mjg@FreeBSD.org>, Konstantin Belousov <kib@freebsd.org> Cc: "freebsd-current@freebsd.org" <freebsd-current@freebsd.org> Subject: Re: panic: Assertion pgrp->pg_jobc > 0 failed at kern_proc.c:816 Message-ID: <aa91221f-cfc6-6ba4-06d7-2251f52dd09c@FreeBSD.org> In-Reply-To: <8f88c925-f12a-6a51-c2b3-c21c55de8dab@FreeBSD.org> References: <8f88c925-f12a-6a51-c2b3-c21c55de8dab@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 12/28/20 12:24 PM, John Baldwin wrote:
> I got this panic again today in a VM when quitting a gdb
> session after killing a child process via 'kill'.
>
> panic: Assertion pgrp->pg_jobc > 0 failed at /git/bhyve/sys/kern/kern_proc.c:816
> cpuid = 1
> time = 1609185862
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00946547f0
> vpanic() at vpanic+0x181/frame 0xfffffe0094654840
> panic() at panic+0x43/frame 0xfffffe00946548a0
> _refcount_update_saturated() at _refcount_update_saturated/frame 0xfffffe00946548e0
> killjobc() at killjobc+0x6a6/frame 0xfffffe0094654940
> exit1() at exit1+0x6af/frame 0xfffffe00946549b0
> sys_sys_exit() at sys_sys_exit+0xd/frame 0xfffffe00946549c0
> amd64_syscall() at amd64_syscall+0x12e/frame 0xfffffe0094654af0
> fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0094654af0
> --- syscall (1, FreeBSD ELF64, sys_sys_exit), rip = 0x8024c8f0a, rsp = 0x7fffffffe358, rbp = 0x7fffffffe370 ---
> KDB: enter: panic
> [ thread pid 44034 tid 102484 ]
> Stopped at kdb_enter+0x37: movq $0,0x10ab066(%rip)
>
> From what I can tell, the child process that was killed via 'kill' has
> not yet exited and is stuck in ptracestop() from fork():
>
> (kgdb) where
> #0 sched_switch (td=0xfffffe0094001a00, flags=<optimized out>) at /git/bhyve/sys/kern/sched_ule.c:2147
> #1 0xffffffff80bf4015 in mi_switch (flags=266) at /git/bhyve/sys/kern/kern_synch.c:542
> #2 0xffffffff80bfeba5 in thread_suspend_switch (td=<optimized out>, p=<optimized out>)
> at /git/bhyve/sys/kern/kern_thread.c:1477
> #3 0xffffffff80bef04b in ptracestop (td=0xfffffe0094001a00, sig=17, si=0x0)
> at /git/bhyve/sys/kern/kern_sig.c:2642
> #4 0xffffffff80ba1a54 in fork_return (td=0xfffffe0094001a00, frame=0xfffffe0094671b00)
> at /git/bhyve/sys/kern/kern_fork.c:1106
> #5 0xffffffff80ba18b0 in fork_exit (callout=0xffffffff80ba1950 <fork_return>, arg=0xfffffe0094001a00,
> frame=0xfffffe0094671b00) at /git/bhyve/sys/kern/kern_fork.c:1069
> #6 <signal handler called>
> #7 0x00000008007b71aa in ?? ()
>
> kgdb can't find the panicking process due to the zombproc removal, so I will
> have to go work on kgdb to recover from that change. :(
I've come up with a shorter reproducer (original was trying to debug a perl script
in OpenSSL's test suite).
Compile this program:
#include <sys/types.h>
#include <sys/wait.h>
#include <err.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int
main(void)
{
pid_t pid, wpid;
pid = fork();
if (pid == -1)
err(1, "fork");
if (pid == 0) {
printf("I'm in the child\n");
exit(1);
}
printf("I'm in the parent\n");
wpid = waitpid(pid, NULL, 0);
if (wpid < 0)
err(1, "waitpid");
return (0);
}
Then in gdb do the following:
# gdb101 ./forktest
...
(gdb) catch fork
Catchpoint 1 (fork)
(gdb) r
Starting program: /mnt/forktest/forktest
Catchpoint 1 (forked process 830), _fork () at _fork.S:4
4 _fork.S: No such file or directory.
(gdb) kill
Kill the program being debugged? (y or n) y
[Inferior 1 (process 828) killed]
(gdb) quit
<panic>
--
John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?aa91221f-cfc6-6ba4-06d7-2251f52dd09c>