From owner-freebsd-current@freebsd.org Tue Dec 29 19:26:40 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6A3A84C8DA1 for ; Tue, 29 Dec 2020 19:26:40 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4D54D014Zpz3Hth; Tue, 29 Dec 2020 19:26:39 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.16.1/8.16.1) with ESMTPS id 0BTJQWE6088608 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Tue, 29 Dec 2020 21:26:35 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 0BTJQWE6088608 Received: (from kostik@localhost) by tom.home (8.16.1/8.16.1/Submit) id 0BTJQWBP088607; Tue, 29 Dec 2020 21:26:32 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 29 Dec 2020 21:26:32 +0200 From: Konstantin Belousov To: John Baldwin Cc: Mateusz Guzik , "freebsd-current@freebsd.org" Subject: Re: panic: Assertion pgrp->pg_jobc > 0 failed at kern_proc.c:816 Message-ID: References: <8f88c925-f12a-6a51-c2b3-c21c55de8dab@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 4D54D014Zpz3Hth X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Dec 2020 19:26:40 -0000 On Mon, Dec 28, 2020 at 12:44:18PM -0800, John Baldwin wrote: > On 12/28/20 12:24 PM, John Baldwin wrote: > > I got this panic again today in a VM when quitting a gdb > > session after killing a child process via 'kill'. > > > > panic: Assertion pgrp->pg_jobc > 0 failed at /git/bhyve/sys/kern/kern_proc.c:816 > > cpuid = 1 > > time = 1609185862 > > KDB: stack backtrace: > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00946547f0 > > vpanic() at vpanic+0x181/frame 0xfffffe0094654840 > > panic() at panic+0x43/frame 0xfffffe00946548a0 > > _refcount_update_saturated() at _refcount_update_saturated/frame 0xfffffe00946548e0 > > killjobc() at killjobc+0x6a6/frame 0xfffffe0094654940 > > exit1() at exit1+0x6af/frame 0xfffffe00946549b0 > > sys_sys_exit() at sys_sys_exit+0xd/frame 0xfffffe00946549c0 > > amd64_syscall() at amd64_syscall+0x12e/frame 0xfffffe0094654af0 > > fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0094654af0 > > --- syscall (1, FreeBSD ELF64, sys_sys_exit), rip = 0x8024c8f0a, rsp = 0x7fffffffe358, rbp = 0x7fffffffe370 --- > > KDB: enter: panic > > [ thread pid 44034 tid 102484 ] > > Stopped at kdb_enter+0x37: movq $0,0x10ab066(%rip) > > > > From what I can tell, the child process that was killed via 'kill' has > > not yet exited and is stuck in ptracestop() from fork(): > > > > (kgdb) where > > #0 sched_switch (td=0xfffffe0094001a00, flags=) at /git/bhyve/sys/kern/sched_ule.c:2147 > > #1 0xffffffff80bf4015 in mi_switch (flags=266) at /git/bhyve/sys/kern/kern_synch.c:542 > > #2 0xffffffff80bfeba5 in thread_suspend_switch (td=, p=) > > at /git/bhyve/sys/kern/kern_thread.c:1477 > > #3 0xffffffff80bef04b in ptracestop (td=0xfffffe0094001a00, sig=17, si=0x0) > > at /git/bhyve/sys/kern/kern_sig.c:2642 > > #4 0xffffffff80ba1a54 in fork_return (td=0xfffffe0094001a00, frame=0xfffffe0094671b00) > > at /git/bhyve/sys/kern/kern_fork.c:1106 > > #5 0xffffffff80ba18b0 in fork_exit (callout=0xffffffff80ba1950 , arg=0xfffffe0094001a00, > > frame=0xfffffe0094671b00) at /git/bhyve/sys/kern/kern_fork.c:1069 > > #6 > > #7 0x00000008007b71aa in ?? () > > > > kgdb can't find the panicking process due to the zombproc removal, so I will > > have to go work on kgdb to recover from that change. :( > > I've come up with a shorter reproducer (original was trying to debug a perl script > in OpenSSL's test suite). > > Compile this program: > > #include > #include > #include > #include > #include > #include > > int > main(void) > { > pid_t pid, wpid; > > pid = fork(); > if (pid == -1) > err(1, "fork"); > if (pid == 0) { > printf("I'm in the child\n"); > exit(1); > } > printf("I'm in the parent\n"); > wpid = waitpid(pid, NULL, 0); > if (wpid < 0) > err(1, "waitpid"); > > return (0); > } > > Then in gdb do the following: > > # gdb101 ./forktest > ... > (gdb) catch fork > Catchpoint 1 (fork) > (gdb) r > Starting program: /mnt/forktest/forktest > > Catchpoint 1 (forked process 830), _fork () at _fork.S:4 > 4 _fork.S: No such file or directory. > (gdb) kill > Kill the program being debugged? (y or n) y > [Inferior 1 (process 828) killed] > (gdb) quit > > https://reviews.freebsd.org/D27816