From owner-freebsd-hackers@freebsd.org Fri May 15 16:23:05 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BB64C2F9ACC for ; Fri, 15 May 2020 16:23:05 +0000 (UTC) (envelope-from freebsd@sysctl.cz) Received: from wes1-so1.wedos.net (wes1-so1.wedos.net [46.28.106.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49NtxN38RRz44W6 for ; Fri, 15 May 2020 16:23:03 +0000 (UTC) (envelope-from freebsd@sysctl.cz) Received: from webmail.wedos.net (wes1-wm3.wedos.net [46.28.106.84]) by wes1-so1.wedos.net (Postfix) with ESMTPSA id 49NtxB62ZQz3Fv; Fri, 15 May 2020 18:22:54 +0200 (CEST) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Fri, 15 May 2020 18:22:54 +0200 From: freebsd@sysctl.cz To: Konstantin Belousov Cc: Freebsd hackers Subject: Re: Debug linux binary with enable linux emulation In-Reply-To: <20200512174935.GK68906@kib.kiev.ua> References: <24f30eaa0597d79ddadc10d6f993f2a0@sysctl.cz> <20200511115656.GF68906@kib.kiev.ua> <67eed7f18b68f67ff04f628c4c367535@sysctl.cz> <20200512174935.GK68906@kib.kiev.ua> Message-ID: <0fd224d0e4bc643ec8980f2cd71d87c2@sysctl.cz> X-Sender: freebsd@sysctl.cz User-Agent: Roundcube Webmail/1.2.4 X-Rspamd-Queue-Id: 49NtxN38RRz44W6 X-Spamd-Bar: +++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of freebsd@sysctl.cz has no SPF policy when checking 46.28.106.43) smtp.mailfrom=freebsd@sysctl.cz X-Spamd-Result: default: False [3.87 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; IP_SCORE(1.05)[ipnet: 46.28.104.0/21(1.42), asn: 197019(3.76), country: CZ(0.09)]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[sysctl.cz]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.92)[0.920,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FROM_NO_DN(0.00)[]; NEURAL_SPAM_LONG(1.00)[0.999,0]; R_SPF_NA(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:197019, ipnet:46.28.104.0/21, country:CZ]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 May 2020 16:23:05 -0000 Dne 2020-05-12 19:49, Konstantin Belousov napsal: > On Tue, May 12, 2020 at 12:49:25AM +0200, freebsd@sysctl.cz wrote: >> Dne 2020-05-11 13:56, Konstantin Belousov napsal: >> > On Mon, May 11, 2020 at 12:28:23AM +0200, freebsd@sysctl.cz wrote: >> > > Hi, >> > > I tried debug with gdb for linux emulation >> > > and have issue with kernel panic. >> > > >> > > kldload linux64.ko >> > > gdb ./Discord or other linux binary >> > > >> > > Fatal trap 12: page fault while in kernel mode >> > > cpuid = 3; apic id = 03 >> > > fault virtual address = 0x18 >> > > fault code = supervisor read data, page not present >> > > instruction pointer = 0x20:0xffffffff82f5b682 >> > > stack pointer = 0x28:0xfffffe00691fd980 >> > > frame pointer = 0x28:0xfffffe00691fd9e0 >> > > code segment = base 0x0, limit 0xfffff, type 0x1b >> > > = DPL 0, pres 1, long 1, def32 0, gran 1 >> > > processor eflags = interrupt enabled, resume, IOPL = 0 >> > > current process = 17392 (fish) >> > > trap number = 12 >> > > panic: page fault >> > > cpuid = 3 >> > > time = 1589132677 >> > > KDB: stack backtrace: >> > > #0 0xffffffff80c1d2f7 at kdb_backtrace+0x67 >> > > #1 0xffffffff80bd062d at vpanic+0x19d >> > > #2 0xffffffff80bd0483 at panic+0x43 >> > > #3 0xffffffff810a7dcc at trap_fatal+0x39c >> > > #4 0xffffffff810a7e19 at trap_pfault+0x49 >> > > #5 0xffffffff810a740f at trap+0x29f >> > > #6 0xffffffff81081bdc at calltrap+0x8 >> > > #7 0xffffffff82f503d1 at linux_thread_detach+0x21 >> > Show the line number for linux_thread_detach+0x21. >> > Or better, compile with INVARIANTS, it should fire an assertion. >> > Then get a core dump. >> > >> > > #8 0xffffffff80be5acf at thread_suspend_check+0x41f >> > > #9 0xffffffff80c32ed9 at ast+0x3b9 >> > > #10 0xffffffff810850e9 at doreti_ast+0x1f >> > > Uptime: 2h56m24s >> > > Dumping 1146 out of 8042 >> > > MB:..2%..12%..21%..31%..41%..51%..62%..72%..81%..91%---<>--- >> > > Copyright (c) 1992-2019 The FreeBSD Project. >> > > >> > > GNU gdb (GDB) 9.1 [GDB v9.1 for FreeBSD] >> > > Copyright (C) 2020 Free Software Foundation, Inc. >> > > License GPLv3+: GNU GPL version 3 or later >> > > >> > > This is free software: you are free to change and redistribute it. >> > > There is NO WARRANTY, to the extent permitted by law. >> > > Type "show copying" and "show warranty" for details. >> > > This GDB was configured as "x86_64-portbld-freebsd12.1". >> > > Type "show configuration" for configuration details. >> > > For bug reporting instructions, please see: >> > > . >> > > Find the GDB manual and other documentation resources online at: >> > > . >> > > >> > > For help, type "help". >> > > Type "apropos word" to search for commands related to "word"... >> > > Reading symbols from /boot/kernel/kernel... >> > > (No debugging symbols found in /boot/kernel/kernel) >> > > 0xffffffff80c01eda in sched_switch () >> > > (kgdb) >> > > (kgdb) >> > > (kgdb) bt >> > > #0 0xffffffff80c01eda in sched_switch () >> > > #1 0xffffffff80bdbfa2 in mi_switch () >> > > #2 0xffffffff80c2bb75 in sleepq_catch_signals () >> > > #3 0xffffffff80c2be64 in sleepq_timedwait_sig () >> > > #4 0xffffffff80bdb9a5 in _sleep () >> > > #5 0xffffffff80bf1ee3 in umtxq_sleep () >> > > #6 0xffffffff80bf1c90 in do_wait () >> > > #7 0xffffffff80bef8fe in __umtx_op_wait_uint_private () >> > > #8 0xffffffff810a8984 in amd64_syscall () >> > > #9 >> > > #10 0x000000080974dedc in ?? () >> > > Backtrace stopped: Cannot access memory at address 0x7fffffffddc8 >> > > >> > > I have now kernel without debug symbols. >> > > >> > > M. >> > > _______________________________________________ >> > > freebsd-emulation@freebsd.org mailing list >> > > https://lists.freebsd.org/mailman/listinfo/freebsd-emulation >> > > To unsubscribe, send any mail to >> > > "freebsd-emulation-unsubscribe@freebsd.org" >> > _______________________________________________ >> > freebsd-hackers@freebsd.org mailing list >> > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> > To unsubscribe, send any mail to >> > "freebsd-hackers-unsubscribe@freebsd.org" >> >> >> >> Hi konstantin, >> I have good news, now we can look detail >> >> (kgdb) bt >> #0 __curthread () at /usr/src/sys/amd64/include/pcpu.h:234 >> #1 doadump (textdump=) at >> /usr/src/sys/kern/kern_shutdown.c:371 >> #2 0xffffffff80bd0228 in kern_reboot (howto=260) at >> /usr/src/sys/kern/kern_shutdown.c:451 >> #3 0xffffffff80bd0689 in vpanic (fmt=, ap=> out>) >> at /usr/src/sys/kern/kern_shutdown.c:877 >> #4 0xffffffff80bd0483 in panic (fmt=) at >> /usr/src/sys/kern/kern_shutdown.c:804 >> #5 0xffffffff810a7dcc in trap_fatal (frame=0xfffffe00634e58c0, >> eva=24) at >> /usr/src/sys/amd64/amd64/trap.c:943 >> #6 0xffffffff810a7e19 in trap_pfault (frame=0xfffffe00634e58c0, >> usermode=0) >> at /usr/src/sys/amd64/amd64/trap.c:767 >> #7 0xffffffff810a740f in trap (frame=0xfffffe00634e58c0) at >> /usr/src/sys/amd64/amd64/trap.c:443 >> #8 >> #9 release_futexes (td=, em=0x0) at >> /usr/src/sys/compat/linux/linux_futex.c:1283 >> #10 0xffffffff82f503d1 in linux_thread_detach (td=0xfffff8014bd935e0) >> at >> /usr/src/sys/compat/linux/linux_fork.c:466 >> #11 0xffffffff80be5acf in thread_suspend_check (return_instead=0) at >> /usr/src/sys/kern/kern_thread.c:1010 >> #12 0xffffffff80c32ed9 in ast (framep=0xfffffe00634e5ac0) at >> /usr/src/sys/kern/subr_trap.c:342 >> #13 0xffffffff810850e9 in doreti_ast () at >> /usr/src/sys/amd64/amd64/exception.S:1149 >> #14 0x0000000800bb7008 in ?? () >> #15 0x000000000000000f in ?? () >> #16 0x0000000000000000 in ?? () >> (kgdb) list 0xffffffff82f503d1 >> Function "0xffffffff82f503d1" not defined. >> (kgdb) list *0xffffffff82f503d1 >> 0xffffffff82f503d1 is in linux_thread_detach >> (/usr/src/sys/compat/linux/linux_fork.c:468). >> warning: Source file is more recent than executable. > This indicates that your source potentially does not match the kernel. > And indeed as seen below, if panic occured on line 468, then it should > be due to em dereference, which should be 0. But then, > release_futexes() > alse dereference em and should cause the same trap earlier. > >> 463 >> 464 LINUX_CTR1(thread_detach, "thread(%d)", em->em_tid); >> 465 >> 466 release_futexes(td, em); >> 467 >> 468 child_clear_tid = em->child_clear_tid; >> 469 >> 470 if (child_clear_tid != NULL) { >> 471 >> 472 LINUX_CTR2(thread_detach, "thread(%d) %p", >> (kgdb) > So you did not recompiled with INVARIANTS ? > > It might be easier if you provide me with a self-contained tarball of > small > linux binary and all required libraries which reproduce the panic. > > If not, ensure that the kernel binary is consistent with the sources, > that INVARIANTS are enabled, and provide backtraces from gdb for all > threads of the paniced process. > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to > "freebsd-hackers-unsubscribe@freebsd.org" Hello konstantin, i have now 12.1 p5 generic kernel and cannot replicate this kernel panic. M.F.