From owner-freebsd-current@freebsd.org Thu May 9 04:02:00 2019 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CEB4D159C37E for ; Thu, 9 May 2019 04:02:00 +0000 (UTC) (envelope-from ler@FreeBSD.org) Received: from thebighonker.lerctr.org (ns-b.lerctr.org [IPv6:2001:470:1f0f:3ad::53:2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "*.lerctr.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6F49474EF5; Thu, 9 May 2019 04:02:00 +0000 (UTC) (envelope-from ler@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=ler2019; h=Message-ID:References:In-Reply-To:Subject:Cc:To:From:Date: Content-Transfer-Encoding:Content-Type:MIME-Version:Sender:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=rfpKDD05W2AlUzlB8KjrU0tSGCU7RH+HQE5IORxu8BQ=; b=YrDqYjEI1qfXiwPmUEgea7y5mk DGa1kTgx2b6F5A+6S6/F0OGCzURJ+s/ippzW9pVe9UugO6z8uvcPm0ZMHG61XyhNyFt0IEV25m5S+ ceZg6VZ1bnfZL43akZPmouU2jQGuyq/pOMRB4lAoJRdS+laB0JiQINiL2oYp67c0ET+aA/K0ZTSoX HEJqxGvIUVakSZ+Zxx0DvSTO+AgZY/TGAMtgWs1ZaDDxNC3PwOftcFwhmyFGSTYUB7Div/AQNWWya s1I00e6qOy4eoG+yiso9QHrbtZVB//MuIG5HZMeDrgExJHLuKxOzBce6mru16tSSOJfkYG8QcfioW 9cb6oCGw==; Received: from thebighonker.lerctr.org ([2001:470:1f0f:3ad:bb:dcff:fe50:d900]:51519 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpsa (TLSv1.3:TLS_AES_256_GCM_SHA384:256) (Exim 4.92 (FreeBSD)) (envelope-from ) id 1hOaFq-000AHg-Ns; Wed, 08 May 2019 23:01:58 -0500 Received: from 2600:1700:210:b180:d8c:cc0b:810:3d61 by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Wed, 08 May 2019 23:01:58 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Wed, 08 May 2019 23:01:58 -0500 From: Larry Rosenman To: Mark Johnston Cc: freebsd-current@freebsd.org, Mark Johnston Subject: Re: Crash loading dtraceall In-Reply-To: <20190509033211.GE11774@raichu> References: <20190508205245.ulbo6fusk3b4py7t@ler-imac.local> <20190508222932.GB11774@raichu> <845dd186ef038d98c1a95a7454e432d2@FreeBSD.org> <20190508225553.GC11774@raichu> <460d563e2fe48bfd90b489015b4c0f9d@FreeBSD.org> <20190509033211.GE11774@raichu> Message-ID: <8418ec8f5e303dce4225a53be88fc49d@FreeBSD.org> X-Sender: ler@FreeBSD.org User-Agent: Roundcube Webmail/1.3.9 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 May 2019 04:02:01 -0000 On 05/08/2019 10:32 pm, Mark Johnston wrote: > On Wed, May 08, 2019 at 05:57:18PM -0500, Larry Rosenman wrote: >> On 05/08/2019 5:55 pm, Mark Johnston wrote: >> > On Wed, May 08, 2019 at 05:47:08PM -0500, Larry Rosenman wrote: >> >> On 05/08/2019 5:29 pm, Mark Johnston wrote: >> >> > On Wed, May 08, 2019 at 03:52:45PM -0500, Larry Rosenman wrote: >> >> >> Greetings, >> >> >> >> >> >> Somewhere between r346483 and r347241 loading dtraceall causes a >> >> >> crash. I have the cores and kernels. >> >> >> >> >> >> It's hard for me to bisect more than this, as the box is remote. >> >> >> >> >> >> What more do you need? (this dump is fropm r347355). >> >> > >> >> > Please visit frame 8 and print *lf. >> >> > >> >> #9 fbt_provide_module_function (lf=0xfffff800020ff000, symindx=30763, >> >> symval=0xfffffe00d74d7e00, opaque=0xfffffe00d74d7e50) at >> >> /usr/src/sys/cddl/dev/fbt/x86/fbt_isa.c:191 >> >> 191 if (*instr == FBT_PUSHL_EBP) >> >> (kgdb) print *lf >> >> $1 = {ops = 0xfffff800020f6000, refs = 202, userrefs = 1, flags = 1, >> >> link = {tqe_next = 0xfffff800020fec00, tqe_prev = 0xffffffff80c767d0 >> >> }, filename = 0xfffff80002101030 "kernel", >> >> pathname = 0xfffff80002104080 "/boot/kernel/kernel", id = 1, >> >> address = >> >> 0xffffffff80200000 "\177ELF\002\001\001\t", size = 17612816, >> >> ctors_addr >> >> = 0x0, ctors_size = 0, ndeps = 0, deps = 0x0, common = {stqh_first = >> >> 0x0, >> >> stqh_last = 0xfffff800020ff070}, modules = {tqh_first = >> >> 0xfffff800020e5800, tqh_last = 0xfffff80002116790}, loaded = {tqe_next >> >> = >> >> 0x0, tqe_prev = 0x0}, loadcnt = 1, nenabled = 0, fbt_nentries = 25062} >> >> (kgdb) >> > >> > And could you show the output of: >> > >> > $ readelf -s /boot/kernel/kernel | grep "30763:" >> > _______________________________________________ >> > freebsd-current@freebsd.org mailing list >> > https://lists.freebsd.org/mailman/listinfo/freebsd-current >> > To unsubscribe, send any mail to >> > "freebsd-current-unsubscribe@freebsd.org" >> >> [root@oldtbh2 /var/crash]# readelf -s /boot/kernel/kernel | grep >> "30763:" >> 30763: ffffffff80791310 75 IFUNC GLOBAL DEFAULT 8 >> x86_rng_store >> [root@oldtbh2 /var/crash]# > > The problem is with the kernel linker's handling of ifuncs. When > enumerating symbols, it replaces ifunc symbol values with the return > value of the resolver but preserves the original symbol size, which is > that of the resolver. I believe this patch will address the panic > you're seeing: > > diff --git a/sys/kern/link_elf.c b/sys/kern/link_elf.c > index 6ceb34d66b74..8bd9a0219a1d 100644 > --- a/sys/kern/link_elf.c > +++ b/sys/kern/link_elf.c > @@ -1350,17 +1350,23 @@ static int > link_elf_symbol_values(linker_file_t lf, c_linker_sym_t sym, > linker_symval_t *symval) > { > + c_linker_sym_t target; > elf_file_t ef; > const Elf_Sym *es; > caddr_t val; > + long diff; > > ef = (elf_file_t)lf; > es = (const Elf_Sym *)sym; > if (es >= ef->symtab && es < (ef->symtab + ef->nchains)) { > symval->name = ef->strtab + es->st_name; > val = (caddr_t)ef->address + es->st_value; > - if (ELF_ST_TYPE(es->st_info) == STT_GNU_IFUNC) > + if (ELF_ST_TYPE(es->st_info) == STT_GNU_IFUNC) { > val = ((caddr_t (*)(void))val)(); > + (void)link_elf_search_symbol(lf, val, &target, &diff); > + if (diff == 0) > + es = (const Elf_Sym *)target; > + } > symval->value = val; > symval->size = es->st_size; > return (0); > @@ -1370,8 +1376,12 @@ link_elf_symbol_values(linker_file_t lf, > c_linker_sym_t sym, > if (es >= ef->ddbsymtab && es < (ef->ddbsymtab + ef->ddbsymcnt)) { > symval->name = ef->ddbstrtab + es->st_name; > val = (caddr_t)ef->address + es->st_value; > - if (ELF_ST_TYPE(es->st_info) == STT_GNU_IFUNC) > + if (ELF_ST_TYPE(es->st_info) == STT_GNU_IFUNC) { > val = ((caddr_t (*)(void))val)(); > + (void)link_elf_search_symbol(lf, val, &target, &diff); > + if (diff == 0) > + es = (const Elf_Sym *)target; > + } > symval->value = val; > symval->size = es->st_size; > return (0); > diff --git a/sys/kern/link_elf_obj.c b/sys/kern/link_elf_obj.c > index ac4cc8c085cb..5ce160a05699 100644 > --- a/sys/kern/link_elf_obj.c > +++ b/sys/kern/link_elf_obj.c > @@ -1240,9 +1240,11 @@ static int > link_elf_symbol_values(linker_file_t lf, c_linker_sym_t sym, > linker_symval_t *symval) > { > + c_linker_sym_t target; > elf_file_t ef; > const Elf_Sym *es; > caddr_t val; > + long diff; > > ef = (elf_file_t) lf; > es = (const Elf_Sym*) sym; > @@ -1250,8 +1252,12 @@ link_elf_symbol_values(linker_file_t lf, > c_linker_sym_t sym, > if (es >= ef->ddbsymtab && es < (ef->ddbsymtab + ef->ddbsymcnt)) { > symval->name = ef->ddbstrtab + es->st_name; > val = (caddr_t)es->st_value; > - if (ELF_ST_TYPE(es->st_info) == STT_GNU_IFUNC) > + if (ELF_ST_TYPE(es->st_info) == STT_GNU_IFUNC) { > val = ((caddr_t (*)(void))val)(); > + (void)link_elf_search_symbol(lf, val, &target, &diff); > + if (diff == 0) > + es = (const Elf_Sym *)target; > + } > symval->value = val; > symval->size = es->st_size; > return 0; It does *NOT*. ⌂69% [ler@oldtbh2.lerctr.org:/var/crash] $ more core.txt.6 oldtbh2.lerctr.org dumped core - see /var/crash/vmcore.6 Wed May 8 22:59:19 CDT 2019 FreeBSD oldtbh2.lerctr.org 13.0-CURRENT FreeBSD 13.0-CURRENT #27 r347355M: Wed May 8 22:49:25 CDT 2019 root@oldtbh2.lerctr.org:/usr/obj/usr/src/amd64.amd64/sys/LER-MINIMAL amd64 panic: page fault GNU gdb (GDB) 8.2.1 [GDB v8.2.1 for FreeBSD] Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd13.0". Type "show configuration" for configuration details. For bug reporting instructions, please see: . Find the GDB manual and other documentation resources online at: . For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /boot/kernel/kernel...Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...done. done. Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 02 fault virtual address = 0x10 fault code = supervisor read data , page not present instruction pointer = 0x20:0xffffffff804be609 stack pointer = 0x28:0xfffffe00d727ddc0 frame pointer = 0x28:0xfffffe00d727dde0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 1523 (kldload) trap number = 12 panic: page fault cpuid = 2 time = 1557374088 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00d727da70 vpanic() at vpanic+0x19d/frame 0xfffffe00d727dac0 panic() at panic+0x43/frame 0xfffffe00d727db20 trap_fatal() at trap_fatal+0x394/frame 0xfffffe00d727db80 trap_pfault() at trap_pfault+0x49/frame 0xfffffe00d727dbe0 trap() at trap+0x2b4/frame 0xfffffe00d727dcf0 calltrap() at calltrap+0x8/frame 0xfffffe00d727dcf0 --- trap 0xc, rip = 0xffffffff804be609, rsp = 0xfffffe00d727ddc0, rbp = 0xfffffe00d727dde0 --- link_elf_symbol_values() at link_elf_symbol_values+0x1e9/frame 0xfffffe00d727dde0 link_elf_each_function_nameval() at link_elf_each_function_nameval+0x64/frame 0xfffffe00d727de40 fbt_provide_module() at fbt_provide_module+0xde/frame 0xfffffe00d727e270 fbt_linker_file_cb() at fbt_linker_file_cb+0x12/frame 0xfffffe00d727e280 linker_file_foreach() at linker_file_foreach+0x52/frame 0xfffffe00d727e2b0 linker_load_module() at linker_load_module+0xbd8/frame 0xfffffe00d727e5e0 linker_load_dependencies() at linker_load_dependencies+0x2fd/frame 0xfffffe00d727e630 link_elf_load_file() at link_elf_load_file+0x105e/frame 0xfffffe00d727e6f0 linker_load_module() at linker_load_module+0x9ef/frame 0xfffffe00d727ea20 kern_kldload() at kern_kldload+0xa7/frame 0xfffffe00d727ea60 sys_kldload() at sys_kldload+0x5b/frame 0xfffffe00d727ea90 amd64_syscall() at amd64_syscall+0x25c/frame 0xfffffe00d727ebb0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00d727ebb0 --- syscall (304, FreeBSD ELF64, sys_kldload), rip = 0x8002de43a, rsp = 0x7fffffffe658, rbp = 0x7fffffffebd0 --- Uptime: 1m41s Dumping 2248 out of 64482 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% __curthread () at /usr/src/sys/amd64/include/pcpu.h:241 241 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (OFFSETOF_CURTHREAD)); (kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu.h:241 #1 doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:383 #2 0xffffffff80496320 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:470 #3 0xffffffff80496799 in vpanic (fmt=, ap=) at /usr/src/sys/kern/kern_shutdown.c:896 #4 0xffffffff804964d3 in panic (fmt=) at /usr/src/sys/kern/kern_shutdown.c:823 #5 0xffffffff80767314 in trap_fatal (frame=0xfffffe00d727dd00, eva=16) at /usr/src/sys/amd64/amd64/trap.c:946 #6 0xffffffff80767379 in trap_pfault (frame=0xfffffe00d727dd00, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:765 #7 0xffffffff80766964 in trap (frame=0xfffffe00d727dd00) at /usr/src/sys/amd64/amd64/trap.c:441 #8 #9 0xffffffff804be609 in link_elf_symbol_values (lf=0xfffff800020ff000, sym=, symval=0xfffffe00d727ddf0) at /usr/src/sys/kern/link_elf.c:1385 #10 0xffffffff804bf8e4 in link_elf_each_function_nameval ( file=0xfffff800020ff000, callback=0xffffffff825cb570 , opaque=0xfffffe00d727de50) at /usr/src/sys/kern/link_elf.c:1519 #11 0xffffffff825ca33e in fbt_provide_module (arg=, lf=0xfffff800020ff000) at /usr/src/sys/cddl/dev/fbt/fbt.c:204 #12 0xffffffff825ca242 in fbt_linker_file_cb (lf=0xffffffff811f8cc8, arg=0x7a39) at /usr/src/sys/cddl/dev/fbt/fbt.c:1103 #13 0xffffffff8046d772 in linker_file_foreach ( predicate=0xffffffff825ca230 , context=0x0) at /usr/src/sys/kern/kern_linker.c:594 #14 0xffffffff8046cb58 in linker_file_sysinit (lf=0xfffff8001cfea000) at /usr/src/sys/kern/kern_linker.c:236 #15 linker_load_file (filename=, result=) at /usr/src/sys/kern/kern_linker.c:462 #16 linker_load_module (kldname=, modname=0xffffffff81d792ae "fbt", parent=, verinfo=, lfpp=0x0) at /usr/src/sys/kern/kern_linker.c:2110 #17 0xffffffff8046f1bd in linker_load_dependencies (lf=0xfffff8001cc82800) at /usr/src/sys/kern/kern_linker.c:2200 #18 0xffffffff80797fde in link_elf_load_file (cls=, filename=0xfffff8001ce1f200 "/boot/kernel/dtraceall.ko", result=0xfffffe00d727e898) at /usr/src/sys/kern/link_elf_obj.c:1010 #19 0xffffffff8046c96f in LINKER_LOAD_FILE ( cls=0xffffffff80acccc0 , filename=, result=0x0) at ./linker_if.h:180 #20 linker_load_file (filename=, result=) at /usr/src/sys/kern/kern_linker.c:447 #21 linker_load_module (kldname=, modname=0xfffff80015e6ec00 "dtraceall", parent=, verinfo=, lfpp=0xfffffe00d727ea38) at /usr/src/sys/kern/kern_linker.c:2110 #22 0xffffffff8046e297 in kern_kldload (td=0xfffff800936415a0, file=, fileid=0xfffffe00d727ea74) at /usr/src/sys/kern/kern_linker.c:1089 #23 0xffffffff8046e35b in sys_kldload (td=0xfffff800936415a0, uap=) at /usr/src/sys/kern/kern_linker.c:1115 #24 0xffffffff80767ddc in syscallenter (td=0xfffff800936415a0) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:135 #25 amd64_syscall (td=0xfffff800936415a0, traced=0) at /usr/src/sys/amd64/amd64/trap.c:1166 #26 #27 0x00000008002de43a in ?? () Backtrace stopped: Cannot access memory at address 0x7fffffffe658 (kgdb) -- Larry Rosenman http://people.freebsd.org/~ler Phone: +1 214-642-9640 E-Mail: ler@FreeBSD.org US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106