Date: Fri, 2 Nov 2018 00:16:23 -0700 From: Mark Millard <marklmi26-fbsd@yahoo.com> To: Konstantin Belousov <kostikbel@gmail.com> Cc: svn-src-head@freebsd.org, Alexander Richardson <arichardson@freebsd.org>, Shawn Webb <shawn.webb@hardenedbsd.org> Subject: Re: svn commit: r339876 - head/libexec/rtld-elf Message-ID: <003A49D7-6E8B-4775-A70B-E0EB44505D4B@yahoo.com> In-Reply-To: <E44F5772-1F8A-40B8-9C4E-B8362B768F37@yahoo.com> References: <8E5A5F3A-F1A7-4702-A2F7-65D74CC5B2E5@yahoo.com> <20181102004101.GI5335@kib.kiev.ua> <E44F5772-1F8A-40B8-9C4E-B8362B768F37@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[The backtrace confirms what I reported to Alexander Richardson and Shawn Webb earlier.] On 2018-Nov-1, at 6:40 PM, Mark Millard <marklmi26-fbsd at yahoo.com> = wrote: > On 2018-Nov-1, at 5:41 PM, Konstantin Belousov <kostikbel at = gmail.com> wrote: >=20 >> On Tue, Oct 30, 2018 at 12:45:13PM -0700, Mark Millard wrote: >>> Konstantin Belousov kostikbel at gmail.com wrote on >>> Tue Oct 30 18:04:04 UTC 2018 : >>>=20 >>>> On Tue, Oct 30, 2018 at 03:32:40PM +0000, Alexander Richardson = wrote: >>>>> On Tue, 30 Oct 2018 at 10:17, Michael Tuexen >>>>> <Michael.Tuexen at macmic.franken.de> wrote: >>>>>>=20 >>>>>>> On 29. Oct 2018, at 22:08, Alex Richardson <arichardson at = FreeBSD.org> wrote: >>>>>>>=20 >>>>>>> Author: arichardson >>>>>>> Date: Mon Oct 29 21:08:02 2018 >>>>>>> New Revision: 339876 >>>>>>> URL: https://svnweb.freebsd.org/changeset/base/339876 >>>>>>>=20 >>>>>>> Log: >>>>>>> rtld: set obj->textsize correctly >>>>>>>=20 >>>>>>> With lld-generated binaries the first PT_LOAD will usually be a = read-only >>>>>>> segment unless you pass --no-rosegment. For those binaries the = textsize is >>>>>>> determined by the next PT_LOAD. To allow both LLD and bfd 2.17 = binaries to >>>>>>> be parsed correctly use the end of the last PT_LOAD that is = marked as >>>>>>> executable instead. >>>>>>>=20 >>>>>>> I noticed that the value was wrong while adding some debug = prints for some rtld >>>>>>> changes for CHERI binaries. `obj->textsize` only seems to be = used by PPC so the >>>>>>> effect is untested. However, the value before was definitely = wrong and the new >>>>>>> result matches the phdrs. >>>>>> I build kernel and world with a revision later than this on a = PPC. Buildword >>>>>> ends up with a world where almost all binaries are = segfaulting.... Especially gdb >>>>>> (but svn, ls or so all segfault). >>>>>>=20 >>>>>> Best regards >>>>>> Michael >>>>>=20 >>>>> This is rather surprising since if anything the range of the = icache >>>>> flush should increase rather than decrease after this change. >>>>>=20 >>>>> I can only see this causing a behaviour change if we actually need = to >>>>> flush more than just the executable segments. >>>>> Is it possible that some binary/library contains a non-executable >>>>> segment as the first PT_LOAD? >>>>> Or is there some linker script that adds custom PHDRS? >>>>>=20 >>>> Could it be that there is a hole between start of the object = mapping and >>>> the last PT_LOADable segment eligible for execution ? >>>=20 >>> [This note may be easier to deal with than the first >>> note that I sent out.] >>>=20 >>> [My examples are from devel/powerpc64-xtoolchain-gcc used >>> to buildworld buildkernel targeting a head -r339076 based >>> powerpc64 environment. I do that on powerpc64 as well.] >>>=20 >>> powerpc64 loads the readonly data and the readonly code in one = PT_LOAD, >>> the first. The 2nd PT_LOAD is for sections without the readonly = status, >>> that includes .got and .plt being spanned. See below from >>> objdump -ph for /bin/ls : >>>=20 >>> Program Header: >>> PHDR off 0x0000000000000040 vaddr 0x0000000010000040 paddr = 0x0000000010000040 align 2**3 >>> filesz 0x0000000000000188 memsz 0x0000000000000188 flags r-- >>> INTERP off 0x00000000000001c8 vaddr 0x00000000100001c8 paddr = 0x00000000100001c8 align 2**0 >>> filesz 0x0000000000000015 memsz 0x0000000000000015 flags r-- >>> LOAD off 0x0000000000000000 vaddr 0x0000000010000000 paddr = 0x0000000010000000 align 2**16 >>> filesz 0x000000000000910c memsz 0x000000000000910c flags r-x >>> LOAD off 0x0000000000009110 vaddr 0x0000000010019110 paddr = 0x0000000010019110 align 2**16 >>> filesz 0x0000000000000ee0 memsz 0x00000000000010e8 flags rw- >>> DYNAMIC off 0x0000000000009138 vaddr 0x0000000010019138 paddr = 0x0000000010019138 align 2**3 >>> filesz 0x00000000000001c0 memsz 0x00000000000001c0 flags rw- >>> NOTE off 0x00000000000001e0 vaddr 0x00000000100001e0 paddr = 0x00000000100001e0 align 2**2 >>> filesz 0x0000000000000030 memsz 0x0000000000000030 flags r-- >>> STACK off 0x0000000000000000 vaddr 0x0000000000000000 paddr = 0x0000000000000000 align 2**4 >>> filesz 0x0000000000000000 memsz 0x0000000000000000 flags rw- >>=20 >> We only need program headers, and we only need them from the object >> which load causes the fault. It might be not the binary but some = library >> that triggers the fault. So the backtrace and some information from = the >> state of the image is needed. >>=20 >> You can build only rtld and use it as the standalone program to = initiate >> the image activation: >> <path>/ld-elf.so.1 /bin/ls >> or similar. >=20 > . . . It stops when the dcbst in __syncicache runs into an address in the p_align 65536 caused hole between the two PT_LOAD's with PF_X. /bin/ls itself has such a hole, as do the .so libraries involved. Details follow. (gdb) run /bin/ls Starting program: = /usr/obj/powerpcvtsc_gcc421/powerpc.powerpc/usr/src/powerpc.powerpc/libexe= c/rtld-elf/ld-elf.so.1.full /bin/ls Program received signal SIGSEGV, Segmentation fault. __syncicache (from=3D0x1800000, len=3D102400) at = /usr/src/lib/libc/powerpc/gen/syncicache.c:94 94 __asm __volatile ("dcbst 0,%0" :: "r"(p)); (gdb) bt #0 __syncicache (from=3D0x1800000, len=3D102400) at = /usr/src/lib/libc/powerpc/gen/syncicache.c:94 #1 0x01012b58 in reloc_non_plt (obj=3D0x41041000, obj_rtld=3D<optimized = out>, flags=3D4, lockstate=3D0x0) at = /usr/src/libexec/rtld-elf/powerpc/reloc.c:330 #2 0x010175f4 in relocate_object (obj=3D0x41041000, bind_now=3D0 = '\000', rtldobj=3D0x103ea88 <obj_rtld>, flags=3D4, lockstate=3D0x0) at = /usr/src/libexec/rtld-elf/rtld.c:2849 #3 0x0101776c in relocate_objects (first=3D<optimized out>, bind_now=3D0 = '\000', rtldobj=3D0x103ea88 <obj_rtld>, flags=3D4, lockstate=3D0x0) at = /usr/src/libexec/rtld-elf/rtld.c:2908 #4 0x0101beb0 in _rtld (sp=3D<optimized out>, exit_proc=3D0xffffdc08, = objp=3D0xffffdc0c) at /usr/src/libexec/rtld-elf/rtld.c:677 #5 0x0101217c in .rtld_start () at = /usr/src/libexec/rtld-elf/powerpc/rtld_start.S:98 Backtrace stopped: frame did not save the PC This is because there are 2 PT_LOAD's that have PF_X but for which there is an address range between that spans may pages for which there is no header entry for, in part a consequence of p_align being 65536 so forcing lots of space between entries 2 and 3 ( /bin/ls example): entry: 2 p_type: PT_LOAD p_offset: 0 p_vaddr: 0x1800000 p_paddr: 0x1800000 p_filesz: 34112 p_memsz: 34112 p_flags: PF_X|PF_R p_align: 65536 entry: 3 p_type: PT_LOAD p_offset: 34112 p_vaddr: 0x1818540 p_paddr: 0x1818540 p_filesz: 316 p_memsz: 1752 p_flags: PF_X|PF_W|PF_R p_align: 65536 ( check 2's p_paddr+ p_memsz vs. 3's p_vaddr Later below the failing address will be shown as between those. ) [powerpc64 only gets one PT_LOAD with PF_X and so can not have space between such PT_LOAD's as things currently are.] The source code related details are as follows and shows that it was processing /bin/ls itself ( obj_main->path =3D 0x41042000 "/bin/ls" and the failing address ) : [The presentation order is not necessarily the same as my exploration order.] (gdb) list 89 off =3D (u_int)from & (cacheline_size - 1); 90 l =3D len +=3D off; 91 p =3D (char *)from - off; 92=09 93 do { 94 __asm __volatile ("dcbst 0,%0" :: "r"(p)); 95 p +=3D cacheline_size; 96 } while ((l -=3D cacheline_size) > 0); 97 __asm __volatile ("sync"); 98 p =3D (char *)from - off; [Reminder for entry 2: p_vaddr: 0x1800000 p_memsz: 34112 0x1800000+34112 =3D 0x1808540 ] (gdb) print p $4 =3D 0x1809000 <error: Cannot access memory at address 0x1809000> (gdb) up #1 0x01012b58 in reloc_non_plt (obj=3D0x41041000, obj_rtld=3D<optimized = out>, flags=3D4, lockstate=3D0x0) at = /usr/src/libexec/rtld-elf/powerpc/reloc.c:330 330 __syncicache(obj->mapbase, obj->textsize); (gdb) list 325 done: 326 if (cache !=3D NULL) 327 free(cache); 328=09 329 /* Synchronize icache for text seg in case we made any = changes */ 330 __syncicache(obj->mapbase, obj->textsize); 331=09 332 return (r); 333 } 334=09 (gdb) up #2 0x010175f4 in relocate_object (obj=3D0x41041000, bind_now=3D0 = '\000', rtldobj=3D0x103ea88 <obj_rtld>, flags=3D4, lockstate=3D0x0) at = /usr/src/libexec/rtld-elf/rtld.c:2849 2849 if (reloc_non_plt(obj, rtldobj, flags, lockstate)) (gdb) list 2844 /* There are relocations to the write-protected text = segment. */ 2845 if (obj->textrel && reloc_textrel_prot(obj, true) !=3D = 0) 2846 return (-1); 2847=09 2848 /* Process the non-PLT non-IFUNC relocations. */ 2849 if (reloc_non_plt(obj, rtldobj, flags, lockstate)) 2850 return (-1); 2851=09 2852 /* Re-protected the text segment. */ 2853 if (obj->textrel && reloc_textrel_prot(obj, false) !=3D = 0) (gdb) up =20 #3 0x0101776c in relocate_objects (first=3D<optimized out>, bind_now=3D0 = '\000', rtldobj=3D0x103ea88 <obj_rtld>, flags=3D4, lockstate=3D0x0) at = /usr/src/libexec/rtld-elf/rtld.c:2908 2908 error =3D relocate_object(obj, bind_now, = rtldobj, flags, (gdb) list 2903=09 2904 for (error =3D 0, obj =3D first; obj !=3D NULL; 2905 obj =3D TAILQ_NEXT(obj, next)) { 2906 if (obj->marker) 2907 continue; 2908 error =3D relocate_object(obj, bind_now, = rtldobj, flags, 2909 lockstate); 2910 if (error =3D=3D -1) 2911 break; 2912 } (gdb) up #4 0x0101beb0 in _rtld (sp=3D<optimized out>, exit_proc=3D0xffffdc08, = objp=3D0xffffdc0c) at /usr/src/libexec/rtld-elf/rtld.c:677 677 if (relocate_objects(obj_main, (gdb) list 672 * block even if they didn't ask for it. 673 */ 674 allocate_tls_offset(entry->obj); 675 } 676=09 677 if (relocate_objects(obj_main, 678 ld_bind_now !=3D NULL && *ld_bind_now !=3D '\0', 679 &obj_rtld, SYMLOOK_EARLY, NULL) =3D=3D -1) 680 rtld_die(); 681=09 (gdb) up =20 #5 0x0101217c in .rtld_start () at = /usr/src/libexec/rtld-elf/powerpc/rtld_start.S:98 98 bl _rtld@plt /* &_start =3D _rtld(sp, = &exit_proc, &obj_main)*/ (gdb) list 93 addi %r3,%r4,-4 /* locate argc ptr, &argv[-1] */ 94=09 95 addi %r4,%r1,8 /* &exit_proc on stack */ 96 addi %r5,%r1,12 /* &obj_main on stack */ 97=09 98 bl _rtld@plt /* &_start =3D _rtld(sp, = &exit_proc, &obj_main)*/ 99 mtlr %r3 100=09 101 /* 102 * Restore args, with new obj/exit proc (gdb) down #4 0x0101beb0 in _rtld (sp=3D<optimized out>, exit_proc=3D0xffffdc08, = objp=3D0xffffdc0c) at /usr/src/libexec/rtld-elf/rtld.c:677 677 if (relocate_objects(obj_main, . . . (gdb) print *obj_main $3 =3D {magic =3D 0, version =3D 0, next =3D {tqe_next =3D 0x41041200, = tqe_prev =3D 0x103ea68 <obj_list>}, path =3D 0x41042000 "/bin/ls", = origin_path =3D 0x0, refcount =3D 1, holdcount =3D 0, dl_refcount =3D 0,=20= mapbase =3D 0x1800000 "\177ELF\001\002\001\t", mapsize =3D 102400, = textsize =3D 102400, vaddrbase =3D 25165824, relocbase =3D 0x0, dynamic = =3D 0x1818554,=20 entry =3D 0x18022e0 "|\b\002\246\224!\377\340=3D \001\202\223\241", = phdr =3D 0x1800034, phsize =3D 224, interp =3D 0x1800114 = "/libexec/ld-elf.so.1", stack_flags =3D 6, tlsindex =3D 0, tlsinit =3D = 0x0,=20 tlsinitsize =3D 0, tlssize =3D 0, tlsoffset =3D 0, tlsalign =3D 0, = relro_page =3D 0x0, relro_size =3D 0, pltgot =3D 0x18186f4, rel =3D 0x0, = relsize =3D 0, rela =3D 0x1801ebc, relasize =3D 1008, pltrel =3D 0x0,=20 pltrelsize =3D 0, pltrela =3D 0x1801f28, pltrelasize =3D 900, symtab =3D= 0x1800798, strtab =3D 0x1801418 "", strsize =3D 2241, verneed =3D = 0x1801e6c, verneednum =3D 1, verdef =3D 0x0, verdefnum =3D 0,=20 versyms =3D 0x1801cda, buckets =3D 0x1800164, nbuckets =3D 197, chains = =3D 0x1800478, nchains =3D 200, nbuckets_gnu =3D 0, symndx_gnu =3D 0, = maskwords_bm_gnu =3D 0, shift2_gnu =3D 0, dynsymcount =3D 200,=20 bloom_gnu =3D 0x0, buckets_gnu =3D 0x0, chain_zero_gnu =3D 0x0, rpath = =3D 0x0, runpath =3D 0x0, needed =3D 0x41042010, needed_filtees =3D 0x0, = needed_aux_filtees =3D 0x0, names =3D {stqh_first =3D 0x0,=20 stqh_last =3D 0x410410f8}, vertab =3D 0x41043080, vernum =3D 6, init = =3D 25174700, fini =3D 25197908, preinit_array =3D 0, init_array =3D 0, = fini_array =3D 0, preinit_array_num =3D 0, init_array_num =3D 0,=20 fini_array_num =3D 0, osrel =3D 1200084, mainprog =3D 1 '\001', rtld =3D= 0 '\000', relocated =3D 1 '\001', ver_checked =3D 1 '\001', textrel =3D = 0 '\000', symbolic =3D 0 '\000', bind_now =3D 0 '\000',=20 traced =3D 0 '\000', jmpslots_done =3D 0 '\000', init_done =3D 0 = '\000', tls_done =3D 1 '\001', phdr_alloc =3D 0 '\000', z_origin =3D 0 = '\000', z_nodelete =3D 0 '\000', z_noopen =3D 0 '\000',=20 z_loadfltr =3D 0 '\000', z_interpose =3D 0 '\000', z_nodeflib =3D 0 = '\000', z_global =3D 0 '\000', ref_nodel =3D 0 '\000', init_scanned =3D = 0 '\000', on_fini_list =3D 0 '\000', dag_inited =3D 0 '\000',=20 filtees_loaded =3D 0 '\000', irelative =3D 0 '\000', gnu_ifunc =3D 0 = '\000', non_plt_gnu_ifunc =3D 0 '\000', crt_no_init =3D 1 '\001', = valid_hash_sysv =3D 1 '\001', valid_hash_gnu =3D 0 '\000',=20 dlopened =3D 0 '\000', marker =3D 0 '\000', unholdfree =3D 0 '\000', = doomed =3D 0 '\000', linkmap =3D {l_addr =3D 0x1800000 = "\177ELF\001\002\001\t", l_name =3D 0x41042000 "/bin/ls", l_ld =3D = 0x1818554,=20 l_next =3D 0x41041334, l_prev =3D 0x0}, dldags =3D {stqh_first =3D = 0x0, stqh_last =3D 0x41041148}, dagmembers =3D {stqh_first =3D 0x0, = stqh_last =3D 0x41041150}, dev =3D 0, ino =3D 0, priv =3D 0x0} =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?003A49D7-6E8B-4775-A70B-E0EB44505D4B>