From owner-svn-src-head@freebsd.org Fri Nov 2 07:26:39 2018 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5F3C710EB9CE for ; Fri, 2 Nov 2018 07:26:39 +0000 (UTC) (envelope-from marklmi26-fbsd@yahoo.com) Received: from sonic303-21.consmr.mail.gq1.yahoo.com (sonic303-21.consmr.mail.gq1.yahoo.com [98.137.64.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C682A79F01 for ; Fri, 2 Nov 2018 07:26:38 +0000 (UTC) (envelope-from marklmi26-fbsd@yahoo.com) X-YMail-OSG: ikN2PUoVM1n9yak7mhssWZhD5i7l47AfYFlkkKFqpwU7acYHC8g1e7jqlPvUAf7 SUaPg4Xj9doK5vGaOKWMjDWzMArrVxXed3qHcbnWbYxloYjOJSHc7ES4SQFEJfpcixemynmDzfFi koM3B6s4rUlStcAqWm_di_4Fe0.bxZRMr7915w1Au2Oloc2sEKh73zbGz1hEsxPjxb9BzNRG8KIG Z8y_4LWPYioJerDUgcOXvX29n1hwBzEJB7DiDKvl0jsMLPzntsj6yXY75D6rjN5KzLO4j9OEeeZw 4GxdnvyAwXtRmw9HjP8aiqiPmTt9oRABTK1IlpHV0LjXvLag0I6PbDSSHzJ92eQJqyiVzhihRn_U WCw3B30F2Hn.V_7kYSz2qR7HZ5HbGvgcVT672gVq7uztqbGZJaJGbsQFprw0vpb2Lf9TE_qZhWgv _VJGB8MiEEwyDB8eubxSpWMqGiMR6evw8Wrb58aC9wY8XCwod8mzRQxWQ8Hf9j7JnipRKulucQyC f0RFc7.OMQdKR4j1QPw7z5sabEzudo7JNGBLtV4H23vefWWNOjgkovePYVKJLv._lsMkznjtDpyK OHOZyl98q_L8GSvXOx5kqIjJMea29MxAo35thd8S1nc0wMftvNdAsz35bsozh0i_Mdx47rED2zmY kCpNHd7Bs3QYWT1dvJpYjHjPAb9_fsMtEYTCmsuwawND7czkbD.erdNGOT_fnBqroKEL1mErTr5q 4uqD8O1g7sZ7y9oIiq1rf5iRz_w2pxlwCHgx1ctxUzpZaMNa53Nda3DXmEXJ0cwt49s39VSH.zSa bRP0Wzin1VDD3X0opguleM37_gN3EdV64hTDKdgJAa._lgR3Od_k84Dvo88a9iqh3QCe37LgmHMT IVeqczny.pwhTEJLNjSRez5q0hlt28ILZp65oZByVLdBTeIIdq__SXLdEK.YbUAp3BoHSWjwBCNL 526imcvJeV09QHRab3Tsq5CXvWxBpBxb8rOjsPUBgse_ersuANKLmGZg9ccCsz8S0prZ0if3rNKv HYCui8gtJWegpXBltrW86p19OQzkVjow_E2c.D3QhN.Y.pGw- Received: from sonic.gate.mail.ne1.yahoo.com by sonic303.consmr.mail.gq1.yahoo.com with HTTP; Fri, 2 Nov 2018 07:26:35 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.25]) ([76.115.7.162]) by smtp430.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID a383864a705eded79eb5e434fc5403a8; Fri, 02 Nov 2018 07:16:23 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: svn commit: r339876 - head/libexec/rtld-elf From: Mark Millard In-Reply-To: Date: Fri, 2 Nov 2018 00:16:23 -0700 Cc: svn-src-head@freebsd.org, Alexander Richardson , Shawn Webb Content-Transfer-Encoding: quoted-printable Message-Id: <003A49D7-6E8B-4775-A70B-E0EB44505D4B@yahoo.com> References: <8E5A5F3A-F1A7-4702-A2F7-65D74CC5B2E5@yahoo.com> <20181102004101.GI5335@kib.kiev.ua> To: Konstantin Belousov X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Nov 2018 07:26:39 -0000 [The backtrace confirms what I reported to Alexander Richardson and Shawn Webb earlier.] On 2018-Nov-1, at 6:40 PM, Mark Millard = wrote: > On 2018-Nov-1, at 5:41 PM, Konstantin Belousov wrote: >=20 >> On Tue, Oct 30, 2018 at 12:45:13PM -0700, Mark Millard wrote: >>> Konstantin Belousov kostikbel at gmail.com wrote on >>> Tue Oct 30 18:04:04 UTC 2018 : >>>=20 >>>> On Tue, Oct 30, 2018 at 03:32:40PM +0000, Alexander Richardson = wrote: >>>>> On Tue, 30 Oct 2018 at 10:17, Michael Tuexen >>>>> wrote: >>>>>>=20 >>>>>>> On 29. Oct 2018, at 22:08, Alex Richardson wrote: >>>>>>>=20 >>>>>>> Author: arichardson >>>>>>> Date: Mon Oct 29 21:08:02 2018 >>>>>>> New Revision: 339876 >>>>>>> URL: https://svnweb.freebsd.org/changeset/base/339876 >>>>>>>=20 >>>>>>> Log: >>>>>>> rtld: set obj->textsize correctly >>>>>>>=20 >>>>>>> With lld-generated binaries the first PT_LOAD will usually be a = read-only >>>>>>> segment unless you pass --no-rosegment. For those binaries the = textsize is >>>>>>> determined by the next PT_LOAD. To allow both LLD and bfd 2.17 = binaries to >>>>>>> be parsed correctly use the end of the last PT_LOAD that is = marked as >>>>>>> executable instead. >>>>>>>=20 >>>>>>> I noticed that the value was wrong while adding some debug = prints for some rtld >>>>>>> changes for CHERI binaries. `obj->textsize` only seems to be = used by PPC so the >>>>>>> effect is untested. However, the value before was definitely = wrong and the new >>>>>>> result matches the phdrs. >>>>>> I build kernel and world with a revision later than this on a = PPC. Buildword >>>>>> ends up with a world where almost all binaries are = segfaulting.... Especially gdb >>>>>> (but svn, ls or so all segfault). >>>>>>=20 >>>>>> Best regards >>>>>> Michael >>>>>=20 >>>>> This is rather surprising since if anything the range of the = icache >>>>> flush should increase rather than decrease after this change. >>>>>=20 >>>>> I can only see this causing a behaviour change if we actually need = to >>>>> flush more than just the executable segments. >>>>> Is it possible that some binary/library contains a non-executable >>>>> segment as the first PT_LOAD? >>>>> Or is there some linker script that adds custom PHDRS? >>>>>=20 >>>> Could it be that there is a hole between start of the object = mapping and >>>> the last PT_LOADable segment eligible for execution ? >>>=20 >>> [This note may be easier to deal with than the first >>> note that I sent out.] >>>=20 >>> [My examples are from devel/powerpc64-xtoolchain-gcc used >>> to buildworld buildkernel targeting a head -r339076 based >>> powerpc64 environment. I do that on powerpc64 as well.] >>>=20 >>> powerpc64 loads the readonly data and the readonly code in one = PT_LOAD, >>> the first. The 2nd PT_LOAD is for sections without the readonly = status, >>> that includes .got and .plt being spanned. See below from >>> objdump -ph for /bin/ls : >>>=20 >>> Program Header: >>> PHDR off 0x0000000000000040 vaddr 0x0000000010000040 paddr = 0x0000000010000040 align 2**3 >>> filesz 0x0000000000000188 memsz 0x0000000000000188 flags r-- >>> INTERP off 0x00000000000001c8 vaddr 0x00000000100001c8 paddr = 0x00000000100001c8 align 2**0 >>> filesz 0x0000000000000015 memsz 0x0000000000000015 flags r-- >>> LOAD off 0x0000000000000000 vaddr 0x0000000010000000 paddr = 0x0000000010000000 align 2**16 >>> filesz 0x000000000000910c memsz 0x000000000000910c flags r-x >>> LOAD off 0x0000000000009110 vaddr 0x0000000010019110 paddr = 0x0000000010019110 align 2**16 >>> filesz 0x0000000000000ee0 memsz 0x00000000000010e8 flags rw- >>> DYNAMIC off 0x0000000000009138 vaddr 0x0000000010019138 paddr = 0x0000000010019138 align 2**3 >>> filesz 0x00000000000001c0 memsz 0x00000000000001c0 flags rw- >>> NOTE off 0x00000000000001e0 vaddr 0x00000000100001e0 paddr = 0x00000000100001e0 align 2**2 >>> filesz 0x0000000000000030 memsz 0x0000000000000030 flags r-- >>> STACK off 0x0000000000000000 vaddr 0x0000000000000000 paddr = 0x0000000000000000 align 2**4 >>> filesz 0x0000000000000000 memsz 0x0000000000000000 flags rw- >>=20 >> We only need program headers, and we only need them from the object >> which load causes the fault. It might be not the binary but some = library >> that triggers the fault. So the backtrace and some information from = the >> state of the image is needed. >>=20 >> You can build only rtld and use it as the standalone program to = initiate >> the image activation: >> /ld-elf.so.1 /bin/ls >> or similar. >=20 > . . . It stops when the dcbst in __syncicache runs into an address in the p_align 65536 caused hole between the two PT_LOAD's with PF_X. /bin/ls itself has such a hole, as do the .so libraries involved. Details follow. (gdb) run /bin/ls Starting program: = /usr/obj/powerpcvtsc_gcc421/powerpc.powerpc/usr/src/powerpc.powerpc/libexe= c/rtld-elf/ld-elf.so.1.full /bin/ls Program received signal SIGSEGV, Segmentation fault. __syncicache (from=3D0x1800000, len=3D102400) at = /usr/src/lib/libc/powerpc/gen/syncicache.c:94 94 __asm __volatile ("dcbst 0,%0" :: "r"(p)); (gdb) bt #0 __syncicache (from=3D0x1800000, len=3D102400) at = /usr/src/lib/libc/powerpc/gen/syncicache.c:94 #1 0x01012b58 in reloc_non_plt (obj=3D0x41041000, obj_rtld=3D, flags=3D4, lockstate=3D0x0) at = /usr/src/libexec/rtld-elf/powerpc/reloc.c:330 #2 0x010175f4 in relocate_object (obj=3D0x41041000, bind_now=3D0 = '\000', rtldobj=3D0x103ea88 , flags=3D4, lockstate=3D0x0) at = /usr/src/libexec/rtld-elf/rtld.c:2849 #3 0x0101776c in relocate_objects (first=3D, bind_now=3D0 = '\000', rtldobj=3D0x103ea88 , flags=3D4, lockstate=3D0x0) at = /usr/src/libexec/rtld-elf/rtld.c:2908 #4 0x0101beb0 in _rtld (sp=3D, exit_proc=3D0xffffdc08, = objp=3D0xffffdc0c) at /usr/src/libexec/rtld-elf/rtld.c:677 #5 0x0101217c in .rtld_start () at = /usr/src/libexec/rtld-elf/powerpc/rtld_start.S:98 Backtrace stopped: frame did not save the PC This is because there are 2 PT_LOAD's that have PF_X but for which there is an address range between that spans may pages for which there is no header entry for, in part a consequence of p_align being 65536 so forcing lots of space between entries 2 and 3 ( /bin/ls example): entry: 2 p_type: PT_LOAD p_offset: 0 p_vaddr: 0x1800000 p_paddr: 0x1800000 p_filesz: 34112 p_memsz: 34112 p_flags: PF_X|PF_R p_align: 65536 entry: 3 p_type: PT_LOAD p_offset: 34112 p_vaddr: 0x1818540 p_paddr: 0x1818540 p_filesz: 316 p_memsz: 1752 p_flags: PF_X|PF_W|PF_R p_align: 65536 ( check 2's p_paddr+ p_memsz vs. 3's p_vaddr Later below the failing address will be shown as between those. ) [powerpc64 only gets one PT_LOAD with PF_X and so can not have space between such PT_LOAD's as things currently are.] The source code related details are as follows and shows that it was processing /bin/ls itself ( obj_main->path =3D 0x41042000 "/bin/ls" and the failing address ) : [The presentation order is not necessarily the same as my exploration order.] (gdb) list 89 off =3D (u_int)from & (cacheline_size - 1); 90 l =3D len +=3D off; 91 p =3D (char *)from - off; 92=09 93 do { 94 __asm __volatile ("dcbst 0,%0" :: "r"(p)); 95 p +=3D cacheline_size; 96 } while ((l -=3D cacheline_size) > 0); 97 __asm __volatile ("sync"); 98 p =3D (char *)from - off; [Reminder for entry 2: p_vaddr: 0x1800000 p_memsz: 34112 0x1800000+34112 =3D 0x1808540 ] (gdb) print p $4 =3D 0x1809000 (gdb) up #1 0x01012b58 in reloc_non_plt (obj=3D0x41041000, obj_rtld=3D, flags=3D4, lockstate=3D0x0) at = /usr/src/libexec/rtld-elf/powerpc/reloc.c:330 330 __syncicache(obj->mapbase, obj->textsize); (gdb) list 325 done: 326 if (cache !=3D NULL) 327 free(cache); 328=09 329 /* Synchronize icache for text seg in case we made any = changes */ 330 __syncicache(obj->mapbase, obj->textsize); 331=09 332 return (r); 333 } 334=09 (gdb) up #2 0x010175f4 in relocate_object (obj=3D0x41041000, bind_now=3D0 = '\000', rtldobj=3D0x103ea88 , flags=3D4, lockstate=3D0x0) at = /usr/src/libexec/rtld-elf/rtld.c:2849 2849 if (reloc_non_plt(obj, rtldobj, flags, lockstate)) (gdb) list 2844 /* There are relocations to the write-protected text = segment. */ 2845 if (obj->textrel && reloc_textrel_prot(obj, true) !=3D = 0) 2846 return (-1); 2847=09 2848 /* Process the non-PLT non-IFUNC relocations. */ 2849 if (reloc_non_plt(obj, rtldobj, flags, lockstate)) 2850 return (-1); 2851=09 2852 /* Re-protected the text segment. */ 2853 if (obj->textrel && reloc_textrel_prot(obj, false) !=3D = 0) (gdb) up =20 #3 0x0101776c in relocate_objects (first=3D, bind_now=3D0 = '\000', rtldobj=3D0x103ea88 , flags=3D4, lockstate=3D0x0) at = /usr/src/libexec/rtld-elf/rtld.c:2908 2908 error =3D relocate_object(obj, bind_now, = rtldobj, flags, (gdb) list 2903=09 2904 for (error =3D 0, obj =3D first; obj !=3D NULL; 2905 obj =3D TAILQ_NEXT(obj, next)) { 2906 if (obj->marker) 2907 continue; 2908 error =3D relocate_object(obj, bind_now, = rtldobj, flags, 2909 lockstate); 2910 if (error =3D=3D -1) 2911 break; 2912 } (gdb) up #4 0x0101beb0 in _rtld (sp=3D, exit_proc=3D0xffffdc08, = objp=3D0xffffdc0c) at /usr/src/libexec/rtld-elf/rtld.c:677 677 if (relocate_objects(obj_main, (gdb) list 672 * block even if they didn't ask for it. 673 */ 674 allocate_tls_offset(entry->obj); 675 } 676=09 677 if (relocate_objects(obj_main, 678 ld_bind_now !=3D NULL && *ld_bind_now !=3D '\0', 679 &obj_rtld, SYMLOOK_EARLY, NULL) =3D=3D -1) 680 rtld_die(); 681=09 (gdb) up =20 #5 0x0101217c in .rtld_start () at = /usr/src/libexec/rtld-elf/powerpc/rtld_start.S:98 98 bl _rtld@plt /* &_start =3D _rtld(sp, = &exit_proc, &obj_main)*/ (gdb) list 93 addi %r3,%r4,-4 /* locate argc ptr, &argv[-1] */ 94=09 95 addi %r4,%r1,8 /* &exit_proc on stack */ 96 addi %r5,%r1,12 /* &obj_main on stack */ 97=09 98 bl _rtld@plt /* &_start =3D _rtld(sp, = &exit_proc, &obj_main)*/ 99 mtlr %r3 100=09 101 /* 102 * Restore args, with new obj/exit proc (gdb) down #4 0x0101beb0 in _rtld (sp=3D, exit_proc=3D0xffffdc08, = objp=3D0xffffdc0c) at /usr/src/libexec/rtld-elf/rtld.c:677 677 if (relocate_objects(obj_main, . . . (gdb) print *obj_main $3 =3D {magic =3D 0, version =3D 0, next =3D {tqe_next =3D 0x41041200, = tqe_prev =3D 0x103ea68 }, path =3D 0x41042000 "/bin/ls", = origin_path =3D 0x0, refcount =3D 1, holdcount =3D 0, dl_refcount =3D 0,=20= mapbase =3D 0x1800000 "\177ELF\001\002\001\t", mapsize =3D 102400, = textsize =3D 102400, vaddrbase =3D 25165824, relocbase =3D 0x0, dynamic = =3D 0x1818554,=20 entry =3D 0x18022e0 "|\b\002\246\224!\377\340=3D \001\202\223\241", = phdr =3D 0x1800034, phsize =3D 224, interp =3D 0x1800114 = "/libexec/ld-elf.so.1", stack_flags =3D 6, tlsindex =3D 0, tlsinit =3D = 0x0,=20 tlsinitsize =3D 0, tlssize =3D 0, tlsoffset =3D 0, tlsalign =3D 0, = relro_page =3D 0x0, relro_size =3D 0, pltgot =3D 0x18186f4, rel =3D 0x0, = relsize =3D 0, rela =3D 0x1801ebc, relasize =3D 1008, pltrel =3D 0x0,=20 pltrelsize =3D 0, pltrela =3D 0x1801f28, pltrelasize =3D 900, symtab =3D= 0x1800798, strtab =3D 0x1801418 "", strsize =3D 2241, verneed =3D = 0x1801e6c, verneednum =3D 1, verdef =3D 0x0, verdefnum =3D 0,=20 versyms =3D 0x1801cda, buckets =3D 0x1800164, nbuckets =3D 197, chains = =3D 0x1800478, nchains =3D 200, nbuckets_gnu =3D 0, symndx_gnu =3D 0, = maskwords_bm_gnu =3D 0, shift2_gnu =3D 0, dynsymcount =3D 200,=20 bloom_gnu =3D 0x0, buckets_gnu =3D 0x0, chain_zero_gnu =3D 0x0, rpath = =3D 0x0, runpath =3D 0x0, needed =3D 0x41042010, needed_filtees =3D 0x0, = needed_aux_filtees =3D 0x0, names =3D {stqh_first =3D 0x0,=20 stqh_last =3D 0x410410f8}, vertab =3D 0x41043080, vernum =3D 6, init = =3D 25174700, fini =3D 25197908, preinit_array =3D 0, init_array =3D 0, = fini_array =3D 0, preinit_array_num =3D 0, init_array_num =3D 0,=20 fini_array_num =3D 0, osrel =3D 1200084, mainprog =3D 1 '\001', rtld =3D= 0 '\000', relocated =3D 1 '\001', ver_checked =3D 1 '\001', textrel =3D = 0 '\000', symbolic =3D 0 '\000', bind_now =3D 0 '\000',=20 traced =3D 0 '\000', jmpslots_done =3D 0 '\000', init_done =3D 0 = '\000', tls_done =3D 1 '\001', phdr_alloc =3D 0 '\000', z_origin =3D 0 = '\000', z_nodelete =3D 0 '\000', z_noopen =3D 0 '\000',=20 z_loadfltr =3D 0 '\000', z_interpose =3D 0 '\000', z_nodeflib =3D 0 = '\000', z_global =3D 0 '\000', ref_nodel =3D 0 '\000', init_scanned =3D = 0 '\000', on_fini_list =3D 0 '\000', dag_inited =3D 0 '\000',=20 filtees_loaded =3D 0 '\000', irelative =3D 0 '\000', gnu_ifunc =3D 0 = '\000', non_plt_gnu_ifunc =3D 0 '\000', crt_no_init =3D 1 '\001', = valid_hash_sysv =3D 1 '\001', valid_hash_gnu =3D 0 '\000',=20 dlopened =3D 0 '\000', marker =3D 0 '\000', unholdfree =3D 0 '\000', = doomed =3D 0 '\000', linkmap =3D {l_addr =3D 0x1800000 = "\177ELF\001\002\001\t", l_name =3D 0x41042000 "/bin/ls", l_ld =3D = 0x1818554,=20 l_next =3D 0x41041334, l_prev =3D 0x0}, dldags =3D {stqh_first =3D = 0x0, stqh_last =3D 0x41041148}, dagmembers =3D {stqh_first =3D 0x0, = stqh_last =3D 0x41041150}, dev =3D 0, ino =3D 0, priv =3D 0x0} =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)