Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 2 Nov 2018 00:16:23 -0700
From:      Mark Millard <marklmi26-fbsd@yahoo.com>
To:        Konstantin Belousov <kostikbel@gmail.com>
Cc:        svn-src-head@freebsd.org, Alexander Richardson <arichardson@freebsd.org>, Shawn Webb <shawn.webb@hardenedbsd.org>
Subject:   Re: svn commit: r339876 - head/libexec/rtld-elf
Message-ID:  <003A49D7-6E8B-4775-A70B-E0EB44505D4B@yahoo.com>
In-Reply-To: <E44F5772-1F8A-40B8-9C4E-B8362B768F37@yahoo.com>
References:  <8E5A5F3A-F1A7-4702-A2F7-65D74CC5B2E5@yahoo.com> <20181102004101.GI5335@kib.kiev.ua> <E44F5772-1F8A-40B8-9C4E-B8362B768F37@yahoo.com>

next in thread | previous in thread | raw e-mail | index | archive | help
[The backtrace confirms what I reported to Alexander Richardson
and Shawn Webb earlier.]

On 2018-Nov-1, at 6:40 PM, Mark Millard <marklmi26-fbsd at yahoo.com> =
wrote:

> On 2018-Nov-1, at 5:41 PM, Konstantin Belousov <kostikbel at =
gmail.com> wrote:
>=20
>> On Tue, Oct 30, 2018 at 12:45:13PM -0700, Mark Millard wrote:
>>> Konstantin Belousov kostikbel at gmail.com wrote on
>>> Tue Oct 30 18:04:04 UTC 2018 :
>>>=20
>>>> On Tue, Oct 30, 2018 at 03:32:40PM +0000, Alexander Richardson =
wrote:
>>>>> On Tue, 30 Oct 2018 at 10:17, Michael Tuexen
>>>>> <Michael.Tuexen at macmic.franken.de> wrote:
>>>>>>=20
>>>>>>> On 29. Oct 2018, at 22:08, Alex Richardson <arichardson at =
FreeBSD.org> wrote:
>>>>>>>=20
>>>>>>> Author: arichardson
>>>>>>> Date: Mon Oct 29 21:08:02 2018
>>>>>>> New Revision: 339876
>>>>>>> URL: https://svnweb.freebsd.org/changeset/base/339876
>>>>>>>=20
>>>>>>> Log:
>>>>>>> rtld: set obj->textsize correctly
>>>>>>>=20
>>>>>>> With lld-generated binaries the first PT_LOAD will usually be a =
read-only
>>>>>>> segment unless you pass --no-rosegment. For those binaries the =
textsize is
>>>>>>> determined by the next PT_LOAD. To allow both LLD and bfd 2.17 =
binaries to
>>>>>>> be parsed correctly use the end of the last PT_LOAD that is =
marked as
>>>>>>> executable instead.
>>>>>>>=20
>>>>>>> I noticed that the value was wrong while adding some debug =
prints for some rtld
>>>>>>> changes for CHERI binaries. `obj->textsize` only seems to be =
used by PPC so the
>>>>>>> effect is untested. However, the value before was definitely =
wrong and the new
>>>>>>> result matches the phdrs.
>>>>>> I build kernel and world with a revision later than this on a =
PPC. Buildword
>>>>>> ends up with a world where almost all binaries are =
segfaulting.... Especially gdb
>>>>>> (but svn, ls or so all segfault).
>>>>>>=20
>>>>>> Best regards
>>>>>> Michael
>>>>>=20
>>>>> This is rather surprising since if anything the range of the =
icache
>>>>> flush should increase rather than decrease after this change.
>>>>>=20
>>>>> I can only see this causing a behaviour change if we actually need =
to
>>>>> flush more than just the executable segments.
>>>>> Is it possible that some binary/library contains a non-executable
>>>>> segment as the first PT_LOAD?
>>>>> Or is there some linker script that adds custom PHDRS?
>>>>>=20
>>>> Could it be that there is a hole between start of the object =
mapping and
>>>> the last PT_LOADable segment eligible for execution ?
>>>=20
>>> [This note may be easier to deal with than the first
>>> note that I sent out.]
>>>=20
>>> [My examples are from devel/powerpc64-xtoolchain-gcc used
>>> to buildworld buildkernel targeting a head -r339076 based
>>> powerpc64 environment. I do that on powerpc64 as well.]
>>>=20
>>> powerpc64 loads the readonly data and the readonly code in one =
PT_LOAD,
>>> the first. The 2nd PT_LOAD is for sections without the readonly =
status,
>>> that includes .got and .plt being spanned. See below from
>>> objdump -ph for /bin/ls :
>>>=20
>>> Program Header:
>>>   PHDR off    0x0000000000000040 vaddr 0x0000000010000040 paddr =
0x0000000010000040 align 2**3
>>>        filesz 0x0000000000000188 memsz 0x0000000000000188 flags r--
>>> INTERP off    0x00000000000001c8 vaddr 0x00000000100001c8 paddr =
0x00000000100001c8 align 2**0
>>>        filesz 0x0000000000000015 memsz 0x0000000000000015 flags r--
>>>   LOAD off    0x0000000000000000 vaddr 0x0000000010000000 paddr =
0x0000000010000000 align 2**16
>>>        filesz 0x000000000000910c memsz 0x000000000000910c flags r-x
>>>   LOAD off    0x0000000000009110 vaddr 0x0000000010019110 paddr =
0x0000000010019110 align 2**16
>>>        filesz 0x0000000000000ee0 memsz 0x00000000000010e8 flags rw-
>>> DYNAMIC off    0x0000000000009138 vaddr 0x0000000010019138 paddr =
0x0000000010019138 align 2**3
>>>        filesz 0x00000000000001c0 memsz 0x00000000000001c0 flags rw-
>>>   NOTE off    0x00000000000001e0 vaddr 0x00000000100001e0 paddr =
0x00000000100001e0 align 2**2
>>>        filesz 0x0000000000000030 memsz 0x0000000000000030 flags r--
>>>  STACK off    0x0000000000000000 vaddr 0x0000000000000000 paddr =
0x0000000000000000 align 2**4
>>>        filesz 0x0000000000000000 memsz 0x0000000000000000 flags rw-
>>=20
>> We only need program headers, and we only need them from the object
>> which load causes the fault.  It might be not the binary but some =
library
>> that triggers the fault.  So the backtrace and some information from =
the
>> state of the image is needed.
>>=20
>> You can build only rtld and use it as the standalone program to =
initiate
>> the image activation:
>> 	<path>/ld-elf.so.1 /bin/ls
>> or similar.
>=20
> . . .

It stops when the dcbst in __syncicache runs into an address in
the p_align 65536 caused hole between the two PT_LOAD's with PF_X.
/bin/ls itself has such a hole, as do the .so libraries involved.

Details follow.

(gdb) run /bin/ls
Starting program: =
/usr/obj/powerpcvtsc_gcc421/powerpc.powerpc/usr/src/powerpc.powerpc/libexe=
c/rtld-elf/ld-elf.so.1.full /bin/ls

Program received signal SIGSEGV, Segmentation fault.
__syncicache (from=3D0x1800000, len=3D102400) at =
/usr/src/lib/libc/powerpc/gen/syncicache.c:94
94			__asm __volatile ("dcbst 0,%0" :: "r"(p));
(gdb) bt
#0  __syncicache (from=3D0x1800000, len=3D102400) at =
/usr/src/lib/libc/powerpc/gen/syncicache.c:94
#1  0x01012b58 in reloc_non_plt (obj=3D0x41041000, obj_rtld=3D<optimized =
out>, flags=3D4, lockstate=3D0x0) at =
/usr/src/libexec/rtld-elf/powerpc/reloc.c:330
#2  0x010175f4 in relocate_object (obj=3D0x41041000, bind_now=3D0 =
'\000', rtldobj=3D0x103ea88 <obj_rtld>, flags=3D4, lockstate=3D0x0) at =
/usr/src/libexec/rtld-elf/rtld.c:2849
#3  0x0101776c in relocate_objects (first=3D<optimized out>, bind_now=3D0 =
'\000', rtldobj=3D0x103ea88 <obj_rtld>, flags=3D4, lockstate=3D0x0) at =
/usr/src/libexec/rtld-elf/rtld.c:2908
#4  0x0101beb0 in _rtld (sp=3D<optimized out>, exit_proc=3D0xffffdc08, =
objp=3D0xffffdc0c) at /usr/src/libexec/rtld-elf/rtld.c:677
#5  0x0101217c in .rtld_start () at =
/usr/src/libexec/rtld-elf/powerpc/rtld_start.S:98
Backtrace stopped: frame did not save the PC

This is because there are 2 PT_LOAD's that have PF_X but
for which there is an address range between that spans
may pages for which there is no header entry for, in part
a consequence of p_align being 65536 so forcing lots of
space between entries 2 and 3 ( /bin/ls example):

entry: 2
       p_type: PT_LOAD
       p_offset: 0
       p_vaddr: 0x1800000
       p_paddr: 0x1800000
       p_filesz: 34112
       p_memsz: 34112
       p_flags: PF_X|PF_R
       p_align: 65536

entry: 3
       p_type: PT_LOAD
       p_offset: 34112
       p_vaddr: 0x1818540
       p_paddr: 0x1818540
       p_filesz: 316
       p_memsz: 1752
       p_flags: PF_X|PF_W|PF_R
       p_align: 65536

( check 2's p_paddr+ p_memsz vs. 3's p_vaddr
  Later below the failing address will be shown
  as between those. )

[powerpc64 only gets one PT_LOAD with PF_X and
so can not have space between such PT_LOAD's
as things currently are.]

The source code related details are as follows
and shows that it was processing /bin/ls itself
( obj_main->path =3D 0x41042000 "/bin/ls" and
  the failing address ) :


[The presentation order is not necessarily the same
as my exploration order.]

(gdb) list
89		off =3D (u_int)from & (cacheline_size - 1);
90		l =3D len +=3D off;
91		p =3D (char *)from - off;
92=09
93		do {
94			__asm __volatile ("dcbst 0,%0" :: "r"(p));
95			p +=3D cacheline_size;
96		} while ((l -=3D cacheline_size) > 0);
97		__asm __volatile ("sync");
98		p =3D (char *)from - off;

[Reminder for entry 2:
       p_vaddr: 0x1800000
       p_memsz: 34112
 0x1800000+34112 =3D 0x1808540
]

(gdb) print p
$4 =3D 0x1809000 <error: Cannot access memory at address 0x1809000>

(gdb) up
#1  0x01012b58 in reloc_non_plt (obj=3D0x41041000, obj_rtld=3D<optimized =
out>, flags=3D4, lockstate=3D0x0) at =
/usr/src/libexec/rtld-elf/powerpc/reloc.c:330
330		__syncicache(obj->mapbase, obj->textsize);
(gdb) list
325	done:
326		if (cache !=3D NULL)
327			free(cache);
328=09
329		/* Synchronize icache for text seg in case we made any =
changes */
330		__syncicache(obj->mapbase, obj->textsize);
331=09
332		return (r);
333	}
334=09
(gdb) up
#2  0x010175f4 in relocate_object (obj=3D0x41041000, bind_now=3D0 =
'\000', rtldobj=3D0x103ea88 <obj_rtld>, flags=3D4, lockstate=3D0x0) at =
/usr/src/libexec/rtld-elf/rtld.c:2849
2849		if (reloc_non_plt(obj, rtldobj, flags, lockstate))
(gdb) list
2844		/* There are relocations to the write-protected text =
segment. */
2845		if (obj->textrel && reloc_textrel_prot(obj, true) !=3D =
0)
2846			return (-1);
2847=09
2848		/* Process the non-PLT non-IFUNC relocations. */
2849		if (reloc_non_plt(obj, rtldobj, flags, lockstate))
2850			return (-1);
2851=09
2852		/* Re-protected the text segment. */
2853		if (obj->textrel && reloc_textrel_prot(obj, false) !=3D =
0)
(gdb) up    =20
#3  0x0101776c in relocate_objects (first=3D<optimized out>, bind_now=3D0 =
'\000', rtldobj=3D0x103ea88 <obj_rtld>, flags=3D4, lockstate=3D0x0) at =
/usr/src/libexec/rtld-elf/rtld.c:2908
2908			error =3D relocate_object(obj, bind_now, =
rtldobj, flags,
(gdb) list
2903=09
2904		for (error =3D 0, obj =3D first;  obj !=3D NULL;
2905		    obj =3D TAILQ_NEXT(obj, next)) {
2906			if (obj->marker)
2907				continue;
2908			error =3D relocate_object(obj, bind_now, =
rtldobj, flags,
2909			    lockstate);
2910			if (error =3D=3D -1)
2911				break;
2912		}
(gdb) up
#4  0x0101beb0 in _rtld (sp=3D<optimized out>, exit_proc=3D0xffffdc08, =
objp=3D0xffffdc0c) at /usr/src/libexec/rtld-elf/rtld.c:677
677	    if (relocate_objects(obj_main,
(gdb) list
672		 * block even if they didn't ask for it.
673		 */
674		allocate_tls_offset(entry->obj);
675	    }
676=09
677	    if (relocate_objects(obj_main,
678	      ld_bind_now !=3D NULL && *ld_bind_now !=3D '\0',
679	      &obj_rtld, SYMLOOK_EARLY, NULL) =3D=3D -1)
680		rtld_die();
681=09
(gdb) up =20
#5  0x0101217c in .rtld_start () at =
/usr/src/libexec/rtld-elf/powerpc/rtld_start.S:98
98		bl      _rtld@plt	/* &_start =3D _rtld(sp, =
&exit_proc, &obj_main)*/
(gdb) list
93	        addi    %r3,%r4,-4	/* locate argc ptr, &argv[-1] */
94=09
95		addi	%r4,%r1,8	/* &exit_proc on stack */
96		addi	%r5,%r1,12	/* &obj_main on stack */
97=09
98		bl      _rtld@plt	/* &_start =3D _rtld(sp, =
&exit_proc, &obj_main)*/
99		mtlr    %r3
100=09
101		/*
102		 * Restore args, with new obj/exit proc
(gdb) down
#4  0x0101beb0 in _rtld (sp=3D<optimized out>, exit_proc=3D0xffffdc08, =
objp=3D0xffffdc0c) at /usr/src/libexec/rtld-elf/rtld.c:677
677	    if (relocate_objects(obj_main,
. . .
(gdb) print *obj_main
$3 =3D {magic =3D 0, version =3D 0, next =3D {tqe_next =3D 0x41041200, =
tqe_prev =3D 0x103ea68 <obj_list>}, path =3D 0x41042000 "/bin/ls", =
origin_path =3D 0x0, refcount =3D 1, holdcount =3D 0, dl_refcount =3D 0,=20=

  mapbase =3D 0x1800000 "\177ELF\001\002\001\t", mapsize =3D 102400, =
textsize =3D 102400, vaddrbase =3D 25165824, relocbase =3D 0x0, dynamic =
=3D 0x1818554,=20
  entry =3D 0x18022e0 "|\b\002\246\224!\377\340=3D \001\202\223\241", =
phdr =3D 0x1800034, phsize =3D 224, interp =3D 0x1800114 =
"/libexec/ld-elf.so.1", stack_flags =3D 6, tlsindex =3D 0, tlsinit =3D =
0x0,=20
  tlsinitsize =3D 0, tlssize =3D 0, tlsoffset =3D 0, tlsalign =3D 0, =
relro_page =3D 0x0, relro_size =3D 0, pltgot =3D 0x18186f4, rel =3D 0x0, =
relsize =3D 0, rela =3D 0x1801ebc, relasize =3D 1008, pltrel =3D 0x0,=20
  pltrelsize =3D 0, pltrela =3D 0x1801f28, pltrelasize =3D 900, symtab =3D=
 0x1800798, strtab =3D 0x1801418 "", strsize =3D 2241, verneed =3D =
0x1801e6c, verneednum =3D 1, verdef =3D 0x0, verdefnum =3D 0,=20
  versyms =3D 0x1801cda, buckets =3D 0x1800164, nbuckets =3D 197, chains =
=3D 0x1800478, nchains =3D 200, nbuckets_gnu =3D 0, symndx_gnu =3D 0, =
maskwords_bm_gnu =3D 0, shift2_gnu =3D 0, dynsymcount =3D 200,=20
  bloom_gnu =3D 0x0, buckets_gnu =3D 0x0, chain_zero_gnu =3D 0x0, rpath =
=3D 0x0, runpath =3D 0x0, needed =3D 0x41042010, needed_filtees =3D 0x0, =
needed_aux_filtees =3D 0x0, names =3D {stqh_first =3D 0x0,=20
    stqh_last =3D 0x410410f8}, vertab =3D 0x41043080, vernum =3D 6, init =
=3D 25174700, fini =3D 25197908, preinit_array =3D 0, init_array =3D 0, =
fini_array =3D 0, preinit_array_num =3D 0, init_array_num =3D 0,=20
  fini_array_num =3D 0, osrel =3D 1200084, mainprog =3D 1 '\001', rtld =3D=
 0 '\000', relocated =3D 1 '\001', ver_checked =3D 1 '\001', textrel =3D =
0 '\000', symbolic =3D 0 '\000', bind_now =3D 0 '\000',=20
  traced =3D 0 '\000', jmpslots_done =3D 0 '\000', init_done =3D 0 =
'\000', tls_done =3D 1 '\001', phdr_alloc =3D 0 '\000', z_origin =3D 0 =
'\000', z_nodelete =3D 0 '\000', z_noopen =3D 0 '\000',=20
  z_loadfltr =3D 0 '\000', z_interpose =3D 0 '\000', z_nodeflib =3D 0 =
'\000', z_global =3D 0 '\000', ref_nodel =3D 0 '\000', init_scanned =3D =
0 '\000', on_fini_list =3D 0 '\000', dag_inited =3D 0 '\000',=20
  filtees_loaded =3D 0 '\000', irelative =3D 0 '\000', gnu_ifunc =3D 0 =
'\000', non_plt_gnu_ifunc =3D 0 '\000', crt_no_init =3D 1 '\001', =
valid_hash_sysv =3D 1 '\001', valid_hash_gnu =3D 0 '\000',=20
  dlopened =3D 0 '\000', marker =3D 0 '\000', unholdfree =3D 0 '\000', =
doomed =3D 0 '\000', linkmap =3D {l_addr =3D 0x1800000 =
"\177ELF\001\002\001\t", l_name =3D 0x41042000 "/bin/ls", l_ld =3D =
0x1818554,=20
    l_next =3D 0x41041334, l_prev =3D 0x0}, dldags =3D {stqh_first =3D =
0x0, stqh_last =3D 0x41041148}, dagmembers =3D {stqh_first =3D 0x0, =
stqh_last =3D 0x41041150}, dev =3D 0, ino =3D 0, priv =3D 0x0}

=3D=3D=3D
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?003A49D7-6E8B-4775-A70B-E0EB44505D4B>