Date: Fri, 27 Oct 2017 20:22:25 +0200 From: Michal Meloun <melounmichal@gmail.com> To: Brooks Davis <brooks@freebsd.org>, Konstantin Belousov <kostikbel@gmail.com> Cc: Dimitry Andric <dim@FreeBSD.org>, src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org, kib@freebsd.org Subject: Re: svn commit: r324938 - head/contrib/jemalloc/include/jemalloc/internal Message-ID: <9b85bde5-c450-94c7-a31e-f2eb7f6ab9ee@freebsd.org> In-Reply-To: <20171027152154.GA31598@spindle.one-eyed-alien.net> References: <201710232131.v9NLV4Rb068825@repo.freebsd.org> <38db6f4e-72b8-6ffd-4529-f15ca32bad54@freebsd.org> <6FD27DFB-5039-4E33-B131-EF5391DD1630@FreeBSD.org> <6eff6e66-4987-8753-105f-b6a5b8234ff3@freebsd.org> <20171027150841.GH2566@kib.kiev.ua> <20171027152154.GA31598@spindle.one-eyed-alien.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Oct 27, 2017 at 19:48:43 +0200, Michal Meloun wrote:
> On 27.10.2017 17:21, Brooks Davis wrote:
>> On Fri, Oct 27, 2017 at 06:08:41PM +0300, Konstantin Belousov
>> wrote:
>>> On Fri, Oct 27, 2017 at 02:53:26PM +0200, Michal Meloun wrote:
>>>> Sorry for top posting That's pity, we have clear problem in
>>>> rtld code :( See:
>>>> ----------------------------------------------------- RESCUE
>>>> WITHOUT JEMALLOC_ALIGNED(16);
>>>> ----------------------------------------------------- Program
>>>> Headers: TLS 0xa732b0 0x00a832b0 0x00a832b0 0x00b40
>>>> 0x011bc R 0x8 Section Headers: 04 .tdata .tbss
>>>> .init_array .fini_array .jcr .got Dump: 00a832b0
>>>> <__je_tsd_tls+0xa832b0>: a832b0: 00000005
>>>>
>>>> GDB (gdb) b tsd_fetch_impl Breakpoint 1 at 0x7c4c08:
>>>> tsd_fetch_impl. (6 locations) (gdb) r Starting program:
>>>> /usr/src/rescue.noalign sh
>>>>
>>>> Breakpoint 1, tsd_fetch_impl (init=true, minimal=false) at
>>>> /usr/src/contrib/jemalloc/include/jemalloc/internal/tsd.h:261
>>>> 261 tsd_t *tsd = tsd_get(init); (gdb) n 263
>>>> if (!init && tsd_get_allocates() && tsd == NULL) {
>>>>
>>>> (gdb) p tsd $1 = (tsd_t *) 0x20c83008
>>>>
>>>> (gdb) p *tsd $2 = {state = 5 '\005', ....
>>>>
>>>> (gdb) p *((tsd_t *)0x00a832b0) $3 = {state = 5 '\005', ...
>>>>
>>>>
>>>>
>>>> ----------------------------------------------------- RESCUE
>>>> WITH JEMALLOC_ALIGNED(16);
>>>> ----------------------------------------------------- Program
>>>> Headers: TLS 0xa732b0 0x00a832b0 0x00a832b0 0x00b40
>>>> 0x011bc R 0x10 Section Headers: 04 .tdata .tbss
>>>> .init_array .fini_array .jcr .got Dump: 00a832b0
>>>> <__je_tsd_tls+0xa832b0>: a832b0: 00000005
>>>>
>>>> GDB (gdb) b tsd_fetch_impl Breakpoint 1 at 0x7c4c08:
>>>> tsd_fetch_impl. (6 locations) (gdb) r Starting program:
>>>> /usr/obj/usr/src/rescue/rescue/rescue sh Breakpoint 1,
>>>> tsd_fetch_impl (init=true, minimal=false) at
>>>> /usr/src/contrib/jemalloc/include/jemalloc/internal/tsd.h:261
>>>> 261 tsd_t *tsd = tsd_get(init); (gdb) n 263
>>>> if (!init && tsd_get_allocates() && tsd == NULL) {
>>>>
>>>> (gdb) p tsd $1 = (tsd_t *) 0x20c83010
>>>>
>>>> (gdb) p *tsd $2 = {state = 0 '\000', ...
>>>>
>>>> (gdb) p *((tsd_t *)0x00a832b0) $3 = {state = 5 '\005', ...
>>>>
>>>> !!!! BUT p *(tsd - 8 bytes) !!!!!!!!!! (gdb) p *((tsd_t
>>>> *)0x20c83008) $4 = {state = 5 '\005', ...
>>>>
>>>> ----------------------------------------------------- So it's
>>>> clear that:
>>>>
>>>> - both binaries are valid, .tdata section have valid data. -
>>>> requested alignment is propagated to binary. - .tdata section
>>>> is properly loaded to memory because p *((tsd_t *)0x00a832b0)
>>>> is {state = 5 '\005' in both cases
>>>>
>>>> - a per thread copy of .tdata respect requested alignment but
>>>> the original data was copied to unaligned address. because for
>>>> aligned binary p *tsd is {state = 0 '\000', ... p *(tsd - 8
>>>> bytes) is {state = 5 '\005'
>>>>
>>>> I'm right? Kib, please, can you help us?
>>>
>>> Does it happen for rescue binary ?
>>>
>>> Note that the binary is linked static, so the problem is in
>>> lib/libc/gen/tls.c and not in rtld. There, I do not see any real
>>> use of the phdr' p_align value.
>>>
> Ahh, right, good catch.
>>> BTW, is rescue linked to libthr ?
> Imho not.
My bad, rescue *is* linked with libthr. And after looking to
libthr/arch/arm/pthread_md.h, I'm confused much much more...
>>
>> There isn't alignment support for TLS in static binaries. I've
>> fixed this in CheriBSD and am planning to upstream the fixes at
>> some point. The fix for variant I is in:
>>
>> https://github.com/CTSRD-CHERI/cheribsd/commit/3cfb124ebb9fdb545dad8436a04dd58c05b33f4b
>>
>>
> The real alignment have no effect for program itself, it's important
> only for AddressSanitizer.
>
> But there is something what's still missing me.
> The rescue binary is statically linked, so whole program have only one
> TLS section and full TLS layout is determined by linker.
> The problematic structure, tsd_tls, is first one in TLS section group:
>
> readelf -a /usr/obj/usr/src/rescue/rescue/rescue | grep TLS
> 518961: 0000000000000000 2880 TLS GLOBAL DEFAULT 9 __je_tsd_tls
> and
>
> objdump -d -j .tdata /usr/obj/usr/src/rescue/rescue/rescue
> 00a832b0 <__je_tsd_tls+0xa832b0>:
> a832b0: 00000005
>
> but disassembled code expect it at TLS ptr + 0x8 (for code without
> JEMALLOC_ALIGNED) or at TLS ptr + 0x10 (for code with JEMALLOC_ALIGNED)
>
> [without JEMALLOC_ALIGNED]
> 7c4d90: e59f0014 ldr r0, [pc, #20] ; 7c4dac
> <tsd_get+0x80>
> 7c4d94: e58d0004 str r0, [sp, #4]
> 7c4d98: eb031d14 bl 88c1f0 <__aeabi_read_tp>
> 7c4d9c: e59dc004 ldr ip, [sp, #4]
> 7c4da0: e080000c add r0, r0, ip
> 7c4da4: e1a0d00b mov sp, fp
> 7c4da8: e8bd8800 pop {fp, pc}
> 7c4dac: 00000008 .word 0x00000008 [0x10 for code
> with JEMALLOC_ALIGNED]
>
> so someone other allocates at least one byte from start of TLS and
> linker is aware about this. But again, .data is first member of TLS
> and tsd_tls is first of .tdata and the 0x5 is value of first member of
> tsd_tls structure (.state = tsd_state_uninitialized).
>
> So who consumes the first bytes in TLS ????
>
> Michal
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?9b85bde5-c450-94c7-a31e-f2eb7f6ab9ee>