Date: Mon, 22 Jul 2024 10:27:20 -0700 From: Mark Millard <marklmi@yahoo.com> To: mmel@freebsd.org Cc: FreeBSD Current <freebsd-current@freebsd.org>, "freebsd-arm@freebsd.org" <freebsd-arm@freebsd.org>, "kib@freebsd.org >> Konstantin Belousov" <kib@freebsd.org> Subject: Re: armv7-on-aarch64 stuck at urdlck Message-ID: <C0B42CBB-8F12-4597-A04B-26F2107E176E@yahoo.com> In-Reply-To: <a1b978fe-ff54-4112-860c-b09500d89d0b@freebsd.org> References: <724db42b-5550-4381-8277-2971e6b3e8f1@freebsd.org> <B5E2275D-21F0-43C8-AF06-A45DB7448D66@yahoo.com> <86185657-e521-466b-89e2-f291aaac10a6@freebsd.org> <0EF18174-8735-46A4-BD71-FFA3472B319F@yahoo.com> <a1b978fe-ff54-4112-860c-b09500d89d0b@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Jul 22, 2024, at 09:41, meloun.michal@gmail.com wrote: > On 22.07.2024 18:26, Mark Millard wrote: >> On Jul 22, 2024, at 06:40, Michal Meloun <meloun.michal@gmail.com> = wrote: >>> On 22.07.2024 13:46, Mark Millard wrote: >>>> On Jul 21, 2024, at 22:59, Michal Meloun <meloun.michal@gmail.com> = wrote: >>>>> I don't want to hijack the original thread, so I'm replying in a = new one. >>>>>=20 >>>>> My tegra track current, has been running 24/7 by building = kernel/world and kde5 in a loop for a few years now. But I have never = encountered the aforementioned lockup in native armv7. >>>>>=20 >>>>> I have seen usermode mutex lockup in arm32 jail on aarch64, but = only very rarely (once a month or so) and all my attempts to reproduce = it in a more deterministic way have failed. Also, I don't think I've = ever seen this with the debug version of libc. >>>>>=20 >>>>> Unfortunately I also failed to reproduce given lockup using = dlopen_test.c, neither on native armv7 or arm32 jail. >>>>>=20 >>>>> Michal Meloun >>>> What is the output of: >>>> # readelf -a /libexec/ld-elf.so.1 | grep -E "(^[^ = 0-9]|.*_rtld_get_stack_prot)" >>>> in your armv7 context(s)? Does it include for likes of: >>>> QUOTE >>>> Symbol table '.symtab' contains 911 entries: >>>> 903: 000000000001b9ac 16 FUNC GLOBAL DEFAULT 11 = _rtld_get_stack_prot >>>> END QUOTE >>>> ` >>>> vs. not? >>>> Note that the "debug version of libc" being involved likely means = that >>>> DEBUG_FLAGS was defined. That in turn likely means that strip is = not >>>> being used. In such a case, I expect that the .symtab entry for >>>> _rtld_get_stack_prot (and more) exists for such a context. >>> At tis time, I have standard (thus stripped, non-debug) version of = runtime linker library installed. Thus it have only dynamic relocation = record for _rtld_get_stack_prot: >>>=20 >>> root@tegra124:~/dlopen_test # readelf -a /libexec/ld-elf.so.1 | grep = -E "(^[^ 0-9]|.*_rtld_get_stack_prot)" >>> ELF Header: >>> Elf file type is DYN (Shared object file) >>> Entry point 0x1449c >>> There are 10 program headers, starting at offset 52 >>> Program Headers: >>> There are 23 section headers, starting at offset 0x1a448: >>> Section Headers: >>> Key to Flags: >>> Dynamic section at offset 0x19fa4 contains 15 entries: >>> Relocation section (.rel.dyn): >>> r_offset r_info r_type st_value st_name >>> Symbol table '.dynsym' contains 27 entries: >>> 5: 000000000001ba0c 16 FUNC GLOBAL DEFAULT 12 = _rtld_get_stack_prot@@FBSDprivate_1.0 (11) >>> Notes at offset 0x00000174 with length 0x00000018: >>> Histogram for bucket list length (total of 6 buckets): >>> Histogram for bucket list length (total of 27 buckets): >>> Version symbol section (.gnu.version): >>> Version definition section (.gnu.version_d): >>> Attribute Section: aeabi >>>=20 >>> ------ >>>=20 >>> root@tegra124:~/dlopen_test # ./dlopen_test >>> root@tegra124:~/dlopen_test # >> Just to be sure . . . >> Did you at some point "pkg install cairo" (or analogous) so that >> the following (or some vintage) were in place? >> # ls -lodT /usr/local/lib/libcairo.so* >> lrwxr-xr-x 1 root wheel - 21 Apr 29 19:45:15 2024 = /usr/local/lib/libcairo.so -> libcairo.so.2.11704.0 >> lrwxr-xr-x 1 root wheel - 21 Apr 29 19:45:15 2024 = /usr/local/lib/libcairo.so.2 -> libcairo.so.2.11704.0 >> -rwxr-xr-x 1 root wheel - 1118272 Apr 29 19:45:15 2024 = /usr/local/lib/libcairo.so.2.11704.0 >> # file /usr/local/lib/libcairo.so.2.11704.0 >> /usr/local/lib/libcairo.so.2.11704.0: ELF 32-bit LSB shared object, = ARM, EABI5 version 1 (FreeBSD), dynamically linked, for FreeBSD 15.0 = (1500018), stripped >> (Installing cairo would also install other things it needs.) >> For the failing contexts, the a.out from dlopen_test.c will only >> hang if the library (and what it requires) is actually there to >> load. > Yep, i have cairo installed (but compiled from sources, not installed = by pkg). And i have verified that dlopen() return success. > In the meantime I tried all combinations (debud/stripped) of ld_elf = and libthr. All combinations work without problems on the native system = and in arm323 jail. Thanks for the information. My personal builds, which are the ones that work in my testing, are built on aarch64 as armv7 instead of on amd64. The known failing ones are built on amd64. But I've no more specific information suggesting a tie to the type of build host for the world used. > Btw, gdb has long had problems with stepping inside ld_elf. It's = better to run the test program without it and connect to the test = program to get the "correct" stack trace. >=20 In part I was deliberately exploring what sequence leads to the hangups vs. lack of hangups and the like: more context than a backtrace of the stuck state can provide. But doing "./a.out &" and then "gdb -p..." to attach to it: _umtx_op () at _umtx_op.S:4 warning: 4 _umtx_op.S: No such file or directory (gdb) bt #0 _umtx_op () at _umtx_op.S:4 #1 0x2036845c in _umtx_op_err (obj=3D0x4, op=3D12, val=3D0, uaddr=3D0x0, = uaddr2=3D0x0) at = /home/pkgbuild/worktrees/main/lib/libsys/_umtx_op_err.c:36 #2 0x20115da8 in __thr_rwlock_rdlock (rwlock=3D0x4, = rwlock@entry=3D0x20137c40, flags=3D3, tsp=3D<optimized out>) at = /home/pkgbuild/worktrees/main/lib/libthr/thread/thr_umtx.c:294 #3 0x2010ebf4 in _thr_rwlock_rdlock (rwlock=3D0x20137c40, flags=3D0, = tsp=3D0x0) at = /home/pkgbuild/worktrees/main/lib/libthr/thread/thr_umtx.h:229 #4 _thr_rtld_rlock_acquire (lock=3D0x20137c40) at = /home/pkgbuild/worktrees/main/lib/libthr/thread/thr_rtld.c:121 #5 0x20060788 in rlock_acquire (lock=3D0x2008af10 <rtld_locks>, = lockstate=3Dlockstate@entry=3D0xffffd114) at = /home/pkgbuild/worktrees/main/libexec/rtld-elf/rtld_lock.c:259 #6 0x20059098 in _rtld_bind (obj=3D0x2008f404, reloff=3D496) at = /home/pkgbuild/worktrees/main/libexec/rtld-elf/rtld.c:1035 #7 0x2005483c in _rtld_bind_start () at = /home/pkgbuild/worktrees/main/libexec/rtld-elf/arm/rtld_start.S:89 #8 0x2005483c in _rtld_bind_start () at = /home/pkgbuild/worktrees/main/libexec/rtld-elf/arm/rtld_start.S:89 #9 0x2005483c in _rtld_bind_start () at = /home/pkgbuild/worktrees/main/libexec/rtld-elf/arm/rtld_start.S:89 . . . It does not seem significantly different than I'd reported for the hungup state. An issue here is that the pkgbase world possibly is -O2 based despite having debug information (but is stripped). This can make details less reliable. So, for example, the rwlock=3D0x4 vs. rwlock@entry=3D0x20137c40 for __thr_rwlock_rdlock could well be suspect. =3D=3D=3D Mark Millard marklmi at yahoo.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?C0B42CBB-8F12-4597-A04B-26F2107E176E>