From owner-freebsd-bugs@freebsd.org Fri Apr 9 19:33:30 2021 Return-Path: Delivered-To: freebsd-bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E47485CCEE7 for ; Fri, 9 Apr 2021 19:33:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4FH7bG5c92z3vlW for ; Fri, 9 Apr 2021 19:33:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id C031E5CCF46; Fri, 9 Apr 2021 19:33:30 +0000 (UTC) Delivered-To: bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BFDEC5CCCF9 for ; Fri, 9 Apr 2021 19:33:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FH7bG40Ffz3w1c for ; Fri, 9 Apr 2021 19:33:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 7363723C88 for ; Fri, 9 Apr 2021 19:33:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 139JXUT2036931 for ; Fri, 9 Apr 2021 19:33:30 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 139JXU49036930 for bugs@FreeBSD.org; Fri, 9 Apr 2021 19:33:30 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 254927] rtld: dangling locks after fix for bug 254774 Date: Fri, 09 Apr 2021 19:33:30 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: vangyzen@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Apr 2021 19:33:31 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D254927 Bug ID: 254927 Summary: rtld: dangling locks after fix for bug 254774 Product: Base System Version: CURRENT Hardware: Any OS: Any Status: New Severity: Affects Only Me Priority: --- Component: bin Assignee: bugs@FreeBSD.org Reporter: vangyzen@FreeBSD.org After the fix for bug 254774, some processes hang due to apparent dangling locks in rtld. Reverting commit d36d68161517 fixes the hang. I first saw the hang with gdk-pixbuf-query-loaders. (This is called from s= ome package post-install scripts, so I first noticed it when running tools/build/beinstall.sh.) Simply run it without arguments. I also see it with thunderbird. Here are some notes from debugging: 98991 0 I+ 0:00.00 `-- /bin/sh ./tools/build/beinstall.sh 58781 0 I+ 0:00.00 `-- pkg upgrade 58782 0 S+ 0:15.73 `-- pkg upgrade 59415 0 I+ 0:00.00 `-- /bin/sh -c set -- libavif-0.9.0_1\nif ! /usr/sbin/service ldconfig restart >/dev/null; then\n\011\011if [ -z "${INSTALL_AS_USER}" ]; then\n\011\011\011exit 1\n\011\011fi\n\011fi\n/usr/local/bin/gdk-pixbuf-query-loaders > /dev/null = 2>&1 && /usr/local/bin/gdk-pixbuf-query-loaders > /usr/local/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache 2>/dev/null || /usr/bin/= true 59428 0 I+ 0:00.02 `-- /usr/local/bin/gdk-pixbuf-query-loaders (gdk-pixbuf-query-lo) [9/41] Extracting librsvg2-rust-2.50.3_2: 100% (hang) [28/41] Extracting libavif-0.9.0_1: 100% (hang) # gdb /usr/local/bin/gdk-pixbuf-query-loaders 58871 GNU gdb (GDB) 10.1 [GDB v10.1 for FreeBSD] [...] (gdb) info thr Id Target Id Frame * 1 LWP 114003 of process 58871 _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:40 2 LWP 178846 of process 58871 "pool-spawner" _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:40 3 LWP 178847 of process 58871 "gmain" 0x000000080071680a in ?? = () from /lib/libc.so.7 (gdb) bt #0 _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:= 40 #1 0x00000008005c6830 in __thr_rwlock_wrlock (rwlock=3D0x8005cac80, tsp=3D) at /usr/src/lib/libthr/thread/thr_umtx.c:327 #2 0x00000008005c0f11 in _thr_rwlock_wrlock (rwlock=3D, tsp=3D) at /usr/src/lib/libthr/thread/thr_umtx.h:241 #3 _thr_rtld_wlock_acquire (lock=3D0x8005cac80) at /usr/src/lib/libthr/thread/thr_rtld.c:143 #4 0x0000000800216b3b in wlock_acquire (lock=3D0x800224640 , lockstate=3Dlockstate@entry=3D0x7ffff= fffca90) at /usr/src/libexec/rtld-elf/rtld_lock.c:248 #5 0x0000000800211e71 in tls_get_addr_slow (dtvp=3D, index=3D3, offset=3D0) at /usr/src/libexec/rtld-elf/rtld.c:4878 #6 0x0000000800210df9 in rtld_fill_dl_phdr_info (obj=3D0x0, phdr_info=3D0x7fffffffcbf0) at /usr/src/libexec/rtld-elf/rtld.c:3896 #7 _rtld_addr_phdr (addr=3D0x802e1a200, phdr_info=3D0x7fffffffcbf0) at /usr/src/libexec/rtld-elf/rtld.c:3781 #8 0x000000080073f565 in __cxa_finalize (dso=3D0x8005cac80) at /usr/src/lib/libc/stdlib/atexit.c:209 #9 0x0000000802d6ae8c in () at /usr/local/lib/libpixman-1.so.0 #10 0x0000000800c17808 in () #11 0x0000000800c14008 in () #12 0x00007fffffffd110 in () #13 0x0000000802e10945 in _fini () at /usr/local/lib/libpixman-1.so.0 #14 0x00007fffffffd110 in () #15 0x00000008002162a4 in objlist_call_fini (list=3D, root=3D, lockstate=3D0x800c1600= 8) at /usr/src/libexec/rtld-elf/rtld.c:2766 (gdb) thr 2 [Switching to thread 2 (LWP 178846 of process 58871)] #0 _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:= 40 40 RSYSCALL_ERR(_umtx_op) (gdb) bt #0 _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:= 40 #1 0x00000008005c6660 in _thr_umtx_timedwait_uint (mtx=3D0x80023d008, id=3Did@entry=3D0, clockid=3D, abstime=3D, shared=3D, shared@entry=3D0) at /usr/src/lib/libthr/thread/thr_umtx.c:236 #2 0x00000008005bd119 in _thr_sleep (curthread=3Dcurthread@entry=3D0x801012500, clockid=3D0, abstime=3Dabstime@entry=3D0x0) at /usr/src/lib/libthr/thread/thr_kern.c:199 #3 0x00000008005b869f in cond_wait_user (cvp=3D0x801018e80, mp=3D0x800c22c88, abstime=3D0x0, cancel=3D1) at /usr/src/lib/libthr/thread/thr_cond.c:320 #4 cond_wait_common (cond=3D, mutex=3D, abstime=3D0x0, cancel= =3D1) at /usr/src/lib/libthr/thread/thr_cond.c:380 #5 0x0000000800377185 in g_cond_wait () at /usr/local/lib/libglib-2.0.so.0 #6 0x00000008002eec35 in () at /usr/local/lib/libglib-2.0.so.0 #7 0x000000080034e1f8 in () at /usr/local/lib/libglib-2.0.so.0 #8 0x000000080034dbf2 in () at /usr/local/lib/libglib-2.0.so.0 #9 0x00000008005b982b in thread_start (curthread=3D0x801012500) at /usr/src/lib/libthr/thread/thr_create.c:292 #10 0x0000000000000000 in () (gdb) thr 3 [Switching to thread 3 (LWP 178847 of process 58871)] #0 0x000000080071680a in ?? () from /lib/libc.so.7 (gdb) btr Undefined command: "btr". Try "help". (gdb) bt #0 0x000000080071680a in () at /lib/libc.so.7 #1 0x00000008005c4f56 in __thr_poll (fds=3D0x801008210, nfds=3D1, timeout= =3D-1) at /usr/src/lib/libthr/thread/thr_syscalls.c:338 #2 0x0000000800323471 in () at /usr/local/lib/libglib-2.0.so.0 #3 0x0000000800323596 in g_main_context_iteration () at /usr/local/lib/libglib-2.0.so.0 #4 0x00000008003255d1 in () at /usr/local/lib/libglib-2.0.so.0 #5 0x000000080034dbf2 in () at /usr/local/lib/libglib-2.0.so.0 #6 0x00000008005b982b in thread_start (curthread=3D0x801012a00) at /usr/src/lib/libthr/thread/thr_create.c:292 #7 0x0000000000000000 in () (gdb) thr 1 [Switching to thread 1 (LWP 114003 of process 58871)] #0 _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:= 40 40 RSYSCALL_ERR(_umtx_op) (gdb) f 5 #5 0x0000000800211e71 in tls_get_addr_slow (dtvp=3D, index= =3D3, offset=3D0) at /usr/src/libexec/rtld-elf/rtld.c:4878 (gdb) p rtld_bind_lock $1 =3D (rtld_lock_t) 0x800224640 (gdb) p *rtld_bind_lock $2 =3D {handle =3D 0x8005cac80, mask =3D 1} (gdb) p *(struct rtld_lock*)rtld_bind_lock.handle $4 =3D {lock =3D {rw_state =3D 1, rw_flags =3D 2, rw_blocked_readers =3D 0, rw_blocked_writers =3D 0, rw_spare =3D {0, 0, 0, 0}}, _pad =3D '\000' } (gdb) p dtv[0] $5 =3D 3 (gdb) p tls_dtv_generation $6 =3D 3 (gdb) p index $7 =3D 3 (gdb) p dtv[index + 1] $8 =3D 0 (gdb) f 3 #3 _thr_rtld_wlock_acquire (lock=3D0x8005cac80) at /usr/src/lib/libthr/thread/thr_rtld.c:143 143 while (_thr_rwlock_wrlock(&l->lock, NULL) !=3D 0) (gdb) set $l =3D (struct rtld_lock *)lock (gdb) p $l->lock $11 =3D {rw_state =3D 1, rw_flags =3D 2, rw_blocked_readers =3D 0, rw_blocked_writers =3D 0, rw_spare =3D {0, 0, 0, 0}} --=20 You are receiving this mail because: You are the assignee for the bug.=