Date: Tue, 13 Nov 2018 00:42:35 -0800 From: Mark Millard <marklmi@yahoo.com> To: Kyle Evans <kevans@freebsd.org> Cc: Sean Bruno <sbruno@freebsd.org>, ports-list freebsd <freebsd-ports@freebsd.org> Subject: Re: FYI: ports head -r484783 poudriere-devel with qemu-arm-static: sometimes hangs between a cc (wait) and its child ld (uwait) Message-ID: <F040FBAC-0224-48B3-9867-39AF3F16768D@yahoo.com> In-Reply-To: <3F2AC9A4-7F77-4999-9E9D-DC7D450D921D@yahoo.com> References: <01EDE371-F1BB-4489-BA28-21C2C68BEE9B@yahoo.com> <CACNAnaGtv-%2BhkHWky_ceZu1-nzxb47QL3%2BcWVM2-fdukVAbTMw@mail.gmail.com> <3F2AC9A4-7F77-4999-9E9D-DC7D450D921D@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2018-Nov-12, at 10:18, Mark Millard <marklmi at yahoo.com> wrote:
> On 2018-Nov-12, at 05:54, Kyle Evans <kevans at freebsd.org> wrote:
>=20
>> On Sun, Nov 11, 2018 at 9:11 PM Mark Millard <marklmi at yahoo.com> =
wrote:
>>>=20
>>> [I still can not produce the problem below on demand.
>>> It seems racy with no fixed context producing the
>>> problem as far as which port is building. But the
>>> general structure of what hangs is the same each
>>> time so far.]
>>>=20
>>> The following is just an FYI for the other
>>> qemu-arm-static tied problem that I regularly run into.
>>> I do not have much useful information so far. It is
>>> not clear how I'd get such information.
>>>=20
>>=20
>> Hi,
>>=20
>> Just so we're clear- in what kind of time frame did you start
>> observing this hang?
>=20
> Unfortunately, I did no qemu-user-static use after
> 2018-Feb-6 until 2018-10-26. My list activity reported
> the problem for the first time on Oct. 26 and I had
> updated before using qemu-arm-static on the 26th.
>=20
> Looks like back on Feb. 6 I was using: =
qemu-user-static-2.11.50.g20171215_3
>=20
> Looks like back on Oct. 26 I was using: =
qemu-user-static-2.11.50.g20180622_1
>=20
> I'm now using qemu-user-static-2.11.50.g20181011 .
>=20
>=20
> For reference:
>=20
> The Feb cross build logs for Feb 6 show things like:
>=20
> =3D>> Building ports-mgmt/poudriere-devel
> build started at Tue Feb 6 17:39:36 PST 2018
> port directory: /usr/ports/ports-mgmt/poudriere-devel
> package name: poudriere-devel-3.2.99.20180202_2
> building for: FreeBSD FBSDFSSDjailVariant 12.0-CURRENT FreeBSD =
12.0-CURRENT r327485M arm
> maintained by: bdrewery@FreeBSD.org
> Makefile ident: $FreeBSD: =
head/ports-mgmt/poudriere-devel/Makefile 461075 2018-02-06 16:33:15Z brd =
$
> Poudriere version: 3.2.99.20180202_2
> Host OSVERSION: 1200054
> Jail OSVERSION: 1200054
>=20
> The amd64 (host) logs before that show for qemu-user-static:
>=20
> =3D>> Building emulators/qemu-user-static
> build started at Sun Feb 4 11:22:59 PST 2018
> port directory: /usr/ports/emulators/qemu-user-static
> package name: qemu-user-static-2.11.50.g20171215_3
> building for: FreeBSD FBSDFSSDjailVariant 12.0-CURRENT FreeBSD =
12.0-CURRENT r327485M amd64
> maintained by: sbruno@FreeBSD.org
> Makefile ident: $FreeBSD: =
head/emulators/qemu-user-static/Makefile 441455 2017-05-22 13:17:38Z =
linimon $
> Poudriere version: 3.2.99.20180202_1
> Host OSVERSION: 1200054
> Jail OSVERSION: 1200054
>=20
> (I normally keep the system source code the same across TARGET_ARCH's,
> with some exceptions for powerpc families.)
>=20
> Oct. 26 shows for qemu-user-static:
>=20
> =3D>> Building emulators/qemu-user-static
> build started at Fri Oct 26 13:55:50 PDT 2018
> port directory: /usr/ports/emulators/qemu-user-static
> package name: qemu-user-static-2.11.50.g20180622_1
> building for: FreeBSD FBSDFSSDjailVariant 12.0-ALPHA8 FreeBSD =
12.0-ALPHA8 #1 r339076:339432M: Mon Oct 22 17:48:28 PDT 2018 =
markmi@FBSDFSSD:/usr/obj/amd64_clang_alt/amd64.amd64/usr/src/amd64.amd64/s=
ys/GENERIC-NODBG amd64
> maintained by: sbruno@FreeBSD.org
> Makefile ident: $FreeBSD: =
head/emulators/qemu-user-static/Makefile 441455 2017-05-22 13:17:38Z =
linimon $
> Poudriere version: 3.2.99.20180511
> Host OSVERSION: 1200084
> Jail OSVERSION: 1200063
>=20
> The armv7 jail context would also be based on the same system source,
> mostly -r339076 source.
>=20
>=20
> Currently for qemu-user-static I'm at:
>=20
> =3D>> Building emulators/qemu-user-static
> build started at Sun Nov 11 14:52:52 PST 2018
> port directory: /usr/ports/emulators/qemu-user-static
> package name: qemu-user-static-2.11.50.g20181011
> building for: FreeBSD FBSDFSSDjailVariant 13.0-CURRENT FreeBSD =
13.0-CURRENT amd64
> maintained by: sbruno@FreeBSD.org
> Makefile ident: $FreeBSD: =
head/emulators/qemu-user-static/Makefile 441455 2017-05-22 13:17:38Z =
linimon $
> Poudriere version: 3.2.99.20181024
> Host OSVERSION: 1300003
> Jail OSVERSION: 1300003
I did some buildworld's inside a bulk -i session.
I got a hangup that was not lld: an emulated
ctfmerge hangup. It too had a thread fanout, but
only 7 threads. The attach/detach sequence did
not start things going. The thread backtraces
look like for the lld example.
(gdb) info threads
Id Target Id Frame=20
* 1 LWP 100502 of process 64885 _umtx_op () at _umtx_op.S:3
2 LWP 101208 of process 64885 _umtx_op_err () at =
/usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37
3 LWP 102353 of process 64885 _umtx_op () at _umtx_op.S:3
4 LWP 101899 of process 64885 _umtx_op () at _umtx_op.S:3
5 LWP 101783 of process 64885 _umtx_op () at _umtx_op.S:3
6 LWP 101902 of process 64885 _umtx_op () at _umtx_op.S:3
7 LWP 101907 of process 64885 _umtx_op () at _umtx_op.S:3
(gdb) bt
#0 _umtx_op () at _umtx_op.S:3
#1 0x0000000060050c34 in _umtx_wait_uint_private (addr=3D<optimized =
out>, target_val=3D<optimized out>, tsz=3D<optimized out>, t=3D<optimized =
out>, where=3D<optimized out>)
at =
/wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b=
sd-user/freebsd/os-thread.c:258
#2 freebsd_lock_umutex (target_addr=3D4139717232, id=3D100502, ts=3D0x0, =
mode=3D<optimized out>) at =
/wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b=
sd-user/freebsd/os-thread.c:890
#3 0x000000006004a768 in do_freebsd__umtx_op (obj=3D4139717232, =
op=3D<optimized out>, val=3D0, uaddr=3D0, target_time=3D0)
at =
/wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b=
sd-user/freebsd/os-thread.h:359
#4 0x0000000060041405 in do_freebsd_syscall (cpu_env=3D0x860596318, =
num=3D454, arg1=3D<optimized out>, arg2=3D<optimized out>, =
arg3=3D<optimized out>, arg4=3D0, arg5=3D0, arg6=3D165556, arg7=3D100502, =
arg8=3D0)
at =
/wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b=
sd-user/syscall.c:1364
#5 0x0000000060038bb3 in target_cpu_loop (env=3D0x860596318) at =
/wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b=
sd-user/arm/target_arch_cpu.h:207
#6 0x0000000060038559 in cpu_loop (env=3D0xf6bf128c) at =
/wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b=
sd-user/main.c:121
#7 0x00000000600397d2 in main (argc=3D-18969, argv=3D0x7fffffffaea0) at =
/wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b=
sd-user/main.c:513
(gdb) thread 2
[Switching to thread 2 (LWP 101208 of process 64885)]
#0 _umtx_op_err () at =
/usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37
37 RSYSCALL_ERR(_umtx_op)
(gdb) bt
#0 _umtx_op_err () at =
/usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37
#1 0x00000000601a3cc1 in _thr_umtx_timedwait_uint (mtx=3D0x8604de008, =
id=3D<optimized out>, clockid=3D<optimized out>, abstime=3D<optimized =
out>, shared=3D<optimized out>)
at /usr/src/lib/libthr/thread/thr_umtx.c:236
#2 0x00000000601ad548 in cond_wait_user (abstime=3D0x0, cancel=3D1, =
cvp=3D<optimized out>, mp=3D<optimized out>) at =
/usr/src/lib/libthr/thread/thr_cond.c:307
#3 cond_wait_common (cond=3D<optimized out>, mutex=3D<optimized out>, =
abstime=3D0x0, cancel=3D1) at /usr/src/lib/libthr/thread/thr_cond.c:367
#4 0x00000000601143cc in qemu_futex_wait (val=3D4294967295, =
ev=3D<optimized out>) at util/qemu-thread-posix.c:350
#5 qemu_event_wait (ev=3D0x626eeec0 <rcu_call_ready_event>) at =
util/qemu-thread-posix.c:445
#6 0x000000006011b3ba in call_rcu_thread (opaque=3D<optimized out>) at =
util/rcu.c:255
#7 0x00000000601ad1c6 in thread_start (curthread=3D0x8604d2500) at =
/usr/src/lib/libthr/thread/thr_create.c:292
#8 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffdfffd000
(gdb) thread 3
[Switching to thread 3 (LWP 102353 of process 64885)]
#0 _umtx_op () at _umtx_op.S:3
3 RSYSCALL(_umtx_op)
(gdb) bt
#0 _umtx_op () at _umtx_op.S:3
#1 0x0000000060050c34 in _umtx_wait_uint_private (addr=3D<optimized =
out>, target_val=3D<optimized out>, tsz=3D<optimized out>, t=3D<optimized =
out>, where=3D<optimized out>)
at =
/wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b=
sd-user/freebsd/os-thread.c:258
#2 freebsd_lock_umutex (target_addr=3D4139717232, id=3D102353, ts=3D0x0, =
mode=3D<optimized out>) at =
/wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b=
sd-user/freebsd/os-thread.c:890
#3 0x000000006004a768 in do_freebsd__umtx_op (obj=3D4139717232, =
op=3D<optimized out>, val=3D0, uaddr=3D0, target_time=3D0)
at =
/wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b=
sd-user/freebsd/os-thread.h:359
#4 0x0000000060041405 in do_freebsd_syscall (cpu_env=3D0x860ca3198, =
num=3D454, arg1=3D<optimized out>, arg2=3D<optimized out>, =
arg3=3D<optimized out>, arg4=3D0, arg5=3D0, arg6=3D-167772160, =
arg7=3D102353, arg8=3D0)
at =
/wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b=
sd-user/syscall.c:1364
#5 0x0000000060038bb3 in target_cpu_loop (env=3D0x860ca3198) at =
/wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b=
sd-user/arm/target_arch_cpu.h:207
#6 0x0000000060038559 in cpu_loop (env=3D0xf6bf128c) at =
/wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b=
sd-user/main.c:121
#7 0x00000000600503ea in new_freebsd_thread_start (arg=3D<optimized =
out>) at =
/wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b=
sd-user/freebsd/os-thread.c:152
#8 0x00000000601ad1c6 in thread_start (curthread=3D0x8604d5200) at =
/usr/src/lib/libthr/thread/thr_create.c:292
#9 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffdfdfc000
PID USERNAME THR PRI NICE SIZE RES SWAP STATE C TIME =
WCPU COMMAND
64885 root 7 20 0 79M 12M 0 uwait 2 0:00 =
0.00% /usr/local/bin/qemu-arm-static ctfmerge -L VERSION -g -o bhnd.kld =
bhnd.o bhnd_subr.o bhnd_bus_if.o bhnd_erom.o bhnd_erom_if
64846 root 1 52 0 10M 1768K 0 wait 22 0:00 =
0.00% sh -e
46319 root 1 20 0 11M 2420K 0 select 8 0:00 =
0.00% /nxb-bin/usr/bin/make all DIRPRFX=3Dbhnd/
Again: not repeatable for when/where --but has a thread
fanout involved.
Note: I had set things up to use the nxb-bin/... instead of
emulated compiling/linking/etc.
Another point: This was a native FreeBSD boot, not
under Hyper-V. But it is the same system and same
storage media as when used under Hyper-V.
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?F040FBAC-0224-48B3-9867-39AF3F16768D>