From owner-freebsd-ports@freebsd.org Tue Nov 13 08:42:44 2018 Return-Path: Delivered-To: freebsd-ports@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A96F1110906B for ; Tue, 13 Nov 2018 08:42:44 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic313-21.consmr.mail.gq1.yahoo.com (sonic313-21.consmr.mail.gq1.yahoo.com [98.137.65.84]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C7E0A79335 for ; Tue, 13 Nov 2018 08:42:43 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: 2hTcAFkVM1k5vWhJo2KbyfOZwu3vBNXDjlmqrI60oYw04D6Q3f41IbjKzlLF8rg JU.YeIXVBEgd5Z3npXyMVQoEEW_JhNSxZAaidhQsGAeLp6h8aMhhOsGDOmgJb7qQmU4BzGY78Xdw a9Qx4ltP2rMnsh4Aa.Pg0k6Q7RFyC1u.89ShDxZU_NtQzuSbLvQi54nA6Fp91fzFnTFhZQrJbueE ct.ndkGtdGDc92PJRk86OkTi8kaXrxWGdh3LBG4T9YuWquO5j8FBd1_BFnFIeqLsEb8eGmaLnf3R assVAfegF51oDV7Y0WNeKs.lIQDG.2pi1z9N9aw5p5zj2.jVVZJVLlHvg0rrd07szqcwsIyxfp.h UQ.xfkF46dgRh_0Ri76lrSvD.pY9aflDaupFDt_ePuYDf_AnrQJM.jICzZBOooS4UccFeeVUyJjA eSaO5JJ5OJ0wRB7bCqLl61rzLjS1TaR.rkQhFnxpXtCAf_yd4X_zRgUrlaiHujHGuDKntSGdrNpp dPyCenxWNcw2vbJOwN1Dks0KpRumlApQsXLo8BZ1CfiEhcYZOgh429fyDLhhWiuxTax3GWkhGCqI dFUUUtzz03XDydYioq_K2bjfA36BTC2T4l5dJcXfB8Qf.xlVbx2zyq4Bt4DW9Thi_0q2jWZM.dd4 AWFTrhuou8qdf6SL_Kp_85xKbBYGJfoDr2x03I28r8DlxmHBDWA3C2tE2Jgl1JsfrxFPX6E7JKMA qBeaMLvg8058jVgig2QAEFKY7s8B5NmcRHNfAElDSJeIFtFGTGKcQlJjX6agb3ZSQLC2GGjW4kDr 9Ew47Z0yx4Nyk_hZ9yL7ijH3_4lSoLBTAeTVVV9PfEuh7rhz2W9w3BvZ5pYPgyH_RK2thlgwUKo6 dPveJ5Q72h5jz6JaDWeWYKkTWzthXCQh._yjXRCZvlEky6Ucr9R2dvhbpoonrya72V8CjFCZBxfZ gRK0DI4kcmX1IdtLZl4wmkn2Ne_aKN51FG0v7wX6h95CCtK_qc7xEwjxgDMHxtfH1huz93FVl.kj l6j99AlVpqaudnql7mFOoQcXT1Y2potnEIOsPf_n4p3EopZo. Received: from sonic.gate.mail.ne1.yahoo.com by sonic313.consmr.mail.gq1.yahoo.com with HTTP; Tue, 13 Nov 2018 08:42:37 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.25]) ([76.115.7.162]) by smtp417.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID eeb4c14f16674c5d1d83f40926ab3511; Tue, 13 Nov 2018 08:42:36 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.1 \(3445.101.1\)) Subject: Re: FYI: ports head -r484783 poudriere-devel with qemu-arm-static: sometimes hangs between a cc (wait) and its child ld (uwait) From: Mark Millard In-Reply-To: <3F2AC9A4-7F77-4999-9E9D-DC7D450D921D@yahoo.com> Date: Tue, 13 Nov 2018 00:42:35 -0800 Cc: Sean Bruno , ports-list freebsd Content-Transfer-Encoding: quoted-printable Message-Id: References: <01EDE371-F1BB-4489-BA28-21C2C68BEE9B@yahoo.com> <3F2AC9A4-7F77-4999-9E9D-DC7D450D921D@yahoo.com> To: Kyle Evans X-Mailer: Apple Mail (2.3445.101.1) X-Rspamd-Queue-Id: C7E0A79335 X-Spamd-Result: default: False [-2.18 / 200.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-0.42)[-0.420,0]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(0.23)[ipnet: 98.137.64.0/21(0.70), asn: 36647(0.56), country: US(-0.09)]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; SUBJECT_ENDS_SPACES(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com]; NEURAL_HAM_MEDIUM(-0.99)[-0.990,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-0.99)[-0.990,0]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[84.65.137.98.list.dnswl.org : 127.0.5.0] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Nov 2018 08:42:44 -0000 On 2018-Nov-12, at 10:18, Mark Millard wrote: > On 2018-Nov-12, at 05:54, Kyle Evans wrote: >=20 >> On Sun, Nov 11, 2018 at 9:11 PM Mark Millard = wrote: >>>=20 >>> [I still can not produce the problem below on demand. >>> It seems racy with no fixed context producing the >>> problem as far as which port is building. But the >>> general structure of what hangs is the same each >>> time so far.] >>>=20 >>> The following is just an FYI for the other >>> qemu-arm-static tied problem that I regularly run into. >>> I do not have much useful information so far. It is >>> not clear how I'd get such information. >>>=20 >>=20 >> Hi, >>=20 >> Just so we're clear- in what kind of time frame did you start >> observing this hang? >=20 > Unfortunately, I did no qemu-user-static use after > 2018-Feb-6 until 2018-10-26. My list activity reported > the problem for the first time on Oct. 26 and I had > updated before using qemu-arm-static on the 26th. >=20 > Looks like back on Feb. 6 I was using: = qemu-user-static-2.11.50.g20171215_3 >=20 > Looks like back on Oct. 26 I was using: = qemu-user-static-2.11.50.g20180622_1 >=20 > I'm now using qemu-user-static-2.11.50.g20181011 . >=20 >=20 > For reference: >=20 > The Feb cross build logs for Feb 6 show things like: >=20 > =3D>> Building ports-mgmt/poudriere-devel > build started at Tue Feb 6 17:39:36 PST 2018 > port directory: /usr/ports/ports-mgmt/poudriere-devel > package name: poudriere-devel-3.2.99.20180202_2 > building for: FreeBSD FBSDFSSDjailVariant 12.0-CURRENT FreeBSD = 12.0-CURRENT r327485M arm > maintained by: bdrewery@FreeBSD.org > Makefile ident: $FreeBSD: = head/ports-mgmt/poudriere-devel/Makefile 461075 2018-02-06 16:33:15Z brd = $ > Poudriere version: 3.2.99.20180202_2 > Host OSVERSION: 1200054 > Jail OSVERSION: 1200054 >=20 > The amd64 (host) logs before that show for qemu-user-static: >=20 > =3D>> Building emulators/qemu-user-static > build started at Sun Feb 4 11:22:59 PST 2018 > port directory: /usr/ports/emulators/qemu-user-static > package name: qemu-user-static-2.11.50.g20171215_3 > building for: FreeBSD FBSDFSSDjailVariant 12.0-CURRENT FreeBSD = 12.0-CURRENT r327485M amd64 > maintained by: sbruno@FreeBSD.org > Makefile ident: $FreeBSD: = head/emulators/qemu-user-static/Makefile 441455 2017-05-22 13:17:38Z = linimon $ > Poudriere version: 3.2.99.20180202_1 > Host OSVERSION: 1200054 > Jail OSVERSION: 1200054 >=20 > (I normally keep the system source code the same across TARGET_ARCH's, > with some exceptions for powerpc families.) >=20 > Oct. 26 shows for qemu-user-static: >=20 > =3D>> Building emulators/qemu-user-static > build started at Fri Oct 26 13:55:50 PDT 2018 > port directory: /usr/ports/emulators/qemu-user-static > package name: qemu-user-static-2.11.50.g20180622_1 > building for: FreeBSD FBSDFSSDjailVariant 12.0-ALPHA8 FreeBSD = 12.0-ALPHA8 #1 r339076:339432M: Mon Oct 22 17:48:28 PDT 2018 = markmi@FBSDFSSD:/usr/obj/amd64_clang_alt/amd64.amd64/usr/src/amd64.amd64/s= ys/GENERIC-NODBG amd64 > maintained by: sbruno@FreeBSD.org > Makefile ident: $FreeBSD: = head/emulators/qemu-user-static/Makefile 441455 2017-05-22 13:17:38Z = linimon $ > Poudriere version: 3.2.99.20180511 > Host OSVERSION: 1200084 > Jail OSVERSION: 1200063 >=20 > The armv7 jail context would also be based on the same system source, > mostly -r339076 source. >=20 >=20 > Currently for qemu-user-static I'm at: >=20 > =3D>> Building emulators/qemu-user-static > build started at Sun Nov 11 14:52:52 PST 2018 > port directory: /usr/ports/emulators/qemu-user-static > package name: qemu-user-static-2.11.50.g20181011 > building for: FreeBSD FBSDFSSDjailVariant 13.0-CURRENT FreeBSD = 13.0-CURRENT amd64 > maintained by: sbruno@FreeBSD.org > Makefile ident: $FreeBSD: = head/emulators/qemu-user-static/Makefile 441455 2017-05-22 13:17:38Z = linimon $ > Poudriere version: 3.2.99.20181024 > Host OSVERSION: 1300003 > Jail OSVERSION: 1300003 I did some buildworld's inside a bulk -i session. I got a hangup that was not lld: an emulated ctfmerge hangup. It too had a thread fanout, but only 7 threads. The attach/detach sequence did not start things going. The thread backtraces look like for the lld example. (gdb) info threads Id Target Id Frame=20 * 1 LWP 100502 of process 64885 _umtx_op () at _umtx_op.S:3 2 LWP 101208 of process 64885 _umtx_op_err () at = /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 3 LWP 102353 of process 64885 _umtx_op () at _umtx_op.S:3 4 LWP 101899 of process 64885 _umtx_op () at _umtx_op.S:3 5 LWP 101783 of process 64885 _umtx_op () at _umtx_op.S:3 6 LWP 101902 of process 64885 _umtx_op () at _umtx_op.S:3 7 LWP 101907 of process 64885 _umtx_op () at _umtx_op.S:3 (gdb) bt #0 _umtx_op () at _umtx_op.S:3 #1 0x0000000060050c34 in _umtx_wait_uint_private (addr=3D, target_val=3D, tsz=3D, t=3D, where=3D) at = /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b= sd-user/freebsd/os-thread.c:258 #2 freebsd_lock_umutex (target_addr=3D4139717232, id=3D100502, ts=3D0x0, = mode=3D) at = /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b= sd-user/freebsd/os-thread.c:890 #3 0x000000006004a768 in do_freebsd__umtx_op (obj=3D4139717232, = op=3D, val=3D0, uaddr=3D0, target_time=3D0) at = /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b= sd-user/freebsd/os-thread.h:359 #4 0x0000000060041405 in do_freebsd_syscall (cpu_env=3D0x860596318, = num=3D454, arg1=3D, arg2=3D, = arg3=3D, arg4=3D0, arg5=3D0, arg6=3D165556, arg7=3D100502, = arg8=3D0) at = /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b= sd-user/syscall.c:1364 #5 0x0000000060038bb3 in target_cpu_loop (env=3D0x860596318) at = /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b= sd-user/arm/target_arch_cpu.h:207 #6 0x0000000060038559 in cpu_loop (env=3D0xf6bf128c) at = /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b= sd-user/main.c:121 #7 0x00000000600397d2 in main (argc=3D-18969, argv=3D0x7fffffffaea0) at = /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b= sd-user/main.c:513 (gdb) thread 2 [Switching to thread 2 (LWP 101208 of process 64885)] #0 _umtx_op_err () at = /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 37 RSYSCALL_ERR(_umtx_op) (gdb) bt #0 _umtx_op_err () at = /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 #1 0x00000000601a3cc1 in _thr_umtx_timedwait_uint (mtx=3D0x8604de008, = id=3D, clockid=3D, abstime=3D, shared=3D) at /usr/src/lib/libthr/thread/thr_umtx.c:236 #2 0x00000000601ad548 in cond_wait_user (abstime=3D0x0, cancel=3D1, = cvp=3D, mp=3D) at = /usr/src/lib/libthr/thread/thr_cond.c:307 #3 cond_wait_common (cond=3D, mutex=3D, = abstime=3D0x0, cancel=3D1) at /usr/src/lib/libthr/thread/thr_cond.c:367 #4 0x00000000601143cc in qemu_futex_wait (val=3D4294967295, = ev=3D) at util/qemu-thread-posix.c:350 #5 qemu_event_wait (ev=3D0x626eeec0 ) at = util/qemu-thread-posix.c:445 #6 0x000000006011b3ba in call_rcu_thread (opaque=3D) at = util/rcu.c:255 #7 0x00000000601ad1c6 in thread_start (curthread=3D0x8604d2500) at = /usr/src/lib/libthr/thread/thr_create.c:292 #8 0x0000000000000000 in ?? () Backtrace stopped: Cannot access memory at address 0x7fffdfffd000 (gdb) thread 3 [Switching to thread 3 (LWP 102353 of process 64885)] #0 _umtx_op () at _umtx_op.S:3 3 RSYSCALL(_umtx_op) (gdb) bt #0 _umtx_op () at _umtx_op.S:3 #1 0x0000000060050c34 in _umtx_wait_uint_private (addr=3D, target_val=3D, tsz=3D, t=3D, where=3D) at = /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b= sd-user/freebsd/os-thread.c:258 #2 freebsd_lock_umutex (target_addr=3D4139717232, id=3D102353, ts=3D0x0, = mode=3D) at = /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b= sd-user/freebsd/os-thread.c:890 #3 0x000000006004a768 in do_freebsd__umtx_op (obj=3D4139717232, = op=3D, val=3D0, uaddr=3D0, target_time=3D0) at = /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b= sd-user/freebsd/os-thread.h:359 #4 0x0000000060041405 in do_freebsd_syscall (cpu_env=3D0x860ca3198, = num=3D454, arg1=3D, arg2=3D, = arg3=3D, arg4=3D0, arg5=3D0, arg6=3D-167772160, = arg7=3D102353, arg8=3D0) at = /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b= sd-user/syscall.c:1364 #5 0x0000000060038bb3 in target_cpu_loop (env=3D0x860ca3198) at = /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b= sd-user/arm/target_arch_cpu.h:207 #6 0x0000000060038559 in cpu_loop (env=3D0xf6bf128c) at = /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b= sd-user/main.c:121 #7 0x00000000600503ea in new_freebsd_thread_start (arg=3D) at = /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/b= sd-user/freebsd/os-thread.c:152 #8 0x00000000601ad1c6 in thread_start (curthread=3D0x8604d5200) at = /usr/src/lib/libthr/thread/thr_create.c:292 #9 0x0000000000000000 in ?? () Backtrace stopped: Cannot access memory at address 0x7fffdfdfc000 PID USERNAME THR PRI NICE SIZE RES SWAP STATE C TIME = WCPU COMMAND 64885 root 7 20 0 79M 12M 0 uwait 2 0:00 = 0.00% /usr/local/bin/qemu-arm-static ctfmerge -L VERSION -g -o bhnd.kld = bhnd.o bhnd_subr.o bhnd_bus_if.o bhnd_erom.o bhnd_erom_if 64846 root 1 52 0 10M 1768K 0 wait 22 0:00 = 0.00% sh -e 46319 root 1 20 0 11M 2420K 0 select 8 0:00 = 0.00% /nxb-bin/usr/bin/make all DIRPRFX=3Dbhnd/ Again: not repeatable for when/where --but has a thread fanout involved. Note: I had set things up to use the nxb-bin/... instead of emulated compiling/linking/etc. Another point: This was a native FreeBSD boot, not under Hyper-V. But it is the same system and same storage media as when used under Hyper-V. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)