From owner-freebsd-ports@freebsd.org Sat Dec 29 03:07:05 2018 Return-Path: Delivered-To: freebsd-ports@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D8647142F0C0 for ; Sat, 29 Dec 2018 03:07:04 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic309-22.consmr.mail.ne1.yahoo.com (sonic309-22.consmr.mail.ne1.yahoo.com [66.163.184.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DF9B381FC8 for ; Sat, 29 Dec 2018 03:07:03 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: vN.8rycVM1lUUsYKTXHjGDtKvOmNC2LkQhdaty7awDKGTP80OorGcqTMrWBY_sL NupIbID1wcJueTm2fbWLHg6tvKxWawp5VPIl0qo0uO8.ar.q5b3hHj.4zzgRqU_vEAeWvalsyDPD h1q._kI4A5GL2d2IkZ7DDjPonIed3S5AlaPCthzDaYaJcWziZrSNKuBYm9qA4U9KMXS_hBVUpNBo OMRlmi5ULjFJWnRwanZNkYR33GeY6ocKQNo5hatOr1xkiwC9o3YeXKDUraQuYd0ts.w4uNYFudJg S0uopk1.LiUKnRd9bNLBPnd47kgdU5kR6Pwv.GUX7tyBmqQRGzbXyxpRCISc0FuzqU2QzQC1rFnU DPfAzom.c4I1aUk.7lYp3OXxUl6iBU1hviPtzimqZGb5aqcRIGKZkxy_TwyjtlRVYX3s1_L1KGo5 SGeMEFmwNhNJ8dvBJT3O78rFG6.FZVw7yj4u3GGqPbEBJLGehRmVc9aVtVWMtrubzB1glWQoK5zy r7CXWqTuE48AVoZ1SKv89dvvEuLOdMIrHb1q1f8Gouc46IhYTzT.vXbVBvNzQSAqUVeTEr3ED9o1 5wh3HvUyDIWJr1DXeCjkKy8WdkB3wqUBsSzZvyTQ_U9PU3vBl1rb0iepamJYoXjRiSIwIj3yx9cB tFNuUhVLzaLlJTI2AUCRTazGXl1m5yJhZV4sz_76mRDcq6huW53Vs7K1LUvd3mkL3_WM2AlynUzl 5keu07fKhTLNmEIhXHF.SrhHHwu_gREQnsxoRPSH_hP_BMIWNTaBd4eqyjVabV4TG3IIJqSpeLRc IqciDRjr7apAhLB7EXP9bEpju3FMmvl3ohA.gCFLEZtv3zzIA2WmY2tpnVa_q9DZU__j314HGdC7 TiOa4A9dw7qgptiP13IXKhQ3jwFAfyfwLCWDHM5mJhKhTyRLu4XqViWJISlL0ZgjnjLgTowRTpDv JkvPzIqzOiiLgfsw8wGlPoo5G.50JwyaihOkQYbU6ibwfKtOkZ0Q3SWtZnzt8io3CuhckzI0hBms P7jaZKmuioB7FrRydMP0vYs7TETghgZTDvBwy0dOco6K1SyQawkWrV3oNnHWSkxMDi4TC5aWbUL6 Z Received: from sonic.gate.mail.ne1.yahoo.com by sonic309.consmr.mail.ne1.yahoo.com with HTTP; Sat, 29 Dec 2018 03:06:56 +0000 Received: from c-67-170-167-181.hsd1.or.comcast.net (EHLO [192.168.1.109]) ([67.170.167.181]) by smtp420.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID a30b0319a9e92381b4c3a8625c6cf7d5; Sat, 29 Dec 2018 02:56:45 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.2 \(3445.102.3\)) Subject: Re: A reliable port cross-build failure (hangup) in my context (amd64->armv7 cross build, with native-tool speedup involved) From: Mark Millard In-Reply-To: Date: Fri, 28 Dec 2018 18:56:43 -0800 Cc: freebsd-emulation@freebsd.org, FreeBSD Current , ports-list freebsd , freebsd-arm , FreeBSD Toolchain Content-Transfer-Encoding: quoted-printable Message-Id: <2E3F6196-4652-40D2-937F-8860B6005A35@yahoo.com> References: <865A13C8-9749-486E-9F79-5EEDDECBE621@yahoo.com> <0154C3AC-D85B-4FCF-BA63-454BC26BC1A2@yahoo.com> <13f5e4dd-33fb-2170-e31a-1b5d5f155869@freebsd.org> To: mmel@freebsd.org X-Mailer: Apple Mail (2.3445.102.3) X-Rspamd-Queue-Id: DF9B381FC8 X-Spamd-Bar: + X-Spamd-Result: default: False [1.14 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCPT_COUNT_FIVE(0.00)[6]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36646, ipnet:66.163.184.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.11)[-0.113,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_SPAM_SHORT(0.93)[0.934,0]; NEURAL_HAM_LONG(-0.25)[-0.251,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(1.08)[ip: (3.39), ipnet: 66.163.184.0/21(1.15), asn: 36646(0.92), country: US(-0.08)]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[148.184.163.66.list.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Dec 2018 03:07:05 -0000 On 2018-Dec-28, at 12:12, Mark Millard wrote: > On 2018-Dec-28, at 05:13, Michal Meloun = wrote: >=20 >> Mark, >> this is known problem with qemu-user-static. >> Emulation of every single interruptible syscall is broken by design = (it >> have signal related races). Theses races cannot be solved without = major >> rewrite of syscall emulation code. >> Unfortunately, nobody actively works on this, I think. >>=20 >=20 > Thanks for the note setting some expectations. >=20 > On the evidence that I have I expect that more is going on than that: >=20 > A) The hang-up always happens and always in the same place. So > it would appear that no race is involved. >=20 > B) (A) is true even for varying the number of builders in parallel > (so other builds also happening) and the number of jobs allowed per > builder. It also fails for only one builder allowed only one process. > (I get traces from that last kind of context.) >=20 > C) The problem started on the package-building servers for armv7 > and armv6 without qemu-user-static having an update (FreeBSD and > cmake had updates, for example). >=20 > D) The problem is only observed for targeting armv7 and armv6 as > far as I can tell. I've never seen it for aarch64, neither my > own builds nor when I looked at the package-building server > history. >=20 > At least that is what got me started. (I've since learned that > qemu-user-static uses fork in place of a requested vfork.) >=20 > My ktrace/kdump experiment yesterday showed something odd for the > kevent that hangs in cmake: >=20 > 93172 qemu-arm-static CALL = kevent(0x3,0x7ffffffe7d40,0x2,0x7ffffffd7d40,0x400,0) > 93172 qemu-arm-static STRU struct kevent[] =3D { { ident=3D6, = filter=3DEVFILT_READ, flags=3D0x1, fflags=3D0, data=3D0, = udata=3D0x0 } > { ident=3D0x0, filter=3D, flags=3D0, = fflags=3D0x8, data=3D0x1ffff, udata=3D0x0 } } >=20 > Note the 0x2 argument to kevent and the apparently-odd 2nd entry in = the struct > kevent[]. The kevent use is from cmake. >=20 > So far I've not identified a signal being delivered at a time that = would seem > to me to be likely to contribute. (But this is not familiar code so my = judgment > is likely not the best.) >=20 > Note: I normally run FreeBSD using a non-debug kernel, even when using > head. (The kernel does have symbols.) The detail of the signal usage involved leading up to the hang-up, starting from just before the "press return" for the "make FLAVOR=3Dqt5" command that I had entered: The only "Interrupted system call" prior to my killing the hung cmake process was (kdump -H -r -S output): 93172 100717 qemu-arm-static CALL = execve[59](0x10392,0x8605051a0,0x860cf5400) 93172 101706 qemu-arm-static RET nanosleep[240] -1 errno 4 = Interrupted system call 93172 100717 qemu-arm-static NAMI "/bin/sh" 93172 100717 sh RET execve[59] JUSTRETURN 93172 100717 sh CALL readlink[58](0x207a65,0x7fffffffccc0,0x400) This is where ninja (via qemu-arm-static) execve's the amd64-native = /bin/sh (to in turn later run cmake via qemu-arm-static). (This was after the fork = [for the requested vfork].) So it is for the close-down of the thread that was in nanosleep. There were no PSIG's and no sigreturn's prior to the kill according to = the kdump output. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)