From owner-freebsd-emulation@freebsd.org Fri Dec 28 13:13:28 2018 Return-Path: Delivered-To: freebsd-emulation@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2738314385F1; Fri, 28 Dec 2018 13:13:28 +0000 (UTC) (envelope-from melounmichal@gmail.com) Received: from mail-wm1-x343.google.com (mail-wm1-x343.google.com [IPv6:2a00:1450:4864:20::343]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 188EA8A45C; Fri, 28 Dec 2018 13:13:27 +0000 (UTC) (envelope-from melounmichal@gmail.com) Received: by mail-wm1-x343.google.com with SMTP id t200so7734110wmt.0; Fri, 28 Dec 2018 05:13:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:reply-to:subject:to:cc:references:openpgp:autocrypt:message-id :date:user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=JdfPZqVXFSTDaypPNx9VwESn1tLKAY4b6HjsrDtJslo=; b=bu9+24h1dO+/JxYYnWaPP2s3+R+cADexDDheDvowoosSR7dkrdH0uWs8hYSGXcLX3d f3TzO3B8Chq7VxLQXTHwVUCDqfrdkXQblWufbzH97QrVrRqzYQqyOGn1fb+/0Ey621/8 8iq4hIC2o5omM5/DmANFR/84UKBIrba2MQYJS9+A+994NZr5FO4vuz3Jxs+mMq6iET9s hhagzZop6Dfrlzin9A1RZrBgiKKARYxWoZVVtBC0Ra6QO5IGbgdBJoFYW6MWiCUH2zsR 3SCd5UhrUSYGQ1mb2ucIIG2C/z7i2SsHlte0Hc4OH57kzEz6+/N+XzmvZwAh2k+qYD38 aXgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:reply-to:subject:to:cc:references:openpgp :autocrypt:message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=JdfPZqVXFSTDaypPNx9VwESn1tLKAY4b6HjsrDtJslo=; b=a5Uv5++JU1JdxuveF+OU7Df6WSK+a8o/XeaasACZLkbwAycdhzPo7rv6QPVm4GxPpw 7F3RMdwFgaIR3x5NwFwzpW8es2lqOWazYA+LWu83TSW0fD0mY37H6d3ofs06p4AMo8Mr yuUjjE6KM0c1TIt2b6tI/mO+799Ij0ZGjXk+qMklD38tfJg9pGh5aVESIh7ZyFG7eZyN 8CWLSLiO7HNgFiJzc7fzc2Jzv3mmLN38V9wgYIjww9n9HCZAWyA/7xVklgrTIaLwdnXs O9rn3633PAiY6abx8vJbScLNqnHGxjxmZ08QZs/IprKF0rO6KSHbhN3VmMyJjNtnqJsk jgkA== X-Gm-Message-State: AA+aEWbgAtq/b0PKDoYDeg5lyrfc9Mv6+BQQ0PL6ftuV/j0Nu4enfqch 9X1hkYUX6b3qwN9BNloZOziFrJUQGys= X-Google-Smtp-Source: ALg8bN6qPuX61IRuZjgq1M3f14uv4qIJOj00Y1i5Amq4Jp30ha/r3H0Wg0DIUqAJrjOy7AsXWV77Lg== X-Received: by 2002:a1c:9a0d:: with SMTP id c13mr24826750wme.41.1546002805646; Fri, 28 Dec 2018 05:13:25 -0800 (PST) Received: from [88.208.79.100] (halouny.humusoft.cz. [88.208.79.100]) by smtp.gmail.com with ESMTPSA id k7sm27601306wrl.51.2018.12.28.05.13.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 28 Dec 2018 05:13:24 -0800 (PST) From: Michal Meloun X-Google-Original-From: Michal Meloun Reply-To: mmel@freebsd.org Subject: Re: A reliable port cross-build failure (hangup) in my context (amd64->armv7 cross build, with native-tool speedup involved) To: Mark Millard , freebsd-emulation@freebsd.org, FreeBSD Current , ports-list freebsd Cc: freebsd-arm , FreeBSD Toolchain References: <865A13C8-9749-486E-9F79-5EEDDECBE621@yahoo.com> <0154C3AC-D85B-4FCF-BA63-454BC26BC1A2@yahoo.com> Openpgp: preference=signencrypt Autocrypt: addr=mmel@freebsd.org; prefer-encrypt=mutual; keydata= mQENBFYuVRkBCADZiwLCCne3wG9b9k+R2Neo5zVo2bLaZRfNNY/v9kg283i0sb1Da4EdEiNT 15El5UyozhphUIbIR/zrVpxF1TvvFdoCyzx6a68bNY2d9dBrDcNDZC+XnyDdHQoobN87DWT1 mRVkmbg9LHZ/SVUOkGYuWyE+8UYeDAcUizuXwSK5zFWmeTyIoWNa68ifrWLfQe0p4x5jC/AI VURCi17p360vU4fhgwoMvEEhrRBWCr4DYHToFjIt2WdBy3GR1qoO0+Xkd6G+OoBULo+XDfgu L2WdPvh0K69F9/LgHkMmG5Il7SCe62QGpG2vaCgRV7BQhLX+kxlvM+WrdRatWRml4Y/3ABEB AAG0IE1pY2hhbCBNZWxvdW4gPG1tZWxAZnJlZWJzZC5vcmc+iQFABBMBCgAqAhsDBQsJCAcD BRUKCQgLBRYDAgEAAh4BAheAAhkBBQJZjBHDBQkHICOqAAoJEGkesmtexaqqIKMIAJ9xTp1w ge86ns2ZYOac5++mAgpFatohSlxYUR3gwud3Y3Ej0eumavpv/C26N6dsLnspwRenKdLbIPKe 0N8lI7CcDBIJGiFyY3c4H79QjIkYpRgbWFyCM85zEyVJpB+U7BhsgXE2uwVjE9RNhEP0KBoj sp357uqq1B1+VUO4GJ+RjdmYSOcNrjR8tTfy02456qovGjJ4JcJBlhyK6GzBKvnZSoA0s+QP OMn3gd8gdomMLEJdS3kTsfhLh2rQPZa9EmzafIyjXrirWq4+4fVFgd8SiMZyyTM+Kz30ZSUe 6SmfaQTQ/WLRIl5jku2uYQWlrRIKT9xaQzRWtZO9UgtXFRG5AQ0EVi5VGQEIALqgRkfS21D/ OqWE9mXfh2bIjrp9uC8T0MCuimbsrAdLKNNorGu2nE+rebgX8n5nYM377HOnalPGyOuXvCbQ 8MFVRdWOHxenJjXJialNdBsOf2wLva3vSSVsdoPzibWDIcJqhBOQ3EuhsILyWSPvYYKEiy95 mfhrDtuTTOAYVR9aNQBOENztB2TDJyMx/qZmtGroGV3N0Hqde/znHPtQO8RG5/FQGMfHMI5G FMuycr1ceHnLo/ovrqAl4TYV+UHSHJ+FDE9dt9wXHclWbWbC0yNugchZq6rho5Jjfv4a2v7P pyn3HoDinh1lWP7hYA0ZNExGHekLnXWVqO/lzGS6bMEAEQEAAYkBJQQYAQoADwIbDAUCWYwR wwUJByAjqgAKCRBpHrJrXsWqqrsrB/4g4ESK5TLxUxi8pLWcLPyvwtN4Fmf7VsCVefkhakaG rDPmfvfnG+OFwN60Xqoni7GBeakl01xwT4RINfvVfShDy6cHpLS7QL/M8pzfulVX38MkVkOD yGZhwjE+jyT/kZNA1Olaw3N3IefHq3brskQ7G4d9oPep2DDbw7C4Q76uOBjxy34JVB0WOsB6 NyMQB9h6LGljQtdEddyUqwnRZzzHiGvp0hPtdYQHQZlqbj4FV9lTRK7a8Ega+y7MgmeMiztG zeXyjNP02r3PRHCPagwa57bPxH2aAh4Q7UzBBZ0GTMm7DLKNtCP58WDxblrrhZ+7kHqGK8Fs bdeUpDdEYLVd Message-ID: <13f5e4dd-33fb-2170-e31a-1b5d5f155869@freebsd.org> Date: Fri, 28 Dec 2018 14:13:25 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.3.3 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 188EA8A45C X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=bu9+24h1; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of melounmichal@gmail.com designates 2a00:1450:4864:20::343 as permitted sender) smtp.mailfrom=melounmichal@gmail.com X-Spamd-Result: default: False [-3.17 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_REPLYTO(0.00)[mmel@freebsd.org]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; RCPT_COUNT_FIVE(0.00)[6]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; FREEMAIL_TO(0.00)[yahoo.com]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(-0.08)[ip: (2.97), ipnet: 2a00:1450::/32(-1.76), asn: 15169(-1.51), country: US(-0.08)]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.980,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_SHORT(-0.11)[-0.110,0]; NEURAL_HAM_LONG(-0.99)[-0.993,0]; MIME_GOOD(-0.10)[text/plain]; REPLYTO_DOM_NEQ_FROM_DOM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[3.4.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[] X-BeenThere: freebsd-emulation@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Development of Emulators of other operating systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Dec 2018 13:13:28 -0000 On 24.12.2018 8:28, Mark Millard wrote: > [I built a FreeBSD head -r340288 context and tried ports head > -r484783 and the problem repeated.] > > On 2018-Dec-22, at 12:55, Mark Millard wrote: > >> [I found my E-mail records reporting successful builds using >> qemu-user-static from ports head -r484783 under FreeBSD >> head -r340287.] >> >> On 2018-Dec-22, at 00:10, Mark Millard wrote: >> >>> [I messed up the freebsd-emulation email address the first time I sent >>> this. I also forgot to indicate the qemu-user-static vintage relationship.] >>> >>> I had been reporting intermittent hang-ups for my amd64->{aarch64,armv7} port cross >>> builds in another message sequence. But it turns out that one thing I ran into >>> has hung-up every time, the same way, for amd64->armv7 cross builds: >>> multimedia/gstreamer1-qt@qt5 . So I extract the material here into a separate report >>> with some updated notes. >>> >>> A little context: I had built from ports head -r484783 before under FreeBSD head >>> -r340287 (as I remember the version). Back then it did not have this problem that it >>> now has under FreeBSD head -r341836 . One ports-specific change was to force perl5.28 >>> as the default instead of perl5.26 originally. In fact this is what drives what is >>> being rebuilt for my experiment that caught this. But I doubt the perl version is >>> important to the problem. The context has a Ryzen Threadripper 1950X and has been >>> tested both for FreeBSD under Hyper-V and for the same media native-booted. Both >>> hang-up at the same point as seen via ps or top. The native tools for cross-build >>> speedup were in use. Cross-builds targeting aarch64 did not get this problem but >>> targeting armv7 did. 121 of 129 armv7 ports built before the hang-up for the first >>> armv7 try. >>> >>> ADDED: The qemu-user-static back with head -r340287 before installing the >>> updated ports would likely be different than the -r484783 vintage. So both >>> FreeBSD and qemu-user-static may have changed over the comparison. >> >> CORRECTION to ADDED: Back on 2018-Nov-11 I reported successful cross-builds >> based on qemu-user-static from ports head -484783 --all built under FreeBSD >> head -r340287 . So the use of the perl5.28 as the forced-default and the >> newer FreeBSD head version -r341836 as the context are the differences here. >> >>> The hang-up: >>> >>> In the port rebuilds targeting armv7, multimedia/gstreamer1-qt@qt5 hung-up and timed >>> out. Looking during the wait in later tries shows something much like (from one of the >>> examples): >>> >>> root 33719 0.0 0.0 12920 3528 0 I 11:40 0:00.03 | | `-- sh: poudriere[FBSDFSSDjailArmV7-default][02]: build_pkg (gstreamer1-qt5-1.2.0_14) (sh) >>> root 41551 0.0 0.0 12920 3520 0 I 11:43 0:00.00 | | `-- sh: poudriere[FBSDFSSDjailArmV7-default][02]: build_pkg (gstreamer1-qt5-1.2.0_14) (sh) >>> root 41552 0.0 0.0 10340 1744 0 IJ 11:43 0:00.01 | | `-- /usr/bin/make -C /usr/ports/multimedia/gstreamer1-qt FLAVOR=qt5 build >>> root 41566 0.0 0.0 10236 1796 0 IJ 11:43 0:00.00 | | `-- /bin/sh -e -c (cd /wrkdirs/usr/ports/multimedia/gstreamer1-qt/work-qt5/.build; if ! /usr/bin/env QT_SELE >>> root 41567 0.0 0.0 89976 12896 0 IJ 11:43 0:00.07 | | `-- /usr/local/bin/qemu-arm-static ninja -j28 -v all >>> root 41585 0.0 0.0 102848 25056 0 IJ 11:43 0:00.10 | | |-- /usr/local/bin/qemu-arm-static /usr/local/bin/cmake -E cmake_autogen /wrkdirs/usr/ports/multimedia/g >>> root 41586 0.0 0.0 102852 25072 0 IJ 11:43 0:00.11 | | `-- /usr/local/bin/qemu-arm-static /usr/local/bin/cmake -E cmake_autogen /wrkdirs/usr/ports/multimedia/g >>> >>> or as top showed it: >>> >>> 41552 root 1 52 0 10M 1744K 0 wait 15 0:00 0.00% /usr/bin/make -C /usr/ports/multimedia/gstreamer1-qt FLAVOR=qt5 build >>> 41566 root 1 52 0 10M 1796K 0 wait 1 0:00 0.00% /bin/sh -e -c (cd /wrkdirs/usr/ports/multimedia/gstreamer1-qt/work-qt5/.build; if ! /usr/bin/env QT_SELECT=qt5 QMAKEMODULES >>> 41567 root 2 52 0 88M 13M 0 select 4 0:00 0.00% /usr/local/bin/qemu-arm-static ninja -j28 -v all >>> 41585 root 2 52 0 100M 24M 0 kqread 8 0:00 0.00% /usr/local/bin/qemu-arm-static /usr/local/bin/cmake -E cmake_autogen /wrkdirs/usr/ports/multimedia/gstreamer1-qt/work-qt5/. >>> 41586 root 2 52 0 100M 24M 0 kqread 22 0:00 0.00% /usr/local/bin/qemu-arm-static /usr/local/bin/cmake -E cmake_autogen /wrkdirs/usr/ports/multimedia/gstreamer1-qt/work-qt5/. >>> >>> So: waiting in kqread trying to run cmake. >>> >>> Unlike some intermittent hang-ups, attaching-then-detaching via gdb does not >>> resume the hung-up processes. Kills of the processes waiting on kqread stop >>> the build. >>> >>> Given the prior ports have been built already, building just >>> multimedia/gstreamer1-qt@qt5 still gets the hang-up at the same point. >>> >>> Building anything that requires multimedia/gstreamer1-qt@qt5 seems to be >>> solidly blocked in my environment. > > > I built a FreeBSD head -r340288 context and tried cross-buiding an > amd64->armv7 ports head -r484783 of my usual ports and the problem > repeated. I also found evidence that originally in the old time frame > I'd disabled part of my originally-intended port builds because of > other problems so multimedia/gstreamer1-qt 's build was not being > tried. > > So the qemu-user-static vintage or content may be what to vary to > narrow down the problem instead of bisecting FreeBSD kernel or world > vintages. clang7 building qemu-user-static or the kernel/world has > been eliminated. > > > (I used -r340288 to match a artifact.ci.freebsd.org build, incorrectly > expecting to bisect via kernel substitutions.) > Mark, this is known problem with qemu-user-static. Emulation of every single interruptible syscall is broken by design (it have signal related races). Theses races cannot be solved without major rewrite of syscall emulation code. Unfortunately, nobody actively works on this, I think. Michal