From owner-freebsd-current@freebsd.org Fri Dec 28 13:13:27 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9116914385EA; Fri, 28 Dec 2018 13:13:27 +0000 (UTC) (envelope-from melounmichal@gmail.com) Received: from mail-wr1-x443.google.com (mail-wr1-x443.google.com [IPv6:2a00:1450:4864:20::443]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7CC4F8A45B; Fri, 28 Dec 2018 13:13:26 +0000 (UTC) (envelope-from melounmichal@gmail.com) Received: by mail-wr1-x443.google.com with SMTP id q18so20884606wrx.9; Fri, 28 Dec 2018 05:13:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:reply-to:subject:to:cc:references:openpgp:autocrypt:message-id :date:user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=JdfPZqVXFSTDaypPNx9VwESn1tLKAY4b6HjsrDtJslo=; b=JlXyZS8YN5HTFZamWSZi/GNS19YmOybUKCFJMINKQKVaP//C1ztYxVtXyTKZZz2C+2 GJ7paIMr0jXXGOvU9MQ/Xg9nhJ1SR2iEgSW7+BpFV+xTZM8CF5f+CsDPLKlBpfHiR1P8 JAbW/dcQeruHDnhYE4ilPEnuX72Vpc7Deop8/Z63CQg0OV40dlPHPvFpwLLDn9bA2QYK s69MIrtAJQE2SmPvPPxTW+NsmjPGFkUfuYWBRvdhN8t4rosVJ/CJjMw+vp/4cVqBTw9U i7WHPAVnY8owgAkEa2jc92xmIjQuaBl9aYkxdFVVzClGtX7FvXllxp1A6VatUfLSypG3 7avQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:reply-to:subject:to:cc:references:openpgp :autocrypt:message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=JdfPZqVXFSTDaypPNx9VwESn1tLKAY4b6HjsrDtJslo=; b=O+vJfwClLyPuDMWJhXsJVjq4zzmNx8PeexPSDIy4XZ68RvjLRXBKcEtDjiqOVPZneS vm3cfDsvS1PU+onqmysYHSJkVEq6IiZdbGnh+xi9TN226bDHgX4PuVdDebJTPheO1WpP avv6xg4y8gWCb2br8VwrbHN1FHktEy+JYpPF94+hDMjsgamOSrtgkBjXQX9Vohs+iiCv Fo5KClZ0fLyonaQls59gP9mdrfthDpFsx6gcHhz3AApLe65PVo0VHJrcYV8RWKSHJGeb W190KZ2P4GJKX2S3O1NcMofkOGFIXwGAOYigwZDrP548MrgMLy3p0oC/fvclZPMoSQj0 5NNA== X-Gm-Message-State: AJcUukc1F1VfEf6e2eYt/J5oiZKAWhhVgBuFspDalaEJ4wtpOjB88CWN gOiZ07DFjBgsES7j4IFgkygi6+pBOkI= X-Google-Smtp-Source: ALg8bN602D5pVEoG3DAlRGEYrm/4TOwzvErBayiIVY8/up7lkYhlJC5D/qx5EVapLYPO2DD3TikwSw== X-Received: by 2002:adf:be8b:: with SMTP id i11mr26586057wrh.235.1546002805216; Fri, 28 Dec 2018 05:13:25 -0800 (PST) Received: from [88.208.79.100] (halouny.humusoft.cz. [88.208.79.100]) by smtp.gmail.com with ESMTPSA id g188sm23795644wmf.32.2018.12.28.05.13.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 28 Dec 2018 05:13:24 -0800 (PST) From: Michal Meloun X-Google-Original-From: Michal Meloun Reply-To: mmel@freebsd.org Subject: Re: A reliable port cross-build failure (hangup) in my context (amd64->armv7 cross build, with native-tool speedup involved) To: Mark Millard , freebsd-emulation@freebsd.org, FreeBSD Current , ports-list freebsd Cc: freebsd-arm , FreeBSD Toolchain References: <865A13C8-9749-486E-9F79-5EEDDECBE621@yahoo.com> <0154C3AC-D85B-4FCF-BA63-454BC26BC1A2@yahoo.com> Openpgp: preference=signencrypt Autocrypt: addr=mmel@freebsd.org; prefer-encrypt=mutual; keydata= mQENBFYuVRkBCADZiwLCCne3wG9b9k+R2Neo5zVo2bLaZRfNNY/v9kg283i0sb1Da4EdEiNT 15El5UyozhphUIbIR/zrVpxF1TvvFdoCyzx6a68bNY2d9dBrDcNDZC+XnyDdHQoobN87DWT1 mRVkmbg9LHZ/SVUOkGYuWyE+8UYeDAcUizuXwSK5zFWmeTyIoWNa68ifrWLfQe0p4x5jC/AI VURCi17p360vU4fhgwoMvEEhrRBWCr4DYHToFjIt2WdBy3GR1qoO0+Xkd6G+OoBULo+XDfgu L2WdPvh0K69F9/LgHkMmG5Il7SCe62QGpG2vaCgRV7BQhLX+kxlvM+WrdRatWRml4Y/3ABEB AAG0IE1pY2hhbCBNZWxvdW4gPG1tZWxAZnJlZWJzZC5vcmc+iQFABBMBCgAqAhsDBQsJCAcD BRUKCQgLBRYDAgEAAh4BAheAAhkBBQJZjBHDBQkHICOqAAoJEGkesmtexaqqIKMIAJ9xTp1w ge86ns2ZYOac5++mAgpFatohSlxYUR3gwud3Y3Ej0eumavpv/C26N6dsLnspwRenKdLbIPKe 0N8lI7CcDBIJGiFyY3c4H79QjIkYpRgbWFyCM85zEyVJpB+U7BhsgXE2uwVjE9RNhEP0KBoj sp357uqq1B1+VUO4GJ+RjdmYSOcNrjR8tTfy02456qovGjJ4JcJBlhyK6GzBKvnZSoA0s+QP OMn3gd8gdomMLEJdS3kTsfhLh2rQPZa9EmzafIyjXrirWq4+4fVFgd8SiMZyyTM+Kz30ZSUe 6SmfaQTQ/WLRIl5jku2uYQWlrRIKT9xaQzRWtZO9UgtXFRG5AQ0EVi5VGQEIALqgRkfS21D/ OqWE9mXfh2bIjrp9uC8T0MCuimbsrAdLKNNorGu2nE+rebgX8n5nYM377HOnalPGyOuXvCbQ 8MFVRdWOHxenJjXJialNdBsOf2wLva3vSSVsdoPzibWDIcJqhBOQ3EuhsILyWSPvYYKEiy95 mfhrDtuTTOAYVR9aNQBOENztB2TDJyMx/qZmtGroGV3N0Hqde/znHPtQO8RG5/FQGMfHMI5G FMuycr1ceHnLo/ovrqAl4TYV+UHSHJ+FDE9dt9wXHclWbWbC0yNugchZq6rho5Jjfv4a2v7P pyn3HoDinh1lWP7hYA0ZNExGHekLnXWVqO/lzGS6bMEAEQEAAYkBJQQYAQoADwIbDAUCWYwR wwUJByAjqgAKCRBpHrJrXsWqqrsrB/4g4ESK5TLxUxi8pLWcLPyvwtN4Fmf7VsCVefkhakaG rDPmfvfnG+OFwN60Xqoni7GBeakl01xwT4RINfvVfShDy6cHpLS7QL/M8pzfulVX38MkVkOD yGZhwjE+jyT/kZNA1Olaw3N3IefHq3brskQ7G4d9oPep2DDbw7C4Q76uOBjxy34JVB0WOsB6 NyMQB9h6LGljQtdEddyUqwnRZzzHiGvp0hPtdYQHQZlqbj4FV9lTRK7a8Ega+y7MgmeMiztG zeXyjNP02r3PRHCPagwa57bPxH2aAh4Q7UzBBZ0GTMm7DLKNtCP58WDxblrrhZ+7kHqGK8Fs bdeUpDdEYLVd Message-ID: Date: Fri, 28 Dec 2018 14:13:25 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.3.3 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 7CC4F8A45B X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=JlXyZS8Y; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of melounmichal@gmail.com designates 2a00:1450:4864:20::443 as permitted sender) smtp.mailfrom=melounmichal@gmail.com X-Spamd-Result: default: False [-4.27 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_REPLYTO(0.00)[mmel@freebsd.org]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; RCPT_COUNT_FIVE(0.00)[6]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.76)[-0.760,0]; FREEMAIL_TO(0.00)[yahoo.com]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(-0.51)[ip: (0.82), ipnet: 2a00:1450::/32(-1.76), asn: 15169(-1.51), country: US(-0.08)]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; REPLYTO_DOM_NEQ_FROM_DOM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[3.4.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Dec 2018 13:13:27 -0000 On 24.12.2018 8:28, Mark Millard wrote: > [I built a FreeBSD head -r340288 context and tried ports head > -r484783 and the problem repeated.] > > On 2018-Dec-22, at 12:55, Mark Millard wrote: > >> [I found my E-mail records reporting successful builds using >> qemu-user-static from ports head -r484783 under FreeBSD >> head -r340287.] >> >> On 2018-Dec-22, at 00:10, Mark Millard wrote: >> >>> [I messed up the freebsd-emulation email address the first time I sent >>> this. I also forgot to indicate the qemu-user-static vintage relationship.] >>> >>> I had been reporting intermittent hang-ups for my amd64->{aarch64,armv7} port cross >>> builds in another message sequence. But it turns out that one thing I ran into >>> has hung-up every time, the same way, for amd64->armv7 cross builds: >>> multimedia/gstreamer1-qt@qt5 . So I extract the material here into a separate report >>> with some updated notes. >>> >>> A little context: I had built from ports head -r484783 before under FreeBSD head >>> -r340287 (as I remember the version). Back then it did not have this problem that it >>> now has under FreeBSD head -r341836 . One ports-specific change was to force perl5.28 >>> as the default instead of perl5.26 originally. In fact this is what drives what is >>> being rebuilt for my experiment that caught this. But I doubt the perl version is >>> important to the problem. The context has a Ryzen Threadripper 1950X and has been >>> tested both for FreeBSD under Hyper-V and for the same media native-booted. Both >>> hang-up at the same point as seen via ps or top. The native tools for cross-build >>> speedup were in use. Cross-builds targeting aarch64 did not get this problem but >>> targeting armv7 did. 121 of 129 armv7 ports built before the hang-up for the first >>> armv7 try. >>> >>> ADDED: The qemu-user-static back with head -r340287 before installing the >>> updated ports would likely be different than the -r484783 vintage. So both >>> FreeBSD and qemu-user-static may have changed over the comparison. >> >> CORRECTION to ADDED: Back on 2018-Nov-11 I reported successful cross-builds >> based on qemu-user-static from ports head -484783 --all built under FreeBSD >> head -r340287 . So the use of the perl5.28 as the forced-default and the >> newer FreeBSD head version -r341836 as the context are the differences here. >> >>> The hang-up: >>> >>> In the port rebuilds targeting armv7, multimedia/gstreamer1-qt@qt5 hung-up and timed >>> out. Looking during the wait in later tries shows something much like (from one of the >>> examples): >>> >>> root 33719 0.0 0.0 12920 3528 0 I 11:40 0:00.03 | | `-- sh: poudriere[FBSDFSSDjailArmV7-default][02]: build_pkg (gstreamer1-qt5-1.2.0_14) (sh) >>> root 41551 0.0 0.0 12920 3520 0 I 11:43 0:00.00 | | `-- sh: poudriere[FBSDFSSDjailArmV7-default][02]: build_pkg (gstreamer1-qt5-1.2.0_14) (sh) >>> root 41552 0.0 0.0 10340 1744 0 IJ 11:43 0:00.01 | | `-- /usr/bin/make -C /usr/ports/multimedia/gstreamer1-qt FLAVOR=qt5 build >>> root 41566 0.0 0.0 10236 1796 0 IJ 11:43 0:00.00 | | `-- /bin/sh -e -c (cd /wrkdirs/usr/ports/multimedia/gstreamer1-qt/work-qt5/.build; if ! /usr/bin/env QT_SELE >>> root 41567 0.0 0.0 89976 12896 0 IJ 11:43 0:00.07 | | `-- /usr/local/bin/qemu-arm-static ninja -j28 -v all >>> root 41585 0.0 0.0 102848 25056 0 IJ 11:43 0:00.10 | | |-- /usr/local/bin/qemu-arm-static /usr/local/bin/cmake -E cmake_autogen /wrkdirs/usr/ports/multimedia/g >>> root 41586 0.0 0.0 102852 25072 0 IJ 11:43 0:00.11 | | `-- /usr/local/bin/qemu-arm-static /usr/local/bin/cmake -E cmake_autogen /wrkdirs/usr/ports/multimedia/g >>> >>> or as top showed it: >>> >>> 41552 root 1 52 0 10M 1744K 0 wait 15 0:00 0.00% /usr/bin/make -C /usr/ports/multimedia/gstreamer1-qt FLAVOR=qt5 build >>> 41566 root 1 52 0 10M 1796K 0 wait 1 0:00 0.00% /bin/sh -e -c (cd /wrkdirs/usr/ports/multimedia/gstreamer1-qt/work-qt5/.build; if ! /usr/bin/env QT_SELECT=qt5 QMAKEMODULES >>> 41567 root 2 52 0 88M 13M 0 select 4 0:00 0.00% /usr/local/bin/qemu-arm-static ninja -j28 -v all >>> 41585 root 2 52 0 100M 24M 0 kqread 8 0:00 0.00% /usr/local/bin/qemu-arm-static /usr/local/bin/cmake -E cmake_autogen /wrkdirs/usr/ports/multimedia/gstreamer1-qt/work-qt5/. >>> 41586 root 2 52 0 100M 24M 0 kqread 22 0:00 0.00% /usr/local/bin/qemu-arm-static /usr/local/bin/cmake -E cmake_autogen /wrkdirs/usr/ports/multimedia/gstreamer1-qt/work-qt5/. >>> >>> So: waiting in kqread trying to run cmake. >>> >>> Unlike some intermittent hang-ups, attaching-then-detaching via gdb does not >>> resume the hung-up processes. Kills of the processes waiting on kqread stop >>> the build. >>> >>> Given the prior ports have been built already, building just >>> multimedia/gstreamer1-qt@qt5 still gets the hang-up at the same point. >>> >>> Building anything that requires multimedia/gstreamer1-qt@qt5 seems to be >>> solidly blocked in my environment. > > > I built a FreeBSD head -r340288 context and tried cross-buiding an > amd64->armv7 ports head -r484783 of my usual ports and the problem > repeated. I also found evidence that originally in the old time frame > I'd disabled part of my originally-intended port builds because of > other problems so multimedia/gstreamer1-qt 's build was not being > tried. > > So the qemu-user-static vintage or content may be what to vary to > narrow down the problem instead of bisecting FreeBSD kernel or world > vintages. clang7 building qemu-user-static or the kernel/world has > been eliminated. > > > (I used -r340288 to match a artifact.ci.freebsd.org build, incorrectly > expecting to bisect via kernel substitutions.) > Mark, this is known problem with qemu-user-static. Emulation of every single interruptible syscall is broken by design (it have signal related races). Theses races cannot be solved without major rewrite of syscall emulation code. Unfortunately, nobody actively works on this, I think. Michal