From owner-freebsd-current@freebsd.org Mon Nov 25 18:26:53 2019 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B1DB81B5572 for ; Mon, 25 Nov 2019 18:26:53 +0000 (UTC) (envelope-from ruslanngaripov@gmail.com) Received: from mail-lj1-x241.google.com (mail-lj1-x241.google.com [IPv6:2a00:1450:4864:20::241]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47MFqc4xDLz3GY1 for ; Mon, 25 Nov 2019 18:26:52 +0000 (UTC) (envelope-from ruslanngaripov@gmail.com) Received: by mail-lj1-x241.google.com with SMTP id m4so17047992ljj.8 for ; Mon, 25 Nov 2019 10:26:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language:content-transfer-encoding; bh=e7WAluMMQSduInbIZQ00awvJQkNGWx2mOA9g/CNjQUw=; b=jecoH1nDMki8X/q0aDY8qHrDRq+1D+pT5sqth7QeDAGfNSR910bcHri6ejqwU4Ik9v 9+vqPDRbjakHC4DKbfX+SC46Dpfr+pE7LrCGFr5yimszd6xSFQLr4uoKQpMx33q1GJZa hGFcmaeBKMULwwOemndOnX5UZsLcB2ht7zZIE1rt8S+nI7KJ8j3gOIDJFGKOoLNnjCQR A+bQNbQr2WAGsyeXub4+2s4ngdwxKlmbq4EHSUTTqOGdSRHSBk4ygd5Ll2+OM/AT3Gy+ KZxTk1SJlwcIJ8sJeYqq1mV7TQzz5TCaqB1kOoxoNKKbD8aavEUItwXBfF/9YkUryPIE MNng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=e7WAluMMQSduInbIZQ00awvJQkNGWx2mOA9g/CNjQUw=; b=LPGOewfhQsvjTzewGt3dD9WDyVclMebH6LwL/IacxpC/1HFHWivaPV2KBn1pb1DqJj n32RnzCetv9YfjLUTR59P0Mf1nhbjKv9Hd2RKAafZea/Tlsfp/GcDbpqyweL8bYYKQSJ xZTQEHI1kowMmieSJo46lbjjX7t9nxW7n0n0xSGBIYKDOqrdGWyZBl9Ch/4fosnBNnC0 1wDho8ICm1NDZ2wQPeMt0bL+ED9NXVNe//dj5sOKDAkrA3aVJvZr4vABjmmag7HQFffP twos/B9UnnjCRm2uwRWxzjuT0ctuxUUfZMUD5W5jQLULizjJbmJk/33CC8FzHJCd96G6 Rw/Q== X-Gm-Message-State: APjAAAU4roTN8OgmcusV6radShi4Almx5Vc2EzokNRui4BwANVvzTg6D jScSqpMDdEo7Uj50+AjxN9iQ9GTrMFk= X-Google-Smtp-Source: APXvYqyG0RzzkuMiWyuewqnQMi1103U1DlgVigb+SbxwFD3FEnRBlJqlut3zY+uP5FDuzG1xcC/M9Q== X-Received: by 2002:a05:651c:1117:: with SMTP id d23mr23871052ljo.90.1574706410524; Mon, 25 Nov 2019 10:26:50 -0800 (PST) Received: from [192.168.1.2] ([46.48.69.183]) by smtp.gmail.com with ESMTPSA id b28sm4983303ljp.9.2019.11.25.10.26.49 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 25 Nov 2019 10:26:49 -0800 (PST) Subject: Re: buildworld on CPU-A, installworld on CPU-B ends up with SIGILL To: Miroslav Lachman <000.fbsd@quip.cz>, freebsd-current@freebsd.org References: <5596338e-134c-9849-de9e-710d3106687f@gmail.com> From: Ruslan Garipov Message-ID: Date: Mon, 25 Nov 2019 23:26:46 +0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.2.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 47MFqc4xDLz3GY1 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=jecoH1nD; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of ruslanngaripov@gmail.com designates 2a00:1450:4864:20::241 as permitted sender) smtp.mailfrom=ruslanngaripov@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(0.00)[ip: (2.89), ipnet: 2a00:1450::/32(-2.70), asn: 15169(-1.96), country: US(-0.05)]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[1.4.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Nov 2019 18:26:53 -0000 On 11/25/2019 10:30 PM, Miroslav Lachman wrote: > Ruslan Garipov wrote on 2019/11/25 15:06: >> Hello. >> >> I want to build kernel and world (of FreeBSD 13.0-CURRENT) on a fast >> virtual machine for other ones (all the virtual machines are hosted on >> VMware EXSi hypervisors, which have different physical CPUs). >> >> After `make -j16 buildworld` has finished successfully on the build >> machine, I get there, for example, >> /usr/obj/usr/src/amd64.amd64/tmp/legacy/bin/install program having the >> shlxq instruction (one from the BMI2 instruction set extensions). This >> eventually causes make installkernel and make installworld to fail with >> SIGILL on a virtual machine which must consume built world and kernel, >> and which is hosted on another ESXi instance, with older physical CPU >> (read: with CPU not implementing shlxq). >> >> The build machine (FreeBSD 13.0-CURRENT r354802) builds (x)install using >> the following commands (a part of buildworld): >> >> $ cc -O2 -pipe -O2 -march=x86-64 -pipe -I/usr/src/contrib/mtree >> -I/usr/src/lib/libnetbsd -DHAVE_STRUCT_STAT_ST_FLAGS=1 -MD >> -MF.depend.xinstall.o -MTxinstall.o -std=gnu99 -Wno-format-zero-length >> -Qunused-arguments -I/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/include >> -c /usr/src/usr.bin/xinstall/xinstall.c -o xinstall.o >> $ cc -O2 -pipe -O2 -march=x86-64 -pipe -I/usr/src/contrib/mtree >> -I/usr/src/lib/libnetbsd -DHAVE_STRUCT_STAT_ST_FLAGS=1 -MD >> -MF.depend.getid.o -MTgetid.o -std=gnu99 -Wno-format-zero-length >> -Qunused-arguments -I/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/include >> -c /usr/src/contrib/mtree/getid.c -o getid.o >> $ cc -O2 -pipe -O2 -march=x86-64 -pipe -I/usr/src/contrib/mtree >> -I/usr/src/lib/libnetbsd -DHAVE_STRUCT_STAT_ST_FLAGS=1 -std=gnu99 >> -Wno-format-zero-length -Qunused-arguments >> -I/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/include -static >> -L/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/lib -o xinstall xinstall.o >> getid.o -L/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/lib/libmd -lmd -legacy >> >> This produces xintstall with `shlxq`s: >> >> $ llvm-objdump --disassemble xinstall | grep -c shlxq >> 135 >> >> I believe statically linked /usr/lib/libmd.a is a stuff which brings >> `shlxq`s into the xinstall. I didn't examine it further, sorry... >> >> My question is: is it possible to buildworld without issuing >> instructions which are native for the host CPU? I have neither >> /etc/make.conf, nor /etc/src.conf on the build machine. Is it possible >> at all for FreeBSD CURRENT to be built outside a host and installed on >> the host later? >> >> Just as a reference: >> >> My build machine has Intel(R) Xeon(R) Gold 6150 CPU that supports BMI2: >> >> # cpucontrol -i 7 /dev/cpuctl0 >> cpuid level 0x7: 0x00000000 0xd19f6ffb 0x00000018 0xbc000000 >> >> (Bit 08 in EBX is set) >> >> , and a consuming machine has Intel(R) Xeon(R) CPU E5-4617 CPU that >> doesn't support BMI2: >> >> # cpucontrol -i 7 /dev/cpuctl0 >> cpuid level 0x7: 0x00000000 0x00000002 0x00000000 0xbc000000 >> >> (Bit 08 in EBX is unset). >> >> Both machines install kernel and world without any problem, if they were >> built locally. > > I didn't tried this with current but I am using it with stable (11.3 at > this time). Building on Xeon E3-1240v3 and installing on many different > machines. Some of them are 10+ years old AMD Opteron, some Xeon E5649, > some 10 years old Intel Pentium. > So at least it worked in the past (11.3 amd64). Did you use this > workflow in the past / did it work? No, unfortunately I didn't. Always built world/kernel on target host. > I remember some issue in the past which was (accidentaly?) fixed by > running "make buildworld && make builkernel && make installkernel && > make installworld" on the build host (to some different DESTDIR) and > then "make installkernel && make installworld" on the target host (build > machine is shared via NFS) Therefore, this trick somehow "fixes" /usr/obj shared on the build machine? I'll try this later. Thanks! > > Miroslav Lachman >