From owner-freebsd-current@freebsd.org Mon Nov 25 17:30:10 2019 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 74B171B3403 for ; Mon, 25 Nov 2019 17:30:10 +0000 (UTC) (envelope-from SRS0=C6/G=ZR=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47MDZ90mg5z4fpQ for ; Mon, 25 Nov 2019 17:30:08 +0000 (UTC) (envelope-from SRS0=C6/G=ZR=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 2437F28459; Mon, 25 Nov 2019 18:30:06 +0100 (CET) Received: from illbsd.quip.test (ip-62-24-92-232.net.upcbroadband.cz [62.24.92.232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id B5A3A28461; Mon, 25 Nov 2019 18:30:04 +0100 (CET) Subject: Re: buildworld on CPU-A, installworld on CPU-B ends up with SIGILL To: Ruslan Garipov , freebsd-current@freebsd.org References: <5596338e-134c-9849-de9e-710d3106687f@gmail.com> From: Miroslav Lachman <000.fbsd@quip.cz> Message-ID: Date: Mon, 25 Nov 2019 18:30:04 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Firefox/52.0 SeaMonkey/2.49.3 MIME-Version: 1.0 In-Reply-To: <5596338e-134c-9849-de9e-710d3106687f@gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 47MDZ90mg5z4fpQ X-Spamd-Bar: ++++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of SRS0=C6/G=ZR=quip.cz=000.fbsd@elsa.codelab.cz has no SPF policy when checking 94.124.105.4) smtp.mailfrom=SRS0=C6/G=ZR=quip.cz=000.fbsd@elsa.codelab.cz X-Spamd-Result: default: False [4.08 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; IP_SCORE(0.89)[ip: (0.40), ipnet: 94.124.104.0/21(0.20), asn: 42000(3.75), country: CZ(0.09)]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[quip.cz]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.99)[0.995,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[4.105.124.94.list.dnswl.org : 127.0.10.0]; NEURAL_SPAM_LONG(1.00)[0.996,0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[000.fbsd@quip.cz,SRS0=C6/G=ZR=quip.cz=000.fbsd@elsa.codelab.cz]; FREEMAIL_TO(0.00)[gmail.com]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ]; FROM_NEQ_ENVFROM(0.00)[000.fbsd@quip.cz,SRS0=C6/G=ZR=quip.cz=000.fbsd@elsa.codelab.cz]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Nov 2019 17:30:10 -0000 Ruslan Garipov wrote on 2019/11/25 15:06: > Hello. > > I want to build kernel and world (of FreeBSD 13.0-CURRENT) on a fast > virtual machine for other ones (all the virtual machines are hosted on > VMware EXSi hypervisors, which have different physical CPUs). > > After `make -j16 buildworld` has finished successfully on the build > machine, I get there, for example, > /usr/obj/usr/src/amd64.amd64/tmp/legacy/bin/install program having the > shlxq instruction (one from the BMI2 instruction set extensions). This > eventually causes make installkernel and make installworld to fail with > SIGILL on a virtual machine which must consume built world and kernel, > and which is hosted on another ESXi instance, with older physical CPU > (read: with CPU not implementing shlxq). > > The build machine (FreeBSD 13.0-CURRENT r354802) builds (x)install using > the following commands (a part of buildworld): > > $ cc -O2 -pipe -O2 -march=x86-64 -pipe -I/usr/src/contrib/mtree > -I/usr/src/lib/libnetbsd -DHAVE_STRUCT_STAT_ST_FLAGS=1 -MD > -MF.depend.xinstall.o -MTxinstall.o -std=gnu99 -Wno-format-zero-length > -Qunused-arguments -I/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/include > -c /usr/src/usr.bin/xinstall/xinstall.c -o xinstall.o > $ cc -O2 -pipe -O2 -march=x86-64 -pipe -I/usr/src/contrib/mtree > -I/usr/src/lib/libnetbsd -DHAVE_STRUCT_STAT_ST_FLAGS=1 -MD > -MF.depend.getid.o -MTgetid.o -std=gnu99 -Wno-format-zero-length > -Qunused-arguments -I/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/include > -c /usr/src/contrib/mtree/getid.c -o getid.o > $ cc -O2 -pipe -O2 -march=x86-64 -pipe -I/usr/src/contrib/mtree > -I/usr/src/lib/libnetbsd -DHAVE_STRUCT_STAT_ST_FLAGS=1 -std=gnu99 > -Wno-format-zero-length -Qunused-arguments > -I/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/include -static > -L/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/lib -o xinstall xinstall.o > getid.o -L/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/lib/libmd -lmd -legacy > > This produces xintstall with `shlxq`s: > > $ llvm-objdump --disassemble xinstall | grep -c shlxq > 135 > > I believe statically linked /usr/lib/libmd.a is a stuff which brings > `shlxq`s into the xinstall. I didn't examine it further, sorry... > > My question is: is it possible to buildworld without issuing > instructions which are native for the host CPU? I have neither > /etc/make.conf, nor /etc/src.conf on the build machine. Is it possible > at all for FreeBSD CURRENT to be built outside a host and installed on > the host later? > > Just as a reference: > > My build machine has Intel(R) Xeon(R) Gold 6150 CPU that supports BMI2: > > # cpucontrol -i 7 /dev/cpuctl0 > cpuid level 0x7: 0x00000000 0xd19f6ffb 0x00000018 0xbc000000 > > (Bit 08 in EBX is set) > > , and a consuming machine has Intel(R) Xeon(R) CPU E5-4617 CPU that > doesn't support BMI2: > > # cpucontrol -i 7 /dev/cpuctl0 > cpuid level 0x7: 0x00000000 0x00000002 0x00000000 0xbc000000 > > (Bit 08 in EBX is unset). > > Both machines install kernel and world without any problem, if they were > built locally. I didn't tried this with current but I am using it with stable (11.3 at this time). Building on Xeon E3-1240v3 and installing on many different machines. Some of them are 10+ years old AMD Opteron, some Xeon E5649, some 10 years old Intel Pentium. So at least it worked in the past (11.3 amd64). Did you use this workflow in the past / did it work? I remember some issue in the past which was (accidentaly?) fixed by running "make buildworld && make builkernel && make installkernel && make installworld" on the build host (to some different DESTDIR) and then "make installkernel && make installworld" on the target host (build machine is shared via NFS) Miroslav Lachman