From nobody Fri Dec 29 20:12:35 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T1xPn5JVKz557Td for ; Fri, 29 Dec 2023 20:12:45 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4T1xPn0Xz6z3QFb; Fri, 29 Dec 2023 20:12:44 +0000 (UTC) (envelope-from freebsd@omnilan.de) Authentication-Results: mx1.freebsd.org; none Received: from mh0.gentlemail.de (mh0.gentlemail.de [78.138.80.135]) by mx0.gentlemail.de (8.15.2/8.15.2) with ESMTP id 3BTKCdRJ035623; Fri, 29 Dec 2023 21:12:39 +0100 (CET) (envelope-from freebsd@omnilan.de) Received: from [172.21.3.1] (s1.omnilan.de [217.91.127.234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id A022036C; Fri, 29 Dec 2023 21:12:39 +0100 (CET) Message-ID: <62cc8fed-6acd-45fb-a138-ac7cd218191a@omnilan.de> Date: Fri, 29 Dec 2023 21:12:35 +0100 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: SIGILL when CPUTYPE set to anyting witjh avx and CFLAGS -O2 Content-Language: en-US To: Dimitry Andric Cc: FreeBSD-STABLE References: <72107B4B-F279-471B-8A8F-5B94C5EEDA47@FreeBSD.org> From: Harry Schmalzbauer Organization: OmniLAN In-Reply-To: <72107B4B-F279-471B-8A8F-5B94C5EEDA47@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:61157, ipnet:2a00:e10:2800::/38, country:DE] X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4T1xPn0Xz6z3QFb On 12/29/23 18:00, Dimitry Andric wrote: > On 29 Dec 2023, at 11:14, Harry Schmalzbauer wrote: >> >> at least since 14-release, I can't compile base/ports with CPUTYPE post nehalem. Any '-march' declaring AVX(*) capability leads to SIGILLing binaries with default -O2 CFLAGS. ... >> >> If I compile without CFLAGS -O2 ("CFLAGS= -pipe"), -march=skylake-avx512 the binary doens't crash with SIGILL. >> >> Is CPUTYPE supposed to stay untouched these days? > > Obviously not, but are you sure your target CPU is correct? If it is not, the compiler may insert unsupported instructions. Ho Dimitry, thanks for your attention! I have to admit that I know much too less about contemporary compilers and CPUs likewise, but target CPU matches -march ;-) I nailed it down to AVX instructions. Starting with 'sandybridge', Intel CPUs provide avx instruction units, and starting with CPUTYPE?=sandybridge, binaries crash on stable/14 (and late 14-current) on the corresponding machines. But only in combination with CFLAGS -O2! > In any case, for the above gdb session, try the command "disassemble" to see which particular instruction it crashed on. That may give more information. Thanks for the hint! This came out: Program terminated with signal SIGILL, Illegal instruction. Privileged opcode. #0 0x000000000041e040 in init_yank () (gdb) disassemble Dump of assembler code for function init_yank: 0x000000000041e030 <+0>: push %rbp 0x000000000041e031 <+1>: mov %rsp,%rbp 0x000000000041e034 <+4>: vpxor %xmm0,%xmm0,%xmm0 0x000000000041e038 <+8>: vmovdqa -0x202c10(%rip),%xmm1 # 0x21b430 => 0x000000000041e040 <+16>: kxnorw %k0,%k0,%k1 0x000000000041e044 <+20>: vpscatterdq %ymm0,0x55f180(,%xmm1,1){%k1} 0x000000000041e04f <+31>: vmovdqa -0x202b77(%rip),%xmm1 # 0x21b4e0 0x000000000041e057 <+39>: kxnorw %k0,%k0,%k1 0x000000000041e05b <+43>: vpscatterdq %ymm0,0x55f180(,%xmm1,1){%k1} 0x000000000041e066 <+54>: vmovdqa -0x202c5e(%rip),%xmm1 # 0x21b410 0x000000000041e06e <+62>: kxnorw %k0,%k0,%k1 0x000000000041e072 <+66>: vpscatterdq %ymm0,0x55f180(,%xmm1,1){%k1} 0x000000000041e07d <+77>: vmovdqa -0x202cf5(%rip),%xmm1 # 0x21b390 0x000000000041e085 <+85>: kxnorw %k0,%k0,%k1 0x000000000041e089 <+89>: vpscatterdq %ymm0,0x55f180(,%xmm1,1){%k1} 0x000000000041e094 <+100>: vmovdqa -0x202b5c(%rip),%xmm1 # 0x21b540 0x000000000041e09c <+108>: kxnorw %k0,%k0,%k1 0x000000000041e0a0 <+112>: vpscatterdq %ymm0,0x55f180(,%xmm1,1){%k1} 0x000000000041e0ab <+123>: vmovdqa -0x202e43(%rip),%xmm1 # 0x21b270 0x000000000041e0b3 <+131>: kxnorw %k0,%k0,%k1 0x000000000041e0b7 <+135>: vpscatterdq %ymm0,0x55f180(,%xmm1,1){%k1} 0x000000000041e0c2 <+146>: vmovdqa -0x202d0a(%rip),%xmm1 # 0x21b3c0 0x000000000041e0ca <+154>: kxnorw %k0,%k0,%k1 0x000000000041e0ce <+158>: vpscatterdq %ymm0,0x55f180(,%xmm1,1){%k1} 0x000000000041e0d9 <+169>: vmovdqa -0x202ac1(%rip),%xmm1 # 0x21b620 0x000000000041e0e1 <+177>: kxnorw %k0,%k0,%k1 0x000000000041e0e5 <+181>: vpscatterdq %ymm0,0x55f180(,%xmm1,1){%k1} 0x000000000041e0f0 <+192>: vmovdqa -0x202d58(%rip),%xmm1 # 0x21b3a0 0x000000000041e0f8 <+200>: kxnorw %k0,%k0,%k1 0x000000000041e0fc <+204>: vpscatterdq %ymm0,0x55f180(,%xmm1,1){%k1} 0x000000000041e107 <+215>: movq $0x0,0x1414ee(%rip) # 0x55f600 0x000000000041e112 <+226>: pop %rbp 0x000000000041e113 <+227>: vzeroupper 0x000000000041e116 <+230>: ret End of assembler dump. Example is from 'cc -c -I. -Iproto -DHAVE_CONFIG_H -DLIBICONV_PLUG -I/usr/local/include -O2 -pipe -march=skylake-avx512 -DLIBICONV_PLUG -fstack-protector-strong -fno-strict-aliasing -D_REENTRANT -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -o objects/alloc.o alloc.c' with CPU: Intel(R) Xeon(R) CPU E3-1275 v6 @ 3.80GHz (3800.00-MHz K8-class CPU) When I 1st noticed, I checked also much more recent CPUs - with the same result - if CPUTUPE is set to any avx-involving optimization together with -O2, binaries crash with SIGILL on Haswell, SkyLake, TigerLake... My last world/ports builds (sucessfully with CPUTYPE?=ivybridge) were with stable-13 (arround 13.1) - and llvm 13 I think - can't remember exactly. Thanks, -harry