From nobody Sun May 22 23:03:39 2022 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6FBAF1B4CB07 for ; Sun, 22 May 2022 23:04:07 +0000 (UTC) (envelope-from jfc@mit.edu) Received: from outgoing-exchange-1.mit.edu (outgoing-exchange-1.mit.edu [18.9.28.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.outgoing-exchange.mit.edu", Issuer "InCommon RSA Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4L5wxy2wQDz4mK2; Sun, 22 May 2022 23:04:06 +0000 (UTC) (envelope-from jfc@mit.edu) Received: from w92exedge3.exchange.mit.edu (W92EXEDGE3.EXCHANGE.MIT.EDU [18.7.73.15]) by outgoing-exchange-1.mit.edu (8.14.7/8.12.4) with ESMTP id 24MN3sVn024749; Sun, 22 May 2022 19:03:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mit.edu; s=outgoing; t=1653260640; bh=hLxAoIzymnU7WUmUyvarQd06kGgp5zPKVDpNCOZOsQA=; h=From:To:CC:Subject:Date:References:In-Reply-To; b=gJdPSlJ/nM/fVthdwbJEKs91q4ik961RY3rHhZ2Gcgb1Y4IbAbQD3oAsV5YqBVoOF vhzF05fEU5HGP3dnSU45mPwHp/ar7SSnADxqBlotJ5a/Bs8GrfxKLDtgxRGTDj5PjK uPyVl8e6NlFPXukWKLbqR/HJXAw8GpcT6xpL5gPBUQWqaXe0+cm7xnRNGjada9ndgv PJbrTN0u6E3hwZpPZoDs7ECJ82KvaBdvmoZZQ97ALHGaUzBn6at8JNfMXgQ65UAjPP 794SmnNgI5t6LlLF3yVxuYF7szhHUCRWHJtcItjsTNG4UZMCDlndNv8omyDg5ZKxrw N2uKoPBvNXXGg== Received: from OC11EXPO29.exchange.mit.edu (18.9.4.102) by w92exedge3.exchange.mit.edu (18.7.73.15) with Microsoft SMTP Server (TLS) id 15.0.1497.36; Sun, 22 May 2022 19:02:53 -0400 Received: from OC11EXPO29.exchange.mit.edu (18.9.4.102) by oc11expo29.exchange.mit.edu (18.9.4.102) with Microsoft SMTP Server (TLS) id 15.0.1497.23; Sun, 22 May 2022 19:03:39 -0400 Received: from OC11EXPO29.exchange.mit.edu ([18.9.4.102]) by oc11expo29.exchange.mit.edu ([18.9.4.102]) with mapi id 15.00.1497.023; Sun, 22 May 2022 19:03:39 -0400 From: John F Carr To: "tuexen@freebsd.org" CC: "freebsd-arm@freebsd.org" Subject: Re: clang14 issue triggering PR264094? Thread-Topic: clang14 issue triggering PR264094? Thread-Index: AQHYbhxeR1Lj4IHeV02A3Hxv+Z3JG60rqmgAgAAcxoA= Date: Sun, 22 May 2022 23:03:39 +0000 Message-ID: References: <41AF8299-1B05-487A-AE34-11BCA460C3B1@freebsd.org> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [108.7.221.50] Content-Type: text/plain; charset="us-ascii" Content-ID: <2B76BC51C6F9634DB4FA48FFB34264FD@exchange.mit.edu> Content-Transfer-Encoding: quoted-printable List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org MIME-Version: 1.0 X-Rspamd-Queue-Id: 4L5wxy2wQDz4mK2 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=mit.edu header.s=outgoing header.b="gJdPSlJ/"; dmarc=pass (policy=none) header.from=mit.edu; spf=pass (mx1.freebsd.org: domain of jfc@mit.edu designates 18.9.28.15 as permitted sender) smtp.mailfrom=jfc@mit.edu X-Spamd-Result: default: False [-2.73 / 15.00]; HAS_XOIP(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:18.9.28.0/24]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[18.9.28.15:from]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[mit.edu,none]; DKIM_TRACE(0.00)[mit.edu:+]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:3, ipnet:18.9.0.0/16, country:US]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; RCVD_COUNT_FIVE(0.00)[5]; R_DKIM_ALLOW(-0.20)[mit.edu:s=outgoing]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; NEURAL_SPAM_SHORT(0.97)[0.974]; DWL_DNSWL_LOW(-1.00)[mit.edu:dkim]; RCVD_IN_DNSWL_NONE(0.00)[18.9.4.102:received]; TO_DN_EQ_ADDR_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-arm] X-ThisMailContainsUnwantedMimeParts: N Status: O Content-Length: 5882 Lines: 151 > On May 22, 2022, at 17:20 , John F Carr wrote: >=20 > On May 22, 2022, at 16:41 , tuexen@freebsd.org wrote: >>=20 >> Dear all, >>=20 >> I'm trying to analyze https://bugs.freebsd.org/bugzilla/show_bug.cgi?id= =3D264094 >>=20 >> The relevant file is: >> https://cgit.freebsd.org/src/tree/sys/netinet/cc/cc_htcp.c >>=20 >> It is interesting that the panic happens on arm64, but not amd64. It doe= s >> happen when using clang14 (most recent version in the main tree), it doe= s >> not happen when using clang13. >> I also does not happen using clang14 when forcing htcp_recalc_beta() not >> to be inlined. >>=20 >> The panic happens when accessing V_htcp_adaptive_backoff in >> https://cgit.freebsd.org/src/tree/sys/netinet/cc/cc_htcp.c#n471 >>=20 >> Since this looks strange to me, I disassembled htcp_recalc_beta() when >> using clang14 and the function not being inlined. This is the relevant >> code: >>=20 >> (kgdb) disassemble htcp_recalc_beta >> Dump of assembler code for function htcp_recalc_beta: >> 0x00000000000113cc <+0>: stp x29, x30, [sp, #-16]! >> 0x00000000000113d0 <+4>: mov x29, sp >> 0x00000000000113d4 <+8>: ldr x8, [x0] ; x8 =3D ccv >> 0x00000000000113d8 <+12>: ldr x9, [x18] ; x9 =3D curthread >> 0x00000000000113dc <+16>: adrp x10, 0x21000 ; x10 =3D ??? >> 0x00000000000113e0 <+20>: ldr x9, [x9, #1368] ; x9 =3D curthread->td_= vnet >> 0x00000000000113e4 <+24>: ldr x10, [x10, #2168] ; x10 =3D ??? >> 0x00000000000113e8 <+28>: ldr x9, [x9, #40] ; x9 =3D curthread->td_= vnet->vnet_data_base >> 0x00000000000113ec <+32>: ldr w9, [x9, x10] ; w9 =3D V_htcp_adaptiv= e_backoff ??? >> 0x00000000000113f0 <+36>: cbz w9, 0x11428 >>=20 >> I don't understand the computations in relation to x10, which is the off= set used to get the relevant variable. >>=20 >> However, this code works. >>=20 >> Looking at the code generated by clang13 when htcp_recalc_beta() is inli= ned, one gets: >>=20 >> 0xffff000150610f28 <+212>: ldr x10, [x0] ; x10 =3D ccv >> 0xffff000150610f2c <+216>: ldr x11, [x18] ; x11 =3D curth= read >> 0xffff000150610f30 <+220>: ldr x11, [x11, #1368] ; x11 =3D curth= read->td_vnet >> 0xffff000150610f34 <+224>: ldr x12, [x11, #40] ; x12 =3D curth= read->td_vnet->vnet_data_base >> 0xffff000150610f38 <+228>: adrp x11, 0xffff000150621000 ; ??? >> 0xffff000150610f3c <+232>: ldr x11, [x11, #2256] ; ??? >> 0xffff000150610f40 <+236>: ldr w12, [x12, x11] >> 0xffff000150610f44 <+240>: cbz w12, 0xffff000150610f7c >>=20 >> It looks similar and it does work. >>=20 >> Now comes the inlined code from clang14: >>=20 >> 0xffff0001016acf28 <+212>: ldr x10, [x0] ; x10 =3D ccv >> 0xffff0001016acf2c <+216>: ldr x11, [x18] ; x11 =3D curthread >> 0xffff0001016acf30 <+220>: ldr x12, [x11, #1368] ; x12 =3D curthread->t= d_vnet >> 0xffff0001016acf34 <+224>: nop >> 0xffff0001016acf38 <+228>: adr x11, 0xffff0001016bd520 >> 0xffff0001016acf3c <+232>: ldr x12, [x12, #40] ; x12 =3D curthread->t= d_vnet->vnet_data_base >> =3D=3D>0xffff0001016acf40 <+236>: ldr w12, [x12, x11] >> 0xffff0001016acf44 <+240>: cbz w12, 0xffff0001016acf7c >>=20 >> The line marked with =3D=3D> is the line where the panic happens. It loo= ks that the offset computation is different. >>=20 >> Is this an issue with clang14? Any idea what is going wrong? >>=20 >> Thanks for any help! >>=20 >> Best regards >> Michael >>=20 >>=20 >=20 > That nop next to the adr instruction makes me think a 32 bit relocation w= ent wrong. These relocations normally consume two instructions but the lin= ker can patch one into a nop if it is not needed. Usually you have a pair = of instructions adrp+adr or, as in the clang13 example, adrp+ld or adrp+st.= The adrp computes a page-aligned address within a 32 bit offset of the P= C and the next instruction has low 12 bits of the address. The problem cou= ld be in the compiler or the linker. What does the assembly or disassemble= d .o look like before it gets linked? >=20 >=20 I have an arm64 running CURRENT so I was able to reproduce the problem. The assembly code is (cc -S) ldr x10, [x0] ldr x11, [x18] ldr x12, [x11, #1368] adrp x11, :got:vnet_entry_htcp_adaptive_backoff ldr x11, [x11, :got_lo12:vnet_entry_htcp_adaptive_backoff] ldr x12, [x12, #40] ldr w12, [x12, x11] cc_htcp.o disassembles to (objdump --disassemble --reloc) 1f4: f940000a ldr x10, [x0] 1f8: f940024b ldr x11, [x18] 1fc: f942ad6c ldr x12, [x11, #1368] 200: 9000000b adrp x11, 0 200: R_AARCH64_ADR_GOT_PAGE vnet_entry_htcp_adaptive_backoff 204: f940016b ldr x11, [x11] 204: R_AARCH64_LD64_GOT_LO12_NC vnet_entry_htcp_adaptive_backoff 208: f940158c ldr x12, [x12, #40] 20c: b86b698c ldr w12, [x12, x11] cc_htcp.ko disassembles to 10f44: f940000a ldr x10, [x0] 10f48: f940024b ldr x11, [x18] 10f4c: f942ad6c ldr x12, [x11, #1368] 10f50: d503201f nop 10f54: 10082f6b adr x11, 21540 10f58: f940158c ldr x12, [x12, #40] 10f5c: b86b698c ldr w12, [x12, x11] An adrp+ld pair, with paired relocations, is being transformed into a nop+a= dr pair. So it computes the address of the variable instead of loading its= value. I am not familiar with ARM relocation codes and can not say if th= e R_AARCH64_ADR_GOT_PAGE+ R_AARCH64_LD64_GOT_LO12_NC combination is correct= here, or whether the :got: and :got_lo12: prefixes in the assembly are cor= rect. I can say that I have seen this sort of behavior due to a linker bug= before on other systems, when the linker assumes a relocation type is only= used with a certain opcode.=