From owner-freebsd-arm@freebsd.org Sun Dec 6 21:31:03 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5FBEB4AC434 for ; Sun, 6 Dec 2020 21:31:03 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic315-55.consmr.mail.gq1.yahoo.com (sonic315-55.consmr.mail.gq1.yahoo.com [98.137.65.31]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Cq0461M3Fz4sW5 for ; Sun, 6 Dec 2020 21:31:01 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1607290259; bh=k8vUXjXzYjKP3m2FB2mCzEn+1UV2uK9AufMMOOvLbLl=; h=Subject:From:Date:To:From:Subject; b=Pefr1WyTA8Y0PYj5f4Fpx497J/xupqmQII4m1oxzRKTCm7+m8JfdnU6Z/A4xSg7OHcaG7fnNGTrKRKKXjPevIEH0bjEPb/GHjv1jJM7/cvn6kTsOAHk/FT1q05CvBPDh6z3a9EOaUNY8idTTWBvEfJxjp8xcTJbWOr5ZiDiYcXRhdO3ogD/L2Vt+LlEGI5t0pj26BU4J0zb0bJKopqWjCYCVRgLDeKyu3fiuICOplb7lHlFIzvK9jYcYPu+DGSjh74ggp+9Ik5vxbBxBr5R2vNbZ6IUsCFta9eJhA0WBjy5oYVOvejrAqN26l70nxV6sBTEyixOwZqwPnuxn2dn9XQ== X-YMail-OSG: XtaWqxcVM1l.VYaWzdANP0b9d3VCuC21AJsxM4VfKbm5gGajBYsiWd2iGJKPznK H7uB2HxKSBrz1zXW2.b.dsq5dobKUEF5QHuaB_sJYZhEA65TdH.3yzQ0nVavtEBC12.fGxLvELBI J1MeR4kfhMT_Lj8lDgpetu8i8ZHeTALwDD2ZhrwlD8z2GdixBTVVgPSwR_cx6QisqMVgeA.38Apy 3RUI8nzhyWzxdDAfin0n_GHQkSZrXskHPe3frJkdemJgH1tt1HRTlSXm_LtaN.E8fIoZktAIWtKc 3BM8.tVERLACdQchFP1A9J3JQLo9In4xEuUntsM3Xkm8len_kmfiICYdQEa63.miYk0eatgh62ak gsnXL2ziRqTJWvTyB60e3Wu_ksMgBRRxDswkZZxWIrWimnhZ7HeyaT_Cn_tZtlqje9crGcc1Lsu3 LhTFt10Rv4GNNpPKqaXNM9UvDc2Z.xBzOCAW21DtaXODxaZ_FY5EVtxhmjjYWz5_JVAVw5TRODHr jc.NKYc0VpGUrDanpxq92zkT_dTshG79fdDVLYLpN_p9KCK0cVcHL3ymeZKwvh8gZz7CEgB9f_aM A3Pd0Zo8YK2RQ3f1QERQFwA368Eb9wSsiUedxGVfKRT_JtdAicazDBq5imm5QJxkp4xSr_wTWk0j z0ZeaFpX.ocnJtbj8Ao.VxkQ7xggrWtRTby7SMs0tMWF9mXwEs2ZhBiRg7gCsAQ_MY9EmQwkvWNS bYPicSkkQlYgBvptI0nsRI72kl9fytd9kG16CvO9fflYhaEebIC_zSh9.aLQihB2ZbYXhRiyg821 j7i.atjjvS_44nyBdKOKqPqyg88AZGoomQuSSuAsCcvefAVurg3AKE2Z65XuTZUcryru1NsX9htm TNKfuY8KJiFMRBzr9rrxdF6xid4QG_o.g_YT.ZDvNglS8w62GtJG2EjHKqhIBx3BDcR8yOScWkX1 xkcfpqUNN6i2IM72htiCX1GfeoNbaTlMvDyFatVMX8_1x6zfeRV618RWsNHG6s_5mRSKvzueJWsw X4gkcwNxOnW2C6dTRANBGTwBYv__39p5yM0kd1m1PNybwYsaH35Voi.CTnuJBjlKYvOG0sbTrtIa 9ItiDpV4c6kWK3YBAvTlnRMaTxhFyVjvrPTIPCEE.DtXQHiwMh0z5V_DD0VZk1AWrWr94ROL16a6 FEIXXFuu.RJ4zqTmpyNgMgCJs7pftjt1bjumOpwl9Q4eSXPBqnpirRlfX9EESkRdr4GEkY7sp6S4 fUPNq.FMgzFXPO7g5Y86KVd0KbQB_Nu.4HsiSFZcnQ56PDIU9uEnspdB__Qahb9qowaZ0a1Vx1SA S.LZIKAE5_YpY_OP1kiYc4I8DGhinWPmnCDeCdZtG7CLC88xjzGOPE6SCFkJGMV5TvQXw1jsdmpv U6z4vAvd7hmTDEHyZC8GHJhK9ZMPC6GPUvHIqyxOo2r5AgkrvWkHR4HkXNIOl68ETGWfbIRlL.sY EoT6FvwC2UX5H7Tw6jvNuMfH6yUakZ2qx0igfPxI2FlYPsshETf97m0f9yIM2oJdWp97LdWPyAqa C06..2KTTlKWBBUOWBP3ylrSi7PuUG.OM3eoNlCfVKk0GUMRR8zaTzMMfZAWqj.rp0zP.DqIPFgI VjEFimjWip4Iilgs378NhRqYJA94GxowdoRLi9chUYHzrmeLp1KsuVVCHQCw0nalq59B4ySujdrh 8_lkIBsSPE1GPOqsWugAc7ft_jZSgMcEjtA0kqFws8XVi82bvMucYST0lueo.asQbg1vt0cClEJb T65k1ui92.41E8QxwD8O_C.i1XH4Ze6g.cVsitXhbp1vEc6JPiIV_zNooZiHd6JLGHh5MQbkDA92 lfWm_JSgRdMpCl7aI9bXStXUj5k3QM5mQQ9quEiVe_Z89W_c7lVH7unRLtp7a8iDFEpdkhQ8QiBc .hj4Ne5KeVgy9sP1NVNxi5I2ZyvjL13IL8j8ScMFQsmghSBATG6Jmx91m7xa_k8hYg0advKMAO4g cuQufzmLtfSxQnzETDwSdIvO9XxNovaBc0HtofZ6BPuyvtQwkxkRbjzzFhG3r.6OZlMg8xsXA9xk QCdMeh8PH7exBuf4b4Q6zCY.Rhy6Vn0n19Lz56WhtGmfDfGwsUCsiC0aqLJYe8DtqS_C04ASJqhB sJaQMa88UikpmOGlf36RoNjnKeb7Q97k3M1JSOmplj1Sep0zpqezfYZGz4iI1J_JzCVy03f8SbZz fk49nJQBj1y6pp9XhOopEPjNKfYOVF3sHGvy22CcUMWZ3tBVFdxBUo5YzPLyb0BSe0B2TEZOerj0 LK11.1x1pI6i.sDc7AJUxB_Ibpb0QqGnqJZXkYZe3gfWESyF_VXp3_L3qD.JcoEUhtujMn0vWkL7 TMp280vO58eQQQFVE8c4YpMBUu0NgYJTr6eFpN6i63LY06_DzV3eAocF4cFumNmL87betxIJfXJ_ y21Mayao- Received: from sonic.gate.mail.ne1.yahoo.com by sonic315.consmr.mail.gq1.yahoo.com with HTTP; Sun, 6 Dec 2020 21:30:59 +0000 Received: by smtp402.mail.bf1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 5e8e03e83d83ecdfa3ecbcd664479caa; Sun, 06 Dec 2020 21:30:54 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.20.0.2.21\)) Subject: Re: ThunderX Panic after r368370 From: Mark Millard In-Reply-To: <91654fc4-8734-d8a7-5309-0400f418438a@freebsd.org> Date: Sun, 6 Dec 2020 13:30:52 -0800 Cc: freebsd-arm Content-Transfer-Encoding: quoted-printable Message-Id: References: <1C3442ED-278E-45B8-9206-0DD24FCBC237@brickporch.com> <4331eee0-74a6-565c-3bec-0051415b2bc1@freebsd.org> <56F0E9EB-0B78-4B0B-830A-48F8AFC5ABE1@yahoo.com> <91654fc4-8734-d8a7-5309-0400f418438a@freebsd.org> To: mmel@freebsd.org X-Mailer: Apple Mail (2.3654.20.0.2.21) X-Rspamd-Queue-Id: 4Cq0461M3Fz4sW5 X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[98.137.65.31:from]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[98.137.65.31:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.31:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.31:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-arm] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Dec 2020 21:31:03 -0000 On 2020-Dec-6, at 03:51, Michal Meloun = wrote: > On 06.12.2020 10:47, Mark Millard wrote: >> On 2020-Dec-6, at 00:17, Michal Meloun = wrote: >>> On 06.12.2020 3:21, Marcel Flores wrote: >>>> Hi All, >>>> Looks like the ThunderX started panicking at boot after r368370: >>>> https://reviews.freebsd.org/rS368370 >>>> =46rom a verbose boot, it looks like it bails in gic0 redistributor = setup(?): >>>> gic0: CPU29 Re-Distributor woke up >>>> gic0: CPU24 enabled CPU interface via system registers >>>> gic0: CPU17 enabled CPU interface via system registers >>>> gic0: CPU29 enabled CPU interface via system registers >>>> done >>>> Full Verbose boot: >>>> https://gist.github.com/mesflores/f026122495c8494d041bce04d30b15bb >>>> I'm not really familiar with the details of the commit, but happy = to test >>>> anything if anyone has any ideas. >>>=20 >>>=20 >>> Hi Marcel >>> are you able to get crashdump and do backtrace? >>> = https://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug.html#= kerneldebug-obtain >>> and >>> = https://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug-gdb.h= tml >>> If not, I'll make some debug patch. >>>=20 >>> It's weird, even though GIC is potentially affected by my patch, in = this case the cpuid numbering was not changed. >> (I've no access to a ThunderX. I just looked for my own curiosity. >> Sorry if this is obvious and so is noise.) >> When I looked at the code it appeared to be the last "->" in >> the following that was dereferencing the nullptr value (via [x8] >> in assembler notation): >> static uint64_t >> its_cmd_prepare(struct its_cmd *cmd, struct its_cmd_desc *desc) >> { >> uint64_t target; >> uint8_t cmd_type; >> u_int size; >> cmd_type =3D desc->cmd_type; >> target =3D ITS_TARGET_NONE; >> switch (cmd_type) { >> case ITS_CMD_MOVI: /* Move interrupt ID to another = collection */ >> target =3D desc->cmd_desc_movi.col->col_target; >> . . . >> In other words: it appeared to me that the above = desc->cmd_desc_movi.col >> evaluated as 0 when used in what was reported. > This is very probably right analysis. But problem is that = cmd_desc_movi.col should not be NULL, is initialized in its_cmd_movi = from sc->sc_its_cols which should be allocated in gicv3_its_attach(). >=20 The following is unlikely to directly contribute to the specific problem's solution but documents an oddity that took my time while looking around related the problem. One (comment?) oddity I ran into looking around: /usr/src/sys/sys/cpuset.h:#define CPU_FFS(p) = BIT_FFS(CPU_SETSIZE, p) but in /usr/src/sys/sys/bitset.h : #define BIT_FFS(_s, p) BIT_FFS_AT((_s), (p), 0) and (comment wrong about start?): /* * Note that `start` and the returned value from BIT_FFS_AT are * 1-based bit indices. */ #define BIT_FFS_AT(_s, p, start) __extension__ ({ = \ . . . In other words, BIT_FFS (and CPU_FFS) provide BIT_FFS_AT with start=3D=3D0= but start is documented to be a 1-based bit index. So, looking into what happens with start=3D=3D0, showing BIT_FFS_AT: #define BIT_FFS_AT(_s, p, start) __extension__ ({ = \ __size_t __i; = \ long __mask; = \ int __bit; = \ = \ __mask =3D ~0UL << ((start) % _BITSET_BITS); = \ __bit =3D 0; = \ for (__i =3D __bitset_word((_s), (start)); = \ __i < __bitset_words((_s)); = \ __i++) { = \ if (((p)->__bits[__i] & __mask) !=3D 0) { = \ __bit =3D ffsl((p)->__bits[__i] & __mask); = \ __bit +=3D __i * _BITSET_BITS; = \ break; = \ } = \ __mask =3D ~0UL; = \ } = \ __bit; = \ }) It looks like this traces to use of: __mask =3D ~0UL << ((start) % _BITSET_BITS); = \ and to use of: #define __bitset_word(_s, n) = \ (__constexpr_cond(__bitset_words((_s)) =3D=3D 1) ? = \ 0 : ((n) / _BITSET_BITS)) So __mask=3D=3D~0UL and __bitset_word((_s), (start))=3D=3D0 . Then for __i=3D=3D0: ((p)->__bits[0] & __mask) !=3D 0 evaluates like ((p)->__bits[0] & ~0UL) !=3D 0 which in turn evaluates like (p)->__bits[0] !=3D 0. =46rom there __bit =3D ffsl((p)->__bits[0] & __mask) would involve (p)->__bits[0] & __mask evaluing like (p)->__bits[0] & ~0UL and that in turn evaluating like just (p)->__bits[0] . Presuming non-zero as a context, effectively for such a context: __bit =3D ffsl((p)->__bits[0]); __bit +=3D 0; which would seem to set __bit correctly. It looks to me like start is 0-based in BIT_FFS_AT, not 1-based. So I expect that the comment is wrong about start. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)