From owner-freebsd-ppc@freebsd.org Wed Jun 5 08:35:41 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B1AF515C991D for ; Wed, 5 Jun 2019 08:35:41 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic309-20.consmr.mail.ne1.yahoo.com (sonic309-20.consmr.mail.ne1.yahoo.com [66.163.184.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9207971895 for ; Wed, 5 Jun 2019 08:35:39 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: OdrJJMUVM1kipM7xllMKZLb93WACgBY5bBQmXA52b85yM.eIaQ9RSwi6_CwH609 kZPu3Ffu3iKNKQxQ9CG900VMO3VjmP1Q7ymmPVlJKeitahi3CIPnL0v08aKOB8EGZaxK09Budr5z T3BxhuN_n19pJCVau241zp2BHSZ4b9qdrNlvCBYjrLt7RnymBiD2Lw0xajDnP3jBlAdffX.VSI1A .0LFMsLgXa2hMdUbSyxzd4bjRVHSZkqIrji.F36NJcQs5Z7LRzDvrjFvDOwlJXrgAfddzqRhNGuN RWqPJfZR43sS4wdCmZCSKyJVTehGew.pEr0CaMBELVT26AMrqADKaawfvHaAQxOJnq2noq2s86Vo 2yK.m7yyagM4cGrzyJn8OxJQ4zKrEeoa7wu5hl.mvCD4qnzJLEeD0xdLeaLbej6xIgB5DW.7XHqw 3g9sAKsqVUSzRECQMPZbRRsqzk6AcsbCBeVHlOdtGxwJXhZjAxV.lObQbotpTAz1ofCczF8E5r5O zZWeHXXFn78P6WwdNRbmup6OHkfxlLIpbby.UXYVEfAmQj7m64XvuuJM_FHG1IqsW.SwKWFdf2h7 16HDyjJX9eDfXkqwvAbEPIuxPiwBWL0zhGkJFkJQjmH5348UXaeu4ADJbb3I3rxAWvpBA9nrivWS 2c0G80QnTuioszUnWTdkq2ZLjgsz5WrmQl1lZmv.BO8wzEir6I_gO7gzhUxrkFwQ5J_bBL_8vcyU rd3G.PORTbIxob9HN0MoPh.Gs578O54VleSQKTTW6yMO8XlKM2DU8jt6KBlOe65YjIlijFADgBbg 0fZdecSjpL0K6IU8JIkvnOFS7vG49dDqiFPxCEGblgoWB7dEpuQvPTG0LsL5VIM9_oZPd55QtT6S uMymARx9GDTrKzvGc9fTwAF3ZafPpstwAmNlYaCCw4J63ay6HNK6tlOuhr3SbeSfgcpIl_BcoY8V gzfmpJh1Z4Zm1Wkxo.dCof2GZlWn3gu8.9AD3JFuuurU7Sh81v5HqJ3U.PQWJTljHzHGj3gfpGhH 1Nsq6zVpaN7Fi1TT3Qipvm0nHn_4cMxCnGVPO6sh.sMoh7M.0DVOV.7HMhMrdPSn7MMOZSYjBfAe G1HnkQIM6mA5hj3mwiy8UxWtFLjqdbPgUwZzfHngFO2nBaWHJsLDz2jUcKdrRCX_iqFINOLHLA2y 2IQgRsJ9HQiZnuMmP7K3KscjXIHMObKbGBAS.5ejeBixgMIh8 Received: from sonic.gate.mail.ne1.yahoo.com by sonic309.consmr.mail.ne1.yahoo.com with HTTP; Wed, 5 Jun 2019 08:35:32 +0000 Received: from c-67-170-167-181.hsd1.or.comcast.net (EHLO [192.168.1.115]) ([67.170.167.181]) by smtp401.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 26e40e16668a539b998751e530f633ae for ; Wed, 05 Jun 2019 08:35:30 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) Subject: Re: crash of 32-bit powerpc -r347549 kernel built via system-clang-8 (crash is while trying to mount the root file system) Date: Wed, 5 Jun 2019 01:35:28 -0700 References: <45D010BF-7654-43A6-8FF4-CCDEEF4004F6@yahoo.com> <4354EA25-69C2-4CAB-8273-62457333BD30@yahoo.com> To: FreeBSD PowerPC ML In-Reply-To: <4354EA25-69C2-4CAB-8273-62457333BD30@yahoo.com> Message-Id: <995DA649-9390-420B-AC95-FFD17079CDA9@yahoo.com> X-Mailer: Apple Mail (2.3445.104.11) X-Rspamd-Queue-Id: 9207971895 X-Spamd-Bar: + X-Spamd-Result: default: False [1.93 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-0.09)[-0.092,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36646, ipnet:66.163.184.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-ppc@freebsd.org]; NEURAL_SPAM_MEDIUM(0.26)[0.258,0]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(1.47)[ip: (4.95), ipnet: 66.163.184.0/21(1.36), asn: 36646(1.09), country: US(-0.06)]; NEURAL_SPAM_LONG(0.81)[0.806,0]; RCVD_IN_DNSWL_NONE(0.00)[146.184.163.66.list.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jun 2019 08:35:42 -0000 On 2019-Jun-3, at 19:40, Mark Millard wrote: > On 2019-Jun-3, at 17:24, Mark Millard wrote: >=20 >> I tried (cross) building a 32-bit powerpc kernel and world = (non-debug)=20 >> with system-clang (on amd64) and use of devel/powerpc64-binutils . = The >> installed kernel panics trying to mount the root file system. >>=20 >> FYI: Typed from picture of screen . . . >>=20 >> Trying to mount root from ufs:/dev/ufs/FBSDG4Srootfs [rw,noatime]... >> panic: getnewbuf_empty: Locked buf 0xd2800000 on free queue. >> . . . >> 0xd6919080: at kdb_backtrace+0x64 >> 0xd69190e0: at vpanic+0x200 >> 0xd6919150: at panic+0x50 >> 0xd6919190: at getnewbuf+0x594 >> 0xd69191f0: at getblkx+0x540 >> 0xd69192a0: at breadn_flags+0x90 >> 0xd69192f0: at ffs_use_bread+0x9c >> 0xd6919330: at readsuper+0x68 >> 0xd6919370: at ffs_sbget+0xcc >> 0xd69193c0: at ffs_mount+0x18b8 >> 0xd69194f0: at vfs_domount+0xa74 >> 0xd69196a0: at vfs_donmount+0x944 >> 0xd6919700: at kernel_mount+0x64 >> 0xd6919740: at parse_mount+0x52c >> 0xd6919840: at vfs_mountroot+0x71c >> 0xd69199b0: at start_init+0x44 >> 0xd6919a10: at fork_exit_0xcc >> 0xd6919a40: at fork_trampoline+0xc >> KDB: enter panic >> [ thread pid 1 tid 100002 ] >> Stopped at kdb_enter+0x74: addi r3,r0,0x0 >>=20 >> This reproduces with each boot attempt. >>=20 >> Replacing the kernel with one built via gcc 4.2.1 and booting >> the result does not panic. >>=20 >>=20 >> FYI for the context of the panic call: >>=20 >> /usr/src/sys/kern/vfs_bio.c : >>=20 >> static struct buf * >> buf_alloc(struct bufdomain *bd) >> { >> struct buf *bp; >> int freebufs; >>=20 >> /* >> * We can only run out of bufs in the buf zone if the average = buf >> * is less than BKVASIZE. In this case the actual wait/block = will >> * come from buf_reycle() failing to flush one of these small = bufs. >> */ >> bp =3D NULL; >> freebufs =3D atomic_fetchadd_int(&bd->bd_freebuffers, -1); >> if (freebufs > 0) >> bp =3D uma_zalloc(buf_zone, M_NOWAIT); >> if (bp =3D=3D NULL) { >> atomic_add_int(&bd->bd_freebuffers, 1); >> bufspace_daemon_wakeup(bd); >> counter_u64_add(numbufallocfails, 1); >> return (NULL); >> } >> /* >> * Wake-up the bufspace daemon on transition below threshold. >> */ >> if (freebufs =3D=3D bd->bd_lofreebuffers) >> bufspace_daemon_wakeup(bd); >>=20 >> if (BUF_LOCK(bp, LK_EXCLUSIVE | LK_NOWAIT, NULL) !=3D 0) >> panic("getnewbuf_empty: Locked buf %p on free queue.", = bp); >=20 >=20 > I tried making a debug kernel build via system-clang-8. It > reports differently but still during getnewbuf being active > on the stack (again typed from a picture): >=20 > Trying to mount root from ufs:/dev/ufs/FBSDG4Srootfs [rw,noatime]... > . . . (ignore witness/diagnostic warnings) . . . > panic: bq_remove: Locked buf 0xd2a00000 not on a queue. > . . . > 0xd6b7bfd0: at kdb_backtrace+0x64 > 0xd6b7c030: at vpanic+0x200 > 0xd6b7c0a0: at panic+0x50 > 0xd6b7c0e0: at bq_remove+01e0 > 0xd6b7c100: at buf_import+0x8c > 0xd6b7c130: at uma_zalloc_arg+0x544 > 0xd6b7c190: at getnewbuf+0x380 > 0xd6b7c1f0: at getblkx+0x620 > 0xd6b7c290: at breadn_flags+0x90 > 0xd6b7c2e0: at ffs_use_bread+0xa8 > 0xd6b7c320: at readsuper+0x68 > 0xd6b7c360: at ffs_sbget+0xcc > 0xd6b7c3b0: at ffs_mount+0xefc > 0xd6b7c4e0: at vfs_domount+0xa754 > 0xd6b7c690: at vfs_donmount+0x78c > 0xd6b7c6f0: at kernel_mount+0x7c > 0xd6b7c730: at parse_mount+0x52c > 0xd6b7c830: at vfs_mountroot+0x660 > 0xd6b7c9a0: at start_init+0x4c > 0xd6b7ca10: at fork_exit_0xb0 > 0xd6b7ca40: at fork_trampoline+0xc >=20 > /usr/src/sys/kern/vfs_bio.c : >=20 > static void > bq_remove(struct bufqueue *bq, struct buf *bp) > { >=20 > CTR3(KTR_BUF, "bq_remove(%p) vp %p flags %X", > bp, bp->b_vp, bp->b_flags); > KASSERT(bp->b_qindex !=3D QUEUE_NONE, > ("bq_remove: buffer %p not on a queue.", bp)); > . . . >=20 > For reference: >=20 > static int > buf_import(void *arg, void **store, int cnt, int domain, int flags) > { > struct buf *bp; > int i; >=20 > BQ_LOCK(&bqempty); > for (i =3D 0; i < cnt; i++) { > bp =3D TAILQ_FIRST(&bqempty.bq_queue); > if (bp =3D=3D NULL) > break; > bq_remove(&bqempty, bp); > store[i] =3D bp; > } > BQ_UNLOCK(&bqempty); >=20 > return (i); > } >=20 >=20 I tried building the debug kernel with KTR for KTR_BUF. Installing and booting the result did not panic. Manually forcing getting to ddb> soon enough and doing "show ktr" did show a bq_remove for 0xd2a00000 (and later activity). =46rom the looks of the KTR_BUF CTRn's, this suggests to me that the access to bp->qindex in bq_remove is racy in some way vs. updates to the value. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)