From owner-freebsd-arm@freebsd.org Sat Dec 19 06:27:40 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 163AE4BD107 for ; Sat, 19 Dec 2020 06:27:40 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic301-20.consmr.mail.gq1.yahoo.com (sonic301-20.consmr.mail.gq1.yahoo.com [98.137.64.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CybPl1BTZz4mFv for ; Sat, 19 Dec 2020 06:27:38 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1608359257; bh=TrEoNJIGoArvvKCVW9hElhNqNnLpwviq0OquqNJ5awl=; h=From:Subject:Date:To:From:Subject; b=dmnMvwi93p70aZyGVmQjD/PEOZJN8X/r+jrrW8yPeqQXucMVtFXnMHTXlcgUCiGJtfxajfrb8rvD9KC4NL7BFv0Cn0jxWpmwmIkHXIAH0pefZo3U+yakkt4UEM0WT2UN2eH1VwxehRqe7fHrpP64wykB5B84bU4VxD5xI7jWyEN3OketG7/rjqyT74I4yfNEMuvHNW7aBMVgLT9Rllzh9XJl3S5a1tMa3OsKw1miwEk2jrNvkUkBTSBR+gLqA5wakUE3GYmbNkejHTEti5Lq4IKfSDKdNw7ueE5eualGmUndKHEZVw/oIJuc575nmO8yK84wdcKPy1ZcLogahjyCVw== X-YMail-OSG: WL5pAc8VM1mVY7nE9U3KvFVqc964rgTYOVjqH0IZSuS.PHxhD1EsrDaEx_JBKTy VBxijpqZBKg_vfKtMQ2GlhjyxpOAzmmOZQKGLgKRjolzhaS9qVFrWxmtAk8rb2c4BXTM.4FaF8mL DBqg5O8YQRGX3lhxVfkUx_COTeXiwajCdubgIwCFKRxtcBJ41S916mtG_3Yd6nvpzsVFazA5yJMl ZcF.Yj__e5VBEBeGLIgtisRNxSyvWCXH0i40AW2oDrn1_qkC.NvJsBa2HNknBVLeLhiAaibESSAu a0zP8DIpccp2YQDs1KEBFBTcg_c4qfFjB7nRn8aY..4oYwGgEG_ueueuZe_uFKgtSNfAXBOGWxrO 2f4ajIe2eWjY6YlmMaWThuz1YgO9Kr5iJdWXWVLzvCDhHGmeZ77EHEjb5T2ctBS8l01vM2hhYOm3 l_n1PQtNs.Eeo.ItaSApeuboHVX8uWiacjEK2FSwqQluQXhPst4D7hmYRKq_mocZgHTIelzLGo7Y l7XmK._myRJbxK9LnAeBnUpINTnHF89Jn.xtyoxHtz50mtjm3iHPOTvObCwUlEqqxEKKJHgF3tKR SOA1f6GAw8VXTErfLR_xMNtV2ryXmf2.rcoG0a_hOiqK1OmtGIxfEnw1eMM1z7H.Im0gbAiqTmkD xl5Yo.MuYXVPYmIkaOf8sy_GuX1Xlx4V5Yu5.YH831V6G1G7xxBOKl9q8aHBbYyvfGEnCXlfKFQg OWU9TxFuhYhjx3mbJvsRrRXSaXNvZCs0FcQbUuU8ZxvroXxB16L1nkF4jGqx79I9YiWRy3qGbWbc pa9pxokEjNErVH_IT5oW2ma3gXa3W6ynhrFnNvrnD_2ZM3LPunBTwsp72Gw89pI69urS5CBIyn4f kRDPlyzzYJAnw4RrKRjNSX9PlSu6AnC9aIMDZjpVkz5l5mt9lrK13lNKtuBd__nzS7y5DvAJCAHo nLaX3yOsIvBMaEyz3pZJLpkzNd.gtH.SkTZp2E_96MF8v0S.LJK3gZ6CHz8VA48_I6ZQtPxnd2A1 o1829nILg4WUP.8r9uXfNOl_l1kBOu.t5dnyRCEL91ZO8E7tfWnJs.fhwHXvaXoBOL.Jt9ZzJ3ql Cq6acCD2467b8JOsLV1fjzmPOdCza.sN37TBXqRVRG3gGMqrXWPXn7mEHXQIixvM.RerC87fbMGW UXE7t0Ve7nInhqkKvWxirkVejEhS8FC3GeeqqTmmeTVFbHx3rBiNgWH8tiICWHMxFKguhOvoXiFB O_6T3m3cbzc6DtI_IQn2NyAEIjRM6iY1WZk_qZc9085zcTue3SghgE2SpqmwVg5U8JGer1td8Q8V mwgZ4oja9p7znvSksDdDI5NhBnCGHyEW2gTGTpLIi41wA7YybWFe7D0PMAtK42hqZ67vTr6t8E52 AQMFmCRAs7rO3ULE5zHOEbALhpUgXFVxoym8XCTHR57uyUbWfsOlHCvIePkjAeLQ.pur9IOelNIE tE_vI_cAbZ9rn6lwVmUV.C3K777KjhWc0Tg7dHucxv_h5RLBxzmXA.HbmionSe5KcqhOkMXxL3J5 BVnfVWFX8JKUpbyiicHy9xXAYByE.WUnE3CVUWD8lfcAWDlbooC2F0dnhvUFAQVLp0EYoY9XKK5M vHi5DW4DRxbvpDSJJyhPYNE7m1W5_4ShYONYgnEwQOqcm4M9OsPa3QQAS6tYEVLVxw_0gHCHqpIP ewlUWGEXVDRQ.Otvjn5IAwc9zHWVVHNKYnN8jN3md87GYLqwb5EoTZGiTiSBJM4lqWIAnDU73IpU 3qy2ClglaPJCg2jnO5jBYsoCnXpsiXxHp3LXgHw6Lm7AF70_YwJdS_wSiAfsyHRJxhafX4QKKR1d DT2DPdaiFPS_MPlyEgyaKHEZ5akzHIVlK9fLReD0_luMdxn8eIe_NIu4VQNHU_S7q8Z6d4vb23wT 1LT.kj0dFc_uOsyZqkzgPwzwwa.bnRB80EkhayYrlo82CEtEZ9PKhOhszNMQx.YKKTqgd8RK71Ay tgmn8GvgrIu7xv63S2tP3a4yPGRGrpaDmb3LSlZ08e6r4ywgRM584NIGF1yb99mgApqZHdb0zRQg OAanKo0tPCc9dLXkm_0PAsRe1mGU3kzt.y2cK.kqJ790ev2yBfdcgdf1cVlHSRgJrla9sJvbVTl8 dls8JIVjmD8jL3Kk0NA0I1a4tHTgb5.rfenzQqAraRa27.K8tfwDg1ECLg5FaUYxx9ettpJCGHlw C2VXQrcVGPM1heOR8odjGN8PBztu3cEzsaB7sPnVoeT83WVvq9KyxCb8hF0m6DykUq.rI0FUnE3f uYEW83JufrRjZAEqvLf5urH1.cyv4sd1wEHoxO0j7SNb3Z5DpEo_0Zuo4XnTp2L9.Iuwcpyr1evZ eFWO1rj.VnlePb9nzrjMnQLm0fRQS8BIBp5Po5G3RW96PZDWTlWvFcHKYWrtnkojWFROL0QjZF0K bI8E54dFumStLzrVjSU6eda8b..BOY1.TwJGO9Bd4.vPoGZL8oaYSxeDmuNUFfdkOXNshT8.86hC 8B6yA62OnXgNwzMpXnn__gzxupGjenB61I.vyn1vMpmMFd5b8bsa5x0BYUfud638Yvz_MTIin2Vf H0dK1fCE2SjeXCgm8nzYArRvc5uV.ELCs_N4KNkgQMVl5anSxU4PDtQXkNfhHESBlBHJKwUE_PgQ vOa66Gms6z.7hID_UgPo6Jw-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic301.consmr.mail.gq1.yahoo.com with HTTP; Sat, 19 Dec 2020 06:27:37 +0000 Received: by smtp411.mail.ne1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 26315aee6695c2f74e6055b656b85d35; Sat, 19 Dec 2020 06:27:31 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.20.0.2.21\)) Subject: debug head -r368500 kernel (for example) : "lock order reversal: (sleepable after non-sleepable)" involving ure0 and a sysctl lock; more Message-Id: Date: Fri, 18 Dec 2020 22:27:28 -0800 To: freebsd-arm , FreeBSD Current X-Mailer: Apple Mail (2.3654.20.0.2.21) References: X-Rspamd-Queue-Id: 4CybPl1BTZz4mFv X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RBL_DBL_DONT_QUERY_IPS(0.00)[98.137.64.146:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[98.137.64.146:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.146:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.146:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-arm] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Dec 2020 06:27:40 -0000 The following is from head -r368500's artifact kernel from: = https://artifact.ci.freebsd.org/snapshot/13.0-CURRENT/r368500/arm64/aarch6= 4/kernel.txz but the same sort of material showed for -r368000 . (I was attempting a bisect for a different issue but the debug kernels did not fail for what I was looking for and all the debug versions that I tried reported similarly to the below.) Note also the mixing in of "uma_zalloc_debug" activity after the initial LOR backtrace, ure0 still involved. Autoloading module: if_ure.ko ure0 on uhub0 ure0: = on usbus0 add host 127.0.0.1: gatelock order reversal: (sleepable after = non-sleepable) 1st 0xffffa00002b2cea0 ure0 (ure0, sleep mutex) @ = /usr/src/sys/dev/usb/usb_request.c:714 2nd 0xffff000000dd6858 sysctl lock (sysctl lock, sleepable rm) @ /usway = lo0 fib 0: route alrr/src/sys/kern/kern_sysctl.c:836 lock order ure0 -> sysctl lock attempted at: #0 0xffff00000056d068 at witness_checkorder+0xc54 #1 0xffff0000004f8f08 at _rm_wlock_debug+0x88 #2 0xffff00000050ee2c at sysctl_add_oid+0x60 #3 0xffff00009e415780 at ure_attach_post+0x1a78 #4 0xffff000000391d6c at ue_attach_post_task+0x3c #5 0xffff0000003826e0 at usb_process+0x10c #6 0xffff0000004baa1c at fork_exit+0x7c #7 0xffff000000816544 at fork_trampoline+0x10 uma_zalloc_debug: zone "malloc-128eady in table " with the following non-sleepable locks held: exclusive sleep mutex ure0 (ure0) r =3D 0 (0xffffa00002b2cea0) locked @ = /usr/src/sys/dev/usb/usb_request.c:714 stack backtrace: #0 0xffff00000056d388 at witness_debugger+0x64 #1 0xffff00000056e518 at witness_warn+0x3ec #2 0xffff000000778f9c at uma_zalloc_debug+0x2c #3 0xffff000000778998 at uma_zalloc_arg+0x2c #4 0xffff0000004d534c at malloc+0xa0 #5 0xffff00000050ee80 at sysctl_add_oid+0xb4 #6 0xffff00009e415780 at ure_attach_post+0x1a78 #7 0xffff000000391d6c at ue_attach_post_task+0x3c #8 0xffff0000003826e0 at usb_process+0x10c #9 0xffff0000004baa1c at fork_exit+0x7c #10 0xffff000000816544 at fork_trampoline+0x10 uma_zalloc_debug: zone "malloc-16" with the following non-sleepable = locks held: exclusive sleep mutex ure0 (ure0) r =3D 0 (0xffffa00002b2cea0) locked @ = /usr/src/sys/dev/usb/usb_request.c:714 stack backtrace: #0 0xffff00000056d388 at witness_debugger+0x64 #1 0xffff00000056e518 at witness_warn+0x3ec #2 0xffff000000778f9c at uma_zalloc_debug+0x2c #3 0xffff000000778998 at uma_zalloc_arg+0x2c #4 0xffff0000004d534c at malloc+0xa0 #5 0xffff0000005f8c80 at strdup+0x2c #6 0xffff00000050eeb8 at sysctl_add_oid+0xec #7 0xffff00009e415780 at ure_attach_post+0x1a78 #8 0xffff000000391d6c at ue_attach_post_task+0x3c #9 0xffff0000003826e0 at usb_process+0x10c #10 0xffff0000004baa1c at fork_exit+0x7c #11 0xffff000000816544 at fork_trampoline+0x10 uma_zalloc_debug: zone "malloc-64" with the following non-sleepable = locks held: exclusive sleep mutex ure0 (ure0) r =3D 0 (0xffffa00002b2cea0) locked @ = /usr/src/sys/dev/usb/usb_request.c:714 stack backtrace: #0 0xffff00000056d388 at witness_debugger+0x64 #1 0xffff00000056e518 at witness_warn+0x3ec #2 0xffff000000778f9c at uma_zalloc_debug+0x2c #3 0xffff000000778998 at uma_zalloc_arg+0x2c #4 0xffff0000004d534c at malloc+0xa0 #5 0xffff0000005f8c80 at strdup+0x2c #6 0xffff00000050eee4 at sysctl_add_oid+0x118 #7 0xffff00009e415780 at ure_attach_post+0x1a78 #8 0xffff000000391d6c at ue_attach_post_task+0x3c #9 0xffff0000003826e0 at usb_process+0x10c #10 0xffff0000004baa1c at fork_exit+0x7c #11 0xffff000000816544 at fork_trampoline+0x10 uma_zalloc_debug: zone "malloc-32" with the following non-sleepable = locks held: exclusive sleep mutex ure0 (ure0) r =3D 0 (0xffffa00002b2cea0) locked @ = /usr/src/sys/dev/usb/usb_request.c:714 stack backtrace: #0 0xffff00000056d388 at witness_debugger+0x64 #1 0xffff00000056e518 at witness_warn+0x3ec #2 0xffff000000778f9c at uma_zalloc_debug+0x2c #3 0xffff000000778998 at uma_zalloc_arg+0x2c #4 0xffff0000004d534c at malloc+0xa0 #5 0xffff00000050ef3c at sysctl_add_oid+0x170 #6 0xffff00009e415780 at ure_attach_post+0x1a78 #7 0xffff000000391d6c at ue_attach_post_task+0x3c #8 0xffff0000003826e0 at usb_process+0x10c #9 0xffff0000004baa1c at fork_exit+0x7c #10 0xffff000000816544 at fork_trampoline+0x10 miibus0: on ure0 rgephy0: PHY 0 on miibus0 add hrgephy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, = 1000baseT-FDX, 1000baseT-FDX-master, auto 0 fib 0: route already iue0: on ure0 The context here is an RPi4 (aarch64 cortex-a72) with: # uname -apKU FreeBSD RPi4B 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r368500: Thu Dec 10 = 07:52:39 UTC 2020 = root@FreeBSD-head-aarch64-build.jail.ci.FreeBSD.org:/usr/obj/usr/src/arm64= .aarch64/sys/GENERIC arm64 aarch64 1300131 1300131 The boot attempts were via uefi/ACPI using https://github.com/pftf/RPi4 v1.21 materials, directly booting from the USB3 SSD, no microsd card involved. Some context in case it contributes something for the above (probably not) . . . The reason for the bisect was: such boot attempts fail to mount route with my non-debug head -r368500 kernel build. (Previously the RPi4 was back at head -r365932 or so.) But my non-debug builds use -mcpu=3Dcortex-a72 . (This combination has caught missing synchronization activity before.) In the failing case the following never shows up: da0 at umass-sim0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SPC-4 SCSI device da0: Serial Number REPLACED da0: 400.000MB/s transfers da0: 228936MB (468862128 512 byte sectors) da0: quirks=3D0x2 and the message: Root mount waiting for: usbus0 repeats indefinitely, unlike historically for my configuration. With the artifact debug kernel instead of the non-debug kernel, the RPi4 boots fine, other things held constant. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)