From owner-freebsd-current@freebsd.org Sat Dec 19 06:44:31 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7D4A64BE0C8 for ; Sat, 19 Dec 2020 06:44:31 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic303-23.consmr.mail.gq1.yahoo.com (sonic303-23.consmr.mail.gq1.yahoo.com [98.137.64.204]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CybnB26Rgz4nXX for ; Sat, 19 Dec 2020 06:44:30 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1608360268; bh=2OkcfCe70t0hZmN0n4HF8nS6jEf/Hq+yfyrM6fYFSPZ=; h=Subject:From:Date:To:From:Subject; b=Mts7YoFU6wBdlmcX9QOES2tgt0enuhPyUzM/T/Z+YfXiibs7VVRIa52kisUAwDAIo5o3rgFtB3nc88Xm9QeM6Jqg7SnniekZDUj14nIZPql02xolvTPnHcIF3QHDv4Y7FkYk7si92CIagGS6EuggxZ6dOGSEcgOKENnX8+6N9TNtPYbriaMzEIfie7E4+YSo8XH7W78xqgllR6OLDkTiKwWnVmyllqhLEwQ5wOJoHiwc1IrOw3g+4cAVpTZX8JFWe71jbN22r0hlmx1xvOFgnP4GnShu2RcBG5kgDBhHkC96eKeB5y/lbMxQpsGenqT73p+EjXysXKlHO7HSyY0YNg== X-YMail-OSG: UZ_2zC4VM1kda.kYcD8xFqygxfMPQUaw5PDXYoXEeBMH4N9S1pS69jW1jLNXqDM S7iIdvtNNka12ajtredDG68KlLBP2Qcd_YEGgkE28wjCUWORIVZKQn3Jtp5yWuH_Mi8R4wuYA_Xn l1dcxtkL1.jOTIzFS9moWQh71rwY_AzXywYmiQY9uwELyqfk2epLGmTm7Mypm50FegqyFN_Tl4f6 9kegoOgRB57FE_XLyxPCseDG.KOgrcs1KgjH2oO51EvPgl37IGO2LUz49WZee4zQlAJxN.RPvutF uWxA_VBWkwQHMkfBWaTDMrRVZnm6RxKS.ZEk5QdkPWrZ86N7i0gLPbuV5_iAEOxoZ0M4zXLY9GrS PAv3UVwSGwwu.X5QT9lmFS34JBNWUL.SbC3UIcwX8SwQ27IlGmrNRj3eFqIO_lCWE9RUw1QO63VB 8ZBbg3bGsRPSU_6jJOYjAwhAq5S8zxfdxh9AbXDhDmSOC4eNcg085txLKoFmMfBp0Pnpkl4xzxyC GR8JkMi8fKTW.YJsxEPZIo1bsCnZhKgQ_Doj4M0AN557XBSRPuYtNefoNqbKNX3GSbEgxsCDLVvZ Ubf2cCGnHZ8iLxZwcZ3e1acKL2e.UhxPnnFbvLjzzSN2f8VLyagqZljMqkPE_vSmTujGddsWNTwI G4At8pOtlpnjzamH97I1XDZvpNJiUBNZh20hIBs.NjxziKEzoVM3uN0Yo7x._deoCSmiV1byqWhS m60yRdtoMODtZeoR2PolL0EJfEfL7cr2kflBgP_XivJBQXun4VGrNfBThUV.uMBAMhGf558l.bSp NFG5LFEdafk0K_ugjIFDfVJpgayh2Hh27H6z2CR3iHqlQUhJlSs3AcyYR9g5l1UDfngc4utO9vH2 CnwSg4kkjVowtIXZJU1zKjgaEvrm6VqZ_wia2FZFBKg_D8V2x355t1FWU5SDaqn_hhVMQMSbTpCr _N7qtiW5.xAVPArsfV94FXI5HiXpTowsFumRT3q72kgJ5oeV3zlXk5c1DzjQ.zlb4qJCm3lKtJZr eooOgMd54kz80t_M.Pe8or9fjGOXRT_M5eZd7qDUixwXqCtwLwQFyKdiwbzwozyywQiHB6P2BnaY r4JCwjlJT5te1nYhs4WBxkJxCb6q.kiWDoLiPoEfMC0xvB9_zPzDgz1L9cIKQCeECnVC4JT.dqHM _c4_ryk2uouzrQ0spv6kV.gOuj.lp64Ni8bW44EEAcBb24urMaoxXr_IG_kMY8LIUvMdh2ZrO0zo GcJnZDEjKDQKVJfFukrCH_t4XyfhO46EWmHdgrpKPE_K4jp03VmQGbTx.iAwj_9yHsOycs4_iaxG sg8tMsud2ih89rBzuqf0yLBmByhAeoTZKBsQT0MZBCyxDop2GJePFKLKed4mWVqJoqW2hF1EJi.O g1k_EZG3EAmuC_vz8kFtFFf_eUO5q7PEk46EkFy.Qe2hu6w7Twn_JAnP2ql4sfa53ahnZaciWxBs xcG5pSChaamWqmpPlSsZycRvhN7Vfg8HMyk5uOWZPMGbFYoQIw67ZV95af48ymYkVFpyDxhfLONH c_sEFzyEaQ8GizLA0QKxzE05q3oDX3sn3sY8Ax8VppecvqMBSllI8bjWLU2d1ulUrvk85rPbUF.. TsbT42ivEO7d0YdqflnG_NYr_FFmmgTS7HtBrm1v2kQkMh2Fq4pt5NUKOewWTJkF38aXTap30Zfv FRT53fAsKa0dVdrXvUOXjxpRghZczWxd.UPY3Q0b_5FHFI0v29fep__Y0nHojF68MSQSTkNYm3hw yzzEd3rW72Rf4AnHWlKYwYjKXxm9NhH5eXJwHuBLCQSBSRmGT04EHwPXKvFUAsuL5nWV2XVCAZty dHLbz69XHzBHRBpOz7YtDEwR20cfr6GXnFCAmn5T7erQFWdR.0i4dyjcWbq6JAAKwwGxvq5EOUHh nhAccR.onu_8L7w4tlUr0P3zhaKL1AjXMxohk2fgmEaoHR9Q2uhGP.Rl9tTEHsmgmqbPELsr0PPf 99SqGogfAIQAhfKhPGwV8C0lEl4CqNyG0q_sl9KyQc2T2mOOsOiFQcFM724hz4PCcb5p.kVcK.RV mueHEcMBDzNhiSyKhxtQuoAvDrfp8oqxFcWi27RUHIrvPHBnFWq4BiavotEXlKEWnBCXDB8l7syE kF4LkLCl3bMI2oSx6Evgs.r2394new52LdhHarGQvjTnvNla4ABbUOUdVkYxLwbqq0KJRgAiggW3 A1LTlpRgep_g0644K8yg8dwmwlqlXKOoEHq7iEm4c8QEmzradViDflojWN8AkOxri316WgSzZVyG xCBCqAUcQRO2e7vrUaYO_G_gESMlLZCCBRojcEnY4aHb2PkTRqw3wZEMA46d1uaxCbEvKb8ysRxA m.83ktAB2U1uDcnBd79d2X4jnxawyfHdAT179zY.br6PMb_k1I8czwdNszEVteUIZNTi2J9g4Yu7 P.BplZ3IkYsuB.pAVTmZfaapFCvrJf3oJrFKMTDjtKCn04ACDG0Y.QCst5YoL9yn7iPuv8w4pboA xWX4xNwgudFlB78HZdAxk9S3fh6r1F7hkGOzzY6BldmGNZyrYgmWLsVE5Cp3oyr9oqOxNyzyA_m6 OYoJ7ftWSDexe9CDA_2fmvhXXp3F1Q6eeDa.kflLY99RBznJU45qc9RCmnJeoJFBJL5vOia2G72v JUdfOLl5yQ.FJPb7te0CXo3A- Received: from sonic.gate.mail.ne1.yahoo.com by sonic303.consmr.mail.gq1.yahoo.com with HTTP; Sat, 19 Dec 2020 06:44:28 +0000 Received: by smtp406.mail.ne1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 9d0c4ab4bd44960de7e21ec0a4bc7f89; Sat, 19 Dec 2020 06:44:26 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.20.0.2.21\)) Subject: Re: debug head -r368500 kernel (for example) : "lock order reversal: (sleepable after non-sleepable)" involving ure0 and a sysctl lock; more From: Mark Millard In-Reply-To: Date: Fri, 18 Dec 2020 22:44:24 -0800 Cc: John-Mark Gurney Content-Transfer-Encoding: quoted-printable Message-Id: <3392879C-F0DC-4F14-9C48-1ADD94214D49@yahoo.com> References: To: freebsd-arm , FreeBSD Current X-Mailer: Apple Mail (2.3654.20.0.2.21) X-Rspamd-Queue-Id: 4CybnB26Rgz4nXX X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RBL_DBL_DONT_QUERY_IPS(0.00)[98.137.64.204:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[98.137.64.204:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.204:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.204:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-current] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Dec 2020 06:44:31 -0000 [I managed to not cc the primary person that I intended but to cc the = secondary person. So this resend just adds jmg and removes hps.] On 2020-Dec-18, at 22:27, Mark Millard wrote: > The following is from head -r368500's artifact kernel from: >=20 > = https://artifact.ci.freebsd.org/snapshot/13.0-CURRENT/r368500/arm64/aarch6= 4/kernel.txz >=20 > but the same sort of material showed for -r368000 . > (I was attempting a bisect for a different issue but > the debug kernels did not fail for what I was looking > for and all the debug versions that I tried reported > similarly to the below.) >=20 > Note also the mixing in of "uma_zalloc_debug" activity > after the initial LOR backtrace, ure0 still involved. >=20 > Autoloading module: if_ure.ko > ure0 on uhub0 > ure0: = on usbus0 > add host 127.0.0.1: gatelock order reversal: (sleepable after = non-sleepable) > 1st 0xffffa00002b2cea0 ure0 (ure0, sleep mutex) @ = /usr/src/sys/dev/usb/usb_request.c:714 > 2nd 0xffff000000dd6858 sysctl lock (sysctl lock, sleepable rm) @ = /usway lo0 fib 0: route alrr/src/sys/kern/kern_sysctl.c:836 > lock order ure0 -> sysctl lock attempted at: > #0 0xffff00000056d068 at witness_checkorder+0xc54 > #1 0xffff0000004f8f08 at _rm_wlock_debug+0x88 > #2 0xffff00000050ee2c at sysctl_add_oid+0x60 > #3 0xffff00009e415780 at ure_attach_post+0x1a78 > #4 0xffff000000391d6c at ue_attach_post_task+0x3c > #5 0xffff0000003826e0 at usb_process+0x10c > #6 0xffff0000004baa1c at fork_exit+0x7c > #7 0xffff000000816544 at fork_trampoline+0x10 > uma_zalloc_debug: zone "malloc-128eady in table > " with the following non-sleepable locks held: > exclusive sleep mutex ure0 (ure0) r =3D 0 (0xffffa00002b2cea0) locked = @ /usr/src/sys/dev/usb/usb_request.c:714 > stack backtrace: > #0 0xffff00000056d388 at witness_debugger+0x64 > #1 0xffff00000056e518 at witness_warn+0x3ec > #2 0xffff000000778f9c at uma_zalloc_debug+0x2c > #3 0xffff000000778998 at uma_zalloc_arg+0x2c > #4 0xffff0000004d534c at malloc+0xa0 > #5 0xffff00000050ee80 at sysctl_add_oid+0xb4 > #6 0xffff00009e415780 at ure_attach_post+0x1a78 > #7 0xffff000000391d6c at ue_attach_post_task+0x3c > #8 0xffff0000003826e0 at usb_process+0x10c > #9 0xffff0000004baa1c at fork_exit+0x7c > #10 0xffff000000816544 at fork_trampoline+0x10 > uma_zalloc_debug: zone "malloc-16" with the following non-sleepable = locks held: > exclusive sleep mutex ure0 (ure0) r =3D 0 (0xffffa00002b2cea0) locked = @ /usr/src/sys/dev/usb/usb_request.c:714 > stack backtrace: > #0 0xffff00000056d388 at witness_debugger+0x64 > #1 0xffff00000056e518 at witness_warn+0x3ec > #2 0xffff000000778f9c at uma_zalloc_debug+0x2c > #3 0xffff000000778998 at uma_zalloc_arg+0x2c > #4 0xffff0000004d534c at malloc+0xa0 > #5 0xffff0000005f8c80 at strdup+0x2c > #6 0xffff00000050eeb8 at sysctl_add_oid+0xec > #7 0xffff00009e415780 at ure_attach_post+0x1a78 > #8 0xffff000000391d6c at ue_attach_post_task+0x3c > #9 0xffff0000003826e0 at usb_process+0x10c > #10 0xffff0000004baa1c at fork_exit+0x7c > #11 0xffff000000816544 at fork_trampoline+0x10 > uma_zalloc_debug: zone "malloc-64" with the following non-sleepable = locks held: > exclusive sleep mutex ure0 (ure0) r =3D 0 (0xffffa00002b2cea0) locked = @ /usr/src/sys/dev/usb/usb_request.c:714 > stack backtrace: > #0 0xffff00000056d388 at witness_debugger+0x64 > #1 0xffff00000056e518 at witness_warn+0x3ec > #2 0xffff000000778f9c at uma_zalloc_debug+0x2c > #3 0xffff000000778998 at uma_zalloc_arg+0x2c > #4 0xffff0000004d534c at malloc+0xa0 > #5 0xffff0000005f8c80 at strdup+0x2c > #6 0xffff00000050eee4 at sysctl_add_oid+0x118 > #7 0xffff00009e415780 at ure_attach_post+0x1a78 > #8 0xffff000000391d6c at ue_attach_post_task+0x3c > #9 0xffff0000003826e0 at usb_process+0x10c > #10 0xffff0000004baa1c at fork_exit+0x7c > #11 0xffff000000816544 at fork_trampoline+0x10 > uma_zalloc_debug: zone "malloc-32" with the following non-sleepable = locks held: > exclusive sleep mutex ure0 (ure0) r =3D 0 (0xffffa00002b2cea0) locked = @ /usr/src/sys/dev/usb/usb_request.c:714 > stack backtrace: > #0 0xffff00000056d388 at witness_debugger+0x64 > #1 0xffff00000056e518 at witness_warn+0x3ec > #2 0xffff000000778f9c at uma_zalloc_debug+0x2c > #3 0xffff000000778998 at uma_zalloc_arg+0x2c > #4 0xffff0000004d534c at malloc+0xa0 > #5 0xffff00000050ef3c at sysctl_add_oid+0x170 > #6 0xffff00009e415780 at ure_attach_post+0x1a78 > #7 0xffff000000391d6c at ue_attach_post_task+0x3c > #8 0xffff0000003826e0 at usb_process+0x10c > #9 0xffff0000004baa1c at fork_exit+0x7c > #10 0xffff000000816544 at fork_trampoline+0x10 > miibus0: on ure0 > rgephy0: PHY 0 on miibus0 > add hrgephy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, = 1000baseT-FDX, 1000baseT-FDX-master, auto > 0 fib 0: route already iue0: on ure0 >=20 >=20 >=20 > The context here is an RPi4 (aarch64 cortex-a72) with: >=20 > # uname -apKU > FreeBSD RPi4B 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r368500: Thu Dec 10 = 07:52:39 UTC 2020 = root@FreeBSD-head-aarch64-build.jail.ci.FreeBSD.org:/usr/obj/usr/src/arm64= .aarch64/sys/GENERIC arm64 aarch64 1300131 1300131 >=20 > The boot attempts were via uefi/ACPI using = https://github.com/pftf/RPi4 > v1.21 materials, directly booting from the USB3 SSD, no microsd card > involved. >=20 >=20 > Some context in case it contributes something for the above > (probably not) . . . >=20 > The reason for the bisect was: such boot attempts fail to mount > route with my non-debug head -r368500 kernel build. (Previously > the RPi4 was back at head -r365932 or so.) But my non-debug > builds use -mcpu=3Dcortex-a72 . (This combination has caught > missing synchronization activity before.) >=20 > In the failing case the following never shows up: >=20 > da0 at umass-sim0 bus 0 scbus0 target 0 lun 0 > da0: Fixed Direct Access SPC-4 SCSI device > da0: Serial Number REPLACED > da0: 400.000MB/s transfers > da0: 228936MB (468862128 512 byte sectors) > da0: quirks=3D0x2 >=20 > and the message: >=20 > Root mount waiting for: usbus0 >=20 > repeats indefinitely, unlike historically for my configuration. >=20 > With the artifact debug kernel instead of the non-debug > kernel, the RPi4 boots fine, other things held constant. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)