From owner-freebsd-hackers@freebsd.org Wed Feb 13 20:33:57 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BE7CB14D70D1 for ; Wed, 13 Feb 2019 20:33:57 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic305-21.consmr.mail.gq1.yahoo.com (sonic305-21.consmr.mail.gq1.yahoo.com [98.137.64.84]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AC0D073011 for ; Wed, 13 Feb 2019 20:33:56 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: _e5UpQUVM1mGUYOgQoQtoxcBiA2fciWs4Aju6Hip1je0x63Bd5hxd_Y2hk20Jwa VruBR8VsUpr0NdWRmTo.yM.fK0NoCJLkoEs4vzNiqVnqRyFbRINmU2TAJDSQUDAfFpbzOS4Vq8Is VNNzd9lDZAkoYUaATzDA.kCK6yEoZ8mGsKdBqMsBYbhj1P2qwV0d0qZl2mYP0kVaS8H6jg1F7CPO 9n1WzJRWzv9eki4IuK6NiF4IIzpKcT64bBJ9b.J9K6SxPPVgWaN65tRMgE_b.5e4EBEun4dF0wEL gqhrwGaEc2NV3b6F.7Th2pw8Bz_ZNxn17LGA8qQnHw_c54Ljd6hoMjlgs.VzAm5lFa5zKOAK7jQs gYLVVd0o6Ci5WLS9bEOiUAFhHQBTA7o7xoQVpDG7kIJ8S5ewvgvlJnNDj0.EmCPT5VdUBqDweWH9 Ecczn2l949Hswuk9neOFn5EN4ersK47HWEhdcB4ECBzhjTpmRyiPaZlZ.LIjfPpAzQ1.As1bIl_I G2RbEo7OxMJ8O_RNb0qn4ZFWKMRBOtLYkYkVCfhABuxrXPw8xzXxAykhMwaCx6E5Elva_fRnvLj_ Cf8NyRPoWFMUcHtd8E9SzSHMFHwkQUS59zOHD.Bw4xghHB6fmn5H3DCXtWIj7vldNzZyjbXnP2MX 4.paaKPRu72FO4IMdlEvbkrYL9tdcfanWOm1LXP9PV5.rfbbTDdON7WIBc26UChbKxjYFiWDgnnv pEAq1EOJu4UdyWvj8hqUvW8Bw2edV6BsHcZABg2pmNGNmEZgm24De.uWWOPfk.pjOCwPgwJbQ9RT J9w6k5JxJ563_kpSJyhAOX1UKvqxGnoZ1feCNdd3Szaav5hikqMl4tXwU5oGYMCp8tFAo2q26Bq4 u1hrpPTgh7cPr3Gevzwo9KtuN.pRicXa6YsBdENodM2vp4Yk695lssXEU63xgi6mjB5Izmd6zRxb MZTAAzyPR1AOF98eUj7B3fSX7bbxJjWXEO6qPZ8h_nkmDCG02W4hiny4qG7wCBQdAPqukvdncEnD _Ktrz6uEKjmEW0rdRfgCQrjyZzstzPTQXyheXyN615r.5pJrs1f3zigE8es78qSrX05Y- Received: from sonic.gate.mail.ne1.yahoo.com by sonic305.consmr.mail.gq1.yahoo.com with HTTP; Wed, 13 Feb 2019 20:33:47 +0000 Received: from c-67-170-167-181.hsd1.or.comcast.net (EHLO [192.168.1.113]) ([67.170.167.181]) by smtp405.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID faac3d4b65fba3935193366089efa6a3; Wed, 13 Feb 2019 20:23:39 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 12.2 \(3445.102.3\)) Subject: Questions with a powerpc64/powerpc context: relaxed use of smp_cpus in umtx_busy vs. relaxed updates to smp_cpus in machine dependent code? Message-Id: <096EABF3-1876-4E0C-9C16-ECF5C068B189@yahoo.com> Date: Wed, 13 Feb 2019 12:23:38 -0800 To: FreeBSD PowerPC ML , freebsd-hackers Hackers X-Mailer: Apple Mail (2.3445.102.3) X-Rspamd-Queue-Id: AC0D073011 X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.95 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.99)[-0.993,0]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; SUBJECT_ENDS_QUESTION(1.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; URIBL_BLOCKED(0.00)[dsl-only.net.multi.uribl.com]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[84.64.137.98.list.dnswl.org : 127.0.5.0]; IP_SCORE(-1.45)[ip: (-8.45), ipnet: 98.137.64.0/21(0.72), asn: 36647(0.57), country: US(-0.07)] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Feb 2019 20:33:58 -0000 Why I ask the questions below (after providing context): There are boot issues on old multi-processor PowerMac G5s that frequently hang up during cpu_mp_unleash --but not always. /usr/src/sys/kern/kern_umtx.c has the following code (note the smp_cpus use in the machine-independent code): static inline void umtxq_busy(struct umtx_key *key) { struct umtxq_chain *uc; =20 uc =3D umtxq_getchain(key); mtx_assert(&uc->uc_lock, MA_OWNED); if (uc->uc_busy) { #ifdef SMP if (smp_cpus > 1) { int count =3D BUSY_SPINS; if (count > 0) { umtxq_unlock(key); while (uc->uc_busy && --count > 0) cpu_spinwait(); umtxq_lock(key); } } #endif while (uc->uc_busy) { uc->uc_waiters++; msleep(uc, &uc->uc_lock, 0, "umtxqb", 0); uc->uc_waiters--; } } uc->uc_busy =3D 1; } The use of smp_cpus here on powerpc would be what is called a std::memory_order_relaxed load in c++ terms. smp_cpus does change during the machine dependent-code cpu_mp_unleash in /usr/src/sys/powerpc/powerpc/mp_machdep.c : static void cpu_mp_unleash(void *dummy) { . . . smp_cpus =3D 0; . . . STAILQ_FOREACH(pc, &cpuhead, pc_allcpu) { . . . if (pc->pc_awake) { if (bootverbose) printf("Adding CPU %d, hwref=3D%jx, = awake=3D%x\n", pc->pc_cpuid, = (uintmax_t)pc->pc_hwref, pc->pc_awake); smp_cpus++; } else . . .=20 } which are relaxed stores. [This dos not appear to be a std::memory_order_consume like context (no dependency ordered before usage).] /usr/src/sys/kern/subr_smp.c does initialize smp_cpus to 1 in its definition. (But it temporarily reverts to zero in the above code.) So far I've not managed to track down examples of specific code (in an objdump of the kernel, say) that matches up using some form(s) of the following to control access order in the various places umtxq_busy is used: lwsync (acquire/release/AcqRel fence or store-release [with load-acquire = code as well]) or: sync (a.k.a. hwsync and sync 0) (sequentially consistent = fence/store/load) Note: smp_cpus is not even volatile so, potentially, for a time a = register could be all that holds the sequence of smp_cpus values before memory is updated later. Nor have I yet found the earliest use of the umtxq_busy code. If it is late enough after cpu_mp_unleash, that might implicitly provide = something that is not a local code structure. Can anyone point me to example(s) of what controls umtxq_busy = necessarily accessing the intended smp_cpus value? =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)