From owner-freebsd-ppc@freebsd.org Sun Mar 31 02:11:43 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 54326156D163 for ; Sun, 31 Mar 2019 02:11:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [96.47.72.132]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E2BD289E92 for ; Sun, 31 Mar 2019 02:11:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by freefall.freebsd.org (Postfix) id BC2BC12721; Sun, 31 Mar 2019 02:11:42 +0000 (UTC) Delivered-To: powerpc@localmail.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [96.47.72.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "mx1.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by freefall.freebsd.org (Postfix) with ESMTPS id B9E8D12720 for ; Sun, 31 Mar 2019 02:11:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7E71189E90 for ; Sun, 31 Mar 2019 02:11:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id B60B639B7 for ; Sun, 31 Mar 2019 02:11:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id x2V2BfPH090582 for ; Sun, 31 Mar 2019 02:11:41 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id x2V2BfUo090578 for powerpc@FreeBSD.org; Sun, 31 Mar 2019 02:11:41 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: powerpc@FreeBSD.org Subject: [Bug 232387] head -r345558 (updated): system crash during kldload if_epair on powerpc64 (for more modern buildworld buildkernel toolchain experiments), Probably also: ipfw.ko linuxkpi.ko siftr.ko Possibly: hwpmc.ko Date: Sun, 31 Mar 2019 02:11:41 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: vimage X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: marklmi26-fbsd@yahoo.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: powerpc@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-Rspamd-Queue-Id: E2BD289E92 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.99 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.99)[-0.992,0]; ASN(0.00)[asn:11403, ipnet:96.47.64.0/20, country:US]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 Mar 2019 02:11:43 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D232387 --- Comment #18 from Mark Millard --- (In reply to Mark Millard from comment #15) For if_epair.ko 's problem, adding: device epair to the kernel configuration to avoid the dynamic load of the if_epair.ko works: # kyua test -k /usr/tests/Kyuafile sys/netipsec/tunnel/aes_cbc_128_hmac_sha1 sys/netipsec/tunnel/aes_cbc_128_hmac_sha1:v4 -> passed [0.626s] sys/netipsec/tunnel/aes_cbc_128_hmac_sha1:v6 -> passed [0.631s] Results file id is usr_tests.20190331-020517-933050 Results saved to /root/.kyua/store/results.usr_tests.20190331-020517-933050= .db 2/2 passed (0 failed) I'll eventually see how far kyua gets this way. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-ppc@freebsd.org Sun Mar 31 05:30:04 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D9D1C154D9F1 for ; Sun, 31 Mar 2019 05:30:03 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic301-30.consmr.mail.ne1.yahoo.com (sonic301-30.consmr.mail.ne1.yahoo.com [66.163.184.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9DC136A280 for ; Sun, 31 Mar 2019 05:30:02 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: vRPplugVM1kBhZVad6HOWyqCubgjMV.9BT6Krxi7JgvqrsgunF.TXhUT4jzy6Wc aXLakdY_B9HslTGckt5q.SIlCwhL1IvSajeQ_htdBmndxct98s.x91WUTwWh5UfW.IqhwHaTLSKr 199ToFgZgSLyLsArsqgnHfM9sq2sGFwHXCVnJGIyGixIbiPgEisEuISLvUr5OU85pQl43LaSu.kC 0PGy_Jxdsg8mDEiTkOQ2PUUFfmSP8rbi7WUzeZfWzOGScvaYDSzx59ExmyTn2LlcnSetPBJ6ctMT TLAyyWAnpx.jSHPKBAT2ifSBWYCRwLqz5Ta35HTmdenQ5LruJolgoHn6Zl4kSBPQoDuHj20lI996 r6JLZi.TTVDMWbOA_Gv1qpk.tRwB7xWr31ncrpYR_ft7tXvdbknWhKDus6bvrnEzttEv4RHpZ2Fq qa5w0SzzvLco7wN7QPsUC6D4aPuweuCHo6tT7AIrVoc6XAHKuODuJLgf4s6Bhuc471e.0hK9a43a nsaT0h97tHJlIlU9YMu7hxrksmQVMKPFK8oNS84YzI3J_UYPk.vdZ_0_FJ8Ux0eqEiRyMkuGLhI7 qIL._rt6ORUFPvklrdoD19tQVK0cU4ewCT_uIl2yX7RL00oZCJtUJNXdV20DCN8zTyv3HSRZT93M v8pGl3xHh5Rrk9ubZ13Nk399gCnkSQCBo6PZdgWCfAM8Pl7iVb7WZz0X3vB8N7_arqNNbCVz07GM Ff2Sb_315.7ty0QcDudPXdroOUS6QwfH8Sm2CJh8EkdKRlS2Fpf138ZN3v_Tsb982JhrYFc._V0k w98A1FzWWScTTs8ALQwSe_ituXueqgc1ELsUb8VyMIlYJ.zrW8_IpmPGkWW41U2v0kw8qEE0wMQY iiw0WW82N8wynhkYL3WFtof.XroPhQtVFy7XFcNWegABJhY38EKFcQk.g3gLL2I4.c_rtItaZk4m kJ5O2GdLfvy00mQ7gr7Ejc0_FuV7hMWD.yiC_PjtDtqELvJHfLXWnJjvF2w.abc8U61BijF6VC4s 8ZXalKVElYKc91ioaMoolSiVWvhuBoNpwqj6doY.PLPgQuDPwCB8Rpfq1Xr1BWc8- Received: from sonic.gate.mail.ne1.yahoo.com by sonic301.consmr.mail.ne1.yahoo.com with HTTP; Sun, 31 Mar 2019 05:30:00 +0000 Received: from c-67-170-167-181.hsd1.or.comcast.net (EHLO [192.168.1.115]) ([67.170.167.181]) by smtp402.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 7f44e7b93e84f93dd40322bc2990992c for ; Sun, 31 Mar 2019 05:29:56 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: I've completed a run of: kyua test -k /usr/tests/Kyuafile on a PowerMac G5 (2 sockets, 2 cores per) Message-Id: Date: Sat, 30 Mar 2019 22:29:55 -0700 To: FreeBSD PowerPC ML X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 9DC136A280 X-Spamd-Bar: + X-Spamd-Result: default: False [1.56 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36646, ipnet:66.163.184.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.12)[-0.119,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_SPAM_SHORT(0.91)[0.906,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-ppc@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(1.21)[ip: (3.66), ipnet: 66.163.184.0/21(1.37), asn: 36646(1.09), country: US(-0.07)]; NEURAL_SPAM_LONG(0.07)[0.072,0]; RCVD_IN_DNSWL_NONE(0.00)[199.184.163.66.list.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 Mar 2019 05:30:04 -0000 For the first time I've had a run of: kyua test -k /usr/tests/Kyuafile complete on a system that had no part built via gcc 4.2.1 . This was based on head -r335558 , cross-built via amd64->powerpc64 , system-clang and devel/powerpc64-binutils . There are fixes and hacks involved. The only file system is ufs (so zfs tests do not work). I had core files redirected for where they go, so any were not found by the tests themselves. I had to build the kernel with 'device epair' to avoid a dynamic-loading crash for dpcpu_off[cpuid] used with one or more .ko internal pcpu_entry_NAMEs , in this case pcpu_entry_epair_dcpu . It reported: =3D=3D=3D> Summary Results read from = /root/.kyua/store/results.usr_tests.20190331-021208-373603.db Test cases: 7504 total, 227 skipped, 37 expected failures, 121 broken, = 54 failed Total time: 6552.974s =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-ppc@freebsd.org Sun Mar 31 05:56:12 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B5DE4154E98B for ; Sun, 31 Mar 2019 05:56:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [96.47.72.132]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4EB376B112 for ; Sun, 31 Mar 2019 05:56:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by freefall.freebsd.org (Postfix) id 330581588C; Sun, 31 Mar 2019 05:56:11 +0000 (UTC) Delivered-To: powerpc@localmail.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "mx1.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by freefall.freebsd.org (Postfix) with ESMTPS id 2FC991588B for ; Sun, 31 Mar 2019 05:56:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CE79F6B110 for ; Sun, 31 Mar 2019 05:56:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id D2E5E5BE5 for ; Sun, 31 Mar 2019 05:56:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id x2V5u9df019329 for ; Sun, 31 Mar 2019 05:56:09 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id x2V5u9ko019328 for powerpc@FreeBSD.org; Sun, 31 Mar 2019 05:56:09 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: powerpc@FreeBSD.org Subject: [Bug 232387] head -r345558 (updated): system crash during kldload if_epair on powerpc64 (for more modern buildworld buildkernel toolchain experiments), Probably also: ipfw.ko linuxkpi.ko siftr.ko Possibly: hwpmc.ko Date: Sun, 31 Mar 2019 05:56:10 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: vimage X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: marklmi26-fbsd@yahoo.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: powerpc@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-Rspamd-Queue-Id: 4EB376B112 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.99 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.99)[-0.986,0]; ASN(0.00)[asn:11403, ipnet:96.47.64.0/20, country:US]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 Mar 2019 05:56:12 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D232387 --- Comment #19 from Mark Millard --- (In reply to Mark Millard from comment #18) For the first time I've had a run of: kyua test -k /usr/tests/Kyuafile complete on a powerpc64 system that had no part built via gcc 4.2.1 . It reported: =3D=3D=3D> Summary Results read from /root/.kyua/store/results.usr_tests.20190331-021208-37360= 3.db Test cases: 7504 total, 227 skipped, 37 expected failures, 121 broken, 54 failed Total time: 6552.974s --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-ppc@freebsd.org Sun Mar 31 18:30:28 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8836815681A3 for ; Sun, 31 Mar 2019 18:30:28 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic317-32.consmr.mail.ne1.yahoo.com (sonic317-32.consmr.mail.ne1.yahoo.com [66.163.184.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 580098C981 for ; Sun, 31 Mar 2019 18:30:26 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: NCRHeL0VM1mnpi9dnliWe0RfXYcRzv4zpLiovvNc7uoq.dkHK1fL.vt7Czet_PO .NyvivqSt42xABmxbkAAeMknQn1P9FVvAwyxa6Ej0.w38r.fv1uMH.kwk2BH2FxF89CwxSyloJ7O b5XpW4vdBxgyzPsxZiag_u.pG0L5CVJQkF1x4JLMPP_q1PyOaW9JibRoItW4g0bYHI.I5ev8gHRD CMBlgcGkddTQq6uQeNiDnAT1WEQW5WujnFoWIPlcyzjpjbeFdukVd3.jIuZ8v1PIKoi50YvoLI.m B.sEzoIyHToqbY73nZlFlNJ49MGLbuS8Oivn6peiFGQ3J_tdoBtYKvqSuKkQtD7ZAFtIdw4qQPTj 6V.L1jFNdoaNfnQT0YBHzBn7DRRaGzBQ0y_ZjjI2tFRcCjR2wbjoVFguZDA5Gn88wr1ECui13Kjv tKoky2Uf83n2hP_0jSLfbfRe9wNoCwwWQOKFIBBOIsHobNSjXdcCSfX6gsP2IX2JNHv0J18TwZTa yabtOvh.o32sIwkQ_zL09PFmwOCBlSFnkaqnDAS2ydz80tE4FX9y7F1Ytd4qyP5BntbXEIZFdPkX 3QanrbgCooQlBkcy5rvlCCyekxurApdBADkRWRb_wQS0z8yWasMgJStAd4z3juWiAZKeNLl9LoKQ .pcLZvswJL7Hsyd4GvKM5wVvbPsl930Jr6IrH0Tu5bBM0HAG7UGmqavoyB0vVt_B0VJitpMe1V.Y qE0MaSA7wLI6xUvSjqwUjGFaMXxghQIZVfzNaKHlZ0tHf0t7AngsOk1JKNban9kTA0DrNT1UK2kS C59wXiE7MRM5k3GxseskJO1Vf0_rOlRnNwqCA46bRDi7_BisD5frpEUpjPuuR8v4TWI9N3v30NUY vuS9BaVZPSw6yoKLWuivLNbEQtIpZQVOrQTLdeUUJSnbWctfX9PRqolP6DmZ_QXD0_9nOv3t14es cVkFdGjuSJSullmM85YGCZVib5A1UH4dIhX.4BnYrruQgGknARtvZx8LlTuza_9A_0du9PVXTFKJ q.39kSLF9KMN6RN1ZpRIJudkVEe72itOrZgVOrUP7dNjatduRXHwztrsfVe0Zwre1sGNnZ45sqA- - Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.ne1.yahoo.com with HTTP; Sun, 31 Mar 2019 18:30:24 +0000 Received: from c-67-170-167-181.hsd1.or.comcast.net (EHLO [192.168.1.115]) ([67.170.167.181]) by smtp404.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID ab054f8c7d1489cf7d03c7cf8815b3a4; Sun, 31 Mar 2019 18:30:23 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Priority: 3 Subject: Fwd: [PATCH] D59694: [PPC64][libunwind] Fix r2 not properly restored Date: Sun, 31 Mar 2019 11:30:22 -0700 References: <5dec87ec91c02753302586cef9d16df6@localhost.localdomain> To: FreeBSD Toolchain , FreeBSD PowerPC ML Message-Id: <8578AE62-9A2E-4FC2-BFB0-D9D52912E853@yahoo.com> Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 580098C981 X-Spamd-Bar: / X-Spamd-Result: default: False [0.29 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; HAS_X_PRIO_THREE(0.00)[3]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36646, ipnet:66.163.184.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.79)[-0.791,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_SPAM_SHORT(0.97)[0.973,0]; NEURAL_HAM_LONG(-0.37)[-0.374,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.99)[ip: (2.46), ipnet: 66.163.184.0/21(1.42), asn: 36646(1.14), country: US(-0.07)]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[43.184.163.66.list.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 Mar 2019 18:30:28 -0000 The below does not deal with building llvm's libunwind for WITH_LIB32=3D (or 32-bit powerpc targeting in general), just with targeting powerpc64 directly. Begin forwarded message: From: Nemanja Ivanovic via Phabricator Subject: [PATCH] D59694: [PPC64][libunwind] Fix r2 not properly restored Date: 2019-March-31 at 05:31:17 PDT . . . nemanjai accepted this revision. nemanjai added a comment. I don't really know much about how libunwind is tested so I can't really = comment on how to write one (or the need for one). But the code looks = good to me. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D59694/new/ https://reviews.llvm.org/D59694 =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-ppc@freebsd.org Mon Apr 1 17:00:24 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4399B15695D2 for ; Mon, 1 Apr 2019 17:00:24 +0000 (UTC) (envelope-from ich@malte.de) Received: from mail.wedel.name (mail.wedel.name [IPv6:2a01:4f8:151:448::b009:4a51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1EAB7776D5 for ; Mon, 1 Apr 2019 17:00:22 +0000 (UTC) (envelope-from ich@malte.de) Received: from [IPv6:2a02:2455:1e1:5500:1436:648c:69de:eae8] (unknown [IPv6:2a02:2455:1e1:5500:1436:648c:69de:eae8]) by mail.wedel.name (Postfix) with ESMTPSA id C14631B089 for ; Mon, 1 Apr 2019 19:00:20 +0200 (CEST) From: Malte Wedel Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 12.2 \(3445.102.3\)) Subject: FreeBSD 12 on Powerbook G4 Message-Id: Date: Mon, 1 Apr 2019 19:00:20 +0200 To: freebsd-ppc@freebsd.org X-Mailer: Apple Mail (2.3445.102.3) X-Rspamd-Queue-Id: 1EAB7776D5 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of ich@malte.de designates 2a01:4f8:151:448::b009:4a51 as permitted sender) smtp.mailfrom=ich@malte.de X-Spamd-Result: default: False [-3.37 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.wedel.name]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-ppc@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; DMARC_NA(0.00)[malte.de]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MX_GOOD(-0.01)[mail.malte.de]; NEURAL_HAM_SHORT(-0.74)[-0.736,0]; IP_SCORE(-0.82)[ipnet: 2a01:4f8::/29(-2.16), asn: 24940(-1.95), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Apr 2019 17:00:24 -0000 Hello, I tried to install FreeBSD 12 on my old PowerBook G4 Titanium. Had to = boot with disabled Firewire by entering the following on the OFW-console=20= set hint.fwohci.0.disabled=3D=E2=80=9C1=E2=80=9D set hint.firewire.0.disabled=3D=E2=80=9C1=E2=80=9D boot After that installation went through without issues. As there are no = binary packages for ppc, I installed ports and tried to build something, = =E2=80=9Cld" failed randomly in configure, making it impossible to = install anything from the ports collection. Calling =E2=80=9Cld = =E2=80=94version=E2=80=9D 50% of the time returns the actual version = info and 50% of the time it returns =E2=80=9CIllegal instruction (core = dumped)=E2=80=9D. Couldn=E2=80=99t install gdb either, to gather further = information. I now installed 11.2 instead, where this issue doesn=E2=80=99t occur. = Did anyone else experience this issue with 12.0? Is there any interest = to further analyze what is going wrong, and how would I proceed to get = further information? Regards, Malte= From owner-freebsd-ppc@freebsd.org Mon Apr 1 18:34:30 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 23773156BF7B for ; Mon, 1 Apr 2019 18:34:30 +0000 (UTC) (envelope-from chmeeedalf@gmail.com) Received: from mail-it1-x12d.google.com (mail-it1-x12d.google.com [IPv6:2607:f8b0:4864:20::12d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2AD4D83C3D for ; Mon, 1 Apr 2019 18:34:29 +0000 (UTC) (envelope-from chmeeedalf@gmail.com) Received: by mail-it1-x12d.google.com with SMTP id w15so786894itc.0 for ; Mon, 01 Apr 2019 11:34:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=RBvXqoFw3wDk/7xfI44L+Viov6B12fAZZIHs1oG0Uqg=; b=Ov1+eoySenNHUvZCKHN0Lew4KG5Miq3/G1yG7F6rSpUmBkrFF4javyZRNnSY52hkQn a81hADKZ7N8hALPOfdHTDSlsIrO0vmFFGI/uNCm/6yxXvY7rtoqaUIl/npe7YNoswNyb OfB37hAn7GGBCJRBIO17Qsp3gTsqdaMtNO4JseyZuEkFmsUd6+O9JJYRImJp8pdJs9tA DKWToYwSmAlUMEMarI6CKGwsqi8MXYidvrtSWH+oFkYJ2UyCS9s2rinQzYggI1JetuWY am5rlzBEY3tO4UOjCjGPHIwo33iUpWWS/7/YwFi+gR52GLxEhTXWRiQsE1lLfySo9/3W GD7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=RBvXqoFw3wDk/7xfI44L+Viov6B12fAZZIHs1oG0Uqg=; b=EVBACNHOqOG9jngNl+DZMlrrhWRv62bzyss11TPs0hov4QqrmedGK7FT9Osok19gLO PPnGo8dlOKRoQYth0WMYsqtHfJwBic0onwphcdP7d67rfRWXogGnnp242/i1YwL38yIv BFw7QEO+pn0pw80ryseD+7UvlMhMbj09UHCYQmv4p38jAObsT2llfQpeDDMw45A88DHc DbI39zvfIW/pVkX06t5j4Z8aGip9etcuIgDfSXQqMR/Dz28Rm2NfqUoebut7Jl1EeYEE xWuSsUG3n2c1Wcjj0gzKAsJUbA5V7s7bPHUceOcd6zCY1/zqKEXH5f7OV01EKj0e9Dyu IOvw== X-Gm-Message-State: APjAAAVC/1nU49gaxqKVY/YOX/oJo/uIcPLNw0Y8IxKwDHsSUPCgvD74 WxGtauK5oI2CxmOWOhAJ/Nl67sT0 X-Google-Smtp-Source: APXvYqy1+B0tiFXeUOGS2wgsZQujz9CnTU5/ubO8VUsPUk70ij/q4vs/rtXEZQWVc2GaBZxNxVcKXw== X-Received: by 2002:a02:924d:: with SMTP id y13mr38556611jag.24.1554143668372; Mon, 01 Apr 2019 11:34:28 -0700 (PDT) Received: from titan.knownspace (173-25-245-129.client.mchsi.com. [173.25.245.129]) by smtp.gmail.com with ESMTPSA id v187sm7695285ita.0.2019.04.01.11.34.27 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 01 Apr 2019 11:34:28 -0700 (PDT) Date: Mon, 1 Apr 2019 13:34:24 -0500 From: Justin Hibbits To: Malte Wedel Cc: freebsd-ppc@freebsd.org Subject: Re: FreeBSD 12 on Powerbook G4 Message-ID: <20190401133424.0615a7e7@titan.knownspace> In-Reply-To: References: X-Mailer: Claws Mail 3.17.3 (GTK+ 2.24.32; powerpc64-portbld-freebsd13.0) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 2AD4D83C3D X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=Ov1+eoyS; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of chmeeedalf@gmail.com designates 2607:f8b0:4864:20::12d as permitted sender) smtp.mailfrom=chmeeedalf@gmail.com X-Spamd-Result: default: False [-6.67 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.89)[-0.892,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_TLS_LAST(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-ppc@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[d.2.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.76)[ip: (-8.72), ipnet: 2607:f8b0::/32(-2.89), asn: 15169(-2.15), country: US(-0.06)] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Apr 2019 18:34:30 -0000 Hi Malte, On Mon, 1 Apr 2019 19:00:20 +0200 Malte Wedel wrote: > Hello, >=20 > I tried to install FreeBSD 12 on my old PowerBook G4 Titanium. Had to > boot with disabled Firewire by entering the following on the > OFW-console=20 >=20 > set hint.fwohci.0.disabled=3D=E2=80=9C1=E2=80=9D > set hint.firewire.0.disabled=3D=E2=80=9C1=E2=80=9D > boot >=20 > After that installation went through without issues. As there are no > binary packages for ppc, I installed ports and tried to build > something, =E2=80=9Cld" failed randomly in configure, making it impossibl= e to > install anything from the ports collection. Calling =E2=80=9Cld =E2=80=94= version=E2=80=9D 50% > of the time returns the actual version info and 50% of the time it > returns =E2=80=9CIllegal instruction (core dumped)=E2=80=9D. Couldn=E2=80= =99t install gdb > either, to gather further information. >=20 > I now installed 11.2 instead, where this issue doesn=E2=80=99t occur. Did > anyone else experience this issue with 12.0? Is there any interest to > further analyze what is going wrong, and how would I proceed to get > further information? >=20 > Regards, > Malte This sounds similar to what I was experiencing on powerpcspe, and fixed in head. Can you try a 13.0-CURRENT snapshot from late November? If that works fine, then I think I know the change that fixes it, and I can MFC it. It probably should be MFC'd anyway, I just forgot to mark it as such, so let it slip through the cracks. The change in question is r340653, so if you're able to apply just that change to 12.0 and recompile your kernel (even cross-compile), and test it, that should confirm my thought. - Justin From owner-freebsd-ppc@freebsd.org Mon Apr 1 20:28:43 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3A7C1156E0FD for ; Mon, 1 Apr 2019 20:28:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D26F987D48 for ; Mon, 1 Apr 2019 20:28:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by freefall.freebsd.org (Postfix) id C5D0913C49; Mon, 1 Apr 2019 20:28:42 +0000 (UTC) Delivered-To: powerpc@localmail.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "mx1.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by freefall.freebsd.org (Postfix) with ESMTPS id C263813C48 for ; Mon, 1 Apr 2019 20:28:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7025F87D46 for ; Mon, 1 Apr 2019 20:28:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id B6A6A1AAB0 for ; Mon, 1 Apr 2019 20:28:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id x31KSfSf012105 for ; Mon, 1 Apr 2019 20:28:41 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id x31KSf8e012104 for powerpc@FreeBSD.org; Mon, 1 Apr 2019 20:28:41 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: powerpc@FreeBSD.org Subject: [Bug 232387] head -r345558 (updated): system crash during kldload if_epair on powerpc64 (for more modern buildworld buildkernel toolchain experiments), Probably also: ipfw.ko linuxkpi.ko siftr.ko Date: Mon, 01 Apr 2019 20:28:41 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: vimage X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: marklmi26-fbsd@yahoo.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: powerpc@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: short_desc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-Rspamd-Queue-Id: D26F987D48 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.98)[-0.981,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Apr 2019 20:28:43 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D232387 Mark Millard changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|head -r345558 (updated): |head -r345558 (updated): |system crash during kldload |system crash during kldload |if_epair on powerpc64 (for |if_epair on powerpc64 (for |more modern buildworld |more modern buildworld |buildkernel toolchain |buildkernel toolchain |experiments), Probably |experiments), Probably |also: ipfw.ko linuxkpi.ko |also: ipfw.ko linuxkpi.ko |siftr.ko Possibly: hwpmc.ko |siftr.ko --- Comment #20 from Mark Millard --- >From what I've seen: /boot/kernel/hwpmc.ko: U dpcpu_off U pcpu_entry_pmc_sampled does not cause problems (because pcpu_entry_pmc_sampled is defined in /boot/kernel/kernel instead of being dynamically loaded). So I've removed hwpmc.ko from the summary. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-ppc@freebsd.org Mon Apr 1 20:29:21 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 618E4156E12E for ; Mon, 1 Apr 2019 20:29:21 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic307-9.consmr.mail.ne1.yahoo.com (sonic307-9.consmr.mail.ne1.yahoo.com [66.163.190.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 571D187D79 for ; Mon, 1 Apr 2019 20:29:20 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: YybEYKAVM1lTdpeEKTR6IZ8lqwxMCG4qH15yGEuY1GaQjneycQ8BSplRX0GAlJy NFoXwadzhdeQ2aShdbQdxBf2QgbxrlhFkAZuXcwpq5LQbGCCCcBc4wPrcIMFs53kYzt_x4Y55VPu pZB4wd9xkA9q3A1R5gNsQUJ74ioMYjlMYVyXdDG2AgKOg9l7X7kNoL66fCzjN7swM_GFlGtZQurB Z0VgyepIGbbKOUhXAWlftSxyREBdXjrtw71yP881H2zoWNHRcrgH.iwNw9D.qRMPcp0bBY6SCBDm N3hw53Y58.GVkEpyanVC6InuBlBWwxDNRBIsx96cMkk91dzeBHFSm3vNY0Yte9I2.PfSkl_YqA0v YfWnN1RgtvzG.Ym1Xn1eQCn2Bp2LwKoK7tc0IAGQDaTYF_yRtvDoPkh5D88E8ncsXZaUanoxyJW6 uzlNjKUX70i0BEmiD1Rxggtespcjfke0KpDhtK69kukFdynifFkOX14_LQ1XC0u5qUWpOZjFH2fm Mr2LCUGiuJqzxKDh7XI0F9oLYXudV0sghPwsyZaPD1uNDMlWhLMwMKN0BHu9xjEdzmPB.KMTGToT CXPu9PVhJoQsWfS1jqERWV0bnExwxmI2FuQhtHUC3n4N2WZM_2rQyBOMEBdD4pzq5ady709SNy7p r.EF9Vtgz3eXiJHlWgJK9OSSn8hhoWtEH4mdYDuHdFgpQ4ycOJz5UWizQtidaDeFdefhWqCC_KfB ZroPasduSss1aylFos2EIMvLH7Kf55MH9TEUCropxw9nyIcU5gOmVin0khys978dgatH0SgTXAkv X2kQUYp8XN8yMR7X7.R2ZkhB3BCCk2fn5Q0Bkx39YPNNQ9m9Havjuw1HPHlW1D9VmeFauv3E8Lu4 _zNY9JKDDrG5r3Wmy7WGe0Gje2kaXMD6HOB4rQlHKGqH33Xadion_Z.DbwqaGwm3IvrAj8k0jIlh h8z_uj3wQAFwGpqwOH6KsHV4Wse_lH1OH2PeUqy8ausbMq9Uq7kg6er3MhEAeRT9AKyFVX0cYLDY GeObJN.tvKYoAsOZGC2vF9lhzYerM4NKbArUBtIOjNGVgkJssKo7yfvnfI2fFB3n.4OaxmUpC3w- - Received: from sonic.gate.mail.ne1.yahoo.com by sonic307.consmr.mail.ne1.yahoo.com with HTTP; Mon, 1 Apr 2019 20:29:18 +0000 Received: from c-67-170-167-181.hsd1.or.comcast.net (EHLO [192.168.1.113]) ([67.170.167.181]) by smtp424.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 385fe7fa83865ad0447fee378a99f3bf; Mon, 01 Apr 2019 20:29:16 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: power9: "Successfully booted an LLVM compiled kernel": Can kernel modules (.ko) be dynamically loaded? Date: Mon, 1 Apr 2019 13:29:14 -0700 References: <53D45B7C-E7F0-4DC9-B09D-D3EFD56122E9@yahoo.com> To: FreeBSD PowerPC ML , Justin Hibbits In-Reply-To: <53D45B7C-E7F0-4DC9-B09D-D3EFD56122E9@yahoo.com> Message-Id: <3BF85ED4-FB88-4660-8D32-C86F06AE8470@yahoo.com> X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 571D187D79 X-Spamd-Bar: ++++ X-Spamd-Result: default: False [4.24 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(1.56)[ip: (5.24), ipnet: 66.163.184.0/21(1.47), asn: 36646(1.17), country: US(-0.06)]; SUBJECT_ENDS_QUESTION(1.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36646, ipnet:66.163.184.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_SPAM_SHORT(0.89)[0.892,0]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; NEURAL_SPAM_MEDIUM(0.53)[0.528,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.77)[0.769,0]; RCVD_IN_DNSWL_NONE(0.00)[32.190.163.66.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Apr 2019 20:29:21 -0000 On 2019-Mar-23, at 12:13, Mark Millard wrote: > As of when clang 5 (and later) became system clang, > I've found that .ko files started using R_PPC64_JMP_SLOT > r_types in .rela.plt sections. (My context, then and > now is still ELFv1 ABI and based on svn's head > materials.) >=20 > Last I checked, FreeBSD did not support R_PPC64_JMP_SLOT > and crashed during a dynamic-load of any(?) .ko that had > such. >=20 > So I've been building the modules that I want to use into > the kernel itself for when I experiment with clang-based > powerpc64 system builds. >=20 > (I've no clue if the R_PPC64_JMP_SLOT use is considered > "KBI" compliant or not vs. if FreeBSD is/was just > incomplete in its coverage of the "KBI" for pwoerpc64: > so which side should change.) >=20 > I recognize that power9 likely is based on the ELFv2 > ABI. Ignore the question: I've tested head -r345758 and clang is producing R_PPC64_JMP_SLOT in .ko files but the dynamic loads are working. I no longer need to build modules into the kernel for this issue. # uname -apKU FreeBSD FBSDG5L 13.0-CURRENT FreeBSD 13.0-CURRENT #8 r345758M: Sun Mar = 31 19:43:35 PDT 2019 = markmi@FBSDFSSD:/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64= /usr/src/powerpc.powerpc64/sys/GENERIC64vtsc-NODBG powerpc powerpc64 = 1300017 1300017 amd64->powerpc64 cross-built with system-clang and = devel/powerpc64-binutils using WITH_LLVM_LIBUNWIND=3D (patched) and WITHOUT_LIB32=3D . Various other = patches/workarounds for other issues are present, unrelated to R_PPC64_JMP_SLOT use in .ko = files. (I exierment with using more modern tool-chains to build for powerpc64 and = powerpc.) The system is an old PowerMac G5 (2 sockets, 2 cores per). I was able to comment out my prior inclusion of filemon and mac_ntpd in the kernel, so I now have: # kldstat Id Refs Address Size Name 1 6 0xc000000000100000 1927578 kernel 2 1 0xe00000008fcd2000 12000 mac_ntpd.ko 3 1 0xe00000008efda000 15000 filemon.ko I've no clue just when the status changed. I have closed bugzilla 224561 as overcome by events. (For other reasons I have to build in epair to allow kyua runs to not crash the system. I've updated bugzilla 232387 with recent information as well.) =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-ppc@freebsd.org Tue Apr 2 21:52:11 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A889915500AC for ; Tue, 2 Apr 2019 21:52:11 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic316-55.consmr.mail.gq1.yahoo.com (sonic316-55.consmr.mail.gq1.yahoo.com [98.137.69.31]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EBD788265E for ; Tue, 2 Apr 2019 21:52:09 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: 8QlUV0YVM1mVeefU1PEQ5Q5OeifmLTGWte.BE7QcM9xe2HzbP7vCh4WGvIsMZac MEeAMOBQlDvKixBXIsBMQARF2qUo7s8jMAKrn4unoev6Pf5F3zizUq.sFyNoKHZoBoY4Mslwf0Yc VBRhkihDFF2gPmFH.s.Je5FyodLguSCaqAKdA6yHR3kZw4qOQLrtQvfYGTx5eX6y1HZOXCcUwRXd 0mmUXhCV8kskj1H0y_yhAdjISBfUjhjQf5f9WU8jYjQSr8i12AsCmNSZBHoOeA_1GdMjeCNQTyhi Trwo82XEiDr2UQDKJ2bfkiKrhbDor7BUOHue0.tKoK5eOqyABZ9wrcopC3D_IR3nJTwZqaQtzfKH O4ld63hXwcMUiGzoI5vGbAqsMPDk1jIijFu8JakoF.C9EKdDJVCdobsclaskwe.DA5Oijy6ORArp bU9SUaYMoRvt8Dax6GOiU6QpH9Esa.eoaIRr1tkx9ptnEH2TEzA5U8mv3Bgj1Z1BxWFn9Ae_FFrv 7iRjavpFh2W0.vm843.tyW1JzuY0NG7DCP2Yd.Etzy3cxWaBhQaoUY0cvQ.lHy6q0Ixtlz.zBqdt YjStbyILk21rts1A71XWTKrsxBhuQhE9Do1A2Aq6KNfJOmRWYBU.TC5fpHA4E2DJJtTW3MIbTqEs P7dgP_qdMaNsdXmUmS41Y2c53HzcYwiHmBVppviM8VCOcdMwyg6Zw2JLddX8ie_t3H6uTkFC3BQW NDc17wREpCp3p8mG80tpXwM3Zs9YJOOk_iKW3ZJU9bPLxyDbdwNHRFNdnrNFrBUCIRWsEwJMzexl GsGng995z.xrPzpP5W8EdCY2R6n_Cd6pizLJpa7mdHxai0HcpP5cYFCx5XkjpEdHhP.6Zc2dkni0 bNUVOBYBadgZAcG4Key8PBcG.wdZml_0q5vETpKBbOdboVIt7wYh.vZ.8O_4VqqspF1PN2qvqdTn WQ35X9b4XvqHMUaSINq72pxWxwfETYfKPWsgCVqPELCd7tNlsz9gPa0.HkMYTFSK5x9oQnqFY4vU NlNgJXSfFmoF52Y152aqxXfpfq8v0elypo9MGPAcNAyK4MByLAG796FBKYg-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic316.consmr.mail.gq1.yahoo.com with HTTP; Tue, 2 Apr 2019 21:52:03 +0000 Received: from c-67-170-167-181.hsd1.or.comcast.net (EHLO [192.168.1.115]) ([67.170.167.181]) by smtp416.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 0a7887cff012312741b2336377531944 for ; Tue, 02 Apr 2019 21:52:02 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: powerpc64 r13 setjmp/longjmp use; 32-bit setjmp/longjmp powerpc r2 use; they look a little odd to me (but operational) Message-Id: <246BE62D-16B7-4D87-BF3C-A5172DF91425@yahoo.com> Date: Tue, 2 Apr 2019 14:52:01 -0700 To: FreeBSD PowerPC ML X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: EBD788265E X-Spamd-Bar: ++ X-Spamd-Result: default: False [2.91 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_SPAM_SHORT(0.48)[0.475,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-ppc@freebsd.org]; NEURAL_SPAM_MEDIUM(0.42)[0.422,0]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(2.08)[ip: (8.76), ipnet: 98.137.64.0/21(0.95), asn: 36647(0.76), country: US(-0.06)]; NEURAL_SPAM_LONG(0.44)[0.441,0]; RCVD_IN_DNSWL_NONE(0.00)[31.69.137.98.list.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Apr 2019 21:52:12 -0000 In looking around to find what to expect for 32-bit powerpc r2 handling I ran into things that look a little odd to me for both powerpc64 (r13) and 32-bit powerpc (r2): setjmp/longjmp handling of those registers. head -r345758 based for the below examples, system-clang did the buildworld's involved. powerpc64 r13 handling for a system-clang-based buildworld: r13 is saved to 72(r6) but never restored from 72(r3). The location for 72(r6) seems to be write only. (Recorded for debugging use? It is not obvious that it should be saved at all, given the _set_tp r13 use related to thread-local storage.) # objdump -d --prefix-addresses = /usr/obj/DESTDIRs/clang-powerpc64-installworld_altbinutils-poud/lib/libc.s= o.7 | egrep -C3 '(_set_tp|.*jmp.*).*(\|[^0-9]72\(r)' | more 00000000000a4d68 <.__getcontextx+0x94> blr ... 00000000000a4d78 <._set_tp> addi r3,r3,28688 00000000000a4d7c <._set_tp+0x4> mr r13,r3 00000000000a4d80 <._set_tp+0x8> blr ... 00000000000a4d90 <.__syncicache> mflr r0 -- 00000000000a50cc <.sigsetjmp+0x4c> stfd f16,240(r6) 00000000000a50d0 <.sigsetjmp+0x50> std r12,64(r6) 00000000000a50d4 <.sigsetjmp+0x54> stfd f17,248(r6) 00000000000a50d8 <.sigsetjmp+0x58> std r13,72(r6) 00000000000a50dc <.sigsetjmp+0x5c> stfd f18,256(r6) 00000000000a50e0 <.sigsetjmp+0x60> std r14,80(r6) 00000000000a50e4 <.sigsetjmp+0x64> stfd f19,264(r6) -- 00000000000a52b0 <.setjmp+0x40> stfd f16,240(r6) 00000000000a52b4 <.setjmp+0x44> std r12,64(r6) 00000000000a52b8 <.setjmp+0x48> stfd f17,248(r6) 00000000000a52bc <.setjmp+0x4c> std r13,72(r6) 00000000000a52c0 <.setjmp+0x50> stfd f18,256(r6) 00000000000a52c4 <.setjmp+0x54> std r14,80(r6) 00000000000a52c8 <.setjmp+0x58> stfd f19,264(r6) -- 00000000000a5474 <._setjmp+0x24> stfd f16,240(r3) 00000000000a5478 <._setjmp+0x28> std r12,64(r3) 00000000000a547c <._setjmp+0x2c> stfd f17,248(r3) 00000000000a5480 <._setjmp+0x30> std r13,72(r3) 00000000000a5484 <._setjmp+0x34> stfd f18,256(r3) 00000000000a5488 <._setjmp+0x38> std r14,80(r3) 00000000000a548c <._setjmp+0x3c> stfd f19,264(r3) (It appears that C++ exception handling should not be saving or adjusting r13 for powerpc64.) 32-bit powerpc handling for a system-clang-based buildworld: r2 is saved to 20(r6) (first going through r9) but restored to r9 via 20(r3), never going back to r2. (Recorded for debugging use? It is not obvious that it should be saved at all [or restored to r9], given the _set_tp r2 use related to thread-local storage.) # objdump -d --prefix-addresses = /usr/obj/DESTDIRs/gcc421-powerpc-installworld/lib/libc.so.7 | egrep -C3 = '(_set_tp|.*jmp.*).*(\|[^0-9]20\(r)' | more 0005bee8 mflr r11 0005beec mfcr r12 0005bef0 mr r10,r1 0005bef4 mr r9,r2 0005bef8 stmw r9,20(r6) 0005befc stfd f14,112(r6) 0005bf00 stfd f15,120(r6) 0005bf04 stfd f16,128(r6) -- 0005bf44 li r3,0 0005bf48 blr 0005bf4c nop 0005bf50 lmw r9,20(r3) 0005bf54 lfd f14,112(r3) 0005bf58 lfd f15,120(r3) 0005bf5c lfd f16,128(r3) -- 0005bffc mflr r11 0005c000 mfcr r12 0005c004 mr r10,r1 0005c008 mr r9,r2 0005c00c stmw r9,20(r6) 0005c010 stfd f14,112(r6) 0005c014 stfd f15,120(r6) 0005c018 stfd f16,128(r6) -- 0005c054 stfd f31,248(r6) 0005c058 li r3,0 0005c05c blr 0005c060 <__longjmp> lmw r9,20(r3) 0005c064 <__longjmp+0x4> lfd f14,112(r3) 0005c068 <__longjmp+0x8> lfd f15,120(r3) 0005c06c <__longjmp+0xc> lfd f16,128(r3) -- 0005c0f0 <_setjmp> mflr r11 0005c0f4 <_setjmp+0x4> mfcr r12 0005c0f8 <_setjmp+0x8> mr r10,r1 0005c0fc <_setjmp+0xc> mr r9,r2 0005c100 <_setjmp+0x10> stmw r9,20(r3) 0005c104 <_setjmp+0x14> stfd f14,112(r3) 0005c108 <_setjmp+0x18> stfd f15,120(r3) 0005c10c <_setjmp+0x1c> stfd f16,128(r3) -- 0005c154 <_setjmp+0x64> nop 0005c158 <_setjmp+0x68> nop 0005c15c <_setjmp+0x6c> nop 0005c160 <_longjmp> lmw r9,20(r3) 0005c164 <_longjmp+0x4> lfd f14,112(r3) 0005c168 <_longjmp+0x8> lfd f15,120(r3) 0005c16c <_longjmp+0xc> lfd f16,128(r3) -- 0005c3a0 b 0005c364 0005c3a4 <_set_tp> stwu r1,-32(r1) 0005c3a8 <_set_tp+0x4> addi r3,r3,28680 0005c3ac <_set_tp+0x8> mr r2,r3 0005c3b0 <_set_tp+0xc> addi r1,r1,32 0005c3b4 <_set_tp+0x10> blr 0005c3b8 <__syncicache> stwu r1,-64(r1) (It appears that C++ exception handling should not be saving or adjusting r2 for 32-bit powerpc.) (I had remembered that there was something odd with 32-bit powerpc r2 handling but had forgotten the details. Until rediscovering the above, it had left me wondering if C++ exception handling should involve adjusting r2 or not for FreeBSD. I've worded some prior list submittals presuming save/restore was appropriate for C++ exception handling. That was wrong.) =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-ppc@freebsd.org Tue Apr 2 22:27:45 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C541E1550C93; Tue, 2 Apr 2019 22:27:44 +0000 (UTC) (envelope-from tuexen@fh-muenster.de) Received: from drew.franken.de (mail-n.franken.de [193.175.24.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.franken.de", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A27B28362E; Tue, 2 Apr 2019 22:27:43 +0000 (UTC) (envelope-from tuexen@fh-muenster.de) Received: from mbp.fritz.box (p54952123.dip0.t-ipconnect.de [84.149.33.35]) (Authenticated sender: macmic) by mail-n.franken.de (Postfix) with ESMTPSA id E10F1721E2826; Wed, 3 Apr 2019 00:27:32 +0200 (CEST) From: Michael Tuexen Message-Id: Content-Type: multipart/signed; boundary="Apple-Mail=_A26B82D4-1AFC-4AF8-A85C-EB10136E809F"; protocol="application/pkcs7-signature"; micalg=sha-256 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed] Date: Wed, 3 Apr 2019 00:27:32 +0200 In-Reply-To: <20190324110138.GR1923@kib.kiev.ua> Cc: Bruce Evans , freebsd-hackers Hackers , FreeBSD PowerPC ML To: Konstantin Belousov References: <20190303111931.GI68879@kib.kiev.ua> <20190303223100.B3572@besplex.bde.org> <20190303161635.GJ68879@kib.kiev.ua> <20190304043416.V5640@besplex.bde.org> <20190304114150.GM68879@kib.kiev.ua> <20190305031010.I4610@besplex.bde.org> <20190306172003.GD2492@kib.kiev.ua> <20190308001005.M2756@besplex.bde.org> <20190307222220.GK2492@kib.kiev.ua> <20190309144844.K1166@besplex.bde.org> <20190324110138.GR1923@kib.kiev.ua> X-Mailer: Apple Mail (2.3445.104.8) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=disabled version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mail-n.franken.de X-Rspamd-Queue-Id: A27B28362E X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-1.40 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; MV_CASE(0.50)[]; HAS_ATTACHMENT(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[mx3.fh-muenster.de,mx2.fh-muenster.de,mx1.fh-muenster.de,mx4.fh-muenster.de]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_IN_DNSWL_LOW(-0.10)[27.24.175.193.list.dnswl.org : 127.0.5.1]; MIME_TRACE(0.00)[0:+,1:+]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:680, ipnet:193.174.0.0/15, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[35.33.149.84.zen.spamhaus.org : 127.0.0.10]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.84)[-0.843,0]; FROM_HAS_DN(0.00)[]; SIGNED_SMIME(-2.00)[]; NEURAL_SPAM_SHORT(0.02)[0.022,0]; RCPT_COUNT_THREE(0.00)[4]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; DMARC_NA(0.00)[fh-muenster.de]; AUTH_NA(1.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.15)[0.153,0]; R_SPF_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; IP_SCORE(0.08)[ipnet: 193.174.0.0/15(0.40), asn: 680(0.02), country: DE(-0.01)]; FREEMAIL_CC(0.00)[optusnet.com.au] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Apr 2019 22:27:45 -0000 --Apple-Mail=_A26B82D4-1AFC-4AF8-A85C-EB10136E809F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On 24. Mar 2019, at 12:01, Konstantin Belousov = wrote: >=20 > On Sat, Mar 09, 2019 at 06:00:14PM +1100, Bruce Evans wrote: >> I more strongly disclike (sic) the more complete merge. The central = APIs >> have even more parameters and reduced type safety to describe objects = as >> (offset, size) pairs. > I changed the patch to be type-safe. Now I like it even more. It = provides > 1. internal > 2. concise > 3. type-safe > API to fetch data from timehands. The implementation needs to be read > only once. Hi, I'm a bit lost... I think this started to fix a problem on G5 PowerMacs. Do you think this patch solves the problem. Should this be tested? Or is this still work in progress or a general improvement not necessary fixing the problem on G5 PowerMacs? Best regards Michael >=20 >> Small delicate loops are ideal for duplicating. They are easier to >> understand individually and short enough to compare without using = diff >> to see gratuitous and substantive differences. Multiple instances = are >> only hard to write and maintain. Since these multiple instances are >> already written, they are only harder to maintain. > This is a good argument to have bintime_off and getthmember unmerged > (there are two small but delicate loops). The API is internal, so it = is > only matter for maintainer, which means that the absence of = duplication > is important. More, all that arguments clearly explain why there = should > be not twenty similar loops scattered over the source. >=20 >>=20 >>>> XX void >>>> XX binuptime(struct bintime *bt) >>>> XX { >>>> XX @@ -361,7 +383,7 @@ >>>> XX th =3D timehands; >>>> XX gen =3D atomic_load_acq_int(&th->th_generation); >>>> XX *bt =3D th->th_offset; >>>> XX - bintime_addx(bt, th->th_scale * tc_delta(th)); >>>> XX + bintime_adddelta(bt, th); >>>> XX atomic_thread_fence_acq(); >>>> XX } while (gen =3D=3D 0 || gen !=3D th->th_generation); >>>> XX } >>>>=20 >>>> This is the kind of non-churning change that I like. >>> Ok. I made all cases where timehands are read, more uniform by >>> moving calculations after the generation loop. This makes the >>> atomic part of the functions easier to see, and loop body has lower >>> chance to hit generation reset. >>=20 >> I think this change is slightly worse: >> - it increases register pressure. 'scale' and 'delta' must be read = in a >> alost program program before the loop exit test. The above order = uses >> them and stores the results to memory, so more registers are free = for >> the exit test. i386 certainly runs out of registers. IIRC, i386 = now >> spills 'gen'. It would have to spill something to load 'gen' or = 'th' >> for the test. > Which does not matter on any modern architecture anyway. >=20 >> - it enlarges the window between reading 'scale' and 'delta' and the >> caller seeing the results. Preemption in this window gives results >> that may be far in the past. > My opinion is that quickly exiting the code and avoid retry is more > important (as in performance) than making an impression that we = protect > against preemption. If preemption is important for the caller, then > the calling place must use some measures like interrupt disabling and > re-checking the time after the operation. Preemption can occur after = the > loop exit with the same consequences. >=20 >> The 'get' name is another problem. I would like all the get*time >> functions and not add new names starting with 'get'. The library >> implementation already doesn't bother optimizing the get*time = functions, >> but always uses the hardware timecounter. >>=20 >> getfoo() is a more natural name than foo_get() for the action of = getting >> foo, but the latter is better for consistency, especially in code = that >> puts the subsystem name first in nearby code. >>=20 >> The get*time functions would be better if they were more like >> time_second. Note that time_second is racy if time_t is too larger >> for the arch so that accesses to it are not atomic, as happens on >> 32-bit arches with premature 64-bit time_t. However, in this 32/64 >> case, the race is only run every 136 years, with the next event >> scheduled in 2038, so this race is even less important now than other >> events scheduled in 2038. Bintimes are 96 or 128 bits, so directly >> copying a global like time_second for them would race every 1/2**32 >> second on 2-bit arches or every 1 second on 64-bit arches. Most of >> the loops on the generation count are for fixing these races, but >> perhaps a simpler method would work. On 64-bit arches with atomic >> 64 accesses on 32-bit boundaries, the following would work: >> - set the lower 32 bits of the fraction to 0, or ignore them >> - load the higher 32 bits of the fraction and the lower 32 bits of = the >> seconds >> - race once every 136 years starting in 2038 reading the higher 32 = bits >> of the seconds non-atomically. >> - alternatively, break instead of racing in 2038 by setting the = higher >> 32 bits to 0. This is the same as using sbintimes instead of = bintimes. >> - drop a few more lower bits by storing a right-shifted value. Right >> shifting by just 1 gives a race frequency of once per 272 years, = with >> the next one in 2006. > It would make sense if the functions were written from scratch, but = since > we already have the generation counts, it is not obvious that such = change > is useful. >=20 > But if we decide to go that route (later), my current patch only > requires exactly one location getthmember() to experiment with and to > change after. So you effectively made yet another, and perhaps most > convincing, argument, for me. >=20 >>=20 >>> diff --git a/sys/kern/kern_tc.c b/sys/kern/kern_tc.c >>> index 2656fb4d22f..8d12847f2cd 100644 >>> --- a/sys/kern/kern_tc.c >>> +++ b/sys/kern/kern_tc.c >>> @@ -200,20 +202,56 @@ tc_delta(struct timehands *th) >>> * the comment in for a description of these 12 = functions. >>> */ >>>=20 >>> -#ifdef FFCLOCK >>> -void >>> -fbclock_binuptime(struct bintime *bt) >>> +static __inline void >>> +bintime_off(struct bintime *bt, u_int off) >>> { >>> struct timehands *th; >>> - unsigned int gen; >>> + struct bintime *btp; >>> + uint64_t scale, x; >>> + u_int delta, gen, large_delta; >>>=20 >>> do { >>> th =3D timehands; >>> gen =3D atomic_load_acq_int(&th->th_generation); >>> - *bt =3D th->th_offset; >>> - bintime_addx(bt, th->th_scale * tc_delta(th)); >>=20 >> You didn't fully obfuscate this by combinining this function with >> getthmember() so as to deduplicate the loop. >>=20 >>> + btp =3D (struct bintime *)((vm_offset_t)th + off); >>=20 >> Ugly conversion to share code. This is technically incorrect. = Improving >> the casts gives: >>=20 >> btp =3D (void *)(uintptr_t)((uintptr_t)(void *)th + off); >>=20 >> but this assumes that arithmetic on the intermediate integer does = what >> is espected. uintptr_t is only guaranteed to work when the = intermediate >> representation held in it is not adjusted. > vm_offset_t has the semantic that is needed for the arithmetic. It is > better uintptr_t for kernel, where we know that all object pointers = are > compatible with vm_offset_t (AKA flat tag-less memory model). >=20 >>=20 >> Fixing the API gives >>=20 >> static __inline void >> bintime_off(struct bintime *btp, struct bintime *base_btp) >>=20 >> where base_btp is &th->th_bintime or &th->th_offset. >>=20 >> (th_offset and th_bintime are badly named. th_offset is really a = base >> time and the offset is tc_delta(). th_bintime is also a base time. >> It is the same as th_offset with another actual offset (the = difference >> between UTC and local time) already added to it as an optimization. = In >> old versions, th_bintime didn't exist, but the related struct members >> th_nanotime and th_microtime existed, since these benefit more from >> not converting on every call. > How could it be &th->th_offset, when th is calculated inside the call = ? > But I did modified the API in this spirit, indeed. It takes the = member > name directly as an argument. >=20 >>=20 >> My old version even documents the struct members, while -current = still >> has no comments. The comments were lost to staticization. My = version >> mostly adds "duh" to the banal comments after recovering them: >>=20 >> XX /* >> XX * XXX rotted comment cloned from . >> XX * >> XX * th_counter is undocumented (duh). >> XX * >> XX * th_adjustment [PPM << 16] which means that the smallest unit of = correction >> XX * you can apply amounts to 481.5 usec/year. >> XX * >> XX * th_scale is undocumented (duh). >> XX * >> XX * th_offset_count is the contents of the counter which = corresponds to the >> XX * >> XX * rest of the offset_* values. >> XX * >> XX * th_offset is undocumented (duh). >> XX * >> XX * th_microtime is undocumented (duh). >> XX * >> XX * th_nanotime is undocumented (duh). >> XX * >> XX * XXX especially massive bitrot here. "three" is now "many"... >> XX * Each timecounter must supply an array of three timecounters. = This is needed >> XX * to guarantee atomicity in the code. Index zero is used to = transport >> XX * modifications, for instance done with sysctl, into the = timecounter being >> XX * used in a safe way. Such changes may be adopted with a delay = of up to 1/HZ. >> XX * Index one and two are used alternately for the actual = timekeeping. >> XX * >> XX * th_generation is undocumented (duh). >> XX * >> XX * th_next is undocumented (duh). >> XX */ >>=20 >>> + *bt =3D *btp; >>> + scale =3D th->th_scale; >>> + delta =3D tc_delta(th); >>> + large_delta =3D th->th_large_delta; >>=20 >> I had forgotten that th_scale is so volatile (it may be adjusted on >> every windup). th_large_delta is equally volatile. So moving the >> calculation outside of the loop gives even more register pressure >> than I noticed above. >>=20 >>> atomic_thread_fence_acq(); >>> } while (gen =3D=3D 0 || gen !=3D th->th_generation); >>> + >>> + if (__predict_false(delta < large_delta)) { >>> + /* Avoid overflow for scale * delta. */ >>> + x =3D (scale >> 32) * delta; >>> + bt->sec +=3D x >> 32; >>> + bintime_addx(bt, x << 32); >>> + bintime_addx(bt, (scale & 0xffffffff) * delta); >>> + } else { >>> + bintime_addx(bt, scale * delta); >>> + } >>> +} >>> + >>> +static __inline void >>> +getthmember(void *out, size_t out_size, u_int off) >>> +{ >>> + struct timehands *th; >>> + u_int gen; >>> + >>> + do { >>> + th =3D timehands; >>> + gen =3D atomic_load_acq_int(&th->th_generation); >>> + memcpy(out, (char *)th + off, out_size); >>=20 >> This isn't so ugly or technically incorrect. Now the object is = generic, >> so the reference to it should be passed as (void *objp, size_t = objsize) >> instead of the type-safe (struct bintime *base_bpt). > _Generic is what gave me a hint how to make the implementation = type-safe. > diff --git a/sys/kern/kern_tc.c b/sys/kern/kern_tc.c > index 2656fb4d22f..4e94f762026 100644 > --- a/sys/kern/kern_tc.c > +++ b/sys/kern/kern_tc.c > @@ -72,6 +72,7 @@ struct timehands { > struct timecounter *th_counter; > int64_t th_adjustment; > uint64_t th_scale; > + u_int th_large_delta; > u_int th_offset_count; > struct bintime th_offset; > struct bintime th_bintime; > @@ -90,6 +91,7 @@ static struct timehands th1 =3D { > static struct timehands th0 =3D { > .th_counter =3D &dummy_timecounter, > .th_scale =3D (uint64_t)-1 / 1000000, > + .th_large_delta =3D 1000000, > .th_offset =3D { .sec =3D 1 }, > .th_generation =3D 1, > .th_next =3D &th1 > @@ -200,20 +202,72 @@ tc_delta(struct timehands *th) > * the comment in for a description of these 12 = functions. > */ >=20 > -#ifdef FFCLOCK > -void > -fbclock_binuptime(struct bintime *bt) > +static __inline void > +bintime_off(struct bintime *bt, u_int off) > { > struct timehands *th; > - unsigned int gen; > + struct bintime *btp; > + uint64_t scale, x; > + u_int delta, gen, large_delta; >=20 > do { > th =3D timehands; > gen =3D atomic_load_acq_int(&th->th_generation); > - *bt =3D th->th_offset; > - bintime_addx(bt, th->th_scale * tc_delta(th)); > + btp =3D (struct bintime *)((vm_offset_t)th + off); > + *bt =3D *btp; > + scale =3D th->th_scale; > + delta =3D tc_delta(th); > + large_delta =3D th->th_large_delta; > atomic_thread_fence_acq(); > } while (gen =3D=3D 0 || gen !=3D th->th_generation); > + > + if (__predict_false(delta >=3D large_delta)) { > + /* Avoid overflow for scale * delta. */ > + x =3D (scale >> 32) * delta; > + bt->sec +=3D x >> 32; > + bintime_addx(bt, x << 32); > + bintime_addx(bt, (scale & 0xffffffff) * delta); > + } else { > + bintime_addx(bt, scale * delta); > + } > +} > +#define GETTHBINTIME(dst, member) = \ > +do { = \ > + _Static_assert(_Generic(((struct timehands *)NULL)->member, = \ > + struct bintime: 1, default: 0) =3D=3D 1, = \ > + "struct timehands member is not of struct bintime type"); = \ > + bintime_off(dst, __offsetof(struct timehands, member)); = \ > +} while (0) > + > +static __inline void > +getthmember(void *out, size_t out_size, u_int off) > +{ > + struct timehands *th; > + u_int gen; > + > + do { > + th =3D timehands; > + gen =3D atomic_load_acq_int(&th->th_generation); > + memcpy(out, (char *)th + off, out_size); > + atomic_thread_fence_acq(); > + } while (gen =3D=3D 0 || gen !=3D th->th_generation); > +} > +#define GETTHMEMBER(dst, member) = \ > +do { = \ > + _Static_assert(_Generic(*dst, = \ > + __typeof(((struct timehands *)NULL)->member): 1, = \ > + default: 0) =3D=3D 1, = \ > + "*dst and struct timehands member have different types"); = \ > + getthmember(dst, sizeof(*dst), __offsetof(struct timehands, = \ > + member)); = \ > +} while (0) > + > +#ifdef FFCLOCK > +void > +fbclock_binuptime(struct bintime *bt) > +{ > + > + GETTHBINTIME(bt, th_offset); > } >=20 > void > @@ -237,16 +291,8 @@ fbclock_microuptime(struct timeval *tvp) > void > fbclock_bintime(struct bintime *bt) > { > - struct timehands *th; > - unsigned int gen; >=20 > - do { > - th =3D timehands; > - gen =3D atomic_load_acq_int(&th->th_generation); > - *bt =3D th->th_bintime; > - bintime_addx(bt, th->th_scale * tc_delta(th)); > - atomic_thread_fence_acq(); > - } while (gen =3D=3D 0 || gen !=3D th->th_generation); > + GETTHBINTIME(bt, th_bintime); > } >=20 > void > @@ -270,100 +316,55 @@ fbclock_microtime(struct timeval *tvp) > void > fbclock_getbinuptime(struct bintime *bt) > { > - struct timehands *th; > - unsigned int gen; >=20 > - do { > - th =3D timehands; > - gen =3D atomic_load_acq_int(&th->th_generation); > - *bt =3D th->th_offset; > - atomic_thread_fence_acq(); > - } while (gen =3D=3D 0 || gen !=3D th->th_generation); > + GETTHMEMBER(bt, th_offset); > } >=20 > void > fbclock_getnanouptime(struct timespec *tsp) > { > - struct timehands *th; > - unsigned int gen; > + struct bintime bt; >=20 > - do { > - th =3D timehands; > - gen =3D atomic_load_acq_int(&th->th_generation); > - bintime2timespec(&th->th_offset, tsp); > - atomic_thread_fence_acq(); > - } while (gen =3D=3D 0 || gen !=3D th->th_generation); > + GETTHMEMBER(&bt, th_offset); > + bintime2timespec(&bt, tsp); > } >=20 > void > fbclock_getmicrouptime(struct timeval *tvp) > { > - struct timehands *th; > - unsigned int gen; > + struct bintime bt; >=20 > - do { > - th =3D timehands; > - gen =3D atomic_load_acq_int(&th->th_generation); > - bintime2timeval(&th->th_offset, tvp); > - atomic_thread_fence_acq(); > - } while (gen =3D=3D 0 || gen !=3D th->th_generation); > + GETTHMEMBER(&bt, th_offset); > + bintime2timeval(&bt, tvp); > } >=20 > void > fbclock_getbintime(struct bintime *bt) > { > - struct timehands *th; > - unsigned int gen; >=20 > - do { > - th =3D timehands; > - gen =3D atomic_load_acq_int(&th->th_generation); > - *bt =3D th->th_bintime; > - atomic_thread_fence_acq(); > - } while (gen =3D=3D 0 || gen !=3D th->th_generation); > + GETTHMEMBER(bt, th_bintime); > } >=20 > void > fbclock_getnanotime(struct timespec *tsp) > { > - struct timehands *th; > - unsigned int gen; >=20 > - do { > - th =3D timehands; > - gen =3D atomic_load_acq_int(&th->th_generation); > - *tsp =3D th->th_nanotime; > - atomic_thread_fence_acq(); > - } while (gen =3D=3D 0 || gen !=3D th->th_generation); > + GETTHMEMBER(tsp, th_nanotime); > } >=20 > void > fbclock_getmicrotime(struct timeval *tvp) > { > - struct timehands *th; > - unsigned int gen; >=20 > - do { > - th =3D timehands; > - gen =3D atomic_load_acq_int(&th->th_generation); > - *tvp =3D th->th_microtime; > - atomic_thread_fence_acq(); > - } while (gen =3D=3D 0 || gen !=3D th->th_generation); > + GETTHMEMBER(tvp, th_microtime); > } > #else /* !FFCLOCK */ > + > void > binuptime(struct bintime *bt) > { > - struct timehands *th; > - u_int gen; >=20 > - do { > - th =3D timehands; > - gen =3D atomic_load_acq_int(&th->th_generation); > - *bt =3D th->th_offset; > - bintime_addx(bt, th->th_scale * tc_delta(th)); > - atomic_thread_fence_acq(); > - } while (gen =3D=3D 0 || gen !=3D th->th_generation); > + GETTHBINTIME(bt, th_offset); > } >=20 > void > @@ -387,16 +388,8 @@ microuptime(struct timeval *tvp) > void > bintime(struct bintime *bt) > { > - struct timehands *th; > - u_int gen; >=20 > - do { > - th =3D timehands; > - gen =3D atomic_load_acq_int(&th->th_generation); > - *bt =3D th->th_bintime; > - bintime_addx(bt, th->th_scale * tc_delta(th)); > - atomic_thread_fence_acq(); > - } while (gen =3D=3D 0 || gen !=3D th->th_generation); > + GETTHBINTIME(bt, th_bintime); > } >=20 > void > @@ -420,85 +413,47 @@ microtime(struct timeval *tvp) > void > getbinuptime(struct bintime *bt) > { > - struct timehands *th; > - u_int gen; >=20 > - do { > - th =3D timehands; > - gen =3D atomic_load_acq_int(&th->th_generation); > - *bt =3D th->th_offset; > - atomic_thread_fence_acq(); > - } while (gen =3D=3D 0 || gen !=3D th->th_generation); > + GETTHMEMBER(bt, th_offset); > } >=20 > void > getnanouptime(struct timespec *tsp) > { > - struct timehands *th; > - u_int gen; > + struct bintime bt; >=20 > - do { > - th =3D timehands; > - gen =3D atomic_load_acq_int(&th->th_generation); > - bintime2timespec(&th->th_offset, tsp); > - atomic_thread_fence_acq(); > - } while (gen =3D=3D 0 || gen !=3D th->th_generation); > + GETTHMEMBER(&bt, th_offset); > + bintime2timespec(&bt, tsp); > } >=20 > void > getmicrouptime(struct timeval *tvp) > { > - struct timehands *th; > - u_int gen; > + struct bintime bt; >=20 > - do { > - th =3D timehands; > - gen =3D atomic_load_acq_int(&th->th_generation); > - bintime2timeval(&th->th_offset, tvp); > - atomic_thread_fence_acq(); > - } while (gen =3D=3D 0 || gen !=3D th->th_generation); > + GETTHMEMBER(&bt, th_offset); > + bintime2timeval(&bt, tvp); > } >=20 > void > getbintime(struct bintime *bt) > { > - struct timehands *th; > - u_int gen; >=20 > - do { > - th =3D timehands; > - gen =3D atomic_load_acq_int(&th->th_generation); > - *bt =3D th->th_bintime; > - atomic_thread_fence_acq(); > - } while (gen =3D=3D 0 || gen !=3D th->th_generation); > + GETTHMEMBER(bt, th_bintime); > } >=20 > void > getnanotime(struct timespec *tsp) > { > - struct timehands *th; > - u_int gen; >=20 > - do { > - th =3D timehands; > - gen =3D atomic_load_acq_int(&th->th_generation); > - *tsp =3D th->th_nanotime; > - atomic_thread_fence_acq(); > - } while (gen =3D=3D 0 || gen !=3D th->th_generation); > + GETTHMEMBER(tsp, th_nanotime); > } >=20 > void > getmicrotime(struct timeval *tvp) > { > - struct timehands *th; > - u_int gen; >=20 > - do { > - th =3D timehands; > - gen =3D atomic_load_acq_int(&th->th_generation); > - *tvp =3D th->th_microtime; > - atomic_thread_fence_acq(); > - } while (gen =3D=3D 0 || gen !=3D th->th_generation); > + GETTHMEMBER(tvp, th_microtime); > } > #endif /* FFCLOCK */ >=20 > @@ -514,15 +469,8 @@ getboottime(struct timeval *boottime) > void > getboottimebin(struct bintime *boottimebin) > { > - struct timehands *th; > - u_int gen; >=20 > - do { > - th =3D timehands; > - gen =3D atomic_load_acq_int(&th->th_generation); > - *boottimebin =3D th->th_boottime; > - atomic_thread_fence_acq(); > - } while (gen =3D=3D 0 || gen !=3D th->th_generation); > + GETTHMEMBER(boottimebin, th_boottime); > } >=20 > #ifdef FFCLOCK > @@ -1038,15 +986,8 @@ getmicrotime(struct timeval *tvp) > void > dtrace_getnanotime(struct timespec *tsp) > { > - struct timehands *th; > - u_int gen; >=20 > - do { > - th =3D timehands; > - gen =3D atomic_load_acq_int(&th->th_generation); > - *tsp =3D th->th_nanotime; > - atomic_thread_fence_acq(); > - } while (gen =3D=3D 0 || gen !=3D th->th_generation); > + GETTHMEMBER(tsp, th_nanotime); > } >=20 > /* > @@ -1464,6 +1405,7 @@ tc_windup(struct bintime *new_boottimebin) > scale +=3D (th->th_adjustment / 1024) * 2199; > scale /=3D th->th_counter->tc_frequency; > th->th_scale =3D scale * 2; > + th->th_large_delta =3D MIN(((uint64_t)1 << 63) / scale, = UINT_MAX); >=20 > /* > * Now that the struct timehands is again consistent, set the = new > _______________________________________________ > freebsd-ppc@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-ppc > To unsubscribe, send any mail to "freebsd-ppc-unsubscribe@freebsd.org" --Apple-Mail=_A26B82D4-1AFC-4AF8-A85C-EB10136E809F Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCEJAw ggTVMIIDvaADAgECAghQTsb1PRG0ZDANBgkqhkiG9w0BAQsFADBxMQswCQYDVQQGEwJERTEcMBoG A1UEChMTRGV1dHNjaGUgVGVsZWtvbSBBRzEfMB0GA1UECxMWVC1UZWxlU2VjIFRydXN0IENlbnRl cjEjMCEGA1UEAxMaRGV1dHNjaGUgVGVsZWtvbSBSb290IENBIDIwHhcNMTQwNzIyMTIwODI2WhcN MTkwNzA5MjM1OTAwWjBaMQswCQYDVQQGEwJERTETMBEGA1UEChMKREZOLVZlcmVpbjEQMA4GA1UE CxMHREZOLVBLSTEkMCIGA1UEAxMbREZOLVZlcmVpbiBQQ0EgR2xvYmFsIC0gRzAxMIIBIjANBgkq hkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA6ZvDZ4X5Da71jVTDllA1PWLpbkztlNcAW5UidNQg6zSP 1uzAMQQLmYHiphTSUqAoI4SLdIkEXlvg4njBeMsWyyg1OXstkEXQ7aAAeny/Sg4bAMOG6VwrMRF7 DPOCJEOMHDiLamgAmu7cT3ir0sYTm3at7t4m6O8Br3QPwQmi9mvOvdPNFDBP9eXjpMhim4IaAycw DQJlYE3t0QkjKpY1WCfTdsZxtpAdxO3/NYZ9bzOz2w/FEcKKg6GUXUFr2NIQ9Uz9ylGs2b3vkoO7 2uuLFlZWQ8/h1RM9ph8nMM1JVNvJEzSacXXFbOqnC5j5IZ0nrz6jOTlIaoytyZn7wxLyvQIDAQAB o4IBhjCCAYIwDgYDVR0PAQH/BAQDAgEGMB0GA1UdDgQWBBRJt8bP6D0ff+pEexMp9/EKcD7eZDAf BgNVHSMEGDAWgBQxw3kbuvVT1xfgiXotF2wKsyudMzASBgNVHRMBAf8ECDAGAQH/AgECMGIGA1Ud IARbMFkwEQYPKwYBBAGBrSGCLAEBBAICMBEGDysGAQQBga0hgiwBAQQDADARBg8rBgEEAYGtIYIs AQEEAwEwDwYNKwYBBAGBrSGCLAEBBDANBgsrBgEEAYGtIYIsHjA+BgNVHR8ENzA1MDOgMaAvhi1o dHRwOi8vcGtpMDMzNi50ZWxlc2VjLmRlL3JsL0RUX1JPT1RfQ0FfMi5jcmwweAYIKwYBBQUHAQEE bDBqMCwGCCsGAQUFBzABhiBodHRwOi8vb2NzcDAzMzYudGVsZXNlYy5kZS9vY3NwcjA6BggrBgEF BQcwAoYuaHR0cDovL3BraTAzMzYudGVsZXNlYy5kZS9jcnQvRFRfUk9PVF9DQV8yLmNlcjANBgkq hkiG9w0BAQsFAAOCAQEAYyAo/ZwhhnK+OUZZOTIlvKkBmw3Myn1BnIZtCm4ssxNZdbEzkhthJxb/ w7LVNYL7hCoBSb1mu2YvssIGXW4/buMBWlvKQ2NclbbhMacf1QdfTeZlgk4y+cN8ekvNTVx07iHy dQLsUj7SyWrTkCNuSWc1vn9NVqTszC/Pt6GXqHI+ybxA1lqkCD3WvILDt7cyjrEsjmpttzUCGc/1 OURYY6ckABCwu/xOr24vOLulV0k/2G5QbyyXltwdRpplic+uzPLl2Z9Tsz6hL5Kp2AvGhB8Exuse 6J99tXulAvEkxSRjETTMWpMgKnmIOiVCkKllO3yG0xIVIyn8LNrMOVtUFzCCBaIwggSKoAMCAQIC BxekJKEJSDMwDQYJKoZIhvcNAQELBQAwWjELMAkGA1UEBhMCREUxEzARBgNVBAoTCkRGTi1WZXJl aW4xEDAOBgNVBAsTB0RGTi1QS0kxJDAiBgNVBAMTG0RGTi1WZXJlaW4gUENBIEdsb2JhbCAtIEcw MTAeFw0xNDA1MjcxNDU0MDlaFw0xOTA3MDkyMzU5MDBaMIHGMQswCQYDVQQGEwJERTEcMBoGA1UE CBMTTm9yZHJoZWluLVdlc3RmYWxlbjERMA8GA1UEBxMITXVlbnN0ZXIxIDAeBgNVBAoTF0ZhY2ho b2Noc2NodWxlIE11ZW5zdGVyMSMwIQYDVQQLExpEYXRlbnZlcmFyYmVpdHVuZ3N6ZW50cmFsZTEd MBsGA1UEAxMURkggTXVlbnN0ZXIgQ0EgLSBHMDExIDAeBgkqhkiG9w0BCQEWEWNhQGZoLW11ZW5z dGVyLmRlMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuHlsrvBs7CL9IqMH9r//QU9E pghTV/3skHuQZ3DpNY+lyJWOW5zbtUubgXt7lYHpIE4d4CclTZWqCHwoAI6gqzSSGjUKuX6/0ui/ LhXmlDvCBfwuER+T+3/R59hlLnhI5iYYPQiNywQIa3wJhBLTZrlXw8nDdjI54MAzcVDUX7l21sbo ZIA6idM7SXmshxoRQ6xsfPHskrceNMcvtHNDhVnVscwRUJQUR55fs0X7Y93PasugWPv3xmgNr1da Cq94eV+nslNU/GJaT9TQ3uG8pagLXl9NbDNkHIrvFAD5zXO0m/d00I4QhUVQyEtwnTegDqcM+WFh JXensgnZhWe6bwIDAQABo4IB/jCCAfowEgYDVR0TAQH/BAgwBgEB/wIBATAOBgNVHQ8BAf8EBAMC AQYwEQYDVR0gBAowCDAGBgRVHSAAMB0GA1UdDgQWBBQK81u85DGA1jVCiabTw8833tHf1zAfBgNV HSMEGDAWgBRJt8bP6D0ff+pEexMp9/EKcD7eZDAcBgNVHREEFTATgRFjYUBmaC1tdWVuc3Rlci5k ZTCBiAYDVR0fBIGAMH4wPaA7oDmGN2h0dHA6Ly9jZHAxLnBjYS5kZm4uZGUvZ2xvYmFsLXJvb3Qt Y2EvcHViL2NybC9jYWNybC5jcmwwPaA7oDmGN2h0dHA6Ly9jZHAyLnBjYS5kZm4uZGUvZ2xvYmFs LXJvb3QtY2EvcHViL2NybC9jYWNybC5jcmwwgdcGCCsGAQUFBwEBBIHKMIHHMDMGCCsGAQUFBzAB hidodHRwOi8vb2NzcC5wY2EuZGZuLmRlL09DU1AtU2VydmVyL09DU1AwRwYIKwYBBQUHMAKGO2h0 dHA6Ly9jZHAxLnBjYS5kZm4uZGUvZ2xvYmFsLXJvb3QtY2EvcHViL2NhY2VydC9jYWNlcnQuY3J0 MEcGCCsGAQUFBzAChjtodHRwOi8vY2RwMi5wY2EuZGZuLmRlL2dsb2JhbC1yb290LWNhL3B1Yi9j YWNlcnQvY2FjZXJ0LmNydDANBgkqhkiG9w0BAQsFAAOCAQEA3kcDNdZKb7kSD7s1ly2qa/2QbQe+ ld3LhZeOcfysdLtN8oweBmgT3MYoZ+D9c+SoUWJAwTKPB15DoGy+fWhelXTpQrqxIGb4ISr1JCjg slnmMUva0xjwZGxojZ9gE1bi18xfKw3+dMpwCLt6LbLTjr/tyH6otacwr2tZzuuJIUAORnefwTcr vmB21n/BEQH/ZXruWu8lSO3L9YAmQB6ViaZFCpn2sMmOLACdoWxmUQb3QAjsa327jHUjsz53k9q5 Zrx/g+zOg5s1Wmy2JOlLQMUIZXXf0/6rB5Fr2llx7dBG/Uk7NhZdNy7OzNzci0C4Wnkd8rDVEWHG hH2gfpcTfjCCBg0wggT1oAMCAQICBxuZiHQ3saMwDQYJKoZIhvcNAQELBQAwgcYxCzAJBgNVBAYT AkRFMRwwGgYDVQQIExNOb3JkcmhlaW4tV2VzdGZhbGVuMREwDwYDVQQHEwhNdWVuc3RlcjEgMB4G A1UEChMXRmFjaGhvY2hzY2h1bGUgTXVlbnN0ZXIxIzAhBgNVBAsTGkRhdGVudmVyYXJiZWl0dW5n c3plbnRyYWxlMR0wGwYDVQQDExRGSCBNdWVuc3RlciBDQSAtIEcwMTEgMB4GCSqGSIb3DQEJARYR Y2FAZmgtbXVlbnN0ZXIuZGUwHhcNMTYwNzA0MDcwNjEzWhcNMTkwNzA0MDcwNjEzWjB8MQswCQYD VQQGEwJERTEgMB4GA1UECgwXRmFjaGhvY2hzY2h1bGUgTXVlbnN0ZXIxMjAwBgNVBAsMKUZhY2hi ZXJlaWNoIEVsZWt0cm90ZWNobmlrIHVuZCBJbmZvcm1hdGlrMRcwFQYDVQQDDA5NaWNoYWVsIFR1 ZXhlbjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMyaGlBt2ZtuF8QP8zYNrGxXC+es PMajIPl+hu1LGHnN2BJ3J5ZMN44BOZw3n6LO1FaAgO8D4xU4/AELecX6VxJZ2zOOSD8uTYO4OnUu 24hkjFUQAj13tT644AKUQMMBpgj7wC52V5Jij+mZX/t1S38/WFiCGnirt4xTNi5OmN4K+VNZfG4x 0msDqFjJX70rF1y09/Mylu1M/Y0tu/I9DqhwDQT4LBOvyyaAlhSJ8Jb8m8YTt5xlOzrXlBmj4pKs 74y7C2IKRw4tFozGX1cf1LVEs2eBCb5iUwXrlcMipwm62sJ38GD00EOlRNTpAM5rDAcgWxMCffek bRv/01whtOkCAwEAAaOCAkcwggJDMEAGA1UdIAQ5MDcwEQYPKwYBBAGBrSGCLAEBBAMFMBEGDysG AQQBga0hgiwCAQQDATAPBg0rBgEEAYGtIYIsAQEEMAkGA1UdEwQCMAAwDgYDVR0PAQH/BAQDAgXg MB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDBDAdBgNVHQ4EFgQU0B2vaoSoEmYAggD04WZF 2hGif3UwHwYDVR0jBBgwFoAUCvNbvOQxgNY1Qomm08PPN97R39cwIAYDVR0RBBkwF4EVdHVleGVu QGZoLW11ZW5zdGVyLmRlMIGIBgNVHR8EgYAwfjA9oDugOYY3aHR0cDovL2NkcDEucGNhLmRmbi5k ZS9maC1tdWVuc3Rlci1jYS9wdWIvY3JsL2NhY3JsLmNybDA9oDugOYY3aHR0cDovL2NkcDIucGNh LmRmbi5kZS9maC1tdWVuc3Rlci1jYS9wdWIvY3JsL2NhY3JsLmNybDCB1wYIKwYBBQUHAQEEgcow gccwMwYIKwYBBQUHMAGGJ2h0dHA6Ly9vY3NwLnBjYS5kZm4uZGUvT0NTUC1TZXJ2ZXIvT0NTUDBH BggrBgEFBQcwAoY7aHR0cDovL2NkcDEucGNhLmRmbi5kZS9maC1tdWVuc3Rlci1jYS9wdWIvY2Fj ZXJ0L2NhY2VydC5jcnQwRwYIKwYBBQUHMAKGO2h0dHA6Ly9jZHAyLnBjYS5kZm4uZGUvZmgtbXVl bnN0ZXItY2EvcHViL2NhY2VydC9jYWNlcnQuY3J0MA0GCSqGSIb3DQEBCwUAA4IBAQBI9v+seJM6 AlSIrmmpopz6zh8QAsqGLJkkY2D0KYFucUY/xZaJTtZxvmWddbKk2903Qhg+vZKOf87PHhip7/4t FSwhxYNSS36WsRJTeUa0f3KkSa28yrIRfWlJATgxfL5X/QQnopjCt34n4221kcsR7LHxBAn37ow+ /2L7WjWDDuOkaM9/ZSCtrN+yFRat1eUVs1Hk7sKT/bfJTsYqzovXitjmCP3YdB40dkuQ6/ZzEdXT bpa4c45RcRnPqKXnxknK0UfRHNHqk15W7dUPVMzSGFUvjhmWPP2wW6a8F1U5sEqfHcoBFC5CGjGy 7Gk2luk3obi/KLrDyZC+dkjhDYEpMYIEOTCCBDUCAQEwgdIwgcYxCzAJBgNVBAYTAkRFMRwwGgYD VQQIExNOb3JkcmhlaW4tV2VzdGZhbGVuMREwDwYDVQQHEwhNdWVuc3RlcjEgMB4GA1UEChMXRmFj aGhvY2hzY2h1bGUgTXVlbnN0ZXIxIzAhBgNVBAsTGkRhdGVudmVyYXJiZWl0dW5nc3plbnRyYWxl MR0wGwYDVQQDExRGSCBNdWVuc3RlciBDQSAtIEcwMTEgMB4GCSqGSIb3DQEJARYRY2FAZmgtbXVl bnN0ZXIuZGUCBxuZiHQ3saMwDQYJYIZIAWUDBAIBBQCgggI3MBgGCSqGSIb3DQEJAzELBgkqhkiG 9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTE5MDQwMjIyMjczMlowLwYJKoZIhvcNAQkEMSIEIGSIIzmA QlMYcCWNiwVDItNMbp2U1LppxYebTqP1GV1zMIHjBgkrBgEEAYI3EAQxgdUwgdIwgcYxCzAJBgNV BAYTAkRFMRwwGgYDVQQIExNOb3JkcmhlaW4tV2VzdGZhbGVuMREwDwYDVQQHEwhNdWVuc3RlcjEg MB4GA1UEChMXRmFjaGhvY2hzY2h1bGUgTXVlbnN0ZXIxIzAhBgNVBAsTGkRhdGVudmVyYXJiZWl0 dW5nc3plbnRyYWxlMR0wGwYDVQQDExRGSCBNdWVuc3RlciBDQSAtIEcwMTEgMB4GCSqGSIb3DQEJ ARYRY2FAZmgtbXVlbnN0ZXIuZGUCBxuZiHQ3saMwgeUGCyqGSIb3DQEJEAILMYHVoIHSMIHGMQsw CQYDVQQGEwJERTEcMBoGA1UECBMTTm9yZHJoZWluLVdlc3RmYWxlbjERMA8GA1UEBxMITXVlbnN0 ZXIxIDAeBgNVBAoTF0ZhY2hob2Noc2NodWxlIE11ZW5zdGVyMSMwIQYDVQQLExpEYXRlbnZlcmFy YmVpdHVuZ3N6ZW50cmFsZTEdMBsGA1UEAxMURkggTXVlbnN0ZXIgQ0EgLSBHMDExIDAeBgkqhkiG 9w0BCQEWEWNhQGZoLW11ZW5zdGVyLmRlAgcbmYh0N7GjMA0GCSqGSIb3DQEBAQUABIIBAIZaQcal 2J/msfopGijjkS2+107hVANwLAQ6CiD93XD/ijJOkTZQLjqzRmSFAFMQGw451uj11sla6SlLLFYI QR3GJEVfyykRs31RS3HAbU8b96QiB4AYCWStAbulwWbJFHTdKZJsZEc4oHCA3OboIw7gjd7c89PF 3siE2nHFJWUlUevrMqg+xSaqkXV1eOhC8l0foNqzlSYyhYw3wdB3mpZS2kDQ6d5VVO0qskqy5Ua9 RW457EF9AXcW3sCJ6k/B6p78uUUkEkNgBZbRkIykqu/VACIffur+2qF49sKEV4yjhSxFNtJKtqVM RZNPcwhMe446prrHZbWU3ThPedtIv8QAAAAAAAA= --Apple-Mail=_A26B82D4-1AFC-4AF8-A85C-EB10136E809F-- From owner-freebsd-ppc@freebsd.org Wed Apr 3 00:26:10 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A3F0315544C0 for ; Wed, 3 Apr 2019 00:26:10 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic314-20.consmr.mail.gq1.yahoo.com (sonic314-20.consmr.mail.gq1.yahoo.com [98.137.69.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BBBC387745 for ; Wed, 3 Apr 2019 00:26:09 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: 7LTWA5QVM1kW3Lb6rCmlym1zAVjEOf5kpfujlRwpU98uzMbUR0dXozL.eCkTYb2 Hg52w2GaMFi_5gRcaQmEEjYYttGg5p1nML0LLGf_EYr64Zasv8oJ.9rI_XiV89RMxYuJ3.bldXxg 5ftaW1fdUkwr36p2Xbbd7_ajVNDzkM7ilrfkfHEHSeuBDWXLgJnpwq3nZ0J_v3tNHES425yfCx6j m2FOrCX3e.pcd9wY0PUEGz9CnAeVTOwuy7rXUUhQ5uMmw5aP7ueOd7ds7qGFLV06F3KiOnle9dgo mz2AGPeF_Co70RlQ5MU8MFGm0eLlCLtqwaZaZVUzAmEOg5F2MzRjlwsde6y8ZnKyIqMMsgYgIAOj uBaOOPWykAuyTzoMzvs3J1AtGwSWMI0zaLZAYynUt_6tbUomxg9zFZPzxwUwibcJdaDr.6w485td CfZmYSHMtOTUIO9DuoeZKzfgwKhwuuA7gOiIMtZ_tWg4ECbun7GAXCPiUUP8L2HTNNSBhlM8O3g8 PuULbzP9UOCMT0MOnwKs4p7ZyHeKVOWFwgoIY051_XwDC6TIA1vczy3m6RdfAK3EOP0KYEgHFGor yMqkAn4HK_5sUoxodMWR1Ut1bZOhBRXXLeM5iKfhj1moqMP.b52TsU4w80VeLZpbtIIObcn1rxVT smhSGpu1erFRgh80sqAYMRXTarEBxJJsJLU36kue9ZmdiM.x73RE13LR5jd9aZ02QgFKGu38AgFO RGAKjf6fxg0cpg4cfF8Ug6Ig8cSV3fCzGEnL.qfRPffVwcrm1FbMSk3NMQ9ibGTt4byQliZfQaJA i.ABdYU7ofx_njt01iaVl3Fbm6SQTCSzc9Z_rETVZplXkDoqdGwtGZnJmOo6P97D5Vr9vcYqH1eX YIu1XlAvj3uuiubavTH67sPvb37yW1fqnrJdvnsHUC36Z417pAQ7F3J66yppNHtm5.ZKjBvBIMvP nmz43X9qfeZr.3h7W0oBxX0LEyjHnGBJRHAQGSnzBE.MJihHobBYZqZhblYGD37ZbgcFjLk6OJM_ 1p9MyyBPrlEHMlGBi7UgOelX7GzuZ8oBZlHLX2NYuAlfsJEMsAL3qnh9riJukBnBrYOF.SVK3N3d A Received: from sonic.gate.mail.ne1.yahoo.com by sonic314.consmr.mail.gq1.yahoo.com with HTTP; Wed, 3 Apr 2019 00:26:07 +0000 Received: from c-67-170-167-181.hsd1.or.comcast.net (EHLO [192.168.1.113]) ([67.170.167.181]) by smtp424.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 935369d1782ce17aa08491c035fe8ef3; Wed, 03 Apr 2019 00:15:57 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed] From: Mark Millard In-Reply-To: Date: Tue, 2 Apr 2019 17:15:56 -0700 Cc: Konstantin Belousov , freebsd-hackers Hackers , Bruce Evans , FreeBSD PowerPC ML Content-Transfer-Encoding: quoted-printable Message-Id: <70B2E795-1FCC-41A3-8F4D-6E6C97379116@yahoo.com> References: <20190303111931.GI68879@kib.kiev.ua> <20190303223100.B3572@besplex.bde.org> <20190303161635.GJ68879@kib.kiev.ua> <20190304043416.V5640@besplex.bde.org> <20190304114150.GM68879@kib.kiev.ua> <20190305031010.I4610@besplex.bde.org> <20190306172003.GD2492@kib.kiev.ua> <20190308001005.M2756@besplex.bde.org> <20190307222220.GK2492@kib.kiev.ua> <20190309144844.K1166@besplex.bde.org> <20190324110138.GR1923@kib.kiev.ua> To: Michael Tuexen X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: BBBC387745 X-Spamd-Bar: +++ X-Spamd-Result: default: False [3.30 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_SPAM_SHORT(0.68)[0.677,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(1.90)[ip: (7.88), ipnet: 98.137.64.0/21(0.95), asn: 36647(0.76), country: US(-0.06)]; NEURAL_SPAM_MEDIUM(0.53)[0.529,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.70)[0.700,0]; RCVD_IN_DNSWL_NONE(0.00)[83.69.137.98.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_POSSIBLE(0.00)[83.69.137.98.rep.mailspike.net : 127.0.0.17]; FREEMAIL_CC(0.00)[gmail.com] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Apr 2019 00:26:10 -0000 On 2019-Apr-2, at 15:27, Michael Tuexen = wrote: >> On 24. Mar 2019, at 12:01, Konstantin Belousov wrote: >>=20 >> On Sat, Mar 09, 2019 at 06:00:14PM +1100, Bruce Evans wrote: >>> I more strongly disclike (sic) the more complete merge. The central = APIs >>> have even more parameters and reduced type safety to describe = objects as >>> (offset, size) pairs. >> I changed the patch to be type-safe. Now I like it even more. It = provides >> 1. internal >> 2. concise >> 3. type-safe >> API to fetch data from timehands. The implementation needs to be = read >> only once. > Hi, >=20 > I'm a bit lost... I think this started to fix a problem on G5 = PowerMacs. > Do you think this patch solves the problem. Should this be tested? > Or is this still work in progress or a general improvement not = necessary > fixing the problem on G5 PowerMacs? >=20 > Best regards > Michael > . . . A much earlier version of the patch made things much worse on the PowerMc G5 (massively long hangups, inability to effectively enter commands). I had to cut power, repair the file system, and undo the separately (not booted from that media). I've not tried again since then. I suspect that while my report prompted the looking into the code, the issues noticed are likely not fixes to the behavior observed on the old PowerMac G5's. I've my own hack/workaround that has my context operational, avoiding the original issue. Definitely not what FreeBSD should be doing. It even involves constants figured out by observation of the failing conditions in the specific context. =46rom what I've seen, my detailed hack-text will not apply to the new code. I'll likely have to work out a variant until the original problem and a solution is identified and provided. Note: The only PowerPC contexts that I've access to are old PowerMacs. I've no clue if other contexts have the original issue or not. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-ppc@freebsd.org Wed Apr 3 07:01:00 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4E58D1560354; Wed, 3 Apr 2019 07:01:00 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6BEA09621A; Wed, 3 Apr 2019 07:00:59 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id x3370lKJ077099 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 3 Apr 2019 10:00:50 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua x3370lKJ077099 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id x3370j6a077098; Wed, 3 Apr 2019 10:00:45 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 3 Apr 2019 10:00:45 +0300 From: Konstantin Belousov To: Michael Tuexen Cc: Bruce Evans , freebsd-hackers Hackers , FreeBSD PowerPC ML Subject: Re: powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed] Message-ID: <20190403070045.GW1923@kib.kiev.ua> References: <20190303161635.GJ68879@kib.kiev.ua> <20190304043416.V5640@besplex.bde.org> <20190304114150.GM68879@kib.kiev.ua> <20190305031010.I4610@besplex.bde.org> <20190306172003.GD2492@kib.kiev.ua> <20190308001005.M2756@besplex.bde.org> <20190307222220.GK2492@kib.kiev.ua> <20190309144844.K1166@besplex.bde.org> <20190324110138.GR1923@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.11.4 (2019-03-13) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Apr 2019 07:01:00 -0000 On Wed, Apr 03, 2019 at 12:27:32AM +0200, Michael Tuexen wrote: > > On 24. Mar 2019, at 12:01, Konstantin Belousov wrote: > > > > On Sat, Mar 09, 2019 at 06:00:14PM +1100, Bruce Evans wrote: > >> I more strongly disclike (sic) the more complete merge. The central APIs > >> have even more parameters and reduced type safety to describe objects as > >> (offset, size) pairs. > > I changed the patch to be type-safe. Now I like it even more. It provides > > 1. internal > > 2. concise > > 3. type-safe > > API to fetch data from timehands. The implementation needs to be read > > only once. > Hi, > > I'm a bit lost... I think this started to fix a problem on G5 PowerMacs. > Do you think this patch solves the problem. Should this be tested? > Or is this still work in progress or a general improvement not necessary > fixing the problem on G5 PowerMacs? It started from a report of issues on G5. The specific issues are bugs on G5, and the posted patches do not fix them. From what I see, the timecounter values were wrapped. This is genuine ppc or G5 issue. The patch fixes time keeping subsystem reaction to the already bad situation, by correctly handling overflow in calculations. This overflow can occur in more reasonable setups as well, e.g. if ddb was activated and interrupts were stopped for prolonged period, even on x86. In addition, my version of the patch reorganizes the code and removes excessive copies of the most delicate loops in lock-less readers. This chunk can be split from the overflow part, but it is not completely trivial. From owner-freebsd-ppc@freebsd.org Wed Apr 3 15:47:41 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E55281571B57; Wed, 3 Apr 2019 15:47:40 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail104.syd.optusnet.com.au (mail104.syd.optusnet.com.au [211.29.132.246]) by mx1.freebsd.org (Postfix) with ESMTP id 2AEBD87E7C; Wed, 3 Apr 2019 15:47:39 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from [192.168.0.102] (c110-21-101-228.carlnfd1.nsw.optusnet.com.au [110.21.101.228]) by mail104.syd.optusnet.com.au (Postfix) with ESMTPS id A9BCF4387D6; Thu, 4 Apr 2019 02:47:35 +1100 (AEDT) Date: Thu, 4 Apr 2019 02:47:34 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Konstantin Belousov cc: Michael Tuexen , Bruce Evans , freebsd-hackers Hackers , FreeBSD PowerPC ML Subject: Re: powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed] In-Reply-To: <20190403070045.GW1923@kib.kiev.ua> Message-ID: <20190404011802.E2390@besplex.bde.org> References: <20190303161635.GJ68879@kib.kiev.ua> <20190304043416.V5640@besplex.bde.org> <20190304114150.GM68879@kib.kiev.ua> <20190305031010.I4610@besplex.bde.org> <20190306172003.GD2492@kib.kiev.ua> <20190308001005.M2756@besplex.bde.org> <20190307222220.GK2492@kib.kiev.ua> <20190309144844.K1166@besplex.bde.org> <20190324110138.GR1923@kib.kiev.ua> <20190403070045.GW1923@kib.kiev.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.2 cv=P6RKvmIu c=1 sm=1 tr=0 cx=a_idp_d a=PalzARQSbocsUSjMRkwAPg==:117 a=PalzARQSbocsUSjMRkwAPg==:17 a=jpOVt7BSZ2e4Z31A5e1TngXxSK0=:19 a=kj9zAlcOel0A:10 a=pGLkceISAAAA:8 a=82MzWkLtsjH0Vdtkr0wA:9 a=bndD0-KsWL0R_FZ2:21 a=4Wq7EeSnkzghBKyU:21 a=CjuIK1q_8ugA:10 X-Rspamd-Queue-Id: 2AEBD87E7C X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.96 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.96)[-0.960,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Apr 2019 15:47:41 -0000 On Wed, 3 Apr 2019, Konstantin Belousov wrote: > On Wed, Apr 03, 2019 at 12:27:32AM +0200, Michael Tuexen wrote: >>> On 24. Mar 2019, at 12:01, Konstantin Belousov wrote: >>> >>> On Sat, Mar 09, 2019 at 06:00:14PM +1100, Bruce Evans wrote: >>>> I more strongly disclike (sic) the more complete merge. The central APIs >>>> have even more parameters and reduced type safety to describe objects as >>>> (offset, size) pairs. >>> I changed the patch to be type-safe. Now I like it even more. It provides >>> 1. internal >>> 2. concise >>> 3. type-safe >>> API to fetch data from timehands. The implementation needs to be read >>> only once. >> Hi, >> >> I'm a bit lost... I think this started to fix a problem on G5 PowerMacs. >> Do you think this patch solves the problem. Should this be tested? >> Or is this still work in progress or a general improvement not necessary >> fixing the problem on G5 PowerMacs? > > It started from a report of issues on G5. The specific issues are > bugs on G5, and the posted patches do not fix them. From what I see, > the timecounter values were wrapped. This is genuine ppc or G5 issue. I am still unhappy with the patch (but haven't got around to replying to the latest version). It is overengineered and doesn't fix the original problem. > The patch fixes time keeping subsystem reaction to the already > bad situation, by correctly handling overflow in calculations. This > overflow can occur in more reasonable setups as well, e.g. if ddb was > activated and interrupts were stopped for prolonged period, even on x86. I have noticed that it mostly fixes this. > In addition, my version of the patch reorganizes the code and removes > excessive copies of the most delicate loops in lock-less readers. This > chunk can be split from the overflow part, but it is not completely > trivial. This is the part that I don't like. I noticed (or better realized) a general problem with multiple timehands. ntpd can slew the clock at up to 500 ppm, and at least an old version of it uses a rate of 50 ppm to fix up fairly small drifts in the milliseconds range. 500 ppm is enormous in CPU cycles -- it is 500 thousand nsec or 2 million cycles at 4GHz. Winding up the timecounter every 1 msec reduces this to only 2000 cycles. More details of ordering and timing for 1 thread: - thread N calls binuptime() and it loads timehands - another or even the same thread runs tc_windup(). This modifies timehands. - thread N is preempted for a long time, but less than the time for updates - thread N checks the generation count. Since this is for the timehands contents and not for the timehands pointer, it hasn't changed, so the old timehands is used - and instant later, the same thread calls binuptime again() and uses the new timehands - now suppose only 2 timehands (as in -current) the worst (?) case of a slew of 500 ppm for the old timehands and -500 ppm for the new timehands and almost the worst case of 10 msec for the oldness of the old timehands relative to the new timehands, with the new timehands about to be updated after 10 msec (assume perfectly periodiodic updates every 10 msec). The calculated times are: old bintime = old_base + (20 msec) * (1 + 500e-6) new base = old_base + 10 msec * (1 + 500e-6) # calc by tc_windup() new bintime = new_base + (10 msec) * (1 - 500e-6) + epsilon error = epsilon - (20 msec) * 500e-6 = epsilon - 10000 nsec Errors in the negative direction are most harmful. ntpd normally doesn't change the skew as much as that in one step, but it is easy for adjtime(2) to change the skew like that and there are no reasonable microadjustments that would accidentally work around this kernel bug (it seems unreasonable to limit the skew to 1 ppm and that still gives an error of epsilon + 20 nsec. phk didn't want to slow down timecounters using extra code to make them them monotonic and coherent (getbinuptime() is incoherent with binuptime() since it former lags the latter by the update interval), but this wouldn't be very slow within a thread. Monotonicity across threads is a larger problem and not helped by using a faked forced monotonic time within threads. So it seems best to fix the above problem by moving the generation count from the timehands contents to the timehands pointer, and maybe also reduce the number of timehands to 1. With 2 timehands, this gives a shorter race: - thread N loads timehands - tc_windup() - thread N preempted - thread N uses old timehands - case tc_windup() completes first: no problem -- thread N checks the generation count on the pointer and loops - case binuptime() completes first: lost a race -- binuptime() is off by approx * . The main point of having multiple timehands (after introducing the per- timehands generation count) is to avoid blocking thread N during the update, but this doesn't actually work, even for only 2 timehands and a global generation count. Bruce From owner-freebsd-ppc@freebsd.org Wed Apr 3 16:11:24 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DE485157276C; Wed, 3 Apr 2019 16:11:24 +0000 (UTC) (envelope-from tuexen@freebsd.org) Received: from drew.franken.de (mail-n.franken.de [193.175.24.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.franken.de", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 78BDF88ED6; Wed, 3 Apr 2019 16:11:24 +0000 (UTC) (envelope-from tuexen@freebsd.org) Received: from [192.168.1.3] (p57BB4B1C.dip0.t-ipconnect.de [87.187.75.28]) (Authenticated sender: macmic) by mail-n.franken.de (Postfix) with ESMTPSA id 6E8E671B63002; Wed, 3 Apr 2019 18:11:20 +0200 (CEST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed] From: Michael Tuexen In-Reply-To: <20190403070045.GW1923@kib.kiev.ua> Date: Wed, 3 Apr 2019 18:11:19 +0200 Cc: Bruce Evans , freebsd-hackers Hackers , FreeBSD PowerPC ML Content-Transfer-Encoding: quoted-printable Message-Id: <8184C845-7721-4F88-974E-3F63F1980FDE@freebsd.org> References: <20190303161635.GJ68879@kib.kiev.ua> <20190304043416.V5640@besplex.bde.org> <20190304114150.GM68879@kib.kiev.ua> <20190305031010.I4610@besplex.bde.org> <20190306172003.GD2492@kib.kiev.ua> <20190308001005.M2756@besplex.bde.org> <20190307222220.GK2492@kib.kiev.ua> <20190309144844.K1166@besplex.bde.org> <20190324110138.GR1923@kib.kiev.ua> <20190403070045.GW1923@kib.kiev.ua> To: Konstantin Belousov X-Mailer: Apple Mail (2.3445.104.8) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=disabled version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mail-n.franken.de X-Rspamd-Queue-Id: 78BDF88ED6 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.96 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.96)[-0.957,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Apr 2019 16:11:25 -0000 > On 3. Apr 2019, at 09:00, Konstantin Belousov = wrote: >=20 > On Wed, Apr 03, 2019 at 12:27:32AM +0200, Michael Tuexen wrote: >>> On 24. Mar 2019, at 12:01, Konstantin Belousov = wrote: >>>=20 >>> On Sat, Mar 09, 2019 at 06:00:14PM +1100, Bruce Evans wrote: >>>> I more strongly disclike (sic) the more complete merge. The = central APIs >>>> have even more parameters and reduced type safety to describe = objects as >>>> (offset, size) pairs. >>> I changed the patch to be type-safe. Now I like it even more. It = provides >>> 1. internal >>> 2. concise >>> 3. type-safe >>> API to fetch data from timehands. The implementation needs to be = read >>> only once. >> Hi, >>=20 >> I'm a bit lost... I think this started to fix a problem on G5 = PowerMacs. >> Do you think this patch solves the problem. Should this be tested? >> Or is this still work in progress or a general improvement not = necessary >> fixing the problem on G5 PowerMacs? >=20 > It started from a report of issues on G5. The specific issues are > bugs on G5, and the posted patches do not fix them. =46rom what I = see, > the timecounter values were wrapped. This is genuine ppc or G5 issue. Thanks a lot for the clarification. Best regards Michael >=20 > The patch fixes time keeping subsystem reaction to the already > bad situation, by correctly handling overflow in calculations. This > overflow can occur in more reasonable setups as well, e.g. if ddb was > activated and interrupts were stopped for prolonged period, even on = x86. >=20 > In addition, my version of the patch reorganizes the code and removes > excessive copies of the most delicate loops in lock-less readers. This > chunk can be split from the overflow part, but it is not completely > trivial. From owner-freebsd-ppc@freebsd.org Thu Apr 4 20:16:04 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ABC471556352 for ; Thu, 4 Apr 2019 20:16:04 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic310-21.consmr.mail.gq1.yahoo.com (sonic310-21.consmr.mail.gq1.yahoo.com [98.137.69.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8A9ED8AC23 for ; Thu, 4 Apr 2019 20:16:03 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: tUl2yRAVM1mWa9EZN.Spbbd98seZjY_PZeD2.x5vKu1m3eempaq_3JfWWx7FY_s fvBY3fOt9Fhg5ve0M.CrRi1sXpQrxrXHOcSce9A4IxZMcNbIpicsSM5DTSaj7wd1maxyMuvOMtgU 0AnY1xYulmHaMv4219_1R0SWjhqtUcP9.31K3zc60yJKeeyFhFMwdNGyjGxo5YGC2Uy9tvr2E7eM itLgBgk11F2J3JqdeOEbk9TT0SdMK52qqfEWO1MUachqz8kTISq5FRHXCSpIPzWJpWF2xXynI1Vw 39j_bGs5luP4YX9bQ2Z0RFd45gt5h7QhPBWGXfso2G3s0c_NwAMZGxPlF59GUuCCKSGmyYUPXnRa blQozlJvxTi9vZ3GZkGkERWYUCfsT88o1P7x2N3m8kOANxfumKc2ESY5WZJjB03GVkhoHNmG4Af4 YaZkZaL.aLp_q.IeV3qeEFbyhk.ZbI2.4ZffDpsj4v4sup_JEAU58mPc94gZsUaX3k9zXPqKfuiR M7ThWGybRThlbxPSUv96VZARbP1MafRTWYu6tolvGMCOZQo314oqK6eAak_VpJMkXqE29NyoYawm LE9mcj42jxE2pmTBG2z6F9.Q8NeNLt8JiXOsPZ.qNb1e2csLotagwgjo1JOQqYLfI.nc5I0xxjKd Fu7LTwjvfIz70zOGH6wzHUP914MoEIL02S.boLroVAXg4tDJHav1J_Kce9HtrKBroioGaNI36jSJ vu9J4G80CyOfvwa.2qvO9P4OL9xvKnDBhiOQ3Ya4D2GMzcgLfQT6BIA0J5I6iJYwRtiNKbLI4RbY P_AntPHcrfIIoFsjahVbSZGRp5k_EIyfyCX_A1pob4ateBe3SBrCzM8ZhcTnBKfX1XVu97vdbE7p Y2puXXEYJ8u2JyzNJh_lSyqCSgRdsyMLCNmNsygAPCiFv1fp6BATXbQV0wYOPMbMrV5RKudE2dXh 6ZusHZ1n6eevT7.bHTQ5qUGqJEZ6bkgeecvlXjtWwWTmXGzhLNQug1Ur8yE_2IzMZd1.v2DYNS6V bjyQH7NX7N2JMb4eJu1cH0rYt9UoVxTYVyBBJrCuoF90sy54_EIRMlZFbGAZB11ZUm14ayC5gxoS CZw-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic310.consmr.mail.gq1.yahoo.com with HTTP; Thu, 4 Apr 2019 20:16:01 +0000 Received: from c-67-170-167-181.hsd1.or.comcast.net (EHLO [192.168.1.113]) ([67.170.167.181]) by smtp420.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 6a5cfc0e336cb9a55c16cba217ec176c; Thu, 04 Apr 2019 20:15:59 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed] From: Mark Millard In-Reply-To: <20190404011802.E2390@besplex.bde.org> Date: Thu, 4 Apr 2019 13:15:58 -0700 Cc: Konstantin Belousov , freebsd-hackers Hackers , FreeBSD PowerPC ML Content-Transfer-Encoding: quoted-printable Message-Id: References: <20190303161635.GJ68879@kib.kiev.ua> <20190304043416.V5640@besplex.bde.org> <20190304114150.GM68879@kib.kiev.ua> <20190305031010.I4610@besplex.bde.org> <20190306172003.GD2492@kib.kiev.ua> <20190308001005.M2756@besplex.bde.org> <20190307222220.GK2492@kib.kiev.ua> <20190309144844.K1166@besplex.bde.org> <20190324110138.GR1923@kib.kiev.ua> <20190403070045.GW1923@kib.kiev.ua> <20190404011802.E2390@besplex.bde.org> To: Bruce Evans X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 8A9ED8AC23 X-Spamd-Bar: / X-Spamd-Result: default: False [0.97 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FREEMAIL_TO(0.00)[optusnet.com.au]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.21)[-0.212,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_SPAM_SHORT(0.70)[0.698,0]; NEURAL_HAM_LONG(-0.10)[-0.102,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(1.09)[ip: (3.81), ipnet: 98.137.64.0/21(0.96), asn: 36647(0.76), country: US(-0.06)]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[147.69.137.98.list.dnswl.org : 127.0.5.0]; FREEMAIL_CC(0.00)[gmail.com] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Apr 2019 20:16:05 -0000 On 2019-Apr-3, at 08:47, Bruce Evans wrote: > . . . >=20 > I noticed (or better realized) a general problem with multiple > timehands. ntpd can slew the clock at up to 500 ppm, and at least an > old version of it uses a rate of 50 ppm to fix up fairly small drifts > in the milliseconds range. 500 ppm is enormous in CPU cycles -- it is > 500 thousand nsec or 2 million cycles at 4GHz. Winding up the = timecounter > every 1 msec reduces this to only 2000 cycles. >=20 > More details of ordering and timing for 1 thread: > - thread N calls binuptime() and it loads timehands > - another or even the same thread runs tc_windup(). This modifies = timehands. > - thread N is preempted for a long time, but less than the time for > updates > - thread N checks the generation count. Since this is for the = timehands > contents and not for the timehands pointer, it hasn't changed, so the > old timehands is used > - and instant later, the same thread calls binuptime again() and uses = the > new timehands - now suppose only 2 timehands (as in -current) the = worst (?) case of a > slew of 500 ppm for the old timehands and -500 ppm for the new = timehands > and almost the worst case of 10 msec for the oldness of the old = timehands > relative to the new timehands, with the new timehands about to be = updated > after 10 msec (assume perfectly periodiodic updates every 10 msec). = The > calculated times are: >=20 > old bintime =3D old_base + (20 msec) * (1 + 500e-6) > new base =3D old_base + 10 msec * (1 + 500e-6) # calc by = tc_windup() > new bintime =3D new_base + (10 msec) * (1 - 500e-6) + epsilon >=20 > error =3D epsilon - (20 msec) * 500e-6 =3D epsilon - 10000 nsec >=20 > Errors in the negative direction are most harmful. ntpd normally = doesn't > change the skew as much as that in one step, but it is easy for = adjtime(2) > to change the skew like that and there are no reasonable = microadjustments > that would accidentally work around this kernel bug (it seems = unreasonable > to limit the skew to 1 ppm and that still gives an error of epsilon + = 20 > nsec. >=20 > phk didn't want to slow down timecounters using extra code to make > them them monotonic and coherent (getbinuptime() is incoherent with > binuptime() since it former lags the latter by the update interval), > but this wouldn't be very slow within a thread. >=20 > Monotonicity across threads is a larger problem and not helped by = using > a faked forced monotonic time within threads. >=20 > So it seems best to fix the above problem by moving the generation = count > from the timehands contents to the timehands pointer, and maybe also > reduce the number of timehands to 1. With 2 timehands, this gives a > shorter race: >=20 > - thread N loads timehands > - tc_windup() > - thread N preempted > - thread N uses old timehands > - case tc_windup() completes first: no problem -- thread N checks the > generation count on the pointer and loops > - case binuptime() completes first: lost a race -- binuptime() is off > by approx * . >=20 > The main point of having multiple timehands (after introducing the = per- > timehands generation count) is to avoid blocking thread N during the > update, but this doesn't actually work, even for only 2 timehands and = a global generation count. >=20 Thanks for the description of an example way that sbinuptime and the like might not give weakly increasing results. Unfortunately, all the multi-socket contexts that I sometimes have access to are old PowerMacs. And, currently, the only such context is the G5 with 2 sockets, 2 cores per socket (powerpc64). So I've not been able to set up other types of examples to see if problems repeat. I do not have access to a single-socket powerpc64 for contrast in that direction. One oddity is that the eventtimer's decrementer and timecounter may change (nearly) together: both change at 33,333,333 Hz, as if they are tied to the same clock (at least on one socket). In case it helps with knowing how analogous your investigations are to the original problem context, I report the following. If you do not care for such information, stop reading here. # grep ntpd /etc/rc.conf ntpd_enable=3D"YES" ntpd_sync_on_start=3D"YES" # sysctl kern.eventtimer kern.eventtimer.periodic: 0 kern.eventtimer.timer: decrementer kern.eventtimer.idletick: 0 kern.eventtimer.singlemul: 2 kern.eventtimer.choice: decrementer(1000) kern.eventtimer.et.decrementer.quality: 1000 kern.eventtimer.et.decrementer.frequency: 33333333 kern.eventtimer.et.decrementer.flags: 7 # vmstat -ai | grep decrementer cpu0:decrementer 4451007 35 cpu3:decrementer 1466010 11 cpu2:decrementer 1481722 12 cpu1:decrementer 1478618 12 (That last is from a basically-idle timeframe.) # sysctl -a | grep hz kern.clockrate: { hz =3D 1000, tick =3D 1000, profhz =3D 8128, stathz =3D = 127 } kern.hz: 1000 # sysctl kern.timecounter kern.timecounter.fast_gettime: 1 kern.timecounter.tick: 1 kern.timecounter.choice: timebase(0) dummy(-1000000) kern.timecounter.hardware: timebase kern.timecounter.alloweddeviation: 5 kern.timecounter.stepwarnings: 0 kern.timecounter.tc.timebase.quality: 0 kern.timecounter.tc.timebase.frequency: 33333333 kern.timecounter.tc.timebase.counter: 1144662532 kern.timecounter.tc.timebase.mask: 4294967295 (The actual Time Base Register (tbr) i s 64 bits and freebsd truncates it down.) # sysctl -a | grep 'cpu.*freq' device cpufreq debug.cpufreq.verbose: 0 debug.cpufreq.lowest: 0 dev.cpufreq.0.%parent: cpu3 dev.cpufreq.0.%pnpinfo:=20 dev.cpufreq.0.%location:=20 dev.cpufreq.0.%driver: cpufreq dev.cpufreq.0.%desc:=20 dev.cpufreq.%parent:=20 dev.cpu.3.freq_levels: 2500/-1 1250/-1 dev.cpu.3.freq: 2500 So 2500 MHz / 33333333 Hz is very near 75 clock periods per timebase counter value. I do sometimes have access to a Ryzen Threadripper 1950X based system: FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs FreeBSD/SMP: 1 package(s) x 2 groups x 2 cache groups x 4 core(s) x 2 = hardware threads but it is single=3Dsocket. It has . . . [It turns out that I've accidentally been running it without ntpd being enabled in /etc.rc.conf . Just fixed for next boot.] # sysctl kern.eventtimer = = kern.eventtimer.periodic: = 0 kern.eventtimer.timer: LAPIC kern.eventtimer.idletick: 0 kern.eventtimer.singlemul: 2 kern.eventtimer.choice: LAPIC(600) HPET(350) HPET1(350) HPET2(350) = i8254(100) RTC(0) kern.eventtimer.et.RTC.quality: 0 kern.eventtimer.et.RTC.frequency: 32768 kern.eventtimer.et.RTC.flags: 17 kern.eventtimer.et.i8254.quality: 100 kern.eventtimer.et.i8254.frequency: 1193182 kern.eventtimer.et.i8254.flags: 1 kern.eventtimer.et.HPET2.quality: 350 kern.eventtimer.et.HPET2.frequency: 14318180 kern.eventtimer.et.HPET2.flags: 3 kern.eventtimer.et.HPET1.quality: 350 kern.eventtimer.et.HPET1.frequency: 14318180 kern.eventtimer.et.HPET1.flags: 3 kern.eventtimer.et.HPET.quality: 350 kern.eventtimer.et.HPET.frequency: 14318180 kern.eventtimer.et.HPET.flags: 3 kern.eventtimer.et.LAPIC.quality: 600 kern.eventtimer.et.LAPIC.frequency: 49907650 kern.eventtimer.et.LAPIC.flags: 7 # vmstat -ai | grep timer irq0: attimer0 0 0 cpu0:timer 609974 14 cpu1:timer 160546 4 cpu2:timer 99803 2 cpu3:timer 158073 4 cpu4:timer 102870 2 cpu5:timer 168433 4 cpu6:timer 163004 4 cpu7:timer 162947 4 cpu8:timer 163501 4 cpu9:timer 160595 4 cpu10:timer 169100 4 cpu11:timer 163888 4 cpu12:timer 144278 3 cpu13:timer 162706 4 cpu14:timer 164161 4 cpu15:timer 167101 4 cpu16:timer 187732 4 cpu17:timer 189228 4 cpu18:timer 179423 4 cpu19:timer 182701 4 cpu20:timer 139658 3 cpu21:timer 186046 4 cpu22:timer 201248 5 cpu23:timer 184823 4 cpu24:timer 186839 4 cpu25:timer 186008 4 cpu26:timer 191473 4 cpu27:timer 182573 4 cpu28:timer 181213 4 cpu29:timer 197659 5 cpu30:timer 188190 4 cpu31:timer 189092 4 (Again: from a basically-idle timeframe.) # sysctl -a | grep hz kern.clockrate: { hz =3D 1000, tick =3D 1000, profhz =3D 8128, stathz =3D = 127 } kern.hz: 1000 debug.psm.hz: 20 # sysctl -a | grep 'cpu.*freq' device cpufreq debug.cpufreq.verbose: 0 debug.cpufreq.lowest: 0 dev.cpufreq.0.%parent: cpu0 dev.cpufreq.0.%pnpinfo:=20 dev.cpufreq.0.%location:=20 dev.cpufreq.0.%driver: cpufreq dev.cpufreq.0.%desc:=20 dev.cpufreq.%parent:=20 dev.cpu.0.freq_levels: 3400/-1 2800/-1 2200/-1 dev.cpu.0.freq: 3400 # sysctl kern.timecounter kern.timecounter.tsc_shift: 1 kern.timecounter.smp_tsc_adjust: 0 kern.timecounter.smp_tsc: 1 kern.timecounter.invariant_tsc: 1 kern.timecounter.fast_gettime: 1 kern.timecounter.tick: 1 kern.timecounter.choice: ACPI-fast(900) i8254(0) HPET(950) TSC-low(1000) = dummy(-1000000) kern.timecounter.hardware: TSC-low kern.timecounter.alloweddeviation: 5 kern.timecounter.stepwarnings: 0 kern.timecounter.tc.ACPI-fast.quality: 900 kern.timecounter.tc.ACPI-fast.frequency: 3579545 kern.timecounter.tc.ACPI-fast.counter: 3941257163 kern.timecounter.tc.ACPI-fast.mask: 4294967295 kern.timecounter.tc.i8254.quality: 0 kern.timecounter.tc.i8254.frequency: 1193182 kern.timecounter.tc.i8254.counter: 54249 kern.timecounter.tc.i8254.mask: 65535 kern.timecounter.tc.HPET.quality: 950 kern.timecounter.tc.HPET.frequency: 14318180 kern.timecounter.tc.HPET.counter: 796737466 kern.timecounter.tc.HPET.mask: 4294967295 kern.timecounter.tc.TSC-low.quality: 1000 kern.timecounter.tc.TSC-low.frequency: 1696860321 kern.timecounter.tc.TSC-low.counter: 3613142695 kern.timecounter.tc.TSC-low.mask: 4294967295 =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-ppc@freebsd.org Fri Apr 5 04:08:07 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3B2FE1567AA7 for ; Fri, 5 Apr 2019 04:08:07 +0000 (UTC) (envelope-from aik@ozlabs.ru) Received: from mail-pf1-x441.google.com (mail-pf1-x441.google.com [IPv6:2607:f8b0:4864:20::441]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D657D7451F for ; Fri, 5 Apr 2019 04:08:04 +0000 (UTC) (envelope-from aik@ozlabs.ru) Received: by mail-pf1-x441.google.com with SMTP id t21so198498pfh.2 for ; Thu, 04 Apr 2019 21:08:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ozlabs-ru.20150623.gappssmtp.com; s=20150623; h=to:from:subject:openpgp:autocrypt:message-id:date:user-agent :mime-version:content-language:content-transfer-encoding; bh=82PD9LUHjTEyj2kih6PYc7WryaOlQPs6Qy/TJHC8FM4=; b=uzsFIiayTQz2mciMAc9ImK02Tz5Xkm4Q6hzyi9+UQ5LJremux82GIyQXI3x811F6Vn rn8Z6XK+H43peRXbv23lBy/iajVOZOU+V2NWgTAJJkj9DRoEaEOqP50olZpwap0gEsAu dyjQP0X3mJCS5hWBWU1SGGN8tvJXmjL1kZdhjhuDzgjo6jW8YMiCzt4RWvwGdssxJQt+ VphIpwGhFKPIjypz3QutreA/4RG861kktActEHkHMpN8aIg9PRbh7THJz9rANMNLyg2x SnT1geEP3qD9I8G04js3RNgjeHppFg5HByT3L+wCpmFgVU+saZyFrssi8T58AUEyiHWK DKFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:subject:openpgp:autocrypt:message-id :date:user-agent:mime-version:content-language :content-transfer-encoding; bh=82PD9LUHjTEyj2kih6PYc7WryaOlQPs6Qy/TJHC8FM4=; b=Ui82CIWIzfHM05UkcbmyDeLuvt8XiJCXT+P0y12QJIibygj/N2ALMbAwEIVMLNc8sE +a8pchxpyt6O4wERAjwIy0G+boyrcc1PP44FYew4CkFlEMSNkaLUUhB6OZ0wtwb1Kx+Y 6O42QUHUSMMM22AwY2FAMU5UETPTQAZGcHwLU9wE+0V/Tf/fRXi+QRfuXJGZ4iO5Q1ND 0DZ7aMI9W9AXgMIZj4qZq1iCgY7rmZuvSAoK8JoweJLmr6IR6x4o8JtmbsquMcU2VS20 vSbtocazl5q2LZ4Dk5aZbu7I7akOhlhlXHzqi8ydkS68S9Dk9IUrqWQiZKsV4nABqRR3 IILw== X-Gm-Message-State: APjAAAVyd3m/ZLhjBx3V4xqZ827/Gcd1AMHOBP/2mTONhZ4amvPrPDtj t4E7Pz8RIsoXsKUqQyaafEmouncw5rE= X-Google-Smtp-Source: APXvYqwCbqfSXVxpNSlmXrlBSsRGMGTybbXVN6AEpX+/LDzEUA6ZrgcjxS4i0sZt9T9D2UzWx0ClAQ== X-Received: by 2002:a62:571b:: with SMTP id l27mr10056998pfb.195.1554437282463; Thu, 04 Apr 2019 21:08:02 -0700 (PDT) Received: from [10.61.2.175] ([122.99.82.10]) by smtp.gmail.com with ESMTPSA id v189sm60724671pgd.77.2019.04.04.21.08.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 04 Apr 2019 21:08:01 -0700 (PDT) To: freebsd-ppc@freebsd.org From: Alexey Kardashevskiy Subject: Fail to boot FreeBSD12 or 13 on POWER9 under PowerKVM Openpgp: preference=signencrypt Autocrypt: addr=aik@ozlabs.ru; keydata= mQINBE+rT0sBEADFEI2UtPRsLLvnRf+tI9nA8T91+jDK3NLkqV+2DKHkTGPP5qzDZpRSH6mD EePO1JqpVuIow/wGud9xaPA5uvuVgRS1q7RU8otD+7VLDFzPRiRE4Jfr2CW89Ox6BF+q5ZPV /pS4v4G9eOrw1v09lEKHB9WtiBVhhxKK1LnUjPEH3ifkOkgW7jFfoYgTdtB3XaXVgYnNPDFo PTBYsJy+wr89XfyHr2Ev7BB3Xaf7qICXdBF8MEVY8t/UFsesg4wFWOuzCfqxFmKEaPDZlTuR tfLAeVpslNfWCi5ybPlowLx6KJqOsI9R2a9o4qRXWGP7IwiMRAC3iiPyk9cknt8ee6EUIxI6 t847eFaVKI/6WcxhszI0R6Cj+N4y+1rHfkGWYWupCiHwj9DjILW9iEAncVgQmkNPpUsZECLT WQzMuVSxjuXW4nJ6f4OFHqL2dU//qR+BM/eJ0TT3OnfLcPqfucGxubhT7n/CXUxEy+mvWwnm s9p4uqVpTfEuzQ0/bE6t7dZdPBua7eYox1AQnk8JQDwC3Rn9kZq2O7u5KuJP5MfludMmQevm pHYEMF4vZuIpWcOrrSctJfIIEyhDoDmR34bCXAZfNJ4p4H6TPqPh671uMQV82CfTxTrMhGFq 8WYU2AH86FrVQfWoH09z1WqhlOm/KZhAV5FndwVjQJs1MRXD8QARAQABtCRBbGV4ZXkgS2Fy ZGFzaGV2c2tpeSA8YWlrQG96bGFicy5ydT6JAjgEEwECACIFAk+rT0sCGwMGCwkIBwMCBhUI AgkKCwQWAgMBAh4BAheAAAoJEIYTPdgrwSC5fAIP/0wf/oSYaCq9PhO0UP9zLSEz66SSZUf7 AM9O1rau1lJpT8RoNa0hXFXIVbqPPKPZgorQV8SVmYRLr0oSmPnTiZC82x2dJGOR8x4E01gK TanY53J/Z6+CpYykqcIpOlGsytUTBA+AFOpdaFxnJ9a8p2wA586fhCZHVpV7W6EtUPH1SFTQ q5xvBmr3KkWGjz1FSLH4FeB70zP6uyuf/B2KPmdlPkyuoafl2UrU8LBADi/efc53PZUAREih sm3ch4AxaL4QIWOmlE93S+9nHZSRo9jgGXB1LzAiMRII3/2Leg7O4hBHZ9Nki8/fbDo5///+ kD4L7UNbSUM/ACWHhd4m1zkzTbyRzvL8NAVQ3rckLOmju7Eu9whiPueGMi5sihy9VQKHmEOx OMEhxLRQbzj4ypRLS9a+oxk1BMMu9cd/TccNy0uwx2UUjDQw/cXw2rRWTRCxoKmUsQ+eNWEd iYLW6TCfl9CfHlT6A7Zmeqx2DCeFafqEd69DqR9A8W5rx6LQcl0iOlkNqJxxbbW3ddDsLU/Y r4cY20++WwOhSNghhtrroP+gouTOIrNE/tvG16jHs8nrYBZuc02nfX1/gd8eguNfVX/ZTHiR gHBWe40xBKwBEK2UeqSpeVTohYWGBkcd64naGtK9qHdo1zY1P55lHEc5Uhlk743PgAnOi27Q ns5zuQINBE+rT0sBEACnV6GBSm+25ACT+XAE0t6HHAwDy+UKfPNaQBNTTt31GIk5aXb2Kl/p AgwZhQFEjZwDbl9D/f2GtmUHWKcCmWsYd5M/6Ljnbp0Ti5/xi6FyfqnO+G/wD2VhGcKBId1X Em/B5y1kZVbzcGVjgD3HiRTqE63UPld45bgK2XVbi2+x8lFvzuFq56E3ZsJZ+WrXpArQXib2 hzNFwQleq/KLBDOqTT7H+NpjPFR09Qzfa7wIU6pMNF2uFg5ihb+KatxgRDHg70+BzQfa6PPA o1xioKXW1eHeRGMmULM0Eweuvpc7/STD3K7EJ5bBq8svoXKuRxoWRkAp9Ll65KTUXgfS+c0x gkzJAn8aTG0z/oEJCKPJ08CtYQ5j7AgWJBIqG+PpYrEkhjzSn+DZ5Yl8r+JnZ2cJlYsUHAB9 jwBnWmLCR3gfop65q84zLXRQKWkASRhBp4JK3IS2Zz7Nd/Sqsowwh8x+3/IUxVEIMaVoUaxk Wt8kx40h3VrnLTFRQwQChm/TBtXqVFIuv7/Mhvvcq11xnzKjm2FCnTvCh6T2wJw3de6kYjCO 7wsaQ2y3i1Gkad45S0hzag/AuhQJbieowKecuI7WSeV8AOFVHmgfhKti8t4Ff758Z0tw5Fpc BFDngh6Lty9yR/fKrbkkp6ux1gJ2QncwK1v5kFks82Cgj+DSXK6GUQARAQABiQIfBBgBAgAJ BQJPq09LAhsMAAoJEIYTPdgrwSC5NYEP/2DmcEa7K9A+BT2+G5GXaaiFa098DeDrnjmRvumJ BhA1UdZRdfqICBADmKHlJjj2xYo387sZpS6ABbhrFxM6s37g/pGPvFUFn49C47SqkoGcbeDz Ha7JHyYUC+Tz1dpB8EQDh5xHMXj7t59mRDgsZ2uVBKtXj2ZkbizSHlyoeCfs1gZKQgQE8Ffc F8eWKoqAQtn3j4nE3RXbxzTJJfExjFB53vy2wV48fUBdyoXKwE85fiPglQ8bU++0XdOr9oyy j1llZlB9t3tKVv401JAdX8EN0++ETiOovQdzE1m+6ioDCtKEx84ObZJM0yGSEGEanrWjiwsa nzeK0pJQM9EwoEYi8TBGhHC9ksaAAQipSH7F2OHSYIlYtd91QoiemgclZcSgrxKSJhyFhmLr QEiEILTKn/pqJfhHU/7R7UtlDAmFMUp7ByywB4JLcyD10lTmrEJ0iyRRTVfDrfVP82aMBXgF tKQaCxcmLCaEtrSrYGzd1sSPwJne9ssfq0SE/LM1J7VdCjm6OWV33SwKrfd6rOtvOzgadrG6 3bgUVBw+bsXhWDd8tvuCXmdY4bnUblxF2B6GOwSY43v6suugBttIyW5Bl2tXSTwP+zQisOJo +dpVG2pRr39h+buHB3NY83NEPXm1kUOhduJUA17XUY6QQCAaN4sdwPqHq938S3EmtVhsuQIN BFq54uIBEACtPWrRdrvqfwQF+KMieDAMGdWKGSYSfoEGGJ+iNR8v255IyCMkty+yaHafvzpl PFtBQ/D7Fjv+PoHdFq1BnNTk8u2ngfbre9wd9MvTDsyP/TmpF0wyyTXhhtYvE267Av4X/BQT lT9IXKyAf1fP4BGYdTNgQZmAjrRsVUW0j6gFDrN0rq2J9emkGIPvt9rQt6xGzrd6aXonbg5V j6Uac1F42ESOZkIh5cN6cgnGdqAQb8CgLK92Yc8eiCVCH3cGowtzQ2m6U32qf30cBWmzfSH0 HeYmTP9+5L8qSTA9s3z0228vlaY0cFGcXjdodBeVbhqQYseMF9FXiEyRs28uHAJEyvVZwI49 CnAgVV/n1eZa5qOBpBL+ZSURm8Ii0vgfvGSijPGbvc32UAeAmBWISm7QOmc6sWa1tobCiVmY SNzj5MCNk8z4cddoKIc7Wt197+X/X5JPUF5nQRvg3SEHvfjkS4uEst9GwQBpsbQYH9MYWq2P PdxZ+xQE6v7cNB/pGGyXqKjYCm6v70JOzJFmheuUq0Ljnfhfs15DmZaLCGSMC0Amr+rtefpA y9FO5KaARgdhVjP2svc1F9KmTUGinSfuFm3quadGcQbJw+lJNYIfM7PMS9fftq6vCUBoGu3L j4xlgA/uQl/LPneu9mcvit8JqcWGS3fO+YeagUOon1TRqQARAQABiQRsBBgBCAAgFiEEZSrP ibrORRTHQ99dhhM92CvBILkFAlq54uICGwICQAkQhhM92CvBILnBdCAEGQEIAB0WIQQIhvWx rCU+BGX+nH3N7sq0YorTbQUCWrni4gAKCRDN7sq0YorTbVVSD/9V1xkVFyUCZfWlRuryBRZm S4GVaNtiV2nfUfcThQBfF0sSW/aFkLP6y+35wlOGJE65Riw1C2Ca9WQYk0xKvcZrmuYkK3DZ 0M9/Ikkj5/2v0vxz5Z5w/9+IaCrnk7pTnHZuZqOh23NeVZGBls/IDIvvLEjpD5UYicH0wxv+ X6cl1RoP2Kiyvenf0cS73O22qSEw0Qb9SId8wh0+ClWet2E7hkjWFkQfgJ3hujR/JtwDT/8h 3oCZFR0KuMPHRDsCepaqb/k7VSGTLBjVDOmr6/C9FHSjq0WrVB9LGOkdnr/xcISDZcMIpbRm EkIQ91LkT/HYIImL33ynPB0SmA+1TyMgOMZ4bakFCEn1vxB8Ir8qx5O0lHMOiWMJAp/PAZB2 r4XSSHNlXUaWUg1w3SG2CQKMFX7vzA31ZeEiWO8tj/c2ZjQmYjTLlfDK04WpOy1vTeP45LG2 wwtMA1pKvQ9UdbYbovz92oyZXHq81+k5Fj/YA1y2PI4MdHO4QobzgREoPGDkn6QlbJUBf4To pEbIGgW5LRPLuFlOPWHmIS/sdXDrllPc29aX2P7zdD/ivHABslHmt7vN3QY+hG0xgsCO1JG5 pLORF2N5XpM95zxkZqvYfC5tS/qhKyMcn1kC0fcRySVVeR3tUkU8/caCqxOqeMe2B6yTiU1P aNDq25qYFLeYxg67D/4w/P6BvNxNxk8hx6oQ10TOlnmeWp1q0cuutccblU3ryRFLDJSngTEu ZgnOt5dUFuOZxmMkqXGPHP1iOb+YDznHmC0FYZFG2KAc9pO0WuO7uT70lL6larTQrEneTDxQ CMQLP3qAJ/2aBH6SzHIQ7sfbsxy/63jAiHiT3cOaxAKsWkoV2HQpnmPOJ9u02TPjYmdpeIfa X2tXyeBixa3i/6dWJ4nIp3vGQicQkut1YBwR7dJq67/FCV3Mlj94jI0myHT5PIrCS2S8LtWX ikTJSxWUKmh7OP5mrqhwNe0ezgGiWxxvyNwThOHc5JvpzJLd32VDFilbxgu4Hhnf6LcgZJ2c Zd44XWqUu7FzVOYaSgIvTP0hNrBYm/E6M7yrLbs3JY74fGzPWGRbBUHTZXQEqQnZglXaVB5V ZhSFtHopZnBSCUSNDbB+QGy4B/E++Bb02IBTGl/JxmOwG+kZUnymsPvTtnNIeTLHxN/H/ae0 c7E5M+/NpslPCmYnDjs5qg0/3ihh6XuOGggZQOqrYPC3PnsNs3NxirwOkVPQgO6mXxpuifvJ DG9EMkK8IBXnLulqVk54kf7fE0jT/d8RTtJIA92GzsgdK2rpT1MBKKVffjRFGwN7nQVOzi4T XrB5p+6ML7Bd84xOEGsj/vdaXmz1esuH7BOZAGEZfLRCHJ0GVCSssg== Message-ID: Date: Fri, 5 Apr 2019 15:07:58 +1100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: D657D7451F X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=ozlabs-ru.20150623.gappssmtp.com header.s=20150623 header.b=uzsFIiay; spf=pass (mx1.freebsd.org: domain of aik@ozlabs.ru designates 2607:f8b0:4864:20::441 as permitted sender) smtp.mailfrom=aik@ozlabs.ru X-Spamd-Result: default: False [-3.72 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[ozlabs-ru.20150623.gappssmtp.com:s=20150623]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-ppc@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[ozlabs.ru]; DKIM_TRACE(0.00)[ozlabs-ru.20150623.gappssmtp.com:+]; MX_GOOD(-0.01)[alt1.aspmx.l.google.com,aspmx.l.google.com,aspmx2.googlemail.com,alt2.aspmx.l.google.com,aspmx3.googlemail.com]; RCVD_IN_DNSWL_NONE(0.00)[1.4.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.63)[-0.632,0]; IP_SCORE(-0.58)[ip: (2.26), ipnet: 2607:f8b0::/32(-2.92), asn: 15169(-2.17), country: US(-0.06)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2019 04:08:07 -0000 Hi! I am trying a freebsd guest on a POWER9 (pvr=004e1201) host with linux+kvm (5.1.0-rc2) and qemu (upstream, 4.0) and something goes wrong - it crashes as (the full output is below): ===== run_interrupt_driven_hooks: still waiting after 300 seconds for xpt_config random: unblocking device. panic: run_interrupt_driven_config_hooks: waited too long cpuid = 0 time = 361 KDB: stack backtrace: 0xe000000000008660: at .kdb_backtrace+0x5c 0xe000000000008790: at .vpanic+0x1b4 0xe000000000008850: at .panic+0x38 0xe0000000000088e0: at .boot_run_interrupt_driven_config_hooks+0x194 0xe0000000000089e0: at .mi_startup+0x1f8 0xe000000000008a80: at btext+0xc4 KDB: enter: panic [ thread pid 0 tid 100000 ] Stopped at .kdb_enter+0x60: ld r2, r1, 0x28 db> ===== The systems were installed from: FreeBSD-12.0-RELEASE-powerpc-powerpc64-dvd1.iso FreeBSD-13.0-CURRENT-powerpc-powerpc64-20190307-r344854-disc1.iso Everything by default (vt100 terminal, etc), IBM vio-scsi disk and cdrom, 8GB RAM, 16MB backing huge pages, the host is running in the hash (HPT) mode. However the exact same disk images + qemu/slof binary + qemu command line + linux kernel do boot on POWER8E (pvr=004b0201) and POWER8NVL (pvr=004c0100) to the login prompt. I told QEMU to enforce XICS interrupt controller mode, POWER8 compatibility (although it does not make sense as FreeBSD does not do "client-architecture-support" RTAS call), what else can I try? While at it, FreeBSD is aware of 004b0201 and 004e1201 but it fails to recognize 004c0100 (the FreeBSD guest still boots just fine): cpu0: Unknown PowerPC CPU revision 0x0100, 3259.00 MHz cpu0: Features c4000000 but in fact architecturally it behaves exactly as IBMPOWER8 (004bxxxx or 004d0000). build/qemu-aikrhel74alt-ppc64/ppc64-softmmu/qemu-system-ppc64 \ -nodefaults \ -chardev stdio,id=STDIO0,signal=off,mux=on \ -device spapr-vty,id=svty0,reg=0x71000110,chardev=STDIO0 \ -mon id=MON0,chardev=STDIO0,mode=readline -nographic -vga none \ img/freebsd12-64G.qcow2 -enable-kvm \ -smp 1 -mem-prealloc -mem-path qemu_hp_16M_node0 -m 8G \ -machine \ pseries,cap-hpt-max-page-size=16M,cap-cfpc=broken,max-cpu-compat=power8,ic-mode=xics \ -snapshot -bios ./slof.bin \ -L /home/aik/t/qemu-ppc64-bios/ \ -trace events=qemu_trace_events -d guest_errors \ -chardev socket,id=SOCKET0,server,nowait,path=qemu.mon.8324 \ -mon chardev=SOCKET0,mode=control SLOF ********************************************************************** QEMU Starting Build Date = Apr 5 2019 13:01:51 FW Version = git-a5b428e1c1eae703 Press "s" to enter Open Firmware. Populating /vdevice methods Populating /vdevice/nvram@71000000 Populating /vdevice/v-scsi@71000001 SCSI: Looking for devices 8000000000000000 DISK : "QEMU QEMU HARDDISK 2.5+" Populating /vdevice/vty@71000110 Populating /pci@800000020000000 No NVRAM common partition, re-initializing... Scanning USB Using default console: /vdevice/vty@71000110 Welcome to Open Firmware Copyright (c) 2004, 2017 IBM Corporation All rights reserved. This program and the accompanying materials are made available under the terms of the BSD License available at http://www.opensource.org/licenses/bsd-license.php Trying to load: from: /vdevice/v-scsi@71000001/disk@8000000000000000 ... Successfully loaded >> FreeBSD/powerpc Open Firmware boot block Boot path: /vdevice/v-scsi@71000001/disk@8000000000000000 Boot loader: /boot/loader Boot volume: /vdevice/v-scsi@71000001/disk@8000000000000000:2 Consoles: Open Firmware console FreeBSD/powerpc64 Open Firmware loader, Revision 0.1 Memory: 8388608KB Booted from: /vdevice/v-scsi@71000001/disk@8000000000000000 Loading /boot/defaults/loader.conf /boot/kernel/kernel data=0x136c550+0x4aa1f0 syms=[0x8+0x165c78+0x8+0x1643cb] /boot/entropy size=0x1000 Hit [Enter] to boot immediately, or any other key for command prompt. Booting [/boot/kernel/kernel]... Kernel entry at 0x1025d0 ... ---<>--- Copyright (c) 1992-2018 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-RELEASE r341666 GENERIC powerpc gcc version 4.2.1 20070831 patched [FreeBSD] VT: init without driver. cpu0: IBM POWER9 revision 2.1, 2250.00 MHz cpu0: Features dc007182 cpu0: Features2 eee00000 real memory = 8544436224 (8148 MB) avail memory = 8258940928 (7876 MB) random: unblocking device. random: entropy device external interface kbd0 at kbdmux0 random: registering fast source PowerISA DARN random number generator random: fast provider: "PowerISA DARN random number generator" ofwbus0: on nexus0 xicp0: on ofwbus0 xicp0: Handling CPUs 0-7 vdevice0: on ofwbus0 uart0: irq 16781585 on vdevice0 vscsi0: irq 16781570 on vdevice0 vscsi0: Queue depth 22 commands pcib0: on ofwbus0 pci0: on pcib0 cpulist0: on ofwbus0 cpu0: on cpulist0 rtas0: on ofwbus0 rtas0: registered as a time-of-day clock, resolution 0.002000s Timecounter "timebase" frequency 512000000 Hz quality 0 Event timer "decrementer" frequency 512000000 Hz quality 1000 Timecounters tick every 1.000 msec usb_needs_explore_all: no devclass run_interrupt_driven_hooks: still waiting after 60 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 120 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 180 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 240 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 300 seconds for xpt_config random: unblocking device. panic: run_interrupt_driven_config_hooks: waited too long cpuid = 0 time = 361 KDB: stack backtrace: 0xe000000000008660: at .kdb_backtrace+0x5c 0xe000000000008790: at .vpanic+0x1b4 0xe000000000008850: at .panic+0x38 0xe0000000000088e0: at .boot_run_interrupt_driven_config_hooks+0x194 0xe0000000000089e0: at .mi_startup+0x1f8 0xe000000000008a80: at btext+0xc4 KDB: enter: panic [ thread pid 0 tid 100000 ] Stopped at .kdb_enter+0x60: ld r2, r1, 0x28 db> -- Alexey From owner-freebsd-ppc@freebsd.org Fri Apr 5 04:38:16 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AED1715695DB; Fri, 5 Apr 2019 04:38:16 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail106.syd.optusnet.com.au (mail106.syd.optusnet.com.au [211.29.132.42]) by mx1.freebsd.org (Postfix) with ESMTP id 8DD5674FED; Fri, 5 Apr 2019 04:38:06 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from [192.168.0.102] (c110-21-101-228.carlnfd1.nsw.optusnet.com.au [110.21.101.228]) by mail106.syd.optusnet.com.au (Postfix) with ESMTPS id C8F303DBE4B; Fri, 5 Apr 2019 15:38:02 +1100 (AEDT) Date: Fri, 5 Apr 2019 15:38:02 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Mark Millard cc: Bruce Evans , Konstantin Belousov , freebsd-hackers Hackers , FreeBSD PowerPC ML Subject: Re: powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed] In-Reply-To: Message-ID: <20190405150236.A959@besplex.bde.org> References: <20190303161635.GJ68879@kib.kiev.ua> <20190304043416.V5640@besplex.bde.org> <20190304114150.GM68879@kib.kiev.ua> <20190305031010.I4610@besplex.bde.org> <20190306172003.GD2492@kib.kiev.ua> <20190308001005.M2756@besplex.bde.org> <20190307222220.GK2492@kib.kiev.ua> <20190309144844.K1166@besplex.bde.org> <20190324110138.GR1923@kib.kiev.ua> <20190403070045.GW1923@kib.kiev.ua> <20190404011802.E2390@besplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.2 cv=FNpr/6gs c=1 sm=1 tr=0 cx=a_idp_d a=PalzARQSbocsUSjMRkwAPg==:117 a=PalzARQSbocsUSjMRkwAPg==:17 a=jpOVt7BSZ2e4Z31A5e1TngXxSK0=:19 a=kj9zAlcOel0A:10 a=PO7r1zJSAAAA:8 a=jA9InAqbobDorSXSjysA:9 a=nJ01U1HSc7Yp0U_T:21 a=jiLLkcfR9Aiq0lNV:21 a=CjuIK1q_8ugA:10 X-Rspamd-Queue-Id: 8DD5674FED X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of brde@optusnet.com.au designates 211.29.132.42 as permitted sender) smtp.mailfrom=brde@optusnet.com.au X-Spamd-Result: default: False [-4.87 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_IN_DNSWL_LOW(-0.10)[42.132.29.211.list.dnswl.org : 127.0.5.1]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[optusnet.com.au]; R_SPF_ALLOW(-0.20)[+ip4:211.29.132.0/23]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[optusnet.com.au]; RCPT_COUNT_FIVE(0.00)[5]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[extmail.optusnet.com.au]; NEURAL_SPAM_SHORT(0.23)[0.232,0]; IP_SCORE(-2.79)[ip: (-7.79), ipnet: 211.28.0.0/14(-3.40), asn: 4804(-2.71), country: AU(-0.04)]; FREEMAIL_TO(0.00)[yahoo.com]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[optusnet.com.au]; ASN(0.00)[asn:4804, ipnet:211.28.0.0/14, country:AU]; FREEMAIL_CC(0.00)[optusnet.com.au]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2019 04:38:16 -0000 On Thu, 4 Apr 2019, Mark Millard wrote: > On 2019-Apr-3, at 08:47, Bruce Evans wrote: >> . . . >> >> I noticed (or better realized) a general problem with multiple >> timehands. ntpd can slew the clock at up to 500 ppm, and at least an >> old version of it uses a rate of 50 ppm to fix up fairly small drifts >> in the milliseconds range. 500 ppm is enormous in CPU cycles -- it is >> 500 thousand nsec or 2 million cycles at 4GHz. Winding up the timecounter >> every 1 msec reduces this to only 2000 cycles. >> >> More details of ordering and timing for 1 thread: >> ... > Thanks for the description of an example way that sbinuptime and > the like might not give weakly increasing results. > > Unfortunately, all the multi-socket contexts that I sometimes have > access to are old PowerMacs. And, currently, the only such context > is the G5 with 2 sockets, 2 cores per socket (powerpc64). So I've > not been able to set up other types of examples to see if problems > repeat. > > I do not have access to a single-socket powerpc64 for contrast in > that direction. Testing 1 socket is time-consuming enough. Do these old systems use the equivalent of an x86 TSC for the timecounter? With multiple sockets, it isn't clear how even a hardware timer independent of the CPUs can be distributed so as to appear to be monotonic on all cors. > One oddity is that the eventtimer's decrementer and timecounter > may change (nearly) together: both change at 33,333,333 Hz, as if > they are tied to the same clock (at least on one socket). I think this is from a normal hardware implementation. On all of my x86 systems with a TSC, the TSC frequency is an exact fractional multiple of the i8254, the ACPI timer (if present) and the HPET (if present). Only the RTC has an independent frequency. The fraction is changed by changing the nominal TSC frequency in the BIOS, but is not changed by temperature variations. This must be because most clocks are derived from a common clock using a PLL. I use this to calibrate all clocks (except the RTC) by calibrating only 1. > In case it helps with knowing how analogous your investigations > are to the original problem context, I report the following. > > If you do not care for such information, stop reading here. > > # grep ntpd /etc/rc.conf > ntpd_enable="YES" > ntpd_sync_on_start="YES" > > # sysctl kern.eventtimer > kern.eventtimer.periodic: 0 > kern.eventtimer.timer: decrementer > kern.eventtimer.idletick: 0 > kern.eventtimer.singlemul: 2 > kern.eventtimer.choice: decrementer(1000) > kern.eventtimer.et.decrementer.quality: 1000 > kern.eventtimer.et.decrementer.frequency: 33333333 > kern.eventtimer.et.decrementer.flags: 7 > > # vmstat -ai | grep decrementer > cpu0:decrementer 4451007 35 > cpu3:decrementer 1466010 11 > cpu2:decrementer 1481722 12 > cpu1:decrementer 1478618 12 Powerpc seems to have a PLL in software too. Event timers don't need to be very precise or accurate. > (That last is from a basically-idle timeframe.) > > # sysctl -a | grep hz > kern.clockrate: { hz = 1000, tick = 1000, profhz = 8128, stathz = 127 } > kern.hz: 1000 x86 is similar. I think synchronization from using PLLs still gives unfair scheduling, but with multiple CPUs and often more cycles than can be used, no one cares about accidental synchronization or bothers to steal cycles using intentional synchronization. > # sysctl kern.timecounter > kern.timecounter.fast_gettime: 1 > kern.timecounter.tick: 1 > kern.timecounter.choice: timebase(0) dummy(-1000000) > kern.timecounter.hardware: timebase > kern.timecounter.alloweddeviation: 5 > kern.timecounter.stepwarnings: 0 > kern.timecounter.tc.timebase.quality: 0 > kern.timecounter.tc.timebase.frequency: 33333333 > kern.timecounter.tc.timebase.counter: 1144662532 > kern.timecounter.tc.timebase.mask: 4294967295 > > (The actual Time Base Register (tbr) i s 64 bits > and freebsd truncates it down.) > > # sysctl -a | grep 'cpu.*freq' > device cpufreq > debug.cpufreq.verbose: 0 > debug.cpufreq.lowest: 0 > dev.cpufreq.0.%parent: cpu3 > dev.cpufreq.0.%pnpinfo: > dev.cpufreq.0.%location: > dev.cpufreq.0.%driver: cpufreq > dev.cpufreq.0.%desc: > dev.cpufreq.%parent: > dev.cpu.3.freq_levels: 2500/-1 1250/-1 > dev.cpu.3.freq: 2500 > > So 2500 MHz / 33333333 Hz is very near 75 clock periods per > timebase counter value. Looks like it is exactly 75. Fractions are especially easy to guess and verify when they are integral. > I do sometimes have access to a Ryzen Threadripper 1950X based system: > > FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs > FreeBSD/SMP: 1 package(s) x 2 groups x 2 cache groups x 4 core(s) x 2 hardware threads > > but it is single=socket. It has . . . > ... Bruce From owner-freebsd-ppc@freebsd.org Fri Apr 5 08:40:19 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 92044156EB7C for ; Fri, 5 Apr 2019 08:40:19 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic316-8.consmr.mail.gq1.yahoo.com (sonic316-8.consmr.mail.gq1.yahoo.com [98.137.69.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 41F70850A8 for ; Fri, 5 Apr 2019 08:40:18 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: VXMvz5oVM1lVot.ZVGNj0UQSOyj6Ni7RoNkSCdjgRVXFBnGiPKOjpE5Zi6a5y.X qN9LPcDwHp_bK4aECSY19vbASlQ0YO1klqDnu_6De9akLMgvCY4nNbZ6unYeC_wflLQIEx5yVUfm W4EDvxzfaYetPY1SdR8LbXte7Z4wXLVQD3ESae70sBu3mxGJUx.RIa20UZv6zveRgRPzHMrUeUZ3 oknukAca4A9fXK8nCmLU35OCV_5MM4WZdi03iY_aEZSI4FpmsvShRoaNL5R5eknvLVVpKLkrfWuL 6j81MDZfSHZpJG_clu5cwlAgU6HJd0u04N5GbLVTuAMXUiE3KuWLBUORsGNU1NJLCBe2AL0Z.NtJ mbwsmxmvvS1T25yWiYRicNbseNcBQfZErelkOeX1yDkKIwwRAr6bE_V2E7Nd5Qxo9MuBrJcc7v6m aL0lkqyehIsJXiQ152wrlQW2idWPHVvo1uGS.SkzZBE1E14e547hGtZnkgzwVZYkDZeZ_zZdkEL9 anTz7ZNwo0U7zlbKV2X8tCdEyGk3gfYcT6.6u9a2813jx.79t4Hprc3xN6U8iliqcD5cQfwWos_p C_ZagEBWfau8GFgDmsURlaW_tt17u7eHad7dUunI2nICXaLRQPXgnFQ52QkuK.IT2_Vyx.mQgJAN cEv.LxZ34lI4jb8jghgcAzSy401vtLOkKc9rsbEeDssJ7g4PESmejxmoIh8XL4Yo12u5PY_r4VJu gkbJdhsj1A7R15S7sF56wAVRK2dBGuSKBiGCPY5scR2.CJBwUezUyTjFGARm9JEgz3xibvqERgLh tR2QwVa79V8ZuD7LLQBaOIZ3yYKgGwkGq53FZMkpKymnhxEaU5RcPIgIMv4CO3uqR5ahJzLV5D1X WNryDviRgCHNAvcZikXZQEA3jzRxZmRbqk4puAhPMrCp7RDIgGcpqRrJF4AvH711D84xHT6SLYwG DLBB.BZ0vFB1o4x64_GgxRARxik1LtuhaSIgD_RKXRihR1TV3DN.CQ7GtG4p4hICocTswXgZDnPL A2dUyvaH8_VLQ5Pn.vLaiO5QkjU6d.Qp7B0vvY7TrCG8FOKVo6ow5Gszm7QBr4SmkyVKwYoq.3o9 b8g-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic316.consmr.mail.gq1.yahoo.com with HTTP; Fri, 5 Apr 2019 08:40:10 +0000 Received: from c-67-170-167-181.hsd1.or.comcast.net (EHLO [192.168.1.113]) ([67.170.167.181]) by smtp428.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 4ebaa5d02eedc5f0d26af1d554561570; Fri, 05 Apr 2019 08:40:08 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed] From: Mark Millard In-Reply-To: <20190405150236.A959@besplex.bde.org> Date: Fri, 5 Apr 2019 01:40:07 -0700 Cc: Konstantin Belousov , freebsd-hackers Hackers , FreeBSD PowerPC ML Content-Transfer-Encoding: quoted-printable Message-Id: References: <20190303161635.GJ68879@kib.kiev.ua> <20190304043416.V5640@besplex.bde.org> <20190304114150.GM68879@kib.kiev.ua> <20190305031010.I4610@besplex.bde.org> <20190306172003.GD2492@kib.kiev.ua> <20190308001005.M2756@besplex.bde.org> <20190307222220.GK2492@kib.kiev.ua> <20190309144844.K1166@besplex.bde.org> <20190324110138.GR1923@kib.kiev.ua> <20190403070045.GW1923@kib.kiev.ua> <20190404011802.E2390@besplex.bde.org> <20190405150236.A959@besplex.bde.org> To: Bruce Evans X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 41F70850A8 X-Spamd-Bar: + X-Spamd-Result: default: False [1.83 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FREEMAIL_TO(0.00)[optusnet.com.au]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_SPAM_SHORT(0.81)[0.808,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(1.28)[ip: (4.75), ipnet: 98.137.64.0/21(0.95), asn: 36647(0.76), country: US(-0.06)]; NEURAL_SPAM_MEDIUM(0.03)[0.028,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.23)[0.226,0]; RCVD_IN_DNSWL_NONE(0.00)[32.69.137.98.list.dnswl.org : 127.0.5.0]; FREEMAIL_CC(0.00)[gmail.com] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2019 08:40:19 -0000 On 2019-Apr-4, at 21:38, Bruce Evans wrote: > On Thu, 4 Apr 2019, Mark Millard wrote: >=20 >> On 2019-Apr-3, at 08:47, Bruce Evans wrote: >>> . . . >>>=20 >>> I noticed (or better realized) a general problem with multiple >>> timehands. ntpd can slew the clock at up to 500 ppm, and at least = an >>> old version of it uses a rate of 50 ppm to fix up fairly small = drifts >>> in the milliseconds range. 500 ppm is enormous in CPU cycles -- it = is >>> 500 thousand nsec or 2 million cycles at 4GHz. Winding up the = timecounter >>> every 1 msec reduces this to only 2000 cycles. >>>=20 >>> More details of ordering and timing for 1 thread: >>> ... >> Thanks for the description of an example way that sbinuptime and >> the like might not give weakly increasing results. >>=20 >> Unfortunately, all the multi-socket contexts that I sometimes have >> access to are old PowerMacs. And, currently, the only such context >> is the G5 with 2 sockets, 2 cores per socket (powerpc64). So I've >> not been able to set up other types of examples to see if problems >> repeat. >>=20 >> I do not have access to a single-socket powerpc64 for contrast in >> that direction. >=20 > Testing 1 socket is time-consuming enough. Do these old systems > use the equivalent of an x86 TSC for the timecounter? With multiple > sockets, it isn't clear how even a hardware timer independent of the > CPUs can be distributed so as to appear to be monotonic on all cors. "The DEC frequency is based on the same implementation-dependent frequency that drives the time base." The frequency may well be fixed by the PowerMac G5 model implementation but is not fixed by the powerpc64 architecture. The Time Base Register (TBR) in a powerpc64 core (cpu in FreeBSD terms) increments at 33,333,333 Hz (nominal) and is 64 bits wide. Its value can be set (mttb instruction) and the boot sequence in FreeBSD does attempt to adjust as the FreeBSD CPU is brought-up/started. mftb is used to read the 64-bit value. FreeBSD masks it down to 32-bits to contribute to the time-counter. (Is that description sufficient for what you were after? I've never seen documentation of how the 33,333,333 MHz is produced.) As FreeBSD supports multi-socket, what are its criteria for a sufficient context for it to work with for supporting sbinuptime and the like? Is FreeBSD supposed to then make it appear that sbinputime and the like are weakly increasing, even as threads migrate between CPUs (cores, hw-threads)? >> One oddity is that the eventtimer's decrementer and timecounter >> may change (nearly) together: both change at 33,333,333 Hz, as if >> they are tied to the same clock (at least on one socket). >=20 > I think this is from a normal hardware implementation. On all of > my x86 systems with a TSC, the TSC frequency is an exact fractional > multiple of the i8254, the ACPI timer (if present) and the HPET (if > present). Only the RTC has an independent frequency. The fraction is > changed by changing the nominal TSC frequency in the BIOS, but is not > changed by temperature variations. This must be because most clocks = are > derived from a common clock using a PLL. I use this to calibrate all > clocks (except the RTC) by calibrating only 1. I'm not aware of the OpenFirmware having any control over the TBR-change frequency behavior. I've no evidence about any variability based on temperature. >> In case it helps with knowing how analogous your investigations >> are to the original problem context, I report the following. >>=20 >> If you do not care for such information, stop reading here. >>=20 >> # grep ntpd /etc/rc.conf >> ntpd_enable=3D"YES" >> ntpd_sync_on_start=3D"YES" >>=20 >> # sysctl kern.eventtimer >> kern.eventtimer.periodic: 0 >> kern.eventtimer.timer: decrementer >> kern.eventtimer.idletick: 0 >> kern.eventtimer.singlemul: 2 >> kern.eventtimer.choice: decrementer(1000) >> kern.eventtimer.et.decrementer.quality: 1000 >> kern.eventtimer.et.decrementer.frequency: 33333333 >> kern.eventtimer.et.decrementer.flags: 7 >>=20 >> # vmstat -ai | grep decrementer >> cpu0:decrementer 4451007 35 >> cpu3:decrementer 1466010 11 >> cpu2:decrementer 1481722 12 >> cpu1:decrementer 1478618 12 >=20 > Powerpc seems to have a PLL in software too. Event timers don't need = to > be very precise or accurate. powerpc64 sets the value to count down from (in the 32-bit DEC regsiter) via the mtdec instruction. I'm not ware of being=20 able to change the frequency that the countdown occurs at on the old PowerMac G5's. (mtdec is a form of mtspr SPRNUM,rs .) [Accessing the DEC's value is via a privileged instruction on powerpc64 vs. a non-privileged one on power, different instruction encodings but the same mnemonic unless mfspr is used directly as the mnemonic. Just a difference in one bit position in the SPUNUM in teh encoding.] >> (That last is from a basically-idle timeframe.) >>=20 >> # sysctl -a | grep hz >> kern.clockrate: { hz =3D 1000, tick =3D 1000, profhz =3D 8128, stathz = =3D 127 } >> kern.hz: 1000 >=20 > x86 is similar. I think synchronization from using PLLs still gives > unfair scheduling, but with multiple CPUs and often more cycles than = can > be used, no one cares about accidental synchronization or bothers to = steal > cycles using intentional synchronization. Good to know. >> # sysctl kern.timecounter >> kern.timecounter.fast_gettime: 1 >> kern.timecounter.tick: 1 >> kern.timecounter.choice: timebase(0) dummy(-1000000) >> kern.timecounter.hardware: timebase >> kern.timecounter.alloweddeviation: 5 >> kern.timecounter.stepwarnings: 0 >> kern.timecounter.tc.timebase.quality: 0 >> kern.timecounter.tc.timebase.frequency: 33333333 >> kern.timecounter.tc.timebase.counter: 1144662532 >> kern.timecounter.tc.timebase.mask: 4294967295 >>=20 >> (The actual Time Base Register (tbr) i s 64 bits >> and freebsd truncates it down.) >>=20 >> # sysctl -a | grep 'cpu.*freq' >> device cpufreq >> debug.cpufreq.verbose: 0 >> debug.cpufreq.lowest: 0 >> dev.cpufreq.0.%parent: cpu3 >> dev.cpufreq.0.%pnpinfo: >> dev.cpufreq.0.%location: >> dev.cpufreq.0.%driver: cpufreq >> dev.cpufreq.0.%desc: >> dev.cpufreq.%parent: >> dev.cpu.3.freq_levels: 2500/-1 1250/-1 >> dev.cpu.3.freq: 2500 >>=20 >> So 2500 MHz / 33333333 Hz is very near 75 clock periods per >> timebase counter value. >=20 > Looks like it is exactly 75. Fractions are especially easy to guess = and > verify when they are integral. I'm not sure what happens for DEC and TBR change frequencies if the 2500 MHz cpu frequency is dropped down to 1250 MHz. But as I understand my context, 1250 MHz is not in use at all, just 2500 MHz. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-ppc@freebsd.org Fri Apr 5 10:13:58 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D115A1570B54; Fri, 5 Apr 2019 10:13:57 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail104.syd.optusnet.com.au (mail104.syd.optusnet.com.au [211.29.132.246]) by mx1.freebsd.org (Postfix) with ESMTP id 3646587EDA; Fri, 5 Apr 2019 10:13:54 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from [192.168.0.102] (c110-21-101-228.carlnfd1.nsw.optusnet.com.au [110.21.101.228]) by mail104.syd.optusnet.com.au (Postfix) with ESMTPS id 0D37343A785; Fri, 5 Apr 2019 21:13:51 +1100 (AEDT) Date: Fri, 5 Apr 2019 21:13:50 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Mark Millard cc: Bruce Evans , Konstantin Belousov , freebsd-hackers Hackers , FreeBSD PowerPC ML Subject: Re: powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed] In-Reply-To: Message-ID: <20190405201055.I2396@besplex.bde.org> References: <20190303161635.GJ68879@kib.kiev.ua> <20190304043416.V5640@besplex.bde.org> <20190304114150.GM68879@kib.kiev.ua> <20190305031010.I4610@besplex.bde.org> <20190306172003.GD2492@kib.kiev.ua> <20190308001005.M2756@besplex.bde.org> <20190307222220.GK2492@kib.kiev.ua> <20190309144844.K1166@besplex.bde.org> <20190324110138.GR1923@kib.kiev.ua> <20190403070045.GW1923@kib.kiev.ua> <20190404011802.E2390@besplex.bde.org> <20190405150236.A959@besplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.2 cv=P6RKvmIu c=1 sm=1 tr=0 cx=a_idp_d a=PalzARQSbocsUSjMRkwAPg==:117 a=PalzARQSbocsUSjMRkwAPg==:17 a=jpOVt7BSZ2e4Z31A5e1TngXxSK0=:19 a=kj9zAlcOel0A:10 a=PO7r1zJSAAAA:8 a=E-FLDMUcsM50LrUROKkA:9 a=CjuIK1q_8ugA:10 X-Rspamd-Queue-Id: 3646587EDA X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of brde@optusnet.com.au designates 211.29.132.246 as permitted sender) smtp.mailfrom=brde@optusnet.com.au X-Spamd-Result: default: False [-6.01 / 15.00]; R_SPF_ALLOW(-0.20)[+ip4:211.29.132.0/23]; FREEMAIL_FROM(0.00)[optusnet.com.au]; RCPT_COUNT_FIVE(0.00)[5]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: extmail.optusnet.com.au]; NEURAL_HAM_SHORT(-0.98)[-0.983,0]; FREEMAIL_TO(0.00)[yahoo.com]; RCVD_NO_TLS_LAST(0.10)[]; RCVD_IN_DNSWL_LOW(-0.10)[246.132.29.211.list.dnswl.org : 127.0.5.1]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[optusnet.com.au]; ASN(0.00)[asn:4804, ipnet:211.28.0.0/14, country:AU]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[optusnet.com.au]; TO_MATCH_ENVRCPT_SOME(0.00)[]; IP_SCORE(-2.72)[ip: (-7.39), ipnet: 211.28.0.0/14(-3.42), asn: 4804(-2.72), country: AU(-0.04)]; FREEMAIL_CC(0.00)[optusnet.com.au]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2019 10:13:58 -0000 On Fri, 5 Apr 2019, Mark Millard wrote: > On 2019-Apr-4, at 21:38, Bruce Evans wrote: > >> On Thu, 4 Apr 2019, Mark Millard wrote: >>> ... >>> Unfortunately, all the multi-socket contexts that I sometimes have >>> access to are old PowerMacs. And, currently, the only such context >>> is the G5 with 2 sockets, 2 cores per socket (powerpc64). So I've >>> not been able to set up other types of examples to see if problems >>> repeat. >>> >>> I do not have access to a single-socket powerpc64 for contrast in >>> that direction. >> >> Testing 1 socket is time-consuming enough. Do these old systems >> use the equivalent of an x86 TSC for the timecounter? With multiple >> sockets, it isn't clear how even a hardware timer independent of the >> CPUs can be distributed so as to appear to be monotonic on all cors. > > "The DEC frequency is based on the same implementation-dependent > frequency that drives the time base." The frequency may well be > fixed by the PowerMac G5 model implementation but is not fixed > by the powerpc64 architecture. > > The Time Base Register (TBR) in a powerpc64 core (cpu in FreeBSD terms) > increments at 33,333,333 Hz (nominal) and is 64 bits wide. Its value > can be set (mttb instruction) and the boot sequence in FreeBSD does > attempt to adjust as the FreeBSD CPU is brought-up/started. mftb is > used to read the 64-bit value. FreeBSD masks it down to 32-bits to > contribute to the time-counter. > > (Is that description sufficient for what you were after? I've never > seen documentation of how the 33,333,333 MHz is produced.) This seems to be equivalent to an x86 TSC. x86 P-state-invariant TSCs apparently use a 100 MHz clock which corresponds even more closely to this, except for historical reasons this clock is scaled and interpolated to a clock resembling the CPU cycle count at a nominal frequency. > As FreeBSD supports multi-socket, what are its criteria for a sufficient > context for it to work with for supporting sbinuptime and the like? Is > FreeBSD supposed to then make it appear that sbinputime and the like are > weakly increasing, even as threads migrate between CPUs (cores, > hw-threads)? CLOCK_MONOTONIC has to be monotonic, but it is only clear what this means within a single thread: sequential clock_gettime() calls must occur in program order and the results must be consistent with that order. Across threads, I think it should mean that the results must be consistent with any order established using any supported ordering methods. >... >>> One oddity is that the eventtimer's decrementer and timecounter >>> may change (nearly) together: both change at 33,333,333 Hz, as if >>> they are tied to the same clock (at least on one socket). >> >> I think this is from a normal hardware implementation. On all of >> my x86 systems with a TSC, the TSC frequency is an exact fractional >> multiple of the i8254, the ACPI timer (if present) and the HPET (if >> present). Only the RTC has an independent frequency. The fraction is >> changed by changing the nominal TSC frequency in the BIOS, but is not >> changed by temperature variations. This must be because most clocks are >> derived from a common clock using a PLL. I use this to calibrate all >> clocks (except the RTC) by calibrating only 1. > > I'm not aware of the OpenFirmware having any control over the > TBR-change frequency behavior. I've no evidence about any variability > based on temperature. Temperature changes usually affect the actual frequency but not the nominal frequency. Bruce From owner-freebsd-ppc@freebsd.org Fri Apr 5 11:35:17 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A17221573ACD; Fri, 5 Apr 2019 11:35:17 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C87618CFE7; Fri, 5 Apr 2019 11:35:16 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id x35BZ6jL002604 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 5 Apr 2019 14:35:09 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua x35BZ6jL002604 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id x35BZ4Vo002581; Fri, 5 Apr 2019 14:35:04 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 5 Apr 2019 14:35:04 +0300 From: Konstantin Belousov To: Bruce Evans Cc: Mark Millard , freebsd-hackers Hackers , FreeBSD PowerPC ML Subject: Re: powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed] Message-ID: <20190405113504.GA1923@kib.kiev.ua> References: <20190307222220.GK2492@kib.kiev.ua> <20190309144844.K1166@besplex.bde.org> <20190324110138.GR1923@kib.kiev.ua> <20190403070045.GW1923@kib.kiev.ua> <20190404011802.E2390@besplex.bde.org> <20190405150236.A959@besplex.bde.org> <20190405201055.I2396@besplex.bde.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20190405201055.I2396@besplex.bde.org> User-Agent: Mutt/1.11.4 (2019-03-13) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2019 11:35:18 -0000 On Fri, Apr 05, 2019 at 09:13:50PM +1100, Bruce Evans wrote: > On Fri, 5 Apr 2019, Mark Millard wrote: > > > On 2019-Apr-4, at 21:38, Bruce Evans wrote: > > > >> On Thu, 4 Apr 2019, Mark Millard wrote: > >>> ... > >>> Unfortunately, all the multi-socket contexts that I sometimes have > >>> access to are old PowerMacs. And, currently, the only such context > >>> is the G5 with 2 sockets, 2 cores per socket (powerpc64). So I've > >>> not been able to set up other types of examples to see if problems > >>> repeat. > >>> > >>> I do not have access to a single-socket powerpc64 for contrast in > >>> that direction. > >> > >> Testing 1 socket is time-consuming enough. Do these old systems > >> use the equivalent of an x86 TSC for the timecounter? With multiple > >> sockets, it isn't clear how even a hardware timer independent of the > >> CPUs can be distributed so as to appear to be monotonic on all cors. > > > > "The DEC frequency is based on the same implementation-dependent > > frequency that drives the time base." The frequency may well be > > fixed by the PowerMac G5 model implementation but is not fixed > > by the powerpc64 architecture. > > > > The Time Base Register (TBR) in a powerpc64 core (cpu in FreeBSD terms) > > increments at 33,333,333 Hz (nominal) and is 64 bits wide. Its value > > can be set (mttb instruction) and the boot sequence in FreeBSD does > > attempt to adjust as the FreeBSD CPU is brought-up/started. mftb is > > used to read the 64-bit value. FreeBSD masks it down to 32-bits to > > contribute to the time-counter. > > > > (Is that description sufficient for what you were after? I've never > > seen documentation of how the 33,333,333 MHz is produced.) > > This seems to be equivalent to an x86 TSC. It is equivalent from the interface PoV. I saw some references that Powers do have per-core PLLs, the best I can find now is https://stackoverflow.com/questions/43844438/are-multiple-clock-domains-common-in-modern-processors For Intel there is no much information, the best guess is that TSC is in uncore and effectively shared by all cores. See also https://software.intel.com/en-us/articles/best-timing-function-for-measuring-ipp-api-timing/ > > x86 P-state-invariant TSCs apparently use a 100 MHz clock which corresponds > even more closely to this, except for historical reasons this clock is > scaled and interpolated to a clock resembling the CPU cycle count at a > nominal frequency. For Intel designs, there is indeed a single-source 100MHz signal which is distributed over all consumers using clock fan-out buffers like DB1900Z. > > > As FreeBSD supports multi-socket, what are its criteria for a sufficient > > context for it to work with for supporting sbinuptime and the like? Is > > FreeBSD supposed to then make it appear that sbinputime and the like are > > weakly increasing, even as threads migrate between CPUs (cores, > > hw-threads)? > > CLOCK_MONOTONIC has to be monotonic, but it is only clear what this means > within a single thread: sequential clock_gettime() calls must occur in > program order and the results must be consistent with that order. Across > threads, I think it should mean that the results must be consistent with > any order established using any supported ordering methods. > > >... > >>> One oddity is that the eventtimer's decrementer and timecounter > >>> may change (nearly) together: both change at 33,333,333 Hz, as if > >>> they are tied to the same clock (at least on one socket). > >> > >> I think this is from a normal hardware implementation. On all of > >> my x86 systems with a TSC, the TSC frequency is an exact fractional > >> multiple of the i8254, the ACPI timer (if present) and the HPET (if > >> present). Only the RTC has an independent frequency. The fraction is > >> changed by changing the nominal TSC frequency in the BIOS, but is not > >> changed by temperature variations. This must be because most clocks are > >> derived from a common clock using a PLL. I use this to calibrate all > >> clocks (except the RTC) by calibrating only 1. > > > > I'm not aware of the OpenFirmware having any control over the > > TBR-change frequency behavior. I've no evidence about any variability > > based on temperature. > > Temperature changes usually affect the actual frequency but not the > nominal frequency. > > Bruce From owner-freebsd-ppc@freebsd.org Fri Apr 5 11:39:23 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 60FC01573DEB; Fri, 5 Apr 2019 11:39:23 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AC98A8D528; Fri, 5 Apr 2019 11:39:22 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id x35BdCH1003695 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 5 Apr 2019 14:39:15 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua x35BdCH1003695 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id x35BdCH7003692; Fri, 5 Apr 2019 14:39:12 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 5 Apr 2019 14:39:12 +0300 From: Konstantin Belousov To: Bruce Evans Cc: Michael Tuexen , freebsd-hackers Hackers , FreeBSD PowerPC ML Subject: Re: powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed] Message-ID: <20190405113912.GB1923@kib.kiev.ua> References: <20190304114150.GM68879@kib.kiev.ua> <20190305031010.I4610@besplex.bde.org> <20190306172003.GD2492@kib.kiev.ua> <20190308001005.M2756@besplex.bde.org> <20190307222220.GK2492@kib.kiev.ua> <20190309144844.K1166@besplex.bde.org> <20190324110138.GR1923@kib.kiev.ua> <20190403070045.GW1923@kib.kiev.ua> <20190404011802.E2390@besplex.bde.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20190404011802.E2390@besplex.bde.org> User-Agent: Mutt/1.11.4 (2019-03-13) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2019 11:39:23 -0000 On Thu, Apr 04, 2019 at 02:47:34AM +1100, Bruce Evans wrote: > I noticed (or better realized) a general problem with multiple > timehands. ntpd can slew the clock at up to 500 ppm, and at least an > old version of it uses a rate of 50 ppm to fix up fairly small drifts > in the milliseconds range. 500 ppm is enormous in CPU cycles -- it is > 500 thousand nsec or 2 million cycles at 4GHz. Winding up the timecounter > every 1 msec reduces this to only 2000 cycles. > > More details of ordering and timing for 1 thread: > - thread N calls binuptime() and it loads timehands > - another or even the same thread runs tc_windup(). This modifies timehands. > - thread N is preempted for a long time, but less than the time for > updates > - thread N checks the generation count. Since this is for the timehands > contents and not for the timehands pointer, it hasn't changed, so the > old timehands is used > - and instant later, the same thread calls binuptime again() and uses the > new timehands > - now suppose only 2 timehands (as in -current) the worst (?) case of a > slew of 500 ppm for the old timehands and -500 ppm for the new timehands > and almost the worst case of 10 msec for the oldness of the old timehands > relative to the new timehands, with the new timehands about to be updated > after 10 msec (assume perfectly periodiodic updates every 10 msec). The > calculated times are: > > old bintime = old_base + (20 msec) * (1 + 500e-6) > new base = old_base + 10 msec * (1 + 500e-6) # calc by tc_windup() > new bintime = new_base + (10 msec) * (1 - 500e-6) + epsilon > > error = epsilon - (20 msec) * 500e-6 = epsilon - 10000 nsec > > Errors in the negative direction are most harmful. ntpd normally doesn't > change the skew as much as that in one step, but it is easy for adjtime(2) > to change the skew like that and there are no reasonable microadjustments > that would accidentally work around this kernel bug (it seems unreasonable > to limit the skew to 1 ppm and that still gives an error of epsilon + 20 > nsec. > > phk didn't want to slow down timecounters using extra code to make > them them monotonic and coherent (getbinuptime() is incoherent with > binuptime() since it former lags the latter by the update interval), > but this wouldn't be very slow within a thread. > > Monotonicity across threads is a larger problem and not helped by using > a faked forced monotonic time within threads. > > So it seems best to fix the above problem by moving the generation count > from the timehands contents to the timehands pointer, and maybe also > reduce the number of timehands to 1. With 2 timehands, this gives a > shorter race: > > - thread N loads timehands > - tc_windup() > - thread N preempted > - thread N uses old timehands > - case tc_windup() completes first: no problem -- thread N checks the > generation count on the pointer and loops > - case binuptime() completes first: lost a race -- binuptime() is off > by approx * . > > The main point of having multiple timehands (after introducing the per- > timehands generation count) is to avoid blocking thread N during the > update, but this doesn't actually work, even for only 2 timehands and > a global generation count. You are describing the generic race between reader and writer. The same race would exist even with one timehand (and/or one global generation counter), where ntp adjustment might come earlier or later of some consumer accessing the timehands. If timehand instance was read before tc_windup() run but code consumed the result after the windup, it might appear as if time went backward, and this cannot be fixed without either re-reading the time after time-depended calculations were done and restarting, or some globabl lock ensuring serialization. From owner-freebsd-ppc@freebsd.org Fri Apr 5 12:52:40 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8572F157664F; Fri, 5 Apr 2019 12:52:40 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail106.syd.optusnet.com.au (mail106.syd.optusnet.com.au [211.29.132.42]) by mx1.freebsd.org (Postfix) with ESMTP id E92F190085; Fri, 5 Apr 2019 12:52:39 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from [192.168.0.102] (c110-21-101-228.carlnfd1.nsw.optusnet.com.au [110.21.101.228]) by mail106.syd.optusnet.com.au (Postfix) with ESMTPS id 2CA103DC95D; Fri, 5 Apr 2019 23:52:29 +1100 (AEDT) Date: Fri, 5 Apr 2019 23:52:27 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Konstantin Belousov cc: Bruce Evans , Michael Tuexen , freebsd-hackers Hackers , FreeBSD PowerPC ML Subject: Re: powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed] In-Reply-To: <20190405113912.GB1923@kib.kiev.ua> Message-ID: <20190405230717.D3383@besplex.bde.org> References: <20190304114150.GM68879@kib.kiev.ua> <20190305031010.I4610@besplex.bde.org> <20190306172003.GD2492@kib.kiev.ua> <20190308001005.M2756@besplex.bde.org> <20190307222220.GK2492@kib.kiev.ua> <20190309144844.K1166@besplex.bde.org> <20190324110138.GR1923@kib.kiev.ua> <20190403070045.GW1923@kib.kiev.ua> <20190404011802.E2390@besplex.bde.org> <20190405113912.GB1923@kib.kiev.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.2 cv=P6RKvmIu c=1 sm=1 tr=0 cx=a_idp_d a=PalzARQSbocsUSjMRkwAPg==:117 a=PalzARQSbocsUSjMRkwAPg==:17 a=jpOVt7BSZ2e4Z31A5e1TngXxSK0=:19 a=kj9zAlcOel0A:10 a=MfOOJuOeLnfdOtmGB0YA:9 a=-f6ZOhMsL1iUIJkq:21 a=gaGQGk8RQ06_Sp8i:21 a=CjuIK1q_8ugA:10 X-Rspamd-Queue-Id: E92F190085 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.98 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.98)[-0.982,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2019 12:52:40 -0000 On Fri, 5 Apr 2019, Konstantin Belousov wrote: > On Thu, Apr 04, 2019 at 02:47:34AM +1100, Bruce Evans wrote: >> I noticed (or better realized) a general problem with multiple >> timehands. ntpd can slew the clock at up to 500 ppm, and at least an >> old version of it uses a rate of 50 ppm to fix up fairly small drifts >> in the milliseconds range. 500 ppm is enormous in CPU cycles -- it is >> 500 thousand nsec or 2 million cycles at 4GHz. Winding up the timecounter >> every 1 msec reduces this to only 2000 cycles. >> ... >> The main point of having multiple timehands (after introducing the per- >> timehands generation count) is to avoid blocking thread N during the >> update, but this doesn't actually work, even for only 2 timehands and >> a global generation count. > > You are describing the generic race between reader and writer. The same > race would exist even with one timehand (and/or one global generation > counter), where ntp adjustment might come earlier or later of some > consumer accessing the timehands. If timehand instance was read before > tc_windup() run but code consumed the result after the windup, it might > appear as if time went backward, and this cannot be fixed without either > re-reading the time after time-depended calculations were done and > restarting, or some globabl lock ensuring serialization. With 1 timehand, its generation count would be global. I think its ordering is strong enough to ensure serialization. I think the fix in the kernel to use a global generation count (with > 1 timehands) is simply s/th->th_generation/tc_generation/g. Oops, that makes multiple timehands useless and gives some blocking. The critical case is when a new timehands is under construction. The old timehands becomes unsafe to use when the writer (tc_windup()) updates the offset. tc_windup() currently sets th_generation to 0 to indicate that the new timehands is unsafe to use. Doing the same with a global tc_generation would give serialization at the cost of busy-waiting for tc_generation to become nonzero. It would indicate that all timehands are unsafe to use. In the library, does it just work to put the global generation count in the shared page? Bruce From owner-freebsd-ppc@freebsd.org Fri Apr 5 13:21:38 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AAA831576E5C; Fri, 5 Apr 2019 13:21:38 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1F96490C20; Fri, 5 Apr 2019 13:21:38 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id x35DLSfb011961 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 5 Apr 2019 16:21:31 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua x35DLSfb011961 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id x35DLSX2011960; Fri, 5 Apr 2019 16:21:28 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 5 Apr 2019 16:21:28 +0300 From: Konstantin Belousov To: Bruce Evans Cc: Michael Tuexen , freebsd-hackers Hackers , FreeBSD PowerPC ML Subject: Re: powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed] Message-ID: <20190405132128.GD1923@kib.kiev.ua> References: <20190306172003.GD2492@kib.kiev.ua> <20190308001005.M2756@besplex.bde.org> <20190307222220.GK2492@kib.kiev.ua> <20190309144844.K1166@besplex.bde.org> <20190324110138.GR1923@kib.kiev.ua> <20190403070045.GW1923@kib.kiev.ua> <20190404011802.E2390@besplex.bde.org> <20190405113912.GB1923@kib.kiev.ua> <20190405230717.D3383@besplex.bde.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20190405230717.D3383@besplex.bde.org> User-Agent: Mutt/1.11.4 (2019-03-13) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2019 13:21:38 -0000 On Fri, Apr 05, 2019 at 11:52:27PM +1100, Bruce Evans wrote: > On Fri, 5 Apr 2019, Konstantin Belousov wrote: > > > On Thu, Apr 04, 2019 at 02:47:34AM +1100, Bruce Evans wrote: > >> I noticed (or better realized) a general problem with multiple > >> timehands. ntpd can slew the clock at up to 500 ppm, and at least an > >> old version of it uses a rate of 50 ppm to fix up fairly small drifts > >> in the milliseconds range. 500 ppm is enormous in CPU cycles -- it is > >> 500 thousand nsec or 2 million cycles at 4GHz. Winding up the timecounter > >> every 1 msec reduces this to only 2000 cycles. > >> ... > >> The main point of having multiple timehands (after introducing the per- > >> timehands generation count) is to avoid blocking thread N during the > >> update, but this doesn't actually work, even for only 2 timehands and > >> a global generation count. > > > > You are describing the generic race between reader and writer. The same > > race would exist even with one timehand (and/or one global generation > > counter), where ntp adjustment might come earlier or later of some > > consumer accessing the timehands. If timehand instance was read before > > tc_windup() run but code consumed the result after the windup, it might > > appear as if time went backward, and this cannot be fixed without either > > re-reading the time after time-depended calculations were done and > > restarting, or some globabl lock ensuring serialization. > > With 1 timehand, its generation count would be global. I think its ordering > is strong enough to ensure serialization. Yes, single timehands result in global generation. But it would not solve the same race appearing in slightly different manner, as I described above. If reader finished while generation number in th was not yet reset, but caller uses the result after tc_windup(), the effect is same as if we have two th's and reader used the outdated one. > > I think the fix in the kernel to use a global generation count (with > 1 > timehands) is simply s/th->th_generation/tc_generation/g. Oops, that > makes multiple timehands useless and gives some blocking. The critical > case is when a new timehands is under construction. The old timehands > becomes unsafe to use when the writer (tc_windup()) updates the offset. > tc_windup() currently sets th_generation to 0 to indicate that the new > timehands is unsafe to use. Doing the same with a global tc_generation > would give serialization at the cost of busy-waiting for tc_generation > to become nonzero. It would indicate that all timehands are unsafe > to use. > > In the library, does it just work to put the global generation count in > the shared page? libc always reload tk_current in the loop, so it works with any number of vdso timehands greater or equal to one. From owner-freebsd-ppc@freebsd.org Fri Apr 5 14:01:25 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6DEF21577F9E; Fri, 5 Apr 2019 14:01:25 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail104.syd.optusnet.com.au (mail104.syd.optusnet.com.au [211.29.132.246]) by mx1.freebsd.org (Postfix) with ESMTP id B4F33928FD; Fri, 5 Apr 2019 14:01:24 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from [192.168.0.102] (c110-21-101-228.carlnfd1.nsw.optusnet.com.au [110.21.101.228]) by mail104.syd.optusnet.com.au (Postfix) with ESMTPS id AD7E5438988; Sat, 6 Apr 2019 01:01:20 +1100 (AEDT) Date: Sat, 6 Apr 2019 01:01:19 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Konstantin Belousov cc: Michael Tuexen , freebsd-hackers Hackers , FreeBSD PowerPC ML Subject: Re: powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed] In-Reply-To: <20190405132128.GD1923@kib.kiev.ua> Message-ID: <20190406003907.C3872@besplex.bde.org> References: <20190306172003.GD2492@kib.kiev.ua> <20190308001005.M2756@besplex.bde.org> <20190307222220.GK2492@kib.kiev.ua> <20190309144844.K1166@besplex.bde.org> <20190324110138.GR1923@kib.kiev.ua> <20190403070045.GW1923@kib.kiev.ua> <20190404011802.E2390@besplex.bde.org> <20190405113912.GB1923@kib.kiev.ua> <20190405230717.D3383@besplex.bde.org> <20190405132128.GD1923@kib.kiev.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.2 cv=P6RKvmIu c=1 sm=1 tr=0 cx=a_idp_d a=PalzARQSbocsUSjMRkwAPg==:117 a=PalzARQSbocsUSjMRkwAPg==:17 a=jpOVt7BSZ2e4Z31A5e1TngXxSK0=:19 a=kj9zAlcOel0A:10 a=xRf97b5gqAnxw9LQlxwA:9 a=q9dmEwd7LJ99Beko:21 a=KBmFWjL1gV5mJfMI:21 a=CjuIK1q_8ugA:10 X-Rspamd-Queue-Id: B4F33928FD X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.94 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.94)[-0.943,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2019 14:01:25 -0000 On Fri, 5 Apr 2019, Konstantin Belousov wrote: > On Fri, Apr 05, 2019 at 11:52:27PM +1100, Bruce Evans wrote: >> On Fri, 5 Apr 2019, Konstantin Belousov wrote: >> >>> On Thu, Apr 04, 2019 at 02:47:34AM +1100, Bruce Evans wrote: >>>> I noticed (or better realized) a general problem with multiple >>>> timehands. ntpd can slew the clock at up to 500 ppm, and at least an >>>> old version of it uses a rate of 50 ppm to fix up fairly small drifts >>>> in the milliseconds range. 500 ppm is enormous in CPU cycles -- it is >>>> 500 thousand nsec or 2 million cycles at 4GHz. Winding up the timecounter >>>> every 1 msec reduces this to only 2000 cycles. >>>> ... >>>> The main point of having multiple timehands (after introducing the per- >>>> timehands generation count) is to avoid blocking thread N during the >>>> update, but this doesn't actually work, even for only 2 timehands and >>>> a global generation count. >>> >>> You are describing the generic race between reader and writer. The same >>> race would exist even with one timehand (and/or one global generation >>> counter), where ntp adjustment might come earlier or later of some >>> consumer accessing the timehands. If timehand instance was read before >>> tc_windup() run but code consumed the result after the windup, it might >>> appear as if time went backward, and this cannot be fixed without either >>> re-reading the time after time-depended calculations were done and >>> restarting, or some globabl lock ensuring serialization. >> >> With 1 timehand, its generation count would be global. I think its ordering >> is strong enough to ensure serialization. > Yes, single timehands result in global generation. But it would not solve > the same race appearing in slightly different manner, as I described above. > If reader finished while generation number in th was not yet reset, but > caller uses the result after tc_windup(), the effect is same as if we > have two th's and reader used the outdated one. You described it too concisely for me to understand :-). I now see that a single generation count doesn't give serialization. I thought that setting the generation to 0 at the start of tc_windup() serialized the reader and writer. But all it does is prevent use of the results of the windup while only some of them are visible. If the setting the generation count to 0 doesn't become before tc_windup() reads the hardware timecounter, then this read may be before other reads using the old timehands, but it needs to be after. A not so good fix for this is to wait a bit after setting the generation count to 0, so that the change becomes visible on all CPUs. Bruce From owner-freebsd-ppc@freebsd.org Fri Apr 5 14:36:29 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 06123157890C; Fri, 5 Apr 2019 14:36:29 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6A7CB94089; Fri, 5 Apr 2019 14:36:28 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id x35EaJdK029276 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 5 Apr 2019 17:36:22 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua x35EaJdK029276 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id x35EaJKI029275; Fri, 5 Apr 2019 17:36:19 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 5 Apr 2019 17:36:19 +0300 From: Konstantin Belousov To: Bruce Evans Cc: Michael Tuexen , freebsd-hackers Hackers , FreeBSD PowerPC ML Subject: Re: powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed] Message-ID: <20190405143619.GF1923@kib.kiev.ua> References: <20190307222220.GK2492@kib.kiev.ua> <20190309144844.K1166@besplex.bde.org> <20190324110138.GR1923@kib.kiev.ua> <20190403070045.GW1923@kib.kiev.ua> <20190404011802.E2390@besplex.bde.org> <20190405113912.GB1923@kib.kiev.ua> <20190405230717.D3383@besplex.bde.org> <20190405132128.GD1923@kib.kiev.ua> <20190406003907.C3872@besplex.bde.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20190406003907.C3872@besplex.bde.org> User-Agent: Mutt/1.11.4 (2019-03-13) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2019 14:36:29 -0000 On Sat, Apr 06, 2019 at 01:01:19AM +1100, Bruce Evans wrote: > On Fri, 5 Apr 2019, Konstantin Belousov wrote: > > > On Fri, Apr 05, 2019 at 11:52:27PM +1100, Bruce Evans wrote: > >> On Fri, 5 Apr 2019, Konstantin Belousov wrote: > >> > >>> On Thu, Apr 04, 2019 at 02:47:34AM +1100, Bruce Evans wrote: > >>>> I noticed (or better realized) a general problem with multiple > >>>> timehands. ntpd can slew the clock at up to 500 ppm, and at least an > >>>> old version of it uses a rate of 50 ppm to fix up fairly small drifts > >>>> in the milliseconds range. 500 ppm is enormous in CPU cycles -- it is > >>>> 500 thousand nsec or 2 million cycles at 4GHz. Winding up the timecounter > >>>> every 1 msec reduces this to only 2000 cycles. > >>>> ... > >>>> The main point of having multiple timehands (after introducing the per- > >>>> timehands generation count) is to avoid blocking thread N during the > >>>> update, but this doesn't actually work, even for only 2 timehands and > >>>> a global generation count. > >>> > >>> You are describing the generic race between reader and writer. The same > >>> race would exist even with one timehand (and/or one global generation > >>> counter), where ntp adjustment might come earlier or later of some > >>> consumer accessing the timehands. If timehand instance was read before > >>> tc_windup() run but code consumed the result after the windup, it might > >>> appear as if time went backward, and this cannot be fixed without either > >>> re-reading the time after time-depended calculations were done and > >>> restarting, or some globabl lock ensuring serialization. > >> > >> With 1 timehand, its generation count would be global. I think its ordering > >> is strong enough to ensure serialization. > > Yes, single timehands result in global generation. But it would not solve > > the same race appearing in slightly different manner, as I described above. > > If reader finished while generation number in th was not yet reset, but > > caller uses the result after tc_windup(), the effect is same as if we > > have two th's and reader used the outdated one. > > You described it too concisely for me to understand :-). > > I now see that a single generation count doesn't give serialization. I > thought that setting the generation to 0 at the start of tc_windup() > serialized the reader and writer. But all it does is prevent use of the > results of the windup while only some of them are visible. If the > setting the generation count to 0 doesn't become before tc_windup() reads > the hardware timecounter, then this read may be before other reads using > the old timehands, but it needs to be after. If we have either single th or global gen counter, current code would become serialized, but this is not what I am about. Lets assume, for the sake of the discussion only, that all readers take the same spinlock as tc_windup (i.e. tc_setclock_mtx). It is still possible that reader unlocked the mutex, tc_windup() kicked in, locked the mutex and moved timehands (as you noted, this might even happen on the same CPU), and only then the reader continues. For consumer of bintime() or any other function' result, it looks exactly the same as if we did not serialized with writer but used outdated timehands. Let me formulate this diffeently: as far as consumer of the bintime() result does not serialize itself with tc_windup(), serializing bintime() itself against tc_windup() does not close the race, but it is not obvious that the race matters. Either we should just accept the race as we currently do, or readers must take the spinlock where the exact value of the current time is important, or readers must re-read the time after doing something important, and redo if the new measuremedtime is outside the acceptable range. > > A not so good fix for this is to wait a bit after setting the generation > count to 0, so that the change becomes visible on all CPUs. > > Bruce From owner-freebsd-ppc@freebsd.org Fri Apr 5 15:26:17 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0774B157A302; Fri, 5 Apr 2019 15:26:17 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail105.syd.optusnet.com.au (mail105.syd.optusnet.com.au [211.29.132.249]) by mx1.freebsd.org (Postfix) with ESMTP id 4AA62965F1; Fri, 5 Apr 2019 15:26:15 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from [192.168.0.102] (c110-21-101-228.carlnfd1.nsw.optusnet.com.au [110.21.101.228]) by mail105.syd.optusnet.com.au (Postfix) with ESMTPS id AC183105BAD5; Sat, 6 Apr 2019 02:26:11 +1100 (AEDT) Date: Sat, 6 Apr 2019 02:26:11 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Konstantin Belousov cc: Michael Tuexen , freebsd-hackers Hackers , FreeBSD PowerPC ML Subject: Re: powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed] In-Reply-To: <20190405143619.GF1923@kib.kiev.ua> Message-ID: <20190406014724.X4174@besplex.bde.org> References: <20190307222220.GK2492@kib.kiev.ua> <20190309144844.K1166@besplex.bde.org> <20190324110138.GR1923@kib.kiev.ua> <20190403070045.GW1923@kib.kiev.ua> <20190404011802.E2390@besplex.bde.org> <20190405113912.GB1923@kib.kiev.ua> <20190405230717.D3383@besplex.bde.org> <20190405132128.GD1923@kib.kiev.ua> <20190406003907.C3872@besplex.bde.org> <20190405143619.GF1923@kib.kiev.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.2 cv=FNpr/6gs c=1 sm=1 tr=0 cx=a_idp_d a=PalzARQSbocsUSjMRkwAPg==:117 a=PalzARQSbocsUSjMRkwAPg==:17 a=jpOVt7BSZ2e4Z31A5e1TngXxSK0=:19 a=kj9zAlcOel0A:10 a=WpN8qSVOMLHOXD53BCgA:9 a=_yIFZNw7G-84PWPD:21 a=tYTChskUWkClw5JK:21 a=CjuIK1q_8ugA:10 X-Rspamd-Queue-Id: 4AA62965F1 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.93 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[]; NEURAL_HAM_SHORT(-0.93)[-0.928,0] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2019 15:26:17 -0000 On Fri, 5 Apr 2019, Konstantin Belousov wrote: > On Sat, Apr 06, 2019 at 01:01:19AM +1100, Bruce Evans wrote: >> On Fri, 5 Apr 2019, Konstantin Belousov wrote: >> >>> On Fri, Apr 05, 2019 at 11:52:27PM +1100, Bruce Evans wrote: >>>> On Fri, 5 Apr 2019, Konstantin Belousov wrote: >>>> >>>>> On Thu, Apr 04, 2019 at 02:47:34AM +1100, Bruce Evans wrote: >>>>>> I noticed (or better realized) a general problem with multiple >>>>>> timehands. ntpd can slew the clock at up to 500 ppm, and at least an >>>>>> old version of it uses a rate of 50 ppm to fix up fairly small drifts >>>>>> in the milliseconds range. 500 ppm is enormous in CPU cycles -- it is >>>>>> 500 thousand nsec or 2 million cycles at 4GHz. Winding up the timecounter >>>>>> every 1 msec reduces this to only 2000 cycles. >>>>>> ... >>>>>> The main point of having multiple timehands (after introducing the per- >>>>>> timehands generation count) is to avoid blocking thread N during the >>>>>> update, but this doesn't actually work, even for only 2 timehands and >>>>>> a global generation count. >>>>> >>>>> You are describing the generic race between reader and writer. The same >>>>> race would exist even with one timehand (and/or one global generation >>>>> counter), where ntp adjustment might come earlier or later of some >>>>> consumer accessing the timehands. If timehand instance was read before >>>>> tc_windup() run but code consumed the result after the windup, it might >>>>> appear as if time went backward, and this cannot be fixed without either >>>>> re-reading the time after time-depended calculations were done and >>>>> restarting, or some globabl lock ensuring serialization. >>>> >>>> With 1 timehand, its generation count would be global. I think its ordering >>>> is strong enough to ensure serialization. >>> Yes, single timehands result in global generation. But it would not solve >>> the same race appearing in slightly different manner, as I described above. >>> If reader finished while generation number in th was not yet reset, but >>> caller uses the result after tc_windup(), the effect is same as if we >>> have two th's and reader used the outdated one. >> >> You described it too concisely for me to understand :-). >> >> I now see that a single generation count doesn't give serialization. I >> thought that setting the generation to 0 at the start of tc_windup() >> serialized the reader and writer. But all it does is prevent use of the >> results of the windup while only some of them are visible. If the >> setting the generation count to 0 doesn't become before tc_windup() reads >> the hardware timecounter, then this read may be before other reads using >> the old timehands, but it needs to be after. > If we have either single th or global gen counter, current code would > become serialized, but this is not what I am about. Lets assume, for No, generation counts used like they are now (or in any way that I can see) can't give serialization. > the sake of the discussion only, that all readers take the same spinlock > as tc_windup (i.e. tc_setclock_mtx). Spinlocks are far too heavyweight. Most of the complications in timecounter locking are to avoid using them. But OK for the discussion. > It is still possible that reader unlocked the mutex, tc_windup() kicked > in, locked the mutex and moved timehands (as you noted, this might > even happen on the same CPU), and only then the reader continues. For > consumer of bintime() or any other function' result, it looks exactly > the same as if we did not serialized with writer but used outdated > timehands. Not with full serialization. The writer tc_windup() is also a reader, and serializing its read gives the necessary monotonicity (for a single thread): - normal reader locks the mutex, reads the timecounter and unlocks. The mutex makes visible all previous writes, so the reader doesn't use a stale timehands. Consumers of bintime(), etc., use this time in the past. - tc_windup() locks the mutex, reads the timecounter hardware and writes the timecounter soft state. The new offset is after all previous times read, since this is serialized. - normal reader as above sees the new state, so it reads times after the time of the windup, so also after the time of previous normal reads. > Let me formulate this diffeently: as far as consumer of the bintime() > result does not serialize itself with tc_windup(), serializing bintime() > itself against tc_windup() does not close the race, but it is not > obvious that the race matters. Readers can easily see times far in the past, but the times increase in program order. > Either we should just accept the race as > we currently do, or readers must take the spinlock where the exact value > of the current time is important, Disabling interrupts should be enough. In my version of 5.2, spinlocks don't disable hardware interrupts and may be preempted by fast interrupt handlers which may be not so fast and take hundreds of usec. Actually, even disabling interrupts might not be enough. A single isa bus read can take at least 138 usec (when it is behind a DMA queue or something). There are also NMI's and SMI's. > or readers must re-read the time after > doing something important, and redo if the new measuremedtime is outside > the acceptable range. This method seems to be essential for robustness. But I don't see any race (for a single thread and no timecounter skew across CPUs). Sloppy readers just see times an unknown but usually small time in the past. Non-sloppy readers can also defend against timecounter skew by binding to 1 CPU. Mutex locking of the timecounter doesn't give monotonic times across threads. It gives some order, but you don't know which. Another mutex or rendezvous is needed to control the order. Bruce From owner-freebsd-ppc@freebsd.org Fri Apr 5 19:35:09 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 28DC0155BB9C for ; Fri, 5 Apr 2019 19:35:09 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic314-20.consmr.mail.gq1.yahoo.com (sonic314-20.consmr.mail.gq1.yahoo.com [98.137.69.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F3C8A712BC for ; Fri, 5 Apr 2019 19:35:07 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: uUzurf0VM1lxFYMoT_VBwfIKKg_s0eLih6LSYHMSNeXWXxiBkoP2rvVrWL0OxJD d8iJGhL1jake0_oyOkaNVOZmCy27aLpZYcM7gofxA3CuOfsqVndeXhVlcWxqhNkdNM1Yj2Cz6dPw wybRMPE2BjOlPvh3l10jl9ZdUuSJBCf4_MZhvroD6pKBdxavkTzMzWSVijZM8xfJ44v73EzoclQ0 h4PJVrGvn9H4aH1WR8DCKi6L1AQcTXiTTSxvSpqrwWx_aypqsI7SOs6VZzly.exRnX8B4v0BLUuy _WLEECk3rs4xPadTY12V32_ljrmWrGBMGcyDBhcFEowc7rY5kGlDZEZCEA1gNqmaBPq3VsSuJsvV qlqUJGPCMiGMvCgI.FUltTTlK4hq_FXG9N.eL6hFmHRz0ILtfUCdUDM3.DXbD8izpDIoYPwfziS2 WYWbyO8S4oTErJbZwXnHwZkDIghitYAWGhOfXXMVQ0r9SGn6iH9KWI14eucdXQxwtZ.had2oHv.L b83htDwmBZXc_XFul3qrF3lIYWBc2URPcFVdCFx1l0WBo.Zpj8IX3vdcpSS5JrNvsPobpp7y9pU. vS5RQplaBuKiRi5VaSmdvTPWieY3pILVoMlud70OKbbXvwpOFApWaPP0r36g2Ar3tDE3.t2SXwfP bvkBTSxPTZ4H1ZKC8sNr8p2aC9Rm6SuvdCvzvoGxVsQRlnu2tQRgOVGBh8xKCFuOLKpOkh.iEviJ uTQbaN92zku1Ca578H5Zqt4R8mNzedj6utmcE7wjoFi9o7XIUx45DNaZ4MDauMWr7UPPD2MdZacC 8qouyPItMtGT0Ad3n4vxBgPp7q0UdLgx2v64bd2pyttqJqMHiyVqOJNzCd3zznyfLDdIIQac9oXf Pc5pDZujnGMyPEjKCezUBD0HjRdHLHCnzXXW1GaCIbUQxTvkmomk1Of_KStVIzRZTzFzlC1RWhLm MG_d33ruDYKLaox2h9wQc_f5RFssNJSSNYBiJ_AoMEdNwxwOPNpIxWcKBePqJTFFEbSzzsHVjlcl Ql30S_V860plIFcyCYqsFMP78XVSMIBQLsKDOBZlFEewacGzYDYlGvALX_YrWwICidLO88SpLZQA 8zA-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic314.consmr.mail.gq1.yahoo.com with HTTP; Fri, 5 Apr 2019 19:35:01 +0000 Received: from c-67-170-167-181.hsd1.or.comcast.net (EHLO [192.168.1.113]) ([67.170.167.181]) by smtp431.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 86af237d517259e6241beadd5b873703; Fri, 05 Apr 2019 19:35:00 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed] From: Mark Millard In-Reply-To: <20190406014724.X4174@besplex.bde.org> Date: Fri, 5 Apr 2019 12:34:59 -0700 Cc: Konstantin Belousov , freebsd-hackers Hackers , Michael Tuexen , FreeBSD PowerPC ML Content-Transfer-Encoding: quoted-printable Message-Id: <48DB5A87-1681-47DE-969F-FA569EBF6DF5@yahoo.com> References: <20190307222220.GK2492@kib.kiev.ua> <20190309144844.K1166@besplex.bde.org> <20190324110138.GR1923@kib.kiev.ua> <20190403070045.GW1923@kib.kiev.ua> <20190404011802.E2390@besplex.bde.org> <20190405113912.GB1923@kib.kiev.ua> <20190405230717.D3383@besplex.bde.org> <20190405132128.GD1923@kib.kiev.ua> <20190406003907.C3872@besplex.bde.org> <20190405143619.GF1923@kib.kiev.ua> <20190406014724.X4174@besplex.bde.org> To: Bruce Evans X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: F3C8A712BC X-Spamd-Bar: +++ X-Spamd-Result: default: False [3.14 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FREEMAIL_TO(0.00)[optusnet.com.au]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_SPAM_SHORT(0.27)[0.270,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(1.90)[ip: (7.87), ipnet: 98.137.64.0/21(0.95), asn: 36647(0.76), country: US(-0.06)]; NEURAL_SPAM_MEDIUM(0.95)[0.948,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.53)[0.533,0]; RCVD_IN_DNSWL_NONE(0.00)[83.69.137.98.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_POSSIBLE(0.00)[83.69.137.98.rep.mailspike.net : 127.0.0.17]; FREEMAIL_CC(0.00)[gmail.com] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2019 19:35:09 -0000 On 2019-Apr-5, at 08:26, Bruce Evans wrote: > On Fri, 5 Apr 2019, Konstantin Belousov wrote: >=20 >> On Sat, Apr 06, 2019 at 01:01:19AM +1100, Bruce Evans wrote: >>> On Fri, 5 Apr 2019, Konstantin Belousov wrote: >>>=20 >>>> On Fri, Apr 05, 2019 at 11:52:27PM +1100, Bruce Evans wrote: >>>>> On Fri, 5 Apr 2019, Konstantin Belousov wrote: >>>>>=20 >>>>>> On Thu, Apr 04, 2019 at 02:47:34AM +1100, Bruce Evans wrote: >>>>>>> I noticed (or better realized) a general problem with multiple >>>>>>> timehands. ntpd can slew the clock at up to 500 ppm, and at = least an >>>>>>> old version of it uses a rate of 50 ppm to fix up fairly small = drifts >>>>>>> in the milliseconds range. 500 ppm is enormous in CPU cycles -- = it is >>>>>>> 500 thousand nsec or 2 million cycles at 4GHz. Winding up the = timecounter >>>>>>> every 1 msec reduces this to only 2000 cycles. >>>>>>> ... >>>>>>> The main point of having multiple timehands (after introducing = the per- >>>>>>> timehands generation count) is to avoid blocking thread N during = the >>>>>>> update, but this doesn't actually work, even for only 2 = timehands and >>>>>>> a global generation count. >>>>>>=20 >>>>>> You are describing the generic race between reader and writer. = The same >>>>>> race would exist even with one timehand (and/or one global = generation >>>>>> counter), where ntp adjustment might come earlier or later of = some >>>>>> consumer accessing the timehands. If timehand instance was read = before >>>>>> tc_windup() run but code consumed the result after the windup, it = might >>>>>> appear as if time went backward, and this cannot be fixed without = either >>>>>> re-reading the time after time-depended calculations were done = and >>>>>> restarting, or some globabl lock ensuring serialization. >>>>>=20 >>>>> With 1 timehand, its generation count would be global. I think = its ordering >>>>> is strong enough to ensure serialization. >>>> Yes, single timehands result in global generation. But it would = not solve >>>> the same race appearing in slightly different manner, as I = described above. >>>> If reader finished while generation number in th was not yet reset, = but >>>> caller uses the result after tc_windup(), the effect is same as if = we >>>> have two th's and reader used the outdated one. >>>=20 >>> You described it too concisely for me to understand :-). >>>=20 >>> I now see that a single generation count doesn't give serialization. = I >>> thought that setting the generation to 0 at the start of tc_windup() >>> serialized the reader and writer. But all it does is prevent use of = the >>> results of the windup while only some of them are visible. If the >>> setting the generation count to 0 doesn't become before tc_windup() = reads >>> the hardware timecounter, then this read may be before other reads = using >>> the old timehands, but it needs to be after. >> If we have either single th or global gen counter, current code would >> become serialized, but this is not what I am about. Lets assume, for >=20 > No, generation counts used like they are now (or in any way that I can > see) can't give serialization. >=20 >> the sake of the discussion only, that all readers take the same = spinlock >> as tc_windup (i.e. tc_setclock_mtx). >=20 > Spinlocks are far too heavyweight. Most of the complications in = timecounter > locking are to avoid using them. But OK for the discussion. >=20 >> It is still possible that reader unlocked the mutex, tc_windup() = kicked >> in, locked the mutex and moved timehands (as you noted, this might >> even happen on the same CPU), and only then the reader continues. For >> consumer of bintime() or any other function' result, it looks exactly >> the same as if we did not serialized with writer but used outdated >> timehands. >=20 > Not with full serialization. The writer tc_windup() is also a reader, = and > serializing its read gives the necessary monotonicity (for a single = thread): > - normal reader locks the mutex, reads the timecounter and unlocks. = The > mutex makes visible all previous writes, so the reader doesn't use a > stale timehands. Consumers of bintime(), etc., use this time in the = past. > - tc_windup() locks the mutex, reads the timecounter hardware and = writes the > timecounter soft state. The new offset is after all previous times = read, > since this is serialized. > - normal reader as above sees the new state, so it reads times after = the > time of the windup, so also after the time of previous normal reads. >=20 >> Let me formulate this diffeently: as far as consumer of the bintime() >> result does not serialize itself with tc_windup(), serializing = bintime() >> itself against tc_windup() does not close the race, but it is not >> obvious that the race matters. >=20 > Readers can easily see times far in the past, but the times increase = in > program order. >=20 >> Either we should just accept the race as >> we currently do, or readers must take the spinlock where the exact = value >> of the current time is important, >=20 > Disabling interrupts should be enough. In my version of 5.2, = spinlocks > don't disable hardware interrupts and may be preempted by fast = interrupt > handlers which may be not so fast and take hundreds of usec. = Actually, > even disabling interrupts might not be enough. A single isa bus read > can take at least 138 usec (when it is behind a DMA queue or = something). > There are also NMI's and SMI's. >=20 >> or readers must re-read the time after >> doing something important, and redo if the new measuremedtime is = outside >> the acceptable range. >=20 > This method seems to be essential for robustness. >=20 > But I don't see any race (for a single thread and no timecounter skew > across CPUs). Sloppy readers just see times an unknown but usually = small > time in the past. Non-sloppy readers can also defend against = timecounter > skew by binding to 1 CPU. >=20 > Mutex locking of the timecounter doesn't give monotonic times across = threads. > It gives some order, but you don't know which. Another mutex or = rendezvous > is needed to control the order. >=20 Just for context for the original problem, in case it helps: The sleepq_timeout went into the case: if (td->td_sleeptimo > sbinuptime() || td->td_sleeptimo =3D=3D 0) = { /* * The thread does not want a timeout (yet). */ and after that the specific sleep did not try again (deleted?), thus the hangup for the sleeping thread. This was with a call backtrace looking like the below at the time: 0xe00000009af7c730: at sleepq_timeout+0x148 0xe00000009af7c7d0: at softclock_call_cc+0x234 0xe00000009af7c910: at callout_process+0x2e0 0xe00000009af7c9f0: at handleevents+0x22c 0xe00000009af7caa0: at timercb+0x340 0xe00000009af7cba0: at decr_intr+0x140 0xe00000009af7cbd0: at powerpc_interrupt+0x268 (I added a call to cause the backtrace to be reported.) For this call chain: timercb gets a "now" value that is passsed along and into callout_process but not to softclock_call_cc or sleepq_timeout . The callout_process is doing CALLOUT_DIRECT handling when it directly calls softclock_call_cc: . . . /* Iterate callwheel from firstb to nowb and then up to lastb. = */ do { sc =3D &cc->cc_callwheel[firstb & callwheelmask]; tmp =3D LIST_FIRST(sc); while (tmp !=3D NULL) { /* Run the callout if present time within = allowed. */ if (tmp->c_time <=3D now) { /* * Consumer told us the callout may be = run * directly from hardware interrupt = context. */ if (tmp->c_iflags & CALLOUT_DIRECT) { #ifdef CALLOUT_PROFILING ++depth_dir; #endif cc_exec_next(cc) =3D LIST_NEXT(tmp, c_links.le); cc->cc_bucket =3D firstb & = callwheelmask; LIST_REMOVE(tmp, c_links.le); softclock_call_cc(tmp, cc, #ifdef CALLOUT_PROFILING &mpcalls_dir, = &lockcalls_dir, NULL, #endif 1); tmp =3D cc_exec_next(cc); cc_exec_next(cc) =3D NULL; } else { . . . =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-ppc@freebsd.org Fri Apr 5 21:31:52 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 03704155F390 for ; Fri, 5 Apr 2019 21:31:52 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic302-3.consmr.mail.bf2.yahoo.com (sonic302-3.consmr.mail.bf2.yahoo.com [74.6.135.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 97B0F762FF for ; Fri, 5 Apr 2019 21:31:51 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: m3n.7wIVM1kc6473I5483ksjM9s0yR8uGzlSWg3L9xpBxZIAEq2e2TeMKoLHYKk 7NKnJVbk8QF3DI37BqEt5tSU9MMVpK3jaRGQDEFAshAa.2WUSG2D3aTuIcC99upH.KxqzdIlbG23 ngKmuMl0UsK2IphWhUuK0IGKPrfHIYZkNspo7WWi0QonLAGn3WkiEaBFH6k5wIKyr5Zbu7JOaqr_ 15OhnGjVKgxms6hxZK5_q3Q4M1i1TwiQ2RBhDx2oLr6e8roMYvz9WOlnTAo.pyr5EIKYzK7ijbL0 F7rKwsZiANSgwsRoEeu16nbrcFsjLbWxUYV35.zbSs6gVPKxDcn49P_o7udmpczqX2zjd.8YgDj4 a2RVOQtWBvxmF8Nxz8yqLuE5rzuPkitj9lxuE5VafuuG6YMOPN546RnmzzA_g2rS6ouDyH9bq1xU _PEJadq1zBmuJtIs0mX1ejPl9mUiwEKKPBy9cyS9aUaYOS6YLv_WCFK26_SW6Rj2yLo9zSugxj1c mQONqVhHD_k6G4K3k0gIbtLhYrTT91CrvhNBQSv6kT_cJDMyTdQrmLRCe6dXKV5.tNTF8x7jB7t. ItXNVgrgBKwn0zr.N_YXPJs140tsiHmBAu1yJhYRxDq7J6tfrT6dJRTMGg0pGcNu_p.SaVTeuerx C.7GhqK5xv9lVhmDjLAacK5rYIh69wcTAkzl3071VzjNPXh939c2KZiFbWaoBkTJfdJRwzXpJQD3 tmWsCjZb1LNhC8NzLbQevSs7r3xwoM19FBJRIEvuaUgDJ05WXpV6WpwocPkYjp18cQ7gb5QMcA62 r7WoOaB7kXabWjJWfw.L.KYvApEPQGKsDLf.O5WcIwwJibt6.cE6XTzWIxL4AYXSC.P3shpiZc0D 702ZzBc9OvDp5GnIkq0KbK7y2nY2QnwNERYxAetyLcRED0vINsu.qJ5WXaYkUo3ge_lpbCWGKq5G pjD6SI4OmU7ayibz2A._z4Go4LqX5ImS3cM.HpdK7yAqj8McxPfhvmawukXAuuOXwUAIxaSmtSkT hustoDi2P_tLfQLDVG6XuaeNweK9IEDmjc7h5E3xUGhtYWd8ATl_FT3coIalTQc.BlaWIXbPp Received: from sonic.gate.mail.ne1.yahoo.com by sonic302.consmr.mail.bf2.yahoo.com with HTTP; Fri, 5 Apr 2019 21:31:50 +0000 Received: from c-67-170-167-181.hsd1.or.comcast.net (EHLO [192.168.1.113]) ([67.170.167.181]) by smtp401.mail.bf1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 7418801b5e1fee9bf1fae3b3fb8f9af6; Fri, 05 Apr 2019 21:31:46 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed] From: Mark Millard In-Reply-To: <20190405113504.GA1923@kib.kiev.ua> Date: Fri, 5 Apr 2019 14:31:43 -0700 Cc: freebsd-hackers Hackers , FreeBSD PowerPC ML Content-Transfer-Encoding: quoted-printable Message-Id: <55604EF6-CB81-4300-8E9E-E3A94504D0B5@yahoo.com> References: <20190307222220.GK2492@kib.kiev.ua> <20190309144844.K1166@besplex.bde.org> <20190324110138.GR1923@kib.kiev.ua> <20190403070045.GW1923@kib.kiev.ua> <20190404011802.E2390@besplex.bde.org> <20190405150236.A959@besplex.bde.org> <20190405201055.I2396@besplex.bde.org> <20190405113504.GA1923@kib.kiev.ua> To: Konstantin Belousov , Bruce Evans X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 97B0F762FF X-Spamd-Bar: ------ X-Spamd-Result: default: False [-6.98 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.98)[-0.983,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2019 21:31:52 -0000 [I found a document covering PLLs for the 970MP's in the old PowerMac G5 involved.] > On 2019-Apr-5, at 04:35, Konstantin Belousov = wrote: >=20 > On Fri, Apr 05, 2019 at 09:13:50PM +1100, Bruce Evans wrote: >> On Fri, 5 Apr 2019, Mark Millard wrote: >>=20 >>> On 2019-Apr-4, at 21:38, Bruce Evans = wrote: >>>=20 >>>> On Thu, 4 Apr 2019, Mark Millard wrote: >>>>> ... >>>>> Unfortunately, all the multi-socket contexts that I sometimes have >>>>> access to are old PowerMacs. And, currently, the only such context >>>>> is the G5 with 2 sockets, 2 cores per socket (powerpc64). So I've >>>>> not been able to set up other types of examples to see if problems >>>>> repeat. >>>>>=20 >>>>> I do not have access to a single-socket powerpc64 for contrast in >>>>> that direction. >>>>=20 >>>> Testing 1 socket is time-consuming enough. Do these old systems >>>> use the equivalent of an x86 TSC for the timecounter? With = multiple >>>> sockets, it isn't clear how even a hardware timer independent of = the >>>> CPUs can be distributed so as to appear to be monotonic on all = cors. >>>=20 >>> "The DEC frequency is based on the same implementation-dependent >>> frequency that drives the time base." The frequency may well be >>> fixed by the PowerMac G5 model implementation but is not fixed >>> by the powerpc64 architecture. >>>=20 >>> The Time Base Register (TBR) in a powerpc64 core (cpu in FreeBSD = terms) >>> increments at 33,333,333 Hz (nominal) and is 64 bits wide. Its value >>> can be set (mttb instruction) and the boot sequence in FreeBSD does >>> attempt to adjust as the FreeBSD CPU is brought-up/started. mftb is >>> used to read the 64-bit value. FreeBSD masks it down to 32-bits to >>> contribute to the time-counter. >>>=20 >>> (Is that description sufficient for what you were after? I've never >>> seen documentation of how the 33,333,333 MHz is produced.) >>=20 >> This seems to be equivalent to an x86 TSC. > It is equivalent from the interface PoV. I saw some references that > Powers do have per-core PLLs, the best I can find now is > = https://stackoverflow.com/questions/43844438/are-multiple-clock-domains-co= mmon-in-modern-processors I finally found a document covering the 970MP in the old PowerMac G5 in question: http://datasheets.chipdb.org/IBM/PowerPC/970/PowerPC-970MP.pdf It has material about the PLL's, pinout, etc. : "This section will help in configuring the PLL and determining SYSCLK = input frequency for PowerPC 970MP systems." Also: 100 MHz to 700 MHz SYSCLK. "SYSCLK minimum frequency is for PLL mode. In PLL bypass mode (BYPASS = low), SYSCLK frequency may be as low as 10MHz." BUS_CFG[0:2], PLL_MULT, and PLL_RANGE[1:0] are involved for PLL mode. Ratio from BUS_CFG: 3:1, 6:1, 12:1 normally use PLL_MULT=3D0 for a PLL = multiplier of 12. Ratio from BUS_CFG: 2:1, 4:1, 8:1 normally use PLL_MULT=3D1 for a PLL = multiplier of 8. (It describes the PLL power supply filtering circuit as well.) So far I've not seen anything that is directly for the rate that the TBR and DEC registers change. > For Intel there is no much information, the best guess is that TSC is = in > uncore and effectively shared by all cores. See also > = https://software.intel.com/en-us/articles/best-timing-function-for-measuri= ng-ipp-api-timing/ >=20 >>=20 >> x86 P-state-invariant TSCs apparently use a 100 MHz clock which = corresponds >> even more closely to this, except for historical reasons this clock = is >> scaled and interpolated to a clock resembling the CPU cycle count at = a >> nominal frequency. > For Intel designs, there is indeed a single-source 100MHz signal which > is distributed over all consumers using clock fan-out buffers like = DB1900Z. >=20 >>=20 >>> As FreeBSD supports multi-socket, what are its criteria for a = sufficient >>> context for it to work with for supporting sbinuptime and the like? = Is >>> FreeBSD supposed to then make it appear that sbinputime and the like = are >>> weakly increasing, even as threads migrate between CPUs (cores, >>> hw-threads)? >>=20 >> CLOCK_MONOTONIC has to be monotonic, but it is only clear what this = means >> within a single thread: sequential clock_gettime() calls must occur = in >> program order and the results must be consistent with that order. = Across >> threads, I think it should mean that the results must be consistent = with >> any order established using any supported ordering methods. >>=20 >>> ... >>>>> One oddity is that the eventtimer's decrementer and timecounter >>>>> may change (nearly) together: both change at 33,333,333 Hz, as if >>>>> they are tied to the same clock (at least on one socket). >>>>=20 >>>> I think this is from a normal hardware implementation. On all of >>>> my x86 systems with a TSC, the TSC frequency is an exact fractional >>>> multiple of the i8254, the ACPI timer (if present) and the HPET (if >>>> present). Only the RTC has an independent frequency. The fraction = is >>>> changed by changing the nominal TSC frequency in the BIOS, but is = not >>>> changed by temperature variations. This must be because most = clocks are >>>> derived from a common clock using a PLL. I use this to calibrate = all >>>> clocks (except the RTC) by calibrating only 1. >>>=20 >>> I'm not aware of the OpenFirmware having any control over the >>> TBR-change frequency behavior. I've no evidence about any = variability >>> based on temperature. >>=20 >> Temperature changes usually affect the actual frequency but not the >> nominal frequency. >>=20 =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-ppc@freebsd.org Sat Apr 6 07:54:15 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 711C01579036 for ; Sat, 6 Apr 2019 07:54:15 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic313-10.consmr.mail.ne1.yahoo.com (sonic313-10.consmr.mail.ne1.yahoo.com [66.163.185.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BEBC36D304 for ; Sat, 6 Apr 2019 07:54:13 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: krgR4qkVM1lp_il.ec8qPQ_FqWpoc9GCtFRDRv.tGcazLsOHleHFCe9kZhtuhMD PIgmzklwHihOlY7Ypad.R8_9H2oRCfy6H8DXhdA_tC5FoGFq0nyQFZnAx4EaArLiQDxSvBBiC6TK hkr8F6sO_iHyHlvt0IgkZ9DPY3cD3Ghnb.WIFxIAJNM9IpjAZWq8LwXhknnrOho1X6_QK61Rj5G7 dxmxcS4dfp1jxTE2OaarF8khua6b8qHy7qCLD1jJujOaUzMH_E5D4R.bB_YdXQP4ToklgO.7FclY oCEDjBT5DYFJIQMP7SBVRgfGll.7jmOHoY_zaUnTDSMO5AWs_C6QyPWsFm5VPiK0e0F2Pa4r9Lmb uyN3lifTuHgwSPUKSAlzCIRRz2sr5fe8ZefU3RdFS4G6KX0OwMEhg8oxKEt3kuM.ypNQIG7wIV1L FfI3mD93WBKaqGPstIBdTSu_xBdvlY2SNxdmfSVPPdd0Zn5A7JgO_CizfIIzL8CChWwbEs_lcNJA 9yLZY5sivju16ORgRtKY4RxiOTAjzkyRswU9MVZ1xKkuLz6sD9WTjTxVfknMIzSAsR9o8nWlt7gZ b9U60miS2dYAEf32lKfEEb5KQ45nJG2Td0yWxjTqWEgZnTeCPjXMPQsc9ZR8h2eJuDN__PZRki7Z RAjwdOd4yse299CwuLnIEoXiFHRvs5PiH7_7y7rlRa9Wy7cYPGTdLTnZG6lNnoNrvv5yP9.ckQLv G9tjg4CgBkviLYZ3YEvQLNoalrDy2KaVIe3RQlaSL9TRS4d7EeT8KmhZQPH7W0wSkF1ivADFlW5K fylhXVOQHV4v5oygYAjhR8FW1F2WxWUqHZJ.IwglVBIBb4.eTsZ9Gr1lRy6hOieMV2PQlRzDhdAe pXtLcwN9eop0iU6PeNLMjShHkbHMwKVWTuBZpVWEdu8O46hxpNe3XV6aHn1ljUNTenG4EoBwUA7c g_sFjAKXuqZlYc7GDLjJcfRMQIdUGhflRdky5bShGdw1zRYRadnOCHwUlcH8cTRFhFjf9zK9STw3 vBuppWsPdKZgtj3RlcRUaSmJiFDtVEtkMNw8xTSgYK5lILMBHtOs0B.lxGuGLoyitl._W7V7z Received: from sonic.gate.mail.ne1.yahoo.com by sonic313.consmr.mail.ne1.yahoo.com with HTTP; Sat, 6 Apr 2019 07:54:05 +0000 Received: from c-67-170-167-181.hsd1.or.comcast.net (EHLO [192.168.1.113]) ([67.170.167.181]) by smtp417.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 71c50158df587c5d8e6f5cfcbcf89622; Sat, 06 Apr 2019 07:54:02 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed] From: Mark Millard In-Reply-To: <55604EF6-CB81-4300-8E9E-E3A94504D0B5@yahoo.com> Date: Sat, 6 Apr 2019 00:54:00 -0700 Cc: freebsd-hackers Hackers , FreeBSD PowerPC ML Content-Transfer-Encoding: quoted-printable Message-Id: <3592A7F1-9EAE-4A33-B51A-678BE104E18C@yahoo.com> References: <20190307222220.GK2492@kib.kiev.ua> <20190309144844.K1166@besplex.bde.org> <20190324110138.GR1923@kib.kiev.ua> <20190403070045.GW1923@kib.kiev.ua> <20190404011802.E2390@besplex.bde.org> <20190405150236.A959@besplex.bde.org> <20190405201055.I2396@besplex.bde.org> <20190405113504.GA1923@kib.kiev.ua> <55604EF6-CB81-4300-8E9E-E3A94504D0B5@yahoo.com> To: Konstantin Belousov , Bruce Evans X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: BEBC36D304 X-Spamd-Bar: +++ X-Spamd-Result: default: False [3.42 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36646, ipnet:66.163.184.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_SPAM_SHORT(0.95)[0.951,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(1.50)[ip: (4.94), ipnet: 66.163.184.0/21(1.46), asn: 36646(1.17), country: US(-0.06)]; NEURAL_SPAM_MEDIUM(0.96)[0.960,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.52)[0.516,0]; RCVD_IN_DNSWL_NONE(0.00)[33.185.163.66.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_POSSIBLE(0.00)[33.185.163.66.rep.mailspike.net : 127.0.0.17] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Apr 2019 07:54:15 -0000 On 2019-Apr-5, at 14:31, Mark Millard wrote: [I found a document covering PLLs for the 970MP's in the old PowerMac G5 involved.] > On 2019-Apr-5, at 04:35, Konstantin Belousov = wrote: >>=20 >> On Fri, Apr 05, 2019 at 09:13:50PM +1100, Bruce Evans wrote: >>> On Fri, 5 Apr 2019, Mark Millard wrote: >>>=20 >>>> On 2019-Apr-4, at 21:38, Bruce Evans = wrote: >>>>=20 >>>>> On Thu, 4 Apr 2019, Mark Millard wrote: >>>>>> ... >>>>>> Unfortunately, all the multi-socket contexts that I sometimes = have >>>>>> access to are old PowerMacs. And, currently, the only such = context >>>>>> is the G5 with 2 sockets, 2 cores per socket (powerpc64). So I've >>>>>> not been able to set up other types of examples to see if = problems >>>>>> repeat. >>>>>>=20 >>>>>> I do not have access to a single-socket powerpc64 for contrast in >>>>>> that direction. >>>>>=20 >>>>> Testing 1 socket is time-consuming enough. Do these old systems >>>>> use the equivalent of an x86 TSC for the timecounter? With = multiple >>>>> sockets, it isn't clear how even a hardware timer independent of = the >>>>> CPUs can be distributed so as to appear to be monotonic on all = cors. >>>>=20 >>>> "The DEC frequency is based on the same implementation-dependent >>>> frequency that drives the time base." The frequency may well be >>>> fixed by the PowerMac G5 model implementation but is not fixed >>>> by the powerpc64 architecture. >>>>=20 >>>> The Time Base Register (TBR) in a powerpc64 core (cpu in FreeBSD = terms) >>>> increments at 33,333,333 Hz (nominal) and is 64 bits wide. Its = value >>>> can be set (mttb instruction) and the boot sequence in FreeBSD does >>>> attempt to adjust as the FreeBSD CPU is brought-up/started. mftb is >>>> used to read the 64-bit value. FreeBSD masks it down to 32-bits to >>>> contribute to the time-counter. >>>>=20 >>>> (Is that description sufficient for what you were after? I've never >>>> seen documentation of how the 33,333,333 MHz is produced.) >>>=20 >>> This seems to be equivalent to an x86 TSC. >> It is equivalent from the interface PoV. I saw some references that >> Powers do have per-core PLLs, the best I can find now is >> = https://stackoverflow.com/questions/43844438/are-multiple-clock-domains-co= mmon-in-modern-processors >=20 > I finally found a document covering the 970MP in the old > PowerMac G5 in question: >=20 > http://datasheets.chipdb.org/IBM/PowerPC/970/PowerPC-970MP.pdf >=20 > It has material about the PLL's, pinout, etc. : >=20 > "This section will help in configuring the PLL and determining SYSCLK = input > frequency for PowerPC 970MP systems." Also: 100 MHz to 700 MHz SYSCLK. > "SYSCLK minimum frequency is for PLL mode. In PLL bypass mode (BYPASS = low), > SYSCLK frequency may be as low as 10MHz." BUS_CFG[0:2], PLL_MULT, and > PLL_RANGE[1:0] are involved for PLL mode. >=20 > Ratio from BUS_CFG: 3:1, 6:1, 12:1 normally use PLL_MULT=3D0 for a PLL = multiplier of 12. > Ratio from BUS_CFG: 2:1, 4:1, 8:1 normally use PLL_MULT=3D1 for a PLL = multiplier of 8. >=20 > (It describes the PLL power supply filtering circuit as well.) >=20 > So far I've not seen anything that is directly for the rate that the = TBR > and DEC registers change. >=20 >> For Intel there is no much information, the best guess is that TSC is = in >> uncore and effectively shared by all cores. See also >> = https://software.intel.com/en-us/articles/best-timing-function-for-measuri= ng-ipp-api-timing/ >>=20 >>>=20 >>> x86 P-state-invariant TSCs apparently use a 100 MHz clock which = corresponds >>> even more closely to this, except for historical reasons this clock = is >>> scaled and interpolated to a clock resembling the CPU cycle count at = a >>> nominal frequency. >> For Intel designs, there is indeed a single-source 100MHz signal = which >> is distributed over all consumers using clock fan-out buffers like = DB1900Z. >>=20 >>>=20 >>>> As FreeBSD supports multi-socket, what are its criteria for a = sufficient >>>> context for it to work with for supporting sbinuptime and the like? = Is >>>> FreeBSD supposed to then make it appear that sbinputime and the = like are >>>> weakly increasing, even as threads migrate between CPUs (cores, >>>> hw-threads)? >>>=20 >>> CLOCK_MONOTONIC has to be monotonic, but it is only clear what this = means >>> within a single thread: sequential clock_gettime() calls must occur = in >>> program order and the results must be consistent with that order. = Across >>> threads, I think it should mean that the results must be consistent = with >>> any order established using any supported ordering methods. >>>=20 >>>> ... >>>>>> One oddity is that the eventtimer's decrementer and timecounter >>>>>> may change (nearly) together: both change at 33,333,333 Hz, as if >>>>>> they are tied to the same clock (at least on one socket). >>>>>=20 >>>>> I think this is from a normal hardware implementation. On all of >>>>> my x86 systems with a TSC, the TSC frequency is an exact = fractional >>>>> multiple of the i8254, the ACPI timer (if present) and the HPET = (if >>>>> present). Only the RTC has an independent frequency. The = fraction is >>>>> changed by changing the nominal TSC frequency in the BIOS, but is = not >>>>> changed by temperature variations. This must be because most = clocks are >>>>> derived from a common clock using a PLL. I use this to calibrate = all >>>>> clocks (except the RTC) by calibrating only 1. >>>>=20 >>>> I'm not aware of the OpenFirmware having any control over the >>>> TBR-change frequency behavior. I've no evidence about any = variability >>>> based on temperature. >>>=20 >>> Temperature changes usually affect the actual frequency but not the >>> nominal frequency. >>>=20 >=20 I found more about the 970MP's TB/DEC rate: "The TBEN input pin can be used as an enable for the internal timebase/decrementer or as an external clock input." Details: HID0 bit 19 =3D 0: update at 1/16th processor core frequency, but only when TBEN is high. HID0 bit 19 =3D 1: update at rising edge of TBEN (must not exceed 1/16th of the core processor's maximum frequency). So the 33,333,333 Hz TB&DEC update rate vs. 2500 MHz would mean that rising edges of TBEN are in use (HID0 bit 19 =3D 1) in the PowerMac G5 context in question. I've no information about how closely matched the TBEN signals are at the 2 sockets, beyond the nominal frequency. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-ppc@freebsd.org Sat Apr 6 08:25:21 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C8D27157A43E for ; Sat, 6 Apr 2019 08:25:21 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic302-20.consmr.mail.gq1.yahoo.com (sonic302-20.consmr.mail.gq1.yahoo.com [98.137.68.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49D7B6F2EF for ; Sat, 6 Apr 2019 08:25:20 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: 4cnKdn8VM1nYoyhy2wjSk5RYKMFVZTzfVtToQCcExGw7cscAKevnD4ij5g3CIU9 viVNlpClSvqK0OXvT2YrjznQdIvkDG7KaC3GknQ2OhUvQMfPwG9.Y9y4o_ZvFuJXalXuP66wv88s lBiFTjGGpMVQjPzDj3TL3FgGd0rP9R6yxRu.plvknkZsPSdn_nmzfz4_OBDD4xh5iT1mvb.xMqzy Mx2.zDXOvlTIKAbqNPYivl2hCq4tpH2oYq0etF3iDA3N9_WRnwRmX2_toCJSdfkSKnoUkKWAITRR Wo_jUDo6y5bnnNUgm1Ti5eshPNQgyqNP9wmNdn4mr8Re187njoQWywv_MMR12B_DagAdyitl5QRX 52eXpa4H65C9kUU4fMXSkLUN9VBaE0zTZsuw5IOm5sy16QKhtNZgTmwtxV.zD84VQiaJkqEhy8Gv P3cVySaEoF4fTwQNAhQzUDaw.tzlZS6eR0Yqm_Ha3__qbfbXvTCffJFvs8wyrw5QRSJWyeFZiFES JEIRw9Q05So3VWcrhPSk7xKbdZgIrgOHvwZs9HUGbejQ1bjOjE4Aw0VqRelhkldu17Ow3ECT09VV kctRcPrPFNnIC_Gl42j7RvBtuYc1gRYjDDBTEGEep0Xgm6PQiv5eynmq3k5TyVATnHdzeo8xv_Jl u0_a8np3rBonMGR5mYRso4Bs6xukNBdB2IlgPQOF7DCApD8a8J3synuuYgp3MIvNbS41uFpvc2Tu DSMVPR1GcyNtr1m6FezGO9ncw9gTl1owzzQfG5fncpZLVH397UukvV.x1FW2TzbRv38KWExnPS8y xUGzY18SvDUFqdwOdQs3GrzTSVI.E3cGzW2g375Wk_IPIBqJQDU5DU.88HikFMNjQriA8Fv_qNVF 2yZRtM7G8XceAyh1iFab_xEpPwnoLuUOUsUx6WaAF_x8FYwHvmSkT4iZsHs6lyXCBE.SlwS41uCx gJma5CzFClBXJY49fqO.1Uao3CJkm.D8OzlFCLsEWXM2JAsL_nsJCFEEAgbmFwVrJq42QoynW0Sx vYHpvudnhVw3c2FWpijRE4dlqtfUW.A2b91YtzTcfph7UC3nBRLYIpvOr1Yu1BLyEb3FNA_c4Xg- - Received: from sonic.gate.mail.ne1.yahoo.com by sonic302.consmr.mail.gq1.yahoo.com with HTTP; Sat, 6 Apr 2019 08:25:18 +0000 Received: from c-67-170-167-181.hsd1.or.comcast.net (EHLO [192.168.1.113]) ([67.170.167.181]) by smtp420.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 75821d711dc554505c099e91b68f42a6; Sat, 06 Apr 2019 08:25:15 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed] From: Mark Millard In-Reply-To: <3592A7F1-9EAE-4A33-B51A-678BE104E18C@yahoo.com> Date: Sat, 6 Apr 2019 01:25:13 -0700 Cc: freebsd-hackers Hackers , FreeBSD PowerPC ML Content-Transfer-Encoding: quoted-printable Message-Id: <7B096BB4-7DBA-4B95-AC1E-DDD9A7DF3B22@yahoo.com> References: <20190307222220.GK2492@kib.kiev.ua> <20190309144844.K1166@besplex.bde.org> <20190324110138.GR1923@kib.kiev.ua> <20190403070045.GW1923@kib.kiev.ua> <20190404011802.E2390@besplex.bde.org> <20190405150236.A959@besplex.bde.org> <20190405201055.I2396@besplex.bde.org> <20190405113504.GA1923@kib.kiev.ua> <55604EF6-CB81-4300-8E9E-E3A94504D0B5@yahoo.com> <3592A7F1-9EAE-4A33-B51A-678BE104E18C@yahoo.com> To: Konstantin Belousov , Bruce Evans X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 49D7B6F2EF X-Spamd-Bar: + X-Spamd-Result: default: False [1.45 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_SPAM_SHORT(0.89)[0.891,0]; NEURAL_HAM_LONG(-0.52)[-0.521,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.86)[ip: (2.66), ipnet: 98.137.64.0/21(0.95), asn: 36647(0.76), country: US(-0.06)]; NEURAL_SPAM_MEDIUM(0.73)[0.727,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[146.68.137.98.list.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Apr 2019 08:25:22 -0000 [Another document was more explicit about 1/16 mode.] On 2019-Apr-6, at 00:54, Mark Millard wrote: > On 2019-Apr-5, at 14:31, Mark Millard wrote: >=20 > [I found a document covering PLLs for the 970MP's in the old > PowerMac G5 involved.] >=20 >> On 2019-Apr-5, at 04:35, Konstantin Belousov = wrote: >>>=20 >>> On Fri, Apr 05, 2019 at 09:13:50PM +1100, Bruce Evans wrote: >>>> On Fri, 5 Apr 2019, Mark Millard wrote: >>>>=20 >>>>> On 2019-Apr-4, at 21:38, Bruce Evans = wrote: >>>>>=20 >>>>>> On Thu, 4 Apr 2019, Mark Millard wrote: >>>>>>> ... >>>>>>> Unfortunately, all the multi-socket contexts that I sometimes = have >>>>>>> access to are old PowerMacs. And, currently, the only such = context >>>>>>> is the G5 with 2 sockets, 2 cores per socket (powerpc64). So = I've >>>>>>> not been able to set up other types of examples to see if = problems >>>>>>> repeat. >>>>>>>=20 >>>>>>> I do not have access to a single-socket powerpc64 for contrast = in >>>>>>> that direction. >>>>>>=20 >>>>>> Testing 1 socket is time-consuming enough. Do these old systems >>>>>> use the equivalent of an x86 TSC for the timecounter? With = multiple >>>>>> sockets, it isn't clear how even a hardware timer independent of = the >>>>>> CPUs can be distributed so as to appear to be monotonic on all = cors. >>>>>=20 >>>>> "The DEC frequency is based on the same implementation-dependent >>>>> frequency that drives the time base." The frequency may well be >>>>> fixed by the PowerMac G5 model implementation but is not fixed >>>>> by the powerpc64 architecture. >>>>>=20 >>>>> The Time Base Register (TBR) in a powerpc64 core (cpu in FreeBSD = terms) >>>>> increments at 33,333,333 Hz (nominal) and is 64 bits wide. Its = value >>>>> can be set (mttb instruction) and the boot sequence in FreeBSD = does >>>>> attempt to adjust as the FreeBSD CPU is brought-up/started. mftb = is >>>>> used to read the 64-bit value. FreeBSD masks it down to 32-bits to >>>>> contribute to the time-counter. >>>>>=20 >>>>> (Is that description sufficient for what you were after? I've = never >>>>> seen documentation of how the 33,333,333 MHz is produced.) >>>>=20 >>>> This seems to be equivalent to an x86 TSC. >>> It is equivalent from the interface PoV. I saw some references that >>> Powers do have per-core PLLs, the best I can find now is >>> = https://stackoverflow.com/questions/43844438/are-multiple-clock-domains-co= mmon-in-modern-processors >>=20 >> I finally found a document covering the 970MP in the old >> PowerMac G5 in question: >>=20 >> http://datasheets.chipdb.org/IBM/PowerPC/970/PowerPC-970MP.pdf >>=20 >> It has material about the PLL's, pinout, etc. : >>=20 >> "This section will help in configuring the PLL and determining SYSCLK = input >> frequency for PowerPC 970MP systems." Also: 100 MHz to 700 MHz = SYSCLK. >> "SYSCLK minimum frequency is for PLL mode. In PLL bypass mode (BYPASS = low), >> SYSCLK frequency may be as low as 10MHz." BUS_CFG[0:2], PLL_MULT, and >> PLL_RANGE[1:0] are involved for PLL mode. >>=20 >> Ratio from BUS_CFG: 3:1, 6:1, 12:1 normally use PLL_MULT=3D0 for a = PLL multiplier of 12. >> Ratio from BUS_CFG: 2:1, 4:1, 8:1 normally use PLL_MULT=3D1 for a PLL = multiplier of 8. >>=20 >> (It describes the PLL power supply filtering circuit as well.) >>=20 >> So far I've not seen anything that is directly for the rate that the = TBR >> and DEC registers change. >>=20 >>> For Intel there is no much information, the best guess is that TSC = is in >>> uncore and effectively shared by all cores. See also >>> = https://software.intel.com/en-us/articles/best-timing-function-for-measuri= ng-ipp-api-timing/ >>>=20 >>>>=20 >>>> x86 P-state-invariant TSCs apparently use a 100 MHz clock which = corresponds >>>> even more closely to this, except for historical reasons this clock = is >>>> scaled and interpolated to a clock resembling the CPU cycle count = at a >>>> nominal frequency. >>> For Intel designs, there is indeed a single-source 100MHz signal = which >>> is distributed over all consumers using clock fan-out buffers like = DB1900Z. >>>=20 >>>>=20 >>>>> As FreeBSD supports multi-socket, what are its criteria for a = sufficient >>>>> context for it to work with for supporting sbinuptime and the = like? Is >>>>> FreeBSD supposed to then make it appear that sbinputime and the = like are >>>>> weakly increasing, even as threads migrate between CPUs (cores, >>>>> hw-threads)? >>>>=20 >>>> CLOCK_MONOTONIC has to be monotonic, but it is only clear what this = means >>>> within a single thread: sequential clock_gettime() calls must occur = in >>>> program order and the results must be consistent with that order. = Across >>>> threads, I think it should mean that the results must be consistent = with >>>> any order established using any supported ordering methods. >>>>=20 >>>>> ... >>>>>>> One oddity is that the eventtimer's decrementer and timecounter >>>>>>> may change (nearly) together: both change at 33,333,333 Hz, as = if >>>>>>> they are tied to the same clock (at least on one socket). >>>>>>=20 >>>>>> I think this is from a normal hardware implementation. On all of >>>>>> my x86 systems with a TSC, the TSC frequency is an exact = fractional >>>>>> multiple of the i8254, the ACPI timer (if present) and the HPET = (if >>>>>> present). Only the RTC has an independent frequency. The = fraction is >>>>>> changed by changing the nominal TSC frequency in the BIOS, but is = not >>>>>> changed by temperature variations. This must be because most = clocks are >>>>>> derived from a common clock using a PLL. I use this to calibrate = all >>>>>> clocks (except the RTC) by calibrating only 1. >>>>>=20 >>>>> I'm not aware of the OpenFirmware having any control over the >>>>> TBR-change frequency behavior. I've no evidence about any = variability >>>>> based on temperature. >>>>=20 >>>> Temperature changes usually affect the actual frequency but not the >>>> nominal frequency. >>>>=20 >>=20 >=20 > I found more about the 970MP's TB/DEC rate: >=20 > "The TBEN input pin can be used as an enable for the internal > timebase/decrementer or as an external clock input." Details: >=20 > HID0 bit 19 =3D 0: update at 1/16th processor core frequency, > but only when TBEN is high. That 1/16th should be of "the full processor frequency, even when the processor itself is running at a lower frequency". The 970FX has 1/8th instead of 1/16th. "Since the mesh clock frequency can be lowered to 1/64th of the full-speed, the time base/decrementer may be increased/decreased by more than one at a time." The decrementer tests for wrap, not for reaching zero. > HID0 bit 19 =3D 1: update at rising edge of TBEN > (must not exceed 1/16th of the core processor's > maximum frequency). >=20 > So the 33,333,333 Hz TB&DEC update rate vs. 2500 MHz would mean > that rising edges of TBEN are in use (HID0 bit 19 =3D 1) in the > PowerMac G5 context in question. >=20 > I've no information about how closely matched the TBEN signals are > at the 2 sockets, beyond the nominal frequency. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-ppc@freebsd.org Sat Apr 6 12:17:04 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C35791558DCD; Sat, 6 Apr 2019 12:17:04 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 14F4A763A7; Sat, 6 Apr 2019 12:17:03 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id x36CGsx5061946 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 6 Apr 2019 15:16:57 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua x36CGsx5061946 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id x36CGoJq061945; Sat, 6 Apr 2019 15:16:50 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 6 Apr 2019 15:16:50 +0300 From: Konstantin Belousov To: Bruce Evans Cc: Michael Tuexen , freebsd-hackers Hackers , FreeBSD PowerPC ML Subject: Re: powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed] Message-ID: <20190406121650.GJ1923@kib.kiev.ua> References: <20190324110138.GR1923@kib.kiev.ua> <20190403070045.GW1923@kib.kiev.ua> <20190404011802.E2390@besplex.bde.org> <20190405113912.GB1923@kib.kiev.ua> <20190405230717.D3383@besplex.bde.org> <20190405132128.GD1923@kib.kiev.ua> <20190406003907.C3872@besplex.bde.org> <20190405143619.GF1923@kib.kiev.ua> <20190406014724.X4174@besplex.bde.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20190406014724.X4174@besplex.bde.org> User-Agent: Mutt/1.11.4 (2019-03-13) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Apr 2019 12:17:05 -0000 On Sat, Apr 06, 2019 at 02:26:11AM +1100, Bruce Evans wrote: > On Fri, 5 Apr 2019, Konstantin Belousov wrote: > > > On Sat, Apr 06, 2019 at 01:01:19AM +1100, Bruce Evans wrote: > >> On Fri, 5 Apr 2019, Konstantin Belousov wrote: > >> > >>> On Fri, Apr 05, 2019 at 11:52:27PM +1100, Bruce Evans wrote: > >>>> On Fri, 5 Apr 2019, Konstantin Belousov wrote: > >>>> > >>>>> On Thu, Apr 04, 2019 at 02:47:34AM +1100, Bruce Evans wrote: > >>>>>> I noticed (or better realized) a general problem with multiple > >>>>>> timehands. ntpd can slew the clock at up to 500 ppm, and at least an > >>>>>> old version of it uses a rate of 50 ppm to fix up fairly small drifts > >>>>>> in the milliseconds range. 500 ppm is enormous in CPU cycles -- it is > >>>>>> 500 thousand nsec or 2 million cycles at 4GHz. Winding up the timecounter > >>>>>> every 1 msec reduces this to only 2000 cycles. > >>>>>> ... > >>>>>> The main point of having multiple timehands (after introducing the per- > >>>>>> timehands generation count) is to avoid blocking thread N during the > >>>>>> update, but this doesn't actually work, even for only 2 timehands and > >>>>>> a global generation count. > >>>>> > >>>>> You are describing the generic race between reader and writer. The same > >>>>> race would exist even with one timehand (and/or one global generation > >>>>> counter), where ntp adjustment might come earlier or later of some > >>>>> consumer accessing the timehands. If timehand instance was read before > >>>>> tc_windup() run but code consumed the result after the windup, it might > >>>>> appear as if time went backward, and this cannot be fixed without either > >>>>> re-reading the time after time-depended calculations were done and > >>>>> restarting, or some globabl lock ensuring serialization. > >>>> > >>>> With 1 timehand, its generation count would be global. I think its ordering > >>>> is strong enough to ensure serialization. > >>> Yes, single timehands result in global generation. But it would not solve > >>> the same race appearing in slightly different manner, as I described above. > >>> If reader finished while generation number in th was not yet reset, but > >>> caller uses the result after tc_windup(), the effect is same as if we > >>> have two th's and reader used the outdated one. > >> > >> You described it too concisely for me to understand :-). > >> > >> I now see that a single generation count doesn't give serialization. I > >> thought that setting the generation to 0 at the start of tc_windup() > >> serialized the reader and writer. But all it does is prevent use of the > >> results of the windup while only some of them are visible. If the > >> setting the generation count to 0 doesn't become before tc_windup() reads > >> the hardware timecounter, then this read may be before other reads using > >> the old timehands, but it needs to be after. > > If we have either single th or global gen counter, current code would > > become serialized, but this is not what I am about. Lets assume, for > > No, generation counts used like they are now (or in any way that I can > see) can't give serialization. > > > the sake of the discussion only, that all readers take the same spinlock > > as tc_windup (i.e. tc_setclock_mtx). > > Spinlocks are far too heavyweight. Most of the complications in timecounter > locking are to avoid using them. But OK for the discussion. > > > It is still possible that reader unlocked the mutex, tc_windup() kicked > > in, locked the mutex and moved timehands (as you noted, this might > > even happen on the same CPU), and only then the reader continues. For > > consumer of bintime() or any other function' result, it looks exactly > > the same as if we did not serialized with writer but used outdated > > timehands. > > Not with full serialization. The writer tc_windup() is also a reader, and > serializing its read gives the necessary monotonicity (for a single thread): > - normal reader locks the mutex, reads the timecounter and unlocks. The > mutex makes visible all previous writes, so the reader doesn't use a > stale timehands. Consumers of bintime(), etc., use this time in the past. > - tc_windup() locks the mutex, reads the timecounter hardware and writes the > timecounter soft state. The new offset is after all previous times read, > since this is serialized. > - normal reader as above sees the new state, so it reads times after the > time of the windup, so also after the time of previous normal reads. > > > Let me formulate this diffeently: as far as consumer of the bintime() > > result does not serialize itself with tc_windup(), serializing bintime() > > itself against tc_windup() does not close the race, but it is not > > obvious that the race matters. > > Readers can easily see times far in the past, but the times increase in > program order. This is true for the current implementation (single-thread monotonic). > > > Either we should just accept the race as > > we currently do, or readers must take the spinlock where the exact value > > of the current time is important, > > Disabling interrupts should be enough. In my version of 5.2, spinlocks > don't disable hardware interrupts and may be preempted by fast interrupt > handlers which may be not so fast and take hundreds of usec. Actually, > even disabling interrupts might not be enough. A single isa bus read > can take at least 138 usec (when it is behind a DMA queue or something). > There are also NMI's and SMI's. > > > or readers must re-read the time after > > doing something important, and redo if the new measuremedtime is outside > > the acceptable range. > > This method seems to be essential for robustness. > > But I don't see any race (for a single thread and no timecounter skew > across CPUs). Sloppy readers just see times an unknown but usually small > time in the past. Non-sloppy readers can also defend against timecounter > skew by binding to 1 CPU. Yes, this is true. > > Mutex locking of the timecounter doesn't give monotonic times across threads. > It gives some order, but you don't know which. Another mutex or rendezvous > is needed to control the order. Define monotonic between threads ? From owner-freebsd-ppc@freebsd.org Sat Apr 6 13:06:52 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5ACA9155C92F for ; Sat, 6 Apr 2019 13:06:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id E72A477A09 for ; Sat, 6 Apr 2019 13:06:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id AB575155C92E; Sat, 6 Apr 2019 13:06:51 +0000 (UTC) Delivered-To: ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 98F88155C92D for ; Sat, 6 Apr 2019 13:06:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 306C777A06 for ; Sat, 6 Apr 2019 13:06:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 54EB61797D for ; Sat, 6 Apr 2019 13:06:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id x36D6odc047043 for ; Sat, 6 Apr 2019 13:06:50 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id x36D6of5047042 for ppc@FreeBSD.org; Sat, 6 Apr 2019 13:06:50 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: ppc@FreeBSD.org Subject: [Bug 235060] [boot] FreeBSD 12.0 Release DVD and CD will not boot on PowerMac G5 Quad Date: Sat, 06 Apr 2019 13:06:49 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: sven.siemsen@me.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: ppc@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Apr 2019 13:06:52 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D235060 Sven Siemsen changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |sven.siemsen@me.com --- Comment #8 from Sven Siemsen --- Same for me. Power Mac G5 quad, 16 GB Yesterday, I did a buildworld and buildkernel with from releng-12.0 with the same result (black screen) --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-ppc@freebsd.org Sat Apr 6 16:21:04 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3B21E156372B; Sat, 6 Apr 2019 16:21:04 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail104.syd.optusnet.com.au (mail104.syd.optusnet.com.au [211.29.132.246]) by mx1.freebsd.org (Postfix) with ESMTP id 3BAE68741F; Sat, 6 Apr 2019 16:21:02 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from [192.168.0.102] (c110-21-101-228.carlnfd1.nsw.optusnet.com.au [110.21.101.228]) by mail104.syd.optusnet.com.au (Postfix) with ESMTPS id 99BAE4383CA; Sun, 7 Apr 2019 02:20:51 +1000 (AEST) Date: Sun, 7 Apr 2019 02:20:49 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Konstantin Belousov cc: Bruce Evans , Michael Tuexen , freebsd-hackers Hackers , FreeBSD PowerPC ML Subject: Re: powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed] In-Reply-To: <20190406121650.GJ1923@kib.kiev.ua> Message-ID: <20190407015735.W2994@besplex.bde.org> References: <20190324110138.GR1923@kib.kiev.ua> <20190403070045.GW1923@kib.kiev.ua> <20190404011802.E2390@besplex.bde.org> <20190405113912.GB1923@kib.kiev.ua> <20190405230717.D3383@besplex.bde.org> <20190405132128.GD1923@kib.kiev.ua> <20190406003907.C3872@besplex.bde.org> <20190405143619.GF1923@kib.kiev.ua> <20190406014724.X4174@besplex.bde.org> <20190406121650.GJ1923@kib.kiev.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.2 cv=P6RKvmIu c=1 sm=1 tr=0 cx=a_idp_d a=PalzARQSbocsUSjMRkwAPg==:117 a=PalzARQSbocsUSjMRkwAPg==:17 a=jpOVt7BSZ2e4Z31A5e1TngXxSK0=:19 a=kj9zAlcOel0A:10 a=lIKfSjjBpJ_XLBo-IhQA:9 a=PV0E4CjrIwHpNLuQ:21 a=bqPXIC2uY7JGT8R7:21 a=CjuIK1q_8ugA:10 X-Rspamd-Queue-Id: 3BAE68741F X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.98 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.98)[-0.980,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Apr 2019 16:21:04 -0000 On Sat, 6 Apr 2019, Konstantin Belousov wrote: > On Sat, Apr 06, 2019 at 02:26:11AM +1100, Bruce Evans wrote: >> On Fri, 5 Apr 2019, Konstantin Belousov wrote: >> >>> On Sat, Apr 06, 2019 at 01:01:19AM +1100, Bruce Evans wrote: >>>> On Fri, 5 Apr 2019, Konstantin Belousov wrote: >>>> >>>>> On Fri, Apr 05, 2019 at 11:52:27PM +1100, Bruce Evans wrote: >>>>>> On Fri, 5 Apr 2019, Konstantin Belousov wrote: >>>>>> >>>>>>> On Thu, Apr 04, 2019 at 02:47:34AM +1100, Bruce Evans wrote: >>>>>>>> I noticed (or better realized) a general problem with multiple >>>>>>>> timehands. ntpd can slew the clock at up to 500 ppm, and at least an >>>>>>>> old version of it uses a rate of 50 ppm to fix up fairly small drifts >>>>>>>> in the milliseconds range. 500 ppm is enormous in CPU cycles -- it is >>>>>>>> 500 thousand nsec or 2 million cycles at 4GHz. Winding up the timecounter >>>>>>>> every 1 msec reduces this to only 2000 cycles. >>>>>>>> ... >>>>>>>> The main point of having multiple timehands (after introducing the per- >>>>>>>> timehands generation count) is to avoid blocking thread N during the >>>>>>>> update, but this doesn't actually work, even for only 2 timehands and >>>>>>>> a global generation count. >>>>>>> >>>>>>> You are describing the generic race between reader and writer. The same >>>>>>> race would exist even with one timehand (and/or one global generation >>>>>>> counter), where ntp adjustment might come earlier or later of some >>>>>>> consumer accessing the timehands. If timehand instance was read before >>>>>>> tc_windup() run but code consumed the result after the windup, it might >>>>>>> appear as if time went backward, and this cannot be fixed without either >>>>>>> re-reading the time after time-depended calculations were done and >>>>>>> restarting, or some globabl lock ensuring serialization. >>>>>> >>>>>> With 1 timehand, its generation count would be global. I think its ordering >>>>>> is strong enough to ensure serialization. >>>>> Yes, single timehands result in global generation. But it would not solve >>>>> the same race appearing in slightly different manner, as I described above. >>>>> If reader finished while generation number in th was not yet reset, but >>>>> caller uses the result after tc_windup(), the effect is same as if we >>>>> have two th's and reader used the outdated one. >>>> >>>> You described it too concisely for me to understand :-). >>>> >>>> I now see that a single generation count doesn't give serialization. I >>>> thought that setting the generation to 0 at the start of tc_windup() >>>> serialized the reader and writer. But all it does is prevent use of the >>>> results of the windup while only some of them are visible. If the >>>> setting the generation count to 0 doesn't become before tc_windup() reads >>>> the hardware timecounter, then this read may be before other reads using >>>> the old timehands, but it needs to be after. >>> If we have either single th or global gen counter, current code would >>> become serialized, but this is not what I am about. Lets assume, for >> >> No, generation counts used like they are now (or in any way that I can >> see) can't give serialization. >> >>> the sake of the discussion only, that all readers take the same spinlock >>> as tc_windup (i.e. tc_setclock_mtx). >> >> Spinlocks are far too heavyweight. Most of the complications in timecounter >> locking are to avoid using them. But OK for the discussion. >> >>> It is still possible that reader unlocked the mutex, tc_windup() kicked >>> in, locked the mutex and moved timehands (as you noted, this might >>> even happen on the same CPU), and only then the reader continues. For >>> consumer of bintime() or any other function' result, it looks exactly >>> the same as if we did not serialized with writer but used outdated >>> timehands. >> >> Not with full serialization. The writer tc_windup() is also a reader, and >> serializing its read gives the necessary monotonicity (for a single thread): >> - normal reader locks the mutex, reads the timecounter and unlocks. The >> mutex makes visible all previous writes, so the reader doesn't use a >> stale timehands. Consumers of bintime(), etc., use this time in the past. >> - tc_windup() locks the mutex, reads the timecounter hardware and writes the >> timecounter soft state. The new offset is after all previous times read, >> since this is serialized. >> - normal reader as above sees the new state, so it reads times after the >> time of the windup, so also after the time of previous normal reads. >> >>> Let me formulate this diffeently: as far as consumer of the bintime() >>> result does not serialize itself with tc_windup(), serializing bintime() >>> itself against tc_windup() does not close the race, but it is not >>> obvious that the race matters. >> >> Readers can easily see times far in the past, but the times increase in >> program order. > This is true for the current implementation (single-thread monotonic). Provided there is no skew across CPUs. I think this is a requirement for any implementation. >> >>> Either we should just accept the race as >>> we currently do, or readers must take the spinlock where the exact value >>> of the current time is important, >> >> Disabling interrupts should be enough. In my version of 5.2, spinlocks >> don't disable hardware interrupts and may be preempted by fast interrupt >> handlers which may be not so fast and take hundreds of usec. Actually, >> even disabling interrupts might not be enough. A single isa bus read >> can take at least 138 usec (when it is behind a DMA queue or something). >> There are also NMI's and SMI's. >> >>> or readers must re-read the time after >>> doing something important, and redo if the new measuremedtime is outside >>> the acceptable range. >> >> This method seems to be essential for robustness. >> >> But I don't see any race (for a single thread and no timecounter skew >> across CPUs). Sloppy readers just see times an unknown but usually small >> time in the past. Non-sloppy readers can also defend against timecounter >> skew by binding to 1 CPU. > Yes, this is true. > >> >> Mutex locking of the timecounter doesn't give monotonic times across threads. >> It gives some order, but you don't know which. Another mutex or rendezvous >> is needed to control the order. > Define monotonic between threads ? The threads establish an order of events using _any_ reasonable method, e.g., memory ordering. Then if the events are a monotonic clock-reading event E0, any event E1, and a monotonic clock-reading event E2, in the order E0 <= E1 <= E2 according to the reasonable method, then E0 must be <= E2 according to the timespec value of the clock. One thread can arrange for E0 <= E1 very easily using only program order. E1 might consist of setting a flag. Another thread watching this flag can't tell when it was changed, but can tell that it was changed after E0, so E2 in the other thread must be after E0, by program order E1 <= E2 in the other thread. More precisely,there are really 2 different events E1. One (E1a) is setting the flag in the first thread and the other (E1b) is observing the change in the other thread. The full order is E0 <= E1a <= E1b <= E2, where E0 <= E1a and E1b <= E2 by program order in each thread and E1a <= E2b by the arrow of time (this doesn't take any memory ordering magic, except in practice you would have to do more to not miss changes when reusing the flag). Establishing an order using external events doesn't work so well, since the events become visible in different threads at different times. If you have a clock good enough for measuring and compensating for the different times, then you should be using it for the timecounter and can't use it for implementing the timecounter :-). Ticks on a system-wide timer is an example of an external event. Another is arrival of packets at an interface. If the packets are read from the interface by separate threads and put into a buffer using any fast method, then they are likely to become out of order. Hardware timestamps on them would allow fixing up the order, but software timestamps made by separate readers don't work at all without using a necessarily slow method to communicate the order. I think the multiple timehands method works perfectly if the frequency or phase is never adjusted. It is just an optimization of the calculation 'result = base + timecounter_value * inverse_frequency', except for negligible rounding errors. We want to do the multiplication using only 64-bit integers, so we replace the RHS by '(base + offset * inverse_freqency) + (timecountere_value - base) * inverse_frequency'. When inverse_frequency is constant, It doesn't matter when the (base, offset) pair is changed provided the change is atomic and not too far in the past for overflow to occur with 64-bit integers. The method gives atomicity, and a bug apparently gives overflow. Races from decreases in the frequency are harmless (for monotonicity) since they increase the inverse frequency so increase time differences so never give negative differences. So only races from increases in the frequency are a problem. I think changes in the frequency are rare (ntpd normally only makes them every 64 seconds) so it wouldn't hurt to wait for them. The problem is know when to wait. Bruce