Date: Tue, 17 Apr 2018 10:06:47 -0400 From: George Mitchell <george+freebsd@m5p.com> To: freebsd-stable@freebsd.org Subject: more data: SCHED_ULE+PREEMPTION is the problem Message-ID: <a166000b-eeee-c92e-5682-87eb3bafa59b@m5p.com> In-Reply-To: <paak11$vcf$1@oper.dinoex.de> References: <pa17m7$82t$1@oper.dinoex.de> <paak11$vcf$1@oper.dinoex.de>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--y8wTAHtxYHx9ysmdBA2s1Y1q2cunFfqS9
Content-Type: multipart/mixed; boundary="KFC0dGtENmQVZ6oo5TwgAyShvsXQBBeeF";
protected-headers="v1"
From: George Mitchell <george+freebsd@m5p.com>
To: freebsd-stable@freebsd.org
Message-ID: <a166000b-eeee-c92e-5682-87eb3bafa59b@m5p.com>
Subject: more data: SCHED_ULE+PREEMPTION is the problem
References: <pa17m7$82t$1@oper.dinoex.de> <paak11$vcf$1@oper.dinoex.de>
In-Reply-To: <paak11$vcf$1@oper.dinoex.de>
--KFC0dGtENmQVZ6oo5TwgAyShvsXQBBeeF
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
On 04/07/18 10:18, Peter wrote:
> Hi all,
> [...]
Thanks for all the investigation!
> 3. kern.sched.preempt_thresh
>=20
> I could make the problem disappear by changing kern.sched.preempt_thres=
h
> =C2=A0from the default 80 to either 11 (i5-3570T) or 7 (p3) or smaller.=
This
> seems to correspond to the disk interrupt threads, which run at intr:12=
> (i5-3570T) or intr:8 (p3).
> [...]
More data. With SCHED_4BSD at FreeBSD 10.4-RELEASE-p8 #0 r331984:
kern.sched.runq_fuzz: 1
kern.sched.ipiwakeup.useloop: 0
kern.sched.ipiwakeup.usemask: 1
kern.sched.ipiwakeup.delivered: 376139898
kern.sched.ipiwakeup.requested: 376137875
kern.sched.ipiwakeup.enabled: 1
kern.sched.slice: 12
kern.sched.quantum: 94488
kern.sched.name: 4BSD
kern.sched.preemption: 1
kern.sched.cpusetsize: 8
With dnetc running on a 6-core AMD CPU from a few years back,
"time make buildworld" yields:
6640.224u 828.874s 2:14:37.73 92.4% 28525+494k 31633+431554io 33192pf=
+0w
I shifted to a GENERIC kernel, FreeBSD 10.4-RELEASE-p8 #0 r332560:
kern.sched.topology_spec: <groups>
<group level=3D"1" cache-level=3D"0">
<cpu count=3D"6" mask=3D"3f">0, 1, 2, 3, 4, 5</cpu>
<children>
<group level=3D"2" cache-level=3D"2">
<cpu count=3D"6" mask=3D"3f">0, 1, 2, 3, 4, 5</cpu>
</group>
</children>
</group>
</groups>
kern.sched.steal_thresh: 2
kern.sched.steal_idle: 1
kern.sched.balance_interval: 127
kern.sched.balance: 1
kern.sched.affinity: 1
kern.sched.idlespinthresh: 157
kern.sched.idlespins: 10000
kern.sched.static_boost: 152
kern.sched.preempt_thresh: 80
kern.sched.interact: 30
kern.sched.slice: 12
kern.sched.quantum: 94488
kern.sched.name: ULE
kern.sched.preemption: 1
kern.sched.cpusetsize: 8
I stupidly typed "make buildworld" without the "time" command, but the
build log started at Mon Apr 16 13:49:12 EDT 2018 and completed at
Tue Apr 17 00:22:23 EDT 2018. You read that right: 2+ hours vs 10 1/2!
So I set "sysctl kern.sched.preempt_thresh=3D5" (a wild guess on my part)=
and started another "time make buildworld". It's still going now, but
subjectively it's still running like molasses. I'll post more results
later after trying sysctl kern.sched.preempt_thresh=3D0.
By the way, over the years that this discussion has been going on, I've
*never* had a response to my question: "What is the workload for which
SCHED_ULE outperforms SCHED_4BSD?" -- George
--KFC0dGtENmQVZ6oo5TwgAyShvsXQBBeeF--
--y8wTAHtxYHx9ysmdBA2s1Y1q2cunFfqS9
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCAAdFiEENdM4ZHktsJW5kKZXwRES3m+p4fkFAlrV/3gACgkQwRES3m+p
4fk36A//TpmVboHGSjflyl0kawaWT8m/fljSn3qWttRmZqSiAilUXpnAAm8rn6gU
5f9SfqmBLtkeyrnZrNCeVA92VSfQb7pp8NkA5jEoXhp+n+qhuj2qEc2M9ddCRaI7
AGoFjcdMUlPipYQpvcL0y2FtAc7yOawm98LgjhIM+cE1pvSY3tAWKeWqqgn6yCp4
jIc89UrJgRuVEGcdYta0Pbr5ahjV7yO2FRb7fuVz8rl9xfOG+p3JaLjQDEGzpiVu
l3YsM/vtTVxqasbCkhM9ueL9ex/Dqy7U1368IyTXXaPvSpCCUg98ThLMUl8My+MQ
kVNGWHXkmLhzzTCoVyvwhP7B+X9OWyl+Dq8Hs+XejpK5PlmejHhLfP2CICQgGmpT
F0zMfJZQCi8b/Yv7GdfhGBCPTMS5wk6xUHkxGZrpSDYf4yS4luovJ+D+fvCeYyQp
y3EHSpFAgXlgMJo/JMYQ3rVULSPBDt9pxDjq5zdmOq7V6BLTL1bzQuR/qyUSnzgS
WtCsj6FMwBS0nYqnaAOMnEb5M8ujajD/Qc2rE7GajgyPen46PLKAZ85foiK3DbBq
yGr0kVtLgUnCk+fvpddjTjMNNrCJCd/QmgeLE4Gk6vo7oQ6EyBG5xeM8WR0q03z2
pUHzDbgf1TW7w9hSy5K7+Cx7CDBLhI4JynjdvL+CeWxgcyt8gFc=
=kEHm
-----END PGP SIGNATURE-----
--y8wTAHtxYHx9ysmdBA2s1Y1q2cunFfqS9--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?a166000b-eeee-c92e-5682-87eb3bafa59b>