From owner-freebsd-stable@freebsd.org Tue Apr 17 14:12:47 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AF5F2FA65F3 for ; Tue, 17 Apr 2018 14:12:47 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Received: from mailhost.m5p.com (mailhost.m5p.com [74.104.188.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "m5p.com", Issuer "Let's Encrypt Authority X3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 425EF76B47 for ; Tue, 17 Apr 2018 14:12:47 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Received: from [IPv6:2001:470:1f07:15ff::1f] (haymarket.m5p.com [IPv6:2001:470:1f07:15ff::1f]) (authenticated bits=0) by mailhost.m5p.com (8.15.2/8.15.2) with ESMTPSA id w3HE6mqM061427 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Tue, 17 Apr 2018 10:06:58 -0400 (EDT) (envelope-from george+freebsd@m5p.com) From: George Mitchell Subject: more data: SCHED_ULE+PREEMPTION is the problem To: freebsd-stable@freebsd.org References: Openpgp: preference=signencrypt Autocrypt: addr=george+freebsd@m5p.com; prefer-encrypt=mutual; keydata= xsFNBFgnLnwBEADAJDiBKQX77LFRz9wZW8mz3KvaQol2nIremcws0F1mz/zgFlk6uhQVtwnL wb4XL5LdFwcNE1+QZzPLcbYWoWQlz0lBw1bMuKAgr0S6V2e0+I0DqhKeslVFctcTwtvT6pnK VLZXO/7ZGAaLzG4K5vSPzgoevU+YI/pxNsVCH2UO/c3jQW63uEt25mIZbCF1Pu4jgp4RhIgF ujn877r/j6OwBwjzRUu3E6ADp+U825d+5YCuQMEH0wIPnn9GTpXvfdKdbwOIl2akqXqs4cnk iATWfK3r6D4mvDEj1OPHlTvJYcfic7aOIiAwmx1C1v78GjXOdOOA0SGffNix3C2/8oZUO1+V Aet4MKpUKkduWSvULhIkHNZ5Nu8SIJOqge8pmtHxuNXAMfMrAjMdjPwwBFLsYg3Xa2E2oJwg ehTauwd/EDJFcVCyDCyCAYOi/BH/+XQyxzgDlY9N9qj9tHqhVPI6XK7t8UVffGiZUq4rHp5J RdOToqiTNC6eCJBczhMIW+DuFvWU9e6W708T1dz0Accn6Lrgk4eRIn3GFPBG+TxnpjAqHsbW 607dcnD3YKAqY4e+khczL4EObhe7dC1v2fmZiAC6Ds3WHR11IfqoUgCkIwJ590Ej+ElygJFF XxI82wtEz9hkeLLvItpyEJNVjppViRW+Dgl/U7ypHB3qDgYjgwARAQABzSBHZW9yZ2UgTWl0 Y2hlbGwgPGdlb3JnZUBtNXAuY29tPsLBlwQTAQgAQQIbIwUJCWYBgAULCQgHAgYVCAkKCwIE FgIDAQIeAQIXgBYhBDXTOGR5LbCVuZCmV8EREt5vqeH5BQJYWXFRAhkBAAoJEMEREt5vqeH5 SRsQAIb/crcxXyAyeokAuTjN+YXkEFdVv/JrOgNXKdCukXt/UGd3nZTcAzpllytDIvIlPTNI 2/nZ5sm6ymeyVwmvkrM/r3sRUib5ftakJpbv0wn2j+eCGAca8IA2frBUg9bEXKcHRJRQCztG cpousHzOziTCRQ/1NfPcIFBNbQMPUVoQ96cJPvM/XfFGOISKKsSI78skHm6Oazh1hLCQTKvy hNNgVpNP/PHCHMbla1+SNgyn35CUelTh153lnkOhw1XyX6IxY4o5Bhcf3YrxAVcoeHq3FEH0 T3ygAdI14VrZGXXitBAMI78QLB0FiWMoPQ3Oddnoh6tlT4djnsc9IW0Tzk6ozvQL7sKdYgO8 ZlIpkBqQQfpSHzwPu9EkOFggPWB9WtP3IajQ0lt1LovqMug4C8APRC2/1cvi+GUIRwjVsop0 ukhlLTTJd3/S4Muh3s87M4Rdek1xpOiMKjYOVaxmhQQ91oz971zJuJJWpX7uUQiXx9oEwCDW TvI5yEuqYLsMUwx3d2iFTr+HbtlBJmF+Vguyrn/a3vFK8P/TH9fMvNeTdln9SuOOa1SAMMcy czOpBYg9RpzNLshUJVrhKzugT59Rl2wsNQsQCUkzgF3f8cZHJyl+8x6t0nuM9LTkMv13YIXS Mde3UOD2EaBhmeIqvC/adQHxpNudvxM1viFJDnTnzsFNBFgnLnwBEAC7kzsZqjBRPonnr/63 C98FSa3LikvqQWygmPSCC9DsFX/fB0CSXIHTrHQ4a7lXdfZyTZcGdxXN+MC8O8thjvVq6WYm CpyOJ/bq4SxOa9cnQSJ5SP9VCmVoDN/3T4ybXFzLAt756kfQ5jsVuP0m6iQ4z918zhZXk+Mo qdwGjYTxsBD02a7m1aeYafyaI2mZ+vdEy0cDhV4PDXI6ThLNAavTPji6ZrBdH5a4LMg30u8v kkNe0eCKvU+0cWb2VQIeddMhhiGSBE2Dv+A4eNe1VZvoGpLDlYdnoHraVFL2GHNFGymj/uRf 1hja5kW9Rekisqby5SpGABwrFFEs7ABpfYG0IRBKbjjG1Cqdfe/R6ETJFvvNOOpCKPAWeqYf Isxv4OHWTmhKihhIanYWCAe1MDLRRbj7UrOTZeia2WJ4v58xbU5rVQoHI/Tzaq86rXzRWITc 5w+kresMad0zpvQ900BdHc8ATNY2aW/Fr5it5OqMvIW6Y4gc8Z95MkQPVc8vj3WzfxuWtZNK 6Wbv4r6Qbiwg1RpY1JoEIoF4OsZJOVMQaB6ezD7xvaa2eY6nRGtq9SoJxo2qvlSbLlKq8NdH VYHhtTbpQ1NfrEJfv5sLX5W5IpoYww48bFYH67+7r1f/W3voBptSgE7qnYAm6Em9bEAKOQEX 8BXwoa54fn03z6TyhwARAQABwsFkBBgBCAAPBQJYJy58AhsMBQkJZgGAAAoJEMEREt5vqeH5 6PcP+JvrMM7ZM8UlnbrY4Er9psPj3ayllRhQFA9h6GNUKYuSzSxOrPaT96s8KUGMCr4jrn1S WFmeeNLLOgSJmQRicMh6LmnKq6WY6UaOfn7Y9O62NUjXfEI3Bw4ID36YCdQ+CJd14r0YOf4M 5F50bvHV3lbzD9TXZPxecHKC2ZUMBbT37tsckWCLL3lzKMsqQLwBUmgBl1NIUc7gyXxiNyxb 6SPVF1NguDDM438mcg9jSRAyjgAk6POUEM8YIXkw0Gg6HF1tNMJJ1xTMBCLYl6fHTtsxJpf9 yo+Hnw346hqYzXn4ytHJ49Ngcre8uhqM1l8iMpa17tEjfalkc1FWR9/qvoowOKtxpvblsy3a YzeEFgIomhLzISz6IafQ3S7Mt5AFlqwN/qQHx0k2V66GzDG0ngZBPROP1sXSpdJzO0zbJQFn MZE3f+y8vXMcE/MBXR7kAdYYApiEMQzVxy9TdQDU3lGLptcPZ1IOntTNFFrvp5NwsKi+6C9i mXtd5kJ1PwhcJYW3/ov/490l60C5SFUL/RZ/NOW8SHFaPcqlGcqIlexFKbzrMQwmYXo95jWB eZ0Qn+raxCUFZNGiwtusyQGBMcpHVJUanOCNd1z4ZbfmhUjDJKC/7YWDunvaDRSukGiRCl6J s8caqXHiVZjx+s76iWzm6AHRP5jg9D6EtTOrGiE= Message-ID: Date: Tue, 17 Apr 2018 10:06:47 -0400 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="y8wTAHtxYHx9ysmdBA2s1Y1q2cunFfqS9" X-Spam-Status: No, score=0.2 required=10.0 tests=HELO_MISC_IP, RP_MATCHES_RCVD autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mattapan.m5p.com X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.6.2 (mailhost.m5p.com [IPv6:2001:470:1f07:15ff::f7]); Tue, 17 Apr 2018 10:07:04 -0400 (EDT) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Apr 2018 14:12:47 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --y8wTAHtxYHx9ysmdBA2s1Y1q2cunFfqS9 Content-Type: multipart/mixed; boundary="KFC0dGtENmQVZ6oo5TwgAyShvsXQBBeeF"; protected-headers="v1" From: George Mitchell To: freebsd-stable@freebsd.org Message-ID: Subject: more data: SCHED_ULE+PREEMPTION is the problem References: In-Reply-To: --KFC0dGtENmQVZ6oo5TwgAyShvsXQBBeeF Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 04/07/18 10:18, Peter wrote: > Hi all, > [...] Thanks for all the investigation! > 3. kern.sched.preempt_thresh >=20 > I could make the problem disappear by changing kern.sched.preempt_thres= h > =C2=A0from the default 80 to either 11 (i5-3570T) or 7 (p3) or smaller.= This > seems to correspond to the disk interrupt threads, which run at intr:12= > (i5-3570T) or intr:8 (p3). > [...] More data. With SCHED_4BSD at FreeBSD 10.4-RELEASE-p8 #0 r331984: kern.sched.runq_fuzz: 1 kern.sched.ipiwakeup.useloop: 0 kern.sched.ipiwakeup.usemask: 1 kern.sched.ipiwakeup.delivered: 376139898 kern.sched.ipiwakeup.requested: 376137875 kern.sched.ipiwakeup.enabled: 1 kern.sched.slice: 12 kern.sched.quantum: 94488 kern.sched.name: 4BSD kern.sched.preemption: 1 kern.sched.cpusetsize: 8 With dnetc running on a 6-core AMD CPU from a few years back, "time make buildworld" yields: 6640.224u 828.874s 2:14:37.73 92.4% 28525+494k 31633+431554io 33192pf= +0w I shifted to a GENERIC kernel, FreeBSD 10.4-RELEASE-p8 #0 r332560: kern.sched.topology_spec: 0, 1, 2, 3, 4, 5 0, 1, 2, 3, 4, 5 kern.sched.steal_thresh: 2 kern.sched.steal_idle: 1 kern.sched.balance_interval: 127 kern.sched.balance: 1 kern.sched.affinity: 1 kern.sched.idlespinthresh: 157 kern.sched.idlespins: 10000 kern.sched.static_boost: 152 kern.sched.preempt_thresh: 80 kern.sched.interact: 30 kern.sched.slice: 12 kern.sched.quantum: 94488 kern.sched.name: ULE kern.sched.preemption: 1 kern.sched.cpusetsize: 8 I stupidly typed "make buildworld" without the "time" command, but the build log started at Mon Apr 16 13:49:12 EDT 2018 and completed at Tue Apr 17 00:22:23 EDT 2018. You read that right: 2+ hours vs 10 1/2! So I set "sysctl kern.sched.preempt_thresh=3D5" (a wild guess on my part)= and started another "time make buildworld". It's still going now, but subjectively it's still running like molasses. I'll post more results later after trying sysctl kern.sched.preempt_thresh=3D0. By the way, over the years that this discussion has been going on, I've *never* had a response to my question: "What is the workload for which SCHED_ULE outperforms SCHED_4BSD?" -- George --KFC0dGtENmQVZ6oo5TwgAyShvsXQBBeeF-- --y8wTAHtxYHx9ysmdBA2s1Y1q2cunFfqS9 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEENdM4ZHktsJW5kKZXwRES3m+p4fkFAlrV/3gACgkQwRES3m+p 4fk36A//TpmVboHGSjflyl0kawaWT8m/fljSn3qWttRmZqSiAilUXpnAAm8rn6gU 5f9SfqmBLtkeyrnZrNCeVA92VSfQb7pp8NkA5jEoXhp+n+qhuj2qEc2M9ddCRaI7 AGoFjcdMUlPipYQpvcL0y2FtAc7yOawm98LgjhIM+cE1pvSY3tAWKeWqqgn6yCp4 jIc89UrJgRuVEGcdYta0Pbr5ahjV7yO2FRb7fuVz8rl9xfOG+p3JaLjQDEGzpiVu l3YsM/vtTVxqasbCkhM9ueL9ex/Dqy7U1368IyTXXaPvSpCCUg98ThLMUl8My+MQ kVNGWHXkmLhzzTCoVyvwhP7B+X9OWyl+Dq8Hs+XejpK5PlmejHhLfP2CICQgGmpT F0zMfJZQCi8b/Yv7GdfhGBCPTMS5wk6xUHkxGZrpSDYf4yS4luovJ+D+fvCeYyQp y3EHSpFAgXlgMJo/JMYQ3rVULSPBDt9pxDjq5zdmOq7V6BLTL1bzQuR/qyUSnzgS WtCsj6FMwBS0nYqnaAOMnEb5M8ujajD/Qc2rE7GajgyPen46PLKAZ85foiK3DbBq yGr0kVtLgUnCk+fvpddjTjMNNrCJCd/QmgeLE4Gk6vo7oQ6EyBG5xeM8WR0q03z2 pUHzDbgf1TW7w9hSy5K7+Cx7CDBLhI4JynjdvL+CeWxgcyt8gFc= =kEHm -----END PGP SIGNATURE----- --y8wTAHtxYHx9ysmdBA2s1Y1q2cunFfqS9--