From nobody Sat Mar 25 22:35:36 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PkYnj6C2Kz41pj9 for ; Sat, 25 Mar 2023 22:35:53 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic301-22.consmr.mail.gq1.yahoo.com (sonic301-22.consmr.mail.gq1.yahoo.com [98.137.64.148]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PkYnj1FYkz48sS for ; Sat, 25 Mar 2023 22:35:53 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679783751; bh=Yol58pV/rLr38QiELsLObJTdfgFajduQyOavu7zNkQc=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=Rjzm2WWqTzHhOTt1vX0FSnipOsUoQNkNNSjVnT1m8zqH/+jo8bvCnlSw3x8gTOpSIY8bQNOt/zV8NQ4cmY6GiUeAA0nbq2HgikL9hFrhfkjHlSFDKaVNVFsMSg++SmVKxb4GFigi1GAVAbhaU0kk283aQNhPbqZx0YEEa05yTNh2ed4j1u3SBkil4263nM5wNBTiLZAd6zmdnMhigIQIhz9we2SH7/HmMe3fulm4ChM264gcWBaonqo+njFjHWWTGyELmBAkeYTbxveZh0eKKM32EkHAwXl4yTI1bkc8ixJ+TAckSDQzOQlnGMeRBUVgLMBlWscYD/VsBy9iC/7/jg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679783751; bh=suhqWNlWwL+8P8iU2V9Ny0PD7NTSxrB3uaflPreqD3b=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=H52vfciWKnC0VEhaK0vuSRtcd1haXKIcECnSsYUHU9QGqJRS1aijEwVfelKP2lJUQ32Yaa/ERh1B8acZOCMtSLXLq/p4lx39EXcpEJ18QAJPk/kkkbFsT7uowaboyt6Nv/3+y82Ng0Z35/vJBnzOdf70JZ2DXAY0qotVNZFc9kM7D239mFcA69tCrR2FadNS/99nUII3PzRqGMqYVIzwmg+9sBBWPT9Tf4G12qTgmKl0UkA7ngvySxNZ8mli426TGX+1FZkpku/uZrovHp96eKzd1UAogiqVZBmtkpVefbifhJ0Zrm4h5pIKsAIv5xYPbMz/B07Rh1neFhl7qDT9+g== X-YMail-OSG: ZVDXDiMVM1nWjjAnJ.hytJcoAHL_BKqHcgaZuZrYEStf5Z1ubVMsgNVD95Af9G0 SdpfClazpFldzePAW5o0uDM3FA7ccpbY3Fmxi0dh_JUiNMzGZPG4VdnsxvlnsoQhueY6HymcnUs3 dpc7pY1yEVbr1TCACvUTFg9BV9P1Ez_wThse96G7FYgBsRgUxdGuxe4WNGZthVhR4lE.fHkpL1kw IuOH9Lfe48mvZwOoUiwPNajxSFFfLaL_4kjdI0xKR.9qHVFBSEqjjOPhYJCZOCNczpU_X.5hrbqx 7SW.oH9v6bgYh0zZcniTUB.8xn.5OPb_ccepGLdeAHaDgvN4t4XSeYK..t69g0JWVTWAT5uFNCLO ZjHOz35AsLvXNlD4LswTglV3GLuBoFjI5o2XR_bXK6Ic26sG.m9_t065tXoUcX3tAjw_OQp6cm_D v7.KyrNcvIS_PslrV7a0kZIfXQaBurwbratiCNcYbbMU5J4MEXpCIkZE.CJKCU4AYiIwPDsuO6Cx vG0AFQUfVsx01cOLJqzu3wcQM4G0PSQTLrcWv7KZptHSONsXw_IqBgN5R.qlYRHzkiTl0rD9sR4M rTj40882FQfUhBEmUoakaatCj8e0vxJ7smVTkWCF4lca7kZV.Hwpm8hNp3iXl9ChIABuI9fOtZ83 RhmVV7LTBiENFkelAyJyCMqg4ofwZsg6mfTKwjUzmGaXN92lm5OY_IznclHM.ri4mYZDDmLIwxX0 yTg9r9ZKcTrpPY7rhsHlB4W0.jbrN.CbmmgWB4f3qzwAaWlp9fu_oM8EEj1R7L42skmtdswM7iJq iWlng2mX7DUc_mskSTnrfbh5RCiLIW5hhyQ0iuoKZS07kULCyUbiFsYJErCeyViMpZ5K.ANhhxa_ RIOdinu3C6LEWALAREC.N0L3A_YiH9w1aKj8X5V4TLt2hpJqZ02w9EaST6W87I_yAIIlri5ge2Si EYGzhLTAx2i8QFDOtkTSt91WNcem72Bwdp3JxykhLzyD5KWWYTR5LSvm7LHiDswqc6cRbN0Upad6 DaHtYUyXxhOYK3XbXi03DuCfKbsKiSNyPJiM_liKGtNR_2i7TOavTixcKTxGNQhYvXErIBitIe8h lXKBagOwrkzE.Cm5ipDCPtjeu.NmEumNDLCRPLlE2ZrQve.f1QN7rCh_1JV_mh4fVRs6AdztlD72 L9ZHGJxiJaSKc4JrP528eMC3OBkBCuzsr3M5y8S2GotkXJO3o.JTedzEDXDFZFw7OxECI0GtAtcY 98jougNa7nOrIjYG9RQ1hiMdQSImokYLaeZXwkcjhTTqw9YJeu4JrDvaGFaU8wiGzQ6X5rvfvy4U iJPANe1d8bXsOOkAx1DL6P83LKsU7BzbXu0IRxtDHu26Py7vaGmyyFqGQDWC0r2CD_QQdNzqx8Oe 47HSmADAjPJl8mMwDEeS2oDA5DZpV5wJlsH3aUm2rZ0.wKpb_AYY.TQWPgfGryjufxBMPj8mFnE5 P2czRD2kakODylTJRxf9G99BGMgUmr_GkZZA.csT7X7d12pmRy.N8dUaKLBTN3edM7EH7nLV4I_M KDogzAzyyi85xT.xwEdxy9mWnqZmfy.PWAMySBkYoUNjPnKaZ4IE_CtPKEuUrPSR5wc4L_x0DC1x UgAdRSpR4Zi9kjw7.lNCaR2I9Hz0MljchQ0BMyUEWJYFEK3AEn_PnP3vgZiHt0E7qHvLrFRADLJo ngOMH3BrCMnDvR7L1E85XkGGrNQIA_Grm5EQ8VeMQ8fCqucEBWtqrKF5fWCGai5uInOTesHkcjAv reCWxxJ1vVfdlPwLh8ydol01X3LZ.m4TzuJiBi8FMtqYatc_94x4fX07jI6SnZ9Ac0HhVjV3jRKQ gUd8mLlUC_sAwIqf6vnjHnuURh5VDG29IZhkOJGkrPf9KPrd2zqQYUhvxnzhJUG.IWDavC2IvmUU q7dRQ.MgS.W2XziYmBZC_Ldkgx1fs6JjX7JA9C1GHYUgjwntt2aQrjtvru7SAp_DYfFSCWPwXvsZ fYORPvvlG1ULjcHA080DD8Cj_gPn7quvMFqnWHkf5v.to9.iRAuOjkcz4aVDNjIb7rHmFtShhHVJ ixq5DINQOQwDMdcncOzwmn4b5k3ZT4PJrFyPlOmwQvtHoL6l39SGL95C_ctT2RxnQiSMhKgPFyaV 8ijiiXzgkQsASVq9ixDgPgRrxE8G80YRbxyXRjA7nY58OwT7P9ieYgcwiH36d.ZYNI.gnup6CwsI YcwuJKl_MrQT_1VjtjKj0ZVoABjiVhSmwwplA8xSD7dfrNavfvDFDo0aCtc.D1fjCNVVoyLnalc_ VSI_pTlWIkuM- X-Sonic-MF: X-Sonic-ID: 92a80676-41ad-4662-942c-061be1f69067 Received: from sonic.gate.mail.ne1.yahoo.com by sonic301.consmr.mail.gq1.yahoo.com with HTTP; Sat, 25 Mar 2023 22:35:51 +0000 Received: by hermes--production-gq1-6cf7749bc8-g5z7v (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID e30ae660f78010d228b43b0325d0c153; Sat, 25 Mar 2023 22:35:46 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Periodic rant about SCHED_ULE From: Mark Millard In-Reply-To: Date: Sat, 25 Mar 2023 15:35:36 -0700 Cc: Mateusz Guzik Content-Transfer-Encoding: quoted-printable Message-Id: <21FEBFC6-7A43-43C6-A888-EC9962D738B8@yahoo.com> References: <5AF26266-5B4C-4A7F-8784-4C6308B6C5CA.ref@yahoo.com> <5AF26266-5B4C-4A7F-8784-4C6308B6C5CA@yahoo.com> To: Peter , FreeBSD Hackers X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Rspamd-Queue-Id: 4PkYnj1FYkz48sS X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N > On Mar 25, 2023, at 14:51, Peter wrote: >=20 > On Sat, Mar 25, 2023 at 01:41:16PM -0700, Mark Millard wrote: > ! On Mar 25, 2023, at 11:58, Peter = wrote: >=20 > ! > !=20 > ! > ! At which point I get the likes of: > ! > !=20 > ! > ! 17129 root 1 68 0 14192Ki 3628Ki RUN 13 = 0:20 3.95% gzip -9 > ! > ! 17128 root 1 20 0 58300Ki 13880Ki pipdwt 18 = 0:00 0.27% tar cvf - / (bsdtar) > ! > ! 17097 root 1 133 0 13364Ki 3060Ki CPU13 13 = 8:05 95.93% sh -c while true; do :; done > ! > !=20 > ! > ! up front. > ! >=20 > ! > Ah. So? To me this doesn't look good. If both jobs are runnable, = they > ! > should each get ~50%. > ! >=20 > ! > ! For reference, I also see the likes of the following from > ! > ! "gstat -spod" (it is a root on ZFS context with PCIe Optane = media): > ! >=20 > ! > So we might assume that indeed both jobs are runable, and the only > ! > significant difference is that one does system calls while the = other > ! > doesn't. > ! >=20 > ! > The point of this all is: identify the malfunction with the most > ! > simple usecase. (And for me here is a malfunction.) > ! > And then, obviousely, fix it. > !=20 > ! I tried the following that still involves pipe-io but avoids > ! file system I/O (so: simplifying even more): > !=20 > ! cat /dev/random | cpuset -l 13 gzip -9 >/dev/null 2>&1 > !=20 > ! mixed with: > !=20 > ! cpuset -l 13 sh -c "while true; do :; done" & > !=20 > ! So far what I've observed is just the likes of: > !=20 > ! 17736 root 1 112 0 13364Ki 3048Ki RUN 13 2:03 = 53.15% sh -c while true; do :; done > ! 17735 root 1 111 0 14192Ki 3676Ki CPU13 13 2:20 = 46.84% gzip -9 > ! 17734 root 1 23 0 12704Ki 2364Ki pipewr 24 0:14 = 4.81% cat /dev/random > !=20 > ! Simplifying this much seems to get a different result. >=20 > Okay, then you have simplified too much and the malfunction is not > visible anymore. >=20 > ! Pipe I/O of itself does not appear to lead to the > ! behavior you are worried about. >=20 > How many bytes does /dev/random deliver in a single read() ? >=20 > ! Trying cat /dev/zero instead ends up similar: > !=20 > ! 17778 root 1 111 0 14192Ki 3672Ki CPU13 13 0:20 = 51.11% gzip -9 > ! 17777 root 1 24 0 12704Ki 2364Ki pipewr 30 0:02 = 5.77% cat /dev/zero > ! 17736 root 1 112 0 13364Ki 3048Ki RUN 13 6:36 = 48.89% sh -c while true; do :; done > !=20 > ! It seems that, compared to using tar and a file system, there > ! is some significant difference in context that leads to the > ! behavioral difference. It would probably be of interest to know > ! what the distinction(s) are in order to have a clue how to > ! interpret the results. >=20 > I can tell you: > With tar, tar can likely not output data from more than one input > file in a single output write(). So, when reading big files, we > get probably 16k or more per system call over the pipe. But if the > files are significantly smaller than that (e.g. in /usr/include), > then we get gzip doing more system calls per time unit. And that > makes a difference, because a system call goes into the scheduler > and reschedules the thread. >=20 > This 95% vs. 5% imbalance is the actual problem that has to be > addressed, because this is not suitable for me, I cannot wait for my > tasks starving along at a tenth of the expected compute only because > some number crunching does also run on the core. >=20 > Now, reading from /dev/random cannot reproduce it. Reading from > tar can reproduce it under certain conditions - and that is all that > is needed. The suggestion that the size of the transfers into the first pipe matters is back up by experiments with the likes of: dd if=3D/dev/zero bs=3D128 | cpuset -l 13 gzip -9 >/dev/null 2>&1 & vs. dd if=3D/dev/zero bs=3D132 | cpuset -l 13 gzip -9 >/dev/null 2>&1 & vs. dd if=3D/dev/zero bs=3D133 | cpuset -l 13 gzip -9 >/dev/null 2>&1 & vs. dd if=3D/dev/zero bs=3D192 | cpuset -l 13 gzip -9 >/dev/null 2>&1 & vs. dd if=3D/dev/zero bs=3D1k | cpuset -l 13 gzip -9 >/dev/null 2>&1 & vs. dd if=3D/dev/zero bs=3D4k | cpuset -l 13 gzip -9 >/dev/null 2>&1 & vs. dd if=3D/dev/zero bs=3D16k | cpuset -l 13 gzip -9 >/dev/null 2>&1 & (just examples) as what is paired up with: cpuset -l 13 sh -c "while true; do :; done" & Such avoids the uncontrolled variability in use of tar against a file system. But an interesting comparison/contrast results from, for example: dd if=3D/dev/zero bs=3D128 | cpuset -l 13 gzip -9 >/dev/null 2>&1 & vs. dd if=3D/dev/random bs=3D128 | cpuset -l 13 gzip -9 >/dev/null 2>&1 & as what is paired with the: cpuset -l 13 sh -c "while true; do :; done" = & At least in my context, the /dev/zero one ends up with: 18251 root 1 68 0 14192Ki 3676Ki RUN 13 0:02 = 1.07% gzip -9 18250 root 1 20 0 12820Ki 2484Ki pipewr 29 0:02 = 1.00% dd if=3D/dev/zero bs=3D128 18177 root 1 135 0 13364Ki 3048Ki CPU13 13 14:47 = 98.93% sh -c while true; do :; done but the /dev/random one ends up with: 18253 root 1 108 0 14192Ki 3676Ki CPU13 13 0:09 = 50.74% gzip -9 18252 root 1 36 0 12820Ki 2488Ki pipewr 30 0:03 = 16.96% dd if=3D/dev/random bs=3D128 18177 root 1 115 0 13364Ki 3048Ki RUN 13 15:45 = 49.26% sh -c while true; do :; done It appears that the CPU time (or more) for the dd feeding the first pipe matters for the overall result, not just the bs=3D value used. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Sun Mar 26 01:17:39 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PkdbR5NGJz4121J for ; Sun, 26 Mar 2023 01:27:15 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PkdbR2RxDz4P9L for ; Sun, 26 Mar 2023 01:27:14 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; none Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.17.1/8.17.1) with ESMTPS id 32Q1R8Dn085482 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sun, 26 Mar 2023 03:27:09 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from uucp@localhost) by uucp.dinoex.org (8.17.1/8.17.1/Submit) with UUCP id 32Q1R8vD085481; Sun, 26 Mar 2023 03:27:08 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (disp-e.intra.daemon.contact [IPv6:fd00:0:0:0:0:0:0:112]) by admn.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32Q1JdT2088530 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK); Sun, 26 Mar 2023 03:19:39 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (localhost [127.0.0.1]) by disp.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32Q1HdA3015531 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sun, 26 Mar 2023 03:17:40 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from pmc@localhost) by disp.intra.daemon.contact (8.17.1/8.17.1/Submit) id 32Q1Hdmf015530; Sun, 26 Mar 2023 03:17:39 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) X-Authentication-Warning: disp.intra.daemon.contact: pmc set sender to pmc@citylink.dinoex.sub.org using -f Date: Sun, 26 Mar 2023 03:17:39 +0200 From: Peter To: Mateusz Guzik Cc: freebsd-hackers@freebsd.org, Mark Millard Subject: Re: Periodic rant about SCHED_ULE Message-ID: References: <5AF26266-5B4C-4A7F-8784-4C6308B6C5CA.ref@yahoo.com> <5AF26266-5B4C-4A7F-8784-4C6308B6C5CA@yahoo.com> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Sun, 26 Mar 2023 03:27:11 +0200 (CEST) X-Rspamd-Queue-Id: 4PkdbR2RxDz4P9L X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Sat, Mar 25, 2023 at 11:09:56PM +0100, Mateusz Guzik wrote: ! So far it's look like it's not syscall usage per se, but "voluntary" ! time spent off cpu. No. ! Any time a thread goes off in a manner different than preemption it ! adds to counters which ULE interprets to mean the thread is "less ! interactive" and threads which do stay on cpu on their own should be ! preferred -- this is how syscall-less hog is considered important. No. ! The mechanism lacks distinction between sleeps on purpose, waitpid, ! yield and similar vs going off cpu due to lock contention or ! initiating i/o. Even then, I don't know if the mechanism works at ! all. No. I've said it often enough: preemption does not schedule, it immediately resumes the thread afterwards. Syscalls do schedule, therefore syscalls get into disadvantage. And my patch does nothing else than schedule the thread after preemption, so that preemption is on par with syscalls. That costs a few cycles, but it might also increase responsiveness slightly. Anyway I need a balanced system, not a superfast one. From nobody Sun Mar 26 01:18:20 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PkdbZ5w5kz411c7 for ; Sun, 26 Mar 2023 01:27:22 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PkdbZ49Krz4PvX for ; Sun, 26 Mar 2023 01:27:22 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; none Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.17.1/8.17.1) with ESMTPS id 32Q1R5L2085434 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sun, 26 Mar 2023 03:27:06 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from uucp@localhost) by uucp.dinoex.org (8.17.1/8.17.1/Submit) with UUCP id 32Q1R5Un085433; Sun, 26 Mar 2023 03:27:05 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (disp-e.intra.daemon.contact [IPv6:fd00:0:0:0:0:0:0:112]) by admn.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32Q1JdT4088530 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK); Sun, 26 Mar 2023 03:19:39 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (localhost [127.0.0.1]) by disp.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32Q1IKSI015536 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sun, 26 Mar 2023 03:18:20 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from pmc@localhost) by disp.intra.daemon.contact (8.17.1/8.17.1/Submit) id 32Q1IKOc015535; Sun, 26 Mar 2023 03:18:20 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) X-Authentication-Warning: disp.intra.daemon.contact: pmc set sender to pmc@citylink.dinoex.sub.org using -f Date: Sun, 26 Mar 2023 03:18:20 +0200 From: Peter To: Mark Millard Cc: FreeBSD Hackers , Mateusz Guzik Subject: Re: Periodic rant about SCHED_ULE Message-ID: References: <5AF26266-5B4C-4A7F-8784-4C6308B6C5CA.ref@yahoo.com> <5AF26266-5B4C-4A7F-8784-4C6308B6C5CA@yahoo.com> <21FEBFC6-7A43-43C6-A888-EC9962D738B8@yahoo.com> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <21FEBFC6-7A43-43C6-A888-EC9962D738B8@yahoo.com> X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Sun, 26 Mar 2023 03:27:08 +0200 (CEST) X-Rspamd-Queue-Id: 4PkdbZ49Krz4PvX X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Sat, Mar 25, 2023 at 03:35:36PM -0700, Mark Millard wrote: ! > On Mar 25, 2023, at 14:51, Peter wrote: ! > ! > On Sat, Mar 25, 2023 at 01:41:16PM -0700, Mark Millard wrote: ! > ! On Mar 25, 2023, at 11:58, Peter wrote: ! > ! > ! > ! ! > ! > ! At which point I get the likes of: ! > ! > ! ! > ! > ! 17129 root 1 68 0 14192Ki 3628Ki RUN 13 0:20 3.95% gzip -9 ! > ! > ! 17128 root 1 20 0 58300Ki 13880Ki pipdwt 18 0:00 0.27% tar cvf - / (bsdtar) ! > ! > ! 17097 root 1 133 0 13364Ki 3060Ki CPU13 13 8:05 95.93% sh -c while true; do :; done ! > ! > ! ! > ! > ! up front. ! > ! > ! > ! > Ah. So? To me this doesn't look good. If both jobs are runnable, they ! > ! > should each get ~50%. ! > ! > ! > ! > ! For reference, I also see the likes of the following from ! > ! > ! "gstat -spod" (it is a root on ZFS context with PCIe Optane media): ! > ! > ! > ! > So we might assume that indeed both jobs are runable, and the only ! > ! > significant difference is that one does system calls while the other ! > ! > doesn't. ! > ! > ! > ! > The point of this all is: identify the malfunction with the most ! > ! > simple usecase. (And for me here is a malfunction.) ! > ! > And then, obviousely, fix it. ! > ! ! > ! I tried the following that still involves pipe-io but avoids ! > ! file system I/O (so: simplifying even more): ! > ! ! > ! cat /dev/random | cpuset -l 13 gzip -9 >/dev/null 2>&1 ! > ! ! > ! mixed with: ! > ! ! > ! cpuset -l 13 sh -c "while true; do :; done" & ! > ! ! > ! So far what I've observed is just the likes of: ! > ! ! > ! 17736 root 1 112 0 13364Ki 3048Ki RUN 13 2:03 53.15% sh -c while true; do :; done ! > ! 17735 root 1 111 0 14192Ki 3676Ki CPU13 13 2:20 46.84% gzip -9 ! > ! 17734 root 1 23 0 12704Ki 2364Ki pipewr 24 0:14 4.81% cat /dev/random ! > ! ! > ! Simplifying this much seems to get a different result. ! > ! > Okay, then you have simplified too much and the malfunction is not ! > visible anymore. ! > ! > ! Pipe I/O of itself does not appear to lead to the ! > ! behavior you are worried about. ! > ! > How many bytes does /dev/random deliver in a single read() ? ! > ! > ! Trying cat /dev/zero instead ends up similar: ! > ! ! > ! 17778 root 1 111 0 14192Ki 3672Ki CPU13 13 0:20 51.11% gzip -9 ! > ! 17777 root 1 24 0 12704Ki 2364Ki pipewr 30 0:02 5.77% cat /dev/zero ! > ! 17736 root 1 112 0 13364Ki 3048Ki RUN 13 6:36 48.89% sh -c while true; do :; done ! > ! ! > ! It seems that, compared to using tar and a file system, there ! > ! is some significant difference in context that leads to the ! > ! behavioral difference. It would probably be of interest to know ! > ! what the distinction(s) are in order to have a clue how to ! > ! interpret the results. ! > ! > I can tell you: ! > With tar, tar can likely not output data from more than one input ! > file in a single output write(). So, when reading big files, we ! > get probably 16k or more per system call over the pipe. But if the ! > files are significantly smaller than that (e.g. in /usr/include), ! > then we get gzip doing more system calls per time unit. And that ! > makes a difference, because a system call goes into the scheduler ! > and reschedules the thread. ! > ! > This 95% vs. 5% imbalance is the actual problem that has to be ! > addressed, because this is not suitable for me, I cannot wait for my ! > tasks starving along at a tenth of the expected compute only because ! > some number crunching does also run on the core. ! > ! > Now, reading from /dev/random cannot reproduce it. Reading from ! > tar can reproduce it under certain conditions - and that is all that ! > is needed. ! ! The suggestion that the size of the transfers into the ! first pipe matters is back up by experiments with the ! likes of: ! ! dd if=/dev/zero bs=128 | cpuset -l 13 gzip -9 >/dev/null 2>&1 & ! vs. ! dd if=/dev/zero bs=132 | cpuset -l 13 gzip -9 >/dev/null 2>&1 & ! vs. ! dd if=/dev/zero bs=133 | cpuset -l 13 gzip -9 >/dev/null 2>&1 & ! vs. ! dd if=/dev/zero bs=192 | cpuset -l 13 gzip -9 >/dev/null 2>&1 & ! vs. ! dd if=/dev/zero bs=1k | cpuset -l 13 gzip -9 >/dev/null 2>&1 & ! vs. ! dd if=/dev/zero bs=4k | cpuset -l 13 gzip -9 >/dev/null 2>&1 & ! vs. ! dd if=/dev/zero bs=16k | cpuset -l 13 gzip -9 >/dev/null 2>&1 & ! ! (just examples) as what is paired up with: ! ! cpuset -l 13 sh -c "while true; do :; done" & ! ! Such avoids the uncontrolled variability in use of tar ! against a file system. ! ! But an interesting comparison/contrast results from, for ! example: ! ! dd if=/dev/zero bs=128 | cpuset -l 13 gzip -9 >/dev/null 2>&1 & ! vs. ! dd if=/dev/random bs=128 | cpuset -l 13 gzip -9 >/dev/null 2>&1 & ! ! as what is paired with the: cpuset -l 13 sh -c "while true; do :; done" & ! ! At least in my context, the /dev/zero one ends up with: ! ! 18251 root 1 68 0 14192Ki 3676Ki RUN 13 0:02 1.07% gzip -9 ! 18250 root 1 20 0 12820Ki 2484Ki pipewr 29 0:02 1.00% dd if=/dev/zero bs=128 ! 18177 root 1 135 0 13364Ki 3048Ki CPU13 13 14:47 98.93% sh -c while true; do :; done ! ! but the /dev/random one ends up with: ! ! 18253 root 1 108 0 14192Ki 3676Ki CPU13 13 0:09 50.74% gzip -9 ! 18252 root 1 36 0 12820Ki 2488Ki pipewr 30 0:03 16.96% dd if=/dev/random bs=128 ! 18177 root 1 115 0 13364Ki 3048Ki RUN 13 15:45 49.26% sh -c while true; do :; done ! ! It appears that the CPU time (or more) for the dd feeding the ! first pipe matters for the overall result, not just the ! bs= value used. This is all fine, but it is of no relevance to me. I'm not an entomologist; I don't want to explore bugs, I just want to kill them. From nobody Sun Mar 26 03:42:02 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pkhfl6X3fz419gr for ; Sun, 26 Mar 2023 03:45:19 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pkhfk4cLSz3NNw for ; Sun, 26 Mar 2023 03:45:18 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of pmc@citylink.dinoex.sub.org designates 2a0b:f840::12 as permitted sender) smtp.mailfrom=pmc@citylink.dinoex.sub.org; dmarc=none Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.17.1/8.17.1) with ESMTPS id 32Q3j6RI085398 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Sun, 26 Mar 2023 05:45:07 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) X-MDaemon-Deliver-To: Received: (from uucp@localhost) by uucp.dinoex.org (8.17.1/8.17.1/Submit) with UUCP id 32Q3j6KD085396 for freebsd-hackers@freebsd.org; Sun, 26 Mar 2023 05:45:06 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (disp-e.intra.daemon.contact [IPv6:fd00:0:0:0:0:0:0:112]) by admn.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32Q3hdTf064576 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK) for ; Sun, 26 Mar 2023 05:43:39 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (localhost [127.0.0.1]) by disp.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32Q3g2ec016814 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Sun, 26 Mar 2023 05:42:02 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from pmc@localhost) by disp.intra.daemon.contact (8.17.1/8.17.1/Submit) id 32Q3g2vN016813 for freebsd-hackers@freebsd.org; Sun, 26 Mar 2023 05:42:02 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) X-Authentication-Warning: disp.intra.daemon.contact: pmc set sender to pmc@citylink.dinoex.sub.org using -f Date: Sun, 26 Mar 2023 05:42:02 +0200 From: Peter To: freebsd-hackers@freebsd.org Subject: Re: Periodic rant about SCHED_ULE Message-ID: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Sun, 26 Mar 2023 05:45:09 +0200 (CEST) X-Spamd-Result: default: False [-2.30 / 15.00]; FAKE_REPLY(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[sub.org]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; TO_DN_NONE(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; HAS_XAW(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; ARC_NA(0.00)[]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4Pkhfk4cLSz3NNw X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N Quoting George Mitchell : On 3/25/23 11:47, Peter wrote: >> You're welcome. Can I get a success/failure report? >> >> >> [...] >Thanks for your help. Regretfully, I have to report that the patch >did not really help in my case (make buildworld while running dnetc). >But I have belatedly realized that it's a really good idea to use >make -j12 to build kernels and the world. With 4BSD and dnetc >running, I can now buildworld in 3512 seconds (down from 20477), and >with ULE in 14193 seconds (down from 50290). Peter's patch got it >down to 13755 seconds. Thank You for testing. So at least it doesn't make things worse. Back then 5 years ago when I made that, I think I've seen a few more strange things in that code, but they didn't bother me immanently at that point. Probably there is more work to do there... From nobody Sun Mar 26 05:01:59 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PkkMG6NTtz41FpV; Sun, 26 Mar 2023 05:02:02 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PkkMG3dcwz3kZ5; Sun, 26 Mar 2023 05:02:02 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679806922; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=L5jVnGuhq1+5mPJJ/6DAYTT59BgVSJ94MbvNU/w5ff0=; b=YKl9e1B2IQUY4cjPcsaXjk0+W+EzbD0MWC3N0ey+c0J3h583BIrKIHk4mhm4ZjCG89HejH NuvIgWBK5JpYhFm9VT4oMA+7Y7AsGctjndAuM+0XfdrtXdgzIv1z3Nt/OVcL/kbFGRKRlr PVezlToLrpa0OXt4EnEG4eXIEphkMS8fYyLzKqtzqY+PlI2e5YE+IBnrJSt0CQKPxxjt5m rvBldA3mw+nrdpz6Iltpr1qOasH/cYn6rdTSRxvDTmkVjcl33WhfU6cEQHUKyI3TxJK1qC DPMUyTPRtiVT8PraDq8oXVmHXooW2rwNuFXTlw+MjaruUT5krfU/VX9bukw7hw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679806922; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=L5jVnGuhq1+5mPJJ/6DAYTT59BgVSJ94MbvNU/w5ff0=; b=DURUdBqlrIKJVZDxFaJTAkpsN7ENVC56y7/YMi2skfT/3AaFuVhGMMy8nKRJ6C2aIc/G+I tD408l0lpQE4liLHihjLoaEVsS2ErMUbGg6XF+EZfMDu+vXaZjHb2KMhp6bzbntft6An9g q/UtoabY1BC2QoCw2EVcL7djg14RKBQtOHBWF0tjMNZ3o1mAXEdGtKNCY2guR8CfaLnPyk NO3Va5ZN94aj6DFfnbGMQ/v4FB/o+bpUru+UOu8HuCXnlJceqAC/+2J1BRTR8eNV2TURsV ffHbno01zxoGkYYKWu7Yu8ueVZqwepsn11sqITm0stXytvZ9WY3Va4DU6pGpTA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679806922; a=rsa-sha256; cv=none; b=NxHryQsoTR7tQ2eMGI8lu3lLfwz9PUrOS5rbjxp6vrbSd+nqOGh3vYwf44x6nYMGZyjohB J70hqmT6VL9vb+v7DfvPDhvcCLg+6FYa5f+jOHVeXZMYbFkDA988W/ScMcvs+jNCZI5SDM OxBGlih3JjxX81BczJyPzFexxH2524bRqbjms+871nMXIlLMymQIMqjM44g+MZ1I67s3Lv kPHI0KLgH6b5pDywWvYarqQKOv2qyynOUJ29fmO/TMixtqA+FInc7CrPquuIcejZYXYzHX rTz5wP2XO552bYUuztNDt/aLmIOGSV3Z1Ml0PIfxH6SmNvlMT46lCEW9sC8e9g== Received: from [IPV6:2001:470:1f1c:a0::2] (tunnel642390-pt.tunnel.tserv1.lon2.ipv6.he.net [IPv6:2001:470:1f1c:a0::2]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: grahamperrin) by smtp.freebsd.org (Postfix) with ESMTPSA id 4PkkMG11hnzndy; Sun, 26 Mar 2023 05:02:02 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) Message-ID: <793bf7cf-3076-f7d2-70f5-8b6ca7ed0c0c@freebsd.org> Date: Sun, 26 Mar 2023 06:01:59 +0100 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 To: freebsd-ports@freebsd.org, FreeBSD hackers Content-Language: en-US From: Graham Perrin Subject: explain_timerfd_create_or_die(3) Organization: FreeBSD Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------c2K5fTi2I5EXyj2090n6EepN" X-ThisMailContainsUnwantedMimeParts: N This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------c2K5fTi2I5EXyj2090n6EepN Content-Type: multipart/mixed; boundary="------------7ObbTG1SKtdQkMELw02j1GHH"; protected-headers="v1" From: Graham Perrin To: freebsd-ports@freebsd.org, FreeBSD hackers Message-ID: <793bf7cf-3076-f7d2-70f5-8b6ca7ed0c0c@freebsd.org> Subject: explain_timerfd_create_or_die(3) --------------7ObbTG1SKtdQkMELw02j1GHH Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 V2hpY2ggcG9ydCBpcyB0aGUgb3JpZ2luIG9mIHRoaXMgbWFudWFsIHBhZ2U/DQoNCjxodHRw czovL21hbi5mcmVlYnNkLm9yZy9jZ2kvbWFuLmNnaT9xdWVyeT1leHBsYWluX3RpbWVyZmRf Y3JlYXRlX29yX2RpZSZzZWt0aW9uPTMmbWFucGF0aD1mcmVlYnNkLXBvcnRzPg0KDQo= --------------7ObbTG1SKtdQkMELw02j1GHH-- --------------c2K5fTi2I5EXyj2090n6EepN Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsF5BAABCAAjFiEEWT/lssMHB+28ly8Kt2dIb0oY1AsFAmQf0ccFAwAAAAAACgkQt2dIb0oY1AtD zA/7BaLeKFBU3Lfwh36LNVzAJ+wbx2ZE7F85keUkkXlyS3o8Cd6p/9kQbZsDvOPpzpeJHFz68ZXd CMesLIU0J5e3flDpDNhZtWE0NnIprnxusRBRZP2QphX++QzyukKGI5I8Wfe8QosKyWegiCnTbNks cy6W74ZwU5ysoII8X08CxKt6YBesFYvaq9+LG0F/XkST4el+9pLpVxLuKRlpiV07FrE+6gcyAzsx DMq6HN9e3rzsTUN7U8efMekT3/ScGpeEI8jlxBmXJ2WrXlnFMdW1asGk2qpQEZNeBfOiB0AtZlM2 lUeSzNINX95zu/nhrhhxlfLFn582mdTgQNoObS8oVXhF/IvnsDTPEGFjiDXj/p5cHDdjsoVpKwg/ 0JMq7lRo3LJc2TSWCL3M6UkXZoH1mNPRdOUa/bNKvntS+lZCT1Wyto+lbO1REry4FPeFFEo6ZDOa VTu/GqgD/LvWVTOE+C5q9sKnJ5B5FXLvgCA3/otPf6pR+iqBYR7hCuhs3IO7L7C+1gKjY2kuvwqP RxN17V0Sn63gourxseZRI27TOL9mrNF3anhB8R/QbA8Nnfv8Gh8TSR+/jJUI1KOo0VZUN+O4XK4f UN8VSGR4rwm/2M8RlGQnTlFOa3IdTuRlq7h9cwbEySnDH0hRwKIIx7FcBsVbg8W2nUVU16rVi0sF gKs= =FpL1 -----END PGP SIGNATURE----- --------------c2K5fTi2I5EXyj2090n6EepN-- From nobody Sun Mar 26 05:24:00 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pkks56PPwz41HBL; Sun, 26 Mar 2023 05:24:25 +0000 (UTC) (envelope-from agh@riseup.net) Received: from mx1.riseup.net (mx1.riseup.net [198.252.153.129]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mx1.riseup.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pkks54FtJz3mw3; Sun, 26 Mar 2023 05:24:25 +0000 (UTC) (envelope-from agh@riseup.net) Authentication-Results: mx1.freebsd.org; none Received: from fews2.riseup.net (fews2-pn.riseup.net [10.0.1.84]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.riseup.net", Issuer "R3" (not verified)) by mx1.riseup.net (Postfix) with ESMTPS id 4Pkkrs03whzDqY8; Sun, 26 Mar 2023 05:24:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=riseup.net; s=squak; t=1679808264; bh=p8x+bAXFYwUIgDkWb3IRSfpkkFjIxq+mJlfAcURzvFo=; h=Date:From:To:Subject:In-Reply-To:References:From; b=LQ03kljVcUaW5F5gYHuhZN4UDzryH6CyrNIJRZ01y4CpLFMSqQQGNGRvexitp5Drs HfvDR0gD02W/ayJX0rAV5VUClAEXnlFvUpI9gtBqfH2yOqtW7H3Pm8b/nPLQSmuKwx mO0zShBM+D5IadB9G+iv0JxadQBP6ppR5YstuFVg= X-Riseup-User-ID: 971BB295A7F580A86F30B1C524DD7FB56040CD29BD88A36A8710DFC13507832E Received: from [127.0.0.1] (localhost [127.0.0.1]) by fews2.riseup.net (Postfix) with ESMTPSA id 4Pkkrr043Gz1y7V; Sun, 26 Mar 2023 05:24:11 +0000 (UTC) Date: Sun, 26 Mar 2023 13:24:00 +0800 From: Alastair Hogge To: freebsd-hackers@freebsd.org, Graham Perrin , freebsd-ports@freebsd.org, FreeBSD hackers Subject: Re: explain_timerfd_create_or_die(3) In-Reply-To: <793bf7cf-3076-f7d2-70f5-8b6ca7ed0c0c@freebsd.org> References: <793bf7cf-3076-f7d2-70f5-8b6ca7ed0c0c@freebsd.org> Message-ID: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary=----RAJ9O6SALHC292D3XI8DBYJR4PPS1A Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4Pkks54FtJz3mw3 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:16652, ipnet:198.252.153.0/24, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N ------RAJ9O6SALHC292D3XI8DBYJR4PPS1A Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Is that https://www=2Efreshports=2Eorg/devel/libexplain/ On 26 March 2023 1:01:59 pm AWST, Graham Perrin wrote: >Which port is the origin of this manual page? > > > --=20 Sent from a device with a tiny bloody screen and no hard keyboard; please = excuse my brevity=2E ------RAJ9O6SALHC292D3XI8DBYJR4PPS1A Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable

= On 26 March 2023 1:01:59 pm AWST, Graham Perrin <grahamperrin@freebsd=2E= org> wrote:
--
Sent from a device with a tiny bloody screen and no hard keyboar= d; please excuse my brevity=2E
------RAJ9O6SALHC292D3XI8DBYJR4PPS1A-- From nobody Sun Mar 26 05:39:25 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PklBQ6Sndz41HmW; Sun, 26 Mar 2023 05:39:26 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PklBQ5lGDz3pXF; Sun, 26 Mar 2023 05:39:26 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679809166; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=4Ie3xOXkOPDM4e44NUD6GAJL0ZHck2av2Nkr7Q2PBx4=; b=rxFCJbaJc3nxTowQzRxQnJ5EWLKFCOTlHiERr0QWc8bEVuxX2kdproLMXfT78v82FO2NXr tqfpQt6GYEkUgs7i0AmDUAEj8Ru59PpTU08sAHODCPlrDrPyLjhKqWjT80eK5c2OmYq9ni uvfiP4H3nHWLa8mIwnk4+RuWxTTu1VaKPUL7ntOIcYKijjpRxnZ/eTVyhhjpAyvEj76bys lX2JhBn4TE415VAqjHKXbdwP0pbSFihVlPUvQUEmOWALzlBz7KmJv+nS/BaPlbLTizucGk EDqVC3JhRnssfdjHDXI/gztuOSa5lvj+FZDlj4bBlyeNoZICpcAKriMEYsPMLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679809166; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=4Ie3xOXkOPDM4e44NUD6GAJL0ZHck2av2Nkr7Q2PBx4=; b=pEW8GGWwjx9X80ulKtTC2POTAvEUT48NJEy1wWV9sAZiM9lrJrRFfVc0l1G5PijjuMilJc nYEC6kQEtZs29ahc5kpn57GlLde65G3AW4YM1Dr3vuTqWmGQb62Ad3prvPhaC1Gog6MM7B hBesc8Q/1SdTSk2973JTjCOH12H28xs/VqEwmUUEwdqSt7WKYL/KFTwwvEvpj3sSQgN4zz p9QQ3thmfgtX+DVGJRyE8+wISykl9KdAOI3gzoIvJOYeST/wycRoWHUIU85GCDVKEnHXWf QG1PxzXL91Fn2u/SHqusdurYQ2XYpjLYIH2TROMq3wHar5Gr+ZAhBECeRaEFag== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679809166; a=rsa-sha256; cv=none; b=sgnw/mKg/tNSE3QV8K1HPtTQdVAg0DmmF20C4Uxqu/DJ+h0888xj3L2noIpaG6h0GVVtU5 et9o/1ZwC6dkwKGr1x4MbmqAdFwYHQxD27B1AN1Q1C5FNjLsaZilelSXuRzId0geCHGDW6 bhK0KOb6cHXPe47o54Zq/z0wKGc8IrJbYZ9SwPktN14CxCuMdh3bey2TRCT8+5Rl/FQe+x U96uKZV+jUno0A17ZTpGdkLrwfEYEkASIO30H4sHYVknDeXS4u8mZkUjtkzE0zQBreHnlv Kpal4HJtE1/A+TrlYdZX1zfR5VmFRLQWxcb7eopqZODvzH9Nbc1Vkm9lcKj39A== Received: from [IPV6:2001:470:1f1c:a0::2] (tunnel642390-pt.tunnel.tserv1.lon2.ipv6.he.net [IPv6:2001:470:1f1c:a0::2]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) (Authenticated sender: grahamperrin) by smtp.freebsd.org (Postfix) with ESMTPSA id 4PklBQ2kLFzpHV; Sun, 26 Mar 2023 05:39:26 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) Message-ID: <77ea7eca-d702-93df-aafd-3c20344a2002@freebsd.org> Date: Sun, 26 Mar 2023 06:39:25 +0100 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 Subject: Re: explain_timerfd_create_or_die(3) Content-Language: en-US To: freebsd-hackers@freebsd.org, freebsd-ports@freebsd.org References: <793bf7cf-3076-f7d2-70f5-8b6ca7ed0c0c@freebsd.org> From: Graham Perrin Organization: FreeBSD In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------IcvEpkVZLpQ4qfd7Pmag2L0K" X-ThisMailContainsUnwantedMimeParts: N This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------IcvEpkVZLpQ4qfd7Pmag2L0K Content-Type: multipart/mixed; boundary="------------iaAMQpjGM0QB3NYwyq7SVOoS"; protected-headers="v1" From: Graham Perrin To: freebsd-hackers@freebsd.org, freebsd-ports@freebsd.org Message-ID: <77ea7eca-d702-93df-aafd-3c20344a2002@freebsd.org> Subject: Re: explain_timerfd_create_or_die(3) References: <793bf7cf-3076-f7d2-70f5-8b6ca7ed0c0c@freebsd.org> In-Reply-To: --------------iaAMQpjGM0QB3NYwyq7SVOoS Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 T24gMjYvMDMvMjAyMyAwNjoyNCwgQWxhc3RhaXIgSG9nZ2Ugd3JvdGU6DQoNCj4gSXMgdGhh dCBodHRwczovL3d3dy5mcmVzaHBvcnRzLm9yZy9kZXZlbC9saWJleHBsYWluLw0KDQoNClRo YW5rIHlvdSENCg0KSSBmb3Jnb3QgdG8gbWVudGlvbiwgSSByYW4gdGhpcyBiZWZvcmUgbXkg Zmlyc3QgZW1haWw6DQoNCg0KJSBscyAtaGxuIC91c3IvbG9jYWwvbWFuL21hbjMvZXhwbGFp bl90aW1lcmZkX2NyZWF0ZV9vcl9kaWUuMy5neg0KLXJ3LXItLXItLSDCoDEgMCDCoDAgwqDC oDEuMksgMTggTWF5IMKgMjAyMiANCi91c3IvbG9jYWwvbWFuL21hbjMvZXhwbGFpbl90aW1l cmZkX2NyZWF0ZV9vcl9kaWUuMy5neg0KJSBwa2cgcHJvdmlkZXMgZXhwbGFpbl90aW1lcmZk X2NyZWF0ZV9vcl9kaWUNCiUNCg0KDQpEaWQgcGtnLXByb3ZpZGVzKDgpIG5vdCBmaW5kIGl0 IGJlY2F1c2UgdGhlcmUncyBjdXJyZW50bHkgbm8gcGFja2FnZSBmb3IgDQpGcmVlQlNEOjE0 OmFtZDY0Pw0KDQoNCihJIGZvcmdvdCB0byB1c2UgcGtnLXdoaWNoKDgpLikNCg0KDQo+DQo+ IE9uIDI2IE1hcmNoIDIwMjMgMTowMTo1OSBwbSBBV1NULCBHcmFoYW0gUGVycmluIA0KPiA8 Z3JhaGFtcGVycmluQGZyZWVic2Qub3JnPiB3cm90ZToNCj4NCj4gICAgIFdoaWNoIHBvcnQg aXMgdGhlIG9yaWdpbiBvZiB0aGlzIG1hbnVhbCBwYWdlPw0KPiAgICAgPGh0dHBzOi8vbWFu LmZyZWVic2Qub3JnL2NnaS9tYW4uY2dpP3F1ZXJ5PWV4cGxhaW5fdGltZXJmZF9jcmVhdGVf b3JfZGllJnNla3Rpb249MyZtYW5wYXRoPWZyZWVic2QtcG9ydHMNCj4gICAgIDxodHRwczov L21hbi5mcmVlYnNkLm9yZy9jZ2kvbWFuLmNnaT9xdWVyeT1leHBsYWluX3RpbWVyZmRfY3Jl YXRlX29yX2RpZSZzZWt0aW9uPTMmbWFucGF0aD1mcmVlYnNkLXBvcnRzPj4NCj4NCj4NCg== --------------iaAMQpjGM0QB3NYwyq7SVOoS-- --------------IcvEpkVZLpQ4qfd7Pmag2L0K Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsF5BAABCAAjFiEEWT/lssMHB+28ly8Kt2dIb0oY1AsFAmQf2o0FAwAAAAAACgkQt2dIb0oY1Avc ww//al9LhE26MgM7zbwOlvlfbNFBk0qEYPUmOnzr+Ea7awW1EvR8Cbhl8mqFY9AvgXwO+c8jHWs0 ROzZPuyEWRovaTgf/KsZ7iqf2un0dyrgIfPx8r9yxN52EnYnyC3FJ1vbIG4GcLuQEnkzE3URiQ9L oqLIWs/jP/Xu5QiTiRur7Hut/CypbkpEsgLcBldRvUhYqjhwM6tUF6dQuEy7XCt/5K6VSly5bRXm UNXPuf3fDyoP2C1In/8LlUkCv3P77OEyXUIeNs1KngQwQneIPCOfzTWdIi84x7oYgQlCFbv7oJtu pzQMczbirZTkHSEXPw8XtO57jAgDood+FFfl7nPtmR4mKAvTxuCRto+QNzvn21109bdiyM/srMjM N7jCOTN/m/3Jn74Nstqx7FzogrY97y7j04jDWtiDJzbdAOxaMl3hfMmoOZfnED+Sk/byGYq0TcMs PIWCN/mlgWh6pXGT1JyPObHvQ6xRMNQhuG9ZnSzBRjdiT71Zq6UeIhz13l9jJefkqenUpUljRIV0 QfvizjRlhfh0ynH0W+Ht+U4Sz48g8C2/bOHW3572mqm20E9juqHviRgR8j+mLB0UmNjVkP9FS2rY d9/mmFnJkF3O51ZabrMBe0YXskqDS5chhFebkaj8j4t0lc+NNv0ZFaRESAMSaYLo7zp4hKwlklNi poA= =dZnR -----END PGP SIGNATURE----- --------------IcvEpkVZLpQ4qfd7Pmag2L0K-- From nobody Sun Mar 26 08:59:22 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PkqdB68xyz41X5b for ; Sun, 26 Mar 2023 08:59:26 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PkqdB5thfz493t for ; Sun, 26 Mar 2023 08:59:26 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679821166; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=yTz/agW53osZAb76j6kZDFiWUA6dUgjkXCVi8eMLyoo=; b=XDk78/vjEvEZKGSf0JpzVkzA19PF+61XpuCE/ST60/AiQsTcn0e5M/a4PbGVC4V2q0URCV s9SektZ58YNTdiS8GuOAq0Lj4Bl7ODxvvY3mzhdPxF9P6uUn9u+l7qXLAVVeKRQOoTiy8N rAwLSg0poLuHFgioOCbd763jD0ZrboGGMSjEEqzzCfFVrD6o+cyrrOHfV8beQoRsf1RsYu Vm04xX749oxi3+Awe+k+5/38bWyhx8n7IhunKh/bHYnQJFmyfkRanCbXK6R38E4j2C/e27 OT1I9TmYYeiO/ibJtSXhRRHjH6/ARFCAOtm0DyzD94IUTPsowAX18uMthbsi/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679821166; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=yTz/agW53osZAb76j6kZDFiWUA6dUgjkXCVi8eMLyoo=; b=moQY+D0LzFRFhnniQbbQtAv8c0RC/Jtw1Jr9d3KvI6XWT5u+1CJRFBmiTyFBZ8y94S91P7 1bgtfcx/gdwZXcZSSujXQVYGQtDiUlxgujVnzmpp1YfYfr8NONrBqguIdWUgD9+w5dF+dL ZSQSrwJASGem9Tn8JZon6ZJZtnAdU8CrGOHI0Azq549AmUGoYlvIigd4S5M2Myappt+bsJ L37IvMDsd1BXkLN9WqWjeEfILoVuIj/F73BWiuut4et8tmHiWRepcmRfCx8PgPfpHXGBCT qDPvclXXiaT5OkYYOrOeCVucRmlbM5NOcJyb9vJj0D+zotxAr1TNNbZuS5to/Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679821166; a=rsa-sha256; cv=none; b=Ia2/PkHdggaWUoPk0uxFZU/+/gonWVUAdmDZfOyQEJs6AoHGL/5VL6PH+tgWvudTlbK9W2 CRLYhJpuKrJ+5KdB3BHu8FdRNwjqhYD/uAL71bXSfTfqz3o+vcQr/d9NkSvzrNapJsUoLi IrBBgoiDzHPU4OeqGR6DdYMwdTJI4ZtS3cjGjhJbCMazQCYY2jXa1BDHF8iwey+Ky+gbp/ GvQMjBy03dlBx7zWcvqRUZ4SCdrv3RDq9NTSuFd0y6AdA8Fy2czH5aOEz0XrWzq6XfBR+r 1Pstn6uI5DHyAfBFf0SCVFV2H0XYd/798FXzvOabp3zjE5wCXXfq6uaPVzDKgQ== Received: from [IPV6:2001:470:1f1c:a0::2] (tunnel642390-pt.tunnel.tserv1.lon2.ipv6.he.net [IPv6:2001:470:1f1c:a0::2]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: grahamperrin) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Pkqd95MBhztZ2 for ; Sun, 26 Mar 2023 08:59:25 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) Message-ID: <4d694678-4421-7c34-7907-657b6f524502@freebsd.org> Date: Sun, 26 Mar 2023 09:59:22 +0100 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 Content-Language: en-US To: FreeBSD hackers From: Graham Perrin Subject: User-Agent: and In-Reply-To: Organization: FreeBSD Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------04FClYkbcFIyYTJjfibmXM2v" X-ThisMailContainsUnwantedMimeParts: N This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------04FClYkbcFIyYTJjfibmXM2v Content-Type: multipart/mixed; boundary="------------lXHHD9Fjd0rtSvEw84v6ov7s"; protected-headers="v1" From: Graham Perrin To: FreeBSD hackers Message-ID: <4d694678-4421-7c34-7907-657b6f524502@freebsd.org> Subject: User-Agent: and In-Reply-To: --------------lXHHD9Fjd0rtSvEw84v6ov7s Content-Type: multipart/mixed; boundary="------------uTJw8bsCB2aEs2Uc9pYw0k17" --------------uTJw8bsCB2aEs2Uc9pYw0k17 Content-Type: multipart/alternative; boundary="------------y9Z9x39mAgkCj9DtP3KhqE7w" --------------y9Z9x39mAgkCj9DtP3KhqE7w Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 Tm90IHRvIHN0YXJ0IGEgbG9uZyBkZWJhdGUgKG9yIHJhbnQpLCBJJ20gY3VyaW91cy4NCg0K TWFyaywgUGV0ZXIsIHBsZWFzZTogd2hhdCBkbyB5b3UgdXNlIHRvIHNlbmQgZW1haWw/DQoN CkEgcmVjZW50IGRpc2N1c3Npb24gaXMgc3BsaXQgYWNyb3NzIHR3ZWx2ZSB0aHJlYWRzOyBz Y3JlZW5zaG90IGF0dGFjaGVkLg0KDQpJIGxvb2tlZCBhdCBoZWFkZXJzIGZvciBhIGhhbmRm dWwgb2YgdGhlIHNwbGl0cy4gSWYgSSdtIG5vdCBtaXN0YWtlbjogDQpuZWl0aGVyIGEgKlVz ZXItQWdlbnQqOiBmaWVsZCwgbm9yIGFuICpJbi1SZXBseS1Ubyo6IGZpZWxkLg0KDQo= --------------y9Z9x39mAgkCj9DtP3KhqE7w Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

Not to start a long debate (or rant), I'm curious.

Mark, Peter, please: what do you use to send email?

A recent discussion is split across twelve threads; screenshot attached.=C2=A0

I looked at headers for a handful of the splits. If I'm not mistaken: neither a User-Agent: field, nor an In-Reply-To= : field.

--------------y9Z9x39mAgkCj9DtP3KhqE7w-- --------------uTJw8bsCB2aEs2Uc9pYw0k17 Content-Type: image/png; name="2023-03-26 09.52.png" Content-Disposition: attachment; filename="2023-03-26 09.52.png" Content-Transfer-Encoding: base64 iVBORw0KGgoAAAANSUhEUgAAAoEAAAC/CAIAAAAO3pEbAAAACXBIWXMAAA7EAAAOxAGVKw4b AAAgAElEQVR4XuydeVwTR//4JyGBEMJ9CHLI5cEpShRQFEQF71q1Cl611PtAsNL6fTz7iLVS rRRB61X9ASpFrVABERVBPABFkBtFQEAMQhBCAoEk5PfH6Bo3gYQAKj7zfvnyFXZnZnPs7Gfn 2HkThELhgQMHfvrpJ4BAIBAIBOIjQsRvQCAQCAQC8VGQHIOzs7PDw8Ozs7PxO3oOj8e7d+/e vXv3eDwefh+iT2Gz2fhN0ujs7ORwOPitCARCEnJUsU9OaWnp/fv3X758id+B+DyQHIMzMjLc 3NwyMjLwO3pOeXm5ioqKiopKeXk5fh+i78jMzPTw8AgODsbv6JY1a9Z4enoyGAz8DgQC8SHy VbFPjra2NpfLra6ufvjw4UdoCO3YsWPVqlVyHEjujAMdyTHY2dk5LS3N2dkZv6OHcLncly9f 6unp6enpvXz5ksvl4lN8oeTm5tLpdDqdPnbs2OnTpwcFBbFYLHyiLjh9+jTM6+rq6u3tHRsb i08hCTU1NS0trUGDBuF3iJGVlUWn03fu3AkAMDQ01NXVpVAo+ES9Jjc398SJE2VlZfgdYqxY sYJOp7969Qq/QzaSkpLCwsJwp1ZHR8fBgwenTZvm5ubm6+t79+5duJ3BYAQGBk6ZMmXatGm7 du1qaGhYv349nU7Pzc0F774Zf39/AADcjpGeno79pq6urt98801kZGRHR4foQSFMJpNOp8+e PRv+GRwcTKfTY2NjYfb169djKbECIUuXLsV2IbrH19eXTqfPmzcP/unu7k6n07du3fphKsk8 ePCATqfv2bMHvwMAAEB4eDj8ORoaGgAAW7ZsodPp7u7uQKSKvX79mk6nr1ixAgBw7do1Op0e Fhb2QSkysHLlSjqdXlNTg9/R1+jo6PD5fAsLCxUVldzcXDmCHI/Hy83NjY2NLSgo6GoLpLOz 8969exwOh0wmwy3l5eU3b94UbYA9e/bs+vXrjx496uzsxDaKZpRYuEAgePLkSUpKSn19Pbax rKzs33//vXv3Lp/Ph1skFo6DzWaHh4fv3LmzsLAQbqmpqUlPT5d4saqrq0tKSrp58yZ2Ae/s 7Lx06dLOnTvj4+M/TCsnJPwGAAAAjo6O2dnZjo6O+B095MWLF1paWvD30NLSevHixfDhw/GJ PhEBAQHe3t5OTk74HX2HqanpggULrl+/Hhsby+Pxfv75Z3yKrpk4caKFhcWVK1eCgoK0tbUn TJiAT/EhVlZWSUlJ+K3S6OpK1HuSk5NjYmKGDRtmaWmJ39ennDlz5vnz5/CCiHH8+PHo6Ggb GxsXF5eMjIyLFy+OHz+ew+EsX768sbFx7ty5FAolMzOTQCCI5hJn6dKlxsbGAAA7O7vKykoA gImJyZw5c27duvXHH3/U1dXJeN3vBlNTUx8fHwCAgYEBft9nTxW38seSjXktOQAAe9VRwSPC TCim+ET9AIvFIhAIVVVVbDa7qamJzWYTCITm5mZ8up7T0tJCIBCEQmFRUdHEiRNLS0sJBAKb zRYIBFgVe/36NT7bZ8nLly8NDQ2xP3V1ddva2srLy3t0EW5oaPD39y8pKYF/Llq06LvvvsNt CQwMhK9LSkpaWlq++uor+OeJEydOnDgBX69cuXLVqlVbt25NT0+HW2xtbU+cOKGoqCiaUfxw gYGBPB4vICAA9ssqKSkdOHDA2dn5v//9b2JiIkxmbW39559/bt++XWLhoiQmJoaEhDQ2NgIA 3N3dbWxsQkJCoqKi4F5PT899+/Zhl4WYmJiDBw/CcK6urv7XX3+xWKwDBw7At6eqqjpr1qy3 5fYCye3g3tDQ0JCTk/P48ePU1NTq6mo9PT24XU9Pr7q6OjU19fHjxzk5OfA28xPi5+cXGhqa kpKC39F36OnpeXt7w8t0fn4+AKC6unrdunUTJ05ctGgR1jiTOMhkY2OzYcOGhQsXAgDu3bvX 2tr6888/T5kyZdasWX/99ZdQKOzo6KDT6du2bTt+/PiUKVPu3buH3d3X1NT4+fm5u7vPnTs3 MjJSKBQCAAoLC729vd3d3c+cOYMdRbQNevny5Xnz5rm5uW3cuLG2tlY0zdSpU9PS0qZNm3bv 3r2ffvppzpw5EyZMWLduHezEXrFihbu7e0xMzPTp02fNmlVWVnb69OmYmBgAwNatW7ds2YIV lZubu3HjxilTpkyZMuXnn3/G7l4BAFFRUVOmTPnmm2/gzWlra+svv/zi5eXl6ekZFBQEB63n zp1Lp9M7Ojpu3bpFp9ODg4PXr1///PlzAIC7u/u5c+ew0uC3vWjRorVr1545cyY4OJhAIJw/ f76xsXHBggU7duzYunVrTEyMtrY2lkUi7u7u8+fPnz9/voaGBtyir6+/YsWK48eP02i06Oho uZvvGHp6evAQ48aNw+/7vKniVno9HMfv5H816JuvBn3D7+SPf2BXxa3Ep+sHWCyWmZkZAKCk pKS0tJREIhkZGbW0tABJ5xiupoi2kLZu3Uqn03NycrAtzc3Nqqqq2traRUVFTU1NdXV15ubm AAA2m919AxoAcOXKFV9f3wkTJsycOTM6Ohq8a3NfuHAhICBg8+bNfD7/4MGDHh4eS5Ys6e8B oGfPnhUWFjY0NDQ0NGAdXQYGBj3tj6RQKGZmZn/99dfly5cpFMrff//N4/FwW+A3DwDIysoC AIwZMwYAUFNTc/LkSQcHh5SUlNGjR58+fbq2tnbq1KlhYWE3b950dHQsKCjAWrpYRvHDtbS0 XL9+PSMjY9myZfHx8UpKSrA66+vr7969+/r16zY2NkVFRQ8fPuyqcIzOzs7z58+rq6u7urrC Lfn5+VFRUTY2NklJSVOmTLlx40ZaWhp4d03W0dFZvHjxjRs3FixY0NzcfPny5eTk5NraWuwm o0/oLgaLhknZZ2kVFBRQqVQ1NbVhw4aNHDkS65Qgk8kjR44cNmyYmpoalUoV/4I+MmZmZkFB QdHR0X3VpSCRjo6Oe/fuAQCGDh0qEAg2b95cWlrq7+9PJpMDAwNra2ujoqImTZp0+/ZtfE4A AABUKhUAoKCgcPDgwatXry5evNjBweHo0aPJyckwQXZ29qlTp8zNzYnEtz8ln8/39/fPzMz0 9vbW0tL6448/kpKSeDxeYGBgeXm5j48PDMk4bt++vX//fgUFhXnz5rW3t6urq4vuffPmzb59 +2BTQCgUzpkzZ9y4cQ8fPvzzzz9hAjabnZCQQKfTGQxGbGzs5MmTXVxcAADLly9fvXo1Vo6G hoaKisqyZcv09fWvXr0q2nB/9uzZrFmzKioqYG/BoUOH/vnnH2dnZxcXl9jY2MOHD2MpRVm1 ahXsft+zZ8+UKVOw7dbW1gCAX375JSgoqLi4WElJCQAAT7nx48fDNKKN4EuXLoWFhYl3+9+7 dy8+Ph67ucagUqlWVlYAgKKiItyunlJfXx8fHx8fH19XV4ff93kTWLzBRsV+jIazBknDf8i2 74zXOqmPDyzegE/XD7BYLGNjYxqNVlRUVFJSYmlpSaFQYG9hV+eYeE25e/duamrq3LlzR40a JVqyUCi0tbUtLi6GzR34Q2NhphuoVKq5ufn3338Ph0KwaVBnzpzJzMy0tLS8cuVKdHT0kCFD 3N3d37x582HuvqSpqam2ttbIyKiqqqq8vBwboiKTyerq6j0602g02t69e+3t7YcMGQLvRNXU 1HBbsK80KyuLRCLB7zMzM1MoFLq6uqqpqbm6ugqFwoyMjBkzZjg7OxMIBB6Pp6SkNGzYMFxG 8cMRicT79+8DADw8PPT19e3s7Gpra1+8eLF+/frZs2dra2ubmpoCABQUFLoqHINIJP7222/n z5/HOudKS0sBAM7Ozjo6Oi4uLkKhsKCgALsme3h4+Pv7a2pqwusJkUhcu3ZtTEyM6KWm90ju i4bEx8djI148Hs/NzS0tLU1qB7W6unpTUxO8eRQHngTl5eW4q/wnwczM7PDhwzt27KioqNi0 aRN+d6/JysqCjRtLS8sNGzbk5+dXVVVNmjTJxcWFxWKVlJTcv39fR0dHW1sba2ZhFBYWHjt2 7PLlywAAFxeXrVu36ujoTJ8+vba2NikpKTU1ddKkSQCAN2/eHD58eMKECQ8ePIAZCwoKKisr x48fv3bt2nHjxvn6+l67dg0OYjk7O69ZsyYrK+vRo0eixwIAXLlyBQDw008/0el03C7IjBkz Nm/eTCAQ4JyUmpqaW7duwe5ZyM8//8zj8a5du/bixQtTU1NjY+MHDx7Y29uPGDECS2Nqanrg wAEAgIaGxt69e0Wz79q1y8jIKCUlpby8/PXr14mJiSoqKnDQ+tatW0lJSdu3b8cSY8BKW1dX 5+7uTqPRsO1r167lcrmx71i6dKm/vz+89Eg88brqxod9BlZWVuJjAbCctrY23PaeUlFRAZtW ISEhsgznfz7cb7qz0vhtxE2sj/3OaO3dN6mxdTEfpup7Wltb+Xw+jJRFRUWtra1WVlZFRUUw Bnd1juFqCpfLDQ4O1tDQwFX85ubmzs5OW1vbc+fOlZSUmJiYqKmpwe2iySTi5eXl5eXV2dlZ WVkZHx9fVVUFt3O53Li4OF1d3bVr1wIAtmzZYmdnl5mZCWch9Ad1dXU6Ojq6urrNzc3t7e1a WlrYLnV1dSaTOWTIEJHkMnHz5k0GgzFlyhQVFRWJWzo6Op48eWJrawtbDnDgVlVVFQAAv0NY AXNzc1euXEmhUEJDQ2GdxWUULxx2/sOi4P+vX7+GIebVq1e3bt3S09MbO3asxMJx4EZ89PX1 AQCFhYVMJvPx48cAAAUFBdw1mc/n//333wQCYdasWXB+8bNnz0QL6SXdxWDRMbbs7GwZZ2nB W8jy8vKuwnBlZaWqqurQoUPxOz4FKioqQUFBO3bsOHLkSJ+HYXNz89GjR1+6dMnR0dHExATe Vt++fRtr9dbX1y9YsGDatGkfZAMAAHDnzp379+8bGRlt3LjRysqKz+c3NDRgM32wiQkaGhq4 8AB36ejoAABgX+vr16/hxm4u8bDzuZshyTlz5sAht19++SUjIwN2DgsEAiyBkpKS6J8SKSkp +e2334qLi+G9nWhftIKCAgBAU1Oztra2urqax+MNGjSIRCIBADQ0NBgMhuyT2gAAFApl27Zt a9asiY2NDQ8Pj4qKmjdvnq6u7vPnz5uamvCpATh16pSDg0NWVpbonClsu+gWDHhRHjx4MH4H AAAArLNBYq+DKGPHjj169Ch+60DjdQejqq0Sv7V/gGeCUCi0sbG5du0al8t1d3cvLS3lcrk8 Hu/58+cSzzFcTWEwGLW1taamprgrdUtLCyy5qakpLS3NysoKnpmytIOTkpJOnjxZXV0Nu7sF AgHMO27cOF1dXfCubmLDc/3Hq1evYCtQ/I6TSqXKMRHs+fPne/fu1dfXx1ZzEt/y5MmT9vZ2 GAsBALADHF4T4K+grKwMADA0NNyzZ09ERMSGDRvCwsLodDouIxArXLwouKW1tXXLli18Pv+X X36BQ7/ihWNlSsTFxcXBwSEjI8PLywsGXWNj42nTpolek/fv319aWurn59dPU1u664sWxdHR ccOGDVIbwQAAMplsb29Po9EkDngwGAwqlWptbY31UX8OwFkY+K29RkdHZ/Pmzerq6pcvX66t rYV1z8PD49E71q1bB7oYD163bl1GRsalS5fmzp2roaFBIpE0NDRgE/bRo0enTp2CyeBpLQqs 7UwmE/tfT08P3ofC7i+JMwZhrm46qeCBTp48eevWrVWrVmGd4V0Bu6dEoywA4Oeff87Lyzt2 7Nhvv/0muh28i1XwDZuZmZFIpKamJoFAwOfzm5qaKBQK/AgAgM7OTtFiYZcy7kBv3rxpb2/X 1NT87rvvYFcVk8mELXI43gMA6M2D0a2trUVFRQQCAY5KiqKurq6goFBfXw9H3WBjCN4SfWGM 05iYz3rfkrvw6mwJu8Ce9r5ft5+AMZhCodja2tbW1jY2NlpbW8PhBhaL1dU5hqsppqamHh4e lZWV2LweSHNzM4VCgX2P+fn5VlZWWMmiycRpamratWsXn8+Pj4+Hk+wwsOFY2IaD1VDqDWtv 6Ojo6OoCSyaTJc7n74aampqNGzcqKyuHhYVpampK3ALejeliodTIyAgAAKMA/N/Y2JjL5erq 6s6aNWvatGl8Ph9Os8JlFC9ctCh4jTIyMuJyuVu2bKmoqPjll1/gjbLEwrtHQUHhzz//PHr0 6PHjx9XU1JSVleFgNnZNDgsLi4uLW7p06fLlyz/I2Xd01w7uDQKBAJ67OBQVFdvb2/FbPx0c Dmfnzp2mpqZ+fn74fX2BsrLyokWL4PzAHTt2GBsbp6WlHT58WFVVNSsrKzQ09OLFi6GhocHB wbBvWSIKCgrTpk2Lj4//8ccfR40adePGjd27d4vOeBTFxsbGyMgoIyPj1KlT8CycOXOmnZ0d hULJyMg4c+bMjRs38HkAmDlz5sOHD3ft2jV+/Pjnz58fOXJEPLqDd3GrvLwcTnesq6uTeAMB 3nXyREdHP3/+fM2aNXAjh8MRCoU5OTmZmZkAANGHAf744w8DAwMGgzFy5EgtLS0vL6+EhIT9 +/cLBAIulztv3jwY8Gpqao4dOyY6QKuvr19WVvbbb7+NHz9+xowZAAA+n79hw4a2tjY46lZZ WamlpTV8+HAzM7PLly/HxcVxOBwKhXL79u3Q0FCsHImkpqbCOV/YDXVdXV1UVNTNmzc5HM6y ZcuwCxAGiURycXG5e/euv7+/oaFhdna2pqamg4MD/LBVVVUhISEAAEVFRThO8fr1azjioKKi IrFH5LPlN6vw8Q/sCESCJXU4AKCQ/SSn+dE9l3x8ur4Gi8E2NjYAABKJZGFhAeMci8Xq5hzD sWrVqpSUlOPHj3t5ecFWlEAgYLPZgwcPptFopqamlZWVVlZWcH4fi8WCERQAoKampqSkVFFR kZGRAW9e09PTXV1dOzs729raUlNT4X1eWVkZbgayk5NTQUFBWFjY8OHDYbH9BJVK5XK58Dtp bm5uaWlRVVWFbWIul4vdzspCS0vL2rVrYY/dgwcP4CyzgIAA0S3u7u4GBgYPHz5UVla2tbWF GSdMmKCpqfnvv/9qamrGxcVpaGjY2dl98803w4cPt7CwiIuLAwDY29sDAEQzih/O3d197ty5 ly5dOn78eH5+fkFBgaurq46Ozg8//PDo0SMnJ6e6uroLFy4YGRkFBwfjCr9x40ZYWNh///vf kSNHvv9IYggEgj/++KOqqmrz5s2DBg2KjIyE1+RXr16dPXt28ODBenp6Fy5c0NbW9vT0xGfu NbK2g3sKi8US7dzHoFKpUu8oPxoVFRVbtmyZNGlSPwVgyMKFCykUSkJCwosXL0JDQ8eOHRsX FxcbG2thYcHn83V1dSWOB+MIDAz8+uuvCwoK/vrrL21tbdH5RDjIZHJISMjo0aMjIiLq6+sD AgK8vLxoNNqePXu0tLQuXrw4efJkfB4AZs6cGRAQQCKREhMTyWQybI+K4+vrO2LEiBs3bjCZ zN27d1MoFNhdLM7s2bMdHR2Li4vv3LmD9eMFBgbq6+tHRkba29t/++23WF4lJSVNTc3Y2Fhb W9vdu3fDlLNnz05LS7t3797XX38Nn9n95ptvNDQ0UlJSFixY8O44YPXq1SYmJmlpaXDiBgCA RCJt2bLF0NAwISEBDqCEhYVRqVQNDY3IyEhXV1fYlzB16lTYRO6GqKio/fv379+/H7tivnjx 4ujRoy0tLf7+/riOa4zdu3fPmDGjoqLi1q1bDg4OYWFhWIcng8GIioqKiorCxiMqKyvhIcLD w98XMRAwoZjec8knComxdTGxdTHKBOo9l/yP8GwSHAWgUChaWlrp6empqalkMhmLwV2dY+IM HTrUw8ODwWDAeyDwrsMZFnXu3Ln09PTRo0djJWMZKRSKr68vmUy+efPm6NGjPTw86urqqqur 16xZ09HRERERsX79ehcXF/GW7rJly9zd3fPz80tLS7u55+492tra8A03NjbW1NRoaWnV1NTA B3KamprEO6i7gclkwgbopUuXDh06dOjQodraWtwWDofDZrOLiopGjx4tWqnDw8ONjIxOnjxp aGgYHh5uYGDg6+v76tWrc+fOqaiobNu2beLEibiM4ofjcDgjRowICgpqbW2Njo52d3eH8yfg FMvMzEyY7OXLl+KFv379+uXLlxUVFW8/jCS2bt26adOm5ubmgwcPLlu2DACAXZPhIWpra3// /fdDhw5JbL30HoKwC2fD2bNnZ82aJdqHlp2dnZGR4ezsLLVHmsfj3blzB97j8Hg82HswaNAg 2D2Sl5c3ceLErrpKPhoVFRU7d+709fX18PDA70MgZANrKGO4ubn107gRAiEj8EkqGxubsrIy e3t7DQ2NpqamvLw8S0vLwsJCFxcXifOVekNeXt6mTZtWrVrV06Vm5M4oC6mpqdu2bUtMTBSd lYbjyZMnFAqlR89M9y1d3iR6enomJyeL9jTKPjUa9vVxudza2loOhwNn+pSUlKioqAwePJhC oXA4HKktv/4mNDR006ZN/bpGB+KL59atW9euXRPdMnjwYBSDEZ8WGo02bNiwZ8+etbe3wyut hoYGl8t99uzZsGHD+jwAAwDs7e1TUlJw0zJkQe6MspCTkzN16tRuAjAAoPtu6o9Al+1gcXrU Ds7NzeVwOGZmZoMHD4ZNXh6PV1tbW1FRoaKi4uDg8MnbwQgEAvEFA7thHRwcaDQam83Ozc01 MzPrah7JF0ljYyOFQpE4Kvr50IMYjEAgEAgEog/przlZCAQCgUAgukdyDJZ9ZUqp8JA/+GPR 1WNC3YD8wQiE7MhRxT45yB/8mSM5Bmcgf/BAQz65KfIHIxAyIl8V++Qgf/BnjuQYjPzBvQRT wyJ/MH6HGMgfDOmPZzO+VJA/WHaQPxgHW8wfzOfzS0pK7t69K27zqxPzBwMA6uvrU1JSnjx5 Iv4IuBxIfjYJ+YP7BFPkD0b+YGmYIn9wz2Ehf7AMIH+wLP7gly9f/vDDD7C1MHz48KioKOyy IO4PHjJkCPS3wpt+Jyenw4cPix+lR0huB/cG5A/GQP5g5A+WCvIHywEL+YOlgfzBuDY6kOQP BgDs27evrKzs119/TUlJCQsLgwEYXpPF/cGdnZ2//fYbiUSKj49fsWJFZmamHC0fHN3FYOQP 7j3IH4z8wd2D/MFywEL+4G5B/mAZ/cF1dXVZWVk6OjrR0dGzZs06dOhQe3t7Xl7elClTfv/9 dw8xf3BVVVVNTY2tra2+vj5cbRRbIlduJPdFQ5A/uJcgfzDyB0sF+YN7CvIHS6UO+YNl8wfD 8Xg2mz1p0iQVFZWkpCQbGxsnJydtbW2sB5cv4g+GHwp+HNEP1Ru6i8GiY2zIHywHyB+MA/mD xUH+4J4CzwQh8gd3DfIHy+gPhleYMWPGLFmyxNbW9t69ewUFBT4+PgkJCVgaUX8wHOQS/1C9 obu+aFGQP1gOkD9YNMoC5A/+skD+YFGQP3gg+oPhPTRsZ8PfBd6TYdfksA/9wTC96DuBczZ7 Q3ft4N4gQP5gAADyByN/MPIH9zVYDEb+4K5A/mBYuFR/sK6urrOzM7xa3r59m0AgTJo0KS8v b9WqVQsXLjQwMBD3B0+cOPHOnTt//vknvLpik8DlRtZ2cE9hIX/wO5A/GPmDsQ5PBvIH9xrk D5YK8gfL7g/eu3evm5tbVFQUh8PZtm2bs7MzjUbT0dEZNGgQPATOH7x79+5JkyZFR0ez2ey9 e/fCLpPe0KWzAfmDEQipYA1lDOQPRnxy4JNUyB+cKoM/+JPT5U0i8gcjEFJB/mDEZwgN+YMB ALL5gz85XbaDxelROzgX+YMRCATi04H8wcgfjEAgEAgEokv6a04WAoFAIBCI7un3GMxD/uCP RVePCXUD8gcjEF82yB/8mdPvMRj5gz8O8slNkT8Ygfiy+Zj+YKFQOG/evIMHD+J3SEPujF8A Xc6L7hO4XO7Lly/h+kQlJSVDhgzBloz5soHLlgIAiESitrb2+PHj/fz8ZHw0/vTp08eOHQMA UCgUIyMjb2/vuXPn4hOJ0SN/8Pr166dPn753715DQ8P6+vr++FFyc3OzsrI8PDykThJesWJF QUHB1atXu1kpsxuSkpLKysrgIrHYxo6OjtDQ0Js3b7a1tVlYWPj6+kJTCoPBOHToUE5ODolE Gjt2rJ+f365du7KyskTXqnR1dQ0JCVm/fj1cvgcCl1WBvymFQjEwMJgzZ86iRYvEtWVMJtPL y8vAwODq1asAgODg4JiYmB07dpiamq5cuVJ0ZUrsJIGMGDEiKioK+xPRDU5OTvDpWx0dndGj R/v5+cE1YXDU1tYmJyebmpq6u7vj9/3PgPmD6+vrc3Nz5ZgPy+PxCgsLKysrLS0t4UoaHR0d RUVFDAbD3NxcVI1QWlpaVVWFaVEEAkFBQQGTybSzs4OLmciSEU4fa2pqcnBwgCtkAQA4HE5e Xp5QKBw5ciRcnlr2dyWKeC4g6X1ilJSUYOt+AwBsbGzgpLbi4mImkzl06FBZLrnd078x+HP2 B38ETJE/GPmDpWGK/MFyQSAQoIw2OTm5qqpK4u1LcXFxWFiYj4/P/2YM7id/8LJlyzZu3Air AwDAx8fnhx9+gK8fPnwI3q06yROT/lpaWkrNmJubu3XrVriou4KCwt69ez09PV++fLlu3Tq4 pv3gwYOPHj1KoVBkf1cY4p8FJyemUCi//vqrqNkwISHhwoUL2J/h4eHa2tp+fn6PHz8GACgo KEAPPZZADvq+L3qg+IM/AsgfjPzBUkH+YPkgEomLFi06ePAghUIpKSlhMpm4ypWbmwsf97hw 4YKbmxuQVPvgsv5paWne3t5nz5798AgDm/7zB1Op1BkzZsTExPz777+qqqoXL17EVgTLzMxU UFAYPXo0AEBc+qurqys1o5qamrOzc3x8/N69ewUCwf/7f/8PAHDy5Mna2trDh7VDTkYAACAA SURBVA+HhobW1taePHmyR+8KQzwXTk6sqKgYHBwsEAiEQmFrayuW8fjx43BxPScnp+jo6MeP H8+fP//69euDBg36448/ulpVUEb6PgYPFH/wxwH5g5E/uHuQP7g3kMlkuMYhgUDAVS4KhQKX XnJ1dT1w4IDE2gcL2bdvH4PBkLpw6QCiX/3BCgoKvr6+5ubmPB6Pz+dbW1tDPRRcFsLGxgb2 FUuU/krNaG5uHhQUpK+vD5dZhJe1Bw8eKCkpjR8/3sXFhUKh3Lt3T/Z3JYp4rq7kxHCJX0xP 9+bNmydPnsDGUmlpKQBg8uTJ2traDg4OPB7v6dOn74/Rc/q+L3oA+YP7G+QPRv5gqSB/sHx0 dnbGxMTcvXuXzWbb2tpWVVXhKldBQYGdnR0AwNjY2NnZOTc3V7z2waJMTEzCwsL6Y1bEp+Ij +INPnTr1559/GhkZwRoNAMjPz+dyuZj+qCvpr9SMEDi4MGfOHD6fz2QytbW1YTxWVVWtr6/v 6OiAUzGkviuJSJUTGxoaVlVVYafEtm3bAADq6urHjh2DlfTx48dDhgyB0Vc82PeIvo/BA8sf 3K8gfzAO5A8WB/mD5UMoFAYHB6urq0+ePDkgICAvLw+IVS5RqxW81OISwBdeXl5fUgAGH8Uf 7ObmRqPRjh49umzZsujoaA0NDTiHccyYMTAB/ErhNQHWdLhFakYAQEJCwsWLF11dXefPn08k EhUVFbFrC5/PJ5FIsHtVlneFlYkhi5w4MDAQJl6wYIGbm5uFhcXff/996tSpiIiIjRs3JiYm njp16uzZs/A+oJf6wr6PwWQy2d7evrCwkMFgiE9WZLzzB+O2f5FAf/CNGzcuX768ZMkSzB+M e4KIzWaLr+C6bt2677//Hr4WCAQkEolGoyUnJ2N9zjCMiRsG5fYHV1ZW1tXVdbWUnag/2N/f f8aMGZ6envhEInTlDy4rKzt16lRjYyN2lkNgrIJv2EzEHywUCuXzB1OpVOgPTkhIqKysZDKZ I0aMyMjISEtLmzhxIgCAw+HAe2c5kNEfTKFQvnh/8BgNZ/jnR/MHQxQUFKCdEAKnJuAqFwy3 8NyQWPvCwsKApEo00OlvfzCXyx06dOjQoUPT09MzMzOLi4tdXFyysrIoFArsQwYi0l8LCwtR 6a/UjOnp6Xv37h01atT+/fvhZcTQ0LCysrK1tZVIJLJYLGNjYwKBIOO7gmViiOeS+D7heLCK ioqhoaGJiQmBQID2w+bmZn19/UuXLsF7vh9++GHo0KHiYa5H9H0MhggGiD+4v0H+YOQPRv7g j4Ctra145YInYXp6ulAo3LBhg3gCfClfCv3qD9bV1Q0KCpo+fbqCgkJ2djaFQhk6dCiHwyks LBwzZgwW+8Wlv9XV1QEBAd1nzM/P//HHH4lEorOzM5yfsXDhwrlz5x4+fPiXX34hEAgCgeCr r76S8V2FhIQ8ePDg9OnTsJEjngtOXMW9T21t7YCAgOzs7EuXLgUFBXV0dIwZM+b69esAADi5 j0QiNTQ0HD16lEgk/vTTT1Ifr+ie/orBLBZL4pADlUrFuoD+R1i4cGFERERCQsKyZctguI2L i6PRaBMmTOD3xB9MJpPT09NzcnIcHBy6+dXJZHJISEhwcHBERISmpiacXAAA2LNnT0hIyMWL F+fPny8+iWDmzJnNzc0XL15MTEy0tbVlMpnYk3mi+Pr6Pnv27MaNG87Ozrt37z558mRXftbZ s2enp6cXFBS0tbUtXrwYDrQEBgYeOHAgMjJywYIF1tbWmPVPoj+YSCSmpaURCARRf3B+fn5K SoqPj8/vv/8O865evbqqqgqmhDEY+oPPnj2bkJDA4/GcnZ39/PyoVCqVSo2MjDx48OCjR48o FIqM/mD4Ys+ePfALgf5gAwMDf3//RYsWfZD6Hbt37z58+HBmZmZJSYmDg0NAQADWz8FgMGCZ ZmZmMAZXVlbu378fAGBgYDCwYjD0BwcWb4BjwPa0UR/HHywREokkXrmsrKzmz5+fmJh49+7d 6dOniyfAl/KlAP3BFAqlsbHx1atXZmZmFRUVAoFAS0urN/5guCUyMvKrr75KS0tjMplWVlYb NmzQ0dFJT08XCASiY7pQ+nvq1Cko/d2+fTuVSpWa8enTpzweDwAA10hQVVX18fHx8fFpbGxM SEgAACxfvnzx4sXV1dWyvKuqqqrnz5+/efMGVkDxzzJmzBjx90kgEOB4MJVKXbJkCeyC1tLS 2rx58/z58/l8/owZM9hstr29/fbt2y0sLGBRctMvzgbeQPAHIxC9B/mDEZ8hH98ffO7cuSNH jpw9e1Z0DqYsyJ1RFg4dOpSdnX3+/Hn8jt6RlpY2fPjwXnZBY0huxPSSAeEPRiB6D/IHIz5D aB/dH7xkyZL58+dLHH/sHrkzykJOTg5cYqFvgT3SfUV/tYORPxiBQCA+IcgfzGAwdHR0uhov +0zolxiMQCAQCARCKn2/ThYCgUAgEAhZkByDs7Ozw8PDs7Oz8Tt6Dg/5gz8WXT0m1A3IH4xA fNkgf/BnjuQYnJGR4ebmBp8u7SXIH/xxQP5gBAIhDvIHf+ZIHqx2dnZOS0tzdn67Ao7cIH8w 8gdLnSSM/MEQ5A+WHeQPlh3kDxZFPBeQ9D4xuvIHl5eXw/WYu1qSWXYkx2BHR8fs7GxHR0f8 jh7yOfuDoffRyckJv6PvQP5g5A+WCvIHywcB+YOlgfzB8DWG+GeRwx9saGgI1z2EW1auXLl2 7VosgRxI7ovuDQPFH+zn5xcaGpqSkoLf0XcgfzDyB0sF+YPlA/mDuwf5g7HCMcRzyeEPrqmp OXnypIODQ0pKyujRo0+fPl1dXS1ykB7TXQwWDZOyz9IaKP5gMzOzoKCg6Ojo+Ph4/L6+A/mD kT+4e5A/uDeQkT9YEsgf3H/+4MzMTKFQ6Orqqqam5urqKhQKezlxSnJfNCQ+Ph4zbPB4PDc3 t7S0NKkd1APIH2xmZnb48OEdO3ZUVFRs2rQJv7vXIH8w8gdLBfmD5QP5g7sB+YP7zx8MAzNM Caf49OieRpzuYrDoGFt2draMs7QGlj9YRUUlKChox44dR44c6fMwbI78wR+C/MHiIH+wfAiR P7hrkD+4//zBcIKLaHqJijnZ6S4Gi+Lo6Ci1BQwhD0B/MIFAkHqtlAPkD8apaZA/+EsC+YM/ W5A/uP/8wVh67H84bVNuZI3BPUUwQPzBHA5n586dpqamfn5++H19AfIHI38w8gd/BJA/WBQq 8gf3mz94woQJmpqa//77r6amZlxcnIaGBrynl5vu5mT1BhaLBecT4aBSqT3qVOxXKioqtmzZ MmnSpH4KwJCFCxdSKJSEhIQXL16EhoaOHTs2Li4uNjbWwsKC3xN/8Ndff11QUPDXX39pa2t3 81ANmUwOCQkZPXp0REREfX09nFxAo9H27NmjpaV18eLFyZMn4/MAMHPmzICAABKJlJiYSCaT YXtUHF9f3xEjRty4cYPJZO7evZtCoXS1Hvrs2bMdHR2Li4vv3LnT0tICNwYGBurr60dGRtrb 23/77bdYXon+4NmzZ6elpd27d0/UH6yhoZGSkrJgwYJ3xwGrV682MTFJS0vDhvegP9jQ0DAh IQEOoISFhVGpVA0NjcjISFdXVzjFUUZ/8P79+/fv3w8nWoN3/uCWlhZ/f39cxzXG7t27Z8yY UVFRcevWLQcHh7CwMKyfg8FgREVFRUVFYT2ilZWV8BDh4eHvixgIQH8wUUiMrYuJrYtRJlA/ uT8YV7mgP7i5ufnu3bvl5eXiCfClfClAfzAAoLGxsaamRktLq6amprGxEQDQG3/woUOHDh06 ZGho+NVXXz148CAuLs7KyiokJERHR+fx48cCSf7g1tZW6OXds2ePjY2N1IzQH9zR0XHs2LFD hw6dOHFCQUHBx8fn22+/ffTo0cOHD6E/WMZ3hfmDYeHiuTgcjvj7JBAIhoaGurq60B8sEAgi IiJ4PB70ByspKYWHhxsZGZ08edLQ0DA8PBze68hNl86Gs2fPzpo1S7QPLTs7OyMjw9nZWWqn NG8g+IMrKip27tzp6+vr4eGB34dAyAbyByM+Q5A/GNJP/uC+RXIjBgDg6emZnJws2tMo+9Ro 2Nf3mfuDQ0NDN23a1K9rdCC+eJA/GPEZQkP+YABAv/mD+5Yu28Hi9KgdjPzBCAQC8QlB/mAG 8gcjEAgEAoHoiv6ak4VAIBAIBKJ7JMdg2VemlAoP+YM/Fl09JtQNyB+MQMiOHFXsk4P8wZ85 kmNwBvIHDzSQPxiB6Ffkq2KfHOQP/syRHIP71h+sp6enp6fXU1/HgCY3N5dOp9Pp9LFjx06f Pj0oKEj2p6JPnz4N87q6unp7e4uLBCTSI38wnU6HSzHDx+B6+XybRHJzc0+cOCG6CkdXiLqb 5CApKSksLAx3anV0dBw8eHDatGlubm6+vr6YoorBYAQGBk6ZMmXatGm7du1qaGhYv349nU7P zc0F774Z+Cwy3I6Rnp6O/aaurq7ffPNNZGSkxPWGmEwmnU7HVhUNDg6m0+mxsbEwu+gjxViB ECgYQMiCr68vnU6fN28e/NPd3Z1Op8vokXzw4AH9nWFMnPDwcPhzQGPNli1b6HS6u7s7EKli r1+/ptPpcCnfa9eu0el0uN5Wj1i5ciWdTpdj2ciegvmDVVRUcnNz5QjDcI5tbGwsJtrp6OjI zc1NSkp6+vSpaEqoAcZWLxAIBE+ePElJScGWBZUlI5vNvnv3bnx8vOiXw+FwHjx4cP/+fazf TvZ3JYp4LiDpfYrS0NDg7e2NLW3E5/OTRbhz586HyXuM5AljyB/cJyB/MPIHSwX5g+WAxWIR CISqqio2m93U1MRms+Fi5vh0PaelpQWuXFtUVDRx4sTS0lICgcBmswUCAVbF4NLTnz/IHwxf Y4h/Fqn+YADA1atXy8rKsGEILpf7n//8B9s7cuTIz26dLOQPxkD+YOQPlgryB8sBi8WCK3WX lJSUlpaSSCQjIyO4HJv4OYarKaLrpW/dupVOp+fk5GBbmpubVVVVtbW1i4qKmpqa6urqoHsG rnqBVTGJXLlyxdfXd8KECTNnzoyOjgbv2twXLlwICAjYvHkzn88/ePCgh4fHkiVL+nsACPmD scIxxHNJ9QcLhcIrV64oKyvjlsd3dHSEy+2dPn1adLscdBeDkT+493QgfzDyB3dLPfIH9xy4 cD+NRisqKiopKbG0tKRQKHC4p6tzTLym3L17NzU1de7cuaNGvfdMsFgsoVAI5W+wwQR/aGy9 1W6gUqnm5ubff/89HArBpkGdOXMmMzPT0tLyypUr0dHRQ4YMcXd3xxZQ7A+QP7iv/MGZmZm1 tbVeXl75+fmnT5/GesIFAsHTp09xa+TJh+S+aAjyB/cS5A9G/mCpIH9wT2ltbeXz+TBSFhUV tba2WllZFRUVwRjc1TmGqylcLjc4OFhDQwNX8Zubmzs7O21tbc+dO1dSUmJiYgINB7J0dHt5 eXl5eXV2dlZWVsbHx0NlFgCAy+XGxcXp6uquXbsWALBlyxY7O7vMzMzc3NwP8vcdyB/cV/7g f/75BwAwd+7cixcvJiQkzJ49G7aLcnNzFy9eDHft2LFDtPCe0l0MFh1jy0b+4J6D/ME4kD9Y HOQP7inwTBAKhTY2NteuXeNyue7u7qWlpVwul8fjPX/+XOI5hqspDAajtrbW1NQUt3BjS0sL LLmpqSktLc3KygqembK0g5OSkk6ePFldXQ27uwUCAcw7btw4aBSF1RAbnus/kD+4T/zBTU1N d+7codFoJSUl8OqdmJi4dOnSo0ePampqqqqqLl++PDY2dvny5SYmJqKH6BHd9UWL4ujouGHD BqmNYPDOH0yj0SQOeDDe+YM/q7Uq4SwM/NZeA/3B6urqly9frq2txQymcCDh0aNH69atA12M B69bty4jI+PSpUtz587V0NAgkUjwZIUZT506BZOJGwbl9gcDALrppBL1B69atQrrDO+KrvzB eXl5x44d++2330S3g679wXw+Xz5/cHt7O/QHQzkS9AcDANLS0mCa3jwYLaM/GADwxfuDsT8/ mj8YxmAKhWJra1tbW9vY2GhtbQ2HG1gsVlfnGK6mmJqaenh4VFZWJiYmim5vbm6mUChwLCM/ P9/KygorWTSZOE1NTbt27eLz+fHx8XCSHQa8yoN3zSxYDaXesPaGjo/iD/b29razs2toaCgu LgYAdOUPBu8uLEbv/MHdZ5ToD25ubm5tbeVyuSwWy8jIiNC1PxhXOA7xXBLfp1Ao5HA4TCaT z+ez2exff/0VdjuHhYURCIRRo0YNHTpUX18f3oJLPTe6p7t2cG8QIH8wAAD5g5E/GPmD+xos BtvY2AAASCSShYUFjHMsFqubcwzHqlWrUlJSjh8/7uXlBTs2BQIBm80ePHgwjUYzNTWtrKy0 srKC8/tYLBaMoAAANTU1JSWlioqKjIwMePOanp7u6ura2dnZ1taWmpoK7/PKyspwM5CdnJwK CgrCwsKGDx+O2TD7A+QPhoX30h/8zz//YMN2a9asyc7OTkxMzM7O3r59+6xZs9ra2goKCvT1 9XvZpytrO7insJA/+B3IH4z8wViHJwP5g3sNHAWgUChaWlrp6empqalkMhmLwV2dY+IMHTrU w8ODwWDAeyDwrsMZFnXu3Ln09PTRo0djJWMZKRSKr68vmUy+efPm6NGjPTw86urqqqur16xZ 09HRERERsX79ehcXF/GW7rJly9zd3fPz80tLS7u55+49yB/cJ/5g+NNDsEuuvb39lClTbt68 mZSUNH78+LCwMImtTdnp0tmA/MEIhFSQPxjxGQKfpEL+YOQPRv5gxBcO8gcjPkNoyB8MAED+ YA7yByMQCMQnAvmDGcgfjEAgEAgEoiv6a04WAoFAIBCI7un3GMxD/uCPRVePCXUD8gcjELIj RxX75CB/8GdOv8dg5A/+OMgnN0X+YARCRuSrYp8c5A/+zOnfGMxF/mDkD5aGqLtJDpA/+H8Q 5A+WHeQPFkU8FwDg2bNn169ff/TokfgagtXV1fv27du5cyfUKDEYjNu3b9+9e7cPuw/7d8LY 5+wP/giYIn8w8gdLwxT5g3sOC/mDZQD5g+FrDPHPsmXLlq1bt2Lr7tnZ2R0/fhwumsblcs+c ORMREQHvWn744YeYmJijR4/COK2hoXHmzBl4ceglfd8OHij+4I8A8gcjf7BUkD9YDljIHywN 5A8WX6dMPFdra+vUqVPDwsJu3rzp6OhYUFBQUFDA5XL5fH5zc/P58+ddXFwwx5Szs/OPP/6Y kpKyfPnypqYmOdo8Eun7GDxQ/MEfhw7kD0b+4G5B/mA5YCF/cLcgf7Ds/uAZM2Y4OzsTCAQe j6ekpGRiYjJ79uxvv/120KBBUVFRv//+O7ZO+IgRIxYsWEChUOAglJ2dnWjhctP3fdEDyB/c 3yB/MPIHSwX5g3sK8gdLBfmDZfcHAwByc3NXrlxJoVD++OMPDQ0NfX19fX19AID4tyQQCNzc 3Hg8np+fnywmX1no+xg8sPzB/QryB+NA/mBxkD+4p8AzQYj8wV2D/MGy+4MBAIaGhnv27ImI iNi4ceORI0ciIyM/zPGezs7O//73v9euXQsNDeXz+b6+vvgUPafv+6LJA9Af3E8gf7BolAXI H/xlgfzBoiB/8AD1B3O5XF1d3VmzZk2bNo3H42VkZMDxYHxOAHg8HoFAmDp16rfffgsAuHPn Dj6FXPR9OxgiGCD+4P4G+YORPxj5g/sWLAYjf3BXIH8wLFyqP3jUqFGBgYHDhw+3sLCIi4sj EAhWVlazZ8/W1dUVty0FBgYymcwJEybA6Dty5EhcAvno+3YwhDUQ/MEfB+QPRv5grMOTgfzB vQb5g6WC/MEy+oNJJJKvr++rV6/OnTunoqLy008/ubm56evrS2zhrFixQllZ+dy5c42Njd7e 3rAXs/f0i7OBNxD8wQhE70H+YMRnCPIHQwa2P7g3wL4+7uftD0Ygeg/yByM+Q2jIHwwA+PL8 wbIDn/fiIH8wAoFAfCKQP5iB/MEIBAKBQCC6or/mZCEQCAQCgegeyTE4Ozs7PDw8Ozsbv6Pn 8JA/+GPR1WNC3YD8wQjElw3yB3/mSI7BGRkZbm5u8OnSXoL8wR8H+eSmyB+MQHzZIH/wZ47k wWpnZ2f4YCV+Rw/hcrkvX76Ek85LSkqGDBkCH7b74oELkAIAiESitrb2+PHj/fz8ZHw0/vTp 08eOHQMAUCgUIyMjb2/vuXPn4hOJ0SN/8Pr166dPn753715DQ8P6+vr++FFyc3OzsrI8PDyk ThJesWJFQUHB1atXu1kpsxuSkpLKysrgcq/Yxo6OjtDQ0Js3b7a1tVlYWPj6+rq6ugIAGAzG oUOHcnJySCTS2LFj/fz8du3alZWVJbpWpaura0hIyPr16+ESehC4rAr8TSkUioGBwZw5cxYt WgTXdhCFyWR6eXkZGBhcvXoVABAcHBwTE7Njxw5TU9OVK1eKrkyJnSSQESNGwDVyEVJxcnKC T9/q6OiMHj3az88PrgmDo7a2Njk52dTU1N3dHb/vfwbMH1xfX5+bmyvHfFgej1dYWFhZWWlp aWlrawsA6OjoKCoqYjAY5ubmcFFMCNQAY1oUgUBQUFDAZDLt7OzgYiayZITTx5qamhwcHOBi WwAADoeTl5cnFApHjhwJF3mW/V2JIp4LAFBdXR0REcHlcv/v//4PW9bi9evXogt6Dx482NbW lsViFRUVsdlsW1tbiaecHEiOwY6OjtnZ2Y6OjvgdPeRz9gcHBAR4e3s7OTnhd/QdpsgfjPzB 0jBF/mC5IBAI0PyanJxcVVUl8faluLg4LCzMx8fnfzMGI38wfI0h/lk2bdqEkwRjMbiwsPA/ //kPltfX15fNZm/btg0O+ZFIpKCgIFFXm9xI7ovuDQPFH+zn5xcaGpqSkoLf0XcgfzDyB0sF +YPlg0gkLlq06ODBgxQKpaSkhMlk4ipXbm4ufNzjwoULbm5uQFLtCwsLo9PpaWlp3t7eZ8+e /fAIAxvkDxZfp0w8V21tLU4SDD68IK9evRouq7d+/Xpra+vFixdfv3790KFDfD7/0qVLWLLe 0F0MFg2Tss/SGij+YDMzs6CgoOjo6Pj4ePy+vgP5g5E/uHuQP7g3kMlk+PQngUDAVS4KhbJ0 6VIAgKur64EDByTWPljIvn37GAyG1IVLBxDIHyyjP1hcEhwVFeXh4YFdkNva2vLz8+E3pqam tnr1ai0tLag8+Bj+4Pj4eMywwePx3Nzc0tLSpHZQDyB/sJmZ2eHDh3fs2FFRUYHTiPYJyB+M /MFSQf5g+ejs7IyJibl79y4cnKuqqsJVroKCAniVNDY2dnZ2zs3NFa99sCgTE5OwsLD+mBXx qUD+YNn9wTAjBnZBhv3hkZGRkZGRCgoKO3bsgOrY9evXP3z40N3dHVPR9JLuYrDoGFt2draM s7QGlj9YRUUlKChox44dR44c6fMwjPzBOJA/WBzkD5YPoVAYHBysrq4+efLkgICAvLw8IFa5 RK1WMCTgEsAXXl5eX1IABsgf3EN/sCjTpk2DF+Q3b94cPXrU2Nj4xYsXGzduDA8Ph5ffFStW DB8+PCoqyt/fPywsDJ+/53QXg0VxdHSU2gKGkMlke3v7wsJCBoMhPnOM8c4fjNv+aSEQCFKv lXIA/cE3bty4fPnykiVLMH8w7gkiNpstvoLrunXrvv/+e/haIBCQSCQajZacnIz1OcMwJm4Y lNsfXFlZWVdX19VSdqL+YH9//xkzZnh6euITidCVP7isrOzUqVONjY2BgYGiu+D3L+4PFgqF 8vmDqVQq9AcnJCRUVlZCf3BGRkZaWtrEiRMBABwOB3cLLDsy+oMpFMoX7w8eo/H2vvyj+YMh CgoK0E4IgVMTcJULhlt4bkisffAaKl6JBjofxx88dOjQ9PT0zMzM4uJiFxeXrvzBFhYWOH9w 9xkl+oMrKytbW1uJRCKLxTI2NiZ07Q/GFQ7LxBDPJQ68IGtqao4aNQp23CooKLS0tAiFwvb2 dicnJ0dHx4sXL2ZkZGDN8d4gawzuKYIB4g/mcDg7d+40NTX18/PD7+sLkD8Y+YORP/gjYGtr K1654EmYnp4uFAo3bNggngBfypcC8gfDwqX6g93d3XGdf5GRkUeOHDlw4EBlZWVcXNzMmTPz 8vIEAsGkSZOioqLOnTs3Z86cioqK9vb2ESNG9D4Ag/6LwSwWS+KQA5VKxbqAPjkVFRW//vrr rFmzsD7e/mDhwoUREREJCQnLli2D4TYuLo5Go02YMIHfE38wmUxOT0/PyclxcHDo5qEaMpkc EhISHBwcERGhqakJ/cEAgD179oSEhFy8eHH+/PlPnz7F5Zo5c2Zzc/PFixcTExNtbW2ZTCb2 ZJ4ovr6+z549u3HjhrOz8+7du0+ePNnVeuizZ89OT08vKChoa2tbvHgxHBAKDAw8cOBAZGTk ggULrK2tMeufRH8wkUhMS0sjEAii/uD8/PyUlBQfH5/ff/8d5l29enVVVRVMCWMw9AefPXs2 ISGBx+M5Ozv7+flRqVQqlRoZGXnw4MFHjx5RKBQZ/cHwxZ49e+AXAv3BBgYG/v7+ixYt+iD1 O3bv3n348OHMzMySkhIHB4eAgACsn4PBYMAyzczMYAyurKzcv38/AMDAwGBgxWDoDw4s3gDH gO1poz6OP1giJBJJvHJZWVnNnz8/MTHx7t2706dPF0+AL+VLAfqDKRRKY2Pjq1evzMzMKioq BAKBlpZWb/zBcEtkZORXX32VlpbGZDKtrKw2bNigo6OTnp4ukOQPPnXqFPQHb9++nUqlSs0I /cEAALhGgqqqqo+Pj4+PT2NjY0JCAgAA+oOrq6tleVeYPxhWQPHPItoBDsEuyJ6engUFBdHR 0WQyed68eX5+fq2trQUFBXFxcXw+393dXfSJj97QpbPh7Nmzs2bNEu1Da7W/qgAAIABJREFU y87OzsjIcHZ2ltopzRsI/uCKioqdO3f6+vp6eHjg9yEQsoH8wYjPEOQPhgxsf7Cnp2dycrJo T6PsU6NhX99n7g8ODQ3dtGlTv67RgfjiQf5gxGcIDfmDAQBfnj+4R+1g5A9GIBCITwjyBzOQ PxiBQCAQCERXvL1BIHxX+cFmBAKBQCAQ/cz7Rrr56Gbstb+T/qaxg45k1YVkMrCN8qGqqJC4 eDgAYMb50pYOKWs4IBAIBKIPCZ5i4mxE+yvn9dknn3iJfoREJK8XvWnsoO0p1ZvGdrmskuzs cjNMfcFKfcHa5fY/NA4hH44GKvd9rVUV8cucds/JWWZ98kshEIjeQCER7n5nPc6472c89YYb 5c0KBOBhph7vM7yn15aecnqOuQ61x4Ovk83U/P6Hr2CSY/CRrLp9HsZHsnqwtLdEDFUVPc3V 40vfxJe+8TRXN1TtgyeaBwpXvYdlr7K995118tIR3jZS/DyQskZuaGadjL0Fd7+zHjmICgA4 l99ws7wH6zjKzlhDFbchqvitkgjxGvL9KF38VpmReCASEeydZPRwpe19X+s/pg0ZpPJ2Ep+X hXqCz/DHq21jFljaD6ICAK56D/OyePvI4wgdSt5aO/h6x4TBRevtH660fbjS9vI3QwEAU8zU nm4cmbXS5s4K650TDbWUJV+ShmpRita/XbUHALBt/OBfPIwBAKqKxOebRmpQ3ueCW7JX2cKj YG8D0d84G9L+WTj08WrbxMXD5w7Hr5TSr6R+a3VtyXD4mkomlm4YGT7dFADA5QtDMxklDW0A ACM1RR/bLmv9Pg+j7xyk15eRg6hp31rB13/NMZ83Qp6PeaO8WZ9GDr5Xm1TWFD3fUo4wbKOr fHyWWfLSEZud3q976DZEdeWHVV5bmWSto9zQ2uOnrt2GqNW38kEXB8JBAMBv7KAEn+GRcy2G asm6wqh4LnUlhV8nG99YOuLgVBOp34ksb0xuJMdg2AXd+45of2f9f5++4fA6ObzOf5++8Xfu +w8gN/E+w6ea9+8V8z8p1ePPFG25/mL7hMHOhtLvjpvbBdGFTPxWaaS+aCluaMNv7Qs8TNUn mOBDY38g8UDL7HXHGdFmnC/xiCjOZbQqk4kAAFcT1Z/djbbdqnI8URD+sK6ZK6XC/13IHHOq YMypgvkXn8EtZY3csacKZ5wvUVVUODLd9IPU8jI5shge5frz9wM6/wsoKhCsdZQnmNAmmNCs dZQVFbpcOqZvGTmIGj7D9NCDV44nCgJvVBH74rAKBGCmoTTDUvozk+pKChpKJF0qCQAwZrAK s42nqfy28RdT1NjYJgAADNWiLO46BquQFfrkPXcPLsynVrJKmW097Y9UUiAcn2V2o7x5Y2Ll dw66Xhbqg1TIYdNNf51s4mP3wQqs44xpGTUt8LWqosKfM83u+1rHLLAcoq4IADDVUPprjvm9 76xvLhvhYfbBQl3jjGkPqlvEDwT3/jjO4M4K6+tL3uZaZq8z2Vzd99/yiqb2P6YNAQBMNFGN XTQs43ubv+dbmmlIfsZJPNf+ycadQuDzT5mlFiXgw8DkY6udv9Yu7VurtG+tzs+z7OqN9RU9 7jeQylRzdV8HXQIBjNBRJhLAqn/L4fb40jcn55hPNbcraWgTCsFfufU3yj/lBSvwRtWF+ZZ7 77y8XNyI39enFNS3Zb5kOxvRMl6yZw/T2OykTyYSrpS8eXuj4zUk6yV7npVW+Zv2Y4/q4ryH 2f+ZDwCgKRL3uhuNMaS18joPPXgFL+5aygq/ew4ZqkXJfMlWUnh7/xTiNST/devpnHoAwGYn /W+stRQIhJOPX/+V+349Mh9b7RE6FIoCkW5IW3y57JfJxsO0KIoKhJBMxoUCJiyksL7VbYja MG3KuXzmH5mMHRMM51tpEggEtyFqXudKOt8tpz1IhSyeHQAwVIty6ZuhRmqKUXkNYQ/rQBcf 4a855vFP3/xT8kZVkZi7xs7qaN6P4wZLPJCRmmIpkwtvkM+8+yybxgw6nv06/3UbACDtxdsK Lwfsjs7dqTX3fa3dh6im9qKc/2UUFQhOhipEQGC1CwAAulSSngrtfg27Q9D3S6/j+MHF4OyT +nvVbABAYX1bYf3be1Dx+iXxJBStdDtuV//iYexooGKopphe1XKvqgUAMFyb8utkYx0Vcs4r zn9Sqtkd75dYJxGBogLxZkXTOGPVuNI3zka0WxUs+uC3C49nfG+z8t9yM02l/7gaalAUbi23 Crj+Iq+uVbxiDlZVjPzaYrg2JTKvAfY4qikpHJhibKdHZbB5P92sev6mV2v67p5ouHykTg2r AwCAFfV3AfPEbPMQVcWXLbIuGc3vFO698xJ+b+lVrLGGtA6BMJfBicir3z/ZRDSli5Hq/Zq3 60n8n+tgBrtjbULFUjudw15D5sU8A0Lh4YxX+a/b3Ieo/jHNdNSJfFjTB6mQKSRiZXOHAgHg DnT9efOc4ZrjjFWnRBabayrFLBg64UxRDoOT+oJVx+FF5NUvsdNWU1JoFwhXx5fXc/hbxxns mGj4/buII4p4rislb1IrWbxO4cUi5mJb/HLutypY/tdfwNcS39gHqXuH5HYwxErn/YrB/k76 zzeN9JehJX5wqsnN8ubYksY1V8sXXy7j8N6evhxe5+LLZWuulseWNN4sbz449YPf7+NT3NDm c7lss5P+ipH4H6DPIRMJXH6nkZriHjejZVeee50r8bJUd3/X+7qWPuhE9ut96S9Fs/yf6+BO ACacKdp0rfKQp4mRmiIAYMcEwzoOb/yZouhCJk0R/9vNHqYxZ5jGnOjS+TFP77+7IcWYOVSz itWx+HLZaw7v9wev3P5f0faU6l0TDbG2yyIb7S3JL+ZGP/V10FVUIASlv/y7sPFSEXNq1Pu4 CADoKvtwbcrKq+Xz/n662lHPTk8ZdPERxOnqQInPmtxN1XZONDQVube11lXOfsV5n6gXtPE7 y9602+pR8TsQsmGpSSECArONz+sUFjW0lb1pFwqBpaas3YO9wX4Q9X41fqFyifWrq5MQq3Tf OegaqJI9o0pmXyi101M+l99AAODoTLOwh3UTzhS18To3jPlgqFKDQmrjd6ZXtYw3VgUAuBip Jj9v1qR80Ji5+rTpPynVZY3cyRHFeXWtEium2xDVH5JfzP376cpRerASbXcdXMPqcD1TdC6/ oZeXRydDmqeF+uEMxmYnfX8n/b/f3ShzeJ23KpoXWL9XGUpFIARYyHnTJuB3Cm9Xsk7l1AvE zC/jjGkP3sXg6ZbqsEvvYhHTRpc6mEaubO6At84UMrGyuR2r6Vgu8QPBcmJLGjsEwpIGbklD 2yRTtfzXbVXNHTANAKBTKMx8yX7N4QsBUCYRyxq5b8v9EPFcN8qbeZ1CAEATVwBfrByle2XR sA/zAdDFG+tD8NdxUS7Mt3y+aST8J/ssraKGtrGGtILXbVj0FYXD6yx43eZkRCvqn+7THlHc 0DbzfOkCa+0DU4zx+/oIMpHgPkR1tIFK6guWp7l6Rg37FZvH5QvTKlvGv+t9vfOClVze3Nz+ wTDwzKGaEU8ahACUMrk5r1onm6kBAKaYq5/PZwIAsl5ymG34bthZwzQvF79pbBPUsnklDfhz sYnLD39YV8fhCQEorG/rFIKUShaRQNB7N84aXcB8zeHXsnlEAuhm5L6r7PHPmpq4glo270EN 29VEFXTxEWQn+xXnm4vP9FRIiYuHH/I0UVNSUFUkUsnEN2IfHABwyNMkZ7Vtzmrbv+d/4MRc ZKMNR2qnWUroQWK28lXEbmXk4NYyq4crba8s/Cx0nB8NfRr5zbuxACM1xZctHax2gTZVytBa 71FVJKoqEl9zeACA/3MdDH9fmiJRYv3q6iTEKp25hlIuo5XXKSxlcpUUiJZaFGtdZXUlhVsV LABA0vNmGGsxtCgkdofgXlXLOGOaqiKRpkgsqm/DxWAcEivm2+rW8r66TbPUuFjYCAC4Ud5s o0dV7cWZucBa6+9CZmolKyqv/vrzpsyX7+9Xsl6yXYykj4uJQyUTPS3Ur5VJUH8CAAxVFQkA wGa3MomopqRQx+YBANoFwuZ2vj7t7VUieemI7a6Gm69VYhldjFTvV3/QYBA9kAFNkcHhwe0M Ns9A9f3KTl+P0Ex70QK7KIZrU7JW2lhqUUIypIyfiubCtsBj5b9uu/7u07kY0RJ8ht/9znrW 0PdjE91/A3LT3anjcPy9AsXfSV/GWVqrr1Yc8jTZNFb/SJbkr2PTWP2WdsEPyVX4HZ+Clg6B z+WyC/MtD0wx/ulmNX537/h1sjGb1/miqX3jtcqSBu7XI7RcjGnwSS0AwO3KtxOpKpvwnU7K JKKqIhGLsg1tfD0VMpVMVCG/3yjOIBXy69a356s42FG0lBX2eRhbaFLqODwiAWDjUtjdXfe3 eV1lx2hq42srkyV+hA8TSqewvm3TtRc6VNKZOeZbnPX3pL1s43dqKpOqWfietB+Sq+C96ggd SsyC97Hw70Lmz2kfdDCIoqdCymVInwHXtSDjLZMji5u40sv5glEmEalk+WNGj2jp6OwQCNWU FAAA++/WhmUxctfYEQAYRCPj6lc3JyFWHdKrWjaMHXSxkDnaQKVd0FnR1O5kSFMhE7FymB9O MtJQVmhpFzDb+I1t/EU22hk1bFa7gEQENEWi6JVdFIkVE1fdqGQiTZEYPsMUtsnKGrk0aROF usHTXH311XIAQMHrNgA+aO1UvGkX7eCUnX0exonPmnIZrfgdAAAAxhnTsI5oWF8kXkZmni91 NVG9tHCYV1QJ/F1cjGmHM16JpsEdSKJU1kZXeYWD7pJ/yuCfpUzu5IiS5SN1Ir+2WPBu5oc4 uFwAAB9bbW1lEhzqynzJhjcrsSVvrpU1NXEFYw1VzsyxSH3Bgr9s99+A3HQXg0UJyWTIOEWr pUOwOr7i+Eyz2cM0rz59g9s7e5gmAOCHG59FAMYQCoUEIO1C23O23aoWHTmo5/ByGK3iwxXi 51gbv5PD69RSJsH7Sh1lUs4rTiuvs0Mg1FBSqMEnf0tDK0/73dwQcbBT2d/JQIFA8IwqAQCI zv6VEanZtamkiiaOxI8AE7yTPsn6hTe08q8+bXIzVQUAFDdwxwxWyavrg2pAUySaa1JyGB/U fwirXaCkQFBSILQLhAAANSUFiY3v/3EYbJ4ulYRFOHMNJTWKQj3nY3xRZY1c+0FUbBgYIrF+ dXUSYpUu6XnT+jGDfnIdzGoXLLvyvJXX2dDK43cKZ54vFa+YAABNCgneb6VXtaylD9p5u5rX KWzldWpSSOwuvLzdV0xIK6+Tw+v84UaV6Lktx20rhKZIlNgNCQDg8DrFh7GkssVZX1WRuDW5 y9vZcUY0rFHRyutktQsGqZCbuAIlBYK6EonBfnsLwusU3q5kdQg6h+tQ7lezTdUVO/idr97t BWIHesXuwNrQg2hkeIjBquQj0023Jle9aH7/hbd0CBKevglw1icRAV/SRxfP5TZEdeVoPZ/L z3AzGNr4nfCkzmW0kogEA5ris0bu/2/vzOOaOtb/P4EEAgQIq+yi4MKqQq6CoKaIgjtqrVCr tagXQatg9ba3dakV9Yr6ldqgdfdXsVKVgmURFZBVAkKNsgiVJbLEsEMIEEhCfn+MnsaTsINb 5/2HL5gzZ44kZ5bnmWeeT7+fwJAZ9JcxQAgEAJ1FOOrahf1aFW8SdSXFX1dYPG0Q/Cdx1JcF SRU8J2MKXIRON1br29d0p6x1ja0OAMBSW3mKgWpieSsAIKOyzWuyNgBgtpm67Dm8xHLe8sla msqKFCWFPpa6FKWX7lwnY4qSIqEP86Wa122krqSkSJCOd+3tdhdTdSVFwlhNJScTSnIFD/Ty J5Q1CybqkBUIYJPD30Gbch+0fYYBfaw6UQEYqyvNHa8Bl5+nHtb6OepPM1AlKoDZZuouQz2L SSYSvp9j8vDVyhdHbbuwtKnLd5qeAgGM11KeO04jvRK/v44obRYQCEBHhUhSIJAUCPpqRAIA pc34TZDR4Nyj+s2O+uO1lBUIwEbv5Y6+3P4l9yWUxk5fVYEAgu48/zqxqqRRAAB42tDZ2Cn6 1E4HAGCqoTRJ57Udbi0ysaVLBABIe96mQlRgVvMBAC1dYukTawCAal63vhpJSZFAUVIYYMe8 V97qO1VPgQDUSArwGIWwR0J61SOkf+6XF3yhvtrL8cFWX8XbVsdW/+Vz9dWIfzUO7jva5KA3 zVAtIJ7dR7Cds6k6thkMAEgoa11lrQMAWD5Zu6i+g8MXLpuk5WCgCgDwsNDUVCaWNAgAADNN /w7jAvIelFDaumySlpIiYYI22UpXJbmCp6tKvLTU4kBaDfNVz93sqG9AISkpEnzsdIvqO0U9 wEJLecHre0+yd/3LSG3vbOP10WV1r1aNNCO1fzvoAwAWTaAaqZMUCGCNnW6zQMRu6ZL9j40g +HF8pLDWU4kokJOWpaJF8MUwDpKOLFa6Kj8vMr/8uAELux1VKlq6/ptUyVgwVomo8FejYO/9 6rZeFs4AgOC0moNupunrrflCcdCd5xy+EACwP63mR4+xqZ9b3S1vfSbTkW4UNY7TUk5YM7lH IrnIqu/twNLPuXU/Lhh7Z83kvBftP+XUelhQixvkezjulLUsmkC9s2by/tRqLHi4t9vr2oW/ fzKRSlYMyXzxV5MA9PInRD5tOr9kPH2sxrk/6zDfndwHZVa17XAyDJlnJpGAO2WtJ7O5AID7 bN6BtJqDbqaGFNJfjYKQB3KsWGlWWWsveHXaZObFQgCAhRY54wtrcY/kXjnvh7ReV7Vbblf8 QDf1cxzTIhCdzK6VHl9yNtpCp8K+lOq4Z80AgLufTYaBGpdY9Wfy6rCaHzbdYsmDar6lFhnu AXP5otJmwRsIigYA/FHSrEpUOLXQfIwaqaFDFPawtkPYI7d/yX0JpSmo6yxv7kpfb6NCIvzV KNhx5zm7tTsgnh38kUkAbUx9h+hIJke6vpaKIrSDs2v4NqefwMJWgVj7dUv3WZMgrbLt/jqr Xwsaf86tHUjHhP/VjC+sO4Q9vzxuYNbwn9Z3Pm/pvrxs/Ppb5VHFTd/NMq5pE2YMYDnIrObb 6qsmV/BmGFO2zTC4yKr/r6vxyWxudg1/hrE6zn/QN2Qi4RsXo7bunswvbAAAj2vbN8ZU4OpM 0Ca3CsTwCAPkcAbn6Dyz9PXW3HZh0J3nAICyJsHeOcZjNZWbBaKtt9nQfTLTVD3+2cvtVbkP +qOk2UpX5e5nk7tEkqA7z1u7xF+7GJpTlUPcX8asBcRXvOALLy8br61CfNYkgMHMyydrf2qn c7v07/XWhml6uLu+dTUyoCj9/slEAIAESGZeLJxmoLpwIvXsn3WiHsnZxeP1VIlVvO5NMeWK CkD2P4a1PHxeajZ8U7xaOlclAIDlZ+cTWSr9ugw8gaW6kmLGF9bQ7a5GUlgySQsAEFPSDN0j V1dYul4qGmAmitHDSlfl1xUWwemc0T6bhEAg3ll0VF6605UUCUfmmookYNc7tlM2BKx0VX77 2HLTH+UH3Ez33K/OruHPMKYc+MhkT3LV2SXjvV8f2IfPognUaQaqwemvLVYGQqzPpM+jy/qI cRkavlP1bPVVdrwbIUf90qsd7BdbcWbxOGOpODQAwHfJVQfdTPudg631VMqaBfpqxNU2ujNM KND/c3bJeGY1/7fChrJmgbWeilwf4Jvk6Dyz/yRWjdIZ5bIvp+CLEP8MLH56jC9CjCij3bmG lo7qjTGQF+xpQ+fFR/UH3EwttJThSJtdw7fQUj7gZnqpd0NcLgP/tAeS+UuWnI02+KIRYtmo ZU8byFcwcHqdg7Nr+LMvF0mXDDw0uqi+U1lR4YSH+SVW/c57ldDkVVdS3DBN74SHeTWvu2gw zpBRYvG1EnzRyDGyXxICgcAY2c7lPk5j10wjoiKBAMDT+s49KVUw0dX7Tmg2l9cl3jbDwEpX 5WlDp5WuSlt3T0RBo3TqnoEwsp82Qpa/fdH4KwgEAoFAIEaTl3aw5JL5a8UIBAKBQCBGGfnn UvLy8sLCwvLy8vAXBo9QKMzMzMzMzBQK8RGJiJGFzx/0FntPT097+8ikfkQgPniG0MXeOiUl JQ8ePKip6fUIAOLtIn8OZjKZc+bMYTKZ+AuDp7y8XE1NTU1Nrbwcn5sCMYJkZ2e7ubmFhITg L/SJn5/f/Pnzudx+guwQCMTQuthbR0dHRyAQVFVVPXz48A0YQrt37960adMQHjTkG9935M/B Tk5OqampTk5O+AuDRCAQ1NTU6Ovr6+vr19TUCAT4I60fKiwWi0aj0Wi06dOnL1iwIDg4mMcb qMTvhQsX4L2urq7e3t7R0dH4GvLQ0NDQ1tYeM6b/hN45OTk0Gm3Pnj0AAGNjYz09PTJ55PPs s1iss2fPlpb+nRauN9avX0+j0V686Oekb28kJCQwGAzcq9Xd3X3s2DFPT885c+b4+vpmZGTA ci6Xu2vXLnd3d09Pz7179zY0NAQEBNBoNBaLBV59MoGBgQAAWI6Rnp6Ofaeurq6rVq26cuVK t7zj3Y2NjTQabcmSJfDXkJAQGo0WHR0Nbw8ICMBqYg1CPvvsM+wSom98fX1pNNqKFSvgr3Q6 nUaj7dy58/Va8snKyqLRaN9//z3+AgAAgLCwMPh1NDQ0AAB27NhBo9HodDqQ6mJ1dXU0Gm39 +vUAgNu3b9NoNAaD8VorA2Djxo00Gq26uresdyOGrq6uSCSysLBQU1NjsVhDmOSEQiGLxYqO ji4oKMAKk5KSNm/eXFLyWmRrT09PZmZme3s7ifTyQE15eXliYqK0Afbs2bM7d+7k5ub29Pyd zkr6RrmPE4vFjx8/Tk5Orq//O6astLT0jz/+yMjIEIlenm6S2zgOPp8fFha2Z8+ewsJCWFJd XZ2ent7bYBUaGrpnz56BD+CDRX5ctKOjY15enqOjI/7CIHn+/Lm2tjb8PrS1tZ8/fz5p0sss rG+doKAgb2/vGTNm4C+MHObm5h9//PGdO3eio6OFQuH+/fvxNXpn9uzZFhYWUVFRwcHBOjo6 s2bNwtd4HSsrq4SEBHxpf/Q2Eg2fu3fvXr9+feLEiZaWlvhrI8qlS5fKysrggIhx5syZiIgI GxsbZ2dnJpN548YNFxeX9vb2devWNTU1eXl5kcnk7OzsV1kze+Wzzz4zNTUFANjZ2bHZbACA mZnZ0qVLk5KSfvzxx9ra2gGO+31gbm7u4+MDADA0NMRfe+epFLD/U7z1SdsjAIC9+rSQyQwz sjm+0ijA4/EIBEJlZSWfz29paeHz+QQCobV1BM4ZtrW1EQgEiURSVFQ0e/bskpISAoHA5/PF YjHWxerq3o80LDU1NcbGxtivenp6nZ2d5eXlgxqEGxoaAgMDi4uL4a+rV6/etWsXACAqKio3 N1dZ+TW93uLi4ra2tmXLlsFfz549e/bsWfjzxo0bN23atHPnzvT0dFhia2t79uxZJSUl6Rvl Pk4oFAYFBUG/rLKy8pEjR5ycnH744Yf4+HhYzdra+ueff/7uu+/kNi5NfHx8aGhoU1MTAIBO p9vY2ISGhoaHh8Or8+fPP3jwoPSwIBAIoqOjVVVV1dTUAADBwcHSRtGCBQsOHDiA/To05NvB w6GhoeHRo0d//vlnSkpKVVWVvr4+LNfX16+qqkpJSfnzzz8fPXoEl5lvkW3btp08eTI5ORl/ YeTQ19f39vaGw3R+fj4AoKqqyt/ff/bs2atXr8aMM7mbTDY2Nlu2bPnkk08AAJmZmR0dHfv3 73d3d1+8ePHFixclEkl3dzeNRvvmm2/OnDnj7u6emZmJre6rq6u3bdtGp9O9vLyuXLkikUgA AIWFhd7e3nQ6/dKlS9hTpG3QyMjIFStWzJkzZ+vWrRzO38ft169fP2/evNTUVE9Pz8zMzK+/ /nrp0qWzZs3y9/eHTuz169fT6fTr168vWLBg8eLFpaWlFy5cuH79OgBg586dO3bswJpisVhb t251d3d3d3ffv38/tnoFAISHh7u7u69atQouTjs6Og4dOuTh4TF//vzg4GC4ae3l5UWj0bq7 u5OSkmg0WkhISEBAQFlZGQCATqdfvXoVaw1+2qtXr968efOlS5dCQkIIBMKvv/7a1NT08ccf 7969e+fOndevX9fR0cFukQudTl+5cuXKlSup1JfJtgwMDNavX3/mzBkKhRIRETFk8x1DX18f PmLmzJn4a+82lQK2x8OZoh7RsjGrlo1ZJeoRuWTZVQrY+HqjAI/HGzduHACguLi4pKSESCSa mJi0tbUBee8YrqdIW0g7d+6k0WiPHj3CSlpbW9XV1XV0dIqKilpaWmpra8ePHw8A4PP5fRvQ AICoqChfX99Zs2YtWrQoIiICvLK5r127FhQUtH37dpFIdOzYMTc3tzVr1oz2BtCzZ88KCwsb GhoaGhowR5ehoeFg/ZFkMnncuHEXL16MjIwkk8m//fZbW1tbTU0Nk8l0cHAwNzeXrpyTkwMA +Ne//gUAqK6uPnfu3NSpU5OTkx0cHC5cuMDhcObNm8dgMBITEx0dHQsKCjBLF7tR7uPu3LnD ZDLXrl0bGxurrKwMu7OBgcG+ffvu3LljY2NTVFT08OHD3hrH6Onp+fXXXzU1NV1dXWFJfn5+ eHi4jY1NQkKCu7v7vXv3UlNTgdSYnJiYyOfzly1bpqioCACYN2/eV1999dVXX82ePRsAMBC/ Y7/0NQdLT5MDj9IqKChQVVXV0NCYOHHilClTMKcEiUSaMmXKxIkTNTQ0VFVVZT+gN8y4ceOC g4MjIiJiY2Px10aO7u7uzMxMAMCECRPEYvH27dtLSkoCAwNJJNKuXbs4HE54ePhHH310//59 /J0AAABUVVUBAIqKiseOHYuJifn000+nTp166tSpu3fvwgp5eXnnz58fP368gsLLr1IkEgUG BmZnZ3t7e2tra//4448JCQlCoXDXrl3l5eU+Pj5wSsZx//79w4dVam6QAAAgAElEQVQPKyoq rlixoqurS1PztWyrzc3NBw8ehKaARCJZunTpzJkzHz58+PPPP8MKfD4/Li6ORqNxudzo6Oi5 c+c6OzsDANatW/fvf/8ba4dKpaqpqa1du9bAwCAmJkbacH/27NnixYsrKiqgt+D48eO///67 k5OTs7NzdHT0iRMnsJrSbNq0CXaD77//3t3dHSu3trYGABw6dCg4OPjp06dwtQ5fORcXF1hH erV78+ZNBoMh6/bPzMyMjY3FFtcYqqqqVlZWAICiotfO0A+B+vr62NjY2NjY2tr+T96/U+x6 usVGzf5fVCcqkRo49psvTDfP0HTZ9XQLvt4owOPxTE1NKRRKUVFRcXGxpaUlmUyG3sLe3jHZ npKRkZGSkuLl5TVt2jTpliUSia2t7dOnT6E1Br9oOMH3jaqq6vjx4zds2AC3QrAwqEuXLmVn Z1taWkZFRUVERIwdO5ZOpzc34/VsRpCWlhYOh2NiYlJZWVleXo5NFSQSSVNTc1BvGoVCOXDg gL29/dixY+FKVEFBISoqCgCwfPny6OjoGzduYJVzcnKIRCL8PLOzsyUSiaurq4aGhqurq0Qi YTKZCxcudHJyIhAIQqFQWVl54sSXer3YjXIf9+DBAwCAm5ubgYGBnZ0dh8N5/vx5QEDAkiVL dHR04DpAUVGxt8YxFBQUjh49+uuvv2LOOehLd3Jy0tXVdXZ2lkgkBQUF0mNyZGQkgUBYsGDB hQsXMjIyZsyY4ePj4+PjU1tbq6CgsHLlSun2h4Z8XzQkNjYW2/ESCoVz5sxJTU3t10GtqanZ 0tICF4+ywJegvLwcN8q/FcaNG3fixIndu3dXVFR8+eWX+MvDJicnBxo3lpaWW7Zsyc/Pr6ys /Oijj5ydnXk8XnFx8YMHD3R1dXV0dDAzC6OwsPD06dORkZEAAGdn5507d+rq6i5YsIDD4SQk JKSkpHz00UcAgObm5hMnTsyaNSsrKwveWFBQwGazXVxcNm/ePHPmTF9f39u3b8NNLCcnJz8/ v5ycnNzcXOlnAQBgp/r6669pNBruEmThwoXbt28nEAgwJqW6ujopKQm6ZyH79+8XCoW3b99+ /vy5ubm5qalpVlaWvb395MmTsTrm5uZHjhwBAFCp1AMHDkjfvnfvXhMTk+Tk5PLy8rq6uvj4 eDU1NbhpnZSUlJCQ8N1332GVMWCnra2tpdPpFMrf+g2bN2+GTiTIZ599FhgYCIceuS9eb258 6DOwsrKS3QuA7XR2DjfbTEVFBTStQkNDR2RZ/cZ40JK20fTljBtfH/2FyeaM5pTo2uuv1xp5 Ojo6RCIRnCmLioo6OjqsrKyKiorgHNzbO4brKQKBICQkhEql4jp+a2trT0+Pra3t1atXi4uL zczMNDQ0YLl0Nbl4eHh4eHj09PSw2ezY2NjKype5EgUCwa1bt/T09DZv3gwA2LFjh52dXXZ2 NoxCGA1qa2t1dXX19PRaW1u7urq0tbWxS5qamo2NjWPHjpWqPiASExO5XK67u7uysnJMTIy6 urqbm9vKlSuJROKqVasAAN3d3Y8fP7a1tYWWA9y4VVdXBwDAzxB2QBaLtXHjRjKZfPLkSdhn cTfiHqempgad/7Ap+G9dXR2cYl68eJGUlKSvrz99+nS5jePA7fgYGBgAAAoLCxsbG//8808A gKKiIjYml5aW5ufnz5w5U1FR8fTp015eXtCAfvjwYUlJibu7+4jsH/U1B0vvseXl5Q0wSgsu IcvLy3ubhtlstrq6+oQJE/AX3gZqamrBwcG7d+/+6aefRnwaHj9+vIODw82bNx0dHc3MzOCy +v79+5jVW19f//HHH3t6er52GwAAgLS0tAcPHpiYmGzdutXKykokEjU0NGCRPlhgApVKxU0P 8JKuri4AAPpa6+rqYGEfQzx0PvfxSi1duhRuuR06dIjJZELnsFj8d0YhZWVl6V/lUlxcfPTo 0adPn8K1nbQvGrp6tLS0OBxOVVWVUCgcM2YMkUgEAFCpVC6XO6iYCDKZ/M033/j5+UVHR4eF hYWHh69YsUJPT6+srKylRY4E9/nz56dOnZqTkyMdM4WVS5dgwEHZyMgIfwEAAADmbJDrdZBm +vTpp06dwpe+b9R1cys72fjS0QG+CRKJxMbG5vbt2wKBgE6nl5SUCAQCoVBYVlYm9x3D9RQu l8vhcMzNzXEjdVtbG2y5paUlNTXVysoKvpkDsYMTEhLOnTtXVVUF3d1isRjeO3PmTD09PfCq b2Lbc6PHixcvoBUou+JUVVUdQiBYWVnZgQMHDAwMvv7664yMjMbGRmtr69jY2IaGBgqFkpKS QqfTHz9+3NXVBedCAAB0gMMxAX4LKioqAABjY+Pvv//+l19+2bJlC4PBoNFouBtxj5PbFCzp 6OjYsWOHSCQ6dOgQ3PqVbRxrUy7Ozs5Tp05lMpkeHh7QEDI1NfX09IRj8vHjxwEAGhoa0ESp qKgoLCy0sbG5cuUKAGDNmjWvtTVU+vJFS+Po6Lhly5Z+jWAAAIlEsre3p1Aocjc8uFyuqqqq tbU15qN+F4BRGPjSYaOrq7t9+3ZNTc3IyEgOhwP7npubW+4r/P39QS/7wf7+/kwm8+bNm15e XlQqlUgkUqlUaMLm5uaeP38eVoOvtTSwtzc2NmL/6uvrw3UodH/JjRiEd/XhpIIPOnfuXFJS 0qZNmzBneG9Aj5/0LAsA2L9//5MnT06fPn306FHpcvBqroL/4XHjxhGJxJaWFrFYLBKJWlpa yGQy/BMAAD09PdLNQpcy7kHNzc1dXV1aWlpffPEFdFU1NjZCixzu9wAAhnMwuqOjo6ioiEAg wF1JaTQ1NRUVFevr6+GuGzSG4JLoA2MmdXY+729L7tqLy8X8AnvK337dUQLOwWQy2dbWlsPh NDU1WVtbw+0GHo/X2zuG6ynm5uZubm5sNhuL64G0traSyWS4l5Gfn29lZYW1LF1NlpaWlr17 94pEotjYWBhkh4Ftx0IbDnbDfhesw6G7u7u3AZZEIsmN5++D6urqrVu3qqioMBgMLS0taJUW FRUdPnwYdk8Y0wT3dLGp1MTEBAAAZwH4r6mpqUAg0NPTW7x4saenp0gkgmFWuBtxj8M1Bcco ExMTgUCwY8eOioqKQ4cOwYWy3Mb7RlFR8eeffz516tSZM2c0NDRUVFTgZjafz5dIJPAvTUhI uHz5MgDg8ePHaWlp5eXlDx48sLW1tbOze72xIdKXHTwcxGIxLmQOoqSk1NXVhS99e7S3t+/Z s8fc3Hzbtm34ayOBiorK6tWrYXzg7t27TU1NU1NTT5w4oa6unpOTc/LkyRs3bpw8eTIkJAT6 luWiqKjo6ekZGxv7n//8Z9q0affu3du3b590xKM0NjY2JiYmTCbz/Pnz8C1ctGiRnZ0dmUxm MpmXLl26d+8e/h4AFi1a9PDhw71797q4uJSVlf3000+yszt4NW+Vl5fDcMfa2lq5CwjwyskT ERFRVlbm5+cHC9vb2yUSyaNHj7KzswEA0ocBfvzxR0NDQy6XO2XKFG1tbQ8Pj7i4uMOHD4vF YoFAsGLFCjjhVVdXnz59WnqD1sDAoLS09OjRoy4uLgsXLgQAiESiLVu2dHZ2wl03Nputra09 adKkcePGRUZG3rp1q729nUwm379//+TJk1g7cklJSYExX9iCura2Njw8PDExsb29fe3atXCM kIZIJDo7O2dkZAQGBhobG+fl5WlpaU2dOhX+sZWVlaGhoQAAJSUluE9RV1cHdxzU1NTkekTe WY5ahblk2REUCJaqkwAAhfzHj1pzM53z8fVGGmwOtrGxAQAQiUQLCws4z/F4vD7eMRybNm1K Tk4+c+aMh4cHtKLEYjGfzzcyMqJQKObm5mw228rKCsb38Xg8OIMCADQ0NJSVlSsqKphMJly8 pqenu7q69vT0dHZ2pqSkwHVeaWkpLgJ5xowZBQUFDAZj0qRJsNlRQlVVVSAQwM+ktbW1ra1N XV0d2sQCgQBbzg6Etra2zZs3Q49dVlZWVlYWnU6HsaIAABqNZmJiAk2Chw8fqqio2Nrawkuz Zs3S0tL6448/tLS0bt26RaVS7ezsVq1aNWnSJAsLi1u3bgEA7O3tcTfKfZyXl9fNmzfPnDmT n59fUFDg6uqqq6v71Vdf5ebmzpgxo7a29tq1ayYmJiEhIbjG7927x2AwfvjhhylTpsD/lVzE YvGPP/5YWVm5ffv2MWPGXLlyBY7JcFMDAJCXl+fn5+fl5eXv7//DDz8AAEbwJOFA7eDBwuPx pJ37GKqqqv2uKN8YFRUVO3bs+Oijj0ZpAoZ88sknZDI5Li7u+fPnJ0+enD59+q1bt6Kjoy0s LEQikZ6entz9YBy7du1avnx5QUHBxYsXdXR0pOOJcJBIpNDQUAcHh19++aW+vj4oKMjDw4NC oXz//ffa2to3btyYO3cu/h4AFi1aFBQURCQS4+PjSSQStEdl8fX1nTx58r179xobG/ft20cm k6G7WJYlS5Y4Ojo+ffo0LS0N8+Pt2rXLwMDgypUr9vb2n3/+OXavsrKylpZWdHS0ra3tvn37 YM0lS5akpqZmZmYuX74cntldtWoVlUpNTk7++OOPXz0H/Pvf/zYzM0tNTYWBGwAAIpG4Y8cO Y2PjuLg4uIHCYDBUVVWpVOqVK1dcXV2hL2HevHm4qE5ZwsPDDx8+fPjwYWzEfP78+alTp9ra 2gIDA3GOa4x9+/YtXLiwoqIiKSlp6tSpDAYDc3hyudzw8PDw8HBsP4LNZsNHhIWF/d3E+4AZ 2TzTOV9BohBdez269roKQTXTOf8NnE2CuwBkMllbWzs9PT0lJYVEImFzcG/vmCwTJkxwc3Pj crlwDQReOZxhU1evXk1PT3dwcMBaxm4kk8m+vr4kEikxMdHBwcHNza22traqqsrPz6+7u/uX X34JCAhwdnaWtXTXrl1Lp9Pz8/NLSkr6WHMPHx0dHfgfbmpqqq6u1tbWrq6uhgdyWlpaZB3U fdDY2AgN0Js3bx4/fvz48ePSDiRsIOLz+UVFRQ4ODtKdOiwszMTE5Ny5c8bGxmFhYYaGhr6+ vi9evLh69aqamto333wze/Zs3I1yHzd58uTg4OCOjo6IiAg6nQ7jJ2CIZXZ2NqxWU1Mj23hd XV1NTU1FRV9yvzt37vzyyy9bW1uPHTu2du1aAIDsmIzF8QEAFBUVJ02aNIJf30vNBuh2l+by 5cuLFy+W9qHl5eUxmUwnJ6d+PdJCoTAtLQ2ucYRCIfQejBkzBrpHnjx5Mnv27N5cJW+MioqK PXv2+Pr6urm54a8hEAMDM5Qx5syZM9pHohGIvoEnqWxsbEpLS+3t7alUaktLy5MnTywtLQsL C52dneXGKw2HJ0+efPnll5s2bRqsgTjkGwdCSkrKN998Ex8fLx2VhuPx48dkMnlQZ6ZHll4X ifPnz7979660p3HgodHQ1ycQCDgcTnt7O4z0KS4uVlNTMzIyIpPJ7e3t/Vp+o83Jkye//PLL Uc3RgfjgSUpKun37tnSJkZERmoMRbxcKhTJx4sRnz551dXXBkZZKpQoEgmfPnk2cOHHEJ2AA gL29fXJyMi4sYyAM+caB8OjRo3nz5vUxAQMA+nZTvwF6tYNlGZQdzGKx2tvbx40bZ2RkBE1e oVDI4XAqKirU1NSmTp361u1gBAKB+ICBbtipU6dSKBQ+n89iscaNG9dbHMkHSVNTE5lMlrsr +u4wiDkYgUAgEAjECDJaMVkIBAKBQCD6Rv4cPPDMlP0iRPrBb4rejgn1AdIPRiA+bJB+8DuO /DmYifSD3zeGJm6K9IMRiA8bpB/8jiM/Lnpk9YNhfqLi4uKxY8fCw3YfPDBtKQBAQUFBR0fH xcVl27ZtAzwaf+HChdOnTwMAyGSyiYmJt7e3l5cXvpIMg9IPDggIgKpbxsbG9fX1o/GlsFis nJwcNze3foOE169fX1BQEBMT00emzD5ISEgoLS2FSWKxwu7u7pMnTyYmJnZ2dlpYWPj6+sJE r1wu9/jx448ePSISidOnT9+2bdvevXtzcnKkc1W6urqGhoYGBATA9D0QmFYFfqdkMtnQ0HDp 0qWrV6+WFUdrbGz08PAwNDSMiYkBAISEhFy/fn337t3m5uYbN26UzkyJvSSQyZMnYxpqiL6Z MWMGPH2rq6vr4OCwbds2mBMGB4fDuXv3rrm5OZ1Ox1/7x4DpB9fX17NYrCHEwwqFwsLCQjab bWlpCTNpZGVlSSfvnDVrFkzpA2WADQ0NsUeUl5fDvMVY6uJnz56Vl5fr6Og4ODhg526lb5R9 HABALBYXFBQ0Njba2dnBvCgAgNLS0qKiIm1tbScnJ3i8WG7jOPh8/v/7f/+Py+V6e3tPmjRJ WjqPTCZDQSSMhoaGrVu3btiwYd68edLlI4j8ORjpB48I5kg/GOkH94c50g8eEgQCAYrR3r17 t7KyUu7y5enTpwwGw8fH5585B4+efvBPP/30119/wRIFBQVMhvW90w8eO3bst99+i12dOnUq bg6OiYkpLS3Ftvm8vLyks23v3r17IAZS38hfKQwHpB+MgfSDkX5wvyD94KGhoKCwevXqY8eO kcnk4uLixsZGXOdisVjwuMe1a9fmzJkD5PU+mNY/NTXV29sb5gT+YBhV/WB4Ceaby8nJwaY6 6Dp6j/SDIY6OjtJ5+CUSSUdHB/whKipKRUVl/vz5sOaGDRugfjBMFD8Qv2O/9DUHI/3g4YP0 g5F+cN8g/eDhQCKRoBOSQCDgOheZTIapl1xdXY8cOSK398FGDh48yOVy+01c+h4x2vrB8FJ1 dXVBQYH0SjrnfdMPhojF4r/++gtLeBcUFDR//vz6+vrs7GwOh+Ph4ZGfn3/hwoX29vYlS5b4 +Ph4eHhUV1ebmZkNf7sW9OaLhiD94GGSg/SD7ZF+cD8g/eCh0dPTc/369YyMDD6fb2trW1lZ ietcBQUFUNnG1NTUycmJxWLJ9j7YlJmZGYPBGI2oiLfFqOoHq6mpwRLohp0wYcLFixdVVFTe R/1gCIvF+vTTTwEAXl5eu3fvNjY2fv78ubKy8u+//w4Lb9y4ERcXt2TJEvi3X79+XSgUfvrp p/1uZg2EvuZgpB88TJB+MA6kHywL0g8eGhKJJCQkRFNTc+7cuUFBQU+ePAEynUta1QqO47gK 8AcPD48PaQIGo6wfDAD49ttvxWLx2LFjd+/ezWQyExISli9f/j7qB5PJ5FOnTmlpaamrq69b t+7WrVvr1q3btWsXAKC5uTktLY1CoRQXF8OhOz4+/vPPP+/q6rpx44aGhsbixYvxzQ2JvuZg aRwdHfu1gCEkEsne3r6wsJDL5coGK3Jf6Qfjyt8uhNHUD753715kZOSaNWsw/WDcCSI+ny+7 ZPP399+wYQP8WSwWE4lECoVy9+5dzBEEpzFZhcEh6wez2eza2treUtlJ6wcHBgYuXLgQ2yOR S2/6waWlpefPn29qaoIvOgb8/GX1gyUSydD0g1VVVaF+cFxcHJvNhvrBTCYzNTUVhl20t7dj K/rBMkD9YDKZ/MHrB/+L+nJd/sb0gyGKiopQnRACQxNwnQtOt/DdkNv7GAwGkNeJ3ndGVT8Y ADBp0iTYPuxTcD2KkwHuWz+4rq7u1KlTTCaTRqPhbpR9HNaUhYXFQPSDpRuHbfYGdIDDv8XI yKi5uZnH48H94MbGRpFIxOfz//e//8HKDAZj7dq1MTExra2t69evH6l120Dn4MEiRvrBAACk H4z0g5F+8BvB1tZWtnPBlzA9PV0ikWzZskW2Ar6VD4VR1Q+2t7f/73//O23aNH19/aioKCKR CN/k91E/OCcn57vvvlu8eHFnZ2dBQYGhoeGECROCgoJyc3OjoqKwPTs/P7+8vLz4+HgCgXD1 6lUikYgpKA+fvmKyhgPSD8ZA+sFIPxjzc3CRfvDoQCQSZTuXlZXVypUrW1tbMzIyysvLZSvg W/lQGFX9YGVlZR8fn5KSkmvXrhkaGh47dmzixInvqX6wvb29u7t7YmJiQkKCi4vLTz/9pKys bGxsrKenJ23mYuMt3EpfuHAhdt5n+PSq2YD0gxGIfkH6wYh3EKQfDBmIfvBbR74RA5B+MAIx AJB+MOIdhIL0gwEAA9MPfuv0agfLMig7GOkHIxAIxFsE6Qcj/WAEAoFAIBC9MloxWQgEAoFA IPpG/hw88MyU/SJE+sFvit6OCfUB0g9GIAbOELrYWwfpB7/jyJ+DmUg/+H0D6QcjEKPK0LrY WwfpB7/jyJ+DR1Y/WF9fX19ff7B6He81LBaLRqPRaLTp06cvWLAgODh44KeiL1y4AO91dXX1 9vaWFRKQy6D0g2k0GkzFLHsSbqRgsVhnz56VzsLRG9LaTUMgISGBwWDgXq3u7u5jx455enrO mTPH19cXk6jicrm7du1yd3f39PTcu3dvQ0NDQEAAjUZjsVjg1ScDzyLDcoz09HTsO3V1dV21 atWVK1fk5htqbGyk0WhYVtGQkBAajRYdHQ1vlz5SjDUIGY2zGR8qvr6+NBptxYoV8Fc6nU6j 0QaoI5mVlUV7pTAmS1hYGPw6oGLNjh07aDQanU4HUl2srq6ORqPBVL63b9+m0Wgw39ag2Lhx I41GG0LayMGC6QerqamxWKwhTHIwxjY6OhoT2snKyrorBZY1HcoAt7e3YyG35eXliYmJ0gbY s2fP7ty5k5ubK52wT/pG2ccBAMRi8ePHj5OTk7EMowCA0tLSP/74IyMjAwurlts4Dj6fHxYW tmfPHijRBgBoaWlhsVjPnj2TzZYoWzkpKWnz5s0lJSWvVxw68s8mIf3gEcEc6Qcj/eD+MEf6 wYOHx+MRCITKyko+n9/S0sLn82Eyc3y9wdPW1gYz1xYVFc2ePbukpIRAIPD5fCgaBrsYTD39 7oP0g/vVD7axsfntt99CQ0Ph0mTixIlhYWFY8jvZygAAmD9LbhbIoSHfDh4OSD8YA+kHI/3g fkH6wUOAx+PBTN3FxcUlJSVEItHExASmY5N9x3A9RdpC2rlzJ41Ge/ToEVbS2tqqrq6uo6NT VFTU0tJSW1sLtWdg1gusi8klKirK19d31qxZixYtioiIAK9s7mvXrgUFBW3fvl0kEh07dszN zW3NmjWjvQGE9IOlzWiIrH5wd3d3aGiopqZmQkLChg0b/vrrr7i4OAAAXHXJig3X1NQwmUwH B4d+U+wNnL7mYKQfPHyQfjDSD+4bpB88BHg8nqmpKYVCKSoqKi4utrS0JJPJcLunt3dMtqdk ZGSkpKR4eXlBvVusZYlEAsXfoDUGv2hs4ukDVVXV8ePHb9iwAW6FYGFQly5dys7OtrS0jIqK ioiIGDt2LJ1OhwIqowTSDx6gfjCJRFJTUxMIBG1tbWKxmEAgWFpawjE5LS1NVmwYarwuX748 Ojr6xo0bWPlwkO+LhiD94GGC9IORfnC/IP3gwdLR0SESieBMWVRU1NHRYWVlVVRUBOfg3t4x XE8RCAQhISFUKhXX8VtbW3t6emxtba9evVpcXGxmZgYVDgbi6Pbw8PDw8Ojp6WGz2bGxsVAy CwAgEAhu3bqlp6e3efNmAMCOHTvs7Oyys7NhFMJogPSDB6gfTCAQDh8+7O/vv3r16p6eHm9v bycnp5aWFjgm4yqLRKKYmBh1dXU3N7eVK1cSicRVq1ZJVxgafc3BSD94mCD9YBxIP1gWpB88 WOCbIJFIbGxsbt++LRAI6HR6SUmJQCAQCoVlZWVy3zFcT+FyuRwOx9zcHDdSt7W1wZZbWlpS U1OtrKzgmzkQOzghIeHcuXNVVVXQ3S0Wi+G9M2fOhIqisBuOYLr/3kD6wQPUD+7s7Dx+/Lie nt6WLVsiIiJ+++23yZMnL168WO6YnJGR0djYaG1tHRsb29DQQKFQUlJS6HQ6vt4g6csXLY2j o+OWLVv6NYLBK/1gCoUid8OD+0o/+J3KVQmjMPClwwbqB2tqakZGRnI4HEzBFO6j5Obm+vv7 g172g/39/ZlM5s2bN728vKhUKpFIpFKp0ITNzc09f/48rCarMDhk/WDwaqEqF2n94E2bNmHO 8N7oTT/4yZMnp0+fPnr0qHQ56F0/WCQSDU0/uKurC+oHQ1cV1A8GAKSmpsI6wzkYPUD9YADA B68fjP36xvSD4RxMJpNtbW05HE5TU5O1tTXcbuDxeL29Y7ieYm5u7ubmxmazsbgeSGtrK5lM hnsZ+fn5VlZWWMvS1WRpaWnZu3evSCSKjY2FQXYY2HYstOFgN+x3wToc3oB+8JQpU6hUKuxT Q9AP9vT0FIlEMMwKd6Ps46SbGoh+sHTjfcNkMktLS+l0+uLFiw8ePCiRSOCGlNwxGZrjRUVF hw8fhuNSeHg4vtLg6csOHg5IPxiC9IORfjDSDx5ZsDkYxqkSiUQLCws4z/F4vD7eMRybNm1K Tk4+c+aMh4cHtKLEYjGfzzcyMqJQKObm5mw228rKCsb38Xg8OIMCADQ0NJSVlSsqKphMJly8 pqenu7q69vT0dHZ2pqSkwHVeaWkpLgJ5xowZBQUFDAZj0qRJmBrmaID0g2Hj/eoHm5qaKigo JCUlmZiYPHnyBABgaWl55coVuWPyJ598gskG02g0ExMTzBYaDgO1gwcLD+kHvwLpByP9YMzh yUX6wcMGWl1kMllbWzs9PT0lJYVEImFzcG/vmCwTJkxwc3PjcrlwDQReOZxhU1evXk1PT3dw cMBaxm4kk8m+vr4kEikxMdHBwcHNza22traqqsrPz6+7u/uXX34JCAhwdnaWtXTXrl1Lp9Pz 8/NLSkr6WHMPH6QfPED9YEtLyyNHjujp6Z06daqgoMDb2zswMHAgY3IfI/Bg6VWzAekHIxD9 ghnKGEg/GPHWgSepkH5wCtIPRvrBiA8bpB+MeAehIP1gAADSD25H+sEIBALxlkD6wUg/GIFA IBAIRK+MVkwWAoFAIBCIvhn1OViI9IPfFL0dE+oDpB+MQKKEGngAAAUiSURBVHzYIP3gd5xR n4ORfvCbYWjipkg/GIH4sHnD+sGIwTK6c7AA6Qcj/eD+QPrBkNE4m/GhMmPGDPiheXp6fvvt t70tIjkczuXLl1NSUvAX/kkMXz8YANDe3p6VlfXgwQNpt1lDQ4O3t7fcnD8AgNDQ0D179mDj HgzUjX5dFRgB+jibNCK8y/rBbwBzpB+M9IP7wxzpBw8JAoEAxWjv3r1bWVkpN2vg06dPGQyG j4/P8JP6vo+MiH4wAKCmpsbf3x+mlDcyMjp16hRMHhkTE1NaWip3C0wgEERHR6uqqkKBB7mq wK/d8A9m5O3g90U/+A2A9IORfnC/IP3goaGgoLB69epjx46RyeTi4uLGxkZc52KxWPC4x7Vr 1+bMmQPk9T6Y1j81NdXb2/vy5cuvP+H9ZqT0gwEA586d43A4J06cOHnyJIfDOXfuHABAIpFE RUWpqKjMnz8fq4kNZYmJiXw+f9myZVC1gixPFRi76x/OyM/B74t+8JsB6Qcj/eC+QfrBw4FE IsEchwQCAde5yGQydO+7uroeOXJEbu+DjRw8eJDL5fabuPQ9YgT1gwEAWVlZysrKLi4uzs7O ZDIZDmjZ2dkcDsfDwyM/P//ChQvt7e35+fnu7u7/93//BwCIjIwkEAgLFiy4cOFCRkYGRZ4q MO4p/1hG3hf9HukHjzZIPxjpB/cL0g8eGj09PdevX8/IyODz+ba2tpWVlbjOVVBQYGdnBwAw NTV1cnJisViyvQ82ZWZmxmAwRiMq4m0xgvrBIpGosbFRR0cHzprq6ur19fXd3d2///47AMDL y+vGjRtxcXFLliyhUCg6Ojr6+vqlpaX5+fkzZ85UVFQ8ffq0l5eXq6srbE1WhBgx8nPw+6Uf PKog/WAcSD9YFqQfPDQkEklISIimpubcuXODgoKg6A2uc0mrWkHhOVwF+IOHh8eHNAGDEdUP JhKJSkpKWNcWiUREIrGjoyMtLY1CoRQXF8NhLT4+/vPPP4+LiwMAHD9+HACgoaEBV/YVFRWF hYU2NjY4VWAEZOTnYBKJZG9vX1hYyOVyoYCdNNxX+sG48g8SqB987969yMjINWvWYPrBuBNE fD5fNoOrv7//hg0b4M9isZhIJFIolLt372I+HDiNySoMDlk/mM1m19bW9pbKTlo/ODAwcOHC hdL7QLL0ph9cWlp6/vz5pqYmXFBGb/rBEolkaPrBqqqqUD84Li6OzWZD/WAmk5mamjp79mwA QHt7+5AX4wPUDyaTyR+8fvC/qE7w1zemHwxRVFSE6oQQGJqA61xwuoXvhtzex2AwgLxO9L4z svrBxsbGbDa7o6NDQUGBx+OZmpo2NjaKRCI+n/+///0P1mEwGGvXru3s7KRQKHC5g/mWHj9+ nJaWpqmpiVMFRkBGfg6GvC/6waMN0g9G+sFIP/gNYGtrK9u54EuYnp4ukUi2bNkiWwHfyofC COoHAwC8vLxOnDhx6NAhAoEgFouXLVtmYWGB7Wf5+fnl5eXFx8cXFRVt3Ljxk08+gftNAIC8 vDw/Pz8vL6/PPvvMx8cHpwrch9ftH8VobYy/F/rBbwakH4z0g5F+8GhDJBJlO5eVldXKlStb W1szMjLKy8tlK+Bb+VAYQf1gAICPj8/nn3+em5v78OHDdevWffrpp9JXsbGIQqHo6upKb3hh Tju5qsBYtX84o6LZ8F7oByMQwwfpByPeQd68fjBiyMg3YoYJ9PW94/rBCMTwQfrBiHcQyhvX D0YMmf8PEkQERYj7gvMAAAAASUVORK5CYII= --------------uTJw8bsCB2aEs2Uc9pYw0k17-- --------------lXHHD9Fjd0rtSvEw84v6ov7s-- --------------04FClYkbcFIyYTJjfibmXM2v Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsF5BAABCAAjFiEEWT/lssMHB+28ly8Kt2dIb0oY1AsFAmQgCWoFAwAAAAAACgkQt2dIb0oY1Au7 DA//QNS4HoY60SY2NgHYok0n7dp0NCbv+sjieri6n0pfxwSN9Q5MLhND+dLigoUKu6ltiSEI4Ut2 XhAb73JsMy/J5MHTHqBo8t9dz6Xi1tbVoFpgnclB3BzaU8UFLENSs92UD1OktZjnhPc7bxA63D24 lOkfZnwv/nkWn64j3MghY10pqbJjrJchjY5yvIAfkVAnKMnEjj3oezS2AGZe+f+AWFGnVltBGaCN gsGVRoKpa/uNmQLQ7TVThrUqYbotg9pAIqbJv3bgxZQ/YuxluixnKz2mBsiDOxexKYyqMk/+41hx Phc0ll/beY2axDPGcpfKpvrmhH8ijVZuA4BU0G0W7GZwJ1tLUsIBNMOB3UCQWoi+zHjpmbt4IJk6 mX0a8eV7eyL3TUM8c6cPxNQrjNSIW0hGCYqwy09Qjb/4tJ3RclusRtovjcLaqePIrfCJBN6Cnhy4 VdZfeycXPtjh/CUyhSJrqYYh/id4heLIH/xxGacNnj6NHcDKQ1M63ID7irU7R93rYow+TWKh1Foj DuQjBQxQaNhJe6WAN7Ij0iHykO04rfb30PS1WBV39yb7VriiluARRMXle31ItABmclSCLYUmmVP+ mGQgrFyRbrM71koR6K0j37bdJWh9wCyzIvUAwjrBAmbh/pFcXMaBwalWwOEhyV4LonB4eSwk9hpA swc= =szA8 -----END PGP SIGNATURE----- --------------04FClYkbcFIyYTJjfibmXM2v-- From nobody Sun Mar 26 17:06:02 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pl2R22hhQz424St for ; Sun, 26 Mar 2023 17:06:22 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic315-54.consmr.mail.gq1.yahoo.com (sonic315-54.consmr.mail.gq1.yahoo.com [98.137.65.30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pl2R06Ddlz3jhG for ; Sun, 26 Mar 2023 17:06:20 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b="FCTde/UC"; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.30 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679850379; bh=MdGaDp+yyvLbpOxae0NxeyWETPfw/gwgSPZ2y06SMxY=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=FCTde/UCPR2Dc2rzdv6gD6Um/PekzGfY1Z19vFNNRGNLaqZ+EHl7De9SYC6w8ESL6ZtjSyNj4+y5By3H6Kge9nW8GiYE1eu6VtASHBdK8aDhuIBUHqfSxhE+l593Z1PD20Fx7OgE4E1kQozDg2C5t//9BKPK0ySw2OwYdYpuNl9GItTQkKS7EDzYtpPYjdrFx7bC+FtPTstdzsO63HfkpzyOqkrHoCa9SPoFsHtdnnbxp9qkk7XqhErbt/OAKnQKo4W9LDLeT6wq3DlhNf+6e6+zt5QAQiL3eNkeOAv03pSX7Ugwl+2anXuL/mHG7SYAHkh5OFHDUSCd5lW/Ch9Y1Q== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679850379; bh=7eJNqTzugicV0dgtpeMyUHg9G0TGzXrEh4/+3QxqEJG=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=VEQjbW3SkoV1cTRPDMuV/+Q7fo7UIFzJTJ8YOcOFPwMtGikxgp4wNTzgoP6LVL3cu1VjtJPeeQ6vQXuICSENqy2juPJh5lzagip9d7HQfydA5oACDmJSssWu87JzSXKjC0MSdryD3oUI6j9LU7gnz3nLkRdesiNqnj0erswrr+X3TQwOGjZ7hm46KQC699H3E+DUvFYQzoPFI4/mnWjOEPRprX+MD0NWsgeoquo9xtvuelOm6GjsdO0I8kt+jowS1PCf6ebHcOT4iUdY3QveK7AhnHA5EtvvukqybdEqUa/35UGgQfyenC1FbhswRoUJuKXW+r/0kNIbbUcU523UHw== X-YMail-OSG: tYlaqwkVM1m7.2sxFOIfyOayzdgV16ja9j._LP0wQ3jrg7hmFuvzOFON3CGkIB3 QzPt0ZIa3WPKrkOkkSAL2DT.T0DSEoSNwZlPDwMpAHyyckdbcrXWkeSQeng3DJqIeRFVOfOgG0_l q8LMc.6VyG7C0_Vf6neCKxfnKbWGxCTvT2ulvrPSEYhzgzKELtgylkim.k3qFgwFXnnbQsWGGzeK 9jLnb2JUhw6HnSGtKPJQPPlj3pcE7cqTQ8A.k_jTe47ZkshMVgNCj_K.nKZSV9OclkN7A7Xvmqwa uJDya4WiJn4ZqYPBzjof2a8ogvGjzk2fRvD90QipfjKPJn_JuFhf5t_wfsgtc9K8EHfc5VU1j9Vr TeAEMEB5Bt05qqfDeqdp_md46AJiY82LhKKKS6pNNzYfuexyOdgVHO3PULreASLTd3.Q3Cadll9e NCraK3_4jydKbZ_fesekQpWJaejQbvvRJRennyZFH23MSk8Ht5H7_n1h_CEA.s516JnAOquCneNz yDKVt0NI3I565FlvCbE1O0mNBF7bb8_7v8n9291zLYwcfY2CAWKKI0O82_bN.3IPayfMWsOIfT72 Cnaxd4evhyELFvt0ybwuu6Pc2R6CxjmM7ky3saui9N7QumvQR7uoTbb8PF2ryZHqRdVtn8E16xk1 7PjOS1mQ4Agw1kka1KC85mLuKeryuFt6bBhIBKEIPJ2TILu4CIadHDon1K7ka6MwgvpXtILOGlKb DDFl2H5h98RQTS1Z_Y5l2n1AtTAvDcquy1PDFhqihf8HP71xiswUDu9dXWacWLvW30ouQcC7uh2v .Rr1NGaiGVmSAHRofKsIil3Yt.tFmRBnqOM55SqGkDnBP_hsoiI25yhdGMrS2xwRaYZBgOp5bmZQ a23jhi74G4XM7pd5DgleYFPcpe7pVLIzH8SBNDaqAP9P_ZzSmq0s1z8bcPskOuy_kRo9x4qIR3OT ddaN7iHd84EAN3Mcq8sus18agsqb0uztMWX6t2T00yM8IVt91kBB1YsqQFeaJqM.13Yvqp3ykC6X 7kKhq4Dj5ZzDkHtVxjecDHH8txF28M_VjGdVQs0v4rmmXF4wCTrFlF5jUn0VDOfBhZuCV8XRLQPH P6RZqclp3GSoXfo05lNrktR6TCAoJa5OJR1r96zwsJR3YQGYwJBJxxhm3DqId.WlbYNgBEJA0lLs Tss75G6Dw0TRkKg_YGPjHqq2m_XWsI7iI8PZPul3ZHAPkqBm6QkAABm8vPcH2eMUweT7dLYCCjCE WrdkQM_XdoPdnRFPByx_O_WIBNp3lTaWSH_OL4v97xUto0VqsVrx0k5FiuevFzLjgNj9tGtSau42 wFbeVMGfZqhCiYKznq.xLY29VsKVGpCgCzxcszcUnHR.zRIIw0I4OQVhaY4l..kt99CFHV6X.ML3 xZU_CYIMKMnrRIln6jkR4vsbTIrv3y3FCoIaMEw.ZP8qyELg_0_aOcHgghPCIgKpj8mF_jXtgar7 Ka1p5zWHPz5Wu6Lu8dnFMAWrtBYUJ3hSVzVeM4C_TrrHfTiuWg3xF975E6X.YllXyfqhGD2eg1tW eNLJD5xqlP3IJBmT3MdU0Q8nXBKoQ8W5zOZv8vmtzjJhosu46hkIbP9X5aYjMFDjM4bKiZsZEACg 8IY00BVeeY4guAJj7ZIGpm8fwKFMsD7lebC9bo7hzuHDFUT9r58zOKCGaIGHCutIrw3i5ir.aS2v ewfDMDnR3ENexDmaJHBGVD.SK4bi7U8OAU0fT7JVfduMjVFeLnHubgeaHzfSsAZVTftVQcHG8INU pwELKHFM0b81gYtFV0Wc1jB7aFSyz1p13npEDdQiTDAPd81r7xswjMivJDYsUnS6DhjH2t8Db0HD lXO5nvA2VJxtHYZ5IxN3kROB2oYE2k3EQXVknvmC4h2sYWJzZixkPVQB1O.NDuYCO3ZyMYDnZgDx 8zgL1jlugmEqmEv5zjy8RYkrCPEtzBo3A0sYM3b9CQwOtHcYSXZNd_ArKM2_WFcZZ8h5ssjBhHmF CxUEJnm3S6Ekegc0BoImASQlpY5W3YyrBV47ng4wCqY3geDLhYmnMm4uaqYuypEcnvWbV7TaLE4V IQDulT4NY.GsWt0.9R6Vf9cu7fjzHJncG44egGfw0DRtWFUXtbeB34gEfOTw6Aek5qexf7kHigM0 KLJbjUzaU6Dz3QDhibIgbeyIS_iBT.JKu.TM65yP9IWenjm6T0ViJStGAcbQqwpDAGEFyk7Jgiov OWP1rUanxLATcNjUPO2Ujd3H0GJb4Xny08V4L2oB1S79xujbbMfO1U_huYymqzRdOz8Ip6UEfoFV UkA-- X-Sonic-MF: X-Sonic-ID: 2e44da7b-f5b1-4bd1-9130-14ef43fb9fdc Received: from sonic.gate.mail.ne1.yahoo.com by sonic315.consmr.mail.gq1.yahoo.com with HTTP; Sun, 26 Mar 2023 17:06:19 +0000 Received: by hermes--production-bf1-777648578f-7gmg8 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID dbb9f6f37342bb5c3cd902a8b92de74d; Sun, 26 Mar 2023 17:06:15 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: RE: User-Agent: and In-Reply-To: Message-Id: Date: Sun, 26 Mar 2023 10:06:02 -0700 To: Graham Perrin , FreeBSD Hackers X-Mailer: Apple Mail (2.3731.400.51.1.1) References: X-Spamd-Result: default: False [-2.92 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.98)[-0.984]; NEURAL_HAM_SHORT(-0.93)[-0.933]; SUBJECT_ENDS_SPACES(0.50)[]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.30:from]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4Pl2R06Ddlz3jhG X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N Graham Perrin wrote on Date: Sun, 26 Mar 2023 08:59:22 UTC : > Not to start a long debate (or rant), I'm curious. >=20 > Mark, Peter, please: what do you use to send email? >=20 > A recent discussion is split across twelve threads; screenshot = attached. >=20 > I looked at headers for a handful of the splits. If I'm not mistaken:=20= > neither a *User-Agent*: field, nor an *In-Reply-To*: field. My subscription to this list is via: Subscribe without receiving mails: = freebsd-hackers+subscribe-nomail@freebsd.org and I read the list via a web browser, not via an E-mail client. (For example, I've not received your query via E-mail so far and I read your query in a web browser.) This ends up meaning that some of my notes to the list are from constructing the note from a web page that I'm viewing. (Like this note.) Other times I've been TO'd or CC'd and have received it in E-mail as well and I've happened to reply to that. Thus a mix occurs overall. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Sun Mar 26 17:36:38 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pl36J2Fk1z425vM for ; Sun, 26 Mar 2023 17:36:56 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic311-24.consmr.mail.gq1.yahoo.com (sonic311-24.consmr.mail.gq1.yahoo.com [98.137.65.205]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pl36G6NvVz3mnF for ; Sun, 26 Mar 2023 17:36:54 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=RpHkTstP; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.205 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679852213; bh=oZ0Qvenw0JdKrDgYzcM4EiGdeIcFmH8Ml6tzHHT40mc=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=RpHkTstPeg/mU9HG5ukVwkPPM/beVbH5wQ8S2RCjbSHwqu/Wj0XoYj9pUmNMoHMdAFT1iB1SN+6vcTGv7e9gm1NHuDgjjQG5MpQMhN5X1HVPbc85zfy9R1zPiEFfsntgYcPicb+L9Xu73e9xzi7qf7M1UnU2hsHedatc+URThR/DI4vo4nkq7U7jEBjVytIV28Y7e8SmoJFbzx1dgPtcnP5/tb+CefbiFg7AcINgLe7Vig1AH5kaY75N+nhVINSdT/YeAp/p/fNMLsnl0/X8SGltMEq3gargM2tCkmRdWbZMIR/1EGeSv3oQ9OEhIZi9Z0pPV+kJiLSQwxNkM9MOXg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679852213; bh=rNuPNte76BFauK64wRk0w/gvPCr2qaRyZdp9O2sNR4u=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=aQd2NbDQq3Cbg3DOGMXUMT7oJw0drVaMX7DmKrYokPfh+HNDFYcHo2YzXDovedxHEhPDEyplhsN303qlkWuUPaizqC6BbfM4senYacmHlge539fF7XDnTPLq1thhqEJP4L535Aum++LZS4OJNfSp71AFsJTQj1aEt8qljFl9xtG68vH7bGUxd2t8FI+XJ2XZTsbFOL5zfBDvmX4quxz+/NLNmNUPWa0N9utyAr1jTr+NTe0rSQgb6ooLbB7Lk7QLs7K/ia9Hq8ucQ7lyFEkKDo9zs3bgZvqyyIGlFN1n7sorSBK+dFZwrwCQtlGyGkr94zsg/EMecRvue3rXPzfS9g== X-YMail-OSG: prE97NgVM1nqUUg6K58eoA3SkgxD.BolELfOO7xA0OI0eipUFPI4Tvu7qgk9D__ 6SLOXYAmk1HKvGeuVim7lKd4QgQs2BedYD3sf2VB6reNgV913VR5cDhvGnbbS5YEiwE3Q4BH568F UNWonJUUPfbuoVHoHAwXBNXj24q0QUJ6isMzIkgHr6cF66LM02gVmlg.PuXdu9EDKzWRuMbP0osM bI.OtTYn8pZCAp1aIZ1qkA4.ys4kZ3_sfBus1_HcIhVPs.yLx4JT0dQ1dFF7lQMldY5sVgKE23DN qWcoF_EXfTLr83w7QU65szo0nvYfxUC5R3rJdVgCIuuERbo1yzgwh34aP7SE2KU79LnD86Bim7H9 2P20sTHjFQWrLUXl9RNtlpEuGhIBAmW_JDKUXUH3Yz6Vp82sRwgp_NzotMN4riT7t1_BTJmSnM1b 5SMblN_e0BKT97D9XSmL6mGGEg4JPoPyBRsdp4jDTOrFlz9AhObf35HwZihhXZq74OTrdCNopUXI G7d4j93CnmdZIH5M0y5BrCpzHALQpGbEA0VXq31IDp.F45ml7ynDhmSySTdzFIl1OeLhvqnvIxvo i_vMFkTkUQqAcMPa.G2X_ABEGPmmPLpsh4peBI1qyqwjWZiWanOLmN8eXNSL0fNqetxrDVHWZI7h Fmg_1uXISE42GLyGYB.NVeCxh2fxUunVJ_2_2kVMKC7HaiA2yjxqCWNT7G2kuvzt.lU8gXjQHWno aLAYMR1j2mXm.XbD.n5mVrkJVzIp3iPZMy3mvsbhCi94X0e1B6P99sG.FT0rvR81a.rtsHH5ESan KbzUYC3LjL9ePKLvzBneyzwspxxsLZCcUt6gv0STcqxYe4NAR4pAZGwHeHluIoVNLyQKMxqR6w7q rBFBUAUnQPzAhqFn5CqI8Ax.X0JXaztsXhukEiZzMPf3ndZ8ZzQD7ug_6yAhOZxQ_A1_bHR2GR4N QlQXeEve7Tk86kkLxL3Rp6MgWQ6ARg8As0J2HgOwbHQZsNRQlS_bSvfzvgBKsQZHEMPItgmu8WhG c49Lcnebx6_bE6P4ex3JpkSrcACPscdReBBnJwgXJvzi7z4rMMUVB2VmtBv988HmPdHnJey9EjuY dziSYUcI7s18VSf9j6KSJ7PHGd6BF23P2h_r9Nad7P1A6oJnuUks8OZQv5In3jWWSAOWa7mmaxBy uikeTSNDLl6106abyp_X3fc1Q4UdLv8TCBABn.O78Fcj_dXqXsyiyMh59hgNbNkmwbHf0V7QoQxH 5lqfGMaSIE4FRo9b80iPQZ4KwuGSS5T_APV5myUJwnUtVeC.NCKCnYF2FIBryMHtVfd24k6CoW9H cicAlvUEKPdB2N10GctyBKBMtw3quGHoszpCaq646.lOhywSAh0NAjoSi6Nzhl8yckD_HKeBeFNh wog_flajuMCZf0VhC.zSvpDkq3Qa1yaMHqVx9VwmMqehMLcUbAGISTh3MPqWk1l.Q2smQkHFB2ok 7uRB2UJ3oFHLoxTLssHEUU4kTdEbctdLyNauq2uXQZo0MvcumolAxLUFoc5tiNhs79kozGAxDje0 8cVh8nUT2nzNrki.7TIFGlMSrm9oS1L7zzRRWb_fsmd4CHwVYdS5MyHhCc_ABVoUUBp.Nv7M49x9 XIbas9nBrMmZVmzPGHjMfjg1Hf.jnIT9uzOaHQ4TqygB0E6lQqhovvSkFy2J054RAamRMHRYUmlm VUDXglMUskctKReeNHIYFbOjp_AX7u4Xd0FiHpjnXnPWPPwsAu7JwkWChuZCY7pJGaUAdSG72iWX yqdLc32PQkJSy8Z_WXAUJbtcHBzSdgSAWGrey1Pl37IL9tFSnZ0bGST33rpUO.2itXNnvuVKr8fw 9P540uQbcPSwGFE.IMwFGSNisvsImNrzkwUEZpoiG8JPfhA8k9IlUZ0bF2tmzau5bMwfxUbbioiF kKDyOHOETzoXVVayVl5Sm9dwvKxPgO8jLmUQZneWBTcFuridHQK6JW52CC4EITOThrI0LTyclukh 5_Unf77T9rHpHgku6LN7crMr11vTyRiIFCPlnheHXaOmFkFXM0UTbe55ddk4hfmQOoMdrNglXODV vys93vMcyFSmwPfCMONFLpAUoG.UdSWbh9617Tvh.ufzcM.m4vsF.Ig8SjL9l8V2IutAcEqCGANS t3tPMa93EpI2LXHsTAlwf2BtVzZw8iYJ2v0.AZYnn_NkPED6n92RFxRy7ogtTx0L2O4qY4cle4ig ue1xxcAzYWPHIXinJBkTkxqd7YFNtaqIUTzg2PUVlox.1uwTfXD2CXA6aTrBx.S7gbBCuadDNfaS NEw-- X-Sonic-MF: X-Sonic-ID: 193924e3-2cdd-45fb-8c32-4986c9306f2e Received: from sonic.gate.mail.ne1.yahoo.com by sonic311.consmr.mail.gq1.yahoo.com with HTTP; Sun, 26 Mar 2023 17:36:53 +0000 Received: by hermes--production-ne1-759c9b8c64-7lgm5 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 7c341a3200bf102a5dad382dac103411; Sun, 26 Mar 2023 17:36:50 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: User-Agent: and In-Reply-To: Date: Sun, 26 Mar 2023 10:36:38 -0700 References: To: Graham Perrin , FreeBSD Hackers In-Reply-To: Message-Id: <4319B7B4-46BB-42F3-BEE4-7FC7A64E2BD7@yahoo.com> X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-3.42 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.98)[-0.979]; NEURAL_HAM_SHORT(-0.94)[-0.945]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.205:from]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4Pl36G6NvVz3mnF X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On Mar 26, 2023, at 10:06, Mark Millard wrote: > Graham Perrin wrote on > Date: Sun, 26 Mar 2023 08:59:22 UTC : >=20 >> Not to start a long debate (or rant), I'm curious. >>=20 >> Mark, Peter, please: what do you use to send email? >>=20 >> A recent discussion is split across twelve threads; screenshot = attached. >>=20 >> I looked at headers for a handful of the splits. If I'm not mistaken:=20= >> neither a *User-Agent*: field, nor an *In-Reply-To*: field. >=20 > My subscription to this list is via: >=20 > Subscribe without receiving mails: = freebsd-hackers+subscribe-nomail@freebsd.org >=20 > and I read the list via a web browser, not via an > E-mail client. (For example, I've not received your > query via E-mail so far and I read your query in > a web browser.) >=20 > This ends up meaning that some of my notes to the list > are from constructing the note from a web page that > I'm viewing. (Like this note.) Other times I've been > TO'd or CC'd and have received it in E-mail as well > and I've happened to reply to that. Thus a mix occurs > overall. Looks like my answer covered In-Reply-To but not User-Agent. It appears that the analogy(s) for User-Agent for my E-mail client might be one or both of: Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) X-Mailer: Apple Mail (2.3731.400.51.1.1) I'll also note that I've always read the list via the likes of: https://lists.freebsd.org/archives/freebsd-hackers/2023-March/date.html So I'd not seen the "by thread" mess. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Sun Mar 26 17:41:41 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pl3D15Dddz426G6 for ; Sun, 26 Mar 2023 17:41:53 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pl3D13yhRz3nwK; Sun, 26 Mar 2023 17:41:53 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679852513; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=/EyYqisfBkkevEaHsOWJSPRRb872yfZy2WsTuC+Yj0o=; b=FotTcwqKQgHzRyuu4k/nTpqBzP++6TbBUH6MVXnZiA5B/Ie9TQMp4Ih7ssykV7ILuh/cm5 3tA88hUYBxvTcHVwpAcWvDQDsQFEvShRIjj4TJXHUnhGSZ/ot9mfec2357H4EsOBqDwDph tDjpNnvyCw4elr1VqdhUhUj9s6mT8uwcy+BjbfJQbVD9KZ0dDejn27pnebx/vkchu1DHDX lhMLLrLRe8kFuTdFWd7bqGY62Wayh1hckiWjtQKt4XAI6x68Dh/Mar8bxcYj2RI61+C3k9 jMAKetKNJYqL0Tvv3WFGTp2Zc/7XWkdCaynqKQHyGqttXuKP6oBpEqCkCShk4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679852513; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=/EyYqisfBkkevEaHsOWJSPRRb872yfZy2WsTuC+Yj0o=; b=j6fzT/gxWpN/C3wZX0Z9Fsu+4rBTSvl1Ovw0gat5iTTMhfj6gpZiLVpwNQup74CQoGUJa5 0EDw1vY4WsvGd5w4WGLzdmIh5/fzXG78uKEqIdPTYemDsFZo+AKHEtojkQ0M3D1ojm8oeM S2+Kg0DjCUv/FPy53gQOm/Yi8Hjr94dXqBj+Nhekt3pIaGQq5WRqT0eG4lHut9Tvs88dAJ CojBfj2uJkJn9d3nU4IUGMfqElSVAMu3kf/NkCyV/YQZiBuCCW8yaSQhmLoWYnPZqr2xJy 5NSNCIrqRjjTU1e6nkLi8yX1azGbZhTzQW8o8p1a8wdVSJ9Vzjwb0i/1EI/3gA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679852513; a=rsa-sha256; cv=none; b=KsIhcHoVysMJh9b5aPP2MH2m0PFZ1330/dIzO+DK8hNtxO1yf4XajZwqeRuZYlAY6TG/cW c24nFVo5sb7p08q6r8zEX8/DxaT7D5CHMAPNytFpVM6yCzdFhKXjAi273XkurNlVjiuKd9 QhnLIp+QRcsI4atKsne86qQbaacicuaiqXPcarWFj/bt1e8V1EmRxK0LA6i2mrfqph2pko IfIC2r7/aeiZNdbfBUUiBfhxvMLTgHHRjFjs3lMlP/T8+GfkaWXmuSGgONAFl+Km6NlPWh t+apkHlfbK7Lg4cH/sAmdppldgYfM6kR1xEAumuSgXupRPcECbdjfdHAVcgbtA== Received: from [IPV6:2001:470:1f1c:a0::2] (tunnel642390-pt.tunnel.tserv1.lon2.ipv6.he.net [IPv6:2001:470:1f1c:a0::2]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: grahamperrin) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Pl3D01t0tz13wS; Sun, 26 Mar 2023 17:41:51 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) Message-ID: Date: Sun, 26 Mar 2023 18:41:41 +0100 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 Subject: Re: User-Agent: and In-Reply-To: To: Mark Millard References: Content-Language: en-US X-Priority: 5 (Lowest) Cc: FreeBSD Hackers From: Graham Perrin Organization: FreeBSD In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------tq5rRbqahfNAHtXGB9ag4CUA" X-ThisMailContainsUnwantedMimeParts: N This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------tq5rRbqahfNAHtXGB9ag4CUA Content-Type: multipart/mixed; boundary="------------sIrtuBmvlamfEfKyZOaebvyF"; protected-headers="v1" From: Graham Perrin To: Mark Millard Cc: FreeBSD Hackers Message-ID: Subject: Re: User-Agent: and In-Reply-To: References: In-Reply-To: --------------sIrtuBmvlamfEfKyZOaebvyF Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 T24gMjYvMDMvMjAyMyAxODowNiwgTWFyayBNaWxsYXJkIHdyb3RlOg0KPiDigKYgU3Vic2Ny aWJlIHdpdGhvdXQgcmVjZWl2aW5nIG1haWxzOmZyZWVic2QtaGFja2VycytzdWJzY3JpYmUt bm9tYWlsQGZyZWVic2Qub3JnDQpPSywgSSB3b25kZXJlZCB3aGV0aGVyIGRpZ2VzdCAob3Ig c29tZXRoaW5nIGxpa2UgdGhhdCkgd2FzIHRoZSANCmV4cGxhbmF0aW9uLiBNYXJrLCB0aGFu a3MgZm9yIGNvbmZpcm1pbmcuDQo= --------------sIrtuBmvlamfEfKyZOaebvyF-- --------------tq5rRbqahfNAHtXGB9ag4CUA Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsF5BAABCAAjFiEEWT/lssMHB+28ly8Kt2dIb0oY1AsFAmQgg9YFAwAAAAAACgkQt2dIb0oY1AuT 8Q/6AmX1OTLEj73HbU9L15LuP277kiOJYCCuwcq24nfFmVNLt0YN4VnIKHrjKsCHb64ARh1ZUNF4 /ZbDsHZtPoTErZ3lbrIftWnqt4dfPKew78R48L+XEFWwAHCWW8nxBHaIFE2Fs436/0NqCrv2i7Tf lfshiMoF7+5tsqaiGz60mEdv+t0k5xdceN2F4WtAnlKVtCmLjweKOnugfbLn2MkgB3I+65bX5WFM TbD/K6bIWDyRPtzgY7bocL9FsHhRkXcpnuV/IFjP/1eyBEmTfi989X6w2UJIf2wHSNSKp5ETFQkp up5EcTZpwDBodvRw7KPWCJcnvYs5Ijj7I/pwRolodnC7uDvaDxAo1mAh0arQ/7uI96WnFAQDOWnW 1tzRnnqi2b2CB+4sblQC8wPdRu89oWPE30eAdnZSylpzyGwUpQAYZM4i9Vs3gztQ6Bht6vCTFHok JAQaUC8o+skGvyNfViawc2l3INlOTrfE1jhNDop2DO8pRgtodzmlxlic2zsdPkaAPdyHvy+W5vvN 1K5hfFdb3hmDJ7u4+mSQnhD0ZBxO1k7KGHmOgxJhcmkLlyeLGDRwxT7YNNGUYRaGIDNvCwFYAye6 WnzBwsgKU8sn4+HPTBZFxmxD585FhOcj6pYeUhGZ3M+MwLPYpAItowNbVusjDcFIW7U7OH6cxbfC vkc= =KarU -----END PGP SIGNATURE----- --------------tq5rRbqahfNAHtXGB9ag4CUA-- From nobody Sun Mar 26 18:24:07 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pl48x1HhYz428dQ for ; Sun, 26 Mar 2023 18:24:17 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Received: from mailhost.m5p.com (mailhost.m5p.com [74.104.188.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "m5p.com", Issuer "R3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pl48w1HnJz3sJp for ; Sun, 26 Mar 2023 18:24:16 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of george+freebsd@m5p.com designates 74.104.188.4 as permitted sender) smtp.mailfrom=george+freebsd@m5p.com; dmarc=none Received: from [IPV6:2001:470:1f07:15ff::26] (court.m5p.com [IPv6:2001:470:1f07:15ff:0:0:0:26]) (authenticated bits=0) by mailhost.m5p.com (8.16.1/8.15.2) with ESMTPSA id 32QIO7LZ033075 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Sun, 26 Mar 2023 14:24:12 -0400 (EDT) (envelope-from george+freebsd@m5p.com) Message-ID: Date: Sun, 26 Mar 2023 14:24:07 -0400 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.4.0 Subject: Re: Periodic rant about SCHED_ULE To: freebsd-hackers@freebsd.org References: Content-Language: en-US From: George Mitchell In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.0 required=10.0 tests=HELO_NO_DOMAIN,NICE_REPLY_A autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mattapan.m5p.com X-Spamd-Result: default: False [-2.85 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_LONG(-1.00)[-0.997]; NEURAL_HAM_SHORT(-0.55)[-0.552]; R_SPF_ALLOW(-0.20)[+a]; MIME_GOOD(-0.10)[text/plain]; R_DKIM_NA(0.00)[]; TAGGED_FROM(0.00)[freebsd]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:701, ipnet:74.104.0.0/16, country:US]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[m5p.com]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4Pl48w1HnJz3sJp X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N On 3/25/23 23:42, Peter wrote: > Quoting George Mitchell : > > On 3/25/23 11:47, Peter wrote: >>> You're welcome. Can I get a success/failure report? >>> >>> >>> [...] >> Thanks for your help. Regretfully, I have to report that the patch >> did not really help in my case (make buildworld while running dnetc). >> But I have belatedly realized that it's a really good idea to use >> make -j12 to build kernels and the world. With 4BSD and dnetc >> running, I can now buildworld in 3512 seconds (down from 20477), and >> with ULE in 14193 seconds (down from 50290). Peter's patch got it >> down to 13755 seconds. > > Thank You for testing. So at least it doesn't make things worse. > Back then 5 years ago when I made that, I think I've seen a few > more strange things in that code, but they didn't bother me immanently > at that point. Probably there is more work to do there... To oversimplify the discussion I've seen so far on this thread: It would be really nice if the schedulers were kernel loadable modules. GSoC project? Whether SCHED_ULE or SCHED_4BSD should be the default scheduler in the GENERIC kernel is a contentious discussion, but perhaps we need to have that discussion. -- George From nobody Mon Mar 27 02:34:21 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PlH2V4Jj7z41XrQ for ; Mon, 27 Mar 2023 02:34:26 +0000 (UTC) (envelope-from obiwac@gmail.com) Received: from mail-ed1-x536.google.com (mail-ed1-x536.google.com [IPv6:2a00:1450:4864:20::536]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PlH2T0qx7z3Qn0; Mon, 27 Mar 2023 02:34:25 +0000 (UTC) (envelope-from obiwac@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=ml8nj5ZK; spf=pass (mx1.freebsd.org: domain of obiwac@gmail.com designates 2a00:1450:4864:20::536 as permitted sender) smtp.mailfrom=obiwac@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ed1-x536.google.com with SMTP id i5so30070564eda.0; Sun, 26 Mar 2023 19:34:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1679884462; h=cc:to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=aphvNDaRP1y/qtXjieCy/eKUyNWj4i9Ff55nJNJD/QA=; b=ml8nj5ZKu5HIzIPnv+g4JUHqLOW5Q9jJb0dXHMpRy5nBjKGyk9iBaqxOiIJh2NP4Sr 7xeUtSXQ2nkwikvdmnyhgIvU2+5F847R0CL2ato5Ah+jvCa9ryRpfNeExf+CQCj1nrdW T60MRIWAnamMYIoenb2gaHrJzRA2Syq0RsIop37b+ZbWaIvqiijJmCaJKkXeVUpOpmee jyiezuorfa35YuP51wlbMdqpbcDnu2PwCbLPUdVB5sGtvHEIAXpvP6LvzXKpg1NCHuXZ f0RY08dtHX9KT9um9NokKc3ujgx5RP44JcC0rSyLX/M9a1jKDgHIznnWq9bJDtqbDE90 EVxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679884462; h=cc:to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=aphvNDaRP1y/qtXjieCy/eKUyNWj4i9Ff55nJNJD/QA=; b=VJ7t0Bbi0bh0zMgddR1aqGJJcuxn/lT3r6B4S3gyr9Qnf4XxhpRsMqATYtfDDkBogb m6roGrUh2LAtAd38AYmMbaEDCQ7eF/Egut/3TTgAwRoHa3RFdM+zRYqD8sV2ueuYOATC 8eoSe6ttGu36QEmmGBklHBBQtxtLVt1tuE+QM0x3Xim6uLLdKrwATm2Ge6DhY0ZgEtu8 9v5ztBYbrGeiLo62cKgESD2kGM+nW6T/4cc1bjNpIKvl/7TjpYVGnQkFMYP/FRhqxido Ku4J5Hw3f703/CDOdxHHcWcSmduqaemTMCwVtzxQS+UVVv9gsgsuyNwzf1e3+Ki4H5gd VU5Q== X-Gm-Message-State: AAQBX9dgWyGTKrlqg2EKxhNvqL4aJF+tUeiUE5jVEHcz96qnXMrtQYwL hkdHc+ecu7WqWrjn5UjUFgw8mwaqiYp7nvKuWShMJogQ X-Google-Smtp-Source: AKy350aud3MIRIO5JoV5qb7/HtqluEBgt7Oi7JzEMMJ+dFmQSvv/gNIPgJhl2Pd41rZeLrkUj2uiJcL40ShSBMRMFyc= X-Received: by 2002:a17:906:94c2:b0:930:f984:c56f with SMTP id d2-20020a17090694c200b00930f984c56fmr4365264ejy.12.1679884462010; Sun, 26 Mar 2023 19:34:22 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:a17:906:478b:b0:8e7:7343:cad1 with HTTP; Sun, 26 Mar 2023 19:34:21 -0700 (PDT) From: obiwac Date: Mon, 27 Mar 2023 02:34:21 +0000 Message-ID: Subject: Search for GSoC mentors for BATMAN To: freebsd-hackers Cc: "mmokhi@FreeBSD.org" Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-3.10 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.99)[-0.994]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; NEURAL_HAM_SHORT(-0.10)[-0.101]; MIME_GOOD(-0.10)[text/plain]; TO_DN_EQ_ADDR_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::536:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4PlH2T0qx7z3Qn0 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N Hey! I had already posted about a Linux KPI-related networking project for FreeBSD for GSoC. melifaro@ redirected me towards mmokhi@ and bz@, with whom I have since followed up. After a short discussion with mmokhi@, I've settled on wanting to bring BATMAN support to FreeBSD, which is an ad-hoc networking protocol developed for use in the Freifunk initiative. As mmokhi@ may not have time to be a full mentor this year, I'm sending this message as a last-minute effort to find someone else who'd be interested in mentoring/co-mentoring this project. If you are, I can send over my current proposal draft and we can discuss things further :D Kind regards, Aymeric From nobody Mon Mar 27 12:13:44 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PlWv008qQz42C12 for ; Mon, 27 Mar 2023 12:13:48 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-ot1-x32d.google.com (mail-ot1-x32d.google.com [IPv6:2607:f8b0:4864:20::32d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PlWty6bVBz3hld; Mon, 27 Mar 2023 12:13:46 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=VDVMJmC6; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2607:f8b0:4864:20::32d as permitted sender) smtp.mailfrom=mjguzik@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ot1-x32d.google.com with SMTP id r40-20020a05683044a800b006a14270bc7eso1078576otv.6; Mon, 27 Mar 2023 05:13:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1679919225; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=QO+Ser1AXpRsgldlo2kK9vLuNgEyHN4IF1pXN2VzGyU=; b=VDVMJmC6E2JrXHRsmpGvMJs4sEgIjIQRTdVuFRu5qKqJRAL6XmhrASzeG1OMDZuV/3 rXQjwSsmHLwKvDE4wkD1UiNa+qHKj1gm6G46bLXvInlBtKWIuRsKpjxWXbxA6jMSXx2h ZYjx7a7qT4cYPdpydywz+pr7KQFGGHD3olnr2IdAd5iL6i77ArtPmJ6Bzh6LW3DfmbNI kF1LeDkRjEXH4RBC+5+nApPxIU8en4A033noBSCN1s5/gLMqIh76gpqweZ9jaLTFrc6p WbNDOJUWS0LFW6jzWZZAa/JyZci8npGVPtBswo96c/m+QfzZWafTROk8JAVWJsM4kNlL 4hsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679919225; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=QO+Ser1AXpRsgldlo2kK9vLuNgEyHN4IF1pXN2VzGyU=; b=iQO1HhmsXXEwD6Z4kWlJhPe1jaBl1Utxxmt6cijFTUvJx9tfc0CsMDeEY8D9yiC7kN 9GbGPKZeyUG5AJrSBeS9OnkOP8GLwMyI7gxWqwalXG489CjFc5widKNtmPMkOO7MMsxO 2Cn2OMMO7Q1R/mrVSiPYl+thHNHocK5uR00svFjFb31Lxvv05SciUXLsOXfaVrOafVMN YJ0SPXgOXSTaUf+SGKT8VMlC+QojB23Lv1rslPUtqglgY3NRSAMHsV0NCgQMCyilBfjd znicRubrlSLkyKimTGEiIbqWnZHcZE40WjknUopHP+Nmwfg9Z3k7p1wZHhr75wJm3mu9 d4cQ== X-Gm-Message-State: AO0yUKU84V4xLapRd8/h7UfcL8aSfvO0GULqxWxG1ywtBop7Tmts47jL 1NxkLiyvsPkXUXWvoVwfIvxdHT1LkA5vOuUEpl8vb+GD X-Google-Smtp-Source: AK7set/pGbosf55toylK3a9gls3v682ZkjbU6snZUeTglzW/+Dc5NCv3mC27Rley/BMCgSiGikBWe/zrDOuFlVwmMp4= X-Received: by 2002:a9d:63c5:0:b0:698:f988:7c30 with SMTP id e5-20020a9d63c5000000b00698f9887c30mr3777695otl.2.1679919225228; Mon, 27 Mar 2023 05:13:45 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:a8a:12c2:0:b0:49c:b071:b1e3 with HTTP; Mon, 27 Mar 2023 05:13:44 -0700 (PDT) In-Reply-To: References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> From: Mateusz Guzik Date: Mon, 27 Mar 2023 14:13:44 +0200 Message-ID: Subject: Re: Periodic rant about SCHED_ULE To: sgk@troutmask.apl.washington.edu Cc: Matthias Andree , freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-2.12 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_SPAM_SHORT(0.88)[0.876]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::32d:from]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DKIM_TRACE(0.00)[gmail.com:+]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; TO_DN_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4PlWty6bVBz3hld X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N On 3/25/23, Mateusz Guzik wrote: > On 3/23/23, Mateusz Guzik wrote: >> On 3/22/23, Mateusz Guzik wrote: >>> On 3/22/23, Steve Kargl wrote: >>>> On Wed, Mar 22, 2023 at 07:31:57PM +0100, Matthias Andree wrote: >>>>> >>>>> Yes, there are reports that FreeBSD is not responsive by default - but >>>>> this >>>>> may make it get overall better throughput at the expense of >>>>> responsiveness, >>>>> because it might be doing fewer context switches. So just complaining >>>>> about >>>>> a longer buildworld without seeing how much dnetc did in the same >>>>> wallclock >>>>> time period is useless. Periodic rant's don't fix this lack of >>>>> information. >>>>> >>>> >>>> I reported the issue with ULE some 15 to 20 years ago. >>>> I gave up reporting the issue. The individuals with the >>>> requisite skills to hack on ULE did not; and yes, I lack >>>> those skills. The path of least resistance is to use >>>> 4BSD. >>>> >>>> % cat a.f90 >>>> ! >>>> ! Silly numerically intensive computation. >>>> ! >>>> program foo >>>> implicit none >>>> integer, parameter :: m = 200, n = 1000, dp = kind(1.d0) >>>> integer i >>>> real(dp) x >>>> real(dp), allocatable :: a(:,:), b(:,:), c(:,:) >>>> call random_init(.true., .true.) >>>> allocate(a(n,n), b(n,n)) >>>> do i = 1, m >>>> call random_number(a) >>>> call random_number(b) >>>> c = matmul(a,b) >>>> x = sum(c) >>>> if (x < 0) stop 'Whoops' >>>> end do >>>> end program foo >>>> % gfortran11 -o z -O3 -march=native a.f90 >>>> % time ./z >>>> 42.16 real 42.04 user 0.09 sys >>>> % cat foo >>>> #! /bin/csh >>>> # >>>> # Launch NCPU+1 images with a 1 second delay >>>> # >>>> foreach i (1 2 3 4 5 6 7 8 9) >>>> ./z & >>>> sleep 1 >>>> end >>>> % ./foo >>>> >>>> In another xterm, you can watch the 9 images. >>>> >>>> % top >>>> st pid: 1709; load averages: 4.90, 1.61, 0.79 up 0+00:56:46 >>>> 11:43:01 >>>> 74 processes: 10 running, 64 sleeping >>>> CPU: 99.9% user, 0.0% nice, 0.1% system, 0.0% interrupt, 0.0% idle >>>> Mem: 369M Active, 187M Inact, 240K Laundry, 889M Wired, 546M Buf, 14G >>>> Free >>>> Swap: 16G Total, 16G Free >>>> >>>> PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU >>>> COMMAND >>>> 1699 kargl 1 56 0 68M 35M RUN 3 0:41 92.60% >>>> z >>>> 1701 kargl 1 56 0 68M 35M RUN 0 0:41 92.33% >>>> z >>>> 1689 kargl 1 56 0 68M 35M CPU5 5 0:47 91.63% >>>> z >>>> 1691 kargl 1 56 0 68M 35M CPU0 0 0:45 89.91% >>>> z >>>> 1695 kargl 1 56 0 68M 35M CPU2 2 0:43 88.56% >>>> z >>>> 1697 kargl 1 56 0 68M 35M CPU6 6 0:42 88.48% >>>> z >>>> 1705 kargl 1 55 0 68M 35M CPU1 1 0:39 88.12% >>>> z >>>> 1703 kargl 1 56 0 68M 35M CPU4 4 0:39 87.86% >>>> z >>>> 1693 kargl 1 56 0 68M 35M CPU7 7 0:45 78.12% >>>> z >>>> >>>> With 4BSD, you see the ./z's with 80% or greater CPU. All the ./z's >>>> exit >>>> after 55-ish seconds. If you try this experiment on ULE, you'll get >>>> NCPU-1 >>>> ./z's with nearly 99% CPU and 2 ./z's with something like 45-ish% as >>>> the >>>> two images ping-pong on one cpu. Back when I was testing ULE vs 4BSD, >>>> this was/is due to ULE's cpu affinity where processes never migrate to >>>> another cpu. Admittedly, this was several years ago. Maybe ULE has >>>> gotten better, but George's rant seems to suggest otherwise. >>>> >>> >>> While I have not tried openmpi yet, I can confirm there is a problem >>> here, but the situtation is not as clear cut as one might think. >>> >>> sched_4bsd *round robins* all workers across all CPUs, which comes at >>> a performance *hit* compared to ule if number of workers is <= CPU >>> count -- here ule maintains affinity, avoiding cache busting. If you >>> slap in $cpu_count + 1 workers, 4bsd keeps the round robin equally >>> penalizing everyone, while ule mostly penalizes a select victim. By >>> the end of it, you get lower total cpu time spent, but higher total >>> real time. See below for an example. >>> >>> Two massive problems with 4bsd, apart from mandatory round robin which >>> also happens to help by accident: >>> 1. it has one *global* lock, meaning the scheduler itself just does >>> not scale and this is visible at modest contemporary scales >>> 2. it does not understand topology -- no accounting done for ht nor >>> numa, but i concede the latter wont be a factor for most people >>> >>> That said, ULE definitely has performance bugs which need to be fixed. >>> At least for the case below, 4BSD just "lucked" into sucking less >>> simply because it is so primitive. >>> >>> I wrote a cpu burning program (memset 1 MB in a loop, with enough >>> iterations to take ~20 seconds on its own). >>> >>> I booted an 8 core bhyve vm, where I made sure to cpuset is to 8 >>> distinct >>> cores. >>> >>> The test runs *9* workers, here is a sample run: >>> >>> 4bsd: >>> 23.18 real 20.81 user 0.00 sys >>> 23.26 real 20.81 user 0.00 sys >>> 23.30 real 20.81 user 0.00 sys >>> 23.34 real 20.82 user 0.00 sys >>> 23.41 real 20.81 user 0.00 sys >>> 23.41 real 20.80 user 0.00 sys >>> 23.42 real 20.80 user 0.00 sys >>> 23.53 real 20.81 user 0.00 sys >>> 23.60 real 20.80 user 0.00 sys >>> 187.31s user 0.02s system 793% cpu 23.606 total >>> >>> ule: >>> 20.67 real 20.04 user 0.00 sys >>> 20.97 real 20.00 user 0.00 sys >>> 21.45 real 20.29 user 0.00 sys >>> 21.51 real 20.22 user 0.00 sys >>> 22.77 real 20.04 user 0.00 sys >>> 22.78 real 20.26 user 0.00 sys >>> 23.42 real 20.04 user 0.00 sys >>> 24.07 real 20.30 user 0.00 sys >>> 24.46 real 20.16 user 0.00 sys >>> 181.41s user 0.07s system 741% cpu 24.465 total >>> >>> It reliably uses 187s user time on 4BSD and 181s on ULE. At the same >>> time it also reliably has massive time imblance between >>> fastest/slowest in terms of total real time between workers *and* ULE >>> reliably uses more total real time to finish the entire thing. >>> >>> iow this is a tradeoff, but most likely a bad one >>> >> >> So I also ran the following setup: 8 core vm doing -j 8 buildkernel, >> while 8 nice -n 20 processes are cpu-bound. After the build ends >> workers report how many ops they did in that time. >> >> ule is way off the reservation here. >> >> unimpeded build takes: 441.39 real 3135.63 user 266.92, similar on >> both schedulers >> >> with cpu hoggers: >> 4bsd: 657.22 real 3152.02 user 253.30 sys [+49%] >> ule: 4427.69 real 3225.33 user 194.86 sys [+903%] >> >> ule spends less time in the kernel, but the time blows up -- over 10 x >> the base line. >> this is clearly a total non-starter. >> >> full stats: >> >> 4bsd: >> hogger pid/ops >> 58315: 5546013 >> 58322: 5557294 >> 58321: 5545052 >> 58313: 5546347 >> 58318: 5537874 >> 58317: 5555303 >> 58323: 5545116 >> 58320: 5548530 >> >> runtimes: >> >> 657.23 real 230.02 user 0.02 sys >> 657.23 real 229.83 user 0.00 sys >> 657.23 real 230.50 user 0.00 sys >> 657.23 real 230.53 user 0.00 sys >> 657.23 real 230.14 user 0.01 sys >> 657.23 real 230.19 user 0.00 sys >> 657.23 real 230.09 user 0.00 sys >> 657.23 real 230.10 user 0.03 sys >> >> kernel build: >> 657.22 real 3152.02 user 253.30 sys >> >> ule: >> hogger pid/ops >> 77794: 95916836 >> 77792: 95909794 >> 77789: 96454064 >> 77796: 95813778 >> 77791: 96728121 >> 77795: 96678291 >> 77798: 97258060 >> 77797: 96347984 >> >> 4427.70 real 4001.94 user 0.10 sys >> 4427.70 real 3980.68 user 0.16 sys >> 4427.70 real 3973.96 user 0.10 sys >> 4427.70 real 3980.11 user 0.13 sys >> 4427.70 real 4012.32 user 0.07 sys >> 4427.70 real 4008.79 user 0.12 sys >> 4427.70 real 4034.77 user 0.09 sys >> 4427.70 real 3995.40 user 0.08 sys >> >> kernel build: >> 4427.69 real 3225.33 user 194.86 sys >> > > added a probe to runq_add*: > diff --git a/sys/kern/kern_switch.c b/sys/kern/kern_switch.c > index aee0c9cbd1ae..db73226547c5 100644 > --- a/sys/kern/kern_switch.c > +++ b/sys/kern/kern_switch.c > @@ -357,6 +357,9 @@ runq_setbit(struct runq *rq, int pri) > rqb->rqb_bits[RQB_WORD(pri)] |= RQB_BIT(pri); > } > > +SDT_PROVIDER_DECLARE(sched); > +SDT_PROBE_DEFINE3(sched, , , runq_add, "struct thread *", "int", "int"); > + > /* > * Add the thread to the queue specified by its priority, and set the > * corresponding status bit. > @@ -378,6 +381,7 @@ runq_add(struct runq *rq, struct thread *td, int flags) > } else { > TAILQ_INSERT_TAIL(rqh, td, td_runq); > } > + SDT_PROBE3(sched, , , runq_add, td, flags, pri); > } > > void > @@ -396,6 +400,7 @@ runq_add_pri(struct runq *rq, struct thread *td, > u_char pri, int flags) > } else { > TAILQ_INSERT_TAIL(rqh, td, td_runq); > } > + SDT_PROBE3(sched, , , runq_add, td, flags, pri); > } > /* > * Return true if there are runnable processes of any priority on the run > > and used this one-liner: > > dtrace -b 16M -x aggsize=32M -x dynvarsize=32M -n 'sched:::runq_add > /args[0]->td_ucred->cr_uid == 2000/ { self->runq_t = timestamp; } > sched:::on-cpu /self->runq_t/ { @runqlat["runq", execname == > "cpuburner-long" ? "cpuburner" : "rest"] = quantize(timestamp - > self->runq_t); self->runq_t = 0; } sched:::on-cpu > /curthread->td_ucred->cr_uid == 2000/ { self->oncpu_t = timestamp; } > sched:::off-cpu /self->oncpu_t/ { @oncpu["oncpu", execname == > "cpuburner-long" ? "cpuburner" : "rest"] = quantize(timestamp - > self->oncpu_t); } tick-300s { exit(0); } > > caped at 5 minutes because dtrace starts running into aggregation drops > > Key takeaway: 4bsd let's the cpu hog stay on cpu for longer than ule, > but when it is taken off, it waits a long time to get back. in > contrast, it gets back on very quick with ule and it is everyone else > waiting big time. > > times in nanoseconds > > 4bsd: > runq cpuburner > value ------------- Distribution ------------- count > 2048 | 0 > 4096 | 2 > 8192 |@@@@@@@@@@ 4343 > 16384 |@@@@@ 2159 > 32768 |@ 363 > 65536 |@@@ 1359 > 131072 | 215 > 262144 | 129 > 524288 | 101 > 1048576 | 132 > 2097152 | 185 > 4194304 |@ 390 > 8388608 |@ 318 > 16777216 |@ 398 > 33554432 |@@ 838 > 67108864 |@@@@@@@ 3025 > 134217728 |@@@@@ 2474 > 268435456 |@@@ 1552 > 536870912 | 153 > 1073741824 | 0 > > runq rest > value ------------- Distribution ------------- count > 2048 | 0 > 4096 | 637 > 8192 |@@@@@@@@@@@@@@@@@@@@@@@@ 364832 > 16384 |@@@ 52982 > 32768 |@ 11613 > 65536 |@@ 34286 > 131072 |@@@ 39734 > 262144 |@@ 23261 > 524288 |@ 21985 > 1048576 |@ 18999 > 2097152 |@ 10789 > 4194304 | 6239 > 8388608 | 4725 > 16777216 | 4598 > 33554432 | 4050 > 67108864 | 5857 > 134217728 | 3979 > 268435456 | 2024 > 536870912 | 2011 > 1073741824 | 1105 > 2147483648 | 841 > 4294967296 | 519 > 8589934592 | 372 > 17179869184 | 133 > 34359738368 | 37 > 68719476736 | 30 > 137438953472 | 1 > 274877906944 | 0 > > oncpu cpuburner > value ------------- Distribution ------------- count > 2048 | 0 > 4096 | 20 > 8192 | 10 > 16384 | 19 > 32768 | 161 > 65536 | 137 > 131072 | 77 > 262144 | 104 > 524288 | 147 > 1048576 |@ 301 > 2097152 |@ 482 > 4194304 |@@@ 1178 > 8388608 |@@@@@@@@@@@@@@@ 6971 > 16777216 |@ 474 > 33554432 |@ 669 > 67108864 |@@@@@@@@@@@@@@@@ 7299 > 134217728 | 14 > 268435456 | 38 > 536870912 | 38 > 1073741824 | 2 > 2147483648 | 2 > 4294967296 | 0 > > oncpu rest > value ------------- Distribution ------------- count > 512 | 0 > 1024 | 102 > 2048 |@@@@@@@@@@@ 146555 > 4096 |@@ 23373 > 8192 |@@@ 45165 > 16384 |@ 14531 > 32768 |@@@@@ 67664 > 65536 |@@@@@@ 80155 > 131072 |@@@@@ 64609 > 262144 |@@ 21509 > 524288 |@@ 23393 > 1048576 |@@ 21058 > 2097152 |@ 7854 > 4194304 |@ 8788 > 8388608 |@ 20242 > 16777216 | 2352 > 33554432 | 2084 > 67108864 | 4388 > 134217728 | 1123 > 268435456 | 755 > 536870912 | 135 > 1073741824 | 2 > 2147483648 | 0 > > ule: > runq cpuburner > value ------------- Distribution ------------- count > 2048 | 0 > 4096 | 2 > 8192 |@@@@@@@@@@@@@@@@@@@@@@@@ 37193 > 16384 |@@@@@ 8612 > 32768 |@ 1481 > 65536 |@@@@@ 8430 > 131072 |@@ 3102 > 262144 |@ 938 > 524288 | 457 > 1048576 |@ 2063 > 2097152 | 257 > 4194304 | 41 > 8388608 | 428 > 16777216 | 12 > 33554432 | 1 > 67108864 | 2 > 134217728 | 0 > > runq rest > value ------------- Distribution ------------- count > 4096 | 0 > 8192 |@ 649 > 16384 |@@@@ 1953 > es > 32768 |@ 539 > 65536 |@@@@@ 2369 > 131072 |@@@@ 1904 > 262144 |@ 471 > 524288 | 131 > 1048576 |@ 458 > 2097152 |@ 443 > 4194304 |@ 547 > 8388608 |@ 632 > 16777216 |@@ 984 > 33554432 |@@@ 1606 > 67108864 |@@@@@@@@ 3894 > 134217728 |@ 511 > 268435456 |@ 489 > 536870912 |@ 445 > 1073741824 |@ 475 > 2147483648 |@ 314 > 4294967296 | 188 > 8589934592 | 136 > 17179869184 | 100 > 34359738368 | 81 > 68719476736 | 39 > 137438953472 | 3 > 274877906944 | 0 > > oncpu cpuburner > value ------------- Distribution ------------- count > 2048 | 0 > 4096 | 13 > 8192 | 311 > 16384 |@@ 2369 > 32768 |@ 853 > 65536 |@ 2302 > 131072 |@ 1302 > 262144 | 518 > 524288 |@ 872 > 1048576 |@ 1172 > 2097152 |@ 1435 > 4194304 |@@ 2706 > 8388608 |@@@@@@@@@@@@@@@@@@@ 29669 > 16777216 |@@ 2733 > 33554432 |@@ 3910 > 67108864 |@@@@@@ 9991 > 134217728 |@ 1800 > 268435456 |@ 840 > 536870912 | 200 > 1073741824 | 15 > 2147483648 | 7 > 4294967296 | 1 > 8589934592 | 0 > > oncpu rest > value ------------- Distribution ------------- count > 512 | 0 > 1024 |@ 419 > 2048 |@@@@@@@@@ 5730 > 4096 |@ 778 > 8192 |@@@@@ 3233 > 16384 |@@ 1400 > 32768 |@@@@ 2739 > 65536 |@@@@@@@@@@@@ 7550 > 131072 |@@@ 1720 > 262144 | 151 > 524288 | 112 > 1048576 |@@@ 1688 > 2097152 | 174 > 4194304 | 86 > 8388608 |@ 411 > 16777216 | 7 > 33554432 | 3 > 67108864 | 0 > I repeat the setup: 8 cores, 8 processes doing cpu-bound stuff while niced to 20 vs make -j buildkernel I had a little more look here, slapped in some hacks as a POC and got an improvement from 67 minutes above to 21. Hacks are: 1. limit hog timeslice to 1 tick so that is more eager to bail 2. always preempt if pri < cpri So far I can confidently state the general problem: ULE penalizes non-cpu hogs for blocking (even if it is not their fault, so to speak) and that bumps their prio past preemption threshold, at which point they can't preempt said hogs (despite hogs having a higher priority). At the same time hogs use their full time slices, while non-hogs get off cpu very early and have to wait a long time to get back on, at least in part due to inability to preempt said hogs. As I mentioned elsewhere in the thread, interactivity scoring takes "voluntary off cpu time" into account. As literally anything but getting preempted counts as "voluntary sleep", workers get shafted for going off cpu while waiting on locks in the kernel. If I/O needs to happen and the thread waits for the result, it most likely does it early in its timeslice and once it's all ready it waits for background hogs to get off cpu -- it can't preempt them. All that said: 1. "interactivity scoring" (see sched_interact_score) I don't know if it makes any sense to begin with. Even if it does, it counts stuff it should not by not differentiating between deliberately going off cpu (e.g., actual sleep) vs just waiting for a file being read. Imagine firefox reading a file from disk and being considered less interactive for it. I don't have a solution for this problem. I *suspect* the way to go would be to explicitly mark xorg/wayland/whatever as "interactive" and have it inherited by its offspring. At the same time it should not follow to stuff spawned in terminals. Not claiming this is perfect, but it does eliminate the guessing game. Even so, 4BSD does not have any mechanism of the sort and reportedly remains usable on a desktop just by providing some degree of fairness. Given that, I suspect the short term solution would whack said scoring altogether and focus on fairness (see below). 2. fairness As explained above doing any offcpu-time inducing work instantly shafts threads versus cpu hogs, even if said hogs are niced way above them. Here I *suspect* position to add in the runqueue should be related to how much slice was left when the thread went off cpu, while making sure that hogs get to run eventually. Not that I have a nice way of implementing this -- maybe a separate queue for known hogs and picking them every n turns or similar. -- Mateusz Guzik From nobody Mon Mar 27 13:28:24 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PlYY65vdFz41X8w for ; Mon, 27 Mar 2023 13:28:26 +0000 (UTC) (envelope-from mandree@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PlYY65Mcqz3sTJ; Mon, 27 Mar 2023 13:28:26 +0000 (UTC) (envelope-from mandree@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679923706; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h/IF62W68WqABKseHo5QXfRri9Qv+g6BJwJOXukT2mY=; b=eFBS0pp36dwLf1EzPxLz3cmhiu/UotU2cw6YnKzgVdAK48jrfFsDeBWhsfJz7lUlo9hrAU E3vnnYG8KxT58ZoZB1oCjgzf2+RmI8K0K0F8WIIo0jevqPub8fbgF3DoTiEKucugnvg1xo yjAosdPQLudK7pSVGZ5GeZukjZzfEoIwb5XrBy7mDf2a3Q4/hqOriHgZNw5dCPtnQ6fwzw ozLtgg1P9o+v6uRCb7SF7+CaYB3m+lFZQksp29pdslvg78UoElMLGn+ceq5a7r3sj7A8SZ LKsWmCKOIypyFU6xKDSxWrp1y/nI15/sqFUb8xKGV2s/AkONy6JwBb8ONJr17w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679923706; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h/IF62W68WqABKseHo5QXfRri9Qv+g6BJwJOXukT2mY=; b=qIhdaGrAnmsCS5wy19xML+0NfDU0xp36m3uZofqF+d4JxIudpZG4XjaTQNSs7/Kr032QJx a4VAaYLHO6VnPymPAyg2T81dc+5qn3YhJFFwQ3V3VIvPCA2e5HdaIddYCKpdV3DB3IIVxG rZYOLWjSNHuEYpd4wJ/KmJxjRjUuINAsIOiZ3+AVutqi4+30+sqct6f+gs/z5RXh1csJMH cPYHN2VeRQvnLYg3s/UC5jzQ5l+g6BwMEEMOvTi/z/tR7/OtvmRRkNvodrIY9tVCtuAJSY nxagozseIY60WyWS8susdidu8Pc4xjuV5zo8sUdheBf14xLeB7BmaZ7IsDIxYg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679923706; a=rsa-sha256; cv=none; b=sC/7ojmIBFMnjRbu0jmHu3x6mOvP/8SxALx6QNgN4QmeAZ/DvdXIWkdyFyHdlZ3CowU97U L2XLGV53bvmpjPYq8Vxyx4n1yFhb07lTAQ5WRpsxomBF9XD+35dKLeXTxg0gae1HJVWDHm jKgfh1NhXq/lEvi1l8LqtXwQULbJoEnqhNptzc2aE/cxhL2Th5vVq5m9O6iwV9VnxXp3i3 rQ6q159foNHyg9i65DarxoU/F3skZ9hoQsAqqhNk3RldNwJy/ybNrm8ZJxs/HuzhOjPrpj MMPa09YdOs3DvO8BCit62rjGvNW0THQZyvcOqCiEgF8shqAjbpNDmUysmqmBxA== Received: from mandree.no-ip.org (pd9e073a3.dip0.t-ipconnect.de [217.224.115.163]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: mandree/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4PlYY63RXLz1Q0X; Mon, 27 Mar 2023 13:28:26 +0000 (UTC) (envelope-from mandree@freebsd.org) Received: from [127.0.0.1] (localhost [127.0.0.1]) by ryzen.an3e.de (Postfix) with ESMTP id 78B2ADC47FE; Mon, 27 Mar 2023 15:28:24 +0200 (CEST) Message-ID: <1d4d589c-eafe-6219-2e6e-884d49b7ff57@FreeBSD.org> Date: Mon, 27 Mar 2023 15:28:24 +0200 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 From: Matthias Andree Content-Language: en-US To: Mateusz Guzik , sgk@troutmask.apl.washington.edu Cc: freebsd-hackers@freebsd.org References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> Organization: FreeBSD.org Subject: Re: Periodic rant about SCHED_ULE In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ThisMailContainsUnwantedMimeParts: N Am 27.03.23 um 14:13 schrieb Mateusz Guzik: > > I repeat the setup: 8 cores, 8 processes doing cpu-bound stuff while > niced to 20 vs make -j buildkernel > > I had a little more look here, slapped in some hacks as a POC and got > an improvement from 67 minutes above to 21. > > Hacks are: > 1. limit hog timeslice to 1 tick so that is more eager to bail > 2. always preempt if pri < cpri > > So far I can confidently state the general problem: ULE penalizes > non-cpu hogs for blocking (even if it is not their fault, so to speak) > and that bumps their prio past preemption threshold, at which point > they can't preempt said hogs (despite hogs having a higher priority). > At the same time hogs use their full time slices, while non-hogs get > off cpu very early and have to wait a long time to get back on, at > least in part due to inability to preempt said hogs. > > As I mentioned elsewhere in the thread, interactivity scoring takes > "voluntary off cpu time" into account. As literally anything but > getting preempted counts as "voluntary sleep", workers get shafted for > going off cpu while waiting on locks in the kernel. > > If I/O needs to happen and the thread waits for the result, it most > likely does it early in its timeslice and once it's all ready it waits > for background hogs to get off cpu -- it can't preempt them. > > All that said: > 1. "interactivity scoring" (see sched_interact_score) > > I don't know if it makes any sense to begin with. Even if it does, it > counts stuff it should not by not differentiating between deliberately > going off cpu (e.g., actual sleep) vs just waiting for a file being > read. Imagine firefox reading a file from disk and being considered > less interactive for it. > > I don't have a solution for this problem. I *suspect* the way to go > would be to explicitly mark xorg/wayland/whatever as "interactive" and > have it inherited by its offspring. At the same time it should not > follow to stuff spawned in terminals. Not claiming this is perfect, > but it does eliminate the guessing game. > > Even so, 4BSD does not have any mechanism of the sort and reportedly > remains usable on a desktop just by providing some degree of fairness. > > Given that, I suspect the short term solution would whack said scoring > altogether and focus on fairness (see below). > > 2. fairness > > As explained above doing any offcpu-time inducing work instantly > shafts threads versus cpu hogs, even if said hogs are niced way above > them. > > Here I *suspect* position to add in the runqueue should be related to > how much slice was left when the thread went off cpu, while making > sure that hogs get to run eventually. Not that I have a nice way of > implementing this -- maybe a separate queue for known hogs and picking > them every n turns or similar. > There is some analogy - not sure if we can learn something from it. TCP (the network's Transmission Control Protocol) has the big overspanning item "fairness between streams", and possibly we need to put this into focus. Apparently we have now established that threads that do not use up their quantum are not scheduled *sooner* which seems to harm both interactivity AND total processing time (because, say, they may want to run a spell checker after receiving a key-press key-release sequence or send off an e-mail after a click on the send button). What I currently fail to understand is why we have not gone to improving SCHED_ULE yet. This seems to be the first time I've paid attention to some details. Have I missed similar discussion in the past? Or have they stalled soonish? -- Matthias Andree FreeBSD ports committer From nobody Mon Mar 27 14:04:31 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PlbFr5Bf0z41d68 for ; Mon, 27 Mar 2023 14:45:20 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PlbFq1lNnz44lm; Mon, 27 Mar 2023 14:45:19 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of pmc@citylink.dinoex.sub.org designates 2a0b:f840::12 as permitted sender) smtp.mailfrom=pmc@citylink.dinoex.sub.org; dmarc=none Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.17.1/8.17.1) with ESMTPS id 32REj6J3066357 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Mon, 27 Mar 2023 16:45:07 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from uucp@localhost) by uucp.dinoex.org (8.17.1/8.17.1/Submit) with UUCP id 32REj6pi066356; Mon, 27 Mar 2023 16:45:06 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (disp-e.intra.daemon.contact [IPv6:fd00:0:0:0:0:0:0:112]) by admn.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32RE4iYl050781 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK); Mon, 27 Mar 2023 16:04:44 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (localhost [127.0.0.1]) by disp.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32RE4V69003712 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Mon, 27 Mar 2023 16:04:32 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from pmc@localhost) by disp.intra.daemon.contact (8.17.1/8.17.1/Submit) id 32RE4VGi003711; Mon, 27 Mar 2023 16:04:31 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) X-Authentication-Warning: disp.intra.daemon.contact: pmc set sender to pmc@citylink.dinoex.sub.org using -f Date: Mon, 27 Mar 2023 16:04:31 +0200 From: Peter To: freebsd-hackers@freebsd.org Cc: Graham Perrin Subject: Re: User-Agent: and In-Reply-To: Message-ID: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Mon, 27 Mar 2023 16:45:09 +0200 (CEST) X-Spamd-Result: default: False [-2.30 / 15.00]; FAKE_REPLY(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-1.00)[-0.997]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; R_DKIM_NA(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; HAS_XAW(0.00)[]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE]; RCVD_COUNT_FIVE(0.00)[5]; ARC_NA(0.00)[]; DMARC_NA(0.00)[sub.org]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4PlbFq1lNnz44lm X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N Quoting Graham Perrin : >

Not to start a long debate (or rant), I'm curious.
>

>

Mark, Peter, please: what do you use to send email?
sendmail. ;) >

>

A recent discussion is split across twelve threads; screenshot > attached.=C2=A0

This is probably because I currently have to create answers from scratch. List postings are received here by a technical account and sorted into a database where messages are decent to read. With the old listserver it was possible to reply to them by simply adding the technical account as the Sender: address. The new "mimi" software does not accept this construct. I will occasionally figure out a better solution, to properly present mails that contain crypto/base64/webpages/whatever, and also to enable direct replies again - but for now this not high on my priorities list, as most people on the list tend to Cc: to the personal mail address anyway, and then References: and In-Reply-To: should get generated. From nobody Mon Mar 27 14:31:27 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PlbGF2HgFz41cwL for ; Mon, 27 Mar 2023 14:45:41 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PlbGD6zBdz45T0; Mon, 27 Mar 2023 14:45:40 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; none Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.17.1/8.17.1) with ESMTPS id 32REj9U8066388 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Mon, 27 Mar 2023 16:45:09 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from uucp@localhost) by uucp.dinoex.org (8.17.1/8.17.1/Submit) with UUCP id 32REj9d4066387; Mon, 27 Mar 2023 16:45:09 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (disp-e.intra.daemon.contact [IPv6:fd00:0:0:0:0:0:0:112]) by admn.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32REVie5064944 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK); Mon, 27 Mar 2023 16:31:44 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (localhost [127.0.0.1]) by disp.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32REVRNc003965 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Mon, 27 Mar 2023 16:31:27 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from pmc@localhost) by disp.intra.daemon.contact (8.17.1/8.17.1/Submit) id 32REVRI9003964; Mon, 27 Mar 2023 16:31:27 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) X-Authentication-Warning: disp.intra.daemon.contact: pmc set sender to pmc@citylink.dinoex.sub.org using -f Date: Mon, 27 Mar 2023 16:31:27 +0200 From: Peter To: freebsd-hackers@freebsd.org Cc: Matthias Andree Subject: Re: Periodic rant about SCHED_ULE Message-ID: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1d4d589c-eafe-6219-2e6e-884d49b7ff57@FreeBSD.org> X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Mon, 27 Mar 2023 16:45:12 +0200 (CEST) X-Rspamd-Queue-Id: 4PlbGD6zBdz45T0 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N Quoting Matthias Andree : >There is some analogy - not sure if we can learn something from it. TCP >(the network's Transmission Control Protocol) has the big overspanning >item "fairness between streams", and possibly we need to put this into >focus. It is a delicate matter there also, and gladly we have plugable modules to do the job. (On my site only 'vegas' with specific parameters does give satisfying results - but then it does). >What I currently fail to understand is why we have not gone to improving >SCHED_ULE yet. This seems to be the first time I've paid attention to >some details. Have I missed similar discussion in the past? Or have >they stalled soonish? Indeed I don't remember these discussion going into the technical details inside the scheduler. Maybe nobody (including me) was eloquent enough to push it there? From nobody Mon Mar 27 14:47:04 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PlbHv0wr9z41dB4 for ; Mon, 27 Mar 2023 14:47:07 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oa1-x2e.google.com (mail-oa1-x2e.google.com [IPv6:2001:4860:4864:20::2e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PlbHt1KFPz46Wh; Mon, 27 Mar 2023 14:47:06 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=P0wJ3Mum; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2001:4860:4864:20::2e as permitted sender) smtp.mailfrom=mjguzik@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oa1-x2e.google.com with SMTP id 586e51a60fabf-17ac5ee3f9cso9472681fac.12; Mon, 27 Mar 2023 07:47:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1679928425; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=5cHxHuVqCYz5YOIGxGLqECG7HrsNvOYj8/k2G4gAgBY=; b=P0wJ3MumKG/UJVuc3dtp3a3MiE+sF4wB6tVW8s377JMnfNZMO7ObQdYrdCXA7zdPtO hYUfrXrqysBCqVVafO5gQk0wlZieuTKiBBNRGTZxTzW7AhTSTfWRdVHcMVmFSozAH/4d lhnmQeNMtMNOqnaltbUxfju18rpTypct5a6C6fGUi5ITI6UoHIPKZBMeZZmeulPx2jA8 RS6Y9mQQluqmExyIJHiE06sAw5RxRCqcQmzgBynGaBnA1SnnCSqBCkR9Iqmq9eoDKagH 9tpWx7Ap5mta2RE2jnv3P0VDIZa6lcpJLlCkpg1GUYkvbM768WXefxhOLfW9Z8JKl1UP z73A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679928425; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=5cHxHuVqCYz5YOIGxGLqECG7HrsNvOYj8/k2G4gAgBY=; b=Du4bEbbUGJrJEO01KA588UkEab9euzksYeeN8pHn40lczs6CQ2iwjbx+SQYFhKEu7Z CTWmFGZiSYZ5QnxqB6C/8V3wWvR+Su39o3XtP4LXZ4vR1J1+LNS4NoW300oOjfVYuHsx CyYvujfVajGPSHfHmMdwwjqs/fJwKUddubqGilZtf+nmTZvqwHPssbulLa+M0Zj7yVUy Sj7g/PuRPVRVALd+jZTx7Qv/OXa1rO3/wY8zJ1Pbp7zc894dVnqAnwr7pylK0cbH48zc VEvlTVQ4/r9E7CFdomJb/zm71y/7C+YQe6EHMSuNgVs+wcFfXlcIKSI7EFqM7b0irsHW fqFA== X-Gm-Message-State: AAQBX9eCL5fQYgkCXmtF8ZILdxJk0BpuOK1YF2m9x5ironbDOk9dY2Vf KOA6fXE61cQSr01Ppd9YNKKMe4XxSHhYuB4nGJCFbGMM X-Google-Smtp-Source: AK7set9a+D/Ng5HVt4/nHxcV8gczAM+Wb//NQh+FV57vs4ULkvMjJhTRwBLe4I35xJxehsjE/nkiD2zOnX7+oQItHPk= X-Received: by 2002:a05:6871:e09:b0:17a:b713:63e9 with SMTP id vj9-20020a0568710e0900b0017ab71363e9mr3994788oab.4.1679928425051; Mon, 27 Mar 2023 07:47:05 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:a8a:12c2:0:b0:49c:b071:b1e3 with HTTP; Mon, 27 Mar 2023 07:47:04 -0700 (PDT) In-Reply-To: References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> From: Mateusz Guzik Date: Mon, 27 Mar 2023 16:47:04 +0200 Message-ID: Subject: Re: Periodic rant about SCHED_ULE To: sgk@troutmask.apl.washington.edu Cc: Matthias Andree , freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-1.58 / 15.00]; URI_HIDDEN_PATH(1.00)[https://people.freebsd.org/~mjg/.junk/ule-poc-hacks-dont-use.diff]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_SPAM_SHORT(0.42)[0.419]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; R_SPF_ALLOW(-0.20)[+ip6:2001:4860:4000::/36]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_NONE(0.00)[2001:4860:4864:20::2e:from]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; ARC_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:2001:4860:4864::/48, country:US]; DKIM_TRACE(0.00)[gmail.com:+]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; TO_DN_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4PlbHt1KFPz46Wh X-Spamd-Bar: - X-ThisMailContainsUnwantedMimeParts: N On 3/27/23, Mateusz Guzik wrote: > On 3/25/23, Mateusz Guzik wrote: >> On 3/23/23, Mateusz Guzik wrote: >>> On 3/22/23, Mateusz Guzik wrote: >>>> On 3/22/23, Steve Kargl wrote: >>>>> On Wed, Mar 22, 2023 at 07:31:57PM +0100, Matthias Andree wrote: >>>>>> >>>>>> Yes, there are reports that FreeBSD is not responsive by default - >>>>>> but >>>>>> this >>>>>> may make it get overall better throughput at the expense of >>>>>> responsiveness, >>>>>> because it might be doing fewer context switches. So just >>>>>> complaining >>>>>> about >>>>>> a longer buildworld without seeing how much dnetc did in the same >>>>>> wallclock >>>>>> time period is useless. Periodic rant's don't fix this lack of >>>>>> information. >>>>>> >>>>> >>>>> I reported the issue with ULE some 15 to 20 years ago. >>>>> I gave up reporting the issue. The individuals with the >>>>> requisite skills to hack on ULE did not; and yes, I lack >>>>> those skills. The path of least resistance is to use >>>>> 4BSD. >>>>> >>>>> % cat a.f90 >>>>> ! >>>>> ! Silly numerically intensive computation. >>>>> ! >>>>> program foo >>>>> implicit none >>>>> integer, parameter :: m = 200, n = 1000, dp = kind(1.d0) >>>>> integer i >>>>> real(dp) x >>>>> real(dp), allocatable :: a(:,:), b(:,:), c(:,:) >>>>> call random_init(.true., .true.) >>>>> allocate(a(n,n), b(n,n)) >>>>> do i = 1, m >>>>> call random_number(a) >>>>> call random_number(b) >>>>> c = matmul(a,b) >>>>> x = sum(c) >>>>> if (x < 0) stop 'Whoops' >>>>> end do >>>>> end program foo >>>>> % gfortran11 -o z -O3 -march=native a.f90 >>>>> % time ./z >>>>> 42.16 real 42.04 user 0.09 sys >>>>> % cat foo >>>>> #! /bin/csh >>>>> # >>>>> # Launch NCPU+1 images with a 1 second delay >>>>> # >>>>> foreach i (1 2 3 4 5 6 7 8 9) >>>>> ./z & >>>>> sleep 1 >>>>> end >>>>> % ./foo >>>>> >>>>> In another xterm, you can watch the 9 images. >>>>> >>>>> % top >>>>> st pid: 1709; load averages: 4.90, 1.61, 0.79 up 0+00:56:46 >>>>> 11:43:01 >>>>> 74 processes: 10 running, 64 sleeping >>>>> CPU: 99.9% user, 0.0% nice, 0.1% system, 0.0% interrupt, 0.0% idle >>>>> Mem: 369M Active, 187M Inact, 240K Laundry, 889M Wired, 546M Buf, 14G >>>>> Free >>>>> Swap: 16G Total, 16G Free >>>>> >>>>> PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU >>>>> COMMAND >>>>> 1699 kargl 1 56 0 68M 35M RUN 3 0:41 92.60% >>>>> z >>>>> 1701 kargl 1 56 0 68M 35M RUN 0 0:41 92.33% >>>>> z >>>>> 1689 kargl 1 56 0 68M 35M CPU5 5 0:47 91.63% >>>>> z >>>>> 1691 kargl 1 56 0 68M 35M CPU0 0 0:45 89.91% >>>>> z >>>>> 1695 kargl 1 56 0 68M 35M CPU2 2 0:43 88.56% >>>>> z >>>>> 1697 kargl 1 56 0 68M 35M CPU6 6 0:42 88.48% >>>>> z >>>>> 1705 kargl 1 55 0 68M 35M CPU1 1 0:39 88.12% >>>>> z >>>>> 1703 kargl 1 56 0 68M 35M CPU4 4 0:39 87.86% >>>>> z >>>>> 1693 kargl 1 56 0 68M 35M CPU7 7 0:45 78.12% >>>>> z >>>>> >>>>> With 4BSD, you see the ./z's with 80% or greater CPU. All the ./z's >>>>> exit >>>>> after 55-ish seconds. If you try this experiment on ULE, you'll get >>>>> NCPU-1 >>>>> ./z's with nearly 99% CPU and 2 ./z's with something like 45-ish% as >>>>> the >>>>> two images ping-pong on one cpu. Back when I was testing ULE vs 4BSD, >>>>> this was/is due to ULE's cpu affinity where processes never migrate to >>>>> another cpu. Admittedly, this was several years ago. Maybe ULE has >>>>> gotten better, but George's rant seems to suggest otherwise. >>>>> >>>> >>>> While I have not tried openmpi yet, I can confirm there is a problem >>>> here, but the situtation is not as clear cut as one might think. >>>> >>>> sched_4bsd *round robins* all workers across all CPUs, which comes at >>>> a performance *hit* compared to ule if number of workers is <= CPU >>>> count -- here ule maintains affinity, avoiding cache busting. If you >>>> slap in $cpu_count + 1 workers, 4bsd keeps the round robin equally >>>> penalizing everyone, while ule mostly penalizes a select victim. By >>>> the end of it, you get lower total cpu time spent, but higher total >>>> real time. See below for an example. >>>> >>>> Two massive problems with 4bsd, apart from mandatory round robin which >>>> also happens to help by accident: >>>> 1. it has one *global* lock, meaning the scheduler itself just does >>>> not scale and this is visible at modest contemporary scales >>>> 2. it does not understand topology -- no accounting done for ht nor >>>> numa, but i concede the latter wont be a factor for most people >>>> >>>> That said, ULE definitely has performance bugs which need to be fixed. >>>> At least for the case below, 4BSD just "lucked" into sucking less >>>> simply because it is so primitive. >>>> >>>> I wrote a cpu burning program (memset 1 MB in a loop, with enough >>>> iterations to take ~20 seconds on its own). >>>> >>>> I booted an 8 core bhyve vm, where I made sure to cpuset is to 8 >>>> distinct >>>> cores. >>>> >>>> The test runs *9* workers, here is a sample run: >>>> >>>> 4bsd: >>>> 23.18 real 20.81 user 0.00 sys >>>> 23.26 real 20.81 user 0.00 sys >>>> 23.30 real 20.81 user 0.00 sys >>>> 23.34 real 20.82 user 0.00 sys >>>> 23.41 real 20.81 user 0.00 sys >>>> 23.41 real 20.80 user 0.00 sys >>>> 23.42 real 20.80 user 0.00 sys >>>> 23.53 real 20.81 user 0.00 sys >>>> 23.60 real 20.80 user 0.00 sys >>>> 187.31s user 0.02s system 793% cpu 23.606 total >>>> >>>> ule: >>>> 20.67 real 20.04 user 0.00 sys >>>> 20.97 real 20.00 user 0.00 sys >>>> 21.45 real 20.29 user 0.00 sys >>>> 21.51 real 20.22 user 0.00 sys >>>> 22.77 real 20.04 user 0.00 sys >>>> 22.78 real 20.26 user 0.00 sys >>>> 23.42 real 20.04 user 0.00 sys >>>> 24.07 real 20.30 user 0.00 sys >>>> 24.46 real 20.16 user 0.00 sys >>>> 181.41s user 0.07s system 741% cpu 24.465 total >>>> >>>> It reliably uses 187s user time on 4BSD and 181s on ULE. At the same >>>> time it also reliably has massive time imblance between >>>> fastest/slowest in terms of total real time between workers *and* ULE >>>> reliably uses more total real time to finish the entire thing. >>>> >>>> iow this is a tradeoff, but most likely a bad one >>>> >>> >>> So I also ran the following setup: 8 core vm doing -j 8 buildkernel, >>> while 8 nice -n 20 processes are cpu-bound. After the build ends >>> workers report how many ops they did in that time. >>> >>> ule is way off the reservation here. >>> >>> unimpeded build takes: 441.39 real 3135.63 user 266.92, similar on >>> both schedulers >>> >>> with cpu hoggers: >>> 4bsd: 657.22 real 3152.02 user 253.30 sys [+49%] >>> ule: 4427.69 real 3225.33 user 194.86 sys [+903%] >>> >>> ule spends less time in the kernel, but the time blows up -- over 10 x >>> the base line. >>> this is clearly a total non-starter. >>> >>> full stats: >>> >>> 4bsd: >>> hogger pid/ops >>> 58315: 5546013 >>> 58322: 5557294 >>> 58321: 5545052 >>> 58313: 5546347 >>> 58318: 5537874 >>> 58317: 5555303 >>> 58323: 5545116 >>> 58320: 5548530 >>> >>> runtimes: >>> >>> 657.23 real 230.02 user 0.02 sys >>> 657.23 real 229.83 user 0.00 sys >>> 657.23 real 230.50 user 0.00 sys >>> 657.23 real 230.53 user 0.00 sys >>> 657.23 real 230.14 user 0.01 sys >>> 657.23 real 230.19 user 0.00 sys >>> 657.23 real 230.09 user 0.00 sys >>> 657.23 real 230.10 user 0.03 sys >>> >>> kernel build: >>> 657.22 real 3152.02 user 253.30 sys >>> >>> ule: >>> hogger pid/ops >>> 77794: 95916836 >>> 77792: 95909794 >>> 77789: 96454064 >>> 77796: 95813778 >>> 77791: 96728121 >>> 77795: 96678291 >>> 77798: 97258060 >>> 77797: 96347984 >>> >>> 4427.70 real 4001.94 user 0.10 sys >>> 4427.70 real 3980.68 user 0.16 sys >>> 4427.70 real 3973.96 user 0.10 sys >>> 4427.70 real 3980.11 user 0.13 sys >>> 4427.70 real 4012.32 user 0.07 sys >>> 4427.70 real 4008.79 user 0.12 sys >>> 4427.70 real 4034.77 user 0.09 sys >>> 4427.70 real 3995.40 user 0.08 sys >>> >>> kernel build: >>> 4427.69 real 3225.33 user 194.86 sys >>> >> >> added a probe to runq_add*: >> diff --git a/sys/kern/kern_switch.c b/sys/kern/kern_switch.c >> index aee0c9cbd1ae..db73226547c5 100644 >> --- a/sys/kern/kern_switch.c >> +++ b/sys/kern/kern_switch.c >> @@ -357,6 +357,9 @@ runq_setbit(struct runq *rq, int pri) >> rqb->rqb_bits[RQB_WORD(pri)] |= RQB_BIT(pri); >> } >> >> +SDT_PROVIDER_DECLARE(sched); >> +SDT_PROBE_DEFINE3(sched, , , runq_add, "struct thread *", "int", "int"); >> + >> /* >> * Add the thread to the queue specified by its priority, and set the >> * corresponding status bit. >> @@ -378,6 +381,7 @@ runq_add(struct runq *rq, struct thread *td, int >> flags) >> } else { >> TAILQ_INSERT_TAIL(rqh, td, td_runq); >> } >> + SDT_PROBE3(sched, , , runq_add, td, flags, pri); >> } >> >> void >> @@ -396,6 +400,7 @@ runq_add_pri(struct runq *rq, struct thread *td, >> u_char pri, int flags) >> } else { >> TAILQ_INSERT_TAIL(rqh, td, td_runq); >> } >> + SDT_PROBE3(sched, , , runq_add, td, flags, pri); >> } >> /* >> * Return true if there are runnable processes of any priority on the >> run >> >> and used this one-liner: >> >> dtrace -b 16M -x aggsize=32M -x dynvarsize=32M -n 'sched:::runq_add >> /args[0]->td_ucred->cr_uid == 2000/ { self->runq_t = timestamp; } >> sched:::on-cpu /self->runq_t/ { @runqlat["runq", execname == >> "cpuburner-long" ? "cpuburner" : "rest"] = quantize(timestamp - >> self->runq_t); self->runq_t = 0; } sched:::on-cpu >> /curthread->td_ucred->cr_uid == 2000/ { self->oncpu_t = timestamp; } >> sched:::off-cpu /self->oncpu_t/ { @oncpu["oncpu", execname == >> "cpuburner-long" ? "cpuburner" : "rest"] = quantize(timestamp - >> self->oncpu_t); } tick-300s { exit(0); } >> >> caped at 5 minutes because dtrace starts running into aggregation drops >> >> Key takeaway: 4bsd let's the cpu hog stay on cpu for longer than ule, >> but when it is taken off, it waits a long time to get back. in >> contrast, it gets back on very quick with ule and it is everyone else >> waiting big time. >> >> times in nanoseconds >> >> 4bsd: >> runq cpuburner >> value ------------- Distribution ------------- count >> 2048 | 0 >> 4096 | 2 >> 8192 |@@@@@@@@@@ 4343 >> 16384 |@@@@@ 2159 >> 32768 |@ 363 >> 65536 |@@@ 1359 >> 131072 | 215 >> 262144 | 129 >> 524288 | 101 >> 1048576 | 132 >> 2097152 | 185 >> 4194304 |@ 390 >> 8388608 |@ 318 >> 16777216 |@ 398 >> 33554432 |@@ 838 >> 67108864 |@@@@@@@ 3025 >> 134217728 |@@@@@ 2474 >> 268435456 |@@@ 1552 >> 536870912 | 153 >> 1073741824 | 0 >> >> runq rest >> value ------------- Distribution ------------- count >> 2048 | 0 >> 4096 | 637 >> 8192 |@@@@@@@@@@@@@@@@@@@@@@@@ 364832 >> 16384 |@@@ 52982 >> 32768 |@ 11613 >> 65536 |@@ 34286 >> 131072 |@@@ 39734 >> 262144 |@@ 23261 >> 524288 |@ 21985 >> 1048576 |@ 18999 >> 2097152 |@ 10789 >> 4194304 | 6239 >> 8388608 | 4725 >> 16777216 | 4598 >> 33554432 | 4050 >> 67108864 | 5857 >> 134217728 | 3979 >> 268435456 | 2024 >> 536870912 | 2011 >> 1073741824 | 1105 >> 2147483648 | 841 >> 4294967296 | 519 >> 8589934592 | 372 >> 17179869184 | 133 >> 34359738368 | 37 >> 68719476736 | 30 >> 137438953472 | 1 >> 274877906944 | 0 >> >> oncpu cpuburner >> value ------------- Distribution ------------- count >> 2048 | 0 >> 4096 | 20 >> 8192 | 10 >> 16384 | 19 >> 32768 | 161 >> 65536 | 137 >> 131072 | 77 >> 262144 | 104 >> 524288 | 147 >> 1048576 |@ 301 >> 2097152 |@ 482 >> 4194304 |@@@ 1178 >> 8388608 |@@@@@@@@@@@@@@@ 6971 >> 16777216 |@ 474 >> 33554432 |@ 669 >> 67108864 |@@@@@@@@@@@@@@@@ 7299 >> 134217728 | 14 >> 268435456 | 38 >> 536870912 | 38 >> 1073741824 | 2 >> 2147483648 | 2 >> 4294967296 | 0 >> >> oncpu rest >> value ------------- Distribution ------------- count >> 512 | 0 >> 1024 | 102 >> 2048 |@@@@@@@@@@@ 146555 >> 4096 |@@ 23373 >> 8192 |@@@ 45165 >> 16384 |@ 14531 >> 32768 |@@@@@ 67664 >> 65536 |@@@@@@ 80155 >> 131072 |@@@@@ 64609 >> 262144 |@@ 21509 >> 524288 |@@ 23393 >> 1048576 |@@ 21058 >> 2097152 |@ 7854 >> 4194304 |@ 8788 >> 8388608 |@ 20242 >> 16777216 | 2352 >> 33554432 | 2084 >> 67108864 | 4388 >> 134217728 | 1123 >> 268435456 | 755 >> 536870912 | 135 >> 1073741824 | 2 >> 2147483648 | 0 >> >> ule: >> runq cpuburner >> value ------------- Distribution ------------- count >> 2048 | 0 >> 4096 | 2 >> 8192 |@@@@@@@@@@@@@@@@@@@@@@@@ 37193 >> 16384 |@@@@@ 8612 >> 32768 |@ 1481 >> 65536 |@@@@@ 8430 >> 131072 |@@ 3102 >> 262144 |@ 938 >> 524288 | 457 >> 1048576 |@ 2063 >> 2097152 | 257 >> 4194304 | 41 >> 8388608 | 428 >> 16777216 | 12 >> 33554432 | 1 >> 67108864 | 2 >> 134217728 | 0 >> >> runq rest >> value ------------- Distribution ------------- count >> 4096 | 0 >> 8192 |@ 649 >> 16384 |@@@@ 1953 >> es >> 32768 |@ 539 >> 65536 |@@@@@ 2369 >> 131072 |@@@@ 1904 >> 262144 |@ 471 >> 524288 | 131 >> 1048576 |@ 458 >> 2097152 |@ 443 >> 4194304 |@ 547 >> 8388608 |@ 632 >> 16777216 |@@ 984 >> 33554432 |@@@ 1606 >> 67108864 |@@@@@@@@ 3894 >> 134217728 |@ 511 >> 268435456 |@ 489 >> 536870912 |@ 445 >> 1073741824 |@ 475 >> 2147483648 |@ 314 >> 4294967296 | 188 >> 8589934592 | 136 >> 17179869184 | 100 >> 34359738368 | 81 >> 68719476736 | 39 >> 137438953472 | 3 >> 274877906944 | 0 >> >> oncpu cpuburner >> value ------------- Distribution ------------- count >> 2048 | 0 >> 4096 | 13 >> 8192 | 311 >> 16384 |@@ 2369 >> 32768 |@ 853 >> 65536 |@ 2302 >> 131072 |@ 1302 >> 262144 | 518 >> 524288 |@ 872 >> 1048576 |@ 1172 >> 2097152 |@ 1435 >> 4194304 |@@ 2706 >> 8388608 |@@@@@@@@@@@@@@@@@@@ 29669 >> 16777216 |@@ 2733 >> 33554432 |@@ 3910 >> 67108864 |@@@@@@ 9991 >> 134217728 |@ 1800 >> 268435456 |@ 840 >> 536870912 | 200 >> 1073741824 | 15 >> 2147483648 | 7 >> 4294967296 | 1 >> 8589934592 | 0 >> >> oncpu rest >> value ------------- Distribution ------------- count >> 512 | 0 >> 1024 |@ 419 >> 2048 |@@@@@@@@@ 5730 >> 4096 |@ 778 >> 8192 |@@@@@ 3233 >> 16384 |@@ 1400 >> 32768 |@@@@ 2739 >> 65536 |@@@@@@@@@@@@ 7550 >> 131072 |@@@ 1720 >> 262144 | 151 >> 524288 | 112 >> 1048576 |@@@ 1688 >> 2097152 | 174 >> 4194304 | 86 >> 8388608 |@ 411 >> 16777216 | 7 >> 33554432 | 3 >> 67108864 | 0 >> > > I repeat the setup: 8 cores, 8 processes doing cpu-bound stuff while > niced to 20 vs make -j buildkernel > > I had a little more look here, slapped in some hacks as a POC and got > an improvement from 67 minutes above to 21. > > Hacks are: > 1. limit hog timeslice to 1 tick so that is more eager to bail > 2. always preempt if pri < cpri > > So far I can confidently state the general problem: ULE penalizes > non-cpu hogs for blocking (even if it is not their fault, so to speak) > and that bumps their prio past preemption threshold, at which point > they can't preempt said hogs (despite hogs having a higher priority). > At the same time hogs use their full time slices, while non-hogs get > off cpu very early and have to wait a long time to get back on, at > least in part due to inability to preempt said hogs. > > As I mentioned elsewhere in the thread, interactivity scoring takes > "voluntary off cpu time" into account. As literally anything but > getting preempted counts as "voluntary sleep", workers get shafted for > going off cpu while waiting on locks in the kernel. > > If I/O needs to happen and the thread waits for the result, it most > likely does it early in its timeslice and once it's all ready it waits > for background hogs to get off cpu -- it can't preempt them. > > All that said: > 1. "interactivity scoring" (see sched_interact_score) > > I don't know if it makes any sense to begin with. Even if it does, it > counts stuff it should not by not differentiating between deliberately > going off cpu (e.g., actual sleep) vs just waiting for a file being > read. Imagine firefox reading a file from disk and being considered > less interactive for it. > > I don't have a solution for this problem. I *suspect* the way to go > would be to explicitly mark xorg/wayland/whatever as "interactive" and > have it inherited by its offspring. At the same time it should not > follow to stuff spawned in terminals. Not claiming this is perfect, > but it does eliminate the guessing game. > > Even so, 4BSD does not have any mechanism of the sort and reportedly > remains usable on a desktop just by providing some degree of fairness. > > Given that, I suspect the short term solution would whack said scoring > altogether and focus on fairness (see below). > > 2. fairness > > As explained above doing any offcpu-time inducing work instantly > shafts threads versus cpu hogs, even if said hogs are niced way above > them. > > Here I *suspect* position to add in the runqueue should be related to > how much slice was left when the thread went off cpu, while making > sure that hogs get to run eventually. Not that I have a nice way of > implementing this -- maybe a separate queue for known hogs and picking > them every n turns or similar. > Aight, now that I had a sober look at the code I think I cracked the case. The runq mechanism used by both 4BSD and ULE provides 64(!) queues, where the priority is divided by said number and that's how you know in which queue to land the thread. When deciding what to run, 4BSD uses runq_choose which iterates all queues from the beginning. This means threads of lower priority keep executing before the rest. In particular cpu hog lands with a high priority, looking worse than make -j 8 buildkernel and only running when there is nothing else ready to get the cpu. While this may sound decent, it is bad -- in principle a steady stream of lower priority threads can starve the hogs indefinitely. The problem was recognized when writing ULE, but improperly fixed -- it ends up distributing all threads within given priority range across the queues and then performing a lookup in a given queue. Here the problem is that while technically everyone does get a chance to run, the threads not using full slices are hosed for the time period as they wait for the hog *a lot*. A hack patch to induce the bogus-but-better 4BSD behavior of draining all runqs before running higher prio threads drops down build time to ~9 minutes, which is shorter than 4BSD. However, the right fix would achieve that *without* introducing starvation potential. I also note the runqs are a massive waste of memory and computing power. I'm going to have to sleep on what to do here. For interested here is the hackery: https://people.freebsd.org/~mjg/.junk/ule-poc-hacks-dont-use.diff sysctl kern.sched.slice_nice=0 sysctl kern.sched.preempt_thresh=400 # arbitrary number higher than any prio -- Mateusz Guzik From nobody Mon Mar 27 16:31:18 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PldcG4YM1z41lGn for ; Mon, 27 Mar 2023 16:31:26 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.95.76.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "troutmask", Issuer "troutmask" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PldcG1nhHz4LZf; Mon, 27 Mar 2023 16:31:26 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Authentication-Results: mx1.freebsd.org; none Received: from troutmask.apl.washington.edu (localhost [127.0.0.1]) by troutmask.apl.washington.edu (8.17.1/8.17.1) with ESMTPS id 32RGVI57077106 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Mon, 27 Mar 2023 09:31:19 -0700 (PDT) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.17.1/8.17.1/Submit) id 32RGVIj2077105; Mon, 27 Mar 2023 09:31:18 -0700 (PDT) (envelope-from sgk) Date: Mon, 27 Mar 2023 09:31:18 -0700 From: Steve Kargl To: Mateusz Guzik Cc: Matthias Andree , freebsd-hackers@freebsd.org Subject: Re: Periodic rant about SCHED_ULE Message-ID: Reply-To: sgk@troutmask.apl.washington.edu References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4PldcG1nhHz4LZf X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:73, ipnet:128.95.0.0/16, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Mon, Mar 27, 2023 at 04:47:04PM +0200, Mateusz Guzik wrote: > (A mssive amount trimmed to keep this short.) > Aight, now that I had a sober look at the code I think I cracked the case. > > The runq mechanism used by both 4BSD and ULE provides 64(!) queues, > where the priority is divided by said number and that's how you know > in which queue to land the thread. > > When deciding what to run, 4BSD uses runq_choose which iterates all > queues from the beginning. This means threads of lower priority keep > executing before the rest. In particular cpu hog lands with a high > priority, looking worse than make -j 8 buildkernel and only running > when there is nothing else ready to get the cpu. While this may sound > decent, it is bad -- in principle a steady stream of lower priority > threads can starve the hogs indefinitely. > > The problem was recognized when writing ULE, but improperly fixed -- > it ends up distributing all threads within given priority range across > the queues and then performing a lookup in a given queue. Here the > problem is that while technically everyone does get a chance to run, > the threads not using full slices are hosed for the time period as > they wait for the hog *a lot*. > > A hack patch to induce the bogus-but-better 4BSD behavior of draining > all runqs before running higher prio threads drops down build time to > ~9 minutes, which is shorter than 4BSD. > > However, the right fix would achieve that *without* introducing > starvation potential. > > I also note the runqs are a massive waste of memory and computing > power. I'm going to have to sleep on what to do here. > > For interested here is the hackery: > https://people.freebsd.org/~mjg/.junk/ule-poc-hacks-dont-use.diff > > sysctl kern.sched.slice_nice=0 > sysctl kern.sched.preempt_thresh=400 # arbitrary number higher than any prio Mateusz, Thanks for taking a deeper look at the schedulers and providing your analysis. If you come up with any patches that you would like to see have additional testing, feel free to ping on or off the mailing list. -- Steve From nobody Mon Mar 27 21:02:23 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Plld71DHSz423lj for ; Mon, 27 Mar 2023 21:02:35 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Plld66TJzz3Qpq for ; Mon, 27 Mar 2023 21:02:34 +0000 (UTC) (envelope-from kevans@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679950954; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=7UDSD+QJAjzxajXqYrKjCwevPc5E6tW+xW/ec8sxJ2Y=; b=x4zUiHnGIZ4exWA7aYVLtJ0W00qVr/KmeGVzG/+VaqQwYm2yiBJYWdRfaGoEHYjoZe4BJ9 8xnizptCwdlrjrCN8XHtOqBlMg+9VdLzqOY6PxXnUrD157GO53HHYmR5GSL4U5otcMkKIi Hg1IhuqO8XAlNfgP/37YUoepNtIsu4679UpXRUV9WZykGSRNjoNz6gfuWuV65MPNFpFEwL gPdGVJO5zlCVul75BE/HGKVOov9+wHCYwLhZnFCKeiUAMUzaHSVggz5BfjRrJgD6m4dtxx MF7x4y8fOZyrISxsq6ZDZ1FQCFaYJMKsXDSS95tSBPu31et+Vjjta8Nh7YPDDw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679950954; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=7UDSD+QJAjzxajXqYrKjCwevPc5E6tW+xW/ec8sxJ2Y=; b=uPrEoS/ogEBGjSK3BCtQgLB1nR6yJ5OJ60xNUeL7Z9vBsS1YQozg0roOSEJNcmlKoQVyNy KvXQEEPbYQmUkeI3WNQHaVQ/7KypNgn8HoNoUeayOwJLtFxmCpczow6RmYV08iKEHteQwh JdcxJZ7Jc0jDTGH6XTvCfOmxhPcXSP2tDWqxJEUdPihXaHmrj2/o2YQ2rasjY8J/IuUqfw Yelxw0MkEGApbaEm69NIVPqlsOibVDDIJJTQzZPWIdp2a7Mz6qiNUBBP0qGYtcOnBlZFXq LjFb6myUvTtSw2zpt7DarRUgp/WrZ+sWuN+1wRt5mpp+jmD7UjPkkJBsNTmmvw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679950954; a=rsa-sha256; cv=none; b=RVWv46EhZIX0cZh0siF0ZBaDe7vZC1kt7eYBciQZJu0JyahkPY4U3hENRLMGikyK0nCOlA sLGSd73PBo6sXfJdvUF+IwrMhcRRO4QYDXj6IEFaNK8nx6k0/p8hWQst5N80Vh570UL/4x su6Z135MVIUDypHVu1M0si6BOmNgw7/7IJUwdj2uZwQZ3C6M1CQfliq90MlZZOaKmzVJok yjQWPbSDCuhs+Bgzv+q+zfPjeaWwzd55KiT0RDGuZoQFgI7sMu9gq9NnC/5XIWIlebsEf/ kZuhzIropFRjHNc5jNd4mOK9a6omA5fBiWpeX6aggybKA19ZnZc6UOSHkzeraQ== Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) (Authenticated sender: kevans) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Plld65X1jzLNw for ; Mon, 27 Mar 2023 21:02:34 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: by mail-qt1-f176.google.com with SMTP id cr18so6108920qtb.0 for ; Mon, 27 Mar 2023 14:02:34 -0700 (PDT) X-Gm-Message-State: AO0yUKUoEGrJQAUdlMMss4Sm9qsAgZS3gEZFjhfGPHO4KTjWFO+8ttJs 8+Ep+gqDkbLH1hHGSi37rlWYChroXd3LTBgUq0s= X-Google-Smtp-Source: AK7set80jUn6h8SlwlE9uEAgDg6JKE0y83ilsjCj7Wj7ctZ41oEshhXbm7lBZN+poVno9A6njoLsKk8gnpWKNihNWsI= X-Received: by 2002:ac8:5f0b:0:b0:3e3:8a0e:495f with SMTP id x11-20020ac85f0b000000b003e38a0e495fmr4896253qta.13.1679950954162; Mon, 27 Mar 2023 14:02:34 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 From: Kyle Evans Date: Mon, 27 Mar 2023 16:02:23 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: flua/makeman reviews To: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-ThisMailContainsUnwantedMimeParts: N Hi, I have a couple of reviews in flight for flua && makeman, if anybody is interested: - https://reviews.freebsd.org/D39083 (flua: lposix: add more useful functions for general purpose scripts) - https://reviews.freebsd.org/D39084 (tools: build: add a rewrite of makeman in lua) D39084 doesn't hook up makeman.lua to the "makeman" target, yet; it simply adds the new version, then we can validate it and replace the existing makeman shell script. The main benefit of the rewrite of makeman is that it applies some super-naive parallelism to makeman; one fork for every option described in src.conf(5). This dropped execution time on one of our high-core count arm64 machines from ~1m40s on average to ~15s. In a VM on an Intel NUC, down from ~2 minutes to 40s. Thanks, Kyle Evans From nobody Mon Mar 27 22:07:34 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pln4J5PNYz427jq for ; Mon, 27 Mar 2023 22:07:44 +0000 (UTC) (envelope-from lorenzo.castellini28@gmail.com) Received: from mail-qt1-x830.google.com (mail-qt1-x830.google.com [IPv6:2607:f8b0:4864:20::830]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pln4H5sntz3sjY for ; Mon, 27 Mar 2023 22:07:43 +0000 (UTC) (envelope-from lorenzo.castellini28@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=CXjUAK2x; spf=pass (mx1.freebsd.org: domain of lorenzo.castellini28@gmail.com designates 2607:f8b0:4864:20::830 as permitted sender) smtp.mailfrom=lorenzo.castellini28@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-qt1-x830.google.com with SMTP id p2so5010939qtw.13 for ; Mon, 27 Mar 2023 15:07:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1679954863; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=3FMtacX09067gspwkVLIuhLu5zX149BEz0FJqc2bQq0=; b=CXjUAK2xV6sL+oSYLNdPUHn/D4hBvsx8ELxhlk0aZV3RaJGuABIyMWpPnMjOnysSWr vnWL55/upsuxMnDp9z1A3T+S1ld4x1NCn12F3awLcF7+niJ+gJAH8mGaN+VOJBRuvm28 IkcjdhXTBct0MCosPIpOrYN9QDoKLvKoo6GzZPxY3zjVZ9wm5GmrUrMWG2jex2SLjv28 DRFdw+WlXnFTq4muapn9LWLV5yC+wiVNKUHeYtAvPcPNnRh2Nv4ZiRX1kQvrl3m9n52h J/ZDixuCf8ynyUoMnGzi+UwDTQOMeuXG1JxYe5McaOHHCNKgiP09DavtDjE47gUenkMD bqjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679954863; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=3FMtacX09067gspwkVLIuhLu5zX149BEz0FJqc2bQq0=; b=Wr89NXOERxjk8KFGuwG7gE/Jf31O7h80dqUL554+c00BPYUWqRLAZ/P3nn7DwAkMRu vTNPDoL5GO3G1+NfUDavs8JALRzu8YAL8FFnNsq1o3HvFJN9ttfGdfFoWWi7f7HtGGQv te5pJiyFA8eufgSgWjecwTJKd3EUh/cUnr870CGzcf1msR4yBrhHYxpLZtsJ/G9otDn9 OvQM7cRMW23BVePDxYBAeWS0xEw05Eo5wx4D1DqQgj80/y2FTbgOLU0VGLY0GWjX7Erj zUY/L/ZK7Eh0Qk7UxP6lgyfBEJW42T+mA2+FJ5PcpLZS4uDo/yFJ//MQ0kmsYYT+Twhc kpbQ== X-Gm-Message-State: AO0yUKXwsu6YQBImAAAtTwjweAuM6N0iRBYQx06UpdQI8p8I6lH7Rr5n qJSidxKLUcpAspZYlHID6rPRfjJgJRt1KroXM0EaKKfXeBnL1P4ZwFY= X-Google-Smtp-Source: AK7set/j39/aWv9QoJAB7oIqSSPptp/Ro1oSuvu3igfxzXF+xpnBFYwjiwE/+erTE5dPVPyt5YXxT3Wdit0ZvknREjk= X-Received: by 2002:a05:622a:199a:b0:3d5:49eb:4d1e with SMTP id u26-20020a05622a199a00b003d549eb4d1emr5042635qtc.1.1679954862771; Mon, 27 Mar 2023 15:07:42 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 From: Lorenzo Castellini Date: Mon, 27 Mar 2023 18:07:34 -0400 Message-ID: Subject: Google Summer of Code To: freebsd-hackers@freebsd.org Content-Type: multipart/alternative; boundary="000000000000d5908005f7e8f8eb" X-Spamd-Result: default: False [-3.88 / 15.00]; NEURAL_HAM_LONG(-1.00)[-0.998]; NEURAL_HAM_SHORT(-0.97)[-0.975]; NEURAL_HAM_MEDIUM(-0.90)[-0.905]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::830:from]; ARC_NA(0.00)[]; TAGGED_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_DN_NONE(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4Pln4H5sntz3sjY X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N --000000000000d5908005f7e8f8eb Content-Type: text/plain; charset="UTF-8" Greetings, I hope this email finds you well. I am Lorenzo Castellini, a senior high school student, with an interest in participating in the Google Summer of Code. I am writing this email, since I am interested in collaborating with you guys on the project, but I am in need of a bit of guidance with the project proposal. I checked out the project list, and I was particularly interested in a project of virtual memory compression. I would like to work with this, maybe extending the project even further or applying some automation to such a process. On another note, I would also like to work with something similar, but with other PC components. Let me know of some ideas according to my interests. Thanks! Sincerely, Lorenzo Castellini Coutin --000000000000d5908005f7e8f8eb Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Greetings,
I hope this email finds you well. I am Lore= nzo Castellini, a senior high school student, with an interest in participa= ting in the Google Summer of Code. I am writing this email, since I am inte= rested in collaborating=C2=A0with you guys on the project, but I am in need= of a bit of guidance with the project proposal. I checked out the project = list, and I was particularly interested in a project of virtual memory comp= ression. I would like to work with this, maybe extending the project even f= urther or applying some automation to such a process. On another note, I wo= uld also like to work with something similar, but with other PC components.= Let me know of some ideas according to my interests. Thanks!

Sincerely,
Lorenzo Castellini Coutin
--000000000000d5908005f7e8f8eb-- From nobody Mon Mar 27 22:23:42 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PlnVz3Pygz42948 for ; Mon, 27 Mar 2023 22:27:23 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PlnVy6W2Bz3vtT; Mon, 27 Mar 2023 22:27:22 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; none Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.17.1/8.17.1) with ESMTPS id 32RMR7BM039300 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Tue, 28 Mar 2023 00:27:07 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from uucp@localhost) by uucp.dinoex.org (8.17.1/8.17.1/Submit) with UUCP id 32RMR7Ds039299; Tue, 28 Mar 2023 00:27:07 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (disp-e.intra.daemon.contact [IPv6:fd00:0:0:0:0:0:0:112]) by admn.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32RMPjdx029781 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK); Tue, 28 Mar 2023 00:25:45 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (localhost [127.0.0.1]) by disp.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32RMNgkL014177 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Tue, 28 Mar 2023 00:23:43 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from pmc@localhost) by disp.intra.daemon.contact (8.17.1/8.17.1/Submit) id 32RMNg2j014176; Tue, 28 Mar 2023 00:23:42 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) X-Authentication-Warning: disp.intra.daemon.contact: pmc set sender to pmc@citylink.dinoex.sub.org using -f Date: Tue, 28 Mar 2023 00:23:42 +0200 From: Peter To: freebsd-hackers@freebsd.org Cc: Mateusz Guzik , Matthias Andree Subject: Re: Periodic rant about SCHED_ULE Message-ID: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Tue, 28 Mar 2023 00:27:10 +0200 (CEST) X-Rspamd-Queue-Id: 4PlnVy6W2Bz3vtT X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N Quoting Mateusz Guzik : >When deciding what to run, 4BSD uses runq_choose which iterates all >queues from the beginning. This means threads of lower priority keep >executing before the rest. In particular cpu hog lands with a high >priority, looking worse than make -j 8 buildkernel and only running >when there is nothing else ready to get the cpu. While this may sound >decent, it is bad -- in principle a steady stream of lower priority >threads can starve the hogs indefinitely. Ideally, while starving, the priority of that hog should occasionally get re-calculated, until it is low enough to run it again. Not sure if that actually happens. >The problem was recognized when writing ULE, but improperly fixed -- >it ends up distributing all threads within given priority range across >the queues and then performing a lookup in a given queue. Here the As I did understand it, the ULE run-queue is circular and works like a revolver barrel. So yes, once in a while everybody gets a shot. >problem is that while technically everyone does get a chance to run, >the threads not using full slices are hosed for the time period as >they wait for the hog *a lot*. Yes, exactly. The hog stays at the pole position until they use up their quantum. From nobody Tue Mar 28 03:19:19 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Plvzw528yz42Ttk for ; Tue, 28 Mar 2023 03:19:24 +0000 (UTC) (envelope-from void@f-m.fm) Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Plvzv4NMTz3CX2 for ; Tue, 28 Mar 2023 03:19:23 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm2 header.b=SWImWw4B; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=idXsMT7K; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 66.111.4.26 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 198875C01DC for ; Mon, 27 Mar 2023 23:19:22 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Mon, 27 Mar 2023 23:19:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :message-id:mime-version:reply-to:sender:subject:subject:to:to; s=fm2; t=1679973562; x=1680059962; bh=ZzdW/9Zwan36bt+C3uo/I6yfU 692vnGN2SvO5MTQRos=; b=SWImWw4BNW48WEQhWPxf8Uw7nvSIPTvk/VXDLdt8Z hVcsXsL2hWtwRz4KfSfcamGJrQbmEOZCIHVSu9nLf0pbSndGSsjKNLOZzHXti6Fz W9eyeLFsBDUWlvPI4OziBRKZ2PJ8AF8BrsbEtahNAzSrDbr3wmMDmVEfafjpmxeJ Mlf+0RiDNTqWZZ/9RF85f7zYVocbMzqVAvlxg+PMWm+OaSSlkW6QcMYUXhv6NzCL sNxuDbRFlj1b1FnIHS5qZvOx/70wFtV96wLOnmCx1DvDIQ5GdhTO84AX2rYtRFW6 XMovw4GMCw86yxGqEJI/RGic32dJQo1V475hIg9t/WTQg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:message-id :mime-version:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t= 1679973562; x=1680059962; bh=ZzdW/9Zwan36bt+C3uo/I6yfU692vnGN2Sv O5MTQRos=; b=idXsMT7KrQO3Dd6nUCSICciNeeQ7hiAortsC0+nc9jqHmMWwEPP 1nNe0MJJRk6eEs/v1CQ2IuEkvt5Gqj7lQ+1TEKNXTnkhlUI7/AzuD37O4M1VNMPt ZBYCYixP/uA4exbRUd/sWpt5SqLxO1Y506j6HS2XzHbyxdtZY6iGTLVcmxzmHrDu H5dYF+zO+n6xPIk6kxBzrwK8maxuFv57Q5n5uqz+ByihF3p2vVD6A3zWKc7evWfi eVSkHJ3wmy3ZGj0E2cxW22/kCLfynp+tiZvMrDf1WBjNEJJSdDpQs7itlii15nE4 uFceoAgJ5B2Ybvd1Amav57arL4UMdoUMukg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdehfedgjedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkgggtugesthdtredttd dtvdenucfhrhhomhepvhhoihguuceovhhoihgusehfqdhmrdhfmheqnecuggftrfgrthht vghrnhepveduffeivdfffffghfegfeejfefftdeiteehteekfefhvdefgfettdeuheegff eunecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepvhho ihgusehfqdhmrdhfmh X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Mon, 27 Mar 2023 23:19:21 -0400 (EDT) Date: Tue, 28 Mar 2023 04:19:19 +0100 From: void To: freebsd-hackers@freebsd.org Subject: what's the Correct git update method keeping local changes Message-ID: Mail-Followup-To: freebsd-hackers@freebsd.org List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline X-Spamd-Result: default: False [-4.17 / 15.00]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; NEURAL_HAM_SHORT(-0.47)[-0.473]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.26]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm2,messagingengine.com:s=fm2]; RWL_MAILSPIKE_GOOD(-0.10)[66.111.4.26:from]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.26:from]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:19151, ipnet:66.111.4.0/24, country:US]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[f-m.fm]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4Plvzv4NMTz3CX2 X-Spamd-Bar: ---- X-ThisMailContainsUnwantedMimeParts: N Hello hackers@ I looked in the porters and developers handbooks and couldn't find reference to a Correct method of working with git, poudriere, the ports tree and local changes for a use case like mine. Right now my workflow looks like this: 1. apply patch either with patch -p0 < patchfile or git apply from the top of the ports tree 2. git stash 3. poudriere ports -u -q 4. git stash pop 5. run the poudriere build then, subsequent poudriere builds need steps 2-5 repeated. I'm wary of git merge/apply because i'm not a dev and so don't want to push changes. but I want to update the ports tree for poudriere with local changes keeping them local. What's the best way? -- From nobody Tue Mar 28 03:40:00 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PlwRq0jcFz42WKy for ; Tue, 28 Mar 2023 03:40:07 +0000 (UTC) (envelope-from void@f-m.fm) Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PlwRp2mzXz3Fk1 for ; Tue, 28 Mar 2023 03:40:06 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm2 header.b=hQeo1Vl5; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=RCOOpyCt; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 66.111.4.26 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 126BA5C01D4 for ; Mon, 27 Mar 2023 23:40:06 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Mon, 27 Mar 2023 23:40:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm2; t=1679974806; x=1680061206; bh=6j 3pWlOZydRJTLPImNG6LNADGbr3Bou7xlj1bXe2Zmg=; b=hQeo1Vl58gi+RJeKLK Ajaa9Qw20Jxcl5OVka2KhdgOoi6RnjuZp1wLiiS25YzNzv6kLuqhJGA7LZ85E+S8 57QwdJhT5m7ePr2BtDlo2o304AXtkBL8W4ZzsqGnmORD8bQXaDIjkSMXFY3JLrHw 467lSRajZ/FxY4okSc0a1UomLnYrAXp6uM6Gkme1Re7dnn1Z7ITxU4H1jNHmSwbr JEcWipejRvObzbASA0zH5GbFuk3HvZl8cyDrAxxoGWzP8tktEHJY033NN2yMjhk2 35zM69KD9qr+nmNGzwyaBqZCAl3r6JCvpnGqCBNVLXMBgiwqE1td00IGdrmQmmtu zVWg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; t=1679974806; x=1680061206; bh=6j3pWlOZydRJT LPImNG6LNADGbr3Bou7xlj1bXe2Zmg=; b=RCOOpyCtbEJk/ExRahmoemPsfhwpT +SkJhI6ysWCE0zqB6cr8f64sz1kyluPgCaVnROruLkfhGcWW9py3/IF4DWKWZFPF VHgHmBfkR+aduZNnTA+8vUiC5JCJDj32xv9gUTvS5R7q1PVoiwHuk+Zw1T+iyu9S jeLYSnaJjI56hhogdZR0N5mq/ClQab3RbLx9wADxV0oWwkvVkWKnb2R3IdlXMpZc 3XqZlVHqz67l49nkUyceDlU4G6Lvsffdq7zIWiSdvlhWyNJoJolh2xC7Xbj5bMNM 794Y8gM0aTIK8Q1NhL/EY2RGjoc4bed6rWrirm7zWJjWKisGslzPynqcQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdehfedgjeeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggujgesthdtro dttddtvdenucfhrhhomhepvhhoihguuceovhhoihgusehfqdhmrdhfmheqnecuggftrfgr thhtvghrnhepieevgfefgeffhffggfeitdejjefhteeffefgleefieevleevjeeuieduge elgffgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhep vhhoihgusehfqdhmrdhfmh X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Mon, 27 Mar 2023 23:40:05 -0400 (EDT) Date: Tue, 28 Mar 2023 04:40:00 +0100 From: void To: freebsd-hackers@freebsd.org Subject: Re: User-Agent: and In-Reply-To: Message-ID: Mail-Followup-To: freebsd-hackers@freebsd.org References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: X-Spamd-Result: default: False [-4.03 / 15.00]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; NEURAL_HAM_SHORT(-0.33)[-0.330]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.26:c]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm2,messagingengine.com:s=fm2]; RWL_MAILSPIKE_GOOD(-0.10)[66.111.4.26:from]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.26:from]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:19151, ipnet:66.111.4.0/24, country:US]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[f-m.fm]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4PlwRp2mzXz3Fk1 X-Spamd-Bar: ---- X-ThisMailContainsUnwantedMimeParts: N On Mon, Mar 27, 2023 at 04:04:31PM +0200, Peter wrote: >list, as most people on the list tend to Cc: to the personal mail >address anyway, I've never understood the practice of Cc: on a list both parties are implicitly subscribed to. -- From nobody Tue Mar 28 03:55:07 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PlwnC4kbjz42WrD for ; Tue, 28 Mar 2023 03:55:11 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from omta001.cacentral1.a.cloudfilter.net (omta001.cacentral1.a.cloudfilter.net [3.97.99.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PlwnB54R7z3HFr for ; Tue, 28 Mar 2023 03:55:10 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of cy.schubert@cschubert.com has no SPF policy when checking 3.97.99.32) smtp.mailfrom=cy.schubert@cschubert.com; dmarc=none Received: from shw-obgw-4002a.ext.cloudfilter.net ([10.228.9.250]) by cmsmtp with ESMTP id gqTmperLjuZMSh0QPpZjvA; Tue, 28 Mar 2023 03:55:09 +0000 Received: from spqr.komquats.com ([70.66.148.124]) by cmsmtp with ESMTPA id h0QOp45psyAOeh0QPp3Bsp; Tue, 28 Mar 2023 03:55:09 +0000 X-Authority-Analysis: v=2.4 cv=e5oV9Il/ c=1 sm=1 tr=0 ts=6422651d a=Cwc3rblV8FOMdVN/wOAqyQ==:117 a=Cwc3rblV8FOMdVN/wOAqyQ==:17 a=kj9zAlcOel0A:10 a=k__wU0fu6RkA:10 a=YxBL1-UpAAAA:8 a=6I5d2MoRAAAA:8 a=EkcXrb_YAAAA:8 a=pyn-8fn7wKyj4KAFgmcA:9 a=CjuIK1q_8ugA:10 a=Ia-lj3WSrqcvXOmTRaiG:22 a=IjZwj45LgO3ly-622nXo:22 a=LK5xJRSDVpKd5WXXoEvA:22 Received: from slippy.cwsent.com (slippy [10.1.1.91]) by spqr.komquats.com (Postfix) with ESMTP id DB8B4E8E for ; Mon, 27 Mar 2023 20:55:07 -0700 (PDT) Received: by slippy.cwsent.com (Postfix, from userid 1000) id 9B8632CD; Mon, 27 Mar 2023 20:55:07 -0700 (PDT) X-Mailer: exmh version 2.9.0 11/07/2018 with nmh-1.8+dev Reply-to: Cy Schubert From: Cy Schubert X-os: FreeBSD X-Sender: cy@cwsent.com X-URL: http://www.cschubert.com/ To: freebsd-hackers@freebsd.org Subject: Re: what's the Correct git update method keeping local changes In-reply-to: References: Comments: In-reply-to void message dated "Tue, 28 Mar 2023 04:19:19 +0100." List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Mon, 27 Mar 2023 20:55:07 -0700 Message-Id: <20230328035507.9B8632CD@slippy.cwsent.com> X-CMAE-Envelope: MS4xfPHYr27aNnC5D2n3wO0y09ZBmLJhuTcpAncVNHrJ/hFK6nV5P0g1Gu5Uv3kn+yTrzI0dU5kS520bqqDiLPmUvwirlaPEwMqCHGvndMw7GP7w9BLs94d1 3j44W4kU0ii39GIrH6wTvSdeK5vHQ9teLt/HLekebFEXvdWNG8ISbMMXCo9299qqJRMs/OZLCEpvU2itFk3iczVskVteJeiUOBI= X-Spamd-Result: default: False [-1.79 / 15.00]; AUTH_NA(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_LONG(-1.00)[-0.999]; NEURAL_HAM_SHORT(-1.00)[-0.996]; MV_CASE(0.50)[]; RCVD_IN_DNSWL_MED(-0.20)[3.97.99.32:from]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[cschubert.com: no valid DMARC record]; FROM_HAS_DN(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; REPLYTO_EQ_FROM(0.00)[]; HAS_REPLYTO(0.00)[Cy.Schubert@cschubert.com]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:16509, ipnet:3.96.0.0/15, country:US]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[3.97.99.32:from] X-Rspamd-Queue-Id: 4PlwnB54R7z3HFr X-Spamd-Bar: - X-ThisMailContainsUnwantedMimeParts: N In message , void writes: > Hello hackers@ > > I looked in the porters and developers handbooks and couldn't find reference > to a Correct method of working with git, poudriere, the ports tree and > local changes for a use case like mine. > > Right now my workflow looks like this: > > 1. apply patch either with patch -p0 < patchfile or git apply from the > top of the ports tree > 2. git stash > 3. poudriere ports -u -q > 4. git stash pop > 5. run the poudriere build That's one way to do it. > > then, subsequent poudriere builds need steps 2-5 repeated. > > I'm wary of git merge/apply because i'm not a dev and so don't want to push > changes. but I want to update the ports tree for poudriere with local changes > keeping them local. > > What's the best way? > -- > I maintain my own branch which contains my patches. In fact I maintain a branch for each patch. I then do a git cherry-pick from each patch branch to my own branch. To update I git fetch followed by a git rebase origin/main while within my branch. This brings my branch up to date. I maintain a separate repo, owned by a separate account, from which my poudriere builds its packages. This is a little extra work but keeps my development work totally separate from anything I'm developing. I do the same with my src tree, which has patches that I have no intention of committing, yet, if ever. -- Cheers, Cy Schubert FreeBSD UNIX: Web: https://FreeBSD.org NTP: Web: https://nwtime.org e^(i*pi)+1=0 From nobody Tue Mar 28 04:12:41 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PlxRl3lstz42YcD for ; Tue, 28 Mar 2023 04:25:07 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from omta001.cacentral1.a.cloudfilter.net (omta001.cacentral1.a.cloudfilter.net [3.97.99.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PlxRl08Npz3KvF for ; Tue, 28 Mar 2023 04:25:07 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of cy.schubert@cschubert.com has no SPF policy when checking 3.97.99.32) smtp.mailfrom=cy.schubert@cschubert.com; dmarc=none Received: from shw-obgw-4004a.ext.cloudfilter.net ([10.228.9.227]) by cmsmtp with ESMTP id goEspecyVuZMSh0hPpZldP; Tue, 28 Mar 2023 04:12:43 +0000 Received: from spqr.komquats.com ([70.66.148.124]) by cmsmtp with ESMTPA id h0hNpq3lf3fOSh0hOp4nsE; Tue, 28 Mar 2023 04:12:43 +0000 X-Authority-Analysis: v=2.4 cv=J8G5USrS c=1 sm=1 tr=0 ts=6422693b a=Cwc3rblV8FOMdVN/wOAqyQ==:117 a=Cwc3rblV8FOMdVN/wOAqyQ==:17 a=kj9zAlcOel0A:10 a=k__wU0fu6RkA:10 a=YxBL1-UpAAAA:8 a=6I5d2MoRAAAA:8 a=EkcXrb_YAAAA:8 a=FPq7eYf2pJhiWMlH9ckA:9 a=CjuIK1q_8ugA:10 a=Ia-lj3WSrqcvXOmTRaiG:22 a=IjZwj45LgO3ly-622nXo:22 a=LK5xJRSDVpKd5WXXoEvA:22 Received: from slippy.cwsent.com (slippy [10.1.1.91]) by spqr.komquats.com (Postfix) with ESMTP id 7D88FEB7 for ; Mon, 27 Mar 2023 21:12:41 -0700 (PDT) Received: by slippy.cwsent.com (Postfix, from userid 1000) id 5D8A7431; Mon, 27 Mar 2023 21:12:41 -0700 (PDT) X-Mailer: exmh version 2.9.0 11/07/2018 with nmh-1.8+dev Reply-to: Cy Schubert From: Cy Schubert X-os: FreeBSD X-Sender: cy@cwsent.com X-URL: http://www.cschubert.com/ To: freebsd-hackers@freebsd.org Subject: Re: User-Agent: and In-Reply-To: In-reply-to: References: Comments: In-reply-to void message dated "Tue, 28 Mar 2023 04:40:00 +0100." List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Mon, 27 Mar 2023 21:12:41 -0700 Message-Id: <20230328041241.5D8A7431@slippy.cwsent.com> X-CMAE-Envelope: MS4xfPOJAgG0GE9yTUIDUQA5a3WEkAoSn6fkcRAdQZujzJU30vWv4zjjRr3uAOj1a4Dzy3ZaWl20t7RDEdH4rDYwY1/u1wo8z2BInspOjQlXdmwIo5/pilWX 4zG1by5sN63itAyT/F72J9iw7aljdL5ON/wKoBunDezpZWxu6D4kuU5UlCLkckg//qjsf5EFtkN4OVDUdbwcDnxpBlKFGNknKtg= X-Spamd-Result: default: False [1.71 / 15.00]; R_BAD_CTE_7BIT(3.50)[7bit]; AUTH_NA(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_LONG(-1.00)[-0.997]; NEURAL_HAM_SHORT(-1.00)[-0.996]; MV_CASE(0.50)[]; RCVD_IN_DNSWL_MED(-0.20)[3.97.99.32:from]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; HAS_REPLYTO(0.00)[Cy.Schubert@cschubert.com]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[cschubert.com: no valid DMARC record]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_NA(0.00)[no SPF record]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; FROM_HAS_DN(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; RWL_MAILSPIKE_POSSIBLE(0.00)[3.97.99.32:from]; FROM_EQ_ENVFROM(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; REPLYTO_EQ_FROM(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:16509, ipnet:3.96.0.0/15, country:US]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; GREYLIST(0.00)[pass,body] X-Rspamd-Queue-Id: 4PlxRl08Npz3KvF X-Spamd-Bar: + X-ThisMailContainsUnwantedMimeParts: N In message , void writes: > On Mon, Mar 27, 2023 at 04:04:31PM +0200, Peter wrote: > > >list, as most people on the list tend to Cc: to the personal mail > >address anyway, > > I've never understood the practice of Cc: on a list both parties are > implicitly subscribed to. Because some of us use procmail, slocal, or other rule processors. Emails sent to the mailing list go into a folder. Emails which have any of my email addresses in the to or cc line will also end up in my inbox folder. I look at inbox before I look at anything else. Emails to or cc me will always get my attention. Emails destined to the mailing list will eventually be read but, depending on workload, $JOB, life, etc., maybe not today. Also a to: or cc: to any of my email addresses will also be sent to my phone in case I'm AFK for a while, also making sure those emails get my immediate attention. I wouldn't doubt others may implement the same policy using procmail or even Outlook (which has powerful processing rules). So, yeah, to and cc while also cc the ML does have its place. Regarding replies, MUAs that don't fill in references will break threading. As an MH user my MUA has been configured to specifically add fill in the references header to make sure the ML will thread properly. Not all MUAs do this and it's annoying. -- Cheers, Cy Schubert FreeBSD UNIX: Web: https://FreeBSD.org NTP: Web: https://nwtime.org e^(i*pi)+1=0 From nobody Tue Mar 28 07:59:45 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pm2CS0KSrz41YnW for ; Tue, 28 Mar 2023 07:59:48 +0000 (UTC) (envelope-from mandree@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pm2CR6ycYz3vYJ for ; Tue, 28 Mar 2023 07:59:47 +0000 (UTC) (envelope-from mandree@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679990388; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Otwc8BfnyP6L59n/ekTfOTZd8OPcXCjQHYETJZBhWgg=; b=jl08vCcQsLjDF6PGQTQJ45Qcb8zL+ayApyNqymKklF7YD1Uoc3eJaV0VyVKLQ23OHhU2sY 9RONb4dYj26xSfl8zUcEqw2iuumgJh663BBcsWYVwAwbT2bf182ziaANMmdH03CMByM+i4 wafOahBqlJBSe3JAZfvvcaRuE9e2GXFa8OuXccvNN76U4lWvWfpQiodvdXLFtrn8X6Ulpi XuGS6IX2puhagMTeIDakH1fsWd+CObHKfbamsICz8i2dBiXhLmprSMJdfalHJqABHlXubg BcTxA9yyvNnYgT5J9lQIidf8fnXxr2AmkA2QzXA1HzIIuQLMADa88/yNG60blA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679990388; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Otwc8BfnyP6L59n/ekTfOTZd8OPcXCjQHYETJZBhWgg=; b=cIDfXlB8nGR5OHD0t/dnsWIOv9X8NzKjMZBk1zODpnc15Olk81gLBLiCMAVFAsU2aIo4XF cudNHxBGkjO0aA+rN6o2G05yVEvsoqo2N64jiwMgJ5GCu/JoDmhlWMEjBDyKlpKg7K5yet 35fpBfzXouLHuKK20Bt2nvDEgURMFdX+yc7QvK2flBf+C49cJ2ilIzJXmrV9kKi/VsaMdv RPWn6wTfOpjCxET859m0mb6V5TJbho9zmRqhhNcMzEiPrR0U4/EiGe1axt19hlZlbSdUUc GLvoznd2NaoYcZgyE7ouinL1fPszfaPyKKfp1kKfSyp/llDTQxyd5e+GwGaxXg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679990388; a=rsa-sha256; cv=none; b=QWkSXokakzA/+bkktyaoM45kzmWa+NlVg/k0BdeAeuUEmR1KmGmblPDJRtzpylC4jXHiIT LRaFyRaOvGD8esTdpEQWubq/hp/Z+AC36DFBiKs092aJ1GNQzEmeObdxEIlOb/SYiyCGax IjUqdV1XLbccCTePTUxINKmoT4JWYWqDBewJMC70bGq9Y5ZHBkAiIAsvv/2pZK4BK/leMR H4klLLzgvxkdWQGGorfLJaM2lb3FX+pETXJKewDPXHIzjLnsdtGZGWi9w5bphKu4GMocZw mvfDEV7LLAAM7/jifTIsuYZRIzFndcBbisYdKVTv4AIb21ruOAT/ikMJ6uLGnw== Received: from mandree.no-ip.org (pd9e073a3.dip0.t-ipconnect.de [217.224.115.163]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: mandree/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Pm2CR53h9zXTg for ; Tue, 28 Mar 2023 07:59:47 +0000 (UTC) (envelope-from mandree@FreeBSD.org) Received: from [127.0.0.1] (localhost [127.0.0.1]) by ryzen.an3e.de (Postfix) with ESMTP id 0DE6BDC62A8 for ; Tue, 28 Mar 2023 09:59:46 +0200 (CEST) Message-ID: <708822f5-2a2f-35f7-8f79-bf48ff644a22@FreeBSD.org> Date: Tue, 28 Mar 2023 09:59:45 +0200 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 Subject: Re: what's the Correct git update method keeping local changes Content-Language: en-US To: freebsd-hackers@freebsd.org References: From: Matthias Andree Organization: FreeBSD.org In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-ThisMailContainsUnwantedMimeParts: N Am 28.03.23 um 05:19 schrieb void: > Hello hackers@ > > I looked in the porters and developers handbooks and couldn't find > reference > to a Correct method of working with git, poudriere, the ports tree and > local changes for a use case like mine. > > Right now my workflow looks like this: > > 1. apply patch either with patch -p0 < patchfile or git apply from the >    top of the ports tree > 2. git stash > 3. poudriere ports -u -q > 4. git stash pop > 5. run the poudriere build > > then, subsequent poudriere builds need steps 2-5 repeated. > > I'm wary of git merge/apply because i'm not a dev and so don't want to push > changes. but I want to update the ports tree for poudriere with local > changes > keeping them local. > > What's the best way? So - there's some discrepancy in your tools there, you are using git to keep local patches, and poudriere to update. This seems odd. Is poudriere using git to update your ports tree? Portsnap? Something else? We don't know. I personally do not use poudriere ports -u, in fact I am maintaining my /usr/ports in Git (but then I occasionally push from my tree into FreeBSD's), and poudriere knows the ports tree with a "null" method, and then 2...4 become a simple "git pull --rebase --autostash". > $ poudriere ports -l > PORTSTREE METHOD TIMESTAMP PATH > default null 2023-03-27 05:59:35 /usr/ports -- Matthias Andree FreeBSD ports committer From nobody Tue Mar 28 08:39:32 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pm35c0j7Wz41bjf for ; Tue, 28 Mar 2023 08:39:48 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ed1-x52c.google.com (mail-ed1-x52c.google.com [IPv6:2a00:1450:4864:20::52c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pm35b1L7Tz41kZ for ; Tue, 28 Mar 2023 08:39:47 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20210112.gappssmtp.com header.s=20210112 header.b=k1ZYDjJW; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2a00:1450:4864:20::52c) smtp.mailfrom=wlosh@bsdimp.com; dmarc=none Received: by mail-ed1-x52c.google.com with SMTP id i5so46586798eda.0 for ; Tue, 28 Mar 2023 01:39:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20210112.gappssmtp.com; s=20210112; t=1679992783; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=XGJPCqVIEBJHySo1gQ+noYX37RvJQ5bDb7G5VAuuNLU=; b=k1ZYDjJWTzu63Bs06+RLZ42756i8VagYWx2NXkp/vv5uyrbEdaoEnPoW5SVnP8UPhv 9sACuM1N2HhvGS01WfuHJgFCl85yXjWnO+qA1ICE/NdnVLwSXIv0dBKr75qBpVrki+/Y bH/HU/DR4WlTzq3JBRNEPGcAf8303W1PKMPxNmYeD16SFi4fzwIxSlXqR8bcobbnX8nC OeaDvOlDiMbd3aiApJSo7de3rXvgnPcQkft3yFWe4vZGApNGIfc7Xf9diz/ZfiNJkdtS D9MRYZhmFP6jcB1AkGsjtIqxmvSCQUUWk3e3Sr9yY2KV8pl0KkuDrwLhJikmgHBRUpCT V5rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679992783; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XGJPCqVIEBJHySo1gQ+noYX37RvJQ5bDb7G5VAuuNLU=; b=PpSgxPUZHAXrFENmNnzRbIyhY5uqTx2VVBanSR4/e0Hh/hfp+HVlBzW9CxPtSQAOSF TMv64aI4rZ1om6922Yiz5ZTejomEqibyncOms1aNH+/XYw7shG5ls7/dVChJWKFCpYX8 DDLaLnNsGJhLlGw2oeLUJNgk0xPI6KplpaAz59WagUY5aoj3NXRdxlOSCqw8gGdUcxE5 zoya6HGWSAeJ3RleWsr5B/oxMTyl5Q1kIYgrqT8llZ8ORzIaih5WPSuE5ONwLzJyKEm6 Uowee4ZSd3+yaE1GlvzeHlMTeM6LVvXATr3VRwoIwik1Tmt86MWvlliEpTX7hgTTznIL iDpA== X-Gm-Message-State: AAQBX9cc71sQOHeIRr63YM0Kt2Y3aNT20lS3J9NdUcnqQkJjcM5uJw7A QzTbEoM2R8pLJcEcFEMGgL4Z0ultworVxXJUn+mTSeo8xmgxcFhW5GA= X-Google-Smtp-Source: AKy350aPPosBgPaN+OAb4wCh2ne/DYcpPG++6prXKBWvIcHIQ8/zEL2uUZlLBxy8KNUJIVXtQvoArCeCwigWWZxpDCs= X-Received: by 2002:a50:8712:0:b0:4fa:123:3b32 with SMTP id i18-20020a508712000000b004fa01233b32mr7484733edb.7.1679992783209; Tue, 28 Mar 2023 01:39:43 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Tue, 28 Mar 2023 10:39:32 +0200 Message-ID: Subject: Re: what's the Correct git update method keeping local changes To: FreeBSD Hackers Content-Type: multipart/alternative; boundary="0000000000001195ca05f7f1cd27" X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20210112.gappssmtp.com:s=20210112]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::52c:from]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:~]; R_SPF_NA(0.00)[no SPF record]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20210112.gappssmtp.com:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[bsdimp.com]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4Pm35b1L7Tz41kZ X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N --0000000000001195ca05f7f1cd27 Content-Type: text/plain; charset="UTF-8" On Tue, Mar 28, 2023, 5:19 AM void wrote: > Hello hackers@ > > I looked in the porters and developers handbooks and couldn't find > reference > to a Correct method of working with git, poudriere, the ports tree and > local changes for a use case like mine. > I use the rebase workflow for pending ports commit work. Each change is a commit that I'm "curating" for later. I update main and then rebase -i in case there is anything weird I need to do. This is true for changes I get from bz or stuff I'm working on. This aversion dates back to mercurial patch queues / stash screwing me over a few times 15 or 20 years ago.. Warner Right now my workflow looks like this: > > 1. apply patch either with patch -p0 < patchfile or git apply from the > top of the ports tree > 2. git stash > 3. poudriere ports -u -q > 4. git stash pop > 5. run the poudriere build > > then, subsequent poudriere builds need steps 2-5 repeated. > > I'm wary of git merge/apply because i'm not a dev and so don't want to push > changes. but I want to update the ports tree for poudriere with local > changes > keeping them local. > > What's the best way? > -- > > --0000000000001195ca05f7f1cd27 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Tue, Mar 28, 2023, 5:19 AM void <void@f-m.fm> wrote:
Hello hackers@

I looked in the porters and developers handbooks and couldn't find refe= rence
to a Correct method of working with git, poudriere, the ports tree and
local changes for a use case like mine.

I use the rebase workflow for pendin= g ports commit work. Each change is a commit that I'm "curating&qu= ot; for later. I update main and then rebase -i in case there is anything w= eird I need to do.

This = is true for changes I get from bz or stuff I'm working on.

This aversion dates back to mercuria= l patch queues / stash screwing me over a few times 15 or 20 years ago..=C2= =A0

Warner

Right now my workflow looks like this:

1. apply patch either with patch -p0 < patchfile or git apply from the <= br> =C2=A0 =C2=A0 top of the ports tree
2. git stash
3. poudriere ports -u -q
4. git stash pop
5. run the poudriere build

then, subsequent poudriere builds need steps 2-5 repeated.

I'm wary of git merge/apply because i'm not a dev and so don't = want to push
changes. but I want to update the ports tree for poudriere with local chang= es
keeping them local.

What's the best way?
--

--0000000000001195ca05f7f1cd27-- From nobody Tue Mar 28 09:02:33 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pm3c4544Kz41d8d for ; Tue, 28 Mar 2023 09:02:44 +0000 (UTC) (envelope-from developer@lorenzosalvadore.it) Received: from mail-4018.proton.ch (mail-4018.proton.ch [185.70.40.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pm3c33Zn8z44fD for ; Tue, 28 Mar 2023 09:02:43 +0000 (UTC) (envelope-from developer@lorenzosalvadore.it) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=lorenzosalvadore.it header.s=protonmail2 header.b=hxggtIGk; spf=pass (mx1.freebsd.org: domain of developer@lorenzosalvadore.it designates 185.70.40.18 as permitted sender) smtp.mailfrom=developer@lorenzosalvadore.it; dmarc=pass (policy=quarantine) header.from=lorenzosalvadore.it Date: Tue, 28 Mar 2023 09:02:33 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lorenzosalvadore.it; s=protonmail2; t=1679994161; x=1680253361; bh=PO2G/e6FG3HdIUPQNUrRlzkug61xv3yVMDj483DXuiY=; h=Date:To:From:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=hxggtIGkxmR7/KRrFuv3YuMLr5yC/vRSTM5oq+2CJ4UKGRi+UIX25nMZwqhm1hoi8 WJIf7wzjIZd+APGBFWQmXkxGimitq2JTdtZg49IyF2mieuuzcPLNXsqr2dWxwtwwYQ 34ntpe3mw5dq84VmZCjVpkn5G+VVrB/qWrnpU8g1ItgtE2B0dnjYG0oGdkar8mGtBk yzD/QlQK+R4dOrxt/MaGJon0KRTv0mSxJXqUCfXCi/0Lb7DMMapU+Sph3sDzVL9XfQ 3qSn9x8GzS/mZswxTJ+LVnTBzDSwOOaf7Xg08Dtp67DGecjpQkGB0/RBa4pP6eZ1HH +aBLrQ//OkxhA== To: freebsd-hackers@freebsd.org From: Lorenzo Salvadore Subject: Re: what's the Correct git update method keeping local changes Message-ID: <4IA4uMvgbs6IP7g48WgKrv0HhDF-NDuLE4SYSuGlJGG7QcCUzl-OSTKiwkjd3GboJO_bE2lQZJ_Y6bRt2lfWw8VY-EtkONHgXg3EX2YLquI=@lorenzosalvadore.it> In-Reply-To: References: <708822f5-2a2f-35f7-8f79-bf48ff644a22@FreeBSD.org> Feedback-ID: 53711648:user:proton List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; DMARC_POLICY_ALLOW(-0.50)[lorenzosalvadore.it,quarantine]; R_SPF_ALLOW(-0.20)[+ip4:185.70.40.0/24:c]; R_DKIM_ALLOW(-0.20)[lorenzosalvadore.it:s=protonmail2]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_ZERO(0.00)[0]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:62371, ipnet:185.70.40.0/24, country:CH]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[lorenzosalvadore.it:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4Pm3c33Zn8z44fD X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N ------- Original Message ------- On Tuesday, March 28th, 2023 at 9:59 AM, Matthias Andree mandree@FreeBSD.or= g wrote: > Am 28.03.23 um 05:19 schrieb void: >=20 > > Hello hackers@ > >=20 > > I looked in the porters and developers handbooks and couldn't find > > reference > > to a Correct method of working with git, poudriere, the ports tree and > > local changes for a use case like mine. > >=20 > > Right now my workflow looks like this: > >=20 > > 1. apply patch either with patch -p0 < patchfile or git apply from the > > top of the ports tree > > 2. git stash > > 3. poudriere ports -u -q > > 4. git stash pop > > 5. run the poudriere build > >=20 > > then, subsequent poudriere builds need steps 2-5 repeated. > >=20 > > I'm wary of git merge/apply because i'm not a dev and so don't want to = push > > changes. but I want to update the ports tree for poudriere with local > > changes > > keeping them local. > >=20 > > What's the best way? >=20 > So - there's some discrepancy in your tools there, you are using git to > keep local patches, and poudriere to update. This seems odd. Is > poudriere using git to update your ports tree? Portsnap? Something > else? We don't know. I agree with Matthias Andree, using poudriere to update is odd. I use git worktrees for my workflow. I have three trees: - in one tree I have an exact copy of upstream; - in another tree I have a copy of upstream with my local patches applied; - the last tree is for testing patches with poudriere before I push them or submit them for review. > I'm wary of git merge/apply because i'm not a dev and so don't want to pu= sh > changes. Do not worry about that: if you are not a dev, you do not have the necessar= y permissions to push any change by accident. You can experiment safely. Cheers, Lorenzo Salvadore From nobody Tue Mar 28 10:00:05 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pm4tK5kmHz41hWN for ; Tue, 28 Mar 2023 10:00:09 +0000 (UTC) (envelope-from zirias@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pm4tK5C33z486q for ; Tue, 28 Mar 2023 10:00:09 +0000 (UTC) (envelope-from zirias@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679997609; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=2u5ZAEamPhYPzdLGyrKE7BBnXqLjlp84X1mCCMk/VTY=; b=Xh53LrM/IYQaTKNLW87POYebMszv4qcJzVJ7E9J6xE/lXIZHjVzYaJZKC07FNAdxWW1acf EuzbygstHkb2ZhoEEoMRJQQPd1SsKJfX2jDlFAlZo046f8lsccPjecwWlQC7rLlhkrKtxO 8G9xnroRM3w7jxhkaSnuS2Zx8jlUpmAPbKIJdVQ8RnPw7t7jTI/c77JWoHGmv+G+XQMEN2 bnur1leEbzb/7nQTOnW/jwesKtxmmwwj0HHJP/l404ugSIoFSskfWsXqZjEoKWGlbSPft2 7qYSi9yvEUd4E1O9QlXbQKqwNORN+7CfKihBJ2XFsO39qKpis33hhyf3NcJPKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679997609; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2u5ZAEamPhYPzdLGyrKE7BBnXqLjlp84X1mCCMk/VTY=; b=HD7bzU2DT3IRp39mjGXp+9ai1TnnXN5Kg7ZQFF+vc9Vnhc9EfSO6JspOX89DGVnJBmGuUT LoDyw3iJ4Ma0wjVBvxT8FpoK6f2vPxJoUvJEHiJRiQpY2XkumWnguarNPm9pMGyRjVPf8Y zNaqDGKs+7vHPsljUXiVbPYnGCV+2WWFRqnzj0Gz/Cgroj7s08uD3KU4wSknTmLVQOAqbO f4tbj1vF0VsY2iWsqXlTwNG/QlxYyYyiO5tYRr7QfFK44nvP6Sei0TliDELOVx9fTU5ZLG HuMMmJ9hsk5cpmQNGAiVVDdOPHn7HsRlrCeFPPChQEc67+SjVho4O1m+ttfCgg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679997609; a=rsa-sha256; cv=none; b=cuQIB8msH2wh/z4nYhTyHmQ83OCRsJDVYXVEOHJ/5Kgf/hwwGk+VYy4HqCeVmHZ3Iw6RH2 cdiamxQxVTHKVOdbKEXnEZNodtgjF5PRmSAV+sfiBUjlQpgZ1Wjm2RmR5croYYTah04kH2 dmevqtQcL7Egj5igoUkjUNFPe+KaZwFqx/m0ZDyngeWje082nvIA+2Wu9/YWpzu4uQHocu wOWUt+44zXHEkUOTtD7sMjwOejnyRBBai6ucp7fTQ9CPzCxPbZFrX4mNNzqRengWfwYIYV qy0j0hYqniZBM4tRhm9Ay3/UWQRkd/Z2onOjX5O17qRtX6yyEuCywu/14QY/3A== Received: from stef.palmen-it.de (stef.palmen-it.de [IPv6:2001:470:1f0b:bbb:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: zirias/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Pm4tK47tlzYtK for ; Tue, 28 Mar 2023 10:00:09 +0000 (UTC) (envelope-from zirias@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=palmen-it.de; s=20200414; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:To:From:Date:Sender:Reply-To:Cc: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=2u5ZAEamPhYPzdLGyrKE7BBnXqLjlp84X1mCCMk/VTY=; b=NjmoAgB5XZfMg+borvrVy0K7fz lhp0+Zpw79WTvFWhVRfe7YWOQEOX/4dDlV2HKLCHiJS5NvJ0ydUnBiH02dqSoZ2apCLD1VYLBDmFL iCScuSqmrQ3AT8sb/GP3O/UwEwO/B7rZadqpvh3AyoJuKvO4thfew73RSQlKFOUl2boZGZxQyU9mS +xwVAEHVe2E906rWy6oXvIkcqtU3qUrYbsr2gJfsK3ATf3cIgUXM13rH6S+1SDH+SrskYn0ymyHZY Hxtn7BhRwV5Mz+Zy1RYGeHRQWIgMs4QcwZA1aJ0yMFKvQVggMbyWMQbbbE00xyqMVcrCXsko4YWej sbS1VVWg==; Received: from [192.168.71.101] (helo=mail.home.palmen-it.de) by stef.palmen-it.de with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1ph67b-002oHS-1G for freebsd-hackers@freebsd.org; Tue, 28 Mar 2023 12:00:07 +0200 Received: from nexus.home.palmen-it.de ([192.168.99.2]) by mail.home.palmen-it.de with esmtpsa (TLS1.3) tls TLS_CHACHA20_POLY1305_SHA256 (Exim 4.95 (FreeBSD)) (envelope-from ) id 1ph67a-000Hqx-LX for freebsd-hackers@freebsd.org; Tue, 28 Mar 2023 10:00:06 +0000 Date: Tue, 28 Mar 2023 12:00:05 +0200 From: Felix Palmen To: freebsd-hackers@freebsd.org Subject: Re: what's the Correct git update method keeping local changes Message-ID: <20230328100005.63necvdkvx4y7zyo@nexus.home.palmen-it.de> Mail-Followup-To: freebsd-hackers@freebsd.org X-Face: /1K@t"h.}e~pR@]c7HorQ!T`F^RJCa'BCr#e>IKA{>C/9OTGB4|xh"y2{?1Z5M i2w"AH^pN_LlHR^{+f',_Np~;.B;!M/bL}*qk]p5*r7F5vW};{:@4u5S?T&f0$7BJ-71Q5SV]:v$`5 A0[DZ:=?S52x8HJ~5@^P_\T@MsjG{R( Organization: FreeBSD.org References: <20230328035507.9B8632CD@slippy.cwsent.com> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="7nkp2fbrsmgnrkf3" Content-Disposition: inline In-Reply-To: <20230328035507.9B8632CD@slippy.cwsent.com> User-Agent: NeoMutt/20220429 X-ThisMailContainsUnwantedMimeParts: N --7nkp2fbrsmgnrkf3 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable * Cy Schubert [20230327 20:55]: > I maintain my own branch which contains my patches. In fact I maintain a= =20 > branch for each patch. I then do a git cherry-pick from each patch branch= =20 > to my own branch. +1 for just maintaining your local branch, that's the cleanest way to do it. Git is pretty flexible, so many variations are possible. I personally do it the other way around, I just have two branches, "local" and "wip", with local containing anything I want to include in my poudriere bulk builds and "wip" containing other development stuff. For putting something up for review, I create a new branch from "main" and cherry-pick the relevant commits from "local" or "wip". Updating my branches is as simple as git checkout local git fetch upstream main:main git rebase main (and similar for my "wip" branch if it currently contains something) There, "upstream" is my remote pointing at the official ports repository hosted on freebsd.org. --=20 Felix Palmen {private} felix@palmen-it.de -- ports committer (mentee) -- {web} http://palmen-it.de {pgp public key} http://palmen-it.de/pub.txt {pgp fingerprint} 6936 13D5 5BBF 4837 B212 3ACC 54AD E006 9879 F231 --7nkp2fbrsmgnrkf3 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iNUEABYKAH0WIQRpNhPVW79IN7ISOsxUreAGmHnyMQUCZCK6nl8UgAAAAAAuAChp c3N1ZXItZnByQG5vdGF0aW9ucy5vcGVucGdwLmZpZnRoaG9yc2VtYW4ubmV0Njkz NjEzRDU1QkJGNDgzN0IyMTIzQUNDNTRBREUwMDY5ODc5RjIzMQAKCRBUreAGmHny McHSAQCM3DBpvOPJVPuKAcBOsLeVC5P4KWuHbO+cC4u36G/tNAEA+KdtWhWb2iwR kLm18iODmUiKMex5dJODNiBxg2gTqAc= =JZFf -----END PGP SIGNATURE----- --7nkp2fbrsmgnrkf3-- From nobody Tue Mar 28 10:12:02 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pm58512Vdz41htX for ; Tue, 28 Mar 2023 10:12:05 +0000 (UTC) (envelope-from zirias@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pm5850TYcz4Cbb for ; Tue, 28 Mar 2023 10:12:05 +0000 (UTC) (envelope-from zirias@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679998325; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=J+okNh1mBP8jr8A3NZBKZBvMChjc1ou+XVJmC8GFk+k=; b=vtYwt+wDfkdAmMcxM5OuiE14yCiEkdBtWI6Ku7u4zdLjONFUeeoHbX0eOLS6R+QFBcAX0h wf7MTzvm38WkIoJC8RDF/uLuDcz/nC6axNyiXkq3F/EHsN8NAHtU0QY17JTLbUGeEXJzMP mQdlPNrGIAcSMamYzL6LvQ3tsadFTpEp8f3YL5d4E+7+CkXtUWs3OxCMNsSrFwpf+7zx/E DggG2TTQQq6E5tpD6e1mcxmpiHRfkL3jrF0yJ6P+S/Pq4jpNdwha9fqNskpbL871r6c3IV Do82Lo2d0ErxLrTBpm/KN3zguiIwzs4lWMTqjQ/JWTCBIKh9KM1DK1C292ZrxA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679998325; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references:dkim-signature; bh=J+okNh1mBP8jr8A3NZBKZBvMChjc1ou+XVJmC8GFk+k=; b=wLsVbsFDKulOxRH6eE6o9TQR7ww/BZxL4VikqHeB8SR/xKLXAOeBEG3CkgvYQL0bBmUuDu 0rRHoDWlStw2rMSuQsDsk5fPgQp5+Wkvu32zxAigW4igMQOZ5jWdIyAZmtcUqZNfEVhqvL PUTKi61pUwDfvfZt/3Ll1n20/WXy6iXqpqRVxoCfTfeiCOqWvFBQV5TG0NqHj4JfVhPEAT CPIQGxBWm1zCBhZhUJAhTH4tG8UwP0ALARWJFyYmdiyInYZMILBSp5PicT7EuBym1qVYpY eKGEYYVXbOuvwFvWtmZYFNniBWGeb0AEDXyLUDIyhmoNy7rXeRdYt7rnywY/vQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679998325; a=rsa-sha256; cv=none; b=hZbNulGoGgtAeP+eITQxuERrx6ETeU8gu8bT+t7GMu48Uot3ls149+vG6+CNdTL22Z4hWj P5Rb/+G8TnYMVJcpWI8rlzFZi7bD5WrPidNhKTM1D0m95XQJOZAGeBfcubP2idZmbgh9b+ y+w3je2nmcYV+BbIl/H0ZH24a9iwFRuCtwDwROj0Clqzn22gsl/HXCeGT/ojI4vYC/yNOg RoMswUYFFLzCtPHOi33CLsU51ijDVefJ8y2Qikv9kihKbTfZb5D87M7TDOEAdZnax6xsKY wPc4l6xKJVs3iZCKQJxPhrQq0iiHRtsviaehbg1ctyPxurk8ym52jppJTWXQqA== Received: from stef.palmen-it.de (stef.palmen-it.de [IPv6:2001:470:1f0b:bbb:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: zirias/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Pm5846fffzZdD for ; Tue, 28 Mar 2023 10:12:04 +0000 (UTC) (envelope-from zirias@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=palmen-it.de; s=20200414; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:To:From:Date:Sender:Reply-To:Cc: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=J+okNh1mBP8jr8A3NZBKZBvMChjc1ou+XVJmC8GFk+k=; b=cUedhMUeDFXOAguFNRjnY3UhTz U0sA/3hzg91NFknxn4bVj1GhXxbtgeho+e5rZw512viZiwqRMQBocf5JbG9aORStzELjIWTUXH9oo Oea7zz50ENsFFi2/A4mEzHH/7EWM0Qs3DERNarZ3vRMGj3ZZMg8s9WnX7CVhFWMnYK7lQWjn5jXcE ffqLv6kFJj/ZocHZSfxJ1F7SIRuD9uX0f7bVIe9xebLW2ZLhD+eSrPx+RLxVTm062vKjTpr4A5A4h 6kIzXBqB+ntWZy0XvJXMElNYmZRUsUguCfm10e6b0a2aGQtGrUfVUz/SyMKS3XXQALXiKylKwBg9T SxUyAEfw==; Received: from [192.168.71.101] (helo=mail.home.palmen-it.de) by stef.palmen-it.de with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1ph6J9-002oNR-LY for freebsd-hackers@freebsd.org; Tue, 28 Mar 2023 12:12:03 +0200 Received: from nexus.home.palmen-it.de ([192.168.99.2]) by mail.home.palmen-it.de with esmtpsa (TLS1.3) tls TLS_CHACHA20_POLY1305_SHA256 (Exim 4.95 (FreeBSD)) (envelope-from ) id 1ph6J9-000Huf-49 for freebsd-hackers@freebsd.org; Tue, 28 Mar 2023 10:12:03 +0000 Date: Tue, 28 Mar 2023 12:12:02 +0200 From: Felix Palmen To: freebsd-hackers@freebsd.org Subject: Re: User-Agent: and In-Reply-To: Message-ID: <20230328101202.wxh3o4bsvxr5qnwb@nexus.home.palmen-it.de> Mail-Followup-To: freebsd-hackers@freebsd.org X-Face: /1K@t"h.}e~pR@]c7HorQ!T`F^RJCa'BCr#e>IKA{>C/9OTGB4|xh"y2{?1Z5M i2w"AH^pN_LlHR^{+f',_Np~;.B;!M/bL}*qk]p5*r7F5vW};{:@4u5S?T&f0$7BJ-71Q5SV]:v$`5 A0[DZ:=?S52x8HJ~5@^P_\T@MsjG{R( Organization: FreeBSD.org References: <20230328041241.5D8A7431@slippy.cwsent.com> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="5rnksy3ueozgq2ym" Content-Disposition: inline In-Reply-To: <20230328041241.5D8A7431@slippy.cwsent.com> User-Agent: NeoMutt/20220429 X-ThisMailContainsUnwantedMimeParts: N --5rnksy3ueozgq2ym Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable * Cy Schubert [20230327 21:12]: > I wouldn't doubt others may implement the same policy using procmail or= =20 > even Outlook (which has powerful processing rules). You could even use dovecot with sieve-filters (that's what I do) ;) > So, yeah, to and cc while also cc the ML does have its place. It certainly does, although for lists I read regularly anyways, a CC to me directly annoys me a bit (one more mail to delete manually)... A pity the Mail-Followup-To: header isn't a standard and therefore only supported by a few MUAs. It solves this problem very nicely, announcing to responders where they should direct their followups to. AFAIK, at least thunderbird, (neo)mutt and gnus support it, probably a few others as well. --=20 Felix Palmen {private} felix@palmen-it.de -- ports committer (mentee) -- {web} http://palmen-it.de {pgp public key} http://palmen-it.de/pub.txt {pgp fingerprint} 6936 13D5 5BBF 4837 B212 3ACC 54AD E006 9879 F231 --5rnksy3ueozgq2ym Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iNUEABYKAH0WIQRpNhPVW79IN7ISOsxUreAGmHnyMQUCZCK9cl8UgAAAAAAuAChp c3N1ZXItZnByQG5vdGF0aW9ucy5vcGVucGdwLmZpZnRoaG9yc2VtYW4ubmV0Njkz NjEzRDU1QkJGNDgzN0IyMTIzQUNDNTRBREUwMDY5ODc5RjIzMQAKCRBUreAGmHny MXx4AP9GGGKdtl4lwD3A9ntvmrhMnOsmXCkzUVDngfs9ybn8FQD9Ez1CJtOA+nY2 xJs2QmYggNwgXn7DxVX/IHAurZ4TjA8= =Un9I -----END PGP SIGNATURE----- --5rnksy3ueozgq2ym-- From nobody Tue Mar 28 10:27:54 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pm5VY6x5mz41jgM for ; Tue, 28 Mar 2023 10:28:05 +0000 (UTC) (envelope-from junchoon@dec.sakura.ne.jp) Received: from www121.sakura.ne.jp (www121.sakura.ne.jp [153.125.133.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pm5VX0sR2z4FSR for ; Tue, 28 Mar 2023 10:28:03 +0000 (UTC) (envelope-from junchoon@dec.sakura.ne.jp) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of junchoon@dec.sakura.ne.jp has no SPF policy when checking 153.125.133.21) smtp.mailfrom=junchoon@dec.sakura.ne.jp; dmarc=none Received: from kalamity.joker.local (123-1-88-210.area1b.commufa.jp [123.1.88.210]) (authenticated bits=0) by www121.sakura.ne.jp (8.16.1/8.16.1/[SAKURA-WEB]/20201212) with ESMTPA id 32SARsux040684 for ; Tue, 28 Mar 2023 19:27:54 +0900 (JST) (envelope-from junchoon@dec.sakura.ne.jp) Date: Tue, 28 Mar 2023 19:27:54 +0900 From: Tomoaki AOKI To: freebsd-hackers@freebsd.org Subject: Re: what's the Correct git update method keeping local changes Message-Id: <20230328192754.a4991f95ea76df9caddd29d5@dec.sakura.ne.jp> In-Reply-To: <708822f5-2a2f-35f7-8f79-bf48ff644a22@FreeBSD.org> References: <708822f5-2a2f-35f7-8f79-bf48ff644a22@FreeBSD.org> Organization: Junchoon corps X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.33; amd64-portbld-freebsd13.2) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-1.48 / 15.00]; AUTH_NA(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.96)[-0.959]; NEURAL_HAM_LONG(-0.92)[-0.919]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:7684, ipnet:153.125.128.0/18, country:JP]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; R_SPF_NA(0.00)[no SPF record]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_TLS_LAST(0.00)[]; HAS_ORG_HEADER(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; DMARC_NA(0.00)[sakura.ne.jp]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4Pm5VX0sR2z4FSR X-Spamd-Bar: - X-ThisMailContainsUnwantedMimeParts: N On Tue, 28 Mar 2023 09:59:45 +0200 Matthias Andree wrote: > Am 28.03.23 um 05:19 schrieb void: > > Hello hackers@ > > > > I looked in the porters and developers handbooks and couldn't find > > reference > > to a Correct method of working with git, poudriere, the ports tree and > > local changes for a use case like mine. > > > > Right now my workflow looks like this: > > > > 1. apply patch either with patch -p0 < patchfile or git apply from the > >    top of the ports tree > > 2. git stash > > 3. poudriere ports -u -q > > 4. git stash pop > > 5. run the poudriere build > > > > then, subsequent poudriere builds need steps 2-5 repeated. > > > > I'm wary of git merge/apply because i'm not a dev and so don't want to push > > changes. but I want to update the ports tree for poudriere with local > > changes > > keeping them local. > > > > What's the best way? > > So - there's some discrepancy in your tools there, you are using git to > keep local patches, and poudriere to update. This seems odd. Is > poudriere using git to update your ports tree? Portsnap? Something > else? We don't know. > > I personally do not use poudriere ports -u, in fact I am maintaining my > /usr/ports in Git (but then I occasionally push from my tree into > FreeBSD's), and poudriere knows the ports tree with a "null" method, and > then 2...4 become a simple "git pull --rebase --autostash". > > > $ poudriere ports -l > > PORTSTREE METHOD TIMESTAMP PATH > > default null 2023-03-27 05:59:35 /usr/ports > > -- > Matthias Andree > FreeBSD ports committer I myself use git to update ports tree and never use `poudriere port -u`. Basically every step is done on /usr/ports. 1. Apply patches I want to try. 2. `git stash` 3. `git pull --ff-only` 4. `portsdb -Uu` 5. `git stash apply` Step 4 is for portupgrade (still use it if tooooooo many update is proposed by `poudriere bulk` but need some ports updated right away (basically small security updates), and pkg_replace doesn't work as intended. This step updates INDEX locally and update portupgrade database. If INDEX is somehow broken, do `portsdb-Fu` before step 5 to fetch last sane (but old) INDEX. Note that I've configured poudriere ports tree as below to use /usr/ports. `poudriere ports -c -f none -M /usr/ports -m null` This is because, I cannot recall which actually was, some ports hesitates to build unless ports tree is /usr/ports and I only use poudriere on bare metal environment. No builds for other archs nor branches. -- Tomoaki AOKI From nobody Tue Mar 28 10:34:33 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pm5f36wphz41kJt for ; Tue, 28 Mar 2023 10:34:35 +0000 (UTC) (envelope-from loo00013@umn.edu) Received: from mta-p8.oit.umn.edu (mta-p8.oit.umn.edu [134.84.196.208]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pm5f301Vsz4GlZ for ; Tue, 28 Mar 2023 10:34:35 +0000 (UTC) (envelope-from loo00013@umn.edu) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=umn.edu header.s=google header.b=Fc7hOSpp; spf=pass (mx1.freebsd.org: domain of loo00013@umn.edu designates 134.84.196.208 as permitted sender) smtp.mailfrom=loo00013@umn.edu; dmarc=pass (policy=reject) header.from=umn.edu Received: from localhost (unknown [127.0.0.1]) by mta-p8.oit.umn.edu (Postfix) with ESMTP id 4Pm5f20Klcz9vC8w for ; Tue, 28 Mar 2023 10:34:34 +0000 (UTC) X-Virus-Scanned: amavisd-new at umn.edu Received: from mta-p8.oit.umn.edu ([127.0.0.1]) by localhost (mta-p8.oit.umn.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LrI3-GHnmjYw for ; Tue, 28 Mar 2023 05:34:33 -0500 (CDT) Received: from mail-il1-f197.google.com (mail-il1-f197.google.com [209.85.166.197]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mta-p8.oit.umn.edu (Postfix) with ESMTPS id 4Pm5f15csLz9vC8r for ; Tue, 28 Mar 2023 05:34:33 -0500 (CDT) DMARC-Filter: OpenDMARC Filter v1.3.2 mta-p8.oit.umn.edu 4Pm5f15csLz9vC8r DKIM-Filter: OpenDKIM Filter v2.11.0 mta-p8.oit.umn.edu 4Pm5f15csLz9vC8r Received: by mail-il1-f197.google.com with SMTP id a17-20020a921a11000000b00325f9878441so4764798ila.7 for ; Tue, 28 Mar 2023 03:34:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=umn.edu; s=google; t=1679999672; h=content-transfer-encoding:in-reply-to:content-language:references :cc:to:subject:from:user-agent:mime-version:date:message-id:from:to :cc:subject:date:message-id:reply-to; bh=YSCIBvy1FjCnWgxFzMyZs1+Y9VQIoWci7el2OdmimUM=; b=Fc7hOSppdZ9gePqk15Z1s3ycBxmChoVF0fHBStoNrV7TDCoYkxp78BqieyTf5pZMkM 6vY1Ox0d3NibFyXCpdCFrq/4k84lftoVwh5+PZb72YdjipDxC1HNKchVJ6NVg1daMM78 lxCT48PaSRGGixjoLHCo1po0K6ND/M0xq/+smM4Li8wiPFoxLBuVyETNk244M98yZQfj EptFM0o6+s/n9taDF5D6vyLHhof2NbMqD3MG8P+JuqqjUeJ+eBcaxJ0JrNdibDi8g4ON zzRqOMhrxth/w08VSToDJ/L+4NNdlLggQiH11hsxJFEMzREfzEqJpY3+tXzibn3Ff52Z dmBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679999672; h=content-transfer-encoding:in-reply-to:content-language:references :cc:to:subject:from:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=YSCIBvy1FjCnWgxFzMyZs1+Y9VQIoWci7el2OdmimUM=; b=Ml+0jpcXOGOx2A4CSCXrmuTlAol9Al04eDSii4EqRpCjsRnA9U5McB1i61S1iTJxym BFt0xK92p3Vy36HjTG86vxTuZRnoS8JL5miLJ+V+GxGWkUSr0sFtXJ3tIr3P7E8wiOeX Eb1Qb9s9yVlpB0jEQOTQ9K+YeLclCzvgqzGsSRnRYlWLW6L+9sPPkbJu/hdkq1wm3dAB MdjXUBWHVAT3X2VmorWGjW9pQe6zwRSEAw2xpgEOWcKKWvjYx9KQu13xFluavacuTibS XGzdDoSu/wzBXscqSqdg2BfLlrwqXU2Ab6/+m7I70EdDGLFGrPm+luf6szkCvhroQvxJ iPag== X-Gm-Message-State: AAQBX9cC4Zf7Cz6jzUk2Y/7q+OU+O+6Lt+/XBaBvLTZKE7HCKV2wUnpN WCJlfmuGYGuIeiz3S3thtiFuSsZOIbTlW7bmnbM1JaYiZqaV8GIu8YtuQmFiNgQbvedptGNwgWD l61tHS0U6jSOw7IIM1N788nNe3BepYh/mSKj8tg== X-Received: by 2002:a6b:da0e:0:b0:75c:8ca2:c9dd with SMTP id x14-20020a6bda0e000000b0075c8ca2c9ddmr1816413iob.13.1679999672640; Tue, 28 Mar 2023 03:34:32 -0700 (PDT) X-Google-Smtp-Source: AKy350YsI/0n7Ws079Y2Ly18QbC3RK3zt7nWDbDgXLz98Uo48ZBjd1v1lSlb9ktCbol6MiybmZkFog== X-Received: by 2002:a6b:da0e:0:b0:75c:8ca2:c9dd with SMTP id x14-20020a6bda0e000000b0075c8ca2c9ddmr1816400iob.13.1679999672314; Tue, 28 Mar 2023 03:34:32 -0700 (PDT) Received: from [192.168.0.67] (97-116-81-233.mpls.qwest.net. [97.116.81.233]) by smtp.gmail.com with ESMTPSA id cl4-20020a0566383d0400b003ab21c8fa84sm9520800jab.121.2023.03.28.03.34.31 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 28 Mar 2023 03:34:31 -0700 (PDT) Message-ID: Date: Tue, 28 Mar 2023 05:34:33 -0500 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 From: Shaun Loo Subject: Re: Periodic rant about SCHED_ULE To: George Mitchell Cc: freebsd-hackers@freebsd.org References: Content-Language: en-US In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-5.70 / 15.00]; DWL_DNSWL_LOW(-1.00)[umn.edu:dkim]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.996]; DMARC_POLICY_ALLOW(-0.50)[umn.edu,reject]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; R_SPF_ALLOW(-0.20)[+ip4:134.84.196.192/27]; R_DKIM_ALLOW(-0.20)[umn.edu:s=google]; RCVD_IN_DNSWL_MED(-0.20)[134.84.196.208:from]; MIME_GOOD(-0.10)[text/plain]; TAGGED_RCPT(0.00)[freebsd]; TO_MATCH_ENVRCPT_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; FROM_HAS_DN(0.00)[]; RCVD_COUNT_FIVE(0.00)[6]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[209.85.166.197:received]; DKIM_TRACE(0.00)[umn.edu:+]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:217, ipnet:134.84.0.0/16, country:US]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4Pm5f301Vsz4GlZ X-Spamd-Bar: ----- X-ThisMailContainsUnwantedMimeParts: N On Sun, Mar 26, 2023 at 1:24 PM George Mitchell > wrote: On 3/25/23 23:42, Peter wrote: > Quoting George Mitchell >: > > On 3/25/23 11:47, Peter wrote: >>> You're welcome. Can I get a success/failure report? >>> >>> >>> [...] >> Thanks for your help.  Regretfully, I have to report that the patch >> did not really help in my case (make buildworld while running dnetc). >> But I have belatedly realized that it's a really good idea to use >> make -j12 to build kernels and the world.  With 4BSD and dnetc >> running, I can now buildworld in 3512 seconds (down from 20477), and >> with ULE in 14193 seconds (down from 50290).  Peter's patch got it >> down to 13755 seconds. > > Thank You for testing. So at least it doesn't make things worse. > Back then 5 years ago when I made that, I think I've seen a few > more strange things in that code, but they didn't bother me immanently > at that point. Probably there is more work to do there... To oversimplify the discussion I've seen so far on this thread: It would be really nice if the schedulers were kernel loadable modules. GSoC project? I'm definitely eager to put forth a GSoC project to make schedulers into kernel modules! I'll be reviewing how to write kernel modules such that I'll be in a good position when GSoC comes. I've been checking this thread on-and off for the past few days so I definitely think I'm in a good position to work on exactly this! Is anyone in this thread willing and able to be my mentor? I'll try to come up with a compelling proposal in the next couple days or so to get an application in right before the April 4th deadline. --Shaun Whether SCHED_ULE or SCHED_4BSD should be the default scheduler in the GENERIC kernel is a contentious discussion, but perhaps we need to have that discussion.                                           -- George From nobody Tue Mar 28 10:37:16 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pm5jd2TMqz41kw1 for ; Tue, 28 Mar 2023 10:37:41 +0000 (UTC) (envelope-from pat@patmaddox.com) Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pm5jc39zfz4Hbt for ; Tue, 28 Mar 2023 10:37:40 +0000 (UTC) (envelope-from pat@patmaddox.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=patmaddox.com header.s=fm1 header.b=c+9M0oBr; dkim=pass header.d=messagingengine.com header.s=fm2 header.b="X voXTDg"; spf=pass (mx1.freebsd.org: domain of pat@patmaddox.com designates 64.147.123.19 as permitted sender) smtp.mailfrom=pat@patmaddox.com; dmarc=none Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.west.internal (Postfix) with ESMTP id 02ED132008FD; Tue, 28 Mar 2023 06:37:36 -0400 (EDT) Received: from imap41 ([10.202.2.91]) by compute2.internal (MEProxy); Tue, 28 Mar 2023 06:37:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=patmaddox.com; h=cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm1; t= 1679999856; x=1680086256; bh=SA/qvP0nmgPuBU6ozigYXXds3XM2oiojxYz lb/Xa2c8=; b=c+9M0oBryZbpU8+VspppuA4bJ+YxXWMQrR5VcBdG94VEO6qdYJM UpYzwpL2Epc21cQMeUJWEzW9R/Osf8H5Sj3G5yKksWswAynUJvWHoQz9r0EAHFP6 k0ZG2SqwsogmFyo7qSkSqhvOf3ljrm3i1sl4lYgW+tk8/3fCKPPNxIgDopmJGE4g 0wjQZ2wSZD9pvdYbWgK4lgcRLcG1m1OPQnQh4l+EqPJBYtcxMCbRtvlh7RB8Is7q Sokq8bP4BQaELe+O4H5ov/Cqt756DLUEQ7ZeK71iqY12yBSCl3A/Q0adxfLJUosZ fgsAU1NfQPE8i4z644Lqa62t66SzC1WomEw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t=1679999856; x= 1680086256; bh=SA/qvP0nmgPuBU6ozigYXXds3XM2oiojxYzlb/Xa2c8=; b=X voXTDgrK2Y5YJTd3jMSB9Kfbk//LJdoj88YCKzPpshPzoX2520hMSZW87CT8kr+o pfGU+BVPkf03s0YVSRKAVLUstA4ZvBjAxYWyE6X1YUOSiRFCSWxcjxI/V/oJPj5X SJPKjRfjlRL0uwX0S+iU7YujkXzORkB42TNGNknu1C3aT7Phe1mTntw5ujLtFmc6 p7jEfOQS+O71y84UVYn/b/W6fFWE9JdLqZ0Te62CtCngHipL6ssXelcXtHIzHBKR yRWR5tdAfEnr94u16nenOEfBFG3q5IAUhHFgxEhEW96qD2UDkpyqP4Dz6tEp1HSa TfN7oCVltQx9zvqQWI1pg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdehgedgfedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfgjfhffhffvufgtgfesth hqredtreerjeenucfhrhhomhepfdfrrghtucforgguughogidfuceophgrthesphgrthhm rgguughogidrtghomheqnecuggftrfgrthhtvghrnhepvedtieetlefgvdeklefhffelke evtefgkeegleejleegveduhfekgefggfdvudeknecuvehluhhsthgvrhfuihiivgeptden ucfrrghrrghmpehmrghilhhfrhhomhepphgrthesphgrthhmrgguughogidrtghomh X-ME-Proxy: Feedback-ID: i8b6c40f9:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 59E092340096; Tue, 28 Mar 2023 06:37:36 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.9.0-alpha0-236-g06c0f70e43-fm-20230313.001-g06c0f70e List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 Message-Id: In-Reply-To: References: Date: Tue, 28 Mar 2023 03:37:16 -0700 From: "Pat Maddox" To: freebsd-hackers@freebsd.org Subject: Re: what's the Correct git update method keeping local changes Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-4.09 / 15.00]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.996]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.19]; R_DKIM_ALLOW(-0.20)[patmaddox.com:s=fm1,messagingengine.com:s=fm2]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.19:from]; XM_UA_NO_VERSION(0.01)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[patmaddox.com:+,messagingengine.com:+]; RCVD_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[patmaddox.com]; FREEFALL_USER(0.00)[pat]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 4Pm5jc39zfz4Hbt X-Spamd-Bar: ---- X-ThisMailContainsUnwantedMimeParts: N On Tue, Mar 28, 2023, at 1:39 AM, Warner Losh wrote: > > I use the rebase workflow for pending ports commit work. Most of the replies say something similar. What are people=E2=80=99s exp= eriences with poudriere overlay dire? Pat From nobody Tue Mar 28 10:52:32 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pm62q3zRYz41lck for ; Tue, 28 Mar 2023 10:52:35 +0000 (UTC) (envelope-from zirias@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pm62q3Xdtz4KGW for ; Tue, 28 Mar 2023 10:52:35 +0000 (UTC) (envelope-from zirias@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1680000755; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Vwhv8UeXBnNxTtSnv7C9fPLKL64bw/NU52YPwF93gEs=; b=BxpMUz4xTyhQANH0gW3j8qwEHB6TOGvKDXX2S+EITHBSwxqkcApq714FOM0jgOSPmwh0qa 85kPGsahBstCMgr6T1+A0f4aDYqB91/bpOMSdO3XpdgScmtAUlpaaIwX0z37wVxbnMYYhA TUvk5ZzaIPhZBZZZMHhqBFQDLsjEXzsgb5QVYqPK9KG6Pb6udrd/zHu9JxSCfbBEUPiQpZ T6E4qvwQn5xt1xu2/3x4C61f6tZTUZvoY/t7CwoXqGX4O6zN9YZgkvVqA05WEw6rYtJ5+J PmeOJ2/bqHsnaF3JjtrDW61hPxYH15MONwlpPKRrAsFIKrhnBlKlHGhrEtecSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1680000755; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Vwhv8UeXBnNxTtSnv7C9fPLKL64bw/NU52YPwF93gEs=; b=Y76xV5CQSrlHBzojGJM3QITAH7PmDycn1eHsXfhIA+mZmefNGNdFcuXErEokv0evYzX4NS A1dka4zc8ZBVXPpZEGp1/vL+kpt2UjA2BA/Lg/35awKTRcQteSGOaM7nnE5/qQyi9qC9Pq jz+6sn+hg1Kf6MALsUtyHWomUq2by3MRcND3eoNsBPr+j5n8EMUkUME6zNrD+XkMvpMBCs zv5kHcSgl+oc38LchWrrrEG8uC6yu8iRpHW+0x66nz2Khfp3n3LD4LHkn/AnWtX2KVU0dL 8bifVivrJiX2j08RWXYDbYRXlQx9VUzy178eJIYO2R1BukPo1yPLZWCKP+IqDw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1680000755; a=rsa-sha256; cv=none; b=sokiHMktE8ZB5GvlO54jsfDnY8KEH2DmKz0URDlY8XJzUnIWPcoTmaurpdj8mAJvRdVvey h84R4saDfpPvsx3hC9C5rfXL7CdZ4v0d9tpQK6FYMPCIqXDmsgrx6Tal1a2CE0VyX96nFd rTosJhnDEiQkglwRD+v/V/QHfpjiiVCXvXY+E1afUK2Y45IMkbIJJiSWp7/SDna2eQ2luE crHo7ivAjkJSEl8MkjHmYbE85YWAUF8U/mdg/oZ6u/AfER3vT0shzuhij60rqzC0D1etxK BPOmcwrjCjkWqdMAErPTgDXY4ibMEwS0ZeCG1YM3XgLrEF0K4VC7jmu0YieVJA== Received: from stef.palmen-it.de (stef.palmen-it.de [IPv6:2001:470:1f0b:bbb:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: zirias/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Pm62q2cM5zbct for ; Tue, 28 Mar 2023 10:52:35 +0000 (UTC) (envelope-from zirias@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=palmen-it.de; s=20200414; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:To:From:Date:Sender:Reply-To:Cc: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=Vwhv8UeXBnNxTtSnv7C9fPLKL64bw/NU52YPwF93gEs=; b=KAvhIVvDia9hMgwEJXUeBN2GoJ YAU6K8vLpa5DgGkLS9MoqcO9aC/pXEqVdvRDA/v01iVkUQRMdrXPbIzFruLymqXCG5y2Qw8dSQSze 4hbVG27RAHp5HiObOR7Qal3cnBDjYKUMaoVlrkQ1WSt7L4lEbiKRiLb9fqQXX5SiasMGhJltUB3Vy T6+miHOoB+YWXlq1Japah5u9waTG6PZ+lDow+tb+n9msR/Ed4iv/rMVTrOZaVsF3WYkIO7rIsvXsS w/wD752WCaE9AcmM8o0vhKqX7ItbU/U4VlraiG5uyvadDjxiZhCDna9b/Cgq13AzRIfZnaMEek/Z7 7O4dqMjg==; Received: from [192.168.71.101] (helo=mail.home.palmen-it.de) by stef.palmen-it.de with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1ph6wL-002oUv-95 for freebsd-hackers@freebsd.org; Tue, 28 Mar 2023 12:52:33 +0200 Received: from nexus.home.palmen-it.de ([192.168.99.2]) by mail.home.palmen-it.de with esmtpsa (TLS1.3) tls TLS_CHACHA20_POLY1305_SHA256 (Exim 4.95 (FreeBSD)) (envelope-from ) id 1ph6wK-000I7N-S0 for freebsd-hackers@freebsd.org; Tue, 28 Mar 2023 10:52:32 +0000 Date: Tue, 28 Mar 2023 12:52:32 +0200 From: Felix Palmen To: freebsd-hackers@freebsd.org Subject: Re: what's the Correct git update method keeping local changes Message-ID: <20230328105232.dfzw4i3gqfgs47s5@nexus.home.palmen-it.de> Mail-Followup-To: freebsd-hackers@freebsd.org X-Face: /1K@t"h.}e~pR@]c7HorQ!T`F^RJCa'BCr#e>IKA{>C/9OTGB4|xh"y2{?1Z5M i2w"AH^pN_LlHR^{+f',_Np~;.B;!M/bL}*qk]p5*r7F5vW};{:@4u5S?T&f0$7BJ-71Q5SV]:v$`5 A0[DZ:=?S52x8HJ~5@^P_\T@MsjG{R( Organization: FreeBSD.org References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="jyc5evhifvn5l6dk" Content-Disposition: inline In-Reply-To: User-Agent: NeoMutt/20220429 X-ThisMailContainsUnwantedMimeParts: N --jyc5evhifvn5l6dk Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable * Pat Maddox [20230328 03:37]: > On Tue, Mar 28, 2023, at 1:39 AM, Warner Losh wrote: > > > > I use the rebase workflow for pending ports commit work. >=20 > Most of the replies say something similar. What are people=E2=80=99s expe= riences with poudriere overlay dire? Overlays are a completely different approach for "local changes" and independent from the SCM used (currently git). My 2=C2=A2: I'd say using local git branches is much more flexible and fits my personal workflow best, as I can just cherry-pick to actually commit something or put it up for review on phab. If I would use overlays locally, I'd have to manually copy over code. IMHO, a good usecase for overlays is if you need some shared staging area for a team, very much like the libreoffice team uses it. --=20 Felix Palmen {private} felix@palmen-it.de -- ports committer (mentee) -- {web} http://palmen-it.de {pgp public key} http://palmen-it.de/pub.txt {pgp fingerprint} 6936 13D5 5BBF 4837 B212 3ACC 54AD E006 9879 F231 --jyc5evhifvn5l6dk Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iNUEABYKAH0WIQRpNhPVW79IN7ISOsxUreAGmHnyMQUCZCLG8F8UgAAAAAAuAChp c3N1ZXItZnByQG5vdGF0aW9ucy5vcGVucGdwLmZpZnRoaG9yc2VtYW4ubmV0Njkz NjEzRDU1QkJGNDgzN0IyMTIzQUNDNTRBREUwMDY5ODc5RjIzMQAKCRBUreAGmHny MVZcAP9I8cN2jVuDzbSHaGm0yWi/TIL7wyqa67RDD78NP92l3QD/ahoV1gpLa563 S5PA4SyrWumTGtApNHPNRAsi/3LGqgs= =4Ygt -----END PGP SIGNATURE----- --jyc5evhifvn5l6dk-- From nobody Tue Mar 28 13:51:40 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PmB1w3fPCz41xx0 for ; Tue, 28 Mar 2023 13:52:04 +0000 (UTC) (envelope-from dch@skunkwerks.at) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PmB1v3Bjjz4XLh for ; Tue, 28 Mar 2023 13:52:03 +0000 (UTC) (envelope-from dch@skunkwerks.at) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=skunkwerks.at header.s=fm3 header.b=iSYUb4Md; dkim=pass header.d=messagingengine.com header.s=fm2 header.b="Z 5IeRyO"; spf=pass (mx1.freebsd.org: domain of dch@skunkwerks.at designates 66.111.4.28 as permitted sender) smtp.mailfrom=dch@skunkwerks.at; dmarc=pass (policy=reject) header.from=skunkwerks.at Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 397725C016D for ; Tue, 28 Mar 2023 09:52:01 -0400 (EDT) Received: from imap44 ([10.202.2.94]) by compute1.internal (MEProxy); Tue, 28 Mar 2023 09:52:01 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=skunkwerks.at; h=cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm3; t= 1680011521; x=1680097921; bh=1iH6HM3vRgf+VfewNo2+3yhXEA+5vveRbuU tAVNGa/o=; b=iSYUb4MdweZtfmaMv7Sxd6wpiYb7cyMLPVbFhCgvhUL44HwgoN0 UB8GC930xGcJJv2F0hQaCl65FRrVsqvxkXf5zX8k+8NjBICkdmQ0k+aBQsgrzGd1 1QXOzVq99g7KvXMvXfE2kuIC8BuQnpLjjHmjvlV8/mb30PvldYfGwJGREWDllXZo UjLQiK/qhntXuaBKdTu5E4VFiNJUjyMaHEtOgJhOdBQmlxvcCqNfjEtGfVjqge5g aMz9AAR3FveyDkaA+2F0yi+Y2o/mhPGGJb6ElOdspMQvANiRudUMUr2Ki6KY2GZB rUj8lPzMXK++Ccl61hS2Ngrh8l/fGnJyAiw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t=1680011521; x= 1680097921; bh=1iH6HM3vRgf+VfewNo2+3yhXEA+5vveRbuUtAVNGa/o=; b=Z 5IeRyO4wB02jnt07SxpTnEqgl+ZbX8Xiac1nLgj/QXb86qHrSeKE8QJT4nyZuMt4 79u/GOXcoqhbGOnCj/CuMBRzqLQ4DgBrBc2jQecP9UCKC45UD8rZ+rXYPVQMRSu7 E2PMTNWxJGL8sYPSgIIYsOty+UXFzDvTF/+NbhUxuSphOHQ/6ez9wVMR5fyoLBBi EMPBGbGDwhdYjuJkId87KXQpUqnXYTZ8uZPKdhhWWzHAiFqNP2ihelRvcZ9blpWu F/5BXIJC2nbzP5ZKfomfNsL4HlYzr6OE8tfAvuKu2/DL+vcjOFi0xzUCUMyPajGX DLebvwswJddWGGClSusRw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdehgedgieekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepofgfggfkjghffffhvffutgfgsehtqhertderreejnecuhfhrohhmpedfffgr vhgvucevohhtthhlvghhuhgsvghrfdcuoegutghhsehskhhunhhkfigvrhhkshdrrghtqe enucggtffrrghtthgvrhhnpedthfethfejkeeihedvhfeiieeiledtheehueetieelhfff leehhffhfeefteejteenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrih hlfhhrohhmpegutghhsehskhhunhhkfigvrhhkshdrrght X-ME-Proxy: Feedback-ID: ic0e84090:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id B663136A007A; Tue, 28 Mar 2023 09:52:00 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.9.0-alpha0-237-g62623e8e3f-fm-20230327.001-g62623e8e List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 Message-Id: <4dd6c6b4-20d6-438e-9a23-5e0613c1f9ea@app.fastmail.com> In-Reply-To: References: Date: Tue, 28 Mar 2023 13:51:40 +0000 From: "Dave Cottlehuber" To: "FreeBSD Hackers" Subject: Re: what's the Correct git update method keeping local changes Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-4.69 / 15.00]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[skunkwerks.at,reject]; R_DKIM_ALLOW(-0.20)[skunkwerks.at:s=fm3,messagingengine.com:s=fm2]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.28]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.28:from]; MIME_GOOD(-0.10)[text/plain]; RWL_MAILSPIKE_GOOD(-0.10)[66.111.4.28:from]; XM_UA_NO_VERSION(0.01)[]; FREEFALL_USER(0.00)[dch]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; ASN(0.00)[asn:19151, ipnet:66.111.4.0/24, country:US]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[skunkwerks.at:+,messagingengine.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4PmB1v3Bjjz4XLh X-Spamd-Bar: ---- X-ThisMailContainsUnwantedMimeParts: N On Tue, 28 Mar 2023, at 10:37, Pat Maddox wrote: > On Tue, Mar 28, 2023, at 1:39 AM, Warner Losh wrote: >> >> I use the rebase workflow for pending ports commit work. > > Most of the replies say something similar. What are people=E2=80=99s=20 > experiences with poudriere overlay dire? > > Pat I think this is largely a matter of familiarity with git. If you have a pile of local ports, in your own category, then an overlay is just fine, but then you have 2 separate repos to manage. As soon as you decide you want to pin an upstream port, or make some customisation that hasn't been accepted upstream yet, then having a single ports tree and rebasing periodically is really nice. You can revert an upstream commit you don't want, or keep an additional one, and it all just floats up the top.=20 Resolving conflicts is mostly a matter of adjusting various PORTREVISION changes, occasionally you need more extensive fixes ofc if the ports or patches it includes shifts too far. I like knowing that my entire environment comes down to 2 commits, one in src repo, another in ports repo, and that's very easy to reproduce. A+ Dave From nobody Tue Mar 28 14:25:39 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PmBmn4B2Cz420sR for ; Tue, 28 Mar 2023 14:25:45 +0000 (UTC) (envelope-from void@f-m.fm) Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PmBmm49q8z4d2W for ; Tue, 28 Mar 2023 14:25:44 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm2 header.b=t3MBHtJ4; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=n2iirFAX; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 64.147.123.24 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.west.internal (Postfix) with ESMTP id 86CF33200900 for ; Tue, 28 Mar 2023 10:25:42 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Tue, 28 Mar 2023 10:25:42 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm2; t=1680013542; x=1680099942; bh=Qg bq7G6HieBL4bO48uDmLob3792QoeirYQ9bXeEQz8Y=; b=t3MBHtJ4ihadX7kO+h ULLQAytweyKhzDnO7XBi3Gxv20Z+SfO9oqOs4BRTKaA4aiD/zph7Ifj4spUAwm20 7l6qso4ZhDYoOjvC3ZlFttXS2UZcESJCG3h+NGhO/IHSerjP3RDp8QvQlu5vgx8K 9x50r3UdCIwLEdNlwzkjRsBrCZGx9h4hfZz0ocekjQP46dPULJ+CV2S3ZdSDAyVW KZUO+sxO9+5VDgV2hVlouEbu4KzoyOjJRtyuDjM+tyiiJojc9Js2JBIUaEKJUG3N Ovqp3RIPJjIoBycRi2j/SSqhYfcYqwR5kdI5yjLuCpseCEsa3EcF9EJHy5Ew7n+9 WVTw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; t=1680013542; x=1680099942; bh=Qgbq7G6HieBL4 bO48uDmLob3792QoeirYQ9bXeEQz8Y=; b=n2iirFAXmSpKzpAoD1Yyq0RTxKdEn YbMs2mEZcVrhvP/08hES7K6r6P7Tya0lZpqumsFhqG+DSKR4OOVZlDS8blO5l3yx M1DA8D2annY8rLSGNVjwFlcmqAPI6Rkvklla6CP5Xg/J4MrwRFo72m6Bg7VwxjJ9 vBtHtKs1D33iORcl6uLWF5o0/fRvLlm8G7MllvNBtjAWsqhFJtpJuN1b65khjEmW 5YcGe0Esbv/YzZ8c3YjnvL9mzfbq4s7EAG9ueWthWjSWFUHIwElShk4o4WvjIdKo xMrBMyzPLxPNdfCyIKMV//tsPixwLbATPs8C5YBwfKNKMWu6FCnttNPhQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdehgedgjeehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggujgesthdtre dttddtvdenucfhrhhomhepvhhoihguuceovhhoihgusehfqdhmrdhfmheqnecuggftrfgr thhtvghrnhepkeeluddvlefhieelfefggffhffektdehleelgfdugfdvgeekjeejuddthe ehgfeunecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhep vhhoihgusehfqdhmrdhfmh X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Tue, 28 Mar 2023 10:25:41 -0400 (EDT) Date: Tue, 28 Mar 2023 15:25:39 +0100 From: void To: freebsd-hackers@freebsd.org Subject: Re: what's the Correct git update method keeping local changes Message-ID: Mail-Followup-To: freebsd-hackers@freebsd.org References: <708822f5-2a2f-35f7-8f79-bf48ff644a22@FreeBSD.org> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <708822f5-2a2f-35f7-8f79-bf48ff644a22@FreeBSD.org> X-Spamd-Result: default: False [-4.35 / 15.00]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.75)[-0.755]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm2,messagingengine.com:s=fm2]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.24]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.24:from]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; RCPT_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; FREEMAIL_FROM(0.00)[f-m.fm]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US]; FROM_HAS_DN(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[64.147.123.24:from]; FROM_EQ_ENVFROM(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4PmBmm49q8z4d2W X-Spamd-Bar: ---- X-ThisMailContainsUnwantedMimeParts: N On Tue, Mar 28, 2023 at 09:59:45AM +0200, Matthias Andree wrote: >Am 28.03.23 um 05:19 schrieb void: > >So - there's some discrepancy in your tools there, you are using git to >keep local patches, and poudriere to update. This seems odd. Is >poudriere using git to update your ports tree? Portsnap? Something >else? We don't know. This is where I was going wrong. poudriere -u was using git+https for updates and was complaining after patches were applied. I've not looked/found what git subcommand 'poudriere ports -u' maps to, yet. Instead of using using 'poudriere ports -u' I'm now using 'git -C $portspath pull' and it's working as expected with the 'default' portstree of poudriere. Thanks everyone -- From nobody Tue Mar 28 14:45:56 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PmCDC2K21z42266 for ; Tue, 28 Mar 2023 14:46:03 +0000 (UTC) (envelope-from void@f-m.fm) Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PmCDB1PMVz3HJL for ; Tue, 28 Mar 2023 14:46:02 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm2 header.b=iPIiJlXO; dkim=pass header.d=messagingengine.com header.s=fm2 header.b="Z Avg6jV"; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 64.147.123.21 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id D56EB320091B for ; Tue, 28 Mar 2023 10:46:00 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Tue, 28 Mar 2023 10:46:00 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-transfer-encoding:content-type:content-type:date:date :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm2; t= 1680014760; x=1680101160; bh=A/A5pXDLsvmMSph+7mQzmZ0mWxMa/0Pp4PM qCIIqwAA=; b=iPIiJlXOkHeiBRnKUKKtDV9/kbo0fAV4hDmp/9736bLBTkNKYVR xP0JyLbGW0DfGhnQk4zjSsijfPp/dMy1q9WfCEDrJetiq3oJ9I4p5YSqVfb2h1dK PIW9FHHdsTRNwaONQEyLEaz7o2WHZjUqaWAbkpeOBzUH2I337o8XGKpuyLJW59yY sFYtVtuYIqI7yyAaZJVPbxow4B+Lu1m/erE5FdqxaTwtx3INj0OwugQNOUd7UeoZ AgVaHB6vYMI+r+uJJBgepZ0T4UMvWGi8+wfdvLkgIxr/LO8ux56I7M9eYvRimSa4 RvEH5Uc+UTbmpc9suHmzKzBk3bQC663hsIA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t=1680014760; x= 1680101160; bh=A/A5pXDLsvmMSph+7mQzmZ0mWxMa/0Pp4PMqCIIqwAA=; b=Z Avg6jVg9yC9w9n4rhRMlTl29cwpaQuGSY76muW4aw1qEW/N7FFpbksxsLvmrDY7v TG2zl8GLuHUBLbyqcsgAPlXPl+v5BkTaFeXfuGPVkakCwcttAXZWRtQSjvX9a5YS tMy8SjdFybY43U3IyV1tM/SbnRZ2//pC+XG2ubyW9eY3KYv8WQsdSmvYzWI4W8PR 8b6i+4eC+Xwxdywe/Sm39RuV1sgwpQ8pTvUUEv7+S5K4/O1rhhKMiF3aV8f/L5Hx AgZYMckOEFFJfaNRN6nkPJrD7fwR71+Y/qP25zfltzqJ4tNVwUWfT95HDI2bfMvt SszYtUsGBjGsNyGgCLsPg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdehgedgjeelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggugfgjsehtke ertddttdejnecuhfhrohhmpehvohhiugcuoehvohhiugesfhdqmhdrfhhmqeenucggtffr rghtthgvrhhnpeduteevgeeggeevieetvdduudetfffffffhteejgffggeevjeffiedute eftedvkeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhm pehvohhiugesfhdqmhdrfhhm X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Tue, 28 Mar 2023 10:45:59 -0400 (EDT) Date: Tue, 28 Mar 2023 15:45:56 +0100 From: void To: freebsd-hackers@freebsd.org Subject: Re: what's the Correct git update method keeping local changes Message-ID: Mail-Followup-To: freebsd-hackers@freebsd.org References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Spamd-Result: default: False [-4.53 / 15.00]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.93)[-0.930]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; MID_RHS_NOT_FQDN(0.50)[]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.21:c]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm2,messagingengine.com:s=fm2]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.21:from]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[f-m.fm]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4PmCDB1PMVz3HJL X-Spamd-Bar: ---- X-ThisMailContainsUnwantedMimeParts: N On Tue, Mar 28, 2023 at 03:37:16AM -0700, Pat Maddox wrote: >On Tue, Mar 28, 2023, at 1:39 AM, Warner Losh wrote: >> >> I use the rebase workflow for pending ports commit work. > >Most of the replies say something similar. What are people’s >experiences with poudriere overlay dire? I use two overlays: [1] for sccache-overlay to speed up those ports requiring rust to build [2] to maintain the last-known-good php74 for systems that can't be updated to a supported php version for $REASONS For systems that require [2], they need ports building in a tree that doesn't have the MOVED file, so it uses a different 'default' tree which has 'git rm MOVED' run on it after it updates but before poudriere bulk runs. -- From nobody Wed Mar 29 01:56:41 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PmV652DzTz41ljM for ; Wed, 29 Mar 2023 01:56:45 +0000 (UTC) (envelope-from jrm@ftfl.ca) Received: from mail-qt1-f175.google.com (mail-qt1-f175.google.com [209.85.160.175]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PmV641y4Fz3ssY for ; Wed, 29 Mar 2023 01:56:43 +0000 (UTC) (envelope-from jrm@ftfl.ca) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of jrm@ftfl.ca designates 209.85.160.175 as permitted sender) smtp.mailfrom=jrm@ftfl.ca; dmarc=none Received: by mail-qt1-f175.google.com with SMTP id cr18so10012018qtb.0 for ; Tue, 28 Mar 2023 18:56:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680055003; h=mime-version:user-agent:message-id:date:references:in-reply-to :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=jglEWxN0KyJ85vFURQh3Y8kP9whCjtcZk4VyhZadLdg=; b=RFPLt22Ks3Qp6cExDBQ6afvzsxsqhnfb6z2WKIbzM4uEHylENP7K3RTaPAfLn3rGCc VZyY/5sYE7ZC0X8Kf6NWSJC54hWb8UpPtkc8VXpOkmJxxBo2E/MjWIhdVj1AX04UoCZW u8k0uKy/LaqoEKOMoqFhaWnA7f4lX/VmeI0k+ANgPJ2auK6q0oZY253uGofh2U2EXWhJ DQ7nKVxlUA40JNIJIONA76R8C8/lQ60UfBcOnxpDLlIrfphCtBokkpm2UphVtGut6eaR vpsFLN8k0dqdUeY1u7Rf2gbtFWFp6LFOCdjYZa8KuqV7BfjwlF4wgEI/zk4D9H7tJYm2 S9SQ== X-Gm-Message-State: AO0yUKU3Bk348cSNr8m8wrhsjNq0uOGR4R0/zTFBVTh8XSXasuaa9g5y t9blOsiMRaD1yZvQmgw9LvZgoA== X-Google-Smtp-Source: AK7set/MwH6dlHpi7t9vKvnnhQS0QD7WkTFbuyu9suvX/NaQFecHmQauYTY+rhwDiL3ZnJ0XxN+Rzg== X-Received: by 2002:ac8:7d0f:0:b0:3d3:cd:623b with SMTP id g15-20020ac87d0f000000b003d300cd623bmr27687208qtb.63.1680055003003; Tue, 28 Mar 2023 18:56:43 -0700 (PDT) Received: from phe.ftfl.ca.ftfl.ca (drmons0544w-156-34-187-123.dhcp-dynamic.fibreop.ns.bellaliant.net. [156.34.187.123]) by smtp.gmail.com with ESMTPSA id w23-20020ac87197000000b003e4dab0776esm3234854qto.40.2023.03.28.18.56.42 (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 28 Mar 2023 18:56:42 -0700 (PDT) From: Joseph Mingrone To: Lorenzo Castellini Cc: freebsd-hackers@freebsd.org, mmokhi@FreeBSD.org Subject: Re: Google Summer of Code In-Reply-To: (Lorenzo Castellini's message of "Mon, 27 Mar 2023 18:07:34 -0400") References: Date: Tue, 28 Mar 2023 22:56:41 -0300 Message-ID: <865yakwc9y.fsf@phe.ftfl.ca> User-Agent: Gnus/5.13 (Gnus v5.13) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain X-Spamd-Result: default: False [-2.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.988]; FORGED_SENDER(0.30)[jrm@FreeBSD.org,jrm@ftfl.ca]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; MIME_GOOD(-0.10)[text/plain]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TAGGED_RCPT(0.00)[]; DMARC_NA(0.00)[freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[209.85.160.175:from]; FREEFALL_USER(0.00)[jrm]; ARC_NA(0.00)[]; FROM_NEQ_ENVFROM(0.00)[jrm@FreeBSD.org,jrm@ftfl.ca]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.160.175:from] X-Rspamd-Queue-Id: 4PmV641y4Fz3ssY X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N On Mon, 2023-03-27 at 18:07, Lorenzo Castellini wrote: > Greetings, > I hope this email finds you well. I am Lorenzo Castellini, a senior high > school student, with an interest in participating in the Google Summer of > Code. I am writing this email, since I am interested in collaborating with > you guys on the project, but I am in need of a bit of guidance with the > project proposal. I checked out the project list, and I was particularly > interested in a project of virtual memory compression. I would like to work > with this, maybe extending the project even further or applying some > automation to such a process. On another note, I would also like to work > with something similar, but with other PC components. Let me know of some > ideas according to my interests. Thanks! > Sincerely, > Lorenzo Castellini Coutin Hi Lorenzo, I don't see any proposed projects to work on virtual memory compression on https://wiki.freebsd.org/SummerOfCodeIdeas. That's not to say that you can't still propose something, but we should determine if this is a viable project. It looks like there were some GSoC projects in this area in 2015 and 2019. https://wiki.freebsd.org/SummerOfCode2015/MemoryCompressionDeduplication https://wiki.freebsd.org/SummerOfCode2019Projects/VirtualMemoryCompression I've copied Mahdi Mokhtari, who mentored the project in 2019. Mahdi, could you share any information about how that project turned out? The repository listed on the wiki returns HTTP 404. Lorenzo, there is some general information about writing a proposal at https://www.freebsd.org/projects/summerofcode/. Joe From nobody Wed Mar 29 02:53:12 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PmWMJ36nLz41pP5 for ; Wed, 29 Mar 2023 02:53:16 +0000 (UTC) (envelope-from sysadmin.lists@mailfence.com) Received: from wilbur.contactoffice.com (wilbur.contactoffice.com [212.3.242.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PmWMJ1z0nz41Tn for ; Wed, 29 Mar 2023 02:53:16 +0000 (UTC) (envelope-from sysadmin.lists@mailfence.com) Authentication-Results: mx1.freebsd.org; none Received: from ichabod.co-bxl (ichabod.co-bxl [10.2.0.36]) by wilbur.contactoffice.com (Postfix) with ESMTP id 1C3351C24; Wed, 29 Mar 2023 04:53:15 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1680058395; s=20210208-e7xh; d=mailfence.com; i=sysadmin.lists@mailfence.com; h=Date:From:To:Message-ID:In-Reply-To:References:Subject:MIME-Version:Content-Type:Content-Transfer-Encoding:Cc; l=1949; bh=e8NcQV5005//bLi+f2qYlsYWggKY7tLR7Rq6h1gaUfQ=; b=QG0oXltj4Udt1wzqlV79FuQC9kEthinZBChGEWCUNWoDtIku18WoRJsrKwXhxlp0 6yCP6BC11HrTsWotPtN+iCwlBpO7fcHMfWoaRpO+dPtbc3MY40NDuadP6rn4Y6X20wN duPy16f8Pqwhxd/ovT4cRJuRM8LpVLWRLuNaONOMCEHe+9BaUdfLSfGkP3NVIZPJ80R yf/lv5T5GILNB9yCfamjMo5jDXxwXcxRpaqL67TmyCAFVZoGfu3Zf0SiD673rhAIYpv VruM+hXDzjM7Gzd3yp32n7J+TlYqe9BxwbFwkv8rdb29H7o3Ab0VXzcDQoydJ22OpNb EJU3LQhFMA== Date: Wed, 29 Mar 2023 04:53:12 +0200 (CEST) From: Sysadmin Lists To: freebsd-hackers@freebsd.org Message-ID: <57699630.208778.1680058392828@ichabod.co-bxl> In-Reply-To: References: Subject: Re: what's the Correct git update method keeping local changes List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: void X-Mailer: ContactOffice Mail X-ContactOffice-Account: com:312482426 X-Rspamd-Queue-Id: 4PmWMJ1z0nz41Tn X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:10753, ipnet:212.3.242.64/26, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N > ---------------------------------------- > From: void > Date: Mar 27, 2023, 8:19:19 PM > To: > Subject: what's the Correct git update method keeping local changes > > What's the best way? > -- > I use the following interactive script to update and maintain local ports mods. It updates the poudriere jail, switches branches when available, and merges changes to my custom branch. #!/usr/local/bin/bash # update poudriere's ports git repos jail_name="FreeBSD:13:amd64" logfile="/usr/local/etc/poudriere.d/quarterly.log" ports_dir="/usr/local/poudriere/ports/quarterly" ports_list="[list of customized category/portname pairs goes here]" curb=$(git -C $ports_dir branch | sed -n 'x;$p') read -p "==> Update jails? [N|y] " update_jails if [[ $update_jails == "y" ]]; then poudriere jail -uj $jail_name fi # prompt to switch branches if local quarter branch and current quarter differ read month year <<< $(date "+%m %Y") awk 'BEGIN { exit "'${curb: -1}'" == int(('$month' / 3) + .9) }' # (true is 1) if [[ $? == 0 ]]; then read -p "==> Change branch? [n|Y] " change_branch if [[ ! $change_branch == "n" ]]; then set -ve newb=$(git -C $ports_dir branch -rl "*${year}Q*" | sed '$!d') git -C $ports_dir fetch git -C $ports_dir checkout ${newb# origin/} git -C $ports_dir worktree list curb=${newb# origin/} set +ve fi fi read -p "==> Update branches? [n|Y] " update_ports if [[ ! $update_ports == "n" ]]; then set -ve poudriere ports -vup latest git -C $ports_dir checkout $curb poudriere ports -vup quarterly | tee $logfile git -C $ports_dir checkout custombranch git -C $ports_dir rebase $curb for port in $ports_list; do if grep --quiet $port $logfile; then rsync -ax $ports_dir/$port/ \ $ports_dir/${port/*\//custom/}/ fi done set +ve fi exit -- Sent with https://mailfence.com Secure and private email From nobody Wed Mar 29 03:40:58 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PmXQR0R9zz41slY for ; Wed, 29 Mar 2023 03:41:03 +0000 (UTC) (envelope-from void@f-m.fm) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PmXQP5pgCz45H5 for ; Wed, 29 Mar 2023 03:41:01 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm2 header.b=n+fbcX1s; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=VYYmRM0w; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 66.111.4.25 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 03AE05C012C for ; Tue, 28 Mar 2023 23:41:01 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Tue, 28 Mar 2023 23:41:01 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm2; t=1680061260; x=1680147660; bh=HF Ae85Lo7GzkhIK7lZ6zB7dwjoMZYxFcMGAkyQUsENU=; b=n+fbcX1sk6pWscOhrE +BGxBcaw/cSVu1y+AyajknnRDjNr31bRi4wrQG5rMOqAqAq0lpdOjYUW7avX25d5 FH7LxoVGLJHl3Bh2aXW4QFeusrL58lxKlSWnL4BVRWpNf6UKG3Hlb744MepNjeA+ mL0QLC1VoHKnHIiDkxCXgN1mpTzBJ6SGulI5OvRPujmXIWu+6mqli+yiyMV+tBY2 ujWH1+ysfw/FioBJ3W84zIxK9EZAY7B33um215O0TRP/Bo95HGzNw7Q6jfbyZMS6 7o7jydYn5suZWFchPNBOlA/BlJvDJ7ev13WFZes6+rJx/rIIy7RpWDwtHqcwQXpb LepQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; t=1680061260; x=1680147660; bh=HFAe85Lo7Gzkh IK7lZ6zB7dwjoMZYxFcMGAkyQUsENU=; b=VYYmRM0wrgNDMs6KwUn20NtrT88TP 8b2wgR9o6ryazisqHdLAQKN6QJlOb8uBRLOWdd/6U/33a5fS6zhZxgsf0v9J3mO6 64Opt5xg1LYVNxQfxH4wSgdgTuPHW1F/+3a/InE8TjCWKSf5pMGTCo1CNxSIZk1t GPO/HnOGvaaXdkfppr1VzMMlFLp8q8oZI9XfwYNXukulhyYKwFLmlv9ETfmemVUq 2Y8NKgC11gQAXkd/CBWA+O8b5wop2HvdQmbx2EKI++bThdwvefSaR2pSTsxfSGPU 0sU7wWoj8qIDCrPC8cBLdRGw1aqrmVMEdT9FUWHiMliccRArONVf1D49A== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdehhedgjeegucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggujgesthdtre dttddtvdenucfhrhhomhepvhhoihguuceovhhoihgusehfqdhmrdhfmheqnecuggftrfgr thhtvghrnhepkeeluddvlefhieelfefggffhffektdehleelgfdugfdvgeekjeejuddthe ehgfeunecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhep vhhoihgusehfqdhmrdhfmh X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Tue, 28 Mar 2023 23:41:00 -0400 (EDT) Date: Wed, 29 Mar 2023 04:40:58 +0100 From: void To: freebsd-hackers@freebsd.org Subject: Re: what's the Correct git update method keeping local changes Message-ID: Mail-Followup-To: freebsd-hackers@freebsd.org References: <57699630.208778.1680058392828@ichabod.co-bxl> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <57699630.208778.1680058392828@ichabod.co-bxl> X-Spamd-Result: default: False [-3.77 / 15.00]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm2,messagingengine.com:s=fm2]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.25]; RWL_MAILSPIKE_GOOD(-0.10)[66.111.4.25:from]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.25:from]; NEURAL_HAM_SHORT(-0.07)[-0.073]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:19151, ipnet:66.111.4.0/24, country:US]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[f-m.fm]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4PmXQP5pgCz45H5 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On Wed, Mar 29, 2023 at 04:53:12AM +0200, Sysadmin Lists wrote: > >I use the following interactive script to update and maintain local ports mods. >It updates the poudriere jail, switches branches when available, and merges >changes to my custom branch. Thank you for posting that script, it looks interesting! -- From nobody Wed Mar 29 22:47:24 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pn1sH43dTz420xm for ; Wed, 29 Mar 2023 22:47:31 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pn1sG4dX0z442V for ; Wed, 29 Mar 2023 22:47:30 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=WUSAavBz; spf=pass (mx1.freebsd.org: domain of grahamperrin@gmail.com designates 2a00:1450:4864:20::32b as permitted sender) smtp.mailfrom=grahamperrin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-wm1-x32b.google.com with SMTP id i5-20020a05600c354500b003edd24054e0so12353565wmq.4 for ; Wed, 29 Mar 2023 15:47:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680130049; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=Pn+TyxYjgAx5Z0xu5Ga4bLGdtqJf3BTyq1Tu8HBz9KQ=; b=WUSAavBz1PR0xU3z2OIHb0sAusQPtOr2br45nrTcv6Jss3ui7T5a6isaIgRc4p7O/N GnNuhb3TDIVojobJLjdW2+DQwJnkOrJhCFY4wEL4D7stsmn4qL7HpshvZlvJ4EmsiH7S Ju9fLEmLCKd0iYs9bjUaDQ1RcqtZF0e3XzQDbwvNKcDy1zJi/dYO7C60aUDL7w4yhBkf mNAgSACpkVhMBqqHI73h/0X0SamTOaFMBc/hS18m/nWB1GC0BQr7aDdV/gKJzr8G3IoK DZJeF193N+HWh8RycIPixLt93uBYBinz00W1RQWsOzjS8TsMaHCUFV7s6tgr4NGnr4jr DZWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680130049; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Pn+TyxYjgAx5Z0xu5Ga4bLGdtqJf3BTyq1Tu8HBz9KQ=; b=kO/2+GN99/PzAMXW/V44+jo2mJ9OEjbTRFOgvi3EtLWs1ukcz4jhdBgoKkemn8ffId BkZXYQL+GD7lMwFRQvRc35fxt2kt2Ji0MVs7u6f0BN5aWNUM/tGpts/nMqYMFoG7m3v1 ZzeZxAQtZIWz2B7T4yR4VXgVzb0s8zz1Jr7eawT3V42kpgKLQj+cMHWgPtpYfVI1LiaF xxSna5C3ZEtOgY2/1rgqK4lXCHyf5dI0r/fFZPsY2VL2qKG+B/d1Nv0mA+SnENMAxI+n UQ5UJ9aaHqjpwUuuhH0qphe97Gd7MdRrORIPJlBjHyZEgYo7qmbR2EYtAP4Bb7T1kVuK eHKw== X-Gm-Message-State: AAQBX9dEQJSuqI2RUf0G/K4QUooX5EVfPdtmpgkWZi07x7HEAoiJa9ZC PX6Sj+NuVvTyU/mTqLvRrzlhp53LyTWjAA== X-Google-Smtp-Source: AKy350aihqmKj/GqNlc+YoekbTygh9kxoRuY+9xyZ3TlaYPido6qNvXLrizCG872LwLrZA4OR8NGYw== X-Received: by 2002:a1c:7209:0:b0:3ef:5e6c:ef03 with SMTP id n9-20020a1c7209000000b003ef5e6cef03mr13303674wmc.16.1680130048822; Wed, 29 Mar 2023 15:47:28 -0700 (PDT) Received: from ?IPV6:2001:470:1f1c:a0::2? (tunnel642390-pt.tunnel.tserv1.lon2.ipv6.he.net. [2001:470:1f1c:a0::2]) by smtp.gmail.com with ESMTPSA id f18-20020a1c6a12000000b003ed2276cd0dsm3586533wmc.38.2023.03.29.15.47.26 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Mar 2023 15:47:27 -0700 (PDT) Message-ID: <5ee400db-bd3c-ee67-643f-d5f24b187834@gmail.com> Date: Wed, 29 Mar 2023 23:47:24 +0100 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 Subject: poudriere ports -u (was: what's the Correct git update method keeping local changes) Content-Language: en-US To: freebsd-hackers@freebsd.org References: <708822f5-2a2f-35f7-8f79-bf48ff644a22@FreeBSD.org> From: Graham Perrin In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-3.97 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.97)[-0.970]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; MIME_GOOD(-0.10)[text/plain]; FREEFALL_USER(0.00)[grahamperrin]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::32b:from]; DKIM_TRACE(0.00)[gmail.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4Pn1sG4dX0z442V X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On 28/03/2023 15:25, void wrote: > … I've not looked/found what git subcommand 'poudriere ports -u' maps > to, yet. … So, for example: /usr/local/bin/git -C /usr/local/poudriere/ports/default pull --rebase -q From nobody Thu Mar 30 05:13:39 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PnBQx6mJVz42SLf for ; Thu, 30 Mar 2023 05:13:45 +0000 (UTC) (envelope-from void@f-m.fm) Received: from wout4-smtp.messagingengine.com (wout4-smtp.messagingengine.com [64.147.123.20]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PnBQw6d4dz3scL for ; Thu, 30 Mar 2023 05:13:44 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm2 header.b=oUBOx1lf; dkim=pass header.d=messagingengine.com header.s=fm2 header.b="m mFPf16"; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 64.147.123.20 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id D474A3200344 for ; Thu, 30 Mar 2023 01:13:42 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Thu, 30 Mar 2023 01:13:42 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-transfer-encoding:content-type:content-type:date:date :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm2; t= 1680153222; x=1680239622; bh=TS1NJSXa4M9iSWhsDhEup/0MkYjdEuBP/jv +pfTTVP8=; b=oUBOx1lfHrVGikNYCVUYz63zxKtjrC21ltfNxHpb6E2rds7te9w F/Zq6grIq6zCxBSwVwIu5FKld9u2SZWew2RjjcHGa0xAknRzd/2sh+hQ8QiIRj44 NU0UhhwRI29miQwJUTSnc/6Rm9pvxWFGYhr217k4Plxp7dQxU5Tw1Lwj9w/esqrB jZ402G+hsAreJzEmHNIv4t6VM7rb2fbdrPTno3EHMKimp7vZFtFp/z+38qZUQR7p O3MSHgCbH52UqvxyJHNOH4uIxYP08j7qL0YatOZwQgunGpu94OeG+5nBo1f9bZ9a KvVSZtR9rxKNryiM4znRqUvDf6ld3yBjf5g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t=1680153222; x= 1680239622; bh=TS1NJSXa4M9iSWhsDhEup/0MkYjdEuBP/jv+pfTTVP8=; b=m mFPf16/EfS40tykO2HaBPzGf+7OPve49oYWKhS1o7InA8PCLvOEeiqqxv76AWK9b Y4gzzxoPymF3oWRRUWjhfdZXKKLDoq+O95uc+Bg2OjdHSU10Ad5LY9eO/Wt8VQTa EJB6bxbvqblDC2g3GyxtaT8z2TaKDEHofb4gfGcSGHj/SAT2k2trpUTRim9mG80c wx38Voro+WGrpL30D0h6NVoslAzOf4PX2CUrAgm8LK701JpO0YNEj/5Fpxy4c5+/ PJyGvxCxuTURd14C/vc2qEVWLovnnAgAZIPPt+HruSTg5J0br5J5BHh5pkXBevSX Qf0p+lq6VDrWZj1wfALWw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdehjedgledvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggugfgjsehtke ertddttdejnecuhfhrohhmpehvohhiugcuoehvohhiugesfhdqmhdrfhhmqeenucggtffr rghtthgvrhhnpeefvdetleffiedtfffgueekleeuffehkeekjeefiedvffejtdelieehtd elgeefieenucffohhmrghinhepghhithhhuhgsrdgtohhmpdhpohhrthhsrdhshhenucev lhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehvohhiugesfh dqmhdrfhhm X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 30 Mar 2023 01:13:41 -0400 (EDT) Date: Thu, 30 Mar 2023 06:13:39 +0100 From: void To: freebsd-hackers@freebsd.org Subject: Re: poudriere ports -u (was: what's the Correct git update method keeping local changes) Message-ID: Mail-Followup-To: freebsd-hackers@freebsd.org References: <708822f5-2a2f-35f7-8f79-bf48ff644a22@FreeBSD.org> <5ee400db-bd3c-ee67-643f-d5f24b187834@gmail.com> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <5ee400db-bd3c-ee67-643f-d5f24b187834@gmail.com> X-Spamd-Result: default: False [-4.60 / 15.00]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; MID_RHS_NOT_FQDN(0.50)[]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.20]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm2,messagingengine.com:s=fm2]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.20:from]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[f-m.fm]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4PnBQw6d4dz3scL X-Spamd-Bar: ---- X-ThisMailContainsUnwantedMimeParts: N On Wed, Mar 29, 2023 at 11:47:24PM +0100, Graham Perrin wrote: >On 28/03/2023 15:25, void wrote: > >> … I've not looked/found what git subcommand 'poudriere ports -u' maps >> to, yet. … > > > >So, for example: > >/usr/local/bin/git -C /usr/local/poudriere/ports/default pull --rebase -q Thanks. I'm updating the ports tree with git now then null mounting it as it avoids the issues I was having. Do you know what poudriere is doing when it's at the "inspecting ports for changes" phase? It's at that stage where the delay is the longest; most recently taking over 10 minutes. -- From nobody Thu Mar 30 14:03:46 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PnQBm3k8zz41p8D for ; Thu, 30 Mar 2023 14:04:00 +0000 (UTC) (envelope-from lorenzo.castellini28@gmail.com) Received: from mail-qk1-x733.google.com (mail-qk1-x733.google.com [IPv6:2607:f8b0:4864:20::733]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PnQBl57c0z3t4Y; Thu, 30 Mar 2023 14:03:59 +0000 (UTC) (envelope-from lorenzo.castellini28@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=nYWblO1h; spf=pass (mx1.freebsd.org: domain of lorenzo.castellini28@gmail.com designates 2607:f8b0:4864:20::733 as permitted sender) smtp.mailfrom=lorenzo.castellini28@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-qk1-x733.google.com with SMTP id z33so5860543qko.6; Thu, 30 Mar 2023 07:03:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680185038; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=AGb6sKcAAKCMX17OmhYeg82xDGXzMy/ACFxVe8I7h7s=; b=nYWblO1hRWbSTGW+GBJbbsqB1bWsWW/DRfZB8VCj8f1raM6g4xXK28ubmxb36n4hia RqEv7kBDeNa6gTgrdrY85t8TV7xbbdidSu9eRVxk10XKg3Nw7HgA3vKO18PstB2vf0Il 4OZ0nM4DCQKWBGhYkUw9YER0RHiVsgB2X7c540ry8OFpilR1J6gIhe9QnQ2RVyHnb7TC fw4evJbLr+F7qjtcVpSXSPbeq+oYh2BHiCc/403suxXOeSYTAulEc4C6MoCoHRxUn28a CB++sMAhbQ1DYvAewGDRqvtnVz6cpsK8B1vGVpFR7wmWlDgCwbTKOIIXxR++z5eQS7+i TUOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680185038; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=AGb6sKcAAKCMX17OmhYeg82xDGXzMy/ACFxVe8I7h7s=; b=tDz+3wyRz7IbewfFPXzNlPemlGQKGlS8ZRP8MfYf1au7987TfXXcEuaTCcO9KaBluX x0+4uUuaRM2lCeueS4j4e/AsdpDs0ZKNqtFO0NoB+E9bRXqcxX47nUkz8ITHAMfEFMlC wrpk0HWNSPRMpxA0XdvyCD5ibKsGzyM18IkPzK5XO90y+iEwc7O2iTjnFqFnOxtQN67V 2zt898W6k+diG6oY5EPz05cUehgSa7PQGOWyKYsjDHQF/HE3YKI88ov/jOGJE3FHr1nP 6PeW8VhhgJXsQHMBGaRMHJF/qq9KV8y4pTvn3ycw+Z+PD984W37bL5Hczwn3gg/1cLgz td9w== X-Gm-Message-State: AO0yUKWNak9x6h8obV5y358his91A8T+Orfga177021TYkWi4Mr1ZN8m s2nQFI9CtM11CZRzewyOuzpg7L/lsfVOjHJbr067c9QBQzjiMQ== X-Google-Smtp-Source: AK7set/qq8kDvMtDuyAioiG1l4mDBQvpUXN5JtKWM4atIoDfzBZy87ys/RQYSOeCZMdTXIlwqitJr+UqBvZ7hgAZDC0= X-Received: by 2002:a05:620a:372b:b0:746:7beb:477e with SMTP id de43-20020a05620a372b00b007467beb477emr4630080qkb.10.1680185037520; Thu, 30 Mar 2023 07:03:57 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <865yakwc9y.fsf@phe.ftfl.ca> In-Reply-To: <865yakwc9y.fsf@phe.ftfl.ca> From: Lorenzo Castellini Date: Thu, 30 Mar 2023 10:03:46 -0400 Message-ID: Subject: Re: Google Summer of Code To: Joseph Mingrone Cc: freebsd-hackers@freebsd.org, mmokhi@freebsd.org Content-Type: multipart/alternative; boundary="00000000000051830f05f81e90e2" X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.996]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::733:from]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; RCVD_TLS_LAST(0.00)[]; TAGGED_FROM(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4PnQBl57c0z3t4Y X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N --00000000000051830f05f81e90e2 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi, I hope all is well. Firstly, I appreciate your time and the email response. Unfortunately, due to the proposal due date being not so far away, I do not think I will make it in time to send a complete proposal. Nonetheless, if I come up with any new ideas that are more complete, I will let you know. Again, thanks for your time and consideration. Sincerely, Lorenzo Castellini Coutin On Tue, Mar 28, 2023 at 9:56=E2=80=AFPM Joseph Mingrone w= rote: > On Mon, 2023-03-27 at 18:07, Lorenzo Castellini < > lorenzo.castellini28@gmail.com> wrote: > > > Greetings, > > I hope this email finds you well. I am Lorenzo Castellini, a senior hig= h > > school student, with an interest in participating in the Google Summer = of > > Code. I am writing this email, since I am interested in collaborating > with > > you guys on the project, but I am in need of a bit of guidance with the > > project proposal. I checked out the project list, and I was particularl= y > > interested in a project of virtual memory compression. I would like to > work > > with this, maybe extending the project even further or applying some > > automation to such a process. On another note, I would also like to wor= k > > with something similar, but with other PC components. Let me know of so= me > > ideas according to my interests. Thanks! > > > Sincerely, > > Lorenzo Castellini Coutin > > Hi Lorenzo, > > I don't see any proposed projects to work on virtual memory compression > on https://wiki.freebsd.org/SummerOfCodeIdeas. That's not to say that > you can't still propose something, but we should determine if this is a > viable project. It looks like there were some GSoC projects in this > area in 2015 and 2019. > > https://wiki.freebsd.org/SummerOfCode2015/MemoryCompressionDeduplication > https://wiki.freebsd.org/SummerOfCode2019Projects/VirtualMemoryCompressio= n > > I've copied Mahdi Mokhtari, who mentored the project in 2019. Mahdi, > could you share any information about how that project turned out? The > repository listed on the wiki returns HTTP 404. > > Lorenzo, there is some general information about writing a proposal at > https://www.freebsd.org/projects/summerofcode/. > > Joe > --00000000000051830f05f81e90e2 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,
I hope all is well. Firstly, I appreciate your tim= e and the email response. Unfortunately, due to the proposal due date being= not so far away, I do not think I will make it in time to send a complete = proposal. Nonetheless, if I come up with any new ideas that are more comple= te, I will let you know. Again, thanks for your time and=C2=A0consideration= .

Sincerely,

Lorenzo Cast= ellini Coutin

On Tue, Mar 28, 2023 at 9:56=E2=80=AFPM Joseph Mingrone = <jrm@freebsd.org> wrote:
On Mon, 2023-03-27 at= 18:07, Lorenzo Castellini <lorenzo.castellini28@gmail.com> wrote:

> Greetings,
> I hope this email finds you well. I am Lorenzo Castellini, a senior hi= gh
> school student, with an interest in participating in the Google Summer= of
> Code. I am writing this email, since I am interested in collaborating = with
> you guys on the project, but I am in need of a bit of guidance with th= e
> project proposal. I checked out the project list, and I was particular= ly
> interested in a project of virtual memory compression. I would like to= work
> with this, maybe extending the project even further or applying some > automation to such a process. On another note, I would also like to wo= rk
> with something similar, but with other PC components. Let me know of s= ome
> ideas according to my interests. Thanks!

> Sincerely,
> Lorenzo Castellini Coutin

Hi Lorenzo,

I don't see any proposed projects to work on virtual memory compression=
on https://wiki.freebsd.org/SummerOfCodeIdeas.=C2=A0 T= hat's not to say that
you can't still propose something, but we should determine if this is a=
viable project.=C2=A0 It looks like there were some GSoC projects in this area in 2015 and 2019.

https://wiki.freebsd.org/Sum= merOfCode2015/MemoryCompressionDeduplication
https://wiki.freebsd.org/S= ummerOfCode2019Projects/VirtualMemoryCompression

I've copied Mahdi Mokhtari, who mentored the project in 2019.=C2=A0 Mah= di,
could you share any information about how that project turned out?=C2=A0 Th= e
repository listed on the wiki returns HTTP 404.

Lorenzo, there is some general information about writing a proposal at
https://www.freebsd.org/projects/summerofcode/.
Joe
--00000000000051830f05f81e90e2-- From nobody Thu Mar 30 15:36:54 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PnSG35pmXz41wS9 for ; Thu, 30 Mar 2023 15:36:59 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oi1-x22d.google.com (mail-oi1-x22d.google.com [IPv6:2607:f8b0:4864:20::22d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PnSG24l9Cz4FJd for ; Thu, 30 Mar 2023 15:36:58 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=ls2zUXMx; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2607:f8b0:4864:20::22d as permitted sender) smtp.mailfrom=mjguzik@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oi1-x22d.google.com with SMTP id w133so14459472oib.1 for ; Thu, 30 Mar 2023 08:36:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680190616; h=to:subject:message-id:date:from:references:in-reply-to:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=a6/TRVrMkEnbxNUpH2qR/hnSibr0QBG6rHKWxogO+IE=; b=ls2zUXMx1CfXE0sLc9uHzx6iRhBYAGMbJ61ir2eAni2dfrB7/xmBSHzBycbL+IGARZ FPdHbXHrlYReZZZwk8Imm/QGIDdgS4OYuXtROc6MpJKbNOiKVM82Lu4OSs2FypqxNyw7 3nESNwgU/5YE3t9NyefI966R+Wew4VvdCSSN6vL2iVkhaG59ohZYOUUDcBp71bZOa0V/ jjNAyxQtuRV488qKO/sSB13Dh2TiHiUD4d2thIQ9oLLm6V1Miab8EKCJZJyln9siM/cV SVUOpnsRcZK0tWOjwVaMhGceepryIfG0R62Rh+Mh7Ot+Qbd6jsOrS0wIC7hYj7lDtrBv 1r9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680190616; h=to:subject:message-id:date:from:references:in-reply-to:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=a6/TRVrMkEnbxNUpH2qR/hnSibr0QBG6rHKWxogO+IE=; b=fHNr1ibx13UxnAZEWhQWbcZYP9wCE8VnxrhdqtQKCikyUp/NQFZ1AbWkDYz9Lb4uCR p/oyCZP1cUCYwzcPx4WA8kTzUzn8cEZayJUt1xEvDYRXYkV71UU5RuJptVuzd3YJ0nSD HTRvfsA9tRT2Mst0hzjloMtc+QA3+SPpa7DNfrzOUZhNAS1daQeq801mZWZATFFjwlTg TlI8ef1naSUqp8GBBHO0hc4WlODG5J7AeVj4NKbAkyUPS9nH3F8rSecIB003OoEVwYFd DbiG2Adx7V9kLrEkdv5aBFMMSkWi58HFs28VQjEjRMotWhAOP8U5ljIdK2cjTki+I6qc jcUQ== X-Gm-Message-State: AO0yUKWa0eHqD5ovXR4p7OHwOr6rFUiPRHw58FZ8tKbF+pcyuevyS9ww ZtXtvFUF06u5vkV4lo0MwQbR+sDH4sZOlrlifRZrClqb X-Google-Smtp-Source: AK7set/YhhS9pD/MvI318T2rnbBheti36mASphEN1EQGJ3eqghq6vkHgXNFxtd4/I7bYpd0iCq+fIqzOOjCZaePY6pU= X-Received: by 2002:aca:d7c1:0:b0:387:13ea:721a with SMTP id o184-20020acad7c1000000b0038713ea721amr6901271oig.4.1680190615302; Thu, 30 Mar 2023 08:36:55 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:ac9:7598:0:b0:49c:b071:b1e3 with HTTP; Thu, 30 Mar 2023 08:36:54 -0700 (PDT) In-Reply-To: References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> From: Mateusz Guzik Date: Thu, 30 Mar 2023 17:36:54 +0200 Message-ID: Subject: Re: Periodic rant about SCHED_ULE To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-3.98 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.99)[-0.985]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::22d:from]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4PnSG24l9Cz4FJd X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N I looked into it a little more, below you can find summary and steps forward. First a general statement: while ULE does have performance bugs, it has better basis than 4BSD to make scheduling decisions. Most notably it understands CPU topology, at least for cases which don't involve big.LITTLE. For any non-freak case where 4BSD performs better, it is a bug in ULE if this is for any reason other than a tradeoff which can be tweaked to line them up. Or more to the point, there should not be any legitimate reason to use 4BSD these days and modulo the bugs below, you are probably losing on performance for doing so. Bugs reported in this thread by others and confirmed by me: 1. failure to load-balance when having n CPUs and n + 1 workers -- the excess one stays on one the same CPU thread continuously penalizing the same victim. as a result total real time to execute a finite computation is longer than in the case of 4BSD 2. unfairness of nice -n 20 threads vs threads going frequently off CPU (e.g., due to I/O) -- after using only a fraction of the slice the victim has to wait for the cpu hog to use up its entire slice, rinse and repeat. This extends a 7+ minute buildkernel to over 67 minutes, not an issue on 4BSD I did not put almost any effort into investigating no 1. There is code which is supposed to rebalance load across CPUs, someone(tm) will have to sit through it -- for all I know the fix is trivial. Fixing number 2 makes *another* bug more acute and it complicates the whole ordeal. Thus, bug reported by me: 3. interactivity scoring is bogus -- originally introduced to detect "interactive" behavior by equating being off CPU with waiting for user input. One part of the problem is that it puts *all* non-preempted off CPU time into one bag: a voluntary sleep. This includes suffering from lock contention in the kernel, lock contention in the program itself, file I/O and so on, none of which has bearing on how interactive or not the program might happen to be. A bigger part of the problem is that at least today, the graphical programs don't even act this way to begin with -- they stay on CPU *a lot*. I asked people to provide me with the output of: dtrace -n 'sched:::on-cpu { @[execname] = lquantize(curthread->td_priority, 0, 224, 1); }' from their laptops/desktops. One finding is that most people (at least those who reported) use firefox. Another finding is that the browser is above the threshold which would be considered "interactive" for vast majority of the time in all reported cases. I booted a 2 thread vm with xfce and decided to click around. Spawned firefox, opened a file manager (Thunar) and from there I opened a movie to play with mpv. As root I spawned make -j 2 buildkernel. it was not particularly good :) I found that mpv spawns a bunch of threads, most notably 2 distinct threads for audio and video output. The one for video got a priority of 175, while the rest had either 88 or 89 -- the lowest for timesharing not considered interactive [note lower is considered better]. At the same time the file manager who was left in the background kept doing evil syscall usage, which as a result bouncing between a regular timesharing priority and one which made it "interactive", even though the program was not touched for minutes. Or to put it differently, the scheduler failed to recognize that mpv is the program to prioritize, all while thinking the background time waster is the thing to look after (so to speak). This brings us to fixing problem 2: currently, due to the existence of said problem, the interactivity scoring woes are less acute -- the venerable make -j example is struggling to get CPU time, as a result messing with real interactive programs to a lesser extent. If that gets fixed, we are in a different boat altogether. I don't see a clean solution. Right now I'm toying with the idea of either: 1. having programs explicitly tell the kernel they are interactive 2. adding a scheduler hook to /dev/dsp -- the observation is that if a program is producing sound it probably should get some cpu time in a timely manner. this would cover audio/video players and web browsers, but would not cover other programs (say a pdf reader). it may be it is good enough though -- Mateusz Guzik From nobody Thu Mar 30 17:44:15 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PnW4x2xQhz425p6 for ; Thu, 30 Mar 2023 17:44:17 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-ot1-x32c.google.com (mail-ot1-x32c.google.com [IPv6:2607:f8b0:4864:20::32c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PnW4w5gCyz4Z14; Thu, 30 Mar 2023 17:44:16 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=UzOPmPii; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2607:f8b0:4864:20::32c as permitted sender) smtp.mailfrom=mjguzik@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ot1-x32c.google.com with SMTP id r40-20020a05683044a800b006a14270bc7eso7092923otv.6; Thu, 30 Mar 2023 10:44:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680198256; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=l1aLaFdXnZ7GoMNUqs+POSTK6pObCykGrNLJHT9FIk4=; b=UzOPmPii9vUfnO5ruyjpag4Kz8lbJGPBSHW2FWJIiIPULct7zP7DNrNgus27JV8Xqh 3bzzYtVSRtTkHRbdrX+mGEj9flBcMu9NTHjOLQQQaBQ9XulPazpi/8yUB/8QedU/Jn6f Z+BpkGY3ZsqVoKGYJvkO1LKCbY+OSfF6V5bhkXd9bkUZSrQVRKOVlGLy8jSYGsKZuqRT rAJgSJckLe++ngw8XsDLlyY6WpBrkDq2VblQvxyDKJ8if5t6nAnfEOauLUaJ7YbcHjdZ 6OPf8vzz93WCInQpR4+IJSmuraHTwWqTa0Ezm0yOIPmdxAMh8fB4zXOE0GAxV8QNnwYj sNVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680198256; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=l1aLaFdXnZ7GoMNUqs+POSTK6pObCykGrNLJHT9FIk4=; b=kGc9++yTrGCw6w7pki+DF/vgzMygqMYsBxkCK0k4tDaGzcvjLuX72N7w8bsDC/08yR ACbyxA5yANhmh51PunPQ+7rFTP2S4K58sjouRtLh3J5XCThb5W4LWvz/aNpmv5rvp6Fc O3qmLeNlE/pZT15TTgulE/RpIL7tZNW5s4LVixTPZLBc6iCwavxHSeoIpvXBsVrEN+0u aPtQl8TqZMtetFcdahyCKeFpJTv1mPVpl2J5ZBBrAQ0aYP5q1YOzQG29zdAXYHic+bOz M1eGrFrD2SfmcxtOH67IM9jMga+vLyyxivI1zeB3w/M9QmjrbbhUPrH4ZZr/ch5Ev6XQ TSnQ== X-Gm-Message-State: AO0yUKWY+9t7nEKvlNnbH9unlKc/TdGPUfOj70MD200JlmjwsW05PKIu 9wkf3fGhrnRcUKlhnNV4lEWfLARiDS2+0ypNyLyiatU4 X-Google-Smtp-Source: AK7set9kBzM75na3sECqTvdVz29yWU8/JEa6p6x/RMbOLnDRq3EaoBm2vIkNTMwrnox9NT5zlnvNlEX98wk0d0C25oc= X-Received: by 2002:a9d:6a5a:0:b0:69f:573:6113 with SMTP id h26-20020a9d6a5a000000b0069f05736113mr7643244otn.2.1680198255872; Thu, 30 Mar 2023 10:44:15 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:ac9:7598:0:b0:49c:b071:b1e3 with HTTP; Thu, 30 Mar 2023 10:44:15 -0700 (PDT) In-Reply-To: References: <2f3dcda0-5135-290a-2dff-683b2e9fe271@FreeBSD.org> <2836EF0C-8B4B-441C-927B-3796457A3865@FreeBSD.org> From: Mateusz Guzik Date: Thu, 30 Mar 2023 19:44:15 +0200 Message-ID: Subject: Re: CFT: snmalloc as libc malloc To: David Chisnall Cc: freebsd-hackers Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; MIME_GOOD(-0.10)[text/plain]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::32c:from]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; FREEMAIL_FROM(0.00)[gmail.com]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4PnW4w5gCyz4Z14 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On 3/5/23, David Chisnall wrote: > Hi, > so i tried to bench but now this fails to compile on main after clang got updated to 15, for example: /usr/include/c++/v1/__functional/hash.h:646:34: error: use of undeclared identifier 'nullptr_t'; did you mean 'nullptr'? struct _LIBCPP_TEMPLATE_VIS hash can you rebase please and sort it out I also note your knob for disabling snmalloc memcpy does not sort out the kludge in rtld. I'm going to respond to the rest later. -- Mateusz Guzik From nobody Thu Mar 30 18:29:41 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PnX5d3CL9z428qQ for ; Thu, 30 Mar 2023 18:29:57 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Received: from mail-pl1-x629.google.com (mail-pl1-x629.google.com [IPv6:2607:f8b0:4864:20::629]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PnX5d1L43z3DWS for ; Thu, 30 Mar 2023 18:29:57 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pl1-x629.google.com with SMTP id u10so18919260plz.7 for ; Thu, 30 Mar 2023 11:29:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kev009.com; s=google; t=1680200996; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=lZCI7zgX8B/tyiJXgPThRp/J/zfWkqv/HD34yiO7+Fw=; b=qjS6tJImPFeWFph7cmx1Fb+DtqCOchewultjsCKD55OGVX3xDy6DP2g+5Rxy/fX9Zd QKOm4pjrSG5J3ytI4isvtb0fNgJ5yxM39MSddUddf+Yro/1vf2odYvfCaFtjmKCvlIfT FWtXJlNuloOkG5pBW205nqbyLZrmm9Lc6cxlY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680200996; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lZCI7zgX8B/tyiJXgPThRp/J/zfWkqv/HD34yiO7+Fw=; b=r+4skcVy/0Br5Z1SRsZfoOnvG5t1lCzPDZwXhSBLiJSowEYla5KkR8tOoXB4rb0wZD kaY7rGjMOMsq4JkZio6un6E+U6zeOndvil4ndBNolwb/Kr/cx6JIF3p4q4InIJO0+0NO lWFqfDiem1X+U4Awcyi0WJ5DBBBXoENdX2k2aUiE3eaPR31oBMC5sbm9A+KQ4ha/C5mV OKzA0k313KyQtzFSkR1oKR49b9Fh47za7srHw/gbhLkZXohTWqi940HNolt8vdOsaS1N jKyNmSg36o/JANYdneOnjI7fTPtBSQeQ2AA9B716LPUK9Y2y+lpqGoOgNSiWXrAt03dc 16SA== X-Gm-Message-State: AAQBX9c3eO9vAizkIagPdistlQkuDPdafYVsoA1vvIDw8wc7fCyQMzf/ W8JcGSDJkSQGSKhPSK45komPsWFed2I4t2N08sYrQgr3aK2cKbtZ X-Google-Smtp-Source: AKy350bBaO5k+DulI5TjcrRsFOYLMGiKyNHhgHIdcGbnr8L0Lg1Z9dDZzUxO1/55DiGoDDhenm01aoOm3MMOzZRPRk0= X-Received: by 2002:a17:90a:1681:b0:23f:63b3:f164 with SMTP id o1-20020a17090a168100b0023f63b3f164mr7173349pja.3.1680200995798; Thu, 30 Mar 2023 11:29:55 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> In-Reply-To: From: Kevin Bowling Date: Thu, 30 Mar 2023 11:29:41 -0700 Message-ID: Subject: Re: Periodic rant about SCHED_ULE To: Mateusz Guzik Cc: freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4PnX5d1L43z3DWS X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Thu, Mar 30, 2023 at 8:37=E2=80=AFAM Mateusz Guzik w= rote: > > I looked into it a little more, below you can find summary and steps forw= ard. > > First a general statement: while ULE does have performance bugs, it > has better basis than 4BSD to make scheduling decisions. Most notably > it understands CPU topology, at least for cases which don't involve > big.LITTLE. For any non-freak case where 4BSD performs better, it is a > bug in ULE if this is for any reason other than a tradeoff which can > be tweaked to line them up. Or more to the point, there should not be > any legitimate reason to use 4BSD these days and modulo the bugs > below, you are probably losing on performance for doing so. An elided simple algorithm for big.LITTLE, from Larry McVoy.. if you run for an entire quantum, flag preference for big core. If you run for less or get punted off, flag for little core preference. > Bugs reported in this thread by others and confirmed by me: > 1. failure to load-balance when having n CPUs and n + 1 workers -- the > excess one stays on one the same CPU thread continuously penalizing > the same victim. as a result total real time to execute a finite > computation is longer than in the case of 4BSD > 2. unfairness of nice -n 20 threads vs threads going frequently off > CPU (e.g., due to I/O) -- after using only a fraction of the slice the > victim has to wait for the cpu hog to use up its entire slice, rinse > and repeat. This extends a 7+ minute buildkernel to over 67 minutes, > not an issue on 4BSD > > I did not put almost any effort into investigating no 1. There is code > which is supposed to rebalance load across CPUs, someone(tm) will have > to sit through it -- for all I know the fix is trivial. > > Fixing number 2 makes *another* bug more acute and it complicates the > whole ordeal. > > Thus, bug reported by me: > 3. interactivity scoring is bogus -- originally introduced to detect > "interactive" behavior by equating being off CPU with waiting for user > input. One part of the problem is that it puts *all* non-preempted off > CPU time into one bag: a voluntary sleep. This includes suffering from > lock contention in the kernel, lock contention in the program itself, > file I/O and so on, none of which has bearing on how interactive or > not the program might happen to be. A bigger part of the problem is > that at least today, the graphical programs don't even act this way to > begin with -- they stay on CPU *a lot*. > > I asked people to provide me with the output of: dtrace -n > 'sched:::on-cpu { @[execname] =3D lquantize(curthread->td_priority, 0, > 224, 1); }' from their laptops/desktops. > > One finding is that most people (at least those who reported) use firefox= . > > Another finding is that the browser is above the threshold which would > be considered "interactive" for vast majority of the time in all > reported cases. > > I booted a 2 thread vm with xfce and decided to click around. Spawned > firefox, opened a file manager (Thunar) and from there I opened a > movie to play with mpv. As root I spawned make -j 2 buildkernel. it > was not particularly good :) > > I found that mpv spawns a bunch of threads, most notably 2 distinct > threads for audio and video output. The one for video got a priority > of 175, while the rest had either 88 or 89 -- the lowest for > timesharing not considered interactive [note lower is considered > better]. > > At the same time the file manager who was left in the background kept > doing evil syscall usage, which as a result bouncing between a regular > timesharing priority and one which made it "interactive", even though > the program was not touched for minutes. > > Or to put it differently, the scheduler failed to recognize that mpv > is the program to prioritize, all while thinking the background time > waster is the thing to look after (so to speak). > > This brings us to fixing problem 2: currently, due to the existence of > said problem, the interactivity scoring woes are less acute -- the > venerable make -j example is struggling to get CPU time, as a result > messing with real interactive programs to a lesser extent. If that > gets fixed, we are in a different boat altogether. > > I don't see a clean solution. > > Right now I'm toying with the idea of either: > 1. having programs explicitly tell the kernel they are interactive > 2. adding a scheduler hook to /dev/dsp -- the observation is that if a > program is producing sound it probably should get some cpu time in a > timely manner. this would cover audio/video players and web browsers, > but would not cover other programs (say a pdf reader). it may be it is > good enough though > > -- > Mateusz Guzik > From nobody Thu Mar 30 18:39:56 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PnXKT0MVXz429Qj for ; Thu, 30 Mar 2023 18:40:13 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Received: from mail-pf1-x436.google.com (mail-pf1-x436.google.com [IPv6:2607:f8b0:4864:20::436]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PnXKR3Mp1z3G1q for ; Thu, 30 Mar 2023 18:40:11 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Authentication-Results: mx1.freebsd.org; dkim=none ("invalid DKIM record") header.d=kev009.com header.s=google header.b=nnwButA3; spf=pass (mx1.freebsd.org: domain of kevin.bowling@kev009.com designates 2607:f8b0:4864:20::436 as permitted sender) smtp.mailfrom=kevin.bowling@kev009.com; dmarc=none Received: by mail-pf1-x436.google.com with SMTP id u38so13198241pfg.10 for ; Thu, 30 Mar 2023 11:40:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kev009.com; s=google; t=1680201610; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=T1GGLY+qy7DEJ9++krIcd5sz7q6HLP+656LY0VbabRQ=; b=nnwButA37V7H2RN/4QcY9mEI/nlZcANCbCtejqFHWOZPcnLme3iqsoYPM/ugB0zdFi Triz0xTvK0Dgsop+Mth9BWBAuolHQn3uxDt3j9bfXOm2ieXHPYFQ1TZvq5zEXuze7LYL KVZP/9PJzZBOpSugUjOxrocm/Tl4gQuIoEmsc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680201610; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=T1GGLY+qy7DEJ9++krIcd5sz7q6HLP+656LY0VbabRQ=; b=zOgzFn8GN8Bx2AlGUUSF7lg8gefnspghyMg4CSDPp+TeRyXVOlzfwNghfXBQTmEMgr lKblFS+heeEM1n7Om6lOTh7e/67JH+mXQNMUHye+mpKOjjO+oBIFzPzVUZ17kRGF8+zg VtEzJ7Qb09IbBhIsclkGBeKa/nIs3LvITE6zVpN3TSGRJfQ4XspDzf6gMMO47Mz+5hwj YU6HsKEIweNp1UE6KiNesrZ63UCSQOWVceh93X98TRq9MJspPt0qLNm2oQ7MWINeSE8I nOv5NYhBdoQhupvO7FbYEann/MKWVA6pzGOMhgSdHEX4ASX/Y6+zDjaE/pNsgZr6rqkk KSLw== X-Gm-Message-State: AAQBX9eVQvpfChToJRrXzp/CrGRy6lwm2706m2vpGwDRGxtdiGxboyQn bOuIZC/Zt8p0E1ZjwmaRWeihJlpoZF798uPEb0i3g32BCoOXPef9 X-Google-Smtp-Source: AKy350ZaGycRCl810GF8hbu6u2Ru+InfRBU3VYCLVvZFlmBg2FO+sFrd6QoUUSufsno5jJPR65LSQTm63aDxBivYv4s= X-Received: by 2002:a63:4d09:0:b0:503:25af:f50d with SMTP id a9-20020a634d09000000b0050325aff50dmr6444345pgb.4.1680201610044; Thu, 30 Mar 2023 11:40:10 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> In-Reply-To: From: Kevin Bowling Date: Thu, 30 Mar 2023 11:39:56 -0700 Message-ID: Subject: Re: Periodic rant about SCHED_ULE To: Mateusz Guzik Cc: freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-2.70 / 15.00]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.98)[-0.981]; NEURAL_HAM_LONG(-0.42)[-0.423]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; R_DKIM_PERMFAIL(0.00)[kev009.com:s=google]; FREEMAIL_TO(0.00)[gmail.com]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::436:from]; DKIM_TRACE(0.00)[kev009.com:~]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TO_DN_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[kev009.com]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4PnXKR3Mp1z3G1q X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N On Thu, Mar 30, 2023 at 11:29=E2=80=AFAM Kevin Bowling wrote: > > On Thu, Mar 30, 2023 at 8:37=E2=80=AFAM Mateusz Guzik = wrote: > > > > I looked into it a little more, below you can find summary and steps fo= rward. > > > > First a general statement: while ULE does have performance bugs, it > > has better basis than 4BSD to make scheduling decisions. Most notably > > it understands CPU topology, at least for cases which don't involve > > big.LITTLE. For any non-freak case where 4BSD performs better, it is a > > bug in ULE if this is for any reason other than a tradeoff which can > > be tweaked to line them up. Or more to the point, there should not be > > any legitimate reason to use 4BSD these days and modulo the bugs > > below, you are probably losing on performance for doing so. > > An elided simple algorithm for big.LITTLE, from Larry McVoy.. if you > run for an entire quantum, flag preference for big core. If you run > for less or get punted off, flag for little core preference. > > > Bugs reported in this thread by others and confirmed by me: > > 1. failure to load-balance when having n CPUs and n + 1 workers -- the > > excess one stays on one the same CPU thread continuously penalizing > > the same victim. as a result total real time to execute a finite > > computation is longer than in the case of 4BSD > > 2. unfairness of nice -n 20 threads vs threads going frequently off > > CPU (e.g., due to I/O) -- after using only a fraction of the slice the > > victim has to wait for the cpu hog to use up its entire slice, rinse > > and repeat. This extends a 7+ minute buildkernel to over 67 minutes, > > not an issue on 4BSD > > > > I did not put almost any effort into investigating no 1. There is code > > which is supposed to rebalance load across CPUs, someone(tm) will have > > to sit through it -- for all I know the fix is trivial. > > > > Fixing number 2 makes *another* bug more acute and it complicates the > > whole ordeal. > > > > Thus, bug reported by me: > > 3. interactivity scoring is bogus -- originally introduced to detect > > "interactive" behavior by equating being off CPU with waiting for user > > input. One part of the problem is that it puts *all* non-preempted off > > CPU time into one bag: a voluntary sleep. This includes suffering from > > lock contention in the kernel, lock contention in the program itself, > > file I/O and so on, none of which has bearing on how interactive or > > not the program might happen to be. A bigger part of the problem is > > that at least today, the graphical programs don't even act this way to > > begin with -- they stay on CPU *a lot*. > > > > I asked people to provide me with the output of: dtrace -n > > 'sched:::on-cpu { @[execname] =3D lquantize(curthread->td_priority, 0, > > 224, 1); }' from their laptops/desktops. > > > > One finding is that most people (at least those who reported) use firef= ox. > > > > Another finding is that the browser is above the threshold which would > > be considered "interactive" for vast majority of the time in all > > reported cases. > > > > I booted a 2 thread vm with xfce and decided to click around. Spawned > > firefox, opened a file manager (Thunar) and from there I opened a > > movie to play with mpv. As root I spawned make -j 2 buildkernel. it > > was not particularly good :) > > > > I found that mpv spawns a bunch of threads, most notably 2 distinct > > threads for audio and video output. The one for video got a priority > > of 175, while the rest had either 88 or 89 -- the lowest for > > timesharing not considered interactive [note lower is considered > > better]. > > > > At the same time the file manager who was left in the background kept > > doing evil syscall usage, which as a result bouncing between a regular > > timesharing priority and one which made it "interactive", even though > > the program was not touched for minutes. > > > > Or to put it differently, the scheduler failed to recognize that mpv > > is the program to prioritize, all while thinking the background time > > waster is the thing to look after (so to speak). > > > > This brings us to fixing problem 2: currently, due to the existence of > > said problem, the interactivity scoring woes are less acute -- the > > venerable make -j example is struggling to get CPU time, as a result > > messing with real interactive programs to a lesser extent. If that > > gets fixed, we are in a different boat altogether. > > > > I don't see a clean solution. One other random anecdote. Windows 11 uses window focus to highly boost scheduling priority in an obviously effective way. I have no idea how difficult something like that would be to fit into the unix world. > > Right now I'm toying with the idea of either: > > 1. having programs explicitly tell the kernel they are interactive > > 2. adding a scheduler hook to /dev/dsp -- the observation is that if a > > program is producing sound it probably should get some cpu time in a > > timely manner. this would cover audio/video players and web browsers, > > but would not cover other programs (say a pdf reader). it may be it is > > good enough though > > > > -- > > Mateusz Guzik > > From nobody Thu Mar 30 18:50:31 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PnXYN6MW0z42B43 for ; Thu, 30 Mar 2023 18:50:32 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-ot1-x32e.google.com (mail-ot1-x32e.google.com [IPv6:2607:f8b0:4864:20::32e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PnXYN4j6xz3HtT for ; Thu, 30 Mar 2023 18:50:32 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ot1-x32e.google.com with SMTP id r17-20020a05683002f100b006a131458abfso7968900ote.2 for ; Thu, 30 Mar 2023 11:50:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680202231; h=content-transfer-encoding:cc:to:subject:message-id:date:from :references:in-reply-to:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=HfVBgOye/IGioI3xBOIodiweQ4NocpBXIGNVsY/5aKg=; b=A3Jnq3fjQYJMZPcfhPTjVznhp16aVYGWqmCMV7qoqnbfvdAdZLXuggZ3YwAmXiL/uJ nOad6XUCiu4aRqeJirs7reJiIGxwAz1TL8Z1V4dd20fU33rLsd8C2QBRSUp0SeabOr4l JyUSmilBe3En1VsWorH+QGj9/ve7cJkl4HIYBiFVJNFViMmp0Ht9ZH81tkwFpz9FBU/T yfGC+ZZbi1iApcQPB2+xZ0cA6sQ/XZmkfINWu+hEoGqsx0Ts2wkU5BI6Ru9q/rPiSnHE R8+h7Xq51RAZhRTYIh3c7qApp9xv7oyJRrE48iCJhQI7mle5CcdTHwTCoEDyV4sLYD9A 6L2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680202231; h=content-transfer-encoding:cc:to:subject:message-id:date:from :references:in-reply-to:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HfVBgOye/IGioI3xBOIodiweQ4NocpBXIGNVsY/5aKg=; b=bBgqgnloHgH5eoiMigRfcgteZjIsy4pcMzH9PG8tY+nJ5q0JOj16knLHN2nz4mmDgL Ot24my0GlTrVe/PGDvg0yIWEwe2QVROxQPxhKABSht5t4fHuD7ZphGyUA4Ns3Cj3n2G1 mOm3dgQk3sgQUP532PF404Uz14G9Fe1TRGbhQWM5Djya7azzsAQ/ZUyeF+azoUsxiyo8 0r+cWiOjUh7rMXFsREzGjKolw+qdpWJDnadJVdNXPCvbhwFJTek9ZM93Twa+l17s7YLX hcYEe6rSqJ94eMkOYl7F/epmMs38zlbAewSKjvGNXzTukNm08znjozT8FhZXSzdDHboD Y34Q== X-Gm-Message-State: AO0yUKXxn6JS/yrTYnasHIVfz1hLI8BmtSHcz4DCyPy7Bm8u7RxDBbDs j1AvVZ5TPUAXPyzd/hI4jAkGdK7VzokgjB+kQvliJMvf X-Google-Smtp-Source: AK7set8grPTQij6aZ2x0lObjOhKxTVcEakMLF/Zxphq5rB517OYEtdzArAg7OGkpkNoRJwksHOFhie4zoqu7tjXUo00= X-Received: by 2002:a9d:63c5:0:b0:698:f988:7c30 with SMTP id e5-20020a9d63c5000000b00698f9887c30mr7928324otl.2.1680202231552; Thu, 30 Mar 2023 11:50:31 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:ac9:7598:0:b0:49c:b071:b1e3 with HTTP; Thu, 30 Mar 2023 11:50:31 -0700 (PDT) In-Reply-To: References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> From: Mateusz Guzik Date: Thu, 30 Mar 2023 20:50:31 +0200 Message-ID: Subject: Re: Periodic rant about SCHED_ULE To: Kevin Bowling Cc: freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4PnXYN4j6xz3HtT X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On 3/30/23, Kevin Bowling wrote: > On Thu, Mar 30, 2023 at 11:29=E2=80=AFAM Kevin Bowling > wrote: >> >> On Thu, Mar 30, 2023 at 8:37=E2=80=AFAM Mateusz Guzik wrote: >> > >> > I looked into it a little more, below you can find summary and steps >> > forward. >> > >> > First a general statement: while ULE does have performance bugs, it >> > has better basis than 4BSD to make scheduling decisions. Most notably >> > it understands CPU topology, at least for cases which don't involve >> > big.LITTLE. For any non-freak case where 4BSD performs better, it is a >> > bug in ULE if this is for any reason other than a tradeoff which can >> > be tweaked to line them up. Or more to the point, there should not be >> > any legitimate reason to use 4BSD these days and modulo the bugs >> > below, you are probably losing on performance for doing so. >> >> An elided simple algorithm for big.LITTLE, from Larry McVoy.. if you >> run for an entire quantum, flag preference for big core. If you run >> for less or get punted off, flag for little core preference. >> >> > Bugs reported in this thread by others and confirmed by me: >> > 1. failure to load-balance when having n CPUs and n + 1 workers -- the >> > excess one stays on one the same CPU thread continuously penalizing >> > the same victim. as a result total real time to execute a finite >> > computation is longer than in the case of 4BSD >> > 2. unfairness of nice -n 20 threads vs threads going frequently off >> > CPU (e.g., due to I/O) -- after using only a fraction of the slice the >> > victim has to wait for the cpu hog to use up its entire slice, rinse >> > and repeat. This extends a 7+ minute buildkernel to over 67 minutes, >> > not an issue on 4BSD >> > >> > I did not put almost any effort into investigating no 1. There is code >> > which is supposed to rebalance load across CPUs, someone(tm) will have >> > to sit through it -- for all I know the fix is trivial. >> > >> > Fixing number 2 makes *another* bug more acute and it complicates the >> > whole ordeal. >> > >> > Thus, bug reported by me: >> > 3. interactivity scoring is bogus -- originally introduced to detect >> > "interactive" behavior by equating being off CPU with waiting for user >> > input. One part of the problem is that it puts *all* non-preempted off >> > CPU time into one bag: a voluntary sleep. This includes suffering from >> > lock contention in the kernel, lock contention in the program itself, >> > file I/O and so on, none of which has bearing on how interactive or >> > not the program might happen to be. A bigger part of the problem is >> > that at least today, the graphical programs don't even act this way to >> > begin with -- they stay on CPU *a lot*. >> > >> > I asked people to provide me with the output of: dtrace -n >> > 'sched:::on-cpu { @[execname] =3D lquantize(curthread->td_priority, 0, >> > 224, 1); }' from their laptops/desktops. >> > >> > One finding is that most people (at least those who reported) use >> > firefox. >> > >> > Another finding is that the browser is above the threshold which would >> > be considered "interactive" for vast majority of the time in all >> > reported cases. >> > >> > I booted a 2 thread vm with xfce and decided to click around. Spawned >> > firefox, opened a file manager (Thunar) and from there I opened a >> > movie to play with mpv. As root I spawned make -j 2 buildkernel. it >> > was not particularly good :) >> > >> > I found that mpv spawns a bunch of threads, most notably 2 distinct >> > threads for audio and video output. The one for video got a priority >> > of 175, while the rest had either 88 or 89 -- the lowest for >> > timesharing not considered interactive [note lower is considered >> > better]. >> > >> > At the same time the file manager who was left in the background kept >> > doing evil syscall usage, which as a result bouncing between a regular >> > timesharing priority and one which made it "interactive", even though >> > the program was not touched for minutes. >> > >> > Or to put it differently, the scheduler failed to recognize that mpv >> > is the program to prioritize, all while thinking the background time >> > waster is the thing to look after (so to speak). >> > >> > This brings us to fixing problem 2: currently, due to the existence of >> > said problem, the interactivity scoring woes are less acute -- the >> > venerable make -j example is struggling to get CPU time, as a result >> > messing with real interactive programs to a lesser extent. If that >> > gets fixed, we are in a different boat altogether. >> > >> > I don't see a clean solution. > > One other random anecdote. Windows 11 uses window focus to highly > boost scheduling priority in an obviously effective way. I have no > idea how difficult something like that would be to fit into the unix > world. > I thought about doing something like that, but I consider it dodgy. Imagine you play some crap from youtube while messing around in a text editor -- I'm pretty sure the former is more prone to disturbance from scheduling changes. Anyhow after sending the above e-mail an actual solution hit me: the X server can tell the kernel what processes connect to it over the unix socket, which again very well may be good enough. In the reports I got I found pulseaudio, this one may need to be patched in a similar manner. --=20 Mateusz Guzik From nobody Thu Mar 30 19:05:01 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PnXtN5tx0z42BgG for ; Thu, 30 Mar 2023 19:05:16 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Received: from mail-pl1-x62c.google.com (mail-pl1-x62c.google.com [IPv6:2607:f8b0:4864:20::62c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PnXtN2ScMz3LSS for ; Thu, 30 Mar 2023 19:05:16 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pl1-x62c.google.com with SMTP id o2so19031121plg.4 for ; Thu, 30 Mar 2023 12:05:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kev009.com; s=google; t=1680203115; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=E1GimjrYwah0l7RBDHGlb8FHPgwgtcqsnvNK3pTYEqw=; b=eOmf3fboHhJsjs+qMB0IqBVziRlnR+R83T1nxYN9vOJiEMIZ3ZtLzIObN0O6mXnT0W wvyyokEu8W0205MSKXom4DNkerWQ6Fdw8o6kXRvexJPTlwbFJiRMzJNp8bELZB5/iMtx 2K3Yx7L0IQ+NzoUVX+nFVsazwUGuqE4TaJfbU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680203115; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=E1GimjrYwah0l7RBDHGlb8FHPgwgtcqsnvNK3pTYEqw=; b=06aW0NKd6L7VliYgZMnHyyyEhUMIaRU33pVQTjZ2I4/ws5xo6J8Rej9dc+PWbSXezw CTMJUEI6vZjlx4D27rICVjelV7arfBaVwgFR3zz1VL4ExzdsyVdYPxwpY32PggMdt5K2 zVBKLbuHM0zcOfixjSqgBVpMPCbpIHOf2XX8bJi1ERxWHBtE0MbXjAokWMGPGZqaPXo9 uRKlkLfbEDh4Ivpbfi7qsN8qoCAP1QeeqmoBEM9VOEsjUUV3/6c+J2pgBcaAyb1X8EtJ xXPBvhBYrsWJtSUeBuxBpIYX1WO+7J3S11BEORRDVqR/c4RApqZWKTQvoyiUnZZiy0Ff AXkg== X-Gm-Message-State: AAQBX9faKDNw5yTrCrREylQViRDNsq2OL1q4d5VUnVWJ0dRB+Mif3+Ic PnLoNLVatSe3Zi04NeGWq8lYadMjusyOsfIDqzhOvw== X-Google-Smtp-Source: AKy350YoMxAe08CkGIrO2m7NEwKZe23wXIhu+G4RlZfC8k3mkAlPkao41m67SLJk3sZN/IBPd7ou0FpygRGyR85XjXs= X-Received: by 2002:a17:902:dace:b0:1a0:41ea:b9ba with SMTP id q14-20020a170902dace00b001a041eab9bamr9376502plx.8.1680203115035; Thu, 30 Mar 2023 12:05:15 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> In-Reply-To: From: Kevin Bowling Date: Thu, 30 Mar 2023 12:05:01 -0700 Message-ID: Subject: Re: Periodic rant about SCHED_ULE To: Mateusz Guzik Cc: freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4PnXtN2ScMz3LSS X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Thu, Mar 30, 2023 at 11:50=E2=80=AFAM Mateusz Guzik = wrote: > > On 3/30/23, Kevin Bowling wrote: > > On Thu, Mar 30, 2023 at 11:29=E2=80=AFAM Kevin Bowling > > wrote: > >> > >> On Thu, Mar 30, 2023 at 8:37=E2=80=AFAM Mateusz Guzik wrote: > >> > > >> > I looked into it a little more, below you can find summary and steps > >> > forward. > >> > > >> > First a general statement: while ULE does have performance bugs, it > >> > has better basis than 4BSD to make scheduling decisions. Most notabl= y > >> > it understands CPU topology, at least for cases which don't involve > >> > big.LITTLE. For any non-freak case where 4BSD performs better, it is= a > >> > bug in ULE if this is for any reason other than a tradeoff which can > >> > be tweaked to line them up. Or more to the point, there should not b= e > >> > any legitimate reason to use 4BSD these days and modulo the bugs > >> > below, you are probably losing on performance for doing so. > >> > >> An elided simple algorithm for big.LITTLE, from Larry McVoy.. if you > >> run for an entire quantum, flag preference for big core. If you run > >> for less or get punted off, flag for little core preference. > >> > >> > Bugs reported in this thread by others and confirmed by me: > >> > 1. failure to load-balance when having n CPUs and n + 1 workers -- t= he > >> > excess one stays on one the same CPU thread continuously penalizing > >> > the same victim. as a result total real time to execute a finite > >> > computation is longer than in the case of 4BSD > >> > 2. unfairness of nice -n 20 threads vs threads going frequently off > >> > CPU (e.g., due to I/O) -- after using only a fraction of the slice t= he > >> > victim has to wait for the cpu hog to use up its entire slice, rinse > >> > and repeat. This extends a 7+ minute buildkernel to over 67 minutes, > >> > not an issue on 4BSD > >> > > >> > I did not put almost any effort into investigating no 1. There is co= de > >> > which is supposed to rebalance load across CPUs, someone(tm) will ha= ve > >> > to sit through it -- for all I know the fix is trivial. > >> > > >> > Fixing number 2 makes *another* bug more acute and it complicates th= e > >> > whole ordeal. > >> > > >> > Thus, bug reported by me: > >> > 3. interactivity scoring is bogus -- originally introduced to detect > >> > "interactive" behavior by equating being off CPU with waiting for us= er > >> > input. One part of the problem is that it puts *all* non-preempted o= ff > >> > CPU time into one bag: a voluntary sleep. This includes suffering fr= om > >> > lock contention in the kernel, lock contention in the program itself= , > >> > file I/O and so on, none of which has bearing on how interactive or > >> > not the program might happen to be. A bigger part of the problem is > >> > that at least today, the graphical programs don't even act this way = to > >> > begin with -- they stay on CPU *a lot*. > >> > > >> > I asked people to provide me with the output of: dtrace -n > >> > 'sched:::on-cpu { @[execname] =3D lquantize(curthread->td_priority, = 0, > >> > 224, 1); }' from their laptops/desktops. > >> > > >> > One finding is that most people (at least those who reported) use > >> > firefox. > >> > > >> > Another finding is that the browser is above the threshold which wou= ld > >> > be considered "interactive" for vast majority of the time in all > >> > reported cases. > >> > > >> > I booted a 2 thread vm with xfce and decided to click around. Spawne= d > >> > firefox, opened a file manager (Thunar) and from there I opened a > >> > movie to play with mpv. As root I spawned make -j 2 buildkernel. it > >> > was not particularly good :) > >> > > >> > I found that mpv spawns a bunch of threads, most notably 2 distinct > >> > threads for audio and video output. The one for video got a priority > >> > of 175, while the rest had either 88 or 89 -- the lowest for > >> > timesharing not considered interactive [note lower is considered > >> > better]. > >> > > >> > At the same time the file manager who was left in the background kep= t > >> > doing evil syscall usage, which as a result bouncing between a regul= ar > >> > timesharing priority and one which made it "interactive", even thoug= h > >> > the program was not touched for minutes. > >> > > >> > Or to put it differently, the scheduler failed to recognize that mpv > >> > is the program to prioritize, all while thinking the background time > >> > waster is the thing to look after (so to speak). > >> > > >> > This brings us to fixing problem 2: currently, due to the existence = of > >> > said problem, the interactivity scoring woes are less acute -- the > >> > venerable make -j example is struggling to get CPU time, as a result > >> > messing with real interactive programs to a lesser extent. If that > >> > gets fixed, we are in a different boat altogether. > >> > > >> > I don't see a clean solution. > > > > One other random anecdote. Windows 11 uses window focus to highly > > boost scheduling priority in an obviously effective way. I have no > > idea how difficult something like that would be to fit into the unix > > world. > > > > I thought about doing something like that, but I consider it dodgy. > Imagine you play some crap from youtube while messing around in a text > editor -- I'm pretty sure the former is more prone to disturbance from > scheduling changes. > > Anyhow after sending the above e-mail an actual solution hit me: the X > server can tell the kernel what processes connect to it over the unix > socket, which again very well may be good enough. > > In the reports I got I found pulseaudio, this one may need to be > patched in a similar manner. Yeah that seems like an easier problem, IMO something like a userspace audio server (or its init script) should be in charge of setting it to RT. > -- > Mateusz Guzik From nobody Thu Mar 30 19:15:56 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PnY6p62CTz42CbR for ; Thu, 30 Mar 2023 19:16:02 +0000 (UTC) (envelope-from guru@unixarea.de) Received: from ms-10.1blu.de (ms-10.1blu.de [178.254.4.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PnY6p48cQz3NWc for ; Thu, 30 Mar 2023 19:16:02 +0000 (UTC) (envelope-from guru@unixarea.de) Authentication-Results: mx1.freebsd.org; none Received: from [80.187.80.3] (helo=localhost.unixarea.de) by ms-10.1blu.de with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phxkd-00023m-2I; Thu, 30 Mar 2023 21:15:59 +0200 Received: from localhost.my.domain (localhost [127.0.0.1]) by localhost.unixarea.de (8.16.1/8.14.9) with ESMTPS id 32UJFv7i017010 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 30 Mar 2023 21:15:57 +0200 (CEST) (envelope-from guru@unixarea.de) Received: (from guru@localhost) by localhost.my.domain (8.16.1/8.14.9/Submit) id 32UJFu0w017009; Thu, 30 Mar 2023 21:15:56 +0200 (CEST) (envelope-from guru@unixarea.de) X-Authentication-Warning: localhost.my.domain: guru set sender to guru@unixarea.de using -f Date: Thu, 30 Mar 2023 21:15:56 +0200 From: Matthias Apitz To: Kevin Bowling Cc: Mateusz Guzik , freebsd-hackers@freebsd.org Subject: Re: Periodic rant about SCHED_ULE Message-ID: Reply-To: Matthias Apitz Mail-Followup-To: Kevin Bowling , Mateusz Guzik , freebsd-hackers@freebsd.org References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Operating-System: FreeBSD 13.0-CURRENT r368166 (amd64) X-message-flag: Mails containing HTML will not be read! Please send only plain text. X-Con-Id: 51246 X-Con-U: 0-guru X-Originating-IP: 80.187.80.3 X-Rspamd-Queue-Id: 4PnY6p48cQz3NWc X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:42730, ipnet:178.254.0.0/19, country:DE] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N Periodic rant about ... ? No, this is an ongoing thread about it and maybe it could come to an end. matthias -- Matthias Apitz, ✉ guru@unixarea.de, http://www.unixarea.de/ +49-176-38902045 Public GnuPG key: http://www.unixarea.de/key.pub No euro for the war! Keinen Euro für den Krieg! ¡No un Euro por la guerra! From nobody Thu Mar 30 20:08:29 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PnZHQ19dBz42GTj for ; Thu, 30 Mar 2023 20:08:34 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qt1-x829.google.com (mail-qt1-x829.google.com [IPv6:2607:f8b0:4864:20::829]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PnZHP6QSSz3lCS for ; Thu, 30 Mar 2023 20:08:33 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-qt1-x829.google.com with SMTP id ga7so19721220qtb.2 for ; Thu, 30 Mar 2023 13:08:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680206913; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:from:to:cc:subject:date:message-id :reply-to; bh=z+nC/WAXNKIIKmyzMSSP/eyJjuMTMi7ZCmVR04txc2Y=; b=DzuaMHwsCKqOptnXHu9jEXRoxaHGmGl+rTMr/u47wmjrFYn5YxYQbfhqXAAVeJIc66 fcl6PSbvX3izG2B4fgcWHnFKQw7IGQppr75nHz6A11Jsxnt8oWjbpff9toCz3cln4T1m pz6r72E3Yfqp2hCn2/qALY+yybMsNbTiGs1PUZ5s1FhiEkvq/sb4cyQyOLrBjmCRTZCT aEv/D7q4Urig408HKwomro+WzqQnKRZMdC2idouYaupPiErgD6exjdvemVPKqR3Fv1xh ICKX6/evW9MK9+Ag4+iujeJdy5hL+YC8/wjnySDzRiPhOghgWbECBVeoL5EvBdRGCjGh jiBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680206913; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=z+nC/WAXNKIIKmyzMSSP/eyJjuMTMi7ZCmVR04txc2Y=; b=2kmR7tr62Y6gpt6aDoqYxl34rvPHbU7EWVCi8O9sCO6+mQ/958vX9G415XevRpAkCK 8ZnpjJ2KO6QYRym+420ZT8wywwhJqnqo2hiVS3P7sejym2bxLVcT4IlYD8DVgy/SOT1j lPplla5+JMblJVeyH7HRmztGZFKeZwqS5yJhXhtpMNTWQwOlyK07ZkObsDvcMbGEKc7K NkJwk3tcVEfJsWb/lqzG+xpgfipZhBeYXhfdPWaiDqSHWOXxe46pGuolH1n9kYO5Guih 15GwcevXv99zcA2fBvnOuSX/IKcrVJLqMaDRfU3eUAUdGAbnLsFQTW4FtyVofpAvww2s 6HhQ== X-Gm-Message-State: AO0yUKWyl9i7sOhtT0D3DBZhpQa6QLzbcKTdzBHYpgDWp1thZ7076INq ABldY6qPw0eAIfvZanGfI60gBES3Pak= X-Google-Smtp-Source: AKy350a9VHtJvKS2mjNf59s5E9ip+1ZDVdRoN7dip55UpDsQpZSoxuzyxrzwYOsiTMPxSqJpQCo5iQ== X-Received: by 2002:ac8:7d8d:0:b0:3bf:d4c3:365d with SMTP id c13-20020ac87d8d000000b003bfd4c3365dmr40337550qtd.14.1680206912989; Thu, 30 Mar 2023 13:08:32 -0700 (PDT) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id u5-20020a37ab05000000b007484a7fa2d2sm126565qke.34.2023.03.30.13.08.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Mar 2023 13:08:32 -0700 (PDT) Date: Thu, 30 Mar 2023 16:08:29 -0400 From: Mark Johnston To: Mateusz Guzik Cc: freebsd-hackers@freebsd.org Subject: Re: Periodic rant about SCHED_ULE Message-ID: References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4PnZHP6QSSz3lCS X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Thu, Mar 30, 2023 at 05:36:54PM +0200, Mateusz Guzik wrote: > I looked into it a little more, below you can find summary and steps forward. > > First a general statement: while ULE does have performance bugs, it > has better basis than 4BSD to make scheduling decisions. Most notably > it understands CPU topology, at least for cases which don't involve > big.LITTLE. For any non-freak case where 4BSD performs better, it is a > bug in ULE if this is for any reason other than a tradeoff which can > be tweaked to line them up. Or more to the point, there should not be > any legitimate reason to use 4BSD these days and modulo the bugs > below, you are probably losing on performance for doing so. > > Bugs reported in this thread by others and confirmed by me: > 1. failure to load-balance when having n CPUs and n + 1 workers -- the > excess one stays on one the same CPU thread continuously penalizing > the same victim. as a result total real time to execute a finite > computation is longer than in the case of 4BSD > 2. unfairness of nice -n 20 threads vs threads going frequently off > CPU (e.g., due to I/O) -- after using only a fraction of the slice the > victim has to wait for the cpu hog to use up its entire slice, rinse > and repeat. This extends a 7+ minute buildkernel to over 67 minutes, > not an issue on 4BSD > > I did not put almost any effort into investigating no 1. There is code > which is supposed to rebalance load across CPUs, someone(tm) will have > to sit through it -- for all I know the fix is trivial. > > Fixing number 2 makes *another* bug more acute and it complicates the > whole ordeal. > > Thus, bug reported by me: > 3. interactivity scoring is bogus -- originally introduced to detect > "interactive" behavior by equating being off CPU with waiting for user > input. One part of the problem is that it puts *all* non-preempted off > CPU time into one bag: a voluntary sleep. This includes suffering from > lock contention in the kernel, lock contention in the program itself, Note that time spent off-CPU on a turnstile is not counted as sleeping for the purpose of interactivity scoring, so this observation applies only to sx, lockmgr and sleepable rm locks. That's not to say that this behaviour is correct, but it doesn't apply to some of the most contended locks unless I'm missing something. > file I/O and so on, none of which has bearing on how interactive or > not the program might happen to be. A bigger part of the problem is > that at least today, the graphical programs don't even act this way to > begin with -- they stay on CPU *a lot*. I think this statement deserves more nuance. I was on a video call just now and firefox was consuming about the equivalent of 20-30% of a CPU across all threads. What kind of graphical programs are you talking about specifically? > I asked people to provide me with the output of: dtrace -n > 'sched:::on-cpu { @[execname] = lquantize(curthread->td_priority, 0, > 224, 1); }' from their laptops/desktops. > > One finding is that most people (at least those who reported) use firefox. > > Another finding is that the browser is above the threshold which would > be considered "interactive" for vast majority of the time in all > reported cases. That is not true of the output that I sent. There, most of the firefox thread samples are in the interactive range [88-135]. Some show an even higher priority, maybe due to priority propagation. > I booted a 2 thread vm with xfce and decided to click around. Spawned > firefox, opened a file manager (Thunar) and from there I opened a > movie to play with mpv. As root I spawned make -j 2 buildkernel. it > was not particularly good :) > > I found that mpv spawns a bunch of threads, most notably 2 distinct > threads for audio and video output. The one for video got a priority > of 175, while the rest had either 88 or 89 -- the lowest for > timesharing not considered interactive [note lower is considered > better]. Presumably all of the video decoding was done in software, since you're running in a VM? On my desktop, mpv does not consume much CPU and is entirely interactive. Your test suggests that you expect ULE to prioritize a CPU hog, which doesn't seem realistic absent some scheduling hints from the user or the program itself. Problem 2 is the opposite problem: timesharing CPU hogs are allowed to starve other timesharing threads. > At the same time the file manager who was left in the background kept > doing evil syscall usage, which as a result bouncing between a regular > timesharing priority and one which made it "interactive", even though > the program was not touched for minutes. > > Or to put it differently, the scheduler failed to recognize that mpv > is the program to prioritize, all while thinking the background time > waster is the thing to look after (so to speak). > > This brings us to fixing problem 2: currently, due to the existence of > said problem, the interactivity scoring woes are less acute -- the > venerable make -j example is struggling to get CPU time, as a result > messing with real interactive programs to a lesser extent. If that > gets fixed, we are in a different boat altogether. > > I don't see a clean solution. > > Right now I'm toying with the idea of either: > 1. having programs explicitly tell the kernel they are interactive I don't see how this can work. It's not just traditional "interactive" programs that benefit from this scoring, it applies also to network servers and other programs which spend most of their time sleeping but want to handle requests with low latency. Such an interface would also let any program request soft realtime scheduling without giving up the ability to monopolize CPU time, which goes against ULE's fairness goals. > 2. adding a scheduler hook to /dev/dsp -- the observation is that if a > program is producing sound it probably should get some cpu time in a > timely manner. this would cover audio/video players and web browsers, On my system at least firefox doesn't open /dev/dsp, it sends audio streams to pulseaudio. > but would not cover other programs (say a pdf reader). it may be it is > good enough though I think some more thorough analysis, using tools like schedgraph or KUtrace[1], is needed to characterize the problems you are reporting with interactivity scoring. It's also not clear how any of this would address the problem that started this thread, wherein two competing timesharing (i.e., non-interactive) workloads get uneven amounts of CPU time. There is absolutely room for improvement in ULE's scheduling decisions. It seems to be common practice to tune various ULE parameters to get better interactive performance, but in general I see no analysis explaining /why/ exactly they help and what goes wrong with the default parameter values in specific workloads. schedgraph is a very useful tool for this sort of thing. Such tools also required to rule out bugs in ULE itself, when looking at abnormal scheduling behaviour. Last year some scheduler races[2] were fixed that apparently hurt system performance on EPYC quite a bit. I was told privately that applying those patches to 13.1 improved IPSec throughput by ~25% on EPYC, and I wouldn't be surprised if there are more improvements to be had which don't involve modifying core heuristics of the scheduler. Either way this requires deeper analysis of ULE's micro-level behaviour; I don't think "interactivity scoring is bogus" is a useful starting point. [1] https://github.com/dicksites/KUtrace Dick Sites' tracing framework, ported to FreeBSD. I have some WIP to integrate parts of it into the base system and ports tree but haven't yet had a good chunk of time to devote to it. I hope to get it done soon. Please let me know if you'd like to try it. [2] There were several more, but: https://cgit.freebsd.org/src/commit/?id=6d3f74a14a83b867c273c6be2599da182a9b9ec7 https://cgit.freebsd.org/src/commit/?id=03f868b163ad46d6f7cb03dc46fb83ca01fb8f69 are the ones that affected ULE specifically. From nobody Thu Mar 30 22:16:02 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pnd6Y3jHdz42QVl for ; Thu, 30 Mar 2023 22:16:05 +0000 (UTC) (envelope-from mandree@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pnd6Y3Hv4z43Z2 for ; Thu, 30 Mar 2023 22:16:05 +0000 (UTC) (envelope-from mandree@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1680214565; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NIzFsZ6tctXPj1TubZX+CYrmVqOISPN2qXtIaMpZPX4=; b=faRDiSAVMqnUlWgT6t7mhxhUBGLrsD9rhMd6AtlaKg53pFJaSJhFx9GWtd1g8pB0qVMvIP uPHSH0rKEmItSVC/XBe6Rxo2jM10cpSKWJiHLi3JjQRmi0P6BvpBPH63g0y9X6i07qKYj8 vyaixdX6ZFNjsRK2eeotNtuVmNPGq5LbWhuL5VJ4mepNZ6Ijczah291bddbukgygsbx9Tw p2SKqFUFLLdjajYBWwZrQ7GcctmpMoxZi0OLZXEmUUI4k6J8qDk1BiWhNCxUqnT508Zfxy yyg86CVWH9j8X3IZBCHDqWJMq2PZP306DU8oYzO9S8BD8CDOuaIj5OnnpdiZwg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1680214565; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NIzFsZ6tctXPj1TubZX+CYrmVqOISPN2qXtIaMpZPX4=; b=eVs+3pK7oeT66qWNzifSLtt2QHkTk0aj2IlC9UuqIP9L/GkqrspwFaom2h+meBnK0nX0Z7 t1canXMR2mVBQJQiaPGx/+eY/iYs5ZNlW9Jo4wVmC8/QJETk/5BT7mLMptgsY3BqieDhC3 ELH9LYcq21i/fdh05L/cKYmgU94FiYHuGf3lqLHrhJ6MVk4P1ONFluk6PzarbrsD0u42/u 8v4u8ytOXuUZ8QSO94UqZaDuuKPiw2X5/ANesxJJS8rwZeorTl5iws2TiQv42WRU5oR0MI DDfU74JjUOoQMVUv8xMHn3InT6MJRbxA2CUSntZ59YT1KyFy1egnuiWAYMQ00g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1680214565; a=rsa-sha256; cv=none; b=bolurTp+sln2MQNDSHrBYoZPYs+Rg/jbDqwQGDVomhQRiFI5UVcWUY+ZRM/ksVdkA4L5a5 cuYRQlzm0A78e9pFxmKmnNK1bi5ELBRHbAMgKAJXQa6fvfOP47Kr0DDeVLuFvmXPMKFhMh NkGXghGR0l07S686ZzQbPepvwoZYcx4JTopv0wiHe3zShNMISQRpmGskBX2I8TRSk+pEkm aSBtdMzu0ieuVuOnPpwOb6gBvdPCSp9Ixf6YRaSOu/H6Z68kK1TuvE/DqIHM5oDyhElwAs GyfWmqmKP4RieYJ4s/mXJuYk4Jb00x4UoYxPafo1wNRScCg3IcWF4QJlv/1ghw== Received: from mandree.no-ip.org (p200300d0271ebd00de65dff1a874e1cc.dip0.t-ipconnect.de [IPv6:2003:d0:271e:bd00:de65:dff1:a874:e1cc]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: mandree/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Pnd6Y1SWvzlMm for ; Thu, 30 Mar 2023 22:16:05 +0000 (UTC) (envelope-from mandree@FreeBSD.org) Received: from [127.0.0.1] (localhost [127.0.0.1]) by ryzen.an3e.de (Postfix) with ESMTP id C3FCADD63B1 for ; Fri, 31 Mar 2023 00:16:02 +0200 (CEST) Message-ID: Date: Fri, 31 Mar 2023 00:16:02 +0200 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 Subject: Re: Periodic rant about SCHED_ULE Content-Language: en-US To: freebsd-hackers@freebsd.org References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> From: Matthias Andree Organization: FreeBSD.org In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ThisMailContainsUnwantedMimeParts: N Am 30.03.23 um 20:50 schrieb Mateusz Guzik: > Anyhow after sending the above e-mail an actual solution hit me: the X > server can tell the kernel what processes connect to it over the unix > socket, which again very well may be good enough. > > In the reports I got I found pulseaudio, this one may need to be > patched in a similar manner. Pulseaudio shouldn't need such stuff since it would either try to go into a real-time scheduling group (which I don't think SCHED_ULE would have), or do a (privileged) renice -11 to have considerable priority over non-privileged stuff. At least on Linux, that is. -- Matthias Andree FreeBSD ports committer From nobody Fri Mar 31 07:08:42 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PnrxP6f08z423PC for ; Fri, 31 Mar 2023 07:08:57 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PnrxP09n0z4DHf for ; Fri, 31 Mar 2023 07:08:56 +0000 (UTC) (envelope-from julian@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=softfail (mx1.freebsd.org: 204.109.63.16 is neither permitted nor denied by domain of julian@freebsd.org) smtp.mailfrom=julian@freebsd.org; dmarc=none Received: from [192.168.0.18] ([73.223.42.188]) (authenticated bits=0) by vps1.elischer.org (8.17.1/8.15.2) with ESMTPSA id 32V78mGt018779 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NOT); Fri, 31 Mar 2023 00:08:48 -0700 (PDT) (envelope-from julian@freebsd.org) X-Authentication-Warning: vps1.elischer.org: Host [73.223.42.188] claimed to be [192.168.0.18] Message-ID: <4b92fb09-7083-f3c4-d503-f34d57d0a6d4@freebsd.org> Date: Fri, 31 Mar 2023 00:08:42 -0700 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 Subject: Re: Periodic rant about SCHED_ULE To: George Mitchell , freebsd-hackers@freebsd.org References: Content-Language: en-US From: Julian Elischer In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-3.09 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.992]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; R_DKIM_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; DMARC_NA(0.00)[freebsd.org]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[julian]; ARC_NA(0.00)[]; ASN(0.00)[asn:36236, ipnet:204.109.60.0/22, country:US]; TO_DN_SOME(0.00)[]; TAGGED_RCPT(0.00)[freebsd]; MID_RHS_MATCH_FROM(0.00)[]; HAS_XAW(0.00)[]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4PnrxP09n0z4DHf X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On 3/26/23 11:24 AM, George Mitchell wrote: > On 3/25/23 23:42, Peter wrote: >> > > To oversimplify the discussion I've seen so far on this thread: > > It would be really nice if the schedulers were kernel loadable modules. > GSoC project? > > Whether SCHED_ULE or SCHED_4BSD should be the default scheduler in the > GENERIC kernel is a contentious discussion, but perhaps we need to have > that discussion.                                           -- George > When I abstracted out teh kernel scheduler ABI/API some time ago (25 years maybe?) I thought about making a loadable module but there were a lot of gotchas that stopped me. I can't even remember what they all were.  Maybe it might be more doable now. From nobody Fri Mar 31 07:28:53 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PnsP806Ckz424HS for ; Fri, 31 Mar 2023 07:29:32 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from mailgate.Leidinger.net (bastille.leidinger.net [89.238.82.207]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) client-signature ECDSA (P-256)) (Client CN "mailgate.leidinger.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PnsP75GgJz4Gx6 for ; Fri, 31 Mar 2023 07:29:31 +0000 (UTC) (envelope-from Alexander@leidinger.net) Authentication-Results: mx1.freebsd.org; none Received: from outgoing.leidinger.net (p5b165130.dip0.t-ipconnect.de [91.22.81.48]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-256) client-signature ECDSA (P-256)) (Client CN "outgoing.leidinger.net", Issuer "R3" (verified OK)) by mailgate.Leidinger.net (Postfix) with ESMTPSA id 98BFD23CA0 for ; Fri, 31 Mar 2023 09:29:26 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1680247766; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=DJrgeO33yXUImLeCSH71V356LZPHEO+dOsivWVSCZ6E=; b=sCzv/n6lBN+BjjfTjluCZlWBB6kh2yAej+r1/YZQKUDLR2dapc6qvdVN6nYdL164F4Y3i6 0/M/Q7qNVLbRxscVVzVYnaA1l5xx6wl5uK83EYZK2cm0Rta7JaiHv8o4Vf+v1oNWwUmhVu JtYbtH7yHlwvtuBmp02jMAzhGb3nmGUrIZhe/Y2jZ5i8f8Ac1A65DVkER4PcSiR/1GYhjN f0uTumIPP0bgJ0jRBxhUZdfW4CEAS/hhA+PKqo82452eK1Jm6QpzkJs0loQm7rjYfP68Se 9exS7HtaII9XzxSOdXE2jone+NpkDWNfWYPnwIne6YPy0FnaAnKGzxm9608ETg== Received: from webmail.leidinger.net (localhost [127.0.0.1]) by outgoing.leidinger.net (Postfix) with ESMTP id E39C83E67 for ; Fri, 31 Mar 2023 09:28:53 +0200 (CEST) Received: from www (uid 80) (envelope-from Alexander@leidinger.net) id a1648 by webmail.leidinger.net (DragonFly Mail Agent v0.13+ on webmail.leidinger.net); Fri, 31 Mar 2023 09:28:53 +0200 Date: Fri, 31 Mar 2023 09:28:53 +0200 Message-ID: <20230331092853.Horde.OjdeEhUbQnY_fAcTedXJHY5@webmail.leidinger.net> From: Alexander Leidinger To: Mateusz Guzik Cc: freebsd-hackers@freebsd.org Subject: Re: Periodic rant about SCHED_ULE References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> In-Reply-To: Accept-Language: de,en Content-Type: multipart/signed; boundary="=_l8AL_ieHTE-W7Y1C0XG6ITV"; protocol="application/pgp-signature"; micalg=pgp-sha256 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 X-Rspamd-Queue-Id: 4PnsP75GgJz4Gx6 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:34240, ipnet:89.238.64.0/18, country:DE] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N This message is in MIME format and has been PGP signed. --=_l8AL_ieHTE-W7Y1C0XG6ITV Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Quoting Mateusz Guzik (from Thu, 30 Mar 2023=20=20 17:36:54=20+0200): > Right now I'm toying with the idea of either: > 1. having programs explicitly tell the kernel they are interactive There is no POSIX interface to do this. I'm not aware of a widespread=20=20 use=20of private interfaces in other unix-like operating systems to do=20= =20 this. I=20consider modifying all interactive programs with FreeBSD specific=20=20 code=20to do so a bad idea. I consider modifying the X server with FreeBSD specific code to do so=20=20 on=20behalf of programs using it a bad idea too (what about interactive=20= =20 non-X-using=20programs I start in an xterm or from the vidconsole?). > 2. adding a scheduler hook to /dev/dsp -- the observation is that if a > program is producing sound it probably should get some cpu time in a > timely manner. this would cover audio/video players and web browsers, > but would not cover other programs (say a pdf reader). it may be it is > good enough though What if I don't have an audio device on a system but run interactive=20=20 programs=20on it? You told it yourself, it favourites a specific class=20= =20 of=20programs. Please think this to the end, you are telling you only=20=20 want=20to priorize a subset of interactive processes and leave others=20=20 alone.=20If you are honest to yourself, you need to say this is a hack,=20= =20 not=20a solution. My suggestion is to first fix the bugs we/you are able to fix (number=20=20 2),=20and then to measure again with a bigger sample size. Small=20=20 evolution=20and re-observation instead of a big bang fix of everything=20= =20 based=20upon assumptions of what impact the smaller fixes may have on=20=20 our=20big population of use cases. Maybe someone has an idea how to factor out lock contention in the=20=20 mean=20time, and we can have a look if this improves something for=20=20 someone=20by providing it as a fix (no matter if gated by a sysctl or=20=20 not)=20and we can re-measure again. Bye, Alexander. --=20 http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF --=_l8AL_ieHTE-W7Y1C0XG6ITV Content-Type: application/pgp-signature Content-Description: Digitale PGP-Signatur Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIzBAABCAAdFiEER9UlYXp1PSd08nWXEg2wmwP42IYFAmQmi7QACgkQEg2wmwP4 2IZ3Yg/9FeOTevrIe8RaVX0s7/+su23CYPjpUhag2atYsQlpqfN8nAxWtr6rKFY3 OEf0taMPygCtCZI66R46gI7HeUNL0E+4ZS9qqZglqvB2vet/zweXht7eQqvW1cAD PglU86lt/CBLgHbL0MLoebc7gjwE8xbcPob+fxOz+8iPTzbCbFabwz005as76ZD8 bokH2TKSm6bKu3Y6y5kqmoTAYJ63UheEGxZ6+GPDWdkxhTvImLD/paQ5pZh4kYvo 2zqDW872ATCvmtNEpEibrXSk07vavWAWVCuLIkBCLOpCsBgAuHGG3UWmjAjOLJiM bNTbqLwuLWY+K/QJ4lh7QIxLfdC8aP/ndoIh3Bw5FF+Q+4Eq+4ki7Q4hdmySTCgJ 11iJAB6i+WXmJfBGNCMLQF0VHyWh3QrVmetfmTEVZzLYSyMESrBgo1tBCcv3BgZc WIhMTMI8v+1OPUahrs63d6mZPIAlC4ZNxqrfmzjyEySGMDApp2tVu6kLDVEItSrW aqVLcQOT/aocvu4xAWnPr0bWyFhsu9oidpgaWgb026/juUBHHAX5Dn6ji5n9uZBR Zqh108VeryLz97aN+ar+Fq7a7xy/gylSo8HikPXpSI4c55964PXVnfajDi4paDAb 8AvF4uHlwqv+/a16JDo5cdUTbMsRkfhEAEuMtIVdPAp1O8jM+mI= =0XFB -----END PGP SIGNATURE----- --=_l8AL_ieHTE-W7Y1C0XG6ITV-- From nobody Fri Mar 31 12:57:51 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pp0h42pxyz42ScZ for ; Fri, 31 Mar 2023 12:57:56 +0000 (UTC) (envelope-from junchoon@dec.sakura.ne.jp) Received: from www121.sakura.ne.jp (www121.sakura.ne.jp [153.125.133.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pp0h33R00z3tPq for ; Fri, 31 Mar 2023 12:57:55 +0000 (UTC) (envelope-from junchoon@dec.sakura.ne.jp) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of junchoon@dec.sakura.ne.jp has no SPF policy when checking 153.125.133.21) smtp.mailfrom=junchoon@dec.sakura.ne.jp; dmarc=none Received: from kalamity.joker.local (123-1-88-210.area1b.commufa.jp [123.1.88.210]) (authenticated bits=0) by www121.sakura.ne.jp (8.16.1/8.16.1/[SAKURA-WEB]/20201212) with ESMTPA id 32VCvpKQ036267 for ; Fri, 31 Mar 2023 21:57:52 +0900 (JST) (envelope-from junchoon@dec.sakura.ne.jp) Date: Fri, 31 Mar 2023 21:57:51 +0900 From: Tomoaki AOKI To: freebsd-hackers@freebsd.org Subject: Re: Periodic rant about SCHED_ULE Message-Id: <20230331215751.166a294f7382c85b545f53a2@dec.sakura.ne.jp> In-Reply-To: References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> Organization: Junchoon corps X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.33; amd64-portbld-freebsd13.2) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-0.58 / 15.00]; URI_HIDDEN_PATH(1.00)[https://people.freebsd.org/~mjg/.junk/ule-poc-hacks-dont-use.diff]; AUTH_NA(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.995]; NEURAL_HAM_LONG(-0.99)[-0.989]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:7684, ipnet:153.125.128.0/18, country:JP]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; R_SPF_NA(0.00)[no SPF record]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[sakura.ne.jp]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_NONE(0.00)[]; HAS_ORG_HEADER(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4Pp0h33R00z3tPq X-Spamd-Bar: / X-ThisMailContainsUnwantedMimeParts: N On Mon, 27 Mar 2023 16:47:04 +0200 Mateusz Guzik wrote: > On 3/27/23, Mateusz Guzik wrote: > > On 3/25/23, Mateusz Guzik wrote: > >> On 3/23/23, Mateusz Guzik wrote: > >>> On 3/22/23, Mateusz Guzik wrote: > >>>> On 3/22/23, Steve Kargl wrote: > >>>>> On Wed, Mar 22, 2023 at 07:31:57PM +0100, Matthias Andree wrote: > >>>>>> (snip) > > > > I repeat the setup: 8 cores, 8 processes doing cpu-bound stuff while > > niced to 20 vs make -j buildkernel > > > > I had a little more look here, slapped in some hacks as a POC and got > > an improvement from 67 minutes above to 21. > > > > Hacks are: > > 1. limit hog timeslice to 1 tick so that is more eager to bail > > 2. always preempt if pri < cpri > > > > So far I can confidently state the general problem: ULE penalizes > > non-cpu hogs for blocking (even if it is not their fault, so to speak) > > and that bumps their prio past preemption threshold, at which point > > they can't preempt said hogs (despite hogs having a higher priority). > > At the same time hogs use their full time slices, while non-hogs get > > off cpu very early and have to wait a long time to get back on, at > > least in part due to inability to preempt said hogs. > > > > As I mentioned elsewhere in the thread, interactivity scoring takes > > "voluntary off cpu time" into account. As literally anything but > > getting preempted counts as "voluntary sleep", workers get shafted for > > going off cpu while waiting on locks in the kernel. > > > > If I/O needs to happen and the thread waits for the result, it most > > likely does it early in its timeslice and once it's all ready it waits > > for background hogs to get off cpu -- it can't preempt them. > > > > All that said: > > 1. "interactivity scoring" (see sched_interact_score) > > > > I don't know if it makes any sense to begin with. Even if it does, it > > counts stuff it should not by not differentiating between deliberately > > going off cpu (e.g., actual sleep) vs just waiting for a file being > > read. Imagine firefox reading a file from disk and being considered > > less interactive for it. > > > > I don't have a solution for this problem. I *suspect* the way to go > > would be to explicitly mark xorg/wayland/whatever as "interactive" and > > have it inherited by its offspring. At the same time it should not > > follow to stuff spawned in terminals. Not claiming this is perfect, > > but it does eliminate the guessing game. > > > > Even so, 4BSD does not have any mechanism of the sort and reportedly > > remains usable on a desktop just by providing some degree of fairness. > > > > Given that, I suspect the short term solution would whack said scoring > > altogether and focus on fairness (see below). > > > > 2. fairness > > > > As explained above doing any offcpu-time inducing work instantly > > shafts threads versus cpu hogs, even if said hogs are niced way above > > them. > > > > Here I *suspect* position to add in the runqueue should be related to > > how much slice was left when the thread went off cpu, while making > > sure that hogs get to run eventually. Not that I have a nice way of > > implementing this -- maybe a separate queue for known hogs and picking > > them every n turns or similar. > > > > Aight, now that I had a sober look at the code I think I cracked the case. > > The runq mechanism used by both 4BSD and ULE provides 64(!) queues, > where the priority is divided by said number and that's how you know > in which queue to land the thread. > > When deciding what to run, 4BSD uses runq_choose which iterates all > queues from the beginning. This means threads of lower priority keep > executing before the rest. In particular cpu hog lands with a high > priority, looking worse than make -j 8 buildkernel and only running > when there is nothing else ready to get the cpu. While this may sound > decent, it is bad -- in principle a steady stream of lower priority > threads can starve the hogs indefinitely. > > The problem was recognized when writing ULE, but improperly fixed -- > it ends up distributing all threads within given priority range across > the queues and then performing a lookup in a given queue. Here the > problem is that while technically everyone does get a chance to run, > the threads not using full slices are hosed for the time period as > they wait for the hog *a lot*. > > A hack patch to induce the bogus-but-better 4BSD behavior of draining > all runqs before running higher prio threads drops down build time to > ~9 minutes, which is shorter than 4BSD. > > However, the right fix would achieve that *without* introducing > starvation potential. > > I also note the runqs are a massive waste of memory and computing > power. I'm going to have to sleep on what to do here. > > For interested here is the hackery: > https://people.freebsd.org/~mjg/.junk/ule-poc-hacks-dont-use.diff > > sysctl kern.sched.slice_nice=0 > sysctl kern.sched.preempt_thresh=400 # arbitrary number higher than any prio > > -- > Mateusz Guzik Thanks for the patch. Applied on top of main, amd64 at commit 9d33a9d96f5a2cd88d0955b5b56ef5058b1706c1, setup 2 sysctls as you mentioned and tested as below *Play flac files by multimedia/audacious via audio/virtual_oss *Running www/firefox (not touched while testing) *Forcibly build lang/rust *Play games/aisleriot at the same time. games/aisleriot runs slower than the situation lang/rust is not in build, but didn't "freeze" and audacious normally played next music on playlist, even on lang/rust is building codes written in rust. This is GREAT advance! Without the patch, compiling rust codes eats up almost 100% of ALL cores, and games/aisleriot often FREEZES SEVERAL MINUTES, and multimedia/audacious needs to wait for, at worst, next music for FEW MINUTES. (Once playback starts, the music is played normally until it ends.) But unfortunately, the patch cannot be applied to stable/13, as some prerequisite commits are not MFC'ed. Missing commits are at least as below. There should be more, as I gave up further tracking and haven't actually merged them to test. commit 954cffe95de1b9d70ed804daa45b7921f0f5c9da [1] ule: Simplistic time-sharing for interrupt threads. commit fea89a2804ad89f5342268a8546a3f9b515b5e6c [2] Add sched_ithread_prio to set the base priority of an interrupt thread. commit 85b46073242d4666e1c9037d52220422449f9584 [3] Deduplicate bus_dma bounce code. [1] https://cgit.freebsd.org/src/commit/?id=954cffe95de1b9d70ed804daa45b7921f0f5c9da [2] https://cgit.freebsd.org/src/commit/?id=fea89a2804ad89f5342268a8546a3f9b515b5e6c [3] https://cgit.freebsd.org/src/commit/?id=85b46073242d4666e1c9037d52220422449f9584 -- Tomoaki AOKI From nobody Fri Mar 31 17:33:58 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pp6pd2jnjz432SJ for ; Fri, 31 Mar 2023 17:34:01 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oi1-x22c.google.com (mail-oi1-x22c.google.com [IPv6:2607:f8b0:4864:20::22c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pp6pd0Z5Yz4VfB for ; Fri, 31 Mar 2023 17:34:01 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oi1-x22c.google.com with SMTP id bj20so17211938oib.3 for ; Fri, 31 Mar 2023 10:34:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680284039; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=k4iaWh+7RFylYYyfRaCaTX6xuyN7FpX3MlLslkdzRzs=; b=LDPZZHNnx/snlX+F6ZS1ESEeckVadCu4V9D6Hj4DeJd+iZDkDRZLg2eEcNh0aR5MDj om9NOn2EzPGKV8gZlqq8kKNU6OSkqs3xwjvQLG9SnD6wsn8FG9XwVtCAXyWiocTt8xV2 CBSJC54k1EozkcMP8JBOfpqYv7YWg4TAs0Nrk+FAY2bzf8kYG2Zsqz5D1XXM2YWKeldG T5Ot61j2Pxh/hLlOxd4edPavTIYjL+jHL0wHgZhkXvPAjpxM+0Xo9DLxnKi6IdnApWjH nP5AtW1+j/kiNsPHGpI3uU8r5hK+RMeXlomB2DiokmoVmxvNahWezWP5iAIxeClfVQ1F dKrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680284039; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=k4iaWh+7RFylYYyfRaCaTX6xuyN7FpX3MlLslkdzRzs=; b=ACB//o/c1A57mOuoOUP6imZg5vc5SVqjiyYVo43CRco4onbxDAerVLLfcBbp/vp6yg gadLTEkbrhTguBa6SZl+C4q/zdtz3vwyBQ9tJfGC/gsThP4xncOhb4RmJsMrKwDQ61fe e7rWvLicgibJOJjwXhj3LY7QpshPtXdAruTE5/8k35U4FmW2GYNeQ+HwRVjJwXkvxr3s paDMvu/qJKjThyvLubIkhcYySi2cxK0e2bqw2GpFAjvkOqP3OtCCaGhxQVo74uAvq8ue c9eIqrXdRAeQtHbWMzgzdxs0VW4YkpiMWrXfzCxbTl3ba8kQ70O/PewqHuAwmcQKcGJK EvGQ== X-Gm-Message-State: AO0yUKW2s+IXR5i3H6Yt80uz3kSc0cOEN4qhJaXUHSkU3EDNunZFxys4 tI+Mh9+tdrLrcLTgOTpwffPs596UXvVNu/+Xw9R7gnVk X-Google-Smtp-Source: AK7set+kUYTzDmv1codbqI6JrftwnWEqbOdE0wGaCvfXAqyEVvQbIFpqjdUL/GGfOwZ/90eVBkui/CYiopxABMynyT0= X-Received: by 2002:a54:4d84:0:b0:387:251b:71fa with SMTP id y4-20020a544d84000000b00387251b71famr8862687oix.4.1680284039554; Fri, 31 Mar 2023 10:33:59 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:ac9:7598:0:b0:49c:b071:b1e3 with HTTP; Fri, 31 Mar 2023 10:33:58 -0700 (PDT) In-Reply-To: <20230331215751.166a294f7382c85b545f53a2@dec.sakura.ne.jp> References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> <20230331215751.166a294f7382c85b545f53a2@dec.sakura.ne.jp> From: Mateusz Guzik Date: Fri, 31 Mar 2023 19:33:58 +0200 Message-ID: Subject: Re: Periodic rant about SCHED_ULE To: Tomoaki AOKI Cc: freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4Pp6pd0Z5Yz4VfB X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On 3/31/23, Tomoaki AOKI wrote: > On Mon, 27 Mar 2023 16:47:04 +0200 > Mateusz Guzik wrote: > >> On 3/27/23, Mateusz Guzik wrote: >> > On 3/25/23, Mateusz Guzik wrote: >> >> On 3/23/23, Mateusz Guzik wrote: >> >>> On 3/22/23, Mateusz Guzik wrote: >> >>>> On 3/22/23, Steve Kargl wrote: >> >>>>> On Wed, Mar 22, 2023 at 07:31:57PM +0100, Matthias Andree wrote: >> >>>>>> > > (snip) > >> > >> > I repeat the setup: 8 cores, 8 processes doing cpu-bound stuff while >> > niced to 20 vs make -j buildkernel >> > >> > I had a little more look here, slapped in some hacks as a POC and got >> > an improvement from 67 minutes above to 21. >> > >> > Hacks are: >> > 1. limit hog timeslice to 1 tick so that is more eager to bail >> > 2. always preempt if pri < cpri >> > >> > So far I can confidently state the general problem: ULE penalizes >> > non-cpu hogs for blocking (even if it is not their fault, so to speak) >> > and that bumps their prio past preemption threshold, at which point >> > they can't preempt said hogs (despite hogs having a higher priority). >> > At the same time hogs use their full time slices, while non-hogs get >> > off cpu very early and have to wait a long time to get back on, at >> > least in part due to inability to preempt said hogs. >> > >> > As I mentioned elsewhere in the thread, interactivity scoring takes >> > "voluntary off cpu time" into account. As literally anything but >> > getting preempted counts as "voluntary sleep", workers get shafted for >> > going off cpu while waiting on locks in the kernel. >> > >> > If I/O needs to happen and the thread waits for the result, it most >> > likely does it early in its timeslice and once it's all ready it waits >> > for background hogs to get off cpu -- it can't preempt them. >> > >> > All that said: >> > 1. "interactivity scoring" (see sched_interact_score) >> > >> > I don't know if it makes any sense to begin with. Even if it does, it >> > counts stuff it should not by not differentiating between deliberately >> > going off cpu (e.g., actual sleep) vs just waiting for a file being >> > read. Imagine firefox reading a file from disk and being considered >> > less interactive for it. >> > >> > I don't have a solution for this problem. I *suspect* the way to go >> > would be to explicitly mark xorg/wayland/whatever as "interactive" and >> > have it inherited by its offspring. At the same time it should not >> > follow to stuff spawned in terminals. Not claiming this is perfect, >> > but it does eliminate the guessing game. >> > >> > Even so, 4BSD does not have any mechanism of the sort and reportedly >> > remains usable on a desktop just by providing some degree of fairness. >> > >> > Given that, I suspect the short term solution would whack said scoring >> > altogether and focus on fairness (see below). >> > >> > 2. fairness >> > >> > As explained above doing any offcpu-time inducing work instantly >> > shafts threads versus cpu hogs, even if said hogs are niced way above >> > them. >> > >> > Here I *suspect* position to add in the runqueue should be related to >> > how much slice was left when the thread went off cpu, while making >> > sure that hogs get to run eventually. Not that I have a nice way of >> > implementing this -- maybe a separate queue for known hogs and picking >> > them every n turns or similar. >> > >> >> Aight, now that I had a sober look at the code I think I cracked the >> case. >> >> The runq mechanism used by both 4BSD and ULE provides 64(!) queues, >> where the priority is divided by said number and that's how you know >> in which queue to land the thread. >> >> When deciding what to run, 4BSD uses runq_choose which iterates all >> queues from the beginning. This means threads of lower priority keep >> executing before the rest. In particular cpu hog lands with a high >> priority, looking worse than make -j 8 buildkernel and only running >> when there is nothing else ready to get the cpu. While this may sound >> decent, it is bad -- in principle a steady stream of lower priority >> threads can starve the hogs indefinitely. >> >> The problem was recognized when writing ULE, but improperly fixed -- >> it ends up distributing all threads within given priority range across >> the queues and then performing a lookup in a given queue. Here the >> problem is that while technically everyone does get a chance to run, >> the threads not using full slices are hosed for the time period as >> they wait for the hog *a lot*. >> >> A hack patch to induce the bogus-but-better 4BSD behavior of draining >> all runqs before running higher prio threads drops down build time to >> ~9 minutes, which is shorter than 4BSD. >> >> However, the right fix would achieve that *without* introducing >> starvation potential. >> >> I also note the runqs are a massive waste of memory and computing >> power. I'm going to have to sleep on what to do here. >> >> For interested here is the hackery: >> https://people.freebsd.org/~mjg/.junk/ule-poc-hacks-dont-use.diff >> >> sysctl kern.sched.slice_nice=0 >> sysctl kern.sched.preempt_thresh=400 # arbitrary number higher than any >> prio >> >> -- >> Mateusz Guzik > > Thanks for the patch. > Applied on top of main, amd64 at commit > 9d33a9d96f5a2cd88d0955b5b56ef5058b1706c1, setup 2 sysctls as you > mentioned and tested as below > > *Play flac files by multimedia/audacious via audio/virtual_oss > *Running www/firefox (not touched while testing) > *Forcibly build lang/rust > *Play games/aisleriot > > at the same time. > games/aisleriot runs slower than the situation lang/rust is not in > build, but didn't "freeze" and audacious normally played next music on > playlist, even on lang/rust is building codes written in rust. > > This is GREAT advance! > Without the patch, compiling rust codes eats up almost 100% of ALL > cores, and games/aisleriot often FREEZES SEVERAL MINUTES, and > multimedia/audacious needs to wait for, at worst, next music for FEW > MINUTES. (Once playback starts, the music is played normally until it > ends.) > thanks for testing the patch is just the bare minimum to prove what the problem is. due to that it runs into problems a real fix will not however, even with the real fix i can't promise optimal behavior > But unfortunately, the patch cannot be applied to stable/13, as some > prerequisite commits are not MFC'ed. there are no prerequisites for merging a proper fix -- Mateusz Guzik From nobody Fri Mar 31 18:09:19 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pp7p72dSLz435Gt for ; Fri, 31 Mar 2023 18:18:39 +0000 (UTC) (envelope-from li-fbsd@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pp7p60G0Tz4bYy for ; Fri, 31 Mar 2023 18:18:37 +0000 (UTC) (envelope-from li-fbsd@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of li-fbsd@citylink.dinoex.sub.org designates 2a0b:f840::12 as permitted sender) smtp.mailfrom=li-fbsd@citylink.dinoex.sub.org; dmarc=none Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.17.1/8.17.1) with ESMTPS id 32VII5c6033642 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Fri, 31 Mar 2023 20:18:06 +0200 (CEST) (envelope-from li-fbsd@citylink.dinoex.sub.org) X-MDaemon-Deliver-To: Received: (from uucp@localhost) by uucp.dinoex.org (8.17.1/8.17.1/Submit) with UUCP id 32VII5os033641 for freebsd-hackers@freebsd.org; Fri, 31 Mar 2023 20:18:05 +0200 (CEST) (envelope-from li-fbsd@citylink.dinoex.sub.org) Received: from admn.intra.daemon.contact (localhost [127.0.0.1]) by admn.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32VI9VTw041675 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Fri, 31 Mar 2023 20:09:31 +0200 (CEST) (envelope-from li-fbsd@citylink.dinoex.sub.org) Received: from intra.daemon.contact (news@localhost) by admn.intra.daemon.contact (8.17.1/8.17.1/Submit) with NNTP id 32VI9J2U041595 for freebsd-hackers@freebsd.org; Fri, 31 Mar 2023 20:09:19 +0200 (CEST) (envelope-from li-fbsd@citylink.dinoex.sub.org) X-Authentication-Warning: admn.intra.daemon.contact: news set sender to li-fbsd@citylink.dinoex.sub.org using -f From: "Peter 'PMc' Much" X-Newsgroups: m2n.fbsd.hackers Subject: Re: Periodic rant about SCHED_ULE Date: Fri, 31 Mar 2023 18:09:19 -0000 (UTC) Message-ID: References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> Injection-Date: Fri, 31 Mar 2023 18:09:19 -0000 (UTC) Injection-Info: admn.intra.daemon.contact; logging-data="5437"; mail-complaints-to="usenet@citylink.dinoex.sub.org" User-Agent: slrn/1.0.3 (FreeBSD) To: freebsd-hackers@freebsd.org X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Fri, 31 Mar 2023 20:18:08 +0200 (CEST) X-Spamd-Result: default: False [-2.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.990]; FORGED_SENDER(0.30)[pmc@citylink.dinoex.sub.org,li-fbsd@citylink.dinoex.sub.org]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; TO_DN_NONE(0.00)[]; R_DKIM_NA(0.00)[]; DMARC_NA(0.00)[sub.org]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_COUNT_THREE(0.00)[4]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE]; TO_MATCH_ENVRCPT_ALL(0.00)[]; HAS_XAW(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; FROM_NEQ_ENVFROM(0.00)[pmc@citylink.dinoex.sub.org,li-fbsd@citylink.dinoex.sub.org] X-Rspamd-Queue-Id: 4Pp7p60G0Tz4bYy X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org On 2023-03-30, Matthias Andree wrote: > Am 30.03.23 um 20:50 schrieb Mateusz Guzik: > >> Anyhow after sending the above e-mail an actual solution hit me: the X >> server can tell the kernel what processes connect to it over the unix >> socket, which again very well may be good enough. >> >> In the reports I got I found pulseaudio, this one may need to be >> patched in a similar manner. > > Pulseaudio shouldn't need such stuff since it would either try to go > into a real-time scheduling group (which I don't think SCHED_ULE would > have), or do a (privileged) renice -11 to have considerable priority > over non-privileged stuff. At least on Linux, that is. > >From what I get, it seems most people with professional audio are using JACK, and JACK runs in rtprio by default: UID PID PPID C PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 1100 1404 1403 2 -51 0 153508 71816 sigwait I; Fri, 31 Mar 2023 18:19:00 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-ot1-x330.google.com (mail-ot1-x330.google.com [IPv6:2607:f8b0:4864:20::330]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pp7pX1JtPz4c9D for ; Fri, 31 Mar 2023 18:19:00 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ot1-x330.google.com with SMTP id o25-20020a9d4119000000b006a11eb19f8eso11112182ote.5 for ; Fri, 31 Mar 2023 11:19:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680286738; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=IsJbrIaMQp3ztIUTNL/PyPWGElVD3qzWSJGjIXGXXtA=; b=JZbCsWwkEYSJfSa5NRxJfRyhHSseRHq1p3fRrOaYdbVaQ014l/PAsDdmAu3HmkiwZ/ EHp5E76A09XyNM+UMl+vUGMB9I/r1sptMSv9mZ45RszED9pXDg76yTj9kEj+/ikElkah nDwYSUlyQxEYN8wRtANbAowtoPgeVwueBfZcLFlXQGWySkA7swN63I6OWvi0blEPgRzB mVnNjOUQxsux22jDfwp6pVw+/GtcOfdwEIo2Cikjny/7m6cGLL2cD+Uqo6W88fV5yFPU RQ0WkCFitAceFdzPk0UxZH2ba5WVZhnVoGp1EQpB3A/yrwPHaAT5cC71TcfOxL8Iw4G2 36BA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680286738; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=IsJbrIaMQp3ztIUTNL/PyPWGElVD3qzWSJGjIXGXXtA=; b=zz+czpkfu33teLKdxtLKhUpOgJjsjK2tn8cVfsVcl2sobxyWnFejdaXN0rYLzTzp9e 6gIwp7DmT8C9LK8MOQCj3ZyEVB9mnsWEaAlsYLZ0dZaJi+/r1dPJBv921zT+IIyb9Mx6 GnjRuKBULebopyJTdePQNEaBtczQm6ImO+s5/e5vaNRJf48/Nz9M/h6oeKeKRstJzELU zUM0btgUCntxRGtc2QdrKI4L6cIf2cc3WfLvd3ZIugMlmg7N/Q3yWiihRw9COSv525rS vucva3eGOnnlJUU/EnfyZxwYbWwV70ki7eWzOYmh1mfNj3i2OiR4ER05+pCtG8Lrirdv rb8A== X-Gm-Message-State: AAQBX9cGs/QdxLgnUcIeymfX4mAXuGXkIwImiHik4KJJsHAkGY23NVb5 Xevgt+lBWzXWilYfwbb3NQzg2Ry0o8Bf2C9h07Yv5+4q X-Google-Smtp-Source: AKy350Zb67WE7/PCWIHvOWXSAiZrSubz0VGeelTfepoo/i+gS86ZCxR8eUQxKwjmdVgPk8JuqFBLUpsn/MgWg6xyN+Q= X-Received: by 2002:a05:6830:39d7:b0:6a1:3fd6:5a0b with SMTP id bt23-20020a05683039d700b006a13fd65a0bmr6152708otb.2.1680286738291; Fri, 31 Mar 2023 11:18:58 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:ac9:7598:0:b0:49c:b071:b1e3 with HTTP; Fri, 31 Mar 2023 11:18:57 -0700 (PDT) In-Reply-To: <20230331092853.Horde.OjdeEhUbQnY_fAcTedXJHY5@webmail.leidinger.net> References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> <20230331092853.Horde.OjdeEhUbQnY_fAcTedXJHY5@webmail.leidinger.net> From: Mateusz Guzik Date: Fri, 31 Mar 2023 20:18:57 +0200 Message-ID: Subject: Re: Periodic rant about SCHED_ULE To: Alexander Leidinger Cc: freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4Pp7pX1JtPz4c9D X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On 3/31/23, Alexander Leidinger wrote: > Quoting Mateusz Guzik (from Thu, 30 Mar 2023 > 17:36:54 +0200): > >> Right now I'm toying with the idea of either: >> 1. having programs explicitly tell the kernel they are interactive > > There is no POSIX interface to do this. I'm not aware of a widespread > use of private interfaces in other unix-like operating systems to do > this. > I consider modifying all interactive programs with FreeBSD specific > code to do so a bad idea. > I consider modifying the X server with FreeBSD specific code to do so > on behalf of programs using it a bad idea too (what about interactive > non-X-using programs I start in an xterm or from the vidconsole?). > Of course there are corner cases like that. I'm all ears for a solution which does not have any. I once more stress how the current code fundamentally fails to asses real programs, mpv being an example -- the thread doing video output was considered least important. The current method of "was off cpu there is interactive" literally does *NOT* work. >> 2. adding a scheduler hook to /dev/dsp -- the observation is that if a >> program is producing sound it probably should get some cpu time in a >> timely manner. this would cover audio/video players and web browsers, >> but would not cover other programs (say a pdf reader). it may be it is >> good enough though > > What if I don't have an audio device on a system but run interactive > programs on it? if they are using X (or wayland), they are marked as such > You told it yourself, it favourites a specific class > of programs. Please think this to the end, you are telling you only > want to priorize a subset of interactive processes and leave others > alone. If you are honest to yourself, you need to say this is a hack, > not a solution. You must have missed the part where I wrote: > I don't see a clean solution. Anyhow, if you can show me a real case of an interactive program you are running in a terminal, which does not use X nor produce sound *and* suffers from interactivity issues, I'm going to look into it closer. Do you expect vim to suffer video tearing? Anyhow even for such a contrived case one can try genuinely devise something. Did I mention the *current* method actively shafts real interactive programs? > > My suggestion is to first fix the bugs we/you are able to fix (number > 2), and then to measure again with a bigger sample size. Small > evolution and re-observation instead of a big bang fix of everything > based upon assumptions of what impact the smaller fixes may have on > our big population of use cases. > An almost bare-minimum fix immediately makes the interactive scoring issues worse, as I tried to explain. This can be damage-controlled in a reasonably easy manner and it may be the result will be tolerable enough for the foreseeable future. > Maybe someone has an idea how to factor out lock contention in the > mean time, and we can have a look if this improves something for > someone by providing it as a fix (no matter if gated by a sysctl or > not) and we can re-measure again. > No idea is needed. Just not have going-off-cpu-because-of-kernel uses have the same flag as going-off-cpu-intentionally. Even then this is not worth pursuing as the fundamental method involving this is bogus. -- Mateusz Guzik From nobody Fri Mar 31 18:41:41 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pp8Jl3G1bz436K1 for ; Fri, 31 Mar 2023 18:41:43 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oi1-x232.google.com (mail-oi1-x232.google.com [IPv6:2607:f8b0:4864:20::232]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pp8Jl17NDz3CBW; Fri, 31 Mar 2023 18:41:43 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oi1-x232.google.com with SMTP id y184so17367550oiy.8; Fri, 31 Mar 2023 11:41:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680288102; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=e6m3VFvjn6U5gb2bh0yNi6qdqU0yMAYZg8gjPGOh8GY=; b=HhyOtlogY1n+kze+7UcG+YuXPh1gIThrI/9nCqaAiAQdOpnbi+NcpV3J20nl03pDst XTgXBo2mMr8C7PXec9K2Kq/UqodZdmMQcr4ajtmLn2cD9eJVDHHNJ+fyp9sPZ5KYtsAh 5KPMzW3iR3FEF830cRamVPxzEclOyw6VdgL2ixJ6lFcq+fql+wFlL7PTluRdSMue0opG z0NJla4wF0X1XSZ3HwNN1yTiyGqGEmlLkEijxWMALMwx8M2EfXVn+C+yn0TobdKfyZxa QM42vtwd1SQDJDIewL78h5dR1hvmHngJ6lOJ8/bNOYpOMnYEfyD2twHTbUiAq6fBcuqY 4kUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680288102; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=e6m3VFvjn6U5gb2bh0yNi6qdqU0yMAYZg8gjPGOh8GY=; b=lW5+nCDsB+djA6A/cM1B/eiJNwcS8IOF24t1mw6koaXSG7sP/wEyl2RJbiPYNTjIZk 8lb5eUgOJXGOo6pM0j+jXvzfBgJz9CRcrTp9RFcBZjf3acwOh4uhYT2tImJ33Jof0Hjy PZpl8QnPmi1/thJT/47bUjc9gvPBMVUlWP9yuR3IpDbb6wzu+aN2LrQ5mWUMwHCA/7Fa T2Pm2gNo3qYI04Vc87LzICZT6kNN5K0qwok+a5vWKTj2N0aoxJdn38VN2mQ9sRRjN1bc 1qPTghqsgJfOPNMhbX4S3i6OZMeKcZgpn1j1b2iTpr9wU7wzVpHqJ5RLkLHeMVbUdCxb G7LQ== X-Gm-Message-State: AO0yUKXb1eMOIdyg9Q12rTASJtfmHXg4mU+fCVmf/l9hBWqmKML7mjpj 01t3/XSF/tNfluFU8lpRbZ79pBv3M5fyhnH2ypK8bhu8 X-Google-Smtp-Source: AK7set/UFo7D8TMqGVp9VdvNVncqxdy5J7kWovQMhS1O0avRcLoxwa640VJh+0B6BVOeTk6Kz29COxnNe3Oh1rbrAs8= X-Received: by 2002:aca:1b09:0:b0:383:f5df:9e85 with SMTP id b9-20020aca1b09000000b00383f5df9e85mr6612233oib.4.1680288101846; Fri, 31 Mar 2023 11:41:41 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:ac9:7598:0:b0:49c:b071:b1e3 with HTTP; Fri, 31 Mar 2023 11:41:41 -0700 (PDT) In-Reply-To: References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> From: Mateusz Guzik Date: Fri, 31 Mar 2023 20:41:41 +0200 Message-ID: Subject: Re: Periodic rant about SCHED_ULE To: Mark Johnston Cc: freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4Pp8Jl17NDz3CBW X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On 3/30/23, Mark Johnston wrote: > On Thu, Mar 30, 2023 at 05:36:54PM +0200, Mateusz Guzik wrote: >> I looked into it a little more, below you can find summary and steps >> forward. >> >> First a general statement: while ULE does have performance bugs, it >> has better basis than 4BSD to make scheduling decisions. Most notably >> it understands CPU topology, at least for cases which don't involve >> big.LITTLE. For any non-freak case where 4BSD performs better, it is a >> bug in ULE if this is for any reason other than a tradeoff which can >> be tweaked to line them up. Or more to the point, there should not be >> any legitimate reason to use 4BSD these days and modulo the bugs >> below, you are probably losing on performance for doing so. >> >> Bugs reported in this thread by others and confirmed by me: >> 1. failure to load-balance when having n CPUs and n + 1 workers -- the >> excess one stays on one the same CPU thread continuously penalizing >> the same victim. as a result total real time to execute a finite >> computation is longer than in the case of 4BSD >> 2. unfairness of nice -n 20 threads vs threads going frequently off >> CPU (e.g., due to I/O) -- after using only a fraction of the slice the >> victim has to wait for the cpu hog to use up its entire slice, rinse >> and repeat. This extends a 7+ minute buildkernel to over 67 minutes, >> not an issue on 4BSD >> >> I did not put almost any effort into investigating no 1. There is code >> which is supposed to rebalance load across CPUs, someone(tm) will have >> to sit through it -- for all I know the fix is trivial. >> >> Fixing number 2 makes *another* bug more acute and it complicates the >> whole ordeal. >> >> Thus, bug reported by me: >> 3. interactivity scoring is bogus -- originally introduced to detect >> "interactive" behavior by equating being off CPU with waiting for user >> input. One part of the problem is that it puts *all* non-preempted off >> CPU time into one bag: a voluntary sleep. This includes suffering from >> lock contention in the kernel, lock contention in the program itself, > > Note that time spent off-CPU on a turnstile is not counted as sleeping > for the purpose of interactivity scoring, so this observation applies > only to sx, lockmgr and sleepable rm locks. That's not to say that this > behaviour is correct, but it doesn't apply to some of the most contended > locks unless I'm missing something. > page busy (massively contested for fork/exec), pipe_lock and even not-locks like waitpid(!) >> file I/O and so on, none of which has bearing on how interactive or >> not the program might happen to be. A bigger part of the problem is >> that at least today, the graphical programs don't even act this way to >> begin with -- they stay on CPU *a lot*. > > I think this statement deserves more nuance. I was on a video call just > now and firefox was consuming about the equivalent of 20-30% of a CPU > across all threads. What kind of graphical programs are you talking > about specifically? > you don't consider 20-30% a lot? >> I asked people to provide me with the output of: dtrace -n >> 'sched:::on-cpu { @[execname] = lquantize(curthread->td_priority, 0, >> 224, 1); }' from their laptops/desktops. >> >> One finding is that most people (at least those who reported) use >> firefox. >> >> Another finding is that the browser is above the threshold which would >> be considered "interactive" for vast majority of the time in all >> reported cases. > > That is not true of the output that I sent. There, most of the firefox > thread samples are in the interactive range [88-135]. Some show an even > higher priority, maybe due to priority propagation. > That's not the interactive range. 88 is PRI_MIN_BATCH Then in tdq_runq_add you can see: if (pri < PRI_MIN_BATCH) { <-- aka 88 ts->ts_runq = &tdq->tdq_realtime; <-- interactive threads go here } else if (pri <= PRI_MAX_BATCH) { <-- this is where firefox is along with make et al ts->ts_runq = &tdq->tdq_timeshare; KASSERT(pri <= PRI_MAX_BATCH && pri >= PRI_MIN_BATCH, ("Invalid priority %d on timeshare runq", pri)); /* * This queue contains only priorities between MIN and MAX * batch. Use the whole queue to represent these values. */ if ((flags & (SRQ_BORROWING|SRQ_PREEMPTED)) == 0) { pri = RQ_NQS * (pri - PRI_MIN_BATCH) / PRI_BATCH_RANGE; pri = (pri + tdq->tdq_idx) % RQ_NQS; /* * This effectively shortens the queue by one so we * can have a one slot difference between idx and * ridx while we wait for threads to drain. */ if (tdq->tdq_ridx != tdq->tdq_idx && pri == tdq->tdq_ridx) pri = (unsigned char)(pri - 1) % RQ_NQS; } else pri = tdq->tdq_ridx; runq_add_pri(ts->ts_runq, td, pri, flags); return; } else ts->ts_runq = &tdq->tdq_idle; >> I booted a 2 thread vm with xfce and decided to click around. Spawned >> firefox, opened a file manager (Thunar) and from there I opened a >> movie to play with mpv. As root I spawned make -j 2 buildkernel. it >> was not particularly good :) >> >> I found that mpv spawns a bunch of threads, most notably 2 distinct >> threads for audio and video output. The one for video got a priority >> of 175, while the rest had either 88 or 89 -- the lowest for >> timesharing not considered interactive [note lower is considered >> better]. > > Presumably all of the video decoding was done in software, since you're > running in a VM? On my desktop, mpv does not consume much CPU and is > entirely interactive. Your test suggests that you expect ULE to > prioritize a CPU hog, which doesn't seem realistic absent some > scheduling hints from the user or the program itself. Problem 2 is the > opposite problem: timesharing CPU hogs are allowed to starve other > timesharing threads. > Now that I pointed out anything >= 88 is *NOT* interactive, are you sure your mpv was considered interactive anyway? I don't expect ULE to prioritize CPU hogs. I'm pointing out how a hog which was a part of an interactive program got shafted, further showing how the method based on counting off cpu time does not work. >> At the same time the file manager who was left in the background kept >> doing evil syscall usage, which as a result bouncing between a regular >> timesharing priority and one which made it "interactive", even though >> the program was not touched for minutes. >> >> Or to put it differently, the scheduler failed to recognize that mpv >> is the program to prioritize, all while thinking the background time >> waster is the thing to look after (so to speak). >> >> This brings us to fixing problem 2: currently, due to the existence of >> said problem, the interactivity scoring woes are less acute -- the >> venerable make -j example is struggling to get CPU time, as a result >> messing with real interactive programs to a lesser extent. If that >> gets fixed, we are in a different boat altogether. >> >> I don't see a clean solution. >> >> Right now I'm toying with the idea of either: >> 1. having programs explicitly tell the kernel they are interactive > > I don't see how this can work. It's not just traditional "interactive" > programs that benefit from this scoring, it applies also to network > servers and other programs which spend most of their time sleeping but > want to handle requests with low latency. > > Such an interface would also let any program request soft realtime > scheduling without giving up the ability to monopolize CPU time, which > goes against ULE's fairness goals. > Clearly it would be gated with some permission, so only available on a desktop for example. Then again see my response else in the thread: x server could be patched to mark threads. And it does not go against any fairness goals -- it very much can be achieved, but one has information who can be put off cpu for a longer time without introducing issues. >> 2. adding a scheduler hook to /dev/dsp -- the observation is that if a >> program is producing sound it probably should get some cpu time in a >> timely manner. this would cover audio/video players and web browsers, > > On my system at least firefox doesn't open /dev/dsp, it sends audio > streams to pulseaudio. > I think I noted elsewhere in the thread that pulseaudio may need the same treatment as the x server. >> but would not cover other programs (say a pdf reader). it may be it is >> good enough though > > I think some more thorough analysis, using tools like schedgraph or > KUtrace[1], is needed to characterize the problems you are reporting > with interactivity scoring. It's also not clear how any of this would > address the problem that started this thread, wherein two competing > timesharing (i.e., non-interactive) workloads get uneven amounts of CPU > time. > I explicitly stated I have not looked into this bit. > There is absolutely room for improvement in ULE's scheduling decisions. > It seems to be common practice to tune various ULE parameters to get > better interactive performance, but in general I see no analysis > explaining /why/ exactly they help and what goes wrong with the default > parameter values in specific workloads. schedgraph is a very useful > tool for this sort of thing. > I tried schedgraph in the past to look at buildkernel and found it does not cope with the amount of threads, at least on my laptop. > Such tools also required to rule out bugs in ULE itself, when looking at > abnormal scheduling behaviour. Last year some scheduler races[2] were > fixed that apparently hurt system performance on EPYC quite a bit. I > was told privately that applying those patches to 13.1 improved IPSec > throughput by ~25% on EPYC, and I wouldn't be surprised if there are > more improvements to be had which don't involve modifying core > heuristics of the scheduler. Either way this requires deeper analysis > of ULE's micro-level behaviour; I don't think "interactivity scoring is > bogus" is a useful starting point. > I provided explicit examples how it marked a background thread as interactive, while the real hard worker (if you will) as not interactive, because said worker was not acting the way ULE expects. A bandaid for the time being will stop shafting processes giving up their time slice early in the batch queue, along with some fairness for the rest who does not (like firefox). I'll hack it up for testing. -- Mateusz Guzik From nobody Fri Mar 31 19:07:58 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pp8v66L1mz437tH for ; Fri, 31 Mar 2023 19:08:02 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qt1-x82b.google.com (mail-qt1-x82b.google.com [IPv6:2607:f8b0:4864:20::82b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pp8v64gZkz3Hcj for ; Fri, 31 Mar 2023 19:08:02 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-qt1-x82b.google.com with SMTP id p2so17555245qtw.13 for ; Fri, 31 Mar 2023 12:08:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680289682; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:from:to:cc:subject:date:message-id :reply-to; bh=I7wivi/BupONQGpatWw+ZGulI+q/KEqq7EYbH2LYjho=; b=m6vCVaaa6ld0y+NlO2pRRhAckWifOBpBkSjVw70IjLtkVr9+MTR6x441DqbjFloZ/c fDlxWChRhx+OpL2W4iefukrQWPx3Rimkwm3SLPfDWb1xecjyP03AbPnh1/yb8bdKVa3F L6O7IMBazi4barqXcn3XJNyziLTGTQtO8QOrXffavdUza+PnETyz1KUyx0XUMXKwxONq Pfv6c9LOVnEmCmjJZWKyLDjgDqmI4RKfYHh854u69TKyOg0S33FqawogC7Bvs4tgwMjn PhAblpuiU1J5auetBPdbAFArUCK59pmttbtxpaT4u7e4JbYc1aXhNnSDZ63SZIo9Pj5W pOZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680289682; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=I7wivi/BupONQGpatWw+ZGulI+q/KEqq7EYbH2LYjho=; b=x7y1g2s1vD3Wp+LUgJ2/D83AR/M8bvVPG2zWLHpm2a5+JfZ/Pr+yIHPHDegOzYFflN qB6huUDhQYxtM61KdyxXHaOKZ7XUvYrs6Q6+aQZyLQDiMTK2fGeoK6lda1YsdTbHS20D QLOsm3G7PySVwFqyHJ3aiKcFEyoM6XsOCuvJEHHuK2vHNMd5hFsoN5ElwugYTci5qfg9 B6yarpGXr9TgbqhpsCoAfe5J2VGdSmpJvJ5EMeprPP8rD6WgpfM0QtC079xe3ceeH5EG iHiCKdyrbngLqRRo4GtS52MlvkkAioV/OK9DUh107qIqdIfX7F5b4zuFIH4Dk2kXmTes fiWQ== X-Gm-Message-State: AO0yUKUPJMWowFRaL+k2LLQQOP5pccYe7zaJam6eBELu1z+SfWILrgCd S9rOHXc5Gp1UfLeOE2+7RoN9woJfKmQ= X-Google-Smtp-Source: AKy350Z/AleDbdlqDwJBJYE86C1ycpNpWNfRTpmDtfsVd/cV8QUFwZ5mpOpeeCLa+GkUUVgGMxIZ0A== X-Received: by 2002:a05:622a:214:b0:3e3:902a:a084 with SMTP id b20-20020a05622a021400b003e3902aa084mr45698794qtx.6.1680289681835; Fri, 31 Mar 2023 12:08:01 -0700 (PDT) Received: from framework ([24.114.52.162]) by smtp.gmail.com with ESMTPSA id y3-20020a37f603000000b007468733cd1fsm860506qkj.58.2023.03.31.12.08.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 31 Mar 2023 12:08:01 -0700 (PDT) Date: Fri, 31 Mar 2023 15:07:58 -0400 From: Mark Johnston To: Mateusz Guzik Cc: freebsd-hackers@freebsd.org Subject: Re: Periodic rant about SCHED_ULE Message-ID: References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4Pp8v64gZkz3Hcj X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Fri, Mar 31, 2023 at 08:41:41PM +0200, Mateusz Guzik wrote: > On 3/30/23, Mark Johnston wrote: > > On Thu, Mar 30, 2023 at 05:36:54PM +0200, Mateusz Guzik wrote: > >> I looked into it a little more, below you can find summary and steps > >> forward. > >> > >> First a general statement: while ULE does have performance bugs, it > >> has better basis than 4BSD to make scheduling decisions. Most notably > >> it understands CPU topology, at least for cases which don't involve > >> big.LITTLE. For any non-freak case where 4BSD performs better, it is a > >> bug in ULE if this is for any reason other than a tradeoff which can > >> be tweaked to line them up. Or more to the point, there should not be > >> any legitimate reason to use 4BSD these days and modulo the bugs > >> below, you are probably losing on performance for doing so. > >> > >> Bugs reported in this thread by others and confirmed by me: > >> 1. failure to load-balance when having n CPUs and n + 1 workers -- the > >> excess one stays on one the same CPU thread continuously penalizing > >> the same victim. as a result total real time to execute a finite > >> computation is longer than in the case of 4BSD > >> 2. unfairness of nice -n 20 threads vs threads going frequently off > >> CPU (e.g., due to I/O) -- after using only a fraction of the slice the > >> victim has to wait for the cpu hog to use up its entire slice, rinse > >> and repeat. This extends a 7+ minute buildkernel to over 67 minutes, > >> not an issue on 4BSD > >> > >> I did not put almost any effort into investigating no 1. There is code > >> which is supposed to rebalance load across CPUs, someone(tm) will have > >> to sit through it -- for all I know the fix is trivial. > >> > >> Fixing number 2 makes *another* bug more acute and it complicates the > >> whole ordeal. > >> > >> Thus, bug reported by me: > >> 3. interactivity scoring is bogus -- originally introduced to detect > >> "interactive" behavior by equating being off CPU with waiting for user > >> input. One part of the problem is that it puts *all* non-preempted off > >> CPU time into one bag: a voluntary sleep. This includes suffering from > >> lock contention in the kernel, lock contention in the program itself, > > > > Note that time spent off-CPU on a turnstile is not counted as sleeping > > for the purpose of interactivity scoring, so this observation applies > > only to sx, lockmgr and sleepable rm locks. That's not to say that this > > behaviour is correct, but it doesn't apply to some of the most contended > > locks unless I'm missing something. > > > > page busy (massively contested for fork/exec), pipe_lock and even > not-locks like waitpid(!) A program that spends most of its time blocked in waitpid, like a shell, interactive or not, should indeed have a higher scheduling priority... > >> file I/O and so on, none of which has bearing on how interactive or > >> not the program might happen to be. A bigger part of the problem is > >> that at least today, the graphical programs don't even act this way to > >> begin with -- they stay on CPU *a lot*. > > > > I think this statement deserves more nuance. I was on a video call just > > now and firefox was consuming about the equivalent of 20-30% of a CPU > > across all threads. What kind of graphical programs are you talking > > about specifically? > > > > you don't consider 20-30% a lot? I would expect a program consuming 20-30% of a CPU to be prioritized higher than a CPU hog. And in my experience, running builds while on a call doesn't hurt anything (usually). Again, there is room for improvement, I don't claim the scheduler is perfect. > >> I asked people to provide me with the output of: dtrace -n > >> 'sched:::on-cpu { @[execname] = lquantize(curthread->td_priority, 0, > >> 224, 1); }' from their laptops/desktops. > >> > >> One finding is that most people (at least those who reported) use > >> firefox. > >> > >> Another finding is that the browser is above the threshold which would > >> be considered "interactive" for vast majority of the time in all > >> reported cases. > > > > That is not true of the output that I sent. There, most of the firefox > > thread samples are in the interactive range [88-135]. Some show an even > > higher priority, maybe due to priority propagation. > > > > That's not the interactive range. 88 is PRI_MIN_BATCH 88 is PRI_MIN_TIMESHARE (on main, stable/13 ranges are different I think). PRI_MIN_BATCH is PRI_MIN_TIMESHARE + PRI_INTERACT_RANGE = 88 + 48 = 136. Everything in [88-135] goes into the realtime queue. > >> I booted a 2 thread vm with xfce and decided to click around. Spawned > >> firefox, opened a file manager (Thunar) and from there I opened a > >> movie to play with mpv. As root I spawned make -j 2 buildkernel. it > >> was not particularly good :) > >> > >> I found that mpv spawns a bunch of threads, most notably 2 distinct > >> threads for audio and video output. The one for video got a priority > >> of 175, while the rest had either 88 or 89 -- the lowest for > >> timesharing not considered interactive [note lower is considered > >> better]. > > > > Presumably all of the video decoding was done in software, since you're > > running in a VM? On my desktop, mpv does not consume much CPU and is > > entirely interactive. Your test suggests that you expect ULE to > > prioritize a CPU hog, which doesn't seem realistic absent some > > scheduling hints from the user or the program itself. Problem 2 is the > > opposite problem: timesharing CPU hogs are allowed to starve other > > timesharing threads. > > > > Now that I pointed out anything >= 88 is *NOT* interactive, are you > sure your mpv was considered interactive anyway? Yes. > I don't expect ULE to prioritize CPU hogs. I'm pointing out how a hog > which was a part of an interactive program got shafted, further > showing how the method based on counting off cpu time does not work. You're saying that interactivity scoring should take into account overall process behaviour instead of just thread behviour? Sure, that could be reasonable. > >> At the same time the file manager who was left in the background kept > >> doing evil syscall usage, which as a result bouncing between a regular > >> timesharing priority and one which made it "interactive", even though > >> the program was not touched for minutes. > >> > >> Or to put it differently, the scheduler failed to recognize that mpv > >> is the program to prioritize, all while thinking the background time > >> waster is the thing to look after (so to speak). > >> > >> This brings us to fixing problem 2: currently, due to the existence of > >> said problem, the interactivity scoring woes are less acute -- the > >> venerable make -j example is struggling to get CPU time, as a result > >> messing with real interactive programs to a lesser extent. If that > >> gets fixed, we are in a different boat altogether. > >> > >> I don't see a clean solution. > >> > >> Right now I'm toying with the idea of either: > >> 1. having programs explicitly tell the kernel they are interactive > > > > I don't see how this can work. It's not just traditional "interactive" > > programs that benefit from this scoring, it applies also to network > > servers and other programs which spend most of their time sleeping but > > want to handle requests with low latency. > > > > Such an interface would also let any program request soft realtime > > scheduling without giving up the ability to monopolize CPU time, which > > goes against ULE's fairness goals. > > > > Clearly it would be gated with some permission, so only available on a > desktop for example. > > Then again see my response else in the thread: x server could be > patched to mark threads. To do what? > And it does not go against any fairness goals -- it very much can be > achieved, but one has information who can be put off cpu for a longer > time without introducing issues. > > >> 2. adding a scheduler hook to /dev/dsp -- the observation is that if a > >> program is producing sound it probably should get some cpu time in a > >> timely manner. this would cover audio/video players and web browsers, > > > > On my system at least firefox doesn't open /dev/dsp, it sends audio > > streams to pulseaudio. > > > > I think I noted elsewhere in the thread that pulseaudio may need the > same treatment as the x server. > > >> but would not cover other programs (say a pdf reader). it may be it is > >> good enough though > > > > I think some more thorough analysis, using tools like schedgraph or > > KUtrace[1], is needed to characterize the problems you are reporting > > with interactivity scoring. It's also not clear how any of this would > > address the problem that started this thread, wherein two competing > > timesharing (i.e., non-interactive) workloads get uneven amounts of CPU > > time. > > > > I explicitly stated I have not looked into this bit. > > > There is absolutely room for improvement in ULE's scheduling decisions. > > It seems to be common practice to tune various ULE parameters to get > > better interactive performance, but in general I see no analysis > > explaining /why/ exactly they help and what goes wrong with the default > > parameter values in specific workloads. schedgraph is a very useful > > tool for this sort of thing. > > > > I tried schedgraph in the past to look at buildkernel and found it > does not cope with the amount of threads, at least on my laptop. > > > Such tools also required to rule out bugs in ULE itself, when looking at > > abnormal scheduling behaviour. Last year some scheduler races[2] were > > fixed that apparently hurt system performance on EPYC quite a bit. I > > was told privately that applying those patches to 13.1 improved IPSec > > throughput by ~25% on EPYC, and I wouldn't be surprised if there are > > more improvements to be had which don't involve modifying core > > heuristics of the scheduler. Either way this requires deeper analysis > > of ULE's micro-level behaviour; I don't think "interactivity scoring is > > bogus" is a useful starting point. > > > > I provided explicit examples how it marked a background thread as > interactive, while the real hard worker (if you will) as not > interactive, because said worker was not acting the way ULE expects. > > A bandaid for the time being will stop shafting processes giving up > their time slice early in the batch queue, along with some fairness > for the rest who does not (like firefox). I'll hack it up for testing. > > -- > Mateusz Guzik From nobody Fri Mar 31 19:43:10 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pp9lg3JhKz439hQ for ; Fri, 31 Mar 2023 19:46:39 +0000 (UTC) (envelope-from jroberson@jroberson.net) Received: from mail-pj1-x102f.google.com (mail-pj1-x102f.google.com [IPv6:2607:f8b0:4864:20::102f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pp9lf2xjCz3M7s for ; Fri, 31 Mar 2023 19:46:38 +0000 (UTC) (envelope-from jroberson@jroberson.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=jroberson-net.20210112.gappssmtp.com header.s=20210112 header.b=WpVaJvV5; spf=none (mx1.freebsd.org: domain of jroberson@jroberson.net has no SPF policy when checking 2607:f8b0:4864:20::102f) smtp.mailfrom=jroberson@jroberson.net; dmarc=none Received: by mail-pj1-x102f.google.com with SMTP id gp15-20020a17090adf0f00b0023d1bbd9f9eso26596733pjb.0 for ; Fri, 31 Mar 2023 12:46:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jroberson-net.20210112.gappssmtp.com; s=20210112; t=1680291997; h=mime-version:message-id:subject:to:from:date:from:to:cc:subject :date:message-id:reply-to; bh=k53qO8umfeBH4v8lZN0+HGQHHHDfUEtknMpgd0FgH74=; b=WpVaJvV5hjcf3Wd59xj3b6QLMcaEoZNh08IHqLPzgRvLv99rEoRBF/u9aGKyjIMJyM Ju9wXXX5nqz2aVag8SZVggKQ3hLeFjqnPPcRjN1iT8tdhDDRkqSLiMFlgwOLOXeAUsqJ DeSZL2+7tlu3be+UN/fCEAOfggwHP+qiHKnFIW+lhov76w1sVr4HmFokR+LYQZUW5wsp meFuBOx7BPQq3GMeDAdfS4wgdkDiSxj+XVD0Mgvg40zCfFsoduPT4qg5/kjLvGr6CwhQ YlsMMsuyslSkbkhJPYrtvaRvsvHmy/WHsA/gJpQ4dIS0uUBnnLahoFkggFtaVjLqz83y kN/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680291997; h=mime-version:message-id:subject:to:from:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=k53qO8umfeBH4v8lZN0+HGQHHHDfUEtknMpgd0FgH74=; b=VqitryxnqJoazUIbbbfVGJbgqdjo3xNxboCJontUUeF/AX7osZJb6sbjr4Xei8X4sp yRw6WvlBdkVy0OwwNJqifLI1r6fvl3PMGiOlyYzRTc1mbr6XjbYgz7XcejtARcLenwIp 8YapeydUlc0WBFDTDZJrpiekZhsO+8siN+EORajP7eX02lVXIyzjW9CXKklXEW6Oc48n +6sKDpiT1D/st9WOlULpksAi+DICfT4tvetQacjM1HeYkJArpPRI7p3oW2RVdhsdTZ89 IqLuy9+JvvUGKUuFileJjOiXHHg+x1i1HkZ59tgmCw32T1tDuvvJMwqXvls71SBuw45r 3wGA== X-Gm-Message-State: AAQBX9c/8WndwuL3rJ6zCR2kcnzUUKUbugsZ/xs7n3n/SG1JZKaWtf55 w0E2wnaZffDslmuHDe02pL5UOaTBpxMxGnHLSbI= X-Google-Smtp-Source: AKy350b7icUjlAepufr/zHTd0eqewIc0CXrRIK3JRU7KpseHWLIUTc+A8j/dGr5cxJseTlQ4Pp9arg== X-Received: by 2002:a17:902:fb85:b0:19e:b088:5900 with SMTP id lg5-20020a170902fb8500b0019eb0885900mr24209553plb.38.1680291997030; Fri, 31 Mar 2023 12:46:37 -0700 (PDT) Received: from [192.168.0.31] (c-98-246-66-2.hsd1.wa.comcast.net. [98.246.66.2]) by smtp.gmail.com with ESMTPSA id u5-20020a656705000000b00502e6bfedc0sm2030711pgf.0.2023.03.31.12.46.35 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 31 Mar 2023 12:46:36 -0700 (PDT) Date: Fri, 31 Mar 2023 12:43:10 -0700 (PDT) From: Jeff Roberson To: freebsd-hackers@freebsd.org Subject: ULE process to resolution Message-ID: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset=US-ASCII X-Spamd-Result: default: False [-3.30 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[jroberson-net.20210112.gappssmtp.com:s=20210112]; MIME_GOOD(-0.10)[text/plain]; R_SPF_NA(0.00)[no SPF record]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::102f:from]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; DKIM_TRACE(0.00)[jroberson-net.20210112.gappssmtp.com:+]; RCVD_COUNT_THREE(0.00)[3]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; DMARC_NA(0.00)[jroberson.net]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4Pp9lf2xjCz3M7s X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N Hi Folks, For those who don't know, I am the original author of ULE. I have not had much time for FreeBSD in recent years but this thread was forwarded to me and I am dishearetened at the state of things. I will give my perspective and propose a path to resolve this systematically. The fundamental benefit of ULE is also the fundamental challenge, That is: N cpu local decisions need to add up to a reasonable approximation of a correct global decision. This is necessary to scale to large core counts, large thread counts, and preserve some affinity. You could permute 4BSD further towards these goals but I posit that you would simply have to work through the same bugs. As I read these threads I can state with a high degree of confidence that many of these tests worked with superior results with ULE at one time. It may be that tradeoffs have changed or exposed weaknesses, it may also be that it's simply been broken over time. I see a large number of commits intended to address point issues and wonder whether we adequately explored the consquences. Indeed I see solutions involving tunables proposed here that will definitively break other cases. I know that CPU tradeoffs have changed. ULE was written in a way that the topology could be annotated and cost of migration can be specified. It is adaptable to this but someone has to put in the effort. The cost function was written in ticks which does not scale down properly and accurate cpu tick counters could now be used for more precise time-keeping for more specific affinity. Over time people have also added additional searches to pickcpu which don't scale well to very high core count systems. NUMA and heterogeneous CPUs are also possible in the graph framework but need further investment. The other thing that has changed over time is the ability of the interactivity score to correctly detect truely interactive applications. When I wrote it you could do a buildworld on a single core or small multi-core system and play mp3s and browse the web without a hiccup. However, web browsers have evolved to be significantly more resource intensive. I'm not sure a heuristic can or should catch this case. We're probably long overdue to add x window focus hints as most other operating systems do. I don't think tossing the interactivity score is really going to produce the desired results. Linux CFS disagrees with me but I have always been able to achieve superior responsiveness with ULE. My intuition is that with an x window focus hint we could dial back the interactive threshold and have better tradeoffs with the soft real-time score. schedgraph is also no longer adequate for modern systems. In my professional life I have taken the same types of data sources and built text based processes on top because graphical representations just can't scale to the number of events and cores for full system scheduling. For complex scheduling issues you need detailed introspection. You're not going to tweak variables and run buildworlds to arrive at success by supposition with any kind of reasonable velocity. The first step to resolving this is to come up with a list of regression tests and catalog how they behave compared to 4BSD. When I wrote the scheduler I also wrote a simple fixed duty cycle program that could be run with different scheduling parameters and report on its cpu usage and latency. Combining many copies of this program you can simulate various kinds of interactions. It is available at people.freebsd.org/~jeff/late.tgz. I know there is also a linux scheduler benchmark that may be worth porting. If someone would start making regression tests I am happy to fix bugs or review bug fixes. Personally I would start from fairness given different nice values on a single CPU, and then multi-cpu. Evaluate allocation with variation on load to core count ratios. It should not take a few hours to iterate through the interesting cases here before going on to more complex questions about buildworld or firefox etc. This would need to be something we carried forward in the source tree and ask people to re-run as part of scheduler CRs or we're just going to find ourselves back in this spot again. I also have a backlog of improvements for large multi-core systems from work I did years ago that have not made it into the tree. And I have an old review for patches to improve the reliability of priority in causing scheduling events that may be germane. If we can collaborate on a testing framework I could trickle these in. Thanks, Jeff