From nobody Sat Apr 29 21:09:17 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q82Cy1RtKz48GHc for ; Sat, 29 Apr 2023 21:09:34 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic309-21.consmr.mail.gq1.yahoo.com (sonic309-21.consmr.mail.gq1.yahoo.com [98.137.65.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q82Cw3QbPz3n81 for ; Sat, 29 Apr 2023 21:09:32 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=PCLmk6CF; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.147 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1682802570; bh=8YcqB/ws+4OyS94d2bsWpKaKQSMb40BWDad0bUhTw44=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=PCLmk6CFbDhKGFirdAJxz1mEJoaDBIb+5/T09kPj2fqrkAtq5GQMxinLrOyCCiH4KCdRbZegvAqA/9+li/wdeZ7lUd9K7DIaBzXRlIXf0RhdGtqKqdeXKhiSjz2XJuPPKMXxQUYFmquo3vWAKJpjnszUgWehOGCUg9nhVroAWg7AqB7GViCS/mpFF1wVVMrdsVffn6WPMCH2xuGEbEqJy0zNsoYzxB1u6J6jgBEi/TH8Lx8jDY2E4SH7t0m5ZNvrquyoB7XLV5aHq9NJ18e1+14B6FPiUMiTn7nhxXm5aKHYk+eP8x2T9K7g38Zw1vADevWxdAalVS3aurltpZsDWw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1682802570; bh=d7WS8dUBZeYB8BCHUqLCFfojSms0EaeIA/VXv4CUSpm=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=UxvdF07mxHFrjvXA9bsvARY2OyTy/7QXpDNPR/nMTRT4ltVFNoSU5BsE0mJwlFdMNYrhvgWic+CGU75VhF7tlMAuk3D2MPVEiumY+EtrKpJ6KaKFEkeTSaU5aoqwuc7fxI4MCwNo8prd1IZ6wZh395Jr+yxXhCuKYEX6NHk4KtUaYZE0cqk3N4Qi8ktFVRr+yxrGVl5Vt64Bjpo5Rn5vfNTXYDc5gBl9Qur03uykpLE86QXcTlXIbDSRz7+gb26KZ1giVQkxQnZ4TV9cN9+vMPwjjXs7/lQGuVOmCO5x2DDDYCeoEmYJBwGl/C8QYvAXjE1OtmijJtpVwPsckLnyFQ== X-YMail-OSG: vMyEWc8VM1likePTgbc6e9nv8fTHT.UXSE0J9dHqC_Rkxmm2FihNrBvogvdgbDH aYCUb8h74wXcO1A.ARNxbCzUwivUpY87s71dP.HkGzZOCwEbsfnF6H_Jd1byO1I6nE4aOBHKQVHD 48kUl6MFnsPyjyoCbDjpqopRbCHsHABaBix5gK5VJcX9ycvc.KintqHAuVKNryxA6VKWJYcBcmd7 4ptQgH97cbCH6jWCborrdfGdB_5E5JN97ZI3WXqdEHRmLCUUtlchJD4Cyu8NNsOaIhxXDRpGHkjL Y.0HrknkkdSSuWMjl3ob.GzRGL9LgPHiN9_xdZftbuceVFFJSkIgolpMayjt9Dzxj_po8zMnu03L 1ytwp2qr6ifule7qJLI9XOuO.yFLdfBWHyGRagA34O_quf8KZx.U.6gLL2fUqg70gGMUDmqcscbI aa1hD_QspEnsO9EOuDXQiKtMjjxyQyJT9_9nEoq0cztQHNyPq5J6ELy8GunUr2o4m.1N3Vh8zig1 1Hv2vJGWW1K4UlBveetvDqcX66KMVIYGcLn6JH_js3VDbvvuQURSTRxFoNxrgfkfmMx1cYJkCHPw X4lmvPYm827rfKsKGQRHTVHPFCzFMI7I3Wigq.VVeJ.aYYY0xBfdUu_ponMQdRZscojFrWecimCQ sAJ5neOWO1F1bKEuSG.XeZZ.nbziwA._zQ_5q1v4ylvUxIuOQibeJV1V5HzGCtc3we2pfAjdLqQM qUgh3yg.r6kw_be2kxnrmFBDgHRx8IYYRa5rl7N78w9TPKB0rCbKGQr_t9ODoCFXIY5nvjFQRUsp eYsExvCT5Y8D6TxWZWf4T1EfxTemHvX8lNJC_n4skad1GU1W33pKhqGXUmsVk0WNedFw4kQrMDTj L25NwC3WqETYj8fj1AjtOlrWevn9F07Tdk_sKMYfFhHK9kBC4SydrGfp4lAJIghJCRlAU.XIoQsO xNnUctwOFjKkDMtqhfFyf6UF.w4jTpPliPmoWvMib5SqGU._k9FBtmv06q9YZ5000Wm.8iS6Mtu2 PYPt3w4fT4Uyxv258pj5REh0XTr02wjN3C3_FiRardeaUpomKUPVZGwaS9Bwc8qnbub2tkb.DleX izR8m52KtOQa1S.eTWtWCTnpWu1CrORAh6_ED4Qwr2DJ2MWj_.e8j2BmpxN.4Z5N3LuCns7MsqDR TywC2JRQxXSvrqa7q5BCNbhuzQJsQvCoLhV41JQauaeNfnh8LVkp2F2Fm1Avh8KzrOvpbWM8KUhH OYmKZf56eEQL_SclIyGE0ERexpp16vp06kZcUuxpzGV5N.pvSyJGA3Q37YyzErnCHMyr7rsiV4gT D.2mUYS0J9_aBSItHl5.8OqH_an1hvWK3.bzWdeFjh_aNXgXVVzNigEWHG5VkSN2jNBo4HTUNqDO aZor7RvKwqJWXQSWBfChlQ65Pd5e7N.U.nA7qlbjJhd_ZzR084sAT_5XlPYYkANlDaXBf62Jmxyp UHD9nWaR7KsbxnHoVPLjCUl4azUh6qWx46snJjHUMjO71Jw7N5.BwV4tkorbQalxtTQ7Ai81ukmd LegF4CNKhvQ5i6CsjdEVxdyKbKBRZ_RCU31PpBdGj6frSynS7QT053ChvJpO5.URhCHJJgKQT6Mm 1fWstW77DzX5bAPXUkng7TF37.AR6DpXWovUieI0ix22WSZZ_wSIUD644L4sExD097EtrZeXC6dd I8USyuZe.2b3WAG9yAPG5mpxINdRv_nYIBld9bLjXq8jPGb6wIiVQcfn6ZauCImu0SNok1ZDoZB3 hs6IDj174XSMD_sBPrLhYzXG_UaF.IkS.s76DMWrx2zpNO5eMizS4UoL_MSxDOK3K0x3m5pQfF9a VEXwqxld2QZ5BBlxn4qtunbDKB.JzIvAr5RV_E5NBAsl7fOC2lDk9tODQFOYJhQQ8MDukKSm6XRt IAPDKX_d.qxOO17vqPjOmh5oSb5ts1RRxdAhzoUzr9_6GjhOrBAecmd42Q5l8zAvD.4s.r2C6VtO NpITJORNdLbeKoEFDf9TE86Ar6.yXn77GX_z.xGtNKNv43svQy02_CSpNg8dHkwhN3cX0FQaapCQ iTdnWWA5qj53W9WdYRiwm4nn609o0TqqbTR4MSNiZN5NN_tvjQCQ14EpEUuIgI4EQY7el4nPANX7 xPy9eaDYsOkSmzCP478CCWbDmtYxSQeSEzG9FrfjDDL5jEpLGJaXHJj0QGvl0ZZrLyS9T6Z8MN8L kxQgfOrBpqcfcgNNHMHXqz4sI2O.WDWebaG9ESc1GiCF4EqKgoqTV6Gj_8aimkyEndR7MbGoxBg5 lcgo- X-Sonic-MF: X-Sonic-ID: f0e58359-f89c-44f2-96c6-3247303c2dc6 Received: from sonic.gate.mail.ne1.yahoo.com by sonic309.consmr.mail.gq1.yahoo.com with HTTP; Sat, 29 Apr 2023 21:09:30 +0000 Received: by hermes--production-ne1-7dbd98dd99-qthmr (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID f994fac8294747fd934df0c16472a593; Sat, 29 Apr 2023 21:09:29 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: armv8.2-A+ tuned FreeBSD kernels and buildworld buildkernel times: an example From: Mark Millard In-Reply-To: <177A2369-1751-4DB5-B316-E140ED156B6E@yahoo.com> Date: Sat, 29 Apr 2023 14:09:17 -0700 Cc: freebsd-arm Content-Transfer-Encoding: quoted-printable Message-Id: <4E8BB159-38D1-4EF0-B486-DF6C8B49D7AB@yahoo.com> References: <177A2369-1751-4DB5-B316-E140ED156B6E@yahoo.com> To: FreeBSD Hackers X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-2.39 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_SPAM_SHORT(0.11)[0.114]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.147:from]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4Q82Cw3QbPz3n81 X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N On Apr 29, 2023, at 12:16, Mark Millard wrote: > Context: all world's and kernel's involved/built are non-debug style. >=20 > Note: clang15 through LLVM main (so far) has errors in both directions > for the features for cortex-a78c. So I also used +flagm+nofp16fml . > (The cortex-x1c also has such problems, but the details are > different.) >=20 > Notation in table below: > CA72: matching world or kernel had been built using -mcpu=3Dcortex-a72 > CA78C: matching world or kernel had been built using = -mcpu=3Dcortex-a78C+flagm+nofp16fml >=20 > System: Windows Dev Kit 2023 (4 cortex-a78c's and 4 cortex-x1c's): > (both: armv8.2-A with a few more modern features) >=20 > Times to build system from scratch (buildworld buildkernel from same > sources) . . . >=20 > System running: World built in: kernel built in: > CA72 kernel, CA72 world 6601 sec 597 sec > CA78C kernel, CA78C world 4680 sec 413 sec > CA78C kernel, CA72 world (chroot) 4715 sec 422 sec >=20 > The CA72/CA72 is from before I'd built the CA78C world and kernel. > All builds used -j8 . None had competing activity on the machine. >=20 > What this suggests is having an explicitly armv8.2+ tuned kernel > makes a notable difference for -j8 buildworld buildkernel times > on aarch64. "Tuned" here includes newer-feature use, so incompatible with the likes of armv8.0-A hardware, for example. The FEAT_LSE atomics use would be an example. But I've done nothing to investigate subsetting the new-feature use to isolate what makes the biggest contributions to the elapsed-time decrease. > The Windows Dev Kit 2023 is the first (and only) armv8.1+ based > system that I've have access to. So testing such properties is > limited to the one context. >=20 > Also, I've not had access to the Windows Dev Kit 2023 for long: > first experiments. >=20 >=20 > Notes on my historically-usual aarch64 builds: >=20 > On cortex-a72 hardware, my context is -mcpu=3Dcortex-a72 based. This > once exposed a lack of sufficient synchronization in a palce in > the USB subsystem. (Running the same system on cortex-a53 hardware > did not fail. Running -mcpu=3Dcortex-a53 based world+kernel on a > cortex-a72 did not fail. A cortex-a53 hardware running the > -mcpu=3Dcortex-a53 based world+kernel did not fail.) >=20 > Until the hardware failed, there was a time when I also had > access to a cortex-a57 FreeBSD system. >=20 > I do not do such -mcpu=3D tailoring on the only FreeBSD amd64 that > I've access to, a ThreadRipper 1950X. I do such only for the lower > end systems that I have access to. My aarch64 access is all to > lower end, not upper end. I should have reported that my recent activity for this is based on: main-n262658-b347c2284603-dirty, b347c2284603 being from late Apr 28, 2023 UTC. (The "-dirty" is from some historical patches that I use.) Some of my activity has been from somewhat earlier but I wanted to pick up another openzfs fix nor 2 that had happened since then.) =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Sun Apr 30 02:44:13 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q89fM46grz48c32 for ; Sun, 30 Apr 2023 02:44:27 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic317-21.consmr.mail.gq1.yahoo.com (sonic317-21.consmr.mail.gq1.yahoo.com [98.137.66.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q89fL5B94z4FJ2 for ; Sun, 30 Apr 2023 02:44:26 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=s0flWfP4; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.66.147 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1682822665; bh=Dg1zRP6K+BkPLSZIDNaWBC+aEEpxAZJKuTftAcqwykQ=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=s0flWfP4B9WE88d56JpOwM21gfvfnRN+Jnkve3V3JuQam+1cFDlHx1uUxTnbrhcm+9AJb3oVkzHN9aIO3cwYZlrx2qx6OM50MhEjCccLLGHuTPorsVB0tkzjyG9+nq+hoNp9bHrJBuK6RrByBdQwJNEtzmDUtYj3yPL8gtzeGWPqfNRxXeuGTM331OYbglut8Gc1ZRHbG+UBLyc9PeG3HSG4oAwpfBcrX/5DnqG7xTEs3B6J4hbbpQgS3hLB2qc9PyO9doeCwIyOU7afSypHMlPXuyAw6QcKy1Yj10yXrsOExL7wIyxGWoLOpQwg5Q42M9HgDUkMK8wSpy0hGwiVIQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1682822665; bh=nd5Ii+dBXUB/ugS8bU8qUae5jVgyBEn1AL++8gTUnta=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=cWxeM5J23bZN7CSYnlcGkq4wIFvPdRlFeABqJvyC8NWp5iYjYfoQAZHmYNbvUWjeEJ0IObn2Ig7fRPcpx+n6DfyujnQPI7PH9rGC1SkWaXfNScXR9c3xP2Od7XkFMt4eso6CS1bPih8N6+wFqNyUcscPWxnfcPhabA/C9daX4VviY2cAhZYcRog8q4W/XZ4uUCKNUwLvyoQF4D41opO4ab6nOaP0v6iz5j1CTs+OnZz1z2mRAneQGFM1btvs8hDhDVQFZBUXSHrPIaCIzcLrKoYYMlkq1mG268a4lOkl5zFvH7SsHaC+kGVupN7nKydnmMI3j3l1p++6hu8Dd0JoFA== X-YMail-OSG: WVuzG08VM1md.qqGM.tX2xSZ0snDT4G9Xsp6Lxrk_hed5Ud7wTtkNBm2wVIVmVx 0TZqzngtAmmloe1JSFVvZokTb53JBd7xCqJ539XlSF7ifXayryJvvC2IaC3T9apRiUEj15piK3U4 JhrbRFm_m0OBK8.Lg1wNTpycC_IYbbTNRC4X8KspLhmGKp4.hXvCTyrH02cOftrrH67VqeNfXo2V Xwi.m8uSQntmVepI3O857EroCqXXSeMtsQhDxzK7koa8lfICYYtJUpjSNY2F8sKJIph624RQGKv3 nQPM5bCJqn8dWQfWNtF40QX9CsuwGe5XELhGEfhGwfTraGz.LM.NsaDNd5CQ0l4UI1HFIfwIzhou vuKt78CgvkAnlYMYcg2n08Zu7orj.0b_6iaED1vANu97Nn6m7fkf6rSI9WpfA3ktRbQ_7oneEBeA N912iAmMANH7n2DYRHWCfaefqUPynrTCjiLTJTdXszz327oLiOJ1AckYzav1daAhidljjWJi18Xa FIp7gIjb.261mcFgEy7lpPimtlf1C8rmrFwjeFul_SWsV4Ds5dS7dcRHt6EykJ7KvgNhtf0qQIyE nLh6Vw2Xf782KX8_kcTgw5I35lXQECqIKDcLAFQohEZszHu0nH3FxxORhsF06UOyaZDxwx_Vu3tc 6iDgnDw.SLHih7sxoarYyywQfPaLto_TwHmzGSmtSFyruSzIFj4zRNDwcLGkatM2cwG_k.QU0gEQ U3KtrqGSaV6kC4tHgKnq67sKQkykuDqtgYo07x9mEWS6ul.6ExuR8GGh4yesM0JZseEOlPUi2idU df0K9Ynca9geYTu2woPXPox21nE2q.mjfT3FUpcQLcAFF5szJo0BmnBgZRYR5dFOfhWJHqQr_LNc _KO_eS7.tpShej0bDpScbaqYDqPITdgc3fQJBb9VcAZYP.UrtW36a8ug7F0OFcrbDggiABL1HUPN kGvzgy3A0N_0EJJI14HW04ro0yZEcUqnkK.KH.GyueKfQkmjaL.fcBEgN6Vva_dl0tMCzAXKG7UP xPMue.B.QFjSOUFqnruQmcNLcbFw14Z23yWlORXraXPyxPlpUd8ofxLn1QK7UwF5ZaP.Ul9xq21u hEtxNrUWqv8r32nYqEMP_d1GIgxVoPXp1g7z97bzOZrB9eG0q0xUz_prkwRkBueG22unZJMhmXJ4 qfjsivJlERgJ2xNg8i4MfEhMWLU4RsHnS2jiNafFnkmyizL7oSj2PaP6nhB6ES1NPwKJPWsnx897 GIvaVF4OPJkM.4qgCx6TbM97QAXMVeyTIp.hK7e3XcMIqghp8XT0TWEJ_9ykC8XFjdK8nEvSkhk4 Qzcb0lgCcVNUBBtqMCv2sdSFWi5I96WXh7PrGOgUrh8Adbi7zxN7GxxB_R_gGq2oWSkEMbeMRt.z McN.U6Zn80cn3su0ijyXJnGbFKr47Kt5hwI1JCc7udBCNV9HIiQMhqMyQA.S4a9O995dhrlLVknz oMM7J9nCHGbUJ.LGEc9DKpcsf8SUxQSG_KOC2bhOqFhje8idH.NXoGyiNpbLegOtp5ozoAi59uWf Kz1tzjFEnmkqcBdOxQ6.laaYolcQJM5MSJA4O5MPCuXk8nrMRTU_XXEeZYpfHFiYrTHkRq4i_vAQ _HPgJ5V8ze6p82PRMnXD5V2HvE7_rfqazu3Ry813gXcLRVCQmoMNbpVU68.6S.XzqpZDqyUOI6Wa YhQ4IJsKG.5hmOxgEpxKJ2hbXzOnVD2EQ7jG2nz1.h5U_sauRKndCaUvb7WDkeXOqkORZmsTpVec 1eU1R1ri75g1tsUvgSVd0gYTSFGW2YkHwRnQEKjqDJK5Ka_.jwfOgkGT9g9qR7hLHQvx63LkUeqC kFmLJUs7I8a1KE9HeFM90gVQVdHkrVQc9tZh3hibEkXjVZAyAdikaHcj5siC8.coSnH.dSaQtbtk ik9KpcaQ.apO1x_huFuZ176RJqqCS78ZTvXHGglsl2uSQElnbAuQTrO72KUgkUfICGID6lEEiG9B vW1RVp9H2fdCv2uYcKme46ryZzuFtQcVCkyT3jRXKy97ZnOJG.7h5RwelCr8G50TPjezC5Jmkvav _Q_AUghz8r_BH5qLbkF_t6tY26EUnFg77ec.JeqY572lbziWPghnRfs9dIkdCM5.P0RU1I4h3PHK wEtnGwiCIfqXfhZH7XcBVnQZ5wK_uUCi_ksvVL43vhUstIu_irZfQNRoz7bRibP4Q9wVgeS9rTFT oYI7xdcCvHP2XJSC2p94IDFfuCMzefmT5HS9H7yHPeASCTKIjo6QgKRgXOdwrVvWuy7m6oJCqzLF m X-Sonic-MF: X-Sonic-ID: 600319dc-fa0d-4f7c-9d89-4504435516d5 Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.gq1.yahoo.com with HTTP; Sun, 30 Apr 2023 02:44:25 +0000 Received: by hermes--production-gq1-546798879c-g88f5 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 1023595f13d6ecaf5c33461fb2a882a1; Sun, 30 Apr 2023 02:44:24 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: armv8.2-A+ tuned FreeBSD kernels vs. poudriere bulk and USB3 media: tx->tx_quiesce_done_cv related blocking of processes? Message-Id: <7AE28A5B-109E-4C26-9D70-BCA5D49CD79D@yahoo.com> Date: Sat, 29 Apr 2023 19:44:13 -0700 To: FreeBSD Hackers , freebsd-arm X-Mailer: Apple Mail (2.3731.400.51.1.1) References: <7AE28A5B-109E-4C26-9D70-BCA5D49CD79D.ref@yahoo.com> X-Spamd-Result: default: False [-2.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_SHORT(-1.00)[-0.996]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_IN_DNSWL_NONE(0.00)[98.137.66.147:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_TLS_LAST(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.66.147:from] X-Rspamd-Queue-Id: 4Q89fL5B94z4FJ2 X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N This is based on: main-n262658-b347c2284603-dirty, b347c2284603 being from late Apr 28, 2023 UTC. (The "-dirty" is from some historical patches that I use.) The build is a non-debug build (but with symbols not stripped). World or Kernel had been built using: -mcpu=3Dcortex-a78C+flagm+nofp16fml just for testing purposes. (Worked nicely for -j8 buildworld buildkernel testing for the 4 cortex-a78c's plus 4 cortex-x1c's present.) Monitoring poudriere bulk related activity via top and gstat -spod I see a lot of the odd result of one process doing something like: CPU4 4 1:39 99.12% /usr/local/sbin/pkg-static create -f tzst -r = /wrkdirs/usr/ports/devel/cmake-core/work/stage while other processes sit in the likes of: tx->tx zcq->z zcw->z zilog- select wait But sometimes there is no CPU bound process and the top CPU process is the likes of: 1.24% [usb{usbus0}] "gstat -spod" basically shows da0 dedicated to write activity most of the time. After: sysctl kern.tty_info_kstacks=3D1 Then using ^T at various times, I see a lot of: load: 0.48 cmd: sh 93914 [tx->tx_quiesce_done_cv] 7534.91r 11.06u = 22.66s 0% 3800k #0 0xffff0000004fd564 at mi_switch+0x104 #1 0xffff000000463f40 at _cv_wait+0x120 #2 0xffff00000153fa34 at txg_wait_open+0xf4 #3 0xffff0000014a40bc at dmu_free_long_range+0x17c #4 0xffff000001448254 at zfs_rmnode+0x64 #9 0xffff000001455678 at zfs_freebsd_inactive+0x48 #10 0xffff0000005fc430 at vinactivef+0x180 #11 0xffff0000005fba50 at vput_final+0x200 #12 0xffff00000060c4d0 at kern_funlinkat+0x320 #13 0xffff00015d6cbbf4 at filemon_wrapper_unlink+0x14 #14 0xffff0000008f8514 at do_el0_sync+0x594 #15 0xffff0000008d4904 at handle_el0_sync+0x40 load: 0.34 cmd: sh 93914 [tx->tx_quiesce_done_cv] 7566.69r 11.06u = 22.66s 0% 3800k #0 0xffff0000004fd564 at mi_switch+0x104 #1 0xffff000000463f40 at _cv_wait+0x120 #2 0xffff00000153fa34 at txg_wait_open+0xf4 #3 0xffff0000014a40bc at dmu_free_long_range+0x17c #4 0xffff000001448254 at zfs_rmnode+0x64 #5 0xffff0000014557c4 at zfs_freebsd_reclaim+0x34 #6 0xffff000000a1340c at VOP_RECLAIM_APV+0x2c #7 0xffff0000005fd6c0 at vgonel+0x450 #8 0xffff0000005fde7c at vrecycle+0x9c #9 0xffff000001455678 at zfs_freebsd_inactive+0x48 #10 0xffff0000005fc430 at vinactivef+0x180 #11 0xffff0000005fba50 at vput_final+0x200 #12 0xffff00000060c4d0 at kern_funlinkat+0x320 #13 0xffff00015d6cbbf4 at filemon_wrapper_unlink+0x14 #14 0xffff0000008f8514 at do_el0_sync+0x594 #15 0xffff0000008d4904 at handle_el0_sync+0x40 load: 0.44 cmd: sh 93914 [tx->tx_quiesce_done_cv] 7693.52r 11.24u = 23.08s 0% 3800k #0 0xffff0000004fd564 at mi_switch+0x104 #1 0xffff000000463f40 at _cv_wait+0x120 #2 0xffff00000153fa34 at txg_wait_open+0xf4 #3 0xffff0000014a40bc at dmu_free_long_range+0x17c #4 0xffff000001448254 at zfs_rmnode+0x64 #5 0xffff0000014557c4 at zfs_freebsd_reclaim+0x34 #6 0xffff000000a1340c at VOP_RECLAIM_APV+0x2c #7 0xffff0000005fd6c0 at vgonel+0x450 #8 0xffff0000005fde7c at vrecycle+0x9c #9 0xffff000001455678 at zfs_freebsd_inactive+0x48 #10 0xffff0000005fc430 at vinactivef+0x180 #11 0xffff0000005fba50 at vput_final+0x200 #12 0xffff00000060c4d0 at kern_funlinkat+0x320 #13 0xffff00015d6cbbf4 at filemon_wrapper_unlink+0x14 #14 0xffff0000008f8514 at do_el0_sync+0x594 #15 0xffff0000008d4904 at handle_el0_sync+0x40 The system (Windows Dev Kit 2023) has 32 GiBytes of RAM. Example output from a top modified to show some "Max[imum]Obs[erved]" information: last pid: 17198; load averages: 0.33, 0.58, 1.06 MaxObs: 15.49, = 8.73, 5.75 = up 0+20:48:10 19:14:49 426 threads: 9 running, 394 sleeping, 1 stopped, 22 waiting, 50 = MaxObsRunning CPU: 0.0% user, 0.0% nice, 0.2% system, 0.1% interrupt, 99.7% idle Mem: 282760Ki Active, 7716Mi Inact, 23192Ki Laundry, 22444Mi Wired, = 2780Ki Buf, 848840Ki Free, 2278Mi MaxObsActive, 22444Mi MaxObsWired, = 22752Mi MaxObs(Act+Wir+Lndry) ARC: 11359Mi Total, 3375Mi MFU, 5900Mi MRU, 993Mi Anon, 93076Ki Header, = 992Mi Other 8276Mi Compressed, 19727Mi Uncompressed, 2.38:1 Ratio Swap: 120832Mi Total, 120832Mi Free, 2301Mi MaxObs(Act+Lndry+SwapUsed), = 22752Mi MaxObs(Act+Wir+Lndry+SwapUsed) The poudriere bulk has 8 builders but has ALLOW_MAKE_JOBS=3Dyes without any explicit settings for the likes of MAKE_JOBS_NUMBER . So it is a configuration that allows a high load average compared to the number of hardware threads (here: cores) in the system. I've rebooted to do a test with filemon not loaded at the time (here it was loaded from prior buildworld buildkernel activity). We will see if it still ends up with such problems. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Sun Apr 30 07:50:02 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q8JRH4mMwz46SFV for ; Sun, 30 Apr 2023 07:50:19 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic304-23.consmr.mail.gq1.yahoo.com (sonic304-23.consmr.mail.gq1.yahoo.com [98.137.68.204]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q8JRG4r8yz3rwQ for ; Sun, 30 Apr 2023 07:50:18 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=PaHrZaCX; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.204 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1682841016; bh=YOoe200HacnbBfywZapJrkgkLiUxWYnsNRKevxbTQFE=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=PaHrZaCXD6R3a/v29DM8qhgEVswwDpJ1tf7Q0ZY727MpRhUxdIzRdzvU4psbPpq/MDsGTXUi+kRMtkML7s2Lb1O3A1srOdhc/L8mL4t75mhHt36L4FPWydQ2e2f1E66TW3SXAolXP2lYnoOeP4bfH2fQ6TDD65MwBEDqLZnFmmhFwvhBGYuBDHV+a/JqJrlYUyhN/netIN4eweJy9OQXEaaJzJp2GJQgsp+YZYgcfB0mRbe9GiPI5okaOxiHwPaL3I4lZBzgP71T0qc2Zwx0uosGyyKvtJToqUNGYWbEL1su94QlLfVrCEcH6M4B88s+ANwtEVLXEJCuvk17RC1aSQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1682841016; bh=kZyvH3ygK1SqV9irDpKYtA84HCwKxKi8/5CQ/A21vWw=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=UrV7huuq2E2OqFhMPhxSHbKiEgOHcfY4h5RFyUiI8ruc24lxmDLbTtDJtp27KEnTTgilvdD1tR3mwa44PoHpa9D70XDGqVSOLMgeF8kbNVHtQVFjjLTcJRCY0K+S9FyTZ9x0ps/x8UDibtlnEt59GrDHVthkrBFA+zCQEY1yoljbF/mITzgDHk7C0/iAgm3vsn2LiZqiEGa/f8Sg3xeOEWc1nFUXYhdV3AKa2UGeSARRrsDnYaCYLBmF5H7b6nNjul1e6BhXs7j0nxIowbeWNPOxDbOheVe0G8Rtoqiv3PrJaLbkXivIZBlRhnp36aU6HCSMoaWxWoQhVy9wdow5Mw== X-YMail-OSG: mFkGbdQVM1lGhrCb7qxlbOZLSA6BJWpCIud8W4te6Bs1gQMNGWKNURFKPekmKzF wt4d6suy0czfJxsuj8.kjepuThLnF8z2tonJ8uyEs6js1MOIr.rKotmq8mtzB3dSaTw_ztRnSgZi hH9tU0fv1IZwu8S96JyiUZwXalMguT58FnhRMiNl4J_mZMFjqV..n1CxC5pzP5liKuVq4tcLytm9 Ks8yFGKa3NHufmvnhCyCvwxEzzbpFNotijYBvvRcdobl0CoACioV8PQfzX6gDP8.uDISoP_T8WyK gcmX3qKG3iMC7GspCK8t.tFziFgs0e4jvNrt2hBu52MKKiYGRy0qJiFrVPK_8_GcRMpVlEUWUvqV 75vAeKx.KsF.fOIosXCq7HeHvZwaM8EpyxrrD4fDakZ.JB7.x8k.H3BbxPWAhT3a8.Md57uvhEij 9b6PKiGB2J5Dp60X.qO6IkxLCekbAb.umq3ZgaqeEOonhIx_zVKn63wmaDaNLfoB4_JmvyHttS9N t743X.EjC9tza1U0KS6PfvIU7SkCpcl.DLisdL06F8hVGuacwJ.WnjM2f_Pp22PJ4qa7SGEVpEGO 0eXT0I7zRtvO0wOQYBv.6pKte7JXqOH61RrFYEGQkVlqIOIDx7KrFLHSVgmjrxzsiBqxSjNiprxS culvrfIHC5VgobxFFUQUPWNSejFGPKUTmFFcrTOGaR_Z8ZEGT0GYfpfLRr3Mkyn6pWRnvMRwpeEW C6vXKiBx9n37I8IOkEfR4R9_hPNLbZ9nu4LdoJ6CxmWN9jJkwSTtspdyQKqj65k6xGxEyN51IfOB YJd.WNf5JDsh18G1VYmmW.cmAmBZ0P8MzkCUVMpIcJRbvbxLdow6NiWAn0XwgQsHefU0BxF7goXc RgRsVYl6543a1NUqM1cPabD2m5g4M1P30D6oCrmT2dturdxy_hYBIXJmDqX8dfqENejASGrDW2oI mxZt54AXGEGxx3J.YjG46BxFGE.oUWbh57MdulcGGrF07g7ydIQH4lOuwtVDH6mIVji1xA52Te8O tRQwDZiLjR8Wy68IDm2a7F0mLIAHPk0br8ysmzGmO2ZjHQcxUaFQ_1LdkJ0euNrnzOtZ6VL70n0X ZkFf4XzQH8EGyjGCdDz1rHjm8nmp3LEoT42WJTiS0nvHXEYzFRzPK73E_Ms0pf_ssEuVz.cp7zKq sMTy03DTpljtrVR1jachMnxOzh5W3zNUCQVpnDtkxTHV4moaOBonEZJxmaCUaXxBqGjEz55hoWP7 JWvi7qJQAwP5_YA1.YAT6bliJO4TP5pgRwIUpRbhYHdzCgf1bhNmNqh13dXqEsU9UU56HvWWRF._ H204DlcetlspUjx0Zfna0fYsgqyOlM7_yvBpzQvw3P7ycdtwmpmt0d7QyEwekV_h.53.QTd6_rU2 bOkpOdr6H7QI7Bebe5MdQ_..BGScau1h21akZ8NNDJ4YDCeRheOlqL6lw14Xz06zmYxxO9Tjnt1Y zIIubZmZ47ROzx_qmJ4J_Kbwjp1B1Wlht0eXb1LmIYsoxFJwmNMel2LNoBn8mwf8z9t1c0f5oma9 p1Xz0wpSLXwcwWJTnRGREpdwO7hKLxdzIoYLzyJS2G4OjE6JH6LxL0ZVxl8aKrb2Xfhfg3Ri_4t0 t_yIhIoFUuDSkO._LU_zfcGq.L.FItrdycFBKHy79RaFKwqoonTskuWY5Xx0sObMPvjMID975Fx. uaRNRidePyHS6j7WPzF2qRBpgaPFIW96lr0Sgu8CHahsr_tXPOpw1pWYiF6n1GLDRoc6PgPmbLwf aNrULHi2StD3OPp6NfVj39a5w5S1RLHHQ7qTLjagqVX1Xp1oB5rg8ewipDciP4oJh5gAMsCITc_v crTklNHCRN.H2iA0jlFPKu3ZTj5a_SlA42FCrCv4L4OvJZ_k2uoGX65yKuaKqQnVK.Lqj68gH37p n_snsMUb9AGoitdQn6wETWTkE6Hekd.q2V9TDlkdioh8EDmQ7G4ZQkQP6yuK57cO8oDIjMU7pXpI 3tAEXSImaS_D3VZG6ot0hNKB1fdCmK3nK7SSzth..8sfIiQ2r0g8K6vt8j5jL3oCXIidsomEwv2g gGhiv3dtn5FyM.k49ocpjjX15_X7lYVrW_9G17XI4a8Lwh025pC63AzgaJw4pQ.uIeoqQQ6L9kPp X8hgpqiLgYY6_.AVSBmD6EOczkh.2GJ1U2kmyBto4KK2DFCWtqSG_tsunF3Mj.fgBaS.EjoGHxTi Xsd8fPQ1IYU7MJ1MRRjxpz7lE597NRAmmhktbNyCq1N2caVodPhHbWHq3z4rMn8WpNe4J6McAVsV Bwh4- X-Sonic-MF: X-Sonic-ID: 110a1bb2-13c1-4169-bc79-053d31d03f1a Received: from sonic.gate.mail.ne1.yahoo.com by sonic304.consmr.mail.gq1.yahoo.com with HTTP; Sun, 30 Apr 2023 07:50:16 +0000 Received: by hermes--production-gq1-546798879c-qtf56 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 887f2c2aaf197c4439425249f4901f60; Sun, 30 Apr 2023 07:50:13 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: armv8.2-A+ tuned FreeBSD kernels vs. poudriere bulk and USB3 media: tx->tx_quiesce_done_cv related blocking of processes? Date: Sun, 30 Apr 2023 00:50:02 -0700 References: <7AE28A5B-109E-4C26-9D70-BCA5D49CD79D@yahoo.com> To: FreeBSD Hackers , freebsd-arm In-Reply-To: <7AE28A5B-109E-4C26-9D70-BCA5D49CD79D@yahoo.com> Message-Id: <02DC03AE-E082-4FB5-AA0D-396F64CC23CB@yahoo.com> X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-2.44 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.94)[-0.944]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.204:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_TLS_LAST(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.204:from] X-Rspamd-Queue-Id: 4Q8JRG4r8yz3rwQ X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N On Apr 29, 2023, at 19:44, Mark Millard wrote: > This is based on: main-n262658-b347c2284603-dirty, b347c2284603 > being from late Apr 28, 2023 UTC. (The "-dirty" is from some > historical patches that I use.) The build is a non-debug build > (but with symbols not stripped). World or Kernel had been built > using: >=20 > -mcpu=3Dcortex-a78C+flagm+nofp16fml >=20 > just for testing purposes. (Worked nicely for -j8 buildworld > buildkernel testing for the 4 cortex-a78c's plus 4 cortex-x1c's > present.) >=20 > Monitoring poudriere bulk related activity via top and gstat -spod > I see a lot of the odd result of one process doing something > like: >=20 > CPU4 4 1:39 99.12% /usr/local/sbin/pkg-static create -f tzst -r = /wrkdirs/usr/ports/devel/cmake-core/work/stage >=20 > while other processes sit in the likes of: >=20 > tx->tx > zcq->z > zcw->z > zilog- > select > wait >=20 > But sometimes there is no CPU bound process and the top CPU process is > the likes of: >=20 > 1.24% [usb{usbus0}] >=20 > "gstat -spod" basically shows da0 dedicated to write activity most > of the time. >=20 > After: sysctl kern.tty_info_kstacks=3D1 > Then using ^T at various times, I see a lot of: >=20 > load: 0.48 cmd: sh 93914 [tx->tx_quiesce_done_cv] 7534.91r 11.06u = 22.66s 0% 3800k > #0 0xffff0000004fd564 at mi_switch+0x104 > #1 0xffff000000463f40 at _cv_wait+0x120 > #2 0xffff00000153fa34 at txg_wait_open+0xf4 > #3 0xffff0000014a40bc at dmu_free_long_range+0x17c > #4 0xffff000001448254 at zfs_rmnode+0x64 > #9 0xffff000001455678 at zfs_freebsd_inactive+0x48 > #10 0xffff0000005fc430 at vinactivef+0x180 > #11 0xffff0000005fba50 at vput_final+0x200 > #12 0xffff00000060c4d0 at kern_funlinkat+0x320 > #13 0xffff00015d6cbbf4 at filemon_wrapper_unlink+0x14 > #14 0xffff0000008f8514 at do_el0_sync+0x594 > #15 0xffff0000008d4904 at handle_el0_sync+0x40 >=20 > load: 0.34 cmd: sh 93914 [tx->tx_quiesce_done_cv] 7566.69r 11.06u = 22.66s 0% 3800k > #0 0xffff0000004fd564 at mi_switch+0x104 > #1 0xffff000000463f40 at _cv_wait+0x120 > #2 0xffff00000153fa34 at txg_wait_open+0xf4 > #3 0xffff0000014a40bc at dmu_free_long_range+0x17c > #4 0xffff000001448254 at zfs_rmnode+0x64 > #5 0xffff0000014557c4 at zfs_freebsd_reclaim+0x34 > #6 0xffff000000a1340c at VOP_RECLAIM_APV+0x2c > #7 0xffff0000005fd6c0 at vgonel+0x450 > #8 0xffff0000005fde7c at vrecycle+0x9c > #9 0xffff000001455678 at zfs_freebsd_inactive+0x48 > #10 0xffff0000005fc430 at vinactivef+0x180 > #11 0xffff0000005fba50 at vput_final+0x200 > #12 0xffff00000060c4d0 at kern_funlinkat+0x320 > #13 0xffff00015d6cbbf4 at filemon_wrapper_unlink+0x14 > #14 0xffff0000008f8514 at do_el0_sync+0x594 > #15 0xffff0000008d4904 at handle_el0_sync+0x40 >=20 > load: 0.44 cmd: sh 93914 [tx->tx_quiesce_done_cv] 7693.52r 11.24u = 23.08s 0% 3800k > #0 0xffff0000004fd564 at mi_switch+0x104 > #1 0xffff000000463f40 at _cv_wait+0x120 > #2 0xffff00000153fa34 at txg_wait_open+0xf4 > #3 0xffff0000014a40bc at dmu_free_long_range+0x17c > #4 0xffff000001448254 at zfs_rmnode+0x64 > #5 0xffff0000014557c4 at zfs_freebsd_reclaim+0x34 > #6 0xffff000000a1340c at VOP_RECLAIM_APV+0x2c > #7 0xffff0000005fd6c0 at vgonel+0x450 > #8 0xffff0000005fde7c at vrecycle+0x9c > #9 0xffff000001455678 at zfs_freebsd_inactive+0x48 > #10 0xffff0000005fc430 at vinactivef+0x180 > #11 0xffff0000005fba50 at vput_final+0x200 > #12 0xffff00000060c4d0 at kern_funlinkat+0x320 > #13 0xffff00015d6cbbf4 at filemon_wrapper_unlink+0x14 > #14 0xffff0000008f8514 at do_el0_sync+0x594 > #15 0xffff0000008d4904 at handle_el0_sync+0x40 >=20 >=20 > The system (Windows Dev Kit 2023) has 32 GiBytes of RAM. Example > output from a top modified to show some "Max[imum]Obs[erved]" > information: >=20 > last pid: 17198; load averages: 0.33, 0.58, 1.06 MaxObs: = 15.49, 8.73, 5.75 = up 0+20:48:10 19:14:49 > 426 threads: 9 running, 394 sleeping, 1 stopped, 22 waiting, 50 = MaxObsRunning > CPU: 0.0% user, 0.0% nice, 0.2% system, 0.1% interrupt, 99.7% idle > Mem: 282760Ki Active, 7716Mi Inact, 23192Ki Laundry, 22444Mi Wired, = 2780Ki Buf, 848840Ki Free, 2278Mi MaxObsActive, 22444Mi MaxObsWired, = 22752Mi MaxObs(Act+Wir+Lndry) > ARC: 11359Mi Total, 3375Mi MFU, 5900Mi MRU, 993Mi Anon, 93076Ki = Header, 992Mi Other > 8276Mi Compressed, 19727Mi Uncompressed, 2.38:1 Ratio > Swap: 120832Mi Total, 120832Mi Free, 2301Mi = MaxObs(Act+Lndry+SwapUsed), 22752Mi MaxObs(Act+Wir+Lndry+SwapUsed) >=20 >=20 > The poudriere bulk has 8 builders but has ALLOW_MAKE_JOBS=3Dyes > without any explicit settings for the likes of MAKE_JOBS_NUMBER . > So it is a configuration that allows a high load average compared > to the number of hardware threads (here: cores) in the system. >=20 >=20 > I've rebooted to do a test with filemon not loaded at the time > (here it was loaded from prior buildworld buildkernel activity). > We will see if it still ends up with such problems. It still ends up with things waiting, but the detailed STATE valiues generally lsited are somewhat different. I also tried a chroot into a world from use of -mcpu=3Dcortex-a72 and got similar results. Suggesting that only the -mcpu=3Dcortex-a78c+flagm+nofp16fml kernel is required to see the issues. This even got some examples like: load: 1.20 cmd: sh 95560 [tx->tx_quiesce_done_cv] 2022.55r 3.21u 9.24s = 0% 3852k #0 0xffff0000004fd564 at mi_switch+0x104 #1 0xffff000000463f40 at _cv_wait+0x120 #2 0xffff000001518a34 at txg_wait_open+0xf4 #3 0xffff00000147d0bc at dmu_free_long_range+0x17c #4 0xffff000001421254 at zfs_rmnode+0x64 #5 0xffff00000142e7c4 at zfs_freebsd_reclaim+0x34 #6 0xffff000000a1340c at VOP_RECLAIM_APV+0x2c #7 0xffff0000005fd6c0 at vgonel+0x450 #8 0xffff0000005fde7c at vrecycle+0x9c #9 0xffff00000142e678 at zfs_freebsd_inactive+0x48 #10 0xffff0000005fc430 at vinactivef+0x180 #11 0xffff0000005fba50 at vput_final+0x200 #12 0xffff00015d8ceab4 at null_reclaim+0x154 #13 0xffff000000a1340c at VOP_RECLAIM_APV+0x2c #14 0xffff0000005fd6c0 at vgonel+0x450 #15 0xffff0000005fde7c at vrecycle+0x9c #16 0xffff00015d8ce8e8 at null_inactive+0x38 #17 0xffff0000005fc430 at vinactivef+0x180 (The chroot use involves null mounts.) which is sort of analogous to the filemon related backtraces I showed earlier. The common part=20 across the examples looks to be #0..#11: #0 0xffff0000004fd564 at mi_switch+0x104 #1 0xffff000000463f40 at _cv_wait+0x120 #2 0xffff00000153fa34 at txg_wait_open+0xf4 #3 0xffff0000014a40bc at dmu_free_long_range+0x17c #4 0xffff000001448254 at zfs_rmnode+0x64 #5 0xffff0000014557c4 at zfs_freebsd_reclaim+0x34 #6 0xffff000000a1340c at VOP_RECLAIM_APV+0x2c #7 0xffff0000005fd6c0 at vgonel+0x450 #8 0xffff0000005fde7c at vrecycle+0x9c #9 0xffff000001455678 at zfs_freebsd_inactive+0x48 #10 0xffff0000005fc430 at vinactivef+0x180 #11 0xffff0000005fba50 at vput_final+0x200 There were a lot more nanslp examples in all the alter testing (i.e, those avoided filemon.ko being loaded). Starting from having pkgclean -A'd the ports, the experiments got about the same number of ports built as of the end of the 1st hour. UFS vs. ZFS? Different media types? . . . So I decided to create and try a UFS test context instead of a ZFS one. But the media that was best to update was a U2 960GB Optane in a USB3 adapter, something that would perform noticeably better than my normal USB3 NVMe drives, even with USB involved. This combination maintained reasonable load averages (instead of having long periods of <1) and finish building 172 ports in the 1st hour, far more than the around 83 each time I tried the other device/ZFS combination. NO evdience of the earlier reported oddities. I should also time a from-scratch buildworld buildkernel. I'll look into setting up another U2 960GB Optane for use in the USB3 adpater, but with ZFS. That should help isolate media vs. file system contributions to the varying behaviors. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Sun Apr 30 10:49:07 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q8NPj2998z46fcC for ; Sun, 30 Apr 2023 10:49:13 +0000 (UTC) (envelope-from void@f-m.fm) Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q8NPh0fcVz46fF for ; Sun, 30 Apr 2023 10:49:12 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm3 header.b=PQ3QMaus; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=CwApRzxi; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 64.147.123.21 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.west.internal (Postfix) with ESMTP id 893D5320076F for ; Sun, 30 Apr 2023 06:49:10 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Sun, 30 Apr 2023 06:49:10 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1682851750; x=1682938150; bh=E7 Y1D1OUopHPVS3dQP0pe0MwGe22U0+Kk15BPSnTga0=; b=PQ3QMausgT9qwqI9ZK wwVxknQvQo0Y0B0RjBBcUhiyR+wLHqVoBQ4bubJP0nrXd4xX3bj7oiq+DZZmiOic RaX/tmxhz/UiXflklz1XVn6lhas4byz3bF3UQCdbG+H9mmvnDHQI2O90i95yDemm dQ8p8ognozHH2MVP2yC2WkgWwuEAwPy1++thJK9C/VpjiYkI9+df7hklF9t4cyhn S1MtTfGl6x2qdHqGaLzX4KCcmpNChXRqop98C2kBxweA+SNwnNXoZgVq2xzbsvJt 9ynMaG2rjG2R96FiWwHI0Svh/ZPJOqBw0XkrE/aDesVIqBS20V/ILcqm31n6BoTv kKvg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1682851750; x=1682938150; bh=E7Y1D1OUopHPV S3dQP0pe0MwGe22U0+Kk15BPSnTga0=; b=CwApRzxiTbOhrINbJBuztO/gAyXYr G+Ea+HWVvZXTWrWcez6nj7yaTW1+B78vH0osexREXnHT2iFRQI6YDvJ6riz3+0uz WgPE9jcnFEWBPf+NUCys4QvOiscc/doHurDag4Ee2cz2buZWFwTzMoipZeV3yuOs fkQRYRw/OvjmDjzCx51cKWosen9RQYAT/MKaNXcfxbEYpLKUudZK3VJBrKO718CT psqTvHm5ByTTzPpdyWSD35cCcvaHN8N/TOcjG/3ta62sYp9I6gzTDfQmDVR4tast NwPsFFPMj/Z/036L0GHU/MqQ01T+7BZVOLnpPptrmkOfsF1CtrPgfeNaQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrfedvvddgfeefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggujgesthdtre dttddtvdenucfhrhhomhepvhhoihguuceovhhoihgusehfqdhmrdhfmheqnecuggftrfgr thhtvghrnhepkeeluddvlefhieelfefggffhffektdehleelgfdugfdvgeekjeejuddthe ehgfeunecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhep vhhoihgusehfqdhmrdhfmh X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Sun, 30 Apr 2023 06:49:09 -0400 (EDT) Date: Sun, 30 Apr 2023 11:49:07 +0100 From: void To: freebsd-hackers@freebsd.org Subject: Re: BHYVE_SNAPSHOT Message-ID: Mail-Followup-To: freebsd-hackers@freebsd.org References: <00edb622-5675-e165-83c8-6b73320f8aa9@madpilot.net> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <00edb622-5675-e165-83c8-6b73320f8aa9@madpilot.net> X-Spamd-Result: default: False [-3.49 / 15.00]; SUBJ_ALL_CAPS(1.05)[14]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.94)[-0.938]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; MID_RHS_NOT_FQDN(0.50)[]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.21]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm3,messagingengine.com:s=fm3]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.21:from]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[f-m.fm]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4Q8NPh0fcVz46fF X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On Sat, Apr 29, 2023 at 01:27:14PM +0200, Guido Falsi wrote: > >The bhyvectl(8) man page has some further information, read about the >--checkpoint and --suspend options. > >bhyve(8) has some details too (-r option). thank you! -- From nobody Sun Apr 30 11:07:16 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q8Nph0k7Pz46gYv for ; Sun, 30 Apr 2023 11:07:24 +0000 (UTC) (envelope-from void@f-m.fm) Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q8Npg1WN1z4CSt for ; Sun, 30 Apr 2023 11:07:23 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm3 header.b=f6BcBwuY; dkim=pass header.d=messagingengine.com header.s=fm3 header.b="d b0S9eY"; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 64.147.123.21 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id 555BD3200906 for ; Sun, 30 Apr 2023 07:07:22 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Sun, 30 Apr 2023 07:07:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-transfer-encoding:content-type:content-type:date:date :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm3; t= 1682852841; x=1682939241; bh=XsxVOpQcqLxbgaZIsXmFIHdpn0UjC8VTFQl eVXi8J+Q=; b=f6BcBwuY9k/D/W7iYPDmVwn+yQKw0SG1V3a+HdMbHcdoLE8q/Bi WH74AaGWXnih9g7QpYz7GYSUonWlF2BTCX9SO1qvGlssZPaAJl2Bd47d5zqpdv60 zdykqdKYsbHlp8F6b7wk64tN/4bdIkhupomL33bFTkTi+El3aM1WdxmOREMgbvJQ 5iZtTW/kO7/IyqiDQGtWn1jXsOkYazZxEs9E5aPI8TlJHgpUrHkgGFLAmNfwzBls lmHwBC3b+/2YQaHgxMuceA36ZDL/6GdfSCUS90bWSZPztKaBn9mazp1nG0+n3RUM SxdWTKqy+TYHll6ioo2X+4hf/yp4UjSbT/g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1682852841; x= 1682939241; bh=XsxVOpQcqLxbgaZIsXmFIHdpn0UjC8VTFQleVXi8J+Q=; b=d b0S9eYRlZdyjzjnOVPAKkFZznNvIJnbUpwppfqnbVk/GheK6wLIapT28HASRLnrm crG3F0IZ8QyBPu/sbwlmgZNaCxjJGhS8rU6IZcm2mUdzLSnyFF9KDEnvWlb4PBoA U6b4FNQHgTUkIXeUzQGMO2Y4QETeceTek548ZtYGdNm6OTKuAdI2No/V8m5j1OKZ RRBhe42LQV4CffcKpvHNglM2adO0Iu8rNKIA/eWNNeTjeQ3fVvNPqgA8Lds1B92t H0fWMha8+a/dB1Ncnr6yBcX3h7XglIIgRPSpHmwsAe65TlgTRzavxxd4VNdxOMwZ 8pXqOmgDHUVd8F/j5VxCA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrfedvvddgfeejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggugfgjsehtke ertddttdejnecuhfhrohhmpehvohhiugcuoehvohhiugesfhdqmhdrfhhmqeenucggtffr rghtthgvrhhnpeetudeghedtledvffeljedtkeevfeevtefgtdelkeelhfduudehgeelue dvveegieenucffohhmrghinhepfhhrvghshhhpohhrthhsrdhorhhgnecuvehluhhsthgv rhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepvhhoihgusehfqdhmrdhfmh X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Sun, 30 Apr 2023 07:07:21 -0400 (EDT) Date: Sun, 30 Apr 2023 12:07:16 +0100 From: void To: freebsd-hackers@freebsd.org Subject: Re: BHYVE_SNAPSHOT Message-ID: Mail-Followup-To: freebsd-hackers@freebsd.org References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Spamd-Result: default: False [-3.52 / 15.00]; SUBJ_ALL_CAPS(1.05)[14]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_SHORT(-0.97)[-0.966]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; MID_RHS_NOT_FQDN(0.50)[]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.21:c]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm3,messagingengine.com:s=fm3]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.21:from]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[f-m.fm]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4Q8Npg1WN1z4CSt X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On Sat, Apr 29, 2023 at 01:29:23PM +0200, Tomek CEDRO wrote: >On Sat, Apr 29, 2023 at 1:17 PM void wrote: >> Where can I read up about the recently introduced BHYVE_SNAPSHOT ? > >Have you tried vm-bhyve? This is really simple frontend to >create/manage/snapshot bhyve vms.. it is using dedicated zfs poll for >vm storage and uses zfs snapshot for the vm snapshot :-) > >https://www.freshports.org/sysutils/vm-bhyve/ Thanks for the suggestion. I've lookled at vm-bhyve in the past but found that manually running vms from a simple script in say screen(8) easier esp. if something breaks. -- From nobody Sun Apr 30 19:26:50 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q8bvJ2Wqwz48CBB; Sun, 30 Apr 2023 19:27:08 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q8bvJ22bwz3xs3; Sun, 30 Apr 2023 19:27:08 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1682882828; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=BTkUC5FW5h8qNu1OxzqdOP6/aFDPedn2xnRR/tJt8M4=; b=e4fSeKGwREL2OrFRIEpXdGGRo3PG3eubKr2ia3GCSL4mvbsGz5XjG6cM9kZWGdZfsNk77n Xln9qHPfoTyB/gUuPKVp7NRT7MCOTRKwq3AKzvBi+gH+CcdothUV9onYQ5Q19bAmlXr2kK J3FsO9NLCtHNVmLAy0XWj9urQZBX8D4eP3X+9Aocpgflbd9Hil9TwlZA68lCzldoC0cEiO 4Hc96PyYxTB2dgcqYywL2yHsf1AkQg2SBXdDEJxsUayxUt3lZRwKvzutpFeW/Bi5o/1znz unXqml4kZKmZy3FOH0i8UUSn9ExO6kolPefe4yfaPERYKUN11oqnRwllx9Dr8g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1682882828; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=BTkUC5FW5h8qNu1OxzqdOP6/aFDPedn2xnRR/tJt8M4=; b=FkI0xe8m/zyyH5PxJQ0yiBUFPAoGxkgVt7tODdhy19DT3KYQxpbTilsYPIraP60ajGhZLU Vj4L0dpW8JyyDOTKTqhzrlNtXOiV9Ah6kry2omBVtLxGUTf73YJEJy3gK+zvxFaiiw8wTt ge6FcIXvO9+IaGMeKf3bM9DAVLQTOWAR+9w2bUVT4LvacD4B4eZ/3YYXQo0J1L48amwaxj XFeo2lLsfpL5pIUOUEIDj/LvECiDTHiVdqqDZ3CJiXp4/k6vnKDOTxjN32Nn1NACOW4TUb 5owF6bgd3dut4gkCUXYb+JdLZPuxUITvxkQyRpCZQBzdNSJ8FN0tL6GcN1gKDQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1682882828; a=rsa-sha256; cv=none; b=lHxe+Yl3loE9bIXt9M+ilRfmn/fBEGt0eUbIXlogkWc/SBnop9RvIT8SWduiSPQXC2C4LK 2xBILJ6p3VS882YVHlp5ZQfmcf1OW1f1/c8UoIrmQRviqL0XMufpylz77IASmkUMZ3UJCP bH8cQU2zM+W0HXnloa3mG+fUK2R6GChlSs8XTCVE6vDVDXm8ctXxGPxaNlEuZnlKp7MyAh IeeySJdK1Jr9wYkMwFBtGltLRRGpFJtlFrzN+oZ9B0xbL1rHQ8zr6vM+kyd6S7LwqBp/8c aobZrOIq+vmr2KMWUEKzu227i9klprcvqmENUBIFiTew/SPoV+1DKlqtoHvAuA== Received: from [IPV6:2001:470:1f1c:a0::2] (tunnel642390-pt.tunnel.tserv1.lon2.ipv6.he.net [IPv6:2001:470:1f1c:a0::2]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: grahamperrin) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Q8bv14l1KzP5b; Sun, 30 Apr 2023 19:26:53 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) Message-ID: <226afa3a-114b-6740-36b1-ebfee4931a22@freebsd.org> Date: Sun, 30 Apr 2023 20:26:50 +0100 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.10.1 Subject: FreeBSD #ai-ml (was: How do I get a new freebsd- mailing list made) Content-Language: en-US From: Graham Perrin To: FreeBSD questions , FreeBSD hackers References: Organization: FreeBSD In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------DgEnRaHtCjPhVpkI5c8BRBjr" X-ThisMailContainsUnwantedMimeParts: N This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------DgEnRaHtCjPhVpkI5c8BRBjr Content-Type: multipart/mixed; boundary="------------8nlAcy0Hzh21SyajiGQ7hT4B"; protected-headers="v1" From: Graham Perrin To: FreeBSD questions , FreeBSD hackers Message-ID: <226afa3a-114b-6740-36b1-ebfee4931a22@freebsd.org> Subject: FreeBSD #ai-ml (was: How do I get a new freebsd- mailing list made) References: In-Reply-To: --------------8nlAcy0Hzh21SyajiGQ7hT4B Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 SSBjYW4gbm8gbG9uZ2VyIGZpbmQgcGFydHMgb2YgdGhpcyB0aHJlYWQgKHNvcnJ5KS4NCg0K Rm9yIHRob3NlIG9mIHlvdSB3aG8gYXJlIGludGVyZXN0ZWQsIHRoZXJlJ3MgYW4gI2FpLW1s IGNoYW5uZWwgaW4gDQpEaXNjb3JkIGZvciBGcmVlQlNEOg0KDQo+ICJBcnRpZmljaWFsIElu dGVsbGlnZW5jZSIgLSAiTWFjaGluZSBMZWFybmluZyIgLSBhbmQgcmVsYXRlZCBjb21wdXRl ciANCj4gc2NpZW5jZXkgdGhpbmdzLg0KDQo= --------------8nlAcy0Hzh21SyajiGQ7hT4B-- --------------DgEnRaHtCjPhVpkI5c8BRBjr Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsF5BAABCAAjFiEEWT/lssMHB+28ly8Kt2dIb0oY1AsFAmROwPoFAwAAAAAACgkQt2dIb0oY1AsO 0BAAnm37CZzPqTeDtk8Dc8aTZu2NA0RFgqaSHgSzyyTDAzobD7eNhuGTgxfb2k29GIgv7hQoYVzE 0Vu2LE9t1xORgrAezb5E2eFzuu4W/EanTYhOdZAxq1hiHBv0SyrCBNzZIo1B6y4H4EjTyBu1V3Hm x+3XEfV6EvJWCwi9aaFwommWVbG/38ew/y0MvWoLZ5Qcp4MuMgFm6bgaNVGQY0SD1DlS/s3uK7JQ 5LRna/hVLCmKu02rpJqrfwj4ABd1QTNIFeCoK415Y9o+9JtC4yXCDwCmAs82exWVI5Ovrj87sAlG vaR3pkSWBAH3fE73UIprAETWaznKjKQlJDiGN+3eC0fliLkCkvLBqkKJnKjxbpFYKs4wnBlqtshH WTca2iWyX4R3b5IzsNYWZatfnVHTWBCMkGcPhlEITe+NmvwkcCssKgoxPjvKP+UJ2FZh4BXj1lLw QkkRT2l0NSUMVX5+O28TugMjNm3zqN57j+G056ostfCFnMjggi6C02y80VxN9I2+Dy0YrwIFEer+ kOhhGmtfn5h3DzVAoj2rILWk0rHxvmRyaFIr909ZNOkcIdjo2SF5XdoUn/ry5/46/8fW7/sU88xx V2VYfkSw6Gyf2il+qzh0BMdxcDYpCJsEySgX31MtdFvRgJSp+OR4X+3cQwqKgZz4mdjHmjBuwLTB wVY= =CNrL -----END PGP SIGNATURE----- --------------DgEnRaHtCjPhVpkI5c8BRBjr-- From nobody Sun Apr 30 21:41:36 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q8ftk3KNrz48LLQ; Sun, 30 Apr 2023 21:41:50 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx1.shrew.net (mx1.shrew.net [38.97.5.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q8ftj1lVYz4GWP; Sun, 30 Apr 2023 21:41:49 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 38.97.5.131 as permitted sender) smtp.mailfrom=mgrooms@shrew.net; dmarc=none Received: from mail.shrew.net (mail.shrew.prv [10.24.10.20]) by mx1.shrew.net (8.15.2/8.15.2) with ESMTP id 33ULfgOY067025; Sun, 30 Apr 2023 16:41:42 -0500 (CDT) (envelope-from mgrooms@shrew.net) Received: from [10.22.200.32] (unknown [136.49.230.220]) by mail.shrew.net (Postfix) with ESMTPSA id 645C2192C40; Sun, 30 Apr 2023 16:41:37 -0500 (CDT) Message-ID: Date: Sun, 30 Apr 2023 16:41:36 -0500 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.10.0 Subject: Re: BHYVE_SNAPSHOT Content-Language: en-US To: freebsd-hackers@freebsd.org, freebsd-virtualization@freebsd.org References: From: Matthew Grooms Cc: elenamihailescu22@gmail.com, Mihai Carabas , gusev.vitaliy@gmail.com In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (mx1.shrew.net [10.24.10.10]); Sun, 30 Apr 2023 16:41:42 -0500 (CDT) X-Spamd-Result: default: False [1.40 / 15.00]; SUBJ_ALL_CAPS(1.05)[14]; NEURAL_HAM_LONG(-1.00)[-0.998]; NEURAL_SPAM_SHORT(0.86)[0.859]; NEURAL_SPAM_MEDIUM(0.79)[0.789]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org,freebsd-hackers@freebsd.org]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:174, ipnet:38.0.0.0/8, country:US]; R_DKIM_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_FIVE(0.00)[5]; FREEMAIL_CC(0.00)[gmail.com]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; TO_DN_SOME(0.00)[]; DMARC_NA(0.00)[shrew.net]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4Q8ftj1lVYz4GWP X-Spamd-Bar: + X-ThisMailContainsUnwantedMimeParts: N On 4/29/23 06:17, void wrote: > Hi, > > Where can I read up about the recently introduced BHYVE_SNAPSHOT ? > > I can only find terse information about what it does (in man 5 src.conf) > but not about how to use it and/or why. This began as sponsored student work with the Politehnica University of Bucharest back in 2016 with an eye toward developing Live MIgration as a feature. The initial bhyve save/restore patch was committed back in 2020 ... https://reviews.freebsd.org/rS360648 Student projects related to bhyve continued until last year, but we gave up due to there being such a massive accumulation of patches with no path to getting them committed. We tried reaching out to project maintainers and the FreeBSD foundation alike for support and feedback, but there seems to be little to no interest. There have also been attempts by companies that rely on these features to and get them improved and committed. Unfortunately they don't appear to be having much luck either ... https://reviews.freebsd.org/D38858#885651 Would you like to see support for VM snapshots in the generic kernel? How about support for saving/restore checkpoints using QCOW2, VMDK via libvdsk? How about support for warm or live migration? How about USB device pass-through? There are experimental patches for all these features that were developed by students at UPB. In a lot of cases, there are open reviews that have been waiting on feedback for ages. Here is a a presentation given by Elena from UPB just last month at the FreeBSD devsumit in Tokyo ... https://wiki.freebsd.org/DevSummit/202303?action=AttachFile&do=view&target=Presentation+-+bhyvecon.pdf The case is quite plain. I'm not sure what the solution is to this problem. I'd love to hear feedback from the community about how I've got this completely wrong and how the course could be corrected. That would be something. -Matthew From nobody Sun Apr 30 23:35:07 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q8jPV4cfcz48SC6 for ; Sun, 30 Apr 2023 23:35:10 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-ot1-x32f.google.com (mail-ot1-x32f.google.com [IPv6:2607:f8b0:4864:20::32f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q8jPT4YxZz4Pwh for ; Sun, 30 Apr 2023 23:35:09 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=TaGlBzn+; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2607:f8b0:4864:20::32f as permitted sender) smtp.mailfrom=mjguzik@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ot1-x32f.google.com with SMTP id 46e09a7af769-6a604259983so1506091a34.2 for ; Sun, 30 Apr 2023 16:35:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1682897708; x=1685489708; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=9ywG0XYiJPtwWFDD4GO6ZBN275lbO81bD8z1yD0mAKI=; b=TaGlBzn+Gy0AkIm55Q7dXrh7KxSeE60nK51eDQS0oee6u3lXoeikqlbPIZB3ABLYw4 o3E3qkrwClOGLycOOQbnlO0eFl1MnnmBz3XgwAdpNAPBtxsJjxlvjyaGWi0uaeG9zto+ YiNvQt1dzfeIO5HFXoeQtrdBB+F+KaXVKtTT7groIFA62uQ2wW7pVktsRekWIcmPZBpo jgozJqeqryk3j4nT9vMm27GDjvD+0Dg3/Yovsdo00FQpi0O8HUHsDXW3IBSl7/C7saxy keGxHYVG5LCgPBZ1xeaux5R90OrKboeZU2nHYCnMiiQZF37MFlLMCB/RfHRvuZqUKYDF QMsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682897708; x=1685489708; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=9ywG0XYiJPtwWFDD4GO6ZBN275lbO81bD8z1yD0mAKI=; b=Ll0d7FGOwask3HJ18eJRZujbrtjHSZzRQTEHVZ/OiSc23dKeWFy/aRUXcKFZPCYdKU psv3KrDswZLlCYMvzJyLlnYcqF22dTE/tdQCm4nPyRcyqJDXF+puujcikCoAWxLq0eUJ 2FOQXfgT7AP8QiJFV3P7GiJfos0fvQlemAyXR3JIobIqUYUCvVlq0l3/fcdv1HZtAhq8 rP0+dU0GWxM/70N9rY4bJs8vXBIn7mFF7Dkih01sTwc2I6+FAzDgdkUPKVcnzIMuhK47 PaHFHsmzvPkQXDoblOue7AAYzR4QwA6KunC1ln5O5JEH/N6EJbYjfhhbQL4ru+qKLMJI Z0lQ== X-Gm-Message-State: AC+VfDx3eWylEpJQLWB4SPihcv+1rNhAH5rTDA3YIfOJUaoLiY70CRHF uu+x5b9aljZRVX+Q1sStEQGY7YYXqFRh+A6BvipHGOUh X-Google-Smtp-Source: ACHHUZ5oLMv3JcVDDOhQLlwZJvCmj64aPUQOSI1E71mgJq4lcpCvOZdUGbCbnGNNbZfyGYBNxzsL+5IkGaLznb2EoF4= X-Received: by 2002:a54:4482:0:b0:38e:eaf:cf1f with SMTP id v2-20020a544482000000b0038e0eafcf1fmr5418907oiv.44.1682897707988; Sun, 30 Apr 2023 16:35:07 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:a8a:1194:0:b0:4d4:94b:7266 with HTTP; Sun, 30 Apr 2023 16:35:07 -0700 (PDT) In-Reply-To: <20230331215751.166a294f7382c85b545f53a2@dec.sakura.ne.jp> References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> <20230331215751.166a294f7382c85b545f53a2@dec.sakura.ne.jp> From: Mateusz Guzik Date: Mon, 1 May 2023 01:35:07 +0200 Message-ID: Subject: Re: Periodic rant about SCHED_ULE To: Tomoaki AOKI Cc: freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-2.98 / 15.00]; URI_HIDDEN_PATH(1.00)[https://people.freebsd.org/~mjg/.junk/ule-poc-hacks-dont-use.diff]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.980]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::32f:from]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4Q8jPT4YxZz4Pwh X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N On 3/31/23, Tomoaki AOKI wrote: > On Mon, 27 Mar 2023 16:47:04 +0200 > Mateusz Guzik wrote: > >> On 3/27/23, Mateusz Guzik wrote: >> > On 3/25/23, Mateusz Guzik wrote: >> >> On 3/23/23, Mateusz Guzik wrote: >> >>> On 3/22/23, Mateusz Guzik wrote: >> >>>> On 3/22/23, Steve Kargl wrote: >> >>>>> On Wed, Mar 22, 2023 at 07:31:57PM +0100, Matthias Andree wrote: >> >>>>>> > > (snip) > >> > >> > I repeat the setup: 8 cores, 8 processes doing cpu-bound stuff while >> > niced to 20 vs make -j buildkernel >> > >> > I had a little more look here, slapped in some hacks as a POC and got >> > an improvement from 67 minutes above to 21. >> > >> > Hacks are: >> > 1. limit hog timeslice to 1 tick so that is more eager to bail >> > 2. always preempt if pri < cpri >> > >> > So far I can confidently state the general problem: ULE penalizes >> > non-cpu hogs for blocking (even if it is not their fault, so to speak) >> > and that bumps their prio past preemption threshold, at which point >> > they can't preempt said hogs (despite hogs having a higher priority). >> > At the same time hogs use their full time slices, while non-hogs get >> > off cpu very early and have to wait a long time to get back on, at >> > least in part due to inability to preempt said hogs. >> > >> > As I mentioned elsewhere in the thread, interactivity scoring takes >> > "voluntary off cpu time" into account. As literally anything but >> > getting preempted counts as "voluntary sleep", workers get shafted for >> > going off cpu while waiting on locks in the kernel. >> > >> > If I/O needs to happen and the thread waits for the result, it most >> > likely does it early in its timeslice and once it's all ready it waits >> > for background hogs to get off cpu -- it can't preempt them. >> > >> > All that said: >> > 1. "interactivity scoring" (see sched_interact_score) >> > >> > I don't know if it makes any sense to begin with. Even if it does, it >> > counts stuff it should not by not differentiating between deliberately >> > going off cpu (e.g., actual sleep) vs just waiting for a file being >> > read. Imagine firefox reading a file from disk and being considered >> > less interactive for it. >> > >> > I don't have a solution for this problem. I *suspect* the way to go >> > would be to explicitly mark xorg/wayland/whatever as "interactive" and >> > have it inherited by its offspring. At the same time it should not >> > follow to stuff spawned in terminals. Not claiming this is perfect, >> > but it does eliminate the guessing game. >> > >> > Even so, 4BSD does not have any mechanism of the sort and reportedly >> > remains usable on a desktop just by providing some degree of fairness. >> > >> > Given that, I suspect the short term solution would whack said scoring >> > altogether and focus on fairness (see below). >> > >> > 2. fairness >> > >> > As explained above doing any offcpu-time inducing work instantly >> > shafts threads versus cpu hogs, even if said hogs are niced way above >> > them. >> > >> > Here I *suspect* position to add in the runqueue should be related to >> > how much slice was left when the thread went off cpu, while making >> > sure that hogs get to run eventually. Not that I have a nice way of >> > implementing this -- maybe a separate queue for known hogs and picking >> > them every n turns or similar. >> > >> >> Aight, now that I had a sober look at the code I think I cracked the >> case. >> >> The runq mechanism used by both 4BSD and ULE provides 64(!) queues, >> where the priority is divided by said number and that's how you know >> in which queue to land the thread. >> >> When deciding what to run, 4BSD uses runq_choose which iterates all >> queues from the beginning. This means threads of lower priority keep >> executing before the rest. In particular cpu hog lands with a high >> priority, looking worse than make -j 8 buildkernel and only running >> when there is nothing else ready to get the cpu. While this may sound >> decent, it is bad -- in principle a steady stream of lower priority >> threads can starve the hogs indefinitely. >> >> The problem was recognized when writing ULE, but improperly fixed -- >> it ends up distributing all threads within given priority range across >> the queues and then performing a lookup in a given queue. Here the >> problem is that while technically everyone does get a chance to run, >> the threads not using full slices are hosed for the time period as >> they wait for the hog *a lot*. >> >> A hack patch to induce the bogus-but-better 4BSD behavior of draining >> all runqs before running higher prio threads drops down build time to >> ~9 minutes, which is shorter than 4BSD. >> >> However, the right fix would achieve that *without* introducing >> starvation potential. >> >> I also note the runqs are a massive waste of memory and computing >> power. I'm going to have to sleep on what to do here. >> >> For interested here is the hackery: >> https://people.freebsd.org/~mjg/.junk/ule-poc-hacks-dont-use.diff >> >> sysctl kern.sched.slice_nice=0 >> sysctl kern.sched.preempt_thresh=400 # arbitrary number higher than any >> prio >> >> -- >> Mateusz Guzik > > Thanks for the patch. > Applied on top of main, amd64 at commit > 9d33a9d96f5a2cd88d0955b5b56ef5058b1706c1, setup 2 sysctls as you > mentioned and tested as below > > *Play flac files by multimedia/audacious via audio/virtual_oss > *Running www/firefox (not touched while testing) > *Forcibly build lang/rust > *Play games/aisleriot > > at the same time. > games/aisleriot runs slower than the situation lang/rust is not in > build, but didn't "freeze" and audacious normally played next music on > playlist, even on lang/rust is building codes written in rust. > > This is GREAT advance! > Without the patch, compiling rust codes eats up almost 100% of ALL > cores, and games/aisleriot often FREEZES SEVERAL MINUTES, and > multimedia/audacious needs to wait for, at worst, next music for FEW > MINUTES. (Once playback starts, the music is played normally until it > ends.) > > But unfortunately, the patch cannot be applied to stable/13, as some > prerequisite commits are not MFC'ed. > Missing commits are at least as below. There should be more, as I > gave up further tracking and haven't actually merged them to test. > > commit 954cffe95de1b9d70ed804daa45b7921f0f5c9da [1] > ule: Simplistic time-sharing for interrupt threads. > > commit fea89a2804ad89f5342268a8546a3f9b515b5e6c [2] > Add sched_ithread_prio to set the base priority of an interrupt > thread. > > commit 85b46073242d4666e1c9037d52220422449f9584 [3] > Deduplicate bus_dma bounce code. > > > [1] > https://cgit.freebsd.org/src/commit/?id=954cffe95de1b9d70ed804daa45b7921f0f5c9da > > [2] > https://cgit.freebsd.org/src/commit/?id=fea89a2804ad89f5342268a8546a3f9b515b5e6c > > [3] > https://cgit.freebsd.org/src/commit/?id=85b46073242d4666e1c9037d52220422449f9584 > > -- > Tomoaki AOKI > > Hello everyone. I sorted out a patch I consider comittable for the time being. IT IS NOT A PANACEA by any means, but it does sort out the most acute problem and should be a win for most people. It also comes with a knob to turn it off. That said, can you test this please: https://people.freebsd.org/~mjg/ule_pickshort.diff works against fresh main. if you are worried about recent zfs woes, just make sure you don't zpool upgrade and will be fine. -- Mateusz Guzik From nobody Mon May 1 00:04:22 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q8k3S5J8zz48V6Y for ; Mon, 1 May 2023 00:04:36 +0000 (UTC) (envelope-from tomek@cedro.info) Received: from mail-yw1-x1136.google.com (mail-yw1-x1136.google.com [IPv6:2607:f8b0:4864:20::1136]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q8k3S3GzNz4TFR for ; Mon, 1 May 2023 00:04:36 +0000 (UTC) (envelope-from tomek@cedro.info) Authentication-Results: mx1.freebsd.org; none Received: by mail-yw1-x1136.google.com with SMTP id 00721157ae682-55a44a2637bso7225567b3.2 for ; Sun, 30 Apr 2023 17:04:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cedro.info; s=google; t=1682899475; x=1685491475; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=WS/+SmDg9E6kfzNV+E23WyBQMGjpljvUh1xy/4KnyDI=; b=h2eawhLNv5F7oRQ3+NSJTcOyP7n0JVkWgTx83uOv0UZ5mXdZY85Y3DMRHUuQUGdF0a qG7aBWbATHaKGjBO6ZT65kIA/+uwkNbqCUmWR/nwcwEHzbDDL3/qedybKjqRUt8+/cOL AdNuW8dEwVeC0GIlruIWN/VaaA2EAtqR+8M9qBevKuFA2IvWlHpYtrdbbjMoYFa4e9Tx cFi/Bgv+V6HnHNUPBTzrOGdIaCQDIEP50IIM2tcGGwWXD2dIxuYSmr3fa43cj1o4nqr1 icisuaAysV/wXYhzUFdrzvbqE0yJlztIiZqQ8tI9ubhrKhHQD0+TXXLj/us0O3DHuFax y23g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682899475; x=1685491475; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=WS/+SmDg9E6kfzNV+E23WyBQMGjpljvUh1xy/4KnyDI=; b=AB6rHH6jyxOxZz2D2hXbR2DnB+/7qxlVNTd+7cJFXcpeiseazvs14yUBJNHkRUBi9s y8ji6ddKcWgPxm4O8u0HsYHavVPyK3TSytlB7o3yrAKvvfAXH8xveNvi2MOmi9un7VfF L29vE/jSJ3N+iebFMC635T0w8G7FqCdrVbOtM5HP8V+9/u29hFOR+YfKdHUh98zwonSw BQa9YsnaxIopeM4m972X9VtrIHGk7sifqo40jUSJRI+upoEJzSV/o1K3qMTfAOHIjcmp z5al5NY6225Vibr6OpW/wJmny4/DBUfHyJdEdvFp9g7mo0FsmcY68+3UsuwtPUQ9KbOa Z2Wg== X-Gm-Message-State: AC+VfDwYqSoo+Ia1soTS3rA/qId2d4/piQoVDZOaJrOi7lEUySaHf2ON K4q4WAld7pbWeW1gSTOkg3rFyA== X-Google-Smtp-Source: ACHHUZ7CP5vDtBRIxvautD8B9t4OVqlfgDANkt/FqORg3gwZ8Ld6bgYtk+zgsla4VgNGuWgWMvw0Aw== X-Received: by 2002:a0d:d8c4:0:b0:559:fb89:e2c4 with SMTP id a187-20020a0dd8c4000000b00559fb89e2c4mr4843838ywe.22.1682899474967; Sun, 30 Apr 2023 17:04:34 -0700 (PDT) Received: from mail-yw1-f181.google.com (mail-yw1-f181.google.com. [209.85.128.181]) by smtp.gmail.com with ESMTPSA id d190-20020a0df4c7000000b0054fcbf35b94sm6943683ywf.87.2023.04.30.17.04.34 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 30 Apr 2023 17:04:34 -0700 (PDT) Received: by mail-yw1-f181.google.com with SMTP id 00721157ae682-54fc6949475so27382557b3.3; Sun, 30 Apr 2023 17:04:34 -0700 (PDT) X-Received: by 2002:a0d:d648:0:b0:55a:535c:5c53 with SMTP id y69-20020a0dd648000000b0055a535c5c53mr1756086ywd.8.1682899473993; Sun, 30 Apr 2023 17:04:33 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Tomek CEDRO Date: Mon, 1 May 2023 02:04:22 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: BHYVE_SNAPSHOT To: Matthew Grooms , Graham Perrin , Ed Maste Cc: FreeBSD Hackers , Virtualisation on FreeBSD , elenamihailescu22@gmail.com, Mihai Carabas , gusev.vitaliy@gmail.com Content-Type: multipart/alternative; boundary="0000000000005719b605fa96918e" X-Rspamd-Queue-Id: 4Q8k3S3GzNz4TFR X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_RCPT(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --0000000000005719b605fa96918e Content-Type: text/plain; charset="UTF-8" o_O -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info On Sun, Apr 30, 2023, 23:42 Matthew Grooms wrote: > On 4/29/23 06:17, void wrote: > > Hi, > > > > Where can I read up about the recently introduced BHYVE_SNAPSHOT ? > > > > I can only find terse information about what it does (in man 5 src.conf) > > but not about how to use it and/or why. > > This began as sponsored student work with the Politehnica University of > Bucharest back in 2016 with an eye toward developing Live MIgration as a > feature. The initial bhyve save/restore patch was committed back in 2020 > ... > > https://reviews.freebsd.org/rS360648 > > Student projects related to bhyve continued until last year, but we gave > up due to there being such a massive accumulation of patches with no > path to getting them committed. We tried reaching out to project > maintainers and the FreeBSD foundation alike for support and feedback, > but there seems to be little to no interest. There have also been > attempts by companies that rely on these features to and get them > improved and committed. Unfortunately they don't appear to be having > much luck either ... > > https://reviews.freebsd.org/D38858#885651 > > Would you like to see support for VM snapshots in the generic kernel? > How about support for saving/restore checkpoints using QCOW2, VMDK via > libvdsk? How about support for warm or live migration? How about USB > device pass-through? There are experimental patches for all these > features that were developed by students at UPB. In a lot of cases, > there are open reviews that have been waiting on feedback for ages. Here > is a a presentation given by Elena from UPB just last month at the > FreeBSD devsumit in Tokyo ... > > > https://wiki.freebsd.org/DevSummit/202303?action=AttachFile&do=view&target=Presentation+-+bhyvecon.pdf > > The case is quite plain. I'm not sure what the solution is to this > problem. I'd love to hear feedback from the community about how I've got > this completely wrong and how the course could be corrected. That would > be something. > > -Matthew > > > > --0000000000005719b605fa96918e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
o_O

--
= CeDeROM, SQ7MHZ, http://www.tomek.c= edro.info

On Sun, Apr 30, 2023, 23:42 Matthew Grooms <mgrooms@shrew.net> wrote:
On 4/29/23 06:17, void wrote:
> Hi,
>
> Where can I read up about the recently introduced BHYVE_SNAPSHOT ?
>
> I can only find terse information about what it does (in man 5 src.con= f)
> but not about how to use it and/or why.

This began as sponsored student work with the Politehnica University of Bucharest back in 2016 with an eye toward developing Live MIgration as a feature. The initial bhyve save/restore patch was committed back in 2020 ..= .

https://reviews.freebsd.org/rS360648

Student projects related to bhyve continued until last year, but we gave up due to there being such a massive accumulation of patches with no
path to getting them committed. We tried reaching out to project
maintainers and the FreeBSD foundation alike for support and feedback,
but there seems to be little to no interest. There have also been
attempts by companies that rely on these features to and get them
improved and committed. Unfortunately they don't appear to be having much luck either ...

https://reviews.freebsd.org/D38858#885651
Would you like to see support for VM snapshots in the generic kernel?
How about support for saving/restore checkpoints using QCOW2, VMDK via
libvdsk? How about support for warm or live migration? How about USB
device pass-through? There are experimental patches for all these
features that were developed by students at UPB. In a lot of cases,
there are open reviews that have been waiting on feedback for ages. Here is a a presentation given by Elena from UPB just last month at the
FreeBSD devsumit in Tokyo ...

https://wiki.freebsd.org/DevSummit/202303?action= =3DAttachFile&do=3Dview&target=3DPresentation+-+bhyvecon.pdf
The case is quite plain. I'm not sure what the solution is to this
problem. I'd love to hear feedback from the community about how I'v= e got
this completely wrong and how the course could be corrected. That would be something.

-Matthew



--0000000000005719b605fa96918e-- From nobody Mon May 1 00:33:36 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q8kjC73Ffz48WD3 for ; Mon, 1 May 2023 00:33:51 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic311-25.consmr.mail.gq1.yahoo.com (sonic311-25.consmr.mail.gq1.yahoo.com [98.137.65.206]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q8kjC2fDcz4Wcc for ; Mon, 1 May 2023 00:33:51 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=XBJX4QLV; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.206 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1682901229; bh=pX018TPo4mKsuDQ3bxYUyNWN8vID7MyjbE1GruXiTWk=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=XBJX4QLVka0i2H66aMH1t8INCWRcaR+krqrKuFUar7QFmE2/qzTSGBtE55qfFvRmIm8v4wcmuCWfTdBSN2Bed3dyyjeOz34xqn2EF8EKRJYf+9CYPTLemvvnPL+fedJvjrewg83AN+laeHdONbC4RwJgiT7dpLYkdvRRBR8jUz7d/K0ujb47RQ0l0QoP13ymqflmZpzHLjzp8LsfpuSwOepmgCaLSUEDCmIkg8S6wHSNtsmqwhAcYSh3ik3NLik6QwYr8NvWFjL1fL7hmKpWd8Sp/VVmDVCkdd4SEZNrKOU2v4bYGgJ8wpuIcQkWT7AEjAqkvfRuEXr4W9xvqcchRw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1682901229; bh=6HzOao75VAf9SYbPegXArZRt+zqYGDxD2fLHjfu4Mfj=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=n326ccuVJJJiunFrPlwwXtpF9K0kZWx8Ql73IFE5D43cd7ijIAdZyRPjIINGg0xhagU2UWf0yU/tuc0XpyV2TTIm7ghxso4AeyIu9AQ7JJaMEFY8ocnWTOsL5Q7J8wGYPtBTfm00xmM6gu5keWOFZfiIqOyGnVURXK59LAYRS5JuAG5nEwJUku6ANz9OfpXJgPzrl2IACvrqUZ8Na0SM7qHJlDZmKtf3ipMJlwBf5wHGPHWjofeYSt9q0xcEnPCziwdr/5vnZLdenwYPvzDkpgw84TZSpGovm9v+0BgaXf5jZzZKTKUQqztEF7/JTqyomJtWKHalzMtWCtVvQbkFpA== X-YMail-OSG: 5DT5xD4VM1kcuCQLjQ_qJ1FeyMQ_NT2kt1.Wz.M9bGl5ftmn0_LDyDjNVjVVpvm 3bRl5346LGpsTv3soSYhgIe3hfvsl7l2AL54NuiTQbjdgnDQhlh1WMSVwz7VuLVo3UUwE4piWdar 1Y8QgkwtipCHV6KgwX_j_ntOhI7HqLargLuiEiS_yn.UdqM8KNI0owPjlBrM9JOltstUFWRwe7Pw NQ7ecWQbQvryzCd71o5W0F9tmXLW7I4kqJbTYoUvhA6WMxxsnUlbTWWghrs3JME4.x1CufM13pHV cOymR.Q9yJ3jQMPMEb5zsBFgfaRnILITEX3G.b.7S4HJupu31Fnv9g9_.Rxyu6iSs33GAD3ddQNH 6WdsSdQ03AlAMpx._kDZl2vbjBfOyZRHAfJXQeMqHAcrEmVCWbpt1KLpzrdOvKcC2XEzT69TrEz. EYg1IM87jLmtSxJDOGD1PS8rXMVcTeODS8vogDf9pE2t0BGBPT_eDG7TsWQVcBaIdLAys0tlhbii 0dYha7kAIGcCCpLbQxwNPDhuAShzlC4t8p8WKssub5HmwDRssTlOwKSBpUzJwaEVq2NrrEDwlE1r sxBl_JJscPYAqcgGbBYddAHwCzaNbJob8xoLAJDKKanmDtMmiK4QF9_rmfQu0vL4f5KrXNI7cF.h TkTJ8vA4Chp_wf1.vueTjPlLL61Tl4hsxZMc4R8X7_mRhwBa5dBY8mHB4WrDB4hV.LG7h3aJvFI. KYixQgNQJuPkF0JYqCruE44xf4VKBxtl2oNHNx7dzWC1aMXxVZnLvx0kxVM4Sud6XgUJ2Y0_.DLZ bxUTomd_AB0JW7ZKJkH5vpfmMFaocfJgaZSvCgjgX3C9zN4wgcohPMoG1YyxI52vlUqEjlPjgP0Y ecCGgURBp7qhAqZPVu0wXz_Y71vfTeFHTxGozot9SO0r7IgLSKKjTQd3E3099wYiglDFTwmsH8r_ oksa9c_TTJGFLQ_HqVnLSFsOHuOQg08IQ8rV.lI6IUcPDhcDouVgWRcEEdx5EdGehNSiQvITJvVi tNKNJo8oTNYwUL_skdDi59XnFDqLBNgBBx1hBgNTHUtnRytHCtS8YugPacWlIjmOGjIN56wqbSdS QnMLEOr43GrVgq3iSN8D6p99sKEikrKWw_deheg4Ow_LQtOhbXqusYagkhgUil4US75UpMgwhbF_ P9a_xpv2jXnfucDYa2Kt.ClYguCpxjwMmxd2dj1Y.vMgUZ0aSNwIOdZH.Cuaz3TZEpOgZ5zpSdhz H_19Uc9vOIqBLRs6UJJ6PATzEyLd7oA_4fZTI4l0BZJ5.orCFmyRFUQ7fYaBW3fsj3BsRmkI8iK4 HE3N_mAupSfNwvvtDDvD9..Xu0oZo_PPNluCruU_pLxPo4B9LRdWUKizTXz0C3tpDRX3kOC9bfYu _LUfTvXSo1fsIrjA3CVA7yrW1cJu1nc4d02_g85sriFH0X1sbW5q.4KGp.lIt.uLubO.2mGdwVmo 0edS.oxb4kocafy3FRheSntdtPQV0HMF36t6AVWmkj6.vAoVHsAQJoOyJ0TTIIWcXPx4MIOqAQg9 vPl2k_eFww1rRhgi9SkEzwIHHbHdWX79hJt1MHJ.GwfNM30fqGIuQVbIlGVdoi.4Vj4ZA4glZH9O rNjFr54r8wKxlak_aFUvLSISAtxWI2VWVOQZiu0a.wn7bvUtScJ5Qbf4o2zqdmx8ocpx8bTxAJdA cndaat3Be7h9Aa8RQDfMhKof84Q5omi5R1iV5PoQEHOodLxQZUfW1.sMKXGjeDRMGJYtL5GHDg6S vJ8zt81nb0leGVAKx8Gti7HOjhqVc6qcEny6fVryuGktLwYEU8zGhZG_vfCafleoeXI.RGDD7pqK CG5tbYT1oFz3VKj023OQLZSt5TBBwrDeUqdHkJroPmonCIb.ZiG1h6wUuPKheFw82rWeTU0Lhezg ifFVTyW84jwJw62V1_QfUrWanUeJVRGnVhy8gypkQZIpJ9W8lOl8ma00VMN.XcrctJ0jcOgE9l9k eVRk0Y.fJaxZNlk06JiWLQRrbB2HyrRo7UO.satOyeji8UOnrwE0YQTzJSpLxINWJLMVmef3Ui2D _Qhep2AtSKf0pMjp5sRd6IL8iUNY_a310O14d3g1Wr2Svkb5ph8GsaJpRjvw1h6PEmDH.DCfYa7x d0.v244STR3ZTf4Bys7mi9Ic5ULJwh.mnn6izdV51w.Is2q2I5i7HED80Ieh2pO5.K6109TiB15a A3r6f7k_FKkolBifdvwojiiiZChdehGBEM2r1EU4gjVVKVmVojlsFSZpMfv6KOqdF1s5KeOk3j4v vgg-- X-Sonic-MF: X-Sonic-ID: d00b87de-aef3-4224-83b9-56bd9d43a311 Received: from sonic.gate.mail.ne1.yahoo.com by sonic311.consmr.mail.gq1.yahoo.com with HTTP; Mon, 1 May 2023 00:33:49 +0000 Received: by hermes--production-gq1-546798879c-sfj59 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 4fbe0c6f94bed2bf88666b843be83362; Mon, 01 May 2023 00:33:47 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: armv8.2-A+ tuned FreeBSD kernels vs. poudriere bulk and USB3 media: tx->tx_quiesce_done_cv related blocking of processes? Date: Sun, 30 Apr 2023 17:33:36 -0700 References: <7AE28A5B-109E-4C26-9D70-BCA5D49CD79D@yahoo.com> <02DC03AE-E082-4FB5-AA0D-396F64CC23CB@yahoo.com> To: FreeBSD Hackers , freebsd-arm In-Reply-To: <02DC03AE-E082-4FB5-AA0D-396F64CC23CB@yahoo.com> Message-Id: X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-2.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_SHORT(-1.00)[-0.998]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.206:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_TLS_LAST(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.206:from] X-Rspamd-Queue-Id: 4Q8kjC2fDcz4Wcc X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N As the evidence this time is largely independent of the details reported previousy, I'm top posting this. The previous ZFS on USB3 results were based on poudriere using "USE_TMPFS=3Ddata", meaning that almost all file I/O was via ZFS to the USB3 media. The UFS on U2 960GB Optane via USB3 adapter results did not not suffer the reported problems, despite "USE_TMPFS=3Ddata". (I let it run to completion.) But this had both a media and a file system difference. This time the results are for just changing poudriere to use "USE_TMPFS=3Dall" instead but back on the original ZFS on USB3 media. The implication is that the vast majority of the file I/O is not via ZFS to the USB3 media. The context has 32 GiBytes of RAM and about 118 GiBytes of swap/paging space. It would need to page if pet run to completion. Observing, the load average is behaving normally: Things are not stuck waiting. "gstat -spod" indicates the ZFS I/O is not sustained (no paging in swap space use yet). First 1 hr: 262 ports built. But this had both a media and a file system difference again. I'm stopping after this, in order to set up the next just- ZFS experiments. Next experiments: I expect to move the ZFS context to U2 960GB Optane media used with the USB3 adapter and to test both "USE_TMPFS=3Ddata" and "USE_TMPSF=3Dall", probably letting USE_TMPFS=3Dall run to completion. If Optane based USE_TMPFS=3Ddata context still has the problem, even the more performance media would have been not enough to avoid what would then appear to be a ZFS problem, two other file systems not having the problem. The Optane based USE_TMPFS=3Dall context should just handle the paging and more rare ZFS I/O quicker. I do not expect problems for this combination, given the UFS on Optane USB3 results and the partial USE_TMPFS=3Dall non-Optane USB3 results. Even with ZFS working, this likely is the more performant type of context for the Windows Dev Kit 2023, given that I'm leaving Windows 11 Pro in place on the internal media. Hypothesis for the original problem: I wonder if ZFS write activity to the USB3 media was largely blocking the ZFS read activity to the same media, causing lots of things to have to spend much time waiting for data instead of making progress, leading to long periods of low load averages. Older material: On Apr 30, 2023, at 00:50, Mark Millard wrote: > On Apr 29, 2023, at 19:44, Mark Millard wrote: >=20 >> This is based on: main-n262658-b347c2284603-dirty, b347c2284603 >> being from late Apr 28, 2023 UTC. (The "-dirty" is from some >> historical patches that I use.) The build is a non-debug build >> (but with symbols not stripped). World or Kernel had been built >> using: >>=20 >> -mcpu=3Dcortex-a78C+flagm+nofp16fml >>=20 >> just for testing purposes. (Worked nicely for -j8 buildworld >> buildkernel testing for the 4 cortex-a78c's plus 4 cortex-x1c's >> present.) >>=20 >> Monitoring poudriere bulk related activity via top and gstat -spod >> I see a lot of the odd result of one process doing something >> like: >>=20 >> CPU4 4 1:39 99.12% /usr/local/sbin/pkg-static create -f tzst = -r /wrkdirs/usr/ports/devel/cmake-core/work/stage >>=20 >> while other processes sit in the likes of: >>=20 >> tx->tx >> zcq->z >> zcw->z >> zilog- >> select >> wait >>=20 >> But sometimes there is no CPU bound process and the top CPU process = is >> the likes of: >>=20 >> 1.24% [usb{usbus0}] >>=20 >> "gstat -spod" basically shows da0 dedicated to write activity most >> of the time. >>=20 >> After: sysctl kern.tty_info_kstacks=3D1 >> Then using ^T at various times, I see a lot of: >>=20 >> load: 0.48 cmd: sh 93914 [tx->tx_quiesce_done_cv] 7534.91r 11.06u = 22.66s 0% 3800k >> #0 0xffff0000004fd564 at mi_switch+0x104 >> #1 0xffff000000463f40 at _cv_wait+0x120 >> #2 0xffff00000153fa34 at txg_wait_open+0xf4 >> #3 0xffff0000014a40bc at dmu_free_long_range+0x17c >> #4 0xffff000001448254 at zfs_rmnode+0x64 >> #9 0xffff000001455678 at zfs_freebsd_inactive+0x48 >> #10 0xffff0000005fc430 at vinactivef+0x180 >> #11 0xffff0000005fba50 at vput_final+0x200 >> #12 0xffff00000060c4d0 at kern_funlinkat+0x320 >> #13 0xffff00015d6cbbf4 at filemon_wrapper_unlink+0x14 >> #14 0xffff0000008f8514 at do_el0_sync+0x594 >> #15 0xffff0000008d4904 at handle_el0_sync+0x40 >>=20 >> load: 0.34 cmd: sh 93914 [tx->tx_quiesce_done_cv] 7566.69r 11.06u = 22.66s 0% 3800k >> #0 0xffff0000004fd564 at mi_switch+0x104 >> #1 0xffff000000463f40 at _cv_wait+0x120 >> #2 0xffff00000153fa34 at txg_wait_open+0xf4 >> #3 0xffff0000014a40bc at dmu_free_long_range+0x17c >> #4 0xffff000001448254 at zfs_rmnode+0x64 >> #5 0xffff0000014557c4 at zfs_freebsd_reclaim+0x34 >> #6 0xffff000000a1340c at VOP_RECLAIM_APV+0x2c >> #7 0xffff0000005fd6c0 at vgonel+0x450 >> #8 0xffff0000005fde7c at vrecycle+0x9c >> #9 0xffff000001455678 at zfs_freebsd_inactive+0x48 >> #10 0xffff0000005fc430 at vinactivef+0x180 >> #11 0xffff0000005fba50 at vput_final+0x200 >> #12 0xffff00000060c4d0 at kern_funlinkat+0x320 >> #13 0xffff00015d6cbbf4 at filemon_wrapper_unlink+0x14 >> #14 0xffff0000008f8514 at do_el0_sync+0x594 >> #15 0xffff0000008d4904 at handle_el0_sync+0x40 >>=20 >> load: 0.44 cmd: sh 93914 [tx->tx_quiesce_done_cv] 7693.52r 11.24u = 23.08s 0% 3800k >> #0 0xffff0000004fd564 at mi_switch+0x104 >> #1 0xffff000000463f40 at _cv_wait+0x120 >> #2 0xffff00000153fa34 at txg_wait_open+0xf4 >> #3 0xffff0000014a40bc at dmu_free_long_range+0x17c >> #4 0xffff000001448254 at zfs_rmnode+0x64 >> #5 0xffff0000014557c4 at zfs_freebsd_reclaim+0x34 >> #6 0xffff000000a1340c at VOP_RECLAIM_APV+0x2c >> #7 0xffff0000005fd6c0 at vgonel+0x450 >> #8 0xffff0000005fde7c at vrecycle+0x9c >> #9 0xffff000001455678 at zfs_freebsd_inactive+0x48 >> #10 0xffff0000005fc430 at vinactivef+0x180 >> #11 0xffff0000005fba50 at vput_final+0x200 >> #12 0xffff00000060c4d0 at kern_funlinkat+0x320 >> #13 0xffff00015d6cbbf4 at filemon_wrapper_unlink+0x14 >> #14 0xffff0000008f8514 at do_el0_sync+0x594 >> #15 0xffff0000008d4904 at handle_el0_sync+0x40 >>=20 >>=20 >> The system (Windows Dev Kit 2023) has 32 GiBytes of RAM. Example >> output from a top modified to show some "Max[imum]Obs[erved]" >> information: >>=20 >> last pid: 17198; load averages: 0.33, 0.58, 1.06 MaxObs: = 15.49, 8.73, 5.75 = up 0+20:48:10 19:14:49 >> 426 threads: 9 running, 394 sleeping, 1 stopped, 22 waiting, 50 = MaxObsRunning >> CPU: 0.0% user, 0.0% nice, 0.2% system, 0.1% interrupt, 99.7% = idle >> Mem: 282760Ki Active, 7716Mi Inact, 23192Ki Laundry, 22444Mi Wired, = 2780Ki Buf, 848840Ki Free, 2278Mi MaxObsActive, 22444Mi MaxObsWired, = 22752Mi MaxObs(Act+Wir+Lndry) >> ARC: 11359Mi Total, 3375Mi MFU, 5900Mi MRU, 993Mi Anon, 93076Ki = Header, 992Mi Other >> 8276Mi Compressed, 19727Mi Uncompressed, 2.38:1 Ratio >> Swap: 120832Mi Total, 120832Mi Free, 2301Mi = MaxObs(Act+Lndry+SwapUsed), 22752Mi MaxObs(Act+Wir+Lndry+SwapUsed) >>=20 >>=20 >> The poudriere bulk has 8 builders but has ALLOW_MAKE_JOBS=3Dyes >> without any explicit settings for the likes of MAKE_JOBS_NUMBER . >> So it is a configuration that allows a high load average compared >> to the number of hardware threads (here: cores) in the system. >>=20 >>=20 >> I've rebooted to do a test with filemon not loaded at the time >> (here it was loaded from prior buildworld buildkernel activity). >> We will see if it still ends up with such problems. >=20 > It still ends up with things waiting, but the detailed STATE > valiues generally lsited are somewhat different. >=20 > I also tried a chroot into a world from use of -mcpu=3Dcortex-a72 > and got similar results. Suggesting that only the > -mcpu=3Dcortex-a78c+flagm+nofp16fml kernel is required to see the > issues. This even got some examples like: >=20 > load: 1.20 cmd: sh 95560 [tx->tx_quiesce_done_cv] 2022.55r 3.21u = 9.24s 0% 3852k > #0 0xffff0000004fd564 at mi_switch+0x104 > #1 0xffff000000463f40 at _cv_wait+0x120 > #2 0xffff000001518a34 at txg_wait_open+0xf4 > #3 0xffff00000147d0bc at dmu_free_long_range+0x17c > #4 0xffff000001421254 at zfs_rmnode+0x64 > #5 0xffff00000142e7c4 at zfs_freebsd_reclaim+0x34 > #6 0xffff000000a1340c at VOP_RECLAIM_APV+0x2c > #7 0xffff0000005fd6c0 at vgonel+0x450 > #8 0xffff0000005fde7c at vrecycle+0x9c > #9 0xffff00000142e678 at zfs_freebsd_inactive+0x48 > #10 0xffff0000005fc430 at vinactivef+0x180 > #11 0xffff0000005fba50 at vput_final+0x200 > #12 0xffff00015d8ceab4 at null_reclaim+0x154 > #13 0xffff000000a1340c at VOP_RECLAIM_APV+0x2c > #14 0xffff0000005fd6c0 at vgonel+0x450 > #15 0xffff0000005fde7c at vrecycle+0x9c > #16 0xffff00015d8ce8e8 at null_inactive+0x38 > #17 0xffff0000005fc430 at vinactivef+0x180 >=20 > (The chroot use involves null mounts.) >=20 > which is sort of analogous to the filemon related > backtraces I showed earlier. The common part=20 > across the examples looks to be #0..#11: >=20 > #0 0xffff0000004fd564 at mi_switch+0x104 > #1 0xffff000000463f40 at _cv_wait+0x120 > #2 0xffff00000153fa34 at txg_wait_open+0xf4 > #3 0xffff0000014a40bc at dmu_free_long_range+0x17c > #4 0xffff000001448254 at zfs_rmnode+0x64 > #5 0xffff0000014557c4 at zfs_freebsd_reclaim+0x34 > #6 0xffff000000a1340c at VOP_RECLAIM_APV+0x2c > #7 0xffff0000005fd6c0 at vgonel+0x450 > #8 0xffff0000005fde7c at vrecycle+0x9c > #9 0xffff000001455678 at zfs_freebsd_inactive+0x48 > #10 0xffff0000005fc430 at vinactivef+0x180 > #11 0xffff0000005fba50 at vput_final+0x200 >=20 > There were a lot more nanslp examples in all > the alter testing (i.e, those avoided filemon.ko > being loaded). >=20 > Starting from having pkgclean -A'd the ports, the > experiments got about the same number of ports built > as of the end of the 1st hour. >=20 >=20 >=20 > UFS vs. ZFS? Different media types? . . . >=20 > So I decided to create and try a UFS test context > instead of a ZFS one. But the media that was best > to update was a U2 960GB Optane in a USB3 > adapter, something that would perform noticeably > better than my normal USB3 NVMe drives, even with > USB involved. >=20 > This combination maintained reasonable load averages > (instead of having long periods of <1) and finish > building 172 ports in the 1st hour, far more than the > around 83 each time I tried the other device/ZFS > combination. NO evdience of the earlier reported > oddities. >=20 > I should also time a from-scratch buildworld > buildkernel. >=20 > I'll look into setting up another U2 960GB Optane > for use in the USB3 adpater, but with ZFS. That > should help isolate media vs. file system > contributions to the varying behaviors. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Mon May 1 00:44:10 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q8kx872B9z48WfM; Mon, 1 May 2023 00:44:12 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oi1-x234.google.com (mail-oi1-x234.google.com [IPv6:2607:f8b0:4864:20::234]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q8kx81Gspz4YjQ; Mon, 1 May 2023 00:44:12 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oi1-x234.google.com with SMTP id 5614622812f47-38dfdc1daa9so1339128b6e.1; Sun, 30 Apr 2023 17:44:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1682901851; x=1685493851; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=mcXwCpT4Vk7YyJ1B6VIHWGfjsi7tjVDvKFYslcLEEf0=; b=P9VID/nuVD1PunzpW+s1yB72WfTbYQFw8kaDnM9p+kkG0deOU56ydXLlvUPz3gljd9 F6m034g2UejdfOpu6kNhrSbTBbTt2d5ZK/xVylul/32S1iGfIoR1wh3T6SQswyEXf48n /y/t5+OHGJ5iXcdeSGSnRA7Zw1EfYTyR6QojXTmNZSZMAhWNx4ZxRLmWOt4GCXGQAICA Zr7Qo+2fxMOtyNnAq9py5ecNrHVMRRVqvd6tU5iauGo6NXGssDnIAzT7CUSXchihuQ7F w32x2pns9b6FGL6TkYW8HLxLXzCJvYWENEJrgTONHfCZNKqDy0lnZVVmns0+2NTgOqLV q5uw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682901851; x=1685493851; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=mcXwCpT4Vk7YyJ1B6VIHWGfjsi7tjVDvKFYslcLEEf0=; b=YbN9dPL4j/aZqV0nAkRkCa5w9SmZ9aLk5BXzyjyVgpntNwxsfNMvPRzXg3fSHE9lH+ 8L2pfSk4TdLqb0hHtmq9brrKKIzahvY54OhQRpcDbo0w1C3NF6fJCDulvFM289pFSo0a 4fhjkMClz+nZ4ENMGoxD6kv2jkcwce/JDfOZ66Y/6R+L38GMa6sbX7+0uEV9Z5lmTMvV DO2Ot+jlHDQrWk528oA/fRKH5VrXmKHrJfEZU5pF+4CaLwM96nekfE496+M8h6CmZw1y biY7b9o9LqzF8sLcrJriQKawhiLf/qTGhZ+fXkuOMWgbq7NgAiTE/HIRaKKWX9ep4TIP 5e2Q== X-Gm-Message-State: AC+VfDzYh77OmsovYz8pBxnaSd3Kulds9lqkG5PFj0NOrIyjrThtyw7J w4q1txjxlsI0l228jcEz2YhZmV79ouktKRPA9rcicn5o X-Google-Smtp-Source: ACHHUZ40rbXa1CytS5bGunpAgnfsTwB0N1LvAFHLZCbsJv+qv+hNIQkl8yZVnD4KSsqOCGkpM6D5jZxUedcyLyHkUIE= X-Received: by 2002:a05:6808:253:b0:389:7b6b:7a2c with SMTP id m19-20020a056808025300b003897b6b7a2cmr6157292oie.44.1682901850774; Sun, 30 Apr 2023 17:44:10 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:a8a:1194:0:b0:4d4:94b:7266 with HTTP; Sun, 30 Apr 2023 17:44:10 -0700 (PDT) In-Reply-To: References: <7AE28A5B-109E-4C26-9D70-BCA5D49CD79D@yahoo.com> <02DC03AE-E082-4FB5-AA0D-396F64CC23CB@yahoo.com> From: Mateusz Guzik Date: Mon, 1 May 2023 02:44:10 +0200 Message-ID: Subject: Re: armv8.2-A+ tuned FreeBSD kernels vs. poudriere bulk and USB3 media: tx->tx_quiesce_done_cv related blocking of processes? To: Mark Millard Cc: FreeBSD Hackers , freebsd-arm Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4Q8kx81Gspz4YjQ X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N can you redo zfs test with: sysctl vfs.zfs.per_txg_dirty_frees_percent=5 On 5/1/23, Mark Millard wrote: > As the evidence this time is largely independent of the details > reported previousy, I'm top posting this. > > The previous ZFS on USB3 results were based on poudriere using > "USE_TMPFS=data", meaning that almost all file I/O was via ZFS > to the USB3 media. > > The UFS on U2 960GB Optane via USB3 adapter results did not not > suffer the reported problems, despite "USE_TMPFS=data". (I let > it run to completion.) But this had both a media and a file > system difference. > > This time the results are for just changing poudriere to use > "USE_TMPFS=all" instead but back on the original ZFS on > USB3 media. The implication is that the vast majority of the > file I/O is not via ZFS to the USB3 media. > > The context has 32 GiBytes of RAM and about 118 GiBytes of > swap/paging space. It would need to page if pet run to > completion. > > Observing, the load average is behaving normally: Things are > not stuck waiting. "gstat -spod" indicates the ZFS I/O is > not sustained (no paging in swap space use yet). > > First 1 hr: 262 ports built. > > But this had both a media and a file system difference again. > > I'm stopping after this, in order to set up the next just- > ZFS experiments. > > > Next experiments: > > I expect to move the ZFS context to U2 960GB Optane media > used with the USB3 adapter and to test both "USE_TMPFS=data" > and "USE_TMPSF=all", probably letting USE_TMPFS=all run to > completion. > > If Optane based USE_TMPFS=data context still has the problem, > even the more performance media would have been not enough to > avoid what would then appear to be a ZFS problem, two other > file systems not having the problem. > > The Optane based USE_TMPFS=all context should just handle the > paging and more rare ZFS I/O quicker. I do not expect problems > for this combination, given the UFS on Optane USB3 results > and the partial USE_TMPFS=all non-Optane USB3 results. Even > with ZFS working, this likely is the more performant type of > context for the Windows Dev Kit 2023, given that I'm leaving > Windows 11 Pro in place on the internal media. > > > Hypothesis for the original problem: > > I wonder if ZFS write activity to the USB3 media was largely > blocking the ZFS read activity to the same media, causing > lots of things to have to spend much time waiting for data > instead of making progress, leading to long periods of low > load averages. > > > Older material: > > On Apr 30, 2023, at 00:50, Mark Millard wrote: > >> On Apr 29, 2023, at 19:44, Mark Millard wrote: >> >>> This is based on: main-n262658-b347c2284603-dirty, b347c2284603 >>> being from late Apr 28, 2023 UTC. (The "-dirty" is from some >>> historical patches that I use.) The build is a non-debug build >>> (but with symbols not stripped). World or Kernel had been built >>> using: >>> >>> -mcpu=cortex-a78C+flagm+nofp16fml >>> >>> just for testing purposes. (Worked nicely for -j8 buildworld >>> buildkernel testing for the 4 cortex-a78c's plus 4 cortex-x1c's >>> present.) >>> >>> Monitoring poudriere bulk related activity via top and gstat -spod >>> I see a lot of the odd result of one process doing something >>> like: >>> >>> CPU4 4 1:39 99.12% /usr/local/sbin/pkg-static create -f tzst -r >>> /wrkdirs/usr/ports/devel/cmake-core/work/stage >>> >>> while other processes sit in the likes of: >>> >>> tx->tx >>> zcq->z >>> zcw->z >>> zilog- >>> select >>> wait >>> >>> But sometimes there is no CPU bound process and the top CPU process is >>> the likes of: >>> >>> 1.24% [usb{usbus0}] >>> >>> "gstat -spod" basically shows da0 dedicated to write activity most >>> of the time. >>> >>> After: sysctl kern.tty_info_kstacks=1 >>> Then using ^T at various times, I see a lot of: >>> >>> load: 0.48 cmd: sh 93914 [tx->tx_quiesce_done_cv] 7534.91r 11.06u 22.66s >>> 0% 3800k >>> #0 0xffff0000004fd564 at mi_switch+0x104 >>> #1 0xffff000000463f40 at _cv_wait+0x120 >>> #2 0xffff00000153fa34 at txg_wait_open+0xf4 >>> #3 0xffff0000014a40bc at dmu_free_long_range+0x17c >>> #4 0xffff000001448254 at zfs_rmnode+0x64 >>> #9 0xffff000001455678 at zfs_freebsd_inactive+0x48 >>> #10 0xffff0000005fc430 at vinactivef+0x180 >>> #11 0xffff0000005fba50 at vput_final+0x200 >>> #12 0xffff00000060c4d0 at kern_funlinkat+0x320 >>> #13 0xffff00015d6cbbf4 at filemon_wrapper_unlink+0x14 >>> #14 0xffff0000008f8514 at do_el0_sync+0x594 >>> #15 0xffff0000008d4904 at handle_el0_sync+0x40 >>> >>> load: 0.34 cmd: sh 93914 [tx->tx_quiesce_done_cv] 7566.69r 11.06u 22.66s >>> 0% 3800k >>> #0 0xffff0000004fd564 at mi_switch+0x104 >>> #1 0xffff000000463f40 at _cv_wait+0x120 >>> #2 0xffff00000153fa34 at txg_wait_open+0xf4 >>> #3 0xffff0000014a40bc at dmu_free_long_range+0x17c >>> #4 0xffff000001448254 at zfs_rmnode+0x64 >>> #5 0xffff0000014557c4 at zfs_freebsd_reclaim+0x34 >>> #6 0xffff000000a1340c at VOP_RECLAIM_APV+0x2c >>> #7 0xffff0000005fd6c0 at vgonel+0x450 >>> #8 0xffff0000005fde7c at vrecycle+0x9c >>> #9 0xffff000001455678 at zfs_freebsd_inactive+0x48 >>> #10 0xffff0000005fc430 at vinactivef+0x180 >>> #11 0xffff0000005fba50 at vput_final+0x200 >>> #12 0xffff00000060c4d0 at kern_funlinkat+0x320 >>> #13 0xffff00015d6cbbf4 at filemon_wrapper_unlink+0x14 >>> #14 0xffff0000008f8514 at do_el0_sync+0x594 >>> #15 0xffff0000008d4904 at handle_el0_sync+0x40 >>> >>> load: 0.44 cmd: sh 93914 [tx->tx_quiesce_done_cv] 7693.52r 11.24u 23.08s >>> 0% 3800k >>> #0 0xffff0000004fd564 at mi_switch+0x104 >>> #1 0xffff000000463f40 at _cv_wait+0x120 >>> #2 0xffff00000153fa34 at txg_wait_open+0xf4 >>> #3 0xffff0000014a40bc at dmu_free_long_range+0x17c >>> #4 0xffff000001448254 at zfs_rmnode+0x64 >>> #5 0xffff0000014557c4 at zfs_freebsd_reclaim+0x34 >>> #6 0xffff000000a1340c at VOP_RECLAIM_APV+0x2c >>> #7 0xffff0000005fd6c0 at vgonel+0x450 >>> #8 0xffff0000005fde7c at vrecycle+0x9c >>> #9 0xffff000001455678 at zfs_freebsd_inactive+0x48 >>> #10 0xffff0000005fc430 at vinactivef+0x180 >>> #11 0xffff0000005fba50 at vput_final+0x200 >>> #12 0xffff00000060c4d0 at kern_funlinkat+0x320 >>> #13 0xffff00015d6cbbf4 at filemon_wrapper_unlink+0x14 >>> #14 0xffff0000008f8514 at do_el0_sync+0x594 >>> #15 0xffff0000008d4904 at handle_el0_sync+0x40 >>> >>> >>> The system (Windows Dev Kit 2023) has 32 GiBytes of RAM. Example >>> output from a top modified to show some "Max[imum]Obs[erved]" >>> information: >>> >>> last pid: 17198; load averages: 0.33, 0.58, 1.06 MaxObs: 15.49, >>> 8.73, 5.75 >>> up 0+20:48:10 19:14:49 >>> 426 threads: 9 running, 394 sleeping, 1 stopped, 22 waiting, 50 >>> MaxObsRunning >>> CPU: 0.0% user, 0.0% nice, 0.2% system, 0.1% interrupt, 99.7% idle >>> Mem: 282760Ki Active, 7716Mi Inact, 23192Ki Laundry, 22444Mi Wired, >>> 2780Ki Buf, 848840Ki Free, 2278Mi MaxObsActive, 22444Mi MaxObsWired, >>> 22752Mi MaxObs(Act+Wir+Lndry) >>> ARC: 11359Mi Total, 3375Mi MFU, 5900Mi MRU, 993Mi Anon, 93076Ki Header, >>> 992Mi Other >>> 8276Mi Compressed, 19727Mi Uncompressed, 2.38:1 Ratio >>> Swap: 120832Mi Total, 120832Mi Free, 2301Mi MaxObs(Act+Lndry+SwapUsed), >>> 22752Mi MaxObs(Act+Wir+Lndry+SwapUsed) >>> >>> >>> The poudriere bulk has 8 builders but has ALLOW_MAKE_JOBS=yes >>> without any explicit settings for the likes of MAKE_JOBS_NUMBER . >>> So it is a configuration that allows a high load average compared >>> to the number of hardware threads (here: cores) in the system. >>> >>> >>> I've rebooted to do a test with filemon not loaded at the time >>> (here it was loaded from prior buildworld buildkernel activity). >>> We will see if it still ends up with such problems. >> >> It still ends up with things waiting, but the detailed STATE >> valiues generally lsited are somewhat different. >> >> I also tried a chroot into a world from use of -mcpu=cortex-a72 >> and got similar results. Suggesting that only the >> -mcpu=cortex-a78c+flagm+nofp16fml kernel is required to see the >> issues. This even got some examples like: >> >> load: 1.20 cmd: sh 95560 [tx->tx_quiesce_done_cv] 2022.55r 3.21u 9.24s 0% >> 3852k >> #0 0xffff0000004fd564 at mi_switch+0x104 >> #1 0xffff000000463f40 at _cv_wait+0x120 >> #2 0xffff000001518a34 at txg_wait_open+0xf4 >> #3 0xffff00000147d0bc at dmu_free_long_range+0x17c >> #4 0xffff000001421254 at zfs_rmnode+0x64 >> #5 0xffff00000142e7c4 at zfs_freebsd_reclaim+0x34 >> #6 0xffff000000a1340c at VOP_RECLAIM_APV+0x2c >> #7 0xffff0000005fd6c0 at vgonel+0x450 >> #8 0xffff0000005fde7c at vrecycle+0x9c >> #9 0xffff00000142e678 at zfs_freebsd_inactive+0x48 >> #10 0xffff0000005fc430 at vinactivef+0x180 >> #11 0xffff0000005fba50 at vput_final+0x200 >> #12 0xffff00015d8ceab4 at null_reclaim+0x154 >> #13 0xffff000000a1340c at VOP_RECLAIM_APV+0x2c >> #14 0xffff0000005fd6c0 at vgonel+0x450 >> #15 0xffff0000005fde7c at vrecycle+0x9c >> #16 0xffff00015d8ce8e8 at null_inactive+0x38 >> #17 0xffff0000005fc430 at vinactivef+0x180 >> >> (The chroot use involves null mounts.) >> >> which is sort of analogous to the filemon related >> backtraces I showed earlier. The common part >> across the examples looks to be #0..#11: >> >> #0 0xffff0000004fd564 at mi_switch+0x104 >> #1 0xffff000000463f40 at _cv_wait+0x120 >> #2 0xffff00000153fa34 at txg_wait_open+0xf4 >> #3 0xffff0000014a40bc at dmu_free_long_range+0x17c >> #4 0xffff000001448254 at zfs_rmnode+0x64 >> #5 0xffff0000014557c4 at zfs_freebsd_reclaim+0x34 >> #6 0xffff000000a1340c at VOP_RECLAIM_APV+0x2c >> #7 0xffff0000005fd6c0 at vgonel+0x450 >> #8 0xffff0000005fde7c at vrecycle+0x9c >> #9 0xffff000001455678 at zfs_freebsd_inactive+0x48 >> #10 0xffff0000005fc430 at vinactivef+0x180 >> #11 0xffff0000005fba50 at vput_final+0x200 >> >> There were a lot more nanslp examples in all >> the alter testing (i.e, those avoided filemon.ko >> being loaded). >> >> Starting from having pkgclean -A'd the ports, the >> experiments got about the same number of ports built >> as of the end of the 1st hour. >> >> >> >> UFS vs. ZFS? Different media types? . . . >> >> So I decided to create and try a UFS test context >> instead of a ZFS one. But the media that was best >> to update was a U2 960GB Optane in a USB3 >> adapter, something that would perform noticeably >> better than my normal USB3 NVMe drives, even with >> USB involved. >> >> This combination maintained reasonable load averages >> (instead of having long periods of <1) and finish >> building 172 ports in the 1st hour, far more than the >> around 83 each time I tried the other device/ZFS >> combination. NO evdience of the earlier reported >> oddities. >> >> I should also time a from-scratch buildworld >> buildkernel. >> >> I'll look into setting up another U2 960GB Optane >> for use in the USB3 adpater, but with ZFS. That >> should help isolate media vs. file system >> contributions to the varying behaviors. > > > > === > Mark Millard > marklmi at yahoo.com > > > -- Mateusz Guzik From nobody Mon May 1 00:45:56 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q8kzN553qz48WpS; Mon, 1 May 2023 00:46:08 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-yw1-x112b.google.com (mail-yw1-x112b.google.com [IPv6:2607:f8b0:4864:20::112b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q8kzN3T4jz4ZmM; Mon, 1 May 2023 00:46:08 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-yw1-x112b.google.com with SMTP id 00721157ae682-55a00da4e53so20829927b3.0; Sun, 30 Apr 2023 17:46:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1682901967; x=1685493967; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=OLH7E+tNyMAOlFTmQ3zboPvehIf56Te4NVP8VPyTTDU=; b=DSQCRPiSGgG3tn6Pzypqt3oN7zB4N+46aUv4gN0FSd84ZbpNOmMnzoRHkDdFIWmQOW b+BDZq/2Khj1R/oE3Lj0P4jqyE2FGGQk/3PM5oiJboDb4i3dFV2d/DpsV1WP/EcX1jXj TFR1S+ENGiGHkj1JmM6RV3IRjEh6rdm83U/NxeuYcB9HN4Ev402b86mmZe1W+b0/H4rA NKCxSFeHiZ7qp/XKGDI8ALm1+cwWNgREHmK1JFVOa0mVCSvOOukTyhdivKqXPVyfUL9u Q84XsN4pFqnp1lGkgkrZSqesQr8eVAnTngFscomJWGo7sdm5xnGZ1mwFvfBeMmDa/RZe cM2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682901967; x=1685493967; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=OLH7E+tNyMAOlFTmQ3zboPvehIf56Te4NVP8VPyTTDU=; b=hCzdzzduSCCt72ySVTpXao+2jgExHj4mwdU7F/p1OZ/nHJaG7LlDK8OHOZwV06VwdF PE/eOajEwmBcX83/k8ghliB4/8wCSQ9Mz+Lvu4uUgm7Z40k0h6zj5dmh94Nhvd0toCPE 0buUO0NjqqNZjIjTmWkzX1DakB4p9EPiWPVcrTVSs3iiwk/q7Xke98Arrlrl9pcMPJ75 1Sk+oj7WV2D4jW9gwvfzo6OGULcXkEJRde+z5eDMfg1TEPIMRMFjzwpo7XVB9Ljgvrtp BsvkuYFcV5yB4BfFOGDYkc5MZ6EUYrBFebn2lrZJWqLQyLoWGR+/bEaK6lBE2zI48i+f g7Ag== X-Gm-Message-State: AC+VfDwL6WJITxNAGiQ3Nr5AKz7coHGeuYia1D86Re4J66lj2YX+6YSr F4Gwgow9XKrG2ZC9aZaXrMfElfnshcnbNnaTMG0= X-Google-Smtp-Source: ACHHUZ7Qmoatazbvvxk4LXs0v60QS2uLutzQkmU3Mq5dBFA2wX00HG/yo843iLvNCwJJKJOzB6Rk5UppoNlQKrfCok0= X-Received: by 2002:a81:71d6:0:b0:559:f52b:7c5f with SMTP id m205-20020a8171d6000000b00559f52b7c5fmr4941081ywc.17.1682901967597; Sun, 30 Apr 2023 17:46:07 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Mario Marietto Date: Mon, 1 May 2023 02:45:56 +0200 Message-ID: Subject: Re: BHYVE_SNAPSHOT To: Tomek CEDRO Cc: Matthew Grooms , Graham Perrin , Ed Maste , FreeBSD Hackers , Virtualisation on FreeBSD , elenamihailescu22@gmail.com, Mihai Carabas , gusev.vitaliy@gmail.com Content-Type: multipart/alternative; boundary="000000000000f8772005fa9725dd" X-Rspamd-Queue-Id: 4Q8kzN3T4jz4ZmM X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_RCPT(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --000000000000f8772005fa9725dd Content-Type: text/plain; charset="UTF-8" whats the meaning of o_O ? Il lun 1 mag 2023, 02:04 Tomek CEDRO ha scritto: > o_O > > -- > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > > On Sun, Apr 30, 2023, 23:42 Matthew Grooms wrote: > >> On 4/29/23 06:17, void wrote: >> > Hi, >> > >> > Where can I read up about the recently introduced BHYVE_SNAPSHOT ? >> > >> > I can only find terse information about what it does (in man 5 src.conf) >> > but not about how to use it and/or why. >> >> This began as sponsored student work with the Politehnica University of >> Bucharest back in 2016 with an eye toward developing Live MIgration as a >> feature. The initial bhyve save/restore patch was committed back in 2020 >> ... >> >> https://reviews.freebsd.org/rS360648 >> >> Student projects related to bhyve continued until last year, but we gave >> up due to there being such a massive accumulation of patches with no >> path to getting them committed. We tried reaching out to project >> maintainers and the FreeBSD foundation alike for support and feedback, >> but there seems to be little to no interest. There have also been >> attempts by companies that rely on these features to and get them >> improved and committed. Unfortunately they don't appear to be having >> much luck either ... >> >> https://reviews.freebsd.org/D38858#885651 >> >> Would you like to see support for VM snapshots in the generic kernel? >> How about support for saving/restore checkpoints using QCOW2, VMDK via >> libvdsk? How about support for warm or live migration? How about USB >> device pass-through? There are experimental patches for all these >> features that were developed by students at UPB. In a lot of cases, >> there are open reviews that have been waiting on feedback for ages. Here >> is a a presentation given by Elena from UPB just last month at the >> FreeBSD devsumit in Tokyo ... >> >> >> https://wiki.freebsd.org/DevSummit/202303?action=AttachFile&do=view&target=Presentation+-+bhyvecon.pdf >> >> The case is quite plain. I'm not sure what the solution is to this >> problem. I'd love to hear feedback from the community about how I've got >> this completely wrong and how the course could be corrected. That would >> be something. >> >> -Matthew >> >> >> >> --000000000000f8772005fa9725dd Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
whats the meaning of o_O ?

Il lun 1 mag 2023, 02:04 Tomek = CEDRO <tomek@cedro.info> ha s= critto:
o_O

On Sun, Apr 30, 2023, 23:42 Matthew Grooms &l= t;mgrooms@shrew.net> wrote:
O= n 4/29/23 06:17, void wrote:
> Hi,
>
> Where can I read up about the recently introduced BHYVE_SNAPSHOT ?
>
> I can only find terse information about what it does (in man 5 src.con= f)
> but not about how to use it and/or why.

This began as sponsored student work with the Politehnica University of Bucharest back in 2016 with an eye toward developing Live MIgration as a feature. The initial bhyve save/restore patch was committed back in 2020 ..= .

https://reviews.freebsd.org/rS360648
Student projects related to bhyve continued until last year, but we gave up due to there being such a massive accumulation of patches with no
path to getting them committed. We tried reaching out to project
maintainers and the FreeBSD foundation alike for support and feedback,
but there seems to be little to no interest. There have also been
attempts by companies that rely on these features to and get them
improved and committed. Unfortunately they don't appear to be having much luck either ...

https://reviews.freebsd.org/D38858#88= 5651

Would you like to see support for VM snapshots in the generic kernel?
How about support for saving/restore checkpoints using QCOW2, VMDK via
libvdsk? How about support for warm or live migration? How about USB
device pass-through? There are experimental patches for all these
features that were developed by students at UPB. In a lot of cases,
there are open reviews that have been waiting on feedback for ages. Here is a a presentation given by Elena from UPB just last month at the
FreeBSD devsumit in Tokyo ...

https://wiki.freebsd.org/DevSummit/20= 2303?action=3DAttachFile&do=3Dview&target=3DPresentation+-+bhyvecon= .pdf

The case is quite plain. I'm not sure what the solution is to this
problem. I'd love to hear feedback from the community about how I'v= e got
this completely wrong and how the course could be corrected. That would be something.

-Matthew



--000000000000f8772005fa9725dd-- From nobody Mon May 1 01:10:49 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q8lX74Fxtz48YGB for ; Mon, 1 May 2023 01:11:03 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ej1-x62f.google.com (mail-ej1-x62f.google.com [IPv6:2a00:1450:4864:20::62f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q8lX72gqHz4d5m for ; Mon, 1 May 2023 01:11:03 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x62f.google.com with SMTP id a640c23a62f3a-94f1a6e66c9so401424666b.2 for ; Sun, 30 Apr 2023 18:11:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20221208.gappssmtp.com; s=20221208; t=1682903461; x=1685495461; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=Q8fXpFKn27V7m9JqCU+c5eN8wHyg+o03ZOHVKHdJOgM=; b=GT1xdVQ56SJ8zPXDD2cbLn5XikHDTucpOUijuiXtaxZBc2SHALUFe6HV6f7DbThH2w lQ9y0KNf55ojKODPOaA+sP8NRa9Zxkq45MTxYc/1N8Y/v7TdkwUxjmletFt4E0NgVkqF YfonuMPOnULV1DCNNuqtrdhljUMbSIi0X7f+BN4N+ZUsMv3anV7jn+l5aK9PurwAXWN8 mU7NB4CVaBxWmhwy81oyIksgvvuVOoQlfP0lITxMXW8Ekezo8ghlSbL0RJrBShFEaK7C 8bFEZpMNtLpQnqBDpNJ/bC+f/71/r1rrLF5Py4YnFpm+ts73JI8CVIyeT2+GAm1H+ejw wQAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682903461; x=1685495461; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Q8fXpFKn27V7m9JqCU+c5eN8wHyg+o03ZOHVKHdJOgM=; b=IfySXcIlUEj/jbjC4jtESLSdDIz0jMly9PSlA6lUNQEalJvnm+4qO2w+/mmHstbnm5 9yakj6DWZ+wb+/MBYcdUPqyrOyIBb19UV1iC/IJYkhW1qxTGnGOu/d8r/kaVeb7ciT+m 3Wu7qNVRmtoIIMjgOxunFrgj8EcPPn08mj4eYzfEnxVGMMDF55yqHxasDMkjusslYErr uuS8T5d0Hx5h0Bq9vPIuX5b/41Sm0epRUpiDoDPr0BTdpR0mJhO3Usrg24A48G99rn/O 3qbMJLMUj+E3QL6D36fKua2/on4Bo386NS6eif7+nsXRb8M+AZcOaY8JHG2QHzl48kAr SlXw== X-Gm-Message-State: AC+VfDwx8ZRu9P0CYID+g7F6toIBoSY6KWABIsTTf2ThQm9ybt+oe7nI exf+dWUicA83Zbb+T14Ii4wfdZqLmU9b5rXOK2TC4Q== X-Google-Smtp-Source: ACHHUZ5tiL1Ty0hNfUHrl0w6K/zx/vpq1GMPm987lLs8nF2bjGIgqsN4DPqqF9We+OVhX40xeNUG4fiVL1ypDgvJCEo= X-Received: by 2002:a17:907:928d:b0:94f:553:6fd6 with SMTP id bw13-20020a170907928d00b0094f05536fd6mr11525832ejc.24.1682903461040; Sun, 30 Apr 2023 18:11:01 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Sun, 30 Apr 2023 19:10:49 -0600 Message-ID: Subject: Re: BHYVE_SNAPSHOT To: Mario Marietto Cc: Tomek CEDRO , Matthew Grooms , Graham Perrin , Ed Maste , FreeBSD Hackers , Virtualisation on FreeBSD , Elena Mihailescu , Mihai Carabas , gusev.vitaliy@gmail.com Content-Type: multipart/alternative; boundary="000000000000fca5b605fa977e0b" X-Rspamd-Queue-Id: 4Q8lX72gqHz4d5m X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_RCPT(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --000000000000fca5b605fa977e0b Content-Type: text/plain; charset="UTF-8" "Wow! That's eye popping" On Sun, Apr 30, 2023, 6:46 PM Mario Marietto wrote: > whats the meaning of o_O ? > > Il lun 1 mag 2023, 02:04 Tomek CEDRO ha scritto: > >> o_O >> >> -- >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info >> >> On Sun, Apr 30, 2023, 23:42 Matthew Grooms wrote: >> >>> On 4/29/23 06:17, void wrote: >>> > Hi, >>> > >>> > Where can I read up about the recently introduced BHYVE_SNAPSHOT ? >>> > >>> > I can only find terse information about what it does (in man 5 >>> src.conf) >>> > but not about how to use it and/or why. >>> >>> This began as sponsored student work with the Politehnica University of >>> Bucharest back in 2016 with an eye toward developing Live MIgration as a >>> feature. The initial bhyve save/restore patch was committed back in 2020 >>> ... >>> >>> https://reviews.freebsd.org/rS360648 >>> >>> Student projects related to bhyve continued until last year, but we gave >>> up due to there being such a massive accumulation of patches with no >>> path to getting them committed. We tried reaching out to project >>> maintainers and the FreeBSD foundation alike for support and feedback, >>> but there seems to be little to no interest. There have also been >>> attempts by companies that rely on these features to and get them >>> improved and committed. Unfortunately they don't appear to be having >>> much luck either ... >>> >>> https://reviews.freebsd.org/D38858#885651 >>> >>> Would you like to see support for VM snapshots in the generic kernel? >>> How about support for saving/restore checkpoints using QCOW2, VMDK via >>> libvdsk? How about support for warm or live migration? How about USB >>> device pass-through? There are experimental patches for all these >>> features that were developed by students at UPB. In a lot of cases, >>> there are open reviews that have been waiting on feedback for ages. Here >>> is a a presentation given by Elena from UPB just last month at the >>> FreeBSD devsumit in Tokyo ... >>> >>> >>> https://wiki.freebsd.org/DevSummit/202303?action=AttachFile&do=view&target=Presentation+-+bhyvecon.pdf >>> >>> The case is quite plain. I'm not sure what the solution is to this >>> problem. I'd love to hear feedback from the community about how I've got >>> this completely wrong and how the course could be corrected. That would >>> be something. >>> >>> -Matthew >>> >>> >>> >>> --000000000000fca5b605fa977e0b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
"Wow! That's eye popping"

On Sun, Apr 30, 20= 23, 6:46 PM Mario Marietto <ma= rietto2008@gmail.com> wrote:
whats the meaning of o_O ?

Il lun 1 mag 2023, 02:04 Tomek= CEDRO <tomek@cedro.info> ha scritto:
o_O

--
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info

On Sun, Apr 30, 2023, 23:42 Matthew Grooms <mgrooms@shrew.n= et> wrote:
On 4/29/23 06:17,= void wrote:
> Hi,
>
> Where can I read up about the recently introduced BHYVE_SNAPSHOT ?
>
> I can only find terse information about what it does (in man 5 src.con= f)
> but not about how to use it and/or why.

This began as sponsored student work with the Politehnica University of Bucharest back in 2016 with an eye toward developing Live MIgration as a feature. The initial bhyve save/restore patch was committed back in 2020 ..= .

https://reviews.freebsd.org/rS3= 60648

Student projects related to bhyve continued until last year, but we gave up due to there being such a massive accumulation of patches with no
path to getting them committed. We tried reaching out to project
maintainers and the FreeBSD foundation alike for support and feedback,
but there seems to be little to no interest. There have also been
attempts by companies that rely on these features to and get them
improved and committed. Unfortunately they don't appear to be having much luck either ...

https://reviews.freebsd.or= g/D38858#885651

Would you like to see support for VM snapshots in the generic kernel?
How about support for saving/restore checkpoints using QCOW2, VMDK via
libvdsk? How about support for warm or live migration? How about USB
device pass-through? There are experimental patches for all these
features that were developed by students at UPB. In a lot of cases,
there are open reviews that have been waiting on feedback for ages. Here is a a presentation given by Elena from UPB just last month at the
FreeBSD devsumit in Tokyo ...

https://wiki.freebsd.org/D= evSummit/202303?action=3DAttachFile&do=3Dview&target=3DPresentation= +-+bhyvecon.pdf

The case is quite plain. I'm not sure what the solution is to this
problem. I'd love to hear feedback from the community about how I'v= e got
this completely wrong and how the course could be corrected. That would be something.

-Matthew



--000000000000fca5b605fa977e0b-- From nobody Mon May 1 01:13:08 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q8lZp2fCdz48YfR for ; Mon, 1 May 2023 01:13:22 +0000 (UTC) (envelope-from tomek@cedro.info) Received: from mail-yb1-xb36.google.com (mail-yb1-xb36.google.com [IPv6:2607:f8b0:4864:20::b36]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q8lZp15dGz4l2D for ; Mon, 1 May 2023 01:13:22 +0000 (UTC) (envelope-from tomek@cedro.info) Authentication-Results: mx1.freebsd.org; none Received: by mail-yb1-xb36.google.com with SMTP id 3f1490d57ef6-b9d8730fe5aso3109652276.1 for ; Sun, 30 Apr 2023 18:13:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cedro.info; s=google; t=1682903601; x=1685495601; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=Aq7V3fWEnjCs/Oidw/9YQ0jLto133q9eo2LE+dD8358=; b=S6EwT0kTjPf2vpOaoq/XKm5HJAenAcTe5VFMkYwIYXVnEV4BO0LnIlAU/mUXhjnHaS edSpI8pAUD0oAvzGDFxLavCkWVkpKbdLXB3cV5DNgzLSQ+eUfF6rZQRMSbBF+SRVLKwL tIn4sylCW51DuZmn42a7KcnbNHsRyN22OpVT3cni3alV+Khj8kjJRcvBCyT/mqpmVabk SR2yr4cNaqU87MKb9Ly0pXrkfRcUTVHQngginYO4bTvK2rWTeSXFEjlKFnZO5xS/qg0d vsC+mtLo/WtUkgAXVRJMFT+IhB2kr2yE+GbejcrEiySuefJvXy/ZtU+KciLIJFgypKl7 bFsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682903601; x=1685495601; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Aq7V3fWEnjCs/Oidw/9YQ0jLto133q9eo2LE+dD8358=; b=anewyQDADoS/sDToDmS7JXvuNFTT5gEX4MAXvjmcD1kORu006NS4OF+hTPc3/znfbP s6B3kIJm2xh3a/z4LW/ki4V+Bk45YjmVmf3oOXvQE+bmyXFFaQ/GT2yw1SNEJ1xuxXx6 fmhQaiixOCOLQtKqD3X/driFXUrQc/np7cwGHpMol0JQQ+jXPHZgE1Y1EvchRQvCoDCh ohAH17I/lD/6ZJzFaAlqX34x94+No/AbGjESt07WBcMl+5HidelvtdqaxnhiTEivZQPG 93jca3jqFyUwzo+5Ucr8GoOtSyuOZb9Eznlgrig9bL+MjFyIuLGjA8Nxdh4n0dQpYD9j /y5w== X-Gm-Message-State: AC+VfDw5XwgqDx/jvkQ6lWmhas6oQ1ATvK7crMQ3h8pvD+fg3iK6BiOl J0JG9xoaLihNjFx0Q1vm25Ke5w== X-Google-Smtp-Source: ACHHUZ6gFx8D0gifXHYVIHt2KvQcMDmInzsATHw/uR/WbetTGSEp+kL0+Vq31UVCvJsekh2r/VJgSQ== X-Received: by 2002:a25:aaab:0:b0:b9a:791f:284e with SMTP id t40-20020a25aaab000000b00b9a791f284emr10869978ybi.29.1682903601154; Sun, 30 Apr 2023 18:13:21 -0700 (PDT) Received: from mail-yb1-f178.google.com (mail-yb1-f178.google.com. [209.85.219.178]) by smtp.gmail.com with ESMTPSA id b185-20020a2567c2000000b00b8f46d74e5dsm1427707ybc.37.2023.04.30.18.13.20 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 30 Apr 2023 18:13:20 -0700 (PDT) Received: by mail-yb1-f178.google.com with SMTP id 3f1490d57ef6-b9a659bd135so3079919276.3; Sun, 30 Apr 2023 18:13:20 -0700 (PDT) X-Received: by 2002:a25:b782:0:b0:b95:2bd5:8f86 with SMTP id n2-20020a25b782000000b00b952bd58f86mr10184115ybh.26.1682903600313; Sun, 30 Apr 2023 18:13:20 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Tomek CEDRO Date: Mon, 1 May 2023 03:13:08 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: BHYVE_SNAPSHOT To: Mario Marietto Cc: Matthew Grooms , Graham Perrin , Ed Maste , FreeBSD Hackers , Virtualisation on FreeBSD , elenamihailescu22@gmail.com, Mihai Carabas , gusev.vitaliy@gmail.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4Q8lZp15dGz4l2D X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_RCPT(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Mon, May 1, 2023 at 2:46=E2=80=AFAM Mario Marietto wrote: > whats the meaning of o_O ? surprise / shock.. can't believe what i see.. in the context of "code is out there but never used" :-P --=20 CeDeROM, SQ7MHZ, http://www.tomek.cedro.info From nobody Mon May 1 01:33:18 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q8m1t0J0Bz48ZSY for ; Mon, 1 May 2023 01:33:22 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oa1-x34.google.com (mail-oa1-x34.google.com [IPv6:2001:4860:4864:20::34]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q8m1r6ZbRz3Dnr for ; Mon, 1 May 2023 01:33:20 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=HGDAazRP; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2001:4860:4864:20::34 as permitted sender) smtp.mailfrom=mjguzik@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oa1-x34.google.com with SMTP id 586e51a60fabf-187edc01fa3so756697fac.3 for ; Sun, 30 Apr 2023 18:33:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1682904799; x=1685496799; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=KBy6mO2frWOe5B/Gn/zSos1xlDijx2XqKxMh9ZdmsKY=; b=HGDAazRPrKJbcPrdQB699N5AFE1Z0igJqY613N4gJK1K8FL63fCb7ELh7YU85b+JWK v9EvdZflbPJCcVfl8wLxw4inFR+kqoeiErNZCaOA1RGoG1DXda4J3p/ulBZiTS5QfG3c Zk8/LL77kNv6vA4QxJIfD3P96KhPge2dEWWIY8TO4KKggHWKY0IghWnaJtXwPTYLmiUW QS1DJfbQ/5ANekSVP1ztWlfH5lilLAOcV93M4YFtIbDbPhOZJJ8aKNfApn0TaNyzl8tr Nwjv7C0356kzFYPG78RLv4np/4u6/2yEeQqHcylsX3OWZKSfX6KRO/nnoIQyDQZLNNh9 P+Mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682904799; x=1685496799; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=KBy6mO2frWOe5B/Gn/zSos1xlDijx2XqKxMh9ZdmsKY=; b=TKa/GRr/zF0YXnk6rTdwx5h1Fj27TfKXB1zKvh7NO423CfDfXvHQ4lNGGGv1GEbvKU 1EiSbwzETBhX1pJE/jdqTkGXIHNYeja4Gp8Ne44v03M9vOc1mjcEj36IhAeWaCBvu0zH +jDGnfM923/7LlLDlp8hjrZlOf4duuODwtS3SOPSJuQBvjCbo4R7NN0WYok+LpjCUrft m+WkKKsFv9w6ZGIS/uX9gWpUnqFiE7MGgjF/Uujo+tuFs9uo/6ExiNX8x+ciXk7awcH2 443IcThaRqeGHW+kI5aX8HF6m8CACfX8U9qYWDaKOdseCPNoFZLLisrTYomO92Q03SDV 8yFQ== X-Gm-Message-State: AC+VfDwS4OGqTeKXS5W6XreTbgS8Gq5sxUkMejulYdM8Cup7hHaTdj5F 1GY7JOVweva75cpE2he0GixogMnjnianmd87vS5Ux2sg X-Google-Smtp-Source: ACHHUZ53TSnzv5BEoG18Ad7ji2N/r7Izhk99ZEj61JvDsbc+9xwcCo/lVh0LdaaVtSUEXDZC3DoOh+hr1fs83obt8vk= X-Received: by 2002:a05:6808:4394:b0:38c:2394:da32 with SMTP id dz20-20020a056808439400b0038c2394da32mr5300460oib.46.1682904798763; Sun, 30 Apr 2023 18:33:18 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:a8a:1194:0:b0:4d4:94b:7266 with HTTP; Sun, 30 Apr 2023 18:33:18 -0700 (PDT) In-Reply-To: References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> <20230331215751.166a294f7382c85b545f53a2@dec.sakura.ne.jp> From: Mateusz Guzik Date: Mon, 1 May 2023 03:33:18 +0200 Message-ID: Subject: Re: Periodic rant about SCHED_ULE To: Tomoaki AOKI Cc: freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-3.00 / 15.00]; URI_HIDDEN_PATH(1.00)[https://people.freebsd.org/~mjg/.junk/ule-poc-hacks-dont-use.diff]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2001:4860:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2001:4860:4864::/48, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2001:4860:4864:20::34:from]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4Q8m1r6ZbRz3Dnr X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N On 5/1/23, Mateusz Guzik wrote: > On 3/31/23, Tomoaki AOKI wrote: >> On Mon, 27 Mar 2023 16:47:04 +0200 >> Mateusz Guzik wrote: >> >>> On 3/27/23, Mateusz Guzik wrote: >>> > On 3/25/23, Mateusz Guzik wrote: >>> >> On 3/23/23, Mateusz Guzik wrote: >>> >>> On 3/22/23, Mateusz Guzik wrote: >>> >>>> On 3/22/23, Steve Kargl wrote: >>> >>>>> On Wed, Mar 22, 2023 at 07:31:57PM +0100, Matthias Andree wrote: >>> >>>>>> >> >> (snip) >> >>> > >>> > I repeat the setup: 8 cores, 8 processes doing cpu-bound stuff while >>> > niced to 20 vs make -j buildkernel >>> > >>> > I had a little more look here, slapped in some hacks as a POC and got >>> > an improvement from 67 minutes above to 21. >>> > >>> > Hacks are: >>> > 1. limit hog timeslice to 1 tick so that is more eager to bail >>> > 2. always preempt if pri < cpri >>> > >>> > So far I can confidently state the general problem: ULE penalizes >>> > non-cpu hogs for blocking (even if it is not their fault, so to speak) >>> > and that bumps their prio past preemption threshold, at which point >>> > they can't preempt said hogs (despite hogs having a higher priority). >>> > At the same time hogs use their full time slices, while non-hogs get >>> > off cpu very early and have to wait a long time to get back on, at >>> > least in part due to inability to preempt said hogs. >>> > >>> > As I mentioned elsewhere in the thread, interactivity scoring takes >>> > "voluntary off cpu time" into account. As literally anything but >>> > getting preempted counts as "voluntary sleep", workers get shafted for >>> > going off cpu while waiting on locks in the kernel. >>> > >>> > If I/O needs to happen and the thread waits for the result, it most >>> > likely does it early in its timeslice and once it's all ready it waits >>> > for background hogs to get off cpu -- it can't preempt them. >>> > >>> > All that said: >>> > 1. "interactivity scoring" (see sched_interact_score) >>> > >>> > I don't know if it makes any sense to begin with. Even if it does, it >>> > counts stuff it should not by not differentiating between deliberately >>> > going off cpu (e.g., actual sleep) vs just waiting for a file being >>> > read. Imagine firefox reading a file from disk and being considered >>> > less interactive for it. >>> > >>> > I don't have a solution for this problem. I *suspect* the way to go >>> > would be to explicitly mark xorg/wayland/whatever as "interactive" and >>> > have it inherited by its offspring. At the same time it should not >>> > follow to stuff spawned in terminals. Not claiming this is perfect, >>> > but it does eliminate the guessing game. >>> > >>> > Even so, 4BSD does not have any mechanism of the sort and reportedly >>> > remains usable on a desktop just by providing some degree of fairness. >>> > >>> > Given that, I suspect the short term solution would whack said scoring >>> > altogether and focus on fairness (see below). >>> > >>> > 2. fairness >>> > >>> > As explained above doing any offcpu-time inducing work instantly >>> > shafts threads versus cpu hogs, even if said hogs are niced way above >>> > them. >>> > >>> > Here I *suspect* position to add in the runqueue should be related to >>> > how much slice was left when the thread went off cpu, while making >>> > sure that hogs get to run eventually. Not that I have a nice way of >>> > implementing this -- maybe a separate queue for known hogs and picking >>> > them every n turns or similar. >>> > >>> >>> Aight, now that I had a sober look at the code I think I cracked the >>> case. >>> >>> The runq mechanism used by both 4BSD and ULE provides 64(!) queues, >>> where the priority is divided by said number and that's how you know >>> in which queue to land the thread. >>> >>> When deciding what to run, 4BSD uses runq_choose which iterates all >>> queues from the beginning. This means threads of lower priority keep >>> executing before the rest. In particular cpu hog lands with a high >>> priority, looking worse than make -j 8 buildkernel and only running >>> when there is nothing else ready to get the cpu. While this may sound >>> decent, it is bad -- in principle a steady stream of lower priority >>> threads can starve the hogs indefinitely. >>> >>> The problem was recognized when writing ULE, but improperly fixed -- >>> it ends up distributing all threads within given priority range across >>> the queues and then performing a lookup in a given queue. Here the >>> problem is that while technically everyone does get a chance to run, >>> the threads not using full slices are hosed for the time period as >>> they wait for the hog *a lot*. >>> >>> A hack patch to induce the bogus-but-better 4BSD behavior of draining >>> all runqs before running higher prio threads drops down build time to >>> ~9 minutes, which is shorter than 4BSD. >>> >>> However, the right fix would achieve that *without* introducing >>> starvation potential. >>> >>> I also note the runqs are a massive waste of memory and computing >>> power. I'm going to have to sleep on what to do here. >>> >>> For interested here is the hackery: >>> https://people.freebsd.org/~mjg/.junk/ule-poc-hacks-dont-use.diff >>> >>> sysctl kern.sched.slice_nice=0 >>> sysctl kern.sched.preempt_thresh=400 # arbitrary number higher than any >>> prio >>> >>> -- >>> Mateusz Guzik >> >> Thanks for the patch. >> Applied on top of main, amd64 at commit >> 9d33a9d96f5a2cd88d0955b5b56ef5058b1706c1, setup 2 sysctls as you >> mentioned and tested as below >> >> *Play flac files by multimedia/audacious via audio/virtual_oss >> *Running www/firefox (not touched while testing) >> *Forcibly build lang/rust >> *Play games/aisleriot >> >> at the same time. >> games/aisleriot runs slower than the situation lang/rust is not in >> build, but didn't "freeze" and audacious normally played next music on >> playlist, even on lang/rust is building codes written in rust. >> >> This is GREAT advance! >> Without the patch, compiling rust codes eats up almost 100% of ALL >> cores, and games/aisleriot often FREEZES SEVERAL MINUTES, and >> multimedia/audacious needs to wait for, at worst, next music for FEW >> MINUTES. (Once playback starts, the music is played normally until it >> ends.) >> >> But unfortunately, the patch cannot be applied to stable/13, as some >> prerequisite commits are not MFC'ed. >> Missing commits are at least as below. There should be more, as I >> gave up further tracking and haven't actually merged them to test. >> >> commit 954cffe95de1b9d70ed804daa45b7921f0f5c9da [1] >> ule: Simplistic time-sharing for interrupt threads. >> >> commit fea89a2804ad89f5342268a8546a3f9b515b5e6c [2] >> Add sched_ithread_prio to set the base priority of an interrupt >> thread. >> >> commit 85b46073242d4666e1c9037d52220422449f9584 [3] >> Deduplicate bus_dma bounce code. >> >> >> [1] >> https://cgit.freebsd.org/src/commit/?id=954cffe95de1b9d70ed804daa45b7921f0f5c9da >> >> [2] >> https://cgit.freebsd.org/src/commit/?id=fea89a2804ad89f5342268a8546a3f9b515b5e6c >> >> [3] >> https://cgit.freebsd.org/src/commit/?id=85b46073242d4666e1c9037d52220422449f9584 >> >> -- >> Tomoaki AOKI >> >> > > Hello everyone. > > I sorted out a patch I consider comittable for the time being. IT IS > NOT A PANACEA by any means, but it does sort out the most acute > problem and should be a win for most people. It also comes with a knob > to turn it off. > > That said, can you test this please: > https://people.freebsd.org/~mjg/ule_pickshort.diff > > works against fresh main. if you are worried about recent zfs woes, > just make sure you don't zpool upgrade and will be fine. > Here is an updated patch: https://people.freebsd.org/~mjg/ule_pickshortv2.diff if you are getting bad results, do: sysctl kern.sched.preempt_bottom=0 and try again. thank you. -- Mateusz Guzik From nobody Mon May 1 01:57:16 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q8mYk6lJXz48bjp for ; Mon, 1 May 2023 01:57:30 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic306-21.consmr.mail.gq1.yahoo.com (sonic306-21.consmr.mail.gq1.yahoo.com [98.137.68.84]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q8mYk2HKhz3Gmm for ; Mon, 1 May 2023 01:57:30 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1682906249; bh=/cmO4ZCOIvGXCD1Fw2ps4OdrhftESgdb7dAOMUjlo6c=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=p8NJJ8Qn6sildfLBF4+3ZcGc7HY+g1R2EgcciFY+xC+uGLLVOR0KfhdYv0e75xkoxXEKJ25cGxZ47PivLzSWb6FElctekK37qMya1kdmfufaWMoW4O8UfDjolQTl9s3DOyBGbf0hZMU/QC9Y2toxqY6/7OVQUmyC+srPgH8Y1YPzprHL3GUG4zOgmb7by9hdrNzb4wBa6xfMH/KExrhWMXX67NHLrN+WjD0iNXacMPHqxRkJh8SO09Q+sHGZe8XDFBrSjrpGEwiWNvFmvxWULMYkftUdd73zKhmxnxxBatsGSEpGNdq6DjtcISIoZ9HyHb+rspkkPTszCFISQWz7Iw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1682906249; bh=IcuLuPRM6Qx6D0iXZoRNleRklEyw1cQ7kmQHLDZt0mv=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=tStQzEkmf4UUpaAbB6aZRXBLMImmFHFcUdGVMzsJIdlE3fPLF5jGcNNSaRQo3HNr0UykCBbVVE+yP7rpl01ORIGiiM6B/5+WdgTC3nnsJENATkb7Kqr+Mw8mQuGslXHUMTW6qQJF4T1fQIOqVZ/lDf9m/KjQPKxc6s9Xu8mYfVOIZN2YjKHdU3rPq9sJ4fnwyCgfKtxKEhKp4ntvhJGPvuGxIwmleBV1dV/JQDkUIu0f0y9N0eUVbdK54hFwXd/3qGpQf5cbT2eUDHgruXAr0j7x7p+uS7knVQ/Q7/JzxzDBCgWiOjKnH3p3VyxyqX/X2mRxiqBNTkX4JBJMCJjA5A== X-YMail-OSG: .MpGC0EVM1lsBfC768LJ4BBmnh5_Q0CBQQSdx9k.E4NFGr285ZVBbl56PbCpy49 T09EvIc4JuojdTq6Gv.zUKIEsEpJD6a4kxYxSlVS8anIxGND3OnOdQkbgN092wWxYhyhZquzxxS0 M1US_zWdnwlBfQXAPPuLziXo.ykFClsNdVEG5347ul9f.XoJejsC0.KWtWvtUIVqc1XZDPmXyZ8m 9PHDHuftbjg5zDfinBzSSAhpeCibcvc4ehd24ePmwV1HjSeW1cD7b2aQs0ORFf1xUlr5613YsTr8 JfZnCx_1Wiszq5h1tVbu_Lc1.JdBE9mI7L6Lu57c37a4_n3RHQrW4Vzp_04TK2KbXOT8tgZ7.46R dfUIF8ZZjQexmHpt417su9r.U0bgraPVUB4BYC5WPCnc0UGhqg0ZqxsPRCsVwoNXpBp76WS7_plT wQcn9BhwhGlZ0YY0bPKVf4HX_4xlFJWc6hxmYf_M63Vz03iD2KPeqbVBS9lbeIz7c1ym55PE9mKB OhPSG0VN0X2E0UCOtSCMFxqyjppaLgN5EA2WfFwwiFBRxvEvpmsaQivr0htWNsrflHYmLlERLdBT .VrC27x.ZA4_HQp00srAY4Gb1XEn03kiBcKqMMpDYFMtqIbGHf3rvj0mv9Sxo5xDBqnDP1bIFqSk z4irFtqVkYx.QNIFICVTX2mbChpLE17DJjUWlOtm1oyoHqxXlx0KvkfDP635Zu1K7l30gGGTE9gt rSTm4y5AhHSBQqQWfSPpU5Sx_8ljZQzEM2qRwvNg05zjMRN__jQESYWLI7Phc_rcjCKbNwpJChd2 RR.t7kskIa5.cld8z2Fq1XnTHs2kQDNJZEOCAbTmXnJ.1gVmYsrMLuQ0YpY6rR2FT8WsFoFg2CqT wTkhCwSSlj4viYjPRsJPThmlKdvoQc56LAFvq.OZdT8k3ANmht1C1u3IrVqcyvvO6MwPWueCvTwK sqdlKTuMC6oD01TXm7OS2WT1xYjfgoAzi8nUbnF9n2UytD.mEgGQ.X.q76BfAFv2qTzEWWr.VWUY mZNFO5khCRjFyDmlxciD31SqLrJ_1M4gCqmkYPeKraZ7AJvny1tkKQGE0Vu7_xqEL5Se6v3Ceufo 5Hd0YpItb8ljNrwbt0Rx47eUlgjRP6HHHhXXua.EdhvAMLbngbtO5v1MYsIKLO9fXKBxBNgyd6pj AqgUifECsbqdNVf5nWEU0YDSZyMzj2020o6R2RuO5g44.3Dew.pV0oBpXFzaS0qvHqIV7I1rF4Qc jFrEFGaQKYKXTTvgh_MuPYyjTO2cCeqXB9OY.bnOwWZl7mTjxmU6Kb.08hMRv3db2chrGCEmx8aV .yrn8P2o7sXIRut4KXt40kK2rCiMIUgJANj6msTdYmTxGAWyp77pY4ovSK.bHnHOHhVN_fEa5YZC qw0Ujmr..lb2S6FLafibS3F5n_lOb5l4Fxt3lvWTGGR3ecu1TnDPaygyQm31NVoY_LQM8nsBGDT0 v8aTOxPF0yb_6XE7.tAjzsxu21C2KsTFZAzpZtnMptPPIZx5IlzNEMT3QeI_hv.3wJVkcaa0usmH zRr.lz7nwdsbCC1I0STxZ_gG2jn3Ifg2U_y7c8zM7DVqcCp_GDxiogQVR2EbyZ55K525o2moIljC c4JNdAm3YcMd94luW4ohync5KTy0zzZKzAox1ODIXIJy3INd8vf4zEhoEVaA22WDN78W05YfFWcS IKIXGE3fS8qI6LgwCSVnUi4V_Wmwxuavpy40hPS40C7qjAttytk_PtcDtOjFBY7gAQB4ELAgiKzr dj7rj3xeJ6A5vygfSuZsZoJN5Q1O1LxVM4HTU0pKb7X.jjFe66Jf3N2N0MPHG1kMjGgj2WD1yfo5 8na3J9dnv9k_JsI.QHZZS_jl8wMr.g0HQB.9fOIqDXySjnHJnP6B15BmGy0rpOfpPd1TWCPiWYuG 8xmyRX6TF79uKS9BfXMWR5.ZxtT7z8sqRIFF8fqFeYrL5.nYxXg.sS0YY0u407ZsBzgVrl83vwkW Hmiu6PUkwfhRzl_6cL29g5Pc6ExDEq2NgF9YzjEZk8W7ZbdCPSd_IHPFqWoqyndUFEVCdEvAtlvc 95k8CQinpmdjKI5SwX994pCWMGNmUSZd3x8KDLsnAzRfsakbvX0mYayXuy.7cBT2QyuIb4Ugbd4b y86NMG2gRJ5c_ybmyeb39zJID8DLj3MAMASuKjUlotPqIhqXnIsCyM4AlKp7uhexdcK6USg8W4c1 YsKXeubGKC1zNEpdGoyIMzjagzCkr4LZlhCqlVE2wTfve51eHOmss8DinHwXUfwml42UVxhA9OYm 1 X-Sonic-MF: X-Sonic-ID: 73c7f526-a957-41be-ac47-57f7d13d9aad Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.gq1.yahoo.com with HTTP; Mon, 1 May 2023 01:57:29 +0000 Received: by hermes--production-gq1-546798879c-qx24x (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID b5c8fa2a09aca77006c3b1692a533c4b; Mon, 01 May 2023 01:57:27 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: armv8.2-A+ tuned FreeBSD kernels vs. poudriere bulk and USB3 media: tx->tx_quiesce_done_cv related blocking of processes? From: Mark Millard In-Reply-To: Date: Sun, 30 Apr 2023 18:57:16 -0700 Cc: FreeBSD Hackers , freebsd-arm Content-Transfer-Encoding: 7bit Message-Id: References: <7AE28A5B-109E-4C26-9D70-BCA5D49CD79D@yahoo.com> <02DC03AE-E082-4FB5-AA0D-396F64CC23CB@yahoo.com> To: Mateusz Guzik X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Rspamd-Queue-Id: 4Q8mYk2HKhz3Gmm X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Apr 30, 2023, at 17:44, Mateusz Guzik wrote: > can you redo zfs test with: > sysctl vfs.zfs.per_txg_dirty_frees_percent=5 Sure. Result summary: Seems to have avoided the sustained periods of low load average activity. Much better for the context. Context: Original ZFS USB3 media. World or Kernel in use had been built (non-debug style) using: -mcpu=cortex-a78C+flagm+nofp16fml Steps for this test . . . # poudriere pkgclean -A -jmain-CA78C . . . # sysctl vfs.zfs.per_txg_dirty_frees_percent=5 vfs.zfs.per_txg_dirty_frees_percent: 30 -> 5 # grep USE_TMPFS= /usr/local/etc/poudriere.conf # EXAMPLE: USE_TMPFS="wrkdir data" USE_TMPFS="data" #USE_TMPFS=all # poudriere bulk -jmain-CA78C -w -f ~/origins/CA78C-origins.txt . . . At 15 minutes into the build: 46 ports in 1st 15 minutes. Load average stayed reasonable for the configuration. At 30 minutes into the build: 102 ports in 1st 30 minutes. Load average still reasonable for the configuration. Looks good compared to before. I've no clue what optimal would be for the context, but vfs.zfs.per_txg_dirty_frees_percent=5 is vastly better for the context than the default 30 was. Thanks. I'm going to stop the test and do the conversion to the U2 960GB Optane media in the USB3 adaptor and then compare USE_TMPFS=data vs. USE_TMPFS=all --but using your vfs.zfs.per_txg_dirty_frees_percent=5 assignment. > On 5/1/23, Mark Millard wrote: >> As the evidence this time is largely independent of the details >> reported previousy, I'm top posting this. >> >> The previous ZFS on USB3 results were based on poudriere using >> "USE_TMPFS=data", meaning that almost all file I/O was via ZFS >> to the USB3 media. >> >> The UFS on U2 960GB Optane via USB3 adapter results did not not >> suffer the reported problems, despite "USE_TMPFS=data". (I let >> it run to completion.) But this had both a media and a file >> system difference. >> >> This time the results are for just changing poudriere to use >> "USE_TMPFS=all" instead but back on the original ZFS on >> USB3 media. The implication is that the vast majority of the >> file I/O is not via ZFS to the USB3 media. >> >> The context has 32 GiBytes of RAM and about 118 GiBytes of >> swap/paging space. It would need to page if pet run to >> completion. >> >> Observing, the load average is behaving normally: Things are >> not stuck waiting. "gstat -spod" indicates the ZFS I/O is >> not sustained (no paging in swap space use yet). >> >> First 1 hr: 262 ports built. >> >> But this had both a media and a file system difference again. >> >> I'm stopping after this, in order to set up the next just- >> ZFS experiments. >> >> >> Next experiments: >> >> I expect to move the ZFS context to U2 960GB Optane media >> used with the USB3 adapter and to test both "USE_TMPFS=data" >> and "USE_TMPSF=all", probably letting USE_TMPFS=all run to >> completion. >> >> If Optane based USE_TMPFS=data context still has the problem, >> even the more performance media would have been not enough to >> avoid what would then appear to be a ZFS problem, two other >> file systems not having the problem. >> >> The Optane based USE_TMPFS=all context should just handle the >> paging and more rare ZFS I/O quicker. I do not expect problems >> for this combination, given the UFS on Optane USB3 results >> and the partial USE_TMPFS=all non-Optane USB3 results. Even >> with ZFS working, this likely is the more performant type of >> context for the Windows Dev Kit 2023, given that I'm leaving >> Windows 11 Pro in place on the internal media. >> >> >> Hypothesis for the original problem: >> >> I wonder if ZFS write activity to the USB3 media was largely >> blocking the ZFS read activity to the same media, causing >> lots of things to have to spend much time waiting for data >> instead of making progress, leading to long periods of low >> load averages. >> >> >> Older material: [I've removed the older material.] === Mark Millard marklmi at yahoo.com From nobody Mon May 1 02:31:23 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q8nJj4WDgz48d9l; Mon, 1 May 2023 02:31:17 +0000 (UTC) (envelope-from rob.fx907@gmail.com) Received: from mail-ej1-x629.google.com (mail-ej1-x629.google.com [IPv6:2a00:1450:4864:20::629]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q8nJj44XLz3KF7; Mon, 1 May 2023 02:31:17 +0000 (UTC) (envelope-from rob.fx907@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x629.google.com with SMTP id a640c23a62f3a-94a342f4c8eso61561466b.0; Sun, 30 Apr 2023 19:31:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1682908276; x=1685500276; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=OZb/SU58VWwmR4byX4uYgVx8EtL+8JbEAI9GQuJXP3o=; b=rCaofLwsLkWba8NXiEaA58TiFBtfk+WxQNr5CkEBwVxRJjnX9QFTetqOFMl0cgbvw1 hu5mRchRnsCyOpUv6XBxCyhrtpK56kq8usE3KRPlGQzwCyLHsI7Jizebo5V4vHgNGhyc 69MGQlYg8K6HLQYdtUNpHvRjG/8k4G+36/X5C2N966CBdgAgyXKppvjKUC2/AnTrNCjX YxC2peMaSHaqBc340AKW93LOsyt0iI3awRhSad5Yj72miCku4Zf0z3T7i3s1Ni7hbktx tleFhuRpdpuJ4EC79pJ62ct9moBw6EQOnlnqd8QWGKCDhlel6gNim9Aff4S/0cXeyGRd M14A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682908276; x=1685500276; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=OZb/SU58VWwmR4byX4uYgVx8EtL+8JbEAI9GQuJXP3o=; b=l88plTLx/H/iwS0BihqK4k5rSYD0aMJ4ihn15K8mXvzzwYQwm1r5j9ukYYH8W2BRa0 uAN2+LM4K2CqgjcpGziyYCIm7Z+PqiWTevd0OasQuEMTqRq+rkPBeSvQwFOMJ9vfc9Mh Rm86OYie0yMvZx08d3LgNkI+NKjwwpGUl2o1dYslx2q4Y/I6v4cr6T9G08cfwbvT3oAo mc4g/HHao+n+fv8+g8oVq529uKFqlZ5OgEapN2MgHgPIByQKHVo/n3KGbqVvz5NMeu+1 xAOB8qYANb3Q0PCWxCdzKhFlUADWzqxhKy2gHEns0OhCkITYloHvrNQmnGtz8oWPgtqB cKSQ== X-Gm-Message-State: AC+VfDx8bT5e1I2PC3u5Fg8mxPbJLtcBwZWXA9fwuVbzdNcePZYfeVqj kHMwEVnrTGyAZFRmoyB27ywWhQU9q7ShO1i9Y/E= X-Google-Smtp-Source: ACHHUZ4DYbtAiLv8F3woIC1hFX3lqGnicM5Cuqon6VPc7QiMv9E+3jcjeNd0Yms+G10V+hfMHvOM7kef8G6iOegbPR0= X-Received: by 2002:a17:906:7483:b0:94e:8b6c:462c with SMTP id e3-20020a170906748300b0094e8b6c462cmr7948246ejl.2.1682908275818; Sun, 30 Apr 2023 19:31:15 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Rob Wing Date: Sun, 30 Apr 2023 18:31:23 -0800 Message-ID: Subject: Re: BHYVE_SNAPSHOT To: Matthew Grooms Cc: freebsd-hackers@freebsd.org, freebsd-virtualization@freebsd.org, elenamihailescu22@gmail.com, Mihai Carabas , gusev.vitaliy@gmail.com Content-Type: multipart/alternative; boundary="000000000000f8434305fa989d84" X-Rspamd-Queue-Id: 4Q8nJj44XLz3KF7 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_RCPT(0.00)[]; TAGGED_FROM(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --000000000000f8434305fa989d84 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hey Matthew, On Sun, Apr 30, 2023 at 1:41=E2=80=AFPM Matthew Grooms = wrote: > > Would you like to see support for VM snapshots in the generic kernel? > Is there a review open that addresses the limitations described in the commit message that brought the snapshot feature in? https://github.com/freebsd/freebsd-src/commit/483d953a86a2507355f8287c5107d= c827a0ff516 > How about support for warm or live migration? This builds off the snapshot work, right? Seems like it'd make more sense to address the current limitations of the snapshot code before extending the functionality off the top of it. > There are experimental patches for all these features that were developed > by students at UPB. In a lot of cases, there are open reviews that have > been waiting on feedback for ages. > In general, most people don't want to review large experimental patches. > The case is quite plain. I'm not sure what the solution is to this > problem. I'd love to hear feedback from the community about how I've got > this completely wrong and how the course could be corrected. That would > be something. > My perspective is that it would have been better to focus student efforts on completing the snapshot feature. By completing the snapshot feature, I mean getting the code into a state where it's compiled in by default and no longer considered an experimental feature. -Rob --000000000000f8434305fa989d84 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hey Matthew,

=
On Sun, Apr 30, 2023 at 1:41=E2=80=AF= PM Matthew Grooms <mgrooms@shrew.ne= t> wrote:

Would you like to see support for VM snapshots in the generic kernel?

Is there a review open that addresses the li= mitations described in the commit message that brought the snapshot feature= in? https://github.com/freebsd/freebsd-src/commit/= 483d953a86a2507355f8287c5107dc827a0ff516
=C2=A0
How about support for warm or live migration?

This builds off the snapshot work, right? Seems like it'd make more = sense to address the current limitations of the snapshot code before extend= ing the functionality off the top of it.
=C2=A0
There are experimental patches fo= r all these features that were developed by students at UPB. In a lot of ca= ses, there are open reviews that have been waiting on feedback for ages.

In general, most people don't want to= review large experimental patches.
=C2=A0
The case is quite plain. I'm not sure what the solution is to this
problem. I'd love to hear feedback from the community about how I'v= e got
this completely wrong and how the course could be corrected. That would be something.

My perspective is that it= would have been better to focus student efforts on completing the snapshot= feature. By completing the snapshot feature, I mean getting the code into = a state where it's compiled in by default and no longer considered an e= xperimental feature.

-Rob
--000000000000f8434305fa989d84-- From nobody Mon May 1 05:16:12 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q8rzQ07S1z47nbM; Mon, 1 May 2023 05:16:34 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: from mail-lf1-x12b.google.com (mail-lf1-x12b.google.com [IPv6:2a00:1450:4864:20::12b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q8rzP4Rcwz3mCh; Mon, 1 May 2023 05:16:33 +0000 (UTC) (envelope-from sodynet1@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x12b.google.com with SMTP id 2adb3069b0e04-4edc63c82d1so2509102e87.0; Sun, 30 Apr 2023 22:16:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1682918190; x=1685510190; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=pnVj+pSJoDZKyly8PkPUhqYDsc/T9aC1aWM1MTY8n2c=; b=ZmeMRSpMIMeI6IRBCjg6/tW4oHYnn/B9yzJgat4wWon/un/cDoHPRQpqvGO8wLIxT1 bMUthv0ry6TW7VnMHD1tQMcm/BIr50u1ZUnCnsLc3ORlEVqLw0JOeSMDUge4LtScEr0V cgX6cVK0OPUYmlV+z4F86MfZhG4xxM3mvJ3TtUH4fGEr2uO/e/KXjQUN5XuoDGICgn7t 2YnaQUtpP1qUSUoCegZBuk1RS6nCkPEhzVPIUBh7KekD9iH8gjryraG6fWpNVoyFMi/Q KanTY337eby6GrKsyv5C+RU8CN6kjUUV0d5QMSWayDiPYKr++9TCJSsX6nN6BFKkxZLl d/2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682918190; x=1685510190; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=pnVj+pSJoDZKyly8PkPUhqYDsc/T9aC1aWM1MTY8n2c=; b=QQWBWk9ZcEMPamgFbQUqo8+t8aaikCjUmbweZV9LEGOozj1V9ylAv2E6+9GFvkGYeq Drn8dk56/+kURMPZBdCWrLNflv5jyhNwE9rXGPe4+czHV5DL1b/tLviH37BTCra3TOvc j75LWvBJQsfhJUu1C2dLd3TpTovlUrQq4bjNcnX9tgwL05UCkhyy2KmBM0eYuappxeqG 5dpmKz9h6kixFTe83Y/P/T0s9lbMBf7YRzOjPrvVWWGbK9zB0MUGWDiExsB2XJ6CWgBE CYx+Rf/TQE5BMrUYVeaJf9C/X4TbV88yBjwNnWnJAp1aHNdn98LJgtPjIPXDw6XoMpVD J/nQ== X-Gm-Message-State: AC+VfDzsxoIdks3DloXDcrygWeXYgaJ+U70w/Nx1tWg4dQwoT7TejmD4 fY4mGx0RF/fAZpsrzgdTTCVlFN4COGvveUpePFpRiMNg X-Google-Smtp-Source: ACHHUZ7ohywURRa62/dhS03fGOWptb+Yi+GLZo5lLWb1hwA35SvVNw+BqlDBlCkczBi0XsIgFC9sM2oNbTOJnhoJ9wE= X-Received: by 2002:ac2:5584:0:b0:4dd:a212:e3ca with SMTP id v4-20020ac25584000000b004dda212e3camr3865536lfg.11.1682918189796; Sun, 30 Apr 2023 22:16:29 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <226afa3a-114b-6740-36b1-ebfee4931a22@freebsd.org> In-Reply-To: <226afa3a-114b-6740-36b1-ebfee4931a22@freebsd.org> From: Sami Halabi Date: Mon, 1 May 2023 08:16:12 +0300 Message-ID: Subject: Re: FreeBSD #ai-ml (was: How do I get a new freebsd- mailing list made) To: Graham Perrin Cc: FreeBSD questions , FreeBSD hackers Content-Type: multipart/alternative; boundary="000000000000e391f705fa9aec6e" X-Rspamd-Queue-Id: 4Q8rzP4Rcwz3mCh X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --000000000000e391f705fa9aec6e Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi, how to join freebsd discord server? IN the wiki all invited seem to be invalid/expired. Sami =D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A =D7=99=D7=95=D7=9D =D7=90=D7=B3, 30 = =D7=91=D7=90=D7=A4=D7=A8=D7=B3 2023, 22:28, =D7=9E=D7=90=D7=AA Graham Perri= n =E2=80=8F< grahamperrin@freebsd.org>: > I can no longer find parts of this thread (sorry). > > For those of you who are interested, there's an #ai-ml channel in > Discord for FreeBSD: > > > "Artificial Intelligence" - "Machine Learning" - and related computer > > sciencey things. > > --000000000000e391f705fa9aec6e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,
how to join freebsd discord server?<= div dir=3D"auto">IN the wiki all invited seem to be invalid/expired.
<= div dir=3D"auto">
Sami

=D7=91=D7=AA= =D7=90=D7=A8=D7=99=D7=9A =D7=99=D7=95=D7=9D =D7=90=D7=B3, 30 =D7=91=D7=90= =D7=A4=D7=A8=D7=B3 2023, 22:28, =D7=9E=D7=90=D7=AA Graham Perrin =E2=80=8F&= lt;grahamperrin@freebsd.org= >:
I can no longer find parts of= this thread (sorry).

For those of you who are interested, there's an #ai-ml channel in
Discord for FreeBSD:

> "Artificial Intelligence" - "Machine Learning" - a= nd related computer
> sciencey things.

--000000000000e391f705fa9aec6e-- From nobody Mon May 1 18:14:49 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9BFX15KGz48qS3; Mon, 1 May 2023 18:14:56 +0000 (UTC) (envelope-from gjb@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9BFX0SJcz4BpD; Mon, 1 May 2023 18:14:56 +0000 (UTC) (envelope-from gjb@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1682964896; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type; bh=982sUHCVyS2oRa34lrZYsnJAj5HelSWghZYUY1qthSo=; b=e2uyA5Q6AD9bbISlvsMI0vyfsPzicaOC7XD5E+MKNMw+fJbaWceMZvlzdSje7Fte3zvK0P FrlCBGI5aTy34N3Dz+5wTSFfvK4ynDogYuDHU07+4H7Pe9GLunqGYvuZ8wa+qJFB5y4jZQ AGgXtDITDYY2ipt7MbmFd5D8qPuzQPrlHrq722VREPrjQtFcXK8wHfZW5+AFgs1cVtj8CU /zEVQmHdbUwT7KG2KE8HJtXHJ6rFxo+joUyYMp3wE6WUWz0Q+FZejk4///8wSc9GK+gHov MNMhEr8Ft5tPFvz+42X4K0aJtvoKtFtcM2+CEf+K7Vhgc9D76xyG0wdHlf/Ujw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1682964896; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type; bh=982sUHCVyS2oRa34lrZYsnJAj5HelSWghZYUY1qthSo=; b=ooBppsiqA/6CY/P+1fmTIFw3W2g+8gW2ykyuUczDT6bUBaTrLiwPe1BtbX9XMx/RefFJSD YIYGtL9gJhFcXL13E8xQzFj0ExuMg/PVctVfqKyxLcR4rQ9H5O/2y3MfrAvrQGO9ix9YNo PiLGJXHVlXPmEsbwXpWA3bksTjNwPL4gedhtJ/Ml8WQlUHKoYhXmPbwc0yA5R0JDRT6glF mjKp5kjfVAzK5hc5fJ9fBKwTTTTY/oV6h3/UC79lKfAU7vl3dml0wxo/Hk/rSGFl6Eqr9V JDaZX6gKs+ZlewGC2KUYJ7sfJ/DM6WW75HHBTeNBhQdJ0Emz9KVYqQ9DrnazAg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1682964896; a=rsa-sha256; cv=none; b=Nt/9fSS1IBJon5alsFoswCLb7BHObHOmSLrR3nV+C6xhEq6d2Ss6idHhxCS9YUAuNLzgob I8yyLUI2FYNsTLc7JD3u8OlhqYwLC+Xg34gzjSLVwXqKWAfD094VGfo6/eyGvoAPyd5zlp qyZNnzE6ZzTiEBk+GB5JPLibv5AI9tUA/MBntGBrlg9mL5nLPnfLruni/ClATJVupto0bD LLiHWMCsQzJVm5rEHSpfA8axTMBJVc3E8G1XQ5/OeWb4XR6VwbkUgo++YHinJWTHZEskMG Y7N0OpSrAtriGeIwd0VfjHeKglNE5kiqOzDVztsGAaUU3ntuEj3EvGlG+5+ppg== Received: from FreeBSD.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by freefall.freebsd.org (Postfix) with ESMTPS id 87671B0E9; Mon, 1 May 2023 18:14:55 +0000 (UTC) (envelope-from gjb@freebsd.org) Date: Mon, 1 May 2023 18:14:49 +0000 From: Glen Barber To: freebsd-arch@freebsd.org, freebsd-hackers@freebsd.org Cc: freebsd-current@freebsd.org, FreeBSD Release Engineering Team , FreeBSD Security Team Subject: Delay in 14.0-RELEASE cycle and blocking items Message-ID: <20230501181449.GJ1219@FreeBSD.org> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="cyV/sMl4KAhiehtf" Content-Disposition: inline X-ThisMailContainsUnwantedMimeParts: N --cyV/sMl4KAhiehtf Content-Type: text/plain; charset=us-ascii Content-Disposition: inline According to the 14.0-RELEASE schedule, the code slush in main and the freeze to the KBI for 14.0 was scheduled for April 25, 2023. As some of you may have noticed, that did not happen. First, and most importantly for 14.0, is the status of the OpenSSL update to version 3. This in itself is reason to delay the schedule until some tangible progress has been made. Yes, some have expressed interest in helping in this area, however at this moment, this is the key blocker. Second is the status of the branch and how it pertains to the recent upstream merge from OpenZFS. Although block_cloning is disabled by default, there have been other regressions discovered (and fixed), but as a whole, I do not feel that we have a solid understanding of the regressions about which we do not know. There is no feasible way we are going to make the branch point of stable/14 in time, with that scheduled for May 12, 2023 with the above points. That said, this is not an all-inclusive list, but the more major items on our radar at the moment. A more up-to-date schedule for the 14.0 release will be published in the near future, though nothing is yet set in stone. Thank you for your patience, and for any help in getting us through these outstanding items. Glen On behalf of: re@ --cyV/sMl4KAhiehtf Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEjRJAPC5sqwhs9k2jAxRYpUeP4pMFAmRQAZkACgkQAxRYpUeP 4pPQJg/8CO1Z12iekJPausELsStIi0rsCxZPtx9wOXAOKWIJmc85ss35UFluPbWy 8iKAciUuHqCTg7GebTPpkbTx/MptAvbO+xFbKfiGPUrRLuLF4DftKvMyaaDWqo7K nUZ5ob3ldLaPkiYGjJ3vayf1HvXTa9bHf8I0eie/fY8fm20Yro00dYGDcr1UoxoQ QPUDJ7hulN11nw6mt/ywHM5gYTe3hT4BphJ+KLu+5tayEvmv7sgFd7F/oXW4BnPo BZsi0jZf49udUxdFpssyzaQsV5fyZYgjeelG/znQ6bQPO9PwE3NJEzTUz19F0MWY lxuJeuyB3iz5ZBEvNLbHXxKjBCTjANN8l2lUU4iQshm1nKIR3SbaTw6P/3L79UwF aP+1OAMpwKT+1xnZRWnrMWmpDEcrWjvXy4aBmpTXobxHT5zpBU1lecKGmpFWpOMp KJEDEEndQlgTZh2KfWE80xIWnnvG7dEkxdIgyHXeYjSrkUqCA8wsZqmV7LOdCFxv rUDjeuhcDQNsgEXMM2e+OTt+TAb7xIagX7cYqTN9ATZjzHcbP65jfBhi7dYF7lhZ Thrj1+u5xzNqc2bbEuZbZBWK0+hp+Z0DoKy/ka3XNguP868+SmsG07KErnLr+KYU j20ZRz5S2iKrmdXtkc2Rn0FY/CUOJievJtIun6dY2tfkUBJNcm0= =aAce -----END PGP SIGNATURE----- --cyV/sMl4KAhiehtf-- From nobody Mon May 1 19:57:01 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9DWj4YHyz48wsy for ; Mon, 1 May 2023 19:57:21 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic317-20.consmr.mail.gq1.yahoo.com (sonic317-20.consmr.mail.gq1.yahoo.com [98.137.66.146]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9DWh13rtz4Q54 for ; Mon, 1 May 2023 19:57:20 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=hBJGCRje; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.66.146 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1682971038; bh=xuLSkyPWETSS+3g7ArElxHEzmCDoZIj4wr/z1L33qJ4=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=hBJGCRjesL+OhLPjvqGZipel5WepPF1987oHJ3N6sdKHmzuVS1FVF+RlMrNBZqJ3ITF1peVtRhA0nRNo4tSPTCneAB5QR+vKYXoPmXqbgh7hepqK2qRIRk8qdxF9CQi3aQwZsxGLEzXrh0uMPZPLB49zCgepdCy235P96div5/QlNxkipp7zXg3GvCnsitJBnCF5fiCr51e7OTPCc/yQqdDGBzai+H3iViWQDUI4ehvK3hINvVDf++GQdLpPrQZtwh63pEpLJ4NZeQG+K0bIv7ZQtV7jjS9xFzCIES8U0d5pD3T1M5LqT9wqe51mio1yI49BgX3oobLTJOrqvWAYeQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1682971038; bh=Ra7XaJuAE/zkOrGuNBU+9mhZ9jsJmpcpFinDNIzoq78=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=MwMXQ3PGAT4NBYHfvk4EGL+XrhVjvxGYrgnOa01hIDCCduhcNSJM3jzpX7RcdmUL3uHin3zQpa/8xKbl+eiO03FXiN3WHG9DhWqXrn6OgUEKJNEhVRuMnUKq+EkkZisdkML7TmVoqwqiJKr59w1v+Tc2OHqNkyp7dBRAeDKmCpXce2LQ7Nm4ncj82B0t6gjH8/qds6IPZ63V/+k3uCLTYsAJJzFnD99ZeEyDU7yLr0VZuRmF9QgvIwhZR8yozs+TZXp3h0VairMAxleNXu2WQmwNQ9IUi1axo4DaR8i2ewflRRIMCMk+zcRwHz0X+Bv+mr2edz5guPw/mr9PhPctCA== X-YMail-OSG: PqQt0j8VM1kkdc7MA5NWMAmvhluv9gq1CFi63iMrijg423MtlF.J.icce02Z7xt gQ9zzd3a3VHJQckL4zdVW_q_TVNRGdr0MEyUhF7t1Qx7j6anSF6RUUsr.LV1joNBKBNezS1NBUt9 w7PWPV.BQd.Gb7NOShDRBRJ6mbqWHs4R9IJOMuSBX1ZJtWwKPle4Y22K6ASaPe9thnrofxuv4cfa PGPw1E9.hy1DqZwsKzeWMS1jAJeWBIk1XZoc8sGs.5YJm3McRaSAJhu_DDDWi9aZc66IMzag4lTC 0T10sB1BsOaQItYMnf5yu_nSSlZjXog8fMahtQZ_J43eHKho85WqkjD_mGrO_LqlIrj9smvCCUIq cQUtc3ASCn5nAGIKyGFfKyaZS..wQRqmVzaOLV4irgWmi.gfj4wT4WGne_pmlwDohW4N6HTVeFTZ 9UiPE8lZP.aiCZG5uv1rznnv_hLvHe7omV0Ol6e_R1rhGylS5.l0f8Pc50yNX7zqP8HPKR6H0WHp JY68ie3_IAV7nfBr5mPL8.PLh.5YxRBK_ew_jYBZ2cN09i_iRDsXEoaKYrc.dX0CbFUXQHhrdJ7S BcsQ9DLiOZRh2qTJJXnifYj30InzWbLazAz46qAOQyQk14ZuTV9zdaYJOsHOTLdchj992LUBR1mO XpOZ6T3ArLrD2il5xv4BtvHDOCDMIvW3KLDo3MxQ1cR1dZwfDfXqiK71.BtfUtgGTfzwKKcyxMAC Z5w72uEusKHKb6SC19zP2cXE1Wy6qxUV4iUihmhdTOyrm95mUrKND6IppmxG_.pKBi9qbmIabzwH fN.I80VwhzfGPdMUo8WBE5PMsbcCOhmRe97QeBeWHH1ibC8FBbhuuz4ofuOzJzCEDrcvEb.nmNyc iMVc2__9T.EefTd4MGAMWc6VWEyhy_8MqaE27QMro.f.p1sXbulu_I6ESJ6TJz15sl0yB0aRcUrp CBsuX9nPFvqqGUJvbBLkRUk60BCIYx.DbaLmD9xPH8zB8kYSIIbh7B96x9XAoHrL0f9WUUfYvymP _3NzoYVjVm6T0iUqRr86FtZTWzWSM.q3xH1ZLP0IQriIa7CbusIOWJOoEIjthMw6Wrmfldn5eX.T 6MRQ6bXhh1gVesqyPGnkd81ThftcWWdyU5qe7uprzqnF9K_iB0KUC70geHy6roDUY9mQn3eDQ8Ij TuO6zk6VxDc9xiRJy4NBZmEGMYkFw9S7aqd4Sl8pRQUYKqX_ns_47_7xv6rqDwHtK0HA6v_uczEq 7GSdP0lsCOlGuuFWCJXljIRHX1V6zD3gnr8KIvvLC_avwcnixfxbhyMWTrgfwAm8ukFEQ6XI0l2L YdAGdxqlRb_Aar8orjp2sKbkJD1cpbs4ZlONUdh2TI765L25lI3EdXaatE2SCYUgz21e5BF2_.1n GldL9xRU8e3LfI1fXMngIA38hEWVqZ7jPwJY_GsaXQN9.392N1BykpTwgWH7pkL1NpjP09MkZ6av SbDnt6mBJVElOFHM0ArglBYj1J2.7zgeKwwhOW3tF.BPJ3EjiLvdo_DK8WwEn_35fI35k2OTcNAY 6S3ZDyU3gL6DLeCHnChzQEWuur3pnv4Wl_YEIMr6mYOQfhiYtTJWu7kPmNTGphl29jnuz9NIs1HF O73m59NRzdqOInt6FHicx6EUvH_mfWoDcC5m0.hjkx70UoYgq6Ta0tsXoQu_YlUvOlkox0vVISKr hsTNnTm1egP.olCArEm51BX9gligWXgwkEuiNEZS5U5YkvFP.hqBng2ljabDgJvaMOKXLTIZ5G7r egmWPCTjLvttW6xdKP8iInW29AZPGBaCRo6avQ0GYRS5WylZ3r350xKQf1EHDv3dvI1Ke3dD7aZw 5uMUKFIs0u4WOKfaw0941gvifchDAQxY7hD.f9aibsNIswtkZPs7670QRr_fGIXrM3TbZpMHT5nA 8u9ydNztTuhcAQnJ.1bX1ViqpFY0uk7wWGECCckg.39TTBzA2bTggxPe4zKEXyTfRPIazcadg491 r.jHSAmpKl3RGcTw8zznYR6aXeCBL_17C68PIq9Z8TxMQMro6RGlKDc.xI1oNMHRHmcKGOz.Iz9R IWKWvV2uI02EKt6Gh2r6GYLPxGBbSaF_NB_IqhEEj7fkr7L_Yf23c35rWC5jVS8.BcfqrRBh2BYj sFRwHySwsOaasMS2GbaFUpzh5RU8gIoQlvksKKz2nWoTEn51yEMQmJijrwga7L4kLswAmbK5AIHc gdoqam.Vydeg4ULWQbRAXHLeVnlMu_EN3tWjyMGrZ3y0F0uBE8xpLQ1Ylw80fV55ttUwyosmFgeC m X-Sonic-MF: X-Sonic-ID: 5b84c1d3-7548-4222-aeb9-79325bd06b9f Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.gq1.yahoo.com with HTTP; Mon, 1 May 2023 19:57:18 +0000 Received: by hermes--production-bf1-5f9df5c5c4-fgkgh (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 7948446c661a58a6abcb434ead47fd04; Mon, 01 May 2023 19:57:13 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: armv8.2-A+ tuned FreeBSD kernels vs. poudriere bulk and USB3 media: tx->tx_quiesce_done_cv related blocking of processes? From: Mark Millard In-Reply-To: Date: Mon, 1 May 2023 12:57:01 -0700 Cc: FreeBSD Hackers , freebsd-arm Content-Transfer-Encoding: 7bit Message-Id: <8F883286-D713-4632-9575-5813E885D125@yahoo.com> References: <7AE28A5B-109E-4C26-9D70-BCA5D49CD79D@yahoo.com> <02DC03AE-E082-4FB5-AA0D-396F64CC23CB@yahoo.com> To: Mateusz Guzik X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-2.42 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_SHORT(-0.92)[-0.918]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_IN_DNSWL_NONE(0.00)[98.137.66.146:from]; RCVD_TLS_LAST(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4Q9DWh13rtz4Q54 X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N On Apr 30, 2023, at 18:57, Mark Millard wrote: > On Apr 30, 2023, at 17:44, Mateusz Guzik wrote: > >> can you redo zfs test with: >> sysctl vfs.zfs.per_txg_dirty_frees_percent=5 > > Sure. > > Result summary: Seems to have avoided the sustained periods > of low load average activity. Much better for the context. > > > Context: Original ZFS USB3 media. World or Kernel in use > had been built (non-debug style) using: > > -mcpu=cortex-a78C+flagm+nofp16fml > > > Steps for this test . . . > > # poudriere pkgclean -A -jmain-CA78C > . . . > > # sysctl vfs.zfs.per_txg_dirty_frees_percent=5 > vfs.zfs.per_txg_dirty_frees_percent: 30 -> 5 > > # grep USE_TMPFS= /usr/local/etc/poudriere.conf > # EXAMPLE: USE_TMPFS="wrkdir data" > USE_TMPFS="data" > #USE_TMPFS=all > > # poudriere bulk -jmain-CA78C -w -f ~/origins/CA78C-origins.txt > . . . > > At 15 minutes into the build: > 46 ports in 1st 15 minutes. Load average stayed reasonable > for the configuration. > > At 30 minutes into the build: > 102 ports in 1st 30 minutes. Load average still reasonable > for the configuration. > > Looks good compared to before. > > > I've no clue what optimal would be for the context, but > > vfs.zfs.per_txg_dirty_frees_percent=5 > > is vastly better for the context than the default 30 was. > > Thanks. > > > I'm going to stop the test and do the conversion to the > U2 960GB Optane media in the USB3 adaptor and then > compare USE_TMPFS=data vs. USE_TMPFS=all --but using your > vfs.zfs.per_txg_dirty_frees_percent=5 assignment. > Took a while to actually get around to stopping the test. It got 186 of the ports built in the 1st hour. (A from scratch build, starting with building pkg.) I finally have started the U2 960GB Optane based tests, currently USE_TMPFS=data . The initial activity looks like it might build about as many ports as the earlier USE_TMPFS=all test (for different media, vfs.zfs.per_txg_dirty_frees_percent being 30). . . . waiting . . . It got 222 of the ports built in the 1st hour, again starting with pkg. That compares to 262 for the earlier USE_TMPFS=all test. None of these ports form the first hour are large, long running port builds, none using large scale amounts of storage space for its builder. (As I build things, rust for example, uses 17GiBytes+ of file system space, more than half of the size of the RAM in the Windows Dev Kit 2023 for USE_TEMPFS=all just for file system content.) Now for a USE_TMPFS=all build test with vfs.zfs.per_txg_dirty_frees_percent being 5 . I may try letting that run to completion. (The configuration has 118 GiBytes of swap for paging activity.) === Mark Millard marklmi at yahoo.com From nobody Mon May 1 21:21:26 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9GP40B9lz492BM for ; Mon, 1 May 2023 21:21:44 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic301-22.consmr.mail.gq1.yahoo.com (sonic301-22.consmr.mail.gq1.yahoo.com [98.137.64.148]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9GP2320Bz3Qfh for ; Mon, 1 May 2023 21:21:42 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b="ioo/gDBX"; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.148 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1682976100; bh=zux7QH1t8qvzq6qDMuP24l22VAwqouhbdhExbhPJ4DU=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=ioo/gDBXuLGuu/bKRlJazmbiYoHrQ70/MU50OC7w3/EureLkZ9/jbHhoFScEBb9wXFLmX2hMw+sSbzPKJvWJ9SDWC3qhEKMfhbI5rC4h/VmS3ORg4U9+0hTMXF/95TWb043Q5fuhxsFgX6fQPvDFCkaOUo4HRE/6BH9K7GoAgoJ8Y53cneQ0PiTHaciPNaC5cQIIyLCE1WIfP7dfF02gKRlIGoTT2oevlYXLL8UOy0q9BAZwuuVgRJjmeedNefZ3Puz05I1xfMKTgmsvg3S+biGiOCbpoz6GK+2FRAV67X9qmAq+1zHeh1Ok73wOiRWPvtgPvgn0DefoCE4YrlhM/Q== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1682976100; bh=wTY/7LVnvXTDGO5RN8EVAKQspxL2Hv+jD6GsNRvFEaM=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=ZwnjuzUs8vA6SKxSaqSv6ONH+YjB3z9nljn7rFYmsR2trA5yNydLiSJLBcxyC7ujbYys5tx5kqXuxLpuddoixyffHDPrmSOceUkXzBPVsUuw/MFEpeZDoqcWvTr6JcLLv1T0xOXIg/4lrJ09L4GBd0s/2kst6SnclofNLaZuXl432aotV4YolmkGKE61S9UBWuZUMQmeR9s7T8MJt5lFwMhR4+k6JuzMwqguwC31d5IETH3YtKzQpnTtnEICXzc/LUulraTdpx4a/dVIs1R6gIkFnoA6WQkL3sdOGoxmY99eNeQDTZm5NZNKFPtaHzsT9WXxOd73EwOzREs35EAtCg== X-YMail-OSG: wQqzf84VM1lRmRofstmjdy0Q0U._o88s3sqEZ__MzeTFIOm1KbrkZXt4AFhtV78 toQNh8VTc75JEjc8bpSeSWOi4lN6dAvyp8PyZQA7YoVUgB4YHM.bTxXfaqjvfEb_mdxqCw_Hwst7 PeQfdaWomGy2e1H3k1JOFzDARqFfsLpWIc5kxSN8Tg18.Z5GSSbLGhc33XgiLfU.t2tZQBe8fz8H IjAsmH33sBSCVGnwNSKdGzQD9KjaEK0jytdlv4tbdI2J.J.1O6GR66SGC3PS6k0Uv.GBN.7IQpDT DoHl9hO0E9gy8BGS66aEGK1SOoy0u3HxZ64PRmJUNiqtQyIDi3dZp3ltvTPN0Xx9u9weyRwQv.ng xR3GJ_Crx5p_GRGcUpNMf6.8MUxIgtBqWsZKNeQJdFvaVezilFm6C52KoatfQOzsTEyIxdQ6mr_E btTumEaTKWiD7x4XlBSYBRwznF_5tDzgRLhoDzVLKFGs0n3XydlKAVo2AI9EiU85K36BVDBDMbT8 Nur8V5Jx1WSOq5rLruCodIqVH_LbQXbxLVxlfEKRBtjY2BDra_pgm8APFOwgsG1nchSYZfYZKd65 egRTFTgqAv7QKLgyfU4hrwsz5hKCSewoGSG0ZsYfkOPA3Fzjkc5Grox1NwkHN7j7G8rbXYCMZLVC Oh_RaNdZI6pCaIKjVxOAUwHNFxnmYmxQTPzrDTbs5VNZXK.y0RPfMfSirCPBGs89jn_CGUwlF_1n SgMeOLU3iQQbIPZ6e24pwBx2A21UnEvDBVMGd1aL8akrKsmIOP5Buy2uDbGqYj9Pv7LBYALUTzRp hPQionw.6nXR41XsJkiZFlbLQxoc0LYL1d0O6xyj3Ok74k2P_hDmMynGjdkC1D9Ae_VRWYOtuGik X7Jj00QNkebhWImVN0zDrFPnfv5ralyK755trDn9gYKBf7b7a_X70MeqFcEZtAzo9JwmcWnc1ngU LgtlEhdNANFbCbS4R4IpMrP6aW6shY5lUxHLm5j727CfQ4FMay5mPpNLVO0OLHN34HZuJyNUjisN 0zKCsiBqzimsLncUE6PuI9y.FZrHQzhpm3kVplgUO6kUrBVTUdm1Ro5OjvUKEI1pQ4DnQRRUg0b3 BcGRMgQ68CTLK1wFoM8NlPGZwZrP_duJcXE6XA_aoZLxlBnKGJlvVPiLUeqqPHVLkshLpnVc5eHh djlhc2.xkVc4pflhtnwcmW2_FoUZ_1WG3L.M7Tr_YETVk3SzCCLixueuivQ4r8PMJCx4W_FM4enV d8xTNWaipcF8k8nBHuBmmJ1XTxXcQgB9JJqZeYUeq9Nm9OpOa5uTELsjmaL4QrZeEVFfdZgcf.Vj 5z0_fHQrCw9AABmnY0NO7c0h.BJgIcu1NQrhZ2OEj3p7A.iTvlTJX2pTi.J.HaxHWf.R7HwJYv26 2zKLqv_0_WDJ.8jYDsCPBDFGgscH0LnJmprbdCDd0Krcmh37fs59a87VMYUQsU49yuzm7_LkEwmQ yDSemiBXenHs9a63Q4BbB3u6R2qxA7pFpcyaWaFUgLM8oVK8W3UqJs3aG30TEAwaf5WDXdA.xHRi iipktYZ5sGxuexc1vgFqrZmYAdH0YcNArCe.6UjhpD_rlssSK76gvF7R11Dcy_b_KX2WIBbk4LSd 1uzZVvPIQAVQADuUAnHJ.VD0daWxi.JD0B.wKAOISFdqLk6GsWxfQdgOMVgks7XvlPfWI8uFLGqJ SQa3ZTbMD5sp9DNtlndGQWqqRNsirHhdL8VT.g6q88gdacExcKF6YtKvma.JEbmxkFV_uROxgkYd i3yWuft03B_VtxoMj20HE3u_ZqJi.irm6zm_A9yXyBXgSGVSu56eAiacxS0S1mxrnsgTd8DUxXES q0N6GFBINYXh45CjSIZYH_m90zubpM08xS.K10_rHk0Nva5AJKTuymK42Tk276vMAjPC41lB9Z.S 85B3FWcpcQvQuXGJ6HKhoKJJmps_rOUAsUZ8rVnkLGd0uqx1UaS59cBKvqWdeG0zOjhITT9YsDwp CeOVxuqjh8AiG.6IoFEfBSfepQbDqu.2IK86GwAuOG0WTZIrr9aQyYtMgu5bGJCNBHV8r.9Ogv1O lEx0ndeRYl1s7KdvKeMjiM5qyfuXaQUNVIzsnI_XRFITKbkj9vSyWtsaNwF6bM9kOq3wBd2wNEfW U8PNZuLaBHT4ZaYLLcy5ORSyd41FOdQNoIWfUOtoffgZy3mkURO2vcWIdeo.wMGgse9lANZ9abIf U6DzoaKLcenJ_5Tf0XMB5nadCQ.h1U.Ve1K39nVw4jiSiF330cwwo.bKvtX9Y2xbZZwo4RndO_hV 1J4k- X-Sonic-MF: X-Sonic-ID: 8f881a25-07ff-4f4f-8d0b-a3b62dbac332 Received: from sonic.gate.mail.ne1.yahoo.com by sonic301.consmr.mail.gq1.yahoo.com with HTTP; Mon, 1 May 2023 21:21:40 +0000 Received: by hermes--production-gq1-546798879c-l2qgj (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 23455dc20bc6a2c6fc14c38aa33e2c08; Mon, 01 May 2023 21:21:37 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: armv8.2-A+ tuned FreeBSD kernels vs. poudriere bulk and USB3 media: tx->tx_quiesce_done_cv related blocking of processes? From: Mark Millard In-Reply-To: <8F883286-D713-4632-9575-5813E885D125@yahoo.com> Date: Mon, 1 May 2023 14:21:26 -0700 Cc: FreeBSD Hackers , freebsd-arm Content-Transfer-Encoding: 7bit Message-Id: References: <7AE28A5B-109E-4C26-9D70-BCA5D49CD79D@yahoo.com> <02DC03AE-E082-4FB5-AA0D-396F64CC23CB@yahoo.com> <8F883286-D713-4632-9575-5813E885D125@yahoo.com> To: Mateusz Guzik X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-1.70 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-0.202]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.148:from]; RCVD_TLS_LAST(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4Q9GP2320Bz3Qfh X-Spamd-Bar: - X-ThisMailContainsUnwantedMimeParts: N On May 1, 2023, at 12:57, Mark Millard wrote: > On Apr 30, 2023, at 18:57, Mark Millard wrote: > >> On Apr 30, 2023, at 17:44, Mateusz Guzik wrote: >> >>> can you redo zfs test with: >>> sysctl vfs.zfs.per_txg_dirty_frees_percent=5 >> >> Sure. >> >> Result summary: Seems to have avoided the sustained periods >> of low load average activity. Much better for the context. >> >> >> Context: Original ZFS USB3 media. World or Kernel in use >> had been built (non-debug style) using: >> >> -mcpu=cortex-a78C+flagm+nofp16fml >> >> >> Steps for this test . . . >> >> # poudriere pkgclean -A -jmain-CA78C >> . . . >> >> # sysctl vfs.zfs.per_txg_dirty_frees_percent=5 >> vfs.zfs.per_txg_dirty_frees_percent: 30 -> 5 >> >> # grep USE_TMPFS= /usr/local/etc/poudriere.conf >> # EXAMPLE: USE_TMPFS="wrkdir data" >> USE_TMPFS="data" >> #USE_TMPFS=all >> >> # poudriere bulk -jmain-CA78C -w -f ~/origins/CA78C-origins.txt >> . . . >> >> At 15 minutes into the build: >> 46 ports in 1st 15 minutes. Load average stayed reasonable >> for the configuration. >> >> At 30 minutes into the build: >> 102 ports in 1st 30 minutes. Load average still reasonable >> for the configuration. >> >> Looks good compared to before. >> >> >> I've no clue what optimal would be for the context, but >> >> vfs.zfs.per_txg_dirty_frees_percent=5 >> >> is vastly better for the context than the default 30 was. >> >> Thanks. >> >> >> I'm going to stop the test and do the conversion to the >> U2 960GB Optane media in the USB3 adaptor and then >> compare USE_TMPFS=data vs. USE_TMPFS=all --but using your >> vfs.zfs.per_txg_dirty_frees_percent=5 assignment. >> > > Took a while to actually get around to stopping the test. > It got 186 of the ports built in the 1st hour. (A from > scratch build, starting with building pkg.) > > I finally have started the U2 960GB Optane based tests, > currently USE_TMPFS=data . The initial activity looks > like it might build about as many ports as the earlier > USE_TMPFS=all test (for different media, > vfs.zfs.per_txg_dirty_frees_percent being 30). > > . . . waiting . . . > > It got 222 of the ports built in the 1st hour, again > starting with pkg. That compares to 262 for the earlier > USE_TMPFS=all test. > > None of these ports form the first hour are large, long > running port builds, none using large scale amounts of > storage space for its builder. (As I build things, rust > for example, uses 17GiBytes+ of file system space, more > than half of the size of the RAM in the Windows Dev Kit > 2023 for USE_TEMPFS=all just for file system content.) > > Now for a USE_TMPFS=all build test with > vfs.zfs.per_txg_dirty_frees_percent being 5 . I may try > letting that run to completion. (The configuration has > 118 GiBytes of swap for paging activity.) USE_TMPFS=all with the U2 960GB Optane USB3 media: It built 270 ports in the first hour. I'm going to let it run to completion (assuming that it does in under, say, 16 hours total for the overall 480 ports). I'll see what it is like when llvm15, llvm16, rust, and possibly gcc13 or such will likely build in overlapping time frames. (I do not build LTO style and, for gcc* do do not even use a bootstrap style. My usage of compiler toolchains between toolchain builds is minor compared to the toolchain builds themselves for the most part and my time preferences for builds are fairly strongly biased to less wait time.) === Mark Millard marklmi at yahoo.com From nobody Tue May 2 06:16:15 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9VG52Z9Hz48Xr7; Tue, 2 May 2023 06:16:29 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx1.shrew.net (mx1.shrew.net [38.97.5.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9VG36Nx6z4R1T; Tue, 2 May 2023 06:16:27 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 38.97.5.131 as permitted sender) smtp.mailfrom=mgrooms@shrew.net; dmarc=none Received: from mail.shrew.net (mail.shrew.prv [10.24.10.20]) by mx1.shrew.net (8.15.2/8.15.2) with ESMTP id 3426GKTM090463; Tue, 2 May 2023 01:16:20 -0500 (CDT) (envelope-from mgrooms@shrew.net) Received: from [10.22.200.32] (unknown [136.49.230.220]) by mail.shrew.net (Postfix) with ESMTPSA id A1D37193926; Tue, 2 May 2023 01:16:15 -0500 (CDT) Content-Type: multipart/alternative; boundary="------------PosPzAVxthavBLdS0gbQcyBE" Message-ID: <89c84bd7-a925-02f4-acbe-12c3000e7007@shrew.net> Date: Tue, 2 May 2023 01:16:15 -0500 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.10.0 Subject: Re: BHYVE_SNAPSHOT Content-Language: en-US To: Rob Wing Cc: freebsd-hackers@freebsd.org, freebsd-virtualization@freebsd.org, elenamihailescu22@gmail.com, Mihai Carabas , gusev.vitaliy@gmail.com References: From: Matthew Grooms In-Reply-To: X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (mx1.shrew.net [10.24.10.10]); Tue, 02 May 2023 01:16:20 -0500 (CDT) X-Spamd-Result: default: False [-0.57 / 15.00]; SUBJ_ALL_CAPS(1.05)[14]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.98)[-0.984]; NEURAL_SPAM_SHORT(0.66)[0.663]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org,freebsd-hackers@freebsd.org]; ASN(0.00)[asn:174, ipnet:38.0.0.0/8, country:US]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_CC(0.00)[freebsd.org,gmail.com]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_SOME(0.00)[]; DMARC_NA(0.00)[shrew.net]; TAGGED_RCPT(0.00)[]; RCPT_COUNT_FIVE(0.00)[6]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4Q9VG36Nx6z4R1T X-Spamd-Bar: / X-ThisMailContainsUnwantedMimeParts: N This is a multi-part message in MIME format. --------------PosPzAVxthavBLdS0gbQcyBE Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 4/30/23 21:31, Rob Wing wrote: > Hey Matthew, > > On Sun, Apr 30, 2023 at 1:41 PM Matthew Grooms wrote: > > > Would you like to see support for VM snapshots in the generic kernel? > > > Is there a review open that addresses the limitations described in the > commit message that brought the snapshot feature in? > https://github.com/freebsd/freebsd-src/commit/483d953a86a2507355f8287c5107dc827a0ff516 Yes. The next set of project goals where not pulled out of thin air. They were selected specifically to address the limitations that prevented snapshots from being in the mainline kernel after the initial commit. That's why patches for AMD CPU, Multiple Devices ( >1 of the same type ), Capsicum and JSON file format for snapshots were developed. They were identified as the major per-requisite for lifting conditional compilation. > How about support for warm or live migration? > > > This builds off the snapshot work, right? Seems like it'd make more > sense to address the current limitations of the snapshot code before > extending the functionality off the top of it. > Yup. See above. I appreciate your input, but the goal of live migration was set in 2016 with a prototype first demonstrated in 2018. How long do you suggest a developer wait without review feedback before moving forward out of tree? > There are experimental patches for all these features that were > developed by students at UPB. In a lot of cases, there are open > reviews that have been waiting on feedback for ages. > > > In general, most people don't want to review large experimental patches. Yup. That approach was attempted with the Warm Migration patches. From slide 17 in Elena's presentation: First review opened in 2021: https://reviews.freebsd.org/D28270 5 reviews from 2022 starting with https://reviews.freebsd.org/D34717 (same feature split in multiple parts) A similar request was made recently to Gusev Vitaliy WRT the multiple device support patch which he took ownership of. Thanks for adding feedback to that review BTW. We'll see how that pans out ... https://reviews.freebsd.org/D35590 > > The case is quite plain. I'm not sure what the solution is to this > problem. I'd love to hear feedback from the community about how > I've got > this completely wrong and how the course could be corrected. That > would > be something. > > > My perspective is that it would have been better to focus student > efforts on completing the snapshot feature. By completing the snapshot > feature, I mean getting the code into a state where it's compiled in > by default and no longer considered an experimental feature. > I'm not sure what more to say hear regarding the snapshot feature or what might have been done in the past. We need a solution for the present. If you have any comments related to the follow up reviews submitted by UPB, I'm sure they'd love to hear them. And lastly: I get that FreeBSD is a non paid volunteer project for most. Without the efforts of folks like Peter, Neel, John and others, there would be no bhyve. I'm not saying that they, as project maintainers, should somehow be doing more. We all have limited time to invest, paid work to do and families to feed. I'm asking if there are other developers that might be willing and able to help with reviews? Is there something the FreeBSD foundation can do help out in situations like these? Thanks, -Matthew --------------PosPzAVxthavBLdS0gbQcyBE Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 4/30/23 21:31, Rob Wing wrote:
Hey Matthew,

On Sun, Apr 30, 2023 at 1:41 PM Matthew Grooms <mgrooms@shrew.net> wrote:

Would you like to see support for VM snapshots in the generic kernel?

Is there a review open that addresses the limitations described in the commit message that brought the snapshot feature in? https://github.com/freebsd/freebsd-src/commit/483d953a86a2507355f8287c5107dc827a0ff516

Yes. The next set of project goals where not pulled out of thin air. They were selected specifically to address the limitations that prevented snapshots from being in the mainline kernel after the initial commit. That's why patches for AMD CPU, Multiple Devices ( >1 of the same type ), Capsicum and JSON file format for snapshots were developed. They were identified as the major per-requisite for lifting conditional compilation.

 
How about support for warm or live migration?

This builds off the snapshot work, right? Seems like it'd make more sense to address the current limitations of the snapshot code before extending the functionality off the top of it.

Yup. See above. I appreciate your input, but the goal of live migration was set in 2016 with a prototype first demonstrated in 2018. How long do you suggest a developer wait without review feedback before moving forward out of tree?

 
There are experimental patches for all these features that were developed by students at UPB. In a lot of cases, there are open reviews that have been waiting on feedback for ages.

In general, most people don't want to review large experimental patches.

Yup. That approach was attempted with the Warm Migration patches. From slide 17 in Elena's presentation:

First review opened in 2021: https://reviews.freebsd.org/D28270
5 reviews from 2022 starting with https://reviews.freebsd.org/D34717 (same feature split in multiple parts)

A similar request was made recently to Gusev Vitaliy WRT the multiple device support patch which he took ownership of. Thanks for adding feedback to that review BTW. We'll see how that pans out ...

https://reviews.freebsd.org/D35590

 
The case is quite plain. I'm not sure what the solution is to this
problem. I'd love to hear feedback from the community about how I've got
this completely wrong and how the course could be corrected. That would
be something.

My perspective is that it would have been better to focus student efforts on completing the snapshot feature. By completing the snapshot feature, I mean getting the code into a state where it's compiled in by default and no longer considered an experimental feature.

I'm not sure what more to say hear regarding the snapshot feature or what might have been done in the past. We need a solution for the present. If you have any comments related to the follow up reviews submitted by UPB, I'm sure they'd love to hear them.

And lastly: I get that FreeBSD is a non paid volunteer project for most. Without the efforts of folks like Peter, Neel, John and others, there would be no bhyve. I'm not saying that they, as project maintainers, should somehow be doing more. We all have limited time to invest, paid work to do and families to feed. I'm asking if there are other developers that might be willing and able to help with reviews? Is there something the FreeBSD foundation can do help out in situations like these?

Thanks,

-Matthew

--------------PosPzAVxthavBLdS0gbQcyBE-- From nobody Tue May 2 07:57:21 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9XVX63k9z48fmX; Tue, 2 May 2023 07:57:24 +0000 (UTC) (envelope-from corvink@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9XVX5WlQz4d0Z; Tue, 2 May 2023 07:57:24 +0000 (UTC) (envelope-from corvink@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1683014244; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=K5i1ZpYw/NE7Tzd2JNdf7HyERA7mNMxRRE0rHYb1CKk=; b=jm/va3Ebu81jahPKXRJelJemSO9EMQtUWvWs8KSN3hyV92BUpa1C9yhwnPo9VDbbBoRsL9 yVaJbGc4odf5B/BuhsIpUrlt5glMGStrkYGYbnqlB5EpoXcKnl7Pa8aNKzUJEiIqkp/JhT VTMOlIfX0swd0uA8QL9ygt/ROW7GUbgynzDM/vA51j+arEe0lNHmr4+yVPGma687UJKmPA 6NICc3lZEJYPQ38G1Q01zvKmonGqBS2hFVxwxqcQyyvPp8ycSQj13MyBjmur8F1MdemOqA ++HyImJWV7/WzpnWbJFcGIC8qm9T6SPcXcfxQHvw4MAwyr1WowNw43Wo/U5Oug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1683014244; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=K5i1ZpYw/NE7Tzd2JNdf7HyERA7mNMxRRE0rHYb1CKk=; b=fthOHJka70ZF3P4VIB23Bp+nMm7ybcrBoQ031r6JfAKqoYeBPIMVWDAOgwJtjFe6vX+uVN tocaO6zE4s9gTrhL0K2JZZf/abEPu1S0PJGhg6oPrwSSq442VD4yZ1OYmF9jF9CDkIbDM3 Ddlh6fOFOVEhg1lOACl73pEyPvqWySVBVMmlqmr6HuLWnYDkXXIOB4i8iUoDpZXxBh0zDi 99MLJBDnlnyznOH1KiHOrzqK2QSpbz5aqCphJkUH1lCuFEpTU6ibgC113LghnzRgDHn1PX wYLNtCJyp5ngJsBmY33LQmutbCSI31rMgSj+2mpXwCeYYFku+ODfZMI1fU0bNQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1683014244; a=rsa-sha256; cv=none; b=LbXd15SyeyhO0Dexxk8PyLJDQ4EqD1PWRjytpNAGWi0Ay3MQhfdfsjQ+RQcNZa5nuWBo8Z 60IRsY08k6xsi35tS3s6KgRw+JjyKTs56g8WqsGDsBEnk1vxVKK0v/PRtEfEr+WkZbR4s3 moRQXQ8KKafhVSByJEbkoeVd8LDBCcLO9kuxqSzET2c5zrCAWpDrSyTyP0n0t6t9YCMH9e cRHnTFdurZdIb/bx/hyxzBuZGj/jHlbag8MSw1ZRh22y2+vJkXL7x254dyU5LvUHwsftMM Ip6G7WBaZbiNlXlf3K08IoBVvDeTViKmt7Mo6c0T0SxJV9jkx1Wx8UtxWtVf/g== Received: from [IPv6:2001:9e8:da52:5e00:4d86:a2e1:ac9d:59c8] (unknown [IPv6:2001:9e8:da52:5e00:4d86:a2e1:ac9d:59c8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: corvink) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Q9XVW5l7Cz16bc; Tue, 2 May 2023 07:57:23 +0000 (UTC) (envelope-from corvink@FreeBSD.org) Message-ID: <3ab4e6d94fb0153fb6ff4a53ac6f53b2eaae0cf7.camel@FreeBSD.org> Subject: Re: BHYVE_SNAPSHOT From: Corvin =?ISO-8859-1?Q?K=F6hne?= To: Matthew Grooms , Rob Wing Cc: freebsd-hackers@freebsd.org, freebsd-virtualization@freebsd.org, elenamihailescu22@gmail.com, Mihai Carabas , gusev.vitaliy@gmail.com Date: Tue, 02 May 2023 09:57:21 +0200 In-Reply-To: <89c84bd7-a925-02f4-acbe-12c3000e7007@shrew.net> References: <89c84bd7-a925-02f4-acbe-12c3000e7007@shrew.net> Content-Type: multipart/signed; micalg="pgp-sha256"; protocol="application/pgp-signature"; boundary="=-1ejwBLksvRG43eWs0Fu8" User-Agent: Evolution 3.46.4 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N --=-1ejwBLksvRG43eWs0Fu8 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, 2023-05-02 at 01:16 -0500, Matthew Grooms wrote: > On 4/30/23 21:31, Rob Wing wrote: > > =C2=A0Hey Matthew, > >=20 > > On Sun, Apr 30, 2023 at 1:41=E2=80=AFPM Matthew Grooms > > wrote: > > >=20 > > > =C2=A0Would you like to see support for VM snapshots in the generic > > > kernel? > >=20 > > Is there a review open that addresses the limitations described in > > the commit message that brought the snapshot feature in? > > https://github.com/freebsd/freebsd-src/commit/483d953a86a2507355f8287c5= 107dc827a0ff516 >=20 > Yes. The next set of project goals where not pulled out of thin air. > They were selected specifically to address the limitations that > prevented snapshots from being in the mainline kernel after the > initial commit. That's why patches for AMD CPU, Multiple Devices ( >1 > of the same type ), Capsicum and JSON file format for snapshots were > developed. They were identified as the major per-requisite for > lifting conditional compilation. > > >=20 > > > =C2=A0How about support for warm or live migration? > >=20 > > This builds off the snapshot work, right? Seems like it'd make more > > sense to address the current limitations of the snapshot code > > before extending the functionality off the top of it. >=20 > Yup. See above. I appreciate your input, but the goal of live > migration was set in 2016 with a prototype first demonstrated in > 2018. How long do you suggest a developer wait without review > feedback before moving forward out of tree? The snapshot feature isn't compiled in by default. So, it's likely that changes break it and only a few people are testing it. We have to focus on getting this into the tree. > > > There are experimental patches for all these features that were > > > developed by students at UPB. In a lot of cases, there are open > > > reviews that have been waiting on feedback for ages. > >=20 > > In general, most people don't want to review large experimental > > patches. >=20 > Yup. That approach was attempted with the Warm Migration patches. > From slide 17 in Elena's presentation: >=20 > =C2=A0First review opened in 2021: https://reviews.freebsd.org/D28270 > =C2=A05 reviews from 2022 starting with https://reviews.freebsd.org/D3471= 7 > (same feature split in multiple parts) > =C2=A0 > =C2=A0A similar request was made recently to Gusev Vitaliy WRT the > multiple device support patch which he took ownership of. Thanks for > adding feedback to that review BTW. We'll see how that pans out ... >=20 > =C2=A0https://reviews.freebsd.org/D35590 >=20 I've already reviewed Vitaliy's multi device support patch=C2=A0and people had more than enough time to complain about it. I'm going to commit it as soon as he splits his commit. =C2=A0=C2=A0 > > > =C2=A0The case is quite plain. I'm not sure what the solution is to > > > this=20 > > > =C2=A0problem. I'd love to hear feedback from the community about how > > > I've got=20 > > > =C2=A0this completely wrong and how the course could be corrected. > > > That would=20 > > > =C2=A0be something. > > >=20 > >=20 > > My perspective is that it would have been better to focus student > > efforts on completing the snapshot feature. By completing the > > snapshot feature, I mean getting the code into a state where it's > > compiled in by default and no longer considered an experimental > > feature. > >=20 > I'm not sure what more to say hear regarding the snapshot feature or > what might have been done in the past. We need a solution for the > present. If you have any comments related to the follow up reviews > submitted by UPB, I'm sure they'd love to hear them. > And lastly: I get that FreeBSD is a non paid volunteer project for > most. Without the efforts of folks like Peter, Neel, John and others, > there would be no bhyve. I'm not saying that they, as project > maintainers, should somehow be doing more. We all have limited time > to invest, paid work to do and families to feed. I'm asking if there > are other developers that might be willing and able to help with > reviews? Is there something the FreeBSD foundation can do help out in > situations like these? > Thanks, > -Matthew > =C2=A0 UPB has developed some interesting features and I'd like to see those in tree. I can take some time to review the patches. Nevertheless, we really need the snapshot feature compiled in by default. Otherwise, it's wasted time for all of us. --=20 Kind regards, Corvin --=-1ejwBLksvRG43eWs0Fu8 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEgvRSla3m2t/H2U9G2FTaVjFeAmoFAmRQwmEACgkQ2FTaVjFe Amq4vxAAumSDeuzxwoIng8rnHPIuaUYlTyQveawd+FEGs32aulOJvBs3hgs0KLNm t8C7uj04JjENwj4USNOVYFh87pfk9L5Qve0pmDbkJHRqI/3nNWj8WGhs3Bk4yWtH /b5yXSsrM8PBXBowKX+nYGOj7LJxayLUof/glSNJgNbCTy0/7v9qtCDLc2BMTfXU BZNVo405iJdaTvktDFTrugApWvpUb8hSETaDknZa8pV6UXmerAcR3VVb9Xfe75fS UWuamynh0s5IEJGA5d6Da30aI16Bv9cTvqNJuC05MlWJ/7Oy/G3YMMjuNW4zK+9U 9ePpgu49ZEl2AZlmGpI8O8zoZMw58Cr3YRVgj1cyEBi0Zv97nbvhCZW9ujvmQ3lQ Ub7GcIxQ6EHYKYik9DXruedH0HPwZB5OPHPRCPYM4YPL/47ayyjUcHkleSCMnc2n nA3Zga+0yr3H0qNHcGs3TzWMhUj4x7wrYf4bOi0y6SlHsFkCvWd1jJybRqbGpPR4 9rY9Dpjp8CrVfVloAe8tLp+Npwxs7ddU/EAplgv1NdaE6yzEXDBYoZ6xAjMynNkr G63mZ/+iq0bxwWtdR2gznJYy6PPCkb5o13tGnIoW6236sMA2iHHQkMrskzq8k75M yGq+0lsSWNt13+9pXsX75dhpMZGs5D4SA/lnTNAAdsFliFdVHMw= =dU7I -----END PGP SIGNATURE----- --=-1ejwBLksvRG43eWs0Fu8-- From nobody Tue May 2 11:00:31 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9cZ52Sn7z48r55; Tue, 2 May 2023 11:00:45 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Received: from mail-lf1-x129.google.com (mail-lf1-x129.google.com [IPv6:2a00:1450:4864:20::129]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9cZ51Mrhz3kK9; Tue, 2 May 2023 11:00:45 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x129.google.com with SMTP id 2adb3069b0e04-4f00d3f98deso27498614e87.0; Tue, 02 May 2023 04:00:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1683025243; x=1685617243; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=Kedj9pEFkDCLD2y7qQ+eu6dhg5vW4Tr+qQw81KkXqec=; b=SVF0KGKMy9QYiqQWUm1oj2V8JOQOuYNIHKnADEqibjxn4r/qHZfxTSP1pCcmqH47yC fU8VW5T5KCOyy5Kb5SardrWBCL0wJF4CzAN/74Pll89Z51UVxIYpI9TLwr+9lEJMre6y SFgsIsp/nzFRIRnRfQnkHyA4ZQkKKUxGx/pqGKXt4SgY97aLvXA7xzQ3gd0JVUwwFgrt PpOH8vUPHRUsb0gczUfP5PYGc000zTti1K6Fp3q844j25NdB9zrT2v3Hss473xnUy0Ak 9Hs0uc2E0RlPvGaD/jmhBihshXNIEE4llP3Kp1D6G7Op+0ZWqnEpYUp87Z1zbM1T4AI5 FWpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683025243; x=1685617243; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Kedj9pEFkDCLD2y7qQ+eu6dhg5vW4Tr+qQw81KkXqec=; b=LJos1MTYNH/XgImyZUXHOe0OyKEhozR11M5M2Pncz6rwxiz4loI7RflUD4d6P2dnje Sl8g4T3WkjsQy6te0thU6qFCHhi6RnKiltVYZcdL3TLndE0CzsAVJcT0l97B7D0u/vpO COdls5zTvi1z6iOCVe3fhZ16GiZ0AAzbBo7OXMLEHcwWYfYqiU3XhtBg0sKk47zLNAxn hsEgLyE8twBxdL3gOCmVBrgezuNJyhNyqHgwudJQ2muFoUHezeBBCF9/AxL+z8NdjZXh Zzo8q2cCHZ1IQfla1tfSGK8qbfB4uSwUhnDhJDZaJ56sJOp1PGyoMCHUY71E9CrfMNrm 1wOw== X-Gm-Message-State: AC+VfDzlZcbOIntSDSFF/CF1KECLTGT2V8vRCHzG48tjvMc6KWEI87cq An0pHtOPvd3Q9AAwqt6NBKonJu0WoOKXUw== X-Google-Smtp-Source: ACHHUZ77Dtxg7EjDYsHu1GG7saceXpD5njMlp+bVj4xmEGz/FVNqjVBj7rYPIeFFRD7mebMqQm3CTQ== X-Received: by 2002:a05:6512:904:b0:4ef:ef9f:f24f with SMTP id e4-20020a056512090400b004efef9ff24fmr4425476lft.13.1683025243070; Tue, 02 May 2023 04:00:43 -0700 (PDT) Received: from smtpclient.apple ([188.187.60.230]) by smtp.gmail.com with ESMTPSA id q3-20020ac25143000000b004ec8ca234cfsm5308457lfd.122.2023.05.02.04.00.42 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 May 2023 04:00:42 -0700 (PDT) From: Vitaliy Gusev Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_B895AB4D-5F14-4A57-B65C-D1DB37F70C9D" List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.500.231\)) Subject: Re: BHYVE_SNAPSHOT Date: Tue, 2 May 2023 14:00:31 +0300 In-Reply-To: <3ab4e6d94fb0153fb6ff4a53ac6f53b2eaae0cf7.camel@FreeBSD.org> Cc: Matthew Grooms , Rob Wing , freebsd-hackers@freebsd.org, freebsd-virtualization@freebsd.org, Elena Mihailescu , Mihai Carabas To: =?utf-8?Q?Corvin_K=C3=B6hne?= References: <89c84bd7-a925-02f4-acbe-12c3000e7007@shrew.net> <3ab4e6d94fb0153fb6ff4a53ac6f53b2eaae0cf7.camel@FreeBSD.org> X-Mailer: Apple Mail (2.3731.500.231) X-Rspamd-Queue-Id: 4Q9cZ51Mrhz3kK9 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_RCPT(0.00)[]; TAGGED_FROM(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_B895AB4D-5F14-4A57-B65C-D1DB37F70C9D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Just add some plans for me: 1. Describe snapshot file format: One file for snapshot. 2. Implement snapshot/resume via nvlist. nvlist implementation brings: Versioning Easy debugging, getting saved values, etc. Validate restored variables: types, sized, etc. Add optional variables without breaking backward compatibility (resume = can be performed with old snapshots) Remove variables without breaking backward compatibility Use one file for snapshot Improve restore command line: "bhyve -r $snapshot=E2=80=9D, i.e. w/o = additional arguments =E2=80=94=E2=80=94 Vitaliy Gusev >>> before extending the functionality off the top of it. >>=20 >> Yup. See above. I appreciate your input, but the goal of live >> migration was set in 2016 with a prototype first demonstrated in >> 2018. How long do you suggest a developer wait without review >> feedback before moving forward out of tree? >=20 > The snapshot feature isn't compiled in by default. So, it's likely = that > changes break it and only a few people are testing it. >=20 > We have to focus on getting this into the tree. >=20 >>>> There are experimental patches for all these features that were >>>> developed by students at UPB. In a lot of cases, there are open >>>> reviews that have been waiting on feedback for ages. >>>=20 >>> In general, most people don't want to review large experimental >>> patches. >>=20 >> Yup. That approach was attempted with the Warm Migration patches. >> =46rom slide 17 in Elena's presentation: >>=20 >> First review opened in 2021: https://reviews.freebsd.org/D28270 >> 5 reviews from 2022 starting with https://reviews.freebsd.org/D34717 >> (same feature split in multiple parts) >> =20 >> A similar request was made recently to Gusev Vitaliy WRT the >> multiple device support patch which he took ownership of. Thanks for >> adding feedback to that review BTW. We'll see how that pans out ... >>=20 >> https://reviews.freebsd.org/D35590 >>=20 >=20 > I've already reviewed Vitaliy's multi device support patch and people > had more than enough time to complain about it. I'm going to commit it > as soon as he splits his commit. > =20 >>>> The case is quite plain. I'm not sure what the solution is to >>>> this=20 >>>> problem. I'd love to hear feedback from the community about how >>>> I've got=20 >>>> this completely wrong and how the course could be corrected. >>>> That would=20 >>>> be something. >>>>=20 >>>=20 >>> My perspective is that it would have been better to focus student >>> efforts on completing the snapshot feature. By completing the >>> snapshot feature, I mean getting the code into a state where it's >>> compiled in by default and no longer considered an experimental >>> feature. >>>=20 >> I'm not sure what more to say hear regarding the snapshot feature or >> what might have been done in the past. We need a solution for the >> present. If you have any comments related to the follow up reviews >> submitted by UPB, I'm sure they'd love to hear them. >> And lastly: I get that FreeBSD is a non paid volunteer project for >> most. Without the efforts of folks like Peter, Neel, John and others, >> there would be no bhyve. I'm not saying that they, as project >> maintainers, should somehow be doing more. We all have limited time >> to invest, paid work to do and families to feed. I'm asking if there >> are other developers that might be willing and able to help with >> reviews? Is there something the FreeBSD foundation can do help out in >> situations like these? >> Thanks, >> -Matthew >> =20 >=20 > UPB has developed some interesting features and I'd like to see those > in tree. I can take some time to review the patches. Nevertheless, we > really need the snapshot feature compiled in by default. Otherwise, > it's wasted time for all of us. >=20 >=20 > --=20 > Kind regards, > Corvin --Apple-Mail=_B895AB4D-5F14-4A57-B65C-D1DB37F70C9D Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Just add some = plans for me:

 1. Describe snapshot file format: = One file for snapshot.

 2. Implement = snapshot/resume via nvlist.

  =  nvlist implementation brings:

  • Versioning
  • Easy debugging, getting = saved values, etc.
  • Validate restored variables: types, sized, = etc.
  • Add optional variables without breaking backward = compatibility (resume can be performed with old = snapshots)
  • Remove variables without breaking backward = compatibility
  • Use one file for snapshot
  • Improve restore = command line:  "bhyve -r $snapshot=E2=80=9D, i.e. w/o additional = arguments

=E2=80=94=E2=80=94
= Vitaliy Gusev

before extending the functionality off = the top of it.

Yup. See above. I appreciate your = input, but the goal of live
migration was set in 2016 with a = prototype first demonstrated in
2018. How long do you suggest a = developer wait without review
feedback before moving forward out of = tree?

The = snapshot feature isn't compiled in by default. So, it's likely = that
changes break it and = only a few people are testing it.

We have = to focus on getting this into the tree.

There are experimental patches for all these features that = were
developed by students at UPB. In a lot of cases, there are = open
reviews that have been waiting on feedback for = ages.

In general, most people don't want to review = large experimental
patches.

Yup. That approach = was attempted with the Warm Migration patches.
=46rom slide 17 in = Elena's presentation:

 First review opened in 2021: = https://reviews.freebsd.org/D28270
 5 reviews from 2022 starting = with https://reviews.freebsd.org/D34717
(same feature split in = multiple parts)
 
 A similar request was made recently = to Gusev Vitaliy WRT the
multiple device support patch which he took = ownership of. Thanks for
adding feedback to that review BTW. We'll = see how that pans out = ...

 https://reviews.freebsd.org/D35590

I've already reviewed Vitaliy's multi = device support patch and people
had = more than enough time to complain about it. I'm going to commit = it
as soon as he splits his = commit.
  
 The = case is quite plain. I'm not sure what the solution is to
this 
 problem. I'd love = to hear feedback from the community about how
I've got 
 this completely = wrong and how the course could be corrected.
That would 
 be = something.


My perspective is that it would have = been better to focus student
efforts on completing the snapshot = feature. By completing the
snapshot feature, I mean getting the code = into a state where it's
compiled in by default and no longer = considered an experimental
feature.

I'm not sure = what more to say hear regarding the snapshot feature or
what might = have been done in the past. We need a solution for the
present. If = you have any comments related to the follow up reviews
submitted by = UPB, I'm sure they'd love to hear them.
And lastly: I get that = FreeBSD is a non paid volunteer project for
most. Without the efforts = of folks like Peter, Neel, John and others,
there would be no bhyve. = I'm not saying that they, as project
maintainers, should somehow be = doing more. We all have limited time
to invest, paid work to do and = families to feed. I'm asking if there
are other developers that might = be willing and able to help with
reviews? Is there something the = FreeBSD foundation can do help out in
situations like = these?
Thanks,
-Matthew
 

UPB has developed some interesting features = and I'd like to see those
in = tree. I can take some time to review the patches. Nevertheless, = we
really need the snapshot = feature compiled in by default. Otherwise,
it's = wasted time for all of us.


-- 
Kind regards,
Corvin

= = --Apple-Mail=_B895AB4D-5F14-4A57-B65C-D1DB37F70C9D-- From nobody Tue May 2 11:00:31 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9cZ65lysz48rCs; Tue, 2 May 2023 11:00:46 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Received: from mail-lf1-x12a.google.com (mail-lf1-x12a.google.com [IPv6:2a00:1450:4864:20::12a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9cZ51gWMz3kDK; Tue, 2 May 2023 11:00:45 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x12a.google.com with SMTP id 2adb3069b0e04-4f00d41df22so27491739e87.1; Tue, 02 May 2023 04:00:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1683025243; x=1685617243; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=Kedj9pEFkDCLD2y7qQ+eu6dhg5vW4Tr+qQw81KkXqec=; b=SVF0KGKMy9QYiqQWUm1oj2V8JOQOuYNIHKnADEqibjxn4r/qHZfxTSP1pCcmqH47yC fU8VW5T5KCOyy5Kb5SardrWBCL0wJF4CzAN/74Pll89Z51UVxIYpI9TLwr+9lEJMre6y SFgsIsp/nzFRIRnRfQnkHyA4ZQkKKUxGx/pqGKXt4SgY97aLvXA7xzQ3gd0JVUwwFgrt PpOH8vUPHRUsb0gczUfP5PYGc000zTti1K6Fp3q844j25NdB9zrT2v3Hss473xnUy0Ak 9Hs0uc2E0RlPvGaD/jmhBihshXNIEE4llP3Kp1D6G7Op+0ZWqnEpYUp87Z1zbM1T4AI5 FWpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683025243; x=1685617243; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Kedj9pEFkDCLD2y7qQ+eu6dhg5vW4Tr+qQw81KkXqec=; b=FR0WRM8/jEImOpkevmvuhPc7x1Jwk6ZhpNln3AQY7nPp1sXkmit1+h6ZUjONt94ktq wCxaUiMUL8JrDaaPtzEdcVAsI3wjgZ9MydRjq3crh8pnHXbG7Nbl5JqcjwNaYs9sUMnl Rxr78LgG4zNl7u+Yhr7Ma7BZOdy3ZRPczdaJSR6t3Mrv9pguq1v50z/GDPC801ZxPF9l rKSjHkRYYIEhjMT97jWbhGBazTySOhXWFoQ/l885WMoAr/ubnlnMAjzY4vYvirGOCcge Xb12WosYmcpBI5ZnH0fgJ/TtY43WL7cPuZlevszotfD0GqBPKS3Hx5YXmPemMIOl9CXx FkOw== X-Gm-Message-State: AC+VfDwjA0byoTAabFas82PbtWHE5Aa9RWv4LqMfY0qiDtWGdbmdWbXZ LP4SydBoivubNfYKxEuU75sJt/PFKCyoJQ== X-Google-Smtp-Source: ACHHUZ5aRGdwhHaiaoV06D14L0shtXjRpbL3GkvbDJnm0KNZgFbF+E9NP/Pt94QwBi7EH8DBb0XkZA== X-Received: by 2002:a05:651c:231b:b0:2a8:bc78:2e3f with SMTP id bi27-20020a05651c231b00b002a8bc782e3fmr6329301ljb.0.1683025242982; Tue, 02 May 2023 04:00:42 -0700 (PDT) Received: from smtpclient.apple ([188.187.60.230]) by smtp.gmail.com with ESMTPSA id y14-20020a2e320e000000b00295da33c42dsm5242725ljy.15.2023.05.02.04.00.41 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 May 2023 04:00:42 -0700 (PDT) From: Vitaliy Gusev Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_B895AB4D-5F14-4A57-B65C-D1DB37F70C9D" List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.500.231\)) Subject: Re: BHYVE_SNAPSHOT Date: Tue, 2 May 2023 14:00:31 +0300 In-Reply-To: <3ab4e6d94fb0153fb6ff4a53ac6f53b2eaae0cf7.camel@FreeBSD.org> Cc: Matthew Grooms , Rob Wing , freebsd-hackers@freebsd.org, freebsd-virtualization@freebsd.org, Elena Mihailescu , Mihai Carabas To: =?utf-8?Q?Corvin_K=C3=B6hne?= References: <89c84bd7-a925-02f4-acbe-12c3000e7007@shrew.net> <3ab4e6d94fb0153fb6ff4a53ac6f53b2eaae0cf7.camel@FreeBSD.org> X-Mailer: Apple Mail (2.3731.500.231) X-Rspamd-Queue-Id: 4Q9cZ51gWMz3kDK X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_RCPT(0.00)[]; TAGGED_FROM(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_B895AB4D-5F14-4A57-B65C-D1DB37F70C9D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Just add some plans for me: 1. Describe snapshot file format: One file for snapshot. 2. Implement snapshot/resume via nvlist. nvlist implementation brings: Versioning Easy debugging, getting saved values, etc. Validate restored variables: types, sized, etc. Add optional variables without breaking backward compatibility (resume = can be performed with old snapshots) Remove variables without breaking backward compatibility Use one file for snapshot Improve restore command line: "bhyve -r $snapshot=E2=80=9D, i.e. w/o = additional arguments =E2=80=94=E2=80=94 Vitaliy Gusev >>> before extending the functionality off the top of it. >>=20 >> Yup. See above. I appreciate your input, but the goal of live >> migration was set in 2016 with a prototype first demonstrated in >> 2018. How long do you suggest a developer wait without review >> feedback before moving forward out of tree? >=20 > The snapshot feature isn't compiled in by default. So, it's likely = that > changes break it and only a few people are testing it. >=20 > We have to focus on getting this into the tree. >=20 >>>> There are experimental patches for all these features that were >>>> developed by students at UPB. In a lot of cases, there are open >>>> reviews that have been waiting on feedback for ages. >>>=20 >>> In general, most people don't want to review large experimental >>> patches. >>=20 >> Yup. That approach was attempted with the Warm Migration patches. >> =46rom slide 17 in Elena's presentation: >>=20 >> First review opened in 2021: https://reviews.freebsd.org/D28270 >> 5 reviews from 2022 starting with https://reviews.freebsd.org/D34717 >> (same feature split in multiple parts) >> =20 >> A similar request was made recently to Gusev Vitaliy WRT the >> multiple device support patch which he took ownership of. Thanks for >> adding feedback to that review BTW. We'll see how that pans out ... >>=20 >> https://reviews.freebsd.org/D35590 >>=20 >=20 > I've already reviewed Vitaliy's multi device support patch and people > had more than enough time to complain about it. I'm going to commit it > as soon as he splits his commit. > =20 >>>> The case is quite plain. I'm not sure what the solution is to >>>> this=20 >>>> problem. I'd love to hear feedback from the community about how >>>> I've got=20 >>>> this completely wrong and how the course could be corrected. >>>> That would=20 >>>> be something. >>>>=20 >>>=20 >>> My perspective is that it would have been better to focus student >>> efforts on completing the snapshot feature. By completing the >>> snapshot feature, I mean getting the code into a state where it's >>> compiled in by default and no longer considered an experimental >>> feature. >>>=20 >> I'm not sure what more to say hear regarding the snapshot feature or >> what might have been done in the past. We need a solution for the >> present. If you have any comments related to the follow up reviews >> submitted by UPB, I'm sure they'd love to hear them. >> And lastly: I get that FreeBSD is a non paid volunteer project for >> most. Without the efforts of folks like Peter, Neel, John and others, >> there would be no bhyve. I'm not saying that they, as project >> maintainers, should somehow be doing more. We all have limited time >> to invest, paid work to do and families to feed. I'm asking if there >> are other developers that might be willing and able to help with >> reviews? Is there something the FreeBSD foundation can do help out in >> situations like these? >> Thanks, >> -Matthew >> =20 >=20 > UPB has developed some interesting features and I'd like to see those > in tree. I can take some time to review the patches. Nevertheless, we > really need the snapshot feature compiled in by default. Otherwise, > it's wasted time for all of us. >=20 >=20 > --=20 > Kind regards, > Corvin --Apple-Mail=_B895AB4D-5F14-4A57-B65C-D1DB37F70C9D Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Just add some = plans for me:

 1. Describe snapshot file format: = One file for snapshot.

 2. Implement = snapshot/resume via nvlist.

  =  nvlist implementation brings:

  • Versioning
  • Easy debugging, getting = saved values, etc.
  • Validate restored variables: types, sized, = etc.
  • Add optional variables without breaking backward = compatibility (resume can be performed with old = snapshots)
  • Remove variables without breaking backward = compatibility
  • Use one file for snapshot
  • Improve restore = command line:  "bhyve -r $snapshot=E2=80=9D, i.e. w/o additional = arguments

=E2=80=94=E2=80=94
= Vitaliy Gusev

before extending the functionality off = the top of it.

Yup. See above. I appreciate your = input, but the goal of live
migration was set in 2016 with a = prototype first demonstrated in
2018. How long do you suggest a = developer wait without review
feedback before moving forward out of = tree?

The = snapshot feature isn't compiled in by default. So, it's likely = that
changes break it and = only a few people are testing it.

We have = to focus on getting this into the tree.

There are experimental patches for all these features that = were
developed by students at UPB. In a lot of cases, there are = open
reviews that have been waiting on feedback for = ages.

In general, most people don't want to review = large experimental
patches.

Yup. That approach = was attempted with the Warm Migration patches.
=46rom slide 17 in = Elena's presentation:

 First review opened in 2021: = https://reviews.freebsd.org/D28270
 5 reviews from 2022 starting = with https://reviews.freebsd.org/D34717
(same feature split in = multiple parts)
 
 A similar request was made recently = to Gusev Vitaliy WRT the
multiple device support patch which he took = ownership of. Thanks for
adding feedback to that review BTW. We'll = see how that pans out = ...

 https://reviews.freebsd.org/D35590

I've already reviewed Vitaliy's multi = device support patch and people
had = more than enough time to complain about it. I'm going to commit = it
as soon as he splits his = commit.
  
 The = case is quite plain. I'm not sure what the solution is to
this 
 problem. I'd love = to hear feedback from the community about how
I've got 
 this completely = wrong and how the course could be corrected.
That would 
 be = something.


My perspective is that it would have = been better to focus student
efforts on completing the snapshot = feature. By completing the
snapshot feature, I mean getting the code = into a state where it's
compiled in by default and no longer = considered an experimental
feature.

I'm not sure = what more to say hear regarding the snapshot feature or
what might = have been done in the past. We need a solution for the
present. If = you have any comments related to the follow up reviews
submitted by = UPB, I'm sure they'd love to hear them.
And lastly: I get that = FreeBSD is a non paid volunteer project for
most. Without the efforts = of folks like Peter, Neel, John and others,
there would be no bhyve. = I'm not saying that they, as project
maintainers, should somehow be = doing more. We all have limited time
to invest, paid work to do and = families to feed. I'm asking if there
are other developers that might = be willing and able to help with
reviews? Is there something the = FreeBSD foundation can do help out in
situations like = these?
Thanks,
-Matthew
 

UPB has developed some interesting features = and I'd like to see those
in = tree. I can take some time to review the patches. Nevertheless, = we
really need the snapshot = feature compiled in by default. Otherwise,
it's = wasted time for all of us.


-- 
Kind regards,
Corvin

= = --Apple-Mail=_B895AB4D-5F14-4A57-B65C-D1DB37F70C9D-- From nobody Tue May 2 15:38:15 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9kk82gNjz495rq; Tue, 2 May 2023 15:38:08 +0000 (UTC) (envelope-from rob.fx907@gmail.com) Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9kk73lkfz4HFB; Tue, 2 May 2023 15:38:07 +0000 (UTC) (envelope-from rob.fx907@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=HgATklFY; spf=pass (mx1.freebsd.org: domain of rob.fx907@gmail.com designates 2a00:1450:4864:20::534 as permitted sender) smtp.mailfrom=rob.fx907@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ed1-x534.google.com with SMTP id 4fb4d7f45d1cf-504d149839bso826512a12.1; Tue, 02 May 2023 08:38:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1683041886; x=1685633886; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=errR5P0dAWMC6MBSES4ce2BPg3tEKzLkevUy/ewpve8=; b=HgATklFY731IyqEuM6nnqsL7XrIQec9by8fB6fMto4i8CZp6B8Vh7lW3EYXu8HqkAA roc04CL0pl6KdryWIsH5gDnOGwrcFt4pApYKEGmsKOGYyEXUFYLwY0upgyCzLa8ke6+/ vbf0tZ+H41aV1shzQ5E42ltVQZspHK5izp3eRnZ9eTPHToutqSNYR2jsAcJmW6WeK9um HMemv51B6VDC5lBdJjooONx46y2wzmSB2vmwlSh+y32Tbm3M3CIccAvXMuUlf+3oG2fY 8HKbcgbDG+3Wx6PlLJXhnkFkHnGeSoq6670sXvw4NZJ2BgzjPODbCnDwxonztQJTPOHh L4Nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683041886; x=1685633886; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=errR5P0dAWMC6MBSES4ce2BPg3tEKzLkevUy/ewpve8=; b=Lra+FUjfruB19cQNp7etQoWmvpAP1BWBk636ge0UqKiXE6e9sAMfdqAIKrIDdd1+q3 RpnPbfrx1zqkPGEKvjxr3/7cpb7nHnFQX5dRfoe9JBJaXc7F+H/+rFvb6IFbXWJy0hIp qCQ5y/ha6AIOPW3kLT4YLk+ZGEIS5Xobj1b1c31D2d3E+yEJutRcYD1dwhVwTcYrvgdi ELpNP5X5qb1hJ5qhOmMonU8IAgoMfrksSRhiugjIHf2lvLpmwNusrdv1cPOSAMJQ86hB z5ZLVSKJINje7Z3IYZByry8LRUShu+kKyy60WYTDXu/zZENkrnhjOUJaeaXHojUefht/ TY6Q== X-Gm-Message-State: AC+VfDwC6NjqVyHFQeg2P5iRLnhxkTXHhtZLiWPTWudLkM6MH566kkIV fF9scA/uq/onXo98464qtRDYemjnKlQkenyJr4gQOgDNgyc= X-Google-Smtp-Source: ACHHUZ7vLi5nj6s+ZRe+hwO2SBVskbyMbKh/VNjgmUpyBWYkyWpFXx21oJg896p71yaIrj+ZptldzoiTGE1VtDcWW2w= X-Received: by 2002:a17:906:7a49:b0:95f:db5f:73b7 with SMTP id i9-20020a1709067a4900b0095fdb5f73b7mr2775421ejo.0.1683041886032; Tue, 02 May 2023 08:38:06 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <89c84bd7-a925-02f4-acbe-12c3000e7007@shrew.net> <3ab4e6d94fb0153fb6ff4a53ac6f53b2eaae0cf7.camel@FreeBSD.org> In-Reply-To: From: Rob Wing Date: Tue, 2 May 2023 07:38:15 -0800 Message-ID: Subject: Re: BHYVE_SNAPSHOT To: Vitaliy Gusev Cc: =?UTF-8?Q?Corvin_K=C3=B6hne?= , Matthew Grooms , freebsd-hackers@freebsd.org, freebsd-virtualization@freebsd.org, Elena Mihailescu , Mihai Carabas Content-Type: multipart/alternative; boundary="000000000000c258dc05fab7b9bf" X-Spamd-Result: default: False [0.12 / 15.00]; SUSPICIOUS_RECIPS(1.50)[]; SUBJ_ALL_CAPS(1.05)[14]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_SPAM_SHORT(0.57)[0.568]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TAGGED_RCPT(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org,freebsd-virtualization@freebsd.org]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::534:from]; ARC_NA(0.00)[]; TAGGED_FROM(0.00)[]; FREEMAIL_CC(0.00)[freebsd.org,shrew.net,gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_SEVEN(0.00)[7]; MID_RHS_MATCH_FROMTLD(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TO_DN_SOME(0.00)[] X-Rspamd-Queue-Id: 4Q9kk73lkfz4HFB X-Spamd-Bar: / X-ThisMailContainsUnwantedMimeParts: N --000000000000c258dc05fab7b9bf Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, May 2, 2023 at 3:00=E2=80=AFAM Vitaliy Gusev wrote: > Just add some plans for me: > > 1. Describe snapshot file format: One file for snapshot. > > 2. Implement snapshot/resume via nvlist. > > *nvlist* implementation brings: > > > - Versioning > - Easy debugging, getting saved values, etc. > - Validate restored variables: types, sized, etc. > - Add optional variables without breaking backward compatibility > (resume can be performed with old snapshots) > - Remove variables without breaking backward compatibility > - Use one file for snapshot > - Improve restore command line: "bhyve -r $snapshot=E2=80=9D, i.e. w/= o > additional arguments > > Do you plan on saving the guest memory into the nvlist? If I have VM with 8 gigs of memory, will the nvlist implementation allocate 8 gigs of memory for the nvlist then write it out to disk? Or..? All the bullet points look good to me. -Rob --000000000000c258dc05fab7b9bf Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Tue, May 2, 2023 at 3:00=E2=80=AFA= M Vitaliy Gusev <gusev.vitali= y@gmail.com> wrote:
Just add some plans for me:

=C2=A01. Des= cribe snapshot file format: One file for snapshot.

=C2=A02. Implement snapshot/resume via nvlist.

= =C2=A0 =C2=A0nvlist implementation brings:

=
  • Versioning
  • Easy debugging, getting saved values, etc.
    • <= li>Validate restored variables: types, sized, etc.
  • Add optional var= iables without breaking backward compatibility (resume can be performed wit= h old snapshots)
  • Remove variables without breaking backward compati= bility
  • Use one file for snapshot
  • Improve restore command li= ne: =C2=A0"bhyve -r $snapshot=E2=80=9D, i.e. w/o additional arguments<= /li>

Do you plan on saving= the guest memory into the nvlist? If I have VM with 8 gigs of memory, will= the nvlist implementation allocate 8 gigs of memory for the nvlist then wr= ite it out to disk? Or..?

All the bullet points lo= ok good to me.

-Rob
--000000000000c258dc05fab7b9bf-- From nobody Tue May 2 16:04:02 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9lJK5rZdz496fl; Tue, 2 May 2023 16:04:17 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Received: from mail-lj1-x22b.google.com (mail-lj1-x22b.google.com [IPv6:2a00:1450:4864:20::22b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9lJJ6PW3z4LP8; Tue, 2 May 2023 16:04:16 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=OcmOsu+W; spf=pass (mx1.freebsd.org: domain of gusev.vitaliy@gmail.com designates 2a00:1450:4864:20::22b as permitted sender) smtp.mailfrom=gusev.vitaliy@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-lj1-x22b.google.com with SMTP id 38308e7fff4ca-2aa39ce5d26so40035301fa.1; Tue, 02 May 2023 09:04:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1683043454; x=1685635454; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=QU8tYAy4WZ9pbgxX9JzfnrW1kukUjmYcxtPRSGSst5I=; b=OcmOsu+WBcaN6sJt5MPJWkCvXdQnYfaKKZMtdOwFEuitSQivYBliqWYFt0xL7YwPNO rfs+nCnMI6lvHb8v0bGPCDpNzQigv29EWubYdbaWrPILVSKTLH24jlVZ9Tsu8AW6OrlD w2XNK445Gl+DpDHpnEZLILu9RwDNxahtL96X2VZcFV/t4z9obUfgB0let5FBx10qdnAA RMrTJKKnSAC0IJnFO8us2mdVzP62r9BwhzdaN8JcLakefeDn0VSTdOKcYEwLqpBhbLpU YDIhrA3WpmMznkSQpc8tAZ9RoZNdnP+DYGsubijHddtcbuqse6crVppJOklN8e433Kyt joPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683043454; x=1685635454; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QU8tYAy4WZ9pbgxX9JzfnrW1kukUjmYcxtPRSGSst5I=; b=DUGskCo+1Uz6EUgXlUgpV68jKjOVOZ9VJmYbnzqL3+u57aB1YzLKxGhZhFAnvtCPDT o0dPwIDmtJUcdupS0sgAISHqu5khgq6XYOSfHKo1VuNpdF/bdQ71CvFWiKVGcn8HizMO OuZaXtb7GlKCsg+Q7i2XaD/t7V6RR/xu3xfWNvKkIgkk1cOwad3cTlowfuXQbr47qc1U qitUNuaXbtGbtlMs0+nAbMlLbo8vh4pJOdi+v+HbXf7b/IRnn2kmz9fZ48ndUQ3347wi ulzLbHnpI/wTDWX9k8a9SRTvfHzktTd9UpfhWFISdLn8HbfZC3Aat6Ox4+6YJqMtnoDj ZpkA== X-Gm-Message-State: AC+VfDxLa5MRlRcNtG2cFSjX2Pc3J4LjY+R5yUYfQ7zQISEYgxfjTxVZ Y5ehXkcql1JmwUESsNKj210= X-Google-Smtp-Source: ACHHUZ6CfOOyQ4QngnjYymTo/zwri0JCT4wG0UEAwWeNTXtVmnU7rHrHeuhlbi2HNO0Xa08UZF96rw== X-Received: by 2002:a2e:98c5:0:b0:2a8:e6f8:301e with SMTP id s5-20020a2e98c5000000b002a8e6f8301emr4595615ljj.28.1683043454199; Tue, 02 May 2023 09:04:14 -0700 (PDT) Received: from smtpclient.apple ([188.187.60.230]) by smtp.gmail.com with ESMTPSA id w4-20020ac25d44000000b004eb0c51780bsm5443260lfd.29.2023.05.02.09.04.13 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 May 2023 09:04:13 -0700 (PDT) From: Vitaliy Gusev Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_07C1A3F1-22E1-41CC-8145-2FE0D588B957" List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.500.231\)) Subject: Re: BHYVE_SNAPSHOT Date: Tue, 2 May 2023 19:04:02 +0300 In-Reply-To: Cc: =?utf-8?Q?Corvin_K=C3=B6hne?= , Matthew Grooms , freebsd-hackers@freebsd.org, freebsd-virtualization@freebsd.org, Elena Mihailescu , Mihai Carabas To: Rob Wing References: <89c84bd7-a925-02f4-acbe-12c3000e7007@shrew.net> <3ab4e6d94fb0153fb6ff4a53ac6f53b2eaae0cf7.camel@FreeBSD.org> X-Mailer: Apple Mail (2.3731.500.231) X-Spamd-Result: default: False [-2.44 / 15.00]; SUBJ_ALL_CAPS(1.05)[14]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.990]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TAGGED_RCPT(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::22b:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org,freebsd-virtualization@freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; TAGGED_FROM(0.00)[]; RCPT_COUNT_SEVEN(0.00)[7]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FREEMAIL_TO(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_CC(0.00)[freebsd.org,shrew.net,gmail.com] X-Rspamd-Queue-Id: 4Q9lJJ6PW3z4LP8 X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_07C1A3F1-22E1-41CC-8145-2FE0D588B957 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On 2 May 2023, at 18:38, Rob Wing wrote: >=20 > Do you plan on saving the guest memory into the nvlist? If I have VM = with 8 gigs of memory, will the nvlist implementation allocate 8 gigs of = memory for the nvlist then write it out to disk? Or..? >=20 > All the bullet points look good to me. >=20 > -Rob Of course no. Guest memory should be saved as is. As possible = improvement - only dirty memory/pages.=20 I was taking about saving with nvlist: device=E2=80=99s variables, = registers, internal data, etc.=20 Overall description will be provided shortly. =E2=80=94=E2=80=94=E2=80=94 Vitaliy Gusev --Apple-Mail=_07C1A3F1-22E1-41CC-8145-2FE0D588B957 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On 2 May = 2023, at 18:38, Rob Wing <rob.fx907@gmail.com> = wrote:

Do you plan on saving the = guest memory into the nvlist? If I have VM with 8 gigs of memory, will = the nvlist implementation allocate 8 gigs of memory for the nvlist then = write it out to disk? Or..?

All the bullet points look good to = me.

-Rob


Of course no. Guest memory should be saved as is. As possible = improvement  -  only dirty = memory/pages. 

I was taking about saving = with nvlist: device=E2=80=99s variables, registers, internal data, = etc. 

Overall description will be provided = shortly.

=E2=80=94=E2=80=94=E2=80=94
Vi= taliy Gusev


= --Apple-Mail=_07C1A3F1-22E1-41CC-8145-2FE0D588B957-- From nobody Tue May 2 19:25:12 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9qm16pKyz49KCp; Tue, 2 May 2023 19:25:05 +0000 (UTC) (envelope-from rob.fx907@gmail.com) Received: from mail-ej1-x62b.google.com (mail-ej1-x62b.google.com [IPv6:2a00:1450:4864:20::62b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9qm11hsFz44sC; Tue, 2 May 2023 19:25:05 +0000 (UTC) (envelope-from rob.fx907@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x62b.google.com with SMTP id a640c23a62f3a-9594916df23so130493866b.1; Tue, 02 May 2023 12:25:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1683055503; x=1685647503; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=LiOIjWmt1NbUmkN9IZ2lHw/4SCRlFTaji52oJE1sx90=; b=MDpEZB2P791/Y+7EPZF+FcImymz5gvGq5hQPaXTGOvD2KrlmFnGAiYyuaVzkiwwpCc OFiwedgo3l4oA3PVAXWPirsAbJtdzwymSrN2ZgR89dNxMDp0xseHNjyRmk33G82y4TrN 5vYsCwisrI1b9j8s4Pb7z9iq0HTov8DpCoezJ9hCmQx3hsIOKiHGjOkVJTBoSFEuo79y if+ffpbW7awlNVjdEdXwR9zu1UJgHciD8SxQC1aPgFRvgv+W1v+eIWwUNPfc1zr+WsZb 5CS2bVSD/Y9v7zMXAAjWVChsV5/BMZOkzb5J9s/MCeL+FA/ll/MBcHXDzkvOCcp7VcdR Jdww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683055503; x=1685647503; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=LiOIjWmt1NbUmkN9IZ2lHw/4SCRlFTaji52oJE1sx90=; b=Ux1lEikvmsMv3xX+yFR+0wxKocy/ur0taVbNk/tF1T/ozlMF6/YDp++xmJXC67znBs LpTP3yDfXNJucrmBNPugT/9SfqNtxNAeOwppHctGoY5DJ3lTgweLTJzgz6eqG2FjiLJo xajW7aBhmD1BVQuWhhKaGxZA/Ru93tVj8hoGjFPugCX/aeRBwd+iI6hHTmK2T5z88kEb MixJHCdTTshOjVoQNgxr8C6uB5zMFVhWWeFe5WPUVgZzjgjQN1r5O0hU6TJ+anDk3YZZ hUxXhodADL9ERIlyf4kvApTtnd3KJxbwlb5nKt8jT0fKiy4NnGr8jQfeXLjmovH+vGhf nOoA== X-Gm-Message-State: AC+VfDxx1RrWsA/7lMNHV7lZqNQ7FxRsC0S5IYRknLvJP/Wq+I+O9dbB og/7mBewDfqwoC85Z8GMJMzfZdKrUmQFhtKDwyEy0MJFqHo= X-Google-Smtp-Source: ACHHUZ7AEZ/Ks403S5obdFn8aj7AsOxHE/74/xdMWWVcwHjtYRzFUWGXczp/1bRnaO6yM6mrbzdsqYETFJBRyr6zYY0= X-Received: by 2002:a17:906:7a01:b0:94f:66af:b1f7 with SMTP id d1-20020a1709067a0100b0094f66afb1f7mr2938797ejo.1.1683055503301; Tue, 02 May 2023 12:25:03 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <89c84bd7-a925-02f4-acbe-12c3000e7007@shrew.net> In-Reply-To: <89c84bd7-a925-02f4-acbe-12c3000e7007@shrew.net> From: Rob Wing Date: Tue, 2 May 2023 11:25:12 -0800 Message-ID: Subject: Re: BHYVE_SNAPSHOT To: Matthew Grooms Cc: freebsd-hackers@freebsd.org, freebsd-virtualization@freebsd.org, elenamihailescu22@gmail.com, Mihai Carabas , gusev.vitaliy@gmail.com Content-Type: multipart/alternative; boundary="0000000000006964a805fabae503" X-Rspamd-Queue-Id: 4Q9qm11hsFz44sC X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_RCPT(0.00)[]; TAGGED_FROM(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --0000000000006964a805fabae503 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, May 1, 2023 at 10:16=E2=80=AFPM Matthew Grooms = wrote: > Yup. See above. I appreciate your input, but the goal of live migration > was set in 2016 with a prototype first demonstrated in 2018. How long do > you suggest a developer wait without review feedback before moving forwar= d > out of tree? > Yea..I'd say a submitter should receive feedback within a week (or so) and when that passes, try rattling the cage of relevant developers to ack a timeline. My comments for the live/warm migration review is to rebase it after the snapshot feature is compiled in by default. > Yup. That approach was attempted with the Warm Migration patches. From > slide 17 in Elena's presentation: > > First review opened in 2021: https://reviews.freebsd.org/D28270 > 5 reviews from 2022 starting with https://reviews.freebsd.org/D34717 > (same feature split in multiple parts) > I understand that, as a submitter, this is frustrating - to spend a non-trivial amount of time preparing/submitting a review and then receive no feedback, not even a "changes too big" or "nope". For future reference, if UPB does another project - might be worth trying to seek out a FreeBSD mentor that could guide the students and be willing to usher the changes in, not unlike a GSOC project. --0000000000006964a805fabae503 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

On Mon, May 1,= 2023 at 10:16=E2=80=AFPM Matthew Grooms <mgrooms@shrew.net> wrote:

= Yup. See above. I appreciate your input, but the goal of live migration was set in 2016 with a prototype first demonstrated in 2018. How long do you suggest a developer wait without review feedback before moving forward out of tree?


Yea..I'd say a submitter should receive feedback w= ithin a week (or so) and when that passes, try rattling the cage of relevan= t developers to ack a timeline. My comments for the live/warm migration rev= iew is to rebase it after the snapshot feature is compiled in by default.
=C2=A0
=

=20

Yup. That approach was attempted with the Warm Migration patches. From slide 17 in Elena's presentation:

First review opened in 2021: https://reviews.freebsd.org/D28270
5 reviews from 2022 starting with http= s://reviews.freebsd.org/D34717 (same feature split in multiple parts)


<= /div>
I understand that, as a submitter, this is frustrating - to spend= a non-trivial amount of time preparing/submitting a review and then receiv= e no feedback, not even a "changes too big" or "nope".<= /div>

For future reference, if UPB does another project = - might be worth trying to seek out a FreeBSD mentor that could guide the s= tudents and be willing to usher the changes in, not unlike a GSOC project.<= br>
--0000000000006964a805fabae503-- From nobody Tue May 2 21:16:48 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9tFB0X0Lz48BMY; Tue, 2 May 2023 21:17:02 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx2.shrew.net (mx2.shrew.net [38.97.5.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9tF919MVz4N1s; Tue, 2 May 2023 21:17:00 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 38.97.5.132 as permitted sender) smtp.mailfrom=mgrooms@shrew.net; dmarc=none Received: from mail.shrew.net (mail.shrew.prv [10.24.10.20]) by mx2.shrew.net (8.15.2/8.15.2) with ESMTP id 342LGsRT089307; Tue, 2 May 2023 16:16:54 -0500 (CDT) (envelope-from mgrooms@shrew.net) Received: from [10.22.200.32] (unknown [136.49.230.220]) by mail.shrew.net (Postfix) with ESMTPSA id 1D8EA193A82; Tue, 2 May 2023 16:16:49 -0500 (CDT) Content-Type: multipart/alternative; boundary="------------YGulZ2VgxDlosOBeXKDZgfz0" Message-ID: <7d649a69-8c68-9b15-7999-2102873c1833@shrew.net> Date: Tue, 2 May 2023 16:16:48 -0500 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.10.0 Subject: Re: BHYVE_SNAPSHOT Content-Language: en-US To: Rob Wing Cc: freebsd-hackers@freebsd.org, freebsd-virtualization@freebsd.org, elenamihailescu22@gmail.com, Mihai Carabas , gusev.vitaliy@gmail.com References: <89c84bd7-a925-02f4-acbe-12c3000e7007@shrew.net> From: Matthew Grooms In-Reply-To: X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (mx2.shrew.net [10.24.10.11]); Tue, 02 May 2023 16:16:54 -0500 (CDT) X-Spamd-Result: default: False [-0.32 / 15.00]; SUBJ_ALL_CAPS(1.05)[14]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.44)[-0.443]; NEURAL_SPAM_MEDIUM(0.37)[0.370]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org,freebsd-hackers@freebsd.org]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:174, ipnet:38.0.0.0/8, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_FIVE(0.00)[6]; FREEMAIL_CC(0.00)[freebsd.org,gmail.com]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; TO_DN_SOME(0.00)[]; DMARC_NA(0.00)[shrew.net]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4Q9tF919MVz4N1s X-Spamd-Bar: / X-ThisMailContainsUnwantedMimeParts: N This is a multi-part message in MIME format. --------------YGulZ2VgxDlosOBeXKDZgfz0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 5/2/23 14:25, Rob Wing wrote: > > For future reference, if UPB does another project - might be worth > trying to seek out a FreeBSD mentor that could guide the students and > be willing to usher the changes in, not unlike a GSOC project. Good idea. That would be wonderful. -Matthew --------------YGulZ2VgxDlosOBeXKDZgfz0 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 7bit
On 5/2/23 14:25, Rob Wing wrote:

For future reference, if UPB does another project - might be worth trying to seek out a FreeBSD mentor that could guide the students and be willing to usher the changes in, not unlike a GSOC project.

Good idea. That would be wonderful.

-Matthew

--------------YGulZ2VgxDlosOBeXKDZgfz0-- From nobody Tue May 2 21:40:14 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9tm74Gsbz48BdP for ; Tue, 2 May 2023 21:40:23 +0000 (UTC) (envelope-from steffen@sdaoden.eu) Received: from sdaoden.eu (sdaoden.eu [217.144.132.164]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9tm62QK4z3D3N for ; Tue, 2 May 2023 21:40:22 +0000 (UTC) (envelope-from steffen@sdaoden.eu) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of steffen@sdaoden.eu designates 217.144.132.164 as permitted sender) smtp.mailfrom=steffen@sdaoden.eu; dmarc=none Date: Tue, 02 May 2023 23:40:14 +0200 Author: Steffen Nurpmeso From: Steffen Nurpmeso To: tech@openbsd.org, tech-userlevel@netbsd.org, freebsd-hackers@freebsd.org Subject: usr.bin/mail: cmd3.c:bangexp(): "borked"?, and BSD fails POSIX compat Message-ID: <20230502214014._zIz6%steffen@sdaoden.eu> Mail-Followup-To: tech@openbsd.org, tech-userlevel@netbsd.org, freebsd-hackers@freebsd.org User-Agent: s-nail v14.9.24-461-g2a5db04708 OpenPGP: id=EE19E1C1F2F7054F8D3954D8308964B51883A0DD; url=https://ftp.sdaoden.eu/steffen.asc; preference=signencrypt BlahBlahBlah: Any stupid boy can crush a beetle. But all the professors in the world can make no bugs. List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [0.47 / 15.00]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_SPAM_MEDIUM(0.77)[0.771]; R_SPF_ALLOW(-0.20)[+a]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_SHORT(-0.00)[-0.005]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_ZERO(0.00)[0]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[sdaoden.eu]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:15987, ipnet:217.144.128.0/20, country:DE]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_NONE(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[] X-Rspamd-Queue-Id: 4Q9tm62QK4z3D3N X-Spamd-Bar: / X-ThisMailContainsUnwantedMimeParts: N Hallo, and sorry for the cross-post, but so all in one (maybe) go. This is about a niche "feature" of mail, the shell command "bang" (! / ~! in compose mode): ? !echo no!bang no!bang ! ? set bang ? ! echo no!bang !echo nobang nobang ! ? ! ! !echo nobang nobang ! ? ! echo no!bang !echo noecho nobangbang noecho nobangbang ! ? In short: they all do it differently, and they all do it "wrong". - Apple mail simply does not act unless "bang" is set: bangexp() simply returns. - Free-, Net- and OpenBSD do not look for a "bang" variable at all, they simply do bang-style expansion whether that is set or not. - V10 mailx (and the code i maintain) do some mix (expand the "last bang" only if "bang" is set, but know about \! escapes and such. POSIX now says / will say If the bang variable is set, each unescaped occurrence of '!' in command shall be replaced with the command executed by the previous ! command or =CB=9C! command escape. thus - All commands entered for ! and ~! shall be stored. - If "bang" is set, an unquoted ! shall be replaced by that storage. The code everywhere is more or less what Kurt A. Shoens wrote as commit ae3dba0da2068b0de91ef163ea95fe774196a501 Author: Kurt A. Schoens AuthorDate: 1980-10-09 02:48:47 -0800 Commit: Kurt A. Schoens CommitDate: 1980-10-09 02:48:47 -0800 Made shell escapes expand !'s to previous command SCCS-vsn: usr.bin/mail/cmd3.c 1.2 It will fail to properly "bang" a second time if some escaped question mark was present in the first round, while(*cp){ ... if (*cp =3D=3D '!') { BANGEXP ... continue; } ... if (*cp =3D=3D '\\' && cp[1] =3D=3D '!') { ... *cp2++ =3D '!'; cp +=3D 2; changed++; [fallthru] } ... *cp2++ =3D *cp++; and it stores and pushes through the second escape for "\!\!". I am thinking of dropping this entirely, as it can properly work only once if some escape took place, and i do not know how to fix this the right way. (And the line editor has history.) --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt) |~~ |..and in spring, hear David Leonard sing.. | |The black bear, The black bear, |blithely holds his own holds himself at leisure |beating it, up and down tossing over his ups and downs with pleasure |~~ |Farewell, dear collar bear From nobody Tue May 2 22:24:41 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9vlK45xBz48FmS for ; Tue, 2 May 2023 22:24:45 +0000 (UTC) (envelope-from steffen@sdaoden.eu) Received: from sdaoden.eu (sdaoden.eu [217.144.132.164]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9vlJ46Vzz3KG8 for ; Tue, 2 May 2023 22:24:44 +0000 (UTC) (envelope-from steffen@sdaoden.eu) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of steffen@sdaoden.eu designates 217.144.132.164 as permitted sender) smtp.mailfrom=steffen@sdaoden.eu; dmarc=none Date: Wed, 03 May 2023 00:24:41 +0200 Author: Steffen Nurpmeso From: Steffen Nurpmeso To: tech@openbsd.org, tech-userlevel@netbsd.org, freebsd-hackers@freebsd.org Subject: Re: usr.bin/mail: cmd3.c:bangexp(): "borked"?, and BSD fails POSIX compat Message-ID: <20230502222441.fUEOs%steffen@sdaoden.eu> In-Reply-To: <20230502214014._zIz6%steffen@sdaoden.eu> References: <20230502214014._zIz6%steffen@sdaoden.eu> Mail-Followup-To: tech@openbsd.org, tech-userlevel@netbsd.org, freebsd-hackers@freebsd.org User-Agent: s-nail v14.9.24-461-g2a5db04708 OpenPGP: id=EE19E1C1F2F7054F8D3954D8308964B51883A0DD; url=https://ftp.sdaoden.eu/steffen.asc; preference=signencrypt BlahBlahBlah: Any stupid boy can crush a beetle. But all the professors in the world can make no bugs. List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [0.31 / 15.00]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_SPAM_MEDIUM(0.71)[0.714]; R_SPF_ALLOW(-0.20)[+a]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_SHORT(-0.10)[-0.099]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_ZERO(0.00)[0]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[sdaoden.eu]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:15987, ipnet:217.144.128.0/20, country:DE]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_NONE(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[] X-Rspamd-Queue-Id: 4Q9vlJ46Vzz3KG8 X-Spamd-Bar: / X-ThisMailContainsUnwantedMimeParts: N Steffen Nurpmeso wrote in <20230502214014._zIz6%steffen@sdaoden.eu>: |Hallo, and sorry for the cross-post, but so all in one (maybe) go. | |This is about a niche "feature" of mail, the shell command "bang" |(! / ~! in compose mode): ... |POSIX now says / will say | | If the bang variable is set, each unescaped occurrence of '!' in | command shall be replaced with the command executed by the | previous ! command or =CB=9C! command escape. | |thus | | - All commands entered for ! and ~! shall be stored. | | - If "bang" is set, an unquoted ! shall be replaced by that | storage. ... |It will fail to properly "bang" a second time[.] Eh, forget this please. Of course not, one simply inserts the old string without expansion. Since vim(1) offers the same "bang"ing, and merges \! to ! (source comment speaks of vi-compat) i keep it. (It actually requires '\!' or \\! ro avoid its expansion in ":echo BLA".) Ciao. static struct str last_bang; struct n_string xbang, *bang; char c; char const *cp_orig; boole bangit, changed; NYD_IN; bangit =3D ok_blook(bang); changed =3D FAL0; cp_orig =3D cp; for(bang =3D n_string_creat(&xbang); (c =3D *cp++) !=3D '\0';){ if(bangit && c =3D=3D '!'){ changed =3D TRU1; if(last_bang.l > 0) bang =3D n_string_push_buf(bang, last_bang.= s, last_bang.l); }else{ if(c =3D=3D '\\' && *cp =3D=3D '!'){ changed =3D TRU1; ++cp; c =3D '!'; } bang =3D n_string_push_c(bang, c); } } if(last_bang.s !=3D NIL) su_FREE(last_bang.s); last_bang.s =3D n_string_cp(bang); last_bang.l =3D bang->s_len; bang =3D n_string_drop_ownership(bang); n_string_gut(bang); if(bangit){ cp =3D last_bang.s; if(changed && (n_psonce & n_PSO_INTERACTIVE)) fprintf(n_stdout, "!%s\n", cp); }else cp =3D cp_orig; NYD_OU; return cp; --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt) |~~ |..and in spring, hear David Leonard sing.. | |The black bear, The black bear, |blithely holds his own holds himself at leisure |beating it, up and down tossing over his ups and downs with pleasure |~~ |Farewell, dear collar bear From nobody Tue May 2 23:41:50 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9xSV1fGbz48KLH for ; Tue, 2 May 2023 23:42:02 +0000 (UTC) (envelope-from junchoon@dec.sakura.ne.jp) Received: from www121.sakura.ne.jp (www121.sakura.ne.jp [153.125.133.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9xSS4FgNz3Qkd for ; Tue, 2 May 2023 23:42:00 +0000 (UTC) (envelope-from junchoon@dec.sakura.ne.jp) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of junchoon@dec.sakura.ne.jp has no SPF policy when checking 153.125.133.21) smtp.mailfrom=junchoon@dec.sakura.ne.jp; dmarc=none Received: from kalamity.joker.local (123-1-88-210.area1b.commufa.jp [123.1.88.210]) (authenticated bits=0) by www121.sakura.ne.jp (8.16.1/8.16.1/[SAKURA-WEB]/20201212) with ESMTPA id 342Nfodc025093 for ; Wed, 3 May 2023 08:41:50 +0900 (JST) (envelope-from junchoon@dec.sakura.ne.jp) Date: Wed, 3 May 2023 08:41:50 +0900 From: Tomoaki AOKI To: freebsd-hackers@freebsd.org Subject: Re: Periodic rant about SCHED_ULE Message-Id: <20230503084150.061d508f44fc1d79b18f0110@dec.sakura.ne.jp> In-Reply-To: References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> <20230331215751.166a294f7382c85b545f53a2@dec.sakura.ne.jp> Organization: Junchoon corps X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.33; amd64-portbld-freebsd13.2) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [1.41 / 15.00]; URI_HIDDEN_PATH(1.00)[https://people.freebsd.org/~mjg/.junk/ule-poc-hacks-dont-use.diff]; AUTH_NA(1.00)[]; NEURAL_HAM_SHORT(-0.95)[-0.950]; NEURAL_HAM_MEDIUM(-0.84)[-0.841]; NEURAL_SPAM_LONG(0.80)[0.799]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; R_SPF_NA(0.00)[no SPF record]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[sakura.ne.jp]; ASN(0.00)[asn:7684, ipnet:153.125.128.0/18, country:JP]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_NONE(0.00)[]; HAS_ORG_HEADER(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4Q9xSS4FgNz3Qkd X-Spamd-Bar: + X-ThisMailContainsUnwantedMimeParts: N On Mon, 1 May 2023 03:33:18 +0200 Mateusz Guzik wrote: > On 5/1/23, Mateusz Guzik wrote: > > On 3/31/23, Tomoaki AOKI wrote: > >> On Mon, 27 Mar 2023 16:47:04 +0200 > >> Mateusz Guzik wrote: > >> > >>> On 3/27/23, Mateusz Guzik wrote: > >>> > On 3/25/23, Mateusz Guzik wrote: > >>> >> On 3/23/23, Mateusz Guzik wrote: > >>> >>> On 3/22/23, Mateusz Guzik wrote: > >>> >>>> On 3/22/23, Steve Kargl wrote: > >>> >>>>> On Wed, Mar 22, 2023 at 07:31:57PM +0100, Matthias Andree wrote: > >>> >>>>>> > >> > >> (snip) > >> > >>> > > >>> > I repeat the setup: 8 cores, 8 processes doing cpu-bound stuff while > >>> > niced to 20 vs make -j buildkernel > >>> > > >>> > I had a little more look here, slapped in some hacks as a POC and got > >>> > an improvement from 67 minutes above to 21. > >>> > > >>> > Hacks are: > >>> > 1. limit hog timeslice to 1 tick so that is more eager to bail > >>> > 2. always preempt if pri < cpri > >>> > > >>> > So far I can confidently state the general problem: ULE penalizes > >>> > non-cpu hogs for blocking (even if it is not their fault, so to speak) > >>> > and that bumps their prio past preemption threshold, at which point > >>> > they can't preempt said hogs (despite hogs having a higher priority). > >>> > At the same time hogs use their full time slices, while non-hogs get > >>> > off cpu very early and have to wait a long time to get back on, at > >>> > least in part due to inability to preempt said hogs. > >>> > > >>> > As I mentioned elsewhere in the thread, interactivity scoring takes > >>> > "voluntary off cpu time" into account. As literally anything but > >>> > getting preempted counts as "voluntary sleep", workers get shafted for > >>> > going off cpu while waiting on locks in the kernel. > >>> > > >>> > If I/O needs to happen and the thread waits for the result, it most > >>> > likely does it early in its timeslice and once it's all ready it waits > >>> > for background hogs to get off cpu -- it can't preempt them. > >>> > > >>> > All that said: > >>> > 1. "interactivity scoring" (see sched_interact_score) > >>> > > >>> > I don't know if it makes any sense to begin with. Even if it does, it > >>> > counts stuff it should not by not differentiating between deliberately > >>> > going off cpu (e.g., actual sleep) vs just waiting for a file being > >>> > read. Imagine firefox reading a file from disk and being considered > >>> > less interactive for it. > >>> > > >>> > I don't have a solution for this problem. I *suspect* the way to go > >>> > would be to explicitly mark xorg/wayland/whatever as "interactive" and > >>> > have it inherited by its offspring. At the same time it should not > >>> > follow to stuff spawned in terminals. Not claiming this is perfect, > >>> > but it does eliminate the guessing game. > >>> > > >>> > Even so, 4BSD does not have any mechanism of the sort and reportedly > >>> > remains usable on a desktop just by providing some degree of fairness. > >>> > > >>> > Given that, I suspect the short term solution would whack said scoring > >>> > altogether and focus on fairness (see below). > >>> > > >>> > 2. fairness > >>> > > >>> > As explained above doing any offcpu-time inducing work instantly > >>> > shafts threads versus cpu hogs, even if said hogs are niced way above > >>> > them. > >>> > > >>> > Here I *suspect* position to add in the runqueue should be related to > >>> > how much slice was left when the thread went off cpu, while making > >>> > sure that hogs get to run eventually. Not that I have a nice way of > >>> > implementing this -- maybe a separate queue for known hogs and picking > >>> > them every n turns or similar. > >>> > > >>> > >>> Aight, now that I had a sober look at the code I think I cracked the > >>> case. > >>> > >>> The runq mechanism used by both 4BSD and ULE provides 64(!) queues, > >>> where the priority is divided by said number and that's how you know > >>> in which queue to land the thread. > >>> > >>> When deciding what to run, 4BSD uses runq_choose which iterates all > >>> queues from the beginning. This means threads of lower priority keep > >>> executing before the rest. In particular cpu hog lands with a high > >>> priority, looking worse than make -j 8 buildkernel and only running > >>> when there is nothing else ready to get the cpu. While this may sound > >>> decent, it is bad -- in principle a steady stream of lower priority > >>> threads can starve the hogs indefinitely. > >>> > >>> The problem was recognized when writing ULE, but improperly fixed -- > >>> it ends up distributing all threads within given priority range across > >>> the queues and then performing a lookup in a given queue. Here the > >>> problem is that while technically everyone does get a chance to run, > >>> the threads not using full slices are hosed for the time period as > >>> they wait for the hog *a lot*. > >>> > >>> A hack patch to induce the bogus-but-better 4BSD behavior of draining > >>> all runqs before running higher prio threads drops down build time to > >>> ~9 minutes, which is shorter than 4BSD. > >>> > >>> However, the right fix would achieve that *without* introducing > >>> starvation potential. > >>> > >>> I also note the runqs are a massive waste of memory and computing > >>> power. I'm going to have to sleep on what to do here. > >>> > >>> For interested here is the hackery: > >>> https://people.freebsd.org/~mjg/.junk/ule-poc-hacks-dont-use.diff > >>> > >>> sysctl kern.sched.slice_nice=0 > >>> sysctl kern.sched.preempt_thresh=400 # arbitrary number higher than any > >>> prio > >>> > >>> -- > >>> Mateusz Guzik > >> > >> Thanks for the patch. > >> Applied on top of main, amd64 at commit > >> 9d33a9d96f5a2cd88d0955b5b56ef5058b1706c1, setup 2 sysctls as you > >> mentioned and tested as below > >> > >> *Play flac files by multimedia/audacious via audio/virtual_oss > >> *Running www/firefox (not touched while testing) > >> *Forcibly build lang/rust > >> *Play games/aisleriot > >> > >> at the same time. > >> games/aisleriot runs slower than the situation lang/rust is not in > >> build, but didn't "freeze" and audacious normally played next music on > >> playlist, even on lang/rust is building codes written in rust. > >> > >> This is GREAT advance! > >> Without the patch, compiling rust codes eats up almost 100% of ALL > >> cores, and games/aisleriot often FREEZES SEVERAL MINUTES, and > >> multimedia/audacious needs to wait for, at worst, next music for FEW > >> MINUTES. (Once playback starts, the music is played normally until it > >> ends.) > >> > >> But unfortunately, the patch cannot be applied to stable/13, as some > >> prerequisite commits are not MFC'ed. > >> Missing commits are at least as below. There should be more, as I > >> gave up further tracking and haven't actually merged them to test. > >> > >> commit 954cffe95de1b9d70ed804daa45b7921f0f5c9da [1] > >> ule: Simplistic time-sharing for interrupt threads. > >> > >> commit fea89a2804ad89f5342268a8546a3f9b515b5e6c [2] > >> Add sched_ithread_prio to set the base priority of an interrupt > >> thread. > >> > >> commit 85b46073242d4666e1c9037d52220422449f9584 [3] > >> Deduplicate bus_dma bounce code. > >> > >> > >> [1] > >> https://cgit.freebsd.org/src/commit/?id=954cffe95de1b9d70ed804daa45b7921f0f5c9da > >> > >> [2] > >> https://cgit.freebsd.org/src/commit/?id=fea89a2804ad89f5342268a8546a3f9b515b5e6c > >> > >> [3] > >> https://cgit.freebsd.org/src/commit/?id=85b46073242d4666e1c9037d52220422449f9584 > >> > >> -- > >> Tomoaki AOKI > >> > >> > > > > Hello everyone. > > > > I sorted out a patch I consider comittable for the time being. IT IS > > NOT A PANACEA by any means, but it does sort out the most acute > > problem and should be a win for most people. It also comes with a knob > > to turn it off. > > > > That said, can you test this please: > > https://people.freebsd.org/~mjg/ule_pickshort.diff > > > > works against fresh main. if you are worried about recent zfs woes, > > just make sure you don't zpool upgrade and will be fine. > > > > Here is an updated patch: > https://people.freebsd.org/~mjg/ule_pickshortv2.diff > > if you are getting bad results, do: > sysctl kern.sched.preempt_bottom=0 > > and try again. > > thank you. > > -- > Mateusz Guzik Tried just a bit, and turned out my workload requires kern.sched.preempt_bottom=0. Tested with previous patch (ule-poc-hacks-dont-use.diff) backed out. At commit d713e0891ff9ab8246245c3206851d486ecfdd37, amd64. What I did for tests: While building lang/rust, in massive rustc phase, *Play FLAC music files using multimedia/audacious *Play youtube movies in www/firefox *edit some junk text on editors/leafpad multimedia/audacious plays via audio/virtual_oss. www/firefox plays sounds via audio/pulseaudio (Backed with audio/virtual_oss.) What happened: Without kern.sched.preempt_bottom=0 (was 135), all sounds are chopping unless any keytypes or mouse actions are done. Text editing was mostly smooth, but always slow. These are regardless kern.sched.preempt_thresh settings. With kern.sched.preempt_bottom=0, *Playing FLAC music files using multimedia/audacious is fine. *Playing youtube movies in www/firefox depends on kern.sched.preempt_thresh setting. The larger the value, the smoother the playback is. Tested with values 21, 80, 121, 224, 400. Note that 224 and 400 looked almost the same with my eyeballs. *Editing texts is faster, but just sometimes cursor movements and editing was not smooth (randomly). Sorry, as I'm basically working on stable/13, I cannot take enough time for testing on main. Thanks in advance! -- Tomoaki AOKI From nobody Thu May 4 09:41:24 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QBpjk0GcMz49tf0 for ; Thu, 4 May 2023 09:41:30 +0000 (UTC) (envelope-from void@f-m.fm) Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4QBpjj10Xqz472t for ; Thu, 4 May 2023 09:41:29 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm3 header.b=TgaV3Tcu; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=ZUU19smE; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 64.147.123.21 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.west.internal (Postfix) with ESMTP id 6AEE6320093B; Thu, 4 May 2023 05:41:27 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Thu, 04 May 2023 05:41:27 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1683193287; x=1683279687; bh=uW VNcvSMVcfSEeAALU1Ig8p5DoIpsZ/Rop2v5aKDsXI=; b=TgaV3TcuKPPVmPUfdB EobdsjSYmtlNAy9Q6ZvJhhUx5byU8uTyAOLHKJBqX+KTK3yA6Ly9LhjxRuyqz8rc q29KiSn3CtyWOa/3rMYYzUuUCTg8T6d+1gcByWu+PdkOta/5oZyCcUSv/5sM68wa V9x2hDhdVSskbxtQgf7KevGQTeC2y/2tpmR6cXMAfdwxBNWJ1RFFSdfFQFwPdjcX q3xISk2IJ4M+GlSj+HH14jOnMOfMsBbYQSmQuAkwE27R1KsjCURfLSWuSPAhK+2K Tcf4FA2MGc4oofJetEhEtOBz6gO93fakmY36sFiX37dncC6oyhMmHktincMQxemg UhDA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1683193287; x=1683279687; bh=uWVNcvSMVcfSE eAALU1Ig8p5DoIpsZ/Rop2v5aKDsXI=; b=ZUU19smEePIAOcxEfetnba+z5qdnh bkejBH34vPpKvjW7jV0En1QgMdXFnYLOm+eGbciFU1HBlgU+tqhUC0dUqCVp0pDi awhzRoZrheMYBI3+J1mM+Cj65sTgCz8qD3m9hjDlsmijXF7ZHlra25xtGSvXxFCK saSMNqJuojWHZAc28J6nS3a+b+6liCjam3KZ5K7kwC5z9yUDt2Ku4W+508VXr91y SQbeMjraB2WB20XnpbW2pFQC+bhOCb7hKvJ+IVvyZqlRDxhGwTox+2jSVsDZbdxC EvFQxBMujs7lvtbZVnSUZJrmojTT8nhqT6Cj2TwUPRBbM4VIyx/4C7XHg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrfeeftddgudekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne goufhushhpvggtthffohhmrghinhculdegledmnecujfgurhepfffhvfevuffkfhggtggu jgesthdtredttddtvdenucfhrhhomhepvhhoihguuceovhhoihgusehfqdhmrdhfmheqne cuggftrfgrthhtvghrnhepheethfffveejudfghefhgfethfevhedvhfeltdegtdejffel ffevudeggfffhedunecuffhomhgrihhnpehgohhoghhlvgdrtghomhenucevlhhushhtvg hrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehvohhiugesfhdqmhdrfhhm X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 4 May 2023 05:41:26 -0400 (EDT) Date: Thu, 4 May 2023 10:41:24 +0100 From: void To: freebsd-hackers@freebsd.org Cc: Matthew Grooms Subject: Re: BHYVE_SNAPSHOT Message-ID: Mail-Followup-To: freebsd-hackers@freebsd.org, Matthew Grooms References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: X-Spamd-Result: default: False [-2.54 / 15.00]; SUBJ_ALL_CAPS(1.05)[14]; NEURAL_HAM_LONG(-1.00)[-1.000]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm3,messagingengine.com:s=fm3]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.21]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.21:from]; MIME_GOOD(-0.10)[text/plain]; NEURAL_SPAM_SHORT(0.01)[0.014]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_COUNT_THREE(0.00)[4]; FREEMAIL_FROM(0.00)[f-m.fm]; TO_DN_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; MIME_TRACE(0.00)[0:+]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4QBpjj10Xqz472t X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N On Sun, Apr 30, 2023 at 04:41:36PM -0500, Matthew Grooms wrote: > >The case is quite plain. I'm not sure what the solution is to this >problem. I'd love to hear feedback from the community about how I've got >this completely wrong and how the course could be corrected. That would >be something. > >-Matthew Hi Matthew, Have you tried participating in bhyve weekly calls? AIUI alternates between 'prodiction users' and 'developers', might be relevant? https://docs.google.com/document/d/1PFUmz6XpTVAGkq5dBe8uaBFV2Y4i-uR88AuiCLIRxIQ/edit Apologies if you already know about this. -- From nobody Thu May 4 11:23:30 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QBrzg4vnTz490xS for ; Thu, 4 May 2023 11:23:43 +0000 (UTC) (envelope-from freebsd@igalic.co) Received: from mail-4323.proton.ch (mail-4323.proton.ch [185.70.43.23]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QBrzd5nk4z4LJY for ; Thu, 4 May 2023 11:23:41 +0000 (UTC) (envelope-from freebsd@igalic.co) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=igalic.co header.s=protonmail header.b=HxyI+ZaJ; spf=pass (mx1.freebsd.org: domain of freebsd@igalic.co designates 185.70.43.23 as permitted sender) smtp.mailfrom=freebsd@igalic.co; dmarc=none Date: Thu, 04 May 2023 11:23:30 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=igalic.co; s=protonmail; t=1683199419; x=1683458619; bh=MId4YCko2ob+GRjl7J2QjglUKRigf8415DylF8GUzoM=; h=Date:To:From:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=HxyI+ZaJWom6OH5no3avFhfa+pLwrXpNEUQCZsaEYwgrxJIAAcNOA4SRKy3Vbp5v8 otKK1VdyxbLOmLAOMdnOcnS1ytFeFQoFMgoqRJi8SMNt48ALGkVGYFGNHTKY0h3QZT krOTuuU6MDjkwhO3FQLB8jFQJfXJXGKK/pfzhZSaoHnrle6w6UO/+9lF69KFK3N2RZ 0uE7xmwTUQHJQUtHvb12EmxigbyJlJLopxlvMHqrhp+qSmKuLOvGbxMIYrpy/Xv5Gt MPJN1AWlAhMW1Dv0OteYieRjQzM4xbZIyjn2tEOIi0HbJF5zi9aMWL88n4iQJxlFXL nVsQ/285IuWTA== To: freebsd-hackers From: =?utf-8?Q?Mina_Gali=C4=87?= Subject: Netlink support in Go Message-ID: In-Reply-To: <4NBmuJiJqEmbbaw1sdfXZ8mTbePxfSWuqEGloGj8iWlhNu8xe7C11MVuYQrsQ7w3U4LG7SlqThCl9IMUZNhv9N_qBlzMgSvrKAgg2zaUd4c=@igalic.co> References: <4NBmuJiJqEmbbaw1sdfXZ8mTbePxfSWuqEGloGj8iWlhNu8xe7C11MVuYQrsQ7w3U4LG7SlqThCl9IMUZNhv9N_qBlzMgSvrKAgg2zaUd4c=@igalic.co> Feedback-ID: 66573723:user:proton List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-2.66 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.996]; R_MIXED_CHARSET(0.83)[subject]; R_DKIM_ALLOW(-0.20)[igalic.co:s=protonmail]; R_SPF_ALLOW(-0.20)[+ip4:185.70.43.0/24]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_ZERO(0.00)[0]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:62371, ipnet:185.70.43.0/24, country:CH]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[igalic.co:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[igalic.co]; RWL_MAILSPIKE_POSSIBLE(0.00)[185.70.43.23:from] X-Rspamd-Queue-Id: 4QBrzd5nk4z4LJY X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N Hey folks, I'm trying to get LXD agent running for FreeBSD, and it's missing a few thi= ngs: https://github.com/lxc/lxd/issues/11603 Now, while the LXD folks are working on removing the necessity for such low= -level access=E2=80=A6 I was wondering if anyone of ye who has worked on Netlink would be interest= ed in extending Go's netlink support to FreeBSD. Kind regards, Mina Gali=C4=87 Try PkgBase: https://alpha.pkgbase.live/ (once i get a hardware sponsor again) From nobody Thu May 4 12:22:20 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QBtHX2Hq9z493jn for ; Thu, 4 May 2023 12:22:32 +0000 (UTC) (envelope-from melifaro@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QBtHX1SjVz4Sv7; Thu, 4 May 2023 12:22:32 +0000 (UTC) (envelope-from melifaro@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1683202952; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yCD1wMv684CFKqaQentR+tKGfsvFEBeiKnU2WgJmWwg=; b=i+TH1Tja4rv2qWlLO3V5VIDJovqcvzUV9EayZYNcPWS+wpbegPS6lZ1fidsZUb650FmV2W W/IK3KgN4lTCJFDaZAAbR9+c/K8WcamgSd1Llv3Q4SntTSxFNo2bih1T79AD4G9PWDDNcR OkuLnDBdFAxcHn/R+zpdUa6kB363tqjUpl81p/hMULhSJMGK5GEUCfKKJqdBoxA2ovAcGf hQ3PMxBsOSMzdeELdRETHaU8ANDxmEzwDGemCWI4LsbqyyqIaJMNheHEBGg8eNzvbCNd9H rIQJ+1ZJ5xC/+8qjPg82dBnUHUmlRbkbz3H+CxTM+K2e3Gfl3F0faSTTesmpSA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1683202952; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yCD1wMv684CFKqaQentR+tKGfsvFEBeiKnU2WgJmWwg=; b=OlEvuYQcATwZn8i0rhJjIkQimM4XbfwY6a1/F33OKxlmq5lirWALWZAkPuR5vZ6s+auRSq YhlxbVJyIyFMFkxHHjHe4rxDTDdSIg9AIL/GaDj97aSy8eZ3X9+29wwdraD02/HS0q8XCo l+TZak3Fu7GaUKFWEvJ/J6k0INbKs7nA66LomOjtbrx7mJ2cwFWYAXOBJyx8fZT1LPXZoe V5w4dK57uoeqGqGQ4m76HJNd+4r/TPLQNa5tRzzN9XbZatrYOst/E0PvcQcrck4ym5AzGo OhjVuv+hAVhFmXvtiWlYn4wz64to+rZ29CAUSythtqTNqgXcCZ2yqOKU53rUtg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1683202952; a=rsa-sha256; cv=none; b=tUyA2r4D16D3n2jCT3jk3pFMrNCH7IZKg/IVfZjTgZ8DJq8lBMzqYVa+YvOcmaxbp41Bpn NFrqk5PwvFIBZx5ZBgbsidqRFZgtn3uNWCsjXziplNZPkc90LLxvqIlZlq/c80Fxggfip3 Pfal8aYpS0e2biMnTopQrnH9y7UuRj8b1YgydXlvTXsuJikPFxti18bI69syRxE4R+HqzU w+hhzf369Ev3n1W7VHOrWde6peKCbkv+Tbda2+TPnKd4ev6k4h4wIwgZkS+g/tep1nFFgC Uz0hUu+DbJVa3UJAF3t8ODGqhJ5RzrYj59A+Ua24J4fcKrv9H/43rn0Qe2d0vw== Received: from smtpclient.apple (unknown [IPv6:2a02:8084:d6bb:510:616f:68ba:2038:32cf]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: melifaro/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4QBtHW5Lnmz19tr; Thu, 4 May 2023 12:22:31 +0000 (UTC) (envelope-from melifaro@freebsd.org) Content-Type: text/plain; charset=utf-8 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Netlink support in Go From: Alexander Chernikov In-Reply-To: Date: Thu, 4 May 2023 13:22:20 +0100 Cc: freebsd-hackers Content-Transfer-Encoding: quoted-printable Message-Id: <4318A2DD-3DEF-4F6F-A9CB-E827752DBF91@FreeBSD.org> References: <4NBmuJiJqEmbbaw1sdfXZ8mTbePxfSWuqEGloGj8iWlhNu8xe7C11MVuYQrsQ7w3U4LG7SlqThCl9IMUZNhv9N_qBlzMgSvrKAgg2zaUd4c=@igalic.co> To: =?utf-8?Q?Mina_Gali=C4=87?= X-Mailer: Apple Mail (2.3731.400.51.1.1) X-ThisMailContainsUnwantedMimeParts: N > On 4 May 2023, at 12:23, Mina Gali=C4=87 wrote: >=20 > Hey folks, >=20 > I'm trying to get LXD agent running for FreeBSD, and it's missing a = few things: >=20 > https://github.com/lxc/lxd/issues/11603 >=20 > Now, while the LXD folks are working on removing the necessity for = such low-level access=E2=80=A6 > I was wondering if anyone of ye who has worked on Netlink would be = interested in extending Go's netlink support to FreeBSD. Few more datapoints here: all user-visible defines in FreeBSD Netlink = implementation matches their counterparts in Linux, so (hopefully) = extending the support is about enabling libgo have those defines from = the FreeBSD headers. The headers layout (and naming) is different from Linux, so that=E2=80=99s= probably the main point where effort is required. >=20 >=20 > Kind regards, >=20 > Mina Gali=C4=87 >=20 > Try PkgBase: https://alpha.pkgbase.live/ > (once i get a hardware sponsor again) >=20 >=20 >=20 From nobody Thu May 4 14:37:48 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QBxHr38Bdz49BHh for ; Thu, 4 May 2023 14:38:00 +0000 (UTC) (envelope-from jo@bruelltuete.com) Received: from email.jo-t.de (seppel.jo-t.de [45.132.244.126]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4QBxHp4wZYz3HsQ for ; Thu, 4 May 2023 14:37:58 +0000 (UTC) (envelope-from jo@bruelltuete.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bruelltuete.com header.s=bruelltuete18a header.b=G0Ly1eBb; spf=pass (mx1.freebsd.org: domain of jo@bruelltuete.com designates 45.132.244.126 as permitted sender) smtp.mailfrom=jo@bruelltuete.com; dmarc=pass (policy=none) header.from=bruelltuete.com DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=bruelltuete.com; s=bruelltuete18a; t=1683211071; bh=0QceyAFx29yVZIk1czKySISYP/io/U0cM7JYe962xBw=; h=Message-ID:Date:MIME-Version:To:From:Subject:From; b=G0Ly1eBb7YwayEaEiF2YBSqeoHjyazi0iClCCqOVj8ar8OZqKcnSmWUnpQVce5u8l o9W9v7Wxxoqm6kjFyxTM8qvRLFPRA1XQhKFBIr6j0O+fJkS7f74J9RaVXKbJ6IKLrM B5tNhTxM9KRQx/bROj6a3zq72g2c9PYQlZGmlAkGjbilQ6gPPPvxJ8vuFYUBqdn0kZ Ec/yWXnbH9unVN8CYifu709Yb+obr/aIJPEGrelk8p3Jd0Z8b95havpMP92UJjyKN8 aujGXWbOJLol3sHOXT+iFXOC4XG+mEZLbu09p3WkJRl3ueYop+joq72dr6aMVvc8qa yzi9PWc4Gijyw== Message-ID: <256af1c8-7272-746f-35ea-0b1f94c2a40c@bruelltuete.com> Date: Thu, 4 May 2023 15:37:48 +0100 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 X-Mozilla-News-Host: news://news.gmane.io:119 Content-Language: en-GB To: freebsd-hackers@FreeBSD.org From: Johannes Totz Subject: Run scripts from rc.suspend Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; DMARC_POLICY_ALLOW(-0.50)[bruelltuete.com,none]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[bruelltuete.com:s=bruelltuete18a]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-hackers@FreeBSD.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_ZERO(0.00)[0]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:197540, ipnet:45.132.244.0/22, country:DE]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[bruelltuete.com:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4QBxHp4wZYz3HsQ X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N Hi everyone, /etc/rc.resume can run rc-scripts with the resume keyword. /etc/rc.suspend does not have a similar feature. Is there a good reason why not? Or is it just a case of nobody-asked-for-it-yet? cheers, Johannes From eugen@grosbein.net Thu May 4 15:59:16 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QBz5w3Z81z49Fmv for ; Thu, 4 May 2023 15:59:32 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QBz5v04wwz3kvt for ; Thu, 4 May 2023 15:59:30 +0000 (UTC) (envelope-from eugen@grosbein.net) Authentication-Results: mx1.freebsd.org; none Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.17.1/8.17.1) with ESMTPS id 344FxIBB054741 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 4 May 2023 15:59:19 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: jo@bruelltuete.com Received: from [10.58.0.11] (dadvw [10.58.0.11] (may be forged)) by eg.sd.rdtc.ru (8.16.1/8.16.1) with ESMTPS id 344FxIYq073430 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Thu, 4 May 2023 22:59:18 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: Run scripts from rc.suspend To: Johannes Totz , freebsd-hackers@FreeBSD.org References: <256af1c8-7272-746f-35ea-0b1f94c2a40c@bruelltuete.com> From: Eugene Grosbein Message-ID: <29e0a1a0-7807-a0f0-05b4-8e36385b8bc6@grosbein.net> Date: Thu, 4 May 2023 22:59:16 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 In-Reply-To: <256af1c8-7272-746f-35ea-0b1f94c2a40c@bruelltuete.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,SHORTCIRCUIT autolearn=disabled version=3.4.6 X-Spam-Report: * -0.0 SHORTCIRCUIT No description available. * -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on hz.grosbein.net X-Rspamd-Queue-Id: 4QBz5v04wwz3kvt X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/32, country:DE] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N 04.05.2023 21:37, Johannes Totz wrote: > Hi everyone, > > /etc/rc.resume can run rc-scripts with the resume keyword. > /etc/rc.suspend does not have a similar feature. > Is there a good reason why not? Or is it just a case of nobody-asked-for-it-yet? The latter. From nobody Thu May 4 18:12:21 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QC23G6PLkz49NQm for ; Thu, 4 May 2023 18:12:26 +0000 (UTC) (envelope-from jo@bruelltuete.com) Received: from email.jo-t.de (seppel.jo-t.de [45.132.244.126]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4QC23G4SLvz4DJy for ; Thu, 4 May 2023 18:12:26 +0000 (UTC) (envelope-from jo@bruelltuete.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=bruelltuete.com; s=bruelltuete18a; t=1683223945; bh=dS/Y+2v5MyBtFcjd9KcxAtZeGuSbJmT1w+VviKozcjU=; h=Message-ID:Date:MIME-Version:Subject:To:References:From:From; b=VU7p3kSEbTUts2SxQbPd3BXpnHeEKnebzMuUabaxp5vOMMUCigcRd5CvVcb7BkGNL Y3cFtRykgI1zgbtzpkFVkqMMLONkdGEQJgQQFxCaAnyQmbtRZAKEO0pCmMpSfcoHv5 AICDSbE4+8E66pGEGnXQiaJJup7Y8VmhyopfBk7VQgS/Yzi/ixbDzbn+bXX/ugEzqu a3za20TyLzHGoCeFiHyzN/GyF7a1zndKfgQJswqyPaiJjxvWOj+Ye7fpYiwIPc7eZa GbDnviOrFmDFRzXRmtHrmBbqxxQ3Gs0MO55JMkSPyRmebYUxrwJtptZ4ADkl8yTl+9 mougbPr40zj/g== Message-ID: <07e1f6fe-baf6-a19d-7365-db75c245341a@bruelltuete.com> Date: Thu, 4 May 2023 19:12:21 +0100 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Subject: Re: Run scripts from rc.suspend Content-Language: en-GB To: Eugene Grosbein , freebsd-hackers@FreeBSD.org References: <256af1c8-7272-746f-35ea-0b1f94c2a40c@bruelltuete.com> <29e0a1a0-7807-a0f0-05b4-8e36385b8bc6@grosbein.net> From: Johannes Totz In-Reply-To: <29e0a1a0-7807-a0f0-05b4-8e36385b8bc6@grosbein.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4QC23G4SLvz4DJy X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:197540, ipnet:45.132.244.0/22, country:DE] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On 04/05/2023 16:59, Eugene Grosbein wrote: > 04.05.2023 21:37, Johannes Totz wrote: >> Hi everyone, >> >> /etc/rc.resume can run rc-scripts with the resume keyword. >> /etc/rc.suspend does not have a similar feature. >> Is there a good reason why not? Or is it just a case of nobody-asked-for-it-yet? > > The latter. Quick hack: https://reviews.freebsd.org/D39965 Works as intended. I would use this as follows: I'm done for the day, press sleep button on my workstation. A new rc.suspend-script runs and sets a wake up time for the next morning. Then the next day the machine wakes up again, runs its cron jobs or whatever and is ready when I'm ready. From nobody Fri May 5 22:29:53 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QClsm6BrVz49tSt for ; Fri, 5 May 2023 22:36:44 +0000 (UTC) (envelope-from li-fbsd@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QClsl2Mkbz3n5b for ; Fri, 5 May 2023 22:36:42 +0000 (UTC) (envelope-from li-fbsd@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of li-fbsd@citylink.dinoex.sub.org designates 2a0b:f840::12 as permitted sender) smtp.mailfrom=li-fbsd@citylink.dinoex.sub.org; dmarc=none Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.17.1/8.17.1) with ESMTPS id 345Ma85S019900 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Sat, 6 May 2023 00:36:08 +0200 (CEST) (envelope-from li-fbsd@citylink.dinoex.sub.org) X-MDaemon-Deliver-To: Received: (from uucp@localhost) by uucp.dinoex.org (8.17.1/8.17.1/Submit) with UUCP id 345Ma8qd019899 for freebsd-hackers@freebsd.org; Sat, 6 May 2023 00:36:08 +0200 (CEST) (envelope-from li-fbsd@citylink.dinoex.sub.org) Received: from admn.intra.daemon.contact (localhost [127.0.0.1]) by admn.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 345MUNx7027353 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Sat, 6 May 2023 00:30:24 +0200 (CEST) (envelope-from li-fbsd@citylink.dinoex.sub.org) Received: from intra.daemon.contact (news@localhost) by admn.intra.daemon.contact (8.17.1/8.17.1/Submit) with NNTP id 345MTr9M027050 for freebsd-hackers@freebsd.org; Sat, 6 May 2023 00:29:53 +0200 (CEST) (envelope-from li-fbsd@citylink.dinoex.sub.org) X-Authentication-Warning: admn.intra.daemon.contact: news set sender to li-fbsd@citylink.dinoex.sub.org using -f From: "Peter 'PMc' Much" X-Newsgroups: m2n.fbsd.hackers Subject: Re: Run scripts from rc.suspend Date: Fri, 5 May 2023 22:29:53 -0000 (UTC) Message-ID: References: <256af1c8-7272-746f-35ea-0b1f94c2a40c@bruelltuete.com> Injection-Date: Fri, 5 May 2023 22:29:53 -0000 (UTC) Injection-Info: admn.intra.daemon.contact; logging-data="75328"; mail-complaints-to="usenet@citylink.dinoex.sub.org" User-Agent: slrn/1.0.3 (FreeBSD) To: freebsd-hackers@freebsd.org X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Sat, 06 May 2023 00:36:11 +0200 (CEST) X-Spamd-Result: default: False [-2.65 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.65)[-0.653]; FORGED_SENDER(0.30)[pmc@citylink.dinoex.sub.org,li-fbsd@citylink.dinoex.sub.org]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[sub.org]; R_DKIM_NA(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; RCPT_COUNT_ONE(0.00)[1]; HAS_XAW(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_DN_NONE(0.00)[]; FROM_NEQ_ENVFROM(0.00)[pmc@citylink.dinoex.sub.org,li-fbsd@citylink.dinoex.sub.org] X-Rspamd-Queue-Id: 4QClsl2Mkbz3n5b X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org On 2023-05-04, Johannes Totz wrote: > Hi everyone, > > /etc/rc.resume can run rc-scripts with the resume keyword. > /etc/rc.suspend does not have a similar feature. > Is there a good reason why not? Or is it just a case of > nobody-asked-for-it-yet? I found there is a rather short timeout, when the susppend script will not complete running child processes and give up (hand over to acpi). OTOH stopping processes may sometimes be fast and sometimes take a while - so the outcome is likely an unpredictable state in the suspend, and some people don't like that.