Date: Tue, 2 May 2023 14:00:31 +0300 From: Vitaliy Gusev <gusev.vitaliy@gmail.com> To: =?utf-8?Q?Corvin_K=C3=B6hne?= <corvink@FreeBSD.org> Cc: Matthew Grooms <mgrooms@shrew.net>, Rob Wing <rob.fx907@gmail.com>, freebsd-hackers@freebsd.org, freebsd-virtualization@freebsd.org, Elena Mihailescu <elenamihailescu22@gmail.com>, Mihai Carabas <mihai.carabas@gmail.com> Subject: Re: BHYVE_SNAPSHOT Message-ID: <ACFE251D-F4ED-4901-A692-618DC64E9B0E@gmail.com> In-Reply-To: <3ab4e6d94fb0153fb6ff4a53ac6f53b2eaae0cf7.camel@FreeBSD.org> References: <ZEz8tU_83QfqbbMu@int21h> <fe221c6a-acb7-ddbd-413d-7039de33e872@shrew.net> <CAF3%2Bn_fN4J4jXH89t8gMOD8QpqAike0Uzrb9wUfKoYv56zQt_w@mail.gmail.com> <89c84bd7-a925-02f4-acbe-12c3000e7007@shrew.net> <3ab4e6d94fb0153fb6ff4a53ac6f53b2eaae0cf7.camel@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_B895AB4D-5F14-4A57-B65C-D1DB37F70C9D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Just add some plans for me: 1. Describe snapshot file format: One file for snapshot. 2. Implement snapshot/resume via nvlist. nvlist implementation brings: Versioning Easy debugging, getting saved values, etc. Validate restored variables: types, sized, etc. Add optional variables without breaking backward compatibility (resume = can be performed with old snapshots) Remove variables without breaking backward compatibility Use one file for snapshot Improve restore command line: "bhyve -r $snapshot=E2=80=9D, i.e. w/o = additional arguments =E2=80=94=E2=80=94 Vitaliy Gusev >>> before extending the functionality off the top of it. >>=20 >> Yup. See above. I appreciate your input, but the goal of live >> migration was set in 2016 with a prototype first demonstrated in >> 2018. How long do you suggest a developer wait without review >> feedback before moving forward out of tree? >=20 > The snapshot feature isn't compiled in by default. So, it's likely = that > changes break it and only a few people are testing it. >=20 > We have to focus on getting this into the tree. >=20 >>>> There are experimental patches for all these features that were >>>> developed by students at UPB. In a lot of cases, there are open >>>> reviews that have been waiting on feedback for ages. >>>=20 >>> In general, most people don't want to review large experimental >>> patches. >>=20 >> Yup. That approach was attempted with the Warm Migration patches. >> =46rom slide 17 in Elena's presentation: >>=20 >> First review opened in 2021: https://reviews.freebsd.org/D28270 >> 5 reviews from 2022 starting with https://reviews.freebsd.org/D34717 >> (same feature split in multiple parts) >> =20 >> A similar request was made recently to Gusev Vitaliy WRT the >> multiple device support patch which he took ownership of. Thanks for >> adding feedback to that review BTW. We'll see how that pans out ... >>=20 >> https://reviews.freebsd.org/D35590 >>=20 >=20 > I've already reviewed Vitaliy's multi device support patch and people > had more than enough time to complain about it. I'm going to commit it > as soon as he splits his commit. > =20 >>>> The case is quite plain. I'm not sure what the solution is to >>>> this=20 >>>> problem. I'd love to hear feedback from the community about how >>>> I've got=20 >>>> this completely wrong and how the course could be corrected. >>>> That would=20 >>>> be something. >>>>=20 >>>=20 >>> My perspective is that it would have been better to focus student >>> efforts on completing the snapshot feature. By completing the >>> snapshot feature, I mean getting the code into a state where it's >>> compiled in by default and no longer considered an experimental >>> feature. >>>=20 >> I'm not sure what more to say hear regarding the snapshot feature or >> what might have been done in the past. We need a solution for the >> present. If you have any comments related to the follow up reviews >> submitted by UPB, I'm sure they'd love to hear them. >> And lastly: I get that FreeBSD is a non paid volunteer project for >> most. Without the efforts of folks like Peter, Neel, John and others, >> there would be no bhyve. I'm not saying that they, as project >> maintainers, should somehow be doing more. We all have limited time >> to invest, paid work to do and families to feed. I'm asking if there >> are other developers that might be willing and able to help with >> reviews? Is there something the FreeBSD foundation can do help out in >> situations like these? >> Thanks, >> -Matthew >> =20 >=20 > UPB has developed some interesting features and I'd like to see those > in tree. I can take some time to review the patches. Nevertheless, we > really need the snapshot feature compiled in by default. Otherwise, > it's wasted time for all of us. >=20 >=20 > --=20 > Kind regards, > Corvin --Apple-Mail=_B895AB4D-5F14-4A57-B65C-D1DB37F70C9D Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"content-type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"overflow-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;">Just add some = plans for me:<div><br></div><div> 1. Describe snapshot file format: = One file for snapshot.</div><div><br></div><div> 2. Implement = snapshot/resume via nvlist.</div><div><br></div><div> = <b>nvlist</b> implementation brings:</div><div><br></div><div><ul = class=3D"MailOutline"><li>Versioning</li><li>Easy debugging, getting = saved values, etc.</li><li>Validate restored variables: types, sized, = etc.</li><li>Add optional variables without breaking backward = compatibility (resume can be performed with old = snapshots)</li><li>Remove variables without breaking backward = compatibility</li><li>Use one file for snapshot</li><li>Improve restore = command line: "bhyve -r $snapshot=E2=80=9D, i.e. w/o additional = arguments</li></ul></div><div><br></div><div>=E2=80=94=E2=80=94</div><div>= Vitaliy Gusev<br><div><br><div><blockquote type=3D"cite"><div><blockquote = type=3D"cite" style=3D"font-family: Helvetica; font-size: 14px; = font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; orphans: auto; text-align: start; text-indent: = 0px; text-transform: none; white-space: normal; widows: auto; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><blockquote type=3D"cite">before extending the functionality off = the top of it.<br></blockquote><br>Yup. See above. I appreciate your = input, but the goal of live<br>migration was set in 2016 with a = prototype first demonstrated in<br>2018. How long do you suggest a = developer wait without review<br>feedback before moving forward out of = tree?<br></blockquote><br style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 14px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 14px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">The = snapshot feature isn't compiled in by default. So, it's likely = that</span><br style=3D"caret-color: rgb(0, 0, 0); font-family: = Helvetica; font-size: 14px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><span style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 14px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;">changes break it and = only a few people are testing it.</span><br style=3D"caret-color: rgb(0, = 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><br style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 14px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 14px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">We have = to focus on getting this into the tree.</span><br style=3D"caret-color: = rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: = normal; font-variant-caps: normal; font-weight: 400; letter-spacing: = normal; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><br style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 14px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><blockquote type=3D"cite" style=3D"font-family: = Helvetica; font-size: 14px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><blockquote type=3D"cite"><blockquote = type=3D"cite">There are experimental patches for all these features that = were<br>developed by students at UPB. In a lot of cases, there are = open<br>reviews that have been waiting on feedback for = ages.<br></blockquote><br>In general, most people don't want to review = large experimental<br>patches.<br></blockquote><br>Yup. That approach = was attempted with the Warm Migration patches.<br>=46rom slide 17 in = Elena's presentation:<br><br> First review opened in 2021: = https://reviews.freebsd.org/D28270<br> 5 reviews from 2022 starting = with https://reviews.freebsd.org/D34717<br>(same feature split in = multiple parts)<br> <br> A similar request was made recently = to Gusev Vitaliy WRT the<br>multiple device support patch which he took = ownership of. Thanks for<br>adding feedback to that review BTW. We'll = see how that pans out = ...<br><br> https://reviews.freebsd.org/D35590<br><br></blockquote><b= r style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><span = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;">I've already reviewed Vitaliy's multi = device support patch and people</span><br style=3D"caret-color: = rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: = normal; font-variant-caps: normal; font-weight: 400; letter-spacing: = normal; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 14px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">had = more than enough time to complain about it. I'm going to commit = it</span><br style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 14px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><span style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 14px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;">as soon as he splits his = commit.</span><br style=3D"caret-color: rgb(0, 0, 0); font-family: = Helvetica; font-size: 14px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><span style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 14px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;"> </span><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><blockquote = type=3D"cite" style=3D"font-family: Helvetica; font-size: 14px; = font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; orphans: auto; text-align: start; text-indent: = 0px; text-transform: none; white-space: normal; widows: auto; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><blockquote type=3D"cite"><blockquote type=3D"cite"> The = case is quite plain. I'm not sure what the solution is to<br>this<span = class=3D"Apple-converted-space"> </span><br> problem. I'd love = to hear feedback from the community about how<br>I've got<span = class=3D"Apple-converted-space"> </span><br> this completely = wrong and how the course could be corrected.<br>That would<span = class=3D"Apple-converted-space"> </span><br> be = something.<br><br></blockquote><br>My perspective is that it would have = been better to focus student<br>efforts on completing the snapshot = feature. By completing the<br>snapshot feature, I mean getting the code = into a state where it's<br>compiled in by default and no longer = considered an experimental<br>feature.<br><br></blockquote>I'm not sure = what more to say hear regarding the snapshot feature or<br>what might = have been done in the past. We need a solution for the<br>present. If = you have any comments related to the follow up reviews<br>submitted by = UPB, I'm sure they'd love to hear them.<br>And lastly: I get that = FreeBSD is a non paid volunteer project for<br>most. Without the efforts = of folks like Peter, Neel, John and others,<br>there would be no bhyve. = I'm not saying that they, as project<br>maintainers, should somehow be = doing more. We all have limited time<br>to invest, paid work to do and = families to feed. I'm asking if there<br>are other developers that might = be willing and able to help with<br>reviews? Is there something the = FreeBSD foundation can do help out in<br>situations like = these?<br>Thanks,<br>-Matthew<br> <br></blockquote><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><span = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;">UPB has developed some interesting features = and I'd like to see those</span><br style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 14px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 14px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">in = tree. I can take some time to review the patches. Nevertheless, = we</span><br style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 14px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><span style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 14px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;">really need the snapshot = feature compiled in by default. Otherwise,</span><br style=3D"caret-color:= rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: = normal; font-variant-caps: normal; font-weight: 400; letter-spacing: = normal; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 14px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">it's = wasted time for all of us.</span><br style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 14px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><br style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 14px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><br style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 14px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 14px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">--<span = class=3D"Apple-converted-space"> </span></span><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><span = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;">Kind regards,</span><br style=3D"caret-color:= rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: = normal; font-variant-caps: normal; font-weight: 400; letter-spacing: = normal; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 14px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline = !important;">Corvin</span></div></blockquote></div><br></div></div></body>= </html>= --Apple-Mail=_B895AB4D-5F14-4A57-B65C-D1DB37F70C9D--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?ACFE251D-F4ED-4901-A692-618DC64E9B0E>