Date: Tue, 2 May 2023 14:00:31 +0300 From: Vitaliy Gusev <gusev.vitaliy@gmail.com> To: =?utf-8?Q?Corvin_K=C3=B6hne?= <corvink@FreeBSD.org> Cc: Matthew Grooms <mgrooms@shrew.net>, Rob Wing <rob.fx907@gmail.com>, freebsd-hackers@freebsd.org, freebsd-virtualization@freebsd.org, Elena Mihailescu <elenamihailescu22@gmail.com>, Mihai Carabas <mihai.carabas@gmail.com> Subject: Re: BHYVE_SNAPSHOT Message-ID: <ACFE251D-F4ED-4901-A692-618DC64E9B0E@gmail.com> In-Reply-To: <3ab4e6d94fb0153fb6ff4a53ac6f53b2eaae0cf7.camel@FreeBSD.org> References: <ZEz8tU_83QfqbbMu@int21h> <fe221c6a-acb7-ddbd-413d-7039de33e872@shrew.net> <CAF3%2Bn_fN4J4jXH89t8gMOD8QpqAike0Uzrb9wUfKoYv56zQt_w@mail.gmail.com> <89c84bd7-a925-02f4-acbe-12c3000e7007@shrew.net> <3ab4e6d94fb0153fb6ff4a53ac6f53b2eaae0cf7.camel@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] Just add some plans for me: 1. Describe snapshot file format: One file for snapshot. 2. Implement snapshot/resume via nvlist. nvlist implementation brings: Versioning Easy debugging, getting saved values, etc. Validate restored variables: types, sized, etc. Add optional variables without breaking backward compatibility (resume can be performed with old snapshots) Remove variables without breaking backward compatibility Use one file for snapshot Improve restore command line: "bhyve -r $snapshot”, i.e. w/o additional arguments —— Vitaliy Gusev >>> before extending the functionality off the top of it. >> >> Yup. See above. I appreciate your input, but the goal of live >> migration was set in 2016 with a prototype first demonstrated in >> 2018. How long do you suggest a developer wait without review >> feedback before moving forward out of tree? > > The snapshot feature isn't compiled in by default. So, it's likely that > changes break it and only a few people are testing it. > > We have to focus on getting this into the tree. > >>>> There are experimental patches for all these features that were >>>> developed by students at UPB. In a lot of cases, there are open >>>> reviews that have been waiting on feedback for ages. >>> >>> In general, most people don't want to review large experimental >>> patches. >> >> Yup. That approach was attempted with the Warm Migration patches. >> From slide 17 in Elena's presentation: >> >> First review opened in 2021: https://reviews.freebsd.org/D28270 >> 5 reviews from 2022 starting with https://reviews.freebsd.org/D34717 >> (same feature split in multiple parts) >> >> A similar request was made recently to Gusev Vitaliy WRT the >> multiple device support patch which he took ownership of. Thanks for >> adding feedback to that review BTW. We'll see how that pans out ... >> >> https://reviews.freebsd.org/D35590 >> > > I've already reviewed Vitaliy's multi device support patch and people > had more than enough time to complain about it. I'm going to commit it > as soon as he splits his commit. > >>>> The case is quite plain. I'm not sure what the solution is to >>>> this >>>> problem. I'd love to hear feedback from the community about how >>>> I've got >>>> this completely wrong and how the course could be corrected. >>>> That would >>>> be something. >>>> >>> >>> My perspective is that it would have been better to focus student >>> efforts on completing the snapshot feature. By completing the >>> snapshot feature, I mean getting the code into a state where it's >>> compiled in by default and no longer considered an experimental >>> feature. >>> >> I'm not sure what more to say hear regarding the snapshot feature or >> what might have been done in the past. We need a solution for the >> present. If you have any comments related to the follow up reviews >> submitted by UPB, I'm sure they'd love to hear them. >> And lastly: I get that FreeBSD is a non paid volunteer project for >> most. Without the efforts of folks like Peter, Neel, John and others, >> there would be no bhyve. I'm not saying that they, as project >> maintainers, should somehow be doing more. We all have limited time >> to invest, paid work to do and families to feed. I'm asking if there >> are other developers that might be willing and able to help with >> reviews? Is there something the FreeBSD foundation can do help out in >> situations like these? >> Thanks, >> -Matthew >> > > UPB has developed some interesting features and I'd like to see those > in tree. I can take some time to review the patches. Nevertheless, we > really need the snapshot feature compiled in by default. Otherwise, > it's wasted time for all of us. > > > -- > Kind regards, > Corvin [-- Attachment #2 --] <html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">Just add some plans for me:<div><br></div><div> 1. Describe snapshot file format: One file for snapshot.</div><div><br></div><div> 2. Implement snapshot/resume via nvlist.</div><div><br></div><div> <b>nvlist</b> implementation brings:</div><div><br></div><div><ul class="MailOutline"><li>Versioning</li><li>Easy debugging, getting saved values, etc.</li><li>Validate restored variables: types, sized, etc.</li><li>Add optional variables without breaking backward compatibility (resume can be performed with old snapshots)</li><li>Remove variables without breaking backward compatibility</li><li>Use one file for snapshot</li><li>Improve restore command line: "bhyve -r $snapshot”, i.e. w/o additional arguments</li></ul></div><div><br></div><div>——</div><div>Vitaliy Gusev<br><div><br><div><blockquote type="cite"><div><blockquote type="cite" style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><blockquote type="cite">before extending the functionality off the top of it.<br></blockquote><br>Yup. See above. I appreciate your input, but the goal of live<br>migration was set in 2016 with a prototype first demonstrated in<br>2018. How long do you suggest a developer wait without review<br>feedback before moving forward out of tree?<br></blockquote><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">The snapshot feature isn't compiled in by default. So, it's likely that</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">changes break it and only a few people are testing it.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">We have to focus on getting this into the tree.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><blockquote type="cite" style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><blockquote type="cite"><blockquote type="cite">There are experimental patches for all these features that were<br>developed by students at UPB. In a lot of cases, there are open<br>reviews that have been waiting on feedback for ages.<br></blockquote><br>In general, most people don't want to review large experimental<br>patches.<br></blockquote><br>Yup. That approach was attempted with the Warm Migration patches.<br>From slide 17 in Elena's presentation:<br><br> First review opened in 2021: https://reviews.freebsd.org/D28270<br> 5 reviews from 2022 starting with https://reviews.freebsd.org/D34717<br>(same feature split in multiple parts)<br> <br> A similar request was made recently to Gusev Vitaliy WRT the<br>multiple device support patch which he took ownership of. Thanks for<br>adding feedback to that review BTW. We'll see how that pans out ...<br><br> https://reviews.freebsd.org/D35590<br><br></blockquote><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">I've already reviewed Vitaliy's multi device support patch and people</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">had more than enough time to complain about it. I'm going to commit it</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">as soon as he splits his commit.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;"> </span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><blockquote type="cite" style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><blockquote type="cite"><blockquote type="cite"> The case is quite plain. I'm not sure what the solution is to<br>this<span class="Apple-converted-space"> </span><br> problem. I'd love to hear feedback from the community about how<br>I've got<span class="Apple-converted-space"> </span><br> this completely wrong and how the course could be corrected.<br>That would<span class="Apple-converted-space"> </span><br> be something.<br><br></blockquote><br>My perspective is that it would have been better to focus student<br>efforts on completing the snapshot feature. By completing the<br>snapshot feature, I mean getting the code into a state where it's<br>compiled in by default and no longer considered an experimental<br>feature.<br><br></blockquote>I'm not sure what more to say hear regarding the snapshot feature or<br>what might have been done in the past. We need a solution for the<br>present. If you have any comments related to the follow up reviews<br>submitted by UPB, I'm sure they'd love to hear them.<br>And lastly: I get that FreeBSD is a non paid volunteer project for<br>most. Without the efforts of folks like Peter, Neel, John and others,<br>there would be no bhyve. I'm not saying that they, as project<br>maintainers, should somehow be doing more. We all have limited time<br>to invest, paid work to do and families to feed. I'm asking if there<br>are other developers that might be willing and able to help with<br>reviews? Is there something the FreeBSD foundation can do help out in<br>situations like these?<br>Thanks,<br>-Matthew<br> <br></blockquote><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">UPB has developed some interesting features and I'd like to see those</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">in tree. I can take some time to review the patches. Nevertheless, we</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">really need the snapshot feature compiled in by default. Otherwise,</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">it's wasted time for all of us.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">--<span class="Apple-converted-space"> </span></span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">Kind regards,</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">Corvin</span></div></blockquote></div><br></div></div></body></html>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?ACFE251D-F4ED-4901-A692-618DC64E9B0E>