Date: Thu, 25 May 2023 18:22:10 +0200 From: Mario Marietto <marietto2008@gmail.com> To: Vitaliy Gusev <gusev.vitaliy@gmail.com> Cc: Tomek CEDRO <tomek@cedro.info>, virtualization@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: BHYVE SNAPSHOT image format proposal Message-ID: <CA%2B1FSijKkCnR5j1N9BN0CbqzzitibK7r9OA84jc2eZ=uJaMW-g@mail.gmail.com> In-Reply-To: <8FE14143-1AA9-418E-A497-FEFB99BF6B9F@gmail.com> References: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> <CAFYkXjng1LWy5wVyTnSo0xrEWOy%2BOx9ZjLcmFqQs5EVpT8J_uA@mail.gmail.com> <AF34E648-2D8A-46C7-82A5-B88006BBB8F6@gmail.com> <CAFYkXjkUjh8gEMv4XZgb2QQW=qM1fhxMoMxRYuc4p6HbBXsDCw@mail.gmail.com> <8FE14143-1AA9-418E-A497-FEFB99BF6B9F@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] Vitaliy, what happens if I clone your repo as source code on my FreeBSD system. Can I test your code directly or not ? Anyway,I think that,before doing this,I need to follow some kind of tutorial,to understand how the workflow is. Otherwise I will be not able to test and stress it. On Thu, May 25, 2023 at 3:40 PM Vitaliy Gusev <gusev.vitaliy@gmail.com> wrote: > > > On 25 May 2023, at 04:30, Tomek CEDRO <tomek@cedro.info> wrote: > > On Wed, May 24, 2023 at 5:11 PM Vitaliy Gusev wrote: > > Protecting requires more efforts and it should be clearly defined: what is > purpose. If > purpose is having checksum with 99.9% reliability, NVLIST HEADER can be > widen > to have “checksum” key/value for a Section. > > > Well, this could be optional but useful to make sure snapshot did not > break somehow for instance backup medium error or something like > that.. even more maybe a way to fix it.. just a design stage idea :- > > > Yes, new format can have checksum of a Section data if implemented. > > > > If purpose is having crypto verification - I believe sha256 program should > be your choice. > > > My question was more specific to availability of that feature > (integrity + repair) rather than specific format :-) > > The use case here is having a virtual machine (it was VirtualBox) with > a bare os installed, plus some common applications, that is snapshoted > at some point in time, then experimented a lot, restored from > snapshot, etc. I had a backup of such vm + snapshot backed up that got > broken somehow. It would be nice to know that something is broken, > what is broken, maybe a way to fix :-) > > > > “Integrity" is a very broad term. What checksum algorithm is fine enough? > > For the instance, ZFS has several options for checksum: > > *checksum*=*on*|*off*|*fletcher2*|*fletcher4*|*sha256*|*noparity*|*sha512* > |*skein*|*edonr* > > > > > Having checksum for a filesystem is strongly recommended. However, If > consider image format, > it doesn’t need to care about consistency in a file itself. As example > (!) - binary files in a system. > They don’t have checksum integrated, validation is done by another program > - pkg or another. > > > > > Why do you need modify snapshot image ? Could you describe more? Do you > modify current 3 snapshot files? > > > Analysis that require ram / nvram modification? Not sure if this is > already possible, but may come handy for experimenting with uefi and > maybe some OS (features) that will not run with unmodified nvram :-P > > > > Sorry I don’t get, why do you need to modify snapshot image, but not > directly vmem on the running > VM? > > Another question, checksum and modifying image - two mutual exclusive > things. > > > > If you are talking about compatibility of a Image format - it should be > compatible in > both directions, at least for not so big format changes. > > If consider overall snapshot/resume compatibility - I believe forward > compatibility > is not case and target. Indeed, why do you need to resume an image > created by > a higher version of a program? > > > This happens quite often. For instance there is a bug in application > and I need to revert to (at least) one step older version. Then I am > unable to work on a file that I just saved (or was autosaved for me). > Firefox profile settings let be the first example. KiCAD file format > is another example (sometimes I need to switch to a devel build to > evade a nasty blocker bug then anyone else that uses a release is > blocked for some months including me myself). > > > Any additional thing has a cost of development, testing and support. > Current > Implementation doesn’t support compatibility at all. Having compatibility > in both > directions can be hard. > > For example, if some variable is removed in bhyve, backward compatibility > is fine, > but forward compatibly is not possible unless that removed variable is > being saved > into a snapshot image just for forward compatibility. And of course, it > should be tested > and verified as worked. > > Do you like that approach? I don’t think so. So I guess only backward > compatibility > should be supported to make the snapshot code simple and robust. > > Thanks, > Vitaliy Gusev > > > -- Mario. [-- Attachment #2 --] <div dir="ltr"><div>Vitaliy,</div><div><br></div><div>what happens if I clone your repo as source code on my FreeBSD system. Can I test your code directly or not ? Anyway,I think that,before doing this,I need to follow some kind of tutorial,to understand how the workflow is. Otherwise I will be not able to test and stress it. <br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, May 25, 2023 at 3:40 PM Vitaliy Gusev <<a href="mailto:gusev.vitaliy@gmail.com">gusev.vitaliy@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><br><div><br><blockquote type="cite"><div>On 25 May 2023, at 04:30, Tomek CEDRO <<a href="mailto:tomek@cedro.info" target="_blank">tomek@cedro.info</a>> wrote:</div><br><div><div>On Wed, May 24, 2023 at 5:11 PM Vitaliy Gusev wrote:<br><blockquote type="cite">Protecting requires more efforts and it should be clearly defined: what is purpose. If<br>purpose is having checksum with 99.9% reliability, NVLIST HEADER can be widen<br>to have “checksum” key/value for a Section.<br></blockquote><br>Well, this could be optional but useful to make sure snapshot did not<br>break somehow for instance backup medium error or something like<br>that.. even more maybe a way to fix it.. just a design stage idea :-</div></div></blockquote><br>Yes, new format can have checksum of a Section data if implemented.</div><div><br><blockquote type="cite"><div><div><br><br><blockquote type="cite">If purpose is having crypto verification - I believe sha256 program should be your choice.<br></blockquote><br>My question was more specific to availability of that feature<br>(integrity + repair) rather than specific format :-)<br><br>The use case here is having a virtual machine (it was VirtualBox) with<br>a bare os installed, plus some common applications, that is snapshoted<br>at some point in time, then experimented a lot, restored from<br>snapshot, etc. I had a backup of such vm + snapshot backed up that got<br>broken somehow. It would be nice to know that something is broken,<br>what is broken, maybe a way to fix :-)<br></div></div></blockquote><div><br></div><div><br></div><div> “Integrity" is a very broad term. What checksum algorithm is fine enough?</div><div> </div><div>For the instance, ZFS has several options for checksum:</div><div><br></div></div><blockquote style="margin:0px 0px 0px 40px;border:medium none;padding:0px"><div><div><p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;background-color:rgb(231,238,238)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(231,238,238);background-color:rgb(0,0,0)"><b>checksum</b></span><span style="font-variant-ligatures:no-common-ligatures">=<b>on</b>|<b>off</b>|<b>fletcher2</b>|<b>fletcher4</b>|<b>sha256</b>|<b>noparity</b>|<b>sha512</b>|<b>skein</b>|<b>edonr</b></span></p></div></div><div><div><p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;background-color:rgb(231,238,238)"><span style="font-variant-ligatures:no-common-ligatures"> </span></p></div></div></blockquote><div><br></div><div>Having checksum for a filesystem is strongly recommended. However, If consider image format,</div><div>it doesn’t need to care about consistency in a file itself. As example (!) - binary files in a system.</div><div>They don’t have checksum integrated, validation is done by another program - pkg or another.</div><div><br></div><div><br></div><div><blockquote type="cite"><div><div><br><br><blockquote type="cite">Why do you need modify snapshot image ? Could you describe more? Do you<br>modify current 3 snapshot files?<br></blockquote><br>Analysis that require ram / nvram modification? Not sure if this is<br>already possible, but may come handy for experimenting with uefi and<br>maybe some OS (features) that will not run with unmodified nvram :-P<br></div></div></blockquote><div><br></div><div><br></div>Sorry I don’t get, why do you need to modify snapshot image, but not directly vmem on the running</div><div>VM?</div><div><br></div><div>Another question, checksum and modifying image - two mutual exclusive things. </div><div><br><blockquote type="cite"><div><div><br><br><blockquote type="cite">If you are talking about compatibility of a Image format - it should be compatible in<br>both directions, at least for not so big format changes.<br><br>If consider overall snapshot/resume compatibility - I believe forward compatibility<br>is not case and target. Indeed, why do you need to resume an image created by<br>a higher version of a program?<br></blockquote><br>This happens quite often. For instance there is a bug in application<br>and I need to revert to (at least) one step older version. Then I am<br>unable to work on a file that I just saved (or was autosaved for me).<br>Firefox profile settings let be the first example. KiCAD file format<br>is another example (sometimes I need to switch to a devel build to<br>evade a nasty blocker bug then anyone else that uses a release is<br>blocked for some months including me myself).<br></div></div></blockquote><div><br></div><div>Any additional thing has a cost of development, testing and support. Current</div><div>Implementation doesn’t support compatibility at all. Having compatibility in both</div><div>directions can be hard.</div><div><br></div><div>For example, if some variable is removed in bhyve, backward compatibility is fine,</div><div>but forward compatibly is not possible unless that removed variable is being saved</div><div>into a snapshot image just for forward compatibility. And of course, it should be tested</div><div>and verified as worked.</div><div><br></div><div>Do you like that approach? I don’t think so. So I guess only backward compatibility</div><div>should be supported to make the snapshot code simple and robust.</div><div><br></div></div><div>Thanks,</div><div>Vitaliy Gusev</div><div><br></div><div><br></div></div></blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature">Mario.<br></div>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B1FSijKkCnR5j1N9BN0CbqzzitibK7r9OA84jc2eZ=uJaMW-g>