From nobody Sun Jul 9 11:01:51 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QzPN629kdz4lr6m for ; Sun, 9 Jul 2023 11:01:58 +0000 (UTC) (envelope-from void@f-m.fm) Received: from wout4-smtp.messagingengine.com (wout4-smtp.messagingengine.com [64.147.123.20]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4QzPN36fSMz3s5B for ; Sun, 9 Jul 2023 11:01:55 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm2 header.b=fUP734Gc; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=dJYGM6DW; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 64.147.123.20 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.west.internal (Postfix) with ESMTP id E7099320099F for ; Sun, 9 Jul 2023 07:01:53 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Sun, 09 Jul 2023 07:01:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm2; t=1688900513; x=1688986913; bh=MH 8X0UEkEE+efuOhlFtakjpuw1ehWZzLYLbc/TR8im8=; b=fUP734GcR1gvDuyvqU AeWAJUKFea0SVMV+011xpYcXGujdov9Gd75Nx1ORQeLw45ttk/0x4/qYoGG/8W+U DVWF4gl59KaQU2X03z+G24q7c9SlVhFdBSbGzbjy+iW1tyKO4lNf7bU8CJ4MuWsv 0LIIHo/FByQgOxyWnkSDXPHKyiguoVqol+FK9VVm6J8vbY0ls5Tdzzm1oFBwXSWT HPCUmF1Xnc2j+XWVm2a/DpveLSuoC1OyCV86BmgLCZO1MMndIfK/dPV6pCoF2gwl yxdsYe+Ed/8MGxMmvcTGH3I/l2CZO6ynEVHD0WY88bHV3FamICy4/Q2K29Zlng9y TbxQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; t=1688900513; x=1688986913; bh=MH8X0UEkEE+ef uOhlFtakjpuw1ehWZzLYLbc/TR8im8=; b=dJYGM6DWUL1yTAJqBaErPX7O+mP71 JiBWxaPUGvUjbP2I4Ca240GgPfmSlLwiPl7FtQWhJrI5lObIabhZZ3aLf9R0tNho rxfZp8VXj6sQa6fhFWDpLmX/3MpwKVrg2daeLUBlo8gEPFHQRnHqqyTSHo/vVrt7 I+Qgx/MXdwlA6/x3AAjGlEy1OD2pggNy8Wg0yI0UJDjT3nOARImM6/12bwpC2dZS iFrrkcNRpBLszZgVUa4U/w7sfRkd3i36uDFEY03i656TpU6nb/AhCbBf9YyPu/p0 U7vjUCM7y1ta/jBp0ORK0HQGaop230ZtNHfAZ+gkGNgXp9SeEHDKeoqag== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrvdehgdefiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehttdertd dttddvnecuhfhrohhmpehvohhiugcuoehvohhiugesfhdqmhdrfhhmqeenucggtffrrght thgvrhhnpeekleduvdelhfeileefgffghfffkedtheellefgudfgvdegkeejjedutdehhe fgueenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehv ohhiugesfhdqmhdrfhhm X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Sun, 9 Jul 2023 07:01:52 -0400 (EDT) Date: Sun, 9 Jul 2023 12:01:51 +0100 From: void To: virtualization@freebsd.org Subject: Re: convert bhyve vm image to something linux lvm/kvm/qemu can use Message-ID: References: <0ef823e3-7076-d64f-3d15-aa1304f76807@syscid.com> List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <0ef823e3-7076-d64f-3d15-aa1304f76807@syscid.com> X-Spamd-Result: default: False [-4.95 / 15.00]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_SHORT(-0.99)[-0.991]; NEURAL_HAM_MEDIUM(-0.99)[-0.989]; NEURAL_HAM_LONG(-0.97)[-0.966]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; RWL_MAILSPIKE_EXCELLENT(-0.40)[64.147.123.20:from]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm2,messagingengine.com:s=fm2]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.20]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.20:from]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[virtualization@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[f-m.fm]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4QzPN36fSMz3s5B X-Spamd-Bar: ---- X-ThisMailContainsUnwantedMimeParts: N On Sat, Jul 08, 2023 at 03:30:55PM +0200, Georg Pfuetzenreuter wrote: >Hi, > >I think qcow2 is the most popular format on KVM. Hi, thanks for this. Good to know but in the end I didn't need to convert. What i did was to transfer the image onto the linux hosting then wrote using dd to a lvm device. in order to get the console working, I needed this line at the top of /boot/loader.conf: console="comconsole" and then, had to comment out the console line and reboot. The console line had to be commented out because otherwise the message "can't exec getty 'none' for port /dev/console: No such file or directory" gets spammed to the console. -- From nobody Sun Jul 9 21:00:52 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qzfg85KHfz4mZpf for ; Sun, 9 Jul 2023 21:00:52 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Qzfg83g4yz3MCw for ; Sun, 9 Jul 2023 21:00:52 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1688936452; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=YmQ5fZXOmHSvgk+h1fKaqXscWZb6uxrDW9+85OYq9gQ=; b=fWNjYFMD/T4CNX2HGFY7VlMOqXoSme9U9u4Xc1su35DcnPZA31hGB0tG4BAG0NgpakyefD quNnPmlVDfz2BDtbmYOBgyrQu4kd68cfMJwNqyE/EaiZ2mhAZ/DPjw+TtmN26A9bAC31uD I8dzS3/oX+6zk/hJq+EVrh4LYxrigiEjWuuGi2jvsW0SMUSPDwA6hSHTYdqCaVh0PGcw6B RQ47P7zzCD7ngNbv8Gt/Z/Ca9ot+/nNU578xGm+/KmZpzDu3McCjn1nFWnjAu2JKmxsGp8 7KxPxzyn7DF9cpGIXbeKvIGKUoeDHVL9P1kyFfRrzSERW7B3dz7u8/cVuq2ScA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1688936452; a=rsa-sha256; cv=none; b=kAy2LF0auG04V2Rc25joxt0i19WCWcdN4ED6LcSqVidiQweQmpFEUDaqvR9ZMFaJe+gad8 fGh9UANHfVwgjPApYEdsH+KQ5b/DSxKlG58FFv7tHIBGoZZkC98qQjYzOjagdm/jwiB+eD 6Yk4u3vRuSHv8LYt347jEsDmeC8HWDMZseN3MFsDRF/P+V/BXACNVHBxEcQ7RdRxAyke7S uPdMOP6lP/KNk1yS9/qfCQKupfgguD00MzcqlNpV8xnK2Hx3eu9MsSVrmXS1A4Geu9W7+W /xo3k+hsxnP8nG4MWTip+Kk5r/pOIExAQwvWryHCfh8/HjmsvkMM+MAMni/CsQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Qzfg82bgxzJDd for ; Sun, 9 Jul 2023 21:00:52 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 369L0q6G096658 for ; Sun, 9 Jul 2023 21:00:52 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 369L0qKk096657 for virtualization@FreeBSD.org; Sun, 9 Jul 2023 21:00:52 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202307092100.369L0qKk096657@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: virtualization@FreeBSD.org Subject: Problem reports for virtualization@FreeBSD.org that need special attention Date: Sun, 9 Jul 2023 21:00:52 +0000 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="16889364521.7E150f.91508" Content-Transfer-Encoding: 7bit X-ThisMailContainsUnwantedMimeParts: N --16889364521.7E150f.91508 Date: Sun, 9 Jul 2023 21:00:52 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- In Progress | 247208 | mpt(4): VMWare virtualized LSI controller panics New | 240945 | [hyper-v] [netvsc] hn network driver incorrectly Open | 244838 | "bectl activate -t" does not honor the -t flag in 3 problems total for which you should take action. --16889364521.7E150f.91508 Date: Sun, 9 Jul 2023 21:00:52 +0000 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8" 
The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.

Status      |    Bug Id | Description
------------+-----------+---------------------------------------------------
In Progress |    247208 | mpt(4): VMWare virtualized LSI controller panics 
New         |    240945 | [hyper-v] [netvsc] hn network driver incorrectly 
Open        |    244838 | "bectl activate -t" does not honor the -t flag in

3 problems total for which you should take action.
--16889364521.7E150f.91508-- From nobody Fri Jul 14 17:39:58 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R2dzJ01Jmz2tryk for ; Fri, 14 Jul 2023 17:40:12 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx1.shrew.net (mx1.shrew.net [38.97.5.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4R2dzH25Mdz4564 for ; Fri, 14 Jul 2023 17:40:11 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 38.97.5.131 as permitted sender) smtp.mailfrom=mgrooms@shrew.net; dmarc=none Received: from mail.shrew.net (mail.shrew.prv [10.24.10.20]) by mx1.shrew.net (8.15.2/8.15.2) with ESMTP id 36EHe4Mj085556 for ; Fri, 14 Jul 2023 12:40:04 -0500 (CDT) (envelope-from mgrooms@shrew.net) Received: from [10.22.200.32] (unknown [136.49.230.220]) by mail.shrew.net (Postfix) with ESMTPSA id E7997194F36 for ; Fri, 14 Jul 2023 12:39:58 -0500 (CDT) Message-ID: <79fabe94-b800-c713-48ea-518da1f74e4d@shrew.net> Date: Fri, 14 Jul 2023 12:39:58 -0500 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 From: Matthew Grooms Subject: Re: BHYVE SNAPSHOT image format proposal To: virtualization@freebsd.org References: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> <6b98da58a5bd8e83bc466efa20b5a900298210aa.camel@FreeBSD.org> <8387AC83-6667-48E5-A3FA-11475EA96A5F@gmail.com> <986A83D8-E0E0-4030-9369-A5B3500E27C6@gmail.com> Content-Language: en-US In-Reply-To: <986A83D8-E0E0-4030-9369-A5B3500E27C6@gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (mx1.shrew.net [10.24.10.10]); Fri, 14 Jul 2023 12:40:04 -0500 (CDT) X-Spamd-Result: default: False [-1.21 / 15.00]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_SPAM_LONG(0.98)[0.984]; NEURAL_HAM_MEDIUM(-0.89)[-0.890]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:174, ipnet:38.0.0.0/8, country:US]; DMARC_NA(0.00)[shrew.net]; RCVD_COUNT_THREE(0.00)[3]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[virtualization@freebsd.org]; TO_DN_NONE(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4R2dzH25Mdz4564 X-Spamd-Bar: - X-ThisMailContainsUnwantedMimeParts: N On 6/7/23 13:25, Vitaliy Gusev wrote: > Hi Corvin, > >> On 6 Jun 2023, at 15:59, Corvin Köhne wrote: >>> ... >> >> We may have different version of the format from the same produce. >> IMHO, it makes sense to have a dedicated IDENT and VERSION field to >> easily figure out >> >> 1) if the producer of the image is known >> 2) if we support that version of the producer >> >> Even if you allocated a huge amount of free space, someone would need >> more. So, what do you think about this format: >> >> +---------------------------------------------------------------------+ >> | IDENT - 16 BYTES                                                    | >> +-------------------+-----------------------+-------------------------+ >> | VERSION - 4 BYTES | NVLIST SIZE - 4 BYTES | NVLIST OFFSET - 8 BYTES | >> +-------------------+-----------------------+-------------------------+ >> | POSSIBLE FREE SPACE (e.g. for custom data, alignment etc.)          | >> +---------------------------------------------------------------------+ >> | NVLIST DATA                                                         | >> +---------------------------------------------------------------------+ >> | POSSIBLE FREE SPACE (for whatever reason)                           | >> +---------------------------------------------------------------------+ >> | SNAPSHOT DATA                                                       | >> +——————————————————————————————————+ >> I agree. The only values that need to exist in the initial header would be the identity and version. This could be as simple as "BHYVE" which is similar to how common network protocols identify themselves ... SSH-protoversion-softwareversion SP comments CR LF Note the use of a carriage return + line feed to signify the identity + version string. It avoids the need to reserve a specific number of bytes for this purpose and offers more free form expression of values. If you want to include a stream subtype, you could add another field resulting in something like ... IDENT-SUBTYPE-VERSION CR LF I think it would also be helpful to borrow the concept of using generic data stream section headers. Instead of expecting a fixed number bytes after the identity + version, you can append section headers that define the type and the length of each. For example ... +-----------------------------------+ | bhyve-checkpoint-1.0\n\r | +-----------------------------------+ | TYPE - 2 BYTES | LENGTH - 8 BYTES | +-----------------------------------+ | DATA ... | +-----------------------------------+ | TYPE - 2 BYTES | LENGTH - 8 BYTES | +-----------------------------------+ | DATA ... | +-----------------------------------+ Example section types: 0x0001 = NVLIST A 0x0002 = NVLIST B 0x0003 = MEMORY 0x0004 = ... 0xFFFF = CHECKSUM Any number of sections could be included in a single file. Sections can be added or without breaking the stream format. Required sections for the subtype could be accounted for and simple integrity checks could be performed based on section length. Checksum validation can be performed by only parsing the trailing checksum section and the rest as raw input. The same basic structure could also be used in a network stream so there is potential for code reuse. > > Note, simple string "BHYVE CHECKPOINT IMAGE” has 22 bytes. So 16 bytes seems > too small. > > So I would not to complicate a header first. > > I would rather describe ideas, conditions and then solutions: > > 1. Need to distinguish snapshot image file from other files. > _Solution_: Header should have "magic id”. > > 2. Need a barrier for resuming if image "is not ours”. Idea is not to > allow to resume images from other producers. > > The reason to have it and use it instead of header versioning: > > Imagine that mainstream has its own implementation and company’s > fork repo has its own implementation. How to *ensure* that the > *versions* in an image file are ours and not somebody’s else? > > _Solution_:  Header should have “Producer id” string. > > Example: Snapshot image file has empty Producer string , but bhyve > has current Producer as “MYCOMPANY”. Strings are not equal, resume > must fail. > It would appear that you'd like the provider to be a string. I'm not sure why this couldn't be added by the provider as a value in an nvlist section. The consumer can filter on that value. Why should it be part of the stream format? > 3. The Rule above does not restrict getting/decoding data from an image > file. It should be possible to look at an image file and analyse > internals, to get/decode values, etc. > _Solution_: Have additional option either in bhyve or bhyvectl to > get into image file. > See my final comments regarding encoding. > 4. Following nvlist header data should have a short information about > image file and its internals. > _Solution_: NV HEADER can have several sections: “config”, “kernel”, > “devices”, “memory”, … > See my final comments regarding encoding. > 5. Versioning of NV HEADER. Idea is to have an information in > advance whether it is possible to be resumed or not. In other words, > before do resume, get information about ability to resume. > > _Solution_: Each Section should have “version”  and “subversion”. > While “version” is responsible for both  types of compatibility: > backward and forward, “subversion” is for forward compatibility only. > > _Rules for check_: >         If bh_version == version && bh_subversion >= subversion  then >                   Bhyve able to resume the Section >         Else >                   Bhyve cannot resume the Section >         Endif > _ > Example 1_: Section in image has “version=1", “subversion=5”,  Bhyve > has “version=1", “subversion=6". That means, bhyve can resume the > Section. > > _Example 2_: The same image Section, but bhyve has “version=1", > “subversion=4". Bhyve cannot resume the Section. > > Example 3: The same image Section, but bhyve has “version=2", > “subversion=5”. Bhyve cannot resume the Section. > > _Rules for increasing versions_: >   -  If during code-change “backward” compatibility is broken, > “version” should be increased and “subversion” is set to 0. >   -  If during code-change “forward” compatibility is broken, > “subversion” should be increased. > More version values could be included I suppose, but it seems overly complicated to me. I see most of the change over time being in nvlist value sections. There's a lot of flexibility and validation that can occur there before pushing version numbers down into individual stream sections, but I guess I could be convinced otherwise. In any case, we need to be careful to not rely too much on version checks. Otherwise it will be difficult to snapshot/restore/migrate between different versions of bhyve. > 6. Other versioning in HEADER is redundant. If something is changed in > the format, “magic id” can be changed appropriately. > _Solution_: “magic id” should be stable and not changed for a long time. > I think that's what the major version would be used for. > > As result I would suggest to give at least 32 bytes for "magic id” / > ident and 32 bytes for “producer id”. > > Format of entire image file can be: > > > +-----------------------------------------------------------------+ > |               HEADER MAGIC ID     - 32 BYTES                    | > +-----------------------------------------------------------------+ > |               HEADER PRODUCER ID  - 32 BYTES                    | > +———————————————----------------————————--—-----------------------+ > |               NVLIST HEADER SIZE  -  4 BYTES                    | > +-----------------------------------------------------------------+ > |               NVLIST HEADER DATA (SECTIONS)                     | > +-----------------------------------------------------------------+ > |                       SNAPSHOT DATA                             | > +-----------------------------------------------------------------+ > > > _MAGIC ID_: should be hardcoded: "BHYVE CHECKPOINT IMAGE”. > > _PRODUCER ID_: can be empty and supported by producer, i.e. reserved. > _ > _ > _NVLIST HEADER SIZE_: has enough dimension, but in general size is less > than 4KB > > _NVLIST HEADER DATA_: Packed nvlist data, contains Sections:  “config”, > “kernel”, “devices”, “memory”, … : > > [config] > >         offset = 0x1000 (4096) > >         size = 0x1f6 (502) > >         type = text > > vers = 1 > subvers = 5 > > [kernel] > >         offset = 0x11f6 (4598) > >         size = 0x19a7 (6567) > >         type = nvlist > > vers = 1 > subvers = 0 > > [devices] > >         offset = 0x2b9d (11165) > >         size = 0x10145ba (16860602) > >         type = nvlist > > vers = 2 > subvers = 1 > > [memory] > >         offset = 0x1200000 (18874368) > >         size = 0x3ce00000 (1021313024) > >         type = pages > > vers = 1 > subvers = 0 > I see a need to define a format for bhyve so it's possible to mix different sections and encodings inside a unified stream. But all the data in your nvlist example above can be easily be represented as text. We already have JSON, YAML, XML, etc ... By adopting an preexisting format, we could retain the snapshot structure instead of lifting it up into the stream format. Even if we decide to break the structure up into different nvlist stream sections, using a common format would allow other tools to more easily parse and validate the structure inside these sections. Isn't that a good thing? Is there a reason we're trying to reinvent the wheel here? Thanks, -Matthew From nobody Fri Jul 14 18:08:19 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R2fZp2Qtqz4mbHc for ; Fri, 14 Jul 2023 18:07:30 +0000 (UTC) (envelope-from rob.fx907@gmail.com) Received: from mail-lf1-x132.google.com (mail-lf1-x132.google.com [IPv6:2a00:1450:4864:20::132]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R2fZp0Vb9z49hh for ; Fri, 14 Jul 2023 18:07:30 +0000 (UTC) (envelope-from rob.fx907@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x132.google.com with SMTP id 2adb3069b0e04-4fbe5161fbeso666496e87.0 for ; Fri, 14 Jul 2023 11:07:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1689358048; x=1691950048; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=ib+JAyGnvAg06chNZ3/vUgNLYUzuYvjPXwnC+7XvDaU=; b=JfK8FpmXwpS2CiJ5Db7coxVTLQUV/lD6mmnZIVBilirHajOoGi7mJowWhxdlfoRnkS OaWpZTKHOvv7Z6NRwAVg8r7GKqJ0pMC78a0JIZbggLAlJltI5ZDy88FP1PY6e5EAPJ2q jrnfYtefYeiSH/j5HNvgvdwaKrAMxHk4UWZ1p6LUBok7X9VwlHwbl2eXweOzGr87a90U dVnr6LTdUUnNsBTmLZHIKqGZp5jpLYgbJuzDLUllroFL/2rcA6DiZqciHcEVcs5p7p37 tNSi3n2FF0bQjJAPgKLxyBtyM4EH8fU1V6nVv/CtvvUfRrR0s8+yzW5eyFYWEd5MVPll EDDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689358048; x=1691950048; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ib+JAyGnvAg06chNZ3/vUgNLYUzuYvjPXwnC+7XvDaU=; b=OHIazIOYAPVTXdGoiwZ9SJS4yPrC5DxrNDrMcj3e8CPNyXtcZSPloe0AhwiDABpgtX kks0D8B2JFI4yn1vt8s7XtkTnZLHEbMDiwgLsXcjK83DuztC5L6k+ejo6bOLs10uXhlV IWQgGvB1FvBVtORH5CgzWK4xb7R8k+JMGURHqm7kIgJk4cAQ9ChI7aqUsa4A+AuT1TZS /+8rxC5OC2nPPAmTE07QjXxM38kZnnzRIvB6+LY1R6KD/AQ8ALtW8YQX8+MKlDoTftWZ iaZDJXYDIYGQiZyET93F5fMOpesq/AqTtnCruPlRMy/n3G+uHPqyNne58TqUOaVFBFO9 T1bg== X-Gm-Message-State: ABy/qLZiDrqQpv0Hm2r7rcyVp0yUExc0mg4aZnlWHI4cbvG/MZql8NXJ noOcOikLV8DJIp3byNazL7rZRWtzn+Xlad9a9Tk= X-Google-Smtp-Source: APBJJlGlObGf3razMdBjC/eWnxv5NMETfu5/PxfdoUFs55PUOOOr1Q/COOLBTAHm2IODwOcNmDbkTb+WpvFm/7ecPc4= X-Received: by 2002:a2e:9016:0:b0:2b6:cd7f:5ea8 with SMTP id h22-20020a2e9016000000b002b6cd7f5ea8mr3423641ljg.1.1689358047628; Fri, 14 Jul 2023 11:07:27 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> <6b98da58a5bd8e83bc466efa20b5a900298210aa.camel@FreeBSD.org> <8387AC83-6667-48E5-A3FA-11475EA96A5F@gmail.com> <986A83D8-E0E0-4030-9369-A5B3500E27C6@gmail.com> <79fabe94-b800-c713-48ea-518da1f74e4d@shrew.net> In-Reply-To: <79fabe94-b800-c713-48ea-518da1f74e4d@shrew.net> From: Rob Wing Date: Fri, 14 Jul 2023 10:08:19 -0800 Message-ID: Subject: Re: BHYVE SNAPSHOT image format proposal To: Matthew Grooms Cc: virtualization@freebsd.org Content-Type: multipart/alternative; boundary="00000000000053d1c006007652af" X-Rspamd-Queue-Id: 4R2fZp0Vb9z49hh X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --00000000000053d1c006007652af Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, Jul 14, 2023 at 9:40=E2=80=AFAM Matthew Grooms = wrote: > > I see a need to define a format for bhyve so it's possible to mix > different sections and encodings inside a unified stream. But all the > data in your nvlist example above can be easily be represented as text. > We already have JSON, YAML, XML, etc ... By adopting an preexisting > format, we could retain the snapshot structure instead of lifting it up > into the stream format. Even if we decide to break the structure up into > different nvlist stream sections, using a common format would allow > other tools to more easily parse and validate the structure inside these > sections. Isn't that a good thing? Is there a reason we're trying to > reinvent the wheel here? > Does JSON support storing binary data? I'm under the impression that it does not. --00000000000053d1c006007652af Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Fri, Jul 14, 2023 at 9:= 40=E2=80=AFAM Matthew Grooms <mgroo= ms@shrew.net> wrote:

I see a need to define a format for bhyve so it's possible to mix
different sections and encodings inside a unified stream. But all the
data in your nvlist example above can be easily be represented as text. We already have JSON, YAML, XML, etc ... By adopting an preexisting
format, we could retain the snapshot structure instead of lifting it up into the stream format. Even if we decide to break the structure up into different nvlist stream sections, using a common format would allow
other tools to more easily parse and validate the structure inside these sections. Isn't that a good thing? Is there a reason we're trying t= o
reinvent the wheel here?

Does JSON supp= ort storing binary data? I'm under the impression that it does not.
=
--00000000000053d1c006007652af-- From nobody Fri Jul 14 18:27:35 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R2g2D0mKXz4mjhC for ; Fri, 14 Jul 2023 18:27:48 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx2.shrew.net (mx2.shrew.net [38.97.5.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4R2g2B575Lz4GN0 for ; Fri, 14 Jul 2023 18:27:46 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 38.97.5.132 as permitted sender) smtp.mailfrom=mgrooms@shrew.net; dmarc=none Received: from mail.shrew.net (mail.shrew.prv [10.24.10.20]) by mx2.shrew.net (8.15.2/8.15.2) with ESMTP id 36EIReFK004584; Fri, 14 Jul 2023 13:27:40 -0500 (CDT) (envelope-from mgrooms@shrew.net) Received: from [10.22.200.32] (unknown [136.49.230.220]) by mail.shrew.net (Postfix) with ESMTPSA id 6B36A194ED9; Fri, 14 Jul 2023 13:27:35 -0500 (CDT) Message-ID: <6e7f3128-b8fa-8cd8-007d-ffd9b89c62d1@shrew.net> Date: Fri, 14 Jul 2023 13:27:35 -0500 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Subject: Re: BHYVE SNAPSHOT image format proposal Content-Language: en-US To: Rob Wing Cc: virtualization@freebsd.org References: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> <6b98da58a5bd8e83bc466efa20b5a900298210aa.camel@FreeBSD.org> <8387AC83-6667-48E5-A3FA-11475EA96A5F@gmail.com> <986A83D8-E0E0-4030-9369-A5B3500E27C6@gmail.com> <79fabe94-b800-c713-48ea-518da1f74e4d@shrew.net> From: Matthew Grooms In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (mx2.shrew.net [10.24.10.11]); Fri, 14 Jul 2023 13:27:40 -0500 (CDT) X-Spamd-Result: default: False [-3.13 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; NEURAL_HAM_LONG(-0.83)[-0.829]; R_SPF_ALLOW(-0.20)[+mx:c]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_TO(0.00)[gmail.com]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:174, ipnet:38.0.0.0/8, country:US]; RCPT_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; BLOCKLISTDE_FAIL(0.00)[136.49.230.220:server fail,38.97.5.132:server fail]; TO_DN_SOME(0.00)[]; DMARC_NA(0.00)[shrew.net]; TAGGED_RCPT(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4R2g2B575Lz4GN0 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On 7/14/23 13:08, Rob Wing wrote: > > > On Fri, Jul 14, 2023 at 9:40 AM Matthew Grooms > wrote: > > > I see a need to define a format for bhyve so it's possible to mix > different sections and encodings inside a unified stream. But all the > data in your nvlist example above can be easily be represented as text. > We already have JSON, YAML, XML, etc ... By adopting an preexisting > format, we could retain the snapshot structure instead of lifting it up > into the stream format. Even if we decide to break the structure up > into > different nvlist stream sections, using a common format would allow > other tools to more easily parse and validate the structure inside > these > sections. Isn't that a good thing? Is there a reason we're trying to > reinvent the wheel here? > > Does JSON support storing binary data? I'm under the impression that it > does not. Hey Rob, I wasn't suggesting that any of the bulk data be encoded. But I think I see the disconnect as I replied to the reply vs the original proposal. I'll try again this weekend when I have more time to read everything again and re-digest. -Matthew From nobody Fri Jul 14 22:37:46 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R2mZp4qbqz4nT5W for ; Fri, 14 Jul 2023 22:37:54 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx1.shrew.net (mx1.shrew.net [38.97.5.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4R2mZn4b2bz3xf6 for ; Fri, 14 Jul 2023 22:37:53 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 38.97.5.131 as permitted sender) smtp.mailfrom=mgrooms@shrew.net; dmarc=none Received: from mail.shrew.net (mail.shrew.prv [10.24.10.20]) by mx1.shrew.net (8.15.2/8.15.2) with ESMTP id 36EMbqkG087396; Fri, 14 Jul 2023 17:37:52 -0500 (CDT) (envelope-from mgrooms@shrew.net) Received: from [10.22.200.32] (unknown [136.49.230.220]) by mail.shrew.net (Postfix) with ESMTPSA id 2BAF5194F0D; Fri, 14 Jul 2023 17:37:47 -0500 (CDT) Message-ID: <3973013d-c183-360f-d7ca-ca822859c23d@shrew.net> Date: Fri, 14 Jul 2023 17:37:46 -0500 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Subject: Re: BHYVE SNAPSHOT image format proposal Content-Language: en-US To: Rob Wing Cc: virtualization@freebsd.org References: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> <6b98da58a5bd8e83bc466efa20b5a900298210aa.camel@FreeBSD.org> <8387AC83-6667-48E5-A3FA-11475EA96A5F@gmail.com> <986A83D8-E0E0-4030-9369-A5B3500E27C6@gmail.com> <79fabe94-b800-c713-48ea-518da1f74e4d@shrew.net> From: Matthew Grooms In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (mx1.shrew.net [10.24.10.10]); Fri, 14 Jul 2023 17:37:52 -0500 (CDT) X-Spamd-Result: default: False [-3.29 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; NEURAL_HAM_LONG(-0.99)[-0.993]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_TO(0.00)[gmail.com]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:174, ipnet:38.0.0.0/8, country:US]; R_DKIM_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_SOME(0.00)[]; DMARC_NA(0.00)[shrew.net]; TAGGED_RCPT(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4R2mZn4b2bz3xf6 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On 7/14/23 13:08, Rob Wing wrote: > > > On Fri, Jul 14, 2023 at 9:40 AM Matthew Grooms > wrote: > > > I see a need to define a format for bhyve so it's possible to mix > different sections and encodings inside a unified stream. But all the > data in your nvlist example above can be easily be represented as text. > We already have JSON, YAML, XML, etc ... By adopting an preexisting > format, we could retain the snapshot structure instead of lifting it up > into the stream format. Even if we decide to break the structure up > into > different nvlist stream sections, using a common format would allow > other tools to more easily parse and validate the structure inside > these > sections. Isn't that a good thing? Is there a reason we're trying to > reinvent the wheel here? > > > Does JSON support storing binary data? I'm under the impression that it > does not. Hi Rob, When we spoke to John Baldwin and others in the community about being able to remove the #ifdef's from the snapshot code, we were told that copying binary data structures to a state file was the wrong approach and that a better method needed to be provided. We agreed. That's why the following work was undertaken to provide a rich file format that describes the component values of device's state instead of codifying the c structure in the file format ... https://reviews.freebsd.org/D29262 When Vitaliy added comments to that review WRT an nvlist based approach, I assumed that meant preserving the decomposition of device information that the UPB team spent so much effort trying to extract. I should have read the original file format proposal email before I replied as I tried too hard to interpret it info through that lens. My mistake. If Vitaliy, and apparently you, favor the continued practice of copying data from c structure pointers directly into files to save device state, I have no more comments on that approach. Maybe John or UPB will chime in here with feedback that's more helpful. -Matthew From nobody Sat Jul 15 00:07:03 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R2pYk2TRXz4n6yN for ; Sat, 15 Jul 2023 00:07:06 +0000 (UTC) (envelope-from rob.fx907@gmail.com) Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R2pYj2zs5z4Hv6 for ; Sat, 15 Jul 2023 00:07:05 +0000 (UTC) (envelope-from rob.fx907@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-wr1-x433.google.com with SMTP id ffacd0b85a97d-31422f20399so562890f8f.1 for ; Fri, 14 Jul 2023 17:07:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1689379624; x=1691971624; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=lIDG824+HsY6Myl8PFO3rB1UaUbtzDt1UklcIoQuSA4=; b=DgqNLtTvxY9k3GgcBVxGS8cvPbtJ8kG1dLo1nxy6LHtlpwz+ZDYiniimDWN/ChmZVU cF70unLC3Iw46L9McLsql7wIUwKIesWbICgOw+mTdtdHLUvyY+A1XsA8t6sFMnhVruVG 3mFg6kv2YifnUvKSm6eYxN9pAa1OPcSSipZNf11IeRuJ1eRLzADokGGZ8Obe1+8NajHR QI5u18zhckEirbYCQVZsXmTi3DZSBpYasppQyYKENSyl3xLU/XBPM6bCkERaAnSVYifd 33rqs2ZYPPKjb2kztQJiUiMt+K8myU0dFqhAybqMG14HqlWta03Mea+++f0QKng6lHZt cYrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689379624; x=1691971624; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=lIDG824+HsY6Myl8PFO3rB1UaUbtzDt1UklcIoQuSA4=; b=R+rSElOEh7SGqgT+n3fPmY+BAJZIjwwk+90CkeZ0MrNxyAqcjpe42R8frbIraSOtUf U7+MxjY4xeyMuDSUiT+d9idUo5jwztJgwfn0HamSqWwOCLev22P4720XlDILxSoD4xDH gs7EFyY8QRWrIEQzkLNqEKC1mH9XFjFkAttVpla17wGfVUABO5v1EuaAQq46m8BydytA EyLyBN6X1piIxkhnrCQ2MTFU9tgQ3bHV8QNzmPyAelMJvPfpvM5FQ/wrMcQ799TBgVLG WNP0Zxe5wAcQn6u5aiQeaKaDPvcC0/wwglbw57K4ANj8oKZqMj6i6owf5sE3cpMYzSkI jhCA== X-Gm-Message-State: ABy/qLbJ/vuo1FUdm0I0L3nrZSP6dbeQhIFWh4qehJZVzUp6uEYjm7Ih ImSN6QADLFgzFmWdZ/ZOTLIY8al+jSpTafG5CDEoMfEm X-Google-Smtp-Source: APBJJlGSIG5ipgB91AUPf1SAqeCdpArRl9Rp28KrnCKaFgeHUfHn+bnV9Q3X7X1CqTkK2/ldhKlQiEX8ny7NqFFJcWo= X-Received: by 2002:a05:600c:474f:b0:3f7:fb5d:6e7a with SMTP id w15-20020a05600c474f00b003f7fb5d6e7amr435135wmo.0.1689379623696; Fri, 14 Jul 2023 17:07:03 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Received: by 2002:a17:907:970d:b0:982:69cd:7365 with HTTP; Fri, 14 Jul 2023 17:07:03 -0700 (PDT) In-Reply-To: <3973013d-c183-360f-d7ca-ca822859c23d@shrew.net> References: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> <6b98da58a5bd8e83bc466efa20b5a900298210aa.camel@FreeBSD.org> <8387AC83-6667-48E5-A3FA-11475EA96A5F@gmail.com> <986A83D8-E0E0-4030-9369-A5B3500E27C6@gmail.com> <79fabe94-b800-c713-48ea-518da1f74e4d@shrew.net> <3973013d-c183-360f-d7ca-ca822859c23d@shrew.net> From: Rob Wing Date: Fri, 14 Jul 2023 16:07:03 -0800 Message-ID: Subject: Re: BHYVE SNAPSHOT image format proposal To: Matthew Grooms Cc: "virtualization@freebsd.org" Content-Type: multipart/alternative; boundary="0000000000005c7d1d06007b5815" X-Rspamd-Queue-Id: 4R2pYj2zs5z4Hv6 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --0000000000005c7d1d06007b5815 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable The review you referenced requires multiple files for a snapshot, which I think is a non-starter. I was trying to ensure that the limitations described in the original snapshot commit message are addressed appropriately - but maybe I've mis-interpreted the message or that I don't fully understand the problem. I don't want give unhelpful or contradictory suggestions to what you've received in the past. The best I can do at this point is defer to those who you have made these agreements with. On Friday, July 14, 2023, Matthew Grooms wrote: > On 7/14/23 13:08, Rob Wing wrote: > >> >> >> On Fri, Jul 14, 2023 at 9:40=E2=80=AFAM Matthew Grooms > > wrote: >> >> >> I see a need to define a format for bhyve so it's possible to mix >> different sections and encodings inside a unified stream. But all th= e >> data in your nvlist example above can be easily be represented as >> text. >> We already have JSON, YAML, XML, etc ... By adopting an preexisting >> format, we could retain the snapshot structure instead of lifting it >> up >> into the stream format. Even if we decide to break the structure up >> into >> different nvlist stream sections, using a common format would allow >> other tools to more easily parse and validate the structure inside >> these >> sections. Isn't that a good thing? Is there a reason we're trying to >> reinvent the wheel here? >> >> >> Does JSON support storing binary data? I'm under the impression that it >> does not. >> > > Hi Rob, > > When we spoke to John Baldwin and others in the community about being abl= e > to remove the #ifdef's from the snapshot code, we were told that copying > binary data structures to a state file was the wrong approach and that a > better method needed to be provided. We agreed. That's why the following > work was undertaken to provide a rich file format that describes the > component values of device's state instead of codifying the c structure i= n > the file format ... > > https://reviews.freebsd.org/D29262 > > When Vitaliy added comments to that review WRT an nvlist based approach, = I > assumed that meant preserving the decomposition of device information tha= t > the UPB team spent so much effort trying to extract. I should have read t= he > original file format proposal email before I replied as I tried too hard = to > interpret it info through that lens. My mistake. > > If Vitaliy, and apparently you, favor the continued practice of copying > data from c structure pointers directly into files to save device state, = I > have no more comments on that approach. Maybe John or UPB will chime in > here with feedback that's more helpful. > > -Matthew > --0000000000005c7d1d06007b5815 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable The review you referenced requires multiple files for a snapshot, which I t= hink is a non-starter.

I was trying to ensure that the l= imitations described in the original snapshot commit message are addressed = appropriately - but maybe I've mis-interpreted the message or that I do= n't fully understand the problem.

I don't = want give unhelpful or contradictory suggestions to what you've receive= d in the past.

The best I can do at this point is = defer to those who you have made these agreements with.

On Friday, J= uly 14, 2023, Matthew Grooms <mgroo= ms@shrew.net> wrote:
On 7/14/23 13= :08, Rob Wing wrote:


On Fri, Jul 14, 2023 at 9:40=E2=80=AFAM Matthew Grooms <mgrooms@shrew.net <mailto:mgrooms@shrew.net&g= t;> wrote:


=C2=A0 =C2=A0 I see a need to define a format for bhyve so it's possibl= e to mix
=C2=A0 =C2=A0 different sections and encodings inside a unified stream. But= all the
=C2=A0 =C2=A0 data in your nvlist example above can be easily be represente= d as text.
=C2=A0 =C2=A0 We already have JSON, YAML, XML, etc ... By adopting an preex= isting
=C2=A0 =C2=A0 format, we could retain the snapshot structure instead of lif= ting it up
=C2=A0 =C2=A0 into the stream format. Even if we decide to break the struct= ure up
=C2=A0 =C2=A0 into
=C2=A0 =C2=A0 different nvlist stream sections, using a common format would= allow
=C2=A0 =C2=A0 other tools to more easily parse and validate the structure i= nside
=C2=A0 =C2=A0 these
=C2=A0 =C2=A0 sections. Isn't that a good thing? Is there a reason we&#= 39;re trying to
=C2=A0 =C2=A0 reinvent the wheel here?


Does JSON support storing binary data? I'm under the impression that it= does not.

Hi Rob,

When we spoke to John Baldwin and others in the community about being able = to remove the #ifdef's from the snapshot code, we were told that copyin= g binary data structures to a state file was the wrong approach and that a = better method needed to be provided. We agreed. That's why the followin= g work was undertaken to provide a rich file format that describes the comp= onent values of device's state instead of codifying the c structure in = the file format ...

https://re= views.freebsd.org/D29262

When Vitaliy added comments to that review WRT an nvlist based approach, I = assumed that meant preserving the decomposition of device information that = the UPB team spent so much effort trying to extract. I should have read the= original file format proposal email before I replied as I tried too hard t= o interpret it info through that lens. My mistake.

If Vitaliy, and apparently you, favor the continued practice of copying dat= a from c structure pointers directly into files to save device state, I hav= e no more comments on that approach. Maybe John or UPB will chime in here w= ith feedback that's more helpful.

-Matthew
--0000000000005c7d1d06007b5815-- From nobody Sat Jul 15 14:01:07 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R394C6HTFz4mfGG for ; Sat, 15 Jul 2023 14:01:15 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx1.shrew.net (mx1.shrew.net [38.97.5.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4R394B5w7dz4Dpt for ; Sat, 15 Jul 2023 14:01:14 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 38.97.5.131 as permitted sender) smtp.mailfrom=mgrooms@shrew.net; dmarc=none Received: from mail.shrew.net (mail.shrew.prv [10.24.10.20]) by mx1.shrew.net (8.15.2/8.15.2) with ESMTP id 36FE1DmZ091160; Sat, 15 Jul 2023 09:01:13 -0500 (CDT) (envelope-from mgrooms@shrew.net) Received: from [10.22.200.32] (unknown [136.49.230.220]) by mail.shrew.net (Postfix) with ESMTPSA id 211D2193A65; Sat, 15 Jul 2023 09:01:08 -0500 (CDT) Message-ID: <3a037482-2e6c-667f-1979-d5b612e506ec@shrew.net> Date: Sat, 15 Jul 2023 09:01:07 -0500 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Subject: Re: BHYVE SNAPSHOT image format proposal Content-Language: en-US To: Rob Wing Cc: "virtualization@freebsd.org" References: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> <6b98da58a5bd8e83bc466efa20b5a900298210aa.camel@FreeBSD.org> <8387AC83-6667-48E5-A3FA-11475EA96A5F@gmail.com> <986A83D8-E0E0-4030-9369-A5B3500E27C6@gmail.com> <79fabe94-b800-c713-48ea-518da1f74e4d@shrew.net> <3973013d-c183-360f-d7ca-ca822859c23d@shrew.net> From: Matthew Grooms In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (mx1.shrew.net [10.24.10.10]); Sat, 15 Jul 2023 09:01:13 -0500 (CDT) X-Spamd-Result: default: False [-3.30 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; ASN(0.00)[asn:174, ipnet:38.0.0.0/8, country:US]; R_DKIM_NA(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; TO_DN_SOME(0.00)[]; DMARC_NA(0.00)[shrew.net]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4R394B5w7dz4Dpt X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On 7/14/23 19:07, Rob Wing wrote: > The review you referenced requires multiple files for a snapshot, which > I think is a non-starter. > We'll overlook the fact that it does attempt to consolidate files ( 2 vs 3 ) and that there's no feedback requesting further consolidation after being open for two years, but noted. > I was trying to ensure that the limitations described in the original > snapshot commit message are addressed appropriately - but maybe I've > mis-interpreted the message or that I don't fully understand the problem. > Since we're trying to ensure that limitations in the original commit message are addressed, let's read that ... https://reviews.freebsd.org/rS360648 "While the current implementation is useful for several uses cases, it has a few limitations. The file format for saving the guest state is tied to the ABI of internal bhyve structures and is not self-describing (in that it does not communicate the set of device models present in the system). In addition, the state saved for some device models closely matches the internal data structures which might prove a challenge for compatibility of snapshot files across a range of bhyve versions." The UPB patch addresses the above. Vitaliy's patch does nothing to address any of it. If one is going to be proposed as an alternative to the other, it better solve the same problems as then some. > I don't want give unhelpful or contradictory suggestions to what you've > received in the past. > The best I can do at this point is defer to those who you have made > these agreements with. > The silence is real. -Matthew