From nobody Thu Jun 1 00:00:28 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QWmVP2Xplz4Xrpm; Thu, 1 Jun 2023 00:00:29 +0000 (UTC) (envelope-from salvadore@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [96.47.72.132]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QWmVP0vgyz463j; Thu, 1 Jun 2023 00:00:29 +0000 (UTC) (envelope-from salvadore@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1685577629; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc; bh=BZDboLs55RS+2CA6Wp814zfKjcwV1ra3lerb9u3JLZI=; b=Ug/wWhRwqGQFAuUrgwT3oBoXaLiPDRtp/L0co+p9rSLgc1OuUUi1Rk030TAFda7iMn2Yug iJMDxXoqpmT9Qa7Y5ZjzLruc33cKDxgYybAyeriFCJ3eJ0eiQPoYkrwgnY5wtWOZZBnea4 FX48Kcp9LhsVy6QgZ+Q7s77Ihc9jlQ7dz6ZfLKFcTfQXCWELM6T5kN8fY7fbdAZbwJFOV5 nMUBwWKYLV+M1Bw6F3J/6qC4+p0cYvWapYv6eJ6ipx3zLkL4wh7U/XMD78YBjdDzPVPRDW vVmLueUvbAhvyxTLspbHVSbpZn0gM64GHRfWQcAgHCL4Nr+RJfecPFekSxNvmA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1685577629; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc; bh=BZDboLs55RS+2CA6Wp814zfKjcwV1ra3lerb9u3JLZI=; b=R7JnTuWhx4x3bxn4CNmmNJxR2taz2FgYJVFxTXvzJs0SR9yc4vfcOcMgQxuObXv05S9cHw BXVJBIFC1TEEgIxkD3ldJKiw5oSmd0sZ1IoFNC0bxsa+8A/n28TUbWtPNEUp8fJZrQvPM8 1o9W56o/ZGg8TkfzNbkqPzl3POUhsutkWlZWMVW73Se897S8rtbJPVPZrU5NyeQPQ4L4eM VGkYEdbQk2jXfgvoenDcbnrQgvKt44BXUSVOBC7NH2znQeWBuCKfZrYb5hPHy312AqWssW 2HLjPPX11bodErlYhTA/DTzXGHSCznxFks24iDWFmwvuK8+VUCmAxXXvkxem9w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1685577629; a=rsa-sha256; cv=none; b=EwsPCVL7RZ9XMLH2wclnuyOtYcTfM+tZ4ol+6MRabMzHHbWjPZ7D9R1HS6MUHFqOT6eon9 6gB/fyPpwgiFDPPQtUiMDgdtrtI2Cjw3UqmX7XgA6Fpp0gQADoSQoH9NiV5SuET97lBvHR RtlqegzxByvtrB2uFFRD8wzjEKE6crmsFC8ia83lwjAghQkl0NoSk+CTAjYGW/2RMt6Bf/ lTr82pAgJm+603wng543NImaoWjhXCGKkt87wHuES8LigoyAJ3V6BJ++25/Kg+lvoL1ZQM Z9PKErBSoDTqkL4ILt4byP2FPSs2WTNEMFtgifb5UKFy11WWRSltythk/A7Z2Q== Received: by freefall.freebsd.org (Postfix, from userid 1472) id D980B91B8; Thu, 1 Jun 2023 00:00:28 +0000 (UTC) To: freebsd-status-calls@FreeBSD.org Subject: Call for 2023Q2 status reports Cc: freebsd-current@FreeBSD.org,freebsd-hackers@FreeBSD.org,devsummit@FreeBSD.org,info@bsdcan.org,soc-students@FreeBSD.org,soc-mentors@FreeBSD.org Message-Id: <20230601000028.D980B91B8@freefall.freebsd.org> Date: Thu, 1 Jun 2023 00:00:28 +0000 (UTC) From: Lorenzo Salvadore X-ThisMailContainsUnwantedMimeParts: N List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Dear FreeBSD Community, The deadline for the next FreeBSD Status Report update is June, 30th 2023 for work done since the last round of quarterly reports: April 2023 - June 2023. I would like to remind you that reports are published on a quarterly basis and are usually collected during the last month of each quarter, You are also welcome to submit them even earlier if you want, and the earlier you submit them, the more time we have for reviewing. Status report submissions do not need to be very long. They may be about anything happening in the FreeBSD project and community, and they provide a great way to inform FreeBSD users and developers about work that is underway or has been completed. Report submissions are not limited to committers; anyone doing anything interesting and FreeBSD related can -- and should -- write one! The following methods are available to submit your reports: * submit a review on Phabricator and add the group "status" to the reviewers list. You should put your reports in the directory doc/website/content/en/status/report-2023-04-2023-06/ (create it if it is missing); * submit a pull request at . You should put your reports in the directory doc/website/content/en/status/report-2023-04-2023-06/ (create it if it is missing); * send an email to status-submissions@FreeBSD.org including your report. An AsciiDoc template is available at . We look forward to seeing your 2023Q2 reports! Thanks, Lorenzo Salvadore (on behalf of status@) From nobody Mon Jun 5 15:32:57 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QZd0X4hWcz4Z1W1; Mon, 5 Jun 2023 15:33:00 +0000 (UTC) (envelope-from corvink@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QZd0X2LmPz4CX2; Mon, 5 Jun 2023 15:33:00 +0000 (UTC) (envelope-from corvink@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1685979180; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=ffG4GMn37fvx7VjU6P5pVAoWbJtSPNujPmA51loml3I=; b=rIlN9HkFMNIfEiuuNY7javycRa6p5Us+UsVxdKDgXRhcQQiKJ0PJcWDmFjDJxQz2ncz4fH C+wiuKA6HQ6HNauxlCiOKmtm2IwjMh6buhwNGRQOdDxKm4RJSScOz17eaVJ0Uqts6gKHBv ZeHXuM1TJbmab72zCxcbt0utIkp6wdqGnwvrPQoY2f8glnHtRYEX3ICSuqnzLsHbbOf9R0 iX5UwVFqDa1HSHxsyTicyMocaSj9hh9LgVw+osABtLz3t73RNGquQk0+6uKHkLbgc64aHl xRKt0aAH+nqItj/0ehrNZ09jT62eYwsO+PiAOJCWSawkKM7lXcR8VEro26TvWA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1685979180; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=ffG4GMn37fvx7VjU6P5pVAoWbJtSPNujPmA51loml3I=; b=M0vP6SSgJYSNRs34+G6bKSUzq7AYFd2/ZxLlL3hPaU9xq2aDZ3AfyMiKWrGApkO0VYs9mT BVN5m4PUvY3Pr1vghhCMUJNx8P2mWjy8p/D7EM/XWE8vi6xloz+v2FrTNG2UZNDk2GnmEc UEGpNsIlBkUdErI+TmJhatS2oVHBT5oRcagTa5etn+pVaQAZECUfv8ZjsoqtHsbCSumEDX axUnmgd8ECL7XUqjB6os7R+IF3efkr6K9IsqLxCq8FsN8f6tJKu13S8bRtOQU7QmIVxseT kvJ8n29KI8Vi/0u/RjLJIo8gRvCUHtOgGST2DYtNOoR9KBmHHo16nDgSxkdOiw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1685979180; a=rsa-sha256; cv=none; b=KZ1ppNh3jeTcmfsWhxeSdnzRIzRAeX2FbnowENOHQtW77KgECiuTgfdsgoW2KyT/fw8Kzq nmDB0Z2B3FTNSPqaDoLLcXh1aKNRUo80mRncjccmXMB98klLepVzCKAkUCtvyJLY3mHEm1 EgSE0jnJyNkeqjrw2XOd34SfPb3qWd1SOwT3OAWnzWIwccK6FRcBzxcBZhmWSfON6AWOgR ppjNiOqeXZ8O5sfFIykeKIbJbEkFX6BOvQP6GyVa4LMRln2bjd+pj31fOxyrekrK6nsUR+ nQy0H6KbJ5cqXW7klZLzAstQZYqIqsrGCxvbzMjU9I16Y+TYQtTDcP58+ycDIw== Received: from [IPv6:2001:9e8:da74:9600:33d3:868a:bb07:336] (unknown [IPv6:2001:9e8:da74:9600:33d3:868a:bb07:336]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: corvink) by smtp.freebsd.org (Postfix) with ESMTPSA id 4QZd0W5GvSzKlW; Mon, 5 Jun 2023 15:32:59 +0000 (UTC) (envelope-from corvink@FreeBSD.org) Message-ID: <6b98da58a5bd8e83bc466efa20b5a900298210aa.camel@FreeBSD.org> Subject: Re: BHYVE SNAPSHOT image format proposal From: Corvin =?ISO-8859-1?Q?K=F6hne?= To: Vitaliy Gusev , virtualization@freebsd.org Cc: freebsd-hackers@freebsd.org Date: Mon, 05 Jun 2023 17:32:57 +0200 In-Reply-To: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> References: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> Content-Type: multipart/signed; micalg="pgp-sha256"; protocol="application/pgp-signature"; boundary="=-fIhsp7U1xeUiBpku0+0r" User-Agent: Evolution 3.46.4 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N --=-fIhsp7U1xeUiBpku0+0r Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: base64 T24gVHVlLCAyMDIzLTA1LTIzIGF0IDE5OjA1ICswMzAwLCBWaXRhbGl5IEd1c2V2IHdyb3RlOgo+ IEhpLAo+IAo+IEhlcmUgaXMgYSBwcm9wb3NhbCBmb3IgYmh5dmUgc25hcHNob3QvY2hlY2twb2lu dCBpbWFnZSBmb3JtYXQKPiBpbXByb3ZlbWVudHMuCj4gCgpIaSBWaXRhbGl5LAoKdGhhbmsgeW91 IHZlcnkgbXVjaCBmb3Igc2VuZGluZyB0aGlzIHByb3Bvc2FsIHRvIHRoZSBtYWlsaW5nIGxpc3Qu Cgo+IEl0IGltcGxpZXMgbW92aW5nIHNuYXBzaG90IGNvZGUgdG8gbnZsaXN0IGVuZ2luZS7CoAo+ IAo+IEN1cnJlbnQgc25hcHNob3QgaW1wbGVtZW50YXRpb24gaGFzIGRpc2FkdmFudGFnZXM6Cj4g Cj4gwqAqIDMgZmlsZXMgcGVyIHNuYXBzaG90OiAubWV0YSwgLmtlcm4sIHZyYW0KPiDCoCogQmlu YXJ5IFN0cmVhbSBmb3JtYXQgb2YgZGF0YS4KPiDCoCogQWRkaW5nIMKgb3B0aW9uYWwgdmFyaWFi bGUgLSBicmVha3MgcmVzdW1lCj4gwqAqIFJlbW92aW5nIHZhcmlhYmxlIC0gYnJlYWtzIHJlc3Vt ZQo+IMKgKiBDaGFuZ2luZyBzYXZlZCBvcmRlciBvZiB2YXJpYWJsZXMgLSBicmVha3MgcmVzdW1l Cj4gwqAqIEhhcmQgdG8gZ2V0IGluZm9ybWF0aW9uIGFib3V0IHdoYXQgaXMgc2F2ZWQgYW5kIGRl Y29kZS4KPiDCoCogSGFyZCB0byBkZWJ1ZyBpZiBzb21ldGhpbmdzIGdvZXMgd3JvbmcKPiDCoCog Tm8gdmVyc2lvbnMuIElmIGNoYW5nZSBjb2RlLCByZXN1bWUgb2YgYW4gb2xkIGltYWdlcyBjYW4g YmUKPiDCoMKgwqBwYXNzZWQsIGJ1dCB3aXRoIFVCLgo+IAo+IE5ldyBudmxpc3QgaW1wbGVtZW50 YXRpb24gc2hvdWxkIHNvbHZlIGFsbCB0aGluZ3MgYWJvdmUuIFRoZSBmaXJzdAo+IHN0ZXAgLQo+ IGltcHJvdmUgc25hcHNob3QvY2hlY2twb2ludCBzYXZpbmcgZm9ybWF0LiBJdCBlbGltaW5hdGVz IHRocmVlIGZpbGVzCj4gdXNhZ2UKPiBwZXIgYSBzbmFwc2hvdC4KPiAKPiAKPiAxLiBCSFlWRSBT TkFQU0hPVCBpbWFnZSBmb3JtYXQ6IMKgCj4gCj4gPiAr4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU 4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU 4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCUKwo+ID4gfCDCoCDCoCDCoEhFQURF UiBQSFlTIC0gNDA5NiBCWVRFUyDCoCDCoCDCoCDCoCB8Cj4gPiAr4oCU4oCU4oCU4oCU4oCU4oCU 4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU 4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCUKwo+ID4gfCDCoCDCoCDC oCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCB8Cj4gPiB8 IMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgREFUQSDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCB8 Cj4gPiB8IMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKg IMKgIMKgIHwKPiA+ICvigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTi gJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTi gJTigJTigJTigJTigJTigJQrCj4gCj4gMi4gSEVBREVSIFBIWVMgZm9ybWF0OsKgCj4gCj4gwqAw IMKgIMKgK+KAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKA lOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKA lOKAlOKAlOKAlOKAlOKAlCvCoAo+IMKgIMKgIMKgIHwgwqAgwqAgwqAgwqBJREVOVCBTVFJJTkcg wqAtIDY0IEJZVEVTIMKgIMKgIMKgIMKgIHwKPiDCoDY0IMKgICvigJTigJTigJTigJTigJTigJTi gJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTi gJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJQrIMKgwqAKPiDC oCDCoCDCoCB8IE5WTElTVCBTSVpFIMKgLSA0IEJZVEVTIMKgIHwgwqBOVkxJU1QgREFUQSB8Cj4g wqA3MiDCoCAr4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU 4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU 4oCU4oCU4oCU4oCU4oCU4oCUKwo+IMKgIMKgIMKgIHwgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAg wqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgfAo+IMKgIMKgIMKgIHwgwqAgwqAg wqAgwqAgwqAgwqAgwqAgTlZMSVNUIERBVEEgwqAgwqAgwqAgwqAgwqAgwqAgwqAgfAo+IMKgIMKg IMKgIHwgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAg wqAgwqAgwqAgfAo+IMKgNDA5NiAr4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU 4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU 4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCUKwo+IAo+ID4gCj4gPiBJREVOVCBTVFJJTkcg LSBFYWNoIHByb2R1Y2VyIGNhbiBzZXQgaXRzIG93biB2YWx1ZSB0byBzcGVjaWZ5Cj4gPiBpbWFn ZS4KPiA+IE5WTElTVCBTSVpFIMKgLSBUaGUgZm9sbG93aW5nIHBhY2tlZCBoZWFkZXIgbnZsaXN0 IGRhdGEgc2l6ZS4KPiA+IE5WTElTVCBEQVRBIC0gUGFja2VkIG52bGlzdCBoZWFkZXIgZGF0YS4K PiA+IAo+ID4gNEtCIHNob3VsZCBiZSBlbm91Z2ggZm9yIHRoZSBIRUFERVIgdG8ga2VlcCBiYXNp YyBpbmZvcm1hdGlvbiBhYm91dAo+ID4gU2VjdGlvbnMuIEhvd2V2ZXIsIGl0IGNhbgo+ID4gYmUg ZW5sYXJnZWQgbGF0ZWx5LCB3aXRob3V0IGJyZWFraW5nIGJhY2t3YXJkIGNvbXBhdGliaWxpdHku wqAKPiA+IAoKSSBjYW4ndCBzZWUgYW4gYWR2YW50YWdlIG9mIHVzaW5nIGEgZml4ZWQgc2l6ZWQg aGVhZGVyIG9mIDRLQi4gWW91IGhhdmUKdG8gcGFyc2UgdGhlIG9mZnNldCBhbmQgc2l6ZSBvZiBl dmVyeSBzZWN0aW9uIGFueXdheXMuIElmIGl0J3MgZm9yCmFsaWdubWVudCByZXF1aXJlbWVudHMg eW91IGNhbiBzdGlsbCBhbGlnbiBhbGwgc2VjdGlvbnMgb24gc2F2ZSBhbmQgc2V0CnRoZSBvZmZz ZXQgYWNjb3JkaW5nbHkuIFNvLCB3aHkgY29tcGxpY2F0aW5nIHRoaW5ncyBieSB1c2luZyBhIGZp eGVkCmhlYWRlciBzaXplPwoKVGhlIElERU5UIFNUUklORyBzZWVtcyB0byBiZSB2ZXJ5IGxhcmdl LiBFdmVuIGEgR1VJRCB3aGljaCBzaG91bGQgYmUgYQpnbG9iYWwgdW5pcXVlIGlkZW50aWZpZXIg dXNlcyBqdXN0IDE2IEJ5dGVzLiBBZGRpdGlvbmFsbHksIGl0IG1pZ2h0IGJlCndvcnRoIHVzaW5n IGEgZGVkaWNhdGVkIGlkZW50IGFuZCB2ZXJzaW9uIGZpZWxkIGZvciBhbiBlYXNpZXIgdmVyc2lv bgpwYXJzaW5nLiBFLmcuOgoKKy0tLS0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0tLS0t Kwp8IElERU5UIC0gNTYgQllURVMgfCBWRVJTSU9OIC0gOCBCWVRFUyB8CistLS0tLS0tLS0tLS0t LS0tLS0rLS0tLS0tLS0tLS0tLS0tLS0tLSsKCklERU5UIC0gIkJIWVZFIENIRUNLUE9JTlQgSU1B R0UiClZFUlNJT04gLSAxIChhcyB1aW50NjRfdCkKCkJ0dzogSSBkb24ndCBjYXJlIGJ1dCBoZXJl IHdlIGNvdWxkIGxlYXZlIHNvbWUgZnJlZSBzcGFjZSBmb3IgcG9zc2libGUKZm9yd2FyZCBjb21w YXRpYmlsaXR5LiBFLmcuOgoKKy0tLS0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0tLS0t Ky0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rCnwgSURFTlQgLSAxNiBCWVRFUyB8IFZFUlNJT04g LSA4IEJZVEVTIHwgX0ZSRUVfU1BBQ0VfIC0gNDAgQllURVMgfAorLS0tLS0tLS0tLS0tLS0tLS0t Ky0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSsKCj4gMy4gTlZM SVNUIEhFQURFUsKgY29uc2lzdHMgb2YgU2VjdGlvbnMgaW4gdGhlIGZvbGxvd2luZyBmb3JtYXQ6 Cj4gCj4gwqAqIE5hbWUgLSBzdHJpbmcKPiDCoCogVHlwZTogwqAgwqBzdHJpbmc6Cj4gwqDCoMKg wqAtIOKAnHRleHQsIMKgIC0gcGxhaW4gdGV4dCwKPiDCoMKgwqDCoC0g4oCcbnZsaXN04oCdIC0g cGFja2VkIG52bGlzdCwKPiDCoMKgwqDCoC0g4oCcYmluYXJ54oCdIC0gcmF3IGJpbmFyeSBkYXRh Lgo+IMKgKiBTaXplIC0gU2l6ZSBvZiBzZWN0aW9uIC0gdWludDY0Cj4gwqAqIE9mZnNldCAtIE9m ZnNldCBpbiBpbWFnZSBmb3JtYXQgLSB1aW50NjQKPiA+IAo+IMKgIMKgIFByZWRlZmluZWQgc2Vj dGlvbnM6IMKg4oCcY29uZmln4oCdLCDigJxkZXZpY2Vz4oCdLCDigJxrZXJuZWzigJ0sIOKAnG1l bW9yeeKAnS7CoAo+IAoKTEdUTS4KCj4gCj4gNC4gRVhBTVBMRToKPiAKPiAKPiDCoElERU5UIFNU UklORzoKPiAKPiDCoCDCoCDCoCDCoCJCSFlWRSBDSEVDS1BPSU5UIElNQUdFIFZFUlNJT04gMSIK PiAKPiDCoE5WTElTVCBIRUFERVI6wqAKPiAKPiDCoCBbY29uZmlnXQo+IMKgIMKgIMKgIMKgIGNv bmZpZy5vZmZzZXQgPSAweDEwMDAgKDQwOTYpCj4gwqAgwqAgwqAgwqAgY29uZmlnLnNpemUgPSAw eDFmNiAoNTAyKQo+IMKgIMKgIMKgIMKgIGNvbmZpZy50eXBlID0gInRleHQiCj4gwqAgW2tlcm5l bF0KPiDCoCDCoCDCoCDCoCBrZXJuZWwub2Zmc2V0ID0gMHgxMWY2ICg0NTk4KQo+IMKgIMKgIMKg IMKgIGtlcm5lbC5zaXplID0gMHgxOWE3ICg2NTY3KQo+IMKgIMKgIMKgIMKgIGtlcm5lbC50eXBl ID0g4oCcbnZsaXN0Igo+IMKgIFtkZXZpY2VzXQo+IMKgIMKgIMKgIMKgIGRldmljZXMub2Zmc2V0 ID0gMHgyYjlkICgxMTE2NSkKPiDCoCDCoCDCoCDCoCBkZXZpY2VzLnNpemUgPSAweDEwMTQ1YmEg KDE2ODYwNjAyKQo+IMKgIMKgIMKgIMKgIGRldmljZXMudHlwZSA9ICJudmxpc3QiCj4gwqAgW21l bW9yeV0KPiDCoCDCoCDCoCDCoCBtZW1vcnkub2Zmc2V0ID0gMHgxMjAwMDAwICgxODg3NDM2OCkK PiDCoCDCoCDCoCDCoCBtZW1vcnkuc2l6ZSA9IDB4M2NlMDAwMDAgKDEwMjEzMTMwMjQpCj4gwqAg wqAgwqAgwqAgbWVtb3J5LnR5cGUgPSDigJxiaW5hcnkiCj4gCj4gwqBTRUNUSU9OUzoKPiAKPiDC oFtzZWN0aW9uICJjb25maWciIHNpemUgMHgxZjYgb2Zmc2V0IDB4MTAwMF06Cj4gPiBtZW1vcnku c2l6ZT0xMDI0TQo+ID4geDg2LnN0cmljdG1zcj10cnVlCj4gPiB4ODYudm1leGl0X29uX2hsdD10 cnVlCj4gPiBjcHVzPTIKPiA+IGFjcGlfdGFibGVzPXRydWUKPiA+IHBjaS4wLjAuMC5kZXZpY2U9 aG9zdGJyaWRnZQo+ID4gcGNpLjAuMzEuMC5kZXZpY2U9bHBjCj4gPiBwY2kuMC40LjAuZGV2aWNl PXZpcnRpby1uZXQKPiA+IHBjaS4wLjQuMC5iYWNrZW5kPXRhcDAKPiA+IHBjaS4wLjcuMC5kZXZp Y2U9ZmJ1Zgo+ID4gcGNpLjAuNy4wLnRjcD0xMC40Mi4wLjc4OjU5MDAKPiA+IHBjaS4wLjcuMC53 PTEwMjQKPiA+IHBjaS4wLjcuMC5oPTc2OAo+ID4gcGNpLjAuNS4wLmRldmljZT1haGNpCj4gPiBw Y2kuMC41LjAucG9ydC4wLnR5cGU9Y2QKPiA+IHBjaS4wLjUuMC5wb3J0LjAucGF0aD0vSVNPL3Vi dW50dS0yMi4wNC4xLWxpdmUtc2VydmVyLWFtZDY0Lmlzbwo+ID4gbHBjLmJvb3Ryb209L3Vzci9s b2NhbC9zaGFyZS91ZWZpLWZpcm13YXJlL0JIWVZFX1VFRkkuZmQKPiA+IGNoZWNrcG9pbnQuZGF0 ZT0iV2VkIEphbiAyNSAyMzo0ODoyOSAyMDIzIgo+ID4gbmFtZT11YnVudHUyMgo+IAoKTm90IHN1 cmUgaWYgaXQncyBqdXN0IGFuIGV4YW1wbGUgZm9yIHRoZSAidGV4dCIgdHlwZS4gYmh5dmUgY29u dmVydHMgaXQKaW50byBhIG52bGlzdCwgc28gaXQgY291bGQgYmUgc2F2ZWQgZGlyZWN0bHkgYXMg bnZsaXN0LgoKQnR3OiBJIHdvdWxkIG9ubHkgaW1wbGVtZW50IHRoZSAidGV4dCIgdHlwZSBpZiB0 aGVyZSdzIGFuIHVzZWNhc2UgdGhhdApjYW4ndCBiZSBzb2x2ZWQgYnkgb25lIG9mIHRoZSBvdGhl ciB0eXBlcy4KCj4gwqBbc2VjdGlvbiAia2VybmVsIiBzaXplIDB4MTlhNyBvZmZzZXQgMHgxMWY2 XToKPiDCoCDCoFt2bV0KPiDCoCDCoCDCoCDCoCB2bS52ZHNfdmVyc2lvbiA9IDB4MSAoMSkKPiDC oCDCoCDCoCDCoCB2bS5jcHUwLmRhdGEoQklOQVJZKTogMDAgMDAgMDAgMDAgMEQgMDAgMDAgMDAg MDEgMDAgMDAgMDAgMDAKPiAwMCAwMCAwMCAuLi7CoCBzaXplPTB4MjgKPiDCoCDCoCDCoCDCoCB2 bS5jcHUxLmRhdGEoQklOQVJZKTogMDAgMDAgMDAgMDAgMEQgMDAgMDAgMDAgMDEgMDAgMDAgMDAg MDAKPiAwMCAwMCAwMCAuLi7CoCBzaXplPTB4MjgKPiDCoCDCoCDCoCDCoCB2bS5jaGVja3BvaW50 X3RzYyA9IDB4ZTJlMGFjNmZiZTQ1NiAoMzk5MTI3MzQ5Njg5NjU5OCkKPiDCoCDCoFtocGV0XQo+ IMKgIMKgIMKgIMKgIGhwZXQudmRzX3ZlcnNpb24gPSAweDEgKDEpCj4gwqAgwqAgwqAgwqAgaHBl dC5kYXRhKEJJTkFSWSk6IDAxIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAw Cj4gMDAgMDAgLi4uwqAgc2l6ZT0weDExOAo+IMKgIMKgW3ZteF0KPiDCoCDCoCDCoCDCoCB2bXgu dmRzX3ZlcnNpb24gPSAweDEgKDEpCj4gwqAgwqAgwqAgwqAgdm14LmNwdV9mZWF0dXJlcyA9IDAg KDApCj4gwqAgwqAgwqAgwqAgdm14LmNwdTAudm14X2RhdGEoQklOQVJZKTogRjAgQ0MgMTUgQjgg RkYgRkYgRkYgRkYgNDAgQjQgMjEKPiBCOSBGRiBGRiBGRiBGRiAuLi7CoCBzaXplPTB4Mjg4Cj4g wqAgwqAgwqAgwqAgdm14LmNwdTEudm14X2RhdGEoQklOQVJZKTogRjAgQ0MgMTUgQjggRkYgRkYg RkYgRkYgMDAgMDAgNjcKPiA0MSBEOCA5QiBGRiBGRiAuLi7CoCBzaXplPTB4Mjg4Cj4gwqAgwqBb aW9hcGljXQo+IMKgIMKgIMKgIMKgIGlvYXBpYy52ZHNfdmVyc2lvbiA9IDB4MSAoMSkKPiDCoCDC oCDCoCDCoCBpb2FwaWMuZGF0YShCSU5BUlkpOiAwMCAwMCAwMSAwMCAwMCAwMCAwMCAwMCAwMCAw MCAwMCAwMCAwMAo+IDAwIDAwIDAwIC4uLsKgIHNpemU9MHgyMDgKPiDCoCDCoFtsYXBpY10KPiDC oCDCoCDCoCDCoCBsYXBpYy52ZHNfdmVyc2lvbiA9IDB4MSAoMSkKPiDCoCDCoCDCoCDCoCBsYXBp Yy5jcHUwLmRhdGEoQklOQVJZKTogMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAK PiAwMCAwMCAwMCAwMCAuLi7CoCBzaXplPTB4NDYwCj4gwqAgwqAgwqAgwqAgbGFwaWMuY3B1MS5k YXRhKEJJTkFSWSk6IDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwCj4gMDAgMDAg MDAgMDAgLi4uwqAgc2l6ZT0weDQ2MAo+IMKgIMKgW2F0cGl0XQo+IMKgIMKgIMKgIMKgIGF0cGl0 LnZkc192ZXJzaW9uID0gMHgxICgxKQo+IMKgIMKgIMKgIMKgIGF0cGl0LmRhdGEoQklOQVJZKTog MDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgNTQgQUQgNTEgOTcgMEYgMEUKPiAwMCAwMCAuLi7CoCBz aXplPTB4YTAKPiDCoCDCoFthdHBpY10KPiDCoCDCoCDCoCDCoCBhdHBpYy52ZHNfdmVyc2lvbiA9 IDB4MSAoMSkKPiDCoCDCoCDCoCDCoCBhdHBpYy5kYXRhKEJJTkFSWSk6IDAxIDAwIDAwIDAwIDAw IDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAxIDAwCj4gMDAgMDAgLi4uwqAgc2l6ZT0weDg0Cj4gwqAg wqBbcG10aW1lcl0KPiDCoCDCoCDCoCDCoCBwbXRpbWVyLnZkc192ZXJzaW9uID0gMHgxICgxKQo+ IMKgIMKgIMKgIMKgIHBtdGltZXIudXB0aW1lID0gMHgyNmZkMTMzZTVjYyAoMjY3OTI3NDQ2NDcx NikKPiDCoCDCoFtydGNdCj4gwqAgwqAgwqAgwqAgcnRjLnZkc192ZXJzaW9uID0gMHgxICgxKQo+ IMKgIMKgIMKgIMKgIHJ0Yy5kYXRhKEJJTkFSWSk6IDBBIDAwIDAwIDAwIDAwIEZFIEZGIEZGIDEw IDM1IDEzIDNEIDQwIEY3Cj4gMTQgMDAgLi4uwqAgc2l6ZT0weDk4Cj4gCj4g4oCUCj4gVGhhbmtz LAo+IFZpdGFsaXkgR3VzZXYKPiAKCkFsbCBpbiBhbGwsIGl0IGxvb2tzIGdvb2QuIEtlZXAgb24g eW91ciB3b3JrIQoKUmVnYXJkcyBjaGVja3N1bSBmZWF0dXJlOgpXZSBzaG91bGQgZm9jdXMgb24g ZW5hYmxpbmcgdGhpcyBmZWF0dXJlIGJ5IGRlZmF1bHQgYmVmb3JlIGFkZGluZwphZHZhbmNlZCBm ZWF0dXJlcy4gU28sIGtlZXAgaXQgc2ltcGxlIGFuZCBzbWFsbC4KClJlZ2FyZHMgZm9yd2FyZCBj b21wYXRpYmlsaXR5OgpCYWNrd2FyZCBjb21wYXRpYmlsaXR5IGlzIHdheSBtb3JlIGltcG9ydGFu dCB0aGFuIGZvcndhcmQKY29tcGF0aWJpbGl0eS4gTmV2ZXJ0aGVsZXNzLCBmb3J3YXJkIGNvbXBh dGliaWxpdHkgd291bGQgYmUgbmljZSB0bwpoYXZlLiBTbywgd2Ugc2hvdWxkIGtlZXAgaXQgaW4g bWluZCB3aGVuIG1vZGlmeWluZyB0aGUgbGF5b3V0LiBGb3IgdGhlCm1vbWVudCwganVzdCBmb2N1 cyBvbiBhIGZvcm1hdCB3aGljaCBpcyBiYWNrd2FyZCBjb21wYXRpYmxlLgoKCi0tIApLaW5kIHJl Z2FyZHMsCkNvcnZpbgo= --=-fIhsp7U1xeUiBpku0+0r Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEgvRSla3m2t/H2U9G2FTaVjFeAmoFAmR+ACkACgkQ2FTaVjFe AmqHiA//Z+21VQdlu4imactssAT4+30yMrPULmDp98ncd7Z+jyixmXNa8+i3b0Wb XuTMBVERSGRxuaUjca2ChG98qlI3liLVgvRuvZqT2mi7VwGaQlAnMaEkL40mxq6M H+y9zUnOXKUGjTHS2nKCJflPihcCx/Nj0uif4WiEePCazmbp/KEOMXbrTlEIUNC/ 39OHjG0q2BPLp1P0c0eF4UJLXr8P0ieeeWDKWSSaLscJs59ON+H0cn8z28AbWxTk KBo3i4UwXS/Ei/VIVmM2I3Xg51CeCEXdlpg8vf+/MNqiQ4/xUvQdDBdbzgUVAmwG fhuKVds/DJa3vJPNHiS/EoLFOeyo5IXVQhowi2OVzDznwRXlmGcGHJ80/XDmHuZr +N6ekQJVOH/OUrNcsFlCWenWmWYeajIq9F2EWBOnnLy6jnieQ6qCC0Tflkn1wvIN FmS1qg2z7Y5OIxOKMOKfK8vckiaaAMrK76GehNjqBI3/7FzO7rMFqz1CcCPfkS9E atf2AaSCIFiBJpTFyj2HZwdlL7nKNh8d7raACyC3G3MRMZgCZ8Fhiyp7t+LL4Dfk F303H2zaFCo6lZR4GpRBsadmxo8QnXPw42eWLDy4kBP1WoA+TMbk8mnK/eONDvYz aYmTEOcJIdIylWK/S2p8bPs1ydeBCF9e6xbK7ErGBTbtnprVKcs= =oFeS -----END PGP SIGNATURE----- --=-fIhsp7U1xeUiBpku0+0r-- From nobody Tue Jun 6 11:25:38 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qb7Sx1vFJz4bD4K; Tue, 6 Jun 2023 11:25:53 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Received: from mail-lf1-x12b.google.com (mail-lf1-x12b.google.com [IPv6:2a00:1450:4864:20::12b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Qb7Sw3SKXz4NyC; Tue, 6 Jun 2023 11:25:52 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x12b.google.com with SMTP id 2adb3069b0e04-4f620583bc2so3609835e87.1; Tue, 06 Jun 2023 04:25:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1686050750; x=1688642750; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=S2earCf2oH9ASFpdicga292JWYO11HZaj/LPmJqHUuA=; b=Y8vaMWlxjye+0P23e8NX9LRfYL5iDZdwqlWM4Rs6Iius/lE8BEEhnloAoUDudu7zJ2 RcLGnNgVqoWWkWAX4p53qBMUIxlOvr8Zv+C+//ETqXs+zSNQZg0JyNU2Z+aRryyT7K4O xYiimRvNDrqhANnrk0Y9fWYfoHaSYZpLWwU2JJ3yShKJcQfalxqugWEhgajFLuSu7Lbi Ek1dxCqTlQNkfJOO/6/SwchevtFzcGul89Fw0EkCF4rBAi2/3auS0LmA5pq63NFVdnhy UPC2RTv47ZamiunLVZ3dj70EstLrAmjRag/SL/MZSufM/jNmpQK9y+XLCWpizD1zWZuu ldRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686050750; x=1688642750; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=S2earCf2oH9ASFpdicga292JWYO11HZaj/LPmJqHUuA=; b=O44EuG8mGCn0Rd/qaZyrp98zR+41Zt6APs6eWredB3A3ScBOMlihimO5AYx39Sno7y WoixrPhQohJOPZ7YBuxANmN8BjdyL3fFt/hGr9Pub2EtGwvvn2nIpeQSCZYFDIFCDW4z 1l7QLv6PREpPasx0DfVaGWR9eS58lJPmEFCu9/QJOwYwQ0HnyXCuKUh1OrIqNhCoXa5L tO2gx33pFqsovxnCCq/1e1afGvK2IFNMtOVSogg8heUWuB0nat7S8UjntTMAWqW09UQk 0dg/yo/q/PVQpTnAJosCNt57LzfAI6cWuUm6rc3+yY9iC4TGUtEtnKp7vIKpxU439gq2 h+pA== X-Gm-Message-State: AC+VfDyMZhVQDOX6Jp/pR3iXc9txnFikkKcYp73saHYV23DJsW4GDdsJ KreFcw4S9P2rPKpHztqHnvX+nl6iKADawA== X-Google-Smtp-Source: ACHHUZ7pmsR6XuIv6HuZQ11l5JE+B/PVTLBH+dgZMdOnDENJSK2FS+MrpDT9E/A60mrTOdZ+MkW16Q== X-Received: by 2002:ac2:44d7:0:b0:4f3:859c:a01d with SMTP id d23-20020ac244d7000000b004f3859ca01dmr796700lfm.69.1686050750023; Tue, 06 Jun 2023 04:25:50 -0700 (PDT) Received: from smtpclient.apple ([188.187.60.230]) by smtp.gmail.com with ESMTPSA id h18-20020a197012000000b004f382ae9892sm1445345lfc.247.2023.06.06.04.25.49 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 06 Jun 2023 04:25:49 -0700 (PDT) From: Vitaliy Gusev Message-Id: <8387AC83-6667-48E5-A3FA-11475EA96A5F@gmail.com> Content-Type: multipart/alternative; boundary="Apple-Mail=_A8750554-1230-4B80-AEEC-C03D927B7A32" List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.500.231\)) Subject: Re: BHYVE SNAPSHOT image format proposal Date: Tue, 6 Jun 2023 14:25:38 +0300 In-Reply-To: <6b98da58a5bd8e83bc466efa20b5a900298210aa.camel@FreeBSD.org> Cc: virtualization@freebsd.org, freebsd-hackers@freebsd.org To: =?utf-8?Q?Corvin_K=C3=B6hne?= References: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> <6b98da58a5bd8e83bc466efa20b5a900298210aa.camel@FreeBSD.org> X-Mailer: Apple Mail (2.3731.500.231) X-Rspamd-Queue-Id: 4Qb7Sw3SKXz4NyC X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_A8750554-1230-4B80-AEEC-C03D927B7A32 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi Corvin,=20 Thanks for your comments and advices.=20 Answers are below, > On 5 Jun 2023, at 18:32, Corvin K=C3=B6hne = wrote: >=20 > On Tue, 2023-05-23 at 19:05 +0300, Vitaliy Gusev wrote: >> 2. HEADER PHYS format:=20 >>=20 >> 0 = +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+=20= >> | IDENT STRING - 64 BYTES | >> 64 = +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+ =20= >> | NVLIST SIZE - 4 BYTES | NVLIST DATA | >> 72 = +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+ >> | | >> | NVLIST DATA | >> | | >> 4096 = +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+ >>=20 >>>=20 >>> IDENT STRING - Each producer can set its own value to specify >>> image. >>> NVLIST SIZE - The following packed header nvlist data size. >>> NVLIST DATA - Packed nvlist header data. >>>=20 >>> 4KB should be enough for the HEADER to keep basic information about >>> Sections. However, it can >>> be enlarged lately, without breaking backward compatibility.=20 >>>=20 >=20 > I can't see an advantage of using a fixed sized header of 4KB. You = have > to parse the offset and size of every section anyways. If it's for > alignment requirements you can still align all sections on save and = set > the offset accordingly. So, why complicating things by using a fixed > header size? You are right about 4KB restriction. I will correct it in updated format = proposal. Idea is to reserve enough space for HEADER and write it after all finished = stages at the beginning=20 of a snapshot file. Implementation (snapshot path) should know estimated maximum size of the = header and can use the possible maximum. Currently 4KB is enough and easily can be increased in the bhyve=E2=80=99s code without any problem.=20 Alignment is useful to debug and looking into snapshot image file. >=20 > The IDENT STRING seems to be very large. Even a GUID which should be a > global unique identifier uses just 16 Bytes. Additionally, it might be > worth using a dedicated ident and version field for an easier version > parsing. E.g.: Intention is to add enough space for the future version (as reservation) = and other producers and companies to specify it=E2=80=99s own ID string with possible add-on = information. So adding 64 bytes for the future is not so huge pay, but can be very useful. During resume, if IDENT string is not the same as in bhyve, resume can = fail before parsing other data, because it could be that internal format is not as expected. I would not to fix IDENT string format and just apply rule: During resume, bhyve compares its own IDENT string and IDENT string from = an Snapshot image. If it is not the same, further assumption about format = cannot be done, and resume should fail. >=20 > +------------------+-------------------+ > | IDENT - 56 BYTES | VERSION - 8 BYTES | > +------------------+-------------------+ >=20 > IDENT - "BHYVE CHECKPOINT IMAGE" > VERSION - 1 (as uint64_t) >=20 > Btw: I don't care but here we could leave some free space for possible > forward compatibility. E.g.: >=20 > +------------------+-------------------+-------------------------+ > | IDENT - 16 BYTES | VERSION - 8 BYTES | _FREE_SPACE_ - 40 BYTES | > = +------------------+-------------------+=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+ ... >> 4. EXAMPLE: >>=20 >>=20 >> IDENT STRING: >>=20 >> "BHYVE CHECKPOINT IMAGE VERSION 1" >>=20 >> NVLIST HEADER:=20 >>=20 >> [config] >> config.offset =3D 0x1000 (4096) >> config.size =3D 0x1f6 (502) >> config.type =3D "text" >>=20 >=20 > Not sure if it's just an example for the "text" type. bhyve converts = it > into a nvlist, so it could be saved directly as nvlist. > Btw: I would only implement the "text" type if there's an usecase that > can't be solved by one of the other types. Intention is to use current engine to dump bhyve=E2=80=99s config and = read config from a file (-k option). Advantage of using =E2=80=9Ctext=E2=80=9D type - simple implementation = and as an example of flexibility of proposed image format. Image file can keep any types = that a producer would like to use: text, nvlist, binary, diff-pages, etc. >=20 > All in all, it looks good. Keep on your work! >=20 > Regards checksum feature: > We should focus on enabling this feature by default before adding > advanced features. So, keep it simple and small. Could you give a more example what you meant about =E2=80=9Cchecksum=E2=80= =9D feature? Did you mean as TAR=E2=80=99s checksum, i.e. only header? >=20 > Regards forward compatibility: > Backward compatibility is way more important than forward > compatibility. Nevertheless, forward compatibility would be nice to > have. So, we should keep it in mind when modifying the layout. For the > moment, just focus on a format which is backward compatible. >=20 It seems that having information about forward compatibility could be = very useful, at least to get it in advance if it is impossible to restore. I = will add it during implementing this format. Thanks, Vitaliy Gusev --Apple-Mail=_A8750554-1230-4B80-AEEC-C03D927B7A32 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi = Corvin, 

Thanks for your comments and = advices. 

Answers are = below,

On 5 Jun 2023, at = 18:32, Corvin K=C3=B6hne <corvink@FreeBSD.org> wrote:

On Tue, 2023-05-23 at 19:05 +0300, Vitaliy = Gusev wrote:
2. HEADER PHYS = format: 

 0 =   =  +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= + 
      |       =  IDENT STRING  - 64 BYTES         = |
 64   = +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+ =   
      | NVLIST SIZE  - 4 = BYTES   |  NVLIST DATA |
 72   = +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+
=       | =                     =                     = |
      |       =         NVLIST DATA         =       |
      |       =                     =               |
 4096 = +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+
=

IDENT STRING - Each producer can set = its own value to specify
image.
NVLIST SIZE  - The following = packed header nvlist data size.
NVLIST DATA - Packed nvlist header = data.

4KB should be enough for the HEADER to keep basic = information about
Sections. However, it can
be enlarged lately, = without breaking backward = compatibility. 


I can't see an advantage of using a fixed = sized header of 4KB. You have
to = parse the offset and size of every section anyways. If it's = for
alignment requirements you can still align all sections on = save and set
the = offset accordingly. So, why complicating things by using a = fixed
header = size?

You are = right about 4KB restriction. I will correct it in updated format = proposal. Idea is
to reserve enough space for HEADER and write = it after all finished stages at the beginning 
of a = snapshot file.

Implementation (snapshot path) = should know estimated maximum size of the header and can
use = the possible maximum. Currently 4KB is enough and easily can = be
increased in the bhyve=E2=80=99s code without any = problem. 

Alignment is useful to debug and = looking into snapshot image file.


The = IDENT STRING seems to be very large. Even a GUID which should be = a
global unique identifier = uses just 16 Bytes. Additionally, it might be
worth using a dedicated ident and version = field for an easier version
parsing. E.g.:

Intention = is to add enough space for the future version (as reservation) and other = producers
and companies to specify it=E2=80=99s own ID string = with possible add-on information. So adding  64 bytes
for = the future is not so huge pay, but can be very = useful.

During resume, if IDENT string is not = the same as in bhyve, resume can fail before parsing
other = data, because it could be that internal format is not as = expected.

I would not to fix IDENT string = format and just apply = rule:

During resume, bhyve = compares its own IDENT string and IDENT string from = an
Snapshot image. If it is not the same, further assumption = about format cannot be done,
and resume should = fail.


+------------------+-------------------+
| IDENT - 56 BYTES | VERSION - 8 BYTES = |
+------------------+-------------------+

IDENT - "BHYVE CHECKPOINT IMAGE"
VERSION - 1 (as uint64_t)

Btw: I don't care but here we could leave = some free space for possible
forward = compatibility. E.g.:

+------------------+-------------------+---------------------= ----+
| IDENT = - 16 BYTES | VERSION - 8 BYTES | _FREE_SPACE_ - 40 BYTES |
+------------------+-------------------+=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94+
...
4. = EXAMPLE:


 IDENT STRING:

      =  "BHYVE CHECKPOINT IMAGE VERSION 1"

 NVLIST = HEADER: 

  [config]
    =     config.offset =3D 0x1000 = (4096)
        config.size =3D 0x1f6 = (502)
        config.type =3D = "text"


Not = sure if it's just an example for the "text" type. bhyve converts = it
into a nvlist, so it = could be saved directly as nvlist.
Btw: I would only implement the "text" type = if there's an usecase that
can't = be solved by one of the other types.


Intentio= n is to use current engine to dump bhyve=E2=80=99s config and read = config
from a file (-k = option).

Advantage of using =E2=80=9Ctext=E2=80=9D= type - simple implementation and as an example
of flexibility = of proposed image format. Image file can keep any types that
a = producer would like to use: text, nvlist, binary, diff-pages, = etc.


All in all, it looks good. Keep on your = work!

Regards = checksum feature:
We = should focus on enabling this feature by default before adding
advanced features. So, keep it simple and = small.

Could you = give a more example what you meant about =E2=80=9Cchecksum=E2=80=9D = feature? Did you mean as
TAR=E2=80=99s checksum, i.e. only = header?



Regards forward compatibility:
Backward compatibility is way more = important than forward
compatibility. Nevertheless, forward compatibility would be = nice to
have. = So, we should keep it in mind when modifying the layout. For = the
moment, = just focus on a format which is backward compatible.


It seems that having = information about forward compatibility could be very
useful, = at least to get it in advance if it is impossible to restore. I will add = it during
implementing this = format.

Thanks,
Vitaliy = Gusev

= --Apple-Mail=_A8750554-1230-4B80-AEEC-C03D927B7A32-- From nobody Tue Jun 6 12:59:53 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qb9YS1bRxz4bW1j; Tue, 6 Jun 2023 12:59:56 +0000 (UTC) (envelope-from corvink@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Qb9YS0nmmz3Mvp; Tue, 6 Jun 2023 12:59:56 +0000 (UTC) (envelope-from corvink@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1686056396; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=1vN98Fh3Nq2DBVmYQPjrAHHKOWlDEDcMwncl7e2Jg5Y=; b=f6CUoyc/OBbnFvWWfC2+fAvDmqls7qX4SkVfUTk+U4hBXwc/15X329WztJu9yAEaDh1MMH SSsBTmnoScxTIuQS5wJX96L8Z9IbkUHO4pfyjiBmHctxwlW1GJWFg66oIRT1VqTrgLqdWC 78zBbo2p0b0DRD7B42Muzdy3xdUu+xcAG2YmK74SHw6/qWVOsy6lCq0eRhB9wOSxHwJqm9 ZMBLPG6RgbFTMkgRMa76/X4T7Lm50qdV0EHkD9AHEjjTeqOQlf7X5UTBQIox4RpSoIigm+ WzcwaxhMbrhTGlgdBjx35NQx6TJZmw6e5WCzSQlKGC9bhcs/ackDaPHnE/xSxg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1686056396; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=1vN98Fh3Nq2DBVmYQPjrAHHKOWlDEDcMwncl7e2Jg5Y=; b=REPd6aNXwtXL2nCWhc2ISveRmzhFfkumKspLqqGGg1TqwZY11ZLvCFyDELcu0cUFgQTmPv ko/hwCHag+4HQf939ly8uHxGh2hUlsFRZgQRt/w90X7eiV3dv9UdAB/ndMr6ZL/wCkBXOX eiq8y3p3j/gL1wd0/ET5gq7Mq/MMGHZnAoH7T0EozI2uCDr/tSLwpu5PoBro+HJGEbLetE QpCr69VhnRCHBAhkjWAjD+6wM87Je0KAACCzoolA7yH2GBS4p6TR1+jUB01ACHIlyFxWjf iXa5HwzZchyfkmN0OCDosIgDlgoyJgy+V5O+VCZVQhuR+B/4TBEx5lPZdrDbKw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1686056396; a=rsa-sha256; cv=none; b=sb0gn9VFSOZvnSFshPQqEg+Y1CLknJtcWZNR9a+ORKBbTYg8eaSqrCUKcD3dEW4s1oFS6E FskZkZHd13R6Y7D1bVbJRahw9MqketO3X16k9zVnh/0lkxpb+bbnggX3rqFIB5WS3FbHNx wUqZa5ZLCCfNiYECKx1HB1BrAwSqwNAbSKQgVwPwl5R9ePwP1uevrJVGpo8POpqqhb14O5 uQ+tu/9a1Zqn/oX03MvfoEpPrJPfRU/EjbT4xN3als+f315ICP+5+e3cCsvu17VCe+ZhwG MBE6pvpqps+UdgFmlRPVGkc8Zyy4mUjRIxkkDaYbnmbRYSXb4HmgUoEWrEjCgA== Received: from [IPv6:2001:9e8:da59:8e00:1c7e:7163:67b2:a7d8] (unknown [IPv6:2001:9e8:da59:8e00:1c7e:7163:67b2:a7d8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: corvink) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Qb9YR3hydzkrr; Tue, 6 Jun 2023 12:59:55 +0000 (UTC) (envelope-from corvink@FreeBSD.org) Message-ID: Subject: Re: BHYVE SNAPSHOT image format proposal From: Corvin =?ISO-8859-1?Q?K=F6hne?= To: Vitaliy Gusev Cc: virtualization@freebsd.org, freebsd-hackers@freebsd.org Date: Tue, 06 Jun 2023 14:59:53 +0200 In-Reply-To: <8387AC83-6667-48E5-A3FA-11475EA96A5F@gmail.com> References: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> <6b98da58a5bd8e83bc466efa20b5a900298210aa.camel@FreeBSD.org> <8387AC83-6667-48E5-A3FA-11475EA96A5F@gmail.com> Content-Type: multipart/signed; micalg="pgp-sha256"; protocol="application/pgp-signature"; boundary="=-EnCgH2AdnOZbtGH9VMEd" User-Agent: Evolution 3.46.4 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N --=-EnCgH2AdnOZbtGH9VMEd Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Vitaliy, thanks for your answers. See comments below. On Tue, 2023-06-06 at 14:25 +0300, Vitaliy Gusev wrote: > Hi Corvin,=C2=A0 >=20 > Thanks for your comments and advices.=C2=A0 >=20 > Answers are below, >=20 > > On 5 Jun 2023, at 18:32, Corvin K=C3=B6hne wrote: > >=20 > > On Tue, 2023-05-23 at 19:05 +0300, Vitaliy Gusev wrote: > > > 2. HEADER PHYS format:=C2=A0 > > >=20 > > > =C2=A00 =C2=A0 =C2=A0+=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94+=C2=A0 > > > =C2=A0 =C2=A0 =C2=A0=C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0IDENT STRING = =C2=A0- 64 BYTES =C2=A0 =C2=A0 =C2=A0 =C2=A0 | > > > =C2=A064 =C2=A0 +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94+ =C2=A0=C2=A0 > > > =C2=A0 =C2=A0 =C2=A0=C2=A0| NVLIST SIZE =C2=A0- 4 BYTES =C2=A0 | =C2= =A0NVLIST DATA | > > > =C2=A072 =C2=A0 +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94+ > > > =C2=A0 =C2=A0 =C2=A0=C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 | > > > =C2=A0 =C2=A0 =C2=A0=C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 NVLIST DATA =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 | > > > =C2=A0 =C2=A0 =C2=A0=C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 | > > > =C2=A04096 +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94+ > > >=20 > > > >=20 > > > > IDENT STRING - Each producer can set its own value to specify > > > > image. > > > > NVLIST SIZE =C2=A0- The following packed header nvlist data size. > > > > NVLIST DATA - Packed nvlist header data. > > > >=20 > > > > 4KB should be enough for the HEADER to keep basic information > > > > about > > > > Sections. However, it can > > > > be enlarged lately, without breaking backward compatibility.=C2=A0 > > > >=20 > >=20 > > I can't see an advantage of using a fixed sized header of 4KB. You > > have > > to parse the offset and size of every section anyways. If it's for > > alignment requirements you can still align all sections on save and > > set > > the offset accordingly. So, why complicating things by using a > > fixed > > header size? >=20 >=20 > You are right about 4KB restriction. I will correct it in updated > format proposal. Idea is > to reserve enough space for HEADER and write it after all finished > stages at the beginning=C2=A0 > of a snapshot file. >=20 > Implementation (snapshot path) should know estimated maximum size of > the header and can > use the possible maximum. Currently 4KB is enough and easily can be > increased in the bhyve=E2=80=99s code without any problem.=C2=A0 >=20 > Alignment is useful to debug and looking into snapshot image file. >=20 Ah okay, I see the issue. Bhyve has to estimate the size. It's fine to use a fixed header size in the code. However, our image format shouldn't be defined that way. > >=20 > > The IDENT STRING seems to be very large. Even a GUID which should > > be a > > global unique identifier uses just 16 Bytes. Additionally, it might > > be > > worth using a dedicated ident and version field for an easier > > version > > parsing. E.g.: >=20 >=20 > Intention is to add enough space for the future version (as > reservation) and other producers > and companies to specify it=E2=80=99s own ID string with possible add-on > information. So adding =C2=A064 bytes > for the future is not so huge pay, but can be very useful. > During resume, if IDENT string is not the same as in bhyve, resume > can fail before parsing > other data, because it could be that internal format is not as > expected. >=20 > I would not to fix IDENT string format and just apply rule: >=20 > > During resume, bhyve compares its own IDENT string and IDENT string > > from an > > Snapshot image. If it is not the same, further assumption about > > format cannot be done, > > and resume should fail. >=20 We may have different version of the format from the same produce. IMHO, it makes sense to have a dedicated IDENT and VERSION field to easily figure out 1) if the producer of the image is known 2) if we support that version of the producer Even if you allocated a huge amount of free space, someone would need more. So, what do you think about this format: +---------------------------------------------------------------------+ | IDENT - 16 BYTES | +-------------------+-----------------------+-------------------------+ | VERSION - 4 BYTES | NVLIST SIZE - 4 BYTES | NVLIST OFFSET - 8 BYTES | +-------------------+-----------------------+-------------------------+ | POSSIBLE FREE SPACE (e.g. for custom data, alignment etc.) | +---------------------------------------------------------------------+ | NVLIST DATA | +---------------------------------------------------------------------+ | POSSIBLE FREE SPACE (for whatever reason) | +---------------------------------------------------------------------+ | SNAPSHOT DATA | +---------------------------------------------------------------------+ > >=20 > > +------------------+-------------------+ > > | IDENT - 56 BYTES | VERSION - 8 BYTES | > > +------------------+-------------------+ > >=20 > > IDENT - "BHYVE CHECKPOINT IMAGE" > > VERSION - 1 (as uint64_t) > >=20 > > Btw: I don't care but here we could leave some free space for > > possible > > forward compatibility. E.g.: > >=20 > > +------------------+-------------------+-------------------------+ > > | IDENT - 16 BYTES | VERSION - 8 BYTES | _FREE_SPACE_ - 40 BYTES | > > +------------------+-------------------+=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94+ > ... > > > 4. EXAMPLE: > > >=20 > > >=20 > > > =C2=A0IDENT STRING: > > >=20 > > > =C2=A0 =C2=A0 =C2=A0 =C2=A0"BHYVE CHECKPOINT IMAGE VERSION 1" > > >=20 > > > =C2=A0NVLIST HEADER:=C2=A0 > > >=20 > > > =C2=A0=C2=A0[config] > > > =C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0config.offset =3D 0x1000 (4096) > > > =C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0config.size =3D 0x1f6 (502) > > > =C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0config.type =3D "text" > > >=20 > >=20 > > Not sure if it's just an example for the "text" type. bhyve > > converts it > > into a nvlist, so it could be saved directly as nvlist. > > Btw: I would only implement the "text" type if there's an usecase > > that > > can't be solved by one of the other types. >=20 >=20 >=20 > Intention is to use current engine to dump bhyve=E2=80=99s config and rea= d > config > from a file (-k option). >=20 > Advantage of using =E2=80=9Ctext=E2=80=9D type - simple implementation an= d as an > example > of flexibility of proposed image format. Image file can keep any > types that > a producer would like to use: text, nvlist, binary, diff-pages, etc. >=20 nvlist is bhyve's internal format. So, implementing it should be even more simple. We don't need an example of how flexible the image format is in our final implementation. Adding it as an example for this thread is fine. > >=20 > > All in all, it looks good. Keep on your work! > >=20 > > Regards checksum feature: > > We should focus on enabling this feature by default before adding > > advanced features. So, keep it simple and small. >=20 >=20 > Could you give a more example what you meant about =E2=80=9Cchecksum=E2= =80=9D > feature? Did you mean as > TAR=E2=80=99s checksum, i.e. only header? >=20 Sry for the confusion. I was referring to the discussion about checksums (like checksums for section data) in this thread. Just ignore my comment and focus on the rest. >=20 > >=20 > > Regards forward compatibility: > > Backward compatibility is way more important than forward > > compatibility. Nevertheless, forward compatibility would be nice to > > have. So, we should keep it in mind when modifying the layout. For > > the > > moment, just focus on a format which is backward compatible. > >=20 >=20 >=20 > It seems that having information about forward compatibility could be > very > useful, at least to get it in advance if it is impossible to restore. > I will add it during > implementing this format. >=20 Forward compatibility would be great but please just ignore it at the moment to speed up the development. We have versioning. So, we can add forward compatibility in a future version. --=20 Kind regards, Corvin --=-EnCgH2AdnOZbtGH9VMEd Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEgvRSla3m2t/H2U9G2FTaVjFeAmoFAmR/LckACgkQ2FTaVjFe AmrMLw//WL+go44XpMM39BYieAgn709rpTE1qMfR7M592bmT3iztxgSDwBkRQamU jbBmP1GE2jcY9/iuaHxTWtv9/iah87IyUZ+j7tm9I3i7aktvd0sRmQaC84GWXdvV Yu/MGpitkLtUoCrg3uU0l+FtaYhyK8I8bP5T1xrMQoj47dlsb9rrKqoMjznYaSPW 2SjmKpSjYKfN14dHGi3HRJ9WmyDtTyUvipb9E4Uma5SfZyXv9TcM7QjTmI0CFHVu 8Dhq+Q4jjoYLPLPk/c0xaZmlaL4QhKE4n9m9gNuH7vkGU4Sm01QU8XLG7VrY/J5J +vmqH1l/3xfIYcJbZ4qQhccG8OznERLQL/eHd7Bj/9fYLNmTjGQ21eCSWl0Lm0bo 8F9wPCOSZ/i7R3w0iBqXluF7ZMe9JmZot8nb3mUKCaChESbb8HExuRHpnnc0VpNr qJFQ2YRmNhwqP69mQtCxf0pM1f4OSCKB/Db6Xs43xbArG9xOEbEgVlr8E57ov+Vr 2Ad/eIBTknZQ93oMGDsB2KrcbfAYVF6egp3fwdYc/m1tyQ22YHS99A/vm30qYywW LfIYUHzhiw5MMpf9mDM+GmIf0YkaQCtMxNn3zBjOCsmR4afDo/DZ/SMRnZgjcZzV Mi/Yrth5DfMdMOZjEANU0Bxp6Wh0tjwB31IiKl5fuFp32+HNX6c= =k+Ep -----END PGP SIGNATURE----- --=-EnCgH2AdnOZbtGH9VMEd-- From nobody Wed Jun 7 18:25:41 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qbwl931CKz4bphZ; Wed, 7 Jun 2023 18:25:57 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Received: from mail-lf1-x134.google.com (mail-lf1-x134.google.com [IPv6:2a00:1450:4864:20::134]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Qbwl81n9lz3pKr; Wed, 7 Jun 2023 18:25:56 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=LLP7uIih; spf=pass (mx1.freebsd.org: domain of gusev.vitaliy@gmail.com designates 2a00:1450:4864:20::134 as permitted sender) smtp.mailfrom=gusev.vitaliy@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-lf1-x134.google.com with SMTP id 2adb3069b0e04-4f6454a21a9so977349e87.3; Wed, 07 Jun 2023 11:25:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1686162353; x=1688754353; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=ntn/ojICyGo375vFkoWg4WzcDPY21XXPKEZyJj98DGI=; b=LLP7uIihKKxPFFASNanKoalRbMbh5ROdK+nbV7Zysgi/c8DF+SIlUoZqV4N/0mJ+E5 HMCIL/n1mtor0SK13u9Uj0wn0hhjFKM65sD7ynORYEY/l1iHhfIxM9SDIdzYjAlGi1sS 7J/H9jlIlxwwooYDcMZywxLvgkjb7TBtvRYwke6ewzkvtgoGxc6yY9WZFyNGTdlB4sRR dqMDT01//N48xgIVjiQZcBmjtCCv/fBG6+JM2yooauWffScxyIQbNVvPNJkAxZVNn8YR SWwSvNd5YtLOBE2bSkH22m3k/RcehYCIp4pjwjND5cu4p9uJye/MGaSubzMQEQFeF6O6 RaoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686162353; x=1688754353; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ntn/ojICyGo375vFkoWg4WzcDPY21XXPKEZyJj98DGI=; b=EtgLV/dZ51iyXV52UaDHhuMWFHmgSnp0q9CUoT8FxqCxFkRCZaRRVkQ4pxlQUyWFhf h2EsrlfhDRHfYU54e2kFT9vqgIXZxfyFVSO8dvrzvddzQakiZVQrH2nTST4P5jhRWlHQ mCkRmlk50g3fB8WhyRiuLEoAhqpTzJDJjtXoTJ4XwnTPofe9YdfssI70x9hs0Lxzyakk 78areW8GOTdC1q/fWErDe0E9z026uhCuc4stXqykG5CmI/IopBOrUet+6/hgrkwJVaQO sASfXmK0/p8g4moWKhRvyrT1l7CB7d8+YxfOqbWv8SUX6MJrQQdn8yniZNqwCgJBcgkx oJOg== X-Gm-Message-State: AC+VfDwpqRncHYYicV7BLpZMsZU1zX9vcRahnu+lsAgqQG70HoNO0brK u+kWGKGwr2lPO7E9s3HgDUZcmZ4I4Hc= X-Google-Smtp-Source: ACHHUZ4fg57EBL71dETyzTYvmfYARRtcAcAgprsRCkkOao+Pk/OMZOGG/RC1kPPiOtnzXiJ1iQ0Fog== X-Received: by 2002:a19:740f:0:b0:4f3:802c:7725 with SMTP id v15-20020a19740f000000b004f3802c7725mr2426920lfe.30.1686162352400; Wed, 07 Jun 2023 11:25:52 -0700 (PDT) Received: from smtpclient.apple ([188.187.60.230]) by smtp.gmail.com with ESMTPSA id c24-20020ac25318000000b004eefdd8b37fsm1876015lfh.194.2023.06.07.11.25.51 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 07 Jun 2023 11:25:51 -0700 (PDT) From: Vitaliy Gusev Message-Id: <986A83D8-E0E0-4030-9369-A5B3500E27C6@gmail.com> Content-Type: multipart/alternative; boundary="Apple-Mail=_95CB93A3-5926-4337-8EBD-2B9AD8CCD631" List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.600.7\)) Subject: Re: BHYVE SNAPSHOT image format proposal Date: Wed, 7 Jun 2023 21:25:41 +0300 In-Reply-To: Cc: virtualization@freebsd.org, freebsd-hackers@freebsd.org To: =?utf-8?Q?Corvin_K=C3=B6hne?= References: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> <6b98da58a5bd8e83bc466efa20b5a900298210aa.camel@FreeBSD.org> <8387AC83-6667-48E5-A3FA-11475EA96A5F@gmail.com> X-Mailer: Apple Mail (2.3731.600.7) X-Spamd-Result: default: False [-3.30 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.98)[-0.978]; NEURAL_HAM_SHORT(-0.82)[-0.818]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::134:from]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; TAGGED_FROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; MLMMJ_DEST(0.00)[virtualization@freebsd.org,freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4Qbwl81n9lz3pKr X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_95CB93A3-5926-4337-8EBD-2B9AD8CCD631 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi Corvin,=20 > On 6 Jun 2023, at 15:59, Corvin K=C3=B6hne = wrote: >> ... >=20 > We may have different version of the format from the same produce. > IMHO, it makes sense to have a dedicated IDENT and VERSION field to > easily figure out >=20 > 1) if the producer of the image is known > 2) if we support that version of the producer >=20 > Even if you allocated a huge amount of free space, someone would need > more. So, what do you think about this format: >=20 > = +---------------------------------------------------------------------+ > | IDENT - 16 BYTES = | > = +-------------------+-----------------------+-------------------------+ > | VERSION - 4 BYTES | NVLIST SIZE - 4 BYTES | NVLIST OFFSET - 8 BYTES = | > = +-------------------+-----------------------+-------------------------+ > | POSSIBLE FREE SPACE (e.g. for custom data, alignment etc.) = | > = +---------------------------------------------------------------------+ > | NVLIST DATA = | > = +---------------------------------------------------------------------+ > | POSSIBLE FREE SPACE (for whatever reason) = | > = +---------------------------------------------------------------------+ > | SNAPSHOT DATA = | > = +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94+ >=20 Note, simple string "BHYVE CHECKPOINT IMAGE=E2=80=9D has 22 bytes. So 16 = bytes seems too small.=20 So I would not to complicate a header first.=20 I would rather describe ideas, conditions and then solutions: Need to distinguish snapshot image file from other files.=20 Solution: Header should have "magic id=E2=80=9D. Need a barrier for resuming if image "is not ours=E2=80=9D. Idea is not = to allow to resume images from other producers. The reason to have it and use it instead of header versioning:=20 Imagine that mainstream has its own implementation and company=E2=80=99s = fork repo has its own implementation. How to ensure that the versions in = an image file are ours and not somebody=E2=80=99s else? Solution: Header should have =E2=80=9CProducer id=E2=80=9D string. Example: Snapshot image file has empty Producer string , but bhyve has = current Producer as =E2=80=9CMYCOMPANY=E2=80=9D. Strings are not equal, = resume must fail. The Rule above does not restrict getting/decoding data from an image = file. It should be possible to look at an image file and analyse = internals, to get/decode values, etc. Solution: Have additional option either in bhyve or bhyvectl to get into = image file. Following nvlist header data should have a short information about image = file and its internals. Solution: NV HEADER can have several sections: =E2=80=9Cconfig=E2=80=9D, = =E2=80=9Ckernel=E2=80=9D, =E2=80=9Cdevices=E2=80=9D, =E2=80=9Cmemory=E2=80= =9D, =E2=80=A6 Versioning of NV HEADER. Idea is to have an information in advance = whether it is possible to be resumed or not. In other words, before do = resume, get information about ability to resume. Solution: Each Section should have =E2=80=9Cversion=E2=80=9D and = =E2=80=9Csubversion=E2=80=9D. While =E2=80=9Cversion=E2=80=9D is = responsible for both types of compatibility: backward and forward, = =E2=80=9Csubversion=E2=80=9D is for forward compatibility only. Rules for check: If bh_version =3D=3D version && bh_subversion >=3D subversion = then Bhyve able to resume the Section Else Bhyve cannot resume the Section Endif Example 1: Section in image has =E2=80=9Cversion=3D1", = =E2=80=9Csubversion=3D5=E2=80=9D, Bhyve has =E2=80=9Cversion=3D1", = =E2=80=9Csubversion=3D6". That means, bhyve can resume the Section. Example 2: The same image Section, but bhyve has =E2=80=9Cversion=3D1", = =E2=80=9Csubversion=3D4". Bhyve cannot resume the Section. Example 3: The same image Section, but bhyve has =E2=80=9Cversion=3D2", = =E2=80=9Csubversion=3D5=E2=80=9D. Bhyve cannot resume the Section. Rules for increasing versions: - If during code-change =E2=80=9Cbackward=E2=80=9D compatibility is = broken, =E2=80=9Cversion=E2=80=9D should be increased and = =E2=80=9Csubversion=E2=80=9D is set to 0. - If during code-change =E2=80=9Cforward=E2=80=9D compatibility is = broken, =E2=80=9Csubversion=E2=80=9D should be increased. Other versioning in HEADER is redundant. If something is changed in the = format, =E2=80=9Cmagic id=E2=80=9D can be changed appropriately. Solution: =E2=80=9Cmagic id=E2=80=9D should be stable and not changed = for a long time. As result I would suggest to give at least 32 bytes for "magic id=E2=80=9D= / ident and 32 bytes for =E2=80=9Cproducer id=E2=80=9D. Format of entire image file can be: +-----------------------------------------------------------------+ | HEADER MAGIC ID - 32 BYTES | +-----------------------------------------------------------------+ | HEADER PRODUCER ID - 32 BYTES | = +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94--------------= --=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= --=E2=80=94-----------------------+ | NVLIST HEADER SIZE - 4 BYTES |=20 +-----------------------------------------------------------------+ | NVLIST HEADER DATA (SECTIONS) | +-----------------------------------------------------------------+ | SNAPSHOT DATA | +-----------------------------------------------------------------+ MAGIC ID: should be hardcoded: "BHYVE CHECKPOINT IMAGE=E2=80=9D. PRODUCER ID: can be empty and supported by producer, i.e. reserved.=20 NVLIST HEADER SIZE: has enough dimension, but in general size is less = than 4KB NVLIST HEADER DATA: Packed nvlist data, contains Sections: = =E2=80=9Cconfig=E2=80=9D, =E2=80=9Ckernel=E2=80=9D, =E2=80=9Cdevices=E2=80= =9D, =E2=80=9Cmemory=E2=80=9D, =E2=80=A6 : [config] offset =3D 0x1000 (4096) size =3D 0x1f6 (502) type =3D text vers =3D 1 subvers =3D 5 [kernel] offset =3D 0x11f6 (4598) size =3D 0x19a7 (6567) type =3D nvlist vers =3D 1 subvers =3D 0 [devices] offset =3D 0x2b9d (11165) size =3D 0x10145ba (16860602) type =3D nvlist vers =3D 2 subvers =3D 1 [memory] offset =3D 0x1200000 (18874368) size =3D 0x3ce00000 (1021313024) type =3D pages vers =3D 1 subvers =3D 0=20 I hope I gained a whole understanding. Thanks, Vitaliy Gusev --Apple-Mail=_95CB93A3-5926-4337-8EBD-2B9AD8CCD631 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi = Corvin, 

On 6 Jun 2023, at 15:59, = Corvin K=C3=B6hne <corvink@FreeBSD.org> = wrote:
...

We may have different version of the = format from the same produce.
IMHO, it makes sense to have a = dedicated IDENT and VERSION field to
easily figure out

1) if = the producer of the image is known
2) if we support that version of = the producer

Even if you allocated a huge amount of free space, = someone would need
more. So, what do you think about this = format:

+----------------------------------------------------------= -----------+
| IDENT - 16 BYTES =             &n= bsp;           &nbs= p;            =             &n= bsp; |
+-------------------+-----------------------+--------------= -----------+
| VERSION - 4 BYTES | NVLIST SIZE - 4 BYTES | NVLIST = OFFSET - 8 BYTES = |
+-------------------+-----------------------+------------------------= -+
| POSSIBLE FREE SPACE (e.g. for custom data, alignment etc.) =          |
+--------------= -------------------------------------------------------+
| NVLIST = DATA =             &n= bsp;           &nbs= p;            =             &n= bsp;      |
+----------------------------= -----------------------------------------+
| POSSIBLE FREE SPACE (for = whatever reason) =             &n= bsp;           &nbs= p; |
+------------------------------------------------------------= ---------+
| SNAPSHOT DATA =             &n= bsp;           &nbs= p;            =             &n= bsp;    |
+=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+


Note, = simple string "BHYVE CHECKPOINT = IMAGE=E2=80=9D has 22 bytes. So 16 bytes seems
too = small. 

So I would not to complicate a header = first. 

I would rather describe ideas, = conditions and then solutions:

  1. Need to distinguish snapshot image file = from other files.
    Solution: Header should have "magic = id=E2=80=9D.

  2. Need a barrier for resuming if image = "is not ours=E2=80=9D. Idea is not to allow to resume images from other = producers.

    The reason to have it and use it instead of header = versioning:

    Imagine that mainstream has its own implementation = and company=E2=80=99s fork repo has its own implementation. How to = ensure that the versions in an image file are ours = and not somebody=E2=80=99s else?

    Solution:  Header = should have =E2=80=9CProducer id=E2=80=9D string.

    Example: = Snapshot image file has empty Producer string , but bhyve has current = Producer as =E2=80=9CMYCOMPANY=E2=80=9D. Strings are not equal, resume = must fail.

  3. The Rule above does not restrict = getting/decoding data from an image file. It should be possible to look = at an image file and analyse internals, to get/decode values, = etc.
    Solution: Have additional option either in bhyve or = bhyvectl to get into image file.

  4. Following nvlist header = data should have a short information about image file and its = internals.
    Solution: NV HEADER can have several sections: = =E2=80=9Cconfig=E2=80=9D, =E2=80=9Ckernel=E2=80=9D, =E2=80=9Cdevices=E2=80= =9D, =E2=80=9Cmemory=E2=80=9D, =E2=80=A6

  5. Versioning of = NV HEADER. Idea is to have an information in advance whether it is = possible to be resumed or not. In other words, before do resume, get = information about ability to resume.

    Solution: Each = Section should have =E2=80=9Cversion=E2=80=9D  and = =E2=80=9Csubversion=E2=80=9D. While =E2=80=9Cversion=E2=80=9D is = responsible for both  types of compatibility: backward and forward, = =E2=80=9Csubversion=E2=80=9D is for forward compatibility = only.

    Rules for check:
            If = bh_version =3D=3D version && bh_subversion >=3D subversion =  then
                    =   Bhyve able to resume the Section
            = Else
                      = Bhyve cannot resume the Section
            = Endif

    Example 1    : Section in image has =E2=80=9Cversion=3D= 1", =E2=80=9Csubversion=3D5=E2=80=9D,  Bhyve has =E2=80=9Cversion=3D1= ", =E2=80=9Csubversion=3D6". That means, bhyve can resume the = Section.

    Example 2: The = same image Section, but bhyve has =E2=80=9Cversion=3D1", = =E2=80=9Csubversion=3D4". Bhyve cannot resume the Section.

    Example 3: The same image Section, = but bhyve has =E2=80=9Cversion=3D2", =E2=80=9Csubversion=3D5=E2=80=9D. = Bhyve cannot resume the Section.

    Rules for increasing = versions:
      -  If during code-change =E2=80=9Cbackward=E2= =80=9D compatibility is broken, =E2=80=9Cversion=E2=80=9D should be = increased and =E2=80=9Csubversion=E2=80=9D is set to 0.
      - =  If during code-change =E2=80=9Cforward=E2=80=9D compatibility is = broken, =E2=80=9Csubversion=E2=80=9D should be = increased.

  6. Other versioning in HEADER is redundant. If = something is changed in the format, =E2=80=9Cmagic id=E2=80=9D can be = changed appropriately.
    Solution: =E2=80=9Cmagic id=E2=80=9D = should be stable and not changed for a long = time.


As result I would suggest to = give at least 32 bytes for "magic id=E2=80=9D / ident and 32 bytes for = =E2=80=9Cproducer id=E2=80=9D.

Format of entire image = file can be:


+--------------------------------------------------------------= ---+
|             =   HEADER MAGIC ID     - 32 BYTES       =              |
+--------------------------------------------------------------= ---+
|             =   HEADER PRODUCER ID  - 32 BYTES         =            |
+=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94--= --------------=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94--=E2=80=94-----------------------+
|   =             NVLIST HEADER SIZE  - =  4 BYTES                   =  | 
+--------------------------------------------------------------= ---+
|             =   NVLIST HEADER DATA (SECTIONS)           =           |
+--------------------------------------------------------------= ---+
|             =           SNAPSHOT DATA       =                     =   = |
+-----------------------------------------------------------------+

MAGIC ID: should be hardcoded: "BHYVE CHECKPOINT = IMAGE=E2=80=9D.

PRODUCER ID: can = be empty and supported by producer, i.e. = reserved. 

NVLIST HEADER SIZE: has = enough dimension, but in general size is less than = 4KB

NVLIST HEADER DATA: Packed nvlist data, = contains Sections:  =E2=80=9Cconfig=E2=80=9D, =E2=80=9Ckernel=E2=80=9D= , =E2=80=9Cdevices=E2=80=9D, =E2=80=9Cmemory=E2=80=9D, =E2=80=A6 = :

[config]

    =     offset =3D 0x1000 (4096)

    =     size =3D 0x1f6 (502)

    =     type =3D text

vers =3D = 1
= subvers =3D 5

[kernel]

    =     offset =3D 0x11f6 (4598)

    =     size =3D 0x19a7 (6567)

    =     type =3D nvlist

vers =3D 1
subvers =3D 0

[devices]

    =     offset =3D 0x2b9d (11165)

    =     size =3D 0x10145ba (16860602)

    =     type =3D nvlist

vers =3D 2
subvers =3D 1

[memory]

    =     offset =3D 0x1200000 (18874368)

    =     size =3D 0x3ce00000 (1021313024)

        type = =3D pages

vers =3D 1
subvers =3D = 0 



I hope = I gained a whole understanding.
Thanks,
Vitaliy = Gusev

= --Apple-Mail=_95CB93A3-5926-4337-8EBD-2B9AD8CCD631-- From nobody Wed Jun 7 19:21:28 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QbyWB3Gvpz4ZpVy for ; Wed, 7 Jun 2023 19:45:42 +0000 (UTC) (envelope-from li-fbsd@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QbyW90gjMz40gm for ; Wed, 7 Jun 2023 19:45:40 +0000 (UTC) (envelope-from li-fbsd@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of li-fbsd@citylink.dinoex.sub.org designates 2a0b:f840::12 as permitted sender) smtp.mailfrom=li-fbsd@citylink.dinoex.sub.org; dmarc=none Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.17.2/8.17.2) with ESMTPS id 357Jj73g092248 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Wed, 7 Jun 2023 21:45:08 +0200 (CEST) (envelope-from li-fbsd@citylink.dinoex.sub.org) X-MDaemon-Deliver-To: Received: (from uucp@localhost) by uucp.dinoex.org (8.17.2/8.17.2/Submit) with UUCP id 357Jj75N092247 for freebsd-hackers@freebsd.org; Wed, 7 Jun 2023 21:45:07 +0200 (CEST) (envelope-from li-fbsd@citylink.dinoex.sub.org) Received: from admn.intra.daemon.contact (localhost [127.0.0.1]) by admn.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 357JLoPJ079895 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Wed, 7 Jun 2023 21:21:50 +0200 (CEST) (envelope-from li-fbsd@citylink.dinoex.sub.org) Received: from intra.daemon.contact (news@localhost) by admn.intra.daemon.contact (8.17.1/8.17.1/Submit) with NNTP id 357JLShb079795 for freebsd-hackers@freebsd.org; Wed, 7 Jun 2023 21:21:28 +0200 (CEST) (envelope-from li-fbsd@citylink.dinoex.sub.org) X-Authentication-Warning: admn.intra.daemon.contact: news set sender to li-fbsd@citylink.dinoex.sub.org using -f From: "Peter 'PMc' Much" X-Newsgroups: m2n.fbsd.hackers Subject: Re: low TCP speed, wrong rtt measurement Date: Wed, 7 Jun 2023 19:21:28 -0000 (UTC) Message-ID: References: <31e31ac3-f336-bbba-59c2-ab2cda918fb9@freebsd.org> Injection-Date: Wed, 7 Jun 2023 19:21:28 -0000 (UTC) Injection-Info: admn.intra.daemon.contact; logging-data="79794"; mail-complaints-to="usenet@citylink.dinoex.sub.org" User-Agent: slrn/1.0.3 (FreeBSD) To: freebsd-hackers@freebsd.org X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Wed, 07 Jun 2023 21:45:10 +0200 (CEST) X-Spamd-Result: default: False [-1.29 / 15.00]; NEURAL_HAM_SHORT(-1.00)[-0.996]; NEURAL_HAM_LONG(-0.67)[-0.674]; NEURAL_SPAM_MEDIUM(0.38)[0.381]; FORGED_SENDER(0.30)[pmc@citylink.dinoex.sub.org,li-fbsd@citylink.dinoex.sub.org]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; BLOCKLISTDE_FAIL(0.00)[2a0b:f840::12:server fail]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[sub.org]; R_DKIM_NA(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; HAS_XAW(0.00)[]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_DN_NONE(0.00)[]; FROM_NEQ_ENVFROM(0.00)[pmc@citylink.dinoex.sub.org,li-fbsd@citylink.dinoex.sub.org] X-Rspamd-Queue-Id: 4QbyW90gjMz40gm X-Spamd-Bar: - X-ThisMailContainsUnwantedMimeParts: N List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Folks, I can now say for certain that there is a flaw in the rtt measurements. On 2023-04-09, Scheffenegger, Richard wrote: > Finally, you could be using SIFTR to track the evolution of the minrtt > value over the course of the session. > Although I suspect ultimately a tcpdump including the tcp header (-s 80) > , and the sifter internal state evolution would be optimal to > understanding when and why the RTT values go off the rails. > At first glance, the ertt module may be prone to miscalculations, when > retransmissions are in play - no special precautions appear to be > present, to distinguish between the originally sent packet, and any > retransmission, nor any filtering of ACKs which come in as duplicates. > Thus there could be a scenario, where an ACK for a spurious > retransmission, e.g. due to reordering, could lead to a wrong baseline > RTT measurement, which is physically impossible on such a long distance > connection... Richard, thanks for Your comments. It looks like You are right, and Your debug advice sounds interesting - but for now I went a different way: I put timestamping into the h_ertt code and reset the minrtt measurement after an hour. I think this is a good idea in any case, as physics might switch during a connection (only the way I did it, straightforward inline comparing the timestamp at every occasion, is not really good performance-wise), and mainly, it does already relieve my problem a bit. On my May upload, the flaw did reproduce. Then I put in that patch. Now on the June upload, the flaw does reproduce again, and the patch fixes it after some time. Here is the evidence: tcp4 0 123692 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 14552 7240 1460 off tcp4 0 123692 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 14552 7240 1460 off tcp4 0 123692 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 8712 7240 1460 off tcp4 0 123692 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 14552 7240 1460 off tcp4 0 123692 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 14552 7240 1460 off tcp4 0 123692 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 14552 7240 1460 off tcp4 0 123692 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 14552 7240 1460 off tcp4 0 123299 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 14552 7240 1460 off tcp4 0 120796 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 14552 7240 1460 off tcp4 0 123692 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 14552 7240 1460 off tcp4 0 123692 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 14552 7240 1460 off tcp4 0 123692 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 14552 7240 1460 off tcp4 0 123692 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 14552 7240 1460 off The cwin appears to be locked at 14552 and doesn't go higher. And then the patch does it's action: Jun 7 15:28:47 edge kernel: [72317] h_ertt reset minrtt 1686087605, was 11 since 1686140926 This runs some 500 miles plus tunnels&filters, so rtt 11 is impossible here. After resetting it, the cwin is normal again: tcp4 0 123692 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 20368 10136 1460 off tcp4 0 123692 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 20368 10136 1460 off tcp4 0 123692 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 14528 10136 1460 off tcp4 0 123692 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 17448 10136 1460 off tcp4 0 123381 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 10160 8688 1460 off tcp4 0 122117 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 14540 8688 1460 off tcp4 0 123692 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 20380 8688 1460 off tcp4 0 123692 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 20380 8688 1460 off tcp4 0 123692 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 15988 10136 1460 off tcp4 0 123692 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 18908 10136 1460 off tcp4 0 123692 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 20368 10136 1460 off tcp4 0 123692 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 20368 10136 1460 off tcp4 0 123692 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 21828 10136 1460 off tcp4 0 113559 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 21828 10136 1460 off tcp4 0 123692 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 23288 10136 1460 off tcp4 0 123692 192.168.97.17.24379 192.168.99.1.9103 ESTABLISHED vegas 23288 10136 1460 off So now, with timestamps provided, it makes sense to look into the tcpdump. The next occasion occurs: Jun 7 19:44:59 edge kernel: [87689] h_ertt reset minrtt 1686087605, was 6 since 1686156298 18:44:56.561344 IP 192.168.99.1.9103 > 192.168.97.17.24379: Flags [.], ack 1271901977, win 1518, options [nop,nop,TS val 3960838455 ecr 2751963567], length 0 18:44:56.561438 IP 192.168.97.17.24379 > 192.168.99.1.9103: Flags [.], seq 1271913561:1271915009, ack 0, win 1027, options [nop,nop,TS val 2751963822 ecr 3960838455], length 1448 18:44:56.561446 IP 192.168.97.17.24379 > 192.168.99.1.9103: Flags [.], seq 1271915009:1271916457, ack 0, win 1027, options [nop,nop,TS val 2751963822 ecr 3960838455], length 1448 18:44:56.833069 IP 192.168.99.1.9103 > 192.168.97.17.24379: Flags [.], ack 1271904873, win 1540, options [nop,nop,TS val 3960838833 ecr 2751963582,nop,nop,sack 2 {1271915009:1271916457}{1271909217:1271910665}], length 0 18:44:56.833166 IP 192.168.97.17.24379 > 192.168.99.1.9103: Flags [.], seq 1271916457:1271917905, ack 0, win 1027, options [nop,nop,TS val 2751964093 ecr 3960838833], length 1448 18:44:56.833175 IP 192.168.97.17.24379 > 192.168.99.1.9103: Flags [.], seq 1271917905:1271919353, ack 0, win 1027, options [nop,nop,TS val 2751964093 ecr 3960838833], length 1448 18:44:56.924990 IP 192.168.99.1.9103 > 192.168.97.17.24379: Flags [.], ack 1271904873, win 1540, options [nop,nop,TS val 3960838924 ecr 2751963582,nop,nop,sack 2 {1271915009:1271917905}{1271909217:1271910665}], length 0 18:44:56.926649 IP 192.168.99.1.9103 > 192.168.97.17.24379: Flags [.], ack 1271904873, win 1540, options [nop,nop,TS val 3960838926 ecr 2751963582,nop,nop,sack 2 {1271915009:1271919353}{1271909217:1271910665}], length 0 18:44:56.926731 IP 192.168.97.17.24379 > 192.168.99.1.9103: Flags [.], seq 1271904873:1271906321, ack 0, win 1027, options [nop,nop,TS val 2751964187 ecr 3960838926], length 1448 18:44:57.939383 IP 192.168.97.17.24379 > 192.168.99.1.9103: Flags [.], seq 1271904873:1271906321, ack 0, win 1027, options [nop,nop,TS val 2751965203 ecr 3960838926], length 1448 18:44:57.988205 IP 192.168.99.1.9103 > 192.168.97.17.24379: Flags [.], ack 1271906321, win 1518, options [nop,nop,TS val 3960839988 ecr 2751965203,nop,nop,sack 2 {1271915009:1271919353}{1271909217:1271910665}], length 0 18:44:57.988316 IP 192.168.97.17.24379 > 192.168.99.1.9103: Flags [.], seq 1271906321:1271907769, ack 0, win 1027, options [nop,nop,TS val 2751965248 ecr 3960839988], length 1448 18:44:57.988326 IP 192.168.97.17.24379 > 192.168.99.1.9103: Flags [.], seq 1271907769:1271909217, ack 0, win 1027, options [nop,nop,TS val 2751965248 ecr 3960839988], length 1448 18:44:58.032928 IP 192.168.99.1.9103 > 192.168.97.17.24379: Flags [.], ack 1271907769, win 1518, options [nop,nop,TS val 3960840033 ecr 2751965248,nop,nop,sack 2 {1271915009:1271919353}{1271909217:1271910665}], length 0 18:44:58.033056 IP 192.168.97.17.24379 > 192.168.99.1.9103: Flags [.], seq 1271907769:1271909217, ack 0, win 1027, options [nop,nop,TS val 2751965293 ecr 3960840033], length 1448 18:44:58.033065 IP 192.168.97.17.24379 > 192.168.99.1.9103: Flags [.], seq 1271910665:1271912113, ack 0, win 1027, options [nop,nop,TS val 2751965293 ecr 3960840033], length 1448 18:44:58.033067 IP 192.168.97.17.24379 > 192.168.99.1.9103: Flags [.], seq 1271912113:1271913561, ack 0, win 1027, options [nop,nop,TS val 2751965293 ecr 3960840033], length 1448 18:44:58.033070 IP 192.168.97.17.24379 > 192.168.99.1.9103: Flags [.], seq 1271913561:1271915009, ack 0, win 1027, options [nop,nop,TS val 2751965293 ecr 3960840033], length 1448 18:44:58.042332 IP 192.168.99.1.9103 > 192.168.97.17.24379: Flags [.], ack 1271910665, win 1495, options [nop,nop,TS val 3960840042 ecr 2751965248,nop,nop,sack 1 {1271915009:1271919353}], length 0 18:44:58.077990 IP 192.168.99.1.9103 > 192.168.97.17.24379: Flags [.], ack 1271910665, win 1540, options [nop,nop,TS val 3960840078 ecr 2751965248,nop,nop,sack 2 {1271907769:1271909217}{1271915009:1271919353}], length 0 18:44:58.090877 IP 192.168.99.1.9103 > 192.168.97.17.24379: Flags [.], ack 1271912113, win 1518, options [nop,nop,TS val 3960840091 ecr 2751965293,nop,nop,sack 1 {1271915009:1271919353}], length 0 18:44:58.090972 IP 192.168.97.17.24379 > 192.168.99.1.9103: Flags [.], seq 1271912113:1271913561, ack 0, win 1027, options [nop,nop,TS val 2751965353 ecr 3960840091], length 1448 Here we sent. 18:44:58.090981 IP 192.168.97.17.24379 > 192.168.99.1.9103: Flags [.], seq 1271913561:1271915009, ack 0, win 1027, options [nop,nop,TS val 2751965353 ecr 3960840091], length 1448 18:44:58.098474 IP 192.168.99.1.9103 > 192.168.97.17.24379: Flags [.], ack 1271913561, win 1518, options [nop,nop,TS val 3960840098 ecr 2751965293,nop,nop,sack 1 {1271915009:1271919353}], length 0 And here we got ACK. This is actually 7.5 ms, well, near enough to 6. 18:44:58.098560 IP 192.168.97.17.24379 > 192.168.99.1.9103: Flags [.], seq 1271919353:1271920801, ack 0, win 1027, options [nop,nop,TS val 2751965358 ecr 3960840098], length 1448 18:44:58.107088 IP 192.168.99.1.9103 > 192.168.97.17.24379: Flags [.], ack 1271919353, win 1450, options [nop,nop,TS val 3960840107 ecr 2751965293], length 0 18:44:58.107167 IP 192.168.97.17.24379 > 192.168.99.1.9103: Flags [.], seq 1271920801:1271922249, ack 0, win 1027, options [nop,nop,TS val 2751965368 ecr 3960840107], length 1448 18:44:58.137900 IP 192.168.99.1.9103 > 192.168.97.17.24379: Flags [.], ack 1271919353, win 1540, options [nop,nop,TS val 3960840138 ecr 2751965293,nop,nop,sack 1 {1271912113:1271913561}], length 0 And here is the DSACK. 18:44:58.147276 IP 192.168.99.1.9103 > 192.168.97.17.24379: Flags [.], ack 1271919353, win 1540, options [nop,nop,TS val 3960840147 ecr 2751965293,nop,nop,sack 1 {1271913561:1271915009}], length 0 18:44:58.235268 IP 192.168.99.1.9103 > 192.168.97.17.24379: Flags [.], ack 1271922249, win 1518, options [nop,nop,TS val 3960840225 ecr 2751965358], length 0 18:44:58.235380 IP 192.168.97.17.24379 > 192.168.99.1.9103: Flags [.], seq 1271922249:1271923697, ack 0, win 1027, options [nop,nop,TS val 2751965497 ecr 3960840225], length 1448 18:44:58.235389 IP 192.168.97.17.24379 > 192.168.99.1.9103: Flags [.], seq 1271923697:1271925145, ack 0, win 1027, options [nop,nop,TS val 2751965497 ecr 3960840225], length 1448 18:44:58.235392 IP 192.168.97.17.24379 > 192.168.99.1.9103: Flags [.], seq 1271925145:1271926593, ack 0, win 1027, options [nop,nop,TS val 2751965497 ecr 3960840225], length 1448 18:44:58.305514 IP 192.168.99.1.9103 > 192.168.97.17.24379: Flags [.], ack 1271925145, win 1518, options [nop,nop,TS val 3960840293 ecr 2751965497], length 0 So, what do we do now? This doesn't look very easy to fix. Any ideas, anybody? cheerio, PMc