From owner-freebsd-virtualization@freebsd.org Sun Mar 3 01:06:18 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AA0FE1519FB8 for ; Sun, 3 Mar 2019 01:06:18 +0000 (UTC) (envelope-from doctor@doctor.nl2k.ab.ca) Received: from doctor.nl2k.ab.ca (doctor.nl2k.ab.ca [204.209.81.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 41E1A8D1C4 for ; Sun, 3 Mar 2019 01:06:17 +0000 (UTC) (envelope-from doctor@doctor.nl2k.ab.ca) Received: from doctor by doctor.nl2k.ab.ca with local (Exim 4.92 (FreeBSD)) (envelope-from ) id 1h0FZt-000PW6-Sv; Sat, 02 Mar 2019 18:06:05 -0700 Date: Sat, 2 Mar 2019 18:06:05 -0700 From: The Doctor To: Victor Sudakov Cc: freebsd-virtualization@freebsd.org Subject: Re: Windows 2019 server Message-ID: <20190303010605.GA97372@doctor.nl2k.ab.ca> References: <20190216123859.GA24315@doctor.nl2k.ab.ca> <40ddfdd6-a1c4-ac00-f088-2880a7d372ce@smeets.xyz> <20190302094829.GA48523@admin.sibptus.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20190302094829.GA48523@admin.sibptus.ru> User-Agent: Mutt/1.11.3 (2019-02-01) X-Rspamd-Queue-Id: 41E1A8D1C4 X-Spamd-Bar: ------ X-Spamd-Result: default: False [-6.97 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.97)[-0.968,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Mar 2019 01:06:18 -0000 On Sat, Mar 02, 2019 at 04:48:29PM +0700, Victor Sudakov wrote: > Florian Smeets via freebsd-virtualization wrote: > > On 16.02.19 13:38, The Doctor via freebsd-virtualization wrote: > > > Anyone got Windows 2019 server running in bhyve ? > > > > > > > Yes, on stable/12 (r343339) using > > https://github.com/churchers/vm-bhyve/wiki/Running-Window. Worked like a > > charm. > > Which kind of storage did you present to the Windows 2019 guest, > paravirtualized or ahcd-hd ? > I was able to install in bhyve FreeBSD 11.2 hoever, the ethernet Driver is not picking up. hints please. -- Member - Liberal International This is doctor@@nl2k.ab.ca Ici doctor@@nl2k.ab.ca Yahweh, Queen & country!Never Satan President Republic!Beware AntiChrist rising! https://www.empire.kred/ROOTNK?t=94a1f39b Look at Psalms 14 and 53 on Atheism Be charitable before wealth makes thee covetous. -Sir Thomas Browne From owner-freebsd-virtualization@freebsd.org Sun Mar 3 02:53:28 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 394DC1520011 for ; Sun, 3 Mar 2019 02:53:28 +0000 (UTC) (envelope-from jtubnor@gmail.com) Received: from mail-ot1-x334.google.com (mail-ot1-x334.google.com [IPv6:2607:f8b0:4864:20::334]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C0DBE6A89F for ; Sun, 3 Mar 2019 02:53:26 +0000 (UTC) (envelope-from jtubnor@gmail.com) Received: by mail-ot1-x334.google.com with SMTP id 98so1481779oty.1 for ; Sat, 02 Mar 2019 18:53:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=0J8Gam9OTiqOIJd3rxNJn3hjpiWyTbGlgMPEW7Mdd3A=; b=IXpP5VMDq3wFIlcMImjeK33TVp7watgzM6rYJnUtJPnUFb0ImdRpshYP35jF7SgDk/ 3DEZjCR4Jbk6CVvZ3fCNx1e56ETVOpJjOFpNrXa71nGx30TOZjJTEcXFZbMJTbD+wNQx u84oRx5tYAMnAHRtjVgk3xDxKWu8EEne2IY/Yk0yn4wTrQl2emys6eR/hmujtEzH+b0s NvffFYSD1yP4Ke1PgvMC9COXotPFok+KlTClr4YcGoCd+ya0KECxrdaPdRrDzckt/i8c rPgfiHzMsNJ3gfTKblXMwpZk/4dpVbC/LTBq6zcfM7n+iVHc4jaXGrb2AHeD6IWD7p6Q 59Ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=0J8Gam9OTiqOIJd3rxNJn3hjpiWyTbGlgMPEW7Mdd3A=; b=szx/SVQTwbrkWYocHjiYmIvAblRQTogjbhrraDpZwSWuMdvVZmNc4lpuT0vjfqUiuj Cptdl0oB6Os8kbp4HzRnohkt21UeXp+xFiW3nkAlblCq7Mc2sJIyjA4JlEdxLr6Bs0Lz RyXNxyFiUI6Z6wTHoA6l38YvKUQrfXSOKQPn/7NENjqPbmYL34KtzuY2w6tnHQTXamjU 5EN8gdTSt96xDjPzbLEf1+r1nM10qIvg1CYYNU+hnb4SPkKcuYPetFUPuDbEeLFu0DVv Tb4i+DGrvr5ggGZ6m4Asg5MbDYz+hgYaP4hDAG+6rkrF7V/6tfvzkbNigYw8srJpRMjJ d4Pw== X-Gm-Message-State: APjAAAV5Jr1gJZXoK60+YwMkVedTXX7CHW4DubAQv/T8W23H1bu6O6ZF avOwZj8pmS9aAe3xPCT3d8Lb3q3MJC3DOf7PKdU= X-Google-Smtp-Source: APXvYqx1mSsVkpQNzUjZF46CrEQWBiDWecyLuMVPyezkVabjO8y/xcwtfIjgOLWwsmF+45K1M98htRkfksqreCA+D5g= X-Received: by 2002:a9d:66cb:: with SMTP id t11mr7658382otm.247.1551581605855; Sat, 02 Mar 2019 18:53:25 -0800 (PST) MIME-Version: 1.0 References: <20190216123859.GA24315@doctor.nl2k.ab.ca> <40ddfdd6-a1c4-ac00-f088-2880a7d372ce@smeets.xyz> <20190302094829.GA48523@admin.sibptus.ru> <20190303010605.GA97372@doctor.nl2k.ab.ca> In-Reply-To: <20190303010605.GA97372@doctor.nl2k.ab.ca> From: Jason Tubnor Date: Sun, 3 Mar 2019 13:53:15 +1100 Message-ID: Subject: Re: Windows 2019 server To: The Doctor Cc: Victor Sudakov , freebsd-virtualization@freebsd.org X-Rspamd-Queue-Id: C0DBE6A89F X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=IXpP5VMD; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jtubnor@gmail.com designates 2607:f8b0:4864:20::334 as permitted sender) smtp.mailfrom=jtubnor@gmail.com X-Spamd-Result: default: False [-5.76 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.03)[-0.031,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_TLS_LAST(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[4.3.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.71)[ip: (-8.79), ipnet: 2607:f8b0::/32(-2.68), asn: 15169(-2.03), country: US(-0.07)]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Mar 2019 02:53:28 -0000 On Sun., 3 Mar. 2019, 12:06 pm The Doctor via freebsd-virtualization, < freebsd-virtualization@freebsd.org> wrote: > > > I was able to install in bhyve FreeBSD 11.2 hoever, the ethernet Driver > is not picking up. > > hints please. > Once you are installed, boot again with the virtio cd on an ahci-cd device, go to devices and you'll see the unknown adaptor, update the driver using the virtio-net ones on the CD. You should be good after that. 2k16 drivers work with 19. Cheers > From owner-freebsd-virtualization@freebsd.org Sun Mar 3 03:29:20 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BADCE1521BF3 for ; Sun, 3 Mar 2019 03:29:20 +0000 (UTC) (envelope-from vas@mpeks.tomsk.su) Received: from admin.sibptus.ru (admin.sibptus.ru [IPv6:2001:19f0:5001:21dc::10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EA3306C09E for ; Sun, 3 Mar 2019 03:29:19 +0000 (UTC) (envelope-from vas@mpeks.tomsk.su) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sibptus.ru; s=20181118; h=In-Reply-To:Message-ID:Subject:To:From:Date; bh=QGwfRWZnDpvj9h4YA7DdLKdRXdgevMVrhWjK9cD3vv8=; b=fGoA06REsI6V/Tb4J29spozfFF w2mcRL9PdY0jh03KpfOTHsaNnH0bx65cua84ApxN0PHUvZrijZJSQJLbeZnnWU86jGp7vovQpAx3/ 7IQAbyQeWbO863WluP/37+6l9yhiaro9v9/lRvZfNImwLqCldmdEGP+fsnRdMdi2iGqU=; Received: from vas by admin.sibptus.ru with local (Exim 4.92 (FreeBSD)) (envelope-from ) id 1h0HoU-000FlA-43 for freebsd-virtualization@freebsd.org; Sun, 03 Mar 2019 10:29:18 +0700 Date: Sun, 3 Mar 2019 10:29:18 +0700 From: Victor Sudakov To: freebsd-virtualization@freebsd.org Subject: Re: Windows 2019 server Message-ID: <20190303032918.GA60561@admin.sibptus.ru> References: <20190216123859.GA24315@doctor.nl2k.ab.ca> <40ddfdd6-a1c4-ac00-f088-2880a7d372ce@smeets.xyz> <20190302094829.GA48523@admin.sibptus.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="J2SCkAp4GZ/dPZZf" Content-Disposition: inline In-Reply-To: X-PGP-Key: http://www.dreamwidth.org/pubkey?user=victor_sudakov X-PGP-Fingerprint: 10E3 1171 1273 E007 C2E9 3532 0DA4 F259 9B5E C634 User-Agent: Mutt/1.11.2 (2019-01-07) Sender: Victor Sudakov X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Mar 2019 03:29:20 -0000 --J2SCkAp4GZ/dPZZf Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Jason Tubnor wrote: >=20 > > Which kind of storage did you present to the Windows 2019 guest, > > paravirtualized or ahcd-hd ? >=20 > ahci-hd is what you want with bhyve and uefi Still no paravirtualized disks support for Windows guests, even in FreeBSD 12? How sad. --=20 Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/ --J2SCkAp4GZ/dPZZf Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEcBAEBAgAGBQJce0oOAAoJEA2k8lmbXsY0fPkH/2Q9tfDuUr5mQ8xcOLvapid5 PVivrGocYbfJqeZRbqOSpHkcWSR7iEXxqCjYCdqsCr4hmAwQHBEg2GlbKPeNH+tC o3HOewN4PQmg9qG2EZcrGaQcGKQtycEicgLyNOoU+eXIenyHpwAlixko7xSFhxoY JJqIJxUXv5KTLxUg6X7Fte7jOrtT+K1QjjQGsq2iEVB0YtPMkgB6N6koR+pWP1tp WvwKNRHaaXN1Q9VZfd2iK39LJ/KV1amjiyUMqhR4lstXLX9zXZrCZ3uxLzcUoREs RCb8Eaifp8n6guFxpxOeCwAPbnBpkBZvWuLxrHFv/hSbuAag03C01C+T+IkSDdc= =2rcB -----END PGP SIGNATURE----- --J2SCkAp4GZ/dPZZf-- From owner-freebsd-virtualization@freebsd.org Sun Mar 3 02:44:12 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 47440151FD3F for ; Sun, 3 Mar 2019 02:44:12 +0000 (UTC) (envelope-from jtubnor@gmail.com) Received: from mail-ot1-x329.google.com (mail-ot1-x329.google.com [IPv6:2607:f8b0:4864:20::329]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B1F536A520 for ; Sun, 3 Mar 2019 02:44:11 +0000 (UTC) (envelope-from jtubnor@gmail.com) Received: by mail-ot1-x329.google.com with SMTP id c18so1443587otl.13 for ; Sat, 02 Mar 2019 18:44:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=EDa97cDj170v7MFFci4HhCOokHKaKNa94hvHYShQLGc=; b=qGjRFGFSG5aua4aUQg0jFv2LK8yAxrcFmxtK+7aHJ7A4yo63rzmiG/EOXPvLVjuZog RKKbT0+Ra1KUy1lW9vGKw8INP198PO1rwQPtxZN+SZvKar2OTY4ZkMYEfZHgyIUybUlT EJ5FqEcM9Tg0bB3UOfu6RELZDTUtI8BsSk08aW363/SxhDYsA/APrGeDxrcFPf1kbk8H eBXMvOPUXn0rWecXnti3k4XoTlnjJMw6FhTgtDAmvK0jxpT04IedCImIf5XOkAE6tsxe ma8ll/Uoxqc7mneQ6+OYpngX8C21FjdlTVdPO1gkUQu8HFtUDHSrftgzWEtcNJxTwYKf ZtWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=EDa97cDj170v7MFFci4HhCOokHKaKNa94hvHYShQLGc=; b=ThJSMPj1TsRUPjhYxufTZ0PkkINViSaBKZKWH92Ly9R5WvZ1eehtR7xeNN/kHV7Wz8 WsRMkJ8wyTAf8SKB3Ge6ajJ2a7FlqPoIHbHGtk3yfu8F6MWFMkqJ44GvS8tm/IItMT9j i9DYLG9gYVOTop0GadM+/R9icYY4F2x8DpeG6+T8LXJjIv+QKtCEXgF3Fl/Wv7+Qd15g YmXy1lUjOpijsuzQ4RPuUx1p1jukckvzd7ujEyAJr9oigY0rREyjW6rLLKLdDQAM5nGn 7KhHJ2pevYDdmhaf67Ro0gYC1FRZ2WRXiCAo7VrbkcLcSGYcVzeuUy3AxTMOBQzexbbJ kZGA== X-Gm-Message-State: APjAAAWsVdq2BUT0tvq7EIRfy5x3E9iYFb4uPxW33AHIU0EHCw8d4sqF mTO2ZkjJcDgL6kCpt2qlKJ+lCWkYaZ1AAEp7lKptwQ== X-Google-Smtp-Source: APXvYqzZJezlO5i0CK/3R+9HVn17sbOxxWRKYE6342Yg7XhkEib+yN7ypn9c0USKMqOhpgHLd+HzqRyiQebZkg373nE= X-Received: by 2002:a9d:66cb:: with SMTP id t11mr7644572otm.247.1551581050822; Sat, 02 Mar 2019 18:44:10 -0800 (PST) MIME-Version: 1.0 References: <20190216123859.GA24315@doctor.nl2k.ab.ca> <40ddfdd6-a1c4-ac00-f088-2880a7d372ce@smeets.xyz> <20190302094829.GA48523@admin.sibptus.ru> In-Reply-To: <20190302094829.GA48523@admin.sibptus.ru> From: Jason Tubnor Date: Sun, 3 Mar 2019 13:44:00 +1100 Message-ID: Subject: Re: Windows 2019 server To: Victor Sudakov Cc: freebsd-virtualization@freebsd.org X-Rspamd-Queue-Id: B1F536A520 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.97 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.97)[-0.968,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Mar 2019 02:44:12 -0000 On Sat., 2 Mar. 2019, 8:48 pm Victor S Which kind of storage did you present to the Windows 2019 guest, paravirtualized or ahcd-hd ? ahci-hd is what you want with bhyve and uefi Cheers From owner-freebsd-virtualization@freebsd.org Mon Mar 4 08:53:13 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 51F73150843F for ; Mon, 4 Mar 2019 08:53:13 +0000 (UTC) (envelope-from elenamihailescu22@gmail.com) Received: from mail-ot1-x344.google.com (mail-ot1-x344.google.com [IPv6:2607:f8b0:4864:20::344]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 24EB88E88B; Mon, 4 Mar 2019 08:53:12 +0000 (UTC) (envelope-from elenamihailescu22@gmail.com) Received: by mail-ot1-x344.google.com with SMTP id b3so3584354otp.4; Mon, 04 Mar 2019 00:53:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=uWE7fUoq1/FWZ8iE8kzGecY2Ipa1RMubPdJ+Dk4F6Ow=; b=IZuskJx2nevpOr05QM5ixYOq6C2EX9VqCp3nRU7DTAlcKmnbyu1djvfrpnVpKtf+fq QUBeI7eeIpYbNMZoNoiyy/WDqS9BevykWiJMsrGh+SYWNqzpLWbR6nCSU5ehjQ8e0K3L gGsoXFYIjwyhvUHIC1+k819wwDcyWtmEH3OjWUQEmFQZYfRcVTIqvYhiXql1ZhtOGaO7 G/PmUBBATpUsWutNuE6buYYzo8Y2yrfemDoTv9NIQadjhYfYHrpI6d2VLFeWh97JEMQH 0ZgnSPuyOpZOsG7r/VFJnxZS/Xq9j0eiVIoCidVlhFGkYISDc54HAPUJOk/8MnZcB8Qv X/vQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=uWE7fUoq1/FWZ8iE8kzGecY2Ipa1RMubPdJ+Dk4F6Ow=; b=n4X8lksDWP8MFsR+CD7A4LJ+bPnSX3Ku3DDadzwIHVsdZJ0DLWsOqWEzgWgkDQeVNH SEDYOpK2MT1sOiMgclHVOj8HEYZRfMYftB/qmYXt1n4iiDw4uQI0ZfT4sF0qDdHR02hy HvBJHSczrU/HrVmntgswqNDxkHLqLbdolfyd9vWMgD5ofauY/JfPxUIBwaNsymoZz0ZD JoRY83PofYYY4aG7TZClvCUdPIMIJuIx7lusJzhUmcYstnfLXlQS8+dvG6Y7ELgYIVy1 AyF2vMEQ7VMWTR6LcabUMFqsO2t4OdmKoY3rifBX9HMr4Bd5p13EsehqZuKVbnSGJhnd Y6MQ== X-Gm-Message-State: APjAAAVoDkC6/KojFPnXJ9EXDldRmad3jG2AOzAj/RageXdEwNLbtbj3 8C9wbnLRFQGCHk0VZuxhTao4D5su0Om84JYfhWIgl06FZco= X-Google-Smtp-Source: APXvYqycnRNgIFNh5rJ7INvNtaaSUKJ1wtpSeDw/trZwNB+wn0xbLnEexnyo25U8a73CHFhJ7w7XaNeXEYy5Gp0XRgo= X-Received: by 2002:a9d:22a4:: with SMTP id y33mr11164987ota.218.1551689590845; Mon, 04 Mar 2019 00:53:10 -0800 (PST) MIME-Version: 1.0 References: <20181016162409.GA5066@raichu> <74db03c7-2873-68a9-1873-f3e3b259e505@rice.edu> <6D4C9C61-259F-4D36-B690-E54563C8E2FF@gmail.com> <25e735cb-187b-8b38-210e-80d28302c7fb@FreeBSD.org> <20181024181331.GH45118@raichu> <20181026215929.GC33894@raichu> <245f33cf-1c71-1c15-80e5-0686e992df9a@FreeBSD.org> In-Reply-To: From: Elena Mihailescu Date: Mon, 4 Mar 2019 10:52:44 +0200 Message-ID: Subject: Re: bhyve guest's memory representation & live migration using COW To: John Baldwin , Mark Johnston Cc: Matthew Grooms , Mihai Carabas , alc@rice.edu, Tycho Nightingale , araujo@freebsd.org, Mihai Carabas , Darius Mihai , Sergiu Weisz , Michael Dexter , alc@freebsd.org, Patrick Mooney , freebsd-virtualization@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 24EB88E88B X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=IZuskJx2; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of elenamihailescu22@gmail.com designates 2607:f8b0:4864:20::344 as permitted sender) smtp.mailfrom=elenamihailescu22@gmail.com X-Spamd-Result: default: False [-2.73 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.994,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_SPAM_SHORT(0.58)[0.584,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCPT_COUNT_TWELVE(0.00)[14]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[4.4.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE(-0.31)[ip: (3.26), ipnet: 2607:f8b0::/32(-2.69), asn: 15169(-2.03), country: US(-0.07)]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] X-Mailman-Approved-At: Mon, 04 Mar 2019 12:15:38 +0000 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Mar 2019 08:53:13 -0000 On Thu, 7 Feb 2019 at 14:04, Elena Mihailescu wrote: > > On Fri, 14 Dec 2018 at 17:02, Elena Mihailescu > wrote: > > > > On Thu, 8 Nov 2018 at 20:11, John Baldwin wrote: > > > > > > [ Adding Patrick Mooney who hacks on bhyve @ Joyent to the cc ] > > > On 11/7/18 2:09 PM, John Baldwin wrote: > > > > On 10/29/18 8:39 AM, Elena Mihailescu wrote: > > > >> On Sat, 27 Oct 2018 at 00:59, Mark Johnston wrote: > > > >>> > > > >>> On Thu, Oct 25, 2018 at 02:06:24PM +0300, Elena Mihailescu wrote: > > > >>>> Hi all, > > > >>>> > > > >>>> > > > >>>> On Thu, 25 Oct 2018 at 02:49, Matthew Grooms wrote: > > > >>>>> > > > >>>>> On 10/24/2018 1:13 PM, Mark Johnston wrote: > > > >>>>>> On Tue, Oct 23, 2018 at 08:44:22AM -0700, John Baldwin wrote: > > > >>>>>>> Mark Johnston and I talked a bit about this at the MeetBSD conference last week. > > > >>>>>>> I had talked to Mark about Alan's note about msync() as I wasn't quite sure what > > > >>>>>>> he meant by that and what he suggested is something that I think is a simpler > > > >>>>>>> version of your proposal. Rather than creating an entirely separate set of page > > > >>>>>>> tables, we could perhaps add a second dirty bit for a vm_page (I think in this > > > >>>>>>> case it would be fine to have a single bit rather than a mask) that would be the > > > >>>>>>> "VMM dirty" bit. Anytime the VM would mark a page as dirty due to checking PTE > > > >>>>>>> bits and propagating them to the vm_page_t it would also set the VMM dirty bit > > > >>>>>>> (I believe you could perhaps just do this in vm_page_dirty?). However, the VMM > > > >>>>>>> dirty bit would only be cleared during a migration before starting the copy of a > > > >>>>>>> page, and the VM system would not clear this bit when it clears its own notion of > > > >>>>>>> dirty bits. Note that this also means that migration scans might need to check > > > >>>>>>> the PTEs and propagate dirty bits "up" via vm_page_dirty() before checking the > > > >>>>>>> VMM dirty bit. This is probably a lot less work than reviving submaps and I > > > >>>>>>> think it might be a small diff if it can be centralized in just vm_page_dirty() > > > >>>>>>> (and if a single bit is ok vs the mask we use for the existing dirty bits). > > > >>>>>>> > > > >>>>>>> Mark, does that description sound accurate? > > > >>>>>> That's basically what I had in mind as well. In my head the scan looks > > > >>>>>> roughly like this: > > > >>>>>> > > > >>>>>> foreach page m in guest physical address space: > > > >>>>>> vm_page_test_dirty(m); > > > >>>>>> if ((m->oflags & PG_VMM_DIRTY) == 0) > > > >>>>>> continue; > > > >>>>>> vm_page_xbusy(m); > > > >>>>>> pmap_remove_write(m); > > > >>>>>> m->oflags &= ~PG_VMM_DIRTY; > > > >>>>>> > > > >>>>>> vm_page_xunbusy(m); > > > >>>>>> > > > >>>>>> Note that the test for the dirty bit is racy; there needs to be a final > > > >>>>>> scan which unconditionally clears PG_RW and tests for PG_M on each page > > > >>>>>> while the VM is paused. > > > >>>>> > > > >>>>> At the risk of sounding painfully ignorant: Does the VMs page list need > > > >>>>> to be locked in order to walk it safely while the VM is running? If so, > > > >>>>> would the 'transfer page' option be too slow to perform inside this > > > >>>>> loop? Does the 'make a copy' alternative entail copying all the dirty > > > >>>>> pages so they can be transferred after dropping the lock? > > > >>>>> > > > >>>> > > > >>>> I also have some questions about this approach: > > > >>>> > > > >>>> 1. As I understood from John, your suggestion is to add a new field in > > > >>>> the vm_page structure for the guest (vmm_dirty). I was looking into > > > >>>> the vm_page structure and I don't think I can use a bit from the > > > >>>> flags, aflags or oflags fields since I will end up mixing the fields' > > > >>>> meanings. > > > >>>> But, if I add a new field, it means I will use additional space which > > > >>>> brings me to the question: this idea sounds a lot like the first idea > > > >>>> that came to mind when I started implementing a migration feature for > > > >>>> bhyve (my idea was to use a counter that would be incremented each > > > >>>> time a page was written). But, I received negative feedback from a lot > > > >>>> of people that it would imply to modify the vm_page structure for just > > > >>>> one use-case (bhyve migration) and that the changes will affect the > > > >>>> entire operating system and that my patch might not be accepted into > > > >>>> the mainstream because of this. What do you think? > > > >>>> > > > >>>> Also, I am worried about the memory overhead this additional bit > > > >>>> (field actually) will bring to the FreeBSD. > > > >>>> > > > >>>> Also, if I should use one of the unused bits in the vm_page flags > > > >>>> (flags, oflags, aflags), it could happen that until my patch will be > > > >>>> accepted, that particular bit will be use for other purposes and a > > > >>>> conflict will occur maybe without realizing. > > > >>> > > > >>> In this hypothetical scheme, the VM subsystem would be aware of the > > > >>> extra bit used to store the vmm_dirty state, so I don't think we need to > > > >>> worry about collisions. Both the "flags" and "oflags" fields have a > > > >>> number of spare bits. We definitely want to avoid adding any new > > > >>> fields to struct vm_page, indeed. > > > >>> > > > >>>> 2. I am not very well documented regarding the physical page tables > > > >>>> that a host would normally use. For the EPTs I know that those bits > > > >>>> are "sticky" (once set, only the software can unset them - the > > > >>>> hardware will only set them and never unset). Is it the same thing for > > > >>>> the normal page tables and only the software can unset them? Or are > > > >>>> there any other hardware mechanisms (like MMU) able to unset them? > > > >>> > > > >>> The access and modified bits (PG_A and PG_M) are only ever cleared by > > > >>> software, regardless of the page table format. > > > >>> > > > >>>> 3. Also, are the set/unset operation strictly related to the vm_page > > > >>>> structure? Is there any (software) mechanism that can set and unset > > > >>>> them and the virtual memory system to not update the dirty flag? > > > >>> > > > >>> I don't quite follow your question, but I'll try to explain the > > > >>> interaction between the pmap and the representation of the accessed > > > >>> and modified bits in struct vm_page. For the modified bit of mappings > > > >>> of a page m, m->dirty just caches the value of the PG_M. If I want to > > > >>> ask the question, "is m dirty?" and m->dirty != 0, then I'm done. > > > >>> If m->dirty == 0, I must called pmap_is_modified(m) to see if any > > > >>> mapping of m has PG_M set, and if so, the value is propagated to > > > >>> m->dirty. The process of cleaning a page has multiple steps: > > > >>> > > > >>> 1. Busy the page > > > >>> 2. Mark its mappings as read-only and clear PG_M from any mappings > > > >>> 3. Write the page's contents to disk > > > >>> 4. Set m->dirty = 0 (assuming the write was successful) > > > >>> 5. Unbusy the page > > > >>> > > > >>> Step 2 will trigger a page fault on any attempt to write to the page. > > > >>> The page fault handler will sleep while the page is busy, so any thread > > > >>> attempting to simultaneously modify the page must wait until step 5, at > > > >>> which point it will proceed and upgrade the mapping's permission from > > > >>> read-only to read-write. Once the write completes, we know that the > > > >>> contents of m are identical to its copy on disk, so the page can be > > > >>> marked clean. > > > >>> > > > >>> The accessed bit is handled a little bit differently; it is used mainly > > > >>> to decide whether to evict a page. The page daemon uses > > > >>> pmap_ts_referenced() to check for and clear PG_A bits in mappings of a > > > >>> page m. If references are found, m->act_count is increased, ensuring > > > >>> that the page won't be evicted right away. PG_A is handled a bit > > > >>> differently from PG_M in that PG_A is cleared once the reference has > > > >>> been reflected in the vm_page struct, while PG_M is left set until the > > > >>> page is cleaned. > > > >>> > > > >>>> 4. Also, I do not understand very well the way the virtual memory > > > >>>> system behaves. I will write what I understood from our previous > > > >>>> conversations and I want you to correct me if I am wrong. > > > >>>> > > > >>>> When a page is written, the hardware (MMU) will set the dirty/modified > > > >>>> bit. From time to time, a virtual management routine will walk through > > > >>>> all the pages from the page tables (all the page table entries to be > > > >>>> more precise) and if the dirty flag is set, then the vm_page that > > > >>>> points to that physical page will be updated (vm_page ->dirty will be > > > >>>> set to VM_PAGE_BITS_ALL). Also, from time to time, other virtual > > > >>>> memory system routines will check the dirty flag from vm_pages and > > > >>>> will commit the changes and will unset the dirty bit (e.g. the page > > > >>>> laundering process). > > > >>> > > > >>> Right, this is what I tried to describe above. > > > >>> > > > >>>> So, the whole idea is to introduce another field into the vm_page that > > > >>>> will be set when the vm_page will be set as dirty, but not cleared > > > >>>> when the vm_page dirty field will be cleared. Because the page > > > >>>> walking/physical dirty bit check is happening from time to time, I > > > >>>> have to do another manual page walk before starting the migration > > > >>>> round to see if other pages have become dirty in the meanwhile (from > > > >>>> the last virtual memory subsystem scan). > > > >>>> > > > >>>> Is that correct? > > > >>> > > > >>> Yes, you can't rely on the VM system to provide an up-to-date value of > > > >>> the page's dirty state. m->dirty is a lazy cache. > > > >>> > > > >>>> 5. Another issue that concerns me is the guest memory dual view. When > > > >>>> the guest will write something in its memory, through the EPT > > > >>>> mechanism, the corresponding page entry from the guest pmap (that has > > > >>>> the EPT type) will be updated and the dirty flag will be set. > > > >>>> > > > >>>> But what happens when the host will write something into the guest > > > >>>> memory (e.g. virtio related operations or when the guest wants to read > > > >>>> something from disk)? A corresponding entry from the host page tables > > > >>>> will point to the same physical page that has also an entry into the > > > >>>> EPTs? > > > >>>> > > > >>>> I am kinda lost here and I am not sure if the proposed algorithm will > > > >>>> cover the case when the host will write something into the guest's > > > >>>> memory. > > > >>> > > > >>> I think it can. Note that vm_page_test_dirty() calls > > > >>> pmap_is_modified(), which searches _all_ mappings of m. In particular, > > > >>> if both the guest and host have mapped the same page, it should catch > > > >>> modifications from either pmap. > > > >>> > > > >>>> 6. I was looking over the Mark's pseudocode idea, and I do not > > > >>>> understand why the write access has to be removed from a page. > > > >>> > > > >>> The idea is to ensure that the page transfer process has a consistent > > > >>> view of the page's contents. It may be unnecessary in your > > > >>> case since there is a final step where the guest is paused and we > > > >>> perform a final scan for modifications. > > > >> > > > >> Thank you for your response. It clears a lot of the questions I had. I > > > >> have an idea about how I could implement this. I'll start the > > > >> implementation and I'll come back with an update as soon as I have > > > >> something. > > > > > > > > I did have one later thought about this scheme which is that swapping has > > > > no way to save/restore the VMM dirty bit. I had originally thought about > > > > just setting a flag on the VM object to enable setting the VMM dirty bit > > > > and setting that on the VM object when it was created. You would then > > > > never swap a page that had the VMM dirty bit set to avoid losing that bit. > > > > However, this would effectively prevent swapping of guest memory pages. > > > > Instead, we could wait and only set the VM object flag at the start of a > > > > migration and then make an initial sweep over the VM object marking all > > > > the pages as "VMM dirty". During the first scan of memory this would > > > > result in copying all of guest memory, but that's ok. After the first > > > > pass, the "VMM dirty" bits would then work "correctly". > > > > > > Another option for swapping that might be simpler would be to just always > > > set the VMM dirty bit when swapping a page back in unconditionally. This > > > would avoid the need for trying to block swapping of a page and is probably > > > less work overall. If you are under memory pressure this might degrade into > > > a case where your live migration can't make forward progress because it > > > swaps out clean pages and then has to swap them back in for the next sweep > > > (because we will have to assume that any swapped out pages are dirty and > > > always treat them as dirty during a scan), but that may also end up being a > > > rare case in practice. > > > > > > -- > > > John Baldwin > > > > > > > Hi all, > > > > As I've discussed with John in the previous bhyve call, I'm writing an > > email on this thread to inform you all about my progress towards > > implementing a live migration feature in bhyve. > > > > I've pretty much implemented the framework, but I have some issues > > regarding the live migration procedure behaviour. > > > > It should be good to inform you that I have some constraints regarding > > the actual implementation: > > - the guest should have wired memory (as John intuited in a previous > > email, there are some issues regarding some pages that are not present > > - for some indexes, vm_page_lookup [1] returns NULL; Considering > > Matthew's suggestion, I've constrained the live migration feature for > > wired memory); > > - the guest should have the memory size less than the lowmem_limit (I > > think it is something around 3GB right now). This is just for > > commodity, if I could correctly live migrate the lowmem segment for a > > guest, then I could quite easily extend the implementation to the > > highmem segment. > > > > ======== ALGORITHM =========== > > This being said, the algorithm should do the following things: > > send_memory [2]: > > 1. // First Round - send all pages > > 2. // use an array to keep the pages that should be migrated > > 3. page_list_indexes = all_guest_pages(); > > 4. lock_all_vCPUs(); > > 5. /* We've already saved the pages that should be migrated, so now > > we could clear the dirty bits for all pages. (see later > > observations)*/ > > 6. clear_migration_dirty_bits(); [3] > > 7. unlock_all_vCPUs(); > > 8. send_pages(); /* see bellow */ > > 9 > > 10. for each non final round do > > 11. page_list_indexes = [] > > 12. lock_all_vCPUs() > > 13. search_dirty_pages(page_list_indexes); [4] > > 14. clear_migration_dirty_bits(); > > 15. unlock_all_vCPUs(); > > 16. send_pages(); /* see bellow */ > > 17.end for > > 18. > > 19. // Final Round > > 20. page_list_indexes = [] > > 21. lock_all_vCPUs() > > 22. search_dirty_pages(page_list_indexes); > > 23. send_pages(); > > 24. ... / send CPU state, kernel structs' state, devices' state > > 25. freeze CPUs; unlock_all_vCPUs() and finish the source guest > > > > send_pages [5]: > > 1. while (has_pages_to_be_sent) > > 2. retrieve_N_pages_from_kernel [6] > > 3. send(pages, /* through */ socket) > > 4. update(has_pages_to_be_sent) > > 5. end while > > > > For the receiving part, the algorithm is simpler: I start a guest, and > > right before the virtual cpus are spinning up, I wait for the > > migration info to be received and update the pages accordingly [7]. > > > > ======== TESTS AND RESULTS =========== > > > > As for the tests, I've tested live migration using the following two scenarios: > > Scenario 1: power on the guest; when login prompt appears live migrate the vm; > > Scenario 2: power on the guest; login as a user (root); execute while > > true; do echo $((i=i+1)); sleep 1; done; > > > > As for the results: > > - When testing Scenario 1, all the guest seemed to be migrated well > > (although sometimes, after the login, the guest would fail - kernel > > panic in guest) > > - When testing Scenario 2 and using a 256MB guest (or a 512MB guest), > > the migration was completed and the guest would continue executing the > > script from where it was stopped by the migration procedure > > - When testing Scenario 2 using a 1GB guest (or a 2GB guest), the > > migration fails, sometimes the sleep get stuck, sometimes I get kernel > > panic in guest ("supervisor page not " or from vmspace_exit() and > > pmap_remove_pages() and so on). > > > > It seems that there are some bugs into the implementation. > > > > ========== QUESTIONS ========== > > > > Sorry for the long intro, but I think that I should give details about > > the implementation so you could have a bigger picture about my > > questions. > > First of all, I would like to know that, if at least conceptually, the > > algorithm is fine (even if it can be improved). > > > > Second of all, I have some doubts regarding the way I should clean the > > migration dirty bits. I need a method that would clean the migration > > dirty bit and as well, the physical modified bit (so I could track the > > page level modifications that happen between two rounds). In the > > current implementation, I use vm_object_page_clean that should clean > > all the object pages (I've seen that it should sync the memory). > > > > Another approach I was trying to implement was to clean the migration > > bit and physical modified flag for each page after I copy it from the > > kernel space into the user space in order to migrate it. I was trying > > to use the pmap_remove_write() as Mark suggested, but it seems that I > > get a host kernel panic this way (something related to the wired > > memory: "panic: pmap_demote_pde: page table page for a wired mapping > > is missing". The backtrace shows that the panic happens when the > > get_page function calls pmap_remove_write and pmap_remove_write calls > > pmap_demote_pde_locked.). I've also tried pmap_clear_modify() but it > > didn't help a lot. > > > > So, what do you think? What should I use? I know that in the current > > implementation it may happen to clean the dirty bit for a page, that > > page to be modified in the meanwhile, transfer it and then transfer it > > again in the next round, but it should add redundancy into the code, > > not errors. > > > > > > =========== LINKS ============== > > > > [1] https://github.com/FreeBSD-UPB/freebsd/blob/projects/bhyve_migration_dev/sys/vm/vm_page.c#L1541 > > [2] Live Migration - Send Memory: > > https://github.com/FreeBSD-UPB/freebsd/blob/projects/bhyve_migration_dev/usr.sbin/bhyve/migration.c#L2294 > > [3] Clear Guest Dirty Bits: > > [3.1] Call vm_page_clear_vmm_dirty_bit: > > https://github.com/FreeBSD-UPB/freebsd/blob/projects/bhyve_migration_dev/sys/vm/vm_page.c#L1354 > > for each of the guest pages > > [3.2] Call vm_object_page_clean(object, 0, 0, OBJPC_SYNC); for > > the object that contains the lowmem segment. The vm_object_page_clean > > function is already implemented in FreeBSD: > > https://github.com/FreeBSD-UPB/freebsd/blob/projects/bhyve_migration_dev/sys/vm/vm_object.c#L883 > > I know that this is not the intended way of using this > > function, but I needed something that should clear the physical > > modified bit without affecting the way the virtual memory subsystem > > behaves. > > [3.3] Here is the graph of vm_object_page_clean function all of > > its usages: http://www.leidinger.net/FreeBSD/dox/vm/html/d4/dfe/vm__object_8h.html#acb8268e5afa032b6213738cceba196a2 > > [4] Search dirty pages: > > [4.1] iterate through all of the guest pages; check if page is > > dirty and if so, update field in array accordingly: > > https://github.com/FreeBSD-UPB/freebsd/blob/projects/bhyve_migration_dev/sys/vm/vm_radix.c#L827 > > [4.2] to test if page is dirty, vm_page_test_vmm_dirty is called: > > https://github.com/FreeBSD-UPB/freebsd/blob/projects/bhyve_migration_dev/sys/vm/vm_page.c#L1360 > > [4.3] vm_page_test_vmm_dirty calls vm_page_test_dirty that could > > call vm_page_dirty and updates VPO_VMM_DIRTY and dirty flag: > > https://github.com/FreeBSD-UPB/freebsd/blob/projects/bhyve_migration_dev/sys/vm/vm_page.h#L735 > > [5] https://github.com/FreeBSD-UPB/freebsd/blob/projects/bhyve_migration_dev/usr.sbin/bhyve/migration.c#L2119 > > [6] Copy object pages from kernel space to userspace > > [6.1] Call vm_object_get_page for each page that I am interested > > in: https://github.com/FreeBSD-UPB/freebsd/blob/projects/bhyve_migration_dev/sys/amd64/vmm/vmm.c#L3362 > > [6.2] vm_object_get_page implementation: > > https://github.com/FreeBSD-UPB/freebsd/blob/projects/bhyve_migration_dev/sys/vm/vm_object.c#L2463 > > [7] https://github.com/FreeBSD-UPB/freebsd/blob/projects/bhyve_migration_dev/sys/vm/vm_object.c#L2483 > > > > Thank you very much for your help, > > Elena > > > Hi all, > > I have some issues related to the dirty bit mechanism to detect the > memory changes between two migration rounds and I wanted to know if > anyone can give me some ideas about the way I should debug this. The > idea is that the migrated memory and the guest memory differ at the > end of the live migration. > > The dirty bit implementation is the following: > - use a bit in vm_page->oflags (mask: 0x80) > - update bit in vm_page_dirty() and vm_page_dirty_KBI. > - when searching pages for migration: > for each page: > vm_page_test_dirty(); > test vmm dirty bit > if bit is set then page is add in a to_be_transmitted list > - when sending pages > for each page in to_be_transmitted list: > xbusy_page; > copy_page_to_local_buffer(); > xunbusy_page; > > As you may see, I do not clear the virtual machine dirty bit after I > copy a page. I think that this is not relevant right now. All the > pages that were ever modified since the guest has been started should > be transferred, but the final memory should not be affected. However, > in this case, the guest memory and the migrated memory differ which > led me to the idea that the memory changes detection algorithm does > not work properly. Moreover, when the number of rounds is 1 (warm > migration), then the migration is correct and the guest memory and the > migrated memory are the same, so I will consider (for this moment) > that the getter and setter functions for guest is OK. > Hi all, I'm writing this e-mail to keep you in the loop with the live migration progress and to ask for advice regarding the process of cleaning the dirty bit. At Rod Grimes's suggestion, I added the virtualization list at CC. I had some issues with the vmm_dirty bit (a bit from oflags) and initially I thought that there some issues related to the fact that in some cases oflags is directly updated (oflags = ) and with the fact that the dirty bit is set otherwise that using vm_page_dirty(), but it seems that the issues I had were from other sources and now, after cleaning the code, and refactored a bit, it seems to work fine. Some of the code snippets that I thought that could affect the migration are at [1]. However, if I dump the guest memory after the live migration (the guest is not yet started after migration) and compare it with the source guest's ctx->baseaddr, there are some differences and I don't know from where they may come from. One of the things I must implement is related to the dirty bit mechanism. For now, I do not clear the dirty bit after I migrate a page. This is not a wrong implementation because in every round, I migrate all the pages that have been modified ever (since the guest was started), but it is not an optimal implementation. To clear the dirty bit, I've tried different mechanism such as: - pmap_remove_write - kernel panic: pde entry not found for a wired mapped page - vm_object_page_clean - vm_map_sync -> kernel panic - pmap_protect - remove write permission, copy page and add write permission -> kernel panic; pmap_demote_pde: page table page for a wired mapping is missing Any idea about a way of forcing a page to be cleared (so the physical dirty bit to be cleared)? [1] https://github.com/freebsd/freebsd/blob/master/sys/vm/vm_page.c#L2627 https://github.com/freebsd/freebsd/blob/master/sys/vm/vm_fault.c#L1761 https://github.com/FreeBSD-UPB/freebsd/blob/master/sys/vm/vm_page.c#L1912 https://github.com/FreeBSD-UPB/freebsd/blob/master/sys/vm/vm_page.c#L2105 https://github.com/FreeBSD-UPB/freebsd/blob/master/sys/vm/vm_page.c#L2117 https://github.com/FreeBSD-UPB/freebsd/blob/master/sys/vm/vm_page.c#L1204 Thank you, Elena From owner-freebsd-virtualization@freebsd.org Thu Mar 7 17:05:30 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7CEED150C042; Thu, 7 Mar 2019 17:05:30 +0000 (UTC) (envelope-from dariusmihaim@gmail.com) Received: from mail-qk1-x732.google.com (mail-qk1-x732.google.com [IPv6:2607:f8b0:4864:20::732]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 06D568F175; Thu, 7 Mar 2019 17:05:29 +0000 (UTC) (envelope-from dariusmihaim@gmail.com) Received: by mail-qk1-x732.google.com with SMTP id y15so9453970qki.8; Thu, 07 Mar 2019 09:05:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=U50AT0OmcUeet4iewjdehC4T4+QSqmB88pRg2bSH5AA=; b=ROBXgVgWyGs/DeULkMUPIczOm82Mb3q0eBzbLw+H1LCl6AaF2zuDbEyr5kJb4ib7WH g9qXa5qpjleICHeYn4xIv5g4Enx2w2kVSRob3qKFKtXpGgru0Fc0UD1Ee+ixBJ5+QQot uO2077VT1rjfQ799pXBjmwGZyIBxIJ6p1zcaCr4/505o7fuEes9Jafc1AS1x1adVJ7jT rdXnomKEiiUoq4Projsu/sY6cM1ADEvLOD6/Y7PTAi2UQNYYJhBHJQ1Bk5zGhvhZKmE7 zRvZRaCYCVJPxhnR34vQmwj2571T1vtet3ozAqQ0QMDUDpWvR7Cv0wt+J7VSApo5idJo xjoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=U50AT0OmcUeet4iewjdehC4T4+QSqmB88pRg2bSH5AA=; b=uQXxzFuapr40DCtRoFD7mKgMjBgp6txz0IHsmbY5ZN20mR74ko9oXI4QihA9yVrWPo d+B6MO4PgZke0TIvBdXX5Yh0a0j4kv4xEVAFIW/TaNyShD/oS+mTp2ZfQhlbf9kDZayU MnRv/nvtwBdlUsQ6JH/8953IlS2Luq0a9skDEd+2qJfELPV480gqkQkKpZ+8qYTVbdLt DEDZGWdAjNwkz2hWZe05D0Kx+KnkF0ANVmgSvbEYs5g60mkxIk3g0X10OkR0iRfY2vwz sYIurnSFmTKVMLtTRafQEfmzqt9iNMxDIKyV11gGmM6Km2cXooDn+ZvTamQKqKuuncFW /nEA== X-Gm-Message-State: APjAAAU+7UVC/yFiMzqbiDXOvwGohwB11Tv0nSNBHg/cN1DAXDtz7cL5 1WV67gIDBx+j5Lz0Bmv6s6jHxqQrjrCHLlr4jbjBE1ZM X-Google-Smtp-Source: APXvYqwbWLfLQUiInuZSilY1+0yDUTCasIqvwgUKEhsYE3RPt0xRiXxgIt3zUcwRww9aeH+1fEKEdMaY3TwFEdOf/KM= X-Received: by 2002:a37:d88:: with SMTP id 130mr10617820qkn.314.1551978328204; Thu, 07 Mar 2019 09:05:28 -0800 (PST) MIME-Version: 1.0 From: Darius Mihai Date: Thu, 7 Mar 2019 19:04:52 +0200 Message-ID: Subject: bhyve - Snapshot Save and Restore Feature Revision To: freebsd-virtualization@freebsd.org, freebsd-amd64@freebsd.org, freebsd-current@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 06D568F175 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=ROBXgVgW; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of dariusmihaim@gmail.com designates 2607:f8b0:4864:20::732 as permitted sender) smtp.mailfrom=dariusmihaim@gmail.com X-Spamd-Result: default: False [-6.75 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2.3.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.79)[ip: (-9.14), ipnet: 2607:f8b0::/32(-2.72), asn: 15169(-2.05), country: US(-0.07)]; NEURAL_HAM_SHORT(-0.94)[-0.941,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Mar 2019 17:05:30 -0000 Hi everyone, We have created revision for the bhyve snapshot save and restore feature on phabricator at https://reviews.freebsd.org/D19495 . Any feedback is greatly appreciated. Thanks, Darius From owner-freebsd-virtualization@freebsd.org Thu Mar 7 19:48:24 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2674D152356E for ; Thu, 7 Mar 2019 19:48:24 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-it1-x136.google.com (mail-it1-x136.google.com [IPv6:2607:f8b0:4864:20::136]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DC3016FFEF; Thu, 7 Mar 2019 19:48:22 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-it1-x136.google.com with SMTP id 188so17768455itb.0; Thu, 07 Mar 2019 11:48:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=jQNZyoUF66ID+S/N7W6HqXHmyKdyONlpJGXNlfqjeTQ=; b=S4cGbFGP2NzsBX3kXc7bNcrqf4L5nTabaDOoXqNDQua8Pl8fUtNdaOqrzuEeySdWe2 u68czIf8yEuK81PJ8AcN0XcnjUo21W0fJ6ZcMHlLZwGk7AY+lbglJVagA/jXDKNK1haS oL74q8+L8g5UNkCon1KvLsQc6TT9G4IheytJldmCIAdHxVcItsCXnXOIMWCmVvdaL0EF WZcRQGFyKr0hBiRVJFL4Ooduka3pK8Z3B0qi6gyl8hUPhLLie6VhhvL15xnLOnzt4D6N 453Smj/cRac9n1f3sTODEf7N7JS0IbHBE1iFvkQNY4saqRNyMqGDrelXvoFF/SGKGsFf pE3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=jQNZyoUF66ID+S/N7W6HqXHmyKdyONlpJGXNlfqjeTQ=; b=C4Blua5tIzMK47ipRKDafJEG4/b+9NRYtBzJPzS2huBWwXRvRzlHFub2qiOk7QKs/k IQdgoDHiVpfI1vBmIcykcL3cvHKVNabRIJNyNLhDQc7bocBAaTSH4KF0nuZINubzCTQW VaScGKicJitDr/YUBVpqd4ojGhvfWkYhCO2TaXrK2+3sWaZtPzSRgsi7BUjWLV/RKkgT 1ogHA18BXBaUqhcCau42jWvzlz8VDysHvxcms5DVXAathEd2r3b2+QWSVLT+AtP+vEPR +OsJdf0PHzP5jqNNaBvzIJf7g6aEty7+jE/r6mbaOMxqoEgQN0VqYQpfrhhvewNtdgFt y/+A== X-Gm-Message-State: APjAAAXrnkZvE79kPJpRr0kiWhPpbgzdmr9EUxGlTBqG0DpFOEOHRL5x +2MaCsLQ96pkq5yRFrQNGx0= X-Google-Smtp-Source: APXvYqwKw/RzyHBAvMniRXUGOVr+SBBHcsn3HjbNbrFVH06HN4jsL/13SmP0o8HejIHucHmuz+y4DQ== X-Received: by 2002:a24:1694:: with SMTP id a142mr6244344ita.22.1551988102050; Thu, 07 Mar 2019 11:48:22 -0800 (PST) Received: from raichu (toroon0560w-lp140-01-69-159-36-102.dsl.bell.ca. [69.159.36.102]) by smtp.gmail.com with ESMTPSA id t68sm2877390ita.4.2019.03.07.11.48.19 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 07 Mar 2019 11:48:20 -0800 (PST) Sender: Mark Johnston Date: Thu, 7 Mar 2019 14:41:39 -0500 From: Mark Johnston To: Elena Mihailescu Cc: John Baldwin , Matthew Grooms , Mihai Carabas , alc@rice.edu, Tycho Nightingale , araujo@freebsd.org, Mihai Carabas , Darius Mihai , Sergiu Weisz , Michael Dexter , alc@freebsd.org, Patrick Mooney , freebsd-virtualization@freebsd.org Subject: Re: bhyve guest's memory representation & live migration using COW Message-ID: <20190307194139.GA56002@raichu> References: <20181024181331.GH45118@raichu> <20181026215929.GC33894@raichu> <245f33cf-1c71-1c15-80e5-0686e992df9a@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.11.2 (2019-01-07) X-Rspamd-Queue-Id: DC3016FFEF X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=S4cGbFGP; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::136 as permitted sender) smtp.mailfrom=markjdb@gmail.com X-Spamd-Result: default: False [-5.03 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.90)[-0.897,0]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; FREEMAIL_TO(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWELVE(0.00)[14]; RCVD_IN_DNSWL_NONE(0.00)[6.3.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.42)[ip: (-7.28), ipnet: 2607:f8b0::/32(-2.72), asn: 15169(-2.05), country: US(-0.07)]; MID_RHS_NOT_FQDN(0.50)[] X-Mailman-Approved-At: Thu, 07 Mar 2019 23:22:30 +0000 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Mar 2019 19:48:24 -0000 On Mon, Mar 04, 2019 at 10:52:44AM +0200, Elena Mihailescu wrote: > I'm writing this e-mail to keep you in the loop with the live > migration progress and to ask for advice regarding the process of > cleaning the dirty bit. At Rod Grimes's suggestion, I added the > virtualization list at CC. > > I had some issues with the vmm_dirty bit (a bit from oflags) and > initially I thought that there some issues related to the fact that in > some cases oflags is directly updated (oflags = ) and with the > fact that the dirty bit is set otherwise that using vm_page_dirty(), > but it seems that the issues I had were from other sources and now, > after cleaning the code, and refactored a bit, it seems to work fine. > Some of the code snippets that I thought that could affect the > migration are at [1]. > > However, if I dump the guest memory after the live migration (the > guest is not yet started after migration) and compare it with the > source guest's ctx->baseaddr, there are some differences and I don't > know from where they may come from. > > One of the things I must implement is related to the dirty bit > mechanism. For now, I do not clear the dirty bit after I migrate a > page. Note that there is code that only calls vm_page_dirty(m) if m->dirty != VM_PAGE_BITS_ALL. One example is in vm_page_advise(). So if the page is dirty from the kernel's point of view and you clear the VMM dirty bit, subsequent modifications may not be propagated to the vm_page state. > This is not a wrong implementation because in every round, I > migrate all the pages that have been modified ever (since the guest > was started), but it is not an optimal implementation. To clear the > dirty bit, I've tried different mechanism such as: > - pmap_remove_write - kernel panic: pde entry not found for a wired mapped page > - vm_object_page_clean > - vm_map_sync -> kernel panic > - pmap_protect - remove write permission, copy page and add write > permission -> kernel panic; pmap_demote_pde: page table page for a > wired mapping is missing > > Any idea about a way of forcing a page to be cleared (so the physical > dirty bit to be cleared)? I'm not sure exactly what you mean, but pmap_clear_modify(m) will clear the modification bit for mappings of m. It seems like you want something roughly like: vm_page_xbusy(m); if (pmap_is_modified(m)) m->dirty = m->vmm_dirty = VM_PAGE_BITS_ALL; pmap_clear_modify(m); vm_page_xunbusy(m); Or do we need to restrict the operation to a specific mapping of m? From owner-freebsd-virtualization@freebsd.org Fri Mar 8 16:57:05 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D662C1523077 for ; Fri, 8 Mar 2019 16:57:04 +0000 (UTC) (envelope-from jennifer@getofficecoffee.host) Received: from alias5s177136.cloud.flynet.pro (alias30s177136.cloud.flynet.pro [85.117.233.65]) by mx1.freebsd.org (Postfix) with ESMTP id B5BB882EFD for ; Fri, 8 Mar 2019 16:57:02 +0000 (UTC) (envelope-from jennifer@getofficecoffee.host) Received: from m1.bizquotes.click (unknown [10.20.15.1]) by alias5s177136.cloud.flynet.pro (Postfix) with ESMTP id 71439419DB for ; Fri, 8 Mar 2019 18:56:51 +0200 (IST) Received: from list.bizquotes.click (unknown [192.168.100.137]) by m1.bizquotes.click (Postfix) with ESMTP id 1D3658073 for ; Fri, 8 Mar 2019 18:12:25 +0200 (IST) DKIM-Filter: OpenDKIM Filter v2.11.0 m1.bizquotes.click 1D3658073 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bizquotes.org; s=zmail; t=1552061545; bh=kS1iCkDMD+iectx8xVbMiHFUXxGiDWDToFT0irmdTiY=; h=To:Subject:Date:From:Reply-To:List-Unsubscribe:From; b=Xj9DCEZstPvrd9YbtPyrKVZ6nsE3QAmEOX7CvrRf/nzXkHBXGwvdGEcpe69b4HRE2 uVGxGjMBPQjbZMU0epw2fw8tjNtPQhetzg0Onqg8tLb1OuIMtvB0og32YtxA+OLI7E NnsAssouMHu9+fJqBOB5O5lPdiZOgyyguy2W5sxQ= To: freebsd-virtualization@freebsd.org Subject: Get Coffee Quotes for Charlie's Choice Message-ID: Date: Fri, 08 Mar 2019 15:37:48 +0000 From: "Jennifer Burke" Reply-To: jennifer@getofficecoffee.host MIME-Version: 1.0 X-Mailer-LID: 41,41 X-Mailer-RecptId: 34790047 X-Mailer-SID: 23765 X-Mailer-Sent-By: 1 X-Rspamd-Queue-Id: B5BB882EFD X-Spamd-Bar: ++++ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bizquotes.org header.s=zmail header.b=Xj9DCEZs; spf=pass (mx1.freebsd.org: domain of jennifer@getofficecoffee.host designates 85.117.233.65 as permitted sender) smtp.mailfrom=jennifer@getofficecoffee.host X-Spamd-Result: default: False [4.94 / 15.00]; HAS_REPLYTO(0.00)[jennifer@getofficecoffee.host]; R_SPF_ALLOW(-0.20)[+ip4:85.117.233.0/24]; TO_DN_NONE(0.00)[]; REPLYTO_ADDR_EQ_FROM(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[bizquotes.org:+]; MX_GOOD(-0.01)[mx20.getofficecoffee.host,mx10.getofficecoffee.host]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(0.00)[country: RU(0.00)]; INTRODUCTION(2.00)[]; ASN(0.00)[asn:51724, ipnet:85.117.233.0/24, country:RU]; HAS_INTERSPIRE_SIG(1.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[bizquotes.org:s=zmail]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_SPAM_SHORT(0.42)[0.422,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; HAS_PHPMAILER_SIG(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; HAS_LIST_UNSUB(-0.01)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_SPAM_MEDIUM(0.98)[0.981,0]; DMARC_NA(0.00)[getofficecoffee.host]; NEURAL_SPAM_LONG(0.96)[0.959,0]; MIME_TRACE(0.00)[0:+,1:+] Content-Type: text/plain; format=flowed; charset="UTF-8" Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Mar 2019 16:57:05 -0000 Hi, My name is Jennifer and I'm with GetOfficeCoffee, a service that compares quotes for Coffee machines and Water and Ice dispensers for offices. Provide your employees or clients fresh brewed coffee and filtered water. In order to compose a Quote, the following information will be needed: * First and Last Name * Company Name * Phone number * Zip Code * Number Of Employees/Users * Service Needed (Coffee/Water/Ice) Finding the perfect vendor for your business won't cost a thing when replying with the above information. Best regards, Jennifer Burke Office Vendor Specialist Jennifer@GetOfficeCoffee.host Unsubscribe by reply back "REMOVE".