From owner-freebsd-virtualization@freebsd.org Sun Mar 17 16:22:35 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A7067154127E for ; Sun, 17 Mar 2019 16:22:35 +0000 (UTC) (envelope-from crowston@protonmail.com) Received: from mail1.protonmail.ch (mail1.protonmail.ch [185.70.40.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.protonmail.ch", Issuer "SwissSign Server Silver CA 2014 - G22" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E708C86516 for ; Sun, 17 Mar 2019 16:22:34 +0000 (UTC) (envelope-from crowston@protonmail.com) Date: Sun, 17 Mar 2019 16:22:29 +0000 To: "freebsd-virtualization@freebsd.org" From: Robert Crowston Reply-To: Robert Crowston Subject: GPU passthrough: mixed success on Linux, not yet on Windows Message-ID: Feedback-ID: 2OVbcR1yHYpdkD8cgQllkFwcuMVZg_LiVMMPvptooFDfHD_03MuQO4ZaF626jWHZYFEhNR2cmIbZ53j4QGWMBQ==:Ext:ProtonMail MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-1.2 required=7.0 tests=ALL_TRUSTED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail.protonmail.ch X-Rspamd-Queue-Id: E708C86516 X-Spamd-Bar: ------- X-Spamd-Result: default: False [-7.76 / 15.00]; HAS_REPLYTO(0.00)[crowston@protonmail.com]; R_SPF_ALLOW(-0.20)[+ip4:185.70.40.0/24]; FREEMAIL_FROM(0.00)[protonmail.com]; DKIM_TRACE(0.00)[protonmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[protonmail.com,quarantine]; MX_GOOD(-0.01)[cached: mailsec.protonmail.ch]; NEURAL_HAM_SHORT(-0.98)[-0.981,0]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(-3.67)[ip: (-9.48), ipnet: 185.70.40.0/24(-4.89), asn: 19905(-3.92), country: US(-0.07)]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[protonmail.com]; ASN(0.00)[asn:19905, ipnet:185.70.40.0/24, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[18.40.70.185.list.dnswl.org : 127.0.5.1]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[protonmail.com:s=default]; REPLYTO_EQ_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_REPLYTO(0.00)[protonmail.com]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_EQ_ADDR_ALL(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Mar 2019 16:22:35 -0000 Hi folks, this is my first post to the group. Apologies for length. I've been experimenting with GPU passthrough on bhyve. For background, the = host system is FreeBSD 12.0-RELEASE on an AMD Ryzen 1700 CPU @ 3.8 GHz, 32 = GB of ECC RAM, with two nVidia GPUs. I'm working with a Linux Debian 9 gues= t and a Windows Server 2019 (desktop experience installed) guest. I also ha= ve a USB controller passed-through for bluetooth and keyboard. With some unpleasant hacks I have succeeded in starting X on the Linux gues= t, passing-through an nVidia GT 710 under the nouveau driver. I can run the= "mate" desktop and glxgears, both of which are smooth at 4K. The Unity Hea= ven benchmark runs at an embarrassing 0.1 fps, and 2160p x264 video in VLC = runs at about 5 fps. Neither appears to be CPU-bound in the host or the gue= st. The hack I had to make: I found that many instructions to access memory-map= ped PCI BARs are not being executed on the CPU in guest mode but are being = passed back for emulation in the hypervisor. This causes an assertion to fa= il inside passthru_write() in pci_passthru.c ["pi->pi_bar[baridx].type =3D= =3D PCIBAR_IO"] because it does not expect to perform memory-mapped IO for = the guest. Examining the to-be-emulated instructions in vmexit_inst_emul() = {e.g., movl (%rdi), %eax}, they look benign to me, and I have no explanatio= n for why the CPU refused to execute them in guest mode. As an amateur work-around, I removed the assertion and instead I obtain the= desired offset into the guest's BAR, calculate what that guest address tra= nslates to in the host's address space, open(2) /dev/mem, mmap(2) over to t= hat address, and perform the write directly. I do a similar trick in passth= ru_read(). Ugly, slow, but functional. This code path is accessed continuously whether or not X is running, with a= n increase in activity when running anything GPU-heavy. Always to bar 1, an= d mostly around the same offsets. I added some logging of this event. It ru= ns at about 100 lines per second while playing video. An excerpt is: ... Unexpected out-of-vm passthrough write #492036 to bar 1 at offset 41100. Unexpected out-of-vm passthrough write #492037 to bar 1 at offset 41100. Unexpected out-of-vm passthrough read #276162 to bar 1 at offset 561280. Unexpected out-of-vm passthrough write #492038 to bar 1 at offset 38028. Unexpected out-of-vm passthrough write #492039 to bar 1 at offset 38028. Unexpected out-of-vm passthrough read #276163 to bar 1 at offset 561184. Unexpected out-of-vm passthrough read #276164 to bar 1 at offset 561184. Unexpected out-of-vm passthrough read #276165 to bar 1 at offset 561184. Unexpected out-of-vm passthrough read #276166 to bar 1 at offset 561184. ... So my question here is, 1. How do I diagnose why the instructions are not being executed in guest m= ode? Some other problems: 2. Once the virtual machine is shut down, the passed-through GPU doesn't ge= t turned off. Whatever message was on the screen in the final throes of Lin= ux's shutdown stays there. Maybe there is a specific detach command which b= hyve or nouveau hasn't yet implemented? Alternatively, maybe I could exploi= t some power management feature to reset the card when bhyve exits. 3. It is not possible to reboot the guest and then start X again without an= intervening host reboot. The text console works fine. Xorg.0.log has a mes= sage like (EE) [drm] Failed to open DRM device for pci:0000:00:06.0: -19 (EE) open /dev/dri/card0: No such file or directory dmesg is not very helpful either.[0] I suspect that this is related to prob= lem (2). 4. There is a known bug in the version of the Xorg server that ships with D= ebian 9, where the switch from an animated mouse cursor back to a static cu= rsor causes the X server to sit in a busy loop of gradually increasing stac= k depth, if the GPU takes too long to communicate with the driver.[1] For m= e, this consistently happens after I type my password into the Debian login= dialog box and eventually (~ 120 minutes) locks up the host by eating all = the swap. A work-around is to replace the guest's animated cursors with sta= tic cursors. The bug is fixed in newer versions of X, but I haven't tested = whether their fix works for me yet. 5. The GPU doesn't come to life until the nouveau driver kicks in. What is = special about the driver? Why doesn't the UEFI open the GPU and send it out= put before the boot? Any idea if the problem is on the UEFI side or the hyp= ervisor side? 6. On Windows, the way Windows probes multi-BAR devices seems to be inconsi= stent with bhyve's model for storing io memory mappings. Specifically, I be= lieve Windows assigns the 0xffffffff sentinel to all BARs on a device in on= e shot, then reads them back and assigns the true addresses afterwards. How= ever, bhyve sees the multiple 0xffffffff assignments to different BARs as a= clash and errors out on the second BAR probe. I removed most of the mmio_r= b_tree error handling in mem.c and this is sufficient for Windows to boot, = and detect and correctly identify the GPU. (A better solution might be to h= andle the initial 0xffffffff write as a special case.) I can then install t= he official nVidia drivers without problem over Remote Desktop. However, th= e GPU never springs into life: I am stuck with a "Windows has stopped this = device because it has reported problems. (Code 43)" error in the device man= ager, a blank screen, and not much else to go on. Is it worth me continuing to hack away at these problems---of course I'm ha= ppy to share anything I come up with---or is there an official solution to = GPU support in the pipe about to make my efforts redundant :)? Thanks, Robert Crowston. --- Footnotes [0] Diff'ing dmesg after successful GPU initialization (+) and after failu= re (-), and cutting out some lines that aren't relevant: nouveau 0000:00:06.0: bios: version 80.28.a6.00.10 +nouveau 0000:00:06.0: priv: HUB0: 085014 ffffffff (1f70820b) nouveau 0000:00:06.0: fb: 1024 MiB DDR3 @@ -466,24 +467,17 @@ nouveau 0000:00:06.0: DRM: DCB conn 00: 00001031 nouveau 0000:00:06.0: DRM: DCB conn 01: 00002161 nouveau 0000:00:06.0: DRM: DCB conn 02: 00000200 -nouveau 0000:00:06.0: disp: chid 0 mthd 0000 data 00000400 00001000 000000= 02 -nouveau 0000:00:06.0: timeout at /build/linux-UEAD6s/linux-4.9.144/drivers= /gpu/drm/nouveau/nvkm/engine/disp/dmacgf119.c:88/gf119_disp_dmac_init()! -nouveau 0000:00:06.0: disp: ch 1 init: c207009b -nouveau: DRM:00000000:0000927c: init failed with -16 -nouveau 0000:00:06.0: timeout at /build/linux-UEAD6s/linux-4.9.144/drivers= /gpu/drm/nouveau/nvkm/engine/disp/dmacgf119.c:54/gf119_disp_dmac_fini()! -nouveau 0000:00:06.0: disp: ch 1 fini: c2071088 -nouveau 0000:00:06.0: timeout at /build/linux-UEAD6s/linux-4.9.144/drivers= /gpu/drm/nouveau/nvkm/engine/disp/dmacgf119.c:54/gf119_disp_dmac_fini()! -nouveau 0000:00:06.0: disp: ch 1 fini: c2071088 +[drm] Supports vblank timestamp caching Rev 2 (21.10.2013). +[drm] Driver supports precise vblank timestamp query. +nouveau 0000:00:06.0: DRM: MM: using COPY for buffer copies +nouveau 0000:00:06.0: DRM: allocated 1920x1080 fb: 0x60000, bo ffff96fdb39= a1800 +fbcon: nouveaufb (fb0) is primary device -nouveau 0000:00:06.0: timeout at /build/linux-UEAD6s/linux-4.9.144/drivers= /gpu/drm/nouveau/nvkm/engine/disp/coregf119.c:187/gf119_disp_core_fini() -nouveau 0000:00:06.0: disp: core fini: 8d0f0088 -[TTM] Finalizing pool allocator -[TTM] Finalizing DMA pool allocator -[TTM] Zone kernel: Used memory at exit: 0 kiB -[TTM] Zone dma32: Used memory at exit: 0 kiB -nouveau: probe of 0000:00:06.0 failed with error -16 +Console: switching to colour frame buffer device 240x67 +nouveau 0000:00:06.0: fb0: nouveaufb frame buffer device +[drm] Initialized nouveau 1.3.1 20120801 for 0000:00:06.0 on minor 0 [1] https://devtalk.nvidia.com/default/topic/1028172/linux/titan-v-ubuntu-1= 6-04lts-and-387-34-driver-crashes-badly/post/5230898/#5230898 From owner-freebsd-virtualization@freebsd.org Mon Mar 18 11:17:46 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D4E1E153A7B3 for ; Mon, 18 Mar 2019 11:17:46 +0000 (UTC) (envelope-from misc-freebsd@talk2dom.com) Received: from server.zx81.pw (server.zx81.pw [IPv6:2001:41c9:1:426::39]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "server.zx81.pw", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 026F784134 for ; Mon, 18 Mar 2019 11:17:45 +0000 (UTC) (envelope-from misc-freebsd@talk2dom.com) Received: from auth (localhost [127.0.0.1]) by server.zx81.pw (8.15.2/8.15.2) with ESMTPSA id x2IBHgxe048740 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO) for ; Mon, 18 Mar 2019 11:17:43 GMT (envelope-from misc-freebsd@talk2dom.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=talk2dom.com; s=zx81; t=1552907863; x=1553512663; bh=IpF0BBcQ7gKnGQGVbNaP+pO5iDi7GT+e+g8TTVg0ayg=; h=Subject:To:References:From:Date:In-Reply-To; b=fcHzRIudPBwiKRwXexEWRv/kBlYGkfx9CYucC8OQ8MmlKhlIT3MWrBgtzbSF+oj6r hy2wZK4Xbedtr6xFmAxCnFZYsEP1nedpKV8ore6EQ23KB0VyKUr4lj2Wp6pKv+xxVx WvQg7hf2dtmILfj+LaGArdA+s+4WHz96hnVPMn/U= Subject: Re: GPU passthrough: mixed success on Linux, not yet on Windows To: freebsd-virtualization@freebsd.org References: From: Dom Message-ID: <283e2783-ac4c-6ab2-465d-de370fa2feff@zx81.pw> Date: Mon, 18 Mar 2019 11:17:42 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.5.3 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Mar 2019 11:17:47 -0000 On 17/03/2019 16:22, Robert Crowston via freebsd-virtualization wrote: > 6. On Windows, [...] the GPU never springs into life: I am stuck with a "Windows has stopped this device because it has reported problems. (Code 43)" error in the device manager, a blank screen, and not much else to go on. > > Is it worth me continuing to hack away at these problems---of course I'm happy to share anything I come up with---or is there an official solution to GPU support in the pipe about to make my efforts redundant :)? > > Thanks, > Robert Crowston. As far as I know, the nVidia drivers actively try to deny use of consumer video cards inside a VM guest and error 43 might be a symptom of this. Some people have managed to get round this with some non-FreeBSD VM setups that hide the VM-guest aspect, or change their GPU's reported hardware/vendor IDs, but I've never managed to recreate this in Bhyve. e.g. see Question 10 at http://vfio.blogspot.com/2014/08/vfiovga-faq.html and also the comments at the bottom of the page. Dom From owner-freebsd-virtualization@freebsd.org Mon Mar 18 15:04:22 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D56E915430AC; Mon, 18 Mar 2019 15:04:21 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5FAFE8CAEF; Mon, 18 Mar 2019 15:04:17 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id 9A95843E4; Mon, 18 Mar 2019 11:04:08 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Mon, 18 Mar 2019 11:04:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zyxst.net; h= date:from:to:cc:subject:message-id:mime-version:content-type; s= fm2; bh=ekkaHxWtDaPGNrZzNoISIO8U7su9amFHDMIIDvwGuM8=; b=sQTo9Ofu rksUUIyY6Re5DcHZp6J6QsB/MMw2mG9vDNNCchUVZnhaO1+XhNE07u2m5HktBQr+ FDDhkctWTm8jouWUkma94f8E1m3gDSndg8aEyGK7mwXPZ8/wFGk/lp3lE0mRaUdK cw0W7erfRzDnEptcOVGi95E0wHCYPeDZbxrFUOmNjCvXe8EB9/erW+0mKOPGGry7 H2SmYWufN9kxXQQDyLHzQF9mq5C6gzd/BkP8SRgdjl34ju0h5T7sU1Sizv/DQY/u ld0wZkORZgnAyhmaH+3sLEiX0kAp1BINtcSeORYXRmIvRDrwAMAd7vnw2OU7uiME 8TmTy98nLiaLTw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:message-id :mime-version:subject:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm2; bh=ekkaHxWtDaPGNrZzNoISIO8U7su9a mFHDMIIDvwGuM8=; b=GSHJjNH48X02sluAk5dp3skglknZjhBGN2aFGvPxW0STH IGlHsI4CrLX8JVGjPa61QJs8iotXL005b/Ied15ysF1SpeJh1lWxsdxV5WoQ/PzW yvLj747Jwgfs1y54uuWuHt0SZPBZJ0+sQDQM8kbo1gA5+V0aiKL9QJI6rObLdcOD mLB0iDncT8L4KyXAaowPa8GRPE+WNB6LgsEheYyS3jI+aEFHf23vWZqKaP4Qx/E4 HwTXHbapHQqV8z5LHsoOvQM9JlsrkausJ1A+ruIpZ20bCSe+yt4hxdzYTbUbksjy Df/AvhE1GoeUog5V9Tl6uk+2gdWbfwaqPIRm92MwA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedutddriedugdejvdcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfggtggufgesghdtreertd ervdenucfhrhhomhepthgvtghhqdhlihhsthhsuceothgvtghhqdhlihhsthhsseiihiig shhtrdhnvghtqeenucfkphepkedvrdejtddrledurddutddunecurfgrrhgrmhepmhgrih hlfhhrohhmpehtvggthhdqlhhishhtshesiiihgihsthdrnhgvthenucevlhhushhtvghr ufhiiigvpedt X-ME-Proxy: Received: from rpi3.zyxst.net (rpi3.zyxst.net [82.70.91.101]) by mail.messagingengine.com (Postfix) with ESMTPA id 52B9E1030F; Mon, 18 Mar 2019 11:04:07 -0400 (EDT) Date: Mon, 18 Mar 2019 15:04:05 +0000 From: tech-lists To: freebsd-virtualization@freebsd.org Cc: freebsd-stable@freebsd.org Subject: bhyve zfs resizing Message-ID: <20190318150404.GB91631@rpi3.zyxst.net> Mail-Followup-To: freebsd-virtualization@freebsd.org, freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="XF85m9dhOBO43t/C" Content-Disposition: inline User-Agent: Mutt/1.11.2 (2019-01-07) X-Rspamd-Queue-Id: 5FAFE8CAEF X-Spamd-Bar: ------- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=zyxst.net header.s=fm2 header.b=sQTo9Ofu; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=GSHJjNH4; spf=pass (mx1.freebsd.org: domain of tech-lists@zyxst.net designates 64.147.123.24 as permitted sender) smtp.mailfrom=tech-lists@zyxst.net X-Spamd-Result: default: False [-7.27 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[zyxst.net:s=fm2,messagingengine.com:s=fm2]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.24]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[zyxst.net]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; MX_GOOD(-0.01)[in2-smtp.messagingengine.com,in1-smtp.messagingengine.com,in2-smtp.messagingengine.com,in1-smtp.messagingengine.com,in2-smtp.messagingengine.com,in1-smtp.messagingengine.com,in2-smtp.messagingengine.com,in1-smtp.messagingengine.com,in2-smtp.messagingengine.com,in1-smtp.messagingengine.com,in2-smtp.messagingengine.com,in1-smtp.messagingengine.com,in2-smtp.messagingengine.com,in1-smtp.messagingengine.com]; DKIM_TRACE(0.00)[zyxst.net:+,messagingengine.com:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.97)[-0.971,0]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:11403, ipnet:64.147.123.0/24, country:US]; IP_SCORE(-1.59)[ipnet: 64.147.123.0/24(-4.45), asn: 11403(-3.44), country: US(-0.07)]; RCVD_IN_DNSWL_LOW(-0.10)[24.123.147.64.list.dnswl.org : 127.0.5.1] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Mar 2019 15:04:22 -0000 --XF85m9dhOBO43t/C Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, Apart from the performance benefit as per the section for bhyve in the handbook, can the size of the zfs-backed guest: 1. be resized from the host? 2. does the guest need to be inactive? 3. can linux guests (or even windows ones) be resized as well? thanks, --=20 J. --XF85m9dhOBO43t/C Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEE8n3tWhxW11Ccvv9/s8o7QhFzNAUFAlyPs1YACgkQs8o7QhFz NAUlFQ//WBtTcdDJmrOiIvkOmwgwS3ShzwAIVajVHwLzFClPPFIY421lWCD1odVK MHVsBcf4gkdCzbQRQB9WLlSbHnY/1jJ7RJLptqpujxMd45Dr3DVKY9CpuQu8Gd4b r/ORT65sMRFFtQGaBACZp2ttcEgQpV05uJgwuhSyyvftFeIfNmB8REZkxvTAlnjO zjlQ03EHdQeNL2TRU+K/D046VJjVaAIvdpReGG85u7UsBpkTuwN55FpXf9qQv5+l 9zPkefzO5taMbhpe3Oz9souFuAvOoOouYFyJJpFi8xF1c04+cL22S+7Ff14WMjDI gCGw4LHD/h/HYh11o8W77T6Xo1asABkZJKS8Q+pPtT6pYdRAfweAScOHkFSgiH8P KVNzDZCK1LyNeG/x3PQIEGHutrxrO7fAVeE9Av9Q78xWC2MckMnM4PQsr67khCRD 2dlzpzOJCWmp84BuHTIhgZfgDc60V1Slk6Ek+pSkdIO061mo9EfM2SfijrT3yCPm UZ0jSW6fVoRksH/y3A1DOiV7+oUHqI3ccdcSUaQZJFRT3mD4UT2QEoh5gn+XhUHg hJ3W8yVQ38rQUO+8GnXonnlvLbxojwJSoFnT2R5BRSiDNwB+Li7RC3EVfHyqGbvT 1oGqKpJ7BzOQ7sK6uzggePmWlHiVw+z9BHPbVX5KL38fuj5yAtg= =LNSz -----END PGP SIGNATURE----- --XF85m9dhOBO43t/C-- From owner-freebsd-virtualization@freebsd.org Mon Mar 18 15:08:45 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7F66B154343E; Mon, 18 Mar 2019 15:08:45 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-lj1-f173.google.com (mail-lj1-f173.google.com [209.85.208.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 84FFD8CDDB; Mon, 18 Mar 2019 15:08:44 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-lj1-f173.google.com with SMTP id n18so12419001ljg.11; Mon, 18 Mar 2019 08:08:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=7M6FM8s1F/5jupaGN3K/AHKe4vjcbT2kz9laQUtEgbc=; b=WjJowxElhZjN/QbCtNAdqdHEsNm6z+9r/Robvt6NVNFWh4P2zxokDzQTMYOHU2cSQg HeyjSG8AgDHpNkCJ2ta0sbvoOpSwh+E0CA6g5Si/fP4ibtIlmglQQbUscG6qUgV2CL+W W+XXG5FjJoJrbqc5M3ODwpEG1/9PIxacPu9euQUX+Bplr0r3ok026OfVhQLRaKYG/nYa aSADLi0ohSmBSBPT0dfR6OUsLYrQ3LoMZaCVJPXpb02WoM5uFmsYuNxN96Z3abQJH3al 84pR36FPnecAt6gHfArTxYsfLVLkNJfDEUGxhVZA/nT4MKK6z0gy1yCzPl7bV6IMWbt2 mxMA== X-Gm-Message-State: APjAAAUHiuYb8/wyZfvbf2N2v7w2UPS0CMb7yV3Iz15Dat4NyGq/sa55 uSY+r/dLB5+BhWXcfaKXDMFC6xLBtB52mgKqxAy5+vCg X-Google-Smtp-Source: APXvYqw7yw6A62K2ODW0etZ5sF4yJwXtruA9nFGh+4l532y8OAAL1UBKnuWfsomSE/RSihW54N34j1Nka6Ng9Jgpfiw= X-Received: by 2002:a2e:8582:: with SMTP id b2mr6017265lji.24.1552921722331; Mon, 18 Mar 2019 08:08:42 -0700 (PDT) MIME-Version: 1.0 References: <20190318150404.GB91631@rpi3.zyxst.net> In-Reply-To: <20190318150404.GB91631@rpi3.zyxst.net> From: Alan Somers Date: Mon, 18 Mar 2019 09:08:31 -0600 Message-ID: Subject: Re: bhyve zfs resizing To: "freebsd-virtualization@freebsd.org" , FreeBSD Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 84FFD8CDDB X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.208.173 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-3.98 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.996,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE(-1.31)[ip: (-0.53), ipnet: 209.85.128.0/17(-3.88), asn: 15169(-2.09), country: US(-0.07)]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[173.208.85.209.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.66)[-0.657,0]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Mar 2019 15:08:45 -0000 On Mon, Mar 18, 2019 at 9:05 AM tech-lists wrote: > > Hi, > > Apart from the performance benefit as per the section for bhyve in the > handbook, can the size of the zfs-backed guest: > > 1. be resized from the host? > 2. does the guest need to be inactive? > 3. can linux guests (or even windows ones) be resized as well? > > thanks, > -- > J. Do you mean using a zvol as the backing store for a VM? If so, then: 1) Yes. You can just do "zfs set volsize" on the host. 2) In theory no, but the guest may need to be rebooted to notice the change. And I'm not sure if the current bhyve code will expose the new size without a reboot or not. 3) Sure. But after you expand the zvol (or before you shrink it), you'll have to change the size of the guest's filesystem using the guest's native tools. -Alan From owner-freebsd-virtualization@freebsd.org Mon Mar 18 15:25:34 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1B3BD1543C8F; Mon, 18 Mar 2019 15:25:34 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7711D8D891; Mon, 18 Mar 2019 15:25:33 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id A978045B0; Mon, 18 Mar 2019 11:25:31 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Mon, 18 Mar 2019 11:25:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zyxst.net; h= date:from:to:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm2; bh=/yn8mfkLnZs8QtHF0yFXVC7vqx0 y05CUtyChPjrpgDo=; b=YG1uDenvgktq+mM0uB0PW31bsX8JTy0rVRR3qH8iWme 9ceTnF32rzvW2lI3oZmAqDSOmKLiKu9a0Gr2yMnv2MZ1+fHRU0mUtXVb4YjQl++C z/aB6hzF4BDsmV+uu4ZF0yCJNMzdIkdAlv5Z5AQNspwcIHru/K0R8xCqVz2mOmjn iO1qW7Ig22mghunpmPevX5tBomIscQ61c2VRNLRw5Wa+doLidTf0DX02lEEJSDvq 7pOiNXrutq9CBzQex3/tq2yutKDQnqkS9Ngam+M3YvT7PvnZcc0J1Q5jV8hvKsXb 3UTUFxmt7eMGEqltPRi9FmEAExJznyBfmbmwS41p87g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=/yn8mf kLnZs8QtHF0yFXVC7vqx0y05CUtyChPjrpgDo=; b=vUC+qNn8qYa0kxRnzoI70J vlxGgGirjAPf0+C1Po7V+nES7rxwex8Z28RsRR/pwuZTn4ctKFDiJooWXyg/HImP d5Tda8PrSpDs3sayRvQXuQnqAF40/2z2ytYSne6eXw+IVHU6CYgA251IoydmkB1M bbjD/3ULKIbRAdrSKqICvMPNiI93pacuJj9Qs4UTZeVwFs3A+1GnpUDyZmwXu+ek nKMz+aFM+bzGIuI6z5bx13Lq6EfHMJO87CE9q/1hRDFBiDwoAtZt2mZeYhjx21Q5 Ek/r1QthPZhzA4W0q6/EtiN2xcBZPMHHZklgI4vnpVTihPfQXYINll7DMI6Tmn4Q == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedutddriedugdejiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjfgesghdtre ertdervdenucfhrhhomhepthgvtghhqdhlihhsthhsuceothgvtghhqdhlihhsthhsseii hiigshhtrdhnvghtqeenucfkphepkedvrdejtddrledurddutddunecurfgrrhgrmhepmh grihhlfhhrohhmpehtvggthhdqlhhishhtshesiiihgihsthdrnhgvthenucevlhhushht vghrufhiiigvpedt X-ME-Proxy: Received: from rpi3.zyxst.net (rpi3.zyxst.net [82.70.91.101]) by mail.messagingengine.com (Postfix) with ESMTPA id 78ECFE4684; Mon, 18 Mar 2019 11:25:30 -0400 (EDT) Date: Mon, 18 Mar 2019 15:25:26 +0000 From: tech-lists To: freebsd-virtualization@freebsd.org, FreeBSD Subject: Re: bhyve zfs resizing Message-ID: <20190318152525.GC91631@rpi3.zyxst.net> Mail-Followup-To: freebsd-virtualization@freebsd.org, FreeBSD References: <20190318150404.GB91631@rpi3.zyxst.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="Pk6IbRAofICFmK5e" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.11.2 (2019-01-07) X-Rspamd-Queue-Id: 7711D8D891 X-Spamd-Bar: ------- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=zyxst.net header.s=fm2 header.b=YG1uDenv; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=vUC+qNn8; spf=pass (mx1.freebsd.org: domain of tech-lists@zyxst.net designates 64.147.123.24 as permitted sender) smtp.mailfrom=tech-lists@zyxst.net X-Spamd-Result: default: False [-7.30 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[zyxst.net:s=fm2,messagingengine.com:s=fm2]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.24]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; DMARC_NA(0.00)[zyxst.net]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[zyxst.net:+,messagingengine.com:+]; RCPT_COUNT_TWO(0.00)[2]; MX_GOOD(-0.01)[cached: in2-smtp.messagingengine.com]; SIGNED_PGP(-2.00)[]; NEURAL_HAM_SHORT(-0.99)[-0.991,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:11403, ipnet:64.147.123.0/24, country:US]; IP_SCORE(-1.60)[ipnet: 64.147.123.0/24(-4.47), asn: 11403(-3.44), country: US(-0.07)]; RCVD_IN_DNSWL_LOW(-0.10)[24.123.147.64.list.dnswl.org : 127.0.5.1] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Mar 2019 15:25:34 -0000 --Pk6IbRAofICFmK5e Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Mar 18, 2019 at 09:08:31AM -0600, Alan Somers wrote: > >Do you mean using a zvol as the backing store for a VM? If so, then: >1) Yes. You can just do "zfs set volsize" on the host. >2) In theory no, but the guest may need to be rebooted to notice the >change. And I'm not sure if the current bhyve code will expose the >new size without a reboot or not. >3) Sure. But after you expand the zvol (or before you shrink it), >you'll have to change the size of the guest's filesystem using the >guest's native tools. Great, that's awesome. Thanks for clarifying=20 --=20 J. --Pk6IbRAofICFmK5e Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEE8n3tWhxW11Ccvv9/s8o7QhFzNAUFAlyPuFsACgkQs8o7QhFz NAUgvg//Z2bpBqYHbpl1U6fzFBF+NOw+j1iqMP46tia9TziO6FUH+B/KUe9HAx0v N+QFyPJO6Y5EFZUdKfnAXPpSe1nCA7q5ueFLNoCgJilg13kxLccsvWh2Wo0QO3R7 kdIcGBDAgBF7DM2mWTZdyLe6fdbzKMhfU5vhsZJPw+HQjdo3dCbrd3s2VynyCVT8 XyXDgpbdbm3N3D56JQdxrc2x2CTy7EjaVbYTxD6rfsyyl4Id6s45A4zcBgibtDiE 9ZKcajQmTuxHRmnRsA/4Oax2LMuoLCTFT4nzkry2k7frjJbnQqUTGY7mmVMeeQNO ZCvGPz/A8RhiW8lg6A6quui9wjWz8ItUhmayRZQ3XzRGi7Z685vJF5DMuyLDWY7r A6FI4Ta+3cvCxPdsfnQXlY9Ufj3cgtkNWbHgHEoGu0p/YU/H6AEFTE9U1WooY2GX rJxc+pzt6AJaDXVuj6uy3BImY8Eq3neupbmcZbdGACW3IXfksiGb6gim2N6vyhY9 GGL/S9kzzQEGP4/rcHkI7XHcaWVm7GZ/jaY2jhROhZ+MxLaYZIRntx1ShBEAtX5b cirLCy9X7nezkk+EQl2KBLux8H52LYD8QHaJVgVFGsDrJw2Ydb+ASRlPhBxjt2H+ 1aQMSu/C9i7FC7H9fptEbQ7aAkbavgHuFUBa6A/Ijo/2YCiCCM8= =3l1R -----END PGP SIGNATURE----- --Pk6IbRAofICFmK5e-- From owner-freebsd-virtualization@freebsd.org Mon Mar 18 17:16:10 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 499DF154793F for ; Mon, 18 Mar 2019 17:16:10 +0000 (UTC) (envelope-from ctuffli@gmail.com) Received: from mail-oi1-f171.google.com (mail-oi1-f171.google.com [209.85.167.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 82A926B718 for ; Mon, 18 Mar 2019 17:16:09 +0000 (UTC) (envelope-from ctuffli@gmail.com) Received: by mail-oi1-f171.google.com with SMTP id u12so10733262oiv.1 for ; Mon, 18 Mar 2019 10:16:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=UmOBIWCnN4fQmtkc89a/wEcxCX22OmpnrWaJ1YwT5xA=; b=EWLk1ilIZAFPQW3HRrm8Wfh3FndufX6/rIch5tlpDHGtTod2wgjez04Lfq5m4VkNPo i58KqlExAcydILjqOKsaZ48Ggn1PIJytnYWhRPeIXTtQ4iecb49HbE4QkOVTHPBO3CI1 bq/YLMP7wHFAgWn37TLoBnuy5vH50pnvIQRl4uqTQOAdf9bV3Ya53fBikVpK8ypTzwOE myaOYgAlkGMTxVuHNI297ecXofpv0FC3QQibcoIIP7SufkNSRpikIoHMIT3yF835tI9d bYuuj7V/5Wb1abqei13xIVruu2rk810pwsNsZt+EZ3cm1p/ozBec8aiKUiAyQo53TpOa uNag== X-Gm-Message-State: APjAAAUFLrOyTuX4/UpoMKGFV118l596PtOEkoeXGRwgremhM4dZm20m A3NRMtPRcrCUL97ff/GsCDUBU5ut X-Google-Smtp-Source: APXvYqz7yhK+bGEPEBsjNjPAZpVZuM3sbjw41F9lMFMDCAW13rZlfq+JkGea1THro2glU/52Tm0IsQ== X-Received: by 2002:aca:3a57:: with SMTP id h84mr10099285oia.162.1552929362001; Mon, 18 Mar 2019 10:16:02 -0700 (PDT) Received: from mail-oi1-f170.google.com (mail-oi1-f170.google.com. [209.85.167.170]) by smtp.gmail.com with ESMTPSA id n3sm10777ota.81.2019.03.18.10.16.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 18 Mar 2019 10:16:01 -0700 (PDT) Received: by mail-oi1-f170.google.com with SMTP id v7so3936570oie.8 for ; Mon, 18 Mar 2019 10:16:01 -0700 (PDT) X-Received: by 2002:aca:407:: with SMTP id 7mr5467206oie.90.1552929361259; Mon, 18 Mar 2019 10:16:01 -0700 (PDT) MIME-Version: 1.0 From: Chuck Tuffli Date: Mon, 18 Mar 2019 10:15:50 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: emulated PCI device BAR size To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 82A926B718 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of ctuffli@gmail.com designates 209.85.167.171 as permitted sender) smtp.mailfrom=ctuffli@gmail.com X-Spamd-Result: default: False [-4.22 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; MIME_TRACE(0.00)[0:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.91)[-0.909,0]; RCVD_IN_DNSWL_NONE(0.00)[171.167.85.209.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[]; FORGED_SENDER(0.30)[chuck@freebsd.org,ctuffli@gmail.com]; DMARC_NA(0.00)[freebsd.org]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[chuck@freebsd.org,ctuffli@gmail.com]; IP_SCORE(-1.30)[ip: (-0.45), ipnet: 209.85.128.0/17(-3.88), asn: 15169(-2.09), country: US(-0.07)]; TO_DOM_EQ_FROM_DOM(0.00)[] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Mar 2019 17:16:10 -0000 In debugging why Windows doesn't like bhyve's NVMe device emulation, another developer mentioned that the NVMe specification defines the minimum size of BAR[0] to be 16K bytes. And while most OS don't enforce this requirement, evidently, Windows does. On the surface, a "minimum BAR size" sounds odd, but the MMIO BAR in NVMe includes the doorbell registers for each queue. And since the maximum number of queues is implementation specific, there isn't a fixed size for the BAR. So my question is how to best fix this. The easiest would be to add a check to the BAR size calculation in pci_nvme.c along the lines of: #define NVME_MMIO_SPACE_MIN (1 << 14) ... pci_membar_sz = MAX( , NVME_MMIO_SPACE_MIN); While this makes Windows happy, are there any implications to this approach other than chewing up more memory than the device actually needs? Alternatively, is there a better mechanism to use in this case? --chuck From owner-freebsd-virtualization@freebsd.org Mon Mar 18 17:56:11 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2CBCD154899C; Mon, 18 Mar 2019 17:56:11 +0000 (UTC) (envelope-from SRS0=eii1=RV=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C4C7C6D0CE; Mon, 18 Mar 2019 17:56:09 +0000 (UTC) (envelope-from SRS0=eii1=RV=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id E31D628416; Mon, 18 Mar 2019 18:56:06 +0100 (CET) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id BC69728417; Mon, 18 Mar 2019 18:56:05 +0100 (CET) Subject: Re: bhyve zfs resizing To: freebsd-virtualization@freebsd.org, FreeBSD References: <20190318150404.GB91631@rpi3.zyxst.net> <20190318152525.GC91631@rpi3.zyxst.net> From: Miroslav Lachman <000.fbsd@quip.cz> Message-ID: Date: Mon, 18 Mar 2019 18:56:03 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Firefox/52.0 SeaMonkey/2.49.3 MIME-Version: 1.0 In-Reply-To: <20190318152525.GC91631@rpi3.zyxst.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: C4C7C6D0CE X-Spamd-Bar: ++++ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [4.24 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_SPAM_SHORT(0.91)[0.914,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[quip.cz]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.96)[0.957,0]; RCVD_COUNT_THREE(0.00)[3]; IP_SCORE(0.21)[ip: (0.50), ipnet: 94.124.104.0/21(0.25), asn: 42000(0.20), country: CZ(0.07)]; MX_GOOD(-0.01)[cached: elsa.codelab.cz]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[4.105.124.94.list.dnswl.org : 127.0.10.0]; NEURAL_SPAM_LONG(0.97)[0.971,0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[000.fbsd@quip.cz,SRS0=eii1=RV=quip.cz=000.fbsd@elsa.codelab.cz]; RECEIVED_SPAMHAUS_PBL(0.00)[209.16.49.86.zen.spamhaus.org : 127.0.0.11]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ]; FROM_NEQ_ENVFROM(0.00)[000.fbsd@quip.cz,SRS0=eii1=RV=quip.cz=000.fbsd@elsa.codelab.cz]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Mar 2019 17:56:11 -0000 tech-lists wrote on 2019/03/18 16:25: > On Mon, Mar 18, 2019 at 09:08:31AM -0600, Alan Somers wrote: >> >> Do you mean using a zvol as the backing store for a VM?  If so, then: >> 1) Yes.  You can just do "zfs set volsize" on the host. >> 2) In theory no, but the guest may need to be rebooted to notice the >> change.  And I'm not sure if the current bhyve code will expose the >> new size without a reboot or not. >> 3) Sure.  But after you expand the zvol (or before you shrink it), >> you'll have to change the size of the guest's filesystem using the >> guest's native tools. I did it 2 month ago on FreeBSD 11.2. On the host with running guest: # zfs set volsize=200G tank1/vol1/bhyve/kotel/disk1 Even if I unmounted disk in the guest it still does not see the new size until I rebooted the guest. After reboot of the guest, you will see corrupted GPT: # gpart show -p vtbd1 => 40 209715120 vtbd1 GPT (200G) [CORRUPT] 40 8 - free - (4.0K) 48 1024 vtbd1p1 freebsd-boot (512K) 1072 976 - free - (488K) 2048 203423744 vtbd1p2 freebsd-ufs (97G) 203425792 6289368 - free - (3.0G) And after running recover, the guest will see the added space # gpart recover vtbd1 vtbd1 recovered # gpart show -p vtbd1 => 40 419430320 vtbd1 GPT (200G) 40 8 - free - (4.0K) 48 1024 vtbd1p1 freebsd-boot (512K) 1072 976 - free - (488K) 2048 203423744 vtbd1p2 freebsd-ufs (97G) 203425792 216004568 - free - (103G) After this, the partition can finally be enlarged # gpart resize -a 1M -s 197G -i 2 vtbd1 # growfs /vol0 Kind regards Miroslav Lachman From owner-freebsd-virtualization@freebsd.org Tue Mar 19 00:23:34 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D08A81551FEC; Tue, 19 Mar 2019 00:23:33 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 849F685B52; Tue, 19 Mar 2019 00:23:32 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id 3E49438AA; Mon, 18 Mar 2019 20:23:30 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Mon, 18 Mar 2019 20:23:30 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zyxst.net; h= date:from:to:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm2; bh=n4DsVE1Vk7yxDlJ8c6KpqfJbQAS XDBUQz0B5HlP/dYI=; b=VO67TFcZPZfPKy58xPuix9pYQyjAL1Mg/WzNcIZh+Kd X+cxcza2r2XINJb2JnR4wKI7FK63byR9qXpKq6LaJ9kNTpl5jEjulRX0EBz5rPkp HHG5zHZng5YNPENmjGjmgaJp4OEzt/EceYNZ4wGY96Eudquqescz38vuMn6lFha+ kRV5eEvf1L1Rpc+/EFG1lypIvDkC8ItK2cD6NnlqiL5QdNqUi+slwv0ZaVAaP9b+ XKKA83mRoztSX88kSHy/xNKQHON0O7FIrVd2/nsSzLf2Nwcsz2fERTn95RbsL8iD MZuPX87bOyI1XMnoigxj6mms69QGz1f0uP8rQSjWdkg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=n4DsVE 1Vk7yxDlJ8c6KpqfJbQASXDBUQz0B5HlP/dYI=; b=LhXMQWmeey+S3sZxiFENKi EDxd4yaeZn6hF3HdoxZiP2OdnV5K8JuwD5NdfJIblBZNxF+KglaO6Dm633hXVUOA xA9Hh99UIxAUL9kt4gay1wBMaYnXkXo/Pe6mtL8N5nX6W3SbfEQ1H/nts9X34vvE +rT6wvCojG6SBI+6y7RbSDCjsEBDe7VoYRCkciKbBXZn3lVaZxgJwKwQEWrbmkOX KweEhv7DpBa5eWnB3VODlxhrx9Kq6KFlKkCGr6aoVoPES76h1GcNjXoMnjMjQoTW ZDeeNNcRkYkDXYj6nziNpYD1rt0f+0bCSlZDsF1icgB33HY2jpKBaHznrOQ4qpkw == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedutddrieefgddufecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjfgesghdtre ertdervdenucfhrhhomhepthgvtghhqdhlihhsthhsuceothgvtghhqdhlihhsthhsseii hiigshhtrdhnvghtqeenucfkphepkedvrdejtddrledurddutddunecurfgrrhgrmhepmh grihhlfhhrohhmpehtvggthhdqlhhishhtshesiiihgihsthdrnhgvthenucevlhhushht vghrufhiiigvpedt X-ME-Proxy: Received: from rpi3.zyxst.net (rpi3.zyxst.net [82.70.91.101]) by mail.messagingengine.com (Postfix) with ESMTPA id 0F420E465F; Mon, 18 Mar 2019 20:23:28 -0400 (EDT) Date: Tue, 19 Mar 2019 00:23:26 +0000 From: tech-lists To: freebsd-virtualization@freebsd.org, FreeBSD Subject: Re: bhyve zfs resizing Message-ID: <20190319002325.GD91631@rpi3.zyxst.net> Mail-Followup-To: freebsd-virtualization@freebsd.org, FreeBSD References: <20190318150404.GB91631@rpi3.zyxst.net> <20190318152525.GC91631@rpi3.zyxst.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="vni90+aGYgRvsTuO" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.11.2 (2019-01-07) X-Rspamd-Queue-Id: 849F685B52 X-Spamd-Bar: ------- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=zyxst.net header.s=fm2 header.b=VO67TFcZ; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=LhXMQWme; spf=pass (mx1.freebsd.org: domain of tech-lists@zyxst.net designates 64.147.123.24 as permitted sender) smtp.mailfrom=tech-lists@zyxst.net X-Spamd-Result: default: False [-7.15 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[zyxst.net:s=fm2,messagingengine.com:s=fm2]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.24]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; DMARC_NA(0.00)[zyxst.net]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; MX_GOOD(-0.01)[cached: in2-smtp.messagingengine.com]; DKIM_TRACE(0.00)[zyxst.net:+,messagingengine.com:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.84)[-0.839,0]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:11403, ipnet:64.147.123.0/24, country:US]; IP_SCORE(-1.60)[ipnet: 64.147.123.0/24(-4.49), asn: 11403(-3.44), country: US(-0.07)]; RCVD_IN_DNSWL_LOW(-0.10)[24.123.147.64.list.dnswl.org : 127.0.5.1] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Mar 2019 00:23:34 -0000 --vni90+aGYgRvsTuO Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Mar 18, 2019 at 06:56:03PM +0100, Miroslav Lachman wrote: [...] Thanks for the example, I've saved it. Ok just one other question, which I might have found the answer to, or might not. I'm new to this virtualising on zfs even though ive used zfs for years. It's basically: I made a zvol, installed 12-R into it,. Where the disks option came up I chose the auto defaults for *ZFS* in the guest. I also selected encryption for both the virtual disk and swap. I think perhaps I shouldn't have done all this together in the same vm because with apache running in it, httpd got wedged (and then everything got wedged. sync wouldn't return). I think the top zfs layer and the encryption layer and the zfs underneath got too busy. Happily the server still responded to a shutdown -r and came back up. It's scrubbing the zpool to be on the safe side. Am I correct? In that I should have used UFS in the guest rather than zfs? Or was it the encryption? thanks, --=20 J. --vni90+aGYgRvsTuO Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEE8n3tWhxW11Ccvv9/s8o7QhFzNAUFAlyQNnIACgkQs8o7QhFz NAVQPw/+M6VlF4Ypb6WbYP9RWYaNjmtM78W2YFnLEzVHTGud2NRALXf5L/da2QJ5 SsRJbV6VshSGs3sWIeYOaCq6JviHBEf8V8UvOHDWSRfztoA35oZGaXntsGhu9gse qea34NzRQSnbgSP84/SGncMTF6SMNQGOzepcO1zpDOVbeidZwvBfq/yvRD98jjoR kBmcCR/z1Tg3Ib1tkXCdR1PHIM5XQE/DzSWxmRQIFk29Pwdif/ptsWKODVJUBAHj NSlLQeT3j8Rq+V3qffAM7xtKsXsb7ga4B+8boGdwM13SPyteCm0FAiVvN0oklkL0 YcaxlWEdpPGLSCK21iRwJJ65azfD4rTa+M7sv4/kcNI10lEBh2EaNw4j9gws87Fx wtQMZTcdljkv2ARYySXD/e6XEsHJ/bs6kwYb7vlgRY4bY7oScgs6CxiM5FijA63k WshkwmRRGZ5XuM8vJGIT6j89bS3cxnWLpv8A6AuLAXfkmqci9J2oPaO0G7i9v6BW +bphPbPCsVPCUAr2SLDMD3j3woGdXnYwb5VaRXqjgPFXj/iLqp100O4HTp3NNsDw LaFiYHZ6tnWCYCHwc0j7cxgndKfEkvsNs69lOZBdOuxX2/47deVu53t+BF0J+bH8 7J2oDjZts9SWbrAEnXQFzeM3HuFpEubxQI5z/lz6glApDlsA8uk= =PkcC -----END PGP SIGNATURE----- --vni90+aGYgRvsTuO-- From owner-freebsd-virtualization@freebsd.org Tue Mar 19 00:28:33 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 22365152A57A; Tue, 19 Mar 2019 00:28:33 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-lf1-f45.google.com (mail-lf1-f45.google.com [209.85.167.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 42E15860E4; Tue, 19 Mar 2019 00:28:32 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-lf1-f45.google.com with SMTP id p1so12995544lfk.9; Mon, 18 Mar 2019 17:28:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=iOhevDRY+hweLBoticrI6QocbmRP1h4+75kC+bSCKto=; b=KVrUWlsg96Wv302lQvQeFz8dE0GzKt2gg0xD0KiuNaYDcoSYJyH7qiIVGQ3uLSNx4b NycD9mK0zHokXGKSxzH3M9mIKqIN+nuanQVBX+t51KNKy/6FTLOG1SQ/KpWb5eqazdD/ RR7IVIxtFz8AGvuw8Y4kmswanSniHS20tuJh9nGnzJf444o+tyah787t9FVtrIWvvu+h FtWHl/kg2/yfZnCnKvU5L9yyCfCJq+/cjUha1LFJtxW9G0MJkW/ZHq1MRCKcuJvRgmBN +lHauUuwZD7PYypfO9KsmdVpDMbSHiAjdOIQKb8xEEA+tL9Xv6qTuW4QAJhHJ5DQI3hF tEYQ== X-Gm-Message-State: APjAAAXGsu7GHbxCI0sLljtjSMWD5e7nnGRmSH0k+8BtOmtywUwEnt3j rNjbqTPAL/wsRjS4ceNBEPshB1cLLPITGr5JBSTQKc/Z X-Google-Smtp-Source: APXvYqzgLR+MrOSBSqJ7l7eUg9B6nSuKVaXCGACinYKM2lJRq5bajWiaPGTDXQAmRpx3PwEq677WIDH8yC/yu5JoBUs= X-Received: by 2002:ac2:43ba:: with SMTP id t26mr11184041lfl.10.1552955304427; Mon, 18 Mar 2019 17:28:24 -0700 (PDT) MIME-Version: 1.0 References: <20190318150404.GB91631@rpi3.zyxst.net> <20190318152525.GC91631@rpi3.zyxst.net> <20190319002325.GD91631@rpi3.zyxst.net> In-Reply-To: <20190319002325.GD91631@rpi3.zyxst.net> From: Alan Somers Date: Mon, 18 Mar 2019 18:28:13 -0600 Message-ID: Subject: Re: bhyve zfs resizing To: "freebsd-virtualization@freebsd.org" , FreeBSD Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 42E15860E4 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.167.45 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-3.85 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE(-1.32)[ip: (-0.59), ipnet: 209.85.128.0/17(-3.87), asn: 15169(-2.10), country: US(-0.07)]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[45.167.85.209.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.51)[-0.512,0]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Mar 2019 00:28:33 -0000 On Mon, Mar 18, 2019 at 6:24 PM tech-lists wrote: > > On Mon, Mar 18, 2019 at 06:56:03PM +0100, Miroslav Lachman wrote: > > [...] > > Thanks for the example, I've saved it. > > Ok just one other question, which I might have found the answer to, or > might not. I'm new to this virtualising on zfs even though ive used zfs > for years. It's basically: > > I made a zvol, installed 12-R into it,. Where the disks option came up I > chose the auto defaults for *ZFS* in the guest. I also selected > encryption for both the virtual disk and swap. I think perhaps I > shouldn't have done all this together in the same vm because with apache > running in it, httpd got wedged (and then everything got wedged. sync > wouldn't return). I think the top zfs layer and the encryption layer and > the zfs underneath got too busy. Happily the server still responded to a > shutdown -r and came back up. It's scrubbing the zpool to be on the safe > side. > > Am I correct? In that I should have used UFS in the guest rather than > zfs? Or was it the encryption? > > thanks, > -- > J. Running ZFS inside of a ZFS-backed guest will be slower than using UFS, but it should work just fine. I doubt that it was the cause of your problem. -Alan From owner-freebsd-virtualization@freebsd.org Tue Mar 19 00:49:34 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CD535152C3F9; Tue, 19 Mar 2019 00:49:34 +0000 (UTC) (envelope-from SRS0=58Nm=RW=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 339C186F97; Tue, 19 Mar 2019 00:49:34 +0000 (UTC) (envelope-from SRS0=58Nm=RW=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id F2F1C28417; Tue, 19 Mar 2019 01:49:22 +0100 (CET) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id D6E1D28416; Tue, 19 Mar 2019 01:49:15 +0100 (CET) Subject: Re: bhyve zfs resizing To: freebsd-virtualization@freebsd.org, FreeBSD References: <20190318150404.GB91631@rpi3.zyxst.net> <20190318152525.GC91631@rpi3.zyxst.net> <20190319002325.GD91631@rpi3.zyxst.net> From: Miroslav Lachman <000.fbsd@quip.cz> Message-ID: <49d8d837-b734-0770-20fd-a439c6436a4d@quip.cz> Date: Tue, 19 Mar 2019 01:49:10 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Firefox/52.0 SeaMonkey/2.49.3 MIME-Version: 1.0 In-Reply-To: <20190319002325.GD91631@rpi3.zyxst.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 339C186F97 X-Spamd-Bar: ++++ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [4.29 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_SPAM_SHORT(0.98)[0.980,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[quip.cz]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.96)[0.962,0]; RCVD_COUNT_THREE(0.00)[3]; IP_SCORE(0.19)[ip: (0.48), ipnet: 94.124.104.0/21(0.24), asn: 42000(0.19), country: CZ(0.07)]; MX_GOOD(-0.01)[cached: elsa.codelab.cz]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[4.105.124.94.list.dnswl.org : 127.0.10.0]; NEURAL_SPAM_LONG(0.97)[0.967,0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[000.fbsd@quip.cz,SRS0=58Nm=RW=quip.cz=000.fbsd@elsa.codelab.cz]; RECEIVED_SPAMHAUS_PBL(0.00)[209.16.49.86.zen.spamhaus.org : 127.0.0.11]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ]; FROM_NEQ_ENVFROM(0.00)[000.fbsd@quip.cz,SRS0=58Nm=RW=quip.cz=000.fbsd@elsa.codelab.cz]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Mar 2019 00:49:35 -0000 tech-lists wrote on 2019/03/19 01:23: > Am I correct? In that I should have used UFS in the guest rather than > zfs? Or was it the encryption? As Alan already wrote - you can use ZFS inside of the guest but I would never choose ZFS in zvol backed guest. I prefere UFS. It is faster and does not need so much memory as ZFS does. My VirtualBox and Bhyve guests are small. Sometimes <1GB of RAM. Sometimes 2GB of RAM and that is very small for ZFS. May be you can try to limit ZFS ARC size in /etc/sysctl.conf or in /boot/loader.conf vfs.zfs.arc_max Choose about 1/4 of your guest's RAM size and test it again. Kind regards Miroslav Lachman From owner-freebsd-virtualization@freebsd.org Tue Mar 19 02:46:41 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 71103153A5EC for ; Tue, 19 Mar 2019 02:46:41 +0000 (UTC) (envelope-from vas@mpeks.tomsk.su) Received: from admin.sibptus.ru (admin.sibptus.ru [IPv6:2001:19f0:5001:21dc::10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A49FA8E473 for ; Tue, 19 Mar 2019 02:46:40 +0000 (UTC) (envelope-from vas@mpeks.tomsk.su) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sibptus.ru; s=20181118; h=Message-ID:Subject:To:From:Date:In-Reply-To; bh=JdKe2TYUcbcASPvurfcBUEw9d+8KaN7PYbtLGqrCQME=; b=kMAQPTNHjKIYJ1+ZIWjd5VanS3 v+STZSW5bfjpwmEFQ/HVcuXODguQJZbejsXlpQL7bba6IruXYTLd7WkD5jq2IAkqigiQlDzcvlmWe vroc8HEOH0XUHMq/YqMZtMbIPmIIo/QsmZrEU6AgKiMscOnOf2jNuJwJoHOBO0yJDokY=; Received: from vas by admin.sibptus.ru with local (Exim 4.92 (FreeBSD)) (envelope-from ) id 1h64ly-0002HY-UA for freebsd-virtualization@freebsd.org; Tue, 19 Mar 2019 09:46:38 +0700 Date: Tue, 19 Mar 2019 09:46:38 +0700 From: Victor Sudakov To: freebsd-virtualization@freebsd.org Subject: bhyve and vfs.zfs.arc_max, and zfs tuning for a hypervisor Message-ID: <20190319024638.GA8193@admin.sibptus.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="bg08WKrSYDhXBjb5" Content-Disposition: inline X-PGP-Key: http://www.dreamwidth.org/pubkey?user=victor_sudakov X-PGP-Fingerprint: 10E3 1171 1273 E007 C2E9 3532 0DA4 F259 9B5E C634 User-Agent: Mutt/1.11.3 (2019-02-01) Sender: Victor Sudakov X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Mar 2019 02:46:41 -0000 --bg08WKrSYDhXBjb5 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Dear Colleagues, I have a FreeBSD 11.2 system with 32G RAM which I'm going to use as a bhyve host with zvols and sparse zvols, with (almost) no services/daemons of its own.=20 Could you please clarify some points for me? 1. Does ARC actually cache zfs volumes (not files/datasets)? 2. If ARC does cache volumes, does this cache make sense on a hypervisor, because guest OSes will probably have their own disk cache anyway. 3. Would it make sense to limit vfs.zfs.arc_max to 1/8 or even less of total RAM, so that most RAM is available to guest machines? 4. What other zfs tuning measures can you suggest for a bhyve hypervisor? --=20 Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/ --bg08WKrSYDhXBjb5 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEcBAEBAgAGBQJckFgOAAoJEA2k8lmbXsY0A24H/3GW0qbH0lZ2em6XApGBXBqX pid3dfr2kBGoNLNlMhdcJgGE1uyouVIfFTrq+rma4ZDE8jWmLd0OBIHLxSH6WvV4 0IoycEKUrJP59StI1zk8UCsBXDYplHxDEMsXGrsdbUFP4BET4r9K4E19vHBak8PQ eiXGEgjD3ctfV7AuaBBJnQqRBvfa+pTakmTqzv081YQFm6kF0DNph35YlLDEWRhv H+cv079/ckHUQl5G+WihmhlRaWYKsGRm7adhMw+BJbQJ0wZf9A3w5HXpvDh0rMYd SnjIqy5QCpqfsueHX0NIZrEMzacu0WnZ9NNRxDTSakMocm4Txdo43X5+YrhsWlE= =uMHF -----END PGP SIGNATURE----- --bg08WKrSYDhXBjb5-- From owner-freebsd-virtualization@freebsd.org Tue Mar 19 08:06:17 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1C0BB154597C for ; Tue, 19 Mar 2019 08:06:17 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8975671336 for ; Tue, 19 Mar 2019 08:06:15 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from hugo10.ka.punkt.de (hugo10.ka.punkt.de [217.29.44.10]) by gate2.intern.punkt.de with ESMTP id x2J86DRB005753; Tue, 19 Mar 2019 09:06:13 +0100 (CET) Received: from [217.29.44.250] ([217.29.44.250]) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id x2J86DOM074797; Tue, 19 Mar 2019 09:06:13 +0100 (CET) (envelope-from hausen@punkt.de) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 12.0 \(3445.100.39\)) Subject: Re: bhyve and vfs.zfs.arc_max, and zfs tuning for a hypervisor From: "Patrick M. Hausen" In-Reply-To: <20190319024638.GA8193@admin.sibptus.ru> Date: Tue, 19 Mar 2019 09:06:11 +0100 Cc: freebsd-virtualization@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <20190319024638.GA8193@admin.sibptus.ru> To: Victor Sudakov X-Mailer: Apple Mail (2.3445.100.39) X-Rspamd-Queue-Id: 8975671336 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.97 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[]; NEURAL_HAM_SHORT(-0.97)[-0.971,0] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Mar 2019 08:06:17 -0000 Hi! > Am 19.03.2019 um 03:46 schrieb Victor Sudakov : > 1. Does ARC actually cache zfs volumes (not files/datasets)? Yes it does. > 2. If ARC does cache volumes, does this cache make sense on a = hypervisor, > because guest OSes will probably have their own disk cache anyway. IMHO not much, because the guest OS is relying on the fact that when it writes it=E2=80=99s own cached data out to =E2=80=9Edisk=E2=80=9C, it = will be committed to stable storage. > 3. Would it make sense to limit vfs.zfs.arc_max to 1/8 or even less of > total RAM, so that most RAM is available to guest machines? Yes if you build your own solution on plain FreeBSD. No if you are = running FreeNAS which already tries to autotune the ARC size according to the memory committed to VMs. > 4. What other zfs tuning measures can you suggest for a bhyve > hypervisor? e.g. zfs set sync=3Dalways zfs/vm if zfs/vm is the dataset under which you create the ZVOLs for your = emulated disks. I=E2=80=99m using this for all my VM =E2=80=9Edisks=E2=80=9C and have = added a 16 GB SLOG device to my spinning disk pool - seems to work great. This is on a home = system. Our new data centre systems feature all NVME SSDs and no spinning rust. So no need for a separate SLOG. HTH, Patrick --=20 punkt.de GmbH Internet - Dienstleistungen - Beratung Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100 76133 Karlsruhe info@punkt.de http://punkt.de AG Mannheim 108285 Gf: Juergen Egeling From owner-freebsd-virtualization@freebsd.org Tue Mar 19 13:08:11 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 61B501550811 for ; Tue, 19 Mar 2019 13:08:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id EA39D84988 for ; Tue, 19 Mar 2019 13:08:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id AAB6C1550810; Tue, 19 Mar 2019 13:08:10 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 94F4E155080F for ; Tue, 19 Mar 2019 13:08:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2B28484985 for ; Tue, 19 Mar 2019 13:08:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 40926B063 for ; Tue, 19 Mar 2019 13:08:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id x2JD89x8072944 for ; Tue, 19 Mar 2019 13:08:09 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id x2JD89jw072943 for virtualization@FreeBSD.org; Tue, 19 Mar 2019 13:08:09 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 236042] Windows Server 2016 Hyper-V snapshot triggers SCSI errors Date: Tue, 19 Mar 2019 13:08:09 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: j.morillo@educationetformation.fr X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Mar 2019 13:08:11 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D236042 j.morillo@educationetformation.fr changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |j.morillo@educationetformat | |ion.fr --- Comment #2 from j.morillo@educationetformation.fr --- I'm also having same problem with different setup: - HyperV 2016 with OPNSense as Guest (FreeBSD 11.1-RELEASE-p17) This VM has a replicate on another HyperV wich often failed (need to force = sync again). Here is FreeBSD log: (da0:storvsc0:0:0:0): WRITE(10). CDB: 2a 00 00 cd ac a8 00 01 00 00 (da0:storvsc0:0:0:0): CAM status: SCSI Status Error (da0:storvsc0:0:0:0): SCSI status: Check Condition (da0:storvsc0:0:0:0): SCSI sense: UNIT ATTENTION asc:3f,2 (Changed operating definition) (da0:storvsc0:0:0:0): Retrying command (per sense data) (da0:storvsc0:0:0:0): WRITE(10). CDB: 2a 00 00 ca 90 68 00 00 40 00 (da0:storvsc0:0:0:0): CAM status: SCSI Status Error (da0:storvsc0:0:0:0): SCSI status: Check Condition (da0:storvsc0:0:0:0): SCSI sense: UNIT ATTENTION asc:3f,2 (Changed operating definition) (da0:storvsc0:0:0:0): Retrying command (per sense data) --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Wed Mar 20 01:52:38 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4B0CF15443B4 for ; Wed, 20 Mar 2019 01:52:38 +0000 (UTC) (envelope-from vas@mpeks.tomsk.su) Received: from admin.sibptus.ru (admin.sibptus.ru [IPv6:2001:19f0:5001:21dc::10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7D30A80451 for ; Wed, 20 Mar 2019 01:52:37 +0000 (UTC) (envelope-from vas@mpeks.tomsk.su) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sibptus.ru; s=20181118; h=In-Reply-To:Message-ID:Subject:To:From:Date; bh=53CR8DbjxNXHgU1PD9OnucoU9ZALOyPksYzTnZIcDJs=; b=Epy191gURcghI2j4DnbfZgpMpP 5Rw3A8fOUF6Ley8Km8obtjXXiQ/P2lwTn/bXUxNJxTySiMVvfjmptRrHlkHsh9XcG//lMxjBThMuT hImLtpyTr0SK0hvUiZNn69uCtv0FGacgpyExxOoHDsLmlJt+xPmyaPKUnsg6WFC37Avk=; Received: from vas by admin.sibptus.ru with local (Exim 4.92 (FreeBSD)) (envelope-from ) id 1h6QPE-0009Hf-Dn for freebsd-virtualization@freebsd.org; Wed, 20 Mar 2019 08:52:36 +0700 Date: Wed, 20 Mar 2019 08:52:36 +0700 From: Victor Sudakov To: freebsd-virtualization@freebsd.org Subject: Re: bhyve and vfs.zfs.arc_max, and zfs tuning for a hypervisor Message-ID: <20190320015236.GA35511@admin.sibptus.ru> References: <20190319024638.GA8193@admin.sibptus.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="YZ5djTAD1cGYuMQK" Content-Disposition: inline In-Reply-To: X-PGP-Key: http://www.dreamwidth.org/pubkey?user=victor_sudakov X-PGP-Fingerprint: 10E3 1171 1273 E007 C2E9 3532 0DA4 F259 9B5E C634 User-Agent: Mutt/1.11.3 (2019-02-01) Sender: Victor Sudakov X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Mar 2019 01:52:38 -0000 --YZ5djTAD1cGYuMQK Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Patrick M. Hausen wrote: >=20 > > 1. Does ARC actually cache zfs volumes (not files/datasets)? >=20 > Yes it does. >=20 > > 2. If ARC does cache volumes, does this cache make sense on a hyperviso= r, > > because guest OSes will probably have their own disk cache anyway. >=20 > IMHO not much, because the guest OS is relying on the fact that when > it writes it=E2=80=99s own cached data out to =E2=80=9Edisk=E2=80=9C, it = will be committed to > stable storage. This is an important point. > > 3. Would it make sense to limit vfs.zfs.arc_max to 1/8 or even less of > > total RAM, so that most RAM is available to guest machines? >=20 > Yes if you build your own solution on plain FreeBSD. No if you are running > FreeNAS which already tries to autotune the ARC size according to the > memory committed to VMs. >=20 > > 4. What other zfs tuning measures can you suggest for a bhyve > > hypervisor? >=20 > e.g. > zfs set sync=3Dalways zfs/vm >=20 > if zfs/vm is the dataset under which you create the ZVOLs for your emulat= ed > disks. Well, bhyve already has an option for this: The block-device-options are: nocache Open the file with O_DIRECT. direct Open the file using O_SYNC. ro Force the file to be opened read-only. I think something like=20 "-s 4:0,virtio-blk,/dev/zvol/zroot/vm/mail/disk0,direct" would do the same? >=20 > I=E2=80=99m using this for all my VM =E2=80=9Edisks=E2=80=9C and have add= ed a 16 GB SLOG device > to my spinning disk pool - seems to work great. This is on a home system. Is SLOG also used by zfs volumes? --=20 Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/ --YZ5djTAD1cGYuMQK Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEbBAEBAgAGBQJckZzkAAoJEA2k8lmbXsY0uukH+JC3YmtDNjPduavG8UcnK1Dy nMPQwavDdOyKK8v37n1K11ftUbdEOaHC+IWrrszuUxc3Ui/hPG6K3FAaLtLtwllp 6p+64AJ9w9t/nIoMfIGJfmU1BMQuE0JeXcJCaohoDXvyMfCxXApdxlK3iRRYa97T 8ch7Bc+f77cYuwvJkj0eW2pSmANObTsj5kcCLDhnWX9EuBoDpE1rmhfB1R1zxJaa 5PYpNChx8tqyb+O/Usex6pnZ9B7pKzO4lf4mDB13yg6ysWYnCGhECi54yZbE146h OMaJsLpE1DxOe3aOCGsx19o5Fi+/mXb4qm1nwWTE5Jv+/zvtlVxjla+UySB0Dg== =dqCO -----END PGP SIGNATURE----- --YZ5djTAD1cGYuMQK-- From owner-freebsd-virtualization@freebsd.org Wed Mar 20 09:34:51 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 94715154D8A9 for ; Wed, 20 Mar 2019 09:34:51 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from smtp-a.userve.net (smtp-outbound.userve.net [217.196.1.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.userve.net", Issuer "Thawte RSA CA 2018" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0F0208D68A for ; Wed, 20 Mar 2019 09:34:50 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from owa.usd-group.com (owa.usd-group.com [217.196.1.2]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp-a.userve.net (Postfix) with ESMTPS id A3828428; Wed, 20 Mar 2019 09:34:41 +0000 (GMT) Received: from SERVER.ad.usd-group.com (192.168.0.1) by SERVER.ad.usd-group.com (192.168.0.1) with Microsoft SMTP Server (TLS) id 15.0.847.32; Wed, 20 Mar 2019 09:34:41 +0000 Received: from SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9]) by SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9%12]) with mapi id 15.00.0847.030; Wed, 20 Mar 2019 09:34:41 +0000 From: Matt Churchyard To: Victor Sudakov , "freebsd-virtualization@freebsd.org" Subject: RE: bhyve and vfs.zfs.arc_max, and zfs tuning for a hypervisor Thread-Topic: bhyve and vfs.zfs.arc_max, and zfs tuning for a hypervisor Thread-Index: AQHU3f4eR9n6J576SkWf85jIrU+hAKYSmT6AgAEp9ACAAHlQEA== Date: Wed, 20 Mar 2019 09:34:41 +0000 Message-ID: <6388ebd5a12949179ff6da33eb107205@SERVER.ad.usd-group.com> References: <20190319024638.GA8193@admin.sibptus.ru> <20190320015236.GA35511@admin.sibptus.ru> In-Reply-To: <20190320015236.GA35511@admin.sibptus.ru> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.0.10] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-Rspamd-Queue-Id: 0F0208D68A X-Spamd-Bar: ------ X-Spamd-Result: default: False [-6.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.99)[-0.989,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Mar 2019 09:34:51 -0000 UGF0cmljayBNLiBIYXVzZW4gd3JvdGU6DQo+IA0KPiA+IDEuIERvZXMgQVJDIGFjdHVhbGx5IGNh Y2hlIHpmcyB2b2x1bWVzIChub3QgZmlsZXMvZGF0YXNldHMpPw0KPiANCj4gWWVzIGl0IGRvZXMu DQoNCkkgZmluZCB0aGlzIGRpc3RpbmN0aW9uIGJldHdlZW4gdm9sdW1lcy9maWxlcy9ldGMgYW5k IHdoYXQgaXMgY2FjaGVkIGNhdXNlcyBjb25mdXNpb24gKGFzIHdlbGwgYXMgInZvbHVtZXMgbm90 IGRhdGFzZXRzIikuDQoNCkJvdGggWlZPTHMgYW5kIFogZmlsZSBzeXN0ZW1zIGFyZSB0eXBlcyBv ZiBkYXRhc2V0LiBBIGRhdGFzZXQgc3RvcmVzIGRhdGEgaW4gcmVjb3JkcyAodXN1YWxseSB1cCB0 byAxMjhrYiBpbiBzaXplKS4NCkl0J3MgdGhlc2UgcmVjb3JkcyB0aGF0IGFyZSBjYWNoZWQgKGFu ZCB0aGF0IG1vc3QgWkZTIGZ1bmN0aW9ucyBzdWNoIGFzIGNvbXByZXNzaW9uL3JhaWR6L3ppbC9l dGMgd29yayB3aXRoKQ0KDQpBcyBmYXIgYXMgdGhlIFpGUyBsb3dlciBsZXZlbHMgYXJlIGNvbmNl cm5lZCwgdGhlcmUgaXMgbm8gZGlmZmVyZW5jZSBiZXR3ZWVuIGEgdm9sdW1lIGFuZCBhIGZpbGUg c3lzdGVtLg0KDQo+IA0KPiA+IDIuIElmIEFSQyBkb2VzIGNhY2hlIHZvbHVtZXMsIGRvZXMgdGhp cyBjYWNoZSBtYWtlIHNlbnNlIG9uIGEgDQo+ID4gaHlwZXJ2aXNvciwgYmVjYXVzZSBndWVzdCBP U2VzIHdpbGwgcHJvYmFibHkgaGF2ZSB0aGVpciBvd24gZGlzayBjYWNoZSBhbnl3YXkuDQo+IA0K PiBJTUhPIG5vdCBtdWNoLCBiZWNhdXNlIHRoZSBndWVzdCBPUyBpcyByZWx5aW5nIG9uIHRoZSBm YWN0IHRoYXQgd2hlbiANCj4gaXQgd3JpdGVzIGl04oCZcyBvd24gY2FjaGVkIGRhdGEgb3V0IHRv IOKAnmRpc2vigJwsIGl0IHdpbGwgYmUgY29tbWl0dGVkIHRvIA0KPiBzdGFibGUgc3RvcmFnZS4N Cg0KTWF5YmUgSSd2ZSBtaXNzZWQgc29tZXRoaW5nIGJ1dCBJIGRvbid0IHF1aXRlIGdldCB0aGUg bGluayBiZXR3ZWVuIHJlYWQgY2FjaGUgKEFSQykgYW5kIGd1ZXN0IHdyaXRlcyBoZXJlPw0KDQo+ IFRoaXMgaXMgYW4gaW1wb3J0YW50IHBvaW50Lg0KDQo+ID4gMy4gV291bGQgaXQgbWFrZSBzZW5z ZSB0byBsaW1pdCB2ZnMuemZzLmFyY19tYXggdG8gMS84IG9yIGV2ZW4gbGVzcyANCj4gPiBvZiB0 b3RhbCBSQU0sIHNvIHRoYXQgbW9zdCBSQU0gaXMgYXZhaWxhYmxlIHRvIGd1ZXN0IG1hY2hpbmVz Pw0KPiANCj4gWWVzIGlmIHlvdSBidWlsZCB5b3VyIG93biBzb2x1dGlvbiBvbiBwbGFpbiBGcmVl QlNELiBObyBpZiB5b3UgYXJlIA0KPiBydW5uaW5nIEZyZWVOQVMgd2hpY2ggYWxyZWFkeSB0cmll cyB0byBhdXRvdHVuZSB0aGUgQVJDIHNpemUgYWNjb3JkaW5nIA0KPiB0byB0aGUgbWVtb3J5IGNv bW1pdHRlZCB0byBWTXMuDQo+IA0KPiA+IDQuIFdoYXQgb3RoZXIgemZzIHR1bmluZyBtZWFzdXJl cyBjYW4geW91IHN1Z2dlc3QgZm9yIGEgYmh5dmUgDQo+ID4gaHlwZXJ2aXNvcj8NCj4gDQo+IGUu Zy4NCj4gCXpmcyBzZXQgc3luYz1hbHdheXMgemZzL3ZtDQo+IA0KPiBpZiB6ZnMvdm0gaXMgdGhl IGRhdGFzZXQgdW5kZXIgd2hpY2ggeW91IGNyZWF0ZSB0aGUgWlZPTHMgZm9yIHlvdXIgDQo+IGVt dWxhdGVkIGRpc2tzLg0KDQo+V2VsbCwgYmh5dmUgYWxyZWFkeSBoYXMgYW4gb3B0aW9uIGZvciB0 aGlzOg0KDQo+VGhlIGJsb2NrLWRldmljZS1vcHRpb25zIGFyZToNCg0KPm5vY2FjaGUgICBPcGVu IHRoZSBmaWxlIHdpdGggT19ESVJFQ1QuDQo+ZGlyZWN0ICAgIE9wZW4gdGhlIGZpbGUgdXNpbmcg T19TWU5DLg0KPnJvICAgICAgICBGb3JjZSB0aGUgZmlsZSB0byBiZSBvcGVuZWQgcmVhZC1vbmx5 Lg0KDQo+SSB0aGluayBzb21ldGhpbmcgbGlrZQ0KPiItcyA0OjAsdmlydGlvLWJsaywvZGV2L3p2 b2wvenJvb3Qvdm0vbWFpbC9kaXNrMCxkaXJlY3QiDQo+d291bGQgZG8gdGhlIHNhbWU/DQoNCj4g DQo+IEnigJltIHVzaW5nIHRoaXMgZm9yIGFsbCBteSBWTSDigJ5kaXNrc+KAnCBhbmQgaGF2ZSBh ZGRlZCBhIDE2IEdCIFNMT0cgDQo+IGRldmljZSB0byBteSBzcGlubmluZyBkaXNrIHBvb2wgLSBz ZWVtcyB0byB3b3JrIGdyZWF0LiBUaGlzIGlzIG9uIGEgaG9tZSBzeXN0ZW0uDQoNCj4gSXMgU0xP RyBhbHNvIHVzZWQgYnkgemZzIHZvbHVtZXM/DQoNCkFzIGFib3ZlLCB0aGUgY29yZSBvZiBaRlMg ZG9lc24ndCBjYXJlIHdoYXQgdHlwZSBvZiBkYXRhc2V0IGl0IGlzIHdvcmtpbmcgd2l0aC4gQVJD L1pJTC9ldGMgYWxsIHdvcmsgZXhhY3RseSB0aGUgc2FtZS4NCg0KLS0NClZpY3RvciBTdWRha292 LCAgVkFTNC1SSVBFLCBWQVM0Ny1SSVBODQoyOjUwMDUvNDlAZmlkb25ldCBodHRwOi8vdmFzLnRv bXNrLnJ1Lw0K From owner-freebsd-virtualization@freebsd.org Wed Mar 20 17:11:41 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9791515596FB for ; Wed, 20 Mar 2019 17:11:41 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id E6BC26FBD0 for ; Wed, 20 Mar 2019 17:11:40 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: by mailman.ysv.freebsd.org (Postfix) id 9E29315596FA; Wed, 20 Mar 2019 17:11:40 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6141515596F9 for ; Wed, 20 Mar 2019 17:11:40 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 222796FB03 for ; Wed, 20 Mar 2019 17:11:32 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mh0.gentlemail.de (mh0.gentlemail.de [IPv6:2a00:e10:2800::a135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id x2KHBTbR013742; Wed, 20 Mar 2019 18:11:29 +0100 (CET) (envelope-from freebsd@omnilan.de) Received: from titan.inop.mo1.omnilan.net (s1.omnilan.de [217.91.127.234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id 60109C60; Wed, 20 Mar 2019 18:11:29 +0100 (CET) Subject: Re: NVMe and Bhyve To: Victor Sudakov , "freebsd-virtualization@freebsd.org" References: <20190207174453.Horde.hWBGfoLlymCpipxLv8WUJo9@app.eeeit.de> <5121268a-1ead-9858-86a7-27f75048feb2@yuripv.net> <20190212084333.GA46685@admin.sibptus.ru> <20190212130506.GA62989@admin.sibptus.ru> <20190215044134.GA52633@admin.sibptus.ru> <20190216041134.GA75357@admin.sibptus.ru> From: Harry Schmalzbauer Organization: OmniLAN Message-ID: <21d661a4-d51d-d685-599f-c5430dea83ff@omnilan.de> Date: Wed, 20 Mar 2019 18:11:28 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: <20190216041134.GA75357@admin.sibptus.ru> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]); Wed, 20 Mar 2019 18:11:29 +0100 (CET) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: ; Sender-helo: mh0.gentlemail.de; ) X-Rspamd-Queue-Id: 222796FB03 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of freebsd@omnilan.de designates 2a00:e10:2800::a130 as permitted sender) smtp.mailfrom=freebsd@omnilan.de X-Spamd-Result: default: False [-6.77 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[omnilan.de]; HAS_ORG_HEADER(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[mx0.gentlemail.de]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.93)[-0.934,0]; IP_SCORE(-3.53)[ip: (-9.24), ipnet: 2a00:e10:2800::/64(-4.71), asn: 25074(-3.69), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:25074, ipnet:2a00:e10:2800::/64, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Mar 2019 17:11:41 -0000 Am 16.02.2019 um 05:11 schrieb Victor Sudakov: > Stefan Bethke wrote: >>> I was just worried that using ahci-hd instead of paravirtualized disks >>> causes suboptimal performance of guests. Do you think there is a hope to get >>> paravirtualized disks support for Windows guests in bhyve? >> I was under the impression that the Windows VirtIO disk driver works >> with the Bhyve virtio-blk device. You need to provide these drivers to >> Windows somehow (for example, as a second CD image during >> installation): >> https://www.linux-kvm.org/page/WindowsGuestDrivers/Download_Drivers > I use NetKVM (VirtIO network) driver from there all the time, because > e1000 is bad. > > However, I've read a number of bhyve resources including > https://wiki.freebsd.org/bhyve/Windows and I've never found a success > story with paravirtualized Windows disk drivers. > > If anyone has been successful with viostor(?), please share your experience. You'll need https://github.com/freenas/os/commit/0e4d6e1826f8aa7041cbeeb4365c797eeec5c5f4 if I remember correctly. I asked for commit, since I'm using this in production for a long time, but once upon a time, there has been a linux-guest specific side effect. Don't know much about it and haven't had a chance to reproduce it. In my bhyve world, any guest runs fine with the higher BLOCKIF_IOV limit. -harry From owner-freebsd-virtualization@freebsd.org Wed Mar 20 23:30:13 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D9CFB153E6CB for ; Wed, 20 Mar 2019 23:30:13 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5C01086739 for ; Wed, 20 Mar 2019 23:30:12 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from hugo10.ka.punkt.de (hugo10.ka.punkt.de [217.29.44.10]) by gate2.intern.punkt.de with ESMTP id x2KNUAMP042638; Thu, 21 Mar 2019 00:30:10 +0100 (CET) Received: from [217.29.46.66] (unassigned [217.29.46.66] (may be forged)) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id x2KNUA9p063833; Thu, 21 Mar 2019 00:30:10 +0100 (CET) (envelope-from hausen@punkt.de) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 12.2 \(3445.102.3\)) Subject: Re: bhyve and vfs.zfs.arc_max, and zfs tuning for a hypervisor From: "Patrick M. Hausen" In-Reply-To: <20190320015236.GA35511@admin.sibptus.ru> Date: Thu, 21 Mar 2019 00:30:09 +0100 Cc: freebsd-virtualization@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <0520BA4F-FA4A-4BE9-85D9-2CECF2018282@punkt.de> References: <20190319024638.GA8193@admin.sibptus.ru> <20190320015236.GA35511@admin.sibptus.ru> To: Victor Sudakov X-Mailer: Apple Mail (2.3445.102.3) X-Rspamd-Queue-Id: 5C01086739 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.99)[-0.989,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Mar 2019 23:30:14 -0000 Hi! > Am 20.03.2019 um 02:52 schrieb Victor Sudakov : > Is SLOG also used by zfs volumes? Yes, but for synchronous writes only, if I=E2=80=99m not mistaken. So fundamentally yes, but in most cases no. Patrick --=20 punkt.de GmbH Internet - Dienstleistungen - Beratung Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100 76133 Karlsruhe info@punkt.de http://punkt.de AG Mannheim 108285 Gf: Juergen Egeling From owner-freebsd-virtualization@freebsd.org Wed Mar 20 23:30:49 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E464F153E881 for ; Wed, 20 Mar 2019 23:30:48 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 81C00867B6 for ; Wed, 20 Mar 2019 23:30:47 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from hugo10.ka.punkt.de (hugo10.ka.punkt.de [217.29.44.10]) by gate2.intern.punkt.de with ESMTP id x2KNUkxh042658; Thu, 21 Mar 2019 00:30:46 +0100 (CET) Received: from [217.29.46.66] (unassigned [217.29.46.66] (may be forged)) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id x2KNUA9q063833; Thu, 21 Mar 2019 00:30:46 +0100 (CET) (envelope-from hausen@punkt.de) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 12.2 \(3445.102.3\)) Subject: Re: bhyve and vfs.zfs.arc_max, and zfs tuning for a hypervisor From: "Patrick M. Hausen" In-Reply-To: <6388ebd5a12949179ff6da33eb107205@SERVER.ad.usd-group.com> Date: Thu, 21 Mar 2019 00:30:45 +0100 Cc: Victor Sudakov , "freebsd-virtualization@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: References: <20190319024638.GA8193@admin.sibptus.ru> <20190320015236.GA35511@admin.sibptus.ru> <6388ebd5a12949179ff6da33eb107205@SERVER.ad.usd-group.com> To: Matt Churchyard X-Mailer: Apple Mail (2.3445.102.3) X-Rspamd-Queue-Id: 81C00867B6 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of hausen@punkt.de designates 217.29.33.131 as permitted sender) smtp.mailfrom=hausen@punkt.de X-Spamd-Result: default: False [-3.15 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.997,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:217.29.32.0/20]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[punkt.de]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: mailin.pluspunkthosting.de]; NEURAL_HAM_SHORT(-0.96)[-0.957,0]; RCVD_IN_DNSWL_NONE(0.00)[131.33.29.217.list.dnswl.org : 127.0.10.0]; IP_SCORE(-0.38)[ipnet: 217.29.32.0/20(-1.06), asn: 16188(-0.85), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16188, ipnet:217.29.32.0/20, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Mar 2019 23:30:49 -0000 Hi Matt, > Am 20.03.2019 um 10:34 schrieb Matt Churchyard = : >>> 2. If ARC does cache volumes, does this cache make sense on a=20 >>> hypervisor, because guest OSes will probably have their own disk = cache anyway. >>=20 >> IMHO not much, because the guest OS is relying on the fact that when=20= >> it writes it=E2=80=99s own cached data out to =E2=80=9Edisk=E2=80=9C, = it will be committed to=20 >> stable storage. >=20 > Maybe I've missed something but I don't quite get the link between = read cache (ARC) and guest writes here? You are correct - I confused ARC and ZIL. I still recommend to set sync=3Dalways for hypervisor =E2=80=9Edisk=E2=80=9C ZVOLs. Kind regards Patrick --=20 punkt.de GmbH Internet - Dienstleistungen - Beratung Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100 76133 Karlsruhe info@punkt.de http://punkt.de AG Mannheim 108285 Gf: Juergen Egeling From owner-freebsd-virtualization@freebsd.org Thu Mar 21 01:47:44 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0FEF81546522 for ; Thu, 21 Mar 2019 01:47:44 +0000 (UTC) (envelope-from mike.gerdts@joyent.com) Received: from mail-pf1-x42e.google.com (mail-pf1-x42e.google.com [IPv6:2607:f8b0:4864:20::42e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0F8328CD49 for ; Thu, 21 Mar 2019 01:47:41 +0000 (UTC) (envelope-from mike.gerdts@joyent.com) Received: by mail-pf1-x42e.google.com with SMTP id 8so3235327pfr.4 for ; Wed, 20 Mar 2019 18:47:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joyent.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=K2z6GqCzoNkhrL9ZT2QTRKiHADBD+R2KKvTNGe9bPCs=; b=M84TeUV1V8hpC/Nv7BmSFxkrmLW16pDFVBHoFyNoQCUfRQ7ew2oxWHC8IBJmGW/MFp ZX3+62kFWKSyqqwzRLctVyTqHf4wIS+AVTFDs1fiaVYeXfXDVb7vWH0/2NL9CJxCkBz5 DaIUVRFAKBUmqNtJ1kB9YW7jZkwMMGwllVXmw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=K2z6GqCzoNkhrL9ZT2QTRKiHADBD+R2KKvTNGe9bPCs=; b=CZf2iuuIWPQaYiWARrpI2oQVbphoiqJs37tFeMRvMihqO2gEFdH661F0q/h15HrFJU QNZKj8x8FwOhL8b0hLIv3uAeeSWgm3iip96ZL0/4j7vCUqgFzRbgAtVqgzS5jNfPJV4X zeDuXfjB71iTeh2umo4+JFT0rttdjC+5kB3L6ZLAXPSf8nBF5hDcTeI4QNvJLNK8Bn5p dYKHOVSL0wA9Y4wLKn9RijL4Ma2qSdT2F43p8zHwbfHgpb84B/OQ7dbg/ArdbZ21n4tf Bbl/Nr+uTTTRpXpykRcwlT3mKQNpE1QTI/ba8z+aiwFd1MgePe41HCDYICzuqx9uizu3 IZEg== X-Gm-Message-State: APjAAAUnYyvrdI+i+/Ib9w9/eQOgxsvD/2qevHvEXSaiolnT0OuyEEk3 NyQVdXhxTU311EViSePLjcGeFfrEf0Af3tFsRlWRYHIQKEE= X-Google-Smtp-Source: APXvYqyj1XoFf0aXdNGgy/dYHzAe4gyE+MRxMt2OiDq00JhgxzoDiaWY78hB6kPJ5VGHEKY4gUatjR90IwIYJLVG+bo= X-Received: by 2002:a65:50c4:: with SMTP id s4mr1008041pgp.33.1553132860604; Wed, 20 Mar 2019 18:47:40 -0700 (PDT) MIME-Version: 1.0 References: <20190319024638.GA8193@admin.sibptus.ru> In-Reply-To: From: Mike Gerdts Date: Wed, 20 Mar 2019 20:47:29 -0500 Message-ID: Subject: Re: bhyve and vfs.zfs.arc_max, and zfs tuning for a hypervisor To: "Patrick M. Hausen" Cc: Victor Sudakov , freebsd-virtualization@freebsd.org X-Rspamd-Queue-Id: 0F8328CD49 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=joyent.com header.s=google header.b=M84TeUV1; spf=pass (mx1.freebsd.org: domain of mike.gerdts@joyent.com designates 2607:f8b0:4864:20::42e as permitted sender) smtp.mailfrom=mike.gerdts@joyent.com X-Spamd-Result: default: False [-6.30 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[joyent.com:s=google]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; DMARC_NA(0.00)[joyent.com]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[joyent.com:+]; MX_GOOD(-0.01)[alt1.aspmx.l.google.com,aspmx.l.google.com,aspmx2.googlemail.com,alt2.aspmx.l.google.com,aspmx3.googlemail.com]; RCVD_IN_DNSWL_NONE(0.00)[e.2.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.91)[-0.907,0]; IP_SCORE(-2.88)[ip: (-9.40), ipnet: 2607:f8b0::/32(-2.81), asn: 15169(-2.12), country: US(-0.07)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Mar 2019 01:47:44 -0000 On Tue, Mar 19, 2019 at 3:07 AM Patrick M. Hausen wrote: > Hi! > > > Am 19.03.2019 um 03:46 schrieb Victor Sudakov : > > 1. Does ARC actually cache zfs volumes (not files/datasets)? > > Yes it does. > > > 2. If ARC does cache volumes, does this cache make sense on a hyperviso= r, > > because guest OSes will probably have their own disk cache anyway. > > IMHO not much, because the guest OS is relying on the fact that when > it writes it=E2=80=99s own cached data out to =E2=80=9Edisk=E2=80=9C, it = will be committed to > stable storage. > I'd recommend caching at least metadata (primarycache=3Dmetadata). The gue= st will not cache zfs metadata and not having metadata in the cache can lead to a big hit in performance. The metadata in question here are things like block pointers that keep track of where the data is at - zfs can't find the data without metadata. I think the key decision as to whether you use primarycache=3Dmetadata or primarycache=3Dall comes down to whether you are after predictable performance or optimal performance. You will likely get worse performance with primarycache=3Dmetaadata (or especially with primarycache=3Dnone), presuming the host has RAM to spare. As you pack the system with more VMs or allocate more disk to existing VMs, you will probably find that primarycache=3Dmetadata leads steadier performance regardless of how much storage is used or how active other VMs are. > > 3. Would it make sense to limit vfs.zfs.arc_max to 1/8 or even less of > > total RAM, so that most RAM is available to guest machines? > > Yes if you build your own solution on plain FreeBSD. No if you are runnin= g > FreeNAS which already tries to autotune the ARC size according to the > memory committed to VMs. > > > 4. What other zfs tuning measures can you suggest for a bhyve > > hypervisor? > > e.g. > zfs set sync=3Dalways zfs/vm > > if zfs/vm is the dataset under which you create the ZVOLs for your emulat= ed > disks. > I'm not sure what the state of this is in FreeBSD, but in SmartOS we allow the guests to benefit from write caching if they negotiate FLUSH. Guests that do negotiate flush are expected to use proper barriers to flush the cache at critical times. When a FLUSH arrives, SmartOS bhyve issues an fsync(). To be clear - SmartOS bhyve is not actually caching writes in memory, it is just delaying transaction group commits. This avoids significant write inflation and associated latency. Support for FLUSH negotiation has greatly improved I/O performance - to the point that some tests show parity with running directly on the host pool. If not already in FreeBSD, this would probably be something of relatively high value to pull in. If you do go the route of sync=3Dalways and primarycache=3D,= be sure your guest block size and host volblocksize match. ZFS (on platforms I'm more familiar with, at least) defaults to volblocksize=3D8k. Most gues= t file systems these days seem to default to a block size of 4 KiB. If the guest file system issues a 4 KiB aligned write, that will turn into a read-modify-write cycle to stitch that 4 KiB guest block into the host's 8 KB block. If the adjacent guest block that is in the same 8 KiB host block is written in the next write, it will also turn into a read-modify-write cycle. If you are using ZFS in the guest, this can be particularly problematic because the guest ZFS will align writes with the guest pool's ashift, not with a guest dataset's recordsize or volblocksize. I discovered this in an extended benchmarking of zfs-on-zfs - primarily with primarycache=3Dmetadat= a and sync=3Dalways. The write inflation was quite significant: 3x was common. I tracked some of this down to alignment issues and part of it was due to sync writes causing the data to be written twice. George Wilson has a great talk where he describes the same issues I hit. https://www.youtube.com/watch?v=3D_-QAnKtIbGc I've mentioned write inflation related to sync writes a few times. One point that I think is poorly understood is that when ZFS is rushed into committing a write with fsync or similar, the immediate write is of ZIL blocks to the intent log. The intent log can be on a separate disk (slog, log=3D) or it can be on the disks that hold the pool's data. When the intent log is on the data disks, the data is written to the same disks multiple times: once as ZIL blocks and once as data blocks. Between these writes there will be full-disk head movement as the uberblocks are updated at the beginning and end of the disk. What I say above is based on experience with kernel zones on Solaris and bhyve on SmartOS. There are enough similarities that I expect bhyve on FreeBSD will be the same, but FreeBSD may have some strange-to-me zfs caching changes. Regards, Mike From owner-freebsd-virtualization@freebsd.org Thu Mar 21 02:38:35 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1F9C71548CDF for ; Thu, 21 Mar 2019 02:38:35 +0000 (UTC) (envelope-from vas@mpeks.tomsk.su) Received: from admin.sibptus.ru (admin.sibptus.ru [IPv6:2001:19f0:5001:21dc::10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1E8DB8EC50 for ; Thu, 21 Mar 2019 02:38:33 +0000 (UTC) (envelope-from vas@mpeks.tomsk.su) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sibptus.ru; s=20181118; h=In-Reply-To:Message-ID:Subject:To:From:Date; bh=/S9FwYoBtoHYmMEBCz1R5wze8Meiy0OPtz2C5WwI4es=; b=D5LQIGKdzB3xMjnLrAfjvwJqK9 vx8LxARzFVUh+jzlM/btQZIACPTcupsv0e55c/lXmpCT1rxM8luoUnigfdai7X9zjqCSpGJ/hjHvP 9YoL4ie4bQsqAENeufAshWCznRH+GOfxQKz/L71qnNIVZ6d+ILT7BOyQkOrGHys5iDJ8=; Received: from vas by admin.sibptus.ru with local (Exim 4.92 (FreeBSD)) (envelope-from ) id 1h6nbD-000F4C-W5 for freebsd-virtualization@freebsd.org; Thu, 21 Mar 2019 09:38:31 +0700 Date: Thu, 21 Mar 2019 09:38:31 +0700 From: Victor Sudakov To: "freebsd-virtualization@freebsd.org" Subject: Re: bhyve and vfs.zfs.arc_max, and zfs tuning for a hypervisor Message-ID: <20190321023831.GA57807@admin.sibptus.ru> References: <20190319024638.GA8193@admin.sibptus.ru> <20190320015236.GA35511@admin.sibptus.ru> <6388ebd5a12949179ff6da33eb107205@SERVER.ad.usd-group.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="bg08WKrSYDhXBjb5" Content-Disposition: inline In-Reply-To: <6388ebd5a12949179ff6da33eb107205@SERVER.ad.usd-group.com> X-PGP-Key: http://www.dreamwidth.org/pubkey?user=victor_sudakov X-PGP-Fingerprint: 10E3 1171 1273 E007 C2E9 3532 0DA4 F259 9B5E C634 User-Agent: Mutt/1.11.4 (2019-03-13) Sender: Victor Sudakov X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Mar 2019 02:38:35 -0000 --bg08WKrSYDhXBjb5 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Matt Churchyard wrote: > >=20 > > > 1. Does ARC actually cache zfs volumes (not files/datasets)? > >=20 > > Yes it does. >=20 > I find this distinction between volumes/files/etc and what is cached > causes confusion (as well as "volumes not datasets"). >=20 > Both ZVOLs and Z file systems are types of dataset. A dataset stores data > in records (usually up to 128kb in size). It's these records that are > cached (and that most ZFS functions such as compression/raidz/zil/etc > work with) >=20 > As far as the ZFS lower levels are concerned, there is no difference > between a volume and a file system. Thank you Matt, this was very instructive. > > > 2. If ARC does cache volumes, does this cache make sense on a=20 > > > hypervisor, because guest OSes will probably have their own disk cach= e anyway. > >=20 > > IMHO not much, because the guest OS is relying on the fact that when=20 > > it writes it=E2=80=99s own cached data out to =E2=80=9Edisk=E2=80=9C, i= t will be committed to=20 > > stable storage. >=20 > Maybe I've missed something but I don't quite get the link between > read cache (ARC) and guest writes here? Maybe there was a confusion between read and write caches, but my question still stands: Does it make sense to cache the same data (for reading too) twice: one time in host's RAM (ZFS ARC) and the other time in guest's RAM (whatever fs the guest uses, all modern OSes have disk caches)? What do VMWare or VirtualBox do for this situation? Do they ever cache their volumes in the hypervisor's RAM? --=20 Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/ --bg08WKrSYDhXBjb5 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEcBAEBAgAGBQJckvknAAoJEA2k8lmbXsY05lYIAJQUH3L5cyzR/TEuzoHE1esh LqSJqNDKDAJllrlKZ9NNPe5g1ylLiEEtc0WwrvesrWecIX20cJqJaamz2uEVk/iC EUWaCKkOEN8LvOoVMMY8GvJ5r0DezdfDyKp01tSNTZY/HlsF63u5UNyNlHXDGYeU pSCZhl+8qWWoZxk51G3znGU03QKPD6R9BFBtp6Yukp/cGb3E95jOb8wmX2z/3CJQ bsY2SUlcGdeRnM+bifDBQaH9oCktcYWqpejEEUNLKx5Gmvz1KuG0X1yrH5mtqanY 3EFm9EPsqvn/QWnAxLqEwBmdoZokLtYKXTUPV16LRLrL8Jy19gklACBNEevc+Yo= =9k/W -----END PGP SIGNATURE----- --bg08WKrSYDhXBjb5-- From owner-freebsd-virtualization@freebsd.org Thu Mar 21 10:24:23 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2EC3C155B48C for ; Thu, 21 Mar 2019 10:24:23 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from smtp-a.userve.net (smtp-outbound.userve.net [217.196.1.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.userve.net", Issuer "Thawte RSA CA 2018" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BE15A84195 for ; Thu, 21 Mar 2019 10:24:22 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from owa.usd-group.com (owa.usd-group.com [217.196.1.2]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp-a.userve.net (Postfix) with ESMTPS id A006E6C4; Thu, 21 Mar 2019 10:24:14 +0000 (GMT) Received: from SERVER.ad.usd-group.com (192.168.0.1) by SERVER.ad.usd-group.com (192.168.0.1) with Microsoft SMTP Server (TLS) id 15.0.847.32; Thu, 21 Mar 2019 10:24:14 +0000 Received: from SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9]) by SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9%12]) with mapi id 15.00.0847.030; Thu, 21 Mar 2019 10:24:14 +0000 From: Matt Churchyard To: Victor Sudakov , "freebsd-virtualization@freebsd.org" Subject: RE: bhyve and vfs.zfs.arc_max, and zfs tuning for a hypervisor Thread-Topic: bhyve and vfs.zfs.arc_max, and zfs tuning for a hypervisor Thread-Index: AQHU3f4eR9n6J576SkWf85jIrU+hAKYSmT6AgAEp9ACAAHlQEIABJdmAgAB9PfA= Date: Thu, 21 Mar 2019 10:24:13 +0000 Message-ID: <99457cf7758b4f7ebacafce5f40af220@SERVER.ad.usd-group.com> References: <20190319024638.GA8193@admin.sibptus.ru> <20190320015236.GA35511@admin.sibptus.ru> <6388ebd5a12949179ff6da33eb107205@SERVER.ad.usd-group.com> <20190321023831.GA57807@admin.sibptus.ru> In-Reply-To: <20190321023831.GA57807@admin.sibptus.ru> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.0.10] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-Rspamd-Queue-Id: BE15A84195 X-Spamd-Bar: ------ X-Spamd-Result: default: False [-6.98 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.98)[-0.975,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Mar 2019 10:24:23 -0000 PiA+IA0KPiA+ID4gMS4gRG9lcyBBUkMgYWN0dWFsbHkgY2FjaGUgemZzIHZvbHVtZXMgKG5vdCBm aWxlcy9kYXRhc2V0cyk/DQo+ID4gDQo+ID4gWWVzIGl0IGRvZXMuDQo+IA0KPiBJIGZpbmQgdGhp cyBkaXN0aW5jdGlvbiBiZXR3ZWVuIHZvbHVtZXMvZmlsZXMvZXRjIGFuZCB3aGF0IGlzIGNhY2hl ZCANCj4gY2F1c2VzIGNvbmZ1c2lvbiAoYXMgd2VsbCBhcyAidm9sdW1lcyBub3QgZGF0YXNldHMi KS4NCj4gDQo+IEJvdGggWlZPTHMgYW5kIFogZmlsZSBzeXN0ZW1zIGFyZSB0eXBlcyBvZiBkYXRh c2V0LiBBIGRhdGFzZXQgc3RvcmVzIA0KPiBkYXRhIGluIHJlY29yZHMgKHVzdWFsbHkgdXAgdG8g MTI4a2IgaW4gc2l6ZSkuICBJdCdzIHRoZXNlIHJlY29yZHMgDQo+IHRoYXQgYXJlIGNhY2hlZCAo YW5kIHRoYXQgbW9zdCBaRlMgZnVuY3Rpb25zIHN1Y2ggYXMgDQo+IGNvbXByZXNzaW9uL3JhaWR6 L3ppbC9ldGMgd29yayB3aXRoKQ0KPiANCj4gQXMgZmFyIGFzIHRoZSBaRlMgbG93ZXIgbGV2ZWxz IGFyZSBjb25jZXJuZWQsIHRoZXJlIGlzIG5vIGRpZmZlcmVuY2UgDQo+IGJldHdlZW4gYSB2b2x1 bWUgYW5kIGEgZmlsZSBzeXN0ZW0uDQoNCj5UaGFuayB5b3UgTWF0dCwgdGhpcyB3YXMgdmVyeSBp bnN0cnVjdGl2ZS4NCg0KPiA+ID4gMi4gSWYgQVJDIGRvZXMgY2FjaGUgdm9sdW1lcywgZG9lcyB0 aGlzIGNhY2hlIG1ha2Ugc2Vuc2Ugb24gYSANCj4gPiA+IGh5cGVydmlzb3IsIGJlY2F1c2UgZ3Vl c3QgT1NlcyB3aWxsIHByb2JhYmx5IGhhdmUgdGhlaXIgb3duIGRpc2sgY2FjaGUgYW55d2F5Lg0K PiA+IA0KPiA+IElNSE8gbm90IG11Y2gsIGJlY2F1c2UgdGhlIGd1ZXN0IE9TIGlzIHJlbHlpbmcg b24gdGhlIGZhY3QgdGhhdCB3aGVuIA0KPiA+IGl0IHdyaXRlcyBpdOKAmXMgb3duIGNhY2hlZCBk YXRhIG91dCB0byDigJ5kaXNr4oCcLCBpdCB3aWxsIGJlIGNvbW1pdHRlZCANCj4gPiB0byBzdGFi bGUgc3RvcmFnZS4NCj4gDQo+IE1heWJlIEkndmUgbWlzc2VkIHNvbWV0aGluZyBidXQgSSBkb24n dCBxdWl0ZSBnZXQgdGhlIGxpbmsgYmV0d2VlbiANCj4gcmVhZCBjYWNoZSAoQVJDKSBhbmQgZ3Vl c3Qgd3JpdGVzIGhlcmU/DQoNCj5NYXliZSB0aGVyZSB3YXMgYSBjb25mdXNpb24gYmV0d2VlbiBy ZWFkIGFuZCB3cml0ZSBjYWNoZXMsIGJ1dCBteSBxdWVzdGlvbiBzdGlsbCBzdGFuZHM6DQoNCj5E b2VzIGl0IG1ha2Ugc2Vuc2UgdG8gY2FjaGUgdGhlIHNhbWUgZGF0YSAoZm9yIHJlYWRpbmcgdG9v KSB0d2ljZTogb25lIHRpbWUgaW4gaG9zdCdzIFJBTSAoWkZTIEFSQykgYW5kIHRoZSBvdGhlciB0 aW1lIGluIGd1ZXN0J3MgUkFNICh3aGF0ZXZlciBmcyB0aGUgZ3Vlc3QgdXNlcywgYWxsIG1vZGVy biBPU2VzIGhhdmUgZGlzayBjYWNoZXMpPw0KDQo+V2hhdCBkbyBWTVdhcmUgb3IgVmlydHVhbEJv eCBkbyBmb3IgdGhpcyBzaXR1YXRpb24/IERvIHRoZXkgZXZlciBjYWNoZSB0aGVpciB2b2x1bWVz IGluIHRoZSBoeXBlcnZpc29yJ3MgUkFNPw0KDQpWaXJ0dWFsYm94IHdvdWxkIGJlIG5vIGRpZmZl cmVudCB0byBiaHl2ZSBpbiB0aGF0IGl0IGRvZXNuJ3QgY2FyZSB3aGF0IHN0b3JhZ2Ugc3lzdGVt IHlvdSBhcmUgdXNpbmcgb3IgaG93IGl0IGlzIGNvbmZpZ3VyZWQsIHRoYXQncyB1cCB0byB0aGUg c3lzdGVtIGFkbWluLg0KSSBiZWxpZXZlIFZNRlMgaXMgbW9yZSBha2luIHRvIG90aGVyICJ0cmFk aXRpb25hbCIgZmlsZSBzeXN0ZW1zLCBhbmQgZG9lc24ndCBkbyBSQU0gY2FjaGluZyB0byBhbnl3 aGVyZSBuZWFyIHRoZSBleHRlbnQgb2YgWkZTLiBJIGRvIHRoaW5rIHlvdSBjYW4gdXNlIFNTRC9O Vk1lL2V0YyBhcyBjYWNoZSBpbiBWTVdhcmUuDQoNCk15IGluaXRpYWwgaW5zdGluY3Qgd291bGQg YmUgdG8ga2VlcCBjYWNoZSBvbiwgYnV0IHJlZHVjZSB0aGUgbGltaXQgdG8gYWxsb2NhdGUgdGhl IG1ham9yaXR5IG9mIHRoZSBSQU0gZm9yIGd1ZXN0cy4gKEknZCBzdGlsbCB3YW50IGF0IGxlYXN0 IDRHQiBhcyBhbiBhYnNvbHV0ZSBtaW5pbXVtIHRob3VnaCwgcHJvYmFibHkgbW9yZSBvbiBzeXN0 ZW1zIHdpdGggMTAwR0IrIHRvdGFsKS4gT2YgY291cnNlIHlvdSBjb3VsZCBwcm9iYWJseSB0ZXN0 IHdpdGggY2FjaGUgc2V0IHRvIGFsbC9tZXRhZGF0YSBhbmQgc2VlIHdoYXQgZWZmZWN0IGl0IGhh cy4gQWRkaW5nIEwyQVJDIG1heSBiZSB1c2VmdWwgaWYgdGhlIG1haW4gcG9vbCBpcyBzcGlubmlu ZyBkaXNrcywgYnV0IHRoZW4gSSd2ZSBoZWFyZCB0aGVyZSdzIGEgcnVsZSBvZiB0aHVtYiBmb3Ig cmVxdWlyaW5nIFggYW1vdW50IG9mIEFSQyBmb3IgWSBhbW91bnQgb2YgTDJBUkMsIGJ1dCBJJ20g bm90IHN1cmUgd2hhdCB0aGF0IHJ1bGUgaXMuDQoNCkknZCBhbHNvIGJlIGludHJpZ3VlZCB0byBr bm93IHdoYXQgdGhlIGxvZ2ljIGluIEZyZWVOQVMgaXMgZm9yIGl0LiBJdCBpcyBzaW1wbHkgYSBj YXNlIG9mICIoYXJjID0gdG90YWxfcmFtIC0gZ3Vlc3RfYWxsb2NhdGVkKSI/DQpJcyB0aGVyZSBh IGxvd2VyIGxpbWl0IGJhc2VkIG9uIGEgcGVyY2VudGFnZSBvciB0b3RhbCBSQU0sIGFuZC9vciBh IGhhcmQgbG93ZXIgbGltaXQ/DQoNCk1hdHQNCg0K From owner-freebsd-virtualization@freebsd.org Thu Mar 21 13:10:34 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 984F2153FA69 for ; Thu, 21 Mar 2019 13:10:34 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2F6328A806 for ; Thu, 21 Mar 2019 13:10:27 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from hugo10.ka.punkt.de (hugo10.ka.punkt.de [217.29.44.10]) by gate2.intern.punkt.de with ESMTP id x2LDAIIn053336; Thu, 21 Mar 2019 14:10:18 +0100 (CET) Received: from [217.29.44.158] ([217.29.44.158]) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id x2LDAHrF089852; Thu, 21 Mar 2019 14:10:18 +0100 (CET) (envelope-from hausen@punkt.de) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.2 \(3445.102.3\)) Subject: Re: bhyve and vfs.zfs.arc_max, and zfs tuning for a hypervisor From: "Patrick M. Hausen" In-Reply-To: <99457cf7758b4f7ebacafce5f40af220@SERVER.ad.usd-group.com> Date: Thu, 21 Mar 2019 14:10:17 +0100 Cc: Victor Sudakov , "freebsd-virtualization@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: <758301FE-77BC-4D98-AF13-CD2A7E2D276A@punkt.de> References: <20190319024638.GA8193@admin.sibptus.ru> <20190320015236.GA35511@admin.sibptus.ru> <6388ebd5a12949179ff6da33eb107205@SERVER.ad.usd-group.com> <20190321023831.GA57807@admin.sibptus.ru> <99457cf7758b4f7ebacafce5f40af220@SERVER.ad.usd-group.com> To: Matt Churchyard X-Mailer: Apple Mail (2.3445.102.3) X-Rspamd-Queue-Id: 2F6328A806 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of hausen@punkt.de designates 217.29.33.131 as permitted sender) smtp.mailfrom=hausen@punkt.de X-Spamd-Result: default: False [-2.93 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.994,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:217.29.32.0/20]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[punkt.de]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[mailin.pluspunkthosting.de,mailin.pluspunkthosting.de]; NEURAL_HAM_SHORT(-0.76)[-0.761,0]; RCVD_IN_DNSWL_NONE(0.00)[131.33.29.217.list.dnswl.org : 127.0.10.0]; IP_SCORE(-0.37)[ipnet: 217.29.32.0/20(-1.02), asn: 16188(-0.82), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16188, ipnet:217.29.32.0/20, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Mar 2019 13:10:34 -0000 Hi all, > Am 21.03.2019 um 11:24 schrieb Matt Churchyard via = freebsd-virtualization : > I'd also be intrigued to know what the logic in FreeNAS is for it. It = is simply a case of "(arc =3D total_ram - guest_allocated)"? > Is there a lower limit based on a percentage or total RAM, and/or a = hard lower limit? The relevant code can be found here: = https://github.com/freenas/freenas/blob/master/src/middlewared/middlewared= /plugins/vm.py Kind regards, Patrick --=20 punkt.de GmbH Internet - Dienstleistungen - Beratung Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100 76133 Karlsruhe info@punkt.de http://punkt.de AG Mannheim 108285 Gf: Juergen Egeling From owner-freebsd-virtualization@freebsd.org Fri Mar 22 00:44:55 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B8C531554615 for ; Fri, 22 Mar 2019 00:44:55 +0000 (UTC) (envelope-from d.scott.phillips@intel.com) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "orsmga106.jf.intel.com", Issuer "COMODO RSA Organization Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B515687B7C for ; Fri, 22 Mar 2019 00:44:53 +0000 (UTC) (envelope-from d.scott.phillips@intel.com) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga106.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Mar 2019 17:44:46 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,254,1549958400"; d="scan'208";a="124803722" Received: from dsp-dsk1.jf.intel.com (HELO localhost) ([10.54.70.10]) by orsmga007.jf.intel.com with ESMTP; 21 Mar 2019 17:44:46 -0700 From: D Scott Phillips To: freebsd-virtualization@freebsd.org Subject: Updating uefi-edk2-bhyve Date: Thu, 21 Mar 2019 17:44:46 -0700 Message-ID: <86muln68ld.fsf@intel.com> MIME-Version: 1.0 Content-Type: text/plain X-Rspamd-Queue-Id: B515687B7C X-Spamd-Bar: -------- Authentication-Results: mx1.freebsd.org; dmarc=pass (policy=none) header.from=intel.com; spf=pass (mx1.freebsd.org: domain of d.scott.phillips@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=d.scott.phillips@intel.com X-Spamd-Result: default: False [-8.79 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:134.134.136.126/32]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[126.136.134.134.list.dnswl.org : 127.0.9.2]; MX_GOOD(-0.01)[mga04.intel.com,mga03.intel.com,mga14.intel.com,mga09.intel.com,mga12.intel.com,mga07.intel.com,mga06.intel.com,mga17.intel.com,mga01.intel.com,mga02.intel.com,mga11.intel.com,mga18.intel.com,mga05.intel.com]; DMARC_POLICY_ALLOW(-0.50)[intel.com,none]; NEURAL_HAM_SHORT(-0.98)[-0.984,0]; IP_SCORE(-1.79)[ipnet: 134.134.136.0/24(-4.94), asn: 4983(-3.96), country: US(-0.07)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:4983, ipnet:134.134.136.0/24, country:US]; MID_RHS_MATCH_FROM(0.00)[]; WHITELIST_SPF_DKIM(-3.00)[intel.com:s:+] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Mar 2019 00:44:55 -0000 Hi freebsd-virtualization, Recently I wanted to be able to do UEFI HTTP Boot in bhyve, so I've rebased the bhyve firmware up to the latest upstream tag, edk2-stable201903. You can find the firmware here: https://gitlab.com/scott-ph/edk2/tree/wip/2019-03/v2-bhyve-rebase-edk2-stable201903 and a ports patch to help with testing here: https://gitlab.com/scott-ph/freebsd-ports/tree/wip/2019-03/v2-uefi-edk2-bhyve I've successfully run FreeBSD, Linux, and Windows with this firmware, and of course HTTP Booting is working. If you're interested you can give it a try, and I'd be glad to hear any reports of bugs you find with the new firmware. Hopefully after some testing we can stabilize this and push it as an update to the port. Thanks, Scott From owner-freebsd-virtualization@freebsd.org Fri Mar 22 01:38:01 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BE22C1555A6B for ; Fri, 22 Mar 2019 01:38:01 +0000 (UTC) (envelope-from rebecca@bluestop.org) Received: from muon.bluestop.org (muon.bluestop.org [65.103.231.193]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 14F6C896B2 for ; Fri, 22 Mar 2019 01:37:59 +0000 (UTC) (envelope-from rebecca@bluestop.org) Received: from muon.bluestop.org (localhost [127.0.0.1]) by muon.bluestop.org (Postfix) with ESMTP id 2DDFD7F395; Thu, 21 Mar 2019 19:39:02 -0600 (MDT) Received: from muon.bluestop.org ([127.0.0.1]) by muon.bluestop.org (muon.bluestop.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id RXoVHMPKhSWS; Thu, 21 Mar 2019 19:39:01 -0600 (MDT) Received: from [10.0.10.197] (unknown [65.103.231.197]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by muon.bluestop.org (Postfix) with ESMTPSA; Thu, 21 Mar 2019 19:39:01 -0600 (MDT) Date: Thu, 21 Mar 2019 19:34:51 -0600 From: rebecca@bluestop.org To: freebsd-virtualization@freebsd.org, D Scott Phillips Message-ID: In-Reply-To: <86muln68ld.fsf@intel.com> References: <86muln68ld.fsf@intel.com> Subject: Re: Updating uefi-edk2-bhyve X-Readdle-Message-ID: ccedded1-94ae-464b-98f2-ce932a80a0c7@Spark MIME-Version: 1.0 X-Rspamd-Queue-Id: 14F6C896B2 X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.25 / 15.00]; ARC_NA(0.00)[]; R_SPF_FAIL(0.00)[-all]; R_DKIM_ALLOW(-0.20)[bluestop.org:s=mail]; RCVD_VIA_SMTP_AUTH(0.00)[]; IP_SCORE(-0.99)[ipnet: 65.100.0.0/14(-4.87), asn: 209(0.01), country: US(-0.07)]; TO_DN_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; URI_COUNT_ODD(1.00)[3]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bluestop.org:+]; RCPT_COUNT_TWO(0.00)[2]; FROM_NO_DN(0.00)[]; MX_GOOD(-0.01)[mail.bluestop.org]; DMARC_POLICY_ALLOW(0.00)[bluestop.org,quarantine]; DMARC_POLICY_ALLOW_WITH_FAILURES(-0.50)[]; NEURAL_HAM_SHORT(-0.95)[-0.953,0]; FROM_EQ_ENVFROM(0.00)[]; MID_RHS_NOT_FQDN(0.50)[]; MIME_TRACE(0.00)[0:+,1:+]; ASN(0.00)[asn:209, ipnet:65.100.0.0/14, country:US]; RCVD_TLS_LAST(0.00)[] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Mar 2019 01:38:02 -0000 This is awesome=21 Updating to a newer edk2 has been on my todo list for = a while. However I don=E2=80=99t see a BhyvePkg in the first link you posted. And = could you confirm that you got HTTP boot working on =46reeBSD as well as = Linux and Windows=3F I thought loader changes were needed for that to wor= k, to enable the kernel to be fetched via http. =E2=80=94 Rebecca Cran On Mar 21, 2019, 6:46 PM -0600, D Scott Phillips , wrote: > Hi freebsd-virtualization, > > Recently I wanted to be able to do UE=46I HTTP Boot in bhyve, so I've > rebased the bhyve firmware up to the latest upstream tag, > edk2-stable201903. You can find the firmware here: > > https://gitlab.com/scott-ph/edk2/tree/wip/2019-03/v2-bhyve-rebase-edk2-= stable201903 > > and a ports patch to help with testing here: > > https://gitlab.com/scott-ph/freebsd-ports/tree/wip/2019-03/v2-uefi-edk2= -bhyve > > I've successfully run =46reeBSD, Linux, and Windows with this firmware,= > and of course HTTP Booting is working. If you're interested you can giv= e > it a try, and I'd be glad to hear any reports of bugs you find with the= > new firmware. Hopefully after some testing we can stabilize this and > push it as an update to the port. Thanks, > > Scott > =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F > freebsd-virtualization=40freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to =22freebsd-virtualization-unsubscribe=40= freebsd.org=22 From owner-freebsd-virtualization@freebsd.org Fri Mar 22 01:44:39 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 801941555E4A for ; Fri, 22 Mar 2019 01:44:39 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: from mail-lf1-x131.google.com (mail-lf1-x131.google.com [IPv6:2a00:1450:4864:20::131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 493ED89EF7 for ; Fri, 22 Mar 2019 01:44:38 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: by mail-lf1-x131.google.com with SMTP id r25so242669lfn.13 for ; Thu, 21 Mar 2019 18:44:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:reply-to:from:date:message-id :subject:to:cc; bh=HMSiyZP1kBF8wcRwPE88oo9JzJ1QmvU/XESrE3vsrRI=; b=Vd6L6c63nbymofu1k5ehRkbnSUA55hgpE0kzxtPgpx0VqH7BU58ee5BW0mWh0r4TRH wMRhGthAfJLAJ/JZBaXOXjxCNngKdUvl1r2bI+Odj8JAqyLOGHCDB34+kvLOLnWEI5mO y+56MalZOZeoraJWoqIfJMQFeRyEhEPSYAAsWBhe+eBdxSEiSAjvtS+YFbYufgM+4moS qVZlFnxFEzAROPveqMIHGIh4Cd1fqHshHtLJuSTokJ+n7ScO/p2rrglCAPI8too4SMBO 8trWE+1psAw250UgSMCgPTFgPUZSmKifwNoB+Lxy5YPacgeCDgpPROTxkNuVfAFMFrDm p79g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:reply-to :from:date:message-id:subject:to:cc; bh=HMSiyZP1kBF8wcRwPE88oo9JzJ1QmvU/XESrE3vsrRI=; b=Ew7PmBVxVbg07KKOsI6qrJtMg7EsFPFDD/2DnKISd/DzpU82XMaRScLqDc9mPBLldX g7VcfcaNLQpj5I7URTIapeUe6X2fnxxgFVQmnC2NkiX4dOGy7ZT21iWZxSUHKZ/wM+Q+ IdnuXu2S0ieYFSJSWe6LJgNHhenhK4K40V5wsFeThQv4Ppbk6g60Of7/ebxmi5g2h9dC ltH/sEWRpKCUdU/ZZ/mO6ITnurIl39ydS172PrJVekxLm9EEz8Jn8wyRIqFtsu5xwSan f4qhHFDpbF1fKx1t5L7RaDrkEDEFVmCgBy2LW1QS8I79bjzTFWIu4lR2amKBxsenVUtF sPDQ== X-Gm-Message-State: APjAAAURz9uYFcoy8APabKCc4Cgv3R0SbAhbxtVwwBLvdvk9BoBJhV3K 3y589chEaip3x3EeEwbUxyxytSwQVStyciTnxj8= X-Google-Smtp-Source: APXvYqzdFOEhBJoLrPr+dQ+bpFS+r4WxTLc2T+Gzo5Rdhh3Tm6rWETJoc7IfyaF9eTDdQyyyHNG14SF5qi76mlcd4tY= X-Received: by 2002:a19:ed19:: with SMTP id y25mr3858328lfy.80.1553219076835; Thu, 21 Mar 2019 18:44:36 -0700 (PDT) MIME-Version: 1.0 References: <86muln68ld.fsf@intel.com> In-Reply-To: <86muln68ld.fsf@intel.com> Reply-To: araujo@freebsd.org From: Marcelo Araujo Date: Fri, 22 Mar 2019 09:44:25 +0800 Message-ID: Subject: Re: Updating uefi-edk2-bhyve To: D Scott Phillips Cc: "freebsd-virtualization@freebsd.org" X-Rspamd-Queue-Id: 493ED89EF7 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=Vd6L6c63; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of araujobsdport@gmail.com designates 2a00:1450:4864:20::131 as permitted sender) smtp.mailfrom=araujobsdport@gmail.com X-Spamd-Result: default: False [-5.74 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; HAS_REPLYTO(0.00)[araujo@freebsd.org]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; URI_COUNT_ODD(1.00)[11]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.99)[-0.992,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; REPLYTO_DOM_NEQ_FROM_DOM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[1.3.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.73)[ip: (-9.10), ipnet: 2a00:1450::/32(-2.38), asn: 15169(-2.13), country: US(-0.07)]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Mar 2019 01:44:39 -0000 Em sex, 22 de mar de 2019 =C3=A0s 08:45, D Scott Phillips < d.scott.phillips@intel.com> escreveu: > Hi freebsd-virtualization, > > Recently I wanted to be able to do UEFI HTTP Boot in bhyve, so I've > rebased the bhyve firmware up to the latest upstream tag, > edk2-stable201903. You can find the firmware here: > > > https://gitlab.com/scott-ph/edk2/tree/wip/2019-03/v2-bhyve-rebase-edk2-st= able201903 > > and a ports patch to help with testing here: > > > https://gitlab.com/scott-ph/freebsd-ports/tree/wip/2019-03/v2-uefi-edk2-b= hyve > > I've successfully run FreeBSD, Linux, and Windows with this firmware, > and of course HTTP Booting is working. If you're interested you can give > it a try, and I'd be glad to hear any reports of bugs you find with the > new firmware. Hopefully after some testing we can stabilize this and > push it as an update to the port. Thanks, > > Scott > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscribe@freebsd.org" > This is a fantastic news!!! I think after more tests done by the community, you should make a PR to this repo: https://github.com/freebsd/uefi-edk2 and then I can update the ports. Well done job, thanks! --=20 --=20 Marcelo Araujo (__)araujo@FreeBSD.org \\\'',)http://www.FreeBSD.org \/ \ ^ Power To Server. .\. /_) From owner-freebsd-virtualization@freebsd.org Fri Mar 22 03:19:46 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 625591557F2E for ; Fri, 22 Mar 2019 03:19:46 +0000 (UTC) (envelope-from rebecca@bluestop.org) Received: from muon.bluestop.org (muon.bluestop.org [65.103.231.193]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 37E0D8D804 for ; Fri, 22 Mar 2019 03:19:45 +0000 (UTC) (envelope-from rebecca@bluestop.org) Received: from muon.bluestop.org (localhost [127.0.0.1]) by muon.bluestop.org (Postfix) with ESMTP id 477BF7F4AF; Thu, 21 Mar 2019 21:20:53 -0600 (MDT) Received: from muon.bluestop.org ([127.0.0.1]) by muon.bluestop.org (muon.bluestop.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 9MflVRjab_cS; Thu, 21 Mar 2019 21:20:52 -0600 (MDT) Received: from photon.int.bluestop.org (unknown [65.103.231.197]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by muon.bluestop.org (Postfix) with ESMTPSA; Thu, 21 Mar 2019 21:20:52 -0600 (MDT) Subject: Re: Updating uefi-edk2-bhyve To: D Scott Phillips , freebsd-virtualization@freebsd.org References: <86muln68ld.fsf@intel.com> From: Rebecca Cran Message-ID: <1fe3ca3f-be70-99db-e7c0-35c9194c97e4@bluestop.org> Date: Thu, 21 Mar 2019 21:19:42 -0600 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.5.3 MIME-Version: 1.0 In-Reply-To: <86muln68ld.fsf@intel.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Rspamd-Queue-Id: 37E0D8D804 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.93 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[bluestop.org:s=mail]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(-0.99)[ipnet: 65.100.0.0/14(-4.87), asn: 209(0.00), country: US(-0.07)]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: mail.bluestop.org]; DKIM_TRACE(0.00)[bluestop.org:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.93)[-0.934,0]; DMARC_POLICY_ALLOW(-0.50)[bluestop.org,quarantine]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:209, ipnet:65.100.0.0/14, country:US]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Mar 2019 03:19:46 -0000 On 3/21/19 6:44 PM, D Scott Phillips wrote: > Hi freebsd-virtualization, > > Recently I wanted to be able to do UEFI HTTP Boot in bhyve, so I've > rebased the bhyve firmware up to the latest upstream tag, > edk2-stable201903. You can find the firmware here: > > https://gitlab.com/scott-ph/edk2/tree/wip/2019-03/v2-bhyve-rebase-edk2-stable201903 Thanks! I realized you've removed BhyvePkg and ported OvmfPkg to run under Bhyve, which is perfect. I've successfully built OvmfPkgX64 with your changes and can boot Bhyve to the UEFI Shell. The RELEASE build with the GCC5 toolset (but using an installation of gcc 4.8) works great for me, though I ran into problems with the DEBUG build and the GCC48 toolset. -- Rebecca Cran From owner-freebsd-virtualization@freebsd.org Fri Mar 22 03:34:39 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5CF62155861F; Fri, 22 Mar 2019 03:34:39 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (unknown [IPv6:2001:470:1f0f:3ad::53:2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "*.lerctr.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2F6868E151; Fri, 22 Mar 2019 03:34:37 +0000 (UTC) (envelope-from ler@lerctr.org) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:References:In-Reply-To:Subject:Cc:To:From:Date: Content-Transfer-Encoding:Content-Type:MIME-Version:Sender:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=TJDBcLeLh0weWT64mU1791/LfMSypBfSk3z93BBnU1c=; b=LszWD306aEoG8jR0t/yKjDsSAw Y/UO/8rIjzZI/sctEAVfyFsmqYBe3eS2ajOGHZXrxY+Krqwz/qBF4sstYZjSy2egJIWr53dGmwn9p IVo0Wp8zHdRccjN1oURvrsXoVL6uw6ZiBz3q9Aw2MCRWDdA5iGvswz1vsSbbKGWuw2TA=; Received: from thebighonker.lerctr.org ([2001:470:1f0f:3ad:bb:dcff:fe50:d900]:11805 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpsa (TLSv1.3:TLS_AES_256_GCM_SHA384:256) (Exim 4.92 (FreeBSD)) (envelope-from ) id 1h7Aww-000DSd-2q; Thu, 21 Mar 2019 22:34:30 -0500 Received: from 2600:1700:210:b18f:e89d:e7f9:22a:2fc4 by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Thu, 21 Mar 2019 22:34:29 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Thu, 21 Mar 2019 22:34:29 -0500 From: Larry Rosenman To: Rebecca Cran Cc: D Scott Phillips , freebsd-virtualization@freebsd.org, owner-freebsd-virtualization@freebsd.org Subject: Re: Updating uefi-edk2-bhyve In-Reply-To: <1fe3ca3f-be70-99db-e7c0-35c9194c97e4@bluestop.org> References: <86muln68ld.fsf@intel.com> <1fe3ca3f-be70-99db-e7c0-35c9194c97e4@bluestop.org> Message-ID: <7e84fd01c3f46268c26f9bab8b9fb9bc@lerctr.org> X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/1.3.8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Mar 2019 03:34:39 -0000 On 03/21/2019 10:19 pm, Rebecca Cran via freebsd-virtualization wrote: > On 3/21/19 6:44 PM, D Scott Phillips wrote: >> Hi freebsd-virtualization, >> >> Recently I wanted to be able to do UEFI HTTP Boot in bhyve, so I've >> rebased the bhyve firmware up to the latest upstream tag, >> edk2-stable201903. You can find the firmware here: >> >> https://gitlab.com/scott-ph/edk2/tree/wip/2019-03/v2-bhyve-rebase-edk2-stable201903 > > > Thanks! I realized you've removed BhyvePkg and ported OvmfPkg to run > under Bhyve, which is perfect. I've successfully built OvmfPkgX64 with > your changes and can boot Bhyve to the UEFI Shell. > > The RELEASE build with the GCC5 toolset (but using an installation of > gcc 4.8) works great for me, though I ran into problems with the DEBUG > build and the GCC48 toolset. out of curiosity, is there any work being done to move this to later LLVM/CLANG? -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: ler@lerctr.org US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106 From owner-freebsd-virtualization@freebsd.org Fri Mar 22 05:30:16 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1F696155AC86; Fri, 22 Mar 2019 05:30:16 +0000 (UTC) (envelope-from rebecca@bluestop.org) Received: from muon.bluestop.org (muon.bluestop.org [65.103.231.193]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A5F006AB08; Fri, 22 Mar 2019 05:30:15 +0000 (UTC) (envelope-from rebecca@bluestop.org) Received: from muon.bluestop.org (localhost [127.0.0.1]) by muon.bluestop.org (Postfix) with ESMTP id 7854D7F595; Thu, 21 Mar 2019 23:31:23 -0600 (MDT) Received: from muon.bluestop.org ([127.0.0.1]) by muon.bluestop.org (muon.bluestop.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id E3DS4sAfhr3J; Thu, 21 Mar 2019 23:31:22 -0600 (MDT) Received: from [10.0.10.197] (unknown [65.103.231.197]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by muon.bluestop.org (Postfix) with ESMTPSA; Thu, 21 Mar 2019 23:31:22 -0600 (MDT) Date: Thu, 21 Mar 2019 23:26:44 -0600 From: rebecca@bluestop.org To: Larry Rosenman Cc: D Scott Phillips , freebsd-virtualization@freebsd.org, owner-freebsd-virtualization@freebsd.org Message-ID: <33fcf111-fe00-4ec7-8a2f-7c53246d756f@Spark> In-Reply-To: <7e84fd01c3f46268c26f9bab8b9fb9bc@lerctr.org> References: <86muln68ld.fsf@intel.com> <1fe3ca3f-be70-99db-e7c0-35c9194c97e4@bluestop.org> <7e84fd01c3f46268c26f9bab8b9fb9bc@lerctr.org> Subject: Re: Updating uefi-edk2-bhyve X-Readdle-Message-ID: 33fcf111-fe00-4ec7-8a2f-7c53246d756f@Spark MIME-Version: 1.0 X-Rspamd-Queue-Id: A5F006AB08 X-Spamd-Bar: ------ X-Spamd-Result: default: False [-6.94 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.94)[-0.942,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Mar 2019 05:30:16 -0000 On Mar 21, 2019, 9:54 PM -0600, Larry Rosenman , wrote:= > out of curiosity, is there any work being done to move this to later > LLVM/CLANG=3F That=E2=80=99s something else I=E2=80=99m hoping to work on, since gcc 4.= 8 is very outdated now. It seems to build fine using gcc 7, but I need to= test that it still runs fine. And that=E2=80=99s complicated by the fact= that the DEBUG build generates a triple fault even with gcc 4.8. =E2=80=94 Rebecca Cran From owner-freebsd-virtualization@freebsd.org Fri Mar 22 12:19:36 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7B01D1544427 for ; Fri, 22 Mar 2019 12:19:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 0E127807BC for ; Fri, 22 Mar 2019 12:19:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id C06301544426; Fri, 22 Mar 2019 12:19:35 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AEBB61544425 for ; Fri, 22 Mar 2019 12:19:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4A7EC807BA for ; Fri, 22 Mar 2019 12:19:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 713911249B for ; Fri, 22 Mar 2019 12:19:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id x2MCJYYp063190 for ; Fri, 22 Mar 2019 12:19:34 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id x2MCJYJZ063181 for virtualization@FreeBSD.org; Fri, 22 Mar 2019 12:19:34 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 231756] Vagrant boxes don't work with vagrant-bhyve by default Date: Fri, 22 Mar 2019 12:19:32 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: feature X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: 0mp@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: keywords cc bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Mar 2019 12:19:36 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D231756 Mateusz Piotrowski <0mp@FreeBSD.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |feature CC| |0mp@FreeBSD.org Status|New |Open --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Fri Mar 22 16:04:50 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 60FDE1549F76 for ; Fri, 22 Mar 2019 16:04:50 +0000 (UTC) (envelope-from d.scott.phillips@intel.com) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "orsmga106.jf.intel.com", Issuer "COMODO RSA Organization Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B3E9B890E1 for ; Fri, 22 Mar 2019 16:04:48 +0000 (UTC) (envelope-from d.scott.phillips@intel.com) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga106.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Mar 2019 09:04:45 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,256,1549958400"; d="scan'208";a="154842341" Received: from dsp-dsk1.jf.intel.com (HELO localhost) ([10.54.70.10]) by fmsmga004.fm.intel.com with ESMTP; 22 Mar 2019 09:04:45 -0700 From: D Scott Phillips To: Rebecca Cran , freebsd-virtualization@freebsd.org Subject: Re: Updating uefi-edk2-bhyve In-Reply-To: <1fe3ca3f-be70-99db-e7c0-35c9194c97e4@bluestop.org> References: <86muln68ld.fsf@intel.com> <1fe3ca3f-be70-99db-e7c0-35c9194c97e4@bluestop.org> Date: Fri, 22 Mar 2019 09:04:45 -0700 Message-ID: <86r2ayao9u.fsf@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: B3E9B890E1 X-Spamd-Bar: -------- Authentication-Results: mx1.freebsd.org; dmarc=pass (policy=none) header.from=intel.com; spf=pass (mx1.freebsd.org: domain of d.scott.phillips@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=d.scott.phillips@intel.com X-Spamd-Result: default: False [-8.79 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:134.134.136.126/32]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[126.136.134.134.list.dnswl.org : 127.0.9.2]; MX_GOOD(-0.01)[cached: mga04.intel.com]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.99)[-0.989,0]; DMARC_POLICY_ALLOW(-0.50)[intel.com,none]; IP_SCORE(-1.79)[ipnet: 134.134.136.0/24(-4.94), asn: 4983(-3.96), country: US(-0.07)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:4983, ipnet:134.134.136.0/24, country:US]; MID_RHS_MATCH_FROM(0.00)[]; WHITELIST_SPF_DKIM(-3.00)[intel.com:s:+] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Mar 2019 16:04:50 -0000 Rebecca Cran writes: > This is awesome! Updating to a newer edk2 has been on my todo list for > a while. > > However I don=E2=80=99t see a BhyvePkg in the first link you posted. And = could > you confirm that you got HTTP boot working on FreeBSD as well as Linux > and Windows? I thought loader changes were needed for that to work, to > enable the kernel to be fetched via http. Sorry, I didn't write that very clearly, what I meant was that I tested local boot with those 3 os's, and then I also tested HTTP Boot and it was working well enough to fetch and start loader_lua.efi. So I haven't fully booted FreeBSD over HTTP, just got the loader running. Just using the loader's nfs support will probably be good enough for my use, so I don't plan on looking at http support for the loader at the moment. > On 3/21/19 6:44 PM, D Scott Phillips wrote: > > Hi freebsd-virtualization, > > > > Recently I wanted to be able to do UEFI HTTP Boot in bhyve, so I've > > rebased the bhyve firmware up to the latest upstream tag, > > edk2-stable201903. You can find the firmware here: > > > > https://gitlab.com/scott-ph/edk2/tree/wip/2019-03/v2-bhyve-rebase-edk2-= stable201903 > > > Thanks! I realized you've removed BhyvePkg and ported OvmfPkg to run > under Bhyve, which is perfect. I've successfully built OvmfPkgX64 with > your changes and can boot Bhyve to the UEFI Shell. Right, my understanding from looking at the history is that BhyvePkg started life as a copy of the OvmfPkg directory. So I rewrote the patches to be deltas against OvmfPkg so that rebasing would be tractable. In the end I chose to leave them that way to make the life of the next rebaser easier. From owner-freebsd-virtualization@freebsd.org Fri Mar 22 16:08:50 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DF753154A090 for ; Fri, 22 Mar 2019 16:08:50 +0000 (UTC) (envelope-from d.scott.phillips@intel.com) Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "fmsmga102.fm.intel.com", Issuer "COMODO RSA Organization Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 61E6489237; Fri, 22 Mar 2019 16:08:49 +0000 (UTC) (envelope-from d.scott.phillips@intel.com) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Mar 2019 09:07:37 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,256,1549958400"; d="scan'208";a="285005973" Received: from dsp-dsk1.jf.intel.com (HELO localhost) ([10.54.70.10]) by orsmga004.jf.intel.com with ESMTP; 22 Mar 2019 09:07:37 -0700 From: D Scott Phillips To: araujo@freebsd.org Cc: "freebsd-virtualization\@freebsd.org" Subject: Re: Updating uefi-edk2-bhyve In-Reply-To: References: <86muln68ld.fsf@intel.com> Date: Fri, 22 Mar 2019 09:07:37 -0700 Message-ID: <86mulmao52.fsf@intel.com> MIME-Version: 1.0 Content-Type: text/plain X-Rspamd-Queue-Id: 61E6489237 X-Spamd-Bar: --------- Authentication-Results: mx1.freebsd.org; dmarc=pass (policy=none) header.from=intel.com; spf=pass (mx1.freebsd.org: domain of d.scott.phillips@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=d.scott.phillips@intel.com X-Spamd-Result: default: False [-9.07 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:192.55.52.93/32]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; IP_SCORE(-1.80)[ipnet: 192.55.52.0/24(-4.96), asn: 4983(-3.96), country: US(-0.07)]; MX_GOOD(-0.01)[cached: mga04.intel.com]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[intel.com,none]; NEURAL_HAM_SHORT(-0.97)[-0.968,0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:4983, ipnet:192.55.52.0/24, country:US]; RCVD_IN_DNSWL_HI(-0.50)[93.52.55.192.list.dnswl.org : 127.0.9.3]; MID_RHS_MATCH_FROM(0.00)[]; WHITELIST_SPF_DKIM(-3.00)[intel.com:s:+] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Mar 2019 16:08:51 -0000 Marcelo Araujo writes: > This is a fantastic news!!! > > I think after more tests done by the community, you should make a PR > to this repo: https://github.com/freebsd/uefi-edk2 and then I can > update the ports. Yep, that sounds like a good plan to me. Is there anything else I can be doing to get a "call for testing" out there, beyond this mail thread? From owner-freebsd-virtualization@freebsd.org Fri Mar 22 16:12:28 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0C342154A34F; Fri, 22 Mar 2019 16:12:28 +0000 (UTC) (envelope-from d.scott.phillips@intel.com) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "fmsmga106.fm.intel.com", Issuer "COMODO RSA Organization Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B86A889620; Fri, 22 Mar 2019 16:12:26 +0000 (UTC) (envelope-from d.scott.phillips@intel.com) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Mar 2019 09:12:19 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,256,1549958400"; d="scan'208";a="154223641" Received: from dsp-dsk1.jf.intel.com (HELO localhost) ([10.54.70.10]) by fmsmga002.fm.intel.com with ESMTP; 22 Mar 2019 09:12:19 -0700 From: D Scott Phillips To: rebecca@bluestop.org, Larry Rosenman Cc: freebsd-virtualization@freebsd.org, owner-freebsd-virtualization@freebsd.org Subject: Re: Updating uefi-edk2-bhyve In-Reply-To: <33fcf111-fe00-4ec7-8a2f-7c53246d756f@Spark> References: <86muln68ld.fsf@intel.com> <1fe3ca3f-be70-99db-e7c0-35c9194c97e4@bluestop.org> <7e84fd01c3f46268c26f9bab8b9fb9bc@lerctr.org> <33fcf111-fe00-4ec7-8a2f-7c53246d756f@Spark> Date: Fri, 22 Mar 2019 09:12:19 -0700 Message-ID: <86k1gqanx8.fsf@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: B86A889620 X-Spamd-Bar: -------- Authentication-Results: mx1.freebsd.org; dmarc=pass (policy=none) header.from=intel.com; spf=pass (mx1.freebsd.org: domain of d.scott.phillips@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=d.scott.phillips@intel.com X-Spamd-Result: default: False [-8.79 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip4:192.55.52.136/32]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[136.52.55.192.list.dnswl.org : 127.0.9.2]; MX_GOOD(-0.01)[cached: mga04.intel.com]; DMARC_POLICY_ALLOW(-0.50)[intel.com,none]; NEURAL_HAM_SHORT(-0.98)[-0.984,0]; IP_SCORE(-1.80)[ipnet: 192.55.52.0/24(-4.96), asn: 4983(-3.96), country: US(-0.07)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:4983, ipnet:192.55.52.0/24, country:US]; MID_RHS_MATCH_FROM(0.00)[]; WHITELIST_SPF_DKIM(-3.00)[intel.com:s:+] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Mar 2019 16:12:28 -0000 rebecca@bluestop.org writes: > On Mar 21, 2019, 9:54 PM -0600, Larry Rosenman , wrote: > > out of curiosity, is there any work being done to move this to later > > LLVM/CLANG? > > That=E2=80=99s something else I=E2=80=99m hoping to work on, since gcc 4.= 8 is very > outdated now. It seems to build fine using gcc 7, but I need to test > that it still runs fine. And that=E2=80=99s complicated by the fact that = the > DEBUG build generates a triple fault even with gcc 4.8. You're seeing this firmware triple fault in the DEBUG build of this new firmware? It seems to be functioning properly for me. Do you get any serial output before the fault? From owner-freebsd-virtualization@freebsd.org Fri Mar 22 20:00:25 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D8D241550378; Fri, 22 Mar 2019 20:00:24 +0000 (UTC) (envelope-from rebecca@bluestop.org) Received: from muon.bluestop.org (muon.bluestop.org [65.103.231.193]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E8801912AD; Fri, 22 Mar 2019 20:00:23 +0000 (UTC) (envelope-from rebecca@bluestop.org) Received: from muon.bluestop.org (localhost [127.0.0.1]) by muon.bluestop.org (Postfix) with ESMTP id 997F37FCB1; Fri, 22 Mar 2019 14:01:30 -0600 (MDT) Received: from muon.bluestop.org ([127.0.0.1]) by muon.bluestop.org (muon.bluestop.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 3voT63tTumfm; Fri, 22 Mar 2019 14:01:29 -0600 (MDT) Received: from photon.int.bluestop.org (unknown [65.103.231.197]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by muon.bluestop.org (Postfix) with ESMTPSA; Fri, 22 Mar 2019 14:01:29 -0600 (MDT) Subject: Re: Updating uefi-edk2-bhyve To: D Scott Phillips , Larry Rosenman Cc: freebsd-virtualization@freebsd.org, owner-freebsd-virtualization@freebsd.org References: <86muln68ld.fsf@intel.com> <1fe3ca3f-be70-99db-e7c0-35c9194c97e4@bluestop.org> <7e84fd01c3f46268c26f9bab8b9fb9bc@lerctr.org> <33fcf111-fe00-4ec7-8a2f-7c53246d756f@Spark> <86k1gqanx8.fsf@intel.com> From: Rebecca Cran Message-ID: <8c63eb87-e3b8-2365-2eaf-a6e36424407c@bluestop.org> Date: Fri, 22 Mar 2019 14:00:21 -0600 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.5.3 MIME-Version: 1.0 In-Reply-To: <86k1gqanx8.fsf@intel.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: E8801912AD X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.91 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[bluestop.org:s=mail]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(-0.99)[ipnet: 65.100.0.0/14(-4.88), asn: 209(-0.01), country: US(-0.07)]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bluestop.org:+]; MX_GOOD(-0.01)[cached: mail.bluestop.org]; DMARC_POLICY_ALLOW(-0.50)[bluestop.org,quarantine]; NEURAL_HAM_SHORT(-0.91)[-0.907,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:209, ipnet:65.100.0.0/14, country:US]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Mar 2019 20:00:25 -0000 On 3/22/19 10:12 AM, D Scott Phillips wrote: > > You're seeing this firmware triple fault in the DEBUG build of this new > firmware? It seems to be functioning properly for me. Do you get any > serial output before the fault? I do get quite a bit of output before the fault. I'm running 13-CURRENT (on an Skylake PC), built today, so there may be some new issues that aren't in a release. I'm building OVMF with "build -p OvmfPkg/OvmfPkgX64.dsc -a X64 -b DEBUG -DDEBUG_ON_SERIAL_PORT=TRUE -t GCC5" root@cube:# bhyve -AHP -s 0:0,hostbridge -s 31:0,lpc -c 4 -m 4G -s 29,fbuf,tcp=0.0.0.0:5900,w=800,h=600 -l bootrom,OVMF_CODE.fd -l com1,stdio guest fbuf frame buffer base: 0x941e00000 [sz 16777216] SecCoreStartupWithStack(0xFFFCC000, 0x820000) SEC: Normal boot DecompressMemFvs: OutputBuffer@A00000+0xBE0090 ScratchBuffer@1600000+0x10000 PcdOvmfDecompressionScratchEnd=0x1610000 Register PPI Notify: DCD0BE23-9586-40F4-B643-06522CED4EDE Install PPI: 8C8CE578-8A3D-4F1C-9935-896185C32DD3 Install PPI: 5473C07A-3DCB-4DCA-BD6F-1E9689E7349A The 0th FV start address is 0x00000820000, size is 0x000E0000, handle is 0x820000 Register PPI Notify: 49EDB1C1-BF21-4761-BB12-EB0031AABB39 Register PPI Notify: EA7CA24B-DED5-4DAD-A389-BF827E8F9B38 Install PPI: B9E0ABFE-5979-4914-977F-6DEE78C278A6 Install PPI: DBE23AA9-A345-4B97-85B6-B226F1617389 DiscoverPeimsAndOrderWithApriori(): Found 0x6 PEI FFS files in the 0th FV Evaluate PEI DEPEX for FFS(9B3ADA4F-AE56-4C24-8DEA-F03B7558AE50)   RESULT = TRUE (Apriori) Loading PEIM 9B3ADA4F-AE56-4C24-8DEA-F03B7558AE50 Loading PEIM at 0x0000082C340 EntryPoint=0x0000082F812 PcdPeim.efi Install PPI: 06E81C58-4AD7-44BC-8390-F10265F72480 Install PPI: 01F34D25-4DE2-23AD-3FF3-36353FF323F1 Install PPI: 4D8B155B-C059-4C8F-8926-06FD4331DB8A Install PPI: A60C6B59-E459-425D-9C69-0BCC9CB27D81 Register PPI Notify: 605EA650-C65C-42E1-BA80-91A52AB618C6 Evaluate PEI DEPEX for FFS(A3610442-E69F-4DF3-82CA-2360C4031A23)   TRUE   END   RESULT = TRUE Loading PEIM A3610442-E69F-4DF3-82CA-2360C4031A23 Loading PEIM at 0x00000831140 EntryPoint=0x00000832848 ReportStatusCodeRouterPei.efi Install PPI: 0065D394-9951-4144-82A3-0AFC8579C251 Install PPI: 229832D3-7A30-4B36-B827-F40CB7D45436 Evaluate PEI DEPEX for FFS(9D225237-FA01-464C-A949-BAABC02D31D0)   PUSH GUID(0065D394-9951-4144-82A3-0AFC8579C251) = TRUE   END   RESULT = TRUE Loading PEIM 9D225237-FA01-464C-A949-BAABC02D31D0 Loading PEIM at 0x000008335C0 EntryPoint=0x00000834B6F StatusCodeHandlerPei.efi Evaluate PEI DEPEX for FFS(222C386D-5ABC-4FB4-B124-FBB82488ACF4)   PUSH GUID(01F34D25-4DE2-23AD-3FF3-36353FF323F1) = TRUE   END   RESULT = TRUE Loading PEIM 222C386D-5ABC-4FB4-B124-FBB82488ACF4 Loading PEIM at 0x00000835BC0 EntryPoint=0x00000835E00 PlatformPei.efi Platform PEIM Loaded CMOS: 00: 50 00 47 00 13 00 06 22 03 19 20 02 00 80 00 00 10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 30: 00 00 20 00 00 BF 00 00 00 00 00 00 00 00 00 00 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 40 00 00 00 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Install PPI: 7408D748-FC8C-4EE6-9288-C4BEC092A410 GetFirstNonAddress: Pci64Base=0x800000000 Pci64Size=0x800000000 PublishPeiMemory: mPhysMemAddressWidth=36 PeiMemoryCap=65800 KB PeiInstallPeiMemory MemoryBegin 0xBBFBE000, MemoryLength 0x4042000 QemuInitializeRam called Reserved variable store memory: 0xBFF7C000; size: 528kb Platform PEI Firmware Volume Initialization Install PPI: 49EDB1C1-BF21-4761-BB12-EB0031AABB39 Notify: PPI Guid: 49EDB1C1-BF21-4761-BB12-EB0031AABB39, Peim notify entry point: 8292C1 The 1th FV start address is 0x00000900000, size is 0x00B00000, handle is 0x900000 Register PPI Notify: EE16160A-E8BE-47A6-820A-C6900DB0250A Temp Stack : BaseAddress=0x818000 Length=0x8000 Temp Heap  : BaseAddress=0x810000 Length=0x8000 Total temporary memory:    65536 bytes.   temporary memory stack ever used:       5096 bytes.   temporary memory heap used for HobList: 6328 bytes.   temporary memory heap occupied by memory pages: 0 bytes. Memory Allocation 0x00000004 0x806000 - 0x806FFF Memory Allocation 0x00000006 0xBFF7C000 - 0xBFFFFFFF Memory Allocation 0x00000004 0x820000 - 0x8FFFFF Memory Allocation 0x00000004 0x900000 - 0x13FFFFF Old Stack size 32768, New stack size 131072 Stack Hob: BaseAddress=0xBBFBE000 Length=0x20000 Heap Offset = 0xBB7CE000 Stack Offset = 0xBB7BE000 TemporaryRamMigration(0x810000, 0xBBFD6000, 0x10000) vm exit[0] reason          VMX rip             0x0000000000827359 inst_length     5 status          0 exit_reason     2 (Triple fault) qualification    0x0000000000000000 inst_type               0 inst_error              0 Abort On the build problem side, if I run "build -p OvmfPkg/OvmfPkgX64.dsc -a X64 -t GCC48 -b RELEASE" then I get an error (that I don't get if I replace GCC48 with GCC5): /usr/local/bin/ld: warning: cannot find entry symbol _ModuleEntryPoint; not setting start address /usr/local/bin/ld: warning: cannot find entry symbol _ModuleEntryPoint; not setting start address "objcopy" /home/bcran/workspace/bhyveedk2/Build/OvmfX64/RELEASE_GCC48/X64/OvmfPkg/AmdSevDxe/AmdSevDxe/DEBUG/AmdSevDxe.dll "objcopy" /home/bcran/workspace/bhyveedk2/Build/OvmfX64/RELEASE_GCC48/X64/OvmfPkg/EmuVariableFvbRuntimeDxe/Fvb/DEBUG/EmuVariableFvbRuntimeDxe.dll objcopy: error: the input file '/home/bcran/workspace/bhyveedk2/Build/OvmfX64/RELEASE_GCC48/X64/OvmfPkg/EmuVariableFvbRuntimeDxe/Fvb/DEBUG/EmuVariableFvbRuntimeDxe.dll' has no sections objcopy: error: the input file '/home/bcran/workspace/bhyveedk2/Build/OvmfX64/RELEASE_GCC48/X64/OvmfPkg/AmdSevDxe/AmdSevDxe/DEBUG/AmdSevDxe.dll' has no sections make: *** [GNUmakefile:383: /home/bcran/workspace/bhyveedk2/Build/OvmfX64/RELEASE_GCC48/X64/OvmfPkg/AmdSevDxe/AmdSevDxe/DEBUG/AmdSevDxe.dll] Error 1 -- Rebecca Cran From owner-freebsd-virtualization@freebsd.org Fri Mar 22 20:25:58 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4689A1550D00; Fri, 22 Mar 2019 20:25:58 +0000 (UTC) (envelope-from d.scott.phillips@intel.com) Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "fmsmga101.fm.intel.com", Issuer "COMODO RSA Organization Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BCE5B9203D; Fri, 22 Mar 2019 20:25:56 +0000 (UTC) (envelope-from d.scott.phillips@intel.com) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Mar 2019 13:25:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,256,1549958400"; d="scan'208";a="125060312" Received: from dsp-dsk1.jf.intel.com (HELO localhost) ([10.54.70.10]) by orsmga007.jf.intel.com with ESMTP; 22 Mar 2019 13:25:48 -0700 From: D Scott Phillips To: Rebecca Cran , Larry Rosenman Cc: freebsd-virtualization@freebsd.org, owner-freebsd-virtualization@freebsd.org Subject: Re: Updating uefi-edk2-bhyve In-Reply-To: <8c63eb87-e3b8-2365-2eaf-a6e36424407c@bluestop.org> References: <86muln68ld.fsf@intel.com> <1fe3ca3f-be70-99db-e7c0-35c9194c97e4@bluestop.org> <7e84fd01c3f46268c26f9bab8b9fb9bc@lerctr.org> <33fcf111-fe00-4ec7-8a2f-7c53246d756f@Spark> <86k1gqanx8.fsf@intel.com> <8c63eb87-e3b8-2365-2eaf-a6e36424407c@bluestop.org> Date: Fri, 22 Mar 2019 13:25:48 -0700 Message-ID: <86ftremzar.fsf@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: BCE5B9203D X-Spamd-Bar: --------- Authentication-Results: mx1.freebsd.org; dmarc=pass (policy=none) header.from=intel.com; spf=pass (mx1.freebsd.org: domain of d.scott.phillips@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=d.scott.phillips@intel.com X-Spamd-Result: default: False [-9.09 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip4:192.55.52.88/32]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: mga04.intel.com]; DMARC_POLICY_ALLOW(-0.50)[intel.com,none]; NEURAL_HAM_SHORT(-0.98)[-0.984,0]; IP_SCORE(-1.80)[ipnet: 192.55.52.0/24(-4.96), asn: 4983(-3.96), country: US(-0.07)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:4983, ipnet:192.55.52.0/24, country:US]; RCVD_IN_DNSWL_HI(-0.50)[88.52.55.192.list.dnswl.org : 127.0.9.3]; MID_RHS_MATCH_FROM(0.00)[]; WHITELIST_SPF_DKIM(-3.00)[intel.com:s:+] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Mar 2019 20:25:58 -0000 Rebecca Cran writes: > On 3/22/19 10:12 AM, D Scott Phillips wrote: > > > > > You're seeing this firmware triple fault in the DEBUG build of this new > > firmware? It seems to be functioning properly for me. Do you get any > > serial output before the fault? > > I do get quite a bit of output before the fault. I'm running 13-CURRENT=20 > (on an Skylake PC), built today, so there may be some new issues that=20 > aren't in a release. > > I'm building OVMF with "build -p OvmfPkg/OvmfPkgX64.dsc -a X64 -b DEBUG=20 > -DDEBUG_ON_SERIAL_PORT=3DTRUE -t GCC5" > > root@cube:# bhyve -AHP -s 0:0,hostbridge -s 31:0,lpc -c 4 -m 4G -s=20 > 29,fbuf,tcp=3D0.0.0.0:5900,w=3D800,h=3D600 -l bootrom,OVMF_CODE.fd -l=20 > com1,stdio guest > > [..snip..] > > Memory Allocation 0x00000004 0x806000 - 0x806FFF > Memory Allocation 0x00000006 0xBFF7C000 - 0xBFFFFFFF > Memory Allocation 0x00000004 0x820000 - 0x8FFFFF > Memory Allocation 0x00000004 0x900000 - 0x13FFFFF > Old Stack size 32768, New stack size 131072 > Stack Hob: BaseAddress=3D0xBBFBE000 Length=3D0x20000 > Heap Offset =3D 0xBB7CE000 Stack Offset =3D 0xBB7BE000 > TemporaryRamMigration(0x810000, 0xBBFD6000, 0x10000) > vm exit[0] > reason=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 VMX > rip=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 0x0000000000827359 > inst_length=C2=A0=C2=A0=C2=A0=C2=A0 5 > status=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0 > exit_reason=C2=A0=C2=A0=C2=A0=C2=A0 2 (Triple fault) > qualification=C2=A0=C2=A0=C2=A0 0x0000000000000000 > inst_type=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 0 > inst_error=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 0 > Abort Hmm, I guess it might be some diference in the code generation between gcc 4.8 and gcc 5. > On the build problem side, if I run "build -p OvmfPkg/OvmfPkgX64.dsc -a=20 > X64 -t GCC48 -b RELEASE" then I get an error (that I don't get if I=20 > replace GCC48 with GCC5): > > /usr/local/bin/ld: warning: cannot find entry symbol _ModuleEntryPoint;=20 > not setting start address > /usr/local/bin/ld: warning: cannot find entry symbol _ModuleEntryPoint;=20 > not setting start address > "objcopy"=20 > /home/bcran/workspace/bhyveedk2/Build/OvmfX64/RELEASE_GCC48/X64/OvmfPkg/A= mdSevDxe/AmdSevDxe/DEBUG/AmdSevDxe.dll > "objcopy"=20 > /home/bcran/workspace/bhyveedk2/Build/OvmfX64/RELEASE_GCC48/X64/OvmfPkg/E= muVariableFvbRuntimeDxe/Fvb/DEBUG/EmuVariableFvbRuntimeDxe.dll > objcopy: error: the input file=20 > '/home/bcran/workspace/bhyveedk2/Build/OvmfX64/RELEASE_GCC48/X64/OvmfPkg/= EmuVariableFvbRuntimeDxe/Fvb/DEBUG/EmuVariableFvbRuntimeDxe.dll'=20 > has no sections > objcopy: error: the input file=20 > '/home/bcran/workspace/bhyveedk2/Build/OvmfX64/RELEASE_GCC48/X64/OvmfPkg/= AmdSevDxe/AmdSevDxe/DEBUG/AmdSevDxe.dll'=20 > has no sections > make: *** [GNUmakefile:383:=20 > /home/bcran/workspace/bhyveedk2/Build/OvmfX64/RELEASE_GCC48/X64/OvmfPkg/A= mdSevDxe/AmdSevDxe/DEBUG/AmdSevDxe.dll]=20 > Error 1 I'm not seeing this problem with the build steps I'm doing, which are just the ones in the ports patch. Perhaps some system version of one of the build tools is getting used at some point? I had to mess with ${PATH} to get the build to work right. I'd say look back through the build log and make sure it's using the expected executables. On my side I'll run the build through poudriere to make sure it's not working due to some specific quirk of my system. From owner-freebsd-virtualization@freebsd.org Fri Mar 22 22:29:58 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A18A61553B8F; Fri, 22 Mar 2019 22:29:58 +0000 (UTC) (envelope-from rebecca@bluestop.org) Received: from muon.bluestop.org (muon.bluestop.org [65.103.231.193]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 45C8A96A61; Fri, 22 Mar 2019 22:29:57 +0000 (UTC) (envelope-from rebecca@bluestop.org) Received: from muon.bluestop.org (localhost [127.0.0.1]) by muon.bluestop.org (Postfix) with ESMTP id 7ACB37FD8C; Fri, 22 Mar 2019 16:31:03 -0600 (MDT) Received: from muon.bluestop.org ([127.0.0.1]) by muon.bluestop.org (muon.bluestop.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id NOtdbY29lRiI; Fri, 22 Mar 2019 16:31:02 -0600 (MDT) Received: from photon.int.bluestop.org (unknown [65.103.231.197]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by muon.bluestop.org (Postfix) with ESMTPSA; Fri, 22 Mar 2019 16:31:02 -0600 (MDT) Subject: Re: Updating uefi-edk2-bhyve To: D Scott Phillips , Larry Rosenman Cc: freebsd-virtualization@freebsd.org, owner-freebsd-virtualization@freebsd.org References: <86muln68ld.fsf@intel.com> <1fe3ca3f-be70-99db-e7c0-35c9194c97e4@bluestop.org> <7e84fd01c3f46268c26f9bab8b9fb9bc@lerctr.org> <33fcf111-fe00-4ec7-8a2f-7c53246d756f@Spark> <86k1gqanx8.fsf@intel.com> <8c63eb87-e3b8-2365-2eaf-a6e36424407c@bluestop.org> <86ftremzar.fsf@intel.com> From: Rebecca Cran Message-ID: Date: Fri, 22 Mar 2019 16:29:54 -0600 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.5.3 MIME-Version: 1.0 In-Reply-To: <86ftremzar.fsf@intel.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 45C8A96A61 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.96 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[bluestop.org:s=mail]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(-0.99)[ipnet: 65.100.0.0/14(-4.88), asn: 209(-0.01), country: US(-0.07)]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: mail.bluestop.org]; DKIM_TRACE(0.00)[bluestop.org:+]; DMARC_POLICY_ALLOW(-0.50)[bluestop.org,quarantine]; NEURAL_HAM_SHORT(-0.96)[-0.959,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:209, ipnet:65.100.0.0/14, country:US]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Mar 2019 22:29:58 -0000 On 3/22/19 2:25 PM, D Scott Phillips wrote: > Hmm, I guess it might be some diference in the code generation between > gcc 4.8 and gcc 5. I've just tested switching from gcc 4.8 to 8.3.0 and everything seems to work fine - both build and runtime - so I think it may be more productive to upgrade and use that instead, since it's a supported version (gcc 4.8 is pretty old now). Would you agree? The DEBUG build did complain about unused definitions and functions in FeatureControl.c though, so I ended up with the following patch: diff --git a/OvmfPkg/PlatformPei/FeatureControl.c b/OvmfPkg/PlatformPei/FeatureControl.c index 09f33fe5b1..051dbbdac0 100644 --- a/OvmfPkg/PlatformPei/FeatureControl.c +++ b/OvmfPkg/PlatformPei/FeatureControl.c @@ -24,7 +24,7 @@  //  // The value to be written to the Feature Control MSR, retrieved from fw_cfg.  // -STATIC UINT64 mFeatureControlValue; +//STATIC UINT64 mFeatureControlValue;^M  /**    Write the Feature Control MSR on an Application Processor or the Boot @@ -36,6 +36,7 @@ STATIC UINT64 mFeatureControlValue;    @param[in,out] WorkSpace  Pointer to the input/output argument workspace                              shared by all processors.  **/ +#if 0^M  STATIC  VOID  EFIAPI @@ -45,7 +46,7 @@ WriteFeatureControl (  {    AsmWriteMsr64 (MSR_CORE2_FEATURE_CONTROL, mFeatureControlValue);  } - +#endif^M  /**    Notification function called when EFI_PEI_MP_SERVICES_PPI becomes available. @@ -57,6 +58,7 @@ WriteFeatureControl (    @return  Status of the notification. The status code returned from this             function is ignored.  **/ +#if 0^M  STATIC  EFI_STATUS  EFIAPI @@ -94,17 +96,17 @@ OnMpServicesAvailable (    WriteFeatureControl (NULL);    return EFI_SUCCESS;  } - +#endif^M  //  // Notification object for registering the callback, for when  // EFI_PEI_MP_SERVICES_PPI becomes available.  // -STATIC CONST EFI_PEI_NOTIFY_DESCRIPTOR mMpServicesNotify = { -  EFI_PEI_PPI_DESCRIPTOR_NOTIFY_CALLBACK | // Flags -  EFI_PEI_PPI_DESCRIPTOR_TERMINATE_LIST, -  &gEfiPeiMpServicesPpiGuid,               // Guid -  OnMpServicesAvailable                    // Notify -}; +//STATIC CONST EFI_PEI_NOTIFY_DESCRIPTOR mMpServicesNotify = {^M +//  EFI_PEI_PPI_DESCRIPTOR_NOTIFY_CALLBACK | // Flags^M +//  EFI_PEI_PPI_DESCRIPTOR_TERMINATE_LIST,^M +//  &gEfiPeiMpServicesPpiGuid,               // Guid^M +//  OnMpServicesAvailable                    // Notify^M +//};^M  VOID  InstallFeatureControlCallback ( From owner-freebsd-virtualization@freebsd.org Sat Mar 23 12:54:27 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E54F9154E994 for ; Sat, 23 Mar 2019 12:54:26 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7D6928FC77 for ; Sat, 23 Mar 2019 12:54:24 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 4A2D6200E3 for ; Sat, 23 Mar 2019 08:54:18 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Sat, 23 Mar 2019 08:54:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zyxst.net; h= date:from:to:subject:message-id:mime-version:content-type; s= fm2; bh=SNj3HRC2RHv8GPDky3/UXB3c/PVEje57bEgakfqWaYM=; b=MfX6FVyD vgZmubdvvLq63VVtPI6ude+oOozyJHLiCnxM4M36K671WykJKdnOiqS7R76qeGsk siFBxerRWEMhtEqtN1bR5fWwccmafA1h63xin/9Wk0NIEgoy5AkVhAKwHRzIMl3I 4LCmxwiOm5fmNeMjysxX4iHuI7yDtdCz77396ExfyhOgHGeMMhjhB4ZbrlimSoS1 XmO1sXJyIAjpT+WTOhvtNNeKmjbruCM0eaWHDRc1nGBYWYqB/DPG2mNOYhQ1Gp10 E7v1w7K2jRhrxk8HRpKsngJJKKdqgKOWSPknHJ8EienYT1lXpODeI79SgiETPrPL lv72RMQFxqtsRA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:message-id :mime-version:subject:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm2; bh=SNj3HRC2RHv8GPDky3/UXB3c/PVEj e57bEgakfqWaYM=; b=wM+FPO/ZkjqGN6fO+i+hJMlnZb7QhfZOR7ET5IeY1N498 uPnHiZ2Fl1JtU4Sr/a7DtCLndozA8mtEY299A8Px61kptDSxQQjbW04n0x6fZVFX yLq4tknOd1aZ4hZLEIhwiW1NhFCfHz7tMakpcGpFjMwlWtzrw/UMIUkyhm4xjw2y UOU3VDKmab/liT2M/XlHNIdNyn7y8bEGagiup26LCBhVzpxzM48NVjuY8V1c4YOQ p4UboEP+75k7mEUAkU2tPViALxjJ/elv4SwabwHaNZkCqy2fzDcfWjIDvjhRlibI yTjw0z362l7SpgP6tzDe6EhTe6KKWbqt5OkmgNh0A== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedutddrjeefgdegjecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfggtggufgesghdtreertd ervdenucfhrhhomhepthgvtghhqdhlihhsthhsuceothgvtghhqdhlihhsthhsseiihiig shhtrdhnvghtqeenucfkphepkedvrdejtddrledurddutddunecurfgrrhgrmhepmhgrih hlfhhrohhmpehtvggthhdqlhhishhtshesiiihgihsthdrnhgvthenucevlhhushhtvghr ufhiiigvpedt X-ME-Proxy: Received: from rpi3.zyxst.net (rpi3.zyxst.net [82.70.91.101]) by mail.messagingengine.com (Postfix) with ESMTPA id 70721E4176 for ; Sat, 23 Mar 2019 08:54:17 -0400 (EDT) Date: Sat, 23 Mar 2019 12:54:15 +0000 From: tech-lists To: freebsd-virtualization@freebsd.org Subject: freebsd guest on azure Message-ID: <20190323125414.GA43521@rpi3.zyxst.net> Mail-Followup-To: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="NzB8fVQJ5HfG6fxh" Content-Disposition: inline User-Agent: Mutt/1.11.2 (2019-01-07) X-Rspamd-Queue-Id: 7D6928FC77 X-Spamd-Bar: --------- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=zyxst.net header.s=fm2 header.b=MfX6FVyD; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=wM+FPO/Z; spf=pass (mx1.freebsd.org: domain of tech-lists@zyxst.net designates 66.111.4.27 as permitted sender) smtp.mailfrom=tech-lists@zyxst.net X-Spamd-Result: default: False [-9.20 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[zyxst.net:s=fm2,messagingengine.com:s=fm2]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.27]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[zyxst.net]; DKIM_TRACE(0.00)[zyxst.net:+,messagingengine.com:+]; MX_GOOD(-0.01)[in2-smtp.messagingengine.com,in1-smtp.messagingengine.com,in2-smtp.messagingengine.com,in1-smtp.messagingengine.com,in2-smtp.messagingengine.com,in1-smtp.messagingengine.com]; NEURAL_HAM_SHORT(-0.96)[-0.965,0]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:11403, ipnet:66.111.4.0/24, country:US]; IP_SCORE(-3.53)[ip: (-9.55), ipnet: 66.111.4.0/24(-4.60), asn: 11403(-3.42), country: US(-0.07)]; RCVD_IN_DNSWL_LOW(-0.10)[27.4.111.66.list.dnswl.org : 127.0.5.1] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Mar 2019 12:54:27 -0000 --NzB8fVQJ5HfG6fxh Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, If you're running a freebsd instance on azure, 1. can you back up the instance -without azure's own tools -or with azures own tools (and I imagine it costs extra) 2. or if you can't directly back up the instance, apart from tarring up the directories you need to keep and downloading them from the instance,=20 what do you do? thanks, --=20 J. --NzB8fVQJ5HfG6fxh Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEE8n3tWhxW11Ccvv9/s8o7QhFzNAUFAlyWLG0ACgkQs8o7QhFz NAUcSw//ZXJBH+0MM9PQu6M63lk84Q3mfVC+dBjFB0+aE+M/kGRtTAaEg4k2bc2T Fzx2o1iXgXyEs+zJQ1wlF8f1Q50gr/TtCDWwEukkrXggyvHvWNUnS9qerJWM8vIQ sd7pWVstoGCcWylD1L92ynEKTl2xcB7XRh7nCWYfMn9qmGbVPtF9wr4bRydDubTK jyRFCSvQnUcrhNYWMMHYSsS7XJB2eO36I3ACmuBR6PmxwWbO23qBUm9M2HbpIzXr W8JFS+NhRVkokAccarV13d2jAzADidj8DqQjU65VDqBq/ccDlyZ5+dwmgSZAIFul F/80+H0lZBzZXAPe7fEOwtYpMNKCtHCs9EdWujTOXWCIC3cLAL78MzjikCWB1Uuj mY2kH+AAG9OVGvYTC5TWJ8kMnR5TAs5tZ2jnoAfk3MvdONpL/GZgq7048HgQhemP Z6DDi2M6LaeLyOKccEVIXX+VB5OLD+ry9pNFypFfFcm4GIm1AR2XVm+4YUORlDvf c4fC3hXMcsGPjgm9fJ2cgAxAI1JVLDiK04NehwVx2xtj4moCJXKmA+B+xwJdxVUT WH6tHOro6BtIKnnWwWztAa4HxjTUJ5UbBh0VZ0nhaN34UCHfBJRx/nBKlBOzI7cd YmLqZxHU76t9PYgo0W9n0IrBoHeWbKJuVdfgg64LW2TWj56wgUw= =pJI8 -----END PGP SIGNATURE----- --NzB8fVQJ5HfG6fxh-- From owner-freebsd-virtualization@freebsd.org Sat Mar 23 13:52:41 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EAEEF15514AD for ; Sat, 23 Mar 2019 13:52:40 +0000 (UTC) (envelope-from frank2@fjl.co.uk) Received: from bs1.fjl.org.uk (bs1.fjl.org.uk [84.45.41.196]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "bs1.fjl.org.uk", Issuer "bs1.fjl.org.uk" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id B11E86B9B9 for ; Sat, 23 Mar 2019 13:52:33 +0000 (UTC) (envelope-from frank2@fjl.co.uk) Received: from [10.80.23.167] ([103.14.185.11]) (authenticated bits=0) by bs1.fjl.org.uk (8.14.4/8.14.4) with ESMTP id x2NDVLgw022002 for ; Sat, 23 Mar 2019 13:31:22 GMT (envelope-from frank2@fjl.co.uk) Date: Sat, 23 Mar 2019 21:31:17 +0800 User-Agent: K-9 Mail for Android In-Reply-To: <20190323125414.GA43521@rpi3.zyxst.net> References: <20190323125414.GA43521@rpi3.zyxst.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Re: freebsd guest on azure To: freebsd-virtualization@freebsd.org From: Frank Leonhardt Message-ID: <610D9FF5-891C-4657-BA2A-1E38D41AFF35@fjl.co.uk> X-Rspamd-Queue-Id: B11E86B9B9 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of frank2@fjl.co.uk designates 84.45.41.196 as permitted sender) smtp.mailfrom=frank2@fjl.co.uk X-Spamd-Result: default: False [-1.16 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.53)[-0.530,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:84.45.41.196]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.85)[-0.850,0]; RCVD_TLS_LAST(0.00)[]; NEURAL_SPAM_SHORT(0.75)[0.750,0]; RCVD_IN_DNSWL_MED(-0.20)[196.41.45.84.list.dnswl.org : 127.0.6.2]; MX_GOOD(-0.01)[bs1.fjl.org.uk]; DMARC_NA(0.00)[fjl.co.uk]; IP_SCORE(-0.02)[country: GB(-0.09)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:25577, ipnet:84.45.0.0/17, country:GB]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Mar 2019 13:52:41 -0000 On 23 March 2019 20:54:15 GMT+08:00, tech-lists w= rote: >Hi, > >If you're running a freebsd instance on azure, > >1=2E can you back up the instance > >-without azure's own tools >-or with azures own tools (and I imagine it costs extra) > >2=2E or if you can't directly back up the instance, apart from tarring up >the directories you need to keep and downloading them from the >instance,=20 >what do you do? First off, if I'm doing this on Azure then someone with deep pockets and a= Microsoft fixation is paying=2E I thought I'd make that clear! As to your question, what I've done is run a ZFS pool and simply send a sn= apshot=2E I can't remember now if I made it boot from ZFS as I prefer to bo= ot from UFS and use ZFS for the rest=2E But I think I did this with Azure e= xactly because I wanted to copy the disk around=2E It was early last year= so you can't quite me on it=2E If you're not sure about sending datasets on ZFS then ask=2E Regards, Frank=2E --=20 Sent from my Android device with K-9 Mail=2E Please excuse my brevity=2E From owner-freebsd-virtualization@freebsd.org Sat Mar 23 18:57:31 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3ED4E1559BB6 for ; Sat, 23 Mar 2019 18:57:31 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A60C077239 for ; Sat, 23 Mar 2019 18:57:29 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 8B31E20ED2 for ; Sat, 23 Mar 2019 14:57:28 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Sat, 23 Mar 2019 14:57:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zyxst.net; h= date:from:to:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm2; bh=f7igJgBwCYcxPfO+IBIrSzyihyi GOtkXScAEMO9ImZU=; b=SidS4q6iH72gRR9l9Fi/uxWoM8c6wjDi2jm69ToQO1H CK8XcGZ3kYZ2mo4fPZXWxok/YBKIIIpeR8LBE8zMfK47ppYdRi9iFCta8x0Gk6Ly +gNohUyaeAcATbulvaaIt5Hf/0YHXolpNju6KmrOcp/EmabCrzfQGYbw5I3h7pQa LNzLbRzDLHhf+vePKeUB52r7U2PtXJXo+Lcu3W+mWsCAx0qSBGs1an40MYFXWIDL 0FaHKb92UyaXJwzolqXn6kdXLax5HTF28iXZLw1g2NIHOdASPCgObBGajg5Zs6uq 8JoQQDDFVzN5dc92J+5l2nTbGF1zxQ+Ptk0BkJFY3jw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=f7igJg BwCYcxPfO+IBIrSzyihyiGOtkXScAEMO9ImZU=; b=lcg5ErnAq4LAzU/DK2SXaw Pt5KKPeq6gIo8zZYILm2r4x0snmaw1DpBIp+MfT7oVQAft2RjWNrGPLbnn7xAy2z 7iRjz019h69ivoNgvdFbVVib1v79H3ewujhp+D36KI/XQo1crzPngTo/eEPP9/AI 7zhfR6aqzVTLlG9fqi5yOCxmoL8/k9Zua1DzniHeu+2d/Y4zgKURsvIXHFiuTb1b 5Q8TCQecbWE4wjmigGbL6tbiXPXXxCVgtDzP46t3J/TFpiJX79uXVG/2LAEEQjMg 5+yjqt25MvcadoKQ2re0bKXkYMofaS9rGKAm1Z0nt3heVEcoOiIoV8h8qIL8v3qw == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedutddrjeefgdduvddvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggujggfsehgtd erredtredvnecuhfhrohhmpehtvggthhdqlhhishhtshcuoehtvggthhdqlhhishhtshes iiihgihsthdrnhgvtheqnecukfhppeekvddrjedtrdeluddruddtudenucfrrghrrghmpe hmrghilhhfrhhomhepthgvtghhqdhlihhsthhsseiihiigshhtrdhnvghtnecuvehluhhs thgvrhfuihiivgeptd X-ME-Proxy: Received: from rpi3.zyxst.net (rpi3.zyxst.net [82.70.91.101]) by mail.messagingengine.com (Postfix) with ESMTPA id 911BF1030F for ; Sat, 23 Mar 2019 14:57:27 -0400 (EDT) Date: Sat, 23 Mar 2019 18:57:25 +0000 From: tech-lists To: freebsd-virtualization@freebsd.org Subject: Re: freebsd guest on azure Message-ID: <20190323185724.GA71155@rpi3.zyxst.net> Mail-Followup-To: freebsd-virtualization@freebsd.org References: <20190323125414.GA43521@rpi3.zyxst.net> <610D9FF5-891C-4657-BA2A-1E38D41AFF35@fjl.co.uk> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="oyUTqETQ0mS9luUI" Content-Disposition: inline In-Reply-To: <610D9FF5-891C-4657-BA2A-1E38D41AFF35@fjl.co.uk> User-Agent: Mutt/1.11.2 (2019-01-07) X-Rspamd-Queue-Id: A60C077239 X-Spamd-Bar: --------- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=zyxst.net header.s=fm2 header.b=SidS4q6i; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=lcg5ErnA; spf=pass (mx1.freebsd.org: domain of tech-lists@zyxst.net designates 66.111.4.27 as permitted sender) smtp.mailfrom=tech-lists@zyxst.net X-Spamd-Result: default: False [-9.24 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[zyxst.net:s=fm2,messagingengine.com:s=fm2]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.27]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[zyxst.net]; MX_GOOD(-0.01)[cached: in2-smtp.messagingengine.com]; DKIM_TRACE(0.00)[zyxst.net:+,messagingengine.com:+]; NEURAL_HAM_SHORT(-1.00)[-0.999,0]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:11403, ipnet:66.111.4.0/24, country:US]; IP_SCORE(-3.54)[ip: (-9.59), ipnet: 66.111.4.0/24(-4.60), asn: 11403(-3.42), country: US(-0.07)]; RCVD_IN_DNSWL_LOW(-0.10)[27.4.111.66.list.dnswl.org : 127.0.5.1] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Mar 2019 18:57:31 -0000 --oyUTqETQ0mS9luUI Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Mar 23, 2019 at 09:31:17PM +0800, Frank Leonhardt wrote: >As to your question, what I've done is run a ZFS pool and simply send a sn= apshot.=20 That's a good idea. I presumed I'd be stuck just with UFS. My most favoured way of doing this is having a zfs-vol backed vm on a freebsd host but it's not my choice. > I can't remember now if I made it boot from ZFS as I prefer to boot from = UFS=20 this is me. I like to boot off something fast and simple. > and use ZFS for the rest. But I think I did this with Azure exactly becau= se =20 > I wanted to copy the disk around. It was early last year so you can't=20 > quite me on it. > > If you're not sure about sending datasets on ZFS then ask. Yes please how are you invoking sending a dataset in this context? For example, are you using compression for the data stream? If so, what sort? Is there anything else pertinent to the azure context when using zfs send? thanks, --=20 J. --oyUTqETQ0mS9luUI Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEE8n3tWhxW11Ccvv9/s8o7QhFzNAUFAlyWgYoACgkQs8o7QhFz NAVK4A//WrRcPcRoZi5QsoKAvazL2rc5LSz2HyVteOP6EJIjGLp+wH/5OZD8Ywj2 hlKZ50Vg30PvgYT/UvFuVswjcxZ8nqjsizYjfcsDL2MHDLzsAP2vPgiaMs0w/vd4 gzMyjplVPIb/tBDBZJ4yl3bSx1a04VX78pgXw9toJJMO0xDtjJIK8cfDe5znPaFt vHoV2mR8RgeVTT0o7WzLf5W6R3MLj6UkCwjzI2BAd6dPCSfBGD6lEuv8/wUWbgLH C5HsqSxdswjdjxvhFNTZY//pHHTBnPVjiR9UFqtCousxvDvbXTUh9Bjs3Mvw8HMy Dg53voaGktm6fxmA+cDQK9+tnATf0LOu8sQ1klFa2/w634Z2LD7Nb0BvH/1crzTn POwMGvonXLqpUOyOUVpeuxr3EuPXqfQ+hyjY7v1foQQf6OB2asiCzdM+prsfQUyd tN62j/nAFAQ9bHuvfm9aso9NqTB+D5LgavqBxrnvO3jf/6maGASnY16kVxGcD69i kVTeAfUzfbRJCSSQBaNQoFWJbqK70NoZl3T/HUM2um2RjLPipkYMttsRz2J5u8Uj Su31LnMiSB56JHxPAVN76FF8T2Fw4alrM6zjeVkefN26WlolXnTw9sSPgIxOHT5K 42GJ54ERnk/M9RpAFoVQmfe8wniF4RMIM/6EnOMkPPnuAX3Pn+U= =qIfD -----END PGP SIGNATURE----- --oyUTqETQ0mS9luUI-- From owner-freebsd-virtualization@freebsd.org Sat Mar 23 21:35:56 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DC757155D802 for ; Sat, 23 Mar 2019 21:35:55 +0000 (UTC) (envelope-from ctuffli@gmail.com) Received: from mail-ot1-f43.google.com (mail-ot1-f43.google.com [209.85.210.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DE44083E9C for ; Sat, 23 Mar 2019 21:35:54 +0000 (UTC) (envelope-from ctuffli@gmail.com) Received: by mail-ot1-f43.google.com with SMTP id d24so4879501otl.11 for ; Sat, 23 Mar 2019 14:35:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=u6lOTcyNnam9+Vs17xD2NKpN3YAhECJThGzDBAVSE6g=; b=CAeZ4IQyhEWipspTQCRPZf2vdr5LLzSd13VUJ6WJKye4Va2ytQS5DgwsmCFVx1p/VP esceUxDjLKYXH121AZpjFlImMMfWbbr9mhPHSimkIoN1+Hs7FCbzIXREGR6HsA4AvExV /QigjOlzxBOF7YCcpbU4P3sGo3Cyoc8dOsfvrPprKm+/MgswkQtox5OzPM3yhPqTFSQ1 /7lkmTmr29kp199iLHwkLNcQ3IlMtXnBbl/WFV2qqp0srLMqSWDWa+elqytzbDBuQWL3 Y88wRj13a7N9q5e5jljjXe5oBC9HPr57NP+M48aE1k65DCN/Z7dP8ydj13sdFb7wxCkC DbiA== X-Gm-Message-State: APjAAAUJ7xkZ4hXktxaFVZWX4lbWi+ySykcg/8VRwQhcEpuZN41PEfqW w5MFBt2Iz0Tygv4KPc/6W34NM5yp X-Google-Smtp-Source: APXvYqznNVLEHkmmgRqypHH/LGME6qIpS9vy2OpRl8uWOJvDIa5pZYZPZN2fo8UyhV9vp2oL+1TuUw== X-Received: by 2002:a9d:1b2f:: with SMTP id l44mr11984620otl.217.1553376947901; Sat, 23 Mar 2019 14:35:47 -0700 (PDT) Received: from mail-ot1-f44.google.com (mail-ot1-f44.google.com. [209.85.210.44]) by smtp.gmail.com with ESMTPSA id q6sm4504167oia.4.2019.03.23.14.35.47 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 23 Mar 2019 14:35:47 -0700 (PDT) Received: by mail-ot1-f44.google.com with SMTP id m10so4930945otp.2 for ; Sat, 23 Mar 2019 14:35:47 -0700 (PDT) X-Received: by 2002:a9d:75cc:: with SMTP id c12mr8057084otl.77.1553376947435; Sat, 23 Mar 2019 14:35:47 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Chuck Tuffli Date: Sat, 23 Mar 2019 14:35:36 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: emulated PCI device BAR size To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: DE44083E9C X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of ctuffli@gmail.com designates 209.85.210.43 as permitted sender) smtp.mailfrom=ctuffli@gmail.com X-Spamd-Result: default: False [-4.32 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.99)[-0.989,0]; FORGED_SENDER(0.30)[chuck@freebsd.org,ctuffli@gmail.com]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[chuck@freebsd.org,ctuffli@gmail.com]; TO_DOM_EQ_FROM_DOM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-1.32)[ip: (-0.55), ipnet: 209.85.128.0/17(-3.86), asn: 15169(-2.13), country: US(-0.07)]; RCVD_IN_DNSWL_NONE(0.00)[43.210.85.209.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[43.210.85.209.rep.mailspike.net : 127.0.0.17] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Mar 2019 21:35:56 -0000 On Mon, Mar 18, 2019 at 10:15 AM Chuck Tuffli wrote: > > In debugging why Windows doesn't like bhyve's NVMe device emulation, > another developer mentioned that the NVMe specification defines the > minimum size of BAR[0] to be 16K bytes. And while most OS don't > enforce this requirement, evidently, Windows does. > > On the surface, a "minimum BAR size" sounds odd, but the MMIO BAR in > NVMe includes the doorbell registers for each queue. And since the > maximum number of queues is implementation specific, there isn't a > fixed size for the BAR. > > So my question is how to best fix this. The easiest would be to add a > check to the BAR size calculation in pci_nvme.c along the lines of: > #define NVME_MMIO_SPACE_MIN (1 << 14) > ... > pci_membar_sz = MAX( , > NVME_MMIO_SPACE_MIN); I went down this route and have a patch up for review (https://reviews.freebsd.org/D19676) if anyone is interested. --chuck From owner-freebsd-virtualization@freebsd.org Sat Mar 23 21:59:22 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 10768155E2B8; Sat, 23 Mar 2019 21:59:22 +0000 (UTC) (envelope-from rebecca@bluestop.org) Received: from muon.bluestop.org (muon.bluestop.org [65.103.231.193]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E1F03849E8; Sat, 23 Mar 2019 21:59:20 +0000 (UTC) (envelope-from rebecca@bluestop.org) Received: from muon.bluestop.org (localhost [127.0.0.1]) by muon.bluestop.org (Postfix) with ESMTP id 8C381C0455; Sat, 23 Mar 2019 16:00:19 -0600 (MDT) Received: from muon.bluestop.org ([127.0.0.1]) by muon.bluestop.org (muon.bluestop.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id gsYg5jBhqRNa; Sat, 23 Mar 2019 16:00:18 -0600 (MDT) Received: from photon.int.bluestop.org (unknown [65.103.231.197]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by muon.bluestop.org (Postfix) with ESMTPSA; Sat, 23 Mar 2019 16:00:18 -0600 (MDT) Subject: Re: Updating uefi-edk2-bhyve To: D Scott Phillips Cc: Larry Rosenman , owner-freebsd-virtualization@freebsd.org, freebsd-virtualization@freebsd.org References: <86muln68ld.fsf@intel.com> <1fe3ca3f-be70-99db-e7c0-35c9194c97e4@bluestop.org> <7e84fd01c3f46268c26f9bab8b9fb9bc@lerctr.org> <33fcf111-fe00-4ec7-8a2f-7c53246d756f@Spark> <86k1gqanx8.fsf@intel.com> <8c63eb87-e3b8-2365-2eaf-a6e36424407c@bluestop.org> <86ftremzar.fsf@intel.com> From: Rebecca Cran Message-ID: <4fb370c4-0358-8ca7-67ee-c98fd83d2740@bluestop.org> Date: Sat, 23 Mar 2019 15:59:11 -0600 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.5.3 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Rspamd-Queue-Id: E1F03849E8 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.97 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[bluestop.org:s=mail]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(-0.99)[ipnet: 65.100.0.0/14(-4.88), asn: 209(-0.02), country: US(-0.07)]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bluestop.org:+]; DMARC_POLICY_ALLOW(-0.50)[bluestop.org,quarantine]; MX_GOOD(-0.01)[mail.bluestop.org]; NEURAL_HAM_SHORT(-0.97)[-0.966,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:209, ipnet:65.100.0.0/14, country:US]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Mar 2019 21:59:22 -0000 On 3/22/19 4:29 PM, Rebecca Cran via freebsd-virtualization wrote: > On 3/22/19 2:25 PM, D Scott Phillips wrote: > >> Hmm, I guess it might be some diference in the code generation between >> gcc 4.8 and gcc 5. > > > I've just tested switching from gcc 4.8 to 8.3.0 and everything seems > to work fine - both build and runtime - so I think it may be more > productive to upgrade and use that instead, since it's a supported > version (gcc 4.8 is pretty old now). Would you agree? I've submitted a couple of merge requests to update your edk2 tree and port for gcc 8: https://gitlab.com/scott-ph/edk2/merge_requests/1 https://gitlab.com/scott-ph/freebsd-ports/merge_requests/1 -- Rebecca Cran From owner-freebsd-virtualization@freebsd.org Sat Mar 23 22:07:44 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 68038155E5EB for ; Sat, 23 Mar 2019 22:07:44 +0000 (UTC) (envelope-from ctuffli@gmail.com) Received: from mail-ot1-f42.google.com (mail-ot1-f42.google.com [209.85.210.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8F39A84E4E for ; Sat, 23 Mar 2019 22:07:43 +0000 (UTC) (envelope-from ctuffli@gmail.com) Received: by mail-ot1-f42.google.com with SMTP id x8so4926710otg.7 for ; Sat, 23 Mar 2019 15:07:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=y/2fPSg+1/qeeQ2PHA59wiw9DdJBw5QbpkJBqLR7gGs=; b=IztBze2EjGEsZ8K9wnY5+b9WmnC68T+m9bHX6w7t4QfqUx3GcWw8+QV9xXt3FZS+SG j8hs/Nh1maHtNxp8a7vF7KNsVHncnUNAdtpE1ZetGH0OENW84hpIn8rxn78zq0247K0m wSdluvNwO+lm3OfPY0hgsQyozJ6iF5JljgBIR1Kaf05ppw3dpSJmciuajr0q1DgB8kiF r/IFttZt8QT3/nbdcxVDd2WEMnDHrCCU7sUmhTKrYOYcQDQ9i7Wzgz2lBH36n6setKhI a+VCaKbktVwnx733EHnimxwMSAR/lFi0rtcZ5H4U43BOTqe7XgtLET4I2g5WHiIjiYUM crbQ== X-Gm-Message-State: APjAAAX6wSumiWSpfq4Qlx7retWi8trgBIlKGw3l3F8zBJjA5s55JzeW 42nkfzdXktOfc8xZOV8XdQGdGj5A X-Google-Smtp-Source: APXvYqx+1iWOI7DwHeizsBmsHWXBHMIM7HLU7Kla5P3b8p+ZvOXaMP4YKZ3RtXVpoW6oyh+HtFWilA== X-Received: by 2002:a9d:73c2:: with SMTP id m2mr11811049otk.338.1553378857289; Sat, 23 Mar 2019 15:07:37 -0700 (PDT) Received: from mail-oi1-f172.google.com (mail-oi1-f172.google.com. [209.85.167.172]) by smtp.gmail.com with ESMTPSA id 65sm4678723ota.35.2019.03.23.15.07.36 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 23 Mar 2019 15:07:36 -0700 (PDT) Received: by mail-oi1-f172.google.com with SMTP id v7so4299095oie.8 for ; Sat, 23 Mar 2019 15:07:36 -0700 (PDT) X-Received: by 2002:aca:7541:: with SMTP id q62mr6803313oic.49.1553378856519; Sat, 23 Mar 2019 15:07:36 -0700 (PDT) MIME-Version: 1.0 From: Chuck Tuffli Date: Sat, 23 Mar 2019 15:07:25 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: bhyve threading model? To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 8F39A84E4E X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of ctuffli@gmail.com designates 209.85.210.42 as permitted sender) smtp.mailfrom=ctuffli@gmail.com X-Spamd-Result: default: False [-3.15 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[42.210.85.209.rep.mailspike.net : 127.0.0.18]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.93)[-0.925,0]; FORGED_SENDER(0.30)[chuck@freebsd.org,ctuffli@gmail.com]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[chuck@freebsd.org,ctuffli@gmail.com]; TO_DOM_EQ_FROM_DOM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; SUBJECT_ENDS_QUESTION(1.00)[]; RCVD_IN_DNSWL_NONE(0.00)[42.210.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-1.21)[ipnet: 209.85.128.0/17(-3.86), asn: 15169(-2.14), country: US(-0.07)]; RCVD_TLS_LAST(0.00)[] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Mar 2019 22:07:44 -0000 In the process of refactoring some of bhyve's NVMe emulation code, I'm noticing there are a handful of mutexes. But this leads me to believe I don't understand how the emulated devices work. Is there a paper / video / code comment / etc. somewhere that explains bhyve's threading model, specifically around the emulated devices? TIA --chuck