From owner-freebsd-virtualization@freebsd.org Sun Dec 3 01:16:24 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AA8BDE6E761 for ; Sun, 3 Dec 2017 01:16:24 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 898213AB1 for ; Sun, 3 Dec 2017 01:16:24 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from Ticonderoga.HML3.ScaleEngine.net (senat1-01.HML3.ScaleEngine.net [209.51.186.5]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id 584D014DFA for ; Sun, 3 Dec 2017 01:16:17 +0000 (UTC) Subject: Re: bhyve uses all available memory during IO-intensive operations To: freebsd-virtualization@freebsd.org References: <59DFCE5F-029F-4585-B0BA-8FABC43357F2@ebureau.com> From: Allan Jude Message-ID: <11e6e55d-9802-c9fc-859c-37c026eaba2b@freebsd.org> Date: Sat, 2 Dec 2017 20:16:16 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <59DFCE5F-029F-4585-B0BA-8FABC43357F2@ebureau.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Dec 2017 01:16:24 -0000 On 12/02/2017 00:23, Dustin Wenz wrote: > I have noticed significant storage amplification for my zvols; that could very well be the reason. I would like to know more about why it happens. > > Since the volblocksize is 512 bytes, I certainly expect extra cpu overhead (and maybe an extra 1k or so worth of checksums for each 128k block in the vm), but how do you get a 10X expansion in stored data? > > What is the recommended zvol block size for a FreeBSD/ZFS guest? Perhaps 4k, to match the most common mass storage sector size? > > - .Dustin > >> On Dec 1, 2017, at 9:18 PM, K. Macy wrote: >> >> One thing to watch out for with chyves if your virtual disk is more >> than 20G is the fact that it uses 512 byte blocks for the zvols it >> creates. I ended up using up 1.4TB only half filling up a 250G zvol. >> Chyves is quick and easy, but it's not exactly production ready. >> >> -M >> >> >> >>> On Thu, Nov 30, 2017 at 3:15 PM, Dustin Wenz wrote: >>> I'm using chyves on FreeBSD 11.1 RELEASE to manage a few VMs (guest OS is also FreeBSD 11.1). Their sole purpose is to house some medium-sized Postgres databases (100-200GB). The host system has 64GB of real memory and 112GB of swap. I have configured each guest to only use 16GB of memory, yet while doing my initial database imports in the VMs, bhyve will quickly grow to use all available system memory and then be killed by the kernel: >>> >>> kernel: swap_pager: I/O error - pageout failed; blkno 1735,size 4096, error 12 >>> kernel: swap_pager: I/O error - pageout failed; blkno 1610,size 4096, error 12 >>> kernel: swap_pager: I/O error - pageout failed; blkno 1763,size 4096, error 12 >>> kernel: pid 41123 (bhyve), uid 0, was killed: out of swap space >>> >>> The OOM condition seems related to doing moderate IO within the VM, though nothing within the VM itself shows high memory usage. This is the chyves config for one of them: >>> >>> bargs -A -H -P -S >>> bhyve_disk_type virtio-blk >>> bhyve_net_type virtio-net >>> bhyveload_flags >>> chyves_guest_version 0300 >>> cpu 4 >>> creation Created on Mon Oct 23 16:17:04 CDT 2017 by chyves v0.2.0 2016/09/11 using __create() >>> loader bhyveload >>> net_ifaces tap51 >>> os default >>> ram 16G >>> rcboot 0 >>> revert_to_snapshot >>> revert_to_snapshot_method off >>> serial nmdm51 >>> template no >>> uuid 8495a130-b837-11e7-b092-0025909a8b56 >>> >>> >>> I've also tried using different bhyve_disk_types, with no improvement. How is it that bhyve can use far more memory that I'm specifying? >>> >>> - .Dustin > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" > Storage amplification usually has to do with ZFS RAID-Z padding. If your ZVOL block size does not make sense with your disk sector size, and RAID-Z level, you can get pretty silly numbers. -- Allan Jude From owner-freebsd-virtualization@freebsd.org Sun Dec 3 01:21:29 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2C53CE6EA11 for ; Sun, 3 Dec 2017 01:21:29 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: from mail-ot0-x232.google.com (mail-ot0-x232.google.com [IPv6:2607:f8b0:4003:c0f::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E14F33CDE; Sun, 3 Dec 2017 01:21:28 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: by mail-ot0-x232.google.com with SMTP id b54so12031990otd.8; Sat, 02 Dec 2017 17:21:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-transfer-encoding; bh=BzA9DPuvJ+cL5IbV/btzQ4k5437smCnh1rMAAkKvbbk=; b=kgYRuSB+MtqzbLoYLaotQz2qcWc5hpxCkpZFJiwo7QWay6klzYF3YZB/kVNQHLFrdo 12hSHWCJ/HZEjgoGXbDoB6fw6XT2cRUPUnXq+l7oQxArQKhiiTcnAfG5cX0WUM3RnHAC 8fcjL9l7vJxfc8iDB5YS5cwtUPzU/JhBaB5NflIa/oSgFgTaPHANGGinDdcRbzz6zCAi +XzQ7okG6KQmZZgFb8V+DKaLDpIsQkmSnHuQu9jm3qZoRmqWew/Zuvro5oLU7B3GNfC8 29m+Lv/aMDXUt0NvqGPmjBVedZCPzHCy11fWKhlIPb4TvqKnEmvXOFLch3bfJSlHi62G caqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-transfer-encoding; bh=BzA9DPuvJ+cL5IbV/btzQ4k5437smCnh1rMAAkKvbbk=; b=YbFDEWVK7I+Vr1ET394+3ZFjoSlbpdLX57IhbJo2BYKSUg1CRowJgaMRmtzhOErPY4 sKnh22BAJVIqciQtN21ebvO9iu45jrIbb9ZkayGPMrfNxQ94ZCZoYujPNvNMr7r5Ao6u KHauMa8Zg4/P6HdN/cgOO3E40bjHB2mOtrRVYz2HhU+jwIYRtjAzLZWPKuIwDebOQWJL GeDGc3rgQwZvS6euHohJiRqFpaOXod71BoqL1ztjPgtpeDSciRCxtuXDoOTu/5q4Futk HrXbB6EKNtthtG3w52HLJ0tc7sWIy8vaGralNMQiekBH1d3Dg0LpAS4NC3kbdwjXD93y BYOw== X-Gm-Message-State: AJaThX4nMguRdxWW5Bi8EOEyRBIpOsSvjjaudJimY/bs9XkZG1oHFW72 0sWgOdGPOf6iGsOa85uV/y0NlAKZPooAdz+DpiwhOw== X-Google-Smtp-Source: AGs4zMYXQZ5tNrCY48ubfUKfkc8XByZFMItyHArt06m3BxX5j2gT7bgLk9J6EeqoiBBqCaJ14ttWD6LWllmU+yc8sCk= X-Received: by 10.157.45.161 with SMTP id g30mr11380674otb.122.1512264087914; Sat, 02 Dec 2017 17:21:27 -0800 (PST) MIME-Version: 1.0 Sender: kmacybsd@gmail.com Received: by 10.157.66.238 with HTTP; Sat, 2 Dec 2017 17:21:27 -0800 (PST) In-Reply-To: <11e6e55d-9802-c9fc-859c-37c026eaba2b@freebsd.org> References: <59DFCE5F-029F-4585-B0BA-8FABC43357F2@ebureau.com> <11e6e55d-9802-c9fc-859c-37c026eaba2b@freebsd.org> From: "K. Macy" Date: Sat, 2 Dec 2017 17:21:27 -0800 X-Google-Sender-Auth: kfEpn_c5IkcAcTkYbTs_tkqshuo Message-ID: Subject: Re: bhyve uses all available memory during IO-intensive operations To: Allan Jude Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Dec 2017 01:21:29 -0000 On Sat, Dec 2, 2017 at 5:16 PM, Allan Jude wrote: > On 12/02/2017 00:23, Dustin Wenz wrote: >> I have noticed significant storage amplification for my zvols; that coul= d very well be the reason. I would like to know more about why it happens. >> >> Since the volblocksize is 512 bytes, I certainly expect extra cpu overhe= ad (and maybe an extra 1k or so worth of checksums for each 128k block in t= he vm), but how do you get a 10X expansion in stored data? >> >> What is the recommended zvol block size for a FreeBSD/ZFS guest? Perhaps= 4k, to match the most common mass storage sector size? >> >> - .Dustin >> >>> On Dec 1, 2017, at 9:18 PM, K. Macy wrote: >>> >>> One thing to watch out for with chyves if your virtual disk is more >>> than 20G is the fact that it uses 512 byte blocks for the zvols it >>> creates. I ended up using up 1.4TB only half filling up a 250G zvol. >>> Chyves is quick and easy, but it's not exactly production ready. >>> >>> -M >>> >>> >>> >>>> On Thu, Nov 30, 2017 at 3:15 PM, Dustin Wenz = wrote: >>>> I'm using chyves on FreeBSD 11.1 RELEASE to manage a few VMs (guest OS= is also FreeBSD 11.1). Their sole purpose is to house some medium-sized Po= stgres databases (100-200GB). The host system has 64GB of real memory and 1= 12GB of swap. I have configured each guest to only use 16GB of memory, yet = while doing my initial database imports in the VMs, bhyve will quickly grow= to use all available system memory and then be killed by the kernel: >>>> >>>> kernel: swap_pager: I/O error - pageout failed; blkno 1735,size= 4096, error 12 >>>> kernel: swap_pager: I/O error - pageout failed; blkno 1610,size= 4096, error 12 >>>> kernel: swap_pager: I/O error - pageout failed; blkno 1763,size= 4096, error 12 >>>> kernel: pid 41123 (bhyve), uid 0, was killed: out of swap space >>>> >>>> The OOM condition seems related to doing moderate IO within the VM, th= ough nothing within the VM itself shows high memory usage. This is the chyv= es config for one of them: >>>> >>>> bargs -A -H -P -S >>>> bhyve_disk_type virtio-blk >>>> bhyve_net_type virtio-net >>>> bhyveload_flags >>>> chyves_guest_version 0300 >>>> cpu 4 >>>> creation Created on Mon Oct 23 16:17:04 CDT 2= 017 by chyves v0.2.0 2016/09/11 using __create() >>>> loader bhyveload >>>> net_ifaces tap51 >>>> os default >>>> ram 16G >>>> rcboot 0 >>>> revert_to_snapshot >>>> revert_to_snapshot_method off >>>> serial nmdm51 >>>> template no >>>> uuid 8495a130-b837-11e7-b092-0025909a8b56 >>>> >>>> >>>> I've also tried using different bhyve_disk_types, with no improvement.= How is it that bhyve can use far more memory that I'm specifying? >>>> >>>> - .Dustin >> _______________________________________________ >> freebsd-virtualization@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >> To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@fre= ebsd.org" >> > > Storage amplification usually has to do with ZFS RAID-Z padding. If your > ZVOL block size does not make sense with your disk sector size, and > RAID-Z level, you can get pretty silly numbers. That's not what I'm talking about here. If your volblocksize is too small you end up using (vastly) more space for indirect blocks than data blocks. -M From owner-freebsd-virtualization@freebsd.org Sun Dec 3 03:59:00 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AE420DEC767 for ; Sun, 3 Dec 2017 03:59:00 +0000 (UTC) (envelope-from jtubnor@gmail.com) Received: from mail-io0-x22e.google.com (mail-io0-x22e.google.com [IPv6:2607:f8b0:4001:c06::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6DE7368F4F; Sun, 3 Dec 2017 03:59:00 +0000 (UTC) (envelope-from jtubnor@gmail.com) Received: by mail-io0-x22e.google.com with SMTP id d21so15246478ioe.7; Sat, 02 Dec 2017 19:59:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=VhQjoas4o8I8RSb+aCyaSr5V4XH7Lm/07xboJQC3Pbw=; b=eiUugssLMckzXZFflJ+ormgh7Q2roVM9RZyF58VTM6T4sqbUYRUvQRuzqjl9dC7BNU ihKq8cB7h0lH86AR5Dc5Lvs5UxamqrwLAgpWXY8ZTRN4TyrzTqZFm11ldOrz/5o2DsLW YT1TEMK1amOJphbGkgDodPwmGmD3g4SFo0r9CJgNimf9T2O+JCOBKo8ZXHRzlfM8lV3t kSuW1vfLmH8bbgRc8QUziQhZkmIeQ+84ormyy0QCFECVLiE+KIUq/R18/k+ZaPt0qZFL O0S7dreIEP6ZVcB/plMeurh831WOP8kQ/xMRXXv0nlGKj2Bz9V6YnpS8XPCQk1s1KWJ2 3R6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=VhQjoas4o8I8RSb+aCyaSr5V4XH7Lm/07xboJQC3Pbw=; b=aJn8x2PzcVqeblngNnz/CtJVlYhHLMRKKnL/udO0VD+8LgVGKBJF+zRGBxkMN4XOFZ SC4SgIA2JLEEgWG1HhFa+SK3Caqel++wLR+Fx8+KnzzR2qS2/ajafnXWaUhXNUuoxOHw FGSv4bD4BCPZcuyrVvXa3eBXY3kJ69RQt8rXdc7MeLifg71VltxyGJagqm2WqFTtHWAZ 749cmkp7IV057wB6B1XSkNnn6aHwl3sbIjCuiNAv0NWzP1S+OflQGX4dL2QlR1oRKHgP htFhJrd4tl4X9uL4m9dMSsQAMAN74Oz79pRWmGIehj4yZ3M+e1RS6oO/PGfOstrWN67I +BIg== X-Gm-Message-State: AJaThX7uw9OxgSGlCBafETaUNjq90+e90TEyOQERTLnrx13k+qygUkNa DDzLmWfICCiQjWPoad86vo7lS5G0n3QafZQNPnCDRg== X-Google-Smtp-Source: AGs4zMaD0JGnoaikM8HyMt7Osljw8/Erj6OjWl7+rp4zUUesFDzyysTfTw7Omguz/9ELmd5DyIG/+hmJL3gosu3FhsA= X-Received: by 10.107.84.1 with SMTP id i1mr18771142iob.261.1512273539650; Sat, 02 Dec 2017 19:58:59 -0800 (PST) MIME-Version: 1.0 Received: by 10.2.69.213 with HTTP; Sat, 2 Dec 2017 19:58:58 -0800 (PST) Received: by 10.2.69.213 with HTTP; Sat, 2 Dec 2017 19:58:58 -0800 (PST) In-Reply-To: References: <59DFCE5F-029F-4585-B0BA-8FABC43357F2@ebureau.com> <11e6e55d-9802-c9fc-859c-37c026eaba2b@freebsd.org> From: Jason Tubnor Date: Sun, 3 Dec 2017 14:58:58 +1100 Message-ID: Subject: Re: bhyve uses all available memory during IO-intensive operations To: "K. Macy" Cc: Allan Jude , freebsd-virtualization@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Dec 2017 03:59:00 -0000 On 3 Dec. 2017 12:21, "K. Macy" wrote: > > Storage amplification usually has to do with ZFS RAID-Z padding. If your > ZVOL block size does not make sense with your disk sector size, and > RAID-Z level, you can get pretty silly numbers. That's not what I'm talking about here. If your volblocksize is too small you end up using (vastly) more space for indirect blocks than data blocks. -M This. I experienced this with chyves and bumped the block size back to 8k. Fixed the issue. Be careful selecting 8k for Windows guests though. Anything over 4k and you can't use any mssql. With some tweaks, I've found vm-bhyve better suited if you can do totally UEFI. I'll be putting up some templates shortly. Cheers, Jason From owner-freebsd-virtualization@freebsd.org Sun Dec 3 04:53:04 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A83C6DEDE17 for ; Sun, 3 Dec 2017 04:53:04 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 87AA36A611; Sun, 3 Dec 2017 04:53:04 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id EF0B913D7C; Sun, 3 Dec 2017 04:53:02 +0000 (UTC) Subject: Re: bhyve uses all available memory during IO-intensive operations To: "K. Macy" Cc: "freebsd-virtualization@freebsd.org" References: <59DFCE5F-029F-4585-B0BA-8FABC43357F2@ebureau.com> <11e6e55d-9802-c9fc-859c-37c026eaba2b@freebsd.org> From: Allan Jude Message-ID: <571ab0b4-ec6c-2bc4-438b-d3dce35cd775@freebsd.org> Date: Sat, 2 Dec 2017 23:53:02 -0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Dec 2017 04:53:04 -0000 On 2017-12-02 20:21, K. Macy wrote: > On Sat, Dec 2, 2017 at 5:16 PM, Allan Jude wrote: >> On 12/02/2017 00:23, Dustin Wenz wrote: >>> I have noticed significant storage amplification for my zvols; that could very well be the reason. I would like to know more about why it happens. >>> >>> Since the volblocksize is 512 bytes, I certainly expect extra cpu overhead (and maybe an extra 1k or so worth of checksums for each 128k block in the vm), but how do you get a 10X expansion in stored data? >>> >>> What is the recommended zvol block size for a FreeBSD/ZFS guest? Perhaps 4k, to match the most common mass storage sector size? >>> >>> - .Dustin >>> >>>> On Dec 1, 2017, at 9:18 PM, K. Macy wrote: >>>> >>>> One thing to watch out for with chyves if your virtual disk is more >>>> than 20G is the fact that it uses 512 byte blocks for the zvols it >>>> creates. I ended up using up 1.4TB only half filling up a 250G zvol. >>>> Chyves is quick and easy, but it's not exactly production ready. >>>> >>>> -M >>>> >>>> >>>> >>>>> On Thu, Nov 30, 2017 at 3:15 PM, Dustin Wenz wrote: >>>>> I'm using chyves on FreeBSD 11.1 RELEASE to manage a few VMs (guest OS is also FreeBSD 11.1). Their sole purpose is to house some medium-sized Postgres databases (100-200GB). The host system has 64GB of real memory and 112GB of swap. I have configured each guest to only use 16GB of memory, yet while doing my initial database imports in the VMs, bhyve will quickly grow to use all available system memory and then be killed by the kernel: >>>>> >>>>> kernel: swap_pager: I/O error - pageout failed; blkno 1735,size 4096, error 12 >>>>> kernel: swap_pager: I/O error - pageout failed; blkno 1610,size 4096, error 12 >>>>> kernel: swap_pager: I/O error - pageout failed; blkno 1763,size 4096, error 12 >>>>> kernel: pid 41123 (bhyve), uid 0, was killed: out of swap space >>>>> >>>>> The OOM condition seems related to doing moderate IO within the VM, though nothing within the VM itself shows high memory usage. This is the chyves config for one of them: >>>>> >>>>> bargs -A -H -P -S >>>>> bhyve_disk_type virtio-blk >>>>> bhyve_net_type virtio-net >>>>> bhyveload_flags >>>>> chyves_guest_version 0300 >>>>> cpu 4 >>>>> creation Created on Mon Oct 23 16:17:04 CDT 2017 by chyves v0.2.0 2016/09/11 using __create() >>>>> loader bhyveload >>>>> net_ifaces tap51 >>>>> os default >>>>> ram 16G >>>>> rcboot 0 >>>>> revert_to_snapshot >>>>> revert_to_snapshot_method off >>>>> serial nmdm51 >>>>> template no >>>>> uuid 8495a130-b837-11e7-b092-0025909a8b56 >>>>> >>>>> >>>>> I've also tried using different bhyve_disk_types, with no improvement. How is it that bhyve can use far more memory that I'm specifying? >>>>> >>>>> - .Dustin >>> _______________________________________________ >>> freebsd-virtualization@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >>> To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" >>> >> >> Storage amplification usually has to do with ZFS RAID-Z padding. If your >> ZVOL block size does not make sense with your disk sector size, and >> RAID-Z level, you can get pretty silly numbers. > > That's not what I'm talking about here. If your volblocksize is too > small you end up using (vastly) more space for indirect blocks than > data blocks. > > -M > In addition, if you have say, 4k sectors, and a RAID-Z2, it means every allocation of 4k or less, requires 12k of disk space. Allocations of 8k are worse in this case, since all allocations must be in units of 1+p, where p is the parity level. So allocating 8kb of space (2x 4k sectors), plus 2x 4k parity sectors = 4 sectors, Rounded up the to the next multiple of 3 is 6. That means 8k of data took: 8kb for data + 8kb for parity + 8kb for padding = 24kb of space. If you were using RAID-Z1, it would have been just 12kb (8kb data, 4kb parity, 0kb padding) Or if you used 16kb record size on the zvol: 4 sectors data, 2 sectors parity = 6, which is a multiple of 3, so no padding required. -- Allan Jude From owner-freebsd-virtualization@freebsd.org Mon Dec 4 13:15:01 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BA672DFC747 for ; Mon, 4 Dec 2017 13:15:01 +0000 (UTC) (envelope-from artemrts@ukr.net) Received: from frv191.fwdcdn.com (frv191.fwdcdn.com [212.42.77.191]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7D159681CF for ; Mon, 4 Dec 2017 13:15:01 +0000 (UTC) (envelope-from artemrts@ukr.net) Received: from frv198.fwdcdn.com ([212.42.77.198]) by frv191.fwdcdn.com with esmtp ID 1eLpvR-000N5w-Jw for freebsd-virtualization@freebsd.org; Mon, 04 Dec 2017 14:32:45 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=ffe; h=Content-Transfer-Encoding:Content-Type:MIME-Version:Message-Id:To: Subject:From:Date:Sender:Reply-To:Cc:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: In-Reply-To:References:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=E0vHXjl3082tF6De+2rOWpONGiylITADid3XpJxvmOs=; b=L2MKWfYPpxxM66tpjGDj0BgKnC WP3kJBoVwWvyZ7Ch+hVED256G2Qb8cg8LS3OekxFAwQP9pxF8lJXoQnRJ3lEUKVPZfDeIs1srR4we Oj/2+niHM89LDmTWucLSZq4KYxLjv+FUZek3nXMZ5MuJ57k8PGwVx2CgVM2J690XdY7Q=; Received: from [10.10.10.52] (helo=frv52.fwdcdn.com) by frv198.fwdcdn.com with smtp ID 1eLpvI-000MkV-Us for freebsd-virtualization@freebsd.org; Mon, 04 Dec 2017 14:32:36 +0200 Date: Mon, 04 Dec 2017 14:32:36 +0200 From: wishmaster Subject: Bhyve. Unable install Windows 7/Server 2008R2 To: freebsd-virtualization@freebsd.org X-Mailer: mail.ukr.net 5.0 Message-Id: <1512390476.593482033.ytqgxtgt@frv52.fwdcdn.com> Received: from artemrts@ukr.net by frv52.fwdcdn.com; Mon, 04 Dec 2017 14:32:36 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: binary X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Dec 2017 13:15:01 -0000 Hi, guys, what happened with bhyve after 11.1-RELEASE. I am unable install neither Windows 7 nor Windows Server 2008R2. I have attempted with HDD as raw file, ZFS dev - without success. root@XXX: uname -a FreeBSD XXX 11.1-STABLE FreeBSD 11.1-STABLE #0 r325503: Tue Nov 7 13:38:44 EET 2017 wishmaster@XXX:/usr/obj/usr/src/sys/MY amd64 Cheers, Vit From owner-freebsd-virtualization@freebsd.org Mon Dec 4 13:23:12 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 882F6DFCACB for ; Mon, 4 Dec 2017 13:23:12 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 27A7168595 for ; Mon, 4 Dec 2017 13:23:11 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mh0.gentlemail.de (ezra.dcm1.omnilan.net [78.138.80.135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id vB4DN9an080306; Mon, 4 Dec 2017 14:23:09 +0100 (CET) (envelope-from freebsd@omnilan.de) Received: from titan.inop.mo1.omnilan.net (s1.omnilan.de [217.91.127.234]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id 070EA21A; Mon, 4 Dec 2017 14:23:08 +0100 (CET) Message-ID: <5A254C3C.3030809@omnilan.de> Date: Mon, 04 Dec 2017 14:23:08 +0100 From: Harry Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: wishmaster CC: freebsd-virtualization@freebsd.org Subject: Re: Bhyve. Unable install Windows 7/Server 2008R2 References: <1512390476.593482033.ytqgxtgt@frv52.fwdcdn.com> In-Reply-To: <1512390476.593482033.ytqgxtgt@frv52.fwdcdn.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: ACL 129 matched, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [78.138.80.130]); Mon, 04 Dec 2017 14:23:09 +0100 (CET) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: 78.138.80.135; Sender-helo: mh0.gentlemail.de; ) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Dec 2017 13:23:12 -0000 Bezglich wishmaster's Nachricht vom 04.12.2017 13:32 (localtime): > Hi, > > guys, what happened with bhyve after 11.1-RELEASE. I am unable install neither Windows 7 nor Windows Server 2008R2. > I have attempted with HDD as raw file, ZFS dev - without success. > > root@XXX: uname -a > FreeBSD XXX 11.1-STABLE FreeBSD 11.1-STABLE #0 r325503: Tue Nov 7 13:38:44 EET 2017 wishmaster@XXX:/usr/obj/usr/src/sys/MY amd64 Since you don't tell us what error you get from the guest setup program, I can only guess that you are using a ZFS volume with a block size of 8k (default). My next guess is that you use BHYVE_UEFI.fd for bootrom. The latter can't boot from 8k drives, Make sure you define something like this: -s 3,virtio-blk,/dev/zvol/hostPool/bhyveVOL/sys/guest1,sectorsize=512/4096 If that power guess didn't hit, you need to provide some details (boot method at very first along with some kind of error description). -harry From owner-freebsd-virtualization@freebsd.org Mon Dec 4 23:19:58 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9F743E6DE22 for ; Mon, 4 Dec 2017 23:19:58 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from internet06.ebureau.com (internet06.ebureau.com [65.127.24.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "internet06.ebureau.com", Issuer "internet06.ebureau.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 79A2D7DE6B for ; Mon, 4 Dec 2017 23:19:58 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from localhost (localhost [127.0.0.1]) by internet06.ebureau.com (Postfix) with ESMTP id DABB08464048 for ; Mon, 4 Dec 2017 17:19:50 -0600 (CST) X-Virus-Scanned: amavisd-new at mydomain = ebureau.com Received: from internet06.ebureau.com ([127.0.0.1]) by localhost (internet06.ebureau.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OXAlnL-ijeGP for ; Mon, 4 Dec 2017 17:19:50 -0600 (CST) Received: from square.office.ebureau.com (unknown [10.10.21.22]) by internet06.ebureau.com (Postfix) with ESMTPSA id 9271E846403B for ; Mon, 4 Dec 2017 17:19:50 -0600 (CST) From: Dustin Wenz Content-Type: multipart/signed; boundary="Apple-Mail=_85D4250D-350C-413F-9A10-3C65744412EA"; protocol="application/pkcs7-signature"; micalg=sha1 Mime-Version: 1.0 (Mac OS X Mail 11.1 \(3445.4.7\)) Subject: Storage overhead on zvols Message-Id: Date: Mon, 4 Dec 2017 17:19:49 -0600 To: freebsd-virtualization@freebsd.org X-Mailer: Apple Mail (2.3445.4.7) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Dec 2017 23:19:58 -0000 --Apple-Mail=_85D4250D-350C-413F-9A10-3C65744412EA Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii I'm starting a new thread based on the previous discussion in "bhyve = uses all available memory during IO-intensive operations" relating to = size inflation of bhyve data stored on zvols. I've done some = experimenting with this, and I think it will be useful for others. The zvols listed here were created with this command: zfs create -o volmode=3Ddev -o volblocksize=3DXk -V 30g = vm00/chyves/guests/myguest/diskY The zvols were created on a raidz1 pool of four disks. For each zvol, I = created a basic zfs filesystem in the guest using all default tuning = (128k recordsize, etc). I then copied the same 8.2GB dataset to each = filesystem. volblocksize size amplification 512B 11.7x 4k 1.45x 8k 1.45x 16k 1.5x 32k 1.65x 64k 1x 128k 1x The worst case is with a 512B volblocksize, where the space used is more = than 11 times the size of the data stored within the guest. The size = efficiency gains are non-linear as I continue from 4k and double the = block sizes; 32k blocks being the second-worst. The amount of wasted = space was minimized by using 64k and 128k blocks. It would appear that 64k is a good choice for volblocksize if you are = using a zvol to back your VM, and the VM is using the virtual device for = a zpool. Incidentally, I believe this is the default when creating VMs = in FreeNAS. - .Dustin --Apple-Mail=_85D4250D-350C-413F-9A10-3C65744412EA Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIEzDCCBMgw ggOwoAMCAQICAUEwDQYJKoZIhvcNAQELBQAwgZkxCzAJBgNVBAYTAlVTMRIwEAYDVQQIEwlNaW5u ZXNvdGExFDASBgNVBAcTC1NhaW50IENsb3VkMRAwDgYDVQQKEwdlQnVyZWF1MRQwEgYDVQQLEwtJ bnRlZ3JhdGlvbjEUMBIGA1UEAxMLZWJ1cmVhdS5jb20xIjAgBgkqhkiG9w0BCQEWE3N1cHBvcnRA ZWJ1cmVhdS5jb20wHhcNMTcwNTA1MTYxNjE1WhcNMjcwNTAzMTYxNjE1WjBKMQswCQYDVQQGEwJV UzEUMBIGA1UEAwwLRHVzdGluIFdlbnoxJTAjBgkqhkiG9w0BCQEWFmR1c3RpbndlbnpAZWJ1cmVh dS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCQ/HJXe7JhUaexqEaxCNVifsue sUMgohgXLmi5YIcbAWhdxTr1PUzKYkeTkL9sYNjXU6uaI2tZMF3hA9gcFjxQIfkKSo31MrYOgMdU xQe0Q+t6Vd4pGAmtDQDwhAsrNGccADp3Yjy4eBtVfkDGdzz1Y8Lbc684TPFcW7i9+U/dDaXlcxeq fyDqiHZ5y8Lp/1M2Ot/Rz7eikJZTAuHOWKs/PEiJIM2JHuhPyNy+mL2oqEWeOcEsKMNzgn7HVt4k Xz2irBAG+cj4WAxWs418l46EEXgur4PvhBXZMl0LJg0TyaxOHbsUam4R4tbKnaZ3HhRkg79k2Had sb6DKbnCw9/1AgMBAAGjggFnMIIBYzAJBgNVHRMEAjAAMAsGA1UdDwQEAwIE8DAnBgNVHSUEIDAe BggrBgEFBQcDAQYIKwYBBQUHAwIGCCsGAQUFBwMEMB0GA1UdDgQWBBTLi/8HUHpbBEt9OtPqQoax AmpaNDCBzgYDVR0jBIHGMIHDgBRnpZeXB5rQYLgsUKqiiBcLIHyu6aGBn6SBnDCBmTELMAkGA1UE BhMCVVMxEjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAOBgNVBAoT B2VCdXJlYXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNvbTEiMCAG CSqGSIb3DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbYIJAMwZcjAWAsWXMDAGCWCGSAGG+EIBBAQj FiFodHRwOi8vd3d3LmVidXJlYXUuY29tL2NhLWNybC5wZW0wDQYJKoZIhvcNAQELBQADggEBAHbO qVdB9raUKXCgZRA/nES5a60dlIaGnIlpgz+Y3SjFt0bcJxoUYhIzumBHk9yjyP4M1DubOphkQpJ4 LNZbAS01cjCxjnC0ZUq5V3FCeaDwrn1qPY+QJGoZPLlhWdJUNu17OpnR7ZfBWlp3/pRhvNU5PCbJ nmF7rnvsqxUFq9oeiV3SmqBux5lwJ7p2Uss5SHSW6g17K/KdTMK1roQr/+rWpxp2233qddDrLpOE xGRlvhEqSa/IZbGC9oiYmsiaG1PefQkadoob5IMIS5/MDpWHUgSHqAj1V/LwcCx0rbt73SazGMND EzHVWhsj+khepB/MG5QGfWP23IGFmvQYWWcxggOQMIIDjAIBATCBnzCBmTELMAkGA1UEBhMCVVMx EjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAOBgNVBAoTB2VCdXJl YXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNvbTEiMCAGCSqGSIb3 DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbQIBQTAJBgUrDgMCGgUAoIIBxTAYBgkqhkiG9w0BCQMx CwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNzEyMDQyMzE5NTBaMCMGCSqGSIb3DQEJBDEW BBQuj8Vux5VGF1eV5QvzsJ2iiC5v5TCBsAYJKwYBBAGCNxAEMYGiMIGfMIGZMQswCQYDVQQGEwJV UzESMBAGA1UECBMJTWlubmVzb3RhMRQwEgYDVQQHEwtTYWludCBDbG91ZDEQMA4GA1UEChMHZUJ1 cmVhdTEUMBIGA1UECxMLSW50ZWdyYXRpb24xFDASBgNVBAMTC2VidXJlYXUuY29tMSIwIAYJKoZI hvcNAQkBFhNzdXBwb3J0QGVidXJlYXUuY29tAgFBMIGyBgsqhkiG9w0BCRACCzGBoqCBnzCBmTEL MAkGA1UEBhMCVVMxEjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAO BgNVBAoTB2VCdXJlYXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNv bTEiMCAGCSqGSIb3DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbQIBQTANBgkqhkiG9w0BAQEFAASC AQCJO3mtHogCOS/EAfrWZu6MwXrvP+4waW/G2r4W9k6qI9hAVJGwbQjAxdZSCTJzjnYg97jBc93A aHO1cb2uo3fKp+T6s29ainCvFfe7uAC+MHwLJ5qPHXk9lQvGpyyA/Fg3FlOzOtbUP14epJ3d317m IajFIYG0xL1oiYTjqzWALeCYq/O45P71hcTS1cIvQyK+10L39xe2RIjr41DyL7UowOZWvrlsHixp wa5nO59uroXbtcWZka7/TTMveCkpZl0MalQKPdlea+BAuuGiLYib7S9T73ggV+xK0S5pBRNyUgdp Bk7fsL6yL6e5IR9xbNn8jRXzm7xC4XprnWPIevd7AAAAAAAA --Apple-Mail=_85D4250D-350C-413F-9A10-3C65744412EA-- From owner-freebsd-virtualization@freebsd.org Mon Dec 4 23:22:11 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A443BE6E0F5 for ; Mon, 4 Dec 2017 23:22:11 +0000 (UTC) (envelope-from dmarquess@gmail.com) Received: from mail-qt0-x22e.google.com (mail-qt0-x22e.google.com [IPv6:2607:f8b0:400d:c0d::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 523C07E250 for ; Mon, 4 Dec 2017 23:22:11 +0000 (UTC) (envelope-from dmarquess@gmail.com) Received: by mail-qt0-x22e.google.com with SMTP id f2so1794456qtj.4 for ; Mon, 04 Dec 2017 15:22:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=s6SKejP6j0k+c1jRM6QVlLpAqfCTxXweukS67+JhF+Y=; b=XNk4jHJj+cLWcQ/+K7D3bUWiw1+vf6wCUMQogZdIV8pYibTd3YyM0klzQQY2OPdjlG bT02vBKXrqiCLtmglRJ1k3Nuhn0g0PTYZ1cX/k4484iHU1fugGmbiilJV5oWxT9U7P6S WYexnoHcLPh6TuX4ftAC+Z26J7HbMGTAogVPDsd/KlzrPBEbkGDdXJS8787LWKSgi0MY dmsDXikZzKiu9URqZtgWa8p7N7OD9DvdYNJJD/P9/sKE0f1byxLDm9SBG/8oBiPG7hGz bGgsV8Beu12gg6KFi/35PvryXbFol6lyDCkpZd9PQNGSEostxA/tjM1uM//KZyTAYhQe bAAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=s6SKejP6j0k+c1jRM6QVlLpAqfCTxXweukS67+JhF+Y=; b=ahXTiEYpTH9AwtsB7Y3ZCqol4IIT3c3fMefMH9ZFyjs7UXcHuL4tNx1RIZ/Y5WUhLH vJNN96FtxhrNntL6FXCDJmmIuw5PYPyurwWMHz2XgRJ3dKFGxAsXIiPJdXb/evvZ7Nzr OArY+WSq/KQjfSdMfXX9adJs8P48cOGCPU6Ne+7UzNq14QY/JgIhqHAcN8eiPwFkeXW9 48jq6kSPzQepqTslbtDGK/e6JEKAjMToeZKdajzAs4oFxKrbFFhZCLCQPfAkjBgOh5bg RHBoeF3hNiFybOLzOEBXj9hsAfTIXRb4MkpNDKWnh1akoEbM/J3eJcLzu4/w2xWXOLVb yl3w== X-Gm-Message-State: AKGB3mIwnwfJPsNDxzYeBPZZxYM46QtCPPgRJg0v4StAxD7St4aXNtws ixAGCd71SuwazL0p0W9FT90ybxuKpu6aQIPjsl4= X-Google-Smtp-Source: AGs4zMahjVWpCBwvAuKa8qaT6sBcIakk37bl+uc2w8p3fGAPvar9Jz9rgjJ6KsOnV9c3zpxmW8i4k83yvRAcj5zilYo= X-Received: by 10.200.47.220 with SMTP id m28mr297651qta.146.1512429730218; Mon, 04 Dec 2017 15:22:10 -0800 (PST) MIME-Version: 1.0 Received: by 10.140.105.53 with HTTP; Mon, 4 Dec 2017 15:22:09 -0800 (PST) In-Reply-To: References: From: Dustin Marquess Date: Mon, 4 Dec 2017 17:22:09 -0600 Message-ID: Subject: Re: Storage overhead on zvols To: Dustin Wenz Cc: FreeBSD virtualization Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Dec 2017 23:22:11 -0000 I doubt it's best practice, and I'm sure I'm just crazy for doing it, but personally I try and match the ZVOL blocksize to whatever the underlaying filesystem's blocksize is. To me that just makes the most logical sense. -Dustin On Mon, Dec 4, 2017 at 5:19 PM, Dustin Wenz wrote: > I'm starting a new thread based on the previous discussion in "bhyve uses= all available memory during IO-intensive operations" relating to size infl= ation of bhyve data stored on zvols. I've done some experimenting with this= , and I think it will be useful for others. > > The zvols listed here were created with this command: > > zfs create -o volmode=3Ddev -o volblocksize=3DXk -V 30g vm00/chyv= es/guests/myguest/diskY > > The zvols were created on a raidz1 pool of four disks. For each zvol, I c= reated a basic zfs filesystem in the guest using all default tuning (128k r= ecordsize, etc). I then copied the same 8.2GB dataset to each filesystem. > > volblocksize size amplification > > 512B 11.7x > 4k 1.45x > 8k 1.45x > 16k 1.5x > 32k 1.65x > 64k 1x > 128k 1x > > The worst case is with a 512B volblocksize, where the space used is more = than 11 times the size of the data stored within the guest. The size effici= ency gains are non-linear as I continue from 4k and double the block sizes;= 32k blocks being the second-worst. The amount of wasted space was minimize= d by using 64k and 128k blocks. > > It would appear that 64k is a good choice for volblocksize if you are usi= ng a zvol to back your VM, and the VM is using the virtual device for a zpo= ol. Incidentally, I believe this is the default when creating VMs in FreeNA= S. > > - .Dustin > From owner-freebsd-virtualization@freebsd.org Mon Dec 4 23:39:34 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 513C5E6E56E for ; Mon, 4 Dec 2017 23:39:34 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2611A7E91A for ; Mon, 4 Dec 2017 23:39:33 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from Ticonderoga.HML3.ScaleEngine.net (senat1-01.HML3.ScaleEngine.net [209.51.186.5]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id 1AC53148AB for ; Mon, 4 Dec 2017 23:39:32 +0000 (UTC) Subject: Re: Storage overhead on zvols To: freebsd-virtualization@freebsd.org References: From: Allan Jude Message-ID: <4185ed66-3f99-30f1-c6b7-b9ee262b7e59@freebsd.org> Date: Mon, 4 Dec 2017 18:39:31 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Dec 2017 23:39:34 -0000 On 12/04/2017 18:19, Dustin Wenz wrote: > I'm starting a new thread based on the previous discussion in "bhyve uses all available memory during IO-intensive operations" relating to size inflation of bhyve data stored on zvols. I've done some experimenting with this, and I think it will be useful for others. > > The zvols listed here were created with this command: > > zfs create -o volmode=dev -o volblocksize=Xk -V 30g vm00/chyves/guests/myguest/diskY > > The zvols were created on a raidz1 pool of four disks. For each zvol, I created a basic zfs filesystem in the guest using all default tuning (128k recordsize, etc). I then copied the same 8.2GB dataset to each filesystem. > > volblocksize size amplification > > 512B 11.7x > 4k 1.45x > 8k 1.45x > 16k 1.5x > 32k 1.65x > 64k 1x > 128k 1x > > The worst case is with a 512B volblocksize, where the space used is more than 11 times the size of the data stored within the guest. The size efficiency gains are non-linear as I continue from 4k and double the block sizes; 32k blocks being the second-worst. The amount of wasted space was minimized by using 64k and 128k blocks. > > It would appear that 64k is a good choice for volblocksize if you are using a zvol to back your VM, and the VM is using the virtual device for a zpool. Incidentally, I believe this is the default when creating VMs in FreeNAS. > > - .Dustin > As I explained a bit in the other thread, this depends a lot on your VDEV configuration. Allocations on RAID-Z* must be padded out to a multiple of 1+p (where p is the parity level) So on RAID-Z1, all allocations must be divisible by 2. Of course any record size less than 4k, on drives with 4k sectors would be rounded up as well. So, with recordsize=512, you would end up using: 4k for data, 4k for parity, with a waste factor of almost 16x. 4k is a bit better Z1: 1 data + 1 parity + 0 padding = 2x Z2: 1 data + 2 parity + 0 padding = 3x Z3: 1 data + 3 parity + 0 padding = 4x 8k can be worse, where the RAID-Z padding comes into play: Z1: 2 data + 1 parity + 1 padding = 2x (expect 1.5x) Z2: 2 data + 2 parity + 2 padding = 3x (expect 2x) Z3: 2 data + 3 parity + 3 padding = 4x (expect 2.x5) Finally, all of these nice even numbers can be be thrown out once you enable compression, and some blocks will compress better than others. An 8k record that fits into one 4k sector, etc. Also consider that 'zfs' commands show size after its calculations of what the expected raid-z parity space consumption will be, but does not consider losses to padding. Whereas numbers given by the 'zpool' command, are raw actual storage. -- Allan Jude From owner-freebsd-virtualization@freebsd.org Tue Dec 5 03:37:45 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 29C1BDEF11C for ; Tue, 5 Dec 2017 03:37:45 +0000 (UTC) (envelope-from amvandemore@gmail.com) Received: from mail-it0-x230.google.com (mail-it0-x230.google.com [IPv6:2607:f8b0:4001:c0b::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E4A3167D48 for ; Tue, 5 Dec 2017 03:37:44 +0000 (UTC) (envelope-from amvandemore@gmail.com) Received: by mail-it0-x230.google.com with SMTP id d137so11968133itc.2 for ; Mon, 04 Dec 2017 19:37:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=449o/xOWRPCMshk98BOpBaD6bOv99GrlC+dRCaAxOOU=; b=djIPqvqoxlE9gRpzVtYMB2JcI1rMg32nQyQ1Dvl8nE/XDnEKz00TBMnOuE5QOyhp2o CpJo0ErthZBC13yPb4nfjZX2ABTl/HkQfLJ/wmiO0DF6FAWXPDYjZwvCbEFiwNMOdGhf FJAxZsnQDOaUI6Of7LOQLGz/9qyJCgM15f3MBqW4smmrkqQoTfHilO2rUvxwyx7r1yRv WBwdVSNu67AhqsNu2VXTw6rBlgJ+y14YRi+V/j3uD5Cl7axKOBb5si6s59ipAi7NN4gC dEnFkJkEvh43iukDW4zbMw0Lb22HTU+2ekJ+vEWBL18430reB29xtKNAhB9iVKQRls0v dc+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=449o/xOWRPCMshk98BOpBaD6bOv99GrlC+dRCaAxOOU=; b=SPkbp3R3kMDmZsWqnK1dnB8UIdjkicH1UJQI3sJvSZefG1r0ZsFsBZQ1d2cZUAn0HC V3nN+8Bgd9n2KP/24+Mpt5r+kDO1tIqee6C3yk+GHKevPV+wXm2DISsOiTwI9uJG41Q3 V8Rz/Kg065yqUfiQ5raR+l9DhX+omj2Jydq58kS5d9qb1K5nGUAH2lRDb6P/vC7pwZQJ yW4d4PgjoGU5GOkYAjYZ/ot5SwrUiejBAGD9eOHAB+GPf9N6iGeP2u4sF282kk+DW+LE qrwZ3qKJOYe9isKkzACY9toUtwIgAVQXAxAdvVe1HT/6TIKS/lCcH5FURgaPQVyHVMSe nvIg== X-Gm-Message-State: AKGB3mKCBBkJxnAfKBSRu/InTEdgGQvXu3jAtaeki3dY3rbU85gp4fmd 3T9Uy/UgkxAv8IwpUT082B2679DJ2mM9os+6/1cSGw== X-Google-Smtp-Source: AGs4zMarholwikG5ftPjCAIlWFrSI6uHvLxbnuDY3ytliDGpWY115s26ZakTYDt55/kbo1Do+yzqjqeLcWbYvDjEwUM= X-Received: by 10.36.225.136 with SMTP id n130mr7325728ith.146.1512445063811; Mon, 04 Dec 2017 19:37:43 -0800 (PST) MIME-Version: 1.0 Received: by 10.2.138.114 with HTTP; Mon, 4 Dec 2017 19:37:43 -0800 (PST) In-Reply-To: References: From: Adam Vande More Date: Mon, 4 Dec 2017 21:37:43 -0600 Message-ID: Subject: Re: Storage overhead on zvols To: Dustin Wenz Cc: FreeBSD virtualization Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Dec 2017 03:37:45 -0000 On Mon, Dec 4, 2017 at 5:19 PM, Dustin Wenz wrote: > I'm starting a new thread based on the previous discussion in "bhyve uses > all available memory during IO-intensive operations" relating to size > inflation of bhyve data stored on zvols. I've done some experimenting with > this, and I think it will be useful for others. > > The zvols listed here were created with this command: > > zfs create -o volmode=dev -o volblocksize=Xk -V 30g > vm00/chyves/guests/myguest/diskY > > The zvols were created on a raidz1 pool of four disks. For each zvol, I > created a basic zfs filesystem in the guest using all default tuning (128k > recordsize, etc). I then copied the same 8.2GB dataset to each filesystem. > > volblocksize size amplification > > 512B 11.7x > 4k 1.45x > 8k 1.45x > 16k 1.5x > 32k 1.65x > 64k 1x > 128k 1x > > The worst case is with a 512B volblocksize, where the space used is more > than 11 times the size of the data stored within the guest. The size > efficiency gains are non-linear as I continue from 4k and double the block > sizes; 32k blocks being the second-worst. The amount of wasted space was > minimized by using 64k and 128k blocks. > > It would appear that 64k is a good choice for volblocksize if you are > using a zvol to back your VM, and the VM is using the virtual device for a > zpool. Incidentally, I believe this is the default when creating VMs in > FreeNAS. > I'm not sure what your purpose is behind the posting, but if its simply a "why this behavior" you can find more detail here as well as some calculation leg work: https://www.delphix.com/blog/delphix-engineering/zfs-raidz-stripe-width-or-how-i-learned-stop-worrying-and-love-raidz -- Adam From owner-freebsd-virtualization@freebsd.org Tue Dec 5 06:33:21 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 68E48DF7AE6 for ; Tue, 5 Dec 2017 06:33:21 +0000 (UTC) (envelope-from artemrts@ukr.net) Received: from frv190.fwdcdn.com (frv190.fwdcdn.com [212.42.77.190]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2BF686D092 for ; Tue, 5 Dec 2017 06:33:20 +0000 (UTC) (envelope-from artemrts@ukr.net) Received: from [10.10.80.12] (helo=frv196.fwdcdn.com) by frv190.fwdcdn.com with esmtp ID 1eM6QJ-0007sN-Jm for freebsd-virtualization@freebsd.org; Tue, 05 Dec 2017 08:09:43 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=ffe; h=Content-Transfer-Encoding:Content-Type:MIME-Version:References: In-Reply-To:Message-Id:Cc:To:Subject:From:Date:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=aPppDFp1OTjZrwvdKKNKc5o89XWCpp9ryVeu6I7s0Qs=; b=D5uCSTELSh7k4C6ULX7TJmHiZy lYFxW3B1l/T7jjIkfdc81wLucLHUV4OudfCPouynP0gVwSYmWFGrIEsxSrydAYDHZYmhNzEPxaht8 QjgdKNh8b8ktE/qYeL0QmyevaGunGcJRpRICWFwiVnwTs+2JsRjd29ZVhThvOlAAgXuk=; Received: from [10.10.10.52] (helo=frv52.fwdcdn.com) by frv196.fwdcdn.com with smtp ID 1eM6QB-0006HD-QB for freebsd-virtualization@freebsd.org; Tue, 05 Dec 2017 08:09:35 +0200 Date: Tue, 05 Dec 2017 08:09:35 +0200 From: wishmaster Subject: Re[2]: Bhyve. Unable install Windows 7/Server 2008R2 To: Harry Schmalzbauer Cc: freebsd-virtualization@freebsd.org X-Mailer: mail.ukr.net 5.0 Message-Id: <1512453994.512842252.ka8bgpu8@frv52.fwdcdn.com> In-Reply-To: <5A254C3C.3030809@omnilan.de> References: <1512390476.593482033.ytqgxtgt@frv52.fwdcdn.com> <5A254C3C.3030809@omnilan.de> X-Reply-Action: reply Received: from artemrts@ukr.net by frv52.fwdcdn.com; Tue, 05 Dec 2017 08:09:35 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: binary X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Dec 2017 06:33:21 -0000 Hi, as I said the problem is still here even without ahci-hd device. Below the command: bhyve -c 2 -s 0,hostbridge -s 3,ahci-hd,/dev/zvol/my_zroot/VM/img/win7/disk0 -s 4,ahci-cd,/VM/distro/win7.iso -s 10,virtio-net,tap0 -l bootrom,/VM/distro/BHYVE_UEFI_20160526.fd -m 4G -H -w -s 29,fbuf,tcp=0.0.0.0:5900,w=800,h=600,wait -s 31,lpc windows7 And device: root@XXX: zfs get all my_zroot/VM/img/win7/disk0 NAME PROPERTY VALUE SOURCE my_zroot/VM/img/win7/disk0 type volume - my_zroot/VM/img/win7/disk0 creation Нд лист. 12 15:22 2017 - my_zroot/VM/img/win7/disk0 used 21,3G - my_zroot/VM/img/win7/disk0 available 652G - my_zroot/VM/img/win7/disk0 referenced 442M - my_zroot/VM/img/win7/disk0 compressratio 1.00x - my_zroot/VM/img/win7/disk0 reservation none default my_zroot/VM/img/win7/disk0 volsize 20G local my_zroot/VM/img/win7/disk0 volblocksize 4K - my_zroot/VM/img/win7/disk0 checksum on default my_zroot/VM/img/win7/disk0 compression off local my_zroot/VM/img/win7/disk0 readonly off default my_zroot/VM/img/win7/disk0 copies 1 default my_zroot/VM/img/win7/disk0 refreservation 21,3G local my_zroot/VM/img/win7/disk0 primarycache all default my_zroot/VM/img/win7/disk0 secondarycache all default my_zroot/VM/img/win7/disk0 usedbysnapshots 0 - my_zroot/VM/img/win7/disk0 usedbydataset 442M - my_zroot/VM/img/win7/disk0 usedbychildren 0 - my_zroot/VM/img/win7/disk0 usedbyrefreservation 20,8G - my_zroot/VM/img/win7/disk0 logbias latency default my_zroot/VM/img/win7/disk0 dedup off default my_zroot/VM/img/win7/disk0 mlslabel - my_zroot/VM/img/win7/disk0 sync standard default my_zroot/VM/img/win7/disk0 refcompressratio 1.00x - my_zroot/VM/img/win7/disk0 written 442M - my_zroot/VM/img/win7/disk0 logicalused 438M - my_zroot/VM/img/win7/disk0 logicalreferenced 438M - my_zroot/VM/img/win7/disk0 volmode dev local my_zroot/VM/img/win7/disk0 snapshot_limit none default my_zroot/VM/img/win7/disk0 snapshot_count none default my_zroot/VM/img/win7/disk0 redundant_metadata all default --- Original message --- From: "Harry Schmalzbauer" Date: 4 December 2017, 15:23:13 > Bezüglich wishmaster's Nachricht vom 04.12.2017 13:32 (localtime): > > Hi, > > > > guys, what happened with bhyve after 11.1-RELEASE. I am unable install neither Windows 7 nor Windows Server 2008R2. > > I have attempted with HDD as raw file, ZFS dev - without success. > > > > root@XXX: uname -a > > FreeBSD XXX 11.1-STABLE FreeBSD 11.1-STABLE #0 r325503: Tue Nov 7 13:38:44 EET 2017 wishmaster@XXX:/usr/obj/usr/src/sys/MY amd64 > > Since you don't tell us what error you get from the guest setup program, > I can only guess that you are using a ZFS volume with a block size of 8k > (default). > My next guess is that you use BHYVE_UEFI.fd for bootrom. > The latter can't boot from 8k drives, > Make sure you define something like this: > -s 3,virtio-blk,/dev/zvol/hostPool/bhyveVOL/sys/guest1,sectorsize=512/4096 > > If that power guess didn't hit, you need to provide some details (boot > method at very first along with some kind of error description). > > -harry > > From owner-freebsd-virtualization@freebsd.org Tue Dec 5 06:54:21 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D5D0DDF8758 for ; Tue, 5 Dec 2017 06:54:21 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id 8CAA96DA16 for ; Tue, 5 Dec 2017 06:54:21 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 9CB0620B1B12 for ; Tue, 5 Dec 2017 16:54:11 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 957C02809CC for ; Tue, 5 Dec 2017 16:54:11 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id ITRHy8PF-Lo6 for ; Tue, 5 Dec 2017 16:54:11 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (c-67-180-92-13.hsd1.ca.comcast.net [67.180.92.13]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id B5D7D28190A; Tue, 5 Dec 2017 16:54:07 +1000 (AEST) Subject: Re: Bhyve. Unable install Windows 7/Server 2008R2 To: wishmaster Cc: Harry Schmalzbauer , freebsd-virtualization@freebsd.org References: <1512390476.593482033.ytqgxtgt@frv52.fwdcdn.com> <5A254C3C.3030809@omnilan.de> <1512453994.512842252.ka8bgpu8@frv52.fwdcdn.com> From: Peter Grehan Message-ID: <176468b0-708d-a0f6-96e9-49c99431e281@freebsd.org> Date: Mon, 4 Dec 2017 22:54:05 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <1512453994.512842252.ka8bgpu8@frv52.fwdcdn.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.2 cv=XtWKARN9 c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=5eVCmCvhg37cu/pjidAGzw==:17 a=IkcTkHD0fZMA:10 a=ocR9PWop10UA:10 a=adB33H22bH1QjIN9SvkA:9 a=QEXdDO2ut3YA:10 wl=host:3 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Dec 2017 06:54:21 -0000 Hi, > bhyve -c 2 -s 0,hostbridge -s 3,ahci-hd,/dev/zvol/my_zroot/VM/img/win7/disk0 For win7/2k8*, the sector size presented to the guest has to be forced to 512 bytes i.e. for the ahci-cd slot, the config would look like: -s 3,ahci-hd,/dev/zvol/my_zroot/VM/img/win7/disk0,sectorsize=512 later, Peter. From owner-freebsd-virtualization@freebsd.org Tue Dec 5 12:49:48 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7E777E68E0C for ; Tue, 5 Dec 2017 12:49:48 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6109379398; Tue, 5 Dec 2017 12:49:47 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id vB5Cnj6C051307; Tue, 5 Dec 2017 04:49:45 -0800 (PST) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id vB5Cnj5f051306; Tue, 5 Dec 2017 04:49:45 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201712051249.vB5Cnj5f051306@pdx.rh.CN85.dnsmgr.net> Subject: Re: Bhyve. Unable install Windows 7/Server 2008R2 In-Reply-To: <176468b0-708d-a0f6-96e9-49c99431e281@freebsd.org> To: Peter Grehan Date: Tue, 5 Dec 2017 04:49:45 -0800 (PST) CC: wishmaster , freebsd-virtualization@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Dec 2017 12:49:48 -0000 > Hi, > > > bhyve -c 2 -s 0,hostbridge -s 3,ahci-hd,/dev/zvol/my_zroot/VM/img/win7/disk0 > > For win7/2k8*, the sector size presented to the guest has to be forced > to 512 bytes i.e. for the ahci-cd slot, the config would look like: > > -s 3,ahci-hd,/dev/zvol/my_zroot/VM/img/win7/disk0,sectorsize=512 Should we start a wiki page on "Guest Quirks"? -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Tue Dec 5 14:25:52 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DF533E6B718 for ; Tue, 5 Dec 2017 14:25:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CCE407C91F for ; Tue, 5 Dec 2017 14:25:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vB5EPo9X069043 for ; Tue, 5 Dec 2017 14:25:52 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 213333] FreeBSD 11-RELEASE fails to boot under KVM/Qemu Date: Tue, 05 Dec 2017 14:25:51 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mainland@apeiron.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Dec 2017 14:25:53 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D213333 mainland@apeiron.net changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mainland@apeiron.net --- Comment #16 from mainland@apeiron.net --- I have the same issue with both 10.4 and 11.1 kernels---I cannot upgrade fr= om 10.3. Setting hw.use_xsave=3D0 in loader.conf does not help. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Tue Dec 5 15:20:46 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 99B76E6D5B7 for ; Tue, 5 Dec 2017 15:20:46 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from internet06.ebureau.com (internet06.ebureau.com [65.127.24.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "internet06.ebureau.com", Issuer "internet06.ebureau.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 722837EA98 for ; Tue, 5 Dec 2017 15:20:46 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from localhost (localhost [127.0.0.1]) by internet06.ebureau.com (Postfix) with ESMTP id 7F9DF846B770; Tue, 5 Dec 2017 09:20:44 -0600 (CST) X-Virus-Scanned: amavisd-new at mydomain = ebureau.com Received: from internet06.ebureau.com ([127.0.0.1]) by localhost (internet06.ebureau.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6M4XFHG0oKYr; Tue, 5 Dec 2017 09:20:44 -0600 (CST) Received: from square.office.ebureau.com (unknown [10.10.21.22]) by internet06.ebureau.com (Postfix) with ESMTPSA id 0BE9C846B763; Tue, 5 Dec 2017 09:20:44 -0600 (CST) From: Dustin Wenz Message-Id: <423F466A-732A-4B04-956E-3CC5F5C47390@ebureau.com> Content-Type: multipart/signed; boundary="Apple-Mail=_1D490CF7-77CD-464F-B3DB-17A2F86D455D"; protocol="application/pkcs7-signature"; micalg=sha1 Mime-Version: 1.0 (Mac OS X Mail 11.1 \(3445.4.7\)) Subject: Re: Storage overhead on zvols Date: Tue, 5 Dec 2017 09:20:43 -0600 In-Reply-To: Cc: FreeBSD virtualization To: Adam Vande More References: X-Mailer: Apple Mail (2.3445.4.7) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Dec 2017 15:20:46 -0000 --Apple-Mail=_1D490CF7-77CD-464F-B3DB-17A2F86D455D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Thanks for linking that resource. The purpose of my posting was to = increase the body of knowledge available to people who are running bhyve = on zfs. It's a versatile way to deploy guests, but I haven't seen much = practical advise about doing it efficiently.=20 Allan's explanation yesterday of how allocations are padded is exactly = the sort of breakdown I could have used when I first started = provisioning VMs. I'm sure other people will find this conversation = useful as well. - .Dustin > On Dec 4, 2017, at 9:37 PM, Adam Vande More = wrote: >=20 > On Mon, Dec 4, 2017 at 5:19 PM, Dustin Wenz = wrote: > I'm starting a new thread based on the previous discussion in "bhyve = uses all available memory during IO-intensive operations" relating to = size inflation of bhyve data stored on zvols. I've done some = experimenting with this, and I think it will be useful for others. >=20 > The zvols listed here were created with this command: >=20 > zfs create -o volmode=3Ddev -o volblocksize=3DXk -V 30g = vm00/chyves/guests/myguest/diskY >=20 > The zvols were created on a raidz1 pool of four disks. For each zvol, = I created a basic zfs filesystem in the guest using all default tuning = (128k recordsize, etc). I then copied the same 8.2GB dataset to each = filesystem. >=20 > volblocksize size amplification >=20 > 512B 11.7x > 4k 1.45x > 8k 1.45x > 16k 1.5x > 32k 1.65x > 64k 1x > 128k 1x >=20 > The worst case is with a 512B volblocksize, where the space used is = more than 11 times the size of the data stored within the guest. The = size efficiency gains are non-linear as I continue from 4k and double = the block sizes; 32k blocks being the second-worst. The amount of wasted = space was minimized by using 64k and 128k blocks. >=20 > It would appear that 64k is a good choice for volblocksize if you are = using a zvol to back your VM, and the VM is using the virtual device for = a zpool. Incidentally, I believe this is the default when creating VMs = in FreeNAS. >=20 > I'm not sure what your purpose is behind the posting, but if its = simply a "why this behavior" you can find more detail here as well as = some calculation leg work: >=20 > = https://www.delphix.com/blog/delphix-engineering/zfs-raidz-stripe-width-or= -how-i-learned-stop-worrying-and-love-raidz >=20 > --=20 > Adam --Apple-Mail=_1D490CF7-77CD-464F-B3DB-17A2F86D455D Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIEzDCCBMgw ggOwoAMCAQICAUEwDQYJKoZIhvcNAQELBQAwgZkxCzAJBgNVBAYTAlVTMRIwEAYDVQQIEwlNaW5u ZXNvdGExFDASBgNVBAcTC1NhaW50IENsb3VkMRAwDgYDVQQKEwdlQnVyZWF1MRQwEgYDVQQLEwtJ bnRlZ3JhdGlvbjEUMBIGA1UEAxMLZWJ1cmVhdS5jb20xIjAgBgkqhkiG9w0BCQEWE3N1cHBvcnRA ZWJ1cmVhdS5jb20wHhcNMTcwNTA1MTYxNjE1WhcNMjcwNTAzMTYxNjE1WjBKMQswCQYDVQQGEwJV UzEUMBIGA1UEAwwLRHVzdGluIFdlbnoxJTAjBgkqhkiG9w0BCQEWFmR1c3RpbndlbnpAZWJ1cmVh dS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCQ/HJXe7JhUaexqEaxCNVifsue sUMgohgXLmi5YIcbAWhdxTr1PUzKYkeTkL9sYNjXU6uaI2tZMF3hA9gcFjxQIfkKSo31MrYOgMdU xQe0Q+t6Vd4pGAmtDQDwhAsrNGccADp3Yjy4eBtVfkDGdzz1Y8Lbc684TPFcW7i9+U/dDaXlcxeq fyDqiHZ5y8Lp/1M2Ot/Rz7eikJZTAuHOWKs/PEiJIM2JHuhPyNy+mL2oqEWeOcEsKMNzgn7HVt4k Xz2irBAG+cj4WAxWs418l46EEXgur4PvhBXZMl0LJg0TyaxOHbsUam4R4tbKnaZ3HhRkg79k2Had sb6DKbnCw9/1AgMBAAGjggFnMIIBYzAJBgNVHRMEAjAAMAsGA1UdDwQEAwIE8DAnBgNVHSUEIDAe BggrBgEFBQcDAQYIKwYBBQUHAwIGCCsGAQUFBwMEMB0GA1UdDgQWBBTLi/8HUHpbBEt9OtPqQoax AmpaNDCBzgYDVR0jBIHGMIHDgBRnpZeXB5rQYLgsUKqiiBcLIHyu6aGBn6SBnDCBmTELMAkGA1UE BhMCVVMxEjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAOBgNVBAoT B2VCdXJlYXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNvbTEiMCAG CSqGSIb3DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbYIJAMwZcjAWAsWXMDAGCWCGSAGG+EIBBAQj FiFodHRwOi8vd3d3LmVidXJlYXUuY29tL2NhLWNybC5wZW0wDQYJKoZIhvcNAQELBQADggEBAHbO qVdB9raUKXCgZRA/nES5a60dlIaGnIlpgz+Y3SjFt0bcJxoUYhIzumBHk9yjyP4M1DubOphkQpJ4 LNZbAS01cjCxjnC0ZUq5V3FCeaDwrn1qPY+QJGoZPLlhWdJUNu17OpnR7ZfBWlp3/pRhvNU5PCbJ nmF7rnvsqxUFq9oeiV3SmqBux5lwJ7p2Uss5SHSW6g17K/KdTMK1roQr/+rWpxp2233qddDrLpOE xGRlvhEqSa/IZbGC9oiYmsiaG1PefQkadoob5IMIS5/MDpWHUgSHqAj1V/LwcCx0rbt73SazGMND EzHVWhsj+khepB/MG5QGfWP23IGFmvQYWWcxggOQMIIDjAIBATCBnzCBmTELMAkGA1UEBhMCVVMx EjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAOBgNVBAoTB2VCdXJl YXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNvbTEiMCAGCSqGSIb3 DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbQIBQTAJBgUrDgMCGgUAoIIBxTAYBgkqhkiG9w0BCQMx CwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNzEyMDUxNTIwNDNaMCMGCSqGSIb3DQEJBDEW BBSiUZAcewvpJ5w262TQiHjTQDFWiDCBsAYJKwYBBAGCNxAEMYGiMIGfMIGZMQswCQYDVQQGEwJV UzESMBAGA1UECBMJTWlubmVzb3RhMRQwEgYDVQQHEwtTYWludCBDbG91ZDEQMA4GA1UEChMHZUJ1 cmVhdTEUMBIGA1UECxMLSW50ZWdyYXRpb24xFDASBgNVBAMTC2VidXJlYXUuY29tMSIwIAYJKoZI hvcNAQkBFhNzdXBwb3J0QGVidXJlYXUuY29tAgFBMIGyBgsqhkiG9w0BCRACCzGBoqCBnzCBmTEL MAkGA1UEBhMCVVMxEjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAO BgNVBAoTB2VCdXJlYXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNv bTEiMCAGCSqGSIb3DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbQIBQTANBgkqhkiG9w0BAQEFAASC AQCBXotEZa74uWViaewdng8UcUseVUjhOLgQy+syE37Aa4I2zM252Ctqdwq5HuI5fpv9xf1Tt8hJ helNC/Z+nFapllc4EAzOrFSbYvxMVPWzEPg53dMJ8yw+GOV4/1fZ/cQUsdb+JbGeSVry06CaixTo sAxQXbedyJBdwDPD9IfJk5vGzguljAbUUuS1QVg8Wf/HuGmzKV/ToPt23wIpasqB6qd3smrXq1UL rjQdGfTOauFX0Mbed+kUKB6T+Jh1a08+nqsWqZuM0i5YSqEkDvf4qbRw638jQzatv38cI18x8RMP I4N2fsvWB4057MadSjm4X/c0BfH8QTC/F8rU19mIAAAAAAAA --Apple-Mail=_1D490CF7-77CD-464F-B3DB-17A2F86D455D-- From owner-freebsd-virtualization@freebsd.org Tue Dec 5 15:22:52 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7410FE6D81E for ; Tue, 5 Dec 2017 15:22:52 +0000 (UTC) (envelope-from paul@redbarn.org) Received: from family.redbarn.org (family.redbarn.org [IPv6:2001:559:8000:cd::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 632AB7ED81 for ; Tue, 5 Dec 2017 15:22:52 +0000 (UTC) (envelope-from paul@redbarn.org) Received: from [192.168.11.48] (200.12.232.153.ap.dti.ne.jp [153.232.12.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by family.redbarn.org (Postfix) with ESMTPSA id 4081461FA2 for ; Tue, 5 Dec 2017 15:22:51 +0000 (UTC) Message-ID: <5A26B9C8.7020005@redbarn.org> Date: Tue, 05 Dec 2017 07:22:48 -0800 From: Paul Vixie User-Agent: Postbox 5.0.20 (Windows/20171012) MIME-Version: 1.0 To: FreeBSD virtualization Subject: Re: Storage overhead on zvols References: <423F466A-732A-4B04-956E-3CC5F5C47390@ebureau.com> In-Reply-To: <423F466A-732A-4B04-956E-3CC5F5C47390@ebureau.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Dec 2017 15:22:52 -0000 the surprising fact that came up in recent threads is that some of you run zfs in your guests. that's quite a bit of unnec'y redundancy and other overheads. i am using UFS in my guests. From owner-freebsd-virtualization@freebsd.org Tue Dec 5 15:30:59 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C5EC3E6DA70 for ; Tue, 5 Dec 2017 15:30:59 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from internet06.ebureau.com (internet06.ebureau.com [65.127.24.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "internet06.ebureau.com", Issuer "internet06.ebureau.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 945037EF04 for ; Tue, 5 Dec 2017 15:30:59 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from localhost (localhost [127.0.0.1]) by internet06.ebureau.com (Postfix) with ESMTP id 26582846BA80; Tue, 5 Dec 2017 09:30:58 -0600 (CST) X-Virus-Scanned: amavisd-new at mydomain = ebureau.com Received: from internet06.ebureau.com ([127.0.0.1]) by localhost (internet06.ebureau.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Mpmqjt4bpUxN; Tue, 5 Dec 2017 09:30:57 -0600 (CST) Received: from square.office.ebureau.com (unknown [10.10.21.22]) by internet06.ebureau.com (Postfix) with ESMTPSA id D0BFB846BA73; Tue, 5 Dec 2017 09:30:57 -0600 (CST) From: Dustin Wenz Message-Id: <32BA4687-AB70-4370-A9BA-EF4F66BF69A6@ebureau.com> Content-Type: multipart/signed; boundary="Apple-Mail=_60E27CC7-C84A-4297-9933-0550FAD7B252"; protocol="application/pkcs7-signature"; micalg=sha1 Mime-Version: 1.0 (Mac OS X Mail 11.1 \(3445.4.7\)) Subject: Re: Storage overhead on zvols Date: Tue, 5 Dec 2017 09:30:57 -0600 In-Reply-To: <5A26B9C8.7020005@redbarn.org> Cc: FreeBSD virtualization To: Paul Vixie References: <423F466A-732A-4B04-956E-3CC5F5C47390@ebureau.com> <5A26B9C8.7020005@redbarn.org> X-Mailer: Apple Mail (2.3445.4.7) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Dec 2017 15:30:59 -0000 --Apple-Mail=_60E27CC7-C84A-4297-9933-0550FAD7B252 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii I'm not using ZFS in my VMs for data integrity (the host already = provides that); it's mainly for the easy creation and management of = filesystems, and the ability to do snapshots for rollback and = replication. Some of my deployments have hundreds of filesystems in an = organized hierarchy, with delegated permissions and automated snapshots, = send/recvs, and clones for various operations. - .Dustin > On Dec 5, 2017, at 9:22 AM, Paul Vixie wrote: >=20 > the surprising fact that came up in recent threads is that some of you = run zfs in your guests. that's quite a bit of unnec'y redundancy and = other overheads. i am using UFS in my guests. > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to = "freebsd-virtualization-unsubscribe@freebsd.org" --Apple-Mail=_60E27CC7-C84A-4297-9933-0550FAD7B252 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIEzDCCBMgw ggOwoAMCAQICAUEwDQYJKoZIhvcNAQELBQAwgZkxCzAJBgNVBAYTAlVTMRIwEAYDVQQIEwlNaW5u ZXNvdGExFDASBgNVBAcTC1NhaW50IENsb3VkMRAwDgYDVQQKEwdlQnVyZWF1MRQwEgYDVQQLEwtJ bnRlZ3JhdGlvbjEUMBIGA1UEAxMLZWJ1cmVhdS5jb20xIjAgBgkqhkiG9w0BCQEWE3N1cHBvcnRA ZWJ1cmVhdS5jb20wHhcNMTcwNTA1MTYxNjE1WhcNMjcwNTAzMTYxNjE1WjBKMQswCQYDVQQGEwJV UzEUMBIGA1UEAwwLRHVzdGluIFdlbnoxJTAjBgkqhkiG9w0BCQEWFmR1c3RpbndlbnpAZWJ1cmVh dS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCQ/HJXe7JhUaexqEaxCNVifsue sUMgohgXLmi5YIcbAWhdxTr1PUzKYkeTkL9sYNjXU6uaI2tZMF3hA9gcFjxQIfkKSo31MrYOgMdU xQe0Q+t6Vd4pGAmtDQDwhAsrNGccADp3Yjy4eBtVfkDGdzz1Y8Lbc684TPFcW7i9+U/dDaXlcxeq fyDqiHZ5y8Lp/1M2Ot/Rz7eikJZTAuHOWKs/PEiJIM2JHuhPyNy+mL2oqEWeOcEsKMNzgn7HVt4k Xz2irBAG+cj4WAxWs418l46EEXgur4PvhBXZMl0LJg0TyaxOHbsUam4R4tbKnaZ3HhRkg79k2Had sb6DKbnCw9/1AgMBAAGjggFnMIIBYzAJBgNVHRMEAjAAMAsGA1UdDwQEAwIE8DAnBgNVHSUEIDAe BggrBgEFBQcDAQYIKwYBBQUHAwIGCCsGAQUFBwMEMB0GA1UdDgQWBBTLi/8HUHpbBEt9OtPqQoax AmpaNDCBzgYDVR0jBIHGMIHDgBRnpZeXB5rQYLgsUKqiiBcLIHyu6aGBn6SBnDCBmTELMAkGA1UE BhMCVVMxEjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAOBgNVBAoT B2VCdXJlYXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNvbTEiMCAG CSqGSIb3DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbYIJAMwZcjAWAsWXMDAGCWCGSAGG+EIBBAQj FiFodHRwOi8vd3d3LmVidXJlYXUuY29tL2NhLWNybC5wZW0wDQYJKoZIhvcNAQELBQADggEBAHbO qVdB9raUKXCgZRA/nES5a60dlIaGnIlpgz+Y3SjFt0bcJxoUYhIzumBHk9yjyP4M1DubOphkQpJ4 LNZbAS01cjCxjnC0ZUq5V3FCeaDwrn1qPY+QJGoZPLlhWdJUNu17OpnR7ZfBWlp3/pRhvNU5PCbJ nmF7rnvsqxUFq9oeiV3SmqBux5lwJ7p2Uss5SHSW6g17K/KdTMK1roQr/+rWpxp2233qddDrLpOE xGRlvhEqSa/IZbGC9oiYmsiaG1PefQkadoob5IMIS5/MDpWHUgSHqAj1V/LwcCx0rbt73SazGMND EzHVWhsj+khepB/MG5QGfWP23IGFmvQYWWcxggOQMIIDjAIBATCBnzCBmTELMAkGA1UEBhMCVVMx EjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAOBgNVBAoTB2VCdXJl YXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNvbTEiMCAGCSqGSIb3 DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbQIBQTAJBgUrDgMCGgUAoIIBxTAYBgkqhkiG9w0BCQMx CwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNzEyMDUxNTMwNTdaMCMGCSqGSIb3DQEJBDEW BBTyR2BU3Alove+rYWhgA21q20K5TzCBsAYJKwYBBAGCNxAEMYGiMIGfMIGZMQswCQYDVQQGEwJV UzESMBAGA1UECBMJTWlubmVzb3RhMRQwEgYDVQQHEwtTYWludCBDbG91ZDEQMA4GA1UEChMHZUJ1 cmVhdTEUMBIGA1UECxMLSW50ZWdyYXRpb24xFDASBgNVBAMTC2VidXJlYXUuY29tMSIwIAYJKoZI hvcNAQkBFhNzdXBwb3J0QGVidXJlYXUuY29tAgFBMIGyBgsqhkiG9w0BCRACCzGBoqCBnzCBmTEL MAkGA1UEBhMCVVMxEjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAO BgNVBAoTB2VCdXJlYXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNv bTEiMCAGCSqGSIb3DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbQIBQTANBgkqhkiG9w0BAQEFAASC AQA6uJxw8XS4AumcsYbnPu7B3Q6auCehqQY/gy0B7ApnLBilwuwWPSVQGxdoBBIhHMkgieuPBb7D TUZlFAZZq6im+rVDDgZFY3jjs7OYe8qUjmu5nFEJGjyNoZoqqgBI2X4xYeAdxkr8On5TMtkRrnXo 3k4kCPUdpQk9q2J4dNSoOZ+qBxEoCKx1vzH+7qTG52Ox+9z4VKN/F3lhPaB0QInIaGFR8usp8LNO cl20uj8Eq43r7wu2VgzZ7voWzEeKIiiWYApbENEGMrA8SqCIawVifT8KBVK0/NjIsT1DcOkRXtg+ 7vdv00yVlmbobuvwZeWNTWpU5zwXKbmHSYQHNyqrAAAAAAAA --Apple-Mail=_60E27CC7-C84A-4297-9933-0550FAD7B252-- From owner-freebsd-virtualization@freebsd.org Tue Dec 5 15:37:14 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 506B9E6DE8D for ; Tue, 5 Dec 2017 15:37:14 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 256617F45E for ; Tue, 5 Dec 2017 15:37:13 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id vB5Fb17T052053; Tue, 5 Dec 2017 07:37:01 -0800 (PST) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id vB5Fb1dv052052; Tue, 5 Dec 2017 07:37:01 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201712051537.vB5Fb1dv052052@pdx.rh.CN85.dnsmgr.net> Subject: Re: Storage overhead on zvols In-Reply-To: <32BA4687-AB70-4370-A9BA-EF4F66BF69A6@ebureau.com> To: Dustin Wenz Date: Tue, 5 Dec 2017 07:37:01 -0800 (PST) CC: Paul Vixie , FreeBSD virtualization X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Dec 2017 15:37:14 -0000 > I'm not using ZFS in my VMs for data integrity (the host already provides that); it's mainly for the easy creation and management of filesystems, and the ability to do snapshots for rollback and replication. Some of my deployments have hundreds of filesystems in an organized hierarchy, with delegated permissions and automated snapshots, send/recvs, and clones for various operations. I architect things in such a way that I have 1 VM used as a NAS that runs zfs and allows all that nice functionality then all the other VM's run with ufs + nfs mounts. And can actually run a VM with no local disk over iPxe netbooting. I find this very flexible and minimally impacting. Though it does make a single point of failure, that could be cured with some redundancy. -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Tue Dec 5 15:41:26 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 61536E6E26E for ; Tue, 5 Dec 2017 15:41:26 +0000 (UTC) (envelope-from paul@redbarn.org) Received: from family.redbarn.org (family.redbarn.org [IPv6:2001:559:8000:cd::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4FB8D7F782 for ; Tue, 5 Dec 2017 15:41:26 +0000 (UTC) (envelope-from paul@redbarn.org) Received: from [192.168.11.48] (200.12.232.153.ap.dti.ne.jp [153.232.12.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by family.redbarn.org (Postfix) with ESMTPSA id D768661FA2 for ; Tue, 5 Dec 2017 15:41:25 +0000 (UTC) Message-ID: <5A26BE25.10409@redbarn.org> Date: Tue, 05 Dec 2017 07:41:25 -0800 From: Paul Vixie User-Agent: Postbox 5.0.20 (Windows/20171012) MIME-Version: 1.0 To: FreeBSD virtualization Subject: Re: Storage overhead on zvols References: <423F466A-732A-4B04-956E-3CC5F5C47390@ebureau.com> <5A26B9C8.7020005@redbarn.org> <32BA4687-AB70-4370-A9BA-EF4F66BF69A6@ebureau.com> In-Reply-To: <32BA4687-AB70-4370-A9BA-EF4F66BF69A6@ebureau.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Dec 2017 15:41:26 -0000 Dustin Wenz wrote: > I'm not using ZFS in my VMs for data integrity (the host already > provides that); it's mainly for the easy creation and management of > filesystems, and the ability to do snapshots for rollback and > replication. snapshot and replication works fine on the host, acting on the zvol. > Some of my deployments have hundreds of filesystems in > an organized hierarchy, with delegated permissions and automated > snapshots, send/recvs, and clones for various operations. what kind of zpool do you use in the guest, to avoid unwanted additional redundancy? did you benchmark the space or time efficiency of ZFS vs. UFS? in some bsd related meeting this year i asked allan jude for a bhyve level null mount, so that we could access at / inside the guest some subtree of the host, and avoid block devices and file systems altogether. right now i have to use nfs for that, which is irritating. -- P Vixie From owner-freebsd-virtualization@freebsd.org Tue Dec 5 15:49:25 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 77BD8E6E4A5 for ; Tue, 5 Dec 2017 15:49:25 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 10E077FBCC for ; Tue, 5 Dec 2017 15:49:24 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from hugo10.ka.punkt.de (hugo10.ka.punkt.de [217.29.44.10]) by gate2.intern.punkt.de with ESMTP id vB5FnLIE037225; Tue, 5 Dec 2017 16:49:21 +0100 (CET) Received: from [217.29.44.110] ([217.29.44.110]) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id vB5FnLLl039272; Tue, 5 Dec 2017 16:49:21 +0100 (CET) (envelope-from hausen@punkt.de) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: Storage overhead on zvols From: "Patrick M. Hausen" In-Reply-To: <5A26BE25.10409@redbarn.org> Date: Tue, 5 Dec 2017 16:49:21 +0100 Cc: FreeBSD virtualization Content-Transfer-Encoding: quoted-printable Message-Id: <44AEC596-6BBA-44FB-92A1-99A0ED239B7A@punkt.de> References: <423F466A-732A-4B04-956E-3CC5F5C47390@ebureau.com> <5A26B9C8.7020005@redbarn.org> <32BA4687-AB70-4370-A9BA-EF4F66BF69A6@ebureau.com> <5A26BE25.10409@redbarn.org> To: Paul Vixie X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Dec 2017 15:49:25 -0000 Hi all, > Am 05.12.2017 um 16:41 schrieb Paul Vixie : > in some bsd related meeting this year i asked allan jude for a bhyve = level null mount, > so that we could access at / inside the guest some subtree of the = host, and avoid block > devices and file systems altogether. right now i have to use nfs for = that, which is irritating. I'm not an FS developer but from experience as an admin that feature - nullfs mounts into a hypervisor - while greatly desired, looks quite nontrivial to implement. Jordan went to 9Pfs for the now discontinued FreeNAS Corral at iX. If it was easy to do at the VFS layer, I doubt they would have gone that way. Kind regards, Patrick --=20 punkt.de GmbH Internet - Dienstleistungen - Beratung Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100 76133 Karlsruhe info@punkt.de http://punkt.de AG Mannheim 108285 Gf: Juergen Egeling From owner-freebsd-virtualization@freebsd.org Tue Dec 5 15:58:26 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 43416E6EB42 for ; Tue, 5 Dec 2017 15:58:26 +0000 (UTC) (envelope-from paul@redbarn.org) Received: from family.redbarn.org (family.redbarn.org [IPv6:2001:559:8000:cd::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2F2A880393 for ; Tue, 5 Dec 2017 15:58:26 +0000 (UTC) (envelope-from paul@redbarn.org) Received: from [192.168.11.48] (200.12.232.153.ap.dti.ne.jp [153.232.12.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by family.redbarn.org (Postfix) with ESMTPSA id 8BD7E61FA2 for ; Tue, 5 Dec 2017 15:58:25 +0000 (UTC) Message-ID: <5A26C220.2000909@redbarn.org> Date: Wed, 06 Dec 2017 00:58:24 +0900 From: Paul Vixie User-Agent: Postbox 5.0.20 (Windows/20171012) MIME-Version: 1.0 To: FreeBSD virtualization Subject: Re: Storage overhead on zvols References: <423F466A-732A-4B04-956E-3CC5F5C47390@ebureau.com> <5A26B9C8.7020005@redbarn.org> <32BA4687-AB70-4370-A9BA-EF4F66BF69A6@ebureau.com> <5A26BE25.10409@redbarn.org> <44AEC596-6BBA-44FB-92A1-99A0ED239B7A@punkt.de> In-Reply-To: <44AEC596-6BBA-44FB-92A1-99A0ED239B7A@punkt.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Dec 2017 15:58:26 -0000 Patrick M. Hausen wrote: > I'm not an FS developer but from experience as an admin that > feature - nullfs mounts into a hypervisor - while greatly desired, > looks quite nontrivial to implement. i think what's called for is a vdd of some kind, similar to the virtual ethernet and virtual disk drivers. yes, it would appear in the guest at the vfs layer. i'm surprised that the qemu community doesn't already have it. this is something virtualbox gets wrong, by the way. it offers something that sounds like what i want, but then implements it as SMB. don't get be wrong -- UFS and NFS work for me, and i love bhyve as-is. -- P Vixie From owner-freebsd-virtualization@freebsd.org Tue Dec 5 16:14:44 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3E5ECE6F4D4 for ; Tue, 5 Dec 2017 16:14:44 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1E3CF1081 for ; Tue, 5 Dec 2017 16:14:43 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id 7113D139BE for ; Tue, 5 Dec 2017 16:14:42 +0000 (UTC) Subject: Re: Storage overhead on zvols To: freebsd-virtualization@freebsd.org References: <423F466A-732A-4B04-956E-3CC5F5C47390@ebureau.com> From: Allan Jude Message-ID: <3496de79-8610-5640-0d2c-22031d7e3e5f@freebsd.org> Date: Tue, 5 Dec 2017 11:14:41 -0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <423F466A-732A-4B04-956E-3CC5F5C47390@ebureau.com> Content-Type: text/plain; charset=windows-1252 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Dec 2017 16:14:44 -0000 On 2017-12-05 10:20, Dustin Wenz wrote: > Thanks for linking that resource. The purpose of my posting was to increase the body of knowledge available to people who are running bhyve on zfs. It's a versatile way to deploy guests, but I haven't seen much practical advise about doing it efficiently. > > Allan's explanation yesterday of how allocations are padded is exactly the sort of breakdown I could have used when I first started provisioning VMs. I'm sure other people will find this conversation useful as well. > > - .Dustin > This subject is covered in detail in chapter 9 (Tuning) of "FreeBSD Mastery: Advanced ZFS", available from http://www.zfsbook.com/ or any finer book store. >> On Dec 4, 2017, at 9:37 PM, Adam Vande More wrote: >> >> On Mon, Dec 4, 2017 at 5:19 PM, Dustin Wenz wrote: >> I'm starting a new thread based on the previous discussion in "bhyve uses all available memory during IO-intensive operations" relating to size inflation of bhyve data stored on zvols. I've done some experimenting with this, and I think it will be useful for others. >> >> The zvols listed here were created with this command: >> >> zfs create -o volmode=dev -o volblocksize=Xk -V 30g vm00/chyves/guests/myguest/diskY >> >> The zvols were created on a raidz1 pool of four disks. For each zvol, I created a basic zfs filesystem in the guest using all default tuning (128k recordsize, etc). I then copied the same 8.2GB dataset to each filesystem. >> >> volblocksize size amplification >> >> 512B 11.7x >> 4k 1.45x >> 8k 1.45x >> 16k 1.5x >> 32k 1.65x >> 64k 1x >> 128k 1x >> >> The worst case is with a 512B volblocksize, where the space used is more than 11 times the size of the data stored within the guest. The size efficiency gains are non-linear as I continue from 4k and double the block sizes; 32k blocks being the second-worst. The amount of wasted space was minimized by using 64k and 128k blocks. >> >> It would appear that 64k is a good choice for volblocksize if you are using a zvol to back your VM, and the VM is using the virtual device for a zpool. Incidentally, I believe this is the default when creating VMs in FreeNAS. >> >> I'm not sure what your purpose is behind the posting, but if its simply a "why this behavior" you can find more detail here as well as some calculation leg work: >> >> https://www.delphix.com/blog/delphix-engineering/zfs-raidz-stripe-width-or-how-i-learned-stop-worrying-and-love-raidz >> >> -- >> Adam > -- Allan Jude From owner-freebsd-virtualization@freebsd.org Tue Dec 5 16:41:31 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3EF36E70238 for ; Tue, 5 Dec 2017 16:41:31 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E951E2863 for ; Tue, 5 Dec 2017 16:41:30 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id vB5GfRGV052311; Tue, 5 Dec 2017 08:41:27 -0800 (PST) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id vB5GfR5I052310; Tue, 5 Dec 2017 08:41:27 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201712051641.vB5GfR5I052310@pdx.rh.CN85.dnsmgr.net> Subject: Re: Storage overhead on zvols In-Reply-To: <5A26BE25.10409@redbarn.org> To: Paul Vixie Date: Tue, 5 Dec 2017 08:41:27 -0800 (PST) CC: FreeBSD virtualization X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Dec 2017 16:41:31 -0000 > > > Dustin Wenz wrote: > > I'm not using ZFS in my VMs for data integrity (the host already > > provides that); it's mainly for the easy creation and management of > > filesystems, and the ability to do snapshots for rollback and > > replication. > > snapshot and replication works fine on the host, acting on the zvol. I suspect he is snapshotting and doing send/recvs of something much less than the zvol, probably some datasetbs, maybe boot envorinments, a snapshot of the whole zvol is ok if your managing data at the VM level, not so good if you got lots of stuff going on inside the VM. > > Some of my deployments have hundreds of filesystems in > > an organized hierarchy, with delegated permissions and automated > > snapshots, send/recvs, and clones for various operations. > > what kind of zpool do you use in the guest, to avoid unwanted additional > redundancy? Just a simple stripe of 1 device would be my guess, though your still gona have metadata redundancy. > > did you benchmark the space or time efficiency of ZFS vs. UFS? > > in some bsd related meeting this year i asked allan jude for a bhyve > level null mount, so that we could access at / inside the guest some > subtree of the host, and avoid block devices and file systems > altogether. right now i have to use nfs for that, which is irritating. This is not as simple as it seems, remember bhyve is just presenting a hardware environment, hardware environments dont have a file system concept per se, unlike jails which are providing a software environment. In effect what your asking for is what NFS does, so use NFS and get over the fact that this is the way to get what you want. Sure you could implement a virt-vfs but I wonder how close the spec of that would be to the spec of NFS. Or maybe thats the answer, implement virt-vfs as a more effecient way to transport nfs calls in and out of the guest. -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Tue Dec 5 16:50:46 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AF3E2E708AE for ; Tue, 5 Dec 2017 16:50:46 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from internet06.ebureau.com (internet06.ebureau.com [65.127.24.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "internet06.ebureau.com", Issuer "internet06.ebureau.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 872E63055 for ; Tue, 5 Dec 2017 16:50:46 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from localhost (localhost [127.0.0.1]) by internet06.ebureau.com (Postfix) with ESMTP id AC13A846DAA1; Tue, 5 Dec 2017 10:50:44 -0600 (CST) X-Virus-Scanned: amavisd-new at mydomain = ebureau.com Received: from internet06.ebureau.com ([127.0.0.1]) by localhost (internet06.ebureau.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nOXEqoy1K9D3; Tue, 5 Dec 2017 10:50:44 -0600 (CST) Received: from square.office.ebureau.com (unknown [10.10.21.22]) by internet06.ebureau.com (Postfix) with ESMTPSA id 1DBE9846DA94; Tue, 5 Dec 2017 10:50:44 -0600 (CST) From: Dustin Wenz Message-Id: <3AABC35B-FA6C-4DC6-B70C-F6D1326A7D25@ebureau.com> Content-Type: multipart/signed; boundary="Apple-Mail=_EB8E0C23-94DF-4B05-B0CF-B54FC3AFB16B"; protocol="application/pkcs7-signature"; micalg=sha1 Mime-Version: 1.0 (Mac OS X Mail 11.1 \(3445.4.7\)) Subject: Re: Storage overhead on zvols Date: Tue, 5 Dec 2017 10:50:43 -0600 In-Reply-To: <201712051641.vB5GfR5I052310@pdx.rh.CN85.dnsmgr.net> Cc: Paul Vixie , FreeBSD virtualization To: "Rodney W. Grimes" References: <201712051641.vB5GfR5I052310@pdx.rh.CN85.dnsmgr.net> X-Mailer: Apple Mail (2.3445.4.7) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Dec 2017 16:50:46 -0000 --Apple-Mail=_EB8E0C23-94DF-4B05-B0CF-B54FC3AFB16B Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On Dec 5, 2017, at 10:41 AM, Rodney W. Grimes = wrote: >=20 >>=20 >>=20 >> Dustin Wenz wrote: >>> I'm not using ZFS in my VMs for data integrity (the host already >>> provides that); it's mainly for the easy creation and management of >>> filesystems, and the ability to do snapshots for rollback and >>> replication. >>=20 >> snapshot and replication works fine on the host, acting on the zvol. >=20 > I suspect he is snapshotting and doing send/recvs of something > much less than the zvol, probably some datasetbs, maybe boot > envorinments, a snapshot of the whole zvol is ok if your managing > data at the VM level, not so good if you got lots of stuff going > on inside the VM. Exactly, it's useful to have control of each filesystem discretely. >>> Some of my deployments have hundreds of filesystems in >>> an organized hierarchy, with delegated permissions and automated >>> snapshots, send/recvs, and clones for various operations. >>=20 >> what kind of zpool do you use in the guest, to avoid unwanted = additional=20 >> redundancy? >=20 > Just a simple stripe of 1 device would be my guess, though your > still gona have metadata redundancy. Also correct; just using the zvol virtual device as a single-disk pool. >>=20 >> did you benchmark the space or time efficiency of ZFS vs. UFS? >>=20 >> in some bsd related meeting this year i asked allan jude for a bhyve=20= >> level null mount, so that we could access at / inside the guest some=20= >> subtree of the host, and avoid block devices and file systems=20 >> altogether. right now i have to use nfs for that, which is = irritating. >=20 > This is not as simple as it seems, remember bhyve is just presenting > a hardware environment, hardware environments dont have a file system > concept per se, unlike jails which are providing a software = environment. >=20 > In effect what your asking for is what NFS does, so use NFS and get > over the fact that this is the way to get what you want. Sure you > could implement a virt-vfs but I wonder how close the spec of that > would be to the spec of NFS. >=20 > Or maybe thats the answer, implement virt-vfs as a more effecient way > to transport nfs calls in and out of the guest. I've not done any deliberate comparisons for latency or throughput. What = I've decided to virtualize does not have any exceptional performance = requirements. If I need the best possible IO, I would lean toward using = jails instead of a hypervisor. - .Dustin --Apple-Mail=_EB8E0C23-94DF-4B05-B0CF-B54FC3AFB16B Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIEzDCCBMgw ggOwoAMCAQICAUEwDQYJKoZIhvcNAQELBQAwgZkxCzAJBgNVBAYTAlVTMRIwEAYDVQQIEwlNaW5u ZXNvdGExFDASBgNVBAcTC1NhaW50IENsb3VkMRAwDgYDVQQKEwdlQnVyZWF1MRQwEgYDVQQLEwtJ bnRlZ3JhdGlvbjEUMBIGA1UEAxMLZWJ1cmVhdS5jb20xIjAgBgkqhkiG9w0BCQEWE3N1cHBvcnRA ZWJ1cmVhdS5jb20wHhcNMTcwNTA1MTYxNjE1WhcNMjcwNTAzMTYxNjE1WjBKMQswCQYDVQQGEwJV UzEUMBIGA1UEAwwLRHVzdGluIFdlbnoxJTAjBgkqhkiG9w0BCQEWFmR1c3RpbndlbnpAZWJ1cmVh dS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCQ/HJXe7JhUaexqEaxCNVifsue sUMgohgXLmi5YIcbAWhdxTr1PUzKYkeTkL9sYNjXU6uaI2tZMF3hA9gcFjxQIfkKSo31MrYOgMdU xQe0Q+t6Vd4pGAmtDQDwhAsrNGccADp3Yjy4eBtVfkDGdzz1Y8Lbc684TPFcW7i9+U/dDaXlcxeq fyDqiHZ5y8Lp/1M2Ot/Rz7eikJZTAuHOWKs/PEiJIM2JHuhPyNy+mL2oqEWeOcEsKMNzgn7HVt4k Xz2irBAG+cj4WAxWs418l46EEXgur4PvhBXZMl0LJg0TyaxOHbsUam4R4tbKnaZ3HhRkg79k2Had sb6DKbnCw9/1AgMBAAGjggFnMIIBYzAJBgNVHRMEAjAAMAsGA1UdDwQEAwIE8DAnBgNVHSUEIDAe BggrBgEFBQcDAQYIKwYBBQUHAwIGCCsGAQUFBwMEMB0GA1UdDgQWBBTLi/8HUHpbBEt9OtPqQoax AmpaNDCBzgYDVR0jBIHGMIHDgBRnpZeXB5rQYLgsUKqiiBcLIHyu6aGBn6SBnDCBmTELMAkGA1UE BhMCVVMxEjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAOBgNVBAoT B2VCdXJlYXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNvbTEiMCAG CSqGSIb3DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbYIJAMwZcjAWAsWXMDAGCWCGSAGG+EIBBAQj FiFodHRwOi8vd3d3LmVidXJlYXUuY29tL2NhLWNybC5wZW0wDQYJKoZIhvcNAQELBQADggEBAHbO qVdB9raUKXCgZRA/nES5a60dlIaGnIlpgz+Y3SjFt0bcJxoUYhIzumBHk9yjyP4M1DubOphkQpJ4 LNZbAS01cjCxjnC0ZUq5V3FCeaDwrn1qPY+QJGoZPLlhWdJUNu17OpnR7ZfBWlp3/pRhvNU5PCbJ nmF7rnvsqxUFq9oeiV3SmqBux5lwJ7p2Uss5SHSW6g17K/KdTMK1roQr/+rWpxp2233qddDrLpOE xGRlvhEqSa/IZbGC9oiYmsiaG1PefQkadoob5IMIS5/MDpWHUgSHqAj1V/LwcCx0rbt73SazGMND EzHVWhsj+khepB/MG5QGfWP23IGFmvQYWWcxggOQMIIDjAIBATCBnzCBmTELMAkGA1UEBhMCVVMx EjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAOBgNVBAoTB2VCdXJl YXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNvbTEiMCAGCSqGSIb3 DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbQIBQTAJBgUrDgMCGgUAoIIBxTAYBgkqhkiG9w0BCQMx CwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNzEyMDUxNjUwNDNaMCMGCSqGSIb3DQEJBDEW BBRqFkf4l9VzQL/9GCAzc82q7JG0qDCBsAYJKwYBBAGCNxAEMYGiMIGfMIGZMQswCQYDVQQGEwJV UzESMBAGA1UECBMJTWlubmVzb3RhMRQwEgYDVQQHEwtTYWludCBDbG91ZDEQMA4GA1UEChMHZUJ1 cmVhdTEUMBIGA1UECxMLSW50ZWdyYXRpb24xFDASBgNVBAMTC2VidXJlYXUuY29tMSIwIAYJKoZI hvcNAQkBFhNzdXBwb3J0QGVidXJlYXUuY29tAgFBMIGyBgsqhkiG9w0BCRACCzGBoqCBnzCBmTEL MAkGA1UEBhMCVVMxEjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAO BgNVBAoTB2VCdXJlYXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNv bTEiMCAGCSqGSIb3DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbQIBQTANBgkqhkiG9w0BAQEFAASC AQBWZeELFOLYPi3obweR2RMGGmtuzE1T2JMJoPE7oAdCHKpWYKiiAQufsCmKnfezrXS6XMRzbXPx GLfAVigvz10XjBkCXO6pJw4FWP6VeWovLagFrulhSn4BEcMy8uGkrgdDc6gN9TKne/TYEPrbBCjX wc8Nj+ovDVeZLwpBSi2uGHpVIEH+eMN4FLVOD29L1rar6hUEej8WKJil0u565DVO+5q6DI82/3YZ BIbUzvXfmCNtXErNmFaMxILCZEPhhlOv7pNrgL4WrfP+poMVuSx9lXPDhEx0JftEzPUDdm77nTfk X8iMINn6Uo8JsPTbyib73087L4akVqOqFGUT+OfUAAAAAAAA --Apple-Mail=_EB8E0C23-94DF-4B05-B0CF-B54FC3AFB16B-- From owner-freebsd-virtualization@freebsd.org Tue Dec 5 16:57:22 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C821DE70C1A for ; Tue, 5 Dec 2017 16:57:22 +0000 (UTC) (envelope-from lars@e.0x20.net) Received: from mail.0x20.net (mail.0x20.net [46.251.251.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "0x20.net", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8FE433697; Tue, 5 Dec 2017 16:57:21 +0000 (UTC) (envelope-from lars@e.0x20.net) Received: from e.0x20.net (mail.0x20.net [46.251.251.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.0x20.net (Postfix) with ESMTPS id A3423A8BA5; Tue, 5 Dec 2017 17:57:12 +0100 (CET) Received: (from lars@localhost) by e.0x20.net (8.15.2/8.15.2/Submit) id vB5GvABW052508; Tue, 5 Dec 2017 17:57:10 +0100 (CET) (envelope-from lars) Date: Tue, 5 Dec 2017 17:57:10 +0100 From: Lars Engels To: "Rodney W. Grimes" Cc: Peter Grehan , freebsd-virtualization@freebsd.org Subject: Re: Bhyve. Unable install Windows 7/Server 2008R2 Message-ID: <20171205165710.GK19238@e.0x20.net> References: <176468b0-708d-a0f6-96e9-49c99431e281@freebsd.org> <201712051249.vB5Cnj5f051306@pdx.rh.CN85.dnsmgr.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <201712051249.vB5Cnj5f051306@pdx.rh.CN85.dnsmgr.net> X-Editor: VIM - Vi IMproved 8.0 User-Agent: Mutt/1.9.1 (2017-09-22) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Dec 2017 16:57:22 -0000 On Tue, Dec 05, 2017 at 04:49:45AM -0800, Rodney W. Grimes wrote: > > Hi, > > > > > bhyve -c 2 -s 0,hostbridge -s 3,ahci-hd,/dev/zvol/my_zroot/VM/img/win7/disk0 > > > > For win7/2k8*, the sector size presented to the guest has to be forced > > to 512 bytes i.e. for the ahci-cd slot, the config would look like: > > > > -s 3,ahci-hd,/dev/zvol/my_zroot/VM/img/win7/disk0,sectorsize=512 > > Should we start a wiki page on "Guest Quirks"? Please do. -- Lars From owner-freebsd-virtualization@freebsd.org Tue Dec 5 17:10:50 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B430BE713F4 for ; Tue, 5 Dec 2017 17:10:50 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 52AA66358E for ; Tue, 5 Dec 2017 17:10:50 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mh0.gentlemail.de (mh0.gentlemail.de [IPv6:2a00:e10:2800::a135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id vB5HAlma099877; Tue, 5 Dec 2017 18:10:47 +0100 (CET) (envelope-from freebsd@omnilan.de) Received: from titan.inop.mo1.omnilan.net (s1.omnilan.de [217.91.127.234]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id DF12743B; Tue, 5 Dec 2017 18:10:46 +0100 (CET) Message-ID: <5A26D316.5060804@omnilan.de> Date: Tue, 05 Dec 2017 18:10:46 +0100 From: Harry Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: Lars Engels CC: "Rodney W. Grimes" , freebsd-virtualization@freebsd.org Subject: Re: Bhyve. Unable install Windows 7/Server 2008R2 References: <176468b0-708d-a0f6-96e9-49c99431e281@freebsd.org> <201712051249.vB5Cnj5f051306@pdx.rh.CN85.dnsmgr.net> <20171205165710.GK19238@e.0x20.net> In-Reply-To: <20171205165710.GK19238@e.0x20.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]); Tue, 05 Dec 2017 18:10:47 +0100 (CET) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: ; Sender-helo: mh0.gentlemail.de; ) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Dec 2017 17:10:50 -0000 Bezglich Lars Engels's Nachricht vom 05.12.2017 17:57 (localtime): > On Tue, Dec 05, 2017 at 04:49:45AM -0800, Rodney W. Grimes wrote: >>> Hi, >>> >>>> bhyve -c 2 -s 0,hostbridge -s 3,ahci-hd,/dev/zvol/my_zroot/VM/img/win7/disk0 >>> For win7/2k8*, the sector size presented to the guest has to be forced >>> to 512 bytes i.e. for the ahci-cd slot, the config would look like: >>> >>> -s 3,ahci-hd,/dev/zvol/my_zroot/VM/img/win7/disk0,sectorsize=512 >> Should we start a wiki page on "Guest Quirks"? > Please do. I'd vote for making ,"sectorsize=512/4096" a artificial default. I'm not entire sure about the different Windows versions, but since BHYVE_UEFI.fd isn't able to access drives other than 512b sector size, I think it doesn't make sense to provide 8k drives by default in case of default ZFS volume. If I remember correctly Win6.3 (8.1/Server2k12) did support 4k and 8k native drives, with the limitation that "windows internal database - mssql" might not be usable, but I couldn't UEFI boot unless I changed sectrosize to 512 and stripezesize to 4k/8k. I guess not may other operating systems support bigger sector sizes as well as FreeBSD does today. So this artifical format might be what most times makes most sense. -harry From owner-freebsd-virtualization@freebsd.org Tue Dec 5 20:21:06 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9249BE800CB for ; Tue, 5 Dec 2017 20:21:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7FED06D533 for ; Tue, 5 Dec 2017 20:21:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vB5KL4fU066842 for ; Tue, 5 Dec 2017 20:21:06 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 213333] FreeBSD 11-RELEASE fails to boot under KVM/Qemu Date: Tue, 05 Dec 2017 20:21:05 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cam@neo-zeon.de X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Dec 2017 20:21:06 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D213333 --- Comment #17 from Cameron --- I've managed to work around this.=20 There's 2 issues here: 1. Rather than using an Opteron_* model in kvm for the CPU, etc. Use kvm64. Others may work as well. I believe this alone will Just Work if you haven't been building from source. 2. Did you build everything/anything from source? You probably don't want to set=20 'CPUTYPE?=3Dnative', etc in /etc/make.conf (or perhaps you can find somethi= ng "safe" to set it to?).=20 If you are building from source AND had CPUTYPE set to something like Opteron_*, you'll probably need to rebuild your *current* working version before you can attempt to switch the CPU model to 'kvm64'. An Opteron optim= ized build won't even boot if you set the CPU model to 'kvm64' IIRC. First, unset CPUTYPE if you have it set and then rebuild _everything_. Rebu= ild world, kernel, and ports. Once you've confirmed that everything still works (you can reboot, your daemons still come up, etc), you should be able to sw= itch your virtual CPU model to kvm64. Once your system runs with with kvm64 set, you can build the newer version = of FreeBSD that you want to move to. To be safe, try the newer kernel first (as you should in general anyway) to confirm that it's working without surprise= s. Seeing as how I found that much more than the kernel and the base system we= re affected, this could very well be a compiler regression for the Opteron tar= get. Anyway, hope this helps! --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Tue Dec 5 21:40:15 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 32D63E83732 for ; Tue, 5 Dec 2017 21:40:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 20A4573284 for ; Tue, 5 Dec 2017 21:40:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vB5LeE5B014896 for ; Tue, 5 Dec 2017 21:40:14 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 215972] Bhyve crash more then 1 cpu AMD Date: Tue, 05 Dec 2017 21:40:14 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: domhauton@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Dec 2017 21:40:15 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D215972 domhauton@gmail.com changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |domhauton@gmail.com --- Comment #22 from domhauton@gmail.com --- Hey, Is there a solution or has there been any progress on debugging this? I'm getting the same issue with a 1100T on Win10 Pro / Win10 Education / Windows Server 2016 Datacenter. I've been trying to setup surveillance software which unfortunately needs m= ore than one core. Many Thanks, Dom --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Tue Dec 5 21:53:33 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 53ED6E8434C for ; Tue, 5 Dec 2017 21:53:33 +0000 (UTC) (envelope-from randy@terbush.org) Received: from mail-oi0-x22d.google.com (mail-oi0-x22d.google.com [IPv6:2607:f8b0:4003:c06::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 245F47435E for ; Tue, 5 Dec 2017 21:53:32 +0000 (UTC) (envelope-from randy@terbush.org) Received: by mail-oi0-x22d.google.com with SMTP id f69so1251336oig.10 for ; Tue, 05 Dec 2017 13:53:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=terbush.org; s=google; h=mime-version:from:date:message-id:subject:to; bh=6X3mXUBs8b7IslxeqrmufsoUEkAHOfSFkaI8wzlGf2Q=; b=QlMwuIc1CdhC3ww1YNPhtDmZjINv1q3POwc3x3RLTlKdQ0W383MfQfsNyhKge3LZy9 3OBj5aF+lZa7SPuX/jZRK/ZnKq0laLp2YTahpR+pXCoTBc1+k223AO3sXA3KXfncsHAX G8V1paHBvZUniLMB27qBueyAV3MMNZKIgmTF4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=6X3mXUBs8b7IslxeqrmufsoUEkAHOfSFkaI8wzlGf2Q=; b=FsVIfHMBC3O+JMObLa1RCn/9O4AcVyTZ7mHy+fPLTSl5ezQ3i6EZ5RnbDURtfi5xyg jigaUwRes3M6s3iLIBDJKchpX8HkZ/rK/QO8LDUWhJVwR2qLpgZTVOLkiS1aP2mH+Sg9 CGtwdv8L5ohkxWqsVZkTQGCoNErDmbM52RcEJXQKOlCZCGla3aXSycZwelz5x3ykm+eV D3H2ZubwAQCFyoPDU6Jfb33ClxSsYuBoxfTopBKjp4Z0vDntC/dRRoCnhV0ecTqIR7xQ aiQRuHRpWyGYXYl5BbUSjDrJnC4rHmOD8B+o9ZdWn0B32PLfqI51yx+qDP07zhGvbct/ rRBw== X-Gm-Message-State: AKGB3mJcwTUfLzmfDGLARz+2n3Daw1FkYoEU7lC5Qj6rPatyXAGA4LMf Q88zI/DDpW3zsRdYdJ5fjHxbuNsv7QC5Gr77jPiNDDtWsME= X-Google-Smtp-Source: AGs4zMaMBibzj5MPRI09zkUfio4piLKD/CdGy7BFcc2rTxnfun+GVxrodiHsCEDIALZiITJorKvgM3QPoW5OTTocJD4= X-Received: by 10.202.189.195 with SMTP id n186mr5779953oif.96.1512510811940; Tue, 05 Dec 2017 13:53:31 -0800 (PST) MIME-Version: 1.0 Received: by 10.168.67.10 with HTTP; Tue, 5 Dec 2017 13:53:01 -0800 (PST) From: Randy Terbush Date: Tue, 5 Dec 2017 14:53:01 -0700 Message-ID: Subject: Recovering an ZFS vm To: "freebsd-virtua." Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Dec 2017 21:53:33 -0000 I have a Centos vm that has suddenly stopped booting. At the console, grub tells me the following if I attempt to list any of the available partitions. error: not a correct XFS inode. error: not a correct XFS inode. error: not a correct XFS inode. error: not a correct XFS inode. error: not a correct XFS inode. Filesystem type xfs, UUID 7652ffda-f7c5-408a-b0ce-b554b66fc2e5 - Partition start at 2048 - Total size 2097152 sectors grub> Is there an easy way to recover this? This has happened more than once. Just so happens there is something on this image I would like to have access to... -- Randy From owner-freebsd-virtualization@freebsd.org Tue Dec 5 22:53:55 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7FC77E85D78 for ; Tue, 5 Dec 2017 22:53:55 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 61B1A76CEC for ; Tue, 5 Dec 2017 22:53:54 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from Ticonderoga.HML3.ScaleEngine.net (senat1-01.HML3.ScaleEngine.net [209.51.186.5]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id 817D213D93 for ; Tue, 5 Dec 2017 22:53:53 +0000 (UTC) Subject: Re: Recovering an ZFS vm To: freebsd-virtualization@freebsd.org References: From: Allan Jude Message-ID: Date: Tue, 5 Dec 2017 17:53:53 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Dec 2017 22:53:55 -0000 On 12/05/2017 16:53, Randy Terbush wrote: > I have a Centos vm that has suddenly stopped booting. At the console, grub > tells me the following if I attempt to list any of the available partitions. > > error: not a correct XFS inode. > error: not a correct XFS inode. > error: not a correct XFS inode. > error: not a correct XFS inode. > error: not a correct XFS inode. > Filesystem type xfs, UUID 7652ffda-f7c5-408a-b0ce-b554b66fc2e5 - Partition > start at 2048 - Total size 2097152 sectors > grub> > > Is there an easy way to recover this? This has happened more than once. > Just so happens there is something on this image I would like to have > access to... > > -- > Randy > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" > Is this grub-bhyve? It does not (yet) support the newer version of XFS with checksums. -- Allan Jude From owner-freebsd-virtualization@freebsd.org Tue Dec 5 23:25:41 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 12FD1E86A41 for ; Tue, 5 Dec 2017 23:25:41 +0000 (UTC) (envelope-from randy@terbush.org) Received: from mail-ot0-x234.google.com (mail-ot0-x234.google.com [IPv6:2607:f8b0:4003:c0f::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CA61A784BF for ; Tue, 5 Dec 2017 23:25:40 +0000 (UTC) (envelope-from randy@terbush.org) Received: by mail-ot0-x234.google.com with SMTP id e74so1815925ote.7 for ; Tue, 05 Dec 2017 15:25:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=terbush.org; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=nZqdYLPKb3N3sJqT3Pvve8+KON/h36nmtBJ5IPMc63I=; b=RHvZe3rjtqTJFLYrc8vhp0Zggziu/rz6X1975wcv6uKonAT772Ldo6ABNgNbQXmQJ3 jDCmz367YKxIIXmo3oCiN5Ku5eixCpJK8XdIG/XvLaMHFJR3kcYJRoPQfuaf0fz95eEa u9L0mrUMXyRIGTQ+GRjh30TTJhay8RlGO5wlc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=nZqdYLPKb3N3sJqT3Pvve8+KON/h36nmtBJ5IPMc63I=; b=e12CPVXnCeIzwTW7mxVxF0hQyvN6HZvP9WWNdMVGqty0L2dfw6O5iup0wA1LeRSzYK O/OIt3+wRIC3W8TMd9gHxqjEUPn3a1xqvSPqIMtxZBh3rzqnpkql39QcYSGcwalUgeM6 H1k+hY1Lrax2JyUiOEpykGQ1OOxfdVd8j75To+XYxY9MPioYPldov3KMDgpvSZFmfvHu 7RsH/vzii8mulLH6fXESu110TBWIUm79sVMDOK+3vH0zEOXWb4M333XpOoSX+AdI4mbL 82iDEF1p8MZnoRTaRVr8s+YeQlwkK+eoJeHAGgKjE3xJMYWLwzmkYsQmAYhXwNpoCJgd pZlg== X-Gm-Message-State: AJaThX6PTR0gbSY4dAkTxDW1hE3LUyQEsOAI6PlJaq9hf2CKIvsXLlMM atyaPkIQCSTu41/TUyAW7o5pjSIZMne+9KdDjiHmmQ== X-Google-Smtp-Source: AGs4zMZex557RU1ny3wLTkMGyWHI30mFt6yGTA9pkdlz7LY3Zvt6GqzJ/0xxXQAHnz+aFxIJudphjwOX6t7VwF4vIsc= X-Received: by 10.157.29.146 with SMTP id y18mr22092620otd.371.1512516339463; Tue, 05 Dec 2017 15:25:39 -0800 (PST) MIME-Version: 1.0 Received: by 10.168.67.10 with HTTP; Tue, 5 Dec 2017 15:25:09 -0800 (PST) In-Reply-To: References: From: Randy Terbush Date: Tue, 5 Dec 2017 16:25:09 -0700 Message-ID: Subject: Re: Recovering an ZFS vm To: Allan Jude Cc: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Dec 2017 23:25:41 -0000 Allan, Confused by the question. This is a VM that has been running. Loader is 'grub'. Not sure if that implies bhyve given it is running on the bhyve hyperviser. I have another VM that was cloned from this one that is running fine. I did just build from stable yesterday and did a reboot which is when I find that this VM does not run. Not sure if it was corrupted because of a bad shutdown caused by the reboot, or if there is something more explainable as you suggest. I appreciate your help. -- Randy On Tue, Dec 5, 2017 at 3:53 PM, Allan Jude wrote: > On 12/05/2017 16:53, Randy Terbush wrote: > > I have a Centos vm that has suddenly stopped booting. At the console, > grub > > tells me the following if I attempt to list any of the available > partitions. > > > > error: not a correct XFS inode. > > error: not a correct XFS inode. > > error: not a correct XFS inode. > > error: not a correct XFS inode. > > error: not a correct XFS inode. > > Filesystem type xfs, UUID 7652ffda-f7c5-408a-b0ce-b554b66fc2e5 - > Partition > > start at 2048 - Total size 2097152 sectors > > grub> > > > > Is there an easy way to recover this? This has happened more than once. > > Just so happens there is something on this image I would like to have > > access to... > > > > -- > > Randy > > _______________________________________________ > > freebsd-virtualization@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > > To unsubscribe, send any mail to "freebsd-virtualization- > unsubscribe@freebsd.org" > > > > Is this grub-bhyve? It does not (yet) support the newer version of XFS > with checksums. > > -- > Allan Jude > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization- > unsubscribe@freebsd.org" > From owner-freebsd-virtualization@freebsd.org Tue Dec 5 23:56:40 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 68BDEE8762D for ; Tue, 5 Dec 2017 23:56:40 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id 20CDC79453 for ; Tue, 5 Dec 2017 23:56:39 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id A2E8E20AE974 for ; Wed, 6 Dec 2017 09:56:36 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 9DEA6280A03 for ; Wed, 6 Dec 2017 09:56:36 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id mVVcZrfpGAwM for ; Wed, 6 Dec 2017 09:56:36 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (96-82-80-65-static.hfc.comcastbusiness.net [96.82.80.65]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 9A4582809A8; Wed, 6 Dec 2017 09:56:34 +1000 (AEST) Subject: Re: Recovering an ZFS vm To: Randy Terbush References: Cc: "freebsd-virtua." From: Peter Grehan Message-ID: <81d05d9d-044a-9cad-40e3-5ddf86da6570@freebsd.org> Date: Tue, 5 Dec 2017 15:56:29 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.2 cv=XtWKARN9 c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=mwgbnDbW7alINpy3vhoKyg==:17 a=IkcTkHD0fZMA:10 a=ocR9PWop10UA:10 a=vCW1xSsoxNASy_69gQsA:9 a=QEXdDO2ut3YA:10 wl=host:3 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Dec 2017 23:56:40 -0000 Hi Randy, > I have a Centos vm that has suddenly stopped booting. At the console, grub > tells me the following if I attempt to list any of the available partitions. > > error: not a correct XFS inode. > error: not a correct XFS inode. > error: not a correct XFS inode. > error: not a correct XFS inode. > error: not a correct XFS inode. > Filesystem type xfs, UUID 7652ffda-f7c5-408a-b0ce-b554b66fc2e5 - Partition > start at 2048 - Total size 2097152 sectors > grub> > > Is there an easy way to recover this? This has happened more than once. > Just so happens there is something on this image I would like to have > access to... Looks like the grub partition was upgraded to the version of XFS that has the CRC feature enabled (7.2 ?). Unfortunately this feature is not understood by grub-bhyve :( One way to recover the disk is to create a new VM with the most recent CentOS, but using UEFI for the bootloader. Then, add this disk to the guest, and from within the guest I think you can run an XFS utility that will disable the use of CRCs on that partition. The proper fix would be for grub-bhyve to be updated to the latest version of grub2, though a workaround is to create guests with UEFI and not use grub-bhyve. later, Peter. From owner-freebsd-virtualization@freebsd.org Wed Dec 6 05:50:34 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 90A75E91A49 for ; Wed, 6 Dec 2017 05:50:34 +0000 (UTC) (envelope-from randy@terbush.org) Received: from mail-oi0-x236.google.com (mail-oi0-x236.google.com [IPv6:2607:f8b0:4003:c06::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 56CF163E99 for ; Wed, 6 Dec 2017 05:50:34 +0000 (UTC) (envelope-from randy@terbush.org) Received: by mail-oi0-x236.google.com with SMTP id j17so1857869oih.3 for ; Tue, 05 Dec 2017 21:50:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=terbush.org; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=qqwq3K09i5kSCOfw6XapPiWhcFDbtv05gpFe+BuLHtM=; b=h/Th5c9oJ34i5UOE5kTa7NdT9L3y3j2m2XXmH3zojvksFhMNyF7QkLZjxOUDy7AmFg H2tLvReSnfxfzutYRlc2UMD+aNQjXfvIlbi1I5savU8YK5JDjGAyyCHCc+xdhInjCLTg x4qIZc+5IzlLOir5K5PWao+CWrEMLz3vbprdo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=qqwq3K09i5kSCOfw6XapPiWhcFDbtv05gpFe+BuLHtM=; b=bxZmszfhz69cBXqApNxY4DHvkoZp4RUErW9GLmlWZKbVxWLT8P1x5pxFx6wQZltUwy G1AM3+WBKv0eS7jz+8gQMsAk3AMr9QZ3fjX1e5u/AQ9dkcT1yFIZ/DCSRXLXt1KTVAex 30t66JoYsIRbuesA/cM7RW5EUjRcQbedH76HXtnfsrJxxdGuiO9EUr9mAzSlks2uMVvA CaaKivBfAFawkx6yd230IWZOsy7qoFVVADYAGTJl+RhpMvm25DAZ6Hjjb+6XtihsGLz5 4YeAFtWBywI27vS5vAkv9bup6Rp3eiwv5PI5HzhXWozYauYIjw9uQZ040if65WFTZAZN 6Cgw== X-Gm-Message-State: AJaThX6EBzjTDLEYZIaH4dW66ck3Rz7CQwsjCxUT2nvTi29L7ekudIuR /BUuRh7dUeKWwKuIM1iifWpchLxMCt/Gc78220cjow== X-Google-Smtp-Source: AGs4zMYu8OK8h/BbMiAZg5kqU4sa9s9PNqjcUcvCzRA3PNNAimSUzHqIY69qw4AzWY8aq7C6e4bTqAAgl1ZsnGyI1MY= X-Received: by 10.202.5.132 with SMTP id 126mr19900119oif.276.1512539433403; Tue, 05 Dec 2017 21:50:33 -0800 (PST) MIME-Version: 1.0 Received: by 10.168.67.10 with HTTP; Tue, 5 Dec 2017 21:50:02 -0800 (PST) In-Reply-To: <81d05d9d-044a-9cad-40e3-5ddf86da6570@freebsd.org> References: <81d05d9d-044a-9cad-40e3-5ddf86da6570@freebsd.org> From: Randy Terbush Date: Tue, 5 Dec 2017 22:50:02 -0700 Message-ID: Subject: Re: Recovering an ZFS vm To: Peter Grehan Cc: "freebsd-virtua." Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Dec 2017 05:50:34 -0000 One of the other VM clones is running. What do I need to do to mount the sparse-zvol dataset that is this disk image that won't boot? I'm still confused as to why one of these VM images would boot and not the other. They are both Centos 7 1708. At any rate, before taking a chance of shutting this image down, I'd appreciate any help to mount this other zvol and make sure the crc feature is disabled. Thanks -- Randy On Tue, Dec 5, 2017 at 4:56 PM, Peter Grehan wrote: > Hi Randy, > > I have a Centos vm that has suddenly stopped booting. At the console, grub >> tells me the following if I attempt to list any of the available >> partitions. >> >> error: not a correct XFS inode. >> error: not a correct XFS inode. >> error: not a correct XFS inode. >> error: not a correct XFS inode. >> error: not a correct XFS inode. >> Filesystem type xfs, UUID 7652ffda-f7c5-408a-b0ce-b554b66fc2e5 - >> Partition >> start at 2048 - Total size 2097152 sectors >> grub> >> >> Is there an easy way to recover this? This has happened more than once. >> Just so happens there is something on this image I would like to have >> access to... >> > > Looks like the grub partition was upgraded to the version of XFS that has > the CRC feature enabled (7.2 ?). Unfortunately this feature is not > understood by grub-bhyve :( > > One way to recover the disk is to create a new VM with the most recent > CentOS, but using UEFI for the bootloader. Then, add this disk to the > guest, and from within the guest I think you can run an XFS utility that > will disable the use of CRCs on that partition. > > The proper fix would be for grub-bhyve to be updated to the latest > version of grub2, though a workaround is to create guests with UEFI and not > use grub-bhyve. > > later, > > Peter. > From owner-freebsd-virtualization@freebsd.org Wed Dec 6 08:45:58 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7B84CE99A67 for ; Wed, 6 Dec 2017 08:45:58 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DF8A56907F for ; Wed, 6 Dec 2017 08:45:57 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from hugo10.ka.punkt.de (hugo10.ka.punkt.de [217.29.44.10]) by gate2.intern.punkt.de with ESMTP id vB68jnd0046713; Wed, 6 Dec 2017 09:45:49 +0100 (CET) Received: from [217.29.44.195] ([217.29.44.195]) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id vB68jmhi067554; Wed, 6 Dec 2017 09:45:48 +0100 (CET) (envelope-from hausen@punkt.de) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: NFS alternatives (was: Re: Storage overhead on zvols) From: "Patrick M. Hausen" In-Reply-To: <201712051641.vB5GfR5I052310@pdx.rh.CN85.dnsmgr.net> Date: Wed, 6 Dec 2017 09:45:47 +0100 Cc: Paul Vixie , FreeBSD virtualization Content-Transfer-Encoding: quoted-printable Message-Id: <4A321A55-23FA-42AB-BF65-3DCA3464307D@punkt.de> References: <201712051641.vB5GfR5I052310@pdx.rh.CN85.dnsmgr.net> To: "Rodney W. Grimes" X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Dec 2017 08:45:58 -0000 Hi all, > Am 05.12.2017 um 17:41 schrieb Rodney W. Grimes = : > In effect what your asking for is what NFS does, so use NFS and get > over the fact that this is the way to get what you want. Sure you > could implement a virt-vfs but I wonder how close the spec of that > would be to the spec of NFS. I figure it should be possible to implement something simpler than NFS that provides full local posix semantics under the constraint that only one "client" is allowed to mount the FS at a time. I see quite a few applications for something like this, specifically in "hyperconvergent" environments. Or vagrant, of course. *scratching head* isn't this what Sun's "network disk" protocol = provided? Kind regards, Patrick --=20 punkt.de GmbH Internet - Dienstleistungen - Beratung Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100 76133 Karlsruhe info@punkt.de http://punkt.de AG Mannheim 108285 Gf: Juergen Egeling From owner-freebsd-virtualization@freebsd.org Wed Dec 6 09:04:49 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5C79DE9A0E6 for ; Wed, 6 Dec 2017 09:04:49 +0000 (UTC) (envelope-from paul.g.webster@googlemail.com) Received: from mail-yw0-x234.google.com (mail-yw0-x234.google.com [IPv6:2607:f8b0:4002:c05::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 074006993D; Wed, 6 Dec 2017 09:04:49 +0000 (UTC) (envelope-from paul.g.webster@googlemail.com) Received: by mail-yw0-x234.google.com with SMTP id y187so1270855ywd.12; Wed, 06 Dec 2017 01:04:48 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=wLLNVYBRYQcJbnsQotT1LpdIKQBhnDtEyOvMm0yNtIQ=; b=XwJwnHA4Sfs6VqXHNlTjw02bYFd/uHirFqYP9lTuq0PHaW2rxxefM2C2IPBzM8RNOe TLpOGfNqkLo8uDpe/X/rfBuY/ou03/g7uDPlnuWy7dDoT8rZsN4CElny10AA4xWNO8p8 zpXl9fGiESfSmMJColRslQIwuab0AYwyc9iLcy2mgP9HPnuFPxTFVx75llUptAHZEpNO /pyiIGUWWGI04qo8KWXM7qFB80O01JYZ4bn/eRIe1chAmzO7wrQ8MT/8CT1hLBVSw458 o3yUtkW33sOGHEwWFbCWkO9QQth6erqQBrqZp4Y4eMoVMLMKXCSfPcUpQ5mGEQbgaxjQ cIyw== X-Gm-Message-State: AJaThX6PiBVKCNYYubLQPhebRU84mzHudt0o0qH3Ct1So4RcOblZrnFR ReSMDjsx3xySxfsqbLx0NnAMd2yzBFgS7ZWGJwo= X-Google-Smtp-Source: AGs4zMbo8dfctKMtdWQXzk+bpRoZeZNLFaVOZ3jtydwLYVnqFMnoBUkl353SUKqoHB/8LQAk5HF6+Ml9pvnQYGjQW1Q= X-Received: by 10.129.118.74 with SMTP id j10mr15271194ywk.152.1512551088006; Wed, 06 Dec 2017 01:04:48 -0800 (PST) MIME-Version: 1.0 Received: by 10.37.165.8 with HTTP; Wed, 6 Dec 2017 01:04:47 -0800 (PST) In-Reply-To: References: <81d05d9d-044a-9cad-40e3-5ddf86da6570@freebsd.org> From: Paul Webster Date: Wed, 6 Dec 2017 09:04:47 +0000 Message-ID: Subject: Re: Recovering an ZFS vm To: Randy Terbush Cc: Peter Grehan , "freebsd-virtua." Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Dec 2017 09:04:49 -0000 if you can get to a system that is running the same kernel, you could build A compativle kernel with xfs in it and what not, stick it on a small '/boot' of your own and include that on your bhyve line, so the kernel is booted and then it mounts your existing system On 6 December 2017 at 05:50, Randy Terbush wrote: > One of the other VM clones is running. What do I need to do to mount the > sparse-zvol dataset that is this disk image that won't boot? > > I'm still confused as to why one of these VM images would boot and not the > other. They are both Centos 7 1708. At any rate, before taking a chance of > shutting this image down, I'd appreciate any help to mount this other zvol > and make sure the crc feature is disabled. > > Thanks > > -- > Randy > > On Tue, Dec 5, 2017 at 4:56 PM, Peter Grehan wrote: > > > Hi Randy, > > > > I have a Centos vm that has suddenly stopped booting. At the console, > grub > >> tells me the following if I attempt to list any of the available > >> partitions. > >> > >> error: not a correct XFS inode. > >> error: not a correct XFS inode. > >> error: not a correct XFS inode. > >> error: not a correct XFS inode. > >> error: not a correct XFS inode. > >> Filesystem type xfs, UUID 7652ffda-f7c5-408a-b0ce-b554b66fc2e5 - > >> Partition > >> start at 2048 - Total size 2097152 sectors > >> grub> > >> > >> Is there an easy way to recover this? This has happened more than once. > >> Just so happens there is something on this image I would like to have > >> access to... > >> > > > > Looks like the grub partition was upgraded to the version of XFS that > has > > the CRC feature enabled (7.2 ?). Unfortunately this feature is not > > understood by grub-bhyve :( > > > > One way to recover the disk is to create a new VM with the most recent > > CentOS, but using UEFI for the bootloader. Then, add this disk to the > > guest, and from within the guest I think you can run an XFS utility that > > will disable the use of CRCs on that partition. > > > > The proper fix would be for grub-bhyve to be updated to the latest > > version of grub2, though a workaround is to create guests with UEFI and > not > > use grub-bhyve. > > > > later, > > > > Peter. > > > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization- > unsubscribe@freebsd.org" > From owner-freebsd-virtualization@freebsd.org Wed Dec 6 09:06:44 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 69830E9A14B for ; Wed, 6 Dec 2017 09:06:44 +0000 (UTC) (envelope-from paul.g.webster@googlemail.com) Received: from mail-yw0-x229.google.com (mail-yw0-x229.google.com [IPv6:2607:f8b0:4002:c05::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1E9DF699B0; Wed, 6 Dec 2017 09:06:44 +0000 (UTC) (envelope-from paul.g.webster@googlemail.com) Received: by mail-yw0-x229.google.com with SMTP id t189so1277799ywg.9; Wed, 06 Dec 2017 01:06:44 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=AQPhIHxCn6lxVXFCmQ8pVgMkQguA38HNxmQY7aPWvsk=; b=RCZqqHyX88sJWWPrTGDbmuXEwyaftYA8r1XglAo8RnmEbeB4/2acdackblFnlHx2AU +18vtEzjwWb4WaM8UU1z1/SBP71C/1aXbBTR2tW7QXl5Lgvv+7sHwotpX+8eJ1xd9hMp /tlP8FDpq59gUzkLs7WPL8XH1MF7EssSdkRQvBLGHKcNsva/+MJvjLkAOwxS4xgE+Puu gegMYZEQ7YoZiBPS9C2AcEbeuoPwuL+CsuYJ4/f5Xb7xUgMn9UjAmJ7vb0rf8Ma0idNl bMD2zmwrNMAvXIruu1aoz4rYV0ChZsKo33WrJ1eUJf0Jc6z1JHZN03kTZtWGJmyywS6r 1qVg== X-Gm-Message-State: AJaThX5tTEkrbwv7Vq7qqKSbmWojJnFuHB9oA2sFND9nOuw+5d0ZP4mC w56PydtWSnfgJwNXdsXmJr4v4N2Fp/sJOprsWbY= X-Google-Smtp-Source: AGs4zMb6viJhfEwTZJqbqID47YyjIZ6Qwsk+kpg+i1HMJs02vZUIYyv0ewHki8En2HE9iXKJhyAqRXbvOFsHtd+U2Xk= X-Received: by 10.129.109.85 with SMTP id i82mr14687423ywc.444.1512551203229; Wed, 06 Dec 2017 01:06:43 -0800 (PST) MIME-Version: 1.0 Received: by 10.37.165.8 with HTTP; Wed, 6 Dec 2017 01:06:42 -0800 (PST) In-Reply-To: References: <81d05d9d-044a-9cad-40e3-5ddf86da6570@freebsd.org> From: Paul Webster Date: Wed, 6 Dec 2017 09:06:42 +0000 Message-ID: Subject: Re: Recovering an ZFS vm To: Randy Terbush Cc: Peter Grehan , "freebsd-virtua." Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Dec 2017 09:06:44 -0000 Or as an alternative that just come to mind if your just wanting to 'save the system' boot the gentoo live DVD from the UEFI loader, that will get you a live XFS supportive shell you can then setup basic networking from and sync your important stuff elsewhere On 6 December 2017 at 09:04, Paul Webster wrote: > if you can get to a system that is running the same kernel, you could > build A compativle kernel with xfs in it and what not, stick it on a small > '/boot' of your own and include that on your bhyve line, so the kernel is > booted and then it mounts your existing system > > On 6 December 2017 at 05:50, Randy Terbush wrote: > >> One of the other VM clones is running. What do I need to do to mount the >> sparse-zvol dataset that is this disk image that won't boot? >> >> I'm still confused as to why one of these VM images would boot and not the >> other. They are both Centos 7 1708. At any rate, before taking a chance of >> shutting this image down, I'd appreciate any help to mount this other zvol >> and make sure the crc feature is disabled. >> >> Thanks >> >> -- >> Randy >> >> On Tue, Dec 5, 2017 at 4:56 PM, Peter Grehan wrote: >> >> > Hi Randy, >> > >> > I have a Centos vm that has suddenly stopped booting. At the console, >> grub >> >> tells me the following if I attempt to list any of the available >> >> partitions. >> >> >> >> error: not a correct XFS inode. >> >> error: not a correct XFS inode. >> >> error: not a correct XFS inode. >> >> error: not a correct XFS inode. >> >> error: not a correct XFS inode. >> >> Filesystem type xfs, UUID 7652ffda-f7c5-408a-b0ce-b554b66fc2e5 - >> >> Partition >> >> start at 2048 - Total size 2097152 sectors >> >> grub> >> >> >> >> Is there an easy way to recover this? This has happened more than once. >> >> Just so happens there is something on this image I would like to have >> >> access to... >> >> >> > >> > Looks like the grub partition was upgraded to the version of XFS that >> has >> > the CRC feature enabled (7.2 ?). Unfortunately this feature is not >> > understood by grub-bhyve :( >> > >> > One way to recover the disk is to create a new VM with the most recent >> > CentOS, but using UEFI for the bootloader. Then, add this disk to the >> > guest, and from within the guest I think you can run an XFS utility that >> > will disable the use of CRCs on that partition. >> > >> > The proper fix would be for grub-bhyve to be updated to the latest >> > version of grub2, though a workaround is to create guests with UEFI and >> not >> > use grub-bhyve. >> > >> > later, >> > >> > Peter. >> > >> _______________________________________________ >> freebsd-virtualization@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >> To unsubscribe, send any mail to "freebsd-virtualization-unsubs >> cribe@freebsd.org" >> > > From owner-freebsd-virtualization@freebsd.org Wed Dec 6 13:04:26 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 517D7EA00EF for ; Wed, 6 Dec 2017 13:04:26 +0000 (UTC) (envelope-from paul@redbarn.org) Received: from family.redbarn.org (family.redbarn.org [IPv6:2001:559:8000:cd::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 40C9D71869 for ; Wed, 6 Dec 2017 13:04:26 +0000 (UTC) (envelope-from paul@redbarn.org) Received: from [100.72.214.65] (25.sub-174-214-3.myvzw.com [174.214.3.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by family.redbarn.org (Postfix) with ESMTPSA id 46D3661FA2; Wed, 6 Dec 2017 13:04:19 +0000 (UTC) Date: Wed, 06 Dec 2017 13:04:09 +0000 User-Agent: K-9 Mail for Android In-Reply-To: <4A321A55-23FA-42AB-BF65-3DCA3464307D@punkt.de> References: <201712051641.vB5GfR5I052310@pdx.rh.CN85.dnsmgr.net> <4A321A55-23FA-42AB-BF65-3DCA3464307D@punkt.de> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Re: NFS alternatives (was: Re: Storage overhead on zvols) To: "Patrick M. Hausen" , "Rodney W. Grimes" CC: FreeBSD virtualization From: P Vix Message-ID: <7ECC808B-0C02-4E2D-974C-D363DCC1A046@redbarn.org> X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Dec 2017 13:04:26 -0000 On December 6, 2017 5:45:47 PM GMT+09:00, "Patrick M=2E Hausen" wrote: >Hi all, > >I see quite a few applications for something like this, specifically >in "hyperconvergent" environments=2E Or vagrant, of course=2E +1=2E >*scratching head* isn't this what Sun's "network disk" protocol >provided? No=2E Nd was like iscsi=2E --=20 Sent from my Android device with K-9 Mail=2E Please excuse my brevity=2E From owner-freebsd-virtualization@freebsd.org Wed Dec 6 14:37:19 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7098BE81B78 for ; Wed, 6 Dec 2017 14:37:19 +0000 (UTC) (envelope-from amvandemore@gmail.com) Received: from mail-io0-x234.google.com (mail-io0-x234.google.com [IPv6:2607:f8b0:4001:c06::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3552A7479A for ; Wed, 6 Dec 2017 14:37:19 +0000 (UTC) (envelope-from amvandemore@gmail.com) Received: by mail-io0-x234.google.com with SMTP id s19so2855504ioa.1 for ; Wed, 06 Dec 2017 06:37:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=vl0+huPO34QfWeEMf7yKiM4C/TEwPYCaNXjbcVQ4QNQ=; b=ehH8qeSjzksMDCKhjLa1KS2Zs3TjiePbe6r5gXS4yJf8b+FPRtiYr9HYZFjgu0IAC2 2RMoz92J9pH1DC5LS4I4HSlEoCaopk18Nolx+g0yXCdg9/SH4sMuKms+7eesdel2wKya WtJyzvIrXPIYd+Xuk6Z1sCRx5X6/9TsPIpdd/U9xAmlaqC4KhA8ByND3Wql0hly8PcgQ gfdlQ20Hzmc/Dfys2jp2b1bywP+8Ri6VvnCGwNb9Xq+RLnr4Ydc4tgV0DDaLMBIEllH8 y9+tlWrvHTvIY8yBcR3hTVsBGJcWlEuPkpYvuzxeB0OfubxOEzm0Z1yaMGyvozsikve0 QVnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=vl0+huPO34QfWeEMf7yKiM4C/TEwPYCaNXjbcVQ4QNQ=; b=dYUyJhw+Bq5Lo3Pr2o7RECaLoHmGC54TIKR4OpI8rZZr5omcFTotFYZf05rOWx4XZa 6R7NvJ/x+y0iAyJ7/DlFYEio4Un+l2Aq5pSov1fxlPRuEzf5t+pqVShVgedxmCXsenc1 qYUByvYwWYMbyZvIZgOq0nREzfqUr3bUhINsWtD1b+Lx0Ys9xyFPKzpKim4W1WmQu8Ub J/pdb9qRMHxUOHhgR2hbYAAf8MpoZH6Ao1OpUf7Nxherl525ie7AHFh8E9+YhXOHXgJy 82d0HEvigmTZp5CEFMgoAf0rIYIGzumDByeaQqbdjGEvGpmO3UROCL/26wFQXWNryMCF Dp9A== X-Gm-Message-State: AJaThX7km1C0wH+tvXtsDmLnGG8w7Z3CHULXNcSFSc9MyhkSJ+mkch2K hUG3HZmaEthUh5/WI/v2A6tuGY9twRAwtO0vD1uNzg== X-Google-Smtp-Source: AGs4zMadBMYIHrKje5p8fQlNay5iktdAIbVMgqZVpmWen2kgsyZpCWlo+0R5ZTSlw7+MIhFfk/oKy1tuN7wvsDqA2Y8= X-Received: by 10.107.175.165 with SMTP id p37mr34797807ioo.32.1512571038300; Wed, 06 Dec 2017 06:37:18 -0800 (PST) MIME-Version: 1.0 Received: by 10.2.138.114 with HTTP; Wed, 6 Dec 2017 06:37:17 -0800 (PST) In-Reply-To: <4A321A55-23FA-42AB-BF65-3DCA3464307D@punkt.de> References: <201712051641.vB5GfR5I052310@pdx.rh.CN85.dnsmgr.net> <4A321A55-23FA-42AB-BF65-3DCA3464307D@punkt.de> From: Adam Vande More Date: Wed, 6 Dec 2017 08:37:17 -0600 Message-ID: Subject: Re: NFS alternatives (was: Re: Storage overhead on zvols) To: "Patrick M. Hausen" Cc: "Rodney W. Grimes" , FreeBSD virtualization Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Dec 2017 14:37:19 -0000 On Wed, Dec 6, 2017 at 2:45 AM, Patrick M. Hausen wrote: > Hi all, > > > Am 05.12.2017 um 17:41 schrieb Rodney W. Grimes < > freebsd-rwg@pdx.rh.CN85.dnsmgr.net>: > > In effect what your asking for is what NFS does, so use NFS and get > > over the fact that this is the way to get what you want. Sure you > > could implement a virt-vfs but I wonder how close the spec of that > > would be to the spec of NFS. > > I figure it should be possible to implement something simpler > than NFS that provides full local posix semantics under the > constraint that only one "client" is allowed to mount the FS > at a time. > > I see quite a few applications for something like this, specifically > in "hyperconvergent" environments. Or vagrant, of course. > > *scratching head* isn't this what Sun's "network disk" protocol provided? > > Kind regards, > Patrick Like this? https://www.freebsd.org/doc/handbook/geom-ggate.html -- Adam From owner-freebsd-virtualization@freebsd.org Wed Dec 6 14:41:00 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4B1E9E81E17 for ; Wed, 6 Dec 2017 14:41:00 +0000 (UTC) (envelope-from paul@redbarn.org) Received: from family.redbarn.org (family.redbarn.org [IPv6:2001:559:8000:cd::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3920474900 for ; Wed, 6 Dec 2017 14:41:00 +0000 (UTC) (envelope-from paul@redbarn.org) Received: from [192.168.11.48] (200.12.232.153.ap.dti.ne.jp [153.232.12.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by family.redbarn.org (Postfix) with ESMTPSA id 6A20961FA2; Wed, 6 Dec 2017 14:40:59 +0000 (UTC) Message-ID: <5A28017B.7000806@redbarn.org> Date: Wed, 06 Dec 2017 23:40:59 +0900 From: Paul Vixie User-Agent: Postbox 5.0.20 (Windows/20171012) MIME-Version: 1.0 To: Adam Vande More CC: "Patrick M. Hausen" , "Rodney W. Grimes" , FreeBSD virtualization Subject: Re: NFS alternatives References: <201712051641.vB5GfR5I052310@pdx.rh.CN85.dnsmgr.net> <4A321A55-23FA-42AB-BF65-3DCA3464307D@punkt.de> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Dec 2017 14:41:00 -0000 Adam Vande More wrote: ... > Like this? > > https://www.freebsd.org/doc/handbook/geom-ggate.html yes, that'd be fine. if i used NFS i'd have to run lockd. -- P Vixie From owner-freebsd-virtualization@freebsd.org Wed Dec 6 17:48:06 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 731E4E87019 for ; Wed, 6 Dec 2017 17:48:06 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 53E327B719 for ; Wed, 6 Dec 2017 17:48:05 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id vB6Hm2KJ057502; Wed, 6 Dec 2017 09:48:02 -0800 (PST) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id vB6Hm13h057501; Wed, 6 Dec 2017 09:48:01 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201712061748.vB6Hm13h057501@pdx.rh.CN85.dnsmgr.net> Subject: Re: NFS alternatives (was: Re: Storage overhead on zvols) In-Reply-To: <4A321A55-23FA-42AB-BF65-3DCA3464307D@punkt.de> To: "Patrick M. Hausen" Date: Wed, 6 Dec 2017 09:48:01 -0800 (PST) CC: Paul Vixie , FreeBSD virtualization X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Dec 2017 17:48:06 -0000 > Hi all, > > > Am 05.12.2017 um 17:41 schrieb Rodney W. Grimes : > > In effect what your asking for is what NFS does, so use NFS and get > > over the fact that this is the way to get what you want. Sure you > > could implement a virt-vfs but I wonder how close the spec of that > > would be to the spec of NFS. > > I figure it should be possible to implement something simpler > than NFS that provides full local posix semantics under the > constraint that only one "client" is allowed to mount the FS > at a time. > > I see quite a few applications for something like this, specifically > in "hyperconvergent" environments. Or vagrant, of course. > > *scratching head* isn't this what Sun's "network disk" protocol provided? nd provided a 512b block device, no file system symatics at all, I believe it did allow 1 writer N readers though. Today you would use iSCSI in place of nd. -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Wed Dec 6 19:51:35 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 89C44E8A75A for ; Wed, 6 Dec 2017 19:51:35 +0000 (UTC) (envelope-from randy@terbush.org) Received: from mail-ot0-x236.google.com (mail-ot0-x236.google.com [IPv6:2607:f8b0:4003:c0f::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5373180417 for ; Wed, 6 Dec 2017 19:51:35 +0000 (UTC) (envelope-from randy@terbush.org) Received: by mail-ot0-x236.google.com with SMTP id v21so4403827oth.6 for ; Wed, 06 Dec 2017 11:51:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=terbush.org; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=Kzj35NZQMPbvL9Oxe41AMFLWj9B7wCnu9JIZ0qKoI7g=; b=LeaAhK1LJnWKozWyCv4uUw3O/tXj9c0QKe8PEQIxkAEh4n91FFRtipvSfv+JS7FKci jw6rDw6rpcYJmhr6nFm22FwPUUstaw5QRUxKBxHS7gyXK+MjXop3IOM1xtLSIYA6GXgB RoQCWjw+5nrhvmVOFpuMx+aXsGBIFHmbLDFSg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=Kzj35NZQMPbvL9Oxe41AMFLWj9B7wCnu9JIZ0qKoI7g=; b=Gs//hl2DxbXBdUZJqgnvalzbXMIDqvMMVugSW2b3dHfXn+eq/p8iiVgyFudsnwf/hy TCaSkVTKuDT8fR3FL6AP+uJE7YPQuUBZie4BNKDn+//rDm3DPkwWtuVN+dUAxb0/Z9aN 2fIrlTQlHrIj6BvVSe40JpGs58d3u/6puuCXaHeHs+enzjtHcCz3kYV3cDo5a7D0xlen G2Yudp1+TRJOgSIC82hTK3FHP5twKRUGctpWbo6W5VjsDtPgJMKZivYGVUEQ3iqj/2LA 80+VNzesCsssGm2RM5986XezCfccOv02Bc2F+pULZWAtpuSecgddgL2LODUBzts6sU2n 3sQw== X-Gm-Message-State: AJaThX7vWWwXDZk/yERHCzO6WTCFc799FOypkClXZ3uAmcBXjzzYOxaw wHqDi5FwJYgZ2Qg1gN31X54m4A+hTRD68aTbBrh4Ag== X-Google-Smtp-Source: AGs4zMbw51W2NteIA5AaJunwcvPPSSWIVa+J2BMROFtNhwLD3hr+QCpksHXS5Jb1hSlzmMGVla3NKpGW+IuO3A6ggXo= X-Received: by 10.157.37.106 with SMTP id j39mr24459556otd.294.1512589894130; Wed, 06 Dec 2017 11:51:34 -0800 (PST) MIME-Version: 1.0 Received: by 10.168.67.10 with HTTP; Wed, 6 Dec 2017 11:51:03 -0800 (PST) In-Reply-To: <81d05d9d-044a-9cad-40e3-5ddf86da6570@freebsd.org> References: <81d05d9d-044a-9cad-40e3-5ddf86da6570@freebsd.org> From: Randy Terbush Date: Wed, 6 Dec 2017 12:51:03 -0700 Message-ID: Subject: Re: Recovering an ZFS vm To: Peter Grehan Cc: "freebsd-virtua." Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Dec 2017 19:51:35 -0000 At the end of this all, it seemed that some change has taken away my need to be very specific about the grub partitions. By removing the grub_run0 and grub_run1 parameters from the vm-bhyve configs and running simply with the one below, (and showing a bit of patients while it repaired partitions), it booted as expected. I also see that once booted, the / filesystem is an ext4. I really cannot explain and it is entirely possible that this is a memory issue on my part. loader="grub" cpu=2 memory=8G network0_type="virtio-net" network0_switch="public" disk0_type="virtio-blk" disk0_name="disk0" disk0_dev="sparse-zvol" I appreciate all of you taking time out to reply. -- Randy On Tue, Dec 5, 2017 at 4:56 PM, Peter Grehan wrote: > Hi Randy, > > I have a Centos vm that has suddenly stopped booting. At the console, grub >> tells me the following if I attempt to list any of the available >> partitions. >> >> error: not a correct XFS inode. >> error: not a correct XFS inode. >> error: not a correct XFS inode. >> error: not a correct XFS inode. >> error: not a correct XFS inode. >> Filesystem type xfs, UUID 7652ffda-f7c5-408a-b0ce-b554b66fc2e5 - >> Partition >> start at 2048 - Total size 2097152 sectors >> grub> >> >> Is there an easy way to recover this? This has happened more than once. >> Just so happens there is something on this image I would like to have >> access to... >> > > Looks like the grub partition was upgraded to the version of XFS that has > the CRC feature enabled (7.2 ?). Unfortunately this feature is not > understood by grub-bhyve :( > > One way to recover the disk is to create a new VM with the most recent > CentOS, but using UEFI for the bootloader. Then, add this disk to the > guest, and from within the guest I think you can run an XFS utility that > will disable the use of CRCs on that partition. > > The proper fix would be for grub-bhyve to be updated to the latest > version of grub2, though a workaround is to create guests with UEFI and not > use grub-bhyve. > > later, > > Peter. > From owner-freebsd-virtualization@freebsd.org Thu Dec 7 12:42:57 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9079E85BA6 for ; Thu, 7 Dec 2017 12:42:57 +0000 (UTC) (envelope-from djfarid@gmail.com) Received: from mail-wm0-x243.google.com (mail-wm0-x243.google.com [IPv6:2a00:1450:400c:c09::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 738463C53 for ; Thu, 7 Dec 2017 12:42:57 +0000 (UTC) (envelope-from djfarid@gmail.com) Received: by mail-wm0-x243.google.com with SMTP id g75so12942251wme.0 for ; Thu, 07 Dec 2017 04:42:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=VxeI1Dr4UwShEiuy9HUeNzKxrmRpf81Y0Jw1DZtvoFs=; b=p+1uTSx0bn/JqLv2LZMFWO4BHVNvq6rJWLW4MwDUOzEZEcgIP9u7ySD+jZa5prj5He 5hBhEqbg2P02f3PCVz7WUdtFuguwugxNUk0THdeh3kWMua+CJ83FKQDYVizPVC/T3UZw Z+aqEKdAsRKbQZ+h4FQAPNi8httxeGQ8nbrYFosAsmm/pc89Cnsf/o119+AcYjWyyzIp 6/yMBcYN5bLuCJO23LZ48M3KpoNLblxyU3bzERnU0U9cg52SSAk0Zvi8hEkGgGAc+hMz MFoH7jhUbwb6/kPf2Tc3ySDlWZwDgGcNfv20ZFX2NauphQtgR8sxpHQ9rtliSEG152VJ ecCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=VxeI1Dr4UwShEiuy9HUeNzKxrmRpf81Y0Jw1DZtvoFs=; b=Efr86YPIuTp0nGgia6KxnMr1Y3K8w0zj4i4oOjEln3+M7KSuCUcUy3niEYacwyFrb9 OxeA5rQCW/LI0gO0EPKwjImOEEng7SUgzpoIUSbRS5BPvDXzw54rW3f547v+GbahYn66 HgB5Fj4WmLbWDiEC7Me8E3hdrqeGYtK55ssE0Jk9YAXDMW7W8C4a6gc4H4arPe25N8VI 8qkERT8PsnMxEUkUJRAVL0tM4QSx1mabwgEOlKziGs6xDke2OFjQ1ekIdjkiH/Uf4G1m 1CUwq49C38fu9iPrGo/WkIyPBRt8GlDkDW19d+63jBsnaYO8bWsj+nOMRzBiRQIbk+Ah TEwg== X-Gm-Message-State: AJaThX5S1JaA8nSTqAbSe6ZUTqLEMkU2399MFcLe8Y0XUyf7YFwhAg6b SUAok+1r9dyp2vXUAjZE524qLl30mp58ez/CHNlYxzrU X-Google-Smtp-Source: AGs4zMYKhcudKqYyIUKj8trfLpAHP1H2/Q7ZMgWYbmUBJ70MStAFP2MS3qdKQZ4kHRzHhxk7A4zwDC+679TJoR4oiPM= X-Received: by 10.80.214.139 with SMTP id r11mr45859681edi.17.1512650575654; Thu, 07 Dec 2017 04:42:55 -0800 (PST) MIME-Version: 1.0 Received: by 10.80.139.70 with HTTP; Thu, 7 Dec 2017 04:42:55 -0800 (PST) From: Farid Joubbi Date: Thu, 7 Dec 2017 13:42:55 +0100 Message-ID: Subject: dhcpd: Failed to send 300 byte long packet To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Dec 2017 12:42:57 -0000 Hi, I am running bhyve on FreBSD 11.1-RELEASE-p4. I use vm to control the virtual machines. I have several machines running. Two of them are configured with the same template, one Linux Mint and one RHEL 7.4. Both of them receive their IP address through DHCP. For some reason the RHEL server is dropping it's address and becomes unavailable on the network after a while. When that happens /var/log/messages on the FreeBSD server is spammed with this: dhcpd: send_packet: Host is down dhcpd: dhcp.c:3976: Failed to send 300 byte long packet over fallback interface. This never happens with the machine running Linux Mint. dhcpd is running on the same FreeBSD server as bhyve. This is the output of "vm vonfigure rhel": uefi="yes" cpu=2 memory=4G network0_type="virtio-net" network0_switch="lan" disk0_type="ahci-hd" disk0_name="disk0.img" graphics="yes" xhci_mouse="yes" graphics_res="1600x900" uuid="cfc832cb-d8c9-11e7-8255-9418823456dc" network0_mac="58:9c:fc:08:21:a8" # vm switch info ------------------------ Virtual Switch: lan ------------------------ type: manual ident: bridge0 vlan: - nat: - physical-ports: - bytes-in: 7839031548 (7.300G) bytes-out: 33012227886 (30.745G) virtual-port device: tap0 vm: rhel virtual-port device: tap1 vm: linuxmint Any ideas of what is causing this?