From owner-freebsd-fs@freebsd.org Fri May 18 16:00:46 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F390CEACCBA for ; Fri, 18 May 2018 16:00:45 +0000 (UTC) (envelope-from toasty@dragondata.com) Received: from mail-io0-x235.google.com (mail-io0-x235.google.com [IPv6:2607:f8b0:4001:c06::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8B1577C6C5 for ; Fri, 18 May 2018 16:00:45 +0000 (UTC) (envelope-from toasty@dragondata.com) Received: by mail-io0-x235.google.com with SMTP id e12-v6so6758277iob.8 for ; Fri, 18 May 2018 09:00:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dragondata.com; s=google; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=fEv525FxNUAP/OJFXFE3seA5+gYwjnXxMEv9e9572w8=; b=U7mYlC9M9YZFcSK7sXCVLlAckH0swxmoY2O+AntFUY0oLv1QN6AA0QmAse6fHh1D1B Syj5VR/S6deZBhj+dRvCAY2CDTqHQTbkbf1saKxd5H1o5yOhYhtMakyAYKYQsDZ9xknu OTHD2fU30Bsn1paQcdG3b2h8Lu27SMTCejgYU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=fEv525FxNUAP/OJFXFE3seA5+gYwjnXxMEv9e9572w8=; b=ROad3pNpijchK5uNKkaRsrriEjHz6UGjk6+6YIAT+rCuinoe3T3EMT6hWCCUW1Oikx i0U3DPbDkaTEI2VluU0YpPC7R5UmJIKdlvl8Q32hPxaQqO5mObBkOt7WfXJjpDySo3LD 7oOewu8zYiiqEzMUKdod7tYns5XCVw/16K9xKlNKv9Eu30Wc82sZaTW2c5hNuDEjzG0g FVTL7/2Kn9QR7/lH9E6sU3wj59nS1nkhm3Iu1lGe51KI1ldeiH5iFPcRnd/Eketl52+I 22lOpKvAAVcLcPBQ5DwC0Zw/qNuV9XxdawL+IoB9jpXAL3/d9O9BktRwmn7ZTKLaXuDQ Uvdw== X-Gm-Message-State: ALKqPwfS18PJRWYvRYGW6HXv+2oSG2EZsIRkfEwT+rIUqG5z6JO5A2gO M3ZQhIRHsip8ld7iUzcqcCYUWj92EVs= X-Google-Smtp-Source: AB8JxZqYCE2dQajwMp9zhf134B2FyyZt+CdAZch44583a5+IFAdmdEoEVyeoaEMrJOqssHz1BdkCrw== X-Received: by 2002:a6b:3e8b:: with SMTP id l133-v6mr10792158ioa.65.1526659244437; Fri, 18 May 2018 09:00:44 -0700 (PDT) Received: from unassigned.v6.your.org ([2001:4978:1:45:c57b:919e:3353:907]) by smtp.gmail.com with ESMTPSA id d130-v6sm4976374iog.46.2018.05.18.09.00.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 May 2018 09:00:43 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.3 \(3445.6.18\)) Subject: Re: Unexpected zvol usage From: Kevin Day In-Reply-To: Date: Fri, 18 May 2018 11:00:41 -0500 Cc: "freebsd-fs@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: <2EC9E6E6-F49A-4DBA-B622-61D250C60B6C@dragondata.com> References: To: Paul Esson X-Mailer: Apple Mail (2.3445.6.18) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 May 2018 16:00:46 -0000 Try: zfs create -b 128k -V > On May 18, 2018, at 10:42 AM, Paul Esson = wrote: >=20 > Hi Folks, >=20 > I have an 11.1-RELEASE system being used as a host for a bhyve guest. = There is a large zpool on the host created from 12 x 10TB HDDs using = raidz2 redundancy with ashift12. I have created a sparse zvol within = the pool using default settings and presented that to the bhyve vm as an = ahci-hd disk type. The guest has a zpool and filesystem dataset built = on this disk. When I start to write to the filesystem on the guest I am = finding that the used/referenced on the host's zvol are more than double = those on the guest. The logicalused/referenced values on the host zvol = are more in line with the equivalent guest values, but my problem is = that the host zvol is likely to fill before I have written all intended = data to the guest. >=20 >=20 > I have included below information from both the host and guest before = and after writing. This output shows that the zvol uses a default 8K = blocksize and that the guest zfs is therefore ashift13. I also tried = creating the zvol with a 4K blocksize and the guest zfs ashift12 so that = 4K blocks were consistent across hosts and guest, but still saw the = amplification on writes to the zvol. >=20 > Any insight greatly appreciated. >=20 >=20 >=20 > HOST >=20 > Zpool > RAIDZ2 12 x HDDs, ashift 12 >=20 > NAME PROPERTY VALUE SOURCE > dc1-hn-01 type filesystem - > dc1-hn-01 creation Mon Apr 23 14:35 2018 - > dc1-hn-01 used 32.0G - > dc1-hn-01 available 78.2T - > dc1-hn-01 referenced 201K - > dc1-hn-01 compressratio 1.00x - > dc1-hn-01 mounted yes - > dc1-hn-01 quota none default > dc1-hn-01 reservation none default > dc1-hn-01 recordsize 128K default > dc1-hn-01 mountpoint /export/data/dc1-hn-01 local > dc1-hn-01 sharenfs off default > dc1-hn-01 checksum on default > dc1-hn-01 compression off default > dc1-hn-01 atime on default > dc1-hn-01 devices on default > dc1-hn-01 exec on default > dc1-hn-01 setuid on default > dc1-hn-01 readonly off default > dc1-hn-01 jailed off default > dc1-hn-01 snapdir hidden default > dc1-hn-01 aclmode discard default > dc1-hn-01 aclinherit restricted default > dc1-hn-01 canmount on default > dc1-hn-01 xattr off temporary > dc1-hn-01 copies 1 default > dc1-hn-01 version 5 - > dc1-hn-01 utf8only off - > dc1-hn-01 normalization none - > dc1-hn-01 casesensitivity sensitive - > dc1-hn-01 vscan off default > dc1-hn-01 nbmand off default > dc1-hn-01 sharesmb off default > dc1-hn-01 refquota none default > dc1-hn-01 refreservation none default > dc1-hn-01 primarycache all default > dc1-hn-01 secondarycache all default > dc1-hn-01 usedbysnapshots 0 - > dc1-hn-01 usedbydataset 201K - > dc1-hn-01 usedbychildren 32.0G - > dc1-hn-01 usedbyrefreservation 0 - > dc1-hn-01 logbias latency default > dc1-hn-01 dedup off default > dc1-hn-01 mlslabel - > dc1-hn-01 sync standard default > dc1-hn-01 refcompressratio 1.00x - > dc1-hn-01 written 201K - > dc1-hn-01 logicalused 2.89G - > dc1-hn-01 logicalreferenced 36.5K - > dc1-hn-01 volmode default default > dc1-hn-01 filesystem_limit none default > dc1-hn-01 snapshot_limit none default > dc1-hn-01 filesystem_count none default > dc1-hn-01 snapshot_count none default > dc1-hn-01 redundant_metadata all default >=20 > NAME AVAIL USED USEDSNAP USEDDS = USEDREFRESERV USEDCHILD > dc1-hn-01 78.2T 32.0G 0 201K = 0 32.0G > dc1-hn-01/vm 78.2T 31.9G 0 990M = 0 30.9G > dc1-hn-01/vm/dc1-olbp-sn-11 78.2T 30.9G 0 238K = 0 30.9G > dc1-hn-01/vm/dc1-olbp-sn-11/disk0 78.2T 30.9G 0 4.35G = 26.6G 0 > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 78.2T 4.50M 0 4.50M = 0 0 >=20 > Sparse ZVOL - baseline >=20 > NAME PROPERTY VALUE = SOURCE > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 type volume = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 creation Fri May 18 = 15:36 2018 - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 used 4.50M = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 available 78.2T = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 referenced 4.50M = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 compressratio 1.00x = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 reservation none = default > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 volsize 28T = local > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 volblocksize 8K = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 checksum on = default > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 compression off = default > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 readonly off = default > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 copies 1 = default > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 refreservation none = default > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 primarycache all = default > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 secondarycache all = default > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbysnapshots 0 = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbydataset 4.50M = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbychildren 0 = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbyrefreservation 0 = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 logbias latency = default > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 dedup off = default > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 mlslabel = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 sync standard = default > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 refcompressratio 1.00x = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 written 4.50M = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 logicalused 1.89M = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 logicalreferenced 1.89M = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 volmode dev = local > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 snapshot_limit none = default > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 snapshot_count none = default > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 redundant_metadata all = default >=20 >=20 > GUEST - baseline >=20 > 1 x vdisk from host ZVOL ashift 13 >=20 > NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV = USEDCHILD > dc1-sn-11 26.9T 632K 0 176K 0 = 456K >=20 > NAME PROPERTY VALUE SOURCE > dc1-sn-11 type filesystem - > dc1-sn-11 creation Fri May 18 15:40 2018 - > dc1-sn-11 used 632K - > dc1-sn-11 available 26.9T - > dc1-sn-11 referenced 176K - > dc1-sn-11 compressratio 1.00x - > dc1-sn-11 mounted yes - > dc1-sn-11 quota none default > dc1-sn-11 reservation none default > dc1-sn-11 recordsize 128K default > dc1-sn-11 mountpoint /export/data/dc1-sn-11 local > dc1-sn-11 sharenfs off default > dc1-sn-11 checksum on default > dc1-sn-11 compression off default > dc1-sn-11 atime on default > dc1-sn-11 devices on default > dc1-sn-11 exec on default > dc1-sn-11 setuid on default > dc1-sn-11 readonly off default > dc1-sn-11 jailed off default > dc1-sn-11 snapdir hidden default > dc1-sn-11 aclmode discard default > dc1-sn-11 aclinherit restricted default > dc1-sn-11 canmount on default > dc1-sn-11 xattr off temporary > dc1-sn-11 copies 1 default > dc1-sn-11 version 5 - > dc1-sn-11 utf8only off - > dc1-sn-11 normalization none - > dc1-sn-11 casesensitivity sensitive - > dc1-sn-11 vscan off default > dc1-sn-11 nbmand off default > dc1-sn-11 sharesmb off default > dc1-sn-11 refquota none default > dc1-sn-11 refreservation none default > dc1-sn-11 primarycache all default > dc1-sn-11 secondarycache all default > dc1-sn-11 usedbysnapshots 0 - > dc1-sn-11 usedbydataset 176K - > dc1-sn-11 usedbychildren 456K - > dc1-sn-11 usedbyrefreservation 0 - > dc1-sn-11 logbias latency default > dc1-sn-11 dedup off default > dc1-sn-11 mlslabel - > dc1-sn-11 sync standard default > dc1-sn-11 refcompressratio 1.00x - > dc1-sn-11 written 176K - > dc1-sn-11 logicalused 49K - > dc1-sn-11 logicalreferenced 11.5K - > dc1-sn-11 volmode default default > dc1-sn-11 filesystem_limit none default > dc1-sn-11 snapshot_limit none default > dc1-sn-11 filesystem_count none default > dc1-sn-11 snapshot_count none default > dc1-sn-11 redundant_metadata all default >=20 > After writing some data to the guest >=20 > HOST ZVOL >=20 > NAME PROPERTY VALUE = SOURCE > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 type volume = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 creation Fri May 18 = 15:36 2018 - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 used 99.7G = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 available 78.1T = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 referenced 99.7G = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 compressratio 1.00x = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 reservation none = default > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 volsize 28T = local > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 volblocksize 8K = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 checksum on = default > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 compression off = default > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 readonly off = default > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 copies 1 = default > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 refreservation none = default > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 primarycache all = default > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 secondarycache all = default > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbysnapshots 0 = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbydataset 99.7G = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbychildren 0 = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbyrefreservation 0 = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 logbias latency = default > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 dedup off = default > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 mlslabel = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 sync standard = default > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 refcompressratio 1.00x = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 written 99.7G = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 logicalused 43.6G = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 logicalreferenced 43.6G = - > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 volmode dev = local > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 snapshot_limit none = default > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 snapshot_count none = default > dc1-hn-01/vm/dc1-olbp-sn-11/disk1 redundant_metadata all = default >=20 > GUEST ZFS >=20 > NAME PROPERTY VALUE SOURCE > dc1-sn-11 type filesystem - > dc1-sn-11 creation Fri May 18 15:40 2018 - > dc1-sn-11 used 44.3G - > dc1-sn-11 available 26.8T - > dc1-sn-11 referenced 176K - > dc1-sn-11 compressratio 1.00x - > dc1-sn-11 mounted no - > dc1-sn-11 quota none default > dc1-sn-11 reservation none default > dc1-sn-11 recordsize 128K default > dc1-sn-11 mountpoint /export/data/dc1-sn-11 local > dc1-sn-11 sharenfs off default > dc1-sn-11 checksum on default > dc1-sn-11 compression off default > dc1-sn-11 atime on default > dc1-sn-11 devices on default > dc1-sn-11 exec on default > dc1-sn-11 setuid on default > dc1-sn-11 readonly off default > dc1-sn-11 jailed off default > dc1-sn-11 snapdir hidden default > dc1-sn-11 aclmode discard default > dc1-sn-11 aclinherit restricted default > dc1-sn-11 canmount on default > dc1-sn-11 xattr on default > dc1-sn-11 copies 1 default > dc1-sn-11 version 5 - > dc1-sn-11 utf8only off - > dc1-sn-11 normalization none - > dc1-sn-11 casesensitivity sensitive - > dc1-sn-11 vscan off default > dc1-sn-11 nbmand off default > dc1-sn-11 sharesmb off default > dc1-sn-11 refquota none default > dc1-sn-11 refreservation none default > dc1-sn-11 primarycache all default > dc1-sn-11 secondarycache all default > dc1-sn-11 usedbysnapshots 0 - > dc1-sn-11 usedbydataset 176K - > dc1-sn-11 usedbychildren 44.3G - > dc1-sn-11 usedbyrefreservation 0 - > dc1-sn-11 logbias latency default > dc1-sn-11 dedup off default > dc1-sn-11 mlslabel - > dc1-sn-11 sync standard default > dc1-sn-11 refcompressratio 1.00x - > dc1-sn-11 written 176K - > dc1-sn-11 logicalused 44.2G - > dc1-sn-11 logicalreferenced 11.5K - > dc1-sn-11 volmode default default > dc1-sn-11 filesystem_limit none default > dc1-sn-11 snapshot_limit none default > dc1-sn-11 filesystem_count none default > dc1-sn-11 snapshot_count none default > dc1-sn-11 redundant_metadata all default >=20 >=20 > Regards, >=20 >=20 > Paul Esson > t +44 (0)118 951 5235 | m +44 (0)776 690 6514 > e paul.esson@redstor.com >=20 >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"