From owner-freebsd-fs@freebsd.org Fri May 18 15:42:09 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 34E3CEABFF4 for ; Fri, 18 May 2018 15:42:09 +0000 (UTC) (envelope-from paul.esson@redstor.com) Received: from EUR01-HE1-obe.outbound.protection.outlook.com (mail-he1eur01on0619.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe1e::619]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT TLS CA 4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 45FDB7B73A for ; Fri, 18 May 2018 15:42:08 +0000 (UTC) (envelope-from paul.esson@redstor.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redstorltd.onmicrosoft.com; s=selector1-redstor-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eVQlzuwzBVGBOaBOfSeCS2noGaOCe2jD1IgYZglPtGc=; b=N4kx1/CjbpuDEtgqailSqGu+oZ+lDvLszojsoaFxhKNbj7Q7SIrClX1B2odtp7qkb8zoJTamxjabwUtPG/JiGtX5mk0rcZ3Q5FR/WWR2XgBVQPf8cZ+rwRha55OtLoCvusOJSxRGGtBSLdSVcfR9x1Pmr3z5x0Ttha7ydAElRi4= Received: from HE1PR0102MB2588.eurprd01.prod.exchangelabs.com (10.170.251.141) by HE1PR0102MB3257.eurprd01.prod.exchangelabs.com (10.167.124.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.776.11; Fri, 18 May 2018 15:42:05 +0000 Received: from HE1PR0102MB2588.eurprd01.prod.exchangelabs.com ([fe80::d508:b6bc:e82:73e]) by HE1PR0102MB2588.eurprd01.prod.exchangelabs.com ([fe80::d508:b6bc:e82:73e%13]) with mapi id 15.20.0755.019; Fri, 18 May 2018 15:42:05 +0000 From: Paul Esson To: "freebsd-fs@freebsd.org" Subject: Unexpected zvol usage Thread-Topic: Unexpected zvol usage Thread-Index: AdPuu8U1d0Yref1cQ2GeqjmmDcH+hQ== Date: Fri, 18 May 2018 15:42:05 +0000 Message-ID: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=paul.esson@redstor.com; x-originating-ip: [81.136.184.115] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; HE1PR0102MB3257; 7:qSuLhukJl/6vNrS+xH+rI6wU+wjAb4K+UBomSsYyBWC+kHIqLovQrIIsJrWm3Gk8a/UJ8hXslHwog/D5fHy7tPgE5VNcbfA2cIzdr9daG5rZdAVrH/PKUgGLvMKdNkE3B4p7JXJUSmtbAb1qnto4EpLjz1NRDpW80lGC9oAUsoncKYBI2Te+7GX4q2VmbFpRRHFG+g3EGad6HeU7rc/L5v099taFzyBZ67oBBgCRfguAPo2frL3G4ZTK4uQpOLCp x-ms-exchange-antispam-srfa-diagnostics: SOS; x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7153060)(7193020); SRVR:HE1PR0102MB3257; x-ms-traffictypediagnostic: HE1PR0102MB3257: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(28532068793085)(155532106045638)(21748063052155); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(5005006)(8121501046)(3002001)(10201501046)(3231254)(944501410)(52105095)(93006095)(93001095)(149027)(150027)(6041310)(20161123560045)(20161123564045)(20161123562045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123558120)(6072148)(201708071742011)(7699016); SRVR:HE1PR0102MB3257; BCL:0; PCL:0; RULEID:; SRVR:HE1PR0102MB3257; x-forefront-prvs: 0676F530A9 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(39850400004)(346002)(39380400002)(396003)(376002)(366004)(189003)(199004)(42174003)(5630700001)(316002)(6506007)(53936002)(102836004)(33656002)(53946003)(97736004)(16200700003)(2351001)(106356001)(186003)(9686003)(55016002)(26005)(59450400001)(105586002)(99286004)(6306002)(54896002)(25786009)(68736007)(236005)(9326002)(7696005)(3480700004)(81156014)(81166006)(2906002)(3280700002)(5250100002)(14454004)(8676002)(6916009)(3660700001)(790700001)(8936002)(7116003)(3846002)(86362001)(6116002)(486006)(6436002)(74316002)(66066001)(2501003)(476003)(5660300001)(2900100001)(44832011)(7736002)(478600001)(5640700003)(579004)(569006); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR0102MB3257; H:HE1PR0102MB2588.eurprd01.prod.exchangelabs.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: redstor.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: EgnPDKRQeLeJPt4+IcNlhtg0JiB7WvGkPUkRVJJlpMBl7k70j5y3OjRLkEnRWeHcNDi6YHKx6TknmVVAo72JFkVzB5N5AOEN+WyCe3H6ZTJYGD3j6YHDZSqNpNmhA+5O/Vp+jIYtjQn08PfudrabGRuLDpQYxOKrusCygc5ZhnkKghvEK5UHoXnlPRbkQaJx spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: c0aedfb4-ccfd-454a-8290-08d5bcd5e88a X-OriginatorOrg: redstor.com X-MS-Exchange-CrossTenant-Network-Message-Id: c0aedfb4-ccfd-454a-8290-08d5bcd5e88a X-MS-Exchange-CrossTenant-originalarrivaltime: 18 May 2018 15:42:05.5584 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 24ac53ae-15a7-4211-afef-61d8f34e2571 X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0102MB3257 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.26 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 May 2018 15:42:09 -0000 Hi Folks, I have an 11.1-RELEASE system being used as a host for a bhyve guest. Ther= e is a large zpool on the host created from 12 x 10TB HDDs using raidz2 red= undancy with ashift12. I have created a sparse zvol within the pool using = default settings and presented that to the bhyve vm as an ahci-hd disk type= . The guest has a zpool and filesystem dataset built on this disk. When I= start to write to the filesystem on the guest I am finding that the used/r= eferenced on the host's zvol are more than double those on the guest. The = logicalused/referenced values on the host zvol are more in line with the eq= uivalent guest values, but my problem is that the host zvol is likely to fi= ll before I have written all intended data to the guest. I have included below information from both the host and guest before and a= fter writing. This output shows that the zvol uses a default 8K blocksize = and that the guest zfs is therefore ashift13. I also tried creating the zv= ol with a 4K blocksize and the guest zfs ashift12 so that 4K blocks were co= nsistent across hosts and guest, but still saw the amplification on writes = to the zvol. Any insight greatly appreciated. HOST Zpool RAIDZ2 12 x HDDs, ashift 12 NAME PROPERTY VALUE SOURCE dc1-hn-01 type filesystem - dc1-hn-01 creation Mon Apr 23 14:35 2018 - dc1-hn-01 used 32.0G - dc1-hn-01 available 78.2T - dc1-hn-01 referenced 201K - dc1-hn-01 compressratio 1.00x - dc1-hn-01 mounted yes - dc1-hn-01 quota none default dc1-hn-01 reservation none default dc1-hn-01 recordsize 128K default dc1-hn-01 mountpoint /export/data/dc1-hn-01 local dc1-hn-01 sharenfs off default dc1-hn-01 checksum on default dc1-hn-01 compression off default dc1-hn-01 atime on default dc1-hn-01 devices on default dc1-hn-01 exec on default dc1-hn-01 setuid on default dc1-hn-01 readonly off default dc1-hn-01 jailed off default dc1-hn-01 snapdir hidden default dc1-hn-01 aclmode discard default dc1-hn-01 aclinherit restricted default dc1-hn-01 canmount on default dc1-hn-01 xattr off temporary dc1-hn-01 copies 1 default dc1-hn-01 version 5 - dc1-hn-01 utf8only off - dc1-hn-01 normalization none - dc1-hn-01 casesensitivity sensitive - dc1-hn-01 vscan off default dc1-hn-01 nbmand off default dc1-hn-01 sharesmb off default dc1-hn-01 refquota none default dc1-hn-01 refreservation none default dc1-hn-01 primarycache all default dc1-hn-01 secondarycache all default dc1-hn-01 usedbysnapshots 0 - dc1-hn-01 usedbydataset 201K - dc1-hn-01 usedbychildren 32.0G - dc1-hn-01 usedbyrefreservation 0 - dc1-hn-01 logbias latency default dc1-hn-01 dedup off default dc1-hn-01 mlslabel - dc1-hn-01 sync standard default dc1-hn-01 refcompressratio 1.00x - dc1-hn-01 written 201K - dc1-hn-01 logicalused 2.89G - dc1-hn-01 logicalreferenced 36.5K - dc1-hn-01 volmode default default dc1-hn-01 filesystem_limit none default dc1-hn-01 snapshot_limit none default dc1-hn-01 filesystem_count none default dc1-hn-01 snapshot_count none default dc1-hn-01 redundant_metadata all default NAME AVAIL USED USEDSNAP USEDDS USEDREFR= ESERV USEDCHILD dc1-hn-01 78.2T 32.0G 0 201K = 0 32.0G dc1-hn-01/vm 78.2T 31.9G 0 990M = 0 30.9G dc1-hn-01/vm/dc1-olbp-sn-11 78.2T 30.9G 0 238K = 0 30.9G dc1-hn-01/vm/dc1-olbp-sn-11/disk0 78.2T 30.9G 0 4.35G = 26.6G 0 dc1-hn-01/vm/dc1-olbp-sn-11/disk1 78.2T 4.50M 0 4.50M = 0 0 Sparse ZVOL - baseline NAME PROPERTY VALUE = SOURCE dc1-hn-01/vm/dc1-olbp-sn-11/disk1 type volume = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 creation Fri May 18 15:36 2= 018 - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 used 4.50M = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 available 78.2T = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 referenced 4.50M = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 compressratio 1.00x = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 reservation none = default dc1-hn-01/vm/dc1-olbp-sn-11/disk1 volsize 28T = local dc1-hn-01/vm/dc1-olbp-sn-11/disk1 volblocksize 8K = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 checksum on = default dc1-hn-01/vm/dc1-olbp-sn-11/disk1 compression off = default dc1-hn-01/vm/dc1-olbp-sn-11/disk1 readonly off = default dc1-hn-01/vm/dc1-olbp-sn-11/disk1 copies 1 = default dc1-hn-01/vm/dc1-olbp-sn-11/disk1 refreservation none = default dc1-hn-01/vm/dc1-olbp-sn-11/disk1 primarycache all = default dc1-hn-01/vm/dc1-olbp-sn-11/disk1 secondarycache all = default dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbysnapshots 0 = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbydataset 4.50M = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbychildren 0 = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbyrefreservation 0 = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 logbias latency = default dc1-hn-01/vm/dc1-olbp-sn-11/disk1 dedup off = default dc1-hn-01/vm/dc1-olbp-sn-11/disk1 mlslabel = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 sync standard = default dc1-hn-01/vm/dc1-olbp-sn-11/disk1 refcompressratio 1.00x = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 written 4.50M = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 logicalused 1.89M = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 logicalreferenced 1.89M = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 volmode dev = local dc1-hn-01/vm/dc1-olbp-sn-11/disk1 snapshot_limit none = default dc1-hn-01/vm/dc1-olbp-sn-11/disk1 snapshot_count none = default dc1-hn-01/vm/dc1-olbp-sn-11/disk1 redundant_metadata all = default GUEST - baseline 1 x vdisk from host ZVOL ashift 13 NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHIL= D dc1-sn-11 26.9T 632K 0 176K 0 456= K NAME PROPERTY VALUE SOURCE dc1-sn-11 type filesystem - dc1-sn-11 creation Fri May 18 15:40 2018 - dc1-sn-11 used 632K - dc1-sn-11 available 26.9T - dc1-sn-11 referenced 176K - dc1-sn-11 compressratio 1.00x - dc1-sn-11 mounted yes - dc1-sn-11 quota none default dc1-sn-11 reservation none default dc1-sn-11 recordsize 128K default dc1-sn-11 mountpoint /export/data/dc1-sn-11 local dc1-sn-11 sharenfs off default dc1-sn-11 checksum on default dc1-sn-11 compression off default dc1-sn-11 atime on default dc1-sn-11 devices on default dc1-sn-11 exec on default dc1-sn-11 setuid on default dc1-sn-11 readonly off default dc1-sn-11 jailed off default dc1-sn-11 snapdir hidden default dc1-sn-11 aclmode discard default dc1-sn-11 aclinherit restricted default dc1-sn-11 canmount on default dc1-sn-11 xattr off temporary dc1-sn-11 copies 1 default dc1-sn-11 version 5 - dc1-sn-11 utf8only off - dc1-sn-11 normalization none - dc1-sn-11 casesensitivity sensitive - dc1-sn-11 vscan off default dc1-sn-11 nbmand off default dc1-sn-11 sharesmb off default dc1-sn-11 refquota none default dc1-sn-11 refreservation none default dc1-sn-11 primarycache all default dc1-sn-11 secondarycache all default dc1-sn-11 usedbysnapshots 0 - dc1-sn-11 usedbydataset 176K - dc1-sn-11 usedbychildren 456K - dc1-sn-11 usedbyrefreservation 0 - dc1-sn-11 logbias latency default dc1-sn-11 dedup off default dc1-sn-11 mlslabel - dc1-sn-11 sync standard default dc1-sn-11 refcompressratio 1.00x - dc1-sn-11 written 176K - dc1-sn-11 logicalused 49K - dc1-sn-11 logicalreferenced 11.5K - dc1-sn-11 volmode default default dc1-sn-11 filesystem_limit none default dc1-sn-11 snapshot_limit none default dc1-sn-11 filesystem_count none default dc1-sn-11 snapshot_count none default dc1-sn-11 redundant_metadata all default After writing some data to the guest HOST ZVOL NAME PROPERTY VALUE = SOURCE dc1-hn-01/vm/dc1-olbp-sn-11/disk1 type volume = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 creation Fri May 18 15:36 2= 018 - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 used 99.7G = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 available 78.1T = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 referenced 99.7G = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 compressratio 1.00x = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 reservation none = default dc1-hn-01/vm/dc1-olbp-sn-11/disk1 volsize 28T = local dc1-hn-01/vm/dc1-olbp-sn-11/disk1 volblocksize 8K = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 checksum on = default dc1-hn-01/vm/dc1-olbp-sn-11/disk1 compression off = default dc1-hn-01/vm/dc1-olbp-sn-11/disk1 readonly off = default dc1-hn-01/vm/dc1-olbp-sn-11/disk1 copies 1 = default dc1-hn-01/vm/dc1-olbp-sn-11/disk1 refreservation none = default dc1-hn-01/vm/dc1-olbp-sn-11/disk1 primarycache all = default dc1-hn-01/vm/dc1-olbp-sn-11/disk1 secondarycache all = default dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbysnapshots 0 = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbydataset 99.7G = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbychildren 0 = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbyrefreservation 0 = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 logbias latency = default dc1-hn-01/vm/dc1-olbp-sn-11/disk1 dedup off = default dc1-hn-01/vm/dc1-olbp-sn-11/disk1 mlslabel = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 sync standard = default dc1-hn-01/vm/dc1-olbp-sn-11/disk1 refcompressratio 1.00x = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 written 99.7G = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 logicalused 43.6G = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 logicalreferenced 43.6G = - dc1-hn-01/vm/dc1-olbp-sn-11/disk1 volmode dev = local dc1-hn-01/vm/dc1-olbp-sn-11/disk1 snapshot_limit none = default dc1-hn-01/vm/dc1-olbp-sn-11/disk1 snapshot_count none = default dc1-hn-01/vm/dc1-olbp-sn-11/disk1 redundant_metadata all = default GUEST ZFS NAME PROPERTY VALUE SOURCE dc1-sn-11 type filesystem - dc1-sn-11 creation Fri May 18 15:40 2018 - dc1-sn-11 used 44.3G - dc1-sn-11 available 26.8T - dc1-sn-11 referenced 176K - dc1-sn-11 compressratio 1.00x - dc1-sn-11 mounted no - dc1-sn-11 quota none default dc1-sn-11 reservation none default dc1-sn-11 recordsize 128K default dc1-sn-11 mountpoint /export/data/dc1-sn-11 local dc1-sn-11 sharenfs off default dc1-sn-11 checksum on default dc1-sn-11 compression off default dc1-sn-11 atime on default dc1-sn-11 devices on default dc1-sn-11 exec on default dc1-sn-11 setuid on default dc1-sn-11 readonly off default dc1-sn-11 jailed off default dc1-sn-11 snapdir hidden default dc1-sn-11 aclmode discard default dc1-sn-11 aclinherit restricted default dc1-sn-11 canmount on default dc1-sn-11 xattr on default dc1-sn-11 copies 1 default dc1-sn-11 version 5 - dc1-sn-11 utf8only off - dc1-sn-11 normalization none - dc1-sn-11 casesensitivity sensitive - dc1-sn-11 vscan off default dc1-sn-11 nbmand off default dc1-sn-11 sharesmb off default dc1-sn-11 refquota none default dc1-sn-11 refreservation none default dc1-sn-11 primarycache all default dc1-sn-11 secondarycache all default dc1-sn-11 usedbysnapshots 0 - dc1-sn-11 usedbydataset 176K - dc1-sn-11 usedbychildren 44.3G - dc1-sn-11 usedbyrefreservation 0 - dc1-sn-11 logbias latency default dc1-sn-11 dedup off default dc1-sn-11 mlslabel - dc1-sn-11 sync standard default dc1-sn-11 refcompressratio 1.00x - dc1-sn-11 written 176K - dc1-sn-11 logicalused 44.2G - dc1-sn-11 logicalreferenced 11.5K - dc1-sn-11 volmode default default dc1-sn-11 filesystem_limit none default dc1-sn-11 snapshot_limit none default dc1-sn-11 filesystem_count none default dc1-sn-11 snapshot_count none default dc1-sn-11 redundant_metadata all default Regards, Paul Esson t +44 (0)118 951 5235 | m +44 (0)776 690 6514 e paul.esson@redstor.com