From owner-freebsd-fs@freebsd.org Thu Feb 23 00:49:45 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 524C5CE738B for ; Thu, 23 Feb 2017 00:49:45 +0000 (UTC) (envelope-from eborisch@gmail.com) Received: from mail-it0-x235.google.com (mail-it0-x235.google.com [IPv6:2607:f8b0:4001:c0b::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 23ACF1CFB for ; Thu, 23 Feb 2017 00:49:45 +0000 (UTC) (envelope-from eborisch@gmail.com) Received: by mail-it0-x235.google.com with SMTP id 203so235740ith.0 for ; Wed, 22 Feb 2017 16:49:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=qQVQfCQ8Y6p23O+/N+wIbrVB58bfzOKWQqNBpQ34XXk=; b=ivmyJhjgxh9HJ2Lqp6RRLnDSm3mNklzUop318JzQ8Gjy8+oyPPaa9zCi90jbjcsR+Z n55K95OpCBVIcXkaRZH6cUnF5RotVbvNjIWIjSdOgrwtfQjXjGDIw/9fFPF68Qa2EVRX tPhUIoc87DXU2OLnyOWK7g1VA1tEaEgIayjWMka8xQvpn9wXtk2b5naNpoQCdirGl6cb 1AUrcBCoMYMgjxcsQYu1w+km5wzZ8VaOoxGw491p4XE6/+2zCUcBoipuLG0uqIBVIgYD lfNe+we/7vRCU9cjYH/Aj76aE6sT29f+VIE8eOHPmAxoWgOMrCxh4U6n1D0QZvgYyI45 CNkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=qQVQfCQ8Y6p23O+/N+wIbrVB58bfzOKWQqNBpQ34XXk=; b=kWdD6BNRDk4EZRgH6g+vejU0VvVgukrLvyxAbvreX579GLk+xpoxkBtvi+M7z1KHIp uSTqUwhKlJwnlgr7XU6fCQJEn48Qzipyx4Yl90vSbGLIRBO/lA0MKfvM9R3WNHlP8XKh Vozg+zdsJyVlx1/K6LMt/Rx4a4G7Og0r8tC+gvP9Q/M+6WpOSkD725JWSYxcI9JveD+g kyf9KbvjZNvK9xX56w88tCwdiMesjmZrf91vuI68LE5TS3RnjWw0nJ/spkHIHKXB4fjp Rg9soeSgFEiQLVe1ewpAFyjlYA+t8s5pwya1V8OUg1+ohYx5xb1ExKMMMjFEwF4xzpNy Rg1A== X-Gm-Message-State: AMke39ms03SuP7tsTlw5pzfxunoTB1VXVdF7/ws8OQ70kd4QAJIm3qxjpSR9+iTAinTxA7N+zsigYHkmVjtb4A== X-Received: by 10.107.3.195 with SMTP id e64mr9275037ioi.18.1487810984437; Wed, 22 Feb 2017 16:49:44 -0800 (PST) MIME-Version: 1.0 Received: by 10.107.183.148 with HTTP; Wed, 22 Feb 2017 16:49:43 -0800 (PST) In-Reply-To: References: <1b54a2fe35407a95edca1f992fa08a71@norman-vivat.ru> From: "Eric A. Borisch" Date: Wed, 22 Feb 2017 18:49:43 -0600 Message-ID: Subject: Re: zfs raidz overhead To: Wiktor Niesiobedzki Cc: "Eugene M. Zheganin" , "freebsd-fs@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Feb 2017 00:49:45 -0000 On Wed, Feb 22, 2017 at 3:50 PM, Wiktor Niesiobedzki wrote: > I can add to this, that this is not only seen on raidz, but also on > mirror pools, such as this: > # zpool status tank > pool: tank > state: ONLINE > scan: scrub repaired 0 in 3h22m with 0 errors on Thu Feb 9 06:47:07 2017 > config: > > NAME STATE READ WRITE CKSUM > tank ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > gpt/tank1.eli ONLINE 0 0 0 > gpt/tank2.eli ONLINE 0 0 0 > > errors: No known data errors > > > When I createted test zvols: > # zfs create -V10gb -o volblocksize=8k tank/tst-8k > # zfs create -V10gb -o volblocksize=16k tank/tst-16k > # zfs create -V10gb -o volblocksize=32k tank/tst-32k > # zfs create -V10gb -o volblocksize=64k tank/tst-64k > # zfs create -V10gb -o volblocksize=128k tank/tst-128k > > # zfs get used tank/tst-8k > NAME PROPERTY VALUE SOURCE > tank/tst-8k used 10.3G - > root@kadlubek:~ # zfs get used tank/tst-16k > NAME PROPERTY VALUE SOURCE > tank/tst-16k used 10.2G - > root@kadlubek:~ # zfs get used tank/tst-32k > NAME PROPERTY VALUE SOURCE > tank/tst-32k used 10.1G - > root@kadlubek:~ # zfs get used tank/tst-64k > NAME PROPERTY VALUE SOURCE > tank/tst-64k used 10.0G - > root@kadlubek:~ # zfs get used tank/tst-128k > NAME PROPERTY VALUE SOURCE > tank/tst-128k used 10.0G - Nope, that all looks correct. There is space reserved for metadata (checksum for example) when you create a zvol. Since checksums are computed by block, it makes sense that more metadata is required for the 8k volblocksize. > I also noted, that snapshots impact used stats far much, than usedbysnapshot value: > zfs get volsize,used,referenced,compressratio,volblocksize, > usedbysnapshots,usedbydataset,usedbychildren > tank/dkr-thinpool > NAME PROPERTY VALUE SOURCE > tank/dkr-thinpool volsize 10G local > tank/dkr-thinpool used 12.0G - > tank/dkr-thinpool referenced 1.87G - > tank/dkr-thinpool compressratio 1.91x - > tank/dkr-thinpool volblocksize 64K - > tank/dkr-thinpool usedbysnapshots 90.4M - > tank/dkr-thinpool usedbydataset 1.87G - > tank/dkr-thinpool usedbychildren 0 - > > > On a 10G volume, filled with 2G of data, and 90M used by snapshosts, > used is 2G. When I destroy the snapshots, used will drop to 10.0G. > It makes sense when you just remember that 'used' is enforcing a contract to the user that 10G of new data can be written to the zvol (refreservation, I'm sure it's set to 10G here), and still have all of the current snapshots will still be available. Breaking that down, usedbysnapshots refers to the data that is no longer referenced by the dataset (overwritten, in the case of a zvol.) Used is the space the the dataset is consuming (i.e. not available for other datasets). This means that 10G (refreservation)+ the referenced property of your earliest snapshot + the written value of all but your latest snapshot + the aforementioned metadata overhead needs to be reserved, or 'used' by the dataset. (I think I got that calculation right.) So even though usedbysnapshots is 90M (90M of blocks have been overwritten), the referenced and written properties of the snapshot accumulate to the space you're seeing here. Once all the snapshots are gone, used drops to (almost) the volsize. Hope that helps. If you want thin provisioning, you can unset the refreservation, or just create the volume with the '-s' option. - Eric