From owner-freebsd-fs@FreeBSD.ORG Mon Dec 9 18:58:48 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2BA78EFD for ; Mon, 9 Dec 2013 18:58:48 +0000 (UTC) Received: from bronze.cs.yorku.ca (bronze.cs.yorku.ca [130.63.95.34]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D299B15E7 for ; Mon, 9 Dec 2013 18:58:47 +0000 (UTC) Received: from [130.63.97.125] (ident=jas) by bronze.cs.yorku.ca with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.76) (envelope-from ) id 1Vq5n3-0006i7-4v; Mon, 09 Dec 2013 13:42:45 -0500 Message-ID: <52A60F25.8040800@cse.yorku.ca> Date: Mon, 09 Dec 2013 13:42:45 -0500 From: Jason Keltz User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.1.0 MIME-Version: 1.0 To: Matthew Ahrens Subject: Re: question about zfs written property References: <52A5F80D.8020206@cse.yorku.ca> In-Reply-To: X-Spam-Score: -1.0 X-Spam-Level: - X-Spam-Report: Content preview: On 12/09/2013 01:24 PM, Matthew Ahrens wrote: > Jason, I'm cc'ing the freebsd mailing list as well. In general that > is a better forum for questions about how to use ZFS. > Thanks.. > > On Mon, Dec 9, 2013 at 9:04 AM, Jason Keltz > wrote: > > Hi.. > > I saw your names on the feature addition in illumos for the > "written" property for ZFS: > > https://www.illumos.org/issues/1645 > > I had a question and was hoping you might have a moment to answer. > > I'm rsyncing data from a Linux-based system to a ZFS backup and > archive server. It's running FreeBSD, but it's the same ZFS code > base as illumos. I'm seeing (what I think are) some weird numbers > looking at ZFS written property ... > > For example: > > Sat Dec 7 01:05:00 EST 2013 sync start > rsync://backup@forest-mrpriv/home9 /local/backup/home9 > (189G/264G/1.41x) > Sat Dec 7 01:33:20 EST 2013 sync finish > rsync://backup@forest-mrpriv/home9 /local/backup/home9 > (190G/265G/1.41x) > Sat Dec 7 01:33:20 EST 2013 sync elapsed 00h:28m:20s > rsync://backup@forest-mrpriv/home9 /local/backup/home9, 514M > unarchived > Sat Dec 7 06:32:58 EST 2013 archive create home9 daily 20131207 > as pool1/backup/home9@20131207, 518M > > In the third line, where you see "514M unarchived", I write out > the property of "written" after the rsync completes. However, > when the archive (just a snapshot) runs (hours later), there's 4 > MB more data!? Nothing touches the data after the rsync completes. > Both lines are probing the same property on the same dataset. > How can they get a different result? > > > If you are getting the "written" property just after the rsync > completes, it's possible that there is still some data "in flight" > inside ZFS. If you run "sync", that should flush out all the dirty > data and update the space accounting. Unfortunately this is only > documented in the description of the "used" property, we should add > similar qualifiers to "available", "referenced", "written", > "logicalreferenced", etc.: > > The [...] Content analysis details: (-1.0 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SHORTCIRCUIT Not all rules were run, due to a shortcircuited rule -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Dec 2013 18:58:48 -0000 On 12/09/2013 01:24 PM, Matthew Ahrens wrote: > Jason, I'm cc'ing the freebsd mailing list as well. In general that > is a better forum for questions about how to use ZFS. > Thanks.. > > On Mon, Dec 9, 2013 at 9:04 AM, Jason Keltz > wrote: > > Hi.. > > I saw your names on the feature addition in illumos for the > "written" property for ZFS: > > https://www.illumos.org/issues/1645 > > I had a question and was hoping you might have a moment to answer. > > I'm rsyncing data from a Linux-based system to a ZFS backup and > archive server. It's running FreeBSD, but it's the same ZFS code > base as illumos. I'm seeing (what I think are) some weird numbers > looking at ZFS written property ... > > For example: > > Sat Dec 7 01:05:00 EST 2013 sync start > rsync://backup@forest-mrpriv/home9 /local/backup/home9 > (189G/264G/1.41x) > Sat Dec 7 01:33:20 EST 2013 sync finish > rsync://backup@forest-mrpriv/home9 /local/backup/home9 > (190G/265G/1.41x) > Sat Dec 7 01:33:20 EST 2013 sync elapsed 00h:28m:20s > rsync://backup@forest-mrpriv/home9 /local/backup/home9, 514M > unarchived > Sat Dec 7 06:32:58 EST 2013 archive create home9 daily 20131207 > as pool1/backup/home9@20131207, 518M > > In the third line, where you see "514M unarchived", I write out > the property of "written" after the rsync completes. However, > when the archive (just a snapshot) runs (hours later), there's 4 > MB more data!? Nothing touches the data after the rsync completes. > Both lines are probing the same property on the same dataset. > How can they get a different result? > > > If you are getting the "written" property just after the rsync > completes, it's possible that there is still some data "in flight" > inside ZFS. If you run "sync", that should flush out all the dirty > data and update the space accounting. Unfortunately this is only > documented in the description of the "used" property, we should add > similar qualifiers to "available", "referenced", "written", > "logicalreferenced", etc.: > > The amount of space used, available, or referenced does > > not take into account pending changes. Pending changes > > are generally accounted for within a few seconds. Com- > > mitting a change to a disk using fsync(3c) or O_SYNC > > does not necessarily guarantee that the space usage > > information is updated immediately. > > Yes. I repeated the test. It takes an additional 3 seconds after my 6 MB rsync completes for "written" to produce the correct result. The result is repeatable. It doesn't appear that running "sync" actually reduces the delay. A sleep() works though :) (I know I could sure use one!) Jason.