Date: Fri, 03 Mar 2017 12:25:02 -0500 From: Allan Jude <allanjude@FreeBSD.org> To: freebsd-current@freebsd.org, "Rodney W. Grimes" <freebsd-rwg@pdx.rh.CN85.dnsmgr.net>, Peter Jeremy <peter@rulingia.com> Cc: freebsd-hackers <freebsd-hackers@freebsd.org>, Subbsd <subbsd@gmail.com>, freebsd-current Current <freebsd-current@freebsd.org>, Ngie Cooper <yaneurabeya@gmail.com>, Alan Somers <asomers@freebsd.org> Subject: Re: effect of strip(1) on du(1) Message-ID: <3FAE8942-2896-4EC6-95C6-D87945E57B29@FreeBSD.org> In-Reply-To: <201703031411.v23EBUdM069969@pdx.rh.CN85.dnsmgr.net> References: <201703031411.v23EBUdM069969@pdx.rh.CN85.dnsmgr.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On March 3, 2017 9:11:30 AM EST, "Rodney W. Grimes" <freebsd-rwg@pdx.rh.CN85.dnsmgr.net> wrote: >-- Start of PGP signed section. >[ Charset ISO-8859-1 unsupported, converting... ] >> On 2017-Mar-02 22:19:10 -0800, "Rodney W. Grimes" ><freebsd-rwg@pdx.rh.CN85.dnsmgr.net> wrote: >> >> du(1) is using fts_read(3), which is based on the stat(2) >information. >> >> The OpenGroup defines st_blocksize as "Number of blocks allocated >for >> >> this object." In the case of ZFS, a write(2) may return before >any >> >> blocks are actually allocated. And thanks to compression, gang >> ... >> >My gut tells me that this is gona cause problems, is it ONLY >> >the st_blocksize data that is incorrect then not such a big >> >problem, or are we returning other meta data that is wrong? >> >> Note that it's st_blocks, not st_blocksize. >Yes, I just ignore that digretion, as well as the digretion into >fts_read >being anything special about this, as it just ends up calling stat(2) >in >the end anyway. > >> >> I did an experiment, writing a (roughly) 113MB file (some data I had >> lying around), close()ing it and then stat()ing it in a loop. This >is >> FreeBSD 10.3 with ZFS and lz4 compression. Over the 26ms following >the >> close(), st_blocks gradually rose from 24169 to 51231. It then >stayed >> stable until 4.968s after the close, when st_blocks again started >> increasing until it stabilized after a total of 5.031s at 87483. >Based >> on this, st_blocks reflects the actual number of blocks physically >> written to disk. None of the other fields in the struct stat vary. > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >Thank you for doing the proper regression test, that satisfies me that >we dont have a lattent bug sitting here and infact what we have is >exposure of the kernel caching, which I might be too thrilled about, >is just how its gona have to be. > >> >> The 5s delay is presumably the TXG delay (since this system is >basically >> unloaded). I'm not sure why it writes roughly ? the data immediately >> and the rest as part of the next TXG write. >> >> >My expectactions of executing a stat(2) call on a file would >> >be that the data returned is valid and stable. I think almost >> >any program would expect that. >> >> I think a case could be made that st_blocks is a valid representation >> of "the number of blocks allocated for this object" - with the number >> increasing as the data is physically written to disk. As for it >being >> stable, consider a (hypothetical) filesystem that can transparently >> migrate data between different storage media, with different >compression >> algorithms etc (ZFS will be able to do this once the mythical block >> rewrite code is written). > >I could counter argue that st_blocks is: >st_blocks The actual number of blocks allocated for the file in > 512-byte units. > >Nothing in that says anything about "on disk". So while this thing >is sitting in memory on the TXG queue we should return the number of >512 byte blocks used by the memory holding the data. >I think that would be the more correct thing than exposing the >fact this thing is setting in a write back cache to userland. Can we compare the results of du with du -A? Du will show compression savings, and -A wont ZFS compresses between the write cache and the disk, so the final size may not be know for 5+ seconds -- Allan Jude
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3FAE8942-2896-4EC6-95C6-D87945E57B29>