From owner-freebsd-fs@freebsd.org Fri Sep 29 06:34:01 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C8CC5E24CEF for ; Fri, 29 Sep 2017 06:34:01 +0000 (UTC) (envelope-from karli@inparadise.se) Received: from mail.inparadise.se (h-246-50.A444.priv.bahnhof.se [155.4.246.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6CDED80016 for ; Fri, 29 Sep 2017 06:34:01 +0000 (UTC) (envelope-from karli@inparadise.se) Received: from localhost (localhost [127.0.0.1]) by mail.inparadise.se (Postfix) with ESMTP id 45D264A9E0; Fri, 29 Sep 2017 08:33:59 +0200 (CEST) Received: from mail.inparadise.se ([127.0.0.1]) by localhost (mail.inparadise.se [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id ujHXEG04xetE; Fri, 29 Sep 2017 08:33:58 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by mail.inparadise.se (Postfix) with ESMTP id A39314B740; Fri, 29 Sep 2017 08:33:58 +0200 (CEST) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.inparadise.se A39314B740 X-Virus-Scanned: amavisd-new at inparadise.se Received: from mail.inparadise.se ([127.0.0.1]) by localhost (mail.inparadise.se [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 02UtlZVP-ynq; Fri, 29 Sep 2017 08:33:58 +0200 (CEST) Received: from vpnpool190-35.anst.uu.se (vpnpool190-35.anst.uu.se [130.238.190.35]) by mail.inparadise.se (Postfix) with ESMTPSA id 85B2E4A633; Fri, 29 Sep 2017 08:33:58 +0200 (CEST) Message-ID: <1506666838.7204.39.camel@inparadise.se> Subject: Re: How to create holes in files ? From: Karli =?ISO-8859-1?Q?Sj=F6berg?= Reply-To: karli@inparadise.se To: Ben RUBSON , Freebsd fs Date: Fri, 29 Sep 2017 08:33:58 +0200 In-Reply-To: <454261D6-0061-4DB9-8184-BDD7061870FB@gmail.com> References: <12ed413d-00d1-4182-bdd5-0e5230d46f01@email.android.com> <4D304428-E4AF-45D4-AB2E-D4B73A7FEE87@gmail.com> <1506664468.7204.37.camel@inparadise.se> <454261D6-0061-4DB9-8184-BDD7061870FB@gmail.com> Organization: InParadise Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.18.5.2-0ubuntu3.2 Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Sep 2017 06:34:01 -0000 On fre, 2017-09-29 at 08:26 +0200, Ben RUBSON wrote: > >=20 > > On 29 Sep 2017, at 07:54, Karli Sj=C3=B6berg > > wrote: > >=20 > > On tor, 2017-09-28 at 22:16 +0200, Ben RUBSON wrote: > > >=20 > > > >=20 > > > >=20 > > > > On 28 Sep 2017, at 20:48, Karli Sj=C3=B6berg > > > > wrote: > > > >=20 > > > > Den 28 sep. 2017 6:47 em skrev Ben RUBSON > > > >: > > > > >=20 > > > > >=20 > > > > > On 28 Sep 2017, at 18:34, Bob Eager wrote: > > > > >=20 > > > > > On Thu, 28 Sep 2017 17:26:09 +0200 > > > > > Fabian Keil wrote: > > > > >=20 > > > > > >=20 > > > > > >=20 > > > > > > Ben RUBSON wrote: > > > > > >=20 > > > > > > >=20 > > > > > > >=20 > > > > > > > I'm trying to make holes in files in C. > > > > > > > Goal is to deallocate huge files on ZFS while (randomly) > > > > > > > reading > > > > > > > them.=C2=A0 > > > > > > My interpretation of the above is that you want to create > > > > > > holes > > > > > > without changing the file size and without affecting data > > > > > > that > > > > > > is located before or after the holes that you want to > > > > > > create. > > > > > >=20 > > > > > > Otherwise you could simply "deallocate" the content with > > > > > > truncate(1). > > > > > If he doesn't mind copying the files, dd(1) will do the job. > > > > > However, I > > > > > expect that doesn't meet his criteria. > > > > Thank you Bob for your suggestion. > > > > You're right goal is to avoid copying data : free space would > > > > not > > > > necessarily > > > > allow this, and as the files I'm working on are some hundreds > > > > of GB > > > > in size, > > > > it would really be a counterproductive long stressing storage > > > > operation. > > > >=20 > > > > Well, correct me if I'm wrong, but wouldn't dd with "seek" > > > > mitigate > > > > the issue of writing out all of the data from beginning to end. > > > > If > > > > you seek from the beginning of the file to the point you want > > > > to > > > > start writing from and use bs to specify how large of a "hole" > > > > you > > > > want, the operation wouldn't take long at all. You would, in my > > > > opinion, achieve exactly what you want, to "create holes in > > > > files". > > > > Am I wrong? > > > I would have liked to do this in C. > > Yepp, I understand that, I was just bringing up the idea. But hey, > > if > > dd can do it, just look in the source of how it does it? > Of course yes, you're right :) >=20 > >=20 > > >=20 > > > And the holes I need to create are not necessarily at the > > > beginning > > > of a file. > > No, you _do not_ have to create holes in the beginning of a file, > > you > > can do it anywhere in the file you like. E.g. it=C2=B4s the trick I u= se > > to > > "clean" hardrives to make them look empty, by only erasing the > > first > > and last 10 MB of the drives, a whole JBOD only takes seconds to > > clean. > > Say you have a 10 MB large file and you want to make a 1 MB large > > hole > > somewhere in the middle of it, you do it like this: > >=20 > > # dd if=3D/dev/zero of=3D/foo/bar.bin bs=3D1M seek=3D7 > >=20 > > It=C2=B4ll make the file look like this inside (hope the ASCII gods a= re > > with > > me): > > =C2=A0____________ > > >=20 > > > _______|_|__| > Unfortunately here storage blocks will be set/written to 0, > they will not be freed/recovered, a file hole will not be created. > Below are some commands which demonstrate this. > The last 3/4 commands show a _newly_ created file with a hole. >=20 > # zfs userspace home > POSIX User=C2=A0=C2=A0root=C2=A0=C2=A036.5K=C2=A0=C2=A0=C2=A0none >=20 > # dd if=3D/dev/random of=3Df bs=3D1M count=3D4 > 4+0 records out >=20 > # zfs userspace home > POSIX User=C2=A0=C2=A0root=C2=A0=C2=A04.04M=C2=A0=C2=A0=C2=A0none >=20 > # dd if=3D/dev/zero of=3Df bs=3D1M count=3D2 seek=3D2 > 2+0 records out >=20 > # zfs userspace home > POSIX User=C2=A0=C2=A0root=C2=A0=C2=A04.04M=C2=A0=C2=A0=C2=A0none >=20 > # rm f >=20 > # dd if=3D/dev/zero of=3Df bs=3D1M count=3D2 seek=3D2 > 2+0 records out >=20 > # zfs userspace home > POSIX User=C2=A0=C2=A0root=C2=A0=C2=A02.04M=C2=A0=C2=A0=C2=A0none >=20 > # ls -lh f > -rw-------=C2=A0=C2=A01 root=C2=A0=C2=A0wheel=C2=A0=C2=A0=C2=A04.0M 29 = Sep 08:19 f Ah, I see, nice demonstration! But wouldn=C2=B4t TRIM take of that though (if that=C2=B4s available)? /K >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"