From owner-freebsd-fs@freebsd.org Fri Sep 29 06:54:32 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A7D71E252D2 for ; Fri, 29 Sep 2017 06:54:32 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wr0-x22a.google.com (mail-wr0-x22a.google.com [IPv6:2a00:1450:400c:c0c::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 383C980914 for ; Fri, 29 Sep 2017 06:54:32 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wr0-x22a.google.com with SMTP id y95so706587wrb.4 for ; Thu, 28 Sep 2017 23:54:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=+oc7Kwi793UQ5uGnppwnd1inFOUXkKhTpJXopADRNvs=; b=tJYElwGbCH6+qLXEa0/jkv3OkDGeC8YM2/UW+k7aCU/hrGlcoa5KR07nNOfXw2DSAV k1DAzUWSqjlF1KU8rJt8rfbxtW0ldCGMAsj+dYYpw0Gc92xyhpq3BXVE7plZnMdyCNqq znqknlogVGQFZ61QZ4yhxwhlDCMuHsI5MLyU/uCV4oHyULy3LnDbCDa+6ju37eczq9Bc POO72FyZtjn6dOanotbjIMXuZsBP1+b94Xtw5qYeQ98f44hr/5+UeY+blbEERNsRGk88 5Ko1xdJUd1Y5Zm22KtqELI0UDSaASGc48OVBhYcNYK8d1+iCUHcwylGSTpIYqXDOvt17 h04A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=+oc7Kwi793UQ5uGnppwnd1inFOUXkKhTpJXopADRNvs=; b=aldf5lKeQWl3EFAS5r4WlDdwT5J6efqG22oEeWVLZafGJIFGyc/VBs4wIZgU3dqYWA I8/MsQdrYg8bjt3OiHkSvHzfFjU/qbyDPtIOeo6/40eFRuWXsm1lqnXZtwZhdENLvqrL eI8f4+nr6C44N5NZhItv6K5+K6HzKmFqjq7dJnX0QV4yZcgdDlTX8QhK8YJfE2wC+QPk Bm+evyH+Ur6in07cvSVqJSqMV9Z9zjVvq/5ml8RHa3Rzw8OyuiR9FFCLFZSAUrhCMVn2 8CU839329B1fyAJqiUzTAdjKeay1VDy8hr64TWNuYs1JUlZDkk/qnstLuw2ed1K3YKml mYsg== X-Gm-Message-State: AHPjjUjrH9MkjFwVEglmFL7I+QhK4BMa3+qlAo4TglsOd6hsczoU56HJ 7xKL3+ZPXfBwOCdZqkyY43o6npQ4 X-Google-Smtp-Source: AOwi7QDz9zRRPenfyiecCtGiq1XMLf3XUn4P3fdz+UoN+HIIkPY5eiq20pXWYJ8aXxhe4hoibzo1Yg== X-Received: by 10.223.155.194 with SMTP id e2mr7504534wrc.49.1506668069566; Thu, 28 Sep 2017 23:54:29 -0700 (PDT) Received: from bens-mac.home (LFbn-MAR-1-445-220.w2-15.abo.wanadoo.fr. [2.15.38.220]) by smtp.gmail.com with ESMTPSA id i136sm3186044wmd.23.2017.09.28.23.54.28 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 28 Sep 2017 23:54:28 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: How to create holes in files ? From: Ben RUBSON In-Reply-To: <1506666838.7204.39.camel@inparadise.se> Date: Fri, 29 Sep 2017 08:54:28 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <7F95ECD3-3284-4808-9182-AF3FE5970333@gmail.com> References: <12ed413d-00d1-4182-bdd5-0e5230d46f01@email.android.com> <4D304428-E4AF-45D4-AB2E-D4B73A7FEE87@gmail.com> <1506664468.7204.37.camel@inparadise.se> <454261D6-0061-4DB9-8184-BDD7061870FB@gmail.com> <1506666838.7204.39.camel@inparadise.se> To: Freebsd fs X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Sep 2017 06:54:32 -0000 > On 29 Sep 2017, at 08:33, Karli Sj=C3=B6berg = wrote: >=20 > On fre, 2017-09-29 at 08:26 +0200, Ben RUBSON wrote: >>>=20 >>> On 29 Sep 2017, at 07:54, Karli Sj=C3=B6berg >>> wrote: >>>=20 >>> On tor, 2017-09-28 at 22:16 +0200, Ben RUBSON wrote: >>>>=20 >>>>>=20 >>>>>=20 >>>>> On 28 Sep 2017, at 20:48, Karli Sj=C3=B6berg >>>>> wrote: >>>>>=20 >>>>> Den 28 sep. 2017 6:47 em skrev Ben RUBSON >>>>> : >>>>>>=20 >>>>>>=20 >>>>>> On 28 Sep 2017, at 18:34, Bob Eager wrote: >>>>>>=20 >>>>>> On Thu, 28 Sep 2017 17:26:09 +0200 >>>>>> Fabian Keil wrote: >>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>> Ben RUBSON wrote: >>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> I'm trying to make holes in files in C. >>>>>>>> Goal is to deallocate huge files on ZFS while (randomly) >>>>>>>> reading >>>>>>>> them.=20 >>>>>>> My interpretation of the above is that you want to create >>>>>>> holes >>>>>>> without changing the file size and without affecting data >>>>>>> that >>>>>>> is located before or after the holes that you want to >>>>>>> create. >>>>>>>=20 >>>>>>> Otherwise you could simply "deallocate" the content with >>>>>>> truncate(1). >>>>>> If he doesn't mind copying the files, dd(1) will do the job. >>>>>> However, I >>>>>> expect that doesn't meet his criteria. >>>>> Thank you Bob for your suggestion. >>>>> You're right goal is to avoid copying data : free space would >>>>> not >>>>> necessarily >>>>> allow this, and as the files I'm working on are some hundreds >>>>> of GB >>>>> in size, >>>>> it would really be a counterproductive long stressing storage >>>>> operation. >>>>>=20 >>>>> Well, correct me if I'm wrong, but wouldn't dd with "seek" >>>>> mitigate >>>>> the issue of writing out all of the data from beginning to end. >>>>> If >>>>> you seek from the beginning of the file to the point you want >>>>> to >>>>> start writing from and use bs to specify how large of a "hole" >>>>> you >>>>> want, the operation wouldn't take long at all. You would, in my >>>>> opinion, achieve exactly what you want, to "create holes in >>>>> files". >>>>> Am I wrong? >>>> I would have liked to do this in C. >>> Yepp, I understand that, I was just bringing up the idea. But hey, >>> if >>> dd can do it, just look in the source of how it does it? >> Of course yes, you're right :) >>=20 >>>=20 >>>>=20 >>>> And the holes I need to create are not necessarily at the >>>> beginning >>>> of a file. >>> No, you _do not_ have to create holes in the beginning of a file, >>> you >>> can do it anywhere in the file you like. E.g. it=C2=B4s the trick I = use >>> to >>> "clean" hardrives to make them look empty, by only erasing the >>> first >>> and last 10 MB of the drives, a whole JBOD only takes seconds to >>> clean. >>> Say you have a 10 MB large file and you want to make a 1 MB large >>> hole >>> somewhere in the middle of it, you do it like this: >>>=20 >>> # dd if=3D/dev/zero of=3D/foo/bar.bin bs=3D1M seek=3D7 >>>=20 >>> It=C2=B4ll make the file look like this inside (hope the ASCII gods = are >>> with >>> me): >>> ____________ >>>>=20 >>>> _______|_|__| >> Unfortunately here storage blocks will be set/written to 0, >> they will not be freed/recovered, a file hole will not be created. >> Below are some commands which demonstrate this. >> The last 3/4 commands show a _newly_ created file with a hole. >>=20 >> # zfs userspace home >> POSIX User root 36.5K none >>=20 >> # dd if=3D/dev/random of=3Df bs=3D1M count=3D4 >> 4+0 records out >>=20 >> # zfs userspace home >> POSIX User root 4.04M none >>=20 >> # dd if=3D/dev/zero of=3Df bs=3D1M count=3D2 seek=3D2 >> 2+0 records out >>=20 >> # zfs userspace home >> POSIX User root 4.04M none >>=20 >> # rm f >>=20 >> # dd if=3D/dev/zero of=3Df bs=3D1M count=3D2 seek=3D2 >> 2+0 records out >>=20 >> # zfs userspace home >> POSIX User root 2.04M none >>=20 >> # ls -lh f >> -rw------- 1 root wheel 4.0M 29 Sep 08:19 f >=20 > Ah, I see, nice demonstration! But wouldn=C2=B4t TRIM take of that = though > (if that=C2=B4s available)? Good question ! I'm however not sure TRIM would notify ZFS about space not consumed. In addition I'm mainly on non-SSD drives :-/ Ben