From nobody Fri Jan 14 12:52:32 2022 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 64CB81946CB8 for ; Fri, 14 Jan 2022 12:52:41 +0000 (UTC) (envelope-from feenberg@nber.org) Received: from mail2.nber.org (mail2.nber.org [198.71.6.79]) by mx1.freebsd.org (Postfix) with ESMTP id 4Jb1RX4sx6z4bgL; Fri, 14 Jan 2022 12:52:39 +0000 (UTC) (envelope-from feenberg@nber.org) Received: from mail2.nber.org (mail2.nber.org [198.71.6.79]) by mail2.nber.org (8.16.1/8.15.2) with ESMTPS id 20ECqWLB042695 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT); Fri, 14 Jan 2022 07:52:32 -0500 (EST) (envelope-from feenberg@nber.org) Date: Fri, 14 Jan 2022 07:52:32 -0500 (EST) From: Daniel Feenberg To: "Kevin P. Neal" cc: "Greg 'groggy' Lehey" , David Christensen , freebsd-questions@freebsd.org Subject: Re: zero filling a storage device (was: dd and mbr) In-Reply-To: Message-ID: References: <77680665-7ddb-23c5-e866-05d112339b60@holgerdanske.com> <20220114023002.GP61872@eureka.lemis.com> List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-KLMS-Rule-ID: 1 X-KLMS-Message-Action: clean X-KLMS-AntiSpam-Status: not scanned, disabled by settings X-KLMS-AntiSpam-Interceptor-Info: not scanned X-KLMS-AntiPhishing: Clean, 1970/01/01 00:00:00 X-KLMS-AntiVirus: Kaspersky Security 8.0 for Linux Mail Server, version 8.0.1.721, bases: 2022/01/14 01:58:00 #13273300 X-KLMS-AntiVirus-Status: Clean, skipped X-Rspamd-Queue-Id: 4Jb1RX4sx6z4bgL X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of feenberg@nber.org designates 198.71.6.79 as permitted sender) smtp.mailfrom=feenberg@nber.org X-Spamd-Result: default: False [-1.57 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; FROM_HAS_DN(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[nber.org]; NEURAL_HAM_LONG(-0.87)[-0.870]; RWL_MAILSPIKE_GOOD(0.00)[198.71.6.79:from]; NEURAL_SPAM_SHORT(1.00)[1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_MED(-0.40)[198.71.6.79:received,198.71.6.79:from]; RCVD_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-questions]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:26287, ipnet:198.71.6.0/23, country:US]; RCVD_TLS_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On Thu, 13 Jan 2022, Kevin P. Neal wrote: > On Fri, Jan 14, 2022 at 01:30:02PM +1100, Greg 'groggy' Lehey wrote: >> On Thursday, 13 January 2022 at 17:26:12 -0800, David Christensen wrote: >>> On 1/11/22 2:02 AM, Roderick wrote: >>> >>> >>> On 1/13/22 12:32 AM, Roderick wrote: >> >> Yes, that's fine. If the block size is larger than the maximum that >> the disk can handle, the system divides it into smaller transfers. >> >>> Suppose I have a storage device with M blocks of size B bytes each. >>> Some of the blocks contain non-zero bytes and other blocks contain only >>> zero bytes. >>> >>> Suppose I have a program with a memory buffer of N blocks filled with >>> zero bytes, where N < M. >>> >>> If the program writes the buffer to the storage device starting at >>> block-aligned location P, where P + N * B <= M * B, will the storage >>> device write zero bytes to the storage device blocks that already >>> contain only zero bytes? >> >> Yes. The storage device doesn't know the current contents of the >> disk. To do so it would first have to read them, which takes far too >> long. >> >>> Is the answer different for a HDD, an SSD, a USB flash drive, an SD >>> card, a compact flash card, etc.? >> >> No. It's also not different for other operating systems. > > Are we certain that an SSD won't at least track that there is nothing > written to a logical block and therefore it must be all zeros? I'm not > 100% that an SSD will always keep a logical block assigned to a physical > block. And I'm not 100% certain that an SSD won't notice that all zeros > are being written to a block and just optimize out the write. > > -- > Kevin P. Neal http://www.pobox.com/~kpn/ > > "What is mathematics? The age-old answer is, of course, that mathematics > is what mathematicians do." - Donald Knuth > Doesn't the filesystem code handle sparse files on its own? This man page: https://www.freebsd.org/cgi/man.cgi?du strongly implies that it does. A block of all zero bytes shouldn't occupy hardly any disk space at all. See, for instance: https://en.wikipedia.org/wiki/Sparse_file Of course, if the purpose of the write is to zero out blocks for security, then writing a sparse file won't do that. Daniel Feenberg