From owner-freebsd-fs@freebsd.org Fri Feb 12 18:26:50 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C044F52C15D for ; Fri, 12 Feb 2021 18:26:50 +0000 (UTC) (envelope-from artem@artem.ru) Received: from smtp43.i.mail.ru (smtp43.i.mail.ru [94.100.177.103]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Dchm941VJz3jgb for ; Fri, 12 Feb 2021 18:26:48 +0000 (UTC) (envelope-from artem@artem.ru) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail3; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:MIME-Version:Date:Message-ID:From:References:To:Subject:From:Subject:Content-Type:Content-Transfer-Encoding:To:Cc; bh=q0x8MIxLXEt+PvCR0ZRxvdO/pl/QOdKnXnoZZP9eiyM=; b=Q65mvaW3ZOF5NS/9BS0G6Bpz4k5xwM4x+Hsd+feWkimRrwnoAdRRUQ+glg0nJU9eueZMWBnJizOkmkIGo+jDVN3GSsKh2JXbEe2j0x0OsNo+zR17V+YdvnCuqlz+PHEF4roP9Ysw9IB6q1a0kB+qALqedmIUQSDqXkQK/t1BsMw=; Received: by smtp43.i.mail.ru with esmtpa (envelope-from ) id 1lAd9R-00035v-Sw for freebsd-fs@freebsd.org; Fri, 12 Feb 2021 21:26:46 +0300 Subject: Re: Reading a corrupted file on ZFS To: freebsd-fs@freebsd.org References: <0ca45adf-8f60-a4c3-6264-6122444a3ffd@denninger.net> <899c6b4f-2368-7ec2-4dfe-fa09fab35447@artem.ru> <20210212165216.2f613482@fabiankeil.de> <10977ffc-f806-69dd-0cef-d4fd4fc5f649@artem.ru> <2f82f113-9ca1-99a9-a433-89e3ae5edcbe@denninger.net> From: Artem Kuchin Message-ID: <2bf4f69c-9d5d-5ff9-0daa-c87515437ca3@artem.ru> Date: Fri, 12 Feb 2021 21:26:46 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 In-Reply-To: <2f82f113-9ca1-99a9-a433-89e3ae5edcbe@denninger.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: ru X-7564579A: EEAE043A70213CC8 X-77F55803: 4F1203BC0FB41BD981647AC6901E234B663C574FBA2C95D46270E7B1163DBA8E182A05F538085040F1CEDBD3B86A42A6193482A14A167BB09288F1F6D503C09A0ABC74C5CED422AB X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7DB7B102DCB413779EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637B100969577675F2D8638F802B75D45FF5571747095F342E8C7A0BC55FA0FE5FCF9EC670273F5E627C2F49087E0EC2885AA1DC8F64F6E07B1389733CBF5DBD5E913377AFFFEAFD269176DF2183F8FC7C091DAD9F922AA71188941B15DA834481FCF19DD082D7633A0EF3E4896CB9E6436389733CBF5DBD5E9D5E8D9A59859A8B6042F1592492B88C6CC7F00164DA146DA6F5DAA56C3B73B237318B6A418E8EAB8D32BA5DBAC0009BE9E8FC8737B5C22492AA437DA6D0FD5573AA81AA40904B5D9CF19DD082D7633A078D18283394535A93AA81AA40904B5D98AA50765F7900637D2C6F5AE34F7A4E5EC76A7562686271EEC990983EF5C032935872C767BF85DA29E625A9149C048EE0A3850AC1BE2E735444A83B712AC01484AD6D5ED66289B524E70A05D1297E1BB35872C767BF85DA227C277FBC8AE2E8BDCE939D40DBB93CA75ECD9A6C639B01B4E70A05D1297E1BBC6867C52282FAC85D9B7C4F32B44FF57DB1D15339D2937A5BD9CCCA9EDD067B1EDA766A37F9254B7 X-C1DE0DAB: 0D63561A33F958A5C7B56ED0BD56E4046116EA3521413FA66950C235509A2DACD59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA75B869F6C56E712840410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D3455049D7B43D89D648BC0524BCE29F807F024EAAFCA46C5456D3CC815255EFBBF1C643457A317ABA21D7E09C32AA3244C9380642911FC89F1490C99C87E444C65E8FBBEFAE1C4874C3EB3F6AD6EA9203E X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojtEL/kbq1PXG1VcPKW9Xuyw== X-Mailru-Sender: 332320C0CE44B500FCA62DA34E843E9D26B4AD4F4D429112C6B6FF5922734A4F821176FCD32AD5350A1FD29A504278DEE66B5C1DBFD5D09D2FFF0A5F0DFA254CD0701747CC0EF98689F635B1FCB23A66AE208404248635DF X-Mras: Ok X-Rspamd-Queue-Id: 4Dchm941VJz3jgb X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=mail.ru header.s=mail3 header.b=Q65mvaW3; dmarc=none; spf=none (mx1.freebsd.org: domain of artem@artem.ru has no SPF policy when checking 94.100.177.103) smtp.mailfrom=artem@artem.ru X-Spamd-Result: default: False [-3.40 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[mail.ru:s=mail3]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[artem.ru]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[94.100.177.103:from:127.0.2.255]; DKIM_TRACE(0.00)[mail.ru:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_NA(0.00)[no SPF record]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[94.100.177.103:from]; ASN(0.00)[asn:47764, ipnet:94.100.176.0/20, country:RU]; MID_RHS_MATCH_FROM(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs]; RCVD_IN_DNSWL_LOW(-0.10)[94.100.177.103:from] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Feb 2021 18:26:50 -0000 12.02.2021 19:37, Karl Denninger пишет: > On 2/12/2021 11:22, Artem Kuchin wrote: >> >> This is frustrating. why..why.. > > You created a synthetic situation that in the real world almost-never > exists (ONE byte modified in all copies in the same allocation block > but all other data in that block is intact and recoverable.) > I could be 1 GB file with ZFS wisth block size of 1MB and with rotten bits within the same 1MB of block on different disks. How i did it is not important, life is unpredictable, i'm not trying to avoid everything. The question is what to do when it happens. And currently the answer is - nothing. > In almost-all actual cases of "bit rot" it's exactly that; random and > by statistics extraordinarily unlikely to hit all copies at once in > the same allocation block.  Therefore, ZFS can and does fix it; UFS or > FAT silently returns the corrupted data, propagates it, and eventually > screws you down the road. In active fs you are right. But if this is a storage disk with movies and photos, then i can just checksum all files with a little script and recheck once in a while. So, for storage perposes i have all ZFS postitives and also can read as much data as i can. Because for long time storage it is more important to have ability read the data in any case. > > The nearly-every-case situation in the real world where a disk goes > physically bad (I've had this happen *dozens* of times over my IT > career) results in the drive being unable to *NEARLY* is not good enough for me. > return the block at all; You mix device blocks and ZFS block. As far as i remember default ZFS block for checksumming is 16K and for big files storage better to have it around 128K. > In short there are very, very few actual "in the wild" failures where > one byte is damaged and the rest surrounding that one byte is intact > and retrievable.  In most cases where an actual failure occurs the > unreadable data constitutes *at least* a physical sector. > "very very few" is enough for me to think about. One more thing. If you have one bad byte in a block of 16K and you have checksum and recalculate it then it is quite possible to just brute force every byte to match the checksum, thus restoring the data. If you have mirror with two different bytes then bute forcing is even ether, Somehow, ZFS slaps my hands and does not allow to be sure that i can restore data when i needed it and decide myself if it is okay or not. For long time storage of big files it now seems better to store it on UFS mirror, checksum each 512bytes blocks of files and store checksums separetelly and run monthly/weekly "scrub". This way i would sleep better. Artem