From owner-freebsd-stable@freebsd.org Wed May 1 02:26:21 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 441811580FA6; Wed, 1 May 2019 02:26:21 +0000 (UTC) (envelope-from michelle@sorbs.net) Received: from hades.sorbs.net (hades.sorbs.net [72.12.213.40]) by mx1.freebsd.org (Postfix) with ESMTP id 2CC0071F82; Wed, 1 May 2019 02:26:19 +0000 (UTC) (envelope-from michelle@sorbs.net) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII; format=flowed Received: from isux.com (gate.mhix.org [203.206.128.220]) by hades.sorbs.net (Oracle Communications Messaging Server 7.0.5.29.0 64bit (built Jul 9 2013)) with ESMTPSA id <0PQT00M540R1Z500@hades.sorbs.net>; Tue, 30 Apr 2019 19:40:16 -0700 (PDT) Subject: Re: ZFS... To: Xin LI Cc: rainer@ultra-secure.de, owner-freebsd-stable@freebsd.org, freebsd-stable , Andrea Venturoli References: <30506b3d-64fb-b327-94ae-d9da522f3a48@sorbs.net> <56833732-2945-4BD3-95A6-7AF55AB87674@sorbs.net> <3d0f6436-f3d7-6fee-ed81-a24d44223f2f@netfence.it> <17B373DA-4AFC-4D25-B776-0D0DED98B320@sorbs.net> <70fac2fe3f23f85dd442d93ffea368e1@ultra-secure.de> <70C87D93-D1F9-458E-9723-19F9777E6F12@sorbs.net> From: Michelle Sullivan Message-id: Date: Wed, 01 May 2019 12:26:15 +1000 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:51.0) Gecko/20100101 Firefox/51.0 SeaMonkey/2.48 In-reply-to: X-Rspamd-Queue-Id: 2CC0071F82 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of michelle@sorbs.net designates 72.12.213.40 as permitted sender) smtp.mailfrom=michelle@sorbs.net X-Spamd-Result: default: False [-2.68 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:hades.sorbs.net]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[sorbs.net]; RCPT_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: battlestar.sorbs.net]; NEURAL_HAM_SHORT(-0.81)[-0.810,0]; RCVD_IN_DNSWL_NONE(0.00)[40.213.12.72.list.dnswl.org : 127.0.10.0]; SUBJ_ALL_CAPS(0.45)[6]; IP_SCORE(-0.62)[ip: (-1.60), ipnet: 72.12.192.0/19(-0.79), asn: 11114(-0.61), country: US(-0.06)]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; CTE_CASE(0.50)[]; ASN(0.00)[asn:11114, ipnet:72.12.192.0/19, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 May 2019 02:26:21 -0000 Xin LI wrote: > > On Tue, Apr 30, 2019 at 5:08 PM Michelle Sullivan > wrote: > > but in my recent experience 2 issues colliding at the same time > results in disaster > > > Do we know exactly what kind of corruption happen to your pool? If > you see it twice in a row, it might suggest a software bug that should > be investigated. > Oh I did spot one interesting bug... though it is benign... Check out the following (note the difference between 'zpool status' and 'zpool status -v'): root@colossus:/mnt # zpool status pool: storage state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Mon Apr 29 20:22:03 2019 6.54T scanned at 0/s, 6.54T issued at 0/s, 28.8T total 445G resilvered, 22.66% done, no estimated completion time config: NAME STATE READ WRITE CKSUM storage ONLINE 0 0 5 raidz2-0 ONLINE 0 0 20 mfid11 ONLINE 0 0 0 mfid10 ONLINE 0 0 0 mfid8 ONLINE 0 0 0 mfid7 ONLINE 0 0 0 mfid0 ONLINE 0 0 0 mfid5 ONLINE 0 0 0 mfid4 ONLINE 0 0 0 mfid3 ONLINE 0 0 0 mfid2 ONLINE 0 0 0 mfid14 ONLINE 0 0 0 mfid15 ONLINE 0 0 0 mfid6 ONLINE 0 0 0 mfid9 ONLINE 0 0 0 mfid13 ONLINE 0 0 0 mfid1 ONLINE 0 0 0 errors: 4 data errors, use '-v' for a list root@colossus:/mnt # zpool status pool: storage state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Mon Apr 29 20:22:03 2019 6.54T scanned at 0/s, 6.54T issued at 0/s, 28.8T total 445G resilvered, 22.66% done, no estimated completion time config: NAME STATE READ WRITE CKSUM storage ONLINE 0 0 5 raidz2-0 ONLINE 0 0 20 mfid11 ONLINE 0 0 0 mfid10 ONLINE 0 0 0 mfid8 ONLINE 0 0 0 mfid7 ONLINE 0 0 0 mfid0 ONLINE 0 0 0 mfid5 ONLINE 0 0 0 mfid4 ONLINE 0 0 0 mfid3 ONLINE 0 0 0 mfid2 ONLINE 0 0 0 mfid14 ONLINE 0 0 0 mfid15 ONLINE 0 0 0 mfid6 ONLINE 0 0 0 mfid9 ONLINE 0 0 0 mfid13 ONLINE 0 0 0 mfid1 ONLINE 0 0 0 errors: 4 data errors, use '-v' for a list root@colossus:/mnt # zpool status -v pool: storage state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Mon Apr 29 20:22:03 2019 6.54T scanned at 0/s, 6.54T issued at 0/s, 28.8T total 445G resilvered, 22.66% done, no estimated completion time config: NAME STATE READ WRITE CKSUM storage ONLINE 0 0 5 raidz2-0 ONLINE 0 0 20 mfid11 ONLINE 0 0 0 mfid10 ONLINE 0 0 0 mfid8 ONLINE 0 0 0 mfid7 ONLINE 0 0 0 mfid0 ONLINE 0 0 0 mfid5 ONLINE 0 0 0 mfid4 ONLINE 0 0 0 mfid3 ONLINE 0 0 0 mfid2 ONLINE 0 0 0 mfid14 ONLINE 0 0 0 mfid15 ONLINE 0 0 0 mfid6 ONLINE 0 0 0 mfid9 ONLINE 0 0 0 mfid13 ONLINE 0 0 0 mfid1 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: :<0x3e> :<0x5d> storage:<0x0> storage@now:<0x0> root@colossus:/mnt # zpool status -v pool: storage state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Mon Apr 29 20:22:03 2019 6.54T scanned at 0/s, 6.54T issued at 0/s, 28.8T total 445G resilvered, 22.66% done, no estimated completion time config: NAME STATE READ WRITE CKSUM storage ONLINE 0 0 7 raidz2-0 ONLINE 0 0 28 mfid11 ONLINE 0 0 0 mfid10 ONLINE 0 0 0 mfid8 ONLINE 0 0 0 mfid7 ONLINE 0 0 0 mfid0 ONLINE 0 0 0 mfid5 ONLINE 0 0 0 mfid4 ONLINE 0 0 0 mfid3 ONLINE 0 0 0 mfid2 ONLINE 0 0 0 mfid14 ONLINE 0 0 0 mfid15 ONLINE 0 0 0 mfid6 ONLINE 0 0 0 mfid9 ONLINE 0 0 0 mfid13 ONLINE 0 0 0 mfid1 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: :<0x3e> :<0x5d> storage:<0x0> storage@now:<0x0> root@colossus:/mnt # zpool status -v pool: storage state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Mon Apr 29 20:22:03 2019 6.54T scanned at 0/s, 6.54T issued at 0/s, 28.8T total 445G resilvered, 22.66% done, no estimated completion time config: NAME STATE READ WRITE CKSUM mfid10 ONLINE 0 0 0 mfid8 ONLINE 0 0 0 mfid7 ONLINE 0 0 0 mfid0 ONLINE 0 0 0 mfid5 ONLINE 0 0 0 mfid4 ONLINE 0 0 0 mfid3 ONLINE 0 0 0 mfid2 ONLINE 0 0 0 mfid14 ONLINE 0 0 0 mfid15 ONLINE 0 0 0 mfid6 ONLINE 0 0 0 mfid9 ONLINE 0 0 0 mfid13 ONLINE 0 0 0 mfid1 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: :<0x3e> :<0x5d> storage:<0x0> storage@now:<0x0> root@colossus:/mnt # zpool status -v pool: storage state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Mon Apr 29 20:22:03 2019 6.54T scanned at 0/s, 6.54T issued at 0/s, 28.8T total 445G resilvered, 22.66% done, no estimated completion time config: NAME STATE READ WRITE CKSUM storage ONLINE 0 0 11 raidz2-0 ONLINE 0 0 44 mfid11 ONLINE 0 0 0 mfid10 ONLINE 0 0 0 mfid8 ONLINE 0 0 0 mfid7 ONLINE 0 0 0 mfid0 ONLINE 0 0 0 mfid5 ONLINE 0 0 0 mfid4 ONLINE 0 0 0 mfid3 ONLINE 0 0 0 mfid2 ONLINE 0 0 0 mfid14 ONLINE 0 0 0 mfid15 ONLINE 0 0 0 mfid6 ONLINE 0 0 0 mfid9 ONLINE 0 0 0 mfid13 ONLINE 0 0 0 mfid1 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: :<0x3e> :<0x5d> storage:<0x0> storage@now:<0x0> root@colossus:/mnt # -- Michelle Sullivan http://www.mhix.org/