From owner-freebsd-questions@freebsd.org Mon Mar 30 18:13:47 2020 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CC92E262663 for ; Mon, 30 Mar 2020 18:13:47 +0000 (UTC) (envelope-from FreeBSD@chroot.pl) Received: from mail.apsz.com.pl (mail.apsz.com.pl [91.217.18.46]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48rgZ91XSnz412V for ; Mon, 30 Mar 2020 18:13:37 +0000 (UTC) (envelope-from FreeBSD@chroot.pl) Received: from chroot.pl (89-74-178-152.dynamic.chello.pl [89.74.178.152]) by mail.apsz.com.pl (Postfix) with ESMTPS id AA105E745A for ; Mon, 30 Mar 2020 20:06:58 +0200 (CEST) From: Lukasz DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=chroot.pl; s=mail; t=1585591618; bh=7L/mVyyuuwee3CQHID8AM7ifI5J5iXZ+xc3dCgC1MWg=; h=From:Subject:References:To:Date:In-Reply-To; b=uzHpwbPNLkZBXQcsn5zOLgvHJeEu1ERLM/NxWIvSP9m6SJRZb+pE2w7T+k3RhqAm8 2YqyDgD43xFc0rePhS89oLsYwox7/AAP2QrX90MseYFSnVm7RxyJ4SolDv35vn8TqX Mqf3QkUwGiTUgCgt7eO5Wc/ScxhWV39uKY/kJ/I8FxNGl7E3G74uOOc3flR+POXB5o 9EpMvrMYO40mwqq3nRCXBqFAr7avFftjZnF+f816l9XTL8HXn6SQidzKEMZz2S9Drd QmZ8gTwD9yz5Y75HgxLkfomHCxff5uSiVJMPBzbYE6KxHsXTizSXQzR0//4ONLIDZB X2NxijYfq1T+w== Subject: Re: replace disk in zpool - solved References: <18a94704-5411-3b44-a525-2ae50121a467@holgerdanske.com> <4a8d409e-ecac-77c8-3ad9-025aefdfb4ef@holgerdanske.com> <20200325081814.GK35528@mithril.foucry.net> To: freebsd-questions@freebsd.org Message-ID: Date: Mon, 30 Mar 2020 20:06:56 +0200 User-Agent: WebMail MIME-Version: 1.0 In-Reply-To: <20200325081814.GK35528@mithril.foucry.net> Content-Type: text/plain; charset=utf-8 Content-Language: pl-PL Content-Transfer-Encoding: 8bit X-Spam-Status: Yes, score=6.3 required=4.0 tests=BAYES_50,KHOP_HELO_FCRDNS, NO_FM_NAME_IP_HOSTN,RDNS_DYNAMIC autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on mail.apsz.com.pl X-Virus-Scanned: clamav-milter 0.102.2 at mail.apsz.com.pl X-Virus-Status: Clean X-Spam-Flag: YES X-Spam-Report: * 2.0 BAYES_50 BODY: Bayes spam probability is 40 to 60% * [score: 0.5967] * 1.7 RDNS_DYNAMIC Delivered to internal network by host with * dynamic-looking rDNS * 0.4 KHOP_HELO_FCRDNS Relay HELO differs from its IP's reverse DNS * 2.2 NO_FM_NAME_IP_HOSTN No From name + hostname using IP address X-Spam-Level: ****** X-Rspamd-Queue-Id: 48rgZ91XSnz412V X-Spamd-Bar: +++ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=chroot.pl header.s=mail header.b=uzHpwbPN; dmarc=pass (policy=none) header.from=chroot.pl; spf=pass (mx1.freebsd.org: domain of FreeBSD@chroot.pl designates 91.217.18.46 as permitted sender) smtp.mailfrom=FreeBSD@chroot.pl X-Spamd-Result: default: False [3.66 / 15.00]; ARC_NA(0.00)[]; GREYLIST(0.00)[pass,body]; R_DKIM_ALLOW(-0.20)[chroot.pl:s=mail]; SPAM_FLAG(5.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:91.217.18.46:c]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; XM_UA_NO_VERSION(0.01)[]; RECEIVED_SPAMHAUS_PBL(0.00)[152.178.74.89.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.11]; NEURAL_HAM_MEDIUM(-0.21)[-0.207,0]; DKIM_TRACE(0.00)[chroot.pl:+]; DMARC_POLICY_ALLOW(-0.50)[chroot.pl,none]; RCVD_IN_DNSWL_NONE(0.00)[46.18.217.91.list.dnswl.org : 127.0.10.0]; NEURAL_HAM_LONG(-0.16)[-0.158,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(0.01)[country: PL(0.06)]; ASN(0.00)[asn:51426, ipnet:91.217.18.0/23, country:PL]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Mar 2020 18:13:47 -0000 Hello, this behavior was due to errors in zpool. Regards, Lukasz On 3/25/20 09:18, Jacques Foucry via freebsd-questions wrote: > Le mardi 24 mars 2020 à 16:47:10 (-0700), David Christensen à écrit: >> On 2020-03-24 14:15, Lukasz wrote: >>> Ohh… I forgot mention: >>> it's 12.1-p3 >>> >>> # zpool status -v mypool >>> pool: mypool >>> state: DEGRADED >>> status: One or more devices has experienced an error resulting in data >>> corruption. Applications may be affected. >>> action: Restore the file in question if possible. Otherwise restore the >>> entire pool from backup. >>> see: http://illumos.org/msg/ZFS-8000-8A >>> scan: resilvered 180G in 0 days 16:00:55 with 2 errors on Sun Mar 22 >>> 05:18:46 2020 >>> config: >>> >>> NAME STATE READ WRITE CKSUM >>> mypool DEGRADED 0 0 2 >>> raidz1-0 DEGRADED 0 0 4 >>> diskid/DISK-WD-WMC1F0521131 ONLINE 0 0 0 >>> replacing-1 DEGRADED 0 0 0 >>> 15838717335844820448 UNAVAIL 0 0 0 was /dev/diskid/DISK-WD-WCC130964640 >>> diskid/DISK-K4JG5D2B ONLINE 0 0 0 >>> ada6 ONLINE 0 0 0 >>> ada1 ONLINE 0 0 0 >>> diskid/DISK-WD-WCC130650055 ONLINE 0 0 0 >>> >>> errors: Permanent errors have been detected in the following files: >>> mypool/XXXXXXXXXXXX >>> >>> Yes, I did exacly as you wrote - removed the failed drive, installed a replacement drive, and issued a 'zpool replace' command. >>> I tried this way to: >>> I disabled running services in that pool, unmounted and mounted it again. Even I exported/imported that pool. >>> It has no readonly property. >>> Of course I have a backup. >> >> >> My guess is that resilvering is stuck because ZFS has encountered data >> corruption. This could be caused by drive(s), cable(s), and/or data port(s) >> (motherboard or expansion card). >> >> >> What was the failure mode of the bad drive? Did you test it in any other >> machines? >> >> >> Are the any items of concern in the SMART reports for the current set of >> drives? Please post anything that looks questionable. >> >> >> Unplug and plug all of your drive power and data cables. Make sure they >> seat well. If unsure about a data cable, replace it with a new, locking >> cable. I have experienced too many problems with red SATA cables. Few, if >> any, are marked with their rated speed (I did mark some StarTech SATA III >> cables). So, I stocked up on various lengths and configurations of Cable >> Matters SATA III cables. They are black, marked "6G", and have locking >> connectors. Now, whenever I am in a system case, I replace most every red >> SATA cable just to be safe. >> >> >> I appears that you have Western Digital hard drives. Download Data >> Lifeguard Diagnostic (DLG) for DOS, burn it to a USB flash drive, boot it, >> and test all of your drives. Please post the results: >> >> https://support.wdc.com/downloads.aspx?p=2 > > If you permit an advice, ALWAYS (when it's possible) buy and use disks from > different brand (mix seagate, WD, etc..) in order to avoid same series and same > MTBF. > > I know this to late in this case, but keep this in mind. > > I know this will not help in this case, please excuse my intervention if it's > inappropriate. >