From nobody Wed May 11 19:24:39 2022 X-Original-To: questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 10F601AD512B for ; Wed, 11 May 2022 19:24:51 +0000 (UTC) (envelope-from dpchrist@holgerdanske.com) Received: from holgerdanske.com (holgerdanske.com [184.105.128.27]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "holgerdanske.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Kz4c169dgz4jTt for ; Wed, 11 May 2022 19:24:49 +0000 (UTC) (envelope-from dpchrist@holgerdanske.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=holgerdanske.com; s=nov-20210719-112354; t=1652297080; bh=0NpqOX6FACF+LQ0MTrcRBVo8Rxd2xuPTOu0RXBfBozg=; h=Received:Message-ID:Date:MIME-Version:User-Agent:Subject: Content-Language:To:References:From:In-Reply-To:Content-Type: Content-Transfer-Encoding; b=LWmZjzzykDOwVdVtVCxi0zdq7mXZhfzgeJ9cfFYi4sRG6AJuoRndzUc4LWo83efQx l8ehySC4mOxCVHvcgDHrDX35NpGXqWIGUL3n7ELgoawb68IQVEn+8zqNJKVy/6i+/w WBEdQH6h6/DB8oSm75wKn2NkwovrihB4wQBHB1JN1RUWSeZ3igThPuAJL6Z2wi16qT qcN1Epxm+f2ANBP7vbqiLyBul95nSNj1Q0LpP/UfHPVctEweCjYdqbVSVw5Zrtxs+t Yz/fx1VAsNGNZyy2wAuDgnMUVyQyWjl7+YEuJCJimN47068BXNmRYKD9tneXe6V9hw 6Ptc341nUc5X42WyGltVcFLBpL0/57KU05KwXMBtpKseAHnpuI9EJa6VZFvrdyvq/7 vwExEY1tbWCB7fLkQ1Ffh4hQw5gQc+qtUrxLfKjN+PueSO5t/GvZKq9Bi/cVpZA69j 7Gu9NDMfAu6WuiRwsbkR6J2zmx+7yDrpALhgx6b8sOEPdPYHxZYdGaGPrPUI9C6DOr 97CorQPwyGJg/FGh+Ra/S5VPzDC48B7PjMdgbwssuCezr6mSIQN9WxVAxrSV27nHnk nT5AqfXe1m4leYHU9aCP3zNlXoYyyVYSOOih+IXhNmsY4V/C4xkjjGENXx5z0K2IeA YYoYfqwguaeGlkh+r2ggKiPU= Received: from 99.100.19.101 (99-100-19-101.lightspeed.frokca.sbcglobal.net [99.100.19.101]) by holgerdanske.com with ESMTPSA (TLS_AES_128_GCM_SHA256:TLSv1.3:Kx=any:Au=any:Enc=AESGCM(128):Mac=AEAD) (SMTP-AUTH username dpchrist@holgerdanske.com, mechanism PLAIN) for ; Wed, 11 May 2022 12:24:40 -0700 Message-ID: Date: Wed, 11 May 2022 12:24:39 -0700 List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.8.0 Subject: Re: ZFS replace a mirrored disk Content-Language: en-US To: questions@freebsd.org References: From: David Christensen In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4Kz4c169dgz4jTt X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=holgerdanske.com header.s=nov-20210719-112354 header.b=LWmZjzzy; dmarc=pass (policy=none) header.from=holgerdanske.com; spf=pass (mx1.freebsd.org: domain of dpchrist@holgerdanske.com designates 184.105.128.27 as permitted sender) smtp.mailfrom=dpchrist@holgerdanske.com X-Spamd-Result: default: False [-2.15 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[holgerdanske.com:s=nov-20210719-112354]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+a]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[questions@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_SPAM_SHORT(0.85)[0.852]; DKIM_TRACE(0.00)[holgerdanske.com:+]; DMARC_POLICY_ALLOW(-0.50)[holgerdanske.com,none]; MLMMJ_DEST(0.00)[questions]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:184.104.0.0/15, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On 5/11/22 03:22, Christos Chatzaras wrote: > When a disk fails and want to replace it with a NEW disk are these commands correct? > > ---- > gpart backup nvd1 | gpart restore -F nvd0 > gmirror forget swap > gmirror insert swap /dev/nvd0p2 > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 nvd0 > zpool replace zroot nvd0 > ---- > > > I try to simulate a disk "failure". I tried the above commands with the SAME disk without success. I believe the "issue" is because the "new" disk is the SAME as before. > > I did these steps: > > 1) Boot with mfsBSD and did "gpart destroy -F nvd0" > 2) Reboot the server in the main OS. > > If the "new" disk is the SAME as before do I have to change the commands to these? > > ---- > gpart backup nvd1 | gpart restore -F nvd0 > gmirror forget swap > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 nvd0 > zpool offline zroot nvd0 > zpool online zroot nvd0 > ---- > > > Also I notice that with the SAME disk "gmirror forget swap" starts rebuilding swap immediately and "gmirror insert swap /dev/nvd0p2" is not needed. Is this the correct behaviour when the "new" disk is the SAME? If you are using ZFS, you should use ZFS commands to replace drives. I had a bad HDD in a ZFS pool about two years ago: 2020-03-01 15:55:19 toor@f3 ~ # zpool status p3 pool: p3 state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://illumos.org/msg/ZFS-8000-2Q scan: none requested config: NAME STATE READ WRITE CKSUM p3 DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 4744083090086529196 UNAVAIL 0 0 0 was /dev/gpt/p3a.eli gpt/p3b.eli ONLINE 0 0 0 errors: No known data errors I physically removed the bad HDD, installed a new HDD, created a GPT partition table, added one large partition, initialized a GELI provider, saved the GELI metadata backup, edited /etc/rc.conf with the revised GELI settings, attached the new GELI provider, and then used zpool(8) to replace the old provider with the new provider: 2020-03-01 15:59:48 toor@f3 ~ # zpool replace p3 4744083090086529196 gpt/p3c.eli 2020-03-01 16:01:12 toor@f3 ~ # zpool status p3 pool: p3 state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Sun Mar 1 16:01:08 2020 2.64G scanned at 159M/s, 0 issued at 0/s, 1.71T total 0 resilvered, 0.00% done, no estimated completion time config: NAME STATE READ WRITE CKSUM p3 DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 replacing-0 UNAVAIL 0 0 0 4744083090086529196 UNAVAIL 0 0 0 was /dev/gpt/p3a.eli gpt/p3c.eli ONLINE 0 0 0 gpt/p3b.eli ONLINE 0 0 0 errors: No known data errors A few weeks later, I scrubbed the pool: 2020-03-26 23:09:16 toor@f3 ~ # zpool scrub p3 This was the pool status when the scrub finished: 2020-03-27 12:52:27 toor@f3 ~ # zpool status p3 pool: p3 state: ONLINE scan: scrub repaired 0 in 0 days 04:42:14 with 0 errors on Fri Mar 27 03:51:36 2020 config: NAME STATE READ WRITE CKSUM p3 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/p3c.eli ONLINE 0 0 0 gpt/p3b.eli ONLINE 0 0 0 errors: No known data errors David