From nobody Thu Sep 5 20:13:30 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X09Xq1NQ9z5Tq0r for ; Thu, 05 Sep 2024 20:13:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X09Xp3BQ2z4hkg for ; Thu, 5 Sep 2024 20:13:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1725567210; a=rsa-sha256; cv=none; b=KN19yfjmDLqGLeVoc2FgG/nKWSGpMViZvclX+e/0D1hSZwpEyeb1Xot4WEXadPQhKO2nHK IJ2XCjfvyrZKgzD/uzaT2V6YTSpWK8CHlLgClUODhQLWXFOU/Ym/a9LBCF+HBbXA4IHTQ/ LJKvYodbpKfNlvvEpAtjxAuk6x+7x5tsG1VMvaXbq0Tavs4+zmb28SeiICP3BhCmYx3NMt GQ0UJyMfJ7TFQzSPohDnvfOblmxnST7hep/Q55fpBHxBbeX8ClwmlvhONgO3pUXGPBm03O Bc23cSJqlhhVtLoQHjv/ayhyHunmUcUBD7iSvdv3TmxhWPdKNrJVBExWr7jDtA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1725567210; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dBRnngJcBl8KBc5B4ZbR9xgX2frwQ87EXUP4pkFZ+bU=; b=hOswFpAVBLDP78G2XBHUNsccV/cK8UlCM48Z9qv7y+ie3pwFJXoAjCP0GMys9q2WmMpD3a yZYzbkZEuObBfSNHUmGPxnGiHESPOAdWVe4JdKkhSX00KKghCvP+lIivR5ZQEaVgApZp11 jAWriJGA1/CzDl12GUwP6pHvaerPutj3nt74ZjtBtxSx1vZN2SYhFM/5APrhVQHr2d/Sjn AD6D2AVH5Z3fqinYx4yqd0/G766QIrwWqD6j6VdatLmWQe8R404EOoovfpBMEDIkBYsJLf +Q8Rsf/futcN122ZWg9yjPhrcVdZdI/zFRwuJkKOxy/TCy2YDYakNpwMlvE+Kg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4X09Xp2VfSzMyq for ; Thu, 5 Sep 2024 20:13:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 485KDUgH005134 for ; Thu, 5 Sep 2024 20:13:30 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 485KDUXw005127 for fs@FreeBSD.org; Thu, 5 Sep 2024 20:13:30 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 281291] zfs: panic: MSan: Uninitialized malloc memory from zfs_acl_chmod Date: Thu, 05 Sep 2024 20:13:30 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to keywords Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D281291 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |fs@FreeBSD.org Keywords| |crash --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Sep 5 21:41:55 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0CVr6fHKz5V2fp for ; Thu, 05 Sep 2024 21:41:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0CVr5dttz43WL for ; Thu, 5 Sep 2024 21:41:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1725572516; a=rsa-sha256; cv=none; b=UcKrGcdTePbH2SU89uBZmvcpdztc1PNI7BSvCl8hODmCvKlto6sEpa0HDIRLlDYoLc+Ggz fvvNqelwvD4YvrlrWrQRgheWdTE1EEbDMH//tzfzSVIWCgbIFtDDCvgC2hPg3OHMakcQi2 S3d+3APe/4bOUwo3D0mLUe5NKYnFtOj+NR8oXXZ2nn1RwrUv7XkGnSdIUl3Wp+6nzhmkXR N0u/3II/cCvi2FFwkrp/YkeuX8g+R+JiupGdhGO3VWA8eFu/udBKvplUgn8MdNEFm0pS6N 1zlDo4l15Xp6MPinbHLb/3hVyvy7HytF229MKGSFm0zW9D/4haiT9tRPWjnkhw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1725572516; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7TVk4mEtAvO4CMSX6t/YW+ewAgoMOpm8qdNPATIaNRs=; b=jBCWduqU0Lt8GwZN6hj2ZxV3XTfyTXNElEz/OOz3OrKsX5rUyIBzRSlyU72QaXzq6r9gvA NovZAqoDLvZVEcct+hqVRxrtHMkiIaWMJVKBM71MEPUT+BCBciam5pUk6Lkvb+jMMgXVWp GE2P8pBFPpZtF1Vegi/9RZviG7fI1NVOLyTql5V1WHyEjUES9NUIZDRCwtsrHlmhv2wmVw VipEtSbrMY+1vb2G+bvTBbFFL2bKc//44MwJS+RNyamftecqSjbkjR6klwljCZJXeanMuP JBPC9GwQ9DOiIZ7oWJLlDGZB7lVo+n0jGqoxRU9MmM0nH16Wd9//K/frErj4IA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4X0CVr5FNfzPX9 for ; Thu, 5 Sep 2024 21:41:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 485LfuBe091979 for ; Thu, 5 Sep 2024 21:41:56 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 485Lfuul091966 for fs@FreeBSD.org; Thu, 5 Sep 2024 21:41:56 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 281291] zfs: panic: MSan: Uninitialized malloc memory from zfs_acl_chmod Date: Thu, 05 Sep 2024 21:41:55 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: asomers@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D281291 --- Comment #1 from Alan Somers --- Code review in progress upstream. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Sep 5 21:42:07 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0CW41RRGz5V2hJ for ; Thu, 05 Sep 2024 21:42:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0CW36c2nz43vg for ; Thu, 5 Sep 2024 21:42:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1725572527; a=rsa-sha256; cv=none; b=JbjN62dC9V9vaYaEFpZ5Q3Q9mgIaiPWDjjjKjA+jcubkWlb8p0oeM/PpOOw8CXDCLkqnwF js+nkGgiNXxeD3+LYwqFqQzC51n14UsS93leXxaXQUmE0ihsW7p5BLbZBqE2MZX6XydAWA pQ/zl65gf+2XqV4lc/RlaKuwGupWAOP1fgWRNmRCL5YNzVUiLS8fh5dPpMqhqYVObULaeM e3Fc5wY9BP4SyJQlmEPPzJ7O2s/krYHszNJ42ejbbOr17oM7a7q1BxQ7WgJFZMFYBAgnw8 CgpB9d5MKKyqceHUAGxbM8WgwJJShFEKyAA6kivobvUPcG3IrHkpUZV7bMrGCA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1725572527; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rZmuLBcSmdGgXDyaX6w+TeqmoXklUFHzUcd2JPpk0x8=; b=UrnfUMyRHfHT5yF1yulagi5hYrcgS0QfY9rN88fbtNlQCWmXO7Dcow7If1XJ02CjJvGLBC VfW387Uh6aqYoMf2wLaw6x0XYqFxOsL9mMFnrdE48pX1dY/Z3bWLn6rq8vMLB3UDA5Rp7H rzh2ls4ggwdixDZgKfCR4yAls4/MwBuAdxq1RL/KwgQQeOz6UqbWfsMx88I+z6MHvPTLfy 02BI8C2XDUDEQZd/9NCCdymVT1XR4M6omw6WcsDfDq3o1pfv3fMqDWI9TXApASBKkOGnED ePajME213lk29dVrEZIIyWeHGOGq6cRGGvPhwz2K5L/ol0B067I1MmFTW2uKdQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4X0CW36Cj6zPZB for ; Thu, 5 Sep 2024 21:42:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 485Lg7eC094107 for ; Thu, 5 Sep 2024 21:42:07 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 485Lg7Pp094097 for fs@FreeBSD.org; Thu, 5 Sep 2024 21:42:07 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 281291] zfs: panic: MSan: Uninitialized malloc memory from zfs_acl_chmod Date: Thu, 05 Sep 2024 21:42:07 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: asomers@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_file_loc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D281291 Alan Somers changed: What |Removed |Added ---------------------------------------------------------------------------- URL| |https://github.com/openzfs/ | |zfs/pull/16511 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Sep 5 21:42:16 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0CWF4Nyzz5V30f for ; Thu, 05 Sep 2024 21:42:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0CWF3HCMz440N for ; Thu, 5 Sep 2024 21:42:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1725572537; a=rsa-sha256; cv=none; b=iOm+X8dUKPuHOi1fLaaIBSDS06cnbsqrHLNbac4+TDr2NaTJink96gm1Uy2gmpJ8f0ZUoz wiSqCvHF+eXERirc0nhs2qdcp7oj+UPRrT8uRPukToCeGs89P37PVgmwjI2yIZMGpEWKK9 Yc2vhRhfnh/rpdEj/yvKWRtDAnpK7euA55q08I7XLqG+W8adacewIQn27KoRCtUmtD7ohK 4IIy6c08Gj4YlkRq+b6Hu95Zzss3mAdCrPEwqPp+sc5K5s/E/IeXtgA235O5Rx7nPG6P1z xOCH6ugQALdSgOv9CWapMnIHW3y1WlplnTOcFJHVK32VeoamDGWSvIjUIZ83Zg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1725572537; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZKZVl+VPI1FY2tF2PXRzVeT3JBV9XES/QwbsyNE9G6k=; b=uVuz5j6trrr5Ac8PhvN+M2w8iLNw9zp04uHoiRCxw9QspmsIUNG/M72eatlM8qgeiUrv9L s8RFnt5V+pz/GxbxRfkgu1z4TcSa6JgS8x8E6dJLV+0PRTE1bLiQXPTSAnPlZmYETuBOJO iNM3F70RFq1qVDD2i9xWae3/MUdJ0fNMg84fnMjwzeeW39hA/Zk9liBWjEundqm+TvSWlj 8qYipbooxR9wtr0WVPcNMFTjHqmcT4AHtcEj+AK1yJ44Y8vdDYDdFgv+0emyLpHXrAT+J/ DACvwwQziBpq6t3iJIuNMJcryCUoaCV+cgrbfgU1Idkxzf0sIR5MT3KJblhvsA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4X0CWF2ngpzQGx for ; Thu, 5 Sep 2024 21:42:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 485LgHKw094541 for ; Thu, 5 Sep 2024 21:42:17 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 485LgHxP094539 for fs@FreeBSD.org; Thu, 5 Sep 2024 21:42:17 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 281291] zfs: panic: MSan: Uninitialized malloc memory from zfs_acl_chmod Date: Thu, 05 Sep 2024 21:42:16 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: asomers@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: asomers@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D281291 Alan Somers changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|fs@FreeBSD.org |asomers@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Sep 6 14:24:10 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0dlg66bgz5VB0D for ; Fri, 06 Sep 2024 14:24:31 +0000 (UTC) (envelope-from cross+freebsd@relay.distal.com) Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0dlf4v2Zz4SbN for ; Fri, 6 Sep 2024 14:24:30 +0000 (UTC) (envelope-from cross+freebsd@relay.distal.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=relay.distal.com header.s=mail header.b=kcl31+z2; dmarc=none; spf=pass (mx1.freebsd.org: domain of cross+freebsd@relay.distal.com designates 168.235.95.80 as permitted sender) smtp.mailfrom=cross+freebsd@relay.distal.com dkim-signature: v=1; a=rsa-sha256; d=relay.distal.com; s=mail; c=relaxed/relaxed; q=dns/txt; h=From:Subject:Date:Message-ID:To:CC:MIME-Version:Content-Type:Content-Transfer-Encoding; bh=6Ajf7bf+sKqz2h4nzIEQibpUSL8PArGNsXdYgjPfa8w=; b=kcl31+z2d1H+vclyQ+zCikQeI2C4O8Vgrz/OYq1TW2K6kfPPaHLT5o1jR9e0iEchJYR5RbgnoVBZSaAd004+N2tM5/KchN7/5tmi0/V2VbUe3d/95yYYkBVbzeN4hm/M7tZ+PF9Dhm8f+xHcy73PL6DRXRKzPVZpZBx6OGc5D4hyYD15z5dT95oJEoCL6W+qT6vSomovSaPdqNd2DpQSxZODfECiEJtfqFgCn/tS1A/ULNOerxiiEinoF4 VxqAbN7hsOCozq9UUDYH+r1OTn5J129eG1cEfVT+p3m9hqXWlWBSHt1cuyGYtpB2mEMLW3qS8JC72CmJoi37PjkTJfJQ== Received: from mail.distal.com (pool-108-51-233-124.washdc.fios.verizon.net [108.51.233.124]) by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256) ; Fri, 6 Sep 2024 14:24:22 +0000 Received: from smtpclient.apple ( [2001:420:c0c4:1001::9f]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id 52c632bb (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Fri, 6 Sep 2024 10:24:21 -0400 (EDT) From: Chris Ross Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3776.700.51\)) Subject: Unable to replace drive in raidz1 Message-Id: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> Date: Fri, 6 Sep 2024 10:24:10 -0400 Cc: Chris Ross To: FreeBSD Filesystems X-Mailer: Apple Mail (2.3776.700.51) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.20 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; FORGED_SENDER(0.30)[cross@distal.com,cross@relay.distal.com]; R_SPF_ALLOW(-0.20)[+a:relay.dynu.com]; R_DKIM_ALLOW(-0.20)[relay.distal.com:s=mail]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[distal.com]; RCPT_COUNT_TWO(0.00)[2]; DKIM_TRACE(0.00)[relay.distal.com:+]; RCVD_TLS_ALL(0.00)[]; TO_DN_ALL(0.00)[]; TAGGED_FROM(0.00)[freebsd]; RCVD_COUNT_TWO(0.00)[2]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[cross@distal.com,cross@relay.distal.com]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:3842, ipnet:168.235.92.0/22, country:US]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; APPLE_MAILER_COMMON(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_RCPT(0.00)[freebsd]; MIME_TRACE(0.00)[0:+] X-Rspamd-Queue-Id: 4X0dlf4v2Zz4SbN Hello. I have searched the interwebs a bit and seen tell of this and = others like it, but I haven=E2=80=99t found a solution. I have a pool with three 3-disk raidz1 vols. I want to replace the = disks in the first vdev with larger disks. I=E2=80=99ve done this before, but = may=E2=80=99ve done something wrong here. I belive I used =E2=80=9Czpool remove tank da3=E2=80=9D, but command = history doesn=E2=80=99t have that. I=E2=80=99ve used many commands since I started. I = might=E2=80=99ve =E2=80=9Czpool offline=E2=80=9Dd the device. I=E2=80=99m sorry I = don=E2=80=99t remember the original command. Then replaced the disk and rebooted. This of course renumbered the disks. :-(. But, finding the new/replacement disk (da10), I try to =E2=80=9Czpool replace tank da3 da10=E2=80=9D. This always produces: cannot replace da3 with da10: already in replacing/spare config; wait = for completion or use 'zpool detach=E2=80=99 Now, I can=E2=80=99t use =E2=80=9Czpool detach=E2=80=9D I=E2=80=99ve = learned because that doesn=E2=80=99t work on zraid. And I can=E2=80=99t tell what it _thinks_ is happening. I = even Did a scrub of the pool and let that finish, but am still seeing the same. I have now: =E2=80=948<=E2=80=948<=E2=80=948<--- >>>>>> zpool status -v tank pool: tank state: DEGRADED status: One or more devices are faulted in response to persistent = errors. Sufficient replicas exist for the pool to continue functioning = in a degraded state. action: Replace the faulted device, or use 'zpool clear' to mark the = device repaired. scan: scrub repaired 0B in 17:14:03 with 0 errors on Fri Sep 6 = 09:08:34 2024 config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 raidz1-0 DEGRADED 0 0 0 da3 FAULTED 0 0 0 external = device fault da1 ONLINE 0 0 0 da2 ONLINE 0 0 0 raidz1-1 ONLINE 0 0 0 diskid/DISK-K1GMBN9D ONLINE 0 0 0 diskid/DISK-K1GMEDMD ONLINE 0 0 0 diskid/DISK-K1GMAX1D ONLINE 0 0 0 raidz1-2 ONLINE 0 0 0 diskid/DISK-3WJDHJ2J ONLINE 0 0 0 diskid/DISK-3WK3G1KJ ONLINE 0 0 0 diskid/DISK-3WJ7ZMMJ ONLINE 0 0 0 errors: No known data errors =E2=80=948<=E2=80=948<=E2=80=948<=E2=80=94 I=E2=80=99ll note that before the switcharoo, the second and third = vdev=E2=80=99s listed =E2=80=9Cda4 da5 da6=E2=80=9D and =E2=80=9Cda7 da8 da9=E2=80=9D. The = moving around of names caused the above, which again I=E2=80=99ve seen before, and am fine with. (Oh, you can see I ran a =E2=80=9Czpool offline -f=E2=80=9D on it most = recently. But that was in the list of things I=E2=80=99ve tried that haven=E2=80=99t = helped.) Please let me know if someone knows how I=E2=80=99ve gotten to this = state, and what I need to do to correct it. What is the "in replacing/spare = config=E2=80=9D meaning? - Chris From nobody Fri Sep 6 14:30:31 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0dtt2JqMz5VBpb for ; Fri, 06 Sep 2024 14:30:46 +0000 (UTC) (envelope-from cross+freebsd@relay.distal.com) Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0dts3l8Tz4WYB for ; Fri, 6 Sep 2024 14:30:45 +0000 (UTC) (envelope-from cross+freebsd@relay.distal.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=relay.distal.com header.s=mail header.b=YAPkIJCX; dmarc=none; spf=pass (mx1.freebsd.org: domain of cross+freebsd@relay.distal.com designates 168.235.95.80 as permitted sender) smtp.mailfrom=cross+freebsd@relay.distal.com dkim-signature: v=1; a=rsa-sha256; d=relay.distal.com; s=mail; c=relaxed/relaxed; q=dns/txt; h=From:Subject:Date:Message-ID:To:MIME-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References; bh=lpoCb6LtSfaD5nMc47AkIQCI+71toQspk6kbP1Elwu4=; b=YAPkIJCX2vG24IHo0WYie+CC1dyiVsruDhIH7p3gY4DjuPRtHyEwBui/0UekYksPcwexYxjFLIfD2K5yKQWQoluLZd6+ycmJDGK/tkDmu0C4sKXeP7mIoxxVPj+4jrRLgotEJuN5nlrAThZIk8c7JLT28GiOdoCibs2W6322cDXjSEeKI6d9liHPHVnFOADmT32zM5MsLJwk8twVN1eBTBUK7AxMa5eRgIrrqa5ysS1/4+1gVyUHgYBAPq t8ccB9rXuSxB7QrypKd9UXzo1YIaHZlqH43jQlHIuu5K6Ni8+SNOwhAMcfdevnqzEBR2VH3ynQS1mVqiPRMRw29GvmCQ== Received: from mail.distal.com (pool-108-51-233-124.washdc.fios.verizon.net [108.51.233.124]) by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256) ; Fri, 6 Sep 2024 14:30:43 +0000 Received: from smtpclient.apple ( [2001:420:c0c4:1001::9f]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id 984b57cb (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO) for ; Fri, 6 Sep 2024 10:30:42 -0400 (EDT) From: Chris Ross Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3776.700.51\)) Subject: Re: Unable to replace drive in raidz1 Date: Fri, 6 Sep 2024 10:30:31 -0400 References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> To: FreeBSD Filesystems In-Reply-To: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> Message-Id: X-Mailer: Apple Mail (2.3776.700.51) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.19 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.991]; FORGED_SENDER(0.30)[cross@distal.com,cross@relay.distal.com]; R_SPF_ALLOW(-0.20)[+a:relay.dynu.com]; R_DKIM_ALLOW(-0.20)[relay.distal.com:s=mail]; MIME_GOOD(-0.10)[text/plain]; TAGGED_FROM(0.00)[freebsd]; RCVD_TLS_ALL(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[distal.com]; TO_DN_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[relay.distal.com:+]; ASN(0.00)[asn:3842, ipnet:168.235.92.0/22, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_NEQ_ENVFROM(0.00)[cross@distal.com,cross@relay.distal.com]; MIME_TRACE(0.00)[0:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; APPLE_MAILER_COMMON(0.00)[]; FROM_HAS_DN(0.00)[] X-Rspamd-Queue-Id: 4X0dts3l8Tz4WYB Oh, sorry. Failed to indicate versions. I was running 13.2 on amd64, = and while facing this problem have updated to 14.1. The current state is 14.1, and I=E2=80=99m still seeing the same, but I started in 13.2 and = saw this. > On Sep 6, 2024, at 10:24, Chris Ross wrote: >=20 > Hello. I have searched the interwebs a bit and seen tell of this and = others > like it, but I haven=E2=80=99t found a solution. >=20 > I have a pool with three 3-disk raidz1 vols. I want to replace the = disks > in the first vdev with larger disks. I=E2=80=99ve done this before, = but may=E2=80=99ve > done something wrong here. >=20 > I belive I used =E2=80=9Czpool remove tank da3=E2=80=9D, but command = history doesn=E2=80=99t > have that. I=E2=80=99ve used many commands since I started. I = might=E2=80=99ve > =E2=80=9Czpool offline=E2=80=9Dd the device. I=E2=80=99m sorry I = don=E2=80=99t remember the original > command. >=20 > Then replaced the disk and rebooted. This of course renumbered the > disks. :-(. But, finding the new/replacement disk (da10), I try to > =E2=80=9Czpool replace tank da3 da10=E2=80=9D. > This always produces: >=20 > cannot replace da3 with da10: already in replacing/spare config; wait = for completion or use 'zpool detach=E2=80=99 >=20 > Now, I can=E2=80=99t use =E2=80=9Czpool detach=E2=80=9D I=E2=80=99ve = learned because that doesn=E2=80=99t work > on zraid. And I can=E2=80=99t tell what it _thinks_ is happening. I = even > Did a scrub of the pool and let that finish, but am still seeing the > same. >=20 > I have now: >=20 > =E2=80=948<=E2=80=948<=E2=80=948<--- >>>>>>> zpool status -v tank > pool: tank > state: DEGRADED > status: One or more devices are faulted in response to persistent = errors. > Sufficient replicas exist for the pool to continue functioning = in a > degraded state. > action: Replace the faulted device, or use 'zpool clear' to mark the = device > repaired. > scan: scrub repaired 0B in 17:14:03 with 0 errors on Fri Sep 6 = 09:08:34 2024 > config: >=20 > NAME STATE READ WRITE CKSUM > tank DEGRADED 0 0 0 > raidz1-0 DEGRADED 0 0 0 > da3 FAULTED 0 0 0 external = device fault > da1 ONLINE 0 0 0 > da2 ONLINE 0 0 0 > raidz1-1 ONLINE 0 0 0 > diskid/DISK-K1GMBN9D ONLINE 0 0 0 > diskid/DISK-K1GMEDMD ONLINE 0 0 0 > diskid/DISK-K1GMAX1D ONLINE 0 0 0 > raidz1-2 ONLINE 0 0 0 > diskid/DISK-3WJDHJ2J ONLINE 0 0 0 > diskid/DISK-3WK3G1KJ ONLINE 0 0 0 > diskid/DISK-3WJ7ZMMJ ONLINE 0 0 0 >=20 > errors: No known data errors > =E2=80=948<=E2=80=948<=E2=80=948<=E2=80=94 >=20 > I=E2=80=99ll note that before the switcharoo, the second and third = vdev=E2=80=99s listed > =E2=80=9Cda4 da5 da6=E2=80=9D and =E2=80=9Cda7 da8 da9=E2=80=9D. The = moving around of names caused the > above, which again I=E2=80=99ve seen before, and am fine with. >=20 > (Oh, you can see I ran a =E2=80=9Czpool offline -f=E2=80=9D on it most = recently. But > that was in the list of things I=E2=80=99ve tried that haven=E2=80=99t = helped.) >=20 > Please let me know if someone knows how I=E2=80=99ve gotten to this = state, and > what I need to do to correct it. What is the "in replacing/spare = config=E2=80=9D > meaning? >=20 > - Chris >=20 From nobody Fri Sep 6 15:32:35 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0gGS6hM0z5VKrb for ; Fri, 06 Sep 2024 15:32:48 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-ot1-f46.google.com (mail-ot1-f46.google.com [209.85.210.46]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0gGS1tX2z4lBH for ; Fri, 6 Sep 2024 15:32:48 +0000 (UTC) (envelope-from asomers@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.210.46 as permitted sender) smtp.mailfrom=asomers@gmail.com Received: by mail-ot1-f46.google.com with SMTP id 46e09a7af769-70f79f75da6so1375389a34.0 for ; Fri, 06 Sep 2024 08:32:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725636767; x=1726241567; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EZUYAKKbBG5qhmeCFbVqNb4w91uYZLEo0RLpXUTTd8o=; b=fR+o3BwetXasvL/1xB3bjb99eVdETEjIdFk6Nz3GtcCgNKcbkorMeV5qiM/V1qj3sm 7GhSXPzX6bAaT1BNuRGKmk++rd0hoqKqcF83YOXcIEGbYt2EeRu7KQ92VtmPuO124PKQ dUxNu1rfsGeKEVMPCAsSyaOrG6drvoIkU4FZL4aF37baa5mbsklpssYqOtMuvhO5sv3j CgMyaLvbnK6r/Dgn6nAVbPzNRKBJ+xrH4ln41A5be3EXAmPZirM8osQVK1+CqvFZ1/lC 16oYJNsWbQ9wYulm1ZY8XSti28D5HEH2+1Q/JebKmN4jZEspBF2K5uziqBi7aDlqZAR6 qF0Q== X-Gm-Message-State: AOJu0YwAWxSn2rMvxEyj9dZEkuxdSPpPHso+n/g0vWzNy2DLsqtGb16V swgLoCHWTF4Lp0KCX0hD3amk9b8FqlsLYgwnsdGcA68y1zz62Wgiy3ybPSkEzwVzE2mbqFpLsAY Z2tDWn0NPWXbXZUgEAW2/CUMFoc6gIw== X-Google-Smtp-Source: AGHT+IHAsiG19ocRXM67BUc9nCwL4yW7Z0hYHTZ7d8opB4QNwQO2xsw9WU7JqcIVEvyTdHW9jtyRCZBk0Geh+jRBLzU= X-Received: by 2002:a05:6808:1450:b0:3db:1661:4d64 with SMTP id 5614622812f47-3e029cfdf0bmr3697143b6e.13.1725636767083; Fri, 06 Sep 2024 08:32:47 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> In-Reply-To: From: Alan Somers Date: Fri, 6 Sep 2024 09:32:35 -0600 Message-ID: Subject: Re: Unable to replace drive in raidz1 To: Chris Ross Cc: FreeBSD Filesystems Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.46 / 15.00]; NEURAL_HAM_LONG(-0.99)[-0.991]; NEURAL_HAM_SHORT(-0.86)[-0.860]; NEURAL_HAM_MEDIUM(-0.61)[-0.612]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; RWL_MAILSPIKE_GOOD(-0.10)[209.85.210.46:from]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), No valid DKIM,none]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; MISSING_XM_UA(0.00)[]; FREEFALL_USER(0.00)[asomers]; FREEMAIL_ENVFROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; RCVD_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; TAGGED_RCPT(0.00)[freebsd]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; RCVD_IN_DNSWL_NONE(0.00)[209.85.210.46:from] X-Rspamd-Queue-Id: 4X0gGS1tX2z4lBH On Fri, Sep 6, 2024 at 8:30=E2=80=AFAM Chris Ross wrote: > > Oh, sorry. Failed to indicate versions. I was running 13.2 on amd64, an= d > while facing this problem have updated to 14.1. The current state is > 14.1, and I=E2=80=99m still seeing the same, but I started in 13.2 and sa= w this. > > > On Sep 6, 2024, at 10:24, Chris Ross wrote: > > > > Hello. I have searched the interwebs a bit and seen tell of this and o= thers > > like it, but I haven=E2=80=99t found a solution. > > > > I have a pool with three 3-disk raidz1 vols. I want to replace the dis= ks > > in the first vdev with larger disks. I=E2=80=99ve done this before, bu= t may=E2=80=99ve > > done something wrong here. > > > > I belive I used =E2=80=9Czpool remove tank da3=E2=80=9D, but command hi= story doesn=E2=80=99t > > have that. I=E2=80=99ve used many commands since I started. I might= =E2=80=99ve > > =E2=80=9Czpool offline=E2=80=9Dd the device. I=E2=80=99m sorry I don= =E2=80=99t remember the original > > command. > > > > Then replaced the disk and rebooted. This of course renumbered the > > disks. :-(. But, finding the new/replacement disk (da10), I try to > > =E2=80=9Czpool replace tank da3 da10=E2=80=9D. > > This always produces: > > > > cannot replace da3 with da10: already in replacing/spare config; wait f= or completion or use 'zpool detach=E2=80=99 > > > > Now, I can=E2=80=99t use =E2=80=9Czpool detach=E2=80=9D I=E2=80=99ve le= arned because that doesn=E2=80=99t work > > on zraid. And I can=E2=80=99t tell what it _thinks_ is happening. I e= ven > > Did a scrub of the pool and let that finish, but am still seeing the > > same. > > > > I have now: > > > > =E2=80=948<=E2=80=948<=E2=80=948<--- > >>>>>>> zpool status -v tank > > pool: tank > > state: DEGRADED > > status: One or more devices are faulted in response to persistent error= s. > > Sufficient replicas exist for the pool to continue functioning in= a > > degraded state. > > action: Replace the faulted device, or use 'zpool clear' to mark the de= vice > > repaired. > > scan: scrub repaired 0B in 17:14:03 with 0 errors on Fri Sep 6 09:08:= 34 2024 > > config: > > > > NAME STATE READ WRITE CKSUM > > tank DEGRADED 0 0 0 > > raidz1-0 DEGRADED 0 0 0 > > da3 FAULTED 0 0 0 external de= vice fault > > da1 ONLINE 0 0 0 > > da2 ONLINE 0 0 0 > > raidz1-1 ONLINE 0 0 0 > > diskid/DISK-K1GMBN9D ONLINE 0 0 0 > > diskid/DISK-K1GMEDMD ONLINE 0 0 0 > > diskid/DISK-K1GMAX1D ONLINE 0 0 0 > > raidz1-2 ONLINE 0 0 0 > > diskid/DISK-3WJDHJ2J ONLINE 0 0 0 > > diskid/DISK-3WK3G1KJ ONLINE 0 0 0 > > diskid/DISK-3WJ7ZMMJ ONLINE 0 0 0 > > > > errors: No known data errors > > =E2=80=948<=E2=80=948<=E2=80=948<=E2=80=94 > > > > I=E2=80=99ll note that before the switcharoo, the second and third vdev= =E2=80=99s listed > > =E2=80=9Cda4 da5 da6=E2=80=9D and =E2=80=9Cda7 da8 da9=E2=80=9D. The m= oving around of names caused the > > above, which again I=E2=80=99ve seen before, and am fine with. > > > > (Oh, you can see I ran a =E2=80=9Czpool offline -f=E2=80=9D on it most = recently. But > > that was in the list of things I=E2=80=99ve tried that haven=E2=80=99t = helped.) > > > > Please let me know if someone knows how I=E2=80=99ve gotten to this sta= te, and > > what I need to do to correct it. What is the "in replacing/spare confi= g=E2=80=9D > > meaning? > > > > - Chris "zpool replace" is indeed the correct command. There's no need to run "zpool offline" first, and "zpool remove" is wrong. Since "zpool replace" is still failing, are you sure that da10 is still the correct device name after all disks got renumbered? If you're sure, then you might run "zdb -l /dev/da10" to see what ZFS thinks is on that disk. -Alan From nobody Fri Sep 6 15:43:38 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0gW86wPVz5VMMg for ; Fri, 06 Sep 2024 15:43:48 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smarthost1.sentex.ca", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0gW82Qg0z4nbb for ; Fri, 6 Sep 2024 15:43:48 +0000 (UTC) (envelope-from mike@sentex.net) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:1::12 as permitted sender) smtp.mailfrom=mike@sentex.net Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.18.1/8.18.1) with ESMTPS id 486FhcuB037239 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL); Fri, 6 Sep 2024 11:43:38 -0400 (EDT) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:d98b:f91d:4232:8049] ([IPv6:2607:f3e0:0:4:d98b:f91d:4232:8049]) by pyroxene2a.sentex.ca (8.18.1/8.15.2) with ESMTPS id 486FhaYr048683 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Fri, 6 Sep 2024 11:43:37 -0400 (EDT) (envelope-from mike@sentex.net) Content-Type: multipart/alternative; boundary="------------AOmL5V0Y0p2bqj1SOXU1V6D0" Message-ID: <69093f39-f673-416e-aac9-59a80b936ab7@sentex.net> Date: Fri, 6 Sep 2024 11:43:38 -0400 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Unable to replace drive in raidz1 To: Chris Ross , FreeBSD Filesystems References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> Content-Language: en-US From: mike tancsa Autocrypt: addr=mike@sentex.net; keydata= xsBNBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAHNHW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+wsCOBBMBCAA4FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAl+pQfkCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQeVOEFl5W rMiN6ggAk3H5vk8QnbvGbb4sinxZt/wDetgk0AOR9NRmtTnPaW+sIJEfGBOz47Xih+f7uWJS j+uvc9Ewn2Z7n8z3ZHJlLAByLVLtcNXGoRIGJ27tevfOaNqgJHBPbFOcXCBBFTx4MYMM4iAZ cDT5vsBTSaM36JZFtHZBKkuFEItbA/N8ZQSHKdTYMIA7A3OCLGbJBqloQ8SlW4MkTzKX4u7R yefAYQ0h20x9IqC5Ju8IsYRFacVZconT16KS81IBceO42vXTN0VexbVF2rZIx3v/NT75r6Vw 0FlXVB1lXOHKydRA2NeleS4NEG2vWqy/9Boj0itMfNDlOhkrA/0DcCurMpnpbM7ATQRcsMzk AQgA1Dpo/xWS66MaOJLwA28sKNMwkEk1Yjs+okOXDOu1F+0qvgE8sVmrOOPvvWr4axtKRSG1 t2QUiZ/ZkW/x/+t0nrM39EANV1VncuQZ1ceIiwTJFqGZQ8kb0+BNkwuNVFHRgXm1qzAJweEt RdsCMohB+H7BL5LGCVG5JaU0lqFU9pFP40HxEbyzxjsZgSE8LwkI6wcu0BLv6K6cLm0EiHPO l5G8kgRi38PS7/6s3R8QDsEtbGsYy6O82k3zSLIjuDBwA9GRaeigGppTxzAHVjf5o9KKu4O7 gC2KKVHPegbXS+GK7DU0fjzX57H5bZ6komE5eY4p3oWT/CwVPSGfPs8jOwARAQABwsB2BBgB CAAgFiEEmuvCXT0aY6hs4SbWeVOEFl5WrMgFAl+pQfkCGwwACgkQeVOEFl5WrMiVqwf9GwU8 c6cylknZX8QwlsVudTC8xr/L17JA84wf03k3d4wxP7bqy5AYy7jboZMbgWXngAE/HPQU95NM aukysSnknzoIpC96XZJ0okLBXVS6Y0ylZQ+HrbIhMpuQPoDweoF5F9wKrsHRoDaUK1VR706X rwm4HUzh7Jk+auuMYfuCh0FVlFBEuiJWMLhg/5WCmcRfiuB6F59ZcUQrwLEZeNhF2XJV4KwB Tlg7HCWO/sy1foE5noaMyACjAtAQE9p5kGYaj+DuRhPdWUTsHNuqrhikzIZd2rrcMid+ktb0 NvtvswzMO059z1YGMtGSqQ4srCArju+XHIdTFdiIYbd7+jeehg== In-Reply-To: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> X-Scanned-By: MIMEDefang 2.86 X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.39 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[199.212.134.19:received]; XM_UA_NO_VERSION(0.01)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; TAGGED_RCPT(0.00)[freebsd]; ARC_NA(0.00)[]; FREEFALL_USER(0.00)[mike]; RCPT_COUNT_TWO(0.00)[2]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_DN_ALL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[sentex.net]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4X0gW82Qg0z4nbb This is a multi-part message in MIME format. --------------AOmL5V0Y0p2bqj1SOXU1V6D0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 9/6/2024 10:24 AM, Chris Ross wrote: > NAME STATE READ WRITE CKSUM > tank DEGRADED 0 0 0 > raidz1-0 DEGRADED 0 0 0 > da3 FAULTED 0 0 0 external device fault > da1 ONLINE 0 0 0 > da2 ONLINE 0 0 0 > raidz1-1 ONLINE 0 0 0 > diskid/DISK-K1GMBN9D ONLINE 0 0 0 > diskid/DISK-K1GMEDMD ONLINE 0 0 0 > diskid/DISK-K1GMAX1D ONLINE 0 0 0 > raidz1-2 ONLINE 0 0 0 > diskid/DISK-3WJDHJ2J ONLINE 0 0 0 > diskid/DISK-3WK3G1KJ ONLINE 0 0 0 > diskid/DISK-3WJ7ZMMJ ONLINE 0 0 0 I would triple check to see what the devices are that are part of the pool.  I wish there was a way to tell zfs to only display one or the other.  So list out what diskid/DISK-K1GMBN9D, diskid/DISK-K1GMEDMD... to diskid/DISK-3WJ7ZMMJ are in terms of /dev/da* actually are.  I have some controllers that will re-order the disks on every reboot.  glabel status and camcontrol devlist should help verify     ---Mike --------------AOmL5V0Y0p2bqj1SOXU1V6D0 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 9/6/2024 10:24 AM, Chris Ross wrote:
	NAME                      STATE     READ WRITE CKSUM
	tank                      DEGRADED     0     0     0
	  raidz1-0                DEGRADED     0     0     0
	    da3                   FAULTED      0     0     0  external device fault
	    da1                   ONLINE       0     0     0
	    da2                   ONLINE       0     0     0
	  raidz1-1                ONLINE       0     0     0
	    diskid/DISK-K1GMBN9D  ONLINE       0     0     0
	    diskid/DISK-K1GMEDMD  ONLINE       0     0     0
	    diskid/DISK-K1GMAX1D  ONLINE       0     0     0
	  raidz1-2                ONLINE       0     0     0
	    diskid/DISK-3WJDHJ2J  ONLINE       0     0     0
	    diskid/DISK-3WK3G1KJ  ONLINE       0     0     0
	    diskid/DISK-3WJ7ZMMJ  ONLINE       0     0     0

I would triple check to see what the devices are that are part of the pool.  I wish there was a way to tell zfs to only display one or the other.  So list out what diskid/DISK-K1GMBN9D, diskid/DISK-K1GMEDMD... to diskid/DISK-3WJ7ZMMJ are in terms of /dev/da* actually are.  I have some controllers that will re-order the disks on every reboot.  glabel status and camcontrol devlist should help verify


    ---Mike

--------------AOmL5V0Y0p2bqj1SOXU1V6D0-- From nobody Fri Sep 6 15:56:25 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0gnx5BVTz5VNqG for ; Fri, 06 Sep 2024 15:56:37 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-yb1-xb30.google.com (mail-yb1-xb30.google.com [IPv6:2607:f8b0:4864:20::b30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0gnx39jsz4r08 for ; Fri, 6 Sep 2024 15:56:37 +0000 (UTC) (envelope-from fjwcash@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-yb1-xb30.google.com with SMTP id 3f1490d57ef6-e1a9dc3efc1so2636579276.2 for ; Fri, 06 Sep 2024 08:56:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725638196; x=1726242996; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=/yM3OBXmyU8OOArhxvmtC2osgjgkl+YZ/cOkdoIPeS0=; b=Qtik9/L0DWwlePrkAivUpRRV00n8PdJl3o8UZy4+KdFdim2tD3PEEJeoBQPIt1TE2n 5EaGMxRnAWsnLm+i2Q59susBG10qibNDp0IBV7T7BwlTa/wHNgSSIr6WyqyNa7T7GPLi oShTJBD3+OQjMk++22SFV6kfA1yI2u1RKtZmbR4ffaYYODeTFMu+MgNm+hM4RlZeHyfm nPXIq86CT7fi6VzEFfp+E2a0g0ouMKaXizkeHE3/YNA1JTSaVCk3ifEzOJoArLUJRHuB CmQUidJN9SxOmDjV4jUX7rHm335nwX/1wDkrYREpgIXDfGk67fZHeSbCPpSXk/gykmb3 ubyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725638196; x=1726242996; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=/yM3OBXmyU8OOArhxvmtC2osgjgkl+YZ/cOkdoIPeS0=; b=c9tLcOm5Cnn1rbFPvw7QllWSTvQJVglPLXGohFL3HY87/D++30OW0ZeoYa33Ed+7us 9Sy9ksdngQIW3fw0CMpv/ArQXNHk01ZgbSr65/NvILULc5a+Qon2cnKCD/fXFx2VI9sa VqoZkKcbktNAyZpzf4xYjvxb9P1Jn6DLbAeHEthOgPgbjcVC53W8IYtMLtRBXpyqWoWD dj7pQrcNJgBBw8+HeMWbL7Vx4xZ27Rm1gfPD+dDwZ6WBIw79qLSA/wvv9iVZ/MKx84cp ASApIWy1HKHKKEH5WuJYfVXy1Heis5QUQm9m1Zl6Mn/rG76/EffSt4ctkhbJHNt1RETh 58Qw== X-Forwarded-Encrypted: i=1; AJvYcCVVzVAsB3BoLE/kRgulWBnGlHg5BTRZg12DqMG1q8DfBYqM9dU84kP9+SRcj2WhMXV+k9WNd7HzDb5p@freebsd.org X-Gm-Message-State: AOJu0YxkDH6Y2cyMoFpOc9ht96wWDpOWEiYG0nGs+CZTAVwPPj5EMgEm EEk96Zr802BW08K1AAfVSSZpWzkbjy5V824/nD8VxV4RPznb/EBLvcIWs9TWVIQGkVjYp5ht7xv 5j3i+1feqmrFRap4afcmLTZgjnc9NTmfo X-Google-Smtp-Source: AGHT+IEnIcSPubevpinMljBMEABEdK/DlDR6MrkT6zH+EeOBoO47lyPFlhdcB1BC81RgsmlyiTjSI6lT3Xy8Q58rHkM= X-Received: by 2002:a05:6902:1791:b0:e1a:aa90:34a with SMTP id 3f1490d57ef6-e1d3487f13dmr3574888276.19.1725638196528; Fri, 06 Sep 2024 08:56:36 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> <69093f39-f673-416e-aac9-59a80b936ab7@sentex.net> In-Reply-To: <69093f39-f673-416e-aac9-59a80b936ab7@sentex.net> From: Freddie Cash Date: Fri, 6 Sep 2024 08:56:25 -0700 Message-ID: Subject: Re: Unable to replace drive in raidz1 To: mike tancsa Cc: Chris Ross , FreeBSD Filesystems Content-Type: multipart/alternative; boundary="000000000000b70c9506217573fa" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_RCPT(0.00)[freebsd]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4X0gnx39jsz4r08 --000000000000b70c9506217573fa Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, Sep 6, 2024 at 8:44=E2=80=AFAM mike tancsa wrote: > On 9/6/2024 10:24 AM, Chris Ross wrote: > > NAME STATE READ WRITE CKSUM > tank DEGRADED 0 0 0 > raidz1-0 DEGRADED 0 0 0 > da3 FAULTED 0 0 0 external device fa= ult > da1 ONLINE 0 0 0 > da2 ONLINE 0 0 0 > raidz1-1 ONLINE 0 0 0 > diskid/DISK-K1GMBN9D ONLINE 0 0 0 > diskid/DISK-K1GMEDMD ONLINE 0 0 0 > diskid/DISK-K1GMAX1D ONLINE 0 0 0 > raidz1-2 ONLINE 0 0 0 > diskid/DISK-3WJDHJ2J ONLINE 0 0 0 > diskid/DISK-3WK3G1KJ ONLINE 0 0 0 > diskid/DISK-3WJ7ZMMJ ONLINE 0 0 0 > > > I would triple check to see what the devices are that are part of the > pool. I wish there was a way to tell zfs to only display one or the > other. So list out what diskid/DISK-K1GMBN9D, diskid/DISK-K1GMEDMD... to > diskid/DISK-3WJ7ZMMJ are in terms of /dev/da* actually are. I have some > controllers that will re-order the disks on every reboot. glabel status > and camcontrol devlist should help verify > You can't tell ZFS specifically to use one form of GEOM ID vs another, but you can tell the whole system which GEOM IDs to not use. Add the following to /boot/loader.conf: kern.geom.label.disk_ident.enable=3D"0" # Disable the auto-generated Disk IDs for disks kern.geom.label.gptid.enable=3D"0" # Disable the auto-generated GPT UUIDs f= or disks kern.geom.label.ufsid.enable=3D"0" # Disable the auto-generated UFS UUIDs f= or filesystems The first line will remove the diskid/DISK-* entries and show the device nodes (daX). The other two lines remove GPT and UFS UUIDs as well. All my ZFS systems have those entries in loader.conf, as I prefer to use GPT partition labels in my pools (gpt/label-name) where I list which specific JBOD chassis and drive bay the HD is located. That way, it doesn't matter if the device nodes are renumbered, as the labels don't change. Makes it much easier to find the specific drive to be replaced, whether in my home server with 6 drives or my backups servers at work with multiple JBODs and 92 drives. --=20 Freddie Cash fjwcash@gmail.com --000000000000b70c9506217573fa Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Fri, Sep 6, 2024 at 8:44=E2=80=AFAM mi= ke tancsa <mike@sentex.net> wr= ote:
=20 =20 =20
On 9/6/2024 10:24 AM, Chris Ross wrote:
	NAME                      STATE     READ WRITE CKSUM
	tank                      DEGRADED     0     0     0
	  raidz1-0                DEGRADED     0     0     0
	    da3                   FAULTED      0     0     0  external device faul=
t
	    da1                   ONLINE       0     0     0
	    da2                   ONLINE       0     0     0
	  raidz1-1                ONLINE       0     0     0
	    diskid/DISK-K1GMBN9D  ONLINE       0     0     0
	    diskid/DISK-K1GMEDMD  ONLINE       0     0     0
	    diskid/DISK-K1GMAX1D  ONLINE       0     0     0
	  raidz1-2                ONLINE       0     0     0
	    diskid/DISK-3WJDHJ2J  ONLINE       0     0     0
	    diskid/DISK-3WK3G1KJ  ONLINE       0     0     0
	    diskid/DISK-3WJ7ZMMJ  ONLINE       0     0     0

I would triple check to see what the devices are that are part of the pool.=C2=A0 I wish there was a way to tell zfs to only display on= e or the other.=C2=A0 So list out what diskid/DISK-K1GMBN9D, diskid/DISK-K1GMEDMD... to diskid/DISK-3WJ7ZMMJ are in terms of /dev/da* actually are.=C2=A0 I have some controllers that will re-ord= er the disks on every reboot.=C2=A0 glabel status and camcontrol devlist should help verify

You can't tell ZFS = specifically to use one form of GEOM ID vs another, but you can tell the wh= ole system which GEOM IDs to not use.=C2=A0 Add the following to /boot/load= er.conf:

kern.geom.label.disk_ident.enable=3D"0"= ; =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # Disable the auto-generated Disk IDs = =C2=A0for disks
kern.geom.label.gptid.enable=3D"0" # Disable = the auto-generated GPT UUIDs for disks
kern.geom.label.ufsid.enable=3D&q= uot;0" # Disable the auto-generated UFS UUIDs for filesystems

The first line will remove the diskid/DISK-* entries and = show the device nodes (daX).=C2=A0 The other two lines remove GPT and UFS U= UIDs as well.

All my ZFS systems have those entrie= s in loader.conf, as I prefer to use GPT partition labels in my pools (gpt/= label-name) where I list which specific JBOD chassis and drive bay the HD i= s located.=C2=A0 That way, it doesn't matter if the device nodes are re= numbered, as the labels don't change.=C2=A0 Makes it much easier to fin= d the specific drive to be replaced, whether in my home server with 6 drive= s or my backups servers at work with multiple JBODs and 92 drives.

--
Freddie Cash
fjwcash@gmail.com
--000000000000b70c9506217573fa-- From nobody Fri Sep 6 16:50:55 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0j0w2NMpz5VVKL for ; Fri, 06 Sep 2024 16:51:12 +0000 (UTC) (envelope-from cross+freebsd@relay.distal.com) Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0j0v60Vgz430d; Fri, 6 Sep 2024 16:51:11 +0000 (UTC) (envelope-from cross+freebsd@relay.distal.com) Authentication-Results: mx1.freebsd.org; none dkim-signature: v=1; a=rsa-sha256; d=relay.distal.com; s=mail; c=relaxed/relaxed; q=dns/txt; h=From:Subject:Date:Message-ID:To:CC:MIME-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References; bh=+vQZwpJzUJyzjDzELQkVWNjqgz2QMiLkmHm0ryW22Js=; b=uN/PdT8mMssH/Ny+0JZuCPUK0v6U3S5MlI0Ks1NuG3O2i8VG2MMBE+kZOBSrmDAVaj2TlW6JWWifBcGwyzMY3kv5TWc/tYDgrufoBDBjSM9J/QGoIGKkwM9Im1Yr2xeW9kPYaujKkjqEBNm441RTiZLsAChEIjiiaUfjG1xS0PHb7wRcPN5XCGwP63SiQR/b2in6k4sBOf7mCyxamdNMg/1YbL6CMy8habBPZnGg+I3WanjZ9Zlzt6GBx/ imRUG0eXRlt31+hC3gP+7NxDaGD5d+a2Ay7Eh4/WmxlEVbePtPFspJO4DbV6rFVmcMz7jS5KrXbZe3VIkUH+/Zt2dpug== Received: from mail.distal.com (pool-108-51-233-124.washdc.fios.verizon.net [108.51.233.124]) by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256) ; Fri, 6 Sep 2024 16:51:09 +0000 Received: from smtpclient.apple ( [2001:420:c0c4:1001::9f]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id 8b2aee94 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Fri, 6 Sep 2024 12:51:08 -0400 (EDT) Content-Type: text/plain; charset=us-ascii List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3776.700.51\)) Subject: Re: Unable to replace drive in raidz1 From: Chris Ross In-Reply-To: Date: Fri, 6 Sep 2024 12:50:55 -0400 Cc: FreeBSD Filesystems Content-Transfer-Encoding: quoted-printable Message-Id: References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> To: Alan Somers , mike tancsa X-Mailer: Apple Mail (2.3776.700.51) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[freebsd]; ASN(0.00)[asn:3842, ipnet:168.235.92.0/22, country:US] X-Rspamd-Queue-Id: 4X0j0v60Vgz430d > On Sep 6, 2024, at 11:32, Alan Somers wrote: >=20 > "zpool replace" is indeed the correct command. There's no need to run > "zpool offline" first, and "zpool remove" is wrong. Since "zpool > replace" is still failing, are you sure that da10 is still the correct > device name after all disks got renumbered? If you're sure, then you > might run "zdb -l /dev/da10" to see what ZFS thinks is on that disk. >=20 I can confirm that da10 is still the new disk I put into place of prior = da3. > On Sep 6, 2024, at 11:43, mike tancsa wrote: > I would triple check to see what the devices are that are part of the = pool. I wish there was a way to tell zfs to only display one or the = other. So list out what diskid/DISK-K1GMBN9D, diskid/DISK-K1GMEDMD... = to diskid/DISK-3WJ7ZMMJ are in terms of /dev/da* actually are. I have = some controllers that will re-order the disks on every reboot. glabel = status and camcontrol devlist should help verify camcontrol devlist lets me know that the three HGST drives making up zraid1-1 are da3,da4,da5 and the three WD drives making up zraid1-2 are da6,da7,da8. So, like before, just moved down a number because the prior da3 went away and a new disk in that physical slot became da10. (da9 is a loose JBOD single with ufs on it, previously da10, in slot 12 of 12) da10 is in fact still the disk in slot3 of the chassis, zdb -l shows the below. I did add and remove it as a spare while trying things, that may be why it shows up this way. - Chris % sudo zdb -l /dev/da10 ------------------------------------ LABEL 0 ------------------------------------ version: 5000 name: 'tank' state: 0 txg: 0 pool_guid: 3456317866677065800 errata: 0 hostid: 2747523522 hostname: 'frizzen02.devit.ciscolabs.com' top_guid: 2495145666029787532 guid: 2495145666029787532 vdev_children: 3 vdev_tree: type: 'disk' id: 0 guid: 2495145666029787532 path: '/dev/da10' phys_path: = 'id1,enc@n584b2612f2c321bd/type@0/slot@3/elmdesc@ArrayDevice03' whole_disk: 1 metaslab_array: 0 metaslab_shift: 0 ashift: 12 asize: 22000965255168 is_log: 0 create_txg: 18008413 features_for_read: com.delphix:hole_birth com.delphix:embedded_data create_txg: 18008413 labels =3D 0 1 2 3= From nobody Fri Sep 6 17:02:14 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0jFx1bMzz5VW9h for ; Fri, 06 Sep 2024 17:02:29 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oi1-f181.google.com (mail-oi1-f181.google.com [209.85.167.181]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0jFv4rmnz45BW for ; Fri, 6 Sep 2024 17:02:27 +0000 (UTC) (envelope-from asomers@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.167.181 as permitted sender) smtp.mailfrom=asomers@gmail.com Received: by mail-oi1-f181.google.com with SMTP id 5614622812f47-3df0e0e5577so1348768b6e.0 for ; Fri, 06 Sep 2024 10:02:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725642146; x=1726246946; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kN5DyQgL9diXk46IhlX0prsAncx02Mo5xQcPaQDhwis=; b=UIOY/dprR4vjPTiQrFtMnPfiA3mFvj6RS137/W4VVmti9tNQQPRjA5NVBfHFbBtfVb 9laJzQalHgARyNCoEqaIkF7XMp1dvIqVvfeTW7Zezn65Qn2BOP/WQuUa+jBit/CIL7f2 Hx6/Rb90CPQC424AJ3GirREN4Cpnh3v7Vsw9E6oEd+7uypcvMfDG2YB2z4RMy07syYVQ lXlvf1Rg9khiSESiZYt2mnM3H2hLdwU278GYDzIYaNeHhHGR39QD3uI9rnrrotJf5rDA pH6hbk2n4335DbTlUk6EeiVpd4Ex6zF5FQPaU+PiYRCdanCSrMWGqB7mrvSQfO9KwF1b dwow== X-Forwarded-Encrypted: i=1; AJvYcCUFamsfzHMkW/qc2XvI12JeXHah1O8PkmBrkkZtKp8yOcaKxqH+xa7ubiLMxzB0XEjj3Mej+vsQgenc@freebsd.org X-Gm-Message-State: AOJu0YxmZIneO6ERyPWhnaIb5HIuzSbsZKz1pr3EpZ88Xy/Mu74wYeGq tqCOtT5zYgWGc+qMWyVKcwvuMcJInInjt37GhnXxHisB6XOgZ4loIljJCVPMiIUoy6pieONV3Nh eNHVHVMV7AH+KTc/HX0ZRGi2PRIuaVQha X-Google-Smtp-Source: AGHT+IE7dQJ6OKrwSMW1f2xMmDSKA7VaiWnGNLrnzVT1LR9XhunOBq8DH6mGk9BEMgzAnfmUk+dDvkVTPpDVn6+n6/k= X-Received: by 2002:a05:6808:3c44:b0:3d9:2aa5:4077 with SMTP id 5614622812f47-3e029ceaa81mr4022046b6e.5.1725642146175; Fri, 06 Sep 2024 10:02:26 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> In-Reply-To: From: Alan Somers Date: Fri, 6 Sep 2024 11:02:14 -0600 Message-ID: Subject: Re: Unable to replace drive in raidz1 To: Chris Ross Cc: mike tancsa , FreeBSD Filesystems Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.90 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), No valid DKIM,none]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; MISSING_XM_UA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; FREEFALL_USER(0.00)[asomers]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; TAGGED_RCPT(0.00)[freebsd]; R_DKIM_NA(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.167.181:from]; RCVD_IN_DNSWL_NONE(0.00)[209.85.167.181:from] X-Rspamd-Queue-Id: 4X0jFv4rmnz45BW On Fri, Sep 6, 2024 at 10:51=E2=80=AFAM Chris Ross wrote: > > > > > On Sep 6, 2024, at 11:32, Alan Somers wrote: > > > > "zpool replace" is indeed the correct command. There's no need to run > > "zpool offline" first, and "zpool remove" is wrong. Since "zpool > > replace" is still failing, are you sure that da10 is still the correct > > device name after all disks got renumbered? If you're sure, then you > > might run "zdb -l /dev/da10" to see what ZFS thinks is on that disk. > > > > I can confirm that da10 is still the new disk I put into place of prior d= a3. > > > > On Sep 6, 2024, at 11:43, mike tancsa wrote: > > I would triple check to see what the devices are that are part of the p= ool. I wish there was a way to tell zfs to only display one or the other. = So list out what diskid/DISK-K1GMBN9D, diskid/DISK-K1GMEDMD... to diskid/D= ISK-3WJ7ZMMJ are in terms of /dev/da* actually are. I have some controller= s that will re-order the disks on every reboot. glabel status and camcontr= ol devlist should help verify > > > camcontrol devlist lets me know that the three HGST drives making up > zraid1-1 are da3,da4,da5 and the three WD drives making up > zraid1-2 are da6,da7,da8. So, like before, just moved down a > number because the prior da3 went away and a new disk in that > physical slot became da10. (da9 is a loose JBOD single with ufs > on it, previously da10, in slot 12 of 12) > > da10 is in fact still the disk in slot3 of the chassis, zdb -l shows > the below. I did add and remove it as a spare while trying things, > that may be why it shows up this way. > > - Chris > > % sudo zdb -l /dev/da10 > ------------------------------------ > LABEL 0 > ------------------------------------ > version: 5000 > name: 'tank' > state: 0 > txg: 0 > pool_guid: 3456317866677065800 > errata: 0 > hostid: 2747523522 > hostname: 'frizzen02.devit.ciscolabs.com' > top_guid: 2495145666029787532 > guid: 2495145666029787532 > vdev_children: 3 > vdev_tree: > type: 'disk' > id: 0 > guid: 2495145666029787532 > path: '/dev/da10' > phys_path: 'id1,enc@n584b2612f2c321bd/type@0/slot@3/elmdesc@Array= Device03' > whole_disk: 1 > metaslab_array: 0 > metaslab_shift: 0 > ashift: 12 > asize: 22000965255168 > is_log: 0 > create_txg: 18008413 > features_for_read: > com.delphix:hole_birth > com.delphix:embedded_data > create_txg: 18008413 > labels =3D 0 1 2 3 This looks like you got into a split-brain situation where the disks have inconsistent labels. Most disks think that da10 is not a member of the pool, but da10 thinks that it is. Perhaps you added it as a spare, then physically removed it, and then did a "zpool remove" to remove the spare from the configuration? If you're very very very sure that there is no data on da10 that you care about, you can do "zpool labelclear -f /dev/da10" From nobody Fri Sep 6 17:50:31 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0kKg69Htz5VMVR for ; Fri, 06 Sep 2024 17:50:47 +0000 (UTC) (envelope-from cross+freebsd@relay.distal.com) Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0kKg1qnSz4F3w; Fri, 6 Sep 2024 17:50:47 +0000 (UTC) (envelope-from cross+freebsd@relay.distal.com) Authentication-Results: mx1.freebsd.org; none dkim-signature: v=1; a=rsa-sha256; d=relay.distal.com; s=mail; c=relaxed/relaxed; q=dns/txt; h=From:Subject:Date:Message-ID:To:CC:MIME-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References; bh=qJURHgQq3Bnoivbpiymn0dmDSeiboh8hmAcGsBkFrD0=; b=YlrFeblUQhHSziLzWtLxMIqCICe+lHYSAZPX6K5qTOxlyKkRIbyBL+tKiqI7xQI7+9YeULphzCaNc/3+m4sOifY9e5gvh6fGZCBw28mSigWBHA277Iwr9aflHEQzShY8TwjKumaXuuE4Oa9DV5XElfxw/CqPgUfOAOIcTf7eQsOV/4O0uLOvJuWUqrNbthCvVkMX1prtZzwc1ZeD/pyfkssmPWTH0c7DqyXuG3jRboh/9gE3hidA8QiDUb +FujLqHx53yTh0TTURm0o0piVA70HIpt8dsHtG/rkr8A7QyywWU+CC6PbgZPFn1RXbFLRBLKsyvA94C1QsVtNvSpZ2ZQ== Received: from mail.distal.com (pool-108-51-233-124.washdc.fios.verizon.net [108.51.233.124]) by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256) ; Fri, 6 Sep 2024 17:50:45 +0000 Received: from smtpclient.apple ( [2001:420:c0c4:1001::9f]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id 325d6f1e (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Fri, 6 Sep 2024 13:50:43 -0400 (EDT) Content-Type: text/plain; charset=utf-8 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3776.700.51\)) Subject: Re: Unable to replace drive in raidz1 From: Chris Ross In-Reply-To: Date: Fri, 6 Sep 2024 13:50:31 -0400 Cc: mike tancsa , FreeBSD Filesystems Content-Transfer-Encoding: quoted-printable Message-Id: <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com> References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> To: Alan Somers X-Mailer: Apple Mail (2.3776.700.51) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[freebsd]; ASN(0.00)[asn:3842, ipnet:168.235.92.0/22, country:US] X-Rspamd-Queue-Id: 4X0kKg1qnSz4F3w > On Sep 6, 2024, at 13:02, Alan Somers wrote: >=20 > This looks like you got into a split-brain situation where the disks > have inconsistent labels. Most disks think that da10 is not a member > of the pool, but da10 thinks that it is. Perhaps you added it as a > spare, then physically removed it, and then did a "zpool remove" to > remove the spare from the configuration? I did configure it as a spare, and remove it as a spare, but I=20 haven=E2=80=99t moved any disks physically since the once when I=20 switched it in. And this problem started before I ever tried adding da10 into the pool as a spare. > If you're very very very > sure that there is no data on da10 that you care about, you can do > "zpool labelclear -f /dev/da10=E2=80=9D I am sure, and I didn=E2=80=99t even need the -f. But, no change. % sudo zpool labelclear /dev/da10 Password: % sudo zdb -l /dev/da10 failed to unpack label 0 failed to unpack label 1 failed to unpack label 2 failed to unpack label 3 % sudo zpool replace tank da3 da10 cannot replace da3 with da10: already in replacing/spare config; wait = for completion or use 'zpool detach' :-( - Chris= From nobody Fri Sep 6 17:59:26 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0kWw0LNRz5VN9Y for ; Fri, 06 Sep 2024 17:59:40 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-vs1-f42.google.com (mail-vs1-f42.google.com [209.85.217.42]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0kWv1FRDz4HB3 for ; Fri, 6 Sep 2024 17:59:39 +0000 (UTC) (envelope-from asomers@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.217.42 as permitted sender) smtp.mailfrom=asomers@gmail.com Received: by mail-vs1-f42.google.com with SMTP id ada2fe7eead31-49bc12c0041so676117137.0 for ; Fri, 06 Sep 2024 10:59:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725645578; x=1726250378; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Fz1K8g7jg6+HCV9yZhd+AyuBrOmuFwH+yOqBBdJnfeI=; b=T3Y02fSG1ObzADaXe/wjS5uDSTGk/XtLqMRGUe22jhCIcx0ZvS2W5/ATFJadHnRd8J DXIrxdt1/gCzYn1Fm+zJt3CMRPChLlnbc1Wqpd6QHpH6ued0K2z1zcPwbrGl/hK+0squ VBbv3AY+pWI/h+fc37yrOSlgEyZi0mQqCFRcGLsFQ4dCuGv4p68g8fr97zmxUOPc73v3 at6NlY6G1JkAArZY6U91hnXS45FMg0NPSiDJDPf0/cLdHXDts9Pn3eJWlnVQQ+iaJYB/ R0t7+B5f00wmnRgWnhGcYH+K+4ebsJwrzTU7Ln+VYIxTAE5OGIJnLP5fJb3DYcbqu8O0 auFw== X-Forwarded-Encrypted: i=1; AJvYcCU95VTD+Ey5KHHxaA+foGfASPoI4aIeNN2GgSEJ/XTbwMwPyRXKsnrQgKaDEk02Ryypelau+oDg5pKR@freebsd.org X-Gm-Message-State: AOJu0YyAKMs/TFtEqbMfHu47AKJgVzHKuxGLt3iP0sdD/qUyoyM6j2wK nYIOa2CyTZCIQBkDALY9f+TL7f2tBkCEIr9arZgvzEpw8wLwRL+2kRKUm0nQWXqpCWMRo/o0vur eyIMk1r9zKuvAbkUyTg45vyikuUA= X-Google-Smtp-Source: AGHT+IGflI4t6Esp/PX6XcR70jXxkugLNwwc227OEZPOCR23B2m5FLyReq9zEosk5WISQyEgSvN12DCll1hddTff81o= X-Received: by 2002:a05:6102:292a:b0:498:cdb0:1d03 with SMTP id ada2fe7eead31-49bde263682mr4144787137.23.1725645578077; Fri, 06 Sep 2024 10:59:38 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com> In-Reply-To: <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com> From: Alan Somers Date: Fri, 6 Sep 2024 11:59:26 -0600 Message-ID: Subject: Re: Unable to replace drive in raidz1 To: Chris Ross Cc: mike tancsa , FreeBSD Filesystems Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.37 / 15.00]; NEURAL_HAM_LONG(-1.00)[-0.995]; NEURAL_HAM_MEDIUM(-0.75)[-0.746]; NEURAL_HAM_SHORT(-0.73)[-0.731]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), No valid DKIM,none]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; MISSING_XM_UA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; FREEFALL_USER(0.00)[asomers]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; TAGGED_RCPT(0.00)[freebsd]; R_DKIM_NA(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.217.42:from]; RCVD_IN_DNSWL_NONE(0.00)[209.85.217.42:from] X-Rspamd-Queue-Id: 4X0kWv1FRDz4HB3 On Fri, Sep 6, 2024 at 11:50=E2=80=AFAM Chris Ross wrote: > > > > > On Sep 6, 2024, at 13:02, Alan Somers wrote: > > > > This looks like you got into a split-brain situation where the disks > > have inconsistent labels. Most disks think that da10 is not a member > > of the pool, but da10 thinks that it is. Perhaps you added it as a > > spare, then physically removed it, and then did a "zpool remove" to > > remove the spare from the configuration? > > I did configure it as a spare, and remove it as a spare, but I > haven=E2=80=99t moved any disks physically since the once when I > switched it in. And this problem started before I ever tried > adding da10 into the pool as a spare. > > > If you're very very very > > sure that there is no data on da10 that you care about, you can do > > "zpool labelclear -f /dev/da10=E2=80=9D > > > I am sure, and I didn=E2=80=99t even need the -f. But, no change. > > % sudo zpool labelclear /dev/da10 > Password: > > % sudo zdb -l /dev/da10 > failed to unpack label 0 > failed to unpack label 1 > failed to unpack label 2 > failed to unpack label 3 > > % sudo zpool replace tank da3 da10 > cannot replace da3 with da10: already in replacing/spare config; wait for= completion or use 'zpool detach' > > > :-( > > - Chris If there is no label on da10, and "zpool status" doesn't show any spares, then I don't know what the problem is. It's possible that /sbin/zpool is printing an incorrect error message; it's fairly notorious for that. You could try to debug it. Other wild guesses include: * maybe da3 is the disk with the out-of-date label. You could try physically removing it before doing "zpool replace" * Since exported pools can't have active spares, you could try exporting the pool and then reimporting it. -Alan From nobody Fri Sep 6 18:06:32 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0kh75YT9z5VPJX for ; Fri, 06 Sep 2024 18:06:47 +0000 (UTC) (envelope-from cross+freebsd@relay.distal.com) Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0kh731Wdz4KYQ; Fri, 6 Sep 2024 18:06:47 +0000 (UTC) (envelope-from cross+freebsd@relay.distal.com) Authentication-Results: mx1.freebsd.org; none dkim-signature: v=1; a=rsa-sha256; d=relay.distal.com; s=mail; c=relaxed/relaxed; q=dns/txt; h=From:Subject:Date:Message-ID:To:CC:MIME-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References; bh=8ROrnENf/y2tprcw3YaIAWUhaj71MeY73/qJy/AcM2Q=; b=NQ6U6zKYo5KQVVwFfGEIXII+Rc/v/t4btgRv6IIlGBQ/GiMQoXrdshdRGCsCt6CvhyqfT+U5i6leb+qjTqxwK3Sk4wsLc7SqHMJoxtSGdujo8N9PsFcX+aDtRJp0nV+FCUoyVDsNIJ00mjLMVDua7WU/2IK9OHGa/TQOINzlDk4LXzoBrzfKV8+rhINqLvVoO7ycn8e72DzyaaUpEgQo0nIheu6biZpjQuPsNZoZIdltfBG0nmvMZjfat1 jecFmWmGFWpSoNIEgGpKxon/WTAq8RBiBXm9J6WZMAD4nwyZfVYzsfm46StQzqQgutv+Jl/SlY6eY+EP7nsWeMmlZuWQ== Received: from mail.distal.com (pool-108-51-233-124.washdc.fios.verizon.net [108.51.233.124]) by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256) ; Fri, 6 Sep 2024 18:06:45 +0000 Received: from smtpclient.apple ( [2001:420:c0c4:1001::9f]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id 9822b920 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Fri, 6 Sep 2024 14:06:43 -0400 (EDT) Content-Type: text/plain; charset=utf-8 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3776.700.51\)) Subject: Re: Unable to replace drive in raidz1 From: Chris Ross In-Reply-To: Date: Fri, 6 Sep 2024 14:06:32 -0400 Cc: mike tancsa , FreeBSD Filesystems Content-Transfer-Encoding: quoted-printable Message-Id: <0CF1E2D7-6C82-4A8B-82C3-A5BF1ED939CF@distal.com> References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com> To: Alan Somers X-Mailer: Apple Mail (2.3776.700.51) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[freebsd]; ASN(0.00)[asn:3842, ipnet:168.235.92.0/22, country:US] X-Rspamd-Queue-Id: 4X0kh731Wdz4KYQ > On Sep 6, 2024, at 13:59, Alan Somers wrote: >=20 > On Fri, Sep 6, 2024 at 11:50=E2=80=AFAM Chris Ross = wrote: >> % sudo zpool replace tank da3 da10 >> cannot replace da3 with da10: already in replacing/spare config; wait = for completion or use 'zpool detach' >>=20 >>=20 >=20 > If there is no label on da10, and "zpool status" doesn't show any > spares, then I don't know what the problem is. It's possible that > /sbin/zpool is printing an incorrect error message; it's fairly > notorious for that. You could try to debug it. Other wild guesses > include: > * maybe da3 is the disk with the out-of-date label. You could try > physically removing it before doing "zpool replace" Well, there is a problem there. /dev/da3, right now, is one of the disks in zraid1-1, the second set. It=E2=80=99s working there, = listed as diskid/??? How can I map the diskid=E2=80=99s listed to the underlying device? either by serial number or da#=E2=80=A6 > * Since exported pools can't have active spares, you could try > exporting the pool and then reimporting it. Okay. Lukcily this is _not_ a root on ZFS system, so that should be doable. Hopefully there aren=E2=80=99t significant risks there? Can I tell ZFS to reconsider the labels in the first zraid1? since it=E2=80=99s still reporting da3 as missing/failed, and that = kernel dev is not what it thinks that is=E2=80=A6 - Chris From nobody Fri Sep 6 18:08:31 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0kkW10zXz5VPWq for ; Fri, 06 Sep 2024 18:08:51 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smarthost1.sentex.ca", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0kkV3Sdqz4Km4; Fri, 6 Sep 2024 18:08:50 +0000 (UTC) (envelope-from mike@sentex.net) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:1::12 as permitted sender) smtp.mailfrom=mike@sentex.net Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.18.1/8.18.1) with ESMTPS id 486I8UFj046147 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL); Fri, 6 Sep 2024 14:08:30 -0400 (EDT) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:d98b:f91d:4232:8049] ([IPv6:2607:f3e0:0:4:d98b:f91d:4232:8049]) by pyroxene2a.sentex.ca (8.18.1/8.15.2) with ESMTPS id 486I8T4s096790 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Fri, 6 Sep 2024 14:08:29 -0400 (EDT) (envelope-from mike@sentex.net) Content-Type: multipart/alternative; boundary="------------FdJV1kYheV3uTe3SnA2Ty6SV" Message-ID: <057e1ec3-58b7-4895-a099-c7d74fbeb865@sentex.net> Date: Fri, 6 Sep 2024 14:08:31 -0400 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Unable to replace drive in raidz1 To: Chris Ross , Alan Somers Cc: FreeBSD Filesystems References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com> <0CF1E2D7-6C82-4A8B-82C3-A5BF1ED939CF@distal.com> Content-Language: en-US From: mike tancsa Autocrypt: addr=mike@sentex.net; keydata= xsBNBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAHNHW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+wsCOBBMBCAA4FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAl+pQfkCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQeVOEFl5W rMiN6ggAk3H5vk8QnbvGbb4sinxZt/wDetgk0AOR9NRmtTnPaW+sIJEfGBOz47Xih+f7uWJS j+uvc9Ewn2Z7n8z3ZHJlLAByLVLtcNXGoRIGJ27tevfOaNqgJHBPbFOcXCBBFTx4MYMM4iAZ cDT5vsBTSaM36JZFtHZBKkuFEItbA/N8ZQSHKdTYMIA7A3OCLGbJBqloQ8SlW4MkTzKX4u7R yefAYQ0h20x9IqC5Ju8IsYRFacVZconT16KS81IBceO42vXTN0VexbVF2rZIx3v/NT75r6Vw 0FlXVB1lXOHKydRA2NeleS4NEG2vWqy/9Boj0itMfNDlOhkrA/0DcCurMpnpbM7ATQRcsMzk AQgA1Dpo/xWS66MaOJLwA28sKNMwkEk1Yjs+okOXDOu1F+0qvgE8sVmrOOPvvWr4axtKRSG1 t2QUiZ/ZkW/x/+t0nrM39EANV1VncuQZ1ceIiwTJFqGZQ8kb0+BNkwuNVFHRgXm1qzAJweEt RdsCMohB+H7BL5LGCVG5JaU0lqFU9pFP40HxEbyzxjsZgSE8LwkI6wcu0BLv6K6cLm0EiHPO l5G8kgRi38PS7/6s3R8QDsEtbGsYy6O82k3zSLIjuDBwA9GRaeigGppTxzAHVjf5o9KKu4O7 gC2KKVHPegbXS+GK7DU0fjzX57H5bZ6komE5eY4p3oWT/CwVPSGfPs8jOwARAQABwsB2BBgB CAAgFiEEmuvCXT0aY6hs4SbWeVOEFl5WrMgFAl+pQfkCGwwACgkQeVOEFl5WrMiVqwf9GwU8 c6cylknZX8QwlsVudTC8xr/L17JA84wf03k3d4wxP7bqy5AYy7jboZMbgWXngAE/HPQU95NM aukysSnknzoIpC96XZJ0okLBXVS6Y0ylZQ+HrbIhMpuQPoDweoF5F9wKrsHRoDaUK1VR706X rwm4HUzh7Jk+auuMYfuCh0FVlFBEuiJWMLhg/5WCmcRfiuB6F59ZcUQrwLEZeNhF2XJV4KwB Tlg7HCWO/sy1foE5noaMyACjAtAQE9p5kGYaj+DuRhPdWUTsHNuqrhikzIZd2rrcMid+ktb0 NvtvswzMO059z1YGMtGSqQ4srCArju+XHIdTFdiIYbd7+jeehg== In-Reply-To: <0CF1E2D7-6C82-4A8B-82C3-A5BF1ED939CF@distal.com> X-Scanned-By: MIMEDefang 2.86 X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.38 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.995]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; RCVD_IN_DNSWL_LOW(-0.10)[199.212.134.19:received]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; XM_UA_NO_VERSION(0.01)[]; TAGGED_RCPT(0.00)[freebsd]; ARC_NA(0.00)[]; FREEFALL_USER(0.00)[mike]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_ALL(0.00)[]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_DN_ALL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[sentex.net]; RCPT_COUNT_THREE(0.00)[3] X-Rspamd-Queue-Id: 4X0kkV3Sdqz4Km4 This is a multi-part message in MIME format. --------------FdJV1kYheV3uTe3SnA2Ty6SV Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 9/6/2024 2:06 PM, Chris Ross wrote: > How can I map the diskid’s listed to the underlying device? > either by serial number or da#… What does glabel status show     ---Mike --------------FdJV1kYheV3uTe3SnA2Ty6SV Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 9/6/2024 2:06 PM, Chris Ross wrote:
How can I map the diskid’s listed to the underlying device? 
either by serial number or da#…

What does

glabel status

show

    ---Mike


    

  


--------------FdJV1kYheV3uTe3SnA2Ty6SV--

From nobody Fri Sep  6 18:10:22 2024
X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0kmX68Wdz5VPJy
	for ; Fri, 06 Sep 2024 18:10:36 +0000 (UTC)
	(envelope-from asomers@gmail.com)
Received: from mail-oi1-f180.google.com (mail-oi1-f180.google.com [209.85.167.180])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "WR4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4X0kmW4pKrz4L9c
	for ; Fri,  6 Sep 2024 18:10:35 +0000 (UTC)
	(envelope-from asomers@gmail.com)
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=freebsd.org (policy=none);
	spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.167.180 as permitted sender) smtp.mailfrom=asomers@gmail.com
Received: by mail-oi1-f180.google.com with SMTP id 5614622812f47-3df0e0e5577so1387987b6e.0
        for ; Fri, 06 Sep 2024 11:10:35 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1725646234; x=1726251034;
        h=content-transfer-encoding:cc:to:subject:message-id:date:from
         :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=5E9VocnPsFQXrzH2F7ELwusDNceTpQj1VKUKD7VWj1A=;
        b=lU4jrIk1lrTA+VaFAOkM8xczlJ/MFw3sysPmvGPLquXWkUCG5KJ7L4wpuqZZ0ThfmO
         /Y3ELhRbCnTWFySC1Ygk9Qe1iOos4UFAVjrwgW70UUXd3wxaHKsEKFaU4/Kctx2TdzUl
         wbDJi3+L43upjpIMymoHubfxcT3KtEu/AUFgCApzlQHyXijsy4+UlpAsTVTJWZWQ15+l
         nDt5DZmq+YbTrkd2aP4XcVQWbrUgH+3hYaX+TESgDdN1P/GBzhND4eqVjccMluyJSeOy
         th1F1KVdMMmJW1fNKqsY9LFJzO/UMLeehZrGiciBLqnPWjpXFLK+4VP63RXyaa6UmIzS
         GZKw==
X-Forwarded-Encrypted: i=1; AJvYcCWbWLLXe/0BGVpMb7r1q6crTJKcjg0q5AqcBdT8eLqTpeQ7JKYh+vyjJhhjz9+QyeUMN9AIywR3C2J3@freebsd.org
X-Gm-Message-State: AOJu0Yx05Qx8IkG1nywSqxjA/6ZK97M2yVOT+3b8l/a96HQT8LbcCc6o
	lW2J0UScDfrf8H/CbHMuXDE1eu6Xp1pSZ7MFL4//exmG+DI25G8EXEzxqcTJvjSzc6pVSXj05mC
	VocPRoyzO4J4yaXnMF7xLtayVqth+evOK
X-Google-Smtp-Source: AGHT+IHMO7htfHnBW+j3BD1A5+kZTmlU0N2aBfqcrqPLQvPB1FxYNseyoh/xqT2q6sJYic8FJ3xyb4D4fN6tXhp9Kic=
X-Received: by 2002:a05:6808:1709:b0:3df:144f:9ef9 with SMTP id
 5614622812f47-3e02a02d73cmr3922999b6e.41.1725646234367; Fri, 06 Sep 2024
 11:10:34 -0700 (PDT)
List-Id: Filesystems 
List-Archive: https://lists.freebsd.org/archives/freebsd-fs
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-fs@FreeBSD.org
MIME-Version: 1.0
References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com>
  
  
 <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com> 
 <0CF1E2D7-6C82-4A8B-82C3-A5BF1ED939CF@distal.com>
In-Reply-To: <0CF1E2D7-6C82-4A8B-82C3-A5BF1ED939CF@distal.com>
From: Alan Somers 
Date: Fri, 6 Sep 2024 12:10:22 -0600
Message-ID: 
Subject: Re: Unable to replace drive in raidz1
To: Chris Ross 
Cc: mike tancsa , FreeBSD Filesystems 
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spamd-Bar: --
X-Spamd-Result: default: False [-2.38 / 15.00];
	NEURAL_HAM_LONG(-1.00)[-0.996];
	NEURAL_HAM_MEDIUM(-0.75)[-0.753];
	NEURAL_HAM_SHORT(-0.73)[-0.735];
	FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com];
	R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17];
	MIME_GOOD(-0.10)[text/plain];
	DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), No valid DKIM,none];
	RCPT_COUNT_THREE(0.00)[3];
	FROM_HAS_DN(0.00)[];
	TO_DN_ALL(0.00)[];
	RCVD_TLS_LAST(0.00)[];
	ARC_NA(0.00)[];
	MISSING_XM_UA(0.00)[];
	MIME_TRACE(0.00)[0:+];
	RCVD_COUNT_ONE(0.00)[1];
	ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US];
	FREEMAIL_ENVFROM(0.00)[gmail.com];
	PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org];
	TO_MATCH_ENVRCPT_SOME(0.00)[];
	FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com];
	FREEFALL_USER(0.00)[asomers];
	MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org];
	TAGGED_RCPT(0.00)[freebsd];
	R_DKIM_NA(0.00)[];
	RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.167.180:from];
	RCVD_IN_DNSWL_NONE(0.00)[209.85.167.180:from]
X-Rspamd-Queue-Id: 4X0kmW4pKrz4L9c

On Fri, Sep 6, 2024 at 12:06=E2=80=AFPM Chris Ross  wrote:
>
>
>
> > On Sep 6, 2024, at 13:59, Alan Somers  wrote:
> >
> > On Fri, Sep 6, 2024 at 11:50=E2=80=AFAM Chris Ross  wrote:
> >> % sudo zpool replace tank da3 da10
> >> cannot replace da3 with da10: already in replacing/spare config; wait =
for completion or use 'zpool detach'
> >>
> >>
> >
> > If there is no label on da10, and "zpool status" doesn't show any
> > spares, then I don't know what the problem is.  It's possible that
> > /sbin/zpool is printing an incorrect error message; it's fairly
> > notorious for that.  You could try to debug it.  Other wild guesses
> > include:
> > * maybe da3 is the disk with the out-of-date label.  You could try
> > physically removing it before doing "zpool replace"
>
> Well, there is a problem there.  /dev/da3, right now, is one of
> the disks in zraid1-1, the second set.  It=E2=80=99s working there, liste=
d
> as diskid/???
>
> How can I map the diskid=E2=80=99s listed to the underlying device?
> either by serial number or da#=E2=80=A6

The gpart command should do it, but I don't recommend that.  Instead,
see below...

>
> > * Since exported pools can't have active spares, you could try
> > exporting the pool and then reimporting it.
>
> Okay.  Lukcily this is _not_ a root on ZFS system, so that should
> be doable.  Hopefully there aren=E2=80=99t significant risks there?
>
> Can I tell ZFS to reconsider the labels in the first zraid1?
> since it=E2=80=99s still reporting da3 as missing/failed, and that kernel
> dev is not what it thinks that is=E2=80=A6

Ahh, this means that there are two different vdevs that can be
described by "da3".  You can still refer to them unambiguously by guid
though.  Do "zpool status -g" to find the guid of the disk that you
want to replace, and then do "zpool replace  /dev/da10"

From nobody Fri Sep  6 18:21:06 2024
X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0l0x3RxTz5VQRD
	for ; Fri, 06 Sep 2024 18:21:21 +0000 (UTC)
	(envelope-from cross+freebsd@relay.distal.com)
Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(Client did not present a certificate)
	by mx1.freebsd.org (Postfix) with ESMTPS id 4X0l0w6DJCz4QXv;
	Fri,  6 Sep 2024 18:21:20 +0000 (UTC)
	(envelope-from cross+freebsd@relay.distal.com)
Authentication-Results: mx1.freebsd.org;
	none
dkim-signature: v=1; a=rsa-sha256; d=relay.distal.com; s=mail;
	c=relaxed/relaxed; q=dns/txt; h=From:Subject:Date:Message-ID:To:CC:MIME-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References;
	bh=71BFKHM9WzggFHIRluEaufEgF4+HLHsigY2bsGv0peU=;
	b=esD8lA5TK1cZDkaLetmQVHyCUXMqIZyzvIzTrqdD/2WDcIbcyRSvlHs28S51SJGxmMtMHNqL3w9si37gL1ICp5C38OZnc60UppOMI1Yo2DIBM2qc8wpyLJ09p27lnEheliWNZ0wQFCxFuwmfuFFpq7vjaBjKLuK4FmyNFP0G7Xl5R0GJnY641tGKRM69NGFC/Ml65L+khhCRq+k3tdu1JGRJnBU64Hh1jabz+PC0CQqeOqmm8lIMBrprnI
	yG/+5tkXTGggxGPmVO3m+/J8rZIW0NnKLdNsV4JpcK+T51jKU0fxnb64O3pX/D6tw7dV+SzhSRudTgWqxXNpE74s6/Ng==
Received: from mail.distal.com (pool-108-51-233-124.washdc.fios.verizon.net [108.51.233.124])
 by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305
 bits=256) ; Fri, 6 Sep 2024 18:21:19 +0000
Received: from smtpclient.apple ( [2001:420:c0c4:1001::9f]) by tristain.distal.com
 (OpenSMTPD) with ESMTPSA id d36464d5 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO);
 Fri, 6 Sep 2024 14:21:17 -0400 (EDT)
Content-Type: text/plain; charset=utf-8
List-Id: Filesystems 
List-Archive: https://lists.freebsd.org/archives/freebsd-fs
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-fs@FreeBSD.org
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3776.700.51\))
Subject: Re: Unable to replace drive in raidz1
From: Chris Ross 
In-Reply-To: 
Date: Fri, 6 Sep 2024 14:21:06 -0400
Cc: mike tancsa , FreeBSD Filesystems 
Content-Transfer-Encoding: quoted-printable
Message-Id: <29003A7C-745D-4A06-8558-AE64310813EA@distal.com>
References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> 
  
  <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com>
  <0CF1E2D7-6C82-4A8B-82C3-A5BF1ED939CF@distal.com>
 
To: Alan Somers 
X-Mailer: Apple Mail (2.3776.700.51)
X-Spamd-Bar: ----
X-Rspamd-Pre-Result: action=no action;
	module=replies;
	Message is reply to one we originated
X-Spamd-Result: default: False [-4.00 / 15.00];
	REPLY(-4.00)[];
	TAGGED_FROM(0.00)[freebsd];
	ASN(0.00)[asn:3842, ipnet:168.235.92.0/22, country:US]
X-Rspamd-Queue-Id: 4X0l0w6DJCz4QXv


> On Sep 6, 2024, at 14:08, mike tancsa  wrote:
>=20
> On 9/6/2024 2:06 PM, Chris Ross wrote:
>> How can I map the diskid=E2=80=99s listed to the underlying =
device?either by serial number or da#=E2=80=A6
>>=20
> What does=20
> glabel status

That shows labels for many drives, though not da1 and da2, the
remaining members of zraid1-0.  Intersting.  But, I hope the below
Is the fix...

> On Sep 6, 2024, at 14:10, Alan Somers  wrote:
>=20
> Ahh, this means that there are two different vdevs that can be
> described by "da3".  You can still refer to them unambiguously by guid
> though.  Do "zpool status -g" to find the guid of the disk that you
> want to replace, and then do "zpool replace  /dev/da10=E2=80=9D


Ahh, okay.  That makes sense.  I have ever only known how to replace =
things
using the key that "zpool status=E2=80=9D shows.  Thanks for that!

Oh.  Trying, that doesn=E2=80=99t work either.  :-/

NAME                      STATE     READ WRITE CKSUM
tank                      DEGRADED     0     0     0
 16506780107187041124    DEGRADED     0     0     0
   9127016430593660128   FAULTED      0     0     0  external device =
fault
   4094297345166589692   ONLINE       0     0     0
   17850258180603290288  ONLINE       0     0     0
[=E2=80=A6]
% sudo zpool replace tank 9127016430593660128 /dev/da10
cannot replace 9127016430593660128 with /dev/da10: already in =
replacing/spare config; wait for completion or use 'zpool detach=E2=80=99
% sudo zpool replace tank 9127016430593660128 diskid/DISK-ZGG0A2PA
cannot replace 9127016430593660128 with diskid/DISK-ZGG0A2PA: already in =
replacing/spare config; wait for completion or use 'zpool detach'

Tried with /dev/da10, and the diskid for da10 reported by glabel status.

         - Chris




From nobody Fri Sep  6 18:35:22 2024
X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0lKV6Bf3z5VSNN
	for ; Fri, 06 Sep 2024 18:35:42 +0000 (UTC)
	(envelope-from cross+freebsd@relay.distal.com)
Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(Client did not present a certificate)
	by mx1.freebsd.org (Postfix) with ESMTPS id 4X0lKT54X5z4Wfg;
	Fri,  6 Sep 2024 18:35:41 +0000 (UTC)
	(envelope-from cross+freebsd@relay.distal.com)
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=relay.distal.com header.s=mail header.b=m+W02yw4;
	dmarc=none;
	spf=pass (mx1.freebsd.org: domain of cross+freebsd@relay.distal.com designates 168.235.95.80 as permitted sender) smtp.mailfrom=cross+freebsd@relay.distal.com
dkim-signature: v=1; a=rsa-sha256; d=relay.distal.com; s=mail;
	c=relaxed/relaxed; q=dns/txt; h=From:Subject:Date:Message-ID:To:CC:MIME-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References;
	bh=8OIQjErI4YO/vMNtKxCPN+XfQf5Oy+u+q0IoYSDzbq8=;
	b=m+W02yw4SZ0G4MDVv7EH0AKage27QefNh8vjED3lACpjZpxijX6sdVNQLAqmoekKU6+MZ2p7/EZw/rTcJNlV0sblMrTHcn4IAqhFFi38NuYbQOXs73s1uATQtt194VBzM1Rg7au2KKPRhcW9mipcLnEwex7JhRsigBj5be/fGo4T85tolfCNeWQe0aQBzshp677fhO0VTvaeMbJ7cXBBkmofHZCLiypfQqNqeHND+oDqg4uR1K6LbMu6ye
	jvMXWmUImS0zxClJr36glort2nEZ3fMJAQ2newFeWW1jDLqHKIDGv31j0ldXuV4Bbe9hSt42mr+/GBEpYPRQ2WRTfEiw==
Received: from mail.distal.com (pool-108-51-233-124.washdc.fios.verizon.net [108.51.233.124])
 by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305
 bits=256) ; Fri, 6 Sep 2024 18:35:35 +0000
Received: from smtpclient.apple ( [2001:420:c0c4:1001::9f]) by tristain.distal.com
 (OpenSMTPD) with ESMTPSA id 2eb1851e (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO);
 Fri, 6 Sep 2024 14:35:34 -0400 (EDT)
Content-Type: text/plain; charset=utf-8
List-Id: Filesystems 
List-Archive: https://lists.freebsd.org/archives/freebsd-fs
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-fs@FreeBSD.org
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3776.700.51\))
Subject: Re: Unable to replace drive in raidz1
From: Chris Ross 
In-Reply-To: <29003A7C-745D-4A06-8558-AE64310813EA@distal.com>
Date: Fri, 6 Sep 2024 14:35:22 -0400
Cc: mike tancsa , FreeBSD Filesystems 
Content-Transfer-Encoding: quoted-printable
Message-Id: <780A9427-C9BA-481F-9139-992B323F863B@distal.com>
References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> 
  
  <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com>
  <0CF1E2D7-6C82-4A8B-82C3-A5BF1ED939CF@distal.com>
  <29003A7C-745D-4A06-8558-AE64310813EA@distal.com>
To: Alan Somers 
X-Mailer: Apple Mail (2.3776.700.51)
X-Spamd-Bar: ---
X-Spamd-Result: default: False [-3.20 / 15.00];
	NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	NEURAL_HAM_LONG(-1.00)[-1.000];
	NEURAL_HAM_SHORT(-1.00)[-0.999];
	FORGED_SENDER(0.30)[cross@distal.com,cross@relay.distal.com];
	R_SPF_ALLOW(-0.20)[+a:relay.dynu.com];
	R_DKIM_ALLOW(-0.20)[relay.distal.com:s=mail];
	MIME_GOOD(-0.10)[text/plain];
	DKIM_TRACE(0.00)[relay.distal.com:+];
	RCVD_TLS_ALL(0.00)[];
	TO_DN_ALL(0.00)[];
	RCPT_COUNT_THREE(0.00)[3];
	TAGGED_FROM(0.00)[freebsd];
	DMARC_NA(0.00)[distal.com];
	FROM_HAS_DN(0.00)[];
	RCVD_VIA_SMTP_AUTH(0.00)[];
	ARC_NA(0.00)[];
	RCVD_COUNT_TWO(0.00)[2];
	FROM_NEQ_ENVFROM(0.00)[cross@distal.com,cross@relay.distal.com];
	ASN(0.00)[asn:3842, ipnet:168.235.92.0/22, country:US];
	TO_MATCH_ENVRCPT_SOME(0.00)[];
	MID_RHS_MATCH_FROM(0.00)[];
	MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org];
	APPLE_MAILER_COMMON(0.00)[];
	MIME_TRACE(0.00)[0:+]
X-Rspamd-Queue-Id: 4X0lKT54X5z4Wfg



> On Sep 6, 2024, at 14:21, Chris Ross  wrote:
> NAME                      STATE     READ WRITE CKSUM
> tank                      DEGRADED     0     0     0
> 16506780107187041124    DEGRADED     0     0     0
>   9127016430593660128   FAULTED      0     0     0  external device =
fault
>   4094297345166589692   ONLINE       0     0     0
>   17850258180603290288  ONLINE       0     0     0
> [=E2=80=A6]
> % sudo zpool replace tank 9127016430593660128 /dev/da10
> cannot replace 9127016430593660128 with /dev/da10: already in =
replacing/spare config; wait for completion or use 'zpool detach=E2=80=99
> % sudo zpool replace tank 9127016430593660128 diskid/DISK-ZGG0A2PA
> cannot replace 9127016430593660128 with diskid/DISK-ZGG0A2PA: already =
in replacing/spare config; wait for completion or use 'zpool detach=E2=80=99=



Another related question.  I can (and just did) =E2=80=9Cglabel =
create=E2=80=9D so the =E2=80=9Cnew=E2=80=9D disk
that is called da10, but is still in chassis drive slot 3 where da3 =
_used_ to be.
Now the new disk can be descirbed as =E2=80=9Clabel/drive03=E2=80=9D.  =
This is what I do with
ext4 labels on Linux machines on similar hardware.

Can I add labels to the ZFS devices without hurting them?  glabel says =
it
overwrites the last sector with label information it manages.  Is there =
space
on a disk that is used entirely for a ZFS device?

      - Chris



From nobody Fri Sep  6 18:39:24 2024
X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0lQ25XG4z5VSvb
	for ; Fri, 06 Sep 2024 18:39:38 +0000 (UTC)
	(envelope-from morganw@gmail.com)
Received: from mail-yw1-x112a.google.com (mail-yw1-x112a.google.com [IPv6:2607:f8b0:4864:20::112a])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "WR4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4X0lQ13TTtz4gWL
	for ; Fri,  6 Sep 2024 18:39:37 +0000 (UTC)
	(envelope-from morganw@gmail.com)
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=gmail.com header.s=20230601 header.b=AY0FLU3R;
	dmarc=pass (policy=none) header.from=gmail.com;
	spf=pass (mx1.freebsd.org: domain of morganw@gmail.com designates 2607:f8b0:4864:20::112a as permitted sender) smtp.mailfrom=morganw@gmail.com
Received: by mail-yw1-x112a.google.com with SMTP id 00721157ae682-699ac6dbf24so21156597b3.3
        for ; Fri, 06 Sep 2024 11:39:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1725647976; x=1726252776; darn=freebsd.org;
        h=content-transfer-encoding:mime-version:message-id:references
         :in-reply-to:user-agent:subject:to:from:date:from:to:cc:subject:date
         :message-id:reply-to;
        bh=aBzhViiCZwByufrm2wgneGHNauaAwpmiF9/bep+2qfw=;
        b=AY0FLU3RktYrcO9uNqgxEuwozSuvkGdm8VeIcVXzVnyxfykmyhX033AfcXxLyrqOaW
         QD5HVgYE5FylmoAVMmsg90TDcdYZhEBSeQ/eK+FRBHdWDk6TTTY4x9u3pwYO818ogPFP
         tbydwttPH0+7QAMsEzCPjSEYHIpa/Vq5T9Fy0Q1J8/D+YBnE5x7vcUCceKjWI2mCCXsK
         NlEgMG1YmD+ebAXWLtZOMD6xZCdcJkFO8kVXO/fBgTHeiCyofPfq6rBQsLn7LBt+/V0N
         BI7/9acmMg8kj3UzRQ0D3ScVVOTymwP6lRH3PnggnA2xcGk8VmX3uvBuX0NdsTrjGw/6
         kzrA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1725647976; x=1726252776;
        h=content-transfer-encoding:mime-version:message-id:references
         :in-reply-to:user-agent:subject:to:from:date:x-gm-message-state:from
         :to:cc:subject:date:message-id:reply-to;
        bh=aBzhViiCZwByufrm2wgneGHNauaAwpmiF9/bep+2qfw=;
        b=E/4VWApcmc3+3qfITrkOlkZUBcmUZy4bDVcXHk4grNY9P0SgHroW8hGIFy92UwT9mn
         +wrkzwW6bA7mylRsOcXbM8wE/U4x788l2ubVUtBZ4sOgKyRtv7UGF9i3/RZz/j6F2IxB
         4ENHrl3ADe58XNwsviwo67PsyJpi5ew6wxDrv3b13E5FX9O7dJPrHp0m2IFuAbTBOMPT
         nZom7AAvWXvmSvRgzQ3joAUAGr2WqcR0xClRhifMnfjI9IZBO4OChnTy1aL5Ij0fxEaE
         OBZcUyvj2aRsxb0PZmzV+Qa2lMYBa2o8w/3gIK/4CPsGSlhvZBzXQFZieQbEcU18TUdx
         l2/g==
X-Gm-Message-State: AOJu0YwXT+ZRMsP6xkyIivQoJhycAXU33E0hXxnBuo/NTKfHaq/2x+4l
	89a1D/hMjn8J/bxM9jux2h2mh42ZV5JQqtBvRB0JR4l9bAo8Zs90XFBfgsPk
X-Google-Smtp-Source: AGHT+IGWhbLUKN3IWoT7P9YXNe5XrqT4+7F9V/sOlEwZidiJ6L+EhXH7Wv5n/HK2L6vbHJOk4QqHlg==
X-Received: by 2002:a05:690c:438e:b0:672:e49d:430e with SMTP id 00721157ae682-6db44dc5400mr29536337b3.15.1725647975991;
        Fri, 06 Sep 2024 11:39:35 -0700 (PDT)
Received: from ?IPv6:::1? ([2600:381:642f:e2a:d8f2:e3e8:b579:a70b])
        by smtp.gmail.com with ESMTPSA id 00721157ae682-6db56337c15sm934087b3.43.2024.09.06.11.39.34
        for 
        (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
        Fri, 06 Sep 2024 11:39:35 -0700 (PDT)
Date: Fri, 06 Sep 2024 13:39:24 -0500
From: Wes Morgan 
To: freebsd-fs@freebsd.org
Subject: Re: Unable to replace drive in raidz1
User-Agent: K-9 Mail for Android
In-Reply-To: <29003A7C-745D-4A06-8558-AE64310813EA@distal.com>
References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com>     <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com>  <0CF1E2D7-6C82-4A8B-82C3-A5BF1ED939CF@distal.com>  <29003A7C-745D-4A06-8558-AE64310813EA@distal.com>
Message-ID: <42346193-AD06-4D26-B0C6-4392953D21A3@gmail.com>
List-Id: Filesystems 
List-Archive: https://lists.freebsd.org/archives/freebsd-fs
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-fs@FreeBSD.org
MIME-Version: 1.0
Content-Type: text/plain;
 charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spamd-Bar: ---
X-Spamd-Result: default: False [-3.92 / 15.00];
	NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	NEURAL_HAM_LONG(-1.00)[-1.000];
	NEURAL_HAM_SHORT(-0.92)[-0.921];
	DMARC_POLICY_ALLOW(-0.50)[gmail.com,none];
	R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36];
	R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601];
	MIME_GOOD(-0.10)[text/plain];
	RCVD_TLS_LAST(0.00)[];
	TO_MATCH_ENVRCPT_ALL(0.00)[];
	ARC_NA(0.00)[];
	MIME_TRACE(0.00)[0:+];
	RCPT_COUNT_ONE(0.00)[1];
	DWL_DNSWL_NONE(0.00)[gmail.com:dkim];
	FREEMAIL_FROM(0.00)[gmail.com];
	FROM_HAS_DN(0.00)[];
	ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US];
	FREEMAIL_ENVFROM(0.00)[gmail.com];
	TO_DN_NONE(0.00)[];
	RCVD_COUNT_TWO(0.00)[2];
	FROM_EQ_ENVFROM(0.00)[];
	DKIM_TRACE(0.00)[gmail.com:+];
	PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org];
	MID_RHS_MATCH_FROM(0.00)[];
	MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org];
	RCVD_VIA_SMTP_AUTH(0.00)[];
	RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::112a:from]
X-Rspamd-Queue-Id: 4X0lQ13TTtz4gWL



On September 6, 2024 1:21:06 PM CDT, Chris Ross  wrote:
>
>> On Sep 6, 2024, at 14:08, mike tancsa  wrote:
>>=20
>> On 9/6/2024 2:06 PM, Chris Ross wrote:
>>> How can I map the diskid=E2=80=99s listed to the underlying device?eit=
her by serial number or da#=E2=80=A6
>>>=20
>> What does=20
>> glabel status
>
>That shows labels for many drives, though not da1 and da2, the
>remaining members of zraid1-0=2E  Intersting=2E  But, I hope the below
>Is the fix=2E=2E=2E
>
>> On Sep 6, 2024, at 14:10, Alan Somers  wrote:
>>=20
>> Ahh, this means that there are two different vdevs that can be
>> described by "da3"=2E  You can still refer to them unambiguously by gui=
d
>> though=2E  Do "zpool status -g" to find the guid of the disk that you
>> want to replace, and then do "zpool replace  /dev/da10=E2=80=9D
>
>
>Ahh, okay=2E  That makes sense=2E  I have ever only known how to replace =
things
>using the key that "zpool status=E2=80=9D shows=2E  Thanks for that!
>
>Oh=2E  Trying, that doesn=E2=80=99t work either=2E  :-/
>
>NAME                      STATE     READ WRITE CKSUM
>tank                      DEGRADED     0     0     0
> 16506780107187041124    DEGRADED     0     0     0
>   9127016430593660128   FAULTED      0     0     0  external device faul=
t
>   4094297345166589692   ONLINE       0     0     0
>   17850258180603290288  ONLINE       0     0     0
>[=E2=80=A6]
>% sudo zpool replace tank 9127016430593660128 /dev/da10
>cannot replace 9127016430593660128 with /dev/da10: already in replacing/s=
pare config; wait for completion or use 'zpool detach=E2=80=99
>% sudo zpool replace tank 9127016430593660128 diskid/DISK-ZGG0A2PA
>cannot replace 9127016430593660128 with diskid/DISK-ZGG0A2PA: already in =
replacing/spare config; wait for completion or use 'zpool detach'
>
>Tried with /dev/da10, and the diskid for da10 reported by glabel status=
=2E
>
>         - Chris
>

You should make the changes to your /boot/loader=2Econf as suggested earli=
er by Freddie Cash and reboot=2E This will eliminate all the confusion with=
 diskid=2E Then run "zpool clear", which, if da3 is still online and not co=
mpletely dead, the pool should come out of the faulted state=2E Check zpool=
 status to look for this alleged replacement in progress=2E If it is truly =
trying to replace a device, it should show up in zpool status with the actu=
al device, or the guid if it can't find the device=2E

If you have initiated a replace, and the replacing disk has now been "lost=
" or unlabeled, you are in a bind=2E I ran into this problem many years ago=
, and I thought it was fixed, but the bug was called something like "can't =
replace a replacing vdev"=2E I ultimately solved my problem by manually edi=
ting a fake vdev to have the same guid as the missing device, restarting th=
e replace and then canceling it before zfs realized it was fake=2E But, I a=
m almost certain that zpool cancel can do this now, with the guid=2E

If da10 has a label that says it is in the pool, it is probably the "repla=
cing" vdev and should be picked up=2E=2E=2E


From nobody Fri Sep  6 18:44:04 2024
X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0lWG0vFBz5VTHJ
	for ; Fri, 06 Sep 2024 18:44:10 +0000 (UTC)
	(envelope-from morganw@gmail.com)
Received: from mail-yw1-x1134.google.com (mail-yw1-x1134.google.com [IPv6:2607:f8b0:4864:20::1134])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "WR4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4X0lWF017Kz4hrl
	for ; Fri,  6 Sep 2024 18:44:09 +0000 (UTC)
	(envelope-from morganw@gmail.com)
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=gmail.com header.s=20230601 header.b=Jad41lJ3;
	dmarc=pass (policy=none) header.from=gmail.com;
	spf=pass (mx1.freebsd.org: domain of morganw@gmail.com designates 2607:f8b0:4864:20::1134 as permitted sender) smtp.mailfrom=morganw@gmail.com
Received: by mail-yw1-x1134.google.com with SMTP id 00721157ae682-6da395fb97aso23635097b3.0
        for ; Fri, 06 Sep 2024 11:44:08 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1725648248; x=1726253048; darn=freebsd.org;
        h=content-transfer-encoding:mime-version:message-id:references
         :in-reply-to:user-agent:subject:to:from:date:from:to:cc:subject:date
         :message-id:reply-to;
        bh=YthFJtUP+UlQtyVeFVmwcWHNnCifRp2cwCZs3i7PvHk=;
        b=Jad41lJ3UlPRMiMhWapdzvCn3NQLJEMbxBcP8bcn8nRDn4hRA11Ei2f+nts9+zmePQ
         4qANPWgqYsFVTrORSaQ6At7PNKBYX06iHMVV10XY5Waamz7BF+c37RjZEsfkwcuwAuwe
         liCtQhpx5LEMquufeKOgcfe+Zw4Y3Q41CpuoyR+VM4tOo/WFr7+ZUVK2yylxrfGRtGDh
         sQVJGipKBba5CvdDIbvzYbGx19CaLqbJ2HrVeKkXZhabGaRIraz45vgVe+J9+TlwatU0
         nLnPtoYuEALDIV7lOaxDE1o/TjOEAlLaseTgF5PVvq1f16tn9fMItvLEWBPQAR/7mfU8
         2nwg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1725648248; x=1726253048;
        h=content-transfer-encoding:mime-version:message-id:references
         :in-reply-to:user-agent:subject:to:from:date:x-gm-message-state:from
         :to:cc:subject:date:message-id:reply-to;
        bh=YthFJtUP+UlQtyVeFVmwcWHNnCifRp2cwCZs3i7PvHk=;
        b=H35qIAYLuhiA4EtUu0wGSXgNpx2qnaFop4OG6UMOnTisHRE/qtGfThCzOl+M0fYHtr
         Dh6MYQRa+rqsF2n0BLiQC/NYge7hrxlfG2pNGmBgL2ZvUCXmIZTRVUK9Ekl2Daaanx7J
         i4gifX6Ow1iRHVxsBRucJJcztShnI/Q8K85GNICeER3dyUJskteM52vMCEqp1D+J/of3
         c3IzM3mTlxjPBfIcHroqyHRVgpajqtHSX8mpWSZJb2TCBtOwJagJpUv7ZjuwSI3cVf/E
         1VQXNSv8Z8D2OBOd9CNugN0XAhR8pZYW5tQR/tBM4/78BKqcf8w5LwQIma8Apdy0VK6w
         dKHg==
X-Gm-Message-State: AOJu0YyDcP1eaJfJEXJ6Y+dQ0M5aqrvGDZBCXfLriLLqnheCzvHo3ogH
	tBTPGjEiIIe0DFl+jyrhOlppYGZCqqs4xpkrawwYrYuzYIeHyHKAvIy7kNUM
X-Google-Smtp-Source: AGHT+IF+ncNmGvoHAMi2IPj25rNic10nxjSSUnzbBrd/GM2T5FIOTcKTOBDDUVKu++W1DYZaMTS1sw==
X-Received: by 2002:a05:690c:660c:b0:669:e266:2c55 with SMTP id 00721157ae682-6db44d62ab7mr45244827b3.5.1725648247898;
        Fri, 06 Sep 2024 11:44:07 -0700 (PDT)
Received: from ?IPv6:::1? ([2600:381:642f:e2a:d8f2:e3e8:b579:a70b])
        by smtp.gmail.com with ESMTPSA id 00721157ae682-6db565a86d6sm927657b3.78.2024.09.06.11.44.06
        for 
        (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
        Fri, 06 Sep 2024 11:44:07 -0700 (PDT)
Date: Fri, 06 Sep 2024 13:44:04 -0500
From: Wes Morgan 
To: freebsd-fs@freebsd.org
Subject: Re: Unable to replace drive in raidz1
User-Agent: K-9 Mail for Android
In-Reply-To: <780A9427-C9BA-481F-9139-992B323F863B@distal.com>
References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com>     <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com>  <0CF1E2D7-6C82-4A8B-82C3-A5BF1ED939CF@distal.com>  <29003A7C-745D-4A06-8558-AE64310813EA@distal.com> <780A9427-C9BA-481F-9139-992B323F863B@distal.com>
Message-ID: 
List-Id: Filesystems 
List-Archive: https://lists.freebsd.org/archives/freebsd-fs
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-fs@FreeBSD.org
MIME-Version: 1.0
Content-Type: text/plain;
 charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spamd-Bar: ---
X-Spamd-Result: default: False [-3.94 / 15.00];
	NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	NEURAL_HAM_LONG(-1.00)[-1.000];
	NEURAL_HAM_SHORT(-0.94)[-0.943];
	DMARC_POLICY_ALLOW(-0.50)[gmail.com,none];
	R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c];
	R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601];
	MIME_GOOD(-0.10)[text/plain];
	RCVD_TLS_LAST(0.00)[];
	TO_MATCH_ENVRCPT_ALL(0.00)[];
	RCPT_COUNT_ONE(0.00)[1];
	MIME_TRACE(0.00)[0:+];
	ARC_NA(0.00)[];
	FREEMAIL_ENVFROM(0.00)[gmail.com];
	FREEMAIL_FROM(0.00)[gmail.com];
	FROM_HAS_DN(0.00)[];
	ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US];
	DWL_DNSWL_NONE(0.00)[gmail.com:dkim];
	TO_DN_NONE(0.00)[];
	RCVD_COUNT_TWO(0.00)[2];
	FROM_EQ_ENVFROM(0.00)[];
	DKIM_TRACE(0.00)[gmail.com:+];
	PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org];
	MID_RHS_MATCH_FROM(0.00)[];
	MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org];
	RCVD_VIA_SMTP_AUTH(0.00)[];
	RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1134:from]
X-Rspamd-Queue-Id: 4X0lWF017Kz4hrl



On September 6, 2024 1:35:22 PM CDT, Chris Ross  wrote:
>
>
>> On Sep 6, 2024, at 14:21, Chris Ross  wrote=
:
>> NAME                      STATE     READ WRITE CKSUM
>> tank                      DEGRADED     0     0     0
>> 16506780107187041124    DEGRADED     0     0     0
>>   9127016430593660128   FAULTED      0     0     0  external device fau=
lt
>>   4094297345166589692   ONLINE       0     0     0
>>   17850258180603290288  ONLINE       0     0     0
>> [=E2=80=A6]
>> % sudo zpool replace tank 9127016430593660128 /dev/da10
>> cannot replace 9127016430593660128 with /dev/da10: already in replacing=
/spare config; wait for completion or use 'zpool detach=E2=80=99
>> % sudo zpool replace tank 9127016430593660128 diskid/DISK-ZGG0A2PA
>> cannot replace 9127016430593660128 with diskid/DISK-ZGG0A2PA: already i=
n replacing/spare config; wait for completion or use 'zpool detach=E2=80=99
>
>
>Another related question=2E  I can (and just did) =E2=80=9Cglabel create=
=E2=80=9D so the =E2=80=9Cnew=E2=80=9D disk
>that is called da10, but is still in chassis drive slot 3 where da3 _used=
_ to be=2E
>Now the new disk can be descirbed as =E2=80=9Clabel/drive03=E2=80=9D=2E  =
This is what I do with
>ext4 labels on Linux machines on similar hardware=2E
>
>Can I add labels to the ZFS devices without hurting them?  glabel says it
>overwrites the last sector with label information it manages=2E  Is there=
 space
>on a disk that is used entirely for a ZFS device?

No, do not do this=2E Do not do anything at all to a device if the entire =
device will be assigned to a pool=2E=20


From nobody Fri Sep  6 18:54:32 2024
X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0llX2p0pz5VVJY
	for ; Fri, 06 Sep 2024 18:54:48 +0000 (UTC)
	(envelope-from cross+freebsd@relay.distal.com)
Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(Client did not present a certificate)
	by mx1.freebsd.org (Postfix) with ESMTPS id 4X0llX1123z4jsT
	for ; Fri,  6 Sep 2024 18:54:47 +0000 (UTC)
	(envelope-from cross+freebsd@relay.distal.com)
Authentication-Results: mx1.freebsd.org;
	none
dkim-signature: v=1; a=rsa-sha256; d=relay.distal.com; s=mail;
	c=relaxed/relaxed; q=dns/txt; h=From:Subject:Date:Message-ID:To:CC:MIME-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References;
	bh=6ahwzrwrw5gza7THBUUTSaiuPSVxdU6Qlwyv3witcGA=;
	b=yZc3KZ5k63CrloPlqt9nlOTEPIlZn+9NtrqWjh98NL+5sReSyltmPdJPuCOJCL9l0AiHH5ljfdoGB2Wh9Pir511aAE9BXpoOm8h3V+ilD236dji81LtO/sJaZ77PejqZE19GuQzf3oGw7CGmeVGfA/y252ZSrom57LnngL8FzbobOaLdfjSAu38mjEoGQwaY/V7vi2D+oQGPiQSynhtqa774lJMwipR7CjdfP707gHtP3KFo5vIYbK6bGU
	2JHuSsYG2NCaZgxTKkJ2OTY3z7T6e/yzUqKCwk1Kr5TgrqyG5r3wMyYnS6lsjAsB/qijR+xw23X2anQ03oArJrqmH/OA==
Received: from mail.distal.com (pool-108-51-233-124.washdc.fios.verizon.net [108.51.233.124])
 by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305
 bits=256) ; Fri, 6 Sep 2024 18:54:45 +0000
Received: from smtpclient.apple ( [2001:420:c0c4:1001::9f]) by tristain.distal.com
 (OpenSMTPD) with ESMTPSA id 6cfd1947 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO);
 Fri, 6 Sep 2024 14:54:43 -0400 (EDT)
Content-Type: text/plain; charset=utf-8
List-Id: Filesystems 
List-Archive: https://lists.freebsd.org/archives/freebsd-fs
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-fs@FreeBSD.org
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3776.700.51\))
Subject: Re: Unable to replace drive in raidz1
From: Chris Ross 
In-Reply-To: <42346193-AD06-4D26-B0C6-4392953D21A3@gmail.com>
Date: Fri, 6 Sep 2024 14:54:32 -0400
Cc: freebsd-fs@freebsd.org
Content-Transfer-Encoding: quoted-printable
Message-Id: 
References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> 
  
  <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com>
  <0CF1E2D7-6C82-4A8B-82C3-A5BF1ED939CF@distal.com>
  <29003A7C-745D-4A06-8558-AE64310813EA@distal.com>
 <42346193-AD06-4D26-B0C6-4392953D21A3@gmail.com>
To: Wes Morgan 
X-Mailer: Apple Mail (2.3776.700.51)
X-Spamd-Bar: ----
X-Rspamd-Pre-Result: action=no action;
	module=replies;
	Message is reply to one we originated
X-Spamd-Result: default: False [-4.00 / 15.00];
	REPLY(-4.00)[];
	TAGGED_FROM(0.00)[freebsd];
	ASN(0.00)[asn:3842, ipnet:168.235.92.0/22, country:US]
X-Rspamd-Queue-Id: 4X0llX1123z4jsT



> On Sep 6, 2024, at 14:39, Wes Morgan  wrote:
>=20
>=20
> You should make the changes to your /boot/loader.conf as suggested =
earlier by Freddie Cash and reboot. This will eliminate all the =
confusion with diskid. Then run "zpool clear", which, if da3 is still =
online and not completely dead, the pool should come out of the faulted =
state. Check zpool status to look for this alleged replacement in =
progress. If it is truly trying to replace a device, it should show up =
in zpool status with the actual device, or the guid if it can't find the =
device.

I saw and appreiciated that response, but didn=E2=80=99t respond on that =
thread because I don=E2=80=99t _want_ to turn all of those things off.  =
At least, I don=E2=80=99t want to refer to everything by the =
auto-numbered da# that I think that will cause.  And, Freddie, your =
comment about GPT partition labels I think doesn=E2=80=99t apply because =
I don=E2=80=99t have GPT on my disks.  Just all one big ZFS device.  =
This is why I=E2=80=99m looking at glabel=E2=80=99s generic labeling =
now.

The former da3 is off-line, out of the chassis.  I replaced a disk in a =
full chassis, having them both online at the same time is not possible.  =
That drive in ZFS=E2=80=99s mind is only faulted because I tried =
=E2=80=9Czpool offline -f=E2=80=9D on it to see if that helped.

> If you have initiated a replace, and the replacing disk has now been =
"lost" or unlabeled, you are in a bind. I ran into this problem many =
years ago, and I thought it was fixed, but the bug was called something =
like "can't replace a replacing vdev". I ultimately solved my problem by =
manually editing a fake vdev to have the same guid as the missing =
device, restarting the replace and then canceling it before zfs realized =
it was fake. But, I am almost certain that zpool cancel can do this now, =
with the guid.

I didn=E2=80=99t initiate a replace until after the disks were =
physically changed.  Although in this conversation realize that things =
likely got confused by the replacement in the kernel=E2=80=99s mind of =
da3 with what used to be da4.  :-/

> If da10 has a label that says it is in the pool, it is probably the =
"replacing" vdev and should be picked up=E2=80=A6

Da10, now also /dev/label/drive03, seems to think it=E2=80=99s in the =
pool somewhere, according to zdb -l.
But I=E2=80=99m not sure if this helps.  And, following your other =
message saying I shouldn=E2=80=99t put labels
on disks that are to be used in their entirety as ZFS devices, I=E2=80=99v=
e deleted that label and
zlabelclear=E2=80=99d this device now.  (since the zfs label still had =
the /dev/label/ path in it)

                       - Chris



From nobody Fri Sep  6 19:16:39 2024
X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0mDs1wkJz5VXTp
	for ; Fri, 06 Sep 2024 19:16:45 +0000 (UTC)
	(envelope-from morganw@gmail.com)
Received: from mail-yb1-xb34.google.com (mail-yb1-xb34.google.com [IPv6:2607:f8b0:4864:20::b34])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "WR4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4X0mDr2nmXz4ms7
	for ; Fri,  6 Sep 2024 19:16:44 +0000 (UTC)
	(envelope-from morganw@gmail.com)
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=gmail.com header.s=20230601 header.b=nI94q35h;
	dmarc=pass (policy=none) header.from=gmail.com;
	spf=pass (mx1.freebsd.org: domain of morganw@gmail.com designates 2607:f8b0:4864:20::b34 as permitted sender) smtp.mailfrom=morganw@gmail.com
Received: by mail-yb1-xb34.google.com with SMTP id 3f1490d57ef6-e1d4368ad91so291233276.0
        for ; Fri, 06 Sep 2024 12:16:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1725650203; x=1726255003; darn=freebsd.org;
        h=content-transfer-encoding:mime-version:message-id:references
         :in-reply-to:user-agent:subject:cc:to:from:date:from:to:cc:subject
         :date:message-id:reply-to;
        bh=YQRPyN5Lr6f5iBwMT31HjdHtgVfmm0u6a+0rFTb7n54=;
        b=nI94q35hxKsRs3AiLT4kWQbWt/yQl+9VyPQ9Od9vzsAyDLbialMKs83RpLrKtgLejI
         LpjM5+oGOjizIGNwFS+X8vEwU6LJqdPRCJhlK4OvmZu6pgsimDvwVN77kI6HNNAf1n+Q
         p9DdabaVaxfqu9PbCFbLWWV8ac2fYHMUQRzZ1DBGfLFx4SlwDxJq5BEySxPY7HYPdeVh
         oSbGf3nOAdO6ViWsjNpuiizbxQM4rK98GQShkw4v3opyj/jgEtfX3ci2wgOVvCAKKHXI
         dT6CYSn87031Dyq2XOa85qNQkCrtfHdPPA3nIbx52ERBbjGcm3ibS5bSXHnxUH8REkrG
         NpzA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1725650203; x=1726255003;
        h=content-transfer-encoding:mime-version:message-id:references
         :in-reply-to:user-agent:subject:cc:to:from:date:x-gm-message-state
         :from:to:cc:subject:date:message-id:reply-to;
        bh=YQRPyN5Lr6f5iBwMT31HjdHtgVfmm0u6a+0rFTb7n54=;
        b=Fp8xOT19EIvmJFxLTiwGSgEJ4H9muy6vS6V1WIPy+3UUpDXQ/nt6PnokD0+TVPYEDe
         y9v1oYXbimNUcptH2UlNxZQbDSIBz2Gempt/lAlDRCwy62znauWnMJ+gpyNiCiZcprqw
         5MfVBXK6U9i6ny5+E9yLr9q6YU0Savzjs9J01e6PSvXMB09J6DTq+uDtciima+JBeIvT
         xlITXqe10Wy/g1DsZAj58NCl29gNyyKsbKpqGdYnPPqH2nf3Njy6ybSA8GjgwhH/wF1n
         DswNSCU6KMsk+OwBlNoKK1iDqZEXgv5ERyq1CfMdS1CdrQOEf7p1Dty4rGLIePX+iJ96
         yDSg==
X-Gm-Message-State: AOJu0YyqdfeCMlCtsGVb/SZF2umQJd+buH9M36aPFV80UHuuzc6sx/Lm
	lzajMJM5jpFKMDK63JszNKPo2jZraatKO/Rde8KSkL1TrcWgn7Ln
X-Google-Smtp-Source: AGHT+IFT03FCDG+mHKAWoxKc3eVS5aykd/F+XQMuqKt7BNrShJYZv3DcETwWZ4/q5rGlipSflm1w0g==
X-Received: by 2002:a05:690c:30c:b0:6d6:b852:ed2e with SMTP id 00721157ae682-6db25f4a488mr94546727b3.2.1725650203451;
        Fri, 06 Sep 2024 12:16:43 -0700 (PDT)
Received: from ?IPv6:::1? ([2600:381:642f:e2a:d8f2:e3e8:b579:a70b])
        by smtp.gmail.com with ESMTPSA id 00721157ae682-6db56337dd3sm1056077b3.42.2024.09.06.12.16.41
        (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
        Fri, 06 Sep 2024 12:16:43 -0700 (PDT)
Date: Fri, 06 Sep 2024 14:16:39 -0500
From: Wes Morgan 
To: Chris Ross 
CC: freebsd-fs@freebsd.org
Subject: Re: Unable to replace drive in raidz1
User-Agent: K-9 Mail for Android
In-Reply-To: 
References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com>     <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com>  <0CF1E2D7-6C82-4A8B-82C3-A5BF1ED939CF@distal.com>  <29003A7C-745D-4A06-8558-AE64310813EA@distal.com> <42346193-AD06-4D26-B0C6-4392953D21A3@gmail.com> 
Message-ID: 
List-Id: Filesystems 
List-Archive: https://lists.freebsd.org/archives/freebsd-fs
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-fs@FreeBSD.org
MIME-Version: 1.0
Content-Type: text/plain;
 charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spamd-Bar: ---
X-Spamd-Result: default: False [-4.00 / 15.00];
	NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	NEURAL_HAM_LONG(-1.00)[-1.000];
	NEURAL_HAM_SHORT(-1.00)[-1.000];
	DMARC_POLICY_ALLOW(-0.50)[gmail.com,none];
	R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601];
	R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36];
	MIME_GOOD(-0.10)[text/plain];
	RCVD_TLS_LAST(0.00)[];
	FROM_HAS_DN(0.00)[];
	ARC_NA(0.00)[];
	MIME_TRACE(0.00)[0:+];
	FREEMAIL_ENVFROM(0.00)[gmail.com];
	RCPT_COUNT_TWO(0.00)[2];
	FREEMAIL_FROM(0.00)[gmail.com];
	TO_DN_SOME(0.00)[];
	DKIM_TRACE(0.00)[gmail.com:+];
	ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US];
	RCVD_VIA_SMTP_AUTH(0.00)[];
	TO_MATCH_ENVRCPT_SOME(0.00)[];
	RCVD_COUNT_TWO(0.00)[2];
	FROM_EQ_ENVFROM(0.00)[];
	DWL_DNSWL_NONE(0.00)[gmail.com:dkim];
	PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org];
	MID_RHS_MATCH_FROM(0.00)[];
	MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org];
	TAGGED_RCPT(0.00)[freebsd];
	RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::b34:from]
X-Rspamd-Queue-Id: 4X0mDr2nmXz4ms7



On September 6, 2024 1:54:32 PM CDT, Chris Ross  wrote:
>
>
>> On Sep 6, 2024, at 14:39, Wes Morgan  wrote:
>>=20
>>=20
>> You should make the changes to your /boot/loader=2Econf as suggested ea=
rlier by Freddie Cash and reboot=2E This will eliminate all the confusion w=
ith diskid=2E Then run "zpool clear", which, if da3 is still online and not=
 completely dead, the pool should come out of the faulted state=2E Check zp=
ool status to look for this alleged replacement in progress=2E If it is tru=
ly trying to replace a device, it should show up in zpool status with the a=
ctual device, or the guid if it can't find the device=2E
>
>I saw and appreiciated that response, but didn=E2=80=99t respond on that =
thread because I don=E2=80=99t _want_ to turn all of those things off=2E  A=
t least, I don=E2=80=99t want to refer to everything by the auto-numbered d=
a# that I think that will cause=2E  And, Freddie, your comment about GPT pa=
rtition labels I think doesn=E2=80=99t apply because I don=E2=80=99t have G=
PT on my disks=2E  Just all one big ZFS device=2E  This is why I=E2=80=99m =
looking at glabel=E2=80=99s generic labeling now=2E

You probably don't want that=2E You will have to use the glabel dev, which=
 will not be the same size as your other devices=2E IIRC you have no contro=
l over what device node the system finds first for the pool=2E Even if you =
use GPT labels, the daXpY device will still exist=2E=20

>The former da3 is off-line, out of the chassis=2E  I replaced a disk in a=
 full chassis, having them both online at the same time is not possible=2E =
 That drive in ZFS=E2=80=99s mind is only faulted because I tried =E2=80=9C=
zpool offline -f=E2=80=9D on it to see if that helped=2E

It sounds like you have replaced the wrong device=2E Check the "zpool hist=
ory" to see what you did=2E=20

In your earlier message, three devices were shown in each raidz, when what=
 you should be seeing is that one raidz has an offline device identified by=
 guid and maybe "was /dev/da3" that is being replaced, along with the repla=
cement device=2E I don't see any of that=2E=20

>> If you have initiated a replace, and the replacing disk has now been "l=
ost" or unlabeled, you are in a bind=2E I ran into this problem many years =
ago, and I thought it was fixed, but the bug was called something like "can=
't replace a replacing vdev"=2E I ultimately solved my problem by manually =
editing a fake vdev to have the same guid as the missing device, restarting=
 the replace and then canceling it before zfs realized it was fake=2E But, =
I am almost certain that zpool cancel can do this now, with the guid=2E
>
>I didn=E2=80=99t initiate a replace until after the disks were physically=
 changed=2E  Although in this conversation realize that things likely got c=
onfused by the replacement in the kernel=E2=80=99s mind of da3 with what us=
ed to be da4=2E  :-/

This is why your zpool history will be helpful=2E What did you actually tr=
y to replace, and what did you mean to replace=2E=20


>> If da10 has a label that says it is in the pool, it is probably the "re=
placing" vdev and should be picked up=E2=80=A6
>
>Da10, now also /dev/label/drive03, seems to think it=E2=80=99s in the poo=
l somewhere, according to zdb -l=2E
>But I=E2=80=99m not sure if this helps=2E  And, following your other mess=
age saying I shouldn=E2=80=99t put labels
>on disks that are to be used in their entirety as ZFS devices, I=E2=80=99=
ve deleted that label and
>zlabelclear=E2=80=99d this device now=2E  (since the zfs label still had =
the /dev/label/ path in it)


From nobody Fri Sep  6 19:34:36 2024
X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0mdm6wVwz5VZd3
	for ; Fri, 06 Sep 2024 19:34:52 +0000 (UTC)
	(envelope-from cross+freebsd@relay.distal.com)
Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(Client did not present a certificate)
	by mx1.freebsd.org (Postfix) with ESMTPS id 4X0mdm3ZfLz4rdG
	for ; Fri,  6 Sep 2024 19:34:52 +0000 (UTC)
	(envelope-from cross+freebsd@relay.distal.com)
Authentication-Results: mx1.freebsd.org;
	none
dkim-signature: v=1; a=rsa-sha256; d=relay.distal.com; s=mail;
	c=relaxed/relaxed; q=dns/txt; h=From:Subject:Date:Message-ID:To:CC:MIME-Version:Content-Type:In-Reply-To:References;
	bh=aQdvkE/DXmOzK1hMVi0fgM5rPPI2Opkqb/rWZV/hS7w=;
	b=sGPsB1mWfsTqkfZPlvtocU2umbPNExm8fzrX9HjN1pycQLQS5s7poYLJrEcYMtK6qx4mSvWKny4MaeoQn7xHyfCBeN8hh3VCJlMCYXwJ598HpZ9/U0EFQtjFNOFpdpJUT3ems3QXTYIrsURBCe6KOfFFh/5Ejb8vpj7ezsWwuXgjelp9ZlSme5JAL/M7VMeTwYT1M7uwGkxUWY0ZEAp7oahdfkjciIWHdrrWVjWbmok+xvRt99n2q21oB4
	aaf4jdsfgSfx4RDQPwJ5kwHn9LrgQQxmebmKvFBdTVC1fXMfd9FNNos1wI8Z9OKy8zITDT2+yUNN8k1zwl4K5SusPgRQ==
Received: from mail.distal.com (pool-108-51-233-124.washdc.fios.verizon.net [108.51.233.124])
 by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305
 bits=256) ; Fri, 6 Sep 2024 19:34:50 +0000
Received: from smtpclient.apple ( [2001:420:c0c4:1001::9f]) by tristain.distal.com
 (OpenSMTPD) with ESMTPSA id 3fc5ded1 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO);
 Fri, 6 Sep 2024 15:34:48 -0400 (EDT)
From: Chris Ross 
Message-Id: 
Content-Type: multipart/alternative; boundary="Apple-Mail=_26064149-B678-4693-B259-B3840725655C"
List-Id: Filesystems 
List-Archive: https://lists.freebsd.org/archives/freebsd-fs
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-fs@FreeBSD.org
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3776.700.51\))
Subject: Re: Unable to replace drive in raidz1
Date: Fri, 6 Sep 2024 15:34:36 -0400
In-Reply-To: 
Cc: freebsd-fs@freebsd.org
To: Wes Morgan 
References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> 
  
  <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com>
  <0CF1E2D7-6C82-4A8B-82C3-A5BF1ED939CF@distal.com>
  <29003A7C-745D-4A06-8558-AE64310813EA@distal.com>
 <42346193-AD06-4D26-B0C6-4392953D21A3@gmail.com> 
 
X-Mailer: Apple Mail (2.3776.700.51)
X-Spamd-Bar: ----
X-Rspamd-Pre-Result: action=no action;
	module=replies;
	Message is reply to one we originated
X-Spamd-Result: default: False [-4.00 / 15.00];
	REPLY(-4.00)[];
	TAGGED_FROM(0.00)[freebsd];
	ASN(0.00)[asn:3842, ipnet:168.235.92.0/22, country:US]
X-Rspamd-Queue-Id: 4X0mdm3ZfLz4rdG


--Apple-Mail=_26064149-B678-4693-B259-B3840725655C
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=utf-8



> On Sep 6, 2024, at 15:16, Wes Morgan  wrote:
>=20
> You probably don't want that. You will have to use the glabel dev, =
which will not be the same size as your other devices. IIRC you have no =
control over what device node the system finds first for the pool. Even =
if you use GPT labels, the daXpY device will still exist.=20

Right.  But if I don=E2=80=99t _use_ those device names, it won=E2=80=99t =
matter.  If I use /dev/label/foo, or /dev/gpt/foo, I=E2=80=99ll just =
always use those.  I just did that with the ufs disk I have since it =
moved names, now it=E2=80=99s "/dev/ufs/drive12=E2=80=9D in /etc/fstab =
et al.

I want to have some sort of label.  I=E2=80=99d rather not have to add a =
partitioning scheme to the disk if I know I=E2=80=99m just going to use =
the whole disk just to get a label, but I suppose if I have to I can.  =
Though I=E2=80=99d have to do it one disk at a time.  :-)

>=20
>> The former da3 is off-line, out of the chassis.  I replaced a disk in =
a full chassis, having them both online at the same time is not =
possible.  That drive in ZFS=E2=80=99s mind is only faulted because I =
tried =E2=80=9Czpool offline -f=E2=80=9D on it to see if that helped.
>=20
> It sounds like you have replaced the wrong device. Check the "zpool =
history" to see what you did.=20
>=20
> In your earlier message, three devices were shown in each raidz, when =
what you should be seeing is that one raidz has an offline device =
identified by guid and maybe "was /dev/da3" that is being replaced, =
along with the replacement device. I don't see any of that.=20

History attached.  There is no replacement device (sub-vdev) until after =
the =E2=80=9Czpool replace=E2=80=9D starts, which it won=E2=80=99t.

>> I didn=E2=80=99t initiate a replace until after the disks were =
physically changed.  Although in this conversation realize that things =
likely got confused by the replacement in the kernel=E2=80=99s mind of =
da3 with what used to be da4.  :-/
>=20
> This is why your zpool history will be helpful. What did you actually =
try to replace, and what did you mean to replace.=20

All of my history since the last previous boot in May.

2024-09-05.09:40:14 zpool offline tank da3
2024-09-05.14:26:44 zpool import -c /etc/zfs/zpool.cache -a -N
2024-09-05.14:32:45 zpool import -c /etc/zfs/zpool.cache -a -N
2024-09-05.14:52:18 zpool offline tank da3
2024-09-05.14:53:51 zpool offline tank da3
2024-09-05.14:59:43 zpool offline -f tank da3
2024-09-05.15:02:53 zpool clear tank
2024-09-05.15:07:41 zpool online tank da3
2024-09-05.15:10:00 zpool add tank spare da10
2024-09-05.15:10:20 zpool offline -f tank da3
2024-09-05.15:35:23 zpool remove tank da10
2024-09-05.15:54:35 zpool scrub tank
2024-09-05.16:01:12 zpool set autoreplace=3Don tank
2024-09-05.16:01:24 zpool set autoexpand=3Don tank
2024-09-05.16:02:16 zpool add -o ashift=3D9 tank spare da10
2024-09-06.10:10:20 zpool remove tank da10

So, I offline=E2=80=99d the disk-to-be-replaced at 09:40 yesterday, then =
I shut the system down, removed that physical device replacing it with a =
larger disk, and rebooted.  I suspect the =E2=80=9Coffline=E2=80=9Ds =
after that are me experimenting when it was telling me it couldn=E2=80=99t=
 start the replace action I was asking for.

The scrub I started yesterday just because the replace says sometihng =
about an operation in progress, so I did that.  It completed with no =
issues, but nothing changed w.r.t. my current problem.

I=E2=80=99m pretty sure the problem here is that the old da3 went away, =
and a new da3 came online as a member of raidz1-1.  The new disk I added =
came online as da10, for some reason.  I had to resolve the issue of the =
UFS disk which used to be da10 now being da9, but that was easy enough.  =
Just unexpected.

      - Chris=

--Apple-Mail=_26064149-B678-4693-B259-B3840725655C
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html; charset=utf-8



On Sep 6, 2024, at 15:16, Wes Morgan =
<morganw@gmail.com> wrote:

You =
probably don't want that. You will have to use the glabel dev, which =
will not be the same size as your other devices. IIRC you have no =
control over what device node the system finds first for the pool. Even =
if you use GPT labels, the daXpY device will still exist. 

Right.  But if I don=E2=80=99=
t _use_ those device names, it won=E2=80=99t matter.  If I use =
/dev/label/foo, or /dev/gpt/foo, I=E2=80=99ll just always use those. =
 I just did that with the ufs disk I have since it moved names, now =
it=E2=80=99s "/dev/ufs/drive12=E2=80=9D in /etc/fstab et =
al.

I want to have some sort of label. =
 I=E2=80=99d rather not have to add a partitioning scheme to the =
disk if I know I=E2=80=99m just going to use the whole disk just to get =
a label, but I suppose if I have to I can.  Though I=E2=80=99d have =
to do it one disk at a time.  :-)


The former da3 is off-line, out of the chassis. =
 I replaced a disk in a full chassis, having them both online at =
the same time is not possible.  That drive in ZFS=E2=80=99s mind is =
only faulted because I tried =E2=80=9Czpool offline -f=E2=80=9D on it to =
see if that helped.

It =
sounds like you have replaced the wrong device. Check the "zpool =
history" to see what you did. 

In your earlier message, =
three devices were shown in each raidz, when what you should be seeing =
is that one raidz has an offline device identified by guid and maybe =
"was /dev/da3" that is being replaced, along with the replacement =
device. I don't see any of that. 

History attached.  There =
is no replacement device (sub-vdev) until after the =E2=80=9Czpool =
replace=E2=80=9D starts, which it won=E2=80=99t.

I didn=E2=80=99t initiate a replace until after =
the disks were physically changed.  Although in this conversation =
realize that things likely got confused by the replacement in the =
kernel=E2=80=99s mind of da3 with what used to be da4. =
 :-/

This is =
why your zpool history will be helpful. What did you actually try to =
replace, and what did you mean to replace. 

All of my history since the last previous boot in =
May.

2024-09-05.09:40:14 zpool offline =
tank da3
2024-09-05.14:26:44 zpool import -c =
/etc/zfs/zpool.cache -a -N
2024-09-05.14:32:45 zpool import -c =
/etc/zfs/zpool.cache -a -N
2024-09-05.14:52:18 zpool offline =
tank da3
2024-09-05.14:53:51 zpool offline tank =
da3
2024-09-05.14:59:43 zpool offline -f tank =
da3
2024-09-05.15:02:53 zpool clear =
tank
2024-09-05.15:07:41 zpool online tank =
da3
2024-09-05.15:10:00 zpool add tank spare =
da10
2024-09-05.15:10:20 zpool offline -f tank =
da3
2024-09-05.15:35:23 zpool remove tank =
da10
2024-09-05.15:54:35 zpool scrub =
tank
2024-09-05.16:01:12 zpool set autoreplace=3Don =
tank
2024-09-05.16:01:24 zpool set autoexpand=3Don =
tank
2024-09-05.16:02:16 zpool add -o ashift=3D9 tank spare =
da10
2024-09-06.10:10:20 zpool remove tank =
da10

So, I offline=E2=80=99d the =
disk-to-be-replaced at 09:40 yesterday, then I shut the system down, =
removed that physical device replacing it with a larger disk, and =
rebooted.  I suspect the =E2=80=9Coffline=E2=80=9Ds after that are =
me experimenting when it was telling me it couldn=E2=80=99t start the =
replace action I was asking for.

The scrub I =
started yesterday just because the replace says sometihng about an =
operation in progress, so I did that.  It completed with no issues, =
but nothing changed w.r.t. my current =
problem.

I=E2=80=99m pretty sure the problem =
here is that the old da3 went away, and a new da3 came online as a =
member of raidz1-1.  The new disk I added came online as da10, for =
some reason.  I had to resolve the issue of the UFS disk which used =
to be da10 now being da9, but that was easy enough.  Just =
unexpected.

      - =
Chris
=

--Apple-Mail=_26064149-B678-4693-B259-B3840725655C--


From nobody Fri Sep  6 21:22:44 2024
X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0q2J5fgcz5W1Yw
	for ; Fri, 06 Sep 2024 21:22:48 +0000 (UTC)
	(envelope-from morganw@gmail.com)
Received: from mail-yw1-x1135.google.com (mail-yw1-x1135.google.com [IPv6:2607:f8b0:4864:20::1135])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "WR4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4X0q2H6n65z47d0
	for ; Fri,  6 Sep 2024 21:22:47 +0000 (UTC)
	(envelope-from morganw@gmail.com)
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=gmail.com header.s=20230601 header.b=mpc3F15G;
	dmarc=pass (policy=none) header.from=gmail.com;
	spf=pass (mx1.freebsd.org: domain of morganw@gmail.com designates 2607:f8b0:4864:20::1135 as permitted sender) smtp.mailfrom=morganw@gmail.com
Received: by mail-yw1-x1135.google.com with SMTP id 00721157ae682-6d6e2b2fbc4so24346747b3.0
        for ; Fri, 06 Sep 2024 14:22:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1725657767; x=1726262567; darn=freebsd.org;
        h=content-transfer-encoding:mime-version:message-id:references
         :in-reply-to:subject:cc:to:from:date:from:to:cc:subject:date
         :message-id:reply-to;
        bh=8YVmtfPB1AL3BXtqQJz4uwQD2aqX2T9fD8RFRHG8U1s=;
        b=mpc3F15Gp71zrtqV8NHA43GGugPMM6K9tl6ZeTHA9LbMXAPAxqUYoO25VZAuvz+sZV
         Qiys9hj/6lcbMTxaq91RlNh2ugdX8En/Dmko1XKF6iXg9PxJosCu2HFYzls/Fjl9hAlF
         e+eeQapzh3EBkV8ptD9LUJe0tDNh66t50qWiQZFTrM/k0qvbYajsVGvJznaRlcDd7DnT
         eIYYT5Cu8T86ikmghycXtPyh2gItMoN09/GiaK7XnVJ3C3+WLyhqcnH5AtTpaHqzxQBo
         opjKcUfcDcJFAv4dJqeWyUKWPP4e4I5+AMvmlYjSjMUe2VlCoBnddiXAwvik2JnnSjdS
         6dGA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1725657767; x=1726262567;
        h=content-transfer-encoding:mime-version:message-id:references
         :in-reply-to:subject:cc:to:from:date:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=8YVmtfPB1AL3BXtqQJz4uwQD2aqX2T9fD8RFRHG8U1s=;
        b=gIEu/ielG0BYzMatbTOo6AS7RKbuvGwmDmvBT+QBPFyoy75xf4BdhqYJU93A/HGXwf
         uybUoPiz6cqui31LT/CpVkATnrw0b/uDe0+e/nJFwy4f5UUyW0yvxCyCxP5OiRX3+AnW
         3yTzMM/isOhCBAovvgpYOU27otRssbSqfrYRVSrzeZ5fOuZaLbX1YQK9oepW3v8/Zlqi
         ZNGqzVeU8iUGP8jFtG31xwfAoRqnexL4GJXxmBbLsXP6HU6cGHmDzaxaBYeh88pLbfW6
         NYTC2AiWLJ7Co+Koc1GeHgOgaeK+Z+ZvoxpgGnKTiV7AAtpcEoU/bjgcSp/THBhIoBHd
         wfcQ==
X-Gm-Message-State: AOJu0YzzN+6olL+yoIH3sKCtjcmwChwbQV/FyPdtN28DdaZdh/pvaYt7
	4pEZ4rl92nZYLBYl5ZcmRl7JQliYSCCfOqf3Sa0bth3YMzsapNzyP6Pb0g==
X-Google-Smtp-Source: AGHT+IG7v+9ovLun9wkSR0Lr/TuTlysIqfmV6nJOCmk2KlTRBUDXaYUdYwYKL7Rm2iFLW0oQ/xSWHQ==
X-Received: by 2002:a05:690c:490c:b0:6b7:3da:cffd with SMTP id 00721157ae682-6db44f164b6mr52649417b3.15.1725657766884;
        Fri, 06 Sep 2024 14:22:46 -0700 (PDT)
Received: from ?IPv6:::1? ([2600:381:642f:e2a:d8f2:e3e8:b579:a70b])
        by smtp.gmail.com with ESMTPSA id 00721157ae682-6db565e601asm1428477b3.143.2024.09.06.14.22.45
        (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
        Fri, 06 Sep 2024 14:22:46 -0700 (PDT)
Date: Fri, 06 Sep 2024 16:22:44 -0500
From: Wes Morgan 
To: Chris Ross 
CC: freebsd-fs@freebsd.org
Subject: Re: Unable to replace drive in raidz1
In-Reply-To: 
References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com>     <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com>  <0CF1E2D7-6C82-4A8B-82C3-A5BF1ED939CF@distal.com>  <29003A7C-745D-4A06-8558-AE64310813EA@distal.com> <42346193-AD06-4D26-B0C6-4392953D21A3@gmail.com>   
Message-ID: <50B791D8-F0CC-431E-93B8-834D57AB3C14@gmail.com>
List-Id: Filesystems 
List-Archive: https://lists.freebsd.org/archives/freebsd-fs
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-fs@FreeBSD.org
MIME-Version: 1.0
Content-Type: text/plain;
 charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spamd-Bar: ---
X-Spamd-Result: default: False [-4.00 / 15.00];
	NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	NEURAL_HAM_LONG(-1.00)[-1.000];
	NEURAL_HAM_SHORT(-1.00)[-1.000];
	DMARC_POLICY_ALLOW(-0.50)[gmail.com,none];
	R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601];
	R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c];
	MIME_GOOD(-0.10)[text/plain];
	RCVD_TLS_LAST(0.00)[];
	RCPT_COUNT_TWO(0.00)[2];
	ARC_NA(0.00)[];
	MIME_TRACE(0.00)[0:+];
	FREEMAIL_ENVFROM(0.00)[gmail.com];
	DKIM_TRACE(0.00)[gmail.com:+];
	FREEMAIL_FROM(0.00)[gmail.com];
	TO_DN_SOME(0.00)[];
	FROM_HAS_DN(0.00)[];
	ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US];
	DWL_DNSWL_NONE(0.00)[gmail.com:dkim];
	TO_MATCH_ENVRCPT_SOME(0.00)[];
	RCVD_COUNT_TWO(0.00)[2];
	FROM_EQ_ENVFROM(0.00)[];
	MISSING_XM_UA(0.00)[];
	PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org];
	MID_RHS_MATCH_FROM(0.00)[];
	MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org];
	RCVD_VIA_SMTP_AUTH(0.00)[];
	TAGGED_RCPT(0.00)[freebsd];
	RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1135:from]
X-Rspamd-Queue-Id: 4X0q2H6n65z47d0

On September 6, 2024 2:34:36 PM CDT, Chris Ross  wrote:
>
>
>> On Sep 6, 2024, at 15:16, Wes Morgan  wrote:
>>=20
>> You probably don't want that=2E You will have to use the glabel dev, wh=
ich will not be the same size as your other devices=2E IIRC you have no con=
trol over what device node the system finds first for the pool=2E Even if y=
ou use GPT labels, the daXpY device will still exist=2E=20
>
>Right=2E  But if I don=E2=80=99t _use_ those device names, it won=E2=80=
=99t matter=2E  If I use /dev/label/foo, or /dev/gpt/foo, I=E2=80=99ll just=
 always use those=2E  I just did that with the ufs disk I have since it mov=
ed names, now it=E2=80=99s "/dev/ufs/drive12=E2=80=9D in /etc/fstab et al=
=2E

The labels are helpful for fstab, but zfs doesn't need fstab=2E In the ear=
ly days of zfs on freebsd the unpartitioned device was recommended; maybe t=
hat's not accurate any longer, but I still follow it for a pool that contai=
ns vdevs with multiple devices (raidz)=2E=20

If you use, e=2Eg=2E, da0 in a pool, you cannot later replace it with a la=
beled device of the same size; it won't have enough sectors=2E=20

>I want to have some sort of label=2E  I=E2=80=99d rather not have to add =
a partitioning scheme to the disk if I know I=E2=80=99m just going to use t=
he whole disk just to get a label, but I suppose if I have to I can=2E  Tho=
ugh I=E2=80=99d have to do it one disk at a time=2E  :-)

ZFS will absolutely find the device if it is readable=2E The label on ever=
y device contains enough metadata to describe the entire vdev (and the pool=
 I believe), including the missing devices=2E It's very good at finding the=
m=2E The clearlabel command was added because it was a pain to get zfs to g=
ive up on a disk that has been repurposed=2E You really don't need the labe=
ls, but if you have trouble figuring out which disk is which, that may be t=
he only way for you to be sure=2E


>>=20
>>> The former da3 is off-line, out of the chassis=2E  I replaced a disk i=
n a full chassis, having them both online at the same time is not possible=
=2E  That drive in ZFS=E2=80=99s mind is only faulted because I tried =E2=
=80=9Czpool offline -f=E2=80=9D on it to see if that helped=2E
>>=20
>> It sounds like you have replaced the wrong device=2E Check the "zpool h=
istory" to see what you did=2E=20
>>=20
>> In your earlier message, three devices were shown in each raidz, when w=
hat you should be seeing is that one raidz has an offline device identified=
 by guid and maybe "was /dev/da3" that is being replaced, along with the re=
placement device=2E I don't see any of that=2E=20
>
>History attached=2E  There is no replacement device (sub-vdev) until afte=
r the =E2=80=9Czpool replace=E2=80=9D starts, which it won=E2=80=99t=2E
>
>>> I didn=E2=80=99t initiate a replace until after the disks were physica=
lly changed=2E  Although in this conversation realize that things likely go=
t confused by the replacement in the kernel=E2=80=99s mind of da3 with what=
 used to be da4=2E  :-/
>>=20
>> This is why your zpool history will be helpful=2E What did you actually=
 try to replace, and what did you mean to replace=2E=20
>
>All of my history since the last previous boot in May=2E
>
>2024-09-05=2E09:40:14 zpool offline tank da3
>2024-09-05=2E14:26:44 zpool import -c /etc/zfs/zpool=2Ecache -a -N
>2024-09-05=2E14:32:45 zpool import -c /etc/zfs/zpool=2Ecache -a -N
>2024-09-05=2E14:52:18 zpool offline tank da3
>2024-09-05=2E14:53:51 zpool offline tank da3
>2024-09-05=2E14:59:43 zpool offline -f tank da3
>2024-09-05=2E15:02:53 zpool clear tank
>2024-09-05=2E15:07:41 zpool online tank da3
>2024-09-05=2E15:10:00 zpool add tank spare da10
>2024-09-05=2E15:10:20 zpool offline -f tank da3
>2024-09-05=2E15:35:23 zpool remove tank da10
>2024-09-05=2E15:54:35 zpool scrub tank
>2024-09-05=2E16:01:12 zpool set autoreplace=3Don tank
>2024-09-05=2E16:01:24 zpool set autoexpand=3Don tank
>2024-09-05=2E16:02:16 zpool add -o ashift=3D9 tank spare da10
>2024-09-06=2E10:10:20 zpool remove tank da10
>
>So, I offline=E2=80=99d the disk-to-be-replaced at 09:40 yesterday, then =
I shut the system down, removed that physical device replacing it with a la=
rger disk, and rebooted=2E  I suspect the =E2=80=9Coffline=E2=80=9Ds after =
that are me experimenting when it was telling me it couldn=E2=80=99t start =
the replace action I was asking for=2E

This is probably where you made your mistake=2E Rebooting shifted another =
device into da3=2E When you tried to offline it, you were probably either t=
argeting a device in a different raidz or one that wasn't in the pool=2E Th=
e output of those original offline commands would have been informative=2E =
You could also check dmesg and map the serial numbers to device assignments=
 to figure out what device moved to da3=2E

>The scrub I started yesterday just because the replace says sometihng abo=
ut an operation in progress, so I did that=2E  It completed with no issues,=
 but nothing changed w=2Er=2Et=2E my current problem=2E
>
>I=E2=80=99m pretty sure the problem here is that the old da3 went away, a=
nd a new da3 came online as a member of raidz1-1=2E  The new disk I added c=
ame online as da10, for some reason=2E  I had to resolve the issue of the U=
FS disk which used to be da10 now being da9, but that was easy enough=2E  J=
ust unexpected=2E

Sounds about right=2E In another message it seemed like the pool had start=
ed an autoreplace=2E So I assume you have zfsd enabled? That is what issues=
 the replace command=2E Strange that it is not anywhere in the pool history=
=2E There should be syslog entries for any actions it took=2E

In your case, it appears that you had two missing devices - the original "=
da3" that was physically removed, and the new da3 that you forced offline=
=2E You added da10 as a spare, when what you needed to do was a replace=2E =
Spare devices do not auto-replace without zfsd running and autoreplace set =
to on=2E

This should all be reported in zpool status=2E In your original message, t=
here is no sign of a replacement in progress or a spare device, assuming yo=
u didn't omit something=2E If the pool is only showing that a single device=
 is missing, and that device is to be replaced by da10, zero out the first =
and last sectors (I think a zfs label is 128k?) to wipe out any labels and =
use the replace command, not spare, e=2Eg=2E "zpool replace tank da3 da10",=
 or use the missing guid as suggested elsewhere=2E This should work based o=
n the information provided=2E





From nobody Fri Sep  6 21:48:57 2024
X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0qcv4rgNz5W3rl
	for ; Fri, 06 Sep 2024 21:49:19 +0000 (UTC)
	(envelope-from cross+freebsd@relay.distal.com)
Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(Client did not present a certificate)
	by mx1.freebsd.org (Postfix) with ESMTPS id 4X0qcv0L9yz4F0X
	for ; Fri,  6 Sep 2024 21:49:18 +0000 (UTC)
	(envelope-from cross+freebsd@relay.distal.com)
Authentication-Results: mx1.freebsd.org;
	none
dkim-signature: v=1; a=rsa-sha256; d=relay.distal.com; s=mail;
	c=relaxed/relaxed; q=dns/txt; h=From:Subject:Date:Message-ID:To:CC:MIME-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References;
	bh=rsuW4lzphMF0STMnv4uG0XZCLSiaD25MFQ3VawRzNfk=;
	b=qLleNXndckLDDwBqpATyZe74RdRNStexX6hGPpAMq8yh1QJqFlUvIDDAYPv0v6BajG6jtN4qsTVESVlqSNfVMEUjjeO1qvrTp2dJwDYPABb59LNl3MpjJMKYuQ1Z67h0RyCjvH4H7p3LQEJG88Qim5pEWy0csar9X8FfRsdVyDRi5214aIHF2unGYSBngRY1eAmrQnje16NfBfDAnvwZjhuwMlWcp4netwkGXD8zJx4JaUgSRYC6H+1b9o
	rN6qIoesekrMf8E/M9Jt9Z2C38FSXWLfdNdNF5H2Jfh5xN+UUtfMfMlS/8lmVX08m/U07xHmz09ked61361DIpW4Ma+w==
Received: from mail.distal.com (pool-108-51-233-124.washdc.fios.verizon.net [108.51.233.124])
 by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305
 bits=256) ; Fri, 6 Sep 2024 21:49:17 +0000
Received: from smtpclient.apple ( [2001:420:c0c4:1001::9f]) by tristain.distal.com
 (OpenSMTPD) with ESMTPSA id 11d8e4f5 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO);
 Fri, 6 Sep 2024 17:49:10 -0400 (EDT)
Content-Type: text/plain; charset=utf-8
List-Id: Filesystems 
List-Archive: https://lists.freebsd.org/archives/freebsd-fs
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-fs@FreeBSD.org
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3776.700.51\))
Subject: Re: Unable to replace drive in raidz1
From: Chris Ross 
In-Reply-To: <50B791D8-F0CC-431E-93B8-834D57AB3C14@gmail.com>
Date: Fri, 6 Sep 2024 17:48:57 -0400
Cc: FreeBSD Filesystems 
Content-Transfer-Encoding: quoted-printable
Message-Id: 
References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> 
  
  <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com>
  <0CF1E2D7-6C82-4A8B-82C3-A5BF1ED939CF@distal.com>
  <29003A7C-745D-4A06-8558-AE64310813EA@distal.com>
 <42346193-AD06-4D26-B0C6-4392953D21A3@gmail.com> 
  
 <50B791D8-F0CC-431E-93B8-834D57AB3C14@gmail.com>
To: Wes Morgan 
X-Mailer: Apple Mail (2.3776.700.51)
X-Spamd-Bar: ----
X-Rspamd-Pre-Result: action=no action;
	module=replies;
	Message is reply to one we originated
X-Spamd-Result: default: False [-4.00 / 15.00];
	REPLY(-4.00)[];
	TAGGED_FROM(0.00)[freebsd];
	ASN(0.00)[asn:3842, ipnet:168.235.92.0/22, country:US]
X-Rspamd-Queue-Id: 4X0qcv0L9yz4F0X



> On Sep 6, 2024, at 17:22, Wes Morgan  wrote:
>=20
> The labels are helpful for fstab, but zfs doesn't need fstab. In the =
early days of zfs on freebsd the unpartitioned device was recommended; =
maybe that's not accurate any longer, but I still follow it for a pool =
that contains vdevs with multiple devices (raidz).=20
>=20
> If you use, e.g., da0 in a pool, you cannot later replace it with a =
labeled device of the same size; it won't have enough sectors.=20

The problem is shown here.  da3 was in a pool.  Then, when the system =
rebooted, da3 was the kernels name for a different device in a different =
pool.  Had I known then how to interact with the guid (status -g), I =
likely would=E2=80=99ve been fine.

>> So, I offline=E2=80=99d the disk-to-be-replaced at 09:40 yesterday, =
then I shut the system down, removed that physical device replacing it =
with a larger disk, and rebooted.  I suspect the =E2=80=9Coffline=E2=80=9D=
s after that are me experimenting when it was telling me it couldn=E2=80=99=
t start the replace action I was asking for.
>=20
> This is probably where you made your mistake. Rebooting shifted =
another device into da3. When you tried to offline it, you were probably =
either targeting a device in a different raidz or one that wasn't in the =
pool. The output of those original offline commands would have been =
informative. You could also check dmesg and map the serial numbers to =
device assignments to figure out what device moved to da3.

I offline=E2=80=99d =E2=80=9Cda3=E2=80=9D before I rebooted.  After =
rebooting, I tried the obvious and correct (i thought) =E2=80=9Czpool =
replace da3 da10=E2=80=9D only to get the error I=E2=80=99ve been =
getting since.  Again, had I known how to use the guid for the device =
that used to be da3 but now isn=E2=80=99t, that might=E2=80=99ve worked. =
 I can=E2=80=99t know now.

Then, while trying to fix the problem, I likely made it worse by trying =
to interact with da3, which in the pools brain was a missing disk in =
raidz1-0, but the kernel also knew /dev/da3 to be a working disk (that =
happened to be one of the drives in raidz1-1).  I feel that zfs did =
something wrong somewhere if it _ever_ tried to talk to /dev/da3 when I =
said =E2=80=9Cda3=E2=80=9D after I rebooted and it found that device to =
be part of raidz1-1, but.


> Sounds about right. In another message it seemed like the pool had =
started an autoreplace. So I assume you have zfsd enabled? That is what =
issues the replace command. Strange that it is not anywhere in the pool =
history. There should be syslog entries for any actions it took.

I don=E2=80=99t think so.  That message about some =E2=80=9Calready in =
replacing/spare config=E2=80=9D came up before anything else.  At which =
point, I=E2=80=99d never had a spare in this pool, and there was no =
replace shown in zpool status.

> In your case, it appears that you had two missing devices - the =
original "da3" that was physically removed, and the new da3 that you =
forced offline. You added da10 as a spare, when what you needed to do =
was a replace. Spare devices do not auto-replace without zfsd running =
and autoreplace set to on.

I did offline =E2=80=9Cda3=E2=80=9D a couple of times, again thinking I =
was working with what zpool showed as =E2=80=9Cda3=E2=80=9D.  If it did =
anything with /dev/da3 there, then I think that may be a bug.  Or, at =
least, something that should be made more clear.  It _didn=E2=80=99t_ =
offline the diskid/DISK-K1GMBN9D from raidz1-1, which is what the kernel =
has at da3.  So.

> This should all be reported in zpool status. In your original message, =
there is no sign of a replacement in progress or a spare device, =
assuming you didn't omit something. If the pool is only showing that a =
single device is missing, and that device is to be replaced by da10, =
zero out the first and last sectors (I think a zfs label is 128k?) to =
wipe out any labels and use the replace command, not spare, e.g. "zpool =
replace tank da3 da10", or use the missing guid as suggested elsewhere. =
This should work based on the information provided.

I=E2=80=99ve never seen a replacement going on, and I have had the new =
disk =E2=80=9Cda10=E2=80=9D as a spare a couple of times while testing.  =
But it wasn=E2=80=99t left there after I determined that that also =
didn=E2=80=99t let me get it replaced into the raidz.

And, that attempt to replace is what I=E2=80=99ve tried many times, with =
multiple id=E2=80=99s.  I have cleared the label on da10 multiple times. =
 That replace doesn=E2=80=99t work, giving this error message in all =
cases.

            - Chris


% glabel status
                                      Name  Status  Components
            diskid/DISK-BTWL503503TW480QGN     N/A  ada0
                                 gpt/l2arc     N/A  ada0p1
gptid/9d00849e-0b82-11ec-a143-84b2612f2c38     N/A  ada0p1
                      diskid/DISK-K1GMBN9D     N/A  da3
                      diskid/DISK-3WJDHJ2J     N/A  da6
                      diskid/DISK-3WK3G1KJ     N/A  da7
                      diskid/DISK-3WJ7ZMMJ     N/A  da8
                      diskid/DISK-K1GMEDMD     N/A  da4
                      diskid/DISK-K1GMAX1D     N/A  da5
                               ufs/drive12     N/A  da9
                      diskid/DISK-ZGG0A2PA     N/A  da10

% zpool status tank
  pool: tank
 state: DEGRADED
status: One or more devices are faulted in response to persistent =
errors.
	Sufficient replicas exist for the pool to continue functioning =
in a
	degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the =
device
	repaired.
  scan: scrub repaired 0B in 17:14:03 with 0 errors on Fri Sep  6 =
09:08:34 2024
config:

	NAME                      STATE     READ WRITE CKSUM
	tank                      DEGRADED     0     0     0
	  raidz1-0                DEGRADED     0     0     0
	    da3                   FAULTED      0     0     0  external =
device fault
	    da1                   ONLINE       0     0     0
	    da2                   ONLINE       0     0     0
	  raidz1-1                ONLINE       0     0     0
	    diskid/DISK-K1GMBN9D  ONLINE       0     0     0
	    diskid/DISK-K1GMEDMD  ONLINE       0     0     0
	    diskid/DISK-K1GMAX1D  ONLINE       0     0     0
	  raidz1-2                ONLINE       0     0     0
	    diskid/DISK-3WJDHJ2J  ONLINE       0     0     0
	    diskid/DISK-3WK3G1KJ  ONLINE       0     0     0
	    diskid/DISK-3WJ7ZMMJ  ONLINE       0     0     0

errors: No known data errors

% sudo zpool replace tank da3 da10
Password:
cannot replace da3 with da10: already in replacing/spare config; wait =
for completion or use 'zpool detach'

% zpool status -g tank
  pool: tank
 state: DEGRADED
status: One or more devices are faulted in response to persistent =
errors.
	Sufficient replicas exist for the pool to continue functioning =
in a
	degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the =
device
	repaired.
  scan: scrub repaired 0B in 17:14:03 with 0 errors on Fri Sep  6 =
09:08:34 2024
config:

	NAME                      STATE     READ WRITE CKSUM
	tank                      DEGRADED     0     0     0
	  16506780107187041124    DEGRADED     0     0     0
	    9127016430593660128   FAULTED      0     0     0  external =
device fault
	    4094297345166589692   ONLINE       0     0     0
	    17850258180603290288  ONLINE       0     0     0
	  5104119975785735782     ONLINE       0     0     0
	    6752552549817423876   ONLINE       0     0     0
	    9072227575611698625   ONLINE       0     0     0
	    13778609510621402511  ONLINE       0     0     0
	  11410204456339324959    ONLINE       0     0     0
	    1083322824660576293   ONLINE       0     0     0
	    12505496659970146740  ONLINE       0     0     0
	    11847701970749615606  ONLINE       0     0     0

errors: No known data errors

% sudo zpool replace tank 9127016430593660128 da10
cannot replace 9127016430593660128 with da10: already in replacing/spare =
config; wait for completion or use 'zpool detach'

% sudo zpool replace tank 9127016430593660128 diskid/DISK-ZGG0A2PA
cannot replace 9127016430593660128 with diskid/DISK-ZGG0A2PA: already in =
replacing/spare config; wait for completion or use 'zpool detach'





From nobody Fri Sep  6 22:02:31 2024
X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0qwP2LqFz5W5B5
	for ; Fri, 06 Sep 2024 22:02:45 +0000 (UTC)
	(envelope-from asomers@gmail.com)
Received: from mail-oi1-f177.google.com (mail-oi1-f177.google.com [209.85.167.177])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "WR4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4X0qwN49XSz4GLt
	for ; Fri,  6 Sep 2024 22:02:44 +0000 (UTC)
	(envelope-from asomers@gmail.com)
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=freebsd.org (policy=none);
	spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.167.177 as permitted sender) smtp.mailfrom=asomers@gmail.com
Received: by mail-oi1-f177.google.com with SMTP id 5614622812f47-3e0059dc06bso1630141b6e.0
        for ; Fri, 06 Sep 2024 15:02:44 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1725660163; x=1726264963;
        h=content-transfer-encoding:cc:to:subject:message-id:date:from
         :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=vaWvswPE7Hfpi9K8aN9u3LR82RVyWXHrDN1FgBF0x+c=;
        b=shxWYJC1zM29FGIx4F+TyPa4jex6DtZ99Ov1RXGJjuhbnFJzierGKdtbX1eecDwkFl
         kLHGNXMmZGCfFUyOqJBi82zpFnYjJYvwS/4BvSVg8cOX9464Wj/uT6FKnDvAZfcGgvCg
         DchfLFfSoh8/4XVrtPVoV99FDLMhN5ofkPY1N8AbySK9PW4gQKMRlJ5pswv6QHm46pwN
         YS9cdhHDYML0zJUgSLWvNNdmnXO9E1NbheKlXOAF/BVe0ei5mM6ZlY9TenYgTenwi+Wd
         4Pe1BXThLeNxjkv1W6TkNbwSyUFjpYPNTGytklXDsC9EgwPM55URtt4MjYLgjBwbaSs/
         NtpQ==
X-Forwarded-Encrypted: i=1; AJvYcCU5tNtrICsJcIBg0e2a6OcY6tj1Yf+YBCQ4/wfy8EdkKTwVdJ6GoZEBQ18uYnZC+gsISbCWXY3RwLRo@freebsd.org
X-Gm-Message-State: AOJu0YyG9Xtcm6tvHV8UdMQ/d0Ln9msXJQOvnK0D8UeQgUGlXtRoRnHi
	Dknf8foLkaulldmt4h2iiMDf4PtkEpjyUKNRAtbHbPZ+/1m8zNMwCxxXBAa10pvSLYf4ppd3B83
	E22YEx05Lf+hn6jevycTJ5VZKvNU=
X-Google-Smtp-Source: AGHT+IGKV2zVeSjhAC3VeL7dBzQUA7P1h6B/xEzlZ4T+xfvu/QWrg4eoeQ/n+QBDE7mhZu449tiuE/4/5YexYgzfP2s=
X-Received: by 2002:a05:6808:6544:b0:3db:331:9d41 with SMTP id
 5614622812f47-3e029ef8ac9mr4598310b6e.23.1725660162888; Fri, 06 Sep 2024
 15:02:42 -0700 (PDT)
List-Id: Filesystems 
List-Archive: https://lists.freebsd.org/archives/freebsd-fs
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-fs@FreeBSD.org
MIME-Version: 1.0
References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com>
  
  
 <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com> 
 <0CF1E2D7-6C82-4A8B-82C3-A5BF1ED939CF@distal.com> 
 <29003A7C-745D-4A06-8558-AE64310813EA@distal.com> <42346193-AD06-4D26-B0C6-4392953D21A3@gmail.com>
  
  <50B791D8-F0CC-431E-93B8-834D57AB3C14@gmail.com>
 
In-Reply-To: 
From: Alan Somers 
Date: Fri, 6 Sep 2024 16:02:31 -0600
Message-ID: 
Subject: Re: Unable to replace drive in raidz1
To: Chris Ross 
Cc: Wes Morgan , FreeBSD Filesystems 
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spamd-Bar: --
X-Spamd-Result: default: False [-2.90 / 15.00];
	NEURAL_HAM_LONG(-1.00)[-1.000];
	NEURAL_HAM_SHORT(-1.00)[-1.000];
	NEURAL_HAM_MEDIUM(-1.00)[-0.999];
	FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com];
	R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17];
	MIME_GOOD(-0.10)[text/plain];
	DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), No valid DKIM,none];
	FROM_HAS_DN(0.00)[];
	RCPT_COUNT_THREE(0.00)[3];
	FREEMAIL_CC(0.00)[gmail.com,freebsd.org];
	RCVD_TLS_LAST(0.00)[];
	ARC_NA(0.00)[];
	TO_DN_ALL(0.00)[];
	MIME_TRACE(0.00)[0:+];
	MISSING_XM_UA(0.00)[];
	FREEFALL_USER(0.00)[asomers];
	FREEMAIL_ENVFROM(0.00)[gmail.com];
	PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org];
	TO_MATCH_ENVRCPT_SOME(0.00)[];
	FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com];
	RCVD_COUNT_ONE(0.00)[1];
	MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org];
	TAGGED_RCPT(0.00)[freebsd];
	R_DKIM_NA(0.00)[];
	ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US];
	RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.167.177:from];
	RCVD_IN_DNSWL_NONE(0.00)[209.85.167.177:from]
X-Rspamd-Queue-Id: 4X0qwN49XSz4GLt

On Fri, Sep 6, 2024 at 3:49=E2=80=AFPM Chris Ross  wrote:
>
>
>
> > On Sep 6, 2024, at 17:22, Wes Morgan  wrote:
> >
> > The labels are helpful for fstab, but zfs doesn't need fstab. In the ea=
rly days of zfs on freebsd the unpartitioned device was recommended; maybe =
that's not accurate any longer, but I still follow it for a pool that conta=
ins vdevs with multiple devices (raidz).
> >
> > If you use, e.g., da0 in a pool, you cannot later replace it with a lab=
eled device of the same size; it won't have enough sectors.
>
> The problem is shown here.  da3 was in a pool.  Then, when the system reb=
ooted, da3 was the kernels name for a different device in a different pool.=
  Had I known then how to interact with the guid (status -g), I likely woul=
d=E2=80=99ve been fine.
>
> >> So, I offline=E2=80=99d the disk-to-be-replaced at 09:40 yesterday, th=
en I shut the system down, removed that physical device replacing it with a=
 larger disk, and rebooted.  I suspect the =E2=80=9Coffline=E2=80=9Ds after=
 that are me experimenting when it was telling me it couldn=E2=80=99t start=
 the replace action I was asking for.
> >
> > This is probably where you made your mistake. Rebooting shifted another=
 device into da3. When you tried to offline it, you were probably either ta=
rgeting a device in a different raidz or one that wasn't in the pool. The o=
utput of those original offline commands would have been informative. You c=
ould also check dmesg and map the serial numbers to device assignments to f=
igure out what device moved to da3.
>
> I offline=E2=80=99d =E2=80=9Cda3=E2=80=9D before I rebooted.  After reboo=
ting, I tried the obvious and correct (i thought) =E2=80=9Czpool replace da=
3 da10=E2=80=9D only to get the error I=E2=80=99ve been getting since.  Aga=
in, had I known how to use the guid for the device that used to be da3 but =
now isn=E2=80=99t, that might=E2=80=99ve worked.  I can=E2=80=99t know now.
>
> Then, while trying to fix the problem, I likely made it worse by trying t=
o interact with da3, which in the pools brain was a missing disk in raidz1-=
0, but the kernel also knew /dev/da3 to be a working disk (that happened to=
 be one of the drives in raidz1-1).  I feel that zfs did something wrong so=
mewhere if it _ever_ tried to talk to /dev/da3 when I said =E2=80=9Cda3=E2=
=80=9D after I rebooted and it found that device to be part of raidz1-1, bu=
t.
>
>
> > Sounds about right. In another message it seemed like the pool had star=
ted an autoreplace. So I assume you have zfsd enabled? That is what issues =
the replace command. Strange that it is not anywhere in the pool history. T=
here should be syslog entries for any actions it took.
>
> I don=E2=80=99t think so.  That message about some =E2=80=9Calready in re=
placing/spare config=E2=80=9D came up before anything else.  At which point=
, I=E2=80=99d never had a spare in this pool, and there was no replace show=
n in zpool status.
>
> > In your case, it appears that you had two missing devices - the origina=
l "da3" that was physically removed, and the new da3 that you forced offlin=
e. You added da10 as a spare, when what you needed to do was a replace. Spa=
re devices do not auto-replace without zfsd running and autoreplace set to =
on.
>
> I did offline =E2=80=9Cda3=E2=80=9D a couple of times, again thinking I w=
as working with what zpool showed as =E2=80=9Cda3=E2=80=9D.  If it did anyt=
hing with /dev/da3 there, then I think that may be a bug.  Or, at least, so=
mething that should be made more clear.  It _didn=E2=80=99t_ offline the di=
skid/DISK-K1GMBN9D from raidz1-1, which is what the kernel has at da3.  So.
>
> > This should all be reported in zpool status. In your original message, =
there is no sign of a replacement in progress or a spare device, assuming y=
ou didn't omit something. If the pool is only showing that a single device =
is missing, and that device is to be replaced by da10, zero out the first a=
nd last sectors (I think a zfs label is 128k?) to wipe out any labels and u=
se the replace command, not spare, e.g. "zpool replace tank da3 da10", or u=
se the missing guid as suggested elsewhere. This should work based on the i=
nformation provided.
>
> I=E2=80=99ve never seen a replacement going on, and I have had the new di=
sk =E2=80=9Cda10=E2=80=9D as a spare a couple of times while testing.  But =
it wasn=E2=80=99t left there after I determined that that also didn=E2=80=
=99t let me get it replaced into the raidz.
>
> And, that attempt to replace is what I=E2=80=99ve tried many times, with =
multiple id=E2=80=99s.  I have cleared the label on da10 multiple times.  T=
hat replace doesn=E2=80=99t work, giving this error message in all cases.
>
>             - Chris
>
>
> % glabel status
>                                       Name  Status  Components
>             diskid/DISK-BTWL503503TW480QGN     N/A  ada0
>                                  gpt/l2arc     N/A  ada0p1
> gptid/9d00849e-0b82-11ec-a143-84b2612f2c38     N/A  ada0p1
>                       diskid/DISK-K1GMBN9D     N/A  da3
>                       diskid/DISK-3WJDHJ2J     N/A  da6
>                       diskid/DISK-3WK3G1KJ     N/A  da7
>                       diskid/DISK-3WJ7ZMMJ     N/A  da8
>                       diskid/DISK-K1GMEDMD     N/A  da4
>                       diskid/DISK-K1GMAX1D     N/A  da5
>                                ufs/drive12     N/A  da9
>                       diskid/DISK-ZGG0A2PA     N/A  da10
>
> % zpool status tank
>   pool: tank
>  state: DEGRADED
> status: One or more devices are faulted in response to persistent errors.
>         Sufficient replicas exist for the pool to continue functioning in=
 a
>         degraded state.
> action: Replace the faulted device, or use 'zpool clear' to mark the devi=
ce
>         repaired.
>   scan: scrub repaired 0B in 17:14:03 with 0 errors on Fri Sep  6 09:08:3=
4 2024
> config:
>
>         NAME                      STATE     READ WRITE CKSUM
>         tank                      DEGRADED     0     0     0
>           raidz1-0                DEGRADED     0     0     0
>             da3                   FAULTED      0     0     0  external de=
vice fault
>             da1                   ONLINE       0     0     0
>             da2                   ONLINE       0     0     0
>           raidz1-1                ONLINE       0     0     0
>             diskid/DISK-K1GMBN9D  ONLINE       0     0     0
>             diskid/DISK-K1GMEDMD  ONLINE       0     0     0
>             diskid/DISK-K1GMAX1D  ONLINE       0     0     0
>           raidz1-2                ONLINE       0     0     0
>             diskid/DISK-3WJDHJ2J  ONLINE       0     0     0
>             diskid/DISK-3WK3G1KJ  ONLINE       0     0     0
>             diskid/DISK-3WJ7ZMMJ  ONLINE       0     0     0
>
> errors: No known data errors
>
> % sudo zpool replace tank da3 da10
> Password:
> cannot replace da3 with da10: already in replacing/spare config; wait for=
 completion or use 'zpool detach'
>
> % zpool status -g tank
>   pool: tank
>  state: DEGRADED
> status: One or more devices are faulted in response to persistent errors.
>         Sufficient replicas exist for the pool to continue functioning in=
 a
>         degraded state.
> action: Replace the faulted device, or use 'zpool clear' to mark the devi=
ce
>         repaired.
>   scan: scrub repaired 0B in 17:14:03 with 0 errors on Fri Sep  6 09:08:3=
4 2024
> config:
>
>         NAME                      STATE     READ WRITE CKSUM
>         tank                      DEGRADED     0     0     0
>           16506780107187041124    DEGRADED     0     0     0
>             9127016430593660128   FAULTED      0     0     0  external de=
vice fault
>             4094297345166589692   ONLINE       0     0     0
>             17850258180603290288  ONLINE       0     0     0
>           5104119975785735782     ONLINE       0     0     0
>             6752552549817423876   ONLINE       0     0     0
>             9072227575611698625   ONLINE       0     0     0
>             13778609510621402511  ONLINE       0     0     0
>           11410204456339324959    ONLINE       0     0     0
>             1083322824660576293   ONLINE       0     0     0
>             12505496659970146740  ONLINE       0     0     0
>             11847701970749615606  ONLINE       0     0     0
>
> errors: No known data errors
>
> % sudo zpool replace tank 9127016430593660128 da10
> cannot replace 9127016430593660128 with da10: already in replacing/spare =
config; wait for completion or use 'zpool detach'
>
> % sudo zpool replace tank 9127016430593660128 diskid/DISK-ZGG0A2PA
> cannot replace 9127016430593660128 with diskid/DISK-ZGG0A2PA: already in =
replacing/spare config; wait for completion or use 'zpool detach'

Another user reports the same error message.  In their case, it's an
inappropriate error message from /sbin/zpool.  Can you try a "zpool
status -v" and "diskinfo -f /dev/da10"?  That will show you if you
have the same problem.  If your pool has a 512B block size but the new
disk is 4kn, then you cannot use it as a replacement.

https://github.com/openzfs/zfs/issues/14730

From nobody Fri Sep  6 22:22:16 2024
X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0rMG0hvMz5W6sT
	for ; Fri, 06 Sep 2024 22:22:34 +0000 (UTC)
	(envelope-from cross+freebsd@relay.distal.com)
Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(Client did not present a certificate)
	by mx1.freebsd.org (Postfix) with ESMTPS id 4X0rMF5Nhnz4LFp;
	Fri,  6 Sep 2024 22:22:33 +0000 (UTC)
	(envelope-from cross+freebsd@relay.distal.com)
Authentication-Results: mx1.freebsd.org;
	none
dkim-signature: v=1; a=rsa-sha256; d=relay.distal.com; s=mail;
	c=relaxed/relaxed; q=dns/txt; h=From:Subject:Date:Message-ID:To:CC:MIME-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References;
	bh=vw98N5dd917PA0j5PXb3yb2eT/MVw//UavmdR5lccNE=;
	b=LcfjEsGxhJ7OeJoZUKZTYACFkBVmLA3DC89qWxMg7jNFIaVeqj9gauxDP3h65UCekuXYL8EFeEy5g6Y+ZxFW94H3+LPseNgevFnsm9FXgLD3M80ZHHCLHBrF6M2qsbI3Gq0xmUGhIQzw8vopKWMiTEkXgzq6M3aTXZ9ZEHmpX+NVNitF2xIW8DJ1O0ODO3smJm2zwxe8ChhXTMzjh7kteL1HCNtg69f7dT4TnG67QvnfhtTwLAniqILdZ4
	0svylL9+XPenb8X4YgimwVniA0gzthpsu/mNNVBi0sHMxYUCAzyYs/Bo7mfMnM/x7k0uw04YrZmGSFct626LjBx1FuRg==
Received: from mail.distal.com (pool-108-51-233-124.washdc.fios.verizon.net [108.51.233.124])
 by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305
 bits=256) ; Fri, 6 Sep 2024 22:22:30 +0000
Received: from smtpclient.apple ( [2001:420:c0c4:1001::9f]) by tristain.distal.com
 (OpenSMTPD) with ESMTPSA id 9cd54e1f (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO);
 Fri, 6 Sep 2024 18:22:28 -0400 (EDT)
Content-Type: text/plain; charset=utf-8
List-Id: Filesystems 
List-Archive: https://lists.freebsd.org/archives/freebsd-fs
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-fs@FreeBSD.org
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3776.700.51\))
Subject: Re: Unable to replace drive in raidz1
From: Chris Ross 
In-Reply-To: 
Date: Fri, 6 Sep 2024 18:22:16 -0400
Cc: Wes Morgan , FreeBSD Filesystems 
Content-Transfer-Encoding: quoted-printable
Message-Id: 
References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> 
  
  <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com>
  <0CF1E2D7-6C82-4A8B-82C3-A5BF1ED939CF@distal.com>
  <29003A7C-745D-4A06-8558-AE64310813EA@distal.com>
 <42346193-AD06-4D26-B0C6-4392953D21A3@gmail.com> 
  
 <50B791D8-F0CC-431E-93B8-834D57AB3C14@gmail.com> 
 
To: Alan Somers 
X-Mailer: Apple Mail (2.3776.700.51)
X-Spamd-Bar: ----
X-Rspamd-Pre-Result: action=no action;
	module=replies;
	Message is reply to one we originated
X-Spamd-Result: default: False [-4.00 / 15.00];
	REPLY(-4.00)[];
	TAGGED_FROM(0.00)[freebsd];
	ASN(0.00)[asn:3842, ipnet:168.235.92.0/22, country:US]
X-Rspamd-Queue-Id: 4X0rMF5Nhnz4LFp



> On Sep 6, 2024, at 18:02, Alan Somers  wrote:
> Another user reports the same error message.  In their case, it's an
> inappropriate error message from /sbin/zpool.  Can you try a "zpool
> status -v" and "diskinfo -f /dev/da10"?  That will show you if you
> have the same problem.  If your pool has a 512B block size but the new
> disk is 4kn, then you cannot use it as a replacement.
>=20
> https://github.com/openzfs/zfs/issues/14730

Yeah, I came across that in my searching.  Although, I thought people=20
had found success adding =E2=80=9C-o ashift=3D9=E2=80=9D to the replace =
command.
I can confirm that this raidz1 is ashift=3D9, and the disks in it 512
byte sectorsize.  The other two raidz=E2=80=99s are ashift=3D12.

diskinfo shows 512 as sectorsize for one of the older disks
(matching model to the removed da3), and 4096 sectorsize
for the new disk.

Is this totally impossible?  I know I have replaced smaller disks with
much larger ones in the past, though it=E2=80=99s been a few years and
I don=E2=80=99t remember the details.  Is there really no way to put a =
4k
block disk in a set with 512 byte blocks?  I thought they were still
able to =E2=80=9Cact=E2=80=9D like 512-byte block disks?  (512e)

               - Chris=


From nobody Fri Sep  6 22:28:31 2024
X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0rVP5jCSz5W7SR
	for ; Fri, 06 Sep 2024 22:28:45 +0000 (UTC)
	(envelope-from asomers@gmail.com)
Received: from mail-vs1-f46.google.com (mail-vs1-f46.google.com [209.85.217.46])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "WR4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4X0rVN0sQxz4MWT
	for ; Fri,  6 Sep 2024 22:28:44 +0000 (UTC)
	(envelope-from asomers@gmail.com)
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=freebsd.org (policy=none);
	spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.217.46 as permitted sender) smtp.mailfrom=asomers@gmail.com
Received: by mail-vs1-f46.google.com with SMTP id ada2fe7eead31-49bc44e52d0so697075137.1
        for ; Fri, 06 Sep 2024 15:28:44 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1725661723; x=1726266523;
        h=content-transfer-encoding:cc:to:subject:message-id:date:from
         :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=ZxN5J7058hwez0kyVB857a4f13F0NHoHHjEfJ4ZYx6I=;
        b=wAi20JAv8gUDbUjMUrTV/ApMWofFl6t+P+TbGhjnJ4L0AeD5XiHznJih/FurnsK2Tr
         /tRwehEOr3GiWgk0u3Cra+9+jEdUyBNB9B4xJuHM/EVmDQ0qhCbnhh07wPXG9GKrY+i+
         n6YnUJAU7V/PdnWQ3ApgY+8XzHoeV0XUFe7tfM3v90fqwT18wp82nOsrkKJYTN0IS1pW
         wIzDOm0MK/y6xVopsNPyWXIS1QaT1eA4sO/9qjR4X0x02pkBP9+u5ySFaK6JC1+SBBd7
         fTCTtiSG2Bxzg9VX06hcfJNrvgom2emnuJhKZBn3AfS9Jg79Guy05HbkvaXdxVXGK/Bl
         lWXQ==
X-Forwarded-Encrypted: i=1; AJvYcCXAhcQMC7v9Sccwi9y6H2mhuDaR9bk3KM+FHyn3Y5x0rWbgaFbSFZdS5mvQn31z7dN8QsFgLC59R9W2@freebsd.org
X-Gm-Message-State: AOJu0Yyq6/Fx/8hrc+o4vUVoHilBz2PwrIvSMGmLPVr+H4J1RRegvnXH
	MpSAEepxrjF7mMRZxxITkAsENTEpc+ZaFZU+XmaBJ45ud3BKuCOA7m5ohsODd5m9aCXO3QSW0LM
	bcnedsIn/lHRpxIYs0RTwH0oK7f7KGX6A
X-Google-Smtp-Source: AGHT+IGaxVqZJ1m3gx0CeQZVuHrQJ2QxzuY3Jd88Ic0EtxGD5pJ0+IzeTnUVfdh0w6Mie8Fqq+lJJE4cbPOIUtLE6Do=
X-Received: by 2002:a05:6102:c0f:b0:49b:e3fd:b6d0 with SMTP id
 ada2fe7eead31-49bedb7464cmr497225137.5.1725661723000; Fri, 06 Sep 2024
 15:28:43 -0700 (PDT)
List-Id: Filesystems 
List-Archive: https://lists.freebsd.org/archives/freebsd-fs
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-fs@FreeBSD.org
MIME-Version: 1.0
References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com>
  
  
 <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com> 
 <0CF1E2D7-6C82-4A8B-82C3-A5BF1ED939CF@distal.com> 
 <29003A7C-745D-4A06-8558-AE64310813EA@distal.com> <42346193-AD06-4D26-B0C6-4392953D21A3@gmail.com>
  
  <50B791D8-F0CC-431E-93B8-834D57AB3C14@gmail.com>
  
 
In-Reply-To: 
From: Alan Somers 
Date: Fri, 6 Sep 2024 16:28:31 -0600
Message-ID: 
Subject: Re: Unable to replace drive in raidz1
To: Chris Ross 
Cc: Wes Morgan , FreeBSD Filesystems 
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spamd-Bar: --
X-Spamd-Result: default: False [-2.90 / 15.00];
	NEURAL_HAM_LONG(-1.00)[-1.000];
	NEURAL_HAM_SHORT(-1.00)[-1.000];
	NEURAL_HAM_MEDIUM(-1.00)[-0.999];
	FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com];
	R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c];
	MIME_GOOD(-0.10)[text/plain];
	DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), No valid DKIM,none];
	FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com];
	RCPT_COUNT_THREE(0.00)[3];
	FREEMAIL_CC(0.00)[gmail.com,freebsd.org];
	TO_DN_ALL(0.00)[];
	ARC_NA(0.00)[];
	RCVD_TLS_LAST(0.00)[];
	MIME_TRACE(0.00)[0:+];
	MISSING_XM_UA(0.00)[];
	FREEFALL_USER(0.00)[asomers];
	FREEMAIL_ENVFROM(0.00)[gmail.com];
	PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org];
	TO_MATCH_ENVRCPT_SOME(0.00)[];
	FROM_HAS_DN(0.00)[];
	RCVD_COUNT_ONE(0.00)[1];
	MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org];
	TAGGED_RCPT(0.00)[freebsd];
	R_DKIM_NA(0.00)[];
	ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US];
	RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.217.46:from];
	RCVD_IN_DNSWL_NONE(0.00)[209.85.217.46:from]
X-Rspamd-Queue-Id: 4X0rVN0sQxz4MWT

On Fri, Sep 6, 2024 at 4:22=E2=80=AFPM Chris Ross  wrote:
>
>
>
> > On Sep 6, 2024, at 18:02, Alan Somers  wrote:
> > Another user reports the same error message.  In their case, it's an
> > inappropriate error message from /sbin/zpool.  Can you try a "zpool
> > status -v" and "diskinfo -f /dev/da10"?  That will show you if you
> > have the same problem.  If your pool has a 512B block size but the new
> > disk is 4kn, then you cannot use it as a replacement.
> >
> > https://github.com/openzfs/zfs/issues/14730
>
> Yeah, I came across that in my searching.  Although, I thought people
> had found success adding =E2=80=9C-o ashift=3D9=E2=80=9D to the replace c=
ommand.
> I can confirm that this raidz1 is ashift=3D9, and the disks in it 512
> byte sectorsize.  The other two raidz=E2=80=99s are ashift=3D12.
>
> diskinfo shows 512 as sectorsize for one of the older disks
> (matching model to the removed da3), and 4096 sectorsize
> for the new disk.
>
> Is this totally impossible?  I know I have replaced smaller disks with
> much larger ones in the past, though it=E2=80=99s been a few years and
> I don=E2=80=99t remember the details.  Is there really no way to put a 4k
> block disk in a set with 512 byte blocks?  I thought they were still
> able to =E2=80=9Cact=E2=80=9D like 512-byte block disks?  (512e)
>
>                - Chris

If it's a 4kn disk, then what you ask for is impossible.  If it's
512e, then it should've already worked.  With diskinfo, a 512e disk
will have 512 sectorsize and 4096 stripesize.  A 4kn disk will show
4096 sectorsize.  If you cannot obtain a 512n or 512e disk, then I'm
afraid that you'll have to recreate your whole pool.

-Alan

From nobody Fri Sep  6 22:53:26 2024
X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0s3C12Hpz5W9rW
	for ; Fri, 06 Sep 2024 22:53:43 +0000 (UTC)
	(envelope-from cross+freebsd@relay.distal.com)
Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(Client did not present a certificate)
	by mx1.freebsd.org (Postfix) with ESMTPS id 4X0s3B4T6Gz4RN1;
	Fri,  6 Sep 2024 22:53:42 +0000 (UTC)
	(envelope-from cross+freebsd@relay.distal.com)
Authentication-Results: mx1.freebsd.org;
	none
dkim-signature: v=1; a=rsa-sha256; d=relay.distal.com; s=mail;
	c=relaxed/relaxed; q=dns/txt; h=From:Subject:Date:Message-ID:To:CC:MIME-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References;
	bh=A03Lf8iBCQGvBUVLskf/eggG6HM0l6C+Q1kJtQFCBL4=;
	b=pLmP9YPoaDRbyNE9lpkrnD2cJ3ebD9BmYOa4641cz1RNl4aeN5cEQvLtfQj/TZPebESlurpjKnKRxb08FS1rqHTJq92EVpvQ7yL4DbUxJFMRCTGipT5ht2XiSCmoGpPICFyMjc3RRDyJd+IVr+Bb23v6Kn239YcuK8kAEUgwttxq0IMdM/IYceaqRyVdslOyBhj04oiCDrZ+sJTcQe7gUOpH9nRmwFQS+E2Q5CCSA1qHHo0LmLlfx3XP61
	pmrYJ6ZfSnNGWgsrTDWk9qgJS3lBaqMXKDZ86wGeCjP0/JUy1qw8HPMrqBFMB1lhMzDIsx6vNdSBxMuFUeG4DiTdDZVQ==
Received: from mail.distal.com (pool-108-51-233-124.washdc.fios.verizon.net [108.51.233.124])
 by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305
 bits=256) ; Fri, 6 Sep 2024 22:53:39 +0000
Received: from smtpclient.apple ( [2001:420:c0c4:1001::9f]) by tristain.distal.com
 (OpenSMTPD) with ESMTPSA id e351909a (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO);
 Fri, 6 Sep 2024 18:53:37 -0400 (EDT)
Content-Type: text/plain; charset=utf-8
List-Id: Filesystems 
List-Archive: https://lists.freebsd.org/archives/freebsd-fs
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-fs@FreeBSD.org
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3776.700.51\))
Subject: Re: Unable to replace drive in raidz1
From: Chris Ross 
In-Reply-To: 
Date: Fri, 6 Sep 2024 18:53:26 -0400
Cc: Wes Morgan , FreeBSD Filesystems 
Content-Transfer-Encoding: quoted-printable
Message-Id: 
References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> 
  
  <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com>
  <0CF1E2D7-6C82-4A8B-82C3-A5BF1ED939CF@distal.com>
  <29003A7C-745D-4A06-8558-AE64310813EA@distal.com>
 <42346193-AD06-4D26-B0C6-4392953D21A3@gmail.com> 
  
 <50B791D8-F0CC-431E-93B8-834D57AB3C14@gmail.com> 
  
 
To: Alan Somers 
X-Mailer: Apple Mail (2.3776.700.51)
X-Spamd-Bar: ----
X-Rspamd-Pre-Result: action=no action;
	module=replies;
	Message is reply to one we originated
X-Spamd-Result: default: False [-4.00 / 15.00];
	REPLY(-4.00)[];
	TAGGED_FROM(0.00)[freebsd];
	ASN(0.00)[asn:3842, ipnet:168.235.92.0/22, country:US]
X-Rspamd-Queue-Id: 4X0s3B4T6Gz4RN1



> On Sep 6, 2024, at 18:28, Alan Somers  wrote:
>=20
> If it's a 4kn disk, then what you ask for is impossible.  If it's
> 512e, then it should've already worked.  With diskinfo, a 512e disk
> will have 512 sectorsize and 4096 stripesize.  A 4kn disk will show
> 4096 sectorsize.  If you cannot obtain a 512n or 512e disk, then I'm
> afraid that you'll have to recreate your whole pool.

Hmm.  Okay.  Though there are 6TB disks in the other array that are
quite old, I=E2=80=99m pretty sure could work in 512e mode, but diskinfo =
shows
a 4096 sectorsize for those too.   Oh well, I could be wrong about
how they work.

I can definately find another drive, likely even of the correct size.
But, that breaks my plan to increase the size of the pool by
replacing disks.  Am I correct that FreeBSD offers no way to
remove raidz vdev=E2=80=99s from a pool?  If it could I could try to =
construct
some sort of new pool a piece at a time.  But if it does not, then
there is nothing I can do I assume.  :-/

Happy to hear other thoughts.  I could move all of the data to
another system and rebuild it with a new vdev, but I=E2=80=99m not sure
if I care to grow the FS enough to do that.  :-)

                 - Chris



From nobody Fri Sep  6 23:09:14 2024
X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0sPP2Hr8z5WBvV
	for ; Fri, 06 Sep 2024 23:09:29 +0000 (UTC)
	(envelope-from asomers@gmail.com)
Received: from mail-ot1-f41.google.com (mail-ot1-f41.google.com [209.85.210.41])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "WR4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4X0sPN0tYGz4SGL
	for ; Fri,  6 Sep 2024 23:09:28 +0000 (UTC)
	(envelope-from asomers@gmail.com)
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=freebsd.org (policy=none);
	spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.210.41 as permitted sender) smtp.mailfrom=asomers@gmail.com
Received: by mail-ot1-f41.google.com with SMTP id 46e09a7af769-709346604a7so1704447a34.1
        for ; Fri, 06 Sep 2024 16:09:28 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1725664167; x=1726268967;
        h=content-transfer-encoding:cc:to:subject:message-id:date:from
         :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=Magv/kTrb2Khi/ybDJpsnT8FbNeiTE6KnVwBUYeu3Tg=;
        b=VgxCQVoQPC/MqRWhPJRdHt8NZhBYK0Q1Yt7DKvBRDtPWmFVI+YwRx8XcR+niBrhbll
         h/y6cHjfn+kk3ubwTy4zX9vSq/AUfjVqRrSt4koRQw83XY0RqkKC/abXshAnTawqHY4Q
         uY6Knd4+lJWjR1/lRhiQOkkbeHcEjtbdMzfKWHrOW+JN7dqCazIrdvyL/sHBn9Y3+rbE
         rNWWBgRFGAxae4xI2vc1M/pbbHO0q1W4tn46oe/c9cDGaYHIQTadY14liLi/DYBB5pX6
         vVXVULHnaYga047sVvT02NJFG+8y4FO+AUZCT9rAwZiC46RPfM5YQ8BQgf2K5wLAGfr/
         KstQ==
X-Forwarded-Encrypted: i=1; AJvYcCWEXpTr0cT2R59Smja6+bfK/c4xKulYiQgSMa6PQKfOxBN7QUr7LF7XgqWkrHjD/hrM36/vEgSGncjn@freebsd.org
X-Gm-Message-State: AOJu0YyY3XFl3pB1IA6khVw7p9wViNtaKry11HZ7kGyLsA307r910c66
	xzyk9UlS0fcqRBUjj0ZgnMLMjryJiNjq2dPJQExrFUTQwHhX3UOCltk8UWeIg3KkLOgw0Kw/hwm
	/T0T3RBYWeihEOGG5weR/r720g6c=
X-Google-Smtp-Source: AGHT+IF4ULGCY/LoLtn9l2TD8W/EjRu4rywsAN7o/Ag8mJzv6AKl4GLgibKBd9sDtirw66t8CHeXWIAlDnqMUmcBG5M=
X-Received: by 2002:a05:6830:7205:b0:710:d773:e35c with SMTP id
 46e09a7af769-710d773e540mr1089770a34.2.1725664166637; Fri, 06 Sep 2024
 16:09:26 -0700 (PDT)
List-Id: Filesystems 
List-Archive: https://lists.freebsd.org/archives/freebsd-fs
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-fs@FreeBSD.org
MIME-Version: 1.0
References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com>
  
  
 <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com> 
 <0CF1E2D7-6C82-4A8B-82C3-A5BF1ED939CF@distal.com> 
 <29003A7C-745D-4A06-8558-AE64310813EA@distal.com> <42346193-AD06-4D26-B0C6-4392953D21A3@gmail.com>
  
  <50B791D8-F0CC-431E-93B8-834D57AB3C14@gmail.com>
  
  
 
In-Reply-To: 
From: Alan Somers 
Date: Fri, 6 Sep 2024 17:09:14 -0600
Message-ID: 
Subject: Re: Unable to replace drive in raidz1
To: Chris Ross 
Cc: Wes Morgan , FreeBSD Filesystems 
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spamd-Bar: -
X-Spamd-Result: default: False [-1.56 / 15.00];
	NEURAL_HAM_LONG(-1.00)[-0.999];
	NEURAL_HAM_SHORT(-0.82)[-0.816];
	FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com];
	NEURAL_SPAM_MEDIUM(0.26)[0.259];
	R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17];
	DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), No valid DKIM,none];
	RWL_MAILSPIKE_GOOD(-0.10)[209.85.210.41:from];
	MIME_GOOD(-0.10)[text/plain];
	FREEMAIL_CC(0.00)[gmail.com,freebsd.org];
	RCPT_COUNT_THREE(0.00)[3];
	TO_DN_ALL(0.00)[];
	RCVD_TLS_LAST(0.00)[];
	MIME_TRACE(0.00)[0:+];
	FREEMAIL_ENVFROM(0.00)[gmail.com];
	ARC_NA(0.00)[];
	MISSING_XM_UA(0.00)[];
	RCVD_COUNT_ONE(0.00)[1];
	PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org];
	TO_MATCH_ENVRCPT_SOME(0.00)[];
	FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com];
	RCVD_IN_DNSWL_NONE(0.00)[209.85.210.41:from];
	MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org];
	TAGGED_RCPT(0.00)[freebsd];
	R_DKIM_NA(0.00)[];
	FREEFALL_USER(0.00)[asomers];
	ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US];
	FROM_HAS_DN(0.00)[]
X-Rspamd-Queue-Id: 4X0sPN0tYGz4SGL

On Fri, Sep 6, 2024 at 4:53=E2=80=AFPM Chris Ross  wrote:
>
>
>
> > On Sep 6, 2024, at 18:28, Alan Somers  wrote:
> >
> > If it's a 4kn disk, then what you ask for is impossible.  If it's
> > 512e, then it should've already worked.  With diskinfo, a 512e disk
> > will have 512 sectorsize and 4096 stripesize.  A 4kn disk will show
> > 4096 sectorsize.  If you cannot obtain a 512n or 512e disk, then I'm
> > afraid that you'll have to recreate your whole pool.
>
> Hmm.  Okay.  Though there are 6TB disks in the other array that are
> quite old, I=E2=80=99m pretty sure could work in 512e mode, but diskinfo =
shows
> a 4096 sectorsize for those too.   Oh well, I could be wrong about
> how they work.
>
> I can definately find another drive, likely even of the correct size.
> But, that breaks my plan to increase the size of the pool by
> replacing disks.  Am I correct that FreeBSD offers no way to
> remove raidz vdev=E2=80=99s from a pool?

Correct.

> If it could I could try to construct
> some sort of new pool a piece at a time.  But if it does not, then
> there is nothing I can do I assume.  :-/
>
> Happy to hear other thoughts.  I could move all of the data to
> another system and rebuild it with a new vdev, but I=E2=80=99m not sure
> if I care to grow the FS enough to do that.  :-)
>
>                  - Chris
>
>

From nobody Sun Sep  8 21:00:33 2024
X-Original-To: fs@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X22Rj4mwRz5W6ZT
	for ; Sun, 08 Sep 2024 21:00:33 +0000 (UTC)
	(envelope-from bugzilla-noreply@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (4096 bits) client-digest SHA256)
	(Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4X22Rj2skRz4Pn4
	for ; Sun,  8 Sep 2024 21:00:33 +0000 (UTC)
	(envelope-from bugzilla-noreply@FreeBSD.org)
ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1725829233; a=rsa-sha256; cv=none;
	b=IAbGhknZZRTHOr22c1eI4NAD2OW8vz23yuGalVoJL49+mXjOXY4eqxO+RJqcZOq5FdKN7K
	Ki5mIqTml5o2KeZ+FUQiCIlzeE6pYlbJMpn44RjCJUffNjQsIK3OZTEecumXE6MShIMZeg
	nYuzw1ziF/GDRHpAV0wnULiVKKsIdcaHq0ViiXF/2fBhIek+M6CueKKBv/kW8FoR83stlh
	DqJxC+RmRy9BlYkeqx4kvpD9J5uvnP2rqCneeRWDj1W/Uf5ZHc11GmklJCo31rC6km2oB2
	MNPyEyUC+ek8dK8nyt1UHV95hfag0ZTi8HMZ/aKFzKcZFmXQcWOA5EclwKPf1w==
ARC-Authentication-Results: i=1;
	mx1.freebsd.org;
	none
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org;
	s=dkim; t=1725829233;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:mime-version:mime-version:content-type:content-type:
	 content-transfer-encoding:content-transfer-encoding;
	bh=EPyZpT5jiz/Zh4yVrTfBgLrBfQpe4j7NqnVzhD/2pbE=;
	b=DXu8Vfi44eCrfAwUu6oxKuXC0y4LzF6JCsNcKu9msYq4LuAT1L9tCUqGPWZxfPoiCzG0Ba
	MHBwsxttap5KKGxTFZ1uZG5OT1X+soAW+r6GcxAZnOmVW8CR30QJ7tKiclN9e9x3QD7FBW
	Kxi6kmcI+w9/Zm2kvn3SOuiHec6CY4ag9obkjGYeEmUWQ6UNR0e0BmPc1Ipm3dCCk37M9C
	LqFpx1nzoZamzbizEVr72f6BBJ0UABvykixPD/IooaS9MyhphUM7CI9Is2gaauI1Fqxab2
	NdBiFhAVPy0otRrI0gWZkYSQRXiSa8s7DJqGCrqZVKQ5Xs7KI61tchJtN8FjDA==
Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
	(Client did not present a certificate)
	by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4X22Rj2JzFzZrY
	for ; Sun,  8 Sep 2024 21:00:33 +0000 (UTC)
	(envelope-from bugzilla-noreply@FreeBSD.org)
Received: from kenobi.freebsd.org ([127.0.1.5])
	by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 488L0XeT070668
	for ; Sun, 8 Sep 2024 21:00:33 GMT
	(envelope-from bugzilla-noreply@FreeBSD.org)
Received: (from bugzilla@localhost)
	by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 488L0Xcx070663
	for fs@FreeBSD.org; Sun, 8 Sep 2024 21:00:33 GMT
	(envelope-from bugzilla-noreply@FreeBSD.org)
Message-Id: <202409082100.488L0Xcx070663@kenobi.freebsd.org>
X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f
From: bugzilla-noreply@FreeBSD.org
To: fs@FreeBSD.org
Subject: Problem reports for fs@FreeBSD.org that need special attention
Date: Sun, 8 Sep 2024 21:00:33 +0000
List-Id: Filesystems 
List-Archive: https://lists.freebsd.org/archives/freebsd-fs
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-fs@FreeBSD.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="17258292330.d5d9f.65988"
Content-Transfer-Encoding: 7bit


--17258292330.d5d9f.65988
Date: Sun, 8 Sep 2024 21:00:33 +0000
MIME-Version: 1.0
Content-Type: text/plain; charset="UTF-8"

To view an individual PR, use:
  https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id).

The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.

Status      |    Bug Id | Description
------------+-----------+---------------------------------------------------
Open        |    231794 | zfs: Panic due to ARC related KVA memory exhausti 
Open        |    251035 | ZFS: Allow 64 bit ZFS to support 32 bit ioctls (W 
Open        |    266409 | 13.1-RELEASE amd64 crashes in: sa_handle_destroy  
Open        |    268162 | zfskeys_enable: each successful load of a key is  
Open        |    269503 | docs.freebsd.org: default vfs.zfs.arc.meta_limit  
Open        |    271384 | zfs_load is not suitably documented               
Open        |    226130 | ZFS: solaris assert: zrl->zr_refcount == 0 (0x1 = 

7 problems total for which you should take action.

--17258292330.d5d9f.65988
Date: Sun, 8 Sep 2024 21:00:33 +0000
MIME-Version: 1.0
Content-Type: text/html; charset="UTF-8"


The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.

Status      |    Bug Id | Description
------------+-----------+---------------------------------------------------
Open        |    231794 | zfs: Panic due to ARC related KVA memory exhausti
Open        |    251035 | ZFS: Allow 64 bit ZFS to support 32 bit ioctls (W
Open        |    266409 | 13.1-RELEASE amd64 crashes in: sa_handle_destroy 
Open        |    268162 | zfskeys_enable: each successful load of a key is 
Open        |    269503 | docs.freebsd.org: default vfs.zfs.arc.meta_limit
Open        |    271384 | zfs_load is not suitably documented
Open        |    226130 | ZFS: solaris assert: zrl->zr_refcount == 0 (0x1 =

7 problems total for which you should take action.


--17258292330.d5d9f.65988--