From owner-freebsd-amd64@FreeBSD.ORG Sun Jul 11 23:00:18 2010 Return-Path: Delivered-To: freebsd-amd64@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D88B41065673 for ; Sun, 11 Jul 2010 23:00:18 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 9BE2A8FC17 for ; Sun, 11 Jul 2010 23:00:18 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o6BN0IJC030805 for ; Sun, 11 Jul 2010 23:00:18 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o6BN0IwZ030804; Sun, 11 Jul 2010 23:00:18 GMT (envelope-from gnats) Resent-Date: Sun, 11 Jul 2010 23:00:18 GMT Resent-Message-Id: <201007112300.o6BN0IwZ030804@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-amd64@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Stuart Morgan Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 02F88106566C for ; Sun, 11 Jul 2010 22:59:39 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21]) by mx1.freebsd.org (Postfix) with ESMTP id AF34E8FC08 for ; Sun, 11 Jul 2010 22:59:38 +0000 (UTC) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o6BMxboq016584 for ; Sun, 11 Jul 2010 22:59:37 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.14.3/8.14.3/Submit) id o6BMxbFX016583; Sun, 11 Jul 2010 22:59:37 GMT (envelope-from nobody) Message-Id: <201007112259.o6BMxbFX016583@www.freebsd.org> Date: Sun, 11 Jul 2010 22:59:37 GMT From: Stuart Morgan To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 X-Mailman-Approved-At: Mon, 12 Jul 2010 02:55:18 +0000 Cc: Subject: amd64/148504: ZFS' zpool does not allow replacing drives to be offlined or removed X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Jul 2010 23:00:18 -0000 >Number: 148504 >Category: amd64 >Synopsis: ZFS' zpool does not allow replacing drives to be offlined or removed >Confidential: no >Severity: non-critical >Priority: medium >Responsible: freebsd-amd64 >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sun Jul 11 23:00:18 UTC 2010 >Closed-Date: >Last-Modified: >Originator: Stuart Morgan >Release: 8.1-PRERELEASE >Organization: >Environment: FreeBSD vault.internal 8.1-PRERELEASE FreeBSD 8.1-PRERELEASE #4: Thu Jul 1 09:47:23 UTC 2010 root@vault.internal:/usr/obj/usr/src/sys/vault amd64 >Description: Consider the ZFSv14 array as shown below: $ zpool status pool: vault state: DEGRADED scrub: none requested config: NAME STATE READ WRITE CKSUM vault DEGRADED 0 0 0 mirror ONLINE 0 0 0 da0.eli ONLINE 0 0 0 da8.eli ONLINE 0 0 0 ad4.eli ONLINE 0 0 0 mirror DEGRADED 0 0 0 da1.eli ONLINE 0 0 0 replacing UNAVAIL 0 32.0K 0 insufficient replicas 176228571956779269 UNAVAIL 0 34.6K 0 was /dev/ad6.eli 1650754158582652028 UNAVAIL 0 34.6K 0 was /dev/da99.eli da9.eli ONLINE 0 0 0 errors: No known data errors It is not possible to remove 176228571956779269 and 1650754158582652028. [root@vault ~]# zpool offline vault 176228571956779269 zcannot offline 176228571956779269: no valid replicas [root@vault ~]# zpool offline vault 1650754158582652028 cannot offline 1650754158582652028: no valid replicas Other combinations of detach, replace etc do not work either. The problem seems to be that the zpool binary will not remove a replacing drive when the replacement drive is removed or cannot complete the replacement operation. Output of zdb: vault version=14 name='vault' state=0 txg=174072 pool_guid=14228437791114878229 hostid=4269824854 hostname='vault.internal' vdev_tree type='root' id=0 guid=14228437791114878229 children[0] type='mirror' id=0 guid=15203813302661836458 metaslab_array=23 metaslab_shift=34 ashift=12 asize=2000394125312 is_log=0 children[0] type='disk' id=0 guid=7238937831460639650 path='/dev/da0.eli' whole_disk=0 DTL=51 children[1] type='disk' id=1 guid=14125934351009663627 path='/dev/da8.eli' whole_disk=0 DTL=53 children[2] type='disk' id=2 guid=8553277371515810839 path='/dev/ad4.eli' whole_disk=0 DTL=169 children[1] type='mirror' id=1 guid=14672070090293526603 metaslab_array=118 metaslab_shift=34 ashift=12 asize=2000394125312 is_log=0 children[0] type='disk' id=0 guid=4020825290461516758 path='/dev/da1.eli' whole_disk=0 DTL=123 children[1] type='replacing' id=1 guid=496235429895718168 whole_disk=0 children[0] type='disk' id=0 guid=176228571956779269 path='/dev/ad6.eli' whole_disk=0 not_present=1 DTL=54 children[1] type='disk' id=1 guid=1650754158582652028 path='/dev/da99.eli' whole_disk=0 not_present=1 DTL=168 children[2] type='disk' id=2 guid=5694121814811722857 path='/dev/da9.eli' whole_disk=0 DTL=120 >How-To-Repeat: 1) Create a pool 2) Create 2 mirrored vdevs 3) Replace one of the devices with another 4) Interrupt the replacement process so that both devices are now unavailable 5) Run zpool status and attempt to remove them. >Fix: N/A >Release-Note: >Audit-Trail: >Unformatted: