From owner-freebsd-fs@freebsd.org Wed Jun 7 08:26:54 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D5948BFE72B for ; Wed, 7 Jun 2017 08:26:54 +0000 (UTC) (envelope-from mckay@freebsd.org) Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by mx1.freebsd.org (Postfix) with ESMTP id 35DD266C8F; Wed, 7 Jun 2017 08:26:53 +0000 (UTC) (envelope-from mckay@freebsd.org) Message-Id: <0410af$1dldvp4@ipmail04.adl6.internode.on.net> X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2A1DwAXtzdZ/2O8Ag5eHAEBBAEBCgEBgmpBLVIQgQ2DJYtZlCCFCI9mLIVyBAICgmtYAQIBAQEBAQJrKIUZBicvIxAIA0YhGB4GE4oTAxQRr3M6h0ANhDIBAQEHjAmCWCmBL4Negk8FiUWGbYFdhHeGdzuHJoc1hmOJLIZNi0GJJFeBCoEBCEaFBxyBdi42iVIBAgM Received: from ppp14-2-188-99.bras1.hba2.internode.on.net (HELO localhost) ([14.2.188.99]) by ipmail04.adl6.internode.on.net with ESMTP; 07 Jun 2017 17:51:11 +0930 From: Stephen McKay To: =?UTF-8?Q?=c5=81ukasz_W=c4=85sikowski?= cc: freebsd-fs@freebsd.org, mckay@FreeBSD.org Subject: Re: Problem with zpool remove of log device References: <9188a169-cd81-f64d-6b9e-0e3c6b4af1bb@wasikowski.net> In-Reply-To: <9188a169-cd81-f64d-6b9e-0e3c6b4af1bb@wasikowski.net> from =?UTF-8?Q?=c5=81ukasz_W=c4=85sikowski?= at "Fri, 26 May 2017 11:40:06 +0200" Date: Wed, 07 Jun 2017 18:21:09 +1000 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jun 2017 08:26:54 -0000 On Friday, 26th May 2017, lukasz@wasikowski.net wrote: >I cant remove log device from pool - operation ends ok, but log device >is still in the pool (bug?). > ># uname -a >FreeBSD xxx.yyy.com 11.0-STABLE FreeBSD 11.0-STABLE #0 r316543: Thu Apr >6 08:22:43 CEST 2017 root@xxx.yyy.com:/usr/obj/usr/src/sys/YYY amd64 > ># zpool status tank >[..snip..] > > NAME STATE READ WRITE CKSUM > tank ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > ada2p3 ONLINE 0 0 0 > ada3p3 ONLINE 0 0 0 > logs > mirror-1 ONLINE 0 0 0 > gpt/tankssdzil0 ONLINE 0 0 0 block size: 512B configured, 4096B native > gpt/tankssdzil1 ONLINE 0 0 0 block size: 512B configured, 4096B native >When I try to remove log device operation ends without errors: > ># zpool remove tank mirror-1; echo $? >0 > >But the log device is still there: >[..snip..] >I'd like to remove it - how should I proceed? Does your system still write to the log? Use "zfs iostat -v 1" to check. I think it is probably no longer be in use and only the final disconnection failed. What does "zpool list -v" tell you? If you have a non-zero ALLOC column for your log mirror and the log is no longer being used then you may have hit an accounting bug in zfs that the zfsonlinux people ran into a while ago. I had this problem when I tried to remove a log mirror from a pool I have been using for years. I solved it by tweaking the zfsonlinux hack a bit and slotting it into 9.3. If you apply this hack be sure to have a full backup first! When I used it, I did my backup and a scrub then booted the hacked kernel, issued the zfs remove command (which succeeded), reverted the kernel, then scrubbed again. All went well. Good luck! Here's the patch against 9.3 (should be close even for 11.0): Index: sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c =================================================================== --- sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c (revision 309860) +++ sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c (working copy) @@ -5446,6 +5446,18 @@ ASSERT(vd == vd->vdev_top); /* + * slog stuck hack - barnes333@gmail.com + * https://github.com/zfsonlinux/zfs/issues/1422 + */ + if (vd->vdev_islog && vd->vdev_removing + && vd->vdev_state == VDEV_STATE_OFFLINE + && vd->vdev_stat.vs_alloc > 0) { + printf("ZFS: slog stuck hack - clearing vs_alloc: %llu\n", + (unsigned long long)vd->vdev_stat.vs_alloc); + vd->vdev_stat.vs_alloc = 0; + } + + /* * Only remove any devices which are empty. */ if (vd->vdev_stat.vs_alloc != 0) Cheers, Stephen.