From owner-freebsd-fs@freebsd.org Mon Jan 18 12:53:09 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 495B34EAB6E for ; Mon, 18 Jan 2021 12:53:09 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DKBXh450dz3t4W for ; Mon, 18 Jan 2021 12:53:08 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-pl1-f180.google.com with SMTP id be12so8610646plb.4 for ; Mon, 18 Jan 2021 04:53:08 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=59Rr+q+Req2J5K5AlZjstLAWzFawKDSNB+UxEHgzGlY=; b=nb1YcVQ3Cp3UJgd690OMMFZx5PNGgEeLfthQq49QXzRmdK8FOGHtUNKSAYDHA8sIKM zJAXpMr+LuOPgFRR1Puwd5EzpoXjrGcQONL2JG7NkQmiKkktnW8Yyt2ImT4UDOvYxlup 2JLOtJR4LhA+cxT4OMxWVSyHe9pQEFWjWuihk291rXiYFFbPw4ZBYX2eFKWDLfSb0lsw MYBClK6LKJ0Yj2MRHM3HWtBGRep7mUCMr5qyDcrPLK6gl1AX0rOzpKUSXjynz/oDCMgb 6kjjwUctlfZ6M3ZZhOUttVp7Lso+UCr8BMO5Yu+iyXJcNJg76r+2uYd/gNbgiE5la+fA bHag== X-Gm-Message-State: AOAM53363+tPG7af3AQt23GpGQrVK5cXeXzoRAvNPcGOsH13/YMLoA4O Lg9Gd1CbjJh1yk4NbTXQRCs6neaqAth7yA== X-Google-Smtp-Source: ABdhPJzWvc36VPhoWNWjvOFCJtuIhTVSfBuIGCiSKpUqItVILCrpd9UKQsmKlMT51fBPiRWnw5S9Ow== X-Received: by 2002:a17:90a:6587:: with SMTP id k7mr26726786pjj.154.1610974386480; Mon, 18 Jan 2021 04:53:06 -0800 (PST) Received: from [192.168.0.24] ([195.64.148.76]) by smtp.googlemail.com with ESMTPSA id e72sm8089344pfh.199.2021.01.18.04.53.03 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 18 Jan 2021 04:53:05 -0800 (PST) Subject: Re: zpool remove not working for metadata special devices To: John Delisle , freebsd-fs References: From: Andriy Gapon Message-ID: <07c28c11-325b-d05c-f078-e4a9b3f0be54@FreeBSD.org> Date: Mon, 18 Jan 2021 14:52:59 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0 Thunderbird/78.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4DKBXh450dz3t4W X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[209.85.214.180:from]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com,freebsd.org]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; R_DKIM_NA(0.00)[]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[209.85.214.180:from]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; SPAMHAUS_ZRD(0.00)[209.85.214.180:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[209.85.214.180:from]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jan 2021 12:53:09 -0000 On 2020-12-28 05:04, John Delisle wrote: > To experiment, I've completed this same procedure using Ubuntu 20.10 (on > the same "hardware", both are identically configured VMs in Azure) . > > The steps above work fine with ZFS on Ubuntu, and I can successfully remove > both data vdev mirrors and the special vdev mirror. This makes me think my > syntax is correct at least.. but for whatever reason the exact same > procedure fails on FreeBSD (both removing a mirror and removing the special > mirror fail with the same error on FreeBSD). I don't think that you did anything wrong and the syntax is certainly correct. I can reproduce the problem as well. Note that if you insert a reboot between the add special and the remove, then the latter works fine, at least it does for me. It looks like there is an issue with recording of spa_min_ashift internal parameter when a special device is first added. > On Sun, Dec 27, 2020 at 6:55 PM John Delisle wrote: > >> I have a pool of mirrors, and added a mirrored special device. Although >> documentation suggests it should be removable, I cannot get zpool to do so. >> >> All top-level vdevs are mirrors, and all have the same sector size and >> ashift. No raidz. >> >> ## Current 12.2 p2 >> root@jmdtest:/ # freebsd-version >> 12.2-RELEASE-p2 >> >> ## Disk info >> root@jmdtest:/ # diskinfo /dev/da[2-9] /dev/da[0-9][0-9] >> /dev/da2 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da3 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da4 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da5 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da6 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da7 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da8 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da9 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da10 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da11 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da12 512 68719476736 134217728 4096 0 >> 8354 255 63 >> /dev/da13 512 68719476736 134217728 4096 0 >> 8354 255 63 >> >> >> ## Create the pool: >> zpool create nebula mirror da2 da3 mirror da4 da5 mirror da6 da7 mirror >> da8 da9 mirror da10 da11 >> >> ## Add the special mirror >> zpool add nebula special mirror da12 da13 >> >> ## zpool status >> root@jmdtest:/ # zpool list -v nebula >> NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP >> HEALTH ALTROOT >> nebula 1.30T 2.57G 1.30T - - 0% 0% 1.00x >> ONLINE - >> mirror 254G 492M 254G - - 0% 0.18% >> da2 - - - - - - - >> da3 - - - - - - - >> mirror 254G 501M 254G - - 0% 0.19% >> da4 - - - - - - - >> da5 - - - - - - - >> mirror 254G 596M 253G - - 0% 0.22% >> da6 - - - - - - - >> da7 - - - - - - - >> mirror 254G 481M 254G - - 0% 0.18% >> da8 - - - - - - - >> da9 - - - - - - - >> mirror 254G 561M 253G - - 0% 0.21% >> da10 - - - - - - - >> da11 - - - - - - - >> special - - - - - - >> mirror 63.5G 2.62M 63.5G - - 0% 0.00% >> da12 - - - - - - - >> da13 - - - - - - - >> root@jmdtest:/ # >> >> ## Remove the special mirror >> root@jmdtest:/ # zpool remove nebula mirror-5 >> cannot remove mirror-5: invalid config; all top-level vdevs must have the >> same sector size and not be raidz. >> >> ## All mirrors, and all the same ashift: >> root@jmdtest:/ # zdb -C | grep -e child -e ashift >> vdev_children: 6 >> children[0]: >> ashift: 12 >> children[0]: >> children[1]: >> children[1]: >> ashift: 12 >> children[0]: >> children[1]: >> children[2]: >> ashift: 12 >> children[0]: >> children[1]: >> children[3]: >> ashift: 12 >> children[0]: >> children[1]: >> children[4]: >> ashift: 12 >> children[0]: >> children[1]: >> children[5]: >> ashift: 12 >> children[0]: >> children[1]: >> root@jmdtest:/ # >> >> What am I doing wrong? -- Andriy Gapon