From owner-freebsd-fs@freebsd.org Sun Dec 27 15:50:36 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4FC2D4BED06 for ; Sun, 27 Dec 2020 15:50:36 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4D3lWZ552Gz4lg5 for ; Sun, 27 Dec 2020 15:50:34 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.16.1/8.16.1) with ESMTPS id 0BRFoK4c042483 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sun, 27 Dec 2020 17:50:24 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 0BRFoK4c042483 Received: (from kostik@localhost) by tom.home (8.16.1/8.16.1/Submit) id 0BRFoKTI042465; Sun, 27 Dec 2020 17:50:20 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 27 Dec 2020 17:50:20 +0200 From: Konstantin Belousov To: Rick Macklem Cc: J David , "freebsd-fs@freebsd.org" Subject: Re: Major issues with nfsv4 Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 4D3lWZ552Gz4lg5 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; HAS_XAW(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[2001:470:d5e7:1::1:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.995]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; TAGGED_RCPT(0.00)[]; R_DKIM_NA(0.00)[]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[2001:470:d5e7:1::1:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Dec 2020 15:50:36 -0000 On Sat, Dec 26, 2020 at 11:10:01PM +0000, Rick Macklem wrote: > Although you have not posted the value for > vfs.deferred_inact, if that value has become > relatively large when the problem occurs, > it might support this theory w.r.t. how this > could happen. > > Two processes in different jails do "stat()" or > similar on the same file in the NFS file system > at basically the same time. > --> They both get shared locked nullfs vnodes, > both of which hold shared locks on the > same lowervp (the NFS client one). > --> They both do vput() on these nullfs vnodes > concurrently. > > If both call vput_final() concurrently, I think both > could have the VOP_LOCK(vp, LK_UPGRADE | LK_INTERLOCK | > LK_NOWAIT) at line #3147 fail, since this will call null_lock() > for both nullfs vnodes and then both null_lock() calls will > do VOP_LOCK(lvp, flags); at line #705. > --> The call fails for both processes, since the other one still > holds the shared lock on the NFS client vnode. > > If I have this right, then both processes end up calling > vdefer_inactive() for the upper nullfs vnodes. > > If this is what is happening, then when does the VOP_INACTIVE() > get called for the lowervp? > > I see vfs_deferred_inactive() in sys/kern/vfs_subr.c, but I do not > know when/how it gets called? Right, vfs_deferred_inactive() is one way which tries to handle missed inactivations. If upon vput() the lock is only shared and upgrade failed, vnode is marked as VI_OWEINACT and put onto 'lazy' list, processed by vfs_sync(MNT_LAZY). It is typically called from syncer, which means each 60 secs. There, if the vnode is still unreferenced, it is inactivated. Another place where inactivation can occur is reclamation. There in vgonel(), we call VOP_INACTIVE() if VI_OWEINACT is set. In principle, this is redundand because correct filesystem must do the same cleanup (and more) at reclamation as at the inactivation. But we also call VOP_CLOSE(FNONBLOCK) before VOP_RECLAIM(). Looking at this from another angle, if inactivation for NFSv4 vnodes is not called longer than 2 minutes, perhaps there is a reference leak. It is not due to VFS forgetting about due VOP_INACTIVE() call. From owner-freebsd-fs@freebsd.org Mon Dec 28 00:55:17 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EF5804CCDC9 for ; Mon, 28 Dec 2020 00:55:17 +0000 (UTC) (envelope-from jdelisle@gmail.com) Received: from mail-pl1-x62d.google.com (mail-pl1-x62d.google.com [IPv6:2607:f8b0:4864:20::62d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D3zc467fbz4TrZ for ; Mon, 28 Dec 2020 00:55:16 +0000 (UTC) (envelope-from jdelisle@gmail.com) Received: by mail-pl1-x62d.google.com with SMTP id r4so4883341pls.11 for ; Sun, 27 Dec 2020 16:55:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=lqtYqKgbuUSzcYEBdO+W4JjUj88KpwnJgdqLkwafChc=; b=Xv+YMbcF9SPT+cZU1xBB4psmYqUtQi9mWhFS5qlQ3V+X6/PBttp9IkLBqvpBDNmb/4 gQR0WP/NuS2Hz1zSt/t4hdqqcMlbWe6fxJGbdozK8/xwskEHqxGpjGgFtPjfVjzkEyjz +cPyNyu68/w+PskZKu+GrqmGB0lwzLl9pNbFAEbZ3+RuN9yW8htqOcmRdrYLklTSFL+K NbBJQs5xkvpmteOJku1JN1M5WkGh0JEcZbCA1OJLgrL4psalNpveKSe1sr33i+ljU0Tp aYIggd9l4uwZ0BT7kMe3XoKIuCnEtnlQcspOlWF18OQc5clIVSYpe8I+lL0Hqbl57ZoC N0Og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=lqtYqKgbuUSzcYEBdO+W4JjUj88KpwnJgdqLkwafChc=; b=aa8bN5hFH/2WBq5zQoBnY/CNeeB8fzddRZS5UYrGcFqXWz9FXqdpQ8JUipLtSKjX3i qiuqOoQzvnjLxmSwzpwg5bKvDLL1H9LA/Wob4yQDu0FxgqAqGgdiKSjUJO1PdRoXJdjA 83hy5YaayJtJfjQi6uF4BmIi+gBRlfe6CCXThObIq52k5UOJUJh3hB+R3ygV0guTofW5 SUpHonXHhbDo1cudEeubT5Dcznqi5+SR6TCsV8uFb4kqwOQqBbJ2xhXRygGXLfd4wLFZ 3kG+4jB98NuLAw2UF/hvc3Xn74Gkz7OCl+s8HHm+XYnzT8fVrrUW9Ni+M5dj2dji9tev gshg== X-Gm-Message-State: AOAM530kuaOxJJwqMLp+1tGS1chBKc1Qr4h72mYGiHZA/Az6W8zbdMK2 g8sHZRPdLhY8bGK3N0gIsqinJ5LH/G4/TdIhxvB8LsxUXCTBpBbh X-Google-Smtp-Source: ABdhPJzwO+81mLdwUXommcPd196kDPUstwYyvjC/mVUpvSgvxif8ysi056vnBGa1AXtswBtr9F5nW5FHPycgCZkUqUU= X-Received: by 2002:a17:90a:1182:: with SMTP id e2mr18192175pja.152.1609116914901; Sun, 27 Dec 2020 16:55:14 -0800 (PST) MIME-Version: 1.0 From: John Delisle Date: Sun, 27 Dec 2020 18:55:02 -0600 Message-ID: Subject: zpool remove not working for metadata special devices To: freebsd-fs X-Rspamd-Queue-Id: 4D3zc467fbz4TrZ X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=Xv+YMbcF; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jdelisle@gmail.com designates 2607:f8b0:4864:20::62d as permitted sender) smtp.mailfrom=jdelisle@gmail.com X-Spamd-Result: default: False [-1.90 / 15.00]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MIME_BASE64_TEXT(0.10)[]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::62d:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::62d:from:127.0.2.255]; NEURAL_SPAM_SHORT(1.00)[1.000]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::62d:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Dec 2020 00:55:18 -0000 I have a pool of mirrors, and added a mirrored special device. Although documentation suggests it should be removable, I cannot get zpool to do so. All top-level vdevs are mirrors, and all have the same sector size and ashift. No raidz. ## Current 12.2 p2 root@jmdtest:/ # freebsd-version 12.2-RELEASE-p2 ## Disk info root@jmdtest:/ # diskinfo /dev/da[2-9] /dev/da[0-9][0-9] /dev/da2 512 274877906944 536870912 4096 0 33418 255 63 /dev/da3 512 274877906944 536870912 4096 0 33418 255 63 /dev/da4 512 274877906944 536870912 4096 0 33418 255 63 /dev/da5 512 274877906944 536870912 4096 0 33418 255 63 /dev/da6 512 274877906944 536870912 4096 0 33418 255 63 /dev/da7 512 274877906944 536870912 4096 0 33418 255 63 /dev/da8 512 274877906944 536870912 4096 0 33418 255 63 /dev/da9 512 274877906944 536870912 4096 0 33418 255 63 /dev/da10 512 274877906944 536870912 4096 0 33418 255 63 /dev/da11 512 274877906944 536870912 4096 0 33418 255 63 /dev/da12 512 68719476736 134217728 4096 0 8354 255 63 /dev/da13 512 68719476736 134217728 4096 0 8354 255 63 ## Create the pool: zpool create nebula mirror da2 da3 mirror da4 da5 mirror da6 da7 mirror da8 da9 mirror da10 da11 ## Add the special mirror zpool add nebula special mirror da12 da13 ## zpool status root@jmdtest:/ # zpool list -v nebula NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT nebula 1.30T 2.57G 1.30T - - 0% 0% 1.00x ONLINE - mirror 254G 492M 254G - - 0% 0.18% da2 - - - - - - - da3 - - - - - - - mirror 254G 501M 254G - - 0% 0.19% da4 - - - - - - - da5 - - - - - - - mirror 254G 596M 253G - - 0% 0.22% da6 - - - - - - - da7 - - - - - - - mirror 254G 481M 254G - - 0% 0.18% da8 - - - - - - - da9 - - - - - - - mirror 254G 561M 253G - - 0% 0.21% da10 - - - - - - - da11 - - - - - - - special - - - - - - mirror 63.5G 2.62M 63.5G - - 0% 0.00% da12 - - - - - - - da13 - - - - - - - root@jmdtest:/ # ## Remove the special mirror root@jmdtest:/ # zpool remove nebula mirror-5 cannot remove mirror-5: invalid config; all top-level vdevs must have the same sector size and not be raidz. ## All mirrors, and all the same ashift: root@jmdtest:/ # zdb -C | grep -e child -e ashift vdev_children: 6 children[0]: ashift: 12 children[0]: children[1]: children[1]: ashift: 12 children[0]: children[1]: children[2]: ashift: 12 children[0]: children[1]: children[3]: ashift: 12 children[0]: children[1]: children[4]: ashift: 12 children[0]: children[1]: children[5]: ashift: 12 children[0]: children[1]: root@jmdtest:/ # What am I doing wrong? From owner-freebsd-fs@freebsd.org Mon Dec 28 03:04:32 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DA6F94CF04E for ; Mon, 28 Dec 2020 03:04:32 +0000 (UTC) (envelope-from jdelisle@gmail.com) Received: from mail-pl1-x62b.google.com (mail-pl1-x62b.google.com [IPv6:2607:f8b0:4864:20::62b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D42TC64YLz4bTG for ; Mon, 28 Dec 2020 03:04:31 +0000 (UTC) (envelope-from jdelisle@gmail.com) Received: by mail-pl1-x62b.google.com with SMTP id g3so5015751plp.2 for ; Sun, 27 Dec 2020 19:04:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=LEvRFwe27g0teePML0OL3cDZoL5QMgBtPuhxFaZv4cY=; b=PW6PwpH7okVLATISUyZtL/qPMW/5/4cqMdv/pbxL8Ph9ECsuT2jX75hdrXMq70nc74 JEw6/aSjDSVn2oIN17iQuoDBEAJ8olt6DrpqqJfZJC82OOcr0rO2xPjokeqTnCNg3cNu M07A7cPgUsu51A3/2OJ3+8m970a+I1Bu2XwEVC4MOwWagoUgm/KfF8pWNSDEStCSbIC9 b6nR6VKGZYt1+RjP9IRZ+SWM7U+UZAXJWXzi5umVMRS+svDk6Oc3+Y4z+L8NNdE4cs6a hQ4kVt3DBqEkbqwwtaGEaa3xTaTTQECAOSYYbR+TXQViiAsgZ/anHsnxvvNszsPDNsKS r+Kg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=LEvRFwe27g0teePML0OL3cDZoL5QMgBtPuhxFaZv4cY=; b=qfjoLbAbCTmMK4huLL54XgW/HVMmo7GGGhgRqFPjaGbyRDAZ3aaDfZQvi8hJqd1xOW bMRmeMLjqamqJQhw6eBeKK7pWef2d6EJhj4+7qrZhqrHhc90nefXXY8Wfi4MshzbQ7wc vZYs1SYfP59Fw+zIqsQMWirs1/7MM+XRkZaPzn4k3MKKcsr1NlEi0bV8Su5iEGp3PFAD zHIoOdoPAlcE4dYqZx8Q2lRmjJfQ2uKOWujpQ1tjpbNMSfwFp+yY7Tc1sIo3+NAmW+Tk Km6kcGmB2UwSG29v3BssQ3rPiLrLdndCzTHVsH6soD0Lk6WOiymzw5fNkKsvtuf+LFJ/ c0oA== X-Gm-Message-State: AOAM5314MtTYP7rzVJBTzi2B5KUqqlFEDpjXXpU4kmsOjIWrqc8R0K0k fliMZwsiLrtrxhD5N5aGpbTS3LJ4IflZlLFlLMgGsPgeFGGjmw== X-Google-Smtp-Source: ABdhPJyb4XnLLzqPbisEMIjM6Cz4XYkd/lrc7LoRrUjRBrbj2os2alASYIl6JkcbNIyJVcckDdPmokJyNDRXJlIjaPk= X-Received: by 2002:a17:90a:d308:: with SMTP id p8mr19094515pju.110.1609124669413; Sun, 27 Dec 2020 19:04:29 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: John Delisle Date: Sun, 27 Dec 2020 21:04:16 -0600 Message-ID: Subject: Re: zpool remove not working for metadata special devices To: freebsd-fs X-Rspamd-Queue-Id: 4D42TC64YLz4bTG X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=PW6PwpH7; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jdelisle@gmail.com designates 2607:f8b0:4864:20::62b as permitted sender) smtp.mailfrom=jdelisle@gmail.com X-Spamd-Result: default: False [-1.90 / 15.00]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MIME_BASE64_TEXT(0.10)[]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::62b:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::62b:from:127.0.2.255]; NEURAL_SPAM_SHORT(1.00)[1.000]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::62b:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Dec 2020 03:04:32 -0000 To experiment, I've completed this same procedure using Ubuntu 20.10 (on the same "hardware", both are identically configured VMs in Azure) . The steps above work fine with ZFS on Ubuntu, and I can successfully remove both data vdev mirrors and the special vdev mirror. This makes me think my syntax is correct at least.. but for whatever reason the exact same procedure fails on FreeBSD (both removing a mirror and removing the special mirror fail with the same error on FreeBSD). Thanks On Sun, Dec 27, 2020 at 6:55 PM John Delisle wrote: > I have a pool of mirrors, and added a mirrored special device. Although > documentation suggests it should be removable, I cannot get zpool to do so. > > All top-level vdevs are mirrors, and all have the same sector size and > ashift. No raidz. > > ## Current 12.2 p2 > root@jmdtest:/ # freebsd-version > 12.2-RELEASE-p2 > > ## Disk info > root@jmdtest:/ # diskinfo /dev/da[2-9] /dev/da[0-9][0-9] > /dev/da2 512 274877906944 536870912 4096 0 > 33418 255 63 > /dev/da3 512 274877906944 536870912 4096 0 > 33418 255 63 > /dev/da4 512 274877906944 536870912 4096 0 > 33418 255 63 > /dev/da5 512 274877906944 536870912 4096 0 > 33418 255 63 > /dev/da6 512 274877906944 536870912 4096 0 > 33418 255 63 > /dev/da7 512 274877906944 536870912 4096 0 > 33418 255 63 > /dev/da8 512 274877906944 536870912 4096 0 > 33418 255 63 > /dev/da9 512 274877906944 536870912 4096 0 > 33418 255 63 > /dev/da10 512 274877906944 536870912 4096 0 > 33418 255 63 > /dev/da11 512 274877906944 536870912 4096 0 > 33418 255 63 > /dev/da12 512 68719476736 134217728 4096 0 > 8354 255 63 > /dev/da13 512 68719476736 134217728 4096 0 > 8354 255 63 > > > ## Create the pool: > zpool create nebula mirror da2 da3 mirror da4 da5 mirror da6 da7 mirror > da8 da9 mirror da10 da11 > > ## Add the special mirror > zpool add nebula special mirror da12 da13 > > ## zpool status > root@jmdtest:/ # zpool list -v nebula > NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP > HEALTH ALTROOT > nebula 1.30T 2.57G 1.30T - - 0% 0% 1.00x > ONLINE - > mirror 254G 492M 254G - - 0% 0.18% > da2 - - - - - - - > da3 - - - - - - - > mirror 254G 501M 254G - - 0% 0.19% > da4 - - - - - - - > da5 - - - - - - - > mirror 254G 596M 253G - - 0% 0.22% > da6 - - - - - - - > da7 - - - - - - - > mirror 254G 481M 254G - - 0% 0.18% > da8 - - - - - - - > da9 - - - - - - - > mirror 254G 561M 253G - - 0% 0.21% > da10 - - - - - - - > da11 - - - - - - - > special - - - - - - > mirror 63.5G 2.62M 63.5G - - 0% 0.00% > da12 - - - - - - - > da13 - - - - - - - > root@jmdtest:/ # > > ## Remove the special mirror > root@jmdtest:/ # zpool remove nebula mirror-5 > cannot remove mirror-5: invalid config; all top-level vdevs must have the > same sector size and not be raidz. > > ## All mirrors, and all the same ashift: > root@jmdtest:/ # zdb -C | grep -e child -e ashift > vdev_children: 6 > children[0]: > ashift: 12 > children[0]: > children[1]: > children[1]: > ashift: 12 > children[0]: > children[1]: > children[2]: > ashift: 12 > children[0]: > children[1]: > children[3]: > ashift: 12 > children[0]: > children[1]: > children[4]: > ashift: 12 > children[0]: > children[1]: > children[5]: > ashift: 12 > children[0]: > children[1]: > root@jmdtest:/ # > > What am I doing wrong? > > >