From owner-freebsd-fs@freebsd.org Thu Jan 17 23:35:03 2019 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0389E1484081 for ; Thu, 17 Jan 2019 23:35:03 +0000 (UTC) (envelope-from jdelisle@gmail.com) Received: from mail-lj1-x22f.google.com (mail-lj1-x22f.google.com [IPv6:2a00:1450:4864:20::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 95C268B236 for ; Thu, 17 Jan 2019 23:35:00 +0000 (UTC) (envelope-from jdelisle@gmail.com) Received: by mail-lj1-x22f.google.com with SMTP id u89-v6so10103612lje.1 for ; Thu, 17 Jan 2019 15:35:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=5gPTchVAZ6G7LKRphqzzPRJng5SLfP+3zK3GG0QfVfY=; b=bL7R623ky3LrdjkBqzeqi5GCJAtvoeyCe7wlPN9vnNCmzldkuqhzLvcK0rUVnL3bBo nyM647R5mwXNSja/AskkT+9hyLZh4OiFL1VzjM85JTerQmAz8dzdG8+zmfTNbcOp0KkU 8intYpKa16xdL9rkewdDT4jogqaeZF7h+TvJE0dlWeadq2RkHCBocqmuN3HZml3+wKt6 lR58c7GJXHMeCB/zfFN+y2rmKEafEQj2YD8quSX4Crftf+88KrdXQkqTQ6NJLnkmcVlt KhMCAalS/zmJs3NDcpFsJM8HcuXEVbaiYPsDXsyFdOoCfl1qgKzDJPy4AQQQRye6a3C1 JytA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=5gPTchVAZ6G7LKRphqzzPRJng5SLfP+3zK3GG0QfVfY=; b=LC637j1LsKXXCVQudUHk3+doqEo5toJmssgcUgka9RPQBoW3UPj542O7CuooDgnqb7 XNgQ9TR6lBaUSnydj3iLC/wLbl/Qj71eKHU39IzzMWfXgzmHzETPS8mSL/AMDVef4HD3 UlBl7YpmzGJuH9KSYUA16rvbGTO+Fx4AjKaJ5SMENWVhrMtwFmtWUA6TGqaXkTv7i6mW nOMXuFtqsSIX1WIU1QGphkezs4oO3Bhw/k8tMKbuAqgLAnvMOagbcLaGcZubmVBJXgnU C9zETxIG5bMIl6NV29BoJCsKVA+QXNMuvqmy8miHtxgV4bLtHlpmCeysiA3HnQSopTyf FPHg== X-Gm-Message-State: AJcUukcGQqalqz0bAmDaTFEB4EYASb6YAllYnxIPaanjZk0cz+sqbEPf ez7Y5A3IcSLI41fRFiFEzpKCbmDyUtG0YR+QqcjAOFFQ X-Google-Smtp-Source: ALg8bN7IxyApnY8Tr+9wH+d4KiOB3rD+R0AN6oiewlSC33kNJF20s//mLAbbVsn9v985oSsi33XrI4lurHErD+RlKHA= X-Received: by 2002:a2e:2a06:: with SMTP id q6-v6mr10373550ljq.37.1547768097507; Thu, 17 Jan 2019 15:34:57 -0800 (PST) MIME-Version: 1.0 From: jdelisle Date: Thu, 17 Jan 2019 17:34:46 -0600 Message-ID: Subject: "zpool remove" failing To: freebsd-fs@freebsd.org X-Rspamd-Queue-Id: 95C268B236 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=bL7R623k; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jdelisle@gmail.com designates 2a00:1450:4864:20::22f as permitted sender) smtp.mailfrom=jdelisle@gmail.com X-Spamd-Result: default: False [-6.47 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_NONE(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MIME_BASE64_TEXT(0.10)[]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.87)[-0.866,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-2.69)[ip: (-9.47), ipnet: 2a00:1450::/32(-2.08), asn: 15169(-1.82), country: US(-0.08)]; RCVD_IN_DNSWL_NONE(0.00)[f.2.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jan 2019 23:35:03 -0000 Hello, I've run into a problem and I'm hoping the knowledgeable folks here can help. I've accidentally added a mirror vdev to a pool, and would like to remove it. Note "mirror-8" in the output down below, it's what I'd like to remove. When I attempt to remove it, I get the following message: [root@omega ~]# zpool remove nebula mirror-8 cannot remove mirror-8: invalid config; all top-level vdevs must have the same sector size and not be raidz. The disks in my pool are all 512 byte sector disks. Any idea what I'm doing wrong? ##### My system's uname -a - it's FreeNAS. [root@omega ~]# uname -a FreeBSD omega.nebula.pw 11.2-STABLE FreeBSD 11.2-STABLE #0 r325575+fc3d65faae6(HEAD): Thu Dec 20 16:12:30 EST 2018 root@nemesis.tn.ixsystems.com:/freenas-releng-final/freenas/_BE/objs/freenas-releng-final/freenas/_BE/os/sys/FreeNAS.amd64 amd64 ##### The pool I accidentally added a mirror vdev to (mirror-8): [root@omega ~]# zpool status -v nebula pool: nebula state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM nebula ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gptid/2fdc125f-15f4-11e9-b1b0-000c29f308bf ONLINE 0 0 0 gptid/30816c63-15f4-11e9-b1b0-000c29f308bf ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 gptid/31403c46-15f4-11e9-b1b0-000c29f308bf ONLINE 0 0 0 gptid/31f1f182-15f4-11e9-b1b0-000c29f308bf ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 gptid/32a4cfa3-15f4-11e9-b1b0-000c29f308bf ONLINE 0 0 0 gptid/3356fcab-15f4-11e9-b1b0-000c29f308bf ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 gptid/341311e6-15f4-11e9-b1b0-000c29f308bf ONLINE 0 0 0 gptid/34c9952c-15f4-11e9-b1b0-000c29f308bf ONLINE 0 0 0 mirror-4 ONLINE 0 0 0 gptid/3580b3ad-15f4-11e9-b1b0-000c29f308bf ONLINE 0 0 0 gptid/364188ba-15f4-11e9-b1b0-000c29f308bf ONLINE 0 0 0 mirror-5 ONLINE 0 0 0 gptid/370908e5-15f4-11e9-b1b0-000c29f308bf ONLINE 0 0 0 gptid/37cf00a5-15f4-11e9-b1b0-000c29f308bf ONLINE 0 0 0 mirror-6 ONLINE 0 0 0 gptid/388e6fef-15f4-11e9-b1b0-000c29f308bf ONLINE 0 0 0 gptid/3945fee7-15f4-11e9-b1b0-000c29f308bf ONLINE 0 0 0 mirror-7 ONLINE 0 0 0 gptid/3a08fb45-15f4-11e9-b1b0-000c29f308bf ONLINE 0 0 0 gptid/3ad7a643-15f4-11e9-b1b0-000c29f308bf ONLINE 0 0 0 mirror-8 ONLINE 0 0 0 da33p1 ONLINE 0 0 0 da34p1 ONLINE 0 0 0 errors: No known data errors [root@omega ~]# ##### My ZFS features etc. [root@omega ~]# zpool get all nebula NAME PROPERTY VALUE SOURCE nebula size 21.8T - nebula capacity 64% - nebula altroot /mnt local nebula health ONLINE - nebula guid 9027025247116299332 default nebula version - default nebula bootfs - default nebula delegation on default nebula autoreplace off default nebula cachefile /data/zfs/zpool.cache local nebula failmode continue local nebula listsnapshots off default nebula autoexpand on local nebula dedupditto 0 default nebula dedupratio 1.00x - nebula free 7.76T - nebula allocated 14.0T - nebula readonly off - nebula comment - default nebula expandsize - - nebula freeing 0 default nebula fragmentation 2% - nebula leaked 0 default nebula bootsize - default nebula checkpoint - - nebula feature@async_destroy enabled local nebula feature@empty_bpobj active local nebula feature@lz4_compress active local nebula feature@multi_vdev_crash_dump enabled local nebula feature@spacemap_histogram active local nebula feature@enabled_txg active local nebula feature@hole_birth active local nebula feature@extensible_dataset enabled local nebula feature@embedded_data active local nebula feature@bookmarks enabled local nebula feature@filesystem_limits enabled local nebula feature@large_blocks enabled local nebula feature@sha512 enabled local nebula feature@skein enabled local nebula feature@device_removal enabled local nebula feature@obsolete_counts enabled local nebula feature@zpool_checkpoint enabled local ###### The two disks in the mirrored vdev I wish to remove: [root@omega ~]# diskinfo -v /dev/da33 /dev/da33 512 # sectorsize 120034123776 # mediasize in bytes (112G) 234441648 # mediasize in sectors 4096 # stripesize 0 # stripeoffset 14593 # Cylinders according to firmware. 255 # Heads according to firmware. 63 # Sectors according to firmware. ATA INTEL SSDSA2CW12 # Disk descr. CVPR1132000R120LGN # Disk ident. id1,enc@n50050cc1020371ce/type@0/slot@13 # Physical path Yes # TRIM/UNMAP support 0 # Rotation rate in RPM Not_Zoned # Zone Mode [root@omega ~]# diskinfo -v /dev/da34 /dev/da34 512 # sectorsize 120034123776 # mediasize in bytes (112G) 234441648 # mediasize in sectors 4096 # stripesize 0 # stripeoffset 14593 # Cylinders according to firmware. 255 # Heads according to firmware. 63 # Sectors according to firmware. ATA INTEL SSDSA2CW12 # Disk descr. CVPR119300D5120LGN # Disk ident. id1,enc@n50050cc1020371ce/type@0/slot@7 # Physical path Yes # TRIM/UNMAP support 0 # Rotation rate in RPM Not_Zoned # Zone Mode [root@omega ~]# ##### This is an example of one of the many disks in my mirrored pairs. They're all the same make/ model of disk, and are 512 byte sectors just like the SSDs above. [root@omega ~]# diskinfo -v /dev/da10 /dev/da10 512 # sectorsize 3000592982016 # mediasize in bytes (2.7T) 5860533168 # mediasize in sectors 0 # stripesize 0 # stripeoffset 364801 # Cylinders according to firmware. 255 # Heads according to firmware. 63 # Sectors according to firmware. HITACHI HUS72303CLAR3000 # Disk descr. YHJEXGHD # Disk ident. id1,enc@n50050cc1020371ce/type@0/slot@f # Physical path No # TRIM/UNMAP support 7200 # Rotation rate in RPM Not_Zoned # Zone Mode [root@omega ~]#