Date: Tue, 31 Jan 2017 11:39:14 -0600 From: Larry Rosenman <ler@FreeBSD.org> To: Steven Hartland <killing@multiplay.co.uk> Cc: Freebsd fs <freebsd-fs@freebsd.org> Subject: Re: 16.0E ExpandSize? -- New Server Message-ID: <22e1bfc5840d972cf93643733682cda1@FreeBSD.org> In-Reply-To: <76fc9505-f681-0de0-fe0c-5624b29de321@multiplay.co.uk> References: <00db0ab7243ce6368c246ae20f9c075a@FreeBSD.org> <1a69057c-dc59-9b78-9762-4f98a071105e@multiplay.co.uk> <ce5a1d39612d694077accda33266a3ab@FreeBSD.org> <ad07e84e-f297-362a-1398-c5503bb56a8d@multiplay.co.uk> <35a9034f91542bb1329ac5104bf3b773@FreeBSD.org> <76fc9505-f681-0de0-fe0c-5624b29de321@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
borg-new /home/ler $ sudo ./vdev-stats.d vdev_path: n/a, vdev_max_asize: 0, vdev_asize: 0 vdev_path: n/a, vdev_max_asize: 11947471798272, vdev_asize: 11947478089728 vdev_path: /dev/mfid4p4, vdev_max_asize: 1991245299712, vdev_asize: 1991245299712 vdev_path: /dev/mfid0p4, vdev_max_asize: 1991246348288, vdev_asize: 1991246348288 vdev_path: /dev/mfid1p4, vdev_max_asize: 1991246348288, vdev_asize: 1991246348288 vdev_path: /dev/mfid3p4, vdev_max_asize: 1991247921152, vdev_asize: 1991247921152 vdev_path: /dev/mfid2p4, vdev_max_asize: 1991246348288, vdev_asize: 1991246348288 vdev_path: /dev/mfid5p4, vdev_max_asize: 1991246348288, vdev_asize: 1991246348288 ^C borg-new /home/ler $ borg-new /home/ler $ zpool list -v NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT zroot 10.8T 94.3G 10.7T 16.0E 0% 0% 1.00x ONLINE - raidz1 10.8T 94.3G 10.7T 16.0E 0% 0% mfid4p4 - - - - - - mfid0p4 - - - - - - mfid1p4 - - - - - - mfid3p4 - - - - - - mfid2p4 - - - - - - mfid5p4 - - - - - - borg-new /home/ler $ On 01/31/2017 11:28 am, Steven Hartland wrote: > What does the attached dtrace script output when you run zpool list -v while its running? > > On 31/01/2017 17:07, Larry Rosenman wrote: > > borg-new /home/ler $ sudo zdb > Password: > zroot: > version: 5000 > name: 'zroot' > state: 0 > txg: 58729 > pool_guid: 11945658884309024932 > hostid: 3619181042 > hostname: 'borg-new' > com.delphix:has_per_vdev_zaps > vdev_children: 1 > vdev_tree: > type: 'root' > id: 0 > guid: 11945658884309024932 > create_txg: 4 > children[0]: > type: 'raidz' > id: 0 > guid: 7596925654112466913 > nparity: 1 > metaslab_array: 42 > metaslab_shift: 36 > ashift: 12 > asize: 11947478089728 > is_log: 0 > create_txg: 4 > com.delphix:vdev_zap_top: 35 > children[0]: > type: 'disk' > id: 0 > guid: 1443238581175429852 > path: '/dev/mfid4p4' > whole_disk: 1 > DTL: 137 > create_txg: 4 > com.delphix:vdev_zap_leaf: 131 > children[1]: > type: 'disk' > id: 1 > guid: 1865792721003775978 > path: '/dev/mfid0p4' > whole_disk: 1 > DTL: 133 > create_txg: 4 > com.delphix:vdev_zap_leaf: 37 > children[2]: > type: 'disk' > id: 2 > guid: 12541720522827927342 > path: '/dev/mfid1p4' > whole_disk: 1 > DTL: 132 > create_txg: 4 > com.delphix:vdev_zap_leaf: 38 > children[3]: > type: 'disk' > id: 3 > guid: 13053934791777776444 > path: '/dev/mfid3p4' > whole_disk: 1 > DTL: 136 > create_txg: 4 > com.delphix:vdev_zap_leaf: 135 > children[4]: > type: 'disk' > id: 4 > guid: 4432707573898874857 > path: '/dev/mfid2p4' > whole_disk: 1 > DTL: 130 > create_txg: 4 > com.delphix:vdev_zap_leaf: 40 > children[5]: > type: 'disk' > id: 5 > guid: 5106293125005422556 > path: '/dev/mfid5p4' > whole_disk: 1 > DTL: 129 > create_txg: 4 > com.delphix:vdev_zap_leaf: 41 > features_for_read: > com.delphix:hole_birth > com.delphix:embedded_data > borg-new /home/ler $ > > On 01/31/2017 10:56 am, Steven Hartland wrote: What does "zdb" (no params) report? > > On 31/01/2017 14:51, Larry Rosenman wrote: borg-new /home/ler $ zpool get expandsize > NAME PROPERTY VALUE SOURCE > zroot expandsize 16.0E - > borg-new /home/ler $ > > On 01/30/2017 5:40 pm, Steven Hartland wrote: > What do you get from: > zpool get expandsize > > On 30/01/2017 21:47, Larry Rosenman wrote: > Greetings, > Have a new server with a recent -CURRENT on it. I see an expandsize of 16.0E, which is insane > because there are only 6 2TB disks. > > borg-new /home/ler $ zpool status > pool: zroot > state: ONLINE > scan: scrub repaired 0 in 0h7m with 0 errors on Mon Jan 30 15:43:27 2017 > config: > > NAME STATE READ WRITE CKSUM > zroot ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > mfid4p4 ONLINE 0 0 0 > mfid0p4 ONLINE 0 0 0 > mfid1p4 ONLINE 0 0 0 > mfid3p4 ONLINE 0 0 0 > mfid2p4 ONLINE 0 0 0 > mfid5p4 ONLINE 0 0 0 > > errors: No known data errors > borg-new /home/ler $ zpool list > NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT > zroot 10.8T 94.2G 10.7T 16.0E 0% 0% 1.00x ONLINE - > borg-new /home/ler $ zpool status > pool: zroot > state: ONLINE > scan: scrub repaired 0 in 0h7m with 0 errors on Mon Jan 30 15:43:27 2017 > config: > > NAME STATE READ WRITE CKSUM > zroot ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > mfid4p4 ONLINE 0 0 0 > mfid0p4 ONLINE 0 0 0 > mfid1p4 ONLINE 0 0 0 > mfid3p4 ONLINE 0 0 0 > mfid2p4 ONLINE 0 0 0 > mfid5p4 ONLINE 0 0 0 > > errors: No known data errors > borg-new /home/ler $ zpool history > History for 'zroot': > cannot show history for pool 'zroot': permission denied > borg-new /home/ler $ sudo zpool history > Password: > History for 'zroot': > 2017-01-25.19:02:15 zpool create -o altroot=/mnt -O compress=lz4 -O atime=off -m none -f zroot raidz1 mfid0p4 mfid1p4 mfid2p4 mfid3p4 mfid4p4 mfid5p4 > 2017-01-25.19:02:15 zfs create -o mountpoint=none zroot/ROOT > 2017-01-25.19:02:15 zfs create -o mountpoint=/ zroot/ROOT/default > 2017-01-25.19:02:15 zfs create -o mountpoint=/tmp -o exec=on -o setuid=off zroot/tmp > 2017-01-25.19:02:15 zfs create -o mountpoint=/usr -o canmount=off zroot/usr > 2017-01-25.19:02:15 zfs create zroot/usr/home > 2017-01-25.19:02:15 zfs create -o setuid=off zroot/usr/ports > 2017-01-25.19:02:15 zfs create zroot/usr/src > 2017-01-25.19:02:15 zfs create -o mountpoint=/var -o canmount=off zroot/var > 2017-01-25.19:02:15 zfs create -o exec=off -o setuid=off zroot/var/audit > 2017-01-25.19:02:15 zfs create -o exec=off -o setuid=off zroot/var/crash > 2017-01-25.19:02:15 zfs create -o exec=off -o setuid=off zroot/var/log > 2017-01-25.19:02:15 zfs create -o atime=on zroot/var/mail > 2017-01-25.19:02:15 zfs create -o setuid=off zroot/var/tmp > 2017-01-25.19:02:15 zfs set mountpoint=/zroot zroot > 2017-01-25.19:02:15 zpool set bootfs=zroot/ROOT/default zroot > 2017-01-25.19:02:15 zpool set cachefile=/mnt/boot/zfs/zpool.cache zroot > 2017-01-25.19:02:19 zfs set canmount=noauto zroot/ROOT/default > 2017-01-26.19:51:23 zpool scrub zroot > 2017-01-29.12:50:56 zpool replace zroot 6435283924725851469 /dev/mfid5p4 > 2017-01-29.12:52:44 zpool scrub zroot > 2017-01-29.13:11:50 zfs create zroot/usr/obj > 2017-01-29.13:26:11 zfs create zroot/ler > 2017-01-29.13:27:58 zfs set mountpoint=/home/ler zroot/ler > 2017-01-29.16:22:07 zpool replace zroot 8173792548235216166 /dev/mfid4p4 > 2017-01-29.16:27:18 zpool scrub zroot > 2017-01-29.16:32:15 zfs create -o mountpoint=/home zroot/home > 2017-01-30.15:36:34 zpool scrub zroot > > borg-new /home/ler $ gpart show > => 40 3905945520 mfid0 GPT (1.8T) > 40 1600 1 efi (800K) > 1640 1024 2 freebsd-boot (512K) > 2664 1432 - free - (716K) > 4096 16777216 3 freebsd-swap (8.0G) > 16781312 3889162240 4 freebsd-zfs (1.8T) > 3905943552 2008 - free - (1.0M) > > => 40 3905945520 mfid1 GPT (1.8T) > 40 1600 1 efi (800K) > 1640 1024 2 freebsd-boot (512K) > 2664 1432 - free - (716K) > 4096 16777216 3 freebsd-swap (8.0G) > 16781312 3889162240 4 freebsd-zfs (1.8T) > 3905943552 2008 - free - (1.0M) > > => 40 3905945520 mfid2 GPT (1.8T) > 40 1600 1 efi (800K) > 1640 1024 2 freebsd-boot (512K) > 2664 1432 - free - (716K) > 4096 16777216 3 freebsd-swap (8.0G) > 16781312 3889162240 4 freebsd-zfs (1.8T) > 3905943552 2008 - free - (1.0M) > > => 40 3905945520 mfid3 GPT (1.8T) > 40 1600 1 efi (800K) > 1640 1024 2 freebsd-boot (512K) > 2664 16777216 3 freebsd-swap (8.0G) > 16779880 3889165680 4 freebsd-zfs (1.8T) > > => 40 3905945520 mfid5 GPT (1.8T) > 40 1600 1 efi (800K) > 1640 1024 2 freebsd-boot (512K) > 2664 1432 - free - (716K) > 4096 16777216 3 freebsd-swap (8.0G) > 16781312 3889162240 4 freebsd-zfs (1.8T) > 3905943552 2008 - free - (1.0M) > > => 40 3905945520 mfid4 GPT (1.8T) > 40 1600 1 efi (800K) > 1640 1024 2 freebsd-boot (512K) > 2664 1432 - free - (716K) > 4096 16777216 3 freebsd-swap (8.0G) > 16781312 3889160192 4 freebsd-zfs (1.8T) > 3905941504 4056 - free - (2.0M) > > borg-new /home/ler $ > > borg-new /home/ler $ uname -aKU > FreeBSD borg-new 12.0-CURRENT FreeBSD 12.0-CURRENT #1 r312893: Fri Jan 27 12:39:05 CST 2017 root@borg-new:/usr/obj/usr/src/sys/GENERIC amd64 1200020 1200020 > borg-new /home/ler $ > > Ideas? > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" -- Larry Rosenman http://people.freebsd.org/~ler [1] Phone: +1 214-642-9640 E-Mail: ler@FreeBSD.org US Mail: 17716 Limpia Crk, Round Rock, TX 78664-7281 -- Larry Rosenman http://people.freebsd.org/~ler Phone: +1 214-642-9640 E-Mail: ler@FreeBSD.org US Mail: 17716 Limpia Crk, Round Rock, TX 78664-7281 Links: ------ [1] http://people.freebsd.org/%7Eler From owner-freebsd-fs@freebsd.org Tue Jan 31 18:30:34 2017 Return-Path: <owner-freebsd-fs@freebsd.org> Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5E4CCCC9A77 for <freebsd-fs@mailman.ysv.freebsd.org>; Tue, 31 Jan 2017 18:30:34 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x236.google.com (mail-wm0-x236.google.com [IPv6:2a00:1450:400c:c09::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 13E3F101C for <freebsd-fs@freebsd.org>; Tue, 31 Jan 2017 18:30:33 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x236.google.com with SMTP id r141so544659wmg.1 for <freebsd-fs@freebsd.org>; Tue, 31 Jan 2017 10:30:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to; bh=+ffx5WybUGxLjd+96UH0FIpyC/mPQEzIu4u/r1FmqkE=; b=oeakjSnqPK7or7ucQfHLMJJa7ICbn5wn3+mncgoORtpg5UkpzCIU1huuha8lqQyUIo 5+wcxXxRISzu8u9ipmqOrMLCzec2VLNK1CvDKZjZB7KG16sU0tlvICEOP66uXvX8Jm2Q 0SO4PwpGa7b9RbvYgkGQA2O2FcqgiT4BDbCiOUQY+53YBd9baEiBMGNicC7HPsXBoFe9 Cd2AUTtKLBlhc867XM0l07oYxAbsVTooMJ2HaE1ysfc4P0V7b1pdGh5aCYpRS3PkXtCJ NDG8XtXYMxr8SUTLmDkbHX0eSjoRRNvEcZ2GFuhMqPKclk+SV3Dq9DZmdIfQyQGp4y8O tykw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=+ffx5WybUGxLjd+96UH0FIpyC/mPQEzIu4u/r1FmqkE=; b=LVu9xTdv9qd6WndL9Y5TenHz7yZb+FnGokqQtwgg9sHwl65tQxE2akxYkxD981yl2s 7toCGNPxAmK/ugAphWynNlbDrmlB+4zsOB3bhDc7SmaV0+lcD7FCLThAh0xdP3jn5NhX oNsHLoEVHi7Fx83drqqm+ADEBr0NBl4wgg6IvjxUoMqAt7JmBOrAJ/TKUH7r9hEwQZsr szlkuHsVMBmijdufyiOviJ/ORxZydW4gZCwwCPhNLShteyd6XtLTnTX2NNLxS9y6X5Yn heLx17qZbt7s1enRD1sLgUxCyCfcHjrQM0lFfH61b2NNc66o9dLDSKjO9OJK5yt84xPH uNIg== X-Gm-Message-State: AIkVDXJYsPYa4IywcJKauJU7T5ej8agecf6q9JUJsx+s3GSfWqcj7kHEHGWIGbvCs+JfI2VF X-Received: by 10.28.58.204 with SMTP id h195mr19587695wma.116.1485887431269; Tue, 31 Jan 2017 10:30:31 -0800 (PST) Received: from [10.10.1.58] ([185.97.61.26]) by smtp.gmail.com with ESMTPSA id 18sm29701725wrb.14.2017.01.31.10.30.30 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 31 Jan 2017 10:30:30 -0800 (PST) Subject: Re: 16.0E ExpandSize? -- New Server To: Larry Rosenman <ler@FreeBSD.org> References: <00db0ab7243ce6368c246ae20f9c075a@FreeBSD.org> <1a69057c-dc59-9b78-9762-4f98a071105e@multiplay.co.uk> <ce5a1d39612d694077accda33266a3ab@FreeBSD.org> <ad07e84e-f297-362a-1398-c5503bb56a8d@multiplay.co.uk> <35a9034f91542bb1329ac5104bf3b773@FreeBSD.org> <76fc9505-f681-0de0-fe0c-5624b29de321@multiplay.co.uk> <22e1bfc5840d972cf93643733682cda1@FreeBSD.org> Cc: Freebsd fs <freebsd-fs@freebsd.org> From: Steven Hartland <killing@multiplay.co.uk> Message-ID: <f2600a53-0dc1-9f41-1405-ed22d96d30cf@multiplay.co.uk> Date: Tue, 31 Jan 2017 18:30:31 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-Version: 1.0 In-Reply-To: <22e1bfc5840d972cf93643733682cda1@FreeBSD.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems <freebsd-fs.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fs>, <mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>; List-Post: <mailto:freebsd-fs@freebsd.org> List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>, <mailto:freebsd-fs-request@freebsd.org?subject=subscribe> X-List-Received-Date: Tue, 31 Jan 2017 18:30:34 -0000 Your issue is the reported vdev_max_asize > vdev_asize: vdev_max_asize: 11947471798272 vdev_asize: 11947478089728 max asize is smaller than asize by 6291456 For raidz1 Xsize should be the smallest disk Xsize * disks so: 1991245299712 * 6 = 11947471798272 So your max asize looks right but asize looks too big Expand Size is calculated by: if (vd->vdev_aux == NULL && tvd != NULL && vd->vdev_max_asize != 0) { vs->vs_esize = P2ALIGN(vd->vdev_max_asize - vd->vdev_asize, 1ULL << tvd->vdev_ms_shift); } So the question is why is asize too big? Given you seem to have some random disk sizes do you have auto expand turned on? On 31/01/2017 17:39, Larry Rosenman wrote: > vdev_path: n/a, vdev_max_asize: 11947471798272, vdev_asize: 11947478089728
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?22e1bfc5840d972cf93643733682cda1>