From owner-freebsd-fs@freebsd.org Tue Jan 31 17:39:15 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 83BAFCCA619 for ; Tue, 31 Jan 2017 17:39:15 +0000 (UTC) (envelope-from ler@FreeBSD.org) Received: from thebighonker.lerctr.org (thebighonker.lerctr.org [IPv6:2001:470:1f0f:3ad:223:7dff:fe9e:6e8a]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "thebighonker.lerctr.org", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5E3A8FAF for ; Tue, 31 Jan 2017 17:39:15 +0000 (UTC) (envelope-from ler@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:References:In-Reply-To:Subject:Cc:To:From:Date: Content-Type:MIME-Version:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=XD6grvZ8kxHE+fT+FwIYF20oBQQW1TQt/wnXYNoUgPg=; b=txFVt99XS0xhcxomKFahxS4WHl XzrPMpXLgq7DAOhvU+sOvZnxnhhOSrW3GwJTCOOe/W5ZNaIYqmZ2dxo+7GduJaGQpAZso/hKgsKAc +N2TgBxIzuIEOH8K9IZ06NwE1Vw+BzKnxInZMk6TAZ/G/ljMc6P1kV8j/5hj9mcKUB9Q=; Received: from thebighonker.lerctr.org ([2001:470:1f0f:3ad:223:7dff:fe9e:6e8a]:18074 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.88 (FreeBSD)) (envelope-from ) id 1cYcOg-00058j-Ih; Tue, 31 Jan 2017 11:39:14 -0600 Received: from proxy.na.alcatel-lucent.com ([135.245.48.75]) by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Tue, 31 Jan 2017 11:39:14 -0600 MIME-Version: 1.0 Date: Tue, 31 Jan 2017 11:39:14 -0600 From: Larry Rosenman To: Steven Hartland Cc: Freebsd fs Subject: Re: 16.0E ExpandSize? -- New Server In-Reply-To: <76fc9505-f681-0de0-fe0c-5624b29de321@multiplay.co.uk> References: <00db0ab7243ce6368c246ae20f9c075a@FreeBSD.org> <1a69057c-dc59-9b78-9762-4f98a071105e@multiplay.co.uk> <35a9034f91542bb1329ac5104bf3b773@FreeBSD.org> <76fc9505-f681-0de0-fe0c-5624b29de321@multiplay.co.uk> Message-ID: <22e1bfc5840d972cf93643733682cda1@FreeBSD.org> X-Sender: ler@FreeBSD.org User-Agent: Roundcube Webmail/1.2.3 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jan 2017 17:39:15 -0000 borg-new /home/ler $ sudo ./vdev-stats.d vdev_path: n/a, vdev_max_asize: 0, vdev_asize: 0 vdev_path: n/a, vdev_max_asize: 11947471798272, vdev_asize: 11947478089728 vdev_path: /dev/mfid4p4, vdev_max_asize: 1991245299712, vdev_asize: 1991245299712 vdev_path: /dev/mfid0p4, vdev_max_asize: 1991246348288, vdev_asize: 1991246348288 vdev_path: /dev/mfid1p4, vdev_max_asize: 1991246348288, vdev_asize: 1991246348288 vdev_path: /dev/mfid3p4, vdev_max_asize: 1991247921152, vdev_asize: 1991247921152 vdev_path: /dev/mfid2p4, vdev_max_asize: 1991246348288, vdev_asize: 1991246348288 vdev_path: /dev/mfid5p4, vdev_max_asize: 1991246348288, vdev_asize: 1991246348288 ^C borg-new /home/ler $ borg-new /home/ler $ zpool list -v NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT zroot 10.8T 94.3G 10.7T 16.0E 0% 0% 1.00x ONLINE - raidz1 10.8T 94.3G 10.7T 16.0E 0% 0% mfid4p4 - - - - - - mfid0p4 - - - - - - mfid1p4 - - - - - - mfid3p4 - - - - - - mfid2p4 - - - - - - mfid5p4 - - - - - - borg-new /home/ler $ On 01/31/2017 11:28 am, Steven Hartland wrote: > What does the attached dtrace script output when you run zpool list -v while its running? > > On 31/01/2017 17:07, Larry Rosenman wrote: > > borg-new /home/ler $ sudo zdb > Password: > zroot: > version: 5000 > name: 'zroot' > state: 0 > txg: 58729 > pool_guid: 11945658884309024932 > hostid: 3619181042 > hostname: 'borg-new' > com.delphix:has_per_vdev_zaps > vdev_children: 1 > vdev_tree: > type: 'root' > id: 0 > guid: 11945658884309024932 > create_txg: 4 > children[0]: > type: 'raidz' > id: 0 > guid: 7596925654112466913 > nparity: 1 > metaslab_array: 42 > metaslab_shift: 36 > ashift: 12 > asize: 11947478089728 > is_log: 0 > create_txg: 4 > com.delphix:vdev_zap_top: 35 > children[0]: > type: 'disk' > id: 0 > guid: 1443238581175429852 > path: '/dev/mfid4p4' > whole_disk: 1 > DTL: 137 > create_txg: 4 > com.delphix:vdev_zap_leaf: 131 > children[1]: > type: 'disk' > id: 1 > guid: 1865792721003775978 > path: '/dev/mfid0p4' > whole_disk: 1 > DTL: 133 > create_txg: 4 > com.delphix:vdev_zap_leaf: 37 > children[2]: > type: 'disk' > id: 2 > guid: 12541720522827927342 > path: '/dev/mfid1p4' > whole_disk: 1 > DTL: 132 > create_txg: 4 > com.delphix:vdev_zap_leaf: 38 > children[3]: > type: 'disk' > id: 3 > guid: 13053934791777776444 > path: '/dev/mfid3p4' > whole_disk: 1 > DTL: 136 > create_txg: 4 > com.delphix:vdev_zap_leaf: 135 > children[4]: > type: 'disk' > id: 4 > guid: 4432707573898874857 > path: '/dev/mfid2p4' > whole_disk: 1 > DTL: 130 > create_txg: 4 > com.delphix:vdev_zap_leaf: 40 > children[5]: > type: 'disk' > id: 5 > guid: 5106293125005422556 > path: '/dev/mfid5p4' > whole_disk: 1 > DTL: 129 > create_txg: 4 > com.delphix:vdev_zap_leaf: 41 > features_for_read: > com.delphix:hole_birth > com.delphix:embedded_data > borg-new /home/ler $ > > On 01/31/2017 10:56 am, Steven Hartland wrote: What does "zdb" (no params) report? > > On 31/01/2017 14:51, Larry Rosenman wrote: borg-new /home/ler $ zpool get expandsize > NAME PROPERTY VALUE SOURCE > zroot expandsize 16.0E - > borg-new /home/ler $ > > On 01/30/2017 5:40 pm, Steven Hartland wrote: > What do you get from: > zpool get expandsize > > On 30/01/2017 21:47, Larry Rosenman wrote: > Greetings, > Have a new server with a recent -CURRENT on it. I see an expandsize of 16.0E, which is insane > because there are only 6 2TB disks. > > borg-new /home/ler $ zpool status > pool: zroot > state: ONLINE > scan: scrub repaired 0 in 0h7m with 0 errors on Mon Jan 30 15:43:27 2017 > config: > > NAME STATE READ WRITE CKSUM > zroot ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > mfid4p4 ONLINE 0 0 0 > mfid0p4 ONLINE 0 0 0 > mfid1p4 ONLINE 0 0 0 > mfid3p4 ONLINE 0 0 0 > mfid2p4 ONLINE 0 0 0 > mfid5p4 ONLINE 0 0 0 > > errors: No known data errors > borg-new /home/ler $ zpool list > NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT > zroot 10.8T 94.2G 10.7T 16.0E 0% 0% 1.00x ONLINE - > borg-new /home/ler $ zpool status > pool: zroot > state: ONLINE > scan: scrub repaired 0 in 0h7m with 0 errors on Mon Jan 30 15:43:27 2017 > config: > > NAME STATE READ WRITE CKSUM > zroot ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > mfid4p4 ONLINE 0 0 0 > mfid0p4 ONLINE 0 0 0 > mfid1p4 ONLINE 0 0 0 > mfid3p4 ONLINE 0 0 0 > mfid2p4 ONLINE 0 0 0 > mfid5p4 ONLINE 0 0 0 > > errors: No known data errors > borg-new /home/ler $ zpool history > History for 'zroot': > cannot show history for pool 'zroot': permission denied > borg-new /home/ler $ sudo zpool history > Password: > History for 'zroot': > 2017-01-25.19:02:15 zpool create -o altroot=/mnt -O compress=lz4 -O atime=off -m none -f zroot raidz1 mfid0p4 mfid1p4 mfid2p4 mfid3p4 mfid4p4 mfid5p4 > 2017-01-25.19:02:15 zfs create -o mountpoint=none zroot/ROOT > 2017-01-25.19:02:15 zfs create -o mountpoint=/ zroot/ROOT/default > 2017-01-25.19:02:15 zfs create -o mountpoint=/tmp -o exec=on -o setuid=off zroot/tmp > 2017-01-25.19:02:15 zfs create -o mountpoint=/usr -o canmount=off zroot/usr > 2017-01-25.19:02:15 zfs create zroot/usr/home > 2017-01-25.19:02:15 zfs create -o setuid=off zroot/usr/ports > 2017-01-25.19:02:15 zfs create zroot/usr/src > 2017-01-25.19:02:15 zfs create -o mountpoint=/var -o canmount=off zroot/var > 2017-01-25.19:02:15 zfs create -o exec=off -o setuid=off zroot/var/audit > 2017-01-25.19:02:15 zfs create -o exec=off -o setuid=off zroot/var/crash > 2017-01-25.19:02:15 zfs create -o exec=off -o setuid=off zroot/var/log > 2017-01-25.19:02:15 zfs create -o atime=on zroot/var/mail > 2017-01-25.19:02:15 zfs create -o setuid=off zroot/var/tmp > 2017-01-25.19:02:15 zfs set mountpoint=/zroot zroot > 2017-01-25.19:02:15 zpool set bootfs=zroot/ROOT/default zroot > 2017-01-25.19:02:15 zpool set cachefile=/mnt/boot/zfs/zpool.cache zroot > 2017-01-25.19:02:19 zfs set canmount=noauto zroot/ROOT/default > 2017-01-26.19:51:23 zpool scrub zroot > 2017-01-29.12:50:56 zpool replace zroot 6435283924725851469 /dev/mfid5p4 > 2017-01-29.12:52:44 zpool scrub zroot > 2017-01-29.13:11:50 zfs create zroot/usr/obj > 2017-01-29.13:26:11 zfs create zroot/ler > 2017-01-29.13:27:58 zfs set mountpoint=/home/ler zroot/ler > 2017-01-29.16:22:07 zpool replace zroot 8173792548235216166 /dev/mfid4p4 > 2017-01-29.16:27:18 zpool scrub zroot > 2017-01-29.16:32:15 zfs create -o mountpoint=/home zroot/home > 2017-01-30.15:36:34 zpool scrub zroot > > borg-new /home/ler $ gpart show > => 40 3905945520 mfid0 GPT (1.8T) > 40 1600 1 efi (800K) > 1640 1024 2 freebsd-boot (512K) > 2664 1432 - free - (716K) > 4096 16777216 3 freebsd-swap (8.0G) > 16781312 3889162240 4 freebsd-zfs (1.8T) > 3905943552 2008 - free - (1.0M) > > => 40 3905945520 mfid1 GPT (1.8T) > 40 1600 1 efi (800K) > 1640 1024 2 freebsd-boot (512K) > 2664 1432 - free - (716K) > 4096 16777216 3 freebsd-swap (8.0G) > 16781312 3889162240 4 freebsd-zfs (1.8T) > 3905943552 2008 - free - (1.0M) > > => 40 3905945520 mfid2 GPT (1.8T) > 40 1600 1 efi (800K) > 1640 1024 2 freebsd-boot (512K) > 2664 1432 - free - (716K) > 4096 16777216 3 freebsd-swap (8.0G) > 16781312 3889162240 4 freebsd-zfs (1.8T) > 3905943552 2008 - free - (1.0M) > > => 40 3905945520 mfid3 GPT (1.8T) > 40 1600 1 efi (800K) > 1640 1024 2 freebsd-boot (512K) > 2664 16777216 3 freebsd-swap (8.0G) > 16779880 3889165680 4 freebsd-zfs (1.8T) > > => 40 3905945520 mfid5 GPT (1.8T) > 40 1600 1 efi (800K) > 1640 1024 2 freebsd-boot (512K) > 2664 1432 - free - (716K) > 4096 16777216 3 freebsd-swap (8.0G) > 16781312 3889162240 4 freebsd-zfs (1.8T) > 3905943552 2008 - free - (1.0M) > > => 40 3905945520 mfid4 GPT (1.8T) > 40 1600 1 efi (800K) > 1640 1024 2 freebsd-boot (512K) > 2664 1432 - free - (716K) > 4096 16777216 3 freebsd-swap (8.0G) > 16781312 3889160192 4 freebsd-zfs (1.8T) > 3905941504 4056 - free - (2.0M) > > borg-new /home/ler $ > > borg-new /home/ler $ uname -aKU > FreeBSD borg-new 12.0-CURRENT FreeBSD 12.0-CURRENT #1 r312893: Fri Jan 27 12:39:05 CST 2017 root@borg-new:/usr/obj/usr/src/sys/GENERIC amd64 1200020 1200020 > borg-new /home/ler $ > > Ideas? > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" -- Larry Rosenman http://people.freebsd.org/~ler [1] Phone: +1 214-642-9640 E-Mail: ler@FreeBSD.org US Mail: 17716 Limpia Crk, Round Rock, TX 78664-7281 -- Larry Rosenman http://people.freebsd.org/~ler Phone: +1 214-642-9640 E-Mail: ler@FreeBSD.org US Mail: 17716 Limpia Crk, Round Rock, TX 78664-7281 Links: ------ [1] http://people.freebsd.org/%7Eler From owner-freebsd-fs@freebsd.org Tue Jan 31 18:30:34 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5E4CCCC9A77 for ; Tue, 31 Jan 2017 18:30:34 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x236.google.com (mail-wm0-x236.google.com [IPv6:2a00:1450:400c:c09::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 13E3F101C for ; Tue, 31 Jan 2017 18:30:33 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x236.google.com with SMTP id r141so544659wmg.1 for ; Tue, 31 Jan 2017 10:30:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to; bh=+ffx5WybUGxLjd+96UH0FIpyC/mPQEzIu4u/r1FmqkE=; b=oeakjSnqPK7or7ucQfHLMJJa7ICbn5wn3+mncgoORtpg5UkpzCIU1huuha8lqQyUIo 5+wcxXxRISzu8u9ipmqOrMLCzec2VLNK1CvDKZjZB7KG16sU0tlvICEOP66uXvX8Jm2Q 0SO4PwpGa7b9RbvYgkGQA2O2FcqgiT4BDbCiOUQY+53YBd9baEiBMGNicC7HPsXBoFe9 Cd2AUTtKLBlhc867XM0l07oYxAbsVTooMJ2HaE1ysfc4P0V7b1pdGh5aCYpRS3PkXtCJ NDG8XtXYMxr8SUTLmDkbHX0eSjoRRNvEcZ2GFuhMqPKclk+SV3Dq9DZmdIfQyQGp4y8O tykw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=+ffx5WybUGxLjd+96UH0FIpyC/mPQEzIu4u/r1FmqkE=; b=LVu9xTdv9qd6WndL9Y5TenHz7yZb+FnGokqQtwgg9sHwl65tQxE2akxYkxD981yl2s 7toCGNPxAmK/ugAphWynNlbDrmlB+4zsOB3bhDc7SmaV0+lcD7FCLThAh0xdP3jn5NhX oNsHLoEVHi7Fx83drqqm+ADEBr0NBl4wgg6IvjxUoMqAt7JmBOrAJ/TKUH7r9hEwQZsr szlkuHsVMBmijdufyiOviJ/ORxZydW4gZCwwCPhNLShteyd6XtLTnTX2NNLxS9y6X5Yn heLx17qZbt7s1enRD1sLgUxCyCfcHjrQM0lFfH61b2NNc66o9dLDSKjO9OJK5yt84xPH uNIg== X-Gm-Message-State: AIkVDXJYsPYa4IywcJKauJU7T5ej8agecf6q9JUJsx+s3GSfWqcj7kHEHGWIGbvCs+JfI2VF X-Received: by 10.28.58.204 with SMTP id h195mr19587695wma.116.1485887431269; Tue, 31 Jan 2017 10:30:31 -0800 (PST) Received: from [10.10.1.58] ([185.97.61.26]) by smtp.gmail.com with ESMTPSA id 18sm29701725wrb.14.2017.01.31.10.30.30 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 31 Jan 2017 10:30:30 -0800 (PST) Subject: Re: 16.0E ExpandSize? -- New Server To: Larry Rosenman References: <00db0ab7243ce6368c246ae20f9c075a@FreeBSD.org> <1a69057c-dc59-9b78-9762-4f98a071105e@multiplay.co.uk> <35a9034f91542bb1329ac5104bf3b773@FreeBSD.org> <76fc9505-f681-0de0-fe0c-5624b29de321@multiplay.co.uk> <22e1bfc5840d972cf93643733682cda1@FreeBSD.org> Cc: Freebsd fs From: Steven Hartland Message-ID: Date: Tue, 31 Jan 2017 18:30:31 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-Version: 1.0 In-Reply-To: <22e1bfc5840d972cf93643733682cda1@FreeBSD.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jan 2017 18:30:34 -0000 Your issue is the reported vdev_max_asize > vdev_asize: vdev_max_asize: 11947471798272 vdev_asize: 11947478089728 max asize is smaller than asize by 6291456 For raidz1 Xsize should be the smallest disk Xsize * disks so: 1991245299712 * 6 = 11947471798272 So your max asize looks right but asize looks too big Expand Size is calculated by: if (vd->vdev_aux == NULL && tvd != NULL && vd->vdev_max_asize != 0) { vs->vs_esize = P2ALIGN(vd->vdev_max_asize - vd->vdev_asize, 1ULL << tvd->vdev_ms_shift); } So the question is why is asize too big? Given you seem to have some random disk sizes do you have auto expand turned on? On 31/01/2017 17:39, Larry Rosenman wrote: > vdev_path: n/a, vdev_max_asize: 11947471798272, vdev_asize: 11947478089728