Date: Tue, 31 Jan 2017 13:05:11 -0600 From: Larry Rosenman <ler@FreeBSD.org> To: Steven Hartland <killing@multiplay.co.uk> Cc: Freebsd fs <freebsd-fs@freebsd.org> Subject: Re: 16.0E ExpandSize? -- New Server Message-ID: <8a710dc75c129f58b0372eeaeca575b5@FreeBSD.org> In-Reply-To: <f2600a53-0dc1-9f41-1405-ed22d96d30cf@multiplay.co.uk> References: <00db0ab7243ce6368c246ae20f9c075a@FreeBSD.org> <1a69057c-dc59-9b78-9762-4f98a071105e@multiplay.co.uk> <ce5a1d39612d694077accda33266a3ab@FreeBSD.org> <ad07e84e-f297-362a-1398-c5503bb56a8d@multiplay.co.uk> <35a9034f91542bb1329ac5104bf3b773@FreeBSD.org> <76fc9505-f681-0de0-fe0c-5624b29de321@multiplay.co.uk> <22e1bfc5840d972cf93643733682cda1@FreeBSD.org> <f2600a53-0dc1-9f41-1405-ed22d96d30cf@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
I've replaced some disks due to failure, and some of the pariition sizes
are different.
autoexpand is off:
borg-new /home/ler $ zpool get all zroot
NAME PROPERTY VALUE SOURCE
zroot size 10.8T -
zroot capacity 0% -
zroot altroot - default
zroot health ONLINE -
zroot guid 11945658884309024932 default
zroot version - default
zroot bootfs zroot/ROOT/default local
zroot delegation on default
zroot autoreplace off default
zroot cachefile - default
zroot failmode wait default
zroot listsnapshots off default
zroot autoexpand off default
zroot dedupditto 0 default
zroot dedupratio 1.00x -
zroot free 10.7T -
zroot allocated 94.3G -
zroot readonly off -
zroot comment - default
zroot expandsize 16.0E -
zroot freeing 0 default
zroot fragmentation 0% -
zroot leaked 0 default
zroot feature@async_destroy enabled local
zroot feature@empty_bpobj active local
zroot feature@lz4_compress active local
zroot feature@multi_vdev_crash_dump enabled local
zroot feature@spacemap_histogram active local
zroot feature@enabled_txg active local
zroot feature@hole_birth active local
zroot feature@extensible_dataset enabled local
zroot feature@embedded_data active local
zroot feature@bookmarks enabled local
zroot feature@filesystem_limits enabled local
zroot feature@large_blocks enabled local
zroot feature@sha512 enabled local
zroot feature@skein enabled local
borg-new /home/ler $
borg-new /home/ler $ gpart show
=> 40 3905945520 mfid0 GPT (1.8T)
40 1600 1 efi (800K)
1640 1024 2 freebsd-boot (512K)
2664 1432 - free - (716K)
4096 16777216 3 freebsd-swap (8.0G)
16781312 3889162240 4 freebsd-zfs (1.8T)
3905943552 2008 - free - (1.0M)
=> 40 3905945520 mfid1 GPT (1.8T)
40 1600 1 efi (800K)
1640 1024 2 freebsd-boot (512K)
2664 1432 - free - (716K)
4096 16777216 3 freebsd-swap (8.0G)
16781312 3889162240 4 freebsd-zfs (1.8T)
3905943552 2008 - free - (1.0M)
=> 40 3905945520 mfid2 GPT (1.8T)
40 1600 1 efi (800K)
1640 1024 2 freebsd-boot (512K)
2664 1432 - free - (716K)
4096 16777216 3 freebsd-swap (8.0G)
16781312 3889162240 4 freebsd-zfs (1.8T)
3905943552 2008 - free - (1.0M)
=> 40 3905945520 mfid3 GPT (1.8T)
40 1600 1 efi (800K)
1640 1024 2 freebsd-boot (512K)
2664 16777216 3 freebsd-swap (8.0G)
16779880 3889165680 4 freebsd-zfs (1.8T)
=> 40 3905945520 mfid5 GPT (1.8T)
40 1600 1 efi (800K)
1640 1024 2 freebsd-boot (512K)
2664 1432 - free - (716K)
4096 16777216 3 freebsd-swap (8.0G)
16781312 3889162240 4 freebsd-zfs (1.8T)
3905943552 2008 - free - (1.0M)
=> 40 3905945520 mfid4 GPT (1.8T)
40 1600 1 efi (800K)
1640 1024 2 freebsd-boot (512K)
2664 1432 - free - (716K)
4096 16777216 3 freebsd-swap (8.0G)
16781312 3889160192 4 freebsd-zfs (1.8T)
3905941504 4056 - free - (2.0M)
borg-new /home/ler $
this system was built last week, and I **CAN** rebuild it if necessary,
but I didn't do anything strange (so I thought :) )
On 01/31/2017 12:30 pm, Steven Hartland wrote:
> Your issue is the reported vdev_max_asize > vdev_asize:
> vdev_max_asize: 11947471798272
> vdev_asize: 11947478089728
>
> max asize is smaller than asize by 6291456
>
> For raidz1 Xsize should be the smallest disk Xsize * disks so:
> 1991245299712 * 6 = 11947471798272
>
> So your max asize looks right but asize looks too big
>
> Expand Size is calculated by:
> if (vd->vdev_aux == NULL && tvd != NULL && vd->vdev_max_asize != 0) {
> vs->vs_esize = P2ALIGN(vd->vdev_max_asize - vd->vdev_asize,
> 1ULL << tvd->vdev_ms_shift);
> }
>
> So the question is why is asize too big?
>
> Given you seem to have some random disk sizes do you have auto expand turned on?
>
> On 31/01/2017 17:39, Larry Rosenman wrote:
>
>> vdev_path: n/a, vdev_max_asize: 11947471798272, vdev_asize: 11947478089728
--
Larry Rosenman http://people.freebsd.org/~ler
Phone: +1 214-642-9640 E-Mail: ler@FreeBSD.org
US Mail: 17716 Limpia Crk, Round Rock, TX 78664-7281
From owner-freebsd-fs@freebsd.org Tue Jan 31 20:37:16 2017
Return-Path: <owner-freebsd-fs@freebsd.org>
Delivered-To: freebsd-fs@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
[IPv6:2001:1900:2254:206a::19:1])
by mailman.ysv.freebsd.org (Postfix) with ESMTP id A914CCCA888
for <freebsd-fs@mailman.ysv.freebsd.org>; Tue, 31 Jan 2017 20:37:16 +0000 (UTC)
(envelope-from killing@multiplay.co.uk)
Received: from mail-wm0-x22f.google.com (mail-wm0-x22f.google.com
[IPv6:2a00:1450:400c:c09::22f])
(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
(Client CN "smtp.gmail.com",
Issuer "Google Internet Authority G2" (verified OK))
by mx1.freebsd.org (Postfix) with ESMTPS id 558A0EE6
for <freebsd-fs@freebsd.org>; Tue, 31 Jan 2017 20:37:16 +0000 (UTC)
(envelope-from killing@multiplay.co.uk)
Received: by mail-wm0-x22f.google.com with SMTP id 196so19653740wmm.1
for <freebsd-fs@freebsd.org>; Tue, 31 Jan 2017 12:37:16 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623;
h=subject:to:references:cc:from:message-id:date:user-agent
:mime-version:in-reply-to;
bh=7dzmzWdF3F6V4hIEJtOWa4t7QG48jkyis8b5/eTMON4=;
b=jN4v/8i2A3yIlPXFFd2TiZxBZPPHUtAhHGM+BHR0/aWAhbRhDYbL2s2jJiAasaMdx8
CextaQHoJS89SxCdCdmWhvM9xfAeNCLEa+mi/UFL46E+5IwaC6EsG0Ymap1sKsIB0P88
UgAuLqQrGOIrQ67cRNDkx3/XMqvySoEGa3i7/DgEVIO1frDystvz+7ZiH+iBAwgj2cSe
L2QKTOojnNvlIg7MuTkUqPOkq4WtFBjVdY4NISCc604l+FBCjFdRsmmhO0EI/16wgRtX
+6yayYgbfpg81wnjItOe0cIdMKAsubC01cwr5+wGXVC0pciwpfJllnB6o0x2Zl6PZwVt
kPSQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:subject:to:references:cc:from:message-id:date
:user-agent:mime-version:in-reply-to;
bh=7dzmzWdF3F6V4hIEJtOWa4t7QG48jkyis8b5/eTMON4=;
b=no1X6v5E3r+H612H9A/Ae5jNOiojykkIZ5vw00S+QT0/mD0TYA8X2vUli9fYw6FWTN
edaVKaBkVZwEedbKqcWgO9qTr0KHrzA190JGxxjD2RPa6r8w9EE02oiYB9FyvL6e22TB
cGq7jZfJykJ9gSGVuLSfKET9QR6CvSZ0jg/x0s49fobZe5kKlrFfYhBW1RVCXuU1wMbf
VrZDjWiXhpvcJR46Qb//8hinrCey59b3YBhzn+T9ntadQM9nNpfe9nAO80Wb0za5SWjF
+TodzKlJVy4Cb+gWLLu/JE0d+c2CpsWG3SPysL/7YRQf76fV1l5KxR0n7i3ZilFyxhlf
3ljg==
X-Gm-Message-State: AIkVDXJrcO2pKQB7Qv45HHZQ0Iiu9Bsk10CX7x2FAWjvxZ+ie6RiAHNY0qJPYRKu/rbZO407
X-Received: by 10.223.171.12 with SMTP id q12mr5831546wrc.74.1485895034517;
Tue, 31 Jan 2017 12:37:14 -0800 (PST)
Received: from [10.10.1.58] ([185.97.61.26])
by smtp.gmail.com with ESMTPSA id l37sm30231742wrc.41.2017.01.31.12.37.13
(version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
Tue, 31 Jan 2017 12:37:13 -0800 (PST)
Subject: Re: 16.0E ExpandSize? -- New Server
To: Larry Rosenman <ler@FreeBSD.org>
References: <00db0ab7243ce6368c246ae20f9c075a@FreeBSD.org>
<1a69057c-dc59-9b78-9762-4f98a071105e@multiplay.co.uk>
<ce5a1d39612d694077accda33266a3ab@FreeBSD.org>
<ad07e84e-f297-362a-1398-c5503bb56a8d@multiplay.co.uk>
<35a9034f91542bb1329ac5104bf3b773@FreeBSD.org>
<76fc9505-f681-0de0-fe0c-5624b29de321@multiplay.co.uk>
<22e1bfc5840d972cf93643733682cda1@FreeBSD.org>
<f2600a53-0dc1-9f41-1405-ed22d96d30cf@multiplay.co.uk>
<8a710dc75c129f58b0372eeaeca575b5@FreeBSD.org>
Cc: Freebsd fs <freebsd-fs@freebsd.org>
From: Steven Hartland <killing@multiplay.co.uk>
Message-ID: <aef02eb0-0888-6fea-a4b8-4033ca56f4a3@multiplay.co.uk>
Date: Tue, 31 Jan 2017 20:37:14 +0000
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101
Thunderbird/45.7.0
MIME-Version: 1.0
In-Reply-To: <8a710dc75c129f58b0372eeaeca575b5@FreeBSD.org>
Content-Type: multipart/mixed; boundary="------------E4FD68EB3E67CE5F2306839B"
X-Content-Filtered-By: Mailman/MimeDel 2.1.23
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>;
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 31 Jan 2017 20:37:16 -0000
This is a multi-part message in MIME format.
--------------E4FD68EB3E67CE5F2306839B
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
In that case based on your zpool history I suspect that the original
mfid4p4 was the same size as mfid0p4 (1991246348288) but its been
replaced with a drive which is (1991245299712), slightly smaller.
This smaller size results in a max_asize of 1991245299712 * 6 instead of
original 1991246348288* 6.
Now given the way min_asize (the value used to check if the device size
is acceptable) is rounded to the the nearest metaslab I believe that
replace would be allowed.
https://github.com/freebsd/freebsd/blob/master/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c#L4947
Now the problem is that on open the calculated asize is only updated if
its expanding:
https://github.com/freebsd/freebsd/blob/master/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c#L1424
The updated dtrace file outputs vdev_min_asize which should confirm my
suspicion about why the replace was allowed.
On 31/01/2017 19:05, Larry Rosenman wrote:
>
> I've replaced some disks due to failure, and some of the pariition
> sizes are different.
>
>
> autoexpand is off:
>
> borg-new /home/ler $ zpool get all zroot
> NAME PROPERTY VALUE SOURCE
> zroot size 10.8T -
> zroot capacity 0% -
> zroot altroot - default
> zroot health ONLINE -
> zroot guid 11945658884309024932 default
> zroot version - default
> zroot bootfs zroot/ROOT/default local
> zroot delegation on default
> zroot autoreplace off default
> zroot cachefile - default
> zroot failmode wait default
> zroot listsnapshots off default
> zroot autoexpand off default
> zroot dedupditto 0 default
> zroot dedupratio 1.00x -
> zroot free 10.7T -
> zroot allocated 94.3G -
> zroot readonly off -
> zroot comment - default
> zroot expandsize 16.0E -
> zroot freeing 0 default
> zroot fragmentation 0% -
> zroot leaked 0 default
> zroot feature@async_destroy enabled local
> zroot feature@empty_bpobj active local
> zroot feature@lz4_compress active local
> zroot feature@multi_vdev_crash_dump enabled local
> zroot feature@spacemap_histogram active local
> zroot feature@enabled_txg active local
> zroot feature@hole_birth active local
> zroot feature@extensible_dataset enabled local
> zroot feature@embedded_data active local
> zroot feature@bookmarks enabled local
> zroot feature@filesystem_limits enabled local
> zroot feature@large_blocks enabled local
> zroot feature@sha512 enabled local
> zroot feature@skein enabled local
> borg-new /home/ler $
>
>
> borg-new /home/ler $ gpart show
> => 40 3905945520 mfid0 GPT (1.8T)
> 40 1600 1 efi (800K)
> 1640 1024 2 freebsd-boot (512K)
> 2664 1432 - free - (716K)
> 4096 16777216 3 freebsd-swap (8.0G)
> 16781312 3889162240 4 freebsd-zfs (1.8T)
> 3905943552 2008 - free - (1.0M)
>
> => 40 3905945520 mfid1 GPT (1.8T)
> 40 1600 1 efi (800K)
> 1640 1024 2 freebsd-boot (512K)
> 2664 1432 - free - (716K)
> 4096 16777216 3 freebsd-swap (8.0G)
> 16781312 3889162240 4 freebsd-zfs (1.8T)
> 3905943552 2008 - free - (1.0M)
>
> => 40 3905945520 mfid2 GPT (1.8T)
> 40 1600 1 efi (800K)
> 1640 1024 2 freebsd-boot (512K)
> 2664 1432 - free - (716K)
> 4096 16777216 3 freebsd-swap (8.0G)
> 16781312 3889162240 4 freebsd-zfs (1.8T)
> 3905943552 2008 - free - (1.0M)
>
> => 40 3905945520 mfid3 GPT (1.8T)
> 40 1600 1 efi (800K)
> 1640 1024 2 freebsd-boot (512K)
> 2664 16777216 3 freebsd-swap (8.0G)
> 16779880 3889165680 4 freebsd-zfs (1.8T)
>
> => 40 3905945520 mfid5 GPT (1.8T)
> 40 1600 1 efi (800K)
> 1640 1024 2 freebsd-boot (512K)
> 2664 1432 - free - (716K)
> 4096 16777216 3 freebsd-swap (8.0G)
> 16781312 3889162240 4 freebsd-zfs (1.8T)
> 3905943552 2008 - free - (1.0M)
>
> => 40 3905945520 mfid4 GPT (1.8T)
> 40 1600 1 efi (800K)
> 1640 1024 2 freebsd-boot (512K)
> 2664 1432 - free - (716K)
> 4096 16777216 3 freebsd-swap (8.0G)
> 16781312 3889160192 4 freebsd-zfs (1.8T)
> 3905941504 4056 - free - (2.0M)
>
> borg-new /home/ler $
>
>
> this system was built last week, and I **CAN** rebuild it if
> necessary, but I didn't do anything strange (so I thought :) )
>
>
>
>
> On 01/31/2017 12:30 pm, Steven Hartland wrote:
>
>> Your issue is the reported vdev_max_asize > vdev_asize:
>> vdev_max_asize: 11947471798272
>> vdev_asize: 11947478089728
>>
>> max asize is smaller than asize by 6291456
>>
>> For raidz1 Xsize should be the smallest disk Xsize * disks so:
>> 1991245299712 * 6 = 11947471798272
>>
>> So your max asize looks right but asize looks too big
>>
>> Expand Size is calculated by:
>> if (vd->vdev_aux == NULL && tvd != NULL && vd->vdev_max_asize != 0) {
>> vs->vs_esize = P2ALIGN(vd->vdev_max_asize - vd->vdev_asize,
>> 1ULL << tvd->vdev_ms_shift);
>> }
>>
>> So the question is why is asize too big?
>>
>> Given you seem to have some random disk sizes do you have auto expand
>> turned on?
>>
>> On 31/01/2017 17:39, Larry Rosenman wrote:
>>> vdev_path: n/a, vdev_max_asize: 11947471798272, vdev_asize:
>>> 11947478089728
>
>
> --
> Larry Rosenman http://people.freebsd.org/~ler
> <http://people.freebsd.org/%7Eler>;
> Phone: +1 214-642-9640 E-Mail: ler@FreeBSD.org
> <mailto:ler@FreeBSD.org>
> US Mail: 17716 Limpia Crk, Round Rock, TX 78664-7281
--------------E4FD68EB3E67CE5F2306839B
Content-Type: text/plain; charset=UTF-8;
name="vdev-stats.d"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
filename="vdev-stats.d"
IyEvdXNyL3NiaW4vZHRyYWNlIC1zCgojcHJhZ21hIEQgb3B0aW9uIHF1aWV0CgpmYnQ6OnZk
ZXZfZ2V0X3N0YXRzOmVudHJ5CnsKCXZkID0gKHZkZXZfdCAqKWFyZzA7CiAgICAgICAgcHJp
bnRmKCJ2ZGV2X3BhdGg6ICVzLCB2ZGV2X21heF9hc2l6ZTogJWx1LCB2ZGV2X2FzaXplOiAl
bHUsIHZkZXZfbWluX2FzaXplOiAlbHVcbiIsCgkJdmQtPnZkZXZfcGF0aCA/IHN0cmluZ29m
KHZkLT52ZGV2X3BhdGgpIDogIm4vYSIsIHZkLT52ZGV2X21heF9hc2l6ZSwgdmQtPnZkZXZf
YXNpemUsIHZkLT52ZGV2X21pbl9hc2l6ZSk7Cn0K
--------------E4FD68EB3E67CE5F2306839B--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?8a710dc75c129f58b0372eeaeca575b5>