From owner-freebsd-questions@freebsd.org Thu Dec 12 12:42:51 2019
Return-Path: <owner-freebsd-questions@freebsd.org>
Delivered-To: freebsd-questions@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1DC941D05D0
for <freebsd-questions@mailman.nyi.freebsd.org>;
Thu, 12 Dec 2019 12:42:51 +0000 (UTC)
(envelope-from Norman.Gray@glasgow.ac.uk)
Received: from plockton.cent.gla.ac.uk (plockton.cent.gla.ac.uk
[130.209.16.75])
(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
(Client did not present a certificate)
by mx1.freebsd.org (Postfix) with ESMTPS id 47YYNp1JYmz4VKf
for <freebsd-questions@freebsd.org>; Thu, 12 Dec 2019 12:42:49 +0000 (UTC)
(envelope-from Norman.Gray@glasgow.ac.uk)
Received: from cas08.campus.gla.ac.uk ([130.209.14.165])
by plockton.cent.gla.ac.uk with esmtp (Exim 4.72)
(envelope-from <Norman.Gray@glasgow.ac.uk>)
id 1ifNnr-0001WJ-DV; Thu, 12 Dec 2019 12:42:47 +0000
Received: from cas07.campus.gla.ac.uk (130.209.14.164) by
cas08.campus.gla.ac.uk (130.209.14.165) with Microsoft SMTP Server (TLS) id
15.0.1497.2; Thu, 12 Dec 2019 12:42:47 +0000
Received: from GBR01-LO2-obe.outbound.protection.outlook.com (104.47.21.53) by
cas07.campus.gla.ac.uk (130.209.14.164) with Microsoft SMTP Server
(TLS) id
15.0.1497.2 via Frontend Transport; Thu, 12 Dec 2019 12:42:46 +0000
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
b=RIvCU3ueHTz0EhI0oz1dwU9tqDoN/Tzs9JqgFSQYtEZ9nCaAVh31sc00iCrqbf68O++7DHxj5XxcEcgk89YzdFxa01ZCq2LAqKiGdZuI4t0nQeuSXBzKwJWUmuYx8hmmBp9TM1xjnWyBYll4+cF0X0ieRiZd6toHw1F3J+Onrsf2mgwSNpQXdH1Cl/OMKFfCLN06rYQg4emZzVnggZ9lBSENAY8+cYZ1yTnZdCnxDVxxYsCVgd7Z2gxCdCZKpgQ3pkf1WzMa3BX/C/IKRELg71DFCK0hxAGR1/rYG35yFtwzXk5GAD3eG7Ak7Uo0r+lO8egK8Nizb74XwGr2IhPoOA==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
s=arcselector9901;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
bh=2/wAIUSDI21Bch9FSmmi9ILJlFqwHwagRoD3BBRvd7w=;
b=Y9mC414TzwnqY2xPF2s80OaZz/bExnkM9z7KqYT85ne0pGHghtcdT2wiGaK4H8V4D8dNfdsva7/hklkWe0EUiQA3Daemez/7RxnjmHAUT5Nxwvpbh3sVHA1Ka7eU2QZC4PSqbaf7f8I2U/f1E6VdF+2eWUVTmqyINWFrJZN6asFxrfRqJYGkjZBOpYRueZe+vK2mENr/g3m8zCpHfdG1tPlmzUql12rNy4JykwOhmIX4BRMPzs8d6ou0Dog5JspZBlZ0s46CxKKNulWt56Yfr+PE4HWMGMT7MJXKGbppixl2RC2vI+MwCudDcmtxqKAQW7S6SbAZaRtdlYKcab84qQ==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
smtp.mailfrom=glasgow.ac.uk; dmarc=pass action=none
header.from=glasgow.ac.uk; dkim=pass header.d=glasgow.ac.uk; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gla.onmicrosoft.com;
s=selector2-gla-onmicrosoft-com;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
bh=2/wAIUSDI21Bch9FSmmi9ILJlFqwHwagRoD3BBRvd7w=;
b=EP0iT6jS+1DOds9KtEGYoIAyBXryScJOjAAF0GmRTMsYc9AnELHWq+GnfMzPAP9HXcBN9PRIoSedez2gREjfEYmSYWxKI/sDmQxJj85YHTHazRJxQGp4AT4QYSHFCX0Ziu0ZA1Yqtgn2Ekiu+4hLYcpXxiEiFNnPKwd8GlpoNFg=
Received: from CWXP265MB0149.GBRP265.PROD.OUTLOOK.COM (10.164.144.19) by
CWXP265MB1158.GBRP265.PROD.OUTLOOK.COM (20.176.46.20) with Microsoft SMTP
Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
15.20.2516.12; Thu, 12 Dec 2019 12:42:45 +0000
Received: from CWXP265MB0149.GBRP265.PROD.OUTLOOK.COM
([fe80::c8c6:434c:2a1a:970f]) by CWXP265MB0149.GBRP265.PROD.OUTLOOK.COM
([fe80::c8c6:434c:2a1a:970f%5]) with mapi id 15.20.2516.019; Thu, 12 Dec 2019
12:42:45 +0000
From: Norman Gray <Norman.Gray@glasgow.ac.uk>
To: David Christensen <dpchrist@holgerdanske.com>
CC: "freebsd-questions@freebsd.org" <freebsd-questions@freebsd.org>
Subject: Re: Adding to a zpool -- different redundancies and risks
Thread-Topic: Adding to a zpool -- different redundancies and risks
Thread-Index: AQHVsEnhosdAKOTbGEOdhXJjA5RdA6e19JcAgAB+IQA=
Date: Thu, 12 Dec 2019 12:42:45 +0000
Message-ID: <5A01F7F7-9326-47E2-BA6E-79A7D3F0889A@glasgow.ac.uk>
References: <6104097C-009B-4E9C-A1D8-A2D0E5FECADF@glasgow.ac.uk>
<09b11639-3303-df6b-f70c-6722caaacee7@holgerdanske.com>
In-Reply-To: <09b11639-3303-df6b-f70c-6722caaacee7@holgerdanske.com>
Accept-Language: en-GB, en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [130.209.45.140]
x-clientproxiedby: LO2P265CA0297.GBRP265.PROD.OUTLOOK.COM
(2603:10a6:600:a5::21) To CWXP265MB0149.GBRP265.PROD.OUTLOOK.COM
(2603:10a6:401:8::19)
x-ms-exchange-messagesentrepresentingtype: 1
x-mailer: MailMate (1.13.1r5671)
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: 6632e226-8cf0-4b18-06be-08d77f00c989
x-ms-traffictypediagnostic: CWXP265MB1158:
x-ms-exchange-transport-forked: True
x-microsoft-antispam-prvs: <CWXP265MB115820C3490E04AA1994485AB2550@CWXP265MB1158.GBRP265.PROD.OUTLOOK.COM>
x-ms-oob-tlc-oobclassifiers: OLM:10000;
x-forefront-prvs: 0249EFCB0B
x-forefront-antispam-report: SFV:NSPM;
SFS:(10009020)(136003)(376002)(39860400002)(396003)(366004)(346002)(189003)(199004)(5660300002)(66946007)(36756003)(786003)(316002)(2616005)(6512007)(66476007)(66556008)(64756008)(71200400001)(44832011)(66446008)(6486002)(4326008)(86362001)(2906002)(8676002)(186003)(6916009)(26005)(33656002)(81156014)(478600001)(53546011)(6506007)(966005)(8936002)(81166006)(52116002);
DIR:OUT; SFP:1101; SCL:1; SRVR:CWXP265MB1158;
H:CWXP265MB0149.GBRP265.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en;
PTR:InfoNoRecords; MX:1; A:1;
received-spf: None (protection.outlook.com: glasgow.ac.uk does not designate
permitted sender hosts)
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: TGKdB+YdVS8o5G9tLy8SIkT23BgvBTmjCFT7jl1fVTYUjq7HOGQvigvYX0gGwEQx0YIHdAKXbY7o5e9T99KUWaWAwJZv/D0wBDkzwfPn5GAeYVliwY2BpBD0+TjRovRe+NbIXdDfM/SjuOvQTaEgJHCF9CCo+XlgEBRGm5/M/FQHqGPbD8UVhjNBVCWW1gkaKQpW1nnlQ911Wo1JVdIoW2kqDqJ+WuVqXrAhw4ysT6y3/JHnS1bUSky8NnpF8inR6g3yRlDzPXxcqrAAwurAHabN+EmjL3ArNUPzQZsGBupJTzo2gbZEMUWRtZzJy9NtfydGU1z/DFLK16eiBp99A2nT6bOMXdcRzni9Xr4ggPNGGEWpkG24c6mWOmYGHFMlsx1oslZjhuhcMhe/QBnAnrRVN31alQ9perYRMtIzEODXECcOadBskivY1FWNMxZe
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 6632e226-8cf0-4b18-06be-08d77f00c989
X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Dec 2019 12:42:45.6071 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 6e725c29-763a-4f50-81f2-2e254f0133c8
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: ue/fWQI7+Klg/uxDk5Kre5zgG7v7VFdAzK5CTkyS1nH6HsYfjIeNr1RgVOzTbBaaG+0zAIW+qZYwp+k9UEa+eiOYm6K0kUo8CUZvWbPdp7s=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CWXP265MB1158
X-OriginatorOrg: glasgow.ac.uk
X-Rspamd-Queue-Id: 47YYNp1JYmz4VKf
X-Spamd-Bar: ----
Authentication-Results: mx1.freebsd.org;
dkim=pass header.d=gla.onmicrosoft.com header.s=selector2-gla-onmicrosoft-com
header.b=EP0iT6jS; dmarc=none;
spf=none (mx1.freebsd.org: domain of Norman.Gray@glasgow.ac.uk has no SPF
policy when checking 130.209.16.75) smtp.mailfrom=Norman.Gray@glasgow.ac.uk
X-Spamd-Result: default: False [-4.89 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[];
NEURAL_HAM_MEDIUM(-1.00)[-1.000,0];
R_DKIM_ALLOW(-0.20)[gla.onmicrosoft.com:s=selector2-gla-onmicrosoft-com];
HAS_XOIP(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[];
IP_SCORE(-0.89)[ipnet: 130.209.0.0/16(-3.89), asn: 786(-0.47), country:
GB(-0.08)]; MIME_GOOD(-0.10)[text/plain];
DMARC_NA(0.00)[glasgow.ac.uk];
RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; RCVD_COUNT_FIVE(0.00)[6];
NEURAL_HAM_LONG(-1.00)[-1.000,0];
TO_MATCH_ENVRCPT_SOME(0.00)[];
RCVD_IN_DNSWL_MED(-0.20)[75.16.209.130.list.dnswl.org : 127.0.11.2];
DKIM_TRACE(0.00)[gla.onmicrosoft.com:+];
RCPT_COUNT_TWO(0.00)[2]; R_SPF_NA(0.00)[];
FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+];
RCVD_TLS_LAST(0.00)[];
ASN(0.00)[asn:786, ipnet:130.209.0.0/16, country:GB];
ARC_ALLOW(-1.00)[i=1]; MID_RHS_MATCH_FROM(0.00)[]
X-BeenThere: freebsd-questions@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: User questions <freebsd-questions.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-questions>,
<mailto:freebsd-questions-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-questions/>
List-Post: <mailto:freebsd-questions@freebsd.org>
List-Help: <mailto:freebsd-questions-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-questions>,
<mailto:freebsd-questions-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 12 Dec 2019 12:42:51 -0000
David, hello.
On 12 Dec 2019, at 5:11, David Christensen wrote:
> Please post:
>
> 1 The 'zpool create ...' command you used to create the existing=20
> pool.
I don't have a note of the exact command, but it would have been=20
something like
zpool create pool raidz2 da{0,1,2,3,4,5,6,7,8} raidz2 da9=20
da1{0,1,2,3,4,5,6,7}
> 2. The output of 'zpool status' for the existing pool.
# zpool status pool
pool: pool
state: ONLINE
status: Some supported features are not enabled on the pool. The pool=20
can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: none requested
config:
NAME STATE READ WRITE CKSUM
pool ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
label/zd032 ONLINE 0 0 0
label/zd033 ONLINE 0 0 0
label/zd034 ONLINE 0 0 0
label/zd035 ONLINE 0 0 0
label/zd036 ONLINE 0 0 0
label/zd037 ONLINE 0 0 0
label/zd038 ONLINE 0 0 0
label/zd039 ONLINE 0 0 0
label/zd040 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
label/zd041 ONLINE 0 0 0
label/zd042 ONLINE 0 0 0
label/zd043 ONLINE 0 0 0
label/zd044 ONLINE 0 0 0
label/zd045 ONLINE 0 0 0
label/zd046 ONLINE 0 0 0
label/zd047 ONLINE 0 0 0
label/zd048 ONLINE 0 0 0
label/zd049 ONLINE 0 0 0
errors: No known data errors
#
(Note: since creating the pool, I realised that gpart labels were a Good=20
Thing, hence exported, labelled, and imported the pool, hence the=20
difference from the da* pool creation).
> 3. The output of 'zpool list' for the existing pool.
# zpool list pool
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP =20
HEALTH ALTROOT
pool 98T 75.2T 22.8T - - 29% 76% 1.00x =20
ONLINE -
> 4. The 'zpool add ...' command you are contemplating.
# zpool add -n pool raidz2 label/zd05{0,1,2,3,4,5}
invalid vdev specification
use '-f' to override the following errors:
mismatched replication level: pool uses 9-way raidz and new vdev uses=20
6-way raidz
The six new disks are 12TB; the 18 original ones 5.5TB.
> So, you have 24 drives in a 24 drive cage?
That's correct -- the maximum the chassis will take.
> What are your space and performance goals?
Not very explicit: TB/currency-unit as high as possible. Performance:=20
bottlenecks are likely to be elsewhere (network, processing power) so no=20
stringent requirements. Though this is a fairly general-purpose data=20
store, a large fraction of the datasets on the machine comprise a number=20
of 10GB single files, served via NFS.
> What are your sustainability goals as drives and/or VDEV's fail?
It doesn't have to be high availability, so if I have a drive failure, I=20
can consider shutting the machine down until a replacement disk arrives=20
and can be resilvered. This is a mirror of data where the masters are=20
elsewhere on the planet, so this machine is 'reliable storage but not=20
backed up' (and the users know this). Thus if I do decide to keep=20
running with one failed disk in one VDEV, and the worst comes to the=20
worst and the whole thing explodes... the world won't end. I will be=20
cross, and users will moan, in either case, but they know this is a=20
problem that can fundamentally be solved with more money.
I'm sure I could be more sophisticated about this (and any suggestions=20
are welcome), but unfortunately I don't have as much time to spend on=20
storage problems as I'd like, so I'd like to avoid creating a setup=20
which is smarter than I'm able to fix!
Best wishes,
Norman
--=20
Norman Gray : https://nxg.me.uk
SUPA School of Physics and Astronomy, University of Glasgow, UK