From owner-freebsd-questions@freebsd.org Thu Dec 12 12:42:51 2019 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1DC941D05D0 for ; Thu, 12 Dec 2019 12:42:51 +0000 (UTC) (envelope-from Norman.Gray@glasgow.ac.uk) Received: from plockton.cent.gla.ac.uk (plockton.cent.gla.ac.uk [130.209.16.75]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47YYNp1JYmz4VKf for ; Thu, 12 Dec 2019 12:42:49 +0000 (UTC) (envelope-from Norman.Gray@glasgow.ac.uk) Received: from cas08.campus.gla.ac.uk ([130.209.14.165]) by plockton.cent.gla.ac.uk with esmtp (Exim 4.72) (envelope-from ) id 1ifNnr-0001WJ-DV; Thu, 12 Dec 2019 12:42:47 +0000 Received: from cas07.campus.gla.ac.uk (130.209.14.164) by cas08.campus.gla.ac.uk (130.209.14.165) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 12 Dec 2019 12:42:47 +0000 Received: from GBR01-LO2-obe.outbound.protection.outlook.com (104.47.21.53) by cas07.campus.gla.ac.uk (130.209.14.164) with Microsoft SMTP Server (TLS) id 15.0.1497.2 via Frontend Transport; Thu, 12 Dec 2019 12:42:46 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RIvCU3ueHTz0EhI0oz1dwU9tqDoN/Tzs9JqgFSQYtEZ9nCaAVh31sc00iCrqbf68O++7DHxj5XxcEcgk89YzdFxa01ZCq2LAqKiGdZuI4t0nQeuSXBzKwJWUmuYx8hmmBp9TM1xjnWyBYll4+cF0X0ieRiZd6toHw1F3J+Onrsf2mgwSNpQXdH1Cl/OMKFfCLN06rYQg4emZzVnggZ9lBSENAY8+cYZ1yTnZdCnxDVxxYsCVgd7Z2gxCdCZKpgQ3pkf1WzMa3BX/C/IKRELg71DFCK0hxAGR1/rYG35yFtwzXk5GAD3eG7Ak7Uo0r+lO8egK8Nizb74XwGr2IhPoOA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2/wAIUSDI21Bch9FSmmi9ILJlFqwHwagRoD3BBRvd7w=; b=Y9mC414TzwnqY2xPF2s80OaZz/bExnkM9z7KqYT85ne0pGHghtcdT2wiGaK4H8V4D8dNfdsva7/hklkWe0EUiQA3Daemez/7RxnjmHAUT5Nxwvpbh3sVHA1Ka7eU2QZC4PSqbaf7f8I2U/f1E6VdF+2eWUVTmqyINWFrJZN6asFxrfRqJYGkjZBOpYRueZe+vK2mENr/g3m8zCpHfdG1tPlmzUql12rNy4JykwOhmIX4BRMPzs8d6ou0Dog5JspZBlZ0s46CxKKNulWt56Yfr+PE4HWMGMT7MJXKGbppixl2RC2vI+MwCudDcmtxqKAQW7S6SbAZaRtdlYKcab84qQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=glasgow.ac.uk; dmarc=pass action=none header.from=glasgow.ac.uk; dkim=pass header.d=glasgow.ac.uk; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gla.onmicrosoft.com; s=selector2-gla-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2/wAIUSDI21Bch9FSmmi9ILJlFqwHwagRoD3BBRvd7w=; b=EP0iT6jS+1DOds9KtEGYoIAyBXryScJOjAAF0GmRTMsYc9AnELHWq+GnfMzPAP9HXcBN9PRIoSedez2gREjfEYmSYWxKI/sDmQxJj85YHTHazRJxQGp4AT4QYSHFCX0Ziu0ZA1Yqtgn2Ekiu+4hLYcpXxiEiFNnPKwd8GlpoNFg= Received: from CWXP265MB0149.GBRP265.PROD.OUTLOOK.COM (10.164.144.19) by CWXP265MB1158.GBRP265.PROD.OUTLOOK.COM (20.176.46.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2516.12; Thu, 12 Dec 2019 12:42:45 +0000 Received: from CWXP265MB0149.GBRP265.PROD.OUTLOOK.COM ([fe80::c8c6:434c:2a1a:970f]) by CWXP265MB0149.GBRP265.PROD.OUTLOOK.COM ([fe80::c8c6:434c:2a1a:970f%5]) with mapi id 15.20.2516.019; Thu, 12 Dec 2019 12:42:45 +0000 From: Norman Gray To: David Christensen CC: "freebsd-questions@freebsd.org" Subject: Re: Adding to a zpool -- different redundancies and risks Thread-Topic: Adding to a zpool -- different redundancies and risks Thread-Index: AQHVsEnhosdAKOTbGEOdhXJjA5RdA6e19JcAgAB+IQA= Date: Thu, 12 Dec 2019 12:42:45 +0000 Message-ID: <5A01F7F7-9326-47E2-BA6E-79A7D3F0889A@glasgow.ac.uk> References: <6104097C-009B-4E9C-A1D8-A2D0E5FECADF@glasgow.ac.uk> <09b11639-3303-df6b-f70c-6722caaacee7@holgerdanske.com> In-Reply-To: <09b11639-3303-df6b-f70c-6722caaacee7@holgerdanske.com> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [130.209.45.140] x-clientproxiedby: LO2P265CA0297.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:a5::21) To CWXP265MB0149.GBRP265.PROD.OUTLOOK.COM (2603:10a6:401:8::19) x-ms-exchange-messagesentrepresentingtype: 1 x-mailer: MailMate (1.13.1r5671) x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 6632e226-8cf0-4b18-06be-08d77f00c989 x-ms-traffictypediagnostic: CWXP265MB1158: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-forefront-prvs: 0249EFCB0B x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(136003)(376002)(39860400002)(396003)(366004)(346002)(189003)(199004)(5660300002)(66946007)(36756003)(786003)(316002)(2616005)(6512007)(66476007)(66556008)(64756008)(71200400001)(44832011)(66446008)(6486002)(4326008)(86362001)(2906002)(8676002)(186003)(6916009)(26005)(33656002)(81156014)(478600001)(53546011)(6506007)(966005)(8936002)(81166006)(52116002); DIR:OUT; SFP:1101; SCL:1; SRVR:CWXP265MB1158; H:CWXP265MB0149.GBRP265.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: glasgow.ac.uk does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: TGKdB+YdVS8o5G9tLy8SIkT23BgvBTmjCFT7jl1fVTYUjq7HOGQvigvYX0gGwEQx0YIHdAKXbY7o5e9T99KUWaWAwJZv/D0wBDkzwfPn5GAeYVliwY2BpBD0+TjRovRe+NbIXdDfM/SjuOvQTaEgJHCF9CCo+XlgEBRGm5/M/FQHqGPbD8UVhjNBVCWW1gkaKQpW1nnlQ911Wo1JVdIoW2kqDqJ+WuVqXrAhw4ysT6y3/JHnS1bUSky8NnpF8inR6g3yRlDzPXxcqrAAwurAHabN+EmjL3ArNUPzQZsGBupJTzo2gbZEMUWRtZzJy9NtfydGU1z/DFLK16eiBp99A2nT6bOMXdcRzni9Xr4ggPNGGEWpkG24c6mWOmYGHFMlsx1oslZjhuhcMhe/QBnAnrRVN31alQ9perYRMtIzEODXECcOadBskivY1FWNMxZe Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: 6632e226-8cf0-4b18-06be-08d77f00c989 X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Dec 2019 12:42:45.6071 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 6e725c29-763a-4f50-81f2-2e254f0133c8 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: ue/fWQI7+Klg/uxDk5Kre5zgG7v7VFdAzK5CTkyS1nH6HsYfjIeNr1RgVOzTbBaaG+0zAIW+qZYwp+k9UEa+eiOYm6K0kUo8CUZvWbPdp7s= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CWXP265MB1158 X-OriginatorOrg: glasgow.ac.uk X-Rspamd-Queue-Id: 47YYNp1JYmz4VKf X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gla.onmicrosoft.com header.s=selector2-gla-onmicrosoft-com header.b=EP0iT6jS; dmarc=none; spf=none (mx1.freebsd.org: domain of Norman.Gray@glasgow.ac.uk has no SPF policy when checking 130.209.16.75) smtp.mailfrom=Norman.Gray@glasgow.ac.uk X-Spamd-Result: default: False [-4.89 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gla.onmicrosoft.com:s=selector2-gla-onmicrosoft-com]; HAS_XOIP(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; IP_SCORE(-0.89)[ipnet: 130.209.0.0/16(-3.89), asn: 786(-0.47), country: GB(-0.08)]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[glasgow.ac.uk]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; RCVD_COUNT_FIVE(0.00)[6]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[75.16.209.130.list.dnswl.org : 127.0.11.2]; DKIM_TRACE(0.00)[gla.onmicrosoft.com:+]; RCPT_COUNT_TWO(0.00)[2]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:786, ipnet:130.209.0.0/16, country:GB]; ARC_ALLOW(-1.00)[i=1]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Dec 2019 12:42:51 -0000 David, hello. On 12 Dec 2019, at 5:11, David Christensen wrote: > Please post: > > 1 The 'zpool create ...' command you used to create the existing=20 > pool. I don't have a note of the exact command, but it would have been=20 something like zpool create pool raidz2 da{0,1,2,3,4,5,6,7,8} raidz2 da9=20 da1{0,1,2,3,4,5,6,7} > 2. The output of 'zpool status' for the existing pool. # zpool status pool pool: pool state: ONLINE status: Some supported features are not enabled on the pool. The pool=20 can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(7) for details. scan: none requested config: NAME STATE READ WRITE CKSUM pool ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 label/zd032 ONLINE 0 0 0 label/zd033 ONLINE 0 0 0 label/zd034 ONLINE 0 0 0 label/zd035 ONLINE 0 0 0 label/zd036 ONLINE 0 0 0 label/zd037 ONLINE 0 0 0 label/zd038 ONLINE 0 0 0 label/zd039 ONLINE 0 0 0 label/zd040 ONLINE 0 0 0 raidz2-1 ONLINE 0 0 0 label/zd041 ONLINE 0 0 0 label/zd042 ONLINE 0 0 0 label/zd043 ONLINE 0 0 0 label/zd044 ONLINE 0 0 0 label/zd045 ONLINE 0 0 0 label/zd046 ONLINE 0 0 0 label/zd047 ONLINE 0 0 0 label/zd048 ONLINE 0 0 0 label/zd049 ONLINE 0 0 0 errors: No known data errors # (Note: since creating the pool, I realised that gpart labels were a Good=20 Thing, hence exported, labelled, and imported the pool, hence the=20 difference from the da* pool creation). > 3. The output of 'zpool list' for the existing pool. # zpool list pool NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP =20 HEALTH ALTROOT pool 98T 75.2T 22.8T - - 29% 76% 1.00x =20 ONLINE - > 4. The 'zpool add ...' command you are contemplating. # zpool add -n pool raidz2 label/zd05{0,1,2,3,4,5} invalid vdev specification use '-f' to override the following errors: mismatched replication level: pool uses 9-way raidz and new vdev uses=20 6-way raidz The six new disks are 12TB; the 18 original ones 5.5TB. > So, you have 24 drives in a 24 drive cage? That's correct -- the maximum the chassis will take. > What are your space and performance goals? Not very explicit: TB/currency-unit as high as possible. Performance:=20 bottlenecks are likely to be elsewhere (network, processing power) so no=20 stringent requirements. Though this is a fairly general-purpose data=20 store, a large fraction of the datasets on the machine comprise a number=20 of 10GB single files, served via NFS. > What are your sustainability goals as drives and/or VDEV's fail? It doesn't have to be high availability, so if I have a drive failure, I=20 can consider shutting the machine down until a replacement disk arrives=20 and can be resilvered. This is a mirror of data where the masters are=20 elsewhere on the planet, so this machine is 'reliable storage but not=20 backed up' (and the users know this). Thus if I do decide to keep=20 running with one failed disk in one VDEV, and the worst comes to the=20 worst and the whole thing explodes... the world won't end. I will be=20 cross, and users will moan, in either case, but they know this is a=20 problem that can fundamentally be solved with more money. I'm sure I could be more sophisticated about this (and any suggestions=20 are welcome), but unfortunately I don't have as much time to spend on=20 storage problems as I'd like, so I'd like to avoid creating a setup=20 which is smarter than I'm able to fix! Best wishes, Norman --=20 Norman Gray : https://nxg.me.uk SUPA School of Physics and Astronomy, University of Glasgow, UK