From owner-freebsd-questions@freebsd.org Wed Dec 11 17:39:08 2019 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2E65A1DC188 for ; Wed, 11 Dec 2019 17:39:08 +0000 (UTC) (envelope-from Norman.Gray@glasgow.ac.uk) Received: from plockton.cent.gla.ac.uk (plockton.cent.gla.ac.uk [130.209.16.75]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47Y4170fRDz4PNq for ; Wed, 11 Dec 2019 17:39:06 +0000 (UTC) (envelope-from Norman.Gray@glasgow.ac.uk) Received: from cas07.campus.gla.ac.uk ([130.209.14.164]) by plockton.cent.gla.ac.uk with esmtp (Exim 4.72) (envelope-from ) id 1if5x2-00010A-QS for freebsd-questions@freebsd.org; Wed, 11 Dec 2019 17:39:04 +0000 Received: from CAS08.campus.gla.ac.uk (130.209.14.165) by cas07.campus.gla.ac.uk (130.209.14.164) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 11 Dec 2019 17:39:04 +0000 Received: from GBR01-LO2-obe.outbound.protection.outlook.com (104.47.21.51) by CAS08.campus.gla.ac.uk (130.209.14.165) with Microsoft SMTP Server (TLS) id 15.0.1497.2 via Frontend Transport; Wed, 11 Dec 2019 17:39:04 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Xex56RdGtwfldii6aVqqZo3IYWGa/H39O7H88MYhTMgFjYDH/HC1naU2VRYN88I/IasoAtUKd5BGR1gBHQK5+HRAGdu9jpf9Qbb/Ee37xKDBQ5j+pmna7t/r4yQNZHUwOEGGBy4tPhav880RUVoOfmAlNFkxrOkQsIuR8PYkh9UQFnn68eKo0ivkJLWuRSBntWlWuISW+UreDFR+GBKpx3dPj9qTXEnwImpTX+smX0yb6ynnbZjvkLl/Z1KOCjRhXQTsOrJzEO7t38dd2oSrROs3yJitmEKQoQY8vhP9CNoQoQmf623UVEGKOPJF8AeWOvGlw7XzEZUZMeO3JWDwcg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=g6mohqefApfwaYoIj7D6IH8PLECMbgw5u7MnfJy2n/c=; b=oT1ZbqQsM+HeoWEyHZsyQWI2lQoDrY7oRtDWDUhYdW5NhebiKBckdo7OZdQLq/7EYtv26IwAdh1/z4IbJEBig9MgWCHumTfAKQA28mpfdRQLjqoXbOItaaUIap7a7zL5w7WNoagAffRkWIH9Gt72uB2QGCjSo4xEk/m1lR23yT2cSrFSRYNTdj0RLeCUg3WodvcpD385euoqgQ2/GBbIzsH7Aa40SKiZKeP8dXv9Ufq2PO5MoEmZU7yzEXLxvS+HKIdq1KoWTyCgkW9hNqxr1H4EJkTm8psxkJvgmUkvRIqN9//CYCiZYNsARzo7l4TsSIzkDtaGx7ekEpPrF7b6mQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=glasgow.ac.uk; dmarc=pass action=none header.from=glasgow.ac.uk; dkim=pass header.d=glasgow.ac.uk; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gla.onmicrosoft.com; s=selector2-gla-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=g6mohqefApfwaYoIj7D6IH8PLECMbgw5u7MnfJy2n/c=; b=Kw8zNU9g2w5jQHYIwdmliKQu3V1TqcGXsOvk0RYrENeyCEVA3Dw+9OepaDzRyakqOxUg0d2u2RxSPx3GCdJl7cniAHViqsYNc+hlyTNoIrk4y/yHzaGLS0ZJw4wD9DHMhPinADnCNIr3FxuLJl88r/Wj0w4474YV+v7HEIQEpks= Received: from CWXP265MB0149.GBRP265.PROD.OUTLOOK.COM (10.164.144.19) by CWXP265MB0070.GBRP265.PROD.OUTLOOK.COM (10.164.145.143) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2516.18; Wed, 11 Dec 2019 17:39:03 +0000 Received: from CWXP265MB0149.GBRP265.PROD.OUTLOOK.COM ([fe80::c8c6:434c:2a1a:970f]) by CWXP265MB0149.GBRP265.PROD.OUTLOOK.COM ([fe80::c8c6:434c:2a1a:970f%5]) with mapi id 15.20.2516.018; Wed, 11 Dec 2019 17:39:03 +0000 From: Norman Gray To: FreeBSD questions list Subject: Adding to a zpool -- different redundancies and risks Thread-Topic: Adding to a zpool -- different redundancies and risks Thread-Index: AQHVsEnhosdAKOTbGEOdhXJjA5RdAw== Date: Wed, 11 Dec 2019 17:39:03 +0000 Message-ID: <6104097C-009B-4E9C-A1D8-A2D0E5FECADF@glasgow.ac.uk> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [130.209.45.140] x-clientproxiedby: LO2P123CA0028.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600::16) To CWXP265MB0149.GBRP265.PROD.OUTLOOK.COM (2603:10a6:401:8::19) x-ms-exchange-messagesentrepresentingtype: 1 x-mailer: MailMate (1.13.1r5671) x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 95b99146-3f72-4b78-2c28-08d77e6103af x-ms-traffictypediagnostic: CWXP265MB0070: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-forefront-prvs: 024847EE92 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(366004)(346002)(396003)(39860400002)(376002)(136003)(189003)(199004)(81166006)(966005)(6506007)(8676002)(8936002)(186003)(81156014)(6512007)(33656002)(5660300002)(66446008)(86362001)(16799955002)(478600001)(2616005)(6486002)(44832011)(6916009)(66556008)(26005)(64756008)(52116002)(36756003)(786003)(316002)(66476007)(2906002)(71200400001)(66946007); DIR:OUT; SFP:1101; SCL:1; SRVR:CWXP265MB0070; H:CWXP265MB0149.GBRP265.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: glasgow.ac.uk does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Q1/XH7ZXlCAJxJBOYViEDW31yvZ3Ce3F7ipsDyxm9PNgxJzFI1RJ1m2Txp/XWcKsWKvRBSeazq14svEtonaa73Jk4lmdcBWcGHY8vFHOBC+wqKU/HuGHKH/MpAk5zVfPuIreRHc8MenWh1e8NukGstU8hj01pQuT99K/pLfHhS0mT+it/Lly2ju4AMdChOX44174jzLp6ow196hI30fDuhawAQfwbolMUkha6z04EIAgmDP4YjL2cYkL6rY258RSlDw8SObqzEGu/253v8BTe14t4GfyK6FzkQYbtIZnxaVTOFH8oG+hYdS0Ko4HwR0YA/ktptll+61OfvAFp/pE43cci62OOIky9W6MoeOUpFU6BaLhTNTJeUHRQ8lSLlgPZI91IVpM7btH0GoVN9yrXIi6wmJzmtPfXbqKUkcnIGFLSe1JhmSAmaLW66vU3BAdS88PIJuDYM9OtCZxPIN4+PMHMS55p2nbM7dzutrWlLs= Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: 95b99146-3f72-4b78-2c28-08d77e6103af X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Dec 2019 17:39:03.6572 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 6e725c29-763a-4f50-81f2-2e254f0133c8 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: wsDzAyDoPpsJ7floLXZyzlQhMeG+lvnV/tAFbE+c5L3dgOLztDtbSTZ9+pkrFiFpbp4oL48Avuqvi+OEu1xv4XqXkCszr9d6zosUGEhCb6Q= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CWXP265MB0070 X-OriginatorOrg: glasgow.ac.uk X-Rspamd-Queue-Id: 47Y4170fRDz4PNq X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gla.onmicrosoft.com header.s=selector2-gla-onmicrosoft-com header.b=Kw8zNU9g; dmarc=none; spf=none (mx1.freebsd.org: domain of Norman.Gray@glasgow.ac.uk has no SPF policy when checking 130.209.16.75) smtp.mailfrom=Norman.Gray@glasgow.ac.uk X-Spamd-Result: default: False [-4.83 / 15.00]; IP_SCORE(-0.83)[ipnet: 130.209.0.0/16(-3.59), asn: 786(-0.47), country: GB(-0.08)]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gla.onmicrosoft.com:s=selector2-gla-onmicrosoft-com]; HAS_XOIP(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_COUNT_FIVE(0.00)[6]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[glasgow.ac.uk]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[75.16.209.130.list.dnswl.org : 127.0.11.2]; DKIM_TRACE(0.00)[gla.onmicrosoft.com:+]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:786, ipnet:130.209.0.0/16, country:GB]; ARC_ALLOW(-1.00)[i=1]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Dec 2019 17:39:08 -0000 Greetings. I'd like to add a new VDEV to a pool, and I'm being warned (slightly to=20 my surprise) that there's a 'mismatched replication level'. I'm trying=20 to get a sense of how much of a risk I'd be running by forcing this with=20 -f. Context: * I currently have two raidz2 VDEVs composed of nine 5.5TB disks=20 (thus 2 x ~40TB available) * I'd like to add another raidz2 VDEV composed of six 12TB disks=20 (thus adding a VDEV of ~48TB, roughly the same size as the other two) --=20 this is what prompts the warning about replication level * The storage is a local mirror which it would be very annoying to=20 lose, but it's not holding unique copies of anything * I don't like using -f options unless I'm pretty damn confident I=20 know what's happening (I wouldn't set this up in quite this way from scratch, but this is an=20 old-ish server, and a small budget windfall has allowed me to max out=20 the remaining available slots with new disks). I can appreciate that the ideal planned setup would, in principle, be to=20 have all the VDEVs be symmetrical, in terms of size and number of disks.=20 Is a VDEV mix merely 'not ideal', or 'not great but you'll be fine',=20 or Bad? My mental model of what's going on suggests that, since the pool simply=20 stripes across the VDEVs, it doesn't have to care how the VDEVs=20 themselves are structured, so that a 9x5.5 raidz2 and a 6x12 raidz2=20 would be roughly equally used, and I can't see why there would be a=20 performance or a utilisation difference between the two (but I still=20 count myself as a ZFS tyro). I can see that there would be a reliability issue if the various VDEVs=20 were different sizes of mirrored ones -- this would create different=20 amounts of resilience, and so the warning makes sense in a 'are you=20 sure?' way. If the VDEVs were different sizes and the pool was=20 mirroring over them, then there would obviously be a utilisation issue. Though both of [1] and [2] illustrate only mixing VDEVs of the same=20 type, [2] says merely that 'When using RAIDZ vdevs, it is also a good=20 idea to keep them at the same width and of the same type.' and=20 illustrates a 4-wide plus 8-wide raidz2 as 'not horrible'. The forum=20 post at [3] asks essentially the same question as this email, but=20 receives a rather oblique answer. The question at [4] gets a confident=20 answer which I don't _think_ makes complete sense. Since [1] and [2] are both more authoritative and match my own=20 understanding, I'm inclined to believe that adding this new VDEV would=20 be less than perfect, but reasonable. Am I deceiving myself? Thanks for any advice you can offer, Norman [1]=20 https://forums.freenas.org/index.php?threads/slideshow-explaining-vdev-zpoo= l-zil-and-l2arc-for-noobs.7775/ [2]=20 https://www.ixsystems.com/community/resources/introduction-to-zfs.111/ [3]=20 https://forums.freebsd.org/threads/zfs-mismatched-repli-levels.28226/ [4] https://serverfault.com/questions/522782/zfs-with-unsymmetric-vdevs --=20 Norman Gray : https://nxg.me.uk SUPA School of Physics and Astronomy, University of Glasgow, UK