From owner-freebsd-current@freebsd.org Tue Jan 12 18:40:43 2021 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 81DFB4E2353 for ; Tue, 12 Jan 2021 18:40:43 +0000 (UTC) (envelope-from joe@via.net) Received: from smtp2.via.net (smtp2.via.net [157.22.3.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "via.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DFfXV3GqTz4V2k for ; Tue, 12 Jan 2021 18:40:42 +0000 (UTC) (envelope-from joe@via.net) Received: from mail.via.net (mail.via.net [157.22.3.34]) by smtp2.via.net (8.15.2/8.14.1-VIANET) with ESMTPS id 10CIedCM010873 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Tue, 12 Jan 2021 10:40:39 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.97.3 at smtp2.via.net Received: from [209.81.2.65] ([209.81.2.65]) by mail.via.net (8.15.2/8.14.1-VIANET) with ESMTP id 10CIedZJ012656; Tue, 12 Jan 2021 10:40:39 -0800 (PST) From: joe mcguckin Message-Id: Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\)) Subject: Re: Preparing ZFS drives Date: Tue, 12 Jan 2021 10:40:28 -0800 In-Reply-To: Cc: FreeBSD-Current To: Freddie Cash References: X-Mailer: Apple Mail (2.3608.120.23.2.4) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (smtp2.via.net [157.22.3.6]); Tue, 12 Jan 2021 10:40:40 -0800 (PST) X-Rspamd-Queue-Id: 4DFfXV3GqTz4V2k X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.85 / 15.00]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[157.22.3.6:from]; FREEFALL_USER(0.00)[joe]; FROM_HAS_DN(0.00)[]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+MX:c]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[via.net]; NEURAL_HAM_LONG(-1.00)[-1.000]; MID_RHS_MATCH_FROM(0.00)[]; SPAMHAUS_ZRD(0.00)[157.22.3.6:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-0.95)[-0.954]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:7091, ipnet:157.22.0.0/16, country:US]; RCVD_TLS_LAST(0.00)[]; MAILMAN_DEST(0.00)[freebsd-current]; RCVD_IN_DNSWL_LOW(-0.10)[157.22.3.6:from] Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2021 18:40:43 -0000 Freddie, Thanks! I have to say that this is the most cogent explanation on how to = prepare ZFS drives that=20 anyone has ever given me. Thanks again! Joe Joe McGuckin ViaNet Communications joe@via.net 650-207-0372 cell 650-213-1302 office 650-969-2124 fax > On Jan 12, 2021, at 10:32 AM, Freddie Cash wrote: >=20 > On Tue, Jan 12, 2021 at 10:10 AM joe mcguckin > wrote: > Folks, >=20 > I want to buy some 16TB drives and raid them up >=20 > How should I label and prepare the drives for ZFS? Someone ought to = write a =E2=80=98cookbook=E2=80=99 on that! >=20 > If these drives will be strictly data drives (not booting from them), = partitioning them is fairly easy. You will want to determine a = labelling system for them. Personally, I like to label the drives using = a grid system (columns are letters, rows are numbers). For systems with = multiple JBODs attached, I include which JBOD chassis their in as well. = For example, a 24-bay chassis would use disk-a1, disk-a2, disk-a3 ... = disk-d4, disk-d5, disk-d6. A system with 2 24-bay JBODs would use = jbod1-a1, jbod1-a2, jbod1-a3 ... jbod2-d4, jbod2-d5, jbod2-d6. So you = label the GPT partition on each disk, and build the pool using the GPT = partition labels. >=20 > gpart create -s gpt da0 > gpart add -t freebsd-zfs -a 1M -l disk-a1 da0 >=20 > gpart create -s gpt da1 > gpart add -t freebsd-zfs -a 1M -l disk-a2 da1 >=20 > And so on. Add 1 disk, partition/label it based on its location. = Then add the next disk. And so on. >=20 > Then use the GPT labels to create the pool (they show up as devices = under the /dev/gpt/ directory): >=20 > zpool create mypool mirror gpt/disk-a1 gpt/disk-a2 mirror gpt/disk-a3 = gpt/disk-a4 mirror gpt/disk-a5 gpt/disk-a6 >=20 >=20 > If you need to boot from these drives (make a root pool), then things = get more complicated. Personally, I'd recommend using the 16 TB drives = strictly for a data pool, and then use some smaller SSDs for a root = pool, in a simple mirror vdev setup. Separate the OS from the data. :) >=20 > Do I need to start the volume on a particular sector boundary? >=20 > The "-a 1M" argument for gpart does it for you. It aligns the = partition at the 1 MB boundary, and figures out which sector of the disk = that corresponds to based on the sector size of the disk (512B or 4K). > =20 > Are the 4096 byte sector drives usable? >=20 > Yeah, they work without issues. Try not to mix 512B and 4K drives = within a single vdev (it'll work, but may cause performance issues). = Mixing them in a pool (a vdev using 512B drives, another vdev using 4K = drives) is okay, so long as you set the vfs.zfs.min_auto_ashift sysctl = to 12 (force the minimum block size used by ZFS to be 4K). That way, in = the future, you can replace the 512B drives with 4K drives without any = performance issues. >=20 > You can check what the ashift value is for each of the vdevs using: = zdb | grep -B5 ashift >=20 > If it shows ashift=3D9 anywhere, then destroy the pool, change the = sysctl value, and recreate the pool. Check to make sure it shows = ashift=3D12 in zdb output. >=20 > --=20 > Freddie Cash > fjwcash@gmail.com