From owner-freebsd-current@freebsd.org Tue Jan 12 18:32:20 2021 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A8F064E1DC3 for ; Tue, 12 Jan 2021 18:32:20 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-qk1-x72f.google.com (mail-qk1-x72f.google.com [IPv6:2607:f8b0:4864:20::72f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DFfLq2kKQz4TDn for ; Tue, 12 Jan 2021 18:32:19 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by mail-qk1-x72f.google.com with SMTP id v126so2734916qkd.11 for ; Tue, 12 Jan 2021 10:32:19 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=yDGnoC+I631kn5CmbGFUp4vEIhuwD38mGwwTaJuzZ54=; b=kAHJcPiCDVcypuNFybYp7jXV+sdf6bd+fkdnjSbWkoDwcHySiP9lNcpNRmk2lLLfO9 1ilfiM7puzaFfKa/qSEiDDZ19mlQeDEiJtaFVxBFynHSPOC72ZzC2OEmRI5OYedicNdt cTY23Tq7OBtnU5uRMOT+YeHNg/G7qqR9on5SzlAF3dYtuAA6V3Pq3jk7C1AZS2JD+dR1 gZQyhhVy1bx7v2egJwFAxk0KYighsmmdExWt1svHasx04vnpN1njReG8+ITEeDl+AIJ2 XUr9qFMvacksAcby2HqjqknJ8bIeWWBojAWzb9TG2tXLO8I76N1HDakum+weXlNqlPbI JqgA== X-Gm-Message-State: AOAM533H5fc2x6x4U45WgmgnM0cdiGY0NL73wcGIHGpy75hf3veRuZz1 ur9abWxuWfRdjPTLYuvg/fVHg7WtHq3QIg0zaKDiSiuAfUA= X-Google-Smtp-Source: ABdhPJy0pO+3gXLMS6tchGLbXU4QnRBZvp9lwxH0Vuprrk5vxl8H3T/IwPp6hAJgdNealN+qLCgTdA+n+4nVVfmKDpE= X-Received: by 2002:a37:7981:: with SMTP id u123mr688775qkc.360.1610476338392; Tue, 12 Jan 2021 10:32:18 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Freddie Cash Date: Tue, 12 Jan 2021 10:32:07 -0800 Message-ID: Subject: Re: Preparing ZFS drives To: joe mcguckin Cc: FreeBSD-Current X-Rspamd-Queue-Id: 4DFfLq2kKQz4TDn X-Spamd-Bar: --- X-Spamd-Result: default: False [-4.00 / 15.00]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::72f:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::72f:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::72f:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-current] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2021 18:32:20 -0000 On Tue, Jan 12, 2021 at 10:10 AM joe mcguckin wrote: > Folks, > > I want to buy some 16TB drives and raid them up > > How should I label and prepare the drives for ZFS? Someone ought to writ= e > a =E2=80=98cookbook=E2=80=99 on that! > If these drives will be strictly data drives (not booting from them), partitioning them is fairly easy. You will want to determine a labelling system for them. Personally, I like to label the drives using a grid system (columns are letters, rows are numbers). For systems with multiple JBODs attached, I include which JBOD chassis their in as well. For example, a 24-bay chassis would use disk-a1, disk-a2, disk-a3 ... disk-d4, disk-d5, disk-d6. A system with 2 24-bay JBODs would use jbod1-a1, jbod1-a2, jbod1-a3 ... jbod2-d4, jbod2-d5, jbod2-d6. So you label the GPT partition on each disk, and build the pool using the GPT partition labels. gpart create -s gpt da0 gpart add -t freebsd-zfs -a 1M -l disk-a1 da0 gpart create -s gpt da1 gpart add -t freebsd-zfs -a 1M -l disk-a2 da1 And so on. Add 1 disk, partition/label it based on its location. Then add the next disk. And so on. Then use the GPT labels to create the pool (they show up as devices under the /dev/gpt/ directory): zpool create mypool mirror gpt/disk-a1 gpt/disk-a2 mirror gpt/disk-a3 gpt/disk-a4 mirror gpt/disk-a5 gpt/disk-a6 If you need to boot from these drives (make a root pool), then things get more complicated. Personally, I'd recommend using the 16 TB drives strictly for a data pool, and then use some smaller SSDs for a root pool, in a simple mirror vdev setup. Separate the OS from the data. :) Do I need to start the volume on a particular sector boundary? > The "-a 1M" argument for gpart does it for you. It aligns the partition at the 1 MB boundary, and figures out which sector of the disk that corresponds to based on the sector size of the disk (512B or 4K). > Are the 4096 byte sector drives usable? > Yeah, they work without issues. Try not to mix 512B and 4K drives within a single vdev (it'll work, but may cause performance issues). Mixing them in a pool (a vdev using 512B drives, another vdev using 4K drives) is okay, so long as you set the vfs.zfs.min_auto_ashift sysctl to 12 (force the minimum block size used by ZFS to be 4K). That way, in the future, you can replace the 512B drives with 4K drives without any performance issues. You can check what the ashift value is for each of the vdevs using: zdb | grep -B5 ashift If it shows ashift=3D9 anywhere, then destroy the pool, change the sysctl value, and recreate the pool. Check to make sure it shows ashift=3D12 in z= db output. --=20 Freddie Cash fjwcash@gmail.com