From owner-freebsd-questions@freebsd.org Fri Sep 17 06:51:48 2021 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 58F476AF055 for ; Fri, 17 Sep 2021 06:51:48 +0000 (UTC) (envelope-from bennett@sdf.org) Received: from mx.sdf.org (mx.sdf.org [205.166.94.24]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mx.sdf.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4H9l432DBcz3DPB for ; Fri, 17 Sep 2021 06:51:47 +0000 (UTC) (envelope-from bennett@sdf.org) Received: from sdf.org (IDENT:bennett@faeroes.freeshell.org [205.166.94.9]) by mx.sdf.org (8.15.2/8.14.5) with ESMTPS id 18H6pY2h007836 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256 bits) verified NO); Fri, 17 Sep 2021 06:51:35 GMT Received: (from bennett@localhost) by sdf.org (8.15.2/8.12.8/Submit) id 18H6p98b023123; Fri, 17 Sep 2021 01:51:09 -0500 (CDT) From: Scott Bennett Message-Id: <202109170651.18H6p98b023123@sdf.org> Date: Fri, 17 Sep 2021 01:51:09 -0500 To: Chris Ross Subject: Re: Device names in ZFS pool Cc: freebsd-questions@freebsd.org User-Agent: Heirloom mailx 12.5 6/20/10 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4H9l432DBcz3DPB X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.80 / 15.00]; FAKE_REPLY(1.00)[]; ARC_NA(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:205.166.94.0/24]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_RCPT(0.00)[freebsd]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[sdf.org,quarantine]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:14361, ipnet:205.166.94.0/24, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-questions] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Sep 2021 06:51:48 -0000 On Thu, 9 Sep 2021 16:48:28 -0400 Chris Ross wrote: >Hey there. So, I created a ZFS pool, a RAID-Z with ?da5 da2 da3?. These are then the device names that appeared in ?zpool status mypool?. However, da3 had errors, and I replaced it by another disk living at da1. Then, I removed the bad disk and moved things in the enclosure. But, after this, some ids seem to have changed, and what was originally ?da5? is now ?da4? and ?zpool status mypool? shows: > FreeBSD normally assigns da device numbers in the order that the devices are detected and identified by the kernel. ada device enumeration appears to happen in the order encountered by the bus-controller-port hardware addresses of the devices. da devices are today most commonly devices attached by USB. Because you gave entire drives to ZFS, your label options are limited to /dev/da?, /dev/diskid/*, and maybe /dev/gptid/*. For the latter, add the following line to /boot/loader.conf. If you had used gpart(8) to partition the devices with GPT layouts, then you would have had the option of using gpt labels as well, which would have appeared in /dev/gpt. For my largest pool, for which I use the first partition on each of six drives (it's a raidz2), I use the partition label to identify the device by its manufacturer and serial number, which makes identifying a problem device both trivial and exact. I don't even need to apply an exterior printed label because the manufacturer has already done so. kern.geom.label.gptid.enable="1" Because the first of those is subject to frequent change upon reboot or detaching and reattaching the device in question, one of the latter is preferable in many situations and especially those involving ZFS. However, it is worth noting that a diskid will change anytime you replace a device in ZFS. In any case, whenever is device in a ZFS pool is offline or faulted, a zpool status command will display that device as a many-digit integer and as, for example, "(was /dev/da5)". In that situation, trying to do, say, a zpool replace mypool da5 or a zpool replace mypool da5 da8 may fail because da5 is not available. It will fail when the device that was on /dev/da5 has been enumerated as, say, da6 on the latest reboot. Similarly, a zpool replace mypool da8 will fail because da8 is/was not part of the pool and is not the device that ZFS is looking for. However, if the number shown in the status output is 112574468553112, then a zpool replace mypool 112574468553112 da8 is what you need. I mostly let ZFS do whatever it wants w.r.t. device names and then just do my best to accommodate it because fighting it rarely succeeds. :-} That said, I would note that many folks prefer to refer to a device when creating a pool or adding/attaching/removing a device by its gptid. I haven't yet done this, so I can't say for sure that this will work for you, but it may cause ZFS to prefer the gptid when recognizing devices while activating or importing a pool, in which case you will have a consistent way to identify each devices in the pool. Many have said, OTOH, that reference by diskid allows them to identify a failed or failing physical device more easily because they can paste a matching label onto the drive. > NAME STATE READ WRITE CKSUM > mypool ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > diskid/DISK-Z1Z946N90000R546NBZH ONLINE 0 0 0 You can, of course, refer to /dev/diskid devices if you like. I don't like having them around, mainly because ZFS seems to refer to devices with multiple names without any consistency of order. If you don't like them being around, you can disable their generation by placing the following line into /boot/loader.conf. kern.geom.label.disk_ident.enable="0" My largest pool only has three drives that are connected (usually) by USB externally, and its other three are on ada devices, whose addresses do not change, so I haven't really had many problems in identifying the drive(s) of interest. I also disable the generation of gptid device names for the same reasons. > da2 ONLINE 0 0 0 > da1 ONLINE 0 0 0 > >If I use zdb, I can see that value stored in the ?path? of each child of the vdev/pool. But, can I _change_ it? I know this is the reason many people suggest not using device names like I did, but, if I switch to gpt labels or the like, I?ll still need to update these in this existing vdev, right? Please be aware that you can only switch to gpt labels by partitioning the drives as GPT-partitioned drives with at least one partition on each that you have labeled. (See gpart(8)'s add and modify subcommands.) And you will only be able to do that in your case by destroying the pool first, partitioning the drive, and then recreating the pool because you have already given the *entire* drive to ZFS. You have left no space for the partition tables, and you cannot reduce the size of a device node already given to ZFS. > >How can I change these path labels? 1. You can reboot, and then see how the device names are assigned that time. 2. You can export the pool and then import it with, for example, zpool import -d /dev/diskid mypool or zpool import -d /dev/gptid mypool to get ZFS to look for ZFS labels on those devices instead of starting its search in /dev. However, if your pool is small, which it presumably is because it is a raidz1 pool, why worry about which device names ZFS uses? You should have little trouble figuring out what it is doing or how to deal with a problem. OTOH, when your pool comprises a dozen or more devices, then it may well be worth using diskid or gptid device names and placing printed or handwritten labels onto the drives. Scott Bennett, Comm. ASMELG, CFIAG ********************************************************************** * Internet: bennett at sdf.org *xor* bennett at freeshell.org * *--------------------------------------------------------------------* * "A well regulated and disciplined militia, is at all times a good * * objection to the introduction of that bane of all free governments * * -- a standing army." * * -- Gov. John Hancock, New York Journal, 28 January 1790 * **********************************************************************