From owner-freebsd-questions@freebsd.org Tue May 14 19:24:32 2019 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 042AB15964E1 for ; Tue, 14 May 2019 19:24:32 +0000 (UTC) (envelope-from hartzell@alerce.com) Received: from griffon.alerce.com (griffon.alerce.com [206.125.171.162]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "griffon.alerce.com", Issuer "griffon.alerce.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 94B2C80FC9; Tue, 14 May 2019 19:24:30 +0000 (UTC) (envelope-from hartzell@alerce.com) Received: from griffon.alerce.com (localhost [127.0.0.1]) by griffon.alerce.com (Postfix) with ESMTP id ADE622842E; Tue, 14 May 2019 12:24:28 -0700 (PDT) Received: from postfix.alerce.com (76-226-160-236.lightspeed.sntcca.sbcglobal.net [76.226.160.236]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by griffon.alerce.com (Postfix) with ESMTPSA id 8BEA92842D; Tue, 14 May 2019 12:24:28 -0700 (PDT) Received: by postfix.alerce.com (Postfix, from userid 501) id 2DBD9200E05A54; Tue, 14 May 2019 12:24:28 -0700 (PDT) From: George Hartzell MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <23771.5612.105696.170743@alice.local> Date: Tue, 14 May 2019 12:24:28 -0700 To: Polytropon Cc: hartzell@alerce.com, Matthew Seaman , freebsd-questions@freebsd.org Subject: Re: Suggestions for working with unstable nvme dev names in AWS In-Reply-To: <20190514210203.3d951fb8.freebsd@edvax.de> References: <23770.10599.687213.86492@alice.local> <08660a2a-489f-8172-22ee-47aeba315986@FreeBSD.org> <23770.58821.826610.399467@alice.local> <20190514210203.3d951fb8.freebsd@edvax.de> X-Mailer: VM 8.2.0b under 26.1 (x86_64-apple-darwin14.5.0) Reply-To: hartzell@alerce.com X-Virus-Scanned: ClamAV using ClamSMTP X-Rspamd-Queue-Id: 94B2C80FC9 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of hartzell@alerce.com designates 206.125.171.162 as permitted sender) smtp.mailfrom=hartzell@alerce.com X-Spamd-Result: default: False [-2.24 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_REPLYTO(0.00)[hartzell@alerce.com]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-0.99)[-0.993,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[alerce.com]; REPLYTO_ADDR_EQ_FROM(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: griffon.alerce.com]; NEURAL_HAM_SHORT(-0.13)[-0.132,0]; NEURAL_HAM_MEDIUM(-0.77)[-0.771,0]; IP_SCORE(-0.03)[asn: 25795(-0.10), country: US(-0.06)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:25795, ipnet:206.125.168.0/21, country:US]; RCVD_TLS_LAST(0.00)[] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 May 2019 19:24:32 -0000 Polytropon writes: > On Tue, 14 May 2019 08:59:01 -0700, George Hartzell wrote: > > Matthew Seaman writes: > > > [...] but if you > > > are using ZFS, then shuffling the disks around should not make any > > > difference. > > > [...] > > Yes, once I have them set up (ZFS or labeled), it doesn't matter what > > device names they end up having. For now I just do the setup by hand, > > poking around a bit. Same trick in the Linux world, you end up > > referring to them by their UUID or .... > > In addition to what Matthew suggested, you could use UFS-IDs > in case the disks are initialized with UFS. You can find more > information here (at the bottom of the page): > [...] Yes. As I mentioned in my response to Matthew, once I have some sort of filesystem/zpool on the device, it's straightforward (TMTOWTDI). The problem is being able to provision the system automatically without user intervention. In the Linux world, I can use e.g. Terraform to set up a pair of additional volumes and tell it to call them `/dev/sdy` and `/dev/sdz`. The Linux magic happens and I get pair of symlinks that I can use in my e.g. Ansible playbooks, that point to whatever the devices came up as when it booted. I build filesystems on the devices, add them via their UUID's to `/etc/fstab` and I'm off and running. I can't [seem to] do this in the FreeBSD world; even if I name the devices `/dev/nvme1` (the fast and big one) and `/dev/nvme2` (the slow and small one), there's no guarantee that they'll have those names when the machine boots. This is a weirdly AWS issue and their peace offering is to stash the requested device name in the device/controller/"hardware" and provide a tool that digs it out. I'm trying to figure out what I can do about it from FreeBSD. Perhaps there's already a solution. Perhaps the nvme driver needs to be extended to provide access to the magic AWS info stash and then something like Amazon Linux's `ebsnvme-id` can pry it out. g.