From owner-freebsd-questions@freebsd.org Tue May 14 02:35:29 2019 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AF04A15A228E for ; Tue, 14 May 2019 02:35:29 +0000 (UTC) (envelope-from hartzell@alerce.com) Received: from griffon.alerce.com (griffon.alerce.com [206.125.171.162]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "griffon.alerce.com", Issuer "griffon.alerce.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 71F16803BF for ; Tue, 14 May 2019 02:35:28 +0000 (UTC) (envelope-from hartzell@alerce.com) Received: from griffon.alerce.com (localhost [127.0.0.1]) by griffon.alerce.com (Postfix) with ESMTP id 883862842E for ; Mon, 13 May 2019 19:35:20 -0700 (PDT) Received: from postfix.alerce.com (76-226-160-236.lightspeed.sntcca.sbcglobal.net [76.226.160.236]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by griffon.alerce.com (Postfix) with ESMTPSA id 614472842D for ; Mon, 13 May 2019 19:35:20 -0700 (PDT) Received: by postfix.alerce.com (Postfix, from userid 501) id BB42B200E02483; Mon, 13 May 2019 19:35:19 -0700 (PDT) From: George Hartzell MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <23770.10599.687213.86492@alice.local> Date: Mon, 13 May 2019 19:35:19 -0700 To: freebsd-questions@freebsd.org Subject: Suggestions for working with unstable nvme dev names in AWS X-Mailer: VM 8.2.0b under 26.1 (x86_64-apple-darwin14.5.0) Reply-To: hartzell@alerce.com X-Virus-Scanned: ClamAV using ClamSMTP X-Rspamd-Queue-Id: 71F16803BF X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of hartzell@alerce.com designates 206.125.171.162 as permitted sender) smtp.mailfrom=hartzell@alerce.com X-Spamd-Result: default: False [-3.10 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_REPLYTO(0.00)[hartzell@alerce.com]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-0.997,0]; RCVD_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[alerce.com]; MX_GOOD(-0.01)[griffon.alerce.com]; NEURAL_HAM_SHORT(-0.78)[-0.780,0]; REPLYTO_ADDR_EQ_FROM(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.985,0]; IP_SCORE(-0.03)[asn: 25795(-0.08), country: US(-0.06)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:25795, ipnet:206.125.168.0/21, country:US]; RCVD_TLS_LAST(0.00)[] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 May 2019 02:35:30 -0000 Newer AWS instances use nvme disk devices. On Linux these devices do not have stable names, the volume that was nvme1 on one boot might be nvme3 the next time you boot. There are a variety of tricks that pry the originally requested device name (e.g. /dev/sdh) out of the device and e.g. use udev rules to make symbolic links between that name and the unstable name. Details are here [1] in the AWS docs. The name of the device that was requested (e.g. /dev/sdh) is stashed in device (controller?) and accessible via a vendor-specific extension, > The device name is available through the NVMe controller > vendor-specific extension (bytes 384:4095 of the controller > identification): Amazon provides a python script in their Linux (here's a googled-up copy [2]) that grubs around and finds it for you. On other Linux's you can roll your own starting from `nvme id-ctrl`. I'm working with the FreeBSD 12 ZFS AMI's and assume that they'll have the same issues with unstable naming that Linux has. After a handful of tries, I haven't yet seen things change order, but the negative result isn't very comforting. I don't see the originally requested name in the output of `nvmecontrol identify /dev/nvme1`, or `nvmecontrol identify -x -v /dev/nvme1` (or `/dev/nvme1ns1`). On a flier, I tried the Linux script [2], but it fails with `[Errno 25] Inappropriate ioctl for device`. I can see a few points of alignment between the Linux script and the FreeBSD nvme driver, e.g. the `NVME_OPC_IDENTIFY` opcode value from the `nvme_admin_opcode` enum seems to match the value of `NVME_ADMIN_IDENTIFY` on line 29 of [2] but that's as far as I've unraveled it. Can anyone speak to the current state of device names for nvme disks on AWS using the FreeBSD 12 AMI's? Is name-stability an issue? If so, is there a work-around? Thanks! g. [1]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nvme-ebs-volumes.html [2]: https://gist.github.com/lbernail/d851e5b06eb32180a4b8ead2ce4f45db [3]: https://github.com/freebsd/freebsd/blob/master/sys/dev/nvme/nvme.h#L616