From nobody Tue May 6 17:31:23 2025 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZsQRr4Dcmz5v8Kw for ; Tue, 06 May 2025 17:31:36 +0000 (UTC) (envelope-from spork@bway.net) Received: from smtp1.bway.net (smtp1.bway.net [216.220.96.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZsQRr2dW2z3qgS for ; Tue, 06 May 2025 17:31:36 +0000 (UTC) (envelope-from spork@bway.net) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (pool-96-225-76-85.nwrknj.fios.verizon.net [96.225.76.85]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: spork@bway.net) by smtp1.bway.net (Postfix) with ESMTPSA id 1B51C13A26; Tue, 6 May 2025 13:31:34 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=bway.net; s=mail; t=1746552694; bh=F2MzAh8QutzYI+/wH2XzipsX7yCVH7DF6HMFb7NL80c=; h=Subject:From:In-Reply-To:Date:Cc:References:To; b=ZB7vA+PnYbMfMF9FG38abBgR8uOzmzxjbX25B0Dv4ZWMJuQ6K3s/7dY00tLK6NYWW 5E2SFVf9sRJq5/k/SgEzJ4XXeOjXJVKlbE015xxqN8jCwr0hJ2gzQBPgO1fdh79kQ/ tgXRlJDeNplsrpreN4RLy8iQTf+IDkjS6IdhejkY= Content-Type: text/plain; charset=utf-8 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.400.131.1.6\)) Subject: Re: NVMe (U.2) hot-swap support status? From: Charles Sprickman In-Reply-To: <01000196a3fb8737-9e591147-f4fd-439d-85bc-5f155be4bebb-000000@email.amazonses.com> Date: Tue, 6 May 2025 13:31:23 -0400 Cc: freebsd-stable@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <5530E3D8-B2F8-4101-91B1-D8BAC17D2FDA@bway.net> References: <2E8A7BC6-6C3F-431F-B6F0-2611577B8028@bway.net> <01000196a3fb8737-9e591147-f4fd-439d-85bc-5f155be4bebb-000000@email.amazonses.com> To: Colin Percival X-Mailer: Apple Mail (2.3826.400.131.1.6) X-Rspamd-Queue-Id: 4ZsQRr2dW2z3qgS X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8059, ipnet:216.220.96.0/19, country:US] X-Spamd-Bar: ---- > On May 6, 2025, at 1:04=E2=80=AFAM, Colin Percival = wrote: >=20 > On 5/5/25 21:54, Charles Sprickman wrote: >> Anyhow, we're trying purchase a few servers (likely this Supermicro: = https://www.supermicro.com/en/products/system/UP/1U/SYS-111E-WR) and = since manufacturers don't seem to validate for FreeBSD these days, I was = hoping for some community input. >> Last time I looked into this NVMe (specifically the "U.2" format for = servers that includes hot swap) was supported in FreeBSD but hot swap = was kind of iffy. I don't think too many FreeBSD users/developers at = that time had the hardware, most people seemed to just be using an = on-board NVMe drive on desktop or laptop systems, and there's not any = real call for hot swap in that segment. >> I have a thread on the forums linked below, but really what I'm = looking to find is some larger orgs that are running NVMe drives and can = confirm hot swap works as expected, if there's any gotchas to be on the = lookout for, etc. Trying not to overcomplicate it, but we don't want to = be stuck with a bunch of servers that might be less stable than our = current SAS/SATA servers. We exclusively use ZFS, if that's useful info. >=20 > I can't speak to *physical* hotplug, but I've put a lot of energy over = the > past few months into making sure that nvme "hotplug" is 100% = functional in > Amazon EC2. If you run into problems it's probably going to be due to > broken firmware, [...] Can you explain a bit here if you've got a few minutes?=20 I'm used to the old world concept of IDE/SCSI/SAS controllers (esp. RAID = controllers) where you've got a lot of processing going on in the card = to go from the drive interface to whatever kind of bus the controller = sits on in the PC. But in the case of NVMe, what even is a "controller"? = Is it not just a card that's combining/arbitrating PCI-e lanes from a = bunch of drives to the PCI-e bus? Where would be the firmware and what = would be the compatibility issues? I admit I don't use any NVMe stuff = anywhere, so I'm not very familiar with it, and especially not with this = setup where we have a "controller" involved... Thanks, Charles [...] > so unless anyone has experience with that specific server > I think the best answer you can get is "it should work but you need to = test > it and find out". >=20 > --=20 > Colin Percival > FreeBSD Release Engineering Lead & EC2 platform maintainer > Founder, Tarsnap | www.tarsnap.com | Online backups for the truly = paranoid >=20 >=20