Date: Fri, 9 May 2025 11:57:32 -0600 From: Warner Losh <imp@bsdimp.com> To: Charles Sprickman <spork@bway.net> Cc: Colin Percival <cperciva@tarsnap.com>, freebsd-stable@freebsd.org Subject: Re: NVMe (U.2) hot-swap support status? Message-ID: <CANCZdfreu4AHa2Go2eYkTMKG=dYG%2B_T4WghpX1qqLCoa18K61g@mail.gmail.com> In-Reply-To: <5530E3D8-B2F8-4101-91B1-D8BAC17D2FDA@bway.net> References: <2E8A7BC6-6C3F-431F-B6F0-2611577B8028@bway.net> <01000196a3fb8737-9e591147-f4fd-439d-85bc-5f155be4bebb-000000@email.amazonses.com> <5530E3D8-B2F8-4101-91B1-D8BAC17D2FDA@bway.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, May 6, 2025 at 11:31=E2=80=AFAM Charles Sprickman <spork@bway.net> = wrote: > > > > On May 6, 2025, at 1:04=E2=80=AFAM, Colin Percival <cperciva@tarsnap.co= m> wrote: > > > > On 5/5/25 21:54, Charles Sprickman wrote: > >> Anyhow, we're trying purchase a few servers (likely this Supermicro: h= ttps://www.supermicro.com/en/products/system/UP/1U/SYS-111E-WR) and since m= anufacturers don't seem to validate for FreeBSD these days, I was hoping fo= r some community input. > >> Last time I looked into this NVMe (specifically the "U.2" format for s= ervers that includes hot swap) was supported in FreeBSD but hot swap was ki= nd of iffy. I don't think too many FreeBSD users/developers at that time ha= d the hardware, most people seemed to just be using an on-board NVMe drive = on desktop or laptop systems, and there's not any real call for hot swap in= that segment. > >> I have a thread on the forums linked below, but really what I'm lookin= g to find is some larger orgs that are running NVMe drives and can confirm = hot swap works as expected, if there's any gotchas to be on the lookout for= , etc. Trying not to overcomplicate it, but we don't want to be stuck with = a bunch of servers that might be less stable than our current SAS/SATA serv= ers. We exclusively use ZFS, if that's useful info. > > > > I can't speak to *physical* hotplug, but I've put a lot of energy over = the > > past few months into making sure that nvme "hotplug" is 100% functional= in > > Amazon EC2. If you run into problems it's probably going to be due to > > broken firmware, > [...] > > Can you explain a bit here if you've got a few minutes? > > I'm used to the old world concept of IDE/SCSI/SAS controllers (esp. RAID = controllers) where you've got a lot of processing going on in the card to g= o from the drive interface to whatever kind of bus the controller sits on i= n the PC. But in the case of NVMe, what even is a "controller"? Is it not j= ust a card that's combining/arbitrating PCI-e lanes from a bunch of drives = to the PCI-e bus? Where would be the firmware and what would be the compati= bility issues? I admit I don't use any NVMe stuff anywhere, so I'm not very= familiar with it, and especially not with this setup where we have a "cont= roller" involved... I've had access to a couple of hotplug chassis / motherboards. For x86, they've just worked for me. While the controller is bundled onto the nvme card, the PCIe bus has protocols to cope with a card being removed. FreeBSD has support for the hotplug standards around this. I've had some dodgy firmware on arm64 systems fail, though. I've not had the time to puzzle ou why.... Warner > Thanks, > > Charles > > [...] > > so unless anyone has experience with that specific server > > I think the best answer you can get is "it should work but you need to = test > > it and find out". > > > > -- > > Colin Percival > > FreeBSD Release Engineering Lead & EC2 platform maintainer > > Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paran= oid > > > > > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfreu4AHa2Go2eYkTMKG=dYG%2B_T4WghpX1qqLCoa18K61g>