Date: Sat, 10 Sep 2016 21:34:13 -0600 From: Warner Losh <imp@bsdimp.com> To: Christoph Pilka <c.pilka@asconix.com> Cc: freebsd-fs@freebsd.org Subject: Re: Server with 40 physical cores, 48 NVMe disks, feel free to test it Message-ID: <CANCZdfoAgHjqgcSMQBbpDBXkA8D7zdceZgnDVvMCPeM0Psg98Q@mail.gmail.com> In-Reply-To: <C6904B7F-D148-47C0-BD17-0A2AF63B5717@asconix.com> References: <C6904B7F-D148-47C0-BD17-0A2AF63B5717@asconix.com>
index | next in thread | previous in thread | raw e-mail
On Sat, Sep 10, 2016 at 2:58 AM, Christoph Pilka <c.pilka@asconix.com> wrote: > Hi, > > we've just been granted a short-term loan of a server from Supermicro with 40 physical cores (plus HTT) and 48 NVMe drives. After a bit of mucking about, we managed to get 11-RC running. A couple of things are preventing the system from being terribly useful: > > - We have to use hw.nvme.force_intx=1 for the server to boot > If we don't, it panics around the 9th NVMe drive with "panic: couldn't find an APIC vector for IRQ...". Increasing hw.nvme.min_cpus_per_ioq brings it further, but it still panics later in the NVMe enumeration/init. hw.nvme.per_cpu_io_queues=0 causes it to panic later (I suspect during ixl init - the box has 4x10gb ethernet ports). John Baldwin has patches that help fix this. > - zfskern seems to be the limiting factor when doing ~40 parallel "dd if=/dev/zer of=<file> bs=1m" on a zpool stripe of all 48 drives. Each drive shows ~30% utilization (gstat), I can do ~14GB/sec write and 16 read. > > - direct writing to the NVMe devices (dd from /dev/zero) gives about 550MB/sec and ~91% utilization per device These are slow drives then if all they can do 600MB/s. The drives we're looking at do 3.2GB/s read and 1.6GB/s write from the drives that we're looking at. 48 drives though. Woof. What's the interconnect? There's enough PCIe lanes for that? 192 lanes? How's that possible? > Obviously, the first item is the most troublesome. The rest is based on entirely synthetic testing and may have little or no actual impact on the server's usability or fitness for our purposes. > > There is nothing but sshd running on the server, and if anyone wants to play around you'll have IPMI access (remote kvm, virtual media, power) and root. Don't think I have enough time to track this all down... Warnerhelp
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfoAgHjqgcSMQBbpDBXkA8D7zdceZgnDVvMCPeM0Psg98Q>