Date: Fri, 09 Sep 2016 14:14:50 -0700 From: Dennis Glatting <freebsd@pki2.com> To: Christoph Pilka <c.pilka@asconix.com>, freebsd-questions@freebsd.org Subject: Re: 40 cores, 48 NVMe disks, feel free to take over Message-ID: <1473455690.58708.93.camel@pki2.com> In-Reply-To: <E264C60F-7317-4D99-882C-8F76191238BE@asconix.com> References: <E264C60F-7317-4D99-882C-8F76191238BE@asconix.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, 2016-09-09 at 22:51 +0200, Christoph Pilka wrote: > Hi, > > we've just been granted a short-term loan of a server from Supermicro > with 40 physical cores (plus HTT) and 48 NVMe drives. After a bit of > mucking about, we managed to get 11-RC running. A couple of things > are preventing the system from being terribly useful: > > - We have to use hw.nvme.force_intx=1 for the server to boot > If we don't, it panics around the 9th NVMe drive with "panic: > couldn't find an APIC vector for IRQ...". Increasing > hw.nvme.min_cpus_per_ioq brings it further, but it still panics later > in the NVMe enumeration/init. hw.nvme.per_cpu_io_queues=0 causes it > to panic later (I suspect during ixl init - the box has 4x10gb > ethernet ports). > > - zfskern seems to be the limiting factor when doing ~40 parallel "dd > if=/dev/zer of=<file> bs=1m" on a zpool stripe of all 48 drives. Each > drive shows ~30% utilization (gstat), I can do ~14GB/sec write and 16 > read. > > - direct writing to the NVMe devices (dd from /dev/zero) gives about > 550MB/sec and ~91% utilization per device > > Obviously, the first item is the most troublesome. The rest is based > on entirely synthetic testing and may have little or no actual impact > on the server's usability or fitness for our purposes. > > There is nothing but sshd running on the server, and if anyone wants > to play around you'll have IPMI access (remote kvm, virtual media, > power) and root. > > Any takers? > I'm curious to know what board you have. I have had FreeBSD, including release 11 candidates, running on SM boards without any trouble although some of them are older boards. I haven't looked at ZFS performance because mine are typically low disk use. That said, my virtual server (also a SM) IOPs suck but so do its disks. I recently found the Intel RAID chip on one SM isn't real RAID, rather it's pseudo RAID but for a few dollars more it could be real RAID. :( It was killing IOPs so I popped in an old LSI board, routed the cables from the Intel chip, and the server is now a happy camper. I then replaced 11-RC with Ubuntu 16.10 due to a specific application but I am also running RAIDz2 under Ubuntu on three trash 2.5T disks (I didn't do this for any reason other than fun). root@Tuck3r:/opt/bin# zpool status pool: opt state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM opt ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 sdc ONLINE 0 0 0 > Wbr > Christoph Pilka > Modirum MDpay > > Sent from my iPhone > _______________________________________________ > freebsd-questions@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freeb > sd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1473455690.58708.93.camel>