Date: Mon, 19 Sep 2016 11:56:58 -0700 From: Adrian Chadd <adrian.chadd@gmail.com> To: "Kevin P. Neal" <kpn@neutralgood.org>, John Baldwin <jhb@freebsd.org> Cc: Christoph Pilka <c.pilka@asconix.com>, FreeBSD Questions <freebsd-questions@freebsd.org> Subject: Re: 40 cores, 48 NVMe disks, feel free to take over Message-ID: <CAJ-Vmon3BwXtjSf8bRmeagUt9gLwpo7bJNs01CDUTPN3h8vOCQ@mail.gmail.com> In-Reply-To: <20160911203502.GA24973@neutralgood.org> References: <E264C60F-7317-4D99-882C-8F76191238BE@asconix.com> <1473455690.58708.93.camel@pki2.com> <CCE56C3D-19ED-4A10-942C-16C5379028C1@asconix.com> <20160911203502.GA24973@neutralgood.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi, I think the nvme allocation issue is known. John? -a On 11 September 2016 at 13:35, Kevin P. Neal <kpn@neutralgood.org> wrote: > On Sat, Sep 10, 2016 at 10:57:07AM +0200, Christoph Pilka wrote: >> Hi, >> >> the server we got to experiment with is the SuperMicro 2028R-NR48N (https://www.supermicro.nl/products/system/2U/2028/SSG-2028R-NR48N.cfm <https://www.supermicro.nl/products/system/2U/2028/SSG-2028R-NR48N.cfm>), the board itself is a X10DSC+ > > The best thing to do is file a bug report. If you don't then your report > will probably fall through the cracks. Include all the info you've posted > so far. > >> //Chris >> >> > On 09 Sep 2016, at 23:14, Dennis Glatting <freebsd@pki2.com> wrote: >> > >> > On Fri, 2016-09-09 at 22:51 +0200, Christoph Pilka wrote: >> >> Hi, >> >> >> >> we've just been granted a short-term loan of a server from Supermicro >> >> with 40 physical cores (plus HTT) and 48 NVMe drives. After a bit of >> >> mucking about, we managed to get 11-RC running. A couple of things >> >> are preventing the system from being terribly useful: >> >> >> >> - We have to use hw.nvme.force_intx=1 for the server to boot >> >> If we don't, it panics around the 9th NVMe drive with "panic: >> >> couldn't find an APIC vector for IRQ...". Increasing >> >> hw.nvme.min_cpus_per_ioq brings it further, but it still panics later >> >> in the NVMe enumeration/init. hw.nvme.per_cpu_io_queues=0 causes it >> >> to panic later (I suspect during ixl init - the box has 4x10gb >> >> ethernet ports). >> >> >> >> - zfskern seems to be the limiting factor when doing ~40 parallel "dd >> >> if=/dev/zer of=<file> bs=1m" on a zpool stripe of all 48 drives. Each >> >> drive shows ~30% utilization (gstat), I can do ~14GB/sec write and 16 >> >> read. >> >> >> >> - direct writing to the NVMe devices (dd from /dev/zero) gives about >> >> 550MB/sec and ~91% utilization per device >> >> >> >> Obviously, the first item is the most troublesome. The rest is based >> >> on entirely synthetic testing and may have little or no actual impact >> >> on the server's usability or fitness for our purposes. >> >> >> >> There is nothing but sshd running on the server, and if anyone wants >> >> to play around you'll have IPMI access (remote kvm, virtual media, >> >> power) and root. >> >> >> >> Any takers? >> >> >> > >> > >> > I'm curious to know what board you have. I have had FreeBSD, including >> > release 11 candidates, running on SM boards without any trouble >> > although some of them are older boards. I haven't looked at ZFS >> > performance because mine are typically low disk use. That said, my >> > virtual server (also a SM) IOPs suck but so do its disks. >> > >> > I recently found the Intel RAID chip on one SM isn't real RAID, rather >> > it's pseudo RAID but for a few dollars more it could be real RAID. :( >> > It was killing IOPs so I popped in an old LSI board, routed the cables >> > from the Intel chip, and the server is now a happy camper. I then >> > replaced 11-RC with Ubuntu 16.10 due to a specific application but I am >> > also running RAIDz2 under Ubuntu on three trash 2.5T disks (I didn't do >> > this for any reason other than fun). >> > >> > root@Tuck3r:/opt/bin# zpool status >> > pool: opt >> > state: ONLINE >> > scan: none requested >> > config: >> > >> > NAME STATE READ WRITE CKSUM >> > opt ONLINE 0 0 0 >> > raidz2-0 ONLINE 0 0 0 >> > sda ONLINE 0 0 0 >> > sdb ONLINE 0 0 0 >> > sdc ONLINE 0 0 0 >> > >> > >> > >> >> Wbr >> >> Christoph Pilka >> >> Modirum MDpay >> >> >> >> Sent from my iPhone >> >> _______________________________________________ >> >> freebsd-questions@freebsd.org <mailto:freebsd-questions@freebsd.org> mailing list >> >> https://lists.freebsd.org/mailman/listinfo/freebsd-questions <https://lists.freebsd.org/mailman/listinfo/freebsd-questions>; >> >> To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freeb >> >> sd.org <http://sd.org/>" >> > _______________________________________________ >> > freebsd-questions@freebsd.org <mailto:freebsd-questions@freebsd.org> mailing list >> > https://lists.freebsd.org/mailman/listinfo/freebsd-questions <https://lists.freebsd.org/mailman/listinfo/freebsd-questions>; >> > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org <mailto:freebsd-questions-unsubscribe@freebsd.org>" >> >> _______________________________________________ >> freebsd-questions@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-questions >> To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org" > -- > Kevin P. Neal http://www.pobox.com/~kpn/ > > "Good grief, I've just noticed I've typed in a rant. Sorry chaps!" > Keir Finlow Bates, circa 1998 > _______________________________________________ > freebsd-questions@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-Vmon3BwXtjSf8bRmeagUt9gLwpo7bJNs01CDUTPN3h8vOCQ>