Date: Tue, 20 Sep 2016 17:12:32 +0000 From: Eric Joyner <erj@freebsd.org> To: John Baldwin <jhb@freebsd.org>, Adrian Chadd <adrian.chadd@gmail.com> Cc: "Kevin P. Neal" <kpn@neutralgood.org>, FreeBSD Questions <freebsd-questions@freebsd.org>, Christoph Pilka <c.pilka@asconix.com> Subject: Re: 40 cores, 48 NVMe disks, feel free to take over Message-ID: <CA%2Bb0zg9_aFHCm1qVnuDCBrJv8E_e_d2YK-2Jy=43wtojy78pvQ@mail.gmail.com> In-Reply-To: <2828115.ibI7SUQqHX@ralph.baldwin.cx> References: <E264C60F-7317-4D99-882C-8F76191238BE@asconix.com> <20160911203502.GA24973@neutralgood.org> <CAJ-Vmon3BwXtjSf8bRmeagUt9gLwpo7bJNs01CDUTPN3h8vOCQ@mail.gmail.com> <2828115.ibI7SUQqHX@ralph.baldwin.cx>
next in thread | previous in thread | raw e-mail | index | archive | help
Christoph, Did you end up filing a bug report? I thought the ixl(4) MSI-X interrupt allocation failure was handled properly in 11, but if it's as you suspect, I might still have to look at it again. On Mon, Sep 19, 2016 at 1:10 PM John Baldwin <jhb@freebsd.org> wrote: > On Monday, September 19, 2016 11:56:58 AM Adrian Chadd wrote: > > Hi, > > > > I think the nvme allocation issue is known. John? > > A kernel with 'options EARLY_AP_STARTUP' (which I plan to enable by default > in HEAD "soon") should boot fine without needing the force_intx hack. The > option is available in 11 but not enabled by default. > > > -a > > > > > > On 11 September 2016 at 13:35, Kevin P. Neal <kpn@neutralgood.org> > wrote: > > > On Sat, Sep 10, 2016 at 10:57:07AM +0200, Christoph Pilka wrote: > > >> Hi, > > >> > > >> the server we got to experiment with is the SuperMicro 2028R-NR48N ( > https://www.supermicro.nl/products/system/2U/2028/SSG-2028R-NR48N.cfm < > https://www.supermicro.nl/products/system/2U/2028/SSG-2028R-NR48N.cfm>), > the board itself is a X10DSC+ > > > > > > The best thing to do is file a bug report. If you don't then your > report > > > will probably fall through the cracks. Include all the info you've > posted > > > so far. > > > > > >> //Chris > > >> > > >> > On 09 Sep 2016, at 23:14, Dennis Glatting <freebsd@pki2.com> wrote: > > >> > > > >> > On Fri, 2016-09-09 at 22:51 +0200, Christoph Pilka wrote: > > >> >> Hi, > > >> >> > > >> >> we've just been granted a short-term loan of a server from > Supermicro > > >> >> with 40 physical cores (plus HTT) and 48 NVMe drives. After a bit > of > > >> >> mucking about, we managed to get 11-RC running. A couple of things > > >> >> are preventing the system from being terribly useful: > > >> >> > > >> >> - We have to use hw.nvme.force_intx=1 for the server to boot > > >> >> If we don't, it panics around the 9th NVMe drive with "panic: > > >> >> couldn't find an APIC vector for IRQ...". Increasing > > >> >> hw.nvme.min_cpus_per_ioq brings it further, but it still panics > later > > >> >> in the NVMe enumeration/init. hw.nvme.per_cpu_io_queues=0 causes it > > >> >> to panic later (I suspect during ixl init - the box has 4x10gb > > >> >> ethernet ports). > > >> >> > > >> >> - zfskern seems to be the limiting factor when doing ~40 parallel > "dd > > >> >> if=/dev/zer of=<file> bs=1m" on a zpool stripe of all 48 drives. > Each > > >> >> drive shows ~30% utilization (gstat), I can do ~14GB/sec write and > 16 > > >> >> read. > > >> >> > > >> >> - direct writing to the NVMe devices (dd from /dev/zero) gives > about > > >> >> 550MB/sec and ~91% utilization per device > > >> >> > > >> >> Obviously, the first item is the most troublesome. The rest is > based > > >> >> on entirely synthetic testing and may have little or no actual > impact > > >> >> on the server's usability or fitness for our purposes. > > >> >> > > >> >> There is nothing but sshd running on the server, and if anyone > wants > > >> >> to play around you'll have IPMI access (remote kvm, virtual media, > > >> >> power) and root. > > >> >> > > >> >> Any takers? > > >> >> > > >> > > > >> > > > >> > I'm curious to know what board you have. I have had FreeBSD, > including > > >> > release 11 candidates, running on SM boards without any trouble > > >> > although some of them are older boards. I haven't looked at ZFS > > >> > performance because mine are typically low disk use. That said, my > > >> > virtual server (also a SM) IOPs suck but so do its disks. > > >> > > > >> > I recently found the Intel RAID chip on one SM isn't real RAID, > rather > > >> > it's pseudo RAID but for a few dollars more it could be real RAID. > :( > > >> > It was killing IOPs so I popped in an old LSI board, routed the > cables > > >> > from the Intel chip, and the server is now a happy camper. I then > > >> > replaced 11-RC with Ubuntu 16.10 due to a specific application but > I am > > >> > also running RAIDz2 under Ubuntu on three trash 2.5T disks (I > didn't do > > >> > this for any reason other than fun). > > >> > > > >> > root@Tuck3r:/opt/bin# zpool status > > >> > pool: opt > > >> > state: ONLINE > > >> > scan: none requested > > >> > config: > > >> > > > >> > NAME STATE READ WRITE CKSUM > > >> > opt ONLINE 0 0 0 > > >> > raidz2-0 ONLINE 0 0 0 > > >> > sda ONLINE 0 0 0 > > >> > sdb ONLINE 0 0 0 > > >> > sdc ONLINE 0 0 0 > > >> > > > >> > > > >> > > > >> >> Wbr > > >> >> Christoph Pilka > > >> >> Modirum MDpay > > >> >> > > >> >> Sent from my iPhone > > >> >> _______________________________________________ > > >> >> freebsd-questions@freebsd.org <mailto: > freebsd-questions@freebsd.org> mailing list > > >> >> https://lists.freebsd.org/mailman/listinfo/freebsd-questions < > https://lists.freebsd.org/mailman/listinfo/freebsd-questions>; > > >> >> To unsubscribe, send any mail to > "freebsd-questions-unsubscribe@freeb > > >> >> sd.org <http://sd.org/>" > > >> > _______________________________________________ > > >> > freebsd-questions@freebsd.org <mailto:freebsd-questions@freebsd.org> > mailing list > > >> > https://lists.freebsd.org/mailman/listinfo/freebsd-questions < > https://lists.freebsd.org/mailman/listinfo/freebsd-questions>; > > >> > To unsubscribe, send any mail to " > freebsd-questions-unsubscribe@freebsd.org <mailto: > freebsd-questions-unsubscribe@freebsd.org>" > > >> > > >> _______________________________________________ > > >> freebsd-questions@freebsd.org mailing list > > >> https://lists.freebsd.org/mailman/listinfo/freebsd-questions > > >> To unsubscribe, send any mail to " > freebsd-questions-unsubscribe@freebsd.org" > > > -- > > > Kevin P. Neal > http://www.pobox.com/~kpn/ > > > > > > "Good grief, I've just noticed I've typed in a rant. Sorry chaps!" > > > Keir Finlow Bates, circa 1998 > > > _______________________________________________ > > > freebsd-questions@freebsd.org mailing list > > > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > > > To unsubscribe, send any mail to " > freebsd-questions-unsubscribe@freebsd.org" > > > -- > John Baldwin > _______________________________________________ > freebsd-questions@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to " > freebsd-questions-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2Bb0zg9_aFHCm1qVnuDCBrJv8E_e_d2YK-2Jy=43wtojy78pvQ>