From owner-freebsd-fs@freebsd.org Sat Sep 10 08:58:27 2016 Return-Path: <owner-freebsd-fs@freebsd.org> Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8B36DBD2396 for <freebsd-fs@mailman.ysv.freebsd.org>; Sat, 10 Sep 2016 08:58:27 +0000 (UTC) (envelope-from c.pilka@asconix.com) Received: from aibo.runbox.com (aibo.runbox.com [91.220.196.211]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 52697EFE for <freebsd-fs@freebsd.org>; Sat, 10 Sep 2016 08:58:26 +0000 (UTC) (envelope-from c.pilka@asconix.com) Received: from [10.9.9.211] (helo=mailfront11.runbox.com) by bars.runbox.com with esmtp (Exim 4.71) (envelope-from <c.pilka@asconix.com>) id 1bie7F-0000pA-0u for freebsd-fs@freebsd.org; Sat, 10 Sep 2016 10:58:25 +0200 Received: from cm-84.211.200.201.getinternet.no ([84.211.200.201] helo=houdini.fritz.box) by mailfront11.runbox.com with esmtpsa (uid:865152 ) (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) id 1bie7D-0005Lu-8j for freebsd-fs@freebsd.org; Sat, 10 Sep 2016 10:58:23 +0200 From: Christoph Pilka <c.pilka@asconix.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: Server with 40 physical cores, 48 NVMe disks, feel free to test it Message-Id: <C6904B7F-D148-47C0-BD17-0A2AF63B5717@asconix.com> Date: Sat, 10 Sep 2016 10:58:22 +0200 To: freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems <freebsd-fs.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fs>, <mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/> List-Post: <mailto:freebsd-fs@freebsd.org> List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>, <mailto:freebsd-fs-request@freebsd.org?subject=subscribe> X-List-Received-Date: Sat, 10 Sep 2016 08:58:27 -0000 Hi, we've just been granted a short-term loan of a server from Supermicro = with 40 physical cores (plus HTT) and 48 NVMe drives. After a bit of = mucking about, we managed to get 11-RC running. A couple of things are = preventing the system from being terribly useful: - We have to use hw.nvme.force_intx=3D1 for the server to boot If we don't, it panics around the 9th NVMe drive with "panic: couldn't = find an APIC vector for IRQ...". Increasing hw.nvme.min_cpus_per_ioq = brings it further, but it still panics later in the NVMe = enumeration/init. hw.nvme.per_cpu_io_queues=3D0 causes it to panic later = (I suspect during ixl init - the box has 4x10gb ethernet ports). - zfskern seems to be the limiting factor when doing ~40 parallel "dd = if=3D/dev/zer of=3D<file> bs=3D1m" on a zpool stripe of all 48 drives. = Each drive shows ~30% utilization (gstat), I can do ~14GB/sec write and = 16 read. - direct writing to the NVMe devices (dd from /dev/zero) gives about = 550MB/sec and ~91% utilization per device=20 Obviously, the first item is the most troublesome. The rest is based on = entirely synthetic testing and may have little or no actual impact on = the server's usability or fitness for our purposes.=20 There is nothing but sshd running on the server, and if anyone wants to = play around you'll have IPMI access (remote kvm, virtual media, power) = and root. Any takers? //Chris=