From owner-freebsd-questions@freebsd.org Fri Sep 9 20:51:48 2016 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DD877BD2D7C for ; Fri, 9 Sep 2016 20:51:48 +0000 (UTC) (envelope-from c.pilka@asconix.com) Received: from aibo.runbox.com (aibo.runbox.com [91.220.196.211]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A6927A41 for ; Fri, 9 Sep 2016 20:51:48 +0000 (UTC) (envelope-from c.pilka@asconix.com) Received: from [10.9.9.212] (helo=mailfront12.runbox.com) by bars.runbox.com with esmtp (Exim 4.71) (envelope-from ) id 1biSlv-0006fG-1A for freebsd-questions@freebsd.org; Fri, 09 Sep 2016 22:51:39 +0200 Received: from 78-176-11.connect.netcom.no ([176.11.176.78] helo=[10.242.175.77]) by mailfront12.runbox.com with esmtpsa (uid:865152 ) (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) id 1biSlh-000050-Vt for freebsd-questions@freebsd.org; Fri, 09 Sep 2016 22:51:26 +0200 From: Christoph Pilka Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (1.0) Subject: 40 cores, 48 NVMe disks, feel free to take over Message-Id: Date: Fri, 9 Sep 2016 22:51:25 +0200 To: freebsd-questions@freebsd.org X-Mailer: iPhone Mail (13G36) X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Sep 2016 20:51:49 -0000 Hi, we've just been granted a short-term loan of a server from Supermicro with 4= 0 physical cores (plus HTT) and 48 NVMe drives. After a bit of mucking about= , we managed to get 11-RC running. A couple of things are preventing the sys= tem from being terribly useful: - We have to use hw.nvme.force_intx=3D1 for the server to boot If we don't, it panics around the 9th NVMe drive with "panic: couldn't find a= n APIC vector for IRQ...". Increasing hw.nvme.min_cpus_per_ioq brings it fur= ther, but it still panics later in the NVMe enumeration/init. hw.nvme.per_cp= u_io_queues=3D0 causes it to panic later (I suspect during ixl init - the bo= x has 4x10gb ethernet ports). - zfskern seems to be the limiting factor when doing ~40 parallel "dd if=3D/= dev/zer of=3D bs=3D1m" on a zpool stripe of all 48 drives. Each drive s= hows ~30% utilization (gstat), I can do ~14GB/sec write and 16 read. - direct writing to the NVMe devices (dd from /dev/zero) gives about 550MB/s= ec and ~91% utilization per device=20 Obviously, the first item is the most troublesome. The rest is based on enti= rely synthetic testing and may have little or no actual impact on the server= 's usability or fitness for our purposes.=20 There is nothing but sshd running on the server, and if anyone wants to play= around you'll have IPMI access (remote kvm, virtual media, power) and root.= Any takers? Wbr Christoph Pilka Modirum MDpay Sent from my iPhone=