From owner-freebsd-questions@freebsd.org Mon Sep 19 18:57:00 2016 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0BF5BBE1FCC for ; Mon, 19 Sep 2016 18:57:00 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-io0-x22f.google.com (mail-io0-x22f.google.com [IPv6:2607:f8b0:4001:c06::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C72FF6D6; Mon, 19 Sep 2016 18:56:59 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-io0-x22f.google.com with SMTP id q92so101477712ioi.1; Mon, 19 Sep 2016 11:56:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=wcYBr0yUBnCCDijQQYyp7b0kTDOApXfl5MMyHjQY31o=; b=bati8Is8kUqUEFc0eKgqWKgFWrq3j1tAvitHHisjNMrtwCeSDEBTw+f04q010YD3s0 oDBgqBA9DTC4kUo6sjzaac5ipNDJJgM/YT8qzuLgumJ1qPEHqx2W7cJshV5whkIPJB48 eyt8crFEy6PU+bEs5O0k7gdngcn/NWtY/tPfurMPLs0qJBh0x1oUXZUZ5Sp4iIUYn1P/ +YbOva+QEK3FlZuwuXiTr+LQ+yu5U2MwwS3LsPcsnLuuI6oAWwl4sf5lbYe1/1y17EBY gWPpV1w0TNkEM6ZskJ1ApNY910n05I8FzLUcNRwto8gPYLP+8L5wG/QFfckCJATNno5T ekqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=wcYBr0yUBnCCDijQQYyp7b0kTDOApXfl5MMyHjQY31o=; b=J9+H8PQpWKZr1qMb+aeENXewFh0o8ozEXxApcqRsp10doNHWAUpXzl330//K3pXtie a/v6AN0BV4Fhre2fzxDrD2f2iiStZjqbm//Qmo4gDZ5EqYomypfxjoNi7Fj22iojxAho UcjewhPSHVmfAV1+OhGl/gIRn9pSLe1xashJWnHHAsv8vYoXxnrZPMynOz/6JfBbZV0u xuJeAAHhbDwB794ActCu2M18zgltkSkaC8YdMff/uMrHqC56jMrO9it4bY2PsS4Eq9Gx h7MmKQH+OB01pmrc3JPVTfEGW5BwqLo2t+eNoiw6Mz2vefsqY2xJEaFp454pcs7KzQRB s47g== X-Gm-Message-State: AE9vXwOxtehdCtRXuD9hL1F1T7XPAgcCop/p379JbVSQNG1GsnqDeoF5b6T7wUxhHb3ITKKjFBR6HVCY2AqQHg== X-Received: by 10.107.137.104 with SMTP id l101mr36595142iod.98.1474311419229; Mon, 19 Sep 2016 11:56:59 -0700 (PDT) MIME-Version: 1.0 Received: by 10.36.141.129 with HTTP; Mon, 19 Sep 2016 11:56:58 -0700 (PDT) In-Reply-To: <20160911203502.GA24973@neutralgood.org> References: <1473455690.58708.93.camel@pki2.com> <20160911203502.GA24973@neutralgood.org> From: Adrian Chadd Date: Mon, 19 Sep 2016 11:56:58 -0700 Message-ID: Subject: Re: 40 cores, 48 NVMe disks, feel free to take over To: "Kevin P. Neal" , John Baldwin Cc: Christoph Pilka , FreeBSD Questions Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Sep 2016 18:57:00 -0000 Hi, I think the nvme allocation issue is known. John? -a On 11 September 2016 at 13:35, Kevin P. Neal wrote: > On Sat, Sep 10, 2016 at 10:57:07AM +0200, Christoph Pilka wrote: >> Hi, >> >> the server we got to experiment with is the SuperMicro 2028R-NR48N (https://www.supermicro.nl/products/system/2U/2028/SSG-2028R-NR48N.cfm ), the board itself is a X10DSC+ > > The best thing to do is file a bug report. If you don't then your report > will probably fall through the cracks. Include all the info you've posted > so far. > >> //Chris >> >> > On 09 Sep 2016, at 23:14, Dennis Glatting wrote: >> > >> > On Fri, 2016-09-09 at 22:51 +0200, Christoph Pilka wrote: >> >> Hi, >> >> >> >> we've just been granted a short-term loan of a server from Supermicro >> >> with 40 physical cores (plus HTT) and 48 NVMe drives. After a bit of >> >> mucking about, we managed to get 11-RC running. A couple of things >> >> are preventing the system from being terribly useful: >> >> >> >> - We have to use hw.nvme.force_intx=1 for the server to boot >> >> If we don't, it panics around the 9th NVMe drive with "panic: >> >> couldn't find an APIC vector for IRQ...". Increasing >> >> hw.nvme.min_cpus_per_ioq brings it further, but it still panics later >> >> in the NVMe enumeration/init. hw.nvme.per_cpu_io_queues=0 causes it >> >> to panic later (I suspect during ixl init - the box has 4x10gb >> >> ethernet ports). >> >> >> >> - zfskern seems to be the limiting factor when doing ~40 parallel "dd >> >> if=/dev/zer of= bs=1m" on a zpool stripe of all 48 drives. Each >> >> drive shows ~30% utilization (gstat), I can do ~14GB/sec write and 16 >> >> read. >> >> >> >> - direct writing to the NVMe devices (dd from /dev/zero) gives about >> >> 550MB/sec and ~91% utilization per device >> >> >> >> Obviously, the first item is the most troublesome. The rest is based >> >> on entirely synthetic testing and may have little or no actual impact >> >> on the server's usability or fitness for our purposes. >> >> >> >> There is nothing but sshd running on the server, and if anyone wants >> >> to play around you'll have IPMI access (remote kvm, virtual media, >> >> power) and root. >> >> >> >> Any takers? >> >> >> > >> > >> > I'm curious to know what board you have. I have had FreeBSD, including >> > release 11 candidates, running on SM boards without any trouble >> > although some of them are older boards. I haven't looked at ZFS >> > performance because mine are typically low disk use. That said, my >> > virtual server (also a SM) IOPs suck but so do its disks. >> > >> > I recently found the Intel RAID chip on one SM isn't real RAID, rather >> > it's pseudo RAID but for a few dollars more it could be real RAID. :( >> > It was killing IOPs so I popped in an old LSI board, routed the cables >> > from the Intel chip, and the server is now a happy camper. I then >> > replaced 11-RC with Ubuntu 16.10 due to a specific application but I am >> > also running RAIDz2 under Ubuntu on three trash 2.5T disks (I didn't do >> > this for any reason other than fun). >> > >> > root@Tuck3r:/opt/bin# zpool status >> > pool: opt >> > state: ONLINE >> > scan: none requested >> > config: >> > >> > NAME STATE READ WRITE CKSUM >> > opt ONLINE 0 0 0 >> > raidz2-0 ONLINE 0 0 0 >> > sda ONLINE 0 0 0 >> > sdb ONLINE 0 0 0 >> > sdc ONLINE 0 0 0 >> > >> > >> > >> >> Wbr >> >> Christoph Pilka >> >> Modirum MDpay >> >> >> >> Sent from my iPhone >> >> _______________________________________________ >> >> freebsd-questions@freebsd.org mailing list >> >> https://lists.freebsd.org/mailman/listinfo/freebsd-questions >> >> To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freeb >> >> sd.org " >> > _______________________________________________ >> > freebsd-questions@freebsd.org mailing list >> > https://lists.freebsd.org/mailman/listinfo/freebsd-questions >> > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org " >> >> _______________________________________________ >> freebsd-questions@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-questions >> To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org" > -- > Kevin P. Neal http://www.pobox.com/~kpn/ > > "Good grief, I've just noticed I've typed in a rant. Sorry chaps!" > Keir Finlow Bates, circa 1998 > _______________________________________________ > freebsd-questions@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org"