From owner-freebsd-questions@freebsd.org Tue Sep 20 17:12:50 2016 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3DD91BE2614 for ; Tue, 20 Sep 2016 17:12:50 +0000 (UTC) (envelope-from ricera10@gmail.com) Received: from mail-yw0-f174.google.com (mail-yw0-f174.google.com [209.85.161.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 032081E56; Tue, 20 Sep 2016 17:12:49 +0000 (UTC) (envelope-from ricera10@gmail.com) Received: by mail-yw0-f174.google.com with SMTP id i129so16564260ywb.0; Tue, 20 Sep 2016 10:12:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=m8q7Z81Yb9IUHZx8BToLYXIa3wdaFIFELtWm2Ear7+M=; b=iEKtheTSc3zXQzR4gp9iSHSIMUAicp2SS19ns9ass8W8VX3sIpLWSLH4nGWcFkMrNN CqEnRD+9TyvD8UdXPyMor/oq2F90kWqYpR/D2upmn3OplpUHgdDYf1H1wQRLPOo0f0l9 Sa/njq0iWiHnWewkSwOdYQBnuOgmC0l9bLRd+KZZ+TTXpwG9StE6J0eYZCxClkpTqg2y aAK7DUIWUtVpVhw9bzg0xxbkq3TlGb1dPEDFGRMbE0GQ6Sm+K4Nr0JgpBoCdbWa6zy8h w8PLNUD045LVP69SwrFPJxkBvWaKB+F8lM9o+l8rRBsFwiAf9XPVrxTaDMRXLuZqR+Ul y6FA== X-Gm-Message-State: AE9vXwNjEqte62KHl79wJkRTr0mIptap4ytrxaG85LCwh7X6XVVn9RMGO63xnmin/6Znog== X-Received: by 10.13.202.133 with SMTP id m127mr25811575ywd.251.1474391563248; Tue, 20 Sep 2016 10:12:43 -0700 (PDT) Received: from mail-yw0-f175.google.com (mail-yw0-f175.google.com. [209.85.161.175]) by smtp.gmail.com with ESMTPSA id s131sm11823971ywg.18.2016.09.20.10.12.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 20 Sep 2016 10:12:42 -0700 (PDT) Received: by mail-yw0-f175.google.com with SMTP id i129so16563809ywb.0; Tue, 20 Sep 2016 10:12:42 -0700 (PDT) X-Received: by 10.13.231.3 with SMTP id q3mr32053692ywe.242.1474391562348; Tue, 20 Sep 2016 10:12:42 -0700 (PDT) MIME-Version: 1.0 References: <20160911203502.GA24973@neutralgood.org> <2828115.ibI7SUQqHX@ralph.baldwin.cx> In-Reply-To: <2828115.ibI7SUQqHX@ralph.baldwin.cx> From: Eric Joyner Date: Tue, 20 Sep 2016 17:12:32 +0000 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: 40 cores, 48 NVMe disks, feel free to take over To: John Baldwin , Adrian Chadd Cc: "Kevin P. Neal" , FreeBSD Questions , Christoph Pilka Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Sep 2016 17:12:50 -0000 Christoph, Did you end up filing a bug report? I thought the ixl(4) MSI-X interrupt allocation failure was handled properly in 11, but if it's as you suspect, I might still have to look at it again. On Mon, Sep 19, 2016 at 1:10 PM John Baldwin wrote: > On Monday, September 19, 2016 11:56:58 AM Adrian Chadd wrote: > > Hi, > > > > I think the nvme allocation issue is known. John? > > A kernel with 'options EARLY_AP_STARTUP' (which I plan to enable by default > in HEAD "soon") should boot fine without needing the force_intx hack. The > option is available in 11 but not enabled by default. > > > -a > > > > > > On 11 September 2016 at 13:35, Kevin P. Neal > wrote: > > > On Sat, Sep 10, 2016 at 10:57:07AM +0200, Christoph Pilka wrote: > > >> Hi, > > >> > > >> the server we got to experiment with is the SuperMicro 2028R-NR48N ( > https://www.supermicro.nl/products/system/2U/2028/SSG-2028R-NR48N.cfm < > https://www.supermicro.nl/products/system/2U/2028/SSG-2028R-NR48N.cfm>), > the board itself is a X10DSC+ > > > > > > The best thing to do is file a bug report. If you don't then your > report > > > will probably fall through the cracks. Include all the info you've > posted > > > so far. > > > > > >> //Chris > > >> > > >> > On 09 Sep 2016, at 23:14, Dennis Glatting wrote: > > >> > > > >> > On Fri, 2016-09-09 at 22:51 +0200, Christoph Pilka wrote: > > >> >> Hi, > > >> >> > > >> >> we've just been granted a short-term loan of a server from > Supermicro > > >> >> with 40 physical cores (plus HTT) and 48 NVMe drives. After a bit > of > > >> >> mucking about, we managed to get 11-RC running. A couple of things > > >> >> are preventing the system from being terribly useful: > > >> >> > > >> >> - We have to use hw.nvme.force_intx=1 for the server to boot > > >> >> If we don't, it panics around the 9th NVMe drive with "panic: > > >> >> couldn't find an APIC vector for IRQ...". Increasing > > >> >> hw.nvme.min_cpus_per_ioq brings it further, but it still panics > later > > >> >> in the NVMe enumeration/init. hw.nvme.per_cpu_io_queues=0 causes it > > >> >> to panic later (I suspect during ixl init - the box has 4x10gb > > >> >> ethernet ports). > > >> >> > > >> >> - zfskern seems to be the limiting factor when doing ~40 parallel > "dd > > >> >> if=/dev/zer of= bs=1m" on a zpool stripe of all 48 drives. > Each > > >> >> drive shows ~30% utilization (gstat), I can do ~14GB/sec write and > 16 > > >> >> read. > > >> >> > > >> >> - direct writing to the NVMe devices (dd from /dev/zero) gives > about > > >> >> 550MB/sec and ~91% utilization per device > > >> >> > > >> >> Obviously, the first item is the most troublesome. The rest is > based > > >> >> on entirely synthetic testing and may have little or no actual > impact > > >> >> on the server's usability or fitness for our purposes. > > >> >> > > >> >> There is nothing but sshd running on the server, and if anyone > wants > > >> >> to play around you'll have IPMI access (remote kvm, virtual media, > > >> >> power) and root. > > >> >> > > >> >> Any takers? > > >> >> > > >> > > > >> > > > >> > I'm curious to know what board you have. I have had FreeBSD, > including > > >> > release 11 candidates, running on SM boards without any trouble > > >> > although some of them are older boards. I haven't looked at ZFS > > >> > performance because mine are typically low disk use. That said, my > > >> > virtual server (also a SM) IOPs suck but so do its disks. > > >> > > > >> > I recently found the Intel RAID chip on one SM isn't real RAID, > rather > > >> > it's pseudo RAID but for a few dollars more it could be real RAID. > :( > > >> > It was killing IOPs so I popped in an old LSI board, routed the > cables > > >> > from the Intel chip, and the server is now a happy camper. I then > > >> > replaced 11-RC with Ubuntu 16.10 due to a specific application but > I am > > >> > also running RAIDz2 under Ubuntu on three trash 2.5T disks (I > didn't do > > >> > this for any reason other than fun). > > >> > > > >> > root@Tuck3r:/opt/bin# zpool status > > >> > pool: opt > > >> > state: ONLINE > > >> > scan: none requested > > >> > config: > > >> > > > >> > NAME STATE READ WRITE CKSUM > > >> > opt ONLINE 0 0 0 > > >> > raidz2-0 ONLINE 0 0 0 > > >> > sda ONLINE 0 0 0 > > >> > sdb ONLINE 0 0 0 > > >> > sdc ONLINE 0 0 0 > > >> > > > >> > > > >> > > > >> >> Wbr > > >> >> Christoph Pilka > > >> >> Modirum MDpay > > >> >> > > >> >> Sent from my iPhone > > >> >> _______________________________________________ > > >> >> freebsd-questions@freebsd.org freebsd-questions@freebsd.org> mailing list > > >> >> https://lists.freebsd.org/mailman/listinfo/freebsd-questions < > https://lists.freebsd.org/mailman/listinfo/freebsd-questions> > > >> >> To unsubscribe, send any mail to > "freebsd-questions-unsubscribe@freeb > > >> >> sd.org " > > >> > _______________________________________________ > > >> > freebsd-questions@freebsd.org > mailing list > > >> > https://lists.freebsd.org/mailman/listinfo/freebsd-questions < > https://lists.freebsd.org/mailman/listinfo/freebsd-questions> > > >> > To unsubscribe, send any mail to " > freebsd-questions-unsubscribe@freebsd.org freebsd-questions-unsubscribe@freebsd.org>" > > >> > > >> _______________________________________________ > > >> freebsd-questions@freebsd.org mailing list > > >> https://lists.freebsd.org/mailman/listinfo/freebsd-questions > > >> To unsubscribe, send any mail to " > freebsd-questions-unsubscribe@freebsd.org" > > > -- > > > Kevin P. Neal > http://www.pobox.com/~kpn/ > > > > > > "Good grief, I've just noticed I've typed in a rant. Sorry chaps!" > > > Keir Finlow Bates, circa 1998 > > > _______________________________________________ > > > freebsd-questions@freebsd.org mailing list > > > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > > > To unsubscribe, send any mail to " > freebsd-questions-unsubscribe@freebsd.org" > > > -- > John Baldwin > _______________________________________________ > freebsd-questions@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to " > freebsd-questions-unsubscribe@freebsd.org" >