From owner-freebsd-drivers@freebsd.org Sun Jan 14 14:50:17 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 79FA5E73B80; Sun, 14 Jan 2018 14:50:17 +0000 (UTC) (envelope-from galtsev@kicp.uchicago.edu) Received: from cosmo.uchicago.edu (cosmo.uchicago.edu [128.135.20.71]) by mx1.freebsd.org (Postfix) with ESMTP id 305437ED5D; Sun, 14 Jan 2018 14:50:16 +0000 (UTC) (envelope-from galtsev@kicp.uchicago.edu) Received: by cosmo.uchicago.edu (Postfix, from userid 48) id 7E4D1CB8D3A; Sun, 14 Jan 2018 08:50:10 -0600 (CST) Received: from 108.68.169.115 (SquirrelMail authenticated user valeri) by cosmo.uchicago.edu with HTTP; Sun, 14 Jan 2018 08:50:10 -0600 (CST) Message-ID: <60145.108.68.169.115.1515941410.squirrel@cosmo.uchicago.edu> In-Reply-To: <4cd39c52-9bf0-ef44-8335-9b4cf6eb6a6b@gjunka.com> References: <061ccfb3-ee6a-71a7-3926-372bb17b3171@kicp.uchicago.edu> <4cd39c52-9bf0-ef44-8335-9b4cf6eb6a6b@gjunka.com> Date: Sun, 14 Jan 2018 08:50:10 -0600 (CST) Subject: Re: Server doesn't boot when 3 PCIe slots are populated From: "Valeri Galtsev" To: "Grzegorz Junka" Cc: freebsd-questions@freebsd.org, freebsd-drivers@freebsd.org Reply-To: galtsev@kicp.uchicago.edu User-Agent: SquirrelMail/1.4.8-5.el5.centos.7 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Jan 2018 14:50:17 -0000 On Sun, January 14, 2018 8:34 am, Grzegorz Junka wrote: > > On 13/01/2018 18:31, Valeri Galtsev wrote: >> >> >> On 01/13/18 10:21, Grzegorz Junka wrote: >>> Hello, >>> >>> I am installing a FreeBSD server based on Supermicro H8SML-iF. There >>> are three PCIe slots to which I installed 2 NVMe drives and one >>> network card Intel I350-T4 (with 4 Ethernet slots). >>> >>> I am observing a strange behavior where the system doesn't boot if >>> all three PCIe slots are populated. It shows this message: >>> >>> nvme0: mem 0xfd8fc000-0xfd8fffff irq 24 at >>> device 0.0 on pci1 >>> nvme0: controller ready did not become 1 within 30000 ms >>> nvme0: did not complete shutdown within 5 seconds of notification >>> >>> The I see a kernel panic/dump and the system reboots after 15 seconds. >>> >>> If I remove one card, either one of the NVMe drives or the network >>> card, the system boots fine. Also, if in BIOS I set PnP OS to YES >>> then sometimes it boots (but not always). If I set PnP OS to NO, and >>> all three cards are installed, the system never boots. >>> >>> When the system boots OK I can see that the network card is reported >>> as 4 separate devices on one of the PCIe slots. I tried different >>> NVMe drives as well as changing which device is installed to which >>> slot but the result seems to be the same in any case. >>> >>> What may be the issue? Amount of power drawn by the hardware? Too >>> many devices not supported by the motherboard? Too many interrupts >>> for the FreeBSD kernel to handle? >> >> That would be my first suspicion. Either total power drawn off the >> power supply. Or total power drawn off the PCI[whichever it is] bus >> power leads. Check if any of the add-on cards have extra power port >> (many video cards do). Card likely will work without extra power >> connected to it, but connecting extra power on the card may solve your >> problem. Next: borrow more powerful power supply and see if that >> resolves the issue. Or temporarily disconnect everything else (like >> all hard drives), and boot with all three cards off live CD, and see >> if that doesn't crash, then it is marginally insufficient power supply. > > Thanks for the suggestion. The power supply was able to power two NVMe > disks and 6 spinning HDD disks without issues in another server. So the > total power should be fine. It may be the PCI bus power leads is causing > problems but then, two NVMe drives wouldn't take more than 5-9W and the > network card even less. PCI Express specification allows much more to be > drawn from each slot. In total the server shouldn't take more than 50-70W, > > I am not saying that it's not because of the power supply, but I think > it would be the least likely at this point. I will try with another > power supply when I find one. Another shot in the dark: some PCI-express slots may be "a pair", i.e. they can only take cards with the same number of signal lanes. Then you may have trouble if one of the cards is, say, x8 another is x4. System board ("motherboard") manual may shed light on this. Incidentally, PS powering successfully different machine not necessarily is also sufficient to power this one. As you said. Good luck! Valeri ++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++