From owner-freebsd-drivers@freebsd.org Sat Jan 13 16:21:33 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 41062E713D6; Sat, 13 Jan 2018 16:21:33 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [88.98.225.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 3A8B16B9BF; Sat, 13 Jan 2018 16:21:31 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from crayon2.yoonka.com (crayon2.yoonka.com [10.70.7.20]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id w0DGLOw9003337 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Sat, 13 Jan 2018 16:21:24 GMT (envelope-from list1@gjunka.com) To: freebsd-questions@freebsd.org, freebsd-drivers@freebsd.org From: Grzegorz Junka Subject: Server doesn't boot when 3 PCIe slots are populated Message-ID: Date: Sat, 13 Jan 2018 16:21:24 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-GB-large X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Jan 2018 16:21:33 -0000 Hello, I am installing a FreeBSD server based on Supermicro H8SML-iF. There are three PCIe slots to which I installed 2 NVMe drives and one network card Intel I350-T4 (with 4 Ethernet slots). I am observing a strange behavior where the system doesn't boot if all three PCIe slots are populated. It shows this message: nvme0: mem 0xfd8fc000-0xfd8fffff irq 24 at device 0.0 on pci1 nvme0: controller ready did not become 1 within 30000 ms nvme0: did not complete shutdown within 5 seconds of notification The I see a kernel panic/dump and the system reboots after 15 seconds. If I remove one card, either one of the NVMe drives or the network card, the system boots fine. Also, if in BIOS I set PnP OS to YES then sometimes it boots (but not always). If I set PnP OS to NO, and all three cards are installed, the system never boots. When the system boots OK I can see that the network card is reported as 4 separate devices on one of the PCIe slots. I tried different NVMe drives as well as changing which device is installed to which slot but the result seems to be the same in any case. What may be the issue? Amount of power drawn by the hardware? Too many devices not supported by the motherboard? Too many interrupts for the FreeBSD kernel to handle? Any help would be greatly appreciated. GregJ From owner-freebsd-drivers@freebsd.org Sat Jan 13 17:56:09 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 09340E762C2; Sat, 13 Jan 2018 17:56:09 +0000 (UTC) (envelope-from m.e.sanliturk@gmail.com) Received: from mail-ot0-x22c.google.com (mail-ot0-x22c.google.com [IPv6:2607:f8b0:4003:c0f::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BE8456F814; Sat, 13 Jan 2018 17:56:08 +0000 (UTC) (envelope-from m.e.sanliturk@gmail.com) Received: by mail-ot0-x22c.google.com with SMTP id a24so7607638otd.4; Sat, 13 Jan 2018 09:56:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=KujieS9t0pmshPCmRP9xZnOJBagSIWldVCM1+zesmDU=; b=rsfigNQnCQoqkCjITZ+CeeuNQy5u19g1kjRaChg8l+xMkgCVhg9tMPcXSXpOVqSt+N 7mGyRpqmh1EsenJB/5iFrxk5HAPW8T3Q2g2WkLiYd805+dZUKrntprDuV2uWZql4abUd yFWS1XnfLE9n2yfod497oZ3ScpDVz8FY7d1tnv4AMqwOtsqKps8H2pExnpya/1l3KmpU 7D65eacAXljvE1wSdtf4E4kV1OI1mszHEcT8WZsW/JXUt5YQJvqChPt5Qhp0Qta7nne8 /52anvR4tT9mOMsNaDO1H2DdqOLzPn4eO2Pu7h81YEh6bGZJP2t6ttN3l3lkW5684Mb3 FTAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=KujieS9t0pmshPCmRP9xZnOJBagSIWldVCM1+zesmDU=; b=UfZW/hUkd+qfaKiDjM8zBNm+0uVf2W4UadPd42SYLvBej1++Nm2EBskEEmyv+26dqU nuN5TAF0vUBxYHBXGClw1JmON3QjQocS64I41eCV6xCP8ylL02kZ/OX2vBWRH5fHC3JI n+YwDh1ZenEa/UboCZpkLbsVpuFij+um50mVCBZWqEeQAdK2rkpO+rClwvXZX28a4bMP adsRH8T5Hp7ihyXY+Q1jlSD11GsXmo9A23MnMSs2dylHoxyY48PKd3lUgXBL7VpwY+Zc bcx2GcKUuOgp+ljAFQBCjNMJWF/5DaTLZbWsU5XnOlaPA1O0+2xniGJVG0w83zkR4CAs t8CA== X-Gm-Message-State: AKwxytexeSBM/gIebGCjG0fB4L5PuNaw5T/xvoEE33azISwABNaSR2RF /BZZHii1wgPDKN9dIkQ8Lm8BbiZ4cdOwLAMHUAw= X-Google-Smtp-Source: ACJfBoue3RcqP8BGWIQRIyPBVyHBd8lhJHC97rE7YYJR3KQOddZNyX6uAyaMNImdE/ODgdqw+55wApQk88k+M4EVh+A= X-Received: by 10.157.55.246 with SMTP id x109mr3424593otb.266.1515866168011; Sat, 13 Jan 2018 09:56:08 -0800 (PST) MIME-Version: 1.0 Received: by 10.157.0.2 with HTTP; Sat, 13 Jan 2018 09:56:07 -0800 (PST) In-Reply-To: References: From: Mehmet Erol Sanliturk Date: Sat, 13 Jan 2018 20:56:07 +0300 Message-ID: Subject: Re: Server doesn't boot when 3 PCIe slots are populated To: Grzegorz Junka Cc: FreeBSD Questions Mailing List , freebsd-drivers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Jan 2018 17:56:09 -0000 On Sat, Jan 13, 2018 at 7:21 PM, Grzegorz Junka wrote: > Hello, > > I am installing a FreeBSD server based on Supermicro H8SML-iF. There are > three PCIe slots to which I installed 2 NVMe drives and one network card > Intel I350-T4 (with 4 Ethernet slots). > > I am observing a strange behavior where the system doesn't boot if all > three PCIe slots are populated. It shows this message: > > nvme0: mem 0xfd8fc000-0xfd8fffff irq 24 at device > 0.0 on pci1 > nvme0: controller ready did not become 1 within 30000 ms > nvme0: did not complete shutdown within 5 seconds of notification > > The I see a kernel panic/dump and the system reboots after 15 seconds. > > If I remove one card, either one of the NVMe drives or the network card, > the system boots fine. Also, if in BIOS I set PnP OS to YES then sometimes > it boots (but not always). If I set PnP OS to NO, and all three cards are > installed, the system never boots. > > When the system boots OK I can see that the network card is reported as 4 > separate devices on one of the PCIe slots. I tried different NVMe drives as > well as changing which device is installed to which slot but the result > seems to be the same in any case. > > What may be the issue? Amount of power drawn by the hardware? Too many > devices not supported by the motherboard? Too many interrupts for the > FreeBSD kernel to handle? > > Any help would be greatly appreciated. > > GregJ > > _______________________________________________ > > >From my experience from other trade marked main boards , an action may be to check manual of your server board to see whether there are rules about use of these slots : Sometimes differently shaped slots are supplied with same ports : If one slot is occupied , the other slot should be left open , or rules about not to insert such a kind of device into a slot , for example , graphic cards . Mehmet Erol Sanliturk From owner-freebsd-drivers@freebsd.org Sat Jan 13 18:31:46 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C95AAE7833F; Sat, 13 Jan 2018 18:31:46 +0000 (UTC) (envelope-from galtsev@kicp.uchicago.edu) Received: from kicp.uchicago.edu (kicp.uchicago.edu [128.135.20.70]) by mx1.freebsd.org (Postfix) with ESMTP id AE1F271298; Sat, 13 Jan 2018 18:31:46 +0000 (UTC) (envelope-from galtsev@kicp.uchicago.edu) Received: from point.uchicago.edu (point.uchicago.edu [128.135.52.6]) by kicp.uchicago.edu (Postfix) with ESMTP id 50C7771803D; Sat, 13 Jan 2018 12:31:40 -0600 (CST) Subject: Re: Server doesn't boot when 3 PCIe slots are populated To: Grzegorz Junka , freebsd-questions@freebsd.org, freebsd-drivers@freebsd.org References: From: Valeri Galtsev Message-ID: <061ccfb3-ee6a-71a7-3926-372bb17b3171@kicp.uchicago.edu> Date: Sat, 13 Jan 2018 12:31:37 -0600 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Jan 2018 18:31:46 -0000 On 01/13/18 10:21, Grzegorz Junka wrote: > Hello, > > I am installing a FreeBSD server based on Supermicro H8SML-iF. There are > three PCIe slots to which I installed 2 NVMe drives and one network card > Intel I350-T4 (with 4 Ethernet slots). > > I am observing a strange behavior where the system doesn't boot if all > three PCIe slots are populated. It shows this message: > > nvme0: mem 0xfd8fc000-0xfd8fffff irq 24 at device > 0.0 on pci1 > nvme0: controller ready did not become 1 within 30000 ms > nvme0: did not complete shutdown within 5 seconds of notification > > The I see a kernel panic/dump and the system reboots after 15 seconds. > > If I remove one card, either one of the NVMe drives or the network card, > the system boots fine. Also, if in BIOS I set PnP OS to YES then > sometimes it boots (but not always). If I set PnP OS to NO, and all > three cards are installed, the system never boots. > > When the system boots OK I can see that the network card is reported as > 4 separate devices on one of the PCIe slots. I tried different NVMe > drives as well as changing which device is installed to which slot but > the result seems to be the same in any case. > > What may be the issue? Amount of power drawn by the hardware? Too many > devices not supported by the motherboard? Too many interrupts for the > FreeBSD kernel to handle? That would be my first suspicion. Either total power drawn off the power supply. Or total power drawn off the PCI[whichever it is] bus power leads. Check if any of the add-on cards have extra power port (many video cards do). Card likely will work without extra power connected to it, but connecting extra power on the card may solve your problem. Next: borrow more powerful power supply and see if that resolves the issue. Or temporarily disconnect everything else (like all hard drives), and boot with all three cards off live CD, and see if that doesn't crash, then it is marginally insufficient power supply. I hope this helps. Valeri > > Any help would be greatly appreciated. > > GregJ > > _______________________________________________ > freebsd-questions@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to > "freebsd-questions-unsubscribe@freebsd.org" -- ++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++