From owner-freebsd-questions@freebsd.org Sun Jan 14 16:18:11 2018 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 190ABE786BE; Sun, 14 Jan 2018 16:18:11 +0000 (UTC) (envelope-from m.e.sanliturk@gmail.com) Received: from mail-ot0-x236.google.com (mail-ot0-x236.google.com [IPv6:2607:f8b0:4003:c0f::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CE2238205A; Sun, 14 Jan 2018 16:18:10 +0000 (UTC) (envelope-from m.e.sanliturk@gmail.com) Received: by mail-ot0-x236.google.com with SMTP id 53so8824148otj.2; Sun, 14 Jan 2018 08:18:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=KsWqC5QJddch9AjI/Cx9DJO284czsu8Nll3HIyfKXbY=; b=gMYGVUk0Ujbf3so6HtVXZzH4Uhv7h9vFZm6j6D5dunxKeNndPxdyzRSFskSY6cikL9 qqbqvfKwDnoXnLzeioE/UfFesZed2r+EacKrPOS3C6orjpZWW+HXeAAbBhP3q5y/i5Cn cUtxb3N7sfsQT/TI1SulliJGLmAfMYjXNfvT8qBp/Y/0GAi9AYhsCXtrjBatiqplCO6S YJT8J6C4/Y66uaQeDvOwaDjXzr0oTTfOLbDAjweI8KwXqFRlfgtYOvZhSEUd2k/MJcTS BKy7gPGirBFFh+3Fp9KwVZpRA7Q/2m3c/klvxZJEv1kVfK4y37m5n5tuc9jsMA4CKe+V te4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=KsWqC5QJddch9AjI/Cx9DJO284czsu8Nll3HIyfKXbY=; b=CUL4pN7Ioy7bK4y8hef9RxkUdS+8BJuLGVXJvd536HMx8O6Td1Jcx2ajifnYTUqFkq WG+JvYtbX4VKXEBlTQSCHtBI4O8gvXKR4eTnQNK/rqjC6q20qtRkB6m8uFJqcVIUhxpP pN80Mw+l5nKEpp2002lk/Zr0D9JdJu9E6uRFTRN3RTYqjQI7BS+qlhgfXkGM4sNVpOur 9fD3hVAEkeI7YUVjSXy9VA0T/zsN033GTnEVSz1ouS38rPgYW9Ms7/vlrUxbuQ0UGhzG aMlNlElkKmH9EjUMkqg23ErHwLpcuChTwJsUTg95r7AcSo0aOsu6xevPMPSoHCtbgVQr zdQg== X-Gm-Message-State: AKwxytcM5UBF8UcwglrlPXktsVnOwZlGb22pKFeo+a0Dwpsal3q9GbW8 w4f/QYNDBbwPoCpH3lgZqWiycMrzFx7nE+KysKxnuA== X-Google-Smtp-Source: ACJfBovlv3K0zJazMWg9DWTu3iZCdlqHwm2xRWCPmXyf8nGcfJk75ZH3qpz+Bupu1wvLgAVYds4Zcb2pTetwkG2eK6U= X-Received: by 10.157.82.96 with SMTP id q32mr6745373otg.376.1515946690099; Sun, 14 Jan 2018 08:18:10 -0800 (PST) MIME-Version: 1.0 Received: by 10.157.0.2 with HTTP; Sun, 14 Jan 2018 08:18:09 -0800 (PST) In-Reply-To: <3d0ad00c-5214-71b0-017b-c2d5ba608e37@gjunka.com> References: <3d0ad00c-5214-71b0-017b-c2d5ba608e37@gjunka.com> From: Mehmet Erol Sanliturk Date: Sun, 14 Jan 2018 19:18:09 +0300 Message-ID: Subject: Re: Server doesn't boot when 3 PCIe slots are populated To: Grzegorz Junka Cc: FreeBSD Questions Mailing List , freebsd-drivers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Jan 2018 16:18:11 -0000 On Sun, Jan 14, 2018 at 5:46 PM, Grzegorz Junka wrote: > > On 13/01/2018 17:56, Mehmet Erol Sanliturk wrote: > >> >> >> On Sat, Jan 13, 2018 at 7:21 PM, Grzegorz Junka > > wrote: >> >> Hello, >> >> I am installing a FreeBSD server based on Supermicro H8SML-iF. >> There are three PCIe slots to which I installed 2 NVMe drives and >> one network card Intel I350-T4 (with 4 Ethernet slots). >> >> I am observing a strange behavior where the system doesn't boot if >> all three PCIe slots are populated. It shows this message: >> >> nvme0: mem 0xfd8fc000-0xfd8fffff irq 24 at >> device 0.0 on pci1 >> nvme0: controller ready did not become 1 within 30000 ms >> nvme0: did not complete shutdown within 5 seconds of notification >> >> The I see a kernel panic/dump and the system reboots after 15 seconds. >> >> If I remove one card, either one of the NVMe drives or the network >> card, the system boots fine. Also, if in BIOS I set PnP OS to YES >> then sometimes it boots (but not always). If I set PnP OS to NO, >> and all three cards are installed, the system never boots. >> >> When the system boots OK I can see that the network card is >> reported as 4 separate devices on one of the PCIe slots. I tried >> different NVMe drives as well as changing which device is >> installed to which slot but the result seems to be the same in any >> case. >> >> What may be the issue? Amount of power drawn by the hardware? Too >> many devices not supported by the motherboard? Too many interrupts >> for the FreeBSD kernel to handle? >> >> Any help would be greatly appreciated. >> >> GregJ >> >> _______________________________________________ >> >> >> >> >> >> From my experience from other trade marked main boards , an action may be >> to check manual of your server board to see whether there are rules about >> use of these slots : Sometimes differently shaped slots are supplied with >> same ports : If one slot is occupied , the other slot should be left open , >> or rules about not to insert such a kind of device into a slot , for >> example , graphic cards . >> >> >> Mehmet Erol Sanliturk >> >> > I checked the manual but couldn't find any restrictions regarding PCIe > ports. It only says how many lanes are available in each slot. Would there > be any obvious BIOS setting that could cause this issue? I tried after > resetting BIOS to default settings but maybe something is set incorrectly > by default? > > GregJ > _______________________________________________ > > http://www.supermicro.com/Aplus/motherboard/Opteron3000/SR56x0/H8SML-iF.cfm H8SML-iF On the above page , click "OS Compatibility" On the following page , click "SR5650" http://www.supermicro.com/Aplus/support/resources/OS/OS_Comp_SR5650.cfm OS Compatibility Chart On the column ( third ) H8SML-7F H8SML-7 H8SML-iF H8SML-i there listed only FreeBSD 8.0 FreeBSD 9.1 >From this list , it may be said that , this mother board date is old , means , it seems that the new OS versions are not tested after currently tested OS versions . To check interaction between operating system and your Supermicro H8SML-iF , select one of the suitable operating system ( Unix class OSes are more suitable ) for you and tested on this card , and try to install it as you like your installed components . If it boots successfully , it means that there is an incompatibility between your FreeBSD and the main board . If no one of them boots , then you may conclude that , there is a problem in your settings . BIOS settings are important , because , OS communicates with the main board through these settings . In manual ( downloaded from the above page : Manual Revision 1.0c Release Date: March 12, 2014 ) , page 4-9 , "PCI/PnP Configuration" is defined . If PnP is selected YES. OS adjusts some device settings . If NO is selected , BIOS adjusts some device settings . When BIOS adjusted device settings are not conforming to OS parameters , the result will be "FAIL" . Therefore , more suitable selection is YES . Another point is that , there are many more BIOS selectable parameters and jumpers about PCI slots and others . There are some BIOS settings for PCI slots : PCI X4 Slot 6 ( page 4-9 ) PCI x8 Slot 7 ( page 4-10 ) Please review these BIOS settings in your manual and set them with respect to your requirements . Mehmet Erol Sanliturk