From owner-freebsd-drivers@freebsd.org Mon Jan 15 09:44:34 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E77EDE6D3BF; Mon, 15 Jan 2018 09:44:34 +0000 (UTC) (envelope-from m.e.sanliturk@gmail.com) Received: from mail-ot0-x22b.google.com (mail-ot0-x22b.google.com [IPv6:2607:f8b0:4003:c0f::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A577F69992; Mon, 15 Jan 2018 09:44:34 +0000 (UTC) (envelope-from m.e.sanliturk@gmail.com) Received: by mail-ot0-x22b.google.com with SMTP id r4so6312468oti.12; Mon, 15 Jan 2018 01:44:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=xJChbAZw+czmM6zsoXu3xJIyn72fbHSPMvojhv++0eU=; b=AOl4vVfpnyOjFd9xReJI6UtTo3BQaGe7EbQCIQHiN9WkAFK+mNwxZYAdfi1vvlJuS4 z0IApI+gnMz/SNMi66SfHWW1Dq6t8KZKjAXcC715XrSNbcdN6/CNjuB3Y7kHDNiPsK1l m3oNQje/sOJ4e0ZbDtKZDJLgw6MCe7Rskg3fgNtDIhQdsaBw58SpM7tf/uZcYIGuvZIm MjT77k3N0IvL7wDucHknC0gZjnGlp28Fk58eCh2UT/yqJqdANGIxgcd8b89Q2RYJt7Zz Wm3dIvdIDDZ9W8unlKlDhUcSjtxsaX5rebHGm+FwOQF74N2mQsMmPPnJOI6NaBWY9hr7 +nNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=xJChbAZw+czmM6zsoXu3xJIyn72fbHSPMvojhv++0eU=; b=HIngeHLIc4Pz6rIuYdJl5Iz8ziOqKgDgk5Q+vhZmIulxJX+0uveEYPyxHhtbXcC0mP xqWzHLdLCATx41RPpTyswY1ac/k7sN6kIAWa7jH5wSUQbiTMvO30OqgiTrg11vF7sDAg FLcPGZu82mU7dbYoCAy6LtxOlbcOELGgx+1PNqqkCtVIKVSxL7duyGP4wQIZ9RAw9qfF bfHkz5N+/Xo+YvrfPj2rp9hZHFFrdcQ5x+wZ0PWH6XqYxxP84umkF4kJHFPDBwz4UJx5 tIEYMyO2mdf4cJP2AVjYUZb2QKNs84JoY/jvyATiLJWL0K2NeTAl1IhugiF2hDwbSo1F pU0w== X-Gm-Message-State: AKwxytfn+iCmquY4hzFEJN06/Ezvqhixi3Plfl4xNW2fnYkcAohDUbAR G8oaRgQ/ONpUEK3l412y9t8F+mhpRKPW/6ceFJI= X-Google-Smtp-Source: ACJfBouAZQnlrFR3Cu+bVwl1dTlZC3D91ShLoWxj/1FHnlUwlRJqEiqXD9OeTBwqpDMtF4tgtzBHVQ3CvrDufX3O4lc= X-Received: by 10.157.74.76 with SMTP id d12mr15010220otj.231.1516009473882; Mon, 15 Jan 2018 01:44:33 -0800 (PST) MIME-Version: 1.0 Received: by 10.157.0.2 with HTTP; Mon, 15 Jan 2018 01:44:33 -0800 (PST) In-Reply-To: <0e582bdb-e1f9-438c-3da2-2bcdc950aab5@gjunka.com> References: <3d0ad00c-5214-71b0-017b-c2d5ba608e37@gjunka.com> <8df1e967-01e0-d3c2-e14c-64c7fc8c66b0@gjunka.com> <0e582bdb-e1f9-438c-3da2-2bcdc950aab5@gjunka.com> From: Mehmet Erol Sanliturk Date: Mon, 15 Jan 2018 12:44:33 +0300 Message-ID: Subject: Re: Server doesn't boot when 3 PCIe slots are populated To: Grzegorz Junka Cc: Warner Losh , FreeBSD Questions Mailing List , freebsd-drivers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Jan 2018 09:44:35 -0000 On Mon, Jan 15, 2018 at 9:44 AM, Grzegorz Junka wrote: > > On 15/01/2018 06:18, Warner Losh wrote: > >> >> >> On Jan 14, 2018 11:05 PM, "Grzegorz Junka" > list1@gjunka.com>> wrote: >> >> >> On 14/01/2018 16:18, Mehmet Erol Sanliturk wrote: >> >> >> >> On Sun, Jan 14, 2018 at 5:46 PM, Grzegorz Junka >> >> >> wrote: >> >> >> On 13/01/2018 17:56, Mehmet Erol Sanliturk wrote: >> >> >> >> On Sat, Jan 13, 2018 at 7:21 PM, Grzegorz Junka >> >> > >> >> >>> wrote: >> >> Hello, >> >> I am installing a FreeBSD server based on >> Supermicro H8SML-iF. >> There are three PCIe slots to which I installed 2 NVMe >> drives and >> one network card Intel I350-T4 (with 4 Ethernet >> slots). >> >> I am observing a strange behavior where the system >> doesn't >> boot if >> all three PCIe slots are populated. It shows this >> message: >> >> nvme0: mem >> 0xfd8fc000-0xfd8fffff irq >> 24 at >> device 0.0 on pci1 >> nvme0: controller ready did not become 1 within >> 30000 ms >> nvme0: did not complete shutdown within 5 seconds of >> notification >> >> The I see a kernel panic/dump and the system >> reboots after >> 15 seconds. >> >> If I remove one card, either one of the NVMe >> drives or the >> network >> card, the system boots fine. Also, if in BIOS I >> set PnP OS >> to YES >> then sometimes it boots (but not always). If I set >> PnP OS >> to NO, >> and all three cards are installed, the system >> never boots. >> >> When the system boots OK I can see that the >> network card is >> reported as 4 separate devices on one of the PCIe >> slots. I >> tried >> different NVMe drives as well as changing which >> device is >> installed to which slot but the result seems to be the >> same in any >> case. >> >> What may be the issue? Amount of power drawn by the >> hardware? Too >> many devices not supported by the motherboard? Too >> many >> interrupts >> for the FreeBSD kernel to handle? >> >> Any help would be greatly appreciated. >> >> GregJ >> >> _______________________________________________ >> >> >> >> >> >> From my experience from other trade marked main boards >> , an >> action may be to check manual of your server board to see >> whether there are rules about use of these slots : >> Sometimes >> differently shaped slots are supplied with same ports >> : If one >> slot is occupied , the other slot should be left open , or >> rules about not to insert such a kind of device into a >> slot , >> for example , graphic cards . >> >> >> Mehmet Erol Sanliturk >> >> >> I checked the manual but couldn't find any restrictions >> regarding >> PCIe ports. It only says how many lanes are available in each >> slot. Would there be any obvious BIOS setting that could cause >> this issue? I tried after resetting BIOS to default >> settings but >> maybe something is set incorrectly by default? >> >> GregJ >> _______________________________________________ >> >> >> >> >> >> http://www.supermicro.com/Aplus/motherboard/Opteron3000/SR56 >> x0/H8SML-iF.cfm >> > 6x0/H8SML-iF.cfm> >> H8SML-iF >> >> >> On the above page , click "OS Compatibility" >> >> >> On the following page , click "SR5650" >> >> http://www.supermicro.com/Aplus/support/resources/OS/OS_Comp >> _SR5650.cfm >> > p_SR5650.cfm> >> OS Compatibility Chart >> >> >> On the column ( third ) >> >> H8SML-7F >> H8SML-7 >> H8SML-iF >> H8SML-i >> >> >> there listed only * >> * >> ** >> * >> * >> * >> * >> >> FreeBSD 8.0 >> FreeBSD 9.1 >> >> From this list , it may be said that , this mother board date >> is old , means , it seems that the new OS versions are not >> tested after currently tested OS versions . >> >> >> To check interaction between operating system and your >> Supermicro H8SML-iF , select one of the suitable operating >> system ( Unix class OSes are more suitable ) for you and >> tested on this card , and try to install it as you like your >> installed components . If it boots successfully , it means >> that there is an incompatibility between your FreeBSD and the >> main board . If no one of them boots , then you may conclude >> that , there is a problem in your settings . >> >> >> BIOS settings are important , because , OS communicates with >> the main board through these settings . >> >> >> In manual ( downloaded from the above page : >> Manual Revision 1.0c >> Release Date: March 12, 2014 ) , page 4-9 , "PCI/PnP >> Configuration" is defined . >> If PnP is selected YES. OS adjusts some device settings . If >> NO is selected , BIOS adjusts some device settings . When BIOS >> adjusted device settings are not conforming to OS parameters , >> the result will be "FAIL" . >> >> Therefore , more suitable selection is YES . >> >> >> Another point is that , there are many more BIOS selectable >> parameters and jumpers about PCI slots and others . >> There are some BIOS settings for PCI slots : >> >> PCI X4 Slot 6 ( page 4-9 ) >> PCI x8 Slot 7 ( page 4-10 ) >> >> >> >> Please review these BIOS settings in your manual and set them >> with respect to your requirements . >> >> >> Thanks Mehmet for looking into this. It's an old motherboard but >> my point is that it boots fine when either: one NVMe and the >> network card, or both NVMe are installed, but not when all three >> are installed. How would that be related to FreeBSD compatibility? >> The chipset and all devices that I am trying to install are >> supported by FreeBSD 11.x. >> >> I just tried booting into a Debian live system and it also didn't >> enumerate NVMe drives properly. This means that it's not FreeBSD >> related and is no longer relevant for this list. I will try to >> play with BIOS settings to see if I can make it work that way. >> Thanks for all the help. >> >> >> >> Nvme drives are weird about power. I distrust the power estimate of 5-9w >> earlier in the thread... given the oddity with debian, it's not too crazy >> to think that. How far does FreeBSD boot though? >> >> > I tried with a different power supply but the outcome was exactly the > same. Sometimes FreeBSD boots fine but one of the NVMe drives is not > visible (i.e. dmesg grep shows only one NVMe). When it doesn't work it > boots up to the point of enumerating drives (SATA, USB, NVMe). Then it > stops at the first NVMe and reboots. > > The funny thing is that very often it's enough to pull out one of the > cards and put it back in. Then the system boots fine with all three cards. > I had that a few times. Once it's booted it works, I can restart the system > and it boots every time. As soon as I power off, unplug from the power > main, wait a few minutes and power it on again, the issue comes back - > can't boot as NVMe can't be enumerated. > > I though it might be caused by the hardware being too cold. I left the > server once overnight but it didn't boot up, it was trying and restarting > the whole night. > > GregJ > > > _______________________________________________ > > The above explanation brings mind to the "impedance mismatch in electronics" problem . ( Please search impedance mismatch in electronics impedance matching in electronics in Internet if you want explanations about them . ) When all of these cards are inserted into slots simultaneously , their accumulated electronic effect may distort behaviour of your mother board circuits or attached card circuit(s) . Therefore , if you can find another NVMe and/or network card , please test their effect . Such tests may be inconclusive because mother board circuits may be affected negatively from "properly" operating add on cards when they are inserted together . If it is feasible for you , you may use USB attached network card(s) to eliminate network card attachment . Or you may use a more capable one NVMe card instead of two smaller NVMe cards , or you may use only one of them , or/and select an SATA SSD . Such a choice would save your investment and produces a working server with a "little" loss when compared to "all" . Mehmet Erol Sanliturk