Date: Mon, 15 Jan 2018 16:47:15 -0600 (CST) From: "Valeri Galtsev" <galtsev@kicp.uchicago.edu> To: "Grzegorz Junka" <list1@gjunka.com> Cc: freebsd-questions@freebsd.org, freebsd-drivers@freebsd.org Subject: Re: Server doesn't boot when 3 PCIe slots are populated Message-ID: <60974.108.68.169.115.1516056435.squirrel@cosmo.uchicago.edu> In-Reply-To: <d005c90f-2276-1934-e773-bc9493192eef@gjunka.com> References: <ecce3fa6-3909-0947-685c-8a412684e99c@gjunka.com> <CAOgwaMsf9zByJYhL3KqpUMW5qKAzQEHpDWcwejY-uK=9swWbUQ@mail.gmail.com> <3d0ad00c-5214-71b0-017b-c2d5ba608e37@gjunka.com> <CAOgwaMsOKrGfGNmRt-C9Skjssj8JPAtFpk8bwG9v55LmaWdoVw@mail.gmail.com> <8df1e967-01e0-d3c2-e14c-64c7fc8c66b0@gjunka.com> <CANCZdfqZ-dogHXBdoyMPLOPs_R-vD%2BwLM-r6sm6ypesd0Nvp4A@mail.gmail.com> <57933.108.68.169.115.1516035394.squirrel@cosmo.uchicago.edu> <d005c90f-2276-1934-e773-bc9493192eef@gjunka.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, January 15, 2018 4:36 pm, Grzegorz Junka wrote:
> [cut]
>
> On 15/01/2018 16:56, Valeri Galtsev wrote:
>> On Mon, January 15, 2018 12:18 am, Warner Losh wrote:
>>>
>>> Thanks Mehmet for looking into this. It's an old motherboard but my
>>> point
>>> is that it boots fine when either: one NVMe and the network card, or
>>> both
>>> NVMe are installed, but not when all three are installed. How would
>>> that
>>> be
>>> related to FreeBSD compatibility? The chipset and all devices that I am
>>> trying to install are supported by FreeBSD 11.x.
>>>
>>> I just tried booting into a Debian live system and it also didn't
>>> enumerate
>>> NVMe drives properly. This means that it's not FreeBSD related and is
>>> no
>>> longer relevant for this list.
>> This sounds to me as having something to do with allocation of PCI
>> address
>> space to talk to devices. Many devices can alternatively use different
>> ranges of addresses, so more than one such device can be attached to the
>> same PCI bus. These two particular devices seem to not be successfully
>> negotiated to use different (not overlapping) ranges of addresses (in
>> presence of some particular third device). Maybe it only happens like
>> that
>> in this particular system board ("motherboard"). I would try the same on
>> different machine. But it is likely that all these three devices do not
>> have non-overlapping address ranges.
>>
>> I hope, someone more knowledgeable that I will chime in.
>
> That was exactly my suspicion, that the network card somehow takes away
> some resources that otherwise would be available to the NVMe cards. Not
> sure if that's possible but would love to hear an informed opinion on
> this.
>
> Note, that as stated somewhere else, a live Debian showed the same issue
> - after booting from live USB only one NVMe and the network card were
> visible. The other NVMe was ignored.
Right. Taking as hypothesis inability to de-tangle these three devices I
would try to replace the most generic one: network card. Hopefully you
have access to some batch of these by different manufacturers with
different chipsets. See if you will find one that will play in the band
with your two NVMe-s.
Good luck!
Valeri
> This would suggest either hardware
> or BIOS problems, so if I had some more information I could try to
> change some of the BIOS settings to see if that would help.
>
> GregJ
>
>> Valeri
>>
>>> I will try to play with BIOS settings to
>>> see
>>> if I can make it work that way. Thanks for all the help.
>>>
>>>
>>>
>>> Nvme drives are weird about power. I distrust the power estimate of
>>> 5-9w
>>> earlier in the thread... given the oddity with debian, it's not too
>>> crazy
>>> to think that. How far does FreeBSD boot though?
>>>
>>> Warner
>>>
>
> _______________________________________________
> freebsd-questions@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to
> "freebsd-questions-unsubscribe@freebsd.org"
>
++++++++++++++++++++++++++++++++++++++++
Valeri Galtsev
Sr System Administrator
Department of Astronomy and Astrophysics
Kavli Institute for Cosmological Physics
University of Chicago
Phone: 773-702-4247
++++++++++++++++++++++++++++++++++++++++
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?60974.108.68.169.115.1516056435.squirrel>