From owner-freebsd-drivers@freebsd.org Mon Jan 15 22:36:45 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 42DA6E7BEB0; Mon, 15 Jan 2018 22:36:45 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [88.98.225.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D82D473472; Mon, 15 Jan 2018 22:36:44 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from ultrabook.yoonka.com (pD9E429BE.dip0.t-ipconnect.de [217.228.41.190]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id w0FMafiT054057 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Mon, 15 Jan 2018 22:36:42 GMT (envelope-from list1@gjunka.com) X-Authentication-Warning: msa1.earth.yoonka.com: Host pD9E429BE.dip0.t-ipconnect.de [217.228.41.190] claimed to be ultrabook.yoonka.com Subject: Re: Server doesn't boot when 3 PCIe slots are populated Cc: freebsd-questions@freebsd.org, freebsd-drivers@freebsd.org References: <3d0ad00c-5214-71b0-017b-c2d5ba608e37@gjunka.com> <8df1e967-01e0-d3c2-e14c-64c7fc8c66b0@gjunka.com> <57933.108.68.169.115.1516035394.squirrel@cosmo.uchicago.edu> From: Grzegorz Junka Message-ID: Date: Mon, 15 Jan 2018 22:36:36 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <57933.108.68.169.115.1516035394.squirrel@cosmo.uchicago.edu> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-GB-large X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Jan 2018 22:36:45 -0000 [cut] On 15/01/2018 16:56, Valeri Galtsev wrote: > On Mon, January 15, 2018 12:18 am, Warner Losh wrote: >> >> Thanks Mehmet for looking into this. It's an old motherboard but my point >> is that it boots fine when either: one NVMe and the network card, or both >> NVMe are installed, but not when all three are installed. How would that >> be >> related to FreeBSD compatibility? The chipset and all devices that I am >> trying to install are supported by FreeBSD 11.x. >> >> I just tried booting into a Debian live system and it also didn't >> enumerate >> NVMe drives properly. This means that it's not FreeBSD related and is no >> longer relevant for this list. > This sounds to me as having something to do with allocation of PCI address > space to talk to devices. Many devices can alternatively use different > ranges of addresses, so more than one such device can be attached to the > same PCI bus. These two particular devices seem to not be successfully > negotiated to use different (not overlapping) ranges of addresses (in > presence of some particular third device). Maybe it only happens like that > in this particular system board ("motherboard"). I would try the same on > different machine. But it is likely that all these three devices do not > have non-overlapping address ranges. > > I hope, someone more knowledgeable that I will chime in. That was exactly my suspicion, that the network card somehow takes away some resources that otherwise would be available to the NVMe cards. Not sure if that's possible but would love to hear an informed opinion on this. Note, that as stated somewhere else, a live Debian showed the same issue - after booting from live USB only one NVMe and the network card were visible. The other NVMe was ignored. This would suggest either hardware or BIOS problems, so if I had some more information I could try to change some of the BIOS settings to see if that would help. GregJ > Valeri > >> I will try to play with BIOS settings to >> see >> if I can make it work that way. Thanks for all the help. >> >> >> >> Nvme drives are weird about power. I distrust the power estimate of 5-9w >> earlier in the thread... given the oddity with debian, it's not too crazy >> to think that. How far does FreeBSD boot though? >> >> Warner >>