From owner-freebsd-current@freebsd.org Mon May 4 18:12:02 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 811262D61EE for ; Mon, 4 May 2020 18:12:02 +0000 (UTC) (envelope-from SRS0=ZcGF=6S=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49G9t93RNPz4Wty; Mon, 4 May 2020 18:12:01 +0000 (UTC) (envelope-from SRS0=ZcGF=6S=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 5AFB828434; Mon, 4 May 2020 20:11:53 +0200 (CEST) Received: from illbsd.quip.test (ip-62-24-92-232.net.upcbroadband.cz [62.24.92.232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id D597C28416; Mon, 4 May 2020 20:11:47 +0200 (CEST) Subject: Re: PCIe NVME drives not detected on Dell R6515 From: Miroslav Lachman <000.fbsd@quip.cz> To: Scott Long Cc: Kurt Jaeger , Warner Losh , FreeBSD-Current References: <0F8BCB8C-DE60-4A34-A4D8-F1BB4B9F906A@samsco.org> <9EF043C1-FF8F-4997-B59A-EC3BF7D1CEEE@samsco.org> <31E8B2BE-BED2-4084-868D-32C48CB3CD6E@samsco.org> <573f5fab-1ef6-151f-18ca-58d3a4a89c72@quip.cz> <07B6763F-C23B-4B7C-B76A-26267AC35453@samsco.org> <20200417194431.GD39563@home.opsec.eu> <148dcdf7-f185-f14f-52ee-d4df3a2a1dc7@quip.cz> <8D8E1F62-AB66-47E1-8444-3D66F8EADA74@samsco.org> <015c7aa8-9385-4219-1bf1-0137f65ed80d@quip.cz> <90C35FEF-690C-4D04-A0D8-D3E5A448C744@samsco.org> <4736a46d-716d-7860-ff56-6c1d7391dbeb@quip.cz> Message-ID: <6e6461d1-fbc4-c306-f71f-54767f2849cb@quip.cz> Date: Mon, 4 May 2020 20:11:42 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 In-Reply-To: <4736a46d-716d-7860-ff56-6c1d7391dbeb@quip.cz> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 49G9t93RNPz4Wty X-Spamd-Bar: +++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of SRS0=ZcGF=6S=quip.cz=000.fbsd@elsa.codelab.cz has no SPF policy when checking 94.124.105.4) smtp.mailfrom=SRS0=ZcGF=6S=quip.cz=000.fbsd@elsa.codelab.cz X-Spamd-Result: default: False [3.87 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; IP_SCORE(0.81)[ip: (0.26), ipnet: 94.124.104.0/21(0.13), asn: 42000(3.56), country: CZ(0.09)]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[quip.cz]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.87)[0.870,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_SPAM_LONG(1.00)[0.996,0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[000.fbsd@quip.cz,SRS0=ZcGF=6S=quip.cz=000.fbsd@elsa.codelab.cz]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ]; FROM_NEQ_ENVFROM(0.00)[000.fbsd@quip.cz,SRS0=ZcGF=6S=quip.cz=000.fbsd@elsa.codelab.cz]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 May 2020 18:12:02 -0000 On 2020-04-27 08:02, Miroslav Lachman wrote: > I don't know what is with Scott. I hope he is well. > Is there somebody else who can help me with this issue? > Scott wrote there are hotplug PCIe buses not probed during boot process. > I am not a developer so I cannot move forward alone. The problem is with PCIe Hot Plug. Hot Plug bus was not enumerated thus no NVME detected. Dan Lukes suggested (privately) to disable hot plugging by this at second stage loader prompt: set hw.pci.enable_pcie_hp=0 Then I was able to boot FreeBSD installer ISO in BIOS mode (I don't know why but this machine is not able to boot FreeBSD ISO media in UEFI mode). Installer sees both NVME disks and installation was successful but it cannot boot - Dell R6515 in BIOS mode does not show NVME drives. Switching to UEFI boot shows disks but they didn't contained EFI partition boot code. When I modified the partitions layout (remove swap, ad efi partition and swap again) it is now able to boot FreeBSD 11.3 amd64 in UEFI mode from NVME disks with Hot Plug disabled in loader.conf. Can somebody look on to it why the bus is not probed when Hot Plug is enabled? I have a few days to run some tests on this HW before it will go in to production. Kind regards Miroslav Lachman