From owner-freebsd-current@freebsd.org Fri Apr 17 16:16:03 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EC77C2BF659 for ; Fri, 17 Apr 2020 16:16:03 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qk1-x733.google.com (mail-qk1-x733.google.com [IPv6:2607:f8b0:4864:20::733]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 493h6C0vgLz4BsQ for ; Fri, 17 Apr 2020 16:16:02 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qk1-x733.google.com with SMTP id l25so2988696qkk.3 for ; Fri, 17 Apr 2020 09:16:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=VqMT/IvR3O/9fZUr2HrVRpfhKCPV00wrPCJrNiQio1A=; b=kszmCuyh3WY1zxGcal+FswMbzIMhz8hocLCMRVbnsu425x0Ha7vwW6syGO+k1t3xIM I0SSOpCaJFvuEdgYXESewOzBq2dxX66gVhkq4P1ywUC2v+XmxZ/+49/2D4Df8x/xwTLx iRsKNreD3ao/zOa5y8TP7FbGFan3/aVannw9Z0cIniUONpWmWwUhZY6Sxb5LBXt06xJ0 bM2xxo9xbheCsJSdXANLGwE5SzPDmv2NUQZ0YtsJUX7Mutumpqk+ifnQqN+pphKNahow SbI5PZ6O8r/UfB0vp6LiMRXOnxC6BU0yPnOuffk4vtXlhTRbkpT2qXJ4pJbUFDDYOpit FQfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=VqMT/IvR3O/9fZUr2HrVRpfhKCPV00wrPCJrNiQio1A=; b=FvYg7dNJHftaoDeFOMhp4YcYGHyRvid7PHXLkuuHYRDykReEgAMEVZPlUW4Nku6gHw J6RqCLojBwcLxNyE5Sw6CZ5lnO7ELXpvFynfRQOU2dHSqAkZZ8YaxHdskJWQgCNlVWNb i0U04yfoAcyYxgglBX9e9eY60dFi3qRNUJ/L+Iwl4AlAP+TOHf8ndiWWQ0dvk5IImmlH FXUEwf+yGKrhIKL8OFHzQ2+Nt1BpSy70Arqr1tlRAQGlbVyd9fsF3WscV2ygwMUu8D8U A8VbLecCHjbmxnfqgOT3BfIDJ252jnsoH3zzS6kUTSl0yjjyppTz32jEdREJ70zb8kk0 MH9A== X-Gm-Message-State: AGi0PuZ7WfCZo2yO71hZVDFm3YVA6OWsiyyPDCLicYLEr9Ou46N223OH jry+RoOu0P3SudKnLge4Go0GmH7wL1rjILZJBaX4+uEvcHc= X-Google-Smtp-Source: APiQypK+RET8k2+flxvOfqufy2gdhdkZO+Mw7/h/zvc0PGTopsZ0voEzDfjWsp8qajP4ToWm1BXLhk8jtmyfV7oGdpo= X-Received: by 2002:a37:6ce:: with SMTP id 197mr3861424qkg.495.1587140161975; Fri, 17 Apr 2020 09:16:01 -0700 (PDT) MIME-Version: 1.0 References: <0F8BCB8C-DE60-4A34-A4D8-F1BB4B9F906A@samsco.org> <9EF043C1-FF8F-4997-B59A-EC3BF7D1CEEE@samsco.org> In-Reply-To: <9EF043C1-FF8F-4997-B59A-EC3BF7D1CEEE@samsco.org> From: Warner Losh Date: Fri, 17 Apr 2020 10:15:50 -0600 Message-ID: Subject: Re: PCIe NVME drives not detected on Dell R6515 To: Scott Long Cc: Miroslav Lachman <000.fbsd@quip.cz>, FreeBSD-Current X-Rspamd-Queue-Id: 493h6C0vgLz4BsQ X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=kszmCuyh; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::733) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-3.98 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCVD_IN_DNSWL_NONE(0.00)[3.3.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-1.98)[ip: (-9.09), ipnet: 2607:f8b0::/32(-0.33), asn: 15169(-0.43), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Apr 2020 16:16:04 -0000 No. It was some kind of extra card (Perteq?) managing things in a Dell server. It may be the VMD/VROC stuff, and that driver may be worth a shot, but I'm thinking not. It looks like Doug's code for that is in the tree, though https://reviews.freebsd.org/rS353380. Or are other changes needed? I've been blessed with either being able to turn this off, or not having to deal... Warner On Fri, Apr 17, 2020 at 9:54 AM Scott Long wrote: > Would that be the Intel VMD/VROC stuff? If so, there=E2=80=99s a driver = for > FreeBSD, but it=E2=80=99s not well tested yet. Will have to dig in furth= er. > > Scott > > > > On Apr 17, 2020, at 9:50 AM, Warner Losh wrote: > > > > > > > > On Fri, Apr 17, 2020 at 9:39 AM Scott Long wrote: > > Can you send me the output of =E2=80=98pciconf -llv=E2=80=99, either in= 12-STABLE or > 13-CURRENT? Also, can you send me the output of =E2=80=98dmesg=E2=80=99? > > > > There was another thread that said there was a raid card in the way... > It would be cool to find a way to get it out of the way... :) > > > > Warner > > > > Thanks, > > Scott > > > > > > > On Apr 17, 2020, at 5:23 AM, Miroslav Lachman <000.fbsd@quip.cz> > wrote: > > > > > > I already asked on stable@ but as I tried it on 13-CURRENT with the > same result I am trying to ask for help here. > > > > > > I have rented dedicated server Dell PowerEdge R6515 with iDRAC access > only. > > > There are 2 NVME drives which I wanted to use as ZFS root pool. > > > > > > They are shown in iDRAC > > > > > > Device Description: PCIe SSD in Slot 1 in Bay 1 > > > Device Protocol: NVMe-MI1.0 > > > Model: Dell Express Flash NVMe P4510 1TB SFF > > > Bus: 130 > > > Manufacturer: INTEL > > > Product ID: a54 > > > Revision: VDV1DP23 > > > Enclosure: PCIe SSD Backplane 1 > > > > > > > > > pciconf -l show many things, some of them are named "noneN@pci..." > but none "nvme" > > > > > > The is printscreen (12.1 but 13-CURRENT is the same) > > > > > > https://ibb.co/tPnymL7 > > > > > > But I booted Linux SystemRescueCd and nvme devices are there visible > in /dev/ > > > https://ibb.co/sj22Nwg > > > > > > There is verbose output of Linux lspci https://ibb.co/dPZTwV1 > > > > > > Linux dmesg contains: > > > nvme nvme0: pci function 0000:81:00.0 > > > nvme nvme1: pci function 0000:82:00.0 > > > nvme nvme0: 32/0/0 default/read/poll queues > > > nvme nvme1: 32/0/0 default/read/poll queues > > > > > > > > > The machine is Dell PowerEdge R6515 with AMD EPYC 7302P > > > > > > > > > More details extracted from iDRAC web interface > > > > > > I found this informations > > > > > > PCIe SSD in Slot 1 in Bay 1 > > > Bus: 82 > > > BusProtocol: PCIE > > > Device: 0 > > > DeviceDescription: PCIe SSD in Slot 1 in Bay 1 > > > DeviceProtocol: NVMe-MI1.0 > > > DeviceType: PCIeSSD > > > DriveFormFactor: 2.5 inch > > > FailurePredicted: NO > > > FQDD: Disk.Bay.1:Enclosure.Internal.0-1 > > > FreeSizeInBytes: Information Not Available > > > Function: 0 > > > HotSpareStatus: Information Not Available > > > InstanceID: Disk.Bay.1:Enclosure.Internal.0-1 > > > Manufacturer: INTEL > > > MaximumCapableSpeed: 8 GT/s > > > MediaType: Solid State Drive > > > Model: Dell Express Flash NVMe P4510 1TB SFF > > > NegotiatedSpeed: 8 GT/s > > > PCIeCapableLinkWidth: x4 > > > PCIeNegotiatedLinkWidth: x4 > > > PrimaryStatus: Ok > > > ProductID: a54 > > > RaidStatus: Information Not Available > > > RAIDType: Unknown > > > RemainingRatedWriteEndurance: 100 % > > > Revision: VDV1DP23 > > > SerialNumber: PHLJxxxxxxWF1PxxxxN > > > SizeInBytes: 1000204886016 > > > Slot: 1 > > > State: Ready > > > SystemEraseCapability: CryptographicErasePD > > > > > > PCIe SSD in Slot 1 in Bay 1 - PCI Device > > > BusNumber: 130 > > > DataBusWidth: 4x or x4 > > > Description: Express Flash NVMe 1.0 TB 2.5" U.2 (P4510) > > > DeviceDescription: PCIe SSD in Slot 1 in Bay 1 > > > DeviceNumber: 0 > > > DeviceType: PCIDevice > > > FQDD: Disk.Bay.1:Enclosure.Internal.0-1 > > > FunctionNumber: 0 > > > InstanceID: Disk.Bay.1:Enclosure.Internal.0-1 > > > LastSystemInventoryTime: 2020-04-17T03:21:39 > > > LastUpdateTime: 2020-03-31T13:55:06 > > > Manufacturer: Intel Corporation > > > PCIDeviceID: 0A54 > > > PCISubDeviceID: 2003 > > > PCISubVendorID: 1028 > > > PCIVendorID: 8086 > > > SlotLength: 2.5 Inch Drive Form Factor > > > SlotType: PCI Express Gen 3 SFF-8639 > > > > > > > > > Can anybody shed some light what the real problem is? > > > > > > Is the hardware not properly detected or is the driver completely > missing? > > > > > > NVME PCIe architecture is out of my knowledge. > > > > > > I really appreciate any help. > > > > > > Kind regards > > > Miroslav Lachman > > > _______________________________________________ > > > freebsd-current@freebsd.org mailing list > > > https://lists.freebsd.org/mailman/listinfo/freebsd-current > > > To unsubscribe, send any mail to " > freebsd-current-unsubscribe@freebsd.org" > > > > _______________________________________________ > > freebsd-current@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-current > > To unsubscribe, send any mail to " > freebsd-current-unsubscribe@freebsd.org" > >