From owner-dev-commits-src-all@freebsd.org Thu Jan 28 23:13:45 2021 Return-Path: Delivered-To: dev-commits-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6A1F34F05FB; Thu, 28 Jan 2021 23:13:45 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DRbr92WFBz4qmp; Thu, 28 Jan 2021 23:13:45 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 49071182FE; Thu, 28 Jan 2021 23:13:45 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 10SNDjSD000824; Thu, 28 Jan 2021 23:13:45 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 10SNDjEJ000823; Thu, 28 Jan 2021 23:13:45 GMT (envelope-from git) Date: Thu, 28 Jan 2021 23:13:45 GMT Message-Id: <202101282313.10SNDjEJ000823@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Doug Ambrisko Subject: git: 0c852bb9b928 - main - Add support for some more Intel VMD controllers. Some of the newer controller have a sparce bus space that can be figured out by probing the HW. This gives the starting bus number. When reading the PCI config. space behind the VMD controller, the offset of the starting bus needs to be subtracted from the bus being read. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: ambrisko X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 0c852bb9b9282b30fd047ac1de398358f33777f4 Auto-Submitted: auto-generated X-BeenThere: dev-commits-src-all@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commit messages for all branches of the src repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Jan 2021 23:13:45 -0000 The branch main has been updated by ambrisko: URL: https://cgit.FreeBSD.org/src/commit/?id=0c852bb9b9282b30fd047ac1de398358f33777f4 commit 0c852bb9b9282b30fd047ac1de398358f33777f4 Author: Doug Ambrisko AuthorDate: 2021-01-28 23:12:14 +0000 Commit: Doug Ambrisko CommitDate: 2021-01-28 23:12:14 +0000 Add support for some more Intel VMD controllers. Some of the newer controller have a sparce bus space that can be figured out by probing the HW. This gives the starting bus number. When reading the PCI config. space behind the VMD controller, the offset of the starting bus needs to be subtracted from the bus being read. Fixed a bug in which in which not all of the devices directly attached to the VMD controller would be probed. On my initial test HW, a switch was found at bus 0, slot 0 and function 0. All of the NVME drives were behind that switch. Now scan for all slots and functions attached to bus 0. If a something was found then run attach after the scan. On detach also go through all slots and functions on bus 0. Tested with device ID's: 0x201d & 0x9a0b Tested by: nc@ MFC after: 7 days PR: 252253 --- sys/dev/vmd/vmd.c | 75 ++++++++++++++++++++++++++++++++++++++++++++------- sys/dev/vmd/vmd.h | 1 + sys/dev/vmd/vmd_bus.c | 73 ++++++++++++++++++++++++++++++------------------- 3 files changed, 113 insertions(+), 36 deletions(-) diff --git a/sys/dev/vmd/vmd.c b/sys/dev/vmd/vmd.c index 616fd4c8373e..80eeda4639fa 100644 --- a/sys/dev/vmd/vmd.c +++ b/sys/dev/vmd/vmd.c @@ -61,16 +61,32 @@ struct vmd_type { u_int16_t vmd_vid; u_int16_t vmd_did; char *vmd_name; + int flags; +#define BUS_RESTRICT 1 }; #define INTEL_VENDOR_ID 0x8086 -#define INTEL_DEVICE_ID_VMD 0x201d -#define INTEL_DEVICE_ID_VMD2 0x28c0 +#define INTEL_DEVICE_ID_201d 0x201d +#define INTEL_DEVICE_ID_28c0 0x28c0 +#define INTEL_DEVICE_ID_467f 0x467f +#define INTEL_DEVICE_ID_4c3d 0x4c3d +#define INTEL_DEVICE_ID_9a0b 0x9a0b + +#define VMD_CAP 0x40 +#define VMD_BUS_RESTRICT 0x1 + +#define VMD_CONFIG 0x44 +#define VMD_BUS_START(x) ((x >> 8) & 0x3) + +#define VMD_LOCK 0x70 static struct vmd_type vmd_devs[] = { - { INTEL_VENDOR_ID, INTEL_DEVICE_ID_VMD, "Intel Volume Management Device" }, - { INTEL_VENDOR_ID, INTEL_DEVICE_ID_VMD2, "Intel Volume Management Device" }, - { 0, 0, NULL } + { INTEL_VENDOR_ID, INTEL_DEVICE_ID_201d, "Intel Volume Management Device", 0 }, + { INTEL_VENDOR_ID, INTEL_DEVICE_ID_28c0, "Intel Volume Management Device", BUS_RESTRICT }, + { INTEL_VENDOR_ID, INTEL_DEVICE_ID_467f, "Intel Volume Management Device", BUS_RESTRICT }, + { INTEL_VENDOR_ID, INTEL_DEVICE_ID_4c3d, "Intel Volume Management Device", BUS_RESTRICT }, + { INTEL_VENDOR_ID, INTEL_DEVICE_ID_9a0b, "Intel Volume Management Device", BUS_RESTRICT }, + { 0, 0, NULL, 0 } }; static int @@ -92,7 +108,7 @@ vmd_probe(device_t dev) t++; } -return (ENXIO); + return (ENXIO); } static void @@ -157,8 +173,12 @@ vmd_read_config(device_t dev, u_int b, u_int s, u_int f, u_int reg, int width) struct vmd_softc *sc; bus_addr_t offset; - offset = (b << 20) + (s << 15) + (f << 12) + reg; sc = device_get_softc(dev); + if (b < sc->vmd_bus_start) + return (0xffffffff); + + offset = ((b - sc->vmd_bus_start) << 20) + (s << 15) + (f << 12) + reg; + switch(width) { case 4: return (bus_space_read_4(sc->vmd_btag, sc->vmd_bhandle, @@ -183,8 +203,11 @@ vmd_write_config(device_t dev, u_int b, u_int s, u_int f, u_int reg, struct vmd_softc *sc; bus_addr_t offset; - offset = (b << 20) + (s << 15) + (f << 12) + reg; sc = device_get_softc(dev); + if (b < sc->vmd_bus_start) + return; + + offset = ((b - sc->vmd_bus_start) << 20) + (s << 15) + (f << 12) + reg; switch(width) { case 4: @@ -271,6 +294,8 @@ vmd_attach(device_t dev) { struct vmd_softc *sc; struct pcib_secbus *bus; + struct vmd_type *t; + uint16_t vid, did; uint32_t bar; int i, j, error; int rid, sec_reg; @@ -325,13 +350,45 @@ vmd_attach(device_t dev) pci_write_config(dev, PCIR_PRIBUS_2, pcib_get_bus(device_get_parent(dev)), 1); + t = vmd_devs; + vid = pci_get_vendor(dev); + did = pci_get_device(dev); + + sc->vmd_bus_start = 0; + while (t->vmd_name != NULL) { + if (vid == t->vmd_vid && + did == t->vmd_did) { + if (t->flags == BUS_RESTRICT) { + if (pci_read_config(dev, VMD_CAP, 2) & + VMD_BUS_RESTRICT) + switch (VMD_BUS_START(pci_read_config( + dev, VMD_CONFIG, 2))) { + case 1: + sc->vmd_bus_start = 128; + break; + case 2: + sc->vmd_bus_start = 224; + break; + case 3: + device_printf(dev, + "Unknown bug offset\n"); + goto fail; + break; + } + } + } + t++; + } + + device_printf(dev, "VMD bus starts at %d\n", sc->vmd_bus_start); + sec_reg = PCIR_SECBUS_1; bus = &sc->vmd_bus; bus->sub_reg = PCIR_SUBBUS_1; bus->sec = vmd_read_config(dev, b, s, f, sec_reg, 1); bus->sub = vmd_read_config(dev, b, s, f, bus->sub_reg, 1); bus->dev = dev; - bus->rman.rm_start = 0; + bus->rman.rm_start = sc->vmd_bus_start; bus->rman.rm_end = PCI_BUSMAX; bus->rman.rm_type = RMAN_ARRAY; snprintf(buf, sizeof(buf), "%s bus numbers", device_get_nameunit(dev)); diff --git a/sys/dev/vmd/vmd.h b/sys/dev/vmd/vmd.h index f86630d72c24..e51448c7733e 100644 --- a/sys/dev/vmd/vmd.h +++ b/sys/dev/vmd/vmd.h @@ -78,6 +78,7 @@ struct vmd_softc void *vmd_intr; struct vmd_irq *vmd_irq; int vmd_msix_count; + uint8_t vmd_bus_start; #ifdef TASK_QUEUE_INTR struct taskqueue *vmd_irq_tq; struct task vmd_irq_task; diff --git a/sys/dev/vmd/vmd_bus.c b/sys/dev/vmd/vmd_bus.c index 74a6faa812b1..5eed6176630f 100644 --- a/sys/dev/vmd/vmd_bus.c +++ b/sys/dev/vmd/vmd_bus.c @@ -71,32 +71,45 @@ vmd_bus_attach(device_t dev) struct pci_devinfo *dinfo; rman_res_t start, end; int b, s, f; + int found; sc = device_get_softc(device_get_parent(dev)); - /* Start at max PCI vmd_domain and work down */ - b = s = f = 0; - dinfo = pci_read_device(device_get_parent(dev), dev, - PCI_DOMAINMAX - device_get_unit(device_get_parent(dev)), - b, s, f); - if (dinfo == NULL) { - device_printf(dev, "Cannot allocate dinfo!\n"); - return (ENOENT); + /* + * Start at max PCI vmd_domain and work down. Only VMD + * starting bus is connect to VMD device directly. Scan al + * lslots and function connected to starting bus. + */ + + b = sc->vmd_bus_start; + + found = 0; + for (s = 0; s < PCI_SLOTMAX; s++) { + for (f = 0; f < PCI_FUNCMAX; f++) { + dinfo = pci_read_device(device_get_parent(dev), dev, + PCI_DOMAINMAX - device_get_unit( + device_get_parent(dev)), b, s, f); + if (dinfo != NULL) { + found = 1; + pci_add_child(dev, dinfo); + + start = rman_get_start(sc->vmd_regs_resource[1]); + end = rman_get_end(sc->vmd_regs_resource[1]); + resource_list_add_next(&dinfo->resources, + SYS_RES_MEMORY, start, end, end - start + 1); + + start = rman_get_start(sc->vmd_io_resource); + end = rman_get_end(sc->vmd_io_resource); + resource_list_add_next(&dinfo->resources, + SYS_RES_IOPORT, start, end, end - start + 1); + + } + } } - pci_add_child(dev, dinfo); - - start = rman_get_start(sc->vmd_regs_resource[1]); - end = rman_get_end(sc->vmd_regs_resource[1]); - resource_list_add_next(&dinfo->resources, SYS_RES_MEMORY, start, end, - end - start + 1); - - start = rman_get_start(sc->vmd_io_resource); - end = rman_get_end(sc->vmd_io_resource); - resource_list_add_next(&dinfo->resources, SYS_RES_IOPORT, start, end, - end - start + 1); - - bus_generic_attach(dev); + if (found) { + bus_generic_attach(dev); + } return (0); } @@ -104,17 +117,23 @@ vmd_bus_attach(device_t dev) static int vmd_bus_detach(device_t dev) { + struct vmd_softc *sc; struct pci_devinfo *dinfo; int b, s, f; device_delete_children(dev); - b = s = f = 0; - dinfo = pci_read_device(device_get_parent(dev), dev, - PCI_DOMAINMAX - device_get_unit(device_get_parent(dev)), - b, s, f); - if (dinfo == NULL) { - resource_list_free(&dinfo->resources); + sc = device_get_softc(device_get_parent(dev)); + b = sc->vmd_bus_start; + + for (s = 0; s < PCI_SLOTMAX; s++) { + for (f = 0; f < PCI_FUNCMAX; f++) { + dinfo = pci_read_device(device_get_parent(dev), dev, + PCI_DOMAINMAX - device_get_unit( + device_get_parent(dev)), b, s, f); + if (dinfo != NULL) + resource_list_free(&dinfo->resources); + } } return (0); }