From owner-dev-commits-src-branches@freebsd.org Wed Sep 1 20:57:33 2021 Return-Path: Delivered-To: dev-commits-src-branches@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AB45D671759; Wed, 1 Sep 2021 20:57:33 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4H0GbK4W6fz3sh2; Wed, 1 Sep 2021 20:57:33 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 7F68F41EE; Wed, 1 Sep 2021 20:57:33 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 181KvXrn047268; Wed, 1 Sep 2021 20:57:33 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 181KvXBv047267; Wed, 1 Sep 2021 20:57:33 GMT (envelope-from git) Date: Wed, 1 Sep 2021 20:57:33 GMT Message-Id: <202109012057.181KvXBv047267@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Alexander Motin Subject: git: 8ee90cc8d3f2 - stable/13 - pci_pci: Support growing windows in bus_adjust_resource for NEW_PCIB MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: mav X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 8ee90cc8d3f2c98e37effece00b998f46d0a3d82 Auto-Submitted: auto-generated X-BeenThere: dev-commits-src-branches@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commits to the stable branches of the FreeBSD src repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Sep 2021 20:57:33 -0000 The branch stable/13 has been updated by mav: URL: https://cgit.FreeBSD.org/src/commit/?id=8ee90cc8d3f2c98e37effece00b998f46d0a3d82 commit 8ee90cc8d3f2c98e37effece00b998f46d0a3d82 Author: Jessica Clarke AuthorDate: 2021-08-07 18:27:32 +0000 Commit: Alexander Motin CommitDate: 2021-09-01 20:55:13 +0000 pci_pci: Support growing windows in bus_adjust_resource for NEW_PCIB If we allocate a new window for a bridge rather than reusing an existing one set up by firmware to cover all the devices then the new window only includes the range needed for the first device to allocate the resource. If a request comes in to adjust this resource in order to extend a downstream window for another device then this will fail as the rman doesn't have any space, so we must first grow the bridge's own window. This is needed to support successfully attaching more than one PCI device on SiFive's HiFive Unmatched, which has the following topology: Root Port <---> Bridge <---> Bridge <-+-> Bridge <---> (Unused) (pcib0) (pcib1) (pcib2) | (pcib3) +-> Bridge <---> xHCI | (pcib4) +-> Bridge <---> M.2 E-key | (pcib5) +-> Bridge <---> M.2 M-key | (pcib6) +-> Bridge <---> x16 slot (pcib7) Without this, the xHCI endpoint successfully attaches but NVMe M.2 M-key endpoint fails to attach as, when its adjacent bridge (pcib6) attempts to allocate a window from its parent (pcib2) on the other side of the switch, its parent attempts to grow its own window by calling bus_adjust_resource on its own parent (pcib1) which fails to call the root port device (pcib0) to request more memory to grow its own window. Had the root port been directly connected to the switch without the bridge in the middle then the existing code would have worked, but the extra hop broke it. Reviewed by: jhb MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D31035 --- sys/dev/pci/pci_pci.c | 77 +++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 62 insertions(+), 15 deletions(-) diff --git a/sys/dev/pci/pci_pci.c b/sys/dev/pci/pci_pci.c index d6fbb06a61ac..83106eaa455b 100644 --- a/sys/dev/pci/pci_pci.c +++ b/sys/dev/pci/pci_pci.c @@ -148,6 +148,30 @@ static int pci_clear_pcib; SYSCTL_INT(_hw_pci, OID_AUTO, clear_pcib, CTLFLAG_RDTUN, &pci_clear_pcib, 0, "Clear firmware-assigned resources for PCI-PCI bridge I/O windows."); +/* + * Get the corresponding window if this resource from a child device was + * sub-allocated from one of our window resource managers. + */ +static struct pcib_window * +pcib_get_resource_window(struct pcib_softc *sc, int type, struct resource *r) +{ + switch (type) { + case SYS_RES_IOPORT: + if (rman_is_region_manager(r, &sc->io.rman)) + return (&sc->io); + break; + case SYS_RES_MEMORY: + /* Prefetchable resources may live in either memory rman. */ + if (rman_get_flags(r) & RF_PREFETCHABLE && + rman_is_region_manager(r, &sc->pmem.rman)) + return (&sc->pmem); + if (rman_is_region_manager(r, &sc->mem.rman)) + return (&sc->mem); + break; + } + return (NULL); +} + /* * Is a resource from a child device sub-allocated from one of our * resource managers? @@ -156,21 +180,11 @@ static int pcib_is_resource_managed(struct pcib_softc *sc, int type, struct resource *r) { - switch (type) { #ifdef PCI_RES_BUS - case PCI_RES_BUS: + if (type == PCI_RES_BUS) return (rman_is_region_manager(r, &sc->bus.rman)); #endif - case SYS_RES_IOPORT: - return (rman_is_region_manager(r, &sc->io.rman)); - case SYS_RES_MEMORY: - /* Prefetchable resources may live in either memory rman. */ - if (rman_get_flags(r) & RF_PREFETCHABLE && - rman_is_region_manager(r, &sc->pmem.rman)) - return (1); - return (rman_is_region_manager(r, &sc->mem.rman)); - } - return (0); + return (pcib_get_resource_window(sc, type, r) != NULL); } static int @@ -2330,11 +2344,44 @@ pcib_adjust_resource(device_t bus, device_t child, int type, struct resource *r, rman_res_t start, rman_res_t end) { struct pcib_softc *sc; + struct pcib_window *w; + int error; sc = device_get_softc(bus); - if (pcib_is_resource_managed(sc, type, r)) - return (rman_adjust_resource(r, start, end)); - return (bus_generic_adjust_resource(bus, child, type, r, start, end)); + + /* + * If the resource wasn't sub-allocated from one of our region + * managers then just pass the request up. + */ + if (!pcib_is_resource_managed(sc, type, r)) + return (bus_generic_adjust_resource(bus, child, type, r, + start, end)); + +#ifdef PCI_RES_BUS + if (type != PCI_RES_BUS) +#endif + { + /* + * Resource is managed and not a secondary bus number, must + * be from one of our windows. + */ + w = pcib_get_resource_window(sc, type, r); + KASSERT(w != NULL, + ("%s: no window for resource (%#jx-%#jx) type %d", + __func__, rman_get_start(r), rman_get_end(r), type)); + + /* + * If our window isn't big enough to grow the sub-allocation + * then we need to expand the window. + */ + if (start < w->base || end > w->limit) { + error = pcib_expand_window(sc, w, type, start, end); + if (error != 0) + return (error); + } + } + + return (rman_adjust_resource(r, start, end)); } int