From nobody Fri Mar 6 17:32:11 2026 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4fSD4J4rQqz6T7kR for ; Fri, 06 Mar 2026 17:32:16 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R12" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4fSD4J2xSqz3JSv for ; Fri, 06 Mar 2026 17:32:16 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1772818336; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=V+gGyl7+/BuByTo3n8k02p0/pOCIz6uhVYsYn2QMZDY=; b=DpgWqmEWurlKo+LCr94SpbJ5Cfn6VUS9F4Y1vsufia2nW4un2AJ9eHD/DqYOUoGwWO38py o+Ngbfyj/UstyEI2d1pOgpRK640knlK5VseQOzTcZH50AoUAdUcFkYeU5IYAy5E+Ui1BAc vDHTqIQ0nFvosCpDvriPBE6C4Hpnsm9V/vCIC1xwf8QnDbtq3GAm6hIw+PzD7+cjjZw97h uX6lBiodC90IE2PSdbijokvCGpOOSfrFdagYBNlhaqP/FD64ZDzjpniE/d+I/4Xz0YvN/e zWHk32URpeZfeg8SfsuFjkYnn9iMKkeVZJAAvdkShoFvwN2oSypyt4mrzzRUNQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1772818336; a=rsa-sha256; cv=none; b=GbGMg1NaDt8qlv5af8SF5HKfxM4TL5ym7zyYwzOGSC33xdo7aTs9CqlBWbXuNK5c2l1huT 5UBv05SN43qaxf9gg5ig5LDr0z9UN+v1D+UrE8eCipKRnSlKsCP80lnJvIMD6VXHc3F7kp HCnuEG7gNKy/YfCWytRcxwuoosEg4rInSZnuwlDDw6qSO+q4/20wFOwlu0g1eyVSgy0V7J KqDBp2eyPgYvp4hS9K+yHdqBGjSrKzubFi16Eg5yxSd4uRFYjWcefjE+uZ2pKRe4pfiit7 /7IvHPsqpdyPUsqvrFOJ7Jm2wjNHBzXoyW9HGx4FfGKBcqYys5/vpEn6UXM/3w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1772818336; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=V+gGyl7+/BuByTo3n8k02p0/pOCIz6uhVYsYn2QMZDY=; b=brsaheB+XJL5rF36Jp4zrB2oNVPcJ/JSdxCUctmUzP0KbR/yqKwiI39tVGmUqNfTg8D++X 2Djh17tsRoCmeFC+Ndb5hnl60rOyOmJTFzHEcq64MHZzSrBaVEHuOxjGwyX55WCwWefAi3 DMgQ5NnXjsMVjuWayxdGs51eveV4W4KAAqhpiT8c+Zwvsf6I7gzKyv1s4rc2K4618/LhED 7MRFfjUZSHq/0V8LJrv4Lm6X3s1wPeYFzKp7ZlAqjvTiZxPoetspQtabnpu+jPfwKGdQhm eUHNRii8j5TpFeZQrL9OiIsiqT4JAwwGOdRxAUc8VbVof9wJez/DMGFCOKy36g== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4fSD4J2ThRzY9j for ; Fri, 06 Mar 2026 17:32:16 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 18db4 by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Fri, 06 Mar 2026 17:32:11 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org Cc: Matt Delco From: Alexander Ziaee Subject: git: b3d9e5013f3e - main - nvme: Don't active memory space until all BARs are configured List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: ziaee X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: b3d9e5013f3e5016ffbd3d3d6091194658af2b92 Auto-Submitted: auto-generated Date: Fri, 06 Mar 2026 17:32:11 +0000 Message-Id: <69ab0f9b.18db4.5196b3f7@gitrepo.freebsd.org> The branch main has been updated by ziaee: URL: https://cgit.FreeBSD.org/src/commit/?id=b3d9e5013f3e5016ffbd3d3d6091194658af2b92 commit b3d9e5013f3e5016ffbd3d3d6091194658af2b92 Author: Matt Delco AuthorDate: 2026-03-06 17:23:03 +0000 Commit: Alexander Ziaee CommitDate: 2026-03-06 17:28:41 +0000 nvme: Don't active memory space until all BARs are configured In the current current behavior the 2nd and 3rd BARs can be activated when they're configured with address zero. This change defers the activation of all BARs until after they've all been configured with an address. This enables FreeBSD on Google Compute Engine C4-LSSD Machines. Sponsored by: Google Tested by: NetApp (previous version) Reviewed by: gallatin, imp Discussed with: jrtc27 (improved error reporting) Differential Revision: https://reviews.freebsd.org/D55541 --- sys/dev/nvme/nvme_pci.c | 44 +++++++++++++++++++++++++++++++++++++------- 1 file changed, 37 insertions(+), 7 deletions(-) diff --git a/sys/dev/nvme/nvme_pci.c b/sys/dev/nvme/nvme_pci.c index 5784c6d1be96..74191df52058 100644 --- a/sys/dev/nvme/nvme_pci.c +++ b/sys/dev/nvme/nvme_pci.c @@ -151,24 +151,28 @@ nvme_pci_probe (device_t device) static int nvme_ctrlr_allocate_bar(struct nvme_controller *ctrlr) { + int error; + ctrlr->resource_id = PCIR_BAR(0); ctrlr->msix_table_resource_id = -1; ctrlr->msix_table_resource = NULL; ctrlr->msix_pba_resource_id = -1; ctrlr->msix_pba_resource = NULL; + /* + * Using RF_ACTIVE will set the Memory Space bit in the PCI command register. + * The remaining BARs will get mapped in before they've been programmed with + * an address. To avoid this we'll not set this flag and instead call + * bus_activate_resource() after all the BARs have been programmed. + */ ctrlr->resource = bus_alloc_resource_any(ctrlr->dev, SYS_RES_MEMORY, - &ctrlr->resource_id, RF_ACTIVE); + &ctrlr->resource_id, 0); if (ctrlr->resource == NULL) { nvme_printf(ctrlr, "unable to allocate pci resource\n"); return (ENOMEM); } - ctrlr->bus_tag = rman_get_bustag(ctrlr->resource); - ctrlr->bus_handle = rman_get_bushandle(ctrlr->resource); - ctrlr->regs = (struct nvme_registers *)ctrlr->bus_handle; - /* * The NVMe spec allows for the MSI-X tables to be placed behind * BAR 4 and/or 5, separate from the control/doorbell registers. @@ -180,7 +184,7 @@ nvme_ctrlr_allocate_bar(struct nvme_controller *ctrlr) if (ctrlr->msix_table_resource_id >= 0 && ctrlr->msix_table_resource_id != ctrlr->resource_id) { ctrlr->msix_table_resource = bus_alloc_resource_any(ctrlr->dev, - SYS_RES_MEMORY, &ctrlr->msix_table_resource_id, RF_ACTIVE); + SYS_RES_MEMORY, &ctrlr->msix_table_resource_id, 0); if (ctrlr->msix_table_resource == NULL) { nvme_printf(ctrlr, "unable to allocate msi-x table resource\n"); return (ENOMEM); @@ -190,13 +194,39 @@ nvme_ctrlr_allocate_bar(struct nvme_controller *ctrlr) ctrlr->msix_pba_resource_id != ctrlr->resource_id && ctrlr->msix_pba_resource_id != ctrlr->msix_table_resource_id) { ctrlr->msix_pba_resource = bus_alloc_resource_any(ctrlr->dev, - SYS_RES_MEMORY, &ctrlr->msix_pba_resource_id, RF_ACTIVE); + SYS_RES_MEMORY, &ctrlr->msix_pba_resource_id, 0); if (ctrlr->msix_pba_resource == NULL) { nvme_printf(ctrlr, "unable to allocate msi-x pba resource\n"); return (ENOMEM); } } + error = bus_activate_resource(ctrlr->dev, ctrlr->resource); + if (error) { + nvme_printf(ctrlr, "unable to activate pci resource: %d\n", error); + return (error); + } + if (ctrlr->msix_table_resource != NULL) { + error = bus_activate_resource(ctrlr->dev, ctrlr->msix_table_resource); + if (error) { + nvme_printf(ctrlr, "unable to activate msi-x table resource: %d\n", + error); + return (error); + } + } + if (ctrlr->msix_pba_resource != NULL) { + error = bus_activate_resource(ctrlr->dev, ctrlr->msix_pba_resource); + if (error) { + nvme_printf(ctrlr, "unable to activate msi-x pba resource: %d\n", + error); + return (error); + } + } + + ctrlr->bus_tag = rman_get_bustag(ctrlr->resource); + ctrlr->bus_handle = rman_get_bushandle(ctrlr->resource); + ctrlr->regs = (struct nvme_registers *)ctrlr->bus_handle; + return (0); }