From nobody Mon Apr 13 19:54:49 2026 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4fvdRM0qzSz6ZdBB for ; Mon, 13 Apr 2026 19:54:55 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R12" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4fvdRL5JGRz3nDy for ; Mon, 13 Apr 2026 19:54:54 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1776110094; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=fNrvrZ8xNVhwd1TWsua2Tyx8kDTZoPVRucs6/QX1L68=; b=GLhqTL4PKltI51y1d0UGZnVgE9pbfFsZCRiagqgWFFPowGEgesyNHMkXLCnZruzBo4l0P4 W7xd0CwPfmPdSUb/wH7Op3dACxZrVAAC9jX2ajpA7oixSV1fu1Lpdx50tnOC5DkrSzN4lP yYbymeJZkp+yNrJI5joQrjmJGgl7d1e5cCPkgQ/atfHVZM0fsjBV7ydbJp8QkptV7SMMmP I+ksqmKFFAxP8cAltDsQDv7b91rlfOTIiwEfmntMh5I7hESFK6uRrrkpHZQJQu1SU2Z6Sa 34jKtOJF8iGIQww+nPnHX8ZrkkxCx4ePYKDUXnMuCeWp1smkoQpMhqKnwje9EQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1776110094; a=rsa-sha256; cv=none; b=Q3r4Syyetr80J1U7f7k2xwxxRXDG+b0s/UXqajVvclVGEeC5dXOowZfP0B5gaPTe4VMVye bmjfz+ekhGJmtzuIsHGV4oCTh39JEF3r4fNVhQvmULKs3aRrjx0raMHVHtu6WxaWe0wvze W5HeBmCvU+N2Mn3UzW4ijgtXm4CJiRXO08Q0ukJnH/wS9YOZ70Cni0wkvATHS3XweBT/Wh 2s4jzizBCFO+p3rLLHLITlR7NB894LDDxWlylqX7YxV3gJGwBlhnIECkDm/eAdBLFOb5UD QOmTD7+11Boz3ydCPM2bKXol9VO/vUpjw89MekrpARNl+G5RbJtOzilRgf7JvQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1776110094; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=fNrvrZ8xNVhwd1TWsua2Tyx8kDTZoPVRucs6/QX1L68=; b=xeMcl2Ow+mhyGdclmWCX1CeqDeWPbgejdsWXitLs6iJ/T7W+2FJZg813pjQxulBiO4S+e9 vXfkFIsaTEe+Md2b6g0IHmUz5fI+30Lr6idUptkHYbdW1Hfo4+vYODcaSUmzs94lFLKEnJ 1sNgQvsHBX1xJhdywrpw8/4Jgw7H70q9QqfNkB3CtEJniroN6wfnYdpPVjQPIVno32AOra 1oJLGP/CjXk2y/ktUF0CV+mksZRZ6otpM5NtdwLvjyC9w4Bc7dNEM2WdTgZ9eHLltTv9Zi zXL/kNS7uPV6kbWr2oqJBHUHiV+d0Xp8c3z3K14OUhXuS1AkW/E4eHUPp4cxGg== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4fvdRL4MyRz37B for ; Mon, 13 Apr 2026 19:54:54 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 407fa by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Mon, 13 Apr 2026 19:54:49 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org Cc: Matt Delco From: Alexander Ziaee Subject: git: 925387f94432 - stable/15 - nvme: Don't active memory space until all BARs are configured List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: ziaee X-Git-Repository: src X-Git-Refname: refs/heads/stable/15 X-Git-Reftype: branch X-Git-Commit: 925387f94432ce80daf9c7ab98e856df2d13657a Auto-Submitted: auto-generated Date: Mon, 13 Apr 2026 19:54:49 +0000 Message-Id: <69dd4a09.407fa.ea754fd@gitrepo.freebsd.org> The branch stable/15 has been updated by ziaee: URL: https://cgit.FreeBSD.org/src/commit/?id=925387f94432ce80daf9c7ab98e856df2d13657a commit 925387f94432ce80daf9c7ab98e856df2d13657a Author: Matt Delco AuthorDate: 2026-03-06 17:23:03 +0000 Commit: Alexander Ziaee CommitDate: 2026-04-13 19:54:16 +0000 nvme: Don't active memory space until all BARs are configured In the current current behavior the 2nd and 3rd BARs can be activated when they're configured with address zero. This change defers the activation of all BARs until after they've all been configured with an address. This enables FreeBSD on Google Compute Engine C4-LSSD Machines. Sponsored by: Google Tested by: NetApp (previous version) Reviewed by: gallatin, imp Discussed with: jrtc27 (improved error reporting) Differential Revision: https://reviews.freebsd.org/D55541 (cherry picked from commit b3d9e5013f3e5016ffbd3d3d6091194658af2b92) --- sys/dev/nvme/nvme_pci.c | 44 +++++++++++++++++++++++++++++++++++++------- 1 file changed, 37 insertions(+), 7 deletions(-) diff --git a/sys/dev/nvme/nvme_pci.c b/sys/dev/nvme/nvme_pci.c index cecb05ca0a92..8e5e64e1f0e0 100644 --- a/sys/dev/nvme/nvme_pci.c +++ b/sys/dev/nvme/nvme_pci.c @@ -151,24 +151,28 @@ nvme_pci_probe (device_t device) static int nvme_ctrlr_allocate_bar(struct nvme_controller *ctrlr) { + int error; + ctrlr->resource_id = PCIR_BAR(0); ctrlr->msix_table_resource_id = -1; ctrlr->msix_table_resource = NULL; ctrlr->msix_pba_resource_id = -1; ctrlr->msix_pba_resource = NULL; + /* + * Using RF_ACTIVE will set the Memory Space bit in the PCI command register. + * The remaining BARs will get mapped in before they've been programmed with + * an address. To avoid this we'll not set this flag and instead call + * bus_activate_resource() after all the BARs have been programmed. + */ ctrlr->resource = bus_alloc_resource_any(ctrlr->dev, SYS_RES_MEMORY, - &ctrlr->resource_id, RF_ACTIVE); + &ctrlr->resource_id, 0); if (ctrlr->resource == NULL) { nvme_printf(ctrlr, "unable to allocate pci resource\n"); return (ENOMEM); } - ctrlr->bus_tag = rman_get_bustag(ctrlr->resource); - ctrlr->bus_handle = rman_get_bushandle(ctrlr->resource); - ctrlr->regs = (struct nvme_registers *)ctrlr->bus_handle; - /* * The NVMe spec allows for the MSI-X tables to be placed behind * BAR 4 and/or 5, separate from the control/doorbell registers. @@ -180,7 +184,7 @@ nvme_ctrlr_allocate_bar(struct nvme_controller *ctrlr) if (ctrlr->msix_table_resource_id >= 0 && ctrlr->msix_table_resource_id != ctrlr->resource_id) { ctrlr->msix_table_resource = bus_alloc_resource_any(ctrlr->dev, - SYS_RES_MEMORY, &ctrlr->msix_table_resource_id, RF_ACTIVE); + SYS_RES_MEMORY, &ctrlr->msix_table_resource_id, 0); if (ctrlr->msix_table_resource == NULL) { nvme_printf(ctrlr, "unable to allocate msi-x table resource\n"); return (ENOMEM); @@ -190,13 +194,39 @@ nvme_ctrlr_allocate_bar(struct nvme_controller *ctrlr) ctrlr->msix_pba_resource_id != ctrlr->resource_id && ctrlr->msix_pba_resource_id != ctrlr->msix_table_resource_id) { ctrlr->msix_pba_resource = bus_alloc_resource_any(ctrlr->dev, - SYS_RES_MEMORY, &ctrlr->msix_pba_resource_id, RF_ACTIVE); + SYS_RES_MEMORY, &ctrlr->msix_pba_resource_id, 0); if (ctrlr->msix_pba_resource == NULL) { nvme_printf(ctrlr, "unable to allocate msi-x pba resource\n"); return (ENOMEM); } } + error = bus_activate_resource(ctrlr->dev, ctrlr->resource); + if (error) { + nvme_printf(ctrlr, "unable to activate pci resource: %d\n", error); + return (error); + } + if (ctrlr->msix_table_resource != NULL) { + error = bus_activate_resource(ctrlr->dev, ctrlr->msix_table_resource); + if (error) { + nvme_printf(ctrlr, "unable to activate msi-x table resource: %d\n", + error); + return (error); + } + } + if (ctrlr->msix_pba_resource != NULL) { + error = bus_activate_resource(ctrlr->dev, ctrlr->msix_pba_resource); + if (error) { + nvme_printf(ctrlr, "unable to activate msi-x pba resource: %d\n", + error); + return (error); + } + } + + ctrlr->bus_tag = rman_get_bustag(ctrlr->resource); + ctrlr->bus_handle = rman_get_bushandle(ctrlr->resource); + ctrlr->regs = (struct nvme_registers *)ctrlr->bus_handle; + return (0); }