From owner-dev-commits-src-branches@freebsd.org Mon Feb 22 16:30:12 2021 Return-Path: Delivered-To: dev-commits-src-branches@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 091BA5445AC; Mon, 22 Feb 2021 16:30:12 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Dknhz6sgyz3sgV; Mon, 22 Feb 2021 16:30:11 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id DE7A5953; Mon, 22 Feb 2021 16:30:11 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 11MGUBcl019533; Mon, 22 Feb 2021 16:30:11 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 11MGUBqS019527; Mon, 22 Feb 2021 16:30:11 GMT (envelope-from git) Date: Mon, 22 Feb 2021 16:30:11 GMT Message-Id: <202102221630.11MGUBqS019527@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Marcin Wojtas Subject: git: 5de955fe436a - releng/13.0 - MFC 1c808fcd859f: Allocate BAR for ENA MSIx vector table MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: mw X-Git-Repository: src X-Git-Refname: refs/heads/releng/13.0 X-Git-Reftype: branch X-Git-Commit: 5de955fe436a208f93359e84a059b74bb0169ce6 Auto-Submitted: auto-generated X-BeenThere: dev-commits-src-branches@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commits to the stable branches of the FreeBSD src repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Feb 2021 16:30:12 -0000 The branch releng/13.0 has been updated by mw: URL: https://cgit.FreeBSD.org/src/commit/?id=5de955fe436a208f93359e84a059b74bb0169ce6 commit 5de955fe436a208f93359e84a059b74bb0169ce6 Author: Michal Krawczyk AuthorDate: 2021-02-18 09:00:58 +0000 Commit: Marcin Wojtas CommitDate: 2021-02-22 16:29:40 +0000 MFC 1c808fcd859f: Allocate BAR for ENA MSIx vector table In the new ENA-based instances like c6gn, the vector table moved to a new PCIe bar - BAR1. Previously it was always located on the BAR0, so the resources were already allocated together with the registers. As the FreeBSD isn't doing any resource allocation behind the scenes, the driver is responsible to allocate them explicitly, before other parts of the OS (like the PCI code allocating MSIx) will be able to access them. To determine dynamically BAR on which the MSIx vector table is present the pci_msix_table_bar() is being used and the new BAR is allocated if needed. Approved by: re (gjb) Submitted by: Michal Krawczyk Obtained from: Semihalf Sponsored by: Amazon, Inc (cherry picked from commit 1c808fcd859f5ce24132a903a4c7c9996e0513b1) (cherry picked from commit e540e45097e54aaae553ea12468801ceab473d47) --- sys/dev/ena/ena.c | 21 +++++++++++++++++++++ sys/dev/ena/ena.h | 4 +++- 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/sys/dev/ena/ena.c b/sys/dev/ena/ena.c index b0e05f23b563..680eb0e9d049 100644 --- a/sys/dev/ena/ena.c +++ b/sys/dev/ena/ena.c @@ -299,6 +299,11 @@ ena_free_pci_resources(struct ena_adapter *adapter) bus_release_resource(pdev, SYS_RES_MEMORY, PCIR_BAR(ENA_REG_BAR), adapter->registers); } + + if (adapter->msix != NULL) { + bus_release_resource(pdev, SYS_RES_MEMORY, + adapter->msix_rid, adapter->msix); + } } static int @@ -3532,6 +3537,7 @@ ena_attach(device_t pdev) struct ena_adapter *adapter; struct ena_com_dev *ena_dev = NULL; uint32_t max_num_io_queues; + int msix_rid; int rid, rc; adapter = device_get_softc(pdev); @@ -3570,6 +3576,20 @@ ena_attach(device_t pdev) goto err_dev_free; } + /* MSIx vector table may reside on BAR0 with registers or on BAR1. */ + msix_rid = pci_msix_table_bar(pdev); + if (msix_rid != rid) { + adapter->msix = bus_alloc_resource_any(pdev, SYS_RES_MEMORY, + &msix_rid, RF_ACTIVE); + if (unlikely(adapter->msix == NULL)) { + device_printf(pdev, + "unable to allocate bus resource: msix!\n"); + rc = ENOMEM; + goto err_pci_free; + } + adapter->msix_rid = msix_rid; + } + ena_dev->bus = malloc(sizeof(struct ena_bus), M_DEVBUF, M_WAITOK | M_ZERO); @@ -3735,6 +3755,7 @@ err_com_free: ena_com_mmio_reg_read_request_destroy(ena_dev); err_bus_free: free(ena_dev->bus, M_DEVBUF); +err_pci_free: ena_free_pci_resources(adapter); err_dev_free: free(ena_dev, M_DEVBUF); diff --git a/sys/dev/ena/ena.h b/sys/dev/ena/ena.h index 28da047e2e1b..f3e92f31341a 100644 --- a/sys/dev/ena/ena.h +++ b/sys/dev/ena/ena.h @@ -41,7 +41,7 @@ #define DRV_MODULE_VER_MAJOR 2 #define DRV_MODULE_VER_MINOR 3 -#define DRV_MODULE_VER_SUBMINOR 0 +#define DRV_MODULE_VER_SUBMINOR 1 #define DRV_MODULE_NAME "ena" @@ -398,6 +398,8 @@ struct ena_adapter { /* OS resources */ struct resource *memory; struct resource *registers; + struct resource *msix; + int msix_rid; struct sx global_lock;