From nobody Sat Jun 25 06:08:34 2022 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 397A986672C; Sat, 25 Jun 2022 06:08:35 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LVNpW0MbYz4hHX; Sat, 25 Jun 2022 06:08:35 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1656137315; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=acp8Z3nfSdniRo2W76I0QwW7kOkFhgQnqGmNPYThnHc=; b=x9+ZGpcRQInFC0ECfid7BqG2so/Qcr5t1GQVkHh88Iz+oOnN5Ao61nm6SiCJaR7tqkAiFd dxAEYT360UGImS6yzzpXawJHXXEw9sluLgcN4iE1ZNsMoICk0pfF+KY8ZlE8sVpuYmkeDT 4HIw10Obz+VE4qkuraldVFvmWXRj3OXGulBmrmAXeobuGUrsTrUe67dV4Ne+HbT2FGmZmT xZP2xisU85FljXfse4UIB7C5jki2Zp/Rfaz+sGY1bVWf1O2+0V0ldgOHpk0x0pvUXNpKEU Hcrv7iyB7r15DIpBfmSE4xvIgqRM98swxf3KY0oyMSywiRofDbc7qCaSprGIFQ== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id D0D2A1F1A2; Sat, 25 Jun 2022 06:08:34 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 25P68Y0l043495; Sat, 25 Jun 2022 06:08:34 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 25P68Yjh043494; Sat, 25 Jun 2022 06:08:34 GMT (envelope-from git) Date: Sat, 25 Jun 2022 06:08:34 GMT Message-Id: <202206250608.25P68Yjh043494@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Alan Cox Subject: git: eeb46578c21a - main - busdma_iommu: Fine-grained locking for the dmamap's map list List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: alc X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: eeb46578c21ad37866f49f3bbb3ac738b44abbf6 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1656137315; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=acp8Z3nfSdniRo2W76I0QwW7kOkFhgQnqGmNPYThnHc=; b=rKwn1Vn4y69iyGfSXw9YbJryRaL6dhQ9qVL0YhcpENzpMiaQtQ18tZ/ZvsG4MBcdHUHewj CnZqERC9KaLQbJ+4N56muhiWNyEPGaW93VxkCLB0Jan+cOb5SCIfmBV8sp1kSKVircQhKp xJdtb0uIlfCl59xyD+wYCmXskXNhtTUeA5CvjS803aOuOg+NvcPxhEp/Ud84kY0GkY281A BFZU3xTv1x7COHlHROc/YyA1iVKkvrPHHO1xA65HMoFXa8a2LfSovWEKlJyuGBNZqlA3aB Llzaz98fwY6p0N7qmBLo2H5UDptugAu2cXe/iMW1MbNhwffhOVHpWaVWavydqg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1656137315; a=rsa-sha256; cv=none; b=hW4zm1GCqeNOUZXjCy6gPx5viwer0s0JvFUz9dYTA+bGyH8eDFdHv29Td5+3+gc5LiGgs2 aX5mREp4ii+KTU/jJo3zSkaWu61NRp2I5dJBHhL9VBGmOisKaPMoKk7jXleQYIfaju8riW lBRwNG/fyVrm2SXzuCHj1qwzc1mH0HW5y3+j0jwLMhUP1LMxOpauf/LA5jtv+HzaDh8NfC 7FP9pcnakl+hVy9J1G1Jg06OYIl0AtoTQgjpiDanFJV3bG+8YhX3lNZ60o3HmF7RJEIAVU I2PVqBD0IhaRDTpOcRZRNL+GuWhGXzk+sK8PEqueHySF/vYcZyE5xubex3JaUA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by alc: URL: https://cgit.FreeBSD.org/src/commit/?id=eeb46578c21ad37866f49f3bbb3ac738b44abbf6 commit eeb46578c21ad37866f49f3bbb3ac738b44abbf6 Author: Alan Cox AuthorDate: 2022-06-22 21:51:47 +0000 Commit: Alan Cox CommitDate: 2022-06-25 05:59:23 +0000 busdma_iommu: Fine-grained locking for the dmamap's map list Introduce fine-grained locking on the dmamap's list of map entries, replacing the use of the domain lock. This is not the most significant source of lock contention, but it is the easiest to address. Reviewed by: kib MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D35557 --- sys/dev/iommu/busdma_iommu.c | 56 +++++++++++++++++++++----------------------- sys/dev/iommu/busdma_iommu.h | 7 ++++++ 2 files changed, 34 insertions(+), 29 deletions(-) diff --git a/sys/dev/iommu/busdma_iommu.c b/sys/dev/iommu/busdma_iommu.c index 42fc0b6c5451..69cf9dd12e7e 100644 --- a/sys/dev/iommu/busdma_iommu.c +++ b/sys/dev/iommu/busdma_iommu.c @@ -458,6 +458,7 @@ iommu_bus_dmamap_create(bus_dma_tag_t dmat, int flags, bus_dmamap_t *mapp) return (ENOMEM); } } + IOMMU_DMAMAP_INIT(map); TAILQ_INIT(&map->map_entries); map->tag = tag; map->locked = true; @@ -473,18 +474,16 @@ iommu_bus_dmamap_destroy(bus_dma_tag_t dmat, bus_dmamap_t map1) { struct bus_dma_tag_iommu *tag; struct bus_dmamap_iommu *map; - struct iommu_domain *domain; tag = (struct bus_dma_tag_iommu *)dmat; map = (struct bus_dmamap_iommu *)map1; if (map != NULL) { - domain = tag->ctx->domain; - IOMMU_DOMAIN_LOCK(domain); + IOMMU_DMAMAP_LOCK(map); if (!TAILQ_EMPTY(&map->map_entries)) { - IOMMU_DOMAIN_UNLOCK(domain); + IOMMU_DMAMAP_UNLOCK(map); return (EBUSY); } - IOMMU_DOMAIN_UNLOCK(domain); + IOMMU_DMAMAP_DESTROY(map); free(map, M_IOMMU_DMAMAP); } tag->map_count--; @@ -625,10 +624,11 @@ iommu_bus_dmamap_load_something1(struct bus_dma_tag_iommu *tag, (uintmax_t)entry->start, (uintmax_t)entry->end, (uintmax_t)buflen1, (uintmax_t)tag->common.maxsegsz)); - IOMMU_DOMAIN_LOCK(domain); + KASSERT((entry->flags & IOMMU_MAP_ENTRY_MAP) != 0, + ("entry %p missing IOMMU_MAP_ENTRY_MAP", entry)); + IOMMU_DMAMAP_LOCK(map); TAILQ_INSERT_TAIL(&map->map_entries, entry, dmamap_link); - entry->flags |= IOMMU_MAP_ENTRY_MAP; - IOMMU_DOMAIN_UNLOCK(domain); + IOMMU_DMAMAP_UNLOCK(map); TAILQ_INSERT_TAIL(unroll_list, entry, unroll_link); segs[seg].ds_addr = entry->start + offset; @@ -651,8 +651,8 @@ iommu_bus_dmamap_load_something(struct bus_dma_tag_iommu *tag, { struct iommu_ctx *ctx; struct iommu_domain *domain; - struct iommu_map_entry *entry, *entry1; - struct iommu_map_entries_tailq unroll_list; + struct iommu_map_entry *entry; + struct iommu_map_entries_tailq entries, unroll_list; int error; ctx = tag->ctx; @@ -662,15 +662,15 @@ iommu_bus_dmamap_load_something(struct bus_dma_tag_iommu *tag, TAILQ_INIT(&unroll_list); error = iommu_bus_dmamap_load_something1(tag, map, ma, offset, buflen, flags, segs, segp, &unroll_list); - if (error != 0) { + if (error != 0 && !TAILQ_EMPTY(&unroll_list)) { /* * The busdma interface does not allow us to report * partial buffer load, so unfortunately we have to * revert all work done. */ - IOMMU_DOMAIN_LOCK(domain); - TAILQ_FOREACH_SAFE(entry, &unroll_list, unroll_link, - entry1) { + TAILQ_INIT(&entries); + IOMMU_DMAMAP_LOCK(map); + TAILQ_FOREACH(entry, &unroll_list, unroll_link) { /* * No entries other than what we have created * during the failed run might have been @@ -678,10 +678,11 @@ iommu_bus_dmamap_load_something(struct bus_dma_tag_iommu *tag, * pglock. */ TAILQ_REMOVE(&map->map_entries, entry, dmamap_link); - TAILQ_REMOVE(&unroll_list, entry, unroll_link); - TAILQ_INSERT_TAIL(&domain->unload_entries, entry, - dmamap_link); + TAILQ_INSERT_TAIL(&entries, entry, dmamap_link); } + IOMMU_DMAMAP_UNLOCK(map); + IOMMU_DOMAIN_LOCK(domain); + TAILQ_CONCAT(&domain->unload_entries, &entries, dmamap_link); IOMMU_DOMAIN_UNLOCK(domain); taskqueue_enqueue(domain->iommu->delayed_taskqueue, &domain->unload_task); @@ -872,9 +873,7 @@ iommu_bus_dmamap_unload(bus_dma_tag_t dmat, bus_dmamap_t map1) struct bus_dmamap_iommu *map; struct iommu_ctx *ctx; struct iommu_domain *domain; -#ifndef IOMMU_DOMAIN_UNLOAD_SLEEP struct iommu_map_entries_tailq entries; -#endif tag = (struct bus_dma_tag_iommu *)dmat; map = (struct bus_dmamap_iommu *)map1; @@ -882,17 +881,17 @@ iommu_bus_dmamap_unload(bus_dma_tag_t dmat, bus_dmamap_t map1) domain = ctx->domain; atomic_add_long(&ctx->unloads, 1); + TAILQ_INIT(&entries); + IOMMU_DMAMAP_LOCK(map); + TAILQ_CONCAT(&entries, &map->map_entries, dmamap_link); + IOMMU_DMAMAP_UNLOCK(map); #if defined(IOMMU_DOMAIN_UNLOAD_SLEEP) IOMMU_DOMAIN_LOCK(domain); - TAILQ_CONCAT(&domain->unload_entries, &map->map_entries, dmamap_link); + TAILQ_CONCAT(&domain->unload_entries, &entries, dmamap_link); IOMMU_DOMAIN_UNLOCK(domain); taskqueue_enqueue(domain->iommu->delayed_taskqueue, &domain->unload_task); #else - TAILQ_INIT(&entries); - IOMMU_DOMAIN_LOCK(domain); - TAILQ_CONCAT(&entries, &map->map_entries, dmamap_link); - IOMMU_DOMAIN_UNLOCK(domain); THREAD_NO_SLEEPING(); iommu_domain_unload(domain, &entries, false); THREAD_SLEEPING_OK(); @@ -1062,13 +1061,12 @@ bus_dma_iommu_load_ident(bus_dma_tag_t dmat, bus_dmamap_t map1, VM_MEMATTR_DEFAULT); } error = iommu_gas_map_region(domain, entry, IOMMU_MAP_ENTRY_READ | - ((flags & BUS_DMA_NOWRITE) ? 0 : IOMMU_MAP_ENTRY_WRITE), - waitok ? IOMMU_MF_CANWAIT : 0, ma); + ((flags & BUS_DMA_NOWRITE) ? 0 : IOMMU_MAP_ENTRY_WRITE) | + IOMMU_MAP_ENTRY_MAP, waitok ? IOMMU_MF_CANWAIT : 0, ma); if (error == 0) { - IOMMU_DOMAIN_LOCK(domain); + IOMMU_DMAMAP_LOCK(map); TAILQ_INSERT_TAIL(&map->map_entries, entry, dmamap_link); - entry->flags |= IOMMU_MAP_ENTRY_MAP; - IOMMU_DOMAIN_UNLOCK(domain); + IOMMU_DMAMAP_UNLOCK(map); } else { iommu_domain_unload_entry(entry, true); } diff --git a/sys/dev/iommu/busdma_iommu.h b/sys/dev/iommu/busdma_iommu.h index 903562ecea85..287da3460a92 100644 --- a/sys/dev/iommu/busdma_iommu.h +++ b/sys/dev/iommu/busdma_iommu.h @@ -49,6 +49,7 @@ struct bus_dmamap_iommu { struct memdesc mem; bus_dmamap_callback_t *callback; void *callback_arg; + struct mtx lock; struct iommu_map_entries_tailq map_entries; TAILQ_ENTRY(bus_dmamap_iommu) delay_link; bool locked; @@ -59,6 +60,12 @@ struct bus_dmamap_iommu { #endif }; +#define IOMMU_DMAMAP_INIT(map) mtx_init(&(map)->lock, \ + "iommu dmamap", NULL, MTX_DEF) +#define IOMMU_DMAMAP_DESTROY(map) mtx_destroy(&(map)->lock) +#define IOMMU_DMAMAP_LOCK(map) mtx_lock(&(map)->lock) +#define IOMMU_DMAMAP_UNLOCK(map) mtx_unlock(&(map)->lock) + #define BUS_DMAMAP_IOMMU_MALLOC 0x0001 #define BUS_DMAMAP_IOMMU_KMEM_ALLOC 0x0002