From nobody Wed Nov 29 01:55:38 2023 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sg2Tk6TXbz52t9S; Wed, 29 Nov 2023 01:55:38 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sg2Tk3Nlxz4DPR; Wed, 29 Nov 2023 01:55:38 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1701222938; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=q4QyyODI5xc/WUeG6tCtj1MxCDDIH6Sj1TdAizuFFOA=; b=FhDvWiZYGS0AHY1hctcfxExc2fR65iw+F2tru5mN4hv4mPOFchjGtRNY7yAgDRqWBlebXR b9M8bAMBJnqmaOMUSPApUrbgesASx42gTZemLKyEiwhwCVPpWwzWIzM1LemHY5J9+ZLAq0 yk4NSQyaJxkAEt5FP2/ujWuT1zHo0O5e4qPBoZCZ0zJJcnpWjGZIbO0KgecN1hevmHu4OY HdVYGeycIP632/+/thS4KjZEGSAWYt9Hp5xWNH6hac5KN8zm6MSqQTxAQapFFhY+yta1r1 I1TtDwroJWRfr2babZVr2Q4itcmJqT1EEPhdXoG/BHCMCTvWil7fQhono3MKvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1701222938; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=q4QyyODI5xc/WUeG6tCtj1MxCDDIH6Sj1TdAizuFFOA=; b=GKtmAvBI0x9kKUg+Yo04S07whANiESQcuLle/972MZTyv0EcoILb0ksaoNI/ex0gTScf7k LzT6aUNMd9CPWdaAOvVcwTwwRExpawpd+FUKR4stSjkYxrY3iIotEXmojJduLdaT+XCNhd kMX7UPkyO6wgeFo8PmM+fUILLQj+nNHYOnAvQcOokMTgT7XvuVcSn/g2ckcpQdT2Wu7uWo LkEn5HiX/BnMKlTfwNIL1iMnVLGEiR32CRNgjZJBNGiFS3jk936zbF5OVWfMYMS5NLPCAO yJiifScbRBFx7RfXsywB4BmeqTvk7Uu69pqJhGkmwOl/+0w+2busXLJIoY+L0g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1701222938; a=rsa-sha256; cv=none; b=MctBg3EiU0e7qnLR7GzjJFA1ztFf0bvw9RvqC0s35QsZ+2kjSy7iDPeIvW7JHfDee9xlYx UkfmkMIsUs/ZajX3sMQVTcKMu6pYalJqkJBLIMLhYWv7FYykqNIvaMxi+fZ238KdOsETPy UgKjI2aXj70PQCBhSllplMbKwsm4XNk2SG3JfeNdduut5oGRU4XY4twGOgAC1oV17WAkl4 HfIL9p5o3bKSx+zeEDT9Bep/JefeJh+G43CpgV6cfh5cObuhIEPogNBsWLX7KQX7Bm9bO/ Abthz7DVuK9rHaCn4Pa/civ1swYjvm0dYayKG502rCmwoRWYdVoDAtOm0eW70w== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Sg2Tk2TRhzn2B; Wed, 29 Nov 2023 01:55:38 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 3AT1tcaU064516; Wed, 29 Nov 2023 01:55:38 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 3AT1tciC064513; Wed, 29 Nov 2023 01:55:38 GMT (envelope-from git) Date: Wed, 29 Nov 2023 01:55:38 GMT Message-Id: <202311290155.3AT1tciC064513@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Warner Losh Subject: git: 91d961356d03 - main - mpi3mr: Honor the dma mask from IOCFacts List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: imp X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 91d961356d03465635c4784fab48acdd1304e1e0 Auto-Submitted: auto-generated The branch main has been updated by imp: URL: https://cgit.FreeBSD.org/src/commit/?id=91d961356d03465635c4784fab48acdd1304e1e0 commit 91d961356d03465635c4784fab48acdd1304e1e0 Author: Warner Losh AuthorDate: 2023-11-29 01:49:49 +0000 Commit: Warner Losh CommitDate: 2023-11-29 01:49:49 +0000 mpi3mr: Honor the dma mask from IOCFacts The number of signficant bits that are decoded are returned in the flags field of the IOCFacts structure from the device. Rather than assume the worst with a pessimal 32-bit maximum, look at this value and pass it along to all the dma map creation requests. A lof of those creations are repetitive and could just inherit from the base tag if we moved to the templated interface. This is called out as desireable future work not done at this time. In addition, due to a chicken and an egg problem, we have to allocate some of the maps with a 32-bit loaddr. These are the ones we need to read iocfacts. And they are fine to be so restricted: they are little used after startup, and when they are used, bouncing is fine. Sponsored by: Netflix Reviewed by: mav Differential Revision: https://reviews.freebsd.org/D42559 --- sys/dev/mpi3mr/mpi3mr.c | 27 +++++++++++++++++++++++++++ sys/dev/mpi3mr/mpi3mr_pci.c | 10 ++++++++++ 2 files changed, 37 insertions(+) diff --git a/sys/dev/mpi3mr/mpi3mr.c b/sys/dev/mpi3mr/mpi3mr.c index a2e43850d9b5..4ee44eec6c9a 100644 --- a/sys/dev/mpi3mr/mpi3mr.c +++ b/sys/dev/mpi3mr/mpi3mr.c @@ -980,6 +980,11 @@ static int mpi3mr_setup_admin_qpair(struct mpi3mr_softc *sc) sc->admin_reply_ephase = 1; if (!sc->admin_req) { + /* + * We need to create the tag for the admin queue to get the + * iofacts to see how many bits the controller decodes. Solve + * this chicken and egg problem by only doing lower 4GB DMA. + */ if (bus_dma_tag_create(sc->mpi3mr_parent_dmat, /* parent */ 4, 0, /* algnmnt, boundary */ BUS_SPACE_MAXADDR_32BIT,/* lowaddr */ @@ -1434,6 +1439,12 @@ static int mpi3mr_issue_iocfacts(struct mpi3mr_softc *sc, MPI3_SGE_FLAGS_END_OF_LIST); + /* + * We can't use sc->dma_loaddr / hiaddr here. We set those only after + * we get the iocfacts. So allocate in the lower 4GB. The amount of + * data is tiny and we don't do this that often, so any bouncing we + * might have to do isn't a cause for concern. + */ if (bus_dma_tag_create(sc->mpi3mr_parent_dmat, /* parent */ 4, 0, /* algnmnt, boundary */ BUS_SPACE_MAXADDR_32BIT,/* lowaddr */ @@ -1668,6 +1679,22 @@ static int mpi3mr_process_factsdata(struct mpi3mr_softc *sc, sc->max_host_ios = sc->facts.max_reqs - (MPI3MR_INTERNALCMDS_RESVD + 1); + /* + * Set the DMA mask for the card. dma_mask is the number of bits that + * can have bits set in them. Translate this into bus_dma loaddr/hiaddr + * args. Add sanity for more bits than address space or other overflow + * situations. + */ + if (sc->facts.dma_mask == 0 || + (sc->facts.dma_mask >= sizeof(bus_addr_t) * 8)) + sc->dma_loaddr = BUS_SPACE_MAXADDR; + else + sc->dma_loaddr = ~((1ull << sc->facts.dma_mask) - 1); + sc->dma_hiaddr = BUS_SPACE_MAXADDR; + mpi3mr_dprint(sc, MPI3MR_INFO, + "dma_mask bits: %d loaddr 0x%jx hiaddr 0x%jx\n", + sc->facts.dma_mask, sc->dma_loaddr, sc->dma_hiaddr); + return retval; } diff --git a/sys/dev/mpi3mr/mpi3mr_pci.c b/sys/dev/mpi3mr/mpi3mr_pci.c index eaf73022291d..11ec98882949 100644 --- a/sys/dev/mpi3mr/mpi3mr_pci.c +++ b/sys/dev/mpi3mr/mpi3mr_pci.c @@ -273,6 +273,16 @@ static int mpi3mr_setup_resources(struct mpi3mr_softc *sc) sc->mpi3mr_btag = rman_get_bustag(sc->mpi3mr_regs_resource); sc->mpi3mr_bhandle = rman_get_bushandle(sc->mpi3mr_regs_resource); + /* + * XXX Perhaps we should move this to after we read iocfacts and use + * that to create the proper parent tag. However, to get the iocfacts + * we need to have a dmatag for both the admin queue and the iocfacts + * DMA transfer. So for now, we just create a 'no restriction' tag and + * use sc->dma_loaddr for all the other tag_create calls to get the + * right value. It would be nice if one could retroactively adjust a + * created tag. The Linux driver effectively does this by setting the + * dma_mask on the device. + */ /* Allocate the parent DMA tag */ if (bus_dma_tag_create(bus_get_dma_tag(dev), /* parent */ 1, 0, /* algnmnt, boundary */