From owner-svn-src-all@freebsd.org Fri Feb 9 03:06:46 2018 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7CFEDF1E0E1; Fri, 9 Feb 2018 03:06:46 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 30A6A6D924; Fri, 9 Feb 2018 03:06:46 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 2B7081F1F1; Fri, 9 Feb 2018 03:06:46 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1936jVV040107; Fri, 9 Feb 2018 03:06:45 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1936jhN040106; Fri, 9 Feb 2018 03:06:45 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201802090306.w1936jhN040106@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Fri, 9 Feb 2018 03:06:45 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-11@freebsd.org Subject: svn commit: r329056 - stable/11/sys/dev/ntb X-SVN-Group: stable-11 X-SVN-Commit-Author: mav X-SVN-Commit-Paths: stable/11/sys/dev/ntb X-SVN-Commit-Revision: 329056 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Feb 2018 03:06:46 -0000 Author: mav Date: Fri Feb 9 03:06:45 2018 New Revision: 329056 URL: https://svnweb.freebsd.org/changeset/base/329056 Log: MFC r328450: Use bus_dmamem_alloc(9) KPI instead of contigmalloc(9). I suppose it should make this code NUMA-aware with recent NUMA drop-in, trying to allocate shared memory buffers from domain closer to NT-bridge. Modified: stable/11/sys/dev/ntb/ntb_transport.c Directory Properties: stable/11/ (props changed) Modified: stable/11/sys/dev/ntb/ntb_transport.c ============================================================================== --- stable/11/sys/dev/ntb/ntb_transport.c Fri Feb 9 00:38:50 2018 (r329055) +++ stable/11/sys/dev/ntb/ntb_transport.c Fri Feb 9 03:06:45 2018 (r329056) @@ -182,6 +182,8 @@ struct ntb_transport_mw { size_t xlat_size; size_t buff_size; /* Rx buff is off virt_addr / dma_addr */ + bus_dma_tag_t dma_tag; + bus_dmamap_t dma_map; caddr_t virt_addr; bus_addr_t dma_addr; }; @@ -1147,10 +1149,25 @@ out: NTB_LINK_DOWN_TIMEOUT * hz / 1000, ntb_transport_link_work, nt); } +struct ntb_load_cb_args { + bus_addr_t addr; + int error; +}; + +static void +ntb_load_cb(void *xsc, bus_dma_segment_t *segs, int nsegs, int error) +{ + struct ntb_load_cb_args *cba = (struct ntb_load_cb_args *)xsc; + + if (!(cba->error = error)) + cba->addr = segs[0].ds_addr; +} + static int ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw, size_t size) { struct ntb_transport_mw *mw = &nt->mw_vec[num_mw]; + struct ntb_load_cb_args cba; size_t xlat_size, buff_size; int rc; @@ -1171,30 +1188,36 @@ ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw, s mw->xlat_size = xlat_size; mw->buff_size = buff_size; - mw->virt_addr = contigmalloc(mw->buff_size, M_NTB_T, M_ZERO, 0, - mw->addr_limit, mw->xlat_align, 0); - if (mw->virt_addr == NULL) { + if (bus_dma_tag_create(bus_get_dma_tag(nt->dev), mw->xlat_align, 0, + mw->addr_limit, BUS_SPACE_MAXADDR, + NULL, NULL, mw->buff_size, 1, mw->buff_size, + 0, NULL, NULL, &mw->dma_tag)) { + ntb_printf(0, "Unable to create MW tag of size %zu/%zu\n", + mw->buff_size, mw->xlat_size); + mw->xlat_size = 0; + mw->buff_size = 0; + return (ENOMEM); + } + if (bus_dmamem_alloc(mw->dma_tag, (void **)&mw->virt_addr, + BUS_DMA_WAITOK | BUS_DMA_ZERO, &mw->dma_map)) { + bus_dma_tag_destroy(mw->dma_tag); ntb_printf(0, "Unable to allocate MW buffer of size %zu/%zu\n", mw->buff_size, mw->xlat_size); mw->xlat_size = 0; mw->buff_size = 0; return (ENOMEM); } - /* TODO: replace with bus_space_* functions */ - mw->dma_addr = vtophys(mw->virt_addr); - - /* - * Ensure that the allocation from contigmalloc is aligned as - * requested. XXX: This may not be needed -- brought in for parity - * with the Linux driver. - */ - if (mw->dma_addr % mw->xlat_align != 0) { - ntb_printf(0, - "DMA memory 0x%jx not aligned to BAR size 0x%zx\n", - (uintmax_t)mw->dma_addr, size); - ntb_free_mw(nt, num_mw); + if (bus_dmamap_load(mw->dma_tag, mw->dma_map, mw->virt_addr, + mw->buff_size, ntb_load_cb, &cba, BUS_DMA_NOWAIT) || cba.error) { + bus_dmamem_free(mw->dma_tag, mw->virt_addr, mw->dma_map); + bus_dma_tag_destroy(mw->dma_tag); + ntb_printf(0, "Unable to load MW buffer of size %zu/%zu\n", + mw->buff_size, mw->xlat_size); + mw->xlat_size = 0; + mw->buff_size = 0; return (ENOMEM); } + mw->dma_addr = cba.addr; /* Notify HW the memory location of the receive buffer */ rc = ntb_mw_set_trans(nt->dev, num_mw, mw->dma_addr, mw->xlat_size); @@ -1216,7 +1239,9 @@ ntb_free_mw(struct ntb_transport_ctx *nt, int num_mw) return; ntb_mw_clear_trans(nt->dev, num_mw); - contigfree(mw->virt_addr, mw->xlat_size, M_NTB_T); + bus_dmamap_unload(mw->dma_tag, mw->dma_map); + bus_dmamem_free(mw->dma_tag, mw->virt_addr, mw->dma_map); + bus_dma_tag_destroy(mw->dma_tag); mw->xlat_size = 0; mw->buff_size = 0; mw->virt_addr = NULL;