From owner-svn-src-head@freebsd.org Tue Sep 3 14:06:12 2019 Return-Path: Delivered-To: svn-src-head@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B1F71DC32E; Tue, 3 Sep 2019 14:06:05 +0000 (UTC) (envelope-from yuripv@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [96.47.72.132]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46N7z12LS0z4PBD; Tue, 3 Sep 2019 14:06:05 +0000 (UTC) (envelope-from yuripv@freebsd.org) Received: by freefall.freebsd.org (Postfix, from userid 1452) id 76A6D1A0D7; Tue, 3 Sep 2019 14:05:57 +0000 (UTC) X-Original-To: yuripv@localmail.freebsd.org Delivered-To: yuripv@localmail.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "mx1.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by freefall.freebsd.org (Postfix) with ESMTPS id 2EA807717; Tue, 2 Apr 2019 19:08:11 +0000 (UTC) (envelope-from owner-src-committers@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D16472E9B; Tue, 2 Apr 2019 19:08:10 +0000 (UTC) (envelope-from owner-src-committers@freebsd.org) Received: by freefall.freebsd.org (Postfix, from userid 538) id 1C3D776AD; Tue, 2 Apr 2019 19:08:10 +0000 (UTC) Delivered-To: src-committers@localmail.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "mx1.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by freefall.freebsd.org (Postfix) with ESMTPS id A9B4376AB for ; Tue, 2 Apr 2019 19:08:07 +0000 (UTC) (envelope-from tychon@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7BDB072E8E; Tue, 2 Apr 2019 19:08:07 +0000 (UTC) (envelope-from tychon@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 668422559; Tue, 2 Apr 2019 19:08:07 +0000 (UTC) (envelope-from tychon@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x32J87SC052414; Tue, 2 Apr 2019 19:08:07 GMT (envelope-from tychon@FreeBSD.org) Received: (from tychon@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x32J87f5052412; Tue, 2 Apr 2019 19:08:07 GMT (envelope-from tychon@FreeBSD.org) Message-Id: <201904021908.x32J87f5052412@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: tychon set sender to tychon@FreeBSD.org using -f From: Tycho Nightingale To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r345813 - head/sys/dev/ioat X-SVN-Group: head X-SVN-Commit-Author: tychon X-SVN-Commit-Paths: head/sys/dev/ioat X-SVN-Commit-Revision: 345813 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Precedence: bulk X-Loop: FreeBSD.org Sender: owner-src-committers@freebsd.org X-Rspamd-Queue-Id: 4D16472E9B X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.95 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; NEURAL_HAM_SHORT(-0.96)[-0.956,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US]; NEURAL_HAM_LONG(-1.00)[-1.000,0] Status: O X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.29 List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Date: Tue, 03 Sep 2019 14:06:12 -0000 X-Original-Date: Tue, 2 Apr 2019 19:08:07 +0000 (UTC) X-List-Received-Date: Tue, 03 Sep 2019 14:06:12 -0000 Author: tychon Date: Tue Apr 2 19:08:06 2019 New Revision: 345813 URL: https://svnweb.freebsd.org/changeset/base/345813 Log: ioat(4) should use bus_dma(9) for the operation source and destination addresses Reviewed by: cem Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D19725 Modified: head/sys/dev/ioat/ioat.c head/sys/dev/ioat/ioat_internal.h Modified: head/sys/dev/ioat/ioat.c ============================================================================== --- head/sys/dev/ioat/ioat.c Tue Apr 2 19:06:25 2019 (r345812) +++ head/sys/dev/ioat/ioat.c Tue Apr 2 19:08:06 2019 (r345813) @@ -320,10 +320,26 @@ err: return (error); } +static inline int +ioat_bus_dmamap_destroy(struct ioat_softc *ioat, const char *func, + bus_dma_tag_t dmat, bus_dmamap_t map) +{ + int error; + + error = bus_dmamap_destroy(dmat, map); + if (error != 0) { + ioat_log_message(0, + "%s: bus_dmamap_destroy failed %d\n", func, error); + } + + return (error); +} + static int ioat_detach(device_t device) { struct ioat_softc *ioat; + int i, error; ioat = DEVICE2SOFTC(device); @@ -359,6 +375,47 @@ ioat_detach(device_t device) bus_release_resource(device, SYS_RES_MEMORY, ioat->pci_resource_id, ioat->pci_resource); + if (ioat->data_tag != NULL) { + for (i = 0; i < 1 << ioat->ring_size_order; i++) { + error = ioat_bus_dmamap_destroy(ioat, __func__, + ioat->data_tag, ioat->ring[i].src_dmamap); + if (error != 0) + return (error); + } + for (i = 0; i < 1 << ioat->ring_size_order; i++) { + error = ioat_bus_dmamap_destroy(ioat, __func__, + ioat->data_tag, ioat->ring[i].dst_dmamap); + if (error != 0) + return (error); + } + + for (i = 0; i < 1 << ioat->ring_size_order; i++) { + error = ioat_bus_dmamap_destroy(ioat, __func__, + ioat->data_tag, ioat->ring[i].src2_dmamap); + if (error != 0) + return (error); + } + for (i = 0; i < 1 << ioat->ring_size_order; i++) { + error = ioat_bus_dmamap_destroy(ioat, __func__, + ioat->data_tag, ioat->ring[i].dst2_dmamap); + if (error != 0) + return (error); + } + + bus_dma_tag_destroy(ioat->data_tag); + } + + if (ioat->data_crc_tag != NULL) { + for (i = 0; i < 1 << ioat->ring_size_order; i++) { + error = ioat_bus_dmamap_destroy(ioat, __func__, + ioat->data_crc_tag, ioat->ring[i].crc_dmamap); + if (error != 0) + return (error); + } + + bus_dma_tag_destroy(ioat->data_crc_tag); + } + if (ioat->ring != NULL) ioat_free_ring(ioat, 1 << ioat->ring_size_order, ioat->ring); @@ -523,6 +580,25 @@ ioat3_attach(device_t device) ioat->hw_desc_ring = hw_desc; + error = bus_dma_tag_create(bus_get_dma_tag(ioat->device), + 1, 0, BUS_SPACE_MAXADDR_40BIT, BUS_SPACE_MAXADDR, NULL, NULL, + ioat->max_xfer_size, 1, ioat->max_xfer_size, 0, NULL, NULL, + &ioat->data_crc_tag); + if (error != 0) { + ioat_log_message(0, "%s: bus_dma_tag_create failed %d\n", + __func__, error); + return (error); + } + + error = bus_dma_tag_create(bus_get_dma_tag(ioat->device), + 1, 0, BUS_SPACE_MAXADDR_48BIT, BUS_SPACE_MAXADDR, NULL, NULL, + ioat->max_xfer_size, 1, ioat->max_xfer_size, 0, NULL, NULL, + &ioat->data_tag); + if (error != 0) { + ioat_log_message(0, "%s: bus_dma_tag_create failed %d\n", + __func__, error); + return (error); + } ioat->ring = malloc(num_descriptors * sizeof(*ring), M_IOAT, M_ZERO | M_WAITOK); @@ -530,6 +606,46 @@ ioat3_attach(device_t device) for (i = 0; i < num_descriptors; i++) { memset(&ring[i].bus_dmadesc, 0, sizeof(ring[i].bus_dmadesc)); ring[i].id = i; + error = bus_dmamap_create(ioat->data_tag, 0, + &ring[i].src_dmamap); + if (error != 0) { + ioat_log_message(0, + "%s: bus_dmamap_create failed %d\n", __func__, + error); + return (error); + } + error = bus_dmamap_create(ioat->data_tag, 0, + &ring[i].dst_dmamap); + if (error != 0) { + ioat_log_message(0, + "%s: bus_dmamap_create failed %d\n", __func__, + error); + return (error); + } + error = bus_dmamap_create(ioat->data_tag, 0, + &ring[i].src2_dmamap); + if (error != 0) { + ioat_log_message(0, + "%s: bus_dmamap_create failed %d\n", __func__, + error); + return (error); + } + error = bus_dmamap_create(ioat->data_tag, 0, + &ring[i].dst2_dmamap); + if (error != 0) { + ioat_log_message(0, + "%s: bus_dmamap_create failed %d\n", __func__, + error); + return (error); + } + error = bus_dmamap_create(ioat->data_crc_tag, 0, + &ring[i].crc_dmamap); + if (error != 0) { + ioat_log_message(0, + "%s: bus_dmamap_create failed %d\n", __func__, + error); + return (error); + } } for (i = 0; i < num_descriptors; i++) { @@ -724,6 +840,12 @@ ioat_process_events(struct ioat_softc *ioat, boolean_t ioat->chan_idx, ioat->tail, dmadesc, dmadesc->callback_fn, dmadesc->callback_arg); + bus_dmamap_unload(ioat->data_tag, desc->src_dmamap); + bus_dmamap_unload(ioat->data_tag, desc->dst_dmamap); + bus_dmamap_unload(ioat->data_tag, desc->src2_dmamap); + bus_dmamap_unload(ioat->data_tag, desc->dst2_dmamap); + bus_dmamap_unload(ioat->data_crc_tag, desc->crc_dmamap); + if (dmadesc->callback_fn != NULL) dmadesc->callback_fn(dmadesc->callback_arg, 0); @@ -999,7 +1121,8 @@ ioat_op_generic(struct ioat_softc *ioat, uint8_t op, { struct ioat_generic_hw_descriptor *hw_desc; struct ioat_descriptor *desc; - int mflags; + bus_dma_segment_t seg; + int mflags, nseg, error; mtx_assert(&ioat->submit_lock, MA_OWNED); @@ -1032,9 +1155,31 @@ ioat_op_generic(struct ioat_softc *ioat, uint8_t op, hw_desc->u.control_generic.fence = 1; hw_desc->size = size; - hw_desc->src_addr = src; - hw_desc->dest_addr = dst; + if (src != 0) { + nseg = -1; + error = _bus_dmamap_load_phys(ioat->data_tag, desc->src_dmamap, + src, size, 0, &seg, &nseg); + if (error != 0) { + ioat_log_message(0, "%s: _bus_dmamap_load_phys" + " failed %d\n", __func__, error); + return (NULL); + } + hw_desc->src_addr = seg.ds_addr; + } + + if (dst != 0) { + nseg = -1; + error = _bus_dmamap_load_phys(ioat->data_tag, desc->dst_dmamap, + dst, size, 0, &seg, &nseg); + if (error != 0) { + ioat_log_message(0, "%s: _bus_dmamap_load_phys" + " failed %d\n", __func__, error); + return (NULL); + } + hw_desc->dest_addr = seg.ds_addr; + } + desc->bus_dmadesc.callback_fn = callback_fn; desc->bus_dmadesc.callback_arg = callback_arg; return (desc); @@ -1102,6 +1247,9 @@ ioat_copy_8k_aligned(bus_dmaengine_t dmaengine, bus_ad struct ioat_dma_hw_descriptor *hw_desc; struct ioat_descriptor *desc; struct ioat_softc *ioat; + bus_size_t src1_len, dst1_len; + bus_dma_segment_t seg; + int nseg, error; ioat = to_ioat_softc(dmaengine); CTR2(KTR_IOAT, "%s channel=%u", __func__, ioat->chan_idx); @@ -1117,19 +1265,57 @@ ioat_copy_8k_aligned(bus_dmaengine_t dmaengine, bus_ad return (NULL); } - desc = ioat_op_generic(ioat, IOAT_OP_COPY, 2 * PAGE_SIZE, src1, dst1, + desc = ioat_op_generic(ioat, IOAT_OP_COPY, 2 * PAGE_SIZE, 0, 0, callback_fn, callback_arg, flags); if (desc == NULL) return (NULL); hw_desc = &ioat_get_descriptor(ioat, desc->id)->dma; - if (src2 != src1 + PAGE_SIZE) { + + src1_len = (src2 != src1 + PAGE_SIZE) ? PAGE_SIZE : 2 * PAGE_SIZE; + nseg = -1; + error = _bus_dmamap_load_phys(ioat->data_tag, + desc->src_dmamap, src1, src1_len, 0, &seg, &nseg); + if (error != 0) { + ioat_log_message(0, "%s: _bus_dmamap_load_phys" + " failed %d\n", __func__, error); + return (NULL); + } + hw_desc->src_addr = seg.ds_addr; + if (src1_len != 2 * PAGE_SIZE) { hw_desc->u.control.src_page_break = 1; - hw_desc->next_src_addr = src2; + nseg = -1; + error = _bus_dmamap_load_phys(ioat->data_tag, + desc->src2_dmamap, src2, PAGE_SIZE, 0, &seg, &nseg); + if (error != 0) { + ioat_log_message(0, "%s: _bus_dmamap_load_phys" + " failed %d\n", __func__, error); + return (NULL); + } + hw_desc->next_src_addr = seg.ds_addr; } - if (dst2 != dst1 + PAGE_SIZE) { + + dst1_len = (dst2 != dst1 + PAGE_SIZE) ? PAGE_SIZE : 2 * PAGE_SIZE; + nseg = -1; + error = _bus_dmamap_load_phys(ioat->data_tag, + desc->dst_dmamap, dst1, dst1_len, 0, &seg, &nseg); + if (error != 0) { + ioat_log_message(0, "%s: _bus_dmamap_load_phys" + " failed %d\n", __func__, error); + return (NULL); + } + hw_desc->dest_addr = seg.ds_addr; + if (dst1_len != 2 * PAGE_SIZE) { hw_desc->u.control.dest_page_break = 1; - hw_desc->next_dest_addr = dst2; + nseg = -1; + error = _bus_dmamap_load_phys(ioat->data_tag, + desc->dst2_dmamap, dst2, PAGE_SIZE, 0, &seg, &nseg); + if (error != 0) { + ioat_log_message(0, "%s: _bus_dmamap_load_phys" + " failed %d\n", __func__, error); + return (NULL); + } + hw_desc->next_dest_addr = seg.ds_addr; } if (g_ioat_debug_level >= 3) @@ -1149,6 +1335,8 @@ ioat_copy_crc(bus_dmaengine_t dmaengine, bus_addr_t ds struct ioat_softc *ioat; uint32_t teststore; uint8_t op; + bus_dma_segment_t seg; + int nseg, error; ioat = to_ioat_softc(dmaengine); CTR2(KTR_IOAT, "%s channel=%u", __func__, ioat->chan_idx); @@ -1201,9 +1389,18 @@ ioat_copy_crc(bus_dmaengine_t dmaengine, bus_addr_t ds hw_desc = &ioat_get_descriptor(ioat, desc->id)->crc32; - if ((flags & DMA_CRC_INLINE) == 0) - hw_desc->crc_address = crcptr; - else + if ((flags & DMA_CRC_INLINE) == 0) { + nseg = -1; + error = _bus_dmamap_load_phys(ioat->data_crc_tag, + desc->crc_dmamap, crcptr, sizeof(uint32_t), 0, + &seg, &nseg); + if (error != 0) { + ioat_log_message(0, "%s: _bus_dmamap_load_phys" + " failed %d\n", __func__, error); + return (NULL); + } + hw_desc->crc_address = seg.ds_addr; + } else hw_desc->u.control.crc_location = 1; if (initialseed != NULL) { @@ -1228,6 +1425,8 @@ ioat_crc(bus_dmaengine_t dmaengine, bus_addr_t src, bu struct ioat_softc *ioat; uint32_t teststore; uint8_t op; + bus_dma_segment_t seg; + int nseg, error; ioat = to_ioat_softc(dmaengine); CTR2(KTR_IOAT, "%s channel=%u", __func__, ioat->chan_idx); @@ -1280,9 +1479,18 @@ ioat_crc(bus_dmaengine_t dmaengine, bus_addr_t src, bu hw_desc = &ioat_get_descriptor(ioat, desc->id)->crc32; - if ((flags & DMA_CRC_INLINE) == 0) - hw_desc->crc_address = crcptr; - else + if ((flags & DMA_CRC_INLINE) == 0) { + nseg = -1; + error = _bus_dmamap_load_phys(ioat->data_crc_tag, + desc->crc_dmamap, crcptr, sizeof(uint32_t), 0, + &seg, &nseg); + if (error != 0) { + ioat_log_message(0, "%s: _bus_dmamap_load_phys" + " failed %d\n", __func__, error); + return (NULL); + } + hw_desc->crc_address = seg.ds_addr; + } else hw_desc->u.control.crc_location = 1; if (initialseed != NULL) { @@ -1321,12 +1529,13 @@ ioat_blockfill(bus_dmaengine_t dmaengine, bus_addr_t d return (NULL); } - desc = ioat_op_generic(ioat, IOAT_OP_FILL, len, fillpattern, dst, + desc = ioat_op_generic(ioat, IOAT_OP_FILL, len, 0, dst, callback_fn, callback_arg, flags); if (desc == NULL) return (NULL); hw_desc = &ioat_get_descriptor(ioat, desc->id)->fill; + hw_desc->src_data = fillpattern; if (g_ioat_debug_level >= 3) dump_descriptor(hw_desc); Modified: head/sys/dev/ioat/ioat_internal.h ============================================================================== --- head/sys/dev/ioat/ioat_internal.h Tue Apr 2 19:06:25 2019 (r345812) +++ head/sys/dev/ioat/ioat_internal.h Tue Apr 2 19:08:06 2019 (r345813) @@ -414,6 +414,11 @@ struct bus_dmadesc { struct ioat_descriptor { struct bus_dmadesc bus_dmadesc; uint32_t id; + bus_dmamap_t src_dmamap; + bus_dmamap_t dst_dmamap; + bus_dmamap_t src2_dmamap; + bus_dmamap_t dst2_dmamap; + bus_dmamap_t crc_dmamap; }; /* Unused by this driver at this time. */ @@ -456,6 +461,9 @@ struct ioat_softc { bus_dma_tag_t hw_desc_tag; bus_dmamap_t hw_desc_map; + + bus_dma_tag_t data_tag; + bus_dma_tag_t data_crc_tag; bus_dma_tag_t comp_update_tag; bus_dmamap_t comp_update_map;