From owner-svn-src-all@freebsd.org Wed Nov 18 14:50:13 2020 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7AB0946934F; Wed, 18 Nov 2020 14:50:13 +0000 (UTC) (envelope-from mw@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Cbm1x2qL9z3jlH; Wed, 18 Nov 2020 14:50:13 +0000 (UTC) (envelope-from mw@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 5424D1360F; Wed, 18 Nov 2020 14:50:13 +0000 (UTC) (envelope-from mw@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 0AIEoDM7016189; Wed, 18 Nov 2020 14:50:13 GMT (envelope-from mw@FreeBSD.org) Received: (from mw@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 0AIEoCba016185; Wed, 18 Nov 2020 14:50:12 GMT (envelope-from mw@FreeBSD.org) Message-Id: <202011181450.0AIEoCba016185@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mw set sender to mw@FreeBSD.org using -f From: Marcin Wojtas Date: Wed, 18 Nov 2020 14:50:12 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r367795 - in head/sys: contrib/ena-com dev/ena X-SVN-Group: head X-SVN-Commit-Author: mw X-SVN-Commit-Paths: in head/sys: contrib/ena-com dev/ena X-SVN-Commit-Revision: 367795 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Nov 2020 14:50:13 -0000 Author: mw Date: Wed Nov 18 14:50:12 2020 New Revision: 367795 URL: https://svnweb.freebsd.org/changeset/base/367795 Log: Fix completion descriptors alignment for the ENA The latest generation hardware requires IO CQ (completion queue) descriptors memory to be aligned to a 4K. It needs that feature for the best performance. Allocating unaligned descriptors will have a big performance impact as the packet processing in a HW won't be optimized properly. For that purpose adjust ena_dma_alloc() to support it. It's a critical fix, especially for the arm64 EC2 instances. Submitted by: Ido Segev Obtained from: Amazon, Inc MFC after: 1 week Differential revision: https://reviews.freebsd.org/D27114 Modified: head/sys/contrib/ena-com/ena_com.c head/sys/contrib/ena-com/ena_com.h head/sys/contrib/ena-com/ena_plat.h head/sys/dev/ena/ena.c Directory Properties: head/sys/contrib/ena-com/ (props changed) Modified: head/sys/contrib/ena-com/ena_com.c ============================================================================== --- head/sys/contrib/ena-com/ena_com.c Wed Nov 18 14:32:48 2020 (r367794) +++ head/sys/contrib/ena-com/ena_com.c Wed Nov 18 14:50:12 2020 (r367795) @@ -449,19 +449,21 @@ static int ena_com_init_io_cq(struct ena_com_dev *ena_ size = io_cq->cdesc_entry_size_in_bytes * io_cq->q_depth; io_cq->bus = ena_dev->bus; - ENA_MEM_ALLOC_COHERENT_NODE(ena_dev->dmadev, - size, - io_cq->cdesc_addr.virt_addr, - io_cq->cdesc_addr.phys_addr, - io_cq->cdesc_addr.mem_handle, - ctx->numa_node, - prev_node); + ENA_MEM_ALLOC_COHERENT_NODE_ALIGNED(ena_dev->dmadev, + size, + io_cq->cdesc_addr.virt_addr, + io_cq->cdesc_addr.phys_addr, + io_cq->cdesc_addr.mem_handle, + ctx->numa_node, + prev_node, + ENA_CDESC_RING_SIZE_ALIGNMENT); if (!io_cq->cdesc_addr.virt_addr) { - ENA_MEM_ALLOC_COHERENT(ena_dev->dmadev, - size, - io_cq->cdesc_addr.virt_addr, - io_cq->cdesc_addr.phys_addr, - io_cq->cdesc_addr.mem_handle); + ENA_MEM_ALLOC_COHERENT_ALIGNED(ena_dev->dmadev, + size, + io_cq->cdesc_addr.virt_addr, + io_cq->cdesc_addr.phys_addr, + io_cq->cdesc_addr.mem_handle, + ENA_CDESC_RING_SIZE_ALIGNMENT); } if (!io_cq->cdesc_addr.virt_addr) { Modified: head/sys/contrib/ena-com/ena_com.h ============================================================================== --- head/sys/contrib/ena-com/ena_com.h Wed Nov 18 14:32:48 2020 (r367794) +++ head/sys/contrib/ena-com/ena_com.h Wed Nov 18 14:50:12 2020 (r367795) @@ -51,6 +51,8 @@ #define ADMIN_CQ_SIZE(depth) ((depth) * sizeof(struct ena_admin_acq_entry)) #define ADMIN_AENQ_SIZE(depth) ((depth) * sizeof(struct ena_admin_aenq_entry)) +#define ENA_CDESC_RING_SIZE_ALIGNMENT (1 << 12) /* 4K */ + /*****************************************************************************/ /*****************************************************************************/ /* ENA adaptive interrupt moderation settings */ Modified: head/sys/contrib/ena-com/ena_plat.h ============================================================================== --- head/sys/contrib/ena-com/ena_plat.h Wed Nov 18 14:32:48 2020 (r367794) +++ head/sys/contrib/ena-com/ena_plat.h Wed Nov 18 14:50:12 2020 (r367795) @@ -106,6 +106,8 @@ extern struct ena_bus_space ebs; #define ENA_ADMQ (1 << 8) /* Detailed info about admin queue. */ #define ENA_NETMAP (1 << 9) /* Detailed info about netmap. */ +#define DEFAULT_ALLOC_ALIGNMENT 8 + extern int ena_log_level; #define ena_trace_raw(level, fmt, args...) \ @@ -285,7 +287,7 @@ typedef uint64_t ena_time_t; void ena_dmamap_callback(void *arg, bus_dma_segment_t *segs, int nseg, int error); int ena_dma_alloc(device_t dmadev, bus_size_t size, ena_mem_handle_t *dma, - int mapflags); + int mapflags, bus_size_t alignment); static inline uint32_t ena_reg_read32(struct ena_bus *bus, bus_size_t offset) @@ -313,19 +315,29 @@ ena_reg_read32(struct ena_bus *bus, bus_size_t offset) (void)(size); \ free(ptr, M_DEVBUF); \ } while (0) -#define ENA_MEM_ALLOC_COHERENT_NODE(dmadev, size, virt, phys, handle, node, \ - dev_node) \ +#define ENA_MEM_ALLOC_COHERENT_NODE_ALIGNED(dmadev, size, virt, phys, \ + handle, node, dev_node, alignment) \ do { \ ((virt) = NULL); \ (void)(dev_node); \ } while (0) -#define ENA_MEM_ALLOC_COHERENT(dmadev, size, virt, phys, dma) \ +#define ENA_MEM_ALLOC_COHERENT_NODE(dmadev, size, virt, phys, handle, \ + node, dev_node) \ + ENA_MEM_ALLOC_COHERENT_NODE_ALIGNED(dmadev, size, virt, \ + phys, handle, node, dev_node, DEFAULT_ALLOC_ALIGNMENT) + +#define ENA_MEM_ALLOC_COHERENT_ALIGNED(dmadev, size, virt, phys, dma, \ + alignment) \ do { \ - ena_dma_alloc((dmadev), (size), &(dma), 0); \ + ena_dma_alloc((dmadev), (size), &(dma), 0, alignment); \ (virt) = (void *)(dma).vaddr; \ (phys) = (dma).paddr; \ } while (0) + +#define ENA_MEM_ALLOC_COHERENT(dmadev, size, virt, phys, dma) \ + ENA_MEM_ALLOC_COHERENT_ALIGNED(dmadev, size, virt, \ + phys, dma, DEFAULT_ALLOC_ALIGNMENT) #define ENA_MEM_FREE_COHERENT(dmadev, size, virt, phys, dma) \ do { \ Modified: head/sys/dev/ena/ena.c ============================================================================== --- head/sys/dev/ena/ena.c Wed Nov 18 14:32:48 2020 (r367794) +++ head/sys/dev/ena/ena.c Wed Nov 18 14:50:12 2020 (r367795) @@ -200,7 +200,7 @@ ena_dmamap_callback(void *arg, bus_dma_segment_t *segs int ena_dma_alloc(device_t dmadev, bus_size_t size, - ena_mem_handle_t *dma , int mapflags) + ena_mem_handle_t *dma, int mapflags, bus_size_t alignment) { struct ena_adapter* adapter = device_get_softc(dmadev); uint32_t maxsize; @@ -214,7 +214,7 @@ ena_dma_alloc(device_t dmadev, bus_size_t size, dma_space_addr = BUS_SPACE_MAXADDR; error = bus_dma_tag_create(bus_get_dma_tag(dmadev), /* parent */ - 8, 0, /* alignment, bounds */ + alignment, 0, /* alignment, bounds */ dma_space_addr, /* lowaddr of exclusion window */ BUS_SPACE_MAXADDR,/* highaddr of exclusion window */ NULL, NULL, /* filter, filterarg */