From owner-svn-src-stable@freebsd.org Tue Aug 27 04:16:43 2019 Return-Path: Delivered-To: svn-src-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4D365C8D3E; Tue, 27 Aug 2019 04:16:43 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46HbDC1Ndbz4LZP; Tue, 27 Aug 2019 04:16:43 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 1107C191EF; Tue, 27 Aug 2019 04:16:43 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x7R4GgQ6007246; Tue, 27 Aug 2019 04:16:42 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x7R4GgSr007244; Tue, 27 Aug 2019 04:16:42 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201908270416.x7R4GgSr007244@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Tue, 27 Aug 2019 04:16:42 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-12@freebsd.org Subject: svn commit: r351539 - in stable/12: share/man/man4 sys/dev/ntb X-SVN-Group: stable-12 X-SVN-Commit-Author: mav X-SVN-Commit-Paths: in stable/12: share/man/man4 sys/dev/ntb X-SVN-Commit-Revision: 351539 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Aug 2019 04:16:43 -0000 Author: mav Date: Tue Aug 27 04:16:42 2019 New Revision: 351539 URL: https://svnweb.freebsd.org/changeset/base/351539 Log: MFC r351105, r351107: Limit memory window usage in ntb_transport to 256MB. This adds safety net for the case of misconfigured NTB with too big memory window, for which we may be unable to allocate a memory buffer, which does not make much sense for the network interface. While there, fix the code to really work with asymmetric window sizes setup. This makes driver just print warning message on boot instead of hanging if too large memory window is configured. Modified: stable/12/share/man/man4/ntb_transport.4 stable/12/sys/dev/ntb/ntb_transport.c Directory Properties: stable/12/ (props changed) Modified: stable/12/share/man/man4/ntb_transport.4 ============================================================================== --- stable/12/share/man/man4/ntb_transport.4 Tue Aug 27 04:15:21 2019 (r351538) +++ stable/12/share/man/man4/ntb_transport.4 Tue Aug 27 04:16:42 2019 (r351539) @@ -1,5 +1,5 @@ .\" -.\" Copyright (c) 2016-2017 Alexander Motin +.\" Copyright (c) 2016-2019 Alexander Motin .\" All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without @@ -25,7 +25,7 @@ .\" .\" $FreeBSD$ .\" -.Dd September 2, 2017 +.Dd August 15, 2019 .Dt NTB_TRANSPORT 4 .Os .Sh NAME @@ -51,6 +51,10 @@ The following tunables are settable from the .It Va hw.ntb_transport.debug_level Driver debug level. The default value is 0, higher means more verbose. +.It Va hw.ntb_transport.max_mw_size +Limits maximum memory window usage. +Allocation of big physically contiguous memory buffer may be a problem, +while too big buffers makes no much sense for low latency network interface. .It Va hint.ntb_transport. Ns Ar X Ns Va .config Configures a set of the transport consumers, separated by commas. Each consumer can be configured as: "[][:]", where: Modified: stable/12/sys/dev/ntb/ntb_transport.c ============================================================================== --- stable/12/sys/dev/ntb/ntb_transport.c Tue Aug 27 04:15:21 2019 (r351538) +++ stable/12/sys/dev/ntb/ntb_transport.c Tue Aug 27 04:16:42 2019 (r351539) @@ -81,7 +81,7 @@ SYSCTL_UINT(_hw_ntb_transport, OID_AUTO, debug_level, static unsigned transport_mtu = 0x10000; -static uint64_t max_mw_size; +static uint64_t max_mw_size = 256*1024*1024; SYSCTL_UQUAD(_hw_ntb_transport, OID_AUTO, max_mw_size, CTLFLAG_RDTUN, &max_mw_size, 0, "If enabled (non-zero), limit the size of large memory windows. " "Both sides of the NTB MUST set the same value here."); @@ -177,14 +177,17 @@ struct ntb_transport_mw { size_t xlat_align; size_t xlat_align_size; bus_addr_t addr_limit; - /* Tx buff is off vbase / phys_addr */ + /* Tx buff is vbase / phys_addr / tx_size */ caddr_t vbase; - size_t buff_size; - /* Rx buff is off virt_addr / dma_addr */ + size_t tx_size; + /* Rx buff is virt_addr / dma_addr / rx_size */ bus_dma_tag_t dma_tag; bus_dmamap_t dma_map; caddr_t virt_addr; bus_addr_t dma_addr; + size_t rx_size; + /* rx_size increased to size alignment requirements of the hardware. */ + size_t buff_size; }; struct ntb_transport_child { @@ -331,7 +334,7 @@ ntb_transport_attach(device_t dev) struct ntb_transport_child **cpp = &nt->child; struct ntb_transport_child *nc; struct ntb_transport_mw *mw; - uint64_t db_bitmap, size; + uint64_t db_bitmap; int rc, i, db_count, spad_count, qp, qpu, qpo, qpt; char cfg[128] = ""; char buf[32]; @@ -374,6 +377,15 @@ ntb_transport_attach(device_t dev) if (rc != 0) goto err; + mw->tx_size = mw->phys_size; + if (max_mw_size != 0 && mw->tx_size > max_mw_size) { + device_printf(dev, "Memory window %d limited from " + "%ju to %ju\n", i, (uintmax_t)mw->phys_size, + max_mw_size); + mw->tx_size = max_mw_size; + } + + mw->rx_size = 0; mw->buff_size = 0; mw->virt_addr = NULL; mw->dma_addr = 0; @@ -388,10 +400,7 @@ ntb_transport_attach(device_t dev) * that NTB windows are symmetric and this allocation remain, * but even if not, we will just reallocate it later. */ - size = mw->phys_size; - if (max_mw_size != 0 && size > max_mw_size) - size = max_mw_size; - ntb_set_mw(nt, i, size); + ntb_set_mw(nt, i, mw->tx_size); } qpu = 0; @@ -556,7 +565,7 @@ ntb_transport_init_queue(struct ntb_transport_ctx *nt, struct ntb_transport_mw *mw; struct ntb_transport_qp *qp; vm_paddr_t mw_base; - uint64_t mw_size, qp_offset; + uint64_t qp_offset; size_t tx_size; unsigned num_qps_mw, mw_num, mw_count; @@ -578,9 +587,8 @@ ntb_transport_init_queue(struct ntb_transport_ctx *nt, num_qps_mw = nt->qp_count / mw_count; mw_base = mw->phys_addr; - mw_size = mw->phys_size; - tx_size = mw_size / num_qps_mw; + tx_size = mw->tx_size / num_qps_mw; qp_offset = tx_size * (qp_num / mw_count); qp->tx_mw = mw->vbase + qp_offset; @@ -1103,11 +1111,7 @@ ntb_transport_link_work(void *arg) /* send the local info, in the opposite order of the way we read it */ for (i = 0; i < nt->mw_count; i++) { - size = nt->mw_vec[i].phys_size; - - if (max_mw_size != 0 && size > max_mw_size) - size = max_mw_size; - + size = nt->mw_vec[i].tx_size; ntb_peer_spad_write(dev, NTBT_MW0_SZ_HIGH + (i * 2), size >> 32); ntb_peer_spad_write(dev, NTBT_MW0_SZ_LOW + (i * 2), size); @@ -1139,6 +1143,7 @@ ntb_transport_link_work(void *arg) val64 |= val; mw = &nt->mw_vec[i]; + mw->rx_size = val64; val64 = roundup(val64, mw->xlat_align_size); if (mw->buff_size != val64) { @@ -1288,7 +1293,7 @@ ntb_transport_setup_qp_mw(struct ntb_transport_ctx *nt else num_qps_mw = nt->qp_count / mw_count; - rx_size = mw->buff_size / num_qps_mw; + rx_size = mw->rx_size / num_qps_mw; qp->rx_buff = mw->virt_addr + rx_size * (qp_num / mw_count); rx_size -= sizeof(struct ntb_rx_info);