From owner-svn-src-all@freebsd.org Wed Dec 12 11:47:31 2018 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 80C0A1339D09; Wed, 12 Dec 2018 11:47:31 +0000 (UTC) (envelope-from hselasky@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0A5448B6CB; Wed, 12 Dec 2018 11:47:31 +0000 (UTC) (envelope-from hselasky@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id E2D9277C6; Wed, 12 Dec 2018 11:47:30 +0000 (UTC) (envelope-from hselasky@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id wBCBlUMJ057914; Wed, 12 Dec 2018 11:47:30 GMT (envelope-from hselasky@FreeBSD.org) Received: (from hselasky@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id wBCBlUIV057909; Wed, 12 Dec 2018 11:47:30 GMT (envelope-from hselasky@FreeBSD.org) Message-Id: <201812121147.wBCBlUIV057909@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: hselasky set sender to hselasky@FreeBSD.org using -f From: Hans Petter Selasky Date: Wed, 12 Dec 2018 11:47:30 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-12@freebsd.org Subject: svn commit: r341908 - stable/12/sys/dev/mlx4/mlx4_en X-SVN-Group: stable-12 X-SVN-Commit-Author: hselasky X-SVN-Commit-Paths: stable/12/sys/dev/mlx4/mlx4_en X-SVN-Commit-Revision: 341908 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 0A5448B6CB X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-0.71 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_SHORT(-0.71)[-0.715,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Dec 2018 11:47:31 -0000 Author: hselasky Date: Wed Dec 12 11:47:29 2018 New Revision: 341908 URL: https://svnweb.freebsd.org/changeset/base/341908 Log: MFC r341547: mlx4en: Remove the DRBR and associated logic in the transmit path. The hardware queues are deep enough currently and using the DRBR and associated callbacks only leads to more task switching in the TX path. The is also a race setting the queue_state which can lead to hung TX rings. Sponsored by: Mellanox Technologies Modified: stable/12/sys/dev/mlx4/mlx4_en/en.h stable/12/sys/dev/mlx4/mlx4_en/mlx4_en_cq.c stable/12/sys/dev/mlx4/mlx4_en/mlx4_en_netdev.c stable/12/sys/dev/mlx4/mlx4_en/mlx4_en_tx.c Directory Properties: stable/12/ (props changed) Modified: stable/12/sys/dev/mlx4/mlx4_en/en.h ============================================================================== --- stable/12/sys/dev/mlx4/mlx4_en/en.h Wed Dec 12 11:45:06 2018 (r341907) +++ stable/12/sys/dev/mlx4/mlx4_en/en.h Wed Dec 12 11:47:29 2018 (r341908) @@ -271,10 +271,8 @@ struct mlx4_en_tx_ring { u32 doorbell_qpn; u8 *buf; u16 poll_cnt; - int blocked; struct mlx4_en_tx_info *tx_info; u8 queue_index; - struct buf_ring *br; u32 last_nr_txbb; struct mlx4_qp qp; struct mlx4_qp_context context; @@ -806,7 +804,6 @@ int mlx4_en_create_rx_ring(struct mlx4_en_priv *priv, void mlx4_en_destroy_rx_ring(struct mlx4_en_priv *priv, struct mlx4_en_rx_ring **pring, u32 size, u16 stride); -void mlx4_en_tx_que(void *context, int pending); void mlx4_en_rx_que(void *context, int pending); int mlx4_en_activate_rx_rings(struct mlx4_en_priv *priv); void mlx4_en_deactivate_rx_ring(struct mlx4_en_priv *priv, Modified: stable/12/sys/dev/mlx4/mlx4_en/mlx4_en_cq.c ============================================================================== --- stable/12/sys/dev/mlx4/mlx4_en/mlx4_en_cq.c Wed Dec 12 11:45:06 2018 (r341907) +++ stable/12/sys/dev/mlx4/mlx4_en/mlx4_en_cq.c Wed Dec 12 11:47:29 2018 (r341908) @@ -44,6 +44,10 @@ static void mlx4_en_cq_event(struct mlx4_cq *cq, enum return; } +static void mlx4_en_tx_que(void *arg, int pending) +{ + +} int mlx4_en_create_cq(struct mlx4_en_priv *priv, struct mlx4_en_cq **pcq, Modified: stable/12/sys/dev/mlx4/mlx4_en/mlx4_en_netdev.c ============================================================================== --- stable/12/sys/dev/mlx4/mlx4_en/mlx4_en_netdev.c Wed Dec 12 11:45:06 2018 (r341907) +++ stable/12/sys/dev/mlx4/mlx4_en/mlx4_en_netdev.c Wed Dec 12 11:47:29 2018 (r341908) @@ -1574,9 +1574,12 @@ static void mlx4_en_restart(struct work_struct *work) if (priv->blocked == 0 || priv->port_up == 0) return; for (i = 0; i < priv->tx_ring_num; i++) { + int watchdog_time; + ring = priv->tx_ring[i]; - if (ring->blocked && - ring->watchdog_time + MLX4_EN_WATCHDOG_TIMEOUT < ticks) + watchdog_time = READ_ONCE(ring->watchdog_time); + if (watchdog_time != 0 && + time_after(ticks, ring->watchdog_time)) goto reset; } return; Modified: stable/12/sys/dev/mlx4/mlx4_en/mlx4_en_tx.c ============================================================================== --- stable/12/sys/dev/mlx4/mlx4_en/mlx4_en_tx.c Wed Dec 12 11:45:06 2018 (r341907) +++ stable/12/sys/dev/mlx4/mlx4_en/mlx4_en_tx.c Wed Dec 12 11:47:29 2018 (r341908) @@ -94,15 +94,6 @@ int mlx4_en_create_tx_ring(struct mlx4_en_priv *priv, mtx_init(&ring->tx_lock.m, "mlx4 tx", NULL, MTX_DEF); mtx_init(&ring->comp_lock.m, "mlx4 comp", NULL, MTX_DEF); - /* Allocate the buf ring */ - ring->br = buf_ring_alloc(MLX4_EN_DEF_TX_QUEUE_SIZE, M_DEVBUF, - M_WAITOK, &ring->tx_lock.m); - if (ring->br == NULL) { - en_err(priv, "Failed allocating tx_info ring\n"); - err = -ENOMEM; - goto err_free_dma_tag; - } - tmp = size * sizeof(struct mlx4_en_tx_info); ring->tx_info = kzalloc_node(tmp, GFP_KERNEL, node); if (!ring->tx_info) { @@ -190,8 +181,6 @@ err_dma_map: err_info: vfree(ring->tx_info); err_ring: - buf_ring_free(ring->br, M_DEVBUF); -err_free_dma_tag: bus_dma_tag_destroy(ring->dma_tag); done: kfree(ring); @@ -206,7 +195,6 @@ void mlx4_en_destroy_tx_ring(struct mlx4_en_priv *priv uint32_t x; en_dbg(DRV, priv, "Destroying tx ring, qpn: %d\n", ring->qpn); - buf_ring_free(ring->br, M_DEVBUF); if (ring->bf_enabled) mlx4_bf_free(mdev->dev, &ring->bf); mlx4_qp_remove(mdev->dev, &ring->qp); @@ -236,8 +224,8 @@ int mlx4_en_activate_tx_ring(struct mlx4_en_priv *priv ring->cons = 0xffffffff; ring->last_nr_txbb = 1; ring->poll_cnt = 0; - ring->blocked = 0; memset(ring->buf, 0, ring->buf_size); + ring->watchdog_time = 0; ring->qp_state = MLX4_QP_STATE_RST; ring->doorbell_qpn = ring->qp.qpn << 8; @@ -429,14 +417,6 @@ static int mlx4_en_process_tx_cq(struct net_device *de wmb(); ring->cons += txbbs_skipped; - /* Wakeup Tx queue if it was stopped and ring is not full */ - if (unlikely(ring->blocked) && !mlx4_en_tx_ring_is_full(ring)) { - ring->blocked = 0; - if (atomic_fetchadd_int(&priv->blocked, -1) == 1) - atomic_clear_int(&dev->if_drv_flags ,IFF_DRV_OACTIVE); - priv->port_stats.wake_queue++; - ring->wake_queue++; - } return (0); } @@ -682,15 +662,6 @@ static int mlx4_en_xmit(struct mlx4_en_priv *priv, int /* check if TX ring is full */ if (unlikely(mlx4_en_tx_ring_is_full(ring))) { - /* every full native Tx ring stops queue */ - if (ring->blocked == 0) - atomic_add_int(&priv->blocked, 1); - /* Set HW-queue-is-full flag */ - atomic_set_int(&ifp->if_drv_flags, IFF_DRV_OACTIVE); - priv->port_stats.queue_stopped++; - ring->blocked = 1; - ring->queue_stopped++; - /* Use interrupts to find out when queue opened */ mlx4_en_arm_cq(priv, priv->tx_cq[tx_ind]); return (ENOBUFS); @@ -956,81 +927,35 @@ tx_drop: } static int -mlx4_en_transmit_locked(struct ifnet *dev, int tx_ind, struct mbuf *m) +mlx4_en_transmit_locked(struct ifnet *ifp, int tx_ind, struct mbuf *mb) { - struct mlx4_en_priv *priv = netdev_priv(dev); - struct mlx4_en_tx_ring *ring; - struct mbuf *next; - int enqueued, err = 0; + struct mlx4_en_priv *priv = netdev_priv(ifp); + struct mlx4_en_tx_ring *ring = priv->tx_ring[tx_ind]; + int err = 0; - ring = priv->tx_ring[tx_ind]; - if ((dev->if_drv_flags & (IFF_DRV_RUNNING | IFF_DRV_OACTIVE)) != - IFF_DRV_RUNNING || priv->port_up == 0) { - if (m != NULL) - err = drbr_enqueue(dev, ring->br, m); - return (err); + if (unlikely((ifp->if_drv_flags & IFF_DRV_RUNNING) == 0 || + READ_ONCE(priv->port_up) == 0)) { + m_freem(mb); + return (ENETDOWN); } - enqueued = 0; - if (m != NULL) - /* - * If we can't insert mbuf into drbr, try to xmit anyway. - * We keep the error we got so we could return that after xmit. - */ - err = drbr_enqueue(dev, ring->br, m); - - /* Process the queue */ - while ((next = drbr_peek(dev, ring->br)) != NULL) { - if (mlx4_en_xmit(priv, tx_ind, &next) != 0) { - if (next == NULL) { - drbr_advance(dev, ring->br); - } else { - drbr_putback(dev, ring->br, next); - } - break; - } - drbr_advance(dev, ring->br); - enqueued++; - if ((dev->if_drv_flags & IFF_DRV_RUNNING) == 0) - break; + if (mlx4_en_xmit(priv, tx_ind, &mb) != 0) { + /* NOTE: m_freem() is NULL safe */ + m_freem(mb); + err = ENOBUFS; + if (ring->watchdog_time == 0) + ring->watchdog_time = ticks + MLX4_EN_WATCHDOG_TIMEOUT; + } else { + ring->watchdog_time = 0; } - - if (enqueued > 0) - ring->watchdog_time = ticks; - return (err); } -void -mlx4_en_tx_que(void *context, int pending) -{ - struct mlx4_en_tx_ring *ring; - struct mlx4_en_priv *priv; - struct net_device *dev; - struct mlx4_en_cq *cq; - int tx_ind; - cq = context; - dev = cq->dev; - priv = dev->if_softc; - tx_ind = cq->ring; - ring = priv->tx_ring[tx_ind]; - - if (priv->port_up != 0 && - (dev->if_drv_flags & IFF_DRV_RUNNING) != 0) { - mlx4_en_xmit_poll(priv, tx_ind); - spin_lock(&ring->tx_lock); - if (!drbr_empty(dev, ring->br)) - mlx4_en_transmit_locked(dev, tx_ind, NULL); - spin_unlock(&ring->tx_lock); - } -} - int mlx4_en_transmit(struct ifnet *dev, struct mbuf *m) { struct mlx4_en_priv *priv = netdev_priv(dev); struct mlx4_en_tx_ring *ring; - struct mlx4_en_cq *cq; int i, err = 0; if (priv->port_up == 0) { @@ -1047,17 +972,15 @@ mlx4_en_transmit(struct ifnet *dev, struct mbuf *m) } ring = priv->tx_ring[i]; - if (spin_trylock(&ring->tx_lock)) { - err = mlx4_en_transmit_locked(dev, i, m); - spin_unlock(&ring->tx_lock); - /* Poll CQ here */ - mlx4_en_xmit_poll(priv, i); - } else { - err = drbr_enqueue(dev, ring->br, m); - cq = priv->tx_cq[i]; - taskqueue_enqueue(cq->tq, &cq->cq_task); - } + spin_lock(&ring->tx_lock); + + err = mlx4_en_transmit_locked(dev, i, m); + spin_unlock(&ring->tx_lock); + + /* Poll CQ here */ + mlx4_en_xmit_poll(priv, i); + #if __FreeBSD_version >= 1100000 if (unlikely(err != 0)) if_inc_counter(dev, IFCOUNTER_IQDROPS, 1); @@ -1072,18 +995,9 @@ void mlx4_en_qflush(struct ifnet *dev) { struct mlx4_en_priv *priv = netdev_priv(dev); - struct mlx4_en_tx_ring *ring; - struct mbuf *m; if (priv->port_up == 0) return; - for (int i = 0; i < priv->tx_ring_num; i++) { - ring = priv->tx_ring[i]; - spin_lock(&ring->tx_lock); - while ((m = buf_ring_dequeue_sc(ring->br)) != NULL) - m_freem(m); - spin_unlock(&ring->tx_lock); - } if_qflush(dev); }