From owner-svn-src-all@freebsd.org Sat Sep 23 16:46:32 2017 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 40706E09126; Sat, 23 Sep 2017 16:46:32 +0000 (UTC) (envelope-from shurd@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1A41A7C9D6; Sat, 23 Sep 2017 16:46:32 +0000 (UTC) (envelope-from shurd@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v8NGkVHv040627; Sat, 23 Sep 2017 16:46:31 GMT (envelope-from shurd@FreeBSD.org) Received: (from shurd@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v8NGkVk5040625; Sat, 23 Sep 2017 16:46:31 GMT (envelope-from shurd@FreeBSD.org) Message-Id: <201709231646.v8NGkVk5040625@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: shurd set sender to shurd@FreeBSD.org using -f From: Stephen Hurd Date: Sat, 23 Sep 2017 16:46:31 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r323954 - head/sys/net X-SVN-Group: head X-SVN-Commit-Author: shurd X-SVN-Commit-Paths: head/sys/net X-SVN-Commit-Revision: 323954 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Sep 2017 16:46:32 -0000 Author: shurd Date: Sat Sep 23 16:46:30 2017 New Revision: 323954 URL: https://svnweb.freebsd.org/changeset/base/323954 Log: Have ifmp_ring_enqueue() abdicate instead of switch to a consumer Move TX out of the enqueue() path. As a result, we need to have ifmp_ring_check_drainage() pick up from the abdicate state. We also need to either enqueue the TX task, or check drainage after calling ifmp_ring_enqueue() to ensure it's sent. This change results in a 30% small packet forwarding improvement. Reviewed by: olivier, sbruno Approved by: sbruno (mentor) Sponsored by: Limelight Networks Differential Revision: https://reviews.freebsd.org/D12439 Modified: head/sys/net/iflib.c head/sys/net/mp_ring.c Modified: head/sys/net/iflib.c ============================================================================== --- head/sys/net/iflib.c Sat Sep 23 13:37:02 2017 (r323953) +++ head/sys/net/iflib.c Sat Sep 23 16:46:30 2017 (r323954) @@ -3515,8 +3515,7 @@ _task_fn_tx(void *context) } if (txq->ift_db_pending) ifmp_ring_enqueue(txq->ift_br, (void **)&txq, 1, TX_BATCH_SIZE); - else - ifmp_ring_check_drainage(txq->ift_br, TX_BATCH_SIZE); + ifmp_ring_check_drainage(txq->ift_br, TX_BATCH_SIZE); if (ctx->ifc_flags & IFC_LEGACY) IFDI_INTR_ENABLE(ctx); else { @@ -3718,16 +3717,14 @@ iflib_if_transmit(if_t ifp, struct mbuf *m) DBG_COUNTER_INC(tx_seen); err = ifmp_ring_enqueue(txq->ift_br, (void **)&m, 1, TX_BATCH_SIZE); + GROUPTASK_ENQUEUE(&txq->ift_task); if (err) { - GROUPTASK_ENQUEUE(&txq->ift_task); /* support forthcoming later */ #ifdef DRIVER_BACKPRESSURE txq->ift_closed = TRUE; #endif ifmp_ring_check_drainage(txq->ift_br, TX_BATCH_SIZE); m_freem(m); - } else if (TXQ_AVAIL(txq) < (txq->ift_size >> 1)) { - GROUPTASK_ENQUEUE(&txq->ift_task); } return (err); Modified: head/sys/net/mp_ring.c ============================================================================== --- head/sys/net/mp_ring.c Sat Sep 23 13:37:02 2017 (r323953) +++ head/sys/net/mp_ring.c Sat Sep 23 16:46:30 2017 (r323954) @@ -454,18 +454,12 @@ ifmp_ring_enqueue(struct ifmp_ring *r, void **items, i do { os.state = ns.state = r->state; ns.pidx_tail = pidx_stop; - ns.flags = BUSY; + if (os.flags == IDLE) + ns.flags = ABDICATED; } while (atomic_cmpset_rel_64(&r->state, os.state, ns.state) == 0); critical_exit(); counter_u64_add(r->enqueues, n); - /* - * Turn into a consumer if some other thread isn't active as a consumer - * already. - */ - if (os.flags != BUSY) - drain_ring_lockless(r, ns, os.flags, budget); - return (0); } #endif @@ -476,7 +470,9 @@ ifmp_ring_check_drainage(struct ifmp_ring *r, int budg union ring_state os, ns; os.state = r->state; - if (os.flags != STALLED || os.pidx_head != os.pidx_tail || r->can_drain(r) == 0) + if ((os.flags != STALLED && os.flags != ABDICATED) || // Only continue in STALLED and ABDICATED + os.pidx_head != os.pidx_tail || // Require work to be available + (os.flags != ABDICATED && r->can_drain(r) == 0)) // Can either drain, or everyone left return; MPASS(os.cidx != os.pidx_tail); /* implied by STALLED */