From owner-svn-src-user@FreeBSD.ORG Tue Nov 6 23:42:54 2012 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8A8BCCCA; Tue, 6 Nov 2012 23:42:54 +0000 (UTC) (envelope-from andre@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) by mx1.freebsd.org (Postfix) with ESMTP id 700A48FC0A; Tue, 6 Nov 2012 23:42:54 +0000 (UTC) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.5/8.14.5) with ESMTP id qA6Ngslk089542; Tue, 6 Nov 2012 23:42:54 GMT (envelope-from andre@svn.freebsd.org) Received: (from andre@localhost) by svn.freebsd.org (8.14.5/8.14.5/Submit) id qA6NgsIa089539; Tue, 6 Nov 2012 23:42:54 GMT (envelope-from andre@svn.freebsd.org) Message-Id: <201211062342.qA6NgsIa089539@svn.freebsd.org> From: Andre Oppermann Date: Tue, 6 Nov 2012 23:42:54 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org Subject: svn commit: r242682 - user/andre/tcp_workqueue/sys/dev/bge X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Nov 2012 23:42:54 -0000 Author: andre Date: Tue Nov 6 23:42:54 2012 New Revision: 242682 URL: http://svnweb.freebsd.org/changeset/base/242682 Log: Change the bge(4) driver to use an interrupt filter and an ithread to handle RX and TX packets. Taskqueue is completely removed. The interrupt filter runs in interrupt context and only masks the NIC interrupt. Or for bge(4) the interrupt is only one-shot anyway so nothing has to be done. The step is left in place for reference. When the filter returns FILTER_SCHEDULE_THREAD the correspoding ithread is run and does the heavy packet lifting and DMA descriptor refilling. The entire setup of the interrupt filter and ithread is done with bus_setup_intr(). To prevent live-lock the ithread tries to yield after an arbitrary number of packets, 10 in this case. The function maybe_yield() takes a look at the number of consumed cycles/ticks and decides whether the ithread still has quantum left or not. If not it gets put onto the run queue and continues after other threads had their fair share. This work isn't complete yet and bge_ithr[_msix] and bge_rxeof need better coordination to be able to run in polling mode under load. Locking may be longer be necessary as there is only ever one ithread that services the DMA queues at least for RX. Depending on how TX is triggered locking may still be required. Theory of operation: intr_filter() disables the interrupt and lets the ithread get scheduled ithr_rxeof() does: while (new packets available in DMA ring) { dequeue and process packets; after a couple packets call maybe_yield(); after a couple packets re-sync DMA ring with HW; /* continue as long as new packets are available. */ } re-enable interrupt; return; This gives us polling efficiency under load while not going into live-lock and at the same time interrupt fast low latency when not under load. This change is not tested yet and committed as checkpoint. Discussed with and explained by: attilio Modified: user/andre/tcp_workqueue/sys/dev/bge/if_bge.c user/andre/tcp_workqueue/sys/dev/bge/if_bgereg.h Modified: user/andre/tcp_workqueue/sys/dev/bge/if_bge.c ============================================================================== --- user/andre/tcp_workqueue/sys/dev/bge/if_bge.c Tue Nov 6 23:25:06 2012 (r242681) +++ user/andre/tcp_workqueue/sys/dev/bge/if_bge.c Tue Nov 6 23:42:54 2012 (r242682) @@ -78,9 +78,9 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include -#include #include #include @@ -403,9 +403,9 @@ static struct mbuf *bge_setup_tso(struct uint16_t *, uint16_t *); static int bge_encap(struct bge_softc *, struct mbuf **, uint32_t *); -static void bge_intr(void *); -static int bge_msi_intr(void *); -static void bge_intr_task(void *, int); +static int bge_intr_filter(void *); +static void bge_ithr_msix(void *); +static void bge_ithr(void *); static void bge_start_locked(struct ifnet *); static void bge_start(struct ifnet *); static int bge_ioctl(struct ifnet *, u_long, caddr_t); @@ -3221,7 +3221,6 @@ bge_attach(device_t dev) sc->bge_dev = dev; BGE_LOCK_INIT(sc, device_get_nameunit(dev)); - TASK_INIT(&sc->bge_intr_task, 0, bge_intr_task, sc); callout_init_mtx(&sc->bge_stat_ch, &sc->bge_mtx, 0); /* @@ -3837,23 +3836,13 @@ again: /* Take advantage of single-shot MSI. */ CSR_WRITE_4(sc, BGE_MSI_MODE, CSR_READ_4(sc, BGE_MSI_MODE) & ~BGE_MSIMODE_ONE_SHOT_DISABLE); - sc->bge_tq = taskqueue_create_fast("bge_taskq", M_WAITOK, - taskqueue_thread_enqueue, &sc->bge_tq); - if (sc->bge_tq == NULL) { - device_printf(dev, "could not create taskqueue.\n"); - ether_ifdetach(ifp); - error = ENOMEM; - goto fail; - } - taskqueue_start_threads(&sc->bge_tq, 1, PI_NET, "%s taskq", - device_get_nameunit(sc->bge_dev)); error = bus_setup_intr(dev, sc->bge_irq, - INTR_TYPE_NET | INTR_MPSAFE, bge_msi_intr, NULL, sc, - &sc->bge_intrhand); + INTR_TYPE_NET | INTR_MPSAFE, bge_intr_filter, + bge_ithr_msix, sc, &sc->bge_intrhand); } else error = bus_setup_intr(dev, sc->bge_irq, - INTR_TYPE_NET | INTR_MPSAFE, NULL, bge_intr, sc, - &sc->bge_intrhand); + INTR_TYPE_NET | INTR_MPSAFE, bge_intr_filter, + bge_ithr, sc, &sc->bge_intrhand); if (error) { ether_ifdetach(ifp); @@ -3888,9 +3877,6 @@ bge_detach(device_t dev) callout_drain(&sc->bge_stat_ch); } - if (sc->bge_tq) - taskqueue_drain(sc->bge_tq, &sc->bge_intr_task); - if (sc->bge_flags & BGE_FLAG_TBI) { ifmedia_removeall(&sc->bge_ifmedia); } else { @@ -3910,9 +3896,6 @@ bge_release_resources(struct bge_softc * dev = sc->bge_dev; - if (sc->bge_tq != NULL) - taskqueue_free(sc->bge_tq); - if (sc->bge_intrhand != NULL) bus_teardown_intr(dev, sc->bge_irq, sc->bge_intrhand); @@ -4221,6 +4204,7 @@ bge_rxeof(struct bge_softc *sc, uint16_t { struct ifnet *ifp; int rx_npkts = 0, stdcnt = 0, jumbocnt = 0; + int pkts = 0; uint16_t rx_cons; rx_cons = sc->bge_rx_saved_considx; @@ -4325,6 +4309,10 @@ bge_rxeof(struct bge_softc *sc, uint16_t if (holdlck != 0) { BGE_UNLOCK(sc); (*ifp->if_input)(ifp, m); + if (++pkts > 10) { + maybe_yield(); + pkts = 0; + } BGE_LOCK(sc); } else (*ifp->if_input)(ifp, m); @@ -4499,7 +4487,7 @@ bge_poll(struct ifnet *ifp, enum poll_cm #endif /* DEVICE_POLLING */ static int -bge_msi_intr(void *arg) +bge_intr_filter(void *arg) { struct bge_softc *sc; @@ -4508,12 +4496,11 @@ bge_msi_intr(void *arg) * This interrupt is not shared and controller already * disabled further interrupt. */ - taskqueue_enqueue(sc->bge_tq, &sc->bge_intr_task); - return (FILTER_HANDLED); + return (FILTER_SCHEDULE_THREAD); } static void -bge_intr_task(void *arg, int pending) +bge_ithr_msix(void *arg) { struct bge_softc *sc; struct ifnet *ifp; @@ -4567,10 +4554,11 @@ bge_intr_task(void *arg, int pending) bge_start_locked(ifp); } BGE_UNLOCK(sc); + return; } static void -bge_intr(void *xsc) +bge_ithr(void *xsc) { struct bge_softc *sc; struct ifnet *ifp; @@ -4648,6 +4636,7 @@ bge_intr(void *xsc) bge_start_locked(ifp); BGE_UNLOCK(sc); + return; } static void Modified: user/andre/tcp_workqueue/sys/dev/bge/if_bgereg.h ============================================================================== --- user/andre/tcp_workqueue/sys/dev/bge/if_bgereg.h Tue Nov 6 23:25:06 2012 (r242681) +++ user/andre/tcp_workqueue/sys/dev/bge/if_bgereg.h Tue Nov 6 23:42:54 2012 (r242682) @@ -3024,8 +3024,6 @@ struct bge_softc { int rxcycles; #endif /* DEVICE_POLLING */ struct bge_mac_stats bge_mac_stats; - struct task bge_intr_task; - struct taskqueue *bge_tq; }; #define BGE_LOCK_INIT(_sc, _name) \