From owner-freebsd-net@FreeBSD.ORG Tue Apr 19 19:06:02 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DD463106566C; Tue, 19 Apr 2011 19:06:02 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: from mail-pv0-f182.google.com (mail-pv0-f182.google.com [74.125.83.182]) by mx1.freebsd.org (Postfix) with ESMTP id A09698FC12; Tue, 19 Apr 2011 19:06:02 +0000 (UTC) Received: by pvg11 with SMTP id 11so3622515pvg.13 for ; Tue, 19 Apr 2011 12:06:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=CybIeKwfCwnNMbZwT5xGskOM+c/VvSU5vtb5+ayftaE=; b=sykRipHXxqEKHlMUf9V/RRZ8Lfj7uPEmKwaP+JkdT1nGWRbY2CYivmzESKXjs+CFI6 8flzdAVOHCT1UIjI4XLhOlJLbIQc1GAXckGwPNyhDGFZTKc5GU1pHbi+AtK8Ex9yXmoR Yjn9Cxup28p1jKuRmmmsMRUmf+EEaGOI+ZwPY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=UNTvTT0T0sD52m9b91PQz9+xT2nBGqKGrl8wL2fXl5WNtp6ugZScA4tR5wa+X0R7pt Hbf3I3MFYIl9I23PuV4uD+gq6Ph6Vo5qY1MPMX5nba6Mnhwx1EjFIasILQgL3g8Ja8Bw +q1YlldSRKtwUT6q/yUzf3wYJtcpN6kYkWZJ0= MIME-Version: 1.0 Received: by 10.68.8.2 with SMTP id n2mr895209pba.245.1303239962149; Tue, 19 Apr 2011 12:06:02 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.68.41.101 with HTTP; Tue, 19 Apr 2011 12:06:02 -0700 (PDT) In-Reply-To: References: Date: Tue, 19 Apr 2011 21:06:02 +0200 X-Google-Sender-Auth: M_RukAfaAGRJWt5hER7u6sg2pAI Message-ID: From: "K. Macy" To: Freddie Cash Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Nikolay Denev , Ingo Flaschberger , freebsd-net@freebsd.org Subject: Re: Routing enhancement - reduce routing table locking X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Apr 2011 19:06:02 -0000 On Tue, Apr 19, 2011 at 8:19 PM, Freddie Cash wrote: > On Tue, Apr 19, 2011 at 7:42 AM, K. Macy wrote: >>> I'm not able to find IFNET_MULTIQUEUE in a recent 8.2-STABLE, is this s= omething >>> present only in HEAD? >> >> It looks like it is now EM_MULTIQUEUE. > > Just curious, how would one enable this to test it? =A0We have igb(4) > interfaces in our new storage boxes, and it would be interesting to > test whether or not it helps in our setup. > It should automatically allocate a queue per core up to the max supported. Post 8.0 it should be enabled by default for igb: #if __FreeBSD_version >=3D 800000 /* ** Multiqueue Transmit driver ** */ static int igb_mq_start(struct ifnet *ifp, struct mbuf *m) { struct adapter *adapter =3D ifp->if_softc; struct igb_queue *que; struct tx_ring *txr; int i =3D 0, err =3D 0; /* Which queue to use */ if ((m->m_flags & M_FLOWID) !=3D 0) i =3D m->m_pkthdr.flowid % adapter->num_queues; txr =3D &adapter->tx_rings[i]; que =3D &adapter->queues[i]; if (IGB_TX_TRYLOCK(txr)) { err =3D igb_mq_start_locked(ifp, txr, m); IGB_TX_UNLOCK(txr); } else { err =3D drbr_enqueue(ifp, txr->br, m); taskqueue_enqueue(que->tq, &que->que_task); } return (err); } static int igb_mq_start_locked(struct ifnet *ifp, struct tx_ring *txr, struct mbuf *m) { struct adapter *adapter =3D txr->adapter; struct mbuf *next; int err =3D 0, enq; IGB_TX_LOCK_ASSERT(txr); if ((ifp->if_drv_flags & (IFF_DRV_RUNNING | IFF_DRV_OACTIVE)) !=3D IFF_DRV_RUNNING || adapter->link_active =3D=3D 0) { if (m !=3D NULL) err =3D drbr_enqueue(ifp, txr->br, m); return (err); } /* Call cleanup if number of TX descriptors low */ if (txr->tx_avail <=3D IGB_TX_CLEANUP_THRESHOLD) igb_txeof(txr); enq =3D 0; if (m =3D=3D NULL) { next =3D drbr_dequeue(ifp, txr->br); } else if (drbr_needs_enqueue(ifp, txr->br)) { if ((err =3D drbr_enqueue(ifp, txr->br, m)) !=3D 0) return (err); next =3D drbr_dequeue(ifp, txr->br); } else next =3D m; /* Process the queue */ while (next !=3D NULL) { if ((err =3D igb_xmit(txr, &next)) !=3D 0) { if (next !=3D NULL) err =3D drbr_enqueue(ifp, txr->br, next); break; } enq++; drbr_stats_update(ifp, next->m_pkthdr.len, next->m_flags); ETHER_BPF_MTAP(ifp, next); if ((ifp->if_drv_flags & IFF_DRV_RUNNING) =3D=3D 0) break; if (txr->tx_avail <=3D IGB_TX_OP_THRESHOLD) { ifp->if_drv_flags |=3D IFF_DRV_OACTIVE; break; } next =3D drbr_dequeue(ifp, txr->br); } if (enq > 0) { /* Set the watchdog */ txr->queue_status =3D IGB_QUEUE_WORKING; txr->watchdog_time =3D ticks; } return (err); } I haven't tested this to make sure there aren't any hidden locking performance issues. Cheers