From owner-freebsd-net@FreeBSD.ORG Fri Oct 19 07:53:07 2012 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 73401864; Fri, 19 Oct 2012 07:53:07 +0000 (UTC) (envelope-from fabien.thomas@netasq.com) Received: from work.netasq.com (gwlille.netasq.com [91.212.116.1]) by mx1.freebsd.org (Postfix) with ESMTP id E61AA8FC14; Fri, 19 Oct 2012 07:53:06 +0000 (UTC) Received: from [10.2.1.1] (unknown [10.2.1.1]) by work.netasq.com (Postfix) with ESMTPSA id 54D6E2705764; Fri, 19 Oct 2012 09:53:05 +0200 (CEST) Subject: Re: ixgbe & if_igb RX ring locking Mime-Version: 1.0 (Apple Message framework v1283) From: Fabien Thomas In-Reply-To: Date: Fri, 19 Oct 2012 09:53:07 +0200 Message-Id: <390AF360-AEC3-495E-881A-1ACCFEF42815@netasq.com> References: <5079A9A1.4070403@FreeBSD.org> <20121013182223.GA73341@onelab2.iet.unipi.it> <5080020E.1010603@networx.ch> To: Jack Vogel X-Mailer: Apple Mail (2.1283) Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "Alexander V. Chernikov" , Luigi Rizzo , net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Oct 2012 07:53:07 -0000 Le 18 oct. 2012 =E0 20:09, Jack Vogel a =E9crit : > On Thu, Oct 18, 2012 at 6:20 AM, Andre Oppermann = wrote: >=20 >> On 13.10.2012 20:22, Luigi Rizzo wrote: >>=20 >>> On Sat, Oct 13, 2012 at 09:49:21PM +0400, Alexander V. Chernikov = wrote: >>>=20 >>>> Hello list! >>>>=20 >>>>=20 >>>> Packets receiving code for both ixgbe and if_igb looks like the >>>> following: >>>>=20 >>>>=20 >>>> ixgbe_msix_que >>>>=20 >>>> -- ixgbe_rxeof() >>>> { >>>> IXGBE_RX_LOCK(rxr); >>>> while >>>> { >>>> get_packet; >>>>=20 >>>> -- ixgbe_rx_input() >>>> { >>>> ++ IXGBE_RX_UNLOCK(rxr); >>>> if_input(packet); >>>> ++ IXGBE_RX_LOCK(rxr); >>>> } >>>>=20 >>>> } >>>> IXGBE_RX_UNLOCK(rxr); >>>> } >>>>=20 >>>> Lines marked with ++ appeared in r209068(igb) and r217593(ixgbe). >>>>=20 >>>> These lines probably do LORs masking (if any) well. >>>> However, such change introduce quite significant performance drop: >>>>=20 >>>> On my routing setup (nearly the same from previous -Intel 10G = thread in >>>> -net) adding lock/unlock causes 2.8MPPS decrease to 2.3MPPS which = is >>>> nearly 20%. >>>>=20 >>>=20 >>> one option could be (same as it is done in the timer >>> routine in dummynet) to build a list of all the packets >>> that need to be sent to if_input(), and then call >>> if_input with the entire list outside the lock. >>>=20 >>> It would be even easier if we modify the various *_input() >>> routines to handle a list of mbufs instead of just one. >>>=20 >>=20 >> Not really. You'd just run into tons of layering complexity. >> Somewhere the decomposition and serialization has to be done. >>=20 >> Perhaps the right place is to dequeue a batch of packets from >> the HW ring and then have a task/thread send it up the stack >> one by one. >>=20 >=20 > I was thinking about how to code this, something like what I did with > the refresh routine, in any case I will experiment with it. This modified version for mq polling create a list of packet that are = injected later (mc is the list). = http://www.gitorious.org/~fabient/freebsd/fabient-freebsd/blobs/work/polln= g_mq_stable_8/sys/dev/ixgbe/ixgbe.c#line4615 >=20 > Jack > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"