Date: Fri, 19 Oct 2012 09:53:07 +0200 From: Fabien Thomas <fabien.thomas@netasq.com> To: Jack Vogel <jfvogel@gmail.com> Cc: "Alexander V. Chernikov" <melifaro@freebsd.org>, Luigi Rizzo <rizzo@iet.unipi.it>, net@freebsd.org Subject: Re: ixgbe & if_igb RX ring locking Message-ID: <390AF360-AEC3-495E-881A-1ACCFEF42815@netasq.com> In-Reply-To: <CAFOYbcnT0tT0S6Q8Eos9oEDoSSZZZSr8GD5f9oO4bgpo-t6vNg@mail.gmail.com> References: <5079A9A1.4070403@FreeBSD.org> <20121013182223.GA73341@onelab2.iet.unipi.it> <5080020E.1010603@networx.ch> <CAFOYbcnT0tT0S6Q8Eos9oEDoSSZZZSr8GD5f9oO4bgpo-t6vNg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Le 18 oct. 2012 =E0 20:09, Jack Vogel a =E9crit :
> On Thu, Oct 18, 2012 at 6:20 AM, Andre Oppermann =
<oppermann@networx.ch>wrote:
>=20
>> On 13.10.2012 20:22, Luigi Rizzo wrote:
>>=20
>>> On Sat, Oct 13, 2012 at 09:49:21PM +0400, Alexander V. Chernikov =
wrote:
>>>=20
>>>> Hello list!
>>>>=20
>>>>=20
>>>> Packets receiving code for both ixgbe and if_igb looks like the
>>>> following:
>>>>=20
>>>>=20
>>>> ixgbe_msix_que
>>>>=20
>>>> -- ixgbe_rxeof()
>>>> {
>>>> IXGBE_RX_LOCK(rxr);
>>>> while
>>>> {
>>>> get_packet;
>>>>=20
>>>> -- ixgbe_rx_input()
>>>> {
>>>> ++ IXGBE_RX_UNLOCK(rxr);
>>>> if_input(packet);
>>>> ++ IXGBE_RX_LOCK(rxr);
>>>> }
>>>>=20
>>>> }
>>>> IXGBE_RX_UNLOCK(rxr);
>>>> }
>>>>=20
>>>> Lines marked with ++ appeared in r209068(igb) and r217593(ixgbe).
>>>>=20
>>>> These lines probably do LORs masking (if any) well.
>>>> However, such change introduce quite significant performance drop:
>>>>=20
>>>> On my routing setup (nearly the same from previous -Intel 10G =
thread in
>>>> -net) adding lock/unlock causes 2.8MPPS decrease to 2.3MPPS which =
is
>>>> nearly 20%.
>>>>=20
>>>=20
>>> one option could be (same as it is done in the timer
>>> routine in dummynet) to build a list of all the packets
>>> that need to be sent to if_input(), and then call
>>> if_input with the entire list outside the lock.
>>>=20
>>> It would be even easier if we modify the various *_input()
>>> routines to handle a list of mbufs instead of just one.
>>>=20
>>=20
>> Not really. You'd just run into tons of layering complexity.
>> Somewhere the decomposition and serialization has to be done.
>>=20
>> Perhaps the right place is to dequeue a batch of packets from
>> the HW ring and then have a task/thread send it up the stack
>> one by one.
>>=20
>=20
> I was thinking about how to code this, something like what I did with
> the refresh routine, in any case I will experiment with it.
This modified version for mq polling create a list of packet that are =
injected later (mc is the list).
=
http://www.gitorious.org/~fabient/freebsd/fabient-freebsd/blobs/work/polln=
g_mq_stable_8/sys/dev/ixgbe/ixgbe.c#line4615
>=20
> Jack
> _______________________________________________
> freebsd-net@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?390AF360-AEC3-495E-881A-1ACCFEF42815>