Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 18 Oct 2012 11:09:16 -0700
From:      Jack Vogel <jfvogel@gmail.com>
To:        Andre Oppermann <oppermann@networx.ch>
Cc:        "Alexander V. Chernikov" <melifaro@freebsd.org>, Luigi Rizzo <rizzo@iet.unipi.it>, net@freebsd.org
Subject:   Re: ixgbe & if_igb RX ring locking
Message-ID:  <CAFOYbcnT0tT0S6Q8Eos9oEDoSSZZZSr8GD5f9oO4bgpo-t6vNg@mail.gmail.com>
In-Reply-To: <5080020E.1010603@networx.ch>
References:  <5079A9A1.4070403@FreeBSD.org> <20121013182223.GA73341@onelab2.iet.unipi.it> <5080020E.1010603@networx.ch>

next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Oct 18, 2012 at 6:20 AM, Andre Oppermann <oppermann@networx.ch>wrote:

> On 13.10.2012 20:22, Luigi Rizzo wrote:
>
>> On Sat, Oct 13, 2012 at 09:49:21PM +0400, Alexander V. Chernikov wrote:
>>
>>> Hello list!
>>>
>>>
>>> Packets receiving code for both ixgbe and if_igb looks like the
>>> following:
>>>
>>>
>>> ixgbe_msix_que
>>>
>>> -- ixgbe_rxeof()
>>>     {
>>>        IXGBE_RX_LOCK(rxr);
>>>          while
>>>          {
>>>             get_packet;
>>>
>>>             -- ixgbe_rx_input()
>>>                {
>>>                   ++ IXGBE_RX_UNLOCK(rxr);
>>>                   if_input(packet);
>>>                   ++ IXGBE_RX_LOCK(rxr);
>>>                }
>>>
>>>          }
>>>        IXGBE_RX_UNLOCK(rxr);
>>>      }
>>>
>>> Lines marked with ++ appeared in r209068(igb) and r217593(ixgbe).
>>>
>>> These lines probably do LORs masking (if any) well.
>>> However, such change introduce quite significant performance drop:
>>>
>>> On my routing setup (nearly the same from previous -Intel 10G thread in
>>> -net) adding lock/unlock causes 2.8MPPS decrease to 2.3MPPS which is
>>> nearly 20%.
>>>
>>
>> one option could be (same as it is done in the timer
>> routine in dummynet) to build a list of all the packets
>> that need to be sent to if_input(), and then call
>> if_input with the entire list outside the lock.
>>
>> It would be even easier if we modify the various *_input()
>> routines to handle a list of mbufs instead of just one.
>>
>
> Not really. You'd just run into tons of layering complexity.
> Somewhere the decomposition and serialization has to be done.
>
> Perhaps the right place is to dequeue a batch of packets from
> the HW ring and then have a task/thread send it up the stack
> one by one.
>

I was thinking about how to code this, something like what I did with
the refresh routine, in any case I will experiment with it.

Jack



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFOYbcnT0tT0S6Q8Eos9oEDoSSZZZSr8GD5f9oO4bgpo-t6vNg>