From owner-freebsd-net@FreeBSD.ORG  Thu Oct 18 13:20:20 2012
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
 by hub.freebsd.org (Postfix) with ESMTP id 7F017E12
 for <net@freebsd.org>; Thu, 18 Oct 2012 13:20:20 +0000 (UTC)
 (envelope-from oppermann@networx.ch)
Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2])
 by mx1.freebsd.org (Postfix) with ESMTP id D06F58FC0C
 for <net@freebsd.org>; Thu, 18 Oct 2012 13:20:19 +0000 (UTC)
Received: (qmail 13162 invoked from network); 18 Oct 2012 14:59:13 -0000
Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2])
 (envelope-sender <oppermann@networx.ch>)
 by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP
 for <rizzo@iet.unipi.it>; 18 Oct 2012 14:59:13 -0000
Message-ID: <5080020E.1010603@networx.ch>
Date: Thu, 18 Oct 2012 15:20:14 +0200
From: Andre Oppermann <oppermann@networx.ch>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64;
 rv:16.0) Gecko/20121010 Thunderbird/16.0.1
MIME-Version: 1.0
To: Luigi Rizzo <rizzo@iet.unipi.it>
Subject: Re: ixgbe & if_igb RX ring locking
References: <5079A9A1.4070403@FreeBSD.org>
 <20121013182223.GA73341@onelab2.iet.unipi.it>
In-Reply-To: <20121013182223.GA73341@onelab2.iet.unipi.it>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Cc: "Alexander V. Chernikov" <melifaro@freebsd.org>,
 Jack Vogel <jfvogel@gmail.com>, net@freebsd.org
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 18 Oct 2012 13:20:20 -0000

On 13.10.2012 20:22, Luigi Rizzo wrote:
> On Sat, Oct 13, 2012 at 09:49:21PM +0400, Alexander V. Chernikov wrote:
>> Hello list!
>>
>>
>> Packets receiving code for both ixgbe and if_igb looks like the following:
>>
>>
>> ixgbe_msix_que
>>
>> -- ixgbe_rxeof()
>>     {
>>        IXGBE_RX_LOCK(rxr);
>>          while
>>          {
>>             get_packet;
>>
>>             -- ixgbe_rx_input()
>>                {
>>                   ++ IXGBE_RX_UNLOCK(rxr);
>>                   if_input(packet);
>>                   ++ IXGBE_RX_LOCK(rxr);
>>                }
>>
>>          }
>>        IXGBE_RX_UNLOCK(rxr);
>>      }
>>
>> Lines marked with ++ appeared in r209068(igb) and r217593(ixgbe).
>>
>> These lines probably do LORs masking (if any) well.
>> However, such change introduce quite significant performance drop:
>>
>> On my routing setup (nearly the same from previous -Intel 10G thread in
>> -net) adding lock/unlock causes 2.8MPPS decrease to 2.3MPPS which is
>> nearly 20%.
>
> one option could be (same as it is done in the timer
> routine in dummynet) to build a list of all the packets
> that need to be sent to if_input(), and then call
> if_input with the entire list outside the lock.
>
> It would be even easier if we modify the various *_input()
> routines to handle a list of mbufs instead of just one.

Not really. You'd just run into tons of layering complexity.
Somewhere the decomposition and serialization has to be done.

Perhaps the right place is to dequeue a batch of packets from
the HW ring and then have a task/thread send it up the stack
one by one.

-- 
Andre