From owner-freebsd-net@FreeBSD.ORG  Sat Oct 13 17:50:20 2012
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: net@freebsd.org
Received: from mx2.freebsd.org (mx2.freebsd.org [69.147.83.53])
 by hub.freebsd.org (Postfix) with ESMTP id 8D8396D
 for <net@freebsd.org>; Sat, 13 Oct 2012 17:50:20 +0000 (UTC)
 (envelope-from melifaro@FreeBSD.org)
Received: from dhcp170-36-red.yandex.net (freefall.freebsd.org [8.8.178.135])
 by mx2.freebsd.org (Postfix) with ESMTP id DB0103B5A54;
 Sat, 13 Oct 2012 17:50:19 +0000 (UTC)
Message-ID: <5079A9A1.4070403@FreeBSD.org>
Date: Sat, 13 Oct 2012 21:49:21 +0400
From: "Alexander V. Chernikov" <melifaro@FreeBSD.org>
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64;
 rv:13.0) Gecko/20120627 Thunderbird/13.0.1
MIME-Version: 1.0
To: net@freebsd.org
Subject: ixgbe & if_igb RX ring locking
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Cc: Jack Vogel <jfvogel@gmail.com>
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sat, 13 Oct 2012 17:50:20 -0000

Hello list!


Packets receiving code for both ixgbe and if_igb looks like the following:


ixgbe_msix_que

-- ixgbe_rxeof()
    {
       IXGBE_RX_LOCK(rxr);
         while
         {
            get_packet;

            -- ixgbe_rx_input()
               {
                  ++ IXGBE_RX_UNLOCK(rxr);
                  if_input(packet);
                  ++ IXGBE_RX_LOCK(rxr);
               }

         }
       IXGBE_RX_UNLOCK(rxr);
     }

Lines marked with ++ appeared in r209068(igb) and r217593(ixgbe).

These lines probably do LORs masking (if any) well.
However, such change introduce quite significant performance drop:

On my routing setup (nearly the same from previous -Intel 10G thread in 
-net) adding lock/unlock causes 2.8MPPS decrease to 2.3MPPS which is 
nearly 20%.

So my questions are:

Can any real LORs happen in some complex setup? (I can't imagine any).
If so: maybe we can somehow avoid/workaround such cases? (and consider 
removing those locks).



-- 
WBR, Alexander