From owner-freebsd-net@FreeBSD.ORG  Mon Oct 15 16:29:28 2012
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: net@FreeBSD.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
 by hub.freebsd.org (Postfix) with ESMTP id 8262FC8E;
 Mon, 15 Oct 2012 16:29:28 +0000 (UTC)
 (envelope-from glebius@FreeBSD.org)
Received: from cell.glebius.int.ru (glebius.int.ru [81.19.64.117])
 by mx1.freebsd.org (Postfix) with ESMTP id ED7628FC14;
 Mon, 15 Oct 2012 16:29:27 +0000 (UTC)
Received: from cell.glebius.int.ru (localhost [127.0.0.1])
 by cell.glebius.int.ru (8.14.5/8.14.5) with ESMTP id q9FGTQx0020725;
 Mon, 15 Oct 2012 20:29:26 +0400 (MSK)
 (envelope-from glebius@FreeBSD.org)
Received: (from glebius@localhost)
 by cell.glebius.int.ru (8.14.5/8.14.5/Submit) id q9FGTQCf020724;
 Mon, 15 Oct 2012 20:29:26 +0400 (MSK)
 (envelope-from glebius@FreeBSD.org)
X-Authentication-Warning: cell.glebius.int.ru: glebius set sender to
 glebius@FreeBSD.org using -f
Date: Mon, 15 Oct 2012 20:29:26 +0400
From: Gleb Smirnoff <glebius@FreeBSD.org>
To: "Alexander V. Chernikov" <melifaro@FreeBSD.org>
Subject: Re: ixgbe & if_igb RX ring locking
Message-ID: <20121015162926.GV89655@FreeBSD.org>
References: <5079A9A1.4070403@FreeBSD.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=koi8-r
Content-Disposition: inline
In-Reply-To: <5079A9A1.4070403@FreeBSD.org>
User-Agent: Mutt/1.5.21 (2010-09-15)
Cc: Jack Vogel <jfvogel@gmail.com>, net@FreeBSD.org
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 15 Oct 2012 16:29:28 -0000

On Sat, Oct 13, 2012 at 09:49:21PM +0400, Alexander V. Chernikov wrote:
A> Packets receiving code for both ixgbe and if_igb looks like the following:
A> ixgbe_msix_que
A> 
A> -- ixgbe_rxeof()
A>     {
A>        IXGBE_RX_LOCK(rxr);
A>          while
A>          {
A>             get_packet;
A> 
A>             -- ixgbe_rx_input()
A>                {
A>                   ++ IXGBE_RX_UNLOCK(rxr);
A>                   if_input(packet);
A>                   ++ IXGBE_RX_LOCK(rxr);
A>                }
A> 
A>          }
A>        IXGBE_RX_UNLOCK(rxr);
A>      }
A> 
A> Lines marked with ++ appeared in r209068(igb) and r217593(ixgbe).
A> 
A> These lines probably do LORs masking (if any) well.
A> However, such change introduce quite significant performance drop:
A> 
A> On my routing setup (nearly the same from previous -Intel 10G thread in 
A> -net) adding lock/unlock causes 2.8MPPS decrease to 2.3MPPS which is 
A> nearly 20%.
A> 
A> So my questions are:
A> 
A> Can any real LORs happen in some complex setup? (I can't imagine any).
A> If so: maybe we can somehow avoid/workaround such cases? (and consider 
A> removing those locks).

To me this unlock/lock looks like a legacy from times, when the driver
had a single mutex for both TX and RX parts.

And removing this re-locking in foo_rxeof() was one of the aims for separate
TX/RX locking.

Really, lurking through history shows that once driver had split its locking
to separate RX and TX part, these unlock/lock was removed. However, later
this unlock/lock was added back:

http://svnweb.freebsd.org/base/head/sys/dev/e1000/if_igb.c?revision=209068&view=markup

, without any comments for the reason it is added back.

-- 
Totus tuus, Glebius.