From owner-freebsd-net@FreeBSD.ORG  Sat Oct 13 18:02:08 2012
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
 by hub.freebsd.org (Postfix) with ESMTP id CC14968C;
 Sat, 13 Oct 2012 18:02:08 +0000 (UTC)
 (envelope-from luigi@onelab2.iet.unipi.it)
Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238])
 by mx1.freebsd.org (Postfix) with ESMTP id 86CB98FC0A;
 Sat, 13 Oct 2012 18:02:08 +0000 (UTC)
Received: by onelab2.iet.unipi.it (Postfix, from userid 275)
 id E17637300A; Sat, 13 Oct 2012 20:22:23 +0200 (CEST)
Date: Sat, 13 Oct 2012 20:22:23 +0200
From: Luigi Rizzo <rizzo@iet.unipi.it>
To: "Alexander V. Chernikov" <melifaro@freebsd.org>
Subject: Re: ixgbe & if_igb RX ring locking
Message-ID: <20121013182223.GA73341@onelab2.iet.unipi.it>
References: <5079A9A1.4070403@FreeBSD.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <5079A9A1.4070403@FreeBSD.org>
User-Agent: Mutt/1.4.2.3i
Cc: Jack Vogel <jfvogel@gmail.com>, net@freebsd.org
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sat, 13 Oct 2012 18:02:08 -0000

On Sat, Oct 13, 2012 at 09:49:21PM +0400, Alexander V. Chernikov wrote:
> Hello list!
> 
> 
> Packets receiving code for both ixgbe and if_igb looks like the following:
> 
> 
> ixgbe_msix_que
> 
> -- ixgbe_rxeof()
>    {
>       IXGBE_RX_LOCK(rxr);
>         while
>         {
>            get_packet;
> 
>            -- ixgbe_rx_input()
>               {
>                  ++ IXGBE_RX_UNLOCK(rxr);
>                  if_input(packet);
>                  ++ IXGBE_RX_LOCK(rxr);
>               }
> 
>         }
>       IXGBE_RX_UNLOCK(rxr);
>     }
> 
> Lines marked with ++ appeared in r209068(igb) and r217593(ixgbe).
> 
> These lines probably do LORs masking (if any) well.
> However, such change introduce quite significant performance drop:
> 
> On my routing setup (nearly the same from previous -Intel 10G thread in 
> -net) adding lock/unlock causes 2.8MPPS decrease to 2.3MPPS which is 
> nearly 20%.

one option could be (same as it is done in the timer
routine in dummynet) to build a list of all the packets
that need to be sent to if_input(), and then call
if_input with the entire list outside the lock.

It would be even easier if we modify the various *_input()
routines to handle a list of mbufs instead of just one.

cheers
luigi

> So my questions are:
> 
> Can any real LORs happen in some complex setup? (I can't imagine any).
> If so: maybe we can somehow avoid/workaround such cases? (and consider 
> removing those locks).
> 
> 
> 
> -- 
> WBR, Alexander
> 
> _______________________________________________
> freebsd-net@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"