From owner-freebsd-net@FreeBSD.ORG  Tue Sep 25 20:19:03 2012
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 69DAF106564A;
	Tue, 25 Sep 2012 20:19:03 +0000 (UTC)
	(envelope-from vijju.singh@gmail.com)
Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com
	[209.85.215.182])
	by mx1.freebsd.org (Postfix) with ESMTP id B28A18FC0A;
	Tue, 25 Sep 2012 20:19:02 +0000 (UTC)
Received: by eaac10 with SMTP id c10so1175336eaa.13
	for <multiple recipients>; Tue, 25 Sep 2012 13:19:01 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type;
	bh=rzFmSPgrXFIQT6Fm1V4hfApbPbCWAtRGpnSXNQ34c68=;
	b=pRYVdd0OdpphLWlyLscwY89moTwBjYbZOUKlzRiZzjcXF0VtCUs3mWWap0DmS1kg+u
	ZQfwT2r6R9u16K3YCWzb5OTkzUIN8uGGcIeSrJe6US7oZDoqiPjEKSTWwdrFEhUu/BQZ
	AkVjko11id95S9fqzVdDCSabbiWrD7UxgtTEkrHiQGlUMVnvO/DgvoCepuBCWuKFL+1k
	tCTuHFpr1T2AaHo+A8L4/rdzXHvefq3er2fTSqdL/YsjyemuF5GGpP28YK2zi5GnfKyY
	H19Hf++fxu838mXwDz9m6eCWozYXX4jRH93U0EiRPwshgbuhAboE5vJYC4f/3yEtJ4bs
	kSxw==
MIME-Version: 1.0
Received: by 10.14.178.72 with SMTP id e48mr22522496eem.1.1348604341455; Tue,
	25 Sep 2012 13:19:01 -0700 (PDT)
Received: by 10.14.219.134 with HTTP; Tue, 25 Sep 2012 13:19:01 -0700 (PDT)
In-Reply-To: <201208170941.54482.jhb@freebsd.org>
References: <CALCNsJSSQSWV7vNVR-Sn8CPDKbUBBLpSH0b-HYMJo3SXvkOY=w@mail.gmail.com>
	<201208161736.47250.jhb@freebsd.org>
	<CAFOYbc=bXv_V4egO94ftkTFoLQ2uM50Sw3X5kgYoyKQcd0s31A@mail.gmail.com>
	<201208170941.54482.jhb@freebsd.org>
Date: Tue, 25 Sep 2012 13:19:01 -0700
Message-ID: <CALCNsJQ740ceDzpd5n7QAALn-uJ-GdWxPTkQJuMJUMTUGJjOUg@mail.gmail.com>
From: Vijay Singh <vijju.singh@gmail.com>
To: John Baldwin <jhb@freebsd.org>
Content-Type: text/plain; charset=ISO-8859-1
Cc: freebsd-net@freebsd.org, Jack Vogel <jfvogel@gmail.com>
Subject: Re: ixgbe rx & tx locks
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 25 Sep 2012 20:19:03 -0000

> Vijay, can you test this to see if it helps with your test case?
>
>> Jack

John, apologies for the delay. I have some data to share now.

With your patch, the transmit side lock contention is all gone.
However I still see receive side contention. I have MSI/X enabled,
with one hw queue per-port.

debug.lock.prof.stats:
lock                      	                  held_at 	
             contested_from 	      instances
e1b:rx(0)                 	(    sys/dev/ixgbe/ixgbe.c:4314) 	(
sys/dev/ixgbe/ixgbe.c:4249) 	  6814
e2b:rx(0)                 	(    sys/dev/ixgbe/ixgbe.c:4314) 	(
sys/dev/ixgbe/ixgbe.c:4249) 	  6962

These are from:

4297 static bool
4298 ixgbe_rxeof(struct ix_queue *que, int count)
4299 {
.....
4314         IXGBE_RX_LOCK(rxr);

and

4220 static __inline void
4221 ixgbe_rx_input(struct rx_ring *rxr, struct ifnet *ifp, struct
mbuf *m, u32 ptype)
4222 {
....
4247         IXGBE_RX_UNLOCK(rxr);
4248         (*ifp->if_input)(ifp, m);
4249         IXGBE_RX_LOCK(rxr);

So it seems like the interrupt handler is still contending with a
taskqueue handler on the receive side.

-vijay

PS: The interface names are custom.