From owner-freebsd-net@FreeBSD.ORG  Wed Oct  1 20:27:24 2014
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
 (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id 08A86F6;
 Wed,  1 Oct 2014 20:27:24 +0000 (UTC)
Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238])
 by mx1.freebsd.org (Postfix) with ESMTP id BCEAB1C3;
 Wed,  1 Oct 2014 20:27:23 +0000 (UTC)
Received: by onelab2.iet.unipi.it (Postfix, from userid 275)
 id 43E2E7300A; Wed,  1 Oct 2014 22:32:23 +0200 (CEST)
Date: Wed, 1 Oct 2014 22:32:23 +0200
From: Luigi Rizzo <rizzo@iet.unipi.it>
To: Adrian Chadd <adrian@freebsd.org>,
 "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>,
 "Alexander V. Chernikov" <melifaro@freebsd.org>,
 Elof Ofel <elofu17@hotmail.com>, jfvogel@gmail.com
Subject: Re: individual queue blocking entire rx unit on ixgbe (Re: How do
 I balance bandwidth over several virtual NICs?)
Message-ID: <20141001203223.GA12122@onelab2.iet.unipi.it>
References: <CA+hQ2+hYY7AWE7jJWAg8uYER2LRSPz8TcVPuamsRBLj_WQyakw@mail.gmail.com>
 <CAJ-Vmo=xiZtKkQEV30Y_c7a6uWD6PbuqzmSSDG12J=w8Z45biA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAJ-Vmo=xiZtKkQEV30Y_c7a6uWD6PbuqzmSSDG12J=w8Z45biA@mail.gmail.com>
User-Agent: Mutt/1.5.20 (2009-06-14)
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 01 Oct 2014 20:27:24 -0000

On Wed, Oct 01, 2014 at 10:37:35AM -0700, Adrian Chadd wrote:
> Hi,
> 
> Try this on -HEAD. There were some recent fixes to ixgbe that haven't
> been RFCed.

I don't have a way to test this on HEAD.

Do you know if there is any change that could be related ?

On 10.1 beta 3 I noticed is that when I open a single queue on an ixgbe
(and with incoming traffic), the rx unit stalls no matter what the
previous state of dev.ix.*.fc (i.e. the DROP_EN bits) or QDE are.

In this state, toggling DROP_EN has no effect, whereas something that seems
effective is setting the QDE bit(s) in the PFQDE register for all queues,
_after_ i open the device.

>From the above my take is the following:

- on NIC reset, the SRRCTL register starts at 0 including DROP_EN;
  same goes for PFQDE.QDE

- setting SRRCTL.DROP_EN must happen before the receive unit is started;

- conversely, toggling PFQDE.QDE has effect even when the receive
  unit has started

- sysctl dev.ix.*.fc sets/clear DROP_EN but at the wrong time
  (the right time seems to be the window between reset and start)

I am going to run more tests to figure out.

cheers
luigi