From owner-freebsd-net@FreeBSD.ORG  Sun Oct 21 18:21:54 2012
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
 by hub.freebsd.org (Postfix) with ESMTP id 3289C9D6;
 Sun, 21 Oct 2012 18:21:54 +0000 (UTC)
 (envelope-from melifaro@FreeBSD.org)
Received: from mail.ipfw.ru (unknown [IPv6:2a01:4f8:120:6141::2])
 by mx1.freebsd.org (Postfix) with ESMTP id B4F908FC0C;
 Sun, 21 Oct 2012 18:21:53 +0000 (UTC)
Received: from v6.mpls.in ([2a02:978:2::5] helo=ws.su29.net)
 by mail.ipfw.ru with esmtpsa (TLSv1:CAMELLIA256-SHA:256)
 (Exim 4.76 (FreeBSD)) (envelope-from <melifaro@FreeBSD.org>)
 id 1TQ0DA-000Ct9-Fk; Sun, 21 Oct 2012 22:25:20 +0400
Message-ID: <50843D39.5000003@FreeBSD.org>
Date: Sun, 21 Oct 2012 22:21:45 +0400
From: "Alexander V. Chernikov" <melifaro@FreeBSD.org>
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64;
 rv:9.0) Gecko/20120121 Thunderbird/9.0
MIME-Version: 1.0
To: Ryan Stone <rysto32@gmail.com>
Subject: Re: ixgbe & if_igb RX ring locking
References: <5079A9A1.4070403@FreeBSD.org> <20121015162926.GV89655@FreeBSD.org>
 <CAFMmRNxT=GWxc6r7B81ENjzwJmfea3016Sh-DxJEGBwybM0QwQ@mail.gmail.com>
 <507D4F11.2030704@FreeBSD.org>
 <CAFMmRNy1h86ZXAH4oK+LOK6_VeTsbo=d6oYk669bisKfHbJ56A@mail.gmail.com>
 <20121016124733.GC89655@glebius.int.ru>
 <CAFMmRNzA23r_a0nrnsUN8ZoHGL5c7OF1-nCqWpN42FLc50d7eg@mail.gmail.com>
In-Reply-To: <CAFMmRNzA23r_a0nrnsUN8ZoHGL5c7OF1-nCqWpN42FLc50d7eg@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Cc: Jack Vogel <jfvogel@gmail.com>, net@freebsd.org
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sun, 21 Oct 2012 18:21:54 -0000

On 16.10.2012 17:20, Ryan Stone wrote:
> On Tue, Oct 16, 2012 at 8:47 AM, Gleb Smirnoff<glebius@freebsd.org>  wrote:
>> Can you please provide hints how can SIOCADDMULTI lead to obtaining RX
>> lock in the stock driver?
>
> It doesn't.  But it does acquire the core lock, and the core lock is
> acquired before the RX lock (in ixgbe_init, for instance).

I still can't get how any of these can happen.

We basically has 2 related places where ixgbe driver interact with OS:

1) forwarding path (ixgbe_rxeof via ithread / taskq)
2) ioctl path

I'm currently talking about multicast only.

So, first can lead us to:

ixgbe_rxeof() (rxr->rx_mtx "ix:rx")
   ether_input()
     ...
     udp_input() (V_udbinfo "udp" READ lock)
       udp_input() (inp->inp_lock "udpinp" READ lock)
>

2) ioctl part can lead to the following locks to be held:
   IXGBE_CORE_LOCK() in every ioctl ("ix?:core" mtx)
     ixgbe_init_locked()
       ixgbe_setup_receive_structures()
         ixgbe_setup_receive_ring() (rxr->rx_mtx "ix:rx")

.. with any possibly locks prepended (like in_multi_mtx or any other)


 > Last time I checked(FreeBSD 8.2), this is not true.  The problematic
 > (and convoluted) ordering is:
 >
 > ix:rx ->  udp ->  udpinp ->  in_multi_mtx ->  ix:core ->  ix:rx
 >
 > udp ->  udpinp ->  in_multi_mtx is defined in subr_witness.c.
Actually subr_witness defines the following:
1) udp -> udpinp -> so_snd
2) udpinp -> in_multi_mtx -> .. -> ..

 >
 > ix:core ->  ix:rx is fairly obvious, and happens in places like 
ixgbe_init.
 >
Yes. But how can I trigger reverted order (e.g. ixgbe_init[_locked]() 
(actually, ioctl) on packet input ?

(And if such place really exists we should consider fixing it, but not 
the driver side)

 > ix:rx ->  udp is also fairly obvious (here's one backtrace):
The same question, where "udp" -> "ix:rx" can happen ?