From owner-freebsd-net@freebsd.org Wed Dec 7 10:09:49 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8763CC6A56F for ; Wed, 7 Dec 2016 10:09:49 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 02C371BA3 for ; Wed, 7 Dec 2016 10:09:48 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id uB7A9g8r016222 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 7 Dec 2016 12:09:42 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua uB7A9g8r016222 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id uB7A9gDc016221; Wed, 7 Dec 2016 12:09:42 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 7 Dec 2016 12:09:42 +0200 From: Konstantin Belousov To: Sepherosa Ziehau Cc: Chris Torek , "freebsd-net@freebsd.org" , rysto32@gmail.com Subject: Re: mutex usage in if_bridge vs other drivers Message-ID: <20161207100942.GU54029@kib.kiev.ua> References: <201612030736.uB37aArG057471@elf.torek.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.7.1 (2016-10-04) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Dec 2016 10:09:49 -0000 On Wed, Dec 07, 2016 at 05:41:51PM +0800, Sepherosa Ziehau wrote: > On Sat, Dec 3, 2016 at 3:36 PM, Chris Torek wrote: > >>... Dropping the lock is entirely the wrong thing to do -- as > >>you note, if we do, then the bridge members can change out from > >>under us. The only path forward is to use an sx lock, but ... > > [snip] > >>In code paths that modify the list of bridge members, hold both the > >>BRIDGE_LOCK and the new sx lock. In the transmit and receive paths, > >>nothing should change. > > > > That should work, but I hate the cost of obtaining two locks -- > > not so much the lock acquire/release cost, but the "mental cost" > > of keeping the lock ordering straight. > > > > The other option I thought of was gathering the list of members > > that need an ioctl done on them in one pass, then doing all the > > ioctls after dropping the lock. But that runs the risk of doing > > ioctls on interfaces that are no longer members of the bridge. > > > > I guess the "mental cost" is not that high, since the lock order > > requirement is guaranteed to be "sx lock first, then mtx lock" > > (because the other way around causes sleeping while holding a > > regular mutex!). > > This is what did for Hyper-V network drivers: > > while (sx_try_xlock(drv_conf_sx) == 0) > DELAY(1000); > > since that mainlock, i.e. configuration lock, is not on hot path for drivers. > > In this way, driver can sleep freely on IF_UP/DOWN/detach path, which > is convenient, though other paths e.g. promisc, add/delmulti still > requires busy-wait instead of sleeping. Consider what happens if some other thread already owns the conf sx lock and tries to acquire the mutex. If the thread blocks execution by whatever mean, be it lock primitive putting the thread off cpu, or thread executing spin loop (as in your case), reversing the order still causes the deadlock.