From owner-freebsd-net@FreeBSD.ORG Mon Oct 15 16:29:28 2012 Return-Path: Delivered-To: net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8262FC8E; Mon, 15 Oct 2012 16:29:28 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebius.int.ru (glebius.int.ru [81.19.64.117]) by mx1.freebsd.org (Postfix) with ESMTP id ED7628FC14; Mon, 15 Oct 2012 16:29:27 +0000 (UTC) Received: from cell.glebius.int.ru (localhost [127.0.0.1]) by cell.glebius.int.ru (8.14.5/8.14.5) with ESMTP id q9FGTQx0020725; Mon, 15 Oct 2012 20:29:26 +0400 (MSK) (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by cell.glebius.int.ru (8.14.5/8.14.5/Submit) id q9FGTQCf020724; Mon, 15 Oct 2012 20:29:26 +0400 (MSK) (envelope-from glebius@FreeBSD.org) X-Authentication-Warning: cell.glebius.int.ru: glebius set sender to glebius@FreeBSD.org using -f Date: Mon, 15 Oct 2012 20:29:26 +0400 From: Gleb Smirnoff To: "Alexander V. Chernikov" Subject: Re: ixgbe & if_igb RX ring locking Message-ID: <20121015162926.GV89655@FreeBSD.org> References: <5079A9A1.4070403@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline In-Reply-To: <5079A9A1.4070403@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Jack Vogel , net@FreeBSD.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2012 16:29:28 -0000 On Sat, Oct 13, 2012 at 09:49:21PM +0400, Alexander V. Chernikov wrote: A> Packets receiving code for both ixgbe and if_igb looks like the following: A> ixgbe_msix_que A> A> -- ixgbe_rxeof() A> { A> IXGBE_RX_LOCK(rxr); A> while A> { A> get_packet; A> A> -- ixgbe_rx_input() A> { A> ++ IXGBE_RX_UNLOCK(rxr); A> if_input(packet); A> ++ IXGBE_RX_LOCK(rxr); A> } A> A> } A> IXGBE_RX_UNLOCK(rxr); A> } A> A> Lines marked with ++ appeared in r209068(igb) and r217593(ixgbe). A> A> These lines probably do LORs masking (if any) well. A> However, such change introduce quite significant performance drop: A> A> On my routing setup (nearly the same from previous -Intel 10G thread in A> -net) adding lock/unlock causes 2.8MPPS decrease to 2.3MPPS which is A> nearly 20%. A> A> So my questions are: A> A> Can any real LORs happen in some complex setup? (I can't imagine any). A> If so: maybe we can somehow avoid/workaround such cases? (and consider A> removing those locks). To me this unlock/lock looks like a legacy from times, when the driver had a single mutex for both TX and RX parts. And removing this re-locking in foo_rxeof() was one of the aims for separate TX/RX locking. Really, lurking through history shows that once driver had split its locking to separate RX and TX part, these unlock/lock was removed. However, later this unlock/lock was added back: http://svnweb.freebsd.org/base/head/sys/dev/e1000/if_igb.c?revision=209068&view=markup , without any comments for the reason it is added back. -- Totus tuus, Glebius.