From owner-freebsd-net@FreeBSD.ORG Mon Oct 15 16:39:26 2012 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 007C22DB; Mon, 15 Oct 2012 16:39:25 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-vb0-f54.google.com (mail-vb0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id 768448FC1B; Mon, 15 Oct 2012 16:39:25 +0000 (UTC) Received: by mail-vb0-f54.google.com with SMTP id v11so6975292vbm.13 for ; Mon, 15 Oct 2012 09:39:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=fCEc0qdOMh4veUmcKP6NL+cBTZTtL7osxYL4tQsWYPo=; b=DB8n0AD7ueC51bAjJKoNBrLpFeiRKj/7AZhVe4AFs/fcymNDWrpmmCq8qgoPv9nOBA Ar/duWzq+N7AQMp83PTIe95gs2HE5iTZiSo3WOmjaN1SrPY6VGbOqrJiyrZea+HpX3Cw 8lG31GqcUaAizpQtQjsEoAUrfC1knuOsrTFs6V9CSaQHbsEB6Vi/GfLXL7VBy2UIrAJd mzOFfkpolsr00DMNgUhGM3TqnSq6UczUY8I8Re6lY8QBFcV+v8+kArMGrnaFJqxlq93f 3WF7uJAviu/U2r1Szl6qQ3aZSzwujQukr52mG0zUcHz4/8UcSInsP4c7+8DUyzME9QBI VHrg== MIME-Version: 1.0 Received: by 10.58.1.101 with SMTP id 5mr7344899vel.40.1350319164396; Mon, 15 Oct 2012 09:39:24 -0700 (PDT) Received: by 10.58.68.8 with HTTP; Mon, 15 Oct 2012 09:39:24 -0700 (PDT) In-Reply-To: <20121015162926.GV89655@FreeBSD.org> References: <5079A9A1.4070403@FreeBSD.org> <20121015162926.GV89655@FreeBSD.org> Date: Mon, 15 Oct 2012 09:39:24 -0700 Message-ID: Subject: Re: ixgbe & if_igb RX ring locking From: Jack Vogel To: Gleb Smirnoff Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "Alexander V. Chernikov" , net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2012 16:39:26 -0000 On Mon, Oct 15, 2012 at 9:29 AM, Gleb Smirnoff wrote: > On Sat, Oct 13, 2012 at 09:49:21PM +0400, Alexander V. Chernikov wrote: > A> Packets receiving code for both ixgbe and if_igb looks like the > following: > A> ixgbe_msix_que > A> > A> -- ixgbe_rxeof() > A> { > A> IXGBE_RX_LOCK(rxr); > A> while > A> { > A> get_packet; > A> > A> -- ixgbe_rx_input() > A> { > A> ++ IXGBE_RX_UNLOCK(rxr); > A> if_input(packet); > A> ++ IXGBE_RX_LOCK(rxr); > A> } > A> > A> } > A> IXGBE_RX_UNLOCK(rxr); > A> } > A> > A> Lines marked with ++ appeared in r209068(igb) and r217593(ixgbe). > A> > A> These lines probably do LORs masking (if any) well. > A> However, such change introduce quite significant performance drop: > A> > A> On my routing setup (nearly the same from previous -Intel 10G thread in > A> -net) adding lock/unlock causes 2.8MPPS decrease to 2.3MPPS which is > A> nearly 20%. > A> > A> So my questions are: > A> > A> Can any real LORs happen in some complex setup? (I can't imagine any). > A> If so: maybe we can somehow avoid/workaround such cases? (and consider > A> removing those locks). > > To me this unlock/lock looks like a legacy from times, when the driver > had a single mutex for both TX and RX parts. > > And removing this re-locking in foo_rxeof() was one of the aims for > separate > TX/RX locking. > > Really, lurking through history shows that once driver had split its > locking > to separate RX and TX part, these unlock/lock was removed. However, later > this unlock/lock was added back: > > > http://svnweb.freebsd.org/base/head/sys/dev/e1000/if_igb.c?revision=209068&view=markup > > , without any comments for the reason it is added back. > > I did not want to add it back, there were problems that constrained me to do so, although its been some time, I'd be happy to do some testing again without and see. Jack