Date: Mon, 15 Oct 2012 09:39:24 -0700 From: Jack Vogel <jfvogel@gmail.com> To: Gleb Smirnoff <glebius@freebsd.org> Cc: "Alexander V. Chernikov" <melifaro@freebsd.org>, net@freebsd.org Subject: Re: ixgbe & if_igb RX ring locking Message-ID: <CAFOYbcmkt%2B3-f7BzfaStUHBFEO0TzJXhYES=42ovkencPoKHJA@mail.gmail.com> In-Reply-To: <20121015162926.GV89655@FreeBSD.org> References: <5079A9A1.4070403@FreeBSD.org> <20121015162926.GV89655@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Oct 15, 2012 at 9:29 AM, Gleb Smirnoff <glebius@freebsd.org> wrote:
> On Sat, Oct 13, 2012 at 09:49:21PM +0400, Alexander V. Chernikov wrote:
> A> Packets receiving code for both ixgbe and if_igb looks like the
> following:
> A> ixgbe_msix_que
> A>
> A> -- ixgbe_rxeof()
> A> {
> A> IXGBE_RX_LOCK(rxr);
> A> while
> A> {
> A> get_packet;
> A>
> A> -- ixgbe_rx_input()
> A> {
> A> ++ IXGBE_RX_UNLOCK(rxr);
> A> if_input(packet);
> A> ++ IXGBE_RX_LOCK(rxr);
> A> }
> A>
> A> }
> A> IXGBE_RX_UNLOCK(rxr);
> A> }
> A>
> A> Lines marked with ++ appeared in r209068(igb) and r217593(ixgbe).
> A>
> A> These lines probably do LORs masking (if any) well.
> A> However, such change introduce quite significant performance drop:
> A>
> A> On my routing setup (nearly the same from previous -Intel 10G thread in
> A> -net) adding lock/unlock causes 2.8MPPS decrease to 2.3MPPS which is
> A> nearly 20%.
> A>
> A> So my questions are:
> A>
> A> Can any real LORs happen in some complex setup? (I can't imagine any).
> A> If so: maybe we can somehow avoid/workaround such cases? (and consider
> A> removing those locks).
>
> To me this unlock/lock looks like a legacy from times, when the driver
> had a single mutex for both TX and RX parts.
>
> And removing this re-locking in foo_rxeof() was one of the aims for
> separate
> TX/RX locking.
>
> Really, lurking through history shows that once driver had split its
> locking
> to separate RX and TX part, these unlock/lock was removed. However, later
> this unlock/lock was added back:
>
>
> http://svnweb.freebsd.org/base/head/sys/dev/e1000/if_igb.c?revision=209068&view=markup
>
> , without any comments for the reason it is added back.
>
> I did not want to add it back, there were problems that constrained me to
do so, although its
been some time, I'd be happy to do some testing again without and see.
Jack
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFOYbcmkt%2B3-f7BzfaStUHBFEO0TzJXhYES=42ovkencPoKHJA>