From owner-svn-src-all@FreeBSD.ORG Thu Dec 27 13:25:10 2012 Return-Path: Delivered-To: svn-src-all@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 17709DC1; Thu, 27 Dec 2012 13:25:10 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebius.int.ru (glebius.int.ru [81.19.69.10]) by mx1.freebsd.org (Postfix) with ESMTP id 8653D8FC17; Thu, 27 Dec 2012 13:25:08 +0000 (UTC) Received: from cell.glebius.int.ru (localhost [127.0.0.1]) by cell.glebius.int.ru (8.14.5/8.14.5) with ESMTP id qBRDP7cr013300; Thu, 27 Dec 2012 17:25:07 +0400 (MSK) (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by cell.glebius.int.ru (8.14.5/8.14.5/Submit) id qBRDP7eE013299; Thu, 27 Dec 2012 17:25:07 +0400 (MSK) (envelope-from glebius@FreeBSD.org) X-Authentication-Warning: cell.glebius.int.ru: glebius set sender to glebius@FreeBSD.org using -f Date: Thu, 27 Dec 2012 17:25:07 +0400 From: Gleb Smirnoff To: Attilio Rao Subject: Re: svn commit: r244732 - head/sys/sys Message-ID: <20121227132507.GY80310@FreeBSD.org> References: <201212271236.qBRCawuU078203@svn.freebsd.org> <20121227124657.GX80310@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: svn-src-head@FreeBSD.org, svn-src-all@FreeBSD.org, src-committers@FreeBSD.org X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Dec 2012 13:25:10 -0000 On Thu, Dec 27, 2012 at 04:55:22AM -0800, Attilio Rao wrote: A> On Thu, Dec 27, 2012 at 4:46 AM, Gleb Smirnoff wrote: A> > Attilio, A> > A> > On Thu, Dec 27, 2012 at 12:36:58PM +0000, Attilio Rao wrote: A> > A> Author: attilio A> > A> Date: Thu Dec 27 12:36:58 2012 A> > A> New Revision: 244732 A> > A> URL: http://svnweb.freebsd.org/changeset/base/244732 A> > A> A> > A> Log: A> > A> br_prod_tail and br_cons_tail members are used as barrier to A> > A> signal bug_ring ownership. However, instructions can be reordered A> > A> around members write leading to stale values for ie. br_prod_bufs. A> > A> A> > A> Use correct memory barriers to ensure proper ordering of the A> > A> ownership tokens updates. A> > A> A> > A> Sponsored by: EMC / Isilon storage division A> > A> MFC after: 2 weeks A> > A> > Have you profiled this? A> > A> > After this change the buf_ring actually gains its own hand-rolled A> > mutex: A> > A> > while (atomic_load_acq_32(&br->br_prod_tail) != prod_head) A> > cpu_spinwait(); A> > A> > The only difference with mutex(9) is that this one isn't monitored A> > by WITNESS. A> A> I think you are not correct. It doesn't disable interrupts (as A> spinlock do) and it doesn't sleep. A> So your analogy is completely off. A> A> Also, on x86 atomic_store_rel_*() is a simple write. The only thing A> that really changes is the atomic_load_acq_*() that introduces a A> locked instruction. This only thing, the locked instruction, affects performance a lot. I suspect strong forwarding degradation after your change. Can you please profile that? A> > The idea behind buf_ring was lockless storing and lockless fetching A> > from a ring and now this vanished. A> > A> > What does your change actually fixes except precise accounting of A> > br_prod_bufs that are actually unused and should be better garbage A> > collected rather than fixed? A> A> The write of br_prod_tail must happens as very last thing, also after A> the whole buf setup. The only way you can enforce this is with a A> memory barrier. I can double-check if we can garbage collect A> br_prod_bufs but this should not be enough yet. Do you have a core file that illustrates that a ring can get into inconsistent state? Your original commit message told only about stale values in br_prod_bufs, and as said the latter is absolutely useless. -- Totus tuus, Glebius.