From owner-svn-src-head@freebsd.org Fri Dec 23 03:40:14 2016 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 31371C8D056; Fri, 23 Dec 2016 03:40:14 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from mail.baldwin.cx (bigwig.baldwin.cx [96.47.65.170]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 01CD614A8; Fri, 23 Dec 2016 03:40:14 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by mail.baldwin.cx (Postfix) with ESMTPSA id 0EC0810AA64; Thu, 22 Dec 2016 22:40:13 -0500 (EST) From: John Baldwin To: Mark Johnston Cc: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: Re: svn commit: r310423 - head/sys/kern Date: Thu, 22 Dec 2016 16:40:38 -0800 Message-ID: <2625364.T1Fo1rRtxp@ralph.baldwin.cx> User-Agent: KMail/4.14.10 (FreeBSD/11.0-PRERELEASE; KDE/4.14.10; amd64; ; ) In-Reply-To: <20161222192601.GA78778@wkstn-mjohnston.west.isilon.com> References: <201612221751.uBMHpim4062786@repo.freebsd.org> <6562460.a4qdZuDa0s@ralph.baldwin.cx> <20161222192601.GA78778@wkstn-mjohnston.west.isilon.com> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.4.3 (mail.baldwin.cx); Thu, 22 Dec 2016 22:40:13 -0500 (EST) X-Virus-Scanned: clamav-milter 0.99.2 at mail.baldwin.cx X-Virus-Status: Clean X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Dec 2016 03:40:14 -0000 On Thursday, December 22, 2016 11:26:01 AM Mark Johnston wrote: > On Thu, Dec 22, 2016 at 10:39:12AM -0800, John Baldwin wrote: > > On Thursday, December 22, 2016 05:51:44 PM Mark Johnston wrote: > > > Author: markj > > > Date: Thu Dec 22 17:51:44 2016 > > > New Revision: 310423 > > > URL: https://svnweb.freebsd.org/changeset/base/310423 > > > > > > Log: > > > Revert part of r300109. > > > > > > The removal of TAILQ_FOREACH_SAFE introduced a small race: when the last > > > thread on a sleepqueue is awoken, it reclaims the sleepqueue and may begin > > > executing on a different CPU before sleepq_resume_thread() returns. This > > > leaves a window during which it may go back to sleep and incorrectly be > > > awoken again by the caller of sleepq_broadcast(). > > > > This is very subtle. > > :( That also means debugging this was a nice catch. :) > > The issue is that the last sleepq_resume_thread transfers > > ownership of 'sq' from the wait channel that the sleepq_broadcast has locked, > > to the thread being resumed. > > Right, that's what I meant by "reclaims the sleepqueue." One other > requirement for hitting the race is that the thread goes back to sleep > on a wait channel that hashes to a different sleepchain, else the > sleepchain lock held by the sleepq_broadcast() caller is, I believe, > sufficient to prevent the reuse of the sleepqueue before the loop has > terminated. > > > I thought about using a local TAILQ_HEAD and > > using TAILQ_CONCAT to move the list of threads out of the sleep queue and then > > walking that list. However, a comment explaining this transfer of ownership > > (and that we can't safely access 'sq' after the last thread is resumed) is > > probably sufficient (but necessary I think). Do you feel like adding one? > > How about: > > Index: subr_sleepqueue.c > =================================================================== > --- subr_sleepqueue.c (revision 310423) > +++ subr_sleepqueue.c (working copy) > @@ -892,7 +892,12 @@ > KASSERT(sq->sq_type == (flags & SLEEPQ_TYPE), > ("%s: mismatch between sleep/wakeup and cv_*", __func__)); > > - /* Resume all blocked threads on the sleep queue. */ > + /* > + * Resume all blocked threads on the sleep queue. The last thread will > + * be given ownership of sq and may re-enqueue itself before > + * sleepq_resume_thread() returns, so we must cache the "next" queue > + * item at the beginning of the final iteration. > + */ > wakeup_swapper = 0; > TAILQ_FOREACH_SAFE(td, &sq->sq_blocked[queue], td_slpq, tdn) { > thread_lock(td); That looks great, thanks! -- John Baldwin