From owner-freebsd-net@freebsd.org Mon Jun 20 09:58:24 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A23BFA79A02 for ; Mon, 20 Jun 2016 09:58:24 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 8D69923F8 for ; Mon, 20 Jun 2016 09:58:24 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id 85801A799FE; Mon, 20 Jun 2016 09:58:24 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 850C9A799FA; Mon, 20 Jun 2016 09:58:24 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebi.us (glebi.us [96.95.210.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "cell.glebi.us", Issuer "cell.glebi.us" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id BB90A23F5; Mon, 20 Jun 2016 09:58:23 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebi.us (localhost [127.0.0.1]) by cell.glebi.us (8.15.2/8.15.2) with ESMTPS id u5K9wM5B094519 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 20 Jun 2016 02:58:22 -0700 (PDT) (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by cell.glebi.us (8.15.2/8.15.2/Submit) id u5K9wM4P094518; Mon, 20 Jun 2016 02:58:22 -0700 (PDT) (envelope-from glebius@FreeBSD.org) X-Authentication-Warning: cell.glebi.us: glebius set sender to glebius@FreeBSD.org using -f Date: Mon, 20 Jun 2016 02:58:22 -0700 From: Gleb Smirnoff To: Julien Charbon Cc: rrs@FreeBSD.org, hselasky@FreeBSD.org, net@FreeBSD.org, current@FreeBSD.org Subject: Re: panic with tcp timers Message-ID: <20160620095822.GJ1076@FreeBSD.org> References: <20160617045319.GE1076@FreeBSD.org> <1f28844b-b4ea-b544-3892-811f2be327b9@freebsd.org> <20160620073917.GI1076@FreeBSD.org> <1d18d0e2-3e42-cb26-928c-2989d0751884@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1d18d0e2-3e42-cb26-928c-2989d0751884@freebsd.org> User-Agent: Mutt/1.6.1 (2016-04-27) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Jun 2016 09:58:24 -0000 On Mon, Jun 20, 2016 at 11:55:55AM +0200, Julien Charbon wrote: J> > On Fri, Jun 17, 2016 at 11:27:39AM +0200, Julien Charbon wrote: J> > J> > Comparing stable/10 and head, I see two changes that could J> > J> > affect that: J> > J> > J> > J> > - callout_async_drain J> > J> > - switch to READ lock for inp info in tcp timers J> > J> > J> > J> > That's why you are in To, Julien and Hans :) J> > J> > J> > J> > We continue investigating, and I will keep you updated. J> > J> > However, any help is welcome. I can share cores. J> > J> > Now, spending some time with cores and adding a bunch of J> > extra CTRs, I have a sequence of events that lead to the J> > panic. In short, the bug is in the callout system. It seems J> > to be not relevant to the callout_async_drain, at least for J> > now. The transition to READ lock unmasked the problem, that's J> > why NetflixBSD 10 doesn't panic. J> > J> > The panic requires heavy contention on the TCP info lock. J> > J> > [CPU 1] the callout fires, tcp_timer_keep entered J> > [CPU 1] blocks on INP_INFO_RLOCK(&V_tcbinfo); J> > [CPU 2] schedules the callout J> > [CPU 2] tcp_discardcb called J> > [CPU 2] callout successfully canceled J> > [CPU 2] tcpcb freed J> > [CPU 1] unblocks... panic J> > J> > When the lock was WLOCK, all contenders were resumed in a J> > sequence they came to the lock. Now, that they are readers, J> > once the lock is released, readers are resumed in a "random" J> > order, and this allows tcp_discardcb to go before the old J> > running callout, and this unmasks the panic. J> J> Highly interesting. I should be able to reproduce that (will be useful J> for testing the corresponding fix). J> J> Fix proposal: If callout_async_drain() returns 0 (fail) (instead of 1 J> (success) here) when the callout cancellation is a success _but_ the J> callout is current running, that should fix it. J> J> For the history: It comes back to my old callout question: J> J> Does _callout_stop_safe() is allowed to return 1 (success) even if the J> callout is still currently running; a.k.a. it is not because you J> successfully cancelled a callout that the callout is not currently running. J> J> We did propose a patch to make _callout_stop_safe() returns 0 (fail) J> when the callout is currently running: J> J> callout_stop() should return 0 when the callout is currently being J> serviced and indeed unstoppable J> https://reviews.freebsd.org/differential/changeset/?ref=62513&whitespace=ignore-most J> J> But this change impacted too many old code paths and was interesting J> only for TCP timers and thus was abandoned. The fix I am working on now is doing exactly that. callout_reset must return 0 if the callout is currently running. What are the old paths impacted? -- Totus tuus, Glebius.