From owner-freebsd-hackers@freebsd.org Thu Aug 11 08:18:07 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7EF0ABB59AF for ; Thu, 11 Aug 2016 08:18:07 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1614415CE for ; Thu, 11 Aug 2016 08:18:06 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u7B8I21L017390 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Thu, 11 Aug 2016 11:18:02 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u7B8I21L017390 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u7B8I2Pk017389; Thu, 11 Aug 2016 11:18:02 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 11 Aug 2016 11:18:02 +0300 From: Konstantin Belousov To: Hooman Fazaeli Cc: Ryan Stone , FreeBSD Hackers Subject: Re: 9.3-RELEASE panic: spin lock held too long Message-ID: <20160811081802.GF83214@kib.kiev.ua> References: <57AB349B.2010805@gmail.com> <20160810141948.GP83214@kib.kiev.ua> <57AB462A.2080608@gmail.com> <57AB632D.4000501@gmail.com> <57ABB512.4030503@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <57ABB512.4030503@gmail.com> User-Agent: Mutt/1.6.1 (2016-04-27) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,FREEMAIL_REPLY,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2016 08:18:07 -0000 On Thu, Aug 11, 2016 at 03:43:22AM +0430, Hooman Fazaeli wrote: > On 2016-08-10 22:10, Ryan Stone wrote: > > On Wed, Aug 10, 2016 at 1:23 PM, Hooman Fazaeli > wrote: > > > > No. I have panics involving 'turnstile lock' (see the original post) and 'sched lock 2' too. > > > > > > That doesn't necessarily mean that the root cause isn't due to sched lock 0 being leaked. You'd have to dig into the cores and look at the chain of dependent locks to be sure. Give the patch a > > try; it should panic quite quickly if it's the issue I am thinking of. > > Sure, I will. > BTW, what do you exactly mean by lock leaking? > > Is there a list for the possible causes of 'spin lock held too long' panics? > I mean, what sorts of coding bugs may cause a thread to hold a spin lock for > a long time? Such a list would provide me an starting point for diagnostics. It is impossible to provide the complete list. Possible causes are: - already mentioned lock leak; - lock recursion (sometimes); - something which delays execution of the protected region, which takes the spinlock for otherwise legitimate reasons and period, eg. infinite or too aggressive looping, e.g. due to a deadlock with spinlocks; NMI with run-away handler; failed and stopped executing core; SMI or hypervisor taking control off the OS on the given CPU, while allowing other thread on other CPU to run and notice that. and so on. > > And, How much long is 'too long'? What is the justification behind > the few million for() loop iterations that _mtx_lock_spin waits > to grab a spin lock? This is purely based on real-life experience on the hardware. If faster CPUs with slower inter-core communication facilities ever appear, the constant might need an adjustment. It is fine for currently fastest hardware, and by design is ok for anything slower.