From owner-freebsd-current@FreeBSD.ORG Mon Nov 24 19:20:50 2014 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6152C597 for ; Mon, 24 Nov 2014 19:20:50 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1on0054.outbound.protection.outlook.com [157.56.110.54]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EA3A530C for ; Mon, 24 Nov 2014 19:20:49 +0000 (UTC) Received: from CO2PR0801MB663.namprd08.prod.outlook.com (10.141.247.26) by CO2PR0801MB0807.namprd08.prod.outlook.com (25.160.7.16) with Microsoft SMTP Server (TLS) id 15.1.26.15; Mon, 24 Nov 2014 17:44:39 +0000 Received: from [172.17.3.251] (63.252.212.99) by CO2PR0801MB663.namprd08.prod.outlook.com (10.141.247.26) with Microsoft SMTP Server (TLS) id 15.1.26.15; Mon, 24 Nov 2014 17:44:36 +0000 Message-ID: <54736E7C.80105@panasas.com> Date: Mon, 24 Nov 2014 12:44:28 -0500 From: "Ellis H. Wilson III" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.8.0 MIME-Version: 1.0 To: Benjamin Kaduk Subject: Re: WITNESS observes 2 LORs on Boot of Release 10.1 References: <546BA9D3.6070007@panasas.com> <546BF3F5.8030109@panasas.com> <546FA1DD.2070109@panasas.com> In-Reply-To: Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [63.252.212.99] X-ClientProxiedBy: BLUPR02CA052.namprd02.prod.outlook.com (25.160.23.170) To CO2PR0801MB663.namprd08.prod.outlook.com (10.141.247.26) X-Microsoft-Antispam: UriScan:;UriScan:; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:;SRVR:CO2PR0801MB663; X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:;SRVR:CO2PR0801MB663; X-Forefront-PRVS: 040513D301 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6009001)(6049001)(189002)(479174003)(24454002)(41574002)(199003)(51704005)(377454003)(107046002)(102836001)(62966003)(101416001)(40100003)(99396003)(122386002)(86362001)(95666004)(77156002)(47776003)(21056001)(77096003)(66066001)(46102003)(20776003)(36756003)(105586002)(33656002)(64706001)(106356001)(23756003)(42186005)(31966008)(50986999)(15975445006)(97736003)(54356999)(92566001)(65956001)(92726001)(2171001)(64126003)(87976001)(50466002)(19580395003)(4396001)(83506001)(120916001)(76176999); DIR:OUT; SFP:1101; SCL:1; SRVR:CO2PR0801MB663; H:[172.17.3.251]; FPR:; SPF:None; MLV:sfv; PTR:InfoNoRecords; MX:1; A:1; LANG:en; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:;SRVR:CO2PR0801MB663; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:;SRVR:CO2PR0801MB0807; X-OriginatorOrg: panasas.com Cc: freebsd-current@freebsd.org X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Nov 2014 19:20:50 -0000 On 11/22/2014 03:51 PM, Benjamin Kaduk wrote: > On Fri, 21 Nov 2014, Ellis H. Wilson III wrote: > >> Before I start, and this is mainly geared to my responder Benjamin Kaduk, >> based on your response, are you suggesting that the cnputc WITNESS panic you >> expected to happen is now completely unavoidable in FreeBSD 10? I.E., is this >> a spinlock that WITNESS falls over each time but that is provably deadlock >> free that the developers have decided cannot be BLESSED for some reason? > > https://lists.freebsd.org/pipermail/freebsd-current/2012-January/031316.html > looks to be a better explanation than the previous link I sent ... in > short, console output is hard. > >> I guess I just can't wrap my head around why we would ever move to a regime >> where SKIPSPIN is the default for testing... That just seems like an open >> invitation for introducing spinlock regressions. > > I don't think anyone made the conscious decision to do that, it just > happened by default as no one spent the time to fix the aforementioned > issue. > >> Moving onto the LORs I'm seeing, a question I have as a newbie to WITNESS >> debugging is how exactly to interpret the output if I see a stacktrace and >> then a LOR output like the following: >> >> lock order reversal: >> 1st 0xffffffff81633d88 entropy harvest mutex (entropy harvest mutex) @ >> /usr/src/sys/dev/random/random_harvestq.c:198 >> 2nd 0xffffffff813b6208 scrlock (scrlock) @ >> /usr/src/sys/dev/syscons/syscons.c:2682 >> >> Does this mean WITNESS has already stored an ordering of #1 harvest_mtx then >> #2 scp->scr_lock, and somewhere somebody tried to lock scp->scr_lock without >> first getting harvest_mtx? Or the reverse (WITNESS previously recorded >> scrlock and then harvest and the lines it spit out were the offenders?) > > I believe it is the latter (the ordering being printed is the bad one > which caused WITNESS to complain). Thanks so much for the additional info Ben. This fleshes out the history of this issue for me significantly. I have filed a bug on this at: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195262 Xin Li was able to identify the ordering that caused the problem and proposed a possible patch to fix it. I can confirm that now I'm booting with solely WITNESS (i.e., not WITNESS_SKIPSPIN) without panic. Thanks! ellis