From owner-freebsd-current@freebsd.org Tue Aug 11 04:33:21 2015 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DE02799DFB4 for ; Tue, 11 Aug 2015 04:33:20 +0000 (UTC) (envelope-from tim@kientzle.com) Received: from mail-pd0-f171.google.com (mail-pd0-f171.google.com [209.85.192.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B2291FB2 for ; Tue, 11 Aug 2015 04:33:20 +0000 (UTC) (envelope-from tim@kientzle.com) Received: by pdco4 with SMTP id o4so79060308pdc.3 for ; Mon, 10 Aug 2015 21:33:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=V1JF5jJHbI2TEMAuYaEoTNQhDbmZ5SuAJocdiIRsK5g=; b=KtIWhGcSUe0XPiNs7h2OLu93wdWmA/lZc6EVCqzgszJNeZ7kX1iNFiu1IfSaEW7Fn9 22XhIR9jhX/CX7/6OGMiHCntEPZS+Gnr/xFqnotRBIYUs3x1vWKgvwyO8uNJV0Q4Xzmi yWLY27VLJjgJeeZUGenLoLKl4mxHvH5FWqykWy9+nD9ABYltzedyLU1WgLzZT5n7LaJV lRCiN3/r+0wFJXfA57LF7+gk3ElkzHemBd+6Agxke3QkrZ7Cd4M/Q8x3eWSJHD1Bndqs XgKoNL1HUOsCqpyRWolw81UO6G7PVR9KYzE9817y2xJAbQmTNXbwK/fxGuW0mpRKMth/ MpUQ== X-Gm-Message-State: ALoCoQktMP4vzH2KWLARLSVOZntBpbWoOGr73XEHiLW8rT/06jGpbb/qvxzD1LNjbviAal/oOQCB X-Received: by 10.70.6.34 with SMTP id x2mr51318431pdx.113.1439267594881; Mon, 10 Aug 2015 21:33:14 -0700 (PDT) Received: from [192.168.1.102] (c-24-6-220-224.hsd1.ca.comcast.net. [24.6.220.224]) by smtp.gmail.com with ESMTPSA id b4sm704338pdn.42.2015.08.10.21.33.13 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 10 Aug 2015 21:33:13 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) Subject: Re: Consistent crash of BeagleBone kernel From: Tim Kientzle In-Reply-To: <20150810073947.GN2072@kib.kiev.ua> Date: Mon, 10 Aug 2015 21:33:11 -0700 Cc: FreeBSD Current Content-Transfer-Encoding: quoted-printable Message-Id: <3CCB9E6A-EC5B-4A25-92B9-0850403BC657@kientzle.com> References: <20150809064704.GH2072@kib.kiev.ua> <350C3412-910F-4B08-A903-A51D18474DAF@kientzle.com> <20150809181004.GK2072@kib.kiev.ua> <20150810073947.GN2072@kib.kiev.ua> To: Konstantin Belousov X-Mailer: Apple Mail (2.2102) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Aug 2015 04:33:21 -0000 > On Aug 10, 2015, at 12:39 AM, Konstantin Belousov = wrote: >=20 > On Sun, Aug 09, 2015 at 05:24:13PM -0700, Tim Kientzle wrote: >>=20 >>> On Aug 9, 2015, at 11:10 AM, Konstantin Belousov = wrote: >>>=20 >>> On Sun, Aug 09, 2015 at 10:53:20AM -0700, Tim Kientzle wrote: >>>>=20 >>>> I suspect the LOR is new. >>>>=20 >>>> It looks like the panic is occurring when WITNESS tries to print = the backtrace for the LOR. I???m not familiar with that code; does it = use the kernel linker? >>>>=20 >>>=20 >>> It indeed locks the linker lock to resolve symbols. So it seems to = be >>> even more useful to make the linker lock recursive locally, then you >>> should be able to see the backtrace for LOR. >>=20 >> Changing the kld_sx lock to recursive, I now see a backtrace for the = ufs/kernel linker LOR. Full trace pasted below. >>=20 >> This is displayed just before the network interfaces; I suspect = it???s being triggered when my startup initializes the urtwn wireless = adapter (which does indeed load a number of kernel modules). >>=20 >> lock order reversal: >> 1st 0xc083ef40 kernel linker (kernel linker) @ = /Users/tim/projects/crochet/src-head/sys/kern/kern_linker.c:1030 >> 2nd 0xc2d63c94 ufs (ufs) @ = /Users/tim/projects/crochet/src-head/sys/kern/vfs_lookup.c:529 > The order, for which witness complained, is in fact the right order. > The linker_load_module() function calls LINKER_LOAD_FILE() with the > kld_sx locked, and linker itself locks module vnode. >=20 > So there was something in your system which exposed the reversed order > vnode->kld_sx before the action. To catch it, keep the modification = to > mark kld_sx as recursive, but also add an item to the order_lists in > the sys/kern/subr_witness.c like this: > {"kernel linker", &lock_class_sx}, > {"ufs", &lock_class_lockmgr}, > {NULL, NULL} > and watch were would it fire. I=E2=80=99ll do this when I get back to that system next week. I have a guess, though: I noticed that the old bufwait/dirhash LOR is = being triggered before this. Could the backtrace from that be teaching = Witness a bogus ufs -> kld ordering? Tim