Date: Mon, 10 Aug 2015 21:33:11 -0700 From: Tim Kientzle <tim@kientzle.com> To: Konstantin Belousov <kostikbel@gmail.com> Cc: FreeBSD Current <freebsd-current@freebsd.org> Subject: Re: Consistent crash of BeagleBone kernel Message-ID: <3CCB9E6A-EC5B-4A25-92B9-0850403BC657@kientzle.com> In-Reply-To: <20150810073947.GN2072@kib.kiev.ua> References: <CC793A5E-83D2-4169-AECB-9690572A00D0@kientzle.com> <20150809064704.GH2072@kib.kiev.ua> <350C3412-910F-4B08-A903-A51D18474DAF@kientzle.com> <20150809181004.GK2072@kib.kiev.ua> <F179DE32-D95B-4817-BBFB-97F2E79D4843@kientzle.com> <20150810073947.GN2072@kib.kiev.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
> On Aug 10, 2015, at 12:39 AM, Konstantin Belousov =
<kostikbel@gmail.com> wrote:
>=20
> On Sun, Aug 09, 2015 at 05:24:13PM -0700, Tim Kientzle wrote:
>>=20
>>> On Aug 9, 2015, at 11:10 AM, Konstantin Belousov =
<kostikbel@gmail.com> wrote:
>>>=20
>>> On Sun, Aug 09, 2015 at 10:53:20AM -0700, Tim Kientzle wrote:
>>>>=20
>>>> I suspect the LOR is new.
>>>>=20
>>>> It looks like the panic is occurring when WITNESS tries to print =
the backtrace for the LOR. I???m not familiar with that code; does it =
use the kernel linker?
>>>>=20
>>>=20
>>> It indeed locks the linker lock to resolve symbols. So it seems to =
be
>>> even more useful to make the linker lock recursive locally, then you
>>> should be able to see the backtrace for LOR.
>>=20
>> Changing the kld_sx lock to recursive, I now see a backtrace for the =
ufs/kernel linker LOR. Full trace pasted below.
>>=20
>> This is displayed just before the network interfaces; I suspect =
it???s being triggered when my startup initializes the urtwn wireless =
adapter (which does indeed load a number of kernel modules).
>>=20
>> lock order reversal:
>> 1st 0xc083ef40 kernel linker (kernel linker) @ =
/Users/tim/projects/crochet/src-head/sys/kern/kern_linker.c:1030
>> 2nd 0xc2d63c94 ufs (ufs) @ =
/Users/tim/projects/crochet/src-head/sys/kern/vfs_lookup.c:529
> The order, for which witness complained, is in fact the right order.
> The linker_load_module() function calls LINKER_LOAD_FILE() with the
> kld_sx locked, and linker itself locks module vnode.
>=20
> So there was something in your system which exposed the reversed order
> vnode->kld_sx before the action. To catch it, keep the modification =
to
> mark kld_sx as recursive, but also add an item to the order_lists in
> the sys/kern/subr_witness.c like this:
> {"kernel linker", &lock_class_sx},
> {"ufs", &lock_class_lockmgr},
> {NULL, NULL}
> and watch were would it fire.
I=E2=80=99ll do this when I get back to that system next week.
I have a guess, though: I noticed that the old bufwait/dirhash LOR is =
being triggered before this. Could the backtrace from that be teaching =
Witness a bogus ufs -> kld ordering?
Tim
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3CCB9E6A-EC5B-4A25-92B9-0850403BC657>