From owner-freebsd-arm@FreeBSD.ORG Tue Sep 30 14:19:58 2014 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 09EBC6D0 for ; Tue, 30 Sep 2014 14:19:58 +0000 (UTC) Received: from mho-02-ewr.mailhop.org (mho-02-ewr.mailhop.org [204.13.248.72]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BD87B176 for ; Tue, 30 Sep 2014 14:19:57 +0000 (UTC) Received: from [73.34.117.227] (helo=ilsoft.org) by mho-02-ewr.mailhop.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1XYyHU-000P4h-Kp; Tue, 30 Sep 2014 14:19:56 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by ilsoft.org (8.14.9/8.14.9) with ESMTP id s8UEJtZE014511; Tue, 30 Sep 2014 08:19:55 -0600 (MDT) (envelope-from ian@FreeBSD.org) X-Mail-Handler: Dyn Standard SMTP by Dyn X-Originating-IP: 73.34.117.227 X-Report-Abuse-To: abuse@dyndns.com (see http://www.dyndns.com/services/sendlabs/outbound_abuse.html for abuse reporting information) X-MHO-User: U2FsdGVkX18ZvQPcsdDII3Em8eB9or9i X-Authentication-Warning: paranoia.hippie.lan: Host revolution.hippie.lan [172.22.42.240] claimed to be [172.22.42.240] Subject: Re: Random Kernel Panic on Dreamplug (FS related) From: Ian Lepore To: Mattia Rossi In-Reply-To: <542AB897.3020309@gmail.com> References: <542559BC.7090100@gmail.com> <20140929040126.GG43300@funkthat.com> <54291B74.5010307@gmail.com> <20140930112937.GU43300@funkthat.com> <542A9EA4.70109@gmail.com> <20140930123010.GZ43300@funkthat.com> <542AB897.3020309@gmail.com> Content-Type: text/plain; charset="us-ascii" Date: Tue, 30 Sep 2014 08:19:55 -0600 Message-ID: <1412086795.66615.363.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-arm X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Sep 2014 14:19:58 -0000 On Tue, 2014-09-30 at 16:05 +0200, Mattia Rossi wrote: > Am 30.09.2014 14:30, schrieb John-Mark Gurney: > > Mattia Rossi wrote this message on Tue, Sep 30, 2014 at 14:14 +0200: > >> Am 30.09.2014 13:29, schrieb John-Mark Gurney: > >>> Mattia Rossi wrote this message on Mon, Sep 29, 2014 at 10:42 +0200: > >>>> Am 29.09.2014 06:01, schrieb John-Mark Gurney: > >>>>> Mattia Rossi wrote this message on Fri, Sep 26, 2014 at 14:19 +0200: > >>>>>> This might be part of the weird FFS issues the Dreamplug has and no-one > >>>>>> knows why they're happening. > >>>>> Are you running w/ FFS journaling? If so, try turning it off, but > >>>>> keeping softupdates on.. > >>>> No journaling, no softupdates. I'll try enabling softupdates next time. > >>>> don't know if it will panic though > >>>>>> data_abort_handler() at data_abort_handler+0x5c0 > >>>>>> pc = 0xc0de7a28 lr = 0xc0dd711c (exception_exit) > >>>>>> sp = 0xde019898 fp = 0xde019a20 > >>>>>> r4 = 0xffffffff r5 = 0xffff1004 > >>>>>> r6 = 0xc3f3f6c0 r7 = 0x00001000 > >>>>>> r8 = 0xc443e880 r9 = 0x00000000 > >>>>>> r10 = 0xc3d69000 > >>>>>> exception_exit() at exception_exit > >>>>>> pc = 0xc0dd711c lr = 0xc0d53828 (ffs_truncate+0xaa8) > >>>>>> sp = 0xde0198e8 fp = 0xde019a20 > >>>>>> r0 = 0xd0238120 r1 = 0x00000e60 > >>>>>> r2 = 0x00000000 r3 = 0x00000000 > >>>>>> r4 = 0x00000120 r5 = 0x00000000 > >>>>>> r6 = 0xc3f3f6c0 r7 = 0x00001000 > >>>>>> r8 = 0xc443e880 r9 = 0x00000000 > >>>>>> r10 = 0xc3d69000 r12 = 0xd0238120 > >>>>>> memset() at memset+0x48 > >>>>>> pc = 0xc0de521c lr = 0xc0d53828 (ffs_truncate+0xaa8) > >>>>>> sp = 0xde0198e8 fp = 0xde019a20 > >>>>>> Unwind failure (no registers changed) > >>>>> No more beyond this? If you could run addr2line on 0xc0d53828 so > >>>>> that we know where in ffs_truncate it's failing, that'd be very > >>>>> nice... > >>>> So I was trying to save the coredump in order to reboot and run > >>>> addr2line, but that failed: > >>>> > >>>> Physical memory: 504 MB > >>>> Dumping 67 MB:(da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 01 d5 1f20 > >>>> 00 00 01 00 > >>>> (da0:umass-sim0:0:0:0): CAM status: Resource Unavailable > >>>> (da0:umass-sim0:0:0:0): Error 5, Retries exhausted > >>>> Aborting dump due to I/O error. > >>>> > >>>> ** DUMP FAILED (ERROR 5) ** > >>>> > >>>> So I guess this error is related to the CAM errors I'm getting from time > >>>> to time. I was hoping that those errors were related to the INVARIANTS > >>>> option that slowed down the system and thus might have triggered CAM > >>>> errors, but obviously the SD Card seems to be the real issue here. > >>>> So no crashdump for further analysis. > >>> That's fine.. w/ the addr2line we have some lines to explore... > >>> > >>>> Interestingly the CAM errors didn't show up on the terminal as other > >>>> times, the kernel just panicked straight away. > >>> Hmm.. that is odd.. someone who knows the SD card layer should look > >>> at this part... It could be that the SD card driver doesn't handle > >>> dumping (there is this global flag that gets set) properly and the driver > >>> needs to behave differently when it's set... > >> I also need to grab a new SD card, just to make sure it's really not the > >> card. > >> > >>>> But I've got the addr2line output, even though I'm not sure it makes any > >>>> difference: > >>>> > >>>> addr2line -f -e /mnt/kernel.debug 0xc0d53828 > >>>> > >>>> ffs_truncate > >>>> /usr/devel/dreamplug/sys/ufs/ffs/ffs_inode.c:321 > >>> can you give me the contents of the line? and a few lines of context > >>> around it? In HEAD's source, this is DOINGASYNC, and there is no call > >>> to memset, nor a variable assignment that would result in memset being > >>> called... > >> Same here.. The file hasn't been changed in a while (Fri, 31 May 2013): > >> > >> ip->i_size = length; > >> DIP_SET(ip, i_size, length); > >> if (bp->b_bufsize == fs->fs_bsize) > >> bp->b_flags |= B_CLUSTEROK; > >> if (flags & IO_SYNC) > >> bwrite(bp); > >> 321: else if (DOINGASYNC(vp)) > >> bdwrite(bp); > >> else > >> bawrite(bp); > >> ip->i_flag |= IN_CHANGE | IN_UPDATE; > >> return (ffs_update(vp, !DOINGASYNC(vp))); > >> > >> No idea what's going on. > > ok, could you send me the output of objdump -dSl, but you only need > > to include the part from XXXXX : to the next XXX: > > line... probably off list as it'll be quite long... > I'm sorry, but given that I just broke all my working worlds using fsck, > I'm not going to be able to do that until I'm back from holidays.... > currently working on the stuff remotely and after today's work day, I'm > not going to be able to get my hands on the dreamplug. > > BTW, for anyone playing with this problem, step one is to edit your /etc/fstab and set the fsck pass number to 0 for all filesystems. There's a risk of filesystem corruption after a crash, but it's smaller than the 100% corruption rate of letting fsck run. :) -- Ian