Date: Tue, 30 Sep 2014 07:46:26 -0600 From: Ian Lepore <ian@FreeBSD.org> To: John-Mark Gurney <jmg@funkthat.com> Cc: freebsd-arm <freebsd-arm@freebsd.org> Subject: Re: Random Kernel Panic on Dreamplug (FS related) Message-ID: <1412084786.66615.356.camel@revolution.hippie.lan> In-Reply-To: <20140930112937.GU43300@funkthat.com> References: <542559BC.7090100@gmail.com> <20140929040126.GG43300@funkthat.com> <54291B74.5010307@gmail.com> <20140930112937.GU43300@funkthat.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 2014-09-30 at 04:29 -0700, John-Mark Gurney wrote: > Mattia Rossi wrote this message on Mon, Sep 29, 2014 at 10:42 +0200: > > > > Am 29.09.2014 06:01, schrieb John-Mark Gurney: > > >Mattia Rossi wrote this message on Fri, Sep 26, 2014 at 14:19 +0200: > > >>This might be part of the weird FFS issues the Dreamplug has and no-one > > >>knows why they're happening. > > >Are you running w/ FFS journaling? If so, try turning it off, but > > >keeping softupdates on.. > > No journaling, no softupdates. I'll try enabling softupdates next time. > > don't know if it will panic though > > > > > >>data_abort_handler() at data_abort_handler+0x5c0 > > >> pc = 0xc0de7a28 lr = 0xc0dd711c (exception_exit) > > >> sp = 0xde019898 fp = 0xde019a20 > > >> r4 = 0xffffffff r5 = 0xffff1004 > > >> r6 = 0xc3f3f6c0 r7 = 0x00001000 > > >> r8 = 0xc443e880 r9 = 0x00000000 > > >> r10 = 0xc3d69000 > > >>exception_exit() at exception_exit > > >> pc = 0xc0dd711c lr = 0xc0d53828 (ffs_truncate+0xaa8) > > >> sp = 0xde0198e8 fp = 0xde019a20 > > >> r0 = 0xd0238120 r1 = 0x00000e60 > > >> r2 = 0x00000000 r3 = 0x00000000 > > >> r4 = 0x00000120 r5 = 0x00000000 > > >> r6 = 0xc3f3f6c0 r7 = 0x00001000 > > >> r8 = 0xc443e880 r9 = 0x00000000 > > >> r10 = 0xc3d69000 r12 = 0xd0238120 > > >>memset() at memset+0x48 > > >> pc = 0xc0de521c lr = 0xc0d53828 (ffs_truncate+0xaa8) > > >> sp = 0xde0198e8 fp = 0xde019a20 > > >>Unwind failure (no registers changed) > > >No more beyond this? If you could run addr2line on 0xc0d53828 so > > >that we know where in ffs_truncate it's failing, that'd be very > > >nice... > > So I was trying to save the coredump in order to reboot and run > > addr2line, but that failed: > > > > Physical memory: 504 MB > > Dumping 67 MB:(da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 01 d5 1f 20 > > 00 00 01 00 > > (da0:umass-sim0:0:0:0): CAM status: Resource Unavailable > > (da0:umass-sim0:0:0:0): Error 5, Retries exhausted > > Aborting dump due to I/O error. > > > > ** DUMP FAILED (ERROR 5) ** > > > > So I guess this error is related to the CAM errors I'm getting from time > > to time. I was hoping that those errors were related to the INVARIANTS > > option that slowed down the system and thus might have triggered CAM > > errors, but obviously the SD Card seems to be the real issue here. > > So no crashdump for further analysis. > > That's fine.. w/ the addr2line we have some lines to explore... > > > Interestingly the CAM errors didn't show up on the terminal as other > > times, the kernel just panicked straight away. > > Hmm.. that is odd.. someone who knows the SD card layer should look > at this part... It could be that the SD card driver doesn't handle > dumping (there is this global flag that gets set) properly and the driver > needs to behave differently when it's set... > On this model of dreamplug, an sdcard is just a usb drive; internally there is a usb<->sd adapter chip, the same one used in many usb multi-format card reader/writers. There's no part of the sdhci/mmc/mmcsd driver stack involved. > > But I've got the addr2line output, even though I'm not sure it makes any > > difference: > > > > addr2line -f -e /mnt/kernel.debug 0xc0d53828 > > > > ffs_truncate > > /usr/devel/dreamplug/sys/ufs/ffs/ffs_inode.c:321 > > can you give me the contents of the line? and a few lines of context > around it? In HEAD's source, this is DOINGASYNC, and there is no call > to memset, nor a variable assignment that would result in memset being > called... > > > >>The sad thing is, that with fsck broken for the dreamplug, I have to > > >>re-format the disk, reinstall everything and recreate the config files > > >>which I didn't manage to copy to a safe place beforehand :-( > > Didn't get around yet on checking whether the fsck issue persists if > > everything is compiled with gcc. Will take a bit, as I'm going to be on > > holidays for the next one and a half weeks. Fact is though, fsck is > > broken on the Dreamplug (ARMv5TE), at least when run on EVERY device > > that is attached via USB and if compiled with CLANG. > > >> > > >>Before I do that I'll leave the system in debugging mode for a few days, > > >>in case someone can help and needs some more information. > > Currently I've run fsck on all the disks/cards that had a working world > > for the dreamplug on it, so they're all gone. Can't do eny debugging > > atm. I'll let you know hoe the gcc build goes once I'm back from > > holidays though. > > Hmmm. this is also worth investigating as it could be that clang is > producing bad code somewhere... > I tested with a gcc build of -current yesterday and got the same results as with clang -- fsck on a freshly-formatted filesystem says it's corrupted, even when the formatting was done on a non-armv4 system. Also, this is not related to usb or sdcards per se. You can get the same results with a sata drive, or even md(4) (but interestingly, some sizes of md disk have problems and some don't). -- Ian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1412084786.66615.356.camel>