Date: Mon, 26 Nov 2012 14:21:51 -0800 From: Garrett Cooper <yanegomi@gmail.com> To: Raymond Jimenez <raymondj@caltech.edu> Cc: freebsd-fs@freebsd.org Subject: Re: ZFS kernel panics due to corrupt DVAs (despite RAIDZ) Message-ID: <CAGH67wRYDg5gagx_Wx0Jji8wwYGgvzuui-yiAe4v8sup_bHzxw@mail.gmail.com> In-Reply-To: <50B3E680.8060606@caltech.edu> References: <50B3E680.8060606@caltech.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Nov 26, 2012 at 2:00 PM, Raymond Jimenez <raymondj@caltech.edu> wrote: > Hello, > > We recently sent our drives out for data recovery (blown drive > electronics), and when we got the new drives/data back, ZFS > started to kernel panic whenever listing certain items in a > directory, or whenever a scrub is close to finishing (~99.97%) > > The zpool worked fine before data recovery, and most of the > files are accessible (only a couple hundred unavailable out of > several million). > > Here's the kernel panic output if I scrub the disk: > > Fatal trap 12: page fault while in kernel mode > cpuid = 0; apic id = 00 > fault virtual address = 0x38 > fault code = supervisor read data, page not present > instruction pointer = 0x20:0xffffffff810792d1 > stack pointer = 0x28:0xffffff8235122720 > frame pointer = 0x28:0xffffff8235122750 > code segment = base 0x0, limit 0xffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 52 (txg_thread_enter) > [thread pid 52 tid 101230 ] > Stopped at vdev_is_dead+0x1: cmpq $0x5, 0x38(%rdi) > > $rdi is zero, so this seems to be just a null pointer exception. > > The vdev setup looks like: > > pool: mfs-zpool004 > state: ONLINE > scan: scrub canceled on Mon Nov 26 05:40:49 2012 > config: > > NAME STATE READ WRITE CKSUM > mfs-zpool004 ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > gpt/lenin3-drive8 ONLINE 0 0 0 > gpt/lenin3-drive9.eli ONLINE 0 0 0 > gpt/lenin3-drive10 ONLINE 0 0 0 > gpt/lenin3-drive11.eli ONLINE 0 0 0 > raidz1-1 ONLINE 0 0 0 > gpt/lenin3-drive12 ONLINE 0 0 0 > gpt/lenin3-drive13.eli ONLINE 0 0 0 > gpt/lenin3-drive14 ONLINE 0 0 0 > gpt/lenin3-drive15.eli ONLINE 0 0 0 > > errors: No known data errors > > The initial scrub fixed some data (~24k) in the early stages, but > also crashed at 99.97%. > > Right now, I'm using an interim work-around patch[1] so that our > users can get files without worrying about crashing the server. > It's a small check in dbuf_findbp() that checks if the DVA that will > be returned has a small (=<16) vdev number, and if not, returns EIO. > This just results in ZFS returning I/O errors for any of the corrupt > files I try to access, which at least lets us get at our data for now. > > My suspicion is that somehow, bad data is getting interpreted as > a block pointer/shift constant, and this sends ZFS into the woods. > I haven't been able to track down how this data could get past > checksum verification, especially with RAIDZ. > > Backtraces: > > (both crashes due to vdev_is_dead() dereferencing a null pointer) > > Scrub crash: > http://wsyntax.com/~raymond/zfs/zfs-scrub-bt.txt > > Prefetch off, ls -al of "/06/chunk_0000000001417E06_00000001.mfs": > http://wsyntax.com/~raymond/zfs/zfs-ls-bt.txt This is missing key details like uname, zpool version, etc. Thanks, -Garrett
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAGH67wRYDg5gagx_Wx0Jji8wwYGgvzuui-yiAe4v8sup_bHzxw>