Date: Fri, 21 Mar 2014 16:07:50 +0600 From: "Eugene M. Zheganin" <emz@norma.perm.ru> To: freebsd-stable <freebsd-stable@freebsd.org> Subject: Re: crash on zpool import - help get data back Message-ID: <532C0F76.8010605@norma.perm.ru> In-Reply-To: <532BF531.1050400@norma.perm.ru> References: <532BEABC.5050808@norma.perm.ru> <532BF531.1050400@norma.perm.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi.
On 21.03.2014 14:15, Eugene M. Zheganin wrote:
>
> After some thinking (speeded up with the superiors running in circles) I
> realized that the root cause is the same and I can apply the tricks
> mentioned above.
I did them, I'm able to import the pool and read some data, but when
trying to read all of it I get panic on 43th gigabyte out of 1200
(attached below). Is there some way to get this hack-patched too ? Just
to save data.
===Cut===
Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 03
fault virtual address = 0x88
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff818a0454
stack pointer = 0x28:0xfffffe01214086e0
frame pointer = 0x28:0xfffffe0121408740
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 1282 (zfs)
trap number = 12
panic: page fault
cpuid = 3
KDB: stack backtrace:
#0 0xffffffff808f01d0 at kdb_backtrace+0x60
#1 0xffffffff808b7ba5 at panic+0x155
#2 0xffffffff80c98f32 at trap_fatal+0x3a2
#3 0xffffffff80c99209 at trap_pfault+0x2c9
#4 0xffffffff80c9899b at trap+0x5bb
#5 0xffffffff80c7fc52 at calltrap+0x8
#6 0xffffffff818aeed5 at zio_checksum_verify+0x65
#7 0xffffffff818abada at zio_execute+0x15a
#8 0xffffffff818ab0b3 at zio_wait+0x23
#9 0xffffffff81839f83 at arc_read+0x783
#10 0xffffffff8184e04f at backup_cb+0x35f
#11 0xffffffff8185243d at traverse_visitbp+0x20d
#12 0xffffffff81852e6f at traverse_dnode+0xef
#13 0xffffffff81852bb7 at traverse_visitbp+0x987
#14 0xffffffff81852613 at traverse_visitbp+0x3e3
#15 0xffffffff81852613 at traverse_visitbp+0x3e3
#16 0xffffffff81852613 at traverse_visitbp+0x3e3
#17 0xffffffff81852613 at traverse_visitbp+0x3e3
Uptime: 10m14s
Dumping 426 out of 4043 MB:..4%..12%..23%..34%..42%..53%..64%..72%..83%..94%
Reading symbols from /boot/kernel/zfs.ko.symbols...done.
Loaded symbols for /boot/kernel/zfs.ko.symbols
Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
Loaded symbols for /boot/kernel/opensolaris.ko.symbols
Reading symbols from /boot/kernel/geom_mirror.ko.symbols...done.
Loaded symbols for /boot/kernel/geom_mirror.ko.symbols
#0 doadump (textdump=<value optimized out>) at pcpu.h:219
219 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) #0 doadump (textdump=<value optimized out>) at pcpu.h:219
#1 0xffffffff808b7820 in kern_reboot (howto=260)
at /usr/src/sys/kern/kern_shutdown.c:452
#2 0xffffffff808b7be4 in panic (fmt=<value optimized out>)
at /usr/src/sys/kern/kern_shutdown.c:759
#3 0xffffffff80c98f32 in trap_fatal (frame=<value optimized out>,
eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:875
#4 0xffffffff80c99209 in trap_pfault (frame=0xfffffe0121408630, usermode=0)
at /usr/src/sys/amd64/amd64/trap.c:692
#5 0xffffffff80c9899b in trap (frame=0xfffffe0121408630)
at /usr/src/sys/amd64/amd64/trap.c:456
#6 0xffffffff80c7fc52 in calltrap ()
at /usr/src/sys/amd64/amd64/exception.S:232
#7 0xffffffff818a0454 in zfs_ereport_start_checksum
(spa=0xfffff80099c0c000,
vd=0x0, zio=0xfffff80117eed000, offset=0, length=512,
arg=<value optimized out>, info=<value optimized out>)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_fm.c:704
#8 0xffffffff818aeed5 in zio_checksum_verify (zio=<value optimized out>)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2858
#9 0xffffffff818abada in zio_execute (zio=0xfffff80117eed000)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1346
#10 0xffffffff818ab0b3 in zio_wait (zio=0xfffff80117eed000)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1370
#11 0xffffffff81839f83 in arc_read (pio=0x0, spa=0xfffff80099c0c000,
bp=<value optimized out>, done=<value optimized out>, private=0x0,
priority=ZIO_PRIORITY_ASYNC_READ, zio_flags=512,
arc_flags=0xffffffff81838f02, zb=0x0)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:3414
#12 0xffffffff8184e04f in backup_cb (spa=0xfffff80099c0c000,
zilog=<value optimized out>, bp=0xfffffe005c9de780,
zb=0xfffffe0121408b08, dnp=<value optimized out>,
arg=0xfffff8006e402900)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c:422
#13 0xffffffff8185243d in traverse_visitbp (td=0xfffffe01214095f0,
dnp=0xfffffe005c9de600, bp=0xfffffe005c9de780, zb=0xfffffe0121408b08)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:250
#14 0xffffffff81852e6f in traverse_dnode (td=0xfffffe01214095f0,
dnp=0xfffffe005c9de600, objset=236, object=104707)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:417
#15 0xffffffff81852bb7 in traverse_visitbp (td=0xfffffe01214095f0,
dnp=0xfffffe005c9de000, bp=0xfffffe0056cf9400, zb=0xfffffe0121408d28)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:309
#16 0xffffffff81852613 in traverse_visitbp (td=0xfffffe01214095f0,
dnp=0xfffff800a68a5000, bp=0xfffffe002c653c80, zb=0xfffffe0121408e58)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284
#17 0xffffffff81852613 in traverse_visitbp (td=0xfffffe01214095f0,
dnp=0xfffff800a68a5000, bp=0xfffffe002c74a000, zb=0xfffffe0121408f88)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284
#18 0xffffffff81852613 in traverse_visitbp (td=0xfffffe01214095f0,
dnp=0xfffff800a68a5000, bp=0xfffffe002c71a000, zb=0xfffffe01214090b8)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284
#19 0xffffffff81852613 in traverse_visitbp (td=0xfffffe01214095f0,
dnp=0xfffff800a68a5000, bp=0xfffffe002c37d000, zb=0xfffffe01214091e8)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284
#20 0xffffffff81852613 in traverse_visitbp (td=0xfffffe01214095f0,
dnp=0xfffff800a68a5000, bp=0xfffffe002c726000, zb=0xfffffe0121409318)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284
#21 0xffffffff81852613 in traverse_visitbp (td=0xfffffe01214095f0,
dnp=0xfffff800a68a5000, bp=0xfffff800a68a5040, zb=0xfffffe01214093d8)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284
#22 0xffffffff81852e04 in traverse_dnode (td=0xfffffe01214095f0,
dnp=0xfffff800a68a5000, objset=236, object=0)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:407
#23 0xffffffff818528c0 in traverse_visitbp (td=0xfffffe01214095f0, dnp=0x0,
bp=0xfffff800a6b4ce80, zb=0xfffffe0121409588)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:338
#24 0xffffffff81851e5c in traverse_impl (spa=<value optimized out>,
ds=<value optimized out>, objset=<value optimized out>,
rootbp=0xfffff800a6b4ce80, txg_start=<value optimized out>,
resume=<value optimized out>, flags=<value optimized out>,
func=0xffffffff8184dcf0 <backup_cb>, arg=0xfffff8006e402900)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:540
#25 0xffffffff81851bf3 in traverse_dataset (ds=0xfffffe01214087b0,
txg_start=0, flags=<value optimized out>, func=0x50,
arg=<value optimized out>)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:563
#26 0xffffffff8184b701 in dmu_send_impl (tag=0xffffffff8192d305,
dp=0xfffff80016151400, ds=0xfffff800a64d6c00,
fromds=<value optimized out>, outfd=-1504875032, fp=0xfffff800a64d6e08)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c:551
#27 0xffffffff8184b3a7 in dmu_send_obj (pool=<value optimized out>,
tosnap=<value optimized out>, fromsnap=<value optimized out>, outfd=1,
fp=0xfffff80086e6e2d0, off=0xfffffe01214097c0)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c:620
#28 0xffffffff818c2efa in zfs_ioc_send (zc=0xfffffe002c7a6000)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:4273
#29 0xffffffff818bf008 in zfsdev_ioctl (dev=<value optimized out>,
zcmd=<value optimized out>, arg=<value optimized out>,
flag=<value optimized out>, td=<value optimized out>)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:5960
#30 0xffffffff807b2f4f in devfs_ioctl_f (fp=0xfffff80086e6e370,
com=3222821404, data=0xfffff80086cd19e0, cred=<value optimized out>,
td=0xfffff80026e1f000) at /usr/src/sys/fs/devfs/devfs_vnops.c:757
#31 0xffffffff8090680e in kern_ioctl (td=0xfffff80026e1f000,
fd=<value optimized out>, com=18446741879539140528) at file.h:319
#32 0xffffffff8090658f in sys_ioctl (td=0xfffff80026e1f000,
uap=0xfffffe0121409a40) at /usr/src/sys/kern/sys_generic.c:702
#33 0xffffffff80c99827 in amd64_syscall (td=0xfffff80026e1f000, traced=0)
at subr_syscall.c:134
#34 0xffffffff80c7ff3b in Xfast_syscall ()
at /usr/src/sys/amd64/amd64/exception.S:391
#35 0x00000008019e413a in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language: auto; currently minimal
(kgdb)
===Cut===
Thanks.
Eugene.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?532C0F76.8010605>