Skip site navigation (1)Skip section navigation (2) 
Date:      Mon, 14 Jul 2014 22:18:18 -0700
From:      Mike Carlson <mike@bayphoto.com>
To:        freebsd-fs@freebsd.org
Subject:   ZFS Panic on 10.0-RELEASE - again
Message-ID:  <53C4B99A.9000508@bayphoto.com>
next in thread | raw e-mail | index | archive | help 

[-- Attachment #1 --]
I posted to the list about a month ago where our ZFS pool panic upon 
mounting

Well, after backing up and restoring from some snapshots, I rebuilt the 
server on July 4th and restored the data.

Today, competely unrelated to zfs, our root volume (UFS2) was running a 
portsnap fetch extract and trigger a separate panic

I assumed the ZFS data was fine, as it was unrelated. That was not the 
case. After re-installing and performing a zpool import, our pool panic 
yet again.

This is a completely different situation, and again, I've lost around 
20TB of data.

Here is the vmcore.0's backtrace:

Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 12
fault virtual address    = 0x50
fault code        = supervisor read data, page not present
instruction pointer    = 0x20:0xffffffff81a85246
stack pointer            = 0x28:0xfffffe104cb5aab0
frame pointer            = 0x28:0xfffffe104cb5aac0
code segment        = base 0x0, limit 0xfffff, type 0x1b
             = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags    = interrupt enabled, resume, IOPL = 0
current process        = 0 (system_taskq_7)
trap number        = 12
panic: page fault
cpuid = 2
KDB: stack backtrace:
#0 0xffffffff808e7dd0 at kdb_backtrace+0x60
#1 0xffffffff808af8b5 at panic+0x155
#2 0xffffffff80c8e692 at trap_fatal+0x3a2
#3 0xffffffff80c8e969 at trap_pfault+0x2c9
#4 0xffffffff80c8e0f6 at trap+0x5e6
#5 0xffffffff80c75392 at calltrap+0x8
#6 0xffffffff81a8b710 at vdev_mirror_child_select+0x70
#7 0xffffffff81a8b254 at vdev_mirror_io_start+0x234
#8 0xffffffff81aa52d4 at zio_vdev_io_start+0x184
#9 0xffffffff81aa26a6 at zio_execute+0x136
#10 0xffffffff81a32dec at arc_read+0x87c
#15 0xffffffff81a4aee3 at traverse_visitbp+0x393
#16 0xffffffff81a4aee3 at traverse_visitbp+0x393
#17 0xffffffff81a4aee3 at traverse_visitbp+0x393
Uptime: 4m27s
Dumping 2286 out of 65496 
MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel/ums.ko.symbols...done.
Loaded symbols for /boot/kernel/ums.ko.symbols
Reading symbols from /boot/kernel/zfs.ko.symbols...done.
Loaded symbols for /boot/kernel/zfs.ko.symbols
Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
Loaded symbols for /boot/kernel/opensolaris.ko.symbols
#0  doadump (textdump=<value optimized out>) at pcpu.h:219
219        __asm("movq %%gs:%1,%0" : "=r" (td)
(kgdb) bt
#0  doadump (textdump=<value optimized out>) at pcpu.h:219
#1  0xffffffff808af530 in kern_reboot (howto=260) at 
/usr/src/sys/kern/kern_shutdown.c:447
#2  0xffffffff808af8f4 in panic (fmt=<value optimized out>) at 
/usr/src/sys/kern/kern_shutdown.c:754
#3  0xffffffff80c8e692 in trap_fatal (frame=<value optimized out>, 
eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:882
#4  0xffffffff80c8e969 in trap_pfault (frame=0xfffffe104cb5aa00, 
usermode=0) at /usr/src/sys/amd64/amd64/trap.c:699
#5  0xffffffff80c8e0f6 in trap (frame=0xfffffe104cb5aa00) at 
/usr/src/sys/amd64/amd64/trap.c:463
#6  0xffffffff80c75392 in calltrap () at 
/usr/src/sys/amd64/amd64/exception.S:232
#7  0xffffffff81a85246 in vdev_validate (vd=0xfffff80116622c10, 
strict=<value optimized out>) at 
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c:1451
#8  0xffffffff81a8b710 in vdev_mirror_io_done (zio=0x20) at 
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:397
#9  0xffffffff81a8b254 in vdev_mirror_io_start (zio=0xfffff80116622c00) 
at 
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:90
#10 0xffffffff81aa52d4 in zio_vdev_io_start (zio=0xfffff80015db43b0) at 
time.h:63
#11 0xffffffff81aa26a6 in zio_execute (zio=0xfffff80015db43b0) at 
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1319
#12 0xffffffff81a32dec in arc_read (pio=0x0, spa=0xfffff80015e0a000, 
bp=<value optimized out>, done=0x2, private=0x0, priority=6, 
zio_flags=0, arc_flags=<value optimized out>,
     zb=0xfffff80116d7b048) at 
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:3371
#13 0xffffffff81a4b8f1 in traverse_prefetcher (spa=0xfffff80015e0a000, 
zilog=0xf01ff, bp=<value optimized out>, zb=<value optimized out>, 
dnp=0xfffff80116622c00, arg=<value optimized out>)
     at 
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:451
#14 0xffffffff81a4ad14 in traverse_visitbp (td=0xfffffe104cb5b900, 
dnp=0xfffffe00112ba800, bp=0xfffffe00112ba980, zb=0xfffffe104cb5ae88)
     at 
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:250
#15 0xffffffff81a4b77f in traverse_dnode (td=0xfffffe104cb5b900, 
dnp=0xfffffe00112ba800, objset=203, object=24767564)
     at 
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:417
#16 0xffffffff81a4b487 in traverse_visitbp (td=0xfffffe104cb5b900, 
dnp=0xfffffe00112b9000, bp=0xfffffe0012145100, zb=0xfffffe104cb5b0a8)
     at 
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:309
#17 0xffffffff81a4aee3 in traverse_visitbp (td=0xfffffe104cb5b900, 
dnp=0xfffff80010f5d800, bp=0xfffffe00121d9f00, zb=0xfffffe104cb5b1d8)
     at 
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284
#18 0xffffffff81a4aee3 in traverse_visitbp (td=0xfffffe104cb5b900, 
dnp=0xfffff80010f5d800, bp=0xfffffe001057b780, zb=0xfffffe104cb5b308)
     at 
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284
#19 0xffffffff81a4aee3 in traverse_visitbp (td=0xfffffe104cb5b900, 
dnp=0xfffff80010f5d800, bp=0xfffffe000eb3c000, zb=0xfffffe104cb5b438)
     at 
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284
#20 0xffffffff81a4aee3 in traverse_visitbp (td=0xfffffe104cb5b900, 
dnp=0xfffff80010f5d800, bp=0xfffffe000eb28000, zb=0xfffffe104cb5b568)
     at 
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284
#21 0xffffffff81a4aee3 in traverse_visitbp (td=0xfffffe104cb5b900, 
dnp=0xfffff80010f5d800, bp=0xfffffe0011694000, zb=0xfffffe104cb5b698)
     at 
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284
#22 0xffffffff81a4aee3 in traverse_visitbp (td=0xfffffe104cb5b900, 
dnp=0xfffff80010f5d800, bp=0xfffff80010f5d840, zb=0xfffffe104cb5b758)
     at 
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284
#23 0xffffffff81a4b714 in traverse_dnode (td=0xfffffe104cb5b900, 
dnp=0xfffff80010f5d800, objset=203, object=0)
     at 
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:407
#24 0xffffffff81a4b190 in traverse_visitbp (td=0xfffffe104cb5b900, 
dnp=0x0, bp=0xfffff80116b3c880, zb=0xfffffe104cb5b8e0)
     at 
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:338
#25 0xffffffff81a4aaf6 in traverse_prefetch_thread 
(arg=0xfffffe104ca9a3a0) at 
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:470
#26 0xffffffff81a21c00 in taskq_run (arg=0xfffff80116cb5780, 
pending=983551) at 
/usr/src/sys/modules/zfs/../../cddl/compat/opensolaris/kern/opensolaris_taskq.c:109
#27 0xffffffff808f5b66 in taskqueue_run_locked 
(queue=0xfffff8001535e800) at /usr/src/sys/kern/subr_taskqueue.c:333
#28 0xffffffff808f63e8 in taskqueue_thread_loop (arg=<value optimized 
out>) at /usr/src/sys/kern/subr_taskqueue.c:535
#29 0xffffffff8088198a in fork_exit (callout=0xffffffff808f6340 
<taskqueue_thread_loop>, arg=0xfffff80015025e80, 
frame=0xfffffe104cb5ba40) at /usr/src/sys/kern/kern_fork.c:995
#30 0xffffffff80c758ce in fork_trampoline () at 
/usr/src/sys/amd64/amd64/exception.S:606
#31 0x0000000000000000 in ?? ()
Current language:  auto; currently minimal


Pool creation info:
zpool create data raidz /dev/da2.nop /dev/da3.nop /dev/da4.nop /dev/da5.nop
zpool add data raidz /dev/da6.nop /dev/da7.nop /dev/da8.nop /dev/da9.nop
zpool add data raidz /dev/da10.nop /dev/da11.nop /dev/da12.nop /dev/da13.nop
zpool add data raidz /dev/da14.nop /dev/da15.nop /dev/da16.nop /dev/da17.nop
zpool add data raidz /dev/da18.nop /dev/da19.nop /dev/da20.nop /dev/da21.nop
zpool add data raidz /dev/da22.nop /dev/da23.nop /dev/da24.nop /dev/da25.nop
zpool add data spare /dev/da26.nop /dev/da27.nop
zpool add data log /dev/gpt/log.nop
zpool add data cache /dev/gpt/cache.nop


The pool has a log and cache, and I suspect the log device is corrupt.

All zfs commands cause a panic (zdb, zfs list, zpool status, etc...)

I'm also wondering if the LSI controller itself is buggy:
mps0: <LSI SAS2004> port 0xc000-0xc0ff mem 
0xfe83c000-0xfe83ffff,0xfe840000-0xfe87ffff irq 28 at device 0.0 on pci1
mps0: Firmware: 15.00.00.00, Driver: 16.00.00.00-fbsd
mps0: IOCCapabilities: 
185c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR>



[-- Attachment #2 --]
0	*H
010	+0	*H
"00e3v=0
	*H
0K10
URootCA10U

Bay Photo Lab10U
California10	UUS0
121023173218Z
271023173218Z0X10UBay Photo People CA10U

Bay Photo Lab10U
California10	UUS0"0
	*H
0
;TąuyK~Zz2M'4
EiTj)yL5"kv7Urn \!SgP;zh>ˊj\VovX<LgfxxkL1CdY\S;z(5TO[)5bu\mBj*
nUh&`Qί;ZxȜF
ԧ@})8}4#dzw&P^=AdT}*4 qS^E)̈cA$XDS]Z/_5M`~ӻRo'Ftw\e.G.3@m"\,c{'Gidv(TQY9zbpˆ9c#Y³Vs|If	ew7I%Grau07hf.;{Jʾx/R1.LT}Կ!kb
o8H	]=}SΈ퉃00U-rfbb,v0U00U#0FNqi$x'{(W+0U00}{http://bayca.bayhoto.local/ejbca/publicweb/webdist/certdist?cmd=crl&issuer=CN=RootCA,O=Bay%20Photo%20Lab,ST=California,C=US0U0
	*H
JMUZ>7gm[z }/.~^J;өƉ-Q_\Όh޲#ԾXL7ph(@`+8W&ib!Qj+ȡ1iT(#^( giZ9c<R꼓e.ݘVѬ峿ۅ8Dh$~mm啠~'\ET&	 a}rMKL0u%HYL
l=`Υ3k[؝Y}$ ss8?~IXKda<==mL[RҠsHBR/*`JfUzA)'0JkArvp#e-{]U
Z`#2Ϡv~.#l7"D=&t^-Q_9Mi
uԒn{Zn!U%r3J;Q׼Di@PNg]&;yw|9B*.L=Ij-)/]'g^U0#0b=0
	*H
0X10UBay Photo People CA10U

Bay Photo Lab10U
California10	UUS0
121023180003Z
141023180003Z0`10
	&,d1306910UMike Carlson10	UIT10U

Bay Photo Lab10	UUS0"0
	*H
0
<ȼ^|=e9KtФ-jI_	%[߲'O%3;=*n((RT	•͐C/\WU@HCjrIU-iE˼|paҨm-4݈amƵbK$"UEkEzd
w.
wG u:B'9!?tdk%%̞N.8C1ަί[
BjF0){C9&pXnĉZuX")3zsS\\D:L׏1Q}1Gzz(d#V3fRoш^CLfQ@S/StX
d5Y3M0ՙQ5ō;pIdV]&d#26zsgM}r#iМ|3)md:}뚁00R+F0D0B+06http://bayca.bayhoto.local/ejbca/publicweb/status/ocsp0U(}awJ״(#0U00U#0-rfbb,v0U00U%0
+0U0mike@bayphoto.com0
	*H
9|&V,*Hd	ƏA~6fFg'^y
I'yy,v}Z	@ᔘ7\F5QA37*LT4VStTe .Dӧ=n}=L\E	{
z7kYs#RO}E`OnL'1M0`Dۋ
rvVuX?s=	+O0:yE?BA̡5|Ʀpp*<FLA36k덝j9b=&)KJSmʐXo@g;V4@ujkX9	@Wh#nl\Y)A
rFGjqtvhu.ճK)L}@41AKz&ȴztÈ6͢j=0*+@;xnc-
WƣLG9X)=
y%]Q@BW
,Άut00MiC0
	*H
0X10UBay Photo People CA10U

Bay Photo Lab10U
California10	UUS0
121023175745Z
141023175745Z0`10
	&,d1306910UMike Carlson10	UIT10U

Bay Photo Lab10	UUS0"0
	*H
0
@vɌAļVAW5:eh$n>b%k7Pwޡ=^CBv2ULLqn6+>A:P#=ѕ[8Z<|&wb(x椉
iҒx9H?~Ɔ-y]jN崡1geAˇwH4w?h!/^Pؕfa5-+%<*/+`ZBCƀno|6'zoe!)@H藱$zѩ+
SXDz(~Bݬe?V
\j;.P,銉[JݦkjY*nȡ5]
hlkz3.Wme/tɧ#	8L%
Ũ%zp	_p)ڜ(C=MYe3S>Tfρ=@	]ڑav&0ۗ;.j'Yk_00R+F0D0B+06http://bayca.bayhoto.local/ejbca/publicweb/status/ocsp0UFO+Rdb`?60U00U#0-rfbb,v0U00http://bayca.bayhoto.local/ejbca/publicweb/webdist/certdist?cmd=crl&issuer=CN=Bay%20Photo%20People%20CA,O=Bay%20Photo%20Lab,ST=California,C=US0U0U%0++0U0mike@bayphoto.com0
	*H
/ungfsy@KLw.cM&6?-Y4++IJYD	C£S_2$eڏPU((̖S~aM0ri~jk2Ւ[n9rn&Bz(MݼIܪ*ȱImu5lr[Q`3͈;l{Z07h$>at)qo\]pJW7*[c%

y1FB)p2͞[~=?!Wd9XY5.bOKUDV[Z98E
^X9n<Hi@C?H+jlۗc݌&yqQ<Ii/
ɣ*B!f<.Re-=Y*?-4;|vj1@+Iܑ=J7%'jMmrSM@GV|:C'ݮ_Lkt61F0B0d0X10UBay Photo People CA10U

Bay Photo Lab10U
California10	UUSMiC0	+0	*H
	1	*H
0	*H
	1
140715051818Z0#	*H
	15@<Q2ɿ0l	*H
	1_0]0	`He*0	`He0
*H
0*H
0
*H
@0+0
*H
(0s	+71f0d0X10UBay Photo People CA10U

Bay Photo Lab10U
California10	UUSb=0u*H
	1fd0X10UBay Photo People CA10U

Bay Photo Lab10U
California10	UUSb=0
	*H
Poi$:
V\=kɣrϙ(~7msz'"vp[J͡iCCH", ŤuxGuKւZ5+Q$Y^5.7a76-v9CG-4]B>#Ee$ezf1,Oas<kPxtBF9W JGJmg#BZ^:>.+$>̴s/lJ}kZdOP׿
^
]5Pgg<(^yǶ^jEPL4iu2Z?fi
b"Q*۝j="UxD-R[wrW6`7tŅ<}9ᔼgL'PHkChv Sb)9;<^#j"?p^1D	-CI}K菈ܛ%=QB<-Y-=i

Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?53C4B99A.9000508>