Date: Mon, 18 May 2015 07:56:01 -0500 From: Larry Rosenman <ler@lerctr.org> To: freebsd-current@freebsd.org Subject: Re: use after free panic ZFS Message-ID: <20150518125600.GA1274@borg.lerctr.org> In-Reply-To: <20150518124247.GA6220@borg.lerctr.org> References: <20150518124247.GA6220@borg.lerctr.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, May 18, 2015 at 07:42:47AM -0500, Larry Rosenman wrote:
> found the following panic this am:
>
> borg.lerctr.org dumped core - see /var/crash/vmcore.5
>
> Sun May 17 23:47:48 CDT 2015
>
> FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #40 r283007: Sat May 16 07:23:43 CDT 2015 root@borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER amd64
>
> panic: Most recently used by solaris
>
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for details.
> This GDB was configured as "amd64-marcel-freebsd"...
>
> Unread portion of the kernel message buffer:
> Memory modified after free 0xfffff808535ea000(120) val=deadc0dd @ 0xfffff808535ea050
> panic: Most recently used by solaris
>
> cpuid = 5
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe100bfb7660
> vpanic() at vpanic+0x189/frame 0xfffffe100bfb76e0
> panic() at panic+0x43/frame 0xfffffe100bfb7740
> mtrash_dtor() at mtrash_dtor/frame 0xfffffe100bfb7760
> uma_zalloc_arg() at uma_zalloc_arg+0x4c2/frame 0xfffffe100bfb77d0
> malloc() at malloc+0x198/frame 0xfffffe100bfb7820
> zfs_range_lock() at zfs_range_lock+0x4a/frame 0xfffffe100bfb7880
> zfs_get_data() at zfs_get_data+0x14c/frame 0xfffffe100bfb78f0
> zil_commit() at zil_commit+0x94c/frame 0xfffffe100bfb7a10
> zfs_freebsd_fsync() at zfs_freebsd_fsync+0xc8/frame 0xfffffe100bfb7a40
> VOP_FSYNC_APV() at VOP_FSYNC_APV+0xf7/frame 0xfffffe100bfb7a70
> sys_fsync() at sys_fsync+0x173/frame 0xfffffe100bfb7ae0
> amd64_syscall() at amd64_syscall+0x25a/frame 0xfffffe100bfb7bf0
> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe100bfb7bf0
> --- syscall (95, FreeBSD ELF64, sys_fsync), rip = 0x801eb5daa, rsp = 0x7fffffffd598, rbp = 0x7fffffffd5b0 ---
> Uptime: 1d14h25m26s
> Dumping 12469 out of 64457 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
>
> Reading symbols from /boot/kernel/linux.ko.symbols...done.
> Loaded symbols for /boot/kernel/linux.ko.symbols
> Reading symbols from /boot/kernel/if_lagg.ko.symbols...done.
> Loaded symbols for /boot/kernel/if_lagg.ko.symbols
> Reading symbols from /boot/kernel/snd_envy24ht.ko.symbols...done.
> Loaded symbols for /boot/kernel/snd_envy24ht.ko.symbols
> Reading symbols from /boot/kernel/snd_spicds.ko.symbols...done.
> Loaded symbols for /boot/kernel/snd_spicds.ko.symbols
> Reading symbols from /boot/kernel/coretemp.ko.symbols...done.
> Loaded symbols for /boot/kernel/coretemp.ko.symbols
> Reading symbols from /boot/kernel/ichsmb.ko.symbols...done.
> Loaded symbols for /boot/kernel/ichsmb.ko.symbols
> Reading symbols from /boot/kernel/smbus.ko.symbols...done.
> Loaded symbols for /boot/kernel/smbus.ko.symbols
> Reading symbols from /boot/kernel/ichwd.ko.symbols...done.
> Loaded symbols for /boot/kernel/ichwd.ko.symbols
> Reading symbols from /boot/kernel/cpuctl.ko.symbols...done.
> Loaded symbols for /boot/kernel/cpuctl.ko.symbols
> Reading symbols from /boot/kernel/crypto.ko.symbols...done.
> Loaded symbols for /boot/kernel/crypto.ko.symbols
> Reading symbols from /boot/kernel/cryptodev.ko.symbols...done.
> Loaded symbols for /boot/kernel/cryptodev.ko.symbols
> Reading symbols from /boot/kernel/dtraceall.ko.symbols...done.
> Loaded symbols for /boot/kernel/dtraceall.ko.symbols
> Reading symbols from /boot/kernel/profile.ko.symbols...done.
> Loaded symbols for /boot/kernel/profile.ko.symbols
> Reading symbols from /boot/kernel/dtrace.ko.symbols...done.
> Loaded symbols for /boot/kernel/dtrace.ko.symbols
> Reading symbols from /boot/kernel/systrace_freebsd32.ko.symbols...done.
> Loaded symbols for /boot/kernel/systrace_freebsd32.ko.symbols
> Reading symbols from /boot/kernel/systrace.ko.symbols...done.
> Loaded symbols for /boot/kernel/systrace.ko.symbols
> Reading symbols from /boot/kernel/sdt.ko.symbols...done.
> Loaded symbols for /boot/kernel/sdt.ko.symbols
> Reading symbols from /boot/kernel/lockstat.ko.symbols...done.
> Loaded symbols for /boot/kernel/lockstat.ko.symbols
> Reading symbols from /boot/kernel/fasttrap.ko.symbols...done.
> Loaded symbols for /boot/kernel/fasttrap.ko.symbols
> Reading symbols from /boot/kernel/fbt.ko.symbols...done.
> Loaded symbols for /boot/kernel/fbt.ko.symbols
> Reading symbols from /boot/kernel/dtnfscl.ko.symbols...done.
> Loaded symbols for /boot/kernel/dtnfscl.ko.symbols
> Reading symbols from /boot/kernel/dtmalloc.ko.symbols...done.
> Loaded symbols for /boot/kernel/dtmalloc.ko.symbols
> Reading symbols from /boot/modules/vboxdrv.ko...done.
> Loaded symbols for /boot/modules/vboxdrv.ko
> Reading symbols from /boot/modules/nvidia.ko...done.
> Loaded symbols for /boot/modules/nvidia.ko
> Reading symbols from /boot/kernel/ipmi.ko.symbols...done.
> Loaded symbols for /boot/kernel/ipmi.ko.symbols
> Reading symbols from /boot/kernel/ipmi_linux.ko.symbols...done.
> Loaded symbols for /boot/kernel/ipmi_linux.ko.symbols
> Reading symbols from /boot/kernel/radeonkms.ko.symbols...done.
> Loaded symbols for /boot/kernel/radeonkms.ko.symbols
> Reading symbols from /boot/kernel/iicbb.ko.symbols...done.
> Loaded symbols for /boot/kernel/iicbb.ko.symbols
> Reading symbols from /boot/kernel/iicbus.ko.symbols...done.
> Loaded symbols for /boot/kernel/iicbus.ko.symbols
> Reading symbols from /boot/kernel/iic.ko.symbols...done.
> Loaded symbols for /boot/kernel/iic.ko.symbols
> Reading symbols from /boot/kernel/drm2.ko.symbols...done.
> Loaded symbols for /boot/kernel/drm2.ko.symbols
> Reading symbols from /boot/kernel/radeonkmsfw_R100_cp.ko.symbols...done.
> Loaded symbols for /boot/kernel/radeonkmsfw_R100_cp.ko.symbols
> Reading symbols from /boot/kernel/uhid.ko.symbols...done.
> Loaded symbols for /boot/kernel/uhid.ko.symbols
> Reading symbols from /boot/kernel/ums.ko.symbols...done.
> Loaded symbols for /boot/kernel/ums.ko.symbols
> Reading symbols from /boot/modules/vboxnetflt.ko...done.
> Loaded symbols for /boot/modules/vboxnetflt.ko
> Reading symbols from /boot/kernel/netgraph.ko.symbols...done.
> Loaded symbols for /boot/kernel/netgraph.ko.symbols
> Reading symbols from /boot/kernel/ng_ether.ko.symbols...done.
> Loaded symbols for /boot/kernel/ng_ether.ko.symbols
> Reading symbols from /boot/modules/vboxnetadp.ko...done.
> Loaded symbols for /boot/modules/vboxnetadp.ko
> #0 doadump (textdump=Unhandled dwarf expression opcode 0x93
> ) at pcpu.h:221
> 221 pcpu.h: No such file or directory.
> in pcpu.h
> (kgdb) #0 doadump (textdump=Unhandled dwarf expression opcode 0x93
> ) at pcpu.h:221
> #1 0xffffffff80a839b5 in kern_reboot (howto=Unhandled dwarf expression opcode 0x93
> )
> at /usr/src/sys/kern/kern_shutdown.c:447
> #2 0xffffffff80a83fa8 in vpanic (fmt=<value optimized out>,
> ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:744
> #3 0xffffffff80a83ff3 in panic (fmt=0x0)
> at /usr/src/sys/kern/kern_shutdown.c:675
> #4 0xffffffff80d13750 in mtrash_ctor (mem=<value optimized out>,
> size=<value optimized out>, arg=<value optimized out>,
> flags=<value optimized out>) at /usr/src/sys/vm/uma_dbg.c:138
> #5 0xffffffff80d0f6d2 in uma_zalloc_arg (zone=0xfffff80ffffc9680, udata=0x0,
> flags=2) at /usr/src/sys/vm/uma_core.c:2197
> #6 0xffffffff80a64158 in malloc (size=<value optimized out>,
> mtp=0xffffffff815e16e0, flags=<value optimized out>) at uma.h:336
> #7 0xffffffff80402b4a in zfs_range_lock (zp=0xfffff8075e835730, off=9158656,
> len=8192, type=Unhandled dwarf expression opcode 0x93
> )
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_rlock.c:432
> #8 0xffffffff8040886c in zfs_get_data (arg=<value optimized out>,
> lr=<value optimized out>,
> buf=0xfffffe0662be8178 <Address 0xfffffe0662be8178 out of bounds>,
> zio=0xfffff80d78b89ac8)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1250
> #9 0xffffffff8041c71c in zil_commit (zilog=0xfffff800185c1400,
> foid=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zil.c:1108
> #10 0xffffffff80410168 in zfs_freebsd_fsync (ap=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:2747
> #11 0xffffffff80fdfcd7 in VOP_FSYNC_APV (vop=<value optimized out>,
> a=<value optimized out>) at vnode_if.c:1328
> #12 0xffffffff80b40883 in sys_fsync (td=0xfffff8011b253940,
> uap=<value optimized out>) at vnode_if.h:549
> #13 0xffffffff80e968da in amd64_syscall (td=0xfffff8011b253940, traced=0)
> at subr_syscall.c:133
> #14 0xffffffff80e767bb in Xfast_syscall ()
> at /usr/src/sys/amd64/amd64/exception.S:395
> #15 0x0000000801eb5daa in ?? ()
> Previous frame inner to this frame (corrupt stack?)
> Current language: auto; currently minimal
> (kgdb)
>
> I have the core.
And, trying to re-compile to pick up the latest, got another one:
borg.lerctr.org dumped core - see /var/crash/vmcore.6
Mon May 18 07:51:57 CDT 2015
FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #40 r283007: Sat May 16 07:23:43 CDT 2015 root@borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER amd64
panic: Most recently used by solaris
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Unread portion of the kernel message buffer:
Memory modified after free 0xfffff80c88f1f980(120) val=deadc0dd @ 0xfffff80c88f1f9c0
panic: Most recently used by solaris
cpuid = 0
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe100c459600
vpanic() at vpanic+0x189/frame 0xfffffe100c459680
panic() at panic+0x43/frame 0xfffffe100c4596e0
mtrash_dtor() at mtrash_dtor/frame 0xfffffe100c459700
uma_zalloc_arg() at uma_zalloc_arg+0x4c2/frame 0xfffffe100c459770
malloc() at malloc+0x198/frame 0xfffffe100c4597c0
zfs_range_lock() at zfs_range_lock+0x4a/frame 0xfffffe100c459820
zfs_freebsd_read() at zfs_freebsd_read+0x1c7/frame 0xfffffe100c4598c0
VOP_READ_APV() at VOP_READ_APV+0xf1/frame 0xfffffe100c4598f0
vn_read() at vn_read+0x237/frame 0xfffffe100c459970
vn_io_fault() at vn_io_fault+0x10a/frame 0xfffffe100c4599f0
dofileread() at dofileread+0x95/frame 0xfffffe100c459a40
kern_readv() at kern_readv+0x68/frame 0xfffffe100c459a90
sys_read() at sys_read+0x63/frame 0xfffffe100c459ae0
amd64_syscall() at amd64_syscall+0x25a/frame 0xfffffe100c459bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe100c459bf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8009638fa, rsp = 0x7fffffffe968, rbp = 0x7fffffffe980 ---
Uptime: 7h59m25s
Dumping 14815 out of 64457 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
Reading symbols from /boot/kernel/linux.ko.symbols...done.
Loaded symbols for /boot/kernel/linux.ko.symbols
Reading symbols from /boot/kernel/if_lagg.ko.symbols...done.
Loaded symbols for /boot/kernel/if_lagg.ko.symbols
Reading symbols from /boot/kernel/snd_envy24ht.ko.symbols...done.
Loaded symbols for /boot/kernel/snd_envy24ht.ko.symbols
Reading symbols from /boot/kernel/snd_spicds.ko.symbols...done.
Loaded symbols for /boot/kernel/snd_spicds.ko.symbols
Reading symbols from /boot/kernel/coretemp.ko.symbols...done.
Loaded symbols for /boot/kernel/coretemp.ko.symbols
Reading symbols from /boot/kernel/ichsmb.ko.symbols...done.
Loaded symbols for /boot/kernel/ichsmb.ko.symbols
Reading symbols from /boot/kernel/smbus.ko.symbols...done.
Loaded symbols for /boot/kernel/smbus.ko.symbols
Reading symbols from /boot/kernel/ichwd.ko.symbols...done.
Loaded symbols for /boot/kernel/ichwd.ko.symbols
Reading symbols from /boot/kernel/cpuctl.ko.symbols...done.
Loaded symbols for /boot/kernel/cpuctl.ko.symbols
Reading symbols from /boot/kernel/crypto.ko.symbols...done.
Loaded symbols for /boot/kernel/crypto.ko.symbols
Reading symbols from /boot/kernel/cryptodev.ko.symbols...done.
Loaded symbols for /boot/kernel/cryptodev.ko.symbols
Reading symbols from /boot/kernel/dtraceall.ko.symbols...done.
Loaded symbols for /boot/kernel/dtraceall.ko.symbols
Reading symbols from /boot/kernel/profile.ko.symbols...done.
Loaded symbols for /boot/kernel/profile.ko.symbols
Reading symbols from /boot/kernel/dtrace.ko.symbols...done.
Loaded symbols for /boot/kernel/dtrace.ko.symbols
Reading symbols from /boot/kernel/systrace_freebsd32.ko.symbols...done.
Loaded symbols for /boot/kernel/systrace_freebsd32.ko.symbols
Reading symbols from /boot/kernel/systrace.ko.symbols...done.
Loaded symbols for /boot/kernel/systrace.ko.symbols
Reading symbols from /boot/kernel/sdt.ko.symbols...done.
Loaded symbols for /boot/kernel/sdt.ko.symbols
Reading symbols from /boot/kernel/lockstat.ko.symbols...done.
Loaded symbols for /boot/kernel/lockstat.ko.symbols
Reading symbols from /boot/kernel/fasttrap.ko.symbols...done.
Loaded symbols for /boot/kernel/fasttrap.ko.symbols
Reading symbols from /boot/kernel/fbt.ko.symbols...done.
Loaded symbols for /boot/kernel/fbt.ko.symbols
Reading symbols from /boot/kernel/dtnfscl.ko.symbols...done.
Loaded symbols for /boot/kernel/dtnfscl.ko.symbols
Reading symbols from /boot/kernel/dtmalloc.ko.symbols...done.
Loaded symbols for /boot/kernel/dtmalloc.ko.symbols
Reading symbols from /boot/modules/vboxdrv.ko...done.
Loaded symbols for /boot/modules/vboxdrv.ko
Reading symbols from /boot/modules/nvidia.ko...done.
Loaded symbols for /boot/modules/nvidia.ko
Reading symbols from /boot/kernel/ipmi.ko.symbols...done.
Loaded symbols for /boot/kernel/ipmi.ko.symbols
Reading symbols from /boot/kernel/ipmi_linux.ko.symbols...done.
Loaded symbols for /boot/kernel/ipmi_linux.ko.symbols
Reading symbols from /boot/kernel/radeonkms.ko.symbols...done.
Loaded symbols for /boot/kernel/radeonkms.ko.symbols
Reading symbols from /boot/kernel/iicbb.ko.symbols...done.
Loaded symbols for /boot/kernel/iicbb.ko.symbols
Reading symbols from /boot/kernel/iicbus.ko.symbols...done.
Loaded symbols for /boot/kernel/iicbus.ko.symbols
Reading symbols from /boot/kernel/iic.ko.symbols...done.
Loaded symbols for /boot/kernel/iic.ko.symbols
Reading symbols from /boot/kernel/drm2.ko.symbols...done.
Loaded symbols for /boot/kernel/drm2.ko.symbols
Reading symbols from /boot/kernel/radeonkmsfw_R100_cp.ko.symbols...done.
Loaded symbols for /boot/kernel/radeonkmsfw_R100_cp.ko.symbols
Reading symbols from /boot/kernel/uhid.ko.symbols...done.
Loaded symbols for /boot/kernel/uhid.ko.symbols
Reading symbols from /boot/kernel/ums.ko.symbols...done.
Loaded symbols for /boot/kernel/ums.ko.symbols
Reading symbols from /boot/modules/vboxnetflt.ko...done.
Loaded symbols for /boot/modules/vboxnetflt.ko
Reading symbols from /boot/kernel/netgraph.ko.symbols...done.
Loaded symbols for /boot/kernel/netgraph.ko.symbols
Reading symbols from /boot/kernel/ng_ether.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_ether.ko.symbols
Reading symbols from /boot/modules/vboxnetadp.ko...done.
Loaded symbols for /boot/modules/vboxnetadp.ko
#0 doadump (textdump=Unhandled dwarf expression opcode 0x93
) at pcpu.h:221
221 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) #0 doadump (textdump=Unhandled dwarf expression opcode 0x93
) at pcpu.h:221
#1 0xffffffff80a839b5 in kern_reboot (howto=Unhandled dwarf expression opcode 0x93
)
at /usr/src/sys/kern/kern_shutdown.c:447
#2 0xffffffff80a83fa8 in vpanic (fmt=<value optimized out>,
ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:744
#3 0xffffffff80a83ff3 in panic (fmt=0x0)
at /usr/src/sys/kern/kern_shutdown.c:675
#4 0xffffffff80d13750 in mtrash_ctor (mem=<value optimized out>,
size=<value optimized out>, arg=<value optimized out>,
flags=<value optimized out>) at /usr/src/sys/vm/uma_dbg.c:138
#5 0xffffffff80d0f6d2 in uma_zalloc_arg (zone=0xfffff80ffffc9680, udata=0x0,
flags=2) at /usr/src/sys/vm/uma_core.c:2197
#6 0xffffffff80a64158 in malloc (size=<value optimized out>,
mtp=0xffffffff815e16e0, flags=<value optimized out>) at uma.h:336
#7 0xffffffff80402b4a in zfs_range_lock (zp=0xfffff806afc8d170,
off=75316383, len=131072, type=Unhandled dwarf expression opcode 0x93
)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_rlock.c:432
#8 0xffffffff8040e517 in zfs_freebsd_read (ap=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:703
#9 0xffffffff80fdf3b1 in VOP_READ_APV (vop=<value optimized out>,
a=<value optimized out>) at vnode_if.c:930
#10 0xffffffff80b461b7 in vn_read (fp=0xfffff8002e1389b0,
uio=0xfffffe100c459ab0, active_cred=<value optimized out>,
flags=<value optimized out>, td=0x0) at vnode_if.h:384
#11 0xffffffff80b425ea in vn_io_fault (fp=0xfffff8002e1389b0,
uio=0xfffffe100c459ab0, active_cred=0x0, flags=0, td=0x0)
at /usr/src/sys/kern/vfs_vnops.c:1167
#12 0xffffffff80ae1525 in dofileread (td=0xfffff804bb5f8940, fd=3,
fp=0xfffff8002e1389b0, auio=0xfffffe100c459ab0,
offset=<value optimized out>, flags=Unhandled dwarf expression opcode 0x93
) at file.h:296
#13 0xffffffff80ae1228 in kern_readv (td=0xfffff804bb5f8940, fd=Unhandled dwarf expression opcode 0x93
)
at /usr/src/sys/kern/sys_generic.c:272
#14 0xffffffff80ae11b3 in sys_read (td=0x0, uap=<value optimized out>)
at /usr/src/sys/kern/sys_generic.c:185
#15 0xffffffff80e968da in amd64_syscall (td=0xfffff804bb5f8940, traced=0)
at subr_syscall.c:133
#16 0xffffffff80e767bb in Xfast_syscall ()
at /usr/src/sys/amd64/amd64/exception.S:395
#17 0x00000008009638fa in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language: auto; currently minimal
(kgdb)
I have BOTH cores.
--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 E-Mail: ler@lerctr.org
US Mail: 108 Turvey Cove, Hutto, TX 78634-5688
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20150518125600.GA1274>