Date: Mon, 17 Oct 2016 09:17:54 -0500 From: Karl Denninger <karl@denninger.net> To: freebsd-stable@freebsd.org Subject: Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE Message-ID: <3d4f25c9-a262-a373-ec7e-755325f8810b@denninger.net>
next in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format.
--------------ms020504020101080608050301
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
This is a situation I've had happen before, and reported -- it appeared
to be a kernel stack overflow, and it has gotten materially worse on
11.0-STABLE.
The issue occurs after some period of time (normally a week or so.) The
system has a mirrored pair of large drives used for backup purposes to
which ZFS snapshots are written using a script that iterates over the
system.
The panic /only /happens when the root filesystem is being sent, and it
appears that the panic itself is being triggered by an I/O pattern on
the /backup /drive -- not the source drives. Zpool scrubs on the source
are clean; I am going to run one now on the backup, but in the past that
has been clean as well.
I now have a *repeatable* panic in that if I attempt a "zfs list -rt all
backup" on the backup volume I get the below panic. A "zfs list"
does*__*not panic the system.
The operating theory previously (after digging through the passed
structures in the dump) was that the ZFS system was attempting to issue
TRIMs on a device that can't do them before the ZFS system realizes this
and stops asking (the backup volume is comprised of spinning rust) but
the appearance of the panic now on the volume when I simply do a "zfs
list -rt all backup" appears to negate that theory since no writes are
performed by that operation, and thus no TRIM calls should be issued.
I can leave the backup volume in the state that causes this for a short
period of time in an attempt to find and fix this.
NewFS.denninger.net dumped core - see /var/crash/vmcore.1
Mon Oct 17 09:02:33 CDT 2016
FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13
r307318M: Fri Oct 14 09:23:46 CDT 2016 =20
karl@NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP amd64
panic: double fault
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you =
are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for detail=
s.
This GDB was configured as "amd64-marcel-freebsd"...
Unread portion of the kernel message buffer:
Fatal double fault
rip =3D 0xffffffff8220d9ec
rsp =3D 0xfffffe066821f000
rbp =3D 0xfffffe066821f020
cpuid =3D 6; apic id =3D 14
panic: double fault
cpuid =3D 6
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
0xfffffe0649d78e30
vpanic() at vpanic+0x182/frame 0xfffffe0649d78eb0
panic() at panic+0x43/frame 0xfffffe0649d78f10
dblfault_handler() at dblfault_handler+0xa2/frame 0xfffffe0649d78f30
Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649d78f30
--- trap 0x17, rip =3D 0xffffffff8220d9ec, rsp =3D 0xfffffe066821f000, rb=
p =3D
0xfffffe066821f020 ---
avl_rotation() at avl_rotation+0xc/frame 0xfffffe066821f020
avl_remove() at avl_remove+0x1c8/frame 0xfffffe066821f070
vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame
0xfffffe066821f530
vdev_queue_io_done() at vdev_queue_io_done+0x83/frame 0xfffffe066821f570
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f5a0
zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f5f0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f650
zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f6a0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f6e0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f710
zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f760
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f7c0
zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f810
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f850
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f880
zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f8d0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f930
zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f980
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f9c0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f9f0
zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fa40
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821faa0
zio_execute() at zio_execute+0x23d/frame 0xfffffe066821faf0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821fb30
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fb60
zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fbb0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fc10
zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fc60
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821fca0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fcd0
zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fd20
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fd80
zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fdd0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821fe10
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fe40
zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fe90
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fef0
zio_execute() at zio_execute+0x23d/frame 0xfffffe066821ff40
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821ff80
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821ffb0
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220000
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220060
zio_execute() at zio_execute+0x23d/frame 0xfffffe06682200b0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682200f0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220120
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220170
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682201d0
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220220
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220260
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220290
zio_execute() at zio_execute+0x23d/frame 0xfffffe06682202e0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220340
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220390
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682203d0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220400
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220450
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682204b0
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220500
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220540
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220570
zio_execute() at zio_execute+0x23d/frame 0xfffffe06682205c0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220620
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220670
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682206b0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682206e0
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220730
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220790
zio_execute() at zio_execute+0x23d/frame 0xfffffe06682207e0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220820
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220850
zio_execute() at zio_execute+0x23d/frame 0xfffffe06682208a0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220900
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220950
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220990
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682209c0
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220a10
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220a70
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220ac0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220b00
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220b30
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220b80
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220be0
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220c30
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220c70
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220ca0
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220cf0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220d50
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220da0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220de0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220e10
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220e60
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220ec0
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220f10
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220f50
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220f80
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220fd0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221030
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221080
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682210c0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682210f0
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221140
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682211a0
zio_execute() at zio_execute+0x23d/frame 0xfffffe06682211f0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221230
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221260
zio_execute() at zio_execute+0x23d/frame 0xfffffe06682212b0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221310
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221360
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682213a0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682213d0
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221420
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221480
zio_execute() at zio_execute+0x23d/frame 0xfffffe06682214d0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221510
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221540
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221590
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682215f0
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221640
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221680
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682216b0
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221700
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221760
zio_execute() at zio_execute+0x23d/frame 0xfffffe06682217b0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682217f0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221820
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221870
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682218d0
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221920
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221960
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221990
zio_execute() at zio_execute+0x23d/frame 0xfffffe06682219e0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221a40
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221a90
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221ad0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221b00
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221b50
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221bb0
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221c00
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221c40
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221c70
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221cc0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221d20
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221d70
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221db0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221de0
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221e30
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221e90
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221ee0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221f20
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221f50
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221fa0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222000
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222050
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222090
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682220c0
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222110
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222170
zio_execute() at zio_execute+0x23d/frame 0xfffffe06682221c0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222200
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222230
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222280
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682222e0
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222330
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222370
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682223a0
zio_execute() at zio_execute+0x23d/frame 0xfffffe06682223f0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222450
zio_execute() at zio_execute+0x23d/frame 0xfffffe06682224a0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682224e0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222510
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222560
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682225c0
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222610
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222650
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222680
zio_execute() at zio_execute+0x23d/frame 0xfffffe06682226d0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222730
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222780
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682227c0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682227f0
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222840
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682228a0
zio_execute() at zio_execute+0x23d/frame 0xfffffe06682228f0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222930
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222960
zio_execute() at zio_execute+0x23d/frame 0xfffffe06682229b0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222a10
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222a60
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222aa0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222ad0
zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222b20
taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame
0xfffffe0668222b80
taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame
0xfffffe0668222bb0
fork_exit() at fork_exit+0x85/frame 0xfffffe0668222bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0668222bf0
--- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 ---
Uptime: 4m29s
Dumping 1497 out of 24531
MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91%
Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
/usr/lib/debug//boot/kernel/zfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from
/usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols from
/usr/lib/debug//boot/kernel/geom_eli.ko.debug...done.
done.
Loaded symbols for /boot/kernel/geom_eli.ko
Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from
/usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done.
done.
Loaded symbols for /boot/kernel/geom_mirror.ko
Reading symbols from /boot/kernel/aesni.ko...Reading symbols from
/usr/lib/debug//boot/kernel/aesni.ko.debug...done.
done.
Loaded symbols for /boot/kernel/aesni.ko
Reading symbols from /boot/kernel/uhid.ko...Reading symbols from
/usr/lib/debug//boot/kernel/uhid.ko.debug...done.
done.
Loaded symbols for /boot/kernel/uhid.ko
Reading symbols from /boot/kernel/ums.ko...Reading symbols from
/usr/lib/debug//boot/kernel/ums.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ums.ko
Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from
/usr/lib/debug//boot/kernel/ipfw.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ipfw.ko
Reading symbols from /boot/kernel/ipfw_nat.ko...Reading symbols from
/usr/lib/debug//boot/kernel/ipfw_nat.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ipfw_nat.ko
Reading symbols from /boot/kernel/libalias.ko...Reading symbols from
/usr/lib/debug//boot/kernel/libalias.ko.debug...done.
done.
Loaded symbols for /boot/kernel/libalias.ko
Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols from
/usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done.
done.
Loaded symbols for /boot/kernel/daemon_saver.ko
#0 doadump (textdump=3D1) at pcpu.h:221
221 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) #0 doadump (textdump=3D1) at pcpu.h:221
#1 0xffffffff80ae9b6e in kern_reboot (howto=3D260)
at /usr/src/sys/kern/kern_shutdown.c:366
#2 0xffffffff80aea13b in vpanic (fmt=3D<value optimized out>,
ap=3D<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759
#3 0xffffffff80ae9f73 in panic (fmt=3D0x0)
at /usr/src/sys/kern/kern_shutdown.c:690
#4 0xffffffff80fd18f2 in dblfault_handler (frame=3D<value optimized out>=
)
at /usr/src/sys/amd64/amd64/trap.c:868
#5 0xffffffff80fb406c in Xdblfault ()
at /usr/src/sys/amd64/amd64/exception.S:292
#6 0xffffffff8220d9ec in avl_rotation (tree=3D0xfffff8004dc54510,
node=3D0xfffff8056e45cdd0, balance=3D2)
at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:304
#7 0xffffffff8220de58 in avl_remove (tree=3D<value optimized out>,
data=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:818
#8 0xffffffff822a1eff in vdev_queue_io_to_issue (vq=3D<value optimized o=
ut>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:439
#9 0xffffffff822a2123 in vdev_queue_io_done (zio=3D<value optimized out>=
)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:907
#10 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e3f33d8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#11 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#12 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e3f33d8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#13 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#14 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out>=
)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#15 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e578000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#16 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#17 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e578000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#18 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#19 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out>=
)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#20 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5ec3d8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#21 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#22 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5ec3d8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#23 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#24 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out>=
)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#25 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e48a000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#26 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#27 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e48a000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#28 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#29 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out>=
)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#30 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e57ab88)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#31 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#32 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e57ab88)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#33 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#34 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out>=
)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#35 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800a69a3000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#36 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#37 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800a69a3000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#38 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#39 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out>=
)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#40 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5c8000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#41 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#42 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5c8000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#43 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#44 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out>=
)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#45 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a23f9000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#46 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#47 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a23f9000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#48 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#49 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out>=
)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#50 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5147b0)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#51 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#52 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5147b0)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#53 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#54 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out>=
)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#55 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e977000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#56 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#57 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e977000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#58 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#59 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out>=
)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#60 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e90b3d8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#61 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#62 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e90b3d8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#63 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#64 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out>=
)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#65 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800b1160b88)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#66 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#67 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800b1160b88)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#68 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#69 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out>=
)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#70 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e4f53d8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#71 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#72 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e4f53d8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#73 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#74 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out>=
)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#75 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e8ebb88)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#76 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#77 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e8ebb88)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#78 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#79 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out>=
)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#80 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e8d9b88)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#81 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#82 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e8d9b88)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#83 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#84 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out>=
)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#85 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a21a03d8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#86 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#87 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a21a03d8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#88 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#89 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out>=
)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#90 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e6903d8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#91 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#92 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e6903d8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#93 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#94 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out>=
)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#95 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a22dc7b0)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#96 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#97 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a22dc7b0)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#98 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#99 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out>=
)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#100 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5fb7b0)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#101 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#102 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5fb7b0)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#103 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#104 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out=
>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#105 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5ae3d8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#106 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#107 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5ae3d8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#108 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#109 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out=
>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#110 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a2131b88)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#111 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#112 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a2131b88)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#113 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#114 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out=
>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#115 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e80b7b0)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#116 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#117 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e80b7b0)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#118 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#119 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out=
>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#120 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800b14b9000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#121 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#122 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800b14b9000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#123 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#124 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out=
>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#125 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a21fa3d8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#126 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#127 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a21fa3d8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#128 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#129 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out=
>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#130 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5333d8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#131 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#132 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5333d8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#133 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#134 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out=
>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#135 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e90ab88)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#136 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#137 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e90ab88)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#138 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#139 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out=
>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#140 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e4047b0)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#141 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#142 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e4047b0)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#143 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#144 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out=
>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#145 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e7083d8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#146 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#147 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e7083d8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#148 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#149 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out=
>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#150 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a23917b0)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#151 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#152 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a23917b0)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#153 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#154 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out=
>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#155 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e636000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#156 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#157 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e636000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#158 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#159 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out=
>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#160 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5afb88)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#161 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#162 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5afb88)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#163 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#164 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out=
>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#165 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800a69523d8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#166 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#167 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800a69523d8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#168 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#169 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out=
>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#170 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e47c000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#171 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#172 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e47c000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#173 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#174 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out=
>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#175 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800b17b23d8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#176 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#177 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800b17b23d8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#178 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#179 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out=
>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#180 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800a6d367b0)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#181 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#182 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800a6d367b0)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#183 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#184 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out=
>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#185 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e99fb88)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#186 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#187 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e99fb88)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#188 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#189 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out=
>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#190 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5227b0)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#191 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#192 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5227b0)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112=
#193 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#194 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized out=
>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#195 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800a6d3c000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137=
#196 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651=
#197 0xffffffff80b4895a in taskqueue_run_locked (queue=3D<value optimized=
out>)
at /usr/src/sys/kern/subr_taskqueue.c:454
#198 0xffffffff80b49b58 in taskqueue_thread_loop (arg=3D<value optimized =
out>)
at /usr/src/sys/kern/subr_taskqueue.c:724
#199 0xffffffff80a9f255 in fork_exit (
callout=3D0xffffffff80b49a70 <taskqueue_thread_loop>,
arg=3D0xfffff8057caa72a0, frame=3D0xfffffe0668222c00)
at /usr/src/sys/kern/kern_fork.c:1040
#200 0xffffffff80fb44ae in fork_trampoline ()
at /usr/src/sys/amd64/amd64/exception.S:611
#201 0x0000000000000000 in ?? ()
Current language: auto; currently minimal
(kgdb)
--=20
Karl Denninger
karl@denninger.net <mailto:karl@denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/
--------------ms020504020101080608050301
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC
Bl8wggZbMIIEQ6ADAgECAgEpMA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G
A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl
bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND
dWRhIFN5c3RlbXMgTExDIENBMB4XDTE1MDQyMTAyMjE1OVoXDTIwMDQxOTAyMjE1OVowWjEL
MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM
TEMxHjAcBgNVBAMTFUthcmwgRGVubmluZ2VyIChPQ1NQKTCCAiIwDQYJKoZIhvcNAQEBBQAD
ggIPADCCAgoCggIBALmEWPhAdphrWd4K5VTvE5pxL3blRQPyGF3ApjUjgtavqU1Y8pbI3Byg
XDj2/Uz9Si8XVj/kNbKEjkRh5SsNvx3Fc0oQ1uVjyCq7zC/kctF7yLzQbvWnU4grAPZ3IuAp
3/fFxIVaXpxEdKmyZAVDhk9az+IgHH43rdJRIMzxJ5vqQMb+n2EjadVqiGPbtG9aZEImlq7f
IYDTnKyToi23PAnkPwwT+q1IkI2DTvf2jzWrhLR5DTX0fUYC0nxlHWbjgpiapyJWtR7K2YQO
aevQb/3vN9gSojT2h+cBem7QIj6U69rEYcEDvPyCMXEV9VcXdcmW42LSRsPvZcBHFkWAJqMZ
Myiz4kumaP+s+cIDaXitR/szoqDKGSHM4CPAZV9Yh8asvxQL5uDxz5wvLPgS5yS8K/o7zDR5
vNkMCyfYQuR6PAJxVOk5Arqvj9lfP3JSVapwbr01CoWDBkpuJlKfpQIEeC/pcCBKknllbMYq
yHBO2TipLyO5Ocd1nhN/nOsO+C+j31lQHfOMRZaPQykXVPWG5BbhWT7ttX4vy5hOW6yJgeT/
o3apynlp1cEavkQRS8uJHoQszF6KIrQMID/JfySWvVQ4ksnfzwB2lRomrdrwnQ4eG/HBS+0l
eozwOJNDIBlAP+hLe8A5oWZgooIIK/SulUAsfI6Sgd8dTZTTYmlhAgMBAAGjgfQwgfEwNwYI
KwYBBQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgw
CQYDVR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIB
DQQfFh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUxRyULenJaFwX
RtT79aNmIB/u5VkwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYw
FIESa2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBPf3cYtmKowmGIYsm6
eBinJu7QVWvxi1vqnBz3KE+HapqoIZS8/PolB/hwiY0UAE1RsjBJ7yEjihVRwummSBvkoOyf
G30uPn4yg4vbJkR9lTz8d21fPshWETa6DBh2jx2Qf13LZpr3Pj2fTtlu6xMYKzg7cSDgd2bO
sJGH/rcvva9Spkx5Vfq0RyOrYph9boshRN3D4tbWgBAcX9POdXCVfJONDxhfBuPHsJ6vEmPb
An+XL5Yl26XYFPiODQ+Qbk44Ot1kt9s7oS3dVUrh92Qv0G3J3DF+Vt6C15nED+f+bk4gScu+
JHT7RjEmfa18GT8DcT//D1zEke1Ymhb41JH+GyZchDRWtjxsS5OBFMzrju7d264zJUFtX7iJ
3xvpKN7VcZKNtB6dLShj3v/XDsQVQWXmR/1YKWZ93C3LpRs2Y5nYdn6gEOpL/WfQFThtfnat
HNc7fNs5vjotaYpBl5H8+VCautKbGOs219uQbhGZLYTv6okuKcY8W+4EJEtK0xB08vqr9Jd0
FS9MGjQE++GWo+5eQxFt6nUENHbVYnsr6bYPQsZH0CRNycgTG9MwY/UIXOf4W034UpR82TBG
1LiMsYfb8ahQJhs3wdf1nzipIjRwoZKT1vGXh/cj3gwSr64GfenURBxaFZA5O1acOZUjPrRT
n3ci4McYW/0WVVA3lDGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMH
RmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExD
MRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5
c3RlbXMgTExDIENBAgEpMA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZI
hvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNjEwMTcxNDE3NTRaME8GCSqGSIb3DQEJBDFCBEBs
owjsHsm1VcxRjJVMpRsb10xXGsHlMObTuzawp9lj4wmJmuVNu0RL4R7AfwttKTJQj1qOh5bJ
O6LRNIoT6khcMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAK
BggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYI
KoZIhvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNV
BAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1z
IExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3Vk
YSBTeXN0ZW1zIExMQyBDQQIBKTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYT
AlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1
ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG
9w0BCQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECASkwDQYJKoZIhvcNAQEBBQAEggIADlUn3EV3
cWZ4ZeXjbn63SwoIPlXRyPO2XUGADSSzqBiSCb+EvzI04dIKaJ1RAwABd9Vi5l6VaAeXktuv
8j7LOe/gkqs6vBhmJsHaf9cA1jCsRQj/GLEBy238SpdM925WLtDkJ4YPaBEuLwyxGBWx+x1a
PQMGG6HFh5D8c3Bb0t/gYNjWvvRYA099TCj2k5/ylbjjEdwymgM8ka2DDpYctw6fD/jEKz7O
H5FrjiRYWW6HazH/u5Y9UzqrNiY20lN2AqXtVs3v8VTXFQnUXXNAub1bBkg2k/tsoODcjakm
Rhrkf51a8L7M5FVacJlVEC2BZCX/dq73WxKqgL1EuhXeLyoxsCPYkqZqqeJlkpSa1n+7j9Ab
xjG9GwUdbgXnGxeFGjXmJKAT5qrN4nyy9iSVLuoMF5xujS/8qecyblRkuvLR4BUm/dzDPF8n
NdkoCs98aK8llt4CxEA3IpGYIkTC9gHWrDSvhrC8uHz2Y9e8P3qFmPvCON3h3SXXN2AQiooZ
B5A5gDV2W4F23Vv1hIrPFaNNj5SUJM7uSuUYENAf3q8YaFg0PWSbXVyv8QT5yXxpWrh2qMQZ
g6+GXrBeYO1JbUzJh/c10gS/N414SOvoQTm7NXFLAbTbr/y0cyRP5MMXrrjXo0Us05eO2KEL
og9TougdNm0JCWPQ/QdTIBUFIbAAAAAAAAA=
--------------ms020504020101080608050301--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3d4f25c9-a262-a373-ec7e-755325f8810b>