Date: Mon, 14 Jul 2014 22:18:18 -0700 From: Mike Carlson <mike@bayphoto.com> To: freebsd-fs@freebsd.org Subject: ZFS Panic on 10.0-RELEASE - again Message-ID: <53C4B99A.9000508@bayphoto.com>
next in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format.
--------------ms050603000305030104030407
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: quoted-printable
I posted to the list about a month ago where our ZFS pool panic upon=20
mounting
Well, after backing up and restoring from some snapshots, I rebuilt the=20
server on July 4th and restored the data.
Today, competely unrelated to zfs, our root volume (UFS2) was running a=20
portsnap fetch extract and trigger a separate panic
I assumed the ZFS data was fine, as it was unrelated. That was not the=20
case. After re-installing and performing a zpool import, our pool panic=20
yet again.
This is a completely different situation, and again, I've lost around=20
20TB of data.
Here is the vmcore.0's backtrace:
Fatal trap 12: page fault while in kernel mode
cpuid =3D 2; apic id =3D 12
fault virtual address =3D 0x50
fault code =3D supervisor read data, page not present
instruction pointer =3D 0x20:0xffffffff81a85246
stack pointer =3D 0x28:0xfffffe104cb5aab0
frame pointer =3D 0x28:0xfffffe104cb5aac0
code segment =3D base 0x0, limit 0xfffff, type 0x1b
=3D DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags =3D interrupt enabled, resume, IOPL =3D 0
current process =3D 0 (system_taskq_7)
trap number =3D 12
panic: page fault
cpuid =3D 2
KDB: stack backtrace:
#0 0xffffffff808e7dd0 at kdb_backtrace+0x60
#1 0xffffffff808af8b5 at panic+0x155
#2 0xffffffff80c8e692 at trap_fatal+0x3a2
#3 0xffffffff80c8e969 at trap_pfault+0x2c9
#4 0xffffffff80c8e0f6 at trap+0x5e6
#5 0xffffffff80c75392 at calltrap+0x8
#6 0xffffffff81a8b710 at vdev_mirror_child_select+0x70
#7 0xffffffff81a8b254 at vdev_mirror_io_start+0x234
#8 0xffffffff81aa52d4 at zio_vdev_io_start+0x184
#9 0xffffffff81aa26a6 at zio_execute+0x136
#10 0xffffffff81a32dec at arc_read+0x87c
#15 0xffffffff81a4aee3 at traverse_visitbp+0x393
#16 0xffffffff81a4aee3 at traverse_visitbp+0x393
#17 0xffffffff81a4aee3 at traverse_visitbp+0x393
Uptime: 4m27s
Dumping 2286 out of 65496=20
MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
Reading symbols from /boot/kernel/ums.ko.symbols...done.
Loaded symbols for /boot/kernel/ums.ko.symbols
Reading symbols from /boot/kernel/zfs.ko.symbols...done.
Loaded symbols for /boot/kernel/zfs.ko.symbols
Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
Loaded symbols for /boot/kernel/opensolaris.ko.symbols
#0 doadump (textdump=3D<value optimized out>) at pcpu.h:219
219 __asm("movq %%gs:%1,%0" : "=3Dr" (td)
(kgdb) bt
#0 doadump (textdump=3D<value optimized out>) at pcpu.h:219
#1 0xffffffff808af530 in kern_reboot (howto=3D260) at=20
/usr/src/sys/kern/kern_shutdown.c:447
#2 0xffffffff808af8f4 in panic (fmt=3D<value optimized out>) at=20
/usr/src/sys/kern/kern_shutdown.c:754
#3 0xffffffff80c8e692 in trap_fatal (frame=3D<value optimized out>,=20
eva=3D<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:882
#4 0xffffffff80c8e969 in trap_pfault (frame=3D0xfffffe104cb5aa00,=20
usermode=3D0) at /usr/src/sys/amd64/amd64/trap.c:699
#5 0xffffffff80c8e0f6 in trap (frame=3D0xfffffe104cb5aa00) at=20
/usr/src/sys/amd64/amd64/trap.c:463
#6 0xffffffff80c75392 in calltrap () at=20
/usr/src/sys/amd64/amd64/exception.S:232
#7 0xffffffff81a85246 in vdev_validate (vd=3D0xfffff80116622c10,=20
strict=3D<value optimized out>) at=20
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs=
/vdev.c:1451
#8 0xffffffff81a8b710 in vdev_mirror_io_done (zio=3D0x20) at=20
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs=
/vdev_mirror.c:397
#9 0xffffffff81a8b254 in vdev_mirror_io_start (zio=3D0xfffff80116622c00)=
=20
at=20
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs=
/vdev_mirror.c:90
#10 0xffffffff81aa52d4 in zio_vdev_io_start (zio=3D0xfffff80015db43b0) at=
=20
time.h:63
#11 0xffffffff81aa26a6 in zio_execute (zio=3D0xfffff80015db43b0) at=20
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs=
/zio.c:1319
#12 0xffffffff81a32dec in arc_read (pio=3D0x0, spa=3D0xfffff80015e0a000, =
bp=3D<value optimized out>, done=3D0x2, private=3D0x0, priority=3D6,=20
zio_flags=3D0, arc_flags=3D<value optimized out>,
zb=3D0xfffff80116d7b048) at=20
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs=
/arc.c:3371
#13 0xffffffff81a4b8f1 in traverse_prefetcher (spa=3D0xfffff80015e0a000, =
zilog=3D0xf01ff, bp=3D<value optimized out>, zb=3D<value optimized out>, =
dnp=3D0xfffff80116622c00, arg=3D<value optimized out>)
at=20
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs=
/dmu_traverse.c:451
#14 0xffffffff81a4ad14 in traverse_visitbp (td=3D0xfffffe104cb5b900,=20
dnp=3D0xfffffe00112ba800, bp=3D0xfffffe00112ba980, zb=3D0xfffffe104cb5ae8=
8)
at=20
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs=
/dmu_traverse.c:250
#15 0xffffffff81a4b77f in traverse_dnode (td=3D0xfffffe104cb5b900,=20
dnp=3D0xfffffe00112ba800, objset=3D203, object=3D24767564)
at=20
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs=
/dmu_traverse.c:417
#16 0xffffffff81a4b487 in traverse_visitbp (td=3D0xfffffe104cb5b900,=20
dnp=3D0xfffffe00112b9000, bp=3D0xfffffe0012145100, zb=3D0xfffffe104cb5b0a=
8)
at=20
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs=
/dmu_traverse.c:309
#17 0xffffffff81a4aee3 in traverse_visitbp (td=3D0xfffffe104cb5b900,=20
dnp=3D0xfffff80010f5d800, bp=3D0xfffffe00121d9f00, zb=3D0xfffffe104cb5b1d=
8)
at=20
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs=
/dmu_traverse.c:284
#18 0xffffffff81a4aee3 in traverse_visitbp (td=3D0xfffffe104cb5b900,=20
dnp=3D0xfffff80010f5d800, bp=3D0xfffffe001057b780, zb=3D0xfffffe104cb5b30=
8)
at=20
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs=
/dmu_traverse.c:284
#19 0xffffffff81a4aee3 in traverse_visitbp (td=3D0xfffffe104cb5b900,=20
dnp=3D0xfffff80010f5d800, bp=3D0xfffffe000eb3c000, zb=3D0xfffffe104cb5b43=
8)
at=20
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs=
/dmu_traverse.c:284
#20 0xffffffff81a4aee3 in traverse_visitbp (td=3D0xfffffe104cb5b900,=20
dnp=3D0xfffff80010f5d800, bp=3D0xfffffe000eb28000, zb=3D0xfffffe104cb5b56=
8)
at=20
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs=
/dmu_traverse.c:284
#21 0xffffffff81a4aee3 in traverse_visitbp (td=3D0xfffffe104cb5b900,=20
dnp=3D0xfffff80010f5d800, bp=3D0xfffffe0011694000, zb=3D0xfffffe104cb5b69=
8)
at=20
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs=
/dmu_traverse.c:284
#22 0xffffffff81a4aee3 in traverse_visitbp (td=3D0xfffffe104cb5b900,=20
dnp=3D0xfffff80010f5d800, bp=3D0xfffff80010f5d840, zb=3D0xfffffe104cb5b75=
8)
at=20
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs=
/dmu_traverse.c:284
#23 0xffffffff81a4b714 in traverse_dnode (td=3D0xfffffe104cb5b900,=20
dnp=3D0xfffff80010f5d800, objset=3D203, object=3D0)
at=20
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs=
/dmu_traverse.c:407
#24 0xffffffff81a4b190 in traverse_visitbp (td=3D0xfffffe104cb5b900,=20
dnp=3D0x0, bp=3D0xfffff80116b3c880, zb=3D0xfffffe104cb5b8e0)
at=20
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs=
/dmu_traverse.c:338
#25 0xffffffff81a4aaf6 in traverse_prefetch_thread=20
(arg=3D0xfffffe104ca9a3a0) at=20
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs=
/dmu_traverse.c:470
#26 0xffffffff81a21c00 in taskq_run (arg=3D0xfffff80116cb5780,=20
pending=3D983551) at=20
/usr/src/sys/modules/zfs/../../cddl/compat/opensolaris/kern/opensolaris_t=
askq.c:109
#27 0xffffffff808f5b66 in taskqueue_run_locked=20
(queue=3D0xfffff8001535e800) at /usr/src/sys/kern/subr_taskqueue.c:333
#28 0xffffffff808f63e8 in taskqueue_thread_loop (arg=3D<value optimized=20
out>) at /usr/src/sys/kern/subr_taskqueue.c:535
#29 0xffffffff8088198a in fork_exit (callout=3D0xffffffff808f6340=20
<taskqueue_thread_loop>, arg=3D0xfffff80015025e80,=20
frame=3D0xfffffe104cb5ba40) at /usr/src/sys/kern/kern_fork.c:995
#30 0xffffffff80c758ce in fork_trampoline () at=20
/usr/src/sys/amd64/amd64/exception.S:606
#31 0x0000000000000000 in ?? ()
Current language: auto; currently minimal
Pool creation info:
zpool create data raidz /dev/da2.nop /dev/da3.nop /dev/da4.nop /dev/da5.n=
op
zpool add data raidz /dev/da6.nop /dev/da7.nop /dev/da8.nop /dev/da9.nop
zpool add data raidz /dev/da10.nop /dev/da11.nop /dev/da12.nop /dev/da13.=
nop
zpool add data raidz /dev/da14.nop /dev/da15.nop /dev/da16.nop /dev/da17.=
nop
zpool add data raidz /dev/da18.nop /dev/da19.nop /dev/da20.nop /dev/da21.=
nop
zpool add data raidz /dev/da22.nop /dev/da23.nop /dev/da24.nop /dev/da25.=
nop
zpool add data spare /dev/da26.nop /dev/da27.nop
zpool add data log /dev/gpt/log.nop
zpool add data cache /dev/gpt/cache.nop
The pool has a log and cache, and I suspect the log device is corrupt.
All zfs commands cause a panic (zdb, zfs list, zpool status, etc...)
I'm also wondering if the LSI controller itself is buggy:
mps0: <LSI SAS2004> port 0xc000-0xc0ff mem=20
0xfe83c000-0xfe83ffff,0xfe840000-0xfe87ffff irq 28 at device 0.0 on pci1
mps0: Firmware: 15.00.00.00, Driver: 16.00.00.00-fbsd
mps0: IOCCapabilities:=20
185c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR>
--------------ms050603000305030104030407
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIITIjCC
BhwwggQEoAMCAQICCGWP9DPVdj3ZMA0GCSqGSIb3DQEBCwUAMEsxDzANBgNVBAMMBlJvb3RD
QTEWMBQGA1UECgwNQmF5IFBob3RvIExhYjETMBEGA1UECAwKQ2FsaWZvcm5pYTELMAkGA1UE
BhMCVVMwHhcNMTIxMDIzMTczMjE4WhcNMjcxMDIzMTczMjE4WjBYMRwwGgYDVQQDDBNCYXkg
UGhvdG8gUGVvcGxlIENBMRYwFAYDVQQKDA1CYXkgUGhvdG8gTGFiMRMwEQYDVQQIDApDYWxp
Zm9ybmlhMQswCQYDVQQGEwJVUzCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBALuW
vZE7FFSO77jEhdT/dZzu0XnPS35aepuvMvCNTagnNMbaf+AKrLZFaVRqBt4f6yl5TDXNEAMI
+CJrg7T9dsoGN1Vygm4bIFzOIRtTZ1A7seemeonyqMto8gYMPhEFicuKavEAFcCFprVcVu5v
yXbEG+tYkx08oUzeZ9LWZnh4a0wxAucbH0PcZFmdXFMbgjt6GSiR8jVUT1t/Dik1+mJ1A4K3
3VyM+G1CaqGIKg2UbvHiVQVoJrpgUc6vO5aWWgDreH+CAKLpyJwCj4BGphfHBPENu5LUp0CL
fa4pA4r+OH0HNBp/I7utZMrIevh3pya34lDJXj0CQWSuVBR9Kp80IHHYGuyPU/sMXkUplsyI
Y4KgQSRYRFNdWoboLwsRu1/2iDUaTR+XYH7Tu5zvCOedUm8ns/wfE7TW6kax5nR36VzuZS6B
kUcuETO1QMxt2SJcHZIftiyTuxC8Y9Z7J6igR39pZPWGywJ2+oPRKFQD/u9RWZqjh9k5euVi
4bVwwog5DqnsymMTuKb0I1nCs5js+rfEVo+mc4gAfEmzk/dmpwkOnGV3oTfASbGtwMIlAEe5
HHJhdTA3jpVo3eQAZi479xZ7SozKvnjt1y9SMQ4uTJXjER/IVH3YC9S/BSFrqWLLDfWEGW84
SBn+ogldDIy3PX3SU7nOiN4Vse2Jg+r7AgMBAAGjgfYwgfMwHQYDVR0OBBYEFC34wnK9ZmLm
B7vV52IeLHaapJ33MA8GA1UdEwEB/wQFMAMBAf8wHwYDVR0jBBgwFoAU00bJTnGPaRmhJAUZ
t4x4J3soVyswgY8GA1UdHwSBhzCBhDCBgaB/oH2Ge2h0dHA6Ly9iYXljYS5iYXlob3RvLmxv
Y2FsL2VqYmNhL3B1YmxpY3dlYi93ZWJkaXN0L2NlcnRkaXN0P2NtZD1jcmwmaXNzdWVyPUNO
PVJvb3RDQSxPPUJheSUyMFBob3RvJTIwTGFiLFNUPUNhbGlmb3JuaWEsQz1VUzAOBgNVHQ8B
Af8EBAMCAYYwDQYJKoZIhvcNAQELBQADggIBAEoDTeb1wVXz2VqFPrc3uaRnbVt6DCB9uBft
L8YRgi5+FF5KHjvTqYHGiS38yVFfXM6MaN6yI7WSwtS+wVhM7Tdw6miwKNBApmCDKzjorNVX
iZUmv88fjGm652LeIfT/oFGJg2oruoad9OfIofjT/jFpiFT7KOgjg4xeKCC0f+9naVo5uWOJ
HTyFm1Kq6ryT+g7mZb7kLvndmJRW0aybn8Dls7/bhThEaOYkwn4dbZ6q0W2I5ZWg9bbh66V+
J9P1XObZA/TkRVTlpSYJ1hAgYX1yTbL360vuTAAwdSWQSFm4TAoXC4BsPdlgy86lDjNrW9id
WX0RJCBzk/FzHhI4Aj9+SVhLl/Vkf9nbz8VhPPK2Az0PPYKy5ARtgOqGTKZbgVKut9Kgc/fn
vUgPhObHQsD1Ui+NKsIGYBFKBhNmVXqMQSn4JzC9x0oDEmv37UGrcut2cCP3ZS17p137VaUM
lQ0RWomju+sPCPFgyCPa/TLPoMZ2334uIxkRbDefvOoXIosORMQ9Jh50XqktUesbhuBfH9Q5
8h8bTWm1Cn/LxXW9qdSSbnta0OAH1G4hwVUlcusSM0o7Ude8tszw6kRpmEDDE8BQjE5nXSY7
wOf1eXfQfDkVQiouTD3l5ElqB98tnCnL/y9dphWoBLmiJwgb/4yWZ/Zewc1V65UFr7LmvcQM
MIIGIzCCBAugAwIBAgIIYpSXgZOT7j0wDQYJKoZIhvcNAQELBQAwWDEcMBoGA1UEAwwTQmF5
IFBob3RvIFBlb3BsZSBDQTEWMBQGA1UECgwNQmF5IFBob3RvIExhYjETMBEGA1UECAwKQ2Fs
aWZvcm5pYTELMAkGA1UEBhMCVVMwHhcNMTIxMDIzMTgwMDAzWhcNMTQxMDIzMTgwMDAzWjBg
MRUwEwYKCZImiZPyLGQBAQwFMTMwNjkxFTATBgNVBAMMDE1pa2UgQ2FybHNvbjELMAkGA1UE
CwwCSVQxFjAUBgNVBAoMDUJheSBQaG90byBMYWIxCzAJBgNVBAYTAlVTMIICIjANBgkqhkiG
9w0BAQEFAAOCAg8AMIICCgKCAgEAoTzIvF66A3wYPWXQFzmjBUuIrHSQGhOcE7rb0KQtar+H
rkmHX56bCAklW5P/pd+yJ08lMwb3CxbTOz0q47JuBv2kKJO1xCgCua26Uvz3VAmfirmWwpXq
zZBDqy/bEIt/XFfiVUC4jriGSEPrtx9q9nJJsb2JVRgtsbcHaaJFu8u8s8p8cLbcYdKobS00
g6+7it2IpIJhxc5tEMa1Yku3kCQiHVVFa9b4H5pFDHpkCrKZ43cuCneiR5kgr47z/3U66kLt
J7Q5IT/i7nThjGQMa/f1JSWet8yeTomKvqkuEAA4o/IWQzEbtxzeps6vWxaCDULjEq69s//S
6PtqiQSmG9ZGFoPYD8/GGd4CMBqgjKopintD5sGTlJ851yZwl9VY/hRuxInp8gTjWrt1gQIB
zlgSgSKnKTN6f+e85XMPU1y/wVz8RJWl8Tr11kzo6vrM14+ruNUxo1Ea3PJ9MUcWenoRKGSU
I/IP94kZVjPkZlJv6tTF0Yi2Gclet/ZDu8vgvkxmUZYdQMGGlgZTCAsvHr37/ov6g51Tf+im
7410EsdYCmSINRGzWQAzlH9NscsW1TAd1Znog1H6NRDExY3ksjvFcKYOjUmkyWT8Vl2oJmT2
IzI23/C3esGL9OZzZ6K84MRNrH1y/yNp75vQnP3JfDMpbb5kkDp95Bu365qBluECAwEAAaOB
6DCB5TBSBggrBgEFBQcBAQRGMEQwQgYIKwYBBQUHMAGGNmh0dHA6Ly9iYXljYS5iYXlob3Rv
LmxvY2FsL2VqYmNhL3B1YmxpY3dlYi9zdGF0dXMvb2NzcDAdBgNVHQ4EFgQUzeso+31hmtp3
soKHShXXtAEo+iMwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBQt+MJyvWZi5ge71ediHix2
mqSd9zAOBgNVHQ8BAf8EBAMCBDAwEwYDVR0lBAwwCgYIKwYBBQUHAwQwHAYDVR0RBBUwE4ER
bWlrZUBiYXlwaG90by5jb20wDQYJKoZIhvcNAQELBQADggIBADnQfCasVgMsKsxIZAOZCbsU
xo9BfsbpoM02p2aP+vPNDLXYRmcnH6ReeeUKSfIn0HmS9XkeHizMEXaC5sV9g4dasdQJQOGU
mDcBnlxGn5fzNVFBM7/RHL83waYq4MCeyP9M7lSiNFZTrnSLVL9lIO0FLrBE06c9bn09kExc
zkXI6Qm+e/MNrnoC3vw3GbH3a7tZCPsQcyNSok99jgPTRb1g9uVPg25M+ScYMU0wv2BE24u1
Dfzwcq52h64TllbzdVg/qOQH1HM96wmU+CtPuzA6eYnWRao/80LfQcyhNZ/jfMB/9xwFwsam
o3Bw7SrSPEatw/tMyEEVMzas5/wZm2uMtab7642d5mr5OWLVPYgmKUscSlNt87vKkFhvn0Cz
Z7O8O79WNMJA0sx1aomn1/ZrWDkd8X/ACUC2Fa3cV4AAzmjytiNu7r2z+GwdXPmvWSlBDXKX
wLSoRkdq5hmYAP3GwXF0dsZo63WJLuCU1bPyERNLKdZM//eX832WgomPs4FA4xg0MUH0S7vJ
eo7K1cTutZEmyLT623p0GcOINs2ir/ZqPTDLKszI7ytAltYaATt4kYUXbmMGGYItDf1X/caj
DoLv2hjBTM5HORZYABC/Kfo9iL4KeYDqAvblJc7qyw+QXdHOUbwc9gQXQJvlQlfjDYvJLKme
zoZ1sMzRBOl0MIIG1zCCBL+gAwIBAgIIGE38aUOyx8EwDQYJKoZIhvcNAQELBQAwWDEcMBoG
A1UEAwwTQmF5IFBob3RvIFBlb3BsZSBDQTEWMBQGA1UECgwNQmF5IFBob3RvIExhYjETMBEG
A1UECAwKQ2FsaWZvcm5pYTELMAkGA1UEBhMCVVMwHhcNMTIxMDIzMTc1NzQ1WhcNMTQxMDIz
MTc1NzQ1WjBgMRUwEwYKCZImiZPyLGQBAQwFMTMwNjkxFTATBgNVBAMMDE1pa2UgQ2FybHNv
bjELMAkGA1UECwwCSVQxFjAUBgNVBAoMDUJheSBQaG90byBMYWIxCzAJBgNVBAYTAlVTMIIC
IjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAuECpC3YUm7GV0xz/DMmMQZ3EvMfvVhFB
77TcVxY1OoBlp2jk0ST0Hel/vp9uPhhi/eAlH89rC9fhNwORUHfeofWhoT3ZXrnjisNDQnb5
MgBV6wVM58PrikwU13FuNrIrPuUeuUE659BQhfgj2j2Pv9GVgBib6NBbHJAIOFo8H3wmd4b9
Yv6RKM145qSJCrJp96jSkoQSwni+jraHGMs55kgFP/3f0X6RpY7GhvItFI95Xauf7R5qTuW0
oamhvjFnlWVBoMuHd5yqzUgL7gQ0wfB31wfBP2ghFPAv715Qv9DYleFmgWE1LRkrt8clCzzk
huj00CrEL+rSK9bDYNpa5AdCQ8aA/bG2x24ApOBvfDYnem+Ytu9lIY7qKZiP+9pASOiXsSSt
g8CH9HoG+9GpK/rDyQ2nuNxTWEQEetwofkLdrGU/VgpcwIZqETsugu+l/FCu3Ogslezpiolb
SqjCF5CU3aZravNZo4+HDriFKm7jyKEb+zUcXcwNH2iwbGt6uTMILslXbWXuhi8Lu3SKyacj
1gn9OMPQTCUKxaiTJXqIcJP0CV9wKcgTn7vanChDgz2fTVm6HOzb2RZlyhYzU4ofuMk+VGbg
tBSDns+B9D1ACZMd58D2XdqRvmGPlnbJEZPkAZyDiR4mp/Aw09uXzOCIHzvqLhiysGon7lkV
a18CAwEAAaOCAZswggGXMFIGCCsGAQUFBwEBBEYwRDBCBggrBgEFBQcwAYY2aHR0cDovL2Jh
eWNhLmJheWhvdG8ubG9jYWwvZWpiY2EvcHVibGljd2ViL3N0YXR1cy9vY3NwMB0GA1UdDgQW
BBRGT4gPK1JkpmK4YAsQgD82sskTGDAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFC34wnK9
ZmLmB7vV52IeLHaapJ33MIGlBgNVHR8EgZ0wgZowgZeggZSggZGGgY5odHRwOi8vYmF5Y2Eu
YmF5aG90by5sb2NhbC9lamJjYS9wdWJsaWN3ZWIvd2ViZGlzdC9jZXJ0ZGlzdD9jbWQ9Y3Js
Jmlzc3Vlcj1DTj1CYXklMjBQaG90byUyMFBlb3BsZSUyMENBLE89QmF5JTIwUGhvdG8lMjBM
YWIsU1Q9Q2FsaWZvcm5pYSxDPVVTMA4GA1UdDwEB/wQEAwIGwDAdBgNVHSUEFjAUBggrBgEF
BQcDAgYIKwYBBQUHAwQwHAYDVR0RBBUwE4ERbWlrZUBiYXlwaG90by5jb20wDQYJKoZIhvcN
AQELBQADggIBAC+1BfV1bmcHZtgM4HN5rZek1vzY2UBLTHevnLDsLmNNASYMuOrm1TYMCI0/
LVkGltk0kq3lAP3hECsrSeH+SlmoRAmSCEOAwqOBU5oTqV/SMiQcZb6K2o/QUAVV6Si2KMyW
U35hnqi9/vVNq4y28RjvMAydcrJpmn5qrWsy1ZLbW+Zu3jneBYjUcm4mQnqaCyjRyE2Z3byi
/4m1sxISSdyqoRsWKsixSdb4bXUdNeNscltRYDMEDODtzYjnO7WebHtaMAEOyfE3n2gkPmGr
dBopA6/Z53GVkG+MXF1wSlc3g38dHSpbY6qJiMkl8/wKCnkcMRPC3hRGQikYprh/oAbpHAyS
q9P0jv/3cB0yzZ5b3n49PxqjwyH93lcRZJQ5pAMMWFk1B7IuYk9LVbJEVsPAmxFbWoS87Tn0
/zi6lkUNXoSgm4fJ+NBY5jnk1m48nPFIs7dpr8wW8qT2C0DiQz/tSKMramzsBduXY67djCZ5
cam1H5pRuaiEPEmrafC91i+wCsmjpMoqykKkuiGOlmY8LhRSZZ7M8qwtpo/0yur94z1Z5gyG
3yo/vZQtNDt8jaJ2ajGR50C48CtJufrckT3o3UrtjvQ3jCUnak2VbXIV6QNTio9NQKVH4+fL
VhoGvc/PC/CifBs6Qyfk3a6nHdVfzwGx8Uytj/Br0AV0p8Y2MYIERjCCBEICAQEwZDBYMRww
GgYDVQQDDBNCYXkgUGhvdG8gUGVvcGxlIENBMRYwFAYDVQQKDA1CYXkgUGhvdG8gTGFiMRMw
EQYDVQQIDApDYWxpZm9ybmlhMQswCQYDVQQGEwJVUwIIGE38aUOyx8EwCQYFKw4DAhoFAKCC
AbcwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTQwNzE1MDUx
ODE4WjAjBgkqhkiG9w0BCQQxFgQUFTVAPPRRMgjWzMfX78m/ngDu1x4wbAYJKoZIhvcNAQkP
MV8wXTALBglghkgBZQMEASowCwYJYIZIAWUDBAECMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMC
AgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIBKDBzBgkrBgEEAYI3
EAQxZjBkMFgxHDAaBgNVBAMME0JheSBQaG90byBQZW9wbGUgQ0ExFjAUBgNVBAoMDUJheSBQ
aG90byBMYWIxEzARBgNVBAgMCkNhbGlmb3JuaWExCzAJBgNVBAYTAlVTAghilJeBk5PuPTB1
BgsqhkiG9w0BCRACCzFmoGQwWDEcMBoGA1UEAwwTQmF5IFBob3RvIFBlb3BsZSBDQTEWMBQG
A1UECgwNQmF5IFBob3RvIExhYjETMBEGA1UECAwKQ2FsaWZvcm5pYTELMAkGA1UEBhMCVVMC
CGKUl4GTk+49MA0GCSqGSIb3DQEBAQUABIICABK5FuRQA2/25er0aYEkOgsKVlw9FJqQ2WsT
7N/Jox5yz5kaKH7ENwTqbXO+q42oeifmwiKsoQv+F5t2cO1bSu/zk82haZtDQ0iHIiwgvuO0
xaR11dJ4RwvWdUux8NaCHVqgpfI1K1EkWZsPXtLiNeQuN48Fu2E35TYfwy2ldjkDosdDz0ct
NF3u5EI+/Z4jRQ6rth1ljCRl+npmpsbWMSxPYR+Lc8/RPAjzmmuEgVC5ApWmoHiNqXTfGbn6
QpPgGEY5/puC/bhXIEpHSm3LD8BnI0Jasq3W3hWzH5per8s6mJcePi6bKyQVptU+zLTIc7sv
bJgQqINKfZfp9cLGDtxrw/Na1OyiZE+KUNe/h9j3zNvYDbvvzt/+mF6u85gNDl2CNX9QpWeL
rWebuMI8KF6CecERx7ZeagZFjKfaUIh/TNHsjjTyDGnmkhio5aJ1zPwenwMyWj9maeENq7iS
0mIiD5fXUSrbnWo9zSLOVXj7RLi2m74VLVKomluIrneZ4dlyCAdX4L02YLGuN4l0AMWFjTx9
Ax3jGDmrqOGUvGdMJ1Co8x26BUjW56RrQ4cG4LnaaJ/ldhMgU2Ie4imMOTuBi788XruhI9IS
6WoGIj9wkV4xiguj/A7pjkQJH5YtQwzSBhAQSZN9S/Loj4jcmyU92dJRn0I81cAtWS09pa1p
AAAAAAAA
--------------ms050603000305030104030407--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?53C4B99A.9000508>