Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 02 Jun 2014 17:37:31 -0700
From:      Mike Carlson <mike@bayphoto.com>
To:        Steven Hartland <killing@multiplay.co.uk>, freebsd-fs@freebsd.org
Subject:   Re: ZFS Kernel Panic on 10.0-RELEASE
Message-ID:  <538D18CB.5020906@bayphoto.com>
In-Reply-To: <F445995D86AA44FB8497296E0D41AC8F@multiplay.co.uk>
References:  <5388D64D.4030400@bayphoto.com> <EC2EA442-56FC-46B4-A1E2-97523029B7B3@mail.turbofuzz.com> <5388E5B4.3030002@bayphoto.com> <538BBEB7.4070008@bayphoto.com> <782C34792E95484DBA631A96FE3BEF20@multiplay.co.uk> <538C9CF3.6070208@bayphoto.com> <16ADD4D9DC73403C9669D8F34FDBD316@multiplay.co.uk> <538CB3EA.9010807@bayphoto.com> <6C6FB182781541CEBF627998B73B1DB4@multiplay.co.uk> <538CC16A.6060207@bayphoto.com> <F959477921CD4552A94BF932A55961F4@multiplay.co.uk> <538CDB7F.2060408@bayphoto.com> <88B3A7562A5F4F9B9EEF0E83BCAD2FB0@multiplay.co.uk> <538CE2B3.8090008@bayphoto.com> <85184EB23AA84607A360E601D03E1741@multiplay.co.uk> <538D0174.6000906@bayphoto.com> <F445995D86AA44FB8497296E0D41AC8F@multiplay.co.uk>

next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format.

--------------ms020807010704030109080200
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: quoted-printable

On 6/2/2014 5:29 PM, Steven Hartland wrote:
>
> ----- Original Message ----- From: "Mike Carlson" <mike@bayphoto.com>
> To: "Steven Hartland" <killing@multiplay.co.uk>; <freebsd-fs@freebsd.or=
g>
> Sent: Monday, June 02, 2014 11:57 PM
> Subject: Re: ZFS Kernel Panic on 10.0-RELEASE
>
>
>> On 6/2/2014 2:15 PM, Steven Hartland wrote:
>>> ----- Original Message ----- From: "Mike Carlson" <mike@bayphoto.com>=

>>>
>>>>> Thats the line I gathered it was on but no I need to know what the =

>>>>> value
>>>>> of vd is, so what you need to do is:
>>>>> print vd
>>>>>
>>>>> If thats valid then:
>>>>> print *vd
>>>>>
>>>> It reports:
>>>>
>>>> (kgdb) print *vd
>>>> No symbol "vd" in current context.
>>>
>>> Dam optimiser :(
>>>
>>>> Should I rebuild the kernel with additional options?
>>>
>>> Likely wont help as kernel with zero optimisations tends to fail
>>> to build in my experience :(
>>>
>>> Can you try applying the attached patch to your src e.g.
>>> cd /usr/src
>>> patch < zfs-dsize-dva-check.patch
>>>
>>> The rebuild, install the kernel and then reproduce the issue again.
>>>
>>> Hopefully it will provide some more information on the cause, but
>>> I suspect you might be seeing the effect os have some corruption.
>>
>> Well, after building the kernel with your patch, installing it and=20
>> booting off of it, the system does not panic.
>>
>> It reports this when I mount the filesystem:
>>
>>    Solaris: WARNING: dva_get_dsize_sync(): bad DVA 131241:2147483648
>>    Solaris: WARNING: dva_get_dsize_sync(): bad DVA 131241:2147483648
>>    Solaris: WARNING: dva_get_dsize_sync(): bad DVA 131241:2147483648
>>
>> Here is the results, I can now mount the file system!
>>
>>    root@working-1:~ # zfs set canmount=3Don zroot/data/working
>>    root@working-1:~ # zfs mount zroot/data/working
>>    root@working-1:~ # df
>>    Filesystem                 1K-blocks       Used Avail Capacity   =20
>> Mounted on
>>    zroot                     2677363378    1207060 2676156318    =20
>> 0%    /
>>    devfs                              1          1 0   100% /dev
>>    /dev/mfid10p1              253911544    2827824 230770800 1%     =20
>> /dump
>>    zroot/home                2676156506        188 2676156318    =20
>> 0%      /home
>>    zroot/data                2676156389         71 2676156318    =20
>> 0%      /mnt/data
>>    zroot/usr/ports/distfiles 2676246609      90291 2676156318    =20
>> 0%      /mnt/usr/ports/distfiles
>>    zroot/usr/ports/packages  2676158702       2384 2676156318    =20
>> 0%      /mnt/usr/ports/packages
>>    zroot/tmp                 2676156812        493 2676156318    =20
>> 0%      /tmp
>>    zroot/usr                 2679746045    3589727 2676156318    =20
>> 0%      /usr
>>    zroot/usr/ports           2676986896     830578 2676156318    =20
>> 0%      /usr/ports
>>    zroot/usr/src             2676643553     487234 2676156318    =20
>> 0%      /usr/src
>>    zroot/var                 2676650671     494353 2676156318    =20
>> 0%      /var
>>    zroot/var/crash           2676156388         69 2676156318    =20
>> 0%      /var/crash
>>    zroot/var/db              2677521200    1364882 2676156318    =20
>> 0%      /var/db
>>    zroot/var/db/pkg          2676198058      41740 2676156318    =20
>> 0%      /var/db/pkg
>>    zroot/var/empty           2676156387         68 2676156318    =20
>> 0%      /var/empty
>>    zroot/var/log             2676168522      12203 2676156318    =20
>> 0%      /var/log
>>    zroot/var/mail            2676157043        725 2676156318    =20
>> 0%      /var/mail
>>    zroot/var/run             2676156508        190 2676156318    =20
>> 0%      /var/run
>>    zroot/var/tmp             2676156389         71 2676156318    =20
>> 0%      /var/tmp
>>    zroot/data/working        7664687468 4988531149 2676156318 65%  =20
>>    /mnt/data/working
>>    root@working-1:~ # ls /mnt/data/working/
>>    DONE_ORDERS             DP2_CMD NEW_MULTI_TESTING PROCESS
>>    RECYCLER                XML_NOTIFICATIONS       XML_REPORTS
>
> That does indeed seem to indicated some on disk corruption.
>
> There are a number of cases in the code which have a similar check but
> I'm afraid I don't know the implications of the corruption your
> seeing but others may.
>
> The attached updated patch will enforce the safe panic in this case
> unless the sysctl vfs.zfs.recover is set to 1 (which can also now be
> done on  the fly).
>
> I'd recommend backing up the data off the pool and restoring it else
> where.
>
> It would be interesting to see the output of the following command
> on your pool:
> zdb -uuumdC <pool>
>
>    Regards
>    Steve
I'm applying that patch and rebuilding the kernel again

Here is the output from zdb -uuumdC:

    zroot:
         version: 28
         name: 'zroot'
         state: 0
         txg: 13
         pool_guid: 9132288035431788388
         hostname: 'amnesia.discdrive.bayphoto.com'
         vdev_children: 1
         vdev_tree:
             type: 'root'
             id: 0
             guid: 9132288035431788388
             children[0]:
                 type: 'raidz'
                 id: 0
                 guid: 15520162542638044402
                 nparity: 2
                 metaslab_array: 31
                 metaslab_shift: 36
                 ashift: 9
                 asize: 9894744555520
                 is_log: 0
                 create_txg: 4
                 children[0]:
                     type: 'disk'
                     id: 0
                     guid: 4289437176706222104
                     path: '/dev/gpt/disk0'
                     phys_path: '/dev/gpt/disk0'
                     whole_disk: 1
                     create_txg: 4
                 children[1]:
                     type: 'disk'
                     id: 1
                     guid: 5369387862706621015
                     path: '/dev/gpt/disk1'
                     phys_path: '/dev/gpt/disk1'
                     whole_disk: 1
                     create_txg: 4
                 children[2]:
                     type: 'disk'
                     id: 2
                     guid: 456749962069636782
                     path: '/dev/gpt/disk2'
                     phys_path: '/dev/gpt/disk2'
                     whole_disk: 1
                     create_txg: 4
                 children[3]:
                     type: 'disk'
                     id: 3
                     guid: 3809413300177228462
                     path: '/dev/gpt/disk3'
                     phys_path: '/dev/gpt/disk3'
                     whole_disk: 1
                     create_txg: 4
                 children[4]:
                     type: 'disk'
                     id: 4
                     guid: 4978694931676882497
                     path: '/dev/gpt/disk4'
                     phys_path: '/dev/gpt/disk4'
                     whole_disk: 1
                     create_txg: 4
                 children[5]:
                     type: 'disk'
                     id: 5
                     guid: 17831739822150458220
                     path: '/dev/gpt/disk5'
                     phys_path: '/dev/gpt/disk5'
                     whole_disk: 1
                     create_txg: 4
                 children[6]:
                     type: 'disk'
                     id: 6
                     guid: 1286918567594965543
                     path: '/dev/gpt/disk6'
                     phys_path: '/dev/gpt/disk6'
                     whole_disk: 1
                     create_txg: 4
                 children[7]:
                     type: 'disk'
                     id: 7
                     guid: 7958718879588658810
                     path: '/dev/gpt/disk7'
                     phys_path: '/dev/gpt/disk7'
                     whole_disk: 1
                     create_txg: 4
                 children[8]:
                     type: 'disk'
                     id: 8
                     guid: 18392960683862755998
                     path: '/dev/gpt/disk8'
                     phys_path: '/dev/gpt/disk8'
                     whole_disk: 1
                     create_txg: 4
                 children[9]:
                     type: 'disk'
                     id: 9
                     guid: 13046629036569375198
                     path: '/dev/gpt/disk9'
                     phys_path: '/dev/gpt/disk9'
                     whole_disk: 1
                     create_txg: 4
                 children[10]:
                     type: 'disk'
                     id: 10
                     guid: 10604061156531251346
                     path: '/dev/gpt/disk11'
                     phys_path: '/dev/gpt/disk11'
                     whole_disk: 1
                     create_txg: 4

I find it strange that it says version 28 when it was upgraded to=20
version 5000

--------------ms020807010704030109080200
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIITIjCC
BhwwggQEoAMCAQICCGWP9DPVdj3ZMA0GCSqGSIb3DQEBCwUAMEsxDzANBgNVBAMMBlJvb3RD
QTEWMBQGA1UECgwNQmF5IFBob3RvIExhYjETMBEGA1UECAwKQ2FsaWZvcm5pYTELMAkGA1UE
BhMCVVMwHhcNMTIxMDIzMTczMjE4WhcNMjcxMDIzMTczMjE4WjBYMRwwGgYDVQQDDBNCYXkg
UGhvdG8gUGVvcGxlIENBMRYwFAYDVQQKDA1CYXkgUGhvdG8gTGFiMRMwEQYDVQQIDApDYWxp
Zm9ybmlhMQswCQYDVQQGEwJVUzCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBALuW
vZE7FFSO77jEhdT/dZzu0XnPS35aepuvMvCNTagnNMbaf+AKrLZFaVRqBt4f6yl5TDXNEAMI
+CJrg7T9dsoGN1Vygm4bIFzOIRtTZ1A7seemeonyqMto8gYMPhEFicuKavEAFcCFprVcVu5v
yXbEG+tYkx08oUzeZ9LWZnh4a0wxAucbH0PcZFmdXFMbgjt6GSiR8jVUT1t/Dik1+mJ1A4K3
3VyM+G1CaqGIKg2UbvHiVQVoJrpgUc6vO5aWWgDreH+CAKLpyJwCj4BGphfHBPENu5LUp0CL
fa4pA4r+OH0HNBp/I7utZMrIevh3pya34lDJXj0CQWSuVBR9Kp80IHHYGuyPU/sMXkUplsyI
Y4KgQSRYRFNdWoboLwsRu1/2iDUaTR+XYH7Tu5zvCOedUm8ns/wfE7TW6kax5nR36VzuZS6B
kUcuETO1QMxt2SJcHZIftiyTuxC8Y9Z7J6igR39pZPWGywJ2+oPRKFQD/u9RWZqjh9k5euVi
4bVwwog5DqnsymMTuKb0I1nCs5js+rfEVo+mc4gAfEmzk/dmpwkOnGV3oTfASbGtwMIlAEe5
HHJhdTA3jpVo3eQAZi479xZ7SozKvnjt1y9SMQ4uTJXjER/IVH3YC9S/BSFrqWLLDfWEGW84
SBn+ogldDIy3PX3SU7nOiN4Vse2Jg+r7AgMBAAGjgfYwgfMwHQYDVR0OBBYEFC34wnK9ZmLm
B7vV52IeLHaapJ33MA8GA1UdEwEB/wQFMAMBAf8wHwYDVR0jBBgwFoAU00bJTnGPaRmhJAUZ
t4x4J3soVyswgY8GA1UdHwSBhzCBhDCBgaB/oH2Ge2h0dHA6Ly9iYXljYS5iYXlob3RvLmxv
Y2FsL2VqYmNhL3B1YmxpY3dlYi93ZWJkaXN0L2NlcnRkaXN0P2NtZD1jcmwmaXNzdWVyPUNO
PVJvb3RDQSxPPUJheSUyMFBob3RvJTIwTGFiLFNUPUNhbGlmb3JuaWEsQz1VUzAOBgNVHQ8B
Af8EBAMCAYYwDQYJKoZIhvcNAQELBQADggIBAEoDTeb1wVXz2VqFPrc3uaRnbVt6DCB9uBft
L8YRgi5+FF5KHjvTqYHGiS38yVFfXM6MaN6yI7WSwtS+wVhM7Tdw6miwKNBApmCDKzjorNVX
iZUmv88fjGm652LeIfT/oFGJg2oruoad9OfIofjT/jFpiFT7KOgjg4xeKCC0f+9naVo5uWOJ
HTyFm1Kq6ryT+g7mZb7kLvndmJRW0aybn8Dls7/bhThEaOYkwn4dbZ6q0W2I5ZWg9bbh66V+
J9P1XObZA/TkRVTlpSYJ1hAgYX1yTbL360vuTAAwdSWQSFm4TAoXC4BsPdlgy86lDjNrW9id
WX0RJCBzk/FzHhI4Aj9+SVhLl/Vkf9nbz8VhPPK2Az0PPYKy5ARtgOqGTKZbgVKut9Kgc/fn
vUgPhObHQsD1Ui+NKsIGYBFKBhNmVXqMQSn4JzC9x0oDEmv37UGrcut2cCP3ZS17p137VaUM
lQ0RWomju+sPCPFgyCPa/TLPoMZ2334uIxkRbDefvOoXIosORMQ9Jh50XqktUesbhuBfH9Q5
8h8bTWm1Cn/LxXW9qdSSbnta0OAH1G4hwVUlcusSM0o7Ude8tszw6kRpmEDDE8BQjE5nXSY7
wOf1eXfQfDkVQiouTD3l5ElqB98tnCnL/y9dphWoBLmiJwgb/4yWZ/Zewc1V65UFr7LmvcQM
MIIGIzCCBAugAwIBAgIIYpSXgZOT7j0wDQYJKoZIhvcNAQELBQAwWDEcMBoGA1UEAwwTQmF5
IFBob3RvIFBlb3BsZSBDQTEWMBQGA1UECgwNQmF5IFBob3RvIExhYjETMBEGA1UECAwKQ2Fs
aWZvcm5pYTELMAkGA1UEBhMCVVMwHhcNMTIxMDIzMTgwMDAzWhcNMTQxMDIzMTgwMDAzWjBg
MRUwEwYKCZImiZPyLGQBAQwFMTMwNjkxFTATBgNVBAMMDE1pa2UgQ2FybHNvbjELMAkGA1UE
CwwCSVQxFjAUBgNVBAoMDUJheSBQaG90byBMYWIxCzAJBgNVBAYTAlVTMIICIjANBgkqhkiG
9w0BAQEFAAOCAg8AMIICCgKCAgEAoTzIvF66A3wYPWXQFzmjBUuIrHSQGhOcE7rb0KQtar+H
rkmHX56bCAklW5P/pd+yJ08lMwb3CxbTOz0q47JuBv2kKJO1xCgCua26Uvz3VAmfirmWwpXq
zZBDqy/bEIt/XFfiVUC4jriGSEPrtx9q9nJJsb2JVRgtsbcHaaJFu8u8s8p8cLbcYdKobS00
g6+7it2IpIJhxc5tEMa1Yku3kCQiHVVFa9b4H5pFDHpkCrKZ43cuCneiR5kgr47z/3U66kLt
J7Q5IT/i7nThjGQMa/f1JSWet8yeTomKvqkuEAA4o/IWQzEbtxzeps6vWxaCDULjEq69s//S
6PtqiQSmG9ZGFoPYD8/GGd4CMBqgjKopintD5sGTlJ851yZwl9VY/hRuxInp8gTjWrt1gQIB
zlgSgSKnKTN6f+e85XMPU1y/wVz8RJWl8Tr11kzo6vrM14+ruNUxo1Ea3PJ9MUcWenoRKGSU
I/IP94kZVjPkZlJv6tTF0Yi2Gclet/ZDu8vgvkxmUZYdQMGGlgZTCAsvHr37/ov6g51Tf+im
7410EsdYCmSINRGzWQAzlH9NscsW1TAd1Znog1H6NRDExY3ksjvFcKYOjUmkyWT8Vl2oJmT2
IzI23/C3esGL9OZzZ6K84MRNrH1y/yNp75vQnP3JfDMpbb5kkDp95Bu365qBluECAwEAAaOB
6DCB5TBSBggrBgEFBQcBAQRGMEQwQgYIKwYBBQUHMAGGNmh0dHA6Ly9iYXljYS5iYXlob3Rv
LmxvY2FsL2VqYmNhL3B1YmxpY3dlYi9zdGF0dXMvb2NzcDAdBgNVHQ4EFgQUzeso+31hmtp3
soKHShXXtAEo+iMwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBQt+MJyvWZi5ge71ediHix2
mqSd9zAOBgNVHQ8BAf8EBAMCBDAwEwYDVR0lBAwwCgYIKwYBBQUHAwQwHAYDVR0RBBUwE4ER
bWlrZUBiYXlwaG90by5jb20wDQYJKoZIhvcNAQELBQADggIBADnQfCasVgMsKsxIZAOZCbsU
xo9BfsbpoM02p2aP+vPNDLXYRmcnH6ReeeUKSfIn0HmS9XkeHizMEXaC5sV9g4dasdQJQOGU
mDcBnlxGn5fzNVFBM7/RHL83waYq4MCeyP9M7lSiNFZTrnSLVL9lIO0FLrBE06c9bn09kExc
zkXI6Qm+e/MNrnoC3vw3GbH3a7tZCPsQcyNSok99jgPTRb1g9uVPg25M+ScYMU0wv2BE24u1
Dfzwcq52h64TllbzdVg/qOQH1HM96wmU+CtPuzA6eYnWRao/80LfQcyhNZ/jfMB/9xwFwsam
o3Bw7SrSPEatw/tMyEEVMzas5/wZm2uMtab7642d5mr5OWLVPYgmKUscSlNt87vKkFhvn0Cz
Z7O8O79WNMJA0sx1aomn1/ZrWDkd8X/ACUC2Fa3cV4AAzmjytiNu7r2z+GwdXPmvWSlBDXKX
wLSoRkdq5hmYAP3GwXF0dsZo63WJLuCU1bPyERNLKdZM//eX832WgomPs4FA4xg0MUH0S7vJ
eo7K1cTutZEmyLT623p0GcOINs2ir/ZqPTDLKszI7ytAltYaATt4kYUXbmMGGYItDf1X/caj
DoLv2hjBTM5HORZYABC/Kfo9iL4KeYDqAvblJc7qyw+QXdHOUbwc9gQXQJvlQlfjDYvJLKme
zoZ1sMzRBOl0MIIG1zCCBL+gAwIBAgIIGE38aUOyx8EwDQYJKoZIhvcNAQELBQAwWDEcMBoG
A1UEAwwTQmF5IFBob3RvIFBlb3BsZSBDQTEWMBQGA1UECgwNQmF5IFBob3RvIExhYjETMBEG
A1UECAwKQ2FsaWZvcm5pYTELMAkGA1UEBhMCVVMwHhcNMTIxMDIzMTc1NzQ1WhcNMTQxMDIz
MTc1NzQ1WjBgMRUwEwYKCZImiZPyLGQBAQwFMTMwNjkxFTATBgNVBAMMDE1pa2UgQ2FybHNv
bjELMAkGA1UECwwCSVQxFjAUBgNVBAoMDUJheSBQaG90byBMYWIxCzAJBgNVBAYTAlVTMIIC
IjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAuECpC3YUm7GV0xz/DMmMQZ3EvMfvVhFB
77TcVxY1OoBlp2jk0ST0Hel/vp9uPhhi/eAlH89rC9fhNwORUHfeofWhoT3ZXrnjisNDQnb5
MgBV6wVM58PrikwU13FuNrIrPuUeuUE659BQhfgj2j2Pv9GVgBib6NBbHJAIOFo8H3wmd4b9
Yv6RKM145qSJCrJp96jSkoQSwni+jraHGMs55kgFP/3f0X6RpY7GhvItFI95Xauf7R5qTuW0
oamhvjFnlWVBoMuHd5yqzUgL7gQ0wfB31wfBP2ghFPAv715Qv9DYleFmgWE1LRkrt8clCzzk
huj00CrEL+rSK9bDYNpa5AdCQ8aA/bG2x24ApOBvfDYnem+Ytu9lIY7qKZiP+9pASOiXsSSt
g8CH9HoG+9GpK/rDyQ2nuNxTWEQEetwofkLdrGU/VgpcwIZqETsugu+l/FCu3Ogslezpiolb
SqjCF5CU3aZravNZo4+HDriFKm7jyKEb+zUcXcwNH2iwbGt6uTMILslXbWXuhi8Lu3SKyacj
1gn9OMPQTCUKxaiTJXqIcJP0CV9wKcgTn7vanChDgz2fTVm6HOzb2RZlyhYzU4ofuMk+VGbg
tBSDns+B9D1ACZMd58D2XdqRvmGPlnbJEZPkAZyDiR4mp/Aw09uXzOCIHzvqLhiysGon7lkV
a18CAwEAAaOCAZswggGXMFIGCCsGAQUFBwEBBEYwRDBCBggrBgEFBQcwAYY2aHR0cDovL2Jh
eWNhLmJheWhvdG8ubG9jYWwvZWpiY2EvcHVibGljd2ViL3N0YXR1cy9vY3NwMB0GA1UdDgQW
BBRGT4gPK1JkpmK4YAsQgD82sskTGDAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFC34wnK9
ZmLmB7vV52IeLHaapJ33MIGlBgNVHR8EgZ0wgZowgZeggZSggZGGgY5odHRwOi8vYmF5Y2Eu
YmF5aG90by5sb2NhbC9lamJjYS9wdWJsaWN3ZWIvd2ViZGlzdC9jZXJ0ZGlzdD9jbWQ9Y3Js
Jmlzc3Vlcj1DTj1CYXklMjBQaG90byUyMFBlb3BsZSUyMENBLE89QmF5JTIwUGhvdG8lMjBM
YWIsU1Q9Q2FsaWZvcm5pYSxDPVVTMA4GA1UdDwEB/wQEAwIGwDAdBgNVHSUEFjAUBggrBgEF
BQcDAgYIKwYBBQUHAwQwHAYDVR0RBBUwE4ERbWlrZUBiYXlwaG90by5jb20wDQYJKoZIhvcN
AQELBQADggIBAC+1BfV1bmcHZtgM4HN5rZek1vzY2UBLTHevnLDsLmNNASYMuOrm1TYMCI0/
LVkGltk0kq3lAP3hECsrSeH+SlmoRAmSCEOAwqOBU5oTqV/SMiQcZb6K2o/QUAVV6Si2KMyW
U35hnqi9/vVNq4y28RjvMAydcrJpmn5qrWsy1ZLbW+Zu3jneBYjUcm4mQnqaCyjRyE2Z3byi
/4m1sxISSdyqoRsWKsixSdb4bXUdNeNscltRYDMEDODtzYjnO7WebHtaMAEOyfE3n2gkPmGr
dBopA6/Z53GVkG+MXF1wSlc3g38dHSpbY6qJiMkl8/wKCnkcMRPC3hRGQikYprh/oAbpHAyS
q9P0jv/3cB0yzZ5b3n49PxqjwyH93lcRZJQ5pAMMWFk1B7IuYk9LVbJEVsPAmxFbWoS87Tn0
/zi6lkUNXoSgm4fJ+NBY5jnk1m48nPFIs7dpr8wW8qT2C0DiQz/tSKMramzsBduXY67djCZ5
cam1H5pRuaiEPEmrafC91i+wCsmjpMoqykKkuiGOlmY8LhRSZZ7M8qwtpo/0yur94z1Z5gyG
3yo/vZQtNDt8jaJ2ajGR50C48CtJufrckT3o3UrtjvQ3jCUnak2VbXIV6QNTio9NQKVH4+fL
VhoGvc/PC/CifBs6Qyfk3a6nHdVfzwGx8Uytj/Br0AV0p8Y2MYIERjCCBEICAQEwZDBYMRww
GgYDVQQDDBNCYXkgUGhvdG8gUGVvcGxlIENBMRYwFAYDVQQKDA1CYXkgUGhvdG8gTGFiMRMw
EQYDVQQIDApDYWxpZm9ybmlhMQswCQYDVQQGEwJVUwIIGE38aUOyx8EwCQYFKw4DAhoFAKCC
AbcwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTQwNjAzMDAz
NzMxWjAjBgkqhkiG9w0BCQQxFgQUcpWCecZx5Ldqot8lVfmaEBIc7kcwbAYJKoZIhvcNAQkP
MV8wXTALBglghkgBZQMEASowCwYJYIZIAWUDBAECMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMC
AgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIBKDBzBgkrBgEEAYI3
EAQxZjBkMFgxHDAaBgNVBAMME0JheSBQaG90byBQZW9wbGUgQ0ExFjAUBgNVBAoMDUJheSBQ
aG90byBMYWIxEzARBgNVBAgMCkNhbGlmb3JuaWExCzAJBgNVBAYTAlVTAghilJeBk5PuPTB1
BgsqhkiG9w0BCRACCzFmoGQwWDEcMBoGA1UEAwwTQmF5IFBob3RvIFBlb3BsZSBDQTEWMBQG
A1UECgwNQmF5IFBob3RvIExhYjETMBEGA1UECAwKQ2FsaWZvcm5pYTELMAkGA1UEBhMCVVMC
CGKUl4GTk+49MA0GCSqGSIb3DQEBAQUABIICAB/JLRR8hug44t8lxVsEfjPguhpAKb7SoNVE
9EjdcreJFQLdJFvWAkehi1pcRkinC6DyZxgb5e3bP79i9bAJTZMjTJLvRMNdaXdxbz+j0OHO
kuUCfgsEmxtqjT+OgLVcM9UY72B7BAdfI3oANDmj26KBgSHocawC/A6DP4lNvH2z2SPiKXZP
+yElIeTKQQnr8YQjkQV/M+/9F7LMxoy5SQ+U6kcmjEA2h1J4ymp3oZSCxYXkNhdTh1GctBiS
Ij1GcnuAkmCXICMTkPqVcI+0I8NaePsF4xsHTfZI9MQYtO/ATc0ppfigm8sE30THKeCJnZKg
WBuDhrDLlx3EeY9odYojge589/OmYdJ3MyjqPcIJevX/mII9k/QSkhmynC7PIpzeSoFrPmg+
qH28IWjmWpO3zPMhCptaYhfW8iUSbAZAEVuYKRA8Tkzk0i59jNvn2JqorUcCmThBGSkXrcEl
rVPdq1vXe8uk6fg/tzToSS4TNrkCiOFUVPfr9SNBbIXUjCT4EHSeu2tAwsYHrG08py/KqjZR
DsrXjrE37Hfn8K0IRz4RRc1KbeDQCngCoJqQ8nLyCTux2r3jhoVOPY+golZa/byW1o0jnVIy
zTqUcpdlA9CX/gN2azB4XCt/mUHh89Es+mzjAvFyISXDm3w47YPJRN7Egg0Z9zWM/3d7ENcA
AAAAAAAA
--------------ms020807010704030109080200--



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?538D18CB.5020906>