Date: Wed, 06 Aug 2008 12:11:05 +0200 From: Sebastiaan van Erk <sebster@sebster.com> To: Jeremy Chadwick <koitsu@FreeBSD.org> Cc: pyunyh@gmail.com, "Andrey V. Elsukov" <bu7cher@yandex.ru>, freebsd-stable@freebsd.org Subject: Re: Stable SATA pci card for FreeBSD 6.x/7.0 Message-ID: <489978B9.5080405@sebster.com> In-Reply-To: <20080806095748.GA52551@eos.sc1.parodius.com> References: <48982B58.4000406@sebster.com> <48992532.9080503@yandex.ru> <489970CC.4000103@sebster.com> <20080806095748.GA52551@eos.sc1.parodius.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format.
--------------ms040501060706050302050501
Content-Type: multipart/mixed; boundary="------------040602010206030305030407"
This is a multi-part message in MIME format.
--------------040602010206030305030407
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Hi,
Ok, those rl1: watchdog timeouts didn't ring a bell with me because I'd
seen them before; however a quick grep in the logs (which date back to
May 25) show no other watchdog timeout matches.
To try and avoid being incomplete again, I'll just attach the full dmesg
below.
Jeremy Chadwick wrote:
> On Wed, Aug 06, 2008 at 11:37:16AM +0200, Sebastiaan van Erk wrote:
>> Yes, good thing you pointed this out, I hadn't seen those yet:
>>
>> Aug 5 11:15:05 piglet kernel: rl1: watchdog timeout
>> Aug 5 11:15:05 piglet kernel: ad6: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=218885455
>> Aug 5 11:15:05 piglet kernel: ad4: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=218885455
>> Aug 5 11:15:10 piglet kernel: rl1: watchdog timeout
>> Aug 5 11:15:31 piglet kernel: rl1: watchdog timeout
>> Aug 5 11:15:31 piglet kernel: ad6: FAILURE - device detached
>> Aug 5 11:15:31 piglet kernel: subdisk6: detached
>> Aug 5 11:15:31 piglet kernel: ad6: detached
>> Aug 5 11:15:31 piglet kernel: rl1: watchdog timeout
>> Aug 5 11:15:31 piglet kernel: rl1: watchdog timeout
>> Aug 5 11:15:31 piglet kernel: ad4: FAILURE - device detached
>> Aug 5 11:15:31 piglet kernel: subdisk4: detached
>> Aug 5 11:15:31 piglet kernel: ad4: detached
>> Aug 5 11:15:31 piglet kernel: GEOM_MIRROR: Device gm1: provider ad6 disconnected.
>> Aug 5 11:15:31 piglet kernel: GEOM_MIRROR: Device gm1: provider ad4 disconnected.
>> Aug 5 11:15:31 piglet kernel: GEOM_MIRROR: Device gm1: provider mirror/gm1 destroyed.
>> Aug 5 11:15:31 piglet kernel: GEOM_MIRROR: Device gm1 destroyed.
>> Aug 5 11:15:31 piglet kernel: g_vfs_done():mirror/gm1s1e[WRITE(offset=111376236544, length=16384)] error = 6
>
> Kudos to Andrey for asking a simple yet incredibly benefitial question.
>
> You have a much greater problem here, and it doesn't look specific to
> your disks. It looks as if an interrupt is stalled or locked. I'm
> willing to bet your rl1 Realtek NIC and your ATA controller (associated
> with disks ad4 and ad6) use the same IRQ. vmstat -i output should help
> clear that up, or dmesg output.
>
> I'll tell you that there have been some watchdog timeout fixes committed
> to rl(4) in recent months, depending upon what specific model and
> revision of Realtek NIC you have. No offence intended, but Realtek is
> definitely the worst of the bunch. I'm willing to bet it's an on-board
> NIC too. :-)
Actually, I have 3 NICs in my PC (all of them in use). My machine is the
server/router in my home network, so it has the onboard vr0 NIC
connected to my ADSL modem, the rl0 nic connected to my internal wired
lan, and the rl1 nic connected to my wireless router (my internal wired
lan is firewalled from the wireless, since I don't really trust wireless
security ;-)).
> I'm CC'ing PYUN Yong-Hyeon here, as he presently maintains/works on the
> rl(4) driver, and might be able to help determine if the Realtek NIC is
> what's causing all of this, or if the ATA chipset (is this the VIA? We
> don't know yet) is causing it first.
>
> Finally, what motherboard brand and model is this, and what BIOS
> revision or version?
I attached the output of dmidecode (and dmesg), hopefully that contains
all you need to know.
BTW: I did a reply all, but I'm not sure if that is the "right" policy
here. If I'm bothering anybody with this and they prefer to only see the
mail on the list, then please let me know!
Regards and thanks for all the help,
Sebastiaan
--------------040602010206030305030407
Content-Type: text/plain;
name="dmesg.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="dmesg.txt"
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.3-PRERELEASE #20: Wed Jan 2 19:48:49 CET 2008
root@piglet.sebster.com:/usr/obj/usr/src/sys/PIGLET
MPTable: <OEM00000 PROD00000000>
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Sempron(tm) Processor 2600+ (1599.83-MHz 686-class CPU)
Origin = "AuthenticAMD" Id = 0x20fc2 Stepping = 2
Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
Features2=0x1<SSE3>
AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow!+,3DNow!>
AMD Features2=0x1<LAHF>
real memory = 1056964608 (1008 MB)
avail memory = 1020919808 (973 MB)
ioapic0: Assuming intbase of 0
ioapic0 <Version 0.3> irqs 0-23 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
hptrr: HPT RocketRAID controller driver v1.1 (Jan 2 2008 19:48:29)
cpu0 on motherboard
pcib0: <MPTable Host-PCI bridge> pcibus 0 on motherboard
pci0: <PCI bus> on pcib0
agp0: <VIA 8380 host to PCI bridge> mem 0xe8000000-0xefffffff at device 0.0 on pci0
pcib1: <MPTable PCI-PCI bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
pci1: <display, VGA> at device 0.0 (no driver attached)
rl0: <RealTek 8139 10/100BaseTX> port 0xd000-0xd0ff mem 0xf6084000-0xf60840ff irq 16 at device 8.0 on pci0
miibus0: <MII bus> on rl0
rlphy0: <RealTek internal media interface> on miibus0
rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
rl0: Ethernet address: 00:50:fc:57:a2:4b
rl1: <RealTek 8139 10/100BaseTX> port 0xd100-0xd1ff mem 0xf6080000-0xf60800ff irq 17 at device 9.0 on pci0
miibus1: <MII bus> on rl1
rlphy1: <RealTek internal media interface> on miibus1
rlphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
rl1: Ethernet address: 00:50:fc:44:23:0e
atapci0: <SiI SiI 3512 SATA150 controller> port 0xd200-0xd207,0xd300-0xd303,0xd400-0xd407,0xd500-0xd503,0xd600-0xd60f mem 0xf6081000-0xf60811ff irq 18 at device 10.0 on pci0
ata2: <ATA channel 0> on atapci0
ata3: <ATA channel 1> on atapci0
atapci1: <VIA 6420 SATA150 controller> port 0xd700-0xd707,0xd800-0xd803,0xd900-0xd907,0xda00-0xda03,0xdb00-0xdb0f,0xdc00-0xdcff irq 20 at device 15.0 on pci0
ata4: <ATA channel 0> on atapci1
ata5: <ATA channel 1> on atapci1
atapci2: <VIA 8237 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xdd00-0xdd0f at device 15.1 on pci0
ata0: <ATA channel 0> on atapci2
ata1: <ATA channel 1> on atapci2
uhci0: <VIA 83C572 USB controller> port 0xde00-0xde1f irq 21 at device 16.0 on pci0
uhci0: [GIANT-LOCKED]
usb0: <VIA 83C572 USB controller> on uhci0
usb0: USB revision 1.0
uhub0: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <VIA 83C572 USB controller> port 0xdf00-0xdf1f irq 21 at device 16.1 on pci0
uhci1: [GIANT-LOCKED]
usb1: <VIA 83C572 USB controller> on uhci1
usb1: USB revision 1.0
uhub1: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2: <VIA 83C572 USB controller> port 0xe000-0xe01f irq 21 at device 16.2 on pci0
uhci2: [GIANT-LOCKED]
usb2: <VIA 83C572 USB controller> on uhci2
usb2: USB revision 1.0
uhub2: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
uhci3: <VIA 83C572 USB controller> port 0xe100-0xe11f irq 21 at device 16.3 on pci0
uhci3: [GIANT-LOCKED]
usb3: <VIA 83C572 USB controller> on uhci3
usb3: USB revision 1.0
uhub3: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub3: 2 ports with 2 removable, self powered
ehci0: <VIA VT6202 USB 2.0 controller> mem 0xf6082000-0xf60820ff irq 21 at device 16.4 on pci0
ehci0: [GIANT-LOCKED]
usb4: EHCI version 1.0
usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
usb4: <VIA VT6202 USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: VIA EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub4: 8 ports with 8 removable, self powered
isab0: <PCI-ISA bridge> at device 17.0 on pci0
isa0: <ISA bus> on isab0
pcm0: <VIA VT8237> port 0xe200-0xe2ff irq 22 at device 17.5 on pci0
pcm0: <VIA Technologies VIA1617A AC97 Codec>
pcm0: <VIA DXS Enabled: DXS 4 / SGD 0 / REC 1>
vr0: <VIA VT6102 Rhine II 10/100BaseTX> port 0xe300-0xe3ff mem 0xf6083000-0xf60830ff irq 23 at device 18.0 on pci0
vr0: Quirks: 0x0
miibus2: <MII bus> on vr0
ukphy0: <Generic IEEE 802.3u media interface> on miibus2
ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
vr0: Ethernet address: 00:13:d3:16:75:97
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xcc000-0xd3fff,0xd4000-0xd87ff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
fdc0: <Enhanced floppy controller> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: [FAST]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
unknown: <PNP0303> can't assign resources (port)
unknown: <PNP0c01> can't assign resources (memory)
unknown: <PNP0c02> can't assign resources (memory)
unknown: <PNP0501> can't assign resources (port)
unknown: <PNP0700> can't assign resources (port)
unknown: <PNP0400> can't assign resources (port)
unknown: <PNP0501> can't assign resources (port)
Timecounter "TSC" frequency 1599832423 Hz quality 800
Timecounters tick every 1.000 msec
hptrr: no controller detected.
ad0: 286188MB <Maxtor 6L300R0 BAH41G10> at ata0-master UDMA133
ad1: 239372MB <Maxtor 6L250R0 BAH41G10> at ata0-slave UDMA133
acd0: DMA limited to UDMA33, device found non-ATA66 cable
acd0: DVDR <LITE-ON DVDRW SHW-1635S/YS0N> at ata1-master UDMA33
ad4: 953869MB <SAMSUNG HD103UJ 1AA01112> at ata2-master SATA150
ad6: 953869MB <SAMSUNG HD103UJ 1AA01112> at ata3-master SATA150
ad8: 239372MB <Maxtor 6L250S0 BANC1G10> at ata4-master SATA150
ad10: 239372MB <Maxtor 6L250S0 BANC1G10> at ata5-master SATA150
GEOM_MIRROR: Device gm1 created (id=560014299).
GEOM_MIRROR: Device gm1: provider ad4 detected.
GEOM_MIRROR: Device gm1: provider ad6 detected.
GEOM_MIRROR: Device gm1: provider ad6 activated.
GEOM_MIRROR: Device gm1: provider mirror/gm1 launched.
GEOM_MIRROR: Device gm1: rebuilding provider ad4.
GEOM_MIRROR: Device gm0 created (id=1464068017).
GEOM_MIRROR: Device gm0: provider ad8 detected.
GEOM_MIRROR: Device gm0: provider ad10 detected.
GEOM_MIRROR: Device gm0: provider ad10 activated.
GEOM_MIRROR: Device gm0: provider ad8 activated.
GEOM_MIRROR: Device gm0: provider mirror/gm0 launched.
Trying to mount root from ufs:/dev/ad0s1a
WARNING: / was not properly dismounted
WARNING: /tmp was not properly dismounted
WARNING: /usr was not properly dismounted
WARNING: /var was not properly dismounted
/var: mount pending error: blocks 4 files 1
WARNING: /shared was not properly dismounted
WARNING: /mirror was not properly dismounted
WARNING: R/W mount of /data denied. Filesystem is not clean - run fsck
rl0: link state changed to UP
rl1: link state changed to UP
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
WARNING: R/W mount of /data denied. Filesystem is not clean - run fsck
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
ukbd0: Logitech HID compliant keyboard, rev 1.10/1.80, addr 2, iclass 3/1
kbd2 at ukbd0
uhid0: Logitech HID compliant keyboard, rev 1.10/1.80, addr 2, iclass 3/1
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
ad4: FAILURE - SMART status=51<READY,DSC,ERROR> error=4<ABORTED>
ad6: FAILURE - SMART status=51<READY,DSC,ERROR> error=4<ABORTED>
ad4: FAILURE - SMART status=51<READY,DSC,ERROR> error=4<ABORTED>
ad4: FAILURE - SMART status=51<READY,DSC,ERROR> error=4<ABORTED>
ad6: FAILURE - SMART status=51<READY,DSC,ERROR> error=4<ABORTED>
GEOM_MIRROR: Device gm1: rebuilding provider ad4 finished.
GEOM_MIRROR: Device gm1: provider ad4 activated.
vr0: link state changed to DOWN
vr0: link state changed to UP
vr0: link state changed to DOWN
vr0: link state changed to UP
--------------040602010206030305030407--
--------------ms040501060706050302050501
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIJUTCC
AwMwggJsoAMCAQICEFN8DarMNuuKJDEtfs0UaqUwDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UE
BhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMT
I1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA4MDYzMDEzNTE1N1oX
DTA5MDYzMDEzNTE1N1owaDEQMA4GA1UEBBMHdmFuIEVyazETMBEGA1UEKhMKU2ViYXN0aWFh
bjEbMBkGA1UEAxMSU2ViYXN0aWFhbiB2YW4gRXJrMSIwIAYJKoZIhvcNAQkBFhNzZWJzdGVy
QHNlYnN0ZXIuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsJDDAeYHVmH/
GVxi+bhFx27dmg++9BdhPJfk8k041sqEqq7oXnR2GT54quY3Ac7A1BuOM2JvoICraGmjud4y
b3EanRnqGIK6iH+VAhhTlV/Owrb2Qm1e13DLxwLp1SocSQl4IrEbF9Y5H3ASdIrE0iFqkpju
nPiiHeNhz3LaI5ipjiluKYoH+F6gPx8njHoaDxPePCkSLg4r0IA0afLM74LVZxCRBZEfyRZS
J6VVUJefKlz91dWSzR/3xSw/rO4u9Ds/Zh7VBUKy3K+YFryHxRpUek0gSepE1b70Q39L9Sqd
M/NZqMvFpwrqgW2Zh2Nh8nqRge90maR4ypBzz3GzLwIDAQABozAwLjAeBgNVHREEFzAVgRNz
ZWJzdGVyQHNlYnN0ZXIuY29tMAwGA1UdEwEB/wQCMAAwDQYJKoZIhvcNAQEFBQADgYEAS1Sk
NMgDVzb0ktO9tPPacV0KdKhTYOHcICVmuDEe2sFHOkjLAI1iAKp640pqJEVqvRnfRcCFJ9hK
koPjjVZ+ui2rVmJWBG6FSloLRS/YYED4tUAw6DQhK61UOpjkpQxjCdm+5bHG/2ZgJAda1j0x
uiN822+xFkcaW/5PQgxSRxcwggMDMIICbKADAgECAhBTfA2qzDbriiQxLX7NFGqlMA0GCSqG
SIb3DQEBBQUAMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAo
UHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBD
QTAeFw0wODA2MzAxMzUxNTdaFw0wOTA2MzAxMzUxNTdaMGgxEDAOBgNVBAQTB3ZhbiBFcmsx
EzARBgNVBCoTClNlYmFzdGlhYW4xGzAZBgNVBAMTElNlYmFzdGlhYW4gdmFuIEVyazEiMCAG
CSqGSIb3DQEJARYTc2Vic3RlckBzZWJzdGVyLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEP
ADCCAQoCggEBALCQwwHmB1Zh/xlcYvm4Rcdu3ZoPvvQXYTyX5PJNONbKhKqu6F50dhk+eKrm
NwHOwNQbjjNib6CAq2hpo7neMm9xGp0Z6hiCuoh/lQIYU5VfzsK29kJtXtdwy8cC6dUqHEkJ
eCKxGxfWOR9wEnSKxNIhapKY7pz4oh3jYc9y2iOYqY4pbimKB/heoD8fJ4x6Gg8T3jwpEi4O
K9CANGnyzO+C1WcQkQWRH8kWUielVVCXnypc/dXVks0f98UsP6zuLvQ7P2Ye1QVCstyvmBa8
h8UaVHpNIEnqRNW+9EN/S/UqnTPzWajLxacK6oFtmYdjYfJ6kYHvdJmkeMqQc89xsy8CAwEA
AaMwMC4wHgYDVR0RBBcwFYETc2Vic3RlckBzZWJzdGVyLmNvbTAMBgNVHRMBAf8EAjAAMA0G
CSqGSIb3DQEBBQUAA4GBAEtUpDTIA1c29JLTvbTz2nFdCnSoU2Dh3CAlZrgxHtrBRzpIywCN
YgCqeuNKaiRFar0Z30XAhSfYSpKD441Wfrotq1ZiVgRuhUpaC0Uv2GBA+LVAMOg0ISutVDqY
5KUMYwnZvuWxxv9mYCQHWtY9MbojfNtvsRZHGlv+T0IMUkcXMIIDPzCCAqigAwIBAgIBDTAN
BgkqhkiG9w0BAQUFADCB0TELMAkGA1UEBhMCWkExFTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTES
MBAGA1UEBxMJQ2FwZSBUb3duMRowGAYDVQQKExFUaGF3dGUgQ29uc3VsdGluZzEoMCYGA1UE
CxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZpc2lvbjEkMCIGA1UEAxMbVGhhd3RlIFBl
cnNvbmFsIEZyZWVtYWlsIENBMSswKQYJKoZIhvcNAQkBFhxwZXJzb25hbC1mcmVlbWFpbEB0
aGF3dGUuY29tMB4XDTAzMDcxNzAwMDAwMFoXDTEzMDcxNjIzNTk1OVowYjELMAkGA1UEBhMC
WkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1Ro
YXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMIGfMA0GCSqGSIb3DQEBAQUAA4GN
ADCBiQKBgQDEpjxVc1X7TrnKmVoeaMB1BHCd3+n/ox7svc31W/Iadr1/DDph8r9RzgHU5VAK
MNcCY1osiRVwjt3J8CuFWqo/cVbLrzwLB+fxH5E2JCoTzyvV84J3PQO+K/67GD4Hv0CAAmTX
p6a7n2XRxSpUhQ9IBH+nttE8YQRAHmQZcmC3+wIDAQABo4GUMIGRMBIGA1UdEwEB/wQIMAYB
Af8CAQAwQwYDVR0fBDwwOjA4oDagNIYyaHR0cDovL2NybC50aGF3dGUuY29tL1RoYXd0ZVBl
cnNvbmFsRnJlZW1haWxDQS5jcmwwCwYDVR0PBAQDAgEGMCkGA1UdEQQiMCCkHjAcMRowGAYD
VQQDExFQcml2YXRlTGFiZWwyLTEzODANBgkqhkiG9w0BAQUFAAOBgQBIjNFQg+oLLswNo2as
Zw9/r6y+whehQ5aUnX9MIbj4Nh+qLZ82L8D0HFAgk3A8/a3hYWLD2ToZfoSxmRsAxRoLgnSe
JVCUYsfbJ3FXJY3dqZw5jowgT2Vfldr394fWxghOrvbqNOUQGls1TXfjViF4gtwhGTXeJLHT
HUb/XV9lTzGCA2QwggNgAgEBMHYwYjELMAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBD
b25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFp
bCBJc3N1aW5nIENBAhBTfA2qzDbriiQxLX7NFGqlMAkGBSsOAwIaBQCgggHDMBgGCSqGSIb3
DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTA4MDgwNjEwMTEwNVowIwYJKoZI
hvcNAQkEMRYEFGG3zudJ2x4EAN7lGaw46MnnswVYMFIGCSqGSIb3DQEJDzFFMEMwCgYIKoZI
hvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3
DQMCAgEoMIGFBgkrBgEEAYI3EAQxeDB2MGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3
dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJl
ZW1haWwgSXNzdWluZyBDQQIQU3wNqsw264okMS1+zRRqpTCBhwYLKoZIhvcNAQkQAgsxeKB2
MGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQu
MSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBDQQIQU3wNqsw2
64okMS1+zRRqpTANBgkqhkiG9w0BAQEFAASCAQCZ0nCXtY6aNbJ/zKludd0J+jCqzALz+Ssb
/41I/wBmRm884ulkQfC2ZNyU5RgB45F/Y7nQIz2XZP7V3sjVbx+YqFYcHWEEdkXPeTu3oxpC
6TrUbW+/ikdaUcg1jXRbJ7h7d19zi5Lg1GIMJIU2iyQXeNn19TqrPwwOpMyNpP5Pu9W4qpPN
bcytrWgzd4rR0/7y8kVWu+fvINJbEUXYahPFRLEY2BmYzZr/dko8rya8IgKvKbPlL7hC09zX
uYWr73IDViUFDsy7Oj1y9cycdpROjhjXF60akpzcAZjVdb4R8igUTB34OQQv4dwEuK8a0TFj
tGoZxi0LdLz8nWV0hB4TAAAAAAAA
--------------ms040501060706050302050501--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?489978B9.5080405>