Date: Sat, 20 Apr 2019 09:39:21 -0500 From: Karl Denninger <karl@denninger.net> To: freebsd-stable@freebsd.org Subject: Re: Concern: ZFS Mirror issues (12.STABLE and firmware 19 .v. 20) Message-ID: <6dc1bad1-05b8-2c65-99d3-61c547007dfe@denninger.net> In-Reply-To: <864062ab-f68b-7e63-c3da-539d1e9714f9@denninger.net> References: <f87f32f2-b8c5-75d3-4105-856d9f4752ef@denninger.net> <c96e31ad-6731-332e-5d2d-7be4889716e1@FreeBSD.org> <9a96b1b5-9337-fcae-1a2a-69d7bb24a5b3@denninger.net> <CACpH0MdLNQ_dqH%2Bto=amJbUuWprx3LYrOLO0rQi7eKw-ZcqWJw@mail.gmail.com> <1866e238-e2a1-ef4e-bee5-5a2f14e35b22@denninger.net> <3d2ad225-b223-e9db-cce8-8250571b92c9@FreeBSD.org> <2bc8a172-6168-5ba9-056c-80455eabc82b@denninger.net> <CACpH0MfmPzEO5BO2kFk8-F1hP9TsXEiXbfa1qxcvB8YkvAjWWw@mail.gmail.com> <2c23c0de-1802-37be-323e-d390037c6a84@denninger.net> <864062ab-f68b-7e63-c3da-539d1e9714f9@denninger.net>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms070906010904010606030101 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 4/13/2019 06:00, Karl Denninger wrote: > On 4/11/2019 13:57, Karl Denninger wrote: >> On 4/11/2019 13:52, Zaphod Beeblebrox wrote: >>> On Wed, Apr 10, 2019 at 10:41 AM Karl Denninger <karl@denninger.net> = wrote: >>> >>> >>>> In this specific case the adapter in question is... >>>> >>>> mps0: <Avago Technologies (LSI) SAS2116> port 0xc000-0xc0ff mem >>>> 0xfbb3c000-0xfbb3ffff,0xfbb40000-0xfbb7ffff irq 30 at device 0.0 on = pci3 >>>> mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd >>>> mps0: IOCCapabilities: >>>> 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,Hos= tDisc> >>>> >>>> Which is indeed a "dumb" HBA (in IT mode), and Zeephod says he conne= cts >>>> his drives via dumb on-MoBo direct SATA connections. >>>> >>> Maybe I'm in good company. My current setup has 8 of the disks conne= cted >>> to: >>> >>> mps0: <Avago Technologies (LSI) SAS2308> port 0xb000-0xb0ff mem >>> 0xfe240000-0xfe24ffff,0xfe200000-0xfe23ffff irq 32 at device 0.0 on p= ci6 >>> mps0: Firmware: 19.00.00.00, Driver: 21.02.00.00-fbsd >>> mps0: IOCCapabilities: >>> 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIX= Index,HostDisc> >>> >>> ... just with a cable that breaks out each of the 2 connectors into 4= >>> SATA-style connectors, and the other 8 disks (plus boot disks and SSD= >>> cache/log) connected to ports on... >>> >>> - ahci0: <ASMedia ASM1062 AHCI SATA controller> port >>> 0xd050-0xd057,0xd040-0xd043,0xd030-0xd037,0xd020-0xd023,0xd000-0xd01f= mem >>> 0xfe900000-0xfe9001ff irq 44 at device 0.0 on pci2 >>> - ahci2: <Marvell 88SE9230 AHCI SATA controller> port >>> 0xa050-0xa057,0xa040-0xa043,0xa030-0xa037,0xa020-0xa023,0xa000-0xa01f= mem >>> 0xfe610000-0xfe6107ff irq 40 at device 0.0 on pci7 >>> - ahci3: <AMD SB7x0/SB8x0/SB9x0 AHCI SATA controller> port >>> 0xf040-0xf047,0xf030-0xf033,0xf020-0xf027,0xf010-0xf013,0xf000-0xf00f= mem >>> 0xfea07000-0xfea073ff irq 19 at device 17.0 on pci0 >>> >>> ... each drive connected to a single port. >>> >>> I can actually reproduce this at will. Because I have 16 drives, whe= n one >>> fails, I need to find it. I pull the sata cable for a drive, determi= ne if >>> it's the drive in question, if not, reconnect, "ONLINE" it and wait f= or >>> resilver to stop... usually only a minute or two. >>> >>> ... if I do this 4 to 6 odd times to find a drive (I can tell, in gen= eral, >>> that a drive is part of the SAS controller or the SATA controllers...= so >>> I'm only looking among 8, ever) ... then I "REPLACE" the problem driv= e. >>> More often than not, the a scrub will find a few problems. In fact, = it >>> appears that the most recent scrub is an example: >>> >>> [1:7:306]dgilbert@vr:~> zpool status >>> pool: vr1 >>> state: ONLINE >>> scan: scrub repaired 32K in 47h16m with 0 errors on Mon Apr 1 23:1= 2:03 >>> 2019 >>> config: >>> >>> NAME STATE READ WRITE CKSUM >>> vr1 ONLINE 0 0 0 >>> raidz2-0 ONLINE 0 0 0 >>> gpt/v1-d0 ONLINE 0 0 0 >>> gpt/v1-d1 ONLINE 0 0 0 >>> gpt/v1-d2 ONLINE 0 0 0 >>> gpt/v1-d3 ONLINE 0 0 0 >>> gpt/v1-d4 ONLINE 0 0 0 >>> gpt/v1-d5 ONLINE 0 0 0 >>> gpt/v1-d6 ONLINE 0 0 0 >>> gpt/v1-d7 ONLINE 0 0 0 >>> raidz2-2 ONLINE 0 0 0 >>> gpt/v1-e0c ONLINE 0 0 0 >>> gpt/v1-e1b ONLINE 0 0 0 >>> gpt/v1-e2b ONLINE 0 0 0 >>> gpt/v1-e3b ONLINE 0 0 0 >>> gpt/v1-e4b ONLINE 0 0 0 >>> gpt/v1-e5a ONLINE 0 0 0 >>> gpt/v1-e6a ONLINE 0 0 0 >>> gpt/v1-e7c ONLINE 0 0 0 >>> logs >>> gpt/vr1log ONLINE 0 0 0 >>> cache >>> gpt/vr1cache ONLINE 0 0 0 >>> >>> errors: No known data errors >>> >>> ... it doesn't say it now, but there were 5 CKSUM errors on one of th= e >>> drives that I had trial-removed (and not on the one replaced). >>> _______________________________________________ >> That is EXACTLY what I'm seeing; the "OFFLINE'd" drive is the one that= , >> after a scrub, comes up with the checksum errors.=C2=A0 It does *not* = flag >> any errors during the resilver and the drives *not* taken offline do n= ot >> (ever) show checksum errors either. >> >> Interestingly enough you have 19.00.00.00 firmware on your card as wel= l >> -- which is what was on mine. >> >> I have flashed my card forward to 20.00.07.00 -- we'll see if it still= >> does it when I do the next swap of the backup set. > Verrrrrryyyyy interesting. > > This drive was last written/read under 19.00.00.00.=C2=A0 Yesterday I s= wapped > it back in.=C2=A0 Note that right now I am running: > > mps0: <Avago Technologies (LSI) SAS2116> port 0xc000-0xc0ff mem > 0xfbb3c000-0xfbb3ffff,0xfbb40000-0xfbb7ffff irq 30 at device 0.0 on pci= 3 > mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd > mps0: IOCCapabilities: > 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDi= sc> > > And, after the scrub completed overnight.... > > [karl@NewFS ~]$ zpool status backup > =C2=A0 pool: backup > =C2=A0state: DEGRADED > status: One or more devices has experienced an unrecoverable error.=C2=A0= An > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 attempt was made to correct = the error.=C2=A0 Applications are unaffected. > action: Determine if the device needs to be replaced, and clear the err= ors > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 using 'zpool clear' or repla= ce the device with 'zpool replace'. > =C2=A0=C2=A0 see: http://illumos.org/msg/ZFS-8000-9P > =C2=A0 scan: scrub repaired 4K in 0 days 06:30:55 with 0 errors on Sat = Apr 13 > 01:42:04 2019 > config: > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 NAME=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 STATE=C2=A0=C2=A0=C2=A0=C2=A0 READ WRITE CKSUM > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 backup=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 DEGRADED=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2= =A0=C2=A0=C2=A0=C2=A0 0 > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 mirror-0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= DEGRADED=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0= =C2=A0=C2=A0 0 > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 gpt/= backup61.eli=C2=A0=C2=A0=C2=A0=C2=A0 ONLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0 > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2650= 799076683778414=C2=A0 OFFLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0= =C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0 was > /dev/gpt/backup62-1.eli > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 gpt/= backup62-2.eli=C2=A0=C2=A0 ONLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2= =A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 1 > > errors: No known data errors > > The OTHER interesting data point is that the resilver *also* posted one= > checksum error, which I cleared before doing the scrub.=C2=A0 Both on t= he > 62-2 device. > > That would be one block in both cases.=C2=A0 The expected was several (= maybe > a half-dozen) checksum errors on 19.00.00.00 during the scrub but *zero= * > during the resilver. > > The unit which was put *into* the vault and is now offline was written > and scrubbed under 20.00.07.00.=C2=A0 The behavior change certainly imp= lies > that there are some differences and again, none of these OFFLINE state > situations are uncontrolled -- in each case the drive is taken offline > intentionally, the geli provider is detached and then the unit has > "camcontrol standby" executed against it before it is yanked, so in > theory at least there should be no way for a unflushed but write-cached= > block to be lost or damaged. > > I smell a rat but it may well be in the 19.00.00.00 firmware on the car= d... I can confirm that 20.00.07.00 does *not* stop this. The previous write/scrub on this device was on 20.00.07.00.=C2=A0 It was swapped back in from the vault yesterday, resilvered without incident, but a scrub says.... root@NewFS:/home/karl # zpool status backup =C2=A0 pool: backup =C2=A0state: DEGRADED status: One or more devices has experienced an unrecoverable error.=C2=A0= An =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 attempt was made to correct th= e error.=C2=A0 Applications are unaffected. action: Determine if the device needs to be replaced, and clear the error= s =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 using 'zpool clear' or replace= the device with 'zpool replace'. =C2=A0=C2=A0 see: http://illumos.org/msg/ZFS-8000-9P =C2=A0 scan: scrub repaired 188K in 0 days 09:40:18 with 0 errors on Sat = Apr 20 08:45:09 2019 config: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 NAME=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 STATE=C2=A0=C2=A0=C2=A0=C2=A0 READ WRITE CKSUM =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 backup=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 DEGRADED=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0= 0=C2=A0=C2=A0=C2=A0=C2=A0 0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 mirror-0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 DEGRADED=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2= =A0=C2=A0=C2=A0 0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 gpt/ba= ckup61.eli=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ONLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 gpt/ba= ckup62-1.eli=C2=A0=C2=A0=C2=A0 ONLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 47 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 132828= 12295755460479=C2=A0 OFFLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0 was /dev/gpt/backup62-2.eli errors: No known data errors So this is firmware-invariant (at least between 19.00.00.00 and 20.00.07.00); the issue persists. Again, in my instance these devices are never removed "unsolicited" so there can't be (or at least shouldn't be able to) unflushed data in the device or kernel cache.=C2=A0 The procedure is and remains: zpool offline ..... geli detach ..... camcontrol standby ... Wait a few seconds for the spindle to spin down. Remove disk. Then of course on the other side after insertion and the kernel has reported "finding" the device: geli attach ... zpool online .... Wait... If this is a boogered TXG that's held in the metadata for the "offline"'d device (maybe "off by one"?) that's potentially bad in that if there is an unknown failure in the other mirror component the resilver will complete but data has been irrevocably destroyed. Granted, this is a very low probability scenario (the area where the bad checksums are has to be where the corruption hits, and it has to happen between the resilver and access to that data.)=C2=A0 Those are long odds = but nonetheless a window of "you're hosed" does appear to exist. --=20 Karl Denninger karl@denninger.net <mailto:karl@denninger.net> /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms070906010904010606030101 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DdgwggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBzAwggUYoAMCAQICEwCg0WvVwekjGFiO 62SckFwepz0wDQYJKoZIhvcNAQELBQAwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3Jp ZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBD QTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQTAeFw0xNzA4MTcyMTIx MjBaFw0yMjA4MTYyMTIxMjBaMFcxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRswGQYDVQQDDBJrYXJsQGRlbm5pbmdlci5uZXQw ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A 16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvWZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg 96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTg y+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYIXgVVPgfZZrbJJb5HWOQpvvhILpPCD3xs YJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMiWapsatKm8mxuOOGOEBhAoTVTwUHlMNTg 6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMbNQm1mWREQhw3axgGLSntjjnznJr5vsvX SYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZMqa20JLAF1YagutDiMRURU23iWS7bA9tM cXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN 5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1ly+5ZOZbxBAZZMod4y4b4FiRUhRI97r9l CxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY2BlA7ExM8XShMd9bRPZrNTokPQPUCWCg CdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEEMDAuMCwGCCsGAQUFBzABhiBodHRwOi8v b2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNVHRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIF oDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCG SAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBDbGllbnQgQ2VydGlmaWNhdGUwHQYDVR0O BBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNVHSMEgcIwgb+AFF3AXsKnjdPND5+bxVEC GKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UE BwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRh IFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYITAORIioIQ zl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcN AQELBQADggIBAJXboPFBMLMtaiUt4KEtJCXlHO/3ZzIUIw/eobWFMdhe7M4+0u3te0sr77QR dcPKR0UeHffvpth2Mb3h28WfN0FmJmLwJk+pOx4u6uO3O0E1jNXoKh8fVcL4KU79oEQyYkbu 2HwbXBU9HbldPOOZDnPLi0whi/sbFHdyd4/w/NmnPgzAsQNZ2BYT9uBNr+jZw4SsluQzXG1X lFL/qCBoi1N2mqKPIepfGYF6drbr1RnXEJJsuD+NILLooTNf7PMgHPZ4VSWQXLNeFfygoOOK FiO0qfxPKpDMA+FHa8yNjAJZAgdJX5Mm1kbqipvb+r/H1UAmrzGMbhmf1gConsT5f8KU4n3Q IM2sOpTQe7BoVKlQM/fpQi6aBzu67M1iF1WtODpa5QUPvj1etaK+R3eYBzi4DIbCIWst8MdA 1+fEeKJFvMEZQONpkCwrJ+tJEuGQmjoQZgK1HeloepF0WDcviiho5FlgtAij+iBPtwMuuLiL shAXA5afMX1hYM4l11JXntle12EQFP1r6wOUkpOdxceCcMVDEJBBCHW2ZmdEaXgAm1VU+fnQ qS/wNw/S0X3RJT1qjr5uVlp2Y0auG/eG0jy6TT0KzTJeR9tLSDXprYkN2l/Qf7/nT6Q03qyE QnnKiBXWAZXveafyU/zYa7t3PTWFQGgWoC4w6XqgPo4KV44OMYIFBzCCBQMCAQEwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBglghkgBZQMEAgMFAKCCAkUw GAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTkwNDIwMTQzOTIx WjBPBgkqhkiG9w0BCQQxQgRASwZdol2oQ6cIII6ZuyV8UWL46NapIjpaYQ93Tyw/rmR4StST nnmRUqn/LiljOQKwdBAOoXplkPaVuSkZUPoMITBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFl AwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3 DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGjBgkrBgEEAYI3EAQxgZUwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTCBpQYLKoZIhvcNAQkQAgsxgZWg gZIwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lz dGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0 ZW1zIExMQyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBgkqhkiG9w0BAQEF AASCAgCz6rmwBBY/dQ0YTtrGwCMybyApZBRkWR4RgT2ePg8t8jfqvRZtgOArY3OD51LkCo5i ieug3e7HLL8kayRbb7Y6g1eQOnj3rapDQN/miwyOeY5kuTe3BGLF71nRmsrRDzXbG+GSAT5x q+3A+QEQJ8J+GlHNMLqejcJBsiVIj7kO7AX9uMvB4VzO6WxXiUKuoESRD7T0HsR6GOcGX4V3 BwtnN7z8c0ZDwDsvVi8JoTorVZc8RfpmHFGJA65KVQOlf1hAP2wLsarXoVx/uuxaNZG1TBJz MNKdNpZJrrzIo+SEzOtQfwvBa+RbQqjfmBccFX3vEdUebrnhivUcraiFxjXyISpUnImH9fU9 JHvGGkzP01GHKSJeXfgcdrRXL08KOvYggg6mLf9VATgIN8xbjP4jaan4Z8QS2ndTxPdxTb7g yfGQEB1/osoCKY6SmPDuf0r83Gk6surZYXTAQcWpKGswC1DEZH1qHR6b8xCmzJpeZ1TQO8eW D7Wfkwf+0yXOt3ikT8UKcNd9zqebNSewVYpCUW0N2WjFYhx3+ejoyZpFABHDEwu2mFZGEP7r m3Gv8dOuveqcukdYBUOjRjhzbE/2AhgHjXVLXlCsIhRp0hp7HlVTyjcvcDSUZMImMMIPtOq5 fmTfYbxsIXda09SMZ7Fh2sv6soymTFN+sjTEpWrl0QAAAAAAAA== --------------ms070906010904010606030101--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6dc1bad1-05b8-2c65-99d3-61c547007dfe>