Date: Sat, 20 Apr 2019 12:35:31 -0500 From: Karl Denninger <karl@denninger.net> To: Steven Hartland <killing@multiplay.co.uk>, freebsd-stable@freebsd.org Subject: Re: Concern: ZFS Mirror issues (12.STABLE and firmware 19 .v. 20) Message-ID: <3f53389a-0cb5-d106-1f64-bbc2123e975c@denninger.net> In-Reply-To: <758d5611-c3cf-82dd-220f-a775a57bdd0b@multiplay.co.uk> References: <f87f32f2-b8c5-75d3-4105-856d9f4752ef@denninger.net> <c96e31ad-6731-332e-5d2d-7be4889716e1@FreeBSD.org> <9a96b1b5-9337-fcae-1a2a-69d7bb24a5b3@denninger.net> <CACpH0MdLNQ_dqH%2Bto=amJbUuWprx3LYrOLO0rQi7eKw-ZcqWJw@mail.gmail.com> <1866e238-e2a1-ef4e-bee5-5a2f14e35b22@denninger.net> <3d2ad225-b223-e9db-cce8-8250571b92c9@FreeBSD.org> <2bc8a172-6168-5ba9-056c-80455eabc82b@denninger.net> <CACpH0MfmPzEO5BO2kFk8-F1hP9TsXEiXbfa1qxcvB8YkvAjWWw@mail.gmail.com> <2c23c0de-1802-37be-323e-d390037c6a84@denninger.net> <864062ab-f68b-7e63-c3da-539d1e9714f9@denninger.net> <6dc1bad1-05b8-2c65-99d3-61c547007dfe@denninger.net> <758d5611-c3cf-82dd-220f-a775a57bdd0b@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms020001040304000308080205 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 4/20/2019 10:50, Steven Hartland wrote: > Have you eliminated geli as possible source? No; I could conceivably do so by re-creating another backup volume set without geli-encrypting the drives, but I do not have an extra set of drives of the capacity required laying around to do that.=C2=A0 I would h= ave to do it with lower-capacity disks, which I can attempt if you think it would help.=C2=A0 I *do* have open slots in the drive backplane to set up= a second "test" unit of this sort.=C2=A0 For reasons below it will take at least a couple of weeks to get good data on whether the problem exists without geli, however. > > I've just setup an old server which has a LSI 2008 running and old FW > (11.0) so was going to have a go at reproducing this. > > Apart from the disconnect steps below is there anything else needed > e.g. read / write workload during disconnect? Yes.=C2=A0 An attempt to recreate this on my sandbox machine using smalle= r disks (WD RE-320s) and a decent amount of read/write activity (tens to ~100 gigabytes) on a root mirror of three disks with one taken offline did not succeed.=C2=A0 It *reliably* appears, however, on my backup volum= es with every drive swap.=C2=A0 The sandbox machine is physically identical other than the physical disks; both are Xeons with ECC RAM in them. The only operational difference is that the backup volume sets have a *lot* of data written to them via zfs send|zfs recv over the intervening period where with "ordinary" activity from I/O (which was the case on my sandbox) the I/O pattern is materially different.=C2=A0 The root pool on = the sandbox where I tried to reproduce it synthetically *is* using geli (in fact it boots native-encrypted.) The "ordinary" resilver on a disk swap typically covers ~2-3Tb and is a ~6-8 hour process. The usual process for the backup pool looks like this: Have 2 of the 3 physical disks mounted; the third is in the bank vault. Over the space of a week, the backup script is run daily.=C2=A0 It first imports the pool and then for each zfs filesystem it is backing up (which is not all of them; I have a few volatile ones that I don't care if I lose, such as object directories for builds and such, plus some that are R/O data sets that are backed up separately) it does: If there is no "...@zfs-base": zfs snapshot -r ...@zfs-base; zfs send -R =2E..@zfs-base | zfs receive -Fuvd $BACKUP else zfs rename -r ...@zfs-base ...@zfs-old zfs snapshot -r ...@zfs-base zfs send -RI ...@zfs-old ...@zfs-base |zfs recv -Fudv $BACKUP =2E... if ok then zfs destroy -vr ...@zfs-old otherwise print a complaint= and stop. When all are complete it then does a "zpool export backup" to detach the pool in order to reduce the risk of "stupid root user" (me) accidents. In short I send an incremental of the changes since the last backup, which in many cases includes a bunch of automatic snapshots that are taken on frequent basis out of the cron.=C2=A0 Typically there are a week= 's worth of these that accumulate between swaps of the disk to the vault, and the offline'd disk remains that way for a week.=C2=A0 I also wait for= the zpool destroy on each of the targets to drain before continuing, as not doing so back in the 9 and 10.x days was a good way to stimulate an instant panic on re-import the next day due to kernel stack page exhaustion if the previous operation destroyed hundreds of gigabytes of snapshots (which does routinely happen as part of the backed up data is Macrium images from PCs, so when a new month comes around the PC's backup routine removes a huge amount of old data from the filesystem.) Trying to simulate the checksum errors in a few hours' time thus far has failed.=C2=A0 But every time I swap the disks on a weekly basis I get a handful of checksum errors on the scrub.=C2=A0 If I export and re-import = the backup mirror after that the counters are zeroed -- the checksum error count does *not* remain across an export/import cycle although the "scrub repaired" line remains. For example after the scrub completed this morning I exported the pool (the script expects the pool exported before it begins) and ran the backup.=C2=A0 When it was complete: root@NewFS:~/backup-zfs # zpool status backup =C2=A0 pool: backup =C2=A0state: DEGRADED status: One or more devices has been taken offline by the administrator. =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Sufficient replicas exist for = the pool to continue functioning in a =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 degraded state. action: Online the device using 'zpool online' or replace the device with= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 'zpool replace'. =C2=A0 scan: scrub repaired 188K in 0 days 09:40:18 with 0 errors on Sat = Apr 20 08:45:09 2019 config: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 NAME=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 STATE=C2=A0=C2=A0=C2=A0=C2=A0 READ WRITE CKSUM =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 backup=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 DEGRADED=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0= 0=C2=A0=C2=A0=C2=A0=C2=A0 0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 mirror-0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 DEGRADED=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2= =A0=C2=A0=C2=A0 0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 gpt/ba= ckup61.eli=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ONLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 gpt/ba= ckup62-1.eli=C2=A0=C2=A0=C2=A0 ONLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 132828= 12295755460479=C2=A0 OFFLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0 was /dev/gpt/backup62-2.eli errors: No known data errors It knows it fixed the checksums but the error count is zero -- I did NOT "zpool clear". This may have been present in 11.2; I didn't run that long enough in this environment to know.=C2=A0 It definitely was *not* present in 11.1 a= nd before; the same data structure and script for backups has been in use for a very long time without any changes and this first appeared when I upgraded from 11.1 to 12.0 on this specific machine, with the exact same physical disks being used for over a year (they're currently 6Tb units; the last change out for those was ~1.5 years ago when I went from 4Tb to 6Tb volumes.)=C2=A0 I have both HGST-NAS and He-Enterprise disks in the rotation and both show identical behavior so it doesn't appear to be related to a firmware problem in one disk .vs. the other (e.g. firmware that fails to flush the on-drive cache before going to standby even though it was told to.) > > mps0: <Avago Technologies (LSI) SAS2008> port 0xe000-0xe0ff mem > 0xfaf3c000-0xfaf3ffff,0xfaf40000-0xfaf7ffff irq 26 at device 0.0 on pci= 3 > mps0: Firmware: 11.00.00.00, Driver: 21.02.00.00-fbsd > mps0: IOCCapabilities: > 185c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR> > > =C2=A0=C2=A0=C2=A0 Regards > =C2=A0=C2=A0=C2=A0 Steve > > On 20/04/2019 15:39, Karl Denninger wrote: >> I can confirm that 20.00.07.00 does *not* stop this. >> The previous write/scrub on this device was on 20.00.07.00.=C2=A0 It w= as >> swapped back in from the vault yesterday, resilvered without incident,= >> but a scrub says.... >> >> root@NewFS:/home/karl # zpool status backup >> =C2=A0=C2=A0 pool: backup >> =C2=A0=C2=A0state: DEGRADED >> status: One or more devices has experienced an unrecoverable error.=C2= =A0 An >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 attempt was made to c= orrect the error.=C2=A0 Applications are >> unaffected. >> action: Determine if the device needs to be replaced, and clear the >> errors >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 using 'zpool clear' o= r replace the device with 'zpool replace'. >> =C2=A0=C2=A0=C2=A0 see: http://illumos.org/msg/ZFS-8000-9P >> =C2=A0=C2=A0 scan: scrub repaired 188K in 0 days 09:40:18 with 0 error= s on Sat Apr >> 20 08:45:09 2019 >> config: >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 NAME=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 STATE=C2=A0=C2=A0=C2=A0=C2=A0 READ WRIT= E CKSUM >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 backup=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 DEGRADED=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0 >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 mirror-0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 DEGRADED=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0= =C2=A0=C2=A0=C2=A0=C2=A0 0 >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= gpt/backup61.eli=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ONLINE=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0 >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= gpt/backup62-1.eli=C2=A0=C2=A0=C2=A0 ONLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 47 >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= 13282812295755460479=C2=A0 OFFLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0= =C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0 was >> /dev/gpt/backup62-2.eli >> >> errors: No known data errors >> >> So this is firmware-invariant (at least between 19.00.00.00 and >> 20.00.07.00); the issue persists. >> >> Again, in my instance these devices are never removed "unsolicited" so= >> there can't be (or at least shouldn't be able to) unflushed data in th= e >> device or kernel cache.=C2=A0 The procedure is and remains: >> >> zpool offline ..... >> geli detach ..... >> camcontrol standby ... >> >> Wait a few seconds for the spindle to spin down. >> >> Remove disk. >> >> Then of course on the other side after insertion and the kernel has >> reported "finding" the device: >> >> geli attach ... >> zpool online .... >> >> Wait... >> >> If this is a boogered TXG that's held in the metadata for the >> "offline"'d device (maybe "off by one"?) that's potentially bad in tha= t >> if there is an unknown failure in the other mirror component the >> resilver will complete but data has been irrevocably destroyed. >> >> Granted, this is a very low probability scenario (the area where the b= ad >> checksums are has to be where the corruption hits, and it has to happe= n >> between the resilver and access to that data.)=C2=A0 Those are long od= ds but >> nonetheless a window of "you're hosed" does appear to exist. >> > --=20 Karl Denninger karl@denninger.net <mailto:karl@denninger.net> /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms020001040304000308080205 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DdgwggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBzAwggUYoAMCAQICEwCg0WvVwekjGFiO 62SckFwepz0wDQYJKoZIhvcNAQELBQAwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3Jp ZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBD QTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQTAeFw0xNzA4MTcyMTIx MjBaFw0yMjA4MTYyMTIxMjBaMFcxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRswGQYDVQQDDBJrYXJsQGRlbm5pbmdlci5uZXQw ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A 16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvWZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg 96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTg y+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYIXgVVPgfZZrbJJb5HWOQpvvhILpPCD3xs YJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMiWapsatKm8mxuOOGOEBhAoTVTwUHlMNTg 6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMbNQm1mWREQhw3axgGLSntjjnznJr5vsvX SYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZMqa20JLAF1YagutDiMRURU23iWS7bA9tM cXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN 5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1ly+5ZOZbxBAZZMod4y4b4FiRUhRI97r9l CxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY2BlA7ExM8XShMd9bRPZrNTokPQPUCWCg CdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEEMDAuMCwGCCsGAQUFBzABhiBodHRwOi8v b2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNVHRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIF oDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCG SAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBDbGllbnQgQ2VydGlmaWNhdGUwHQYDVR0O BBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNVHSMEgcIwgb+AFF3AXsKnjdPND5+bxVEC GKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UE BwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRh IFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYITAORIioIQ zl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcN AQELBQADggIBAJXboPFBMLMtaiUt4KEtJCXlHO/3ZzIUIw/eobWFMdhe7M4+0u3te0sr77QR dcPKR0UeHffvpth2Mb3h28WfN0FmJmLwJk+pOx4u6uO3O0E1jNXoKh8fVcL4KU79oEQyYkbu 2HwbXBU9HbldPOOZDnPLi0whi/sbFHdyd4/w/NmnPgzAsQNZ2BYT9uBNr+jZw4SsluQzXG1X lFL/qCBoi1N2mqKPIepfGYF6drbr1RnXEJJsuD+NILLooTNf7PMgHPZ4VSWQXLNeFfygoOOK FiO0qfxPKpDMA+FHa8yNjAJZAgdJX5Mm1kbqipvb+r/H1UAmrzGMbhmf1gConsT5f8KU4n3Q IM2sOpTQe7BoVKlQM/fpQi6aBzu67M1iF1WtODpa5QUPvj1etaK+R3eYBzi4DIbCIWst8MdA 1+fEeKJFvMEZQONpkCwrJ+tJEuGQmjoQZgK1HeloepF0WDcviiho5FlgtAij+iBPtwMuuLiL shAXA5afMX1hYM4l11JXntle12EQFP1r6wOUkpOdxceCcMVDEJBBCHW2ZmdEaXgAm1VU+fnQ qS/wNw/S0X3RJT1qjr5uVlp2Y0auG/eG0jy6TT0KzTJeR9tLSDXprYkN2l/Qf7/nT6Q03qyE QnnKiBXWAZXveafyU/zYa7t3PTWFQGgWoC4w6XqgPo4KV44OMYIFBzCCBQMCAQEwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBglghkgBZQMEAgMFAKCCAkUw GAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTkwNDIwMTczNTMx WjBPBgkqhkiG9w0BCQQxQgRATpkGJK6sJ0iZw/PgzmFGRWATiGfd4pd4wr/vpRGSAUNJR3l4 y1dQJv0vJ/Vx8pzztupUuUCh9VLjGKwrVIbXgTBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFl AwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3 DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGjBgkrBgEEAYI3EAQxgZUwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTCBpQYLKoZIhvcNAQkQAgsxgZWg gZIwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lz dGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0 ZW1zIExMQyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBgkqhkiG9w0BAQEF AASCAgAwpE340/dt2YXvRUi6Vq4Ehbs1DpKL/bYpjrDoam4C/MHfXVM92XufPJkcM2JRXTe5 PNekyDlEHyrk1Xw9cfTpI5rSiMsU0gaAwwujn7PztN4j07/Dx+6ZDSctA74geT5jo6kagkDb sEypDtwuLHrMmHKklnWg3rN4MXOLIeyeLZ3zGCXXsekm33Nh9zhrelOhpXYUZY0K8EwbZwrC /sMW+W2X9mS7E+3UPClY/4qogXV5aD3B3NxN56aNgJ7nLO35IDJ8QMaREpzbMEGM5Z4gXcrL ATsq8PhBwl5dsppau5NyfEn97f2OZnhr8zRPscu6tyBf8WjVF7aXwmpZRrsvW6bEh4s+AYKh yNlaXWqEbQLQ+SSKSLCugyHPm9T+gbugkgepDR6bq6lUaXSvtiU3GqUCNByNWyDe8savT8XM i6OeWwwlYtEWgXNLqU5+ybyQdBu9KnsWXHO0G9AgUSi7JIjbvCo8vJLWwmGV5aygPzJLKK1T MYFatN63C5fhALERo/GsugoUrl23F3UxWLDJVvs/iL2YdtJjVOMxZ5Dyv7Xnask1k96boQW3 0CuPOZkChmp6i5fxeao6HUa0MDV+uWYBb2j8ejYwtMm7FoaVvMRjb4g3BR7lFa6ckN27IV0l YlHM3Hn0/KalV9UmIVtW/ruBuiconAo8ow9TGwezvQAAAAAAAA== --------------ms020001040304000308080205--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3f53389a-0cb5-d106-1f64-bbc2123e975c>