Date: Fri, 12 Feb 2021 14:49:01 -0500 From: Karl Denninger <karl@denninger.net> To: freebsd-fs@freebsd.org Subject: Re: Reading a corrupted file on ZFS Message-ID: <5aa03138-1dc8-5a9c-1be6-d47ed22fc0cf@denninger.net> In-Reply-To: <2bf4f69c-9d5d-5ff9-0daa-c87515437ca3@artem.ru> References: <da892eeb-233f-551f-2faa-62f42c3c1d5b@artem.ru> <0ca45adf-8f60-a4c3-6264-6122444a3ffd@denninger.net> <899c6b4f-2368-7ec2-4dfe-fa09fab35447@artem.ru> <20210212165216.2f613482@fabiankeil.de> <10977ffc-f806-69dd-0cef-d4fd4fc5f649@artem.ru> <2f82f113-9ca1-99a9-a433-89e3ae5edcbe@denninger.net> <2bf4f69c-9d5d-5ff9-0daa-c87515437ca3@artem.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms050304070202000400010902 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable On 2/12/2021 13:26, Artem Kuchin wrote: > 12.02.2021 19:37, Karl Denninger =D0=BF=D0=B8=D1=88=D0=B5=D1=82: >> On 2/12/2021 11:22, Artem Kuchin wrote: >>> >>> This is frustrating. why..why.. >> >> You created a synthetic situation that in the real world almost-never = >> exists (ONE byte modified in all copies in the same allocation block=20 >> but all other data in that block is intact and recoverable.) >> > I could be 1 GB file with ZFS wisth block size of 1MB and with rotten=20 > bits within the same 1MB of block on different disks. How i did it is=20 > not important, life is unpredictable, i'm not trying to avoid=20 > everything. The question is what to do when it happens. And currently=20 > the answer is - nothing. > The answer to a problem that does not present itself statistically=20 speaking in the real world is the last one you worry about.=C2=A0 Worry a= bout=20 all the other ones and cover them first. I have had literal *hundreds* of drives fail in my time in IT.=C2=A0 I us= ed=20 to go change them in office machines (IBM PCs and clones -- the original = ones) 2-3 times a week at one of our larger customers.=C2=A0 This was in = the=20 day of 2,7RLL encoding if you were lucky for capacity reasons (MFM if=20 not)..... yeah, that far back and before.=C2=A0 Of course when you have a= =20 building full of these things as they did they break on a fairly regular = basis and hopefully you've thought the implications of that out and how=20 to recover from it. The *least* disruptive failures were sector-sized and occasionally I'd=20 get the frantic phone call from some place that had no backups and was=20 utterly desperate because something like their master inventory file was = on that disk.=C2=A0 There were times I was able to recover it with a lot = of=20 work and luck but when it was truly hosed and no amount of being willing = to wait would get you one good read I was *never* able to narrow the=20 data loss down to less than one sector irrespective of what the customer = was willing to pay.=C2=A0 Ever.=C2=A0 It sure wasn't cheap having actual = recovery=20 done (the clean-room style places can do it too, but the same issue=20 applies there; gone is gone, like it or not.) Nowdays modern drives frequently claim they have 63 (or 255)=20 sectors/track but that's bollocks; their actual internal organization is = completely opaque beyond the firmware in the drive in no small part=20 because a larger circumference can hold more data bits in a given=20 encoding than a smaller one, so the physical sectors on a given track is = not constant.=C2=A0 Most modern "larger" disks have several *hundred* sec= tors=20 per physical track, in many cases they are physically 4k sectors and=20 loss of servo information can result in the entire track being=20 unrecoverable. As sizes go up so does the size of data at risk from a given event.=C2=A0= =20 It's inherent in complex encoding schemes; the more you encode in a=20 symbol the more you lose when the symbol is corrupted to the point it=20 cannot be recovered. 128kb is all you lost?=C2=A0 How quaint. >> In almost-all actual cases of "bit rot" it's exactly that; random and = >> by statistics extraordinarily unlikely to hit all copies at once in=20 >> the same allocation block.=C2=A0 Therefore, ZFS can and does fix it; U= FS=20 >> or FAT silently returns the corrupted data, propagates it, and=20 >> eventually screws you down the road. > > In active fs you are right. But if this is a storage disk with movies=20 > and photos, then i can just checksum all files with a little script=20 > and recheck once in a while. So, for storage > > perposes i have all ZFS postitives and also can read as much data as i = > can. Because for long time storage it is more important to have=20 > ability read the data in any case. > I do that now with ZFS.=C2=A0 In some cases (long-term online but=20 nearly-read/only "cool" storage that is spun down when not being=20 accessed) I have backups that are only mounted and verified once a=20 year.=C2=A0 Mount the backup volumes and scrub them, for this specific re= ason=20 -- to detect the very unlikely but possible chance that two bits will=20 get flipped in the identical checksummed domain (ZFS block size) on two=20 or more copies.=C2=A0 If it happens you're hosed and "cold" storage (disk= s in=20 a vault for a year) is where patrol scrubs are most-likely to miss it=20 because you're not doing them on a regular basis since the disks are=20 physically in a different place.=C2=A0 Those backups are for all intents = and=20 purposes a fire insurance policy. I've yet to need them but that doesn't mean I'll stop maintaining them.=C2= =A0=20 I've also yet to have one fail verification but I'm sure it'll=20 eventually happen; that's why THOSE are redundant too. >> >> The nearly-every-case situation in the real world where a disk goes=20 >> physically bad (I've had this happen *dozens* of times over my IT=20 >> career) results in the drive being unable to=20 > > *NEARLY* is not good enough for me. > Then make backups and make THOSE ZFS mirrors (which is what I do),=20 dismount them and put them in a second location.=C2=A0 That's what backup= s=20 are for and ZFS makes that pretty easy and efficient with snapshots and=20 send/receive.=C2=A0 ZFS also makes segmenting data into "live" (updated a= lot=20 and currently), "cool" (updated once in a while) and "online but cold"=20 (either R/O or close to it) and moving data from one classification to=20 another (which happens often over time) and that in turn allows you to=20 *easily* build a backup strategy that works for all of that and yet=20 doesn't involve trying to cart a petabyte around in your car to and from = the bank vault.=C2=A0 It also allows a snapshot capacity tailored to each= =20 requirement so an "oops" (as opposed to hardware failures) can be=20 trivially recovered from which is not easy to do with UFS at all. If you have a 0.001 (1 in a thousand) risk over some period of time and=20 and have a second copy off-site with the same risk the odds of both=20 failing at the same time by other than physical calamity that takes out=20 both locations are multiplicative. Increase your redundancy as required=20 until you're comfortable.=C2=A0 If "Tsar Bomba" gets dropped on my locati= on,=20 well..... :-) I've had the ugly and wildly-improbable nightmare scenario happen to me=20 in real life when I ran MCSNet -- a disk adapter that went insane=20 internally and scribbled on *every attached device* at once.=C2=A0 It=20 destroyed the data on ALL of them.=C2=A0 If I didn't have a backup I woul= d=20 have been *done* right there.=C2=A0 This was before ZFS and the pucker fa= ctor=20 on that was considerable. >> return the block at all;=20 > > You mix device blocks and ZFS block. As far as i remember default ZFS=20 > block for checksumming is 16K and for big files storage better to have = > it around 128K. No I don't.=C2=A0 A block is a block is a block; they're different sizes = depending on application and where in the stack of "things" between the=20 physical disk and the application doing the reading or writing. In some=20 cases I run small block sizes on ZFS, in others large block sizes.=C2=A0 = Depends on the workload.=C2=A0 But I have control of that; I have no cont= rol=20 over what the physical media does inside its firmware. In some cases its = idea of a "block" is small, in others its large, and in still others=20 (SSDs in particular) how big it is depends on which block.=C2=A0 Scramble= an=20 allocation table in an SSD's controller and frequently everything on the = drive is gone *at once.*=C2=A0 The scope of damage if you get an uncorrec= ted=20 error off a given media depends on many things (e.g. encoding in use,=20 the media's sector size, is the error in the data or is in the servo=20 information on the platter if the device is spinning rust, if it's an=20 SSD is the error in the data block or is it in the allocation/mapping=20 table storage, etc.) >> In short there are very, very few actual "in the wild" failures where = >> one byte is damaged and the rest surrounding that one byte is intact=20 >> and retrievable.=C2=A0 In most cases where an actual failure occurs th= e=20 >> unreadable data constitutes *at least* a physical sector. >> > "very very few" is enough for me to think about. Then keep backups and make sure THEY are redundant too.=C2=A0 Do patrol s= cans=20 (e.g. scrubs) on those on some reasonable basis so you are comfortable=20 that the backups are good. > > One more thing. If you have one bad byte in a block of 16K and you=20 > have checksum and recalculate it then it is quite possible to just=20 > brute force every byte to match the checksum, thus restoring the data. True.=C2=A0 But as I said across an unbelievable number of failures that = I've=20 seen in my close to 40 years doing this stuff I've yet to have *one*=20 instance in which a media failure took out less than at least one=20 physical sector of the device and if it does, I still have one more=20 error of the *same sort* that has to happen in the same logical block on = the other vdev of the mirror, otherwise it gets detected and fixed with=20 no harm and no foul.=C2=A0 I'm sure it's possible for me to get screwed b= ut=20 then again it's also possible for me to get hit by an asteroid while=20 getting my mail -- it's just statistically improbable to the point that=20 I don't worry about it very much. I'm a LOT more worried about a failure on enough *other* vdevs during a=20 resilver to lose redundancy AND data, which can really hose you. That is = a hell of a lot more-likely statistically and if it happens you better=20 have backups because that second failure, being unrelated to the first,=20 hoses you *hard*. > > If you have mirror with two different bytes then bute forcing is even=20 > ether, > > Somehow, ZFS slaps my hands and does not allow to be sure that i can=20 > restore data when i needed it and decide myself if it is okay or not. > > For long time storage of big files it now seems better to store it on=20 > UFS mirror, checksum each 512bytes blocks of files and store checksums = > separetelly and run monthly/weekly "scrub". This way i would sleep=20 > better. > For the love of God no.=C2=A0 For openers how do you detect a corrupted=20 checksum file and know it's *the file* as opposed to the data? There are good reasons to run UFS instead of ZFS in many cases but this=20 is definitely not one of them. --=20 Karl Denninger karl@denninger.net <mailto:karl@denninger.net> /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms050304070202000400010902 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DdgwggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBzAwggUYoAMCAQICEwCg0WvVwekjGFiO 62SckFwepz0wDQYJKoZIhvcNAQELBQAwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3Jp ZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBD QTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQTAeFw0xNzA4MTcyMTIx MjBaFw0yMjA4MTYyMTIxMjBaMFcxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRswGQYDVQQDDBJrYXJsQGRlbm5pbmdlci5uZXQw ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A 16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvWZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg 96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTg y+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYIXgVVPgfZZrbJJb5HWOQpvvhILpPCD3xs YJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMiWapsatKm8mxuOOGOEBhAoTVTwUHlMNTg 6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMbNQm1mWREQhw3axgGLSntjjnznJr5vsvX SYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZMqa20JLAF1YagutDiMRURU23iWS7bA9tM cXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN 5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1ly+5ZOZbxBAZZMod4y4b4FiRUhRI97r9l CxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY2BlA7ExM8XShMd9bRPZrNTokPQPUCWCg CdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEEMDAuMCwGCCsGAQUFBzABhiBodHRwOi8v b2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNVHRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIF oDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCG SAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBDbGllbnQgQ2VydGlmaWNhdGUwHQYDVR0O BBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNVHSMEgcIwgb+AFF3AXsKnjdPND5+bxVEC GKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UE BwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRh IFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYITAORIioIQ zl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcN AQELBQADggIBAJXboPFBMLMtaiUt4KEtJCXlHO/3ZzIUIw/eobWFMdhe7M4+0u3te0sr77QR dcPKR0UeHffvpth2Mb3h28WfN0FmJmLwJk+pOx4u6uO3O0E1jNXoKh8fVcL4KU79oEQyYkbu 2HwbXBU9HbldPOOZDnPLi0whi/sbFHdyd4/w/NmnPgzAsQNZ2BYT9uBNr+jZw4SsluQzXG1X lFL/qCBoi1N2mqKPIepfGYF6drbr1RnXEJJsuD+NILLooTNf7PMgHPZ4VSWQXLNeFfygoOOK FiO0qfxPKpDMA+FHa8yNjAJZAgdJX5Mm1kbqipvb+r/H1UAmrzGMbhmf1gConsT5f8KU4n3Q IM2sOpTQe7BoVKlQM/fpQi6aBzu67M1iF1WtODpa5QUPvj1etaK+R3eYBzi4DIbCIWst8MdA 1+fEeKJFvMEZQONpkCwrJ+tJEuGQmjoQZgK1HeloepF0WDcviiho5FlgtAij+iBPtwMuuLiL shAXA5afMX1hYM4l11JXntle12EQFP1r6wOUkpOdxceCcMVDEJBBCHW2ZmdEaXgAm1VU+fnQ qS/wNw/S0X3RJT1qjr5uVlp2Y0auG/eG0jy6TT0KzTJeR9tLSDXprYkN2l/Qf7/nT6Q03qyE QnnKiBXWAZXveafyU/zYa7t3PTWFQGgWoC4w6XqgPo4KV44OMYIFBzCCBQMCAQEwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBglghkgBZQMEAgMFAKCCAkUw GAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMjEwMjEyMTk0OTAx WjBPBgkqhkiG9w0BCQQxQgRAYtgRkui2lAaKtA9v25XS0jJDm7OZg8mKNd/WeMqwNdWTpanl CwoHRlIvpe/Xo96V8EiEaeDUd3Ge5KSSy6d8cDBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFl AwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3 DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGjBgkrBgEEAYI3EAQxgZUwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTCBpQYLKoZIhvcNAQkQAgsxgZWg gZIwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lz dGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0 ZW1zIExMQyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBgkqhkiG9w0BAQEF AASCAgBW09FJa7hvqQ0+8LWyV62ccGkpG0lv6EUl274OGaZ/eUMvLNiTTN/5TQvCNWNzOj5Y Fdw6euX4g64PV2zmvzNib8tlisYEUIAPTkAtuWSg0Jp9KYggO/PrqPmN5OE1FA13GF9BxE0M lfMdPwmUAelDcD76RPwNmvYZCT+PLejGhMZM3vN5dk/fsuRDwjUjWDoO1Q7fTq6/ePhvXtmB VuQBVnj3uvrNPTwaSdm4bdO1tax9oMQcAqoIfsdSUb6wEMIRDLln6LscIwBLnqZWCchnb+mw +znRTKTxE8pG1p8Pqg2dEeSdvA7xINDRMkQ3/c18I/5R+NyYsV2myIBAd1UYJMZyk6BoTlJ8 EMLVxGNYrCMczsY5DrMzMVChDSDod+6S8ePpPhKkpNy0y7+MWpvZSVd7aUcjFtH0S040L7PP 308FUTIeNNUpEFFihSGfoOG9acurplBPVZyGBAv/DVRAfjojl2Z7Bw4Caq84KCOq0GoNWXeJ h3EK83taC1fTnoGKDOh2+1846q8wTlqoVERd3zSmPDm+n2dlaSTV9VSFAzo89f2RZ+yicxc4 qU1Z8tlnMGa/AWoL/amxeK5ApnHyfRMauUVSzbPZU08bu/Kp3NfUkt2qFN4rLkP2Nf1GB6m/ 8D+HtjRZMbN+DuNmW3JzWDSf0hfStXyyoc4Z1IvgZgAAAAAAAA== --------------ms050304070202000400010902--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5aa03138-1dc8-5a9c-1be6-d47ed22fc0cf>