Date: Fri, 03 May 2013 17:03:32 -0700 From: Mike Carlson <mike@bayphoto.com> To: Adam Nowacki <nowakpl@platinum.linux.pl> Cc: freebsd-fs@freebsd.org Subject: Re: zfs issue - disappearing data Message-ID: <51845054.3020302@bayphoto.com> In-Reply-To: <51843D0E.2020907@platinum.linux.pl> References: <5183F739.2040908@bayphoto.com> <51843D0E.2020907@platinum.linux.pl>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms050303070303060602090004 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable Interesting. Is that why zdb shows so many objects? Is this a configuration mistake, and would it lead to data loss? Can I provide any addition information ? Mike C On 5/3/2013 3:41 PM, Adam Nowacki wrote: > Looks like we have a leak with extended attributes: > > # zfs create -o mountpoint=3D/test root/test > # touch /test/file1 > # setextattr user test abc /test/file1 > # zdb root/test > Object lvl iblk dblk dsize lsize %full type > 8 1 16K 512 0 512 0.00 ZFS plain file > 9 1 16K 512 1K 512 100.00 ZFS directory > 10 1 16K 512 512 512 100.00 ZFS plain file > > object 8 - the file, > object 9 - extended attributes directory, > object 10 - value of the 'test' extended attribute > > # rm /test/file1 > # zdb root/test > > Object lvl iblk dblk dsize lsize %full type > 10 1 16K 512 512 512 100.00 ZFS plain file > > objects 8 and 9 are deleted, object 10 is still there (leaked). > > On 2013-05-03 19:43, Mike Carlson wrote: >> We had a critical issue with a zfs server that exports shares via samb= a >> (3.5) last night >> >> system info: >> uname -a >> >> FreeBSD zfs-1.discdrive.bayphoto.com 9.1-RELEASE FreeBSD 9.1-RELEA= SE >> #0 r243825: Tue Dec 4 09:23:10 UTC 2012 >> root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 >> >> zpool history: >> >> History for 'data': >> 2013-02-25.17:11:37 zpool create data raidz /dev/gpt/disk1.nop >> /dev/gpt/disk2.nop /dev/gpt/disk3.nop /dev/gpt/disk4.nop >> 2013-02-25.17:11:41 zpool add data raidz /dev/gpt/disk5.nop >> /dev/gpt/disk6.nop /dev/gpt/disk7.nop /dev/gpt/disk8.nop >> 2013-02-25.17:11:47 zpool add data raidz /dev/gpt/disk9.nop >> /dev/gpt/disk10.nop /dev/gpt/disk11.nop /dev/gpt/disk12.nop >> 2013-02-25.17:11:53 zpool add data raidz /dev/gpt/disk13.nop >> /dev/gpt/disk14.nop /dev/gpt/disk15.nop /dev/gpt/disk16.nop >> 2013-02-25.17:11:57 zpool add data raidz /dev/gpt/disk17.nop >> /dev/gpt/disk18.nop /dev/gpt/disk19.nop /dev/gpt/disk20.nop >> 2013-02-25.17:12:02 zpool add data raidz /dev/gpt/disk21.nop >> /dev/gpt/disk22.nop /dev/gpt/disk23.nop /dev/gpt/disk24.nop >> 2013-02-25.17:12:08 zpool add data spare /dev/gpt/disk25.nop >> /dev/gpt/disk26.nop >> 2013-02-25.17:12:15 zpool add data log /dev/gpt/log.nop >> 2013-02-25.17:12:19 zfs set checksum=3Dfletcher4 data >> 2013-02-25.17:12:22 zfs set compression=3Dlzjb data >> 2013-02-25.17:12:25 zfs set aclmode=3Dpassthrough data >> 2013-02-25.17:12:30 zfs set aclinherit=3Dpassthrough data >> 2013-02-25.17:13:25 zpool export data >> 2013-02-25.17:15:33 zpool import -d /dev/gpt data >> 2013-03-01.12:31:58 zpool add data cache /dev/gpt/cache.nop >> 2013-03-15.12:22:22 zfs create data/XML_WORKFLOW >> 2013-03-27.12:05:42 zfs create data/IMAGEQUIX >> 2013-03-27.13:32:54 zfs create data/ROES_ORDERS >> 2013-03-27.13:32:59 zfs create data/ROES_PRINTABLES >> 2013-03-27.13:33:21 zfs destroy data/ROES_PRINTABLES >> 2013-03-27.13:33:26 zfs create data/ROES_PRINTABLE >> >> We had a file structure drop off: >> >> /data/XML_WORKFLOW/XML_ORDERS/ >> >> around 5/2/2012 @ 17:00 >> >> In that directory, there were a few thousand directories (containing >> images and a couple metadata text/xml files) >> >> What is odd, is doing a du -h in the parent XML_WORKFLOW directory, on= ly >> reports ~150MB: >> >> # find . -type f |wc -l >> 86 >> # du -sh . >> 130M . >> >> >> however, df reports 1.5GB: >> >> # df -h . >> Filesystem Size Used Avail Capacity Mounted on >> data/XML_WORKFLOW 28T 1.5G 28T 0% /data/XML_WORKFLOW >> >> zdb -d shows: >> >> # zdb -d data/XML_WORKFLOW >> Dataset data/XML_WORKFLOW [ZPL], ID 139, cr_txg 339633, 1.53G, >> 212812 objects >> >> Digging further into zdb, the path is missing for most of those object= s: >> >> # zdb -ddddd data/XML_WORKFLOW 635248 >> Dataset data/XML_WORKFLOW [ZPL], ID 139, cr_txg 339633, 1.53G, >> 212812 objects, rootbp DVA[0]=3D<5:b274264000:2000> >> DVA[1]=3D<0:b4d81a8000:2000> [L0 DMU objset] fletcher4 lzjb LE >> contiguous unique double size=3D800L/200P birth=3D1202311L/1202311= P >> fill=3D212812=20 >> cksum=3D16d24fb5aa:6c2e0aff6bc:129af90fe2eff:2612f938c5292b >> >> Object lvl iblk dblk dsize lsize %full type >> 635248 1 16K 512 6.00K 512 100.00 ZFS plain file >> 168 bonus System attributes >> dnode flags: USED_BYTES USERUSED_ACCOUNTED >> dnode maxblkid: 0 >> path ???<object#635248> >> uid 11258 >> gid 10513 >> atime Thu May 2 17:31:26 2013 >> mtime Thu May 2 17:31:26 2013 >> ctime Thu May 2 17:31:26 2013 >> crtime Thu May 2 17:13:58 2013 >> gen 1197180 >> mode 100600 >> size 52 >> parent 635247 >> links 1 >> pflags 40800000005 >> Indirect blocks: >> 0 L0 3:a9da05a000:2000 200L/200P F=3D1 B=3D1197391/1197391 >> >> segment [0000000000000000, 0000000000000200) size 512 >> >> The application that writes to this volume runs on a windows client, s= o >> far, it has exhibited identical behavior across two zfs servers, but n= ot >> on a generic windows server 2003 network share. >> >> The question is, what is happening to the data. Is it a samba issue? I= s >> it ZFS? I've enabled the samba full_audit module to track file >> deletions, so I should have more information on that side. >> >> If anyone has seen similar behavior please let me know >> >> Mike C > --------------ms050303070303060602090004 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIITIjCC BhwwggQEoAMCAQICCGWP9DPVdj3ZMA0GCSqGSIb3DQEBCwUAMEsxDzANBgNVBAMMBlJvb3RD QTEWMBQGA1UECgwNQmF5IFBob3RvIExhYjETMBEGA1UECAwKQ2FsaWZvcm5pYTELMAkGA1UE BhMCVVMwHhcNMTIxMDIzMTczMjE4WhcNMjcxMDIzMTczMjE4WjBYMRwwGgYDVQQDDBNCYXkg UGhvdG8gUGVvcGxlIENBMRYwFAYDVQQKDA1CYXkgUGhvdG8gTGFiMRMwEQYDVQQIDApDYWxp Zm9ybmlhMQswCQYDVQQGEwJVUzCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBALuW vZE7FFSO77jEhdT/dZzu0XnPS35aepuvMvCNTagnNMbaf+AKrLZFaVRqBt4f6yl5TDXNEAMI +CJrg7T9dsoGN1Vygm4bIFzOIRtTZ1A7seemeonyqMto8gYMPhEFicuKavEAFcCFprVcVu5v yXbEG+tYkx08oUzeZ9LWZnh4a0wxAucbH0PcZFmdXFMbgjt6GSiR8jVUT1t/Dik1+mJ1A4K3 3VyM+G1CaqGIKg2UbvHiVQVoJrpgUc6vO5aWWgDreH+CAKLpyJwCj4BGphfHBPENu5LUp0CL fa4pA4r+OH0HNBp/I7utZMrIevh3pya34lDJXj0CQWSuVBR9Kp80IHHYGuyPU/sMXkUplsyI Y4KgQSRYRFNdWoboLwsRu1/2iDUaTR+XYH7Tu5zvCOedUm8ns/wfE7TW6kax5nR36VzuZS6B kUcuETO1QMxt2SJcHZIftiyTuxC8Y9Z7J6igR39pZPWGywJ2+oPRKFQD/u9RWZqjh9k5euVi 4bVwwog5DqnsymMTuKb0I1nCs5js+rfEVo+mc4gAfEmzk/dmpwkOnGV3oTfASbGtwMIlAEe5 HHJhdTA3jpVo3eQAZi479xZ7SozKvnjt1y9SMQ4uTJXjER/IVH3YC9S/BSFrqWLLDfWEGW84 SBn+ogldDIy3PX3SU7nOiN4Vse2Jg+r7AgMBAAGjgfYwgfMwHQYDVR0OBBYEFC34wnK9ZmLm B7vV52IeLHaapJ33MA8GA1UdEwEB/wQFMAMBAf8wHwYDVR0jBBgwFoAU00bJTnGPaRmhJAUZ t4x4J3soVyswgY8GA1UdHwSBhzCBhDCBgaB/oH2Ge2h0dHA6Ly9iYXljYS5iYXlob3RvLmxv Y2FsL2VqYmNhL3B1YmxpY3dlYi93ZWJkaXN0L2NlcnRkaXN0P2NtZD1jcmwmaXNzdWVyPUNO PVJvb3RDQSxPPUJheSUyMFBob3RvJTIwTGFiLFNUPUNhbGlmb3JuaWEsQz1VUzAOBgNVHQ8B Af8EBAMCAYYwDQYJKoZIhvcNAQELBQADggIBAEoDTeb1wVXz2VqFPrc3uaRnbVt6DCB9uBft L8YRgi5+FF5KHjvTqYHGiS38yVFfXM6MaN6yI7WSwtS+wVhM7Tdw6miwKNBApmCDKzjorNVX iZUmv88fjGm652LeIfT/oFGJg2oruoad9OfIofjT/jFpiFT7KOgjg4xeKCC0f+9naVo5uWOJ HTyFm1Kq6ryT+g7mZb7kLvndmJRW0aybn8Dls7/bhThEaOYkwn4dbZ6q0W2I5ZWg9bbh66V+ J9P1XObZA/TkRVTlpSYJ1hAgYX1yTbL360vuTAAwdSWQSFm4TAoXC4BsPdlgy86lDjNrW9id WX0RJCBzk/FzHhI4Aj9+SVhLl/Vkf9nbz8VhPPK2Az0PPYKy5ARtgOqGTKZbgVKut9Kgc/fn vUgPhObHQsD1Ui+NKsIGYBFKBhNmVXqMQSn4JzC9x0oDEmv37UGrcut2cCP3ZS17p137VaUM lQ0RWomju+sPCPFgyCPa/TLPoMZ2334uIxkRbDefvOoXIosORMQ9Jh50XqktUesbhuBfH9Q5 8h8bTWm1Cn/LxXW9qdSSbnta0OAH1G4hwVUlcusSM0o7Ude8tszw6kRpmEDDE8BQjE5nXSY7 wOf1eXfQfDkVQiouTD3l5ElqB98tnCnL/y9dphWoBLmiJwgb/4yWZ/Zewc1V65UFr7LmvcQM MIIGIzCCBAugAwIBAgIIYpSXgZOT7j0wDQYJKoZIhvcNAQELBQAwWDEcMBoGA1UEAwwTQmF5 IFBob3RvIFBlb3BsZSBDQTEWMBQGA1UECgwNQmF5IFBob3RvIExhYjETMBEGA1UECAwKQ2Fs aWZvcm5pYTELMAkGA1UEBhMCVVMwHhcNMTIxMDIzMTgwMDAzWhcNMTQxMDIzMTgwMDAzWjBg MRUwEwYKCZImiZPyLGQBAQwFMTMwNjkxFTATBgNVBAMMDE1pa2UgQ2FybHNvbjELMAkGA1UE CwwCSVQxFjAUBgNVBAoMDUJheSBQaG90byBMYWIxCzAJBgNVBAYTAlVTMIICIjANBgkqhkiG 9w0BAQEFAAOCAg8AMIICCgKCAgEAoTzIvF66A3wYPWXQFzmjBUuIrHSQGhOcE7rb0KQtar+H rkmHX56bCAklW5P/pd+yJ08lMwb3CxbTOz0q47JuBv2kKJO1xCgCua26Uvz3VAmfirmWwpXq zZBDqy/bEIt/XFfiVUC4jriGSEPrtx9q9nJJsb2JVRgtsbcHaaJFu8u8s8p8cLbcYdKobS00 g6+7it2IpIJhxc5tEMa1Yku3kCQiHVVFa9b4H5pFDHpkCrKZ43cuCneiR5kgr47z/3U66kLt J7Q5IT/i7nThjGQMa/f1JSWet8yeTomKvqkuEAA4o/IWQzEbtxzeps6vWxaCDULjEq69s//S 6PtqiQSmG9ZGFoPYD8/GGd4CMBqgjKopintD5sGTlJ851yZwl9VY/hRuxInp8gTjWrt1gQIB zlgSgSKnKTN6f+e85XMPU1y/wVz8RJWl8Tr11kzo6vrM14+ruNUxo1Ea3PJ9MUcWenoRKGSU I/IP94kZVjPkZlJv6tTF0Yi2Gclet/ZDu8vgvkxmUZYdQMGGlgZTCAsvHr37/ov6g51Tf+im 7410EsdYCmSINRGzWQAzlH9NscsW1TAd1Znog1H6NRDExY3ksjvFcKYOjUmkyWT8Vl2oJmT2 IzI23/C3esGL9OZzZ6K84MRNrH1y/yNp75vQnP3JfDMpbb5kkDp95Bu365qBluECAwEAAaOB 6DCB5TBSBggrBgEFBQcBAQRGMEQwQgYIKwYBBQUHMAGGNmh0dHA6Ly9iYXljYS5iYXlob3Rv LmxvY2FsL2VqYmNhL3B1YmxpY3dlYi9zdGF0dXMvb2NzcDAdBgNVHQ4EFgQUzeso+31hmtp3 soKHShXXtAEo+iMwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBQt+MJyvWZi5ge71ediHix2 mqSd9zAOBgNVHQ8BAf8EBAMCBDAwEwYDVR0lBAwwCgYIKwYBBQUHAwQwHAYDVR0RBBUwE4ER bWlrZUBiYXlwaG90by5jb20wDQYJKoZIhvcNAQELBQADggIBADnQfCasVgMsKsxIZAOZCbsU xo9BfsbpoM02p2aP+vPNDLXYRmcnH6ReeeUKSfIn0HmS9XkeHizMEXaC5sV9g4dasdQJQOGU mDcBnlxGn5fzNVFBM7/RHL83waYq4MCeyP9M7lSiNFZTrnSLVL9lIO0FLrBE06c9bn09kExc zkXI6Qm+e/MNrnoC3vw3GbH3a7tZCPsQcyNSok99jgPTRb1g9uVPg25M+ScYMU0wv2BE24u1 Dfzwcq52h64TllbzdVg/qOQH1HM96wmU+CtPuzA6eYnWRao/80LfQcyhNZ/jfMB/9xwFwsam o3Bw7SrSPEatw/tMyEEVMzas5/wZm2uMtab7642d5mr5OWLVPYgmKUscSlNt87vKkFhvn0Cz Z7O8O79WNMJA0sx1aomn1/ZrWDkd8X/ACUC2Fa3cV4AAzmjytiNu7r2z+GwdXPmvWSlBDXKX wLSoRkdq5hmYAP3GwXF0dsZo63WJLuCU1bPyERNLKdZM//eX832WgomPs4FA4xg0MUH0S7vJ eo7K1cTutZEmyLT623p0GcOINs2ir/ZqPTDLKszI7ytAltYaATt4kYUXbmMGGYItDf1X/caj DoLv2hjBTM5HORZYABC/Kfo9iL4KeYDqAvblJc7qyw+QXdHOUbwc9gQXQJvlQlfjDYvJLKme zoZ1sMzRBOl0MIIG1zCCBL+gAwIBAgIIGE38aUOyx8EwDQYJKoZIhvcNAQELBQAwWDEcMBoG A1UEAwwTQmF5IFBob3RvIFBlb3BsZSBDQTEWMBQGA1UECgwNQmF5IFBob3RvIExhYjETMBEG A1UECAwKQ2FsaWZvcm5pYTELMAkGA1UEBhMCVVMwHhcNMTIxMDIzMTc1NzQ1WhcNMTQxMDIz MTc1NzQ1WjBgMRUwEwYKCZImiZPyLGQBAQwFMTMwNjkxFTATBgNVBAMMDE1pa2UgQ2FybHNv bjELMAkGA1UECwwCSVQxFjAUBgNVBAoMDUJheSBQaG90byBMYWIxCzAJBgNVBAYTAlVTMIIC IjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAuECpC3YUm7GV0xz/DMmMQZ3EvMfvVhFB 77TcVxY1OoBlp2jk0ST0Hel/vp9uPhhi/eAlH89rC9fhNwORUHfeofWhoT3ZXrnjisNDQnb5 MgBV6wVM58PrikwU13FuNrIrPuUeuUE659BQhfgj2j2Pv9GVgBib6NBbHJAIOFo8H3wmd4b9 Yv6RKM145qSJCrJp96jSkoQSwni+jraHGMs55kgFP/3f0X6RpY7GhvItFI95Xauf7R5qTuW0 oamhvjFnlWVBoMuHd5yqzUgL7gQ0wfB31wfBP2ghFPAv715Qv9DYleFmgWE1LRkrt8clCzzk huj00CrEL+rSK9bDYNpa5AdCQ8aA/bG2x24ApOBvfDYnem+Ytu9lIY7qKZiP+9pASOiXsSSt g8CH9HoG+9GpK/rDyQ2nuNxTWEQEetwofkLdrGU/VgpcwIZqETsugu+l/FCu3Ogslezpiolb SqjCF5CU3aZravNZo4+HDriFKm7jyKEb+zUcXcwNH2iwbGt6uTMILslXbWXuhi8Lu3SKyacj 1gn9OMPQTCUKxaiTJXqIcJP0CV9wKcgTn7vanChDgz2fTVm6HOzb2RZlyhYzU4ofuMk+VGbg tBSDns+B9D1ACZMd58D2XdqRvmGPlnbJEZPkAZyDiR4mp/Aw09uXzOCIHzvqLhiysGon7lkV a18CAwEAAaOCAZswggGXMFIGCCsGAQUFBwEBBEYwRDBCBggrBgEFBQcwAYY2aHR0cDovL2Jh eWNhLmJheWhvdG8ubG9jYWwvZWpiY2EvcHVibGljd2ViL3N0YXR1cy9vY3NwMB0GA1UdDgQW BBRGT4gPK1JkpmK4YAsQgD82sskTGDAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFC34wnK9 ZmLmB7vV52IeLHaapJ33MIGlBgNVHR8EgZ0wgZowgZeggZSggZGGgY5odHRwOi8vYmF5Y2Eu YmF5aG90by5sb2NhbC9lamJjYS9wdWJsaWN3ZWIvd2ViZGlzdC9jZXJ0ZGlzdD9jbWQ9Y3Js Jmlzc3Vlcj1DTj1CYXklMjBQaG90byUyMFBlb3BsZSUyMENBLE89QmF5JTIwUGhvdG8lMjBM YWIsU1Q9Q2FsaWZvcm5pYSxDPVVTMA4GA1UdDwEB/wQEAwIGwDAdBgNVHSUEFjAUBggrBgEF BQcDAgYIKwYBBQUHAwQwHAYDVR0RBBUwE4ERbWlrZUBiYXlwaG90by5jb20wDQYJKoZIhvcN AQELBQADggIBAC+1BfV1bmcHZtgM4HN5rZek1vzY2UBLTHevnLDsLmNNASYMuOrm1TYMCI0/ LVkGltk0kq3lAP3hECsrSeH+SlmoRAmSCEOAwqOBU5oTqV/SMiQcZb6K2o/QUAVV6Si2KMyW U35hnqi9/vVNq4y28RjvMAydcrJpmn5qrWsy1ZLbW+Zu3jneBYjUcm4mQnqaCyjRyE2Z3byi /4m1sxISSdyqoRsWKsixSdb4bXUdNeNscltRYDMEDODtzYjnO7WebHtaMAEOyfE3n2gkPmGr dBopA6/Z53GVkG+MXF1wSlc3g38dHSpbY6qJiMkl8/wKCnkcMRPC3hRGQikYprh/oAbpHAyS q9P0jv/3cB0yzZ5b3n49PxqjwyH93lcRZJQ5pAMMWFk1B7IuYk9LVbJEVsPAmxFbWoS87Tn0 /zi6lkUNXoSgm4fJ+NBY5jnk1m48nPFIs7dpr8wW8qT2C0DiQz/tSKMramzsBduXY67djCZ5 cam1H5pRuaiEPEmrafC91i+wCsmjpMoqykKkuiGOlmY8LhRSZZ7M8qwtpo/0yur94z1Z5gyG 3yo/vZQtNDt8jaJ2ajGR50C48CtJufrckT3o3UrtjvQ3jCUnak2VbXIV6QNTio9NQKVH4+fL VhoGvc/PC/CifBs6Qyfk3a6nHdVfzwGx8Uytj/Br0AV0p8Y2MYIERjCCBEICAQEwZDBYMRww GgYDVQQDDBNCYXkgUGhvdG8gUGVvcGxlIENBMRYwFAYDVQQKDA1CYXkgUGhvdG8gTGFiMRMw EQYDVQQIDApDYWxpZm9ybmlhMQswCQYDVQQGEwJVUwIIGE38aUOyx8EwCQYFKw4DAhoFAKCC AbcwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTMwNTA0MDAw MzMyWjAjBgkqhkiG9w0BCQQxFgQUt3X2CKPiwt17MQ8ymreizAOzwwowbAYJKoZIhvcNAQkP MV8wXTALBglghkgBZQMEASowCwYJYIZIAWUDBAECMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMC AgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIBKDBzBgkrBgEEAYI3 EAQxZjBkMFgxHDAaBgNVBAMME0JheSBQaG90byBQZW9wbGUgQ0ExFjAUBgNVBAoMDUJheSBQ aG90byBMYWIxEzARBgNVBAgMCkNhbGlmb3JuaWExCzAJBgNVBAYTAlVTAghilJeBk5PuPTB1 BgsqhkiG9w0BCRACCzFmoGQwWDEcMBoGA1UEAwwTQmF5IFBob3RvIFBlb3BsZSBDQTEWMBQG A1UECgwNQmF5IFBob3RvIExhYjETMBEGA1UECAwKQ2FsaWZvcm5pYTELMAkGA1UEBhMCVVMC CGKUl4GTk+49MA0GCSqGSIb3DQEBAQUABIICACpfUGNUSIms8C0LqNDSLzkDFDrT4cgrnXuC WDHT+Sem3Erbux17eGoWTqtb+oUfuYFah5FsYSIF66Y/d6Wwot8g36WjWYHtN6daOHG2N4tY SsGAxkQ4oeCbhav0bVg2WglInnl4pXKw3Av7CHsEoyqzmccb/+2CR5zA0AgkRc5EPRyI9Ksd COOlQhFTDRz/35WPIBNdegqcGQreeyutcfwCPddrlxJRyHAgWXMsb0i+/FjyPU9YPvCLZpt2 AAeIOAkhsXZnrYa0et/MzDWvccRjGo03T9MaJm3GS+QV2EahEOvlRwBoY3amskYUyejkKu+6 dsRUJb4JwzT5NR25ViiSCO9ZTP+gLagPFbNf0E9sTmePkqD6linrwOoyiSLW3B3DS416K3ce eHTGmsWC8XWjzqqWn2vklEAsJ4ht5+D44viYmV4P/tHbe3X4DZPlt5X3Ogx58atp7I/DgkpR ZPwsrxApcswy/ZwwU4hWaV7UEqUOsAwAvfobkZxASiZSP7km/1XzbJpbxE9jmlt7ZHGaFI1O 58Ra2x9HUFUgob/FFlDBOwOAZDdYAi6F4/tAVqvDj63U/Hsh5RzjWOhlt5/S91ksLz0wFieA XwH7rWyQIbbfrPR4QRXnHROO6cQbZBPzrkSsq/Psv8CzAJlccQr7OACEFm7+fPetX5goIhkW AAAAAAAA --------------ms050303070303060602090004--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?51845054.3020302>