Date: Tue, 4 Jun 2013 13:09:08 +0530 From: Ajit Jain <ajit.jain@cloudbyte.com> To: freebsd-fs <freebsd-fs@freebsd.org>, Steven Hartland <killing@multiplay.co.uk> Subject: Fwd: seeing data corruption with zfs trim functionality Message-ID: <CAA71u6ZmZNKOHECqX=cEuVLFNfZkTCD6yUaz%2BhnG2GKsUHVp7A@mail.gmail.com> In-Reply-To: <CAA71u6a8d5b5CdaAp50HLGmNvK7p1PBJM6yH8AisCsSf%2B8U3-A@mail.gmail.com> References: <CAA71u6Y5dKZ9O0rqxCpx-9t7DYgTnPZSoNy-iHOnmzrOUYp%2Bvw@mail.gmail.com> <CAA71u6Zh7BbbdC=utqfR2MD1Nn=9euUDXHKqqu9NyBG-Jx%2B=Ow@mail.gmail.com> <9681E07546D348168052D4FC5365B4CD@multiplay.co.uk> <CAA71u6ZuO9CF0ECFS4z07-E5qPea-6SfNwkvhr_g6pFT5MV5yQ@mail.gmail.com> <CAA71u6YKGHDRVg6W_xnCNaA68bJvAZ2Lkp-UisiPqb1vKjJhfA@mail.gmail.com> <3E9CA9334E6F433A8F135ACD5C237340@multiplay.co.uk> <CAA71u6YZAKrmfTLU32f8UmYecmydwiqRT-OrR1ukZ9V6PGsU%2Bw@mail.gmail.com> <A05ACD84EB974E80B7142CE9982E479C@multiplay.co.uk> <93D0677B373A452BAF58C8EA6823783D@multiplay.co.uk> <CAA71u6bZ_4fb9FxYSwcrHBBApkZog30iQJGyTERi-xFMksud1g@mail.gmail.com> <35ABA7AAEB7F4D86A1ED54C4C47FEB49@multiplay.co.uk> <CAA71u6ahzRai=uUp5L6nDQxxEZC=d5jd4jBBfPNa2k29OwTZDg@mail.gmail.com> <2C2F5CAAE72B4658BFA09E4694A21375@multiplay.co.uk> <CAA71u6a3TJ_sO3Q%2BiJa8EHKE2iM0MKh31D37pGAoua7QU_6xYg@mail.gmail.com> <6E4EBFE196274519B847A47A062950EE@multiplay.co.uk> <CAA71u6bZqYcyW-3RAQj9zjYcWp%2BUXPa4KhH4__nY=S6EuVVR-w@mail.gmail.com> <F71FEDB8BA5142C5A3A0F72DF75A6421@multiplay.co.uk> <CAA71u6a8d5b5CdaAp50HLGmNvK7p1PBJM6yH8AisCsSf%2B8U3-A@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--001a11c1ed8a702b9904de4f2ed7 Content-Type: text/plain; charset=ISO-8859-1 Hi Steven, I am not able to send full output file to freebsd-fs. I am just sending the error file in this mail and will send you another mail which contain to full untar output. regards, ajit ---------- Forwarded message ---------- From: Ajit Jain <ajit.jain@cloudbyte.com> Date: Mon, Jun 3, 2013 at 11:51 PM Subject: Re: seeing data corruption with zfs trim functionality To: Steven Hartland <killing@multiplay.co.uk> Cc: freebsd-fs <freebsd-fs@freebsd.org> Hi Steven, untar of the tarball is throwing the error below: tar: Error exit delayed from previous errors. I have download the file from the link 3 times, every time I am seeing the same issue. Please find the tar output file and error (grep from the tar output file) attached with mail. checksum of tar ball (after unzip, on freebsd) is: root@everest:/pool_9stable/obj_src/new # cksum stable-9-r251096.tar 2972813925 3474278400 stable-9-r251096.tar regards, ajit On Fri, May 31, 2013 at 4:12 AM, Steven Hartland <killing@multiplay.co.uk>wrote: > Tar archive of /usr/src and /usr/obj with built world and GENERIC kernel > for ams64 can be found here:- > http://blog.multiplay.co.uk/**dropzone/freebsd/stable-9-**r251096.tar.gz<http://blog.multiplay.co.uk/dropzone/freebsd/stable-9-r251096.tar.gz>; > > This is based off r251096 with current proposed MFC of CAM BIO_DELETE & > ZFS TRIM. > > > Regards > Steve > ----- Original Message ----- From: "Ajit Jain" <ajit.jain@cloudbyte.com> > > > Hi Steven, >> >> That would be really great. I'll install build provided by you and can >> quickly >> update the result. I am kind of feeling that I am asking too much of fever >> from you. >> >> thanks for the help and bearing me, >> ajit >> >> >> On Wed, May 29, 2013 at 6:39 PM, Steven Hartland <killing@multiplay.co.uk >> >**wrote: >> >> Unfortunately FS corruption is a serious matters so even though I'm >>> 99.99% >>> convinced there isn't a problem I'd still prefer to confirm this was >>> indeed >>> an issue with your code base and not an issue with the current code prior >>> to MFC'ing. >>> >>> Would a pre-patched stable/9 source / build help. If so I can look at >>> making >>> that available for you. >>> >>> >>> Regards >>> Steve >>> >>> ----- Original Message ----- From: "Ajit Jain" <ajit.jain@cloudbyte.com> >>> >>> >>> Hi Steven, >>> >>>> >>>> Sorry for the long delay, but might delay even further. >>>> I think the reason for the corruption was, my code >>>> was not updated specially cam directory. >>>> >>>> I request please do not stop just because of the issue I reported. >>>> I'll update my src tree and rerun the experiments I was running >>>> if I see some issue then probably we fix the bug rather then stopping >>>> for MFC. >>>> >>>> thanks, >>>> ajit >>>> >>>> >>>> >>>> On Wed, May 29, 2013 at 5:19 PM, Steven Hartland < >>>> killing@multiplay.co.uk >>>> >**wrote: >>>> >>>> >>>> Sorry to pester, but any update on this Ajit? >>>> >>>>> >>>>> I ask as its currently blocking the MFC of TRIM to stable/8 & 9 and >>>>> I've >>>>> been >>>>> unable to reproduce this issue even with your testing code on working >>>>> FW >>>>> versions. >>>>> >>>>> >>>>> Regards >>>>> Steve >>>>> >>>>> ----- Original Message ----- From: "Ajit Jain" < >>>>> ajit.jain@cloudbyte.com> >>>>> >>>>> >>>>> Sure Steven, >>>>> >>>>> I'll apply the patches and update ASAP. >>>>>> >>>>>> thanks >>>>>> ajit >>>>>> >>>>>> >>>>>> On Thu, May 23, 2013 at 3:03 PM, Steven Hartland < >>>>>> killing@multiplay.co.uk >>>>>> >**wrote: >>>>>> >>>>>> >>>>>> I've attacked the two patch sets I'm looking to MFC to stable-9, one >>>>>> >>>>>> adds BIO_DELETE CAM changes and the other is ZFS TRIM support. >>>>>>> >>>>>>> They should both apply cleanly to stable-9, if you could test with >>>>>>> those on your machine and let me know. >>>>>>> >>>>>>> Regards >>>>>>> Steve >>>>>>> >>>>>>> ----- Original Message ----- From: "Ajit Jain" < >>>>>>> ajit.jain@cloudbyte.com> >>>>>>> >>>>>>> >>>>>>> Hi Steven, >>>>>>> >>>>>>> >>>>>>> FW version on the setup is P15. >>>>>>>> I will upgrade the FW to P16, but I think my >>>>>>>> best bet will be to update code base to 9 stable as unlike you, >>>>>>>> I was seeing corruption for all three delete methods. >>>>>>>> >>>>>>>> thanks >>>>>>>> ajit >>>>>>>> >>>>>>>> On Sat, May 18, 2013 at 4:15 AM, Steven Hartland < >>>>>>>> killing@multiplay.co.uk >>>>>>>> >**wrote: >>>>>>>> >>>>>>>> >>>>>>>> ----- Original Message ----- From: "Steven Hartland" < >>>>>>>> >>>>>>>> killing@multiplay.co.uk> >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> After initially seeing not issues, our overnight monitoring >>>>>>>>> started >>>>>>>>> >>>>>>>>> moaning >>>>>>>>> >>>>>>>>>> big time on the test box. So we checked and there was zpool >>>>>>>>>> corruption >>>>>>>>>> as >>>>>>>>>> well >>>>>>>>>> as a missing boot loader and a corrupt GPT, so I believe we have >>>>>>>>>> reproduced >>>>>>>>>> your issue. >>>>>>>>>> >>>>>>>>>> After recovering the machine I created 3 pools on 3 different >>>>>>>>>> disks >>>>>>>>>> each >>>>>>>>>> running a different delete_method. >>>>>>>>>> >>>>>>>>>> We then re-ran the tests which resulted in the pool running with >>>>>>>>>> delete_method >>>>>>>>>> WS16 being so broken it had suspended IO. A reboot resulted in it >>>>>>>>>> once >>>>>>>>>> again >>>>>>>>>> reporting no partition table via gpart. >>>>>>>>>> >>>>>>>>>> A third test run again produced a corrupt pool for WS16. >>>>>>>>>> >>>>>>>>>> I've conducted a preliminary review of the CAM WS16 code path >>>>>>>>>> along >>>>>>>>>> with >>>>>>>>>> SBC-3 >>>>>>>>>> spec which didn't identify any obvious issues. >>>>>>>>>> >>>>>>>>>> Given we're both using LSI 2008 based controllers it could be FW >>>>>>>>>> issue >>>>>>>>>> specific >>>>>>>>>> to WS16 but that's just speculation atm, so I'll continue to >>>>>>>>>> investigate. >>>>>>>>>> >>>>>>>>>> If you could re-test you end without using WS16 to see if you can >>>>>>>>>> reproduce the >>>>>>>>>> problem with either UNMAP or ATA_TRIM that would be a very useful >>>>>>>>>> data >>>>>>>>>> point. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> After much playing I narrow down a test case of one delete which >>>>>>>>>> was >>>>>>>>>> >>>>>>>>>> causing >>>>>>>>> disc corruption for us (deleted the partition table instead of data >>>>>>>>> in >>>>>>>>> the middle of the disk). >>>>>>>>> >>>>>>>>> The conclusion is LSI 2008 HBA with FW below P13 will eat the data >>>>>>>>> on >>>>>>>>> your >>>>>>>>> SATA >>>>>>>>> disks if you use WS16 due to the following bug:- >>>>>>>>> SCGCQ00230159 (DFCT) - Write same command to a SATA drive that >>>>>>>>> doesn't >>>>>>>>> support >>>>>>>>> SCT write same may write wrong region. >>>>>>>>> >>>>>>>>> After updating here to P16, which we would generally be running, >>>>>>>>> but >>>>>>>>> test >>>>>>>>> box >>>>>>>>> was new and hadnt updated yet the corruption issue is no longer >>>>>>>>> reproducable. >>>>>>>>> >>>>>>>>> So Ajit please check your FW version, I'm hoping to here your on >>>>>>>>> something >>>>>>>>> below P13, P12 possibly? >>>>>>>>> >>>>>>>>> If so then this is your issue, to fix simply update to P16 and the >>>>>>>>> problem >>>>>>>>> should be gone. >>>>>>>>> >>>>>>>>> >>>>>>>>> Regards >>>>>>>>> Steve >>>>>>>>> >>>>>>>>> >>>>>>>>> ==============================**********================== >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> This e.mail is private and confidential between Multiplay (UK) Ltd. >>>>>>>>> and >>>>>>>>> the person or entity to whom it is addressed. In the event of >>>>>>>>> misdirection, >>>>>>>>> the recipient is prohibited from using, copying, printing or >>>>>>>>> otherwise >>>>>>>>> disseminating it or any information contained in it. >>>>>>>>> In the event of misdirection, illegible or incomplete transmission >>>>>>>>> please >>>>>>>>> telephone +44 845 868 1337 >>>>>>>>> or return the E.mail to postmaster@multiplay.co.uk. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ==============================********================== >>>>>>>>> >>>>>>>> >>>>>>>> This e.mail is private and confidential between Multiplay (UK) >>>>>>> Ltd. and >>>>>>> the person or entity to whom it is addressed. In the event of >>>>>>> misdirection, >>>>>>> the recipient is prohibited from using, copying, printing or >>>>>>> otherwise >>>>>>> disseminating it or any information contained in it. >>>>>>> In the event of misdirection, illegible or incomplete transmission >>>>>>> please >>>>>>> telephone +44 845 868 1337 >>>>>>> or return the E.mail to postmaster@multiplay.co.uk. >>>>>>> >>>>>>> >>>>>>> >>>>>>> ==============================******================== >>>>>> >>>>> This e.mail is private and confidential between Multiplay (UK) Ltd. and >>>>> the person or entity to whom it is addressed. In the event of >>>>> misdirection, >>>>> the recipient is prohibited from using, copying, printing or otherwise >>>>> disseminating it or any information contained in it. >>>>> In the event of misdirection, illegible or incomplete transmission >>>>> please >>>>> telephone +44 845 868 1337 >>>>> or return the E.mail to postmaster@multiplay.co.uk. >>>>> >>>>> >>>>> >>>>> >>>> ==============================****================== >>> This e.mail is private and confidential between Multiplay (UK) Ltd. and >>> the person or entity to whom it is addressed. In the event of >>> misdirection, >>> the recipient is prohibited from using, copying, printing or otherwise >>> disseminating it or any information contained in it. >>> In the event of misdirection, illegible or incomplete transmission please >>> telephone +44 845 868 1337 >>> or return the E.mail to postmaster@multiplay.co.uk. >>> >>> >>> >> > ==============================**================== > This e.mail is private and confidential between Multiplay (UK) Ltd. and > the person or entity to whom it is addressed. In the event of misdirection, > the recipient is prohibited from using, copying, printing or otherwise > disseminating it or any information contained in it. > In the event of misdirection, illegible or incomplete transmission please > telephone +44 845 868 1337 > or return the E.mail to postmaster@multiplay.co.uk. > > --001a11c1ed8a702b9904de4f2ed7 Content-Type: application/octet-stream; name=error_log Content-Disposition: attachment; filename=error_log Content-Transfer-Encoding: base64 X-Attachment-Id: f_hhhzf8rb1 eCB1c3Ivb2JqL3Vzci9zcmMvc2hhcmUvem9uZWluZm8vYnVpbGRkaXIvRXVyb3BlL1Nrb3BqZTog Q2FuJ3QgY3JlYXRlICd1c3Ivb2JqL3Vzci9zcmMvc2hhcmUvem9uZWluZm8vYnVpbGRkaXIvRXVy b3BlL1Nrb3BqZScKeCB1c3Ivb2JqL3Vzci9zcmMvc2hhcmUvem9uZWluZm8vYnVpbGRkaXIvRXVy b3BlL1phZ3JlYjogQ2FuJ3QgY3JlYXRlICd1c3Ivb2JqL3Vzci9zcmMvc2hhcmUvem9uZWluZm8v YnVpbGRkaXIvRXVyb3BlL1phZ3JlYicKeCB1c3Ivb2JqL3Vzci9zcmMvc2hhcmUvem9uZWluZm8v YnVpbGRkaXIvRXVyb3BlL0lzbGVfb2ZfTWFuOiBDYW4ndCBjcmVhdGUgJ3Vzci9vYmovdXNyL3Ny Yy9zaGFyZS96b25laW5mby9idWlsZGRpci9FdXJvcGUvSXNsZV9vZl9NYW4nCnggdXNyL29iai91 c3Ivc3JjL3NoYXJlL3pvbmVpbmZvL2J1aWxkZGlyL0V1cm9wZS9HdWVybnNleTogQ2FuJ3QgY3Jl YXRlICd1c3Ivb2JqL3Vzci9zcmMvc2hhcmUvem9uZWluZm8vYnVpbGRkaXIvRXVyb3BlL0d1ZXJu c2V5Jwp4IHVzci9vYmovdXNyL3NyYy9zaGFyZS96b25laW5mby9idWlsZGRpci9FdXJvcGUvSmVy c2V5OiBDYW4ndCBjcmVhdGUgJ3Vzci9vYmovdXNyL3NyYy9zaGFyZS96b25laW5mby9idWlsZGRp ci9FdXJvcGUvSmVyc2V5Jwp4IHVzci9vYmovdXNyL3NyYy9zaGFyZS96b25laW5mby9idWlsZGRp ci9FdXJvcGUvUG9kZ29yaWNhOiBDYW4ndCBjcmVhdGUgJ3Vzci9vYmovdXNyL3NyYy9zaGFyZS96 b25laW5mby9idWlsZGRpci9FdXJvcGUvUG9kZ29yaWNhJwp4IHVzci9vYmovdXNyL3NyYy9zaGFy ZS96b25laW5mby9idWlsZGRpci9FdXJvcGUvVmF0aWNhbjogQ2FuJ3QgY3JlYXRlICd1c3Ivb2Jq L3Vzci9zcmMvc2hhcmUvem9uZWluZm8vYnVpbGRkaXIvRXVyb3BlL1ZhdGljYW4nCnggdXNyL29i ai91c3Ivc3JjL3NoYXJlL3pvbmVpbmZvL2J1aWxkZGlyL0V1cm9wZS9ManVibGphbmE6IENhbid0 IGNyZWF0ZSAndXNyL29iai91c3Ivc3JjL3NoYXJlL3pvbmVpbmZvL2J1aWxkZGlyL0V1cm9wZS9M anVibGphbmEnCnggdXNyL29iai91c3Ivc3JjL3NoYXJlL3pvbmVpbmZvL2J1aWxkZGlyL0V1cm9w ZS9adXJpY2g6IENhbid0IGNyZWF0ZSAndXNyL29iai91c3Ivc3JjL3NoYXJlL3pvbmVpbmZvL2J1 aWxkZGlyL0V1cm9wZS9adXJpY2gnCnggdXNyL29iai91c3Ivc3JjL3NoYXJlL3pvbmVpbmZvL2J1 aWxkZGlyL0V1cm9wZS9Sb21lOiBDYW4ndCBjcmVhdGUgJ3Vzci9vYmovdXNyL3NyYy9zaGFyZS96 b25laW5mby9idWlsZGRpci9FdXJvcGUvUm9tZScKeCB1c3Ivb2JqL3Vzci9zcmMvc2hhcmUvem9u ZWluZm8vYnVpbGRkaXIvRXVyb3BlL0JlbGdyYWRlOiBDYW4ndCBjcmVhdGUgJ3Vzci9vYmovdXNy L3NyYy9zaGFyZS96b25laW5mby9idWlsZGRpci9FdXJvcGUvQmVsZ3JhZGUnCnggdXNyL29iai91 c3Ivc3JjL3NoYXJlL3pvbmVpbmZvL2J1aWxkZGlyL0V1cm9wZS9QcmFndWU6IENhbid0IGNyZWF0 ZSAndXNyL29iai91c3Ivc3JjL3NoYXJlL3pvbmVpbmZvL2J1aWxkZGlyL0V1cm9wZS9QcmFndWUn CnggdXNyL29iai91c3Ivc3JjL3NoYXJlL3pvbmVpbmZvL2J1aWxkZGlyL0V1cm9wZS9NYXJpZWhh bW46IENhbid0IGNyZWF0ZSAndXNyL29iai91c3Ivc3JjL3NoYXJlL3pvbmVpbmZvL2J1aWxkZGly L0V1cm9wZS9NYXJpZWhhbW4nCnggdXNyL29iai91c3Ivc3JjL3NoYXJlL3pvbmVpbmZvL2J1aWxk ZGlyL0FyY3RpYy9Mb25neWVhcmJ5ZW46IENhbid0IGNyZWF0ZSAndXNyL29iai91c3Ivc3JjL3No YXJlL3pvbmVpbmZvL2J1aWxkZGlyL0FyY3RpYy9Mb25neWVhcmJ5ZW4nCnggdXNyL29iai91c3Iv c3JjL3NoYXJlL3pvbmVpbmZvL2J1aWxkZGlyL0FzaWEvSXN0YW5idWw6IENhbid0IGNyZWF0ZSAn dXNyL29iai91c3Ivc3JjL3NoYXJlL3pvbmVpbmZvL2J1aWxkZGlyL0FzaWEvSXN0YW5idWwnCngg dXNyL29iai91c3Ivc3JjL3NoYXJlL3pvbmVpbmZvL2J1aWxkZGlyL0FzaWEvTmljb3NpYTogQ2Fu J3QgY3JlYXRlICd1c3Ivb2JqL3Vzci9zcmMvc2hhcmUvem9uZWluZm8vYnVpbGRkaXIvQXNpYS9O aWNvc2lhJwp4IHVzci9vYmovdXNyL3NyYy9zaGFyZS96b25laW5mby9idWlsZGRpci9BbnRhcmN0 aWNhL01jTXVyZG86IENhbid0IGNyZWF0ZSAndXNyL29iai91c3Ivc3JjL3NoYXJlL3pvbmVpbmZv L2J1aWxkZGlyL0FudGFyY3RpY2EvTWNNdXJkbycKeCB1c3Ivb2JqL3Vzci9zcmMvc2hhcmUvem9u ZWluZm8vYnVpbGRkaXIvQW1lcmljYS9Mb3dlcl9QcmluY2VzOiBDYW4ndCBjcmVhdGUgJ3Vzci9v YmovdXNyL3NyYy9zaGFyZS96b25laW5mby9idWlsZGRpci9BbWVyaWNhL0xvd2VyX1ByaW5jZXMn CnggdXNyL29iai91c3Ivc3JjL3NoYXJlL3pvbmVpbmZvL2J1aWxkZGlyL0FtZXJpY2EvTmV3X1lv cms6IENhbid0IGNyZWF0ZSAndXNyL29iai91c3Ivc3JjL3NoYXJlL3pvbmVpbmZvL2J1aWxkZGly L0FtZXJpY2EvTmV3X1lvcmsnCnggdXNyL29iai91c3Ivc3JjL3NoYXJlL3pvbmVpbmZvL2J1aWxk ZGlyL0FtZXJpY2EvS3JhbGVuZGlqazogQ2FuJ3QgY3JlYXRlICd1c3Ivb2JqL3Vzci9zcmMvc2hh cmUvem9uZWluZm8vYnVpbGRkaXIvQW1lcmljYS9LcmFsZW5kaWprJwp4IHVzci9vYmovdXNyL3Ny Yy9zaGFyZS96b25laW5mby9idWlsZGRpci9BbWVyaWNhL01hcmlnb3Q6IENhbid0IGNyZWF0ZSAn dXNyL29iai91c3Ivc3JjL3NoYXJlL3pvbmVpbmZvL2J1aWxkZGlyL0FtZXJpY2EvTWFyaWdvdCcK eCB1c3Ivb2JqL3Vzci9zcmMvc2hhcmUvem9uZWluZm8vYnVpbGRkaXIvQW1lcmljYS9TaGlwcm9j azogQ2FuJ3QgY3JlYXRlICd1c3Ivb2JqL3Vzci9zcmMvc2hhcmUvem9uZWluZm8vYnVpbGRkaXIv QW1lcmljYS9TaGlwcm9jaycKeCB1c3Ivb2JqL3Vzci9zcmMvc2hhcmUvem9uZWluZm8vYnVpbGRk aXIvQW1lcmljYS9TdF9CYXJ0aGVsZW15OiBDYW4ndCBjcmVhdGUgJ3Vzci9vYmovdXNyL3NyYy9z aGFyZS96b25laW5mby9idWlsZGRpci9BbWVyaWNhL1N0X0JhcnRoZWxlbXknCnggdXNyL29iai91 c3Ivc3JjL3NoYXJlL3pvbmVpbmZvL2J1aWxkZGlyL0V0Yy9Vbml2ZXJzYWw6IENhbid0IGNyZWF0 ZSAndXNyL29iai91c3Ivc3JjL3NoYXJlL3pvbmVpbmZvL2J1aWxkZGlyL0V0Yy9Vbml2ZXJzYWwn CnggdXNyL29iai91c3Ivc3JjL3NoYXJlL3pvbmVpbmZvL2J1aWxkZGlyL0V0Yy9HTVQwOiBDYW4n dCBjcmVhdGUgJ3Vzci9vYmovdXNyL3NyYy9zaGFyZS96b25laW5mby9idWlsZGRpci9FdGMvR01U MCcKeCB1c3Ivb2JqL3Vzci9zcmMvc2hhcmUvem9uZWluZm8vYnVpbGRkaXIvRXRjL0dNVCswOiBD YW4ndCBjcmVhdGUgJ3Vzci9vYmovdXNyL3NyYy9zaGFyZS96b25laW5mby9idWlsZGRpci9FdGMv R01UKzAnCnggdXNyL29iai91c3Ivc3JjL3NoYXJlL3pvbmVpbmZvL2J1aWxkZGlyL0V0Yy9HTVQ6 IENhbid0IGNyZWF0ZSAndXNyL29iai91c3Ivc3JjL3NoYXJlL3pvbmVpbmZvL2J1aWxkZGlyL0V0 Yy9HTVQnCnggdXNyL29iai91c3Ivc3JjL3NoYXJlL3pvbmVpbmZvL2J1aWxkZGlyL0V0Yy9HcmVl bndpY2g6IENhbid0IGNyZWF0ZSAndXNyL29iai91c3Ivc3JjL3NoYXJlL3pvbmVpbmZvL2J1aWxk ZGlyL0V0Yy9HcmVlbndpY2gnCnggdXNyL29iai91c3Ivc3JjL3NoYXJlL3pvbmVpbmZvL2J1aWxk ZGlyL0V0Yy9adWx1OiBDYW4ndCBjcmVhdGUgJ3Vzci9vYmovdXNyL3NyYy9zaGFyZS96b25laW5m by9idWlsZGRpci9FdGMvWnVsdScKeCB1c3Ivb2JqL3Vzci9zcmMvc2hhcmUvem9uZWluZm8vYnVp bGRkaXIvRXRjL1VUQzogQ2FuJ3QgY3JlYXRlICd1c3Ivb2JqL3Vzci9zcmMvc2hhcmUvem9uZWlu Zm8vYnVpbGRkaXIvRXRjL1VUQycKeCB1c3Ivb2JqL3Vzci9zcmMvdG1wL2xlZ2FjeS91c3IvbGli ZXhlYy9tYWtld2hhdGlzLmxvY2FsOiBDYW4ndCBjcmVhdGUgJ3Vzci9vYmovdXNyL3NyYy90bXAv bGVnYWN5L3Vzci9saWJleGVjL21ha2V3aGF0aXMubG9jYWwnCnggdXNyL29iai91c3Ivc3JjL3Rt cC91c3IvYmluL2djcHA6IENhbid0IGNyZWF0ZSAndXNyL29iai91c3Ivc3JjL3RtcC91c3IvYmlu L2djcHAnCnggdXNyL29iai91c3Ivc3JjL3RtcC91c3IvYmluL0NDOiBDYW4ndCBjcmVhdGUgJ3Vz ci9vYmovdXNyL3NyYy90bXAvdXNyL2Jpbi9DQycKeCB1c3Ivb2JqL3Vzci9zcmMvdG1wL3Vzci9i aW4vY2M6IENhbid0IGNyZWF0ZSAndXNyL29iai91c3Ivc3JjL3RtcC91c3IvYmluL2NjJwp4IHVz ci9vYmovdXNyL3NyYy90bXAvdXNyL2Jpbi9jKys6IENhbid0IGNyZWF0ZSAndXNyL29iai91c3Iv c3JjL3RtcC91c3IvYmluL2MrKycKeCB1c3Ivb2JqL3Vzci9zcmMvdG1wL3Vzci9saWIvbGliZmwu YTogQ2FuJ3QgY3JlYXRlICd1c3Ivb2JqL3Vzci9zcmMvdG1wL3Vzci9saWIvbGliZmwuYScKeCB1 c3Ivb2JqL3Vzci9zcmMvdG1wL3Vzci9saWIvbGlibG4uYTogQ2FuJ3QgY3JlYXRlICd1c3Ivb2Jq L3Vzci9zcmMvdG1wL3Vzci9saWIvbGlibG4uYScKeCB1c3Ivb2JqL3Vzci9zcmMvbGliMzIvdXNy L2xpYjMyL2xpYmZsX3AuYTogQ2FuJ3QgY3JlYXRlICd1c3Ivb2JqL3Vzci9zcmMvbGliMzIvdXNy L2xpYjMyL2xpYmZsX3AuYScKeCB1c3Ivb2JqL3Vzci9zcmMvbGliMzIvdXNyL2xpYjMyL2xpYmZs LmE6IENhbid0IGNyZWF0ZSAndXNyL29iai91c3Ivc3JjL2xpYjMyL3Vzci9saWIzMi9saWJmbC5h Jwp4IHVzci9vYmovdXNyL3NyYy9saWIzMi91c3IvbGliMzIvbGlibC5hOiBDYW4ndCBjcmVhdGUg J3Vzci9vYmovdXNyL3NyYy9saWIzMi91c3IvbGliMzIvbGlibC5hJwp4IHVzci9vYmovdXNyL3Ny Yy9saWIzMi91c3IvbGliMzIvbGlibG5fcC5hOiBDYW4ndCBjcmVhdGUgJ3Vzci9vYmovdXNyL3Ny Yy9saWIzMi91c3IvbGliMzIvbGlibG5fcC5hJwo= --001a11c1ed8a702b9904de4f2ed7--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAA71u6ZmZNKOHECqX=cEuVLFNfZkTCD6yUaz%2BhnG2GKsUHVp7A>