From owner-ctm-users@freebsd.org Mon Mar 14 11:42:52 2016 Return-Path: Delivered-To: ctm-users@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 77F9DAD0F71 for ; Mon, 14 Mar 2016 11:42:52 +0000 (UTC) (envelope-from stephen@missouri.edu) Received: from um-nip3-missouri-out.um.umsystem.edu (um-nip3-missouri-out.um.umsystem.edu [198.209.49.163]) (using TLSv1.2 with cipher RC4-SHA (128/128 bits)) (Client CN "um-tip1.um.umsystem.edu", Issuer "InCommon RSA Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 180093E9 for ; Mon, 14 Mar 2016 11:42:51 +0000 (UTC) (envelope-from stephen@missouri.edu) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: =?us-ascii?q?A2ASBQCtouZW/9KeoM9dg0ZSbQa8KhmFd?= =?us-ascii?q?AKBKDwQAQEBAQEBAWQcC4RCAQEEeBECAQgYCRYPCQMCAQIBICUCBA0IAQEXiAm?= =?us-ascii?q?2dwGEDB6JXH6EOIQ8BZdLAYVtlxeOfTcrggMZgUhqiWUBfQEBAQ?= X-IPAS-Result: =?us-ascii?q?A2ASBQCtouZW/9KeoM9dg0ZSbQa8KhmFdAKBKDwQAQEBAQE?= =?us-ascii?q?BAWQcC4RCAQEEeBECAQgYCRYPCQMCAQIBICUCBA0IAQEXiAm2dwGEDB6JXH6EO?= =?us-ascii?q?IQ8BZdLAYVtlxeOfTcrggMZgUhqiWUBfQEBAQ?= Received: from um-ncas5.um.umsystem.edu ([207.160.158.210]) by um-nip3-exch-relay.um.umsystem.edu with ESMTP; 14 Mar 2016 06:41:40 -0500 Received: from UM-MBX-N02.um.umsystem.edu ([169.254.5.130]) by UM-NCAS5.um.umsystem.edu ([207.160.158.210]) with mapi id 14.03.0266.001; Mon, 14 Mar 2016 06:41:40 -0500 From: "Montgomery-Smith, Stephen" To: "ctm-users@freebsd.org" Subject: Re: Space problems with svn-cur Thread-Topic: Space problems with svn-cur Thread-Index: AQHRfEXQLqoGv/Vz/E+rbmbq4GS10J9W88CAgABXiYCAAJDkAIABTAqA Date: Mon, 14 Mar 2016 11:41:39 +0000 Message-ID: <56E6A373.2040901@missouri.edu> References: <201603120959.u2C9x6jU014905@fire.js.berklix.net> <56E4BBE1.9080007@missouri.edu> <56E5135F.70302@inse.ru> <56E58CEA.1000009@missouri.edu> In-Reply-To: <56E58CEA.1000009@missouri.edu> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 x-originating-ip: [207.160.158.194] Content-Type: text/plain; charset="Windows-1252" Content-ID: <8A86103C7C5B754D86435580B0726546@missouri.edu> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Mar 2016 11:42:52 -0000 On 03/13/2016 10:53 AM, Montgomery-Smith, Stephen wrote: > On 03/13/2016 01:14 AM, Roman Kurakin wrote: >> Hi, >> >> On 03/13/2016 04:01 AM, Montgomery-Smith, Stephen wrote: >>> On 03/12/2016 03:59 AM, Julian H. Stacey wrote: >>>> "Montgomery-Smith, Stephen" wrote: >>>>> The svn repository is now getting so large that the CTM generating >>>>> computer is running out of disk space when it creates the xEmpty delt= as >>>>> for svn-cur. The easist band-aid fix is for me to simply stop >>>>> generating these xEmpty deltas. Since CTM is no longer FreeBSD >>>>> approved, I'm going to assume that no-one new is going to join, and >>>>> hence this is not a problem. >>>>> >>>>> Comments? >>>> Hi Stephen >>>> There's plenty of space on ctm.berklix.org >>> The issue is that there isn't enough space to create the xEmpty deltas. >>> There is plenty of space to store the deltas. >>> >>> The svn repositories add up to about 35GB. Fully compressed, they >>> shrink to about 4GB. >>> >>> I'll look into other solutions. But I also get a feeling I am putting = a >>> lot of effort into something that is only used by a very few people. >> The xEmpty is the best way to start clean new copy on a new host. >> The solution could be a big deltas relative the last xEmpty. >=20 > This is the problem. Internally the svn xEmpty deltas look like this: >=20 > CTMTR > >=20 > where the big number is the size of the following huge tar file. What I > have been doing is to create the huge tar file in a tmp directory, and > then find its size, and then copy this file onto the end of the xEmpty > file. This meant that tmp had to be able to take 25GB (the size of the > ports repository). >=20 > Having big deltas relative to the last xEmpty is not a solution, because > I still have to store the repository of every 100th delta. That's 25GB > whichever way you look at it. >=20 > What I am trying now is this. I create the tar file twice - once to get > the size, and the second time to append it to the xEmpty file. This > relies on tar creating the same size file both times. Experiments I > recently tried suggest this should be OK. This seems to be working. I created an xEmpty for 4406. I also found out it takes about 5 hours to create an xEmpty for svn-cur. Almost all that time is spent by xz doing the compression.