From owner-ctm-users@freebsd.org Sun Mar 13 01:02:35 2016 Return-Path: Delivered-To: ctm-users@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2F5A0A92A7C for ; Sun, 13 Mar 2016 01:02:35 +0000 (UTC) (envelope-from stephen@missouri.edu) Received: from mst-rip6-missouri-out.um.umsystem.edu (mst-rip6-missouri-out.um.umsystem.edu [198.209.50.149]) (using TLSv1.2 with cipher RC4-SHA (128/128 bits)) (Client CN "um-tip1.um.umsystem.edu", Issuer "InCommon RSA Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DB6D0CCB for ; Sun, 13 Mar 2016 01:02:34 +0000 (UTC) (envelope-from stephen@missouri.edu) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: =?us-ascii?q?A2AlBQB4uuRW/8aeoM9dgz5SbQa8JRmFd?= =?us-ascii?q?AKBITwQAQEBAQEBAWQnhEIBAQR4ARACAQgYCRYPCQMCAQIBICUCBA0BBwEBF4g?= =?us-ascii?q?JuGQBhAABAQEBBgEBAQEBAQEZilqIdAWXSwGFbZcXjn03K4IDGYFIaoh2AX0BA?= =?us-ascii?q?QE?= X-IPAS-Result: =?us-ascii?q?A2AlBQB4uuRW/8aeoM9dgz5SbQa8JRmFdAKBITwQAQEBAQE?= =?us-ascii?q?BAWQnhEIBAQR4ARACAQgYCRYPCQMCAQIBICUCBA0BBwEBF4gJuGQBhAABAQEBB?= =?us-ascii?q?gEBAQEBAQEZilqIdAWXSwGFbZcXjn03K4IDGYFIaoh2AX0BAQE?= Received: from um-tcas2.um.umsystem.edu ([207.160.158.198]) by mst-rip6-exch-relay.um.umsystem.edu with ESMTP; 12 Mar 2016 19:01:23 -0600 Received: from UM-MBX-N02.um.umsystem.edu ([169.254.5.130]) by UM-TCAS2.um.umsystem.edu ([207.160.158.198]) with mapi id 14.03.0266.001; Sat, 12 Mar 2016 19:01:22 -0600 From: "Montgomery-Smith, Stephen" To: "Julian H. Stacey" CC: "ctm-users@freebsd.org" Subject: Re: Space problems with svn-cur Thread-Topic: Space problems with svn-cur Thread-Index: AQHRfEXQLqoGv/Vz/E+rbmbq4GS10J9W88CA Date: Sun, 13 Mar 2016 01:01:22 +0000 Message-ID: <56E4BBE1.9080007@missouri.edu> References: <201603120959.u2C9x6jU014905@fire.js.berklix.net> In-Reply-To: <201603120959.u2C9x6jU014905@fire.js.berklix.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 x-originating-ip: [207.160.158.194] Content-Type: text/plain; charset="Windows-1252" Content-ID: <3B60EA037F50DA479DBA33BD5D05E050@missouri.edu> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Mar 2016 01:02:35 -0000 On 03/12/2016 03:59 AM, Julian H. Stacey wrote: > "Montgomery-Smith, Stephen" wrote: >> The svn repository is now getting so large that the CTM generating >> computer is running out of disk space when it creates the xEmpty deltas >> for svn-cur. The easist band-aid fix is for me to simply stop >> generating these xEmpty deltas. Since CTM is no longer FreeBSD >> approved, I'm going to assume that no-one new is going to join, and >> hence this is not a problem. >> >> Comments? >=20 > Hi Stephen > There's plenty of space on ctm.berklix.org=20 The issue is that there isn't enough space to create the xEmpty deltas. There is plenty of space to store the deltas. The svn repositories add up to about 35GB. Fully compressed, they shrink to about 4GB. I'll look into other solutions. But I also get a feeling I am putting a lot of effort into something that is only used by a very few people. Stephen From owner-ctm-users@freebsd.org Sun Mar 13 15:53:24 2016 Return-Path: Delivered-To: ctm-users@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 070D1ACF2AF for ; Sun, 13 Mar 2016 15:53:24 +0000 (UTC) (envelope-from stephen@missouri.edu) Received: from mst-rip5-missouri-out.um.umsystem.edu (mst-rip5-missouri-out.um.umsystem.edu [198.209.50.135]) (using TLSv1.2 with cipher RC4-SHA (128/128 bits)) (Client CN "um-tip1.um.umsystem.edu", Issuer "InCommon RSA Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 953479AC for ; Sun, 13 Mar 2016 15:53:23 +0000 (UTC) (envelope-from stephen@missouri.edu) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: =?us-ascii?q?A2ARBQCsi+VW/9CeoM9dgz5SbQa8JhmFd?= =?us-ascii?q?AKBGjwQAQEBAQEBAWQnhEIBAQR4ARACAQgOCgkWDwkDAgECASAlAgQBDAEHAQE?= =?us-ascii?q?XiAm5DAGEAAEBAQcBAQEBAQEBGYlcfoQ4hDwFl0sBhW2XF459NyuCAxmBSGqId?= =?us-ascii?q?gF9AQEB?= X-IPAS-Result: =?us-ascii?q?A2ARBQCsi+VW/9CeoM9dgz5SbQa8JhmFdAKBGjwQAQEBAQE?= =?us-ascii?q?BAWQnhEIBAQR4ARACAQgOCgkWDwkDAgECASAlAgQBDAEHAQEXiAm5DAGEAAEBA?= =?us-ascii?q?QcBAQEBAQEBGYlcfoQ4hDwFl0sBhW2XF459NyuCAxmBSGqIdgF9AQEB?= Received: from um-ncas4.um.umsystem.edu ([207.160.158.208]) by mst-rip5-exch-relay.um.umsystem.edu with ESMTP; 13 Mar 2016 10:53:15 -0500 Received: from UM-MBX-N02.um.umsystem.edu ([169.254.5.130]) by UM-NCAS4.um.umsystem.edu ([207.160.158.208]) with mapi id 14.03.0266.001; Sun, 13 Mar 2016 10:53:15 -0500 From: "Montgomery-Smith, Stephen" To: Roman Kurakin , "Julian H. Stacey" CC: "ctm-users@freebsd.org" Subject: Re: Space problems with svn-cur Thread-Topic: Space problems with svn-cur Thread-Index: AQHRfEXQLqoGv/Vz/E+rbmbq4GS10J9W88CAgABXiYCAAJDkAA== Date: Sun, 13 Mar 2016 15:53:14 +0000 Message-ID: <56E58CEA.1000009@missouri.edu> References: <201603120959.u2C9x6jU014905@fire.js.berklix.net> <56E4BBE1.9080007@missouri.edu> <56E5135F.70302@inse.ru> In-Reply-To: <56E5135F.70302@inse.ru> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 x-originating-ip: [207.160.158.194] Content-Type: text/plain; charset="Windows-1252" Content-ID: <49A835A063D2514C855898D69086957F@missouri.edu> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Mar 2016 15:53:24 -0000 On 03/13/2016 01:14 AM, Roman Kurakin wrote: > Hi, >=20 > On 03/13/2016 04:01 AM, Montgomery-Smith, Stephen wrote: >> On 03/12/2016 03:59 AM, Julian H. Stacey wrote: >>> "Montgomery-Smith, Stephen" wrote: >>>> The svn repository is now getting so large that the CTM generating >>>> computer is running out of disk space when it creates the xEmpty delta= s >>>> for svn-cur. The easist band-aid fix is for me to simply stop >>>> generating these xEmpty deltas. Since CTM is no longer FreeBSD >>>> approved, I'm going to assume that no-one new is going to join, and >>>> hence this is not a problem. >>>> >>>> Comments? >>> Hi Stephen >>> There's plenty of space on ctm.berklix.org >> The issue is that there isn't enough space to create the xEmpty deltas. >> There is plenty of space to store the deltas. >> >> The svn repositories add up to about 35GB. Fully compressed, they >> shrink to about 4GB. >> >> I'll look into other solutions. But I also get a feeling I am putting a >> lot of effort into something that is only used by a very few people. > The xEmpty is the best way to start clean new copy on a new host. > The solution could be a big deltas relative the last xEmpty. This is the problem. Internally the svn xEmpty deltas look like this: CTMTR where the big number is the size of the following huge tar file. What I have been doing is to create the huge tar file in a tmp directory, and then find its size, and then copy this file onto the end of the xEmpty file. This meant that tmp had to be able to take 25GB (the size of the ports repository). Having big deltas relative to the last xEmpty is not a solution, because I still have to store the repository of every 100th delta. That's 25GB whichever way you look at it. What I am trying now is this. I create the tar file twice - once to get the size, and the second time to append it to the xEmpty file. This relies on tar creating the same size file both times. Experiments I recently tried suggest this should be OK. Note the xEmpty file is compressed on the fly, so this file doesn't take a lot of space - only 4GB or so. The other possibility is to ask for more disk space. Stephen= From owner-ctm-users@freebsd.org Mon Mar 14 11:42:52 2016 Return-Path: Delivered-To: ctm-users@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 77F9DAD0F71 for ; Mon, 14 Mar 2016 11:42:52 +0000 (UTC) (envelope-from stephen@missouri.edu) Received: from um-nip3-missouri-out.um.umsystem.edu (um-nip3-missouri-out.um.umsystem.edu [198.209.49.163]) (using TLSv1.2 with cipher RC4-SHA (128/128 bits)) (Client CN "um-tip1.um.umsystem.edu", Issuer "InCommon RSA Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 180093E9 for ; Mon, 14 Mar 2016 11:42:51 +0000 (UTC) (envelope-from stephen@missouri.edu) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: =?us-ascii?q?A2ASBQCtouZW/9KeoM9dg0ZSbQa8KhmFd?= =?us-ascii?q?AKBKDwQAQEBAQEBAWQcC4RCAQEEeBECAQgYCRYPCQMCAQIBICUCBA0IAQEXiAm?= =?us-ascii?q?2dwGEDB6JXH6EOIQ8BZdLAYVtlxeOfTcrggMZgUhqiWUBfQEBAQ?= X-IPAS-Result: =?us-ascii?q?A2ASBQCtouZW/9KeoM9dg0ZSbQa8KhmFdAKBKDwQAQEBAQE?= =?us-ascii?q?BAWQcC4RCAQEEeBECAQgYCRYPCQMCAQIBICUCBA0IAQEXiAm2dwGEDB6JXH6EO?= =?us-ascii?q?IQ8BZdLAYVtlxeOfTcrggMZgUhqiWUBfQEBAQ?= Received: from um-ncas5.um.umsystem.edu ([207.160.158.210]) by um-nip3-exch-relay.um.umsystem.edu with ESMTP; 14 Mar 2016 06:41:40 -0500 Received: from UM-MBX-N02.um.umsystem.edu ([169.254.5.130]) by UM-NCAS5.um.umsystem.edu ([207.160.158.210]) with mapi id 14.03.0266.001; Mon, 14 Mar 2016 06:41:40 -0500 From: "Montgomery-Smith, Stephen" To: "ctm-users@freebsd.org" Subject: Re: Space problems with svn-cur Thread-Topic: Space problems with svn-cur Thread-Index: AQHRfEXQLqoGv/Vz/E+rbmbq4GS10J9W88CAgABXiYCAAJDkAIABTAqA Date: Mon, 14 Mar 2016 11:41:39 +0000 Message-ID: <56E6A373.2040901@missouri.edu> References: <201603120959.u2C9x6jU014905@fire.js.berklix.net> <56E4BBE1.9080007@missouri.edu> <56E5135F.70302@inse.ru> <56E58CEA.1000009@missouri.edu> In-Reply-To: <56E58CEA.1000009@missouri.edu> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 x-originating-ip: [207.160.158.194] Content-Type: text/plain; charset="Windows-1252" Content-ID: <8A86103C7C5B754D86435580B0726546@missouri.edu> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Mar 2016 11:42:52 -0000 On 03/13/2016 10:53 AM, Montgomery-Smith, Stephen wrote: > On 03/13/2016 01:14 AM, Roman Kurakin wrote: >> Hi, >> >> On 03/13/2016 04:01 AM, Montgomery-Smith, Stephen wrote: >>> On 03/12/2016 03:59 AM, Julian H. Stacey wrote: >>>> "Montgomery-Smith, Stephen" wrote: >>>>> The svn repository is now getting so large that the CTM generating >>>>> computer is running out of disk space when it creates the xEmpty delt= as >>>>> for svn-cur. The easist band-aid fix is for me to simply stop >>>>> generating these xEmpty deltas. Since CTM is no longer FreeBSD >>>>> approved, I'm going to assume that no-one new is going to join, and >>>>> hence this is not a problem. >>>>> >>>>> Comments? >>>> Hi Stephen >>>> There's plenty of space on ctm.berklix.org >>> The issue is that there isn't enough space to create the xEmpty deltas. >>> There is plenty of space to store the deltas. >>> >>> The svn repositories add up to about 35GB. Fully compressed, they >>> shrink to about 4GB. >>> >>> I'll look into other solutions. But I also get a feeling I am putting = a >>> lot of effort into something that is only used by a very few people. >> The xEmpty is the best way to start clean new copy on a new host. >> The solution could be a big deltas relative the last xEmpty. >=20 > This is the problem. Internally the svn xEmpty deltas look like this: >=20 > CTMTR > >=20 > where the big number is the size of the following huge tar file. What I > have been doing is to create the huge tar file in a tmp directory, and > then find its size, and then copy this file onto the end of the xEmpty > file. This meant that tmp had to be able to take 25GB (the size of the > ports repository). >=20 > Having big deltas relative to the last xEmpty is not a solution, because > I still have to store the repository of every 100th delta. That's 25GB > whichever way you look at it. >=20 > What I am trying now is this. I create the tar file twice - once to get > the size, and the second time to append it to the xEmpty file. This > relies on tar creating the same size file both times. Experiments I > recently tried suggest this should be OK. This seems to be working. I created an xEmpty for 4406. I also found out it takes about 5 hours to create an xEmpty for svn-cur. Almost all that time is spent by xz doing the compression.