From owner-ctm-users@FreeBSD.ORG Tue Aug 9 13:14:07 2011 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 36D88106566C for ; Tue, 9 Aug 2011 13:14:07 +0000 (UTC) (envelope-from utisoft@gmail.com) Received: from mail-iy0-f172.google.com (mail-iy0-f172.google.com [209.85.210.172]) by mx1.freebsd.org (Postfix) with ESMTP id 08FB88FC13 for ; Tue, 9 Aug 2011 13:14:06 +0000 (UTC) Received: by iye7 with SMTP id 7so297771iye.17 for ; Tue, 09 Aug 2011 06:14:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:from:date:x-google-sender-auth:message-id :subject:to:content-type; bh=BcClqRnHTfJQYPTkF3t13F8isxZr+mlbXrvS2ip3zH8=; b=Vzz4f/mexwuCs0phYhJlinFfBD5z/i8uJwWNhrTOc/nAqVpJcw16ZMoTewnXmHRR75 Ols2X+5Be7Klo672Tb8c3zWOc4wRyVi7Kut8QZuQAZeQmlj+/1X0l0JjrN9+ib7IZCoP Odzs369jW1CG96cpcTa3BwxK1YROMjlVtHW6U= Received: by 10.231.25.217 with SMTP id a25mr7420923ibc.52.1312894176073; Tue, 09 Aug 2011 05:49:36 -0700 (PDT) MIME-Version: 1.0 Sender: utisoft@gmail.com Received: by 10.231.42.169 with HTTP; Tue, 9 Aug 2011 05:49:06 -0700 (PDT) From: Chris Rees Date: Tue, 9 Aug 2011 13:49:06 +0100 X-Google-Sender-Auth: 19Y6KfHs7M7LzKspujNilEV9B4M Message-ID: To: ctm-users@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: SVN repo X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Aug 2011 13:14:07 -0000 Hi Stephen, Have you any plans to start distributing the subversion repo? I keep hearing little hints about the Other Two Trees being moved too... If you need any more hands on it I'll be happy to help. Chris From owner-ctm-users@FreeBSD.ORG Tue Aug 9 16:31:27 2011 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 15EED1065670 for ; Tue, 9 Aug 2011 16:31:27 +0000 (UTC) (envelope-from stephen@missouri.edu) Received: from wilberforce.math.missouri.edu (wilberforce.math.missouri.edu [128.206.184.213]) by mx1.freebsd.org (Postfix) with ESMTP id D5F6A8FC12 for ; Tue, 9 Aug 2011 16:31:26 +0000 (UTC) Received: from [128.206.184.213] (wilberforce.math.missouri.edu [128.206.184.213]) by wilberforce.math.missouri.edu (8.14.4/8.14.4) with ESMTP id p79GH1tD025935; Tue, 9 Aug 2011 11:17:01 -0500 (CDT) (envelope-from stephen@missouri.edu) Message-ID: <4E415D7D.40002@missouri.edu> Date: Tue, 09 Aug 2011 11:17:01 -0500 From: Stephen Montgomery-Smith User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.19) Gecko/20110805 SeaMonkey/2.0.14 MIME-Version: 1.0 To: Chris Rees References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "ctm-users@freebsd.org" Subject: Re: SVN repo X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Aug 2011 16:31:27 -0000 Chris Rees wrote: > Hi Stephen, > > Have you any plans to start distributing the subversion repo? I keep > hearing little hints about the Other Two Trees being moved too... If > you need any more hands on it I'll be happy to help. > > Chris I hadn't thought about it, but it is a good idea. A good start would be for you to tell me some basic commands by which I can fetch the whole subversion tree, and how I can regularly update it. I know almost nothing about svn. (Also on cvs I have access to cvsup-master.freebsd.org, and it would be nice to have the equivalent for svn.) Setting it up so that CTM creates regular updates of the svn repository will then be very straightforward for me, as the infrastructure is essentially all there. Also, if the svn repository is big (as big as the cvs repository), I probably need to warn the mirrors of the big files that will arrive. One other thing - the computer that generates the CTM deltas is now behind the University of Missouri firewall. If I were to be run down by a car tomorrow, I don't know who would be in place to take it over. It probably couldn't be run from the University of Missouri. I hope I have many more years of life in front of me, but I thought I should warn people of the possibility. From owner-ctm-users@FreeBSD.ORG Tue Aug 9 18:04:28 2011 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1F132106564A for ; Tue, 9 Aug 2011 18:04:28 +0000 (UTC) (envelope-from utisoft@gmail.com) Received: from mail-iy0-f172.google.com (mail-iy0-f172.google.com [209.85.210.172]) by mx1.freebsd.org (Postfix) with ESMTP id DFD9D8FC0C for ; Tue, 9 Aug 2011 18:04:27 +0000 (UTC) Received: by iye7 with SMTP id 7so370726iye.17 for ; Tue, 09 Aug 2011 11:04:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=W+aTLTOC+CSSTEl3//ejW/vi9ubqqv034oIDigjagcM=; b=X9uebjAFGbxqX/aM0jt9VMkXa6/Vmor8c9G7N9ZeOMWRftuVNKHQidZqDDUpXV1l4o SicVJJiJa8Rctp8yQMuKaeckzXY+m8weKhCzPPMjrgijxScLPwUaCHUg47MHBafChUrS oLF7A6q1HuG0igqcMw6jbEuPpYU/Nhq3Ln+Os= Received: by 10.231.91.73 with SMTP id l9mr7916250ibm.76.1312913067096; Tue, 09 Aug 2011 11:04:27 -0700 (PDT) MIME-Version: 1.0 Sender: utisoft@gmail.com Received: by 10.231.42.169 with HTTP; Tue, 9 Aug 2011 11:03:57 -0700 (PDT) In-Reply-To: <4E415D7D.40002@missouri.edu> References: <4E415D7D.40002@missouri.edu> From: Chris Rees Date: Tue, 9 Aug 2011 19:03:57 +0100 X-Google-Sender-Auth: f9bkm61gAWAbg8_rzIw_O33yJuU Message-ID: To: Stephen Montgomery-Smith Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: "ctm-users@freebsd.org" Subject: Re: SVN repo X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Aug 2011 18:04:28 -0000 On 9 August 2011 17:17, Stephen Montgomery-Smith wro= te: > Chris Rees wrote: >> >> Hi Stephen, >> >> Have you any plans to start distributing the subversion repo? I keep >> hearing little hints about the Other Two Trees being moved too... If >> you need any more hands on it I'll be happy to help. >> >> Chris > > > I hadn't thought about it, but it is a good idea. > > A good start would be for you to tell me some basic commands by which I c= an > fetch the whole subversion tree, and how I can regularly update it. =A0I = know > almost nothing about svn. =A0(Also on cvs I have access to > cvsup-master.freebsd.org, and it would be nice to have the equivalent for > svn.) > > Setting it up so that CTM creates regular updates of the svn repository w= ill > then be very straightforward for me, as the infrastructure is essentially > all there. > > Also, if the svn repository is big (as big as the cvs repository), I > probably need to warn the mirrors of the big files that will arrive. Hahaha, good one. The svn repo (just for src) is enormous -- the seed file is 1GB alone. > One other thing - the computer that generates the CTM deltas is now behin= d > the University of Missouri firewall. =A0If I were to be run down by a car > tomorrow, I don't know who would be in place to take it over. =A0It proba= bly > couldn't be run from the University of Missouri. =A0I hope I have many mo= re > years of life in front of me, but I thought I should warn people of the > possibility. Aren't you pessimistic? I can think of many people we're pretty screwed without -- it's not an uncommon worry for the project, but far less of a problem for us than for you as an individual :P I'll have a look around to see how feasable this is, but unfortunately since the size issue you've pointed out may be a problem I think we may have to kiss CTM goodbye once cvs goes... Chris From owner-ctm-users@FreeBSD.ORG Tue Aug 9 18:17:31 2011 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 231AF106567B; Tue, 9 Aug 2011 18:17:31 +0000 (UTC) (envelope-from stephen@missouri.edu) Received: from wilberforce.math.missouri.edu (wilberforce.math.missouri.edu [128.206.184.213]) by mx1.freebsd.org (Postfix) with ESMTP id C13758FC29; Tue, 9 Aug 2011 18:17:30 +0000 (UTC) Received: from [127.0.0.1] (wilberforce.math.missouri.edu [128.206.184.213]) by wilberforce.math.missouri.edu (8.14.4/8.14.4) with ESMTP id p79IHT0Q032802; Tue, 9 Aug 2011 13:17:29 -0500 (CDT) (envelope-from stephen@missouri.edu) Message-ID: <4E4179B9.6010808@missouri.edu> Date: Tue, 09 Aug 2011 13:17:29 -0500 From: Stephen Montgomery-Smith User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110617 Thunderbird/3.1.11 MIME-Version: 1.0 To: Chris Rees References: <4E415D7D.40002@missouri.edu> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "ctm-users@freebsd.org" Subject: Re: SVN repo X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Aug 2011 18:17:31 -0000 On 08/09/2011 01:03 PM, Chris Rees wrote: > On 9 August 2011 17:17, Stephen Montgomery-Smith wrote: >> Chris Rees wrote: >>> >>> Hi Stephen, >>> >>> Have you any plans to start distributing the subversion repo? I keep >>> hearing little hints about the Other Two Trees being moved too... If >>> you need any more hands on it I'll be happy to help. >>> >>> Chris >> >> >> I hadn't thought about it, but it is a good idea. >> >> A good start would be for you to tell me some basic commands by which I can >> fetch the whole subversion tree, and how I can regularly update it. I know >> almost nothing about svn. (Also on cvs I have access to >> cvsup-master.freebsd.org, and it would be nice to have the equivalent for >> svn.) >> >> Setting it up so that CTM creates regular updates of the svn repository will >> then be very straightforward for me, as the infrastructure is essentially >> all there. >> >> Also, if the svn repository is big (as big as the cvs repository), I >> probably need to warn the mirrors of the big files that will arrive. > > Hahaha, good one. The svn repo (just for src) is enormous -- the seed > file is 1GB alone. The cvs repos are about 1.2G (compressed). There are about three or four *xEmpty* files in in cvs-cur at any one time. So adding another three or four of these might only double the size of CTM. I think it is doable. Tell me a command sequence for downloading the svn repo, and I can do a few preliminary experiments to see how big it would all be. From owner-ctm-users@FreeBSD.ORG Tue Aug 9 19:08:16 2011 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 34528106564A for ; Tue, 9 Aug 2011 19:08:16 +0000 (UTC) (envelope-from utisoft@gmail.com) Received: from mail-iy0-f172.google.com (mail-iy0-f172.google.com [209.85.210.172]) by mx1.freebsd.org (Postfix) with ESMTP id F30048FC17 for ; Tue, 9 Aug 2011 19:08:15 +0000 (UTC) Received: by iye7 with SMTP id 7so460974iye.17 for ; Tue, 09 Aug 2011 12:08:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=H0ybpblNrYgyUJkxUJGQ9UdK1Ev4yEN109ZfVB+U89E=; b=f7jEyJSJbBtLKDa9SAvIEVQVZwqT+6+QXYMRZ2/8Lkvh+8czjHFckVTRvmd+F+LF90 ZmaxE4LtEwXxEGqlRzn8LiYUw/G6HMHYJUeJlpBJHjYVJpcYFunvo475gznkIKcaRgkV hLkGDOHNUPr5mwTgW+LXoufXV9XGdy9el7tQE= Received: by 10.231.25.217 with SMTP id a25mr7977158ibc.52.1312916895092; Tue, 09 Aug 2011 12:08:15 -0700 (PDT) MIME-Version: 1.0 Sender: utisoft@gmail.com Received: by 10.231.42.169 with HTTP; Tue, 9 Aug 2011 12:07:44 -0700 (PDT) In-Reply-To: <4E4179B9.6010808@missouri.edu> References: <4E415D7D.40002@missouri.edu> <4E4179B9.6010808@missouri.edu> From: Chris Rees Date: Tue, 9 Aug 2011 20:07:44 +0100 X-Google-Sender-Auth: JJOqcXu-mIpxK-P_ykt73jY3Bsk Message-ID: To: Stephen Montgomery-Smith Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: "ctm-users@freebsd.org" Subject: Re: SVN repo X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Aug 2011 19:08:16 -0000 On 9 August 2011 19:17, Stephen Montgomery-Smith wro= te: > On 08/09/2011 01:03 PM, Chris Rees wrote: >> >> On 9 August 2011 17:17, Stephen Montgomery-Smith >> =A0wrote: >>> >>> Chris Rees wrote: >>>> >>>> Hi Stephen, >>>> >>>> Have you any plans to start distributing the subversion repo? I keep >>>> hearing little hints about the Other Two Trees being moved too... If >>>> you need any more hands on it I'll be happy to help. >>>> >>>> Chris >>> >>> >>> I hadn't thought about it, but it is a good idea. >>> >>> A good start would be for you to tell me some basic commands by which I >>> can >>> fetch the whole subversion tree, and how I can regularly update it. =A0= I >>> know >>> almost nothing about svn. =A0(Also on cvs I have access to >>> cvsup-master.freebsd.org, and it would be nice to have the equivalent f= or >>> svn.) >>> >>> Setting it up so that CTM creates regular updates of the svn repository >>> will >>> then be very straightforward for me, as the infrastructure is essential= ly >>> all there. >>> >>> Also, if the svn repository is big (as big as the cvs repository), I >>> probably need to warn the mirrors of the big files that will arrive. >> >> Hahaha, good one. The svn repo (just for src) is enormous -- the seed >> file is 1GB alone. > > The cvs repos are about 1.2G (compressed). =A0There are about three or fo= ur > *xEmpty* files in in cvs-cur at any one time. =A0So adding another three = or > four of these might only double the size of CTM. =A0I think it is doable. > > Tell me a command sequence for downloading the svn repo, and I can do a f= ew > preliminary experiments to see how big it would all be. Although I brought this up, I'm not technically a svn user -- just interested in future compatibility. I'm signing off for today now I'm afraid, but the instructions are at [1], I'll get a command sequence tomorrow if it's complicated :) Chris [1] http://wiki.freebsd.org/SubversionPrimer#head-a9bf2ef502d986092afa9606a= 2619735b87ded60 From owner-ctm-users@FreeBSD.ORG Tue Aug 9 19:22:13 2011 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6A434106566C; Tue, 9 Aug 2011 19:22:13 +0000 (UTC) (envelope-from stephen@missouri.edu) Received: from wilberforce.math.missouri.edu (wilberforce.math.missouri.edu [128.206.184.213]) by mx1.freebsd.org (Postfix) with ESMTP id 1ADC98FC15; Tue, 9 Aug 2011 19:22:12 +0000 (UTC) Received: from [127.0.0.1] (wilberforce.math.missouri.edu [128.206.184.213]) by wilberforce.math.missouri.edu (8.14.4/8.14.4) with ESMTP id p79JMBaW036547; Tue, 9 Aug 2011 14:22:11 -0500 (CDT) (envelope-from stephen@missouri.edu) Message-ID: <4E4188E3.4050007@missouri.edu> Date: Tue, 09 Aug 2011 14:22:11 -0500 From: Stephen Montgomery-Smith User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110617 Thunderbird/3.1.11 MIME-Version: 1.0 To: Chris Rees References: <4E415D7D.40002@missouri.edu> <4E4179B9.6010808@missouri.edu> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "ctm-users@freebsd.org" Subject: Re: SVN repo X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Aug 2011 19:22:13 -0000 On 08/09/2011 02:07 PM, Chris Rees wrote: > On 9 August 2011 19:17, Stephen Montgomery-Smith wrote: >> On 08/09/2011 01:03 PM, Chris Rees wrote: >>> >>> On 9 August 2011 17:17, Stephen Montgomery-Smith >>> wrote: >>>> >>>> Chris Rees wrote: >>>>> >>>>> Hi Stephen, >>>>> >>>>> Have you any plans to start distributing the subversion repo? I keep >>>>> hearing little hints about the Other Two Trees being moved too... If >>>>> you need any more hands on it I'll be happy to help. >>>>> >>>>> Chris >>>> >>>> >>>> I hadn't thought about it, but it is a good idea. >>>> >>>> A good start would be for you to tell me some basic commands by which I >>>> can >>>> fetch the whole subversion tree, and how I can regularly update it. I >>>> know >>>> almost nothing about svn. (Also on cvs I have access to >>>> cvsup-master.freebsd.org, and it would be nice to have the equivalent for >>>> svn.) >>>> >>>> Setting it up so that CTM creates regular updates of the svn repository >>>> will >>>> then be very straightforward for me, as the infrastructure is essentially >>>> all there. >>>> >>>> Also, if the svn repository is big (as big as the cvs repository), I >>>> probably need to warn the mirrors of the big files that will arrive. >>> >>> Hahaha, good one. The svn repo (just for src) is enormous -- the seed >>> file is 1GB alone. >> >> The cvs repos are about 1.2G (compressed). There are about three or four >> *xEmpty* files in in cvs-cur at any one time. So adding another three or >> four of these might only double the size of CTM. I think it is doable. >> >> Tell me a command sequence for downloading the svn repo, and I can do a few >> preliminary experiments to see how big it would all be. > > Although I brought this up, I'm not technically a svn user -- just > interested in future compatibility. > > I'm signing off for today now I'm afraid, but the instructions are at > [1], I'll get a command sequence tomorrow if it's complicated :) > > Chris > > [1] http://wiki.freebsd.org/SubversionPrimer#head-a9bf2ef502d986092afa9606a2619735b87ded60 Those instructions will do nicely. I will do some preliminary trials, and see how well it fits with CTM. One thing I see is that svn stores a lot of stuff in databases. I wonder how well this will work with "diff" which is how CTM creates its deltas. I presume that two databases with exactly the same contents may be represented by completely different files, because the data won't be stored in a linear fashion, but according to index files whose exact structure might be random (depending upon the order in which items are placed into the database). As you say, it may well be incompatible with CTM. Stephen From owner-ctm-users@FreeBSD.ORG Wed Aug 10 03:06:08 2011 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C18EE106566C for ; Wed, 10 Aug 2011 03:06:08 +0000 (UTC) (envelope-from stephen@missouri.edu) Received: from wilberforce.math.missouri.edu (wilberforce.math.missouri.edu [128.206.184.213]) by mx1.freebsd.org (Postfix) with ESMTP id 8D3B58FC1A for ; Wed, 10 Aug 2011 03:06:08 +0000 (UTC) Received: from [127.0.0.1] (wilberforce.math.missouri.edu [128.206.184.213]) by wilberforce.math.missouri.edu (8.14.4/8.14.4) with ESMTP id p7A367Ae063536 for ; Tue, 9 Aug 2011 22:06:07 -0500 (CDT) (envelope-from stephen@missouri.edu) Message-ID: <4E41F59F.9020201@missouri.edu> Date: Tue, 09 Aug 2011 22:06:07 -0500 From: Stephen Montgomery-Smith User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110617 Thunderbird/3.1.11 MIME-Version: 1.0 To: "ctm-users@freebsd.org" Content-Type: multipart/mixed; boundary="------------090306070104000700080905" Subject: Use of xz instead of gzip X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 03:06:08 -0000 This is a multi-part message in MIME format. --------------090306070104000700080905 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit It seems that xz has much better compression than gzip. How do people feel about moving CTM to use xz instead of gzip? This will give about a 40% reduction in the size of the CTM files. In particular, if I start putting SVN on CTM, this will save about 3G. It requires two things: First, the attached patch needs to be applied to /usr/src. (Any committers willing to commit it? It needs to MFC'ed rather quickly if I start using xz.) Secondly, if your version of FreeBSD is old (like 7.x or older) you need to install the archivers/xz port. If you don't, when you attempt to apply ctm to a "xz" compressed file, it replies with a rather helpful message "xz not found." Stephen --------------090306070104000700080905 Content-Type: text/plain; name="ddd" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="ddd" diff -ur ~/FreeBSD-8/src/usr.sbin/ctm/ctm/ctm.c usr.sbin/ctm/ctm/ctm.c --- /home/stephen/FreeBSD-8/src/usr.sbin/ctm/ctm/ctm.c 2011-08-09 16:38:22.000000000 -0500 +++ usr.sbin/ctm/ctm/ctm.c 2011-08-09 21:11:31.000000000 -0500 @@ -211,6 +211,12 @@ strcat(p,filename); f = popen(p,"r"); if(!f) { warn("%s", p); return Exit_Garbage; } + } else if(p && !strcmp(p,".xz")) { + p = alloca(20 + strlen(filename)); + strcpy(p,"xz -dc < "); + strcat(p,filename); + f = popen(p,"r"); + if(!f) { warn("%s", p); return Exit_Garbage; } } else { p = 0; f = fopen(filename,"r"); --------------090306070104000700080905-- From owner-ctm-users@FreeBSD.ORG Wed Aug 10 04:16:16 2011 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 74B68106564A; Wed, 10 Aug 2011 04:16:16 +0000 (UTC) (envelope-from stephen@missouri.edu) Received: from wilberforce.math.missouri.edu (wilberforce.math.missouri.edu [128.206.184.213]) by mx1.freebsd.org (Postfix) with ESMTP id 3E97C8FC16; Wed, 10 Aug 2011 04:16:15 +0000 (UTC) Received: from [127.0.0.1] (wilberforce.math.missouri.edu [128.206.184.213]) by wilberforce.math.missouri.edu (8.14.4/8.14.4) with ESMTP id p7A4GEDE067814; Tue, 9 Aug 2011 23:16:15 -0500 (CDT) (envelope-from stephen@missouri.edu) Message-ID: <4E42060E.4020808@missouri.edu> Date: Tue, 09 Aug 2011 23:16:14 -0500 From: Stephen Montgomery-Smith User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110617 Thunderbird/3.1.11 MIME-Version: 1.0 To: Chris Rees References: <4E415D7D.40002@missouri.edu> <4E4179B9.6010808@missouri.edu> <4E4188E3.4050007@missouri.edu> In-Reply-To: <4E4188E3.4050007@missouri.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "ctm-users@freebsd.org" Subject: rep-cache.db: was SVN repo X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 04:16:16 -0000 One problem with putting SVN on CTM seems to be the file base/db/repo-cache.db. I have no idea what this file is for, or even if it is necessary. I have done a lot of googling, and I'm not finding much useful information. This is a huge file (about 85M). It is not a text file, so every time it is changed, the next CTM delta contains the entirety of this file. With compression, this will make each delta about 40M. Does anyone know what this file is good for? By the way, updating the src CTM's should be a breeze using the svn depository. From owner-ctm-users@FreeBSD.ORG Wed Aug 10 05:48:22 2011 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D987A1065675 for ; Wed, 10 Aug 2011 05:48:22 +0000 (UTC) (envelope-from peter.jeremy@alcatel-lucent.com) Received: from ihemail3.lucent.com (ihemail3.lucent.com [135.245.0.37]) by mx1.freebsd.org (Postfix) with ESMTP id A00B98FC22 for ; Wed, 10 Aug 2011 05:48:22 +0000 (UTC) Received: from usnavsmail4.ndc.alcatel-lucent.com (usnavsmail4.ndc.alcatel-lucent.com [135.3.39.12]) by ihemail3.lucent.com (8.13.8/IER-o) with ESMTP id p7A5Wb1d023340 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK) for ; Wed, 10 Aug 2011 00:32:37 -0500 (CDT) Received: from unixmail.au.alcatel-lucent.com (unixmail.au.alcatel-lucent.com [139.188.42.130]) by usnavsmail4.ndc.alcatel-lucent.com (8.14.3/8.14.3/GMO) with ESMTP id p7A5WWMs015634 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Wed, 10 Aug 2011 00:32:36 -0500 Received: from insmb.au.alcatel-lucent.com (insmb.au.alcatel-lucent.com [139.188.42.184]) by unixmail.au.alcatel-lucent.com (8.13.8+Sun/8.13.3) with ESMTP id p7A5WWEM021566 for ; Wed, 10 Aug 2011 15:32:32 +1000 (EST) Received: from pjdesk.au.alcatel-lucent.com (pjdesk.au.alcatel-lucent.com [139.188.2.2]) by insmb.au.alcatel-lucent.com (8.13.8+Sun/8.13.8) with ESMTP id p7A5Owbt028676; Wed, 10 Aug 2011 15:24:59 +1000 (EST) X-Bogosity: Ham, spamicity=0.000000 Received: from pjdesk.au.alcatel-lucent.com (localhost [127.0.0.1]) by pjdesk.au.alcatel-lucent.com (8.14.4/8.14.4) with ESMTP id p7A5Oj9x047351; Wed, 10 Aug 2011 15:24:45 +1000 (EST) (envelope-from peter.jeremy@alcatel-lucent.com) Received: (from pjeremy@localhost) by pjdesk.au.alcatel-lucent.com (8.14.4/8.14.4/Submit) id p7A5OipY047350; Wed, 10 Aug 2011 15:24:44 +1000 (EST) (envelope-from peter.jeremy@alcatel-lucent.com) Date: Wed, 10 Aug 2011 15:24:44 +1000 From: Peter Jeremy To: Stephen Montgomery-Smith Message-ID: <20110810052444.GM6142@pjdesk.au.alcatel-lucent.com> References: <4E41F59F.9020201@missouri.edu> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="zS7rBR6csb6tI2e1" Content-Disposition: inline In-Reply-To: <4E41F59F.9020201@missouri.edu> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.21 (2010-09-15) X-Scanned-By: MIMEDefang 2.57 on 135.245.2.37 X-Scanned-By: MIMEDefang 2.64 on 135.3.39.12 Cc: "ctm-users@freebsd.org" Subject: Re: Use of xz instead of gzip X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 05:48:23 -0000 --zS7rBR6csb6tI2e1 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2011-Aug-10 11:06:07 +0800, Stephen Montgomery-Smith wrote: >It seems that xz has much better compression than gzip. How do people=20 >feel about moving CTM to use xz instead of gzip? Seems like a reasonable idea except for the 7.x issues. Have you considered bzip2? >First, the attached patch needs to be applied to /usr/src. (Any=20 >committers willing to commit it? It needs to MFC'ed rather quickly if I= =20 >start using xz.) You're unlikely to find any here apart from yourself. >Secondly, if your version of FreeBSD is old (like 7.x or older) you need= =20 >to install the archivers/xz port. If you don't, when you attempt to=20 >apply ctm to a "xz" compressed file, it replies with a rather helpful=20 >message "xz not found." I can see this presenting a problem on 7.x (and earlier, but they are no longer supported) because the base system shouldn't have dependencies on ports. The options would seem to be: 1) Stick with gzip for 7-src and earlier (as well as ports until 7.x goes EOL) 2) Make ctm a port (so it can have a dependency on xz) 3) If dispensation is granted, ensure the dependency is documented in UPDATING and ctm.1 when it is MFC'd to 7.x and before. And, of course, the commits containing the patch need to be distributed before the format is changed (which creates issues for the xEmpty files because they need to work on -RELEASE systems). --=20 Peter Jeremy --zS7rBR6csb6tI2e1 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (FreeBSD) iEYEARECAAYFAk5CFhwACgkQ/opHv/APuId96ACgpGiTajuZ2Jfo/mM0N6WC8kWt MCsAnjzZRQhNcT1zBE2sj//V35NPqEsk =ulq0 -----END PGP SIGNATURE----- --zS7rBR6csb6tI2e1-- From owner-ctm-users@FreeBSD.ORG Wed Aug 10 12:52:50 2011 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 294551065687 for ; Wed, 10 Aug 2011 12:52:50 +0000 (UTC) (envelope-from jhs@berklix.com) Received: from tower.berklix.org (tower.berklix.org [83.236.223.114]) by mx1.freebsd.org (Postfix) with ESMTP id 6C6538FC1C for ; Wed, 10 Aug 2011 12:52:49 +0000 (UTC) Received: from park.js.berklix.net (p5DCBD8E2.dip.t-dialin.net [93.203.216.226]) (authenticated bits=0) by tower.berklix.org (8.14.2/8.14.2) with ESMTP id p7ACRd2r068785; Wed, 10 Aug 2011 12:27:40 GMT (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (fire.js.berklix.net [192.168.91.41]) by park.js.berklix.net (8.13.8/8.13.8) with ESMTP id p7ACRU1G034376; Wed, 10 Aug 2011 14:27:30 +0200 (CEST) (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (localhost [127.0.0.1]) by fire.js.berklix.net (8.14.4/8.14.4) with ESMTP id p7ACREGJ076842; Wed, 10 Aug 2011 12:27:20 GMT (envelope-from jhs@fire.js.berklix.net) Message-Id: <201108101227.p7ACREGJ076842@fire.js.berklix.net> To: Peter Jeremy From: "Julian H. Stacey" Organization: http://www.berklix.com BSD Unix Linux Consultancy, Munich Germany User-agent: EXMH on FreeBSD http://www.berklix.com/free/ X-URL: http://www.berklix.com In-reply-to: Your message "Wed, 10 Aug 2011 15:24:44 +1000." <20110810052444.GM6142@pjdesk.au.alcatel-lucent.com> Date: Wed, 10 Aug 2011 14:27:14 +0200 Sender: jhs@berklix.com Cc: "ctm-users@freebsd.org" Subject: Re: Use of xz instead of gzip X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 12:52:50 -0000 > I can see this presenting a problem on 7.x (and earlier, but they are > no longer supported) because the base system shouldn't have dependencies > on ports. The options would seem to be: > 1) Stick with gzip for 7-src and earlier (as well as ports until 7.x > goes EOL) (Not that I'm suggesting for or against Option 1, but it needs modifying, Applying the same logic, append to Option 1: "as well as cvs-cur" (as cvs can be used to extract eg 4 & ports & doc as well as head). As the ctm collection comprises: cvs-cur ports-cur src-4 src-5 src-6 src-7 src-8 src-cur Option 1 thus becomes "Stick with gzip for all except src-8 & src-cur" Whether a CTM src-stable stream is EOL is irrelevant. All that matters is whether extractor hosts have xz or not. One does not need to run ctm on same `uname -r` as one extracts for, eg my CTM extractor runs on 8.3, but I extract for all of cvs-cur ports-cur src-4 src-5 src-6 src-7 src-8 src-cur Some (4, maybe 6) have long been EOL, but I use them for reasons beyond remit of this thread. > 2) Make ctm a port (so it can have a dependency on xz) I cant imagine many CTM users saying "please kick CTM out of src/" ;-) > 3) If dispensation is granted, Dispensation for what ? > ensure the dependency is documented in > UPDATING and ctm.1 when it is MFC'd to 7.x and before. Man ctm says: "The ctm command appeared in FreeBSD 2.1" In theory someone could be extracting on a really old host ? We might be lucky & find all people are extracting on 8.x ? We won't know unless we ask, Stephen could request on ctm-announce@ eg: "People extracting CTM on old systems are requested to run uname -r & if not 8.something-RELEASE or current, report to ....@.... Personaly, I'd think an acceptable middle course could be to switch to whatever reduces the CTM bandwidth the most & put a note in man ctm: "If running FreeBSD-x or earlier you need to cd /usr/ports/archivers/xz ; make install" Cheers, Julian -- Julian Stacey, BSD Unix Linux C Sys Eng Consultants Munich http://berklix.com Reply below, not above; Indent with "> "; Cumulative like a play script. Format: Plain text. Not HTML, multipart/alternative, base64, quoted-printable. From owner-ctm-users@FreeBSD.ORG Wed Aug 10 13:17:02 2011 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 415AA106566B for ; Wed, 10 Aug 2011 13:17:02 +0000 (UTC) (envelope-from utisoft@gmail.com) Received: from mail-yw0-f54.google.com (mail-yw0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id F184A8FC12 for ; Wed, 10 Aug 2011 13:17:01 +0000 (UTC) Received: by ywm39 with SMTP id 39so752087ywm.13 for ; Wed, 10 Aug 2011 06:17:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=VkYCrcdZSmZGOj7MUO0SnrkWXb1oE2gza3tXEe9q12Q=; b=higZaLYlwDDAtIVgK8umRddzouXqtPcaFdQcyff4n4L85I415IpZQhRXWoeHVP26QE 3DC1CMrZGacYh3k5wnyJ9OZcnXfCUGUdC6Oo2ieM8enTbcXzM5krCZdYczZT6mOzwfPO CIWpd2M4CJKd7MYc8CiFbulw5/sfVwcU9ZrhQ= MIME-Version: 1.0 Received: by 10.43.134.133 with SMTP id ic5mr8418415icc.293.1312982221265; Wed, 10 Aug 2011 06:17:01 -0700 (PDT) Received: by 10.231.42.169 with HTTP; Wed, 10 Aug 2011 06:17:01 -0700 (PDT) Received: by 10.231.42.169 with HTTP; Wed, 10 Aug 2011 06:17:01 -0700 (PDT) In-Reply-To: <201108101227.p7ACREGJ076842@fire.js.berklix.net> References: <20110810052444.GM6142@pjdesk.au.alcatel-lucent.com> <201108101227.p7ACREGJ076842@fire.js.berklix.net> Date: Wed, 10 Aug 2011 14:17:01 +0100 Message-ID: From: Chris Rees To: "Julian H. Stacey" Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: "ctm-users@freebsd.org" Subject: Re: Use of xz instead of gzip X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 13:17:02 -0000 On 10 Aug 2011 13:53, "Julian H. Stacey" wrote: > > > I can see this presenting a problem on 7.x (and earlier, but they are > > no longer supported) because the base system shouldn't have dependencies > > on ports. The options would seem to be: > > 1) Stick with gzip for 7-src and earlier (as well as ports until 7.x > > goes EOL) > > (Not that I'm suggesting for or against Option 1, but it needs modifying, > Applying the same logic, append to Option 1: "as well as cvs-cur" > (as cvs can be used to extract eg 4 & ports & doc as well as head). > As the ctm collection comprises: > cvs-cur ports-cur src-4 src-5 src-6 src-7 src-8 src-cur > Option 1 thus becomes "Stick with gzip for all except src-8 & src-cur" > > Whether a CTM src-stable stream is EOL is irrelevant. > All that matters is whether extractor hosts have xz or not. > One does not need to run ctm on same `uname -r` as one extracts for, > eg my CTM extractor runs on 8.3, but I extract for all of > cvs-cur ports-cur src-4 src-5 src-6 src-7 src-8 src-cur > Some (4, maybe 6) have long been EOL, but I use them for reasons beyond > remit of this thread. > > > > 2) Make ctm a port (so it can have a dependency on xz) > > I cant imagine many CTM users saying "please kick CTM out of src/" ;-) > > > > 3) If dispensation is granted, > > Dispensation for what ? > > > > ensure the dependency is documented in > > UPDATING and ctm.1 when it is MFC'd to 7.x and before. > > Man ctm says: "The ctm command appeared in FreeBSD 2.1" > In theory someone could be extracting on a really old host ? > We might be lucky & find all people are extracting on 8.x ? > We won't know unless we ask, > Stephen could request on ctm-announce@ eg: > "People extracting CTM on old systems are requested to run > uname -r > & if not 8.something-RELEASE or current, report to ....@.... > > Personaly, I'd think an acceptable middle course could be to switch to > whatever reduces the CTM bandwidth the most & put a note in man ctm: > "If running FreeBSD-x or earlier you need to > cd /usr/ports/archivers/xz ; make install" > How does a manpage entry help? It'll only appear on later versions, where It's irrelevant, and not appear on earlier versions, where it would help! Perhaps a note could be put into the Handbook entry? Chris From owner-ctm-users@FreeBSD.ORG Wed Aug 10 14:32:03 2011 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0AC961065825 for ; Wed, 10 Aug 2011 14:32:03 +0000 (UTC) (envelope-from stephen@missouri.edu) Received: from wilberforce.math.missouri.edu (wilberforce.math.missouri.edu [128.206.184.213]) by mx1.freebsd.org (Postfix) with ESMTP id CA0478FC16 for ; Wed, 10 Aug 2011 14:32:02 +0000 (UTC) Received: from [127.0.0.1] (wilberforce.math.missouri.edu [128.206.184.213]) by wilberforce.math.missouri.edu (8.14.4/8.14.4) with ESMTP id p7AEVxLL004092; Wed, 10 Aug 2011 09:31:59 -0500 (CDT) (envelope-from stephen@missouri.edu) Message-ID: <4E42965F.60501@missouri.edu> Date: Wed, 10 Aug 2011 09:31:59 -0500 From: Stephen Montgomery-Smith User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110617 Thunderbird/3.1.11 MIME-Version: 1.0 To: Peter Jeremy References: <4E41F59F.9020201@missouri.edu> <20110810052444.GM6142@pjdesk.au.alcatel-lucent.com> In-Reply-To: <20110810052444.GM6142@pjdesk.au.alcatel-lucent.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "ctm-users@freebsd.org" Subject: Re: Use of xz instead of gzip X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 14:32:03 -0000 On 08/10/2011 12:24 AM, Peter Jeremy wrote: > On 2011-Aug-10 11:06:07 +0800, Stephen Montgomery-Smith wrote: >> It seems that xz has much better compression than gzip. How do people >> feel about moving CTM to use xz instead of gzip? > > Seems like a reasonable idea except for the 7.x issues. Have you > considered bzip2? I created a .0001 delta for svn. It is about 5G. When compressed with gzip, it goes down to 1.8G. When compressed with bzip2, it goes down to 1.4G. When compressed with xz, it goes down to 1G. These are some huge differences in my opinion. >> First, the attached patch needs to be applied to /usr/src. (Any >> committers willing to commit it? It needs to MFC'ed rather quickly if I >> start using xz.) > > You're unlikely to find any here apart from yourself. I am merely a ports committer. And now that I have been in this "exclusive club" for a few weeks, it bestows far less power and romance than an outsider might imagine. >> Secondly, if your version of FreeBSD is old (like 7.x or older) you need >> to install the archivers/xz port. If you don't, when you attempt to >> apply ctm to a "xz" compressed file, it replies with a rather helpful >> message "xz not found." > > I can see this presenting a problem on 7.x (and earlier, but they are > no longer supported) because the base system shouldn't have dependencies > on ports. The options would seem to be: > 1) Stick with gzip for 7-src and earlier (as well as ports until 7.x > goes EOL) By far most of CTM space is taken by the cvs-cur xEmpty files. And these are agnostic to which version of FreeBSD is being used. Stephen From owner-ctm-users@FreeBSD.ORG Wed Aug 10 14:38:12 2011 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 593FE106564A; Wed, 10 Aug 2011 14:38:12 +0000 (UTC) (envelope-from stephen@missouri.edu) Received: from wilberforce.math.missouri.edu (wilberforce.math.missouri.edu [128.206.184.213]) by mx1.freebsd.org (Postfix) with ESMTP id 1FA7D8FC0C; Wed, 10 Aug 2011 14:38:11 +0000 (UTC) Received: from [127.0.0.1] (wilberforce.math.missouri.edu [128.206.184.213]) by wilberforce.math.missouri.edu (8.14.4/8.14.4) with ESMTP id p7AEcBSF004488; Wed, 10 Aug 2011 09:38:11 -0500 (CDT) (envelope-from stephen@missouri.edu) Message-ID: <4E4297D3.5020108@missouri.edu> Date: Wed, 10 Aug 2011 09:38:11 -0500 From: Stephen Montgomery-Smith User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110617 Thunderbird/3.1.11 MIME-Version: 1.0 To: Chris Rees References: <4E415D7D.40002@missouri.edu> <4E4179B9.6010808@missouri.edu> <4E4188E3.4050007@missouri.edu> <4E42060E.4020808@missouri.edu> In-Reply-To: <4E42060E.4020808@missouri.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "ctm-users@freebsd.org" Subject: Re: rep-cache.db: was SVN repo X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 14:38:12 -0000 On 08/09/2011 11:16 PM, Stephen Montgomery-Smith wrote: > One problem with putting SVN on CTM seems to be the file > base/db/repo-cache.db. > > I have no idea what this file is for, or even if it is necessary. I > have done a lot of googling, and I'm not finding much useful information. > > This is a huge file (about 85M). It is not a text file, so every time > it is changed, the next CTM delta contains the entirety of this file. > With compression, this will make each delta about 40M. > > Does anyone know what this file is good for? I tried deleting base/db/repo-cache.db. It seems to be rather important, in that "svn update" didn't work when svn-ing from this repository. I am going to conclude, at least for now, that svn on CTM is impractical. Or at least that CTM will need some complete rethinking. If FreeBSD were to switch to svn, and stop supporting cvs, how many people would be negatively impacted by not having svn on CTM? Stephen From owner-ctm-users@FreeBSD.ORG Wed Aug 10 14:43:40 2011 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E20E41065670 for ; Wed, 10 Aug 2011 14:43:40 +0000 (UTC) (envelope-from stephen@missouri.edu) Received: from wilberforce.math.missouri.edu (wilberforce.math.missouri.edu [128.206.184.213]) by mx1.freebsd.org (Postfix) with ESMTP id ABDBB8FC1D for ; Wed, 10 Aug 2011 14:43:40 +0000 (UTC) Received: from [127.0.0.1] (wilberforce.math.missouri.edu [128.206.184.213]) by wilberforce.math.missouri.edu (8.14.4/8.14.4) with ESMTP id p7AEhUcT004797; Wed, 10 Aug 2011 09:43:30 -0500 (CDT) (envelope-from stephen@missouri.edu) Message-ID: <4E429912.7090900@missouri.edu> Date: Wed, 10 Aug 2011 09:43:30 -0500 From: Stephen Montgomery-Smith User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110617 Thunderbird/3.1.11 MIME-Version: 1.0 To: Chris Rees References: <20110810052444.GM6142@pjdesk.au.alcatel-lucent.com> <201108101227.p7ACREGJ076842@fire.js.berklix.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "ctm-users@freebsd.org" , "Julian H. Stacey" Subject: Re: Use of xz instead of gzip X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 14:43:41 -0000 On 08/10/2011 08:17 AM, Chris Rees wrote: > On 10 Aug 2011 13:53, "Julian H. Stacey" wrote: >> Personaly, I'd think an acceptable middle course could be to switch to >> whatever reduces the CTM bandwidth the most& put a note in man ctm: >> "If running FreeBSD-x or earlier you need to >> cd /usr/ports/archivers/xz ; make install" >> > > How does a manpage entry help? It'll only appear on later versions, where > It's irrelevant, and not appear on earlier versions, where it would help! > Perhaps a note could be put into the Handbook entry? How about a much more helpful message from ctm if xz isn't available? One that gives the message that Julian was considering putting into man ctm? (By the way, if ctm.c gets MFC'ed to earlier versions, so will ctm.1. The problem will be created only if xz is not MFC'ed to earlier versions.) Stephen From owner-ctm-users@FreeBSD.ORG Wed Aug 10 14:51:06 2011 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DB8AA106566C for ; Wed, 10 Aug 2011 14:51:06 +0000 (UTC) (envelope-from jhs@berklix.com) Received: from tower.berklix.org (tower.berklix.org [83.236.223.114]) by mx1.freebsd.org (Postfix) with ESMTP id 6CCAE8FC0C for ; Wed, 10 Aug 2011 14:51:06 +0000 (UTC) Received: from park.js.berklix.net (p5DCBD8E2.dip.t-dialin.net [93.203.216.226]) (authenticated bits=0) by tower.berklix.org (8.14.2/8.14.2) with ESMTP id p7AEp4Vw069807; Wed, 10 Aug 2011 14:51:05 GMT (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (fire.js.berklix.net [192.168.91.41]) by park.js.berklix.net (8.13.8/8.13.8) with ESMTP id p7AEouEb034982; Wed, 10 Aug 2011 16:50:56 +0200 (CEST) (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (localhost [127.0.0.1]) by fire.js.berklix.net (8.14.4/8.14.4) with ESMTP id p7AEofmu081200; Wed, 10 Aug 2011 14:50:46 GMT (envelope-from jhs@fire.js.berklix.net) Message-Id: <201108101450.p7AEofmu081200@fire.js.berklix.net> To: Stephen Montgomery-Smith From: "Julian H. Stacey" Organization: http://www.berklix.com BSD Unix Linux Consultancy, Munich Germany User-agent: EXMH on FreeBSD http://www.berklix.com/free/ X-URL: http://www.berklix.com In-reply-to: Your message "Wed, 10 Aug 2011 09:43:30 CDT." <4E429912.7090900@missouri.edu> Date: Wed, 10 Aug 2011 16:50:41 +0200 Sender: jhs@berklix.com Cc: "ctm-users@freebsd.org" , Chris Rees Subject: Re: Use of xz instead of gzip X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 14:51:06 -0000 > How about a much more helpful message from ctm if xz isn't available? > One that gives the message that Julian was considering putting into man ctm? Both would be best :-) Run time message most importantly. A note in man ctm also nice :-) Cheers, Julian -- Julian Stacey, BSD Unix Linux C Sys Eng Consultants Munich http://berklix.com Reply below, not above; Indent with "> "; Cumulative like a play script. Format: Plain text. Not HTML, multipart/alternative, base64, quoted-printable. From owner-ctm-users@FreeBSD.ORG Wed Aug 10 15:03:41 2011 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 07A991065670 for ; Wed, 10 Aug 2011 15:03:41 +0000 (UTC) (envelope-from jhs@berklix.com) Received: from tower.berklix.org (tower.berklix.org [83.236.223.114]) by mx1.freebsd.org (Postfix) with ESMTP id 7D6D58FC1B for ; Wed, 10 Aug 2011 15:03:39 +0000 (UTC) Received: from park.js.berklix.net (p5DCBD8E2.dip.t-dialin.net [93.203.216.226]) (authenticated bits=0) by tower.berklix.org (8.14.2/8.14.2) with ESMTP id p7AF3Xd0069912; Wed, 10 Aug 2011 15:03:33 GMT (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (fire.js.berklix.net [192.168.91.41]) by park.js.berklix.net (8.13.8/8.13.8) with ESMTP id p7AF3Ptq035027; Wed, 10 Aug 2011 17:03:25 +0200 (CEST) (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (localhost [127.0.0.1]) by fire.js.berklix.net (8.14.4/8.14.4) with ESMTP id p7AF3A5L081336; Wed, 10 Aug 2011 15:03:15 GMT (envelope-from jhs@fire.js.berklix.net) Message-Id: <201108101503.p7AF3A5L081336@fire.js.berklix.net> To: Chris Rees From: "Julian H. Stacey" Organization: http://www.berklix.com BSD Unix Linux Consultancy, Munich Germany User-agent: EXMH on FreeBSD http://www.berklix.com/free/ X-URL: http://www.berklix.com In-reply-to: Your message "Wed, 10 Aug 2011 14:17:01 BST." Date: Wed, 10 Aug 2011 17:03:10 +0200 Sender: jhs@berklix.com Cc: "ctm-users@freebsd.org" Subject: Re: Use of xz instead of gzip X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 15:03:41 -0000 Hi, Reference: > From: Chris Rees > Date: Wed, 10 Aug 2011 14:17:01 +0100 > Message-id: Chris Rees wrote: > --20cf307f3aaee51c1704aa267e44 > Content-Type: text/plain; charset=ISO-8859-1 > > On 10 Aug 2011 13:53, "Julian H. Stacey" wrote: > > > > > I can see this presenting a problem on 7.x (and earlier, but they are > > > no longer supported) because the base system shouldn't have dependencies > > > on ports. The options would seem to be: > > > 1) Stick with gzip for 7-src and earlier (as well as ports until 7.x > > > goes EOL) > > > > (Not that I'm suggesting for or against Option 1, but it needs modifying, > > Applying the same logic, append to Option 1: "as well as cvs-cur" > > (as cvs can be used to extract eg 4 & ports & doc as well as head). > > As the ctm collection comprises: > > cvs-cur ports-cur src-4 src-5 src-6 src-7 src-8 src-cur > > Option 1 thus becomes "Stick with gzip for all except src-8 & src-cur" > > > > Whether a CTM src-stable stream is EOL is irrelevant. > > All that matters is whether extractor hosts have xz or not. > > One does not need to run ctm on same `uname -r` as one extracts for, > > eg my CTM extractor runs on 8.3, but I extract for all of > > cvs-cur ports-cur src-4 src-5 src-6 src-7 src-8 src-cur > > Some (4, maybe 6) have long been EOL, but I use them for reasons beyond > > remit of this thread. > > > > > > > 2) Make ctm a port (so it can have a dependency on xz) > > > > I cant imagine many CTM users saying "please kick CTM out of src/" ;-) > > > > > > > 3) If dispensation is granted, > > > > Dispensation for what ? > > > > > > > ensure the dependency is documented in > > > UPDATING and ctm.1 when it is MFC'd to 7.x and before. > > > > Man ctm says: "The ctm command appeared in FreeBSD 2.1" > > In theory someone could be extracting on a really old host ? > > We might be lucky & find all people are extracting on 8.x ? > > We won't know unless we ask, > > Stephen could request on ctm-announce@ eg: > > "People extracting CTM on old systems are requested to run > > uname -r > > & if not 8.something-RELEASE or current, report to ....@.... > > > > Personaly, I'd think an acceptable middle course could be to switch to > > whatever reduces the CTM bandwidth the most & put a note in man ctm: > > "If running FreeBSD-x or earlier you need to > > cd /usr/ports/archivers/xz ; make install" > > > > How does a manpage entry help? It'll only appear on later versions, where > It's irrelevant, and not appear on earlier versions, where it would help! I see what you mean, but often I read a manual on 1 host while running a prog on another. Notleast cos newer manuals sometimes list more limitations, eg this. Adding a note to manual would be useful, As above would Stephen's run time err. message. > Perhaps a note could be put into the Handbook entry? Sure, good idea. though not all have handbook, not all on line, (though personaly I mirror handbook to laptop for ref. before traveling). > Content-Type: text/html; charset=ISO-8859-1 > Content-Transfer-Encoding: quoted-printable Duplicate ;-). Cheers, Julian -- Julian Stacey, BSD Unix Linux C Sys Eng Consultants Munich http://berklix.com Reply below, not above; Indent with "> "; Cumulative like a play script. Format: Plain text. Not HTML, multipart/alternative, base64, quoted-printable. From owner-ctm-users@FreeBSD.ORG Wed Aug 10 22:50:51 2011 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6DC70106566C for ; Wed, 10 Aug 2011 22:50:51 +0000 (UTC) (envelope-from stephen@missouri.edu) Received: from wilberforce.math.missouri.edu (wilberforce.math.missouri.edu [128.206.184.213]) by mx1.freebsd.org (Postfix) with ESMTP id 218888FC0C for ; Wed, 10 Aug 2011 22:50:50 +0000 (UTC) Received: from [127.0.0.1] (wilberforce.math.missouri.edu [128.206.184.213]) by wilberforce.math.missouri.edu (8.14.4/8.14.4) with ESMTP id p7AMonkn034539 for ; Wed, 10 Aug 2011 17:50:50 -0500 (CDT) (envelope-from stephen@missouri.edu) Message-ID: <4E430B4A.4050603@missouri.edu> Date: Wed, 10 Aug 2011 17:50:50 -0500 From: Stephen Montgomery-Smith User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110617 Thunderbird/3.1.11 MIME-Version: 1.0 To: "ctm-users@freebsd.org" References: <4E41F59F.9020201@missouri.edu> In-Reply-To: <4E41F59F.9020201@missouri.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Use of xz instead of gzip X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 22:50:51 -0000 Comments welcome: http://www.freebsd.org/cgi/query-pr.cgi?pr=159665 I probably won't switch to bzip2 or xz compressed CTM deltas soon. But it is good to prepare for this. (The only reason I was considering hurrying with this proposal was if CTM included svn. Now that I have postponed this indefinitely, the hurry is over.) From owner-ctm-users@FreeBSD.ORG Wed Aug 10 23:04:55 2011 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 794DB106564A; Wed, 10 Aug 2011 23:04:55 +0000 (UTC) (envelope-from jhs@berklix.com) Received: from tower.berklix.org (tower.berklix.org [83.236.223.114]) by mx1.freebsd.org (Postfix) with ESMTP id E27B58FC0A; Wed, 10 Aug 2011 23:04:54 +0000 (UTC) Received: from park.js.berklix.net (p57BCF480.dip.t-dialin.net [87.188.244.128]) (authenticated bits=0) by tower.berklix.org (8.14.2/8.14.2) with ESMTP id p7AN4qvx073361; Wed, 10 Aug 2011 23:04:53 GMT (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (fire.js.berklix.net [192.168.91.41]) by park.js.berklix.net (8.13.8/8.13.8) with ESMTP id p7AN4f0A036981; Thu, 11 Aug 2011 01:04:41 +0200 (CEST) (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (localhost [127.0.0.1]) by fire.js.berklix.net (8.14.4/8.14.4) with ESMTP id p7AN4Avk086473; Wed, 10 Aug 2011 23:04:27 GMT (envelope-from jhs@fire.js.berklix.net) Message-Id: <201108102304.p7AN4Avk086473@fire.js.berklix.net> To: Stephen Montgomery-Smith From: "Julian H. Stacey" Organization: http://www.berklix.com BSD Unix Linux Consultancy, Munich Germany User-agent: EXMH on FreeBSD http://www.berklix.com/free/ X-URL: http://www.berklix.com In-reply-to: Your message "Wed, 10 Aug 2011 09:38:11 CDT." <4E4297D3.5020108@missouri.edu> Date: Thu, 11 Aug 2011 01:04:10 +0200 Sender: jhs@berklix.com Cc: "ctm-users@freebsd.org" Subject: Re: rep-cache.db: was SVN repo X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 23:04:55 -0000 Hi Stephen & all > I created a .0001 delta for svn. It is about 5G. When compressed with > gzip, it goes down to 1.8G. When compressed with bzip2, it goes down to > 1.4G. When compressed with xz, it goes down to 1G. These are some huge > differences in my opinion. Yup ! Too big to ignore. Just in case, did you possibly overlook ensuring comparing like with like ? eg: man gzip -9, --best These options change the compression level used, with the -1 option being the fastest, with less compression, and the -9 option being the slowest, with optimal com- pression. The default compression level is 6. BTW I have : printenv | grep -i gzip # GZIP=--best man bzip2: And --best merely selects the default behaviour. man xz -0 ... -9 Select a compression preset level. The default is -6. ...... > If FreeBSD were to switch to svn, and stop supporting cvs, how many > people would be negatively impacted by not having svn on CTM? I'd miss it. I could live without, I could doubtless switch to running csup of ctm then svn, (but dont much want to punch another hole in firewall, figure packet proxying (cos gate disk too small), &/or fix my ftp proxy (only my http proxy works) I could fix or reconfig all, but I'd miss it :-), ctm via email, fed to local cvs tree has always been just so nice & efficient, allowing lots of offline for max security &/or better response for other interactive stuff, & quick load to laptop for travel, whatever. > I tried deleting base/db/repo-cache.db. I have the book Version Control with Subversion 2nd Ed. but never read it, Maybe you might find an SVN expert to discuss with on one of the SVN lists at http://lists.freebsd.org/mailman/listinfo maybe http://lists.freebsd.org/mailman/listinfo/svn-src-svnadmin Thanks for all the CTM work you do for us ! Cheers, Julian -- Julian Stacey, BSD Unix Linux C Sys Eng Consultants Munich http://berklix.com Reply below, not above; Indent with "> "; Cumulative like a play script. Format: Plain text. Not HTML, multipart/alternative, base64, quoted-printable. From owner-ctm-users@FreeBSD.ORG Wed Aug 10 23:21:35 2011 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BE98F106566B; Wed, 10 Aug 2011 23:21:35 +0000 (UTC) (envelope-from stephen@missouri.edu) Received: from wilberforce.math.missouri.edu (wilberforce.math.missouri.edu [128.206.184.213]) by mx1.freebsd.org (Postfix) with ESMTP id 6F3BA8FC14; Wed, 10 Aug 2011 23:21:35 +0000 (UTC) Received: from [127.0.0.1] (wilberforce.math.missouri.edu [128.206.184.213]) by wilberforce.math.missouri.edu (8.14.4/8.14.4) with ESMTP id p7ANLRdU037763; Wed, 10 Aug 2011 18:21:27 -0500 (CDT) (envelope-from stephen@missouri.edu) Message-ID: <4E431277.6070400@missouri.edu> Date: Wed, 10 Aug 2011 18:21:27 -0500 From: Stephen Montgomery-Smith User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110617 Thunderbird/3.1.11 MIME-Version: 1.0 To: "Julian H. Stacey" References: <201108102304.p7AN4Avk086473@fire.js.berklix.net> In-Reply-To: <201108102304.p7AN4Avk086473@fire.js.berklix.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "ctm-users@freebsd.org" Subject: Re: rep-cache.db: was SVN repo X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 23:21:35 -0000 On 08/10/2011 06:04 PM, Julian H. Stacey wrote: > Hi Stephen& all > >> I created a .0001 delta for svn. It is about 5G. When compressed with >> gzip, it goes down to 1.8G. When compressed with bzip2, it goes down to >> 1.4G. When compressed with xz, it goes down to 1G. These are some huge >> differences in my opinion. > > Yup ! Too big to ignore. Just in case, > did you possibly overlook ensuring comparing like with like ? eg: > > man gzip > -9, --best These options change the compression level used, with > the -1 option being the fastest, with less compression, > and the -9 option being the slowest, with optimal com- > pression. The default compression level is 6. > BTW I have : printenv | grep -i gzip # GZIP=--best > > man bzip2: > And --best merely selects the default behaviour. > > man xz > -0 ... -9 > Select a compression preset level. The default is -6. > ...... > The CTM's are created with "gzip -9." For bzip2 and xz I simply checked default behavior. But bzip2 is already best by default. Tests I have done in the past suggest that the difference created by adjusting these numbers is not very great. I think what happens is that the compression program splits the file into parts, and then compresses each part. The -n options simply say how big those parts can be. As you can imagine, making these parts super big will bring diminishing returns. > >> If FreeBSD were to switch to svn, and stop supporting cvs, how many >> people would be negatively impacted by not having svn on CTM? > > I'd miss it. I could live without, > I could doubtless switch to running csup of ctm then svn, > (but dont much want to punch another hole in firewall, I get the impression that svn is via ssh. Do you have ssh blocked in your firewall? > figure packet proxying (cos gate disk too small), > &/or fix my ftp proxy (only my http proxy works) > I could fix or reconfig all, but I'd miss it :-), > ctm via email, fed to local cvs tree has always been just so > nice& efficient, allowing lots of offline for max security&/or > better response for other interactive stuff,& quick load to laptop > for travel, whatever. > > >> I tried deleting base/db/repo-cache.db. > > I have the book Version Control with Subversion 2nd Ed. but never read it, I found an online book on svn. I think it was this book. I searched this book for rep-cache.db, but it came up blank. > Maybe you might find an SVN expert to discuss with on one of the SVN lists at > http://lists.freebsd.org/mailman/listinfo > maybe > http://lists.freebsd.org/mailman/listinfo/svn-src-svnadmin I might do that, but probably not in the near future. > > Thanks for all the CTM work you do for us ! > > Cheers, > Julian From owner-ctm-users@FreeBSD.ORG Thu Aug 11 00:50:02 2011 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3A6FD106566B for ; Thu, 11 Aug 2011 00:50:02 +0000 (UTC) (envelope-from jhs@berklix.com) Received: from tower.berklix.org (tower.berklix.org [83.236.223.114]) by mx1.freebsd.org (Postfix) with ESMTP id 6851E8FC0C for ; Thu, 11 Aug 2011 00:50:00 +0000 (UTC) Received: from park.js.berklix.net (pD9FBFE86.dip.t-dialin.net [217.251.254.134]) (authenticated bits=0) by tower.berklix.org (8.14.2/8.14.2) with ESMTP id p7B0nufp073890; Thu, 11 Aug 2011 00:49:57 GMT (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (fire.js.berklix.net [192.168.91.41]) by park.js.berklix.net (8.13.8/8.13.8) with ESMTP id p7B0nhLE037448; Thu, 11 Aug 2011 02:49:45 +0200 (CEST) (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (localhost [127.0.0.1]) by fire.js.berklix.net (8.14.4/8.14.4) with ESMTP id p7B0nLRw017277; Thu, 11 Aug 2011 00:49:26 GMT (envelope-from jhs@fire.js.berklix.net) Message-Id: <201108110049.p7B0nLRw017277@fire.js.berklix.net> To: Stephen Montgomery-Smith From: "Julian H. Stacey" Organization: http://www.berklix.com BSD Unix Linux Consultancy, Munich Germany User-agent: EXMH on FreeBSD http://www.berklix.com/free/ X-URL: http://www.berklix.com In-reply-to: Your message "Wed, 10 Aug 2011 18:21:27 CDT." <4E431277.6070400@missouri.edu> Date: Thu, 11 Aug 2011 02:49:21 +0200 Sender: jhs@berklix.com Cc: "ctm-users@freebsd.org" Subject: Re: rep-cache.db: was SVN repo X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 00:50:02 -0000 Stephen Montgomery-Smith wrote: > On 08/10/2011 06:04 PM, Julian H. Stacey wrote: > > Hi Stephen& all > > > >> I created a .0001 delta for svn. It is about 5G. When compressed with > >> gzip, it goes down to 1.8G. When compressed with bzip2, it goes down to > >> 1.4G. When compressed with xz, it goes down to 1G. These are some huge > >> differences in my opinion. > > > > Yup ! Too big to ignore. Just in case, > > did you possibly overlook ensuring comparing like with like ? eg: > > > > man gzip > > -9, --best These options change the compression level used, with > > the -1 option being the fastest, with less compression, > > and the -9 option being the slowest, with optimal com- > > pression. The default compression level is 6. > > BTW I have : printenv | grep -i gzip # GZIP=--best > > > > man bzip2: > > And --best merely selects the default behaviour. > > > > man xz > > -0 ... -9 > > Select a compression preset level. The default is -6. > > ...... > > > > The CTM's are created with "gzip -9." For bzip2 and xz I simply checked > default behavior. But bzip2 is already best by default. OK, so that means if anything iif xv were turned to max, it would tilt the balance even more to ward xv, if anything. > > Tests I have done in the past suggest that the difference created by > adjusting these numbers is not very great. I think what happens is that > the compression program splits the file into parts, and then compresses > each part. The -n options simply say how big those parts can be. As > you can imagine, making these parts super big will bring diminishing > returns. OK. > I get the impression that svn is via ssh. Do you have ssh blocked in > your firewall? No ssh is not blocked. Err, well actually half blocked :-) Being paranoid I only let ssh packets go to know regular hosts. It never occured to me run svn remotely, maybe I'll try some day. ( I'm so used to running cvs localy, & I dont think much of DSL connectivity, which my ISP quotes as <= 16 Mbit/sec but I've not seen more than 5 & slow interactive ie latency I suppose - A competitor selling optic fibre approached our building 2 months after I'd signed a 2 year contract for copper - Murphy's law ) > >> I tried deleting base/db/repo-cache.db. > > > > I have the book Version Control with Subversion 2nd Ed. but never read it, > > I found an online book on svn. I think it was this book. I searched > this book for rep-cache.db, but it came up blank. I started looking in back of book index but time presses. I can't see a string in cd /usr/ports/devel/subversion ; make patch ... find ... grep or in http://subversion.apache.org/ http://www.google.com/search?sitesearch=subversion.apache.org&q=repo-cache.db&Search=Go http://subversion.apache.org/faq.html > > Maybe you might find an SVN expert to discuss with on one of the SVN lists at > > http://lists.freebsd.org/mailman/listinfo > > maybe > > http://lists.freebsd.org/mailman/listinfo/svn-src-svnadmin > > I might do that, but probably not in the near future. Yup, Tempus Fugit. Cheers, Julian -- Julian Stacey, BSD Unix Linux C Sys Eng Consultants Munich http://berklix.com Reply below, not above; Indent with "> "; Cumulative like a play script. Format: Plain text. Not HTML, multipart/alternative, base64, quoted-printable. From owner-ctm-users@FreeBSD.ORG Thu Aug 11 04:31:13 2011 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 237D2106566C for ; Thu, 11 Aug 2011 04:31:13 +0000 (UTC) (envelope-from stephen@missouri.edu) Received: from wilberforce.math.missouri.edu (wilberforce.math.missouri.edu [128.206.184.213]) by mx1.freebsd.org (Postfix) with ESMTP id E34648FC0A for ; Thu, 11 Aug 2011 04:31:12 +0000 (UTC) Received: from [127.0.0.1] (wilberforce.math.missouri.edu [128.206.184.213]) by wilberforce.math.missouri.edu (8.14.4/8.14.4) with ESMTP id p7B4VBOp004156 for ; Wed, 10 Aug 2011 23:31:11 -0500 (CDT) (envelope-from stephen@missouri.edu) Message-ID: <4E435B10.9040907@missouri.edu> Date: Wed, 10 Aug 2011 23:31:12 -0500 From: Stephen Montgomery-Smith User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110617 Thunderbird/3.1.11 MIME-Version: 1.0 To: ctm-users@freebsd.org References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: SVN repo X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 04:31:13 -0000 On 08/09/2011 07:49 AM, Chris Rees wrote: > Hi Stephen, > > Have you any plans to start distributing the subversion repo? I keep > hearing little hints about the Other Two Trees being moved too... If > you need any more hands on it I'll be happy to help. > > Chris OK, I think I have figured out how to do this, at least in principle. svn has all its data organized into revisions. These are numbered, starting at 1, and going on up to something like 224771. Every time someone commits to the svn repository, the revision number is incremented. There is the command: svnadmin dump whereever-the-repository-is \ -r 224703:224771 --incremental --deltas > afile so that if someone else has a repository going up to revision 224702, then they can update to revision 224771 with the command: svnadmin load whereever-the-other-one-is < afile. Furthermore, afile will just contain diffs needed to go from 224702 to 224771, and so it is not going to be larger than it needs to be. Obviously a wrapper can be written around these commands so that afile is sent out be email, and ctm and mkCTM can be written so that svn updates can be sent out by CTM. I'm still experimenting with this. And I don't think I will work seriously on this until the FreeBSD project gets close to abandoning cvs altogether. But I think that with some work, this can be made to work. Stephen From owner-ctm-users@FreeBSD.ORG Thu Aug 11 21:12:46 2011 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DA748106564A for ; Thu, 11 Aug 2011 21:12:46 +0000 (UTC) (envelope-from utisoft@gmail.com) Received: from mail-iy0-f172.google.com (mail-iy0-f172.google.com [209.85.210.172]) by mx1.freebsd.org (Postfix) with ESMTP id A67E68FC14 for ; Thu, 11 Aug 2011 21:12:46 +0000 (UTC) Received: by iye7 with SMTP id 7so4156308iye.17 for ; Thu, 11 Aug 2011 14:12:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type:content-transfer-encoding; bh=2ICt5VOKnPsMlNi4jsJO5SV7U19yWy/eq86wBCH2FWw=; b=rLL+DMaJKDWCDHrYtv/uPTT9+FuHmqzQ+I9cmslmAWGvaDcz3sWpnGYrohPTeFE6T4 BzoAYvukE3jFyeuAUcG/IKu49soKs9YM7jma/od82NfWVgf2irhDbS65FgXWLretLDj0 Jd6VB52n3olNR/8Uww9KjoWHM+swXPverzqg0= Received: by 10.231.207.158 with SMTP id fy30mr257393ibb.63.1313097166101; Thu, 11 Aug 2011 14:12:46 -0700 (PDT) MIME-Version: 1.0 Received: by 10.231.182.77 with HTTP; Thu, 11 Aug 2011 14:12:16 -0700 (PDT) In-Reply-To: <201108101503.p7AF3A5L081336@fire.js.berklix.net> References: <201108101503.p7AF3A5L081336@fire.js.berklix.net> From: Chris Rees Date: Thu, 11 Aug 2011 22:12:16 +0100 Message-ID: To: "Julian H. Stacey" Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: "ctm-users@freebsd.org" Subject: Re: Use of xz instead of gzip X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 21:12:47 -0000 On 10 August 2011 16:03, Julian H. Stacey wrote: > Hi, > Reference: >> From: =A0 =A0 =A0 =A0 Chris Rees >> Date: =A0 =A0 =A0 =A0 Wed, 10 Aug 2011 14:17:01 +0100 >> Message-id: =A0 > > Chris Rees wrote: >> --20cf307f3aaee51c1704aa267e44 >> Content-Type: text/plain; charset=3DISO-8859-1 >> >> On 10 Aug 2011 13:53, "Julian H. Stacey" wrote: >> > >> > > I can see this presenting a problem on 7.x (and earlier, but they ar= e >> > > no longer supported) because the base system shouldn't have dependen= cies >> > > on ports. =A0The options would seem to be: >> > > 1) Stick with gzip for 7-src and earlier (as well as ports until 7.x >> > > =A0 =A0goes EOL) >> > >> > (Not that I'm suggesting for or against Option 1, but it needs modifyi= ng, >> > Applying the same logic, append to Option 1: "as well as cvs-cur" >> > =A0 =A0 =A0 =A0(as cvs can be used to extract eg 4 & ports & doc as we= ll as head). >> > As the ctm collection comprises: >> > =A0 =A0 =A0 =A0 cvs-cur ports-cur src-4 src-5 src-6 src-7 src-8 src-cu= r >> > Option 1 thus becomes "Stick with gzip for all except src-8 & src-cur" >> > >> > Whether a CTM src-stable stream is EOL is irrelevant. >> > All that matters is whether extractor hosts have xz or not. >> > =A0One does not need to run ctm on same `uname -r` as one extracts for= , >> > eg my CTM extractor runs on 8.3, but I extract for all of >> > =A0 =A0 =A0 =A0cvs-cur ports-cur src-4 src-5 src-6 src-7 src-8 src-cur >> > Some (4, maybe 6) have long been EOL, but I use them for reasons beyon= d >> > remit of this thread. >> > >> > >> > > 2) Make ctm a port (so it can have a dependency on xz) >> > >> > I cant imagine many CTM users saying "please kick CTM out of src/" ;-) >> > >> > >> > > 3) If dispensation is granted, >> > >> > Dispensation for what ? >> > >> > >> > > =A0 =A0ensure the dependency is documented in >> > > =A0 =A0UPDATING and ctm.1 when it is MFC'd to 7.x and before. >> > >> > Man ctm says: "The ctm command appeared in FreeBSD 2.1" >> > In theory someone could be extracting on a really old host ? >> > We might be lucky & find all people are extracting on 8.x ? >> > We won't know unless we ask, >> > Stephen could request on ctm-announce@ eg: >> > =A0"People extracting CTM on old systems are requested to run >> > =A0 =A0 =A0 =A0uname -r >> > =A0& if not 8.something-RELEASE or current, report to ....@.... >> > >> > Personaly, I'd think an acceptable middle course could be to switch to >> > whatever reduces the CTM bandwidth the most & put a note in man ctm: >> > =A0 =A0 =A0 =A0"If running FreeBSD-x or earlier you need to >> > =A0 =A0 =A0 =A0 cd /usr/ports/archivers/xz ; make install" >> > >> >> How does a manpage entry help? It'll only appear on later versions, wher= e >> It's irrelevant, and not appear on earlier versions, where it would help= ! > > I see what you mean, but often I read a manual on 1 host while > running a prog on another. Notleast cos newer manuals sometimes > list more limitations, eg this. Adding a note to manual would be useful, > > As above would Stephen's run time err. message. > >> Perhaps a note could be put into the Handbook entry? > > Sure, good idea. though not all have handbook, not all on line, > (though personaly I mirror handbook to laptop for ref. before traveling). > >> Content-Type: text/html; charset=3DISO-8859-1 >> Content-Transfer-Encoding: quoted-printable > > Duplicate ;-). > > Cheers, > Julian Suitably chastised, I'm sorry. Blame the gmail Android app :( Chris --=20 Chris Rees =A0 =A0 =A0 =A0 =A0| FreeBSD Developer crees@FreeBSD.org =A0 | http://people.freebsd.org/~crees