From owner-ctm-users@FreeBSD.ORG Mon Jan 28 19:30:53 2013 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 4221E77E for ; Mon, 28 Jan 2013 19:30:53 +0000 (UTC) (envelope-from ike@blackskyresearch.net) Received: from rs149.luxsci.com (rs149.luxsci.com [64.49.224.181]) by mx1.freebsd.org (Postfix) with ESMTP id D9DD46E4 for ; Mon, 28 Jan 2013 19:30:52 +0000 (UTC) Received: from rs149.luxsci.com (localhost.localdomain [127.0.0.1]) by rs149.luxsci.com (8.14.4/8.13.8) with ESMTP id r0SJUoZC017033 for ; Mon, 28 Jan 2013 14:30:50 -0500 Received: (from root@localhost) by rs149.luxsci.com (8.14.4/8.13.8/Submit) id r0SJU2rv015491 for ctm-users@freebsd.org; Mon, 28 Jan 2013 19:30:02 GMT Received: (from sender 74627) (rs149.luxsci.com [127.0.0.1]) by LuxSci SP; Mon, 28 Jan 2013 19:30:02 +0000 From: "Isaac (.ike) Levy" Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Subject: CTM index on servers Date: Mon, 28 Jan 2013 14:29:39 -0500 To: ctm-users@freebsd.org X-Lux-Comment: Message r0SJTdBX014869 sent by user #74627 Message-Id: <1359401402-8482577.26305758.fr0SJTdBX014869@rs149.luxsci.com> X-Comment: LuxSci SP Message ID - 1359401402-8482577.26305758 X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jan 2013 19:30:53 -0000 Hi Stephen, ctm-users, New to this list, coming from a lively thread on freebsd-stable@ For me and a lot of admins/users, fetching sources just got a lot harder = for a *lot* of admins/users. A page was raised to try to help cull = facts out of all the confusing details, here: https://wiki.freebsd.org/UsersFetchingSource At this point the page may be more confusing itself :) -- Enter: CTM I'm delighted to learn about ctm(1) usage, and quite interested in using = it to fetch REL and RELENG sources, and keep it up to date (to = redistribute internally across a cluster of servers). I know a *ton* of *BSD admins who love c[v]sup, but I don't know any who = use ctm(1) any more, so I'm sorry if my questions are naive. On Jan 28, 2013, at 1:53 PM, Stephen Montgomery-Smith wrote: >> On the FTP server, is there an index of which CTM delta files are on = the FTP server? I'd like to automate fetching just the new deltas... >>=20 >> If there was perhaps 1 file with a consistent name, I could fetch = that on a nightly basis and fetch the other CTM files. >=20 > I can easily put in something like this. Maybe the output of ls in = each directory? >=20 > If you have more ideas on this, sign up to the ctm-users mailing list, = and we can discuss exactly what is wanted. I'm very interested in seeing some kind of 'index' file put in place, a = simple file which lists the ctm delta (and xEmpty) files. This way, users could do something in a shell/cron job, like: fetch ftp://ftp.freebsd.org/pub/FreeBSD/development/CTM/src-9/index # # diff against the last ctm file, or a directory listing containing the = delta files # download/apply any ctm delta files which have not yet been applied -- It's an open question weather or not the entire CTM dir should have the = index, or the individual releases (or both?) e.g. /pub/FreeBSD/development/CTM/src-9/index - vs - /pub/FreeBSD/development/CTM/index (I assume it would be appropriate also to provide sig files for the = index, as well.) -- The index could be as simple as: $ echo "`date +%s`" ; find . -type f -name 'src*' | sort | sed = 's/.*\///' 1359382951 src-9.0400xEmpty.gz src-9.0401.gz src-9.0401.gz.sig src-9.0402.gz src-9.0402.gz.sig src-9.0403.gz src-9.0403.gz.sig $=20 Could get fancier as necessary later, (perhaps date modified for each = file, etc=85) -- With some sort of index, the ease-of-use functionality of c[v]sup could = be trivially replicated in sh(1). Even just as a prototype, this is a task I will enthusiastically do, = even if only for my own use- (I started something already which culls = FTP tarballs, a less than ideal solution). What do yall' think? Best, .ike From owner-ctm-users@FreeBSD.ORG Mon Jan 28 20:51:51 2013 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 68978FF for ; Mon, 28 Jan 2013 20:51:51 +0000 (UTC) (envelope-from stephen@missouri.edu) Received: from wilberforce.math.missouri.edu (wilberforce.math.missouri.edu [128.206.184.213]) by mx1.freebsd.org (Postfix) with ESMTP id 38BF8A89 for ; Mon, 28 Jan 2013 20:51:50 +0000 (UTC) Received: from [127.0.0.1] (wilberforce.math.missouri.edu [128.206.184.213]) by wilberforce.math.missouri.edu (8.14.6/8.14.6) with ESMTP id r0SKpnSq085296 for ; Mon, 28 Jan 2013 14:51:50 -0600 (CST) (envelope-from stephen@missouri.edu) Message-ID: <5106E4E5.5060905@missouri.edu> Date: Mon, 28 Jan 2013 14:51:49 -0600 From: Stephen Montgomery-Smith User-Agent: Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/20130106 Thunderbird/17.0.2 MIME-Version: 1.0 To: ctm-users@freebsd.org Subject: Re: CTM index on servers References: <1359401402-8482577.26305758.fr0SJTdBX014869@rs149.luxsci.com> In-Reply-To: <1359401402-8482577.26305758.fr0SJTdBX014869@rs149.luxsci.com> X-Enigmail-Version: 1.5 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jan 2013 20:51:51 -0000 On 01/28/2013 01:29 PM, Isaac (.ike) Levy wrote: > Hi Stephen, ctm-users, > > New to this list, coming from a lively thread on freebsd-stable@ > > For me and a lot of admins/users, fetching sources just got a lot harder for a *lot* of admins/users. A page was raised to try to help cull facts out of all the confusing details, here: > > https://wiki.freebsd.org/UsersFetchingSource > > At this point the page may be more confusing itself :) > > -- > Enter: CTM > > I'm delighted to learn about ctm(1) usage, and quite interested in using it to fetch REL and RELENG sources, and keep it up to date (to redistribute internally across a cluster of servers). > > I know a *ton* of *BSD admins who love c[v]sup, but I don't know any who use ctm(1) any more, so I'm sorry if my questions are naive. > > On Jan 28, 2013, at 1:53 PM, Stephen Montgomery-Smith wrote: >>> On the FTP server, is there an index of which CTM delta files are on the FTP server? I'd like to automate fetching just the new deltas... >>> >>> If there was perhaps 1 file with a consistent name, I could fetch that on a nightly basis and fetch the other CTM files. >> >> I can easily put in something like this. Maybe the output of ls in each directory? >> >> If you have more ideas on this, sign up to the ctm-users mailing list, and we can discuss exactly what is wanted. > > I'm very interested in seeing some kind of 'index' file put in place, a simple file which lists the ctm delta (and xEmpty) files. > > This way, users could do something in a shell/cron job, like: > > fetch ftp://ftp.freebsd.org/pub/FreeBSD/development/CTM/src-9/index > # > # diff against the last ctm file, or a directory listing containing the delta files > # download/apply any ctm delta files which have not yet been applied > > -- > It's an open question weather or not the entire CTM dir should have the index, or the individual releases (or both?) > > e.g. > /pub/FreeBSD/development/CTM/src-9/index > - vs - > /pub/FreeBSD/development/CTM/index > > (I assume it would be appropriate also to provide sig files for the index, as well.) > > -- > The index could be as simple as: > > $ echo "`date +%s`" ; find . -type f -name 'src*' | sort | sed 's/.*\///' > 1359382951 > src-9.0400xEmpty.gz > src-9.0401.gz > src-9.0401.gz.sig > src-9.0402.gz > src-9.0402.gz.sig > src-9.0403.gz > src-9.0403.gz.sig > $ > > Could get fancier as necessary later, (perhaps date modified for each file, etc…) > > -- > With some sort of index, the ease-of-use functionality of c[v]sup could be trivially replicated in sh(1). > Even just as a prototype, this is a task I will enthusiastically do, even if only for my own use- (I started something already which culls FTP tarballs, a less than ideal solution). > > What do yall' think? The index idea would be very easy for me to implement. Unless anyone has a serious objection, I will probably go ahead and implement it as above. Except I will not include the time stamp in the file, as the file already has a modification time on it. I will probably sign the index file as well, but I don't think it is necessary, as if that information is corrupted it is unlikely to do any damage. But ... I use a script like this: I store all the ctm files in /usr/home/stephen/ctm or subdirectories thereof. This script sees what is already there, and then tries to fetch any future ctm deltas. #!/bin/sh cd /usr/home/stephen/ctm/incoming bases=`find /usr/home/stephen/ctm -type f | sed -n -E -e 's+.*/++' -e 's/\.[[:digit:]]{4}\.gz//p' | sort -u` for b in $bases; do while true; do index=`find /usr/home/stephen/ctm -type f -name "$b*" | sed -n -E 's/.*([[:digit:]]{4}).*/\1/p' | sort -n | tail -1 | xargs expr 1 + | xargs printf "%04d"` fetch ftp://ftp.freebsd.org/pub/CTM/$b/$b.$index.gz || break done done echo ls From owner-ctm-users@FreeBSD.ORG Mon Jan 28 21:24:55 2013 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 70F06762 for ; Mon, 28 Jan 2013 21:24:55 +0000 (UTC) (envelope-from ike@blackskyresearch.net) Received: from rs149.luxsci.com (rs149.luxsci.com [64.49.224.181]) by mx1.freebsd.org (Postfix) with ESMTP id ECC6FCF5 for ; Mon, 28 Jan 2013 21:24:54 +0000 (UTC) Received: from rs149.luxsci.com (localhost.localdomain [127.0.0.1]) by rs149.luxsci.com (8.14.4/8.13.8) with ESMTP id r0SLOqks005159; Mon, 28 Jan 2013 16:24:52 -0500 Received: (from root@localhost) by rs149.luxsci.com (8.14.4/8.13.8/Submit) id r0SLO3Jp003942; Mon, 28 Jan 2013 21:24:03 GMT Received: (from sender 74627) (rs149.luxsci.com [127.0.0.1]) by LuxSci SP; Mon, 28 Jan 2013 21:24:02 +0000 Subject: Re: CTM index on servers Content-Type: text/plain; charset=windows-1252 From: "Isaac (.ike) Levy" In-Reply-To: <5106E4E5.5060905@missouri.edu> Date: Mon, 28 Jan 2013 16:23:43 -0500 Content-Transfer-Encoding: quoted-printable References: <1359401402-8482577.26305758.fr0SJTdBX014869@rs149.luxsci.com> <5106E4E5.5060905@missouri.edu> To: Stephen Montgomery-Smith X-Lux-Comment: Message r0SLNgQZ003594 sent by user #74627 Message-Id: <1359408243-9173061.43936173.fr0SLNgQZ003594@rs149.luxsci.com> X-Comment: LuxSci SP Message ID - 1359408243-9173061.43936173 Cc: ctm-users@freebsd.org X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jan 2013 21:24:55 -0000 On Jan 28, 2013, at 3:51 PM, Stephen Montgomery-Smith wrote: > On 01/28/2013 01:29 PM, Isaac (.ike) Levy wrote: >> With some sort of index, the ease-of-use functionality of c[v]sup = could be trivially replicated in sh(1). >> Even just as a prototype, this is a task I will enthusiastically do, = even if only for my own use- (I started something already which culls = FTP tarballs, a less than ideal solution). >>=20 >> What do yall' think? >=20 > The index idea would be very easy for me to implement. Unless anyone > has a serious objection, I will probably go ahead and implement it as > above. Except I will not include the time stamp in the file, as the > file already has a modification time on it. =20 Fair enough, it makes the index far simpler to parse. > I will probably sign the > index file as well, but I don't think it is necessary, as if that > information is corrupted it is unlikely to do any damage. This is correct, a .sig not necessary. However if my program can check the sig, I can have my fetcher bail = early if the sig isn't kosher- and therefore avoid possible bandwidth = hogging/DOS- (my bandwidth, or the project servers!). > But ... I use a script like this: >=20 > I store all the ctm files in /usr/home/stephen/ctm or subdirectories > thereof. This script sees what is already there, and then tries to > fetch any future ctm deltas. >=20 >=20 > #!/bin/sh >=20 > cd /usr/home/stephen/ctm/incoming >=20 > bases=3D`find /usr/home/stephen/ctm -type f | > sed -n -E -e 's+.*/++' -e 's/\.[[:digit:]]{4}\.gz//p' | sort -u` >=20 > for b in $bases; do > while true; do > index=3D`find /usr/home/stephen/ctm -type f -name "$b*" | > sed -n -E 's/.*([[:digit:]]{4}).*/\1/p' | > sort -n | tail -1 | xargs expr 1 + | xargs printf "%04d"` > fetch ftp://ftp.freebsd.org/pub/CTM/$b/$b.$index.gz || break > done > done >=20 > echo > ls Thanks for that shell bit!!! Best, .ike From owner-ctm-users@FreeBSD.ORG Tue Jan 29 00:47:38 2013 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 9EBB6191 for ; Tue, 29 Jan 2013 00:47:38 +0000 (UTC) (envelope-from stephen@missouri.edu) Received: from wilberforce.math.missouri.edu (wilberforce.math.missouri.edu [128.206.184.213]) by mx1.freebsd.org (Postfix) with ESMTP id 4F763A92 for ; Tue, 29 Jan 2013 00:47:37 +0000 (UTC) Received: from [127.0.0.1] (wilberforce.math.missouri.edu [128.206.184.213]) by wilberforce.math.missouri.edu (8.14.6/8.14.6) with ESMTP id r0T0lY2x000526 for ; Mon, 28 Jan 2013 18:47:35 -0600 (CST) (envelope-from stephen@missouri.edu) Message-ID: <51071C26.2030703@missouri.edu> Date: Mon, 28 Jan 2013 18:47:34 -0600 From: Stephen Montgomery-Smith User-Agent: Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/20130106 Thunderbird/17.0.2 MIME-Version: 1.0 To: ctm-users@freebsd.org Subject: Re: CTM index on servers References: <1359401402-8482577.26305758.fr0SJTdBX014869@rs149.luxsci.com> In-Reply-To: <1359401402-8482577.26305758.fr0SJTdBX014869@rs149.luxsci.com> X-Enigmail-Version: 1.5 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jan 2013 00:47:38 -0000 On 01/28/2013 01:29 PM, Isaac (.ike) Levy wrote: > Hi Stephen, ctm-users, > > New to this list, coming from a lively thread on freebsd-stable@ > > For me and a lot of admins/users, fetching sources just got a lot harder for a *lot* of admins/users. A page was raised to try to help cull facts out of all the confusing details, here: > > https://wiki.freebsd.org/UsersFetchingSource > > At this point the page may be more confusing itself :) > > -- > Enter: CTM > > I'm delighted to learn about ctm(1) usage, and quite interested in using it to fetch REL and RELENG sources, and keep it up to date (to redistribute internally across a cluster of servers). > > I know a *ton* of *BSD admins who love c[v]sup, but I don't know any who use ctm(1) any more, so I'm sorry if my questions are naive. > > On Jan 28, 2013, at 1:53 PM, Stephen Montgomery-Smith wrote: >>> On the FTP server, is there an index of which CTM delta files are on the FTP server? I'd like to automate fetching just the new deltas... >>> >>> If there was perhaps 1 file with a consistent name, I could fetch that on a nightly basis and fetch the other CTM files. >> >> I can easily put in something like this. Maybe the output of ls in each directory? Look in http://web.missouri.edu/~stephen/CTM and the subdirectories, and tell me if you like the way I implemented INDEX. From owner-ctm-users@FreeBSD.ORG Tue Jan 29 13:53:04 2013 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 39A5B5E3 for ; Tue, 29 Jan 2013 13:53:04 +0000 (UTC) (envelope-from ike@blackskyresearch.net) Received: from rs149.luxsci.com (rs149.luxsci.com [64.49.224.181]) by mx1.freebsd.org (Postfix) with ESMTP id CEA5CD38 for ; Tue, 29 Jan 2013 13:53:03 +0000 (UTC) Received: from rs149.luxsci.com (localhost.localdomain [127.0.0.1]) by rs149.luxsci.com (8.14.4/8.13.8) with ESMTP id r0TDqupg020472; Tue, 29 Jan 2013 08:52:57 -0500 Received: (from root@localhost) by rs149.luxsci.com (8.14.4/8.13.8/Submit) id r0TDq2kr019445; Tue, 29 Jan 2013 13:52:02 GMT Received: (from sender 74627) (rs149.luxsci.com [127.0.0.1]) by LuxSci SP; Tue, 29 Jan 2013 13:52:02 +0000 Subject: Re: CTM index on servers Content-Type: text/plain; charset=us-ascii From: "Isaac (.ike) Levy" In-Reply-To: <51071C26.2030703@missouri.edu> Date: Tue, 29 Jan 2013 08:51:39 -0500 Content-Transfer-Encoding: quoted-printable References: <1359401402-8482577.26305758.fr0SJTdBX014869@rs149.luxsci.com> <51071C26.2030703@missouri.edu> To: Stephen Montgomery-Smith X-Lux-Comment: Message r0TDpc0p018870 sent by user #74627 Message-Id: <1359467522-7158796.08576218.fr0TDpc0p018870@rs149.luxsci.com> X-Comment: LuxSci SP Message ID - 1359467522-7158796.08576218 Cc: ctm-users@freebsd.org X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jan 2013 13:53:04 -0000 On Jan 28, 2013, at 7:47 PM, Stephen Montgomery-Smith wrote: > On 01/28/2013 01:29 PM, Isaac (.ike) Levy wrote: >> Hi Stephen, ctm-users, >>=20 >> New to this list, coming from a lively thread on freebsd-stable@ >>=20 >> For me and a lot of admins/users, fetching sources just got a lot = harder for a *lot* of admins/users. A page was raised to try to help = cull facts out of all the confusing details, here: >>=20 >> https://wiki.freebsd.org/UsersFetchingSource >>=20 >> At this point the page may be more confusing itself :) >>=20 >> -- >> Enter: CTM >>=20 >> I'm delighted to learn about ctm(1) usage, and quite interested in = using it to fetch REL and RELENG sources, and keep it up to date (to = redistribute internally across a cluster of servers). >>=20 >> I know a *ton* of *BSD admins who love c[v]sup, but I don't know any = who use ctm(1) any more, so I'm sorry if my questions are naive. >>=20 >> On Jan 28, 2013, at 1:53 PM, Stephen Montgomery-Smith wrote: >>>> On the FTP server, is there an index of which CTM delta files are = on the FTP server? I'd like to automate fetching just the new deltas... >>>>=20 >>>> If there was perhaps 1 file with a consistent name, I could fetch = that on a nightly basis and fetch the other CTM files. >>>=20 >>> I can easily put in something like this. Maybe the output of ls in = each directory? >=20 > Look in http://web.missouri.edu/~stephen/CTM and the subdirectories, = and > tell me if you like the way I implemented INDEX. Absolutely spectacular, I started using it to hack out a basic 'ctmup' = in sh last night. Thank you so much! I'll post back to list as soon as I have it = rough/working, (this week/weekend). Best, .ike