From owner-ctm-users@FreeBSD.ORG Mon Jan 28 21:24:55 2013 Return-Path: Delivered-To: ctm-users@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 70F06762 for ; Mon, 28 Jan 2013 21:24:55 +0000 (UTC) (envelope-from ike@blackskyresearch.net) Received: from rs149.luxsci.com (rs149.luxsci.com [64.49.224.181]) by mx1.freebsd.org (Postfix) with ESMTP id ECC6FCF5 for ; Mon, 28 Jan 2013 21:24:54 +0000 (UTC) Received: from rs149.luxsci.com (localhost.localdomain [127.0.0.1]) by rs149.luxsci.com (8.14.4/8.13.8) with ESMTP id r0SLOqks005159; Mon, 28 Jan 2013 16:24:52 -0500 Received: (from root@localhost) by rs149.luxsci.com (8.14.4/8.13.8/Submit) id r0SLO3Jp003942; Mon, 28 Jan 2013 21:24:03 GMT Received: (from sender 74627) (rs149.luxsci.com [127.0.0.1]) by LuxSci SP; Mon, 28 Jan 2013 21:24:02 +0000 Subject: Re: CTM index on servers Content-Type: text/plain; charset=windows-1252 From: "Isaac (.ike) Levy" In-Reply-To: <5106E4E5.5060905@missouri.edu> Date: Mon, 28 Jan 2013 16:23:43 -0500 Content-Transfer-Encoding: quoted-printable References: <1359401402-8482577.26305758.fr0SJTdBX014869@rs149.luxsci.com> <5106E4E5.5060905@missouri.edu> To: Stephen Montgomery-Smith X-Lux-Comment: Message r0SLNgQZ003594 sent by user #74627 Message-Id: <1359408243-9173061.43936173.fr0SLNgQZ003594@rs149.luxsci.com> X-Comment: LuxSci SP Message ID - 1359408243-9173061.43936173 Cc: ctm-users@freebsd.org X-BeenThere: ctm-users@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: CTM User discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jan 2013 21:24:55 -0000 On Jan 28, 2013, at 3:51 PM, Stephen Montgomery-Smith wrote: > On 01/28/2013 01:29 PM, Isaac (.ike) Levy wrote: >> With some sort of index, the ease-of-use functionality of c[v]sup = could be trivially replicated in sh(1). >> Even just as a prototype, this is a task I will enthusiastically do, = even if only for my own use- (I started something already which culls = FTP tarballs, a less than ideal solution). >>=20 >> What do yall' think? >=20 > The index idea would be very easy for me to implement. Unless anyone > has a serious objection, I will probably go ahead and implement it as > above. Except I will not include the time stamp in the file, as the > file already has a modification time on it. =20 Fair enough, it makes the index far simpler to parse. > I will probably sign the > index file as well, but I don't think it is necessary, as if that > information is corrupted it is unlikely to do any damage. This is correct, a .sig not necessary. However if my program can check the sig, I can have my fetcher bail = early if the sig isn't kosher- and therefore avoid possible bandwidth = hogging/DOS- (my bandwidth, or the project servers!). > But ... I use a script like this: >=20 > I store all the ctm files in /usr/home/stephen/ctm or subdirectories > thereof. This script sees what is already there, and then tries to > fetch any future ctm deltas. >=20 >=20 > #!/bin/sh >=20 > cd /usr/home/stephen/ctm/incoming >=20 > bases=3D`find /usr/home/stephen/ctm -type f | > sed -n -E -e 's+.*/++' -e 's/\.[[:digit:]]{4}\.gz//p' | sort -u` >=20 > for b in $bases; do > while true; do > index=3D`find /usr/home/stephen/ctm -type f -name "$b*" | > sed -n -E 's/.*([[:digit:]]{4}).*/\1/p' | > sort -n | tail -1 | xargs expr 1 + | xargs printf "%04d"` > fetch ftp://ftp.freebsd.org/pub/CTM/$b/$b.$index.gz || break > done > done >=20 > echo > ls Thanks for that shell bit!!! Best, .ike