From owner-freebsd-hubs@FreeBSD.ORG Tue Nov 9 09:45:47 2010 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1F9DB106566B for ; Tue, 9 Nov 2010 09:45:47 +0000 (UTC) (envelope-from kamikaze@bsdforen.de) Received: from mail.server1.bsdforen.de (bsdforen.de [82.193.243.81]) by mx1.freebsd.org (Postfix) with ESMTP id A7F9D8FC08 for ; Tue, 9 Nov 2010 09:45:46 +0000 (UTC) Received: from mobileKamikaze.norad (vpn-cl-162-46.rz.uni-karlsruhe.de [141.3.162.46]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mail.server1.bsdforen.de (Postfix) with ESMTPSA id 0E3267E88B for ; Tue, 9 Nov 2010 10:28:08 +0100 (CET) Message-ID: <4CD91428.5090002@bsdforen.de> Date: Tue, 09 Nov 2010 10:28:08 +0100 From: Dominic Fandrey User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-GB; rv:1.9.1.15) Gecko/20101028 Thunderbird/3.0.10 MIME-Version: 1.0 To: freebsd-hubs@freebsd.org X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Subject: /pub/FreeBSD or /FreeBSD X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Nov 2010 09:45:47 -0000 Hello, I'm asking about the official layout policy for mirrors. I have an application that does parallel package downloads from a list of mirrors and the different Layouts cause me some trouble. A lot of mirrors locate everything in /FreeBSD instead of /pub/FreeBSD. Mirrors that don't follow the /pub/FreeBSD layout are currently subject to unnecessary load caused by my application and cause significant CPU load on my side. My application does only one download from each server at a time. If a job fails the next available job is dispatched to the server (and the failed job will try a different one as soon as one is availble), so failing mirrors are subject to the highest frequency of requests. Is there an official policy for the mirror layout? Of course I can add some layout probing, but it would significantly slow down the startup phase of my application, which is a nuisance for small use cases. Regards P.S.: I'm not on this list, so please CC me. If someone suggests being on this list is useful for developing package management software I will consider subscribing, though. -- A: Because it fouls the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing on usenet and in e-mail? From owner-freebsd-hubs@FreeBSD.ORG Tue Nov 9 10:21:13 2010 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2712C106566B for ; Tue, 9 Nov 2010 10:21:13 +0000 (UTC) (envelope-from otto@informatik.rwth-aachen.de) Received: from mta-1.ms.rz.rwth-aachen.de (mta-1.ms.rz.RWTH-Aachen.DE [134.130.7.72]) by mx1.freebsd.org (Postfix) with ESMTP id C8EF38FC1D for ; Tue, 9 Nov 2010 10:21:12 +0000 (UTC) MIME-version: 1.0 Received: from ironport-out-1.rz.rwth-aachen.de ([134.130.5.40]) by mta-1.ms.rz.RWTH-Aachen.de (Sun Java(tm) System Messaging Server 6.3-7.04 (built Sep 26 2008)) with ESMTP id <0LBM00BR23DBXM40@mta-1.ms.rz.RWTH-Aachen.de> for freebsd-hubs@freebsd.org; Tue, 09 Nov 2010 10:51:11 +0100 (CET) X-IronPort-AV: E=Sophos;i="4.59,174,1288566000"; d="asc'?scan'208";a="80353173" Received: from smarthost-1.ms.rz.rwth-aachen.de (HELO smarthost.rwth-aachen.de) ([134.130.7.89]) by ironport-in-1.rz.rwth-aachen.de with ESMTP; Tue, 09 Nov 2010 10:51:11 +0100 Received: from server-mail.halifax.rwth-aachen.de (ip2-199.halifax.RWTH-Aachen.DE [137.226.109.199]) by smarthost.rwth-aachen.de (8.14.4+Sun/8.13.8/1) with ESMTP id oA99pA7E018056; Tue, 09 Nov 2010 10:51:10 +0100 (CET) Received: from daedalus.informatik.rwth-aachen.de ([137.226.194.64]) by server-mail.halifax.rwth-aachen.de with smtp (Exim 4.69) (envelope-from ) id 1PFkr8-0007kl-JJ; Tue, 09 Nov 2010 10:51:10 +0100 Received: (nullmailer pid 4732 invoked by uid 550); Tue, 09 Nov 2010 09:51:10 +0000 Date: Tue, 09 Nov 2010 10:51:10 +0100 From: Carsten Otto To: Dominic Fandrey Message-id: <20101109095110.GG2741@daedalus.informatik.rwth-aachen.de> Mail-followup-to: Dominic Fandrey , freebsd-hubs@freebsd.org, FTP References: <4CD91428.5090002@bsdforen.de> Content-type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary=2WS97oupGEGbYNpW Content-disposition: inline In-reply-to: <4CD91428.5090002@bsdforen.de> X-GnuGP-Key: http://c-otto.de/pubkey.asc User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-hubs@freebsd.org Subject: Re: /pub/FreeBSD or /FreeBSD X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: FTP List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Nov 2010 10:21:13 -0000 --2WS97oupGEGbYNpW Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Nov 09, 2010 at 10:28:08AM +0100, Dominic Fandrey wrote: > Is there an official policy for the mirror layout? We (ftp.halifax.rwth-aachen.de) serve /pub/FreeBSD although we do not use this layout for the other projects. I guess we did this because of some official policy. Best regards, --=20 Carsten Otto otto@informatik.rwth-aachen.de LuFG Informatik 2 http://verify.rwth-aachen.de/otto/ RWTH Aachen phone: +49 241 80-21211 --2WS97oupGEGbYNpW Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkzZGY4ACgkQjUF4jpCSQBTrHACgjau+vS/cBF2wxAZNwxXZhico dPEAoKSHTOieYuC6Lo9XRw5a52NX83OC =tsu4 -----END PGP SIGNATURE----- --2WS97oupGEGbYNpW-- From owner-freebsd-hubs@FreeBSD.ORG Tue Nov 9 12:17:40 2010 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 17B45106567A for ; Tue, 9 Nov 2010 12:17:40 +0000 (UTC) (envelope-from kamikaze@bsdforen.de) Received: from mail.server1.bsdforen.de (bsdforen.de [82.193.243.81]) by mx1.freebsd.org (Postfix) with ESMTP id C90538FC13 for ; Tue, 9 Nov 2010 12:17:39 +0000 (UTC) Received: from mobileKamikaze.norad (vpn-cl-162-46.rz.uni-karlsruhe.de [141.3.162.46]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mail.server1.bsdforen.de (Postfix) with ESMTPSA id 6E8947E8CB; Tue, 9 Nov 2010 13:17:38 +0100 (CET) Message-ID: <4CD93BE1.6070705@bsdforen.de> Date: Tue, 09 Nov 2010 13:17:37 +0100 From: Dominic Fandrey User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-GB; rv:1.9.1.15) Gecko/20101028 Thunderbird/3.0.10 MIME-Version: 1.0 To: freebsd-hubs@freebsd.org, FTP References: <4CD91428.5090002@bsdforen.de> <20101109095110.GG2741@daedalus.informatik.rwth-aachen.de> In-Reply-To: <20101109095110.GG2741@daedalus.informatik.rwth-aachen.de> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: Subject: Re: /pub/FreeBSD or /FreeBSD X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Nov 2010 12:17:40 -0000 On 09/11/2010 10:51, Carsten Otto wrote: > On Tue, Nov 09, 2010 at 10:28:08AM +0100, Dominic Fandrey wrote: >> Is there an official policy for the mirror layout? > > We (ftp.halifax.rwth-aachen.de) serve /pub/FreeBSD although we do not > use this layout for the other projects. I guess we did this because of > some official policy. I think I might have made a mistake here. I checked again and my estimate for half the servers was a) grossly exaggerated and b) the issue is more present over http. Those don't work: http://ftp.de.freebsd.org/pub/FreeBSD/ http://ftp3.de.freebsd.org/pub/FreeBSD/ http://ftp4.de.freebsd.org/pub/FreeBSD/ http://ftp6.de.freebsd.org/pub/FreeBSD/ (no HTTP at all) These work: ftp://ftp.de.freebsd.org/pub/FreeBSD/ ftp://ftp3.de.freebsd.org/pub/FreeBSD/ ftp://ftp6.de.freebsd.org/pub/FreeBSD/ So, I have three distinct problems with half the German mirrors, two offer /pub/FreeBSD only through FTP, but not HTTP. One (ftp4.de) does not offer /pub/FreeBSD at all. And one does not offer HTTP access, so if I switched back to FTP, 3 quarters of my problem would disappear and I'd have 8 working mirrors instead of 5 out of 9. So I should probably justify my use of HTTP: - Less latency (important for small downloads) - 8-stable/All/automounter-1.4.3.tbz with the same connection: HTTP: 0.20s, 0.20s, 0.20s FTP: 0.39s, 0.62s, 0.39s, 0.39s (I'd throw the 0.62s away as a glitch) - It needs 77 packages to install firefox, 70 of these are below 1.5m, which is a small file by broadband standards, i.e. for 90% of the packages latency is an important parameter - Only one connection per file instead of two (a server can run out of ports twice as fast with FTP), unless I am very mistaken about how HTTP and FTP work Regards -- A: Because it fouls the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing on usenet and in e-mail? From owner-freebsd-hubs@FreeBSD.ORG Tue Nov 9 12:30:51 2010 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4EA9D1065673 for ; Tue, 9 Nov 2010 12:30:51 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (lurza.secnetix.de [IPv6:2a01:170:102f::2]) by mx1.freebsd.org (Postfix) with ESMTP id C32CD8FC15 for ; Tue, 9 Nov 2010 12:30:50 +0000 (UTC) Received: from lurza.secnetix.de (localhost [127.0.0.1]) by lurza.secnetix.de (8.14.3/8.14.3) with ESMTP id oA9CUXqk075333; Tue, 9 Nov 2010 13:30:48 +0100 (CET) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.14.3/8.14.3/Submit) id oA9CUXVV075331; Tue, 9 Nov 2010 13:30:33 +0100 (CET) (envelope-from olli) From: Oliver Fromme Message-Id: <201011091230.oA9CUXVV075331@lurza.secnetix.de> To: kamikaze@bsdforen.de (Dominic Fandrey) Date: Tue, 9 Nov 2010 13:30:32 +0100 (CET) In-Reply-To: <4CD91428.5090002@bsdforen.de> X-Mailer: ELM [version 2.5 PL8] MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.3.5 (lurza.secnetix.de [127.0.0.1]); Tue, 09 Nov 2010 13:30:48 +0100 (CET) Cc: freebsd-hubs@freebsd.org Subject: Re: /pub/FreeBSD or /FreeBSD X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Nov 2010 12:30:51 -0000 Dominic Fandrey wrote: > I have an application that does parallel package downloads from > a list of mirrors and the different Layouts cause me some trouble. > > A lot of mirrors locate everything in /FreeBSD instead of > /pub/FreeBSD. Mirrors that don't follow the /pub/FreeBSD layout > are currently subject to unnecessary load caused by my application > and cause significant CPU load on my side. Traditionally the FreeBSD tree was placed in /pub/FreeBSD because that's the location where sysinstall looks for the installation sets and packages. It can be a symlink, too. You can find the set of paths where sysinstall looks for the installation data on FTP servers at the beginning of the file src/usr.sbin/sysinstall/ftp.c (reformatted): /* List of sub directories to look for under a given FTP server. */ const char *ftp_dirs[] = { ".", "releases/"MACHINE, "snapshots/"MACHINE, "pub/FreeBSD", "pub/FreeBSD/releases/"MACHINE, "pub/FreeBSD/snapshots/"MACHINE, NULL }; where MACHINE is "i386", "amd64" etc. The first of those directories that does exists and contains a subdirectory naming the desired release (e.g. 8.2-RELEASE) will be used to download the installation data. Typically, that's the second to last entry: /pub/FreeBSD/releases/8.2-RELEASE A similar algorithm is used to locate packages, distfiles for ports, documentation files and possibly other data. There are some mirrors that have the FreeBSD data somewhere else, including /FreeBSD, for example. If one of those is used for installation with sysinstall, the FTP path has to be changed manually in the options screen of sysinstall. I think all of the "official" mirrors (i.e. those who can be found in the .freebsd.org domain) use /pub/FreeBSD, so it works with sysinstall out of the box, without having to modify any options. As mentioned above, it can also be a symlink. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "Perl will consistently give you what you want, unless what you want is consistency." -- Larry Wall From owner-freebsd-hubs@FreeBSD.ORG Tue Nov 9 12:48:33 2010 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AE44E1065670 for ; Tue, 9 Nov 2010 12:48:33 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (lurza.secnetix.de [IPv6:2a01:170:102f::2]) by mx1.freebsd.org (Postfix) with ESMTP id 2A0378FC12 for ; Tue, 9 Nov 2010 12:48:32 +0000 (UTC) Received: from lurza.secnetix.de (localhost [127.0.0.1]) by lurza.secnetix.de (8.14.3/8.14.3) with ESMTP id oA9CmFWd075924; Tue, 9 Nov 2010 13:48:30 +0100 (CET) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.14.3/8.14.3/Submit) id oA9CmFB9075923; Tue, 9 Nov 2010 13:48:15 +0100 (CET) (envelope-from olli) From: Oliver Fromme Message-Id: <201011091248.oA9CmFB9075923@lurza.secnetix.de> To: kamikaze@bsdforen.de (Dominic Fandrey) Date: Tue, 9 Nov 2010 13:48:15 +0100 (CET) In-Reply-To: <4CD93BE1.6070705@bsdforen.de> X-Mailer: ELM [version 2.5 PL8] MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.3.5 (lurza.secnetix.de [127.0.0.1]); Tue, 09 Nov 2010 13:48:31 +0100 (CET) Cc: freebsd-hubs@freebsd.org Subject: Re: /pub/FreeBSD or /FreeBSD X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Nov 2010 12:48:33 -0000 Dominic Fandrey wrote: > I think I might have made a mistake here. I checked again and my > estimate for half the servers was a) grossly exaggerated and b) > the issue is more present over http. > > Those don't work: > http://ftp.de.freebsd.org/pub/FreeBSD/ > http://ftp3.de.freebsd.org/pub/FreeBSD/ > http://ftp4.de.freebsd.org/pub/FreeBSD/ > http://ftp6.de.freebsd.org/pub/FreeBSD/ (no HTTP at all) > > These work: > ftp://ftp.de.freebsd.org/pub/FreeBSD/ > ftp://ftp3.de.freebsd.org/pub/FreeBSD/ > ftp://ftp6.de.freebsd.org/pub/FreeBSD/ > > So, I have three distinct problems with half the German mirrors, > two offer /pub/FreeBSD only through FTP, but not HTTP. It is desirable that all FTP mirrors also offer the data via HTTP (and possibly also rsync or other protocols), but it is not a "must". So, it is not surprising that some mirrors do not support HTTP. > One (ftp4.de) does not offer /pub/FreeBSD at all. I consider that a bug. By the way, there's a mailing list specifically for the German mirrors: de-bsd-hubs@de.freebsd.org (posting in either German or English is ok). > And one does not offer HTTP access, so if I switched back to FTP, > 3 quarters of my problem would disappear and I'd have 8 working > mirrors instead of 5 out of 9. > > So I should probably justify my use of HTTP: > - Less latency (important for small downloads) > - 8-stable/All/automounter-1.4.3.tbz with the same connection: > HTTP: 0.20s, 0.20s, 0.20s > FTP: 0.39s, 0.62s, 0.39s, 0.39s (I'd throw the 0.62s away as a glitch) > - It needs 77 packages to install firefox, 70 of these are below 1.5m, > which is a small file by broadband standards, i.e. for 90% of > the packages latency is an important parameter It depends very much on how the protocols are implemented by server and client. If you use HTTP and *both* server and client support keep-alive, all files can be downloaded with a single connection. Otherwise you need n connections (e.g. 77 connections for 77 files). Some clients also open several connections in parallel. With FTP, you have to use one control connection, and one data connection per file, so you have n+1 connections total (e.g. 78 connections for 77 files). So it's not that much different from the non-keep-alive case. I think FreeBSD's fetch(3) does not support keep-alive. Besides the number of connections, the overhead for FTP is slightly larger. That's because HTTP is a simple request- reply protocol, while FTP requires an additional round-trip because of the negotiation of the port number for the data connection (PORT or PASV commands, depending on whether active or passive connections are used). Since FreeBSD's fetch(3) (used by sysinstall, pkg_add and the ports framework) does neither support keep-alive nor parallel downloads, the difference between HTTP and FTP is very small in practice. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd Python is executable pseudocode. Perl is executable line noise. From owner-freebsd-hubs@FreeBSD.ORG Tue Nov 9 13:33:40 2010 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2F0301065675 for ; Tue, 9 Nov 2010 13:33:40 +0000 (UTC) (envelope-from kamikaze@bsdforen.de) Received: from mail.server1.bsdforen.de (bsdforen.de [82.193.243.81]) by mx1.freebsd.org (Postfix) with ESMTP id A58F58FC15 for ; Tue, 9 Nov 2010 13:33:39 +0000 (UTC) Received: from mobileKamikaze.norad (vpn-cl-162-46.rz.uni-karlsruhe.de [141.3.162.46]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mail.server1.bsdforen.de (Postfix) with ESMTPSA id 6AEC885FCC; Tue, 9 Nov 2010 14:33:38 +0100 (CET) Message-ID: <4CD94DB1.3030809@bsdforen.de> Date: Tue, 09 Nov 2010 14:33:37 +0100 From: Dominic Fandrey User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-GB; rv:1.9.1.15) Gecko/20101028 Thunderbird/3.0.10 MIME-Version: 1.0 To: Oliver Fromme References: <201011091248.oA9CmFB9075923@lurza.secnetix.de> In-Reply-To: <201011091248.oA9CmFB9075923@lurza.secnetix.de> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-hubs@freebsd.org Subject: Re: /pub/FreeBSD or /FreeBSD X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Nov 2010 13:33:40 -0000 On 09/11/2010 13:48, Oliver Fromme wrote: > > Dominic Fandrey wrote: > > I think I might have made a mistake here. I checked again and my > > estimate for half the servers was a) grossly exaggerated and b) > > the issue is more present over http. > > > > Those don't work: > > http://ftp.de.freebsd.org/pub/FreeBSD/ > > http://ftp3.de.freebsd.org/pub/FreeBSD/ > > http://ftp4.de.freebsd.org/pub/FreeBSD/ > > http://ftp6.de.freebsd.org/pub/FreeBSD/ (no HTTP at all) > > > > These work: > > ftp://ftp.de.freebsd.org/pub/FreeBSD/ > > ftp://ftp3.de.freebsd.org/pub/FreeBSD/ > > ftp://ftp6.de.freebsd.org/pub/FreeBSD/ > > > > So, I have three distinct problems with half the German mirrors, > > two offer /pub/FreeBSD only through FTP, but not HTTP. > > It is desirable that all FTP mirrors also offer the data > via HTTP (and possibly also rsync or other protocols), > but it is not a "must". So, it is not surprising that > some mirrors do not support HTTP. 3 of these 4 however have HTTP support. Shouldn't they follow the expected pattern if they support HTTP? Probably their HTTP server just doesn't follow Symlinks. > > One (ftp4.de) does not offer /pub/FreeBSD at all. > > I consider that a bug. Thank you. > By the way, there's a mailing list specifically for the > German mirrors: de-bsd-hubs@de.freebsd.org (posting in > either German or English is ok). I also encountered this with other mirrors, I just didn't bother to test them all. Maybe I'll do a small probing script and provide a complete list. > > And one does not offer HTTP access, so if I switched back to FTP, > > 3 quarters of my problem would disappear and I'd have 8 working > > mirrors instead of 5 out of 9. > > > > So I should probably justify my use of HTTP: > > - Less latency (important for small downloads) > > - 8-stable/All/automounter-1.4.3.tbz with the same connection: > > HTTP: 0.20s, 0.20s, 0.20s > > FTP: 0.39s, 0.62s, 0.39s, 0.39s (I'd throw the 0.62s away as a glitch) > > - It needs 77 packages to install firefox, 70 of these are below 1.5m, > > which is a small file by broadband standards, i.e. for 90% of > > the packages latency is an important parameter > > It depends very much on how the protocols are implemented > by server and client. If you use HTTP and *both* server > and client support keep-alive, all files can be downloaded > with a single connection. Otherwise you need n connections > (e.g. 77 connections for 77 files). Some clients also open > several connections in parallel. > > With FTP, you have to use one control connection, and one > data connection per file, so you have n+1 connections total > (e.g. 78 connections for 77 files). So it's not that much > different from the non-keep-alive case. I think FreeBSD's > fetch(3) does not support keep-alive. The client is fetch, my statements were based on that. My application tries to always have one running download per mirror (if there are enough files to be downloaded, at least). I'd have to give up the single queue approach if I wanted to use a keep-alive capability, if fetch supported it. And I think single queue is the best approach to make the best out of my internet connection. > ... > > Since FreeBSD's fetch(3) (used by sysinstall, pkg_add and > the ports framework) does neither support keep-alive nor > parallel downloads, the difference between HTTP and FTP is > very small in practice. I measured that it doesn't matter at all for downloads like openoffice, this supports your assessment. -- A: Because it fouls the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing on usenet and in e-mail? From owner-freebsd-hubs@FreeBSD.ORG Tue Nov 9 13:42:34 2010 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E93731065693 for ; Tue, 9 Nov 2010 13:42:34 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (lurza.secnetix.de [IPv6:2a01:170:102f::2]) by mx1.freebsd.org (Postfix) with ESMTP id 519ED8FC0A for ; Tue, 9 Nov 2010 13:42:34 +0000 (UTC) Received: from lurza.secnetix.de (localhost [127.0.0.1]) by lurza.secnetix.de (8.14.3/8.14.3) with ESMTP id oA9DgFfG078409; Tue, 9 Nov 2010 14:42:30 +0100 (CET) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.14.3/8.14.3/Submit) id oA9DgF5V078407; Tue, 9 Nov 2010 14:42:15 +0100 (CET) (envelope-from olli) From: Oliver Fromme Message-Id: <201011091342.oA9DgF5V078407@lurza.secnetix.de> To: kamikaze@bsdforen.de (Dominic Fandrey) Date: Tue, 9 Nov 2010 14:42:15 +0100 (CET) In-Reply-To: <4CD94DB1.3030809@bsdforen.de> X-Mailer: ELM [version 2.5 PL8] MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.3.5 (lurza.secnetix.de [127.0.0.1]); Tue, 09 Nov 2010 14:42:31 +0100 (CET) Cc: freebsd-hubs@freebsd.org Subject: Re: /pub/FreeBSD or /FreeBSD X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Nov 2010 13:42:35 -0000 Dominic Fandrey wrote: > On 09/11/2010 13:48, Oliver Fromme wrote: > > It is desirable that all FTP mirrors also offer the data > > via HTTP (and possibly also rsync or other protocols), > > but it is not a "must". So, it is not surprising that > > some mirrors do not support HTTP. > > 3 of these 4 however have HTTP support. Shouldn't they follow the > expected pattern if they support HTTP? Yes, that would be desirable. The HTTP code in sysinstall uses the same paths as the FTP code. > Probably their HTTP server just doesn't follow Symlinks. Maybe, I don't know. I can only say for sure that it works fine on ftp7.de.freebsd.org because I wrote the CGI program that implements the HTTP access to the FTP tree. :-) > I also encountered this with other mirrors, I just didn't bother > to test them all. Maybe I'll do a small probing script and provide > a complete list. You're not the first one who does that. :-) Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "I have stopped reading Stephen King novels. Now I just read C code instead." -- Richard A. O'Keefe From owner-freebsd-hubs@FreeBSD.ORG Tue Nov 9 13:53:41 2010 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BA5AD106564A for ; Tue, 9 Nov 2010 13:53:41 +0000 (UTC) (envelope-from kamikaze@bsdforen.de) Received: from mail.server1.bsdforen.de (bsdforen.de [82.193.243.81]) by mx1.freebsd.org (Postfix) with ESMTP id 770198FC08 for ; Tue, 9 Nov 2010 13:53:41 +0000 (UTC) Received: from mobileKamikaze.norad (vpn-cl-162-46.rz.uni-karlsruhe.de [141.3.162.46]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mail.server1.bsdforen.de (Postfix) with ESMTPSA id 3517C7E8CB; Tue, 9 Nov 2010 14:53:39 +0100 (CET) Message-ID: <4CD95263.2070208@bsdforen.de> Date: Tue, 09 Nov 2010 14:53:39 +0100 From: Dominic Fandrey User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-GB; rv:1.9.1.15) Gecko/20101028 Thunderbird/3.0.10 MIME-Version: 1.0 To: Oliver Fromme References: <201011091230.oA9CUXVV075331@lurza.secnetix.de> In-Reply-To: <201011091230.oA9CUXVV075331@lurza.secnetix.de> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-hubs@freebsd.org Subject: Re: /pub/FreeBSD or /FreeBSD X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Nov 2010 13:53:41 -0000 On 09/11/2010 13:30, Oliver Fromme wrote: > > Dominic Fandrey wrote: > > I have an application that does parallel package downloads from > > a list of mirrors and the different Layouts cause me some trouble. > > > > A lot of mirrors locate everything in /FreeBSD instead of > > /pub/FreeBSD. Mirrors that don't follow the /pub/FreeBSD layout > > are currently subject to unnecessary load caused by my application > > and cause significant CPU load on my side. > > Traditionally the FreeBSD tree was placed in /pub/FreeBSD > because that's the location where sysinstall looks for the > installation sets and packages. It can be a symlink, too. > > You can find the set of paths where sysinstall looks for > the installation data on FTP servers at the beginning of > the file src/usr.sbin/sysinstall/ftp.c (reformatted): > > /* List of sub directories to look for under a given FTP server. */ > const char *ftp_dirs[] = { > ".", > "releases/"MACHINE, > "snapshots/"MACHINE, > "pub/FreeBSD", > "pub/FreeBSD/releases/"MACHINE, > "pub/FreeBSD/snapshots/"MACHINE, > NULL > }; > > where MACHINE is "i386", "amd64" etc. The first of those > directories that does exists and contains a subdirectory > naming the desired release (e.g. 8.2-RELEASE) will be used > to download the installation data. Typically, that's the > second to last entry: /pub/FreeBSD/releases/8.2-RELEASE Thanks a lot, I'm currently using: /pub/FreeBSD/ports/$ARCH/packages-$BRANCH/Latest for the PACKAGESITE logic. I don't know what to make of this, yet. But I think this might turn out to be useful information. > A similar algorithm is used to locate packages, distfiles > for ports, documentation files and possibly other data. I never intended probing in my architecture. It will cause me some headaches, but I think I will have to add it. > There are some mirrors that have the FreeBSD data somewhere > else, including /FreeBSD, for example. If one of those is > used for installation with sysinstall, the FTP path has to > be changed manually in the options screen of sysinstall. > > I think all of the "official" mirrors (i.e. those who can > be found in the .freebsd.org domain) use /pub/FreeBSD, so > it works with sysinstall out of the box, without having to > modify any options. As mentioned above, it can also be a > symlink. Does sysinstall support http, too? I'd expect it to if it uses fetch(3). Any way, I think I have to write a script to discern the magnitude of the issue. Regards, Dominic -- A: Because it fouls the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing on usenet and in e-mail? From owner-freebsd-hubs@FreeBSD.ORG Tue Nov 9 13:59:53 2010 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4927C1065670 for ; Tue, 9 Nov 2010 13:59:53 +0000 (UTC) (envelope-from kamikaze@bsdforen.de) Received: from mail.server1.bsdforen.de (bsdforen.de [82.193.243.81]) by mx1.freebsd.org (Postfix) with ESMTP id 062908FC12 for ; Tue, 9 Nov 2010 13:59:52 +0000 (UTC) Received: from mobileKamikaze.norad (vpn-cl-162-46.rz.uni-karlsruhe.de [141.3.162.46]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mail.server1.bsdforen.de (Postfix) with ESMTPSA id C4BB685FA4; Tue, 9 Nov 2010 14:59:51 +0100 (CET) Message-ID: <4CD953D7.5090001@bsdforen.de> Date: Tue, 09 Nov 2010 14:59:51 +0100 From: Dominic Fandrey User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-GB; rv:1.9.1.15) Gecko/20101028 Thunderbird/3.0.10 MIME-Version: 1.0 To: Oliver Fromme References: <201011091342.oA9DgF5V078407@lurza.secnetix.de> In-Reply-To: <201011091342.oA9DgF5V078407@lurza.secnetix.de> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-hubs@freebsd.org Subject: Re: /pub/FreeBSD or /FreeBSD X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Nov 2010 13:59:53 -0000 On 09/11/2010 14:42, Oliver Fromme wrote: > > Dominic Fandrey wrote: > > On 09/11/2010 13:48, Oliver Fromme wrote: > > > It is desirable that all FTP mirrors also offer the data > > > via HTTP (and possibly also rsync or other protocols), > > > but it is not a "must". So, it is not surprising that > > > some mirrors do not support HTTP. > > > > 3 of these 4 however have HTTP support. Shouldn't they follow the > > expected pattern if they support HTTP? > > Yes, that would be desirable. The HTTP code in sysinstall > uses the same paths as the FTP code. > > > Probably their HTTP server just doesn't follow Symlinks. > > Maybe, I don't know. I can only say for sure that it works > fine on ftp7.de.freebsd.org because I wrote the CGI program > that implements the HTTP access to the FTP tree. :-) > > > I also encountered this with other mirrors, I just didn't bother > > to test them all. Maybe I'll do a small probing script and provide > > a complete list. > > You're not the first one who does that. :-) Oh, maybe I /should/ register to the list. A quick search yielded no results, what would be the right keywords to search the mailing list for? Regards, Dominic -- A: Because it fouls the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing on usenet and in e-mail? From owner-freebsd-hubs@FreeBSD.ORG Tue Nov 9 15:39:27 2010 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 47E5A106566C for ; Tue, 9 Nov 2010 15:39:27 +0000 (UTC) (envelope-from jason@wilma.widomaker.com) Received: from wilma.widomaker.com (wilma.widomaker.com [204.17.220.5]) by mx1.freebsd.org (Postfix) with ESMTP id 1701C8FC16 for ; Tue, 9 Nov 2010 15:39:27 +0000 (UTC) Received: from wilma.widomaker.com ([204.17.220.5]) by wilma.widomaker.com with esmtp (Exim 4.69 (FreeBSD)) (envelope-from ) id 1PFq3J-000KiD-NZ; Tue, 09 Nov 2010 10:24:11 -0500 Received: (from jason@localhost) by wilma.widomaker.com (8.14.3/8.14.3/Submit) id oA9FNLYA059986; Tue, 9 Nov 2010 11:23:21 -0400 (AST) (envelope-from jason) Date: Tue, 9 Nov 2010 11:23:18 -0400 From: Jason Harris To: Dominic Fandrey Message-ID: <20101109152318.GA59055@laptop> References: <201011091248.oA9CmFB9075923@lurza.secnetix.de> <4CD94DB1.3030809@bsdforen.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Q68bSM7Ycu6FN28Q" Content-Disposition: inline In-Reply-To: <4CD94DB1.3030809@bsdforen.de> User-Agent: Mutt/1.5.19 (2009-01-05) X-Spam-Score: -1.4 (-) X-Spam-Report: This message has been scanned & scored by widomaker.com's mail servers. The information added to this message is to allow you to chose whether or not you accept this message. For more information, contact helpdesk@widomaker.com or see http://www.widomaker.com/filterspam.html Content analysis details: (-1.4 points, 8.0 required) ALL_TRUSTED=-1.44,AWL=0.048 Cc: freebsd-hubs@freebsd.org, Oliver Fromme Subject: Re: /pub/FreeBSD or /FreeBSD X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Nov 2010 15:39:27 -0000 --Q68bSM7Ycu6FN28Q Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Nov 09, 2010 at 02:33:37PM +0100, Dominic Fandrey wrote: > On 09/11/2010 13:48, Oliver Fromme wrote: > > It depends very much on how the protocols are implemented > > by server and client. If you use HTTP and *both* server > > and client support keep-alive, all files can be downloaded > > with a single connection. Otherwise you need n connections > > (e.g. 77 connections for 77 files). Some clients also open > > several connections in parallel. I have been testing aria2c (ports/www/aria2) for parallel-fetching (1 connection per site, up to 32 sites) FreeBSD ports distfiles with the following ~/.aria2/aria2.conf: server-stat-if=3Dserver-stats server-stat-of=3Dserver-stats min-split-size=3D1M allow-overwrite=3Dtrue uri-selector=3Dadaptive max-concurrent-downloads=3D32 timeout=3D15 connect-timeout=3D15 lowest-speed-limit=3D40K remote-time=3Dtrue server-stat-timeout=3D86313600 split=3D32 enable-http-pipelining=3Dtrue use-head=3Dtrue event-poll=3Dkqueue file-allocation=3Dnone log-level=3Dnotice (NB: This creates ./server-stats, usually in /usr/ports/distfiles, with a few changes to /etc/make.conf and a patched ports/Mk/bsd.port.mk.) It works quite well, often saturating my 4Mbit (down) link. > My application tries to always have one running download per > mirror (if there are enough files to be downloaded, at least). >=20 > I'd have to give up the single queue approach if I wanted to > use a keep-alive capability, if fetch supported it. And I > think single queue is the best approach to make the best out > of my internet connection. You can control instances of aria2c with XML-RPC (samples included in the man page). That would keep connections open and actively downloading for as long as possible. If you use one instance per mirror, use a different server-stats and you can check each mirror's overall performance when each aria2c finally exit()s. > I measured that it doesn't matter at all for downloads like > openoffice, this supports your assessment. OpenOffice uses MirrorBrain ( http://www.mirrorbrain.org/users/ ), which generates Metalink and BitTorrent files for built-in parallel downloading. Also, try "make fetch-url-list | xargs aria2c" in ports/www/firefox to see what is possible with a small patch to FreeBSD's existing ports infrastructure. --=20 Jason Harris | PGP: This _is_ PGP-signed, isn't it? jharris@widomaker.com _|_ Got photons? (TM), (C) 2004 --Q68bSM7Ycu6FN28Q Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iJ0EARECAF0FAkzZZ2ZWGGh0dHA6Ly9rZXlzZXJ2ZXIua2pzbC5jb206MTEzNzEv cGtzL2xvb2t1cD9vcD1nZXQmc2VhcmNoPTB4RDM5REEwRTMmd2VoYXZleW91bm93 PXRydWUACgkQSypIl9OdoON0QwCdFu/VizMoA6Q0aIo7BtTu8PiZ+54AoLC9pAmm ho75uRyCaC3GWRFVh7C5 =0v+w -----END PGP SIGNATURE----- --Q68bSM7Ycu6FN28Q-- From owner-freebsd-hubs@FreeBSD.ORG Tue Nov 9 17:22:56 2010 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6B9131065670 for ; Tue, 9 Nov 2010 17:22:56 +0000 (UTC) (envelope-from kensmith@buffalo.edu) Received: from localmailC.acsu.buffalo.edu (localmailC.acsu.buffalo.edu [128.205.5.204]) by mx1.freebsd.org (Postfix) with ESMTP id 3B7098FC21 for ; Tue, 9 Nov 2010 17:22:55 +0000 (UTC) Received: from localmailC.acsu.buffalo.edu (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id 096E92EEC4 for ; Tue, 9 Nov 2010 12:03:36 -0500 (EST) Received: from localmailC.acsu.buffalo.edu (localhost [127.0.0.1]) by localmailC.acsu.buffalo.edu (Postfix) with ESMTP id 8FB302F019 for ; Tue, 9 Nov 2010 12:03:35 -0500 (EST) Received: from mweb1.acsu.buffalo.edu (mweb1.acsu.buffalo.edu [128.205.5.238]) by localmailC.acsu.buffalo.edu (Prefixe) with ESMTP id 885622EEC4 for ; Tue, 9 Nov 2010 12:03:35 -0500 (EST) Received: from [128.205.32.76] (bauer.cse.buffalo.edu [128.205.32.76]) by mweb1.acsu.buffalo.edu (Postfix) with ESMTP id 5D08A5B0038 for ; Tue, 9 Nov 2010 12:03:35 -0500 (EST) From: Ken Smith To: freebsd-hubs@freebsd.org Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="=-N9CXoLS6O8omU2RB/zAm" Date: Tue, 09 Nov 2010 12:03:35 -0500 Message-ID: <1289322215.32929.25.camel@bauer.cse.buffalo.edu> Mime-Version: 1.0 X-Mailer: Evolution 2.28.2 FreeBSD GNOME Team Port X-PM-EL-Spam-Prob: : 8% Subject: Cleanup of FTP site X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Nov 2010 17:22:56 -0000 --=-N9CXoLS6O8omU2RB/zAm Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Just a quick heads-up to say we'll be cleaning out some of the older stuff on the FTP site this coming Thursday (November 11th, 2010) in preparation for the upcoming 7.4/8.2 release. I have made sure the ftp-archive site has caught up to all of the releases thus far. I'll be removing from the main FTP site the release and package bits for the 7.2 and 8.0 releases. I'll also remove the ISO files for some of the older releases, I usually keep several of those on the main FTP site even after the FTP-install and package trees get removed. Thanks. =20 --=20 Ken Smith - From there to here, from here to | kensmith@buffalo.edu there, funny things are everywhere. | - Theodor Geisel | --=-N9CXoLS6O8omU2RB/zAm Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (FreeBSD) iEYEABECAAYFAkzZftwACgkQ/G14VSmup/YhUQCeNbCARfIEdoy3bzIujQQcQDOq pUUAniZr6cuL01fHQ+WpN4eK6z9s09sf =TllC -----END PGP SIGNATURE----- --=-N9CXoLS6O8omU2RB/zAm-- From owner-freebsd-hubs@FreeBSD.ORG Thu Nov 11 16:16:26 2010 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AC1511065674 for ; Thu, 11 Nov 2010 16:16:26 +0000 (UTC) (envelope-from kensmith@buffalo.edu) Received: from localmailC.acsu.buffalo.edu (localmail.buffalo.edu [128.205.5.204]) by mx1.freebsd.org (Postfix) with ESMTP id 797748FC16 for ; Thu, 11 Nov 2010 16:16:26 +0000 (UTC) Received: from localmailC.acsu.buffalo.edu (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id A078710816; Thu, 11 Nov 2010 11:16:25 -0500 (EST) Received: from localmailC.acsu.buffalo.edu (localhost [127.0.0.1]) by localmailC.acsu.buffalo.edu (Postfix) with ESMTP id 863CC27597; Thu, 11 Nov 2010 11:16:24 -0500 (EST) Received: from mweb1.acsu.buffalo.edu (mweb1.acsu.buffalo.edu [128.205.5.238]) by localmailC.acsu.buffalo.edu (Prefixe) with ESMTP id 72C0427D7A; Thu, 11 Nov 2010 11:16:24 -0500 (EST) Received: from [128.205.32.76] (bauer.cse.buffalo.edu [128.205.32.76]) by mweb1.acsu.buffalo.edu (Postfix) with ESMTP id 5981F5B0038; Thu, 11 Nov 2010 11:16:24 -0500 (EST) From: Ken Smith To: Patrick Bihan-Faou In-Reply-To: <4CC81EE7.4020607@teambox.fr> References: <4CC81EE7.4020607@teambox.fr> Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="=-rRGy5FqieyPbAEqefZe7" Date: Thu, 11 Nov 2010 11:16:23 -0500 Message-ID: <1289492183.40288.37.camel@bauer.cse.buffalo.edu> Mime-Version: 1.0 X-Mailer: Evolution 2.28.2 FreeBSD GNOME Team Port X-PM-EL-Spam-Prob: : 8% Cc: freebsd-hubs@freebsd.org, freebsd-questions@freebsd.org Subject: Re: Sorry state of the rsync based CVS,replication X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Nov 2010 16:16:26 -0000 --=-rRGy5FqieyPbAEqefZe7 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable On Wed, 2010-10-27 at 14:45 +0200, Patrick Bihan-Faou wrote: > We use FreeBSD extensively and keep a local mirror of the CVS=20 > repository. Up until recently things where working properly with the=20 > various servers listed in=20 > http://www.freebsd.org/doc/handbook/mirrors-rsync.html, but sometime=20 > during the summer ftp13.freebsd.org did not respond anymore and since=20 > then rsync replication is broken. >=20 > The main issue (besides the removal of ftp13.freebsd.org) is that most=20 > rsync sources refuse to replicate the content of the .Attic directories=20 > in the CVS tree. This means that performing a check-out on ports using a=20 > tag usually won't work as some files will not be there anymore. >=20 > Here are the typical logs I get using most rsync servers: >=20 > rsync: opendir=20 > "/3/freebsd-core/development/FreeBSD-CVS/ports/chinese/pcmanx/files/Attic= "=20 > (in vol) failed: Permission denied (13) >=20 > At this moment the only rsync server that provides an adequate=20 > replication of the CVS repository is ftp2.tw.FreeBSD.org. As others have reported this was caused by the permissions on the Attic directories not including world read permission. For sites where it was working it's actually an indication they're not following "best practices" for a mirror site. It's typically a bad idea to have the thing that allows access to the content of the mirror site running with the same credentials as what keeps the mirror site up to date. We don't use the 'feature' that allow for (pre-staging content that the world shouldn't have access to for a period of time, allowing the mirror sites to get fully populated before the release date) but I know of other projects that do. The ftp-master machines don't have that in place because they're not public and they need to allow the blessed mirrors access to everything (for the purposes of pre-staging, if we were actually using that feature...). The development/ section of the FTP site is something I hadn't looked at before so it took me a little time to find what populates it and investigate a little. I *think* the issue with the Attic directories not including world-read permissions was either an issue with a badly formed chmod(1) done a long time ago or an issue with the mechanism that populates that portion of the FTP site missing a umask setting in the script that does it some time back in history (it's there now). Not all of the Attic directories had the wrong permissions, it seemed to stop some time in 2007. I adjusted the permissions on ftp-master so hopefully this issue is fixed. However ... > We are moving to svn and svnsync for the freebsd source tree (and I am=20 > happy with this), but the ports do not seem to be available using SVN=20 > (or not in a documented way). >=20 > Can something be done to restore RSYNC mirroring of the CVS tree to a=20 > working state ? The FTP site desperately needs to go on a diet so we're poking around to see if there is some stuff that can be dropped. This section of the site is a candidate for being removed. As you say the ports are not available in SVN but I'm curious why you use the content from the FTP site instead of just using a CVSUP mirror. Is there some benefit to it? We would sort of like to stop providing this as part of the FTP site if there really isn't any benefit to it over using the cvsup mirror infrastructure which won't be going away any time soon. Thanks. --=20 Ken Smith - From there to here, from here to | kensmith@buffalo.edu there, funny things are everywhere. | - Theodor Geisel | --=-rRGy5FqieyPbAEqefZe7 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (FreeBSD) iEYEABECAAYFAkzcFtcACgkQ/G14VSmup/bbsACeJrhgAozbBHDUoKbHlMQcUozm STgAnR0cVSOwDHvo5obcUITETkP46A7T =BwSW -----END PGP SIGNATURE----- --=-rRGy5FqieyPbAEqefZe7-- From owner-freebsd-hubs@FreeBSD.ORG Thu Nov 11 19:39:09 2010 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 995CB106564A for ; Thu, 11 Nov 2010 19:39:09 +0000 (UTC) (envelope-from toasty@dragondata.com) Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id 5DB0B8FC19 for ; Thu, 11 Nov 2010 19:39:09 +0000 (UTC) Received: by gyg13 with SMTP id 13so277523gyg.13 for ; Thu, 11 Nov 2010 11:39:08 -0800 (PST) Received: by 10.42.229.133 with SMTP id ji5mr1219247icb.128.1289504345188; Thu, 11 Nov 2010 11:39:05 -0800 (PST) Received: from vpn177.ord02.your.org (vpn177.ord02.your.org [204.9.55.177]) by mx.google.com with ESMTPS id gy41sm2776133ibb.23.2010.11.11.11.39.03 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 11 Nov 2010 11:39:03 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v1081) Content-Type: text/plain; charset=us-ascii From: Kevin Day In-Reply-To: <1289492183.40288.37.camel@bauer.cse.buffalo.edu> Date: Thu, 11 Nov 2010 13:39:01 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <2E6CACA4-04B1-44E3-9874-2D2BFFE53900@dragondata.com> References: <4CC81EE7.4020607@teambox.fr> <1289492183.40288.37.camel@bauer.cse.buffalo.edu> To: Ken Smith X-Mailer: Apple Mail (2.1081) Cc: freebsd-hubs@freebsd.org Subject: Re: Sorry state of the rsync based CVS,replication X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Nov 2010 19:39:09 -0000 On Nov 11, 2010, at 10:16 AM, Ken Smith wrote: >>=20 >=20 > As others have reported this was caused by the permissions on the > Attic directories not including world read permission. For sites > where it was working it's actually an indication they're not following > "best practices" for a mirror site. It's typically a bad idea to have > the thing that allows access to the content of the mirror site running > with the same credentials as what keeps the mirror site up to date. = We > don't use the 'feature' that allow for (pre-staging content that the > world shouldn't have access to for a period of time, allowing the = mirror > sites to get fully populated before the release date) but I know of > other projects that do. The ftp-master machines don't have that in > place because they're not public and they need to allow the blessed > mirrors access to everything (for the purposes of pre-staging, if we > were actually using that feature...). Just to avoid making this even more confusing to those wondering why it = was working in my post: This was working on ftp3.us because we grab a copy of the cvs repository = via cvsup, and present that as an rsyncable module.=20 Works: rsync -rav = ftp3.us.freebsd.org::FreeBSD-CVS/ncvs/ports/chinese/pcmanx/files/Attic . Doesn't: rsync -rav = ftp3.us.freebsd.org::FreeBSD/development/FreeBSD-CVS/ports/chinese/pcmanx/= files/Attic . receiving file list ... rsync: opendir = "/development/FreeBSD-CVS/ports/chinese/pcmanx/files/Attic" (in FreeBSD) = failed: Permission denied (13) "FreeBSD" is our mirror of ftp-master, and "FreeBSD-CVS" is a mirror of = our cvsup repo that's we get via the cvsup protocol(correct = permissions), not rsync mirrored from ftp-master.us(has unreadable attic = directories). -- Kevin From owner-freebsd-hubs@FreeBSD.ORG Fri Nov 12 08:47:46 2010 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 986A1106566B; Fri, 12 Nov 2010 08:47:46 +0000 (UTC) (envelope-from patrick.bihan-faou@teambox.fr) Received: from smtp.teambox.fr (dedibox.teambox.fr [88.191.109.88]) by mx1.freebsd.org (Postfix) with ESMTP id 2C1158FC16; Fri, 12 Nov 2010 08:47:45 +0000 (UTC) Received: from crest.teambox.fr (crest.mindstep.com [88.167.204.204]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: teambox) by smtp.teambox.fr (Postfix) with ESMTPSA id 18778A243FD; Fri, 12 Nov 2010 09:47:43 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by kawa.local.mindstep.fr (Postfix) with ESMTP id 5E2FDFDBE68; Fri, 12 Nov 2010 09:47:43 +0100 (CET) (envelope-from patrick.bihan-faou@teambox.fr) Received: from kawa.local.mindstep.fr ([127.0.0.1]) by localhost (kawa.local.mindstep.fr [127.0.0.1]) (amavisd-new, port 10024) with LMTP id kjPQmNw5w6yc; Fri, 12 Nov 2010 09:47:43 +0100 (CET) Received: from [127.0.0.1] (unknown [192.168.25.162]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by kawa.local.mindstep.fr (Postfix) with ESMTP id 3031CFDB8E9; Fri, 12 Nov 2010 09:47:43 +0100 (CET) (envelope-from patrick.bihan-faou@teambox.fr) Message-ID: <4CDCFF30.60307@teambox.fr> Date: Fri, 12 Nov 2010 09:47:44 +0100 From: Patrick Bihan-Faou Organization: TeamBox SARL User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; fr; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6 MIME-Version: 1.0 To: Ken Smith References: <4CC81EE7.4020607@teambox.fr> <1289492183.40288.37.camel@bauer.cse.buffalo.edu> In-Reply-To: <1289492183.40288.37.camel@bauer.cse.buffalo.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hubs@freebsd.org, freebsd-questions@freebsd.org Subject: Re: Sorry state of the rsync based CVS,replication X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Nov 2010 08:47:46 -0000 Hi, [...] Regarding the permission of the Attic subdirs in place > The development/ section of the FTP site is something I hadn't looked > at before so it took me a little time to find what populates it and > investigate a little. I *think* the issue with the Attic directories > not including world-read permissions was either an issue with a badly > formed chmod(1) done a long time ago or an issue with the mechanism > that populates that portion of the FTP site missing a umask setting > in the script that does it some time back in history (it's there now). > Not all of the Attic directories had the wrong permissions, it seemed > to stop some time in 2007. > > I adjusted the permissions on ftp-master so hopefully this issue is > fixed. However ... > Great news. I'll check various rsync source later and see if the situation improves. >> We are moving to svn and svnsync for the freebsd source tree (and I am >> happy with this), but the ports do not seem to be available using SVN >> (or not in a documented way). >> >> Can something be done to restore RSYNC mirroring of the CVS tree to a >> working state ? > The FTP site desperately needs to go on a diet so we're poking around > to see if there is some stuff that can be dropped. This section of > the site is a candidate for being removed. As you say the ports are > not available in SVN but I'm curious why you use the content from the > FTP site instead of just using a CVSUP mirror. Is there some benefit > to it? We would sort of like to stop providing this as part of the > FTP site if there really isn't any benefit to it over using the > cvsup mirror infrastructure which won't be going away any time soon. > The benefit is organisational to us. We did use cvsup in the past but it has been a pain to maintain as all the other external sources we keep in sync with use rsync. That combined with the requirement for m3 etc. for the sole purpose of syncing the CVS tree when there are alternatives (rsync for the freebsd cvs tree and more recently svnsync) made the switch to rsync a no brainer (note that this decision was taken long before csup came to life). Don't take this as flamebait, because I have no intention in starting a war on this particular issue, but as good as cvsup is, this is unfortunately a fairly isolated tool that, from my prospective (which is necessarily biaised and incomplete), does not offer any feature compelling enough to prefer it over rsync in our case. That position is by essence just a personal view, applicable to me only and not to anybody else. Also I have to admit that now that the m3 dependency is gone with csup, it becomes easier to return to it. Now if the plan is to eliminate rsync CVS mirroring (and I am in no position to criticize such a decision that is very well justified in your mail), we will of course adapt and go to one of the supported methods for CVS tree mirroring. If this means cvsup only, then we will do it without any hard feeling. I already figured that we must be almost the only ones to use rsync for CVS tree mirroring (otherwise that issue would have been detected and fixed a long time ago), and I don't expect (nor demand) the project to maintain a service for just one user, this would be ridiculous and unreasonable. Anyways, thank you very much for taking the time to look at this issue and hopefully having fixed it. This is really greatly appreciated. Best regards, Patrick. From owner-freebsd-hubs@FreeBSD.ORG Fri Nov 12 17:37:15 2010 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 869BF106566B for ; Fri, 12 Nov 2010 17:37:15 +0000 (UTC) (envelope-from zeising@lysator.liu.se) Received: from mail.lysator.liu.se (mail.lysator.liu.se [130.236.254.3]) by mx1.freebsd.org (Postfix) with ESMTP id 3AC208FC15 for ; Fri, 12 Nov 2010 17:37:15 +0000 (UTC) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id BF5EE40003 for ; Fri, 12 Nov 2010 18:11:38 +0100 (CET) Received: by mail.lysator.liu.se (Postfix, from userid 1674) id B081840007; Fri, 12 Nov 2010 18:11:38 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.1.7-deb3 (2006-10-05) on bernadotte.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-0.8 required=5.0 tests=ALL_TRUSTED, AWL, NO_REAL_NAME autolearn=disabled version=3.1.7-deb3 Received: from webmail.lysator.liu.se (mail.lysator.liu.se [130.236.254.3]) by mail.lysator.liu.se (Postfix) with ESMTP id 7C2B240003 for ; Fri, 12 Nov 2010 18:11:38 +0100 (CET) Received: from 79.136.90.99 (SquirrelMail authenticated user zeising) by webmail.lysator.liu.se with HTTP; Fri, 12 Nov 2010 18:11:38 +0100 (CET) Message-ID: <59297.79.136.90.99.1289581898.squirrel@webmail.lysator.liu.se> Date: Fri, 12 Nov 2010 18:11:38 +0100 (CET) From: zeising@lysator.liu.se To: freebsd-hubs@freebsd.org User-Agent: SquirrelMail/1.4.9a MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal X-Virus-Scanned: ClamAV using ClamSMTP Subject: New rsync mirror X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Nov 2010 17:37:15 -0000 Hi! ftp4.se.freebsd.org has grown rsync capabilities as well. Feel free to use it! To sync the FreeBSD ftp mirror use something like rsync -av --delete rsync://ftp4.se.freebsd.org::FreeBSD /local/path rsync rsync://ftp4.se.freebsd.org:: gives a list of all available modules It listens on both IPv4 and IPv6 addresses, and is located in Linköping, Sweden, on the SUNET backbone. Feel free to contact me if you have questions or comments, or if something seems to be broken. Best Regards! Niclas Zeising ftp-master for ftp.lysator.liu.se From owner-freebsd-hubs@FreeBSD.ORG Sat Nov 13 08:25:54 2010 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8549510656F4 for ; Sat, 13 Nov 2010 08:25:54 +0000 (UTC) (envelope-from grarpamp@gmail.com) Received: from mail-pv0-f182.google.com (mail-pv0-f182.google.com [74.125.83.182]) by mx1.freebsd.org (Postfix) with ESMTP id 5D79A8FC21 for ; Sat, 13 Nov 2010 08:25:54 +0000 (UTC) Received: by pvc22 with SMTP id 22so783598pvc.13 for ; Sat, 13 Nov 2010 00:25:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:date:message-id :subject:from:to:content-type; bh=wK3Rq5uxKNf0Ipz8dTi6s3C9zZO1PgWb46KMPMEJLuM=; b=DTDeQMCV/stpRJzOf4T/FkwyYmubbElkba6GuhS+D5KUJ0SYDCPTcFVpEaXIpJEgDH NhcUspqpkks/bXoXM/8XdiXxx6U6cOTKHSlG1loOned/M7pvD9bmhCHwGZhyz/G+Gp2h MtFq9ZzJsLnuW4Q8hirbKnqcdnq3X/AI38caE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=SvI3PIc6LvYi0R/Ph964pAyZuf+0S2iEtaczxodGAMaJWxrdrJ0ZDPmbQLhhO7g3jX qdgB6umCCPRFtS2+3jHfbxj/mFZxC8BhlvP6JscFITTNc0z+JjWDvoK8sW7W1n7Yzr7Q iHKBW6+iGUHvHmmhXgVc4uuSIGzt/dShUcCvY= MIME-Version: 1.0 Received: by 10.142.225.9 with SMTP id x9mr2946911wfg.116.1289635317668; Sat, 13 Nov 2010 00:01:57 -0800 (PST) Received: by 10.142.178.2 with HTTP; Sat, 13 Nov 2010 00:01:57 -0800 (PST) Date: Sat, 13 Nov 2010 03:01:57 -0500 Message-ID: From: grarpamp To: freebsd-hubs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: CVSup/csup vs. rsync [also git] X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Nov 2010 08:25:54 -0000 re: Sorry state of the rsync based CVS,replication " The benefit is organisational to us. We did use cvsup in the past but it has been a pain to maintain as all the other external sources we keep in sync with use rsync. That combined with the requirement for m3 etc. for the sole purpose of syncing the CVS tree when there are alternatives (rsync for the freebsd cvs tree and more recently svnsync) made the switch to rsync a no brainer (note that this decision was taken long before csup came to life). [snip more commonality] " I can second this. No other major project that I know of is seriously using cvsup... on the contrary, it's being abandoned. The FreeBSD cvsup mirrors are commonly missing distributions. And I can't see any real win over rsync -Haxi [-cz] --delete. While glad to see the dep on m3 go with csup, I still wondered aloud when it came out instead of rsync. I guess my thought is that if you're going to be using supfiles to do arbitrary tag/date checkouts, use the actual repo tool to talk directly to the repo. If you're just going to be saying give me release_x, releng_x, or head till the day I die... supply those via a looped check out on the mirror master and rsync those. Which when doing more than one tag wastes resources on both sides and between... so ultimately... These days, CPU, disk, and even net are cheap and the repo is small compared to that. I can't see any reason why people would *not* just mirror the entire repo and then checkout whatever revision they want from that. And there are long term efficiences with that too... mostly in build scripts and not having to update multiple trees over the wire. I've done perf testing that didn't make me fear rsync in full repo copy mode, be it in CPU, or disk, or net. The only real trick is making sure the master is setup to offer a commit atomic consistent view to the rsync daemon. It's just an intermediate copy, zfs snapshot, etc. And then of course there is git :) Would love to see those crypto hashes from repo to release, gitweb, etc. But I guess for some reason SVN won out for the time being back then. That's ok for now suppose. On the plus from the do away with legacy side... As SVN is the primary source for years now, I would support dropping CVS altogether just as soon as SVN replication is adequate and not worry about doubly supporting CVS anymore. CTM, etc. Having SVN via svn, svnsync, and rsync would be nice, perhaps even optionally over ssh. > The FTP site desperately needs to go on a diet Is that due to CPU, disk or net? Are stats available? > Don't take this as flamebait Ditto. Cheers :) I do hope to be able to provide an update of internal git+rsync (and even old CVS/SVN) tests that might inspire.