From owner-freebsd-hubs Mon Mar 31 16:56:17 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id QAA25052 for hubs-outgoing; Mon, 31 Mar 1997 16:56:17 -0800 (PST) Received: from time.cdrom.com (root@time.cdrom.com [204.216.27.226]) by freefall.freebsd.org (8.8.5/8.8.5) with ESMTP id QAA25028; Mon, 31 Mar 1997 16:56:08 -0800 (PST) Received: from time.cdrom.com (jkh@localhost [127.0.0.1]) by time.cdrom.com (8.8.5/8.6.9) with ESMTP id QAA13143; Mon, 31 Mar 1997 16:56:08 -0800 (PST) To: hubs@freebsd.org cc: www@freebsd.org Subject: Unified download stats [and better presentation of same] Date: Mon, 31 Mar 1997 16:56:07 -0800 Message-ID: <13139.859856167@time.cdrom.com> From: "Jordan K. Hubbard" Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk I've been getting more and more emails lately from folks asking me why we're not doing more to call attention to all of FreeBSD's cool features, it's growing popularity in the marketplace and so on. I sometimes wonder this too, and occasionally I even wander over to Red Hat's site and look at all the (paid, full-time) energy that they've managed to pour into their site and I get a little depressed about it, so rather than cry into my milk I decided to start doing at least *something* to make better use of our statistical information in marketing ourselves, and the first step (IMHO) is to get a better feel for how many copies of FreeBSD are being downloaded each day with the eventual intention of putting this information up on a web page. As a very rough and first-order attempt at this, for the last week or so I've been having ftp.freebsd.org send me little messages like this: From: "Root wcarchive.cdrom.com" Message-Id: <199703311111.DAA07942@wcarchive.cdrom.com> To: freebsd-stats@freebsd.org FreeBSD downloads over last 24 hours (up to Mar 31 03:01) from ftp.FreeBSD.org: 3.0-970209-SNAP 14 copies. 2.1.7.1-RELEASE 53 copies. 2.2.1-RELEASE 139 copies. and I have found it to be informative for keeping track of which releases seem to be popular and what seems to drive demand. It only shows ONE mirror site, however, and this is a drawback which we need to address before such statistics will become truly useful. We also need to make sure of the accuracy of our statistics, and this means that any mirrors which are mirroring things multiple times (I seem to recall Christoph Kukulies having some problems with this?) need to be identified and their duplicate entries filtered out of the raw data. I'm already suspicious that our numbers from ftp.freebsd.org may be inflated by as much as 10% due to bogus repetitive mirroring, and if anyone has any ideas on filtering this out of 500MB weekly xferlogs (I figure you'd need at least a week's granularity to do proper detection), I'm more than open to suggestions. I really do want these stats to be as accurate as possible, neither under or over inflated, since bogus stats help noone. I also need to somehow collect stats for all the mirrors, run once a day and sent to the ``freebsd-stats@freebsd.org'' alias I've just created on freefall. This currently fans out to just David Greenman, myself and Poul-Henning but it could just as easily be a mailing list for those interested in the same information. Ideally, one of our webfolk would also collect the info and chart it on the "FreeBSD Bragging Page" one of us (even if it has to be me :) will be putting up soon, right? :-) So, to summarize: 1. Anyone have a good way of detecting mirror looping from wu-ftpd style xferlogs? This would help both our bandwidth and our stats. 2. Would those hubs who are willing to participate in this be willing to run a little something from /etc/daily (or whatever crontab driven thing they have on their mirror machine) which summarizes the stats and sends it to the freebsd-stats@freebsd.org redistribution point? For stats collection, I'm currently just using this braindead little shell script: #!/bin/sh if [ $? -lt 1 ]; then file=/var/log/xferlog.day else file=$1 fi date=`ls -l $file | awk '{print $6 " " $7 " " $8}'` echo "FreeBSD downloads over last 24 hours (up to $date) from ftp.FreeBSD.org:" awk '/.16\/FreeBSD\/.*\/bin\/bin.aa/ { files[$9]++; } END { for (i in files) if (files[i] > 1) printf("\t%s\t%4d copies.\n", i, files[i]); }' $file | sed -e 's;/.16/FreeBSD/\(.*\)/bin/bin.aa;\1;' It's hard-coded for ftp.freebsd.org's FTP setup, of course, and the output it emits is more designed for humans than machine tabulation. Perhaps the stats counter we eventually come up with should have both? A nice textual summary and then a little embedded comment somewhere in it which just contains the raw numbers for a counter / web page generator's benefit would be ideal, I'd think. The ultimate goal here is, of course, to have a page at www.freebsd.org (and its mirrors) which shows the world-wide weekly download count, perhaps even with a little pie-chart showing it by country, and CD sales to some rough approximation. Assuming that the numbers are impressive, this would then be excellent fodder for people giving talks or trying to talk ISVs and other vendor into supporting FreeBSD in some way. Comments? Any perl programmers out here who'd care to give me an assist in producing the more definitive daily stats summarizer? :-) Jordan From owner-freebsd-hubs Mon Mar 31 18:42:29 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id SAA01782 for hubs-outgoing; Mon, 31 Mar 1997 18:42:29 -0800 (PST) Received: from panda.hilink.com.au (panda.hilink.com.au [203.2.144.5]) by freefall.freebsd.org (8.8.5/8.8.5) with ESMTP id SAA01775; Mon, 31 Mar 1997 18:42:23 -0800 (PST) Received: (from danny@localhost) by panda.hilink.com.au (8.8.5/8.7.3) id NAA27747; Tue, 1 Apr 1997 13:01:30 +1000 (EST) Date: Tue, 1 Apr 1997 13:01:29 +1000 (EST) From: "Daniel O'Callaghan" To: "Jordan K. Hubbard" cc: hubs@FreeBSD.org, www@FreeBSD.org Subject: Re: Unified download stats [and better presentation of same] In-Reply-To: <13139.859856167@time.cdrom.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-hubs@FreeBSD.org X-Loop: FreeBSD.org Precedence: bulk On Mon, 31 Mar 1997, Jordan K. Hubbard wrote: > As a very rough and first-order attempt at this, for the last week or > so I've been having ftp.freebsd.org send me little messages like this: > > From: "Root wcarchive.cdrom.com" > Message-Id: <199703311111.DAA07942@wcarchive.cdrom.com> > To: freebsd-stats@freebsd.org > > FreeBSD downloads over last 24 hours (up to Mar 31 03:01) from ftp.FreeBSD.org: > 3.0-970209-SNAP 14 copies. > 2.1.7.1-RELEASE 53 copies. > 2.2.1-RELEASE 139 copies. Looks good. > shows ONE mirror site, however, and this is a drawback which we need > to address before such statistics will become truly useful. We also > need to make sure of the accuracy of our statistics, and this means > that any mirrors which are mirroring things multiple times (I seem to > recall Christoph Kukulies having some problems with this?) need to be > identified and their duplicate entries filtered out of the raw data. > I also need to somehow collect stats for all the mirrors, run once a > day and sent to the ``freebsd-stats@freebsd.org'' alias I've just > 1. Anyone have a good way of detecting mirror looping from > wu-ftpd style xferlogs? > This would help both our bandwidth and our stats. I see you have made 221/usr/src/release/sysinstall/ftp.c announce itself as 'installer' in the password/e-mail address field.) Would it be possible for cooperating hubs to announce themselves consistently in the password/e-mail field too? That would certainly make it easier. Danny From owner-freebsd-hubs Mon Mar 31 22:36:17 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id WAA13835 for hubs-outgoing; Mon, 31 Mar 1997 22:36:17 -0800 (PST) Received: from time.cdrom.com (root@time.cdrom.com [204.216.27.226]) by freefall.freebsd.org (8.8.5/8.8.5) with ESMTP id WAA13821; Mon, 31 Mar 1997 22:36:12 -0800 (PST) Received: from time.cdrom.com (jkh@localhost [127.0.0.1]) by time.cdrom.com (8.8.5/8.6.9) with ESMTP id WAA14072; Mon, 31 Mar 1997 22:35:42 -0800 (PST) To: "Daniel O'Callaghan" cc: hubs@FreeBSD.org, www@FreeBSD.org Subject: Re: Unified download stats [and better presentation of same] In-reply-to: Your message of "Tue, 01 Apr 1997 13:01:29 +1000." Date: Mon, 31 Mar 1997 22:35:42 -0800 Message-ID: <14068.859876542@time.cdrom.com> From: "Jordan K. Hubbard" Sender: owner-hubs@FreeBSD.org X-Loop: FreeBSD.org Precedence: bulk > I see you have made 221/usr/src/release/sysinstall/ftp.c > announce itself as 'installer' in the password/e-mail address field.) > Would it be possible for cooperating hubs to announce themselves > consistently in the password/e-mail field too? That would certainly make > it easier. Yes, that would, though we can probably just as easily ID them by hostname. :-) My primary hope at this point is that we can at least combine logging information and come up with a single consistent format for it. Since I haven't out and out made a proposal on this format yet, here's my shot at it: Each cooperating hub site would scan its logs once every 24 hours and spit out a single email message to freebsd-stats@freebsd.org containing a record like this: FTP mirror name date information ... Where the number of entries in .. is somewhat variable depending on what's on the mirror sites. Jordan From owner-freebsd-hubs Wed Apr 2 14:44:35 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id OAA00651 for hubs-outgoing; Wed, 2 Apr 1997 14:44:35 -0800 (PST) Received: from kalypso.cybercom.net (kalypso.cybercom.net [206.28.134.5]) by freefall.freebsd.org (8.8.5/8.8.5) with ESMTP id OAA00642 for ; Wed, 2 Apr 1997 14:44:32 -0800 (PST) Received: from shell1.cybercom.net (rhuff@shell1.cybercom.net [206.28.134.6]) by kalypso.cybercom.net (8.8.5/8.8.5) with ESMTP id RAA00367 for ; Wed, 2 Apr 1997 17:44:31 -0500 (EST) Received: (from rhuff@localhost) by shell1.cybercom.net (8.8.5/8.8.5) id RAA18390; Wed, 2 Apr 1997 17:44:26 -0500 (EST) Date: Wed, 2 Apr 1997 17:44:26 -0500 (EST) From: Robert Huff Message-Id: <199704022244.RAA18390@shell1.cybercom.net> To: hubs@freebsd.org Subject: bad files at ftp6.freebsd.org Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hello: I think someone needs to check the integrity of the files, particulatly the packages, at ftp6.freebsd.org. I've had a number of instances where packages retrieved from there wouldn't install properly, whereas the same package on ftp.freebsd.org would. Robert Huff From owner-freebsd-hubs Thu Apr 3 14:40:38 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id OAA21229 for hubs-outgoing; Thu, 3 Apr 1997 14:40:38 -0800 (PST) Received: from shrimp.dataplex.net (shrimp.dataplex.net [208.2.87.3]) by freefall.freebsd.org (8.8.5/8.8.5) with ESMTP id OAA21217; Thu, 3 Apr 1997 14:40:33 -0800 (PST) Received: from [204.69.236.50] (GATEWAY.SKIPSTONE.COM [198.214.10.129]) by shrimp.dataplex.net (8.8.5/8.8.5) with SMTP id QAA27032; Thu, 3 Apr 1997 16:40:30 -0600 (CST) Date: 3 Apr 97 16:42:59 -0600 Subject: Re: Where are the src-cur and ports-cur CTM distributions stored? From: "Richard Wackerbarth" To: "Julian H. Stacey" Cc: freebsd-hubs@freebsd.org X-Mailer: Cyberdog/2.0b1 MIME-Version: 1.0 Message-Id: Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk On Thu, Apr 3, 1997 7:38 AM, Julian H. Stacey wrote: >Hi, Reference: >> Subject: Re: Where are the src-cur and ports-cur CTM distributions stored? > >ftp:eel.dataplex.net/pub/FreeBSD/CTM/src-cur If you must refer to my archive, please use the reference ftp://ftp.dataplex.net/pub/FreeBSD/CTM In the long run, we MUST get some archive site for more general usage. I do not have the bandwidth to support much usage of the ftp capability and will have to terminate it if the usage becomes excessive. From owner-freebsd-hubs Thu Apr 3 15:14:37 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id PAA23975 for hubs-outgoing; Thu, 3 Apr 1997 15:14:37 -0800 (PST) Received: from time.cdrom.com (root@time.cdrom.com [204.216.27.226]) by freefall.freebsd.org (8.8.5/8.8.5) with ESMTP id PAA23965; Thu, 3 Apr 1997 15:14:33 -0800 (PST) Received: from time.cdrom.com (jkh@localhost [127.0.0.1]) by time.cdrom.com (8.8.5/8.6.9) with ESMTP id PAA16690; Thu, 3 Apr 1997 15:14:32 -0800 (PST) To: "Richard Wackerbarth" cc: "Julian H. Stacey" , freebsd-hubs@freebsd.org Subject: Re: Where are the src-cur and ports-cur CTM distributions stored? In-reply-to: Your message of "03 Apr 1997 16:42:59 CST." Date: Thu, 03 Apr 1997 15:14:31 -0800 Message-ID: <16686.860109271@time.cdrom.com> From: "Jordan K. Hubbard" Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > If you must refer to my archive, please use the reference > ftp://ftp.dataplex.net/pub/FreeBSD/CTM > > In the long run, we MUST get some archive site for more general usage. > I do not have the bandwidth to support much usage of the ftp capability and > will have to terminate it if the usage becomes excessive. I've free'd up some space on ftp.freebsd.org - where do you want them? Jordan From owner-freebsd-hubs Thu Apr 3 17:18:20 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id RAA03082 for hubs-outgoing; Thu, 3 Apr 1997 17:18:20 -0800 (PST) Received: from smtp.utexas.edu (smtp.utexas.edu [128.83.126.2]) by freefall.freebsd.org (8.8.5/8.8.5) with SMTP id RAA03070 for ; Thu, 3 Apr 1997 17:18:15 -0800 (PST) Received: (qmail-queue invoked from smtpd); 4 Apr 1997 01:18:14 -0000 Received: from mail.utexas.edu (128.83.126.1) by smtp.utexas.edu with SMTP; 4 Apr 1997 01:18:14 -0000 Received: from [208.2.87.4] (cod.dataplex.net [208.2.87.4]) by mail.utexas.edu with ESMTP id TAA10111; Thu, 3 Apr 1997 19:18:08 -0600 (CST) X-Sender: rkw@shrimp.dataplex.net Message-Id: In-Reply-To: <16686.860109271@time.cdrom.com> References: Your message of "03 Apr 1997 16:42:59 CST." Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Thu, 3 Apr 1997 19:16:25 -0600 To: "Jordan K. Hubbard" From: Richard Wackerbarth Subject: Re: Where are the src-cur and ports-cur CTM distributions stored? Cc: "Julian H. Stacey" , freebsd-hubs@freebsd.org Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk At 5:14 PM -0600 4/3/97, Jordan K. Hubbard wrote: >> If you must refer to my archive, please use the reference >> ftp://ftp.dataplex.net/pub/FreeBSD/CTM >> >> In the long run, we MUST get some archive site for more general usage. >> I do not have the bandwidth to support much usage of the ftp capability and >> will have to terminate it if the usage becomes excessive. > >I've free'd up some space on ftp.freebsd.org - where do you want them? Thanks for stepping forward.. I tend to prefer grouping all the ctm archives in one directory rather than spreading them into various distribution trees. If it is desirable to have them associated with the distrimution, a symbolic link can allow both strategies to appear to exist. I would suggest that you parallel the structure that I am using. IOW, /pub/FreeBSD/CTM/src-2.2 , /pub/FreeBSD/CTM/src-cur , etc. To initialize things, you may simply copy my CTM tree. For the long term, I suggest that you subscribe to each of the lists so that the distibutions are archived automatically as src-2.2 is presently handled. From owner-freebsd-hubs Thu Apr 3 18:07:26 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id SAA06507 for hubs-outgoing; Thu, 3 Apr 1997 18:07:26 -0800 (PST) Received: from time.cdrom.com (root@time.cdrom.com [204.216.27.226]) by freefall.freebsd.org (8.8.5/8.8.5) with ESMTP id SAA06501 for ; Thu, 3 Apr 1997 18:07:22 -0800 (PST) Received: from time.cdrom.com (jkh@localhost [127.0.0.1]) by time.cdrom.com (8.8.5/8.6.9) with ESMTP id SAA17715 for ; Thu, 3 Apr 1997 18:07:27 -0800 (PST) To: hubs@freebsd.org Subject: Things moving around on ftp://ftp.freebsd.org/pub/FreeBSD Date: Thu, 03 Apr 1997 18:07:27 -0800 Message-ID: <17711.860119647@time.cdrom.com> From: "Jordan K. Hubbard" Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk We've overflowed our 1st 4Gb partition there and now need to go to a 2nd, but that means splitting off some of the bits which were traditionally confined to one filesystem, and I'm still not sure what the effects on the mirrors is going to be. If you mirror individual collections, like pub/FreeBSD/2.1.7.1-RELEASE/, then you should be OK since mirror should follow the symlink. If you're grabbing all of pub/FreeBSD, however, what you're going to see afterwards is a link: 2.1.7.1-RELEASE -> ../../.3/FreeBSD/2.1.7.1-RELEASE I'm not quite sure what the "best" work-around for this is, but I just thought I'd mention it. Sorry folks, but it had to happen sooner or later - the ports and packages are huge! ;-) This doesn't effect users who get their bits from ftp.freebsd.org directly, of course, only those who mirror it. Jordan From owner-freebsd-hubs Thu Apr 3 18:27:41 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id SAA07960 for hubs-outgoing; Thu, 3 Apr 1997 18:27:41 -0800 (PST) Received: from thelab.hub.org (hal-ns1-32.netcom.ca [207.181.94.96]) by freefall.freebsd.org (8.8.5/8.8.5) with ESMTP id SAA07951 for ; Thu, 3 Apr 1997 18:27:34 -0800 (PST) Received: from thelab.hub.org (localhost [127.0.0.1]) by thelab.hub.org (8.8.5/8.8.2) with SMTP id WAA24069; Thu, 3 Apr 1997 22:27:20 -0400 (AST) Date: Thu, 3 Apr 1997 22:27:20 -0400 (AST) From: "Marc G. Fournier" To: "Jordan K. Hubbard" cc: hubs@freebsd.org Subject: Re: Things moving around on ftp://ftp.freebsd.org/pub/FreeBSD In-Reply-To: <17711.860119647@time.cdrom.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk On Thu, 3 Apr 1997, Jordan K. Hubbard wrote: > We've overflowed our 1st 4Gb partition there and now need to go to a > 2nd, but that means splitting off some of the bits which were > traditionally confined to one filesystem, and I'm still not sure what > the effects on the mirrors is going to be. If you mirror individual > collections, like pub/FreeBSD/2.1.7.1-RELEASE/, then you should be OK > since mirror should follow the symlink. If you're grabbing all > of pub/FreeBSD, however, what you're going to see afterwards is a link: > 2.1.7.1-RELEASE -> ../../.3/FreeBSD/2.1.7.1-RELEASE > > I'm not quite sure what the "best" work-around for this is, but > I just thought I'd mention it. Sorry folks, but it had to happen > sooner or later - the ports and packages are huge! ;-) > How about providing a listing of symlinks that can be setup in the mirror files individually? My familiarity with mirror isn't that great, but I believe that the first thing that would happen would be when you do the initial mirror, it would try to rm the subdirectories that are the result of the symlinks, but I also believe there are various 'exclude' directives that could be used for that... The only thing that the mirrors would require would be a listing of those subdirectories that are symlink'd so that appropriate mirror 'branches' could be created... Just a thought... Marc G. Fournier Systems Administrator @ hub.org primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org From owner-freebsd-hubs Thu Apr 3 19:24:05 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id TAA11344 for hubs-outgoing; Thu, 3 Apr 1997 19:24:05 -0800 (PST) Received: from time.cdrom.com (root@time.cdrom.com [204.216.27.226]) by freefall.freebsd.org (8.8.5/8.8.5) with ESMTP id TAA11339 for ; Thu, 3 Apr 1997 19:24:02 -0800 (PST) Received: from time.cdrom.com (jkh@localhost [127.0.0.1]) by time.cdrom.com (8.8.5/8.6.9) with ESMTP id TAA18048; Thu, 3 Apr 1997 19:24:03 -0800 (PST) To: "Marc G. Fournier" cc: hubs@freebsd.org Subject: Re: Things moving around on ftp://ftp.freebsd.org/pub/FreeBSD In-reply-to: Your message of "Thu, 03 Apr 1997 22:27:20 -0400." Date: Thu, 03 Apr 1997 19:24:03 -0800 Message-ID: <18045.860124243@time.cdrom.com> From: "Jordan K. Hubbard" Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > The only thing that the mirrors would require would be a listing > of those subdirectories that are symlink'd so that appropriate mirror > 'branches' could be created... Well, they can get that for themselves anytime - it's not like the information's a secret or anything. ;-) Jordan From owner-freebsd-hubs Thu Apr 3 19:31:26 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id TAA11624 for hubs-outgoing; Thu, 3 Apr 1997 19:31:26 -0800 (PST) Received: from thelab.hub.org (hal-ns1-32.netcom.ca [207.181.94.96]) by freefall.freebsd.org (8.8.5/8.8.5) with ESMTP id TAA11616 for ; Thu, 3 Apr 1997 19:31:20 -0800 (PST) Received: from thelab.hub.org (localhost [127.0.0.1]) by thelab.hub.org (8.8.5/8.8.2) with SMTP id XAA24625; Thu, 3 Apr 1997 23:30:51 -0400 (AST) Date: Thu, 3 Apr 1997 23:30:51 -0400 (AST) From: "Marc G. Fournier" To: "Jordan K. Hubbard" cc: hubs@freebsd.org Subject: Re: Things moving around on ftp://ftp.freebsd.org/pub/FreeBSD In-Reply-To: <18045.860124243@time.cdrom.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk On Thu, 3 Apr 1997, Jordan K. Hubbard wrote: > > The only thing that the mirrors would require would be a listing > > of those subdirectories that are symlink'd so that appropriate mirror > > 'branches' could be created... > > Well, they can get that for themselves anytime - it's not like > the information's a secret or anything. ;-) No, I realize that...but the idea should be to make the transition as painful as possible for everyone, no? It would be easier for you to create a list and send it out once to the list then to have everyone ftp'ng into the site to check out what the configuration looks like, no? :) It could also act as a quick confirmation that the "dreadful deed" is done? :) Marc G. Fournier Systems Administrator @ hub.org primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org From owner-freebsd-hubs Thu Apr 3 19:32:29 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id TAA11676 for hubs-outgoing; Thu, 3 Apr 1997 19:32:29 -0800 (PST) Received: from time.cdrom.com (root@time.cdrom.com [204.216.27.226]) by freefall.freebsd.org (8.8.5/8.8.5) with ESMTP id TAA11671 for ; Thu, 3 Apr 1997 19:32:27 -0800 (PST) Received: from time.cdrom.com (jkh@localhost [127.0.0.1]) by time.cdrom.com (8.8.5/8.6.9) with ESMTP id TAA18120; Thu, 3 Apr 1997 19:32:29 -0800 (PST) To: "Marc G. Fournier" cc: hubs@freebsd.org Subject: Re: Things moving around on ftp://ftp.freebsd.org/pub/FreeBSD In-reply-to: Your message of "Thu, 03 Apr 1997 23:30:51 -0400." Date: Thu, 03 Apr 1997 19:32:28 -0800 Message-ID: <18116.860124748@time.cdrom.com> From: "Jordan K. Hubbard" Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > No, I realize that...but the idea should be to make the transition > as painful as possible for everyone, no? It would be easier for you to Absolutely. :-) Jordan From owner-freebsd-hubs Thu Apr 3 19:39:59 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id TAA12067 for hubs-outgoing; Thu, 3 Apr 1997 19:39:59 -0800 (PST) Received: from root.com (implode.root.com [198.145.90.17]) by freefall.freebsd.org (8.8.5/8.8.5) with ESMTP id TAA12062 for ; Thu, 3 Apr 1997 19:39:53 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by root.com (8.8.5/8.6.5) with SMTP id TAA08543; Thu, 3 Apr 1997 19:40:42 -0800 (PST) Message-Id: <199704040340.TAA08543@root.com> X-Authentication-Warning: implode.root.com: localhost [127.0.0.1] didn't use HELO protocol To: "Marc G. Fournier" cc: "Jordan K. Hubbard" , hubs@freebsd.org Subject: Re: Things moving around on ftp://ftp.freebsd.org/pub/FreeBSD In-reply-to: Your message of "Thu, 03 Apr 1997 23:30:51 -0400." From: David Greenman Reply-To: dg@root.com Date: Thu, 03 Apr 1997 19:40:42 -0800 Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk >On Thu, 3 Apr 1997, Jordan K. Hubbard wrote: > >> > The only thing that the mirrors would require would be a listing >> > of those subdirectories that are symlink'd so that appropriate mirror >> > 'branches' could be created... >> >> Well, they can get that for themselves anytime - it's not like >> the information's a secret or anything. ;-) > > No, I realize that...but the idea should be to make the transition >as painful as possible for everyone, no? It would be easier for you to >create a list and send it out once to the list then to have everyone >ftp'ng into the site to check out what the configuration looks like, no? :) > > It could also act as a quick confirmation that the "dreadful deed" >is done? :) The main problem with the notification thing is that we move archives around frequently on wcarchive (due to space shortages) and we have too many archives to even think about trying to notify people about this every time it occurs. There really needs to be a more automatic mechanism for dealing with this. I know that mirror can be told to follow all symlinks, but this is bad when you use symlinks within a filesystem (ports -> current/ports) because it will cause the mirror site to needlessly consume massive amounts of disk space. Mirror should be smarter about this. It should be able to tell when a symlink point outside of the current hierarchy and then follow the symlink when it sees that. Hmmm...maybe it already has support for that? -DG David Greenman Core-team/Principal Architect, The FreeBSD Project From owner-freebsd-hubs Thu Apr 3 19:44:24 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id TAA12285 for hubs-outgoing; Thu, 3 Apr 1997 19:44:24 -0800 (PST) Received: from thelab.hub.org (hal-ns1-32.netcom.ca [207.181.94.96]) by freefall.freebsd.org (8.8.5/8.8.5) with ESMTP id TAA12274 for ; Thu, 3 Apr 1997 19:44:17 -0800 (PST) Received: from thelab.hub.org (localhost [127.0.0.1]) by thelab.hub.org (8.8.5/8.8.2) with SMTP id XAA24691; Thu, 3 Apr 1997 23:44:08 -0400 (AST) Date: Thu, 3 Apr 1997 23:44:08 -0400 (AST) From: "Marc G. Fournier" To: "Jordan K. Hubbard" cc: hubs@freebsd.org Subject: Re: Things moving around on ftp://ftp.freebsd.org/pub/FreeBSD In-Reply-To: <18116.860124748@time.cdrom.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk On Thu, 3 Apr 1997, Jordan K. Hubbard wrote: > > No, I realize that...but the idea should be to make the transition > > as painful as possible for everyone, no? It would be easier for you to > > Absolutely. :-) > Oh good...I was getting nervous there that I should have kept my mouth shut on this *grin* Marc G. Fournier Systems Administrator @ hub.org primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org From owner-freebsd-hubs Thu Apr 3 19:47:55 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id TAA12371 for hubs-outgoing; Thu, 3 Apr 1997 19:47:55 -0800 (PST) Received: from thelab.hub.org (hal-ns1-32.netcom.ca [207.181.94.96]) by freefall.freebsd.org (8.8.5/8.8.5) with ESMTP id TAA12364 for ; Thu, 3 Apr 1997 19:47:51 -0800 (PST) Received: from thelab.hub.org (localhost [127.0.0.1]) by thelab.hub.org (8.8.5/8.8.2) with SMTP id XAA24717; Thu, 3 Apr 1997 23:46:57 -0400 (AST) Date: Thu, 3 Apr 1997 23:46:57 -0400 (AST) From: "Marc G. Fournier" To: David Greenman cc: "Jordan K. Hubbard" , hubs@freebsd.org Subject: Re: Things moving around on ftp://ftp.freebsd.org/pub/FreeBSD In-Reply-To: <199704040340.TAA08543@root.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk On Thu, 3 Apr 1997, David Greenman wrote: > this. I know that mirror can be told to follow all symlinks, but this is > bad when you use symlinks within a filesystem (ports -> current/ports) because > it will cause the mirror site to needlessly consume massive amounts of disk > space. You lost me on this one...how does that consume massive amounts of disk space? You aren't *using* any more space, only moving the same amount of space to a different area...no? Marc G. Fournier Systems Administrator @ hub.org primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org From owner-freebsd-hubs Thu Apr 3 19:56:24 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id TAA12596 for hubs-outgoing; Thu, 3 Apr 1997 19:56:24 -0800 (PST) Received: from root.com (implode.root.com [198.145.90.17]) by freefall.freebsd.org (8.8.5/8.8.5) with ESMTP id TAA12591 for ; Thu, 3 Apr 1997 19:56:21 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by root.com (8.8.5/8.6.5) with SMTP id TAA08681; Thu, 3 Apr 1997 19:57:15 -0800 (PST) Message-Id: <199704040357.TAA08681@root.com> X-Authentication-Warning: implode.root.com: localhost [127.0.0.1] didn't use HELO protocol To: "Marc G. Fournier" cc: "Jordan K. Hubbard" , hubs@freebsd.org Subject: Re: Things moving around on ftp://ftp.freebsd.org/pub/FreeBSD In-reply-to: Your message of "Thu, 03 Apr 1997 23:46:57 -0400." From: David Greenman Reply-To: dg@root.com Date: Thu, 03 Apr 1997 19:57:15 -0800 Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk >On Thu, 3 Apr 1997, David Greenman wrote: > >> this. I know that mirror can be told to follow all symlinks, but this is >> bad when you use symlinks within a filesystem (ports -> current/ports) because >> it will cause the mirror site to needlessly consume massive amounts of disk >> space. > > You lost me on this one...how does that consume massive amounts of >disk space? You aren't *using* any more space, only moving the same amount of >space to a different area...no? Mirror will follow the symlink and individually create the directory hierarchies that it points to. Thus "ports -> current/ports" and "new_ports -> current/ports" will create *three* copies of "ports" and no symlinks will be created. ...but you have to tell it to do this (I forget the option). Normally it just creates symlinks that are within the current tree and ignores those that point outside of the current tree. -DG David Greenman Core-team/Principal Architect, The FreeBSD Project From owner-freebsd-hubs Thu Apr 3 20:05:23 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id UAA12990 for hubs-outgoing; Thu, 3 Apr 1997 20:05:23 -0800 (PST) Received: from thelab.hub.org (hal-ns1-32.netcom.ca [207.181.94.96]) by freefall.freebsd.org (8.8.5/8.8.5) with ESMTP id UAA12984 for ; Thu, 3 Apr 1997 20:05:17 -0800 (PST) Received: from thelab.hub.org (localhost [127.0.0.1]) by thelab.hub.org (8.8.5/8.8.2) with SMTP id AAA24817; Fri, 4 Apr 1997 00:04:56 -0400 (AST) Date: Fri, 4 Apr 1997 00:04:56 -0400 (AST) From: "Marc G. Fournier" To: David Greenman cc: "Jordan K. Hubbard" , hubs@freebsd.org Subject: Re: Things moving around on ftp://ftp.freebsd.org/pub/FreeBSD In-Reply-To: <199704040357.TAA08681@root.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk On Thu, 3 Apr 1997, David Greenman wrote: > >On Thu, 3 Apr 1997, David Greenman wrote: > > > >> this. I know that mirror can be told to follow all symlinks, but this is > >> bad when you use symlinks within a filesystem (ports -> current/ports) because > >> it will cause the mirror site to needlessly consume massive amounts of disk > >> space. > > > > You lost me on this one...how does that consume massive amounts of > >disk space? You aren't *using* any more space, only moving the same amount of > >space to a different area...no? > > Mirror will follow the symlink and individually create the directory > hierarchies that it points to. Thus "ports -> current/ports" and "new_ports > -> current/ports" will create *three* copies of "ports" and no symlinks > will be created. ...but you have to tell it to do this (I forget the option). Okay, now I understand what you are referring too...thanks for the explanation :) Marc G. Fournier Systems Administrator @ hub.org primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org From owner-freebsd-hubs Fri Apr 4 01:26:32 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id BAA28504 for hubs-outgoing; Fri, 4 Apr 1997 01:26:32 -0800 (PST) Received: from icparc-pc-zl.doc.ic.ac.uk (icparc-pc-zl.doc.ic.ac.uk [146.169.24.24]) by freefall.freebsd.org (8.8.5/8.8.5) with SMTP id BAA28499 for ; Fri, 4 Apr 1997 01:26:28 -0800 (PST) Received: from triumph.doc.ic.ac.uk [146.169.24.1] by icparc-pc-zl.doc.ic.ac.uk with smtp (Exim 1.61 #1) id 0wD4Hw-0000c8-00; Fri, 4 Apr 1997 09:24:00 +0100 Message-ID: <3344C8E4.C6DC9D8@doc.ic.ac.uk> Date: Fri, 04 Apr 1997 10:24:52 +0100 From: Lee McLoughlin Organization: IC-Parc, Imperial College, University of London. U.K. X-Mailer: Mozilla 3.0 (X11; I; SunOS 4.1.3_U1 sun4m) MIME-Version: 1.0 To: "Jordan K. Hubbard" CC: hubs@freebsd.org Subject: Re: Things moving around on ftp://ftp.freebsd.org/pub/FreeBSD References: <17711.860119647@time.cdrom.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Jordan K. Hubbard wrote: > > We've overflowed our 1st 4Gb partition there and now need to go to a > 2nd, but that means splitting off some of the bits which were > traditionally confined to one filesystem, and I'm still not sure what > the effects on the mirrors is going to be. If you mirror individual > collections, like pub/FreeBSD/2.1.7.1-RELEASE/, then you should be OK > since mirror should follow the symlink. If you're grabbing all > of pub/FreeBSD, however, what you're going to see afterwards is a link: > 2.1.7.1-RELEASE -> ../../.3/FreeBSD/2.1.7.1-RELEASE > > I'm not quite sure what the "best" work-around for this is, but > I just thought I'd mention it. Sorry folks, but it had to happen > sooner or later - the ports and packages are huge! ;-) > > This doesn't effect users who get their bits from ftp.freebsd.org > directly, of course, only those who mirror it. > > Jordan Could you, perhaps, mount the new disk inside the FreeBSD area? That way any full mirror would still get it all? -- Lee McLoughlin. Phone: +44 171 594 8388 IC-Parc, Imperial College, Fax: +44 171 594 8432 South Kensington, London. SW7 2BZ. UK. Email: L.McLoughlin@doc.ic.ac.uk From owner-freebsd-hubs Fri Apr 4 01:38:10 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id BAA29073 for hubs-outgoing; Fri, 4 Apr 1997 01:38:10 -0800 (PST) Received: from time.cdrom.com (root@time.cdrom.com [204.216.27.226]) by freefall.freebsd.org (8.8.5/8.8.5) with ESMTP id BAA29066 for ; Fri, 4 Apr 1997 01:38:06 -0800 (PST) Received: from time.cdrom.com (jkh@localhost [127.0.0.1]) by time.cdrom.com (8.8.5/8.6.9) with ESMTP id BAA20555; Fri, 4 Apr 1997 01:38:01 -0800 (PST) To: Lee McLoughlin cc: hubs@freebsd.org Subject: Re: Things moving around on ftp://ftp.freebsd.org/pub/FreeBSD In-reply-to: Your message of "Fri, 04 Apr 1997 10:24:52 +0100." <3344C8E4.C6DC9D8@doc.ic.ac.uk> Date: Fri, 04 Apr 1997 01:38:01 -0800 Message-ID: <20551.860146681@time.cdrom.com> From: "Jordan K. Hubbard" Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > Could you, perhaps, mount the new disk inside the FreeBSD area? That > way any full mirror would still get it all? Wurgh, mount spaghetti! :-( I don't think that David would go for it, to be perfectly honest. Isn't it easier just to split the mirror? Jordan From owner-freebsd-hubs Fri Apr 4 01:50:03 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id BAA29519 for hubs-outgoing; Fri, 4 Apr 1997 01:50:03 -0800 (PST) Received: from root.com (implode.root.com [198.145.90.17]) by freefall.freebsd.org (8.8.5/8.8.5) with ESMTP id BAA29488 for ; Fri, 4 Apr 1997 01:50:00 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by root.com (8.8.5/8.6.5) with SMTP id BAA10745; Fri, 4 Apr 1997 01:50:54 -0800 (PST) Message-Id: <199704040950.BAA10745@root.com> X-Authentication-Warning: implode.root.com: localhost [127.0.0.1] didn't use HELO protocol To: "Jordan K. Hubbard" cc: Lee McLoughlin , hubs@freebsd.org Subject: Re: Things moving around on ftp://ftp.freebsd.org/pub/FreeBSD In-reply-to: Your message of "Fri, 04 Apr 1997 01:38:01 PST." <20551.860146681@time.cdrom.com> From: David Greenman Reply-To: dg@root.com Date: Fri, 04 Apr 1997 01:50:54 -0800 Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk >> Could you, perhaps, mount the new disk inside the FreeBSD area? That >> way any full mirror would still get it all? > >Wurgh, mount spaghetti! :-( > >I don't think that David would go for it, to be perfectly honest. >Isn't it easier just to split the mirror? Right - David wouldn't go for it. -DG David Greenman Core-team/Principal Architect, The FreeBSD Project From owner-freebsd-hubs Fri Apr 4 02:44:04 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id CAA01202 for hubs-outgoing; Fri, 4 Apr 1997 02:44:04 -0800 (PST) Received: from icparc-pc-zl.doc.ic.ac.uk (icparc-pc-zl.doc.ic.ac.uk [146.169.24.24]) by freefall.freebsd.org (8.8.5/8.8.5) with SMTP id CAA01146 for ; Fri, 4 Apr 1997 02:43:19 -0800 (PST) Received: from triumph.doc.ic.ac.uk [146.169.24.1] by icparc-pc-zl.doc.ic.ac.uk with smtp (Exim 1.61 #1) id 0wD6QY-0000eb-00; Fri, 4 Apr 1997 11:41:02 +0100 Message-ID: <3344DACF.26FB5C33@doc.ic.ac.uk> Date: Fri, 04 Apr 1997 11:41:19 +0100 From: Lee McLoughlin Organization: IC-Parc, Imperial College, University of London. U.K. X-Mailer: Mozilla 3.0 (X11; I; SunOS 4.1.3_U1 sun4m) MIME-Version: 1.0 To: dg@root.com CC: "Jordan K. Hubbard" , hubs@freebsd.org Subject: Re: Things moving around on ftp://ftp.freebsd.org/pub/FreeBSD References: <199704040950.BAA10745@root.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk David Greenman wrote: > > >> Could you, perhaps, mount the new disk inside the FreeBSD area? That > >> way any full mirror would still get it all? > > > >Wurgh, mount spaghetti! :-( > > > >I don't think that David would go for it, to be perfectly honest. > >Isn't it easier just to split the mirror? > > Right - David wouldn't go for it. Bah! OK plan2: Meta-disk the area, personally I really like Veritas on my Solaris boxes, to make one *BIG* disk. Here on SunSITE.doc.ic.ac.uk we use Veritas to create a single 180 Gigabyte disk out of 40, or so, disks. This isn't a frivolous suggestion. I've been running SunSITE for ages and I grew to *really* hate shuffling areas between disks to balance the space. If you decide against it I have some patches for wu-ftpd that gets pwd to report where you chdir'd to not where you physically are. So if FreeBSD/fred/joe is a symlink to ../../../.01/joe then cd FreeBSD/fred/joe pwd -> FreeBSD/fred/joe This doesn't help with mirroring at all but can help avoid confusing users. Lee -- Lee McLoughlin. Phone: +44 171 594 8388 IC-Parc, Imperial College, Fax: +44 171 594 8432 South Kensington, London. SW7 2BZ. UK. Email: L.McLoughlin@doc.ic.ac.uk From owner-freebsd-hubs Fri Apr 4 02:50:39 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id CAA01396 for hubs-outgoing; Fri, 4 Apr 1997 02:50:39 -0800 (PST) Received: from root.com (implode.root.com [198.145.90.17]) by freefall.freebsd.org (8.8.5/8.8.5) with ESMTP id CAA01391 for ; Fri, 4 Apr 1997 02:50:35 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by root.com (8.8.5/8.6.5) with SMTP id CAA11084; Fri, 4 Apr 1997 02:50:49 -0800 (PST) Message-Id: <199704041050.CAA11084@root.com> X-Authentication-Warning: implode.root.com: localhost [127.0.0.1] didn't use HELO protocol To: Lee McLoughlin cc: "Jordan K. Hubbard" , hubs@freebsd.org Subject: Re: Things moving around on ftp://ftp.freebsd.org/pub/FreeBSD In-reply-to: Your message of "Fri, 04 Apr 1997 11:41:19 +0100." <3344DACF.26FB5C33@doc.ic.ac.uk> From: David Greenman Reply-To: dg@root.com Date: Fri, 04 Apr 1997 02:50:49 -0800 Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk >> Right - David wouldn't go for it. > >Bah! > >OK plan2: Meta-disk the area, personally I really like Veritas on my >Solaris boxes, to make one *BIG* disk. Here on SunSITE.doc.ic.ac.uk we >use Veritas to create a single 180 Gigabyte disk out of 40, or so, >disks. > >This isn't a frivolous suggestion. I've been running SunSITE for ages >and I grew to *really* hate shuffling areas between disks to balance the >space. We don't have Veritas available and I'm not willing to use CCD due to too high a drive failure rate. >If you decide against it I have some patches for wu-ftpd that gets pwd >to report where you chdir'd to not where you physically are. So if >FreeBSD/fred/joe is a symlink to ../../../.01/joe then > cd FreeBSD/fred/joe > pwd -> FreeBSD/fred/joe > >This doesn't help with mirroring at all but can help avoid confusing >users. Sounds pretty evil and in any case, we're not using wu-ftpd. Look, if people can just deal with this for the short term, we'll eventually move FreeBSD to a 9GB drive and this will be a non-issue again. -DG David Greenman Core-team/Principal Architect, The FreeBSD Project From owner-freebsd-hubs Fri Apr 4 03:08:10 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id DAA01783 for hubs-outgoing; Fri, 4 Apr 1997 03:08:10 -0800 (PST) Received: from kremvax.demos.su (kremvax.demos.su [194.87.0.20]) by freefall.freebsd.org (8.8.5/8.8.5) with SMTP id DAA01774 for ; Fri, 4 Apr 1997 03:08:03 -0800 (PST) Received: by kremvax.demos.su (8.6.13/D) from 0@sinbin.demos.su [194.87.0.31] with ESMTP id OAA12124; Fri, 4 Apr 1997 14:03:01 +0300 Received: by sinbin.demos.su id PAA02673; (8.6.12/D) Fri, 4 Apr 1997 15:01:53 +0400 Message-Id: <199704041101.PAA02673@sinbin.demos.su> Subject: Re: Things moving around on ftp://ftp.freebsd.org/pub/FreeBSD To: lmjm@doc.ic.ac.uk (Lee McLoughlin) Date: Fri, 4 Apr 1997 15:01:53 +0400 (MSD) Cc: dg@root.com, jkh@time.cdrom.com, hubs@freebsd.org In-Reply-To: <3344DACF.26FB5C33@doc.ic.ac.uk> from "Lee McLoughlin" at Apr 4, 97 11:41:19 am From: "Mikhail A. Sokolov" X-Class: Fast Organization: Demos Company, Ltd. Reply-To: mishania@demos.su X-Mailer: ELM [version 2.4 PL24 ME7a] Content-Type: text Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > OK plan2: Meta-disk the area, personally I really like Veritas on my There's no Veritas available for FreeBSD, isn't there? As for ccd.. well, I already saw exact situation when one of two ccd'ed disk crashes, what happens to the whole information placed on both. > FreeBSD/fred/joe is a symlink to ../../../.01/joe then > cd FreeBSD/fred/joe > pwd -> FreeBSD/fred/joe > Lee McLoughlin. Phone: +44 171 594 8388 Btw, guys, what's with idea to split distfiles or at last clean out those which are 'old' ? And, maybe it would be better idea to move packages+distfiles onto that other new disk you have there, rather than moving -RELEASE? -mishania From owner-freebsd-hubs Fri Apr 4 03:15:36 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id DAA02077 for hubs-outgoing; Fri, 4 Apr 1997 03:15:36 -0800 (PST) Received: from icparc-pc-zl.doc.ic.ac.uk (icparc-pc-zl.doc.ic.ac.uk [146.169.24.24]) by freefall.freebsd.org (8.8.5/8.8.5) with SMTP id DAA02072 for ; Fri, 4 Apr 1997 03:15:31 -0800 (PST) Received: from triumph.doc.ic.ac.uk [146.169.24.1] by icparc-pc-zl.doc.ic.ac.uk with smtp (Exim 1.61 #1) id 0wD6xG-0000f8-00; Fri, 4 Apr 1997 12:14:51 +0100 Message-ID: <3344E2BC.5DC51F25@doc.ic.ac.uk> Date: Fri, 04 Apr 1997 12:15:08 +0100 From: Lee McLoughlin Organization: IC-Parc, Imperial College, University of London. U.K. X-Mailer: Mozilla 3.0 (X11; I; SunOS 4.1.3_U1 sun4m) MIME-Version: 1.0 To: dg@root.com CC: "Jordan K. Hubbard" , hubs@freebsd.org Subject: Re: Things moving around on ftp://ftp.freebsd.org/pub/FreeBSD References: <199704041050.CAA11084@root.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk David Greenman wrote: > > We don't have Veritas available and I'm not willing to use CCD due to > too high a drive failure rate. CCD? > >If you decide against it I have some patches for wu-ftpd that gets pwd > >to report where you chdir'd to not where you physically are. So if > >FreeBSD/fred/joe is a symlink to ../../../.01/joe then > > cd FreeBSD/fred/joe > > pwd -> FreeBSD/fred/joe > > > >This doesn't help with mirroring at all but can help avoid confusing > >users. > > Sounds pretty evil and in any case, we're not using wu-ftpd. I don't think its evil, several shells now do it (eg: bash). > Look, if > people can just deal with this for the short term, we'll eventually move > FreeBSD to a 9GB drive and this will be a non-issue again. Sorry if you felt I was trying to get at you, or something. Its just from past experience I've found these things help. -- Lee McLoughlin. Phone: +44 171 594 8388 IC-Parc, Imperial College, Fax: +44 171 594 8432 South Kensington, London. SW7 2BZ. UK. Email: L.McLoughlin@doc.ic.ac.uk From owner-freebsd-hubs Fri Apr 4 03:17:35 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id DAA02141 for hubs-outgoing; Fri, 4 Apr 1997 03:17:35 -0800 (PST) Received: from time.cdrom.com (root@time.cdrom.com [204.216.27.226]) by freefall.freebsd.org (8.8.5/8.8.5) with ESMTP id DAA02136 for ; Fri, 4 Apr 1997 03:17:32 -0800 (PST) Received: from time.cdrom.com (jkh@localhost [127.0.0.1]) by time.cdrom.com (8.8.5/8.6.9) with ESMTP id DAA00297; Fri, 4 Apr 1997 03:17:00 -0800 (PST) To: mishania@demos.su cc: lmjm@doc.ic.ac.uk (Lee McLoughlin), dg@root.com, hubs@freebsd.org Subject: Re: Things moving around on ftp://ftp.freebsd.org/pub/FreeBSD In-reply-to: Your message of "Fri, 04 Apr 1997 15:01:53 +0400." <199704041101.PAA02673@sinbin.demos.su> Date: Fri, 04 Apr 1997 03:17:00 -0800 Message-ID: <293.860152620@time.cdrom.com> From: "Jordan K. Hubbard" Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > Btw, guys, what's with idea to split distfiles or at last clean out those whi ch > are 'old' ? And, maybe it would be better idea to move packages+distfiles ont o > that other new disk you have there, rather than moving -RELEASE? I'm flexible. What would cause the least chaos, folks? We have 4GB of stuff and 8GB of space - the various slices of pie are different sizes. How do you divvy it up most effectively? This sounds like one of those math problems we get in U.S. elementary school. ;-) Jordan From owner-freebsd-hubs Fri Apr 4 03:40:16 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id DAA02951 for hubs-outgoing; Fri, 4 Apr 1997 03:40:16 -0800 (PST) Received: from kremvax.demos.su (kremvax.demos.su [194.87.0.20]) by freefall.freebsd.org (8.8.5/8.8.5) with SMTP id DAA02946 for ; Fri, 4 Apr 1997 03:40:13 -0800 (PST) Received: by kremvax.demos.su (8.6.13/D) from 0@sinbin.demos.su [194.87.0.31] with ESMTP id OAA21760; Fri, 4 Apr 1997 14:39:00 +0300 Received: by sinbin.demos.su id PAA11708; (8.6.12/D) Fri, 4 Apr 1997 15:38:11 +0400 Message-Id: <199704041138.PAA11708@sinbin.demos.su> Subject: Re: Things moving around on ftp://ftp.freebsd.org/pub/FreeBSD To: jkh@time.cdrom.com (Jordan K. Hubbard) Date: Fri, 4 Apr 1997 15:38:10 +0400 (MSD) Cc: mishania@demos.su, lmjm@doc.ic.ac.uk, dg@root.com, hubs@freebsd.org In-Reply-To: <293.860152620@time.cdrom.com> from "Jordan K. Hubbard" at Apr 4, 97 03:17:00 am From: "Mikhail A. Sokolov" X-Class: Fast Organization: Demos Company, Ltd. Reply-To: mishania@demos.su X-Mailer: ELM [version 2.4 PL24 ME7a] Content-Type: text Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk >>Btw, guys, what's with idea to split distfiles or at last clean out those whi >>are 'old' ? And, maybe it would be better idea to move packages+distfiles ont > > that other new disk you have there, rather than moving -RELEASE? > > I'm flexible. What would cause the least chaos, folks? We have 4GB > of stuff and 8GB of space - the various slices of pie are different > sizes. How do you divvy it up most effectively? Well, regarding sizes, if I were you I'd make this 2nd 4gb to have packages/ distfiles and releases to stay on another disk. In this case you just have symbolic links to packages stuff. Almost all proggies, used to mirror, fetch & ncftp's can easilly deal with such placement. Uhm, that's what I have now here anyhow, - we're used to use 4gb disks; I never heard users complaining on that topic. > This sounds like one of those math problems we get in U.S. elementary > school. ;-) Heh, as for me I never went to ^^^^^^^^^^^^^^^^^ school ;-) > Jordan -mishania From owner-freebsd-hubs Fri Apr 4 04:18:03 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id EAA04183 for hubs-outgoing; Fri, 4 Apr 1997 04:18:03 -0800 (PST) Received: from root.com (implode.root.com [198.145.90.17]) by freefall.freebsd.org (8.8.5/8.8.5) with ESMTP id EAA04178 for ; Fri, 4 Apr 1997 04:18:01 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by root.com (8.8.5/8.6.5) with SMTP id EAA11522; Fri, 4 Apr 1997 04:18:55 -0800 (PST) Message-Id: <199704041218.EAA11522@root.com> X-Authentication-Warning: implode.root.com: localhost [127.0.0.1] didn't use HELO protocol To: Lee McLoughlin cc: "Jordan K. Hubbard" , hubs@freebsd.org Subject: Re: Things moving around on ftp://ftp.freebsd.org/pub/FreeBSD In-reply-to: Your message of "Fri, 04 Apr 1997 12:15:08 +0100." <3344E2BC.5DC51F25@doc.ic.ac.uk> From: David Greenman Reply-To: dg@root.com Date: Fri, 04 Apr 1997 04:18:55 -0800 Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk >> We don't have Veritas available and I'm not willing to use CCD due to >> too high a drive failure rate. > >CCD? "Concantenated Disk" driver. It's a pseudo disk device that lets one create simple volume sets from individual disk drives. It is usually used to create large pseudo disks from an array of smaller ones. It has no 'parity' bit or other volume fault tolerance mechanisms, however, so if one drive fails you lose the entire volume. >> Look, if >> people can just deal with this for the short term, we'll eventually move >> FreeBSD to a 9GB drive and this will be a non-issue again. > >Sorry if you felt I was trying to get at you, or something. Its just >from >past experience I've found these things help. sunsite.doc.ic.ac.uk does indeed look impressive. It's obvious that someone has spent a lot of money on it. :-) ...we have to work with what we have, however, and I think our current operations and policies are generally working well. -DG David Greenman Core-team/Principal Architect, The FreeBSD Project From owner-freebsd-hubs Fri Apr 4 05:31:23 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id FAA07661 for hubs-outgoing; Fri, 4 Apr 1997 05:31:23 -0800 (PST) Received: from Campino.Informatik.RWTH-Aachen.DE (campino.Informatik.RWTH-Aachen.DE [137.226.116.240]) by freefall.freebsd.org (8.8.5/8.8.5) with ESMTP id FAA07654 for ; Fri, 4 Apr 1997 05:31:20 -0800 (PST) Received: from gilberto.physik.rwth-aachen.de (gilberto.physik.rwth-aachen.de [137.226.31.2]) by Campino.Informatik.RWTH-Aachen.DE (RBI-Z-5/8.6.12) with ESMTP id PAA12053; Fri, 4 Apr 1997 15:31:36 +0200 (MET DST) Received: (from kuku@localhost) by gilberto.physik.rwth-aachen.de (8.8.5/8.6.9) id PAA07548; Fri, 4 Apr 1997 15:45:18 +0200 (MET DST) Message-ID: Date: Fri, 4 Apr 1997 15:45:17 +0200 From: kuku@gilberto.physik.rwth-aachen.de (Christoph Kukulies) To: jkh@time.cdrom.com (Jordan K. Hubbard) Cc: hubs@freebsd.org Subject: Re: Things moving around on ftp://ftp.freebsd.org/pub/FreeBSD References: <17711.860119647@time.cdrom.com> X-Mailer: Mutt 0.58e Mime-Version: 1.0 In-Reply-To: <17711.860119647@time.cdrom.com>; from Jordan K. Hubbard on Apr 3, 1997 18:07:27 -0800 Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Jordan K. Hubbard writes: > We've overflowed our 1st 4Gb partition there and now need to go to a > 2nd, but that means splitting off some of the bits which were > traditionally confined to one filesystem, and I'm still not sure what > the effects on the mirrors is going to be. If you mirror individual > collections, like pub/FreeBSD/2.1.7.1-RELEASE/, then you should be OK I switched to mirroring single pieces although I have all /pub/FreeBSD in a single (dual drive) 6GB ccd device so it wouldn't affect ftp.de.freebsd.org. (Machine is a AmdK5/133 w 64MB and 128 MB swap). I switched because mirror 2.8 (with perl5.003) *never*, really *never*, finished mirroring the whole tree at once. I suspect it a problem deep in mirror and/or perl. One time I let the mirror process run for 5 days with debug on and it turned out that it needed 5 minutes to decide for the next Get (the actual file transfer time was negeglible). This resulted in a predicted (at 40,000 files to fetch - give some worse case) time of 158 days (theoretically). I was suspecting some 2^n effect in searching or db looking up. Anyway, now breaking it into pieces again seems to get through. So my question here - sorry that I jumped into this thread: Is *anyone* out there mirroring the whole tree in one piece using mirror *and* is using a FreeBSD-x.x.x machine for this? I would be very interested in knowing this. Some sites I know of use big Sun machines and they don't seem to suffer from this problem. > since mirror should follow the symlink. If you're grabbing all Does it? I'm not sure. mirror has a variable: follow_local_symlink='' ( define empty by default) and I'm not sure what this means, local from the view of mirror would be the machine mirror runs on from what I understand. > of pub/FreeBSD, however, what you're going to see afterwards is a link: > 2.1.7.1-RELEASE -> ../../.3/FreeBSD/2.1.7.1-RELEASE Is there a possibility to dump some part of the current FreeBSD tree? Like 2.1.6? > > I'm not quite sure what the "best" work-around for this is, but > I just thought I'd mention it. Sorry folks, but it had to happen > sooner or later - the ports and packages are huge! ;-) > > This doesn't effect users who get their bits from ftp.freebsd.org > directly, of course, only those who mirror it. > > Jordan -- --Chris Christoph P. U. Kukulies kuku@gil.physik.rwth-aachen.de From owner-freebsd-hubs Fri Apr 4 09:07:38 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id JAA19812 for hubs-outgoing; Fri, 4 Apr 1997 09:07:38 -0800 (PST) Received: from icparc-pc-zl.doc.ic.ac.uk (icparc-pc-zl.doc.ic.ac.uk [146.169.24.24]) by freefall.freebsd.org (8.8.5/8.8.5) with SMTP id JAA19798 for ; Fri, 4 Apr 1997 09:06:55 -0800 (PST) Received: from triumph.doc.ic.ac.uk [146.169.24.1] by icparc-pc-zl.doc.ic.ac.uk with smtp (Exim 1.61 #1) id 0wDCRJ-0000j3-00; Fri, 4 Apr 1997 18:06:13 +0100 Message-ID: <33453517.2781E494@doc.ic.ac.uk> Date: Fri, 04 Apr 1997 18:06:31 +0100 From: Lee McLoughlin Organization: IC-Parc, Imperial College, University of London. U.K. X-Mailer: Mozilla 3.0 (X11; I; SunOS 4.1.3_U1 sun4m) MIME-Version: 1.0 To: Christoph Kukulies CC: "Jordan K. Hubbard" , hubs@freebsd.org Subject: Re: Things moving around on ftp://ftp.freebsd.org/pub/FreeBSD References: <17711.860119647@time.cdrom.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Christoph Kukulies wrote: > > I switched to mirroring single pieces although I have all /pub/FreeBSD > in a single (dual drive) 6GB ccd device so it wouldn't affect > ftp.de.freebsd.org. (Machine is a AmdK5/133 w 64MB and 128 MB swap). > > I switched because mirror 2.8 (with perl5.003) *never*, really *never*, > finished mirroring the whole tree at once. I suspect it a problem > deep in mirror and/or perl. One time I let the mirror process run for 5 days > with debug on and it turned out that it needed 5 minutes to decide > for the next Get (the actual file transfer time was negeglible). This > resulted in a predicted (at 40,000 files to fetch - give some worse case) > time of 158 days (theoretically). I was suspecting some 2^n effect in > searching or db looking up. Anyway, now breaking it into pieces again > seems to get through. Personally I blame perl :-> The next, perl, release of mirror will have an option to mirror directory by directory. My worry is that this will require an 'ls -l' to be run in each directory on the server - this could prove to be crippling to some servers. For FreeBSD it could mean replacing one 'get ls-lR' by 19,000 calls to 'ls -l'! Anyhow this will be discussed further on the mirror mailing list. > > So my question here - sorry that I jumped into this thread: > > Is *anyone* out there mirroring the whole tree in one piece using mirror > *and* is using a FreeBSD-x.x.x machine for this? I would be very > interested in knowing this. Some sites I know of use big Sun machines > and they don't seem to suffer from this problem. > > > since mirror should follow the symlink. If you're grabbing all > > Does it? I'm not sure. mirror has a variable: follow_local_symlink='' > ( define empty by default) and I'm not sure what this means, local from > the view of mirror would be the machine mirror runs on from what I > understand. I could use this on my system here in the UK to have a symlink called 'incoming' pointing elsewhere. But mirror would treat this like a real directory and NOT remove it to replace by a real directory as it normally would do. The trick that some people do is to add the 'L' flag to the flags_recursive so that when ls is run all the remote symlinks are expanded out. However if symlinks inside the area point to one of their parent directories this results in ls going into an infinite loop. -- Lee McLoughlin. Phone: +44 171 594 8388 IC-Parc, Imperial College, Fax: +44 171 594 8432 South Kensington, London. SW7 2BZ. UK. Email: L.McLoughlin@doc.ic.ac.uk From owner-freebsd-hubs Fri Apr 4 09:15:46 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id JAA20274 for hubs-outgoing; Fri, 4 Apr 1997 09:15:46 -0800 (PST) Received: from icparc-pc-zl.doc.ic.ac.uk (icparc-pc-zl.doc.ic.ac.uk [146.169.24.24]) by freefall.freebsd.org (8.8.5/8.8.5) with SMTP id JAA20258 for ; Fri, 4 Apr 1997 09:15:41 -0800 (PST) Received: from triumph.doc.ic.ac.uk [146.169.24.1] by icparc-pc-zl.doc.ic.ac.uk with smtp (Exim 1.61 #1) id 0wDCYg-0000jA-00; Fri, 4 Apr 1997 18:13:50 +0100 Message-ID: <334536E1.446B9B3D@doc.ic.ac.uk> Date: Fri, 04 Apr 1997 18:14:09 +0100 From: Lee McLoughlin Organization: IC-Parc, Imperial College, University of London. U.K. X-Mailer: Mozilla 3.0 (X11; I; SunOS 4.1.3_U1 sun4m) MIME-Version: 1.0 To: dg@root.com CC: "Jordan K. Hubbard" , hubs@freebsd.org Subject: Re: Things moving around on ftp://ftp.freebsd.org/pub/FreeBSD References: <199704041218.EAA11522@root.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk David Greenman wrote: > >CCD? > > "Concantenated Disk" driver. It's a pseudo disk device that lets one > create simple volume sets from individual disk drives. It is usually used to > create large pseudo disks from an array of smaller ones. It has no 'parity' > bit or other volume fault tolerance mechanisms, however, so if one drive > fails you lose the entire volume. Not come across it before, thanks for the info. I've only used concat' filestores where I'm sure the disks are good and dumps are taken regularly!!! > sunsite.doc.ic.ac.uk does indeed look impressive. It's obvious that > someone has spent a lot of money on it. :-) ...we have to work with what > we have, however, and I think our current operations and policies are > generally working well. SunSITE is worth a bob or two! Now if only I could convince people to pay staff salaries to run it I'd be happy!! BTW we are only too happy to help house things like the BSD area. The problem for most people is that they are in the US and our SunSITE is in the UK. -- Lee McLoughlin. Phone: +44 171 594 8388 IC-Parc, Imperial College, Fax: +44 171 594 8432 South Kensington, London. SW7 2BZ. UK. Email: L.McLoughlin@doc.ic.ac.uk