From owner-freebsd-hubs@FreeBSD.ORG Fri Aug 8 16:27:15 2003 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C74D437B401 for ; Fri, 8 Aug 2003 16:27:15 -0700 (PDT) Received: from lurza.secnetix.de (mail2.secnetix.de [195.143.231.20]) by mx1.FreeBSD.org (Postfix) with ESMTP id 32B6F43FB1 for ; Fri, 8 Aug 2003 16:27:14 -0700 (PDT) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (oxqvcp@localhost [127.0.0.1]) by lurza.secnetix.de (8.12.8p1/8.12.8) with ESMTP id h78NQwgC065235; Sat, 9 Aug 2003 01:26:59 +0200 (CEST) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.12.8p1/8.12.8/Submit) id h78NQsuu065234; Sat, 9 Aug 2003 01:26:54 +0200 (CEST) From: Oliver Fromme Message-Id: <200308082326.h78NQsuu065234@lurza.secnetix.de> To: jonny@jonny.eng.br (=?ISO-8859-1?Q?Jo=E3o_Carlos_Mendes_Lu=EDs?=) Date: Sat, 9 Aug 2003 01:26:54 +0200 (CEST) In-Reply-To: <3F2A9595.5010302@jonny.eng.br> from "=?ISO-8859-1?Q?Jo=E3o_Carlos_Mendes_Lu=EDs?=" at Aug 01, 2003 01:30:13 PM X-Mailer: ELM [version 2.5 PL3] MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit cc: freebsd-hubs@freebsd.org cc: Ken Smith cc: Garrett Wollman Subject: Re: Requirements Final Draft Attempt #2 :-/ X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD Distributions Hubs: mail sup ftp List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Aug 2003 23:27:16 -0000 João Carlos Mendes Luís wrote: > Oliver Fromme wrote: > > [...] > > All of that might sound complicated, but it really isn't > > that bad. The collector can be a small shell script using > > /usr/bin/fetch or automated ncftp. The extension to > > sysinstall shouldn't be too difficult either. I would be > > willing to work on the collector thing at least, and maybe > > also on the sysinstall part (if time permits). > > You should not need to get the files, only list them. Correct, that's what I meant. > Wouldn't this be much easier in perl? I do not like perl, but that task is indeed a bit heavy for a shell script. When I had a few minutes of free time, I coded a rough draft of such a script. I took the liberty to write it in Python. It doesn't do any DNS zone transfers, so it is not affected by reluctant DNS servers. Instead, the list of CC subdomains is hardcoded. For each subdomain, it starts at ftp.$DOMAIN, ftp1.$DOMAIN and increases the number until a host cannot be resolved. That seems to be a better approach than relying on zone transfers. The script produces ASCII output in a very simple format. It could easily be fed into a database (maybe I'll try it with PostgreSQL when I have a few more minutes of free time). I also wrote a web page which reads that ASCII file and presents the data in HTML tables. It doesn't look very pretty (I'm a programmer, not an artist), but serves well as a rough draft and proof-of-concept. http://www.secnetix.de/~olli/fbsd-ftp/ Regards Oliver -- Oliver Fromme, secnetix GmbH & Co KG, Oettingenstr. 2, 80538 München Any opinions expressed in this message may be personal to the author and may not necessarily reflect the opinions of secnetix in any way. "People who claim Windows is superior to Unix are the same people who'd argue that you better use your hand instead of toilet paper to wipe your ass. I can hear them now - 'It's colourful and it's intuitive and easy to use and even a child could do it.'" -- (unknown) From owner-freebsd-hubs@FreeBSD.ORG Fri Aug 8 17:22:57 2003 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5BA2437B401 for ; Fri, 8 Aug 2003 17:22:57 -0700 (PDT) Received: from arthur.nitro.dk (port324.ds1-khk.adsl.cybercity.dk [212.242.113.79]) by mx1.FreeBSD.org (Postfix) with ESMTP id A409643F85 for ; Fri, 8 Aug 2003 17:22:55 -0700 (PDT) (envelope-from simon@arthur.nitro.dk) Received: by arthur.nitro.dk (Postfix, from userid 1000) id 8B8BD10BFA8; Sat, 9 Aug 2003 02:22:54 +0200 (CEST) Date: Sat, 9 Aug 2003 02:22:54 +0200 From: "Simon L. Nielsen" To: Oliver Fromme Message-ID: <20030809002252.GB53916@FreeBSD.org> References: <3F2A9595.5010302@jonny.eng.br> <200308082326.h78NQsuu065234@lurza.secnetix.de> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="DBIVS5p969aUjpLe" Content-Disposition: inline In-Reply-To: <200308082326.h78NQsuu065234@lurza.secnetix.de> User-Agent: Mutt/1.5.4i cc: freebsd-hubs@freebsd.org Subject: Re: Requirements Final Draft Attempt #2 :-/ X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD Distributions Hubs: mail sup ftp List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Aug 2003 00:22:57 -0000 --DBIVS5p969aUjpLe Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable [CC trimmed] On 2003.08.09 01:26:54 +0200, Oliver Fromme wrote: >=20 > Jo=E3o Carlos Mendes Lu=EDs wrote: > > Oliver Fromme wrote: > > > [...] > > > All of that might sound complicated, but it really isn't > > > that bad. The collector can be a small shell script using > > > /usr/bin/fetch or automated ncftp. The extension to > > > sysinstall shouldn't be too difficult either. I would be > > > willing to work on the collector thing at least, and maybe > > > also on the sysinstall part (if time permits). > > > > Wouldn't this be much easier in perl? >=20 > I do not like perl, but that task is indeed a bit heavy for > a shell script. >=20 > When I had a few minutes of free time, I coded a rough draft > of such a script. I took the liberty to write it in Python. This is really cool! I think it would be very useful to have something like this available, and updated regularly. > It doesn't do any DNS zone transfers, so it is not affected > by reluctant DNS servers. Instead, the list of CC subdomains > is hardcoded. For each subdomain, it starts at ftp.$DOMAIN, > ftp1.$DOMAIN and increases the number until a host cannot be > resolved. That seems to be a better approach than relying > on zone transfers. One little problem is when a number is skipped. E.g. your script doesn't pick up ftp3.dk.freebsd.org, since there is no ftp2. I don't know if there are any other special cases like this, but it's not that important right now. --=20 Simon L. Nielsen FreeBSD Documentation Team --DBIVS5p969aUjpLe Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (FreeBSD) iD8DBQE/ND7ch9pcDSc1mlERAvDTAJ9Cqi7AOdIWQ+tvKrq+uBqbSbLuNACgpFX6 D9tpEc7YXV7BW0NFppFerqo= =6POV -----END PGP SIGNATURE----- --DBIVS5p969aUjpLe-- From owner-freebsd-hubs@FreeBSD.ORG Sat Aug 9 10:18:53 2003 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 91D6C37B401; Sat, 9 Aug 2003 10:18:53 -0700 (PDT) Received: from lurza.secnetix.de (mail2.secnetix.de [195.143.231.20]) by mx1.FreeBSD.org (Postfix) with ESMTP id 48B2643F85; Sat, 9 Aug 2003 10:18:51 -0700 (PDT) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (mjktux@localhost [127.0.0.1]) by lurza.secnetix.de (8.12.8p1/8.12.8) with ESMTP id h79HImgC094214; Sat, 9 Aug 2003 19:18:49 +0200 (CEST) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.12.8p1/8.12.8/Submit) id h79HImZ1094212; Sat, 9 Aug 2003 19:18:48 +0200 (CEST) From: Oliver Fromme Message-Id: <200308091718.h79HImZ1094212@lurza.secnetix.de> To: simon@freebsd.org (Simon L. Nielsen) Date: Sat, 9 Aug 2003 19:18:48 +0200 (CEST) In-Reply-To: <20030809002252.GB53916@FreeBSD.org> from "Simon L. Nielsen" at Aug 09, 2003 02:22:54 AM X-Mailer: ELM [version 2.5 PL3] MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit cc: freebsd-hubs@freebsd.org Subject: Re: Requirements Final Draft Attempt #2 :-/ X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD Distributions Hubs: mail sup ftp List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Aug 2003 17:18:53 -0000 Simon L. Nielsen wrote: > On 2003.08.09 01:26:54 +0200, Oliver Fromme wrote: > [...] > > It doesn't do any DNS zone transfers, so it is not affected > > by reluctant DNS servers. Instead, the list of CC subdomains > > is hardcoded. For each subdomain, it starts at ftp.$DOMAIN, > > ftp1.$DOMAIN and increases the number until a host cannot be > > resolved. That seems to be a better approach than relying > > on zone transfers. > > One little problem is when a number is skipped. E.g. your script > doesn't pick up ftp3.dk.freebsd.org, since there is no ftp2. I don't > know if there are any other special cases like this, but it's not that > important right now. I didn't know that there are "holes" in some countries' list of servers. In Germany, when one server goes away, we just duplicate another server, so there are no holes. That's why ftp3.de.freebsd.org and ftp6.de.freebsd.org point to the same machine. However, it seems that there is no "official" requirement to avoid gaps in the list. But that's not a big problem. I changed the script to try two more times, so it can jump over (small) gaps, at the expense of a few more DNS lookups. I re-ran the collector, and now ftp3.dk is in the list, among 45 other hosts that were missing before. :) Another possibility would be to try a zone transfer first, and if that fails, fall back to the existing algorithm. But I'm not sure if it is worth it. Regards Oliver -- Oliver Fromme, secnetix GmbH & Co KG, Oettingenstr. 2, 80538 München Any opinions expressed in this message may be personal to the author and may not necessarily reflect the opinions of secnetix in any way. "Any sufficiently advanced technology is indistinguishable from magic." Arthur C. Clarke's Third Law, from Profiles of the Future From owner-freebsd-hubs@FreeBSD.ORG Sat Aug 9 10:48:23 2003 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8486E37B401 for ; Sat, 9 Aug 2003 10:48:23 -0700 (PDT) Received: from electra.cse.Buffalo.EDU (electra.cse.Buffalo.EDU [128.205.32.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id BBCAF43FCB for ; Sat, 9 Aug 2003 10:48:22 -0700 (PDT) (envelope-from kensmith@cse.Buffalo.EDU) Received: from electra.cse.Buffalo.EDU (kensmith@localhost [127.0.0.1]) h79HmMbr022546 for ; Sat, 9 Aug 2003 13:48:22 -0400 (EDT) Received: (from kensmith@localhost) by electra.cse.Buffalo.EDU (8.12.9/8.12.9/Submit) id h79HmM7L022545 for freebsd-hubs@freebsd.org; Sat, 9 Aug 2003 13:48:22 -0400 (EDT) Date: Sat, 9 Aug 2003 13:48:22 -0400 From: Ken Smith To: freebsd-hubs@freebsd.org Message-ID: <20030809174822.GA22328@electra.cse.Buffalo.EDU> References: <20030809002252.GB53916@FreeBSD.org> <200308091718.h79HImZ1094212@lurza.secnetix.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200308091718.h79HImZ1094212@lurza.secnetix.de> User-Agent: Mutt/1.4.1i Subject: Re: Requirements Final Draft Attempt #2 :-/ X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD Distributions Hubs: mail sup ftp List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Aug 2003 17:48:23 -0000 On Sat, Aug 09, 2003 at 07:18:48PM +0200, Oliver Fromme wrote: [ Very nice work! :-] > However, it seems that there is no "official" requirement to > avoid gaps in the list. But that's not a big problem. This seems true - my DNS scan done earlier had gaps too. > I changed the script to try two more times, so it can jump > over (small) gaps, at the expense of a few more DNS lookups. > I re-ran the collector, and now ftp3.dk is in the list, among > 45 other hosts that were missing before. :) > > Another possibility would be to try a zone transfer first, > and if that fails, fall back to the existing algorithm. > But I'm not sure if it is worth it. I settled on iterating up to 20 and stopping there for what I did, which was strictly to generate a text-based list I could start with for mapping out the sites. I didn't do anything even slightly as nice as this. I did notice one spot your algorithm will have problems. The Hong Kong zone is, I think, using a wildcard for ftp sites. DNS queries all the way up to 20 came back with an answer and almost all of them were the same site. That was the only CC based zone that seemed to work that way - the rest were more like what you expect. I can provide you with a slightly larger list of CC based zones if you want to hardcode the existing list. Or I can provide you with what I used to "seed" my DNS scan with. Basically I took the list if IANA defined country codes and first probed to see if anything was defined for those. My probe threw out the ones that the result of "host -t any ${cc}.freebsd.org" returned an error code for and it then probed for ftp sites inside the remaining list. I can provide you with the list of CC's designated by IANA or I can give you the current list of CC-based zones in freebsd.org, or both. Whatever would be most useful. :-) Thanks a lot, this is great. -- Ken Smith - From there to here, from here to | kensmith@cse.buffalo.edu there, funny things are everywhere. | - Theodore Geisel | From owner-freebsd-hubs@FreeBSD.ORG Sat Aug 9 21:22:09 2003 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5476D37B401 for ; Sat, 9 Aug 2003 21:22:09 -0700 (PDT) Received: from coe.ufrj.br (roma.coe.ufrj.br [146.164.53.65]) by mx1.FreeBSD.org (Postfix) with ESMTP id C1D4F43FDF for ; Sat, 9 Aug 2003 21:22:06 -0700 (PDT) (envelope-from jonny@jonny.eng.br) Received: from jonny.eng.br (unknown [200.165.175.135]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by coe.ufrj.br (Postfix) with ESMTP id 5414857D832; Sun, 10 Aug 2003 01:22:01 -0300 (BRT) Message-ID: <3F35C87A.4050004@jonny.eng.br> Date: Sun, 10 Aug 2003 01:22:18 -0300 From: =?ISO-8859-1?Q?Jo=E3o_Carlos_Mendes_Lu=EDs?= User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4) Gecko/20030624 Netscape/7.1 (ax) X-Accept-Language: pt-br, en-us, en, pt MIME-Version: 1.0 To: Oliver Fromme References: <200308082326.h78NQsuu065234@lurza.secnetix.de> In-Reply-To: <200308082326.h78NQsuu065234@lurza.secnetix.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit cc: freebsd-hubs@freebsd.org cc: Ken Smith cc: Garrett Wollman Subject: Re: Requirements Final Draft Attempt #2 :-/ X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD Distributions Hubs: mail sup ftp List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Aug 2003 04:22:09 -0000 Hi Oliver, Oliver Fromme wrote: > João Carlos Mendes Luís wrote: > > Oliver Fromme wrote: > > > [...] > > > All of that might sound complicated, but it really isn't > > > that bad. The collector can be a small shell script using > > > /usr/bin/fetch or automated ncftp. The extension to > > > sysinstall shouldn't be too difficult either. I would be > > > willing to work on the collector thing at least, and maybe > > > also on the sysinstall part (if time permits). > > > > You should not need to get the files, only list them. > > Correct, that's what I meant. > > > Wouldn't this be much easier in perl? > > I do not like perl, but that task is indeed a bit heavy for > a shell script. > > When I had a few minutes of free time, I coded a rough draft > of such a script. I took the liberty to write it in Python. No problem with python. I was just worried about bourne shell and fetch. > It doesn't do any DNS zone transfers, so it is not affected > by reluctant DNS servers. Instead, the list of CC subdomains > is hardcoded. For each subdomain, it starts at ftp.$DOMAIN, > ftp1.$DOMAIN and increases the number until a host cannot be > resolved. That seems to be a better approach than relying > on zone transfers. Good, and even better after the DNS hole fix. > The script produces ASCII output in a very simple format. > It could easily be fed into a database (maybe I'll try it > with PostgreSQL when I have a few more minutes of free time). > > I also wrote a web page which reads that ASCII file and > presents the data in HTML tables. It doesn't look very > pretty (I'm a programmer, not an artist), but serves well > as a rough draft and proof-of-concept. > > http://www.secnetix.de/~olli/fbsd-ftp/ I really liked it. It has even helped me find problems in my own mirror! ;-) Here's what it found about the mirror I maintain: tp.br.freebsd.org FTP Install Trees 4-STABLE - i386 - BAD 4.8-RELEASE - i386 4.8-STABLE - i386 - BAD 5.1-RELEASE - i386 CD ISO9660 Images 4.8-RELEASE - i386 5.1-RELEASE - i386 Software Packages 4.7-RELEASE - i386 - BAD 4.8-RELEASE - i386 - 4-stable and 4-8stable were there only to hold links to latest packages (and sometime ago, XFree86 links), to allow usage of /stand/sysinstall after cvsup recompiling. - 4.7 packages was a temporary link I have done, to 4.8 packages, for the same reason above, that I forgot to remove. These may be used as an example of some changes that could be useful: - After discovering some "content directory", verify it's contents to see if it's true. This is a little intrusive, but could be done by verifying the presence and size of some file, for example. - If a problem is found, send an email to the CC.freebsd.br hostmaster, and to postmaster@ftpX.CC.freebsd.org. Also, do not list it to the public. The discussion about ftp holes and links reminded me about other possible problem: Some ftp DNS names may point to multiple IP Addresses (I've seen this already, but do not know if it's valid), and some multiple DNS names could point to the same IP (ftp3.de and ftp6.de example). Your program is surely the best place to check for this, and probably list same IPs only once. In the overal, I really liked it! Thanks a lot for your help!!! Jonny -- João Carlos Mendes Luís - Networking Engineer - jonny@jonny.eng.br -- "the West won the world not by the superiority of its ideas or values or religion but rather by its superiority in applying organized violence. Westerners often forget this fact, non-Westerners never do." -- Samuel P. Huntington