Date: Fri, 12 Apr 2002 14:30:21 +0200 From: Daniel Lang <dl@leo.org> To: kuriyama@freebsd.org, logo@freebsd.org, freebsd-doc@freebsd.org Subject: Update to hubs article Message-ID: <20020412123021.GI24607@atrbg11.informatik.tu-muenchen.de>
next in thread | raw e-mail | index | archive | help
--EVF5PPMfhYS0aIcm Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi, here's another update to the hubs article in: doc/en_US.ISO8859-1/articles/hubs/article.sgml I've sent part's of this a while ago, but it did not get committed. Anyway, I've updated it according to the recent changes regarding email-adress entities, etc. The patch fixes a little bit of speech and adds another hint section about, when to mirror and reasonable schedules. Best regards, Daniel -- IRCnet: Mr-Spock - Eddie would go! - Daniel Lang * dl@leo.org * +49 89 289 25735 * http://www.leo.org/~dl/ --EVF5PPMfhYS0aIcm Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="article.sgml.diff" --- article.sgml.orig Fri Apr 12 14:15:08 2002 +++ article.sgml Fri Apr 12 14:23:21 2002 @@ -50,7 +50,7 @@ Depending on the set of releases, architectures, and degree of completeness you want to mirror, a huge amount of disk space may be consumed. Also keep in mind, - that official mirrors are probably required to be + that <emphasis>official</emphasis> mirrors are probably required to be complete. The CVS repository and the web pages should always be mirrored completely. Also note, that the numbers stated here, are reflecting the current @@ -101,7 +101,22 @@ indeed consume some CPU. For AnonCVS it can even be required to set up a MFS of at least 300 MB, so you need to take this into account - for your memory requirements. + for your memory requirements. The following + are just examples to give you a very rough hint. + </para> + <para> + For a moderately visited site, that offers + <application>rsync</application>, you might + consider a current CPU with around 800Mhz - 1 GHz, + and at least 512MB RAM. This is probably the + minimum you want for an <emphasis>official</emphasis> + site. + </para> + <para> + For a frequently used site, a Dual Athlon MP 1800, + with 2GB RAM may be more appropriate. + (Ok, this would be a very smooth configuration, + if you put like DDR-RAM into it ... :-)) </para> <para> You also want to consider a fast disk subsystem. @@ -113,9 +128,9 @@ <title>Services to offer</title> <para> This section describes the services that are - required for you to offer, and those that are + required for you to offer and those, that are optional. It also gives hints, which software - can be used to provide this service. + can be used to provide this particular service. </para> <sect3 id="mirror-serv-ftp"> <title>FTP (required for FTP fileset)</title> @@ -131,11 +146,11 @@ the path <filename>/pub/FreeBSD</filename>. </para> <para> - There is lot of software available which + There is a lot of software available which can be set up to allow anonymous FTP (in alphabetical order). <itemizedlist> - <listitem><para><command>/usr/libexec/ftpd</command>: FreeBSD's own ftpd, + <listitem><para><command>/usr/libexec/ftpd</command>: FreeBSD's own ftpd will probably do a very good job. Be sure to read &man.ftpd.8;.</para> </listitem> <listitem> @@ -156,32 +171,33 @@ <listitem><para><filename role="package">ftp/twoftpd</filename>: same as above</para></listitem> <listitem><para><filename role="package">ftp/vsftpd</filename>: the <quote>very secure</quote> ftpd</para></listitem> <listitem> - <para><filename role="package">ftp/wu-ftpd</filename>: The widely used ftpd from Washington + <para><filename role="package">ftp/wu-ftpd</filename>: The widely used <application>ftpd</application> from Washington University. It became sort of infamous, because of the huge amount of security issues that became obvious in wu-ftpd. - It also around for a long time, which can be positive + It is also around for a long time. This can be seen as positive (experience) or negative (lots of bloated code).</para> </listitem> </itemizedlist> - FreeBSD's ftpd, proftpd, wu-ftpd and maybe ncftpd are among the + FreeBSD's <application>ftpd</application>, <application>proftpd</application>, + <application>wu-ftpd</application> and maybe <application>ncftpd</application> are among the most used ones. The others are probably not that much widespread. </para> </sect3> <sect3 id="mirror-serv-rsync"> <title>RSYNC (optional for FTP fileset)</title> <para> - Rsync is often also offered for convenience, for the + <application>Rsync</application> is often also offered for convenience, for the contents of the FTP area of FreeBSD. The protocol is different from FTP in many ways, and - overall, it can be stated, that it is much more - bandwidth friendly, as only differences between files + overall it can be stated, that it is much more + bandwidth friendly, since only differences between files are transferred, not whole files. - Rsync does require significant amount of memory for + <application>Rsync</application> does require significant amount of memory for each instance. The size depends on the size of the synced module in terms of number of directories and - files. Rsync can use <command>rsh</command> and + files. <application>Rsync</application> can use <command>rsh</command> and <command>ssh</command> (now default) as a transport, - or use it's own protocol for stand-alone access + or use it's own transport for stand-alone access (this is the preferred method for public rsync servers). Authentication, connection limits, and other restrictions may be applied. There is just one software package @@ -197,10 +213,10 @@ If you want to offer the FreeBSD webpages, you need to install a webserver a.k.a <application>httpd</application>. You may offer the FTP fileset via HTTP, as well, if you like. - Some argue HTTP is more efficient for download, but + Some argue HTTP is more efficient for downloads, but I cannot tell. The most commonly - used httpd is Apache, although there are others around, - take a look at <filename>/usr/ports/www</filename>. + used httpd is <application>Apache</application>, although there are others around. + Take a look at <filename>/usr/ports/www</filename>. <itemizedlist> <listitem><para><filename role="package">www/apache13</filename></para></listitem> </itemizedlist> @@ -209,19 +225,20 @@ <sect3 id="mirror-serv-cvsup"> <title>CVSup (desired for CVS repository)</title> <para> - CVSup is a very efficient way of distributing files. - It works similar as rsync, but was specially designed for + <application>CVSup</application> is a very efficient way of distributing files. + It works similar to rsync, but was specially designed for the use with CVS repositories. If you want to offer the FreeBSD CVS repository, you really want to consider - offering it via CVSup. Still it is possible to offer - the CVS repository via AnonCVS, FTP, Rsync or HTTP, but - people would not benefit from CVSup access. - CVSup was developed by &a.jdp;. + offering it via <application>CVSup</application>. Still it is possible to offer + the CVS repository via <application>AnonCVS</application>, FTP, + <application>Rsync</application> or HTTP, but + people would benefit more from <application>CVSup</application> access. + <application>CVSup</application> was developed by &a.jdp;. It is a bit tricky to install on non-FreeBSD platforms, since it is written in Modula-3 and therefore requires - a Modula-3 environmen. John Polstra has built a + a Modula-3 environment. John Polstra has built a stripped down version of M3, that is sufficient to - run CVSup, and can be installed much easier. + run <application>CVSup</application>, and can be installed much easier. See <ulink url="http://www.polstra.com/projects/freeware/ezm3/">Ezm3</ulink>; for details. Related ports are: @@ -254,29 +271,26 @@ CVSup is known to eat some CPU on the server as on the client, since it needs to compare lots of files. </para> - <note> - <para> - Please have look at the <application>CVSup</application> - like &man.cvsup.1; and consider using the <option>-s</option> - option, as it can reduce the amount of work to be done - a lot. - </para> - </note> </sect3> <sect3 id="mirror-anoncvs"> <title>AnonCVS (optional for CVS repository)</title> <para> If you have the CVS repository, you may want to offer - anonymous CVS access. There is not that much demand for it, + anonymous CVS access. A short warning first: + There is not that much demand for it, and it requires some experience and you need to know, - what you are doing. Generally there are two ways, how + what you are doing. + </para> + <para> + Generally there are two ways, how to access a CVS repository remotely: via <emphasis>pserver</emphasis> or via <command>ssh</command> (we don't consider <command>rsh</command>). For anonymous access, <emphasis>pserver</emphasis> is very well suited, but some still offer <command>ssh</command> - access as well. There is some custom crafted wrapper - program around, to be used as a login-shell for the + access as well. There is a custom crafted + <ulink url="ftp://ftp.freebsd.org/pub/FreeBSD/development/FreeBSD-CVS/anoncvs.shar">wrapper</ulink>; + in the CVS repository, to be used as a login-shell for the anonymous ssh account. It does a chroot, and therefore requires the CVS repository to be available under the anonymous user's homedirectory, which may not be possible @@ -291,7 +305,7 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --allow-root=/home/ncvs pserver </programlisting> See the manpage for details of the options. See also the cvs <emphasis>info</emphasis> - page, about how additional ways, to make sure, access is read-only. + page, about additional ways to make sure, access is read-only. It is advisable, that you create an unprivileged account, preferrably called <username>anoncvs</username>. Also you need to create a file <filename>passwd</filename> @@ -352,7 +366,7 @@ </itemizedlist> <filename role="package">ftp/mirror</filename> was very popular, but seemed - to have some drawbacks, as its written in perl, + to have some drawbacks, as it is written in &man.perl.1;, and did have real problems on mirroring large directories like a FreeBSD site. There are rumours, that the current version has fixed this, by allowing @@ -360,7 +374,7 @@ the directory structure. </para> <para> - In general FTP is not really good for mirroring, since it transfers + In general FTP is not really good for mirroring, since it transferes each whole file, if it has changed, and does not create a single data stream, that will benefit from a large TCP congestion window. @@ -375,8 +389,8 @@ <application>rsync</application> is already mentioned in <xref linkend="mirror-serv-rsync">. Since <application>rsync</application> access is not - required, your preferred upstream site may not allow it, - but it already widely used, so chances are small, that + required, your preferred upstream site may not allow it. + Since it is quite common, though, chances are small, that you cannot use it. You can always consider using an upstream server, that offers it, just for the benefits of rsync. @@ -387,7 +401,7 @@ machine, most admins impose limitations on their server. For a mirror, you should ask the site maintainer you are syncing from, about their policy, and maybe - an exception for your host, since you are a mirror. + an exception for your host (since you are a mirror). </para> </note> A command line to mirror FreeBSD could look like that: @@ -433,13 +447,21 @@ way to mirror the archive, in terms of efficiency, but it is only available from few sites. In fact I just know <hostid>ftp-master.freebsd.org</hostid> for sure. + <note id="mirror-cvsup-s-option"> + <para> + Please have look at the <application>CVSup</application> documentation + like &man.cvsup.1; and consider using the <option>-s</option> + option, as it can reduce the amount of work to be done + a lot. + </para> + </note> </para> </sect3> </sect2> <sect2 id="mirror-cvs"> <title>Mirroring the CVS repository</title> <para> - Again you have various possibilities, but the most + Again you have various possibilies, but the most recommended one, is to use <link linkend="mirror-cvs-cvsup">CVSup</link>. </para> <sect3 id="mirror-cvs-cvsup"> @@ -469,6 +491,13 @@ You should also have a look at <filename>/usr/share/examples/cvsup</filename> </para> + <note> + <para> + Please don't forget to consider the hint, + mentioned in <link linkend="mirror-cvsup-s-option">this note</link> + above. + </para> + </note> </sect3> <sect3 id="mirror-cvs-other"> <title>Using other methods</title> @@ -478,7 +507,7 @@ anyway. Since most sites offer the CVS repository as part of the FTP fileset under the path <filename>/pub/FreeBSD/development/FreeBSD-CVS</filename>, - the following methods can of course be used. + the following methods could be used. <itemizedlist> <listitem><para><application>FTP</application></para></listitem> <listitem><para><application>RSYNC</application></para></listitem> @@ -493,7 +522,7 @@ <important> <para> - AFAICT, you can <emphasis>NOT</emphasis> AnonCVS to + AFAIK, you can <emphasis>NOT</emphasis> use AnonCVS to mirror the CVS repository, since CVS does not allow you to access the repository itself, but only checked out versions of the modules. @@ -591,7 +620,7 @@ <important> <para> The building of the documentation, as well as lots - of side issues is documented itself in: + of side issues, is documented itself in: <ulink url="http://www.freebsd.org/doc/en_US.ISO8859-1/books/fdp-primer/">fdp-primer</ulink>. Please read this piece of documentation, especially if you have problems, building the documentation. @@ -605,6 +634,52 @@ </para> </sect3> </sect2> + <sect2 id="mirror-how-often"> + <title>How often should I mirror?</title> + <para> + Every mirror should be updated on a regular + basis. You will certainly need some script + framework for it, that will be called by + &man.cron.8;. Since nearly every admin + does this his own way, we cannot give + specific instructions. It could work + like this: + </para> + <procedure> + <step> + <para> + Put the command to run your mirroring application + in a script. Use of a plain <command>/bin/sh</command> + script is recommended. + </para> + </step> + <step> + <para> + Add some output redirections, so diagnostic + messages are logged to a file. + </para> + </step> + <step> + <para> + Test, if your script works. Check the logs. + </para> + </step> + <step> + <para> + Use &man.crontab.1; to add the script the + appropriate user's &man.crontab.5;. + </para> + </step> + </procedure> + <para> + Here are some recommended schedules: + <itemizedlist> + <listitem><para>FTP fileset: daily</para></listitem> + <listitem><para>CVS repository: daily to hourly</para></listitem> + <listitem><para>WWW pages: daily</para></listitem> + </itemizedlist> + </para> + </sect2> </sect1> <sect1 id="mirror-where"> <title>Where to mirror from</title> @@ -617,15 +692,15 @@ <para> Mirrors are organised by country. All official mirrors have a DNS entry of the form - <hostid>ftpX.CC.freebsd.org</hostid>. Whith + <hostid>ftpN.CC.freebsd.org</hostid>. Whith <emphasis>CC</emphasis> (i.e. country code) beeing the <emphasis>top level domain</emphasis> of the country, where this mirror is located; - and <emphasis>X</emphasis> is a number, - telling that the host would be the <emphasis>Xth</emphasis> + and <emphasis>N</emphasis> is a number, + telling that the host would be the <emphasis>Nth</emphasis> mirror in that country. - (Same applies to <hostid>cvsupX.CC.freebsd.org</hostid>, - <hostid>wwwX.CC.freebsd.org</hostid>, etc.) + (Same applies to <hostid>cvsupN.CC.freebsd.org</hostid>, + <hostid>wwwN.CC.freebsd.org</hostid>, etc.) There are mirrors with no <emphasis>CC</emphasis> part. These are usually located in the US, but don't need to. <hostid>ftp.freebsd.org</hostid> is currently @@ -718,13 +793,12 @@ for access. If you are already an <emphasis>official</emphasis> mirror, this certainly helps you getting access. In any other case make sure your country really needs another mirror. - If it already has three or more, ask - the &a.hubs; first. + If it already has three or more, ask the &a.hubs; first. </para> <para> - There are just two master sites, for the FTP fileset and - for the CVS repository (the webpages and docs are obtained from CVS, - so there is no need for it). + There are just two master sites, one for the FTP fileset and + one for the CVS repository (the webpages and docs are + obtained from CVS, so there is no need for it). </para> <sect4 id="mirror-where-master-ftp"> <title>ftp-master.freebsd.org</title> @@ -769,7 +843,7 @@ Set up the required authentication by following <ulink url="http://people.freebsd.org/~jdp/cvpasswd/">these instructions</ulink>. Make sure you specify the server as - <hostid>freefall.freebsd.org</hostid> on the cvpasswd + <hostid>freefall.freebsd.org</hostid> on the <command>cvpasswd</command> command line, as described in this document, even when you are contacting <hostid>cvsup-master.freebsd.org</hostid> @@ -851,7 +925,7 @@ <sect2 id="mirror-official-become"> <title>How to become official then?</title> <para> - An interesting questions, especially, since the state + An interesting question, especially, since the state of beeing official comes with some benefits, like a much higher bill from your ISP, as more people will be using your site. Also it may be a key requirement, to get access @@ -889,13 +963,14 @@ If there is no subdomain delegated, yet, for your country, you probably need to contact <email>hostmaster@freebsd.org</email>, - however, you can try the &a.hubs; first. + however, you can try the &a.hubs; + first. </para> </step> <step> <para> - Then you can ask the &a.doc;, or the &a.hubs; to add your - mirror site to the mirror list in the + Then you can ask the &a.doc; or the the &a.hubs; + to add your mirror site to the mirror list in the <ulink url="http://www.freebsd.org/handbook">FreeBSD Handbook</ulink>. Make sure you tell them the email address, to list as the maintainer of the site. --EVF5PPMfhYS0aIcm-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-doc" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20020412123021.GI24607>