Date: Sat, 17 May 2014 02:50:17 +0000 (UTC) From: Gavin Atkinson <gavin@FreeBSD.org> To: doc-committers@freebsd.org, svn-doc-all@freebsd.org, svn-doc-head@freebsd.org Subject: svn commit: r44849 - head/en_US.ISO8859-1/articles/hubs Message-ID: <201405170250.s4H2oHex018827@svn.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: gavin Date: Sat May 17 02:50:17 2014 New Revision: 44849 URL: http://svnweb.freebsd.org/changeset/doc/44849 Log: Start to bring this up-to-date: - Drop all references to providing CVSup mirrors - Drop all references to providing anonymous CVS mirrors - Drop all references to mirroring using CVSup, instead use rsync - Update the recommended arguments to rsync in the examples - Updatre size estimates for the individual parts - Change one reference to CVS to instead mention SVN Note that we are still not accepting applications for new mirrors of this style. Modified: head/en_US.ISO8859-1/articles/hubs/article.xml Modified: head/en_US.ISO8859-1/articles/hubs/article.xml ============================================================================== --- head/en_US.ISO8859-1/articles/hubs/article.xml Fri May 16 19:28:26 2014 (r44848) +++ head/en_US.ISO8859-1/articles/hubs/article.xml Sat May 17 02:50:17 2014 (r44849) @@ -57,7 +57,7 @@ and degree of completeness you want to mirror, a huge amount of disk space may be consumed. Also keep in mind that <emphasis>official</emphasis> mirrors are probably required to be - complete. The CVS repository and the web pages should + complete. The web pages should always be mirrored completely. Also note that the numbers stated here are reflecting the current state (at &rel2.current;-RELEASE/&rel.current;-RELEASE). Further development and @@ -67,10 +67,9 @@ Here are some approximate figures: </para> <itemizedlist> - <listitem><para>Full FTP Distribution: 1.1 TB</para></listitem> - <listitem><para>CVS repository: 5.4 GB</para></listitem> - <listitem><para>CTM deltas: 3.2 GB</para></listitem> - <listitem><para>Web pages: 463 MB</para></listitem> + <listitem><para>Full FTP Distribution: 1.4 TB</para></listitem> + <listitem><para>CTM deltas: 10 GB</para></listitem> + <listitem><para>Web pages: 1GB</para></listitem> </itemizedlist> <para> The current disk usage of FTP Distribution can be found at @@ -113,7 +112,7 @@ </para> <para> For a moderately visited site that offers - <application>Rsync</application>, you might + <application>rsync</application>, you might consider a current CPU with around 800MHz - 1 GHz, and at least 512MB RAM. This is probably the minimum you want for an <emphasis>official</emphasis> @@ -127,7 +126,7 @@ </para> <para> You also want to consider a fast disk subsystem. - Operations on the CVS repository require a fast + Operations on the SVN repository require a fast disk subsystem (RAID is highly advised). A SCSI controller that has a cache of its own can also speed up things since most of these services incur a @@ -181,17 +180,9 @@ </listitem> <listitem><para><package>ftp/twoftpd</package>: As above.</para></listitem> <listitem><para><package>ftp/vsftpd</package>: The <quote>very secure</quote> ftpd.</para></listitem> - <listitem> - <para><package>ftp/wu-ftpd</package>: The ftpd from Washington - University. It has become infamous, because of the huge - amount of security issues that have been found in it. - If you do choose to use this software be sure to - keep it up to date. - </para> - </listitem> </itemizedlist> - <para>FreeBSD's <application>ftpd</application>, <application>proftpd</application>, - <application>wu-ftpd</application> and maybe <application>ncftpd</application> + <para>FreeBSD's <application>ftpd</application>, <application>proftpd</application> + and maybe <application>ncftpd</application> are among the most commonly used FTPds. The others do not have a large userbase among mirror sites. One thing to consider is that you may need flexibility in limiting @@ -260,109 +251,6 @@ </listitem> </itemizedlist> </sect3> - <sect3 xml:id="mirror-serv-cvsup"> - <title>CVSup (desired for CVS repository)</title> - <para> - <application>CVSup</application> is a very efficient way of distributing files. - It works similar to <application>rsync</application>, but was specially designed for - use with CVS repositories. If you want to offer the - FreeBSD CVS repository, you really want to consider - offering it via <application>CVSup</application>. It is possible to offer - the CVS repository via <application>AnonCVS</application>, FTP, - <application>rsync</application> or HTTP, but - people would benefit much more from <application>CVSup</application> access. - <application>CVSup</application> was developed by &a.jdp.email;. - It is a bit tricky to install on non-FreeBSD platforms, - since it is written in Modula-3 and therefore requires - a Modula-3 environment. &a.jdp; has built a - stripped down version of M3 that is sufficient to - run <application>CVSup</application>, and can be installed much easier. - See <link xlink:href="http://www.polstra.com/projects/freeware/ezm3/">Ezm3</link>; - for details. Related ports are:</para> - - <itemizedlist> - <listitem> - <para><package>net/cvsup</package>: The native CVSup port (client and server) - which requires <package>lang/ezm3</package> now.</para> - </listitem> - <listitem> - <para><package>net/cvsup-mirror</package>: The CVSup mirror kit, which requires - <package>net/cvsup-without-gui</package>, and configures it mirror-ready. Some - site administrators may want a different setup though. - </para> - </listitem> - </itemizedlist> - - <para>There are a few more like - <package>net/cvsup-without-gui</package> you might want to have - a look at. If you prefer a static binary package, take a look - <link xlink:href="http://people.FreeBSD.org/~jdp/s1g/">here</link>. - This page still refers to the S1G bug that was present - in <application>CVSup</application>. Maybe - John will set up a generic download-site to get - static binaries for various platforms. - </para> - <para> - It is possible to use <application>CVSup</application> to offer - any kind of fileset, not just CVS repositories, - but configuration can be complex. - <application>CVSup</application> is known to eat some CPU on both the server and the - client, since it needs to compare lots of files. - </para> - </sect3> - <sect3 xml:id="mirror-anoncvs"> - <title>AnonCVS (optional for CVS repository)</title> - <para> - If you have the CVS repository, you may want to offer - anonymous CVS access. A short warning first: - There is not much demand for it, - it requires some experience, and you need to know - what you are doing. - </para> - <para> - Generally there are two ways - to access a CVS repository remotely: via - <emphasis>pserver</emphasis> or via <command>ssh</command> - (we do not consider <command>rsh</command>). - For anonymous access, <emphasis>pserver</emphasis> is - very well suited, but some still offer <command>ssh</command> - access as well. There is a custom crafted - <link xlink:href="ftp://ftp.FreeBSD.org/pub/FreeBSD/development/FreeBSD-CVS/anoncvs.shar">wrapper</link>; - in the CVS repository, to be used as a login-shell for the - anonymous ssh account. It does a chroot, and therefore - requires the CVS repository to be available under the - anonymous user's home-directory. This may not be possible - for all sites. If you just offer <emphasis>pserver</emphasis> - this restriction does not apply, but you may run with - more security risks. You do not need to install any special - software, since &man.cvs.1; comes with - FreeBSD. You need to enable access via <command>inetd</command>, - so add an entry into your <filename>/etc/inetd.conf</filename> - like this:</para> - <programlisting> -cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --allow-root=/home/ncvs pserver - </programlisting> - <para>See the manpage for details of the options. Also see the CVS <emphasis>info</emphasis> - page about additional ways to make sure access is read-only. - It is advised that you create an unprivileged account, - preferably called <systemitem class="username">anoncvs</systemitem>. - Also you need to create a file <filename>passwd</filename> - in your <filename>/home/ncvs/CVSROOT</filename> and assign a - CVS password (empty or <literal>anoncvs</literal>) to that user. - The directory <filename>/anoncvstmp</filename> is a special - purpose memory based file system. It is not required but - advised since &man.cvs.1; creates a shadow directory - structure in your <filename>/tmp</filename> which is - not used after the operation but slows things - dramatically if real disk operations are required. - Here is an excerpt from <filename>/etc/fstab</filename>, - how to set up such a MFS:</para> - <programlisting> -/dev/da0s1b /anoncvstmp mfs rw,-s=786432,-b=4096,-f=512,-i=560,-c=3,-m=0,nosuid 0 0 - </programlisting> - <para>This is (of course) tuned a lot, and was suggested by &a.jdp.email;. - </para> - </sect3> </sect2> </sect1> <sect1 xml:id="mirror-howto"> @@ -374,8 +262,8 @@ cvspserver stream tcp nowait root /usr/b the various parts of FreeBSD, what tools to use, and where to mirror from. </para> - <sect2 xml:id="mirror-ftp"> - <title>FTP</title> + <sect2 xml:id="mirror-ftp-rsync"> + <title>Mirroring the FTP site</title> <para> The FTP area is the largest amount of data that needs to be mirrored. It includes the <emphasis>distribution @@ -388,39 +276,8 @@ cvspserver stream tcp nowait root /usr/b for various FreeBSD versions, and various architectures. </para> - <sect3 xml:id="mirror-ftp-ftp"> - <title>With FTP mirror</title> <para> - You can use a <application>FTP mirror</application> - program to get the files. Some of the most commonly used are:</para> - <itemizedlist> - <listitem><para><package>ftp/mirror</package></para></listitem> - <listitem><para><package>ftp/ftpmirror</package></para></listitem> - <listitem><para><package>ftp/emirror</package></para></listitem> - <listitem><para><package>ftp/spegla</package></para></listitem> - <listitem><para><package>ftp/omi</package></para></listitem> - <listitem><para><package>ftp/wget</package></para></listitem> - </itemizedlist> - - <para><package>ftp/mirror</package> was very popular, but seemed - to have some drawbacks, as it is written in &man.perl.1;, - and had real problems with mirroring large - directories like a FreeBSD site. There are rumors that - the current version has fixed this by allowing - a different algorithm for comparing - the directory structure to be specified. - </para> - <para> - In general FTP is not really good for mirroring. It transfers - the whole file if it has changed, and does - not create a single data stream which would benefit from - a large TCP congestion window. - </para> - </sect3> - <sect3 xml:id="mirror-ftp-rsync"> - <title>With rsync</title> - <para> - A better way to mirror the FTP area is <application>rsync</application>. + The best way to mirror the FTP area is <application>rsync</application>. You can install the port <package>net/rsync</package> and then use rsync to sync with your upstream host. <application>rsync</application> is already mentioned @@ -440,7 +297,7 @@ cvspserver stream tcp nowait root /usr/b </para> </note> <para>A command line to mirror FreeBSD might look like:</para> - <screen>&prompt.user; <userinput>rsync -vaz --delete ftp4.de.FreeBSD.org::FreeBSD/ /pub/FreeBSD/</userinput> + <screen>&prompt.user; <userinput>rsync -vaHz --delete rsync://ftp4.de.FreeBSD.org/FreeBSD/ /pub/FreeBSD/</userinput> </screen> <para>Consult the documentation for <application>rsync</application>, which is also available at @@ -453,188 +310,15 @@ cvspserver stream tcp nowait root /usr/b want to set up a script framework that calls such a command via &man.cron.8;. </para> - </sect3> - <sect3 xml:id="mirror-ftp-cvsup"> - <title>With CVSup</title> - <para> - A few sites, including the one-and-only <systemitem class="fqdomainname">ftp-master.FreeBSD.org</systemitem> - even offer <application>CVSup</application> to mirror the contents of - the FTP space. You need to install a <application>CVSup</application> - client, preferably from the port <package>net/cvsup</package>. - (Also reread <xref linkend="mirror-serv-cvsup"/>.) - A sample <filename>supfile</filename> suitable for <systemitem class="fqdomainname">ftp-master.FreeBSD.org</systemitem> - looks like this:</para> - <programlisting> - # - # FreeBSD archive supfile from master server - # - *default host=ftp-master.FreeBSD.org - *default base=/usr - *default prefix=/pub - #*default release=all - *default delete use-rel-suffix - *default umask=002 - - # If your network link is a T1 or faster, comment out the following line. - #*default compress - - FreeBSD-archive release=all preserve - </programlisting> - - <para>It seems <application>CVSup</application> would be the best - way to mirror the archive in terms of efficiency, but - it is only available from few sites.</para> - <note xml:id="mirror-cvsup-s-option"> - <para> - Please have look at the <application>CVSup</application> documentation - like &man.cvsup.1; and consider using the <option>-s</option> - option. This reduces I/O operations by assuming the - recorded information about each file is correct.</para> - </note> - </sect3> - </sect2> - <sect2 xml:id="mirror-cvs"> - <title>Mirroring the CVS repository</title> - <para>There are various ways to mirror the CVS repository. - <application>CVSup</application> is the most common method.</para> - - <sect3 xml:id="mirror-cvs-cvsup"> - <title>Using CVSup</title> - <para> - <application>CVSup</application> is described in some - detail in <xref linkend="mirror-serv-cvsup"/> and <xref linkend="mirror-ftp-cvsup"/>. - </para> - <para>It is very easy to setup a - <application>CVSup</application> mirror. Installing - <package>net/cvsup-mirror</package> will - make sure all of the needed programs are installed and then - gather all the needed information to configure the mirror.</para> - <note> - <para> - Please do not forget to consider the hint - mentioned in <link linkend="mirror-cvsup-s-option">this note</link> - above. - </para> - </note> - </sect3> - <sect3 xml:id="mirror-cvs-other"> - <title>Using other methods</title> - <para> - Using other methods than <application>CVSup</application> is - generally not recommended. We describe them in short here - anyway. Since most sites offer the CVS repository as - part of the FTP fileset under the path - <filename>/pub/FreeBSD/development/FreeBSD-CVS</filename>, - the following methods could be used.</para> - <itemizedlist> - <listitem><para><application>FTP</application></para></listitem> - <listitem><para><application>Rsync</application></para></listitem> - <listitem><para><application>HTTP</application></para></listitem> - </itemizedlist> - - <important> - <para>AnonCVS cannot be used to mirror the CVS repository - since CVS does not allow you to access the repository - itself, only checked out versions of the modules.</para> - </important> - </sect3> </sect2> <sect2 xml:id="mirror-www"> <title>Mirroring the WWW pages</title> <para> - The best way is to check out the <emphasis>www</emphasis> - distribution from CVS. If you have a local mirror of the - CVS repository, it is as easy as:</para> - <screen>&prompt.user; <userinput>cvs -d /home/ncvs co www</userinput></screen> - <para>and a <emphasis>cronjob</emphasis>, that calls <command>cvs up -d -P</command> - on a regular basis, maybe just after your repository was updated. - Of course, the files need to remain in a directory available - for public WWW access. The installation and configuration of a - web server is not discussed here. - </para> - - <para> - If you do not have a local repository, you can use - <application>CVSup</application> to maintain an <quote>up to date copy</quote> - of the www pages. A sample supfile can be found in - <filename>/usr/share/examples/cvsup/www-supfile</filename> and - could look like this:</para> - <programlisting> - # - # WWW module supfile for FreeBSD - # - *default host=cvsup3.de.FreeBSD.org - *default base=/usr - *default prefix=/usr/local - *default release=cvs tag=. - *default delete use-rel-suffix - - # If your network link is a T1 or faster, comment out the following line. - *default compress - - # This collection retrieves the www/ tree of the FreeBSD repository - www - </programlisting> - <para> - Using <package>ftp/wget</package> or other web-mirror tools is - not recommended. - </para> - <sect3 xml:id="mirror-www-doc"> - <title>Mirroring the FreeBSD documentation</title> - <para> - Since the documentation is referenced a lot from the - web pages, it is recommended that you mirror the - FreeBSD documentation as well. However, this is not - as trivial as the www-pages alone. - </para> - <para> - First of all, you should get the doc sources, - again preferably via <application>CVSup</application>. - Here is a corresponding sample supfile:</para> - <programlisting> - # - # FreeBSD documentation supfile - # - *default host=cvsup3.de.FreeBSD.org - *default base=/usr - *default prefix=/usr/share - *default release=cvs tag=. - *default delete use-rel-suffix - - # If your network link is a T1 or faster, comment out the following line. - #*default compress - - # This will retrieve the entire doc branch of the FreeBSD repository. - # This includes the handbook, FAQ, and translations thereof. - doc-all - </programlisting> - <para> - Then you need to install a couple of ports. - You are lucky, there is a meta-port: - <package>textproc/docproj</package> to do the work - for you. You need to set up some - environment variables, like - <literal>SGML_CATALOG_FILES</literal>. - Also have a look at your <filename>/etc/make.conf</filename> - (copy <filename>/usr/share/examples/etc/make.conf</filename> if - you do not have one), and look at the - <literal>DOC_LANG</literal> variable. - Now you are probably ready to run <command>make</command> - in your doc directory (<filename>/usr/share/doc</filename> - by default) and build the documentation. - Again you need to make it accessible for your web server - and make sure the links point to the right location.</para> - <important> - <para> - The building of the documentation, as well as lots - of side issues, is documented itself in the - <link xlink:href="&url.books.fdp-primer;">&os; Documentation - Project Primer</link>. - Please read this piece of documentation, especially if you - have problems building the documentation. - </para> - </important> - </sect3> + The FreeBSD website should only be mirrored via + <application>rsync</application>.</para> + <para>A command line to mirror the FreeBSD web site might look like:</para> + <screen>&prompt.user; <userinput>rsync -vaHz --delete rsync://bit0.us-west.freebsd.org/FreeBSD-www-data/ /usr/local/www/</userinput> + </screen> </sect2> <sect2 xml:id="mirror-how-often"> <title>How often should I mirror?</title> @@ -683,7 +367,6 @@ cvspserver stream tcp nowait root /usr/b Here are some recommended schedules:</para> <itemizedlist> <listitem><para>FTP fileset: daily</para></listitem> - <listitem><para>CVS repository: hourly</para></listitem> <listitem><para>WWW pages: daily</para></listitem> </itemizedlist> </sect2> @@ -708,7 +391,7 @@ cvspserver stream tcp nowait root /usr/b <emphasis>N</emphasis> is a number, telling that the host would be the <emphasis>Nth</emphasis> mirror in that country. - (Same applies to <systemitem>cvsupN.CC.FreeBSD.org</systemitem>, + (Same applies to <systemitem>wwwN.CC.FreeBSD.org</systemitem>, etc.) There are mirrors with no <emphasis>CC</emphasis> part. These are the mirror sites that are very well connected and @@ -812,9 +495,7 @@ cvspserver stream tcp nowait root /usr/b that. </para> <para> - There are three master sites for the FTP fileset and - one for the CVS repository (the web pages and docs are - obtained from CVS, so there is no need for master). + There is one master site for the FTP fileset. </para> <sect4 xml:id="mirror-where-master-ftp"> <title>ftp-master.FreeBSD.org</title> @@ -823,11 +504,9 @@ cvspserver stream tcp nowait root /usr/b </para> <para> <systemitem>ftp-master.FreeBSD.org</systemitem> provides - <application>rsync</application> and <application>CVSup</application> + <application>rsync</application> access, in addition to FTP. - Refer to <xref linkend="mirror-ftp-rsync"/> and - <xref linkend="mirror-ftp-cvsup"/> how to access - via these protocols. + Refer to <xref linkend="mirror-ftp-rsync"/>. </para> <para> Mirrors are also encouraged to allow <application>rsync</application> @@ -835,32 +514,6 @@ cvspserver stream tcp nowait root /usr/b <emphasis>Tier-1</emphasis>-mirrors. </para> </sect4> - <sect4 xml:id="mirror-where-master-cvsup"> - <title>cvsup-master.FreeBSD.org</title> - <para> - This is the master site for the CVS repository. - </para> - <para> - <systemitem>cvsup-master.FreeBSD.org</systemitem> provides - <application>CVSup</application> access only. - See <xref linkend="mirror-cvs-cvsup"/> for details. - </para> - <para> - To get access, you need to contact the &a.cvsup-master;. - Make sure you read the - <link xlink:href="http://people.FreeBSD.org/~jdp/cvsup-access/">FreeBSD CVSup Access Policy</link> - first! - </para> - <para> - Set up the required authentication by following - <link xlink:href="http://people.FreeBSD.org/~jdp/cvpasswd/">these - instructions</link>. Make sure you specify the server as - <systemitem>freefall.FreeBSD.org</systemitem> on the <command>cvpasswd</command> - command line, as described in this document, - even when you are contacting - <systemitem>cvsup-master.FreeBSD.org</systemitem> - </para> - </sect4> </sect3> </sect2> </sect1> @@ -897,14 +550,6 @@ cvspserver stream tcp nowait root /usr/b what <emphasis>official tier-1 mirrors</emphasis> are required to. All other official mirrors can consider this a big <emphasis>should</emphasis>.</para> - <note> - <para> - The following applies mainly to the FTP fileset, - since a CVS repository should always be mirrored - completely, and the web pages are a case of - its own. - </para> - </note> <para> Tier-1 mirrors are required to:</para> <itemizedlist>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201405170250.s4H2oHex018827>