Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 30 Nov 2001 10:10:28 +0200
From:      "Patrick O'Reilly" <patrick@mip.co.za>
To:        "Joseph Wright" <JWRIGHT@mbakercorp.com>, <freebsd-questions@FreeBSD.ORG>
Subject:   RE: Mirroring apache web sites
Message-ID:  <NDBBIMKICMDGDMNOOCAIMEFNEAAA.patrick@mip.co.za>
In-Reply-To: <sc065319.020@mbakercorp.com>

next in thread | previous in thread | raw e-mail | index | archive | help
> From: Joseph Wright
> Sent: 29 November 2001 22:24
>
> I am wondering what is the best way to mirror two websites.
> I have a website inside my network on a freebsd box(this is
> where we develop) and what I want to do is everynight push
> the changes to a website I have on my dmz. I would like the
> process to be automatic probably through cron. Should this be
> done with ftp or rsync or ????.    Any ideas
>

Joseph,

I have scripted a little solution which uses ftp.  One issue I had to
deal with, which I suspect you will have to deal with too, is that an
automated synchronisation may take across stuff that is still being
tested, or has not yet been approved.  Here is my solution, be it crude,
but it works fine for me.

1) On the development web server I set up an FTP account which has the
same root as the DocumentRoot of the www stuff.  I'll leave it up to you
to apply whatever security you find necessary.  You probably already
have a suitable ftp account which is used by the html editing tool the
web developers use to access the web documents.

2) I require the developers to simply add filenames of any newly changed
or added files to a file in that same root.  A command as simple as:
# echo "mynewfile.html" >> _UPLOADS.txt
does the trick.  You might script that to ensure no one accidentally
does it with just one > :)  Naturally, they should only add the file
name to the list when it is ready to publish.

3) On the LIVE web server I created the script (below), and added it to
cron to run periodically (in my case it is "30 * * * *").  Since this
script only fetches across the changes (based on _UPLOADS.txt), not the
whole website, the load is very low.  Therefore, if you want to run it
much more frequently it will not really place a heavy burden on either
server.  The script looks like this:
---------------------------------------
#!/bin/bash
# Script to load updated web site for genesis (gmed)

# this is just a place to do some work
cd /var/mip

# first, fetch the _UPLOADS.txt list
(
    echo "ascii"
    echo "get _UPLOADS.txt"
    echo "bye"
) | ftp -i devwww.mydomain.com

# if there is a list, then go to it!
if [ -f ./_UPLOADS.txt ]
then
    # build the full FTP script on the fly
    (
        # use binary in case there are images, etc
        echo "bin"
        # get each file specified
        for file in `cat ./_UPLOADS.txt`
        do
            # get the files and put them in a temp
            # directory locally
            echo "get $file ./gmed/$file"
        done
        # remove the list off the dev server
        echo "delete _UPLOADS.txt"
        # be polite
        echo "bye"
    ) | ftp -i devwww.mydomain.com

    # remove the list locally too now that we are done
    rm ./_UPLOADS.txt

    # move the whole lot from the temp directory to
    # the LIVE DocumentRoot directory
    cp -r ./gmed/* /www/html/gmed
    # and clean up after yourself
    rm -f ./gmed/*

fi
# That's all folks!
---------------------------------------

Of course, you will need a .netrc file to enable the FTP login to work:
---------------------------------------
machine devwww.mydomain.com login ftpsync password sesame
---------------------------------------

Note that the script does not check for existence of the directories in
the temporary workspace - you can add automated directory building into
the script, or just make sure they exist before you use the script.

Like I said, crude, but effective ;)  I hope this helps.


BTW: If you need to keep an audit of when changes have happened, you
could simply replace:
---------------------------------------
    # remove the list locally too now that we are done
    rm ./_UPLOADS.txt
---------------------------------------
with:
---------------------------------------
    # copy the _UPLOADS.txt file to the audit directory
    timestamp=`date +%Y%m%d%H%M`
    mv ./_UPLOADS.txt ./audit/$timestamp_UPLOADS.txt
---------------------------------------

Have fun,
Patrick.


To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-questions" in the body of the message




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?NDBBIMKICMDGDMNOOCAIMEFNEAAA.patrick>