From owner-freebsd-questions@FreeBSD.ORG Fri Oct 28 18:59:46 2011 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CA053106564A for ; Fri, 28 Oct 2011 18:59:46 +0000 (UTC) (envelope-from tundra2bear@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id 4A9148FC14 for ; Fri, 28 Oct 2011 18:59:46 +0000 (UTC) Received: by faar19 with SMTP id r19so5578278faa.13 for ; Fri, 28 Oct 2011 11:59:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=+fDaGPKXMk7R2klWt2SyAE0Se9/IqBtp0F5f9H1I5vI=; b=lpRXtuUzoDUdWjMfRcvxkHS9FsKxILE2/oSu6VEY9z0m+Yw3HQqdOEiDwKaeLWjbFH INHNKBRAoJzOAlvaf13qfp/NAF/dlBHJUbQIV/f7QYiwnQ4HVHI3MS9okAJUuxMqLYah BpxBZwhQeG+4+1BdBT/pABXev+EHDft6hUMe4= Received: by 10.223.4.215 with SMTP id 23mr7976220fas.8.1319828385145; Fri, 28 Oct 2011 11:59:45 -0700 (PDT) MIME-Version: 1.0 Received: by 10.223.103.200 with HTTP; Fri, 28 Oct 2011 11:59:24 -0700 (PDT) In-Reply-To: References: From: Peter Kryszkiewicz Date: Fri, 28 Oct 2011 11:59:24 -0700 Message-ID: To: "C. P. Ghost" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-questions@freebsd.org Subject: Re: rsync and the ports tree X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Oct 2011 18:59:46 -0000 Sounds like a well-thought out backup strategy. I've started to use your methods here, and I'm building ports I need at the same time, but the wireless here is not password protected so is incredibly slow as there are lots of leaches on the system. But while I'm plodding along, I have a few more questions: On Wed, Oct 26, 2011 at 8:07 AM, C. P. Ghost wrote: > On Wed, Oct 26, 2011 at 12:39 PM, Peter Kryszkiewicz > wrote: > > I have several machines installed in my temporary location and only my > > laptop gets the internet through wireless. So far I've been building > ports > > on the other machines by rsync'ing the distfiles from the laptop as I > need > > them (all machines have the same FreeBSD 8.2 installed). > > > > The problem comes after I did a 'portupgrade -a' on the laptop. To ensure > > the other ports trees are in sync, can I rsync the /usr/ports directory > to > > the other machines? Since some of them are different architectures (amd64 > > multicore for instance) I ran into situations where the distfiles are > > different (for gcc for example). > > First of all, rsync is working perfectly if you want to > distribute /usr/ports/distfiles, /usr/ports to your internal > machines, even when they are not of the same architecture. > I'm doing this with a BIG farm of servers running i386, amd64, > and sparc64 for a long, long time. > > You only need to make sure to rsync the *union* of your > /usr/ports/distfiles directories, or else it won't work. > > Say, on amd64 you have > /usr/ports/distfiles/some-distfile-for-amd64-only.tar.bz2 > and on i386 you have > /usr/ports/distfiles/some-distfile-for-i386-only.tar.bz2 > > Yes, that happens every now and then. > > So you have to rsync both ways, so that you end up with > /usr/ports/distfiles/some-distfile-for-amd64-only.tar.bz2 > /usr/ports/distfiles/some-distfile-for-i386-only.tar.bz2 > on both i386 and amd64 machines. > I've done that, it works well especially for the architecture differences. gcc requires an additional distfile for the amd64 build). > > The catch is: look out for rsync's --delete flag! When some > port managers delete old/stale distfiles, they may also delete > distfiles for the *other* arches because they (rightly) think > they are not needed here... and when you then rsync with --delete, > that would (wrongly) propagate such deletes to those arches, > and you end up with missing distfiles on the targets. > > Since I have more than just two arches, I use a slightly different > 2-layer workflow: > > 0. I have 3 servers that are allowed to fetch files from the outside: > i386-master, amd64-master, sparc64-master. > and a whole bunch of i386-slave-NNN, amd64-slave-NNN and > sparc64-slave-NNN machines that would duplicate from their > relative masters via rsync. > > On all -master(s), I keep $DISTFILES outside of /usr/ports > (on /usr/local/distfiles, with a symbolic link in /usr/ports > /usr/ports/distfiles -> /usr/local/distfiles) > > Initial update of i386-master, as usual: > > 1. On i386-master, csup /usr/ports. > Run portmaster as usual to upgrade everything. > This may delete old stale distfiles and non-i386-distfiles. > This may fetch additional generic and i386-specific distfiles. > > Copy the new /usr/ports (without distfiles) to the other > arch masters: > > 2. rsync -av --delete i386-master:/usr/ports to amd64-master > and sparc64-master. CAUTION: Use --delete is okay, but only > because distfiles are not under /usr/ports, so as not to nuke > non-i386-specific distfiles of the other arches. > > Copy i386-master's NEW distfiles to the other arch masters: > > 3. rsync -av i386-master:/usr/local/distfiles to amd64-master > and sparc64-master. BEWARE: Don't use --delete here! > Do this to copy new generic distfiles (and i386) from > the i386-master build to amd64-master and sparc64-master. > > Update amd64-master and sparc64-master's ports as usual: > > 4. On amd64-master, run portmaster as usual to upgrade everything. > This may delete old stale distfiles and non-amd64-distfiles. > This may fetch additional (generic and) amd64-specific-distfiles. > > 5. On sparc64-master, run portmaster as usual to upgrade everything. > This may delete old stale distfiles and non-sparc64-distfiles. > This may fetch additional (generic and) sparc64-specific-distfiles. > > At this point, i386-master, amd64-master and sparc64-master are > fully updated, and their /usr/local/distfiles directories are up > to date w.r.t. their specific architectures. Now, copy everything > from the masters to the slaves: > > 6. On every i386-slave-NNN, rsync -av --delete: > /usr/ports, /usr/local (including /usr/local/distfiles), > /var/db/pkg, /var/db/ports > from i386-master. > > 7. On every amd64-slave-NNN, rsync -av --delete: > /usr/ports, /usr/local (including /usr/local/distfiles) > /var/db/pkg, /var/db/ports > from amd64-master. > I can see the need to sync /var/db/ports, but isn't /var/db/pkg specific to each machine? Same with /usr/local/distfiles (as far as the need to sync across machines) but /usr/local/ in general is also specific to each machine and is huge, storage-wise. I put git repositories there, like the kdenlive and ffmpeg sources I like to follow on graphics machines, while other machines have TeX and family installed. > > 8. On every sparc64-slave-NNN, rsync -av --delete: > /usr/ports, /usr/local (including /usr/local/distfiles) > /var/db/pkg, /var/db/ports > from sparc64-master. > > You may also need to update entries in /etc and /usr/local/etc > on the slaves. > > > If not rsync, what is the best way to keep multiple ports trees on > different > > hardware in sync, assuming everything runs FreeBSD 8.2? > > > > regards, > > Peter Kryszkiewicz > > -cpghost. > > -- > Cordula's Web. http://www.cordula.ws/ >