From owner-freebsd-questions@FreeBSD.ORG Wed Oct 26 15:07:54 2011 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 97E7B1065781 for ; Wed, 26 Oct 2011 15:07:54 +0000 (UTC) (envelope-from cpghost@cordula.ws) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 5A4568FC14 for ; Wed, 26 Oct 2011 15:07:53 +0000 (UTC) Received: by vcbfo13 with SMTP id fo13so2329352vcb.13 for ; Wed, 26 Oct 2011 08:07:53 -0700 (PDT) MIME-Version: 1.0 Received: by 10.220.116.68 with SMTP id l4mr377994vcq.182.1319641673212; Wed, 26 Oct 2011 08:07:53 -0700 (PDT) Received: by 10.220.194.194 with HTTP; Wed, 26 Oct 2011 08:07:53 -0700 (PDT) X-Originating-IP: [93.221.182.5] In-Reply-To: References: Date: Wed, 26 Oct 2011 17:07:53 +0200 Message-ID: From: "C. P. Ghost" To: Peter Kryszkiewicz Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-questions@freebsd.org Subject: Re: rsync and the ports tree X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Oct 2011 15:07:54 -0000 On Wed, Oct 26, 2011 at 12:39 PM, Peter Kryszkiewicz wrote: > I have several machines installed in my temporary location and only my > laptop gets the internet through wireless. So far I've been building ports > on the other machines by rsync'ing the distfiles from the laptop as I need > them (all machines have the same FreeBSD 8.2 installed). > > The problem comes after I did a 'portupgrade -a' on the laptop. To ensure > the other ports trees are in sync, can I rsync the /usr/ports directory to > the other machines? Since some of them are different architectures (amd64 > multicore for instance) I ran into situations where the distfiles are > different (for gcc for example). First of all, rsync is working perfectly if you want to distribute /usr/ports/distfiles, /usr/ports to your internal machines, even when they are not of the same architecture. I'm doing this with a BIG farm of servers running i386, amd64, and sparc64 for a long, long time. You only need to make sure to rsync the *union* of your /usr/ports/distfiles directories, or else it won't work. Say, on amd64 you have /usr/ports/distfiles/some-distfile-for-amd64-only.tar.bz2 and on i386 you have /usr/ports/distfiles/some-distfile-for-i386-only.tar.bz2 Yes, that happens every now and then. So you have to rsync both ways, so that you end up with /usr/ports/distfiles/some-distfile-for-amd64-only.tar.bz2 /usr/ports/distfiles/some-distfile-for-i386-only.tar.bz2 on both i386 and amd64 machines. The catch is: look out for rsync's --delete flag! When some port managers delete old/stale distfiles, they may also delete distfiles for the *other* arches because they (rightly) think they are not needed here... and when you then rsync with --delete, that would (wrongly) propagate such deletes to those arches, and you end up with missing distfiles on the targets. Since I have more than just two arches, I use a slightly different 2-layer workflow: 0. I have 3 servers that are allowed to fetch files from the outside: i386-master, amd64-master, sparc64-master. and a whole bunch of i386-slave-NNN, amd64-slave-NNN and sparc64-slave-NNN machines that would duplicate from their relative masters via rsync. On all -master(s), I keep $DISTFILES outside of /usr/ports (on /usr/local/distfiles, with a symbolic link in /usr/ports /usr/ports/distfiles -> /usr/local/distfiles) Initial update of i386-master, as usual: 1. On i386-master, csup /usr/ports. Run portmaster as usual to upgrade everything. This may delete old stale distfiles and non-i386-distfiles. This may fetch additional generic and i386-specific distfiles. Copy the new /usr/ports (without distfiles) to the other arch masters: 2. rsync -av --delete i386-master:/usr/ports to amd64-master and sparc64-master. CAUTION: Use --delete is okay, but only because distfiles are not under /usr/ports, so as not to nuke non-i386-specific distfiles of the other arches. Copy i386-master's NEW distfiles to the other arch masters: 3. rsync -av i386-master:/usr/local/distfiles to amd64-master and sparc64-master. BEWARE: Don't use --delete here! Do this to copy new generic distfiles (and i386) from the i386-master build to amd64-master and sparc64-master. Update amd64-master and sparc64-master's ports as usual: 4. On amd64-master, run portmaster as usual to upgrade everything. This may delete old stale distfiles and non-amd64-distfiles. This may fetch additional (generic and) amd64-specific-distfiles. 5. On sparc64-master, run portmaster as usual to upgrade everything. This may delete old stale distfiles and non-sparc64-distfiles. This may fetch additional (generic and) sparc64-specific-distfiles. At this point, i386-master, amd64-master and sparc64-master are fully updated, and their /usr/local/distfiles directories are up to date w.r.t. their specific architectures. Now, copy everything from the masters to the slaves: 6. On every i386-slave-NNN, rsync -av --delete: /usr/ports, /usr/local (including /usr/local/distfiles), /var/db/pkg, /var/db/ports from i386-master. 7. On every amd64-slave-NNN, rsync -av --delete: /usr/ports, /usr/local (including /usr/local/distfiles) /var/db/pkg, /var/db/ports from amd64-master. 8. On every sparc64-slave-NNN, rsync -av --delete: /usr/ports, /usr/local (including /usr/local/distfiles) /var/db/pkg, /var/db/ports from sparc64-master. You may also need to update entries in /etc and /usr/local/etc on the slaves. > If not rsync, what is the best way to keep multiple ports trees on different > hardware in sync, assuming everything runs FreeBSD 8.2? > > regards, > Peter Kryszkiewicz -cpghost. -- Cordula's Web. http://www.cordula.ws/