From owner-freebsd-ports@FreeBSD.ORG Sun Oct 19 08:53:35 2003 Return-Path: Delivered-To: freebsd-ports@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2C92116A4B3 for ; Sun, 19 Oct 2003 08:53:35 -0700 (PDT) Received: from kraid.nerim.net (smtp-100-sunday.nerim.net [62.4.16.100]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1236643F93 for ; Sun, 19 Oct 2003 08:53:34 -0700 (PDT) (envelope-from espie@smtp.nerim.net) Received: from tetto.gentiane.org (espie.gentiane.org [62.212.102.210]) by kraid.nerim.net (Postfix) with ESMTP id 2CC7340E6F; Sun, 19 Oct 2003 17:53:31 +0200 (CEST) Received: from tetto.gentiane.org (espie@localhost [127.0.0.1]) by tetto.gentiane.org (8.12.9/8.12.1) with ESMTP id h9JFrT8m031482; Sun, 19 Oct 2003 17:53:30 +0200 (CEST) Received: (from espie@localhost) by tetto.gentiane.org (8.12.9/8.12.1/Submit) id h9JFrRpl013823; Sun, 19 Oct 2003 17:53:27 +0200 (CEST) Date: Sun, 19 Oct 2003 17:53:24 +0200 From: Marc Espie To: tech-pkg@netbsd.org, ports@openbsd.org, freebsd-ports@freebsd.org Message-ID: <20031019155324.GA28256@tetto.gentiane.org> Mail-Followup-To: tech-pkg@netbsd.org, ports@openbsd.org, freebsd-ports@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.1i Subject: Unification of distfiles X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: espie@nerim.net List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Oct 2003 15:53:35 -0000 I think trying to unify distfiles somehow is a great idea. If I remember right, both Free and Net have mechanisms similar to DIST_SUBDIR, so it just means that we ought to try to have similar naming there. I'm all in favor for greater communication in that area, and possibly people advertizing distfiles mirrors that would hold stuff for all three BSDs. Now, for stuff that exists over in OpenBSD land, that might be of interest to other people. One other issue which has been fairly annoying as far as OpenBSD was concerned, is distfiles changing checksums without changing names. We've had a fairly thorough mechanism for dealing with this in place for a long while, look at the code handling REFETCH in our bsd.port.mk, plus the /usr/ports/infrastructure/fetch/link-checksums script. Basically, every mirror should run this once in a while. This establishes alternate access methods for files, based on md5/sha1/rmd160 checksums. The price to pay is that the mirror ought to be located on a system with unix filesystem semantics (and preferably a single partition) to avoid duplication of files. We also have a `mirror-maker' target that knows how to create a master Makefile in /usr/ports/distfiles, that is 10000+ lines long, and can be used to do automated operations on all distfiles (fetching them, checking checksums, verifying mirrors). The fact that it is a Makefile makes it very easy to run operations in parallel and diminish network bandwidth bottlenecks for grabbing a file. I've been very happy with this redesign of our infrastructure (note that the Makefile is portable enough that Bob Beck actually runs it under Solaris on a sunsite for a distfiles mirror), and I believe it could be useful to other people.... Feel free to ignore this, if you feel offended by the idea that OpenBSD may have done something right.