From owner-freebsd-ports-bugs@FreeBSD.ORG Fri Jan 2 12:01:38 2009 Return-Path: Delivered-To: freebsd-ports-bugs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9AF991065672 for ; Fri, 2 Jan 2009 12:01:38 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) Received: from 0.mx.codelabs.ru (0.mx.codelabs.ru [144.206.177.45]) by mx1.freebsd.org (Postfix) with ESMTP id 2685B8FC1F for ; Fri, 2 Jan 2009 12:01:38 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) DomainKey-Signature: a=rsa-sha1; q=dns; c=simple; s=one; d=codelabs.ru; h=Received:Date:From:To:Cc:Subject:Message-ID:Reply-To:References:MIME-Version:Content-Type:Content-Disposition:Content-Transfer-Encoding:In-Reply-To:Sender; b=Z1QuEJf1qosWMSah3swFxwwPUM/lQnmSDHBjkokXHtv8wlMJ8zeNoIfaVZUJDjJCnr2E91JbqxKYFsKuQKbfityrH4iK9VgO+y4lNqT2vFMJ6rW/cyT9MYmvoyoB9ObnMsYMz47dMRRquycwB3qD+hxXubW/e6T384NQJynnPiQ=; Received: from phoenix.codelabs.ru (ppp91-77-173-142.pppoe.mtu-net.ru [91.77.173.142]) by 0.mx.codelabs.ru with esmtpsa (TLSv1:AES256-SHA:256) id 1LIiie-000K7w-S3; Fri, 02 Jan 2009 15:01:37 +0300 Date: Fri, 2 Jan 2009 15:01:35 +0300 From: Eygene Ryabinkin To: pav@FreeBSD.org Message-ID: References: <200812311824.mBVIOUXb046088@freefall.freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable In-Reply-To: <200812311824.mBVIOUXb046088@freefall.freebsd.org> Sender: rea-fbsd@codelabs.ru Cc: freebsd-ports-bugs@FreeBSD.org Subject: Re: ports/129763: [patch] add HTTP locations of OpenBSD mirrors to bsd.sites.mk X-BeenThere: freebsd-ports-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: rea-fbsd@codelabs.ru List-Id: Ports bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Jan 2009 12:01:39 -0000 Pav, good day. Wed, Dec 31, 2008 at 06:24:30PM +0000, pav@FreeBSD.org wrote: > I think this is somewhat excessive - can we perhaps trim the list to, say= , 20 > best http and 20 best ftp sites? Fair enough. I had trimmed the list, the patch to the previous one goes below: --- bsd.sites.mk-trim-the-list-of-OpenBSD-master-sites.patch begins here --- =46rom 797c667f3bd824e1d9ffee59d8120286068bc9ed Mon Sep 17 00:00:00 2001 =46rom: Eygene Ryabinkin Date: Fri, 2 Jan 2009 14:49:26 +0300 Subject: [PATCH] bsd.sites.mk: trim the list of OpenBSD master sites and re= order it Following suggestion of Pav Lucistnik, I had left slightly more then 20 sites for each protocol and sticked first- and second-level sites to the beginning of the list: these sites should provide higher quality of service, so people who will not use master site randomization, will hit the primary sites first. The list was trimmed using the following strategy: I left only big download sites that are powered by universities, ISPs or entities like this -- chances are greater then such mirror providers will deliver higher service levels. For sites that appeared both at HTTP and FTP mirrors, I had left only one protocol flavour -- chances are high that if, for example, FTP service will be down, then HTTP will be down too. And I had tried to select more different geographical locations. Signed-off-by: Eygene Ryabinkin --- Mk/bsd.sites.mk | 134 ++++++++++-----------------------------------------= --- 1 files changed, 25 insertions(+), 109 deletions(-) diff --git a/Mk/bsd.sites.mk b/Mk/bsd.sites.mk index 7253603..a9cc42f 100644 --- a/Mk/bsd.sites.mk +++ b/Mk/bsd.sites.mk @@ -855,144 +855,60 @@ MASTER_SITE_NVIDIA+=3D \ ftp://download1.nvidia.com/%SUBDIR%/ .endif =20 +# First- and second-level sites come first. Others go later. +# +# Mirrors that are accessible both by HTTP and FTP should be specified +# only for one protocol -- this provides better redundancy: if +# FTP-accessible part of the mirror fails, HTTP-accessible one will +# likely to fail too. And vice-versa. .if !defined(IGNORE_MASTER_SITE_OPENBSD) MASTER_SITE_OPENBSD+=3D \ - http://ftp.heanet.ie/pub/OpenBSD/%SUBDIR%/ \ - http://ftp.belnet.be/packages/openbsd/%SUBDIR%/ \ - http://mirrors.nic.funet.fi/pub/OpenBSD/%SUBDIR%/ \ - http://mirror.public-internet.co.uk/ftp/OpenBSD/%SUBDIR%/ \ http://anga.funkfeuer.at/ftp/pub/OpenBSD/%SUBDIR%/ \ http://spargel.kd85.com/ftp/pub/OpenBSD/%SUBDIR%/ \ http://ftp.eu.openbsd.org/pub/OpenBSD/%SUBDIR%/ \ http://obsd.cec.mtu.edu/pub/OpenBSD/%SUBDIR%/ \ - http://openbsd.noc.jgm.gov.ar/pub/OpenBSD/%SUBDIR%/ \ - http://ftp.iinet.net.au/pub/OpenBSD/%SUBDIR%/ \ + ftp://ftp.openbsd.org/pub/OpenBSD/%SUBDIR%/ \ + ftp://openbsd.informatik.uni-erlangen.de/pub/OpenBSD/%SUBDIR%/ \ + ftp://ftp5.usa.openbsd.org/pub/OpenBSD/%SUBDIR%/ \ + ftp://ftp3.usa.openbsd.org/pub/OpenBSD/%SUBDIR%/ \ + ftp://rt.fm/pub/OpenBSD/%SUBDIR%/ \ + http://ftp.heanet.ie/pub/OpenBSD/%SUBDIR%/ \ + http://ftp.belnet.be/packages/openbsd/%SUBDIR%/ \ + http://mirrors.nic.funet.fi/pub/OpenBSD/%SUBDIR%/ \ + http://mirror.public-internet.co.uk/ftp/OpenBSD/%SUBDIR%/ \ http://mirror.pacific.net.au/OpenBSD/%SUBDIR%/ \ - http://mirror.internode.on.net/pub/OpenBSD/%SUBDIR%/ \ - http://mirror.aarnet.edu.au/pub/OpenBSD/%SUBDIR%/ \ - http://ftp.wu-wien.ac.at/pub/OpenBSD/%SUBDIR%/ \ - http://ftp.das.ufsc.br/pub/OpenBSD/%SUBDIR%/ \ - http://mirrors.uol.com.br/pub/OpenBSD/%SUBDIR%/ \ - http://ivete.fis.unb.br/mirror/OpenBSD/%SUBDIR%/ \ - http://gulus.usherbrooke.ca/pub/distro/OpenBSD/%SUBDIR%/ \ http://ftp.openbsd.dk/pub/OpenBSD/%SUBDIR%/ \ - http://ftp.dkuug.dk/pub/OpenBSD/%SUBDIR%/ \ - http://ftp.aso.ee/pub/OpenBSD/%SUBDIR%/ \ http://ftp.estpak.ee/pub/OpenBSD/%SUBDIR%/ \ - http://ftp.jyu.fi/ftp/pub/OpenBSD/%SUBDIR%/ \ - http://ftp.arcane-networks.fr/pub/OpenBSD/%SUBDIR%/ \ - http://ftp-stud.fht-esslingen.de/pub/OpenBSD/%SUBDIR%/ \ - http://mirror.roothell.org/pub/OpenBSD/%SUBDIR%/ \ - http://artfiles.org/openbsd/%SUBDIR%/ \ - http://ftp.duth.gr/pub/OpenBSD/%SUBDIR%/ \ - http://ftp.cc.uoc.gr/mirrors/OpenBSD/%SUBDIR%/ \ - http://ftp.esat.net/pub/OpenBSD/%SUBDIR%/ \ - http://ftp.jaist.ac.jp/pub/OpenBSD/%SUBDIR%/ \ - http://ftp.kaist.ac.kr/pub/OpenBSD/%SUBDIR%/ \ - http://mirrors.gamevil.com/pub/OpenBSD/%SUBDIR%/ \ - http://mirror.hostfuss.com/pub/OpenBSD/%SUBDIR%/ \ - http://jane.tihlde.org/pub/OpenBSD/%SUBDIR%/ \ - http://ftp.task.gda.pl/pub/OpenBSD/%SUBDIR%/ \ - http://piotrkosoft.net/pub/OpenBSD/%SUBDIR%/ \ - http://ftp.fmed.uc.pt/pub/OpenBSD/%SUBDIR%/ \ + http://mirror.internode.on.net/pub/OpenBSD/%SUBDIR%/ \ http://ftp.chg.ru/pub/OpenBSD/%SUBDIR%/ \ - http://www.obsd.si/pub/OpenBSD/%SUBDIR%/ \ - http://sunsite.rediris.es/pub/OpenBSD/%SUBDIR%/ \ - http://ftp.udc.es/OpenBSD/%SUBDIR%/ \ + http://ftp.arcane-networks.fr/pub/OpenBSD/%SUBDIR%/ \ http://ftp.netbsd.se/OpenBSD/%SUBDIR%/ \ - http://mirror.switch.ch/ftp/pub/OpenBSD/%SUBDIR%/ \ - http://ftp.tcc.edu.tw/pub/OpenBSD/%SUBDIR%/ \ http://openbsd.cs.pu.edu.tw/%SUBDIR%/ \ + http://ftp.wu-wien.ac.at/pub/OpenBSD/%SUBDIR%/ \ http://www.mirrorservice.org/pub/OpenBSD/%SUBDIR%/ \ - http://ftp.plig.net/pub/OpenBSD/%SUBDIR%/ \ - http://openbsd.berkeley.edu/pub/OpenBSD/%SUBDIR%/ \ - http://mirror.planetunix.net/pub/OpenBSD/%SUBDIR%/ \ - http://mirrors.24-7-solutions.net/pub/OpenBSD/%SUBDIR%/ \ - http://carroll.cac.psu.edu/pub/OpenBSD/%SUBDIR%/ \ - http://openbsd.mirrors.hoobly.com/%SUBDIR%/ \ - http://filedump.se.rit.edu/pub/OpenBSD/%SUBDIR%/ \ - http://mirror.rit.edu/pub/OpenBSD/%SUBDIR%/ \ - http://openbsd.mirror.frontiernet.net/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.openbsd.org/pub/OpenBSD/%SUBDIR%/ \ - ftp://anga.funkfeuer.at/pub/OpenBSD/%SUBDIR%/ \ - ftp://spargel.kd85.com/pub/OpenBSD/%SUBDIR%/ \ - ftp://openbsd.informatik.uni-erlangen.de/pub/OpenBSD/%SUBDIR%/ \ + http://mirror.switch.ch/ftp/pub/OpenBSD/%SUBDIR%/ \ + http://ftp.jaist.ac.jp/pub/OpenBSD/%SUBDIR%/ \ + http://ftp.kaist.ac.kr/pub/OpenBSD/%SUBDIR%/ \ + http://ftp.cc.uoc.gr/mirrors/OpenBSD/%SUBDIR%/ \ + http://mirror.hostfuss.com/pub/OpenBSD/%SUBDIR%/ \ + http://ivete.fis.unb.br/mirror/OpenBSD/%SUBDIR%/ \ ftp://ftp.stacken.kth.se/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp5.usa.openbsd.org/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp3.usa.openbsd.org/pub/OpenBSD/%SUBDIR%/ \ - ftp://rt.fm/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.openbsd.md5.com.ar/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.it.net.au/mirrors/OpenBSD/%SUBDIR%/ \ - ftp://ftp.iinet.net.au/pub/OpenBSD/%SUBDIR%/ \ - ftp://mirror.pacific.net.au/OpenBSD/%SUBDIR%/ \ - ftp://mirror.internode.on.net/pub/OpenBSD/%SUBDIR%/ \ - ftp://mirror.aarnet.edu.au/pub/OpenBSD/%SUBDIR%/ \ - ftp://playboy.wu-wien.ac.at/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.scarlet.be/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.belnet.be/packages/OpenBSD/%SUBDIR%/ \ - ftp://ftp.das.ufsc.br/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.bg.openbsd.org/pub/OpenBSD/%SUBDIR%/ \ - ftp://openbsd.arcticnetwork.ca/pub/OpenBSD/%SUBDIR%/ \ ftp://ftp.ca.openbsd.org/pub/OpenBSD/%SUBDIR%/ \ - ftp://gulus.usherbrooke.ca/pub/distro/OpenBSD/%SUBDIR%/ \ ftp://ftp.freebsdchina.org/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.openbsd.dk/pub/OpenBSD/%SUBDIR%/ \ ftp://ftp.dkuug.dk/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.aso.ee/pub/OpenBSD/%SUBDIR%/ \ - ftp://mirrors.nic.funet.fi/pub/OpenBSD/%SUBDIR%/ \ ftp://ftp.jyu.fi/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.arcane-networks.fr/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.crans.org/pub/OpenBSD/%SUBDIR%/ \ ftp://ftp.irisa.fr/pub/OpenBSD/%SUBDIR%/ \ ftp://openbsd.ftp.fu-berlin.de/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.spline.de/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.freenet.de/pub/ftp.openbsd.org/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp-stud.fht-esslingen.de/pub/OpenBSD/%SUBDIR%/ \ ftp://mirror.roothell.org/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.bytemine.net/pub/OpenBSD/%SUBDIR%/ \ - ftp://filoktitis.noc.uoa.gr/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.physics.auth.gr/pub/OpenBSD/%SUBDIR%/ \ ftp://ftp.duth.gr/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.cc.uoc.gr/mirrors/OpenBSD/%SUBDIR%/ \ - ftp://ftp.fsn.hu/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.esat.net/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.heanet.ie/pub/OpenBSD/%SUBDIR%/ \ - ftp://mirror.inter.net.il/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.unina.it/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.jaist.ac.jp/pub/OpenBSD/%SUBDIR%/ \ ftp://ftp.nara.wide.ad.jp/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.jp.openbsd.org/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.kddlabs.co.jp/OpenBSD/%SUBDIR%/ \ - ftp://ftp.kaist.ac.kr/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.secure.lv/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.bsd.lv/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.calyx.nl/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.nluug.nl/pub/OpenBSD/%SUBDIR%/ \ - ftp://mirror.hostfuss.com/pub/OpenBSD/%SUBDIR%/ \ ftp://ftp.inet.no/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.uninett.no/pub/OpenBSD/%SUBDIR%/ \ - ftp://jane.tihlde.org/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.task.gda.pl/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.fmed.uc.pt/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.chg.ru/pub/OpenBSD/%SUBDIR%/ \ + ftp://ftp.nluug.nl/pub/OpenBSD/%SUBDIR%/ \ ftp://ftp.gamma.ru/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.obsd.si/pub/OpenBSD/%SUBDIR%/ \ ftp://ftp.rediris.es/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.udc.es/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.df.lth.se/pub/OpenBSD/%SUBDIR%/ \ ftp://ftp.su.se/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.btradianz.se/pub/OpenBSD/%SUBDIR%/ \ - ftp://mirror.switch.ch/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.enderunix.org/pub/OpenBSD/%SUBDIR%/ \ - ftp://gaia.colocall.net/pub/OpenBSD/%SUBDIR%/ \ - ftp://mirror.public-internet.co.uk/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.mirrorservice.org/pub/OpenBSD/%SUBDIR%/ \ - ftp://mirror.planetunix.net/pub/OpenBSD/%SUBDIR%/ \ ftp://osmirrors.cerias.purdue.edu/pub/OpenBSD/%SUBDIR%/ \ ftp://mirror.iawnet.sandia.gov/pub/OpenBSD/%SUBDIR%/ \ - ftp://ftp.cse.buffalo.edu/pub/OpenBSD/%SUBDIR%/ \ - ftp://mirrors.24-7-solutions.net/pub/OpenBSD/%SUBDIR%/ \ - ftp://openbsd.mirrors.pair.com/%SUBDIR%/ \ ftp://carroll.cac.psu.edu/pub/OpenBSD/%SUBDIR%/ \ ftp://openbsd.mirrors.tds.net/pub/OpenBSD/%SUBDIR%/ .endif --=20 1.6.0.5 --- bsd.sites.mk-trim-the-list-of-OpenBSD-master-sites.patch ends here --- During the testing of this patch I had created small script that will permit one to check the master mirrors from bsd.sites.mk. The script goes below, may be it will be useful for someone else and even could be dropped to the 'tools/' directory. --- verify-master-sites.sh begins here --- #!/bin/sh # Simple script for verification that all mirrors have the given # file on the given subdirectory. # # Eygene Ryabinkin, rea-fbsd at codelabs dot ru, 2009. SITES_MK=3Dbsd.sites.mk usage () { cat >&2 << EOF Usage: $0 make-variable path-to-file I will use file '$SITES_MK' and 'make-variable' to get the list of the bsd.sites.mk-style master sites. Then I'll take the argument 'path-to-file' and will try to download it from each mirror in turn. If there will be any download errors, they will be printed to the standard output. EOF } if ! [ -f "$SITES_MK" ]; then echo "I must be called from the directory with file '$SITES_MK'" >&2 exit 2 fi if [ -z "$1" -o -z "$2" ]; then usage exit 2 fi MASTER_LIST=3D`make -f "$SITES_MK" -V "$1"` if [ -z "$MASTER_LIST" ]; then cat >&2 << EOF I wasn't able to extract master site list from the file '$SITES_MK' using the variable '$1'. Exiting. EOF exit 2 fi SUBDIR=3D`dirname "$2"` if [ "X$SUBDIR" =3D "X." ]; then SUBDIR=3D"" fi FILE=3D`basename "$2"` RETVAL=3D0 for i in $MASTER_LIST; do URL=3D`echo "$i" | sed -e's|%SUBDIR%|'"$SUBDIR"'|g'` fetch -o /dev/null -qp "$URL""$FILE" || RETVAL=3D1 done exit "$RETVAL" --- verify-master-sites.sh ends here --- --=20 Eygene _ ___ _.--. # \`.|\..----...-'` `-._.-'_.-'` # Remember that it is hard / ' ` , __.--' # to read the on-line manual )/' _/ \ `-_, / # while single-stepping the kernel. `-'" `"\_ ,_.-;_.-\_ ', fsc/as # _.-'_./ {_.' ; / # -- FreeBSD Developers handbook {_.-``-' {_/ #