Date: Wed, 22 Oct 2003 01:52:56 +0200 From: Clement Laforet <sheepkiller@cultdeadsheep.org> To: ports@FreeBSD.org Subject: [RFC]: add a basic "closest mirrors finder" to make fetch Message-ID: <20031022015256.57519ab5.sheepkiller@cultdeadsheep.org>
next in thread | raw e-mail | index | archive | help
Hi !
It has already been discussed many times and I've never seen a patch.
The script is based on ping response (which is not a good idea ;)) and
sorts "make 'CKSUMFILES+=${DISTFILES}' master-sites-${group}" output
(PATCH_SITES is supported too).
There is some dirty thing (like the way of getting rid of ftp.FreeBSD.org
mirror), so it surely needs more investigations.
I really hope getting some feedback :-)
Usage:
cd <to your favorite ports>
make ENABLE_CLOSEST_MIRROR=yes fetch
you can use FETCH_DEBUG=yes too, the sorted list will be printed.
clem
============================================
Sample output:
[root@lucifer|(884)| apache13-ssl]# make ENABLE_CLOSEST_MIRROR=yes fetch
>> apache_1.3.28.tar.gz doesn't seem to exist in /usr/ports/distfiles/apache.
Looking for closest mirror, please wait...
>> Attempting to fetch from ftp://mir1.ovh.net/ftp.apache.org/dist/httpd/.
Receiving apache_1.3.28.tar.gz (2388111 bytes): 100% (ETA 00:00)
2388111 bytes transferred in 58.7 seconds (39.75 kBps)
>> apache_1.3.28+ssl_1.49.tar.gz doesn't seem to exist in /usr/ports/distfiles/apache.
Looking for closest mirror, please wait...
>> Attempting to fetch from ftp://opensores.thebunker.net/pub/mirrors/apache-ssl/.
Receiving apache_1.3.28+ssl_1.49.tar.gz (56266 bytes): 100%
56266 bytes transferred in 1.2 seconds (46.41 kBps)
>> no_zombies.patch doesn't seem to exist in /usr/ports/distfiles/apache.
Looking for closest mirror, please wait...
>> Attempting to fetch from ftp://apache.mirrors.rossfell.co.uk/mirrors/apache/dist/httpd/patches/apply_to_1.3.28/.
Receiving no_zombies.patch (889 bytes): 100%
889 bytes transferred in 0.0 seconds (38.37 kBps)
-----------------
With FETCH_DEBUG=yes:
[root@lucifer|(887)| apache13-ssl]# make ENABLE_CLOSEST_MIRROR=yes FETCH_DEBUG=yes fetch
>> apache_1.3.28.tar.gz doesn't seem to exist in /usr/ports/distfiles/apache.
Looking for closest mirror, please wait...
*** DEBUG *** SITES:
ftp://mir1.ovh.net/ftp.apache.org/dist/httpd/ ftp://ftp.apache.de/mirrors/dev.apache.org/dist/httpd/ ftp://ftp.plig.net/pub/apache/dist/httpd/ ftp://ftp-stud.fht-esslingen.de/pub/Mirrors/ftp.apache.org/dist/httpd/ ftp://ftp.cs.tu-berlin.de/pub/net/www/apache/ ftp://gd.tuwien.ac.at/pub/infosys/servers/http/apache/dist/httpd/ ftp://ftp.gin.cz/pub/mirrors/www.apache.org/dist/httpd/ ftp://ftp.kappa.ro/pub/mirrors/ftp.apache.org/dist/httpd/ ftp://ftp.chg.ru/pub/WWW/apache/dist/httpd/ ftp://ftp.task.gda.pl/pub/www/apache/dist/httpd/ ftp://sunsite.srce.hr/mirrors/ftp.apache.org/apache/dist/httpd/ ftp://sunsite.icm.edu.pl/pub/www/apache/dist/httpd/ ftp://apache.rinet.ru/pub/mirror/apache.org/dist/httpd/ ftp://ftp.tux.org/pub/net/apache/dist/httpd/ http://nagoya.apache.org/mirror/httpd/ http://www.apache.org/dist/httpd/ ftp://ftp.pop-mg.com.br/pub/apache/dist/httpd/ ftp://ftp.flirble.org/pub/web/apache/dist/httpd/ ftp://ftp.planetmirror.com/pub/apache/dist/httpd/ ftp://ftp.saix.net/pu
b/apache/dist/httpd/ ftp://ftp.epix.net/pub/apache/httpd/ ftp://xenia.sote.hu/pub/mirrors/www.apache.org/httpd/ ftp://ftp.ccs.neu.edu/net/mirrors/Apache/dist/httpd/ ftp://ftp.apache.inetcosmos.org/pub/apache/dist/httpd/ ftp://apache.theomnistore.com/mirror/apache/dist/httpd/ ftp://ftp.esat.net/mirrors/ftp.apache.org/httpd/ ftp://ftp.sogang.ac.kr/www/apache/dist/httpd/ ftp://ftp.heanet.ie/mirrors/www.apache.org/dist/httpd/ ftp://ftp.rhnet.is/pub/apache/httpd/ ftp://apache.secsup.org/pub/apache/dist/httpd/ ftp://apache.mirrors.rossfell.co.uk/mirrors/apache/dist/httpd/ ftp://ftp.sunet.se/pub/www/servers/apache/dist/httpd/ http://www.t.ring.gr.jp/archives/net/apache/dist/httpd/ ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/apache/
>> Attempting to fetch from ftp://mir1.ovh.net/ftp.apache.org/dist/httpd/.
Receiving apache_1.3.28.tar.gz (2388111 bytes): 100% (ETA 00:00)
2388111 bytes transferred in 40.3 seconds (57.91 kBps)
>> apache_1.3.28+ssl_1.49.tar.gz doesn't seem to exist in /usr/ports/distfiles/apache.
Looking for closest mirror, please wait...
*** DEBUG *** SITES:
ftp://opensores.thebunker.net/pub/mirrors/apache-ssl/ ftp://ftp.it.net.au/mirrors/crypto/SSL/Apache-SSL/ ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/apache/
>> Attempting to fetch from ftp://opensores.thebunker.net/pub/mirrors/apache-ssl/.
Receiving apache_1.3.28+ssl_1.49.tar.gz (56266 bytes): 100%
56266 bytes transferred in 0.6 seconds (96.49 kBps)
>> no_zombies.patch doesn't seem to exist in /usr/ports/distfiles/apache.
Looking for closest mirror, please wait...
*** DEBUG *** SITES:
ftp://ftp.apache.de/mirrors/dev.apache.org/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp-stud.fht-esslingen.de/pub/Mirrors/ftp.apache.org/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.heanet.ie/mirrors/www.apache.org/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.apache.inetcosmos.org/pub/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://sunsite.srce.hr/mirrors/ftp.apache.org/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.esat.net/mirrors/ftp.apache.org/httpd/patches/apply_to_1.3.28/ ftp://ftp.gin.cz/pub/mirrors/www.apache.org/dist/httpd/patches/apply_to_1.3.28/ ftp://xenia.sote.hu/pub/mirrors/www.apache.org/httpd/patches/apply_to_1.3.28/ ftp://ftp.kappa.ro/pub/mirrors/ftp.apache.org/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.flirble.org/pub/web/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://apache.theomnistore.com/mirror/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.rhnet.is/pub/apache/httpd/patches/apply_to_1.3.28/ ftp://ftp.task.gda.pl/pub/www/apache/dist/h
ttpd/patches/apply_to_1.3.28/ ftp://ftp.tux.org/pub/net/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://apache.rinet.ru/pub/mirror/apache.org/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.epix.net/pub/apache/httpd/patches/apply_to_1.3.28/ ftp://sunsite.icm.edu.pl/pub/www/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.ccs.neu.edu/net/mirrors/Apache/dist/httpd/patches/apply_to_1.3.28/ http://nagoya.apache.org/mirror/httpd/patches/apply_to_1.3.28/ ftp://ftp.pop-mg.com.br/pub/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://apache.mirrors.rossfell.co.uk/mirrors/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.sogang.ac.kr/www/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.planetmirror.com/pub/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.saix.net/pub/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://apache.secsup.org/pub/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.chg.ru/pub/WWW/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.cs.tu-berlin.de/
pub/net/www/apache/patches/apply_to_1.3.28/ ftp://ftp.plig.net/pub/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://gd.tuwien.ac.at/pub/infosys/servers/http/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://mir1.ovh.net/ftp.apache.org/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.forthnet.gr/pub/www/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.sunet.se/pub/www/servers/apache/dist/httpd/patches/apply_to_1.3.28/ http://www.apache.org/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/apache/
>> Attempting to fetch from ftp://ftp.apache.de/mirrors/dev.apache.org/dist/httpd/patches/apply_to_1.3.28/.
Receiving no_zombies.patch (889 bytes): 100%
889 bytes transferred in 0.0 seconds (28.35 kBps)
============================================
Patch:
Index: Mk/bsd.port.mk
===================================================================
RCS file: /WORK/REPO/ports/Mk/bsd.port.mk,v
retrieving revision 1.468
diff -u -r1.468 bsd.port.mk
--- Mk/bsd.port.mk 3 Sep 2003 19:50:24 -0000 1.468
+++ Mk/bsd.port.mk 21 Oct 2003 23:31:11 -0000
@@ -839,11 +839,13 @@
LS?= /bin/ls
MKDIR?= /bin/mkdir -p
MV?= /bin/mv
+PING?= /sbin/ping
REALPATH?= /bin/realpath
RM?= /bin/rm
RMDIR?= /bin/rmdir
SED?= /usr/bin/sed
SETENV?= /usr/bin/env
+SORT?= /usr/bin/sort
SH?= /bin/sh
STRIP_CMD?= /usr/bin/strip
SU_CMD?= /usr/bin/su root -c
@@ -2748,11 +2750,26 @@
fi \
done; \
___MASTER_SITES_TMP= ; \
- SORTED_MASTER_SITES_CMD_TMP="echo ${_MASTER_SITE_OVERRIDE} `echo $${__MASTER_SITES_TMP} | ${AWK} '${MASTER_SORT_AWK:S|\\|\\\\|g}'` ${_MASTER_SITE_BACKUP}" ; \
+ SORTED_MASTER_SITES_CMD_TMP="echo ${_MASTER_SITE_OVERRIDE} `echo $${__MASTER_SITES_TMP} | ${AWK} '${MASTER_SORT_AWK:S|\\|\\\\|g}'`" ; \
+ WITH_BACKUP_SITE=yes;\
else \
SORTED_MASTER_SITES_CMD_TMP="${SORTED_MASTER_SITES_DEFAULT_CMD}" ; \
fi ; \
- for site in `eval $$SORTED_MASTER_SITES_CMD_TMP`; do \
+ if [ ! -z ${ENABLE_CLOSEST_MIRROR} ] ;\
+ then \
+ ${ECHO_CMD} "Looking for closest mirror, please wait...";\
+ SITES=$$(for site in `eval $$SORTED_MASTER_SITES_CMD_TMP`;\
+ do host=`${ECHO_CMD} $$site | ${AWK} -F\/ '{print $$3;}'`;\
+ time=`${PING} -Q -t 5 -c 1 $$host | ${AWK} -F\/ '$$1 ~ /^round/ {print $$6;}' 2> /dev/null` ;\
+ ${ECHO} $$time $$site;\
+ done | ${SORT} -n | ${AWK} '$$2 ~ /^(ftp|http)/ {print $$2;}');\
+ SITES="$$SITES ${_MASTER_SITE_BACKUP}"; \
+ else \
+ SITES=`eval $$SORTED_MASTER_SITES_CMD_TMP`; \
+ [ "x$$WITH_BACKUP_SITE" = "x" ] || SITES="$$SITES ${_MASTER_SITE_BACKUP}"; \
+ fi;\
+ [ ! -z ${FETCH_DEBUG} ] && ( ${ECHO} '*** DEBUG *** SITES:' ; ${ECHO} $$SITES) ;\
+ for site in $$SITES; do \
${ECHO_MSG} ">> Attempting to fetch from $${site}."; \
DIR=${DIST_SUBDIR}; \
CKSIZE=`${GREP} "^SIZE ($${DIR:+$$DIR/}$$file)" ${MD5_FILE} | ${AWK} '{print $$4}'`; \
@@ -2801,11 +2818,26 @@
fi \
done; \
___PATCH_SITES_TMP= ; \
- SORTED_PATCH_SITES_CMD_TMP="echo ${_MASTER_SITE_OVERRIDE} `echo $${__PATCH_SITES_TMP} | ${AWK} '${MASTER_SORT_AWK:S|\\|\\\\|g}'` ${_MASTER_SITE_BACKUP}" ; \
+ SORTED_PATCH_SITES_CMD_TMP="echo ${_MASTER_SITE_OVERRIDE} `echo $${__PATCH_SITES_TMP} | ${AWK} '${MASTER_SORT_AWK:S|\\|\\\\|g}'`" ; \
+ WITH_BACKUP_SITE=yes;\
else \
SORTED_PATCH_SITES_CMD_TMP="${SORTED_PATCH_SITES_DEFAULT_CMD}" ; \
fi ; \
- for site in `eval $$SORTED_PATCH_SITES_CMD_TMP`; do \
+ if [ ! -z ${ENABLE_CLOSEST_MIRROR} ] ;\
+ then \
+ ${ECHO_CMD} "Looking for closest mirror, please wait...";\
+ SITES=$$(for site in `eval $$SORTED_PATCH_SITES_CMD_TMP`;\
+ do host=`${ECHO} $$site | ${AWK} -F\/ '{print $$3;}'`;\
+ time=`${PING} -Q -t 10 -c 2 $$host | ${AWK} -F\/ '$$1 ~ /^round-trip/ {print $$6;}' 2> /dev/null` ;\
+ ${ECHO} $$time $$site ; \
+ done | ${GREP} -v ${_MASTER_SITE_BACKUP}| ${SORT} -n | ${AWK} '$$2 ~ /^(ftp|http)/ {print $$2;}');\
+ SITES="$$SITES ${_MASTER_SITE_BACKUP}"; \
+ else \
+ SITES=`eval $$SORTED_PATCH_SITES_CMD_TMP | ${GREP} -v ${_MASTER_SITE_BACKUP}`; \
+ [ "x$$WITH_BACKUP_SITE" = "x" ] || SITES="$$SITES ${_MASTER_SITE_BACKUP}"; \
+ fi;\
+ [ ! -z ${FETCH_DEBUG} ] && ( ${ECHO} '*** DEBUG *** SITES:' ; ${ECHO} $$SITES) ;\
+ for site in $$SITES ; do \
${ECHO_MSG} ">> Attempting to fetch from $${site}."; \
DIR=${DIST_SUBDIR}; \
CKSIZE=`${GREP} "^SIZE ($${DIR:+$$DIR/}$$file)" ${MD5_FILE} | ${AWK} '{print $$4}'`; \
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20031022015256.57519ab5.sheepkiller>