Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 22 Oct 2003 01:52:56 +0200
From:      Clement Laforet <sheepkiller@cultdeadsheep.org>
To:        ports@FreeBSD.org
Subject:   [RFC]: add a basic "closest mirrors finder" to make fetch
Message-ID:  <20031022015256.57519ab5.sheepkiller@cultdeadsheep.org>

next in thread | raw e-mail | index | archive | help
Hi !

It has already been discussed many times and I've never seen a patch.
The script is based on ping response (which is not a good idea ;)) and
sorts "make 'CKSUMFILES+=${DISTFILES}' master-sites-${group}" output 
(PATCH_SITES is supported too).
There is some dirty thing (like the way of getting rid of ftp.FreeBSD.org 
mirror), so it surely needs more investigations.
I really hope getting some feedback :-)

Usage:
cd <to your favorite ports>
make ENABLE_CLOSEST_MIRROR=yes fetch
you can use FETCH_DEBUG=yes too, the sorted list will be printed.

clem 

============================================
Sample output:

[root@lucifer|(884)| apache13-ssl]# make ENABLE_CLOSEST_MIRROR=yes  fetch
>> apache_1.3.28.tar.gz doesn't seem to exist in /usr/ports/distfiles/apache.
Looking for closest mirror, please wait...
>> Attempting to fetch from ftp://mir1.ovh.net/ftp.apache.org/dist/httpd/.
Receiving apache_1.3.28.tar.gz (2388111 bytes): 100% (ETA 00:00)  
2388111 bytes transferred in 58.7 seconds (39.75 kBps)
>> apache_1.3.28+ssl_1.49.tar.gz doesn't seem to exist in /usr/ports/distfiles/apache.
Looking for closest mirror, please wait...
>> Attempting to fetch from ftp://opensores.thebunker.net/pub/mirrors/apache-ssl/.
Receiving apache_1.3.28+ssl_1.49.tar.gz (56266 bytes): 100%
56266 bytes transferred in 1.2 seconds (46.41 kBps)
>> no_zombies.patch doesn't seem to exist in /usr/ports/distfiles/apache.
Looking for closest mirror, please wait...
>> Attempting to fetch from ftp://apache.mirrors.rossfell.co.uk/mirrors/apache/dist/httpd/patches/apply_to_1.3.28/.
Receiving no_zombies.patch (889 bytes): 100%
889 bytes transferred in 0.0 seconds (38.37 kBps)

-----------------

With FETCH_DEBUG=yes:

[root@lucifer|(887)| apache13-ssl]# make ENABLE_CLOSEST_MIRROR=yes FETCH_DEBUG=yes fetch
>> apache_1.3.28.tar.gz doesn't seem to exist in /usr/ports/distfiles/apache.
Looking for closest mirror, please wait...
*** DEBUG *** SITES:
ftp://mir1.ovh.net/ftp.apache.org/dist/httpd/ ftp://ftp.apache.de/mirrors/dev.apache.org/dist/httpd/ ftp://ftp.plig.net/pub/apache/dist/httpd/ ftp://ftp-stud.fht-esslingen.de/pub/Mirrors/ftp.apache.org/dist/httpd/ ftp://ftp.cs.tu-berlin.de/pub/net/www/apache/ ftp://gd.tuwien.ac.at/pub/infosys/servers/http/apache/dist/httpd/ ftp://ftp.gin.cz/pub/mirrors/www.apache.org/dist/httpd/ ftp://ftp.kappa.ro/pub/mirrors/ftp.apache.org/dist/httpd/ ftp://ftp.chg.ru/pub/WWW/apache/dist/httpd/ ftp://ftp.task.gda.pl/pub/www/apache/dist/httpd/ ftp://sunsite.srce.hr/mirrors/ftp.apache.org/apache/dist/httpd/ ftp://sunsite.icm.edu.pl/pub/www/apache/dist/httpd/ ftp://apache.rinet.ru/pub/mirror/apache.org/dist/httpd/ ftp://ftp.tux.org/pub/net/apache/dist/httpd/ http://nagoya.apache.org/mirror/httpd/ http://www.apache.org/dist/httpd/ ftp://ftp.pop-mg.com.br/pub/apache/dist/httpd/ ftp://ftp.flirble.org/pub/web/apache/dist/httpd/ ftp://ftp.planetmirror.com/pub/apache/dist/httpd/ ftp://ftp.saix.net/pu
 b/apache/dist/httpd/ ftp://ftp.epix.net/pub/apache/httpd/ ftp://xenia.sote.hu/pub/mirrors/www.apache.org/httpd/ ftp://ftp.ccs.neu.edu/net/mirrors/Apache/dist/httpd/ ftp://ftp.apache.inetcosmos.org/pub/apache/dist/httpd/ ftp://apache.theomnistore.com/mirror/apache/dist/httpd/ ftp://ftp.esat.net/mirrors/ftp.apache.org/httpd/ ftp://ftp.sogang.ac.kr/www/apache/dist/httpd/ ftp://ftp.heanet.ie/mirrors/www.apache.org/dist/httpd/ ftp://ftp.rhnet.is/pub/apache/httpd/ ftp://apache.secsup.org/pub/apache/dist/httpd/ ftp://apache.mirrors.rossfell.co.uk/mirrors/apache/dist/httpd/ ftp://ftp.sunet.se/pub/www/servers/apache/dist/httpd/ http://www.t.ring.gr.jp/archives/net/apache/dist/httpd/ ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/apache/
>> Attempting to fetch from ftp://mir1.ovh.net/ftp.apache.org/dist/httpd/.
Receiving apache_1.3.28.tar.gz (2388111 bytes): 100% (ETA 00:00)  
2388111 bytes transferred in 40.3 seconds (57.91 kBps)
>> apache_1.3.28+ssl_1.49.tar.gz doesn't seem to exist in /usr/ports/distfiles/apache.
Looking for closest mirror, please wait...
*** DEBUG *** SITES:
ftp://opensores.thebunker.net/pub/mirrors/apache-ssl/ ftp://ftp.it.net.au/mirrors/crypto/SSL/Apache-SSL/ ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/apache/
>> Attempting to fetch from ftp://opensores.thebunker.net/pub/mirrors/apache-ssl/.
Receiving apache_1.3.28+ssl_1.49.tar.gz (56266 bytes): 100%
56266 bytes transferred in 0.6 seconds (96.49 kBps)
>> no_zombies.patch doesn't seem to exist in /usr/ports/distfiles/apache.
Looking for closest mirror, please wait...
*** DEBUG *** SITES:
ftp://ftp.apache.de/mirrors/dev.apache.org/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp-stud.fht-esslingen.de/pub/Mirrors/ftp.apache.org/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.heanet.ie/mirrors/www.apache.org/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.apache.inetcosmos.org/pub/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://sunsite.srce.hr/mirrors/ftp.apache.org/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.esat.net/mirrors/ftp.apache.org/httpd/patches/apply_to_1.3.28/ ftp://ftp.gin.cz/pub/mirrors/www.apache.org/dist/httpd/patches/apply_to_1.3.28/ ftp://xenia.sote.hu/pub/mirrors/www.apache.org/httpd/patches/apply_to_1.3.28/ ftp://ftp.kappa.ro/pub/mirrors/ftp.apache.org/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.flirble.org/pub/web/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://apache.theomnistore.com/mirror/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.rhnet.is/pub/apache/httpd/patches/apply_to_1.3.28/ ftp://ftp.task.gda.pl/pub/www/apache/dist/h
 ttpd/patches/apply_to_1.3.28/ ftp://ftp.tux.org/pub/net/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://apache.rinet.ru/pub/mirror/apache.org/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.epix.net/pub/apache/httpd/patches/apply_to_1.3.28/ ftp://sunsite.icm.edu.pl/pub/www/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.ccs.neu.edu/net/mirrors/Apache/dist/httpd/patches/apply_to_1.3.28/ http://nagoya.apache.org/mirror/httpd/patches/apply_to_1.3.28/ ftp://ftp.pop-mg.com.br/pub/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://apache.mirrors.rossfell.co.uk/mirrors/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.sogang.ac.kr/www/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.planetmirror.com/pub/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.saix.net/pub/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://apache.secsup.org/pub/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.chg.ru/pub/WWW/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.cs.tu-berlin.de/
 pub/net/www/apache/patches/apply_to_1.3.28/ ftp://ftp.plig.net/pub/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://gd.tuwien.ac.at/pub/infosys/servers/http/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://mir1.ovh.net/ftp.apache.org/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.forthnet.gr/pub/www/apache/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.sunet.se/pub/www/servers/apache/dist/httpd/patches/apply_to_1.3.28/ http://www.apache.org/dist/httpd/patches/apply_to_1.3.28/ ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/apache/
>> Attempting to fetch from ftp://ftp.apache.de/mirrors/dev.apache.org/dist/httpd/patches/apply_to_1.3.28/.
Receiving no_zombies.patch (889 bytes): 100%
889 bytes transferred in 0.0 seconds (28.35 kBps)


============================================
Patch:


Index: Mk/bsd.port.mk
===================================================================
RCS file: /WORK/REPO/ports/Mk/bsd.port.mk,v
retrieving revision 1.468
diff -u -r1.468 bsd.port.mk
--- Mk/bsd.port.mk	3 Sep 2003 19:50:24 -0000	1.468
+++ Mk/bsd.port.mk	21 Oct 2003 23:31:11 -0000
@@ -839,11 +839,13 @@
 LS?=		/bin/ls
 MKDIR?=		/bin/mkdir -p
 MV?=		/bin/mv
+PING?=		/sbin/ping
 REALPATH?=	/bin/realpath
 RM?=		/bin/rm
 RMDIR?=		/bin/rmdir
 SED?=		/usr/bin/sed
 SETENV?=	/usr/bin/env
+SORT?=		/usr/bin/sort
 SH?=		/bin/sh
 STRIP_CMD?=	/usr/bin/strip
 SU_CMD?=	/usr/bin/su root -c
@@ -2748,11 +2750,26 @@
 					fi \
 				done; \
 				___MASTER_SITES_TMP= ; \
-				SORTED_MASTER_SITES_CMD_TMP="echo ${_MASTER_SITE_OVERRIDE} `echo $${__MASTER_SITES_TMP} | ${AWK} '${MASTER_SORT_AWK:S|\\|\\\\|g}'` ${_MASTER_SITE_BACKUP}" ; \
+				SORTED_MASTER_SITES_CMD_TMP="echo ${_MASTER_SITE_OVERRIDE} `echo $${__MASTER_SITES_TMP} | ${AWK} '${MASTER_SORT_AWK:S|\\|\\\\|g}'`" ; \
+				WITH_BACKUP_SITE=yes;\
 			else \
 				SORTED_MASTER_SITES_CMD_TMP="${SORTED_MASTER_SITES_DEFAULT_CMD}" ; \
 			fi ; \
-			for site in `eval $$SORTED_MASTER_SITES_CMD_TMP`; do \
+			if [ ! -z ${ENABLE_CLOSEST_MIRROR} ] ;\
+			then \
+				${ECHO_CMD} "Looking for closest mirror, please wait...";\
+				SITES=$$(for site in `eval $$SORTED_MASTER_SITES_CMD_TMP`;\
+				do host=`${ECHO_CMD} $$site | ${AWK} -F\/ '{print $$3;}'`;\
+				time=`${PING} -Q -t 5 -c 1 $$host | ${AWK} -F\/ '$$1 ~ /^round/ {print $$6;}' 2> /dev/null` ;\
+				${ECHO} $$time $$site;\
+				done | ${SORT} -n | ${AWK} '$$2 ~ /^(ftp|http)/ {print $$2;}');\
+				SITES="$$SITES ${_MASTER_SITE_BACKUP}"; \
+			else \
+				SITES=`eval $$SORTED_MASTER_SITES_CMD_TMP`; \
+				[ "x$$WITH_BACKUP_SITE" = "x" ] ||  SITES="$$SITES ${_MASTER_SITE_BACKUP}"; \
+			fi;\
+			[ ! -z ${FETCH_DEBUG} ] && ( ${ECHO} '*** DEBUG *** SITES:' ; ${ECHO} $$SITES) ;\
+			for site in $$SITES; do \
 			    ${ECHO_MSG} ">> Attempting to fetch from $${site}."; \
 				DIR=${DIST_SUBDIR}; \
 				CKSIZE=`${GREP} "^SIZE ($${DIR:+$$DIR/}$$file)" ${MD5_FILE} | ${AWK} '{print $$4}'`; \
@@ -2801,11 +2818,26 @@
 					fi \
 				done; \
 				___PATCH_SITES_TMP= ; \
-				SORTED_PATCH_SITES_CMD_TMP="echo ${_MASTER_SITE_OVERRIDE} `echo $${__PATCH_SITES_TMP} | ${AWK} '${MASTER_SORT_AWK:S|\\|\\\\|g}'` ${_MASTER_SITE_BACKUP}" ; \
+				SORTED_PATCH_SITES_CMD_TMP="echo ${_MASTER_SITE_OVERRIDE} `echo $${__PATCH_SITES_TMP} | ${AWK} '${MASTER_SORT_AWK:S|\\|\\\\|g}'`" ; \
+				WITH_BACKUP_SITE=yes;\
 			else \
 				SORTED_PATCH_SITES_CMD_TMP="${SORTED_PATCH_SITES_DEFAULT_CMD}" ; \
 			fi ; \
-			for site in `eval $$SORTED_PATCH_SITES_CMD_TMP`; do \
+			if [ ! -z ${ENABLE_CLOSEST_MIRROR} ] ;\
+			then \
+				${ECHO_CMD} "Looking for closest mirror, please wait...";\
+				SITES=$$(for site in `eval $$SORTED_PATCH_SITES_CMD_TMP`;\
+				do host=`${ECHO} $$site | ${AWK} -F\/ '{print $$3;}'`;\
+				time=`${PING} -Q -t 10 -c 2 $$host | ${AWK} -F\/ '$$1 ~ /^round-trip/ {print $$6;}' 2> /dev/null` ;\
+				${ECHO} $$time $$site ; \
+				done  | ${GREP} -v ${_MASTER_SITE_BACKUP}| ${SORT} -n | ${AWK} '$$2 ~ /^(ftp|http)/ {print $$2;}');\
+				SITES="$$SITES ${_MASTER_SITE_BACKUP}"; \
+			else \
+				SITES=`eval $$SORTED_PATCH_SITES_CMD_TMP | ${GREP} -v ${_MASTER_SITE_BACKUP}`; \
+				[ "x$$WITH_BACKUP_SITE" = "x" ] ||  SITES="$$SITES ${_MASTER_SITE_BACKUP}"; \
+			fi;\
+			[ ! -z ${FETCH_DEBUG} ] && ( ${ECHO} '*** DEBUG *** SITES:' ; ${ECHO} $$SITES) ;\
+			for site in $$SITES ; do \
 			    ${ECHO_MSG} ">> Attempting to fetch from $${site}."; \
 				DIR=${DIST_SUBDIR}; \
 				CKSIZE=`${GREP} "^SIZE ($${DIR:+$$DIR/}$$file)" ${MD5_FILE} | ${AWK} '{print $$4}'`; \



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20031022015256.57519ab5.sheepkiller>