Date: Mon, 10 Nov 2008 18:56:16 +0300 (MSK) From: Eygene Ryabinkin <rea-fbsd@codelabs.ru> To: FreeBSD-gnats-submit@freebsd.org Subject: ports/128754: [port infrastructure] implement master sites randomization Message-ID: <20081110155616.DA66A1AF424@void.codelabs.ru> Resent-Message-ID: <200811101600.mAAG0Are069585@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 128754
>Category: ports
>Synopsis: [port infrastructure] implement master sites randomization
>Confidential: no
>Severity: non-critical
>Priority: low
>Responsible: freebsd-ports-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Mon Nov 10 16:00:09 UTC 2008
>Closed-Date:
>Last-Modified:
>Originator: Eygene Ryabinkin
>Release: FreeBSD 7.1-PRERELEASE i386
>Organization:
Code Labs
>Environment:
System: FreeBSD 7.1-PRERELEASE i386
>Description:
Today I was hit by the very bad connectivity with twaren.net and
nchc.dl.sourceforge.net (the first site in the SF mirrors list in the
FreeBSD ports .mk files) is hosted by Taiwan REN. So, I decided to
implement simple randomization that will enable to evenly distribute
the downloads between SF mirrors.
I had made it only for the SF mirrors, but it can be done for the other
well-known download locations as well.
>How-To-Repeat:
Look at ${PORTSDIR}/Mk/bsd.sites.mk: it uses statical assignment to
the MASTER_SITE_SOURCEFORGE.
>Fix:
The following patch works for me. It places simple script rnd.awk to
the ${PORTSDIR}/Mk -- I don't think that this is the most appropriate
place to put it, but I can't think of another place now -- the script is
too ports-specific to put it into, for example, /usr/bin/.
--- 0001-Implement-randomization-of-main-SourceForge-sites.patch begins here ---
>From d48aa3982a5ca6192fad561462c3ba9da2b38cfa Mon Sep 17 00:00:00 2001
From: Eygene Ryabinkin <rea-fbsd@codelabs.ru>
Date: Mon, 10 Nov 2008 18:38:01 +0300
The reasoning behind this is simple: today I had experienced very bad
connectivity with twaren.net and it is the first mirror in the list.
So, for automated upgrades of ports that are using SourceForge mirrors,
the download was very time-consuming. Such round-robin randomization
of the download sites helps to improve the situation.
Signed-off-by: Eygene Ryabinkin <rea-fbsd@codelabs.ru>
---
Mk/bsd.commands.mk | 1 +
Mk/bsd.sites.mk | 9 ++++++---
Mk/rnd.awk | 14 ++++++++++++++
3 files changed, 21 insertions(+), 3 deletions(-)
create mode 100644 Mk/rnd.awk
diff --git a/Mk/bsd.commands.mk b/Mk/bsd.commands.mk
index aa8bde8..0b2a58b 100644
--- a/Mk/bsd.commands.mk
+++ b/Mk/bsd.commands.mk
@@ -69,6 +69,7 @@ PASTE?= /usr/bin/paste
PAX?= /bin/pax
PRINTF?= /usr/bin/printf
PW?= /usr/sbin/pw
+RANDLIST?= ${AWK} -f ${PORTSDIR}/Mk/rnd.awk
REALPATH?= /bin/realpath
RM?= /bin/rm
RMDIR?= /bin/rmdir
diff --git a/Mk/bsd.sites.mk b/Mk/bsd.sites.mk
index 41b3f93..bd9a2a6 100644
--- a/Mk/bsd.sites.mk
+++ b/Mk/bsd.sites.mk
@@ -1188,20 +1188,23 @@ MASTER_SITE_SAVANNAH+= \
.if !defined(IGNORE_MASTER_SITE_SOURCEFORGE)
.for mirror in nchc kent easynews ufpr umn mesh heanet
-MASTER_SITE_SOURCEFORGE+= \
+MASTER_SITES_SOURCEFORGE+= \
http://${mirror}.dl.sourceforge.net/sourceforge/%SUBDIR%/
.endfor
+MASTER_SITE_SOURCEFORGE != ${RANDLIST} ${MASTER_SITES_SOURCEFORGE}
.endif
# official sf.net mirrors that don't mirror all projects, check
# http://prdownloads.sourceforge.net/%SUBDIR%/
.if !defined(IGNORE_MASTER_SITE_SOURCEFORGE_EXTENDED)
.for mirror in easynews switch puzzle belnet osdn ovh keihanna
-MASTER_SITE_SOURCEFORGE_EXTENDED+= \
+MASTER_SITES_SOURCEFORGE_EXTENDED+= \
http://${mirror}.dl.sourceforge.net/sourceforge/%SUBDIR%/
.endfor
-MASTER_SITE_SOURCEFORGE_EXTENDED+= \
+MASTER_SITES_SOURCEFORGE_EXTENDED+= \
${MASTER_SITE_SOURCEFORGE}
+MASTER_SITE_SOURCEFORGE_EXTENDED != \
+ ${RANDLIST} ${MASTER_SITES_SOURCEFORGE_EXTENDED}
.endif
.if !defined(IGNORE_MASTER_SITE_SOURCEFORGE_JP)
diff --git a/Mk/rnd.awk b/Mk/rnd.awk
new file mode 100644
index 0000000..fbfcad7
--- /dev/null
+++ b/Mk/rnd.awk
@@ -0,0 +1,14 @@
+BEGIN {
+# Need to drop a couple of initial rand() values: they tend
+# to be around 0.8 - 0.9, so for fairly small array lenght
+# they will produce identical values at the beginning.
+ srand(); rand(); rand(); rand(); rand();
+ for (i = ARGC - 1; i > 0; i--) {
+ j = int(10*ARGC*rand()) % i + 1;
+ if (j == i) continue;
+ t = ARGV[i]; ARGV[i] = ARGV[j]; ARGV[j] = t;
+ }
+
+ for (i = 1; i < ARGC; i++)
+ print ARGV[i];
+}
--
1.6.0.3
--- 0001-Implement-randomization-of-main-SourceForge-sites.patch ends here ---
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20081110155616.DA66A1AF424>