From owner-freebsd-ports-bugs@FreeBSD.ORG Mon Nov 10 16:00:11 2008 Return-Path: Delivered-To: freebsd-ports-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1571D106567B; Mon, 10 Nov 2008 16:00:11 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id A87898FC21; Mon, 10 Nov 2008 16:00:10 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id mAAG0A8N069592; Mon, 10 Nov 2008 16:00:10 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id mAAG0Are069585; Mon, 10 Nov 2008 16:00:10 GMT (envelope-from gnats) Resent-Date: Mon, 10 Nov 2008 16:00:10 GMT Resent-Message-Id: <200811101600.mAAG0Are069585@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@freebsd.org (GNATS Filer) Resent-To: freebsd-ports-bugs@FreeBSD.org Resent-Cc: ports@freebsd.org, pav@freebsd.org, gabor@freebsd.org Resent-Reply-To: FreeBSD-gnats-submit@freebsd.org, Eygene Ryabinkin Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 868EE1065690 for ; Mon, 10 Nov 2008 15:56:19 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) Received: from 0.mx.codelabs.ru (0.mx.codelabs.ru [144.206.177.45]) by mx1.freebsd.org (Postfix) with ESMTP id 378DD8FC0A for ; Mon, 10 Nov 2008 15:56:18 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) Received: from void.codelabs.ru (void.codelabs.ru [144.206.177.25]) by 0.mx.codelabs.ru with esmtps (TLSv1:CAMELLIA256-SHA:256) id 1KzZ7h-000Mtc-8f for FreeBSD-gnats-submit@freebsd.org; Mon, 10 Nov 2008 18:56:17 +0300 Message-Id: <20081110155616.DA66A1AF424@void.codelabs.ru> Date: Mon, 10 Nov 2008 18:56:16 +0300 (MSK) From: Eygene Ryabinkin To: FreeBSD-gnats-submit@freebsd.org X-Send-Pr-Version: 3.113 X-GNATS-Notify: ports@freebsd.org, pav@freebsd.org, gabor@freebsd.org Cc: Subject: ports/128754: [port infrastructure] implement master sites randomization X-BeenThere: freebsd-ports-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Eygene Ryabinkin List-Id: Ports bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Nov 2008 16:00:11 -0000 >Number: 128754 >Category: ports >Synopsis: [port infrastructure] implement master sites randomization >Confidential: no >Severity: non-critical >Priority: low >Responsible: freebsd-ports-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Mon Nov 10 16:00:09 UTC 2008 >Closed-Date: >Last-Modified: >Originator: Eygene Ryabinkin >Release: FreeBSD 7.1-PRERELEASE i386 >Organization: Code Labs >Environment: System: FreeBSD 7.1-PRERELEASE i386 >Description: Today I was hit by the very bad connectivity with twaren.net and nchc.dl.sourceforge.net (the first site in the SF mirrors list in the FreeBSD ports .mk files) is hosted by Taiwan REN. So, I decided to implement simple randomization that will enable to evenly distribute the downloads between SF mirrors. I had made it only for the SF mirrors, but it can be done for the other well-known download locations as well. >How-To-Repeat: Look at ${PORTSDIR}/Mk/bsd.sites.mk: it uses statical assignment to the MASTER_SITE_SOURCEFORGE. >Fix: The following patch works for me. It places simple script rnd.awk to the ${PORTSDIR}/Mk -- I don't think that this is the most appropriate place to put it, but I can't think of another place now -- the script is too ports-specific to put it into, for example, /usr/bin/. --- 0001-Implement-randomization-of-main-SourceForge-sites.patch begins here --- >From d48aa3982a5ca6192fad561462c3ba9da2b38cfa Mon Sep 17 00:00:00 2001 From: Eygene Ryabinkin Date: Mon, 10 Nov 2008 18:38:01 +0300 The reasoning behind this is simple: today I had experienced very bad connectivity with twaren.net and it is the first mirror in the list. So, for automated upgrades of ports that are using SourceForge mirrors, the download was very time-consuming. Such round-robin randomization of the download sites helps to improve the situation. Signed-off-by: Eygene Ryabinkin --- Mk/bsd.commands.mk | 1 + Mk/bsd.sites.mk | 9 ++++++--- Mk/rnd.awk | 14 ++++++++++++++ 3 files changed, 21 insertions(+), 3 deletions(-) create mode 100644 Mk/rnd.awk diff --git a/Mk/bsd.commands.mk b/Mk/bsd.commands.mk index aa8bde8..0b2a58b 100644 --- a/Mk/bsd.commands.mk +++ b/Mk/bsd.commands.mk @@ -69,6 +69,7 @@ PASTE?= /usr/bin/paste PAX?= /bin/pax PRINTF?= /usr/bin/printf PW?= /usr/sbin/pw +RANDLIST?= ${AWK} -f ${PORTSDIR}/Mk/rnd.awk REALPATH?= /bin/realpath RM?= /bin/rm RMDIR?= /bin/rmdir diff --git a/Mk/bsd.sites.mk b/Mk/bsd.sites.mk index 41b3f93..bd9a2a6 100644 --- a/Mk/bsd.sites.mk +++ b/Mk/bsd.sites.mk @@ -1188,20 +1188,23 @@ MASTER_SITE_SAVANNAH+= \ .if !defined(IGNORE_MASTER_SITE_SOURCEFORGE) .for mirror in nchc kent easynews ufpr umn mesh heanet -MASTER_SITE_SOURCEFORGE+= \ +MASTER_SITES_SOURCEFORGE+= \ http://${mirror}.dl.sourceforge.net/sourceforge/%SUBDIR%/ .endfor +MASTER_SITE_SOURCEFORGE != ${RANDLIST} ${MASTER_SITES_SOURCEFORGE} .endif # official sf.net mirrors that don't mirror all projects, check # http://prdownloads.sourceforge.net/%SUBDIR%/ .if !defined(IGNORE_MASTER_SITE_SOURCEFORGE_EXTENDED) .for mirror in easynews switch puzzle belnet osdn ovh keihanna -MASTER_SITE_SOURCEFORGE_EXTENDED+= \ +MASTER_SITES_SOURCEFORGE_EXTENDED+= \ http://${mirror}.dl.sourceforge.net/sourceforge/%SUBDIR%/ .endfor -MASTER_SITE_SOURCEFORGE_EXTENDED+= \ +MASTER_SITES_SOURCEFORGE_EXTENDED+= \ ${MASTER_SITE_SOURCEFORGE} +MASTER_SITE_SOURCEFORGE_EXTENDED != \ + ${RANDLIST} ${MASTER_SITES_SOURCEFORGE_EXTENDED} .endif .if !defined(IGNORE_MASTER_SITE_SOURCEFORGE_JP) diff --git a/Mk/rnd.awk b/Mk/rnd.awk new file mode 100644 index 0000000..fbfcad7 --- /dev/null +++ b/Mk/rnd.awk @@ -0,0 +1,14 @@ +BEGIN { +# Need to drop a couple of initial rand() values: they tend +# to be around 0.8 - 0.9, so for fairly small array lenght +# they will produce identical values at the beginning. + srand(); rand(); rand(); rand(); rand(); + for (i = ARGC - 1; i > 0; i--) { + j = int(10*ARGC*rand()) % i + 1; + if (j == i) continue; + t = ARGV[i]; ARGV[i] = ARGV[j]; ARGV[j] = t; + } + + for (i = 1; i < ARGC; i++) + print ARGV[i]; +} -- 1.6.0.3 --- 0001-Implement-randomization-of-main-SourceForge-sites.patch ends here --- >Release-Note: >Audit-Trail: >Unformatted: