Date: Tue, 18 Mar 2008 13:32:22 GMT From: Anton Statutov <astatutov@gmail.com> To: freebsd-gnats-submit@FreeBSD.org Subject: ports/121828: NEW PORT: sysutils/getdelta, it allows to dramatically reduce download size of ports source files Message-ID: <200803181332.m2IDWMh5085140@www.freebsd.org> Resent-Message-ID: <200803181340.m2IDe1R9077098@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 121828
>Category: ports
>Synopsis: NEW PORT: sysutils/getdelta, it allows to dramatically reduce download size of ports source files
>Confidential: no
>Severity: non-critical
>Priority: medium
>Responsible: freebsd-ports-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: change-request
>Submitter-Id: current-users
>Arrival-Date: Tue Mar 18 13:40:01 UTC 2008
>Closed-Date:
>Last-Modified:
>Originator: Anton Statutov
>Release: 6.3
>Organization:
>Environment:
FreeBSD we.bs 6.3-RELEASE-p1 FreeBSD 6.3-RELEASE-p1 #0: Wed Feb 13 02:40:56 UTC 2008 root@i386-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC i386
>Description:
This is port of Gentoo 'getdelta.sh' script which replaces standard fetch-command and uses Deltup to reduce download size of ports source files. This is done by downloading only binary diffs of updated port sources whenever possible.
This is the second PR, mainly done by Andrew R. Muhametshin. The first PR 'ports/120463' was very raw and now can be ignored.
>From now on, this port depends on sysutils/deltup 0.4.4 and therefore can be added to ports collection only after 'ports/121517'.
>How-To-Repeat:
>Fix:
Patch attached with submission follows:
# This is a shell archive. Save it in a file, remove anything before
# this line, and then unpack it by entering "sh file". Note, it may
# create directories; files and directories will be owned by you and
# have default permissions.
#
# This archive contains:
#
# /usr/ports/sysutils/getdelta
# /usr/ports/sysutils/getdelta/files
# /usr/ports/sysutils/getdelta/files/patch-getdelta.sh
# /usr/ports/sysutils/getdelta/Makefile
# /usr/ports/sysutils/getdelta/pkg-descr
# /usr/ports/sysutils/getdelta/pkg-message
# /usr/ports/sysutils/getdelta/pkg-plist
# /usr/ports/sysutils/getdelta/distinfo
#
echo c - /usr/ports/sysutils/getdelta
mkdir -p /usr/ports/sysutils/getdelta > /dev/null 2>&1
echo c - /usr/ports/sysutils/getdelta/files
mkdir -p /usr/ports/sysutils/getdelta/files > /dev/null 2>&1
echo x - /usr/ports/sysutils/getdelta/files/patch-getdelta.sh
sed 's/^X//' >/usr/ports/sysutils/getdelta/files/patch-getdelta.sh << 'END-of-/usr/ports/sysutils/getdelta/files/patch-getdelta.sh'
X--- ./getdelta.sh.orig 2008-03-15 23:50:36.000000000 +0300
X+++ ./getdelta.sh 2008-03-15 23:50:04.000000000 +0300
X@@ -1,4 +1,4 @@
X-#!/bin/sh
X+#!/usr/local/bin/bash
X # getdelta.sh
X # A download-wrapper script for gentoo that tries to get dtu files
X # created by deltup instead of downloading complete source-files
X@@ -22,9 +22,35 @@
X # 59 Temple Place, Suite 330
X # Boston, MA 02111, USA.
X
X+
X VERSION="0.7"
X
X+
X # Changelog
X+# version 0.7.8-freebsd_4 2008/03/15
X+# Anton Statutov:
X+# - version revision
X+# version 0.7.8-freebsd_3 2008/03/11
X+# Andrew R. Muhametshin:
X+# - fixed litte bug
X+# - fixed 'handled situation when required file already exists' - added the real check.
X+# version 0.7.8-freebsd_2 2008/03/11
X+# Andrew R. Muhametshin:
X+# - added refetch feature
X+# - added 'tgz' mask
X+# - added check of FETCH_CMD parameters
X+# - distfiles path fixed
X+# - fixed 'date' calls
X+# - some work on CHECK_OLD_FILE
X+# Anton Statutov:
X+# - fixed litte ouput bug
X+# version 0.7.8-freebsd_1 2008/02/12
X+# - handled situation when required file already exists in distfiles
X+# version 0.7.8-freebsd 2008/02/09 by Anton Statutov stant@bestyle.ru
X+# - replaced "stat -c %s" to "stat -f "%z"
X+# - replaced "/etc/deltup" to "/usr/local/etc/deltup"
X+# - replaced $1 to $2 because $1 is the size of file on freebsd
X+# - some minor changes
X # version 0.7.8 2007/07/13
X # - added option -O to call of emerge when finding mirrors
X # - replaced some calls of coreutils with bash-internals
X@@ -63,7 +89,7 @@
X # instead of asking the server - the server does not use this parameter - it just
X # exists to create unique request-URLS (as proposed by bodo "bothie" thiesen)
X # version 0.6.9 2005/03/11
X-# exit with exitcode of wget to signal to portage if
X+# - exit with exitcode of wget to signal to portage if
X # download was successful
X # - handle metadata.dtd as exception (repoman uses FETCHCOMMAND)
X # - you can disable fetching of dtu-files now by setting
X@@ -208,7 +234,9 @@
X # This file: #
X ####################################################
X
X-GETDELTA_CONFIGFILE=/etc/deltup/getdelta.rc
X+PREFIX=/usr/local
X+GETDELTA_CONFIGFILE="${PREFIX}/etc/deltup/getdelta.rc"
X+DISTDIR=$(pwd)
X #
X
X splitversion(){
X@@ -368,6 +396,7 @@
X # ignore some strings in any filename
X echo $(sed -e "s/\.bz2$//g" \
X -e "s/\.gz$//g" \
X+ -e "s/\.tgz$//g" \
X -e "s/[0-9]//g" \
X -e "s/pre//g" \
X -e "s/preview//g" \
X@@ -378,18 +407,34 @@
X -e "s/ //g" <<< $MASK_FILENAME)
X }
X
X+check_file(){
X+ _return=0
X+ _sfx1="$(rev <<< $1 | cut -d. -f1 | rev)"
X+ _sfx2="$(rev <<< $1 | cut -d. -f2 | rev)"
X+ if [ ${_sfx2} = "tar" -o ${_sfx1} = "tgz" -o ${_sfx1} = "tbz" ]
X+ then
X+ output "Testing file integrity with tar.\n"
X+ if ! tar -tf $1 >/dev/null
X+ then
X+ output "${RED}reported an error while testing ${CYAN}$1${RED} - so this file is unusable.\n"
X+ _return=1
X+ fi
X+ fi
X+ return ${_return}
X+}
X+
X # create or update a config-file
X
X add_to_configfile KDE_MIRROR "ftp://ftp.kde.org/pub/kde/stable" "we de not get kde-deltas from a delta-up-server, since kde provides own xdelta-files"
X add_to_configfile LOCAL_MIRROR "" "set this to one or more (space separated) URI ending with '/' if you want to check one or more local mirror(s) first\n# most people just leave it empty."
X add_to_configfile DELTUP_SERVER "http://linux01.gwdg.de/~nlissne/deltup.php" "deltup-server to use"
X-add_to_configfile FETCH "/usr/bin/wget -t 1 --passive-ftp" "command to use for downloading"
X+add_to_configfile FETCH "${PREFIX}/bin/wget -t 1 --passive-ftp" "command to use for downloading"
X add_to_configfile QUEUERETRY 15 "number of seconds to wait before a queued request is retried"
X add_to_configfile MAXIMUM_ACCEPTABLE_QUEUEPOS "15" "the maximum queuepos you would accept (if higher download full archive instead)"
X add_to_configfile QUEUETIMEOUT 900 "when a dtu-request is queued - how long should we wait max. before downloading the original archive instead (in seconds)"
X add_to_configfile CHECK_OLD_FILE "false" "set to \"true\", if you want getdelta.sh to use Pkunk's integrity check for the old file before downloading dtu-files"
X add_to_configfile REMOVE_OLD "false" "set to \"true\", if you want getdelta.sh to delete the old file, if patch was succesful"
X-add_to_configfile DO_NOT_REMOVE "/etc/deltup/do_not_remove" "a list of files not to be removed by REMOVE_OLD feature"
X+add_to_configfile DO_NOT_REMOVE "${PREFIX}/etc/deltup/do_not_remove" "a list of files not to be removed by REMOVE_OLD feature"
X add_to_configfile REMOVE_INCOMPLETE_OLD_FILES "false" "set this to \"true\" if you want getdelta.sh to delete old versions that seems to be corrupt,\n# or to \"false\" if you want to delete them manually\n# note: getdelta.sh will not use these files anyway"
X add_to_configfile VERBOSITY true "set to \"true\", if you want verbose outputs (later to be set to a level [0-3])"
X add_to_configfile COLOR true "set to \"true\", if you want colorful messages, \"false\" if not."
X@@ -424,23 +469,6 @@
X add_to_donotremove "^rp-pppoe"
X
X
X-if [ -z $1 ]
X-then
X- COLOR=true
X- echo -e "${YELLOW}getdelta.sh version ${VERSION}"
X- echo "This script has to be called like this:"
X- echo -e "${CYAN}$0 <URI>"
X- echo -e "\n${YELLOW}To use it, you should just put the following line into your /etc/make.conf"
X- echo -e "${GREEN}FETCHCOMMAND=\"$0 \\\${URI}\""
X- echo -e "\n${YELLOW}There is a config-file ${CYAN}${GETDELTA_CONFIGFILE}${YELLOW} with some variables to control the behaviour of this script."
X- echo -e "Edit it to your needs.${NORMAL}"
X- exit 1
X-fi
X-# include variables from gentoo make.globals and make.conf
X-source /etc/make.globals
X-source /etc/make.conf
X-
X-
X if ${COLOR}
X then
X RED="\033[01;31m"
X@@ -459,14 +487,32 @@
X CYAN=""
X NORMAL=""
X fi
X+
X grep -q "getdelta.sh" <<< "${RESUMECOMMAND}" &&
X output "${RED}do NOT set RESUMECOMMAND to use getdelta.sh" &&
X output "use getdelta.sh for your FETCHCOMMAND, only." &&
X sleep 5 && exit 1
X
X-pushd $DISTDIR >/dev/null 2>/dev/null
X-ORIG_URI=$1
X+
X+if [[ "${1%%-S*}" ]] ; then ORIG_URI=$1 ; else ORIG_URI=$2 ; fi
X NEW_FILE=$(basename $ORIG_URI)
X+TMP_DWN_FILE="${DISTDIR}/.download_tmp-$NEW_FILE"
X+
X+
X+if [ -z "${ORIG_URI}" -a -z "${NEW_FILE}" ]
X+then
X+ COLOR=true
X+ echo -e "${YELLOW}getdelta.sh version ${VERSION}"
X+ echo "This script has to be called like this:"
X+ echo -e "${CYAN}$0 <URI>"
X+ echo -e "\n${YELLOW}To use it, you should just put the following line into your /etc/make.conf"
X+ echo -e "${GREEN}FETCH_CMD=\"$0\""
X+ echo -e "\n${YELLOW}There is a config-file ${CYAN}${GETDELTA_CONFIGFILE}${YELLOW} with some variables to control the behaviour of this script."
X+ echo -e "Edit it to your needs.${NORMAL}"
X+ exit 1
X+fi
X+
X+pushd $DISTDIR >/dev/null 2>/dev/null
X
X # repoman downloads metadata.dtd with FETCHCOMMAND
X # this should not be done with getdelta - so just fetch the file and exit
X@@ -578,7 +624,12 @@
X if ! [ -z "$filelist" ]
X then
X # we have got a list of candidates in $filelist now. find the best match .
X- output "${GREEN}We have the following candidates to choose from \n${YELLOW}`sed -e \"s/\ /\\n/g\" <<< $filelist` \n"
X+ output "${GREEN}We have the following candidates to choose from:"
X+ for _name in $filelist
X+ do
X+ output "${YELLOW}${_name}"
X+ done
X+ output ""
X
X # find matching part of filename - first: frontmatch
X x=0;
X@@ -588,7 +639,7 @@
X do
X for ((i=0;i<${#a[@]};i++))
X do
X- [ ${a[0]:${x}:1} != ${a[$i]:${x}:1} ] && match=$x
X+ [[ ${a[0]:${x}:1} != ${a[$i]:${x}:1} ]] && match=$x
X done
X ((x++))
X done
X@@ -601,7 +652,7 @@
X do
X for ((i=0;i<${#a[@]};i++))
X do
X- [ ${a[0]:${#a[0]}-${x}:1} != ${a[$i]:${#a[$i]}-${x}:1} ] && match=$x
X+ [[ ${a[0]:${#a[0]}-${x}:1} != ${a[$i]:${#a[$i]}-${x}:1} ]] && match=$x
X done
X ((x++))
X done
X@@ -652,31 +703,23 @@
X output "${GREEN}The best of all is ... ${CYAN}${best_candidate}\n"
X output "${YELLOW}Checking if this file is OK.\n"
X
X- # this part is based on Pkunk's code posted on http://bugs.gentoo.org/show_bug.cgi?id=63525
X- # but with some changes
X FILE_IS_CORRUPT=false
X- if $CHECK_OLD_FILE
X- then
X- file_digest=$(grep -h ${best_candidate} ${FILESDIR}/digest-* | sed -n 1p)
X- if [ "$file_digest" ]
X+
X+ if [ "${best_candidate}" = "${NEW_FILE}" ]
X then
X- file_md5=$(cut -d ' ' -f2 <<< "$file_digest")
X- file_origsize=$(cut -d ' ' -f4 <<< "$file_digest")
X- file_currentsize=$(stat -c %s ${best_candidate})
X- if [ $file_currentsize -ne $file_origsize ]
X+ output "${RED}Required file already exists... \n"
X+ if ! check_file "${best_candidate}"
X then
X- output "${RED}Found ${best_candidate}, but filesize ${CYAN}${file_currentsize} ${RED} does not match ${CYAN}${file_origsize} (found in digest-file)\n"
X- FILE_IS_CORRUPT=true
X- fi
X+ exit 1
X else
X- if [ $(rev <<< ${best_candidate} | cut -d. -f2 | rev) = "tar" ]
X- then
X- output "${YELLOW}Could not find a digest-file for ${CYAN}${best_candidate}${YELLOW}. Testing file integrity with tar.\n"
X- if ! tar -tf ${best_candidate} >/dev/null
X- then
X- output "${RED}reported an error while testing ${CYAN}${best_candidate}${RED} - so this file is unusable.\n"
X- FILE_IS_CORRUPT=true
X+ output "${RED}md5 checksum is not correct ${CYAN}\"$DISTDIR/${best_candidate}\"${RED} - but this file is usable.\n"
X+ exit 0
X fi
X+ fi
X+
X+ if $CHECK_OLD_FILE
X+ then
X+ check_file "${best_candidate}" || FILE_IS_CORRUPT=true
X
X if $FILE_IS_CORRUPT && $REMOVE_INCOMPLETE_OLD_FILES
X then
X@@ -684,27 +727,24 @@
X remove ${best_candidate}
X fi
X fi
X- fi
X- fi
X+
X # end of file-corruption check for $best_candidate found in distfiles
X if ! $FILE_IS_CORRUPT
X then
X
X- QUERY_URL=$(GENTOO_MIRRORS="" emerge -fOp =${CATEGORY}/${PF} 2>&1 |
X- sed -e "s/ /\\n/g" | egrep "(http|ftp)://" |
X- grep "${NEW_FILE}" | tail -n 1)
X+ QUERY_URL=${ORIG_URI}
X query="?have=${best_candidate}&want=${NEW_FILE}&url=${QUERY_URL}&version=${VERSION}&time=$(date +%s)"
X output "${GREEN}Trying to download ${YELLOW}${best_candidate}-${NEW_FILE}.dtu\n"
X
X # Remember where we are, and go to a new dir there we can work
X- tmp_dwn_dest="${DISTDIR}/.getdelta-`date +%N`-tmp"
X+ tmp_dwn_dest="${DISTDIR}/.getdelta-`date +%s`_${RANDOM}-tmp"
X mkdir ${tmp_dwn_dest}
X # If user abort Ctrl+C (signal 2), remove tmp-dir; enabable trap again and send it again to stop wget
X trap "rm -r ${tmp_dwn_dest}; trap 2; kill -2 $$" 2
X pushd ${tmp_dwn_dest} >/dev/null 2>&1
X
X # thanks to MATSUI Fe2+ Tetsushi for idea and patch
X- FILESIZE=$(stat -c %s "${DISTDIR}/${best_candidate}")
X+ FILESIZE=$(stat -f "%z" "${DISTDIR}/${best_candidate}")
X let TIMELIMIT=${FILESIZE}/${BANDWIDTH}
X [[ $TIMELIMIT -lt $QUEUETIMEOUT ]] && QUEUETIMEOUT=$TIMELIMIT
X
X@@ -771,10 +811,10 @@
X if [ -f ${best_candidate}-${NEW_FILE}.dtu ]
X then
X output "${GREEN}Successfully fetched the dtu-file - let's build ${NEW_FILE}...\n"
X- downloadsize=$(stat -c %s ${best_candidate}-${NEW_FILE}.dtu)
X+ downloadsize=$(stat -f "%z" ${best_candidate}-${NEW_FILE}.dtu)
X if deltup -p -v -D ${DISTDIR} ${best_candidate}-${NEW_FILE}.dtu
X then
X- newsize=$(stat -c %s ${NEW_FILE})
X+ newsize=$(stat -f "%z" ${NEW_FILE})
X let savedsize=${newsize}-${downloadsize}
X let percent=${savedsize}*100/${newsize}
X unit="bytes"
X@@ -824,12 +864,22 @@
X
X # Ok, once we are here, we should have got the delta (and used it)
X # or we still have to download the full file
X+
X if ! [ -f ${DISTDIR}/${NEW_FILE} ]
X then
X output "${RED}The dtu could not be fetched,${YELLOW} downloading full file from original URL\n"
X- $FETCH $ORIG_URI
X+
X+ $FETCH -O "${TMP_DWN_FILE}" -c "$ORIG_URI"
X+
X # remember we had a fallback to use correct exitcode for portage
X FALLBACK=$?
X+
X+ if [ $FALLBACK = 0 ] ; then
X+ output "${YELOW}Download complete\n"
X+ mv "${TMP_DWN_FILE}" "${DISTDIR}/${NEW_FILE}"
X+ fi
X+else
X+ [ -f "${TMP_DWN_FILE}" ] && rm "${TMP_DWN_FILE}"
X fi
X
X
END-of-/usr/ports/sysutils/getdelta/files/patch-getdelta.sh
echo x - /usr/ports/sysutils/getdelta/Makefile
sed 's/^X//' >/usr/ports/sysutils/getdelta/Makefile << 'END-of-/usr/ports/sysutils/getdelta/Makefile'
X# Ports collection makefile for: getdelta
X# Date created: 15 December 2007
X# Whom: Anton Statutov <astatutov@gmail.com>
X#
X# $FreeBSD$
X#
X
XPORTNAME= getdelta
XPORTVERSION= 0.7.8
XPORTREVISION= 4
XCATEGORIES= sysutils
XMASTER_SITES= ${MASTER_SITE_GENTOO} \
X http://132.207.4.160/distfiles/ \
X http://distfiles.gentoo.org/distfiles/ \
X http://linux01.gwdg.de/~nlissne/
XMASTER_SITE_SUBDIR= distfiles
X
XMAINTAINER= astatutov@gmail.com
XCOMMENT= Script for reducing bandwidth load while upgrading ports disfiles
X
XRUN_DEPENDS= deltup>=0.4.4:${PORTSDIR}/sysutils/deltup \
X wget:${PORTSDIR}/ftp/wget \
X bash:${PORTSDIR}/shells/bash
X
XUSE_BZIP2= yes
XNO_WRKSUBDIR= yes
XNO_BUILD= yes
X
X.include <bsd.port.pre.mk>
X
Xdo-install:
X ${INSTALL} ${WRKSRC}/getdelta.sh ${PREFIX}/bin/getdelta.sh
X
Xpost-install:
X @${CAT} ${PKGMESSAGE}
X
X.include <bsd.port.post.mk>
END-of-/usr/ports/sysutils/getdelta/Makefile
echo x - /usr/ports/sysutils/getdelta/pkg-descr
sed 's/^X//' >/usr/ports/sysutils/getdelta/pkg-descr << 'END-of-/usr/ports/sysutils/getdelta/pkg-descr'
XThis is port of Gentoo getdelta.sh.
XGetdelta uses Deltup to reduce bandwidth load while upgrading ports distfiles.
X
XWWW: http://linux01.gwdg.de/~nlissne/
X
X- Anton Statutov
Xastatutov@gmail.com
END-of-/usr/ports/sysutils/getdelta/pkg-descr
echo x - /usr/ports/sysutils/getdelta/pkg-message
sed 's/^X//' >/usr/ports/sysutils/getdelta/pkg-message << 'END-of-/usr/ports/sysutils/getdelta/pkg-message'
X=========================================================================================
XATTENTION!
XTo use Getdelta for upgrading ports you need to add the following string into /etc/make.conf:
XFETCH_CMD=/usr/local/bin/getdelta.sh
X
XTo do this you can simply run the following command:
Xecho "FETCH_CMD=/usr/local/bin/getdelta.sh" >> /etc/make.conf
X=========================================================================================
END-of-/usr/ports/sysutils/getdelta/pkg-message
echo x - /usr/ports/sysutils/getdelta/pkg-plist
sed 's/^X//' >/usr/ports/sysutils/getdelta/pkg-plist << 'END-of-/usr/ports/sysutils/getdelta/pkg-plist'
Xbin/getdelta.sh
X
END-of-/usr/ports/sysutils/getdelta/pkg-plist
echo x - /usr/ports/sysutils/getdelta/distinfo
sed 's/^X//' >/usr/ports/sysutils/getdelta/distinfo << 'END-of-/usr/ports/sysutils/getdelta/distinfo'
XMD5 (getdelta-0.7.8.tar.bz2) = e6feab0fe6a71f63e5a693c67ef7fd1e
XSHA256 (getdelta-0.7.8.tar.bz2) = 72bd5dac59e86a5c78217a15e3198df513152dc784a8685b1a8c2c5efdb34217
XSIZE (getdelta-0.7.8.tar.bz2) = 10335
END-of-/usr/ports/sysutils/getdelta/distinfo
exit
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200803181332.m2IDWMh5085140>