Date: Fri, 17 Mar 2006 02:02:44 +0600 From: Vadim Goncharov <vadim_nuclight@mail.ru> To: freebsd-ports@freebsd.org Subject: Getting distfiles for entire tree Message-ID: <1965362278.20060317020244@mail.ru>
next in thread | raw e-mail | index | archive | help
Hello freebsd-ports, I need to maintain a local distfiles mirror (all N-ty gigabytes) for almost entire ports tree (some lang categories excluded). Also, I have limited access to Internet - not all distfiles could be fetched (however eventually they'll do). So, as I can't use plain ``cd /usr/ports && make fetch'', I've made a script, which, for speeding up, fetches each category in parallel. Script was simple, so it hasn't got all, mainly because of interactive ports, etc. So I searched a little and, after reading some tips, modified script - commented ``find'' and inserted ``daemon''. Here it is: #!/bin/sh CATEGORIES="accessibility archivers astro audio benchmarks biology cad comms converters databases deskutils devel dns editors emulators finance ftp games graphics irc japanese java lang mail math mbone misc multimedia net net-im net-mgmt net-p2p news palm print russian science security shells sysutils textproc ukrainian www x11 x11-clocks x11-fm x11-fonts x11-servers x11-themes x11-toolkits x11-wm" LOG=/data/FreeBSD/fetchlog # vars for fetch export HTTP_TIMEOUT=30 export FTP_TIMEOUT=30 for i in $CATEGORIES ; do cd /usr/ports/$i || exit 1 # find . -type d -mindepth 1 -maxdepth 1 -exec sh -c "cd {} && make fetch" \; > ${LOG}-$i 2>&1 & daemon sh -c "BATCH=yes NO_IGNORE=yes PACKAGE_BUILDING=yes make -k checksum" > ${LOG}-$i 2>&1 done First version (find) forked 300 processes at beginning. Second (daemon) forked 400, and, as it was going, I saw strange things, e.g. error messages in logs like that: ===> abclock Abort trap *** Error code 134 (continuing) or ===> xtu => MD5 Checksum OK for xtu-1.2.tar.gz. => SHA256 Checksum OK for xtu-1.2.tar.gz. *** Signal 6 (continuing) `checksum' not remade because of errors. Why did it happen ? Lack of resources? But I ran it as root, limits were soft, and at least that 400 processes forked! But then some anothe shit happened - system began to build and install packages without checking if they were actually installed, e.g. it made mess installing expat2 over expat-1.95 (I had to deal with it manually later), so I was forced to interrupt the script. So, what happened (why it began to install, why errors?) and how should I create local distfiles mirror ? -- Best regards, Vadim Goncharov ICQ UIN 166852181 mailto:vadim_nuclight@mail.ru
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1965362278.20060317020244>