From owner-svn-src-user@freebsd.org Fri Dec 2 16:43:17 2016 Return-Path: Delivered-To: svn-src-user@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8B86CC630DF for ; Fri, 2 Dec 2016 16:43:17 +0000 (UTC) (envelope-from uqs@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6627A1204; Fri, 2 Dec 2016 16:43:17 +0000 (UTC) (envelope-from uqs@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id uB2GhGnP012772; Fri, 2 Dec 2016 16:43:16 GMT (envelope-from uqs@FreeBSD.org) Received: (from uqs@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id uB2GhGwg012771; Fri, 2 Dec 2016 16:43:16 GMT (envelope-from uqs@FreeBSD.org) Message-Id: <201612021643.uB2GhGwg012771@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: uqs set sender to uqs@FreeBSD.org using -f From: Ulrich Spoerlein Date: Fri, 2 Dec 2016 16:43:16 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org Subject: svn commit: r309420 - user/uqs/git_conv X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2016 16:43:17 -0000 Author: uqs Date: Fri Dec 2 16:43:16 2016 New Revision: 309420 URL: https://svnweb.freebsd.org/changeset/base/309420 Log: Add svn sync script copied from the cluster machines. Turns out svnsync is non-atomic and we now have bad metadata in the git conversion as export on Github. Ceterum censeo SVN esse delendam Added: user/uqs/git_conv/dosync.sh (contents, props changed) Added: user/uqs/git_conv/dosync.sh ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ user/uqs/git_conv/dosync.sh Fri Dec 2 16:43:16 2016 (r309420) @@ -0,0 +1,99 @@ +#! /bin/sh +HOME=/home/svnmir +export HOME +PATH=/usr/local/bin:$PATH +export PATH + +# svnsync is non-atomic. Every commit is done in two distinct steps. +# 1: the commit is replayed as a transaction. +# 2: after the commit, the metadata is copied as a *separate* transaction. +# This makes svnsync -> svnsync chains annoying as there is a brief window +# between #1 and #2 where the repo is unlocked and bogus metadata is +# visible. (!!!) +# As a workaround, recheck the svn:author property of the last few +# revisions and re-copy as needed. This reduces the window to a few +# seconds. Still not good but better than no recovery at all. + +#MAILTO=clusteradm-updates@FreeBSD.org +#@reboot /usr/sbin/daemon -cf lockf -s -t 0 /home/svnmir/dosync.lock /home/svnmir/dosync.sh +#* * * * * /usr/sbin/daemon -cf lockf -s -t 0 /home/svnmir/dosync.lock /home/svnmir/dosync.sh +#7 * * * * /home/svnmir/update.sh + +usage() { + echo "Usage: $0 [-1] [-l logdir ] [-r repodirs] [-s setlist]" 1>&2 + exit 1 +} + +umask 002 +me=$(id -un) + +once=false +logdir=/var/log +repodirs=/home/svn +setlist="" + +while getopts "1l:r:s:" _opt; do + case "$_opt" in + 1) once=true ;; + l) logdir="$OPTARG" ;; + r) repodirs="$OPTARG" ;; + s) setlist="$OPTARG" ;; + *) usage ;; +esac +done +shift $(($OPTIND - 1)) +if [ $# -ne 0 ]; then + usage +fi + +if [ -z "${setlist}" ]; then + for r in base doc ports socsvn; do + if [ -d ${repodirs}/$r ]; then + setlist="${setlist} $r" + fi + done +fi + +# Only ever run this under the dosync.lock +for r in ${setlist}; do + echo "Startup: $(date)" >> ${logdir}/svnsync-$r.log +done + +t0=$(date +%s) + +# Pack repo once at startup +for r in ${setlist}; do + svnadmin pack ${repodirs}/$r >> ${logdir}/svnsync-$r.log 2>&1 +done + +while : +do + for r in ${setlist}; do + locked=$(svn propget --revprop -r 0 svn:sync-lock file://${repodirs}/$r 2>/dev/null) + if [ -n "$locked" ]; then + svn propdel --revprop -r 0 svn:sync-lock file://${repodirs}/$r >>${logdir}/svnsync-$r.log 2>&1 + fi + oldrev=$(svn propget --revprop -r 0 svn:sync-last-merged-rev file://${repodirs}/$r 2>>${logdir}/svnsync-$r.log) + svnsync --non-interactive sync file://${repodirs}/$r >> ${logdir}/svnsync-$r.log 2>&1 + newrev=$(svn propget --revprop -r 0 svn:sync-last-merged-rev file://${repodirs}/$r 2>>${logdir}/svnsync-$r.log) + oldrev=$(($oldrev - 10)) + numrev=$(($newrev - $oldrev + 1)) + for rev in $(jot $numrev $oldrev); do + a=$(svn propget --revprop -r $rev svn:author file://${repodirs}/$r 2>>${logdir}/svnsync-$r.log) + if [ "x$a" == "x$me" ]; then + svnsync copy-revprops -r $rev file://${repodirs}/$r >> ${logdir}/svnsync-$r.log 2>&1 + fi + done + done + if $once; then + exit 0 + fi + # exit and restart once an hour in case of a script update + now=$(date +%s) + elapsed=$(( $now - $t0 )) + # slightly less than 1 hour so we catch the cron restart asap + if [ ${elapsed} -gt 3590 ]; then + exit 0 + fi + sleep 5 +done From owner-svn-src-user@freebsd.org Fri Dec 2 16:49:27 2016 Return-Path: Delivered-To: svn-src-user@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0E954C63175 for ; Fri, 2 Dec 2016 16:49:27 +0000 (UTC) (envelope-from uqs@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BD8A213BA; Fri, 2 Dec 2016 16:49:26 +0000 (UTC) (envelope-from uqs@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id uB2GnP4D013065; Fri, 2 Dec 2016 16:49:25 GMT (envelope-from uqs@FreeBSD.org) Received: (from uqs@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id uB2GnPcL013059; Fri, 2 Dec 2016 16:49:25 GMT (envelope-from uqs@FreeBSD.org) Message-Id: <201612021649.uB2GnPcL013059@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: uqs set sender to uqs@FreeBSD.org using -f From: Ulrich Spoerlein Date: Fri, 2 Dec 2016 16:49:25 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org Subject: svn commit: r309421 - in user/uqs/git_conv: base-head.git base-head.git/.git doc-head.git/.git freebsd-base.git freebsd-doc.git freebsd-ports.git ports-head.git/.git src-head.git/.git X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2016 16:49:27 -0000 Author: uqs Date: Fri Dec 2 16:49:25 2016 New Revision: 309421 URL: https://svnweb.freebsd.org/changeset/base/309421 Log: Update git configs to reflect reality - gitorious and code.google.com are gone - only push master and stable/* to bitbucket, we're close to the 2GB limit (essentially, the bitbucket mirror will stop working very soon) - rename "src" to "base" to match the ill designed SVN repo names - use fake github hostnames, as we no longer can use the same SSH key for multiple repos (wtf?) Added: user/uqs/git_conv/base-head.git/ user/uqs/git_conv/base-head.git/.git/ user/uqs/git_conv/base-head.git/.git/config - copied, changed from r309420, user/uqs/git_conv/src-head.git/.git/config Deleted: user/uqs/git_conv/src-head.git/.git/config Modified: user/uqs/git_conv/doc-head.git/.git/config user/uqs/git_conv/freebsd-base.git/config user/uqs/git_conv/freebsd-doc.git/config user/uqs/git_conv/freebsd-ports.git/config user/uqs/git_conv/ports-head.git/.git/config Copied and modified: user/uqs/git_conv/base-head.git/.git/config (from r309420, user/uqs/git_conv/src-head.git/.git/config) ============================================================================== --- user/uqs/git_conv/src-head.git/.git/config Fri Dec 2 16:43:16 2016 (r309420, copy source) +++ user/uqs/git_conv/base-head.git/.git/config Fri Dec 2 16:49:25 2016 (r309421) @@ -7,12 +7,8 @@ rewriteRoot = svn+ssh://svn.freebsd.org/base url = file:///home/svn/base fetch = head:refs/remotes/trunk +[push] + default = matching [remote "github"] url = src.github.com:freebsd/freebsd.git push = +refs/remotes/trunk:refs/heads/svn_head -[remote "gitorious"] - url = gitorious.org:freebsd/freebsd-head.git - push = +refs/heads/master:refs/heads/master -[remote "googlecode"] - url = https://code.google.com/p/freebsd-head - push = +refs/heads/master:refs/heads/master Modified: user/uqs/git_conv/doc-head.git/.git/config ============================================================================== --- user/uqs/git_conv/doc-head.git/.git/config Fri Dec 2 16:43:16 2016 (r309420) +++ user/uqs/git_conv/doc-head.git/.git/config Fri Dec 2 16:49:25 2016 (r309421) @@ -8,6 +8,8 @@ url = file:///home/svn/doc fetch = head:refs/remotes/trunk branches = {www}:refs/remotes/* +[push] + default = matching [remote "github"] url = doc.github.com:freebsd/freebsd-doc.git push = +refs/remotes/trunk:refs/heads/svn_head Modified: user/uqs/git_conv/freebsd-base.git/config ============================================================================== --- user/uqs/git_conv/freebsd-base.git/config Fri Dec 2 16:43:16 2016 (r309420) +++ user/uqs/git_conv/freebsd-base.git/config Fri Dec 2 16:49:25 2016 (r309421) @@ -2,21 +2,19 @@ repositoryformatversion = 0 filemode = true bare = true +[push] + default = matching [remote "github"] - url = src.github.com:freebsd/freebsd.git + url = base.github.com:freebsd/freebsd.git push = +refs/heads/master:refs/heads/master + push = +refs/heads/projects/*:refs/heads/projects/* push = +refs/heads/release/*:refs/heads/release/* push = +refs/heads/releng/*:refs/heads/releng/* push = +refs/heads/stable/*:refs/heads/stable/* - push = +refs/heads/projects/*:refs/heads/projects/* push = +refs/heads/user/*:refs/heads/user/* push = +refs/notes/*:refs/notes/* [remote "bitbucket"] url = ssh://git@bitbucket.org/freebsd/freebsd-base.git push = +refs/heads/master:refs/heads/master - push = +refs/heads/release/*:refs/heads/release/* - push = +refs/heads/releng/*:refs/heads/releng/* push = +refs/heads/stable/*:refs/heads/stable/* - push = +refs/heads/projects/*:refs/heads/projects/* - push = +refs/heads/user/*:refs/heads/user/* push = +refs/notes/*:refs/notes/* Modified: user/uqs/git_conv/freebsd-doc.git/config ============================================================================== --- user/uqs/git_conv/freebsd-doc.git/config Fri Dec 2 16:43:16 2016 (r309420) +++ user/uqs/git_conv/freebsd-doc.git/config Fri Dec 2 16:49:25 2016 (r309421) @@ -2,9 +2,11 @@ repositoryformatversion = 0 filemode = true bare = true +[push] + default = matching [remote "github"] url = doc.github.com:freebsd/freebsd-doc.git push = +refs/heads/master:refs/heads/master - push = +refs/heads/release/*:refs/heads/release/* push = +refs/heads/projects/*:refs/heads/projects/* + push = +refs/heads/release/*:refs/heads/release/* push = +refs/notes/*:refs/notes/* Modified: user/uqs/git_conv/freebsd-ports.git/config ============================================================================== --- user/uqs/git_conv/freebsd-ports.git/config Fri Dec 2 16:43:16 2016 (r309420) +++ user/uqs/git_conv/freebsd-ports.git/config Fri Dec 2 16:49:25 2016 (r309421) @@ -2,9 +2,11 @@ repositoryformatversion = 0 filemode = true bare = true +[push] + default = matching [remote "github"] url = ports.github.com:freebsd/freebsd-ports.git - push = +refs/heads/master:refs/heads/master push = +refs/heads/branches/*:refs/heads/branches/* + push = +refs/heads/master:refs/heads/master push = +refs/heads/projects/*:refs/heads/projects/* push = +refs/notes/*:refs/notes/* Modified: user/uqs/git_conv/ports-head.git/.git/config ============================================================================== --- user/uqs/git_conv/ports-head.git/.git/config Fri Dec 2 16:43:16 2016 (r309420) +++ user/uqs/git_conv/ports-head.git/.git/config Fri Dec 2 16:49:25 2016 (r309421) @@ -7,6 +7,8 @@ rewriteRoot = svn+ssh://svn.freebsd.org/ports url = file:///home/svn/ports fetch = head:refs/remotes/trunk +[push] + default = matching [remote "github"] url = ports.github.com:freebsd/freebsd-ports.git push = +refs/remotes/trunk:refs/heads/svn_head From owner-svn-src-user@freebsd.org Fri Dec 2 16:51:10 2016 Return-Path: Delivered-To: svn-src-user@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5403CC6320A for ; Fri, 2 Dec 2016 16:51:10 +0000 (UTC) (envelope-from uqs@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 085A9160A; Fri, 2 Dec 2016 16:51:09 +0000 (UTC) (envelope-from uqs@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id uB2Gp9al014171; Fri, 2 Dec 2016 16:51:09 GMT (envelope-from uqs@FreeBSD.org) Received: (from uqs@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id uB2Gp9L4014170; Fri, 2 Dec 2016 16:51:09 GMT (envelope-from uqs@FreeBSD.org) Message-Id: <201612021651.uB2Gp9L4014170@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: uqs set sender to uqs@FreeBSD.org using -f From: Ulrich Spoerlein Date: Fri, 2 Dec 2016 16:51:09 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org Subject: svn commit: r309422 - user/uqs/git_conv/src-head.git X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2016 16:51:10 -0000 Author: uqs Date: Fri Dec 2 16:51:09 2016 New Revision: 309422 URL: https://svnweb.freebsd.org/changeset/base/309422 Log: Sigh, remove empty directories explicitly. Ceterum censeo SVN esse delendam Deleted: user/uqs/git_conv/src-head.git/ From owner-svn-src-user@freebsd.org Fri Dec 2 17:06:38 2016 Return-Path: Delivered-To: svn-src-user@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B1498C63B94 for ; Fri, 2 Dec 2016 17:06:38 +0000 (UTC) (envelope-from uqs@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 67FE81AAE; Fri, 2 Dec 2016 17:06:38 +0000 (UTC) (envelope-from uqs@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id uB2H6bhv022334; Fri, 2 Dec 2016 17:06:37 GMT (envelope-from uqs@FreeBSD.org) Received: (from uqs@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id uB2H6bSq022333; Fri, 2 Dec 2016 17:06:37 GMT (envelope-from uqs@FreeBSD.org) Message-Id: <201612021706.uB2H6bSq022333@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: uqs set sender to uqs@FreeBSD.org using -f From: Ulrich Spoerlein Date: Fri, 2 Dec 2016 17:06:37 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org Subject: svn commit: r309423 - user/uqs/git_conv/base-head.git X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2016 17:06:38 -0000 Author: uqs Date: Fri Dec 2 17:06:37 2016 New Revision: 309423 URL: https://svnweb.freebsd.org/changeset/base/309423 Log: Ignore everything but the git config in this dir. Ceterum censeo SVN esse delendam Modified: Directory Properties: user/uqs/git_conv/base-head.git/ (props changed) From owner-svn-src-user@freebsd.org Fri Dec 2 17:08:29 2016 Return-Path: Delivered-To: svn-src-user@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 26F05C63BEF for ; Fri, 2 Dec 2016 17:08:29 +0000 (UTC) (envelope-from uqs@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D139C1C08; Fri, 2 Dec 2016 17:08:28 +0000 (UTC) (envelope-from uqs@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id uB2H8Su7022444; Fri, 2 Dec 2016 17:08:28 GMT (envelope-from uqs@FreeBSD.org) Received: (from uqs@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id uB2H8SOn022443; Fri, 2 Dec 2016 17:08:28 GMT (envelope-from uqs@FreeBSD.org) Message-Id: <201612021708.uB2H8SOn022443@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: uqs set sender to uqs@FreeBSD.org using -f From: Ulrich Spoerlein Date: Fri, 2 Dec 2016 17:08:28 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org Subject: svn commit: r309424 - user/uqs/git_conv/base-head.git X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2016 17:08:29 -0000 Author: uqs Date: Fri Dec 2 17:08:27 2016 New Revision: 309424 URL: https://svnweb.freebsd.org/changeset/base/309424 Log: You got to be kidding me ... Modified: Directory Properties: user/uqs/git_conv/base-head.git/ (props changed)