From owner-freebsd-hackers@FreeBSD.ORG Sat Aug 9 06:45:41 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3CA331065674 for ; Sat, 9 Aug 2008 06:45:41 +0000 (UTC) (envelope-from keramida@freebsd.org) Received: from igloo.linux.gr (igloo.linux.gr [62.1.205.36]) by mx1.freebsd.org (Postfix) with ESMTP id 828AD8FC0C for ; Sat, 9 Aug 2008 06:45:40 +0000 (UTC) (envelope-from keramida@freebsd.org) Received: from kobe.laptop (adsl19-159.kln.forthnet.gr [77.49.146.159]) (authenticated bits=128) by igloo.linux.gr (8.14.3/8.14.3/Debian-5) with ESMTP id m796RXnX002366 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Sat, 9 Aug 2008 09:27:39 +0300 Received: from kobe.laptop (kobe.laptop [127.0.0.1]) by kobe.laptop (8.14.2/8.14.2) with ESMTP id m796RXiR085518; Sat, 9 Aug 2008 09:27:33 +0300 (EEST) (envelope-from keramida@freebsd.org) Received: (from keramida@localhost) by kobe.laptop (8.14.2/8.14.2/Submit) id m796RWsb085517; Sat, 9 Aug 2008 09:27:32 +0300 (EEST) (envelope-from keramida@freebsd.org) From: Giorgos Keramidas To: "Navdeep Parhar" References: Date: Sat, 09 Aug 2008 09:27:24 +0300 In-Reply-To: (Navdeep Parhar's message of "Fri, 8 Aug 2008 11:47:15 -0700") Message-ID: <87tzdu1z5f.fsf@kobe.laptop> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.0.60 (berkeley-unix) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha1; protocol="application/pgp-signature" X-MailScanner-ID: m796RXnX002366 X-Hellug-MailScanner: Found to be clean X-Hellug-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=-4.273, required 5, autolearn=not spam, ALL_TRUSTED -1.80, AWL 0.13, BAYES_00 -2.60) X-Hellug-MailScanner-From: keramida@freebsd.org X-Spam-Status: No Cc: freebsd-hackers@freebsd.org Subject: Re: mercurial working copy of FreeBSD src X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Aug 2008 06:45:41 -0000 --=-=-= Content-Transfer-Encoding: quoted-printable On Fri, 8 Aug 2008 11:47:15 -0700, "Navdeep Parhar" wro= te: > Hello everyone, > > I'm looking for the fastest way to get a full mercurial repository of > HEAD. Do you really want the *FULL* history of head? It's probably going to be in the order of a couple of hundred of MB, or even more. If you can live with an Hg repository that only includes the 2008 history of 'HEAD', then please keep reading, you may like the rest of this post :) > I tried using hgsvn and the tree at http://svn.freebsd.org/base/head > but it looks like the first hgpullsvn operation will take days (literally= ): hgpullsvn is slow. Really really *REALLY* slow... It's probably a better idea to get a recent version of the mercurial/crew branch. This includes support for the svn-1.5.X bindings, and you can convert parts of the history of the /base/head branch. I have a local hg tree with the history of the 'head' branch since 2008-01-01 on my laptop, which I periodically resync from svn with: keramida@kobe:/home/keramida$ cd /hg/bsd keramida@kobe:/hg/bsd$ ./pull-head.sh scanning source... sorting... converting... [lots of output snipped] keramida@kobe:/hg/bsd/head$ hg shortlog --limit 2 4201:99f1acdec12f | 2008-08-09 04:08 +0000 | imp: Rather than waiting a= fixed amount of time, which might not be enough 4200:d5553c08bf90 | 2008-08-09 03:54 +0000 | imp: Change -1 to 0xffffff= fful since the interface returns uint32_t. (1) A bit of background =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D I didn't really want to convert the full history of the 'head' branch From=20Subversion, because I have a local Subversion repository for that. Being able to rebase my local in-progress patches on an Hg clone _did_ seem useful at the time, so that's why I converted only the history of 'head' from the start of 2008. The initial pull was done by pulling a clean copy of the first commit of 2008 and then importing changesets on top of that with: % cd /hg/bsd % \time hg convert --config convert.svn.startrev=3D175026 \ --config convert.svn.trunk=3D'head' \ --config convert.svn.branches=3D'' \ --config convert.svn.tags=3D'' file:///home/svn/base/ head The initial pull took a bit of time, but far less than hgimportsvn. Subsequent pull-head.sh runs take literally seconds to pull from a local Subversion repository mirror. The speed is much much better than hgpullsvn, because the Hg convert extension doesn't parse the output of shell commands, but it links `natively' (through SWIG) with the subversion libraries. The conversion process is interruptible, and if you keep the 'SHA map' of the initial conversion you can keep pulling changesets into the conversion clone as many times as necessary with something like: % cd /hg/bsd % cat -n pull-head.sh 1 #!/bin/sh 2 3 hg convert \ 4 --config convert.svn.trunk=3D'head' \ 5 --config convert.svn.branches=3D'' \ 6 --config convert.svn.tags=3D'' \ 7 file:///home/svn/base/ head % You don't need the `convert.svn.startrev=3D175026' option in the second and subsequent runs, because the conversion clone already has a file called `.hg/shamap' and the conversion picks up from the place it stopped the previous time. (2) Bootstrapping your own clone =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D Everything above is of `encyclopedical' nature, because now that I've done the conversion once, you can just use a bundle file and a matching copy of my `.hg/shamap' file to bootstrap a conversion clone of your own. You can fetch an Hg bundle and a matching shamap with the history of 'head' in 2008 from: http://people.freebsd.org/~keramida/mercurial/head.hg http://people.freebsd.org/~keramida/mercurial/head.shamap The checksums of these files should be: MD5 (head.hg) =3D 4978723c560bc48c390be1634c7d36a4 MD5 (head.shamap) =3D 8e5957657fb499ed3a3575fc921e3029 SHA256 (head.hg) =3D ab073135925fdb54a2820077a5a4d75f4306d44a217b7c30d6= b8719a9a5008d4 SHA256 (head.shamap) =3D 04c6648bd108d3550ffb126c321b8faf2f5dda9f891787= bb57ae178efa271893 The head.hg bundle is around 92 MB and you can pull it directly into an empty Hg workspace: % hg init head % cd head ; hg pull /var/tmp/head.hg Then copy the `head.shamap' file in `.hg/shamap' and you are ready to go. You can start cloning this tree, and you can also run "hg convert" to incrementally pull the history of the svn/head branch, as many times as necessary. The `pull-head.sh' script I included above will make subsequent pulls less tedious by saving you the typing of all the =2D-config options to get the svn->hg branch names right. (3) Important disclaimer =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Note that this is *NOT* an official branch of the FreeBSD source tree, so if you plan to use the head.hg bundle, this email is _not_ really a promise to keep uploading `bootstrap bundles'. You are also the sole person responsible for comparing the source tree checked out from the Hg clone with a subversion or cvs checkout and making sure that it matches. So have fun if you like it, but please don't sue me or something. HTH, Giorgos --=-=-= Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkidONQACgkQ1g+UGjGGA7airwCdH/amlNwn4VT4I1YTMv6qviO7 mzQAnROJvk5ubBgPjJK/8uaTye6CVC2+ =pQjl -----END PGP SIGNATURE----- --=-=-=--