From owner-freebsd-git@freebsd.org Sat Dec 3 18:12:37 2016 Return-Path: Delivered-To: freebsd-git@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C5DBAC65C5B for ; Sat, 3 Dec 2016 18:12:37 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-qk0-x234.google.com (mail-qk0-x234.google.com [IPv6:2607:f8b0:400d:c09::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 788B4633 for ; Sat, 3 Dec 2016 18:12:37 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-qk0-x234.google.com with SMTP id x190so309983576qkb.0 for ; Sat, 03 Dec 2016 10:12:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=JCrsbZLY6OD/8cnnLEAgxMscnEtcJWC3nXDzJvBIJfw=; b=lY6UCjssXVbqvv1fMt4AGQg6iqLFYrEml06v1AaDiE38dN+tWPyvSFLuVBr2ZwPsGp 8TUobRAfW8v5Q3RPO1f/NgvjzoT9Fc6b+mk5F0xKzc4HLXt7z6tNObWwbYYO1Xl97p56 vaQPZH3lblu8RGIRho74hrbGlitHIiubBkGqyJW+pzuipmN282fAggla3LACkkj6yIOJ eaF+hXlywJIea2dvndN/b/5fLGj6H4YZLMcZsRhYHO7BY8t9ZCtcHhr+2nSN6etiPdN4 6XCxfoy5BXcuzTJlS5y28X7RVFcisiYad75QeMKC48sXD/qjwUuh3+jhfqWIQduuj8wy k+pw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=JCrsbZLY6OD/8cnnLEAgxMscnEtcJWC3nXDzJvBIJfw=; b=WTzUWrQpeBQ0PfOEBIqbUxwR+zBu0jZw0GrwAcwQnX0lKk8pRi+YPyfCqDAMnoVXQ2 LXpY1TZ25F+cO1GhyN5zJhwYOX82LYL/Vkk77I1Y39Td19YVGlNISnM4FK2lVw26wNvo JXWjePAsGgAPpJUz0uC35m4+BQaEyM/kGfam+p0s7alCu92ZovIufpAsHvfrE2pXY2Cm NxdhPFdsUqzs0Cu4PbBGwpGjRBcB0u6JB0Tw/7fBw1YPX+VBoRBSn95Gltjfo5wasUUB 0+8Ul3tv0k59aRmFQnqSr/3HOWykOiiXFvYlsNUCbRF70SRLOV40wuzVxKHr4OBsp/Bg dQgA== X-Gm-Message-State: AKaTC00OOloaspRzVc5iRD7jrqDtgAh8pVUN32Y8DHzEC8YHU1pqLNYWt8ovzwNDnTyeal2T X-Received: by 10.55.78.147 with SMTP id c141mr49283118qkb.4.1480788756429; Sat, 03 Dec 2016 10:12:36 -0800 (PST) Received: from mutt-hardenedbsd (pool-100-16-218-231.bltmmd.fios.verizon.net. [100.16.218.231]) by smtp.gmail.com with ESMTPSA id y23sm5395417qta.40.2016.12.03.10.12.35 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 03 Dec 2016 10:12:35 -0800 (PST) Date: Sat, 3 Dec 2016 13:12:33 -0500 From: Shawn Webb To: Ulrich Sp??rlein Cc: freebsd-git@freebsd.org Subject: Re: Git mirroring halted for freebsd-base Message-ID: <20161203181233.GC82677@mutt-hardenedbsd> References: <20161203172151.GA82677@mutt-hardenedbsd> <20161203172910.GB82677@mutt-hardenedbsd> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="pAwQNkOnpTn9IO2O" Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD mutt-hardenedbsd 12.0-CURRENT-HBSD FreeBSD 12.0-CURRENT-HBSD X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x6A84658F52456EEE User-Agent: Mutt/1.6.1 (2016-04-27) X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Dec 2016 18:12:37 -0000 --pAwQNkOnpTn9IO2O Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Dec 03, 2016 at 06:45:32PM +0100, Ulrich Sp??rlein wrote: > 2016-12-03 18:29 GMT+01:00 Shawn Webb : > > On Sat, Dec 03, 2016 at 06:25:29PM +0100, Ulrich Sp??rlein wrote: > >> 2016-12-03 18:21 GMT+01:00 Shawn Webb : > >> > On Sat, Dec 03, 2016 at 12:42:56PM +0100, Ulrich Sp??rlein wrote: > >> >> 2016-12-02 11:07 GMT+01:00 Ulrich Sp??rlein : > >> >> > The conversion process started chewing up 100% cpu without making= much progress, first attempts to rectify this have failed. > >> >> > > >> >> > The svn2git conversion and pushes to github have been halted. Pus= hes to bitbucket also have been halted (we're reaching the 2GB limit impose= d by bitbucket). > >> >> > > >> >> > I'll update this thread in about 24h. > >> >> > Uli > >> >> > >> >> Service is fully restored now for the github mirror, sorry for the = downtime. > >> >> Bitbucket will stop working soon because of the size limitations. > >> >> Speak up if you require this mirror to be kept up-to-date. > >> >> > >> >> Cheers, > >> >> Uli > >> > > >> > Looks like it might be easier for some downstream projects to fully > >> > recreate their ports repositories from scratch than to try to merge = =66rom > >> > upstream. > >> > >> What are you referring to here? > > > > The ports repo at https://github.com/freebsd/freebsd-ports was force > > pushed. Now attempts at merging in upstream's ports tree into > > hardenedbsd's causes merge conflicts for hundreds of files, including > > files we didn't change. > > > > So I'm forced to either inspect hundreds of files, manually merging in > > the changes or recreate our ports tree from scratch, re-importing our > > changes in a single atomic commit. The second option sounds more > > appealing, though we'd lose the entire history of our changes. > > > > Additionally, anyone downstream from HardenedBSD might have to do the > > same. Domino affect. >=20 > I see. This shouldn't have happend, but as svnsync is > non-transactional, we picked up some bad SVN metadata that made it > into ports and base repos about a year ago. The SVN corruption was > promptly fixed (I didn't ask for this), but that now leaves us with no > way to actually re-do the conversion from scratch, as you'd need a > corrupted SVN repo to produce the same results. >=20 > You should be able to simply merge whatever "official" commit you last > merged to with whatever the new "official" commit is now. This only > affected metadata, so you'll get a clean merge (no conflicts) but you > end up depending on 2x the history for about a year or so. Shouldn't > be that much of a problem. Ask your local git wizard on how to do this > best. >=20 > >> > What caused the issue? What is going to be done to prevent it from > >> > happening again? > >> > >> I have no root cause, other than bitbucket changing permissions and > >> somehow git ending up using 100% CPU for most of the operations. > > > > So no guarantees this massive screw-up won't happen again? >=20 > I said this before, and I'll say it again. This is a best-effort > conversion and we're at the mercy of whatever SVN fucks up next. I > provided clear instructions as to how to do the conversion in-house, > and guess how many people actually wrote to me that they end up with > different SHA hashes on github than they can produce in-house for both > src and ports? >=20 > What would be your guess? >=20 > Exactly, 0 people have done the in-house conversion and have compared > this to github. I could have put all kinds of backdoors in FreeBSD on > github and not a single soul would've noticed. >=20 > So if you depend on it, I would very much appreciate if you could do > the same conversion in-house and report any drift as soon as possible, > because it's a mess otherwise, as you can see. >=20 > Any thoughts on how to fix this for src would also be appreciated, all > I can think of is either pushing 2 heads and telling people to > migrate, or doing the switchover on a flag day. Hey Uli, Sorry for the harsher tone earlier. I'm a bit stressed and it was unfair of me to use that tone. I'm grateful for your efforts. I understand that supporting git isn't an official service provided by FreeBSD. One item on my Christmas wishlst would be to have official support for a read-only git mirror of the various FreeBSD projects (mainly src and ports). Thanks, --=20 Shawn Webb Cofounder and Security Engineer HardenedBSD GPG Key ID: 0x6A84658F52456EEE GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE --pAwQNkOnpTn9IO2O Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJYQwsPAAoJEGqEZY9SRW7uIEYQAMXxWAhapwBqnIi2VCxt8nav cbcwIzIqZy9bbk9BQ5Va9NSFUZNsM89vZe2cq9Surdzb+ydxaYkQLuEXXhIkaNYc DDHfJZ//SyeEj7zFm0jDfLBv8OyrZzmNPSlWTcBJGr8mPK8nny+Yrj850HObnH7v w/cbGI5X5zYQr5JmtU+MNXcaWVmoSVLjmwxj84mnzcrzSX4mNH7vKRpcGB2I/rfa bPOhe9mm9ap2mYr6vckOxahrWRSykLkFHOeMiERQO6XOPeGnvZvdX8mENowVkX6U W3VGRyCyl9zlkd+y6YVWymYWJOcNakY1/eZgFmTOqVoMSA27xZvE7g7qwfXn2cSo lpgXUTnBbdYykpY5+cEuzTTnXk5RdO7fDUqhGmMpTscM/VkxpP9RL+/HNSm2KvYX bcuZF05CvqNIen85Bwhxi0pjrgCLqKnzsF4eIiXhaqhueqOUlJO4U7USLeM+YVw/ 5MWaWS/ZAiQlI3/Xw56h9ymRtiGA6A64NTkaF9w/d9M86vPwMX8KtXdQwCD/TYP5 koiUBWbHBRWPTZz8EI6/hwvWut7f1CUMmyTgNpHze5D/PIPuHdTGh79W57rUuDsg 4CY5BXKnOAqfhI79V9JDzLel39Kl/UqIt76TKtuw/wc1zooHBQMLbx0fITYlyaXP cRCLpEbBMewS2JRFjldr =dfpg -----END PGP SIGNATURE----- --pAwQNkOnpTn9IO2O--