Date: Sat, 3 Dec 2016 13:12:33 -0500 From: Shawn Webb <shawn.webb@hardenedbsd.org> To: Ulrich Sp??rlein <uqs@freebsd.org> Cc: freebsd-git@freebsd.org Subject: Re: Git mirroring halted for freebsd-base Message-ID: <20161203181233.GC82677@mutt-hardenedbsd> In-Reply-To: <CAJ9axoRRN%2B0H2Aptipe4mGFwOjSnbq1-xVB5%2B8SF0e2%2BNC7-1w@mail.gmail.com> References: <CAJ9axoQH0FuaRghpc3OVTgxT42ykObTni3m9EgHsDKnKFfH7dA@mail.gmail.com> <CAJ9axoR5RFFpX3eTKxg0piLn4UidNrW9jCm-gFJS=m0xVOumzA@mail.gmail.com> <20161203172151.GA82677@mutt-hardenedbsd> <CAJ9axoSthwAEn9R-b=LdnOGL%2BoXagp8jcU_RfykxRebqLh6SYA@mail.gmail.com> <20161203172910.GB82677@mutt-hardenedbsd> <CAJ9axoRRN%2B0H2Aptipe4mGFwOjSnbq1-xVB5%2B8SF0e2%2BNC7-1w@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--pAwQNkOnpTn9IO2O Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Dec 03, 2016 at 06:45:32PM +0100, Ulrich Sp??rlein wrote: > 2016-12-03 18:29 GMT+01:00 Shawn Webb <shawn.webb@hardenedbsd.org>: > > On Sat, Dec 03, 2016 at 06:25:29PM +0100, Ulrich Sp??rlein wrote: > >> 2016-12-03 18:21 GMT+01:00 Shawn Webb <shawn.webb@hardenedbsd.org>: > >> > On Sat, Dec 03, 2016 at 12:42:56PM +0100, Ulrich Sp??rlein wrote: > >> >> 2016-12-02 11:07 GMT+01:00 Ulrich Sp??rlein <uqs@freebsd.org>: > >> >> > The conversion process started chewing up 100% cpu without making= much progress, first attempts to rectify this have failed. > >> >> > > >> >> > The svn2git conversion and pushes to github have been halted. Pus= hes to bitbucket also have been halted (we're reaching the 2GB limit impose= d by bitbucket). > >> >> > > >> >> > I'll update this thread in about 24h. > >> >> > Uli > >> >> > >> >> Service is fully restored now for the github mirror, sorry for the = downtime. > >> >> Bitbucket will stop working soon because of the size limitations. > >> >> Speak up if you require this mirror to be kept up-to-date. > >> >> > >> >> Cheers, > >> >> Uli > >> > > >> > Looks like it might be easier for some downstream projects to fully > >> > recreate their ports repositories from scratch than to try to merge = =66rom > >> > upstream. > >> > >> What are you referring to here? > > > > The ports repo at https://github.com/freebsd/freebsd-ports was force > > pushed. Now attempts at merging in upstream's ports tree into > > hardenedbsd's causes merge conflicts for hundreds of files, including > > files we didn't change. > > > > So I'm forced to either inspect hundreds of files, manually merging in > > the changes or recreate our ports tree from scratch, re-importing our > > changes in a single atomic commit. The second option sounds more > > appealing, though we'd lose the entire history of our changes. > > > > Additionally, anyone downstream from HardenedBSD might have to do the > > same. Domino affect. >=20 > I see. This shouldn't have happend, but as svnsync is > non-transactional, we picked up some bad SVN metadata that made it > into ports and base repos about a year ago. The SVN corruption was > promptly fixed (I didn't ask for this), but that now leaves us with no > way to actually re-do the conversion from scratch, as you'd need a > corrupted SVN repo to produce the same results. >=20 > You should be able to simply merge whatever "official" commit you last > merged to with whatever the new "official" commit is now. This only > affected metadata, so you'll get a clean merge (no conflicts) but you > end up depending on 2x the history for about a year or so. Shouldn't > be that much of a problem. Ask your local git wizard on how to do this > best. >=20 > >> > What caused the issue? What is going to be done to prevent it from > >> > happening again? > >> > >> I have no root cause, other than bitbucket changing permissions and > >> somehow git ending up using 100% CPU for most of the operations. > > > > So no guarantees this massive screw-up won't happen again? >=20 > I said this before, and I'll say it again. This is a best-effort > conversion and we're at the mercy of whatever SVN fucks up next. I > provided clear instructions as to how to do the conversion in-house, > and guess how many people actually wrote to me that they end up with > different SHA hashes on github than they can produce in-house for both > src and ports? >=20 > What would be your guess? >=20 > Exactly, 0 people have done the in-house conversion and have compared > this to github. I could have put all kinds of backdoors in FreeBSD on > github and not a single soul would've noticed. >=20 > So if you depend on it, I would very much appreciate if you could do > the same conversion in-house and report any drift as soon as possible, > because it's a mess otherwise, as you can see. >=20 > Any thoughts on how to fix this for src would also be appreciated, all > I can think of is either pushing 2 heads and telling people to > migrate, or doing the switchover on a flag day. Hey Uli, Sorry for the harsher tone earlier. I'm a bit stressed and it was unfair of me to use that tone. I'm grateful for your efforts. I understand that supporting git isn't an official service provided by FreeBSD. One item on my Christmas wishlst would be to have official support for a read-only git mirror of the various FreeBSD projects (mainly src and ports). Thanks, --=20 Shawn Webb Cofounder and Security Engineer HardenedBSD GPG Key ID: 0x6A84658F52456EEE GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE --pAwQNkOnpTn9IO2O Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJYQwsPAAoJEGqEZY9SRW7uIEYQAMXxWAhapwBqnIi2VCxt8nav cbcwIzIqZy9bbk9BQ5Va9NSFUZNsM89vZe2cq9Surdzb+ydxaYkQLuEXXhIkaNYc DDHfJZ//SyeEj7zFm0jDfLBv8OyrZzmNPSlWTcBJGr8mPK8nny+Yrj850HObnH7v w/cbGI5X5zYQr5JmtU+MNXcaWVmoSVLjmwxj84mnzcrzSX4mNH7vKRpcGB2I/rfa bPOhe9mm9ap2mYr6vckOxahrWRSykLkFHOeMiERQO6XOPeGnvZvdX8mENowVkX6U W3VGRyCyl9zlkd+y6YVWymYWJOcNakY1/eZgFmTOqVoMSA27xZvE7g7qwfXn2cSo lpgXUTnBbdYykpY5+cEuzTTnXk5RdO7fDUqhGmMpTscM/VkxpP9RL+/HNSm2KvYX bcuZF05CvqNIen85Bwhxi0pjrgCLqKnzsF4eIiXhaqhueqOUlJO4U7USLeM+YVw/ 5MWaWS/ZAiQlI3/Xw56h9ymRtiGA6A64NTkaF9w/d9M86vPwMX8KtXdQwCD/TYP5 koiUBWbHBRWPTZz8EI6/hwvWut7f1CUMmyTgNpHze5D/PIPuHdTGh79W57rUuDsg 4CY5BXKnOAqfhI79V9JDzLel39Kl/UqIt76TKtuw/wc1zooHBQMLbx0fITYlyaXP cRCLpEbBMewS2JRFjldr =dfpg -----END PGP SIGNATURE----- --pAwQNkOnpTn9IO2O--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20161203181233.GC82677>