Date: Sat, 3 Dec 2016 19:16:08 +0100 From: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= <uqs@freebsd.org> To: Shawn Webb <shawn.webb@hardenedbsd.org> Cc: freebsd-git@freebsd.org Subject: Re: Git mirroring halted for freebsd-base Message-ID: <CAJ9axoRNMaBD6xfnVMbpdQZkB6Gc0nqYGuiMGBEmE2f%2B0r7G5A@mail.gmail.com> In-Reply-To: <20161203181233.GC82677@mutt-hardenedbsd> References: <CAJ9axoQH0FuaRghpc3OVTgxT42ykObTni3m9EgHsDKnKFfH7dA@mail.gmail.com> <CAJ9axoR5RFFpX3eTKxg0piLn4UidNrW9jCm-gFJS=m0xVOumzA@mail.gmail.com> <20161203172151.GA82677@mutt-hardenedbsd> <CAJ9axoSthwAEn9R-b=LdnOGL%2BoXagp8jcU_RfykxRebqLh6SYA@mail.gmail.com> <20161203172910.GB82677@mutt-hardenedbsd> <CAJ9axoRRN%2B0H2Aptipe4mGFwOjSnbq1-xVB5%2B8SF0e2%2BNC7-1w@mail.gmail.com> <20161203181233.GC82677@mutt-hardenedbsd>
next in thread | previous in thread | raw e-mail | index | archive | help
2016-12-03 19:12 GMT+01:00 Shawn Webb <shawn.webb@hardenedbsd.org>: > On Sat, Dec 03, 2016 at 06:45:32PM +0100, Ulrich Sp??rlein wrote: >> 2016-12-03 18:29 GMT+01:00 Shawn Webb <shawn.webb@hardenedbsd.org>: >> > On Sat, Dec 03, 2016 at 06:25:29PM +0100, Ulrich Sp??rlein wrote: >> >> 2016-12-03 18:21 GMT+01:00 Shawn Webb <shawn.webb@hardenedbsd.org>: >> >> > On Sat, Dec 03, 2016 at 12:42:56PM +0100, Ulrich Sp??rlein wrote: >> >> >> 2016-12-02 11:07 GMT+01:00 Ulrich Sp??rlein <uqs@freebsd.org>: >> >> >> > The conversion process started chewing up 100% cpu without making much progress, first attempts to rectify this have failed. >> >> >> > >> >> >> > The svn2git conversion and pushes to github have been halted. Pushes to bitbucket also have been halted (we're reaching the 2GB limit imposed by bitbucket). >> >> >> > >> >> >> > I'll update this thread in about 24h. >> >> >> > Uli >> >> >> >> >> >> Service is fully restored now for the github mirror, sorry for the downtime. >> >> >> Bitbucket will stop working soon because of the size limitations. >> >> >> Speak up if you require this mirror to be kept up-to-date. >> >> >> >> >> >> Cheers, >> >> >> Uli >> >> > >> >> > Looks like it might be easier for some downstream projects to fully >> >> > recreate their ports repositories from scratch than to try to merge from >> >> > upstream. >> >> >> >> What are you referring to here? >> > >> > The ports repo at https://github.com/freebsd/freebsd-ports was force >> > pushed. Now attempts at merging in upstream's ports tree into >> > hardenedbsd's causes merge conflicts for hundreds of files, including >> > files we didn't change. >> > >> > So I'm forced to either inspect hundreds of files, manually merging in >> > the changes or recreate our ports tree from scratch, re-importing our >> > changes in a single atomic commit. The second option sounds more >> > appealing, though we'd lose the entire history of our changes. >> > >> > Additionally, anyone downstream from HardenedBSD might have to do the >> > same. Domino affect. >> >> I see. This shouldn't have happend, but as svnsync is >> non-transactional, we picked up some bad SVN metadata that made it >> into ports and base repos about a year ago. The SVN corruption was >> promptly fixed (I didn't ask for this), but that now leaves us with no >> way to actually re-do the conversion from scratch, as you'd need a >> corrupted SVN repo to produce the same results. >> >> You should be able to simply merge whatever "official" commit you last >> merged to with whatever the new "official" commit is now. This only >> affected metadata, so you'll get a clean merge (no conflicts) but you >> end up depending on 2x the history for about a year or so. Shouldn't >> be that much of a problem. Ask your local git wizard on how to do this >> best. >> >> >> > What caused the issue? What is going to be done to prevent it from >> >> > happening again? >> >> >> >> I have no root cause, other than bitbucket changing permissions and >> >> somehow git ending up using 100% CPU for most of the operations. >> > >> > So no guarantees this massive screw-up won't happen again? >> >> I said this before, and I'll say it again. This is a best-effort >> conversion and we're at the mercy of whatever SVN fucks up next. I >> provided clear instructions as to how to do the conversion in-house, >> and guess how many people actually wrote to me that they end up with >> different SHA hashes on github than they can produce in-house for both >> src and ports? >> >> What would be your guess? >> >> Exactly, 0 people have done the in-house conversion and have compared >> this to github. I could have put all kinds of backdoors in FreeBSD on >> github and not a single soul would've noticed. >> >> So if you depend on it, I would very much appreciate if you could do >> the same conversion in-house and report any drift as soon as possible, >> because it's a mess otherwise, as you can see. >> >> Any thoughts on how to fix this for src would also be appreciated, all >> I can think of is either pushing 2 heads and telling people to >> migrate, or doing the switchover on a flag day. > > Hey Uli, > > Sorry for the harsher tone earlier. I'm a bit stressed and it was unfair > of me to use that tone. > > I'm grateful for your efforts. I understand that supporting git isn't an > official service provided by FreeBSD. One item on my Christmas wishlst > would be to have official support for a read-only git mirror of the > various FreeBSD projects (mainly src and ports). > No worries, I feel your pain. My Friday evening was ruined too :( If you have the resources, *please* try a conversion run of all three repos too and report your findings. We need to make sure that the conversion is now more robust and we need to detect drift earlier :/ Cheers, Uli
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ9axoRNMaBD6xfnVMbpdQZkB6Gc0nqYGuiMGBEmE2f%2B0r7G5A>