From owner-freebsd-git@freebsd.org Sat Dec 3 18:16:11 2016 Return-Path: Delivered-To: freebsd-git@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 763D7C65D33 for ; Sat, 3 Dec 2016 18:16:11 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: from mail-lf0-x244.google.com (mail-lf0-x244.google.com [IPv6:2a00:1450:4010:c07::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EE97E842 for ; Sat, 3 Dec 2016 18:16:10 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: by mail-lf0-x244.google.com with SMTP id o141so24974674lff.1 for ; Sat, 03 Dec 2016 10:16:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=/JuyaC0pFM27BemYGlwg+1vWmGnRZ49VRHeXmK65pY8=; b=AOZ4ESiFotEIdlc4ik2ufbkSc/oXs+WuSL453qED/d2e/1/sABEU23+5PwaQLPGx5U 7KHTF9BEOFkhDmx3mL61KEQ0JScKc0O3wU4K2u0NES2eNcck2xeAvhaF/kZhI1/rKoIX YzQVgzhhvS4msnJqw6p8tBHGrFwxiBu8n2AsyPRkIJTjLeKvArMI9sK6kfgz/dhAiZWs MvgOQX7ea6fsQ8RBb4JjJ8MoWhtkZK83PCW/cYzc8b8bjXxr3na6zX6lJkyPh8EIQKOc TGAsniXuiKEcFLq1xmakMHFP3CiJlMuob/taZpORoYiofpSmBfYjVt7oZuK/zSEiUP1L qm+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=/JuyaC0pFM27BemYGlwg+1vWmGnRZ49VRHeXmK65pY8=; b=fX1nJdmIkWZPkrhkYdY4z0la+7MTE4g8i2eZH//XKi8M8Ioex36CsYV7EBjpWTwr+P 4lvHN189GcJGge0OZMp6x+Fzg44vNhB23OV0Ozoy/a6FPGsCIXLvn9rlhtYsOCENxULL QzVXpAEjkLLVxlHhNL964Ak9cJ4CWh42ufCBqfW7KH6CvaVq9fcG6X7nVKqd9Igo1ri6 v7uussFy8Y/CXmhRisZMdDojotbGE5ORhDf52Klao067ajjqDUinZJzGcuikXrnYp8jp Y/CZNqTV6vl6wEZVr8Z8VhBvWjz93a3mDo7W9hzhJSAeQN9DZbmKNqgS/0D4Ga0YJ9Cg t10g== X-Gm-Message-State: AKaTC00HY0DhJI9rHX6FqmraFTCPNWHYHqSQ4frgxSffcaXNNwonZW06/8ruz975USi1tWtU2DNqGjOXDXgEzA== X-Received: by 10.46.0.229 with SMTP id e98mr25610569lji.33.1480788968816; Sat, 03 Dec 2016 10:16:08 -0800 (PST) MIME-Version: 1.0 Sender: uspoerlein@gmail.com Received: by 10.25.196.140 with HTTP; Sat, 3 Dec 2016 10:16:08 -0800 (PST) In-Reply-To: <20161203181233.GC82677@mutt-hardenedbsd> References: <20161203172151.GA82677@mutt-hardenedbsd> <20161203172910.GB82677@mutt-hardenedbsd> <20161203181233.GC82677@mutt-hardenedbsd> From: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Date: Sat, 3 Dec 2016 19:16:08 +0100 X-Google-Sender-Auth: GYnRR1Tu-n__0yXjrsPs-G1Otrw Message-ID: Subject: Re: Git mirroring halted for freebsd-base To: Shawn Webb Cc: freebsd-git@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Dec 2016 18:16:11 -0000 2016-12-03 19:12 GMT+01:00 Shawn Webb : > On Sat, Dec 03, 2016 at 06:45:32PM +0100, Ulrich Sp??rlein wrote: >> 2016-12-03 18:29 GMT+01:00 Shawn Webb : >> > On Sat, Dec 03, 2016 at 06:25:29PM +0100, Ulrich Sp??rlein wrote: >> >> 2016-12-03 18:21 GMT+01:00 Shawn Webb : >> >> > On Sat, Dec 03, 2016 at 12:42:56PM +0100, Ulrich Sp??rlein wrote: >> >> >> 2016-12-02 11:07 GMT+01:00 Ulrich Sp??rlein : >> >> >> > The conversion process started chewing up 100% cpu without making much progress, first attempts to rectify this have failed. >> >> >> > >> >> >> > The svn2git conversion and pushes to github have been halted. Pushes to bitbucket also have been halted (we're reaching the 2GB limit imposed by bitbucket). >> >> >> > >> >> >> > I'll update this thread in about 24h. >> >> >> > Uli >> >> >> >> >> >> Service is fully restored now for the github mirror, sorry for the downtime. >> >> >> Bitbucket will stop working soon because of the size limitations. >> >> >> Speak up if you require this mirror to be kept up-to-date. >> >> >> >> >> >> Cheers, >> >> >> Uli >> >> > >> >> > Looks like it might be easier for some downstream projects to fully >> >> > recreate their ports repositories from scratch than to try to merge from >> >> > upstream. >> >> >> >> What are you referring to here? >> > >> > The ports repo at https://github.com/freebsd/freebsd-ports was force >> > pushed. Now attempts at merging in upstream's ports tree into >> > hardenedbsd's causes merge conflicts for hundreds of files, including >> > files we didn't change. >> > >> > So I'm forced to either inspect hundreds of files, manually merging in >> > the changes or recreate our ports tree from scratch, re-importing our >> > changes in a single atomic commit. The second option sounds more >> > appealing, though we'd lose the entire history of our changes. >> > >> > Additionally, anyone downstream from HardenedBSD might have to do the >> > same. Domino affect. >> >> I see. This shouldn't have happend, but as svnsync is >> non-transactional, we picked up some bad SVN metadata that made it >> into ports and base repos about a year ago. The SVN corruption was >> promptly fixed (I didn't ask for this), but that now leaves us with no >> way to actually re-do the conversion from scratch, as you'd need a >> corrupted SVN repo to produce the same results. >> >> You should be able to simply merge whatever "official" commit you last >> merged to with whatever the new "official" commit is now. This only >> affected metadata, so you'll get a clean merge (no conflicts) but you >> end up depending on 2x the history for about a year or so. Shouldn't >> be that much of a problem. Ask your local git wizard on how to do this >> best. >> >> >> > What caused the issue? What is going to be done to prevent it from >> >> > happening again? >> >> >> >> I have no root cause, other than bitbucket changing permissions and >> >> somehow git ending up using 100% CPU for most of the operations. >> > >> > So no guarantees this massive screw-up won't happen again? >> >> I said this before, and I'll say it again. This is a best-effort >> conversion and we're at the mercy of whatever SVN fucks up next. I >> provided clear instructions as to how to do the conversion in-house, >> and guess how many people actually wrote to me that they end up with >> different SHA hashes on github than they can produce in-house for both >> src and ports? >> >> What would be your guess? >> >> Exactly, 0 people have done the in-house conversion and have compared >> this to github. I could have put all kinds of backdoors in FreeBSD on >> github and not a single soul would've noticed. >> >> So if you depend on it, I would very much appreciate if you could do >> the same conversion in-house and report any drift as soon as possible, >> because it's a mess otherwise, as you can see. >> >> Any thoughts on how to fix this for src would also be appreciated, all >> I can think of is either pushing 2 heads and telling people to >> migrate, or doing the switchover on a flag day. > > Hey Uli, > > Sorry for the harsher tone earlier. I'm a bit stressed and it was unfair > of me to use that tone. > > I'm grateful for your efforts. I understand that supporting git isn't an > official service provided by FreeBSD. One item on my Christmas wishlst > would be to have official support for a read-only git mirror of the > various FreeBSD projects (mainly src and ports). > No worries, I feel your pain. My Friday evening was ruined too :( If you have the resources, *please* try a conversion run of all three repos too and report your findings. We need to make sure that the conversion is now more robust and we need to detect drift earlier :/ Cheers, Uli