From owner-freebsd-git@freebsd.org Mon Sep 23 21:09:56 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AC3C312122C for ; Mon, 23 Sep 2019 21:09:56 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: from mail-ot1-x341.google.com (mail-ot1-x341.google.com [IPv6:2607:f8b0:4864:20::341]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46ccQq2z9Sz4XtQ for ; Mon, 23 Sep 2019 21:09:55 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: by mail-ot1-x341.google.com with SMTP id f21so13413285otl.13 for ; Mon, 23 Sep 2019 14:09:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=Zz/pWADifwVGe0Z6GOtM7tcSFSPgGi13pHzDneXGFDU=; b=XFqfzL/l/UR0zcK8ZJB+GkwE1Lpt3WsoYUwssPaaOvVXhpmcgedstS4aiv3MII322r eH8+mgmkjWT8psQRnfhQ9PWFFoBGJkFNKeWixa5Z9myPSlnU7A7DaFeNqNQjRptHdmL8 deuqe25D9pfz/u7taFsb3LMFPAylbdcapk4RjC9NZHQW1GbHO8iKN+ffoxqzysB3nwSY J4D67jOyi6I84zxyww6v+v7XLWhfJw6TAma+l6yxSuU6KCbIcGqnqFBDL5EBvPLfwGiv 53HmGr+Sv1eZD/N+49y4i+nJR8mNC92CHAclah1EhwBE2f7fZHL8Qg0bd7V7XXqi0isS uHbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=Zz/pWADifwVGe0Z6GOtM7tcSFSPgGi13pHzDneXGFDU=; b=KurLkdvA1YPAhA190jKj6E8XjgX0/rXD+myeahLWufKGfGGNm/ADXkhAwJgltm6HW8 2ONM1n+jLNgXJna+0hev/ikfFpVIVpBhTL76Tdk/TGRI+dqnnle3ZY7uslrMpMLmt4xD VSa0Fc6Xs+kg7oUd/pHwY3FQmqeMwfoB0fBoKQEpqp/0ZW5LolDFZzEtk+OuHVgB98Fo HuPBNGGAL9XHMGfXDENPBm6GSW7gx5oBzUHKvu2OyZZxRUmyPctGtvl8+ok3XFPx5HuX 7JbdzAnCxoX2ZGmMJ+kUmo8RcGfzH2NqIAq769oeAfRlDI4Gsi7xov49qRZb/G3hEvts IYmw== X-Gm-Message-State: APjAAAWrOTigFML/oWzTNWY7RJe1H0LzwFt0p8rTaA+kQOp337WgYBAN PQxoY9VawNOE31o0QNXohbx5sPXv21CYfjfRP+oXZyO2i/0= X-Google-Smtp-Source: APXvYqzaKLHT1zvbALWRs0OBhuO0+e9DM0lSn7oloQKQMMmkP5+26pByt9X6LUDOKwQLPuYdUZy3k6UsGfxBwxc8QW4= X-Received: by 2002:a9d:be4:: with SMTP id 91mr224510oth.111.1569272994088; Mon, 23 Sep 2019 14:09:54 -0700 (PDT) MIME-Version: 1.0 References: <20190923183424.ebnghzf67mx56aom@mutt-hbsd> In-Reply-To: From: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Date: Mon, 23 Sep 2019 23:09:39 +0200 Message-ID: Subject: Re: Service disruption: git converter currently down To: Warner Losh Cc: Shawn Webb , freebsd-git@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 46ccQq2z9Sz4XtQ X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=XFqfzL/l; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of uspoerlein@gmail.com designates 2607:f8b0:4864:20::341 as permitted sender) smtp.mailfrom=uspoerlein@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[1.4.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(0.00)[ip: (3.06), ipnet: 2607:f8b0::/32(-2.64), asn: 15169(-2.20), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2019 21:09:56 -0000 Am Mo., 23. Sept. 2019 um 21:06 Uhr schrieb Warner Losh : > > > > On Mon, Sep 23, 2019, 8:42 PM Ulrich Sp=C3=B6rlein = wrote: >> >> Am Mo., 23. Sept. 2019 um 20:34 Uhr schrieb Shawn Webb >> : >> > >> > Hey Ulrich, >> > >> > I appreciate your hard work in maintaining the git mirror. Work like >> > this can sometimes go unthanked. I want to take a moment to show >> > appreciation for you and the FreeBSD project in maintaining the git >> > mirror. >> > >> > I do have a few concerns with what was stated in your email. I've >> > written my concerns inline. I hope this discussion is a positive one, >> > wherein upstream and downstream can effectively come to a conclusion. >> > >> > On Mon, Sep 23, 2019 at 08:16:25PM +0200, Ulrich Sp??rlein wrote: >> > > Am Mo., 23. Sept. 2019 um 19:51 Uhr schrieb Sean Chittenden >> > > : >> > > >> >> > > >> Please note however, that more "garbage" metadata escaped from SV= N into >> > > >> github, meaning 3rd parties have a hard time re-running the conve= rsion and >> > > >> making sure that it matches SVN down to the metadata (i.e. timest= amps). >> > > >> >> > > >> Eventually, this will have to be re-rolled and a new "master" bra= nch will >> > > >> be force-pushed into github. There's no timeline for this yet. >> > > > >> > > > >> > > > Wait, what? Can you elaborate? >> > > > >> > > > Discussion of a force-push to github has occurred a few times and = been explicitly ruled out because most of our corporate citizens use github= to integrate changes from FreeBSD. Rerolling master was universally rejec= ted when we socialized wanting to do this due to the level of disruption th= is would cause. The feedback was that this would be a high-cost, low-value= operation. In the tradeoffs of purity vs pragmatism, pragmatism wins ever= y time (that is the FreeBSD way). >> > > > >> > > > -sc >> > > >> > > >> > > This is not just about pragmatism and the disruption it would cause = is >> > > vastly overblown by people who don't seem to know much about the git >> > > storage model. >> > > >> > > There *is* garbage metadata in the published version on github, ther= e >> > > *is* a disclaimer on https://wiki.freebsd.org/GitWorkflow since >> > > forever, and the cost of switching from 1 published branch to anothe= r >> > > is literally: >> > > >> > > - git diff origin/broken_master mybranch > mybranch.patch >> > > - git checkout -b fixed_branch origin/fixed_master >> > > - patch < mybranch.patch >> > >> > Such a workflow breaks historical accuracy. Instead of `git annotate` >> > showing the history properly, it's now based on an "epoch commit". >> > Sure such a commit brings the branch to a working condition, but at >> > the cost of history. >> >> Is there really that much value in having "git blame" work in that envir= onment? >> My mental model is of short-lived branches that get upstreamed, so I'm b= iased >> towards this not being all that big of a problem (for some at least). > > > Yes. There absolutely is. > >> > > >> > > It should also be possible to merge both broken and fixed master int= o >> > > your branch (at the exact same SVN revision in time) and then you ca= n >> > > follow fixed_master going forward. You'll schlepp around double the >> > > commit history, but not tree objects. >> > > If you want to retain history, you can upstream the changes prior to >> > > the switch >> > >> > I so wish that were possible for certain downstream projects. We're >> > unable to upstream the majority of our work. To argue "upstream your >> > work and you won't be affected" is to choose an argument that does not >> > reflect the reality of a growing portion of FreeBSD's downstream >> > consumers: the inability to work effectively with upstream. >> >> :/ >> >> I'm 80% sure that you can just merge both branches and things will be fi= ne >> (though the exact incantation will surely be black magic). I'd love to >> try this on >> an actual repo though, I don't have the time to craft some test repo to = verify >> this assumption, and then find out that other repos are different). > > > One might be able to do this, but this is really quite a lot to ask. I've= done a lot of git rebase in connection with qemu. The hard part us knowing= hash X is for rev r123 in branch 1, but hash Y In branch 2. Once you can a= utomate that, for various mappings, a script to rebase old to new becomes p= ossible which will lessen the impact, but not eliminate it. Hmm, I think that's the trivial part. All commits record the SVN revision in the notes after all. See: % git show commit 2e105280ca193f7bafe103652bb1249704ba25f6 (HEAD -> master) Author: bdrewery Date: 2015-10-15 20:46:34 +0000 Remove unneeded libg++ reference that came in with r267511 based on a removed comment. Sponsored by: EMC / Isilon Storage Division Notes: svn path=3D/head/; revision=3D289388 So all that is left is matching up 2 notes in 2 branches (or 2 tree objects in 2 branches, actually). What's the tree on a specific branch: % git log -1 --format=3D%T master 9c96d76028084fe6b8077292fc428388a22f07f0 Search for it on a different branch (using the same master here, I don't have a repo handy with 2 different but identical branches) % git rev-list master | while read hash; do case `git log -1 --format=3D%T $hash` in 9c96d76028084fe6b8077292fc428388a22f07f0) echo "found commit $hash also pointing to tree 9c96d76028084fe6b8077292fc428388a22f07f0";; esac; done found commit 2e105280ca193f7bafe103652bb1249704ba25f6 also pointing to tree 9c96d76028084fe6b8077292fc428388a22f07f0 Except you can probably write that quicker also, maybe just dump all commit =E2=86=92 tree mappings and do something with that. As I said, it should be straightforward as long as you know a thing or two about git. > >> Also, I'll be offline in the coming weeks, so don't expect immediate res= ponses >> from me going forward. I already had to spent most of my weekend to patc= h >> things up and had to cancel other plans. > > > I think it is safe to view this discussion as just don't force push yet a= s we are in no position to mitigate the large disruption this would cause. = If you aren't force pushing master, there is no time urgency to solve this. > > Finally, and most importantly, thank you for all your time and efforts he= re. They are much appreciated. Any quibbling over future plans shouldn't de= tract from that. > > Warner There will not be a surprise force push if I can help it. That's what I just sacrificed a weekend for, after all :( Should (!!) the project ever decide to abandon SVN, then I would suggest to roll that into a re-do of the conversion, and also only start the conversion with the first commit to SVN, because the cvs2svn conversion also has a bunch of garbage in there, and we should rather not claim that those commits actually every happened in the way and form they are currently recorded in SVN. hth Uli