From owner-freebsd-git@freebsd.org Mon Sep 23 21:40:41 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B5E27121BB2 for ; Mon, 23 Sep 2019 21:40:41 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qt1-x835.google.com (mail-qt1-x835.google.com [IPv6:2607:f8b0:4864:20::835]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46cd6J27L5z4Zf1 for ; Mon, 23 Sep 2019 21:40:40 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qt1-x835.google.com with SMTP id n7so19045277qtb.6 for ; Mon, 23 Sep 2019 14:40:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ZFdLQ7dS+sUW+9X2ENRQ0bJJo0/Pe3X4kuz51f7+PNw=; b=nUPIo4kNmEkqtXJEWD4BCFphN8npz9tMbdX7YPU3wFbSrscj+kKjl9n/5XaLpUAnCS eo4U8IJT2U7klb3wpgMBCQwvdMNl9AyCISYrKxv2XVITsucDIErRpR9miDWHjXRjQ339 fElpvgJpqYM+RG1ldWuCzvAwAkqKmgGRgKZ8pozpz1lMk4K7n9PfDSOpGNl9EzLMw69S Qxo/MSnddaW0L2bBlVVwL+EdBLOsdKquoXZLw3gOrbAyuxkba3ge6R+Hr4OE82zTbORo dcSQTS2PaLfM2eLi/Dihy8gIdrqh2gzfKaCfgdhNSEKt/TIkNxne2ibAMSUCGJd5CtWa lRcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ZFdLQ7dS+sUW+9X2ENRQ0bJJo0/Pe3X4kuz51f7+PNw=; b=QYVBFamgQOPcmTNv54xtcL0iJUbUldRgljmcvLuDzbGqDvSABwAeBctRq8j5h2Wngy R0eaqwz793BHaKKdgBvqhr1ld4oajSTBQjvxaBJ081NPbTdAoTOPBLMu3aON613YbnrX mYdSwCnL/IUCnpbUUeg1l/8h2rxJPSeYqFqOzp385VOlgJcgIY1Gry0eLHpFKglJtwnZ SWOM7X6OZvYk0fD4WDkPbV3gq7jzw2v6sY+ZzVLCG/dP4et+l8OXXxjWnhWGzMQ72iaT NyarDezXKdBvvsEMM4gHDCAFOCFmuBv/KmoyZrv+plGOhFteFJX6pMUvUKPOooksjGB9 cYNQ== X-Gm-Message-State: APjAAAW/E8Twu2Y8Zd87vEOMotccpONXvkhxcq1mTj34GiS/tEHwFCvy Jz0Dv7KFoYQ8lBkLHjLkwmIHCfJe8dadndCD5FIvYQ== X-Google-Smtp-Source: APXvYqzlsC82lU+7ue8WPw7M6lHkdK09nNK0xJ4W2lpe9Aww1/LvD4JDI9no3BMCE8dgRzH8awIQhAlMFpM12L3UMyc= X-Received: by 2002:ac8:2bca:: with SMTP id n10mr2378195qtn.242.1569274838533; Mon, 23 Sep 2019 14:40:38 -0700 (PDT) MIME-Version: 1.0 References: <20190923183424.ebnghzf67mx56aom@mutt-hbsd> In-Reply-To: From: Warner Losh Date: Mon, 23 Sep 2019 23:40:25 +0200 Message-ID: Subject: Re: Service disruption: git converter currently down To: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Cc: Shawn Webb , freebsd-git@freebsd.org X-Rspamd-Queue-Id: 46cd6J27L5z4Zf1 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=nUPIo4kN; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::835) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-3.86 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; URI_COUNT_ODD(1.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCVD_IN_DNSWL_NONE(0.00)[5.3.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; FREEMAIL_TO(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-2.86)[ip: (-9.39), ipnet: 2607:f8b0::/32(-2.64), asn: 15169(-2.20), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2019 21:40:41 -0000 On Mon, Sep 23, 2019, 11:09 PM Ulrich Sp=C3=B6rlein = wrote: > Am Mo., 23. Sept. 2019 um 21:06 Uhr schrieb Warner Losh : > > > > > > > > On Mon, Sep 23, 2019, 8:42 PM Ulrich Sp=C3=B6rlein > wrote: > >> > >> Am Mo., 23. Sept. 2019 um 20:34 Uhr schrieb Shawn Webb > >> : > >> > > >> > Hey Ulrich, > >> > > >> > I appreciate your hard work in maintaining the git mirror. Work like > >> > this can sometimes go unthanked. I want to take a moment to show > >> > appreciation for you and the FreeBSD project in maintaining the git > >> > mirror. > >> > > >> > I do have a few concerns with what was stated in your email. I've > >> > written my concerns inline. I hope this discussion is a positive one= , > >> > wherein upstream and downstream can effectively come to a conclusion= . > >> > > >> > On Mon, Sep 23, 2019 at 08:16:25PM +0200, Ulrich Sp??rlein wrote: > >> > > Am Mo., 23. Sept. 2019 um 19:51 Uhr schrieb Sean Chittenden > >> > > : > >> > > >> > >> > > >> Please note however, that more "garbage" metadata escaped from > SVN into > >> > > >> github, meaning 3rd parties have a hard time re-running the > conversion and > >> > > >> making sure that it matches SVN down to the metadata (i.e. > timestamps). > >> > > >> > >> > > >> Eventually, this will have to be re-rolled and a new "master" > branch will > >> > > >> be force-pushed into github. There's no timeline for this yet. > >> > > > > >> > > > > >> > > > Wait, what? Can you elaborate? > >> > > > > >> > > > Discussion of a force-push to github has occurred a few times an= d > been explicitly ruled out because most of our corporate citizens use gith= ub > to integrate changes from FreeBSD. Rerolling master was universally > rejected when we socialized wanting to do this due to the level of > disruption this would cause. The feedback was that this would be a > high-cost, low-value operation. In the tradeoffs of purity vs pragmatism= , > pragmatism wins every time (that is the FreeBSD way). > >> > > > > >> > > > -sc > >> > > > >> > > > >> > > This is not just about pragmatism and the disruption it would caus= e > is > >> > > vastly overblown by people who don't seem to know much about the g= it > >> > > storage model. > >> > > > >> > > There *is* garbage metadata in the published version on github, > there > >> > > *is* a disclaimer on https://wiki.freebsd.org/GitWorkflow since > >> > > forever, and the cost of switching from 1 published branch to > another > >> > > is literally: > >> > > > >> > > - git diff origin/broken_master mybranch > mybranch.patch > >> > > - git checkout -b fixed_branch origin/fixed_master > >> > > - patch < mybranch.patch > >> > > >> > Such a workflow breaks historical accuracy. Instead of `git annotate= ` > >> > showing the history properly, it's now based on an "epoch commit". > >> > Sure such a commit brings the branch to a working condition, but at > >> > the cost of history. > >> > >> Is there really that much value in having "git blame" work in that > environment? > >> My mental model is of short-lived branches that get upstreamed, so I'm > biased > >> towards this not being all that big of a problem (for some at least). > > > > > > Yes. There absolutely is. > > > >> > > > >> > > It should also be possible to merge both broken and fixed master > into > >> > > your branch (at the exact same SVN revision in time) and then you > can > >> > > follow fixed_master going forward. You'll schlepp around double th= e > >> > > commit history, but not tree objects. > >> > > If you want to retain history, you can upstream the changes prior = to > >> > > the switch > >> > > >> > I so wish that were possible for certain downstream projects. We're > >> > unable to upstream the majority of our work. To argue "upstream your > >> > work and you won't be affected" is to choose an argument that does n= ot > >> > reflect the reality of a growing portion of FreeBSD's downstream > >> > consumers: the inability to work effectively with upstream. > >> > >> :/ > >> > >> I'm 80% sure that you can just merge both branches and things will be > fine > >> (though the exact incantation will surely be black magic). I'd love to > >> try this on > >> an actual repo though, I don't have the time to craft some test repo t= o > verify > >> this assumption, and then find out that other repos are different). > > > > > > One might be able to do this, but this is really quite a lot to ask. > I've done a lot of git rebase in connection with qemu. The hard part us > knowing hash X is for rev r123 in branch 1, but hash Y In branch 2. Once > you can automate that, for various mappings, a script to rebase old to ne= w > becomes possible which will lessen the impact, but not eliminate it. > > Hmm, I think that's the trivial part. All commits record the SVN > revision in the notes after all. See: > > % git show > commit 2e105280ca193f7bafe103652bb1249704ba25f6 (HEAD -> master) > Author: bdrewery > Date: 2015-10-15 20:46:34 +0000 > > Remove unneeded libg++ reference that came in with r267511 based > on a removed > comment. > > Sponsored by: EMC / Isilon Storage Division > > Notes: > svn path=3D/head/; revision=3D289388 > > > So all that is left is matching up 2 notes in 2 branches (or 2 tree > objects in 2 branches, actually). > > What's the tree on a specific branch: > % git log -1 --format=3D%T master > 9c96d76028084fe6b8077292fc428388a22f07f0 > > Search for it on a different branch (using the same master here, I > don't have a repo handy with 2 different but identical branches) > % git rev-list master | while read hash; do case `git log -1 > --format=3D%T $hash` in 9c96d76028084fe6b8077292fc428388a22f07f0) echo > "found commit $hash also pointing to tree > 9c96d76028084fe6b8077292fc428388a22f07f0";; esac; done > found commit 2e105280ca193f7bafe103652bb1249704ba25f6 also pointing to > tree 9c96d76028084fe6b8077292fc428388a22f07f0 > > Except you can probably write that quicker also, maybe just dump all > commit =E2=86=92 tree mappings and do something with that. > > As I said, it should be straightforward as long as you know a thing or > two about git. > And I'm saying that it needs to be simple and easy to convert. I know a lot about git and the above is neither simple nor easy. We need a bulletproof script to rebase current branches into the new branch. For maximum safety, this should be at identical branch points (the less safe rebasing to the new branch is, I'll grant, relatively easy). So, the above is a good sketch to get started, but no where near sufficient given our current user base. And I've not even thought through the implications of people that forked from our repo on github and what the change means to them (or if it is really all the same problem). I'm willing to stick my hand up here to be involved in any automation. I have a enough to loose in WIP work alone to be motivated that this process just work... > > >> Also, I'll be offline in the coming weeks, so don't expect immediate > responses > >> from me going forward. I already had to spent most of my weekend to > patch > >> things up and had to cancel other plans. > > > > > > I think it is safe to view this discussion as just don't force push yet > as we are in no position to mitigate the large disruption this would caus= e. > If you aren't force pushing master, there is no time urgency to solve thi= s. > > > > Finally, and most importantly, thank you for all your time and efforts > here. They are much appreciated. Any quibbling over future plans shouldn'= t > detract from that. > > > > Warner > > There will not be a surprise force push if I can help it. That's what > I just sacrificed a weekend for, after all :( > And we all owe you big for that. Thank you! Should (!!) the project ever decide to abandon SVN, then I would > suggest to roll that into a re-do of the > conversion, and also only start the conversion with the first commit > to SVN, because the cvs2svn conversion > also has a bunch of garbage in there, and we should rather not claim > that those commits actually every happened > in the way and form they are currently recorded in SVN. > Rerolling has issues. Not rerolling also has issues. We should collectively decide on which set of issues we want. Warner hth > Uli >