From owner-freebsd-git@freebsd.org Thu Sep 26 17:26:19 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AEE7112E371 for ; Thu, 26 Sep 2019 17:26:19 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qk1-x743.google.com (mail-qk1-x743.google.com [IPv6:2607:f8b0:4864:20::743]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46fMKR2nY5z4ZjQ for ; Thu, 26 Sep 2019 17:26:19 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qk1-x743.google.com with SMTP id f16so2445386qkl.9 for ; Thu, 26 Sep 2019 10:26:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=OLWLAdr6J9snFuvFCS3wz/bcM16s1Xry7KpGGywl1jI=; b=LO6sKgEnQypNPS9CfIKbO8QAu2UKQ9tF0EpT459RO0dhqf+4T1HzeYYNimF/CwA31w MbWc+97+MQ0vAWniz0Yu/CHaArqY4oDfW4rhogM5GTPEQm/0lyHQSUBUnH1ajKAhBumD qVEMHAQqwlkGoI/wyq0DYsFR5okyaxnbK3M+0gkAgGaQUN0Olmecma68DfryUC5n0sW6 2ArmhkWJQHCS88P4FEbI0AKZRPx/LMcTDSm0VFfjcLWQeBBchE18dTdJaiK0oV4VSQ8i 0tvfWB6X06Apgaw06Lc70Xh9KvcqMFFB5J6P9sKOuBE96u8d+xs6tn8UZQnvtM41Sm8B xrrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=OLWLAdr6J9snFuvFCS3wz/bcM16s1Xry7KpGGywl1jI=; b=kW4Uy+foxKCgWBe9Zkj+YJxYEzAvFsXsGpSYBFz3OqHVCWVgwTB2bMu/EO8rGFJXk1 YWggS64w2S5gvNv2g6p33TZ24j64MzqjeDTGKxWDsYCnN+qh1enGIrhYdbTNYQHn2boM 7EbBXCnat52mciEJYh/hYFWrMPbdbzHtGx5A/VIbHPotdjzwMGRLTXZ2dY8K/vOTI5Fg zhn9AS4d3YMATNcchzUOBPuoYP0ASTh9HTAIyKM8K1jEVxgcuFE2rgltCLtN6fmBI8Hk Ef6wV8iL4DPaBU6IOG0jO16m0E99ek0fSsj6W9lUBB08wTg5z1W9ghPSYfVFNqdDp96J Pe1Q== X-Gm-Message-State: APjAAAX4yC2vp7S8XzjSMBUxBfLA3jZ8OyNIwJRx8+MU6ZmxMuo3lgNN qLfUD4RlKCoFaeVd68683mdR0KLWjWOUWVoKzLEEZg== X-Google-Smtp-Source: APXvYqwdcZx8QAZJPVq3ZsAk4BQd9CDGvcEhxtrV/lbe0Df4PK5eOMS9z9xAfLGJ3k+GxOhjnyLnltINRTnPGxoUyeU= X-Received: by 2002:a05:620a:6af:: with SMTP id i15mr4454654qkh.380.1569518778354; Thu, 26 Sep 2019 10:26:18 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Thu, 26 Sep 2019 11:26:07 -0600 Message-ID: Subject: Re: Service disruption: git converter currently down To: Ed Maste Cc: Sean Chittenden , freebsd-git@freebsd.org, =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= X-Rspamd-Queue-Id: 46fMKR2nY5z4ZjQ X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=LO6sKgEn; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::743) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-2.43 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCVD_IN_DNSWL_NONE(0.00)[3.4.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-0.43)[ip: (2.67), ipnet: 2607:f8b0::/32(-2.60), asn: 15169(-2.18), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Sep 2019 17:26:19 -0000 On Thu, Sep 26, 2019 at 8:26 AM Ed Maste wrote: > On Wed, 25 Sep 2019 at 15:50, Warner Losh wrote: > > > > git log always requires added care. There's not actually 9000 commits > there. The tree looks fine topologically. Its purely an artifact of git log. > > This seems to be getting into a philosophical discussion of what it > means for a commit to exist. But, given the constraints in the way git > represents commits the history crafted by the svn-git exporter indeed > shows thousands of "phantom" commits. The converter should (and with > uqs tweaks, would) represent the offending commit here as if it were a > cherry pick, not a merge. > Yes. This is the artifact of 'git log' I was talking about. Unlike subversion, it will log both sides of the merge commit back to the common ancestor unless you tell it to do otherwise, which is why we're seeing both legs of the tree. But git also has no way to represent cherry-picks. It just does them as regular commits. > In order to really represent this correctly we need to add to git > metadata tracking file operations. Recording that path d1/f1 was > copied from d2/f2 at some hash would allow us to properly represent > this case as well as renames/moves. > Right. We're not going to get that since the 'git' way is to divine that the copies happened. Unlike subversion, git doesn't use paths for version control... > >> git log --first-parent isn't really a solution here either, because > >> there are cases where one legitimately does want history from both > >> parents, especially working in downstream projects. > > > > I'm pretty sure it would be fine, even in that case. > > It's not fine, because it omits the commits I want to see. > The --first-parent actually mirrors what svn log shows today. What commits do you think that it omits? > >> > I'd offer the opinion that needing to know about things like git log > --first-parent vs having to rebase every single downstream fork, > >> > >> We won't need to rebase every fork - in no case should the path > >> forward be worse than uqs's suggestion of a merge from both old/new > >> conversions. > > > > IMHO, uqs suggestion is a complete non-starter, at least the "git diff | > git patch" one. It destroys all local history, commit messages, etc. Except > for the most trivial cases, it's not really going to fly with our users. > His other, followup ones might be workable into scripts. > > diff | patch is not the suggestion; the suggestion is to perform a > merge from the "new" conversion. Other options (e.g. some sort of > scripted commit replay) are at least no worse than that base case. > I'm curious how the 'merge' operation would work with such a remote common ancestor, especially with the complicated glitches in the currently exported tree. I'll have to play around with this since I have a similar situation with my git-svn trees that have identical commits to the github commits, but with different hashes (due to the differing ways git-svn and the github converter export the rXXXXX number), though maybe not with such a deep common ancestor that the old and new conversion trees would have. For the rebase workflow, I agree, it's pretty simple to graft a series of commits to a new branch, even one without a common ancestor. I do it all the time to move things back and forth between my git from github tree and my git from git-svn trees depending on my needs. > I'm not sure you can merge, as there's no common ancestor that's recent > enough to give it a chance at succeeding (since the different exports would > have different hashes starting fairly early in our history). My experience > with qemu is that long-lived merge-updated branches become quite difficult > to cope with after a while. It took me three weeks to sort out that > relatively simple repo. > > In fact, the merge works fine, even with completely unrelated > histories. You can try this by merging 'svn_head' (from git svn) to > 'master' (from svn2git), using `git merge --allow-unrelated-histories > origin/svn_head`. The resulting history has two copies of every > commit, but the file contents are unchanged over the merge. > I'll have to try that to see how well it works. I'd not used allow-unrelated-histories and had frequently run into this issue. In the past, I'd been warned off using that flag, but I'll give it another try. > If you try this in a tree with changes (i.e., try applying it to a > long-running merge-based branch) every modified file will result in a > conflict, but they can be trivially resolved in favour of the first > version. From that point on merging from the "new" conversion will > work as expected. > OK. I'll have to give it a spin to see where it takes me. > > A rebase has a chance of working for people following a 'rebase' work > flow. > > Indeed, for rebase workflow it's fairly straightforward. > > > However, for people like CHERIBSD who follow a 'merge from upstream' > model which never rebases (since that would be anti-social to their down > streams), I'm having trouble understanding how that could work. At work, we > basically do the merge from upstream with collapse model, which I'm having > trouble seeing how to move from old hashes to new. I'd like to know what > the plan for that would be and would happily test any solution there with a > copy of our repo. I'd even be happy to run experiments in advance of there > being something more public available to see what options do or don't work. > > Could you expand on the "merge from upstream with collapse" - > specifically, can you provide an example command used when merging > from FreeBSD? > We basically have an upstream called 'FreeBSD' that we fetch into our git repo: % git fetch FreeBSD master and then we create a beanch % git checkout -b merge-branch-rXXXXXX then we do the merge: % git subtree -P FreeBSD merge FreeBSD/master $HASH # resolve conflicts % git commit % git push then use use stash's pull-request to manage the landing into our master, but it's effectively a % git checkout master % git merge merge-branch-rXXXXXX which results in a fairly ugly master for us, which is the other reason I know the difference between git log and svn log behaviors so well :(. Effectively, the merges from upstream show up as a single merge commit, plus a number of follow-on fix-up commits. git subtree is both awesome and evil. Warnre Warner