Date: Sat, 10 Apr 2021 19:15:49 +0200 From: Martin Matuska <mm@FreeBSD.org> To: Warner Losh <imp@bsdimp.com> Cc: Ed Maste <emaste@freebsd.org>, freebsd-git <freebsd-git@freebsd.org>, Xin Li <delphij@freebsd.org>, Ryan Moeller <freqlabs@freebsd.org>, Alexander Motin <mav@freebsd.org>, Mateusz Guzik <mjg@freebsd.org> Subject: Re: OpenZFS branch tracking policy Message-ID: <f8d7a7f3-63a2-434f-054c-fadb9131cf82@FreeBSD.org> In-Reply-To: <CANCZdfp3EJ%2BbrNM02Sfzu_Y42VDEADiApFaX0V9bu_jb5NWd4w@mail.gmail.com> References: <21c7313e-315c-ec48-9437-e0a3d4ec14d2@FreeBSD.org> <CANCZdfopOxm-HTYkVPHkEweHw-F%2BA9mk3Vv26x4t3MEAVEd2gQ@mail.gmail.com> <CAPyFy2DS=nsE3-JQdqQC797xQhAiBACkuyA%2BcxkcRY0yeB_6=w@mail.gmail.com> <CANCZdfoPm0tfDpBTU8ORy-_Oa-tkiNX0_MeAdJn0T5ZJdQe6MQ@mail.gmail.com> <41924e9d-9d61-6646-6c8f-e4458f94296e@FreeBSD.org> <30f529c1-6087-e704-8cc7-0c48a40b7430@FreeBSD.org> <CANCZdfp3EJ%2BbrNM02Sfzu_Y42VDEADiApFaX0V9bu_jb5NWd4w@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Here are some of the facts: - In my merge, there are 15 conflicting files due to changes in FreeBSD=20 (add/add) - Some of the changes have already been upstreamed in later revisions of = openzfs than 891568c99 - A significant majority of the diffs is subject for upstreaming. The=20 ideal state would be to have all changes upstreamed. Sometimes changes=20 get upstreamed with modifications. - In general our developers open pull requests and commit to OpenZFS,=20 then we merge the changes What our developers would like is to use a "git blame" on=20 sys/contrib/openzfs/something to see the history path from OpenZFS. I agree that the merge commits should be more verbose, ideally=20 containing a "git log --oneline" of the commits since last merge. If a do a "squashed" merge like you described with bzip2, then I do not=20 import the history from OpenZFS. That way we don't need that at all and=20 can continue working the way we did until now. What you say about adding "unnecessary" history - since the common=20 development at OpenZFS the majority of commits directly affects FreeBSD. = Only "Linux-Only" and "CI-related" commits are not relevant for FreeBSD. I have updated my example branch how it may look like with more detailed = commit messages, nicely clickable from github: https://github.com/mmatuska/freebsd-src/tree/openzfs_master_merged So the the current question is quite simple, we can do one of the followi= ng: a) do the unsquashed merge I suggest that imports the openzfs history -=20 this will make the commits very transparent, future merges and upstream=20 tracking very easy and --allow-unrelated-history flag is not required=20 anymore. The "common" part of the histories in main and stable/13 will=20 be identical. b) if that is not desired or we are undecided I will continue the way we = go now until a better solution is found. In that case I will fork a=20 second vendor branch (vendor/openzfs-2.1) that starts with the latest=20 common commit of openzfs/master and openzfs/zfs-2.1-release and will=20 merge (or cherry-pick?) from this branch directly to stable/13. As an=20 alternative to merging, git cherry-pick supports -Xsubtree=3D as well. Best regards, mm On 10. 4. 2021 0:15, Warner Losh wrote: > > > On Fri, Apr 2, 2021 at 6:44 PM Martin Matuska <mm@freebsd.org=20 > <mailto:mm@freebsd.org>> wrote: > > I have prepared an example merged branch here: > https://github.com/mmatuska/freebsd-src/tree/openzfs_master_merged > <https://github.com/mmatuska/freebsd-src/tree/openzfs_master_merged= > > > The magical command was: > git merge -s subtree -Xsubtree=3D"sys/contrib/openzfs" 891568c99 > --allow-unrelated-histories > > Luckily, our current diff is manageable. > > > So I did this for bzip2 using approximately: > > git add remove bzip2 <url> > git fetch bzip2 > git merge -s subtree -Xsubtree=3Dcontrib/bzip2 bzip2/master=20 > --allow-unrelated-histories --squash > > [1] At this point I resolved conflicts, where were the entire files=20 > since I guess I didn't bootstrap right to the last merge. There were 4 = > files in conflict. > > Then I did a git add of all the files in conflict and a git commit. > > This produced a good commit. since it was a squash commit, there were=20 > no issues. > > However, it turns out I botched the commit at point [1] above. So I=20 > ran this again and got a conflict for the whole file that I'd removed=20 > a blank line from. > > So, this looks like it could be workable, but does lead me to a few=20 > questions: > > (1) How do we do this so that the conflicts aren't add/add conflicts?=20 > Is there some way to bootstrap this? > (2) Do we need to keep track of the last merge point and use that in=20 > merging the next one in? > (3) I assume we keep track of FreeBSD diffs in a branch off <url> and=20 > we merge that instead of master. > (4) What do we do about adjustments to the build that are needed? > (5) Do we need to host a FreeBSD-specific repo with this stuff, maybe=20 > with tags we don't want widely pushed to ease the next merge? Eg, make = > this the first case of a 'vendor repo' that we then pull squash=20 > commits from so that the vendor repo can track upstream, but not=20 > otherwise be pushed to all our users.... > > Finally, how did you deal with [1] producing so many full-file add/add = > conflicts? Oh, and what kind of commit message when things merge do=20 > you suggest? I rather like your 'bring in hash XXXX branch blah,=20 > here's the important highlights' emails and think that would be a good = > first cut at advice on what to put in these. > > This suggests the current answer is 'seems doable, but we need to=20 > document it and come up with recommendations for how to do it'. > > Warner > > On 3. 4. 2021 1:37, Martin Matuska wrote: > > Hi Warner and Ed, > > > > 2.1-release has already been branched. The stable branch policy i= n > > OpenZFS is somewhat strange, they make a staging branch for each > > patchlevel release, but the commits are continuous. > > > > To have some idea how big the repo history is: > > > > $ git rev-list master --count > > 6662 > > > > $ git rev-list zfs-2.1-release --count > > 6650 > > > > master and zfs-2.1-release have 6650 common commits at the=C2=A0 = moment > > > > $ git log master | wc -l > > 129868 > > > > (linecount - 4 * revcount) / revcount =3D linecount / revcount - = 4 =3D > > 15,4938 comment lines per commit on average > > > > Initial commit was made in Feb 26, 2008. > > > > Yearly commit counts: > > > > $ git log master | grep -c -E '^Date:.* 2020 -[0-9]+$' > > 666 > > > > $ git log master | grep -c -E '^Date:.* 2019 -[0-9]+$' > > 535 > > > > $git log master | grep -c -E '^Date:.* 2018 -[0-9]+$' > > 428 > > > > Martin > > > > On 2. 4. 2021 20:15, Warner Losh wrote: > >> > >> > >> On Fri, Apr 2, 2021 at 11:56 AM Ed Maste <emaste@freebsd.org > <mailto:emaste@freebsd.org> > >> <mailto:emaste@freebsd.org <mailto:emaste@freebsd.org>>> wrote: > >> > >> =C2=A0=C2=A0=C2=A0 On Fri, 2 Apr 2021 at 11:50, Warner Losh <imp= @bsdimp.com > <mailto:imp@bsdimp.com> > >> =C2=A0=C2=A0=C2=A0 <mailto:imp@bsdimp.com <mailto:imp@bsdimp.com= >>> wrote: > >> =C2=A0=C2=A0=C2=A0 > > >> =C2=A0=C2=A0=C2=A0 > We'd always hoped that we'd be able to do s= ubtree merges from > >> =C2=A0=C2=A0=C2=A0 upstreams > >> =C2=A0=C2=A0=C2=A0 > that use git into FreeBSD. The big worry, t= hough, was > that this > >> =C2=A0=C2=A0=C2=A0 would > >> =C2=A0=C2=A0=C2=A0 > needless bloat the repo with a lot of histo= ry. We don't want, > >> =C2=A0=C2=A0=C2=A0 for example, > >> =C2=A0=C2=A0=C2=A0 > all of LLVM's history in the tree. We'd alw= ays > anticipated that > >> =C2=A0=C2=A0=C2=A0 there'd be > >> =C2=A0=C2=A0=C2=A0 > some things we'd just accept the history fo= r, since it is > >> similar in > >> =C2=A0=C2=A0=C2=A0 > character to the vendor branches (though of= course a bit > more). > >> > >> =C2=A0=C2=A0=C2=A0 Note that if we do want to avoid bringing in = the full > history `git > >> =C2=A0=C2=A0=C2=A0 subtree merge` supports a `--squash` option. = This brings in > the > >> set of > >> =C2=A0=C2=A0=C2=A0 upstream changes as a single commit, without = bringing along the > >> =C2=A0=C2=A0=C2=A0 associated history. We will need to do more e= xperimentation to > >> confirm > >> =C2=A0=C2=A0=C2=A0 that the full process, including bootstrappin= g, will work > as we > >> want. > >> =C2=A0=C2=A0=C2=A0 Assuming this all works it should allow us to= forgo the use > of a > >> =C2=A0=C2=A0=C2=A0 FreeBSD-specific vendor branch in src. > >> > >> =C2=A0=C2=A0=C2=A0 We've discussed mirroring any such 3rd-party = source in some > >> =C2=A0=C2=A0=C2=A0 FreeBSD-controlled repository. This would all= ow the project to > >> retain > >> =C2=A0=C2=A0=C2=A0 a full copy of the history, but avoid bloatin= g src with it. > >> > >> =C2=A0=C2=A0=C2=A0 I agree with Warner that we may want a differ= ent policy (full > >> history > >> =C2=A0=C2=A0=C2=A0 or snapshots) for different contrib sources. > >> > >> > >> Good points Ed. I'd forgotten about --squash. > >> > >> Martin, what's your timeline for wanting to implement these > things? > >> I'm unfamiliar with the OpenZFS schedules. > >> > >> Warner > > _______________________________________________ > > freebsd-git@freebsd.org <mailto:freebsd-git@freebsd.org> mailing > list > > https://lists.freebsd.org/mailman/listinfo/freebsd-git > <https://lists.freebsd.org/mailman/listinfo/freebsd-git>; > > To unsubscribe, send any mail to > "freebsd-git-unsubscribe@freebsd.org > <mailto:freebsd-git-unsubscribe@freebsd.org>" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?f8d7a7f3-63a2-434f-054c-fadb9131cf82>