Date: Thu, 19 Nov 2020 14:00:36 -0500 From: Dan Langille <dan@langille.org> To: Marc Branchaud <marcnarc@gmail.com> Cc: freebsd-git@freebsd.org Subject: Re: Monitoring commits on all branches Message-ID: <BE9B078D-633B-4340-9A79-03B0FA60C431@langille.org> In-Reply-To: <3c9f6285-ae7c-1062-2dd3-42f8c953a230@gmail.com> References: <197541CC-FEA7-4B4C-936E-66A5625BB64C@langille.org> <3c9f6285-ae7c-1062-2dd3-42f8c953a230@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> On Nov 19, 2020, at 11:16 AM, Marc Branchaud <marcnarc@gmail.com> =
wrote:
>=20
> On 2020-11-18 8:49 p.m., Dan Langille wrote:
>> How can a repo be monitored for commits on all branches?
>> I know how to ask a given branch: do you have any commits after =
foo_hash?
>> How do I:
>> * get a list of all commits since foo_hash
>=20
> A quick a note about Warner's reply:
>=20
>> git log $hash..HEAD
>=20
> "HEAD" is just a git nickname for "whatever you have currently =
checked-out" (which can be a branch, a tag, or "detached" commit SHA =
ID).
Mathieu mentioned: git log $foo_hash...branch_name
That was the first time I've seen that used. All previous suggestions =
were HEAD, not branch_name. But they are all the same in this context?
>> * know which branch each of those commits was on (e.g. master, =
branches/2020Q4)
>=20
> Unfortunately you'll find most normal git advice to be a bit =
frustrating with the FreeBSD repos, because FreeBSD doesn't work the way =
most people use git. Specifically, the FreeBSD project does not ever =
merge branches (in the git sense of the word "merge"). Things would be =
very, very much easier if the FreeBSD project were to use git-style =
merging. I believe there are discussions underway about adjusting the =
whole MFC process for the git world. I admit that part of my motivation =
in writing this message is to provide grist for that mill.
>=20
> Fortunately even without git-merged branches, there are still git =
tools that help, though they're not as precise as one would like.
>=20
> Let's look at a concrete example with the beta ports git repo (which I =
just cloned), and compare the 2020Q4 and main branches. I'll start with =
some overall exploration, then address your specific question.
>=20
> There are 298 commits in the 2020Q4 branch. I know this because
> git merge-base origin/main origin/branches/2020Q4
> tells me where 2020Q4 branched off of main: commit 5dbe4e5f775ea2. =
And
> git rev-list 5dbe4e5f775ea2..origin/branches/2020Q4 | wc -l
> says "299". (The "rev-list" command is a bare-bones version of "log" =
that only lists commit SHA IDs.)
[examples snipped]
I followed that.
I took the merge information as background and good-to-know, because =
FreshPorts won't be doing any merging. It just needs a good "git =
checkout" working copy.
Sorry for such a long reply.
* How FreshPorts extracts data
FreshPorts is only interested in a snapshot of the repo with respect to =
a given commit. It works on the 'repo as a whole' to extract values =
from the ports which were affected by that commit. Case in point: a =
commit to a parent port might affect any or all of the child ports. All =
the child ports need to be refreshed.
I am quickly concluding that FreshPorts must decide in advance what git =
branches it will pay attention to. At present, it follows all branches.
* FreshPorts (without git) uses email to create XML
When moving FreshPorts from subversion to git, one of the goals was to =
avoid relying on email to know that a commit has occurred. That is how =
FreshPorts has always worked. The email (from the CVS commit) was parsed =
and XML created. This code was updated for SVN. The XML is then used to =
load the commit into the FreshPorts database which then drives the =
website contents.
When I started the GIT conversion, there was no commit email. "git log =
$foo_hash...HEAD" is how FreshPorts knows what commits to process.
One positive aspect of email approach: it identified the branch. So far, =
I can't see how I can process the repo as a whole and see every commit =
and know what branch it was on.
* Polling git
It is beginning to sound like the FreshPorts git code for detecting =
incoming commits will be:
Every N minutes, do this:
for each repo in REPOS
for branch in BRANCHES
cd to the directory for that repo
git checkout branch
git log $branch_last_hash...HEAD
for each of those commits
process the commit
end for
end for
end for
At present, the REPOS and BRANCHES are:
* freebsd BRANCHES=3D"master branches/2020Q4 branches/2020Q3 =
branches/2020Q2 ...etc"
* freebsd-ports BRANCHES=3D"master stable/12 stable/11"
* freebsd-doc BRANCHES=3D"master"
Some might ask:
* Why not just master and latest-quarterly for freebsd-ports?
* Because commits to older branches sometimes occur (or at least I =
thought I saw one once)
Commit hooks might also help, but I'm not sure if that will make things =
easier or complicate everything
* When new branches arrive
It is vital that FreshPorts remain automated as much as it can be. At =
present, under SVN, I might have to fix things perhaps 5 or 6 times a =
year, usually because a commit did not get processed.
Keeping that in mind, I do not yet know how to handle the following =
situations:
A new branch comes out.
* Automation might be possible for ports quarterly branches
* FreshPorts has to know there is a new branch
* BRANCHES needs to be updated
* I don't see that it can be automated for stable/*
* need to handle 'git checkout branch' when branch does not exist?
* Once branch exists, how do you find out about the commits when you =
have no starting point for 'git log'?
Right now, a new quarterly branch is noticed when the first commit email =
comes through. FreshPorts then does an 'svn co' for that branch.
I'm hoping someone has good ideas for my edge cases.
Thank you.
=E2=80=94=20
Dan Langille
http://langille.org/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?BE9B078D-633B-4340-9A79-03B0FA60C431>