From owner-freebsd-git@freebsd.org Thu Nov 19 21:29:28 2020 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7EFB44754D0 for ; Thu, 19 Nov 2020 21:29:28 +0000 (UTC) (envelope-from oliver.pntr@gmail.com) Received: from mail-ot1-x329.google.com (mail-ot1-x329.google.com [IPv6:2607:f8b0:4864:20::329]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CcXr73Pplz3RD9 for ; Thu, 19 Nov 2020 21:29:27 +0000 (UTC) (envelope-from oliver.pntr@gmail.com) Received: by mail-ot1-x329.google.com with SMTP id n12so3253880otk.0 for ; Thu, 19 Nov 2020 13:29:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=CEXE3YNFRtXRbagOh7eRz67sH9dgxIN8V5ok+wJ1d7Y=; b=W3mtirgV+9ENqFjnT1qBdieOe9SWc5dgEBOFWDKRcav8ikKSffBxe+OY1rnauAJ1jY XVVeg26Y+zO6gKnR2fgrrDVljVMl18fbfr0bErW0uG2dvY6aggj6fX/xgK97UCjsOz71 RkW6RduSza/tK8uNSilMNXuuK/64DnPbqHkZNeaNFuAy/AxfEKVggLLFDjfR7HL0xQGy lnSFKQioJPeW+Oi2SSFUpCdYJor7oZrV3p9ELsJlwCyO3a5ILzsCKTHJtHCGFW11BY7/ G1Rvw12AMmwwo4cvCqBYBkXzaqtdrDp/cqEkW4H61F0SKU0KWTRKM6iMLODGBJgrbT7q VsCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=CEXE3YNFRtXRbagOh7eRz67sH9dgxIN8V5ok+wJ1d7Y=; b=NXnjrXIWeEo5YVh7IvhfnywN6SOkRLOgKA1DanYgTuFC2kM46mkS5MucYgcJAknX5z A/bB2XaRBF1B1p7SuNUHGGb0ffTLlKgKYDI/JOOadOQxc2D/QiVaGFNdPa1O/oTqsXLT ZaNMEImzKH3B+NBszn8UTcw9RaNINQWhRG7jQRAKTDPUh8ZGuhgaqcmpet0PQEFKZzJR NVRcqD88fBjAPmdixxDxxrjxpFVhCIdMH19sSQUIGK/tZvFTZhD6lpXvisErRRUyVJur a2CLMjaSd3LOGLfiXLOog5nqvbys9c9J9tZkWScKjx4lrrABJk4ijhr0A+uDDBDzHrlR yC1w== X-Gm-Message-State: AOAM533EqFrhoWmFk74GzZ4eyN7gvo2SCZEIh87A85bvO1l9pEu4cR9r 6LbDzoWmhHaL0VYDH9N2vVEbx0wBH6Rvj6+MTJLkVehg X-Google-Smtp-Source: ABdhPJyNLoJ7vnp3+dTAe8/mdmG77h8TnQpOEQy4EbKLsOl4dEnq9GW5lBv8mFl0FQ0C6Dre1JY1n/GkRJUF1cnVj24= X-Received: by 2002:a9d:620d:: with SMTP id g13mr11067008otj.56.1605821366287; Thu, 19 Nov 2020 13:29:26 -0800 (PST) MIME-Version: 1.0 Received: by 2002:a4a:51c1:0:0:0:0:0 with HTTP; Thu, 19 Nov 2020 13:29:25 -0800 (PST) In-Reply-To: References: <197541CC-FEA7-4B4C-936E-66A5625BB64C@langille.org> <3c9f6285-ae7c-1062-2dd3-42f8c953a230@gmail.com> From: Oliver Pinter Date: Thu, 19 Nov 2020 22:29:25 +0100 Message-ID: Subject: Re: Monitoring commits on all branches To: Dan Langille Cc: Marc Branchaud , "freebsd-git@freebsd.org" X-Rspamd-Queue-Id: 4CcXr73Pplz3RD9 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=W3mtirgV; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of oliverpntr@gmail.com designates 2607:f8b0:4864:20::329 as permitted sender) smtp.mailfrom=oliverpntr@gmail.com X-Spamd-Result: default: False [-3.46 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.46)[-0.457]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::329:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::329:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::329:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-git]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Nov 2020 21:29:28 -0000 Hi! On Thursday, November 19, 2020, Dan Langille wrote: > > On Nov 19, 2020, at 11:16 AM, Marc Branchaud wrote= : > > > > On 2020-11-18 8:49 p.m., Dan Langille wrote: > >> How can a repo be monitored for commits on all branches? > >> I know how to ask a given branch: do you have any commits after > foo_hash? > >> How do I: > >> * get a list of all commits since foo_hash > > > > A quick a note about Warner's reply: > > > >> git log $hash..HEAD > > > > "HEAD" is just a git nickname for "whatever you have currently > checked-out" (which can be a branch, a tag, or "detached" commit SHA ID). > > Mathieu mentioned: git log $foo_hash...branch_name > > That was the first time I've seen that used. All previous suggestions wer= e > HEAD, not branch_name. But they are all the same in this context? > > >> * know which branch each of those commits was on (e.g. master, > branches/2020Q4) > > > > Unfortunately you'll find most normal git advice to be a bit frustratin= g > with the FreeBSD repos, because FreeBSD doesn't work the way most people > use git. Specifically, the FreeBSD project does not ever merge branches > (in the git sense of the word "merge"). Things would be very, very much > easier if the FreeBSD project were to use git-style merging. I believe > there are discussions underway about adjusting the whole MFC process for > the git world. I admit that part of my motivation in writing this messag= e > is to provide grist for that mill. > > > > Fortunately even without git-merged branches, there are still git tools > that help, though they're not as precise as one would like. > > > > Let's look at a concrete example with the beta ports git repo (which I > just cloned), and compare the 2020Q4 and main branches. I'll start with > some overall exploration, then address your specific question. > > > > There are 298 commits in the 2020Q4 branch. I know this because > > git merge-base origin/main origin/branches/2020Q4 > > tells me where 2020Q4 branched off of main: commit 5dbe4e5f775ea2. And > > git rev-list 5dbe4e5f775ea2..origin/branches/2020Q4 | wc -l > > says "299". (The "rev-list" command is a bare-bones version of "log" > that only lists commit SHA IDs.) > > [examples snipped] > > I followed that. > > I took the merge information as background and good-to-know, because > FreshPorts won't be doing any merging. It just needs a good "git checkout= " > working copy. > > Sorry for such a long reply. > > > * How FreshPorts extracts data > > FreshPorts is only interested in a snapshot of the repo with respect to a > given commit. It works on the 'repo as a whole' to extract values from t= he > ports which were affected by that commit. Case in point: a commit to a > parent port might affect any or all of the child ports. All the child > ports need to be refreshed. > > I am quickly concluding that FreshPorts must decide in advance what git > branches it will pay attention to. At present, it follows all branches. > > > * FreshPorts (without git) uses email to create XML > > When moving FreshPorts from subversion to git, one of the goals was to > avoid relying on email to know that a commit has occurred. That is how > FreshPorts has always worked. The email (from the CVS commit) was parsed > and XML created. This code was updated for SVN. The XML is then used to > load the commit into the FreshPorts database which then drives the websit= e > contents. > > When I started the GIT conversion, there was no commit email. "git log > $foo_hash...HEAD" is how FreshPorts knows what commits to process. > > One positive aspect of email approach: it identified the branch. So far, = I > can't see how I can process the repo as a whole and see every commit and > know what branch it was on. > > > * Polling git > > It is beginning to sound like the FreshPorts git code for detecting > incoming commits will be: > > Every N minutes, do this: > > for each repo in REPOS > for branch in BRANCHES > cd to the directory for that repo > git checkout branch > git log $branch_last_hash...HEAD > for each of those commits > process the commit > end for > end for > end for > > At present, the REPOS and BRANCHES are: > > * freebsd BRANCHES=3D"master branches/2020Q4 branches/2020Q3 branches/202= 0Q2 > ...etc" > * freebsd-ports BRANCHES=3D"master stable/12 stable/11" > * freebsd-doc BRANCHES=3D"master" > > Some might ask: > > * Why not just master and latest-quarterly for freebsd-ports? > * Because commits to older branches sometimes occur (or at least I though= t > I saw one once) > > Commit hooks might also help, but I'm not sure if that will make things > easier or complicate everything > > > * When new branches arrive > > It is vital that FreshPorts remain automated as much as it can be. At > present, under SVN, I might have to fix things perhaps 5 or 6 times a yea= r, > usually because a commit did not get processed. > > Keeping that in mind, I do not yet know how to handle the following > situations: > > A new branch comes out. > > * Automation might be possible for ports quarterly branches > * FreshPorts has to know there is a new branch > * BRANCHES needs to be updated > * I don't see that it can be automated for stable/* > * need to handle 'git checkout branch' when branch does not exist? > * Once branch exists, how do you find out about the commits when you have > no starting point for 'git log'? > > Right now, a new quarterly branch is noticed when the first commit email > comes through. FreshPorts then does an 'svn co' for that branch. > > > > I'm hoping someone has good ideas for my edge cases. Try the 'git branch -rv' command, cache its output in a file, and select the lines with diffed to the new run of previous one. > > Thank you. > > =E2=80=94 > Dan Langille > http://langille.org/ > > > > > _______________________________________________ > freebsd-git@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-git > To unsubscribe, send any mail to "freebsd-git-unsubscribe@freebsd.org" >