From owner-freebsd-git@freebsd.org Fri Apr 9 22:15:36 2021 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8DC945C02CF for ; Fri, 9 Apr 2021 22:15:36 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qt1-x82d.google.com (mail-qt1-x82d.google.com [IPv6:2607:f8b0:4864:20::82d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FHCBH4sDrz4hn2 for ; Fri, 9 Apr 2021 22:15:35 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qt1-x82d.google.com with SMTP id i19so5411710qtv.7 for ; Fri, 09 Apr 2021 15:15:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=RpQbKCVYolg2YM51njqzYl4akm03N6MZjxdg9LiEnVc=; b=Zuc1k9ec94MCz8MMw5+1jMu7AuPLbbCVBh0sA2mVejj9Hf95JLryG2uZAZvWumSE7M 3Dk6O0g5QfaBTvBcSrqs+RQGC/zy3FmgfB/03wSZZpAgTvV/FUbbOYdVC+4lK9lWEv4Q nW2NqG4G3lSBT9K4iwFqjPTexdDPAlolhTX/LIf/uhzhE0I/t81OccgfzflfQiGX3J95 xJhBq3CPOM1tM3MYx5O0F2gfMhxLR1i1hHhn5O+plBtKwNiDChFvlUeRNi8ianwAQyHA litHAf2TxntTsRUtmi3qx3OueCB8dnYMA9KauxOuPDCfpO+4N37HO/YnFiqSRETi4tEl 9zlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=RpQbKCVYolg2YM51njqzYl4akm03N6MZjxdg9LiEnVc=; b=KJVEiOlqsBM03HAFwVf1JL/NCMF6KwswPJdPgrTWeuyEtExlrCRZinEuPbw8pIaZtS AIpU5HUN7+fYNx1yaomFOBzzeHo/dgIES83EUUuJCe+ewBEMylVaE8AYiphxLFNULqaS 9Sle7wmqEbjCaddLJwwsGqOM8WYfWqGG+hdz8E83tQ6bhdSmjaf6ApNUKAkWG0SiACwT tYT6XiB9yd57oVhTlqr7hIXeQGlw7txfOndALYjThlz4CnlCx2K8+zzykI+mmWDgNN71 Q+0xRovUZaxa+JtKGVydquQ94KtnxIRjDZv/n4PGWD/FCN40kOEdnde2ufKQnYXsDwhw eoIw== X-Gm-Message-State: AOAM532eDzd4qy1pdUiLYg2dPVtat9OTLrSAYFkrX5WJai1JEuXD8Ybi LZppxcEMaT/OQqeI6bKllDBYAbHFPo7hRw880+3K3g== X-Google-Smtp-Source: ABdhPJzTEElgZcJrL+fXSGdvCXXP0RD2j/Wz9lM/moGyrmva+1KgO0qRmNRI0mGtfw8dtYshKO8UnQm3UeaFBSCuotA= X-Received: by 2002:a05:622a:3c8:: with SMTP id k8mr14955222qtx.101.1618006534686; Fri, 09 Apr 2021 15:15:34 -0700 (PDT) MIME-Version: 1.0 References: <21c7313e-315c-ec48-9437-e0a3d4ec14d2@FreeBSD.org> <41924e9d-9d61-6646-6c8f-e4458f94296e@FreeBSD.org> <30f529c1-6087-e704-8cc7-0c48a40b7430@FreeBSD.org> In-Reply-To: <30f529c1-6087-e704-8cc7-0c48a40b7430@FreeBSD.org> From: Warner Losh Date: Fri, 9 Apr 2021 16:15:23 -0600 Message-ID: Subject: Re: OpenZFS branch tracking policy To: Martin Matuska Cc: Ed Maste , freebsd-git , Xin Li , Ryan Moeller , Alexander Motin , Mateusz Guzik X-Rspamd-Queue-Id: 4FHCBH4sDrz4hn2 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=Zuc1k9ec; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::82d) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; MAILMAN_DEST(0.00)[freebsd-git]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::82d:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; NEURAL_HAM_SHORT(-1.00)[-0.998]; RCPT_COUNT_SEVEN(0.00)[7]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::82d:from]; R_SPF_NA(0.00)[no SPF record]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::82d:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Apr 2021 22:15:36 -0000 On Fri, Apr 2, 2021 at 6:44 PM Martin Matuska wrote: > I have prepared an example merged branch here: > https://github.com/mmatuska/freebsd-src/tree/openzfs_master_merged > > The magical command was: > git merge -s subtree -Xsubtree="sys/contrib/openzfs" 891568c99 > --allow-unrelated-histories > > Luckily, our current diff is manageable. > So I did this for bzip2 using approximately: git add remove bzip2 git fetch bzip2 git merge -s subtree -Xsubtree=contrib/bzip2 bzip2/master --allow-unrelated-histories --squash [1] At this point I resolved conflicts, where were the entire files since I guess I didn't bootstrap right to the last merge. There were 4 files in conflict. Then I did a git add of all the files in conflict and a git commit. This produced a good commit. since it was a squash commit, there were no issues. However, it turns out I botched the commit at point [1] above. So I ran this again and got a conflict for the whole file that I'd removed a blank line from. So, this looks like it could be workable, but does lead me to a few questions: (1) How do we do this so that the conflicts aren't add/add conflicts? Is there some way to bootstrap this? (2) Do we need to keep track of the last merge point and use that in merging the next one in? (3) I assume we keep track of FreeBSD diffs in a branch off and we merge that instead of master. (4) What do we do about adjustments to the build that are needed? (5) Do we need to host a FreeBSD-specific repo with this stuff, maybe with tags we don't want widely pushed to ease the next merge? Eg, make this the first case of a 'vendor repo' that we then pull squash commits from so that the vendor repo can track upstream, but not otherwise be pushed to all our users.... Finally, how did you deal with [1] producing so many full-file add/add conflicts? Oh, and what kind of commit message when things merge do you suggest? I rather like your 'bring in hash XXXX branch blah, here's the important highlights' emails and think that would be a good first cut at advice on what to put in these. This suggests the current answer is 'seems doable, but we need to document it and come up with recommendations for how to do it'. Warner On 3. 4. 2021 1:37, Martin Matuska wrote: > > Hi Warner and Ed, > > > > 2.1-release has already been branched. The stable branch policy in > > OpenZFS is somewhat strange, they make a staging branch for each > > patchlevel release, but the commits are continuous. > > > > To have some idea how big the repo history is: > > > > $ git rev-list master --count > > 6662 > > > > $ git rev-list zfs-2.1-release --count > > 6650 > > > > master and zfs-2.1-release have 6650 common commits at the moment > > > > $ git log master | wc -l > > 129868 > > > > (linecount - 4 * revcount) / revcount = linecount / revcount - 4 = > > 15,4938 comment lines per commit on average > > > > Initial commit was made in Feb 26, 2008. > > > > Yearly commit counts: > > > > $ git log master | grep -c -E '^Date:.* 2020 -[0-9]+$' > > 666 > > > > $ git log master | grep -c -E '^Date:.* 2019 -[0-9]+$' > > 535 > > > > $git log master | grep -c -E '^Date:.* 2018 -[0-9]+$' > > 428 > > > > Martin > > > > On 2. 4. 2021 20:15, Warner Losh wrote: > >> > >> > >> On Fri, Apr 2, 2021 at 11:56 AM Ed Maste >> > wrote: > >> > >> On Fri, 2 Apr 2021 at 11:50, Warner Losh >> > wrote: > >> > > >> > We'd always hoped that we'd be able to do subtree merges from > >> upstreams > >> > that use git into FreeBSD. The big worry, though, was that this > >> would > >> > needless bloat the repo with a lot of history. We don't want, > >> for example, > >> > all of LLVM's history in the tree. We'd always anticipated that > >> there'd be > >> > some things we'd just accept the history for, since it is > >> similar in > >> > character to the vendor branches (though of course a bit more). > >> > >> Note that if we do want to avoid bringing in the full history `git > >> subtree merge` supports a `--squash` option. This brings in the > >> set of > >> upstream changes as a single commit, without bringing along the > >> associated history. We will need to do more experimentation to > >> confirm > >> that the full process, including bootstrapping, will work as we > >> want. > >> Assuming this all works it should allow us to forgo the use of a > >> FreeBSD-specific vendor branch in src. > >> > >> We've discussed mirroring any such 3rd-party source in some > >> FreeBSD-controlled repository. This would allow the project to > >> retain > >> a full copy of the history, but avoid bloating src with it. > >> > >> I agree with Warner that we may want a different policy (full > >> history > >> or snapshots) for different contrib sources. > >> > >> > >> Good points Ed. I'd forgotten about --squash. > >> > >> Martin, what's your timeline for wanting to implement these things? > >> I'm unfamiliar with the OpenZFS schedules. > >> > >> Warner > > _______________________________________________ > > freebsd-git@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-git > > To unsubscribe, send any mail to "freebsd-git-unsubscribe@freebsd.org" >