From owner-freebsd-git@freebsd.org Tue May 18 17:49:52 2021 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 964FC652BB3 for ; Tue, 18 May 2021 17:49:52 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qk1-x729.google.com (mail-qk1-x729.google.com [IPv6:2607:f8b0:4864:20::729]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Fl3Rg6z0Lz3mqX; Tue, 18 May 2021 17:49:51 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-qk1-x729.google.com with SMTP id 76so10083410qkn.13; Tue, 18 May 2021 10:49:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=IGEAw1+MEvcY4icFkNgG8X9BU0gYd2cG+FmRmPh8Rk0=; b=PwGnlOlvPZx6LatTQ+AsShxZb/UQI84btYpHuXVznu8CObYk8tkAnnpN1YJ2Y6AaC9 wQo1V67oceg/bgs6kyC8JuBfohS3rIf5atw6wezapYBJJ4+huR4AWm4DDV8ug7N2rUFn XlZuM6DMF/vP3AXpqZ+9Vmu7v3XFbcPsY0B5LC/VnX/c5ebH3zRX6GxM5FZOoEbhIXvZ lZ6PexhcIkjSdnr26DajOY+Bp4Y5zGJwX/zXKSLxb8hgpY77OiudqYqCOyyzTDSiLx59 tINutCOgd/CI320hH0D+q1CFptf66oxaP3UfWNepnh1rR7pWIVCRlcr8uZ4+E/XCJh5w BVnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition :content-transfer-encoding:in-reply-to; bh=IGEAw1+MEvcY4icFkNgG8X9BU0gYd2cG+FmRmPh8Rk0=; b=k4hr+GAOJRXvfMJih/B385yzBT/5GAYnpoQE1Ln3IuuWxxWe3Ca/kitnGsC4G/J+XC eqKGyb1iIPCLul0OkGLhNCOSnjyLjOAFu/JAywJ9aTShJvtD+c42bfNjyju1B2z4NRLb zmMmD8a4bPb6QGI2vP1ofXolUdMbiSQlN8rLdO68x9pEm4zcqLTKrX0S9fYGH5HeRJhA xhUjRZ8UobNl8TYZUJrG33qtFTts3zk1kiW/ZycAunp4kIQc6OBA1y0uulBCDvKNzT56 97epDdaFkDr2iMJFQD9UWqrEf1K4sEoASJBjiXqeGAa7Tub8qhhA4d/oLo3yikmdhoYs exWQ== X-Gm-Message-State: AOAM532OAK7ofKjJRSNmYYN4iOcjER+Bs2pGcEItgNsyXB++78tNCjsz l8ohYV68W69J+//3dJP791rKw6EG0VhpaA== X-Google-Smtp-Source: ABdhPJxSmvcEBZX3H3pC0vb+fiC2SNamfooLSnnfSsJ5pV9OvZntV11DV9oaX6DjMPA0v/n07tFd3Q== X-Received: by 2002:a37:6388:: with SMTP id x130mr6917881qkb.485.1621360190758; Tue, 18 May 2021 10:49:50 -0700 (PDT) Received: from nuc ([142.126.159.38]) by smtp.gmail.com with ESMTPSA id 75sm13073297qkl.119.2021.05.18.10.49.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 May 2021 10:49:50 -0700 (PDT) Sender: Mark Johnston Date: Tue, 18 May 2021 13:46:47 -0400 From: Mark Johnston To: Ulrich =?iso-8859-1?Q?Sp=F6rlein?= Cc: freebsd-git@freebsd.org Subject: Re: vendor/illumos merges Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Queue-Id: 4Fl3Rg6z0Lz3mqX X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=PwGnlOlv; dmarc=none; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::729 as permitted sender) smtp.mailfrom=markjdb@gmail.com X-Spamd-Result: default: False [-0.64 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.95)[-0.948]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::729:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.98)[-0.980]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[freebsd.org]; NEURAL_SPAM_MEDIUM(0.99)[0.985]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::729:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::729:from]; MID_RHS_NOT_FQDN(0.50)[]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-git] X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 May 2021 17:49:52 -0000 On Sun, Apr 25, 2021 at 04:02:22PM +0200, Ulrich Spörlein wrote: > On Sat, 2021-04-24 at 11:08:58 -0400, Mark Johnston wrote: > >On Sat, Apr 24, 2021 at 12:44:40PM +0200, Ulrich Spörlein wrote: > >> On Fri, 2021-04-23 at 17:26:33 -0400, Mark Johnston wrote: > >> >Hi, > >> > > >> >Now that FreeBSD uses OpenZFS as the upstream for ZFS, vendor/illumos is > >> >mostly unused. However, we still use illumos as an upstream for CTF > >> >tools and DTrace, though there haven't been any imports in a while. > >> > > >> >illumos has put a lot of work into their CTF toolchain, and I'd like to > >> >import that. There are a couple of snags that I'd appreciate some > >> >guidance on. > >> > > >> >First, I believe I should delete now-unused ZFS code from the vendor > >> >branch and merge the result to main. I did this locally and got an > >> >empty merge, which is what I'd expect. Is there any problem with this? > >> > >> Why would you record this empty merge? If you clean up vendor/foo, just > >> do that but don't merge a no-op back into main (nothing changed, after > >> all). > > > >Ok, I guess there is no reason to merge that change separately. It > >will end up being merged with subsequent imports though. > > > >> >Second, with Subversion we had both vendor/illumos and > >> >vendor-sys/illumos, and now we just have the former, seemingly with > >> >sys/* bits imported from vendor-sys. Some of the upstream commits touch > >> >both userspace and kernel bits, but the merge targets for these in > >> >FreeBSD are different: cddl/contrib/opensolaris vs. > >> >sys/cddl/contrib/opensolaris. How should I merge into main in this > >> >case? I don't really see any options other than to split each offending > >> >upstream commit into two parts, one for userspace and one for the > >> >kernel, and merge them separately. > >> > > >> >If it helps to look at the branch where I staged the upstream commits, > >> >I've pushed it to vendor/illumos2 in https://github.com/markjdb/freebsd > >> >. > >> > >> Can you clarify why the merging of the two might be an issue? Note that > >> unlike subversion, in git there's no "merge a certain subtree" handling, > >> all that is recorded is a tree of some form and then a set of parents or > >> ancestor commits. (git is a content tracker, not really a VCS :) > >> > >> I was under the impression that userland and kernel imports/merges need > >> to happen at the same time anyway, so I assume you would import all the > >> bits under vendor/foo in 1 commit and then merge them in 1 commit into > >> main. Is that not how it goes? > > > >How can I do that with git subtree merge? Suppose an illumos commit > >modifies cmd/dtrace/foo.c (userspace) and uts/common/dtrace/foo.c > >(kernel). That maps to cddl/contrib/opensolaris/cmd/dtrace/foo.c and > >sys/cddl/contrib/opensolaris/uts/common/dtrace/foo.c in FreeBSD, > >respectively. So to do a subtree merge, I need to use distinct prefixes > >depending on whether I'm importing userspace or kernel changes. When > >they are mixed together, it's not clear to me how I can merge at all. > > > >I see that for OpenZFS we keep all code, including userspace code, under > >sys/contrib/openzfs, so it doesn't have this problem. > > I don't think you want a subtree merge, especially as things are > scattered all over the place. Also note that none of this subtree magic > is in any way recorded in the git data, all it does is help you with the > 3-way merges (or whatever). > > So I would do: > - import whatever you need into contrib/foo, commit normally. > - munge /usr/src to have every kernel and userland stuff (not sure what > other merge tools exist, just make sure to copy over file deletions as > well :). You could rsync --del two times with the right source/dest > pairs, or export a diff/patch from step 1 and apply it under the right > prefixes. test, test, test. > - write out this tree to git using: git write-tree > - then commit this using: git commit-tree -m "my message" -p HEAD -p > origin/vendor/illumos > - bump main to point to that hash using git update-ref > - git log --graph and inspect the hell out of this > - git push, then curse that we disallow merge commits and you need to > `git pull --rebase` to advance to the latest published head and that > might mess up your merge commit pretty bad :( > > > Maybe 2x git subtree merge + then rewriting and squashing them into 1 > would work. But I fear it will record 3 parents, not 2 parents. > > Whatever you do, maybe please push to your private Github clone or our > dev repo first and tell us where to look, so we can inspect whether it > looks ok. I followed your suggestions and have what looks like a clean result. Basically I did two subtree merges and committed the result. I pushed it here: https://github.com/markjdb/freebsd/tree/ff/illumos-merge The merge is the second-last commit on that branch. Rebasing the merge is painful, partly because there are some old ZFS commits in the illumos vendor branch which git tries to merge. Rename detection leads to some really weird conflicts as well. I'm not sure how to handle this: there's a lot of room to make mistakes when rebasing, so I would want to be careful and do extra testing before pushing the final merge, but during that window it's likely that I will end up having to rebase again. I should perhaps remove all ZFS bits from the vendor branch, and merge it into main first before importing other stuff.