Date: Thu, 20 Oct 2016 18:23:44 +0000 (UTC) From: Ryan Stone <rstone@FreeBSD.org> To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r307680 - in head/tools/tools: . git Message-ID: <201610201823.u9KINiwr031764@repo.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: rstone Date: Thu Oct 20 18:23:44 2016 New Revision: 307680 URL: https://svnweb.freebsd.org/changeset/base/307680 Log: Add some tools to simplify the use of git Add some scripts that wraps some FreeBSD Project infrastructure and simplifies using them with git. The scripts are: - arcgit, which creates a series of reviews in Differential - importgit, which applies a series of git commits to svn Differential Revision: https://reviews.freebsd.org/D2071 Added: head/tools/tools/git/ head/tools/tools/git/HOWTO (contents, props changed) head/tools/tools/git/arcgit (contents, props changed) head/tools/tools/git/importgit (contents, props changed) Modified: head/tools/tools/README Modified: head/tools/tools/README ============================================================================== --- head/tools/tools/README Thu Oct 20 17:28:52 2016 (r307679) +++ head/tools/tools/README Thu Oct 20 18:23:44 2016 (r307680) @@ -29,6 +29,7 @@ gdb_regofs A simple tool that prints out platforms only. genericize Turn a kernel config into something that can more easily be diffed against the appropriate GENERIC. +git Tools to simplify the use of git by committers. hcomp Compress header files by removing comments and whitespace. html-mv Rename HTML generated filenames to human readable filenames. ifinfo Uses the interface MIB to print out all the information Added: head/tools/tools/git/HOWTO ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/tools/tools/git/HOWTO Thu Oct 20 18:23:44 2016 (r307680) @@ -0,0 +1,144 @@ +# $FreeBSD$ + +This directory contains tools intended to help committers use git when +interacting with standard FreeBSD project resources like Differential or svn. + +I. arcgit + +arcgit is a wrapper script around the arc command line tool that simplifies the +automatic creation of a series of Differential reviews from a series of git +commits. The intended workflow is: + +1. Create a series of commits in git. Each commit will be a separate review, so + try to make each commit as self-contained as possible. The series of commits + should demonstrate a logical progression towards your end goal. For example, + one commit might refactor existing code to expose a new API without changing + any current functionality. A subsequent commit could then introduce your new + code that uses the new API. + + It usually will not be helpful to present your code in the order in which it + was actually written and can actually be harmful. For example, if you + introduced a bug early in your development process that you fixed in a + subsequent commit, it is a waste of your reviewer's time to have them review + old code with known bugs. Instead, it would probably be best to squash the + bug fix into the commit that introduced it, so that the bug is never + presented to your reviewers in any review. + + The commit headline and commit message will be imported verbatim into + Differential, so try to give each commit a meaningful commit message that + gives your reviewers the necessary context to understand your change. + +2. Create your reviews bu running this command in your git repo: + $ arcgit -r C1~..C2 -R reviewer -T testplan + + C1 should be the first commit that you want reviewed, and C2 should be the + last commit that you want reviewed. You may add multiple reviewers by + specifying the -R option multiple times. You can CC (AKA subscribe) people + to a review with the -C option. Note that if you subscribe a mailing list + to a review, the mailing list will be emailed for every comment or change + made to each review. Please be judicious when subscibing mailing lists to + reviews. It may be better to instead send a single email to the appropriate + list announcing all of the reviews and giving a short summary of the change + as a whole, along with a link to the individual reviews. + +3. When you need to make a change and upload it to a review, use the following + process. First, check out the branch corresponding to the review (arcgit + automatically creates this branch for every review that it creates): + + $ git checkout review_D1234 + + Next, make your change and perform whatever testing is necessary. Commit it + to your repository with this command: + + $ git commit --fixup HEAD + + You can upload the change to the code review by running this command in your + repository while (ensure that you are still on the review_D1234 branch): + + $ arc diff --update D1234 review_D1234_base + + When you run arc, it will pull up your editor and give you a chance to + change the message that will be shown in Differential for this upload. It's + recommended that you change it from the default "fixup! ...." as that does + not give your reviewers an indication of what changes were made. It's not + recommended that you give the code review fixes meaningful commit messages + directly because we will be using git's autosquash feature in the next step, + which depends on the fixup! tag being present. + + Do not merge in other branches, or rebase your review branches at this point. + Any changes that are made will show up in your reviews, and that will create + extra noise that your reviewers will have to ignore. If a reviewer requests + a change that requires your commit to be based off of a later version of + head, I would suggest deferring the change from the review and creating a + new review with the change once you hit step 5. + +4. Once the reviews have been approved, you need to prepare your patch series + to be committed. This involves squashing the fixes made in code review + back into the original commit that they applied to. This gives you a clean + series of commits that are ready to be commited back to svn. + + First, merge each of your review branches back into your main development + branch. For example: + + $ git checkout myfeature_dev + $ for branch in review_D1234 review_D1235 review_D1236; do \ + git merge $branch || git mergetool -y || break; done + + Next, do an interactive rebase to squash your code review fixes back into the + main commit: + + $ git rebase -i -k review_D1234_base + + review_D1234 should be the name of the *first* review that was created for + you in step 2. For every commit, your editor will be pulled up and you will + be given one last chance to edit your commit message. Make sure that you fill + in the "Reviewed by:" tag indicating who accepted the review. This would + be a good point to add other tags like MFC after:, Sponsored by:, etc. + + The rebase will not introduce any actual changes to your code if done + properly. You can use this command to double-check that no changes were + inadvertently made here: + + $ git diff myfeature_dev@{1} + +5. Finally, you should get up to date with the latest changes from head: + + $ git pull --rebase origin master + + It is not recommended that you combine this step with the rebase done in the + previous step. The reason for this is that if you perform an interactive + rebase that changes the commit that you branch from, it becomes much more + difficult to use the reflog to confirm that the interactive rebase did not + introduce unwanted changes. + + At this point, you are ready to commit your changes to head. The importgit + script can be used to import your commits directly into git. + +II. importgit + +importgit is a script that can take a series of commits from git and commit them +to a svn repository. The script uses the git commit messages for the svn commit +message, which allows importgit to be fully automated. This does mean that once +you start importgit, it will start commit things to svn without giving any +further chance to sanity check what it's doing. + +importgit only supports importing commits that add or modify files. It does not +support importing commits that rename or delete files, to ensure that git's +rename detection heuristics do not introduce an error in the import process. +importgit also does not support importing merge commits. Only linear history +can be imported into svn. + +importgit must be run from a clean subversion checkout. You should ensure that +the working tree is up-to-date with "svn up" before running importgit. +importgit will run svn directly, so make sure that your ssh-agent is running +and has your ssh key loaded already. Run importgit as follows: + + $ importgit -r D1~..D2 -g /path/to/git/repo + +This will import every commit between D1 and D2, including both D1 and D2. The +invocation is very similar to the invocation given to arcgit but there is an +important point to note. When you rebased your commits as you followed steps 4 +and 5, the commit hashes of all of your commits changed, including C1 and C2. +You must go back and find the new commit hashes of your commits to pass to +importgit. Passing -r C1~..C2 would import your commits as they were *before* +your code review fixes were applied. Added: head/tools/tools/git/arcgit ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/tools/tools/git/arcgit Thu Oct 20 18:23:44 2016 (r307680) @@ -0,0 +1,214 @@ +#!/bin/sh +# +# Copyright (c) 2015 Ryan Stone. All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE +# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. +# +# $FreeBSD$ + +# This script is used to submit a series of git commits to Differential. Each +# commit is submitted as a separate review. For each review, this script will +# create a branch called review_DXXXX (e.g. review_D2185 for Differential +# revision D2185). When you need to make a change to a review, checkout the +# review_D2185 branch, commit your change with "git commit --fixup HEAD". To\ +# upload the change to Differential, use the command: +# $ arc diff --update D2185 review_D2185_base +# +# When your reviews are complete, merge all of the review_DXXXX branches +# together, and then do a git rebase -ik to meld the code review fixes into the +# commit that they fixed. Now you have a clean series of patches to commit to +# svn. + +usage() +{ + echo "Usage: arcgit <-c commit | -r commit1~..commit2> [-R reviewer] " >&2 + echo " [-C subscriber] [-T testplan] [-n]" >&2 +} + +error() +{ + echo "$@" >&2 + usage + rm -f $phab_before $phab_after $arc_msg + exit 1 +} + +create_review() +{ + local commit phab_id arc_dir + unset phab_before phab_after arc_msg + commit=$1 + + phab_before=`mktemp -t arcoutput` + phab_after=`mktemp -t arcoutput` + echo "Create review for '`git show $commit -s --oneline`'" + + if [ -n "$dry_run" ] + then + return + fi + + git checkout $commit > /dev/null || error "Could not checkout $commit" + + arc_dir="$(git rev-parse --show-toplevel)/.git/arc" + arc_msg="$arc_dir/create-message" + mkdir -p $arc_dir + git show -s --format='%B' HEAD > $arc_msg + echo >> $arc_msg + echo "Test Plan:" >> $arc_msg + cat $test_plan >> $arc_msg + echo >> $arc_msg + echo "Reviewers:" >> $arc_msg + echo "$reviewers" >> $arc_msg + echo >> $arc_msg + echo "Subscribers:" >> $arc_msg + echo "$cc_list" >> $arc_msg + echo >> $arc_msg + + arc list > $phab_before + yes | env EDITOR=true arc diff --create --allow-untracked HEAD~ + arc list > $phab_after + + headline="$(git show $commit -s --format=%s)" + phab_id=`comm -13 "$phab_before" "$phab_after" | fgrep "$headline" \ + | egrep -o 'D[0-9]+:' | tr -d ':'` + + if [ -z "$phab_id" ] + then + error "Could not get review ID" + fi + + git branch review_${phab_id}_base HEAD~ + + git checkout -b review_$phab_id + cat - <<EOF | git commit --allow-empty -F - +squash! $headline + +Differential Revision: https://reviews.freebsd.org/$phab_id +Reviewed by: +EOF +} + +unset range test_plan reviewers cc_list dry_run + +while getopts ":c:C:nr:R:T:" o +do + case "$o" in + c) + range="${OPTARG}~..${OPTARG}" + ;; + C) + if [ -z "$cc_list" ] + then + cc_list="$OPTARG" + else + cc_list="$cc_list, $OPTARG" + fi + ;; + n) + dry_run=1 + ;; + r) + range=$OPTARG + ;; + R) + if [ -z "$reviewers" ] + then + reviewers="$OPTARG" + else + reviewers="$reviewers, $OPTARG" + fi + ;; + T) + test_plan=$OPTARG + ;; + *) + error "Unrecognized argument '-$OPTARG'" + esac +done + +shift $((OPTIND - 1)) +OPTIND=1 + +if [ -n "$1" ] +then + error "Unrecognized argument $1" +fi + +if [ -z "$range" ] +then + error "-c or -r argument is mandatory" +fi + +if [ -n "$test_plan" -a ! -r "$test_plan" ] +then + error "$test_plan is not readable" +fi + +if ! type git > /dev/null 2> /dev/null +then + error "Install devel/git first" +fi + +if ! type arc > /dev/null 2> /dev/null +then + error "Install devel/arcanist first" +fi + +git update-index -q --refresh +if ! git diff-index --quiet --cached HEAD +then + error "index is unclean" +fi + +if ! git diff-files --quiet +then + error "Working directory is unclean" +fi + +if git ls-files --other --error-unmatch . > /dev/null 2> /dev/null +then + error "Working directory contains untracked files" +fi + +# We have to do a git checkout in order to run arc, so save the original branch +# so that we can check it out again once we're done. +if ! orig_branch=$(git symbolic-ref --short -q HEAD) +then + orig_branch=$(git show -s --pretty='%H' HEAD) +fi + +git log --format=%H $range | tail -r | while read -r commit +do + create_review $commit < /dev/null +done + +# Note that due to the use of the pipeline above, the body of the while loop +# above runs in a subshell. If it exits with an error, execution resumes +# here rather than exiting the script, so we have to cache the right exit code +# and return it when we're done cleaning up. +code=$? + +git checkout $orig_branch + +exit $code + Added: head/tools/tools/git/importgit ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/tools/tools/git/importgit Thu Oct 20 18:23:44 2016 (r307680) @@ -0,0 +1,182 @@ +#!/bin/sh +# +# Copyright (c) 2015 Ryan Stone. All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE +# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. +# +# $FreeBSD$ + +usage() +{ + echo "Usage: importgit <-c commit | -r c1..c2> -g /path/to/git/repo [-n]" >&2 +} + +error() +{ + local print_usage + + if [ "$1" = "-u" ] + then + shift + print_usage=1 + else + print_usage= + fi + + echo "$@" >&2 + if [ -n "$print_usage" ] + then + usage + fi + exit 1 +} + +unset git range commit dry_run + +while getopts ":c:g:nr:" o +do + case "$o" in + c) + range="${OPTARG}~..${OPTARG}" + ;; + g) + git_repo=$OPTARG + ;; + n) + dry_run=1 + ;; + r) + range=$OPTARG + ;; + *) + error -u "Unrecognized argument '-$OPTARG'" + esac +done + +shift $((OPTIND - 1)) +OPTIND=1 + +if [ -n "$1" ] +then + error -u "Unrecognized argument $1" +fi + +if [ -z "$range" ] +then + error -u "-c or -r argument is mandatory" +fi + +if ! echo "$range" | egrep -qs '^[^.]+\.\.[^.]*$' +then + error -u "$range is not a range of commits. Did you mean '-c $range'?" +fi + +if [ -z "$git_repo" ] +then + error -u "-g <repo> argument is mandatory" +fi + +git="$git_repo/.git" + +if [ ! -d "$git" ] +then + error "$git_repo does not seem to be a git repo" +fi + +if ! type git > /dev/null 2> /dev/null +then + error "Install devel/git first" +fi + +if ! type svn > /dev/null 2> /dev/null +then + error "Install devel/subversion first" +fi + +if [ -n "$(svn status)" ] +then + error "Working tree is not clean" +fi + +if ! svn --non-interactive ls > /dev/null +then + error "Could not communicate with svn server. Is your ssh key loaded?" +fi + +git --git-dir=$git log --format=%H $range | tail -r | while read -r commit +do + echo "Applying `git --git-dir=$git show -s --oneline $commit`" + + if [ -n "$(git --git-dir=$git show --diff-filter=CDRTUXB $commit)" ] + then + error "Commit performed unsupported change (e.g. delete/rename)" + fi + + if [ "$(git --git-dir=$git show -s --format=%P $commit | wc -w)" -ne 1 ] + then + error "Cannot import merge commits" + fi + + git --git-dir=$git diff --diff-filter=A --name-only \ + ${commit}~..$commit | while read -r newfile + do + if [ -f "$newfile" ] + then + error "New file $newfile already exists in tree" + fi + done + + # The previous while loop ran in a subshell, so we have to check if it + # exited with an error and bail out if so. + ret=$? + if [ "$ret" -ne 0 ] + then + exit $ret + fi + + if [ -n "$dry_run" ] + then + continue + fi + + git --git-dir=$git show $commit | patch -p 1 -s || \ + error "Failed to apply patch" + + git --git-dir=$git diff --diff-filter=A --name-only \ + ${commit}~..$commit | while read -r newfile + do + svn add --parents --depth=infinity $newfile || \ + error "Failed to add new file" + done + + # The previous while loop ran in a subshell, so we have to check if it + # exited with an error and bail out if so. + ret=$? + if [ "$ret" -ne 0 ] + then + exit $ret + fi + + git --git-dir=$git show -s --format='%B' $commit | svn commit -F - || \ + error "Failed to commit" +done +
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201610201823.u9KINiwr031764>