From owner-svn-src-all@freebsd.org Thu Nov 26 14:42:17 2020 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 23F2E4AE140; Thu, 26 Nov 2020 14:42:17 +0000 (UTC) (envelope-from uqs@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ChgT50bmZz4YwX; Thu, 26 Nov 2020 14:42:17 +0000 (UTC) (envelope-from uqs@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 0665324E9C; Thu, 26 Nov 2020 14:42:17 +0000 (UTC) (envelope-from uqs@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 0AQEgGfo006731; Thu, 26 Nov 2020 14:42:16 GMT (envelope-from uqs@FreeBSD.org) Received: (from uqs@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 0AQEgG80006729; Thu, 26 Nov 2020 14:42:16 GMT (envelope-from uqs@FreeBSD.org) Message-Id: <202011261442.0AQEgG80006729@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: uqs set sender to uqs@FreeBSD.org using -f From: =?UTF-8?Q?Ulrich_Sp=c3=b6rlein?= Date: Thu, 26 Nov 2020 14:42:16 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r368056 - in head: .github/workflows tools/build X-SVN-Group: head X-SVN-Commit-Author: uqs X-SVN-Commit-Paths: in head: .github/workflows tools/build X-SVN-Commit-Revision: 368056 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Nov 2020 14:42:17 -0000 Author: uqs Date: Thu Nov 26 14:42:16 2020 New Revision: 368056 URL: https://svnweb.freebsd.org/changeset/base/368056 Log: GH Actions: Use pre-installed clang packages Also fix the run by setting up the environment in non-deprecated way. Always run with --debug to understand better what sort of stuff is happening in the background. Also split out the bmake bootstrap stage (takes about 31s on ubuntu, but 1m14 on macOS?) Drops the dependency on coreutils (realpath, nproc) and thus (?) fixes macOS to be just as fast (4 logical cores vs 2 physical cores before, go figure.) Reviewed by: arichardson Modified: head/.github/workflows/cross-bootstrap-tools.yml head/tools/build/make.py Modified: head/.github/workflows/cross-bootstrap-tools.yml ============================================================================== --- head/.github/workflows/cross-bootstrap-tools.yml Thu Nov 26 13:31:57 2020 (r368055) +++ head/.github/workflows/cross-bootstrap-tools.yml Thu Nov 26 14:42:16 2020 (r368056) @@ -1,4 +1,4 @@ -name: Cross-build CI +name: Cross-build Kernel on: push: @@ -8,28 +8,51 @@ on: jobs: build: - name: ${{ matrix.os }} + name: ${{ matrix.os }} (${{ matrix.compiler }}) runs-on: ${{ matrix.os }} strategy: + fail-fast: false matrix: - os: [ubuntu-18.04, ubuntu-20.04, macOS-latest] + include: + # TODO: both Ubuntu and macOS have bmake packages, we should try them instead of bootstrapping our own copy. + - os: ubuntu-20.04 + compiler: clang-9 + cross-bindir: /usr/lib/llvm-9/bin + pkgs: bmake libarchive-dev + - os: ubuntu-20.04 + compiler: clang-10 + cross-bindir: /usr/lib/llvm-10/bin + pkgs: bmake libarchive-dev + - os: macOS-latest + compiler: clang-11 + #cross-bindir: /usr/local/Cellar/llvm/11.0.0/bin # script figures this out automatically + pkgs: bmake libarchive llvm@11 steps: - uses: actions/checkout@v2 - - name: install LLVM+libarchive (Ubuntu) + - name: install packages (Ubuntu) + if: runner.os == 'Linux' run: | - wget -O /tmp/llvm.sh https://apt.llvm.org/llvm.sh - chmod +x /tmp/llvm.sh - sudo /tmp/llvm.sh 11 - sudo apt install -y libarchive-dev - echo "::set-env name=EXTRA_MAKE_ARGS::--cross-bindir=/usr/lib/llvm-11/bin" - if: ${{ startsWith(matrix.os, 'ubuntu') }} - - name: install LLVM+libarchive (macOS) - run: brew install llvm coreutils libarchive xz - if: ${{ startsWith(matrix.os, 'macOS') }} - - name: create build dir - run: rm -rf ../build && mkdir -p ../build + sudo apt-get update --quiet || true + sudo apt-get -yq --no-install-suggests --no-install-recommends install ${{ matrix.pkgs }} + - name: install packages (macOS) + if: runner.os == 'macOS' + run: | + brew update --quiet || true + brew install ${{ matrix.pkgs }} + - name: create environment + run: | + echo "GITHUB_WORKSPACE = $GITHUB_WORKSPACE" + if [ -n "${{ matrix.cross-bindir }}" ]; then + echo "EXTRA_BUILD_ARGS=--cross-bindir=${{ matrix.cross-bindir }}" >> $GITHUB_ENV + fi + mkdir -p ../build + echo "MAKEOBJDIRPREFIX=${PWD%/*}/build" >> $GITHUB_ENV + # heh, works on Linux/BSD/macOS ... + echo "NPROC=`getconf _NPROCESSORS_ONLN 2>/dev/null || getconf NPROCESSORS_ONLN 2>/dev/null || echo 1`" >> $GITHUB_ENV + - name: bootstrap bmake + run: ./tools/build/make.py --debug $EXTRA_BUILD_ARGS TARGET=amd64 TARGET_ARCH=amd64 -n - name: make kernel-toolchain - run: env MAKEOBJDIRPREFIX=`realpath ../build` ./tools/build/make.py $EXTRA_MAKE_ARGS TARGET=amd64 TARGET_ARCH=amd64 kernel-toolchain -s -j$(nproc) + run: ./tools/build/make.py --debug $EXTRA_BUILD_ARGS TARGET=amd64 TARGET_ARCH=amd64 kernel-toolchain -s -j$NPROC - name: make buildkernel - run: env MAKEOBJDIRPREFIX=`realpath ../build` ./tools/build/make.py $EXTRA_MAKE_ARGS TARGET=amd64 TARGET_ARCH=amd64 KERNCONF=GENERIC NO_MODULES=yes buildkernel -s -j$(nproc) + run: ./tools/build/make.py --debug $EXTRA_BUILD_ARGS TARGET=amd64 TARGET_ARCH=amd64 KERNCONF=GENERIC NO_MODULES=yes buildkernel -s -j$NPROC $EXTRA_MAKE_ARGS Modified: head/tools/build/make.py ============================================================================== --- head/tools/build/make.py Thu Nov 26 13:31:57 2020 (r368055) +++ head/tools/build/make.py Thu Nov 26 14:42:16 2020 (r368056) @@ -115,6 +115,9 @@ def check_required_make_env_var(varname, binary_name, " does not exist") new_env_vars[varname] = guess debug("Inferred", varname, "as", guess) + global parsed_args + if parsed_args.debug: + run([guess, "--version"]) def default_cross_toolchain(): @@ -188,6 +191,9 @@ if __name__ == "__main__": if parsed_args.host_compiler_type == "gcc": default_cc, default_cxx, default_cpp = ("gcc", "g++", "cpp") + # FIXME: this should take values like `clang-9` and then look for + # clang-cpp-9, etc. Would alleviate the need to set the bindir on + # ubuntu/debian at least. elif parsed_args.host_compiler_type == "clang": default_cc, default_cxx, default_cpp = ( "clang", "clang++", "clang-cpp") @@ -235,7 +241,5 @@ if __name__ == "__main__": shlex.quote(s) for s in [str(bmake_binary)] + bmake_args) debug("Running `env ", env_cmd_str, " ", make_cmd_str, "`", sep="") os.environ.update(new_env_vars) - if parsed_args.debug: - input("Press enter to continue...") os.chdir(str(source_root)) os.execv(str(bmake_binary), [str(bmake_binary)] + bmake_args)