From owner-dev-commits-src-all@freebsd.org Fri Oct 1 21:00:01 2021 Return-Path: Delivered-To: dev-commits-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D6C686B5D9B; Fri, 1 Oct 2021 21:00:01 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HLjDK5ZSbz3HgD; Fri, 1 Oct 2021 21:00:01 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id A04B92655A; Fri, 1 Oct 2021 21:00:01 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 191L01jx027166; Fri, 1 Oct 2021 21:00:01 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 191L018T027156; Fri, 1 Oct 2021 21:00:01 GMT (envelope-from git) Date: Fri, 1 Oct 2021 21:00:01 GMT Message-Id: <202110012100.191L018T027156@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Li-Wen Hsu Subject: git: fbece7609573 - main - Update Azure release bits MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: lwhsu X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: fbece7609573bd51080e949df03fa3d803a637ae Auto-Submitted: auto-generated X-BeenThere: dev-commits-src-all@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commit messages for all branches of the src repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Oct 2021 21:00:01 -0000 The branch main has been updated by lwhsu: URL: https://cgit.FreeBSD.org/src/commit/?id=fbece7609573bd51080e949df03fa3d803a637ae commit fbece7609573bd51080e949df03fa3d803a637ae Author: Li-Wen Hsu AuthorDate: 2021-10-01 20:59:10 +0000 Commit: Li-Wen Hsu CommitDate: 2021-10-01 20:59:10 +0000 Update Azure release bits Imports the changes for building official images on Azure Marketplace, which fulfill the requirements of Azure and FreeBSD cloud images like disk layout and UEFI for Gen2 VM, along with some minor improvements like configurations to speed up booting. "CLOUDWARE" list will be updated after some more collaborations with re completed. Reviewed by: re (gjb) Sponsored by: The FreeBSD Foundation Technical assistance from: Microsoft Differential Revision: https://reviews.freebsd.org/D23804 --- release/Makefile.azure | 55 +++++++++++++++++++++++++++++--------------- release/Makefile.vm | 2 +- release/tools/azure.conf | 57 ++++++++++++++++++++++++++++++++++------------ release/tools/vmimage.subr | 7 +++++- 4 files changed, 87 insertions(+), 34 deletions(-) diff --git a/release/Makefile.azure b/release/Makefile.azure index 32e91833b766..b8e7418f35f7 100644 --- a/release/Makefile.azure +++ b/release/Makefile.azure @@ -11,8 +11,8 @@ AZURE_UPLOAD_TGTS= azure-check-depends \ CLEANFILES+= ${AZURE_UPLOAD_TGTS} .if defined(AZURE_UPLOAD_CONF) && !empty(AZURE_UPLOAD_CONF) -. for VAR in _STORAGE _ACCOUNT _KEY -AZURE${VAR}!= grep -E ^AZURE${VAR} ${AZURE_UPLOAD_CONF} | awk -F' ' '{print $$2}' +. for VAR in _STORAGE _ACCOUNT _RESOURCEGROUP _KEY _LOCATION +AZURE${VAR}!= grep -E ^AZURE${VAR}= ${AZURE_UPLOAD_CONF} | awk '{print $$2}' . endfor .endif @@ -20,34 +20,53 @@ AZURE${VAR}!= grep -E ^AZURE${VAR} ${AZURE_UPLOAD_CONF} | awk -F' ' '{print $$2} SNAPSHOT_DATE!= date +-${BUILDDATE} .endif -AZURE_TARGET:= ${OSRELEASE}${SNAPSHOT_DATE}.vhd +AZURE_TARGET:= ${OSRELEASE}${SNAPSHOT_DATE} + +START_DATE!= date -v-1d -I -u +EXPIRY_DATE!= date -v+1m -I -u azure-upload: ${AZURE_UPLOAD_TGTS} azure-check-depends: -.for VAR in _STORAGE _ACCOUNT _KEY +.for VAR in _STORAGE _ACCOUNT _RESOURCEGROUP _KEY . if !defined(AZURE${VAR}) || empty(AZURE${VAR}) @echo "Variable AZURE${VAR} cannot be empty." @false . endif .endfor -.if !exists(/usr/local/bin/azure) -. if !exists(/usr/local/bin/npm) -. if !exists(${PORTSDIR}/www/npm/Makefile) -. if !exists(/usr/local/sbin/pkg-static) +.if !exists(/usr/local/bin/az) +. if !exists(${PORTSDIR}/sysutils/py-azure-cli/Makefile) +. if !exists(/usr/local/sbin/pkg-static) env ASSUME_ALWAYS_YES=yes pkg bootstrap -yf -. endif - env ASSUME_ALWAYS_YES=yes pkg install -y www/npm -. else - env UNAME_r=${UNAME_r} make -C ${PORTSDIR}/www/npm BATCH=1 all install clean . endif + env ASSUME_ALWAYS_YES=yes pkg install -y py38-azure-cli +. else + env UNAME_r=${UNAME_r} make -C ${PORTSDIR}/sysutils/py-azure-cli BATCH=1 all install clean . endif - npm install -g azure-cli .endif azure-do-upload: - /usr/local/bin/azure storage blob upload \ - ${AZURE_IMG} ${AZURE_STORAGE} ${AZURE_TARGET} \ - -t page -a ${AZURE_ACCOUNT} -k "${AZURE_KEY}" - touch ${.OBJDIR}/${.TARGET} - + /usr/local/bin/az storage blob upload \ + --account-name ${AZURE_ACCOUNT} --account-key ${AZURE_KEY} \ + --container-name ${AZURE_STORAGE} --type page \ + --file ${AZURE_IMG} --name ${AZURE_TARGET}.vhd + @echo "The disk access URL with shared access signature is:" + @echo + @echo -n https://${AZURE_ACCOUNT}.blob.core.windows.net/${AZURE_STORAGE}/${AZURE_TARGET}.vhd? + @/usr/local/bin/az storage container generate-sas \ + --account-name ${AZURE_ACCOUNT} --account-key ${AZURE_KEY} \ + --name ${AZURE_STORAGE} \ + --permissions lr \ + --start ${START_DATE} \ + --expiry ${EXPIRY_DATE} | cut -d '"' -f 2 + @echo + @echo "Please go to Microsoft Partner Center to create a new offer and publish it:" + @echo + @echo "https://partner.microsoft.com/dashboard/commercial-marketplace/overview" + @echo + @echo "After the new offer status is live, you can delete the disk file with:" + @echo + @echo /usr/local/bin/az storage blob delete \ + --account-name ${AZURE_ACCOUNT} --account-key ${AZURE_KEY} \ + --container-name ${AZURE_STORAGE} --name ${AZURE_TARGET}.vhd + @echo diff --git a/release/Makefile.vm b/release/Makefile.vm index 2240a1e5fc57..72b150022ef7 100644 --- a/release/Makefile.vm +++ b/release/Makefile.vm @@ -174,8 +174,8 @@ cloudware-release: ${MAKE} -C ${.CURDIR} ${.MAKEFLAGS} ${CLOUDTARGETS} .endif -.include "${.CURDIR}/Makefile.ec2" .include "${.CURDIR}/Makefile.azure" +.include "${.CURDIR}/Makefile.ec2" .include "${.CURDIR}/Makefile.gce" .include "${.CURDIR}/Makefile.vagrant" .include "${.CURDIR}/Makefile.inc1" diff --git a/release/tools/azure.conf b/release/tools/azure.conf index c88651a07745..a9126ed8c389 100644 --- a/release/tools/azure.conf +++ b/release/tools/azure.conf @@ -3,25 +3,54 @@ # $FreeBSD$ # +# Convention of Linux type VM on Azure is 30G +export VMSIZE=30g + # Set to a list of packages to install. -# Example: -#export VM_EXTRA_PACKAGES="www/apache24" -export VM_EXTRA_PACKAGES="${VM_EXTRA_PACKAGES} sysutils/azure-agent" +export VM_EXTRA_PACKAGES="azure-agent python python3 firstboot-freebsd-update firstboot-pkgs" # Set to a list of third-party software to enable in rc.conf(5). -# Example: -#export VM_RC_LIST="apache24" -export VM_RC_LIST= +export VM_RC_LIST="ntpd sshd waagent firstboot_freebsd_update firstboot_pkgs" + +# No swap space; waagent will allocate swap space on the resource disk. +# See ResourceDisk.EnableSwap and ResourceDisk.SwapSizeMB in waagent.conf +export NOSWAP=YES + +# https://docs.microsoft.com/en-us/azure/marketplace/azure-vm-create-certification-faq#vm-images-must-have-1mb-free-space +export VM_BOOTPARTSOFFSET=1M vm_extra_pre_umount() { - chroot ${DESTDIR} ln -s /usr/local/sbin/waagent /usr/sbin/waagent - chroot ${DESTDIR} /usr/local/sbin/waagent -verbose -install - yes | chroot ${DESTDIR} /usr/local/sbin/waagent -deprovision - echo 'sshd_enable="YES"' >> ${DESTDIR}/etc/rc.conf - echo 'ifconfig_hn0="SYNCDHCP"' >> ${DESTDIR}/etc/rc.conf - echo 'waagent_enable="YES"' >> ${DESTDIR}/etc/rc.conf - echo 'console="comconsole vidconsole"' >> ${DESTDIR}/boot/loader.conf - echo 'comconsole_speed="115200"' >> ${DESTDIR}/boot/loader.conf + # The firstboot_pkgs rc.d script will download the repository + # catalogue and install or update pkg when the instance first + # launches, so these files would just be replaced anyway; removing + # them from the image allows it to boot faster. + mount -t devfs devfs ${DESTDIR}/dev + chroot ${DESTDIR} ${EMULATOR} env ASSUME_ALWAYS_YES=yes \ + /usr/sbin/pkg delete -f -y pkg + umount ${DESTDIR}/dev + rm ${DESTDIR}/var/db/pkg/repo-*.sqlite + + yes | chroot ${DESTDIR} ${EMULATOR} /usr/local/sbin/waagent -deprovision + + cat << EOF >> ${DESTDIR}/etc/rc.conf +ifconfig_hn0="SYNCDHCP" +ntpd_sync_on_start="YES" +EOF + + cat << EOF >> ${DESTDIR}/boot/loader.conf +autoboot_delay="-1" +beastie_disable="YES" +loader_logo="none" +hw.memtest.tests="0" +console="comconsole efi vidconsole" +comconsole_speed="115200" +boot_multicons="YES" +boot_serial="YES" +mlx4en_load="YES" +mlx5en_load="YES" +EOF + + touch ${DESTDIR}/firstboot rm -f ${DESTDIR}/etc/resolv.conf diff --git a/release/tools/vmimage.subr b/release/tools/vmimage.subr index c2b2e21200f1..0d9202431822 100644 --- a/release/tools/vmimage.subr +++ b/release/tools/vmimage.subr @@ -195,6 +195,11 @@ vm_create_disk() { SWAPOPT="-p freebsd-swap/swapfs::${SWAPSIZE}" fi + if [ -n "${VM_BOOTPARTSOFFSET}" ]; then + BOOTPARTSOFFSET=":${VM_BOOTPARTSOFFSET}" + fi + + BOOTFILES="$(env TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH} \ WITH_UNIFIED_OBJDIR=yes \ make -C ${WORLDDIR}/stand -V .OBJDIR)" @@ -204,7 +209,7 @@ vm_create_disk() { amd64:amd64 | i386:i386) ESP=yes BOOTPARTS="-b ${BOOTFILES}/i386/pmbr/pmbr \ - -p freebsd-boot/bootfs:=${BOOTFILES}/i386/gptboot/gptboot" + -p freebsd-boot/bootfs:=${BOOTFILES}/i386/gptboot/gptboot${BOOTPARTSOFFSET}" ROOTFSPART="-p freebsd-ufs/rootfs:=${VMBASE}" MAKEFSARGS="-B little" ;;