From owner-dev-commits-src-all@freebsd.org Thu Mar 4 14:31:32 2021 Return-Path: Delivered-To: dev-commits-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BFBCE54BB94; Thu, 4 Mar 2021 14:31:32 +0000 (UTC) (envelope-from manu@bidouilliste.com) Received: from mx.blih.net (mx.blih.net [212.83.155.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mx.blih.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DrtbQ5rNXz3r3f; Thu, 4 Mar 2021 14:31:30 +0000 (UTC) (envelope-from manu@bidouilliste.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bidouilliste.com; s=mx; t=1614868281; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oWD396mLPxuM0jE7LZLMGvkx/KbG2Ev00kEHXdG5mHA=; b=ptT5Uh3L0XqXvmhPdCJZbYsqSnAejG0sKVLwW7he5L6TXJOLY5OM5Pa+edCxso3IVRuj7c KQlsjhLWcp/vVq+ezHQlao6XYTTG9Kge9aU7ktnMQM6iPGXvPh6uYPQvWE2LYuO8VqeIA0 96xnkngDFIGh8gpBDohBzU80Hb/5XfY= Received: from amy (lfbn-idf2-1-644-4.w86-247.abo.wanadoo.fr [86.247.100.4]) by mx.blih.net (OpenSMTPD) with ESMTPSA id f5d8143d (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 4 Mar 2021 14:31:21 +0000 (UTC) Date: Thu, 4 Mar 2021 15:31:21 +0100 From: Emmanuel Vadot To: Nathan Whitehorn Cc: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org Subject: Re: git: 1ca8842f3ad9 - main - Use makefs(8) in release VM-image generation instead of md(4) and newfs. Message-Id: <20210304153121.f06b268899a145ce97b21784@bidouilliste.com> In-Reply-To: <202102250225.11P2PE1I011966@gitrepo.freebsd.org> References: <202102250225.11P2PE1I011966@gitrepo.freebsd.org> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.33; amd64-portbld-freebsd14.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4DrtbQ5rNXz3r3f X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bidouilliste.com header.s=mx header.b=ptT5Uh3L; dmarc=pass (policy=none) header.from=bidouilliste.com; spf=pass (mx1.freebsd.org: domain of manu@bidouilliste.com designates 212.83.155.74 as permitted sender) smtp.mailfrom=manu@bidouilliste.com X-Spamd-Result: default: False [-3.50 / 15.00]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[bidouilliste.com:s=mx]; FREEFALL_USER(0.00)[manu]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; MV_CASE(0.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; R_SPF_ALLOW(-0.20)[+mx]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; SPAMHAUS_ZRD(0.00)[212.83.155.74:from:127.0.2.255]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DKIM_TRACE(0.00)[bidouilliste.com:+]; DMARC_POLICY_ALLOW(-0.50)[bidouilliste.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[212.83.155.74:from]; ASN(0.00)[asn:12876, ipnet:212.83.128.0/19, country:FR]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[dev-commits-src-all,dev-commits-src-main] X-BeenThere: dev-commits-src-all@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commit messages for all branches of the src repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Mar 2021 14:31:32 -0000 On Thu, 25 Feb 2021 02:25:14 GMT Nathan Whitehorn wrote: > The branch main has been updated by nwhitehorn: > > URL: https://cgit.FreeBSD.org/src/commit/?id=1ca8842f3ad9725863c9affc044d1974a51818a9 > > commit 1ca8842f3ad9725863c9affc044d1974a51818a9 > Author: Nathan Whitehorn > AuthorDate: 2021-02-25 02:16:56 +0000 > Commit: Nathan Whitehorn > CommitDate: 2021-02-25 02:16:56 +0000 > > Use makefs(8) in release VM-image generation instead of md(4) and newfs. > > Using makefs instead reduces the privileges needed to build VM images, > simplifies the script (no need to copy files to a fresh image at the end), > and improves portability by allowing generation of cross-endian images. > As a result of the last, this patch also adds support for generation of > powerpc64 and powerpc64le VM images. > > No other changes to the output. Tested and working for both amd64 and > powerpc64 targets. > > Reviewed by: gjb > Differential Revision: https://reviews.freebsd.org/D28912 > --- > release/scripts/mk-vmimage.sh | 2 - > release/tools/vmimage.subr | 177 ++++++++++++++++++------------------------ > 2 files changed, 77 insertions(+), 102 deletions(-) > > diff --git a/release/scripts/mk-vmimage.sh b/release/scripts/mk-vmimage.sh > index cf795e04655b..d5985ceb0d25 100755 > --- a/release/scripts/mk-vmimage.sh > +++ b/release/scripts/mk-vmimage.sh > @@ -93,8 +93,6 @@ main() { > . "${VMCONFIG}" > fi > > - ROOTLABEL="gpt" > - > vm_create_base > vm_install_base > vm_extra_install_base > diff --git a/release/tools/vmimage.subr b/release/tools/vmimage.subr > index 7bd971013656..6e0c1ea633e2 100644 > --- a/release/tools/vmimage.subr > +++ b/release/tools/vmimage.subr > @@ -12,70 +12,17 @@ scriptdir=$(dirname $(realpath $0)) > export PATH="/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin" > trap "cleanup" INT QUIT TRAP ABRT TERM > > -write_partition_layout() { > - if [ -z "${NOSWAP}" ]; then > - SWAPOPT="-p freebsd-swap/swapfs::${SWAPSIZE}" > - fi > - > - BOOTFILES="$(env TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH} \ > - WITH_UNIFIED_OBJDIR=yes \ > - make -C ${WORLDDIR}/stand -V .OBJDIR)" > - BOOTFILES="$(realpath ${BOOTFILES})" > - > - case "${TARGET}:${TARGET_ARCH}" in > - amd64:amd64 | i386:i386) > - ESP=yes > - SCHEME=gpt > - BOOTPARTS="-b ${BOOTFILES}/i386/pmbr/pmbr \ > - -p freebsd-boot/bootfs:=${BOOTFILES}/i386/gptboot/gptboot" > - ROOTFSPART="-p freebsd-ufs/rootfs:=${VMBASE}" > - ;; > - arm64:aarch64 | riscv:riscv64*) > - ESP=yes > - SCHEME=gpt > - BOOTPARTS= > - ROOTFSPART="-p freebsd-ufs/rootfs:=${VMBASE}" > - ;; > - powerpc:powerpc*) > - ESP=no > - SCHEME=apm > - BOOTPARTS="-p apple-boot/bootfs:=${BOOTFILES}/powerpc/boot1.chrp/boot1.hfs" > - ROOTFSPART="-p freebsd-ufs/rootfs:=${VMBASE}" > - ;; > - *) > - echo "vmimage.subr: unsupported target '${TARGET}:${TARGET_ARCH}'" >&2 > - exit 1 > - ;; > - esac > - > - if [ ${ESP} = "yes" ]; then > - # Create an ESP > - espfilename=$(mktemp /tmp/efiboot.XXXXXX) > - make_esp_file ${espfilename} ${fat32min} ${BOOTFILES}/efi/loader_lua/loader_lua.efi > - BOOTPARTS="${BOOTPARTS} -p efi/efiesp:=${espfilename}" > - > - # Add this to fstab, requires temporarily remounting the fs > - mddev=$(mdconfig -f ${VMBASE}) > - mount /dev/${mddev} ${DESTDIR} > - mkdir -p ${DESTDIR}/boot/efi > - echo "/dev/${ROOTLABEL}/efiesp /boot/efi msdosfs rw 2 2" \ > - >> ${DESTDIR}/etc/fstab > - umount ${DESTDIR} > - mdconfig -d -u ${mddev} > - fi > - > - mkimg -s ${SCHEME} -f ${VMFORMAT} \ > - ${BOOTPARTS} \ > - ${SWAPOPT} \ > - ${ROOTFSPART} \ > - -o ${VMIMAGE} > - > - if [ ${ESP} = "yes" ]; then > - rm ${espfilename} > - fi > - > - return 0 > -} > +# Platform-specific large-scale setup > +# Most platforms use GPT, so put that as default, then special cases > +PARTSCHEME=gpt > +ROOTLABEL="gpt" > +case "${TARGET}:${TARGET_ARCH}" in > + powerpc:powerpc*) > + PARTSCHEME=mbr > + ROOTLABEL="ufs" > + NOSWAP=yes # Can't label swap partition with MBR, so no swap > + ;; > +esac > > err() { > printf "${@}\n" > @@ -87,10 +34,6 @@ cleanup() { > if [ -c "${DESTDIR}/dev/null" ]; then > umount_loop ${DESTDIR}/dev 2>/dev/null > fi > - umount_loop ${DESTDIR} > - if [ ! -z "${mddev}" ]; then > - mdconfig -d -u ${mddev} > - fi > > return 0 > } > @@ -100,42 +43,12 @@ vm_create_base() { > # written to the formatted disk image with mkimg(1). > > mkdir -p ${DESTDIR} > - truncate -s ${VMSIZE} ${VMBASE} > - mddev=$(mdconfig -f ${VMBASE}) > - newfs -L rootfs /dev/${mddev} > - mount /dev/${mddev} ${DESTDIR} > > return 0 > } > > vm_copy_base() { > - # Creates a new UFS root filesystem and copies the contents of the > - # current root filesystem into it. This produces a "clean" disk > - # image without any remnants of files which were created temporarily > - # during image-creation and have since been deleted (e.g., downloaded > - # package archives). > - > - mkdir -p ${DESTDIR}/old > - mdold=$(mdconfig -f ${VMBASE}) > - mount /dev/${mdold} ${DESTDIR}/old > - > - truncate -s ${VMSIZE} ${VMBASE}.tmp > - mkdir -p ${DESTDIR}/new > - mdnew=$(mdconfig -f ${VMBASE}.tmp) > - newfs -L rootfs /dev/${mdnew} > - mount /dev/${mdnew} ${DESTDIR}/new > - > - tar -cf- -C ${DESTDIR}/old . | tar -xUf- -C ${DESTDIR}/new > - > - umount_loop /dev/${mdold} > - rmdir ${DESTDIR}/old > - mdconfig -d -u ${mdold} > - > - umount_loop /dev/${mdnew} > - rmdir ${DESTDIR}/new > - tunefs -n enable /dev/${mdnew} > - mdconfig -d -u ${mdnew} > - mv ${VMBASE}.tmp ${VMBASE} > + # Defunct > } > > vm_install_base() { > @@ -276,7 +189,71 @@ vm_create_disk() { > echo "Creating image... Please wait." > echo > > - write_partition_layout || return 1 > + if [ -z "${NOSWAP}" ]; then > + SWAPOPT="-p freebsd-swap/swapfs::${SWAPSIZE}" > + fi > + > + BOOTFILES="$(env TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH} \ > + WITH_UNIFIED_OBJDIR=yes \ > + make -C ${WORLDDIR}/stand -V .OBJDIR)" > + BOOTFILES="$(realpath ${BOOTFILES})" > + > + case "${TARGET}:${TARGET_ARCH}" in > + amd64:amd64 | i386:i386) > + ESP=yes > + BOOTPARTS="-b ${BOOTFILES}/i386/pmbr/pmbr \ > + -p freebsd-boot/bootfs:=${BOOTFILES}/i386/gptboot/gptboot" > + ROOTFSPART="-p freebsd-ufs/rootfs:=${VMBASE}" > + MAKEFSARGS="-B little" > + ;; > + arm64:aarch64 | riscv:riscv64*) > + ESP=yes > + BOOTPARTS= > + ROOTFSPART="-p freebsd-ufs/rootfs:=${VMBASE}" > + MAKEFSARGS="-B little" > + ;; > + powerpc:powerpc*) > + ESP=no > + BOOTPARTS="-p prepboot:=${BOOTFILES}/powerpc/boot1.chrp/boot1.elf -a 1" > + ROOTFSPART="-p freebsd:=${VMBASE}" > + if [ ${TARGET_ARCH} = powerpc64le ]; then > + MAKEFSARGS="-B little" > + else > + MAKEFSARGS="-B big" > + fi > + ;; > + *) > + echo "vmimage.subr: unsupported target '${TARGET}:${TARGET_ARCH}'" >&2 > + exit 1 > + ;; > + esac > + > + if [ ${ESP} = "yes" ]; then > + # Create an ESP > + espfilename=$(mktemp /tmp/efiboot.XXXXXX) > + make_esp_file ${espfilename} ${fat32min} ${BOOTFILES}/efi/loader_lua/loader_lua.efi > + BOOTPARTS="${BOOTPARTS} -p efi/efiesp:=${espfilename}" > + > + # Add this to fstab > + mkdir -p ${DESTDIR}/boot/efi > + echo "/dev/${ROOTLABEL}/efiesp /boot/efi msdosfs rw 2 2" \ > + >> ${DESTDIR}/etc/fstab > + fi > + > + echo "Building filesystem... Please wait." > + makefs ${MAKEFSARGS} -o label=rootfs -o version=2 -o softupdates=1 \ > + -s ${VMSIZE} ${VMBASE} ${DESTDIR} There should be some -f usage here to reserve some inodes. I'm not remembering the details exactly (same for bapt) but when we were at Gandi we revert using makefs and went back to use mdconfig/newfs because of some problems related to inodes number that -f couldn't fix. I haven't tested myself but I think that the VM image are busted with that change if they do growfs as makefs by default use the same number of inodes as the number of files in the image. Cheers, > + echo "Building final disk image... Please wait." > + mkimg -s ${PARTSCHEME} -f ${VMFORMAT} \ > + ${BOOTPARTS} \ > + ${SWAPOPT} \ > + ${ROOTFSPART} \ > + -o ${VMIMAGE} > + > + if [ ${ESP} = "yes" ]; then > + rm ${espfilename} > + fi > > return 0 > } -- Emmanuel Vadot