Date: Thu, 31 Aug 2023 07:45:41 -0500 From: "Drew Gallatin" <gallatin@freebsd.org> To: "Alexander Motin" <mav@FreeBSD.org>, "Gleb Smirnoff" <glebius@freebsd.org>, "Martin Matuska" <mm@freebsd.org> Cc: src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org Subject: Re: git: 315ee00fa961 - main - zfs: merge openzfs/zfs@804414aad Message-ID: <07faf861-9186-47d1-992a-91d483ea4e9c@app.fastmail.com> In-Reply-To: <c09c92df-90f5-8c94-4125-9e33262bc686@FreeBSD.org> References: <202308270509.37R596B5048298@gitrepo.freebsd.org> <ZO_aOaf-eGiCMCKy@cell.glebi.us> <c09c92df-90f5-8c94-4125-9e33262bc686@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--06e1cace14e44ba6b615e2ddd8eabb7c
Content-Type: text/plain
On Wed, Aug 30, 2023, at 8:01 PM, Alexander Motin wrote:
> Hi Gleb,
>
> It is the first time I see a panic like this. I'll think about it
> tomorrow. But I'd appreciate any information on what is your workload
> and what are you doing related to ZIL (O_SYNC, fsync(), sync=always,
> etc) to trigger it? What is your pool configuration?
I'm not Gleb, but this was something at $WORK, so I can perhaps help. I've included the output of zpool status, and all non-default settings in the zpool. Note that we don't use a ZIL device.
Thanks,
Drew
# zpool status
pool: zroot
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
nda0p2 ONLINE 0 0 0
nda1p2 ONLINE 0 0 0
nda2p2 ONLINE 0 0 0
spares
nda3p2 AVAIL
errors: No known data errors
# zpool get all | grep -v default
NAME PROPERTY VALUE SOURCE
zroot size 149G -
zroot capacity 14% -
zroot health ONLINE -
zroot guid 5099990586719219765 -
zroot bootfs zroot/ROOT/NCD-7827-freebsd-2023-08-29-merge-oca-r202308291935 local
zroot dedupratio 1.00x -
zroot free 127G -
zroot allocated 22.3G -
zroot readonly off -
zroot expandsize - -
zroot freeing 0 -
zroot fragmentation 28% -
zroot leaked 0 -
zroot checkpoint - -
zroot load_guid 7612195721260396606 -
zroot bcloneused 0 -
zroot bclonesaved 0 -
zroot bcloneratio 1.00x -
zroot feature@async_destroy enabled local
zroot feature@empty_bpobj active local
zroot feature@lz4_compress active local
zroot feature@multi_vdev_crash_dump enabled local
zroot feature@spacemap_histogram active local
zroot feature@enabled_txg active local
zroot feature@hole_birth active local
zroot feature@extensible_dataset active local
zroot feature@embedded_data active local
zroot feature@bookmarks enabled local
zroot feature@filesystem_limits enabled local
zroot feature@large_blocks enabled local
zroot feature@large_dnode enabled local
zroot feature@sha512 enabled local
zroot feature@skein enabled local
zroot feature@edonr disabled local
zroot feature@userobj_accounting active local
zroot feature@encryption enabled local
zroot feature@project_quota active local
zroot feature@device_removal enabled local
zroot feature@obsolete_counts enabled local
zroot feature@zpool_checkpoint enabled local
zroot feature@spacemap_v2 active local
zroot feature@allocation_classes enabled local
zroot feature@resilver_defer enabled local
zroot feature@bookmark_v2 enabled local
zroot feature@redaction_bookmarks enabled local
zroot feature@redacted_datasets enabled local
zroot feature@bookmark_written enabled local
zroot feature@log_spacemap active local
zroot feature@livelist enabled local
zroot feature@device_rebuild enabled local
zroot feature@zstd_compress enabled local
zroot feature@draid enabled local
zroot feature@zilsaxattr disabled local
zroot feature@head_errlog disabled local
zroot feature@blake3 disabled local
zroot feature@block_cloning disabled local
zroot feature@vdev_zaps_v2 disabled local
> On 30.08.2023 20:09, Gleb Smirnoff wrote:
> > Hi Martin,
> >
> > we hit this panic with this update:
> >
> > panic: VERIFY(avl_is_empty(&lwb->lwb_vdev_tree)) failed
> >
> > cpuid = 8
> > time = 1693432004
> > KDB: stack backtrace:
> > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe09519aa800
> > vpanic() at vpanic+0x132/frame 0xfffffe09519aa930
> > spl_panic() at spl_panic+0x3a/frame 0xfffffe09519aa990
> > zil_free_lwb() at zil_free_lwb+0x1de/frame 0xfffffe09519aa9c0
> > zil_sync() at zil_sync+0x3cb/frame 0xfffffe09519aaaa0
> > [Wed Aug 30 21:46:44 2023]dmu_objset_sync() at dmu_objset_sync+0x51b/frame 0xfffffe09519aab70
> > dsl_pool_sync() at dsl_pool_sync+0x11d/frame 0xfffffe09519aabf0
> > spa_sync() at spa_sync+0xc68/frame 0xfffffe09519aae20
> > txg_sync_thread() at txg_sync_thread+0x2eb/frame 0xfffffe09519aaef0
> > fork_exit() at fork_exit+0x82/frame 0xfffffe09519aaf30
> > fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe09519aaf30
> > --- trap 0x26d90a2f, rip = 0xbba54c81b25fd8d4, rsp = 0xff60c2f6f9874bcd, rbp = 0x37940e3e9793bd40 --
> >
> > I have two cores dumped for this panic.
> >
> > Our previous version is based on 9c0a2d522f3a18fc795a7dd6266d2be2aff4b084, which
> > I guess has the most recent ZFS merge openzfs/zfs@009d3288d.
> >
> > On Sun, Aug 27, 2023 at 05:09:06AM +0000, Martin Matuska wrote:
> > M> The branch main has been updated by mm:
> > M>
> > M> URL: https://cgit.FreeBSD.org/src/commit/?id=315ee00fa9616b0a192b6834911f98bcf5316a6b <https://cgit.freebsd.org/src/commit/?id=315ee00fa9616b0a192b6834911f98bcf5316a6b>;
> > M>
> > M> commit 315ee00fa9616b0a192b6834911f98bcf5316a6b
> > M> Merge: b3a714652ff0 804414aad224
> > M> Author: Martin Matuska <mm@FreeBSD.org>
> > M> AuthorDate: 2023-08-26 21:20:04 +0000
> > M> Commit: Martin Matuska <mm@FreeBSD.org>
> > M> CommitDate: 2023-08-26 21:51:42 +0000
> > M>
> > M> zfs: merge openzfs/zfs@804414aad
> > M>
> > M> Notable upstream pull request merges:
> > M> #15024 Add missed DMU_PROJECTUSED_OBJECT prefetch
> > M> #15029 Do not request data L1 buffers on scan prefetch
> > M> #15036 FreeBSD: catch up to __FreeBSD_version 1400093
> > M> #15039 Fix raw receive with different indirect block size
> > M> #15047 FreeBSD: Fix build on stable/13 after 1302506
> > M> #15049 Fix the ZFS checksum error histograms with larger record sizes
> > M> #15052 Reduce bloat in ereport.fs.zfs.checksum events
> > M> #15056 Avoid extra snprintf() in dsl_deadlist_merge()
> > M> #15061 Ignore pool ashift property during vdev attachment
> > M> #15063 Don't panic if setting vdev properties is unsupported for this vdev type
> > M> #15067 spa_min_alloc should be GCD, not min
> > M> #15071 Add explicit prefetches to bpobj_iterate()
> > M> #15072 Adjust prefetch parameters
> > M> #15076 Refactor dmu_prefetch()
> > M> #15079 set autotrim default to 'off' everywhere
> > M> #15080 ZIL: Fix config lock deadlock
> > M> #15088 metaslab: tuneable to better control force ganging
> > M> #15096 Avoid waiting in dmu_sync_late_arrival()
> > M> #15097 BRT should return EOPNOTSUPP
> > M> #15103 Remove zl_issuer_lock from zil_suspend()
> > M> #15107 Remove fastwrite mechanism
> > M> #15113 libzfs: sendrecv: send_progress_thread: handle SIGINFO/SIGUSR1
> > M> #15122 ZIL: Second attempt to reduce scope of zl_issuer_lock
> > M> #15129 zpool_vdev_remove() should handle EALREADY error return
> > M> #15132 ZIL: Replay blocks without next block pointer
> > M> #15148 zfs_clone_range should return descriptive error codes
> > M> #15153 ZIL: Avoid dbuf_read() before dmu_sync()
> > M> #15172 copy_file_range: fix fallback when source create on same txg
> > M> #15180 Update outdated assertion from zio_write_compress
> > M>
> > M> Obtained from: OpenZFS
> > M> OpenZFS commit: 804414aad224b432590afe3f9ec114ffb49e0f13
> > M>
> > M> Makefile.inc1 | 5 +-
> > M> cddl/lib/libzfs/Makefile | 1 +
> > M> cddl/lib/libzfs/Makefile.depend | 1 +
> > M> rescue/rescue/Makefile | 2 +-
> > M> rescue/rescue/Makefile.depend | 1 +
> > M> share/mk/src.libnames.mk | 2 +-
> > M> sys/contrib/openzfs/META | 6 +-
> > M> sys/contrib/openzfs/cmd/zdb/zdb.c | 77 ++-
> > M> sys/contrib/openzfs/cmd/zed/agents/zfs_mod.c | 2 -
> > M> sys/contrib/openzfs/cmd/zed/agents/zfs_retire.c | 5 +
> > M> .../openzfs/cmd/zed/zed.d/statechange-slot_off.sh | 61 ++
> > M> sys/contrib/openzfs/cmd/zed/zed.d/zed.rc | 5 +
> > M> sys/contrib/openzfs/cmd/ztest.c | 2 +-
> > M> sys/contrib/openzfs/config/Shellcheck.am | 3 +-
> > M> sys/contrib/openzfs/config/kernel-blkdev.m4 | 138 +++-
> > M> .../config/kernel-block-device-operations.m4 | 35 +-
> > M> .../openzfs/config/kernel-filemap-splice-read.m4 | 25 +
> > M> .../openzfs/config/kernel-register_sysctl_table.m4 | 27 +
> > M> .../config/kernel-vfs-extended-file_range.m4 | 50 ++
> > M> .../openzfs/config/kernel-vfs-file_range.m4 | 164 +++++
> > M> sys/contrib/openzfs/config/kernel-vfs-iov_iter.m4 | 26 +-
> > M> sys/contrib/openzfs/config/kernel.m4 | 16 +
> > M> sys/contrib/openzfs/contrib/debian/changelog | 6 +
> > M> .../contrib/debian/openzfs-zfs-test.install | 2 -
> > M> .../contrib/debian/openzfs-zfsutils.install | 2 +
> > M> .../contrib/dracut/90zfs/zfs-env-bootfs.service.in | 11 +-
> > M> .../dracut/90zfs/zfs-rollback-bootfs.service.in | 2 +-
> > M> .../openzfs/include/os/freebsd/spl/sys/vnode.h | 6 +-
> > M> .../include/os/linux/kernel/linux/blkdev_compat.h | 17 +
> > M> .../openzfs/include/os/linux/spl/sys/kmem_cache.h | 8 +
> > M> .../openzfs/include/os/linux/spl/sys/types.h | 2 +-
> > M> sys/contrib/openzfs/include/os/linux/spl/sys/uio.h | 12 +
> > M> sys/contrib/openzfs/include/os/linux/zfs/sys/zpl.h | 53 ++
> > M> sys/contrib/openzfs/include/sys/bpobj.h | 2 +-
> > M> sys/contrib/openzfs/include/sys/brt.h | 1 +
> > M> sys/contrib/openzfs/include/sys/dmu.h | 5 +
> > M> sys/contrib/openzfs/include/sys/dmu_impl.h | 2 -
> > M> sys/contrib/openzfs/include/sys/dmu_zfetch.h | 2 -
> > M> sys/contrib/openzfs/include/sys/fm/fs/zfs.h | 4 -
> > M> sys/contrib/openzfs/include/sys/metaslab.h | 3 -
> > M> sys/contrib/openzfs/include/sys/metaslab_impl.h | 6 +-
> > M> sys/contrib/openzfs/include/sys/spa_impl.h | 1 +
> > M> sys/contrib/openzfs/include/sys/vdev_impl.h | 2 +-
> > M> sys/contrib/openzfs/include/sys/zil_impl.h | 44 +-
> > M> sys/contrib/openzfs/include/sys/zio.h | 1 -
> > M> sys/contrib/openzfs/include/sys/zio_checksum.h | 2 -
> > M> sys/contrib/openzfs/lib/libzfs/Makefile.am | 2 +-
> > M> sys/contrib/openzfs/lib/libzfs/libzfs_pool.c | 6 +
> > M> sys/contrib/openzfs/lib/libzfs/libzfs_sendrecv.c | 95 ++-
> > M> sys/contrib/openzfs/man/man4/zfs.4 | 10 +-
> > M> sys/contrib/openzfs/man/man8/zfs-send.8 | 18 +-
> > M> sys/contrib/openzfs/man/man8/zpool-events.8 | 23 +-
> > M> sys/contrib/openzfs/module/Kbuild.in | 1 +
> > M> .../openzfs/module/os/freebsd/zfs/zfs_vnops_os.c | 10 +-
> > M> sys/contrib/openzfs/module/os/linux/spl/spl-proc.c | 26 +-
> > M> .../openzfs/module/os/linux/zfs/vdev_disk.c | 65 +-
> > M> .../openzfs/module/os/linux/zfs/zfs_ctldir.c | 26 +-
> > M> .../openzfs/module/os/linux/zfs/zfs_vfsops.c | 7 +
> > M> .../openzfs/module/os/linux/zfs/zfs_vnops_os.c | 9 +-
> > M> .../openzfs/module/os/linux/zfs/zfs_znode.c | 8 +
> > M> .../openzfs/module/os/linux/zfs/zpl_ctldir.c | 2 +-
> > M> sys/contrib/openzfs/module/os/linux/zfs/zpl_file.c | 44 +-
> > M> .../openzfs/module/os/linux/zfs/zpl_file_range.c | 272 ++++++++
> > M> .../openzfs/module/os/linux/zfs/zpl_super.c | 39 +-
> > M> sys/contrib/openzfs/module/os/linux/zfs/zvol_os.c | 28 +-
> > M> sys/contrib/openzfs/module/zfs/bpobj.c | 49 +-
> > M> sys/contrib/openzfs/module/zfs/brt.c | 43 +-
> > M> sys/contrib/openzfs/module/zfs/dbuf.c | 11 +-
> > M> sys/contrib/openzfs/module/zfs/dmu.c | 127 ++--
> > M> sys/contrib/openzfs/module/zfs/dmu_recv.c | 22 +-
> > M> sys/contrib/openzfs/module/zfs/dmu_zfetch.c | 12 +-
> > M> sys/contrib/openzfs/module/zfs/dnode.c | 31 +-
> > M> sys/contrib/openzfs/module/zfs/dsl_deadlist.c | 14 +-
> > M> sys/contrib/openzfs/module/zfs/dsl_scan.c | 17 +-
> > M> sys/contrib/openzfs/module/zfs/metaslab.c | 83 +--
> > M> sys/contrib/openzfs/module/zfs/spa_log_spacemap.c | 4 +-
> > M> sys/contrib/openzfs/module/zfs/spa_misc.c | 1 +
> > M> sys/contrib/openzfs/module/zfs/vdev.c | 90 ++-
> > M> sys/contrib/openzfs/module/zfs/vdev_indirect.c | 2 +-
> > M> sys/contrib/openzfs/module/zfs/vdev_raidz.c | 2 +-
> > M> sys/contrib/openzfs/module/zfs/zfs_fm.c | 33 +-
> > M> sys/contrib/openzfs/module/zfs/zfs_vnops.c | 46 +-
> > M> sys/contrib/openzfs/module/zfs/zil.c | 719 +++++++++------------
> > M> sys/contrib/openzfs/module/zfs/zio.c | 45 +-
> > M> sys/contrib/openzfs/module/zfs/zio_checksum.c | 2 -
> > M> sys/contrib/openzfs/module/zfs/zvol.c | 8 +-
> > M> sys/contrib/openzfs/tests/runfiles/linux.run | 11 +
> > M> .../openzfs/tests/test-runner/bin/zts-report.py.in | 18 +
> > M> sys/contrib/openzfs/tests/zfs-tests/cmd/.gitignore | 1 +
> > M> .../openzfs/tests/zfs-tests/cmd/Makefile.am | 1 +
> > M> .../openzfs/tests/zfs-tests/cmd/clonefile.c | 333 ++++++++++
> > M> sys/contrib/openzfs/tests/zfs-tests/cmd/readmmap.c | 1 +
> > M> .../openzfs/tests/zfs-tests/include/commands.cfg | 1 +
> > M> .../openzfs/tests/zfs-tests/tests/Makefile.am | 14 +
> > M> .../functional/block_cloning/block_cloning.kshlib | 54 ++
> > M> .../block_cloning/block_cloning_copyfilerange.ksh | 60 ++
> > M> .../block_cloning_copyfilerange_cross_dataset.ksh | 65 ++
> > M> .../block_cloning_copyfilerange_fallback.ksh | 86 +++
> > M> ...ock_cloning_copyfilerange_fallback_same_txg.ksh | 66 ++
> > M> .../block_cloning_copyfilerange_partial.ksh | 68 ++
> > M> .../block_cloning_disabled_copyfilerange.ksh | 60 ++
> > M> .../block_cloning_disabled_ficlone.ksh | 50 ++
> > M> .../block_cloning_disabled_ficlonerange.ksh | 50 ++
> > M> .../block_cloning/block_cloning_ficlone.ksh | 56 ++
> > M> .../block_cloning/block_cloning_ficlonerange.ksh | 56 ++
> > M> .../block_cloning_ficlonerange_partial.ksh | 64 ++
> > M> .../tests/functional/block_cloning/cleanup.ksh | 34 +
> > M> .../tests/functional/block_cloning/setup.ksh | 36 ++
> > M> .../cli_root/zpool_attach/attach-o_ashift.ksh | 30 +-
> > M> .../cli_root/zpool_replace/replace-o_ashift.ksh | 32 +-
> > M> .../cli_root/zpool_replace/replace_prop_ashift.ksh | 24 +-
> > M> sys/modules/zfs/zfs_config.h | 53 +-
> > M> sys/modules/zfs/zfs_gitrev.h | 2 +-
> > M> 113 files changed, 3322 insertions(+), 906 deletions(-)
> > M>
> > M> diff --cc Makefile.inc1
> > M> index 8838d31c795a,000000000000..f740cc5abcca
> > M> mode 100644,000000..100644
> > M> --- a/Makefile.inc1
> > M> +++ b/Makefile.inc1
> > M> @@@ -1,3692 -1,0 +1,3695 @@@
> > M> +#
> > M> +#
> > M> +# Make command line options:
> > M> +# -DNO_CLEANDIR run ${MAKE} clean, instead of ${MAKE} cleandir
> > M> +# -DNO_CLEAN do not clean at all
> > M> +# -DDB_FROM_SRC use the user/group databases in src/etc instead of
> > M> +# the system database when installing.
> > M> +# -DNO_SHARE do not go into share subdir
> > M> +# -DKERNFAST define NO_KERNEL{CONFIG,CLEAN,OBJ}
> > M> +# -DNO_KERNELCONFIG do not run config in ${MAKE} buildkernel
> > M> +# -DNO_KERNELCLEAN do not run ${MAKE} clean in ${MAKE} buildkernel
> > M> +# -DNO_KERNELOBJ do not run ${MAKE} obj in ${MAKE} buildkernel
> > M> +# -DNO_ROOT install without using root privilege
> > M> +# -DWITHOUT_CTF do not run the DTrace CTF conversion tools on built objects
> > M> +# LOCAL_DIRS="list of dirs" to add additional dirs to the SUBDIR list
> > M> +# LOCAL_ITOOLS="list of tools" to add additional tools to the ITOOLS list
> > M> +# LOCAL_LIB_DIRS="list of dirs" to add additional dirs to libraries target
> > M> +# LOCAL_MTREE="list of mtree files" to process to allow local directories
> > M> +# to be created before files are installed
> > M> +# LOCAL_LEGACY_DIRS="list of dirs" to add additional dirs to the legacy
> > M> +# target
> > M> +# LOCAL_BSTOOL_DIRS="list of dirs" to add additional dirs to the
> > M> +# bootstrap-tools target
> > M> +# LOCAL_TOOL_DIRS="list of dirs" to add additional dirs to the build-tools
> > M> +# target
> > M> +# LOCAL_XTOOL_DIRS="list of dirs" to add additional dirs to the
> > M> +# cross-tools target
> > M> +# METALOG="path to metadata log" to write permission and ownership
> > M> +# when NO_ROOT is set. (default: ${DESTDIR}/${DISTDIR}/METALOG,
> > M> +# check /etc/make.conf for DISTDIR)
> > M> +# TARGET="machine" to crossbuild world for a different machine type
> > M> +# TARGET_ARCH= may be required when a TARGET supports multiple endians
> > M> +# BUILDENV_SHELL= shell to launch for the buildenv target (def:${SHELL})
> > M> +# WORLD_FLAGS= additional flags to pass to make(1) during buildworld
> > M> +# KERNEL_FLAGS= additional flags to pass to make(1) during buildkernel
> > M> +# SUBDIR_OVERRIDE="list of dirs" to build rather than everything.
> > M> +# All libraries and includes, and some build tools will still build.
> > M> +
> > M> +#
> > M> +# The intended user-driven targets are:
> > M> +# buildworld - rebuild *everything*, including glue to help do upgrades
> > M> +# installworld- install everything built by "buildworld"
> > M> +# checkworld - run test suite on installed world
> > M> +# doxygen - build API documentation of the kernel
> > M> +#
> > M> +# Standard targets (not defined here) are documented in the makefiles in
> > M> +# /usr/share/mk. These include:
> > M> +# obj depend all install clean cleandepend cleanobj
> > M> +
> > M> +.if !defined(TARGET) || !defined(TARGET_ARCH)
> > M> +.error Both TARGET and TARGET_ARCH must be defined.
> > M> +.endif
> > M> +
> > M> +.if make(showconfig) || make(test-system-*)
> > M> +_MKSHOWCONFIG= t
> > M> +.endif
> > M> +
> > M> +SRCDIR?= ${.CURDIR}
> > M> +LOCALBASE?= /usr/local
> > M> +TIME_ENV ?= time env
> > M> +
> > M> +.include "share/mk/src.tools.mk"
> > M> +
> > M> +# Cross toolchain changes must be in effect before bsd.compiler.mk
> > M> +# so that gets the right CC, and pass CROSS_TOOLCHAIN to submakes.
> > M> +.if defined(CROSS_TOOLCHAIN)
> > M> +.if exists(${LOCALBASE}/share/toolchains/${CROSS_TOOLCHAIN}.mk)
> > M> +.include "${LOCALBASE}/share/toolchains/${CROSS_TOOLCHAIN}.mk"
> > M> +.elif exists(${CROSS_TOOLCHAIN})
> > M> +.include "${CROSS_TOOLCHAIN}"
> > M> +.else
> > M> +.error CROSS_TOOLCHAIN ${CROSS_TOOLCHAIN} not found
> > M> +.endif
> > M> +CROSSENV+=CROSS_TOOLCHAIN="${CROSS_TOOLCHAIN}"
> > M> +.elif defined(UNIVERSE_TOOLCHAIN)
> > M> +UNIVERSE_TOOLCHAIN_PATH?=${HOST_OBJTOP}/tmp/usr/bin
> > M> +XCC?="${UNIVERSE_TOOLCHAIN_PATH}/cc"
> > M> +XCXX?="${UNIVERSE_TOOLCHAIN_PATH}/c++"
> > M> +XCPP?="${UNIVERSE_TOOLCHAIN_PATH}/cpp"
> > M> +XLD?="${UNIVERSE_TOOLCHAIN_PATH}/ld"
> > M> +.endif
> > M> +.if defined(CROSS_TOOLCHAIN_PREFIX)
> > M> +CROSS_COMPILER_PREFIX?=${CROSS_TOOLCHAIN_PREFIX}
> > M> +.endif
> > M> +
> > M> +XCOMPILERS= CC CXX CPP
> > M> +.for COMPILER in ${XCOMPILERS}
> > M> +.if defined(CROSS_COMPILER_PREFIX)
> > M> +X${COMPILER}?= ${CROSS_COMPILER_PREFIX}${${COMPILER}}
> > M> +.else
> > M> +X${COMPILER}?= ${${COMPILER}}
> > M> +.endif
> > M> +.endfor
> > M> +# If a full path to an external cross compiler is given, don't build
> > M> +# a cross compiler.
> > M> +.if ${XCC:N${CCACHE_BIN}:M/*}
> > M> +MK_CLANG_BOOTSTRAP= no
> > M> +# Make sure sub-makes see the option as disabled so the hack in bsd.sys.mk to
> > M> +# work around incompatible headers in Clang's resource directory is enabled.
> > M> +.MAKEOVERRIDES+= MK_CLANG_BOOTSTRAP
> > M> +.endif
> > M> +
> > M> +# Pull in compiler metadata from buildworld/toolchain if possible to avoid
> > M> +# running CC from bsd.compiler.mk.
> > M> +.if make(installworld) || make(install) || make(distributeworld) || \
> > M> + make(stageworld)
> > M> +.-include "${OBJTOP}/toolchain-metadata.mk"
> > M> +.if !defined(_LOADED_TOOLCHAIN_METADATA)
> > M> +.error A build is required first. You may have the wrong MAKEOBJDIRPREFIX set.
> > M> +.endif
> > M> +.endif
> > M> +
> > M> +# Pull in COMPILER_TYPE and COMPILER_FREEBSD_VERSION early. Pull it from the
> > M> +# tree to be friendlier to foreign OS builds. It's safe to do so unconditionally
> > M> +# here since we will always have the right make, unlike in src/Makefile
> > M> +# Don't include bsd.linker.mk yet until XBINUTILS is handled (after src.opts.mk)
> > M> +_NO_INCLUDE_LINKERMK= t
> > M> +# We also want the X_COMPILER* variables if we are using an external toolchain.
> > M> +_WANT_TOOLCHAIN_CROSS_VARS= t
> > M> +.include "share/mk/bsd.compiler.mk"
> > M> +.undef _NO_INCLUDE_LINKERMK
> > M> +.undef _WANT_TOOLCHAIN_CROSS_VARS
> > M> +# src.opts.mk depends on COMPILER_FEATURES
> > M> +.include "share/mk/src.opts.mk"
> > M> +
> > M> +.if ${TARGET} == ${MACHINE}
> > M> +TARGET_CPUTYPE?=${CPUTYPE}
> > M> +.else
> > M> +TARGET_CPUTYPE?=
> > M> +.endif
> > M> +.if !empty(TARGET_CPUTYPE)
> > M> +_TARGET_CPUTYPE=${TARGET_CPUTYPE}
> > M> +.else
> > M> +_TARGET_CPUTYPE=dummy
> > M> +.endif
> > M> +.if ${TARGET} == "arm"
> > M> +.if ${TARGET_CPUTYPE:M*soft*} == ""
> > M> +TARGET_TRIPLE_ABI= gnueabihf
> > M> +.else
> > M> +TARGET_TRIPLE_ABI= gnueabi
> > M> +.endif
> > M> +.endif
> > M> +MACHINE_TRIPLE_ABI?= unknown
> > M> +MACHINE_TRIPLE?=${MACHINE_ARCH:S/amd64/x86_64/}-${MACHINE_TRIPLE_ABI}-freebsd${OS_REVISION}
> > M> +TARGET_TRIPLE_ABI?= unknown
> > M> +TARGET_TRIPLE?= ${TARGET_ARCH:S/amd64/x86_64/}-${TARGET_TRIPLE_ABI}-freebsd${OS_REVISION}
> > M> +KNOWN_ARCHES?= aarch64/arm64 \
> > M> + amd64 \
> > M> + armv6/arm \
> > M> + armv7/arm \
> > M> + i386 \
> > M> + powerpc \
> > M> + powerpc64/powerpc \
> > M> + powerpc64le/powerpc \
> > M> + powerpcspe/powerpc \
> > M> + riscv64/riscv
> > M> +
> > M> +.if ${TARGET} == ${TARGET_ARCH}
> > M> +_t= ${TARGET}
> > M> +.else
> > M> +_t= ${TARGET_ARCH}/${TARGET}
> > M> +.endif
> > M> +.for _t in ${_t}
> > M> +.if empty(KNOWN_ARCHES:M${_t})
> > M> +.error Unknown target ${TARGET_ARCH}:${TARGET}.
> > M> +.endif
> > M> +.endfor
> > M> +
> > M> +.if ${TARGET_ARCH} == "amd64"
> > M> +LIBCOMPAT_INCLUDE_DIRS+= i386
> > M> +.elif ${TARGET_ARCH} == "aarch64"
> > M> +LIBCOMPAT_INCLUDE_DIRS+= arm
> > M> +.endif
> > M> +
> > M> +.if ${.MAKE.OS} != "FreeBSD"
> > M> +CROSSBUILD_HOST=${.MAKE.OS}
> > M> +.if ${.MAKE.OS} != "Linux" && ${.MAKE.OS} != "Darwin"
> > M> +.warning Unsupported crossbuild system: ${.MAKE.OS}. Build will probably fail!
> > M> +.endif
> > M> +# We need to force NO_ROOT/DB_FROM_SRC builds when building on other operating
> > M> +# systems since the BSD.foo.dist specs contain users and groups that do not
> > M> +# exist by default on a Linux/MacOS system.
> > M> +NO_ROOT:= 1
> > M> +DB_FROM_SRC:= 1
> > M> +.export NO_ROOT
> > M> +.endif
> > M> +
> > M> +# If all targets are disabled for system llvm then don't expect it to work
> > M> +# for cross-builds.
> > M> +.if !defined(TOOLS_PREFIX) && ${MK_LLVM_TARGET_ALL} == "no" && \
> > M> + ${MACHINE} != ${TARGET} && ${MACHINE_ARCH} != ${TARGET_ARCH} && \
> > M> + !make(showconfig)
> > M> +MK_SYSTEM_COMPILER= no
> > M> +MK_SYSTEM_LINKER= no
> > M> +.endif
> > M> +
> > M> +# Handle external binutils.
> > M> +.if defined(CROSS_TOOLCHAIN_PREFIX)
> > M> +CROSS_BINUTILS_PREFIX?=${CROSS_TOOLCHAIN_PREFIX}
> > M> +.endif
> > M> +XBINUTILS= AS AR ELFCTL LD NM OBJCOPY RANLIB SIZE STRINGS STRIPBIN
> > M> +.for BINUTIL in ${XBINUTILS}
> > M> +.if defined(CROSS_BINUTILS_PREFIX) && \
> > M> + exists(${CROSS_BINUTILS_PREFIX}/${${BINUTIL}})
> > M> +X${BINUTIL}?= ${CROSS_BINUTILS_PREFIX:C,/*$,,}/${${BINUTIL}}
> > M> +.else
> > M> +X${BINUTIL}?= ${${BINUTIL}}
> > M> +.endif
> > M> +.endfor
> > M> +
> > M> +# If a full path to an external linker is given, don't build lld.
> > M> +.if ${XLD:M/*}
> > M> +MK_LLD_BOOTSTRAP= no
> > M> +.endif
> > M> +
> > M> +# We also want the X_LINKER* variables if we are using an external toolchain.
> > M> +_WANT_TOOLCHAIN_CROSS_VARS= t
> > M> +.include "share/mk/bsd.linker.mk"
> > M> +.undef _WANT_TOOLCHAIN_CROSS_VARS
> > M> +
> > M> +# Begin WITH_SYSTEM_COMPILER / WITH_SYSTEM_LD
> > M> +
> > M> +# WITH_SYSTEM_COMPILER - Pull in needed values and make a decision.
> > M> +
> > M> +# Check if there is a local compiler that can satisfy as an external compiler.
> > M> +# Which compiler is expected to be used?
> > M> +.if ${MK_CLANG_BOOTSTRAP} == "yes"
> > M> +WANT_COMPILER_TYPE= clang
> > M> +.else
> > M> +WANT_COMPILER_TYPE=
> > M> +.endif
> > M> +
> > M> +.if !defined(WANT_COMPILER_FREEBSD_VERSION) && !make(showconfig) && \
> > M> + !make(test-system-linker)
> > M> +.if ${WANT_COMPILER_TYPE} == "clang"
> > M> +WANT_COMPILER_FREEBSD_VERSION_FILE= lib/clang/freebsd_cc_version.h
> > M> +WANT_COMPILER_FREEBSD_VERSION!= \
> > M> + awk '$$2 == "FREEBSD_CC_VERSION" {printf("%d\n", $$3)}' \
> > M> + ${SRCDIR}/${WANT_COMPILER_FREEBSD_VERSION_FILE} || echo unknown
> > M> +WANT_COMPILER_VERSION_FILE= lib/clang/include/clang/Basic/Version.inc
> > M> +WANT_COMPILER_VERSION!= \
> > M> + awk '$$2 == "CLANG_VERSION" {split($$3, a, "."); print a[1] * 10000 + a[2] * 100 + a[3]}' \
> > M> + ${SRCDIR}/${WANT_COMPILER_VERSION_FILE} || echo unknown
> > M> +.endif
> > M> +.export WANT_COMPILER_FREEBSD_VERSION WANT_COMPILER_VERSION
> > M> +.endif # !defined(WANT_COMPILER_FREEBSD_VERSION)
> > M> +
> > M> +# It needs to be the same revision as we would build for the bootstrap.
> > M> +# If the expected vs CC is different then we can't skip.
> > M> +# GCC cannot be used for cross-arch yet. For clang we pass -target later if
> > M> +# TARGET_ARCH!=MACHINE_ARCH.
> > M> +.if ${MK_SYSTEM_COMPILER} == "yes" && \
> > M> + defined(WANT_COMPILER_FREEBSD_VERSION) && \
> > M> + ${MK_CLANG_BOOTSTRAP} == "yes" && \
> > M> + !make(xdev*) && \
> > M> + ${X_COMPILER_TYPE} == ${WANT_COMPILER_TYPE} && \
> > M> + (${X_COMPILER_TYPE} == "clang" || ${TARGET_ARCH} == ${MACHINE_ARCH}) && \
> > M> + ${X_COMPILER_VERSION} == ${WANT_COMPILER_VERSION} && \
> > M> + ${X_COMPILER_FREEBSD_VERSION} == ${WANT_COMPILER_FREEBSD_VERSION}
> > M> +# Everything matches, disable the bootstrap compiler.
> > M> +MK_CLANG_BOOTSTRAP= no
> > M> +USING_SYSTEM_COMPILER= yes
> > M> +.endif # ${WANT_COMPILER_TYPE} == ${COMPILER_TYPE}
> > M> +
> > M> +# WITH_SYSTEM_LD - Pull in needed values and make a decision.
> > M> +
> > M> +# Check if there is a local linker that can satisfy as an external linker.
> > M> +# Which linker is expected to be used?
> > M> +.if ${MK_LLD_BOOTSTRAP} == "yes"
> > M> +WANT_LINKER_TYPE= lld
> > M> +.else
> > M> +WANT_LINKER_TYPE=
> > M> +.endif
> > M> +
> > M> +.if !defined(WANT_LINKER_FREEBSD_VERSION) && !make(showconfig) && \
> > M> + !make(test-system-compiler)
> > M> +.if ${WANT_LINKER_TYPE} == "lld"
> > M> +WANT_LINKER_FREEBSD_VERSION_FILE= lib/clang/include/lld/Common/Version.inc
> > M> +WANT_LINKER_FREEBSD_VERSION!= \
> > M> + awk '$$2 == "LLD_FREEBSD_VERSION" {print $$3}' \
> > M> + ${SRCDIR}/${WANT_LINKER_FREEBSD_VERSION_FILE} || echo unknown
> > M> +WANT_LINKER_VERSION_FILE= lib/clang/include/lld/Common/Version.inc
> > M> +WANT_LINKER_VERSION!= \
> > M> + awk '$$2 == "LLD_VERSION_STRING" {gsub("\"", "", $$3); split($$3, a, "."); print a[1] * 10000 + a[2] * 100 + a[3]}' \
> > M> + ${SRCDIR}/${WANT_LINKER_VERSION_FILE} || echo unknown
> > M> +.else
> > M> +WANT_LINKER_FREEBSD_VERSION_FILE=
> > M> +WANT_LINKER_FREEBSD_VERSION=
> > M> +.endif
> > M> +.export WANT_LINKER_FREEBSD_VERSION WANT_LINKER_VERSION
> > M> +.endif # !defined(WANT_LINKER_FREEBSD_VERSION)
> > M> +
> > M> +.if ${MK_SYSTEM_LINKER} == "yes" && \
> > M> + defined(WANT_LINKER_FREEBSD_VERSION) && \
> > M> + (${MK_LLD_BOOTSTRAP} == "yes") && \
> > M> + !make(xdev*) && \
> > M> + ${X_LINKER_TYPE} == ${WANT_LINKER_TYPE} && \
> > M> + ${X_LINKER_VERSION} == ${WANT_LINKER_VERSION} && \
> > M> + ${X_LINKER_FREEBSD_VERSION} == ${WANT_LINKER_FREEBSD_VERSION}
> > M> +# Everything matches, disable the bootstrap linker.
> > M> +MK_LLD_BOOTSTRAP= no
> > M> +USING_SYSTEM_LINKER= yes
> > M> +.endif # ${WANT_LINKER_TYPE} == ${LINKER_TYPE}
> > M> +
> > M> +# WITH_SYSTEM_COMPILER / WITH_SYSTEM_LINKER - Handle defaults and debug.
> > M> +USING_SYSTEM_COMPILER?= no
> > M> +USING_SYSTEM_LINKER?= no
> > M> +
> > M> +TEST_SYSTEM_COMPILER_VARS= \
> > M> + USING_SYSTEM_COMPILER MK_SYSTEM_COMPILER \
> > M> + MK_CROSS_COMPILER MK_CLANG_BOOTSTRAP \
> > M> + WANT_COMPILER_TYPE WANT_COMPILER_VERSION WANT_COMPILER_VERSION_FILE \
> > M> + WANT_COMPILER_FREEBSD_VERSION WANT_COMPILER_FREEBSD_VERSION_FILE \
> > M> + CC COMPILER_TYPE COMPILER_FEATURES COMPILER_VERSION \
> > M> + COMPILER_FREEBSD_VERSION \
> > M> + XCC X_COMPILER_TYPE X_COMPILER_FEATURES X_COMPILER_VERSION \
> > M> + X_COMPILER_FREEBSD_VERSION
> > M> +TEST_SYSTEM_LINKER_VARS= \
> > M> + USING_SYSTEM_LINKER MK_SYSTEM_LINKER \
> > M> + MK_LLD_BOOTSTRAP \
> > M> + WANT_LINKER_TYPE WANT_LINKER_VERSION WANT_LINKER_VERSION_FILE \
> > M> + WANT_LINKER_FREEBSD_VERSION WANT_LINKER_FREEBSD_VERSION_FILE \
> > M> + LD LINKER_TYPE LINKER_FEATURES LINKER_VERSION \
> > M> + LINKER_FREEBSD_VERSION \
> > M> + XLD X_LINKER_TYPE X_LINKER_FEATURES X_LINKER_VERSION \
> > M> + X_LINKER_FREEBSD_VERSION
> > M> +
> > M> +.for _t in compiler linker
> > M> +test-system-${_t}: .PHONY
> > M> +.for v in ${TEST_SYSTEM_${_t:tu}_VARS}
> > M> + ${_+_}@printf "%-35s= %s\n" "${v}" "${${v}}"
> > M> +.endfor
> > M> +.endfor
> > M> +.if (make(buildworld) || make(buildkernel) || make(kernel-toolchain) || \
> > M> + make(toolchain) || make(_cross-tools))
> > M> +.if ${USING_SYSTEM_COMPILER} == "yes"
> > M> +.info SYSTEM_COMPILER: Determined that CC=${CC} matches the source tree. Not bootstrapping a cross-compiler.
> > M> +.elif ${MK_CLANG_BOOTSTRAP} == "yes"
> > M> +.info SYSTEM_COMPILER: libclang will be built for bootstrapping a cross-compiler.
> > M> +.endif
> > M> +.if ${USING_SYSTEM_LINKER} == "yes"
> > M> +.info SYSTEM_LINKER: Determined that LD=${LD} matches the source tree. Not bootstrapping a cross-linker.
> > M> +.elif ${MK_LLD_BOOTSTRAP} == "yes"
> > M> +.info SYSTEM_LINKER: libclang will be built for bootstrapping a cross-linker.
> > M> +.endif
> > M> +.endif
> > M> +
> > M> +# End WITH_SYSTEM_COMPILER / WITH_SYSTEM_LD
> > M> +
> > M> +# Store some compiler metadata for use in installworld where we don't
> > M> +# want to invoke CC at all.
> > M> +_TOOLCHAIN_METADATA_VARS= COMPILER_VERSION \
> > M> + COMPILER_TYPE \
> > M> + COMPILER_FEATURES \
> > M> + COMPILER_FREEBSD_VERSION \
> > M> + COMPILER_RESOURCE_DIR \
> > M> + LINKER_VERSION \
> > M> + LINKER_FEATURES \
> > M> + LINKER_TYPE \
> > M> + LINKER_FREEBSD_VERSION
> > M> +toolchain-metadata.mk: .PHONY .META
> > M> + @: > ${.TARGET}
> > M> + @echo ".info Using cached toolchain metadata from build at $$(hostname) on $$(date)" \
> > M> + > ${.TARGET}
> > M> + @echo "_LOADED_TOOLCHAIN_METADATA=t" >> ${.TARGET}
> > M> +.for v in ${_TOOLCHAIN_METADATA_VARS}
> > M> + @echo "${v}=${${v}}" >> ${.TARGET}
> > M> + @echo "X_${v}=${X_${v}}" >> ${.TARGET}
> > M> +.endfor
> > M> + @echo ".export ${_TOOLCHAIN_METADATA_VARS}" >> ${.TARGET}
> > M> + @echo ".export ${_TOOLCHAIN_METADATA_VARS:C,^,X_,}" >> ${.TARGET}
> > M> +
> > M> +
> > M> +# We must do lib/ and libexec/ before bin/ in case of a mid-install error to
> > M> +# keep the users system reasonably usable. For static->dynamic root upgrades,
> > M> +# we don't want to install a dynamic binary without rtld and the needed
> > M> +# libraries. More commonly, for dynamic root, we don't want to install a
> > M> +# binary that requires a newer library version that hasn't been installed yet.
> > M> +# This ordering is not a guarantee though. The only guarantee of a working
> > M> +# system here would require fine-grained ordering of all components based
> > M> +# on their dependencies.
> > M> +.if !empty(SUBDIR_OVERRIDE)
> > M> +SUBDIR= ${SUBDIR_OVERRIDE}
> > M> +.else
> > M> +SUBDIR= lib libexec
> > M> +# Add LOCAL_LIB_DIRS, but only if they will not be picked up as a SUBDIR
> > M> +# of a LOCAL_DIRS directory. This allows LOCAL_DIRS=foo and
> > M> +# LOCAL_LIB_DIRS=foo/lib to behave as expected.
> > M> +.for _DIR in ${LOCAL_DIRS:M*/} ${LOCAL_DIRS:N*/:S|$|/|}
> > M> +_REDUNDANT_LIB_DIRS+= ${LOCAL_LIB_DIRS:M${_DIR}*}
> > M> +.endfor
> > M> +.for _DIR in ${LOCAL_LIB_DIRS}
> > M> +.if ${_DIR} == ".WAIT" || (empty(_REDUNDANT_LIB_DIRS:M${_DIR}) && exists(${.CURDIR}/${_DIR}/Makefile))
> > M> +SUBDIR+= ${_DIR}
> > M> +.endif
> > M> +.endfor
> > M> +.if !defined(NO_ROOT) && (make(installworld) || make(install))
> > M> +# Ensure libraries are installed before progressing.
> > M> +SUBDIR+=.WAIT
> > M> +.endif
> > M> +SUBDIR+=bin
> > M> +.if ${MK_CDDL} != "no"
> > M> +SUBDIR+=cddl
> > M> +.endif
> > M> +SUBDIR+=gnu include
> > M> +.if ${MK_KERBEROS} != "no"
> > M> +SUBDIR+=kerberos5
> > M> +.endif
> > M> +.if ${MK_RESCUE} != "no"
> > M> +SUBDIR+=rescue
> > M> +.endif
> > M> +SUBDIR+=sbin
> > M> +.if ${MK_CRYPT} != "no"
> > M> +SUBDIR+=secure
> > M> +.endif
> > M> +.if !defined(NO_SHARE)
> > M> +SUBDIR+=share
> > M> +.endif
> > M> +.if ${MK_BOOT} != "no"
> > M> +SUBDIR+=stand
> > M> +.endif
> > M> +SUBDIR+=sys usr.bin usr.sbin
> > M> +.if ${MK_TESTS} != "no"
> > M> +SUBDIR+= tests
> > M> +.endif
> > M> +
> > M> +# Local directories are built in parallel with the base system directories.
> > M> +# Users may insert a .WAIT directive at the beginning or elsewhere within
> > M> +# the LOCAL_DIRS and LOCAL_LIB_DIRS lists as needed.
> > M> +.for _DIR in ${LOCAL_DIRS}
> > M> +.if ${_DIR} == ".WAIT" || exists(${.CURDIR}/${_DIR}/Makefile)
> > M> +SUBDIR+= ${_DIR}
> > M> +.endif
> > M> +.endfor
> > M> +
> > M> +# We must do etc/ last as it hooks into building the man whatis file
> > M> +# by calling 'makedb' in share/man. This is only relevant for
> > M> +# install/distribute so they build the whatis file after every manpage is
> > M> +# installed.
> > M> +.if make(installworld) || make(install)
> > M> +SUBDIR+=.WAIT
> > M> +.endif
> > M> +SUBDIR+=etc
> > M> +
> > M> +.endif # !empty(SUBDIR_OVERRIDE)
> > M> +
> > M> +.if defined(NOCLEAN)
> > M> +.warning The src.conf WITHOUT_CLEAN option can now be used instead of NOCLEAN.
> > M> +MK_CLEAN:= no
> > M> +.endif
> > M> +.if defined(NO_CLEAN)
> > M> +.info The src.conf WITHOUT_CLEAN option can now be used instead of NO_CLEAN.
> > M> +MK_CLEAN:= no
> > M> +.endif
> > M> +.if defined(NO_CLEANDIR)
> > M> +CLEANDIR= clean cleandepend
> > M> +.else
> > M> +CLEANDIR= cleandir
> > M> +.endif
> > M> +
> > M> +.if defined(WORLDFAST)
> > M> +MK_CLEAN:= no
> > M> +NO_OBJWALK= t
> > M> +.endif
> > M> +
> > M> +.if ${MK_META_MODE} == "yes"
> > M> +# If filemon is used then we can rely on the build being incremental-safe.
> > M> +# The .meta files will also track the build command and rebuild should
> > M> +# it change.
> > M> +.if empty(.MAKE.MODE:Mnofilemon)
> > M> +MK_CLEAN:= no
> > M> +.endif
> > M> +.endif
> > M> +.if defined(NO_OBJWALK) || ${MK_AUTO_OBJ} == "yes"
> > M> +NO_OBJWALK= t
> > M> +NO_KERNELOBJ= t
> > M> +.endif
> > M> +.if !defined(NO_OBJWALK)
> > M> +_obj= obj
> > M> +.endif
> > M> +
> > M> +LOCAL_TOOL_DIRS?=
> > M> +PACKAGEDIR?= ${DESTDIR}/${DISTDIR}
> > M> +
> > M> +.if empty(SHELL:M*csh*)
> > M> +BUILDENV_SHELL?=${SHELL}
> > M> +.else
> > M> +BUILDENV_SHELL?=/bin/sh
> > M> +.endif
> > M> +
> > M> +.if !defined(_MKSHOWCONFIG)
> > M> +.if !defined(VCS_REVISION) || empty(VCS_REVISION)
> > M> +.if !defined(SVNVERSION_CMD) || empty(SVNVERSION_CMD)
> > M> +. for _D in ${PATH:S,:, ,g}
> > M> +. if exists(${_D}/svnversion)
> > M> +SVNVERSION_CMD?=${_D}/svnversion
> > M> +. endif
> > M> +. if exists(${_D}/svnliteversion)
> > M> +SVNVERSION_CMD?=${_D}/svnliteversion
> > M> +. endif
> > M> +. endfor
> > M> +.endif
> > M> +.if defined(SVNVERSION_CMD) && !empty(SVNVERSION_CMD)
> > M> +_VCS_REVISION?= $$(eval ${SVNVERSION_CMD} ${SRCDIR})
> > M> +. if !empty(_VCS_REVISION)
> > M> +VCS_REVISION= $$(echo r${_VCS_REVISION})
> > M> +.export VCS_REVISION
> > M> +. endif
> > M> +.endif
> > M> +.endif
> > M> +
> > M> +.if !defined(GIT_CMD) || empty(GIT_CMD)
> > M> +. for _P in /usr/bin /usr/local/bin
> > M> +. if exists(${_P}/git)
> > M> +GIT_CMD= ${_P}/git
> > M> +. endif
> > M> +. endfor
> > M> +.export GIT_CMD
> > M> +.endif
> > M> +
> > M> +.if !defined(OSRELDATE)
> > M> +.if exists(/usr/include/osreldate.h)
> > M> +OSRELDATE!= awk '/^\#define[[:space:]]*__FreeBSD_version/ { print $$3 }' \
> > M> + /usr/include/osreldate.h
> > M> +.else
> > M> +OSRELDATE= 0
> > M> +.endif
> > M> +.export OSRELDATE
> > M> +.endif
> > M> +
> > M> +# Set VERSION for CTFMERGE to use via the default CTFFLAGS=-L VERSION.
> > M> +.for _V in BRANCH REVISION TYPE
> > M> +.if !defined(_${_V})
> > M> +_${_V}!= eval $$(awk '/^${_V}=/{print}' ${SRCTOP}/sys/conf/newvers.sh); echo $$${_V}
> > M> +.export _${_V}
> > M> +.endif
> > M> +.endfor
> > M> +.if !defined(SRCRELDATE)
> > M> +SRCRELDATE!= awk '/^\#define[[:space:]]*__FreeBSD_version/ { print $$3 }' \
> > M> + ${SRCDIR}/sys/sys/param.h
> > M> +.export SRCRELDATE
> > M> +.endif
> > M> +.if !defined(VERSION)
> > M> +VERSION= FreeBSD ${_REVISION}-${_BRANCH:C/-p[0-9]+$//} ${TARGET_ARCH} ${SRCRELDATE}
> > M> +.export VERSION
> > M> +.endif
> > M> +MAJOR_REVISION= ${_REVISION:R}
> > M> +
> > M> +.if !defined(PKG_VERSION)
> > M> +_STRTIMENOW= %Y%m%d%H%M%S
> > M> +_TIMENOW= ${_STRTIMENOW:gmtime}
> > M> +.if ${_BRANCH:MCURRENT*} || ${_BRANCH:MSTABLE*} || ${_BRANCH:MPRERELEASE*}
> > M> +EXTRA_REVISION= .snap${_TIMENOW}
> > M> +.elif ${_BRANCH:MALPHA*}
> > M> +EXTRA_REVISION= .a${_BRANCH:C/ALPHA([0-9]+).*/\1/}.${_TIMENOW}
> > M> +.elif ${_BRANCH:MBETA*}
> > M> +EXTRA_REVISION= .b${_BRANCH:C/BETA([0-9]+).*/\1/}.${_TIMENOW}
> > M> +.elif ${_BRANCH:MRC*}
> > M> +EXTRA_REVISION= .rc${_BRANCH:C/RC([0-9]+).*/\1/}.${_TIMENOW}
> > M> +.elif ${_BRANCH:M*-p*}
> > M> +EXTRA_REVISION= p${_BRANCH:C/.*-p([0-9]+$)/\1/}
> > M> +.endif
> > M> +PKG_VERSION:= ${MAJOR_REVISION}${EXTRA_REVISION:C/[[:space:]]//g}
> > M> +.endif
> > M> +.endif # !defined(PKG_VERSION)
> > M> +
> > M> +.if !defined(PKG_TIMESTAMP)
> > M> +TIMEEPOCHNOW= %s
> > M> +SOURCE_DATE_EPOCH= ${TIMEEPOCHNOW:gmtime}
> > M> +.else
> > M> +SOURCE_DATE_EPOCH= ${PKG_TIMESTAMP}
> > M> +.endif
> > M> +
> > M> +PKG_NAME_PREFIX?= FreeBSD
> > M> +PKG_MAINTAINER?= re@FreeBSD.org
> > M> +PKG_WWW?= https://www.FreeBSD.org <https://www.freebsd.org/>;
> > M> +.export PKG_NAME_PREFIX
> > M> +.export PKG_MAINTAINER
> > M> +.export PKG_WWW
> > M> +
> > M> +.if !defined(_MKSHOWCONFIG)
> > M> +_CPUTYPE!= MAKEFLAGS= CPUTYPE=${_TARGET_CPUTYPE} ${MAKE} -f /dev/null \
> > M> + -m ${.CURDIR}/share/mk MK_AUTO_OBJ=no -V CPUTYPE
> > M> +.if ${_CPUTYPE} != ${_TARGET_CPUTYPE}
> > M> +.error CPUTYPE global should be set with ?=.
> > M> +.endif
> > M> +.endif
> > M> +.if make(buildworld)
> > M> +BUILD_ARCH!= uname -p
> > M> +# On some Linux systems uname -p returns "unknown" so skip this check there.
> > M> +# This check only exists to tell people to use TARGET_ARCH instead of
> > M> +# MACHINE_ARCH so skipping it when crossbuilding on non-FreeBSD should be fine.
> > M> +.if ${MACHINE_ARCH} != ${BUILD_ARCH} && ${.MAKE.OS} == "FreeBSD"
> > M> +.error To cross-build, set TARGET_ARCH.
> > M> +.endif
> > M> +.endif
> > M> +WORLDTMP?= ${OBJTOP}/tmp
> > M> +BPATH= ${CCACHE_WRAPPER_PATH_PFX}${WORLDTMP}/legacy/usr/sbin:${WORLDTMP}/legacy/usr/bin:${WORLDTMP}/legacy/bin:${WORLDTMP}/legacy/usr/libexec
> > M> +XPATH= ${WORLDTMP}/bin:${WORLDTMP}/usr/sbin:${WORLDTMP}/usr/bin
> > M> +
> > M> +# When building we want to find the cross tools before the host tools in ${BPATH}.
> > M> +# We also need to add UNIVERSE_TOOLCHAIN_PATH so that we can find the shared
> > M> +# toolchain files (clang, lld, etc.) during make universe/tinderbox
> > M> +STRICTTMPPATH= ${XPATH}:${BPATH}:${UNIVERSE_TOOLCHAIN_PATH}
> > M> +# We should not be using tools from /usr/bin accidentally since this could cause
> > M> +# the build to break on other systems that don't have that tool. For now we
> > M> +# still allow using the old behaviour (inheriting $PATH) if
> > M> +# BUILD_WITH_STRICT_TMPPATH is set to 0 but this will eventually be removed.
> > M> +
> > M> +# Currently strict $PATH can cause build failures. Once the remaining issues
> > M> +# have been resolved it will be turned on by default.
> > M> +BUILD_WITH_STRICT_TMPPATH?=0
> > M> +.if defined(CROSSBUILD_HOST)
> > M> +# When building on non-FreeBSD we can't rely on the tools in /usr/bin being compatible
> > M> +# with what FreeBSD expects. Therefore we only use tools from STRICTTMPPATH
> > M> +# during the world build stage. We build most tools during the bootstrap-tools
> > M> +# phase but symlink host tools that are known to work instead of building them
> > M> +BUILD_WITH_STRICT_TMPPATH:=1
> > M> +.endif
> > M> +.if ${BUILD_WITH_STRICT_TMPPATH} != 0
> > M> +TMPPATH= ${STRICTTMPPATH}
> > M> +.else
> > M> +TMPPATH= ${STRICTTMPPATH}:${PATH}
> > M> +.endif
> > M> +
> > M> +#
> > M> +# Avoid running mktemp(1) unless actually needed.
> > M> +# It may not be functional, e.g., due to new ABI
> > M> +# when in the middle of installing over this system.
> > M> +#
> > M> +.if make(distributeworld) || make(installworld) || make(stageworld)
> > M> +.if ${BUILD_WITH_STRICT_TMPPATH} != 0
> > M> +MKTEMP=${WORLDTMP}/legacy/usr/bin/mktemp
> > M> +.if !exists(${MKTEMP})
> > M> +.error mktemp binary doesn't exist in expected location: ${MKTEMP}
> > M> +.endif
> > M> +.else
> > M> +MKTEMP=mktemp
> > M> +.endif
> > M> +INSTALLTMP!= ${MKTEMP} -d -u -t install
> > M> +
> > M> +.if ${.MAKE.OS} == "FreeBSD"
> > M> +# When building on FreeBSD we always copy the host tools instead of linking
> > M> +# into INSTALLTMP to avoid issues with incompatible libraries (see r364030).
> > M> +# Note: we could create links if we don't intend to update the current machine.
> > M> +INSTALLTMP_COPY_HOST_TOOL=cp
> > M> +.else
> > M> +# However, this is not necessary on Linux/macOS. Additionally, copying the host
> > M> +# tools to another directory with cp results in AMFI Launch Constraint
> > M> +# Violations on macOS Ventura as part of its System Integrity Protection.
> > M> +INSTALLTMP_COPY_HOST_TOOL=ln -s
> > M> +.endif
> > M> +.endif
> > M> +
> > M> +.if make(stagekernel) || make(distributekernel)
> > M> +TAGS+= kernel
> > M> +PACKAGE= kernel
> > M> +.endif
> > M> +
> > M> +#
> > M> +# Building a world goes through the following stages
> > M> +#
> > M> +# 1. legacy stage [BMAKE]
> > M> +# This stage is responsible for creating compatibility
> > M> +# shims that are needed by the bootstrap-tools,
> > M> +# build-tools and cross-tools stages. These are generally
> > M> +# APIs that tools from one of those three stages need to
> > M> +# build that aren't present on the host.
> > M> +# 1. bootstrap-tools stage [BMAKE]
> > M> +# This stage is responsible for creating programs that
> > M> +# are needed for backward compatibility reasons. They
> > M> +# are not built as cross-tools.
> > M> +# 2. build-tools stage [TMAKE]
> > M> +# This stage is responsible for creating the object
> > M> +# tree and building any tools that are needed during
> > M> +# the build process. Some programs are listed during
> > M> +# this phase because they build binaries to generate
> > M> +# files needed to build these programs. This stage also
> > M> +# builds the 'build-tools' target rather than 'all'.
> > M> +# 3. cross-tools stage [XMAKE]
> > M> +# This stage is responsible for creating any tools that
> > M> +# are needed for building the system. A cross-compiler is one
> > M> +# of them. This differs from build tools in two ways:
> > M> +# 1. the 'all' target is built rather than 'build-tools'
> > M> +# 2. these tools are installed into TMPPATH for stage 4.
> > M> +# 4. world stage [WMAKE]
> > M> +# This stage actually builds the world.
> > M> +# 5. install stage (optional) [IMAKE]
> > M> +# This stage installs a previously built world.
> > M> +#
> > M> +
> > M> +BOOTSTRAPPING?= 0
> > M> +# Keep these in sync
> > M> +MINIMUM_SUPPORTED_OSREL?= 1104001
> > M> +MINIMUM_SUPPORTED_REL?= 11.4
> > M> +
> > M> +# Common environment for world related stages
> > M> +CROSSENV+= \
> > M> + MACHINE_ARCH=${TARGET_ARCH} \
> > M> + MACHINE=${TARGET} \
> > M> + CPUTYPE=${TARGET_CPUTYPE}
> > M> +.if ${MK_META_MODE} != "no"
> > M> +# Don't rebuild build-tools targets during normal build.
> > M> +CROSSENV+= BUILD_TOOLS_META=.NOMETA
> > M> +.endif
> > M> +.if defined(TARGET_CFLAGS)
> > M> +CROSSENV+= ${TARGET_CFLAGS}
> > M> +.endif
> > M> +.if (${TARGET} != ${MACHINE} && !defined(WITH_LOCAL_MODULES)) || \
> > M> + defined(WITHOUT_LOCAL_MODULES)
> > M> +CROSSENV+= LOCAL_MODULES=
> > M> +.endif
> > M> +
> > M> +BOOTSTRAPPING_OSRELDATE?=${OSRELDATE}
> > M> +
> > M> +# bootstrap-tools stage
> > M> +BMAKEENV= INSTALL="sh ${.CURDIR}/tools/install.sh" \
> > M> + TOOLS_PREFIX=${TOOLS_PREFIX_UNDEF:U${WORLDTMP}} \
> > M> + PATH=${BPATH:Q}:${PATH:Q} \
> > M> + WORLDTMP=${WORLDTMP} \
> > M> + MAKEFLAGS="-m ${.CURDIR}/tools/build/mk ${.MAKEFLAGS}"
> > M> +# need to keep this in sync with targets/pseudo/bootstrap-tools/Makefile
> > M> +BSARGS= DESTDIR= \
> > M> + OBJTOP='${WORLDTMP}/obj-tools' \
> > M> + OBJROOT='$${OBJTOP}/' \
> > M> + UNIVERSE_TOOLCHAIN_PATH=${UNIVERSE_TOOLCHAIN_PATH} \
> > M> + MAKEOBJDIRPREFIX= \
> > M> + BOOTSTRAPPING=${BOOTSTRAPPING_OSRELDATE} \
> > M> + BWPHASE=${.TARGET:C,^_,,} \
> > M> + -DNO_CPU_CFLAGS \
> > M> + -DNO_LINT \
> > M> + -DNO_PIC \
> > M> + -DNO_SHARED \
> > M> + MK_ASAN=no \
> > M> + MK_CTF=no \
> > M> + MK_CLANG_EXTRAS=no \
> > M> + MK_CLANG_FORMAT=no \
> > M> + MK_CLANG_FULL=no \
> > M> + MK_HTML=no \
> > M> + MK_MAN=no \
> > M> + MK_PROFILE=no \
> > M> + MK_RETPOLINE=no \
> > M> + MK_SSP=no \
> > M> + MK_TESTS=no \
> > M> + MK_UBSAN=no \
> > M> + MK_WERROR=no \
> > M> + MK_INCLUDES=yes \
> > M> + MK_MAN_UTILS=yes
> > M> +
> > M> +BMAKE= \
> > M> + ${TIME_ENV} ${BMAKEENV} ${MAKE} ${WORLD_FLAGS} -f Makefile.inc1 \
> > M> + ${BSARGS}
> > M> +.if empty(.MAKEOVERRIDES:MMK_LLVM_TARGET_ALL)
> > M> +BMAKE+= MK_LLVM_TARGET_ALL=no
> > M> +.endif
> > M> +
> > M> +# build-tools stage
> > M> +TMAKE= \
> > M> + ${TIME_ENV} ${BMAKEENV} ${MAKE} ${WORLD_FLAGS} -f Makefile.inc1 \
> > M> + TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH} \
> > M> + DESTDIR= \
> > M> + BOOTSTRAPPING=${BOOTSTRAPPING_OSRELDATE} \
> > M> + BWPHASE=${.TARGET:C,^_,,} \
> > M> + -DNO_CPU_CFLAGS \
> > M> + -DNO_LINT \
> > M> + MK_ASAN=no \
> > M> + MK_CTF=no \
> > M> + MK_CLANG_EXTRAS=no \
> > M> + MK_CLANG_FORMAT=no \
> > M> + MK_CLANG_FULL=no \
> > M> + MK_LLDB=no \
> > M> + MK_RETPOLINE=no \
> > M> + MK_SSP=no \
> > M> + MK_TESTS=no \
> > M> + MK_UBSAN=no \
> > M> + MK_WERROR=no
> > M> +
> > M> +# cross-tools stage
> > M> +# TOOLS_PREFIX set in BMAKE
> > M> +XMAKE= ${BMAKE} \
> > M> + TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH} \
> > M> + MK_LLDB=no \
> > M> + MK_LLVM_BINUTILS=no \
> > M> + MK_TESTS=no
> > M> +
> > M> +# kernel-tools stage
> > M> +KTMAKEENV= INSTALL="sh ${.CURDIR}/tools/install.sh" \
> > M> + PATH=${BPATH:Q}:${PATH:Q} \
> > M> + WORLDTMP=${WORLDTMP} \
> > M> + MAKEFLAGS="-m ${.CURDIR}/tools/build/mk ${.MAKEFLAGS}"
> > M> +
> > M> +KTMAKE= ${TIME_ENV} \
> > M> + TOOLS_PREFIX=${TOOLS_PREFIX_UNDEF:U${WORLDTMP}} \
> > M> + ${KTMAKEENV} ${MAKE} ${WORLD_FLAGS} -f Makefile.inc1 \
> > M> + DESTDIR= \
> > M> + OBJTOP='${WORLDTMP}/obj-kernel-tools' \
> > M> + OBJROOT='$${OBJTOP}/' \
> > M> + UNIVERSE_TOOLCHAIN_PATH=${UNIVERSE_TOOLCHAIN_PATH} \
> > M> + MAKEOBJDIRPREFIX= \
> > M> + BOOTSTRAPPING=${BOOTSTRAPPING_OSRELDATE} \
> > M> + -DNO_CPU_CFLAGS \
> > M> + -DNO_LINT \
> > M> + -DNO_PIC \
> > M> + -DNO_SHARED \
> > M> + MK_CTF=no \
> > M> + MK_HTML=no \
> > M> + MK_MAN=no \
> > M> + MK_PROFILE=no \
> > M> + MK_SSP=no \
> > M> + MK_RETPOLINE=no \
> > M> + MK_WERROR=no
> > M> +
> > M> +# world stage
> > M> +WMAKEENV= ${CROSSENV} \
> > M> + INSTALL="${INSTALL_CMD} -U" \
> > M> + PATH=${TMPPATH:Q} \
> > M> + SYSROOT=${WORLDTMP}
> > M> +
> > M> +# make hierarchy
> > M> *** 5431 LINES SKIPPED ***
> >
>
> --
> Alexander Motin
>
--06e1cace14e44ba6b615e2ddd8eabb7c
Content-Type: text/html
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE html><html><head><title></title><style type=3D"text/css">p.Mso=
Normal,p.MsoNoSpacing{margin:0}</style></head><body><div><br></div><div>=
<br></div><div>On Wed, Aug 30, 2023, at 8:01 PM, Alexander Motin wrote:<=
br></div><blockquote type=3D"cite" id=3D"qt" style=3D""><div>Hi Gleb,<br=
></div><div><br></div><div>It is the first time I see a panic like this.=
I'll think about it <br></div><div>tomorrow. But I'd a=
ppreciate any information on what is your workload <br></div><div>a=
nd what are you doing related to ZIL (O_SYNC, fsync(), sync=3Dalways,&nb=
sp;<br></div><div>etc) to trigger it? What is your pool configurat=
ion?<br></div></blockquote><div><br></div><div>I'm not Gleb, but this wa=
s something at $WORK, so I can perhaps help. I've included the out=
put of zpool status, and all non-default settings in the zpool. No=
te that we don't use a ZIL device.<br></div><div><br></div><div>Thanks,<=
br></div><div><br></div><div>Drew<br></div><div><br></div><div><br></div=
><div># zpool status<br></div><div> pool: zroot<br></div><div>stat=
e: ONLINE<br></div><div>status: Some supported and requested features ar=
e not enabled on the pool.<br></div><div>The pool can still be used, but=
some features are unavailable.<br></div><div>action: Enable all feature=
s using 'zpool upgrade'. Once this is done,<br></div><div>the pool may n=
o longer be accessible by software that does not support<br></div><div>t=
he features. See zpool-features(7) for details.<br></div><div>config:<br=
></div><div><br></div><div>NAME  =
; STATE READ WRITE CKSUM<br></div><div>zroot&nbs=
p; ONLINE &nb=
sp; 0 0 0<br></div><div>=
raidz1-0 ONLINE 0 =
0 0<br></div><div>  =
; nda0p2 ONLINE 0 &=
nbsp; 0 0<br></div><div> =
nda1p2 ONLINE 0 &n=
bsp; 0 0<br></div><div> &=
nbsp; nda2p2 ONLINE 0 &nb=
sp; 0 0<br></div><div>spares<br></di=
v><div> nda3p2 AVAIL <br></div>=
<div><br></div><div>errors: No known data errors<br></div><div><br></div=
><div># zpool get all | grep -v default<br></div><div>NAME P=
ROPERTY  =
; VALU=
E  =
;  =
;  =
;  =
; SOURCE<br>=
</div><div>zroot size &nb=
sp; &nb=
sp; 149G &nbs=
p; &nbs=
p; &nbs=
p; &nbs=
p; &nbs=
p; -<br></div><div>zroot capac=
ity &nb=
sp; 14% =
;  =
;  =
;  =
;  =
; -<br=
></div><div>zroot health =
=
ONLINE  =
;  =
;  =
;  =
;  =
; -<br></div><div>zroot guid &n=
bsp; &n=
bsp; 5099990=
586719219765 =
=
=
-<br></div>=
<div>zroot bootfs &=
nbsp; &=
nbsp; zroot/ROOT/NCD-7827-freebsd-2023-08-29-merge-oca=
-r202308291935 local<br></div><div>zroot dedupratio &nb=
sp; &nb=
sp; 1.00x &nb=
sp; &nb=
sp; &nb=
sp; &nb=
sp; &nb=
sp; -<br></div><div>zroot free =
=
=
127G &=
nbsp; &=
nbsp; &=
nbsp; &=
nbsp; &=
nbsp; -<br></div><div>zroot allocated  =
;  =
; 22.3G  =
;  =
;  =
;  =
;  =
; -<br></div><div>zroot readonly &nbs=
p; &nbs=
p; off =
=
=
=
=
-<br></div><div>zroot&n=
bsp; expandsize &nb=
sp; - &=
nbsp; &=
nbsp; &=
nbsp; &=
nbsp; &=
nbsp; &=
nbsp; -<br></div><div>zroot freeing &=
nbsp; &=
nbsp; 0  =
;  =
;  =
;  =
;  =
; -<br></div><div>zroot&=
nbsp; fragmentation  =
; 28% &=
nbsp; &=
nbsp; &=
nbsp; &=
nbsp; &=
nbsp; -<br></div><div>zr=
oot leaked &n=
bsp; &n=
bsp; 0 =
=
=
=
=
-<br></div><div>zroot checkpoint&nb=
sp; &nb=
sp; - &=
nbsp; &=
nbsp; &=
nbsp; &=
nbsp; &=
nbsp; -<br></div><=
div>zroot load_guid  =
;  =
; 7612195721260396606 &nb=
sp; &nb=
sp; &nb=
sp; &nb=
sp; -<br></div><div>zroot bcloneused =
=
0 &nbs=
p; &nbs=
p; &nbs=
p; &nbs=
p; &nbs=
p; -<br></div><div>zroot bclonesaved=
=
0 &nbs=
p; &nbs=
p; &nbs=
p; &nbs=
p; &nbs=
p; -<br></div><div=
>zroot bcloneratio =
1.00x=
=
=
=
=
-<br></div>=
<div>zroot feature@async_destroy &nbs=
p; enabled &n=
bsp; &n=
bsp; &n=
bsp; &n=
bsp; &n=
bsp; local<br></div><div>zroot feature@empty_bpobj &nbs=
p; active &nb=
sp; &nb=
sp; &nb=
sp; &nb=
sp; &nb=
sp; local<br></div><div>zroot =
feature@lz4_compress &nbs=
p; active &nb=
sp; &nb=
sp; &nb=
sp; &nb=
sp; lo=
cal<br></div><div>zroot feature@multi_vdev_crash_dump enable=
d  =
;  =
;  =
;  =
; local<br></div><div>zr=
oot feature@spacemap_histogram active =
;  =
;  =
;  =
;  =
; local<br></div><div>zr=
oot feature@enabled_txg &=
nbsp; active =
=
=
=
=
local<br></div><div>zroot feature@hole_birth &nb=
sp; active&n=
bsp; &n=
bsp; &n=
bsp; &n=
bsp; &n=
bsp; local<br></div><div=
>zroot feature@extensible_dataset active&n=
bsp; &n=
bsp; &n=
bsp; &n=
bsp; &n=
bsp; local<br></div><div=
>zroot feature@embedded_data &n=
bsp; active &=
nbsp; &=
nbsp; &=
nbsp; &=
nbsp; &=
nbsp; local<br></div><div>zroot feature@bookmarks  =
; enabled&nb=
sp; &nb=
sp; &nb=
sp; &nb=
sp; &nb=
sp; local<br></div><div>zroot&=
nbsp; feature@filesystem_limits enabled&nb=
sp; &nb=
sp; &nb=
sp; &nb=
sp; &nb=
sp; local<br></div><div>zroot&=
nbsp; feature@large_blocks &nbs=
p; enabled &n=
bsp; &n=
bsp; &n=
bsp; &n=
bsp; l=
ocal<br></div><div>zroot feature@large_dnode &nbs=
p; enabled &n=
bsp; &n=
bsp; &n=
bsp; &n=
bsp; &n=
bsp; local<br></div><div>zroot feature@sha=
512 &nb=
sp; enabled &=
nbsp; &=
nbsp; &=
nbsp; &=
nbsp; &=
nbsp; local<br></div><div>zroot feature@skein &nb=
sp; &nb=
sp; enabled &=
nbsp; &=
nbsp; &=
nbsp; &=
nbsp; =
local<br></div><div>zroot feature@edonr &nb=
sp; &nb=
sp; disabled =
=
=
=
local<br></div><d=
iv>zroot feature@userobj_accounting active=
=
=
=
=
local<br></div><d=
iv>zroot feature@encryption &nb=
sp; enabled &=
nbsp; &=
nbsp; &=
nbsp; &=
nbsp; &=
nbsp; local<br></div><div>zroot feature@project_quota&=
nbsp; active =
=
=
=
=
local<br></div><div>zroot&nbs=
p; feature@device_removal  =
; enabled &nb=
sp; &nb=
sp; &nb=
sp; &nb=
sp; local<br></div=
><div>zroot feature@obsolete_counts &=
nbsp; enabled  =
;  =
;  =
;  =
; loca=
l<br></div><div>zroot feature@zpool_checkpoint &n=
bsp; enabled =
=
=
=
=
local<br></div><div>zroot feature@spacemap_v2 &n=
bsp; active &=
nbsp; &=
nbsp; &=
nbsp; &=
nbsp; &=
nbsp; local<br></div><div>zroot feat=
ure@allocation_classes enabled =
=
=
=
=
local<br></div><div>zroot feature@r=
esilver_defer enabled&nb=
sp; &nb=
sp; &nb=
sp; &nb=
sp; &nb=
sp; local<br></div><div>zroot&=
nbsp; feature@bookmark_v2  =
; enabled &nb=
sp; &nb=
sp; &nb=
sp; &nb=
sp; &nb=
sp; local<br></div><div>zroot feature@redaction_bookmarks &nb=
sp; enabled &=
nbsp; &=
nbsp; &=
nbsp; &=
nbsp; local<=
br></div><div>zroot feature@redacted_datasets &nb=
sp; enabled &=
nbsp; &=
nbsp; &=
nbsp; &=
nbsp; local<=
br></div><div>zroot feature@bookmark_written &nbs=
p; enabled &n=
bsp; &n=
bsp; &n=
bsp; &n=
bsp; l=
ocal<br></div><div>zroot feature@log_spacemap &nb=
sp; active &n=
bsp; &n=
bsp; &n=
bsp; &n=
bsp; &n=
bsp; local<br></div><div>zroot feature@liv=
elist &=
nbsp; enabled  =
;  =
;  =
;  =
;  =
; local<br></div><div>zroot feature@device_rebuild &nbs=
p; enabled &n=
bsp; &n=
bsp; &n=
bsp; &n=
bsp; &n=
bsp; local<br></div><div>zroot feature@zstd_compress&n=
bsp; enabled =
=
=
=
=
local<br></div><div>zroot fea=
ture@draid &n=
bsp; enabled =
=
=
=
=
local<br></div><div>zroot feature@zilsaxa=
ttr &nb=
sp; disabled =
=
=
=
local<br></div><d=
iv>zroot feature@head_errlog &n=
bsp; disabled  =
;  =
;  =
;  =
;  =
; local<br></div><div>zroot feature@blake3 =
=
disabled &nb=
sp; &nb=
sp; &nb=
sp; &nb=
sp; local<br></div=
><div>zroot feature@block_cloning &nb=
sp; disabled =
=
=
=
=
local<br></div><div>zroot feature@vdev_zaps_v2 &=
nbsp; disabled &nbs=
p; &nbs=
p; &nbs=
p; &nbs=
p; &nbs=
p; local<br></div><div><br></div><div><br></div><block=
quote type=3D"cite" id=3D"qt" style=3D""><div>On 30.08.2023 20:09, Gleb =
Smirnoff wrote:<br></div><div>> Hi Martin,<br></div=
><div>> <br></div><div>> we hit this panic with this update:<=
br></div><div>> <br></div><div>> panic: VERIFY(avl_is_empty(&=
amp;lwb->lwb_vdev_tree)) failed<br></div><div>> <br></div><di=
v>> cpuid =3D 8<br></div><div>> time =3D 1693432004<br></div><div>=
> KDB: stack backtrace:<br></div><div>> db_trace_self_wrapper() at=
db_trace_self_wrapper+0x2b/frame 0xfffffe09519aa800<br></div><div>> =
vpanic() at vpanic+0x132/frame 0xfffffe09519aa930<br></div><div>> spl=
_panic() at spl_panic+0x3a/frame 0xfffffe09519aa990<br></div><div>> z=
il_free_lwb() at zil_free_lwb+0x1de/frame 0xfffffe09519aa9c0<br></div><d=
iv>> zil_sync() at zil_sync+0x3cb/frame 0xfffffe09519aaaa0<br></div><=
div>> [Wed Aug 30 21:46:44 2023]dmu_objset_sync() at dmu_objset_sync+=
0x51b/frame 0xfffffe09519aab70<br></div><div>> dsl_pool_sync() at dsl=
_pool_sync+0x11d/frame 0xfffffe09519aabf0<br></div><div>> spa_sync() =
at spa_sync+0xc68/frame 0xfffffe09519aae20<br></div><div>> txg_sync_t=
hread() at txg_sync_thread+0x2eb/frame 0xfffffe09519aaef0<br></div><div>=
> fork_exit() at fork_exit+0x82/frame 0xfffffe09519aaf30<br></div><di=
v>> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe09519aaf30=
<br></div><div>> --- trap 0x26d90a2f, rip =3D 0xbba54c81b25fd8d4, rsp=
=3D 0xff60c2f6f9874bcd, rbp =3D 0x37940e3e9793bd40 --<br></div><div>>=
; <br></div><div>> I have two cores dumped for this panic.<br></=
div><div>> <br></div><div>> Our previous version is based on =
9c0a2d522f3a18fc795a7dd6266d2be2aff4b084, which<br></div><div>> I gue=
ss has the most recent ZFS merge openzfs/zfs@009d3288d.<br></div><div>&g=
t; <br></div><div>> On Sun, Aug 27, 2023 at 05:09:06AM +0000, Ma=
rtin Matuska wrote:<br></div><div>> M> The branch main has been up=
dated by mm:<br></div><div>> M><br></div><div>> M> URL: =
;<a href=3D"https://cgit.FreeBSD.org/src/commit/?id=3D315ee00fa9616b0a19=
2b6834911f98bcf5316a6b">https://cgit.FreeBSD.org/src/commit/?id=3D315ee0=
0fa9616b0a192b6834911f98bcf5316a6b</a><br></div><div>> M><br></div=
><div>> M> commit 315ee00fa9616b0a192b6834911f98bcf5316a6b<br></di=
v><div>> M> Merge: b3a714652ff0 804414aad224<br></div><div>> M&=
gt; Author: Martin Matuska <<a href=3D"mailto=
:mm@FreeBSD.org">mm@FreeBSD.org</a>><br></div><div>> M> AuthorD=
ate: 2023-08-26 21:20:04 +0000<br></div><div>> M> Commit: &nb=
sp; Martin Matuska <<a href=3D"mailto:mm@FreeBSD.org">mm@=
FreeBSD.org</a>><br></div><div>> M> CommitDate: 2023-08-26 21:5=
1:42 +0000<br></div><div>> M><br></div><div>> M> =
zfs: merge openzfs/zfs@804414aad<br></div><div>> M><b=
r></div><div>> M> Notable upstream pull re=
quest merges:<br></div><div>> M> &nbs=
p; #15024 Add missed DMU_PROJECTUSED_OBJECT prefetch<br></div><div>> =
M> #15029 Do not request data L1 =
buffers on scan prefetch<br></div><div>> M>  =
; #15036 FreeBSD: catch up to __FreeBSD_version 1400093<br><=
/div><div>> M> #15039 Fix raw =
receive with different indirect block size<br></div><div>> M> =
; #15047 FreeBSD: Fix build on stable/13 a=
fter 1302506<br></div><div>> M>  =
; #15049 Fix the ZFS checksum error histograms with larger record sizes<=
br></div><div>> M> #15052 Redu=
ce bloat in ereport.fs.zfs.checksum events<br></div><div>> M> =
; #15056 Avoid extra snprintf() in dsl_dea=
dlist_merge()<br></div><div>> M> &nbs=
p; #15061 Ignore pool ashift property during vdev attachment<br></div><d=
iv>> M> #15063 Don't panic if =
setting vdev properties is unsupported for this vdev type<br></div><div>=
> M> #15067 spa_min_alloc shou=
ld be GCD, not min<br></div><div>> M>  =
; #15071 Add explicit prefetches to bpobj_iterate()<br></div><div>=
> M> #15072 Adjust prefetch pa=
rameters<br></div><div>> M> #1=
5076 Refactor dmu_prefetch()<br></div><div>> M> &=
nbsp; #15079 set autotrim default to 'off' everywhere<br></d=
iv><div>> M> #15080 ZIL: Fix c=
onfig lock deadlock<br></div><div>> M> &nbs=
p; #15088 metaslab: tuneable to better control force ganging<br></=
div><div>> M> #15096 Avoid wai=
ting in dmu_sync_late_arrival()<br></div><div>> M> &nbs=
p; #15097 BRT should return EOPNOTSUPP<br></div><div>&=
gt; M> #15103 Remove zl_issuer_lo=
ck from zil_suspend()<br></div><div>> M> &n=
bsp; #15107 Remove fastwrite mechanism<br></div><div>> M>&nb=
sp; #15113 libzfs: sendrecv: send_progress=
_thread: handle SIGINFO/SIGUSR1<br></div><div>> M> &nbs=
p; #15122 ZIL: Second attempt to reduce scope of zl_is=
suer_lock<br></div><div>> M> #=
15129 zpool_vdev_remove() should handle EALREADY error return<br></div><=
div>> M> #15132 ZIL: Replay bl=
ocks without next block pointer<br></div><div>> M> &nbs=
p; #15148 zfs_clone_range should return descriptive er=
ror codes<br></div><div>> M> #=
15153 ZIL: Avoid dbuf_read() before dmu_sync()<br></div><div>> M>&=
nbsp; #15172 copy_file_range: fix fallback=
when source create on same txg<br></div><div>> M> &nbs=
p; #15180 Update outdated assertion from zio_write_com=
press<br></div><div>> M><br></div><div>> M>  =
; Obtained from: OpenZFS<br></div><div>> M>  =
; OpenZFS commit: 804414aad224b432590afe3f9ec114ffb49e0f13<b=
r></div><div>> M><br></div><div>> M> Makefile.inc1&nbs=
p; &nbs=
p; &nbs=
p; &nbs=
p; | 5 +-<br></div><div>> M> cddl/lib/libzfs/Mak=
efile &=
nbsp; &=
nbsp; | 1 +<br></div><div>> M> cddl/=
lib/libzfs/Makefile.depend &nbs=
p; |&n=
bsp; 1 +<br></div><div>> M> rescue/rescue/Makefile&nbs=
p; &nbs=
p; &nbs=
p; | 2 +-<br></div><div>> M> r=
escue/rescue/Makefile.depend &n=
bsp; &n=
bsp; | 1 +<br></div><div>> M> share/mk/src=
.libnames.mk =
=
| 2 +-<br></div><div>> M>&nbs=
p; sys/contrib/openzfs/META &nb=
sp; &nb=
sp; | 6 +-<br></div><div=
>> M> sys/contrib/openzfs/cmd/zdb/zdb.c &nb=
sp; &nb=
sp; | 77 ++-<br></div><div>> M> sys/contrib/open=
zfs/cmd/zed/agents/zfs_mod.c | =
2 -<br></div><div>> M> sys/contrib/openzfs/cmd/zed/ag=
ents/zfs_retire.c | 5 +<br></div><div>>=
M> .../openzfs/cmd/zed/zed.d/statechange-slot_off.sh |&n=
bsp; 61 ++<br></div><div>> M> sys/contrib/openzfs/cmd/zed/ze=
d.d/zed.rc |=
5 +<br></div><div>> M> sys/contrib/openzfs/cmd/=
ztest.c  =
; | 2 +-<br>=
</div><div>> M> sys/contrib/openzfs/config/Shellcheck.am&nbs=
p; | 3=
+-<br></div><div>> M> sys/contrib/openzfs/config/kernel-blk=
dev.m4 | 138 +++-<br></div><di=
v>> M> .../config/kernel-block-device-operations.m4 &nb=
sp; | 35 +-<br></div><div>> M> =
.../openzfs/config/kernel-filemap-splice-read.m4 | 25=
+<br></div><div>> M> .../openzfs/config/kernel-register_sys=
ctl_table.m4 | 27 +<br></div><div>> M> .../config/kern=
el-vfs-extended-file_range.m4 | =
; 50 ++<br></div><div>> M> .../openzfs/config/kernel-vfs-fil=
e_range.m4 | 164 +++++<br></di=
v><div>> M> sys/contrib/openzfs/config/kernel-vfs-iov_iter.m=
4 | 26 +-<br></div><div>> M> sys/contrib/openzfs=
/config/kernel.m4 &=
nbsp; | 16 +<br></div><div>> M> =
; sys/contrib/openzfs/contrib/debian/changelog &n=
bsp; | 6 +<br></div><div>> M> .../contrib/=
debian/openzfs-zfs-test.install  =
; | 2 -<br></div><div>> M> .../contrib/debian/op=
enzfs-zfsutils.install | =
2 +<br></div><div>> M> .../contrib/dracut/90zfs/zfs-e=
nv-bootfs.service.in | 11 +-<br></div><div>> M> .../dr=
acut/90zfs/zfs-rollback-bootfs.service.in |  =
; 2 +-<br></div><div>> M> .../openzfs/include/os/freebsd/spl=
/sys/vnode.h | 6 +-<br></div><div>&g=
t; M> .../include/os/linux/kernel/linux/blkdev_compat.h |=
17 +<br></div><div>> M> .../openzfs/include/os/linux/=
spl/sys/kmem_cache.h | 8 +<br></div><div>> M>&nb=
sp; .../openzfs/include/os/linux/spl/sys/types.h =
| 2 +-<br></div><div>> M> sys/contr=
ib/openzfs/include/os/linux/spl/sys/uio.h | 12 +<br></div><div>>=
; M> sys/contrib/openzfs/include/os/linux/zfs/sys/zpl.h | =
53 ++<br></div><div>> M> sys/contrib/openzfs/include/sys/bp=
obj.h =
| 2 +-<br></div><div>> M> sys/contrib/openzfs/in=
clude/sys/brt.h &nb=
sp; | 1 +<br></div><div>> M> s=
ys/contrib/openzfs/include/sys/dmu.h =
| 5 +<br></div><d=
iv>> M> sys/contrib/openzfs/include/sys/dmu_impl.h &nbs=
p; | 2 -<br></div><div>&=
gt; M> sys/contrib/openzfs/include/sys/dmu_zfetch.h =
| 2 -<br></div><div>> M> =
; sys/contrib/openzfs/include/sys/fm/fs/zfs.h &nb=
sp; | 4 -<br></div><div>> M> sys/con=
trib/openzfs/include/sys/metaslab.h &=
nbsp; | 3 -<br></div><div>> M> sys/contrib=
/openzfs/include/sys/metaslab_impl.h | 6 +=
-<br></div><div>> M> sys/contrib/openzfs/include/sys/spa_imp=
l.h | 1 +<br=
></div><div>> M> sys/contrib/openzfs/include/sys/vdev_impl.h=
| 2 +-<br></div><=
div>> M> sys/contrib/openzfs/include/sys/zil_impl.h &nb=
sp; | 44 +-<br></div><div>>=
M> sys/contrib/openzfs/include/sys/zio.h &nbs=
p; | 1=
-<br></div><div>> M> sys/contrib/openzfs/include/sys/zio_ch=
ecksum.h | 2 -<br></div><div>> M&=
gt; sys/contrib/openzfs/lib/libzfs/Makefile.am &n=
bsp; | 2 +-<br></div><div>> M>=
sys/contrib/openzfs/lib/libzfs/libzfs_pool.c &nb=
sp; | 6 +<br></div><div>> M> sys/con=
trib/openzfs/lib/libzfs/libzfs_sendrecv.c | 95 ++-<br>=
</div><div>> M> sys/contrib/openzfs/man/man4/zfs.4 &nbs=
p; &nbs=
p; | 10 +-<br></div><div>> M> sys/contrib/=
openzfs/man/man8/zfs-send.8 &nb=
sp; | 18 +-<br></div><div>> M> sys/c=
ontrib/openzfs/man/man8/zpool-events.8 &nbs=
p; | 23 +-<br></div><div>> M> sys/contrib/openzf=
s/module/Kbuild.in =
| 1 +<br></div><div>> M>=
; .../openzfs/module/os/freebsd/zfs/zfs_vnops_os.c |&n=
bsp; 10 +-<br></div><div>> M> sys/contrib/openzfs/module/os/=
linux/spl/spl-proc.c | 26 +-<br></div><div>> M> .../op=
enzfs/module/os/linux/zfs/vdev_disk.c  =
; | 65 +-<br></div><div>> M> .../openzfs/module/=
os/linux/zfs/zfs_ctldir.c | 26=
+-<br></div><div>> M> .../openzfs/module/os/linux/zfs/zfs_v=
fsops.c | 7 +<br></div><=
div>> M> .../openzfs/module/os/linux/zfs/zfs_vnops_os.c =
; | 9 +-<br></div><div>> M> ..=
./openzfs/module/os/linux/zfs/zfs_znode.c &=
nbsp; | 8 +<br></div><div>> M> .../openzfs=
/module/os/linux/zfs/zpl_ctldir.c |&=
nbsp; 2 +-<br></div><div>> M> sys/contrib/openzfs/modu=
le/os/linux/zfs/zpl_file.c | 44 +-<br></div><div>> M> =
.../openzfs/module/os/linux/zfs/zpl_file_range.c | 272 +++++=
+++<br></div><div>> M> .../openzfs/module/os/linux/zfs/zpl_s=
uper.c | 39 +-<br></div>=
<div>> M> sys/contrib/openzfs/module/os/linux/zfs/zvol_os.c&=
nbsp; | 28 +-<br></div><div>> M> sys/contrib/openzfs/m=
odule/zfs/bpobj.c &=
nbsp; | 49 +-<br></div><div>> M> sys/contr=
ib/openzfs/module/zfs/brt.c &nb=
sp; | 43 +-<br></div><div>>=
M> sys/contrib/openzfs/module/zfs/dbuf.c &nbs=
p; | 11 +-<b=
r></div><div>> M> sys/contrib/openzfs/module/zfs/dmu.c =
=
| 127 ++--<br></div><div>> M> sys/contrib/openzfs/mod=
ule/zfs/dmu_recv.c =
| 22 +-<br></div><div>> M> sys/contrib/openzfs/module=
/zfs/dmu_zfetch.c | 12 +=
-<br></div><div>> M> sys/contrib/openzfs/module/zfs/dnode.c&=
nbsp; =
| 31 +-<br></div><div>> M> sys/contrib/openzfs/module/=
zfs/dsl_deadlist.c | 14 +-<br></div>=
<div>> M> sys/contrib/openzfs/module/zfs/dsl_scan.c &nb=
sp; | 17 +-<br></div><di=
v>> M> sys/contrib/openzfs/module/zfs/metaslab.c =
| 83 +--<br></div><div>=
> M> sys/contrib/openzfs/module/zfs/spa_log_spacemap.c =
| 4 +-<br></div><div>> M> sys/contrib/openzfs/m=
odule/zfs/spa_misc.c &nbs=
p; | 1 +<br></div><div>> M> sys/contrib/openzfs/=
module/zfs/vdev.c &=
nbsp; | 90 ++-<br></div><div>> M> sy=
s/contrib/openzfs/module/zfs/vdev_indirect.c |&n=
bsp; 2 +-<br></div><div>> M> sys/contrib/openzfs/modul=
e/zfs/vdev_raidz.c | &nbs=
p; 2 +-<br></div><div>> M> sys/contrib/openzfs/module/zfs/zf=
s_fm.c =
| 33 +-<br></div><div>> M> sys/contrib/openzfs/module=
/zfs/zfs_vnops.c | =
46 +-<br></div><div>> M> sys/contrib/openzfs/module/zfs/zil=
.c &nbs=
p; | 719 +++++++++------------<br></div><div>> M> =
; sys/contrib/openzfs/module/zfs/zio.c &nbs=
p; | 45 +-<br></di=
v><div>> M> sys/contrib/openzfs/module/zfs/zio_checksum.c&nb=
sp; | 2 -<br></div><div>> M>&n=
bsp; sys/contrib/openzfs/module/zfs/zvol.c =
| 8 +-<br><=
/div><div>> M> sys/contrib/openzfs/tests/runfiles/linux.run&=
nbsp; | 11 +<br></div><div>> M>=
; .../openzfs/tests/test-runner/bin/zts-report.py.in | 18 +<=
br></div><div>> M> sys/contrib/openzfs/tests/zfs-tests/cmd/.=
gitignore | 1 +<br></div><div>> M> .../openzfs/t=
ests/zfs-tests/cmd/Makefile.am =
| 1 +<br></div><div>> M> .../openzfs/tests/zfs-=
tests/cmd/clonefile.c | 333 ++=
++++++++<br></div><div>> M> sys/contrib/openzfs/tests/zfs-te=
sts/cmd/readmmap.c | 1 +<br></div><div>> M> .../=
openzfs/tests/zfs-tests/include/commands.cfg | 1=
+<br></div><div>> M> .../openzfs/tests/zfs-tests/tests/Make=
file.am | 14 +<br></div><div>> M&=
gt; .../functional/block_cloning/block_cloning.kshlib | =
; 54 ++<br></div><div>> M> .../block_cloning/block_cloning_c=
opyfilerange.ksh | 60 ++<br></div><div>> M> .../=
block_cloning_copyfilerange_cross_dataset.ksh | 65 ++<br></d=
iv><div>> M> .../block_cloning_copyfilerange_fallback.ksh&nb=
sp; | 86 +++<br></div><div>> M>=
; ...ock_cloning_copyfilerange_fallback_same_txg.ksh | 66 ++=
<br></div><div>> M> .../block_cloning_copyfilerange_partial.=
ksh | 68 ++<br></div><di=
v>> M> .../block_cloning_disabled_copyfilerange.ksh &nb=
sp; | 60 ++<br></div><div>> M> =
.../block_cloning_disabled_ficlone.ksh &nb=
sp; | 50 ++<br></div><div>>=
M> .../block_cloning_disabled_ficlonerange.ksh &nbs=
p; | 50 ++<br></div><div>> M> =
.../block_cloning/block_cloning_ficlone.ksh  =
; | 56 ++<br></div><div>> M> .../block_clo=
ning/block_cloning_ficlonerange.ksh | 56 ++<br></div><=
div>> M> .../block_cloning_ficlonerange_partial.ksh &nb=
sp; | 64 ++<br></div><div>>=
M> .../tests/functional/block_cloning/cleanup.ksh &=
nbsp; | 34 +<br></div><div>> M> .../tests/functi=
onal/block_cloning/setup.ksh | =
36 ++<br></div><div>> M> .../cli_root/zpool_attach/attach-o=
_ashift.ksh | 30 +-<br></div><div>&g=
t; M> .../cli_root/zpool_replace/replace-o_ashift.ksh &nbs=
p; | 32 +-<br></div><div>> M> .../cli_root/zpool=
_replace/replace_prop_ashift.ksh | 24 +-<br></div><div>> M>&=
nbsp; sys/modules/zfs/zfs_config.h &n=
bsp; &n=
bsp; | 53 +-<br></div><div>> M> sys/=
modules/zfs/zfs_gitrev.h =
=
| 2 +-<br></div><div>> M> 113 files=
changed, 3322 insertions(+), 906 deletions(-)<br></div><div>> M><=
br></div><div>> M> diff --cc Makefile.inc1<br></div><div>> M>=
; index 8838d31c795a,000000000000..f740cc5abcca<br></div><div>> M>=
mode 100644,000000..100644<br></div><div>> M> --- a/Makefile.inc1=
<br></div><div>> M> +++ b/Makefile.inc1<br></div><div>> M> @=
@@ -1,3692 -1,0 +1,3695 @@@<br></div><div>> M> +#<br></div><=
div>> M> +#<br></div><div>> M> +# Make command l=
ine options:<br></div><div>> M> +# -DNO_CLEANDIR run ${MAKE}=
clean, instead of ${MAKE} cleandir<br></div><div>> M> +# -D=
NO_CLEAN do not clean at all<br></div><div>> M> +# -DDB_FROM=
_SRC use the user/group databases in src/etc instead of<br></div><div>&g=
t; M> +# the system database when installing=
.<br></div><div>> M> +# -DNO_SHARE do not go into share subd=
ir<br></div><div>> M> +# -DKERNFAST define NO_KERNEL{CONFIG,=
CLEAN,OBJ}<br></div><div>> M> +# -DNO_KERNELCONFIG do not ru=
n config in ${MAKE} buildkernel<br></div><div>> M> +# -DNO_K=
ERNELCLEAN do not run ${MAKE} clean in ${MAKE} buildkernel<br></div><div=
>> M> +# -DNO_KERNELOBJ do not run ${MAKE} obj in ${MAKE} bu=
ildkernel<br></div><div>> M> +# -DNO_ROOT install without us=
ing root privilege<br></div><div>> M> +# -DWITHOUT_CTF do no=
t run the DTrace CTF conversion tools on built objects<br></div><div>>=
; M> +# LOCAL_DIRS=3D"list of dirs" to add additional dirs to t=
he SUBDIR list<br></div><div>> M> +# LOCAL_ITOOLS=3D"list of=
tools" to add additional tools to the ITOOLS list<br></div><div>> M&=
gt; +# LOCAL_LIB_DIRS=3D"list of dirs" to add additional dirs to l=
ibraries target<br></div><div>> M> +# LOCAL_MTREE=3D"list of=
mtree files" to process to allow local directories<br></div><div>> M=
> +# to be created before files are installe=
d<br></div><div>> M> +# LOCAL_LEGACY_DIRS=3D"list of dirs" t=
o add additional dirs to the legacy<br></div><div>> M> +# &n=
bsp; target<br></div><div>> M> +# LOCAL_BSTOOL_D=
IRS=3D"list of dirs" to add additional dirs to the<br></div><div>> M&=
gt; +# bootstrap-tools target<br></div><div>>=
; M> +# LOCAL_TOOL_DIRS=3D"list of dirs" to add additional dirs=
to the build-tools<br></div><div>> M> +# =
target<br></div><div>> M> +# LOCAL_XTOOL_DIRS=3D"list of di=
rs" to add additional dirs to the<br></div><div>> M> +# &nbs=
p; cross-tools target<br></div><div>> M> +# META=
LOG=3D"path to metadata log" to write permission and ownership<br></div>=
<div>> M> +# when NO_ROOT is set. (=
default: ${DESTDIR}/${DISTDIR}/METALOG,<br></div><div>> M> +=
# check /etc=
/make.conf for DISTDIR)<br></div><div>> M> +# TARGET=3D"mach=
ine" to crossbuild world for a different machine type<br></div><div>>=
M> +# TARGET_ARCH=3D may be required when a TARGET supports mu=
ltiple endians<br></div><div>> M> +# BUILDENV_SHELL=3D shell=
to launch for the buildenv target (def:${SHELL})<br></div><div>> M&g=
t; +# WORLD_FLAGS=3D additional flags to pass to make(1) during bu=
ildworld<br></div><div>> M> +# KERNEL_FLAGS=3D additional fl=
ags to pass to make(1) during buildkernel<br></div><div>> M> =
+# SUBDIR_OVERRIDE=3D"list of dirs" to build rather than everything.<br=
></div><div>> M> +# All libraries and inc=
ludes, and some build tools will still build.<br></div><div>> M>&n=
bsp; +<br></div><div>> M> +#<br></div><div>> M> =
+# The intended user-driven targets are:<br></div><div>> M> =
+# buildworld - rebuild *everything*, including glue to help do up=
grades<br></div><div>> M> +# installworld- install everythin=
g built by "buildworld"<br></div><div>> M> +# checkworld&nbs=
p; - run test suite on installed world<br></div><div>> M> +#=
doxygen - build API documentation of the kernel=
<br></div><div>> M> +#<br></div><div>> M> +# Sta=
ndard targets (not defined here) are documented in the makefiles in<br><=
/div><div>> M> +# /usr/share/mk. These include:<br></d=
iv><div>> M> +# obj depend all install clean cleandepend cl=
eanobj<br></div><div>> M> +<br></div><div>> M> +=
.if !defined(TARGET) || !defined(TARGET_ARCH)<br></div><div>> M>&n=
bsp; +.error Both TARGET and TARGET_ARCH must be defined.<br></div><div>=
> M> +.endif<br></div><div>> M> +<br></div><div>=
> M> +.if make(showconfig) || make(test-system-*)<br></div><=
div>> M> +_MKSHOWCONFIG=3D t<br></div><div>> M> =
+.endif<br></div><div>> M> +<br></div><div>> M> =
+SRCDIR?=3D ${.CURDIR}<br></div><div>> M> +LOCALBASE?=3D /us=
r/local<br></div><div>> M> +TIME_ENV ?=3D time env<br></div>=
<div>> M> +<br></div><div>> M> +.include "share/=
mk/src.tools.mk"<br></div><div>> M> +<br></div><div>> M&g=
t; +# Cross toolchain changes must be in effect before bsd.compile=
r.mk<br></div><div>> M> +# so that gets the right CC, and pa=
ss CROSS_TOOLCHAIN to submakes.<br></div><div>> M> +.if defi=
ned(CROSS_TOOLCHAIN)<br></div><div>> M> +.if exists(${LOCALB=
ASE}/share/toolchains/${CROSS_TOOLCHAIN}.mk)<br></div><div>> M>&nb=
sp; +.include "${LOCALBASE}/share/toolchains/${CROSS_TOOLCHAIN}.mk"<br><=
/div><div>> M> +.elif exists(${CROSS_TOOLCHAIN})<br></div><d=
iv>> M> +.include "${CROSS_TOOLCHAIN}"<br></div><div>> M&=
gt; +.else<br></div><div>> M> +.error CROSS_TOOLCHAIN =
${CROSS_TOOLCHAIN} not found<br></div><div>> M> +.endif<br><=
/div><div>> M> +CROSSENV+=3DCROSS_TOOLCHAIN=3D"${CROSS_TOOLC=
HAIN}"<br></div><div>> M> +.elif defined(UNIVERSE_TOOLCHAIN)=
<br></div><div>> M> +UNIVERSE_TOOLCHAIN_PATH?=3D${HOST_OBJTO=
P}/tmp/usr/bin<br></div><div>> M> +XCC?=3D"${UNIVERSE_TOOLCH=
AIN_PATH}/cc"<br></div><div>> M> +XCXX?=3D"${UNIVERSE_TOOLCH=
AIN_PATH}/c++"<br></div><div>> M> +XCPP?=3D"${UNIVERSE_TOOLC=
HAIN_PATH}/cpp"<br></div><div>> M> +XLD?=3D"${UNIVERSE_TOOLC=
HAIN_PATH}/ld"<br></div><div>> M> +.endif<br></div><div>>=
M> +.if defined(CROSS_TOOLCHAIN_PREFIX)<br></div><div>> M&g=
t; +CROSS_COMPILER_PREFIX?=3D${CROSS_TOOLCHAIN_PREFIX}<br></div><d=
iv>> M> +.endif<br></div><div>> M> +<br></div><d=
iv>> M> +XCOMPILERS=3D CC CXX CPP<br></div><div>> M>&n=
bsp; +.for COMPILER in ${XCOMPILERS}<br></div><div>> M> +.if=
defined(CROSS_COMPILER_PREFIX)<br></div><div>> M> +X${COMPI=
LER}?=3D ${CROSS_COMPILER_PREFIX}${${COMPILER}}<br></div><div>> M>=
+.else<br></div><div>> M> +X${COMPILER}?=3D ${${COMPI=
LER}}<br></div><div>> M> +.endif<br></div><div>> M>&nb=
sp; +.endfor<br></div><div>> M> +# If a full path to an exte=
rnal cross compiler is given, don't build<br></div><div>> M> =
+# a cross compiler.<br></div><div>> M> +.if ${XCC:N${CCACH=
E_BIN}:M/*}<br></div><div>> M> +MK_CLANG_BOOTSTRAP=3D no<br>=
</div><div>> M> +# Make sure sub-makes see the option as dis=
abled so the hack in bsd.sys.mk to<br></div><div>> M> +# wor=
k around incompatible headers in Clang's resource directory is enabled.<=
br></div><div>> M> +.MAKEOVERRIDES+=3D MK_CLANG_BOOTSTRAP<br=
></div><div>> M> +.endif<br></div><div>> M> +<br=
></div><div>> M> +# Pull in compiler metadata from buildworl=
d/toolchain if possible to avoid<br></div><div>> M> +# runni=
ng CC from bsd.compiler.mk.<br></div><div>> M> +.if make(ins=
tallworld) || make(install) || make(distributeworld) || \<br></div><div>=
> M> + make(stageworld)<br></div><div>>=
M> +.-include "${OBJTOP}/toolchain-metadata.mk"<br></div><div>=
> M> +.if !defined(_LOADED_TOOLCHAIN_METADATA)<br></div><div=
>> M> +.error A build is required first. You may have =
the wrong MAKEOBJDIRPREFIX set.<br></div><div>> M> +.endif<b=
r></div><div>> M> +.endif<br></div><div>> M> +<b=
r></div><div>> M> +# Pull in COMPILER_TYPE and COMPILER_FREE=
BSD_VERSION early. Pull it from the<br></div><div>> M> +# tr=
ee to be friendlier to foreign OS builds. It's safe to do so uncondition=
ally<br></div><div>> M> +# here since we will always have th=
e right make, unlike in src/Makefile<br></div><div>> M> +# D=
on't include bsd.linker.mk yet until XBINUTILS is handled (after src.opt=
s.mk)<br></div><div>> M> +_NO_INCLUDE_LINKERMK=3D t<br></div=
><div>> M> +# We also want the X_COMPILER* variables if we a=
re using an external toolchain.<br></div><div>> M> +_WANT_TO=
OLCHAIN_CROSS_VARS=3D t<br></div><div>> M> +.include "share/=
mk/bsd.compiler.mk"<br></div><div>> M> +.undef _NO_INCLUDE_L=
INKERMK<br></div><div>> M> +.undef _WANT_TOOLCHAIN_CROSS_VAR=
S<br></div><div>> M> +# src.opts.mk depends on COMPILER_FEAT=
URES<br></div><div>> M> +.include "share/mk/src.opts.mk"<br>=
</div><div>> M> +<br></div><div>> M> +.if ${TARG=
ET} =3D=3D ${MACHINE}<br></div><div>> M> +TARGET_CPUTYPE?=3D=
${CPUTYPE}<br></div><div>> M> +.else<br></div><div>> M>=
; +TARGET_CPUTYPE?=3D<br></div><div>> M> +.endif<br></=
div><div>> M> +.if !empty(TARGET_CPUTYPE)<br></div><div>>=
M> +_TARGET_CPUTYPE=3D${TARGET_CPUTYPE}<br></div><div>> M&g=
t; +.else<br></div><div>> M> +_TARGET_CPUTYPE=3Ddummy<=
br></div><div>> M> +.endif<br></div><div>> M> +.=
if ${TARGET} =3D=3D "arm"<br></div><div>> M> +.if ${TARGET_C=
PUTYPE:M*soft*} =3D=3D ""<br></div><div>> M> +TARGET_TRIPLE_=
ABI=3D gnueabihf<br></div><div>> M> +.else<br></div><div>>=
; M> +TARGET_TRIPLE_ABI=3D gnueabi<br></div><div>> M>&nbs=
p; +.endif<br></div><div>> M> +.endif<br></div><div>> M&g=
t; +MACHINE_TRIPLE_ABI?=3D unknown<br></div><div>> M> =
+MACHINE_TRIPLE?=3D${MACHINE_ARCH:S/amd64/x86_64/}-${MACHINE_TRIPLE_ABI}=
-freebsd${OS_REVISION}<br></div><div>> M> +TARGET_TRIPLE_ABI=
?=3D unknown<br></div><div>> M> +TARGET_TRIPLE?=3D ${TARGET_=
ARCH:S/amd64/x86_64/}-${TARGET_TRIPLE_ABI}-freebsd${OS_REVISION}<br></di=
v><div>> M> +KNOWN_ARCHES?=3D aarch64/arm64 \<br></div><div>=
> M> + amd64 \<br></div><div>> M> + armv6/arm =
\<br></div><div>> M> + armv7/arm \<br></div><div>> M>=
+ i386 \<br></div><div>> M> + powerpc \<br></div><d=
iv>> M> + powerpc64/powerpc \<br></div><div>> M> =
; + powerpc64le/powerpc \<br></div><div>> M> + powerpcspe/=
powerpc \<br></div><div>> M> + riscv64/riscv<br></div><div>=
> M> +<br></div><div>> M> +.if ${TARGET} =3D=3D =
${TARGET_ARCH}<br></div><div>> M> +_t=3D ${TARGET}<br></div=
><div>> M> +.else<br></div><div>> M> +_t=3D ${T=
ARGET_ARCH}/${TARGET}<br></div><div>> M> +.endif<br></div><d=
iv>> M> +.for _t in ${_t}<br></div><div>> M> +.i=
f empty(KNOWN_ARCHES:M${_t})<br></div><div>> M> +.error Unkn=
own target ${TARGET_ARCH}:${TARGET}.<br></div><div>> M> +.en=
dif<br></div><div>> M> +.endfor<br></div><div>> M>&nbs=
p; +<br></div><div>> M> +.if ${TARGET_ARCH} =3D=3D "amd64"<b=
r></div><div>> M> +LIBCOMPAT_INCLUDE_DIRS+=3D i386<br></div>=
<div>> M> +.elif ${TARGET_ARCH} =3D=3D "aarch64"<br></div><d=
iv>> M> +LIBCOMPAT_INCLUDE_DIRS+=3D arm<br></div><div>> M=
> +.endif<br></div><div>> M> +<br></div><div>> M=
> +.if ${.MAKE.OS} !=3D "FreeBSD"<br></div><div>> M> =
; +CROSSBUILD_HOST=3D${.MAKE.OS}<br></div><div>> M> +.if ${.=
MAKE.OS} !=3D "Linux" && ${.MAKE.OS} !=3D "Darwin"<br></div><div=
>> M> +.warning Unsupported crossbuild system: ${.MAKE.OS}. =
Build will probably fail!<br></div><div>> M> +.endif<br></di=
v><div>> M> +# We need to force NO_ROOT/DB_FROM_SRC builds w=
hen building on other operating<br></div><div>> M> +# system=
s since the BSD.foo.dist specs contain users and groups that do not<br><=
/div><div>> M> +# exist by default on a Linux/MacOS system.<=
br></div><div>> M> +NO_ROOT:=3D 1<br></div><div>> M>&n=
bsp; +DB_FROM_SRC:=3D 1<br></div><div>> M> +.export NO_ROOT<=
br></div><div>> M> +.endif<br></div><div>> M> +<=
br></div><div>> M> +# If all targets are disabled for system=
llvm then don't expect it to work<br></div><div>> M> +# for=
cross-builds.<br></div><div>> M> +.if !defined(TOOLS_PREFIX=
) && ${MK_LLVM_TARGET_ALL} =3D=3D "no" && \<br></div><di=
v>> M> + ${MACHINE} !=3D ${TARGET} &&a=
mp; ${MACHINE_ARCH} !=3D ${TARGET_ARCH} && \<br></div><div>> =
M> + !make(showconfig)<br></div><div>> M&g=
t; +MK_SYSTEM_COMPILER=3D no<br></div><div>> M> +MK_SY=
STEM_LINKER=3D no<br></div><div>> M> +.endif<br></div><div>&=
gt; M> +<br></div><div>> M> +# Handle external binu=
tils.<br></div><div>> M> +.if defined(CROSS_TOOLCHAIN_PREFIX=
)<br></div><div>> M> +CROSS_BINUTILS_PREFIX?=3D${CROSS_TOOLC=
HAIN_PREFIX}<br></div><div>> M> +.endif<br></div><div>> M=
> +XBINUTILS=3D AS AR ELFCTL LD NM OBJCOPY RANLIB SIZE STRINGS =
STRIPBIN<br></div><div>> M> +.for BINUTIL in ${XBINUTILS}<br=
></div><div>> M> +.if defined(CROSS_BINUTILS_PREFIX) &&a=
mp; \<br></div><div>> M> + exists(${CROSS_=
BINUTILS_PREFIX}/${${BINUTIL}})<br></div><div>> M> +X${BINUT=
IL}?=3D ${CROSS_BINUTILS_PREFIX:C,/*$,,}/${${BINUTIL}}<br></div><div>>=
; M> +.else<br></div><div>> M> +X${BINUTIL}?=3D ${$=
{BINUTIL}}<br></div><div>> M> +.endif<br></div><div>> M&g=
t; +.endfor<br></div><div>> M> +<br></div><div>> M&=
gt; +# If a full path to an external linker is given, don't build =
lld.<br></div><div>> M> +.if ${XLD:M/*}<br></div><div>> M=
> +MK_LLD_BOOTSTRAP=3D no<br></div><div>> M> +.endi=
f<br></div><div>> M> +<br></div><div>> M> +# We =
also want the X_LINKER* variables if we are using an external toolchain.=
<br></div><div>> M> +_WANT_TOOLCHAIN_CROSS_VARS=3D t<br></di=
v><div>> M> +.include "share/mk/bsd.linker.mk"<br></div><div=
>> M> +.undef _WANT_TOOLCHAIN_CROSS_VARS<br></div><div>> =
M> +<br></div><div>> M> +# Begin WITH_SYSTEM_COMPIL=
ER / WITH_SYSTEM_LD<br></div><div>> M> +<br></div><div>> =
M> +# WITH_SYSTEM_COMPILER - Pull in needed values and make a d=
ecision.<br></div><div>> M> +<br></div><div>> M> =
+# Check if there is a local compiler that can satisfy as an external c=
ompiler.<br></div><div>> M> +# Which compiler is expected to=
be used?<br></div><div>> M> +.if ${MK_CLANG_BOOTSTRAP} =3D=3D=
"yes"<br></div><div>> M> +WANT_COMPILER_TYPE=3D clang<br></=
div><div>> M> +.else<br></div><div>> M> +WANT_CO=
MPILER_TYPE=3D<br></div><div>> M> +.endif<br></div><div>>=
M> +<br></div><div>> M> +.if !defined(WANT_COMPILE=
R_FREEBSD_VERSION) && !make(showconfig) && \<br></div><d=
iv>> M> + !make(test-system-linker)<br></d=
iv><div>> M> +.if ${WANT_COMPILER_TYPE} =3D=3D "clang"<br></=
div><div>> M> +WANT_COMPILER_FREEBSD_VERSION_FILE=3D lib/cla=
ng/freebsd_cc_version.h<br></div><div>> M> +WANT_COMPILER_FR=
EEBSD_VERSION!=3D \<br></div><div>> M> + awk '$$2 =3D=3D "FR=
EEBSD_CC_VERSION" {printf("%d\n", $$3)}' \<br></div><div>> M> =
; + ${SRCDIR}/${WANT_COMPILER_FREEBSD_VERSION_FILE} || echo unknown<br><=
/div><div>> M> +WANT_COMPILER_VERSION_FILE=3D lib/clang/incl=
ude/clang/Basic/Version.inc<br></div><div>> M> +WANT_COMPILE=
R_VERSION!=3D \<br></div><div>> M> + awk '$$2 =3D=3D "CLANG_=
VERSION" {split($$3, a, "."); print a[1] * 10000 + a[2] * 100 + a[3]}' \=
<br></div><div>> M> + ${SRCDIR}/${WANT_COMPILER_VERSION_FILE=
} || echo unknown<br></div><div>> M> +.endif<br></div><div>&=
gt; M> +.export WANT_COMPILER_FREEBSD_VERSION WANT_COMPILER_VER=
SION<br></div><div>> M> +.endif # !defined(WANT_COMPILER_FRE=
EBSD_VERSION)<br></div><div>> M> +<br></div><div>> M>&=
nbsp; +# It needs to be the same revision as we would build for the boot=
strap.<br></div><div>> M> +# If the expected vs CC is differ=
ent then we can't skip.<br></div><div>> M> +# GCC cannot be =
used for cross-arch yet. For clang we pass -target later if<br></d=
iv><div>> M> +# TARGET_ARCH!=3DMACHINE_ARCH.<br></div><div>&=
gt; M> +.if ${MK_SYSTEM_COMPILER} =3D=3D "yes" && \<br>=
</div><div>> M> + defined(WANT_COMPILER_FR=
EEBSD_VERSION) && \<br></div><div>> M> + =
${MK_CLANG_BOOTSTRAP} =3D=3D "yes" && \<br></div><div>>=
; M> + !make(xdev*) && \<br></div><di=
v>> M> + ${X_COMPILER_TYPE} =3D=3D ${WANT_=
COMPILER_TYPE} && \<br></div><div>> M> + =
(${X_COMPILER_TYPE} =3D=3D "clang" || ${TARGET_ARCH} =3D=3D ${MAC=
HINE_ARCH}) && \<br></div><div>> M> + &nb=
sp; ${X_COMPILER_VERSION} =3D=3D ${WANT_COMPILER_VERSION} && \<b=
r></div><div>> M> + ${X_COMPILER_FREEBSD_V=
ERSION} =3D=3D ${WANT_COMPILER_FREEBSD_VERSION}<br></div><div>> M>=
+# Everything matches, disable the bootstrap compiler.<br></div><=
div>> M> +MK_CLANG_BOOTSTRAP=3D no<br></div><div>> M>&=
nbsp; +USING_SYSTEM_COMPILER=3D yes<br></div><div>> M> +.end=
if # ${WANT_COMPILER_TYPE} =3D=3D ${COMPILER_TYPE}<br></div><div>> M&=
gt; +<br></div><div>> M> +# WITH_SYSTEM_LD - Pull in n=
eeded values and make a decision.<br></div><div>> M> +<br></=
div><div>> M> +# Check if there is a local linker that can s=
atisfy as an external linker.<br></div><div>> M> +# Which li=
nker is expected to be used?<br></div><div>> M> +.if ${MK_LL=
D_BOOTSTRAP} =3D=3D "yes"<br></div><div>> M> +WANT_LINKER_TY=
PE=3D lld<br></div><div>> M> +.else<br></div><div>> M>=
; +WANT_LINKER_TYPE=3D<br></div><div>> M> +.endif<br><=
/div><div>> M> +<br></div><div>> M> +.if !define=
d(WANT_LINKER_FREEBSD_VERSION) && !make(showconfig) && \=
<br></div><div>> M> + !make(test-system-co=
mpiler)<br></div><div>> M> +.if ${WANT_LINKER_TYPE} =3D=3D "=
lld"<br></div><div>> M> +WANT_LINKER_FREEBSD_VERSION_FILE=3D=
lib/clang/include/lld/Common/Version.inc<br></div><div>> M> =
+WANT_LINKER_FREEBSD_VERSION!=3D \<br></div><div>> M> + awk=
'$$2 =3D=3D "LLD_FREEBSD_VERSION" {print $$3}' \<br></div><div>> M&g=
t; + ${SRCDIR}/${WANT_LINKER_FREEBSD_VERSION_FILE} || echo unknown=
<br></div><div>> M> +WANT_LINKER_VERSION_FILE=3D lib/clang/i=
nclude/lld/Common/Version.inc<br></div><div>> M> +WANT_LINKE=
R_VERSION!=3D \<br></div><div>> M> + awk '$$2 =3D=3D "LLD_VE=
RSION_STRING" {gsub("\"", "", $$3); split($$3, a, "."); print a[1] * 100=
00 + a[2] * 100 + a[3]}' \<br></div><div>> M> + ${SRCDIR}/${=
WANT_LINKER_VERSION_FILE} || echo unknown<br></div><div>> M> =
+.else<br></div><div>> M> +WANT_LINKER_FREEBSD_VERSION_FILE=
=3D<br></div><div>> M> +WANT_LINKER_FREEBSD_VERSION=3D<br></=
div><div>> M> +.endif<br></div><div>> M> +.expor=
t WANT_LINKER_FREEBSD_VERSION WANT_LINKER_VERSION<br></div><div>> M&g=
t; +.endif # !defined(WANT_LINKER_FREEBSD_VERSION)<br></div><div>&=
gt; M> +<br></div><div>> M> +.if ${MK_SYSTEM_LINKER=
} =3D=3D "yes" && \<br></div><div>> M> + =
defined(WANT_LINKER_FREEBSD_VERSION) && \<br></div><div>&=
gt; M> + (${MK_LLD_BOOTSTRAP} =3D=3D "yes") &=
amp;& \<br></div><div>> M> + !make(xde=
v*) && \<br></div><div>> M> + ${X_=
LINKER_TYPE} =3D=3D ${WANT_LINKER_TYPE} && \<br></div><div>> =
M> + ${X_LINKER_VERSION} =3D=3D ${WANT_LINKER=
_VERSION} && \<br></div><div>> M> +  =
; ${X_LINKER_FREEBSD_VERSION} =3D=3D ${WANT_LINKER_FREEBSD_VERSION}<br><=
/div><div>> M> +# Everything matches, disable the bootstrap =
linker.<br></div><div>> M> +MK_LLD_BOOTSTRAP=3D no<br></div>=
<div>> M> +USING_SYSTEM_LINKER=3D yes<br></div><div>> M&g=
t; +.endif # ${WANT_LINKER_TYPE} =3D=3D ${LINKER_TYPE}<br></div><d=
iv>> M> +<br></div><div>> M> +# WITH_SYSTEM_COMP=
ILER / WITH_SYSTEM_LINKER - Handle defaults and debug.<br></div><div>>=
; M> +USING_SYSTEM_COMPILER?=3D no<br></div><div>> M>&nbs=
p; +USING_SYSTEM_LINKER?=3D no<br></div><div>> M> +<br></div=
><div>> M> +TEST_SYSTEM_COMPILER_VARS=3D \<br></div><div>>=
; M> + USING_SYSTEM_COMPILER MK_SYSTEM_COMPILER \<br></div><div=
>> M> + MK_CROSS_COMPILER MK_CLANG_BOOTSTRAP \<br></div><div=
>> M> + WANT_COMPILER_TYPE WANT_COMPILER_VERSION WANT_COMPIL=
ER_VERSION_FILE \<br></div><div>> M> + WANT_COMPILER_FREEBSD=
_VERSION WANT_COMPILER_FREEBSD_VERSION_FILE \<br></div><div>> M>&n=
bsp; + CC COMPILER_TYPE COMPILER_FEATURES COMPILER_VERSION \<br></div><d=
iv>> M> + COMPILER_FREEBSD_VERSION \<br></div><div>> M>=
; + XCC X_COMPILER_TYPE X_COMPILER_FEATURES X_COMPILER_VERSION \<b=
r></div><div>> M> + X_COMPILER_FREEBSD_VERSION<br></div><div=
>> M> +TEST_SYSTEM_LINKER_VARS=3D \<br></div><div>> M>=
+ USING_SYSTEM_LINKER MK_SYSTEM_LINKER \<br></div><div>> M>=
+ MK_LLD_BOOTSTRAP \<br></div><div>> M> + WANT_LINKER=
_TYPE WANT_LINKER_VERSION WANT_LINKER_VERSION_FILE \<br></div><div>> =
M> + WANT_LINKER_FREEBSD_VERSION WANT_LINKER_FREEBSD_VERSION_FI=
LE \<br></div><div>> M> + LD LINKER_TYPE LINKER_FEATURES LIN=
KER_VERSION \<br></div><div>> M> + LINKER_FREEBSD_VERSION \<=
br></div><div>> M> + XLD X_LINKER_TYPE X_LINKER_FEATURES X_L=
INKER_VERSION \<br></div><div>> M> + X_LINKER_FREEBSD_VERSIO=
N<br></div><div>> M> +<br></div><div>> M> +.for =
_t in compiler linker<br></div><div>> M> +test-system-${_t}:=
.PHONY<br></div><div>> M> +.for v in ${TEST_SYSTEM_${_t:tu}=
_VARS}<br></div><div>> M> + ${_+_}@printf "%-35s=3D %s\n" "$=
{v}" "${${v}}"<br></div><div>> M> +.endfor<br></div><div>>=
; M> +.endfor<br></div><div>> M> +.if (make(buildwo=
rld) || make(buildkernel) || make(kernel-toolchain) || \<br></div><div>&=
gt; M> + make(toolchain) || make(_cross-tools=
))<br></div><div>> M> +.if ${USING_SYSTEM_COMPILER} =3D=3D "=
yes"<br></div><div>> M> +.info SYSTEM_COMPILER: Determined t=
hat CC=3D${CC} matches the source tree. Not bootstrapping a cross-=
compiler.<br></div><div>> M> +.elif ${MK_CLANG_BOOTSTRAP} =3D=
=3D "yes"<br></div><div>> M> +.info SYSTEM_COMPILER: libclan=
g will be built for bootstrapping a cross-compiler.<br></div><div>> M=
> +.endif<br></div><div>> M> +.if ${USING_SYSTEM_LI=
NKER} =3D=3D "yes"<br></div><div>> M> +.info SYSTEM_LINKER: =
Determined that LD=3D${LD} matches the source tree. Not bootstrapp=
ing a cross-linker.<br></div><div>> M> +.elif ${MK_LLD_BOOTS=
TRAP} =3D=3D "yes"<br></div><div>> M> +.info SYSTEM_LINKER: =
libclang will be built for bootstrapping a cross-linker.<br></div><div>&=
gt; M> +.endif<br></div><div>> M> +.endif<br></div>=
<div>> M> +<br></div><div>> M> +# End WITH_SYSTE=
M_COMPILER / WITH_SYSTEM_LD<br></div><div>> M> +<br></div><d=
iv>> M> +# Store some compiler metadata for use in installwo=
rld where we don't<br></div><div>> M> +# want to invoke CC a=
t all.<br></div><div>> M> +_TOOLCHAIN_METADATA_VARS=3D COMPI=
LER_VERSION \<br></div><div>> M> + COMPILER_TYPE \<br></d=
iv><div>> M> + COMPILER_FEATURES \<br></div><div>> M&g=
t; + COMPILER_FREEBSD_VERSION \<br></div><div>> M> =
+ COMPILER_RESOURCE_DIR \<br></div><div>> M> + LINKER_=
VERSION \<br></div><div>> M> + LINKER_FEATURES \<br></div=
><div>> M> + LINKER_TYPE \<br></div><div>> M> =
+ LINKER_FREEBSD_VERSION<br></div><div>> M> +toolchain-m=
etadata.mk: .PHONY .META<br></div><div>> M> + @: > ${.TAR=
GET}<br></div><div>> M> + @echo ".info Using cached toolchai=
n metadata from build at $$(hostname) on $$(date)" \<br></div><div>> =
M> + > ${.TARGET}<br></div><div>> M>=
; + @echo "_LOADED_TOOLCHAIN_METADATA=3Dt" >> ${.TARGET}<br>=
</div><div>> M> +.for v in ${_TOOLCHAIN_METADATA_VARS}<br></=
div><div>> M> + @echo "${v}=3D${${v}}" >> ${.TARGET}<b=
r></div><div>> M> + @echo "X_${v}=3D${X_${v}}" >> ${.T=
ARGET}<br></div><div>> M> +.endfor<br></div><div>> M>&=
nbsp; + @echo ".export ${_TOOLCHAIN_METADATA_VARS}" >> ${.TARGET}<=
br></div><div>> M> + @echo ".export ${_TOOLCHAIN_METADATA_VA=
RS:C,^,X_,}" >> ${.TARGET}<br></div><div>> M> +<br></d=
iv><div>> M> +<br></div><div>> M> +# We must do =
lib/ and libexec/ before bin/ in case of a mid-install error to<br></div=
><div>> M> +# keep the users system reasonably usable. =
For static->dynamic root upgrades,<br></div><div>> M> +#=
we don't want to install a dynamic binary without rtld and the needed<b=
r></div><div>> M> +# libraries. More commonly, for dyn=
amic root, we don't want to install a<br></div><div>> M> +# =
binary that requires a newer library version that hasn't been installed =
yet.<br></div><div>> M> +# This ordering is not a guarantee =
though. The only guarantee of a working<br></div><div>> M>&n=
bsp; +# system here would require fine-grained ordering of all component=
s based<br></div><div>> M> +# on their dependencies.<br></di=
v><div>> M> +.if !empty(SUBDIR_OVERRIDE)<br></div><div>> =
M> +SUBDIR=3D ${SUBDIR_OVERRIDE}<br></div><div>> M> =
+.else<br></div><div>> M> +SUBDIR=3D lib libexec<br></div><=
div>> M> +# Add LOCAL_LIB_DIRS, but only if they will not be=
picked up as a SUBDIR<br></div><div>> M> +# of a LOCAL_DIRS=
directory. This allows LOCAL_DIRS=3Dfoo and<br></div><div>> M&=
gt; +# LOCAL_LIB_DIRS=3Dfoo/lib to behave as expected.<br></div><d=
iv>> M> +.for _DIR in ${LOCAL_DIRS:M*/} ${LOCAL_DIRS:N*/:S|$=
|/|}<br></div><div>> M> +_REDUNDANT_LIB_DIRS+=3D =
${LOCAL_LIB_DIRS:M${_DIR}*}<br></div><div>> M> +.endf=
or<br></div><div>> M> +.for _DIR in ${LOCAL_LIB_DIRS}<br></d=
iv><div>> M> +.if ${_DIR} =3D=3D ".WAIT" || (empty(_REDUNDAN=
T_LIB_DIRS:M${_DIR}) && exists(${.CURDIR}/${_DIR}/Makefile))<br>=
</div><div>> M> +SUBDIR+=3D ${_DIR}<br></div><div>> M>=
+.endif<br></div><div>> M> +.endfor<br></div><div>>=
; M> +.if !defined(NO_ROOT) && (make(installworld) || m=
ake(install))<br></div><div>> M> +# Ensure libraries are ins=
talled before progressing.<br></div><div>> M> +SUBDIR+=3D.WA=
IT<br></div><div>> M> +.endif<br></div><div>> M> =
+SUBDIR+=3Dbin<br></div><div>> M> +.if ${MK_CDDL} !=3D "no"=
<br></div><div>> M> +SUBDIR+=3Dcddl<br></div><div>> M>=
+.endif<br></div><div>> M> +SUBDIR+=3Dgnu include<br>=
</div><div>> M> +.if ${MK_KERBEROS} !=3D "no"<br></div><div>=
> M> +SUBDIR+=3Dkerberos5<br></div><div>> M> +.e=
ndif<br></div><div>> M> +.if ${MK_RESCUE} !=3D "no"<br></div=
><div>> M> +SUBDIR+=3Drescue<br></div><div>> M> =
+.endif<br></div><div>> M> +SUBDIR+=3Dsbin<br></div><div>>=
; M> +.if ${MK_CRYPT} !=3D "no"<br></div><div>> M> =
+SUBDIR+=3Dsecure<br></div><div>> M> +.endif<br></div><div>&=
gt; M> +.if !defined(NO_SHARE)<br></div><div>> M> +=
SUBDIR+=3Dshare<br></div><div>> M> +.endif<br></div><div>>=
; M> +.if ${MK_BOOT} !=3D "no"<br></div><div>> M> +=
SUBDIR+=3Dstand<br></div><div>> M> +.endif<br></div><div>>=
; M> +SUBDIR+=3Dsys usr.bin usr.sbin<br></div><div>> M>&n=
bsp; +.if ${MK_TESTS} !=3D "no"<br></div><div>> M> +SUBDIR+=3D=
tests<br></div><div>> M> +.endif<br></div><div>> M>&n=
bsp; +<br></div><div>> M> +# Local directories are built in =
parallel with the base system directories.<br></div><div>> M> =
; +# Users may insert a .WAIT directive at the beginning or elsewhere wi=
thin<br></div><div>> M> +# the LOCAL_DIRS and LOCAL_LIB_DIRS=
lists as needed.<br></div><div>> M> +.for _DIR in ${LOCAL_D=
IRS}<br></div><div>> M> +.if ${_DIR} =3D=3D ".WAIT" || exist=
s(${.CURDIR}/${_DIR}/Makefile)<br></div><div>> M> +SUBDIR+=3D=
${_DIR}<br></div><div>> M> +.endif<br></div><div>> M>=
+.endfor<br></div><div>> M> +<br></div><div>> M>=
; +# We must do etc/ last as it hooks into building the man whatis=
file<br></div><div>> M> +# by calling 'makedb' in share/man=
. This is only relevant for<br></div><div>> M> +# inst=
all/distribute so they build the whatis file after every manpage is<br><=
/div><div>> M> +# installed.<br></div><div>> M> =
+.if make(installworld) || make(install)<br></div><div>> M> =
+SUBDIR+=3D.WAIT<br></div><div>> M> +.endif<br></div><div>&g=
t; M> +SUBDIR+=3Detc<br></div><div>> M> +<br></div>=
<div>> M> +.endif # !empty(SUBDIR_OVERRIDE)<br></div><div>&g=
t; M> +<br></div><div>> M> +.if defined(NOCLEAN)<br=
></div><div>> M> +.warning The src.conf WITHOUT_CLEAN option=
can now be used instead of NOCLEAN.<br></div><div>> M> +MK_=
CLEAN:=3D no<br></div><div>> M> +.endif<br></div><div>> M=
> +.if defined(NO_CLEAN)<br></div><div>> M> +.info =
The src.conf WITHOUT_CLEAN option can now be used instead of NO_CLEAN.<b=
r></div><div>> M> +MK_CLEAN:=3D no<br></div><div>> M>&=
nbsp; +.endif<br></div><div>> M> +.if defined(NO_CLEANDIR)<b=
r></div><div>> M> +CLEANDIR=3D clean cleandepend<br></div><d=
iv>> M> +.else<br></div><div>> M> +CLEANDIR=3D c=
leandir<br></div><div>> M> +.endif<br></div><div>> M>&=
nbsp; +<br></div><div>> M> +.if defined(WORLDFAST)<br></div>=
<div>> M> +MK_CLEAN:=3D no<br></div><div>> M> +N=
O_OBJWALK=3D t<br></div><div>> M> +.endif<br></div><div>>=
M> +<br></div><div>> M> +.if ${MK_META_MODE} =3D=3D=
"yes"<br></div><div>> M> +# If filemon is used then we can =
rely on the build being incremental-safe.<br></div><div>> M> =
+# The .meta files will also track the build command and rebuild should=
<br></div><div>> M> +# it change.<br></div><div>> M>&n=
bsp; +.if empty(.MAKE.MODE:Mnofilemon)<br></div><div>> M> +M=
K_CLEAN:=3D no<br></div><div>> M> +.endif<br></div><div>>=
M> +.endif<br></div><div>> M> +.if defined(NO_OBJW=
ALK) || ${MK_AUTO_OBJ} =3D=3D "yes"<br></div><div>> M> +NO_O=
BJWALK=3D t<br></div><div>> M> +NO_KERNELOBJ=3D t<br></div><=
div>> M> +.endif<br></div><div>> M> +.if !define=
d(NO_OBJWALK)<br></div><div>> M> +_obj=3D obj<br></div><div=
>> M> +.endif<br></div><div>> M> +<br></div><div=
>> M> +LOCAL_TOOL_DIRS?=3D<br></div><div>> M> +P=
ACKAGEDIR?=3D ${DESTDIR}/${DISTDIR}<br></div><div>> M> +<br>=
</div><div>> M> +.if empty(SHELL:M*csh*)<br></div><div>> =
M> +BUILDENV_SHELL?=3D${SHELL}<br></div><div>> M> +=
.else<br></div><div>> M> +BUILDENV_SHELL?=3D/bin/sh<br></div=
><div>> M> +.endif<br></div><div>> M> +<br></div=
><div>> M> +.if !defined(_MKSHOWCONFIG)<br></div><div>> M=
> +.if !defined(VCS_REVISION) || empty(VCS_REVISION)<br></div><=
div>> M> +.if !defined(SVNVERSION_CMD) || empty(SVNVERSION_C=
MD)<br></div><div>> M> +. for _D in ${PATH:S,:, ,g}<br></div=
><div>> M> +. if exists(${_D}/svnversion)<br></div><di=
v>> M> +SVNVERSION_CMD?=3D${_D}/svnversion<br></div><div>>=
; M> +. endif<br></div><div>> M> +. if =
exists(${_D}/svnliteversion)<br></div><div>> M> +SVNVERSION_=
CMD?=3D${_D}/svnliteversion<br></div><div>> M> +. endi=
f<br></div><div>> M> +. endfor<br></div><div>> M> =
; +.endif<br></div><div>> M> +.if defined(SVNVERSION_CMD) &a=
mp;& !empty(SVNVERSION_CMD)<br></div><div>> M> +_VCS_REV=
ISION?=3D $$(eval ${SVNVERSION_CMD} ${SRCDIR})<br></div><div>> M>&=
nbsp; +. if !empty(_VCS_REVISION)<br></div><div>> M> +VCS_RE=
VISION=3D $$(echo r${_VCS_REVISION})<br></div><div>> M> +.ex=
port VCS_REVISION<br></div><div>> M> +. endif<br></div><div>=
> M> +.endif<br></div><div>> M> +.endif<br></div=
><div>> M> +<br></div><div>> M> +.if !defined(GI=
T_CMD) || empty(GIT_CMD)<br></div><div>> M> +. for _P in /us=
r/bin /usr/local/bin<br></div><div>> M> +. if exists($=
{_P}/git)<br></div><div>> M> +GIT_CMD=3D ${_P}/g=
it<br></div><div>> M> +. endif<br></div><div>> M>=
; +. endfor<br></div><div>> M> +.export GIT_CMD<br></d=
iv><div>> M> +.endif<br></div><div>> M> +<br></d=
iv><div>> M> +.if !defined(OSRELDATE)<br></div><div>> M&g=
t; +.if exists(/usr/include/osreldate.h)<br></div><div>> M>&=
nbsp; +OSRELDATE!=3D awk '/^\#define[[:space:]]*__FreeBSD_version/ { pri=
nt $$3 }' \<br></div><div>> M> + /usr/include/osreldate.h<b=
r></div><div>> M> +.else<br></div><div>> M> +OSR=
ELDATE=3D 0<br></div><div>> M> +.endif<br></div><div>> M&=
gt; +.export OSRELDATE<br></div><div>> M> +.endif<br><=
/div><div>> M> +<br></div><div>> M> +# Set VERSI=
ON for CTFMERGE to use via the default CTFFLAGS=3D-L VERSION.<br></div><=
div>> M> +.for _V in BRANCH REVISION TYPE<br></div><div>>=
M> +.if !defined(_${_V})<br></div><div>> M> +_${_V=
}!=3D eval $$(awk '/^${_V}=3D/{print}' ${SRCTOP}/sys/conf/newvers.sh); e=
cho $$${_V}<br></div><div>> M> +.export _${_V}<br></div><div=
>> M> +.endif<br></div><div>> M> +.endfor<br></d=
iv><div>> M> +.if !defined(SRCRELDATE)<br></div><div>> M&=
gt; +SRCRELDATE!=3D awk '/^\#define[[:space:]]*__FreeBSD_version/ =
{ print $$3 }' \<br></div><div>> M> + ${SRCDIR}/sys/sys/par=
am.h<br></div><div>> M> +.export SRCRELDATE<br></div><div>&g=
t; M> +.endif<br></div><div>> M> +.if !defined(VERS=
ION)<br></div><div>> M> +VERSION=3D FreeBSD ${_REVISION}-${_=
BRANCH:C/-p[0-9]+$//} ${TARGET_ARCH} ${SRCRELDATE}<br></div><div>> M&=
gt; +.export VERSION<br></div><div>> M> +.endif<br></d=
iv><div>> M> +MAJOR_REVISION=3D ${_REVISION:R}<br></div><div=
>> M> +<br></div><div>> M> +.if !defined(PKG_VER=
SION)<br></div><div>> M> +_STRTIMENOW=3D %Y%m%d%H%M%S<br></d=
iv><div>> M> +_TIMENOW=3D ${_STRTIMENOW:gmtime}<br></div><di=
v>> M> +.if ${_BRANCH:MCURRENT*} || ${_BRANCH:MSTABLE*} || $=
{_BRANCH:MPRERELEASE*}<br></div><div>> M> +EXTRA_REVISION=3D=
.snap${_TIMENOW}<br></div><div>> M> +.elif ${_BRANCH:MALPHA=
*}<br></div><div>> M> +EXTRA_REVISION=3D .a${_BRANCH:C/ALPHA=
([0-9]+).*/\1/}.${_TIMENOW}<br></div><div>> M> +.elif ${_BRA=
NCH:MBETA*}<br></div><div>> M> +EXTRA_REVISION=3D .b${_BRANC=
H:C/BETA([0-9]+).*/\1/}.${_TIMENOW}<br></div><div>> M> +.eli=
f ${_BRANCH:MRC*}<br></div><div>> M> +EXTRA_REVISION=3D .rc$=
{_BRANCH:C/RC([0-9]+).*/\1/}.${_TIMENOW}<br></div><div>> M> =
+.elif ${_BRANCH:M*-p*}<br></div><div>> M> +EXTRA_REVISION=3D=
p${_BRANCH:C/.*-p([0-9]+$)/\1/}<br></div><div>> M> +.endif<=
br></div><div>> M> +PKG_VERSION:=3D ${MAJOR_REVISION}${EXTRA=
_REVISION:C/[[:space:]]//g}<br></div><div>> M> +.endif<br></=
div><div>> M> +.endif # !defined(PKG_VERSION)<br></div><div>=
> M> +<br></div><div>> M> +.if !defined(PKG_TIME=
STAMP)<br></div><div>> M> +TIMEEPOCHNOW=3D %s<br></div><div=
>> M> +SOURCE_DATE_EPOCH=3D ${TIMEEPOCHNOW:gmtime}<br></div>=
<div>> M> +.else<br></div><div>> M> +SOURCE_DATE=
_EPOCH=3D ${PKG_TIMESTAMP}<br></div><div>> M> +.endif<br></d=
iv><div>> M> +<br></div><div>> M> +PKG_NAME_PREF=
IX?=3D FreeBSD<br></div><div>> M> +PKG_MAINTAINER?=3D <a hre=
f=3D"mailto:re@FreeBSD.org">re@FreeBSD.org</a><br></div><div>> M>&=
nbsp; +PKG_WWW?=3D <a href=3D"https://www.FreeBSD.org">https://www.FreeB=
SD.org</a><br></div><div>> M> +.export PKG_NAME_PREFIX<br></=
div><div>> M> +.export PKG_MAINTAINER<br></div><div>> M&g=
t; +.export PKG_WWW<br></div><div>> M> +<br></div><div=
>> M> +.if !defined(_MKSHOWCONFIG)<br></div><div>> M>&=
nbsp; +_CPUTYPE!=3D MAKEFLAGS=3D CPUTYPE=3D${_TARGET_CPUTYPE} ${MAKE} -f=
/dev/null \<br></div><div>> M> + -m ${.CURDIR}/share/mk MK=
_AUTO_OBJ=3Dno -V CPUTYPE<br></div><div>> M> +.if ${_CPUTYPE=
} !=3D ${_TARGET_CPUTYPE}<br></div><div>> M> +.error CPUTYPE=
global should be set with ?=3D.<br></div><div>> M> +.endif<=
br></div><div>> M> +.endif<br></div><div>> M> +.=
if make(buildworld)<br></div><div>> M> +BUILD_ARCH!=3D uname=
-p<br></div><div>> M> +# On some Linux systems uname -p ret=
urns "unknown" so skip this check there.<br></div><div>> M> =
+# This check only exists to tell people to use TARGET_ARCH instead of<b=
r></div><div>> M> +# MACHINE_ARCH so skipping it when crossb=
uilding on non-FreeBSD should be fine.<br></div><div>> M> +.=
if ${MACHINE_ARCH} !=3D ${BUILD_ARCH} && ${.MAKE.OS} =3D=3D "Fre=
eBSD"<br></div><div>> M> +.error To cross-build, set TARGET_=
ARCH.<br></div><div>> M> +.endif<br></div><div>> M>&nb=
sp; +.endif<br></div><div>> M> +WORLDTMP?=3D ${OBJTOP}/tmp<b=
r></div><div>> M> +BPATH=3D ${CCACHE_WRAPPER_PATH_PFX}${WOR=
LDTMP}/legacy/usr/sbin:${WORLDTMP}/legacy/usr/bin:${WORLDTMP}/legacy/bin=
:${WORLDTMP}/legacy/usr/libexec<br></div><div>> M> +XPATH=3D=
${WORLDTMP}/bin:${WORLDTMP}/usr/sbin:${WORLDTMP}/usr/bin<br></div><div=
>> M> +<br></div><div>> M> +# When building we w=
ant to find the cross tools before the host tools in ${BPATH}.<br></div>=
<div>> M> +# We also need to add UNIVERSE_TOOLCHAIN_PATH so =
that we can find the shared<br></div><div>> M> +# toolchain =
files (clang, lld, etc.) during make universe/tinderbox<br></div><div>&g=
t; M> +STRICTTMPPATH=3D ${XPATH}:${BPATH}:${UNIVERSE_TOOLCHAIN_=
PATH}<br></div><div>> M> +# We should not be using tools fro=
m /usr/bin accidentally since this could cause<br></div><div>> M>&=
nbsp; +# the build to break on other systems that don't have that tool. =
For now we<br></div><div>> M> +# still allow using the old b=
ehaviour (inheriting $PATH) if<br></div><div>> M> +# BUILD_W=
ITH_STRICT_TMPPATH is set to 0 but this will eventually be removed.<br><=
/div><div>> M> +<br></div><div>> M> +# Currently=
strict $PATH can cause build failures. Once the remaining issues<br></d=
iv><div>> M> +# have been resolved it will be turned on by d=
efault.<br></div><div>> M> +BUILD_WITH_STRICT_TMPPATH?=3D0<b=
r></div><div>> M> +.if defined(CROSSBUILD_HOST)<br></div><di=
v>> M> +# When building on non-FreeBSD we can't rely on the =
tools in /usr/bin being compatible<br></div><div>> M> +# wit=
h what FreeBSD expects. Therefore we only use tools from STRICTTMPPATH<b=
r></div><div>> M> +# during the world build stage. We build =
most tools during the bootstrap-tools<br></div><div>> M> +# =
phase but symlink host tools that are known to work instead of building =
them<br></div><div>> M> +BUILD_WITH_STRICT_TMPPATH:=3D1<br><=
/div><div>> M> +.endif<br></div><div>> M> +.if $=
{BUILD_WITH_STRICT_TMPPATH} !=3D 0<br></div><div>> M> +TMPPA=
TH=3D ${STRICTTMPPATH}<br></div><div>> M> +.else<br></div><d=
iv>> M> +TMPPATH=3D ${STRICTTMPPATH}:${PATH}<br></div><div>&=
gt; M> +.endif<br></div><div>> M> +<br></div><div>&=
gt; M> +#<br></div><div>> M> +# Avoid running mktem=
p(1) unless actually needed.<br></div><div>> M> +# It may no=
t be functional, e.g., due to new ABI<br></div><div>> M> +# =
when in the middle of installing over this system.<br></div><div>> M&=
gt; +#<br></div><div>> M> +.if make(distributeworld) |=
| make(installworld) || make(stageworld)<br></div><div>> M> =
+.if ${BUILD_WITH_STRICT_TMPPATH} !=3D 0<br></div><div>> M> =
+MKTEMP=3D${WORLDTMP}/legacy/usr/bin/mktemp<br></div><div>> M>&nbs=
p; +.if !exists(${MKTEMP})<br></div><div>> M> +.error mktemp=
binary doesn't exist in expected location: ${MKTEMP}<br></div><div>>=
M> +.endif<br></div><div>> M> +.else<br></div><div=
>> M> +MKTEMP=3Dmktemp<br></div><div>> M> +.endi=
f<br></div><div>> M> +INSTALLTMP!=3D ${MKTEMP} -d -u -t inst=
all<br></div><div>> M> +<br></div><div>> M> +.if=
${.MAKE.OS} =3D=3D "FreeBSD"<br></div><div>> M> +# When bui=
lding on FreeBSD we always copy the host tools instead of linking<br></d=
iv><div>> M> +# into INSTALLTMP to avoid issues with incompa=
tible libraries (see r364030).<br></div><div>> M> +# Note: w=
e could create links if we don't intend to update the current machine.<b=
r></div><div>> M> +INSTALLTMP_COPY_HOST_TOOL=3Dcp<br></div><=
div>> M> +.else<br></div><div>> M> +# However, t=
his is not necessary on Linux/macOS. Additionally, copying the host<br><=
/div><div>> M> +# tools to another directory with cp results=
in AMFI Launch Constraint<br></div><div>> M> +# Violations =
on macOS Ventura as part of its System Integrity Protection.<br></div><d=
iv>> M> +INSTALLTMP_COPY_HOST_TOOL=3Dln -s<br></div><div>>=
; M> +.endif<br></div><div>> M> +.endif<br></div><d=
iv>> M> +<br></div><div>> M> +.if make(stagekern=
el) || make(distributekernel)<br></div><div>> M> +TAGS+=3D =
kernel<br></div><div>> M> +PACKAGE=3D kernel<br></div><div>&=
gt; M> +.endif<br></div><div>> M> +<br></div><div>&=
gt; M> +#<br></div><div>> M> +# Building a world go=
es through the following stages<br></div><div>> M> +#<br></d=
iv><div>> M> +# 1. legacy stage [BMAKE]<br></div><div>> M=
> +# This stage is responsible for creating compatibility<br></=
div><div>> M> +# shims that are needed by the bootstrap-tool=
s,<br></div><div>> M> +# build-tools and cross-tools stages.=
These are generally<br></div><div>> M> +# APIs that tools f=
rom one of those three stages need to<br></div><div>> M> +# =
build that aren't present on the host.<br></div><div>> M> +#=
1. bootstrap-tools stage [BMAKE]<br></div><div>> M> +# This=
stage is responsible for creating programs that<br></div><div>> M>=
; +# are needed for backward compatibility reasons. They<br></div>=
<div>> M> +# are not built as cross-tools.<br></div><div>>=
; M> +# 2. build-tools stage [TMAKE]<br></div><div>> M>&n=
bsp; +# This stage is responsible for creating the object<br></div><div>=
> M> +# tree and building any tools that are needed during<b=
r></div><div>> M> +# the build process. Some programs are li=
sted during<br></div><div>> M> +# this phase because they bu=
ild binaries to generate<br></div><div>> M> +# files needed =
to build these programs. This stage also<br></div><div>> M> =
+# builds the 'build-tools' target rather than 'all'.<br></div><div>>=
M> +# 3. cross-tools stage [XMAKE]<br></div><div>> M>&nb=
sp; +# This stage is responsible for creating any tools that<br></div><d=
iv>> M> +# are needed for building the system. A cross-compi=
ler is one<br></div><div>> M> +# of them. This differs from =
build tools in two ways:<br></div><div>> M> +# 1. the 'all' =
target is built rather than 'build-tools'<br></div><div>> M> =
+# 2. these tools are installed into TMPPATH for stage 4.<br></div><div=
>> M> +# 4. world stage [WMAKE]<br></div><div>> M>&nbs=
p; +# This stage actually builds the world.<br></div><div>> M>&nbs=
p; +# 5. install stage (optional) [IMAKE]<br></div><div>> M> =
+# This stage installs a previously built world.<br></div><div>> M&g=
t; +#<br></div><div>> M> +<br></div><div>> M>&nb=
sp; +BOOTSTRAPPING?=3D 0<br></div><div>> M> +# Keep these in=
sync<br></div><div>> M> +MINIMUM_SUPPORTED_OSREL?=3D 110400=
1<br></div><div>> M> +MINIMUM_SUPPORTED_REL?=3D 11.4<br></di=
v><div>> M> +<br></div><div>> M> +# Common envir=
onment for world related stages<br></div><div>> M> +CROSSENV=
+=3D \<br></div><div>> M> + MACHINE_ARCH=3D${TARGET_ARCH} \=
<br></div><div>> M> + MACHINE=3D${TARGET} \<br></div><div>&=
gt; M> + CPUTYPE=3D${TARGET_CPUTYPE}<br></div><div>> M>&=
nbsp; +.if ${MK_META_MODE} !=3D "no"<br></div><div>> M> +# D=
on't rebuild build-tools targets during normal build.<br></div><div>>=
M> +CROSSENV+=3D BUILD_TOOLS_META=3D.NOMETA<br></div><div>>=
M> +.endif<br></div><div>> M> +.if defined(TARGET_=
CFLAGS)<br></div><div>> M> +CROSSENV+=3D ${TARGET_CFLAGS}<br=
></div><div>> M> +.endif<br></div><div>> M> +.if=
(${TARGET} !=3D ${MACHINE} && !defined(WITH_LOCAL_MODULES)) || =
\<br></div><div>> M> + defined(WITHOUT_LOC=
AL_MODULES)<br></div><div>> M> +CROSSENV+=3D LOCAL_MODULES=3D=
<br></div><div>> M> +.endif<br></div><div>> M> +=
<br></div><div>> M> +BOOTSTRAPPING_OSRELDATE?=3D${OSRELDATE}=
<br></div><div>> M> +<br></div><div>> M> +# boot=
strap-tools stage<br></div><div>> M> +BMAKEENV=3D INSTALL=3D=
"sh ${.CURDIR}/tools/install.sh" \<br></div><div>> M> + TOO=
LS_PREFIX=3D${TOOLS_PREFIX_UNDEF:U${WORLDTMP}} \<br></div><div>> M>=
; + PATH=3D${BPATH:Q}:${PATH:Q} \<br></div><div>> M> =
+ WORLDTMP=3D${WORLDTMP} \<br></div><div>> M> + MAKEFLAGS=3D=
"-m ${.CURDIR}/tools/build/mk ${.MAKEFLAGS}"<br></div><div>> M>&nb=
sp; +# need to keep this in sync with targets/pseudo/bootstrap-tools/Mak=
efile<br></div><div>> M> +BSARGS=3D DESTDIR=3D \<br></div><=
div>> M> + OBJTOP=3D'${WORLDTMP}/obj-tools' \<br></div><div=
>> M> + OBJROOT=3D'$${OBJTOP}/' \<br></div><div>> M>&=
nbsp; + UNIVERSE_TOOLCHAIN_PATH=3D${UNIVERSE_TOOLCHAIN_PATH} \<br></div=
><div>> M> + MAKEOBJDIRPREFIX=3D \<br></div><div>> M>=
+ BOOTSTRAPPING=3D${BOOTSTRAPPING_OSRELDATE} \<br></div><div>>=
; M> + BWPHASE=3D${.TARGET:C,^_,,} \<br></div><div>> M>&=
nbsp; + -DNO_CPU_CFLAGS \<br></div><div>> M> + -DNO_LINT \=
<br></div><div>> M> + -DNO_PIC \<br></div><div>> M>&n=
bsp; + -DNO_SHARED \<br></div><div>> M> + MK_ASAN=3Dno \<b=
r></div><div>> M> + MK_CTF=3Dno \<br></div><div>> M>&=
nbsp; + MK_CLANG_EXTRAS=3Dno \<br></div><div>> M> + MK_CLA=
NG_FORMAT=3Dno \<br></div><div>> M> + MK_CLANG_FULL=3Dno \<=
br></div><div>> M> + MK_HTML=3Dno \<br></div><div>> M>=
; + MK_MAN=3Dno \<br></div><div>> M> + MK_PROFILE=3D=
no \<br></div><div>> M> + MK_RETPOLINE=3Dno \<br></div><div=
>> M> + MK_SSP=3Dno \<br></div><div>> M> + MK_=
TESTS=3Dno \<br></div><div>> M> + MK_UBSAN=3Dno \<br></div>=
<div>> M> + MK_WERROR=3Dno \<br></div><div>> M> =
+ MK_INCLUDES=3Dyes \<br></div><div>> M> + MK_MAN_UTILS=3D=
yes<br></div><div>> M> +<br></div><div>> M> +BMA=
KE=3D \<br></div><div>> M> + ${TIME_ENV} ${BMAKEENV} ${MAK=
E} ${WORLD_FLAGS} -f Makefile.inc1 \<br></div><div>> M> + $=
{BSARGS}<br></div><div>> M> +.if empty(.MAKEOVERRIDES:MMK_LL=
VM_TARGET_ALL)<br></div><div>> M> +BMAKE+=3D MK_LLVM_TARGET=
_ALL=3Dno<br></div><div>> M> +.endif<br></div><div>> M>=
; +<br></div><div>> M> +# build-tools stage<br></div><=
div>> M> +TMAKE=3D \<br></div><div>> M> + ${TI=
ME_ENV} ${BMAKEENV} ${MAKE} ${WORLD_FLAGS} -f Makefile.inc1 \<br></div><=
div>> M> + TARGET=3D${TARGET} TARGET_ARCH=3D${TARGET_ARCH} =
\<br></div><div>> M> + DESTDIR=3D \<br></div><div>> M>=
; + BOOTSTRAPPING=3D${BOOTSTRAPPING_OSRELDATE} \<br></div><div>&g=
t; M> + BWPHASE=3D${.TARGET:C,^_,,} \<br></div><div>> M>=
+ -DNO_CPU_CFLAGS \<br></div><div>> M> + -DNO_LINT =
\<br></div><div>> M> + MK_ASAN=3Dno \<br></div><div>> M&=
gt; + MK_CTF=3Dno \<br></div><div>> M> + MK_CLANG_EX=
TRAS=3Dno \<br></div><div>> M> + MK_CLANG_FORMAT=3Dno \<br>=
</div><div>> M> + MK_CLANG_FULL=3Dno \<br></div><div>> M=
> + MK_LLDB=3Dno \<br></div><div>> M> + MK_RETPOL=
INE=3Dno \<br></div><div>> M> + MK_SSP=3Dno \<br></div><div=
>> M> + MK_TESTS=3Dno \<br></div><div>> M> + M=
K_UBSAN=3Dno \<br></div><div>> M> + MK_WERROR=3Dno<br></div=
><div>> M> +<br></div><div>> M> +# cross-tools s=
tage<br></div><div>> M> +# TOOLS_PREFIX set in BMAKE<br></di=
v><div>> M> +XMAKE=3D ${BMAKE} \<br></div><div>> M>&n=
bsp; + TARGET=3D${TARGET} TARGET_ARCH=3D${TARGET_ARCH} \<br></div><div>=
> M> + MK_LLDB=3Dno \<br></div><div>> M> + MK_=
LLVM_BINUTILS=3Dno \<br></div><div>> M> + MK_TESTS=3Dno<br>=
</div><div>> M> +<br></div><div>> M> +# kernel-t=
ools stage<br></div><div>> M> +KTMAKEENV=3D INSTALL=3D"sh ${=
.CURDIR}/tools/install.sh" \<br></div><div>> M> + PATH=3D${=
BPATH:Q}:${PATH:Q} \<br></div><div>> M> + WORLDTMP=3D${WORL=
DTMP} \<br></div><div>> M> + MAKEFLAGS=3D"-m ${.CURDIR}/too=
ls/build/mk ${.MAKEFLAGS}"<br></div><div>> M> +<br></div><di=
v>> M> +KTMAKE=3D ${TIME_ENV} \<br></div><div>> M>&nb=
sp; + TOOLS_PREFIX=3D${TOOLS_PREFIX_UNDEF:U${WORLDTMP}} \<br></div><div=
>> M> + ${KTMAKEENV} ${MAKE} ${WORLD_FLAGS} -f Makefile.inc=
1 \<br></div><div>> M> + DESTDIR=3D \<br></div><div>> M&=
gt; + OBJTOP=3D'${WORLDTMP}/obj-kernel-tools' \<br></div><div>>=
; M> + OBJROOT=3D'$${OBJTOP}/' \<br></div><div>> M> =
; + UNIVERSE_TOOLCHAIN_PATH=3D${UNIVERSE_TOOLCHAIN_PATH} \<br></div><di=
v>> M> + MAKEOBJDIRPREFIX=3D \<br></div><div>> M>&nbs=
p; + BOOTSTRAPPING=3D${BOOTSTRAPPING_OSRELDATE} \<br></div><div>> M&=
gt; + -DNO_CPU_CFLAGS \<br></div><div>> M> + -DNO_LI=
NT \<br></div><div>> M> + -DNO_PIC \<br></div><div>> M&g=
t; + -DNO_SHARED \<br></div><div>> M> + MK_CTF=3Dno =
\<br></div><div>> M> + MK_HTML=3Dno \<br></div><div>> M&=
gt; + MK_MAN=3Dno \<br></div><div>> M> + MK_PROFILE=3D=
no \<br></div><div>> M> + MK_SSP=3Dno \<br></div><div>> =
M> + MK_RETPOLINE=3Dno \<br></div><div>> M> + MK_=
WERROR=3Dno<br></div><div>> M> +<br></div><div>> M>&nb=
sp; +# world stage<br></div><div>> M> +WMAKEENV=3D ${CROSSEN=
V} \<br></div><div>> M> + INSTALL=3D"${INSTALL_CMD} -U" \<b=
r></div><div>> M> + PATH=3D${TMPPATH:Q} \<br></div><div>>=
; M> + SYSROOT=3D${WORLDTMP}<br></div><div>> M> +<=
br></div><div>> M> +# make hierarchy<br></div><div>> M>=
; *** 5431 LINES SKIPPED ***<br></div><div>> <br></div><div><br>=
</div><div>-- <br></div><div>Alexander Motin<br></div><div><br></di=
v></blockquote><div><br></div></body></html>
--06e1cace14e44ba6b615e2ddd8eabb7c--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?07faf861-9186-47d1-992a-91d483ea4e9c>