From nobody Sat Jul 11 19:30:24 2026 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4gyJh13PhQz6lWxs for ; Sat, 11 Jul 2026 19:30:25 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR1" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4gyJh1057Fz3Zqt for ; Sat, 11 Jul 2026 19:30:25 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1783798225; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ApDORUVmWyrc/Yvzw/gZ9cSSW2pOZI/QY4eWZ8Qt4wo=; b=x/Ex5Xu6+9lvPCn3mFikVByQeCNKlwuSNcaace7RLV5r8X2lnxRp8PfuILqgeEpKD3C9e+ +6Ar04C6th9OkhkACmiPcTUp071q5epJUCj8bnkI2esBMltuynQ0UWFZ9ZGLn6v994eFy2 mXA3tW65BOFDKfmJ+oOoaxAJcrshVLxRmEiCeasOnCVJN3Z56kfnRiePAv+HVIfjxUMaOx jYgGBRFjBvbhBy8025sW5HewNXF965uAyX6YkHgg4px2NtaHZOFsdvgVM5NHOps5zb2L12 /G0K+Uq50mF2kWsszrRZU+wLoZ76wGx1tsXDyqtnNZGcq0grC2EFIiZD9042lQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1783798225; a=rsa-sha256; cv=none; b=Ar3wsPTXjvMF+FRkANdXItSKB4Zw2cDKEGR3Q3QpYJ3dxhj8tBgKeeDTWTC+DW7e1s7OSB KrMB1a6ptoAy9xlmws+LrOCWdot/AiY8/k4AnCUEIgrJSwnLA1y8Raj9Hk9/4j1THdGAIt 50Fw5EiUJheed4ddT6wLhtHJC4lrM2H20dwtfj0hzCUfJYYo94O+UOm6HdKsEmPyemlBOj rRM3nvUI8Fl+JsZqjkdAP5XxOW+x5iH+8Y4/h1wi8MzE1w5ycHU1e4YoVvdMHwgTJMwmAF 2YTlJ+wMT4OxUoAJi8P0Z5OpSbVAoLoL1wM7prScgnwJb+TibKcoO07BUFEZAQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1783798225; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ApDORUVmWyrc/Yvzw/gZ9cSSW2pOZI/QY4eWZ8Qt4wo=; b=a311gSvZBhhLh5ukMM28EMc06PS6WpJK0Li+KT3c4WeSl+415rpIDZsZEDV/m+pGAdbLbS 9B0HoPR+7xbxShqfjByAs7wZcvNNG0kwyVULIurZl+qYKomsWI728eP21Yt0jCH8XQK+xS uMrT0ZXrVkadAFKZMJ0ZTsv9WgFbYiFswwN6qvtUwSRubFZgGJ13HK4ooX4HehHHJ/o1nx OVDlzG2mAmZbUQcmQZJiAt0u3FPj/uq9otvYlleofmh6toX2cag072aJmOO2sG4BFdrGZs oYJspEbDJGHx+UohW8KWM7pAJq/7YbwHqNbM7hp2dQRqcCw6If7N0ciCJnCSKA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4gyJh05MJ3zt9t for ; Sat, 11 Jul 2026 19:30:24 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 26e5e by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Sat, 11 Jul 2026 19:30:24 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Warner Losh Subject: git: dc6c193bb4a6 - main - loader.efi: Add xz decompression List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: imp X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: dc6c193bb4a67e2e9ee97d1e3ac3e2950bd16979 Auto-Submitted: auto-generated Date: Sat, 11 Jul 2026 19:30:24 +0000 Message-Id: <6a5299d0.26e5e.70437550@gitrepo.freebsd.org> The branch main has been updated by imp: URL: https://cgit.FreeBSD.org/src/commit/?id=dc6c193bb4a67e2e9ee97d1e3ac3e2950bd16979 commit dc6c193bb4a67e2e9ee97d1e3ac3e2950bd16979 Author: Warner Losh AuthorDate: 2026-07-11 14:44:40 +0000 Commit: Warner Losh CommitDate: 2026-07-11 19:28:47 +0000 loader.efi: Add xz decompression We can now decompress .xz compressed memory disks, like FreeBSD-15.1-RELEASE-amd64-disc1.iso.xz Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D58073 --- stand/defs.mk | 5 ++ stand/efi/loader/Makefile | 2 +- stand/efi/loader/decompress.c | 101 ++++++++++++++++++++++++++-- stand/efi/loader/decompress.h | 2 +- stand/libsa/Makefile | 7 ++ sys/contrib/xz-embedded/freebsd/xz_config.h | 11 +-- 6 files changed, 117 insertions(+), 11 deletions(-) diff --git a/stand/defs.mk b/stand/defs.mk index 05108404c538..e3908678b53a 100644 --- a/stand/defs.mk +++ b/stand/defs.mk @@ -190,6 +190,11 @@ BZIP2_CFLAGS=-I${SRCTOP}/contrib/bzip2 -DBZ_NO_STDIO -DBZ_NO_COMPRESS # ZSTD client cflags ZSTD_CFLAGS=-I${SYSDIR}/contrib/zstd/lib +# XZ flags +XZ_DIR=${SRCTOP}/sys/contrib/xz-embedded +XZ_CFLAGS=-DXZ_USE_CRC64 -I${XZ_DIR}/freebsd -I${XZ_DIR}/linux/include/linux + + # The boot loader build uses dd status=none, where possible, for reproducible # build output (since performance varies from run to run). Trouble is that # option was recently (10.3) added to FreeBSD and is non-standard. Only use it diff --git a/stand/efi/loader/Makefile b/stand/efi/loader/Makefile index 164b16ff2a59..b29f0b8f8259 100644 --- a/stand/efi/loader/Makefile +++ b/stand/efi/loader/Makefile @@ -47,7 +47,7 @@ HAVE_ZFS= yes CFLAGS.bootinfo.c += -I$(SRCTOP)/sys/teken CFLAGS.bootinfo.c += -I${SRCTOP}/contrib/pnglite -CFLAGS.decompress.c += ${ZLIB_CFLAGS} ${BZIP2_CFLAGS} ${ZSTD_CFLAGS} +CFLAGS.decompress.c += ${ZLIB_CFLAGS} ${BZIP2_CFLAGS} ${ZSTD_CFLAGS} ${XZ_CFLAGS} CFLAGS.framebuffer.c += -I$(SRCTOP)/sys/teken CFLAGS.framebuffer.c += -I${SRCTOP}/contrib/pnglite CFLAGS.main.c += -I$(SRCTOP)/sys/teken diff --git a/stand/efi/loader/decompress.c b/stand/efi/loader/decompress.c index 5c16f7d9bb2a..fb3c6314e4b6 100644 --- a/stand/efi/loader/decompress.c +++ b/stand/efi/loader/decompress.c @@ -11,7 +11,8 @@ #include #include -#ifdef LOADER_ZFS_SUPPORT /* ZSTD only available with ZFS */ +#include +#ifdef LOADER_ZFS_SUPPORT /* ZSTD and lzma only available with ZFS */ #include #endif #include @@ -32,6 +33,7 @@ struct decomp_state union { z_stream zstrm; bz_stream bzstrm; + struct xz_dec *xzstrm; #ifdef LOADER_ZFS_SUPPORT ZSTD_DStream *zstdstrm; #endif @@ -63,10 +65,10 @@ what_compressed(uint8_t *buf, size_t len) return (zstd); } if (memcmp(buf, "\xfd""7zXZ\x00", 6) == 0) { - printf("xz -- unsupproted\n"); - } else { - printf("Not compressed\n"); + printf("xz\n"); + return (xz); } + printf("Not compressed\n"); return (none); } @@ -220,7 +222,7 @@ bzip2_step(decomp_state *dctx, uint8_t *buf, size_t len, size_t offset) if (ret == BZ_STREAM_END) return (done); - if (ret != Z_OK) + if (ret != BZ_OK) return (err); if (dctx->buf_cur < dctx->buf_end) /* Have output space */ return (ok); @@ -247,6 +249,90 @@ bzip2_fini(decomp_state *dctx, bool flush) free_buffer(dctx); } +/* + * XZ support + */ +static EFI_STATUS +xz_init(decomp_state *dctx, uint8_t *first_buf, size_t buflen, size_t size_hint) +{ + /* + * Assume 4x compression, but start at 64MB + */ + dctx->size = max(size_hint * 4, M(64)); + EFI_STATUS status = alloc_buffer(dctx, dctx->size); + if (EFI_ERROR(status)) + return (status); + xz_crc32_init(); + xz_crc64_init(); + dctx->xzstrm = xz_dec_init(XZ_DYNALLOC, (uint32_t)-1); + return (dctx->xzstrm != NULL ? EFI_SUCCESS : EFI_VOLUME_CORRUPTED); +} + + +static enum step_return +xz_step(decomp_state *dctx, uint8_t *buf, size_t len, size_t offset) +{ + struct xz_dec *strm = dctx->xzstrm; + size_t outlen = dctx->buf_end - dctx->buf_cur; + struct xz_buf b = { .in = buf, .in_size = len, .in_pos = 0, + .out = dctx->buf_cur, .out_size = outlen, .out_pos = 0 }; + int ret; + + ret = xz_dec_run(strm, &b); + dctx->buf_cur += b.out_pos; + + if (ret == XZ_STREAM_END) + return (done); + if (ret != XZ_OK) { + switch(ret) { + case XZ_MEM_ERROR: + printf("xz no memory "); + break; + case XZ_DATA_ERROR: + printf("xz file corrupted "); + break; + case XZ_FORMAT_ERROR: + printf("xz format not found "); + break; + case XZ_OPTIONS_ERROR: + printf("unsupported xz option "); + break; + case XZ_MEMLIMIT_ERROR: + printf("xz dictionary too small "); + break; + default: + printf("xz step error %d ", ret); + break; + } + printf(" len %d offset %d\n", (int)len, (int)offset); + return (err); + } + if (dctx->buf_cur < dctx->buf_end) /* Have output space */ + return (ok); + + /* + * We're out of space, grow the buffer and try again if there's buffer + * space. We try again recursively since we know that will usually go + * only 1 deep. + */ + if (EFI_ERROR(grow_buffer(dctx))) + return (err); + if (b.in_pos == b.in_size) + return (ok); + size_t consumed = b.in_pos; + return (xz_step(dctx, buf + consumed, len - consumed, offset + consumed)); +} + +static void +xz_fini(decomp_state *dctx, bool flush) +{ + xz_dec_end(dctx->xzstrm); + dctx->xzstrm = NULL; + if (!flush) + return; + free_buffer(dctx); +} + /* * ZSTD supprot */ @@ -373,6 +459,11 @@ decomp_init(uint8_t *buf, size_t buflen, size_t size_hint) dctx->step = bzip2_step; dctx->fini = bzip2_fini; break; + case xz: + dctx->init = xz_init; + dctx->step = xz_step; + dctx->fini = xz_fini; + break; #ifdef LOADER_ZFS_SUPPORT case zstd: dctx->init = zstd_init; diff --git a/stand/efi/loader/decompress.h b/stand/efi/loader/decompress.h index 43c48078f822..f9d4fe445028 100644 --- a/stand/efi/loader/decompress.h +++ b/stand/efi/loader/decompress.h @@ -6,7 +6,7 @@ #pragma once -enum compression { none, zlib, bzip2, zstd }; +enum compression { none, zlib, bzip2, xz, zstd }; enum step_return { ok, done, err }; typedef struct decomp_state decomp_state; diff --git a/stand/libsa/Makefile b/stand/libsa/Makefile index a0bc47f343b5..ab7a0f5a59f2 100644 --- a/stand/libsa/Makefile +++ b/stand/libsa/Makefile @@ -110,6 +110,13 @@ SRCS+= ${i} SRCS+= lz4.c CFLAGS.lz4.c+= -I${SRCTOP}/sys/cddl/contrib/opensolaris/common/lz4 +# xz decompression +.PATH: ${XZ_DIR}/linux/lib/xz +.for i in xz_crc32.c xz_crc64.c xz_dec_bcj.c xz_dec_lzma2.c xz_dec_stream.c xz.c +CFLAGS.${i}+=${XZ_CFLAGS} +SRCS+= ${i} +.endfor + # io routines SRCS+= closeall.c dev.c ioctl.c nullfs.c stat.c mount.c \ fstat.c close.c lseek.c open.c read.c write.c readdir.c preload.c diff --git a/sys/contrib/xz-embedded/freebsd/xz_config.h b/sys/contrib/xz-embedded/freebsd/xz_config.h index 7fb9a07b5982..4667b64805af 100644 --- a/sys/contrib/xz-embedded/freebsd/xz_config.h +++ b/sys/contrib/xz-embedded/freebsd/xz_config.h @@ -27,10 +27,17 @@ #ifndef __FREEBSD_XZ_CONFIG_H__ #define __FREEBSD_XZ_CONFIG_H__ +#if defined(_KERNEL) #include #include #include #include +#elif defined(_STANDALONE) +#include "stand.h" +#include +#else +/* nothing */ +#endif #include #include "xz_malloc.h" @@ -61,10 +68,6 @@ #define memeq(a, b, size) (memcmp((a), (b), (size)) == 0) #define memzero(buf, size) bzero((buf), (size)) -#ifndef min -# define min(x, y) MIN((x), (y)) -#endif - #define min_t(type, x, y) min((x), (y)) #define get_le32(ptr) le32toh(*(const uint32_t *)(ptr))