From nobody Thu Jun 26 11:47:50 2025 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4bScPf4rH6z60SjJ; Thu, 26 Jun 2025 11:47:50 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4bScPf2yBzz3hbp; Thu, 26 Jun 2025 11:47:50 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1750938470; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=6nmTXCRTEuYNT6dqV0IP4rmDIsuLWGaXzJVOWj9r7Sw=; b=m5F7GfnCM1y2o3PSdjVQtnTXrlSwERp2xwbGdLT9skXLhVZ/NbouZ52QQwSI1aw678mhuY ohcPUcxwmnSRFp8dU6ecnzwVFX6I6Hqb7+iP8w5M90qtU/dk9/xDzxGd43sCAODrndHEH3 e5vzGsWCL2/NwyJKHIX6o+g+Shosgykpc8dqv5kJ887PRkCjygcJlDx1ziWoD4iuG1APPf NiB+G4/3DtKoBzdwPIvJHdLgZQFoETHh5ODJCvpWvkZvC9fU4mJ891YR4bdfqZwwOqZPdv tz/wEABIwp+7lfB/EGRYxAjHpvHdm7nZke7IAbFYBmTizm+1IuQIVTvVnsvJXQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1750938470; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=6nmTXCRTEuYNT6dqV0IP4rmDIsuLWGaXzJVOWj9r7Sw=; b=Pg3byRvoWa7WPX3Rgo4+e114TbbuyRDbjma2+U8/GqR6LmLS/2AdMI9U4570W0gWcAcIpn CfoaYQRNJQ5EVnlZhv0BgatnstISXqzR2f2/9FQo9/1bq6/PasnHdKRLe9KbhMdlaHwRw+ 1cQ4OnGh0vz2nVmSm4s8nri11yQNRRDT+HrYox2wWGcItddHCJw87D7qmoWlQhEpDqqbJy NSmc5muI+cbemwkHGe1otL88iT4r7UbsV0jIdjlrwo5rm+fxoiQilKybo5R52K9zaC9p0H mKsB5aRlMC/nfuXDu8E4qdYaWN4y6ESv0TzGd9lic5sRiZptbpNap8HYj97iLA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1750938470; a=rsa-sha256; cv=none; b=eSVEAkSmVF+IQYRfmj1NRQd/LJ81hzQ//2lYYvnLIAc7ocXsracSSB5HHF3ctbosyUg61t 3IS4Ghl9v5TyoHVChRcxhkhxVL2tPiaFu7fWYmAsoyK+TwTflCwX83Xtsx55RykxVmOcH5 FSy0Gcnaw66nQ0T45z1i36qlNPB5mHjJbsAPkXU5CXax5vuCgQpJzp10NN4fxXbk48iAHW wKZzhYJJH0qf1/lAKA+FGANSdb7ZpxQXtTH33orhZo28ygHicIqhQa+J+koMKk7nMjo6fn +wzp4xY5QMngnUZrGMX8scN0FDC3uC/4VvLhtH0rmggf37RTkFYwTFuwT6+fgA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4bScPf24mcz12HL; Thu, 26 Jun 2025 11:47:50 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 55QBloew059786; Thu, 26 Jun 2025 11:47:50 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 55QBlo6r059781; Thu, 26 Jun 2025 11:47:50 GMT (envelope-from git) Date: Thu, 26 Jun 2025 11:47:50 GMT Message-Id: <202506261147.55QBlo6r059781@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Baptiste Daroussin Subject: git: 19a7ea3cc4de - main - nuageinit: implement write_files List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: bapt X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 19a7ea3cc4de5af80e2913fda70bd65ad72835c0 Auto-Submitted: auto-generated The branch main has been updated by bapt: URL: https://cgit.FreeBSD.org/src/commit/?id=19a7ea3cc4de5af80e2913fda70bd65ad72835c0 commit 19a7ea3cc4de5af80e2913fda70bd65ad72835c0 Author: Baptiste Daroussin AuthorDate: 2025-06-26 11:32:07 +0000 Commit: Baptiste Daroussin CommitDate: 2025-06-26 11:47:37 +0000 nuageinit: implement write_files write_files is a list of files that should be created at the first boot each file content can be either plain text or encoded in base64 (note that cloudinit specify that gzip is supported, but we do not support it yet.) All other specifier from cloudinit should work: by default all files will juste overwrite exesiting files except if "append" is set to true, permissions, ownership can be specified. The files are create before packages are being installed and user created. if "defer" is set to true then the file is being created after packages installation and package manupulation. This feature is requested for KDE's CI. --- libexec/nuageinit/nuage.lua | 88 +++++++++++++++++++++++++++++++++++- libexec/nuageinit/nuageinit | 25 +++++++++- libexec/nuageinit/nuageinit.7 | 38 +++++++++++++++- libexec/nuageinit/tests/Makefile | 1 + libexec/nuageinit/tests/addfile.lua | 71 +++++++++++++++++++++++++++++ libexec/nuageinit/tests/nuage.sh | 10 +++- libexec/nuageinit/tests/nuageinit.sh | 37 ++++++++++++++- 7 files changed, 264 insertions(+), 6 deletions(-) diff --git a/libexec/nuageinit/nuage.lua b/libexec/nuageinit/nuage.lua index deb441ee25ba..cdc0fc6cf2a7 100644 --- a/libexec/nuageinit/nuage.lua +++ b/libexec/nuageinit/nuage.lua @@ -7,6 +7,39 @@ local unistd = require("posix.unistd") local sys_stat = require("posix.sys.stat") local lfs = require("lfs") +local function decode_base64(input) + local b = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/' + input = string.gsub(input, '[^'..b..'=]', '') + + local result = {} + local bits = '' + + -- convert all characters in bits + for i = 1, #input do + local x = input:sub(i, i) + if x == '=' then + break + end + local f = b:find(x) - 1 + for j = 6, 1, -1 do + bits = bits .. (f % 2^j - f % 2^(j-1) > 0 and '1' or '0') + end + end + + for i = 1, #bits, 8 do + local byte = bits:sub(i, i + 7) + if #byte == 8 then + local c = 0 + for j = 1, 8 do + c = c + (byte:sub(j, j) == '1' and 2^(8 - j) or 0) + end + table.insert(result, string.char(c)) + end + end + + return table.concat(result) +end + local function warnmsg(str, prepend) if not str then return @@ -441,6 +474,58 @@ local function upgrade_packages() return run_pkg_cmd("upgrade") end +local function addfile(file, defer) + if type(file) ~= "table" then + return false, "Invalid object" + end + if defer and not file.defer then + return true + end + if not defer and file.defer then + return true + end + if not file.path then + return false, "No path provided for the file to write" + end + local content = nil + if file.content then + if file.encoding then + if file.encoding == "b64" or file.encoding == "base64" then + content = decode_base64(file.content) + else + return false, "Unsupported encoding: " .. file.encoding + end + else + content = file.content + end + end + local mode = "w" + if file.append then + mode = "a" + end + + local root = os.getenv("NUAGE_FAKE_ROOTDIR") + if not root then + root = "" + end + local filepath = root .. file.path + local f = assert(io.open(filepath, mode)) + if content then + f:write(content) + end + f:close() + if file.permissions then + -- convert from octal to decimal + local perm = tonumber(file.permissions, 8) + sys_stat.chmod(file.path, perm) + end + if file.owner then + local owner, group = string.match(file.owner, "([^:]+):([^:]+)") + unistd.chown(file.path, owner, group) + end + return true +end + local n = { warn = warnmsg, err = errmsg, @@ -456,7 +541,8 @@ local n = { install_package = install_package, update_packages = update_packages, upgrade_packages = upgrade_packages, - addsudo = addsudo + addsudo = addsudo, + addfile = addfile } return n diff --git a/libexec/nuageinit/nuageinit b/libexec/nuageinit/nuageinit index 5af1b84c1848..84133d4373c5 100755 --- a/libexec/nuageinit/nuageinit +++ b/libexec/nuageinit/nuageinit @@ -188,6 +188,25 @@ local function install_packages(packages) end end +local function write_files(files, defer) + if not files then + return + end + for n, file in pairs(files) do + local r, errstr = nuage.addfile(file, defer) + if not r then + nuage.warn("Skipping write_files entry number " .. n .. ": " .. errstr) + end + end +end + +local function write_files_not_defered(obj) + write_files(obj.write_files, false) +end + +local function write_files_defered(obj) + write_files(obj.write_files, true) +end -- Set network configuration from user_data local function network_config(obj) if obj.network == nil then return end @@ -456,13 +475,15 @@ if line == "#cloud-config" then ssh_authorized_keys, network_config, ssh_pwauth, - runcmd + runcmd, + write_files_not_defered, } local post_network_calls = { packages, users, - chpasswd + chpasswd, + write_files_defered, } f = io.open(ni_path .. "/" .. ud) diff --git a/libexec/nuageinit/nuageinit.7 b/libexec/nuageinit/nuageinit.7 index 1d2f83fe62e0..3bb440ebac95 100644 --- a/libexec/nuageinit/nuageinit.7 +++ b/libexec/nuageinit/nuageinit.7 @@ -2,7 +2,7 @@ .\" .\" Copyright (c) 2025 Baptiste Daroussin .\" -.Dd June 16, 2025 +.Dd June 26, 2025 .Dt NUAGEINIT 7 .Os .Sh NAME @@ -239,6 +239,42 @@ where x is a number, then the password is considered encrypted, otherwise the password is considered plaintext. .El .El +.It Ic write_files +An array of objects representing files to be created at first boot. +The files are being created before the installation of any packages +and the creation of the users. +The only mandatory field is: +.Ic path . +It accepts the following keys for each objects: +.Bl -tag -width "permissions" +.It Ic content +The content to be written to the file. +If this key is not existing then an empty file will be created. +.It Ic encoding +Specifiy the encoding used for content. +If not specified, then plain text is considered. +Only +.Ar b64 +and +.Ar base64 +are supported for now. +.It Ic path +The path of the file to be created. +.Pq Note intermerdiary directories will not be created . +.It Ic permissions +A string representing the permission of the file in octal. +.It Ic owner +A string representing the owner, two forms are possible: +.Ar user +or +.Ar user:group . +.It Ic append +A boolean to specify the content should be appended to the file if the file +exists. +.It Ic defer +A boolean to specify that the files should be created after the packages are +installed and the users are created. +.El .El .Sh EXAMPLES Here is an example of a YAML configuration for diff --git a/libexec/nuageinit/tests/Makefile b/libexec/nuageinit/tests/Makefile index bb2f0d7c747e..c69bc28a4c86 100644 --- a/libexec/nuageinit/tests/Makefile +++ b/libexec/nuageinit/tests/Makefile @@ -16,5 +16,6 @@ ${PACKAGE}FILES+= dirname.lua ${PACKAGE}FILES+= err.lua ${PACKAGE}FILES+= sethostname.lua ${PACKAGE}FILES+= warn.lua +${PACKAGE}FILES+= addfile.lua .include diff --git a/libexec/nuageinit/tests/addfile.lua b/libexec/nuageinit/tests/addfile.lua new file mode 100644 index 000000000000..98d020e557c0 --- /dev/null +++ b/libexec/nuageinit/tests/addfile.lua @@ -0,0 +1,71 @@ +#!/bin/libexec/flua + +local n = require("nuage") +local lfs = require("lfs") + +local f = { + content = "plop" +} + +local r, err = n.addfile(f, false) +if r or err ~= "No path provided for the file to write" then + n.err("addfile should not accept a file to write without a path") +end + +local function addfile_and_getres(file) + local r, err = n.addfile(file, false) + if not r then + n.err(err) + end + local root = os.getenv("NUAGE_FAKE_ROOTDIR") + if not root then + root = "" + end + local filepath = root .. file.path + local resf = assert(io.open(filepath, "r")) + local str = resf:read("*all") + resf:close() + return str +end + +-- simple file +f.path="/tmp/testnuage" +local str = addfile_and_getres(f) +if str ~= f.content then + n.err("Invalid file content") +end + +-- the file is overwriten +f.content = "test" + +str = addfile_and_getres(f) +if str ~= f.content then + n.err("Invalid file content, not overwritten") +end + +-- try to append now +f.content = "more" +f.append = true + +str = addfile_and_getres(f) +if str ~= "test" .. f.content then + n.err("Invalid file content, not appended") +end + +-- base64 +f.content = "YmxhCg==" +f.encoding = "base64" +f.append = false + +str = addfile_and_getres(f) +if str ~= "bla\n" then + n.err("Invalid file content, base64 decode") +end + +-- b64 +f.encoding = "b64" +str = addfile_and_getres(f) +if str ~= "bla\n" then + n.err("Invalid file content, b64 decode") + print("==>" .. str .. "<==") +end diff --git a/libexec/nuageinit/tests/nuage.sh b/libexec/nuageinit/tests/nuage.sh index f2753d6d91e6..56651c8c5bb7 100644 --- a/libexec/nuageinit/tests/nuage.sh +++ b/libexec/nuageinit/tests/nuage.sh @@ -1,5 +1,5 @@ #- -# Copyright (c) 2022 Baptiste Daroussin +# Copyright (c) 2022-2025 Baptiste Daroussin # # SPDX-License-Identifier: BSD-2-Clause # @@ -11,6 +11,7 @@ atf_test_case addsshkey atf_test_case adduser atf_test_case adduser_passwd atf_test_case addgroup +atf_test_case addfile sethostname_body() { @@ -73,6 +74,12 @@ addgroup_body() atf_check -o inline:"impossible_groupname:*:1001:\n" grep impossible_groupname etc/group } +addfile_body() +{ + mkdir tmp + atf_check /usr/libexec/flua $(atf_get_srcdir)/addfile.lua +} + atf_init_test_cases() { atf_add_test_case sethostname @@ -80,4 +87,5 @@ atf_init_test_cases() atf_add_test_case adduser atf_add_test_case adduser_passwd atf_add_test_case addgroup + atf_add_test_case addfile } diff --git a/libexec/nuageinit/tests/nuageinit.sh b/libexec/nuageinit/tests/nuageinit.sh index 44830f67e4c8..639c87181f95 100644 --- a/libexec/nuageinit/tests/nuageinit.sh +++ b/libexec/nuageinit/tests/nuageinit.sh @@ -1,5 +1,5 @@ #- -# Copyright (c) 2022 Baptiste Daroussin +# Copyright (c) 2022-2025 Baptiste Daroussin # # SPDX-License-Identifier: BSD-2-Clause # @@ -29,6 +29,7 @@ atf_test_case config2_userdata_update_packages atf_test_case config2_userdata_upgrade_packages atf_test_case config2_userdata_shebang atf_test_case config2_userdata_fqdn_and_hostname +atf_test_case config2_userdata_write_files setup_test_adduser() { @@ -847,6 +848,39 @@ EOF fi } +config2_userdata_write_files_body() +{ + mkdir -p media/nuageinit + setup_test_adduser + printf "{}" > media/nuageinit/meta_data.json + cat > media/nuageinit/user_data <