From nobody Wed Dec 3 09:09:38 2025 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4dLsKZ1mLKz6J00F for ; Wed, 03 Dec 2025 09:09:54 +0000 (UTC) (envelope-from wschnr@googlemail.com) Received: from mail-wr1-f45.google.com (mail-wr1-f45.google.com [209.85.221.45]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4dLsKY180jz3fgY for ; Wed, 03 Dec 2025 09:09:53 +0000 (UTC) (envelope-from wschnr@googlemail.com) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of wschnr@googlemail.com designates 209.85.221.45 as permitted sender) smtp.mailfrom=wschnr@googlemail.com Received: by mail-wr1-f45.google.com with SMTP id ffacd0b85a97d-42e2ddb8a13so1607112f8f.0 for ; Wed, 03 Dec 2025 01:09:53 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764752991; x=1765357791; h=content-transfer-encoding:to:subject:message-id:date:from :mime-version:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=fnGqSEbdgVovUwtTyg465IXP3JTCgFXdLOjOn2fN3Mc=; b=T2wdOXIoOsBp4Rr2kLA8SIYgKJK5ZxqfuiUvt6N3ppRQK6g8T5iqXEUlf/h0eULnJu vtJXTOz4cOCW3v0V1kim2d5ecuLImz749WY07qWGes2KHhbRcWMy0kERIGCW3uQxyEkg qzHMAIR9JjXcsusxX3wplA3GYIfCIHCZ4HRSvNO4rQqUMhBHInscxRwmay268OgirPLD lHpFKt2cVbuQ+9SMmkwAJbrfvEbFJ1ra2EPNVppX7sgGmbwnsuqB4kYlMgNWWjJXloIe IhsLlySOsZFGambHZQMMm6AfBedkm85TKxkA7X98L74yeFjV5cwn/FCtoo7PIlfvFhC9 92WQ== X-Gm-Message-State: AOJu0YzDxOMoNkNXCkHNAhP4kINSALSBXQiRoN12skOlZATju2TYukfG //mLUtpBpuD3+yt6i0gZS/LTV7iXWCbVFQPeZsnH7cRDKhvY8+QY6z5BaYD9WXz5sbKpgsbH9wh ONxTgjmEtlBEi6zjPPkpH9V3YdM1m6bYdHvarxAs= X-Gm-Gg: ASbGncuuJTQ0N2/N2558/JVXawT219lDQbAaVbS35RTJI4Eh+iu4JDWMILYM/h/s8cJ bAkGB4j8oFWSF+yRErGlGHw2Bzr9G1Db6FMO3uyA0UJo9+fIRWs3jcq+5AN+T8H+yF5iJrWHiYR jt6d2JzHz0RTVSBpje5/vCTvvkutj5PEb3pgUuUqPpa/BTooI6TgyIbfmC9zt8Gyli1GBCFetDQ j+CcazMVVyHBF4ELfPoJiKPnjNvjl2eIOICtTrnfxFbxdzqawAM+mcammnfi2WbeVdxBo/zlkAv 9s1+L6+4Q6ce8kubSZkL7k2H8dliaSgtKy4u6RB72niKurftYUY= X-Google-Smtp-Source: AGHT+IHV7ktsWmXV3yDb/RU2eg+LUVoBv1IlPdi4mj7ubFP5z6c+R5z8mElWx8SA61y1WvcXXTTNvGxUcjqxa5aUVlU= X-Received: by 2002:a05:6000:611:b0:429:d1a8:3fa2 with SMTP id ffacd0b85a97d-42f731c553fmr1354499f8f.48.1764752990413; Wed, 03 Dec 2025 01:09:50 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 From: Wolfram Schneider Date: Wed, 3 Dec 2025 10:09:38 +0100 X-Gm-Features: AWmQ_bmgjJJCfsp1lXW7DX0-GDMBgOdqNUuvKkNqJqkU5S1uu70qTlc0SOpgllQ Message-ID: Subject: bsdtar -J --options xz:threads=0' uses only *one* CPU To: freebsd-current Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.51 / 15.00]; NEURAL_HAM_MEDIUM(-0.95)[-0.951]; NEURAL_HAM_SHORT(-0.84)[-0.836]; NEURAL_HAM_LONG(-0.82)[-0.822]; FORGED_SENDER(0.30)[wosch@freebsd.org,wschnr@googlemail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), No valid DKIM,none]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; MISSING_XM_UA(0.00)[]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FREEMAIL_ENVFROM(0.00)[googlemail.com]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; FROM_NEQ_ENVFROM(0.00)[wosch@freebsd.org,wschnr@googlemail.com]; RCVD_COUNT_ONE(0.00)[1]; R_DKIM_NA(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.221.45:from]; BLOCKLISTDE_FAIL(0.00)[209.85.221.45:server fail]; TO_DOM_EQ_FROM_DOM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[209.85.221.45:from] X-Rspamd-Queue-Id: 4dLsKY180jz3fgY I noticed after a recent change that `make src.txz' is very slow. It turns out that `bsdtar -J' uses only *one* CPU, and the thread option has no effect. Either multithreaded xz is not enabled in bsdtar, or the xz block size is too small. How to repeat: # as a test directory use /usr/bin - ca. 21MByte cd /; tar cf - usr/bin | wc -c 21585920 # bsdtar -J: only one CPU used time -p tar -J --options xz:threads=3D0 -cf - usr/bin | wc -c 5765120 real 9.38 user 9.28 sys 0.08 # tar | xz: only one CPU used time -p tar -cf - usr/bin | xz -T0 | wc -c 5759376 real 10.22 user 9.77 sys 0.54 # tar | xz + block size 1MB, seven times faster time -p tar -cf - usr/bin | xz -T0 --block-size=3D1MiB| wc -c 6372068 real 1.55 user 10.93 sys 13.01 # now running with 2*21MB, tar | xz =3D> two CPUs are used time -p tar -cf - usr/bin usr/bin | xz -T0 | wc -c 11515212 real 11.97 user 20.13 sys 0.37 I guess this is due the default xz block size of ca. 24MB on FreeBSD (3 =C3=97 8 MB dict size =3D 24 MB) Any thoughts on how to use more than one CPU with tar -J? PS: on debian bsdtar -J works fine. Strange. PPS: found this very old bug report: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D233543 In lib/libarchive/Makefile.inc there is no setup for multithreaded xz compression (-DHAVE_LZMA_STREAM_ENCODER_MT=3D1). If I add it, bsdtar uses more than one CPU for data > 20MB. Is there still a reason not to enable multithreaded xz compression in libarchive? -Wolfram -- Wolfram Schneider https://wolfram.schneider.org