Date: Wed, 3 Dec 2025 10:09:38 +0100 From: Wolfram Schneider <wosch@freebsd.org> To: freebsd-current <freebsd-current@freebsd.org> Subject: bsdtar -J --options xz:threads=0' uses only *one* CPU Message-ID: <CAMWY7CCLSUVui6HEQ0eCaMO=LHU%2BUq=CiuEiaqmrhJ3Dfs5zrw@mail.gmail.com>
index | next in thread | raw e-mail
I noticed after a recent change that `make src.txz' is very slow. It turns out that `bsdtar -J' uses only *one* CPU, and the thread option has no effect. Either multithreaded xz is not enabled in bsdtar, or the xz block size is too small. How to repeat: # as a test directory use /usr/bin - ca. 21MByte cd /; tar cf - usr/bin | wc -c 21585920 # bsdtar -J: only one CPU used time -p tar -J --options xz:threads=0 -cf - usr/bin | wc -c 5765120 real 9.38 user 9.28 sys 0.08 # tar | xz: only one CPU used time -p tar -cf - usr/bin | xz -T0 | wc -c 5759376 real 10.22 user 9.77 sys 0.54 # tar | xz + block size 1MB, seven times faster time -p tar -cf - usr/bin | xz -T0 --block-size=1MiB| wc -c 6372068 real 1.55 user 10.93 sys 13.01 # now running with 2*21MB, tar | xz => two CPUs are used time -p tar -cf - usr/bin usr/bin | xz -T0 | wc -c 11515212 real 11.97 user 20.13 sys 0.37 I guess this is due the default xz block size of ca. 24MB on FreeBSD (3 × 8 MB dict size = 24 MB) Any thoughts on how to use more than one CPU with tar -J? PS: on debian bsdtar -J works fine. Strange. PPS: found this very old bug report: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=233543 In lib/libarchive/Makefile.inc there is no setup for multithreaded xz compression (-DHAVE_LZMA_STREAM_ENCODER_MT=1). If I add it, bsdtar uses more than one CPU for data > 20MB. Is there still a reason not to enable multithreaded xz compression in libarchive? -Wolfram -- Wolfram Schneider <wosch@FreeBSD.org> https://wolfram.schneider.orghome | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAMWY7CCLSUVui6HEQ0eCaMO=LHU%2BUq=CiuEiaqmrhJ3Dfs5zrw>