Date: Sat, 23 Dec 2023 13:00:44 +0100 From: Marek Zarychta <zarychtam@plan-b.pwste.edu.pl> To: freebsd-current@freebsd.org Subject: Re: Proposal: Disable compression of newsyslog by default Message-ID: <3c13fd10-997e-40a7-823e-e37a4d481739@plan-b.pwste.edu.pl> In-Reply-To: <8a585a40-0e78-4dbd-9701-aa0926ccde19@quip.cz> References: <ace7d170-acd1-4dc8-8016-a08c261d91f0@delphij.net> <8a585a40-0e78-4dbd-9701-aa0926ccde19@quip.cz>
next in thread | previous in thread | raw e-mail | index | archive | help
W dniu 23.12.2023 o 09:51, Miroslav Lachman pisze: > On 23/12/2023 08:18, Xin Li wrote: >> Hi, >> >> Inspired by D42961, I propose that we move forward with disabling the >> compression by default in newsyslog, as implemented in >> https://reviews.freebsd.org/D43169 >> >> Historically, newsyslog has compressed rotated log files to save disk >> space. This approach was valuable in the early days where storage >> space was limited. However, the landscape has changed >> significantly. Modern file systems, such as ZFS, now offer native >> compression capabilities. Additionally, the widespread availability >> of larger hard drives has diminished the necessity for additional >> compression. Notably, the need to decompress log files for pattern >> searches poses a significant inconvenience, further questioning the >> utility of this legacy feature. >> >> In commit 906748d208d3, flags J, X, Y, Z can now indicate that a log >> file is eligible for compression rather than directly enforcing it. >> It allows for a more flexible approach, wherein the actual >> compression method can be set to "none" or specified as one among >> bzip2, gzip, xz, or zstd. >> >> Therefore I would propose that we change the default compression >> setting to "none" in FreeBSD 15.0. This change reflects our >> adaptation to the evolving technological environment and user needs. >> It also aligns with the broader initiative to modernize our systems >> while maintaining flexibility and efficiency. >> >> I look forward to your thoughts and feedback on this proposal. Thank you for this change and future MFCs. Log files and an urge to process them differ a lot between setups. I guess some people would back completely disabling compression, and some would be against this proposal. There are also small VMs with limited storage formatted with UFS where current defaults: "-c legacy" and using bzip2 (J from newsyslog.conf) to compress logs seems to be perfectly suited. > > I don't think anything needs to be changed on newsyslog. Those who > want to disable compression can do so in the "default" newsyslog.conf > file. Why force this change in the newsyslog code? > I also don't think that the log file should not be compressed even on > a compressed filesystem. Compressed log files can be handled by tools > like zcat, zless, zgrep, etc. without first decompressing the log file. When newsyslog has to deal only with small/medium files and logs are just stored, if processed then very rarely, the compression method is not very relevant, but the situation changes if you have to deal with larger log files, which have to be processed after rotation. While testing this change, I have done a very simple benchmark with the largest log file which I have to rotate daily. It looks to me that having a compressed log file sometimes makes sense even if you have to process these logs in place. Please take a look at the result of running zcat over the compressed file and the timings for the original, uncompressed one. All the files are stored on ZFS lz4 compressed dataset lying on mirrored, not so fast, SSD drives. [r-b] /tmp# /usr/bin/time zcat flowd.0.zst > /dev/null 3.22 real 3.03 user 0.18 sys [r-b] /tmp# /usr/bin/time zcat flowd.0.zst.orig > /dev/null 3.40 real 3.24 user 0.15 sys [r-b] /tmp# /usr/bin/time zcat flowd.0.gz > /dev/null 6.58 real 6.46 user 0.12 sys [r-b] /tmp# /usr/bin/time zcat flowd.0.xz > /dev/null 18.15 real 16.97 user 1.18 sys [r-b] /tmp# /usr/bin/time zcat flowd.0.bz2 > /dev/null 45.47 real 45.33 user 0.12 sys [r-b] /tmp# /usr/bin/time cat flowd.0 > /dev/null 3.35 real 0.25 user 3.10 sys [r-b] /tmp# ls -l flowd.0* -rw------- 1 root wheel 3823756280 Dec 23 09:56 flowd.0 -rw------- 1 root wheel 195036235 Dec 23 09:56 flowd.0.bz2 -rw------- 1 root wheel 305250854 Dec 23 09:56 flowd.0.gz -rw------- 1 root wheel 180642412 Dec 23 09:56 flowd.0.xz -rw------- 1 root wheel 252470186 Dec 23 09:56 flowd.0.zst (zstd with --long --adapt -T0) -rw------- 1 root wheel 297184160 Dec 23 09:56 flowd.0.zst.orig (pre 906748d208d3 zstd options) > Text log files can still grow to large sizes, and if you have a daily > backup job over a long distance network, if you use a protocol without > own compression, it is still better to have compressed log files than > uncompressed on a compressed filesystem. To me, it's the same as > compressing large database dumps and not relying on filesystem > compression. > YMMV, but I really don't see any benefit of changing the newsyslog > code, just change defaults in newsyslog.conf. > There is a clear benefit for me, users who don't want their logs to be compressed, will have just to alter one line in /etc/crontab: # Rotate log files every hour, if necessary. 0 * * * * root newsyslog -c none instead of changing /etc/newsyslog.conf line by line. Thank you for the new newsyslog(8) -c option providing this feature ! With kind regards -- Marek Zarychta
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3c13fd10-997e-40a7-823e-e37a4d481739>