Date: Wed, 10 Jan 2024 07:50:05 -0600 From: Mike Karels <mike@karels.net> To: Xin LI <delphij@gmail.com> Cc: Xin LI <delphij@freebsd.org>, src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org Subject: Re: git: 2f036705f337 - main - Document the two recent newsyslog(8) change (-c option and <compress> configuration option). Message-ID: <E6DFD9F9-99C3-4581-83F2-6FED3497E245@karels.net> In-Reply-To: <CAGMYy3uOjXY5LaVuJkXnwyFq0h7VEesaMGDvJ2i4J6VcNWnR4g@mail.gmail.com> References: <202312290846.3BT8kOiO029918@gitrepo.freebsd.org> <90D0905E-AA46-4351-AEE0-9ED9D835DB50@karels.net> <CAGMYy3uOjXY5LaVuJkXnwyFq0h7VEesaMGDvJ2i4J6VcNWnR4g@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 9 Jan 2024, at 11:54, Xin LI wrote: > Hi, Mike, > > On Fri, Dec 29, 2023 at 7:25 AM Mike Karels <mike@karels.net> wrote: > >> On 29 Dec 2023, at 2:46, Xin LI wrote: >> >>> The branch main has been updated by delphij: >> [...] > >> >>> +61174ad88e33: >>> + newsyslog(8) now supports specifying a global compression method >> directly >>> + at the beginning of the newsyslog.conf file, which will make >> newsyslog(8) >>> + to behave like the corresponding option was passed to the newly >> added >>> + '-c' option. For example: >>> + >>> + <compress> none >>> + >>> +906748d208d3: >>> + newsyslog(8) now accepts a new option, '-c' which overrides all >> historical >>> + compression flags by treating their meaning as "treat the file as >> compressible" >>> + rather than "compress the file with that specific method." >>> + >>> + The following choices are available: >>> + * none: Do not compress, regardless of flag. >>> + * legacy: Historical behavior (J=bzip2, X=xz, Y=zstd, Z=gzip). >>> + * bzip2, xz, zstd, gzip: apply the specified compression method. >>> + >>> + We plan to change the default to 'none' in FreeBSD 15.0. >> >> Sorry not to have noticed this in the review; it was only when I saw this >> message that it sunk in that we now have *three* ways to specify >> compression, >> and I'm not even sure what the precedence is. I would have thought that >> > > The current precedence is that command line takes highest priority; when > it's not present, <compress> overrides all lines with any compression > method. > > The meaning of J/X/Y/Z is changed to "this file is compressible", unless > newsyslog is in legacy mode. It's intentional to discourage the use of > different compression types for different files (because the current > newsyslog.conf(5) format uses one character to represent compression method > and without a change to e.g. UCL or some other configuration language, > supporting additional compression methods would be much more messier). Not if you adopted my suggestion of adding a letter to specify the default compression as specified by <compress>. And then J/X/Y/Z would not be overloaded. Why do you want to discourage the use of different compression types for different files? If the feature wasn't there and was being considered, I would say it wasn't worth doing. But it's there, and can't be removed anytime soon. I see no cost in retaining it indefinitely. >> <compress> would replace -c. It's a mess if the config file has entries >> that specify J and X flags as well as none, the config file has >> <compress> zstd, and the -c option is given as well. We now have a knob >> > > newsyslog would issue a warning if both -c is given and the configuration > file has <compress>. > > Part of the reason to introduce <compress> was that it would move the > configuration to one place (newsyslog.conf) and there were concerns that > having -c in newsyslog in /etc/crontab is counterintuitive (as raised in > some earlier discussion). > > >> to override the knob to override a knob. The only reason to keep -c that >> I can think of is to specify a different compression in a single >> invocation, >> but as noted, changing compression requires manual operations that make >> it unreasonable to change it invocation by invocation. >> > > The intention of -c was to be added to /etc/crontab, but with the > introduction of <compress> it's probably a good idea to just remove the > command line option or at least leave it undocumented? I agree that -c should be removed. Retaining it as an undocumented option seems like a recipe for confusion, and there is not much utility in varying the compression type from one invocation to the next. Mixing types without manually converting the existing files would lead to a mess. >> Also, using the -c option or <compress> (except with legacy) loses the >> ability to specify different compression types for different files. >> > > This (the ability of specifying different compression types for different > files) is what I deliberately wanted to get rid of. > > Compression types only make a meaningful difference when files are > sufficiently large, and the purpose of newsyslog is to not allow log files > to grow to an unmanageable size. Rotating files when they are large and > also requesting to compress them means the system would have prolonged > "bursts" of CPU workload at integral hours, which would be even worse for > busy systems where logs tend to have more contents. That's a tradeoff that different people (and systems) might differ on. For systems that produce large log files and are not CPU-constrained, it's a perfectly reasonable choice. I have a firewall that produces large logs and runs on a virtual machine, so the disk is not that large. >> People have objected to the term "legacy" as well. >> > > I don't agree. > > Calling it "legacy" would send a clear signal that the historical way is > discouraged (for people who are using a different compression method, > merging upstream change would be a very painful process). We already have > 4 letters to represent 4 different compression methods, that's almost 1/3 > of all valid configuration letters, and we should stop doing that because > the legacy way is causing problems and won't scale when new compression > methods are added. See my point above. If a new letter was adopted for "the default", then new compression methods would not require a letter, they could be limited to specification with <compress>. >> I still think it would be much better to add an option letter to select >> the default compression as specified by <compress>. This would eliminate >> the need for "legacy", and it would add the ability to have both a global >> default and an exception. I think the redefinition of the existing flags >> to have different meanings if <compress> is given is messy. >> > > I insist the default should be anything except "legacy" -- legacy is for > backward compatibility and should serve that purpose only. I think that bzip2 is the only reasonable default, as switching a system automatically on an upgrade would leave a mess of old compressed files unless people convert manually. Providing a conversion script would be a help though. I don't know why you are so determined to change the default. Changing to "none" is reasonable on many systems, changing to a better method is fine (with manual conversion), and "legacy" methods would include all currently- supported methods. That doesn't sound like legacy to me. It's mostly the configuration mechanism that would be legacy. > The entry for -c says that we plan to change the default to "none" in 15.0. >> Hopefully that would be done via <compress> and not -c. However, there >> > > Yes, my intention was to add a line to newsyslog.conf ( > https://reviews.freebsd.org/D43169 ) instead of adding it to /etc/crontab. > > >> was significant pushback on "none" being the default. >> > > I think we should provide a reasonable default. Compression was useful > back in the 90's, but not so much nowadays and makes grep'ing the logs > harder (and for those who want compression, they could choose e.g. 'bzip2' > anyways). This change would also make porter's lives a lot easier as they > could install their newsyslog.conf.d files with e.g. 'J' to indicate that > the file can be compressed, without messing up with the administrator's > decision to not compress or compress it with some other compressor. Or, using my proposal, they could use the new option letter for the default. I think the new default newsyslog.conf should have "<compress> bzip2" and use the a new option letter in all the existing entries. That would make it easier to change the default on individual systems without extra hassle in changing the default; only one line would change. As noted above, using compression is perfectly reasonable on many systems. Almost all of my servers run on virtual machines, so don't have unlimited disk space. And support for compression is unlikely to be removed, so pushing for its use to be reduced doesn't seem useful. Mike
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?E6DFD9F9-99C3-4581-83F2-6FED3497E245>