Date: Tue, 09 Jan 2024 11:19:48 +0100 From: Olivier Certner <olce@freebsd.org> To: Xin LI <delphij@freebsd.org>, Mike Karels <mike@karels.net> Cc: src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org Subject: Re: git: 2f036705f337 - main - Document the two recent newsyslog(8) change (-c option and <compress> configuration option). Message-ID: <2683023.poxlI1A5LX@ravel> In-Reply-To: <90D0905E-AA46-4351-AEE0-9ED9D835DB50@karels.net> References: <202312290846.3BT8kOiO029918@gitrepo.freebsd.org> <90D0905E-AA46-4351-AEE0-9ED9D835DB50@karels.net>
next in thread | previous in thread | raw e-mail | index | archive | help
--nextPart6134885.r39cKavRk3 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8"; protected-headers="v1" From: Olivier Certner <olce@freebsd.org> To: Xin LI <delphij@freebsd.org>, Mike Karels <mike@karels.net> Date: Tue, 09 Jan 2024 11:19:48 +0100 Message-ID: <2683023.poxlI1A5LX@ravel> In-Reply-To: <90D0905E-AA46-4351-AEE0-9ED9D835DB50@karels.net> MIME-Version: 1.0 Hi, Sorry not to have gotten to that earlier. I had initially expressed on IRC that I found the newsyslog changes great, = but now reading Mike's arguments and proposals I have serious doubts on the= current approach. > Sorry not to have noticed this in the review; it was only when I saw this > message that it sunk in that we now have *three* ways to specify compress= ion, > and I'm not even sure what the precedence is. I would have thought that > <compress> would replace -c. It's a mess if the config file has entries > that specify J and X flags as well as none, the config file has > <compress> zstd, and the -c option is given as well. We now have a knob > to override the knob to override a knob. The only reason to keep -c that > I can think of is to specify a different compression in a single invocati= on, > but as noted, changing compression requires manual operations that make > it unreasonable to change it invocation by invocation. I agree. Two possibilies that I can think of from here: Remove '-c' or mak= e it enable compression regardless of the log files' individual settings. =20 > I still think it would be much better to add an option letter to select > the default compression as specified by <compress>. This would eliminate > the need for "legacy", and it would add the ability to have both a global > default and an exception. I think the redefinition of the existing flags > to have different meanings if <compress> is given is messy. I didn't think about that at first. I agree. If people want to be able to override compression settings globally, which = I find useful, one could introduce another directive such as <compress_over= ride> taking a boolean to request to apply the <compress> option regardless= of the individual compression letters. Another possibility is just to rename "<compress>" to "<compress_override>"= (so, this time, not a boolean) and keep its current behavior. This would = match one of the suggestions above about '-c', but then there's the questio= n of which one takes precedence, and I think that the command-line specific= ation should prevail (for practical purposes and POLA). =20 > The entry for -c says that we plan to change the default to "none" in 15.= 0. > Hopefully that would be done via <compress> and not -c. However, there > was significant pushback on "none" being the default. I think the default should be "no <compress_override>", i.e., no directive.= This may plea for having "none" mean "don't change anything" (as if the d= irective wasn't there) and have something else to deactivate compression, s= uch as "no_compression" (which is really an override). If "none" is confus= ing, then just forego it completely, and have 'newsyslog' plain fail on it = (but keep "no_compression" as just described). If there is consensus, I'd then change the 'J' flag currently used for all = log files to the new chosen flag for generic compression, and have <compres= s_override> set to "bzip2" in a first step (for POLA). Then, it could be c= hanged to something else, e.g., 'zstd'. Setting it to 'none' seems to me the worst solution (but far from being the= end of the world). More deeply, I remember having seen at least two claims that using filesyst= em's compression is better, without arguments. I don't agree with that in = practice. The only advantage of in-filesystem compression, besides the adm= inistrative simplification that you can also get with the override above, i= s to get O(1) random access to big log files, and I don't see any compellin= g and common use case for it. You certainly want to get to the end of the = current log quickly, but that one precisely is not handled by 'newsyslog' a= nd stays uncompressed (at the application level). When you want to search = for strings or patterns, you have to grep the whole file anyway. You may w= ant to immediately reach the end of some historical log file, e.g., when ma= nually going back in time from the current log, but this should have neglig= ible latency, and if it doesn't, than just use more and smaller log archive= s. Same thing if you have a more sophisticated setup with an index of log = text: Jumping to a particular location in the log file should have negligib= le latency, else apply the same recipe. If your setup with index requires = a single, never rotated, log file, then you're not even using 'newsyslog' i= n the first place (or should not). Although I agree that in this case usin= g a compressed filesystem (or a randomly accessible archive) can make sense= (if your index doesn't already cover the results expected from your search= es), I very much doubt this is a common setup. Moreover, using in-filesystem compression can lead to degrading the compres= sion ratio, since the compression method on ZFS is chosen per dataset, whic= h includes a bunch of other files and use cases preventing the administrato= r from choosing the best, and slowest, compression methods. To avoid this = problem, one can use a separate dataset for /var/log (anyone?), but changin= g this on already running systems is a greater burden than just changing th= e compression settings in the 'newsyslog' configuration files. I'd like people who disagree with this to present arguments for their case,= if for nothing else to share their experience and best practices on log ma= nagement. Thanks and regards. =2D-=20 Olivier Certner --nextPart6134885.r39cKavRk3 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part. Content-Transfer-Encoding: 7Bit -----BEGIN PGP SIGNATURE----- iQIzBAABCQAdFiEEmNCxHjkosai0LYIujKEwQJceJicFAmWdHcQACgkQjKEwQJce JifhlA//ehOBZnBNEhdAuixFk+drmEXD5UMclOJZV84mpsZRH/McOZ6dKs1Maq9M 91gvyQI3waZADOWkPkE1iyTywm1GgpvUq7JuQ1QzITOLPl3shQCgn6sxUOCNG4ly wKh/L6w5bvAqaFN8zj/RBpSeXfPADFowZ7tUNPZQlr4sGpMO34fp0KEL29rQQVxI wyPTEepfYDDewhxtOOPwJXs3uwMfS3UHAHlnZv0FjYH/oEPXF2plc5X1mDuw1SGg nFE2kRrfA1kZf8yu4XspLRZ/ADaCHko0xw3DE11gxMElQ/9QKZ3vk1yOwtim8UVN h4maCUVo6QHylbIR8nhDu+VdSBZb2wavoVFom7zfBd5xWgzzzBKRVc48JSMMqXMb aRi8Jk4fDQqZ0heH5abtE8R1sFKxDojYHj57hKJXZvcrs7jEqwdL8N1UT4MHGbyA b9E0iroTdcDD7EGvm7wd3z24bPb9pj5mJ1XOa5CYvPqSlNMpAfxTlu/041is3xSh 82+mT2UckYtVlp71xzDFipFZ5sLdwZD8/JkD6yojzN7CphhsBBf+dbVzDi2KqZzP SR/7xr6+J4cqZlRXIPFL59aNLcwIBBg0hcjBc6T4yiG/qf85fvMbxveQI7tHz18g Gpq4izGlh/UFmXz91BoAn1FV7D/zzELTfrUbynLBsNM69MSz7/w= =PJpJ -----END PGP SIGNATURE----- --nextPart6134885.r39cKavRk3--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2683023.poxlI1A5LX>