From nobody Tue Jan 9 10:19:48 2024 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T8Rkl1tH7z571dR; Tue, 9 Jan 2024 10:19:59 +0000 (UTC) (envelope-from olce@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T8Rkl0Bjrz4V4l; Tue, 9 Jan 2024 10:19:59 +0000 (UTC) (envelope-from olce@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1704795599; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=z3YdflRZyBSeEL44OapQ6KDY+nqS8MTIvxlJ++jaaM4=; b=YtmYIDqeQM+9EjgPgNB+3cPfy0tPR/YMr0onQC1UNC5ZKWRIK/k9YpSKxfRw0t/+P+MS0o pE3xXGW6/mHzrh+BlJSSszktUR77ory4hcKQt74egVB5hmW0bKBNjYhDCyfa/tiDjJ2wpf M//YLtUwy9tfOSVxWFuCCh29XppTpDSy1KNO/XtpkvHgbmO/AyUO8GfXT0QNBXqlkSrNvq hRwYXFdjUe1czGHWz6YaJBFgM/tVxeuCQxSJBWW31YgZJ41vdUUzr+txSxQOSbY4nd63Ep BPYxhQnN2+R1SPxMaGs+GNp0cRLKPzbR2CPESNpJjNYbQsbu+B6pHTTjkMTu/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1704795599; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=z3YdflRZyBSeEL44OapQ6KDY+nqS8MTIvxlJ++jaaM4=; b=AqQDxxML7bnyBQGiO0HsKVPGS/d9uYbMHYrUXJO56rNgpXT+GJ7tv7V9wm1jHwM1T+9MH3 B2xqEdia9xhl3B2PsZVUWFIcc5OGCcT5I6xiKQGuedoq5h4KawDjH6UDmxz5Qzj5+h8qLi bhrfkz2Cwa+aIiJrNb3P3QgSukzdIWqVTME158HhmN9abbVjTjd9wvKbKxptM/Ybv0EDgO Nvk2u7RTlhOdrQp6XxaAzTcVDVlsJwggvZLRd/5k4uBEjLLwFzR6e0R9ZbolAb0jeuWTwz amNljoRoV/eEUOF+EYQZ1Yz7OKtxBFtzdoOKcLk1MWHZkgkcTos0l9x+z3K8Aw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1704795599; a=rsa-sha256; cv=none; b=ga4dDMWQ4805mi2UIGlbib80kHGNpYNsnroy5jtb2Qk11FWg8qfK+Qiv6gwE37nRmBqu1a qK2GwQWuF3nMm5PRivJpJDEWSt7Qv1OGurFo6Cio0pEV8g4BDNJjNqC6fsu43m6sytcOTx 3v0KEBkGsdLd6X5x3iBKVwf05X0/PieMks/vU9uv7JcYN1hTSEQuChnaAX7kJarCzyr9ji IFKDVOup8bUINY6bs/alGDJZORq9ZIxqj/uuKmvCfL92BFO0iUOt0NdGBq7TSL8XDghwAB eWCvm0pJ3vHYQBN+8N7YtE/RCVxQjSLkDOo7jchMC3CKcB0zt5F6lage5SR7xQ== Received: from ravel.localnet (lfbn-nic-1-525-172.w90-118.abo.wanadoo.fr [90.118.140.172]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: olce/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4T8Rkk1qXmzJw2; Tue, 9 Jan 2024 10:19:58 +0000 (UTC) (envelope-from olce@freebsd.org) From: Olivier Certner To: Xin LI , Mike Karels Cc: src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org Subject: Re: git: 2f036705f337 - main - Document the two recent newsyslog(8) change (-c option and configuration option). Date: Tue, 09 Jan 2024 11:19:48 +0100 Message-ID: <2683023.poxlI1A5LX@ravel> In-Reply-To: <90D0905E-AA46-4351-AEE0-9ED9D835DB50@karels.net> References: <202312290846.3BT8kOiO029918@gitrepo.freebsd.org> <90D0905E-AA46-4351-AEE0-9ED9D835DB50@karels.net> List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart6134885.r39cKavRk3"; micalg="pgp-sha384"; protocol="application/pgp-signature" --nextPart6134885.r39cKavRk3 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8"; protected-headers="v1" From: Olivier Certner To: Xin LI , Mike Karels Date: Tue, 09 Jan 2024 11:19:48 +0100 Message-ID: <2683023.poxlI1A5LX@ravel> In-Reply-To: <90D0905E-AA46-4351-AEE0-9ED9D835DB50@karels.net> MIME-Version: 1.0 Hi, Sorry not to have gotten to that earlier. I had initially expressed on IRC that I found the newsyslog changes great, = but now reading Mike's arguments and proposals I have serious doubts on the= current approach. > Sorry not to have noticed this in the review; it was only when I saw this > message that it sunk in that we now have *three* ways to specify compress= ion, > and I'm not even sure what the precedence is. I would have thought that > would replace -c. It's a mess if the config file has entries > that specify J and X flags as well as none, the config file has > zstd, and the -c option is given as well. We now have a knob > to override the knob to override a knob. The only reason to keep -c that > I can think of is to specify a different compression in a single invocati= on, > but as noted, changing compression requires manual operations that make > it unreasonable to change it invocation by invocation. I agree. Two possibilies that I can think of from here: Remove '-c' or mak= e it enable compression regardless of the log files' individual settings. =20 > I still think it would be much better to add an option letter to select > the default compression as specified by . This would eliminate > the need for "legacy", and it would add the ability to have both a global > default and an exception. I think the redefinition of the existing flags > to have different meanings if is given is messy. I didn't think about that at first. I agree. If people want to be able to override compression settings globally, which = I find useful, one could introduce another directive such as taking a boolean to request to apply the option regardless= of the individual compression letters. Another possibility is just to rename "" to ""= (so, this time, not a boolean) and keep its current behavior. This would = match one of the suggestions above about '-c', but then there's the questio= n of which one takes precedence, and I think that the command-line specific= ation should prevail (for practical purposes and POLA). =20 > The entry for -c says that we plan to change the default to "none" in 15.= 0. > Hopefully that would be done via and not -c. However, there > was significant pushback on "none" being the default. I think the default should be "no ", i.e., no directive.= This may plea for having "none" mean "don't change anything" (as if the d= irective wasn't there) and have something else to deactivate compression, s= uch as "no_compression" (which is really an override). If "none" is confus= ing, then just forego it completely, and have 'newsyslog' plain fail on it = (but keep "no_compression" as just described). If there is consensus, I'd then change the 'J' flag currently used for all = log files to the new chosen flag for generic compression, and have set to "bzip2" in a first step (for POLA). Then, it could be c= hanged to something else, e.g., 'zstd'. Setting it to 'none' seems to me the worst solution (but far from being the= end of the world). More deeply, I remember having seen at least two claims that using filesyst= em's compression is better, without arguments. I don't agree with that in = practice. The only advantage of in-filesystem compression, besides the adm= inistrative simplification that you can also get with the override above, i= s to get O(1) random access to big log files, and I don't see any compellin= g and common use case for it. You certainly want to get to the end of the = current log quickly, but that one precisely is not handled by 'newsyslog' a= nd stays uncompressed (at the application level). When you want to search = for strings or patterns, you have to grep the whole file anyway. You may w= ant to immediately reach the end of some historical log file, e.g., when ma= nually going back in time from the current log, but this should have neglig= ible latency, and if it doesn't, than just use more and smaller log archive= s. Same thing if you have a more sophisticated setup with an index of log = text: Jumping to a particular location in the log file should have negligib= le latency, else apply the same recipe. If your setup with index requires = a single, never rotated, log file, then you're not even using 'newsyslog' i= n the first place (or should not). Although I agree that in this case usin= g a compressed filesystem (or a randomly accessible archive) can make sense= (if your index doesn't already cover the results expected from your search= es), I very much doubt this is a common setup. Moreover, using in-filesystem compression can lead to degrading the compres= sion ratio, since the compression method on ZFS is chosen per dataset, whic= h includes a bunch of other files and use cases preventing the administrato= r from choosing the best, and slowest, compression methods. To avoid this = problem, one can use a separate dataset for /var/log (anyone?), but changin= g this on already running systems is a greater burden than just changing th= e compression settings in the 'newsyslog' configuration files. I'd like people who disagree with this to present arguments for their case,= if for nothing else to share their experience and best practices on log ma= nagement. Thanks and regards. =2D-=20 Olivier Certner --nextPart6134885.r39cKavRk3 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part. Content-Transfer-Encoding: 7Bit -----BEGIN PGP SIGNATURE----- iQIzBAABCQAdFiEEmNCxHjkosai0LYIujKEwQJceJicFAmWdHcQACgkQjKEwQJce JifhlA//ehOBZnBNEhdAuixFk+drmEXD5UMclOJZV84mpsZRH/McOZ6dKs1Maq9M 91gvyQI3waZADOWkPkE1iyTywm1GgpvUq7JuQ1QzITOLPl3shQCgn6sxUOCNG4ly wKh/L6w5bvAqaFN8zj/RBpSeXfPADFowZ7tUNPZQlr4sGpMO34fp0KEL29rQQVxI wyPTEepfYDDewhxtOOPwJXs3uwMfS3UHAHlnZv0FjYH/oEPXF2plc5X1mDuw1SGg nFE2kRrfA1kZf8yu4XspLRZ/ADaCHko0xw3DE11gxMElQ/9QKZ3vk1yOwtim8UVN h4maCUVo6QHylbIR8nhDu+VdSBZb2wavoVFom7zfBd5xWgzzzBKRVc48JSMMqXMb aRi8Jk4fDQqZ0heH5abtE8R1sFKxDojYHj57hKJXZvcrs7jEqwdL8N1UT4MHGbyA b9E0iroTdcDD7EGvm7wd3z24bPb9pj5mJ1XOa5CYvPqSlNMpAfxTlu/041is3xSh 82+mT2UckYtVlp71xzDFipFZ5sLdwZD8/JkD6yojzN7CphhsBBf+dbVzDi2KqZzP SR/7xr6+J4cqZlRXIPFL59aNLcwIBBg0hcjBc6T4yiG/qf85fvMbxveQI7tHz18g Gpq4izGlh/UFmXz91BoAn1FV7D/zzELTfrUbynLBsNM69MSz7/w= =PJpJ -----END PGP SIGNATURE----- --nextPart6134885.r39cKavRk3--