Date: Tue, 9 Jan 2024 09:54:39 -0800 From: Xin LI <delphij@gmail.com> To: Mike Karels <mike@karels.net> Cc: Xin LI <delphij@freebsd.org>, src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org Subject: Re: git: 2f036705f337 - main - Document the two recent newsyslog(8) change (-c option and <compress> configuration option). Message-ID: <CAGMYy3uOjXY5LaVuJkXnwyFq0h7VEesaMGDvJ2i4J6VcNWnR4g@mail.gmail.com> In-Reply-To: <90D0905E-AA46-4351-AEE0-9ED9D835DB50@karels.net> References: <202312290846.3BT8kOiO029918@gitrepo.freebsd.org> <90D0905E-AA46-4351-AEE0-9ED9D835DB50@karels.net>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000d1cc79060e870261 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi, Mike, On Fri, Dec 29, 2023 at 7:25=E2=80=AFAM Mike Karels <mike@karels.net> wrote= : > On 29 Dec 2023, at 2:46, Xin LI wrote: > > > The branch main has been updated by delphij: > [...] > > > +61174ad88e33: > > + newsyslog(8) now supports specifying a global compression method > directly > > + at the beginning of the newsyslog.conf file, which will make > newsyslog(8) > > + to behave like the corresponding option was passed to the newly > added > > + '-c' option. For example: > > + > > + <compress> none > > + > > +906748d208d3: > > + newsyslog(8) now accepts a new option, '-c' which overrides all > historical > > + compression flags by treating their meaning as "treat the file as > compressible" > > + rather than "compress the file with that specific method." > > + > > + The following choices are available: > > + * none: Do not compress, regardless of flag. > > + * legacy: Historical behavior (J=3Dbzip2, X=3Dxz, Y=3Dzstd, Z=3D= gzip). > > + * bzip2, xz, zstd, gzip: apply the specified compression method. > > + > > + We plan to change the default to 'none' in FreeBSD 15.0. > > Sorry not to have noticed this in the review; it was only when I saw this > message that it sunk in that we now have *three* ways to specify > compression, > and I'm not even sure what the precedence is. I would have thought that > The current precedence is that command line takes highest priority; when it's not present, <compress> overrides all lines with any compression method. The meaning of J/X/Y/Z is changed to "this file is compressible", unless newsyslog is in legacy mode. It's intentional to discourage the use of different compression types for different files (because the current newsyslog.conf(5) format uses one character to represent compression method and without a change to e.g. UCL or some other configuration language, supporting additional compression methods would be much more messier). > <compress> would replace -c. It's a mess if the config file has entries > that specify J and X flags as well as none, the config file has > <compress> zstd, and the -c option is given as well. We now have a knob > newsyslog would issue a warning if both -c is given and the configuration file has <compress>. Part of the reason to introduce <compress> was that it would move the configuration to one place (newsyslog.conf) and there were concerns that having -c in newsyslog in /etc/crontab is counterintuitive (as raised in some earlier discussion). > to override the knob to override a knob. The only reason to keep -c that > I can think of is to specify a different compression in a single > invocation, > but as noted, changing compression requires manual operations that make > it unreasonable to change it invocation by invocation. > The intention of -c was to be added to /etc/crontab, but with the introduction of <compress> it's probably a good idea to just remove the command line option or at least leave it undocumented? > Also, using the -c option or <compress> (except with legacy) loses the > ability to specify different compression types for different files. > This (the ability of specifying different compression types for different files) is what I deliberately wanted to get rid of. Compression types only make a meaningful difference when files are sufficiently large, and the purpose of newsyslog is to not allow log files to grow to an unmanageable size. Rotating files when they are large and also requesting to compress them means the system would have prolonged "bursts" of CPU workload at integral hours, which would be even worse for busy systems where logs tend to have more contents. > People have objected to the term "legacy" as well. > I don't agree. Calling it "legacy" would send a clear signal that the historical way is discouraged (for people who are using a different compression method, merging upstream change would be a very painful process). We already have 4 letters to represent 4 different compression methods, that's almost 1/3 of all valid configuration letters, and we should stop doing that because the legacy way is causing problems and won't scale when new compression methods are added. > I still think it would be much better to add an option letter to select > the default compression as specified by <compress>. This would eliminate > the need for "legacy", and it would add the ability to have both a global > default and an exception. I think the redefinition of the existing flags > to have different meanings if <compress> is given is messy. > I insist the default should be anything except "legacy" -- legacy is for backward compatibility and should serve that purpose only. The entry for -c says that we plan to change the default to "none" in 15.0. > Hopefully that would be done via <compress> and not -c. However, there > Yes, my intention was to add a line to newsyslog.conf ( https://reviews.freebsd.org/D43169 ) instead of adding it to /etc/crontab. > was significant pushback on "none" being the default. > I think we should provide a reasonable default. Compression was useful back in the 90's, but not so much nowadays and makes grep'ing the logs harder (and for those who want compression, they could choose e.g. 'bzip2' anyways). This change would also make porter's lives a lot easier as they could install their newsyslog.conf.d files with e.g. 'J' to indicate that the file can be compressed, without messing up with the administrator's decision to not compress or compress it with some other compressor. --000000000000d1cc79060e870261 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><div class=3D"gmail_default" style=3D"fon= t-family:monospace,monospace">Hi, Mike,</div></div><br><div class=3D"gmail_= quote"><div dir=3D"ltr" class=3D"gmail_attr">On Fri, Dec 29, 2023 at 7:25= =E2=80=AFAM Mike Karels <<a href=3D"mailto:mike@karels.net">mike@karels.= net</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"marg= in:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1e= x">On 29 Dec 2023, at 2:46, Xin LI wrote:<br> <br> > The branch main has been updated by delphij:<br> <span class=3D"gmail_default" style=3D"font-family:monospace,monospace">[..= .]</span>=C2=A0</blockquote><blockquote class=3D"gmail_quote" style=3D"marg= in:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1e= x">><br> > +61174ad88e33:<br> > +=C2=A0 =C2=A0 =C2=A0newsyslog(8) now supports specifying a global com= pression method directly<br> > +=C2=A0 =C2=A0 =C2=A0at the beginning of the newsyslog.conf file, whic= h will make newsyslog(8)<br> > +=C2=A0 =C2=A0 =C2=A0to behave like the corresponding option was passe= d to the newly added<br> > +=C2=A0 =C2=A0 =C2=A0'-c' option. For example:<br> > +<br> > +=C2=A0 =C2=A0 =C2=A0<compress> none<br> > +<br> > +906748d208d3:<br> > +=C2=A0 =C2=A0 =C2=A0newsyslog(8) now accepts a new option, '-c= 9; which overrides all historical<br> > +=C2=A0 =C2=A0 =C2=A0compression flags by treating their meaning as &q= uot;treat the file as compressible"<br> > +=C2=A0 =C2=A0 =C2=A0rather than "compress the file with that spe= cific method."<br> > +<br> > +=C2=A0 =C2=A0 =C2=A0The following choices are available:<br> > +=C2=A0 =C2=A0 =C2=A0 * none: Do not compress, regardless of flag.<br> > +=C2=A0 =C2=A0 =C2=A0 * legacy: Historical behavior (J=3Dbzip2, X=3Dxz= , Y=3Dzstd, Z=3Dgzip).<br> > +=C2=A0 =C2=A0 =C2=A0 * bzip2, xz, zstd, gzip: apply the specified com= pression method.<br> > +<br> > +=C2=A0 =C2=A0 =C2=A0We plan to change the default to 'none' i= n FreeBSD 15.0.<br> <br> Sorry not to have noticed this in the review; it was only when I saw this<b= r> message that it sunk in that we now have *three* ways to specify compressio= n,<br> and I'm not even sure what the precedence is.=C2=A0 I would have though= t that<br></blockquote><div><br></div><div><div class=3D"gmail_default" sty= le=3D"font-family:monospace,monospace">The current precedence is that comma= nd line takes highest priority; when it's not present, <compress>= overrides all lines with any compression method.</div><div class=3D"gmail_= default" style=3D"font-family:monospace,monospace"><br></div><div class=3D"= gmail_default" style=3D"font-family:monospace,monospace">The meaning of J/X= /Y/Z is changed to "this file is compressible", unless newsyslog = is in legacy mode.=C2=A0 It's intentional to discourage the use of diff= erent compression types for different files (because the current newsyslog.= conf(5) format uses one character to represent compression method and witho= ut a change to e.g. UCL or some other configuration language, supporting ad= ditional compression methods would be much more messier).</div></div><div>= =C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0= .8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <compress> would replace -c.=C2=A0 It's a mess if the config file= has entries<br> that specify J and X flags as well as none, the config file has<br> <compress> zstd, and the -c option is given as well.=C2=A0 We now hav= e a knob<br></blockquote><div><br></div><div><div class=3D"gmail_default" s= tyle=3D"font-family:monospace,monospace">newsyslog would issue a warning if= both -c is given and the configuration file has <compress>.</div></d= iv><div class=3D"gmail_default" style=3D"font-family:monospace,monospace"><= br></div><div class=3D"gmail_default" style=3D"font-family:monospace,monosp= ace">Part of the reason to introduce <compress> was that it would mov= e the configuration to one place (newsyslog.conf) and there were concerns t= hat having -c in newsyslog in /etc/crontab is counterintuitive=C2=A0(as rai= sed in some earlier discussion).</div><div>=C2=A0</div><blockquote class=3D= "gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(2= 04,204,204);padding-left:1ex"> to override the knob to override a knob. The only reason to keep -c that<br= > I can think of is to specify a different compression in a single invocation= ,<br> but as noted, changing compression requires manual operations that make<br> it unreasonable to change it invocation by invocation.<br></blockquote><div= ><br></div><div><div class=3D"gmail_default" style=3D"font-family:monospace= ,monospace">The intention of -c was to be added to /etc/crontab, but with t= he introduction of <compress> it's probably a good idea to just r= emove the command line option or at least leave it undocumented?<span style= =3D"font-family:Arial,Helvetica,sans-serif">=C2=A0</span></div></div><div>= =C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0= .8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Also, using the -c option or <compress> (except with legacy) loses th= e<br> ability to specify different compression types for different files.<br></bl= ockquote><div><br></div><div><div class=3D"gmail_default" style=3D"font-fam= ily:monospace,monospace">This (the ability of specifying different=C2=A0com= pression types for different files) is what I deliberately wanted to get ri= d of.</div></div><div class=3D"gmail_default" style=3D"font-family:monospac= e,monospace"><br></div><div class=3D"gmail_default" style=3D"font-family:mo= nospace,monospace">Compression types only make a meaningful difference when= files are sufficiently large, and the purpose of newsyslog is to not allow= log files to grow to an unmanageable size.=C2=A0 Rotating files when they = are large and also requesting to compress them means the system would have = prolonged "bursts" of CPU workload at integral hours, which would= be even worse for busy systems where logs tend to have more contents.</div= ><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px= 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> People have objected to the term "legacy" as well.<br></blockquot= e><div><br></div><div><div class=3D"gmail_default" style=3D"font-family:mon= ospace,monospace">I don't agree.</div><div class=3D"gmail_default" styl= e=3D"font-family:monospace,monospace"><br></div><div class=3D"gmail_default= " style=3D"font-family:monospace,monospace">Calling it "legacy" w= ould send a clear signal that the historical way is discouraged (for people= who are using a different compression method, merging upstream change woul= d be a very painful process).=C2=A0 We already have 4 letters to represent = 4 different compression methods, that's almost 1/3 of all valid configu= ration letters, and we should stop doing that because the legacy way is cau= sing problems and won't scale when new compression methods are added.</= div></div><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margi= n:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex= "> I still think it would be much better to add an option letter to select<br> the default compression as specified by <compress>.=C2=A0 This would = eliminate<br> the need for "legacy", and it would add the ability to have both = a global<br> default and an exception.=C2=A0 I think the redefinition of the existing fl= ags<br> to have different meanings if <compress> is given is messy.<br></bloc= kquote><div><br></div><div><div class=3D"gmail_default" style=3D"font-famil= y:monospace,monospace">I insist the default should be anything except "= ;legacy" -- legacy is for backward compatibility and should serve that= purpose only.</div></div><div><br></div><blockquote class=3D"gmail_quote" = style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);pa= dding-left:1ex">The entry for -c says that we plan to change the default to= "none" in 15.0.<br> Hopefully that would be done via <compress> and not -c.=C2=A0 However= , there<br></blockquote><div><br></div><div><div class=3D"gmail_default" st= yle=3D"font-family:monospace,monospace">Yes, my intention was to add a line= to newsyslog.conf ( <a href=3D"https://reviews.freebsd.org/D43169">https:/= /reviews.freebsd.org/D43169</a> ) instead of adding it to /etc/crontab.</di= v></div><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:= 0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> was significant pushback on "none" being the default.<br></blockq= uote><div><br></div><div class=3D"gmail_default" style=3D"font-family:monos= pace,monospace">I think we should provide a reasonable default.=C2=A0 Compr= ession was useful back in the 90's, but not so much nowadays and makes = grep'ing=C2=A0the logs harder (and for those who want compression, they= could choose e.g. 'bzip2' anyways).=C2=A0 This change would also m= ake porter's lives a lot easier as they could install their newsyslog.c= onf.d files with e.g. 'J' to indicate that the file can be compress= ed, without messing up with the administrator's decision to not compres= s or compress it with some other compressor.</div></div></div> --000000000000d1cc79060e870261--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAGMYy3uOjXY5LaVuJkXnwyFq0h7VEesaMGDvJ2i4J6VcNWnR4g>