From nobody Tue Jan 9 17:54:39 2024 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T8dqg2dq6z564Tc; Tue, 9 Jan 2024 17:54:55 +0000 (UTC) (envelope-from delphij@gmail.com) Received: from mail-lf1-x12a.google.com (mail-lf1-x12a.google.com [IPv6:2a00:1450:4864:20::12a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T8dqf3RkXz4PDb; Tue, 9 Jan 2024 17:54:54 +0000 (UTC) (envelope-from delphij@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=lRPldvO7; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of delphij@gmail.com designates 2a00:1450:4864:20::12a as permitted sender) smtp.mailfrom=delphij@gmail.com Received: by mail-lf1-x12a.google.com with SMTP id 2adb3069b0e04-50e759ece35so3507777e87.3; Tue, 09 Jan 2024 09:54:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704822891; x=1705427691; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=5mZsy/JTUe+sTCTNaF1cZMd04lfgT7jHMhpTfTfIa30=; b=lRPldvO7m7q4P0yV0Btujq6v8aSDEImgf5gJIuwq1pch8XHKIn1+8oYfxQUOLL3yQY pL13onNo4+siUPV70vxpenTlWmKu2vVYRMe9DAFVDVumsM/bgDSvEmW6E9VhFzb1Ffsc 7dgx5iAbw8oHtzpqitde7sMtk3Nt25DiYKl7XG5LIBDhsh/4Vu/hinALMmiW1GBN1bzb 019//eMkQlHkPOyljUPPIoPkZ0yRXwWZxqKoXJuMu0EOQ+zTWtftOF6w0SogXXfujIOY +muwNuz4Y63CCvEyPirynWKcW62Nfn1tX/OPz0Yy5YMLnKq4RmX5DL/L7hSLkLfpZD3u TSqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704822891; x=1705427691; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=5mZsy/JTUe+sTCTNaF1cZMd04lfgT7jHMhpTfTfIa30=; b=mfMhAxe18iDCHxU/8oQS+/Ly5rPdiSWMWA4GM+tzUtMahU5WjhtvVIAyedRuhNcWiN /PxcdoE/glUNODN8XIjaHOb6N1Kfb7QmK7rO6NXmsgRjBjGHa1KJOpHH/q8/ByWihjom uMwZuLspbyCvOto44wKuI1vjxrc1GzEt0Atl2u3pNiaRGUdNIsFqtP9Kgm2IHiysXCzz AThU5jQ/BnUQClIa5omcotZAHjO9i2JnSJCQFL04vPfjswdC5wKDrh2KN8xhfYFJ1rC1 3vgvQOIbuxvtDEJCPwku+xut16J2dVwI8Tt0MRn54SvkyXE0lNdo7RU2xNE8+gj1OoKs kNGQ== X-Gm-Message-State: AOJu0YyJxMr+aVLepMdCPB3Ba68vE1Im/ldJGF1OJPgyGniij1jYO8ND j+mLtNxa47PI+kUEGEWB05c318saVBFtSmoT8V+DynzrPFE= X-Google-Smtp-Source: AGHT+IG8aXxgrVvl2Wd1LwlER1KlD2iwoAucSXli1V7r6SC4sKK5TUyHgSLM1viGlY9PskjVeB8giFoi+FuQEy6HwV4= X-Received: by 2002:a05:6512:312d:b0:50e:7224:ad4f with SMTP id p13-20020a056512312d00b0050e7224ad4fmr1940673lfd.59.1704822890939; Tue, 09 Jan 2024 09:54:50 -0800 (PST) List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 References: <202312290846.3BT8kOiO029918@gitrepo.freebsd.org> <90D0905E-AA46-4351-AEE0-9ED9D835DB50@karels.net> In-Reply-To: <90D0905E-AA46-4351-AEE0-9ED9D835DB50@karels.net> From: Xin LI Date: Tue, 9 Jan 2024 09:54:39 -0800 Message-ID: Subject: Re: git: 2f036705f337 - main - Document the two recent newsyslog(8) change (-c option and configuration option). To: Mike Karels Cc: Xin LI , src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org Content-Type: multipart/alternative; boundary="000000000000d1cc79060e870261" X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.95 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.95)[-0.954]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::12a:from]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TO_DN_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MLMMJ_DEST(0.00)[dev-commits-src-all@freebsd.org,dev-commits-src-main@freebsd.org]; FREEFALL_USER(0.00)[delphij]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCPT_COUNT_FIVE(0.00)[5]; FREEMAIL_ENVFROM(0.00)[gmail.com] X-Rspamd-Queue-Id: 4T8dqf3RkXz4PDb --000000000000d1cc79060e870261 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi, Mike, On Fri, Dec 29, 2023 at 7:25=E2=80=AFAM Mike Karels wrote= : > On 29 Dec 2023, at 2:46, Xin LI wrote: > > > The branch main has been updated by delphij: > [...] > > > +61174ad88e33: > > + newsyslog(8) now supports specifying a global compression method > directly > > + at the beginning of the newsyslog.conf file, which will make > newsyslog(8) > > + to behave like the corresponding option was passed to the newly > added > > + '-c' option. For example: > > + > > + none > > + > > +906748d208d3: > > + newsyslog(8) now accepts a new option, '-c' which overrides all > historical > > + compression flags by treating their meaning as "treat the file as > compressible" > > + rather than "compress the file with that specific method." > > + > > + The following choices are available: > > + * none: Do not compress, regardless of flag. > > + * legacy: Historical behavior (J=3Dbzip2, X=3Dxz, Y=3Dzstd, Z=3D= gzip). > > + * bzip2, xz, zstd, gzip: apply the specified compression method. > > + > > + We plan to change the default to 'none' in FreeBSD 15.0. > > Sorry not to have noticed this in the review; it was only when I saw this > message that it sunk in that we now have *three* ways to specify > compression, > and I'm not even sure what the precedence is. I would have thought that > The current precedence is that command line takes highest priority; when it's not present, overrides all lines with any compression method. The meaning of J/X/Y/Z is changed to "this file is compressible", unless newsyslog is in legacy mode. It's intentional to discourage the use of different compression types for different files (because the current newsyslog.conf(5) format uses one character to represent compression method and without a change to e.g. UCL or some other configuration language, supporting additional compression methods would be much more messier). > would replace -c. It's a mess if the config file has entries > that specify J and X flags as well as none, the config file has > zstd, and the -c option is given as well. We now have a knob > newsyslog would issue a warning if both -c is given and the configuration file has . Part of the reason to introduce was that it would move the configuration to one place (newsyslog.conf) and there were concerns that having -c in newsyslog in /etc/crontab is counterintuitive (as raised in some earlier discussion). > to override the knob to override a knob. The only reason to keep -c that > I can think of is to specify a different compression in a single > invocation, > but as noted, changing compression requires manual operations that make > it unreasonable to change it invocation by invocation. > The intention of -c was to be added to /etc/crontab, but with the introduction of it's probably a good idea to just remove the command line option or at least leave it undocumented? > Also, using the -c option or (except with legacy) loses the > ability to specify different compression types for different files. > This (the ability of specifying different compression types for different files) is what I deliberately wanted to get rid of. Compression types only make a meaningful difference when files are sufficiently large, and the purpose of newsyslog is to not allow log files to grow to an unmanageable size. Rotating files when they are large and also requesting to compress them means the system would have prolonged "bursts" of CPU workload at integral hours, which would be even worse for busy systems where logs tend to have more contents. > People have objected to the term "legacy" as well. > I don't agree. Calling it "legacy" would send a clear signal that the historical way is discouraged (for people who are using a different compression method, merging upstream change would be a very painful process). We already have 4 letters to represent 4 different compression methods, that's almost 1/3 of all valid configuration letters, and we should stop doing that because the legacy way is causing problems and won't scale when new compression methods are added. > I still think it would be much better to add an option letter to select > the default compression as specified by . This would eliminate > the need for "legacy", and it would add the ability to have both a global > default and an exception. I think the redefinition of the existing flags > to have different meanings if is given is messy. > I insist the default should be anything except "legacy" -- legacy is for backward compatibility and should serve that purpose only. The entry for -c says that we plan to change the default to "none" in 15.0. > Hopefully that would be done via and not -c. However, there > Yes, my intention was to add a line to newsyslog.conf ( https://reviews.freebsd.org/D43169 ) instead of adding it to /etc/crontab. > was significant pushback on "none" being the default. > I think we should provide a reasonable default. Compression was useful back in the 90's, but not so much nowadays and makes grep'ing the logs harder (and for those who want compression, they could choose e.g. 'bzip2' anyways). This change would also make porter's lives a lot easier as they could install their newsyslog.conf.d files with e.g. 'J' to indicate that the file can be compressed, without messing up with the administrator's decision to not compress or compress it with some other compressor. --000000000000d1cc79060e870261 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi, Mike,

On Fri, Dec 29, 2023 at 7:25= =E2=80=AFAM Mike Karels <mike@karels.= net> wrote:
On 29 Dec 2023, at 2:46, Xin LI wrote:

> The branch main has been updated by delphij:
[..= .]=C2=A0
>
> +61174ad88e33:
> +=C2=A0 =C2=A0 =C2=A0newsyslog(8) now supports specifying a global com= pression method directly
> +=C2=A0 =C2=A0 =C2=A0at the beginning of the newsyslog.conf file, whic= h will make newsyslog(8)
> +=C2=A0 =C2=A0 =C2=A0to behave like the corresponding option was passe= d to the newly added
> +=C2=A0 =C2=A0 =C2=A0'-c' option. For example:
> +
> +=C2=A0 =C2=A0 =C2=A0<compress> none
> +
> +906748d208d3:
> +=C2=A0 =C2=A0 =C2=A0newsyslog(8) now accepts a new option, '-c= 9; which overrides all historical
> +=C2=A0 =C2=A0 =C2=A0compression flags by treating their meaning as &q= uot;treat the file as compressible"
> +=C2=A0 =C2=A0 =C2=A0rather than "compress the file with that spe= cific method."
> +
> +=C2=A0 =C2=A0 =C2=A0The following choices are available:
> +=C2=A0 =C2=A0 =C2=A0 * none: Do not compress, regardless of flag.
> +=C2=A0 =C2=A0 =C2=A0 * legacy: Historical behavior (J=3Dbzip2, X=3Dxz= , Y=3Dzstd, Z=3Dgzip).
> +=C2=A0 =C2=A0 =C2=A0 * bzip2, xz, zstd, gzip: apply the specified com= pression method.
> +
> +=C2=A0 =C2=A0 =C2=A0We plan to change the default to 'none' i= n FreeBSD 15.0.

Sorry not to have noticed this in the review; it was only when I saw this message that it sunk in that we now have *three* ways to specify compressio= n,
and I'm not even sure what the precedence is.=C2=A0 I would have though= t that

The current precedence is that comma= nd line takes highest priority; when it's not present, <compress>= overrides all lines with any compression method.

The meaning of J/X= /Y/Z is changed to "this file is compressible", unless newsyslog = is in legacy mode.=C2=A0 It's intentional to discourage the use of diff= erent compression types for different files (because the current newsyslog.= conf(5) format uses one character to represent compression method and witho= ut a change to e.g. UCL or some other configuration language, supporting ad= ditional compression methods would be much more messier).
= =C2=A0
<compress> would replace -c.=C2=A0 It's a mess if the config file= has entries
that specify J and X flags as well as none, the config file has
<compress> zstd, and the -c option is given as well.=C2=A0 We now hav= e a knob

newsyslog would issue a warning if= both -c is given and the configuration file has <compress>.
<= br>
Part of the reason to introduce <compress> was that it would mov= e the configuration to one place (newsyslog.conf) and there were concerns t= hat having -c in newsyslog in /etc/crontab is counterintuitive=C2=A0(as rai= sed in some earlier discussion).
=C2=A0
to override the knob to override a knob. The only reason to keep -c that I can think of is to specify a different compression in a single invocation= ,
but as noted, changing compression requires manual operations that make
it unreasonable to change it invocation by invocation.

The intention of -c was to be added to /etc/crontab, but with t= he introduction of <compress> it's probably a good idea to just r= emove the command line option or at least leave it undocumented?=C2=A0
= =C2=A0
Also, using the -c option or <compress> (except with legacy) loses th= e
ability to specify different compression types for different files.

This (the ability of specifying different=C2=A0com= pression types for different files) is what I deliberately wanted to get ri= d of.

Compression types only make a meaningful difference when= files are sufficiently large, and the purpose of newsyslog is to not allow= log files to grow to an unmanageable size.=C2=A0 Rotating files when they = are large and also requesting to compress them means the system would have = prolonged "bursts" of CPU workload at integral hours, which would= be even worse for busy systems where logs tend to have more contents.
=C2=A0
People have objected to the term "legacy" as well.

I don't agree.

Calling it "legacy" w= ould send a clear signal that the historical way is discouraged (for people= who are using a different compression method, merging upstream change woul= d be a very painful process).=C2=A0 We already have 4 letters to represent = 4 different compression methods, that's almost 1/3 of all valid configu= ration letters, and we should stop doing that because the legacy way is cau= sing problems and won't scale when new compression methods are added.
=C2=A0
I insist the default should be anything except "= ;legacy" -- legacy is for backward compatibility and should serve that= purpose only.

The entry for -c says that we plan to change the default to= "none" in 15.0.
Hopefully that would be done via <compress> and not -c.=C2=A0 However= , there

Yes, my intention was to add a line= to newsyslog.conf ( https:/= /reviews.freebsd.org/D43169 ) instead of adding it to /etc/crontab.
=C2=A0
was significant pushback on "none" being the default.

I think we should provide a reasonable default.=C2=A0 Compr= ession was useful back in the 90's, but not so much nowadays and makes = grep'ing=C2=A0the logs harder (and for those who want compression, they= could choose e.g. 'bzip2' anyways).=C2=A0 This change would also m= ake porter's lives a lot easier as they could install their newsyslog.c= onf.d files with e.g. 'J' to indicate that the file can be compress= ed, without messing up with the administrator's decision to not compres= s or compress it with some other compressor.
--000000000000d1cc79060e870261--