From owner-freebsd-embedded@freebsd.org Wed Jun 26 21:54:20 2019 Return-Path: Delivered-To: freebsd-embedded@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EEECF15D3231 for ; Wed, 26 Jun 2019 21:54:19 +0000 (UTC) (envelope-from kudzu@tenebras.com) Received: from mail-qt1-x836.google.com (mail-qt1-x836.google.com [IPv6:2607:f8b0:4864:20::836]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5A66F89904 for ; Wed, 26 Jun 2019 21:54:18 +0000 (UTC) (envelope-from kudzu@tenebras.com) Received: by mail-qt1-x836.google.com with SMTP id w40so311581qtk.0 for ; Wed, 26 Jun 2019 14:54:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenebras-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=8aYIJQjVjT7TohBYroHJ56nJZ8HGrQQjSPKvTE3llbc=; b=wL3Tn4JUpPlUK5LF7G1tHwr7ljgZHkZ29SblfenvYDhoZkv7wajZ0GezzlQoIe5h8F DqcDCbvaMxmuSCfvOZgd05XgxFQuQ1TUqqmCuyzzmWDAhu+ZJ8bRaLCo8yFrdf23HvkW R7qTy6HCwazDUQvqhqKHE/TdZieTiTB9WFpw1wBUEOVab5rVdoZPmVfyAQ2gv7l1h+Oz IdXcPJX8k8/H904SinsUmCgg4hPrJcU1aET0/iWQ90NWeq/H++F32BokLKudOCf4uWrs 4f82UIBExrbbfybM/lCt0wSLRN4H7YcFlhROEkvDzvSXUDn4o5yGZFpx2RHYu0eNk9eE zt/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=8aYIJQjVjT7TohBYroHJ56nJZ8HGrQQjSPKvTE3llbc=; b=OPX+v7d+kXLXDDOvCvn1M1yuib9SKhfA8tBfhMO2PjmaS7Qy4VUWZQGWjK9nHymcLJ 05TZjaTVTZevr3Pjq2gbGaILRGHwp8Zws5TkQubbbbkWgSQuYoAXv3PgtlhY1Uy/aVhW ZlRH1ATfb01XKsfZFew6iYbKuWEp+864T/oMvuqZB+TRz3kpkLGuw5SZ+ggUnkLEeN3N bGtgzqcMBQdOCGbz46JqTDsQ2eZNPzvzpHWa2DzzR3wPJQ/yvlnhIS0K9Z+XQLyIT4Fu 8y0FWISpY6ooeDyBaRoUYymzO7wS0pNwCtvGTPfe7Qc+PmFFDvhQOlbr3MGQn6o5e+hW m6yg== X-Gm-Message-State: APjAAAWoEkArfzGGeSXW0Dx1spTPAYILnWB61N+CL7UpeLuJl2m3iz85 UISKh8XD/WWo+ZU91JU2bZjfhQnXRxM1KLPp5OsIB7X+lkVIpg== X-Google-Smtp-Source: APXvYqxbpoSfzou+nCi+NHqdwGRqwB5/QSVM/fPdWljGdeJYOKcXZ/QXstHH9EC5cI/6BQYn/EFUoCxYe3ktYI0H61c= X-Received: by 2002:ac8:3014:: with SMTP id f20mr131815qte.69.1561586057565; Wed, 26 Jun 2019 14:54:17 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Michael Sierchio Date: Wed, 26 Jun 2019 14:53:41 -0700 Message-ID: Subject: Re: Limiting the size of syslogd output files using options in syslog.conf To: Ian Lepore Cc: freebsd-hackers@freebsd.org, freebsd-embedded@freebsd.org X-Rspamd-Queue-Id: 5A66F89904 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=tenebras-com.20150623.gappssmtp.com header.s=20150623 header.b=wL3Tn4JU X-Spamd-Result: default: False [-6.23 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[tenebras-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-embedded@freebsd.org]; DMARC_NA(0.00)[tenebras.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[tenebras-com.20150623.gappssmtp.com:+]; MX_GOOD(-0.01)[alt1.aspmx.l.google.com,aspmx.l.google.com,aspmx2.googlemail.com,alt2.aspmx.l.google.com,aspmx3.googlemail.com]; RCVD_IN_DNSWL_NONE(0.00)[6.3.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.96)[-0.958,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE(-2.97)[ip: (-9.30), ipnet: 2607:f8b0::/32(-3.14), asn: 15169(-2.34), country: US(-0.06)] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Jun 2019 21:54:20 -0000 You can pipe any syslog facility to a program. My /etc/syslog.conf has auth.*;authpriv.* |/usr/local/bin/authlog because I do some processing, but also rate-limiting and alerting, before writing to /var/log/auth.log You could just as easily write auth.*;authpriv.* | fifolog_writer /var/log/auth.fifolog Again, I object to modifying syslogd to add a feature that already exists in the modular framework of pipes and utilities and specialized tools like fifolog. =E2=80=93 M On Wed, Jun 26, 2019 at 7:50 AM Ian Lepore wrote: > On Tue, 2019-06-25 at 16:06 -0700, Michael Sierchio wrote: > > Apologies, but I don't understand the point of this at all. I object > > especially to idea of patching syslogd when EVERYTHING you seek can > > be > > accomplished with appropriate settings in /etc/newsyslog.conf and > > changing > > its entry in /etc/crontab to run more frequently. You can control the > > size > > and number of files there. > > > > You can also do remote syslog. > > > > Do you need to rotate logs more than once per minute? > > > > > > Newsyslog handles expected log growth. I guess you've never had a > hardware device or other problem unexpectedly trigger the writing of > hundreds or thousands of lines of output per minute to syslog? It > happens. You write apps that try to strike a sensible balance between > logging important events and not logging so much it leads to problems, > then something unexpected happens and your carefully crafted logging > starts happening at 200hz and actually contributes to the problem. > > We've been using this code at $work since 2003; it appears in dozens of > different products ranging from lab instruments to national timescale > systems. People using the products experiencing such problems just > want the device to keep working as best it can. They don't want > repeated mysterious reboots because of things like /var filling up. > (In the case of products like lab instruments, they don't really > perceive the thing to be a computer running an OS at all.) > > And most of all, when something does go wrong and they get our support > engineers to look into it, they don't want to hear "We can't figure out > what happened originally to cause this problem because all the logging > got rotated out of existence." Newsyslog does exactly the opposite of > what you want in that situation... it purposely destroys the valuable > information about the original problem while carefully preserving the > useless fallout that follows. > > Over the years we've upstreamed every freebsd customization we've ever > done at $work except this one. The only reason I've been carrying this > one local mod is because it seemed too specialized to be useful to > anyone else. Then yesterday someone working on an embedded-system > product asked me on irc how to solve the problem of unexpected log > spewage, so I offered the solution we've been using. > > -- Ian > > > > > On Tue, Jun 25, 2019 at 3:25 PM Ian Lepore wrote: > > > > > I've posted a review of a small syslogd change which lets you set a > > > limit on the size of syslogd output files in /etc/syslog.conf. The > > > idea is to prevent filling up a filesystem on emmc or sdcard or > > > other > > > small storage device on an embedded system with unexpected logging > > > triggered by some error or failing hardware. > > > > > > If you're interested, please see the review at > > > > > > https://reviews.freebsd.org/D20764 > > > > > > -- Ian > > > > > > _______________________________________________ > > > freebsd-embedded@freebsd.org mailing list > > > https://lists.freebsd.org/mailman/listinfo/freebsd-embedded > > > To unsubscribe, send any mail to " > > > freebsd-embedded-unsubscribe@freebsd.org > > > " > > > > > > > > > --=20 "Well," Brahm=C4=81 said, "even after ten thousand explanations, a fool is = no wiser, but an intelligent person requires only two thousand five hundred." - The Mah=C4=81bh=C4=81rata