From owner-freebsd-pkg@freebsd.org Mon Feb 13 23:00:32 2017 Return-Path: Delivered-To: freebsd-pkg@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B93B9CDD9EC for ; Mon, 13 Feb 2017 23:00:32 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 976BEC5B for ; Mon, 13 Feb 2017 23:00:32 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 93E63CDD9EB; Mon, 13 Feb 2017 23:00:32 +0000 (UTC) Delivered-To: pkg@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 93889CDD9EA for ; Mon, 13 Feb 2017 23:00:32 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-yb0-x231.google.com (mail-yb0-x231.google.com [IPv6:2607:f8b0:4002:c09::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4A857C58; Mon, 13 Feb 2017 23:00:32 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-yb0-x231.google.com with SMTP id 123so31235694ybe.3; Mon, 13 Feb 2017 15:00:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=5NybLPjZu3XktJPP/R8bwEh8OGdviE/1tr6q9HV8oA8=; b=ZjoDOUvIGoeqjkErADN149SweCF8Ihh3hLjxH+j8bYPaH2KZI+5XXyhaSn6/FEm6dj y3+AwvchcSpPG1fogD00kG+t6GcwwcQgD9bs95UvOvZx3zU3wq82//3aGUc5SJle5GQ6 nZFxsv0oqvhYa5+UZB2X6zNXpdOTnXoLdcjpfUWOH8R/xsTqQWRv5HDLgAJzx1URHi2d D79XoPN3HhnXUhjLBhh7BQ5p4ZTvDhEJX6NMTb963n3Tf4c4+UUy0u+YcsEoIkH4kGB7 8By4WFIqRwSopxI0T4rCLu1qAwzWu3pWkSkZRklqqpfLg05I9yHUX4okCr84n8DomPw3 CO5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=5NybLPjZu3XktJPP/R8bwEh8OGdviE/1tr6q9HV8oA8=; b=G7bxGlkVWnyD8LyFfPZea26lez8l7JDLqBkhgxwhh9+5l1Z0QBc9CCvEMlwYICE3K/ LITy3uFRpdUX6EX/ZLbpkAFgB3jV6Pa1gQijj2vPNgFL70hrS53z4utM1QX43LrN4eYP z18+J2QWORl9wGZrXNoOqe25VxAsMPpEC/fktwIfPfwuswNW6379lSUxVtYGlFSKzXxT SbZ7QtuISI6JSFcdEOfFfrrxqAtNynycnZv/ZoD5SNKpqqSBq8wonn6j40eYQ7nT8wCq H0Wkv5FPJV9SqRkdLf/ykN4/xwiJcSUEpTpn8d8TQbE5a43ooBBhOl7LXP3mbA/jR2Sq ZHjQ== X-Gm-Message-State: AMke39mZG71FRYLKF4DQG1gFZurgmblxuu4W6qmzVykE2Y2k+BD/3gqV6uJ6iD7GnuAKf6/rtNYv0nsW9TLX5A== X-Received: by 10.37.171.139 with SMTP id v11mr18321381ybi.25.1487026831395; Mon, 13 Feb 2017 15:00:31 -0800 (PST) MIME-Version: 1.0 Sender: asomers@gmail.com Received: by 10.129.38.133 with HTTP; Mon, 13 Feb 2017 15:00:31 -0800 (PST) In-Reply-To: <201702132233.v1DMXf4P092101@slippy.cwsent.com> References: <201702132233.v1DMXf4P092101@slippy.cwsent.com> From: Alan Somers Date: Mon, 13 Feb 2017 16:00:31 -0700 X-Google-Sender-Auth: r9REFL-buM_tatHR1J1U4G0JUcU Message-ID: Subject: Re: Bug 217055 - Consolidate random sleeps in periodic scripts To: Cy Schubert Cc: scrappy@freebsd.org, Brian Somers , freebsd-bugzilla@ayaken.net, Cy Schubert , pkg@freebsd.org, =?UTF-8?Q?Dag=2DErling_Sm=C3=B8rgrav?= Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-pkg@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Binary package management and package tools discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Feb 2017 23:00:32 -0000 On Mon, Feb 13, 2017 at 3:33 PM, Cy Schubert wrote: > In message om> > , Alan Somers writes: >> On Mon, Feb 13, 2017 at 2:25 PM, Cy Schubert wrote >> : >> > In message > c >> > om> >> > , Alan Somers writes: >> >> On Mon, Feb 13, 2017 at 12:01 AM, Cy Schubert w >> rot >> >> e: >> >> > In message > il. >> >> c >> >> > om> >> >> > , Alan Somers writes: >> >> >> I propose that we remove the various anti-congestion sleeps from >> >> >> different periodic scripts, and add a single anti-congestion sleep to >> >> >> the very beginning. Does this sound like a good idea to all of you? >> >> >> >> >> >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=217055 >> >> > >> >> > I think the problem with the sleeps is simply the sleeps. My original pl >> an >> >> > to put my sleep/fetch in the background was shot down by some who though >> t >> >> > it wasn't simple enough. >> >> > >> >> > Secondly, we don't need sleeps every boot. Ntpd for example only needs a >> >> > sleep twice a year max to fetch a new leapfile so, to have a sleep every >> >> > boot would be annoying. >> >> > >> >> > The best solution to replace sleeps would be to put a list of files:URLs >> >> > into a queue to be fetched by fetcher script which would fetch only need >> ed >> >> > files that boot (or in the case of ntp via periodic.conf twice a year). >> >> > >> >> > A single script with a queue of files to fetch with one anti-congestion >> >> > sleep, preferably in the background. >> >> > >> >> > NTP, btw can (will) use the leapfile in /etc/ntp until a fresher copy is >> >> > fetched. >> >> > >> >> > Let's remove all fetching functions from the various rc scripts and queu >> e >> >> > them up early in a fetcher rc script, preferably in the background if at >> >> > all possible. >> >> > >> >> > >> >> > -- >> >> > Cheers, >> >> > Cy Schubert >> >> > FreeBSD UNIX: Web: http://www.FreeBSD.org >> >> > >> >> > The need of the many outweighs the greed of the few. >> >> >> >> Unfortunately that won't work, Cy. Some scripts may need to >> >> dynamically determine what files to fetch, in a way that we can't do >> >> in a single separate fetcher script. Worse, some scripts, like >> >> 300.statistics from sysutils/bsdstats, need to _post_ a URL, not get >> >> one. >> > >> > Diverse requirements cannot be addressed by one knob. To assume that >> > various applications all have the same sleep requirement won't work. >> >> Can you think any any periodic script whose sleep needs couldn't be >> satisified by a single sleep at the beginning of the periodic run? I >> can't. All sleeps I know of in /etc/periodic and >> /usr/local/etc/periodic are for the purposes of reducing congestion >> spikes on a server somewhere. The only way a single sleep could be >> insufficient is if the random time interval is too small. > > Yes. ntpd. It doesn't need a sleep every time periodic is run. It only > needs a sleep once during the 30 day period prior to leap-second file > expiry. To impose a sleep for ntpd every time periodic is run is a waste of > time. > >> >> > >> > I suppose we could have an optional single sleep script but we can't >> > summarily remove all sleeps and assume all rc and periodic scripts sleep >> > for some, one or possibly no applications requiring a sleep at any given >> > time. >> >> What? I can't make sense of that sentence. > > What I'm saying is if you want a global sleep, have it as an option, not a > replacement that forces a sleep when none is needed. > >> >> > We can have a general sleep but removing the option of others would >> > be counter productive. It doesn't make sense to have an arbitrary sleep >> > just in case a subsequent script might need it. >> >> Nothing in /etc/periodic needs to be run at a precise time, so adding >> a sleep won't hurt anything. And if the sleep is configurable, a >> sysadmin can always disable it. Also, from an anticongestion >> standpoint it's objectively less good to chain multiple sleeps >> together instead of using a single longer sleep. The reason is >> because when you add several uniformly distributed random variables, >> the result approaches a normal distribution with a peak in the middle. >> But for anticongestion purposes, a uniform distribution is really what >> you want. >> >> > If we have to, let's either >> > reduce the length of the sleeps or put better yet background them. >> >> I don't like the idea of backgrounding parts of the periodic scripts, >> for three reasons. One, it's complicated. Two, it prevents >> periodic(8) from sending a single status email. Three, periodic(8) >> might start the next day's run before the previous day's is complete. > > It is more complicated, yes. However a sleep delays sending the email and > agreed you don't want it running the next day. > >> >> > >> > What's motivating this? Server? Laptop? >> >> Servers mostly. A confounding issue is Bug 210188 - periodic daily >> sleeps even when invoked from a terminal . When I run periodic by >> hand, it still sleeps. It would be easier to fix that bug if only one >> sleep were involved instead of several. > > That can be solved by, > if tty >/dev/null 2>&1; then > SLEEP_TIME=0 > else > SLEEP_TIME=some algorithm to randomize sleep > fi > > This could be put in rc.subr or some other common place and sourced by > scripts that need it. Of course this doesn't address ports that issue > sleep. Nor would a common 000 sleep script either. Future ports and porters > will continue to put random sleeps in ports as well. That's a good idea. Currently, no periodic script includes /etc/rc.subr, but we can (and already do) define functions in /etc/defaults/periodic.conf. I can put an anticongest function in there. It can use a /var/run file in conjunction with an environment variable set by periodic(8) to ensure that the sleep happens no more than once for each invocation of periodic(8). Also, I don't think ports is going to be a big problem. Currently I only know of two ports whose periodic scripts include a sleep. One is bsdstat, whose periodic script is entirely contained within the ports tree. The other is pkg, which is also controlled by the FreeBSD project. So it should be straightforward to convert them. > > The above avoids imposing a sleep on every periodic script whether it wants > it or not just in case a periodic script MIGHT just need a sleep. > > I've added des to the cc list. He will be interested in this discussion. > > > -- > Cheers, > Cy Schubert > FreeBSD UNIX: Web: http://www.FreeBSD.org > > The need of the many outweighs the greed of the few. > >