From owner-freebsd-pkg@freebsd.org Mon Feb 13 22:33:52 2017 Return-Path: Delivered-To: freebsd-pkg@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 53E93CDD31A for ; Mon, 13 Feb 2017 22:33:52 +0000 (UTC) (envelope-from cy.schubert@komquats.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 354C81D0A for ; Mon, 13 Feb 2017 22:33:52 +0000 (UTC) (envelope-from cy.schubert@komquats.com) Received: by mailman.ysv.freebsd.org (Postfix) id 31DBDCDD319; Mon, 13 Feb 2017 22:33:52 +0000 (UTC) Delivered-To: pkg@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2FD6DCDD318 for ; Mon, 13 Feb 2017 22:33:52 +0000 (UTC) (envelope-from cy.schubert@komquats.com) Received: from smtp-out-so.shaw.ca (smtp-out-so.shaw.ca [64.59.136.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id B84BF1D08; Mon, 13 Feb 2017 22:33:51 +0000 (UTC) (envelope-from cy.schubert@komquats.com) Received: from spqr.komquats.com ([96.50.22.10]) by shaw.ca with SMTP id dPBlcg1Kpsa1kdPBmcGbrW; Mon, 13 Feb 2017 15:33:44 -0700 X-Authority-Analysis: v=2.2 cv=W+NIbVek c=1 sm=1 tr=0 a=jvE2nwUzI0ECrNeyr98KWA==:117 a=jvE2nwUzI0ECrNeyr98KWA==:17 a=kj9zAlcOel0A:10 a=n2v9WMKugxEA:10 a=BWvPGDcYAAAA:8 a=6I5d2MoRAAAA:8 a=YxBL1-UpAAAA:8 a=rY95N4Q9iyJ0nQcwCmkA:9 a=1DPAgBlFnT_m26bI:21 a=7LHWGA_iQQ2NYw5u:21 a=CjuIK1q_8ugA:10 a=pxhY87DP9d2VeQe4joPk:22 a=IjZwj45LgO3ly-622nXo:22 a=Ia-lj3WSrqcvXOmTRaiG:22 Received: from slippy.cwsent.com (slippy [10.1.1.91]) by spqr.komquats.com (Postfix) with ESMTPS id 85FD5BF4; Mon, 13 Feb 2017 14:33:41 -0800 (PST) Received: from slippy (localhost [127.0.0.1]) by slippy.cwsent.com (8.15.2/8.15.2) with ESMTP id v1DMXf4P092101; Mon, 13 Feb 2017 14:33:41 -0800 (PST) (envelope-from Cy.Schubert@cschubert.com) Message-Id: <201702132233.v1DMXf4P092101@slippy.cwsent.com> X-Mailer: exmh version 2.8.0 04/21/2012 with nmh-1.6 Reply-to: Cy Schubert From: Cy Schubert X-os: FreeBSD X-Sender: cy@cwsent.com X-URL: http://www.cschubert.com/ To: Alan Somers cc: Cy Schubert , scrappy@freebsd.org, Brian Somers , freebsd-bugzilla@ayaken.net, Cy Schubert , pkg@freebsd.org, des@freebsd.org Subject: Re: Bug 217055 - Consolidate random sleeps in periodic scripts In-Reply-To: Message from Alan Somers of "Mon, 13 Feb 2017 15:07:17 -0700." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Mon, 13 Feb 2017 14:33:41 -0800 X-CMAE-Envelope: MS4wfKW+mpQV2MWAI3Y3IV37aqeg1jx47V+eniX2LaT8JNl16TzygwLx2eTliv1qDUx/O8TE0HwDPJwg1plG1372j4hs27Yqsy8M/9Nj7D5XYj7KQFp415gS uM9UvmxmKNYplodB17lOuxMKt7H9MjMdxrd4GIYAUeCohFc61Zw9+b1gXIIZl+OufodSwMUClqstAEI+qKH5a3rLjJMo34MQhXDftaTTy9iQqDKqmayFmuTs oZg0O6t5X/XYrX4Rfg68CNX/pok0gY8kh5oKqEGp2mVzUsnqDyQcFd91GoOx06sgr9fKexba5/rWRm0Wvbqvwd4D3mPztQjkb1a+7k4mpT49Rq8p8cE6W1So 1f78NgH+ X-BeenThere: freebsd-pkg@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Binary package management and package tools discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Feb 2017 22:33:52 -0000 In message , Alan Somers writes: > On Mon, Feb 13, 2017 at 2:25 PM, Cy Schubert wrote > : > > In message c > > om> > > , Alan Somers writes: > >> On Mon, Feb 13, 2017 at 12:01 AM, Cy Schubert w > rot > >> e: > >> > In message il. > >> c > >> > om> > >> > , Alan Somers writes: > >> >> I propose that we remove the various anti-congestion sleeps from > >> >> different periodic scripts, and add a single anti-congestion sleep to > >> >> the very beginning. Does this sound like a good idea to all of you? > >> >> > >> >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=217055 > >> > > >> > I think the problem with the sleeps is simply the sleeps. My original pl > an > >> > to put my sleep/fetch in the background was shot down by some who though > t > >> > it wasn't simple enough. > >> > > >> > Secondly, we don't need sleeps every boot. Ntpd for example only needs a > >> > sleep twice a year max to fetch a new leapfile so, to have a sleep every > >> > boot would be annoying. > >> > > >> > The best solution to replace sleeps would be to put a list of files:URLs > >> > into a queue to be fetched by fetcher script which would fetch only need > ed > >> > files that boot (or in the case of ntp via periodic.conf twice a year). > >> > > >> > A single script with a queue of files to fetch with one anti-congestion > >> > sleep, preferably in the background. > >> > > >> > NTP, btw can (will) use the leapfile in /etc/ntp until a fresher copy is > >> > fetched. > >> > > >> > Let's remove all fetching functions from the various rc scripts and queu > e > >> > them up early in a fetcher rc script, preferably in the background if at > >> > all possible. > >> > > >> > > >> > -- > >> > Cheers, > >> > Cy Schubert > >> > FreeBSD UNIX: Web: http://www.FreeBSD.org > >> > > >> > The need of the many outweighs the greed of the few. > >> > >> Unfortunately that won't work, Cy. Some scripts may need to > >> dynamically determine what files to fetch, in a way that we can't do > >> in a single separate fetcher script. Worse, some scripts, like > >> 300.statistics from sysutils/bsdstats, need to _post_ a URL, not get > >> one. > > > > Diverse requirements cannot be addressed by one knob. To assume that > > various applications all have the same sleep requirement won't work. > > Can you think any any periodic script whose sleep needs couldn't be > satisified by a single sleep at the beginning of the periodic run? I > can't. All sleeps I know of in /etc/periodic and > /usr/local/etc/periodic are for the purposes of reducing congestion > spikes on a server somewhere. The only way a single sleep could be > insufficient is if the random time interval is too small. Yes. ntpd. It doesn't need a sleep every time periodic is run. It only needs a sleep once during the 30 day period prior to leap-second file expiry. To impose a sleep for ntpd every time periodic is run is a waste of time. > > > > > I suppose we could have an optional single sleep script but we can't > > summarily remove all sleeps and assume all rc and periodic scripts sleep > > for some, one or possibly no applications requiring a sleep at any given > > time. > > What? I can't make sense of that sentence. What I'm saying is if you want a global sleep, have it as an option, not a replacement that forces a sleep when none is needed. > > > We can have a general sleep but removing the option of others would > > be counter productive. It doesn't make sense to have an arbitrary sleep > > just in case a subsequent script might need it. > > Nothing in /etc/periodic needs to be run at a precise time, so adding > a sleep won't hurt anything. And if the sleep is configurable, a > sysadmin can always disable it. Also, from an anticongestion > standpoint it's objectively less good to chain multiple sleeps > together instead of using a single longer sleep. The reason is > because when you add several uniformly distributed random variables, > the result approaches a normal distribution with a peak in the middle. > But for anticongestion purposes, a uniform distribution is really what > you want. > > > If we have to, let's either > > reduce the length of the sleeps or put better yet background them. > > I don't like the idea of backgrounding parts of the periodic scripts, > for three reasons. One, it's complicated. Two, it prevents > periodic(8) from sending a single status email. Three, periodic(8) > might start the next day's run before the previous day's is complete. It is more complicated, yes. However a sleep delays sending the email and agreed you don't want it running the next day. > > > > > What's motivating this? Server? Laptop? > > Servers mostly. A confounding issue is Bug 210188 - periodic daily > sleeps even when invoked from a terminal . When I run periodic by > hand, it still sleeps. It would be easier to fix that bug if only one > sleep were involved instead of several. That can be solved by, if tty >/dev/null 2>&1; then SLEEP_TIME=0 else SLEEP_TIME=some algorithm to randomize sleep fi This could be put in rc.subr or some other common place and sourced by scripts that need it. Of course this doesn't address ports that issue sleep. Nor would a common 000 sleep script either. Future ports and porters will continue to put random sleeps in ports as well. The above avoids imposing a sleep on every periodic script whether it wants it or not just in case a periodic script MIGHT just need a sleep. I've added des to the cc list. He will be interested in this discussion. -- Cheers, Cy Schubert FreeBSD UNIX: Web: http://www.FreeBSD.org The need of the many outweighs the greed of the few.