From owner-freebsd-pkg@freebsd.org Tue Feb 14 06:47:37 2017 Return-Path: Delivered-To: freebsd-pkg@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BAF94CD1DCC for ; Tue, 14 Feb 2017 06:47:37 +0000 (UTC) (envelope-from cy.schubert@komquats.com) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 9C0D41EA3 for ; Tue, 14 Feb 2017 06:47:37 +0000 (UTC) (envelope-from cy.schubert@komquats.com) Received: by mailman.ysv.freebsd.org (Postfix) id 9B791CD1DC9; Tue, 14 Feb 2017 06:47:37 +0000 (UTC) Delivered-To: pkg@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9B257CD1DC8 for ; Tue, 14 Feb 2017 06:47:37 +0000 (UTC) (envelope-from cy.schubert@komquats.com) Received: from smtp-out-so.shaw.ca (smtp-out-so.shaw.ca [64.59.136.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 3B4161EA1; Tue, 14 Feb 2017 06:47:36 +0000 (UTC) (envelope-from cy.schubert@komquats.com) Received: from spqr.komquats.com ([96.50.22.10]) by shaw.ca with SMTP id dWtgcUINZC3JIdWthc1Dm7; Mon, 13 Feb 2017 23:47:35 -0700 X-Authority-Analysis: v=2.2 cv=XbT59Mx5 c=1 sm=1 tr=0 a=jvE2nwUzI0ECrNeyr98KWA==:117 a=jvE2nwUzI0ECrNeyr98KWA==:17 a=kj9zAlcOel0A:10 a=n2v9WMKugxEA:10 a=BWvPGDcYAAAA:8 a=6I5d2MoRAAAA:8 a=YxBL1-UpAAAA:8 a=7ObVzcR8vaCAqjd2oWsA:9 a=n-FW0LE3nq8Ec3Nx:21 a=a41V20e25Bq65R66:21 a=CjuIK1q_8ugA:10 a=pxhY87DP9d2VeQe4joPk:22 a=IjZwj45LgO3ly-622nXo:22 a=Ia-lj3WSrqcvXOmTRaiG:22 Received: from slippy.cwsent.com (slippy [10.1.1.91]) by spqr.komquats.com (Postfix) with ESMTPS id EA348AB7; Mon, 13 Feb 2017 22:47:31 -0800 (PST) Received: from slippy (localhost [127.0.0.1]) by slippy.cwsent.com (8.15.2/8.15.2) with ESMTP id v1E6lVd8078946; Mon, 13 Feb 2017 22:47:31 -0800 (PST) (envelope-from Cy.Schubert@cschubert.com) Message-Id: <201702140647.v1E6lVd8078946@slippy.cwsent.com> X-Mailer: exmh version 2.8.0 04/21/2012 with nmh-1.6 Reply-to: Cy Schubert From: Cy Schubert X-os: FreeBSD X-Sender: cy@cwsent.com X-URL: http://www.cschubert.com/ To: Alan Somers cc: Cy Schubert , scrappy@freebsd.org, Brian Somers , freebsd-bugzilla@ayaken.net, Cy Schubert , pkg@freebsd.org, =?UTF-8?Q?Dag=2DErling_Sm=C3=B8rgrav?= Subject: Re: Bug 217055 - Consolidate random sleeps in periodic scripts In-Reply-To: Message from Alan Somers of "Mon, 13 Feb 2017 16:00:31 -0700." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Mon, 13 Feb 2017 22:47:31 -0800 X-CMAE-Envelope: MS4wfA99TX4z/7L/d1CSs1BIXqfIly8fmUeBwvXs/IPG82XHyCh08KkbYrylJajefSpoz2zoqSM6s4+c9b8WOztp1Y4ZYMPWSWf6sW8l4SdbbdLyKLLufxH/ SYDLDl9sBO3MpOq7QXVh3MGqOb2PGA2bW4ZhPCuIKCgoLCqavfMIt1YSvEg9a+jnrkNUNIjbe4btNZl1n3iSDPjc+zV+2VbsrZgZ/WW+VFlSHJxWfgGnfM5R RvI9c7DDlOZO+hd6YymAbtLftqD9mCSx4UelKaWo04Mv8O8ioAuX2nx98y6DwndfG6xCshEI9tAKnUskzPkx/OB2Y1Qg33LyP+bEW+m0AeSGT2VfhZeB63Av cXFEDeJs X-BeenThere: freebsd-pkg@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Binary package management and package tools discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Feb 2017 06:47:37 -0000 In message , Alan Somers writes: > On Mon, Feb 13, 2017 at 3:33 PM, Cy Schubert wrote > : > > In message c > > om> > > , Alan Somers writes: > >> On Mon, Feb 13, 2017 at 2:25 PM, Cy Schubert wr > ote > >> : > >> > In message il. > >> c > >> > om> > >> > , Alan Somers writes: > >> >> On Mon, Feb 13, 2017 at 12:01 AM, Cy Schubert > w > >> rot > >> >> e: > >> >> > In message gma > >> il. > >> >> c > >> >> > om> > >> >> > , Alan Somers writes: > >> >> >> I propose that we remove the various anti-congestion sleeps from > >> >> >> different periodic scripts, and add a single anti-congestion sleep t > o > >> >> >> the very beginning. Does this sound like a good idea to all of you? > >> >> >> > >> >> >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=217055 > >> >> > > >> >> > I think the problem with the sleeps is simply the sleeps. My original > pl > >> an > >> >> > to put my sleep/fetch in the background was shot down by some who tho > ugh > >> t > >> >> > it wasn't simple enough. > >> >> > > >> >> > Secondly, we don't need sleeps every boot. Ntpd for example only need > s a > >> >> > sleep twice a year max to fetch a new leapfile so, to have a sleep ev > ery > >> >> > boot would be annoying. > >> >> > > >> >> > The best solution to replace sleeps would be to put a list of files:U > RLs > >> >> > into a queue to be fetched by fetcher script which would fetch only n > eed > >> ed > >> >> > files that boot (or in the case of ntp via periodic.conf twice a year > ). > >> >> > > >> >> > A single script with a queue of files to fetch with one anti-congesti > on > >> >> > sleep, preferably in the background. > >> >> > > >> >> > NTP, btw can (will) use the leapfile in /etc/ntp until a fresher copy > is > >> >> > fetched. > >> >> > > >> >> > Let's remove all fetching functions from the various rc scripts and q > ueu > >> e > >> >> > them up early in a fetcher rc script, preferably in the background if > at > >> >> > all possible. > >> >> > > >> >> > > >> >> > -- > >> >> > Cheers, > >> >> > Cy Schubert > >> >> > FreeBSD UNIX: Web: http://www.FreeBSD.org > >> >> > > >> >> > The need of the many outweighs the greed of the few. > >> >> > >> >> Unfortunately that won't work, Cy. Some scripts may need to > >> >> dynamically determine what files to fetch, in a way that we can't do > >> >> in a single separate fetcher script. Worse, some scripts, like > >> >> 300.statistics from sysutils/bsdstats, need to _post_ a URL, not get > >> >> one. > >> > > >> > Diverse requirements cannot be addressed by one knob. To assume that > >> > various applications all have the same sleep requirement won't work. > >> > >> Can you think any any periodic script whose sleep needs couldn't be > >> satisified by a single sleep at the beginning of the periodic run? I > >> can't. All sleeps I know of in /etc/periodic and > >> /usr/local/etc/periodic are for the purposes of reducing congestion > >> spikes on a server somewhere. The only way a single sleep could be > >> insufficient is if the random time interval is too small. > > > > Yes. ntpd. It doesn't need a sleep every time periodic is run. It only > > needs a sleep once during the 30 day period prior to leap-second file > > expiry. To impose a sleep for ntpd every time periodic is run is a waste of > > time. > > > >> > >> > > >> > I suppose we could have an optional single sleep script but we can't > >> > summarily remove all sleeps and assume all rc and periodic scripts sleep > >> > for some, one or possibly no applications requiring a sleep at any given > >> > time. > >> > >> What? I can't make sense of that sentence. > > > > What I'm saying is if you want a global sleep, have it as an option, not a > > replacement that forces a sleep when none is needed. > > > >> > >> > We can have a general sleep but removing the option of others would > >> > be counter productive. It doesn't make sense to have an arbitrary sleep > >> > just in case a subsequent script might need it. > >> > >> Nothing in /etc/periodic needs to be run at a precise time, so adding > >> a sleep won't hurt anything. And if the sleep is configurable, a > >> sysadmin can always disable it. Also, from an anticongestion > >> standpoint it's objectively less good to chain multiple sleeps > >> together instead of using a single longer sleep. The reason is > >> because when you add several uniformly distributed random variables, > >> the result approaches a normal distribution with a peak in the middle. > >> But for anticongestion purposes, a uniform distribution is really what > >> you want. > >> > >> > If we have to, let's either > >> > reduce the length of the sleeps or put better yet background them. > >> > >> I don't like the idea of backgrounding parts of the periodic scripts, > >> for three reasons. One, it's complicated. Two, it prevents > >> periodic(8) from sending a single status email. Three, periodic(8) > >> might start the next day's run before the previous day's is complete. > > > > It is more complicated, yes. However a sleep delays sending the email and > > agreed you don't want it running the next day. > > > >> > >> > > >> > What's motivating this? Server? Laptop? > >> > >> Servers mostly. A confounding issue is Bug 210188 - periodic daily > >> sleeps even when invoked from a terminal . When I run periodic by > >> hand, it still sleeps. It would be easier to fix that bug if only one > >> sleep were involved instead of several. > > > > That can be solved by, > > if tty >/dev/null 2>&1; then > > SLEEP_TIME=0 > > else > > SLEEP_TIME=some algorithm to randomize sleep > > fi > > > > This could be put in rc.subr or some other common place and sourced by > > scripts that need it. Of course this doesn't address ports that issue > > sleep. Nor would a common 000 sleep script either. Future ports and porters > > will continue to put random sleeps in ports as well. > > That's a good idea. Currently, no periodic script includes > /etc/rc.subr, but we can (and already do) define functions in > /etc/defaults/periodic.conf. I can put an anticongest function in > there. It can use a /var/run file in conjunction with an environment > variable set by periodic(8) to ensure that the sleep happens no more > than once for each invocation of periodic(8). > > Also, I don't think ports is going to be a big problem. Currently I > only know of two ports whose periodic scripts include a sleep. One is > bsdstat, whose periodic script is entirely contained within the ports > tree. The other is pkg, which is also controlled by the FreeBSD > project. So it should be straightforward to convert them. Hi Alan, Looking at ntp, it backgrounds itself: (sleep $(jot -r 1 0 3600); service ntpd onefetch) & To facilitate debugging, I can commit the following, if you don't mind. Index: periodic/daily/480.leapfile-ntpd =================================================================== --- periodic/daily/480.leapfile-ntpd (revision 313710) +++ periodic/daily/480.leapfile-ntpd (working copy) @@ -13,6 +13,7 @@ case "$daily_ntpd_leapfile_enable" in [Yy][Ee][Ss]) + tty >/dev/null 2>&1 && daily_ntpd_avoid_congestion=NO case "$daily_ntpd_avoid_congestion" in [Yy][Ee][Ss]) # Avoid dogpiling -- Cheers, Cy Schubert FreeBSD UNIX: Web: http://www.FreeBSD.org The need of the many outweighs the greed of the few.