Date: Fri, 17 Feb 2017 10:47:23 -0600 From: Dustin Wenz <dustinwenz@ebureau.com> To: Walter Cramer <wfc@mintsol.com> Cc: FreeBSD <freebsd-stable@freebsd.org>, Alan Somers <asomers@freebsd.org> Subject: Re: Jailed periodic daily scripts smashing CPU Message-ID: <7D7C8824-319C-4891-8B35-82F9B461FE5D@ebureau.com> In-Reply-To: <20170216174749.V86669@mulder.mintsol.com> References: <321260F8-95D8-4C21-90B5-FDB0F6FF98F9@ebureau.com> <CAOtMX2hb%2BmoGWZFuUjrhO-A6NPg5zRsgs3QDRA2Ja1eATwUCsQ@mail.gmail.com> <46236914-07E5-4128-947E-E2EBD04542BE@ebureau.com> <20170216174749.V86669@mulder.mintsol.com>
next in thread | previous in thread | raw e-mail | index | archive | help
If I needed to go down the road of spreading out the daily maintenance = over a longer period of time, I'd probably use your solution of building = in a delay based on the jail id, or possibly Guy Tabrar's extreme jitter = example. The pkg-backup job appears somewhat flawed, in my view. The xz = compressor is totally CPU bound for a moderate number of jails . Sure, = running it at low priority isn't as cache friendly, but it's certainly = better then the host becoming unresponsive. The other issue is that in = most cases the package archive doesn't change, and so running a daily = backup without checking for changes is wasteful. In my case, ZFS handles = all the rolling backups that I could ever need, so using pkg-backup for = that purpose is redundant. - .Dustin > On Feb 16, 2017, at 5:44 PM, Walter Cramer <wfc@mintsol.com> wrote: >=20 > Adding something like: >=20 > 'sleep $(( $(sysctl -n security.jail.param.jid) * 15 )) && ' >=20 > in front of more resource-intensive commands in /etc/crontab can = reliably spread out the load from a larger number of jails. >=20 > (But if you start and stop jails frequently enough to spread out the = current list of jail id numbers, this method degrades.) >=20 > Low priority for 'periodic daily' jobs might not help much, due to = disk saturation, CPU cache thrashing, etc. > -Walter >=20 > On Thu, 16 Feb 2017, Dustin Wenz wrote: >=20 >> The biggest offender that I see is = /usr/local/etc/periodic/daily/411.pkg-backup >>=20 >> During the high CPU event, my process list contains hundreds of = these: >>=20 >> 83811 - RJ 0:03.42 xz -c >> 83816 - S 0:00.02 /usr/local/sbin/pkg shell .dump >> 83818 - SJ 0:00.02 /usr/local/sbin/pkg shell .dump >> 83820 - SJ 0:00.03 /usr/local/sbin/pkg shell .dump >> 83824 - RJ 0:03.41 xz -c >> 83831 - RJ 0:03.58 xz -c >>=20 >> I could probably get away with disabling that in my case. >>=20 >> However, instead of jitter, I think I'd prefer if the periodic jobs = ran at a lower priority than my user processes. Either through nice, or = idprio. I want them to get done as fast as possible, but I don't want = them to affect my application. >>=20 >> - .Dustin >>=20 >>=20 >>> On Feb 16, 2017, at 4:20 PM, Alan Somers <asomers@freebsd.org> = wrote: >>>=20 >>> Is the problem caused by newsyslog or by the periodic scripts? >>> Newsyslog normally runs from cron directly, not through periodic. = In >>> any case, here are a few suggestions: >>> 1) Turn on cron jitter, as you suggested. Even if 60s isn't enough, >>> it can't hurt. >>> 2) Try gz compression instead of xz compression to see if it's = faster >>> 3) Manually edit the jails' /etc/crontab files to hardcode some >>> variability into their newsyslog and/or periodic run times >>> 4) If the problem is actually being caused by periodic instead of >>> newsyslog, tell me which script it is and how much jitter you want. = I >>> am coincidentally changing how periodic manages jitter right now. >>>=20 >>> -Alan >>>=20 >>> On Thu, Feb 16, 2017 at 2:47 PM, Dustin Wenz = <dustinwenz@ebureau.com> wrote: >>>> I have a number of servers with roughly 60 jails running on each of = them. On these hosts, I've had to disable the periodic security scans = due to overly high disk load when they run (which is redundant in jails = anyway). However, I still have an issue at 3:01am where the CPU is = consumed by dozens of 'xz -c' processes. This is apparently daily log = rolling, which I can't exactly disable. >>>>=20 >>>> The effect is that our processing applications experience a major = slowdown for about 15 seconds every morning, which is just enough that = it's starting to get people's attention. >>>>=20 >>>> What is the best way to mitigate this? I'm aware of the cron jitter = feature, but I'm not sure of the 60-second jitter maximum would be = enough (especially if I wanted to start utilizing more jails). >>>>=20 >>>> - .Dustin >>>> _______________________________________________ >>>> freebsd-stable@freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>>> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >>=20 >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >=20 > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7D7C8824-319C-4891-8B35-82F9B461FE5D>