Date: Sat, 18 Feb 2017 15:35:23 +0000 From: Gary Palmer <gpalmer@freebsd.org> To: Dustin Wenz <dustinwenz@ebureau.com> Cc: Walter Cramer <wfc@mintsol.com>, FreeBSD <freebsd-stable@freebsd.org>, Alan Somers <asomers@freebsd.org> Subject: Re: Jailed periodic daily scripts smashing CPU Message-ID: <20170218153523.GA51196@in-addr.com> In-Reply-To: <7D7C8824-319C-4891-8B35-82F9B461FE5D@ebureau.com> References: <321260F8-95D8-4C21-90B5-FDB0F6FF98F9@ebureau.com> <CAOtMX2hb%2BmoGWZFuUjrhO-A6NPg5zRsgs3QDRA2Ja1eATwUCsQ@mail.gmail.com> <46236914-07E5-4128-947E-E2EBD04542BE@ebureau.com> <20170216174749.V86669@mulder.mintsol.com> <7D7C8824-319C-4891-8B35-82F9B461FE5D@ebureau.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Feb 17, 2017 at 10:47:23AM -0600, Dustin Wenz wrote: > If I needed to go down the road of spreading out the daily maintenance over a longer period of time, I'd probably use your solution of building in a delay based on the jail id, or possibly Guy Tabrar's extreme jitter example. > > The pkg-backup job appears somewhat flawed, in my view. The xz compressor is totally CPU bound for a moderate number of jails . Sure, running it at low priority isn't as cache friendly, but it's certainly better then the host becoming unresponsive. The other issue is that in most cases the package archive doesn't change, and so running a daily backup without checking for changes is wasteful. In my case, ZFS handles all the rolling backups that I could ever need, so using pkg-backup for that purpose is redundant. > Hi, At least in my copy of the /usr/local/etc/periodic/daily/411.pkg-backup script, there is an option to enable backing up the pkg db in the jails from the host OS. If you put daily_backup_pkgng_enable=NO into the periodic.conf in the jails and daily_backup_pkg_jails=YES in the host there should be no more than a single thread consuming CPU doing the pkg backups as they're done sequentially rather than in parallel. In general I think a method of forcing a jail to start it's jobs later based on JID (or some other metric as JID could get rather large and sparse if you stop/start jails a lot) is a good idea that should be looked at for inclusion in base Regards, Gary > > > On Feb 16, 2017, at 5:44 PM, Walter Cramer <wfc@mintsol.com> wrote: > > > > Adding something like: > > > > 'sleep $(( $(sysctl -n security.jail.param.jid) * 15 )) && ' > > > > in front of more resource-intensive commands in /etc/crontab can reliably spread out the load from a larger number of jails. > > > > (But if you start and stop jails frequently enough to spread out the current list of jail id numbers, this method degrades.) > > > > Low priority for 'periodic daily' jobs might not help much, due to disk saturation, CPU cache thrashing, etc. > > -Walter > > > > On Thu, 16 Feb 2017, Dustin Wenz wrote: > > > >> The biggest offender that I see is /usr/local/etc/periodic/daily/411.pkg-backup > >> > >> During the high CPU event, my process list contains hundreds of these: > >> > >> 83811 - RJ 0:03.42 xz -c > >> 83816 - S 0:00.02 /usr/local/sbin/pkg shell .dump > >> 83818 - SJ 0:00.02 /usr/local/sbin/pkg shell .dump > >> 83820 - SJ 0:00.03 /usr/local/sbin/pkg shell .dump > >> 83824 - RJ 0:03.41 xz -c > >> 83831 - RJ 0:03.58 xz -c > >> > >> I could probably get away with disabling that in my case. > >> > >> However, instead of jitter, I think I'd prefer if the periodic jobs ran at a lower priority than my user processes. Either through nice, or idprio. I want them to get done as fast as possible, but I don't want them to affect my application. > >> > >> - .Dustin > >> > >> > >>> On Feb 16, 2017, at 4:20 PM, Alan Somers <asomers@freebsd.org> wrote: > >>> > >>> Is the problem caused by newsyslog or by the periodic scripts? > >>> Newsyslog normally runs from cron directly, not through periodic. In > >>> any case, here are a few suggestions: > >>> 1) Turn on cron jitter, as you suggested. Even if 60s isn't enough, > >>> it can't hurt. > >>> 2) Try gz compression instead of xz compression to see if it's faster > >>> 3) Manually edit the jails' /etc/crontab files to hardcode some > >>> variability into their newsyslog and/or periodic run times > >>> 4) If the problem is actually being caused by periodic instead of > >>> newsyslog, tell me which script it is and how much jitter you want. I > >>> am coincidentally changing how periodic manages jitter right now. > >>> > >>> -Alan > >>> > >>> On Thu, Feb 16, 2017 at 2:47 PM, Dustin Wenz <dustinwenz@ebureau.com> wrote: > >>>> I have a number of servers with roughly 60 jails running on each of them. On these hosts, I've had to disable the periodic security scans due to overly high disk load when they run (which is redundant in jails anyway). However, I still have an issue at 3:01am where the CPU is consumed by dozens of 'xz -c' processes. This is apparently daily log rolling, which I can't exactly disable. > >>>> > >>>> The effect is that our processing applications experience a major slowdown for about 15 seconds every morning, which is just enough that it's starting to get people's attention. > >>>> > >>>> What is the best way to mitigate this? I'm aware of the cron jitter feature, but I'm not sure of the 60-second jitter maximum would be enough (especially if I wanted to start utilizing more jails). > >>>> > >>>> - .Dustin > >>>> _______________________________________________ > >>>> freebsd-stable@freebsd.org mailing list > >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable > >>>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > >> > >> _______________________________________________ > >> freebsd-stable@freebsd.org mailing list > >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable > >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > > > > _______________________________________________ > > freebsd-stable@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20170218153523.GA51196>