Date: Thu, 16 Feb 2017 16:40:42 -0600 From: Dustin Wenz <dustinwenz@ebureau.com> To: Alan Somers <asomers@freebsd.org> Cc: FreeBSD <freebsd-stable@freebsd.org> Subject: Re: Jailed periodic daily scripts smashing CPU Message-ID: <46236914-07E5-4128-947E-E2EBD04542BE@ebureau.com> In-Reply-To: <CAOtMX2hb%2BmoGWZFuUjrhO-A6NPg5zRsgs3QDRA2Ja1eATwUCsQ@mail.gmail.com> References: <321260F8-95D8-4C21-90B5-FDB0F6FF98F9@ebureau.com> <CAOtMX2hb%2BmoGWZFuUjrhO-A6NPg5zRsgs3QDRA2Ja1eATwUCsQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
The biggest offender that I see is = /usr/local/etc/periodic/daily/411.pkg-backup During the high CPU event, my process list contains hundreds of these: 83811 - RJ 0:03.42 xz -c 83816 - S 0:00.02 /usr/local/sbin/pkg shell .dump 83818 - SJ 0:00.02 /usr/local/sbin/pkg shell .dump 83820 - SJ 0:00.03 /usr/local/sbin/pkg shell .dump 83824 - RJ 0:03.41 xz -c 83831 - RJ 0:03.58 xz -c I could probably get away with disabling that in my case. However, instead of jitter, I think I'd prefer if the periodic jobs ran = at a lower priority than my user processes. Either through nice, or = idprio. I want them to get done as fast as possible, but I don't want = them to affect my application. - .Dustin > On Feb 16, 2017, at 4:20 PM, Alan Somers <asomers@freebsd.org> wrote: >=20 > Is the problem caused by newsyslog or by the periodic scripts? > Newsyslog normally runs from cron directly, not through periodic. In > any case, here are a few suggestions: > 1) Turn on cron jitter, as you suggested. Even if 60s isn't enough, > it can't hurt. > 2) Try gz compression instead of xz compression to see if it's faster > 3) Manually edit the jails' /etc/crontab files to hardcode some > variability into their newsyslog and/or periodic run times > 4) If the problem is actually being caused by periodic instead of > newsyslog, tell me which script it is and how much jitter you want. I > am coincidentally changing how periodic manages jitter right now. >=20 > -Alan >=20 > On Thu, Feb 16, 2017 at 2:47 PM, Dustin Wenz <dustinwenz@ebureau.com> = wrote: >> I have a number of servers with roughly 60 jails running on each of = them. On these hosts, I've had to disable the periodic security scans = due to overly high disk load when they run (which is redundant in jails = anyway). However, I still have an issue at 3:01am where the CPU is = consumed by dozens of 'xz -c' processes. This is apparently daily log = rolling, which I can't exactly disable. >>=20 >> The effect is that our processing applications experience a major = slowdown for about 15 seconds every morning, which is just enough that = it's starting to get people's attention. >>=20 >> What is the best way to mitigate this? I'm aware of the cron jitter = feature, but I'm not sure of the 60-second jitter maximum would be = enough (especially if I wanted to start utilizing more jails). >>=20 >> - .Dustin >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?46236914-07E5-4128-947E-E2EBD04542BE>