From owner-freebsd-stable@freebsd.org Fri Feb 17 16:47:27 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D4FAACE3025 for ; Fri, 17 Feb 2017 16:47:27 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from internet06.ebureau.com (internet06.ebureau.com [65.127.24.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "internet06.ebureau.com", Issuer "internet06.ebureau.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id B246C1556; Fri, 17 Feb 2017 16:47:27 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from localhost (localhost [127.0.0.1]) by internet06.ebureau.com (Postfix) with ESMTP id 18CED703335F; Fri, 17 Feb 2017 10:47:25 -0600 (CST) X-Virus-Scanned: amavisd-new at mydomain = ebureau.com Received: from internet06.ebureau.com ([127.0.0.1]) by localhost (internet06.ebureau.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id T4XwSA5PVV-E; Fri, 17 Feb 2017 10:47:24 -0600 (CST) Received: from square.office.ebureau.com (unknown [10.10.20.22]) by internet06.ebureau.com (Postfix) with ESMTPSA id 49E2D7033355; Fri, 17 Feb 2017 10:47:24 -0600 (CST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 10.2 \(3259\)) Subject: Re: Jailed periodic daily scripts smashing CPU From: Dustin Wenz In-Reply-To: <20170216174749.V86669@mulder.mintsol.com> Date: Fri, 17 Feb 2017 10:47:23 -0600 Cc: FreeBSD , Alan Somers Content-Transfer-Encoding: quoted-printable Message-Id: <7D7C8824-319C-4891-8B35-82F9B461FE5D@ebureau.com> References: <321260F8-95D8-4C21-90B5-FDB0F6FF98F9@ebureau.com> <46236914-07E5-4128-947E-E2EBD04542BE@ebureau.com> <20170216174749.V86669@mulder.mintsol.com> To: Walter Cramer X-Mailer: Apple Mail (2.3259) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Feb 2017 16:47:27 -0000 If I needed to go down the road of spreading out the daily maintenance = over a longer period of time, I'd probably use your solution of building = in a delay based on the jail id, or possibly Guy Tabrar's extreme jitter = example. The pkg-backup job appears somewhat flawed, in my view. The xz = compressor is totally CPU bound for a moderate number of jails . Sure, = running it at low priority isn't as cache friendly, but it's certainly = better then the host becoming unresponsive. The other issue is that in = most cases the package archive doesn't change, and so running a daily = backup without checking for changes is wasteful. In my case, ZFS handles = all the rolling backups that I could ever need, so using pkg-backup for = that purpose is redundant. - .Dustin > On Feb 16, 2017, at 5:44 PM, Walter Cramer wrote: >=20 > Adding something like: >=20 > 'sleep $(( $(sysctl -n security.jail.param.jid) * 15 )) && ' >=20 > in front of more resource-intensive commands in /etc/crontab can = reliably spread out the load from a larger number of jails. >=20 > (But if you start and stop jails frequently enough to spread out the = current list of jail id numbers, this method degrades.) >=20 > Low priority for 'periodic daily' jobs might not help much, due to = disk saturation, CPU cache thrashing, etc. > -Walter >=20 > On Thu, 16 Feb 2017, Dustin Wenz wrote: >=20 >> The biggest offender that I see is = /usr/local/etc/periodic/daily/411.pkg-backup >>=20 >> During the high CPU event, my process list contains hundreds of = these: >>=20 >> 83811 - RJ 0:03.42 xz -c >> 83816 - S 0:00.02 /usr/local/sbin/pkg shell .dump >> 83818 - SJ 0:00.02 /usr/local/sbin/pkg shell .dump >> 83820 - SJ 0:00.03 /usr/local/sbin/pkg shell .dump >> 83824 - RJ 0:03.41 xz -c >> 83831 - RJ 0:03.58 xz -c >>=20 >> I could probably get away with disabling that in my case. >>=20 >> However, instead of jitter, I think I'd prefer if the periodic jobs = ran at a lower priority than my user processes. Either through nice, or = idprio. I want them to get done as fast as possible, but I don't want = them to affect my application. >>=20 >> - .Dustin >>=20 >>=20 >>> On Feb 16, 2017, at 4:20 PM, Alan Somers = wrote: >>>=20 >>> Is the problem caused by newsyslog or by the periodic scripts? >>> Newsyslog normally runs from cron directly, not through periodic. = In >>> any case, here are a few suggestions: >>> 1) Turn on cron jitter, as you suggested. Even if 60s isn't enough, >>> it can't hurt. >>> 2) Try gz compression instead of xz compression to see if it's = faster >>> 3) Manually edit the jails' /etc/crontab files to hardcode some >>> variability into their newsyslog and/or periodic run times >>> 4) If the problem is actually being caused by periodic instead of >>> newsyslog, tell me which script it is and how much jitter you want. = I >>> am coincidentally changing how periodic manages jitter right now. >>>=20 >>> -Alan >>>=20 >>> On Thu, Feb 16, 2017 at 2:47 PM, Dustin Wenz = wrote: >>>> I have a number of servers with roughly 60 jails running on each of = them. On these hosts, I've had to disable the periodic security scans = due to overly high disk load when they run (which is redundant in jails = anyway). However, I still have an issue at 3:01am where the CPU is = consumed by dozens of 'xz -c' processes. This is apparently daily log = rolling, which I can't exactly disable. >>>>=20 >>>> The effect is that our processing applications experience a major = slowdown for about 15 seconds every morning, which is just enough that = it's starting to get people's attention. >>>>=20 >>>> What is the best way to mitigate this? I'm aware of the cron jitter = feature, but I'm not sure of the 60-second jitter maximum would be = enough (especially if I wanted to start utilizing more jails). >>>>=20 >>>> - .Dustin >>>> _______________________________________________ >>>> freebsd-stable@freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>>> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >>=20 >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >=20 > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org"