From owner-freebsd-stable@freebsd.org Thu Feb 16 22:40:45 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 95B64CE2343 for ; Thu, 16 Feb 2017 22:40:45 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from internet06.ebureau.com (internet06.ebureau.com [65.127.24.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "internet06.ebureau.com", Issuer "internet06.ebureau.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 73DEC14CB; Thu, 16 Feb 2017 22:40:45 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from localhost (localhost [127.0.0.1]) by internet06.ebureau.com (Postfix) with ESMTP id 2919D7024EB2; Thu, 16 Feb 2017 16:40:43 -0600 (CST) X-Virus-Scanned: amavisd-new at mydomain = ebureau.com Received: from internet06.ebureau.com ([127.0.0.1]) by localhost (internet06.ebureau.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qltUR5edoKj0; Thu, 16 Feb 2017 16:40:42 -0600 (CST) Received: from square.office.ebureau.com (unknown [10.10.20.22]) by internet06.ebureau.com (Postfix) with ESMTPSA id 89CE27024EA8; Thu, 16 Feb 2017 16:40:42 -0600 (CST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 10.2 \(3259\)) Subject: Re: Jailed periodic daily scripts smashing CPU From: Dustin Wenz In-Reply-To: Date: Thu, 16 Feb 2017 16:40:42 -0600 Cc: FreeBSD Content-Transfer-Encoding: quoted-printable Message-Id: <46236914-07E5-4128-947E-E2EBD04542BE@ebureau.com> References: <321260F8-95D8-4C21-90B5-FDB0F6FF98F9@ebureau.com> To: Alan Somers X-Mailer: Apple Mail (2.3259) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Feb 2017 22:40:45 -0000 The biggest offender that I see is = /usr/local/etc/periodic/daily/411.pkg-backup During the high CPU event, my process list contains hundreds of these: 83811 - RJ 0:03.42 xz -c 83816 - S 0:00.02 /usr/local/sbin/pkg shell .dump 83818 - SJ 0:00.02 /usr/local/sbin/pkg shell .dump 83820 - SJ 0:00.03 /usr/local/sbin/pkg shell .dump 83824 - RJ 0:03.41 xz -c 83831 - RJ 0:03.58 xz -c I could probably get away with disabling that in my case. However, instead of jitter, I think I'd prefer if the periodic jobs ran = at a lower priority than my user processes. Either through nice, or = idprio. I want them to get done as fast as possible, but I don't want = them to affect my application. - .Dustin > On Feb 16, 2017, at 4:20 PM, Alan Somers wrote: >=20 > Is the problem caused by newsyslog or by the periodic scripts? > Newsyslog normally runs from cron directly, not through periodic. In > any case, here are a few suggestions: > 1) Turn on cron jitter, as you suggested. Even if 60s isn't enough, > it can't hurt. > 2) Try gz compression instead of xz compression to see if it's faster > 3) Manually edit the jails' /etc/crontab files to hardcode some > variability into their newsyslog and/or periodic run times > 4) If the problem is actually being caused by periodic instead of > newsyslog, tell me which script it is and how much jitter you want. I > am coincidentally changing how periodic manages jitter right now. >=20 > -Alan >=20 > On Thu, Feb 16, 2017 at 2:47 PM, Dustin Wenz = wrote: >> I have a number of servers with roughly 60 jails running on each of = them. On these hosts, I've had to disable the periodic security scans = due to overly high disk load when they run (which is redundant in jails = anyway). However, I still have an issue at 3:01am where the CPU is = consumed by dozens of 'xz -c' processes. This is apparently daily log = rolling, which I can't exactly disable. >>=20 >> The effect is that our processing applications experience a major = slowdown for about 15 seconds every morning, which is just enough that = it's starting to get people's attention. >>=20 >> What is the best way to mitigate this? I'm aware of the cron jitter = feature, but I'm not sure of the 60-second jitter maximum would be = enough (especially if I wanted to start utilizing more jails). >>=20 >> - .Dustin >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org"