From owner-freebsd-stable@freebsd.org Thu Feb 16 22:20:31 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 84316CE2BFA for ; Thu, 16 Feb 2017 22:20:31 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-yw0-x231.google.com (mail-yw0-x231.google.com [IPv6:2607:f8b0:4002:c05::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4398328F0 for ; Thu, 16 Feb 2017 22:20:31 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-yw0-x231.google.com with SMTP id u68so15386363ywg.0 for ; Thu, 16 Feb 2017 14:20:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-transfer-encoding; bh=qZZcPuAWsj5FnDYTN6qnQIvSt77KM7p3UE1P6hH2mSY=; b=FPn7g8BkjF2fWK7E+UxB4+/2X1+X93tkG95+FfFhOP2UgT8TuFw4ouJnKWhXmx4V6V uX3pTowCVcWzBsTAgsfVdGzppzX3UcmUNmq6oXahkeErjZ7aOm/8mDecXpQQ3RYZyVvo Qs5jzDFlHowCPSMM90MydAfQkpOs9Pt56ifNeYw8RzDelw7mlLGRedT6Og0Ig6LtBeus s4vmXeUXsDM0iVKP0T8OFMUC5u6FQgJ8cl3mttVHuoJtNvR6yxcg0HxOrFYefSspxzQ6 edckJuRfLqD5sy4dRWQIdgigMx9WKXZ5jAAePS+mb3OXcU9tlFbz+1l4iMnCPZ0TOr2z HeRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-transfer-encoding; bh=qZZcPuAWsj5FnDYTN6qnQIvSt77KM7p3UE1P6hH2mSY=; b=Ud7PWlDNM0EIt8lF7eZd14LsjrbmW3mIbNnYE8Vi280cDh+WATNsR4MMvWiCJ9Zkes hU2W610bvqBdw6cLAh9SzH6uL/JonUiutfEif7DTEJlTrmbxOszbs3gPWwsQJLeV6NI7 2/4s6Fsbw1W5vG7HPqS4lC+Dioct5SsAUZ+o+zBAt36XhUoKm+OYQs66dQgTP8BWl2Cf IsEU8YMD0C5U5DabU6HQ2R0eIoP15CHgEKSy/4wv67hKzEuVvKCwd8nOoZUSAC6vlGVX 0yXVbhIUje4w6Xw2v4D8pth1Z3cy+a+Elth/KgwnLTzU02hHVgP6tpANpQt3+ExayU2G BNeg== X-Gm-Message-State: AMke39kXVqqeY0oy9ARfvZpg8Dg2xMCZ1kzeSEG4D7tEnYcZALwRbNdmwdJWSKmiS2KC/YrGejxwE+XVqhZHuw== X-Received: by 10.129.173.71 with SMTP id l7mr3360539ywk.351.1487283630304; Thu, 16 Feb 2017 14:20:30 -0800 (PST) MIME-Version: 1.0 Sender: asomers@gmail.com Received: by 10.129.38.133 with HTTP; Thu, 16 Feb 2017 14:20:29 -0800 (PST) In-Reply-To: <321260F8-95D8-4C21-90B5-FDB0F6FF98F9@ebureau.com> References: <321260F8-95D8-4C21-90B5-FDB0F6FF98F9@ebureau.com> From: Alan Somers Date: Thu, 16 Feb 2017 15:20:29 -0700 X-Google-Sender-Auth: 2iiu2dNvJ8hbbUeYPgMh27cpolE Message-ID: Subject: Re: Jailed periodic daily scripts smashing CPU To: Dustin Wenz Cc: FreeBSD Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Feb 2017 22:20:31 -0000 Is the problem caused by newsyslog or by the periodic scripts? Newsyslog normally runs from cron directly, not through periodic. In any case, here are a few suggestions: 1) Turn on cron jitter, as you suggested. Even if 60s isn't enough, it can't hurt. 2) Try gz compression instead of xz compression to see if it's faster 3) Manually edit the jails' /etc/crontab files to hardcode some variability into their newsyslog and/or periodic run times 4) If the problem is actually being caused by periodic instead of newsyslog, tell me which script it is and how much jitter you want. I am coincidentally changing how periodic manages jitter right now. -Alan On Thu, Feb 16, 2017 at 2:47 PM, Dustin Wenz wrote= : > I have a number of servers with roughly 60 jails running on each of them.= On these hosts, I've had to disable the periodic security scans due to ove= rly high disk load when they run (which is redundant in jails anyway). Howe= ver, I still have an issue at 3:01am where the CPU is consumed by dozens of= 'xz -c' processes. This is apparently daily log rolling, which I can't exa= ctly disable. > > The effect is that our processing applications experience a major slowdow= n for about 15 seconds every morning, which is just enough that it's starti= ng to get people's attention. > > What is the best way to mitigate this? I'm aware of the cron jitter featu= re, but I'm not sure of the 60-second jitter maximum would be enough (espec= ially if I wanted to start utilizing more jails). > > - .Dustin > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"