From owner-freebsd-stable@freebsd.org Wed Dec 9 10:24:11 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B3BD99D4CF5 for ; Wed, 9 Dec 2015 10:24:11 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from smtp.rlwinm.de (smtp.rlwinm.de [IPv6:2a01:4f8:201:31ef::e]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7F01D178A for ; Wed, 9 Dec 2015 10:24:11 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from crest.local (unknown [87.253.189.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.rlwinm.de (Postfix) with ESMTPSA id 0BDDAB29E for ; Wed, 9 Dec 2015 11:24:08 +0100 (CET) Subject: Re: Periodic jobs triggering panics in 10.1 and 10.2 To: freebsd-stable@freebsd.org References: <34FA7D40-8758-460D-AC14-20B21D2E3F8D@ebureau.com> <1449619470.31831.9.camel@michaeleichorn.com> From: Jan Bramkamp Message-ID: <56680148.9070601@rlwinm.de> Date: Wed, 9 Dec 2015 11:24:08 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: <1449619470.31831.9.camel@michaeleichorn.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Dec 2015 10:24:11 -0000 On 09/12/15 01:04, Michael B. Eichorn wrote: > On Tue, 2015-12-08 at 16:31 -0600, Dustin Wenz wrote: >> I suspect this is a zfs bug that is triggered by the access patterns >> in the periodic scripts. There is significant load on the system when >> the scheduled processes start, because all jails execute the same >> scripts at the same time. >> >> I've been able to alleviate this problem by disabling the security >> scans within the jails, but leave it enabled on the root host. > > To avoid the problem of jails all starting things at the same time, use > the cron(8) flags -j and -J to set a 'jitter' which will cause cron to > sleep for a random period of specified duration (60 sec max). Cron > flags can be set using the rc.conf variable 'cron_flags'. While jitter would reduce the resource contention a thundering herd of cronjobs shouldn't cause the kernel to divide by zero. Spreading the load by introducing jitter to cronjobs might hide the problem, but it still needs further analysis. @Dustin Wenz: Can you reproduce the problem and file a PR to track this?