From owner-freebsd-stable@FreeBSD.ORG Fri Mar 23 11:09:54 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 68A0E106564A for ; Fri, 23 Mar 2012 11:09:54 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta07.emeryville.ca.mail.comcast.net (qmta07.emeryville.ca.mail.comcast.net [76.96.30.64]) by mx1.freebsd.org (Postfix) with ESMTP id 4ECEA8FC14 for ; Fri, 23 Mar 2012 11:09:54 +0000 (UTC) Received: from omta02.emeryville.ca.mail.comcast.net ([76.96.30.19]) by qmta07.emeryville.ca.mail.comcast.net with comcast id oz3L1i0040QkzPwA7z8oiG; Fri, 23 Mar 2012 11:08:48 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta02.emeryville.ca.mail.comcast.net with comcast id oz8n1i00L1t3BNj8Nz8oTg; Fri, 23 Mar 2012 11:08:48 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id D4291102C1A; Fri, 23 Mar 2012 04:08:47 -0700 (PDT) Date: Fri, 23 Mar 2012 04:08:47 -0700 From: Jeremy Chadwick To: freebsd-stable@freebsd.org Message-ID: <20120323110847.GA12111@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Subject: Debugging periodic scripts X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Mar 2012 11:09:54 -0000 (Please keep me CC'd as I'm not subscribed) Hi folks, Does anyone know how to go about debugging periodic scripts, such as getting useful debug output from start to finish? Basically the situation is this: - We have 5 systems which are RELENG_8 (some 8.2-STABLE, and a couple 8.3-PRERELEASE). These are all bare metal boxes, not VMs. - All the machines have WITHOUT_IPFILTER=true defined in src.conf. - All the machines also have this in their /etc/periodic.conf : daily_status_security_ipfdenied_enable="no" - All the machines run ntpd and do not have clock skew problems or other odd anomalies (they all work great hardware-wise) or filesystem issues (all are using UFS2). On 2 of the systems, /etc/periodic/security/510.ipfdenied gets run during "periodic security" even though it's explicitly shut off in periodic.conf. Thus on these 2 systems, our security mails contain this line: ipfstat: not found I've checked permissions on everything I can think of (from / all the way down) but it all looks fine. I even wrote a small forloop to check all the systems' periodic.conf files and ensure the ipfdenied_enable line is proper (no weird trailing or preceding spaces, high-bit characters, DOS CRs, etc.) and they all check out (1 line, 44 characters long). One of the boxes was even recently rebuilt from scratch (full format + OS reinstall); it exhibited this problem prior to the rebuild, as well as after the rebuild. None of the systems have any unique changes to /root dotfiles nor the shell adjustments in things like /etc/profile, /etc/csh*, etc.. I've tried doing this: (sh -x /etc/periodic/security/510.ipfdenied >& /dev/stdout) | grep ipfdenied Which returns exactly what I would expect: + daily_status_security_ipfwdenied_enable=YES + daily_status_security_ipfdenied_enable=YES + daily_status_security_ipfwlimit_enable=YES + daily_status_security_ipf6denied_enable=YES + daily_status_security_ipfdenied_enable=no The first 4 come from /etc/defaults/periodic.conf, the last comes from /etc/periodic.conf. Running /etc/periodic/security/510.ipfdenied from a root shell results in no output. Editing /etc/periodic/security/510.ipfdenied's hashbang line to use -x doesn't change the behaviour either (maybe stderr gets sent to /dev/null?), whether I run it by hand as a script or via "periodic security". Other settings in periodic.conf are in fact honoured, such as daily_status_smart_enable and some others, so I'm inclined to believe periodic.conf is indeed being read. I don't know what's making this situation. I haven't resorted to using ktrace yet but will down the road assuming nobody has any other ideas. Otherwise something tells me I'm going to have to go look at the periodic source code to figure out what's going on under the hood. Thoughts/ideas? -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |