From owner-freebsd-stable@FreeBSD.ORG Wed May 4 21:42:37 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6A8BB1065670 for ; Wed, 4 May 2011 21:42:37 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta13.emeryville.ca.mail.comcast.net (qmta13.emeryville.ca.mail.comcast.net [76.96.27.243]) by mx1.freebsd.org (Postfix) with ESMTP id 4FE108FC08 for ; Wed, 4 May 2011 21:42:37 +0000 (UTC) Received: from omta17.emeryville.ca.mail.comcast.net ([76.96.30.73]) by qmta13.emeryville.ca.mail.comcast.net with comcast id fZge1g0041afHeLADZidic; Wed, 04 May 2011 21:42:37 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta17.emeryville.ca.mail.comcast.net with comcast id fZia1g00C1t3BNj8dZiadn; Wed, 04 May 2011 21:42:35 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 0234D102C36; Wed, 4 May 2011 14:42:34 -0700 (PDT) Date: Wed, 4 May 2011 14:42:33 -0700 From: Jeremy Chadwick To: "Mikhail T." Message-ID: <20110504214233.GA97428@icarus.home.lan> References: <4DC1A1CC.4080901@aldan.algebra.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4DC1A1CC.4080901@aldan.algebra.com> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org Subject: Re: syslogd spinning the CPU, not logging... X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 May 2011 21:42:37 -0000 On Wed, May 04, 2011 at 02:58:20PM -0400, Mikhail T. wrote: > On FreeBSD/amd64 8.2-STABLE as of Sun Feb 27. I dismissed this > problem the first time (a week or two ago), but just saw it again. > Something triggers syslogd into spinning all available CPU -- while > not logging anything. > > Attempts to log anything -- such as by invoking logger(1) -- simply > hang. sendmail hangs too -- now new e-mail is coming. It is not > pretty. > > ktrace-ing the process yields empty file -- it is not doing > system-calls, while spinning. > > Attaching the debugger several times shows the same stack: > > #0 0x00000008007cb1d6 in _pthread_mutex_init_calloc_cb () from /lib/libc.so.7 > #1 0x00000008007cd9fb in _pthread_mutex_init_calloc_cb () from /lib/libc.so.7 > #2 0x00000008007d4825 in free () from /lib/libc.so.7 > #3 0x00000008007c5aab in catopen () from /lib/libc.so.7 > #4 0x00000008007c5410 in strerror_r () from /lib/libc.so.7 > #5 0x00000008007c556d in strerror () from /lib/libc.so.7 > #6 0x0000000000403c10 in ?? () > ... > > Among the logging-destinations in my syslog.conf there is a program, > that exits sometimes -- but never "too fast". Normally, syslogd > simply restarts it without an issue... > > Any ideas? Thanks! If ktrace doesn't show any syscalls (did you use "ktrace -i -t+"?), then I'm not sure how the process is actually spinning (100% CPU). "procstat -kk" on the PID of syslogd might be more useful in this case. The output is mainly useful for kernel folks. You may end up having to drop to ddb to examine what's going on within the pid. I'm not sure if attaching gdb to a running process on FreeBSD works with things like threads, etc.. Meaning I'm not sure if the above stack trace is indicative of a problem or not. No offence to anyone intended, but I try to avoid gdb on FreeBSD wherever possible (it's so old that I don't know how useful it is for extreme debugging). I've never seen syslogd spiral out of control on any of our systems, but all logging destinations in syslog.conf on our systems are files on local disks. If you can find a way to reliably reproduce the issue that would be useful, otherwise this sounds like one of those things that's going to be a problem to track down. :-( -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB |