From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 26 15:31:36 2010 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BB65C106566B for ; Fri, 26 Mar 2010 15:31:36 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 8C7A88FC1C for ; Fri, 26 Mar 2010 15:31:36 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 338DD46B09; Fri, 26 Mar 2010 11:31:36 -0400 (EDT) Received: from jhbbsd.localnet (smtp.hudson-trading.com [209.249.190.9]) by bigwig.baldwin.cx (Postfix) with ESMTPA id 44A0B8A025; Fri, 26 Mar 2010 11:31:35 -0400 (EDT) From: John Baldwin To: freebsd-hackers@freebsd.org Date: Fri, 26 Mar 2010 09:41:29 -0400 User-Agent: KMail/1.12.1 (FreeBSD/7.3-CBSD-20100217; KDE/4.3.1; amd64; ; ) References: <7B9397B189EB6E46A5EE7B4C8A4BB7CB3B5AACBE@MBX03.exg5.exghost.com> <86iq8jl0d7.fsf@ds4.des.no> <7B9397B189EB6E46A5EE7B4C8A4BB7CB3B5AACDB@MBX03.exg5.exghost.com> In-Reply-To: <7B9397B189EB6E46A5EE7B4C8A4BB7CB3B5AACDB@MBX03.exg5.exghost.com> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201003260941.29154.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0.1 (bigwig.baldwin.cx); Fri, 26 Mar 2010 11:31:35 -0400 (EDT) X-Virus-Scanned: clamav-milter 0.95.1 at bigwig.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-1.7 required=4.2 tests=AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on bigwig.baldwin.cx Cc: Dag-Erling =?iso-8859-15?q?Sm=F8rgrav?= , Peter Steele Subject: Re: Puzzling stack trace X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Mar 2010 15:31:36 -0000 On Friday 26 March 2010 9:32:46 am Peter Steele wrote: > >Type "frame 9" and see what it says. If the bug is easily reproducable, try reproducing it with a debugging version of libc (buildworld with > >DEBUG_FLAGS=-g) > > This crash happened at a production customer site--we have the core and the matching binary and our logs for the application that crashed but that's all. We've never seen this particular crash before and cannot reproduce it. The fscanf() call that failed is repeated on a continual basis as part of a monitoring thread, so literally thousands of this exact same call have been made without incident. > > The "frame 9" command doesn't show anything useful: > > (gdb) frame 9 > #9 0x000000080127374b in memchr () from /lib/libc.so.7 > > That's it. And yes, the stack trace appears to be wrong. Even the trace starting from the vsscanf call is wrong. It says that __srget() is the next function in the stack but vsscanf() doesn't call __srget(): > > int > vsscanf(const char * __restrict str, const char * __restrict fmt, > __va_list ap) > { > FILE f; > > f._file = -1; > f._flags = __SRD; > f._bf._base = f._p = (unsigned char *)str; > f._bf._size = f._r = strlen(str); > f._read = eofread; > f._ub._base = NULL; > f._lb._base = NULL; > f._orientation = 0; > memset(&f._mbstate, 0, sizeof(mbstate_t)); > return (__svfscanf(&f, fmt, ap)); > } > > So it seems our application went completely out to lunch. This is concerning. Keep in mind that libc doesn't include symbols for static functions, so gdb will just report the name of the preceding non-static function in the .text section. If you have a source tree that matches this libc exactly you can just build a new libc with debugging (make DEBUG_FLAGS=-g) and use that with gdb to find the real stack trace in many cases. Also, you should see if __svfscanf() calls __srget(). The __svfscanf() call frame may not show up in gdb if the compiler re-used the callframe from vsscanf for __svfscanf() as an optimization. -- John Baldwin