From owner-freebsd-current@freebsd.org Fri Mar 18 06:15:03 2016 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8711FAD3A6F for ; Fri, 18 Mar 2016 06:15:03 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 17F671D04 for ; Fri, 18 Mar 2016 06:15:02 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u2I6Evqs079484 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Fri, 18 Mar 2016 08:14:57 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u2I6Evqs079484 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u2I6Ev05079483; Fri, 18 Mar 2016 08:14:57 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 18 Mar 2016 08:14:57 +0200 From: Konstantin Belousov To: "Oleg V. Nauman" Cc: freebsd-current@freebsd.org Subject: Re: Fatal error 'mutex is on list' at line 139 in file /usr/src/lib/libthr/thread/thr_mutex.c (errno = 35) Message-ID: <20160318061457.GV1741@kib.kiev.ua> References: <5093647.qxI0C33PyG@asus.theweb.org.ua> <20160318013051.GT1741@kib.kiev.ua> <2526381.30oznMcJoQ@asus.theweb.org.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2526381.30oznMcJoQ@asus.theweb.org.ua> User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Mar 2016 06:15:03 -0000 On Fri, Mar 18, 2016 at 07:57:30AM +0200, Oleg V. Nauman wrote: > On Friday 18 March 2016 03:30:51 Konstantin Belousov wrote: > > On Thu, Mar 17, 2016 at 09:40:11PM +0200, Oleg V. Nauman wrote: > > > I'm experiencing issue with threaded applications ( including many KDE > > > > > > applications ) after recent CURRENT update. Attempt to start application > > > cause Abort trap (6) with "Fatal error 'mutex is on list' at line 139 in > > > file /usr/src/lib/libthr/thread/thr_mutex.c (errno = 35)" output on > > > console. > > > > > > FreeBSD 11.0-CURRENT amd64 r296887 > > > > > > Backtrace from application looks like > > > > > > Application: KGpg (kgpg), signal: Abort trap > > > [Switching to Thread 810815000 (LWP 100865/kgpg)] > > > [KCrash Handler] > > > #8 0x0000000807176d6a in thr_kill () from /lib/libc.so.7 > > > #9 0x0000000807176d3b in __raise (s=6) at > > > /usr/src/lib/libc/gen/raise.c:52 > > > #10 0x0000000807176ca9 in abort () at /usr/src/lib/libc/stdlib/abort.c:65 > > > #11 0x0000000806e9e62c in _thread_exit (fname=, > > > lineno=, msg=) at > > > /usr/src/lib/libthr/thread/thr_exit.c:182 > > > #12 0x0000000806e996fc in mutex_lock_common (m=, > > > abstime=, cvattach=) at > > > /usr/src/lib/libthr/thread/thr_mutex.c:139 > > > #13 0x0000000806e98c20 in __pthread_mutex_timedlock (mutex=0x814600008, > > > abstime=0x7fffffff5a58) at /usr/src/lib/libthr/thread/thr_mutex.c:566 > > > > From the frame 13, please do > > p *mutex > > p *m > > p **mutex > > Got few cores today morning, for example: > > gdb /usr/local/bin/akonadi_notes_agent akonadi_notes_agent.core > .... ( list of many shared libraries loaded by application ) > > 0 0x0000000805d08d6a in thr_kill () from /lib/libc.so.7 > [New Thread 817615000 (LWP 100295/)] > (gdb) bt > #0 0x0000000805d08d6a in thr_kill () from /lib/libc.so.7 > #1 0x0000000805d08d3b in __raise (s=6) at /usr/src/lib/libc/gen/raise.c:52 > #2 0x0000000805d08ca9 in abort () at /usr/src/lib/libc/stdlib/abort.c:65 > #3 0x0000000805a3062c in _thread_exit (fname=, > lineno=, msg=) > at /usr/src/lib/libthr/thread/thr_exit.c:182 > #4 0x0000000805a2b6fc in mutex_lock_common (m=, > abstime=, cvattach=) > at /usr/src/lib/libthr/thread/thr_mutex.c:139 > #5 0x0000000805a2ac20 in __pthread_mutex_timedlock (mutex=0x81b200008, > abstime=0x7fffffffd4f8) at /usr/src/lib/libthr/thread/thr_mutex.c:566 > #6 0x000000080448b560 in KSharedDataCache::setTimestamp () > from /usr/local/lib/libkdecore.so.5 > #7 0x000000080448b7df in KSharedDataCache::setTimestamp () > from /usr/local/lib/libkdecore.so.5 > #8 0x0000000804487192 in KSharedDataCache::insert () from > /usr/local/lib/libkdecore.so.5 > #9 0x0000000802c1912e in KIconLoader::drawOverlays () from > /usr/local/lib/libkdeui.so.5 > #10 0x0000000802c15e3b in KIconLoader::loadIcon () from > /usr/local/lib/libkdeui.so.5 > #11 0x0000000802c14366 in KIconEffect::overlay () from > /usr/local/lib/libkdeui.so.5 > #12 0x00000008034aff3d in QIcon::pixmap () from > /usr/local/lib/qt4/libQtGui.so.4 > #13 0x000000080349d2d6 in QWidgetPrivate::setWindowIcon_sys () > from /usr/local/lib/qt4/libQtGui.so.4 > #14 0x00000008034590e7 in QWidget::event () from > /usr/local/lib/qt4/libQtGui.so.4 > #15 0x000000080340a43c in QApplicationPrivate::notify_helper () > from /usr/local/lib/qt4/libQtGui.so.4 > #16 0x000000080340ca61 in QApplication::notify () from > /usr/local/lib/qt4/libQtGui.so.4 > #17 0x0000000802c724b7 in KApplication::notify () from > /usr/local/lib/libkdeui.so.5 > #18 0x0000000804d8eaa6 in QCoreApplication::notifyInternal () > from /usr/local/lib/qt4/libQtCore.so.4 > #19 0x00000008034084e1 in QApplication::setWindowIcon () from > /usr/local/lib/qt4/libQtGui.so.4 > #20 0x0000000802c74725 in KApplication::iceIOErrorHandler () from > /usr/local/lib/libkdeui.so.5 > #21 0x0000000802c72a9b in KApplication::KApplication () from > /usr/local/lib/libkdeui.so.5 > #22 0x0000000802c728f7 in KApplication::KApplication () from > /usr/local/lib/libkdeui.so.5 > #23 0x000000000040b10c in ?? () > #24 0x000000000040a74f in ?? () > #25 0x000000080063a000 in ?? () > #26 0x0000000000000000 in ?? () > (gdb) f 5 > #5 0x0000000805a2ac20 in __pthread_mutex_timedlock (mutex=0x81b200008, > abstime=0x7fffffffd4f8) at /usr/src/lib/libthr/thread/thr_mutex.c:566 > 566 ret = mutex_lock_common(m, abstime, 0); > Current language: auto; currently minimal > (gdb) p *mutex > $1 = 0x8000000000000001 This means that the mutex is shared, which is very strange. > (gdb) p *m > $2 = {m_lock = {m_owner = 1, m_flags = 2147483648, m_ceilings = 0x81b200010, > m_spare = 0x81b200018}, m_flags = 0, m_owner = 0, m_count = 0, m_spinloops > = 0, > m_yieldloops = 0, m_qe = {tqe_next = 0x0, tqe_prev = 0x1}, m_pqe = { > tqe_next = 0x71100a00000, tqe_prev = 0x100000000000}} The values are not reasonable. At least m_lock.m_owner is wrong, while m_owner indicates unowned lock. The m_lock.m_flags is UMUTEX_CONTESTED, lacking USYNC_PROCESS_SHARED bit. And of course, m_qe/m_pqe links are garbage, which triggered the panic. It almost looks as the *m is gargage memory, but not quite. > (gdb) p **mutex > Cannot access memory at address 0x8000000000000001 > > > > > > What non-KDE apps exhibit the behaviour ? > > I will try to find some simple application experiencing the same issue. Yes, please. It would be significantly easier to diagnose the problem if the minimal example is provided. If not, please pack one crashing app and all it non-system libraries and provide the tarball to me.