From owner-freebsd-current@FreeBSD.ORG Mon May 21 03:28:15 2012 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DEB1D106566B; Mon, 21 May 2012 03:28:15 +0000 (UTC) (envelope-from listlog2011@gmail.com) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 48A368FC0C; Mon, 21 May 2012 03:28:14 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q4L3SCnK095137; Mon, 21 May 2012 03:28:13 GMT (envelope-from listlog2011@gmail.com) Message-ID: <4FB9B649.8020906@gmail.com> Date: Mon, 21 May 2012 11:28:09 +0800 From: David Xu User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: davidxu@freebsd.org References: <4F9E9E06.4070004@entel.upc.edu> <4FB88925.4070008@gmail.com> <20120520172419.GQ2358@deviant.kiev.zoral.com.ua> <4FB9AE7E.6090109@gmail.com> In-Reply-To: <4FB9AE7E.6090109@gmail.com> Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: 7bit Cc: Konstantin Belousov , Alberto Villa , Gustau P?rez i Querol , FreeBSD current Subject: Re: RFC: jemalloc: qdbus sigsegv in malloc_init X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: davidxu@freebsd.org List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 May 2012 03:28:16 -0000 On 2012/5/21 10:54, David Xu wrote: > On 2012/5/21 1:24, Konstantin Belousov wrote: >> On Sun, May 20, 2012 at 06:42:35PM +0200, Alberto Villa wrote: >>> On Sun, May 20, 2012 at 8:03 AM, David Xu >>> wrote: >>>> qdbus segfaults on my machine too, I tracked it down, and found the >>>> problem >>>> is in QT, >>>> it deleted current_thread_data_key, but it still uses it in some >>>> cxa hooks, >>>> I applied the >>>> following patch, and it works fine. >>> Thanks for the analysis David! >>> >>>> I think the bug depends on linking order in QT library ? if the >>>> qthread_unix.cpp is linked >>>> as lastest module, the key will be deleted after all cxa hooks run, >>>> then it >>>> will be fine, >>>> otherwise, it would crash. >>> Is this really possible? >> No, I do not think it is possible. >> >> The only possibility for something weird happen is for >> atexit/__cxa_atexit >> functions to be registered from another atexit function, and then we >> indeed could call the newly registered function too late. >> >> I wonder if the following hack makes any change in the observed >> behaviour. >> >> diff --git a/lib/libc/stdlib/atexit.c b/lib/libc/stdlib/atexit.c >> index 511172a..bab850c 100644 >> --- a/lib/libc/stdlib/atexit.c >> +++ b/lib/libc/stdlib/atexit.c >> @@ -72,6 +72,7 @@ struct atexit { >> }; >> >> static struct atexit *__atexit; /* points to head of LIFO >> stack */ >> +static int atexit_gen; >> >> /* >> * Register the function described by 'fptr' to be called at >> application >> @@ -107,6 +108,7 @@ atexit_register(struct atexit_fn *fptr) >> __atexit = p; >> } >> p->fns[p->ind++] = *fptr; >> + atexit_gen++; >> _MUTEX_UNLOCK(&atexit_mutex); >> return 0; >> } >> @@ -162,7 +164,7 @@ __cxa_finalize(void *dso) >> struct dl_phdr_info phdr_info; >> struct atexit *p; >> struct atexit_fn fn; >> - int n, has_phdr; >> + int atexit_gen_prev, n, has_phdr; >> >> if (dso != NULL) >> has_phdr = _rtld_addr_phdr(dso,&phdr_info); >> @@ -170,6 +172,8 @@ __cxa_finalize(void *dso) >> has_phdr = 0; >> >> _MUTEX_LOCK(&atexit_mutex); >> +retry: >> + atexit_gen_prev = atexit_gen; >> for (p = __atexit; p; p = p->next) { >> for (n = p->ind; --n>= 0;) { >> if (p->fns[n].fn_type == ATEXIT_FN_EMPTY) >> @@ -196,6 +200,8 @@ __cxa_finalize(void *dso) >> _MUTEX_LOCK(&atexit_mutex); >> } >> } >> + if (atexit_gen_prev != atexit_gen) >> + goto retry; >> _MUTEX_UNLOCK(&atexit_mutex); >> if (dso == NULL) >> _MUTEX_DESTROY(&atexit_mutex); > I have tried your patch, it does not fix the problem. As I said, it > is a bug in QT, > the bug is pthread key current_thread_data_key is deleted by a global > C++ object > too early, other C++ global objects still need this pthread key. The > following procedure > shows how I found the problem: > > davidxu@xyf:~%gdb qdbus > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and > you are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for > details. > This GDB was configured as "i386-marcel-freebsd"...(no debugging > symbols found)... > (gdb) break __cxa_finalize > Function "__cxa_finalize" not defined. > Make breakpoint pending on future shared library load? (y or [n]) y > Breakpoint 1 (__cxa_finalize) pending. > (gdb) run > Starting program: /usr/local/bin/qdbus > (no debugging symbols found)...(no debugging symbols found)...(no > debugging symbols found)...(no debugging symbols found)...(no > debugging symbols found)...(no debugging symbols found)...(no > debugging symbols found)...[New LWP 100077] > (no debugging symbols found)...(no debugging symbols found)...(no > debugging symbols found)...(no debugging symbols found)...(no > debugging symbols found)...(no debugging symbols found)...(no > debugging symbols found)...(no debugging symbols found)...Breakpoint 2 > at 0x2864ac26 > Pending breakpoint "__cxa_finalize" resolved > (no debugging symbols found)...[New Thread 29007300 (LWP 100077/qdbus)] > (no debugging symbols found)...:1.0 > org.gnome.SessionManager > :1.11 > :1.111 > :1.12 > :1.13 > org.gtk.vfs.Daemon > :1.143 > :1.15 > org.pulseaudio.Server > :1.17 > org.gnome.Panel > :1.18 > :1.19 > :1.20 > org.gtk.Private.HalVolumeMonitor > :1.21 > org.gtk.Private.GPhoto2VolumeMonitor > :1.22 > :1.24 > org.gnome.ScreenSaver > :1.25 > :1.27 > :1.28 > :1.29 > :1.30 > :1.31 > org.gnome.panel.applet.WnckletFactory > :1.32 > :1.33 > :1.34 > :1.35 > org.gnome.panel.applet.CPUFreqAppletFactory > :1.36 > org.gnome.panel.applet.NotificationAreaAppletFactory > :1.37 > org.gnome.panel.applet.MultiLoadAppletFactory > :1.38 > :1.39 > :1.4 > org.gnome.GConf > :1.41 > org.gnome.panel.applet.ClockAppletFactory > :1.49 > :1.5 > org.gnome.SettingsDaemon > :1.50 > :1.53 > :1.64 > :1.7 > org.freedesktop.secrets > org.gnome.keyring > :1.75 > org.gtk.vfs.Metadata > :1.76 > org.gnome.Terminal.Display_0_0 > :1.77 > org.freedesktop.DBus > [Switching to Thread 29007300 (LWP 100077/qdbus)] > > Breakpoint 2, 0x2864ac26 in __cxa_finalize () from /lib/libc.so.7 > (gdb) print current_thread_data_key > $1 = 0 > (gdb) thread tsd > Key 0, destructor 0x281d77f0 <_Z27destroy_current_thread_dataPv> > Key 1, destructor 0x28732dc0 > Key 2, destructor 0x28726a00 > Key 3, destructor 0x0 > > > Here you can find that the function destroy_current_thread_data() is > registered. > > > (gdb) bt > #0 0x2864ac26 in __cxa_finalize () from /lib/libc.so.7 > #1 0x285efe1a in exit () from /lib/libc.so.7 > #2 0x08051db5 in main () > (gdb) break QThreadData::current() > Breakpoint 3 at 0x281d7856 > (gdb) info breakpoints > Num Type Disp Enb Address What > 2 breakpoint keep y 0x2864ac26 <__cxa_finalize+6> > breakpoint already hit 1 time > 3 breakpoint keep y 0x281d7856 > (gdb) delete 2 > (gdb) cont > Continuing. > > Breakpoint 3, 0x281d7856 in QThreadData::current () from > /usr/local/lib/qt4/libQtCore.so.4 > (gdb) bt > #0 0x281d7856 in QThreadData::current () from > /usr/local/lib/qt4/libQtCore.so.4 > #1 0x281d4747 in QThread::currentThread () from > /usr/local/lib/qt4/libQtCore.so.4 > #2 0x28097248 in QDBusConnectionPrivate::deleteYourself () from > /usr/local/lib/qt4/libQtDBus.so.4 > #3 0x2808f2ea in QDBusConnection::~QDBusConnection () from > /usr/local/lib/qt4/libQtDBus.so.4 > #4 0x2864ad8f in __cxa_finalize () from /lib/libc.so.7 > #5 0x285efe1a in exit () from /lib/libc.so.7 > #6 0x08051db5 in main () > (gdb) thread tsd > Key 1, destructor 0x0 > Key 2, destructor 0x0 > Key 3, destructor 0x0 > > Here you can see the destroy_current_thread_data() was executed, and > unregistered. > the key current_thread_key_data which is index 0 is deleted. > > > (gdb) cont > Continuing. > > Breakpoint 3, 0x281d7856 in QThreadData::current () from > /usr/local/lib/qt4/libQtCore.so.4 > (gdb) bt > #0 0x281d7856 in QThreadData::current () from > /usr/local/lib/qt4/libQtCore.so.4 > #1 0x282f58e3 in QObject::QObject () from > /usr/local/lib/qt4/libQtCore.so.4 > #2 0x281d4710 in QThread::QThread () from > /usr/local/lib/qt4/libQtCore.so.4 > #3 0x281d5a9e in QAdoptedThread::QAdoptedThread () from > /usr/local/lib/qt4/libQtCore.so.4 > #4 0x281d7934 in QThreadData::current () from > /usr/local/lib/qt4/libQtCore.so.4 > #5 0x281d4747 in QThread::currentThread () from > /usr/local/lib/qt4/libQtCore.so.4 > #6 0x28097248 in QDBusConnectionPrivate::deleteYourself () from > /usr/local/lib/qt4/libQtDBus.so.4 > #7 0x2808f2ea in QDBusConnection::~QDBusConnection () from > /usr/local/lib/qt4/libQtDBus.so.4 > #8 0x2864ad8f in __cxa_finalize () from /lib/libc.so.7 > #9 0x285efe1a in exit () from /lib/libc.so.7 > #10 0x08051db5 in main () > > now the stupid code starts to create a new thread... > > (gdb) cont > Continuing. > > Breakpoint 3, 0x281d7856 in QThreadData::current () from > /usr/local/lib/qt4/libQtCore.so.4 > (gdb) bt > #0 0x281d7856 in QThreadData::current () from > /usr/local/lib/qt4/libQtCore.so.4 > #1 0x282f58e3 in QObject::QObject () from > /usr/local/lib/qt4/libQtCore.so.4 > #2 0x281d4710 in QThread::QThread () from > /usr/local/lib/qt4/libQtCore.so.4 > #3 0x281d5a9e in QAdoptedThread::QAdoptedThread () from > /usr/local/lib/qt4/libQtCore.so.4 > #4 0x281d7934 in QThreadData::current () from > /usr/local/lib/qt4/libQtCore.so.4 > #5 0x282f58e3 in QObject::QObject () from > /usr/local/lib/qt4/libQtCore.so.4 > #6 0x281d4710 in QThread::QThread () from > /usr/local/lib/qt4/libQtCore.so.4 > #7 0x281d5a9e in QAdoptedThread::QAdoptedThread () from > /usr/local/lib/qt4/libQtCore.so.4 > #8 0x281d7934 in QThreadData::current () from > /usr/local/lib/qt4/libQtCore.so.4 > #9 0x281d4747 in QThread::currentThread () from > /usr/local/lib/qt4/libQtCore.so.4 > #10 0x28097248 in QDBusConnectionPrivate::deleteYourself () from > /usr/local/lib/qt4/libQtDBus.so.4 > #11 0x2808f2ea in QDBusConnection::~QDBusConnection () from > /usr/local/lib/qt4/libQtDBus.so.4 > #12 0x2864ad8f in __cxa_finalize () from /lib/libc.so.7 > #13 0x285efe1a in exit () from /lib/libc.so.7 > #14 0x08051db5 in main () > (gdb) > #0 0x281d7856 in QThreadData::current () from > /usr/local/lib/qt4/libQtCore.so.4 > #1 0x282f58e3 in QObject::QObject () from > /usr/local/lib/qt4/libQtCore.so.4 > #2 0x281d4710 in QThread::QThread () from > /usr/local/lib/qt4/libQtCore.so.4 > #3 0x281d5a9e in QAdoptedThread::QAdoptedThread () from > /usr/local/lib/qt4/libQtCore.so.4 > #4 0x281d7934 in QThreadData::current () from > /usr/local/lib/qt4/libQtCore.so.4 > #5 0x282f58e3 in QObject::QObject () from > /usr/local/lib/qt4/libQtCore.so.4 > #6 0x281d4710 in QThread::QThread () from > /usr/local/lib/qt4/libQtCore.so.4 > #7 0x281d5a9e in QAdoptedThread::QAdoptedThread () from > /usr/local/lib/qt4/libQtCore.so.4 > #8 0x281d7934 in QThreadData::current () from > /usr/local/lib/qt4/libQtCore.so.4 > #9 0x281d4747 in QThread::currentThread () from > /usr/local/lib/qt4/libQtCore.so.4 > #10 0x28097248 in QDBusConnectionPrivate::deleteYourself () from > /usr/local/lib/qt4/libQtDBus.so.4 > #11 0x2808f2ea in QDBusConnection::~QDBusConnection () from > /usr/local/lib/qt4/libQtDBus.so.4 > #12 0x2864ad8f in __cxa_finalize () from /lib/libc.so.7 > #13 0x285efe1a in exit () from /lib/libc.so.7 > #14 0x08051db5 in main () > (gdb) > > dead-loop in QT library until the stack overflow. > > As I said, it depends on ordering the global objects are destructed, > if the object which deleting > the current_thread_data_key is destructed lastly, the problem wont > happen, but now > it is destructed too early. I believe there is no specification said > that which C++ object should be > destructed first if they are in different compiled module and then are > linked together to generated > a shared object, .so file. > > Now let me dig into qthread_unix.cpp, see how QThreadData::current() works: QThreadData *QThreadData::current() { QThreadData *data = get_thread_data(); if (!data) { void *a; if (QInternal::activateCallbacks(QInternal::AdoptCurrentThread, &a)) { QThread *adopted = static_cast(a); Q_ASSERT(adopted); data = QThreadData::get2(adopted); set_thread_data(data); adopted->d_func()->running = true; adopted->d_func()->finished = false; static_cast(adopted)->init(); } else { data = new QThreadData; QT_TRY { set_thread_data(data); data->thread = new QAdoptedThread(data); } QT_CATCH(...) { clear_thread_data(); data->deref(); data = 0; QT_RETHROW; } data->deref(); } if (!QCoreApplicationPrivate::theMainThread) QCoreApplicationPrivate::theMainThread = data->thread; } return data; } it calls get_thread_data(), if it returns NULL, it create a new thread, and try to set the new thread as "current thread data", it calls set_thread_data(). let's see how get_thread_data() and set_thread_data() work : static QThreadData *get_thread_data() { #ifdef Q_OS_SYMBIAN return reinterpret_cast(Dll::Tls()); #else pthread_once(¤t_thread_data_once, create_current_thread_data_key); return reinterpret_cast(pthread_getspecific(current_thread_data_key)); #endif } static void set_thread_data(QThreadData *data) { #ifdef Q_OS_SYMBIAN qt_symbian_throwIfError(Dll::SetTls(data)); #endif pthread_once(¤t_thread_data_once, create_current_thread_data_key); pthread_setspecific(current_thread_data_key, data); } They just use pthread_getspecific and pthread_setspecific, the current_thread_data_key was only created once which is guarded by pthread_once(), but as you know, the key has already been deleted by Q_DESTRUCTOR_FUNCTION(destroy_current_thread_data_key) which is a global object which has been destructed early, the key is no longer recreated, it is a stale key. So pthread_setspecific should fail, and pthread_getspecific would return NULL, this causes QThreadData::current() create a new thread, it seems QAdoptedThread also calls QThreadData::current(), so it is a recursion, the recursion is never ended until stack overflow.