From owner-freebsd-stable@FreeBSD.ORG Mon May 22 03:51:03 2006 Return-Path: <owner-freebsd-stable@FreeBSD.ORG> X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9A61616A420 for <freebsd-stable@freebsd.org>; Mon, 22 May 2006 03:51:03 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from fw.zoral.com.ua (ll-227.216.82.212.sovam.net.ua [212.82.216.227]) by mx1.FreeBSD.org (Postfix) with ESMTP id C994243D45 for <freebsd-stable@freebsd.org>; Mon, 22 May 2006 03:51:02 +0000 (GMT) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by fw.zoral.com.ua (8.13.4/8.13.4) with ESMTP id k4M3ouvk090224 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 22 May 2006 06:50:56 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.13.6/8.13.6) with ESMTP id k4M3ou4E036210; Mon, 22 May 2006 06:50:56 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.13.6/8.13.6/Submit) id k4M3otlB036209; Mon, 22 May 2006 06:50:55 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 22 May 2006 06:50:55 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: m m <needacoder@gmail.com> Message-ID: <20060522035055.GH54541@deviant.kiev.zoral.com.ua> References: <1e4841eb0605111757t36fdf8cfv267799a17dffc650@mail.gmail.com> <20060514195510.783fa765@kan.dnsalias.net> <1e4841eb0605152058n69d4e7b8m27d3944d9af8a346@mail.gmail.com> <1e4841eb0605211013u30d83617k73ea8551480a50a9@mail.gmail.com> <20060521211607.GG54541@deviant.kiev.zoral.com.ua> <1e4841eb0605211522n74d79ac9gee392857620e90f1@mail.gmail.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="pE2VAHO2njSJCslu" Content-Disposition: inline In-Reply-To: <1e4841eb0605211522n74d79ac9gee392857620e90f1@mail.gmail.com> User-Agent: Mutt/1.4.2.1i X-Virus-Scanned: ClamAV version 0.88.1, clamav-milter version 0.88.1 on fw.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-1.4 required=5.0 tests=ALL_TRUSTED autolearn=failed version=3.1.1 X-Spam-Checker-Version: SpamAssassin 3.1.1 (2006-03-10) on fw.zoral.com.ua Cc: freebsd-stable@freebsd.org, Alexander Kabaev <kabaev@gmail.com> Subject: Re: improper handling of dlpened's C++/atexit() code? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org> List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>, <mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable> List-Post: <mailto:freebsd-stable@freebsd.org> List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help> List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>, <mailto:freebsd-stable-request@freebsd.org?subject=subscribe> X-List-Received-Date: Mon, 22 May 2006 03:51:03 -0000 --pE2VAHO2njSJCslu Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, May 21, 2006 at 06:22:34PM -0400, m m wrote: > n 5/21/06, Konstantin Belousov <kostikbel@gmail.com> wrote: > >> >Program received signal SIGSEGV, Segmentation fault. > >> >0x00000000 in ?? () > >> >(gdb) bt > >> >#0 0x00000000 in ?? () > >> >#1 0x294c0ad8 in __do_global_dtors_aux () from > >> >/usr/local/lib/perl5/5.8.8/mach/auto/Sys/Syslog/Syslog.so > >> >#2 0x294c1d4c in _fini () from > >> >/usr/local/lib/perl5/5.8.8/mach/auto/Sys/Syslog/Syslog.so > >> >#3 0x280b4c80 in ?? () > >> >#4 0x280aaab8 in ?? () from /libexec/ld-elf.so.1 > >> >#5 0xbfbfe6e8 in ?? () > >> >#6 0x2808dca6 in objlist_call_fini (list=3D0x280a96d8) at > >> >/usr/src/libexec/rtld-elf/rtld.c:1336 > >> >#7 0x2808e1d4 in rtld_exit () at /usr/src/libexec/rtld-elf/rtld.c:15= 28 > >> >#8 0x281d58ea in __cxa_finalize (dso=3D0x0) at > >> >/usr/src/lib/libc/stdlib/atexit.c:184 > >> >#9 0x281d55ba in exit (status=3D0) at /usr/src/lib/libc/stdlib/exit.= c:69 > >> >#10 0x0805d0cb in clean_child_exit () > >> >#11 0x0805ea77 in just_die () > >> >#12 0x0805ea9a in usr1_handler () > >> >#13 0xbfbfffb4 in ?? () > >> >#14 0x0000001e in ?? () > >> >#15 0x00000000 in ?? () > >> >#16 0xbfbfe7c0 in ?? () > >> >#17 0x00000002 in ?? () > >> >#18 0x0805ea80 in just_die () > >> >#19 0x0806011e in child_main () > >> >#20 0x080607de in make_child () > >> >#21 0x08060868 in startup_children () > >> >#22 0x08060e81 in standalone_main () > >> >#23 0x08061702 in main () > > > >Could you, please, put somewhere: > >1. /usr/local/lib/perl5/5.8.8/mach/auto/Sys/Syslog/Syslog.so > >2. output of lsof -p <some apache child process pid> for apache running > >in your usual configuration. > > > >Also, could you run the apache with LD_PRELOAD=3D/usr/lib/libstdc++.so.5 > >and report whether the problem persists ? >=20 > Konstantin, > Thank you for looking into this. >=20 > lsof: http://www.savefile.com/files/6494253 > Syslog.so: http://www.savefile.com/files/2163369 >=20 > Although it's not an indicator of certainty (I have had it exit > cleanly in the past), it appears that running Apache with LD_PRELOAD > of libstdc++ does allow it to exit cleanly. Please let me know how I > can further assist. If it would make things easier - I can provide > access to a jail on this machine which exhibits the same behavior. Ok, I have a theory how it happens. Investigation of your instance of Syslog.so shows that crash happens at the following code of /usr/lib/crtbeginS.o: 282: if (__deregister_frame_info) 283: __deregister_frame_info (__EH_FRAME_BEGIN__); (this comes in from contrib/gcc/crtstuff.c, lines 282-283). Symbol __deregister_frame_info is weak and undefined in all your DSOs except libstdc++.so.5. This symbol provides part of the C++ runtime support for exception handling, and reasonably included from c++ runtime support library. Both lines 282 and 283 produce dynamic relocations in final DSO, but line 282 implies R_386_GLOB_DAT, and 283 - R386_JUMP_SLOT (for PLT). First relocation is resolved immediately on DSO load, second one is resolved on demand. My theory is that, at the time of loading Syslog.so, libstdc++.so.5 is loaded in the process, resulting in first relocation being satisfied by rtld immediately. But, at the time exit() processing comes to _fini() function of Syslog.so, libstdc++.so.5 is unloaded. And weak PLT relocation is resolved to 0. As result we got the frame #0 from your trace. This theory is confirmed by presence of libstdc++ in lsof output. Please, check that it does not show up at the time of crash dump by using "show shared" gdb command on crash dump. Short-time fix is to use LD_PRELOAD hack. The real solution would be to mark the libstdc++ DSO as unloadable and implement support for unloadable DSO in rtld (BTW, I think this is also needed for threading libraries libpthread and libthr for the same reason). I know that glibc dynamic loader has support for this feature. P.S. Apache seems to call exit(3) from the signal handler. This is wrong. --pE2VAHO2njSJCslu Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEcTUfC3+MBN1Mb4gRAh9jAJ427LSdYSEtPiQ8BOq9hPxF7HLaUwCg2g2a ZslOU1A5F1ERrTUJEl4JMlQ= =fZkR -----END PGP SIGNATURE----- --pE2VAHO2njSJCslu--