From owner-freebsd-stable@FreeBSD.ORG Sat Feb 23 21:43:27 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EA21C16A403 for ; Sat, 23 Feb 2008 21:43:27 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay03.kiev.sovam.com (relay03.kiev.sovam.com [62.64.120.201]) by mx1.freebsd.org (Postfix) with ESMTP id 9762E13C448 for ; Sat, 23 Feb 2008 21:43:22 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.226] (helo=skuns.kiev.zoral.com.ua) by relay03.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1JT29O-0004zn-1Q; Sat, 23 Feb 2008 23:43:20 +0200 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by skuns.kiev.zoral.com.ua (8.14.1/8.14.1) with ESMTP id m1NLgqdX020240 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 23 Feb 2008 23:42:52 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m1NLhExQ017409; Sat, 23 Feb 2008 23:43:14 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2/Submit) id m1NLhDIt017284; Sat, 23 Feb 2008 23:43:13 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 23 Feb 2008 23:43:13 +0200 From: Kostik Belousov To: "Arun Balakrishnan (WT01 - Computing, Storage & Software Products)" Message-ID: <20080223214313.GF57756@deviant.kiev.zoral.com.ua> References: <47C00A1B.5030708@wipro.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="gDKptnH9ef5/6wqy" Content-Disposition: inline In-Reply-To: <47C00A1B.5030708@wipro.com> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: ClamAV version 0.91.2, clamav-milter version 0.91.2 on skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.3 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on skuns.kiev.zoral.com.ua X-Scanner-Signature: 6bea4dd5e7a98ea238edf764fb0a12e1 X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Header: Not Detected X-SpamTest-Info: Profiles 2291 [Feb 22 2008] X-SpamTest-Info: helo_type=3 X-SpamTest-Method: none X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0278], KAS30/Release Cc: kan@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Memory Leak under FreeBSD 6.0 RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Feb 2008 21:43:28 -0000 --gDKptnH9ef5/6wqy Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Feb 23, 2008 at 05:27:15PM +0530, Arun Balakrishnan (WT01 - Computi= ng, Storage & Software Products) wrote: >=20 > Hi, > We are currently working on a project wherein we are porting a library > from GNU/Linux to FreeBSD 6.0 - RELEASE 32-bit and 64-bit. As part of > the standard memory leak tests, we noticed that the ported library is > leaking memory. After lots of analysis we found something very > strange. Just repeatedly loading and unloading our library was itself > throwing up a leak. We are able to reproduce a similar leak using the > following steps: > 1. SimpleLib.cpp - Simple dummy library > 2. LibLoader.cpp - Utility to repeatedly load the library > 3. Compile as mentioned > 4. Run under Valgrind for multiple times (31 times in our example. > Hard coded for simpilicity) > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3DSimpleLib.cpp=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > #include > #include > class CLeaker > { > public: > CLeaker() { }; > virtual ~CLeaker() { }; > }; > CLeaker obj; > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3DLibLoader.cpp=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > #include "stdio.h" > #include "dlfcn.h" > #include > #include > #include > int main() > { > int i =3D 0; > int loop =3D 31; > while (i { > i++; > void *handle =3D dlopen(argv[1], RTLD_LAZY); > if ( !handle ) > exit(1); > dlclose(handle); > } > return 0; > } > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > =3D=3D > Compilation: > g++ -shared -Wl,-soname,SimpleLib.so -o SimpleLib.so SimpleLib.cpp -g > g++ -o LibLoader_FreeBSD LibLoader.cpp -g > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > =3D=3D=3D > Execution: > valgrind --trace-pthread=3Dall --show-below-main=3Dyes > --show-reachable=3Dyes --leak-check=3Dyes ./LibLoader_FreeBSD > ./SimpleLib.so > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > =3D=3D=3D > Output: (snipped off irrelevant portions) > =3D=3D1155=3D=3D ERROR SUMMARY: 0 errors from 0 contexts (suppressed: = 0 from > 0) > =3D=3D1155=3D=3D malloc/free: in use at exit: 520 bytes in 1 blocks. > =3D=3D1155=3D=3D malloc/free: 1 allocs, 0 frees, 520 bytes allocated. > =3D=3D1155=3D=3D For counts of detected errors, rerun with: -v > =3D=3D1155=3D=3D searching for pointers to 1 not-freed blocks. > =3D=3D1155=3D=3D checked 2140912 bytes. > =3D=3D1155=3D=3D > =3D=3D1155=3D=3D 520 bytes in 1 blocks are still reachable in loss rec= ord 1 of > 1 > =3D=3D1155=3D=3D at 0x3C032183: malloc (in > /usr/local/lib/valgrind/vgpreload_memcheck.so) > =3D=3D1155=3D=3D by 0x3C1CB018: (within /lib/libc.so.6) > =3D=3D1155=3D=3D by 0x3C1CB206: __cxa_atexit (in /lib/libc.so.6) > =3D=3D1155=3D=3D by 0x3C1F0898: ??? > =3D=3D1155=3D=3D > =3D=3D1155=3D=3D LEAK SUMMARY: > =3D=3D1155=3D=3D definitely lost: 0 bytes in 0 blocks. > =3D=3D1155=3D=3D possibly lost: 0 bytes in 0 blocks. > =3D=3D1155=3D=3D still reachable: 520 bytes in 1 blocks. > =3D=3D1155=3D=3D suppressed: 0 bytes in 0 blocks. > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > =3D=3D=3D > Queries: > 1. As seen in the Valgrind output, there is a 520bytes leak. This > happens only after around 31 loops and keeps increasing. By 100 loops, > the leak goes up to 1560 bytes. In our situation with our library, the > 520bytes leak starts by the third iteration itself and by around 23 > iterations it reaches 5KB. We are really stumped as to what could be > the possible reason for this leak? Where is the malloc called from? > Why only after executing 31 times? Executing the same code under > GNU/Linux does not show any leak even for over 1000 iterations. > 2. While executing this without Valgrind, in another terminal we did a > "ps -Aopid,rss | grep LibLoader_" continuously in a loop and saw that > the RSS (resident set size) field value keeps increasing by 4KB every > now and then. The same experiment on GNU/Linux shows that RSS remains > at the same value. What could be the cause for the ever rising RSS > value? > Any help in this regard would be really helpful. Thanks in advance. > Rgds, > ~Arun The valgrind report points to memory used by the atexit_register() for keeping the information on the functions registered by means of atexit(3) and __cxa_atexit(). See the lib/libc/stdlib/atexit.c. In your (non-compilable) example, __cxa_atexit() is used by shared objects to register the destructor for global objects to be called at the dso unload. The handling of the memory is complicated because atexit() specification states that: - functions shall be called in the reverse order of their registration; - at least 32 functions can be registered with atexit(). The current implementation never frees the struct atexit to try to conform to the requirement of order. The static __atexit0, intended to guarantee success of the first 32 atexit() calls, may not guarantee it, because the space can be consumed by the interleaved __cxa_atexit() instead. Patch below may help with the libc leak. diff --git a/lib/libc/stdlib/atexit.c b/lib/libc/stdlib/atexit.c index 05dad84..8389637 100644 --- a/lib/libc/stdlib/atexit.c +++ b/lib/libc/stdlib/atexit.c @@ -41,6 +41,7 @@ __FBSDID("$FreeBSD: src/lib/libc/stdlib/atexit.c,v 1.8 20= 07/01/09 00:28:09 imp E #include #include #include +#include #include "atexit.h" #include "un-namespace.h" =20 @@ -56,7 +57,7 @@ static pthread_mutex_t atexit_mutex =3D PTHREAD_MUTEX_INI= TIALIZER; #define _MUTEX_UNLOCK(x) if (__isthreaded) _pthread_mutex_unlock(x) =20 struct atexit { - struct atexit *next; /* next in list */ + LIST_ENTRY(atexit) link; int ind; /* next index in this table */ struct atexit_fn { int fn_type; /* ATEXIT_? from above */ @@ -69,7 +70,10 @@ struct atexit { } fns[ATEXIT_SIZE]; /* the table itself */ }; =20 -static struct atexit *__atexit; /* points to head of LIFO stack */ +/* Head of LIFO stack */ +LIST_HEAD(, atexit) __atexit =3D LIST_HEAD_INITIALIZER(__atexit); +static struct atexit __atexit0; /* one guaranteed table */ +static unsigned long __atexit_gen; =20 /* * Register the function described by 'fptr' to be called at application @@ -79,30 +83,33 @@ static struct atexit *__atexit; /* points to head of L= IFO stack */ static int atexit_register(struct atexit_fn *fptr) { - static struct atexit __atexit0; /* one guaranteed table */ struct atexit *p; + unsigned long old__atexit_gen; =20 _MUTEX_LOCK(&atexit_mutex); - if ((p =3D __atexit) =3D=3D NULL) - __atexit =3D p =3D &__atexit0; - else while (p->ind >=3D ATEXIT_SIZE) { - struct atexit *old__atexit; - old__atexit =3D __atexit; - _MUTEX_UNLOCK(&atexit_mutex); - if ((p =3D (struct atexit *)malloc(sizeof(*p))) =3D=3D NULL) - return (-1); - _MUTEX_LOCK(&atexit_mutex); - if (old__atexit !=3D __atexit) { - /* Lost race, retry operation */ + if (LIST_EMPTY(&__atexit)) { + p =3D &__atexit0; + LIST_INSERT_HEAD(&__atexit, p, link); + } else { + retry: + p =3D LIST_FIRST(&__atexit); + if (p->ind >=3D ATEXIT_SIZE) { + old__atexit_gen =3D __atexit_gen; _MUTEX_UNLOCK(&atexit_mutex); - free(p); + if ((p =3D (struct atexit *)malloc(sizeof(*p))) =3D=3D NULL) + return (-1); _MUTEX_LOCK(&atexit_mutex); - p =3D __atexit; - continue; + if (old__atexit_gen !=3D __atexit_gen) { + /* Lost race, retry operation */ + _MUTEX_UNLOCK(&atexit_mutex); + free(p); + _MUTEX_LOCK(&atexit_mutex); + goto retry; + } + p->ind =3D 0; + LIST_INSERT_HEAD(&__atexit, p, link); + __atexit_gen++; } - p->ind =3D 0; - p->next =3D __atexit; - __atexit =3D p; } p->fns[p->ind++] =3D *fptr; _MUTEX_UNLOCK(&atexit_mutex); @@ -119,7 +126,7 @@ atexit(void (*func)(void)) int error; =20 fn.fn_type =3D ATEXIT_FN_STD; - fn.fn_ptr.std_func =3D func;; + fn.fn_ptr.std_func =3D func; fn.fn_arg =3D NULL; fn.fn_dso =3D NULL; =20 @@ -138,7 +145,7 @@ __cxa_atexit(void (*func)(void *), void *arg, void *dso) int error; =20 fn.fn_type =3D ATEXIT_FN_CXA; - fn.fn_ptr.cxa_func =3D func;; + fn.fn_ptr.cxa_func =3D func; fn.fn_arg =3D arg; fn.fn_dso =3D dso; =20 @@ -154,32 +161,55 @@ __cxa_atexit(void (*func)(void *), void *arg, void *d= so) void __cxa_finalize(void *dso) { - struct atexit *p; - struct atexit_fn fn; - int n; + struct atexit *p, *p1, cp; + struct atexit_fn *fn; + int i, n, inuse; + unsigned long orig__atexit_gen; =20 _MUTEX_LOCK(&atexit_mutex); - for (p =3D __atexit; p; p =3D p->next) { + restart: + inuse =3D 0; + LIST_FOREACH_SAFE(p, &__atexit, link, p1) { + cp.ind =3D 0; for (n =3D p->ind; --n >=3D 0;) { if (p->fns[n].fn_type =3D=3D ATEXIT_FN_EMPTY) continue; /* already been called */ - if (dso !=3D NULL && dso !=3D p->fns[n].fn_dso) + if (dso !=3D NULL && dso !=3D p->fns[n].fn_dso) { + inuse =3D 1; continue; /* wrong DSO */ - fn =3D p->fns[n]; + } + cp.fns[cp.ind++] =3D p->fns[n]; /* Mark entry to indicate that this particular handler has already been called. */ p->fns[n].fn_type =3D ATEXIT_FN_EMPTY; - _MUTEX_UNLOCK(&atexit_mutex); - =09 + } + if (!inuse && p !=3D &__atexit0) { + LIST_REMOVE(p, link); + __atexit_gen++; + } else { + /* + * The current entry cannot be removed, and so + * any consequent entries. + */ + inuse =3D 1; + p =3D NULL; + } + orig__atexit_gen =3D __atexit_gen; + _MUTEX_UNLOCK(&atexit_mutex); + free(p); + for (i =3D 0; i < cp.ind; i++) { + fn =3D &cp.fns[i]; /* Call the function of correct type. */ - if (fn.fn_type =3D=3D ATEXIT_FN_CXA) - fn.fn_ptr.cxa_func(fn.fn_arg); - else if (fn.fn_type =3D=3D ATEXIT_FN_STD) - fn.fn_ptr.std_func(); - _MUTEX_LOCK(&atexit_mutex); + if (fn->fn_type =3D=3D ATEXIT_FN_CXA) + fn->fn_ptr.cxa_func(fn->fn_arg); + else if (fn->fn_type =3D=3D ATEXIT_FN_STD) + fn->fn_ptr.std_func(); } + _MUTEX_LOCK(&atexit_mutex); + if (orig__atexit_gen !=3D __atexit_gen) + goto restart; } _MUTEX_UNLOCK(&atexit_mutex); } --gDKptnH9ef5/6wqy Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (FreeBSD) iEYEARECAAYFAkfAk3EACgkQC3+MBN1Mb4iYzgCgnIjlPDHsOllc5U33+sV5hceS L9MAnjXVjleHRdMhZhSytwoF6tr6Uk+2 =Dz5Y -----END PGP SIGNATURE----- --gDKptnH9ef5/6wqy--