From owner-freebsd-hackers@FreeBSD.ORG Thu Jan 10 21:03:23 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 67CA24BB; Thu, 10 Jan 2013 21:03:23 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id D2BF78C2; Thu, 10 Jan 2013 21:03:22 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.6/8.14.6) with ESMTP id r0AL3Dnr065186; Thu, 10 Jan 2013 23:03:13 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.4 kib.kiev.ua r0AL3Dnr065186 Received: (from kostik@localhost) by tom.home (8.14.6/8.14.6/Submit) id r0AL3DwD065185; Thu, 10 Jan 2013 23:03:13 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 10 Jan 2013 23:03:13 +0200 From: Konstantin Belousov To: Alfred Perlstein Subject: Re: malloc+utrace, tracking memory leaks in a running program. Message-ID: <20130110210313.GY2561@kib.kiev.ua> References: <50D52B10.1060205@mu.org> <50EE6281.7030602@mu.org> <50EE6630.2010902@mu.org> <20130110073854.GQ2561@kib.kiev.ua> <50EEDB5E.2010906@mu.org> <20130110180514.GS2561@kib.kiev.ua> <50EF0892.104@mu.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="KQ2iXOoQ638mtNze" Content-Disposition: inline In-Reply-To: <50EF0892.104@mu.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: Daniel Eischen , hackers@freebsd.org, Jason Evans X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jan 2013 21:03:23 -0000 --KQ2iXOoQ638mtNze Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jan 10, 2013 at 01:29:38PM -0500, Alfred Perlstein wrote: > On 1/10/13 1:05 PM, Konstantin Belousov wrote: > > On Thu, Jan 10, 2013 at 10:16:46AM -0500, Alfred Perlstein wrote: > >> On 1/10/13 2:38 AM, Konstantin Belousov wrote: > >>> On Thu, Jan 10, 2013 at 01:56:48AM -0500, Alfred Perlstein wrote: > >>>> Here are more convenient links that give diffs against FreeBSD and > >>>> jemalloc for the proposed changes: > >>>> > >>>> FreeBSD: > >>>> https://github.com/alfredperlstein/freebsd/compare/13e7228d5b83c8fcf= c63a0803a374212018f6b68~1...utrace2 > >>>> > >>> Why do you need to expedite the records through the ktrace at all ? > >>> Wouldn't direct write(2)s to a file allow for better performance > >>> due to not stressing kernel memory allocator and single writing threa= d ? > >>> Also, the malloc coupling to the single-system interface would be > >>> prevented. > >>> > >>> I believe that other usermode tracers also behave in the similar way, > >>> using writes and not private kernel interface. > >>> > >>> Also, what /proc issues did you mentioned ? There is > >>> sysctl kern.proc.vmmap which is much more convenient than /proc/pid/m= ap > >>> and does not require /proc mounted. > >>> > >>>> jemalloc: > >>>> https://github.com/alfredperlstein/jemalloc/compare/master...utrace2 > >>>> > >> Konstantin, you are right, it is a strange thing this utrace. I am not > >> sure why it was done this way. > >> > >> You are correct in that much more efficient system could be made using > >> writes gathered into a single write(2). > > Even without writes gathering, non-coalesced writes should be faster th= an > > utrace. > > > >> Do you think there is any reason they may have re-used the kernel paths > >> for ktrace even at the cost of efficiency? > > I can only speculate. The utracing of the malloc calls in the context > > of the ktrace stream is useful for the human reading the trace. Instead > > of seeing the sequence of unexplanaible calls allocating and freeing > > memory, you would see something more penetrable. For example, you would > > see accept/malloc/read/write/free, which could be usefully interpreted > > as network server serving the client. > > > > This context is not needed for a leak detector. > Now I may be wrong here, but I think it's an artifact of someone=20 > noticing how useful fitting this into the ktrace system and leveraging=20 > existing code. >=20 > Even though there are significant performance deficiencies, the actual=20 > utility of the existing framework may have been such a stepping stool=20 > towards tracing that it was just used. >=20 > Right now the code already exists, however it logs just {operation,=20 > size, ptr}, example: > malloc, 512, -> 0xdeadbeef > free, 0, 0xdeadbeef > realloc, 512, 0 -> 0xdeadc0de > realloc, 1024, 0xdeadc0de -> 0xffff0000 > free, 0, 0xffff0000 >=20 > What do you think of just adding the address of the caller of=20 > malloc/free/realloc to these already existing tracepoints? In most real-world applications I saw, malloc() was not a function called to do the allocation. Usually, there is either an app-specific wrapper, or the language runtime system which calls malloc(), e.g. the new operator for the C++ code. Than, the caller address becomes constant for the whole duration of the program run. What would be useful is the full backtrace of each allocation. The tools like libunwind are indeed optimized for this usage pattern. =46rom this POV, the libc malloc(3) might be better offering a set of the well-defined hooks for a pluggable tracer to utilize. I am on the fence there, you could override the malloc/free without hooks, by the ELF symbol interposing technique, but hooks would also offer features not easily implementable with the interposing. --KQ2iXOoQ638mtNze Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ7yyQAAoJEJDCuSvBvK1B+ykP/3o2xJU5OBePwhJepRkiE8AL +y5m9TDUwJ8abK+lnyDvq2c+uK3AiaanTvM/fqrnvq6Y8oN1cw6l7dQKWR5Eja9p bjB5lo80FfZYM5lvdZm51inNPW+E/8QeEeib/LEm9XkTbHmTzRdlH3W63mGNjRSA alYtLcYuCnWD7Y/XXXB1P53Lt0kiE4Ldsu48cprcWbOtShiyrnW2WbNoOItyMG7y 5Fnn6SmRMGvwCZBiuxY4kYcL6Vbx5SNrxDp1rkyY2b7nJVQADbb3vLAjaDTg4eIJ GxCU3F2NTNKkU+hk6wXDFMBIeQozbQ41IupdpqbUTFG+guLCdtFSV6Bjk2+Izq8m i3jc4E3+OvZVWq9zCbKgB7uR7XpVf7fsE8PPhzQvVFMeNokH+XZvw1r//m+09fQz NOTjkjFIe2QtO+5DEcP9b8Aanxwov5YF7zh23mZmtM/G46OzlpJ+qf+SuiEymtzs zwQ9HfFwCQ3GvFIAS4ObjJ6ClWn4+LvJpilFkby0+mggzNGeuahaw32/92k8F3ow Q2r6pLSfLELgB/kAZP4vtLQmGt5E+gs7R/at/cguk3smiapFp4T0HzVso9FZufHh hhAuQacm2ywOO+lT6BAFQ7mNpq+n9DPQC0OVv0lt5kHNZQIHO0OMtEMbz+3nAKsk i8jMPEIyto6tvUlb5puN =K1XI -----END PGP SIGNATURE----- --KQ2iXOoQ638mtNze--