From owner-freebsd-python@FreeBSD.ORG Mon Mar 21 12:52:18 2011 Return-Path: Delivered-To: freebsd-python@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4BB07106566B for ; Mon, 21 Mar 2011 12:52:18 +0000 (UTC) (envelope-from andymac@bullseye.apana.org.au) Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by mx1.freebsd.org (Postfix) with ESMTP id C46B58FC1F for ; Mon, 21 Mar 2011 12:52:17 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AosHAMPjhk15LdWF/2dsb2JhbACYcox+eMAjhWMEkDg Received: from ppp121-45-213-133.lns20.cbr1.internode.on.net (HELO bullseye.apana.org.au) ([121.45.213.133]) by ipmail06.adl6.internode.on.net with ESMTP; 21 Mar 2011 23:06:33 +1030 Received: from [192.168.63.10] (tenring.andymac.org [192.168.63.10]) by bullseye.apana.org.au (8.14.2/8.14.2) with ESMTP id p2LCq6U5020178 for ; Mon, 21 Mar 2011 23:52:07 +1100 (EST) (envelope-from andymac@bullseye.andymac.org) Message-ID: <4D8740D0.1020005@bullseye.andymac.org> Date: Mon, 21 Mar 2011 23:13:04 +1100 From: Andrew MacIntyre User-Agent: Thunderbird 2.0.0.23 (OS/2/20090822) MIME-Version: 1.0 To: freebsd-python@freebsd.org References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Python malloc issue? X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Mar 2011 12:52:18 -0000 Amit Dev wrote: > I'm observing a strange memory usage pattern with python strings on > Freebsd. Consider > the following session. Idea is to create a list which holds some > strings so that cumulative characters in the list is 100MB. > >>>> l = [] >>>> for i in xrange(100000): > ... l.append(str(i) * (1000/len(str(i)))) > > This uses around 100MB of memory as expected and 'del l' will clear that. > >>>> for i in xrange(20000): > ... l.append(str(i) * (5000/len(str(i)))) > > This is using 165MB of memory. I really don't understand where the > additional memory usage is coming from. [Size of both lists are same] > > Python 2.6.4 on FreeBSD 7.2. On Linux and windows both uses around > 100mb memory only. A lot depends on the behavior of the system malloc(), and also the reporting tools. Python's internal allocator is only used for objects up to 256 bytes; over that the request is passed to the platform malloc(). At a guess, a lot of your requests in the second case are over 4096 bytes (a single page of memory) and so the allocator is probably padding the requests out to 2 pages, which is roughly in proportion with the memory consumption you report. jemalloc's behaviour can be tailored a bit - you might want to read the man page. -- ------------------------------------------------------------------------- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac@bullseye.apana.org.au (pref) | Snail: PO Box 370 andymac@pcug.org.au (alt) | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia