Date: Thu, 7 Dec 2000 12:59:55 +0100 (CET) From: Remy Nonnenmacher <remy@boostworks.com> To: dillon@earth.backplane.com Cc: phk@critter.freebsd.dk, freebsd-hackers@FreeBSD.ORG Subject: Re: free() not freing pagedirs pages. Message-ID: <200012071200.NAA35634@luxren2.boostworks.com> In-Reply-To: <200012062139.eB6LdkE99051@earth.backplane.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 6 Dec, Matt Dillon wrote:
> :
> :OK. In fact my problem was just a printf that allocated a buffer via
> :__smakebuf at the very last moment (when all memory was allocated).
> :This prevent free() to give back all previous pages up to this one. The
> :problem was _not_ in malloc.c.
> :
> :Anyway, i learned a lot from hacking the source to catch the caller.
> :Thanks.
> :
> :RN.
> :IhM
>
> Not to mention library routines which might malloc() something and
> keep it around.
>
> I find that the best way to allocate a large chunk of memory is to
> use mmap( ... MAP_ANON ... ). That way you have complete control
> over the memory. You need only page-align the request size
> (getpagesize() helps there). You can then manage the memory with
> madvise(), and free it completely with munmap().
>
> -Matt
>
Well, I may think using this solution if it remains portable between
Unixes. I finally tracked down the problem, after suppressing the
reason to call __smakebuf and tooling malloc.c.
What happens is that malloc() uses the pages to store pginfo chains. If
all memory is used, it allocates high addresses pages and (seems
to) keeps these pages when all memory have been freed().
Here is a case: In the following example, about 60 Mbytes have been
allocated then freed. Only one structure remains allocated at the very
beginning of the memory (It is the first one allocated).
page_dir size is 69632 bytes (17408 entries).
Quiet all page_dir entries are either MALLOC_NOT_MINE (mainly backed
to system, at end of page_dir) or MALLOC_FREE:
(gdb) p *page_dir@17408
$33 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x815b000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1,
0x2, 0x3 <repeats 19 times>, 0x1 <repeats 65 times>, 0x815b000,
0x1 <repeats 111 times>, 0x81cb000, 0x1 <repeats 416 times>, 0x836c000,
0x1 <repeats 675 times>, 0x8610000, 0x1 <repeats 3228 times>, 0x92ad000,
0x1 <repeats 1181 times>, 0x974b000, 0x1 <repeats 587 times>, 0x9997000,
0x1 <repeats 1238 times>, 0x9e6e000, 0x1 <repeats 8398 times>,
0x0 <repeats 1468 times>}
Here is the chain if non-magic entries:
(gdb) p *page_dir[5]
$22 = {next = 0x81cb000, page = 0x815b000, size = 32, shift = 5, free = 121,
total = 127, bits = {4294963088}}
(gdb) p *page_dir[5]->next
$23 = {next = 0x836c000, page = 0x81cb000, size = 32, shift = 5, free = 126,
total = 127, bits = {4294967294}}
(gdb) p *page_dir[5]->next->next
$24 = {next = 0x8610000, page = 0x836c000, size = 32, shift = 5, free = 126,
total = 127, bits = {4294967294}}
(gdb) p *page_dir[5]->next->next->next
$25 = {next = 0x92ad000, page = 0x8610000, size = 32, shift = 5, free = 125,
total = 127, bits = {4294901758}}
(gdb) p *page_dir[5]->next->next->next->next
$26 = {next = 0x974b000, page = 0x92ad000, size = 32, shift = 5, free = 126,
total = 127, bits = {4294967294}}
(gdb) p *page_dir[5]->next->next->next->next->next
$27 = {next = 0x9997000, page = 0x974b000, size = 32, shift = 5, free = 126,
total = 127, bits = {4294967294}}
(gdb) p *page_dir[5]->next->next->next->next->next->next
$28 = {next = 0x9e6e000, page = 0x9997000, size = 32, shift = 5, free = 126,
total = 127, bits = {4294967294}}
(gdb) p *page_dir[5]->next->next->next->next->next->next->next
$29 = {next = 0x0, page = 0x9e6e000, size = 32, shift = 5, free = 126,
total = 127, bits = {4294967294}}
So the program appears as a memory hog on the 'top' view, from a
user perspective. I agree that letting malloc madvise() usage hints of
pages to the kernel would prevent/help swapping but it is difficult to
explain in comprehensible words to many customers.
Thanks for any hint about this situation.
RN.
IhM
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200012071200.NAA35634>