Date: Sun, 25 Feb 2001 23:09:41 -0800 (PST) From: Matt Dillon <dillon@earth.backplane.com> To: seebs@plethora.net (Peter Seebach) Cc: freebsd-hackers@FreeBSD.ORG Subject: Re: Setting memory allocators for library functions. Message-ID: <200102260709.f1Q79fq30005@earth.backplane.com> References: <200102251559.f1PFx2627103@guild.plethora.net>
next in thread | previous in thread | raw e-mail | index | archive | help
:In message <3A98EE37.7B0B6CE0@newsguy.com>, "Daniel C. Sobral" writes:
:>OTOH, the *only* way to get non-overcommit to FreeBSD is for someone who
:>*wants* that feature to sit down and code it. It won't happen otherwise.
:
:So, out of idle curiousity: If, somewhere down the road, I know the kernel
:well enough to attempt such a thing, what would the interest level be in
:merging such a feature?
:
:-s
The problem is a whole lot more complex then you think. Dealing with
overcommit is not simply counting mapped pages, there are all sorts
of issues involved. But the biggest gotcha is that putting in
overcommit protection will not actually save your system from
dying a terrible death. It in fact makes it *more* likely that the
system will die a horrible death, because with overcommit protection
you wind up pre-reserving resources that would otherwise be reserved on
demand (and often might not ever be used by the programs mapping
those resources). Not only that, but overcommit protection does not
protect you from thrashing, and quite often the system will become
unusable long before it actually runs out of memory+swap. This is
true whether you have configured swap or not... it is perfectly easy
to thrash a swapless system if the vast majority of pages are clean
(e.g. you mmap() a lot of files read-only).
A simple example of this is mmap(... PROT_READ|PROT_WRITE, MAP_PRIVATE),
or even simply the static data and bss space associated with a program
binary. Such memory only takes up physical space if you dirty it...you
can *read* the memory all you want without having to reserve any
resources, the OS can throw the physical page away at any time and reload
it form the backing file. But the moment you dirty the memory the OS
must allocate real resources to it ... either a physical page, or swap
(and it can flip between the two). Many such mappings never actually
require dirtying the underlying page and thus never actually take any
resources. But with overcommit protection you have to reserve the
resources immediately, even if the program will never use them. The
result is that your system is likely to run out of memory long before
it would without the overcommit protection. There are many other
issues involved as well that are more complex, such as what you have
to do when a program fork()s.
So before you get into tring to 'protect' yourself by implementing
overcommit protection, you need to think long and hard on what you
are actually protecting yourself from... and what you aren't. People
seem to believe that edge cases such as allocating memory and then
the system faulting the process later because it couldn't allocate the
actual backing store later is of paramount importance but they seem
to forget that by the time you reach that situation, you are generally
already past the point of no return. Overcommit protection doesn't
even come close to being able to prevent your system from getting
into an unrecoverable situation and it certainly is no substitute for
writing the software correctly (such that the memory failure can't
occur in the first place).
-Matt
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200102260709.f1Q79fq30005>