From owner-freebsd-hackers Sun Feb 25 23:10:18 2001 Delivered-To: freebsd-hackers@freebsd.org Received: from earth.backplane.com (earth-nat-cw.backplane.com [208.161.114.67]) by hub.freebsd.org (Postfix) with ESMTP id 9FB8D37B491 for ; Sun, 25 Feb 2001 23:10:14 -0800 (PST) (envelope-from dillon@earth.backplane.com) Received: (from dillon@localhost) by earth.backplane.com (8.11.2/8.9.3) id f1Q79fq30005; Sun, 25 Feb 2001 23:09:41 -0800 (PST) (envelope-from dillon) Date: Sun, 25 Feb 2001 23:09:41 -0800 (PST) From: Matt Dillon Message-Id: <200102260709.f1Q79fq30005@earth.backplane.com> To: seebs@plethora.net (Peter Seebach) Cc: freebsd-hackers@FreeBSD.ORG Subject: Re: Setting memory allocators for library functions. References: <200102251559.f1PFx2627103@guild.plethora.net> Sender: owner-freebsd-hackers@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG :In message <3A98EE37.7B0B6CE0@newsguy.com>, "Daniel C. Sobral" writes: :>OTOH, the *only* way to get non-overcommit to FreeBSD is for someone who :>*wants* that feature to sit down and code it. It won't happen otherwise. : :So, out of idle curiousity: If, somewhere down the road, I know the kernel :well enough to attempt such a thing, what would the interest level be in :merging such a feature? : :-s The problem is a whole lot more complex then you think. Dealing with overcommit is not simply counting mapped pages, there are all sorts of issues involved. But the biggest gotcha is that putting in overcommit protection will not actually save your system from dying a terrible death. It in fact makes it *more* likely that the system will die a horrible death, because with overcommit protection you wind up pre-reserving resources that would otherwise be reserved on demand (and often might not ever be used by the programs mapping those resources). Not only that, but overcommit protection does not protect you from thrashing, and quite often the system will become unusable long before it actually runs out of memory+swap. This is true whether you have configured swap or not... it is perfectly easy to thrash a swapless system if the vast majority of pages are clean (e.g. you mmap() a lot of files read-only). A simple example of this is mmap(... PROT_READ|PROT_WRITE, MAP_PRIVATE), or even simply the static data and bss space associated with a program binary. Such memory only takes up physical space if you dirty it...you can *read* the memory all you want without having to reserve any resources, the OS can throw the physical page away at any time and reload it form the backing file. But the moment you dirty the memory the OS must allocate real resources to it ... either a physical page, or swap (and it can flip between the two). Many such mappings never actually require dirtying the underlying page and thus never actually take any resources. But with overcommit protection you have to reserve the resources immediately, even if the program will never use them. The result is that your system is likely to run out of memory long before it would without the overcommit protection. There are many other issues involved as well that are more complex, such as what you have to do when a program fork()s. So before you get into tring to 'protect' yourself by implementing overcommit protection, you need to think long and hard on what you are actually protecting yourself from... and what you aren't. People seem to believe that edge cases such as allocating memory and then the system faulting the process later because it couldn't allocate the actual backing store later is of paramount importance but they seem to forget that by the time you reach that situation, you are generally already past the point of no return. Overcommit protection doesn't even come close to being able to prevent your system from getting into an unrecoverable situation and it certainly is no substitute for writing the software correctly (such that the memory failure can't occur in the first place). -Matt To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message