Date: Mon, 26 Feb 2001 09:06:34 -0600 From: seebs@plethora.net (Peter Seebach) To: freebsd-hackers@freebsd.org Subject: Re: Setting memory allocators for library functions. Message-ID: <200102261506.f1QF6Y607763@guild.plethora.net> In-Reply-To: Your message of "Sun, 25 Feb 2001 23:09:41 PST." <200102260709.f1Q79fq30005@earth.backplane.com>
next in thread | previous in thread | raw e-mail | index | archive | help
In message <200102260709.f1Q79fq30005@earth.backplane.com>, Matt Dillon writes: > The problem is a whole lot more complex then you think. Dealing with > overcommit is not simply counting mapped pages, there are all sorts > of issues involved. But the biggest gotcha is that putting in > overcommit protection will not actually save your system from > dying a terrible death. No, but it may allow me to stop and save some work and then gracefully suicide, rather than being segfaulted. I have seen plenty of designs which are able to say "give me another 32MB of memory. Oh, you can't? Okay, I'll do this the other way." Most importantly, though, I would rather be able to save my work and *THEN* abort because there's insufficient memory. Let's compare FreeBSD's handling of this situation to the handling we get under one of the most unstable pieces of crap imaginable: MacOS. When I run out of memory on my Mac, you know what happens? I get little warning boxes saying there isn't enough memory to complete a given operation. Almost all the time, I am then allowed to continue *using* the applications I have open. Indeed, the application which *gave* me the error message is still able to run... long enough for me to save my work and close a few windows. Apparently, the "desirable" behavior under FreeBSD would be for the application to believe it had memory, then suddenly abort without warning, and with no chance to save my work. >It in fact makes it *more* likely that the > system will die a horrible death, because with overcommit protection > you wind up pre-reserving resources that would otherwise be reserved on > demand (and often might not ever be used by the programs mapping > those resources). Sure. But I want to know *UP FRONT* when I have problems. If I am trying to allocate too much memory, for God's sake, *TELL ME*. Give me warnings and errors. Make my program thrash, so I start wondering where the swap space went, and I can try to fix the problem, but don't just randomly kill processes that may or may not have done anything "wrong". >Not only that, but overcommit protection does not > protect you from thrashing, and quite often the system will become > unusable long before it actually runs out of memory+swap. I am okay with saving my work slowly. I am not okay with not saving my work. > A simple example of this is mmap(... PROT_READ|PROT_WRITE, MAP_PRIVATE), I am not sure that any promises have ever been made about the stability of mmap()'d memory. :) > So before you get into tring to 'protect' yourself by implementing > overcommit protection, you need to think long and hard on what you > are actually protecting yourself from... and what you aren't. People > seem to believe that edge cases such as allocating memory and then > the system faulting the process later because it couldn't allocate the > actual backing store later is of paramount importance but they seem > to forget that by the time you reach that situation, you are generally > already past the point of no return. This is quite possibly the case. On the other hand, in theory, the system could limit the total number of dirtyable pages such that it has a guarantee of enough space to swap... or at least promise to only kill programs that haven't had the "don't kill me" flag set. There are real applications, if only a few, where this would make a substantial difference. There are lots where it would at least provide warm fuzzies, and a checkbox conformance item that some one may have specified. >Overcommit protection doesn't > even come close to being able to prevent your system from getting > into an unrecoverable situation and it certainly is no substitute for > writing the software correctly (such that the memory failure can't > occur in the first place). Unfortunately, as has been made abundantly clear, you can't write software such that a memory failure can't occur in the first place. If someone else writes a program that allocates a giant chunk of memory, then slowly goes through dirtying pages, it is quite likely that a correctly written program will eventually be killed. If, indeed, FreeBSD might also kill a program for dirtying bss space, you can't write *ANY* program "correctly"; you can always get a SIGSEGV for accessing an object that was in no way distinguishable from an object you have permission to access. -s To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200102261506.f1QF6Y607763>