Date: Mon, 26 Feb 2001 16:49:46 +0900 From: "Daniel C. Sobral" <dcs@newsguy.com> To: Matt Dillon <dillon@earth.backplane.com> Cc: Archie Cobbs <archie@dellroad.org>, Warner Losh <imp@village.org>, Peter Seebach <seebs@plethora.net>, freebsd-hackers@FreeBSD.ORG Subject: Re: Setting memory allocators for library functions. Message-ID: <3A9A0A9A.E4D31F97@newsguy.com> References: <200102260529.f1Q5T8413011@curve.dellroad.org> <200102260628.f1Q6SYX29811@earth.backplane.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Matt Dillon wrote: > > But its all a crock. It simply isn't possible to gracefully handle > an out of memory condition. All sorts of side effects occur when > the system runs out of memory, even *with* overcommit protection. > In fact, all sorts of side effects occur even when the system > *doesn't* run out of memory, but instead just starts to thrash swap. > All sorts of side effects occur if the system starts to get cornered > memory-wise even if you don't *have* any swap. The number of possible > combinations of effects is near infinite and nearly impossible to > program against. Simply put, the 'overcommit' argument doesn't > actually solve the problem in any meaningful way. Any significantly > sized program that actually handled every possible combination and > side effect of an out-of-memory condition gracefully would have so much > conditional garbage cluttering it up that the rest of the code would > be obscured in a haze of if()'s. That's an assumption. > There ain't no magic bullet here, folks. By the time you get a memory > failure, even with no overcommit, it is far too late to save the day. Scientific computation. If, at one point, no more memory can be allocated, you back off, save the present results, and try again later. > There is only one correct way to handle an out of memory condition and > that is to make sure it doesn't happen... so when it does happen you > can treat it as a fatal error and scream bloody murder. It's a whole lot > easier to design a program with bounded, deterministic memory use (e.g. > database X requires Y kilobytes of memory per client instance) and > control that use at the edges rather then to try to deal with memory > failures gracefully in the deepest darkest corners of the program. > And it's a whole lot more reliable, too. > > When I write such a program, if I care about bounding memory use > I do it at the edges. I don't pollute low level allocation routines > (like strdup(), small structural allocations, etc...) with all sorts > of conditionals to gracefully back out of a memory failure. It's a huge > waste of time. I just wrap the routines... safe_strdup(), safe_malloc(), > safe_asprintf()... and the wrappers scream bloody murder and exit if > they get a failure. It's that simple... and far more reliable. You assume too much. Quite a few of the more efficient garbage collection algorithms depend on knowing when the memory has become full. You keep allocating, and when malloc() returns NULL, *then* you run the garbage collector, free some space, and try again. If malloc() doesn't work, quite a few very efficient garbage collection algorithms become impossible to implement. Just because *you* don't see how one thing can work, it doesn't mean it can't work. As I have trivially shown above. Honestly, I think non-overcommit is a mistake and your approach is much better, but it's not the only approach and there _are_ valid approaches that depend on not overcommitting, and I really hate having to defend non-overcommit against such bogus arguments. You don't implement it because you don't think it's worth your time and you are not being paid to do so. That's a perfectly good reason, and if people would just stick with it the threads about this would die much sooner. -- Daniel C. Sobral (8-DCS) dcs@newsguy.com dcs@freebsd.org capo@kzinti.bsdconspiracy.net Acabou o hipismo-arte. Mas a desculpa brasileira mais ouvida em Sydney e' que nao tem mais cavalo bobo por ai'. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3A9A0A9A.E4D31F97>