Date: Tue, 27 Feb 2001 04:14:10 +0900 From: "Daniel C. Sobral" <dcs@newsguy.com> To: Matt Dillon <dillon@earth.backplane.com> Cc: Archie Cobbs <archie@dellroad.org>, Warner Losh <imp@village.org>, Peter Seebach <seebs@plethora.net>, freebsd-hackers@FreeBSD.ORG Subject: Re: Setting memory allocators for library functions. Message-ID: <3A9AAB02.793A197A@newsguy.com> References: <200102260529.f1Q5T8413011@curve.dellroad.org> <200102260628.f1Q6SYX29811@earth.backplane.com> <3A9A0A9A.E4D31F97@newsguy.com> <200102261755.f1QHtvr34064@earth.backplane.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Matt Dillon wrote: > > :Matt Dillon wrote: > :> > :.. > :> the system runs out of memory, even *with* overcommit protection. > :> In fact, all sorts of side effects occur even when the system > :... > : > :That's an assumption. > > Ha. Right. Go through any piece of significant code and just see how > much goes flying out the window because the code wants to simply assume > things work. Then try coding conditionals all the way through to fix > it... and don't forget you need to propogate the error condition back > up the procedure chain too so the original caller knows why it failed. <sarcasm>Perhaps you should re-acquaint yourself with exception handlers, as you seem to have forgotten them since you last worked with your C compiler. You know, the kind of thing where you can put a longjmp() in a proxy malloc(), and keep a setjmp() at the appropriate functional level so that if _any_ allocation fails down that path the whole function is cancelled?</sarcasm> > : > :> There ain't no magic bullet here, folks. By the time you get a memory > :> failure, even with no overcommit, it is far too late to save the day. > : > :Scientific computation. If, at one point, no more memory can be > :allocated, you back off, save the present results, and try again later. > > This is irrelevant to the conversation. It has nothing to do with > overcommit in the context it is being discussed. It has everything to do with overcommit in the context it is being discussed. I might be mistaken on exactly who, but I think it was Peter Seebach himself who once complained about his scientific application dieing suddenly under out of memory conditions on FreeBSD, though working perfectly on Solaris. Said application allocated memory if possible, and, if not, saved temporary results and went do something else until more memory became available. Now, you say "By the time you get a memory failure, even with no overcommit, it is far too late to save the day". This example shows that, no, you are wrong, it is not far too late. It is possible, at this point, to deal with the lack of more memory. I'll give you one more example. Protocol validation. It is often impossible to test all possible permutations of a protocol's dialog, but being able to go as deep as possible on execution tree and then, when you go out of memory, giving up on that path, backing down and continuing elsewhere let you get a partial validation, which is not enough to prove a protocol is correct but might well be enough to prove it is incorrect. This is a real application, and one in which an out of memory condition is not only handled but even expected. > :You assume too much. Quite a few of the more efficient garbage > :collection algorithms depend on knowing when the memory has become full. > > This has nothing to do with overcommit in the context it is being > discussed. In fact, this has nothing to do with OS memory management > at all -- all garbage collected languages have their own infrastructure > to determine when memory pressure requires collecting. And, of course, those whose infrastructure depends on a malloc() returning NULL indicating the heap is full will not work on FreeBSD. (<sarcasm>You do recall that many of these languages are written in C, don't you?</sarcasm>) It has everything to do with overcommit. In this particular case, not only there _is_ something to do when the out of memory condition arise, but the very algorithm depends on it arising. And this is for some algorithms. Other algorithms require unbounded space for the collection but can handle out of memory conditions if they arise. > :You keep allocating, and when malloc() returns NULL, *then* you run the > :garbage collector, free some space, and try again. If malloc() doesn't > :work, quite a few very efficient garbage collection algorithms become > :impossible to implement. > > First, that's bullshit. Most garbage collection implementations > require the memory 'size' to be hardwired. Second, that's bullshit Garbage Collection: Algorithms for Automatic Memory Management, Richard Jones and Rafael Lins. Bullshit is what you just said. > because any program relying on that sort of operation had damn well > better have its datasize limit set to something reasonable, or the > garbage collector is going to run the system out of swap before it > decides it needs to do a run through. Yes, indeed that happens, and dealing with swapped out objects is one topic of study in gc algorithms. For some applications, it would result in trashing. For others, most of the swapped out pages consist entirely of garbage. > :Just because *you* don't see how one thing can work, it doesn't mean it > :can't work. As I have trivially shown above. > > You haven't shown anything above. Garbage collectors do not work that > way and you really should know it. You are obviously acquainted with an all too small subset of garbage collection algorithms. > :Honestly, I think non-overcommit is a mistake and your approach is much > :better, but it's not the only approach and there _are_ valid approaches > :that depend on not overcommitting, and I really hate having to defend > :non-overcommit against such bogus arguments. > > You've completely ignored the point that overcommit has nothing whatsoever > to do with memory pressure. You are assuming that overcommit is some > sort of magic bullet that will solve the memory pressure handling problem, > and it is nothing of the sort. Some applications *do* handle memory pressure, but they depend on being _told_ one is out of memory. -- Daniel C. Sobral (8-DCS) dcs@newsguy.com dcs@freebsd.org capo@kzinti.bsdconspiracy.net Acabou o hipismo-arte. Mas a desculpa brasileira mais ouvida em Sydney e' que nao tem mais cavalo bobo por ai'. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3A9AAB02.793A197A>