Date: Mon, 26 Feb 2001 11:40:19 -0800 (PST) From: Matt Dillon <dillon@earth.backplane.com> To: "Daniel C. Sobral" <dcs@newsguy.com> Cc: Archie Cobbs <archie@dellroad.org>, Warner Losh <imp@village.org>, Peter Seebach <seebs@plethora.net>, freebsd-hackers@FreeBSD.ORG Subject: Re: Setting memory allocators for library functions. Message-ID: <200102261940.f1QJeJi38115@earth.backplane.com> References: <200102260529.f1Q5T8413011@curve.dellroad.org> <200102260628.f1Q6SYX29811@earth.backplane.com> <3A9A0A9A.E4D31F97@newsguy.com> <200102261755.f1QHtvr34064@earth.backplane.com> <3A9AAB02.793A197A@newsguy.com>
next in thread | previous in thread | raw e-mail | index | archive | help
:> :...
:> :
:> :That's an assumption.
:>
:> Ha. Right. Go through any piece of significant code and just see how
:> much goes flying out the window because the code wants to simply assume
:> things work. Then try coding conditionals all the way through to fix
:> it... and don't forget you need to propogate the error condition back
:> up the procedure chain too so the original caller knows why it failed.
:
:<sarcasm>Perhaps you should re-acquaint yourself with exception
:handlers, as you seem to have forgotten them since you last worked with
:your C compiler. You know, the kind of thing where you can put a
:longjmp() in a proxy malloc(), and keep a setjmp() at the appropriate
:functional level so that if _any_ allocation fails down that path the
:whole function is cancelled?</sarcasm>
<sarcasm> ... just try *proving* that sort of code. Good luck! I'm
trying to imagine how to QA code that tries to deal with memory failures
at any point and uses longjump, and I'm failing.
:It has everything to do with overcommit in the context it is being
:discussed. I might be mistaken on exactly who, but I think it was Peter
:Seebach himself who once complained about his scientific application
:dieing suddenly under out of memory conditions on FreeBSD, though
:working perfectly on Solaris. Said application allocated memory if
:possible, and, if not, saved temporary results and went do something
:else until more memory became available.
Said application was poorly written, then. Even on solaris if you
actually run the system out of memory you can blow up other unrelated
processes. To depend on that sort of operation is just plain dumb.
At *best*, even on solaris, you can set the datasize limit for the
process and try to manage memory that way. It is still a bad idea
though because there's a chance that any libc call you make inside
your core code, even something is innocuous as a printf(), may fail.
The proper way to manage memory with this sort of application is to
specify a hard limit in the program itself, and have the program
keep track of its own useage and save itself off if it hits the
hard limit. Alternatively you can monitor system load and install
a signal handler to cause the program to save itself off and exit if
the load gets too high... something that will occur long before
the system actually runs out of memory.
:I'll give you one more example. Protocol validation. It is often
:impossible to test all possible permutations of a protocol's dialog, but
:being able to go as deep as possible on execution tree and then, when
:you go out of memory, giving up on that path, backing down and
:continuing elsewhere let you get a partial validation, which is not
:enough to prove a protocol is correct but might well be enough to prove
:it is incorrect. This is a real application, and one in which an out of
:memory condition is not only handled but even expected.
This has nothing to do with memory overcommit. Nothing at all. What
is your definition of out-of-memory? When swap runs out, or when the
system starts to thrash? What is the point of running a scientific
calculation if the machine turns into a sludge pile and would otherwise
cause the calculation to take years to complete instead of days?
You've got a whole lot more issues to deal with then simple memory
overcommit, and you are ignoring them completely.
:And, of course, those whose infrastructure depends on a malloc()
:returning NULL indicating the heap is full will not work on FreeBSD.
:(<sarcasm>You do recall that many of these languages are written in C,
:don't you?</sarcasm>)
Bullshit. If you care, a simple wrapper will do what you want. Modern
systems tend to have huge amounts of swap. Depending on malloc to
fail with unbounded resources in an overcommit OR a non-overcommit case
is stupid, because the system will be thrashing heavily long before it
even gets to that point.
Depending on malloc() to fail by setting an appropriate datasize limit
resource is more reasonable, and malloc() does work as expected if you
do that.
:It has everything to do with overcommit. In this particular case, not
:only there _is_ something to do when the out of memory condition arise,
:but the very algorithm depends on it arising.
It has nothing to do with overcommit. You are confusing overcommit
with hard-datasize limits, which can be set with a simple 'limit'
command.
:
:Garbage Collection: Algorithms for Automatic Memory Management, Richard
:Jones and Rafael Lins. Bullshit is what you just said.
None of which requires overcommit. None of which would actually
work in a real-world situation with or without overcommit if you do
not hard-limit the memory resource for the program in the first place.
You are again making the mistake of assuming that not having overcommit
will magically solve all your problems. It doesn't even come close.
You think these garbage collection algorithms work by running the
system out of VM and then backing off? That's pure nonsense.
-Matt
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200102261940.f1QJeJi38115>