Date: Thu, 15 Apr 1999 19:05:19 -0700 From: "David Schwartz" <davids@webmaster.com> To: "Daniel C. Sobral" <dcs@newsguy.com> Cc: <chat@FreeBSD.ORG> Subject: RE: swap-related problems Message-ID: <001101be87ad$94130900$021d85d1@whenever.youwant.to> In-Reply-To: <37167B00.3B40F1D7@newsguy.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> David Schwartz wrote: > > > > > First, all this discussion centers on a program P being able to > > > allocate all available memory. If a program doesn't try to do that, > > > it will never run into a malloc(3) returning NULL, pre-allocate or > > > not. Are we agreed on this? > > > > What does malloc returning NULL have to do with anything? *sigh* > > That would be the failed malloc(). It just happen to be the origin > and, as far as I knew, the subject of this thread. If it changed > along the way, sorry, I missed it. The idea is to make malloc return NULL (or fork) when the alternative would be to overcommit. So it has to do with malloc returning NULL _before_ the operating system runs into a serious overcommit situation. > > We're talking about a well-behaved program, P, that > gets killed because of > > overcommitting caused by some combination of its behavior and > the behavior > > of other processes. > > Sure, and then you say: > > ... > > an emergency pool. If the operating system returns NULL when we > call malloc, > > we defer memory-intensive tasks for later. In extreme cases, we > may refuse > ... > > and > > > No. A process can handle a failed malloc or fork by > doing something much > > less drastic than death. The kernel cannot. > > So, are we talking about malloc() returning NULL or not? We are talking about forcing malloc to return NULL. Not about what to do when/if malloc returns NULL. > If we are not setting a limit to a process' size, and a malloc > fails, then the memory will be full. Thus, what I said in "first". > > And let me repeat here, if this well behaved program does not > allocate all memory, it stands to reason that no malloc it tried > could have failed. If no malloc is failing, it is moot what the > program could do if it did fail. Correct. The idea is not to fail malloc but to prevent an overcommit situation from killing the process. The beauty of this setup is it can work even if no malloc ever fails. > > > Second, run program P, using pre-allocation. At some point in the > > > execution of P, all memory will allocated to it, and there will be > > > no more free memory available. Correct? > > > > No. It's possible that P will allocate all the memory > it needs at startup, > > and it will be a very small amount. What do you mean by "all memory will > > allocated to it"? > > If it allocates a small amount of memory, the memory won't get full, > and thus no process will get killed. Unless you have something else > behaving badly. If you have something else behaving badly, limit the > datasize of *that*. If you don't want to limit the datasize of that, > then call that P, and go back to "first". The problem is, you can't solve a problem in X by fixing the entire rest of the world. Not only that, but there is no sane algorithm for doing so. Suppose you have 64Mb of real memory, 512Mb of swap, and 2,500 users. You tell me what you limit them to. > As an aside, if P does not expand to fill all memory, but you are > still facing a situation where processes get killed because of > overcommit, it is simply that you don't have enough memory to run > what you are trying to run. No, it's simply that the operating system allowed a dangerous situation it could have avoided. > > It is false that something needs to get killed. It's > entirely possible that > > had the operating system anticipated this situation and handled it more > > smoothly, say by failing to fork or failing to malloc, other > well-behaved > > programs would have reduced their memory load. > > > > The idea is to make it possible for well behaved > programs to avoid this > > situation by anticipating it earlier, the moment the operating > system began > > to overcommit. > > Err, excuse me? Let's use a "numeric example", as we used to ask our > algebra professor (only to have it say "Given three numbers, a, b > and c, ..." :). > > We have 64 Mb of memory. We start running processes. At some point, > they will try to allocate more than 64 Mb of memory. Umm, are you assuming no swap? Without swap this whole thing doesn't work. Let's assume 512Mb of swap. Okay, so we try to allocate more than 64Mb of memory. > If at this point we make the malloc calls fail, we are > pre-allocating everything. This doesn't work because your > applications get to use a lot less memory than they would otherwise. > A few very specialized systems have use for that. Unix is not among > these systems, so pre-allocating systems are not relevant to this > discussion. Umm, we don't need to make mallocs fail. We can always swap dirty pages. So we're fine. > Thus, we let the applications allocate more than 64 Mb of memory > before making mallocs fail. This is overcommitting. At this point it > is theoretically possible that all memory will get touched, causing > a process to die. In practice, that is unlikely. > > Ok, so let the applications grow some more. At some point the system > will get dangerously close to actually facing a memory starvation > situation. Before that happens, the system will start failing > mallocs, so the situation does not worsen. Of course, the system > must be smart enough to let enough free memory so that normal > dirtying of pages already allocated won't cause the memory > starvation, even without any new mallocs. Exactly. > It is simple to decide what point is that. When you finish this as > your Ph.D. thesis, please send us a reference to your algorithm. The point is simple -- as soon as you have not enough swap to back all shared, copy on write pages. As an optimization, you can make it so that only 'critical' processes are backed by swap. It's up to the user to determine which process are critical. > > For example, one of the programs written by the company > I work for has an > > internal memory tracker. We pre-allocate a few megabytes of > memory to use as > > an emergency pool. If the operating system returns NULL when we > call malloc, > > we defer memory-intensive tasks for later. In extreme cases, we > may refuse > > to accept new incoming connections. Because of that, we can > avoid running > > into the situation where something needs to be killed. > > If the operating system returns NULL, it is either using the > algorithm you'll describe in your Ph.D. thesis, or the memory is > exhausted. If the later, some other process, which is not using > pre-allocated memory, might dirty a page causing something to get > killed. Right, some other process might dirty a page, but that shouldn't cause my well-behaved process to get killed. My process would prefer to have its malloc fail. In fact, if my malloc fails, I can even un-mmap some non-critical stuff (shrinking caches), saving the operating system from having to kill _anything_. All the operating system would have to do is refuse my malloc or fork if it doesn't have enough swap for that process to dirty every page. That's when you are overcommitted. > > The problem is, if the operating system incorrectly > assumes that every > > malloc or fork is vital, it sets itself up for a situation later where a > > copy-on-write will result in a process needing to be killed. > This need is > > solely the result of the operating system causing mallocs and forks to > > succeed in the past where their failure may not have been fatal > to anything. > > I agree. I eagerly await to read your thesis on how an OS can decide > when to stop. When it hasn't enough swap to back all 'vital' process' copy-on-write or fill-on-demand pages. The definition of vital is up to the implementation. 'owned by root' would be one definition. 'That successfully executed a certain system call' would be another. > > You are assuming that processes can do better than the > kernel on a failed > > allocation. This is so obviously false that I can't believe that you are > > even advancing it. > > Much on the contrary. I'm assuming the kernel cannot arbitrarily > choose something other than memory full condition to fail on a > malloc(), because it doesn't know anything about what the process > needs or not. That's an absolutely retarded assumption, and certainly not the assumption that many UNIXes use. > > No. A process can handle a failed malloc or fork by > doing something much > > less drastic than death. The kernel cannot. > > Well... ok, what would you have the kernel do? Please describe the > entire scenario, not just one process. How much real memory exists? > How much got allocated? When did the mallocs start to fail? How was > each process memory allocated (pre-allocated or on-demand). For the 400th time, the mallocs start to fail in 'critical' processes as soon as the kernel doesn't have enough physical memory plus swap to back all their allocations. The tradeoff is as follows: the process agrees not to take as much memory as the operating system might be willing to allocate to it and in exchange, the operating system agrees not to kill off the process due to memory conditions beyond that process' control. This gives the operating system the benefit of failing mallocs early and perhaps preventing it from having to kill off processes at all. This gives the vital process the benefit of not being killed off due to overcommitment -- the idea is to accept fewer resources in exchange for those resources being guaranteed. If the process is smart enough to do something sensible when malloc fails, such as return non-critical memory to the operating system, everybody wins. If it's not, nobody loses, the process simply gets less memory than it would otherwise want. Odds are, if it's well behaved, it won't be the one the kernel would have killed off anyway. Are you deliberately being dense? Or do you have some weird hidden agenda? DS To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-chat" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?001101be87ad$94130900$021d85d1>