From owner-freebsd-hackers Tue Jul 13 12:25:10 1999 Delivered-To: freebsd-hackers@freebsd.org Received: from mail.netbsd.org (redmail.netbsd.org [155.53.200.193]) by hub.freebsd.org (Postfix) with SMTP id CABC014C8E for ; Tue, 13 Jul 1999 12:25:08 -0700 (PDT) (envelope-from cgd@netbsd.org) Received: (qmail 4202 invoked by uid 1000); 13 Jul 1999 19:24:36 -0000 To: Matthew Dillon Cc: freebsd-hackers@FreeBSD.ORG, tech-userlevel@netbsd.org Subject: Re: Replacement for grep(1) (part 2) References: <199907131753.KAA22111@lestat.nas.nasa.gov> <199907131813.LAA79534@apollo.backplane.com> From: cgd@netbsd.org (Chris G. Demetriou) Date: 13 Jul 1999 12:24:35 -0700 In-Reply-To: Matthew Dillon's message of Tue, 13 Jul 1999 11:13:49 -0700 (PDT) Message-ID: <873dys1hfw.fsf@redmail.redback.com> Lines: 87 X-Mailer: Gnus v5.5/Emacs 20.2 Sender: owner-freebsd-hackers@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG [cc list trimmed because it was getting ... insane, and it's not like this is a critical point. It's just beating up a topic which has been beaten up by many others.] Matthew Dillon writes: > Thus it makes little sense to try to disallow overcommit. It gains you > absolutely nothing, and forces you to waste huge amounts of disk space. > > To anyone who is paranoid about it: Fine, then just allocate an insane > amount of swap and forget about it. It would be no more then you would > have to allocate anyway if we were to actually disallow overcommits! But > with overcommits allowed, your box will never come close to using that > much swap. This may be a decent answer for the workstation world, but it's not so good for more restricted systems. Further, your claim that disallowing overcommit gains you absolutely nothing is simply false. It gains you two things (which are at least immediately obvious to me): * Certain knowledge that (if the system is implemented correctly) the system will never have to kill a process (or take similar corrective action) due to overcommit, and that attempts to allocate more backing store resources than are present will fail. If the programs you're using do reasonable error handling when out-of memory situations crop up, then this can be a very useful propery to have if you're building a reliable system. It's not necessarily a matter of gigs of swap space. It means that you know that you can safely run in, say, 64MB or memory, with _no_ swap, if your programs don't overallocate ridiculously. The only alternative to avoiding overcommit that seems plausibly enforceable by the OS/system designer is using resource limits and then planning based on the maximum resource limits and number of processes you allow. However, that results in a ridiculously loose upper bound and a fair bit of extra work tuning the limits. The obvious workstation example is a bit grotesque: 1GB max datasize by default, maxproc 532, and you need > .5TB to be _sure_ that no process will get killed for lack of resources. A more realistic embedded system might allow 64 processes, two of which need 10MB each and the rest of which need 300K-1MB each. If you do this naively, you end up "needing" 640MB to guarantee safely. If you tweak a bunch of things with some special-purpose resource limiting code, you can get it down to 96MB or so to guarantee safety. Only by going to a _great_ deal of extra effort -- which could be avoided by just disabling overcommit -- you can't be sure the system will operate safely in, say, 48MB or even 64MB, both of which may well be adequate. Using resource limits to solve this problem is ... not the right answer (even if various systems implemented resource limits correctly). * protection against bogosity. I may run a system in which all of the processes are effectively unlimited (i.e. have huge resource limits), but I know within a tolerance what the actual expected usage of the system is. If a program mallocs (or equivalently allocates with mmap) it should be told up front that that's not possible (or that it is), and filling that memory in later simply should not result in its death or in the death of another process. This becomes especially important when the process making the bogus request is a "critical" process or when the other process that happens to be killed is a critical process. Certain knowledge about system operation, and deterministinc system behaviour in the face of programming bugs is exactly what the desire for overcommit is about. Overcommit avoidance may not be useful for your particular uses of these UNIX-like systems. However, if you think that it's not useful to anybody who uses them (or that people who think it's useful are deluding themselves 8-), then you're sorely mistaken and have a ... very wrong-headed attitude about why people find such features useful. cgd -- Chris Demetriou - cgd@netbsd.org - http://www.netbsd.org/People/Pages/cgd.html Disclaimer: Not speaking for NetBSD, just expressing my own opinion. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message