From owner-freebsd-hackers Tue Jul 13 15:12:59 1999 Delivered-To: freebsd-hackers@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [209.157.86.2]) by hub.freebsd.org (Postfix) with ESMTP id 8512F14FA8; Tue, 13 Jul 1999 15:12:47 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.9.3/8.9.1) id PAA81234; Tue, 13 Jul 1999 15:12:14 -0700 (PDT) (envelope-from dillon) Date: Tue, 13 Jul 1999 15:12:14 -0700 (PDT) From: Matthew Dillon Message-Id: <199907132212.PAA81234@apollo.backplane.com> To: cgd@netbsd.org (Chris G. Demetriou) Cc: Jason Thorpe , "Brian F. Feldman" , Noriyuki Soda , bright@rush.net, dcs@newsguy.com, freebsd-hackers@FreeBSD.ORG, jon@oaktree.co.uk, tech-userlevel@netbsd.org Subject: Re: Replacement for grep(1) (part 2) References: <199907132110.OAA23817@lestat.nas.nasa.gov> <199907132114.OAA80781@apollo.backplane.com> <877lo4z0pe.fsf@redmail.redback.com> Sender: owner-freebsd-hackers@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG : :Matthew Dillon writes: : :> If you don't have the disk necessary for a standard overcommit model to :> work, you definitely do not have the disk necessary for a non-overcommit :> model to work. : :I'd _really_ like to know how you figure this. : :text data bss dec hex filename :45024 4096 392 49512 c168 /bin/cat :311264 12288 9900 333452 5168c /bin/sh :212960 4096 28492 245548 3bf2c inetd.static :458720 12288 73716 544724 84fd4 sendmail.static : :None of these are particularly huge. Dynamically linked binaries :inflate things somewhat, of course, but even there: : :442336 12288 35780 490404 77ba4 /usr/lib/libc.so.12.40 : :the data and bss per library just aren't that huge. : :Of course, in addition to the sizes of the data and bss sections, you :need to make sure you've got enough extra for dynamically allocated :data, and for stack pages, and for a few other tidbits of dynamically :allocated storage, but the end result is a system which, even without :overcommit, can fit into a reasonable amount of backing store. The text size of a program is irrelevant, because swap is never allocated for it. The data and BSS are only relevant when they are modified. The only thing swap is ever used for is the dynamic allocation of memory. There are three ways to do it: sbrk(), mmap(... MAP_ANON), or mmap(... MAP_PRIVATE). Dynamic allocation of memory can occur under a huge number of conditions. The actual physical allocation of dynamic memory - what is known as a copy-on-write - cannot be predicted from the potential allocation of memory. The most obvious example of this is a fork(). There is a lot of hidden 'potential' VM that you haven't considered. For example, if the resource limit for a process's stack is 8MB, then the process can potentially allocate 8MB of stack even though it may actually only allocate 32K of stack. When a process forks, the child process can potentially modify just about every single non-text page that was owned by the parent process, causing a copy-on-write to occur. The dynamic potential can run into the megabytes but most child processes only modify a small percentage of those pages. :So, I just went back and looked at my mail folder for this NetBSD :mailing list (tech-userlevel), which has about a week and a half's :worth of messages. : :Nowhere did I see what amounts to anything other than hand-waving :claims that you'll have to allocate much, much more backing store than :you currently need to, and claims that that's unacceptable for general :purpose computing environments. If you have a more specific analysis :that you'd like me/us to read, please point us at it more specifically. You are welcome to bring up real-life situations as examples. That is what I do. But look who is doing the hand-waving now? :* not all the world's a general purpose computing environment, Which is meaningless handwaving. Again, you are welcome to point out your own real-life situations. :* while you certainly need to allocate more backing store than you :would with overcommit, it's _not_ ridiculously more for most :applications(+), and, finally, Based on what? I am basing my information on the VM reservation made by programs, and I am demonstrating specific points showing how those reservations occur. For example, the default 8MB resource limit for the stack segment. I am also demonstrating the level of control that is possible with resource limits. If you are doing an embedded system you can easily guarentee memory utilization even in an overcommit model by properly setting the memory resource limits for each process. If the system did not have any swap, I would do this for every single program I run on that system. If you are going to argue the point, work out the numbers and present them. I had to deal with a reservation model on our old SGI's running 5.3 for almost a year. I know what I'm talking about and I can point to real-life cases that demonstrate it. Certainly there are many different situations... you are welcome to bring up other real-life situations as examples. :* even if you are not willing to pay that price, there _are_ people :who are quite willing to pay that price to get the benefits that they :see (whether it's a matter of perception or not, from their :perspective they may as well be real) of such a scheme. Quite true. In the embedded world we preallocate memory and shape the programs to what is available in the system. But if we run out of memory we usually panic and reboot - because the code is designed to NOT run out of memory and thus running out of memory is a catastrophic situation. It is not appropriate to test every single allocation for a failure... that results in bulky code and provides no real benefit since the code is already designed to not overcommit the embedded system's memory. If it does, you reboot. The most critical embedded systems, such as those used in satellites, avoid dynamic allocation as much as possible in order to guarentee that no memory failures occur. This works under UNIX as well. If you run your programs and touch all your potentially modifiable pages you have effectively reserved the memory. :page from the system, it's almost certainly going write something to :it, and, while there are undoubtedly a few pages that aren't written :to, they are by far the majority. And, of course, once the page has :been written, it's no longer reserved, it's committed. 8-) Swap is used solely for dynamically modified pages of data that is not otherwise backed (e.g. not backed by a file, for example). :I would honestly love to know: what do you see huge numbers of :reserved pages being reserved for, if they're not actually being :committed, by 'average' UNIX applications (for any definition of :average that excludes applications which do memory based computation :on sparse dasta). : :cgd :-- :Chris Demetriou - cgd@netbsd.org - http://www.netbsd.org/People/Pages/cgd.html Stack, hidden VM objects after a multi-way fork, MAP_PRIVATE memory maps, private memory pools (e.g. for web servers, java VM's, and so forth). -Matt Matthew Dillon To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message