From owner-freebsd-stable Fri Sep 19 01:22:57 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.7/8.8.7) id BAA24407 for stable-outgoing; Fri, 19 Sep 1997 01:22:57 -0700 (PDT) Received: from mailgate.greenhills.co.uk (mailgate.greenhills.co.uk [195.11.194.180]) by hub.freebsd.org (8.8.7/8.8.7) with SMTP id BAA24395 for ; Fri, 19 Sep 1997 01:22:51 -0700 (PDT) Received: (qmail 10792 invoked by uid 982); 19 Sep 1997 08:20:36 -0000 Message-ID: <19970919092036.25119@webcrawler.com> Date: Fri, 19 Sep 1997 09:20:36 +0100 From: Martijn Koster To: Martijn Koster Cc: freebsd-stable@FreeBSD.ORG Subject: Re: [mak@webcrawler.com: malloc.c problems in 2.2 stable?] References: <19970912064740.36419@webcrawler.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary=NzB8fVQJ5HfG6fxh X-Mailer: Mutt 0.76 In-Reply-To: <19970912064740.36419@webcrawler.com>; from Martijn Koster on Fri, Sep 12, 1997 at 06:47:40AM +0100 Sender: owner-freebsd-stable@FreeBSD.ORG X-Loop: FreeBSD.org Precedence: bulk --NzB8fVQJ5HfG6fxh Content-Type: text/plain; charset=us-ascii On Fri, Sep 12, 1997 at 06:47:40AM +0100, Martijn Koster wrote: > I've justed finished tracking down swap filling on one of my > 2.2-stable production machines to procmail (delivering an only 8M > message), then to realloc. > A search of the mailing lists turned up John Fieber > 's suggestions in June of moving to 3.0-current > malloc.c, and that indeed fixed things. Cool! Nope, it improves things, but doesn't fix the problem completely. Procmail reallocs an area of memory in steps of 16K when reading a message, and the FreeBSD malloc doesn't cope with that -- way before you'd expect to you run out of swap. For example a 64M machine with 128M swap fills up before it gets to reallocing 21M). phk@freebsd.org explains that this should not be considered a malloc bug, but an ineffective allocation strategy on the part of procmail which should be fixed (suggestions and details appended). I agree with the latter, but not necessarily with the former. I had a look at the procmail, the code is in question is in src/pipes.c:readdyn(), and isn't pretty. I ran out of time, and am giving up. I'll report to the procmail mailing list, maybe they can do something, but I won't be holding my breath. -- Martijn Koster, m.koster@pobox.com --NzB8fVQJ5HfG6fxh Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=phk2 >From phk@critter.freebsd.dk Thu Sep 18 14:32:51 1997 Return-Path: Delivered-To: m.koster@greenhills.co.uk Received: (qmail 7055 invoked from network); 18 Sep 1997 14:32:50 -0000 Received: from punt-2b.mail.demon.net (HELO relay-6.mail.demon.net) (194.217.242.6) by mindy.greenhills.co.uk with SMTP; 18 Sep 1997 14:32:50 -0000 Received: from punt-2.mail.demon.net by mailstore for m.koster@greenhills.co.uk id 874591153:12:05626:2; Thu, 18 Sep 97 14:59:13 BST Received: from mail.webcrawler.com ([204.62.245.201]) by punt-2.mail.demon.net id aa0526954; 18 Sep 97 14:58 BST Received: (qmail 11312 invoked by uid 982); 18 Sep 1997 13:54:49 -0000 Delivered-To: mak@webcrawler.com Received: (qmail 11309 invoked from network); 18 Sep 1997 13:54:47 -0000 Received: from critter.freebsd.dk (195.8.129.26) by mail.webcrawler.com with SMTP; 18 Sep 1997 13:54:47 -0000 Received: from critter.freebsd.dk (localhost.cybercity.dk [127.0.0.1]) by critter.freebsd.dk (8.8.7/8.8.7) with ESMTP id PAA03511; Thu, 18 Sep 1997 15:57:51 +0200 (CEST) To: Martijn Koster cc: Andre Albsmeier Subject: Re: procmail dies, why? In-reply-to: Your message of "Thu, 18 Sep 1997 14:25:43 BST." <19970918142543.12647@webcrawler.com> Date: Thu, 18 Sep 1997 15:57:51 +0200 Message-ID: <3509.874591071@critter.freebsd.dk> From: Poul-Henning Kamp Status: RO X-Status: A Content-Length: 5001 Lines: 180 There isn't much that can be done in this case, it's a singular stupid way of allocating memory. I would need to do page flipping AND be very lucky to be able to make this work decently. A far better way to do the job is to: mul = 16 size = 64K; malloc (size) while (not enough) /* don't mess around */ size *= mul realloc(size) if (mul > 2) mul <<= 1; Don't worry about overallocation, it's practically free until you actually access the memory. even doing malloc (32M) right away would be cheaper than the current stuff. But even smarter would be to pipe the stuff into a temporary file and them mmap it... Procmail is stupid, and should be fixed. Poul-Henning In message <19970918142543.12647@webcrawler.com>, Martijn Koster writes: > >[phk: we briefly discussed this last week. Can I once again ask for help?] > > >On Thu, Sep 18, 1997 at 01:27:24PM +0200, Andre Albsmeier wrote: > >> I have here 8 2.2-STABLE machines and all behave equal. realloc or >> procmail screws up the machine although with the new malloc.c >> the system doesn't crash any more. > >You are right; malloc 1.32 improves the situation, but something is >still weird. > >Poul -- can you reproduce this? Any ideas what's up? > >When I run rea.c with 100M argument on -stable (with malloc.c-1.32) on >a 64M machine (132.96-MHz 586-class CPU), I get to 21M, when the >machine has run out of swap (138M). One -current on a 64M machine >(199.43-MHz 686-class CPU) I get to 25M, by then the machine has run >out of swap (138M). This process takes some 12 minutes (!). > >This code runs fine up to 100M on Irix 6.2 (Indy, 96M, 2 secs) and >Solaris 2.5 (Sparc5, 64M, 10 secs). This doesn't make FreeBSD look >good :-( > >Below is all the information I think may be relevant for reproduction. > >-- Martijn > >mak@pitch$ cat rea.c >#include >#include >#include > >int >main(int argc, char* argv[]) >{ > size_t size = 0; > char *p = NULL; > int max = 8404992; > > if (argc > 2) { > fprintf(stderr, "Usage: %s maxmem\n", argv[0]); > exit(-1); > } > if (argc > 1) > max = atoi(argv[1]); > > while(1) { > > size += 16384; > if (size > max) > break; > > fprintf(stderr, "realloc(%u)\n", size); > if ((p = realloc(p, size)) == NULL) { > fprintf(stderr, "out of memory\n"); > exit(-1); > } > } > > { > int c; > printf("done. press return to quit\n"); > read(0, &c, 1); > } >} >mak@pitch$ uname -a >FreeBSD pitch.webcrawler.com 2.2-STABLE FreeBSD 2.2-STABLE #0: Tue Sep 9 16:0 >4:50 PDT 1997 mak@pitch.webcrawler.com:/usr/src/sys/compile/PITCH i386 >mak@pitch$ dmesg | grep 'real memory' >real memory = 67108864 (65536K bytes) >mak@pitch$ grep '\$Id' /usr/src/lib/libc/stdlib/malloc.c > * $Id: malloc.c,v 1.32 1997/08/31 05:59:39 phk Exp $ >mak@pitch$ ls -l /etc/malloc.conf >ls: /etc/malloc.conf: No such file or directory >mak@pitch$ cc -c /usr/src/lib/libc/stdlib/malloc.c >mak@pitch$ cc -c rea.c >mak@pitch$ cc -o rea rea.o malloc.o >mak@pitch$ sudo su >Password: ># bash >bash-2.00# for i in -c -d -f -l -m -n -p -s -t -u -v >> do >> echo $i >> ulimit $i unlimited >> done >-c >-d >-f >-l >-m >-p >bash: ulimit: cannot modify limit: Invalid argument >-s >-t >-u >-v >bash: ulimit: cannot modify limit: Invalid argument >bash-2.00# ulimit -a >core file size (blocks) unlimited >data seg size (kbytes) 131072 >file size (blocks) unlimited >max locked memory (kbytes) unlimited >max memory size (kbytes) unlimited >open files 360 >pipe size (512 bytes) 1 >stack size (kbytes) 65536 >cpu time (seconds) unlimited >max user processes 179 >virtual memory (kbytes) 196608 >bash-2.00# ./rea 104857600 ># ... many lines of output >realloc(21823488) >out of memory >bash-2.00# > ># At the time, top showed: > > PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND >13061 root -18 0 42756K 34196K swread 13:10 21.51% 21.51% rea >11882 mak 10 0 776K 436K wait 0:23 0.84% 0.84% bash >last pid: 16331; load averages: 1.50, 1.70, 1.36 22:32:54 >32 processes: 1 running, 31 sleeping > >Mem: 26M Active, 9660K Inact, 19M Wired, 5876K Cache, 7616K Buf, 532K Free >Swap: 138M Total, 137M Used, 1684K Free, 99% Inuse > > > PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND >13061 root -18 0 42756K 35776K swread 13:10 20.56% 20.56% rea >11882 mak 10 0 776K 436K wait 0:23 0.88% 0.88% bash >head: Cannot allocate memory >kvm_open: Cannot allocate memory >kvm_open: Cannot allocate memory >head: Cannot allocate memory >head: Cannot allocate memory >kvm_open: Cannot allocate memory >kvm_open: Cannot allocate memory >head: Cannot allocate memory > -- Poul-Henning Kamp FreeBSD coreteam member phk@FreeBSD.ORG "Real hackers run -current on their laptop." --NzB8fVQJ5HfG6fxh--