Date: Fri, 19 Sep 1997 09:20:36 +0100 From: Martijn Koster <mak@webcrawler.com> To: Martijn Koster <mak@webcrawler.com> Cc: freebsd-stable@FreeBSD.ORG Subject: Re: [mak@webcrawler.com: malloc.c problems in 2.2 stable?] Message-ID: <19970919092036.25119@webcrawler.com> In-Reply-To: <19970912064740.36419@webcrawler.com>; from Martijn Koster on Fri, Sep 12, 1997 at 06:47:40AM %2B0100 References: <19970912064740.36419@webcrawler.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--NzB8fVQJ5HfG6fxh
Content-Type: text/plain; charset=us-ascii
On Fri, Sep 12, 1997 at 06:47:40AM +0100, Martijn Koster wrote:
> I've justed finished tracking down swap filling on one of my
> 2.2-stable production machines to procmail (delivering an only 8M
> message), then to realloc.
> A search of the mailing lists turned up John Fieber
> <jfieber@indiana.edu>'s suggestions in June of moving to 3.0-current
> malloc.c, and that indeed fixed things. Cool!
Nope, it improves things, but doesn't fix the problem completely.
Procmail reallocs an area of memory in steps of 16K when reading a
message, and the FreeBSD malloc doesn't cope with that -- way before
you'd expect to you run out of swap. For example a 64M machine with
128M swap fills up before it gets to reallocing 21M).
phk@freebsd.org explains that this should not be considered a malloc
bug, but an ineffective allocation strategy on the part of procmail
which should be fixed (suggestions and details appended). I agree with
the latter, but not necessarily with the former.
I had a look at the procmail, the code is in question is in
src/pipes.c:readdyn(), and isn't pretty. I ran out of time, and am
giving up. I'll report to the procmail mailing list, maybe they can do
something, but I won't be holding my breath.
-- Martijn Koster, m.koster@pobox.com
--NzB8fVQJ5HfG6fxh
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename=phk2
>From phk@critter.freebsd.dk Thu Sep 18 14:32:51 1997
Return-Path: <phk@critter.freebsd.dk>
Delivered-To: m.koster@greenhills.co.uk
Received: (qmail 7055 invoked from network); 18 Sep 1997 14:32:50 -0000
Received: from punt-2b.mail.demon.net (HELO relay-6.mail.demon.net) (194.217.242.6)
by mindy.greenhills.co.uk with SMTP; 18 Sep 1997 14:32:50 -0000
Received: from punt-2.mail.demon.net by mailstore for m.koster@greenhills.co.uk
id 874591153:12:05626:2; Thu, 18 Sep 97 14:59:13 BST
Received: from mail.webcrawler.com ([204.62.245.201]) by punt-2.mail.demon.net
id aa0526954; 18 Sep 97 14:58 BST
Received: (qmail 11312 invoked by uid 982); 18 Sep 1997 13:54:49 -0000
Delivered-To: mak@webcrawler.com
Received: (qmail 11309 invoked from network); 18 Sep 1997 13:54:47 -0000
Received: from critter.freebsd.dk (195.8.129.26)
by mail.webcrawler.com with SMTP; 18 Sep 1997 13:54:47 -0000
Received: from critter.freebsd.dk (localhost.cybercity.dk [127.0.0.1])
by critter.freebsd.dk (8.8.7/8.8.7) with ESMTP id PAA03511;
Thu, 18 Sep 1997 15:57:51 +0200 (CEST)
To: Martijn Koster <mak@webcrawler.com>
cc: Andre Albsmeier <Andre.Albsmeier@mchp.siemens.de>
Subject: Re: procmail dies, why?
In-reply-to: Your message of "Thu, 18 Sep 1997 14:25:43 BST."
<19970918142543.12647@webcrawler.com>
Date: Thu, 18 Sep 1997 15:57:51 +0200
Message-ID: <3509.874591071@critter.freebsd.dk>
From: Poul-Henning Kamp <phk@critter.freebsd.dk>
Status: RO
X-Status: A
Content-Length: 5001
Lines: 180
There isn't much that can be done in this case, it's a singular stupid
way of allocating memory.
I would need to do page flipping AND be very lucky to be able to make
this work decently.
A far better way to do the job is to:
mul = 16
size = 64K;
malloc (size)
while (not enough)
/* don't mess around */
size *= mul
realloc(size)
if (mul > 2)
mul <<= 1;
Don't worry about overallocation, it's practically free until you
actually access the memory. even doing
malloc (32M)
right away would be cheaper than the current stuff.
But even smarter would be to pipe the stuff into a temporary file and
them mmap it...
Procmail is stupid, and should be fixed.
Poul-Henning
In message <19970918142543.12647@webcrawler.com>, Martijn Koster writes:
>
>[phk: we briefly discussed this last week. Can I once again ask for help?]
>
>
>On Thu, Sep 18, 1997 at 01:27:24PM +0200, Andre Albsmeier wrote:
>
>> I have here 8 2.2-STABLE machines and all behave equal. realloc or
>> procmail screws up the machine although with the new malloc.c
>> the system doesn't crash any more.
>
>You are right; malloc 1.32 improves the situation, but something is
>still weird.
>
>Poul -- can you reproduce this? Any ideas what's up?
>
>When I run rea.c with 100M argument on -stable (with malloc.c-1.32) on
>a 64M machine (132.96-MHz 586-class CPU), I get to 21M, when the
>machine has run out of swap (138M). One -current on a 64M machine
>(199.43-MHz 686-class CPU) I get to 25M, by then the machine has run
>out of swap (138M). This process takes some 12 minutes (!).
>
>This code runs fine up to 100M on Irix 6.2 (Indy, 96M, 2 secs) and
>Solaris 2.5 (Sparc5, 64M, 10 secs). This doesn't make FreeBSD look
>good :-(
>
>Below is all the information I think may be relevant for reproduction.
>
>-- Martijn
>
>mak@pitch$ cat rea.c
>#include <stdio.h>
>#include <stdlib.h>
>#include <unistd.h>
>
>int
>main(int argc, char* argv[])
>{
> size_t size = 0;
> char *p = NULL;
> int max = 8404992;
>
> if (argc > 2) {
> fprintf(stderr, "Usage: %s maxmem\n", argv[0]);
> exit(-1);
> }
> if (argc > 1)
> max = atoi(argv[1]);
>
> while(1) {
>
> size += 16384;
> if (size > max)
> break;
>
> fprintf(stderr, "realloc(%u)\n", size);
> if ((p = realloc(p, size)) == NULL) {
> fprintf(stderr, "out of memory\n");
> exit(-1);
> }
> }
>
> {
> int c;
> printf("done. press return to quit\n");
> read(0, &c, 1);
> }
>}
>mak@pitch$ uname -a
>FreeBSD pitch.webcrawler.com 2.2-STABLE FreeBSD 2.2-STABLE #0: Tue Sep 9 16:0
>4:50 PDT 1997 mak@pitch.webcrawler.com:/usr/src/sys/compile/PITCH i386
>mak@pitch$ dmesg | grep 'real memory'
>real memory = 67108864 (65536K bytes)
>mak@pitch$ grep '\$Id' /usr/src/lib/libc/stdlib/malloc.c
> * $Id: malloc.c,v 1.32 1997/08/31 05:59:39 phk Exp $
>mak@pitch$ ls -l /etc/malloc.conf
>ls: /etc/malloc.conf: No such file or directory
>mak@pitch$ cc -c /usr/src/lib/libc/stdlib/malloc.c
>mak@pitch$ cc -c rea.c
>mak@pitch$ cc -o rea rea.o malloc.o
>mak@pitch$ sudo su
>Password:
># bash
>bash-2.00# for i in -c -d -f -l -m -n -p -s -t -u -v
>> do
>> echo $i
>> ulimit $i unlimited
>> done
>-c
>-d
>-f
>-l
>-m
>-p
>bash: ulimit: cannot modify limit: Invalid argument
>-s
>-t
>-u
>-v
>bash: ulimit: cannot modify limit: Invalid argument
>bash-2.00# ulimit -a
>core file size (blocks) unlimited
>data seg size (kbytes) 131072
>file size (blocks) unlimited
>max locked memory (kbytes) unlimited
>max memory size (kbytes) unlimited
>open files 360
>pipe size (512 bytes) 1
>stack size (kbytes) 65536
>cpu time (seconds) unlimited
>max user processes 179
>virtual memory (kbytes) 196608
>bash-2.00# ./rea 104857600
># ... many lines of output
>realloc(21823488)
>out of memory
>bash-2.00#
>
># At the time, top showed:
>
> PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
>13061 root -18 0 42756K 34196K swread 13:10 21.51% 21.51% rea
>11882 mak 10 0 776K 436K wait 0:23 0.84% 0.84% bash
>last pid: 16331; load averages: 1.50, 1.70, 1.36 22:32:54
>32 processes: 1 running, 31 sleeping
>
>Mem: 26M Active, 9660K Inact, 19M Wired, 5876K Cache, 7616K Buf, 532K Free
>Swap: 138M Total, 137M Used, 1684K Free, 99% Inuse
>
>
> PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
>13061 root -18 0 42756K 35776K swread 13:10 20.56% 20.56% rea
>11882 mak 10 0 776K 436K wait 0:23 0.88% 0.88% bash
>head: Cannot allocate memory
>kvm_open: Cannot allocate memory
>kvm_open: Cannot allocate memory
>head: Cannot allocate memory
>head: Cannot allocate memory
>kvm_open: Cannot allocate memory
>kvm_open: Cannot allocate memory
>head: Cannot allocate memory
>
--
Poul-Henning Kamp FreeBSD coreteam member
phk@FreeBSD.ORG "Real hackers run -current on their laptop."
--NzB8fVQJ5HfG6fxh--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?19970919092036.25119>