Date: Sat, 31 Jan 2004 14:04:05 -0800 (PST) From: dpk <dpk@dpk.net> To: Jorn Argelo <jorn@wcborstel.nl> Cc: questions@freebsd.org Subject: Re: Memory leak on 5.1-RELEASE? Message-ID: <Pine.BSF.4.58.0401311354210.98773@shared10.hosting.flyingcroc.net> In-Reply-To: <012701c3e8ea$d6436c40$050110ac@computerjorn> References: <Pine.BSF.4.58.0401301550310.98773@shared10.hosting.flyingcroc.net> <012701c3e8ea$d6436c40$050110ac@computerjorn>
next in thread | previous in thread | raw e-mail | index | archive | help
Here's the top output from this morning:
Mem: 1300M Active, 292M Inact, 295M Wired, 104M Cache, 112M Buf, 5116K Free
Swap: 1024M Total, 8036K Used, 1016M Free
At this time there are ~130 processes, whereas in the previous one
there were ~76, with more memory marked as "Active". vmstat says this:
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr da0 ac0 in sy cs us sy id
0 0 1 848048 100880 937 113 23 0 1503 248 0 0 2678 0 8211 2 4 95
0 0 1 830192 110484 340 4 1 0 2719 0 1 0 2574 0 5917 1 5 93
avm is supposed to be the number of active virtual pages. The page size
being 4096, according to hw.pagesize, would actually come up to around
3.4GB. I'm not sure what this number refers to - maybe it's memory + swap?
The whole reason I'm bringing this up with the list is that, ultimately,
this is causing the machine to crash, after running out of swap. If I
could identify where this Active memory has gone, I may be able to recover
the "lost" memory and prevent the server from swapping itself to death.
Here's the ps -ax output - note that nfsd is running, but it's not being
used. It does have an NFS client mounted, but from "vmstat -m", it isn't
using any memory at all (and it would be in Buf I think?)
PID TT STAT TIME COMMAND
0 ?? DLs 0:00.82 (swapper)
0 ?? ZW 0:00.00 (httpd)
1 ?? ILs 0:00.08 /sbin/init --
2 ?? DL 0:13.59 (g_event)
3 ?? DL 3:41.95 (g_up)
4 ?? DL 4:56.15 (g_down)
5 ?? DL 1:41.88 (pagedaemon)
6 ?? DL 0:00.00 (vmdaemon)
7 ?? DL 16:25.54 (pagezero)
8 ?? DL 2:07.11 (bufdaemon)
9 ?? DL 0:01.51 (vnlru)
10 ?? DL 0:00.00 (ktrace)
11 ?? RL 0:00.00 (idle: cpu3)
12 ?? RL 5037:43.41 (idle: cpu2)
13 ?? RL 0:00.00 (idle: cpu1)
14 ?? RL 4281:37.99 (idle: cpu0)
15 ?? WL 127:48.20 (swi1: net)
16 ?? WL 6:02.49 (swi7: tty:sio clock)
18 ?? DL 1:26.21 (random)
20 ?? WL 1:39.12 (swi3: cambio)
23 ?? WL 0:00.00 (irq14: ata0)
26 ?? DL 0:00.02 (usb0)
27 ?? DL 0:00.00 (usbtask)
30 ?? WL 2:13.96 (irq16: ahc0)
31 ?? WL 0:00.00 (irq17: ahc1)
32 ?? WL 58:17.09 (irq18: em0)
33 ?? WL 0:00.00 (irq1: atkbd0)
34 ?? WL 0:00.00 (irq6: fdc0)
35 ?? WL 0:00.00 (swi0: tty:sio)
40 ?? DL 13:22.17 (syncer)
41 ?? IL 3:06.26 (nfsiod 0)
42 ?? IL 2:47.50 (nfsiod 1)
43 ?? IL 2:28.19 (nfsiod 2)
44 ?? IL 3:37.52 (nfsiod 3)
148 ?? Is 0:00.00 adjkerntz -i
257 ?? Ss 0:02.08 /usr/sbin/syslogd -s
352 ?? Ss 0:00.25 /usr/sbin/rpcbind
375 ?? Is 0:00.01 /usr/sbin/mountd -r
377 ?? Is 0:00.02 nfsd: master (nfsd)
380 ?? I 0:00.00 nfsd: server (nfsd)
381 ?? I 0:00.00 nfsd: server (nfsd)
382 ?? I 0:00.00 nfsd: server (nfsd)
383 ?? I 0:00.00 nfsd: server (nfsd)
394 ?? Ss 0:00.35 /usr/sbin/usbd
443 ?? Is 0:02.00 /usr/sbin/sshd -u0
449 ?? Ss 0:06.17 sendmail: accepting connections (sendmail)
452 ?? Is 0:00.13 sendmail: Queue runner@00:30:00 for /var/spool/client
466 ?? Is 0:01.25 /usr/sbin/cron
829 ?? Ss 0:02.75 proftpd: (accepting connections) (proftpd)
855 ?? Is 0:00.00 /usr/sbin/inetd -wW
21727 ?? Ss 2:11.11 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
31784 ?? S 0:00.23 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
52298 ?? S 0:00.13 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
52989 ?? S 0:00.18 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
53224 ?? S 0:00.11 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
54180 ?? S 0:00.18 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
54282 ?? S 0:00.13 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
54310 ?? S 0:00.13 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
55563 ?? S 0:00.11 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
55970 ?? S 0:00.16 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56261 ?? S 0:00.13 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56369 ?? S 0:00.10 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56422 ?? S 0:00.04 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56424 ?? S 0:00.18 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56465 ?? S 0:00.11 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56491 ?? S 0:00.17 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56523 ?? S 0:00.12 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56575 ?? Rs 0:00.07 sshd: toor@ttyp0 (sshd)
56606 ?? S 0:00.33 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56617 ?? S 0:00.15 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56729 ?? I 0:00.13 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56763 ?? S 0:00.11 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56768 ?? S 0:00.09 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56779 ?? S 0:00.09 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56786 ?? S 0:00.05 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56788 ?? S 0:00.00 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56789 ?? S 0:00.12 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56790 ?? S 0:00.08 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56800 ?? S 0:00.09 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56809 ?? I 0:00.01 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56811 ?? S 0:00.13 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56840 ?? S 0:00.15 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56843 ?? S 0:00.04 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56845 ?? S 0:00.03 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56846 ?? S 0:00.02 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56852 ?? S 0:00.02 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56853 ?? S 0:00.05 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56864 ?? S 0:00.11 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56865 ?? S 0:00.17 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56866 ?? S 0:00.06 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56869 ?? S 0:00.06 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56870 ?? S 0:00.06 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56871 ?? S 0:00.05 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56872 ?? S 0:00.09 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56873 ?? S 0:00.03 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56874 ?? S 0:00.04 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56875 ?? S 0:00.07 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56876 ?? S 0:00.03 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56877 ?? S 0:00.03 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56878 ?? S 0:00.04 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56879 ?? S 0:00.07 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56880 ?? S 0:00.07 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56881 ?? S 0:00.06 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56882 ?? S 0:00.04 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56883 ?? S 0:00.05 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56884 ?? S 0:00.03 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56885 ?? S 0:00.03 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56886 ?? S 0:00.04 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56887 ?? S 0:00.12 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56888 ?? S 0:00.02 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56889 ?? S 0:00.03 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56890 ?? S 0:00.03 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56891 ?? S 0:00.03 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56892 ?? S 0:00.02 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56893 ?? S 0:00.04 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56894 ?? S 0:00.03 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56896 ?? S 0:00.00 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56897 ?? S 0:00.02 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56898 ?? S 0:00.03 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56900 ?? S 0:00.03 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
56577 p0 Ss 0:00.02 -sh (sh)
56904 p0 R+ 0:00.00 ps -ax
3574 v0 Is+ 0:00.00 /usr/libexec/getty Pc ttyv0
868 v1 Is+ 0:00.00 /usr/libexec/getty Pc ttyv1
869 v2 Is+ 0:00.00 /usr/libexec/getty Pc ttyv2
870 v3 Is+ 0:00.00 /usr/libexec/getty Pc ttyv3
871 v4 Is+ 0:00.00 /usr/libexec/getty Pc ttyv4
872 v5 Is+ 0:00.00 /usr/libexec/getty Pc ttyv5
873 v6 Is+ 0:00.00 /usr/libexec/getty Pc ttyv6
874 v7 Is+ 0:00.00 /usr/libexec/getty Pc ttyv7
520 con- I 0:02.81 /usr/local/sbin/snmpd
534 con- I 0:00.01 /bin/sh /usr/local/bin/mysqld_safe --datadir=/usr/loc
566 con- S 138:26.09 /usr/local/libexec/mysqld --basedir=/usr/local --data
According to top and ps, mysql is only using around 59M VSZ.
On Sun, 1 Feb 2004, Jorn Argelo wrote:
> Are you sure that there isn't anything else running? Why don't you give us
> an ps -ax output? I don't think there's a memory leak in 5.1, since I've
> seen running 5.1 just fine on a PE2650 with 2 GB RAM. You shouldn't
> rely on top too much acually. Vmstat is a better program when looking
> at memory.
>
> Cheers,
>
> Jorn
> ----- Original Message -----
> From: "dpk" <dpk@dpk.net>
> To: <questions@freebsd.org>
> Sent: Saturday, January 31, 2004 1:10 AM
> Subject: Memory leak on 5.1-RELEASE?
>
>
> > (I'm not a member of the list; please Cc me on any replies.)
> >
> > We're running Apache 1.3.28 on a 5.1-RELEASE machine. It's a Dell PE 2650
> > w/ 2GB RAM. The site contains a lot of large files (multi-megabyte) -
> > otherwise there's nothing unusual running.
> >
> > The Active memory use, according to top, seems rather high:
> >
> > last pid: 21487; load averages: 0.19, 0.33, 0.32 up 2+16:45:20
> 15:52:21
> > 76 processes: 1 running, 75 sleeping
> > CPU states: 0.5% user, 0.0% nice, 4.0% system, 1.4% interrupt, 94.2%
> idle
> > Mem: 1413M Active, 187M Inact, 299M Wired, 93M Cache, 112M Buf, 2632K Free
> > Swap: 1024M Total, 21M Used, 1003M Free, 2% Inuse
> >
> > We can't seem to get the Active number down much, even after stopping
> > Apache it still stays around 1100M. There's no shared memory in use, and
> > nothing in vmstat -m seems to indicate where the "missing" memory is. top,
> > sorting by "size", does not indicate anything unusual either.
> >
> > sysctl vm.vmtotal says:
> >
> > vm.vmtotal:
> > System wide totals computed every five seconds: (values in kilobytes)
> > ===============================================
> > Processes: (RUNQ: 1 Disk Wait: 0 Page Wait: 0 Sleep: 76)
> > Virtual Memory: (Total: 8172K, Active 636472K)
> > Real Memory: (Total: 2051312K Active 389176K)
> > Shared Virtual Memory: (Total: 16436K Active: 11760K)
> > Shared Real Memory: (Total: 6004K Active: 4436K)
> > Free Memory Pages: 79228K
> >
> > whereas on other servers, the Real Memory "Active" number seems to match
> > the one found in top, on this one it is about 1GB lower.
> >
> > A similar machine running Apache on 5.1-R, generally serving smaller
> > files, has the same problem in a smaller scale (about 640M even when
> > Apache is stopped).
> >
> > Are there any other data that I should send to help diagnose this problem,
> > or any programs I can run to try and track this stray memory use down?
> >
> > - dpk
> > _______________________________________________
> > freebsd-questions@freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> > To unsubscribe, send any mail to
> "freebsd-questions-unsubscribe@freebsd.org"
> >
>
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.BSF.4.58.0401311354210.98773>