Date: Thu, 23 Feb 2006 01:36:06 +0100 From: Viktor Vasilev <viktor.vasilev@stud.tu-darmstadt.de> To: hackers@freebsd.org Subject: Looking for advice on chasing a kernel memory leak Message-ID: <200602230136.06689.viktor.vasilev@stud.tu-darmstadt.de>
next in thread | raw e-mail | index | archive | help
--Boundary-00=_2NQ/DNCe++IlWBH Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Content-Disposition: inline Hello, I'm having trouble with a WARP (http://www.pcengines.ch/wrap.htm) board running m0n0wall v1.21 (stripped down FreeBSD 4.11-RELEASE-p13). It's serving as an internet gateway and the problem is, that after some time it starts blocking traffic and doesn't do NAT anymore. The box is handling very low traffic volume and is mostly idle. I've enabled logging to a remote machine and around the time the trouble happens, there are messages like these: Jan 15 04:02:25 gw /kernel: ipf_nattable_max reduced to -96 Jan 15 04:02:49 gw /kernel: ipf_nattable_max reduced to -94 Jan 15 04:02:49 gw /kernel: ipf_nattable_max reduced to -94 Jan 15 04:04:31 gw /kernel: ipf_nattable_max reduced to -92 An inspection of the ipfilter code shows that kmem_alloc is failing: http://fxr.watson.org/fxr/source/contrib/ipfilter/netinet/ip_nat.c?v=RELENG4#L1197 A reboot fixes the things until the same thing happens again in three or so weeks. Since that happened a couple of times, I've monitored RAM usage and see a constant growth of the wired memory. After reboot top reports: Mem: 4312K Active, 3684K Inact, 5960K Wired, 4848K Buf, 99M Free Now after two weeks: Mem: 5044K Active, 3824K Inact, 22M Wired, 5856K Buf, 82M Free vmstat -m output (attached) clearly shows that there's a huge amount of M_TEMP memory held and growing: temp 30370 15148K 15169K 19166K 16804822 0 0 16,32,64,128,256,512,1K,4K,8K,32K,256K,512K I don't know how to find out who actually allocates this memory. Any ideas? ps auxwww output, kernel config and dmesg are attached. Cheers, Viktor --Boundary-00=_2NQ/DNCe++IlWBH Content-Type: text/plain; charset="us-ascii"; name="wrap-dmesg.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="wrap-dmesg.txt" Copyright (c) 1992-2005 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 4.11-RELEASE-p13 #0: Sat Nov 26 12:17:56 CET 2005 root@fb411.neon1.net:/usr/src/sys/compile/M0N0WALL_WRAP Timecounter "i8254" frequency 1193182 Hz CPU: NSC Geode (266.64-MHz 586-class CPU) Origin = "Geode by NSC" Id = 0x540 Stepping = 0 DIR=0x81b7 Features=0x808131<FPU,TSC,MSR,CX8,CMOV,MMX> real memory = 134217728 (131072K bytes) avail memory = 116117504 (113396K bytes) Preloaded elf kernel "kernel" at 0xc0e03000. Preloaded mfs_root "/mfsroot" at 0xc0e030a8. md0: Preloaded image </mfsroot> 11534336 bytes at 0xc0301df0 md1: Malloc disk npx0: <math processor> on motherboard npx0: INT 16 interface pcib0: <Host to PCI bridge> on motherboard pci0: <PCI bus> on pcib0 sis0: <NatSemi DP83815 10/100BaseTX> port 0x1000-0x10ff mem 0x80000000-0x80000fff irq 10 at device 14.0 on pci0 sis0: Ethernet address: 00:0d:b9:02:c4:d0 miibus0: <MII bus> on sis0 ukphy0: <Generic IEEE 802.3u media interface> on miibus0 ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto sis1: <NatSemi DP83815 10/100BaseTX> port 0x1400-0x14ff mem 0x80040000-0x80040fff irq 9 at device 15.0 on pci0 sis1: Ethernet address: 00:0d:b9:02:c4:d1 miibus1: <MII bus> on sis1 ukphy1: <Generic IEEE 802.3u media interface> on miibus1 ukphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto sis2: <NatSemi DP83815 10/100BaseTX> port 0x1800-0x18ff mem 0x80080000-0x80080fff irq 11 at device 16.0 on pci0 sis2: Ethernet address: 00:0d:b9:02:c4:d2 miibus2: <MII bus> on sis2 ukphy2: <Generic IEEE 802.3u media interface> on miibus2 ukphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto isab0: <PCI to ISA bridge (vendor=100b device=0510)> port 0xf600-0xf63f,0xf400-0xf43f at device 18.0 on pci0 isa0: <ISA bus> on isab0 chip1: <PCI to Other bridge (vendor=100b device=0511)> port 0xf000-0xf0ff at device 18.1 on pci0 atapci0: <Generic PCI ATA controller> port 0xfc00-0xfc0f at device 18.2 on pci0 ata0: at 0x1f0 irq 14 on atapci0 pci0: <unknown card> (vendor=0x100b, dev=0x0503) at 18.3 chip2: <PCI to Other bridge (vendor=100b device=0515)> port 0xf200-0xf23f at device 18.5 on pci0 orm0: <Option ROM> at iomem 0xe0000-0xe7fff on isa0 pmtimer0 on isa0 sio0 at port 0x3f8-0x3ff irq 4 flags 0x30 on isa0 sio0: type 16550A, console BRIDGE 020214 loaded IPsec: Initialized Security Association Processing. IP Filter: v3.4.35 initialized. Default = block all, Logging = enabled ad0: 123MB <PRCF1002> [251/16/63] at ata0-master PIO4 Mounting root from ufs:/dev/md0c stray irq 7 --Boundary-00=_2NQ/DNCe++IlWBH Content-Type: text/plain; charset="us-ascii"; name="wrap-vmstat-m.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="wrap-vmstat-m.txt" Memory statistics by bucket size Size In Use Free Requests HighWater Couldfree 16 573 195 11103972 1280 0 32 563 77 2902431 640 0 64 1828 28 25873127 320 228322 128 1070 114 18465333 160 0 256 1382 2378 7479540 80 305488 512 29603 13 3252875 40 10200 1K 39 5 5045708 20 0 2K 25 3 88792 10 17358 4K 5 1 42295 5 0 8K 22 0 383 5 220 16K 4 0 12 5 0 32K 2 0 2 5 0 256K 1 0 1 5 0 512K 0 0 2 5 0 Memory usage type by bucket size Size Type(s) 16 nexusdev, p1003.1b, key mgmt, netgraph, routetbl, ether_multi, vnodes, mount, pcb, soname, atexit, proc-args, kld, ATA generic, rman, MD disk, bus, sysctl, temp, devbuf 32 in_multi, netgraph, routetbl, ether_multi, ifaddr, BPF, vnodes, cluster_save buffer, pcb, soname, proc-args, sigio, kld, taskqueue, eventhandler, bus, sysctl, uidinfo, subproc, pgrp, temp, devbuf 64 isadev, netgraph, routetbl, ether_multi, ifaddr, vnodes, cluster_save buffer, vfscache, lockf, pcb, proc-args, file, AD driver, rman, eventhandler, bus, subproc, session, AR driver, temp, devbuf 128 ZONE, key mgmt, netgraph, routetbl, ifaddr, vnodes, mount, soname, ttys, zombie, proc-args, dev_t, timecounter, kld, bus, uidinfo, cred, temp, devbuf 256 FFS node, newblk, netgraph, routetbl, ifaddr, vnodes, ttys, proc-args, kqueue, file desc, bus, subproc, temp, devbuf 512 crypto, UFS mount, netgraph, mount, BIO buffer, ptys, file desc, ATA generic, msg, bus, proc, AR driver, temp, devbuf 1K Export Host, netgraph, BIO buffer, kqueue, file desc, AD driver, sem, ioctlops, MD disk, bus, temp 2K UFS mount, netgraph, ifaddr, BIO buffer, pcb, AR driver 4K mbuf, pagedep, sem, msg, temp 8K VM pgdata, syncache, netgraph, bus, temp 16K UFS ihash, inodedep, shm, msg, bus 32K vfscache, temp 256K temp 512K temp Memory statistics by type Type Kern Type InUse MemUse HighUse Limit Requests Limit Limit Size(s) nexusdev 4 1K 1K 19166K 4 0 0 16 mbuf 1 4K 4K 19166K 1 0 0 4K crypto 1 1K 1K 19166K 1 0 0 512 isadev 6 1K 1K 19166K 6 0 0 64 ZONE 15 2K 2K 19166K 15 0 0 128 VM pgdata 1 8K 8K 19166K 1 0 0 8K UFS mount 6 9K 9K 19166K 61275 0 0 512,2K UFS ihash 1 16K 16K 19166K 1 0 0 16K FFS node 798 200K 200K 19166K 103331 0 0 256 newblk 1 1K 1K 19166K 1 0 0 256 inodedep 1 16K 16K 19166K 1 0 0 16K pagedep 1 4K 4K 19166K 1 0 0 4K p1003.1b 1 1K 1K 19166K 1 0 0 16 key mgmt 72 7K 18K 19166K 6899993 0 0 16,128 syncache 1 8K 8K 19166K 1 0 0 8K Export Host 1 1K 1K 19166K 1 0 0 1K in_multi 26 1K 1K 19166K 26 0 0 32 netgraph 912 215K 219K 19166K 804800 0 0 16,32,64,128,256,512,1K,2K,8K routetbl 52 7K 11K 19166K 1809 0 0 16,32,64,128,256 ether_multi 55 2K 2K 19166K 55 0 0 16,32,64 ifaddr 47 12K 12K 19166K 440 0 0 32,64,128,256,2K BPF 21 1K 1K 19166K 373 0 0 32 vnodes 18 5K 5K 19166K 224 0 0 16,32,64,128,256 mount 3 2K 2K 19166K 20428 0 0 16,128,512 cluster_save buffer 0 0K 1K 19166K 11 0 0 32,64 vfscache 1190 107K 107K 19166K 145578 0 0 64,32K BIO buffer 28 28K 30K 19166K 81825 0 0 512,1K,2K lockf 2 1K 1K 19166K 48 0 0 64 pcb 48 6K 6K 19166K 2229913 0 0 16,32,64,2K soname 1 1K 1K 19166K 5273087 0 0 16,32,128 ptys 1 1K 1K 19166K 1 0 0 512 ttys 90 12K 12K 19166K 168 0 0 128,256 atexit 1 1K 1K 19166K 1 0 0 16 zombie 1 1K 1K 19166K 2609899 0 0 128 proc-args 13 1K 2K 19166K 2525828 0 0 16,32,64,128,256 shm 1 12K 12K 19166K 1 0 0 16K kqueue 0 0K 2K 19166K 3338072 0 0 256,1K sigio 1 1K 1K 19166K 1 0 0 32 file 188 12K 13K 19166K 19935274 0 0 64 file desc 21 6K 7K 19166K 2610192 0 0 256,512,1K dev_t 607 76K 76K 19166K 607 0 0 128 timecounter 5 1K 1K 19166K 5 0 0 128 kld 4 1K 1K 19166K 1288 0 0 16,32,128 AD driver 1 1K 2K 19166K 285926 0 0 64,1K sem 3 6K 6K 19166K 3 0 0 1K,4K ATA generic 2 1K 1K 19166K 2 0 0 16,512 msg 4 25K 25K 19166K 4 0 0 512,4K,16K rman 35 2K 2K 19166K 392 0 0 16,64 ioctlops 0 0K 1K 19166K 10 0 0 1K taskqueue 2 1K 1K 19166K 2 0 0 32 MD disk 3 3K 3K 19166K 3 0 0 16,1K eventhandler 12 1K 1K 19166K 12 0 0 32,64 bus 325 22K 35K 19166K 757 0 0 16,32,64,128,256,512,1K,8K,16K sysctl 0 0K 1K 19166K 52002 0 0 16,32 uidinfo 3 1K 1K 19166K 5 0 0 32,128 cred 3 1K 1K 19166K 5184723 0 0 128 subproc 44 3K 3K 19166K 5219840 0 0 32,64,256 proc 2 1K 1K 19166K 2 0 0 512 session 8 1K 1K 19166K 32 0 0 64 pgrp 8 1K 1K 19166K 32 0 0 32 AR driver 1 1K 3K 19166K 3 0 0 64,512,2K temp 30370 15148K 15169K 19166K 16804822 0 0 16,32,64,128,256,512,1K,4K,8K,32K,256K,512K devbuf 44 5K 5K 19166K 61313 0 0 16,32,64,128,256,512 Memory Totals: In Use Free Requests 15991K 638K 74254473 --Boundary-00=_2NQ/DNCe++IlWBH Content-Type: text/plain; charset="us-ascii"; name="M0N0WALL_WRAP" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="M0N0WALL_WRAP" machine i386 cpu I586_CPU ident M0N0WALL_WRAP maxusers 0 options INCLUDE_CONFIG_FILE #makeoptions DEBUG=-g #Build kernel with gdb(1) debug symbols makeoptions MODULES_OVERRIDE="dummynet if_tap if_vlan ipfw" options INET #InterNETworking options FAST_IPSEC options FFS #Berkeley Fast Filesystem options FFS_ROOT #FFS usable as root device [keep this!] options SOFTUPDATES #Enable FFS soft updates support options MFS #Memory Filesystem options MD_ROOT #MD is a potential root device options PROCFS #Process filesystem options COMPAT_43 #Compatible with BSD 4.3 [KEEP THIS!] options SCSI_DELAY=15000 #Delay (in ms) before probing SCSI options UCONSOLE #Allow users to grab the console options KTRACE #ktrace(1) support options SYSVSHM #SYSV-style shared memory options SYSVMSG #SYSV-style message queues options SYSVSEM #SYSV-style semaphores options P1003_1B #Posix P1003_1B real-time extensions options _KPOSIX_PRIORITY_SCHEDULING options ICMP_BANDLIM #Rate limit bad replies options HZ=1000 options IPFILTER options IPFILTER_LOG options IPFILTER_DEFAULT_BLOCK options IPSTATE_SIZE=42859 options IPSTATE_MAX=30000 options IPFILTER_MSSCLAMP_FORCE options IPFIREWALL_DEFAULT_TO_ACCEPT options BRIDGE options DEVICE_POLLING options NO_SWAPPING device isa device pci # ATA and ATAPI devices device ata device atadisk # ATA disk drives options ATA_STATIC_ID #Static device numbering options ATA_DISABLE_SLAVE # Floating point support - do not disable. device npx0 at nexus? port IO_NPX irq 13 # Power management support (see LINT for more options) device apm0 at nexus? disable flags 0x20 # Advanced Power Management # Serial (COM) ports device sio0 at isa? port IO_COM1 flags 0x30 irq 4 device sio1 at isa? disable port IO_COM2 irq 3 device sio2 at isa? disable port IO_COM3 irq 5 device sio3 at isa? disable port IO_COM4 irq 9 # PCI Ethernet NICs that use the common MII bus controller code. # NOTE: Be sure to keep the 'device miibus' line in order to use these NICs! device miibus # MII bus support device sis # Silicon Integrated Systems SiS 900/SiS 7016 # WaveLAN/IEEE 802.11 wireless NICs. Note: the WaveLAN/IEEE really # exists only as a PCMCIA device, so there is no ISA attachment needed # and resources will always be dynamically assigned by the pccard code. device wi # Aironet 4500/4800 802.11 wireless NICs. Note: the declaration below will # work for PCMCIA and PCI cards, as well as ISA cards set to ISA PnP # mode (the factory default). If you set the switches on your ISA # card for a manually chosen I/O address and IRQ, you must specify # those parameters here. device an # Pseudo devices - the number indicates how many units to allocate. pseudo-device loop # Network loopback pseudo-device ether # Ethernet support pseudo-device tun # Packet tunnel. pseudo-device pty # Pseudo-ttys (telnet etc) pseudo-device md # Memory "disks" pseudo-device gif # IPv6 and IPv4 tunneling # The `bpf' pseudo-device enables the Berkeley Packet Filter. # Be aware of the administrative consequences of enabling this! pseudo-device bpf #Berkeley packet filter options NETGRAPH #netgraph(4) system options NETGRAPH_ASYNC options NETGRAPH_BPF options NETGRAPH_ETHER options NETGRAPH_IFACE options NETGRAPH_KSOCKET options NETGRAPH_L2TP options NETGRAPH_MPPC_ENCRYPTION options NETGRAPH_PPP options NETGRAPH_PPPOE options NETGRAPH_PPTPGRE options NETGRAPH_SOCKET options NETGRAPH_TEE options NETGRAPH_UI options NETGRAPH_VJC pseudo-device crypto pseudo-device cryptodev device hifn --Boundary-00=_2NQ/DNCe++IlWBH Content-Type: text/plain; charset="us-ascii"; name="warp-ps-auxwww.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="warp-ps-auxwww.txt" USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 14609 0.0 0.0 0 0 ?? Z 9:55AM 0:00.00 (mini_httpd) root 14613 0.0 0.8 1332 968 ?? SN 9:55AM 0:00.02 sh -c ps auxwww root 14612 0.0 0.3 876 340 ?? S 9:55AM 0:00.01 sleep 1 root 14610 0.0 1.5 2376 1776 ?? S 9:55AM 0:00.02 /usr/local/sbin/mini_httpd -S -E /var/etc/cert.pem -c **.php|**.cgi -u root -maxproc 16 -i /var/run/mini_httpd.pid root 14608 0.0 2.9 3952 3364 ?? SN 9:55AM 0:00.31 /usr/local/bin/php exec.php root 8561 0.0 1.4 2436 1612 ?? SNs 9:03AM 0:01.19 /usr/local/sbin/mpd -b -d /var/etc/mpd-vpn -p /var/run/mpd-vpn.pid pptpd nobody 59248 0.0 0.7 1040 808 ?? IN 2:34AM 0:01.13 /usr/local/sbin/dnsmasq root 150 0.0 0.8 1336 904 ?? I 1Jan00 0:00.03 /bin/sh /etc/rc.initial console root 106 0.0 0.7 1332 840 d0- S 1Jan00 70:36.84 /bin/sh /usr/local/bin/runmsntp.sh /var/run/runmsntp.pid /var/run/msntp.pid 300 ntp.example.net root 98 0.0 1.4 2292 1644 ?? Ss 1Jan00 1:37.09 /usr/local/sbin/mini_httpd -S -E /var/etc/cert.pem -c **.php|**.cgi -u root -maxproc 16 -i /var/run/mini_httpd.pid root 95 0.0 0.7 1036 768 ?? Ss 1Jan00 1:30.45 /usr/sbin/syslogd -s -f /var/etc/syslog.conf root 80 0.0 1.1 1456 1224 ?? Ss 1Jan00 2:08.59 /sbin/ipmon -sD root 69 0.0 1.1 2256 1288 ?? Ss 1Jan00 1:02.11 /usr/local/sbin/mpd -b -d /var/etc -p /var/run/mpd.pid pppoe root 7 0.0 0.0 0 0 ?? DL 1Jan00 0:14.51 (vnlru) root 6 0.0 0.0 0 0 ?? DL 1Jan00 1:03.47 (syncer) root 5 0.0 0.0 0 0 ?? DL 1Jan00 0:12.63 (bufdaemon) root 4 0.0 0.0 0 0 ?? DL 1Jan00 0:02.88 (pagedaemon) root 3 0.0 0.0 0 0 ?? DL 1Jan00 0:00.00 (taskqueue) root 2 0.0 0.0 0 0 ?? DL 1Jan00 0:00.00 (cryptoret) root 1 0.0 0.6 1060 696 ?? ILs 1Jan00 0:04.17 /sbin/init -- root 0 0.0 0.0 0 0 ?? DLs 1Jan00 0:00.00 (swapper) root 14614 0.0 0.6 1076 676 ?? RN 9:55AM 0:00.00 ps auxwww --Boundary-00=_2NQ/DNCe++IlWBH--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200602230136.06689.viktor.vasilev>