From owner-freebsd-hackers@FreeBSD.ORG  Thu Feb 23 00:36:11 2006
Return-Path: <owner-freebsd-hackers@FreeBSD.ORG>
X-Original-To: hackers@freebsd.org
Delivered-To: freebsd-hackers@FreeBSD.ORG
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 026CA16A422
	for <hackers@freebsd.org>; Thu, 23 Feb 2006 00:36:10 +0000 (GMT)
	(envelope-from viktor.vasilev@stud.tu-darmstadt.de)
Received: from lnx130.hrz.tu-darmstadt.de (lnx130.hrz.tu-darmstadt.de
	[130.83.174.24])
	by mx1.FreeBSD.org (Postfix) with ESMTP id E9D8843D45
	for <hackers@freebsd.org>; Thu, 23 Feb 2006 00:36:09 +0000 (GMT)
	(envelope-from viktor.vasilev@stud.tu-darmstadt.de)
Received: from mailserver3.hrz.tu-darmstadt.de (lnx115.hrz.tu-darmstadt.de
	[130.83.174.27])
	by lnx130.hrz.tu-darmstadt.de (8.12.10/8.12.10) with ESMTP id
	k1N0a84x028811
	for <hackers@freebsd.org>; Thu, 23 Feb 2006 01:36:08 +0100
Received: from [130.83.20.203] (helo=ABC216.ram1st.wh.tu-darmstadt.de)
	by mailserver3.hrz.tu-darmstadt.de with esmtpsa
	(TLS-1.0:RSA_ARCFOUR_MD5:16) (Exim 4.50) id 1FC4Sl-00038K-Qs
	for hackers@freebsd.org; Thu, 23 Feb 2006 01:36:07 +0100
From: Viktor Vasilev <viktor.vasilev@stud.tu-darmstadt.de>
To: hackers@freebsd.org
Date: Thu, 23 Feb 2006 01:36:06 +0100
User-Agent: KMail/1.8.3
MIME-Version: 1.0
Content-Type: Multipart/Mixed;
  boundary="Boundary-00=_2NQ/DNCe++IlWBH"
Message-Id: <200602230136.06689.viktor.vasilev@stud.tu-darmstadt.de>
Cc: 
Subject: Looking for advice on chasing a kernel memory leak
X-BeenThere: freebsd-hackers@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Technical Discussions relating to FreeBSD
	<freebsd-hackers.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>, 
	<mailto:freebsd-hackers-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-hackers>
List-Post: <mailto:freebsd-hackers@freebsd.org>
List-Help: <mailto:freebsd-hackers-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>,
	<mailto:freebsd-hackers-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 23 Feb 2006 00:36:11 -0000

--Boundary-00=_2NQ/DNCe++IlWBH
Content-Type: text/plain;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Hello,

I'm having trouble with a WARP (http://www.pcengines.ch/wrap.htm) board 
running m0n0wall v1.21 (stripped down FreeBSD 4.11-RELEASE-p13). It's serving 
as an internet gateway and the problem is, that after some time it starts 
blocking traffic and doesn't do NAT anymore. The box is handling very low 
traffic volume and is mostly idle. I've enabled logging to a remote machine 
and around the time the trouble happens, there are messages like these:

  Jan 15 04:02:25 gw /kernel: ipf_nattable_max reduced to -96
  Jan 15 04:02:49 gw /kernel: ipf_nattable_max reduced to -94
  Jan 15 04:02:49 gw /kernel: ipf_nattable_max reduced to -94
  Jan 15 04:04:31 gw /kernel: ipf_nattable_max reduced to -92

An inspection of the ipfilter code shows that kmem_alloc is failing:

  
http://fxr.watson.org/fxr/source/contrib/ipfilter/netinet/ip_nat.c?v=RELENG4#L1197

A reboot fixes the things until the same thing happens again in three or so 
weeks. Since that happened a couple of times, I've monitored RAM usage and 
see a constant growth of the wired memory. After reboot top reports:

  Mem: 4312K Active, 3684K Inact, 5960K Wired, 4848K Buf, 99M Free

Now after two weeks:

  Mem: 5044K Active, 3824K Inact, 22M Wired, 5856K Buf, 82M Free

vmstat -m output (attached) clearly shows that there's a huge amount of M_TEMP 
memory held and growing:

  temp 30370 15148K  15169K 19166K 16804822    0     0  
16,32,64,128,256,512,1K,4K,8K,32K,256K,512K

I don't know how to find out who actually allocates this memory. Any ideas?

ps auxwww output, kernel config and dmesg are attached.


Cheers,
Viktor

--Boundary-00=_2NQ/DNCe++IlWBH
Content-Type: text/plain;
  charset="us-ascii";
  name="wrap-dmesg.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename="wrap-dmesg.txt"

Copyright (c) 1992-2005 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD 4.11-RELEASE-p13 #0: Sat Nov 26 12:17:56 CET 2005
    root@fb411.neon1.net:/usr/src/sys/compile/M0N0WALL_WRAP
Timecounter "i8254"  frequency 1193182 Hz
CPU: NSC Geode (266.64-MHz 586-class CPU)
  Origin = "Geode by NSC"  Id = 0x540  Stepping = 0  DIR=0x81b7
  Features=0x808131<FPU,TSC,MSR,CX8,CMOV,MMX>
real memory  = 134217728 (131072K bytes)
avail memory = 116117504 (113396K bytes)
Preloaded elf kernel "kernel" at 0xc0e03000.
Preloaded mfs_root "/mfsroot" at 0xc0e030a8.
md0: Preloaded image </mfsroot> 11534336 bytes at 0xc0301df0
md1: Malloc disk
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcib0: <Host to PCI bridge> on motherboard
pci0: <PCI bus> on pcib0
sis0: <NatSemi DP83815 10/100BaseTX> port 0x1000-0x10ff mem 0x80000000-0x80000fff irq 10 at device 14.0 on pci0
sis0: Ethernet address: 00:0d:b9:02:c4:d0
miibus0: <MII bus> on sis0
ukphy0: <Generic IEEE 802.3u media interface> on miibus0
ukphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
sis1: <NatSemi DP83815 10/100BaseTX> port 0x1400-0x14ff mem 0x80040000-0x80040fff irq 9 at device 15.0 on pci0
sis1: Ethernet address: 00:0d:b9:02:c4:d1
miibus1: <MII bus> on sis1
ukphy1: <Generic IEEE 802.3u media interface> on miibus1
ukphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
sis2: <NatSemi DP83815 10/100BaseTX> port 0x1800-0x18ff mem 0x80080000-0x80080fff irq 11 at device 16.0 on pci0
sis2: Ethernet address: 00:0d:b9:02:c4:d2
miibus2: <MII bus> on sis2
ukphy2: <Generic IEEE 802.3u media interface> on miibus2
ukphy2:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
isab0: <PCI to ISA bridge (vendor=100b device=0510)> port 0xf600-0xf63f,0xf400-0xf43f at device 18.0 on pci0
isa0: <ISA bus> on isab0
chip1: <PCI to Other bridge (vendor=100b device=0511)> port 0xf000-0xf0ff at device 18.1 on pci0
atapci0: <Generic PCI ATA controller> port 0xfc00-0xfc0f at device 18.2 on pci0
ata0: at 0x1f0 irq 14 on atapci0
pci0: <unknown card> (vendor=0x100b, dev=0x0503) at 18.3
chip2: <PCI to Other bridge (vendor=100b device=0515)> port 0xf200-0xf23f at device 18.5 on pci0
orm0: <Option ROM> at iomem 0xe0000-0xe7fff on isa0
pmtimer0 on isa0
sio0 at port 0x3f8-0x3ff irq 4 flags 0x30 on isa0
sio0: type 16550A, console
BRIDGE 020214 loaded
IPsec: Initialized Security Association Processing.
IP Filter: v3.4.35 initialized.  Default = block all, Logging = enabled
ad0: 123MB <PRCF1002> [251/16/63] at ata0-master PIO4
Mounting root from ufs:/dev/md0c
stray irq 7

--Boundary-00=_2NQ/DNCe++IlWBH
Content-Type: text/plain;
  charset="us-ascii";
  name="wrap-vmstat-m.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename="wrap-vmstat-m.txt"

Memory statistics by bucket size
Size   In Use   Free   Requests  HighWater  Couldfree
  16      573    195   11103972    1280          0
  32      563     77    2902431     640          0
  64     1828     28   25873127     320     228322
 128     1070    114   18465333     160          0
 256     1382   2378    7479540      80     305488
 512    29603     13    3252875      40      10200
  1K       39      5    5045708      20          0
  2K       25      3      88792      10      17358
  4K        5      1      42295       5          0
  8K       22      0        383       5        220
 16K        4      0         12       5          0
 32K        2      0          2       5          0
256K        1      0          1       5          0
512K        0      0          2       5          0

Memory usage type by bucket size
Size  Type(s)
  16  nexusdev, p1003.1b, key mgmt, netgraph, routetbl, ether_multi,
          vnodes, mount, pcb, soname, atexit, proc-args, kld, ATA generic,
          rman, MD disk, bus, sysctl, temp, devbuf
  32  in_multi, netgraph, routetbl, ether_multi, ifaddr, BPF, vnodes,
          cluster_save buffer, pcb, soname, proc-args, sigio, kld, taskqueue,
          eventhandler, bus, sysctl, uidinfo, subproc, pgrp, temp, devbuf
  64  isadev, netgraph, routetbl, ether_multi, ifaddr, vnodes,
          cluster_save buffer, vfscache, lockf, pcb, proc-args, file,
          AD driver, rman, eventhandler, bus, subproc, session, AR driver,
          temp, devbuf
 128  ZONE, key mgmt, netgraph, routetbl, ifaddr, vnodes, mount, soname,
          ttys, zombie, proc-args, dev_t, timecounter, kld, bus, uidinfo, cred,
          temp, devbuf
 256  FFS node, newblk, netgraph, routetbl, ifaddr, vnodes, ttys,
          proc-args, kqueue, file desc, bus, subproc, temp, devbuf
 512  crypto, UFS mount, netgraph, mount, BIO buffer, ptys, file desc,
          ATA generic, msg, bus, proc, AR driver, temp, devbuf
  1K  Export Host, netgraph, BIO buffer, kqueue, file desc, AD driver, sem,
          ioctlops, MD disk, bus, temp
  2K  UFS mount, netgraph, ifaddr, BIO buffer, pcb, AR driver
  4K  mbuf, pagedep, sem, msg, temp
  8K  VM pgdata, syncache, netgraph, bus, temp
 16K  UFS ihash, inodedep, shm, msg, bus
 32K  vfscache, temp
256K  temp
512K  temp

Memory statistics by type                          Type  Kern
        Type  InUse MemUse HighUse  Limit Requests Limit Limit Size(s)
     nexusdev     4     1K      1K 19166K        4    0     0  16
         mbuf     1     4K      4K 19166K        1    0     0  4K
       crypto     1     1K      1K 19166K        1    0     0  512
       isadev     6     1K      1K 19166K        6    0     0  64
         ZONE    15     2K      2K 19166K       15    0     0  128
    VM pgdata     1     8K      8K 19166K        1    0     0  8K
    UFS mount     6     9K      9K 19166K    61275    0     0  512,2K
    UFS ihash     1    16K     16K 19166K        1    0     0  16K
     FFS node   798   200K    200K 19166K   103331    0     0  256
       newblk     1     1K      1K 19166K        1    0     0  256
     inodedep     1    16K     16K 19166K        1    0     0  16K
      pagedep     1     4K      4K 19166K        1    0     0  4K
     p1003.1b     1     1K      1K 19166K        1    0     0  16
     key mgmt    72     7K     18K 19166K  6899993    0     0  16,128
     syncache     1     8K      8K 19166K        1    0     0  8K
  Export Host     1     1K      1K 19166K        1    0     0  1K
     in_multi    26     1K      1K 19166K       26    0     0  32
     netgraph   912   215K    219K 19166K   804800    0     0  16,32,64,128,256,512,1K,2K,8K
     routetbl    52     7K     11K 19166K     1809    0     0  16,32,64,128,256
  ether_multi    55     2K      2K 19166K       55    0     0  16,32,64
       ifaddr    47    12K     12K 19166K      440    0     0  32,64,128,256,2K
          BPF    21     1K      1K 19166K      373    0     0  32
       vnodes    18     5K      5K 19166K      224    0     0  16,32,64,128,256
        mount     3     2K      2K 19166K    20428    0     0  16,128,512
cluster_save buffer     0     0K      1K 19166K       11    0     0  32,64
     vfscache  1190   107K    107K 19166K   145578    0     0  64,32K
   BIO buffer    28    28K     30K 19166K    81825    0     0  512,1K,2K
        lockf     2     1K      1K 19166K       48    0     0  64
          pcb    48     6K      6K 19166K  2229913    0     0  16,32,64,2K
       soname     1     1K      1K 19166K  5273087    0     0  16,32,128
         ptys     1     1K      1K 19166K        1    0     0  512
         ttys    90    12K     12K 19166K      168    0     0  128,256
       atexit     1     1K      1K 19166K        1    0     0  16
       zombie     1     1K      1K 19166K  2609899    0     0  128
    proc-args    13     1K      2K 19166K  2525828    0     0  16,32,64,128,256
          shm     1    12K     12K 19166K        1    0     0  16K
       kqueue     0     0K      2K 19166K  3338072    0     0  256,1K
        sigio     1     1K      1K 19166K        1    0     0  32
         file   188    12K     13K 19166K 19935274    0     0  64
    file desc    21     6K      7K 19166K  2610192    0     0  256,512,1K
        dev_t   607    76K     76K 19166K      607    0     0  128
  timecounter     5     1K      1K 19166K        5    0     0  128
          kld     4     1K      1K 19166K     1288    0     0  16,32,128
    AD driver     1     1K      2K 19166K   285926    0     0  64,1K
          sem     3     6K      6K 19166K        3    0     0  1K,4K
  ATA generic     2     1K      1K 19166K        2    0     0  16,512
          msg     4    25K     25K 19166K        4    0     0  512,4K,16K
         rman    35     2K      2K 19166K      392    0     0  16,64
     ioctlops     0     0K      1K 19166K       10    0     0  1K
    taskqueue     2     1K      1K 19166K        2    0     0  32
      MD disk     3     3K      3K 19166K        3    0     0  16,1K
 eventhandler    12     1K      1K 19166K       12    0     0  32,64
          bus   325    22K     35K 19166K      757    0     0  16,32,64,128,256,512,1K,8K,16K
       sysctl     0     0K      1K 19166K    52002    0     0  16,32
      uidinfo     3     1K      1K 19166K        5    0     0  32,128
         cred     3     1K      1K 19166K  5184723    0     0  128
      subproc    44     3K      3K 19166K  5219840    0     0  32,64,256
         proc     2     1K      1K 19166K        2    0     0  512
      session     8     1K      1K 19166K       32    0     0  64
         pgrp     8     1K      1K 19166K       32    0     0  32
    AR driver     1     1K      3K 19166K        3    0     0  64,512,2K
         temp 30370 15148K  15169K 19166K 16804822    0     0  16,32,64,128,256,512,1K,4K,8K,32K,256K,512K
       devbuf    44     5K      5K 19166K    61313    0     0  16,32,64,128,256,512

Memory Totals:  In Use       Free    Requests
                15991K       638K    74254473

--Boundary-00=_2NQ/DNCe++IlWBH
Content-Type: text/plain;
  charset="us-ascii";
  name="M0N0WALL_WRAP"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename="M0N0WALL_WRAP"


machine		i386
cpu 		I586_CPU
ident		M0N0WALL_WRAP
maxusers	0
options		INCLUDE_CONFIG_FILE

#makeoptions	DEBUG=-g		#Build kernel with gdb(1) debug symbols
makeoptions	MODULES_OVERRIDE="dummynet if_tap if_vlan ipfw"

options 	INET			#InterNETworking
options		FAST_IPSEC
options 	FFS			#Berkeley Fast Filesystem
options 	FFS_ROOT		#FFS usable as root device [keep this!]
options 	SOFTUPDATES		#Enable FFS soft updates support
options 	MFS			#Memory Filesystem
options 	MD_ROOT			#MD is a potential root device
options 	PROCFS			#Process filesystem
options 	COMPAT_43		#Compatible with BSD 4.3 [KEEP THIS!]
options 	SCSI_DELAY=15000	#Delay (in ms) before probing SCSI
options 	UCONSOLE		#Allow users to grab the console
options 	KTRACE			#ktrace(1) support
options 	SYSVSHM			#SYSV-style shared memory
options 	SYSVMSG			#SYSV-style message queues
options 	SYSVSEM			#SYSV-style semaphores
options 	P1003_1B		#Posix P1003_1B real-time extensions
options 	_KPOSIX_PRIORITY_SCHEDULING
options		ICMP_BANDLIM		#Rate limit bad replies

options		HZ=1000

options		IPFILTER
options		IPFILTER_LOG
options		IPFILTER_DEFAULT_BLOCK
options		IPSTATE_SIZE=42859
options		IPSTATE_MAX=30000
options		IPFILTER_MSSCLAMP_FORCE
options		IPFIREWALL_DEFAULT_TO_ACCEPT

options		BRIDGE
options		DEVICE_POLLING

options		NO_SWAPPING

device		isa
device		pci

# ATA and ATAPI devices
device		ata
device		atadisk			# ATA disk drives
options 	ATA_STATIC_ID		#Static device numbering
options		ATA_DISABLE_SLAVE

# Floating point support - do not disable.
device		npx0	at nexus? port IO_NPX irq 13

# Power management support (see LINT for more options)
device		apm0	at nexus? disable flags 0x20 # Advanced Power Management

# Serial (COM) ports
device		sio0	at isa? port IO_COM1 flags 0x30 irq 4
device		sio1	at isa? disable port IO_COM2 irq 3
device		sio2	at isa? disable port IO_COM3 irq 5
device		sio3	at isa? disable port IO_COM4 irq 9

# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
device		miibus		# MII bus support
device		sis		# Silicon Integrated Systems SiS 900/SiS 7016

# WaveLAN/IEEE 802.11 wireless NICs. Note: the WaveLAN/IEEE really
# exists only as a PCMCIA device, so there is no ISA attachment needed
# and resources will always be dynamically assigned by the pccard code.
device		wi

# Aironet 4500/4800 802.11 wireless NICs. Note: the declaration below will
# work for PCMCIA and PCI cards, as well as ISA cards set to ISA PnP
# mode (the factory default). If you set the switches on your ISA
# card for a manually chosen I/O address and IRQ, you must specify
# those parameters here.
device		an

# Pseudo devices - the number indicates how many units to allocate.
pseudo-device	loop		# Network loopback
pseudo-device	ether		# Ethernet support
pseudo-device	tun		# Packet tunnel.
pseudo-device	pty		# Pseudo-ttys (telnet etc)
pseudo-device	md		# Memory "disks"
pseudo-device	gif		# IPv6 and IPv4 tunneling

# The `bpf' pseudo-device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
pseudo-device	bpf		#Berkeley packet filter

options 	NETGRAPH		#netgraph(4) system
options 	NETGRAPH_ASYNC
options		NETGRAPH_BPF
options 	NETGRAPH_ETHER
options 	NETGRAPH_IFACE
options 	NETGRAPH_KSOCKET
options 	NETGRAPH_L2TP
options 	NETGRAPH_MPPC_ENCRYPTION
options 	NETGRAPH_PPP
options 	NETGRAPH_PPPOE
options 	NETGRAPH_PPTPGRE
options 	NETGRAPH_SOCKET
options 	NETGRAPH_TEE
options 	NETGRAPH_UI
options 	NETGRAPH_VJC

pseudo-device	crypto
pseudo-device	cryptodev
device       	hifn

--Boundary-00=_2NQ/DNCe++IlWBH
Content-Type: text/plain;
  charset="us-ascii";
  name="warp-ps-auxwww.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename="warp-ps-auxwww.txt"

USER     PID %CPU %MEM   VSZ  RSS  TT  STAT STARTED      TIME COMMAND
root   14609  0.0  0.0     0    0  ??  Z     9:55AM   0:00.00  (mini_httpd)
root   14613  0.0  0.8  1332  968  ??  SN    9:55AM   0:00.02 sh -c ps auxwww
root   14612  0.0  0.3   876  340  ??  S     9:55AM   0:00.01 sleep 1
root   14610  0.0  1.5  2376 1776  ??  S     9:55AM   0:00.02 /usr/local/sbin/mini_httpd -S -E /var/etc/cert.pem -c **.php|**.cgi -u root -maxproc 16 -i /var/run/mini_httpd.pid
root   14608  0.0  2.9  3952 3364  ??  SN    9:55AM   0:00.31 /usr/local/bin/php exec.php
root    8561  0.0  1.4  2436 1612  ??  SNs   9:03AM   0:01.19 /usr/local/sbin/mpd -b -d /var/etc/mpd-vpn -p /var/run/mpd-vpn.pid pptpd
nobody 59248  0.0  0.7  1040  808  ??  IN    2:34AM   0:01.13 /usr/local/sbin/dnsmasq
root     150  0.0  0.8  1336  904  ??  I     1Jan00   0:00.03 /bin/sh /etc/rc.initial console
root     106  0.0  0.7  1332  840  d0- S     1Jan00  70:36.84 /bin/sh /usr/local/bin/runmsntp.sh /var/run/runmsntp.pid /var/run/msntp.pid 300  ntp.example.net
root      98  0.0  1.4  2292 1644  ??  Ss    1Jan00   1:37.09 /usr/local/sbin/mini_httpd -S -E /var/etc/cert.pem -c **.php|**.cgi -u root -maxproc 16 -i /var/run/mini_httpd.pid
root      95  0.0  0.7  1036  768  ??  Ss    1Jan00   1:30.45 /usr/sbin/syslogd -s -f /var/etc/syslog.conf
root      80  0.0  1.1  1456 1224  ??  Ss    1Jan00   2:08.59 /sbin/ipmon -sD
root      69  0.0  1.1  2256 1288  ??  Ss    1Jan00   1:02.11 /usr/local/sbin/mpd -b -d /var/etc -p /var/run/mpd.pid pppoe
root       7  0.0  0.0     0    0  ??  DL    1Jan00   0:14.51  (vnlru)
root       6  0.0  0.0     0    0  ??  DL    1Jan00   1:03.47  (syncer)
root       5  0.0  0.0     0    0  ??  DL    1Jan00   0:12.63  (bufdaemon)
root       4  0.0  0.0     0    0  ??  DL    1Jan00   0:02.88  (pagedaemon)
root       3  0.0  0.0     0    0  ??  DL    1Jan00   0:00.00  (taskqueue)
root       2  0.0  0.0     0    0  ??  DL    1Jan00   0:00.00  (cryptoret)
root       1  0.0  0.6  1060  696  ??  ILs   1Jan00   0:04.17 /sbin/init --
root       0  0.0  0.0     0    0  ??  DLs   1Jan00   0:00.00  (swapper)
root   14614  0.0  0.6  1076  676  ??  RN    9:55AM   0:00.00 ps auxwww

--Boundary-00=_2NQ/DNCe++IlWBH--