Date: Tue, 30 Dec 2008 12:50:03 -0500 (EST) From: Jason DiCioccio <jd@ods.org> To: FreeBSD-gnats-submit@FreeBSD.org Subject: kern/130059: Leaking 50k mbufs/hour Message-ID: <20081230175003.710001B663@update.ods.org> Resent-Message-ID: <200812301800.mBUI09jo085199@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 130059
>Category: kern
>Synopsis: Leaking 50k mbufs/hour
>Confidential: no
>Severity: serious
>Priority: high
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Tue Dec 30 18:00:09 UTC 2008
>Closed-Date:
>Last-Modified:
>Originator: Jason DiCioccio
>Release: FreeBSD 7.1-RC2 i386
>Organization:
>Environment:
System: FreeBSD update.ods.org 7.1-RC2 FreeBSD 7.1-RC2 #5: Sat Dec 27 21:16:55 EST 2008 geniusj@update.ods.org:/usr/obj/usr/src/sys/UPDATE i386
FreeBSD 7.1-RC2 #5: Sat Dec 27 21:16:55 EST 2008
geniusj@update.ods.org:/usr/obj/usr/src/sys/UPDATE
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Pentium(R) 4 CPU 2.80GHz (2800.11-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0xf29 Stepping = 9
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x4400<CNXT-ID,xTPR>
real memory = 1071906816 (1022 MB)
avail memory = 1039556608 (991 MB)
acpi0: <INTEL D845GVS1> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, 3fe00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> mem 0xf0000000-0xf7ffffff,0xffa80000-0xffafffff irq 11 at device 2.0 on pci0
agp0: <Intel 82845M (845M GMCH) SVGA controller> on vgapci0
agp0: detected 892k stolen memory
agp0: aperture size is 128M
pci0: <serial bus, USB> at device 29.0 (no driver attached)
pci0: <serial bus, USB> at device 29.1 (no driver attached)
pci0: <serial bus, USB> at device 29.2 (no driver attached)
pci0: <serial bus, USB> at device 29.7 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci1: <ACPI PCI bus> on pcib1
em0: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0xdc00-0xdc3f mem 0xff8e0000-0xff8fffff,0xff8c0000-0xff8dffff irq 7 at device 1.0 on pci1
em0: [FILTER]
em0: Ethernet address: 00:1b:21:26:13:f2
pci1: <network, ethernet> at device 8.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH4 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Sleep Button> on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
cryptosoft0: <software crypto> on motherboard
orm0: <ISA Option ROMs> at iomem 0xcb800-0xcc7ff,0xcc800-0xcd7ff,0xcd800-0xce7ff pnpid ORM0000 on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 8250 or not responding
sio0: [FILTER]
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 2800112132 Hz quality 800
Timecounters tick every 1.000 msec
IPsec: Initialized Security Association Processing.
ad0: 76319MB <WDC WD800BB-00FJA0 13.03G13> at ata0-master UDMA100
Trying to mount root from ufs:/dev/ad0s1a
-- KERNEL CONFIG --
machine i386
cpu I686_CPU
ident UPDATE
# To statically compile in device wiring instead of /boot/device.hints
#hints "GENERIC.hints" # Default places to look for devices.
makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols
options SCHED_ULE # 4BSD scheduler
options PREEMPTION # Enable kernel thread preemption
options INET # InterNETworking
options INET6 # IPv6 communications protocols
options IPSEC
options IPSEC_FILTERTUNNEL
options FFS # Berkeley Fast Filesystem
options GEOM_PART_GPT # GUID Partition Tables.
options GEOM_LABEL # GEOM Labelization
options SOFTUPDATES # Enable FFS soft updates support
options UFS_ACL # Support for access control lists
options UFS_DIRHASH # Improve performance on big directories
options MD_ROOT # MD is a potential root device
options NFSCLIENT # Network Filesystem Client
options NFSSERVER # Network Filesystem Server
options NFS_ROOT # NFS usable as /, requires NFSCLIENT
options MSDOSFS # MSDOS Filesystem
options CD9660 # ISO 9660 Filesystem
options PROCFS # Process filesystem (requires PSEUDOFS)
options PSEUDOFS # Pseudo-filesystem framework
options SCTP # Stream Transmittion Control Protocol
options COMPAT_43TTY # Compatible with BSD 4.3 [KEEP THIS!]
options COMPAT_FREEBSD4 # Compatible with FreeBSD4
options COMPAT_FREEBSD5 # Compatible with FreeBSD5
options COMPAT_FREEBSD6 # Compatible with FreeBSD6
options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI
#options AUDIT # Audit Framework
options KTRACE # ktrace(1) support
options STACK # stack(9) support
options SYSVSHM # SYSV-style shared memory
options SYSVMSG # SYSV-style message queues
options SYSVSEM # SYSV-style semaphores
options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
options AHC_REG_PRETTY_PRINT # Print register bitfields in debug
# output. Adds ~128k to driver.
options AHD_REG_PRETTY_PRINT # Print register bitfields in debug
# output. Adds ~215k to driver.
options ADAPTIVE_GIANT # Giant mutex is adaptive.
options STOP_NMI # Stop CPUs using NMI instead of IPI
options HZ=1000
options SHMMAXPGS=131072
options SEMMNI=128
options SEMMNS=512
options SEMUME=100
options SEMMNU=256
# Bus support.
device eisa
device pci
# ATA and ATAPI devices
device ata
device atadisk # ATA disk drives
options ATA_STATIC_ID # Static device numbering
# atkbdc0 controls both the keyboard and the PS/2 mouse
device atkbdc # AT keyboard controller
device atkbd # AT keyboard
device psm # PS/2 mouse
device vga # VGA video card driver
device splash # Splash screen and screen saver support
# syscons is the default console driver, resembling an SCO console
device sc
device agp # support several AGP chipsets
# Serial (COM) ports
device sio # 8250, 16[45]50 based serial ports
device uart # Generic UART driver
device em # Intel Gigabit Ethernet
# Pseudo devices.
device loop # Network loopback
device random # Entropy device
device ether # Ethernet support
device ppp # Kernel PPP
device tun # Packet tunnel.
device pty # Pseudo-ttys (telnet etc)
device md # Memory "disks"
# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device bpf # Berkeley packet filter
# Crypto Support
device crypto
device cryptodev
-- ifconfig --
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 00:1b:21:26:13:f2
inet6 fe80::21b:21ff:fe26:13f2%em0 prefixlen 64 scopeid 0x1
inet 66.246.72.188 netmask 0xffffff00 broadcast 66.246.72.255
media: Ethernet 100baseTX <full-duplex> (100baseTX <half-duplex>)
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
inet 64.247.11.248 netmask 0xffffffff
inet 64.247.11.249 netmask 0xffffffff
inet 64.247.11.250 netmask 0xffffffff
inet 64.247.11.251 netmask 0xffffffff
inet 64.247.11.252 netmask 0xffffffff
inet 64.247.11.253 netmask 0xffffffff
inet 64.247.11.254 netmask 0xffffffff
inet 64.247.11.255 netmask 0xffffffff
inet 66.29.58.64 netmask 0xffffffff
inet 66.29.58.65 netmask 0xffffffff
inet 66.29.58.66 netmask 0xffffffff
inet 66.29.58.67 netmask 0xffffffff
inet 66.29.58.68 netmask 0xffffffff
inet 66.29.58.69 netmask 0xffffffff
inet 66.29.58.70 netmask 0xffffffff
inet 192.168.15.1 netmask 0xffffffff
inet6 2001:470:1f07:208::beef:cafe prefixlen 64
gre0: flags=9051<UP,POINTOPOINT,RUNNING,LINK0,MULTICAST> metric 0 mtu 1476
tunnel inet 66.246.72.188 --> 70.162.202.242
inet6 fe80::21b:21ff:fe26:13f2%gre0 prefixlen 64 scopeid 0x3
inet 10.8.8.5 --> 10.8.8.6 netmask 0xfffffffc
gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1280
tunnel inet 66.246.72.188 --> 209.51.161.14
inet6 fe80::21b:21ff:fe26:13f2%gif0 prefixlen 64 scopeid 0x4
inet6 2001:470:1f06:208::2 --> 2001:470:1f06:208::1 prefixlen 128
tap0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 00:bd:de:1c:01:00
inet6 fe80::2bd:deff:fe1c:100%tap0 prefixlen 64 scopeid 0x5
inet 10.8.8.1 netmask 0xffffff00 broadcast 10.8.8.255
inet6 2001:470:89e1::1 prefixlen 112
Opened by PID 1267
tun10: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
inet6 fe80::21b:21ff:fe26:13f2%tun10 prefixlen 64 scopeid 0x6
inet 10.8.0.1 --> 10.8.0.2 netmask 0xffffffff
Opened by PID 1300
>Description:
Basically, the box will run fine for about 36 hours, after which it will panic.
Failing to allocate kmem:
Panic String: kmem_malloc(4096): kmem_map too small: 335544320 total allocated
When you take a look at netstat -m, you'll see something like this:
68264/226/68490 mbufs in use (current/cache/total)
256/134/390/25600 mbuf clusters in use (current/cache/total/max)
256/128 mbuf+clusters out of packet secondary zone in use (current/cache)
0/39/39/12800 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
17578K/480K/18058K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/35/6656 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
143 requests for I/O initiated by sendfile
0 calls to protocol drain routines
the current/total mbuf numbers will continually grow until they reach over a
million. bytes allocated to network will also grow (current/total) until it
reaches over 400MB (I increased kmem_size to stave off the inevitable a little
longer)
Code like the following:
while true; do
OLDMBUFS="`vmstat -z | grep mbuf: | awk -F, '{print $3}'`"
sleep 1800
NEWMBUFS="`vmstat -z | grep mbuf: | awk -F, '{print $3}'`"
DIFF=`expr ${NEWMBUFS} - ${OLDMBUFS}`
date >> ${LOG}
echo "New mbufs since last check: ${DIFF}" >> ${LOG}
done
Produces:
Tue Dec 23 18:46:24 EST 2008
New mbufs since last check: 24064
Tue Dec 23 19:16:24 EST 2008
New mbufs since last check: 23775
Tue Dec 23 19:46:25 EST 2008
New mbufs since last check: 23567
Tue Dec 23 20:16:25 EST 2008
New mbufs since last check: 23322
I've tried both fxp0 and em0 with the same result. I've tried swapped everything
but the hard drive as well. This problem was occurring in both 7.0-RELEASE-p6
and 7.1-RC1-2.
>How-To-Repeat:
There's a lot going on on this box as far as networking features being used..
I'm using IPv4, IPv6, if_gif, if_tap, if_tun, etc.. So it's hard to narrow it
down to one thing. The latest thing I had done was to add an AAAA record for
our primary domain. However, the IPv6 traffic coming in is fairly light.
>Fix:
No idea. But I'd love it if there were a way that I could remotely dig into
these mbufs and find out what traffic is triggering the problem. I have
vmcore files lying around that I could use.
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20081230175003.710001B663>