Date: Mon, 9 Mar 1998 16:43:20 -0600 From: Karl Denninger <karl@mcs.net> To: current@FreeBSD.ORG Cc: hackers@FreeBSD.ORG Subject: Oh, how I hate it when I find leaks in mbuf clusters :-) Message-ID: <19980309164320.63724@mcs.net>
next in thread | raw e-mail | index | archive | help
Hi folks,
Heh, dig this nice problem......
This morning I took down our 100BaseTX hub to replace it with an Intel 510T
switch. It helped a lot :-)
As a consequence, the servers sat unterminated on that interface for some
5-10 minutes. When I got done with the switch install I plugged it all
in and turned the switch on, thinking nothing of it.
BIG MISTAKE!
It turns out that for some reason the NFS server code was queueing mbufs -
and never getting them off the stack. Almost a dozen hours later, under the
heaviest load of the day, down goes Mr. Server with a "out of mbuf clusters
- increase MAXUSERS" error.
Say what? (There's no way it legitimately ran out).
It comes back up, and is running happily with 400-500 mbufs in use. When it
panic'd it had to have over 15,000 of 'em on the stack.
So I go check its sister machine (which hasn't died - yet) and find:
$ netstat -m
20228/20544 mbufs in use:
20185 mbufs allocated to data
43 mbufs allocated to packet headers
2419/2686 mbuf clusters in use
7940 Kbytes allocated to network (92% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
Yikes!
Obviously, I'm going to have to shut it down before it shuts down for me :-)
Anyone have ideas on this one? "dmesg" for the kernel is:
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
FreeBSD 3.0-CURRENT #0: Sat Nov 22 13:41:31 CST 1997
karl@Codebase.mcs.net:/usr/src/sys/compile/MCS_NEWS
CPU: Pentium Pro (199.43-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0x617 Stepping=7
Features=0xfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV>
real memory = 100663296 (98304K bytes)
avail memory = 94986240 (92760K bytes)
Probing for devices on PCI bus 0:
Correcting Natoma config for non-SMP
chip0: <Intel 82440FX (Natoma) PCI and memory controller> rev 0x02 on pci0.0.0
chip1: <Intel 82371SB PCI to ISA bridge> rev 0x01 on pci0.7.0
ide_pci0: <Intel PIIX3 Bus-master IDE controller> rev 0x00 on pci0.7.1
de0: <Digital 21140 Fast Ethernet> rev 0x12 int a irq 9 on pci0.11.0
de0: SMC 9332DST 21140 [10-100Mb/s] pass 1.2
de0: address 00:00:c0:2d:57:ed
de0: enabling 10baseT port
de1: <Digital 21140 Fast Ethernet> rev 0x12 int a irq 10 on pci0.12.0
de1: SMC 9332DST 21140 [10-100Mb/s] pass 1.2
de1: address 00:00:c0:35:57:ed
de1: enabling 100baseTX port
ahc0: <Adaptec 2940 Ultra SCSI host adapter> rev 0x00 int a irq 11 on pci0.13.0
ahc0: aic7880 Wide Channel, SCSI Id=7, 16 SCBs
ahc0: waiting for scsi devices to settle
scbus0 at ahc0 bus 0
ahc0: target 0 Tagged Queuing Device
sd0 at scbus0 target 0 lun 0
sd0: <CMD TECH CRD-5440 C1-1> type 0 fixed SCSI 2
sd0: Direct-Access 34729MB (71124992 512 byte sectors)
sd0: with 4341 cyls, 128 heads, and an average 128 sectors/track
Probing for devices on the ISA bus:
sc0 at 0x60-0x6f irq 1 on motherboard
sc0: VGA color <16 virtual consoles, flags=0x0>
ed0 not found at 0x280
ed1 not found at 0x300
sio0 at 0x3f8-0x3ff irq 4 on isa
sio0: type 16550A
sio1 at 0x2f8-0x2ff irq 3 on isa
sio1: type 16550A
lpt0 at 0x3bc-0x3c3 irq 7 on isa
lpt0: Interrupt-driven port
lp0: TCP/IP capable interface
lpt1 not found
fdc0 at 0x3f0-0x3f7 irq 6 drq 2 on isa
fdc0: FIFO enabled, 8 bytes threshold
fd0: 1.44MB 3.5in
wdreset: error1: 0x0
wdreset: error1: 0x0
wdc0 not found at 0x1f0
aha0 not found at 0x330
aic0 not found at 0x340
npx0 on motherboard
npx0: INT 16 interface
ccd0-3: Concatenated disk drivers
CC'd to hackers and -current; I've no idea where the "right" place for this
one is, as this kernel isn't really "that" current.
--
--
Karl Denninger (karl@MCS.Net)| MCSNet - Serving Chicagoland and Wisconsin
http://www.mcs.net/ | T1's from $600 monthly to FULL DS-3 Service
| NEW! K56Flex support on ALL modems
Voice: [+1 312 803-MCS1 x219]| EXCLUSIVE NEW FEATURE ON ALL PERSONAL ACCOUNTS
Fax: [+1 312 803-4929] | *SPAMBLOCK* Technology now included at no cost
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-current" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?19980309164320.63724>