From owner-freebsd-hackers Mon Mar 9 15:05:26 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id PAA23355 for freebsd-hackers-outgoing; Mon, 9 Mar 1998 15:05:26 -0800 (PST) (envelope-from owner-freebsd-hackers@FreeBSD.ORG) Received: from Kitten.mcs.com (Kitten.mcs.com [192.160.127.90]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id PAA23254; Mon, 9 Mar 1998 15:05:01 -0800 (PST) (envelope-from karl@Mars.mcs.net) Received: from Mars.mcs.net (karl@Mars.mcs.net [192.160.127.85]) by Kitten.mcs.com (8.8.7/8.8.2) with ESMTP id QAA26773; Mon, 9 Mar 1998 16:43:20 -0600 (CST) Received: (from karl@localhost) by Mars.mcs.net (8.8.7/8.8.2) id QAA11086; Mon, 9 Mar 1998 16:43:20 -0600 (CST) Message-ID: <19980309164320.63724@mcs.net> Date: Mon, 9 Mar 1998 16:43:20 -0600 From: Karl Denninger To: current@FreeBSD.ORG Cc: hackers@FreeBSD.ORG Subject: Oh, how I hate it when I find leaks in mbuf clusters :-) Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 0.84 Sender: owner-freebsd-hackers@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Hi folks, Heh, dig this nice problem...... This morning I took down our 100BaseTX hub to replace it with an Intel 510T switch. It helped a lot :-) As a consequence, the servers sat unterminated on that interface for some 5-10 minutes. When I got done with the switch install I plugged it all in and turned the switch on, thinking nothing of it. BIG MISTAKE! It turns out that for some reason the NFS server code was queueing mbufs - and never getting them off the stack. Almost a dozen hours later, under the heaviest load of the day, down goes Mr. Server with a "out of mbuf clusters - increase MAXUSERS" error. Say what? (There's no way it legitimately ran out). It comes back up, and is running happily with 400-500 mbufs in use. When it panic'd it had to have over 15,000 of 'em on the stack. So I go check its sister machine (which hasn't died - yet) and find: $ netstat -m 20228/20544 mbufs in use: 20185 mbufs allocated to data 43 mbufs allocated to packet headers 2419/2686 mbuf clusters in use 7940 Kbytes allocated to network (92% in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines Yikes! Obviously, I'm going to have to shut it down before it shuts down for me :-) Anyone have ideas on this one? "dmesg" for the kernel is: Copyright (c) 1982, 1986, 1989, 1991, 1993 The Regents of the University of California. All rights reserved. FreeBSD 3.0-CURRENT #0: Sat Nov 22 13:41:31 CST 1997 karl@Codebase.mcs.net:/usr/src/sys/compile/MCS_NEWS CPU: Pentium Pro (199.43-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x617 Stepping=7 Features=0xfbff real memory = 100663296 (98304K bytes) avail memory = 94986240 (92760K bytes) Probing for devices on PCI bus 0: Correcting Natoma config for non-SMP chip0: rev 0x02 on pci0.0.0 chip1: rev 0x01 on pci0.7.0 ide_pci0: rev 0x00 on pci0.7.1 de0: rev 0x12 int a irq 9 on pci0.11.0 de0: SMC 9332DST 21140 [10-100Mb/s] pass 1.2 de0: address 00:00:c0:2d:57:ed de0: enabling 10baseT port de1: rev 0x12 int a irq 10 on pci0.12.0 de1: SMC 9332DST 21140 [10-100Mb/s] pass 1.2 de1: address 00:00:c0:35:57:ed de1: enabling 100baseTX port ahc0: rev 0x00 int a irq 11 on pci0.13.0 ahc0: aic7880 Wide Channel, SCSI Id=7, 16 SCBs ahc0: waiting for scsi devices to settle scbus0 at ahc0 bus 0 ahc0: target 0 Tagged Queuing Device sd0 at scbus0 target 0 lun 0 sd0: type 0 fixed SCSI 2 sd0: Direct-Access 34729MB (71124992 512 byte sectors) sd0: with 4341 cyls, 128 heads, and an average 128 sectors/track Probing for devices on the ISA bus: sc0 at 0x60-0x6f irq 1 on motherboard sc0: VGA color <16 virtual consoles, flags=0x0> ed0 not found at 0x280 ed1 not found at 0x300 sio0 at 0x3f8-0x3ff irq 4 on isa sio0: type 16550A sio1 at 0x2f8-0x2ff irq 3 on isa sio1: type 16550A lpt0 at 0x3bc-0x3c3 irq 7 on isa lpt0: Interrupt-driven port lp0: TCP/IP capable interface lpt1 not found fdc0 at 0x3f0-0x3f7 irq 6 drq 2 on isa fdc0: FIFO enabled, 8 bytes threshold fd0: 1.44MB 3.5in wdreset: error1: 0x0 wdreset: error1: 0x0 wdc0 not found at 0x1f0 aha0 not found at 0x330 aic0 not found at 0x340 npx0 on motherboard npx0: INT 16 interface ccd0-3: Concatenated disk drivers CC'd to hackers and -current; I've no idea where the "right" place for this one is, as this kernel isn't really "that" current. -- -- Karl Denninger (karl@MCS.Net)| MCSNet - Serving Chicagoland and Wisconsin http://www.mcs.net/ | T1's from $600 monthly to FULL DS-3 Service | NEW! K56Flex support on ALL modems Voice: [+1 312 803-MCS1 x219]| EXCLUSIVE NEW FEATURE ON ALL PERSONAL ACCOUNTS Fax: [+1 312 803-4929] | *SPAMBLOCK* Technology now included at no cost To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message