From owner-freebsd-stable Thu Mar 23 15:22:33 2000 Delivered-To: freebsd-stable@freebsd.org Received: from blockhead.mincom.com (blockhead2.mincom.com [203.15.57.33]) by hub.freebsd.org (Postfix) with ESMTP id 7EE7F37C64F for ; Thu, 23 Mar 2000 15:21:20 -0800 (PST) (envelope-from philh@mincom.com) Received: (from uucp@localhost) by blockhead.mincom.com (8.9.3/8.9.3) id JAA03988 for ; Fri, 24 Mar 2000 09:21:07 +1000 (EST) (envelope-from philh@mincom.com) Received: from porthole.mincom.oz.au(172.17.100.2) via SMTP by blockhead.mincom.oz.au, id smtpdjR3986; Fri Mar 24 09:21:00 2000 Received: (from philh@localhost) by porthole.mincom.oz.au (8.9.3/8.9.3/mincom) id JAA24183 for stable@freebsd.org; Fri, 24 Mar 2000 09:21:00 +1000 (EST) (envelope-from philh) Date: Fri, 24 Mar 2000 09:20:59 +1000 From: Phil Homewood To: stable@freebsd.org Subject: System lockup, 3.4-STABLE, two machines Message-ID: <20000324092059.R13576@mincom.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 0.95.5i Sender: owner-freebsd-stable@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG The last two days have seen two lockups on one of our firewall machines. Box is pingable, accepts TCP connections but does nothing more. syslog shows it logging "MARK" every five minutes even when unresponsive. First thought: DoS attack. After cvsupping and recompiling kernel and world yesterday (and forgetting to do the sysctl's for restrict_rst and drop_synfin, d'oh!) it still locked up - but the running kernel had no DDB to investigate. (Now corrected.) So, I decide just for sanity to update one of our other firewall machines as well (as it didn't have the restrict_rst, drop_synfin and icmp_bandlim options). Halfway through "make update" it, too, locked up. Fortunately, this one has DDB compiled in, and I was able to escape to debugger. "trace" shows: db> trace Debugger(c0202755) at Debugger+0x37 scgetc(c0233d44,2,0,c0233d44,0) at scgetc+0x7d8 sckbdevent(c0233d44,0,0,0,c82ab360) at sckbdevent+0x58 atkbd_intr(c0233d44,0,0,0,c01cca5e) at atkbd_intr+0x1f atkbd_isa_intr(0,80000000,80000010,10,0) at atkbd_isa_intr+0x20 Xresume1() at Xresume1+0x2b --- interrupt, eip = 0xc01d588d, esp = 0xc020e244, ebp = 0 --- default_halt() at default_halt+0x1 I have a "ps" dump as well as some "show" output and a vmcore if it helps. Panic forced, system is now dumping (after "syncing disks" failed.) Does the above mean anything to anyone? We did see this behaviour a few months ago but since the last cvsup mid-January it's been behaving. The fact that two machines (similar configs) crashed within a short time of each other is making me think DoS, however it could be something benign in network code that is triggering the fault (if one firewall is unreachable, connections fall back to one of the others...) dmesg from the first machine (that crashed twice, without DDB): Copyright (c) 1992-1999 FreeBSD Inc. Copyright (c) 1982, 1986, 1989, 1991, 1993 The Regents of the University of California. All rights reserved. FreeBSD 3.4-STABLE #6: Thu Mar 23 13:47:35 MST 2000 root@:/usr/src/sys/compile/DENFW Timecounter "i8254" frequency 1193182 Hz Timecounter "TSC" frequency 400910905 Hz CPU: Pentium II/Xeon/Celeron (400.91-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x652 Stepping = 2 Features=0x183f9ff real memory = 100663296 (98304K bytes) avail memory = 95154176 (92924K bytes) Preloaded elf kernel "kernel" at 0xc025b000. Pentium Pro MTRR support enabled Probing for devices on PCI bus 0: chip0: rev 0x03 on pci0.0.0 chip1: rev 0x03 on pci0.1.0 chip2: rev 0x02 on pci0.7.0 ide_pci0: rev 0x01 on pci0.7.1 chip3: rev 0x02 on pci0.7.3 xl0: <3Com 3c905B-TX Fast Etherlink XL> rev 0x30 int a irq 10 on pci0.16.0 xl0: Ethernet address: 00:50:04:af:54:4e xl0: autoneg complete, link status good (half-duplex, 10Mbps) xl1: <3Com 3c905-TX Fast Etherlink XL> rev 0x00 int a irq 12 on pci0.17.0 xl1: Ethernet address: 00:60:97:93:61:a3 xl1: autoneg complete, link status good (half-duplex, 10Mbps) Probing for devices on PCI bus 1: vga0: rev 0x7a on pci1.0.0 Probing for devices on the ISA bus: sc0 on isa sc0: VGA color <16 virtual consoles, flags=0x0> atkbdc0 at 0x60-0x6f on motherboard atkbd0 irq 1 on isa sio0 at 0x3f8-0x3ff irq 4 flags 0x10 on isa sio0: type 16550A, console sio1 at 0x2f8-0x2ff irq 3 on isa sio1: type 16550A fdc0 at 0x3f0-0x3f7 irq 6 drq 2 on isa fdc0: FIFO enabled, 8 bytes threshold fd0: 1.44MB 3.5in wdc0 at 0x1f0-0x1f7 irq 14 on isa wdc0: unit 0 (wd0): wd0: 2014MB (4124736 sectors), 4092 cyls, 16 heads, 63 S/T, 512 B/S ppc0 at 0x378 irq 7 on isa ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppi0: on ppbus 0 plip0: on ppbus 0 vga0 at 0x3b0-0x3df maddr 0xa0000 msize 131072 on isa npx0 on motherboard npx0: INT 16 interface IP packet filtering initialized, divert disabled, rule-based forwarding disabled, default to deny, logging limited to 100 packets/entry by default changing root device to wd0s1a and the second (the one where I got the ddb output above): Copyright (c) 1992-1999 FreeBSD Inc. Copyright (c) 1982, 1986, 1989, 1991, 1993 The Regents of the University of California. All rights reserved. FreeBSD 3.4-STABLE #2: Tue Jan 18 09:46:22 EST 2000 root@blocker.mincom.oz.au:/usr/src/sys/compile/BLOCKER Timecounter "i8254" frequency 1193182 Hz Timecounter "TSC" frequency 400911726 Hz CPU: Celeron (400.91-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x665 Stepping = 5 Features=0x183f9ff real memory = 134217728 (131072K bytes) avail memory = 127754240 (124760K bytes) Preloaded elf kernel "kernel" at 0xc027e000. Pentium Pro MTRR support enabled Probing for devices on PCI bus 0: chip0: rev 0x03 on pci0.0.0 chip1: rev 0x03 on pci0.1.0 chip2: rev 0x02 on pci0.7.0 ide_pci0: rev 0x01 on pci0.7.1 chip3: rev 0x02 on pci0.7.3 xl0: <3Com 3c905B-TX Fast Etherlink XL> rev 0x30 int a irq 11 on pci0.14.0 xl0: Ethernet address: 00:10:5a:72:43:a0 xl0: autoneg complete, link status good (half-duplex, 10Mbps) xl1: <3Com 3c905B-TX Fast Etherlink XL> rev 0x24 int a irq 10 on pci0.18.0 xl1: Ethernet address: 00:10:4b:c5:b6:14 xl1: autoneg complete, link status good (half-duplex, 10Mbps) ahc0: rev 0x00 int a irq 15 on pci0.19.0 ahc0: aic7890/91 Wide Channel A, SCSI Id=7, 16/255 SCBs xl2: <3Com 3c905B-TX Fast Etherlink XL> rev 0x24 int a irq 15 on pci0.20.0 xl2: Ethernet address: 00:10:4b:c5:c3:4d xl2: autoneg complete, link status good (half-duplex, 10Mbps) Probing for devices on PCI bus 1: vga0: rev 0x7a on pci1.0.0 Probing for devices on the ISA bus: sc0 on isa sc0: VGA color <16 virtual consoles, flags=0x0> atkbdc0 at 0x60-0x6f on motherboard atkbd0 irq 1 on isa sio0 at 0x3f8-0x3ff irq 4 flags 0x10 on isa sio0: type 16550A, console sio1 at 0x2f8-0x2ff irq 3 on isa sio1: type 16550A fdc0 at 0x3f0-0x3f7 irq 6 drq 2 on isa fdc0: FIFO enabled, 8 bytes threshold fd0: 1.44MB 3.5in wdc0 at 0x1f0-0x1f7 irq 14 on isa wdc0: unit 0 (atapi): , removable, dma, iordis acd0: drive speed 6890KB/sec, 128KB cache acd0: supported read types: CD-R, CD-RW, CD-DA acd0: Audio: play, 255 volume levels acd0: Mechanism: ejectable tray acd0: Medium: no/blank disc inside, unlocked ppc0 not found vga0 at 0x3b0-0x3df maddr 0xa0000 msize 131072 on isa npx0 on motherboard npx0: INT 16 interface IP packet filtering initialized, divert disabled, rule-based forwarding disabled, logging limited to 100 packets/entry by default Waiting 2 seconds for SCSI devices to settle cda0 at ahc0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-3 device da0: 80.000MB/s transfers (40.000MHz, offset 31, 16bit), Tagged Queueing Enabled da0: 8761MB (17942584 512 byte sectors: 255H 63S/T 1116C) da1 at ahc0 bus 0 target 1 lun 0 da1: Fixed Direct Access SCSI-3 device da1: 80.000MB/s transfers (40.000MHz, offset 31, 16bit), Tagged Queueing Enabled da1: 8761MB (17942584 512 byte sectors: 255H 63S/T 1116C) hanging root device to da0s1a (The hardware was sourced from two different vendors - the firewalls are in two different countries.) I'm off to build world on the second machine in the hope that the TCP/ICMP changes recently will halt this game. But I'd like to know if anyone else has seen this, or knows whether I'm heading in the right direction with the DoS theory. Thanks in advance for any advice... -- This transmission is for the intended addressee only and is confidential information. If you have received this transmission in error, please delete it and notify the sender. The contents of this email are the opinion of the writer and are not endorsed by Mincom Ltd unless expressly stated otherwise. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message