Date: Fri, 24 Mar 2000 09:20:59 +1000 From: Phil Homewood <philh@mincom.com> To: stable@freebsd.org Subject: System lockup, 3.4-STABLE, two machines Message-ID: <20000324092059.R13576@mincom.com>
next in thread | raw e-mail | index | archive | help
The last two days have seen two lockups on one of our firewall
machines. Box is pingable, accepts TCP connections but does
nothing more. syslog shows it logging "MARK" every five minutes
even when unresponsive.
First thought: DoS attack. After cvsupping and recompiling
kernel and world yesterday (and forgetting to do the sysctl's
for restrict_rst and drop_synfin, d'oh!) it still locked up -
but the running kernel had no DDB to investigate. (Now
corrected.)
So, I decide just for sanity to update one of our other
firewall machines as well (as it didn't have the restrict_rst,
drop_synfin and icmp_bandlim options). Halfway through "make
update" it, too, locked up. Fortunately, this one has DDB
compiled in, and I was able to escape to debugger.
"trace" shows:
db> trace
Debugger(c0202755) at Debugger+0x37
scgetc(c0233d44,2,0,c0233d44,0) at scgetc+0x7d8
sckbdevent(c0233d44,0,0,0,c82ab360) at sckbdevent+0x58
atkbd_intr(c0233d44,0,0,0,c01cca5e) at atkbd_intr+0x1f
atkbd_isa_intr(0,80000000,80000010,10,0) at atkbd_isa_intr+0x20
Xresume1() at Xresume1+0x2b
--- interrupt, eip = 0xc01d588d, esp = 0xc020e244, ebp = 0 ---
default_halt() at default_halt+0x1
I have a "ps" dump as well as some "show" output and a vmcore if
it helps.
Panic forced, system is now dumping (after "syncing disks" failed.)
Does the above mean anything to anyone? We did see this behaviour
a few months ago but since the last cvsup mid-January it's been
behaving. The fact that two machines (similar configs) crashed
within a short time of each other is making me think DoS, however
it could be something benign in network code that is triggering
the fault (if one firewall is unreachable, connections fall back
to one of the others...)
dmesg from the first machine (that crashed twice, without DDB):
Copyright (c) 1992-1999 FreeBSD Inc.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
FreeBSD 3.4-STABLE #6: Thu Mar 23 13:47:35 MST 2000
root@:/usr/src/sys/compile/DENFW
Timecounter "i8254" frequency 1193182 Hz
Timecounter "TSC" frequency 400910905 Hz
CPU: Pentium II/Xeon/Celeron (400.91-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0x652 Stepping = 2
Features=0x183f9ff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR>
real memory = 100663296 (98304K bytes)
avail memory = 95154176 (92924K bytes)
Preloaded elf kernel "kernel" at 0xc025b000.
Pentium Pro MTRR support enabled
Probing for devices on PCI bus 0:
chip0: <Intel 82443BX host to PCI bridge> rev 0x03 on pci0.0.0
chip1: <Intel 82443BX host to AGP bridge> rev 0x03 on pci0.1.0
chip2: <Intel 82371AB PCI to ISA bridge> rev 0x02 on pci0.7.0
ide_pci0: <Intel PIIX4 Bus-master IDE controller> rev 0x01 on pci0.7.1
chip3: <Intel 82371AB Power management controller> rev 0x02 on pci0.7.3
xl0: <3Com 3c905B-TX Fast Etherlink XL> rev 0x30 int a irq 10 on pci0.16.0
xl0: Ethernet address: 00:50:04:af:54:4e
xl0: autoneg complete, link status good (half-duplex, 10Mbps)
xl1: <3Com 3c905-TX Fast Etherlink XL> rev 0x00 int a irq 12 on pci0.17.0
xl1: Ethernet address: 00:60:97:93:61:a3
xl1: autoneg complete, link status good (half-duplex, 10Mbps)
Probing for devices on PCI bus 1:
vga0: <ATI model 475a graphics accelerator> rev 0x7a on pci1.0.0
Probing for devices on the ISA bus:
sc0 on isa
sc0: VGA color <16 virtual consoles, flags=0x0>
atkbdc0 at 0x60-0x6f on motherboard
atkbd0 irq 1 on isa
sio0 at 0x3f8-0x3ff irq 4 flags 0x10 on isa
sio0: type 16550A, console
sio1 at 0x2f8-0x2ff irq 3 on isa
sio1: type 16550A
fdc0 at 0x3f0-0x3f7 irq 6 drq 2 on isa
fdc0: FIFO enabled, 8 bytes threshold
fd0: 1.44MB 3.5in
wdc0 at 0x1f0-0x1f7 irq 14 on isa
wdc0: unit 0 (wd0): <ST32122A>
wd0: 2014MB (4124736 sectors), 4092 cyls, 16 heads, 63 S/T, 512 B/S
ppc0 at 0x378 irq 7 on isa
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppi0: <generic parallel i/o> on ppbus 0
plip0: <PLIP network interface> on ppbus 0
vga0 at 0x3b0-0x3df maddr 0xa0000 msize 131072 on isa
npx0 on motherboard
npx0: INT 16 interface
IP packet filtering initialized, divert disabled, rule-based forwarding disabled, default to deny, logging limited to 100 packets/entry by default
changing root device to wd0s1a
and the second (the one where I got the ddb output above):
Copyright (c) 1992-1999 FreeBSD Inc.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
FreeBSD 3.4-STABLE #2: Tue Jan 18 09:46:22 EST 2000
root@blocker.mincom.oz.au:/usr/src/sys/compile/BLOCKER
Timecounter "i8254" frequency 1193182 Hz
Timecounter "TSC" frequency 400911726 Hz
CPU: Celeron (400.91-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0x665 Stepping = 5
Features=0x183f9ff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR>
real memory = 134217728 (131072K bytes)
avail memory = 127754240 (124760K bytes)
Preloaded elf kernel "kernel" at 0xc027e000.
Pentium Pro MTRR support enabled
Probing for devices on PCI bus 0:
chip0: <Intel 82443BX host to PCI bridge> rev 0x03 on pci0.0.0
chip1: <Intel 82443BX host to AGP bridge> rev 0x03 on pci0.1.0
chip2: <Intel 82371AB PCI to ISA bridge> rev 0x02 on pci0.7.0
ide_pci0: <Intel PIIX4 Bus-master IDE controller> rev 0x01 on pci0.7.1
chip3: <Intel 82371AB Power management controller> rev 0x02 on pci0.7.3
xl0: <3Com 3c905B-TX Fast Etherlink XL> rev 0x30 int a irq 11 on pci0.14.0
xl0: Ethernet address: 00:10:5a:72:43:a0
xl0: autoneg complete, link status good (half-duplex, 10Mbps)
xl1: <3Com 3c905B-TX Fast Etherlink XL> rev 0x24 int a irq 10 on pci0.18.0
xl1: Ethernet address: 00:10:4b:c5:b6:14
xl1: autoneg complete, link status good (half-duplex, 10Mbps)
ahc0: <Adaptec 2940 Ultra2 SCSI adapter> rev 0x00 int a irq 15 on pci0.19.0
ahc0: aic7890/91 Wide Channel A, SCSI Id=7, 16/255 SCBs
xl2: <3Com 3c905B-TX Fast Etherlink XL> rev 0x24 int a irq 15 on pci0.20.0
xl2: Ethernet address: 00:10:4b:c5:c3:4d
xl2: autoneg complete, link status good (half-duplex, 10Mbps)
Probing for devices on PCI bus 1:
vga0: <ATI model 4757 graphics accelerator> rev 0x7a on pci1.0.0
Probing for devices on the ISA bus:
sc0 on isa
sc0: VGA color <16 virtual consoles, flags=0x0>
atkbdc0 at 0x60-0x6f on motherboard
atkbd0 irq 1 on isa
sio0 at 0x3f8-0x3ff irq 4 flags 0x10 on isa
sio0: type 16550A, console
sio1 at 0x2f8-0x2ff irq 3 on isa
sio1: type 16550A
fdc0 at 0x3f0-0x3f7 irq 6 drq 2 on isa
fdc0: FIFO enabled, 8 bytes threshold
fd0: 1.44MB 3.5in
wdc0 at 0x1f0-0x1f7 irq 14 on isa
wdc0: unit 0 (atapi): <ATAPI CD-ROM DRIVE 40X MAXIMUM/N0AP>, removable, dma, iordis
acd0: drive speed 6890KB/sec, 128KB cache
acd0: supported read types: CD-R, CD-RW, CD-DA
acd0: Audio: play, 255 volume levels
acd0: Mechanism: ejectable tray
acd0: Medium: no/blank disc inside, unlocked
ppc0 not found
vga0 at 0x3b0-0x3df maddr 0xa0000 msize 131072 on isa
npx0 on motherboard
npx0: INT 16 interface
IP packet filtering initialized, divert disabled, rule-based forwarding disabled, logging limited to 100 packets/entry by default
Waiting 2 seconds for SCSI devices to settle
cda0 at ahc0 bus 0 target 0 lun 0
da0: <QUANTUM ATLAS IV 9 WLS 0808> Fixed Direct Access SCSI-3 device
da0: 80.000MB/s transfers (40.000MHz, offset 31, 16bit), Tagged Queueing Enabled
da0: 8761MB (17942584 512 byte sectors: 255H 63S/T 1116C)
da1 at ahc0 bus 0 target 1 lun 0
da1: <QUANTUM ATLAS IV 9 WLS 0808> Fixed Direct Access SCSI-3 device
da1: 80.000MB/s transfers (40.000MHz, offset 31, 16bit), Tagged Queueing Enabled
da1: 8761MB (17942584 512 byte sectors: 255H 63S/T 1116C)
hanging root device to da0s1a
(The hardware was sourced from two different vendors - the firewalls
are in two different countries.)
I'm off to build world on the second machine in the hope that the
TCP/ICMP changes recently will halt this game. But I'd like to know
if anyone else has seen this, or knows whether I'm heading in the
right direction with the DoS theory. Thanks in advance for any advice...
--
This transmission is for the intended addressee only and is confidential
information. If you have received this transmission in error, please delete
it and notify the sender. The contents of this email are the opinion of the
writer and are not endorsed by Mincom Ltd unless expressly stated otherwise.
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-stable" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20000324092059.R13576>