From owner-freebsd-current Fri Nov 1 7:32:46 2002 Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DAC3637B401 for ; Fri, 1 Nov 2002 07:32:33 -0800 (PST) Received: from prg.traveller.cz (prg.traveller.cz [193.85.2.77]) by mx1.FreeBSD.org (Postfix) with ESMTP id C178443E88 for ; Fri, 1 Nov 2002 07:32:31 -0800 (PST) (envelope-from mime@traveller.cz) Received: from prg.traveller.cz (localhost [127.0.0.1]) by prg.traveller.cz (8.12.2[KQ/pukvis]/8.12.2-prg) with ESMTP id gA1FWUkK006692 for ; Fri, 1 Nov 2002 16:32:30 +0100 (CET) Received: from localhost (mime@localhost) by prg.traveller.cz (8.12.2[KQ/pukvis]/8.12.2-prg/submit) with ESMTP id gA1FWUka006689 for ; Fri, 1 Nov 2002 16:32:30 +0100 (CET) Date: Fri, 1 Nov 2002 16:32:30 +0100 (CET) From: Michal Mertl To: current@freebsd.org Subject: crash with network load (in tcp syncache ?) Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-current@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG I'm getting panics on SMP -CURRENT while running apachebench (binary ab from apache distribution, not the Perl one) against httpd on the machine. The panics don't occur when I have WITNESS and INVARIANTS turned on. I'm running apache server from ports with no special configuration. I'm running (on different machine) ./ab -c 100000 http://host/index.html. The panics occur after bit more than the number of connections I have set as kern.ipc.maxsockets (17000 by default, 30000 incresed) - probably because the first ones expire (when I set net.inet.tcp.msl to 5000 I can't make it panic - the number of tcpcb in vm.zone doesn't grow past about 6500 - it doesn't go much down even long after the benchmark run finishes but that's ok I suppose). I had while 1; sysctl -a |grep tcpcb sleep 1 end running and the output was like this .... tcpcb: 604, 30000, 28551, 57, 28354 tcpcb: 604, 30000, 29301, 57, 29104 tcpcb: 604, 30000, 29926, 56, 29729 - and then panic into ddb with backtrace below (the one posted here is actually from kern.ipc.maxsockets being 17000) My kernel config is basically GENERIC with stripped HW the machine doesn't dontain. ----- verbose booting ----- /boot/kernel/kernel text=0x209c10 data=0x2ae18+0x3d54c syms=[0x4+0x2da40+0x4+0x37495] Hit [Enter] to boot immediately, or any other key for command prompt. Booting [/boot/kernel/kernel]... /boot/kernel/acpi.ko text=0x380ec data=0x1a38+0xae8 syms=[0x4+0x56e0+0x4+0x733b] SMAP type=01 base=00000000 00000000 len=00000000 0009f000 SMAP type=02 base=00000000 0009f000 len=00000000 00001000 SMAP type=02 base=00000000 000f0000 len=00000000 00010000 SMAP type=01 base=00000000 00100000 len=00000000 1fefd000 SMAP type=03 base=00000000 1fffd000 len=00000000 00002000 SMAP type=04 base=00000000 1ffff000 len=00000000 00001000 SMAP type=02 base=00000000 fec00000 len=00000000 00001000 SMAP type=02 base=00000000 fee00000 len=00000000 00001000 SMAP type=02 base=00000000 ffff0000 len=00000000 00010000 Copyright (c) 1992-2002 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 5.0-CURRENT #0: Thu Oct 31 15:34:37 CET 2002 mime@mimet.eunet.cz:/usr/obj/usr/src/sys/TESTIK Preloaded elf kernel "/boot/kernel/kernel" at 0xc0422000. Preloaded elf module "/boot/kernel/acpi.ko" at 0xc04220a8. Calibrating clock(s) ... TSC clock: 751634930 Hz, i8254 clock: 1193071 Hz CLK_USE_I8254_CALIBRATION not specified - using default frequency Timecounter "i8254" frequency 1193182 Hz CLK_USE_TSC_CALIBRATION not specified - using old calibration method CPU: Pentium III/Pentium III Xeon/Celeron (751.71-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x683 Stepping = 3 Features=0x383fbff real memory = 536858624 (524276K bytes) Physical memory chunk(s): 0x00001000 - 0x0009dfff, 643072 bytes (157 pages) 0x0044c000 - 0x1ffbcfff, 532090880 bytes (129905 pages) avail memory = 515866624 (503776K bytes) Programming 24 pins in IOAPIC #0 IOAPIC #0 intpin 2 -> irq 0 FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): apic id: 1, version: 0x00040011, at 0xfee00000 cpu1 (AP): apic id: 0, version: 0x00040011, at 0xfee00000 io0 (APIC): apic id: 2, version: 0x00170011, at 0xfec00000 bios32: Found BIOS32 Service Directory header at 0xc00f9e20 bios32: Entry = 0xf0530 (c00f0530) Rev = 0 Len = 1 pcibios: PCI BIOS entry at 0xf0000+0x730 pnpbios: Found PnP BIOS data at 0xc00fd270 pnpbios: Entry = f0000:d2a0 Rev = 1.0 pnpbios: OEM ID cd041 Other BIOS signatures found: Initializing GEOMetry subsystem null: random: mem: Pentium Pro MTRR support enabled SMP: CPU0 bsp_apic_configure(): lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000010 SVR: 0x000001ff npx0: on motherboard npx0: INT 16 interface acpi0: on motherboard pci_open(1): mode 1 addr port (0x0cf8) is 0x80002358 pci_open(1a): mode1res=0x80000000 (0x80000000) pci_cfgcheck: device 0 [class=060000] [hdr=00] is there (id=71908086) Using $PIR table, 6 entries at 0xc00f0d20 PCI-Only Interrupts: none Location Bus Device Pin Link IRQs slot 1 0 12 A 0x60 3 4 5 7 9 10 11 12 slot 1 0 12 B 0x61 3 4 5 7 9 10 11 12 slot 1 0 12 C 0x62 3 4 5 7 9 10 11 12 slot 1 0 12 D 0x63 3 4 5 7 9 10 11 12 slot 2 0 11 A 0x61 3 4 5 7 9 10 11 12 slot 2 0 11 B 0x62 3 4 5 7 9 10 11 12 slot 2 0 11 C 0x63 3 4 5 7 9 10 11 12 slot 2 0 11 D 0x60 3 4 5 7 9 10 11 12 slot 3 0 10 A 0x62 3 4 5 7 9 10 11 12 slot 3 0 10 B 0x63 3 4 5 7 9 10 11 12 slot 3 0 10 C 0x60 3 4 5 7 9 10 11 12 slot 3 0 10 D 0x61 3 4 5 7 9 10 11 12 slot 4 0 9 A 0x63 3 4 5 7 9 10 11 12 slot 4 0 9 B 0x60 3 4 5 7 9 10 11 12 slot 4 0 9 C 0x61 3 4 5 7 9 10 11 12 slot 4 0 9 D 0x62 3 4 5 7 9 10 11 12 embedded 0 4 A 0x60 3 4 5 7 9 10 11 12 embedded 0 4 B 0x61 3 4 5 7 9 10 11 12 embedded 0 4 C 0x62 3 4 5 7 9 10 11 12 embedded 0 4 D 0x63 3 4 5 7 9 10 11 12 embedded 0 1 A 0x60 3 4 5 7 9 10 11 12 embedded 0 1 B 0x61 3 4 5 7 9 10 11 12 embedded 0 1 C 0x62 3 4 5 7 9 10 11 12 embedded 0 1 D 0x63 3 4 5 7 9 10 11 12 acpi0: power button is handled as a fixed feature programming model. ACPI timer looks BAD min = 2, max = 5, width = 4 ACPI timer looks BAD min = 2, max = 5, width = 4 ACPI timer looks BAD min = 2, max = 5, width = 4 ACPI timer looks BAD min = 2, max = 16776973, width = 16776972 ACPI timer looks BAD min = 2, max = 5, width = 4 ACPI timer looks BAD min = 2, max = 5, width = 4 ACPI timer looks BAD min = 1, max = 5, width = 5 ACPI timer looks BAD min = 2, max = 5, width = 4 ACPI timer looks BAD min = 1, max = 6, width = 6 ACPI timer looks BAD min = 2, max = 5, width = 4 Timecounter "ACPI-safe" frequency 3579545 Hz acpi_timer0: <24-bit timer at 3.579545MHz> port 0xe408-0xe40b on acpi0 acpi_cpu0: on acpi0 acpi_cpu1: on acpi0 acpi_button0: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 ---- initial configuration ------------------------ ---- before setting priority for links ------------ ---- before fixup boot-disabled links ------------- ---- after fixup boot-disabled links -------------- ---- arbitrated configuration --------------------- pci0: on pcib0 pci0: physical bus=0 map[10]: type 3, range 32, base e4000000, size 26, enabled found-> vendor=0x8086, dev=0x7190, revid=0x03 bus=0, slot=0, func=0 class=06-00-00, hdrtype=0x00, mfdev=0 cmdreg=0x0106, statreg=0x2210, cachelnsz=0 (dwords) lattimer=0x40 (1920 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) found-> vendor=0x8086, dev=0x7191, revid=0x03 bus=0, slot=1, func=0 class=06-04-00, hdrtype=0x01, mfdev=0 cmdreg=0x0117, statreg=0x0220, cachelnsz=0 (dwords) lattimer=0x40 (1920 ns), mingnt=0x88 (34000 ns), maxlat=0x00 (0 ns) found-> vendor=0x8086, dev=0x7110, revid=0x02 bus=0, slot=4, func=0 class=06-01-00, hdrtype=0x00, mfdev=1 cmdreg=0x000f, statreg=0x0280, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) map[20]: type 4, range 32, base 0000d800, size 4, port disabled found-> vendor=0x8086, dev=0x7111, revid=0x01 bus=0, slot=4, func=1 class=01-01-80, hdrtype=0x00, mfdev=0 cmdreg=0x0000, statreg=0x0280, cachelnsz=0 (dwords) lattimer=0x20 (960 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) IOAPIC #0 intpin 19 -> irq 2 Freeing (NOT implemented) redirected PCI irq 10. map[20]: type 4, range 32, base 0000d400, size 5, enabled found-> vendor=0x8086, dev=0x7112, revid=0x01 bus=0, slot=4, func=2 class=0c-03-00, hdrtype=0x00, mfdev=0 cmdreg=0x0005, statreg=0x0280, cachelnsz=0 (dwords) lattimer=0x20 (960 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=d, irq=2 map[90]: type 4, range 32, base 0000e800, size 4, enabled found-> vendor=0x8086, dev=0x7113, revid=0x02 bus=0, slot=4, func=3 class=06-80-00, hdrtype=0x00, mfdev=0 cmdreg=0x0003, statreg=0x0280, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) Freeing (NOT implemented) redirected PCI irq 10. map[10]: type 1, range 32, base d7800000, size 12, enabled map[14]: type 4, range 32, base 0000d000, size 6, enabled map[18]: type 1, range 32, base d7000000, size 20, enabled found-> vendor=0x8086, dev=0x1229, revid=0x08 bus=0, slot=9, func=0 class=02-00-00, hdrtype=0x00, mfdev=0 cmdreg=0x0017, statreg=0x0290, cachelnsz=8 (dwords) lattimer=0x20 (960 ns), mingnt=0x08 (2000 ns), maxlat=0x38 (14000 ns) intpin=a, irq=2 powerspec 2 supports D0 D1 D2 D3 current D0 found-> vendor=0x1044, dev=0xa500, revid=0x01 bus=0, slot=10, func=0 class=06-04-00, hdrtype=0x01, mfdev=1 cmdreg=0x0006, statreg=0x04b0, cachelnsz=8 (dwords) lattimer=0x20 (960 ns), mingnt=0x04 (1000 ns), maxlat=0x00 (0 ns) IOAPIC #0 intpin 18 -> irq 10 Freeing (NOT implemented) redirected PCI irq 11. map[10]: type 3, range 32, base e0000000, size 25, enabled found-> vendor=0x1044, dev=0xa501, revid=0x01 bus=0, slot=10, func=1 class=0e-00-01, hdrtype=0x00, mfdev=1 cmdreg=0x0116, statreg=0x04b0, cachelnsz=8 (dwords) lattimer=0x40 (1920 ns), mingnt=0x01 (250 ns), maxlat=0x01 (250 ns) intpin=a, irq=10 powerspec 2 supports D0 D3 current D0 agp0: mem 0xe4000000-0xe7ffffff at device 0.0 on pci0 agp0: allocating GATT for aperture of size 64M pcib1: at device 1.0 on pci0 pcib1: secondary bus 1 pcib1: subordinate bus 1 pcib1: I/O decode 0xe000-0xdfff pcib1: memory decode 0xd8000000-0xdfefffff pcib1: prefetched decode 0xe3f00000-0xe3ffffff pci1: on pcib1 pci1: physical bus=1 map[10]: type 1, range 32, base d8000000, size 26, enabled found-> vendor=0x5333, dev=0x8a13, revid=0x02 bus=1, slot=0, func=0 class=03-00-00, hdrtype=0x00, mfdev=0 cmdreg=0x0007, statreg=0x0210, cachelnsz=0 (dwords) lattimer=0x40 (1920 ns), mingnt=0x04 (1000 ns), maxlat=0xff (63750 ns) powerspec 1 supports D0 D1 D2 D3 current D0 pci1: at device 0.0 (no driver attached) isab0: at device 4.0 on pci0 isa0: on isab0 atapci0: port 0xd800-0xd80f at device 4.1 on pci0 atapci0: Busmastering DMA not supported ata0: iobase=0x01f0 altiobase=0x03f6 bmaddr=0x0000 ata0: at 0x1f0 irq 14 on atapci0 ata1: iobase=0x0170 altiobase=0x0376 bmaddr=0x0000 ata1: at 0x170 irq 15 on atapci0 pci0: at device 4.2 (no driver attached) intpm0: port 0xe800-0xe80f irq 9 at device 4.3 on pci0 intpm0: I/O mapped e800 intpm0: intr IRQ 9 enabled revision 0 smbus0: on intsmb0 smb0: on smbus0 intpm0: PM I/O mapped e400 fxp0: port 0xd000-0xd03f mem 0xd7000000-0xd70fffff,0xd7800000-0xd7800fff irq 2 at device 9.0 on pci0 fxp0: using memory space register mapping fxp0: Ethernet address 00:02:b3:2b:9e:0e fxp0: PCI IDs: 8086 1229 8086 000c 0008 fxp0: Dynamic Standby mode is disabled inphy0: on miibus0 inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto bpf: fxp0 attached pcib2: at device 10.0 on pci0 pcib2: secondary bus 2 pcib2: subordinate bus 2 pcib2: I/O decode 0x0-0x0 pcib2: memory decode 0x100000-0xfffff pcib2: prefetched decode 0x100000-0xfffff pci2: on pcib2 pci2: physical bus=2 asr0: mem 0xe0000000-0xe1ffffff irq 10 at device 10.1 on pci0 asr0: major=154 Freeing (NOT implemented) redirected PCI irq 11. asr0: ADAPTEC 3210S FW Rev. 370F, 2 channel, 256 CCBs, Protocol I2O fdc0: port 0x3f7,0x3f2-0x3f5 irq 6 drq 2 on acpi0 fdc0: FIFO enabled, 8 bytes threshold fd0: <2880-KB 3.5" drive (in 1440-KB mode)> on fdc0 drive 0 unknown: not probed (disabled) sio0: irq maps: 0x41 0x51 0x41 0x41 sio0 port 0x3f8-0x3ff irq 4 on acpi0 sio0: type 16550A, console sio1: irq maps: 0x41 0x49 0x41 0x41 sio1 port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A unknown: not probed (disabled) atkbdc0: port 0x64,0x60 irq 1 on acpi0 atkbd0: flags 0x1 irq 1 on atkbdc0 atkbd: the current kbd controller command byte 0047 atkbd: keyboard ID 0x41ab (2) kbdc: RESET_KBD return code:00fa kbdc: RESET_KBD status:00aa kbd0 at atkbd0 kbd0: atkbd0, AT 101/102 (2), config:0x1, flags:0x1d0000 psm0: unable to allocate IRQ psmcpnp0 irq 12 on acpi0 psm0: current command byte:0047 kbdc: TEST_AUX_PORT status:0000 kbdc: RESET_AUX return code:00fa kbdc: RESET_AUX status:00aa kbdc: RESET_AUX ID:0000 psm: status 00 02 64 psm: status 00 03 06 psm: status 00 33 55 psm: data 08 00 00 psm: status 00 02 64 psm0: irq 12 on atkbdc0 psm0: model NetMouse/NetScroll Optical, device ID 0-00, 3 buttons psm0: config:00000000, flags:00000000, packet size:4 psm0: syncmask:08, syncbits:08 unknown: not probed (disabled) unknown: not probed (disabled) ata: ata0 already exists; skipping it ata: ata1 already exists; skipping it atkbdc: atkbdc0 already exists; skipping it fdc: fdc0 already exists; skipping it sio: sio0 already exists; skipping it sio: sio1 already exists; skipping it Trying Read_Port at 203 Trying Read_Port at 243 Trying Read_Port at 283 Trying Read_Port at 2c3 Trying Read_Port at 303 Trying Read_Port at 343 Trying Read_Port at 383 Trying Read_Port at 3c3 sc: sc0 already exists; skipping it vga: vga0 already exists; skipping it isa_probe_children: disabling PnP devices isa_probe_children: probing non-PnP devices orm0: