From owner-freebsd-bugs@FreeBSD.ORG Sun Jul 10 14:20:26 2005 Return-Path: X-Original-To: freebsd-bugs@hub.freebsd.org Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0EEB416A41F for ; Sun, 10 Jul 2005 14:20:26 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 68F7143D48 for ; Sun, 10 Jul 2005 14:20:25 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.3/8.13.3) with ESMTP id j6AEKPlE021787 for ; Sun, 10 Jul 2005 14:20:25 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.13.3/8.13.1/Submit) id j6AEKPAN021780; Sun, 10 Jul 2005 14:20:25 GMT (envelope-from gnats) Resent-Date: Sun, 10 Jul 2005 14:20:25 GMT Resent-Message-Id: <200507101420.j6AEKPAN021780@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Blaz Zupan Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 802E616A41C for ; Sun, 10 Jul 2005 14:13:25 +0000 (GMT) (envelope-from blaz@titanic.medinet.si) Received: from titanic.medinet.si (titanic.medinet.si [212.18.42.5]) by mx1.FreeBSD.org (Postfix) with ESMTP id B2EBE43D55 for ; Sun, 10 Jul 2005 14:13:24 +0000 (GMT) (envelope-from blaz@titanic.medinet.si) Received: by titanic.medinet.si (Postfix, from userid 1000) id 4264E55475; Sun, 10 Jul 2005 16:13:23 +0200 (CEST) Message-Id: <20050710141323.4264E55475@titanic.medinet.si> Date: Sun, 10 Jul 2005 16:13:23 +0200 (CEST) From: Blaz Zupan To: FreeBSD-gnats-submit@FreeBSD.org X-Send-Pr-Version: 3.113 Cc: Subject: kern/83220: Daily crashes on 5.4 SMP (with backtrace) X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Blaz Zupan List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Jul 2005 14:20:26 -0000 >Number: 83220 >Category: kern >Synopsis: Daily crashes on 5.4 SMP (with backtrace) >Confidential: no >Severity: critical >Priority: high >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sun Jul 10 14:20:24 GMT 2005 >Closed-Date: >Last-Modified: >Originator: Blaz Zupan >Release: FreeBSD 5.4-RELEASE >Organization: >Environment: System: FreeBSD in-4.mail.amis.net 5.4-RELEASE-p3 FreeBSD 5.4-RELEASE-p3 #0: Tue Jul 5 18:37:15 CEST 2005 blaz@bigbrother.amis.net:/usr/obj/usr/src5/sys/DL380 i386 Copyright (c) 1992-2005 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 5.4-RELEASE-p3 #0: Tue Jul 5 18:37:15 CEST 2005 blaz@bigbrother.amis.net:/usr/obj/usr/src5/sys/DL380 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(TM) CPU 3.06GHz (3050.08-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf29 Stepping = 9 Features=0xbfebfbff Hyperthreading: 2 logical CPUs real memory = 1073717248 (1023 MB) avail memory = 1045372928 (996 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 6 cpu3 (AP): APIC ID: 7 MADT: Forcing active-low polarity and level trigger for SCI ioapic0 irqs 0-15 on motherboard ioapic1 irqs 16-31 on motherboard ioapic2 irqs 32-47 on motherboard ioapic3 irqs 48-63 on motherboard npx0: on motherboard npx0: INT 16 interface acpi0: on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 1000 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x920-0x923 on acpi0 cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 pcib0: on acpi0 pci0: on pcib0 pci0: at device 3.0 (no driver attached) pci0: at device 4.0 (no driver attached) pci0: at device 4.2 (no driver attached) isab0: at device 15.0 on pci0 isa0: on isab0 atapci0: port 0x2000-0x200f,0x376,0x170-0x177,0x3f6,0x1f0-0x1f7 at device 15.1 on pci0 ata0: channel #0 on atapci0 ata1: channel #1 on atapci0 ohci0: mem 0xf5ef0000-0xf5ef0fff irq 7 at device 15.2 on pci0 usb0: OHCI version 1.0, legacy support usb0: SMM does not respond, resetting usb0: on ohci0 usb0: USB revision 1.0 uhub0: (0x1166) OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 4 ports with 4 removable, self powered pcib1: on acpi0 pci1: on pcib1 ciss0: port 0x3000-0x30ff mem 0xf7cf0000-0xf7cf3fff,0xf7dc0000-0xf7dfffff irq 30 at device 3.0 on pci1 pcib2: on acpi0 pci2: on pcib2 bge0: mem 0xf7ef0000-0xf7efffff irq 29 at device 1.0 on pci2 miibus0: on bge0 brgphy0: on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto bge0: Ethernet address: 00:0e:7f:20:22:91 bge1: mem 0xf7ee0000-0xf7eeffff irq 31 at device 2.0 on pci2 miibus1: on bge1 brgphy1: on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto bge1: Ethernet address: 00:0e:7f:20:22:90 pcib3: on acpi0 pci3: on pcib3 pcib4: on acpi0 pci6: on pcib4 pci6: at device 30.0 (no driver attached) acpi_tz0: on acpi0 atkbdc0: port 0x64,0x60 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 psm0: irq 12 on atkbdc0 psm0: model Generic PS/2 mouse, device ID 0 sio0: port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A fdc0: port 0x3f2-0x3f5 irq 6 drq 2 on acpi0 fd0: <1440-KB 3.5" drive> on fdc0 drive 0 orm0: at iomem 0xee000-0xeffff,0xcc000-0xcd7ff,0xc8000-0xcbfff,0xc0000-0xc7fff on isa0 pmtimer0 on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 10.000 msec IP Filter: v3.4.35 initialized. Default = pass all, Logging = enabled acd0: CDROM at ata0-master PIO4 SMP: AP CPU #2 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! da0 at ciss0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-0 device da0: 135.168MB/s transfers da0: 69455MB (142245120 512 byte sectors: 255H 32S/T 17432C) Kernel config file: machine i386 cpu I686_CPU ident DL380 options SCHED_4BSD # 4BSD scheduler options INET # InterNETworking options INET6 # IPv6 communications protocols options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_ACL # Support for access control lists options UFS_DIRHASH # Improve performance on big directories options MD_ROOT # MD is a potential root device options GEOM_GPT # GUID Partition Tables. options COMPAT_43 # Compatible with BSD 4.3 [KEEP THIS!] options COMPAT_FREEBSD4 # Compatible with FreeBSD4 options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI options KTRACE # ktrace(1) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options KBD_INSTALL_CDEV # install a CDEV entry in /dev options ADAPTIVE_GIANT # Giant mutex is adaptive. options NMBCLUSTERS=12000 options IPFILTER options IPFILTER_LOG options SMP options INCLUDE_CONFIG_FILE options KDB_STOP_NMI options KDB options DDB makeoptions DEBUG=-g #Build kernel with gdb(1) debug symbols device apic # I/O APIC device isa device eisa device pci device fdc device ata device atapicd # ATAPI CDROM drives options ATA_STATIC_ID # Static device numbering device scbus # SCSI bus (required for SCSI) device da # Direct Access (disks) device ciss # Compaq Smart RAID 5* device atkbdc # AT keyboard controller device atkbd # AT keyboard device psm # PS/2 mouse device vga # VGA video card driver device sc device agp # support several AGP chipsets device npx device pmtimer device sio # 8250, 16[45]50 based serial ports device miibus # MII bus support device bge # Broadcom BCM570xx Gigabit Ethernet device loop # Network loopback device mem # Memory and kernel memory devices device io # I/O device device random # Entropy device device ether # Ethernet support device pty # Pseudo-ttys (telnet etc) device md # Memory "disks" device bpf # Berkeley packet filter device ohci # OHCI PCI->USB interface device usb # USB Bus (required) device ukbd # Keyboard device ums # Mouse >Description: Crash happens with both HTT turned on and off in the BIOS. Machine is a heavily loaded incoming mail server, running postfix, amavisd-new and F-Secure under Linux emulation. ipfilter is running. See also this thread on freebsd-stable: http://lists.freebsd.org/pipermail/freebsd-stable/2005-July/016767.html Below is a backtrace on a crashdump. Crashdump and kernel compiled with -g are available on request. [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". #0 doadump () at pcpu.h:159 159 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:159 #1 0xc044b006 in db_fncall (dummy1=0, dummy2=0, dummy3=-1067606609, dummy4=0xe4b6c9d0 "üɶä(\205]Àèɶäìɶä\222\a") at /usr/src5/sys/ddb/db_command.c:531 #2 0xc044ae14 in db_command (last_cmdp=0xc0674644, cmd_table=0x0, aux_cmd_tablep=0xc064226c, aux_cmd_tablep_end=0xc0642270) at /usr/src5/sys/ddb/db_command.c:349 #3 0xc044aedc in db_command_loop () at /usr/src5/sys/ddb/db_command.c:455 #4 0xc044ca75 in db_trap (type=12, code=0) at /usr/src5/sys/ddb/db_main.c:221 #5 0xc04e6599 in kdb_trap (type=12, code=0, tf=0xe4b6cb3c) at /usr/src5/sys/kern/subr_kdb.c:468 #6 0xc05f4c79 in trap_fatal (frame=0xe4b6cb3c, eva=36) at /usr/src5/sys/i386/i386/trap.c:812 #7 0xc05f43e9 in trap (frame= {tf_fs = -1040580584, tf_es = -1029439472, tf_ds = 16, tf_edi = -1038000128, tf_esi = -1066898900, tf_ebp = -457782384, tf_isp = -457782424, tf_ebx = -1040530304, tf_edx = -1040524364, tf_ecx = -1040524544, tf_eax = 0, tf_trapno = 12, tf_err = 0, tf_eip = -1068574101, tf_cs = 8, tf_eflags = 65683, tf_esp = 180, tf_ss = 0}) at /usr/src5/sys/i386/i386/trap.c:255 #8 0xc05e283a in calltrap () at /usr/src5/sys/i386/i386/exception.s:140 #9 0xc1fa0018 in ?? () #10 0xc2a40010 in ?? () #11 0x00000010 in ?? () #12 0xc2216000 in ?? () #13 0xc0686a2c in tcbinfo () #14 0xe4b6cb90 in ?? () #15 0xe4b6cb68 in ?? () #16 0xc1fac480 in ?? () #17 0xc1fadbb4 in ?? () #18 0xc1fadb00 in ?? () #19 0x00000000 in ?? () #20 0x0000000c in ?? () #21 0x00000000 in ?? () #22 0xc04eda6b in propagate_priority (td=0xc2216000) at /usr/src5/sys/kern/subr_turnstile.c:243 #23 0xc04ee225 in turnstile_wait (ts=0xc1fadb00, lock=0xc0686a2c, owner=0xc2216000) at /usr/src5/sys/kern/subr_turnstile.c:556 #24 0xc04c5ced in _mtx_lock_sleep (m=0xc0686a2c, td=0xc1fac480, opts=0, file=0x0, line=0) at /usr/src5/sys/kern/kern_mutex.c:552 #25 0xc0559ad8 in tcp_usr_rcvd (so=0x0, flags=0) at /usr/src5/sys/netinet/tcp_usrreq.c:602 #26 0xc0506103 in soreceive (so=0xc27bf798, psa=0x0, uio=0xe4b6cc88, mp0=0x0, controlp=0x0, flagsp=0x0) at /usr/src5/sys/kern/uipc_socket.c:1395 #27 0xc04f4bd9 in soo_read (fp=0x0, uio=0xe4b6cc88, active_cred=0xc2884a80, flags=0, td=0xc1fac480) at /usr/src5/sys/kern/sys_socket.c:91 #28 0xc04ee865 in dofileread (td=0xc1fac480, fp=0xc2e17bb0, fd=10, buf=0x0, nbyte=4096, offset=Unhandled dwarf expression opcode 0x93 ) at file.h:233 #29 0xc04ee72f in read (td=0xc1fac480, uap=0xe4b6cd14) at /usr/src5/sys/kern/sys_generic.c:107 #30 0xc05f4fe7 in syscall (frame= {tf_fs = 47, tf_es = 47, tf_ds = -1078001617, tf_edi = 10, tf_esi = 300, tf_ebp = -1077942168, tf_isp = -457781900, tf_ebx = 134822152, tf_edx = 0, tf_ecx = 10, tf_eax = 3, tf_trapno = 0, tf_err = 2, tf_eip = 672556795, tf_cs = 31, tf_eflags = 658, tf_esp = -1077942212, tf_ss = 47}) at /usr/src5/sys/i386/i386/trap.c:1009 #31 0xc05e288f in Xint0x80_syscall () at /usr/src5/sys/i386/i386/exception.s:201 #32 0x0000002f in ?? () #33 0x0000002f in ?? () #34 0xbfbf002f in ?? () #35 0x0000000a in ?? () #36 0x0000012c in ?? () #37 0xbfbfe868 in ?? () #38 0xe4b6cd74 in ?? () #39 0x08093908 in ?? () #40 0x00000000 in ?? () #41 0x0000000a in ?? () #42 0x00000003 in ?? () #43 0x00000000 in ?? () #44 0x00000002 in ?? () #45 0x281666fb in ?? () #46 0x0000001f in ?? () #47 0x00000292 in ?? () #48 0xbfbfe83c in ?? () #49 0x0000002f in ?? () #50 0x00000000 in ?? () #51 0x00000000 in ?? () #52 0x00000000 in ?? () #53 0x00000000 in ?? () #54 0x2c75b000 in ?? () #55 0xc22de000 in ?? () #56 0xc1fac480 in ?? () #57 0xe4b6ccac in ?? () #58 0xe4b6cc94 in ?? () #59 0xc1f26000 in ?? () #60 0xc04ded13 in sched_switch (td=0x12c, newtd=0x8093908, flags=Cannot access memory at address 0xbfbfe878 ) at /usr/src5/sys/kern/sched_4bsd.c:881 Previous frame inner to this frame (corrupt stack?) (kgdb) quit >How-To-Repeat: Only happens on SMP boxes, a very simmilar HP DL380 G3 box with only a single processor has currently an uptime of 12 days, while a SMP box crashes at least once daily. >Fix: Unknown >Release-Note: >Audit-Trail: >Unformatted: