From owner-freebsd-amd64@FreeBSD.ORG Thu Jun 30 20:03:03 2005 Return-Path: X-Original-To: freebsd-amd64@freebsd.org Delivered-To: freebsd-amd64@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A908616A41C for ; Thu, 30 Jun 2005 20:03:03 +0000 (GMT) (envelope-from rob@hudson-trading.com) Received: from daemon.mistermishap.net (167-49.nyc.dsl.access.net [166.84.167.49]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0DF6443D1D for ; Thu, 30 Jun 2005 20:03:02 +0000 (GMT) (envelope-from rob@hudson-trading.com) Received: from daemon.mistermishap.net (localhost.mistermishap.net [127.0.0.1]) by daemon.mistermishap.net (8.12.9/8.12.9) with ESMTP id j5UK32mU038887 for ; Thu, 30 Jun 2005 16:03:02 -0400 (EDT) (envelope-from rob@hudson-trading.com) Received: from localhost (rob@localhost) by daemon.mistermishap.net (8.12.9/8.12.9/Submit) with ESMTP id j5UK32fs038884 for ; Thu, 30 Jun 2005 16:03:02 -0400 (EDT) X-Authentication-Warning: daemon.mistermishap.net: rob owned process doing -bs Date: Thu, 30 Jun 2005 16:03:02 -0400 (EDT) From: Rob Watt X-X-Sender: rob@daemon.mistermishap.net To: freebsd-amd64@freebsd.org Message-ID: <20050630160225.D38285@daemon.mistermishap.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: fatal trap 12 in pagedaemon on dual-core opteron machine X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Jun 2005 20:03:03 -0000 I've been having stability problems in a new dual-core opteron machine. When it first crashed we were running a number of multicast applications that were listening to, recording, and rebroadcasting data. I have been able to re-simulate the data play upto the crash and can recreate the panic 1 out of every 5 or 6 replays. the hardware is: - 2 275 (2.2Ghz) dual-core processors - tyan 2881 k8sr motherboard (bios 2.05v) - 2 Gb DDR 400Mhz PC3200 Registered ECC memory - SEAGATE ST3146807LC scsi drive - intel dual-port gigabit ethernet card I have experienced the problem in both 5.4-RELEASE and STABLE (stable as-of today). I am running a custom SMP kernel. The most recent panic message was: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x88 fault code = supervisor read, page not present instruction pointer = 0x8:0xffffffff802af0da stack pointer = 0x10:0xffffffffb1c16a60 frame pointer = 0x10:0xffffff005b8cd000 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 63 (pagedaemon) trap number = 12 panic: page fault cpuid = 0 boot() called on cpu#0 Uptime: 1h3m10s Dumping 2047 MB 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256 272 288 304 320 336 352 368 384 400 416 432 448 464 480 496 512 528 544 560 576 592 608 624 640 656 672 688 704 720 736 752 768 784 800 816 832 848 864 880 896 912 928 944 960 976 992 1008 1024 1040 1056 1072 1088 1104 1120 1136 1152 1168 1184 1200 1216 1232 1248 1264 1280 1296 1312 1328 1344 1360 1376 1392 1408 1424 1440 1456 1472 1488 1504 1520 1536 1552 1568 1584 1600 1616 1632 1648 1664 1680 1696 1712 1728 1744 1760 1776 1792 1808 1824 1840 1856 1872 1888 1904 1920 1936 1952 1968 1984 2000 2016 2032 Every time it crashes the current process is pagedaemon, and the instruction pointer points to 'thread_fini'. kgdb shows: host1# kgdb /usr/obj/usr/src/sys/LOCAL/kernel.debug /usr/tmp/crash/vmcore.6 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] This GDB was configured as "amd64-marcel-freebsd". #0 doadump () at pcpu.h:167 167 pcpu.h: No such file or directory. in pcpu.h (kgdb) where #0 doadump () at pcpu.h:167 #1 0x0000000000000000 in ?? () #2 0xffffffff802a2bd7 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:410 #3 0xffffffff802a340f in panic (fmt=0xffffff007b78c500 "\uffff\022y{") at /usr/src/sys/kern/kern_shutdown.c:566 #4 0xffffffff80412f8a in trap_fatal (frame=0xffffff007b78c500, eva=18446742976269456104) at /usr/src/sys/amd64/amd64/trap.c:639 #5 0xffffffff804132af in trap_pfault (frame=0xffffffffb1c169b0, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:562 #6 0xffffffff80413553 in trap (frame= {tf_rdi = -1097427386128, tf_rsi = -1097440115456, tf_rdx = 99343, tf_rcx = 0, tf_r8 = 0, tf_r9 = 0, tf_rax = 99343, tf_rbx = 0, tf_rbp = -1097975672832, tf_r10 = 4503599627366400, tf_r11 = 3424, tf_r12 = 4, tf_r13 = 4, tf_r14 = -1098264600680, tf_r15 = -1097364252144, tf_trapno = 12, tf_addr = 136, tf_flags = -1098264600680, tf_err = 0, tf_rip = -2144669478, tf_cs = 8, tf_rflags = 66050, tf_rsp = -1312724368, tf_ss = 16}) at /usr/src/sys/amd64/amd64/trap.c:341 #7 0xffffffff80400c0b in calltrap () at /usr/src/sys/amd64/amd64/exception.S:171 #8 0xffffff007c3b00f0 in ?? () #9 0xffffff007b78c500 in ?? () #10 0x000000000001840f in ?? () #11 0x0000000000000000 in ?? () #12 0x0000000000000000 in ?? () #13 0x0000000000000000 in ?? () #14 0x000000000001840f in ?? () #15 0x0000000000000000 in ?? () #16 0xffffff005b8cd000 in ?? () #17 0x000ffffffffff000 in ?? () #18 0x0000000000000d60 in ?? () #19 0x0000000000000004 in ?? () #20 0x0000000000000004 in ?? () #21 0xffffff004a541f98 in ?? () #22 0xffffff007ffe5a10 in ?? () #23 0x000000000000000c in ?? () #24 0x0000000000000088 in ?? () #25 0xffffff004a541f98 in ?? () #26 0x0000000000000000 in ?? () #27 0xffffffff802af0da in thread_fini (mem=0x0, size=0) at /usr/src/sys/kern/kern_thread.c:271 #28 0x0000000000000010 in ?? () #29 0x0000000000000001 in ?? () #30 0xffffff007ffe5a00 in ?? () #31 0xffffff005b8cdf98 in ?? () #32 0xffffffff803f67ff in zone_drain (zone=0x8) at /usr/src/sys/vm/uma_core.c:749 #33 0xffffffff803f43b6 in zone_foreach (zfunc=0xffffffff803f6630 ) at /usr/src/sys/vm/uma_core.c:1494 #34 0xffffffff803f7fc9 in uma_reclaim () at /usr/src/sys/vm/uma_core.c:2623 #35 0xffffffff803f1dac in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:674 #36 0xffffffff802898cc in fork_exit (callout=0xffffffff803f17b0 , arg=0x0, frame=0xffffffffb1c16c50) at /usr/src/sys/kern/kern_fork.c:791 #37 0xffffffff80400e0e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:296 #38 0x0000000000000000 in ?? () #39 0x0000000000000000 in ?? () #40 0x0000000000000001 in ?? () #41 0x0000000000000000 in ?? () #42 0x0000000000000000 in ?? () #43 0x0000000000000000 in ?? () #44 0x0000000000000000 in ?? () #45 0x0000000000000000 in ?? () #46 0x0000000000000000 in ?? () #47 0x0000000000000000 in ?? () #48 0x0000000000000000 in ?? () #49 0x0000000000000000 in ?? () #50 0x0000000000000000 in ?? () #51 0x0000000000000000 in ?? () #52 0x0000000000000000 in ?? () #53 0x0000000000000000 in ?? () #54 0x0000000000000000 in ?? () #55 0x0000000000000000 in ?? () #56 0x0000000000000000 in ?? () #57 0x0000000000000000 in ?? () #58 0x0000000000000000 in ?? () #59 0x0000000000000000 in ?? () #60 0x0000000000000000 in ?? () #61 0x0000000000000000 in ?? () #62 0x0000000000000000 in ?? () #63 0x0000000000000000 in ?? () #64 0x0000000000000000 in ?? () #65 0x0000000000000000 in ?? () #66 0x0000000000000000 in ?? () #67 0x0000000000000000 in ?? () #68 0x0000000000000000 in ?? () #69 0x0000000000000000 in ?? () #70 0x00000000007ff000 in ?? () #71 0xffffffff8062fad4 in vm_page_max_wired () #72 0xffffffff8062f1c0 in vm_page_queue_free_mtx () #73 0x0000000000000001 in ?? () #74 0xffffff007b7912e8 in ?? () #75 0xffffff007b7d5000 in ?? () #76 0xffffffffb1c16a08 in ?? () #77 0xffffff007b78c500 in ?? () #78 0xffffffff802b777c in sched_switch (td=0x0, newtd=0x0, flags=1) at /usr/src/sys/kern/sched_4bsd.c:881 #79 0x0000000000000000 in ?? () #80 0x0000000000000000 in ?? () #81 0x0000000000000000 in ?? () #82 0x0000000000000000 in ?? () #83 0x0000000000000000 in ?? () #84 0x0000000000000000 in ?? () #85 0x0000000000000000 in ?? () #86 0x0000000000000000 in ?? () #87 0x0000000000000000 in ?? () #88 0x0000000000000000 in ?? () #89 0x0000000000000000 in ?? () #90 0x0000000000000000 in ?? () #91 0x0000000000000000 in ?? () #92 0x0000000000000000 in ?? () #93 0x0000000000000000 in ?? () #94 0x0000000000000000 in ?? () #95 0x0000000000000000 in ?? () #96 0x0000000000000000 in ?? () #97 0x0000000000000000 in ?? () #98 0x0000000000000000 in ?? () #99 0x0000000000000000 in ?? () #100 0x0000000000000000 in ?? () #101 0x0000000000000000 in ?? () #102 0x0000000000000000 in ?? () #103 0x0000000000000000 in ?? () #104 0x0000000000000000 in ?? () #105 0x0000000000000000 in ?? () #106 0x0000000000000000 in ?? () #107 0x0000000000000000 in ?? () #108 0x0000000000000000 in ?? () #109 0x0000000000000000 in ?? () #110 0x0000000000000000 in ?? () #111 0x0000000000000000 in ?? () #112 0x0000000000000000 in ?? () #113 0x0000000000000000 in ?? () #114 0x0000000000000000 in ?? () #115 0x0000000000000000 in ?? () #116 0x0000000000000000 in ?? () #117 0x0000000000000000 in ?? () #118 0x0000000000000000 in ?? () #119 0x0000000000000000 in ?? () #120 0x0000000000000000 in ?? () #121 0x0000000000000000 in ?? () #122 0x0000000000000000 in ?? () #123 0x0000000000000000 in ?? () #124 0x0000000000000000 in ?? () #125 0x0000000000000000 in ?? () #126 0x0000000000000000 in ?? () #127 0x0000000000000000 in ?? () #128 0x0000000000000000 in ?? () #129 0x0000000000000000 in ?? () #130 0x0000000000000000 in ?? () #131 0x0000000000000000 in ?? () #132 0x0000000000000000 in ?? () #133 0x0000000000000000 in ?? () #134 0x0000000000000000 in ?? () #135 0x0000000000000000 in ?? () #136 0x0000000000000000 in ?? () #137 0x0000000000000000 in ?? () #138 0x0000000000000000 in ?? () #139 0x0000000000000000 in ?? () #140 0x0000000000000000 in ?? () #141 0x0000000000000000 in ?? () #142 0x0000000000000000 in ?? () #143 0x0000000000000000 in ?? () #144 0x0000000000000000 in ?? () #145 0x0000000000000000 in ?? () #146 0x0000000000000000 in ?? () #147 0x0000000000000000 in ?? () #148 0x0000000000000000 in ?? () #149 0x0000000000000000 in ?? () #150 0x0000000000000000 in ?? () Cannot access memory at address 0xffffffffb1c17000 kernel config: machine amd64 cpu HAMMER ident CUSTOM makeoptions DEBUG=-g options SCHED_4BSD # 4BSD scheduler options INET # InterNETworking options INET6 # IPv6 communications protocols options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_ACL # Support for access control lists options UFS_DIRHASH # Improve performance on big directories options MD_ROOT # MD is a potential root device options NFSCLIENT # Network Filesystem Client options NFSSERVER # Network Filesystem Server options NFS_ROOT # NFS usable as /, requires NFSCLIENT options NTFS # NT File System options MSDOSFS # MSDOS Filesystem options CD9660 # ISO 9660 Filesystem options PROCFS # Process filesystem (requires PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options GEOM_GPT # GUID Partition Tables. options COMPAT_43 # Needed by COMPAT_LINUX32 options COMPAT_IA32 # Compatible with i386 binaries options COMPAT_FREEBSD4 # Compatible with FreeBSD4 options COMPAT_LINUX32 # Compatible with i386 linux binaries options SCSI_DELAY=15000 # Delay (in ms) before probing SCSI options KTRACE # ktrace(1) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options KBD_INSTALL_CDEV # install a CDEV entry in /dev options AHC_REG_PRETTY_PRINT # Print register bitfields in debug # output. Adds ~128k to driver. options AHD_REG_PRETTY_PRINT # Print register bitfields in debug options ADAPTIVE_GIANT # Giant mutex is adaptive. options SMP device atpic # 8259A compatability options LINPROCFS # Cannot be a module yet. device acpi device isa device pci device fdc device ata device atadisk # ATA disk drives device ataraid # ATA RAID drives device atapicd # ATAPI CDROM drives device atapifd # ATAPI floppy drives device atapist # ATAPI tape drives options ATA_STATIC_ID # Static device numbering device ahc # AHA2940 and onboard AIC7xxx devices device ahd # AHA39320/29320 and onboard AIC79xx devices device aic # Adaptec 15[012]x SCSI adapters, AIC-6[23]60. device scbus # SCSI bus (required for SCSI) device ch # SCSI media changers device da # Direct Access (disks) device sa # Sequential Access (tape etc) device cd # CD device pass # Passthrough device (direct SCSI access) device ses # SCSI Environmental Services (and SAF-TE) device atkbdc # AT keyboard controller device atkbd # AT keyboard device psm # PS/2 mouse device vga # VGA video card driver device splash # Splash screen and screen saver support device sc device sio # 8250, 16[45]50 based serial ports device ppc device ppbus # Parallel port bus (required) device lpt # Printer device ppi # Parallel port interface device device em # Intel PRO/1000 adapter Gigabit Ethernet Card device miibus # MII bus support device bge # Broadcom BCM570xx Gigabit Ethernet device loop # Network loopback device mem # Memory and kernel memory devices device io # I/O device device random # Entropy device device ether # Ethernet support device sl # Kernel SLIP device ppp # Kernel PPP device tun # Packet tunnel. device pty # Pseudo-ttys (telnet etc) device md # Memory "disks" device gif # IPv6 and IPv4 tunneling device faith # IPv6-to-IPv4 relaying (translation) device bpf # Berkeley packet filter device uhci # UHCI PCI->USB interface device ohci # OHCI PCI->USB interface device usb # USB Bus (required) device ugen # Generic device uhid # "Human Interface Devices" device ukbd # Keyboard device ulpt # Printer device umass # Disks/Mass storage - Requires scbus and da device ums # Mouse any thoughts? thanks. - Rob Watt