From owner-freebsd-stable@FreeBSD.ORG Thu Aug 21 09:10:57 2003 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5061A16A4BF; Thu, 21 Aug 2003 09:10:57 -0700 (PDT) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by mx1.FreeBSD.org (Postfix) with ESMTP id 070A843FEC; Thu, 21 Aug 2003 09:10:55 -0700 (PDT) (envelope-from mike@sentex.net) Received: from simian.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.12.9/8.12.8) with ESMTP id h7LGApFb038258; Thu, 21 Aug 2003 12:10:52 -0400 (EDT) (envelope-from mike@sentex.net) Message-Id: <5.2.0.9.0.20030821115110.07f28e08@209.112.4.2> X-Sender: mdtpop@209.112.4.2 (Unverified) X-Mailer: QUALCOMM Windows Eudora Version 5.2.0.9 Date: Thu, 21 Aug 2003 12:13:19 -0400 To: stable@freebsd.org From: Mike Tancsa In-Reply-To: <5.2.0.9.0.20030812161901.09b02558@209.112.4.2> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Virus-Scanned: By Sentex Communications (lava/20020517) Subject: Re: panic with today's stable (sort of resolved) PAE related ? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Aug 2003 16:10:57 -0000 OK, I cvsup'd back to August 7th (default date=2003.08.07.00.00.00), buildworld / kernel etc and the box has been stable for almost 24hrs now. Prior to that, it would be rebooting a few times a day. The panics would be like the one below. I also tried a stable up to the 19th, but still panics. Below are some more panics and some output tegge requested ------------------------------------------------- (kgdb) bt #0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487 #1 0xc01618fc in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:316 #2 0xc0161d49 in panic (fmt=0xc02db8cc "%s") at /usr/src/sys/kern/kern_shutdown.c:595 #3 0xc028cb94 in trap_fatal (frame=0xe9cade3c, eva=16) at /usr/src/sys/i386/i386/trap.c:974 #4 0xc028c829 in trap_pfault (frame=0xe9cade3c, usermode=0, eva=16) at /usr/src/sys/i386/i386/trap.c:867 #5 0xc028c38f in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = -372637680, tf_edi = 8, tf_esi = 1143, tf_ebp = -372580732, tf_isp = -372580760, tf_ebx = -366795420, tf_edx = -4, tf_ecx = 93583622, tf_eax = 524281, tf_trapno = 12, tf_err = 0, tf_eip = -1071252840, tf_cs = 8, tf_eflags = 66182, tf_esp = -372353324, tf_ss = 1143}) at /usr/src/sys/i386/i386/trap.c:466 #6 0xc025fa98 in vm_page_lookup (object=0xea232564, pindex=1143) at /usr/src/sys/vm/vm_page.c:515 #7 0xc0257846 in vm_fault (map=0xea15b140, vaddr=148422656, fault_type=1 '\001', fault_flags=0) at /usr/src/sys/vm/vm_fault.c:426 #8 0xc028c7bb in trap_pfault (frame=0xe9cadfa8, usermode=1, eva=148424256) at /usr/src/sys/i386/i386/trap.c:847 #9 0xc028c23b in trap (frame={tf_fs = -1078001617, tf_es = 142606383, tf_ds = -1078001617, tf_edi = 135422592, tf_esi = 141122551, tf_ebp = -1077937232, tf_isp = -372580396, tf_ebx = 672134608, tf_edx = 148424256, tf_ecx = 758396440, tf_eax = 148306460, tf_trapno = 12, tf_err = 4, tf_eip = 671848534, tf_cs = 31, tf_eflags = 66050, tf_esp = -1077937256, tf_ss = 47}) at /usr/src/sys/i386/i386/trap.c:377 #10 0x280b9856 in ?? () #11 0x280e61a6 in ?? () #12 0x280e6674 in ?? () #13 0x280e6568 in ?? () #14 0x280b780c in ?? () #15 0x2807ef5d in ?? () #16 0x280ea9ab in ?? () #17 0x8048e8b in ?? () #18 0x8048d5a in ?? () (kgdb) up 6 #6 0xc025fa98 in vm_page_lookup (object=0xea232564, pindex=1143) at /usr/src/sys/vm/vm_page.c:515 515 for (m = *bucket; m != NULL; m = m->hnext) { (kgdb) print ((struct vm_page *) 0xc1331138)->object $1 = 0xe991f284 (kgdb) print ((struct vm_page *) 0xc1331138)->object[0] $2 = {object_list = {tqe_next = 0xe8eb7564, tqe_prev = 0xe9586e04}, shadow_head = {lh_first = 0x0}, shadow_list = { le_next = 0xe7fccbdc, le_prev = 0xe94bb234}, memq = {tqh_first = 0xc0de94a0, tqh_last = 0xc1179770}, generation = 192, size = 74, ref_count = 1, shadow_count = 0, hash_rand = -441882534, type = 1 '\001', flags = 8576, pg_color = 22, paging_in_progress = 0, resident_page_count = 51, backing_object = 0xe7fa233c, backing_object_offset = 0, pager_object_list = { tqe_next = 0xe825f000, tqe_prev = 0xe98818e8}, handle = 0x0, un_pager = {vnp = {vnp_size = 3}, devp = {devp_pglist = { tqh_first = 0x3, tqh_last = 0x0}}, swp = {swp_bcount = 3}}} (kgdb) print ((struct vm_page *) 0xc1331138)->pindex $3 = 61 (kgdb) print vm_page_hash_mask $4 = 524287 (kgdb) print vm_page_buckets $5 = (struct vm_page **) 0xc041d000 (kgdb) print vm_page_bucket_count $6 = 524288 (kgdb) print vm_page_buckets + vm_page_bucket_count $7 = (struct vm_page **) 0xc061d000 (kgdb) print vm_page_array $8 = 0xc061d000 (kgdb) print vm_page_array_size $9 = 261111 (kgdb) print vm_page_array + vm_page_array_size $10 = (struct vm_page *) 0xc150dde4 (kgdb) print m $11 = 0x0 (kgdb) and (kgdb) bt #0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487 #1 0xc01618fc in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:316 #2 0xc0161d49 in panic (fmt=0xc02db8cc "%s") at /usr/src/sys/kern/kern_shutdown.c:595 #3 0xc028cb94 in trap_fatal (frame=0xe85c0e3c, eva=12582934) at /usr/src/sys/i386/i386/trap.c:974 #4 0xc028c829 in trap_pfault (frame=0xe85c0e3c, usermode=0, eva=12582934) at /usr/src/sys/i386/i386/trap.c:867 #5 0xc028c38f in trap (frame={tf_fs = 16, tf_es = -390856688, tf_ds = -390856688, tf_edi = 134873088, tf_esi = 84, tf_ebp = -396620156, tf_isp = -396620184, tf_ebx = -400233508, tf_edx = 12582914, tf_ecx = 20568904, tf_eax = 524283, tf_trapno = 12, tf_err = 0, tf_eip = -1071252840, tf_cs = 8, tf_eflags = 66050, tf_esp = 0, tf_ss = -391880320}) at /usr/src/sys/i386/i386/trap.c:466 #6 0xc025fa98 in vm_page_lookup (object=0xe824ebdc, pindex=84) at /usr/src/sys/vm/vm_page.c:515 #7 0xc02575ea in vm_fault (map=0xe8a46180, vaddr=134873088, fault_type=1 '\001', fault_flags=0) at /usr/src/sys/vm/vm_fault.c:292 #8 0xc028c7bb in trap_pfault (frame=0xe85c0fa8, usermode=1, eva=134876672) at /usr/src/sys/i386/i386/trap.c:847 #9 0xc028c23b in trap (frame={tf_fs = 47, tf_es = 135004207, tf_ds = -1078001617, tf_edi = -1077936692, tf_esi = 672112688, tf_ebp = -1077936992, tf_isp = -396619820, tf_ebx = 672134608, tf_edx = 134621060, tf_ecx = 672140532, tf_eax = 134876672, tf_trapno = 12, tf_err = 4, tf_eip = 671837895, tf_cs = 31, tf_eflags = 66050, tf_esp = -1077937080, tf_ss = 47}) at /usr/src/sys/i386/i386/trap.c:377 #10 0x280b6ec7 in ?? () #11 0x2807ef5d in ?? () #12 0x280ea9ab in ?? () #13 0x8048e8b in ?? () #14 0x8048d5a in ?? () (kgdb) up 6 #6 0xc025fa98 in vm_page_lookup (object=0xe824ebdc, pindex=84) at /usr/src/sys/vm/vm_page.c:515 515 for (m = *bucket; m != NULL; m = m->hnext) { (kgdb) print ((struct vm_page *) 0xc1331138)->object $1 = 0xc0318d6c (kgdb) print ((struct vm_page *) 0xc1331138)->object[0] $2 = {object_list = {tqe_next = 0xc0318e24, tqe_prev = 0xc0317f0c}, shadow_head = {lh_first = 0x0}, shadow_list = {le_next = 0x0, le_prev = 0xc0318c60}, memq = {tqh_first = 0xc0934a54, tqh_last = 0xc08f37a4}, generation = 68154, size = 68091, ref_count = 1, shadow_count = 0, hash_rand = -170022, type = 1 '\001', flags = 8576, pg_color = 2, paging_in_progress = 0, resident_page_count = 68026, backing_object = 0x0, backing_object_offset = 0, pager_object_list = {tqe_next = 0xe7dfd8fc, tqe_prev = 0xc0318be8}, handle = 0x0, un_pager = {vnp = {vnp_size = 5}, devp = {devp_pglist = {tqh_first = 0x5, tqh_last = 0x0}}, swp = {swp_bcount = 5}}} (kgdb) print ((struct vm_page *) 0xc1331138)->pindex $3 = 36689 (kgdb) print vm_page_hash_mask $4 = 524287 (kgdb) print vm_page_buckets $5 = (struct vm_page **) 0xc041d000 (kgdb) print vm_page_bucket_count $6 = 524288 (kgdb) print vm_page_buckets + vm_page_bucket_count $7 = (struct vm_page **) 0xc061d000 (kgdb) print vm_page_array $8 = 0xc061d000 (kgdb) print vm_page_array_size $9 = 261111 (kgdb) print vm_page_array + vm_page_array_size $10 = (struct vm_page *) 0xc150dde4 (kgdb) print m $11 = 0x0 (kgdb) ---Mike ns4# cat /boot/loader.conf # -- sysinstall generated deltas -- # userconfig_script_load="YES" hw.ata.wc="1" hw.physmem="1020m" ns4# ns4# gdb -k /kernel.debug vmcore.3 GNU gdb 4.18 (FreeBSD) Copyright 1998 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-unknown-freebsd"...Deprecated bfd_read called at /usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line 2627 in elfstab_build_psymtabs Deprecated bfd_read called at /usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line 933 in fill_symbuf IdlePTD at phsyical address 0x003ad000 initial pcb at physical address 0x0030cb40 panicstr: vm_page_remove(): page not found in hash panic messages: --- panic: vm_page_remove(): page not found in hash syncing disks... 14 stray irq 7 2 2 2 2 2 2 2 7 done Uptime: 14h21m23s twe0: failed to delete unit 0 #0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487 487 if (dumping++) { (kgdb) bt #0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487 #1 0xc01618fc in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:316 #2 0xc0161d49 in panic (fmt=0xc02d2f60 "vm_page_remove(): page not found in hash") at /usr/src/sys/kern/kern_shutdown.c:595 #3 0xc025fa13 in vm_page_remove (m=0xc06b0378) at /usr/src/sys/vm/vm_page.c:461 #4 0xc02600ac in vm_page_free_toq (m=0xc06b0378) at /usr/src/sys/vm/vm_page.c:1104 #5 0xc025de22 in vm_object_terminate (object=0xe8712d4c) at /usr/src/sys/vm/vm_page.h:514 #6 0xc025dce0 in vm_object_deallocate (object=0xe8712d4c) at /usr/src/sys/vm/vm_object.c:399 #7 0xc025af21 in vm_map_entry_delete (map=0xe85ba100, entry=0xe8605780) at /usr/src/sys/vm/vm_map.c:2054 #8 0xc025b0d2 in vm_map_delete (map=0xe85ba100, start=0, end=3217031168) at /usr/src/sys/vm/vm_map.c:2174 #9 0xc025b161 in vm_map_remove (map=0xe85ba100, start=0, end=3217031168) at /usr/src/sys/vm/vm_map.c:2199 #10 0xc0159c3d in exit1 (p=0xe84fa8a0, rv=11) at /usr/src/sys/kern/kern_exit.c:226 #11 0xc01638bb in sigexit (p=0xe84fa8a0, sig=11) at /usr/src/sys/kern/kern_sig.c:1504 #12 0xc016362c in postsig (sig=11) at /usr/src/sys/kern/kern_sig.c:1407 #13 0xc028c5db in trap (frame={tf_fs = -1078001617, tf_es = -1078001617, tf_ds = -1078001617, tf_edi = 135456085, tf_esi = 135455758, tf_ebp = -1077948736, tf_isp = -397344812, tf_ebx = 135455744, tf_edx = 135455744, tf_ecx = 0, tf_eax = 0, tf_trapno = 12, tf_err = 0, tf_eip = 134919195, tf_cs = 31, tf_eflags = 66195, tf_esp = -1077948744, tf_ss = 47}) at /usr/src/sys/i386/i386/trap.c:174 #14 0x80ab41b in ?? () Cannot access memory at address 0xbfbfcec0. (kgdb) up 5 #5 0xc025de22 in vm_object_terminate (object=0xe8712d4c) at /usr/src/sys/vm/vm_page.h:514 514 vm_page_free_toq(m); (kgdb) print ((struct vm_page *) 0xc1331138)->object $1 = 0x0 (kgdb) print ((struct vm_page *) 0xc1331138)->object[0] Cannot access memory at address 0x0. (kgdb) print ((struct vm_page *) 0xc1331138)->pindex $2 = 74 (kgdb) print vm_page_hash_mask $3 = 524287 (kgdb) print vm_page_buckets $4 = (struct vm_page **) 0xc041d000 (kgdb) print vm_page_bucket_count $5 = 524288 (kgdb) print vm_page_buckets + vm_page_bucket_count $6 = (struct vm_page **) 0xc061d000 (kgdb) print vm_page_array $7 = 0xc061d000 (kgdb) print vm_page_array_size $8 = 261111 (kgdb) print vm_page_array + vm_page_array_size $9 = (struct vm_page *) 0xc150dde4 (kgdb) print m $10 = 0x0 (kgdb) dmesg now looks like ns4# dmesg Copyright (c) 1992-2003 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 4.8-STABLE #0: Tue Aug 12 11:15:45 EDT 2003 mdtancsa@ns4.recycle.net:/usr/obj/usr/src/sys/ns4 Timecounter "i8254" frequency 1193182 Hz Timecounter "TSC" frequency 2398857928 Hz CPU: Intel(R) Pentium(R) 4 CPU 2.40GHz (2398.86-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf27 Stepping = 7 Features=0xbfebfbff real memory = 1069547520 (1044480K bytes) config> q avail memory = 1037451264 (1013136K bytes) Preloaded elf kernel "kernel" at 0xc038e000. Preloaded userconfig_script "/boot/kernel.conf" at 0xc038e09c. Pentium Pro MTRR support enabled md0: Malloc disk Using $PIR table, 7 entries at 0xc00fded0 npx0: on motherboard npx0: INT 16 interface pcib0: on motherboard pci0: on pcib0 agp0: mem 0xd8d00000-0xd8d7ffff,0xd0000000-0xd7ffffff irq 12 at device 2.0 on pci0 agp0: detected 892k stolen memory agp0: aperture size is 128M pcib1: at device 30.0 on pci0 pci1: on pcib1 fxp0: port 0xc000-0xc01f mem 0xd8800000-0xd88fffff,0xd8c00000-0xd8c00fff irq 12 at device 0.0 on pci1 fxp0: Ethernet address 00:a0:c9:e7:a6:e6 inphy0: on miibus0 inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp1: port 0xc400-0xc43f mem 0xd8a00000-0xd8afffff,0xd8b00000-0xd8b00fff irq 10 at device 1.0 on pci1 fxp1: Ethernet address 00:02:b3:07:fd:8d inphy1: on miibus1 inphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto twe0: <3ware Storage Controller> port 0xc800-0xc80f irq 15 at device 5.0 on pci1 twe0: 2 ports, Firmware FE6X 1.02.00.029, BIOS BEXX 1.07.00.009 fxp2: port 0xcc00-0xcc3f mem 0xd8b01000-0xd8b01fff irq 11 at device 8.0 on pci1 fxp2: Ethernet address 00:01:80:38:46:36 inphy2: on miibus2 inphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto isab0: at device 31.0 on pci0 isa0: on isab0 pci0: (vendor=0x8086, dev=0x24c3) at 31.3 irq 10 orm0: