From owner-freebsd-questions@FreeBSD.ORG Sun Mar 16 03:35:22 2014 Return-Path: Delivered-To: questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 41A469E6 for ; Sun, 16 Mar 2014 03:35:22 +0000 (UTC) Received: from mail-yk0-x230.google.com (mail-yk0-x230.google.com [IPv6:2607:f8b0:4002:c07::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 01EF287A for ; Sun, 16 Mar 2014 03:35:21 +0000 (UTC) Received: by mail-yk0-f176.google.com with SMTP id 19so11358224ykq.7 for ; Sat, 15 Mar 2014 20:35:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=ZAKlaW84hK/CJjo8gN/tK32vrO5JSfEVYs1WwrNirTw=; b=gwUFREQ7SUtcRg5mS4kCoBQnOkGg6wXn4q30XIbeys8NACwb9tvRKX/YJLuRhNA+5U rObocLVpaNGUKHkbFXSR1JQceVhvYU/rTA8Sb2odQ3OlN2TrgukQ6l9FZWp/lS774LwB O571MhIME/eCRmUYgRh/YnwSl7209l7ihVO1qMPKhYXpsaroD6WTgEQg31Jy3N6gXS7m daBr6qj6pgxzIcsqLbBzDDi40Fm/CafcHyxQp5lhkhbGqF8gZbcc9LSWLUFx//FZUGZ3 oPB1bGYrodtwd4WRbNnKC/oS7Q6HeZwHGST4sSil+2Ktw5L5aqN8sninQk1XYOq6RAMR bbBQ== MIME-Version: 1.0 X-Received: by 10.236.86.226 with SMTP id w62mr2731248yhe.94.1394940920803; Sat, 15 Mar 2014 20:35:20 -0700 (PDT) Received: by 10.170.135.140 with HTTP; Sat, 15 Mar 2014 20:35:20 -0700 (PDT) In-Reply-To: References: Date: Sun, 16 Mar 2014 10:35:20 +0700 Message-ID: Subject: Re: Frequent Page Faults From: Olivier Nicole To: Doug Hardie Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: "questions@freebsd.org FreeBSD" X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Mar 2014 03:35:22 -0000 Hi, Wild guess, but if you could have someone checking the hardware for you: dirty memory connector, dust clogged CPU fan... As you said, the system has been running for over 4 years, this could very much be hardware. Olivier On Sun, Mar 16, 2014 at 7:45 AM, Doug Hardie wrote: > I have a system running: > > FreeBSD zoon.lafn.org 7.2-RELEASE-p5 FreeBSD 7.2-RELEASE-p5 #3: Thu Aug 1= 9 20:09:11 PDT 2010 > > This morning it started crashing frequently. The system had no issues pr= ior to today. Sometimes it auto reboots, others it just hangs. The consol= e messages are always very similar. I do have core dumps for 3 of them. A= t first it appeared that the problem was being caused by an attack on port = 110. While the attack was in process, the system would stay up for only a = few minutes. After discovering and blocking the attack, the system remaine= d up a couple hours but crashed while I was watching it. I am beginning to= suspect a HW issue that was worsened by the load of the attack, but not ca= used directly by the attack. > > I have been in the process of building a new set of disks for this system= using 9.2, but thats not complete yet. In addition it will take a couple = days to get the disks on site. Its a remote facility. I found several ref= erences to issues with this problem and 7.2 that have apparently been fixed= . However, I have been running 7.2 since it first came out on this system = without any similar issues. Actually, I don't recall any issues at all wit= h 7.2. The source for this system no longer exists. My development system= has been upgraded to 9.2. Its a modified kernel with some of the older pr= ocessors commented out and includes QUOTA and ALTQ. I don't recall any oth= er changes. > > I can hurry up the setup of the 9.2 system, but it would be at least unti= l Wed before it could be installed at tried. If that would correct the pro= blem that would be great. However, I have a concern that there is also a H= W issue here and am not sure how to identify such. My review of the dumps = shows that acpi is always involved. Don't know for sure what that implies = though. I don't believe the os has degraded and it has not been touched si= nce 2010. That pretty much leaves me with a HW issue. Any ideas where the= problem is will be appreciated. > > Here are a couple of the dumps. The first one was while the attack was i= n progress. The second after it was terminated. > > -------------------------------------------------------------- > > zoon# kgdb /boot/kernel/kernel vmcore.0 > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you = are > welcome to change it and/or distribute copies of it under certain conditi= ons. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for detail= s. > This GDB was configured as "i386-marcel-freebsd"... > > Unread portion of the kernel message buffer: > 3) > trap number =3D 12 > panic: page fault > cpuid =3D 3 > Uptime: 3m48s > Physical memory: 1993 MB > Dumping 187 MB: 172 156 140 124 > > Fatal trap 12: page fault while in kernel mode > cpuid =3D 1; apic id =3D 01 > fault virtual address =3D 0x4 > fault code =3D supervisor write, page not present > instruction pointer =3D 0x20:0xc0c72c00 > stack pointer =3D 0x28:0xe566faf4 > frame pointer =3D 0x28:0xe566fb14 > code segment =3D base 0x0, limit 0xfffff, type 0x1b > =3D DPL 0, pres 1, def32 1, gran 1 > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > current process =3D 4 (g_down) > trap number =3D 12 > 108 92 76 60 44 28 12 > > Reading symbols from /boot/kernel/fdescfs.ko...Reading symbols from /boot= /kernel/fdescfs.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/fdescfs.ko > Reading symbols from /boot/kernel/pflog.ko...Reading symbols from /boot/k= ernel/pflog.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/pflog.ko > Reading symbols from /boot/kernel/pf.ko...Reading symbols from /boot/kern= el/pf.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/pf.ko > Reading symbols from /boot/kernel/acpi.ko...Reading symbols from /boot/ke= rnel/acpi.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/acpi.ko > #0 doadump () at pcpu.h:196 > 196 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) where > #0 doadump () at pcpu.h:196 > #1 0xc07a5e27 in boot (howto=3D260) at /usr2/src/sys/kern/kern_shutdown.= c:418 > #2 0xc07a60f9 in panic (fmt=3DVariable "fmt" is not available. > ) at /usr2/src/sys/kern/kern_shutdown.c:574 > #3 0xc0aa792c in trap_fatal (frame=3D0xc4ff8c48, eva=3D1361334589) > at /usr2/src/sys/i386/i386/trap.c:939 > #4 0xc0aa7b90 in trap_pfault (frame=3D0xc4ff8c48, usermode=3D0, eva=3D13= 61334589) > at /usr2/src/sys/i386/i386/trap.c:852 > #5 0xc0aa8512 in trap (frame=3D0xc4ff8c48) at /usr2/src/sys/i386/i386/tr= ap.c:530 > #6 0xc0a8d62b in calltrap () at /usr2/src/sys/i386/i386/exception.s:159 > #7 0xc0e21715 in acpi_cpu_c1 () > at /usr2/src/sys/modules/acpi/acpi/../../../i386/acpica/acpi_machdep.= c:550 > #8 0xc0e1a594 in acpi_cpu_idle () > at /usr2/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_cpu.c:943 > #9 0xc0a97f78 in cpu_idle () at /usr2/src/sys/i386/i386/machdep.c:1183 > #10 0xc07c7904 in sched_idletd (dummy=3D0x0) > at /usr2/src/sys/kern/sched_ule.c:2681 > #11 0xc07808d9 in fork_exit (callout=3D0xc07c7640 , arg=3D0= x0, > frame=3D0xc4ff8d38) at /usr2/src/sys/kern/kern_fork.c:810 > #12 0xc0a8d6a0 in fork_trampoline () at /usr2/src/sys/i386/i386/exception= .s:264 > > > > > -------------------------------------------------------------- > > > zoon# kgdb /boot/kernel/kernel vmcore.2 > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you = are > welcome to change it and/or distribute copies of it under certain conditi= ons. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for detail= s. > This GDB was configured as "i386-marcel-freebsd"... > > Unread portion of the kernel message buffer: > > Fatal double fault: > eip =3D 0xc0e21715 > esp =3D 0xc4ff8d80 > ebp =3D 0xc4ff8c88 > cpuid =3D 3; apic id =3D 03 > panic: double fault > cpuid =3D 3 > Uptime: 2h19m49s > Physical memory: 1993 MB > Dumping 187 MB: 172 156 140 124 108 92 76 60 44 28 12 > > Reading symbols from /boot/kernel/fdescfs.ko...Reading symbols from /boot= /kernel/fdescfs.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/fdescfs.ko > Reading symbols from /boot/kernel/pflog.ko...Reading symbols from /boot/k= ernel/pflog.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/pflog.ko > Reading symbols from /boot/kernel/pf.ko...Reading symbols from /boot/kern= el/pf.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/pf.ko > Reading symbols from /boot/kernel/acpi.ko...Reading symbols from /boot/ke= rnel/acpi.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/acpi.ko > #0 doadump () at pcpu.h:196 > 196 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) up > #1 0xc07a5e27 in boot (howto=3D260) at /usr2/src/sys/kern/kern_shutdown.= c:418 > 418 /usr2/src/sys/kern/kern_shutdown.c: No such file or directory. > in /usr2/src/sys/kern/kern_shutdown.c > (kgdb) down > #0 doadump () at pcpu.h:196 > 196 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) where > #0 doadump () at pcpu.h:196 > #1 0xc07a5e27 in boot (howto=3D260) at /usr2/src/sys/kern/kern_shutdown.= c:418 > #2 0xc07a60f9 in panic (fmt=3DVariable "fmt" is not available. > ) at /usr2/src/sys/kern/kern_shutdown.c:574 > #3 0xc0aa763b in dblfault_handler () at /usr2/src/sys/i386/i386/trap.c:9= 72 > #4 0xc0e21715 in acpi_cpu_c1 () > at /usr2/src/sys/modules/acpi/acpi/../../../i386/acpica/acpi_machdep.= c:550 > #5 0xc0e1a594 in acpi_cpu_idle () > at /usr2/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_cpu.c:943 > #6 0xc0a97f78 in cpu_idle () at /usr2/src/sys/i386/i386/machdep.c:1183 > #7 0xc07c7904 in sched_idletd (dummy=3D0x0) > at /usr2/src/sys/kern/sched_ule.c:2681 > #8 0xc07808d9 in fork_exit (callout=3D0xc07c7640 , arg=3D0= x0, > frame=3D0xc4ff8d38) at /usr2/src/sys/kern/kern_fork.c:810 > #9 0xc0a8d6a0 in fork_trampoline () at /usr2/src/sys/i386/i386/exception= .s:264 > (kgdb) > > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.o= rg"