Date: Sun, 17 May 2009 07:06:57 -0400 From: Doug Lee <dgl@dlee.org> To: freebsd-questions@freebsd.org Subject: 4.11 panic every 23 hours 55 minutes or so Message-ID: <20090517110657.GC2706@mini.local>
next in thread | raw e-mail | index | archive | help
One of the weirder things I've seen in a while here...
OS: FreeBSD 4.11 (yeah I know, old, but generally stable)
CPU: Intel(R) Pentium(R) 4 CPU 2.00GHz
real memory = 536608768 (524032K bytes)
Hds: IDE
Problem: Ever since a suspitious power outage (I say suspitious
because we think a surge was also involved), this box has been
exhibiting kernel panics about every 23 hours 55 minutes, give or
take about 4 minutes either way. Obviously hardware is suspect,
and hopefully in line for upgrade; but as FreeBSD has always proven
so stable for me, I'm curious what on earth could cause this sort
of regular panic?
It's not time of day; if I reboot at 2:00 AM, 3:55 PM, or any other
time, it's 23:55 or so later I get a panic, whenever that may be.
I think this rules out cron jobs, external attacks, and load-based
issues.
I'll provide the latest two panic logs below, followed by a few
rounds of output from `top -Sd1000', which is `top' with system
processes shown and dumb-terminal output, logged via ssh onto another
system so I could see the last report before the panic. I have a
longer `top' log but I'll send that only on request to avoid an
even huger post. Curiously, this is one of the few times the uptime
went past 24 hours, and also shows the current process as something
other than Idle. I wonder if running `top' actually changed the
results.
Any info would be most welcome. Please Cc me on responses.
----- Panic from yesterday (double panic; this is typical) -----
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0xe4f5eb02
fault code = supervisor read, page not present
instruction pointer = 0x8:0xc02af097
stack pointer = 0x10:0xc04be8d4
frame pointer = 0x10:0xc04be8d8
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = Idle
interrupt mask =
trap number = 12
panic: page fault
syncing disks...
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0x30
fault code = supervisor read, page not present
instruction pointer = 0x8:0xc0377070
stack pointer = 0x10:0xc04be6f4
frame pointer = 0x10:0xc04be6fc
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = Idle
interrupt mask = bio
trap number = 12
panic: page fault
Uptime: 23h59m5s
Automatic reboot in 15 seconds - press a key on the console to abort
----- Panic from today (just one this time, NOT typical) -----
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0xe694f564
fault code = supervisor read, page not present
instruction pointer = 0x8:0xc02a7b73
stack pointer = 0x10:0xdb394da4
frame pointer = 0x10:0xdb394e28
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 505 (nmbd)
interrupt mask = none
trap number = 12
panic: page fault
syncing disks... 1
done
Uptime: 1d0h1m36s
Automatic reboot in 15 seconds - press a key on the console to abort
Rebooting...
----- top -Sd10000 from today (stopping at the above panic) -----
last pid: 9015; load averages: 0.02, 0.02, 0.00 up 1+00:01:56 06:37:45
76 processes: 1 running, 74 sleeping, 1 zombie
CPU states: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idle
Mem: 95M Active, 264M Inact, 63M Wired, 21M Cache, 60M Buf, 55M Free
Swap: 250M Total, 250M Free
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
491 mysql 2 0 44788K 19244K poll 0:57 0.00% 0.00% mysqld
9 root 18 0 0K 0K syncer 0:16 0.00% 0.00% syncer
330 ssbdev 2 0 9744K 8188K poll 0:12 0.00% 0.00% python
248 root 2 0 1336K 868K select 0:04 0.00% 0.00% ntpd
224 root 2 0 464K 252K select 0:03 0.00% 0.00% natd
416 root 2 0 3188K 1948K select 0:03 0.00% 0.00% sendmail
6408 root 10 0 2868K 2292K nanslp 0:03 0.00% 0.00% perl5.8.8
308 root 2 0 9884K 5732K select 0:02 0.00% 0.00% httpd
6515 root 10 0 16548K 15872K nanslp 0:02 0.00% 0.00% perl5.8.8
505 root 2 0 5364K 1796K select 0:02 0.00% 0.00% nmbd
6738 root 10 0 16548K 15872K nanslp 0:01 0.00% 0.00% perl5.8.8
650 dlee 2 0 5336K 1836K select 0:01 0.00% 0.00% sshd
7778 root 10 0 16532K 15868K nanslp 0:01 0.00% 0.00% perl5.8.8
8307 root 10 0 16532K 15868K nanslp 0:01 0.00% 0.00% perl5.8.8
245 bind 2 0 2484K 1856K select 0:01 0.00% 0.00% named
241 root 2 0 1000K 668K select 0:01 0.00% 0.00% syslogd
8236 root 10 0 16516K 15856K nanslp 0:01 0.00% 0.00% perl5.8.8
10 root -2 0 0K 0K vlruwt 0:01 0.00% 0.00% vnlru
last pid: 9015; load averages: 0.02, 0.02, 0.00 up 1+00:01:58 06:37:47
76 processes: 1 running, 74 sleeping, 1 zombie
CPU states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
Mem: 95M Active, 264M Inact, 63M Wired, 21M Cache, 60M Buf, 55M Free
Swap: 250M Total, 250M Free
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
491 mysql 2 0 44788K 19244K poll 0:57 0.00% 0.00% mysqld
9 root 18 0 0K 0K syncer 0:16 0.00% 0.00% syncer
330 ssbdev 2 0 9744K 8188K poll 0:12 0.00% 0.00% python
248 root 2 0 1336K 868K select 0:04 0.00% 0.00% ntpd
224 root 2 0 464K 252K select 0:03 0.00% 0.00% natd
416 root 2 0 3188K 1948K select 0:03 0.00% 0.00% sendmail
6408 root 10 0 2868K 2292K nanslp 0:03 0.00% 0.00% perl5.8.8
308 root 2 0 9884K 5732K select 0:02 0.00% 0.00% httpd
6515 root 10 0 16548K 15872K nanslp 0:02 0.00% 0.00% perl5.8.8
505 root 2 0 5364K 1804K select 0:02 0.00% 0.00% nmbd
6738 root 10 0 16548K 15872K nanslp 0:01 0.00% 0.00% perl5.8.8
650 dlee 2 0 5336K 1836K select 0:01 0.00% 0.00% sshd
7778 root 10 0 16532K 15868K nanslp 0:01 0.00% 0.00% perl5.8.8
8307 root 10 0 16532K 15868K nanslp 0:01 0.00% 0.00% perl5.8.8
245 bind 2 0 2484K 1856K select 0:01 0.00% 0.00% named
241 root 2 0 1000K 668K select 0:01 0.00% 0.00% syslogd
8236 root 10 0 16516K 15856K nanslp 0:01 0.00% 0.00% perl5.8.8
10 root -2 0 0K 0K vlruwt 0:01 0.00% 0.00% vnlru
last pid: 9015; load averages: 0.02, 0.02, 0.00 up 1+00:02:00 06:37:49
76 processes: 1 running, 74 sleeping, 1 zombie
CPU states: 0.4% user, 0.0% nice, 0.4% system, 0.4% interrupt, 98.8% idle
Mem: 95M Active, 264M Inact, 63M Wired, 21M Cache, 60M Buf, 55M Free
Swap: 250M Total, 250M Free
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
491 mysql 2 0 44788K 19244K poll 0:57 0.00% 0.00% mysqld
9 root 18 0 0K 0K syncer 0:16 0.00% 0.00% syncer
330 ssbdev 2 0 9744K 8188K poll 0:12 0.00% 0.00% python
248 root 2 0 1336K 868K select 0:04 0.00% 0.00% ntpd
224 root 2 0 464K 252K select 0:03 0.00% 0.00% natd
416 root 2 0 3188K 1948K select 0:03 0.00% 0.00% sendmail
6408 root 10 0 2868K 2292K nanslp 0:03 0.00% 0.00% perl5.8.8
308 root 2 0 9884K 5732K select 0:02 0.00% 0.00% httpd
6515 root 10 0 16548K 15872K nanslp 0:02 0.00% 0.00% perl5.8.8
505 root 2 0 5364K 1804K select 0:02 0.00% 0.00% nmbd
6738 root 10 0 16548K 15872K nanslp 0:01 0.00% 0.00% perl5.8.8
650 dlee 2 0 5336K 1836K select 0:01 0.00% 0.00% sshd
7778 root 10 0 16532K 15868K nanslp 0:01 0.00% 0.00% perl5.8.8
8307 root 10 0 16532K 15868K nanslp 0:01 0.00% 0.00% perl5.8.8
245 bind 2 0 2484K 1856K select 0:01 0.00% 0.00% named
241 root 2 0 1000K 668K select 0:01 0.00% 0.00% syslogd
8236 root 10 0 16516K 15856K nanslp 0:01 0.00% 0.00% perl5.8.8
10 root -2 0 0K 0K vlruwt 0:01 0.00% 0.00% vnlru
last pid: 9015; load averages: 0.02, 0.02, 0.00 up 1+00:02:02 06:37:51
76 processes: 1 running, 74 sleeping, 1 zombie
CPU states: 0.4% user, 0.0% nice, 0.0% system, 0.0% interrupt, 99.6% idle
Mem: 95M Active, 264M Inact, 63M Wired, 21M Cache, 60M Buf, 55M Free
Swap: 250M Total, 250M Free
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
491 mysql 2 0 44788K 19244K poll 0:57 0.00% 0.00% mysqld
9 root 18 0 0K 0K syncer 0:16 0.00% 0.00% syncer
330 ssbdev 2 0 9744K 8188K poll 0:12 0.00% 0.00% python
248 root 2 0 1336K 868K select 0:04 0.00% 0.00% ntpd
224 root 2 0 464K 252K select 0:03 0.00% 0.00% natd
416 root 2 0 3188K 1948K select 0:03 0.00% 0.00% sendmail
6408 root 10 0 2868K 2292K nanslp 0:03 0.00% 0.00% perl5.8.8
308 root 2 0 9884K 5732K select 0:02 0.00% 0.00% httpd
6515 root 10 0 16548K 15872K nanslp 0:02 0.00% 0.00% perl5.8.8
505 root 2 0 5364K 1804K select 0:02 0.00% 0.00% nmbd
6738 root 10 0 16548K 15872K nanslp 0:01 0.00% 0.00% perl5.8.8
650 dlee 2 0 5336K 1836K select 0:01 0.00% 0.00% sshd
7778 root 10 0 16532K 15868K nanslp 0:01 0.00% 0.00% perl5.8.8
8307 root 10 0 16532K 15868K nanslp 0:01 0.00% 0.00% perl5.8.8
245 bind 2 0 2484K 1856K select 0:01 0.00% 0.00% named
241 root 2 0 1000K 668K select 0:01 0.00% 0.00% syslogd
8236 root 10 0 16516K 15856K nanslp 0:01 0.00% 0.00% perl5.8.8
10 root -2 0 0K 0K vlruwt 0:01 0.00% 0.00% vnlru
last pid: 9015; load averages: 0.02, 0.02, 0.00 up 1+00:02:04 06:37:53
76 processes: 1 running, 74 sleeping, 1 zombie
CPU states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
Mem: 95M Active, 264M Inact, 63M Wired, 21M Cache, 60M Buf, 55M Free
Swap: 250M Total, 250M Free
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
491 mysql 2 0 44788K 19244K poll 0:57 0.00% 0.00% mysqld
9 root 18 0 0K 0K syncer 0:16 0.00% 0.00% syncer
330 ssbdev 2 0 9744K 8188K poll 0:12 0.00% 0.00% python
248 root 2 0 1336K 868K select 0:04 0.00% 0.00% ntpd
224 root 2 0 464K 252K select 0:03 0.00% 0.00% natd
416 root 2 0 3188K 1948K select 0:03 0.00% 0.00% sendmail
6408 root 10 0 2868K 2292K nanslp 0:03 0.00% 0.00% perl5.8.8
308 root 2 0 9884K 5732K select 0:02 0.00% 0.00% httpd
6515 root 10 0 16548K 15872K nanslp 0:02 0.00% 0.00% perl5.8.8
505 root 2 0 5364K 1804K select 0:02 0.00% 0.00% nmbd
6738 root 10 0 16548K 15872K nanslp 0:01 0.00% 0.00% perl5.8.8
650 dlee 2 0 5336K 1836K select 0:01 0.00% 0.00% sshd
7778 root 10 0 16532K 15868K nanslp 0:01 0.00% 0.00% perl5.8.8
8307 root 10 0 16532K 15868K nanslp 0:01 0.00% 0.00% perl5.8.8
245 bind 2 0 2484K 1856K select 0:01 0.00% 0.00% named
241 root 2 0 1000K 668K select 0:01 0.00% 0.00% syslogd
8236 root 10 0 16516K 15856K nanslp 0:01 0.00% 0.00% perl5.8.8
10 root -2 0 0K 0K vlruwt 0:01 0.00% 0.00% vnlru
last pid: 9015; load averages: 0.02, 0.02, 0.00 up 1+00:02:06 06:37:55
76 processes: 1 running, 74 sleeping, 1 zombie
CPU states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
Mem: 95M Active, 264M Inact, 63M Wired, 21M Cache, 60M Buf, 55M Free
Swap: 250M Total, 250M Free
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
491 mysql 2 0 44788K 19244K poll 0:57 0.00% 0.00% mysqld
9 root 18 0 0K 0K syncer 0:16 0.00% 0.00% syncer
330 ssbdev 2 0 9744K 8188K poll 0:12 0.00% 0.00% python
248 root 2 0 1336K 868K select 0:04 0.00% 0.00% ntpd
224 root 2 0 464K 252K select 0:03 0.00% 0.00% natd
416 root 2 0 3188K 1948K select 0:03 0.00% 0.00% sendmail
6408 root 10 0 2868K 2292K nanslp 0:03 0.00% 0.00% perl5.8.8
308 root 2 0 9884K 5732K select 0:02 0.00% 0.00% httpd
6515 root 10 0 16548K 15872K nanslp 0:02 0.00% 0.00% perl5.8.8
505 root 2 0 5364K 1804K select 0:02 0.00% 0.00% nmbd
6738 root 10 0 16548K 15872K nanslp 0:01 0.00% 0.00% perl5.8.8
650 dlee 2 0 5336K 1836K select 0:01 0.00% 0.00% sshd
7778 root 10 0 16532K 15868K nanslp 0:01 0.00% 0.00% perl5.8.8
8307 root 10 0 16532K 15868K nanslp 0:01 0.00% 0.00% perl5.8.8
245 bind 2 0 2484K 1856K select 0:01 0.00% 0.00% named
241 root 2 0 1000K 668K select 0:01 0.00% 0.00% syslogd
8236 root 10 0 16516K 15856K nanslp 0:01 0.00% 0.00% perl5.8.8
10 root -2 0 0K 0K vlruwt 0:01 0.00% 0.00% vnlru
[Boom!]
--
Doug Lee dgl@dlee.org http://www.dlee.org
SSB BART Group doug.lee@ssbbartgroup.com http://www.ssbbartgroup.com
"Is your cucumber bitter? Throw it away. Are there briars in your
path? Turn aside. That is enough. Do not go on to say, `Why were
things of this sort ever brought into the world?'"
--Marcus Aurelius
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20090517110657.GC2706>