Date: Tue, 1 Mar 2005 21:29:34 -0800 From: Dariusz Kulinski <d.kulinski@gmail.com> To: Doug White <dwhite@gumbysoft.com>, freebsd-current@freebsd.org Subject: Re: FreeBSD 5.3 crash (core with debug symbols available) Message-ID: <1872760286.20050301212934@takeda.tk> In-Reply-To: <20050301184735.O72408@carver.gumbysoft.com> References: <549575862.20050226230200@takeda.tk> <20050228210235.C62607@carver.gumbysoft.com> <20050301104030.W68845@carver.gumbysoft.com> <1119242149.20050301105816@takeda.tk> <20050301184735.O72408@carver.gumbysoft.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hello Doug,
Tuesday, March 1, 2005, 6:59:42 PM, you wrote:
> Some more exploration is necessary. Can you make the crashdump and debug
> kernel available? Also, what was running when this panic tripped? ("info
> threads" in kgdb may be useful.)
This crash occured while doing backup on a live system (snapshot was
enabled to get consistent backup).
But it might not be related to the snapshot, today (Tuesday) I noticed
on morning that system crashed again. I posted information about
that before (Message-ID: <596413574.20050301104045@takeda.tk>). This
crash also occured in same line in the code, but the difference is
that it happened when I wasn't around. So it's hard to me to tell if
the line is just coincidence, or the crashes are related.
From=20the logfile (of the second crash (not this one)):
[...]
Mar 1 07:18:32 chinatsu httpd: PHP Fatal error: Allowed memory size
of 8388608 bytes exhausted (tried to allocate 240 bytes) in
/usr/local/www/phpwiki-1.3.10/lib/WikiUserNew.php on line 1012=20
Mar 1 09:59:26 chinatsu syslogd: restart
Mar 1 09:59:26 chinatsu syslogd: kernel boot file is /boot/kernel/kernel
[...]
Looks like it happened somewhere after 7:18:32 (9:59:26 was when I
went to the console and asked to dump core file)
I showed the log file, because the message is strange. Maybe is related
to the crash. Please note that this is from the second crash that
occured today, not the one we're analyzing currently.
As for the list of threads from the first core dump, here it goes:
(kgdb) info threads
149 TID 100050 sched_switch (td=3D0xc1695000, newtd=3D0xc14e04b0, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
148 TID 100254 sched_switch (td=3D0xc2d9a640, newtd=3D0xc14e3e10, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
147 TID 100176 sched_switch (td=3D0xc1d0b960, newtd=3D0xc211e000, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
146 TID 100219 sched_switch (td=3D0xc1f9ae10, newtd=3D0xc14e04b0, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
kgdb: kvm_read: invalid address (d1791da0)
145 TID 100262 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d136ada0)
144 TID 100104 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1454da0)
143 TID 100143 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d14e8da0)
142 TID 100200 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d14b0da0)
141 TID 100158 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1317da0)
140 TID 100095 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d137cda0)
139 TID 100110 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1794da0)
138 TID 100263 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1842da0)
137 TID 100271 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1851da0)
136 TID 100276 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d168eda0)
135 TID 100305 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d16a0da0)
134 TID 100311 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1394da0)
133 TID 100118 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d173bda0)
132 TID 100333 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1314da0)
131 TID 100094 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1869da0)
130 TID 100284 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d142dda0)
129 TID 100130 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d14a1da0)
128 TID 100153 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1445da0)
127 TID 100138 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d14b3da0)
126 TID 100159 0x00000000 in ?? ()
125 TID 100172 sched_switch (td=3D0xc1d0b320, newtd=3D0xc1addaf0, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
kgdb: kvm_read: invalid address (d1732da0)
124 TID 100330 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d179dda0)
123 TID 100266 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d13ecda0)
122 TID 100129 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1618da0)
121 TID 100242 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d152fda0)
120 TID 100204 0x00000000 in ?? ()
119 TID 100220 sched_switch (td=3D0xc211e000, newtd=3D0xc2e59af0, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
kgdb: kvm_read: invalid address (d172cda0)
118 TID 100328 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1592da0)
117 TID 100192 0x00000000 in ?? ()
116 TID 100277 sched_switch (td=3D0xc2e59af0, newtd=3D0xc14e04b0, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
kgdb: kvm_read: invalid address (d130bda0)
115 TID 100091 0x00000000 in ?? ()
114 TID 100214 sched_switch (td=3D0xc1f9a640, newtd=3D0xc18db640, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
113 TID 100152 sched_switch (td=3D0xc1c54320, newtd=3D0xc2d9a640, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
112 TID 100066 sched_switch (td=3D0xc1696960, newtd=3D0xc14e04b0, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
111 TID 100150 sched_switch (td=3D0xc1c54000, newtd=3D0xc1696960, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
110 TID 100173 sched_switch (td=3D0xc1d0b4b0, newtd=3D0xc14e04b0, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
kgdb: kvm_read: invalid address (d130eda0)
109 TID 100092 0x00000000 in ?? ()
108 TID 100052 sched_switch (td=3D0xc1695320, newtd=3D0xc1f9a640, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
kgdb: kvm_read: invalid address (d1685da0)
107 TID 100302 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d09a6da0)
106 TID 100054 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d16a3da0)
105 TID 100312 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d175ada0)
104 TID 100339 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1694da0)
103 TID 100307 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d165dda0)
102 TID 100294 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d167cda0)
101 TID 100299 0x00000000 in ?? ()
100 TID 100210 sched_switch (td=3D0xc1f9a000, newtd=3D0xc1d0b000, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
99 TID 100170 sched_switch (td=3D0xc1d0b000, newtd=3D0xc1cd6000, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
98 TID 100132 sched_switch (td=3D0xc1adc320, newtd=3D0xc19dbc80, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
97 TID 100327 sched_switch (td=3D0xc189daf0, newtd=3D0xc1adc320, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
96 TID 100338 sched_switch (td=3D0xc19dbc80, newtd=3D0xc1c54000, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
95 TID 100285 sched_switch (td=3D0xc2e5a7d0, newtd=3D0xc1695960, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
94 TID 100310 sched_switch (td=3D0xc1cd6000, newtd=3D0xc2e5a7d0, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
kgdb: kvm_read: invalid address (d15cfda0)
93 TID 100240 0x00000000 in ?? ()
92 TID 100195 sched_switch (td=3D0xc2a4a7d0, newtd=3D0xc2d9a640, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
kgdb: kvm_read: invalid address (d09afda0)
91 TID 100057 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1708da0)
90 TID 100322 0x00000000 in ?? ()
89 TID 100320 sched_switch (td=3D0xc189d000, newtd=3D0xc14e04b0, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
kgdb: kvm_read: invalid address (d1574da0)
88 TID 100182 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d16acda0)
87 TID 100315 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1595da0)
86 TID 100193 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d09cdda0)
85 TID 100067 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d15a7da0)
84 TID 100199 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d158cda0)
83 TID 100190 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d12c8da0)
82 TID 100084 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1589da0)
81 TID 100189 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d12a1da0)
80 TID 100071 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d168bda0)
79 TID 100304 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d183fda0)
78 TID 100270 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1571da0)
77 TID 100181 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1682da0)
76 TID 100301 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d12bfda0)
75 TID 100081 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1654da0)
74 TID 100291 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d15e5da0)
73 TID 100217 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d144eda0)
72 TID 100141 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d13e0da0)
71 TID 100125 0x00000000 in ?? ()
70 TID 100156 sched_switch (td=3D0xc1c54960, newtd=3D0xc2d9a640, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
69 TID 100114 sched_switch (td=3D0xc18db640, newtd=3D0xc17c2c80, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
68 TID 100115 sched_switch (td=3D0xc18db7d0, newtd=3D0xc14e04b0, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
kgdb: kvm_read: invalid address (d14bfda0)
67 TID 100163 0x00000000 in ?? ()
66 TID 100144 sched_switch (td=3D0xc1add640, newtd=3D0xc2d9a640, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
65 TID 100147 sched_switch (td=3D0xc1addaf0, newtd=3D0xc18db640, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
kgdb: kvm_read: invalid address (d143fda0)
64 TID 100136 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1308da0)
63 TID 100090 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d09b2da0)
62 TID 100058 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d131dda0)
61 TID 100097 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1370da0)
60 TID 100106 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d13e6da0)
59 TID 100127 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d12b3da0)
58 TID 100077 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1379da0)
57 TID 100109 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (cbd2eda0)
56 TID 100024 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1364da0)
55 TID 100102 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d136dda0)
54 TID 100105 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d13e3da0)
53 TID 100126 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1311da0)
52 TID 100093 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1376da0)
51 TID 100108 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d09b8da0)
50 TID 100060 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d09b5da0)
49 TID 100059 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d09c1da0)
48 TID 100063 0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d09beda0)
47 TID 100062 0x00000000 in ?? ()
46 TID 100056 sched_switch (td=3D0xc1695960, newtd=3D0xc14e8e10, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
45 TID 100023 sched_switch (td=3D0xc14e84b0, newtd=3D0xc14e8c80, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
44 TID 100049 sched_switch (td=3D0xc1692e10, newtd=3D0xc14e84b0, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
43 TID 100048 sched_switch (td=3D0xc1692c80, newtd=3D0xc1692e10, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
42 TID 100047 sched_switch (td=3D0xc1692af0, newtd=3D0xc1692c80, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
41 TID 100046 sched_switch (td=3D0xc1692960, newtd=3D0xc1692af0, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
40 TID 100045 sched_switch (td=3D0xc16927d0, newtd=3D0xc18db7d0, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
39 TID 100044 sched_switch (td=3D0xc1692640, newtd=3D0xc2d9a640, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
38 TID 100043 sched_switch (td=3D0xc16924b0, newtd=3D0xc14e84b0, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
37 TID 100042 sched_switch (td=3D0xc1692320, newtd=3D0xc14e04b0, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
36 TID 100041 sched_switch (td=3D0xc1692190, newtd=3D0xc14e0c80, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
35 TID 100040 sched_switch (td=3D0xc1692000, newtd=3D0xc14e8c80, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
34 TID 100012 sched_switch (td=3D0xc14e3320, newtd=3D0xc14e8c80, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
33 TID 100011 sched_switch (td=3D0xc14e3190, newtd=3D0xc14e8960, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
32 TID 100035 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2=
06
31 TID 100034 sched_switch (td=3D0xc150b640, newtd=3D0xc14e3190, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
30 TID 100033 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2=
06
29 TID 100032 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2=
06
28 TID 100031 sched_switch (td=3D0xc150b190, newtd=3D0xc18d87d0, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
27 TID 100030 sched_switch (td=3D0xc150b000, newtd=3D0xc1d0b320, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
26 TID 100029 sched_switch (td=3D0xc14e8e10, newtd=3D0xc1c54000, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
25 TID 100028 sched_switch (td=3D0xc14e8c80, newtd=3D0xc14e04b0, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
24 TID 100027 sched_switch (td=3D0xc14e8af0, newtd=3D0xc1692000, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
23 TID 100026 sched_switch (td=3D0xc14e8960, newtd=3D0xc14e8c80, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
22 TID 100000 sched_switch (td=3D0xc14e0000, newtd=3D0xc14e8190, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
21 TID 100022 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2=
06
* 20 TID 100021 doadump () at pcpu.h:159
19 TID 100020 sched_switch (td=3D0xc14e8000, newtd=3D0xc150b190, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
18 TID 100019 sched_switch (td=3D0xc14e3e10, newtd=3D0xc14e8c80, flags=
=3D1) at /usr/src/sys/kern/sched_4bsd.c:866
17 TID 100018 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2=
06
16 TID 100017 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2=
06
15 TID 100016 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2=
06
14 TID 100015 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2=
06
13 TID 100014 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2=
06
12 TID 100013 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2=
06
11 TID 100010 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2=
06
10 TID 100009 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2=
06
9 TID 100008 sched_switch (td=3D0xc14e0c80, newtd=3D0xc1695320, flags=3D=
1) at /usr/src/sys/kern/sched_4bsd.c:866
8 TID 100007 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:206
7 TID 100006 sched_switch (td=3D0xc14e0960, newtd=3D0xc150b000, flags=3D=
1) at /usr/src/sys/kern/sched_4bsd.c:866
6 TID 100005 sched_switch (td=3D0xc14e07d0, newtd=3D0xc14e8190, flags=3D=
1) at /usr/src/sys/kern/sched_4bsd.c:866
5 TID 100004 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:206
4 TID 100003 sched_switch (td=3D0xc14e04b0, newtd=3D0xc14e8190, flags=3D=
1) at /usr/src/sys/kern/sched_4bsd.c:866
3 TID 100002 sched_switch (td=3D0xc14e0320, newtd=3D0xc17c2000, flags=3D=
1) at /usr/src/sys/kern/sched_4bsd.c:866
2 TID 100001 sched_switch (td=3D0xc14e0190, newtd=3D0xc1b954b0, flags=3D=
1) at /usr/src/sys/kern/sched_4bsd.c:866
1 TID 0 sched_switch (td=3D0xc07203c0, newtd=3D0xc14e8af0, flags=3D1) at=
/usr/src/sys/kern/sched_4bsd.c:866
(kgdb)
I can provide kernel.debug, but as for publishing core dump... I
belive there is a lot of sensitive data (besides that is a core dump
of a kernel, the crash occured while doing a backup). The worst thing
is that it's not only my data but also data of my users.
Isn't there a better way?
--=20
Best regards,
Dariusz mailto:d.kulinski@gmail.com
http://www.takeda.tk
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1872760286.20050301212934>