From owner-freebsd-current@FreeBSD.ORG Wed Mar 2 05:29:40 2005 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2919416A4CF for ; Wed, 2 Mar 2005 05:29:40 +0000 (GMT) Received: from chinatsu.takeda.tk (node-402413e2.sna.onnet.us.uu.net [64.36.19.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3494443D2D for ; Wed, 2 Mar 2005 05:29:39 +0000 (GMT) (envelope-from d.kulinski@gmail.com) Received: from takeda.lan (takeda.lan [10.0.0.3]) (authenticated bits=0) by chinatsu.takeda.tk (8.13.1/8.13.1) with ESMTP id j225Tb7f004807 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NO); Tue, 1 Mar 2005 21:29:38 -0800 (PST) (envelope-from d.kulinski@gmail.com) Date: Tue, 1 Mar 2005 21:29:34 -0800 From: Dariusz Kulinski X-Priority: 3 (Normal) Message-ID: <1872760286.20050301212934@takeda.tk> To: Doug White , freebsd-current@freebsd.org In-Reply-To: <20050301184735.O72408@carver.gumbysoft.com> References: <549575862.20050226230200@takeda.tk> <20050228210235.C62607@carver.gumbysoft.com> <20050301104030.W68845@carver.gumbysoft.com> <1119242149.20050301105816@takeda.tk> <20050301184735.O72408@carver.gumbysoft.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Virus-Scanned: ClamAV 0.83/742/Tue Mar 1 17:05:59 2005 on chinatsu.takeda.tk X-Virus-Status: Clean Subject: Re: FreeBSD 5.3 crash (core with debug symbols available) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: freebsd-current@freebsd.org List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Mar 2005 05:29:40 -0000 Hello Doug, Tuesday, March 1, 2005, 6:59:42 PM, you wrote: > Some more exploration is necessary. Can you make the crashdump and debug > kernel available? Also, what was running when this panic tripped? ("info > threads" in kgdb may be useful.) This crash occured while doing backup on a live system (snapshot was enabled to get consistent backup). But it might not be related to the snapshot, today (Tuesday) I noticed on morning that system crashed again. I posted information about that before (Message-ID: <596413574.20050301104045@takeda.tk>). This crash also occured in same line in the code, but the difference is that it happened when I wasn't around. So it's hard to me to tell if the line is just coincidence, or the crashes are related. From=20the logfile (of the second crash (not this one)): [...] Mar 1 07:18:32 chinatsu httpd: PHP Fatal error: Allowed memory size of 8388608 bytes exhausted (tried to allocate 240 bytes) in /usr/local/www/phpwiki-1.3.10/lib/WikiUserNew.php on line 1012=20 Mar 1 09:59:26 chinatsu syslogd: restart Mar 1 09:59:26 chinatsu syslogd: kernel boot file is /boot/kernel/kernel [...] Looks like it happened somewhere after 7:18:32 (9:59:26 was when I went to the console and asked to dump core file) I showed the log file, because the message is strange. Maybe is related to the crash. Please note that this is from the second crash that occured today, not the one we're analyzing currently. As for the list of threads from the first core dump, here it goes: (kgdb) info threads 149 TID 100050 sched_switch (td=3D0xc1695000, newtd=3D0xc14e04b0, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 148 TID 100254 sched_switch (td=3D0xc2d9a640, newtd=3D0xc14e3e10, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 147 TID 100176 sched_switch (td=3D0xc1d0b960, newtd=3D0xc211e000, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 146 TID 100219 sched_switch (td=3D0xc1f9ae10, newtd=3D0xc14e04b0, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 kgdb: kvm_read: invalid address (d1791da0) 145 TID 100262 0x00000000 in ?? () kgdb: kvm_read: invalid address (d136ada0) 144 TID 100104 0x00000000 in ?? () kgdb: kvm_read: invalid address (d1454da0) 143 TID 100143 0x00000000 in ?? () kgdb: kvm_read: invalid address (d14e8da0) 142 TID 100200 0x00000000 in ?? () kgdb: kvm_read: invalid address (d14b0da0) 141 TID 100158 0x00000000 in ?? () kgdb: kvm_read: invalid address (d1317da0) 140 TID 100095 0x00000000 in ?? () kgdb: kvm_read: invalid address (d137cda0) 139 TID 100110 0x00000000 in ?? () kgdb: kvm_read: invalid address (d1794da0) 138 TID 100263 0x00000000 in ?? () kgdb: kvm_read: invalid address (d1842da0) 137 TID 100271 0x00000000 in ?? () kgdb: kvm_read: invalid address (d1851da0) 136 TID 100276 0x00000000 in ?? () kgdb: kvm_read: invalid address (d168eda0) 135 TID 100305 0x00000000 in ?? () kgdb: kvm_read: invalid address (d16a0da0) 134 TID 100311 0x00000000 in ?? () kgdb: kvm_read: invalid address (d1394da0) 133 TID 100118 0x00000000 in ?? () kgdb: kvm_read: invalid address (d173bda0) 132 TID 100333 0x00000000 in ?? () kgdb: kvm_read: invalid address (d1314da0) 131 TID 100094 0x00000000 in ?? () kgdb: kvm_read: invalid address (d1869da0) 130 TID 100284 0x00000000 in ?? () kgdb: kvm_read: invalid address (d142dda0) 129 TID 100130 0x00000000 in ?? () kgdb: kvm_read: invalid address (d14a1da0) 128 TID 100153 0x00000000 in ?? () kgdb: kvm_read: invalid address (d1445da0) 127 TID 100138 0x00000000 in ?? () kgdb: kvm_read: invalid address (d14b3da0) 126 TID 100159 0x00000000 in ?? () 125 TID 100172 sched_switch (td=3D0xc1d0b320, newtd=3D0xc1addaf0, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 kgdb: kvm_read: invalid address (d1732da0) 124 TID 100330 0x00000000 in ?? () kgdb: kvm_read: invalid address (d179dda0) 123 TID 100266 0x00000000 in ?? () kgdb: kvm_read: invalid address (d13ecda0) 122 TID 100129 0x00000000 in ?? () kgdb: kvm_read: invalid address (d1618da0) 121 TID 100242 0x00000000 in ?? () kgdb: kvm_read: invalid address (d152fda0) 120 TID 100204 0x00000000 in ?? () 119 TID 100220 sched_switch (td=3D0xc211e000, newtd=3D0xc2e59af0, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 kgdb: kvm_read: invalid address (d172cda0) 118 TID 100328 0x00000000 in ?? () kgdb: kvm_read: invalid address (d1592da0) 117 TID 100192 0x00000000 in ?? () 116 TID 100277 sched_switch (td=3D0xc2e59af0, newtd=3D0xc14e04b0, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 kgdb: kvm_read: invalid address (d130bda0) 115 TID 100091 0x00000000 in ?? () 114 TID 100214 sched_switch (td=3D0xc1f9a640, newtd=3D0xc18db640, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 113 TID 100152 sched_switch (td=3D0xc1c54320, newtd=3D0xc2d9a640, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 112 TID 100066 sched_switch (td=3D0xc1696960, newtd=3D0xc14e04b0, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 111 TID 100150 sched_switch (td=3D0xc1c54000, newtd=3D0xc1696960, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 110 TID 100173 sched_switch (td=3D0xc1d0b4b0, newtd=3D0xc14e04b0, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 kgdb: kvm_read: invalid address (d130eda0) 109 TID 100092 0x00000000 in ?? () 108 TID 100052 sched_switch (td=3D0xc1695320, newtd=3D0xc1f9a640, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 kgdb: kvm_read: invalid address (d1685da0) 107 TID 100302 0x00000000 in ?? () kgdb: kvm_read: invalid address (d09a6da0) 106 TID 100054 0x00000000 in ?? () kgdb: kvm_read: invalid address (d16a3da0) 105 TID 100312 0x00000000 in ?? () kgdb: kvm_read: invalid address (d175ada0) 104 TID 100339 0x00000000 in ?? () kgdb: kvm_read: invalid address (d1694da0) 103 TID 100307 0x00000000 in ?? () kgdb: kvm_read: invalid address (d165dda0) 102 TID 100294 0x00000000 in ?? () kgdb: kvm_read: invalid address (d167cda0) 101 TID 100299 0x00000000 in ?? () 100 TID 100210 sched_switch (td=3D0xc1f9a000, newtd=3D0xc1d0b000, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 99 TID 100170 sched_switch (td=3D0xc1d0b000, newtd=3D0xc1cd6000, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 98 TID 100132 sched_switch (td=3D0xc1adc320, newtd=3D0xc19dbc80, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 97 TID 100327 sched_switch (td=3D0xc189daf0, newtd=3D0xc1adc320, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 96 TID 100338 sched_switch (td=3D0xc19dbc80, newtd=3D0xc1c54000, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 95 TID 100285 sched_switch (td=3D0xc2e5a7d0, newtd=3D0xc1695960, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 94 TID 100310 sched_switch (td=3D0xc1cd6000, newtd=3D0xc2e5a7d0, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 kgdb: kvm_read: invalid address (d15cfda0) 93 TID 100240 0x00000000 in ?? () 92 TID 100195 sched_switch (td=3D0xc2a4a7d0, newtd=3D0xc2d9a640, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 kgdb: kvm_read: invalid address (d09afda0) 91 TID 100057 0x00000000 in ?? () kgdb: kvm_read: invalid address (d1708da0) 90 TID 100322 0x00000000 in ?? () 89 TID 100320 sched_switch (td=3D0xc189d000, newtd=3D0xc14e04b0, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 kgdb: kvm_read: invalid address (d1574da0) 88 TID 100182 0x00000000 in ?? () kgdb: kvm_read: invalid address (d16acda0) 87 TID 100315 0x00000000 in ?? () kgdb: kvm_read: invalid address (d1595da0) 86 TID 100193 0x00000000 in ?? () kgdb: kvm_read: invalid address (d09cdda0) 85 TID 100067 0x00000000 in ?? () kgdb: kvm_read: invalid address (d15a7da0) 84 TID 100199 0x00000000 in ?? () kgdb: kvm_read: invalid address (d158cda0) 83 TID 100190 0x00000000 in ?? () kgdb: kvm_read: invalid address (d12c8da0) 82 TID 100084 0x00000000 in ?? () kgdb: kvm_read: invalid address (d1589da0) 81 TID 100189 0x00000000 in ?? () kgdb: kvm_read: invalid address (d12a1da0) 80 TID 100071 0x00000000 in ?? () kgdb: kvm_read: invalid address (d168bda0) 79 TID 100304 0x00000000 in ?? () kgdb: kvm_read: invalid address (d183fda0) 78 TID 100270 0x00000000 in ?? () kgdb: kvm_read: invalid address (d1571da0) 77 TID 100181 0x00000000 in ?? () kgdb: kvm_read: invalid address (d1682da0) 76 TID 100301 0x00000000 in ?? () kgdb: kvm_read: invalid address (d12bfda0) 75 TID 100081 0x00000000 in ?? () kgdb: kvm_read: invalid address (d1654da0) 74 TID 100291 0x00000000 in ?? () kgdb: kvm_read: invalid address (d15e5da0) 73 TID 100217 0x00000000 in ?? () kgdb: kvm_read: invalid address (d144eda0) 72 TID 100141 0x00000000 in ?? () kgdb: kvm_read: invalid address (d13e0da0) 71 TID 100125 0x00000000 in ?? () 70 TID 100156 sched_switch (td=3D0xc1c54960, newtd=3D0xc2d9a640, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 69 TID 100114 sched_switch (td=3D0xc18db640, newtd=3D0xc17c2c80, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 68 TID 100115 sched_switch (td=3D0xc18db7d0, newtd=3D0xc14e04b0, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 kgdb: kvm_read: invalid address (d14bfda0) 67 TID 100163 0x00000000 in ?? () 66 TID 100144 sched_switch (td=3D0xc1add640, newtd=3D0xc2d9a640, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 65 TID 100147 sched_switch (td=3D0xc1addaf0, newtd=3D0xc18db640, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 kgdb: kvm_read: invalid address (d143fda0) 64 TID 100136 0x00000000 in ?? () kgdb: kvm_read: invalid address (d1308da0) 63 TID 100090 0x00000000 in ?? () kgdb: kvm_read: invalid address (d09b2da0) 62 TID 100058 0x00000000 in ?? () kgdb: kvm_read: invalid address (d131dda0) 61 TID 100097 0x00000000 in ?? () kgdb: kvm_read: invalid address (d1370da0) 60 TID 100106 0x00000000 in ?? () kgdb: kvm_read: invalid address (d13e6da0) 59 TID 100127 0x00000000 in ?? () kgdb: kvm_read: invalid address (d12b3da0) 58 TID 100077 0x00000000 in ?? () kgdb: kvm_read: invalid address (d1379da0) 57 TID 100109 0x00000000 in ?? () kgdb: kvm_read: invalid address (cbd2eda0) 56 TID 100024 0x00000000 in ?? () kgdb: kvm_read: invalid address (d1364da0) 55 TID 100102 0x00000000 in ?? () kgdb: kvm_read: invalid address (d136dda0) 54 TID 100105 0x00000000 in ?? () kgdb: kvm_read: invalid address (d13e3da0) 53 TID 100126 0x00000000 in ?? () kgdb: kvm_read: invalid address (d1311da0) 52 TID 100093 0x00000000 in ?? () kgdb: kvm_read: invalid address (d1376da0) 51 TID 100108 0x00000000 in ?? () kgdb: kvm_read: invalid address (d09b8da0) 50 TID 100060 0x00000000 in ?? () kgdb: kvm_read: invalid address (d09b5da0) 49 TID 100059 0x00000000 in ?? () kgdb: kvm_read: invalid address (d09c1da0) 48 TID 100063 0x00000000 in ?? () kgdb: kvm_read: invalid address (d09beda0) 47 TID 100062 0x00000000 in ?? () 46 TID 100056 sched_switch (td=3D0xc1695960, newtd=3D0xc14e8e10, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 45 TID 100023 sched_switch (td=3D0xc14e84b0, newtd=3D0xc14e8c80, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 44 TID 100049 sched_switch (td=3D0xc1692e10, newtd=3D0xc14e84b0, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 43 TID 100048 sched_switch (td=3D0xc1692c80, newtd=3D0xc1692e10, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 42 TID 100047 sched_switch (td=3D0xc1692af0, newtd=3D0xc1692c80, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 41 TID 100046 sched_switch (td=3D0xc1692960, newtd=3D0xc1692af0, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 40 TID 100045 sched_switch (td=3D0xc16927d0, newtd=3D0xc18db7d0, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 39 TID 100044 sched_switch (td=3D0xc1692640, newtd=3D0xc2d9a640, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 38 TID 100043 sched_switch (td=3D0xc16924b0, newtd=3D0xc14e84b0, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 37 TID 100042 sched_switch (td=3D0xc1692320, newtd=3D0xc14e04b0, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 36 TID 100041 sched_switch (td=3D0xc1692190, newtd=3D0xc14e0c80, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 35 TID 100040 sched_switch (td=3D0xc1692000, newtd=3D0xc14e8c80, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 34 TID 100012 sched_switch (td=3D0xc14e3320, newtd=3D0xc14e8c80, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 33 TID 100011 sched_switch (td=3D0xc14e3190, newtd=3D0xc14e8960, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 32 TID 100035 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2= 06 31 TID 100034 sched_switch (td=3D0xc150b640, newtd=3D0xc14e3190, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 30 TID 100033 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2= 06 29 TID 100032 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2= 06 28 TID 100031 sched_switch (td=3D0xc150b190, newtd=3D0xc18d87d0, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 27 TID 100030 sched_switch (td=3D0xc150b000, newtd=3D0xc1d0b320, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 26 TID 100029 sched_switch (td=3D0xc14e8e10, newtd=3D0xc1c54000, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 25 TID 100028 sched_switch (td=3D0xc14e8c80, newtd=3D0xc14e04b0, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 24 TID 100027 sched_switch (td=3D0xc14e8af0, newtd=3D0xc1692000, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 23 TID 100026 sched_switch (td=3D0xc14e8960, newtd=3D0xc14e8c80, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 22 TID 100000 sched_switch (td=3D0xc14e0000, newtd=3D0xc14e8190, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 21 TID 100022 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2= 06 * 20 TID 100021 doadump () at pcpu.h:159 19 TID 100020 sched_switch (td=3D0xc14e8000, newtd=3D0xc150b190, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 18 TID 100019 sched_switch (td=3D0xc14e3e10, newtd=3D0xc14e8c80, flags= =3D1) at /usr/src/sys/kern/sched_4bsd.c:866 17 TID 100018 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2= 06 16 TID 100017 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2= 06 15 TID 100016 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2= 06 14 TID 100015 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2= 06 13 TID 100014 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2= 06 12 TID 100013 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2= 06 11 TID 100010 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2= 06 10 TID 100009 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2= 06 9 TID 100008 sched_switch (td=3D0xc14e0c80, newtd=3D0xc1695320, flags=3D= 1) at /usr/src/sys/kern/sched_4bsd.c:866 8 TID 100007 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:206 7 TID 100006 sched_switch (td=3D0xc14e0960, newtd=3D0xc150b000, flags=3D= 1) at /usr/src/sys/kern/sched_4bsd.c:866 6 TID 100005 sched_switch (td=3D0xc14e07d0, newtd=3D0xc14e8190, flags=3D= 1) at /usr/src/sys/kern/sched_4bsd.c:866 5 TID 100004 fork_trampoline () at /usr/src/sys/i386/i386/exception.s:206 4 TID 100003 sched_switch (td=3D0xc14e04b0, newtd=3D0xc14e8190, flags=3D= 1) at /usr/src/sys/kern/sched_4bsd.c:866 3 TID 100002 sched_switch (td=3D0xc14e0320, newtd=3D0xc17c2000, flags=3D= 1) at /usr/src/sys/kern/sched_4bsd.c:866 2 TID 100001 sched_switch (td=3D0xc14e0190, newtd=3D0xc1b954b0, flags=3D= 1) at /usr/src/sys/kern/sched_4bsd.c:866 1 TID 0 sched_switch (td=3D0xc07203c0, newtd=3D0xc14e8af0, flags=3D1) at= /usr/src/sys/kern/sched_4bsd.c:866 (kgdb) I can provide kernel.debug, but as for publishing core dump... I belive there is a lot of sensitive data (besides that is a core dump of a kernel, the crash occured while doing a backup). The worst thing is that it's not only my data but also data of my users. Isn't there a better way? --=20 Best regards, Dariusz mailto:d.kulinski@gmail.com http://www.takeda.tk