Date: Tue, 24 May 2011 11:06:43 +0300 From: Andriy Gapon <avg@FreeBSD.org> To: Joerg Wunsch <joerg_wunsch@uriah.heep.sax.de> Cc: freebsd-stable@FreeBSD.org, Joerg Wunsch <j@uriah.heep.sax.de> Subject: Re: RELENG_8: panic: wrong offset 4096 for sectorsize 2352 Message-ID: <4DDB6713.8070109@FreeBSD.org> In-Reply-To: <20110524072618.GB2110@uriah.heep.sax.de> References: <20110524055408.GA2110@uriah.heep.sax.de> <4DDB54A3.2050205@FreeBSD.org> <20110524072618.GB2110@uriah.heep.sax.de>
next in thread | previous in thread | raw e-mail | index | archive | help
on 24/05/2011 10:26 Joerg Wunsch said the following:
> As Andriy Gapon wrote:
>
>>> panic: wrong offset 4096 for sectorsize 2352
>>>
>>> Any ideas why this happens, and how to avoid it?
>
>> Backtrace would be a first thing.
>
> OK, here we go (the core has been dumped from within a serial console
> BREAK DDB entry, I'm omitting the frames related to that):
>
> #16 0xc0537352 in _cv_wait (cvp=0xc6e6bcd4, lock=0xc6e6bdd4) at /usr/src/sys/kern/kern_condvar.c:96
> #17 0xc0aa8a13 in usb_process (arg=0xc6e6bccc)
> at /usr/src/sys/modules/usb/usb/../../../dev/usb/usb_process.c:183
> #18 0xc054f948 in fork_exit (callout=0xc0aa88e0 <usb_process>, arg=0xc6e6bccc, frame=0xc6a1ad28)
> at /usr/src/sys/kern/kern_fork.c:865
> #19 0xc077fd34 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:275
Unfortunately this seems like a completely unrelated thread. And what you write
below explains why. Maybe you can find the right thread (with the original
panic call in it) in this crash dump or re-produce the problem later.
> After the initial panic, I typed "c" in DDB, in the assumption
> it would proceed with a coredump, but it didn't. That's why I
> hit BREAK again, and forced a dump through the "panic" DDB
> command. Now, I'm no longer sure whether the frames above do
> really relate to the mentioned panic string.
>
>> Information from a frame that called panic
>> would the next thing.
>
> (kgdb) up 16
> #16 0xc0537352 in _cv_wait (cvp=0xc6e6bcd4, lock=0xc6e6bdd4) at /usr/src/sys/kern/kern_condvar.c:96
> 96 {
> (kgdb) l
> 91 * sleeping and will be held on return. It is recommended that the mutex be
> 92 * held when cv_signal or cv_broadcast are called.
> 93 */
> 94 void
> 95 _cv_wait(struct cv *cvp, struct lock_object *lock)
> 96 {
> 97 WITNESS_SAVE_DECL(lock_witness);
> 98 struct lock_class *class;
> 99 struct thread *td;
> 100 int lock_state;
> (kgdb) up
> #17 0xc0aa8a13 in usb_process (arg=0xc6e6bccc)
> at /usr/src/sys/modules/usb/usb/../../../dev/usb/usb_process.c:183
> 183 cv_wait(&up->up_cv, up->up_mtx);
> (kgdb) l
> 178 if (up->up_dsleep) {
> 179 up->up_dsleep = 0;
> 180 cv_broadcast(&up->up_drain);
> 181 }
> 182 up->up_msleep = 1;
> 183 cv_wait(&up->up_cv, up->up_mtx);
> 184 }
> 185
> 186 up->up_ptr = NULL;
> 187 cv_signal(&up->up_cv);
> (kgdb) up
> #18 0xc054f948 in fork_exit (callout=0xc0aa88e0 <usb_process>, arg=0xc6e6bccc, frame=0xc6a1ad28)
> at /usr/src/sys/kern/kern_fork.c:865
> 865 callout(arg, frame);
> (kgdb) l
> 860 * cpu_set_fork_handler intercepts this function call to
> 861 * have this call a non-return function to stay in kernel mode.
> 862 * initproc has its own fork handler, but it does return.
> 863 */
> 864 KASSERT(callout != NULL, ("NULL callout in fork_exit"));
> 865 callout(arg, frame);
> 866
> 867 /*
> 868 * Check if a kernel thread misbehaved and returned from its main
> 869 * function.
>
> Here is the panic message from the DDB console:
>
> panic: wrong offset 4096 for sectorsize 2352
> cpuid = 0
> KDB: enter: panic
> [thread pid 2 tid 100009 ]
> Stopped at kdb_enter+0x3a: movl $0,kdb_why
>
> as well as the output of the DDB "ps" command:
>
> db> ps
> pid ppid pgrp uid state wmesg wchan cmd
> 70698 70661 70661 0 S+ g_waitid 0xc088bcec cd-discid
> 70661 2486 70661 0 S+ piperd 0xc98eb498 sh
> 70615 1450 1416 8 S nanslp 0xc088e224 sleep
> 70466 1706 1706 125 R smtpd
> 70160 1706 1706 125 S select 0xc8281624 pickup
> 68504 1706 1706 125 S select 0xc8a17324 anvil
> 56974 1330 1330 58 S lockf 0xced9f7c0 perl5.8.9
> 56769 1330 1330 58 S lockf 0xcc0a8400 perl5.8.9
> 56578 1330 1330 58 S lockf 0xc7517900 perl5.8.9
> 56268 1330 1330 58 S accept 0xc7ef103e perl5.8.9
> 56078 1330 1330 58 S lockf 0xc8a88a00 perl5.8.9
> 52965 52961 2222 107 S (threaded) mysql-query-browser
> 100291 S ucond 0xc8a1c880 mysql-query-browser
> 100289 S select 0xc821f324 initial thread
> 52961 1 2222 107 S wait 0xce70aac0 sh
> 49078 2262 49078 107 S+ pause 0xc8be8b18 screen
> 38746 1591 1591 80 S lockf 0xc8a219c0 httpd
> 35437 1591 1591 80 S lockf 0xc8a41440 httpd
> 35436 1591 1591 80 S kqread 0xcf21f300 httpd
> 35435 1591 1591 80 S lockf 0xc8a38c00 httpd
> 27055 1591 1591 80 S lockf 0xc8aa0b40 httpd
> 10767 1 2222 107 S select 0xc8a14264 xpdf
> 99867 1591 1591 80 S lockf 0xc8a31140 httpd
> 91459 1 91459 32001 Ss select 0xc7517024 dbus-daemon
> 91458 1 91454 32001 S+ select 0xcd0bec64 dbus-launch
> 54346 1591 1591 80 S lockf 0xcc4d3ac0 httpd
> 51334 51314 51334 107 S+ (threaded) qlandkartegt
> 100221 S select 0xc8aa6664 initial thread
> 51314 51312 51314 107 Ss+ ttyin 0xcd207470 tcsh
> 51312 1 2222 107 S select 0xc8a38ea4 xterm
> 29249 26397 29249 107 S+ select 0xc8a5c164 initial thread
> 26397 26395 26397 107 Ss+ ttyin 0xc929e670 tcsh
> 26395 1 2222 107 S select 0xc8a15664 xterm
> 25354 25352 25354 107 Ss+ ttyin 0xc93a5e70 tcsh
> 25352 1 2222 107 S select 0xc821f964 xterm
> 16153 1591 1591 80 S lockf 0xc8a31240 httpd
> 16151 1591 1591 80 S lockf 0xcc4e0880 httpd
> 15141 1591 1591 80 S lockf 0xcee65400 httpd
> 13309 13304 13303 32001 S+ (threaded) soffice.bin
> 100237 S ucond 0xc8aa0740 soffice.bin
> 100235 S select 0xc8a9e4e4 soffice.bin
> 100236 S accept 0xc898a9e6 soffice.bin
> 100232 S ucond 0xc8a35b80 soffice.bin
> 100165 S select 0xc82eb264 initial thread
> 13304 13303 13303 32001 S+ wait 0xc9177ac0 sh
> 13303 5638 13303 32001 S+ wait 0xc897c2b0 sh
> 8734 5638 8734 32001 S+ select 0xc81fd464 initial thread
> 7892 2273 7892 107 Ss+ ttyin 0xc8feda70 tcsh
> 7560 1706 1706 125 S select 0xc8a396e4 tlsmgr
> 5638 5635 5638 32001 Ss+ ttyin 0xc89cba70 bash
> 5635 5634 5635 32001 Ss select 0xc8a21724 xterm
> 5634 5632 5632 32001 S select 0xc7266264 sshd
> 5632 1635 5632 0 Ss sbwait 0xc8f96bfc sshd
> 4468 1 2222 107 S select 0xc8a401e4 initial thread
> 4466 4462 2222 107 R (threaded) firefox-bin
> 100271 S ucond 0xc8aad980 firefox-bin
> 100274 S ucond 0xc8a41840 firefox-bin
> 100234 S ucond 0xc8a3ea40 firefox-bin
> 100233 S ucond 0xc8a34200 firefox-bin
> 100231 S ucond 0xc72e3d80 firefox-bin
> 100230 S ucond 0xc8a38ac0 firefox-bin
> 100229 CanRun firefox-bin
> 100228 S ucond 0xc8a5f8c0 firefox-bin
> 100227 S select 0xc8a3b964 firefox-bin
> 100171 S select 0xc821faa4 initial thread
> 4462 4458 2222 107 S wait 0xc8be7560 sh
> 4458 1 2222 107 S wait 0xc8bcbac0 sh
> 3728 3653 3728 32001 S+ select 0xc8a613e4 mutt
> 3653 3650 3653 32001 Ss+ wait 0xc8be82b0 bash
> 3650 3649 3650 32001 Ss select 0xc8a5fea4 xterm
> 3649 3645 3645 32001 S select 0xc8a3f4e4 sshd
> 3645 1635 3645 0 Ss sbwait 0xc8306728 sshd
> 2486 2485 2486 0 S+ pause 0xc897d308 tcsh
> 2485 2251 2485 107 S+ wait 0xc7f41560 su
> 2277 2275 2277 107 S+ select 0xc8a5cb64 mutt
> 2275 2273 2275 107 Ss+ pause 0xc897c868 tcsh
> 2273 1 2273 107 Ss select 0xc821f624 screen
> 2262 2260 2262 107 Ss+ pause 0xc809bb18 tcsh
> 2260 1 2222 107 S select 0xc81fdbe4 xterm
> 2251 2240 2251 107 Ss+ pause 0xc809a868 tcsh
> 2249 1 2222 107 S select 0xc8220764 xload
> 2248 1 2222 107 S select 0xc8a61124 xclock
> 2245 2222 2222 107 S select 0xc821f864 FvwmPager
> 2244 2222 2222 107 S select 0xc8280ea4 GoodStuff
> 2242 2222 2222 107 S select 0xc82eb9a4 xconsole
> 2241 2222 2222 107 R initial thread
> 2240 2222 2222 107 S select 0xc8280464 xterm
> 2239 2222 2222 107 S select 0xc8aae064 beforelight
> 2225 2222 2225 107 Ss select 0xc821f2e4 ssh-agent
> 2222 1871 2222 107 Ss select 0xc8a449a4 fvwm
> 2184 1803 2184 0 S+ ttyin 0xc6fa8a70 tcsh
> 1916 1812 1808 0 S select 0xc7ffcca4 initial thread
> 1871 1720 1871 0 Ss wait 0xc74f9810 xdm
> 1848 1720 1848 0 Ss select 0xc8281164 initial thread
> 1847 1720 1847 0 Ss select 0xc821f2a4 xdm
> 1835 1812 1808 0 S kqread 0xc8219c00 initial thread
> 1816 1812 1808 0 R initial thread
> 1812 1808 1808 0 S select 0xc81fe5a4 initial thread
> 1811 1 1811 0 Ss (threaded) console-kit-daemon
> 100175 S ucond 0xc6f258c0 console-kit-daemon
> 100121 S select 0xc8280424 console-kit-daemon
> 1808 1 1808 560 Ss (threaded) hald
> 100176 S piperd 0xc7348ab8 hald
> 100096 D g_waitfo 0xced9d680 hald
> 1803 1 1803 0 Ss+ wait 0xc8270560 login
> 1802 1 1802 0 Ss+ ttyin 0xc6fa7070 getty
> 1801 1 1801 0 Ss+ ttyin 0xc6fa7270 getty
> 1800 1 1800 0 Ss+ ttyin 0xc6efe870 getty
> 1799 1 1799 0 Ss+ ttyin 0xc6efea70 getty
> 1798 1 1798 0 Ss+ ttyin 0xc6efee70 getty
> 1756 1 1756 0 Ss select 0xc81fd2a4 inetd
> 1723 1720 1723 0 Ss select 0xc8281724 initial thread
> 1720 1 1720 0 Ss select 0xc8280664 xdm
> 1711 1706 1706 125 S select 0xc821f8a4 qmgr
> 1706 1 1706 0 Ss select 0xc72673e4 master
> 1647 1 1647 0 Ss nanslp 0xc088e224 cron
> 1635 1 1635 0 Ss select 0xc7ffc824 sshd
> 1591 1 1591 0 Ss select 0xc75292a4 httpd
> 1538 1537 1537 0 S select 0xc7ffc8e4 conserver
> 1537 1 1537 0 Ss select 0xc7ffcea4 conserver
> 1526 1350 1350 0 S select 0xc7529aa4 perl
> 1525 1350 1350 0 S select 0xc7529e24 perl
> 1519 1 1519 0 Ss kqread 0xc8019780 cupsd
> 1487 1 1487 556 Ss select 0xc7ffcc64 dbus-daemon
> 1450 1 1416 8 S+ wait 0xc809a2b0 sh
> 1449 1 1449 8 Rs innd
> 1427 1373 1370 89 S+ (threaded) mysqld
> 100140 S ucond 0xc7266a00 mysqld
> 100136 S sigwait 0xe96d4bb0 mysqld
> 100139 S ucond 0xc7ffd240 mysqld
> 100138 S select 0xc7ffd4a4 mysqld
> 100137 S select 0xc7ffd164 mysqld
> 100128 S ucond 0xc7528640 mysqld
> 100127 S ucond 0xc70dc680 mysqld
> 100126 S ucond 0xc7517a00 mysqld
> 100125 S ucond 0xc7527840 mysqld
> 100064 S select 0xc7529764 initial thread
> 1373 1 1370 89 S+ wait 0xc7f3f560 sh
> 1350 1 1350 0 Ss select 0xc7518464 perl
> 1330 1 1330 58 Ss select 0xc7517b64 perl5.8.9
> 1322 1311 1309 88 S piperd 0xc7347620 unlinkd
> 1321 1311 1309 88 S sbwait 0xc7ef1bfc dnsserver
> 1320 1311 1309 88 S sbwait 0xc7530d98 dnsserver
> 1319 1311 1309 88 S sbwait 0xc7ef13f0 dnsserver
> 1318 1311 1309 88 S sbwait 0xc7ee7a60 dnsserver
> 1317 1311 1309 88 S sbwait 0xc7ef2254 dnsserver
> 1311 1309 1309 88 S kqread 0xc7f6ce00 squid
> 1309 1 1309 88 Ss wait 0xc7f3f000 squid
> 1283 1 1283 67 Ss select 0xc72e3d64 dhcpd
> 1224 1 1224 0 Ss select 0xc7528e64 powerd
> 1216 1 1216 0 Rs ntpd
> 1052 1 1052 0 Ss (threaded) apcupsd
> 100145 S select 0xc6fab564 apcupsd
> 100070 S select 0xc7517724 apcupsd
> 1029 1028 1028 0 S (threaded) nfsd
> 100116 S rpcsvc 0xc7517410 nfsd: service
> 100115 S rpcsvc 0xc7517550 nfsd: service
> 100114 S rpcsvc 0xc7527010 nfsd: service
> 100113 S rpcsvc 0xc75174d0 nfsd: master
> 1028 1 1028 0 Ss select 0xc72e2a64 nfsd
> 1018 1 1018 0 Ss select 0xc7517364 mountd
> 926 1 926 0 Ss select 0xc7527a24 rpcbind
> 909 1 909 53 Rs (threaded) named
> 100100 RunQ named
> 100099 S ucond 0xc7527140 named
> 100098 S ucond 0xc7517b80 named
> 100097 S ucond 0xc7527200 named
> 100075 S sigwait 0xe95debb0 named
> 817 1 817 0 Ss select 0xc75285a4 syslogd
> 636 1 636 0 Ss select 0xc75270a4 devd
> 604 1 604 0 Rs moused
> 464 455 455 64 S bpf 0xc7538400 pflogd
> 455 1 455 0 Ss sbwait 0xc751e728 pflogd
> 452 0 0 0 SL pftm 0xc760fb40 [pfpurge]
> 21 0 0 0 SL sdflush 0xc08a7000 [softdepflush]
> 20 0 0 0 SL vlruwt 0xc6dabac0 [vnlru]
> 19 0 0 0 SL syncer 0xc08a1a34 [syncer]
> 18 0 0 0 SL psleep 0xc08a1768 [bufdaemon]
> 17 0 0 0 SL pgzero 0xc08a7cd4 [pagezero]
> 16 0 0 0 SL psleep 0xc08a78fc [vmdaemon]
> 15 0 0 0 SL psleep 0xc08a78c4 [pagedaemon]
> 9 0 0 0 SL ccb_scan 0xc0877454 [xpt_thrd]
> 8 0 0 0 SL - 0xc6fb3100 [gv_worker]
> 7 0 0 0 SL waiting_ 0xc08a3218 [sctp_iterator]
> 6 0 0 0 SL - 0xc6efe63c [fdc0]
> 5 0 0 0 SL idle 0xc6f32000 [aic_recovery0]
> 14 0 0 0 RL (threaded) usb
> 100033 D - 0xc6e6bd34 [usbus1]
> 100032 RunQ [usbus1]
> 100031 Run CPU 0 [usbus1]
> 100030 D - 0xc6e6bca4 [usbus1]
> 100028 D - 0xc6e62b5c [usbus0]
> 100027 D - 0xc6e62b2c [usbus0]
> 100026 Run CPU 1 [usbus0]
> 100025 RunQ [usbus0]
> 13 0 0 0 SL - 0xc088e084 [yarrow]
> 4 0 0 0 SL - 0xc088bdc4 [g_down]
> 3 0 0 0 SL - 0xc088bdc0 [g_up]
> 2 0 0 0 RL [g_event]
> 12 0 0 0 RL (threaded) intr
> 100044 I [irq7: ppc0]
> 100041 RunQ [irq257: re0]
> 100040 I [irq20: atapci1]
> 100039 I [irq15: ata1]
> 100038 I [irq14: ata0]
> 100037 RunQ [swi0: uart uart+++*]
> 100036 RunQ [irq17: sym1 puc0]
> 100035 I [irq16: sym0 ahd0]
> 100034 I [irq256: hdac0]
> 100029 L *ehci0 0xc7f46280 [irq22: ehci0]
> 100024 L *ohci0 0xc6cc9280 [irq21: ohci0]
> 100023 I [irq9: acpi0]
> 100021 I [swi5: +]
> 100019 I [swi2: cambio]
> 100015 I [swi6: task queue]
> 100013 I [swi6: Giant taskq]
> 100008 I [swi1: netisr 0]
> 100007 I [swi3: vm]
> 100006 RunQ [swi4: clock]
> 100005 RunQ [swi4: clock]
> 11 0 0 0 RL (threaded) idle
> 100004 CanRun [idle: cpu0]
> 100003 CanRun [idle: cpu1]
> 1 0 1 0 SLs wait 0xc6d2bac0 [init]
> 10 0 0 0 SL audit_wo 0xc08a69a0 [audit]
> 0 0 0 0 SLs (threaded) kernel
> 100117 D - 0xc75279c0 [aiod_bio taskq]
> 100022 D - 0xc6dfc200 [thread taskq]
> 100020 D - 0xc6dfc2c0 [kqueue taskq]
> 100018 D - 0xc6dfc400 [acpi_task_2]
> 100017 D - 0xc6dfc400 [acpi_task_1]
> 100016 D - 0xc6dfc400 [acpi_task_0]
> 100014 D - 0xc6dfc740 [ffs_trim taskq]
> 100000 D sched 0xc088be80 [swapper]
>
> I hope that somehow makes sense ... I guess I could repeat the panic
> again by re-issuing the abcde command, but that will have to wait
> until tonight.
>
--
Andriy Gapon
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4DDB6713.8070109>