Date: Mon, 22 Mar 1999 23:38:33 -0700 (MST) From: "Kenneth D. Merry" <ken@plutotech.com> To: mckusick@McKusick.COM (Kirk McKusick) Cc: stable@freebsd.org, dillon@freebsd.org Subject: Re: quintuple softupdates panic in -stable Message-ID: <199903230638.XAA71643@panzer.plutotech.com> In-Reply-To: <199903230229.SAA10662@flamingo.McKusick.COM> from Kirk McKusick at "Mar 22, 1999 6:29:11 pm"
next in thread | previous in thread | raw e-mail | index | archive | help
Kirk McKusick wrote...
> From: "Kenneth D. Merry" <ken@plutotech.com>
> Subject: quintuple softupdates panic in -stable
> To: stable@FreeBSD.ORG
> Date: Mon, 22 Mar 1999 14:51:10 -0700 (MST)
> Cc: mckusick@mckusick.com, dillon@FreeBSD.ORG
>
> I just got this panic on a very heavily loaded -stable
> server. It is RELENG_3 as of about March 10th.
>
> I noticed that Kirk checked in some softupdates bug fixes
> on March 1st/2nd, but they didn't go into RELENG_3. I'm
> not sure if this panic would be fixed by those changes.
>
> I wasn't able to compile a kernel with debugging symbols
> that would correspond to the one that was running on the
> machine in question, so there are no symbols. Sorry.
>
> Ken
> --
> Kenneth Merry
> ken@plutotech.com
>
> ...
>
> panic: allocdirect_check: old 0 != new 397448 || lbn 12 >= 12
>
> I have never gotten this panic before. The fixes that I put in in
> early March would probably not have prevented it (though unless I
> knew what did cause this panic, I can't say that with complete
> assurance). At any rate, it would be very useful if you could
> reproducce this panic with a symbolled kernel. For starters, I
> would like to know the contents of oldadp and newadp (the second
> and third parameters to allocdirect_merge). It would also be
> useful to see the elments on the list headed by adphead (the
> first parameter to allocdirect_merge). The follow-on panics are
> the usual trash that crops up when trying to sync the filesystem
> while in the middle of a soft updates panic.
Okay, I believe I have managed to recompile a debugging kernel using the
same sources and from the same path. It seems like the symbols are more or
less okay.
I think I should also mention that at the time the panic occurred, one of
the partitions (out of 5) with softupdates enabled had filled up under
primarily NFS write activity. I suppose that could make a difference.
If you want more information from this core dump, just let me know. If
you'd like me to install another kernel and see if I get another panic, let
me know as well. I'll need to schedule downtime for the server ahead of
time to install the kernel.
Thanks,
Ken
--
Kenneth Merry
ken@plutotech.com
=============================================================================
gdb -k kernel.debug /var/crash/vmcore.4
GDB is free software and you are welcome to distribute copies of it
under certain conditions; type "show copying" to see the conditions.
There is absolutely no warranty for GDB; type "show warranty" for details.
GDB 4.16 (i386-unknown-freebsd),
Copyright 1996 Free Software Foundation, Inc...
IdlePTD 2813952
initial pcb at 241958
panicstr: softdep_disk_write_complete: lock is held
panic messages:
---
panic: allocdirect_check: old 0 != new 397448 || lbn 12 >= 12
syncing disks... panic: softdep_lock: locking against myself
panic: softdep_disk_write_complete: lock is held
panic: softdep_disk_write_complete: lock is held
panic: softdep_disk_write_complete: lock is held
dumping to dev 20401, offset 327680
dump 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1
---
#0 boot (howto=260) at ../../kern/kern_shutdown.c:285
285 dumppcb.pcb_cr3 = rcr3();
(kgdb) where
#0 boot (howto=260) at ../../kern/kern_shutdown.c:285
#1 0xf0146548 in at_shutdown (
function=0xf02255f8 <__set_sysctl__debug_sym_sysctl___debug_rush_requests+2020>, arg=0xf5a6db20, queue=-173614304) at ../../kern/kern_shutdown.c:446
#2 0xf01d414b in softdep_disk_write_complete (bp=0xf5a6db20)
at ../../ufs/ffs/ffs_softdep.c:2931
#3 0xf01658c5 in biodone (bp=0xf5a6db20) at ../../kern/vfs_bio.c:1911
#4 0xf0120f79 in dadone (periph=0xf3113180, done_ccb=0xf31c1600)
at ../../cam/scsi/scsi_da.c:1313
#5 0xf011cca3 in camisr (queue=0xf0240cc4) at ../../cam/cam_xpt.c:6001
#6 0xf011ca99 in swi_cambio () at ../../cam/cam_xpt.c:5905
#7 0xf011a194 in xpt_polled_action (start_ccb=0xf9d10da8)
at ../../cam/cam_xpt.c:3217
#8 0xf0121445 in dashutdown (howto=260, arg=0x0)
at ../../cam/scsi/scsi_da.c:1603
#9 0xf0146297 in boot (howto=260) at ../../kern/kern_shutdown.c:280
#10 0xf0146548 in at_shutdown (
function=0xf02255f8 <__set_sysctl__debug_sym_sysctl___debug_rush_requests+2020>, arg=0xf5a886f0, queue=-173504784) at ../../kern/kern_shutdown.c:446
#11 0xf01d414b in softdep_disk_write_complete (bp=0xf5a886f0)
at ../../ufs/ffs/ffs_softdep.c:2931
#12 0xf01658c5 in biodone (bp=0xf5a886f0) at ../../kern/vfs_bio.c:1911
#13 0xf0120f79 in dadone (periph=0xf3113180, done_ccb=0xf3394200)
at ../../cam/scsi/scsi_da.c:1313
#14 0xf011cca3 in camisr (queue=0xf0240cc4) at ../../cam/cam_xpt.c:6001
#15 0xf011ca99 in swi_cambio () at ../../cam/cam_xpt.c:5905
#16 0xf011a194 in xpt_polled_action (start_ccb=0xf9d11284)
at ../../cam/cam_xpt.c:3217
#17 0xf0121445 in dashutdown (howto=260, arg=0x0)
at ../../cam/scsi/scsi_da.c:1603
#18 0xf0146297 in boot (howto=260) at ../../kern/kern_shutdown.c:280
#19 0xf0146548 in at_shutdown (
function=0xf02255f8 <__set_sysctl__debug_sym_sysctl___debug_rush_requests+2020>, arg=0xf5a31358, queue=-173862056) at ../../kern/kern_shutdown.c:446
#20 0xf01d414b in softdep_disk_write_complete (bp=0xf5a31358)
at ../../ufs/ffs/ffs_softdep.c:2931
#21 0xf01658c5 in biodone (bp=0xf5a31358) at ../../kern/vfs_bio.c:1911
#22 0xf0120f79 in dadone (periph=0xf3113180, done_ccb=0xf3397800)
at ../../cam/scsi/scsi_da.c:1313
#23 0xf011cca3 in camisr (queue=0xf0240cc4) at ../../cam/cam_xpt.c:6001
#24 0xf011ca99 in swi_cambio () at ../../cam/cam_xpt.c:5905
#25 0xf011a194 in xpt_polled_action (start_ccb=0xf9d11760)
at ../../cam/cam_xpt.c:3217
#26 0xf0121445 in dashutdown (howto=260, arg=0x0)
at ../../cam/scsi/scsi_da.c:1603
#27 0xf0146297 in boot (howto=260) at ../../kern/kern_shutdown.c:280
#28 0xf0146548 in at_shutdown (
function=0xf0224c1c <__set_sysuninit_set_sym_M_DIRREM_uninit_sys_uninit+4>, arg=0xf6555900, queue=-103736980) at ../../kern/kern_shutdown.c:446
#29 0xf01d112f in acquire_lock (lk=0xf02399e0)
at ../../ufs/ffs/ffs_softdep.c:264
#30 0xf01d3f4f in initiate_write_inodeblock (inodedep=0xf3a99b80,
bp=0xf5a2e518) at ../../ufs/ffs/ffs_softdep.c:2821
#31 0xf01d3cd3 in softdep_disk_io_initiation (bp=0xf5a2e518)
at ../../ufs/ffs/ffs_softdep.c:2680
#32 0xf01766b9 in spec_strategy (ap=0xf9d119c8)
at ../../miscfs/specfs/spec_vnops.c:539
#33 0xf0175e81 in spec_vnoperate (ap=0xf9d119c8)
at ../../miscfs/specfs/spec_vnops.c:129
#34 0xf01df489 in ufs_vnoperatespec (ap=0xf9d119c8)
at ../../ufs/ufs/ufs_vnops.c:2312
#35 0xf0163aff in bwrite (bp=0xf5a2e518) at vnode_if.h:891
#36 0xf016814e in vop_stdbwrite (ap=0xf9d11a30) at ../../kern/vfs_default.c:296
#37 0xf0167f99 in vop_defaultop (ap=0xf9d11a30) at ../../kern/vfs_default.c:130
#38 0xf0175e81 in spec_vnoperate (ap=0xf9d11a30)
at ../../miscfs/specfs/spec_vnops.c:129
#39 0xf01df489 in ufs_vnoperatespec (ap=0xf9d11a30)
at ../../ufs/ufs/ufs_vnops.c:2312
#40 0xf016453c in vfs_bio_awrite (bp=0xf5a2e518) at vnode_if.h:1145
#41 0xf01d909a in ffs_fsync (ap=0xf9d11a94) at ../../ufs/ffs/ffs_vnops.c:205
#42 0xf01d7544 in ffs_sync (mp=0xf311c000, waitfor=2, cred=0xf075b000,
p=0xf025f118) at vnode_if.h:499
#43 0xf016ca43 in sync (p=0xf025f118, uap=0x0) at ../../kern/vfs_syscalls.c:549
#44 0xf0146119 in boot (howto=256) at ../../kern/kern_shutdown.c:203
#45 0xf0146548 in at_shutdown (
function=0xf022500b <__set_sysctl__debug_sym_sysctl___debug_rush_requests+503>, arg=0x0, queue=397448) at ../../kern/kern_shutdown.c:446
#46 0xf01d216b in allocdirect_merge (adphead=0xf35eecc4, newadp=0xf32d1ec0,
oldadp=0xf3691bc0) at ../../ufs/ffs/ffs_softdep.c:1264
#47 0xf01d4d15 in merge_inode_lists (inodedep=0xf35eec80)
at ../../ufs/ffs/ffs_softdep.c:3555
#48 0xf01d4bdc in softdep_update_inodeblock (ip=0xf33d2600, bp=0xf5a946e0,
waitfor=0) at ../../ufs/ffs/ffs_softdep.c:3503
#49 0xf01d01e4 in ffs_update (vp=0xfa1d0e40, waitfor=0)
at ../../ufs/ffs/ffs_inode.c:105
#50 0xf01d9f25 in ufs_inactive (ap=0xf9d11c00) at ../../ufs/ufs/ufs_inode.c:92
#51 0xf01df459 in ufs_vnoperate (ap=0xf9d11c00)
at ../../ufs/ufs/ufs_vnops.c:2294
#52 0xf016abab in vput (vp=0xfa1d0e40) at vnode_if.h:767
#53 0xf019dbc6 in nfsrv_write (nfsd=0xf38d0700, slp=0xf30da300,
procp=0xf9cd0c80, mrq=0xf9d11e1c) at ../../nfs/nfs_serv.c:906
#54 0xf01b3a88 in nfssvc_nfsd (nsd=0xf9d11e7c,
argp=0x8071d9c <Address 0x8071d9c out of bounds>, p=0xf9cd0c80)
at ../../nfs/nfs_syscalls.c:656
#55 0xf01b3385 in nfssvc (p=0xf9cd0c80, uap=0xf9d11f84)
at ../../nfs/nfs_syscalls.c:342
#56 0xf0203a88 in syscall (frame={tf_es = 39, tf_ds = 39, tf_edi = 20,
tf_esi = 0, tf_ebp = -272638524, tf_isp = -103735340, tf_ebx = 0,
tf_edx = -272638920, tf_ecx = 0, tf_eax = 155, tf_trapno = 12,
tf_err = 2, tf_eip = 134519060, tf_cs = 31, tf_eflags = 642,
tf_esp = -272638912, tf_ss = 39}) at ../../i386/i386/trap.c:1118
#57 0xf01fa41c in Xint0x80_syscall ()
#58 0x80480e9 in ?? ()
(kgdb) up 46
#46 0xf01d216b in allocdirect_merge (adphead=0xf35eecc4, newadp=0xf32d1ec0,
oldadp=0xf3691bc0) at ../../ufs/ffs/ffs_softdep.c:1264
1264 panic("allocdirect_check: old %d != new %d || lbn %ld >= %d",
(kgdb) list
1259 panic("allocdirect_merge: lock not held");
1260 #endif
1261 if (newadp->ad_oldblkno != oldadp->ad_newblkno ||
1262 newadp->ad_oldsize != oldadp->ad_newsize ||
1263 newadp->ad_lbn >= NDADDR)
1264 panic("allocdirect_check: old %d != new %d || lbn %ld >= %d",
1265 newadp->ad_oldblkno, oldadp->ad_newblkno, newadp->ad_lbn,
1266 NDADDR);
1267 newadp->ad_oldblkno = oldadp->ad_oldblkno;
1268 newadp->ad_oldsize = oldadp->ad_oldsize;
(kgdb) print oldadp
$1 = (struct allocdirect *) 0x0
(kgdb) print newadp
$2 = (struct allocdirect *) 0x0
(kgdb) print (struct allocdirect *)0xf32d1ec0
$3 = (struct allocdirect *) 0xf32d1ec0
(kgdb) print *((struct allocdirect *)0xf32d1ec0)
$4 = {ad_list = {wk_list = {le_next = 0xf3691bc0, le_prev = 0xf5aaba04},
wk_type = 4, wk_state = 32769}, ad_next = {tqe_next = 0xf3691bc0,
tqe_prev = 0xf35eecc4}, ad_lbn = 12, ad_newblkno = 397448,
ad_oldblkno = 0, ad_newsize = 8192, ad_oldsize = 0, ad_deps = {
le_next = 0xf3691bc0, le_prev = 0xf31c8310}, ad_buf = 0xf5ab1438,
ad_inodedep = 0xf35eec80, ad_freefrag = 0x0}
(kgdb) print *((struct allocdirect *)0xf3691bc0)
$5 = {ad_list = {wk_list = {le_next = 0x0, le_prev = 0xf32d1ec0}, wk_type = 4,
wk_state = 32769}, ad_next = {tqe_next = 0x0, tqe_prev = 0xf32d1ecc},
ad_lbn = 12, ad_newblkno = 397448, ad_oldblkno = 0, ad_newsize = 8192,
ad_oldsize = 0, ad_deps = {le_next = 0x0, le_prev = 0xf32d1ee8},
ad_buf = 0xf5ab1438, ad_inodedep = 0xf35eec80, ad_freefrag = 0x0}
(kgdb) print adphead
$6 = (struct allocdirectlst *) 0xf35eecc4
(kgdb) print *adphead
$7 = {tqh_first = 0xf32d1ec0, tqh_last = 0xf3691bcc}
(kgdb) print *adphead->tqh_first
$8 = {ad_list = {wk_list = {le_next = 0xf3691bc0, le_prev = 0xf5aaba04},
wk_type = 4, wk_state = 32769}, ad_next = {tqe_next = 0xf3691bc0,
tqe_prev = 0xf35eecc4}, ad_lbn = 12, ad_newblkno = 397448,
ad_oldblkno = 0, ad_newsize = 8192, ad_oldsize = 0, ad_deps = {
le_next = 0xf3691bc0, le_prev = 0xf31c8310}, ad_buf = 0xf5ab1438,
ad_inodedep = 0xf35eec80, ad_freefrag = 0x0}
(kgdb) print *adphead->tqh_first.ad_next->tqe_next
$9 = {ad_list = {wk_list = {le_next = 0x0, le_prev = 0xf32d1ec0}, wk_type = 4,
wk_state = 32769}, ad_next = {tqe_next = 0x0, tqe_prev = 0xf32d1ecc},
ad_lbn = 12, ad_newblkno = 397448, ad_oldblkno = 0, ad_newsize = 8192,
ad_oldsize = 0, ad_deps = {le_next = 0x0, le_prev = 0xf32d1ee8},
ad_buf = 0xf5ab1438, ad_inodedep = 0xf35eec80, ad_freefrag = 0x0}
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-stable" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199903230638.XAA71643>