Date: Wed, 11 Apr 2007 12:53:32 +0200 From: Kai <kai@xs4all.nl> To: stable@freebsd.org Subject: panic: bundirty (nfs related) Message-ID: <20070411105332.GC7847@xs4all.nl>
next in thread | raw e-mail | index | archive | help
Hello all,
We're running into regular panics on our webserver after upgrading
from 4.x to 6.2-stable:
# kgdb kernel.debug /usr/local/var/crash/vmcore.3
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so:
# Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
panic: bundirty: buffer 0xdc408bc0 still on queue 1
cpuid = 0
Uptime: 13h5m38s
Dumping 3326 MB (2 chunks)
chunk 0: 1MB (156 pages) ... ok
chunk 1: 3327MB (851552 pages) 3311 3295 3279 3263 3247 3231 3215 3199
3183 3167 3151 3135 3119 3103 3087 3071 3055 3039 3023 3007 2991 2975 2959
2943 2927 2911 2895 2879 2863 2847 2831 2815 2799 2783 2767 2751 2735 2719
2703 2687 2671 2655 2639 2623 2607 2591 2575 2559 2543 2527 2511 2495 2479
2463 2447 2431 2415 2399 2383 2367 2351 2335 2319 2303 2287 2271 2255 2239
2223 2207 2191 2175 2159 2143 2127 2111 2095 2079 2063 2047 2031 2015 1999
1983 1967 1951 1935 1919 1903 1887 1871 1855 1839 1823 1807 1791 1775 1759
1743 1727 1711 1695 1679 1663 1647 1631 1615 1599 1583 1567 1551 1535 1519
1503 1487 1471 1455 1439 1423 1407 1391 1375 1359 1343 1327 1311 1295 1279
1263 1247 1231 1215 1199 1183 1167 1151 1135 1119 1103 1087 1071 1055 1039
1023 1007 991 975 959 943 927 911 895 879 863 847 831 815 799 783 767 751
735 719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463
447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175
159 143 127 111 95 79 63 47 31 15
#0 doadump () at pcpu.h:165
165 __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) bt
#0 doadump () at pcpu.h:165
#1 0xc06708e8 in boot (howto=260) at ../../../kern/kern_shutdown.c:409
#2 0xc0670bca in panic (
fmt=0xc08f7fec "bundirty: buffer %p still on queue %d")
at ../../../kern/kern_shutdown.c:565
#3 0xc06b3409 in bundirty (bp=0xdc408bc0) at ../../../kern/vfs_bio.c:1036
#4 0xc06b3d6c in brelse (bp=0xdc408bc0) at ../../../kern/vfs_bio.c:1351
#5 0xc075e18e in nfs_writebp (bp=0xdc408bc0, force=1, td=0xcaa0e000)
at ../../../nfsclient/nfs_vnops.c:2982
#6 0xc075e3f8 in nfs_bwrite (bp=0xdc408bc0) at pcpu.h:162
#7 0xc06b5b7d in getblk (vp=0xc9908440, blkno=8438, size=6037, slpflag=256,
slptimeo=0, flags=0) at buf.h:412
#8 0xc0751299 in nfs_getcacheblk (vp=0xc9908440, bn=8438, size=6037,
td=0xcaa0e000) at ../../../nfsclient/nfs_bio.c:1253
#9 0xc0750e62 in nfs_write (ap=0x0) at ../../../nfsclient/nfs_bio.c:1069
#10 0xc08894f2 in VOP_WRITE_APV (vop=0xc097fc00, a=0xec2babf4)
at vnode_if.c:698
#11 0xc06cea9a in vn_write (fp=0xd01ec360, uio=0xec2bacbc,
active_cred=0xc8e71900, flags=0, td=0xcaa0e000) at vnode_if.h:372
#12 0xc0692867 in dofilewrite (td=0xcaa0e000, fd=5, fp=0xd01ec360,
auio=0xec2bacbc, offset=Unhandled dwarf expression opcode 0x93
) at file.h:252
#13 0xc069270b in kern_writev (td=0xcaa0e000, fd=5, auio=0xec2bacbc)
at ../../../kern/sys_generic.c:402
#14 0xc0692631 in write (td=0xcaa0e000, uap=0x0)
at ../../../kern/sys_generic.c:326
#15 0xc087842f in syscall (frame=
{tf_fs = 136970299, tf_es = 1210646587, tf_ds = -1078001605, tf_edi =
#136986624, tf_esi = 1210674680, tf_ebp = -1077942696, tf_isp = -332681884,
tf_ebx = 1210584448, tf_edx = 4096, tf_ecx = 0, tf_eax = 4, tf_trapno = 0,
#tf_err = 2, tf_eip = 1210524463, tf_cs = 51, tf_eflags = 530, tf_esp =
-1077942724, tf_ss = 59}) at ../../../i386/i386/trap.c:983
#16 0xc08649ef in Xint0x80_syscall () at ../../../i386/i386/exception.s:200
#17 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)
dmesg of one of the hosts having this problem:
FreeBSD 6.2-RELEASE-p3 #2: Thu Apr 5 14:15:19 CEST 2007
kai@webserver:/usr/src/sys/i386/compile/SMP-DEBUG
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU 5130 @ 2.00GHz (2000.08-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0x6f6 Stepping = 6
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x4e33d<SSE3,RSVD2,MON,DS_CPL,VMX,TM2,<b9>,CX16,<b14>,<b15>,<b18>>
AMD Features=0x20100000<NX,LM>
AMD Features2=0x1<LAHF>
Cores per package: 2
real memory = 3489005568 (3327 MB)
avail memory = 3414216704 (3256 MB)
ACPI APIC Table: <PTLTD APIC >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 6
cpu3 (AP): APIC ID: 7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
[...]
Uname reports 6.2-RELEASE-p3. Its a custom kernel:
# higher limits
options MAXDSIZ="(1024UL*1024*1024)"
options MAXSSIZ="(1024UL*1024*1024)"
options DFLDSIZ="(256UL*1024*1024)"
options SMP
makeoptions DEBUG=-g
options INVARIANTS
options INVARIANT_SUPPORT
options DIAGNOSTIC
Its running apache-1.3.37, serving pages from an NFS server (netapp) over
nfs v3. Other mount options are: rw,bg,intr,resvport,nosuid.
Anyone interested in helping me out?
Regards,
Kai
--
This was an above the .signature production
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070411105332.GC7847>