Date: Thu, 17 Jun 2010 18:35:24 +0300 From: Dmitry Pryanishnikov <lynx.ripe@gmail.com> To: freebsd-fs@freebsd.org Subject: Fwd: kern/147890: [ufs] [regression] ufs-related lock problem in RELENG_8 (18.04.2010 -> 20.04.2010) Message-ID: <AANLkTikgFovKkHlL612G4DSAPfMzZ6K6-hUMABmLSObw@mail.gmail.com> In-Reply-To: <AANLkTintNiqxw_MrlsRpR3lHNJ-qq7zdaXBQuR4zQLaQ@mail.gmail.com> References: <201006162220.o5GMK3uM002900@freefall.freebsd.org> <20100617072841.GY13238@deviant.kiev.zoral.com.ua> <AANLkTintNiqxw_MrlsRpR3lHNJ-qq7zdaXBQuR4zQLaQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
I'm sorry for not replying to all recipients... ---------- Forwarded message ---------- From: Dmitry Pryanishnikov <lynx.ripe@gmail.com> Date: 2010/6/17 Subject: Re: kern/147890: [ufs] [regression] ufs-related lock problem in RELENG_8 (18.04.2010 -> 20.04.2010) To: Kostik Belousov <kostikbel@gmail.com> Hello! 2010/6/17 Kostik Belousov <kostikbel@gmail.com>: > On Wed, Jun 16, 2010 at 10:20:03PM +0000, Dmitry Pryanishnikov wrote: >> The following reply was made to PR kern/147890; it has been noted by GNA= TS. >> >> From: Dmitry Pryanishnikov <lynx.ripe@gmail.com> >> To: Bruce Cran <bruce@cran.org.uk> >> Cc: bug-followup@freebsd.org >> Subject: Re: kern/147890: [ufs] [regression] ufs-related lock problem in >> =A0 =A0 =A0 RELENG_8 (18.04.2010 -> 20.04.2010) >> Date: Thu, 17 Jun 2010 01:12:17 +0300 >> >> =A0Hello! >> >> =A02010/6/16 Bruce Cran <bruce@cran.org.uk>: >> =A0> Could you run the commands shown in >> =A0> http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handboo= k/kerneldebug-deadlocks.html >> =A0> to get more information to help debug the problem please? >> >> =A0 =A0Done - I've rebuilt the kernel from the sources as of 20.04.2010 >> =A0with the missing debugging options, repeated the problem, turned > I suspect you do not. In particular, DEBUG_VFS_LOCKS, DEBUG_LOCKS > and WITNESS seems to be missed. =A0Actually they are present - here is the complete config of this kernel: # # DP, BSF8, 2010-06-16 ident =A0 =A0 =A0 =A0 =A0 LYNX makeoptions =A0 =A0 DEBUG=3D-g =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0#Build kernel= with gdb(1) debug symbols # This allows you to actually store this configuration file into # the kernel binary itself. See config(8) for more details. # options =A0 =A0 =A0 =A0 INCLUDE_CONFIG_FILE =A0 =A0 # Include this file in = kernel ##################################################################### # CPU OPTIONS cpu =A0 =A0 =A0 =A0 =A0 =A0 I686_CPU =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0# aka P= entium Pro(tm) ##################################################################### # Scheduler options: options =A0 =A0 =A0 =A0 SCHED_ULE ##################################################################### # SMP OPTIONS: options =A0 =A0 =A0 =A0 SMP =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 # Symme= tric MultiProcessor Kernel # SMP Debugging Options: options =A0 =A0 =A0 =A0 PREEMPTION =A0 =A0 =A0 =A0 =A0 =A0 =A0# Enable kern= el thread preemption options =A0 =A0 =A0 =A0 MUTEX_DEBUG options =A0 =A0 =A0 =A0 WITNESS device =A0 =A0 =A0 =A0 =A0apic =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0# I/O= apic ##################################################################### # COMPATIBILITY OPTIONS options =A0 =A0 =A0 =A0 COMPAT_FREEBSD4 options =A0 =A0 =A0 =A0 COMPAT_FREEBSD5 options =A0 =A0 =A0 =A0 COMPAT_FREEBSD6 options =A0 =A0 =A0 =A0 COMPAT_FREEBSD7 options =A0 =A0 =A0 =A0 SYSVSHM options =A0 =A0 =A0 =A0 SYSVSEM options =A0 =A0 =A0 =A0 SYSVMSG ##################################################################### # DEBUGGING OPTIONS options =A0 =A0 =A0 =A0 KDB options =A0 =A0 =A0 =A0 KDB_TRACE options =A0 =A0 =A0 =A0 DDB options =A0 =A0 =A0 =A0 DDB_NUMSYM options =A0 =A0 =A0 =A0 BREAK_TO_DEBUGGER =A0 =A0 =A0 # BREAK on a serial c= onsole goes to DDB options =A0 =A0 =A0 =A0 SYSCTL_DEBUG options =A0 =A0 =A0 =A0 KTRACE =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0#kernel t= racing options =A0 =A0 =A0 =A0 INVARIANTS options =A0 =A0 =A0 =A0 INVARIANT_SUPPORT options =A0 =A0 =A0 =A0 DEBUG_LOCKS options =A0 =A0 =A0 =A0 DEBUG_VFS_LOCKS options =A0 =A0 =A0 =A0 DIAGNOSTIC options =A0 =A0 =A0 =A0 DEADLKRES =A0 =A0 =A0 =A0 =A0 =A0 =A0 #software dea= dlock resolver thread ##################################################################### # NETWORKING OPTIONS options =A0 =A0 =A0 =A0 INET =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0#Intern= et communications protocols options =A0 =A0 =A0 =A0 INET6 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 #IPv6 com= munications protocols options =A0 =A0 =A0 =A0 IPSEC =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 #IP secur= ity (requires device crypto) options =A0 =A0 =A0 =A0 IPSEC_NAT_T =A0 =A0 =A0 =A0 =A0 =A0 #NAT-T support,= UDP encap of ESP device =A0 =A0 =A0 =A0 =A0ether =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 #Generi= c Ethernet device =A0 =A0 =A0 =A0 =A0loop =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0#Netw= ork loopback device device =A0 =A0 =A0 =A0 =A0bpf =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 #Berk= eley packet filter device =A0 =A0 =A0 =A0 =A0enc =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 #IPse= c interface options =A0 =A0 =A0 =A0 IPFIREWALL =A0 =A0 =A0 =A0 =A0 =A0 =A0#firewall options =A0 =A0 =A0 =A0 IPFIREWALL_VERBOSE =A0 =A0 =A0#enable logging to sy= slogd(8) options =A0 =A0 =A0 =A0 IPFIREWALL_VERBOSE_LIMIT=3D100 =A0 =A0#limit verbos= ity options =A0 =A0 =A0 =A0 IPFIREWALL_FORWARD =A0 =A0 =A0#packet destination c= hanges options =A0 =A0 =A0 =A0 IPFIREWALL_NAT =A0 =A0 =A0 =A0 =A0#ipfw kernel nat = support options =A0 =A0 =A0 =A0 LIBALIAS =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0# libalias = library, performing NAT options =A0 =A0 =A0 =A0 BPF_JITTER =A0 =A0 =A0 =A0 =A0 =A0 =A0#support for = BPF just-in-time compiler ##################################################################### # FILESYSTEM OPTIONS options =A0 =A0 =A0 =A0 FFS =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 #Fast f= ilesystem options =A0 =A0 =A0 =A0 SOFTUPDATES options =A0 =A0 =A0 =A0 UFS_EXTATTR options =A0 =A0 =A0 =A0 UFS_ACL options =A0 =A0 =A0 =A0 UFS_DIRHASH options =A0 =A0 =A0 =A0 UFS_GJOURNAL =A0 =A0 =A0 =A0 =A0 =A0# Gjournal-base= d UFS journaling support. options =A0 =A0 =A0 =A0 QUOTA =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 #enable d= isk quotas options =A0 =A0 =A0 =A0 SUIDDIR =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 #The suiddi= r hack ##################################################################### # POSIX P1003.1B options =A0 =A0 =A0 =A0 _KPOSIX_PRIORITY_SCHEDULING options =A0 =A0 =A0 =A0 P1003_1B_SEMAPHORES ##################################################################### # MISCELLANEOUS DEVICES AND OPTIONS device =A0 =A0 =A0 =A0 =A0pty =A0 =A0 =A0 =A0 =A0 =A0 #Pseudo ttys # Size of the kernel message buffer. =A0Should be N * pagesize. options =A0 =A0 =A0 =A0 MSGBUF_SIZE=3D131072 ##################################################################### # crypto subsystem device =A0 =A0 =A0 =A0 =A0crypto =A0 =A0 =A0 =A0 =A0# core crypto support ##################################################################### # HARDWARE DEVICE CONFIGURATION # Bus support. device =A0 =A0 =A0 =A0 =A0pci # The keyboard controller; it controls the keyboard and the PS/2 mouse. device =A0 =A0 =A0 =A0 =A0atkbdc =A0 =A0 =A0 =A0 =A0# AT keyboard controlle= r device =A0 =A0 =A0 =A0 =A0atkbd =A0 =A0 =A0 =A0 =A0 # AT keyboard device =A0 =A0 =A0 =A0 =A0psm =A0 =A0 =A0 =A0 =A0 =A0 # PS/2 mouse options =A0 =A0 =A0 =A0 KBD_INSTALL_CDEV =A0 =A0 =A0 =A0# install a CDEV en= try in /dev device =A0 =A0 =A0 =A0 =A0vga =A0 =A0 =A0 =A0 =A0 =A0 # VGA video card driv= er device =A0 =A0 =A0 =A0 =A0splash =A0 =A0 =A0 =A0 =A0# Splash screen and scr= een saver support options =A0 =A0 =A0 =A0 X86BIOS =A0 =A0 =A0 =A0 # x86 real mode BIOS emulat= or, required by atkbdc/dpms/vesa options =A0 =A0 =A0 =A0 PRINTF_BUFR_SIZE=3D128 =A0 =A0# Prevent printf outp= ut being interspersed. # The syscons console driver (SCO color console compatible). device =A0 =A0 =A0 =A0 =A0sc options =A0 =A0 =A0 =A0 SC_HISTORY_SIZE=3D20000 options =A0 =A0 =A0 =A0 SC_TWOBUTTON_MOUSE #options =A0 =A0 =A0 =A0TEKEN_UTF8 =A0 =A0 =A0 =A0 =A0 =A0 =A0# UTF-8 outpu= t handling #options =A0 =A0 =A0 =A0TEKEN_XTERM =A0 =A0 =A0 =A0 =A0 =A0 # xterm-style t= erminal emulation device =A0 =A0 =A0 =A0 =A0uart =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0# Gen= eric UART driver # ATA and ATAPI devices device =A0 =A0 =A0 =A0 =A0ata device =A0 =A0 =A0 =A0 =A0atadisk =A0 =A0 =A0 =A0 # ATA disk drives options =A0 =A0 =A0 =A0 ATA_STATIC_ID =A0 # Static device numbering # SCSI peripherals device =A0 =A0 =A0 =A0 =A0scbus =A0 =A0 =A0 =A0 =A0 # SCSI bus (required fo= r SCSI) device =A0 =A0 =A0 =A0 =A0da =A0 =A0 =A0 =A0 =A0 =A0 =A0# Direct Access (di= sks) device =A0 =A0 =A0 =A0 =A0cd =A0 =A0 =A0 =A0 =A0 =A0 =A0# CD device =A0 =A0 =A0 =A0 =A0pass =A0 =A0 =A0 =A0 =A0 =A0# Passthrough device = (direct SCSI access) # Sound options =A0 =A0 =A0 =A0 SND_PCM_64 =A0 =A0 =A0# Process 32bit samples throu= gh 64bit integer Also I've commented out 'debug.witness.watch=3D0' in /boot/loader.conf before booting this kernel. > >> =A0ddb(4) capture buffer on, issued recommended ddb commands, then calle= d >> =A0'doadump'; then rebooted and extracted capture buffer contents from >> =A0the crashdump using ddb(8). > Due to missed options, we cannot trace the lock chains and see why the > issue happens. =A0Please specify what options are still missing in my config. >> >> =A0db> ps >> >> =A0 =A0pid =A0ppid =A0pgrp =A0 uid =A0 state =A0 wmesg =A0 =A0 wchan =A0= =A0cmd >> =A0 2143 =A01846 =A02143 =A0 =A0 0 =A0S+ =A0 =A0 =A0ufs =A0 =A0 =A00xcd4= 66dc8 sync >> =A0 2131 =A0 =A0 1 =A01911 =A01000 =A0S =A0 =A0 =A0 ufs =A0 =A0 =A00xcd4= 66dc8 initial thread > ... >> =A0 =A0 18 =A0 =A0 0 =A0 =A0 0 =A0 =A0 0 =A0SL =A0 =A0 =A0ufs =A0 =A0 = =A00xcd466dc8 [syncer] > .... >> =A0Tracing command syncer pid 18 tid 100067 td 0xc7383000 >> =A0sched_switch(c7383000,0,104,191,e660f504,...) at 0xc05292c0 =3D sched= _switch+0x330 >> =A0mi_switch(104,0,c0745097,1eb,50,...) at 0xc0510680 =3D mi_switch+0x20= 0 >> =A0sleepq_switch(c7383000,0,c0745097,260,50,...) at 0xc054028f =3D >> =A0sleepq_switch+0x15f >> =A0sleepq_wait(cd466dc8,50,c07393ba,4,0,...) at 0xc0540f63 =3D sleepq_wa= it+0x63 >> =A0__lockmgr_args(cd466dc8,80100,cd466e30,0,0,...) at 0xc04f087f =3D >> =A0__lockmgr_args+0xb1f >> =A0ffs_lock(e732cb34,c054944b,c0763314,80100,cd466d70,...) at 0xc067e9f1 >> =A0=3D ffs_lock+0xa1 >> =A0VOP_LOCK1_APV(c07a47c0,e732cb34,c73830a4,c07b7a20,cd466d70,...) at >> =A00xc070c945 =3D VOP_LOCK1_APV+0xb5 >> =A0_vn_lock(cd466d70,80100,c074e6ce,82b,4,...) at 0xc05a2fd8 =3D _vn_loc= k+0x78 >> =A0vget(cd466d70,80100,c7383000,39a,c763b870,...) at 0xc05969db =3D vget= +0xbb >> =A0qsync(c763b870,0,c0761f41,552,4,...) at 0xc0687db7 =3D qsync+0x197 >> =A0ffs_sync(c763b870,3,c074e6ce,d6b,c763b870,...) at 0xc067a139 =3D ffs_= sync+0x349 >> =A0sync_fsync(e732cc7c,c0772c36,c76b8778,e732cc7c,c76b86b8,...) at >> =A00xc059812f =3D sync_fsync+0x18f >> =A0VOP_FSYNC_APV(c0799bc0,e732cc7c,c074e6ce,6a5,c7383000,...) at >> =A00xc070a545 =3D VOP_FSYNC_APV+0xc5 >> =A0sync_vnode(c093b618,c093b604,3e8,6d4,4e20,...) at 0xc059840b =3D sync= _vnode+0x16b >> =A0sched_sync(0,e732cd38,c073bc0b,343,c759f7f8,...) at 0xc0598753 =3D >> =A0sched_sync+0x273 >> =A0fork_exit(c05984e0,0,e732cd38) at 0xc04dddf8 =3D fork_exit+0xb8 >> =A0fork_trampoline() at 0xc06d6b80 =3D fork_trampoline+0x8 >> =A0--- trap 0, eip =3D 0, esp =3D 0xe732cd70, ebp =3D 0 --- > Do you have custoom kernel build, in particular, are quotas compiled in ? =A0I've posted my config during the initial PR submission, and re-posting updated version now. Yes, QUOTA compiled in. It is turned on for /usr. > What are the drives ? ad0: 152626MB <SAMSUNG SP1614N TM100-30> at ata0-master UDMA100 ad14: 953869MB <Seagate ST31000520AS CC32> at ata7-master UDMA100 SATA 3Gb/= s dmitry@lynx$ mount -t ufs /dev/ad0s3a on / (ufs, local, synchronous) /dev/ad0s3d on /usr (ufs, local, with quotas, soft-updates) /dev/ad14s4a on /databig (ufs, local, soft-updates) > Having to pull the generic information about the problematic system > does not make debugging faster. =A0I'm ready to provide any additional information - just not sure what else is significant for this problem. -- Sincerely, Dmitry nic-hdl: LYNX-RIPE
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?AANLkTikgFovKkHlL612G4DSAPfMzZ6K6-hUMABmLSObw>