From owner-freebsd-stable@FreeBSD.ORG Mon Dec 9 11:55:29 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6B8C7C22 for ; Mon, 9 Dec 2013 11:55:29 +0000 (UTC) Received: from mail-we0-x229.google.com (mail-we0-x229.google.com [IPv6:2a00:1450:400c:c03::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id EA55E132C for ; Mon, 9 Dec 2013 11:55:28 +0000 (UTC) Received: by mail-we0-f169.google.com with SMTP id w61so3354310wes.0 for ; Mon, 09 Dec 2013 03:55:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=hR2AIPUIgHsywWzl8u86krdLHYh6EK58JinMilzT0Rs=; b=NFuNrvmGOGl5AHnQafDHPUgAzQ2szy/91aNSvhlogwkAPd/jLTYS69VULGwn48OWgd FWSEHFOiG7Y+k2xgASRvacYIqDs5QklfC9JRQbd/A452RmlUUR6ZCra9choMbllMkgXv MTrgG6sNYwQFBJFWiarXSMy4v5luB1hTc68EWOOu3a6hD6OG9c5mlmjtpc7H4F4wAIeU Mw2krE+ozTq+jhvYn65Wkhtz9oqm21OJ6DEOwoNlyhz0ACoGBviQf3tFGtdlmdUgaX8Z gMABRtMgg0L9EELHOpAxLhwwlQeECU7lnBSCJgDSthaIpGF6AUkajo6IzER7MfBent/+ hQog== MIME-Version: 1.0 X-Received: by 10.194.62.8 with SMTP id u8mr5593604wjr.68.1386590127310; Mon, 09 Dec 2013 03:55:27 -0800 (PST) Received: by 10.217.89.138 with HTTP; Mon, 9 Dec 2013 03:55:27 -0800 (PST) In-Reply-To: References: <20131206201329.GK59496@kib.kiev.ua> <20131207183044.GA19655@x2.osted.lan> Date: Mon, 9 Dec 2013 13:55:27 +0200 Message-ID: Subject: Re: stable/10 r258317 - UFS lock problem From: =?ISO-8859-1?Q?=D6zkan_KIRIK?= To: Peter Holm Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: Konstantin Belousov , freebsd-stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Dec 2013 11:55:29 -0000 Hi, I compiled GENERIC kernel with; makeoptions DEBUG=3D-g options DDB options INVARIANTS options INVARIANT_SUPPORT options WITNESS options WITNESS_SKIPSPIN options DEBUG_LOCKS options DEBUG_VFS_LOCKS options DIAGNOSTIC When I run portsnap fetch extract, I catched these situations on console: lock order reversal: 1st Oxfffffe0080531888 bufwait (bufwait) =95 ,usr/src/sys/kern/vfs bio.c:3050 2nd Oxfffff8000d48a400 dirhash (dirhash) /usr/src/sys/ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: kdb_trace_self_wrapper() at db_trace_self_wrapper.0x2b/frame Oxfffffe009aaa3700 db_backtrace() at kdb backtrace.0x39/frame Oxfffffe009aaa37b0 witness_checkorder() at witness_checkorder.Oxd23/frame Oxfffffe009aaa3840 _sx_xlock() at _sx xlock.0x75/frame Oxfffffe009aaa3880 ufsdirhash_remove() at ufsdirhash remove.0x47/frame Oxfffffe009aaa38b0 ufs_dirremove() at ufs_dirremove.0x11b/frame Oxfffffe009aaa3900 ufs_rmdir() at ufs_rmdir.Oxe3/frame Oxfffffe009aaa3940 VOP_RMDIR_APV() at VOP RHDIR APV.0x12d/frame Oxfffffe009aaa3970 kern_rmdirat() at kern rmdirat.0x1b8/frame Oxfffffe009aaa3ae0 amd64_syscall() at amd64_syscal1.0x282/frame Oxfffffe009aaa3bf0 Xfast_syscall() at Xfast_syscall.Oxfb/frame Oxfffffe009aaa3bf0 --- syscall (137. FreeBSD ELF64. sys_rmdir). rip =3D Ox80088alfa. rsp =3D Ox7fffffffd998, rbp =3D Ox7fffffffda60 --- "ps" output over ssh is below: # ps ax | grep D+ 29730 0 D+ 0:00.00 tar -xz --numeric-owner -f /var/db/portsnap/files/cafbda170898ff0143598e1363c08fc3ac82ba2f4d43334fc53b= 251dbcd28063.gz -C /usr/ports/games/highmoon/ (bsdtar) On Sun, Dec 8, 2013 at 9:38 PM, =D6zkan KIRIK wrote= : > Hi, > > You're right Peter. Now i am compiling with DDB option. > But > http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kern= eldebug-deadlocks.html page, > dont tell about the "DDB" option. > > Thank you, > > > On Sat, Dec 7, 2013 at 8:30 PM, Peter Holm wrote: > >> On Sat, Dec 07, 2013 at 08:24:32AM +0200, =D6zkan KIRIK wrote: >> > Hi, >> > >> > I'm trying to compile a debug kernel with these options : >> > >> > makeoptions DEBUG=3D-g >> > options INVARIANTS >> > options INVARIANT_SUPPORT >> > options WITNESS >> > options DEBUG_LOCKS >> > options DEBUG_VFS_LOCKS >> > options DIAGNOSTIC >> > >> > While compiling; compilations fails due to this error : >> > /usr/src/sys/kern/subr_vmem.c:1357:1: error: unused function 'vmem_dum= p' >> > [-Werror, -Wunused-function] >> > >> >> Did you forget "options DDB" ? >> >> - Peter >> >> > >> > I'll comment out this function and recompiled kernel. >> > Now, while booting, kernel panics. Error line is below; KDB stack >> > backtrace is attached as PNG screenshot. >> > >> > "panic: mtx_lock_spin: recursed on non-recursive mutex cnput_mtx @ >> > /usr/src/sys/kern/kern_cons.c:500" >> > >> > I think, i hit an other bug. >> > >> > Best wishes, >> > >> > Ozkan KIRIK >> > ePati Information Technologies >> > http://www.epati.com.tr/ >> > >> > >> > On Fri, Dec 6, 2013 at 10:13 PM, Konstantin Belousov < >> kostikbel@gmail.com>wrote: >> > >> > > On Fri, Dec 06, 2013 at 08:10:55PM +0200, ?zkan KIRIK wrote: >> > > > Hi, >> > > > >> > > > I am using FreeBSD 10.0-BETA3 r258317 GENERIC kernel. >> > > > >> > > > I run "portsnap fetch extract". >> > > > While extracting files, extracting operation hangs, and i cant kil= l >> > > process. >> > > > >> > > > Output of : "ps ax | grep D+" ( uninterruptable process list ) >> > > > % ps ax | grep D+ >> > > > 28933 0 D+ 0:00.01 tar -xz --numeric-owner -f >> > > > >> > > >> /var/db/portsnap/files/77d41f10d2832f8450e4e02a4db5c0a6131c97d15076ed0c7= 6e761c9ce58338d.gz >> > > > -C /usr/ports/emulators/qemu/ (bsdtar) >> > > > 29051 1 S+ 0:00.00 grep D+ >> > > > % >> > > > >> > > > >> > > > Because of D flag, process cannot be interrupted even by SIGKILL. >> > > > I cannot remove usr/ports folder while this process on D+ state. >> > > > If I run rm -r /usr/ports, rm process is flagged as D+ also. >> > > > >> > > > I tested this situation on both vmware workstation and different >> real >> > > > hardware installations. ( Sun Fire X4150, HP DL380G4, Sun X3 >> servers. ) >> > > > >> > > > Problem is repeatable. If you install FreeBSD on ZFS root, problem >> > > doesn't >> > > > occur. >> > > > >> > > > >> > > > # mount >> > > > /dev/da0p2 on / (ufs, local, journaled soft-updates) >> > > > devfs on /dev (devfs, local, multilabel) >> > > >> > > See >> > > >> > > >> http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/ker= neldebug-deadlocks.html >> > > >> > _______________________________________________ >> > >