From owner-freebsd-current@FreeBSD.ORG  Tue May 14 17:15:50 2013
Return-Path: <owner-freebsd-current@FreeBSD.ORG>
Delivered-To: current@freebsd.org
Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115])
 by hub.freebsd.org (Postfix) with ESMTP id 3962EDB1
 for <current@freebsd.org>; Tue, 14 May 2013 17:15:50 +0000 (UTC)
 (envelope-from roger.pau@citrix.com)
Received: from SMTP.EU.CITRIX.COM (smtp.eu.citrix.com [46.33.159.39])
 by mx1.freebsd.org (Postfix) with ESMTP id D08B7BE6
 for <current@freebsd.org>; Tue, 14 May 2013 17:15:49 +0000 (UTC)
X-IronPort-AV: E=Sophos;i="4.87,552,1363132800"; 
   d="scan'208";a="4575826"
Received: from lonpmailmx01.citrite.net ([10.30.203.162])
 by LONPIPO01.EU.CITRIX.COM with ESMTP/TLS/RC4-MD5; 14 May 2013 17:06:01 +0000
Received: from Roger-2.local (10.30.249.38) by LONPMAILMX01.citrite.net
 (10.30.203.162) with Microsoft SMTP Server id 8.3.298.1; Tue, 14 May 2013
 18:15:48 +0100
Message-ID: <51927143.4080102@citrix.com>
Date: Tue, 14 May 2013 19:15:47 +0200
From: =?ISO-8859-1?Q?Roger_Pau_Monn=E9?= <roger.pau@citrix.com>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7;
 rv:17.0) Gecko/20130328 Thunderbird/17.0.5
MIME-Version: 1.0
To: Konstantin Belousov <kostikbel@gmail.com>
Subject: Re: FreeBSD-HEAD gets stuck on vnode operations
References: <5190CBEC.5000704@citrix.com> <5190F9A0.3000005@citrix.com>
 <20130513150018.GL3047@kib.kiev.ua> <5192618D.8070501@citrix.com>
 <20130514163149.GS3047@kib.kiev.ua>
In-Reply-To: <20130514163149.GS3047@kib.kiev.ua>
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 7bit
Cc: "current@freebsd.org" <current@freebsd.org>
X-BeenThere: freebsd-current@freebsd.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: Discussions about the use of FreeBSD-current
 <freebsd-current.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-current>, 
 <mailto:freebsd-current-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-current>
List-Post: <mailto:freebsd-current@freebsd.org>
List-Help: <mailto:freebsd-current-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-current>,
 <mailto:freebsd-current-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 14 May 2013 17:15:50 -0000

On 14/05/13 18:31, Konstantin Belousov wrote:
> On Tue, May 14, 2013 at 06:08:45PM +0200, Roger Pau Monn? wrote:
>> On 13/05/13 17:00, Konstantin Belousov wrote:
>>> On Mon, May 13, 2013 at 04:33:04PM +0200, Roger Pau Monn? wrote:
>>>> On 13/05/13 13:18, Roger Pau Monn? wrote:
>>
>> Thanks for taking a look,
>>
>>>> I would like to explain this a little bit more, the syncer process
>>>> doesn't get blocked on the _mtx_trylock_flags_ call, it just continues
>>>> looping forever in what seems to be an endless loop around
>>>> mnt_vnode_next_active/ffs_sync. Also while in this state there is no
>>>> noticeable disk activity, so I'm unsure of what is happening.
>>> How many CPUs does your VM have ?
>>
>> 7 vCPUs, but I've also seen this issue with 4 and 16 vCPUs.
>>
>>>
>>> The loop you describing means that other thread owns the vnode
>>> interlock. Can you track what this thread does ? E.g. look at the
>>> vp->v_interlock.mtx_lock, which is basically a pointer to the struct
>>> thread owning the mutex, clear low bits as needed. Then you can
>>> inspect the thread and get a backtrace.
>>
>> There are no other threads running, only syncer is running on CPU 1 (see
>> ps in previous email). All other CPUs are idle, and as seen from the ps
>> quite a lot of threads are blocked in vnode related operations, either
>> "*Name Cac", "*vnode_fr" or "*vnode in". I've also attached the output
>> of alllocks in the previous email.
> This is not useful.  You need to look at the mutex which fails the
> trylock operation in the mnt_vnode_next_active(), see who owns it,
> and then 'unwind' the locking dependencies from there.

Sorry, now I get it, let's see if I can find the locked vnodes and the
thread that owns them...