From owner-freebsd-current@FreeBSD.ORG Tue May 14 16:08:47 2013 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id B4768CC2 for ; Tue, 14 May 2013 16:08:47 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received: from SMTP.EU.CITRIX.COM (smtp.eu.citrix.com [46.33.159.39]) by mx1.freebsd.org (Postfix) with ESMTP id 582EF6B2 for ; Tue, 14 May 2013 16:08:46 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.87,552,1363132800"; d="scan'208";a="4572635" Received: from lonpmailmx01.citrite.net ([10.30.203.162]) by LONPIPO01.EU.CITRIX.COM with ESMTP/TLS/RC4-MD5; 14 May 2013 15:59:01 +0000 Received: from Roger-2.local (10.30.249.38) by LONPMAILMX01.citrite.net (10.30.203.162) with Microsoft SMTP Server id 8.3.298.1; Tue, 14 May 2013 17:08:46 +0100 Message-ID: <5192618D.8070501@citrix.com> Date: Tue, 14 May 2013 18:08:45 +0200 From: =?ISO-8859-1?Q?Roger_Pau_Monn=E9?= User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/20130328 Thunderbird/17.0.5 MIME-Version: 1.0 To: Konstantin Belousov Subject: Re: FreeBSD-HEAD gets stuck on vnode operations References: <5190CBEC.5000704@citrix.com> <5190F9A0.3000005@citrix.com> <20130513150018.GL3047@kib.kiev.ua> In-Reply-To: <20130513150018.GL3047@kib.kiev.ua> Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: 7bit Cc: "current@freebsd.org" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 May 2013 16:08:47 -0000 On 13/05/13 17:00, Konstantin Belousov wrote: > On Mon, May 13, 2013 at 04:33:04PM +0200, Roger Pau Monn? wrote: >> On 13/05/13 13:18, Roger Pau Monn? wrote: Thanks for taking a look, >> I would like to explain this a little bit more, the syncer process >> doesn't get blocked on the _mtx_trylock_flags_ call, it just continues >> looping forever in what seems to be an endless loop around >> mnt_vnode_next_active/ffs_sync. Also while in this state there is no >> noticeable disk activity, so I'm unsure of what is happening. > How many CPUs does your VM have ? 7 vCPUs, but I've also seen this issue with 4 and 16 vCPUs. > > The loop you describing means that other thread owns the vnode > interlock. Can you track what this thread does ? E.g. look at the > vp->v_interlock.mtx_lock, which is basically a pointer to the struct > thread owning the mutex, clear low bits as needed. Then you can > inspect the thread and get a backtrace. There are no other threads running, only syncer is running on CPU 1 (see ps in previous email). All other CPUs are idle, and as seen from the ps quite a lot of threads are blocked in vnode related operations, either "*Name Cac", "*vnode_fr" or "*vnode in". I've also attached the output of alllocks in the previous email. > > Does the loop you described stuck on the same vnode during the whole > lock-step time, or is the progress made, possibly slowly ? I'm not sure how to measure "progress", but indeed the syncer process is not locked, it is iterating over mnt_vnode_next_active. > > I suppose that your HEAD is recent. Last commit in my local repository is: Date: Tue, 7 May 2013 12:39:14 +0000 Subject: [PATCH] By request, add an arrow from NetBSD-0.8 to FreeBSD-1.0. While here, add a few more NetBSD versions to the tree itself. Submitted by: Alan Barrett Submitted by: Thomas Klausner