From owner-freebsd-fs@FreeBSD.ORG Thu Apr 4 16:21:08 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 1BC4776B for ; Thu, 4 Apr 2013 16:21:08 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 6603EE7F for ; Thu, 4 Apr 2013 16:21:07 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id TAA10872; Thu, 04 Apr 2013 19:21:05 +0300 (EEST) (envelope-from avg@FreeBSD.org) Message-ID: <515DA870.6050006@FreeBSD.org> Date: Thu, 04 Apr 2013 19:21:04 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130313 Thunderbird/17.0.4 MIME-Version: 1.0 To: Kevin Day Subject: Re: kern/177536: zfs livelock (deadlock) with high write-to-disk load References: <201304041540.r34Fe1Ka057203@freefall.freebsd.org> <515DA525.3020006@FreeBSD.org> In-Reply-To: X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Apr 2013 16:21:08 -0000 on 04/04/2013 19:12 Kevin Day said the following: > > On Apr 4, 2013, at 11:07 AM, Andriy Gapon wrote: > >> on 04/04/2013 19:05 Kevin Day said the following: [a lot] >> >> One link: https://wiki.freebsd.org/AvgZfsDeadlockDebug > > > Sorry, should have mentioned i've seen this. > > I've tried procstat, but i've failed at keeping it cached long enough to use it > when this happens. If I try running it, the shell gets stuck in 'ufs'. I'm > going to add a cron job to just run procstat periodically so hopefully i can > run it without it needing to touch the disks. Well, stuck ufs points towards to the storage subsystem. You can create a memory disk+fs (see mdconfig, mdmfs) and place some tools there. That way you may be able to get more info and also check if it's VFS or some other common that gets stuck or if it's the real storage indeed. > I built a more debug friendly kernel and tried to drop to ddb when this > happened, but it didn't provide anything useful. When I hit 'enter' on > 'alltrace' it hard locked without printing anything. > > I haven't been able to trigger a core dump, and this system has no serial or > firewire ports for live debugging. > > -- Andriy Gapon