From owner-freebsd-fs@FreeBSD.ORG Sat Mar 7 10:19:48 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2E78ABCA for ; Sat, 7 Mar 2015 10:19:48 +0000 (UTC) Received: from mail-wi0-f172.google.com (mail-wi0-f172.google.com [209.85.212.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B8AFC9E8 for ; Sat, 7 Mar 2015 10:19:46 +0000 (UTC) Received: by wiwl15 with SMTP id l15so8896023wiw.4 for ; Sat, 07 Mar 2015 02:19:39 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=s4yYymKU/BVWcCsl/8d47iIhlkMAYbn1rZ8Pv/sXg00=; b=LLJgFbp8Z60+r/rCA0pys+ty+FWfEcZ73296AvgjLhjbbXX97J7n1aAPr9wNRbHgfj gTW8vfnqVAHzoFb8YVggFUbFZPBW0lLawU7/1Sd1uZJf8zMZNsiOfBdXuleuptAx8G3F dKGKDqR/GPUb/3NPecu/Uch3BtdpIhrJ8s4VX6lG566fr2L+jd5meFQl5lMeueIR37HA Ih44m+UqrULN//Hf6sydrQtfguh6NLxy3Lm2Ob6DeF7XUGzQzdFIxh05BHMTszuHtvuI nOpHfMFWwSgg3SaOCN3eA6XW91dFCD81BrSCXoJJAE0VpHguH3N6OIl9O5xOwkOtwtB0 bvWg== X-Gm-Message-State: ALoCoQmruR6BOK89gAYSlcmGDgE3VqV5xqmvdwrUwiM01OKeFO4lMpgeFab6DYH9kwGVLaXOrev+ X-Received: by 10.180.39.33 with SMTP id m1mr41698981wik.26.1425723579497; Sat, 07 Mar 2015 02:19:39 -0800 (PST) Received: from [10.10.1.68] (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by mx.google.com with ESMTPSA id ka1sm18774143wjc.2.2015.03.07.02.19.38 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 07 Mar 2015 02:19:38 -0800 (PST) Message-ID: <54FAD0B2.8060601@multiplay.co.uk> Date: Sat, 07 Mar 2015 10:19:30 +0000 From: Steven Hartland User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: ZFS Deadlock? References: In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Mar 2015 10:19:48 -0000 There are lots of fixes for ZFS issues in 10 so I'd recommend updated to 10.1-RELEASE to see if you issue is already resolved there. On 07/03/2015 03:13, Nick Sivo wrote: > Hi, > > One of our servers occasionally exhibits strange behavior under heavy > IO load. I think, based on the output from procstat -kk -a, it may be > a ZFS or VFS deadlock. Certain operations, including anything > involving the ZFS commands like zfs and zpool will hang. Running ls at > the root of a ZFS filesystem will also hang. Trying to access > snapshots in the .zfs/ folder will hang. None of these hung processes > can be killed. Eventually the machine will panic, if we don't reboot > it first, but that can take days after we start seeing this issue. > Strangely, our primary application (Hacker News) will keep running > without interruption until the panic. > > Details of three occurrences can be found at > https://gist.github.com/kogir/acbd6d0e28ade0ee3aac > > For the ones this month, it's on: > 9.3-RELEASE-p10 FreeBSD 9.3-RELEASE-p10 #0: Tue Feb 24 21:28:03 UTC 2015 > > Those from October of last year were running an earlier 9.3 (exact > version unknown). The same hardware running 9.2 was solid for months > at a time. We never saw this issue on 9.2. > > top output from the dying box right now: > > last pid: 48083; load averages: 0.24, 0.31, 0.27 > 120 processes: 1 running, 119 sleeping > CPU: 5.6% user, 0.0% nice, 1.7% system, 0.2% interrupt, 92.5% idle > Mem: 5722M Active, 249M Inact, 67G Wired, 352K Cache, 51G Free > ARC: 32G Total, 14G MFU, 8824M MRU, 52M Anon, 1800M Header, 7962M Other > Swap: > > I'd show you the zpool configuration, but that would hang. We're not > using L2ARC or deduplication. > > In any case, it's happening more frequently (twice this week), so I'd > like to get to the bottom of it if I can. Does this look like it could > be a filesystem issue? This will undoubtedly happen again. Is there > more information I should try to collect? > > Thanks for your time and ideas/help you throw my way :) > > Best, > Nick > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"