From owner-freebsd-fs@FreeBSD.ORG Thu Apr 4 16:05:22 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id D9C78226 for ; Thu, 4 Apr 2013 16:05:22 +0000 (UTC) (envelope-from toasty@dragondata.com) Received: from mail-ob0-x234.google.com (mail-ob0-x234.google.com [IPv6:2607:f8b0:4003:c01::234]) by mx1.freebsd.org (Postfix) with ESMTP id A297CD77 for ; Thu, 4 Apr 2013 16:05:22 +0000 (UTC) Received: by mail-ob0-f180.google.com with SMTP id wo10so2753441obc.11 for ; Thu, 04 Apr 2013 09:05:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dragondata.com; s=google; h=x-received:content-type:mime-version:subject:from:in-reply-to:date :cc:content-transfer-encoding:message-id:references:to:x-mailer; bh=TQFP4tDfzlXYOh85U2QWmdG8n/DzbNIzlgZPozIVOlI=; b=LAEKpbS7Dv799DK57ow32lY+Q1xaMvdnF+J0nQzYYm1tuPImUg+zIiHsISMCOoxrYJ c9guFAv6r8xJ/LRM5zoG41vLclHd+FGJ8NQs68Vgyt9BmTOje4Os7IOV48LwYHXSdg2S +Ro9tQAWnliW4tjC5QlQ4z3eSGHAhusoMbrmc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:content-type:mime-version:subject:from:in-reply-to:date :cc:content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=TQFP4tDfzlXYOh85U2QWmdG8n/DzbNIzlgZPozIVOlI=; b=NMS2tKJbtArUEwRv0rg+CiBHrzn0zNmsbVDQqw+AQvSqWUVQQdd7U80uho4n1qIPgw cjDapFV7yAz2on3kuPQH2ohfQ9diKiPkPD1YnO6DDST1CeHCds0wd1pbdiziCepJkFkm y8xoOhqAwqsOCgyaU94/sCk/PBSfngIt74IIKSfsMyKkBiZ3DZP1UzKnVLbHCXm6AWOV /U5ntXUYfT0ITuA7nfie3DP+5A8Lue6ccvrfMuUNPg9AHrRSvFJKTQHIUC3M8CbuMRf0 T0hrhpiS34vAfEHMNM1OTvgZxGBT5ep4LBQlM/X71n7My7Lv1yDPeB/WoyStwkuNC6iJ ocyw== X-Received: by 10.60.3.200 with SMTP id e8mr4777121oee.94.1365091522044; Thu, 04 Apr 2013 09:05:22 -0700 (PDT) Received: from vpn132.rw1.your.org (vpn132.rw1.your.org. [204.9.51.132]) by mx.google.com with ESMTPS id qk4sm6964383obc.5.2013.04.04.09.05.19 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 04 Apr 2013 09:05:20 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: kern/177536: zfs livelock (deadlock) with high write-to-disk load From: Kevin Day In-Reply-To: <201304041540.r34Fe1Ka057203@freefall.freebsd.org> Date: Thu, 4 Apr 2013 11:05:17 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: References: <201304041540.r34Fe1Ka057203@freefall.freebsd.org> To: Andriy Gapon X-Mailer: Apple Mail (2.1503) X-Gm-Message-State: ALoCoQnO4Lb8JRoiOr7RkQRXqi+PHG844F3MwLzLcLTwVTN6a6EC8/kmbLFa7DO1wpEQo/Bi3APl Cc: freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Apr 2013 16:05:22 -0000 I'm not sure if I'm experiencing the same thing, but I'm chiming-in in = case this helps someone. We have a server that's configured as such: 9.1-RELEASE amd64, dual Opteron CPU, 64GB of memory. nVidia nForce MCP55 SATA controller * 1x 240GB SSD used for ZFS l2arc mps LSI 9207-8e (LSISAS2308 chip> * Connected to 4 external enclosures, each with 24 3TB drives for a = total of 96 3TB drives, running ZFS in a JBOD configuration twa 3ware 9650SE-12i * Connected 1:1 (no expander) to 12 internal 500GB drives, running UFS = for / and a secondary UFS filesystem When there's very heavy write load to the giant ZFS filesystem (>2gbps = of total incoming data being written), eventually I reach some kind of = deadlock, where I can't do anything that touches any of the block = devices. Processes that attempt to access any filesystem (ZFS or UFS) will get = stuck in 'ufs', 'getblk', 'vnread', or 'tx->tx'. A shell is still = responsive, and I can run commands as long as they're cached. Trying to = run something that wasn't already cached prior to the problem will hang = that shell. 'gstat' shows that most(all?) of the disk devices have outstanding = requests waiting, but a busy percentage of 0% and no activity happening. This only seems to happen under heavy ZFS writes. Heavy ZFS reads, or = heavy UFS writes do not trigger this. Slowing down the ZFS writes will = prevent the problem from occurring. At first I thought this was a controller hang, but after seeing that = devices on three different controllers are all ending up stuck with = outstanding requests is making me a bit confused as to how this could = even happen. Nothing gets logged to the console when this happens.=20 Things I've tried already: 1) Remove the SSD entirely 2) zfs set sync=3Ddisabled fs 3) Letting the system wait (90 minutes) to see if this recovers. 4) Swapped the motherboard/CPUs/memory for an identically configured = system 5) Switched from an LSI 9280 (mpt) to an LSI 9207 (mps) 6) Updated firmware on the storage cards, updated the BIOS on the = motherboard Fair disclosure, these Opterons do have the TLB bug (AMD errata 298), = but the BIOS has a workaround for it which is enabled. We've got dozens = of identical systems to this and aren't experiencing any weird hangs or = anything elsewhere, so I'm assuming this is not it. The problem is that this is a production system that doesn't give me a = lot of time for troubleshooting before I'm forced to reboot it. I'm = going to try to get procstat to stay in the cache so that next time this = happens I can try running it. If there's anything else anyone would like = me to capture when this happens again I'm happy to try.=20 -- Kevin