Date: Sat, 07 Mar 2015 10:19:30 +0000 From: Steven Hartland <killing@multiplay.co.uk> To: freebsd-fs@freebsd.org Subject: Re: ZFS Deadlock? Message-ID: <54FAD0B2.8060601@multiplay.co.uk> In-Reply-To: <CAM72HBb0C-DQszMZFH706rP4hJK_V9dJdVzmtSJqqafue2ud9Q@mail.gmail.com> References: <CAM72HBb0C-DQszMZFH706rP4hJK_V9dJdVzmtSJqqafue2ud9Q@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
There are lots of fixes for ZFS issues in 10 so I'd recommend updated to 10.1-RELEASE to see if you issue is already resolved there. On 07/03/2015 03:13, Nick Sivo wrote: > Hi, > > One of our servers occasionally exhibits strange behavior under heavy > IO load. I think, based on the output from procstat -kk -a, it may be > a ZFS or VFS deadlock. Certain operations, including anything > involving the ZFS commands like zfs and zpool will hang. Running ls at > the root of a ZFS filesystem will also hang. Trying to access > snapshots in the .zfs/ folder will hang. None of these hung processes > can be killed. Eventually the machine will panic, if we don't reboot > it first, but that can take days after we start seeing this issue. > Strangely, our primary application (Hacker News) will keep running > without interruption until the panic. > > Details of three occurrences can be found at > https://gist.github.com/kogir/acbd6d0e28ade0ee3aac > > For the ones this month, it's on: > 9.3-RELEASE-p10 FreeBSD 9.3-RELEASE-p10 #0: Tue Feb 24 21:28:03 UTC 2015 > > Those from October of last year were running an earlier 9.3 (exact > version unknown). The same hardware running 9.2 was solid for months > at a time. We never saw this issue on 9.2. > > top output from the dying box right now: > > last pid: 48083; load averages: 0.24, 0.31, 0.27 > 120 processes: 1 running, 119 sleeping > CPU: 5.6% user, 0.0% nice, 1.7% system, 0.2% interrupt, 92.5% idle > Mem: 5722M Active, 249M Inact, 67G Wired, 352K Cache, 51G Free > ARC: 32G Total, 14G MFU, 8824M MRU, 52M Anon, 1800M Header, 7962M Other > Swap: > > I'd show you the zpool configuration, but that would hang. We're not > using L2ARC or deduplication. > > In any case, it's happening more frequently (twice this week), so I'd > like to get to the bottom of it if I can. Does this look like it could > be a filesystem issue? This will undoubtedly happen again. Is there > more information I should try to collect? > > Thanks for your time and ideas/help you throw my way :) > > Best, > Nick > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?54FAD0B2.8060601>