From owner-freebsd-stable@FreeBSD.ORG Wed Oct 15 18:59:38 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4CB1E106569B for ; Wed, 15 Oct 2008 18:59:38 +0000 (UTC) (envelope-from heliocentric@gmail.com) Received: from mail-gx0-f16.google.com (mail-gx0-f16.google.com [209.85.217.16]) by mx1.freebsd.org (Postfix) with ESMTP id E43BF8FC1C for ; Wed, 15 Oct 2008 18:59:37 +0000 (UTC) (envelope-from heliocentric@gmail.com) Received: by gxk9 with SMTP id 9so6306238gxk.19 for ; Wed, 15 Oct 2008 11:59:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=pyrk8lWefSWWQE2ZKHO12VFa+kii3ikEuISInyP8dDg=; b=I4lIySzjMay74lgYEnINoq+CC+m9VBLfQFWZDTmkQfV6aorLx8EtRqNj/i2RwHHf7v Pr8qyeYHDy7HNkRdvq8hIiszLxkjCuagsfBf8gy70c3+o25YjyUvQrnu37UA2wXv95XJ LnTJ2MFgWeHZWcN5UB9z0z6qZmKNZuqmissTM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=ut6LiRGnzLAjTYrNZgJe81/qjXzZbbjwlc4zLs4YtlefhO4Kfwjk2m1OCAX7W7zyGU RxW0Zvmg7zodPI66a7MMo/0sNfMs6CQZiFyS2lYoyJOTqQhqlMJ4IZQahBZCsAt2YPH0 4besoDdKEX9Ch5qPhusGDshRLi7khOLN8XtUQ= Received: by 10.90.32.14 with SMTP id f14mr1548509agf.94.1224097176633; Wed, 15 Oct 2008 11:59:36 -0700 (PDT) Received: by 10.90.100.18 with HTTP; Wed, 15 Oct 2008 11:59:36 -0700 (PDT) Message-ID: Date: Wed, 15 Oct 2008 14:59:36 -0400 From: heliocentric@gmail.com To: "Peter Jeremy" In-Reply-To: <20081015105338.GA75226@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20081015082428.GE26536@server.vk2pj.dyndns.org> <20081015083538.GA72190@icarus.home.lan> <20081015085843.GF26536@server.vk2pj.dyndns.org> <20081015090848.GA72852@icarus.home.lan> <20081015094809.GG26536@server.vk2pj.dyndns.org> <20081015105338.GA75226@icarus.home.lan> Cc: freebsd-stable@freebsd.org Subject: Re: System hanging during dump X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Oct 2008 18:59:38 -0000 >> Based on progress reports from both dump and my fifo process, the >> snapshot removal began about 10 hours _after_ the system froze >> (during this time, dump wrote about 143GB). Given the timeline, >> it's fairly clear that neither mksnap_ffs nor the 'rm snapshot' >> were running at the time the system froze. I am therefore quite >> confident that the problem I saw is not related to either creation >> or removal of snapshots. >> >> I have been using FreeBSD snapshots for many years and am quite >> familiar with their quirks. I have never seen this particular >> problem before. (And FWIW, I _am_ using Doug Ambrisko's patch to >> ffs_snapshot.c). After upgrading my backup server to 8-CURRENT, I've noticed LORs during write activity that happens while a snapshot is being read. I believe these are responsible for some of the deadlocks in a subset of filesystem operations during some usages of snapshots I've encountered, but I can't confirm (as I am inexperienced at deeply debugging locking issues). There is also two LORs involving the same locks that occur during a snapshot removal, which is most likely the cause of the 'hang' that seems to be reported for that case. Perhaps a 'show locks' with an 8-CURRENT kernel with WITNESS enabled will shed light on the problem? As most of the filesystem locking doesn't use lockmgr in 7-STABLE, it would be silent with that kernel.