From owner-freebsd-fs@FreeBSD.ORG Mon Aug 22 11:02:16 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 20271106571A for ; Mon, 22 Aug 2011 11:02:16 +0000 (UTC) (envelope-from luke@digital-crocus.com) Received: from mail.digital-crocus.com (node2.digital-crocus.com [91.209.244.128]) by mx1.freebsd.org (Postfix) with ESMTP id CDCFD8FC08 for ; Mon, 22 Aug 2011 11:02:15 +0000 (UTC) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=dkselector; d=hybrid-logic.co.uk; h=Received:Received:Subject:From:Reply-To:To:In-Reply-To:References:Content-Type:Organization:Date:Message-ID:Mime-Version:X-Mailer:Content-Transfer-Encoding:X-Spam-Score:X-Digital-Crocus-Maillimit:X-Authenticated-Sender:X-Complaints:X-Admin:X-Abuse; b=IeMj7+jmeGDbthx1M3z1l0ERikzEJRvOE0KvIsvvMZG+xtN061ZYZbExJvjYMyc44WI1kgyz7ISWP7x8KbYqjLKoRw330lHUbRDwq7SaEj6JxGfmHICsnqIdbyW65g2g; Received: from luke by mail.digital-crocus.com with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1QvSFv-00032h-HY for freebsd-fs@freebsd.org; Mon, 22 Aug 2011 12:01:23 +0100 Received: from 127cr.net ([78.105.122.99] helo=[192.168.1.23]) by mail.digital-crocus.com with esmtpa (Exim 4.69 (FreeBSD)) (envelope-from ) id 1QvSFv-00032b-A8 for freebsd-fs@freebsd.org; Mon, 22 Aug 2011 12:01:23 +0100 From: Luke Marsden To: freebsd-fs@freebsd.org In-Reply-To: <201108221015.p7MAFHpi048670@freefall.freebsd.org> References: <201108221015.p7MAFHpi048670@freefall.freebsd.org> Content-Type: text/plain; charset="UTF-8" Organization: Hybrid Web Cluster Date: Mon, 22 Aug 2011 12:02:11 +0100 Message-ID: <1314010931.3477.138.camel@pow> Mime-Version: 1.0 X-Mailer: Evolution 2.32.2 Content-Transfer-Encoding: 7bit X-Spam-Score: -1.0 X-Digital-Crocus-Maillimit: done X-Authenticated-Sender: luke X-Complaints: abuse@digital-crocus.com X-Admin: admin@digital-crocus.com X-Abuse: abuse@digital-crocus.com (Please include full headers in abuse reports) Subject: Re: kern/157728: [zfs] zfs (v28) incremental receive may leave behind temporary clones X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: luke@hybrid-logic.co.uk List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Aug 2011 11:02:16 -0000 On Mon, 2011-08-22 at 10:15 +0000, mm@FreeBSD.org wrote: > Synopsis: [zfs] zfs (v28) incremental receive may leave behind temporary clones > > State-Changed-From-To: open->closed > State-Changed-By: mm > State-Changed-When: Mon Aug 22 10:15:16 UTC 2011 > State-Changed-Why: > Resolved. Thanks! Brilliant, thanks for fixing this! Do you have any thoughts about what might have caused the other issue I reported, the deadlock? From my email of the 15th July (mfsbsd-se-8.2-zfsv28-amd64 19.06.2011): The biggest issue was a DEADLOCK which occurs quite reliably with a given sequence of events in short succession, on a chroot filesystem with many snapshots and a MySQL socket and nullfs mounts inside it: 1. Force unmount the nullfs mounts which are mounted on top of it 2. Close the MySQL socket in /tmp 3. Force unmount the actual filesystem (even if there are open FDs) 4. 'zfs rename' the filesystem into our 'trash' filesystem (which I understand consists of a clone, promote and destroy) The entire ZFS subsystem then hangs on any new I/O. Here is a procstat of the zfs rename process which hangs after the force unmount: 25674 100871 zfs initial thread mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 txg_wait_synced+0x85 dsl_sync_task_group_wait+0x128 dsl_sync_task_do+0x54 dsl_dir_rename+0x8f dsl_dataset_rename+0x272 zfsdev_ioctl+0xe6 devfs_ioctl_f+0x7b kern_ioctl +0x102 ioctl+0xfd syscallenter+0x1e5 syscall+0x4b Xfast_syscall+0xe2 Unfortunately it's not easy to reproduce, it only seems to happen in an environment which is under load with a lot of datasets and a lot of zfs operations happening concurrently on other datasets. I spent two days trying to reproduce it in self-contained test environments but had no luck, so I'm now reporting it anyway. -- Best Regards, Luke Marsden CTO, Hybrid Logic Ltd. Web: http://www.hybrid-cluster.com/ Hybrid Web Cluster - cloud web hosting Mobile: +1-415-449-1165 (US) / +447791750420 (UK)