From owner-freebsd-stable@FreeBSD.ORG  Sun Nov 30 01:05:50 2008
Return-Path: <owner-freebsd-stable@FreeBSD.ORG>
Delivered-To: freebsd-stable@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id C3C2C1065673;
	Sun, 30 Nov 2008 01:05:50 +0000 (UTC)
	(envelope-from petefrench@ticketswitch.com)
Received: from constantine.ticketswitch.com (constantine.ticketswitch.com
	[IPv6:2002:57e0:1d4e:1::3])
	by mx1.freebsd.org (Postfix) with ESMTP id 8993C8FC1C;
	Sun, 30 Nov 2008 01:05:50 +0000 (UTC)
	(envelope-from petefrench@ticketswitch.com)
Received: from dilbert.rattatosk ([10.64.50.6] helo=dilbert.ticketswitch.com)
	by constantine.ticketswitch.com with esmtps (TLSv1:AES256-SHA:256)
	(Exim 4.69 (FreeBSD)) (envelope-from <petefrench@ticketswitch.com>)
	id 1L6aku-0000ZR-Rc; Sun, 30 Nov 2008 01:05:48 +0000
Received: from petefrench by dilbert.ticketswitch.com with local (Exim 4.69
	(FreeBSD)) (envelope-from <petefrench@ticketswitch.com>)
	id 1L6aku-000NAk-Q1; Sun, 30 Nov 2008 01:05:48 +0000
To: pjd@FreeBSD.org
In-Reply-To: <20081130000059.GC1494@garage.freebsd.pl>
Message-Id: <E1L6aku-000NAk-Q1@dilbert.ticketswitch.com>
From: Pete French <petefrench@ticketswitch.com>
Date: Sun, 30 Nov 2008 01:05:48 +0000
Cc: freebsd-stable@freebsd.org
Subject: Re: Curious failure of ZFS snapshots
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>, 
	<mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable>
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
	<mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sun, 30 Nov 2008 01:05:50 -0000

> It's not file system on-disk structure fault, as far as I understand,
> because reboot fixes it. I belive it's how you access the snapshots.

I was about to say "a reboot doesnt fix it for me" - as I swear i tried
that before, but I have discovered you are right, as I just rebooted the
server and it is fine.

Here is what I am doing - this script is run with an argument '7am' or '7pm'
once per day. the mysql database is a slave replication from a master,
so there is a continuous trickle of data into it. The symbolic
links are there so you can connect to the mysql server and access
'xxx-7am' or 'xxx-7pm' to get a previous version of database 'xxx'.
In case its not obvious, the filesystem 'tank/zfs' is mounted on the
director '/var/db/mysql'. If you run this for a few cycles it should
preseumably break for you too.

-pete.

-------------------------------------------------------------------

#!/bin/sh

SUFFIX=$1
FS_NAME=tank/mysql
DB_DIR=/var/db/mysql

# First we should make sure we are root
if test `/usr/bin/whoami` != "root"
then
        /bin/echo You must be root to run this script
        exit
fi

# Check that we have a suffix
if test "${SUFFIX}" = ""
then
        /bin/echo "usage: $0 <suffix>"
        exit
fi

# Stop mysql, unlink the links and destroy the snapshot
/usr/local/etc/rc.d/mysql-server stop > /dev/null
/bin/rm -f ${DB_DIR}/*-${SUFFIX}
/sbin/zfs destroy ${FS_NAME}@${SUFFIX}

# Create the new (dated) snapshot
/bin/date > ${DB_DIR}/SNAPSHOT_DATE
/sbin/zfs snapshot ${FS_NAME}@${SUFFIX}
/bin/rm -f ${DB_DIR}/SNAPSHOT_DATE

# Find all mysql subdirectories and link them
/usr/bin/find ${DB_DIR}/ -name '[a-z]*' -type d -maxdepth 1 -print | while read DIR
do
        BASE_DIR=`/usr/bin/basename ${DIR}`
        /bin/ln -s ${DB_DIR}/.zfs/snapshot/${SUFFIX}/${BASE_DIR} ${DIR}-${SUFFIX}
done

# Restart mysql
/usr/local/etc/rc.d/mysql-server start > /dev/null