Date: Fri, 7 Jun 2013 00:12:39 +0200 From: mxb <mxb@alumni.chalmers.se> To: Outback Dingo <outbackdingo@gmail.com> Cc: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Re: zpool export/import on failover - The pool metadata is corrupted Message-ID: <016B635E-4EDC-4CDF-AC58-82AC39CBFF56@alumni.chalmers.se> In-Reply-To: <CAKYr3zyPLpLau8xsv3fCkYrpJVzS0tXkyMn4E2aLz29EMBF9cA@mail.gmail.com> References: <D7F099CB-855F-43F8-ACB5-094B93201B4B@alumni.chalmers.se> <CAKYr3zyPLpLau8xsv3fCkYrpJVzS0tXkyMn4E2aLz29EMBF9cA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Then MASTER goes down, CARP on the second node goes MASTER (devd.conf, =
and script for lifting):
root@nfs2:/root # cat /etc/devd.conf
notify 30 {
match "system" "IFNET";
match "subsystem" "carp0";
match "type" "LINK_UP";
action "/etc/zfs_switch.sh active";
};
notify 30 {
match "system" "IFNET";
match "subsystem" "carp0";
match "type" "LINK_DOWN";
action "/etc/zfs_switch.sh backup";
};
root@nfs2:/root # cat /etc/zfs_switch.sh
#!/bin/sh
DATE=3D`date +%Y%m%d`
HOSTNAME=3D`hostname`
ZFS_POOL=3D"jbod"
case $1 in
active)
echo "Switching to ACTIVE and importing ZFS" | mail -s =
''$DATE': '$HOSTNAME' switching to ACTIVE' root
sleep 10
/sbin/zpool import -f jbod
/etc/rc.d/mountd restart
/etc/rc.d/nfsd restart
;;
backup)
echo "Switching to BACKUP and exporting ZFS" | mail -s =
''$DATE': '$HOSTNAME' switching to BACKUP' root
/sbin/zpool export jbod
/etc/rc.d/mountd restart
/etc/rc.d/nfsd restart
;;
*)
exit 0
;;
esac
This works, most of the time, but sometimes I'm forced to re-create =
pool. Those machines suppose to go into prod.
Loosing pool(and data inside it) stops me from deploy this setup.
//mxb
On 6 jun 2013, at 22:06, Outback Dingo <outbackdingo@gmail.com> wrote:
>=20
>=20
>=20
> On Thu, Jun 6, 2013 at 3:24 PM, mxb <mxb@alumni.chalmers.se> wrote:
>=20
> Hello list,
>=20
> I have two-head ZFS setup with external disk enclosure over SAS =
expander.
> This is a failover setup with CARP and devd triggering spool =
export/import.
> One of two nodes is preferred master.
>=20
> Then master is rebooted, devd kicks in as of CARP becomes master and =
the second node picks up ZFS-disks from external enclosure.
> Then master comes back, CARP becomes master, devd kicks in and pool =
gets exported from the second node and imported on the first one.
>=20
> However, I have experienced metadata corruption several times with =
this setup.
> Note, that ZIL(mirrored) resides on external enclosure. Only L2ARC is =
both local and external - da1,da2, da13s2, da14s2
>=20
> root@nfs2:/root # zpool import
> pool: jbod
> id: 17635654860276652744
> state: FAULTED
> status: The pool metadata is corrupted.
> action: The pool cannot be imported due to damaged devices or data.
> see: http://illumos.org/msg/ZFS-8000-72
> config:
>=20
> jbod FAULTED corrupted data
> raidz3-0 ONLINE
> da3 ONLINE
> da4 ONLINE
> da5 ONLINE
> da6 ONLINE
> da7 ONLINE
> da8 ONLINE
> da9 ONLINE
> da10 ONLINE
> da11 ONLINE
> da12 ONLINE
> cache
> da1
> da2
> da13s2
> da14s2
> logs
> mirror-1 ONLINE
> da13s1 ONLINE
> da14s1 ONLINE
>=20
> Any ideas what is going on?
>=20
> Best case scenerio, both nodes tried to import the disks =
simultaneously, split brain condition, or disks appear out of order and =
no using labels, we had a similar situation with geom_multipath, theres =
no quorum disks knowledge yet for FreeBSD using zfs in this =
configuration, we ran it similarly for a while, until we realized =
through research it was bad karma to let carp and devd control nodes =
without a fool proof way to be sure nodes were ready to export/import. =
though you did state
>=20
> "Then master comes back, CARP becomes master, devd kicks in and pool =
gets exported from the second node and imported on the first one." be =
nice to see how you managed that with scripts... even if both nodes =
booted simultaneously their both going to "fight" for master and try to =
import that pool.
>=20
>=20
> //mxb
>=20
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
>=20
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?016B635E-4EDC-4CDF-AC58-82AC39CBFF56>