Date: Wed, 3 Mar 2010 16:31:40 -0800 From: Freddie Cash <fjwcash@gmail.com> To: fs@freebsd.org Subject: HAST: split-brain -- how to force one side to become primary? Message-ID: <b269bc571003031631s1c4cbde1rb3f54b9189d95ea8@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
According to the wiki, when a split-brain situation arises, I should be able
to stop hastd on one side, write changes to the /dev/hast/* providers on the
primary to increment the localcnt valye, and then bring up the secondary
hastd. The locacnt/remotecnt values will be different, and everything will
start to re-sync.
However, this doesn't seem to work. Or, maybe I'm not doing things right to
make it work. Or maybe I've completely misunderstood how it all works.
(Nah, that can never happen. roll-eyes) :)
/dev/hast/* is used to form a raidz1 vdev as part of pool "hapool". There's
a single 1 GB zvol created, that is exported via iSCSI (net/istgt). I can
mount the iSCSI disk on a Linux client, partition it, format it using XFS,
and write data to it. Using only hast2 node as primary, I've written out 10
MB of new data, and verified that the data is there via "zfs list" on hast2,
and multiple mount/unmount cycles on the client. Yet localcnt never
increments beyond 1 (remotecnt is 0).
Is there a way to forcibly increment localcnt on one node, so that bringing
up hastd on the other node will correctly come up as secondary, and start a
sync?
Or, do I have to manually re-do the HAST setup on one side? Or zero out the
base/physical disk underneath HAST?
/etc/hast.conf (only listen line is different between nodes):
# Global section
control /var/run/hastctl
listen 172.20.0.1
replication memsync
# Resource section
resource disk01 {
on hast1 {
local /dev/label/disk01
remote 172.20.0.2
}
on hast2 {
local /dev/label/disk01
remote 172.20.0.1
}
}
resource disk02 {
on hast1 {
local /dev/label/disk02
remote 172.20.0.2
}
on hast2 {
local /dev/label/disk02
remote 172.20.0.1
}
}
resource disk03 {
on hast1 {
local /dev/label/disk03
remote 172.20.0.2
}
on hast2 {
local /dev/label/disk03
remote 172.20.0.1
}
}
resource disk04 {
on hast1 {
local /dev/label/disk04
remote 172.20.0.2
}
on hast2 {
local /dev/label/disk04
remote 172.20.0.1
}
}
hastctl dump on hast1:
resource: disk01
datasize: 2147478528
extentsize: 2097152
keepdirty: 64
localoff: 4608
resuid: 1224151284752404553
localcnt: 1
remotecnt: 0
prevrole: primary
resource: disk02
datasize: 2147478528
extentsize: 2097152
keepdirty: 64
localoff: 4608
resuid: 10884849062207686761
localcnt: 1
remotecnt: 0
prevrole: primary
resource: disk03
datasize: 2147478528
extentsize: 2097152
keepdirty: 64
localoff: 4608
resuid: 14443609578994823508
localcnt: 1
remotecnt: 0
prevrole: primary
resource: disk04
datasize: 2147478528
extentsize: 2097152
keepdirty: 64
localoff: 4608
resuid: 1365498106518463540
localcnt: 1
remotecnt: 0
prevrole: primary
hastctl dump on hast2:
resource: disk01
datasize: 2147478528
extentsize: 2097152
keepdirty: 64
localoff: 4608
resuid: 1224151284752404553
localcnt: 1
remotecnt: 0
prevrole: primary
resource: disk02
datasize: 2147478528
extentsize: 2097152
keepdirty: 64
localoff: 4608
resuid: 10884849062207686761
localcnt: 1
remotecnt: 0
prevrole: primary
resource: disk03
datasize: 2147478528
extentsize: 2097152
keepdirty: 64
localoff: 4608
resuid: 14443609578994823508
localcnt: 1
remotecnt: 0
prevrole: primary
resource: disk04
datasize: 2147478528
extentsize: 2097152
keepdirty: 64
localoff: 4608
resuid: 1365498106518463540
localcnt: 1
remotecnt: 0
prevrole: primary
--
Freddie Cash
fjwcash@gmail.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?b269bc571003031631s1c4cbde1rb3f54b9189d95ea8>