From owner-freebsd-fs@FreeBSD.ORG Tue Mar 1 04:01:55 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4EECA106566C for ; Tue, 1 Mar 2011 04:01:55 +0000 (UTC) (envelope-from ethernext@gmail.com) Received: from mail-qy0-f175.google.com (mail-qy0-f175.google.com [209.85.216.175]) by mx1.freebsd.org (Postfix) with ESMTP id 064938FC19 for ; Tue, 1 Mar 2011 04:01:54 +0000 (UTC) Received: by qyk35 with SMTP id 35so2699392qyk.13 for ; Mon, 28 Feb 2011 20:01:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:date:x-google-sender-auth :message-id:subject:from:to:content-type; bh=H1r6ph9IRLXr3Fiepfr8GZuNxirs85FE+mpFuGFxqTY=; b=LETpwF3UJ5MlxkKTvEHEDDE2hbL6JhZD6nD8oQ+VkouCv83vqimC26/Wro2Mf+RYo6 xjdSiKto6vI0zk7ruFlsmDN2pkr8DeU5ndutl1BW9QCFbrvR8Q7q8i1zG56lhVibeeL3 hssnVrB/zGyyr+tFYMfSYn/rcqpe1khFfjZQg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:content-type; b=J9IOzJvYnfHE8kT+0zb69PDC/1DGDOBByXqpkDYWgv5Vm+0N6uG39H3RwTK8i54g24 x1vp0+cLItO+aR2NuWR8YZFz192JIlZF59ahdcFhXWi+UZLJMhIejJiHzI0zMpHy46dJ HNq3IYiA6mg4bfPIuWt5MJs6ukRDBsoOSukXA= MIME-Version: 1.0 Received: by 10.229.81.15 with SMTP id v15mr4783724qck.209.1298950524987; Mon, 28 Feb 2011 19:35:24 -0800 (PST) Sender: ethernext@gmail.com Received: by 10.229.102.67 with HTTP; Mon, 28 Feb 2011 19:35:24 -0800 (PST) Date: Mon, 28 Feb 2011 22:35:24 -0500 X-Google-Sender-Auth: tXb92c1hwHap1e3IkqyK2-XcdPk Message-ID: From: Bill Desjardins To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: HAST, zvols, istgt and carp working... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Mar 2011 04:01:55 -0000 Hello All, as an experiment today I setup a couple 8-stable guests on vmware ESXI to test hast with zfs, carp and istgt for a redundant nas system I am putting together. I havent seen any mention that anyone has used hast to mirror a zfs zvol so I figured I would try it and atleast my proof of concept seems to work just fine. is anyone doing this and using it in a production environment? heres how I setup the testing environment... - created (2) 8-stable hosts on esxi 4.1: hast-1 & hast-2 (os on da0) on both hast-1 and hast-2 - added 4 x 8GB disk's to each (da1 - da4) - glabel'd disks disk1 - disk4 - zpool create tank mirror disk1 disk2 mirror disk3 disk4 - zfs create -p -s -b 64k -V 4G tank/hzvol.1 hast.conf on each resource tank_hzvol.1 { local /dev/zvol/tank/hzvol.1 on hast-1 { remote x.x.x.9 } on hast-2 { remote x.x.x.8 } } on hast-1 and hast-2 > hastd_enable="YES" in rc.conf > hastctl create tank_hzvol.1 > /etc/rc.d/hastd start on hast-2 > hastctl role secondary tank_hzvol.1 on hast-1 > hastctl role primary tank_hzvol.1 hastctl status reports all is well so far... next I configured istgt identically on hast-1 and hast-2 for the hast device >> LUN0 Storage /dev/hast/tank_hzvol.1 3G istgt was started (istgt onestart) and the zvol target was setup on another vmware esxi server which then was formatted as a vmfs volume. I created a 2GB disk on this volume and added it to another 8-stable host as a ufs disk mounted on /mnt. so far going good, everything working as expected. to test hast replication, I created a few 200MB files on the host with the ufs vmdk volume and seen traffic over the hast network from hast-1 to hast-2. on hast-1, the zvol size reflected correct sparse disk space usage, but hast-2 showed the full 4GB zvol allocated which I suspect is due to hast. to test failover of the isci zvol target from hast-1 to hast-2: on hast-1 > istgt stop > hastctl role secondary tank_hzvol.1 on hast-2 > hastctl role primary tank_hzvol.1 > istgt onestart NOTE: carp does not seem to work on esxi for me so between hast-1 and hast-2 I manually moved the IP for istgt to hast-2. the result was that the istgt hast zvol successfully failed over to hast-2 with only a brief stall while I manually performed the failover process. I only performed the ideal manual failover scenario for proof of concept. I will be testing this on 2 real development servers later this week for a more complete understanding. I see some real advantages for zvols only hast: ++++++++++++++++++++++++++++++++++++++++++++++++ + no need to hast each individual disk in the zpool so you can access all available storage on either storage unit + maintaining storage units remains functionally consistent between them + once setup, zvols are easily migrated to new storage environments in real-time since there is only a single zvol hast resource to replicate. (no need to have all matching zpool hast members, just reconfigure the primary zvol hast resource to point to a new secondary server and swap roles/failover when ready) + can have active hast zvols on each unit to distribute IO + no need for zpool export/import on failover + hast easily added to current zvols + retains performance of entire zpool + zpool can be expanded without changing hast config + minimizes hast replication traffic between storage units + hast split-brain localized to specific zvol's + can use ufs on hast zvol resource for things like samba and nfs cons ------------------------------------------- - performace impact (???) - each hast zvol requires distinct application configurations (more confgurations to deal with/screw up) - zfs sparse volumes seem not to be working correctly via hast (???) - expanding zvol requires hastctl create, init, startup plus may need application specific changes/restart. - other methods needed to replicate data in rest of pool - possible long rebuild time on large zvols? - snapshots / rollbacks (???) - many more??? my main question is if using hast to replicate a zvol is a supported configuration and what are the possible drawbacks? Its more than likely I am overlooking some very basic requirement/restrictions and am blatantly wrong in all this, but if it can perform, I think its a big+ for freebsd and zfs useability as a nas server. thoughts? comments? criticisms? :) Best, Bill