From owner-freebsd-fs@FreeBSD.ORG Sat Mar 16 19:48:38 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 3E8BA78D for ; Sat, 16 Mar 2013 19:48:38 +0000 (UTC) (envelope-from marck@rinet.ru) Received: from woozle.rinet.ru (woozle.rinet.ru [195.54.192.68]) by mx1.freebsd.org (Postfix) with ESMTP id CC5D1FB9 for ; Sat, 16 Mar 2013 19:48:37 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by woozle.rinet.ru (8.14.5/8.14.5) with ESMTP id r2GJmQd4009240 for ; Sat, 16 Mar 2013 23:48:26 +0400 (MSK) (envelope-from marck@rinet.ru) Date: Sat, 16 Mar 2013 23:48:26 +0400 (MSK) From: Dmitry Morozovsky To: freebsd-fs@FreeBSD.org Subject: HA iSCSI target on ZFS: model Message-ID: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-NCC-RegID: ru.rinet X-OpenPGP-Key-ID: 6B691B03 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (woozle.rinet.ru [0.0.0.0]); Sat, 16 Mar 2013 23:48:26 +0400 (MSK) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Mar 2013 19:48:38 -0000 Dear colleagues, I'm currently plan to architect and deploy test HA iSCSI target using two FreeBSD hosts, and want to listen to your comments. What I already did: - two hosts, booted from internal USB stick, with 4 HDDs and one SSD each - LACP-based laggs while links are connected to different half of a clustered switch, with the mtu of 9000 - 2 carps (thanks to araujo@ help, -current, i.e. interface property, not clone) holding shared addresses - disk layout such as root@cthulhu4:/usr/local/etc/istgt# gpart show -l => 34 1953525101 ada0 GPT (931G) 34 2014 - free - (1M) 2048 1952448512 1 ct4-0 (931G) 1952450560 1074575 - free - (524M) => 34 1953522988 ada1 GPT (931G) 34 2014 - free - (1M) 2048 1952448512 1 ct4-1 (931G) 1952450560 1072462 - free - (523M) => 34 1953525101 ada2 GPT (931G) 34 2014 - free - (1M) 2048 1952448512 1 ct4-2 (931G) 1952450560 1074575 - free - (524M) => 34 1953525101 ada3 GPT (931G) 34 2014 - free - (1M) 2048 1952448512 1 ct4-3 (931G) 1952450560 1074575 - free - (524M) => 34 234441581 ada4 GPT (111G) 34 2014 - free - (1M) 2048 2097152 1 ct3-zil4 (1.0G) 2099200 2097152 2 ct4-zil4 (1.0G) 4196352 230244352 3 ct4-cache (109G) 234440704 911 - free - (455k) => 0 4005886 da0 BSD (1.9G) 0 16 - free - (8.0k) 16 4005870 1 (null) (1.9G) (da0 is USB, ada0-ada3 HDDs, ada4 SSD) - 2 hast sets such as root@cthulhu4:/usr/local/etc/istgt# hastctl status Name Status Role Components d0 complete secondary /dev/ada0p1 cthulhu3 d1 complete secondary /dev/ada1p1 cthulhu3 d2 complete primary /dev/ada2p1 cthulhu3 d3 complete primary /dev/ada3p1 cthulhu3 zil3 complete secondary /dev/ada4p1 cthulhu3 zil4 complete primary /dev/ada4p2 cthulhu3 - 2 ZFS setups like root@cthulhu4:/usr/local/etc/istgt# zpool status pool: ct4 state: ONLINE scan: scrub repaired 0 in 0h0m with 0 errors on Thu Mar 14 17:58:48 2013 config: NAME STATE READ WRITE CKSUM ct4 ONLINE 0 0 0 hast/d2 ONLINE 0 0 0 hast/d3 ONLINE 0 0 0 logs hast/zil4 ONLINE 0 0 0 cache gpt/ct4-cache ONLINE 0 0 0 Now, it's time to create exportable entities. I think of creating thin (non-preallocated) ZFS volumes in the pools, and sharing them via istgt. What zfs properties would be appropriate for this? I'm thinking at least about volblocksize=4k (main usage will be vSphere), but not sure about it. And, more importantly, what about sync property? Did I miss something obvious? (and yes, supporting scripts for checking paired resource to be alive are all to be written...) Thanks in advance! -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: marck@FreeBSD.org ] ------------------------------------------------------------------------ *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck@rinet.ru *** ------------------------------------------------------------------------