From owner-freebsd-fs@FreeBSD.ORG Tue Aug 10 02:14:23 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7F064106566B; Tue, 10 Aug 2010 02:14:23 +0000 (UTC) (envelope-from thomas@gibfest.dk) Received: from mail.tyknet.dk (mail.tyknet.dk [213.150.42.155]) by mx1.freebsd.org (Postfix) with ESMTP id D21858FC12; Tue, 10 Aug 2010 02:14:22 +0000 (UTC) Received: from [10.32.67.59] (fw.int.webpartner.dk [213.150.34.98]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mail.tyknet.dk (Postfix) with ESMTPSA id 97C7F638DFC; Tue, 10 Aug 2010 04:14:21 +0200 (CEST) X-DKIM: OpenDKIM Filter v1.1.2 mail.tyknet.dk 97C7F638DFC DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=gibfest.dk; s=default; t=1281406461; bh=SlJEfu7lrQh/DiPOb7yaCqsqLnz+uXDlmjMao/JHJzg=; h=Message-ID:Date:From:MIME-Version:To:CC:Subject:References: In-Reply-To:Content-Type:Content-Transfer-Encoding; b=I0/STZWabsqQpZQY6v7eLquKorJkKIOzZshzNWvphXPHBjDnx4Mx24bHJxH+VugCb YfoxlCS6mkSvSX55TyLZ88+stoQJM0Go7lONRVQgcC//KbKRLgvYezjglWTmKUDigb qC6JKoCstad4zjnQjQv9aXox6zm1de/67Wgcdu/0= Message-ID: <4C60B5FD.9080603@gibfest.dk> Date: Tue, 10 Aug 2010 04:14:21 +0200 From: Thomas Steen Rasmussen User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.11) Gecko/20100711 Lightning/1.0b1 Thunderbird/3.0.6 MIME-Version: 1.0 To: pjd@FreeBSD.org References: <4C57E20E.2030908@gibfest.dk> <20100806135001.GF1710@garage.freebsd.pl> <4C5ECA78.6010803@gibfest.dk> In-Reply-To: <4C5ECA78.6010803@gibfest.dk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: HAST initial sync speed X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Aug 2010 02:14:23 -0000 On 08-08-2010 17:17, Thomas Steen Rasmussen wrote: > On 06-08-2010 15:50, Pawel Jakub Dawidek wrote: >> On Tue, Aug 03, 2010 at 11:31:58AM +0200, Thomas Rasmussen wrote: >> >>> Hello list, >>> >>> I finally got my ZFS/HAST setup up and running, or trying to at least. >>> I am wondering how fast the initial HAST sync normally is - I created >>> these 4 HAST providers yesterday on 4 146 gig drives, and they still each >>> have over 90 gigabytes 'dirty' today. The machines are powerful (dell >>> r710) and are otherwise idle, and they are connected to the same gigabit >>> switch. >>> >>> I can supply details about any part of the configuration if needed, but I >>> just wanted to ask if you guys believe something is wrong here. I can't help >>> but think, if the initial sync takes 24+ hours, then if I ever need to >>> replace one of the servers, I will be without redundancy until the new >>> server reaches 0 'dirty' bytes, correct ? >>> >> Correct, but synchronizartion should take much, much less time. >> Is dirty count actually decreasing? >> >> > Hello, > > Yes it was decreasing steadily but very slowly. It finished between thursday > evening and friday morning, and the dirty count is now 0. All in all it took over > 72 hours. It was transferring around 20mbits while doing this. However, if I > copied a large file to the primary HAST node, it would use up a lot more > bandwidth. It is like HAST was synchronizing the "empty space" with lower > priority or something. Does that make any sense ? The servers are not in > production so I can perform any testing needed. Thank you for your reply. > > Regards > > Thomas Steen Rasmussen > Hello again, I just wanted to include the configs here for completeness: /etc/hast.conf: ----------------------------- resource hasthd4 { local /dev/label/hd4 on server1 { remote 192.168.0.15 } on server2 { remote 192.168.0.14 } } resource hasthd5 { local /dev/label/hd5 on server1 { remote 192.168.0.15 } on server2 { remote 192.168.0.14 } } resource hasthd6 { local /dev/label/hd6 on server1 { remote 192.168.0.15 } on server2 { remote 192.168.0.14 } } resource hasthd7 { local /dev/label/hd7 on server1 { remote 192.168.0.15 } on server2 { remote 192.168.0.14 } } ----------------------------- To create the setup I ran the following commands on both servers: glabel label ssd0 /dev/mfid1 glabel label ssd1 /dev/mfid2 glabel label hd4 /dev/mfid3 glabel label hd5 /dev/mfid4 glabel label hd6 /dev/mfid5 glabel label hd7 /dev/mfid6 And on server2: [root@server2 ~]# hastctl create hasthd4 [root@server2 ~]# hastctl create hasthd5 [root@server2 ~]# hastctl create hasthd6 [root@server2 ~]# hastctl create hasthd7 [root@server2 ~]# /etc/rc.d/hastd start [root@server2 ~]# hastctl role secondary all And on server1: [root@server1 ~]# hastctl create hasthd4 [root@server1 ~]# hastctl create hasthd5 [root@server1 ~]# hastctl create hasthd6 [root@server1 ~]# hastctl create hasthd7 [root@server1 ~]# /etc/rc.d/hastd start [root@server1 ~]# hastctl role primary all This made the HAST devices appear on server1 under /dev/hast/ Then I created the ZFS filesystem on top, on server1: zpool create hatank raidz2 /dev/hast/hasthd4 /dev/hast/hasthd5 /dev/hast/hasthd6 /dev/hast/hasthd7 cache /dev/label/ssd0 /dev/label/ssd1 This resulted in the following "hastctl status" output, on server1: hasthd4: role: primary provname: hasthd4 localpath: /dev/label/hd4 extentsize: 2097152 keepdirty: 64 remoteaddr: 192.168.0.15 replication: memsync status: complete dirty: 146051956736 bytes hasthd5: role: primary provname: hasthd5 localpath: /dev/label/hd5 extentsize: 2097152 keepdirty: 64 remoteaddr: 192.168.0.15 replication: memsync status: complete dirty: 146045665280 bytes hasthd6: role: primary provname: hasthd6 localpath: /dev/label/hd6 extentsize: 2097152 keepdirty: 64 remoteaddr: 192.168.0.15 replication: memsync status: complete dirty: 146047762432 bytes hasthd7: role: primary provname: hasthd7 localpath: /dev/label/hd7 extentsize: 2097152 keepdirty: 64 remoteaddr: 192.168.0.15 replication: memsync status: complete dirty: 146047762432 bytes -------------------------------------------------- The problem again is simply that the initial synchronization took way too long. If I copy a large file to the primary HAST server now it syncs very quickly. I am open for any input, I obviously can't really use HAST before this problem is solved. Thank you again. Thomas Steen Rasmussen