From owner-freebsd-fs@FreeBSD.ORG Fri Aug 1 19:36:05 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 970A02D9 for ; Fri, 1 Aug 2014 19:36:05 +0000 (UTC) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "smarthost.sentex.ca", Issuer "smarthost.sentex.ca" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 715E72BB6 for ; Fri, 1 Aug 2014 19:36:05 +0000 (UTC) Received: from [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a] (saphire3.sentex.ca [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a]) by smarthost1.sentex.ca (8.14.9/8.14.9) with ESMTP id s71JZwQQ015817; Fri, 1 Aug 2014 15:35:58 -0400 (EDT) (envelope-from mike@sentex.net) Message-ID: <53DBEC15.2040406@sentex.net> Date: Fri, 01 Aug 2014 15:35:49 -0400 From: Mike Tancsa Organization: Sentex Communications User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.0 MIME-Version: 1.0 To: Mikolaj Golub Subject: Re: HAST slowness ? References: <53DAB322.1050703@sentex.net> <20140801183005.GA3286@gmail.com> In-Reply-To: <20140801183005.GA3286@gmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.74 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Aug 2014 19:36:05 -0000 On 8/1/2014 2:30 PM, Mikolaj Golub wrote: > On Thu, Jul 31, 2014 at 05:20:34PM -0400, Mike Tancsa wrote: >> I am trying to setup my first HAST setup and for some reason, traffic >> sync seems to be really slow. If I do something simple like >> >> >> #dd if=/dev/zero of=/hastvirt/test count=2000 bs=1024k >> >> on the primary box, traffic on the secondary never gets much beyond a >> Mb/s ?!?! >> > > It looks you are writing to a file-system, not directly to HAST > device? What is this FS? I see two HAST devices? Are they combined in > one FS? How? What if you write directly to a HAST device instead? Hi Mikolaj, Thank you for responding. I have 2 servers, both with two disks. Each have 2 zfs partitions The first zfs partition is the operating system ada0p4 and ada1p4 are for hast on each of the boxes # zpool status pool: hastvirt state: ONLINE scan: scrub repaired 0 in 0h5m with 0 errors on Thu Jul 31 17:34:49 2014 config: NAME STATE READ WRITE CKSUM hastvirt ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 hast/disk1 ONLINE 0 0 0 hast/disk2 ONLINE 0 0 0 The idea if one box dies, the other can take over and mount hastvirt > > Also note, "hole" compression is enabled by default, so your test with > copying zero blocks is not very good. You might be limited here by > RTT, because in memsync mode, on every "compressed" block sent HAST > waits for response before proceeding with the next block. Perhaps this indeed was the issue. I tried importing a windows VM and it seemed to go much faster. I am now seeing traffic across the HAST dedicated NIC at close to 200Mb % ifstat -b -i vlan5 vlan5 Kbps in Kbps out 262365.4 6407.79 235609.5 5757.97 217813.4 5380.81 144054.8 3606.39 150899.8 3882.17 240806.7 5908.37 238943.7 5718.61 Looking at gstat on the master, it currently has maxed out the disk writing so I guess the network is keeping up. # hastctl status Name Status Role Components disk1 complete primary /dev/ada0p4 1.1.1.1 disk2 complete primary /dev/ada1p4 1.1.1.1 hastctl list disk1: role: primary provname: disk1 localpath: /dev/ada0p4 extentsize: 2097152 (2.0MB) keepdirty: 64 remoteaddr: 1.1.1.1 replication: memsync status: complete workerpid: 1021 dirty: 4194304 (4.0MB) statistics: reads: 387530 writes: 309272 deletes: 11 flushes: 480 activemap updates: 19121 local errors: read: 0, write: 0, delete: 11, flush: 0 queues: local: 0, send: 0, recv: 0, done: 0, idle: 255 disk2: role: primary provname: disk2 localpath: /dev/ada1p4 extentsize: 2097152 (2.0MB) keepdirty: 64 remoteaddr: 1.1.1.1 replication: memsync status: complete workerpid: 1023 dirty: 4194304 (4.0MB) statistics: reads: 387008 writes: 309251 deletes: 11 flushes: 480 activemap updates: 19118 local errors: read: 0, write: 0, delete: 11, flush: 0 queues: local: 0, send: 0, recv: 0, done: 0, idle: 255 Are there any good hast tuning guides beyond the wiki you recommend ? Should anything be tuned beyond the defaults ? Thanks again ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/