From owner-freebsd-stable@FreeBSD.ORG Thu May 6 12:11:51 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 23988106566B for ; Thu, 6 May 2010 12:11:51 +0000 (UTC) (envelope-from bsam@ipt.ru) Received: from services.ipt.ru (services.ipt.ru [194.62.233.110]) by mx1.freebsd.org (Postfix) with ESMTP id 75CFD8FC14 for ; Thu, 6 May 2010 12:11:50 +0000 (UTC) Received: from bb.ipt.ru ([194.62.233.89]) by services.ipt.ru with esmtp (Exim 4.54 (FreeBSD)) id 1O9zvf-000HWy-JE; Thu, 06 May 2010 16:11:47 +0400 From: Boris Samorodov To: Michal References: <4BE2874D.10909@ionic.co.uk> Date: Thu, 06 May 2010 16:11:47 +0400 In-Reply-To: <4BE2874D.10909@ionic.co.uk> (michal@ionic.co.uk's message of "Thu, 06 May 2010 10:09:33 +0100") Message-ID: <73575852@bb.ipt.ru> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: freebsd-stable@freebsd.org Subject: Re: iscsi, zfs, RAIS = Cheap SAN...maybe X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 May 2010 12:11:51 -0000 Hi! On Thu, 06 May 2010 10:09:33 +0100 Michal wrote: > Many thanks...even if you just read this beast of an e-mail This is not an answer to your questions but my tiny experience in this area. I created two USB 2G storages with FreeBSD current i386: node001 and node002. Each system has one 250G hard disk. Both nodes export their disks via iscsi: ----- node001% sudo istgtcontrol list lun0 storage "/dev/ad0" 250059350016 DONE LIST command ----- node002% sudo istgtcontrol list lun0 storage "/dev/ad4" 250059350016 DONE LIST command ----- I used my workstation as a controller. And since it's an i386 (FreeBSD-CURRENT) I didn't try zfs but created a gmirror: ----- da0 at iscsi0 bus 0 scbus7 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 238475MB (488397168 512 byte sectors: 255H 63S/T 30401C) da1 at iscsi0 bus 0 scbus7 target 1 lun 0 da1: Fixed Direct Access SCSI-5 device da1: 238475MB (488397168 512 byte sectors: 255H 63S/T 30401C) GEOM_MIRROR: Device mirror/storage0 launched (2/2). ----- bsam% gmirror status Name Status Components mirror/storage0 COMPLETE da0 da1 bsam% mount -p | grep storage /dev/mirror/storage0 /storage ufs rw 2 2 ----- While using 100Mb/s network connection I've got 7-8 MB/s writes (after some sysctl tuning, 3-4 MB/s without). And switching to 1Gb/s network connection (Intel Gigabit Ethernet Controllers) gave me something like 20 Mb/s writes (no tuning): ----- bsam% dd if=/dev/zero of=/storage/file.2G bs=1M count=2000 2000+0 records in 2000+0 records out 2097152000 bytes transferred in 108.479201 secs (19332296 bytes/sec) bsam% netstat -w10 input (Total) output packets errs idrops bytes packets errs bytes colls 21 0 0 1658 9 0 582 0 44 0 0 5425 35 0 3014 0 70470 0 0 5640905 114909 0 184072616 0 157670 0 0 11966753 257062 0 390774311 0 175056 0 0 11951631 285343 0 390099162 0 156502 0 0 11867216 255072 0 387223400 0 174907 0 0 11947580 285064 0 390607312 0 155607 0 0 11800874 253530 0 385202710 0 173359 0 0 11835884 282798 0 386524530 0 157604 0 0 11922283 256966 0 389324902 0 174224 0 0 11919850 284396 0 390212650 0 173677 0 0 11853356 283110 0 386944770 0 174934 0 0 11932229 285361 0 389785504 0 69574 0 0 4584810 113056 0 148735429 0 282 0 0 22196 374 0 452062 0 20 0 0 2054 14 0 2473 0 ----- node001% netstat -w10 input (Total) output packets errs idrops bytes packets errs bytes colls 12 0 0 1008 1 0 178 0 17 0 0 1626 2 0 292 0 60498 0 0 88413959 37258 0 2616474 0 139627 0 0 204095754 86076 0 6064930 0 139188 0 0 203450154 85816 0 6047488 0 138486 0 0 202411672 85363 0 6000534 0 139258 0 0 203561033 85848 0 6049360 0 137636 0 0 201181818 84866 0 5981524 0 138262 0 0 202064896 85169 0 6004090 0 139297 0 0 203633800 85829 0 6031458 0 138884 0 0 203011098 85495 0 6025390 0 138280 0 0 202089488 85166 0 6004198 0 139272 0 0 203583962 85828 0 6047758 0 58465 0 0 85346106 36192 0 2545632 0 193 0 0 236228 132 0 10360 0 ----- node002% netstat -w10 input (Total) output packets errs idrops bytes packets errs bytes colls 12 0 0 1008 1 0 178 0 16 0 0 1566 2 0 292 0 73703 0 0 107730061 44956 0 3134984 0 139420 0 0 203862166 85058 0 5899360 0 139295 0 0 203674472 85024 0 5896468 0 138129 0 0 201945040 84345 0 5850658 0 139068 0 0 203351261 84873 0 5886130 0 138214 0 0 202097218 84293 0 5846170 0 137860 0 0 201540738 84120 0 5835754 0 138751 0 0 202889156 84668 0 5871694 0 139260 0 0 203626362 84907 0 5888854 0 138468 0 0 202435390 84534 0 5863900 0 139210 0 0 203557426 84865 0 5885938 0 45288 0 0 66099374 27684 0 1924360 0 193 0 0 236228 136 0 10624 0 11 0 0 1574 3 0 518 0 ----- Well, there were no tuning at all since I didn't have much spare time to exteriment. Here is some more info about hard and soft. Controller: ----- bsam% uname -a FreeBSD bsam.sem.ipt.ru 9.0-CURRENT FreeBSD 9.0-CURRENT #2 r207498: Sun May 2 16:19:56 MSD 2010 bsam@bsam.sem.ipt.ru:/home/bsam/FreeBSD/base/head/obj/storage/src/sys/BB i386 age0@pci0:2:0:0: class=0x020000 card=0x82261043 chip=0x10481969 rev=0xb0 hdr=0x00 vendor = 'Attansic (Now owned by Atheros)' device = 'Gigabit Ethernet 10/100/1000 Base-T Controller (Atheros L1)' class = network subclass = ethernet cap 01[40] = powerspec 2 supports D0 D3 current D0 cap 05[48] = MSI supports 1 message, 64 bit enabled with 1 message cap 10[58] = PCI-Express 1 endpoint max data 128(128) link x1(x1) ----- Node001: ----- node001% uname -a FreeBSD node001.sem.ipt.ru 9.0-CURRENT FreeBSD 9.0-CURRENT #13 r206496: Mon Apr 12 18:28:45 MSD 2010 bsam@bsam.sem.ipt.ru:/m/home/bsam/FreeBSD/base/head/obj/m/home/bsam/FreeBSD/base/head/src/sys/BB i386 em0@pci0:2:1:0: class=0x020000 card=0x002e8086 chip=0x100e8086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = 'Gigabit Ethernet Controller (82540EM)' class = network subclass = ethernet cap 01[dc] = powerspec 2 supports D0 D3 current D0 cap 07[e4] = PCI-X supports 2048 burst read, 1 split transaction cap 05[f0] = MSI supports 1 message, 64 bit ----- Node002: ----- node002% uname -a FreeBSD node002.sem.ipt.ru 9.0-CURRENT FreeBSD 9.0-CURRENT #13 r206496: Mon Apr 12 18:28:45 MSD 2010 bsam@bsam.sem.ipt.ru:/m/home/bsam/FreeBSD/base/head/obj/m/home/bsam/FreeBSD/base/head/src/sys/BB i386 em0@pci0:0:10:0: class=0x020000 card=0x13768086 chip=0x107c8086 rev=0x05 hdr=0x00 vendor = 'Intel Corporation' device = 'Gigabit Ethernet Controller (Copper) rev 5 (82541PI)' class = network subclass = ethernet cap 01[dc] = powerspec 2 supports D0 D3 current D0 cap 07[e4] = PCI-X supports 2048 burst read, 1 split transaction ----- -- WBR, Boris Samorodov (bsam) Research Engineer, http://www.ipt.ru Telephone & Internet SP FreeBSD Committer, http://www.FreeBSD.org The Power To Serve