From owner-freebsd-stable@FreeBSD.ORG Tue Jan 12 20:49:54 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0BB62106568D; Tue, 12 Jan 2010 20:49:54 +0000 (UTC) (envelope-from dan.naumov@gmail.com) Received: from mail-gx0-f209.google.com (mail-gx0-f209.google.com [209.85.217.209]) by mx1.freebsd.org (Postfix) with ESMTP id 90D248FC26; Tue, 12 Jan 2010 20:49:53 +0000 (UTC) Received: by gxk1 with SMTP id 1so10836664gxk.14 for ; Tue, 12 Jan 2010 12:49:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=RQo+tMLSHoZmsRGH7RPaf/BsGx44obwXIScAEjxDxVQ=; b=sg9JEyAtt00OHphC3ARFnSHWKtYSH19FUQ6inGd1ijIier6YOr+CEvl0HLegNfL5Bc aBv3ySdV52AgbPpVBLt3eA7vqrXVfUqoyrp09RZ4oOuy/cr+IpECIjIA27Uevfh7wZvI 5MUGlxqqdPqCO0e0ryXAKbxIuAaOR7JYjcTMU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=rYkhcpb1YGZBwhK1CmQ4uzd5sb141vVL8wXIUbk2t87AdXaVYMZNO21NjO4VUxYgHT T0zH/RgRJrTCdY2UKkB9EVqozGXxhAV5uh/RZmMODniWYc65ggseFwGehj+jYHJL0Wej VG6ExaRLMCYj2RYdGP8pFo2SIxiySG2gwol0Q= MIME-Version: 1.0 Received: by 10.100.63.16 with SMTP id l16mr789170ana.47.1263329386146; Tue, 12 Jan 2010 12:49:46 -0800 (PST) In-Reply-To: <201001122127.56716.freebsd@o2.pl> References: <201001120100.16631.freebsd@o2.pl> <201001122127.56716.freebsd@o2.pl> Date: Tue, 12 Jan 2010 22:49:46 +0200 Message-ID: From: Dan Naumov To: freebsd@o2.pl, FreeBSD-STABLE Mailing List , freebsd-questions@freebsd.org, freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Subject: Re: ZFS on top of GELI X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2010 20:49:54 -0000 2010/1/12 Rafa=C5=82 Jackiewicz : >>Thanks, could you do the same, but using 2 .eli vdevs mirrorred >>together in a zfs mirror? >> >>- Sincerely, >>Dan Naumov > > Hi, > > Proc: Intell Atom 330 (2x1.6Ghz) - 1 package(s) x 2 core(s) x 2 HTT threa= ds > Chipset: Intel 82945G > Sys: 8.0-RELEASE FreeBSD 8.0-RELEASE #0 > empty file: /boot/loader.conf > Hdd: > =C2=A0 ad4: 953869MB at ata2-master SATA150 > =C2=A0 ad6: 953869MB at ata3-master SATA150 > Geli: > =C2=A0 geli init -s 4096 -K /etc/keys/ad4s2.key /dev/ad4s2 > =C2=A0 geli init -s 4096 -K /etc/keys/ad6s2.key /dev/ad6s2 > > > Results: > **************************************************** > > *** single drive =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0write MB/s =C2=A0 =C2=A0 =C2=A0read =C2=A0MB/s > eli.journal.ufs2 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A023 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A014 > eli.zfs =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 19 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A036 > > > *** mirror =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0write MB/s =C2=A0 =C2=A0 =C2=A0re= ad =C2=A0MB/s > mirror.eli.journal.ufs2 23 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A016 > eli.zfs =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 31 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A040 > zfs =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 83 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A079 > > > *** degraded mirror =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 write MB/s = =C2=A0 =C2=A0 =C2=A0read MB/s > mirror.eli.journal.ufs2 16 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A09 > eli.zfs =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 56 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A040 > zfs =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 86 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A071 > > **************************************************** Thanks a lot for your numbers, the relevant part for me was this: *** mirror write MB/s read MB/s eli.zfs 31 40 zfs 83 79 *** degraded mirror write MB/s read MB/s eli.zfs 56 40 zfs 86 71 31 mb/s writes and 40 mb/s reads is something that I guess I could potentially live with. I am guessing the main problem of stacking ZFS on top of geli like this is the fact that writing to a mirror requires double the CPU use, because we have to encrypt all written data twice (once to each disk) instead of encrypting first and then writing the encrypted data to 2 disks as would be the case if we had crypto sitting on top of ZFS instead of ZFS sitting on top of crypto. I now have to reevaluate my planned use of an SSD though, I was planning to use a 40gb partition on an Intel 80GB X25-M G2 as a dedicated L2ARC device for a ZFS mirror of 2 x 2tb disks. However these numbers make it quite obvious that I would already be CPU-starved at 40-50mb/s throughput on the encrypted ZFS mirror, so adding an l2arc SSD, while improving latency, would do really nothing for actual disk read speeds, considering the l2arc itself would too, have to sit on top of a GELI device. - Sincerely, Dan Naumov