From owner-freebsd-geom@FreeBSD.ORG Mon Jul 30 19:56:30 2007 Return-Path: Delivered-To: geom@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C46AB16A41A for ; Mon, 30 Jul 2007 19:56:30 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (arm132.internetdsl.tpnet.pl [83.17.198.132]) by mx1.freebsd.org (Postfix) with ESMTP id 662B713C45B for ; Mon, 30 Jul 2007 19:56:30 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 147EF4569A; Mon, 30 Jul 2007 21:27:42 +0200 (CEST) Received: from localhost (154.81.datacomsa.pl [195.34.81.154]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 6DB3645684; Mon, 30 Jul 2007 21:27:37 +0200 (CEST) Date: Mon, 30 Jul 2007 21:26:54 +0200 From: Pawel Jakub Dawidek To: Dominic Bishop Message-ID: <20070730192654.GO1092@garage.freebsd.pl> References: <46AA50B4.9080901@fluffles.net> <20070727210032.0140413C457@mx1.freebsd.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="fNagykWcDoSVAmSd" Content-Disposition: inline In-Reply-To: <20070727210032.0140413C457@mx1.freebsd.org> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 7.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-2.6 required=3.0 tests=BAYES_00 autolearn=ham version=3.0.4 Cc: geom@FreeBSD.org Subject: Re: Increasing GELI performance X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jul 2007 19:56:30 -0000 --fNagykWcDoSVAmSd Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jul 27, 2007 at 10:00:35PM +0100, Dominic Bishop wrote: > I just tried your suggestion of geli on the raw device and it is no better > at all: >=20 > dd if=3D/dev/da0.eli of=3D/dev/null bs=3D1m count=3D1000 > 1000+0 records in > 1000+0 records out > 1048576000 bytes transferred in 29.739186 secs (35259069 bytes/sec) >=20 > dd if=3D/dev/zero of=3D/dev/da0.eli bs=3D1m count=3D1000 > 1000+0 records in > 1000+0 records out > 1048576000 bytes transferred in 23.501061 secs (44618241 bytes/sec) >=20 > Using top -S with 1s refresh to list the geli processes whilst doing this= it > seems only one of them is doing anything at any given time, the others are > sitting in a state of "geli:w", I assume that is a truncation of somethin= g, > maybe geli:wait at a guess. No matter how many cores/cpus you have if you run single-threaded application. What you do exactly is: 1. Send read of 128kB. 2. One of geli threads picks it up, decrypts and sends it back. 3. Send next read of 128kB. 4. One of geli threads picks it up, decrypts and sends it back. =2E.. All threads will be used when there are more threads accessing provider. > For comparison, here are the same tests done on the unencrypted raw devic= e: >=20 > dd if=3D/dev/da0 of=3D/dev/null bs=3D1m count=3D1000 > 1000+0 records in > 1000+0 records out > 1048576000 bytes transferred in 5.802704 secs (180704717 bytes/sec) >=20 > dd if=3D/dev/zero of=3D/dev/da0 bs=3D1m count=3D1000 > 1000+0 records in > 1000+0 records out > 1048576000 bytes transferred in 4.026869 secs (260394859 bytes/sec) You can see how much CPU power your one core has by doing something like this: # kldload geom_zero # geli onetime -s 4096 gzero # sysctl kern.geom.zero.clear=3D0 # dd if=3D/dev/gzero.eli of=3D/dev/null bs=3D1m count=3D4096 /dev/gzero is similar to /dev/zero, but it is a GEOM provider. This experiment will show you how much encryption throughput you can get from one geli thread, without disk overhead. --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --fNagykWcDoSVAmSd Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFGrjt+ForvXbEpPzQRAg6sAJsHLS1W5ZZVjdXmW1uuPJEzgKoylgCfX8wy 5DupTlMsgJZjhdKcApiNLEU= =Iq2i -----END PGP SIGNATURE----- --fNagykWcDoSVAmSd--