From owner-freebsd-geom@FreeBSD.ORG Mon Jul 30 11:08:19 2007 Return-Path: Delivered-To: freebsd-geom@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 34B9A16A469 for ; Mon, 30 Jul 2007 11:08:19 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 1F9BF13C480 for ; Mon, 30 Jul 2007 11:08:19 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.1/8.14.1) with ESMTP id l6UB8IQq040604 for ; Mon, 30 Jul 2007 11:08:18 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.1/8.14.1/Submit) id l6UB8HS7040600 for freebsd-geom@FreeBSD.org; Mon, 30 Jul 2007 11:08:17 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 30 Jul 2007 11:08:17 GMT Message-Id: <200707301108.l6UB8HS7040600@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-geom@FreeBSD.org Cc: Subject: Current problem reports assigned to you X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jul 2007 11:08:19 -0000 Current FreeBSD problem reports Critical problems Serious problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/73177 geom kldload geom_* causes panic due to memory exhaustion o kern/76538 geom [gbde] nfs-write on gbde partition stalls and continue o kern/83464 geom [geom] [patch] Unhandled malloc failures within libgeo o kern/84556 geom [geom] GBDE-encrypted swap causes panic at shutdown o kern/87544 geom [gbde] mmaping large files on a gbde filesystem deadlo o kern/89102 geom [geom_vfs] [panic] panic when forced unmount FS from u o bin/90093 geom fdisk(8) incapable of altering in-core geometry o kern/90582 geom [geom_mirror] [panic] Restore cause panic string (ffs_ o kern/98034 geom [geom] dereference of NULL pointer in acd_geom_detach o kern/104389 geom [geom] [patch] sys/geom/geom_dump.c doesn't encode XML o kern/113419 geom [geom] geom fox multipathing not failing back o misc/113543 geom [geom] [patch] geom(8) utilities don't work inside the o kern/113957 geom [gmirror] gmirror is intermittently reporting a degrad 13 problems total. Non-critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- o bin/78131 geom gbde "destroy" not working. o kern/79251 geom [2TB] newfs fails on 2.6TB gbde device o kern/94632 geom [geom] Kernel output resets input while GELI asks for f kern/105390 geom [geli] filesystem on a md backed by sparse file with s o kern/107707 geom [geom] [patch] add new class geom_xbox360 to slice up p bin/110705 geom gmirror control utility does not exit with correct exi o kern/113790 geom [patch] enable the Camellia block cipher on GEOM ELI ( o kern/113837 geom [geom] unable to access 1024 sector size storage o kern/113885 geom [geom] [patch] improved gmirror balance algorithm o kern/114532 geom GEOM_MIRROR shows up in kldstat even if compiled in th 10 problems total. From owner-freebsd-geom@FreeBSD.ORG Mon Jul 30 13:46:01 2007 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A573816A418 for ; Mon, 30 Jul 2007 13:46:01 +0000 (UTC) (envelope-from fernan.aguero@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.169]) by mx1.freebsd.org (Postfix) with ESMTP id 2CD1713C469 for ; Mon, 30 Jul 2007 13:46:00 +0000 (UTC) (envelope-from fernan.aguero@gmail.com) Received: by ug-out-1314.google.com with SMTP id o4so1218119uge for ; Mon, 30 Jul 2007 06:45:59 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=F+65xJtRVZkmJ7Z/XJeZUvNmGczfu3a0QcW3aJOZ8L+AdgV7c7OvNe7YDVEGOf5mqFnoIBePe/LlQ484s6ioiQiwnqO8R8EDDoX0HXvKRwyyAUZALlOQVaxOnJ/tniKjcncE56V8IUD+pPHAVyo9lmYInc8IZB6cIwJyeL83UMM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=XnAKeiPrxurbU1WezMKnHpkdDsMN+90zKPQZaJSGbSkI6GMUoVth2BYKeuMb2U4SroFSoGiv3HLfZlIk5lXth/+GkVDdSmWIeNmQqzNgrkpNk6NUnwtilmkcKVN6GHdJiPmERyaSgdRAwK/HbOCYZ9v+orcrWrrhsAcAaZrto2A= Received: by 10.78.123.4 with SMTP id v4mr1511922huc.1185803159566; Mon, 30 Jul 2007 06:45:59 -0700 (PDT) Received: by 10.78.25.16 with HTTP; Mon, 30 Jul 2007 06:45:59 -0700 (PDT) Message-ID: <520894aa0707300645g23165798yec9737b93c264c2b@mail.gmail.com> Date: Mon, 30 Jul 2007 10:45:59 -0300 From: "Fernan Aguero" To: Fluffles In-Reply-To: <46AB2C1D.9090207@fluffles.net> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <520894aa0707272126g42b88d41q95477f2d1dd3689b@mail.gmail.com> <46AB2C1D.9090207@fluffles.net> Cc: freebsd-geom@freebsd.org Subject: Re: boot from second disk (gmirror device) ... problems X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jul 2007 13:46:01 -0000 On 7/28/07, Fluffles wrote: > Fernan Aguero wrote: > > After setting up /etc/fstab and /boot/loader.conf in my > > /dev/mirror/gm0a partition > > rebooting brings me to the boot loader prompt because: > > 1:ad(2,a)/boot/loader > > is not a valid label or /boot/loader does not exist > > > > On a boot partition you need to have a slice, so you probably should use: > > 1:ad(2,1,a)/boot/loader > > this will try to boot from ad2s1a, a likely name for a boot partition on the secondary master PATA disk. I can confirm now that this doesn't work. I do have a slice on the second disk (there is only one slice, ad2s1), and fdisk tells me that this slice is marked as 'active'. Also, there are 8 partitions (a, b, d, e, f, g, h) with partition 'a' containing an exact copy of the partition 'a' in ad0s2a. rho# fdisk ad2 ******* Working on device /dev/ad2 ******* parameters extracted from in-core disklabel are: cylinders=57461 heads=16 sectors/track=255 (4080 blks/cyl) Figures below won't work with BIOS for partitions not in cyl 1 parameters to be used for BIOS calculations are: cylinders=57461 heads=16 sectors/track=255 (4080 blks/cyl) Media sector size is 512 Warning: BIOS sector numbering starts with sector 1 Information from DOS bootblock is: The data for partition 1 is: sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD) start 255, size 234432465 (114468 Meg), flag 80 (active) beg: cyl 0/ head 1/ sector 1; end: cyl 114/ head 15/ sector 63 > You may also press "?" at the prompt and get a list of devices. I don't get a list of devices. When /boot.config contains something that doesn't work (1:ad(2,a); 1:ad(2,1,a); 1:ad(1,a) ...) then pressing '?' does nothing (i.e. prints an 'Incorrect label' or some similar error message) However, when I type something that at least works partiallty (i.e. points at the right disk (which is always the first disk in my case) then pressing '?' gives me a list of directories under / (but not a list of devices). In this case I can type 'boot/loader' at the boot: prompt and boot fine (but from the first disk). I have put all combinations you can imagine into /boot.config to try to boot from the second disk. No luck whatsoever. BTW, this is on FreeBSD-6.2 (RELEASE), on a MSI KM4MV motherboard. http://www.msicomputer.com/product/p_spec.asp?model=KM4M-V > Also, for your /etc/fstab you may wish to use a label instead. Read up on man glabel on how to set these up. The idea is that, by giving a disk a unique label, you can setup your /etc/fstab without worrying on what controller/cable/connector te disk is on. So doesn't matter if its ad2 or ad8, it will show up as /dev/label/mydisk and you can setup your /etc/fstab accordingly. Thanks for the tip. Would these labels work for the boot loader? E.g. can I label my second disk (ad2) or the geom mirror device (gm0) and use that in boot.config? Thanks, Fernan > Good luck! > > - Veronica > > -- -- fernan From owner-freebsd-geom@FreeBSD.ORG Mon Jul 30 19:56:30 2007 Return-Path: Delivered-To: geom@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C46AB16A41A for ; Mon, 30 Jul 2007 19:56:30 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (arm132.internetdsl.tpnet.pl [83.17.198.132]) by mx1.freebsd.org (Postfix) with ESMTP id 662B713C45B for ; Mon, 30 Jul 2007 19:56:30 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 147EF4569A; Mon, 30 Jul 2007 21:27:42 +0200 (CEST) Received: from localhost (154.81.datacomsa.pl [195.34.81.154]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 6DB3645684; Mon, 30 Jul 2007 21:27:37 +0200 (CEST) Date: Mon, 30 Jul 2007 21:26:54 +0200 From: Pawel Jakub Dawidek To: Dominic Bishop Message-ID: <20070730192654.GO1092@garage.freebsd.pl> References: <46AA50B4.9080901@fluffles.net> <20070727210032.0140413C457@mx1.freebsd.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="fNagykWcDoSVAmSd" Content-Disposition: inline In-Reply-To: <20070727210032.0140413C457@mx1.freebsd.org> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 7.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-2.6 required=3.0 tests=BAYES_00 autolearn=ham version=3.0.4 Cc: geom@FreeBSD.org Subject: Re: Increasing GELI performance X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jul 2007 19:56:30 -0000 --fNagykWcDoSVAmSd Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jul 27, 2007 at 10:00:35PM +0100, Dominic Bishop wrote: > I just tried your suggestion of geli on the raw device and it is no better > at all: >=20 > dd if=3D/dev/da0.eli of=3D/dev/null bs=3D1m count=3D1000 > 1000+0 records in > 1000+0 records out > 1048576000 bytes transferred in 29.739186 secs (35259069 bytes/sec) >=20 > dd if=3D/dev/zero of=3D/dev/da0.eli bs=3D1m count=3D1000 > 1000+0 records in > 1000+0 records out > 1048576000 bytes transferred in 23.501061 secs (44618241 bytes/sec) >=20 > Using top -S with 1s refresh to list the geli processes whilst doing this= it > seems only one of them is doing anything at any given time, the others are > sitting in a state of "geli:w", I assume that is a truncation of somethin= g, > maybe geli:wait at a guess. No matter how many cores/cpus you have if you run single-threaded application. What you do exactly is: 1. Send read of 128kB. 2. One of geli threads picks it up, decrypts and sends it back. 3. Send next read of 128kB. 4. One of geli threads picks it up, decrypts and sends it back. =2E.. All threads will be used when there are more threads accessing provider. > For comparison, here are the same tests done on the unencrypted raw devic= e: >=20 > dd if=3D/dev/da0 of=3D/dev/null bs=3D1m count=3D1000 > 1000+0 records in > 1000+0 records out > 1048576000 bytes transferred in 5.802704 secs (180704717 bytes/sec) >=20 > dd if=3D/dev/zero of=3D/dev/da0 bs=3D1m count=3D1000 > 1000+0 records in > 1000+0 records out > 1048576000 bytes transferred in 4.026869 secs (260394859 bytes/sec) You can see how much CPU power your one core has by doing something like this: # kldload geom_zero # geli onetime -s 4096 gzero # sysctl kern.geom.zero.clear=3D0 # dd if=3D/dev/gzero.eli of=3D/dev/null bs=3D1m count=3D4096 /dev/gzero is similar to /dev/zero, but it is a GEOM provider. This experiment will show you how much encryption throughput you can get from one geli thread, without disk overhead. --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --fNagykWcDoSVAmSd Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFGrjt+ForvXbEpPzQRAg6sAJsHLS1W5ZZVjdXmW1uuPJEzgKoylgCfX8wy 5DupTlMsgJZjhdKcApiNLEU= =Iq2i -----END PGP SIGNATURE----- --fNagykWcDoSVAmSd-- From owner-freebsd-geom@FreeBSD.ORG Mon Jul 30 20:35:41 2007 Return-Path: Delivered-To: geom@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 19A1716A419; Mon, 30 Jul 2007 20:35:41 +0000 (UTC) (envelope-from etc@fluffles.net) Received: from auriate.fluffles.net (cust.95.160.adsl.cistron.nl [195.64.95.160]) by mx1.freebsd.org (Postfix) with ESMTP id C0C6A13C45E; Mon, 30 Jul 2007 20:35:40 +0000 (UTC) (envelope-from etc@fluffles.net) Received: from 195-241-125-45.dsl.ip.tiscali.nl ([195.241.125.45] helo=[10.0.0.18]) by auriate.fluffles.net with esmtpa (Exim 4.66 (FreeBSD)) (envelope-from ) id 1IFbxc-000MqH-KF; Mon, 30 Jul 2007 22:35:24 +0200 Message-ID: <46AE4B94.8010107@fluffles.net> Date: Mon, 30 Jul 2007 22:35:32 +0200 From: Fluffles User-Agent: Thunderbird 2.0.0.5 (X11/20070716) MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <46AA50B4.9080901@fluffles.net> <20070727210032.0140413C457@mx1.freebsd.org> <20070730192654.GO1092@garage.freebsd.pl> In-Reply-To: <20070730192654.GO1092@garage.freebsd.pl> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Cc: geom@FreeBSD.org, Dominic Bishop Subject: Re: Increasing GELI performance X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jul 2007 20:35:41 -0000 Pawel Jakub Dawidek wrote: > On Fri, Jul 27, 2007 at 10:00:35PM +0100, Dominic Bishop wrote: > >> I just tried your suggestion of geli on the raw device and it is no better >> at all: >> >> dd if=/dev/da0.eli of=/dev/null bs=1m count=1000 >> 1000+0 records in >> 1000+0 records out >> 1048576000 bytes transferred in 29.739186 secs (35259069 bytes/sec) >> >> dd if=/dev/zero of=/dev/da0.eli bs=1m count=1000 >> 1000+0 records in >> 1000+0 records out >> 1048576000 bytes transferred in 23.501061 secs (44618241 bytes/sec) >> >> Using top -S with 1s refresh to list the geli processes whilst doing this it >> seems only one of them is doing anything at any given time, the others are >> sitting in a state of "geli:w", I assume that is a truncation of something, >> maybe geli:wait at a guess. >> > > No matter how many cores/cpus you have if you run single-threaded > application. What you do exactly is: > 1. Send read of 128kB. > 2. One of geli threads picks it up, decrypts and sends it back. > 3. Send next read of 128kB. > 4. One of geli threads picks it up, decrypts and sends it back. > ... > > All threads will be used when there are more threads accessing provider. > But isn't it true that the UFS filesystem utilizes read-ahead and with that a multiple I/O queue depth (somewhere between 7 to 9 queued I/O's) - even when using something like dd to sequentially read a file on a mounted filesystem ? Then this read-ahead will cause multiple I/O request coming in and geom_eli can use multiple threads to maximize I/O throughput. Maybe Dominic can try playing with the "vfs.read_max" sysctl variable. - Veronica From owner-freebsd-geom@FreeBSD.ORG Tue Jul 31 11:46:48 2007 Return-Path: Delivered-To: geom@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 40B4616A418 for ; Tue, 31 Jul 2007 11:46:48 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (arm132.internetdsl.tpnet.pl [83.17.198.132]) by mx1.freebsd.org (Postfix) with ESMTP id CC31E13C45D for ; Tue, 31 Jul 2007 11:46:47 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 2179A487F7; Tue, 31 Jul 2007 13:46:46 +0200 (CEST) Received: from localhost (154.81.datacomsa.pl [195.34.81.154]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id BD599487F5; Tue, 31 Jul 2007 13:46:41 +0200 (CEST) Date: Tue, 31 Jul 2007 13:45:56 +0200 From: Pawel Jakub Dawidek To: Fluffles Message-ID: <20070731114555.GQ1092@garage.freebsd.pl> References: <46AA50B4.9080901@fluffles.net> <20070727210032.0140413C457@mx1.freebsd.org> <20070730192654.GO1092@garage.freebsd.pl> <46AE4B94.8010107@fluffles.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="/d7X7C0hV/blnKmH" Content-Disposition: inline In-Reply-To: <46AE4B94.8010107@fluffles.net> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 7.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-2.6 required=3.0 tests=BAYES_00 autolearn=ham version=3.0.4 Cc: geom@FreeBSD.org, Dominic Bishop Subject: Re: Increasing GELI performance X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jul 2007 11:46:48 -0000 --/d7X7C0hV/blnKmH Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jul 30, 2007 at 10:35:32PM +0200, Fluffles wrote: > Pawel Jakub Dawidek wrote: > >No matter how many cores/cpus you have if you run single-threaded > >application. What you do exactly is: > >1. Send read of 128kB. > >2. One of geli threads picks it up, decrypts and sends it back. > >3. Send next read of 128kB. > >4. One of geli threads picks it up, decrypts and sends it back. > >... > > > >All threads will be used when there are more threads accessing provider. > > =20 >=20 > But isn't it true that the UFS filesystem utilizes read-ahead and with=20 > that a multiple I/O queue depth (somewhere between 7 to 9 queued I/O's)= =20 > - even when using something like dd to sequentially read a file on a=20 > mounted filesystem ? Then this read-ahead will cause multiple I/O=20 > request coming in and geom_eli can use multiple threads to maximize I/O= =20 > throughput. Maybe Dominic can try playing with the "vfs.read_max" sysctl= =20 > variable. You are right in general, but if you reread e-mail I was answering to, you will see that the author was reading from/writing to GEOM provider, not file system. --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --/d7X7C0hV/blnKmH Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFGryDzForvXbEpPzQRAgjpAKCMxdqXm0qNVI58S0TzWhusIsJoLwCaAy6P GUF5MHF9O83N+6ieA2Yo/N4= =rcFt -----END PGP SIGNATURE----- --/d7X7C0hV/blnKmH-- From owner-freebsd-geom@FreeBSD.ORG Tue Jul 31 13:25:51 2007 Return-Path: Delivered-To: geom@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 67C9916A417; Tue, 31 Jul 2007 13:25:51 +0000 (UTC) (envelope-from etc@fluffles.net) Received: from auriate.fluffles.net (cust.95.160.adsl.cistron.nl [195.64.95.160]) by mx1.freebsd.org (Postfix) with ESMTP id 075A913C4B0; Tue, 31 Jul 2007 13:25:50 +0000 (UTC) (envelope-from etc@fluffles.net) Received: from 195-241-125-45.dsl.ip.tiscali.nl ([195.241.125.45] helo=[10.0.0.18]) by auriate.fluffles.net with esmtpa (Exim 4.66 (FreeBSD)) (envelope-from ) id 1IFrjD-000Lrf-7r; Tue, 31 Jul 2007 15:25:35 +0200 Message-ID: <46AF3857.30600@fluffles.net> Date: Tue, 31 Jul 2007 15:25:43 +0200 From: Fluffles User-Agent: Thunderbird 2.0.0.5 (X11/20070716) MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <46AA50B4.9080901@fluffles.net> <20070727210032.0140413C457@mx1.freebsd.org> <20070730192654.GO1092@garage.freebsd.pl> <46AE4B94.8010107@fluffles.net> <20070731114555.GQ1092@garage.freebsd.pl> In-Reply-To: <20070731114555.GQ1092@garage.freebsd.pl> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Cc: geom@FreeBSD.org, Dominic Bishop Subject: Re: Increasing GELI performance X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jul 2007 13:25:51 -0000 Pawel Jakub Dawidek wrote: > On Mon, Jul 30, 2007 at 10:35:32PM +0200, Fluffles wrote: > >> Pawel Jakub Dawidek wrote: >> >>> No matter how many cores/cpus you have if you run single-threaded >>> application. What you do exactly is: >>> 1. Send read of 128kB. >>> 2. One of geli threads picks it up, decrypts and sends it back. >>> 3. Send next read of 128kB. >>> 4. One of geli threads picks it up, decrypts and sends it back. >>> ... >>> >>> All threads will be used when there are more threads accessing provider. >>> >>> >> But isn't it true that the UFS filesystem utilizes read-ahead and with >> that a multiple I/O queue depth (somewhere between 7 to 9 queued I/O's) >> - even when using something like dd to sequentially read a file on a >> mounted filesystem ? Then this read-ahead will cause multiple I/O >> request coming in and geom_eli can use multiple threads to maximize I/O >> throughput. Maybe Dominic can try playing with the "vfs.read_max" sysctl >> variable. >> > > You are right in general, but if you reread e-mail I was answering to, > you will see that the author was reading from/writing to GEOM provider, > not file system. > Ah yes you're right. Though he might also have tested on a mounted filesystem, his email does not explicitely specify this. So he should re-run his experiment: geli onetime /dev/da0 newfs /dev/da0 mkdir /test mount /dev/da0 /test dd if=/dev/zero of=/test/zerofile.000 bs=1m count=2000 (write score) dd if=/test/zerofile.000 of=/dev/null bs=1m (read score) That *should* give him higher performance. Also, dominic might try increasing the block size, using "newfs -b 32768 /dev/da0". Without it, it seems it hits a performance roof on about ~130MB/s. I once wrote about this on the mailinglist where Bruce Evans questioned the usefulness of a blocksize higher than 16KB. I still have to investigate this further, it's on my to-do list. Deeplink: http://lists.freebsd.org/pipermail/freebsd-fs/2006-October/002298.html I tried to recreate a test scenario myself, using 4 disks in a striping configuration (RAID0), first reading and writing on the raw .eli device, then on a mounted filesystem: ** raw device # dd if=/dev/stripe/data.eli of=/dev/null bs=1m count=2000 2097152000 bytes transferred in 57.949793 secs (36189120 bytes/sec) # dd if=/dev/zero of=/dev/stripe/data.eli bs=1m count=2000 1239416832 bytes transferred in 35.168374 secs (35242370 bytes/sec) ** mounted default newfs # dd if=/dev/zero of=/test/zerofile.000 bs=1m count=2000 2097152000 bytes transferred in 47.843614 secs (43833478 bytes/sec) # dd if=/test/zerofile.000 of=/dev/null bs=1m count=2000 2097152000 bytes transferred in 50.328749 secs (41669067 bytes/sec) This was on a simple single core Sempron K8 CPU, but already there's a difference with the multiple queue depth VFS/UFS provides. Good luck Dominic be sure to post again when you have new scores! I'm interested to see how far you can push GELI with a quadcore. :) - Veronica From owner-freebsd-geom@FreeBSD.ORG Tue Jul 31 17:27:37 2007 Return-Path: Delivered-To: geom@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CD8D516A417 for ; Tue, 31 Jul 2007 17:27:37 +0000 (UTC) (envelope-from dom@bishnet.net) Received: from carrick.bishnet.net (unknown [IPv6:2001:618:400::54ea:1138]) by mx1.freebsd.org (Postfix) with ESMTP id C755513C45D for ; Tue, 31 Jul 2007 17:27:36 +0000 (UTC) (envelope-from dom@bishnet.net) Received: from cpc1-warw1-0-0-cust384.sol2.cable.ntl.com ([86.20.169.129] helo=magellan.dom.bishnet.net ident=mailnull) by carrick.bishnet.net with esmtps (TLSv1:AES256-SHA:256) (Exim 4.66 (FreeBSD)) (envelope-from ) id 1IFvVK-000BCI-CV for geom@FreeBSD.org; Tue, 31 Jul 2007 18:27:30 +0100 Received: from deimos.dom.bishnet.net ([192.168.3.100] helo=deimos) by magellan.dom.bishnet.net with esmtp (Exim 4.67 (FreeBSD)) (envelope-from ) id 1IFvVJ-000JFl-T2 for geom@FreeBSD.org; Tue, 31 Jul 2007 18:27:29 +0100 From: "Dominic Bishop" To: Date: Tue, 31 Jul 2007 18:27:48 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook, Build 11.0.5510 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3138 Thread-Index: AcfTdlDiek5NvbSQQAi9Bc8kiEuhngAIa+Cw In-Reply-To: <46AF3857.30600@fluffles.net> X-Bishnet-MailScanner-Information: Contact postmaster@bishnet.net X-Bishnet-MailScanner-VirusCheck: Found to be clean X-Bishnet-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=-0.955, required 5, autolearn=not spam, AWL 1.51, BAYES_00 -2.60, FORGED_RCVD_HELO 0.14, SPF_PASS -0.00) X-Bishnet-MailScanner-From: dom@bishnet.net Message-Id: <20070731172736.C755513C45D@mx1.freebsd.org> Cc: Subject: RE: Increasing GELI performance X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jul 2007 17:27:37 -0000 > -----Original Message----- > From: Fluffles [mailto:etc@fluffles.net] > Sent: 31 July 2007 14:26 > To: Pawel Jakub Dawidek > Cc: Dominic Bishop; geom@FreeBSD.org > Subject: Re: Increasing GELI performance > > Pawel Jakub Dawidek wrote: > > On Mon, Jul 30, 2007 at 10:35:32PM +0200, Fluffles wrote: > > > >> Pawel Jakub Dawidek wrote: > >> > >>> No matter how many cores/cpus you have if you run single-threaded > >>> application. What you do exactly is: > >>> 1. Send read of 128kB. > >>> 2. One of geli threads picks it up, decrypts and sends it back. > >>> 3. Send next read of 128kB. > >>> 4. One of geli threads picks it up, decrypts and sends it back. > >>> ... > >>> > >>> All threads will be used when there are more threads > accessing provider. > >>> > >>> > >> But isn't it true that the UFS filesystem utilizes read-ahead and > >> with that a multiple I/O queue depth (somewhere between 7 > to 9 queued > >> I/O's) > >> - even when using something like dd to sequentially read a > file on a > >> mounted filesystem ? Then this read-ahead will cause multiple I/O > >> request coming in and geom_eli can use multiple threads to > maximize > >> I/O throughput. Maybe Dominic can try playing with the > "vfs.read_max" > >> sysctl variable. > >> > > > > You are right in general, but if you reread e-mail I was > answering to, > > you will see that the author was reading from/writing to GEOM > > provider, not file system. > > > > Ah yes you're right. Though he might also have tested on a > mounted filesystem, his email does not explicitely specify > this. So he should re-run his experiment: > > geli onetime /dev/da0 > newfs /dev/da0 > mkdir /test > mount /dev/da0 /test > dd if=/dev/zero of=/test/zerofile.000 bs=1m count=2000 (write > score) dd if=/test/zerofile.000 of=/dev/null bs=1m (read score) > > That *should* give him higher performance. Also, dominic > might try increasing the block size, using "newfs -b 32768 > /dev/da0". Without it, it seems it hits a performance roof on > about ~130MB/s. I once wrote about this on the mailinglist > where Bruce Evans questioned the usefulness of a blocksize > higher than 16KB. I still have to investigate this further, > it's on my to-do list. Deeplink: > http://lists.freebsd.org/pipermail/freebsd-fs/2006-October/002298.html > > I tried to recreate a test scenario myself, using 4 disks in > a striping configuration (RAID0), first reading and writing > on the raw .eli device, then on a mounted filesystem: > > ** raw device > # dd if=/dev/stripe/data.eli of=/dev/null bs=1m count=2000 > 2097152000 bytes transferred in 57.949793 secs (36189120 > bytes/sec) # dd if=/dev/zero of=/dev/stripe/data.eli bs=1m count=2000 > 1239416832 bytes transferred in 35.168374 secs (35242370 bytes/sec) > > ** mounted default newfs > # dd if=/dev/zero of=/test/zerofile.000 bs=1m count=2000 > 2097152000 bytes transferred in 47.843614 secs (43833478 > bytes/sec) # dd if=/test/zerofile.000 of=/dev/null bs=1m > count=2000 2097152000 bytes transferred in 50.328749 secs > (41669067 bytes/sec) > > This was on a simple single core Sempron K8 CPU, but already > there's a difference with the multiple queue depth VFS/UFS provides. > > Good luck Dominic be sure to post again when you have new > scores! I'm interested to see how far you can push GELI with > a quadcore. :) > > - Veronica > Unfortunately I can no longer do testing as I hit a deadline and had to put the boxes into production, so can only do non destructive tests now. It is interesting you should mention non default block sizes though as I did try this and got some very nasty problems. I initially tried the following settings: geli init -s 8192 /dev/da0p2 newfs -b 65536 -f 8192 -n -U -m1 /dev/da0p2.eli The reasoning for this is that newfs uses the geli sector size for fragment size anyway by default (quite logically) and the newfs man page suggested a ratio of 8:1 block:fragment size as an ideal ratio, also newfs wouldn't allow a larger than 65536 block, so overall this seemed like the ideal setting for best performance. My data load would be all files into MBs so any space lost by larger fragmentss was pretty irrelevant. This all *appeared* to be working fine, however after transferring a couple of hundred GB onto the filesystem, unmounting and doing a fsck I got a ton of filesystem errors of all kinds, a few examples: ** Phase 1 - Check Blocks and Sizes PARTIALLY ALLOCATED INODE I=3523664 UNKNOWN FILE TYPE I=3523696 ** Phase 2 - Check Pathnames DIRECTORY CORRUPTED I=3523603 OWNER=dom MODE=40755 SIZE=32256 MTIME=Jul 29 15:10 2007 DIR=/DATA/TEST ** Phase 3 - Check Connectivity UNREF DIR I=18855983 OWNER=dom MODE=40755 SIZE=1024 MTIME=Jul 29 14:39 2007 ** Phase 4 - Check Reference Counts LINK COUNT FILE I=3523760 OWNER=4017938749 MODE=24015 SIZE=0 MTIME=Jan 1 00:00 1970 COUNT -9594 SHOULD BE 1 I was running out of time by this point so I went back to settings I used in the past on other boxes, taken from the geli man page a sector size of 4096 to geli and no settings for newfs, leaving it to use a block size of 16384 and fragment size of 4096. After putting a few hundred GB on the filesystem a fsck passed perfectly. Unfortunately I didn't have time to see which of the geli/newfs settings specifically was causing a problem or if it was just the combination of the two. I am reasonably confident it wasn't a hardware issue as this happened the same on 4 machines, 3 of these were identical dual 3.0Ghz netburst xeons running RELENG_6 i386, the other was Quad 1.6Ghz conroe based xeon cores running RELENG_6 amd64, the src was RELENG_6 as of the 27th on all machines. At no point were any errors logged to messages or console.log The one common factor amongst all machines was a 3ware 9550SXU-12 as the underlying raw device, but as it has worked flawlessly with the different geli/newfs parameters I highly doubt this was the problem. Regards Dominic Bishop