From owner-freebsd-fs@FreeBSD.ORG Thu Jan 13 22:31:45 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7CB7D1065674; Thu, 13 Jan 2011 22:31:45 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.garage.freebsd.pl (60.wheelsystems.com [83.12.187.60]) by mx1.freebsd.org (Postfix) with ESMTP id E6B678FC16; Thu, 13 Jan 2011 22:31:43 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 7629745C9C; Thu, 13 Jan 2011 23:31:41 +0100 (CET) Received: from localhost (89-73-192-49.dynamic.chello.pl [89.73.192.49]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 2834C45C8A; Thu, 13 Jan 2011 23:31:35 +0100 (CET) Date: Thu, 13 Jan 2011 23:31:25 +0100 From: Pawel Jakub Dawidek To: Chris Forgeron Message-ID: <20110113223125.GA2330@garage.freebsd.pl> References: <20101213214556.GC2038@garage.freebsd.pl> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="0F1p//8PRICkK4MW" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-0.6 required=4.5 tests=BAYES_00,RCVD_IN_SORBS_DUL autolearn=no version=3.0.4 Cc: "freebsd-fs@freebsd.org" , "freebsd-current@freebsd.org" Subject: Re: My ZFS v28 Testing Experience X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Jan 2011 22:31:45 -0000 --0F1p//8PRICkK4MW Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Jan 12, 2011 at 11:03:19PM -0400, Chris Forgeron wrote: > I've been testing out the v28 patch code for a month now, and I've yet to= report any real issues other than what is mentioned below.=20 >=20 > I'll detail some of the things I've tested, hopefully the stability of v2= 8 in FreeBSD will convince others to give it a try so the final release of = v28 will be as solid as possible. >=20 > I've been using FreeBSD 9.0-CURRENT as of Dec 12th, and 8.2PRE as of Dec = 16th >=20 > What's worked well: >=20 > - I've made and destroyed small raidz's (3-5 disks), large 26 disk raid-1= 0's, and a large 20 disk raid-50. > - I've upgraded from v15, zfs 4, no issues on the different arrays noted = above > - I've confirmed that a v15 or v28 pool will import into Solaris 11 Expre= ss, and vice versa, with the exception about dual log or cache devices note= d below.=20 > - I've run many TB of data through the ZFS storage via benchmarks from my= VM's connected via NFS, to simple copies inside the same pool, or copies f= rom one pool to another.=20 > - I've tested pretty much every compression level, and changing them as I= tweak my setup and try to find the best blend. > - I've added and subtracted many a log and cache device, some in failed s= tates from hot-removals, and the pools always stayed intact. Thank you very much for all your testing, that's really a valuable contribution. I'll be happy to work with you on tracking down the bottleneck in ZFSv28. > Issues: >=20 > - Import of pools with multiple cache or log devices. (May be a very mino= r point) >=20 > A v28 pool created in Solaris 11 Express with 2 or more log devices, or 2= or more cache devices won't import in FreeBSD 9. This also applies to a po= ol that is created in FreeBSD, is imported in Solaris to have the 2 log dev= ices added there, then exported and attempted to be imported back in FreeBS= D. No errors, zpool import just hangs forever. If I reboot into Solaris, im= port the pool, remove the dual devices, then reboot into FreeBSD, I can the= n import the pool without issue. A single cache, or log device will import = just fine. Unfortunately I deleted my witness-enabled FreeBSD-9 drive, so I= can't easily fire it back up to give more debug info. I'm hoping some kind= soul will attempt this type of transaction and report more detail to the l= ist. >=20 > Note - I just decided to try adding 2 cache devices to a raidz pool in Fr= eeBSD, export, and then importing, all without rebooting. That seems to wor= k. BUT - As soon as you try to reboot FreeBSD with this pool staying active= , it hangs on boot. Booting into Solaris, removing the 2 cache devices, the= n booting back into FreeBSD then works. Something is kept in memory between= exporting then importing that allows this to work. =20 Unfortunately I'm unable to reproduce this. It works for me with 2 cache and 2 log vdevs. I tried to reboot, etc. My test exactly looks like this: # zpool create tank raidz ada0 ada1 # zpool add tank cache ada0 ada1 # zpool export tank # kldunload zfs # zpool import tank # reboot > - Speed. (More of an issue, but what do we do?) >=20 > Wow, it's much slower than Solaris 11 Express for transactions. I do unde= rstand that Solaris will have a slight advantage over any port of ZFS. All = of my speed tests are made with a kernel without debug, and yes, these are = -CURRENT and -PRE releases, but the speed difference is very large. Before we go any further could you please confirm that you commented out this line in sys/modules/zfs/Makefile: CFLAGS+=3D-DDEBUG=3D1 This turns all kind of ZFS debugging and slows it down a lot, but for the correctness testing is invaluable. This will be turned off once we import ZFS into FreeBSD-CURRENT. BTW. In my testing Solaris 11 Express is much, much slower than FreeBSD/ZFSv28. And by much I mean two or more times in some tests. I was wondering if they have some debug turned on in Express. > At first, I thought it may be more of an issue with the ix0/Intel X520DA2= 10Gbe drivers that I'm using, since the bulk of my tests are over NFS (I'm= going to use this as a SAN via NFS, so I test in that environment).=20 >=20 > But - I did a raw cp command from one pool to another of several TB. I ex= ecuted the same command under FreeBSD as I did under Solaris 11 Express. Wh= en executed in FreeBSD, the copy took 36 hours. With a fresh destination po= ol of the same settings/compression/etc under Solaris, the copy took 7.5 ho= urs.=20 When you turn off compression (because it turns all-zero blocks into holes) you can test it by simply: # dd if=3D/dev/zero of=3D//zero bs=3D1m --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --0F1p//8PRICkK4MW Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAk0vfT0ACgkQForvXbEpPzTvxwCgib/g/1XuwWzSXj325r/keAwA sHMAn2hW/6V3HJU2mFd3YKdvARFy0xv3 =sy15 -----END PGP SIGNATURE----- --0F1p//8PRICkK4MW--