From owner-freebsd-fs@FreeBSD.ORG Sun Jun 21 02:18:41 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0C266106566C for ; Sun, 21 Jun 2009 02:18:41 +0000 (UTC) (envelope-from dan.naumov@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.245]) by mx1.freebsd.org (Postfix) with ESMTP id B4DFA8FC12 for ; Sun, 21 Jun 2009 02:18:40 +0000 (UTC) (envelope-from dan.naumov@gmail.com) Received: by an-out-0708.google.com with SMTP id c3so1114449ana.13 for ; Sat, 20 Jun 2009 19:18:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=zKi4tlT7swuDCfrpmD6GKndxMzdfaWjRhEukODozAvA=; b=SyLYhh7dFkF+39UXB7396yJ8HQt+UVHr3p5F1GXzsZYGzJY6VAWlDkZNr5IDPBdIqR LCFkNpjPICVTrTou96vGUKy7mKGv/JYXhKgowGT9AObZBlsKt2HZcwQJHjByVgVRJfPv T4dEsX7ev/7uKt9/gPcBGtYfROMq7tiEUYT5g= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=ix+8lfLk3MwnUN5y+1C7FcrdOoNGP4G1IvGDkyw8eBsr0h2/yYtX3p4XWNygNspo6F bc42WFLEmOZ3MN9PV5BVjm36HocZGEYXBpFLnQf8GtyJcFY8glWHiDz9JaCwio5WZC8o 52dSGfckm56997+HK20QDb2t0WSP0ufwY71ZE= MIME-Version: 1.0 Received: by 10.100.126.15 with SMTP id y15mr6079406anc.71.1245550720009; Sat, 20 Jun 2009 19:18:40 -0700 (PDT) In-Reply-To: <3c1674c90906201808t1854dd46n82213fbd0c1c254c@mail.gmail.com> References: <20090620231130.GA88907@owl.midgard.homeip.net> <3c1674c90906201808t1854dd46n82213fbd0c1c254c@mail.gmail.com> Date: Sun, 21 Jun 2009 05:18:39 +0300 Message-ID: From: Dan Naumov To: Kip Macy Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org, Erik Trulsson Subject: Re: ufs2 / softupdates / ZFS / disk write cache X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jun 2009 02:18:41 -0000 I decided to do some performance tests of my own, "bonnie -s 4096" was used to obtain the results. Note that these results should be used to compare "write cache on" to "write cache off" and not to compare UFS2 vs ZFS, as the testing was done on different parts of the same physical disk (the UFS2 partition resides on the first 16gb of disk and ZFS pool takes the remaining ~1,9tb) and I am also using rather conservative ZFS tunables. UFS2 with write cache: -------Sequential Output-------- ---Sequential Input-- --Rand= om-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seek= s--- Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec = %CPU 4096 55457 95.9 91630 46.7 36264 37.5 46565 74.0 84751 33.7 164.3 = 10.3 UFS2 without write cache: -------Sequential Output-------- ---Sequential Input-- --Rand= om-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seek= s--- Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec = %CPU 4096 4938 46.9 4685 18.0 4288 21.8 17453 34.0 74232 31.6 165.0 = 9.9 As we can clearly see, the performance diffence between having disk cache enabled and disabled is _ENORMOUS_. In the case of sequential block write on UFS2, the performance loss is a staggering 94,89%. More surprinsingly, even reading seems to be affected in a noticable way, per char reads suffer a 62,62% penalty while block reads take a 12,42% hit. Moving on to testing ZFS with and without disk cache enabled: ZFS with write cache (384M ARC, 1GB max kmem): -------Sequential Output-------- ---Sequential Input-- --Rand= om-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seek= s--- Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec = %CPU 4096 25972 66.1 45026 40.6 34269 36.0 46371 86.5 93973 34.6 84.5 = 8.5 ZFS without write cache (384M ARC, 1GB max kmem): -------Sequential Output-------- ---Sequential Input-- --Rand= om-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seek= s--- Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec = %CPU 4096 2399 6.7 2258 3.5 2290 3.9 34380 66.1 85971 32.8 56.7 = 6.1 =09 Uh oh.... After some digging around, I found the following quote: "ZFS is designed to work with storage devices that manage a disk-level cache. ZFS commonly asks the storage device to ensure that data is safely placed on stable storage by requesting a cache flush." at http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide I guess this might be somewhat related to why in the "disk cache disabled" scenario, ZFS suffers bigger losses than UFS2. It is quite obvious at this point that disabling disk cache in order have softupdates live in harmony with disks "lying" about whether disk cache contents have actually been committed to the disk in not in any way, shape or form a viable solution to the problem. On a sidenote, is there any way I can test whether *MY* disk is truthful about writing cache to disk or not? In the past (this was during my previous foray into the FreeBSD world, circa-2001/2002) I have suffered severe data corruption (leading to an unbootable system) using UFS2 + softupdates on 2 different occasions due to power losses and this past experience has me very worried about the proper way to configure my system to avoid such incidents in the future. - Sincerely, Dan Naumov On Sun, Jun 21, 2009 at 4:08 AM, Kip Macy wrote: >> >> My guess is that it will be quite noticable, but that is only a guess. >> (Keep in mind that UFS+softupdates does quite a bit of write-caching on = its >> own, so just switching to ZFS is unlikely to improve write performance >> significantly compared to using UFS.) > > > That all depends on how much the drive relies on the write cache for > batching writes to disk. Soft updates does a lot of small random > writes for metadata updates which will likely be heavily penalized by > the absence of write caching. On my SSD, which unfortunately turned > out to be camera grade flash, with FFS the system was unusable when > doing large numbers of metadata updates, svn checkouts would take > hours. I postulated that ZFS would map well to the large erase blocks, > so I destroyed /usr and recreated a zpool in its place. I now get > random write performance =A0better than FFS, "I lived happily ever > after." > > I don't know if ZFS will provide the same benefit in your situation. > My point is just that FFS+SU and ZFS are "apples and oranges." > > Please note that I've taken -stable off of the the CC, ZFS has been > getting a lot of mailing list traffic lately and I've been hearing > groans from certain quarters about it drowning out other discussions. > Let's try to keep the discussions to freebsd-fs. > > > Thanks, > Kip