From owner-freebsd-fs@FreeBSD.ORG Wed Feb 16 03:35:22 2005 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2D06016A4CE for ; Wed, 16 Feb 2005 03:35:22 +0000 (GMT) Received: from wproxy.gmail.com (wproxy.gmail.com [64.233.184.199]) by mx1.FreeBSD.org (Postfix) with ESMTP id 70D3D43D41 for ; Wed, 16 Feb 2005 03:35:21 +0000 (GMT) (envelope-from sdrhodus@gmail.com) Received: by wproxy.gmail.com with SMTP id 69so25775wri for ; Tue, 15 Feb 2005 19:35:20 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:references; b=DCd6WEMU4id+2PQeSGfQLe381Q8IMgcveUuXC4ll6r4z68Qr4WPqX9NsJmPzmRhsgdXBZ/xyusnBRSrg0upNJM/3Bg0XtBzJAAzoHKmXyXzfnfAhfxh+2inBYB6EGAGFWfdBpHtDyId8f7g19UM5NEzQ/jn67JhXUycWNbSSixE= Received: by 10.54.43.28 with SMTP id q28mr96460wrq; Tue, 15 Feb 2005 19:35:20 -0800 (PST) Received: by 10.54.25.20 with HTTP; Tue, 15 Feb 2005 19:35:20 -0800 (PST) Message-ID: Date: Tue, 15 Feb 2005 22:35:20 -0500 From: David Rhodus To: Scott Long In-Reply-To: <420AD3A7.3000102@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit References: <20050210030119.GD29396@alzatex.com> <420AD3A7.3000102@freebsd.org> cc: freebsd-fs@freebsd.org Subject: Re: Journalling FS and Soft Updates comparision X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: drhodus@machdep.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Feb 2005 03:35:22 -0000 On Wed, 09 Feb 2005 20:23:19 -0700, Scott Long wrote: > Loren M. Lang wrote: > > > Traditionally, filesystems have been designed with the idea that the > > data will always be written to disk safely and not much effort was put > > into making then > > > > Journalling Filesystems and Soft Updates are two different techniques > > designed to solve the problem of keeping data consistent even in the > > case of a major interruption like a power blackout. Both work solely on > > the meta data, not the real data. > > This isn't always true. There are journaling implementations that > journal the data as well as the metadata. > > > This means increasing a file's size > > is protected, but not neccessarily the data that's being written. (Does > > this also mean that the data will be written to free space before the > > file size is increased so extraneous data won't be left in the file?) > > Journally works be recording in a special place on the hard drive called > > the journal every meta data change that it is about to execute before it > > does it, then it updates all the meta data and finally marks the journal > > completed. Soft updates are simply a way to order meta data so that it > > happens in a safe order. An example is moving file a from directory x to > > directory y would first delete file a from dir x, then add it to dir y. > > If a crash happens in the middle, then the data becomes lost right? > > > > Part of the reordering of metadata in softupdates involves generating > dependency graphs that prevent data loss like this. > In all filesystems there is a dependency graph for action ordering. A filesystem is a set of events and actions. E.g. if you think of a filesystem in terms of this graph, there are node relationships with commutative and assoviative properties. Softupdates takes specific advantage of the commutative and associative properties of these node updates and imposes queue order semantics on scheduling of actions. > > Now this shouldn't be a big deal since it's harmless to anything else, > > just some free space is eaten up. Since all meta data updates have this > > same kind of harmless behavior, that why fsck can be done in the > > background now instead of foreground. > > The theory of softupdates is that whatever metadata made it to disk > before shutdown/crash is consistent enough to be trusted after just a > quick preen. The rest of the background checking is just to clean up > blocks that became unallocated but weren't committed. > > > > > Now comparing the two, perfomance wise journalling has an advantage > > since every group of meta data updates that are written to the journal > > at the same time can be reordered to optimize the disk performance. The > > disk head just has to move across the disk in order instead of seeking > > back and forth. Now this performance is usually lost because the > > journal is constantly needing to be updated and it probably lies in one > > small ares of the disk. The other benefit of the journal is very quick > > fsck times since all it has do to it see what the journal was updating > > and make sure it all completed. Soft updates still require a full fsck, > > but since it can be done in the background unlike journalling, it mean > > even faster startup time, but more cpu and i/o time spent on it. Now if > > the journal of a journalling fs could be kept somewhere else, say, in > > some kind of nvram, then journalling might be overall more efficient as > > far as disk i/o and cpu time than soft updates. > > Performance between softupdates and journalling is still hotly debated, > and your statements border on the 'flaimbait' side of the argument. > > > > > I'm mainly just trying to get an understanding of these two techniques, > > not neccessarily saying one is better. In the real world, it's probably > > very dependent on many other things like lot of random access vs. > > sequential, many files and file ops per seconds, vs. mostly read-only > > with noatime set, etc. > > Softupdates really aren't a whole lot different from journalling. Both No, journalling and soft updates are orthogonal technologies; they do not solve the same problem space, although there is some minor overlap. Soft updates is not able to solve all the problems which journaling can. > turn metadata operations into a sequence of ordered atomic updates. The > only difference is that journalling writes these updates to the on-disk > journal right away and then commits them later on, and softupdates keeps > (most of) them in RAM and then commits them later on. You are correct > that journalling has a key advantage in that a fsck, either foreground > or background, is not strictly required after an unexpected shutdown. > For further information, I'd suggest reading: > > http://www.mckusick.com/softdep/index.html > > Scott -- -David Steven David Rhodus From owner-freebsd-fs@FreeBSD.ORG Wed Feb 16 03:51:46 2005 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 332F816A4CE for ; Wed, 16 Feb 2005 03:51:46 +0000 (GMT) Received: from mail.freebsd.org.cn (dns3.freebsd.org.cn [61.129.66.75]) by mx1.FreeBSD.org (Postfix) with SMTP id 77C2043D1D for ; Wed, 16 Feb 2005 03:51:44 +0000 (GMT) (envelope-from delphij@frontfree.net) Received: (qmail 35040 invoked by uid 0); 16 Feb 2005 03:43:03 -0000 Received: from unknown (HELO beastie.frontfree.net) (219.239.99.7) by mail.freebsd.org.cn with SMTP; 16 Feb 2005 03:43:03 -0000 Received: from localhost (localhost.frontfree.net [127.0.0.1]) by beastie.frontfree.net (Postfix) with ESMTP id AF1CB133689; Wed, 16 Feb 2005 11:51:40 +0800 (CST) Received: from beastie.frontfree.net ([127.0.0.1]) by localhost (beastie.frontfree.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 89659-18; Wed, 16 Feb 2005 11:51:29 +0800 (CST) Received: from localhost.localdomain (unknown [61.135.152.194]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by beastie.frontfree.net (Postfix) with ESMTP id B5BE71335CD; Wed, 16 Feb 2005 11:51:26 +0800 (CST) From: Xin LI To: drhodus@machdep.com In-Reply-To: References: <20050210030119.GD29396@alzatex.com> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-+/P+cvkImi35BVrjmzcD" Organization: The FreeBSD Simplified Chinese Project Date: Wed, 16 Feb 2005 11:50:10 +0800 Message-Id: <1108525810.676.21.camel@spirit> Mime-Version: 1.0 X-Mailer: Evolution 2.0.3 FreeBSD GNOME Team Port X-Virus-Scanned: by amavisd-new at frontfree.net cc: freebsd-fs@freebsd.org cc: Scott Long Subject: Re: Journalling FS and Soft Updates comparision X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: delphij@delphij.net List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Feb 2005 03:51:46 -0000 --=-+/P+cvkImi35BVrjmzcD Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi, David, =E5=9C=A8 2005-02-15=E4=BA=8C=E7=9A=84 22:35 -0500=EF=BC=8CDavid Rhodus=E5= =86=99=E9=81=93=EF=BC=9A > > Softupdates really aren't a whole lot different from journalling. Both >=20 > No, journalling and soft updates are orthogonal technologies; they > do not solve the same problem space, although there is some minor > overlap. Soft updates is not able to solve all the problems which > journaling can. Would you please give some examples? I think a file system journalling is different from what we have seen in applications, e.g. databases, since the primary goal of having a journal is to make the file system recoverable to last checkpoint, is that correct, or maybe I have missed something? Thanks! Cheers, --=20 Xin LI http://www.delphij.net/ --=-+/P+cvkImi35BVrjmzcD Content-Type: application/pgp-signature; name=signature.asc Content-Description: =?UTF-8?Q?=E8=BF=99=E6=98=AF=E4=BF=A1=E4=BB=B6=E7=9A=84=E6=95=B0?= =?UTF-8?Q?=E5=AD=97=E7=AD=BE=E5=90=8D=E9=83=A8?= =?UTF-8?Q?=E5=88=86?= -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (FreeBSD) iD8DBQBCEsLy/cVsHxFZiIoRAvUjAJ9zSU6zZr7O/Q1D/s2uPsSsAhfKjwCgigcB UqzWwdE5jUtDSbvCzLh7Nr0= =jv2Y -----END PGP SIGNATURE----- --=-+/P+cvkImi35BVrjmzcD-- From owner-freebsd-fs@FreeBSD.ORG Wed Feb 16 13:12:10 2005 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3BA2F16A4CE for ; Wed, 16 Feb 2005 13:12:10 +0000 (GMT) Received: from wproxy.gmail.com (wproxy.gmail.com [64.233.184.206]) by mx1.FreeBSD.org (Postfix) with ESMTP id C8BAC43D66 for ; Wed, 16 Feb 2005 13:12:09 +0000 (GMT) (envelope-from sdrhodus@gmail.com) Received: by wproxy.gmail.com with SMTP id 69so81849wri for ; Wed, 16 Feb 2005 05:12:09 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:references; b=DncbJTSa/vPnTLdf6IjAoVxiaBX+kf3upvaTn2kzZsR0cNvnBDF6ZWIIgpRERPbCgH/IHdC8Sivuma0S/ctG8W/QLDeaJXwZZDGrmJsYXii/CxNYa6fqc2gvEIpC3WJXAwKgX9nUMGsQzIcXpiYPilhNnAKJTHstBVnuKF0lt9k= Received: by 10.54.47.39 with SMTP id u39mr215127wru; Wed, 16 Feb 2005 05:12:09 -0800 (PST) Received: by 10.54.25.20 with HTTP; Wed, 16 Feb 2005 05:12:08 -0800 (PST) Message-ID: Date: Wed, 16 Feb 2005 13:12:08 +0000 From: David Rhodus To: delphij@delphij.net In-Reply-To: <1108525810.676.21.camel@spirit> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable References: <20050210030119.GD29396@alzatex.com> <1108525810.676.21.camel@spirit> cc: freebsd-fs@freebsd.org cc: Scott Long Subject: Re: Journalling FS and Soft Updates comparision X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: drhodus@machdep.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Feb 2005 13:12:10 -0000 On Wed, 16 Feb 2005 11:50:10 +0800, Xin LI wrote: > Hi, David, >=20 > =E5=9C=A8 2005-02-15=E4=BA=8C=E7=9A=84 22:35 -0500=EF=BC=8CDavid Rhodus= =E5=86=99=E9=81=93=EF=BC=9A > > > Softupdates really aren't a whole lot different from journalling. Bo= th > > > > No, journalling and soft updates are orthogonal technologies; they > > do not solve the same problem space, although there is some minor > > overlap. Soft updates is not able to solve all the problems which > > journaling can. >=20 > Would you please give some examples? I think a file system journalling > is different from what we have seen in applications, e.g. databases, > since the primary goal of having a journal is to make the file system > recoverable to last checkpoint, is that correct, or maybe I have missed > something? >=20 > Thanks! >=20 > Cheers, > -- > Xin LI http://www.delphij.net/ Don't try and think of a filesystem as a database, in practice a filesystem is not a database. If you consider all filesystems in the context of a graph, then what you end up with is: 1.) a graph 2.) a list of node relationships 3.) a finite state automation model for node conflict resolution Though here is a example of why a journalled filesystem performs betters. With todays disk drives a DC power failure can result in an entire track getting trashed. Soft updates optimizes for sector writing not track aligned writing. Journalling can journal on the basis of track sized extents. This is one case were a journalled filesystem can recover whereas soft updates could not, from a power failure, if resolution is needed. If this were to happen on a disk protected by soft updates, damage may have accrued. --=20 -David Steven David Rhodus From owner-freebsd-fs@FreeBSD.ORG Fri Feb 18 20:42:51 2005 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B752116A4CE for ; Fri, 18 Feb 2005 20:42:51 +0000 (GMT) Received: from sccrmhc13.comcast.net (sccrmhc13.comcast.net [204.127.202.64]) by mx1.FreeBSD.org (Postfix) with ESMTP id EB12543D2F for ; Fri, 18 Feb 2005 20:42:50 +0000 (GMT) (envelope-from tbonius@comcast.net) Received: from ostros (c-24-18-102-54.client.comcast.net[24.18.102.54]) by comcast.net (sccrmhc13) with SMTP id <2005021820424901600sbacfe>; Fri, 18 Feb 2005 20:42:50 +0000 Message-ID: <00f201c515fa$8caef250$4300a8c0@home.lan> From: "Thomas Foster" To: Date: Fri, 18 Feb 2005 12:43:45 -0800 MIME-Version: 1.0 X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2180 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2180 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.1 Subject: UFS2 Recovery Questions X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Feb 2005 20:42:51 -0000 Please excuse me if this is not the correct forum in which to ask this = question and please try to bear with me. I was hoping to understand a = few things about attempting to retrieve information from a drive on my = FreeBSD 5.3 system. I recently was rearranging my web server content and accidentally = removed a symbolic link recursively root@host # rm -rf music The music link pointed to a directory that existed on a spserate drive = mounted as /storage music -> /storage/Music/Mp3s I immediately unmounted the drive in question and started to comb = through resources looking for any utilities in attempt to recover this = data. I had not been able to install my DDS4 Tape Backup system yet, as = it had not arrived yet.. so restoring from backup is not a solution. I discovered various tools including the Coroner's Toolkit, and of = course.. Sleuthkit. Sleuthkit was very well documented and seemed to = offer some insight into the possibility of recovering the music files in = question. I immediately went to work installing sleuthkit from FreeBSD ports, and = even attempting to get Autopsy running (though I believe the port is = broken because I get a warning telling me the sleuthkit executables were = not found). I found documentation on the sleuthkit website about data = recovery: http://www.sleuthkit.org/sleuthkit/docs/ref_fs.html Now, the Manual Unix File Recovery Section gives some very helpful = information in tracking down the directory in question. When combing through the disk itself.. I find the directory I am looking = for: root@host # fls -r /dev/ad1s1d | grep Mp3s + d/- * 8007680: Mp3s This looks promising.. it looks like the associated inode for the Mp3s = directory is located at 8007680. I scoured the whole drive and could = not see any files under that directory though.. this might might be a = problem. root@host # fsstat /dev/ad1s1d Group 340: Last Written: Sun Feb 6 16:14:14 2005 Inode Range: 8007680 - 8031231 Fragment Range: 31989920 - 32084007 Super Block: 31989960 - 31989967 Group Desc: 31989968 - 31989975 Inode Table: 31989976 - 31992919 Data Fragments: 31989920 - 31989959, 31992920 - 32084007 Global Summary (from the superblock summary area): Num of Dirs: 1 Num of Avail Blocks: 7280 Num of Avail Inodes: 23550 Num of Avail Frags: 7 Local Summary (from the group descriptor): Num of Dirs: 1 Num of Avail Blocks: 7280 Num of Avail Inodes: 23550 Num of Avail Frags: 7 Last Block Allocated: 32061400 Last Fragment Allocated: 32061400 Last Inode Allocated: 8007680 Awesome... i think... there are the inode ranges... but where are the = block ranges listed in the documentation..? Also.. I comb through the = dates, and it appears that this was the last write to the drive... so I = assume that is when i deleted the directory. root@host # istat -b 2 /dev/ad1s1d 8007680 inode: 8007680 Not Allocated Group: 340 uid / gid: 65534 / 0 mode: ---------- size: 0 num of links: 0 Inode Times: Accessed: Sun Feb 6 16:12:08 2005 File Modified: Sun Feb 6 16:13:44 2005 Inode Modified: Sun Feb 6 16:13:44 2005 Direct Blocks: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Well this doesnt look good... 0 direct blocks.. i don't see this = covered. Now the documentation explains that I should image the inode = addresses for the group in question.. but the example given is an Ext3 = file systemm which contains the information: Group: 1:=20 Inode Range: 15393 - 30784=20 Block Range: 32768 - 65535=20 Super Block: 32768 - 32768=20 Group Descriptor Table: 32769 - 32769=20 Data bitmap: 32770 - 32770=20 Inode bitmap: 32771 - 32771=20 Inode Table: 32772 - 33252=20 Data Blocks: 33253 - 65535=20 I do not have a block range listed for UFS, and was curious if I should = "dls" the Inode range listed? Keep in mind that this dive is not mounted... and has not been altered = since the time of the directory deletion. I am anxious to use = "foremost" to attempt and recover the files via their hex header and = footer information. I setup my foremost.conf with the required = information: mp3 n 30000000 \xff\xfb\x92\x04\x00\x00\x00\x00\x00\x69\x05\ = \xa4\x00\x00\x00\x00\x00\x00\x34\x80\x00\x00 Now the questions is.. are the files even recoverable? Is this a lost = cause? Any additional information required, any comments or suggestions, = anything would be helpful. I thank you for your time in the matter. Thomas Foster http://www.section6.net