From owner-freebsd-fs Wed Nov 28 17:52: 9 2001 Delivered-To: freebsd-fs@freebsd.org Received: from jazz.viagenie.qc.ca (jazz.viagenie.qc.ca [206.123.31.2]) by hub.freebsd.org (Postfix) with ESMTP id DEB8937B41A for ; Wed, 28 Nov 2001 17:52:05 -0800 (PST) Received: from classic (boitepostale.viagenie.qc.ca [206.123.31.3]) by jazz.viagenie.qc.ca (Viagenie/8.11.0) with ESMTP id fAT22Wa71140 for ; Wed, 28 Nov 2001 21:02:32 -0500 (EST) Date: Wed, 28 Nov 2001 20:54:52 -0500 From: Marc Blanchet To: freebsd-fs@freebsd.org Subject: tgz filesystem Message-ID: <224030000.1006998892@classic> X-Mailer: Mulberry/2.1.1 (Linux/x86) MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Sender: owner-freebsd-fs@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Hi, - I would like to use a tar-gzip filesystem that mounts a tgz file. The=20 idea here is to browse through some large archives that are tgz. For=20 example: all RFC and internet-drafts in a tgz file, that I don't=20 necessarily want to decompress before finding the right file. I would use something like: mount -t tgz ietf.tgz /usr/home/myaccount/ietf and then could be able to do whatever I want in this mounted directory=20 without actually copying the files. - is this already done? if yes, where can I find info? - if not, any pointers on how to start programming this? - forgive me if this was discussed before. I looked in the archive, found=20 nothing. Marc. ------------------------------------------ Marc Blanchet Viag=E9nie tel: +1-418-656-9254x225 ------------------------------------------ http://www.freenet6.net: IPv6 connectivity ------------------------------------------ http://www.normos.org: IETF(RFC,draft), IANA,W3C,... standards. ------------------------------------------ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-fs" in the body of the message From owner-freebsd-fs Wed Nov 28 18: 4:22 2001 Delivered-To: freebsd-fs@freebsd.org Received: from elvis.mu.org (elvis.mu.org [216.33.66.196]) by hub.freebsd.org (Postfix) with ESMTP id 79E3C37B417 for ; Wed, 28 Nov 2001 18:04:21 -0800 (PST) Received: by elvis.mu.org (Postfix, from userid 1192) id 1984081D04; Wed, 28 Nov 2001 20:04:16 -0600 (CST) Date: Wed, 28 Nov 2001 20:04:16 -0600 From: Alfred Perlstein To: Marc Blanchet Cc: freebsd-fs@freebsd.org Subject: Re: tgz filesystem Message-ID: <20011128200416.Q46769@elvis.mu.org> References: <224030000.1006998892@classic> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <224030000.1006998892@classic>; from Marc.Blanchet@viagenie.qc.ca on Wed, Nov 28, 2001 at 08:54:52PM -0500 Sender: owner-freebsd-fs@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org * Marc Blanchet [011128 19:52] wrote: > Hi, > - I would like to use a tar-gzip filesystem that mounts a tgz file. The This would be nearly useless, a .tgz file is a tar file (which has no TOC) in a gzipp'd stream. Meaning that browsing it is pretty much impossible without extracting the whole thing anyhow. -- -Alfred Perlstein [alfred@freebsd.org] 'Instead of asking why a piece of software is using "1970s technology," start asking why software is ignoring 30 years of accumulated wisdom.' http://www.morons.org/rants/gpl-harmful.php3 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-fs" in the body of the message From owner-freebsd-fs Thu Nov 29 7:29:55 2001 Delivered-To: freebsd-fs@freebsd.org Received: from jazz.viagenie.qc.ca (jazz.viagenie.qc.ca [206.123.31.2]) by hub.freebsd.org (Postfix) with ESMTP id DBD3A37B443 for ; Thu, 29 Nov 2001 07:29:49 -0800 (PST) Received: from classic (boitepostale.viagenie.qc.ca [206.123.31.3]) by jazz.viagenie.qc.ca (Viagenie/8.11.0) with ESMTP id fATFe9a79046; Thu, 29 Nov 2001 10:40:09 -0500 (EST) Date: Thu, 29 Nov 2001 10:32:26 -0500 From: Marc Blanchet To: Alfred Perlstein Cc: freebsd-fs@freebsd.org Subject: Re: tgz filesystem Message-ID: <278520000.1007047945@classic> In-Reply-To: <20011128200416.Q46769@elvis.mu.org> References: <224030000.1006998892@classic> <20011128200416.Q46769@elvis.mu.org> X-Mailer: Mulberry/2.1.1 (Linux/x86) MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Sender: owner-freebsd-fs@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org -- mercredi, novembre 28, 2001 20:04:16 -0600 Alfred Perlstein=20 wrote/a =E9crit: > * Marc Blanchet [011128 19:52] wrote: >> Hi, >> - I would like to use a tar-gzip filesystem that mounts a tgz file. The > > This would be nearly useless, a .tgz file is a tar file (which has no = TOC) > in a gzipp'd stream. Meaning that browsing it is pretty much impossible > without extracting the whole thing anyhow. good point. well taken. however, the extraction would only be done during the mount time (with the=20 impact of some significant mount delay before the fs is ready if the file=20 is big and the processor is slow). But, if the fs is not mounted, then the=20 space of the temporary copy is not used, and also it keeps the data in the=20 compressed format. Think about the intent: - right now, I have archives of data in the popular tgz format. to actually = search through it and work on the data without knowing exactly in advance=20 which file I'm looking for, I essentially have to decompress the whole=20 thing and then work on it. One can argue that you could do some kind of tar = | grep | .... However, this is always a one shot and limited in terms of=20 searching. In fact, I do that often and the result is that I needed often=20 to do the tar|grep many times before finding the right thing; so instead, I = decompress the whole thing and then work on it. Then after, I have to rm=20 -rf all. - mounting the file as a fs would do the trick more cleanly, more easily. I might not be convincing anybody, but this would be for me at least a good = academic exercise.... ;-))) Could I get at least some advice on how to start programming this=20 (references, code examples,...)? Is the stackable fs software (fist) a good = starting point? Are the null/umap fs the good starting point instead? Marc. > > -- > -Alfred Perlstein [alfred@freebsd.org] > 'Instead of asking why a piece of software is using "1970s technology," > start asking why software is ignoring 30 years of accumulated wisdom.' > http://www.morons.org/rants/gpl-harmful.php3 ------------------------------------------ Marc Blanchet Viag=E9nie tel: +1-418-656-9254x225 ------------------------------------------ http://www.freenet6.net: IPv6 connectivity ------------------------------------------ http://www.normos.org: IETF(RFC,draft), IANA,W3C,... standards. ------------------------------------------ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-fs" in the body of the message From owner-freebsd-fs Thu Nov 29 7:50:26 2001 Delivered-To: freebsd-fs@freebsd.org Received: from srv1.cosmo-project.de (srv1.cosmo-project.de [213.83.6.106]) by hub.freebsd.org (Postfix) with ESMTP id 7EC3837B449 for ; Thu, 29 Nov 2001 07:50:21 -0800 (PST) Received: (from uucp@localhost) by srv1.cosmo-project.de (8.11.0/8.11.0) with UUCP id fATFoD177949; Thu, 29 Nov 2001 16:50:13 +0100 (CET) Received: from mail.cicely.de (cicely20.cicely.de [10.1.1.22]) by cicely5.cicely.de (8.12.1/8.12.1) with ESMTP id fATFk1AD071163; Thu, 29 Nov 2001 16:46:01 +0100 (CET)?g (envelope-from ticso@cicely8.cicely.de) Received: from cicely8.cicely.de (cicely8.cicely.de [10.1.2.10]) by mail.cicely.de (8.11.0/8.11.0) with ESMTP id fATFjsL06930; Thu, 29 Nov 2001 16:46:00 +0100 (CET) Received: (from ticso@localhost) by cicely8.cicely.de (8.11.4/8.11.4) id fATFjrZ53062; Thu, 29 Nov 2001 16:45:54 +0100 (CET) (envelope-from ticso) Date: Thu, 29 Nov 2001 16:45:52 +0100 From: Bernd Walter To: Marc Blanchet Cc: Alfred Perlstein , freebsd-fs@FreeBSD.ORG Subject: Re: tgz filesystem Message-ID: <20011129164552.B50731@cicely8.cicely.de> References: <224030000.1006998892@classic> <20011128200416.Q46769@elvis.mu.org> <278520000.1007047945@classic> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <278520000.1007047945@classic> User-Agent: Mutt/1.3.23i X-Operating-System: FreeBSD cicely8.cicely.de 5.0-CURRENT i386 Sender: owner-freebsd-fs@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Thu, Nov 29, 2001 at 10:32:26AM -0500, Marc Blanchet wrote: > however, the extraction would only be done during the mount time (with the > impact of some significant mount delay before the fs is ready if the file > is big and the processor is slow). But, if the fs is not mounted, then the > space of the temporary copy is not used, and also it keeps the data in the > compressed format. > > Think about the intent: > - right now, I have archives of data in the popular tgz format. to actually > search through it and work on the data without knowing exactly in advance > which file I'm looking for, I essentially have to decompress the whole > thing and then work on it. One can argue that you could do some kind of tar > | grep | .... However, this is always a one shot and limited in terms of > searching. In fact, I do that often and the result is that I needed often > to do the tar|grep many times before finding the right thing; so instead, I > decompress the whole thing and then work on it. Then after, I have to rm > -rf all. > - mounting the file as a fs would do the trick more cleanly, more easily. > > I might not be convincing anybody, but this would be for me at least a good > academic exercise.... ;-))) > > Could I get at least some advice on how to start programming this > (references, code examples,...)? Is the stackable fs software (fist) a good > starting point? Are the null/umap fs the good starting point instead? Write a mount_tar shell script, which untars it to /tmp/something and does a softlink on the mountpoint. Write a umount_tar shell script which untangles /tmp/something and the softlink On reboot /tmp is cleaned anyway if you put it on an mfs/md filesystem. If you like you can also use a separate partiotion from /tmp, which is only writeable by root, so you don't have to wory about tempnames that much. You can also put it into an automounter configuration so you don't have to manualy mount it. -- B.Walter COSMO-Project http://www.cosmo-project.de ticso@cicely.de Usergroup info@cosmo-project.de To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-fs" in the body of the message From owner-freebsd-fs Thu Nov 29 8:28:13 2001 Delivered-To: freebsd-fs@freebsd.org Received: from bilver.wjv.com (spdsl-033.wanlogistics.net [63.209.115.33]) by hub.freebsd.org (Postfix) with ESMTP id 9891537B42A for ; Thu, 29 Nov 2001 08:23:45 -0800 (PST) Received: (from bv@localhost) by bilver.wjv.com (8.11.6/8.11.6) id fATFkSN82603; Thu, 29 Nov 2001 10:46:28 -0500 (EST) (envelope-from bv) Date: Thu, 29 Nov 2001 10:46:27 -0500 From: Bill Vermillion To: Marc Blanchet Cc: freebsd-fs@FreeBSD.ORG Subject: Re: tgz filesystem Message-ID: <20011129104627.A82474@wjv.com> Reply-To: bv@wjv.com References: <224030000.1006998892@classic> <20011128200416.Q46769@elvis.mu.org> <278520000.1007047945@classic> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit User-Agent: Mutt/1.2.5i In-Reply-To: <278520000.1007047945@classic>; from Marc.Blanchet@viagenie.qc.ca on Thu, Nov 29, 2001 at 10:32:26AM -0500 Organization: W.J.Vermillion / Orlando - Winter Park Sender: owner-freebsd-fs@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Thu, Nov 29, 2001 at 10:32:26AM -0500, Marc Blanchet thus spoke: > > -- mercredi, novembre 28, 2001 20:04:16 -0600 Alfred Perlstein > wrote/a écrit: > > * Marc Blanchet [011128 19:52] wrote: > >> - I would like to use a tar-gzip filesystem that mounts a tgz > >> file. The > > This would be nearly useless, a .tgz file is a tar file (which > > has no TOC) in a gzipp'd stream. Meaning that browsing it is > > pretty much impossible without extracting the whole thing > > anyhow. > good point. well taken. A few years ago when drives were smaller, SCO had implemented a file system called DTFS - DeskTop File System. It was a compressing file system so the space available for storage was about twice the real space. Never went very far as the drives got bigger but it worked well for such things a lap tops with 200MB drives for instance. > however, the extraction would only be done during the mount time > (with the impact of some significant mount delay before the fs is > ready if the file is big and the processor is slow). But, if the > fs is not mounted, then the space of the temporary copy is not > used, and also it keeps the data in the compressed format. The problem is you want to mount a file, and it's file SYSTEMS that we mount. > Think about the intent: - right now, I have archives of data in > the popular tgz format. to actually search through it and work > on the data without knowing exactly in advance which file I'm > looking for, I essentially have to decompress the whole thing and > then work on it. One can argue that you could do some kind of tar > | grep | .... However, this is always a one shot and limited in > terms of searching. In fact, I do that often and the result is > that I needed often to do the tar|grep many times before finding > the right thing; so instead, I decompress the whole thing and then > work on it. Then after, I have to rm -rf all. Ah. Now I see what you want to do. And there is an easier way. > - mounting the file as a fs would do the trick more cleanly, more > easily. But how about doing it this way - assuming you have a wildcard match for files and wanted to search for names in the archive, you would just do this: zcat | tar tvf - | grep | less -e > I might not be convincing anybody, but this would be for me at > least a good academic exercise.... ;-))) > Could I get at least some advice on how to start programming this > (references, code examples,...)? Is the stackable fs software > (fist) a good starting point? Are the null/umap fs the good > starting point instead? Files aren't going to look much like file systems. An analogy might be a book doesn't look much like a library where a book is a file and a library is a file system. Bill To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-fs" in the body of the message From owner-freebsd-fs Thu Nov 29 8:31:38 2001 Delivered-To: freebsd-fs@freebsd.org Received: from ns.caldera.de (ns.caldera.de [212.34.180.1]) by hub.freebsd.org (Postfix) with ESMTP id B7DAD37B433 for ; Thu, 29 Nov 2001 08:31:35 -0800 (PST) Received: (from hch@localhost) by ns.caldera.de (8.11.1/8.11.1) id fATGUds19959; Thu, 29 Nov 2001 17:30:39 +0100 Date: Thu, 29 Nov 2001 17:30:39 +0100 From: Christoph Hellwig To: Bill Vermillion Cc: Marc Blanchet , freebsd-fs@FreeBSD.ORG Subject: Re: tgz filesystem Message-ID: <20011129173039.A19261@caldera.de> References: <224030000.1006998892@classic> <20011128200416.Q46769@elvis.mu.org> <278520000.1007047945@classic> <20011129104627.A82474@wjv.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <20011129104627.A82474@wjv.com>; from bv@wjv.com on Thu, Nov 29, 2001 at 10:46:27AM -0500 Sender: owner-freebsd-fs@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Thu, Nov 29, 2001 at 10:46:27AM -0500, Bill Vermillion wrote: > A few years ago when drives were smaller, SCO had implemented a > file system called DTFS - DeskTop File System. It was a > compressing file system so the space available for storage was > about twice the real space. Never went very far as the drives got > bigger but it worked well for such things a lap tops with 200MB > drives for instance. OpenServer still has it - most work was done by plc, though. Christoph -- Of course it doesn't work. We've performed a software upgrade. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-fs" in the body of the message From owner-freebsd-fs Sat Dec 1 15: 4:52 2001 Delivered-To: freebsd-fs@freebsd.org Received: from double.inode.org (cpu1532.adsl.bellglobal.com [206.47.27.13]) by hub.freebsd.org (Postfix) with ESMTP id 8082E37B417 for ; Sat, 1 Dec 2001 15:04:48 -0800 (PST) Received: from coup.inode.org (coup.inode.org [172.16.2.203]) by double.inode.org (8.11.0/8.11.1) with SMTP id fB1N4k444196; Sat, 1 Dec 2001 18:04:47 -0500 (EST) From: David Jones Date: Sat, 1 Dec 2001 18:04:46 -0500 X-Mailer: KMail [version 1.1.99] Content-Type: text/plain; charset="iso-8859-1" To: freebsd-fs@freebsd.org Cc: dej@inode.org Subject: SoftUpdates semantics and data integrity MIME-Version: 1.0 Message-Id: <01120118044609.17440@coup.inode.org> Content-Transfer-Encoding: quoted-printable Sender: owner-freebsd-fs@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org As I understand it, metadata on a soft updates file system is not flushed= =20 right away, and delays of up to one minute before flushing are possible. = =20 However, soft updates will ensure that the file system will reflect a sta= te=20 that it had at some point in time before a crash. At a former employer, I wrote a database-type application that went to gr= eat=20 pains to achieve atomic transactions, even in the event of a crash. I am= =20 wondering if soft updates changes the things I have to do to achieve such= =20 reliability in the future. Answers to the following are appreciated, for both sync-metadata FFS and = soft=20 updates FFS: 1. What must I do in software to ensure that data written to a file has m= ade=20 it to disk? Is close() sufficient, or must I fsync() as well? Is the da= ta=20 guaranteed to be there after fsync() returns? (It isn't with sync()). 2. What must I do to ensure that a rename() has either succeeded or faile= d? =20 "Succeeded" means "the target name refers to the new file, even if the sy= stem=20 crashes right now". 3. If I do two filesystem operations one after the other, can I be assure= d=20 that the first one will be committed safely to disk when the second compl= etes? I don't care about performance necessarily in these cases; I just want to= be=20 sure that a data transaction is guaranteed to be on the disk come hell or= =20 high water in certain, critical cases. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-fs" in the body of the message