From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 25 18:28:48 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6E10716A405 for ; Mon, 25 Feb 2008 18:28:48 +0000 (UTC) (envelope-from mwm@mired.org) Received: from mired.org (bhuda.mired.org [66.92.153.74]) by mx1.freebsd.org (Postfix) with SMTP id 0E26113C45B for ; Mon, 25 Feb 2008 18:28:47 +0000 (UTC) (envelope-from mwm@mired.org) Received: (qmail 80554 invoked from network); 25 Feb 2008 18:27:34 -0000 Received: from unknown (HELO mbook-fbsd) (192.168.195.2) by 0 with SMTP; 25 Feb 2008 18:27:34 -0000 Date: Mon, 25 Feb 2008 13:28:00 -0500 From: Mike Meyer To: "Martin Laabs" Message-ID: <20080225132800.7954bcf7@mbook-fbsd> In-Reply-To: References: Organization: Meyer Consulting X-Mailer: Claws Mail 3.0.2 (GTK+ 2.12.5; amd64-portbld-freebsd7.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 25 Feb 2008 18:37:25 +0000 Cc: freebsd-hackers@freebsd.org Subject: Re: emulate an end-of-media X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Feb 2008 18:28:48 -0000 On Mon, 25 Feb 2008 13:36:17 +0100 "Martin Laabs" wrote: > Hi, > > I'll write a script that back up my data on dvd-r and dvd-rams. > (And some incremental backups also at some internet storage > services.) > Therefore I'd like to use dump. It is not too hard to create > dump-volumes with fixed size through the -B option. But now > I'd like to use some sort of compression. (bzip2 seems far to be > to slow. Maybe I'll use gzip or compress) > However - with compression I can't use the -B switch of dump > because I can't be sure about the compression ratio. This means > I have to use the '-a' switch. With that switch enabled, dump > starts a new volume after it detects an EOM signal. > > Because dump itself does not support compression I have to > build a pipe like that: > dump -aL0 -f - /MYFILESYSTEM|compress -c|aespipe ...|cdrecord dev=... - > This obvisouly does not work. I tried to wrote a script that > close the pipe after the amount of compressed data reached > 4.6GB. But if I close the pipe in that script dump aborts > (because of the broken pipe) at all instead of just starting > a new volume. You might want to play with the -P option to dump. Your above could be written as: dump -aL0 -P 'compress -c' /MYFILESYSTEM | cdrecord dev=... - Assuming that compress -c & cdrecord play nice (which your magic device solution also requires), then all you need to do is make "-a" DTRT when the pipe to it's -P option is closed. Dealing with buffering could be an interesting problem in all these cases. > What I'd need is a magaic pipe device that converts a ordenary > close at the one side to an EOM at the other. > Then I could do something like this: > > dump -aL0 -f /dev/MyMagicDevice & > dd if=/dev/MyMagicDevice bs=1M count=4600|./compress -c|cdrecord dev=... - > > Before I try to write a kernel module that will do the > job (I never wrote a freebsd kernel module before but > I think it is some fun to learn that.) I want to ask if > anyone see another solution for that problem. (Maybe > patching dump - but I don't want to fudge in the CORE > source.) Your magic pipe device might be more generally useful, but I'm not sure how many commands use EOM for anything at all. On the other hand, having dump -a and -P work as expected (doesn't look like they do as of 6.2) seems to be a good thing as well. > PS: Splitting up the huge (maybe compressed) dump file (also in > stream operation) is not a good idea beacause I'd need to insert > all the media - even if I resore only a single file. Well, if -B worked on compressed output, then you could split it up on volumes, which wouldn't be quite so bad. So add making -P and -B play nice together (again, they don't seem to as of 6.2) as a possible solution. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information.