From owner-freebsd-questions@FreeBSD.ORG Fri Oct 5 08:26:57 2007 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 72A5516A417 for ; Fri, 5 Oct 2007 08:26:57 +0000 (UTC) (envelope-from jorn@wcborstel.com) Received: from mail.wcborstel.com (www.wcborstel.com [82.93.93.17]) by mx1.freebsd.org (Postfix) with ESMTP id DC54D13C455 for ; Fri, 5 Oct 2007 08:26:56 +0000 (UTC) (envelope-from jorn@wcborstel.com) Received: from mail.wcborstel.com (localhost [10.0.0.2]) by mail.wcborstel.com (Postfix) with ESMTP id 85EA14347C1; Fri, 5 Oct 2007 10:11:17 +0200 (CEST) Received: by mail.wcborstel.com (Postfix, from userid 58) id 36EE34347C0; Fri, 5 Oct 2007 10:11:17 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on mail.wcborstel.com X-Spam-Level: X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,BAYES_00, HTML_MESSAGE autolearn=ham version=3.2.3 X-Spam-Report: * -1.8 ALL_TRUSTED Passed through trusted hosts only via SMTP * -2.6 BAYES_00 BODY: Bayesian spam probability is 0 to 1% * [score: 0.0000] * 0.0 HTML_MESSAGE BODY: HTML included in message Received: from [192.168.1.6] (unknown [192.168.1.6]) by mail.wcborstel.com (Postfix) with ESMTP id 87DAE4347BE; Fri, 5 Oct 2007 10:11:13 +0200 (CEST) Message-ID: <4705F12B.1000501@wcborstel.com> Date: Fri, 05 Oct 2007 10:09:15 +0200 From: Jorn Argelo User-Agent: Thunderbird 2.0.0.6 (Windows/20070728) MIME-Version: 1.0 To: Steve Bertrand References: <4704DFF3.9040200@ibctech.ca> <200710041458.22743.wundram@beenic.net> <20071003200013.GD45244@demeter.hydra> <47054A1D.2000701@ibctech.ca> In-Reply-To: <47054A1D.2000701@ibctech.ca> X-Virus-Scanned: ClamAV using ClamSMTP Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: FreeBSD Questions Subject: Re: Managing very large files X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Oct 2007 08:26:57 -0000 Steve Bertrand wrote: >>> man 1 split >>> >>> (esp. -l) >>> >> That's probably the best option for a one-shot deal like this. On the >> other hand, Perl itself provides the ability to go through a file one >> line at a time, so you could just read a line, operate, write a line (to >> a new file) as needed, over and over, until you get through the whole >> file. >> >> The real problem would be reading the whole file into a variable (or even >> multiple variables) at once. >> > > This is what I am afraid of. Just out of curiosity, if I did try to read > the entire file into a Perl variable all at once, would the box panic, > or as the saying goes 'what could possibly go wrong'? > > Steve > Check out Tie::File on CPAN. This Perl module treats every line in a file as an array element, and the array element is loaded into memory when it's being requested. In other words: This will work great with huge files such as these, as not the entire file is loaded into memory at once. http://search.cpan.org/~mjd/Tie-File-0.96/lib/Tie/File.pm Jorn > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org" > >