From owner-freebsd-questions@FreeBSD.ORG Fri Oct 5 12:33:49 2007 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DE3A716A468 for ; Fri, 5 Oct 2007 12:33:49 +0000 (UTC) (envelope-from iaccounts@ibctech.ca) Received: from pearl.ibctech.ca (pearl.ibctech.ca [208.70.104.210]) by mx1.freebsd.org (Postfix) with ESMTP id 6A0C613C469 for ; Fri, 5 Oct 2007 12:33:49 +0000 (UTC) (envelope-from iaccounts@ibctech.ca) Received: (qmail 96579 invoked by uid 1002); 5 Oct 2007 12:33:48 -0000 Received: from iaccounts@ibctech.ca by pearl.ibctech.ca by uid 89 with qmail-scanner-1.22 (spamassassin: 2.64. Clear:RC:1(208.70.104.100):. Processed in 14.173306 secs); 05 Oct 2007 12:33:48 -0000 Received: from unknown (HELO ?192.168.30.110?) (steve@ibctech.ca@208.70.104.100) by pearl.ibctech.ca with (DHE-RSA-AES256-SHA encrypted) SMTP; 5 Oct 2007 12:33:34 -0000 Message-ID: <47062F2B.60208@ibctech.ca> Date: Fri, 05 Oct 2007 08:33:47 -0400 From: Steve Bertrand User-Agent: Thunderbird 2.0.0.6 (Windows/20070728) MIME-Version: 1.0 To: Jorn Argelo References: <4704DFF3.9040200@ibctech.ca> <200710041458.22743.wundram@beenic.net> <20071003200013.GD45244@demeter.hydra> <47054A1D.2000701@ibctech.ca> <4705F12B.1000501@wcborstel.com> In-Reply-To: <4705F12B.1000501@wcborstel.com> X-Enigmail-Version: 0.95.3 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: FreeBSD Questions Subject: Re: Managing very large files X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Oct 2007 12:33:50 -0000 > Check out Tie::File on CPAN. This Perl module treats every line in a > file as an array element, and the array element is loaded into memory > when it's being requested. In other words: This will work great with > huge files such as these, as not the entire file is loaded into memory > at once. > > http://search.cpan.org/~mjd/Tie-File-0.96/lib/Tie/File.pm Thanks everyone who replied to me regarding this issue. The above appears to be my best approach. Although I have not the time yet to look into Tie::Find (and I've never used that module before) but I will. So long as I can read chunks of the file, load the data into variables (I like the array approach above) and process each array independently without loading all of them at once into memory, and without having to load the entire file into memory. Tks! Steve