From owner-freebsd-questions@FreeBSD.ORG Thu Oct 4 20:30:19 2007 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DD5B816A469 for ; Thu, 4 Oct 2007 20:30:19 +0000 (UTC) (envelope-from perrin@apotheon.com) Received: from outbound-mail-38.bluehost.com (outbound-mail-38.bluehost.com [69.89.20.192]) by mx1.freebsd.org (Postfix) with SMTP id B6AB613C4E8 for ; Thu, 4 Oct 2007 20:30:19 +0000 (UTC) (envelope-from perrin@apotheon.com) Received: (qmail 32279 invoked by uid 0); 4 Oct 2007 20:30:19 -0000 Received: from unknown (HELO box183.bluehost.com) (69.89.25.183) by mailproxy2.bluehost.com with SMTP; 4 Oct 2007 20:30:19 -0000 Received: from c-24-9-123-251.hsd1.co.comcast.net ([24.9.123.251] helo=demeter.hydra) by box183.bluehost.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.68) (envelope-from ) id 1IdXKs-0007ly-Dl for freebsd-questions@freebsd.org; Thu, 04 Oct 2007 14:30:18 -0600 Received: from demeter.hydra (localhost [127.0.0.1]) by demeter.hydra (8.13.6/8.13.6) with ESMTP id l93Mnsu9046185 for ; Wed, 3 Oct 2007 16:49:54 -0600 (MDT) (envelope-from perrin@apotheon.com) Received: (from ren@localhost) by demeter.hydra (8.13.6/8.13.6/Submit) id l93Mns3d046184 for freebsd-questions@freebsd.org; Wed, 3 Oct 2007 16:49:54 -0600 (MDT) (envelope-from perrin@apotheon.com) X-Authentication-Warning: demeter.hydra: ren set sender to perrin@apotheon.com using -f Date: Wed, 3 Oct 2007 16:49:53 -0600 From: Chad Perrin To: FreeBSD Questions Message-ID: <20071003224953.GA46149@demeter.hydra> Mail-Followup-To: FreeBSD Questions References: <4704DFF3.9040200@ibctech.ca> <200710041458.22743.wundram@beenic.net> <20071003200013.GD45244@demeter.hydra> <47054A1D.2000701@ibctech.ca> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <47054A1D.2000701@ibctech.ca> User-Agent: Mutt/1.4.2.3i X-Identified-User: {737:box183.bluehost.com:apotheon:apotheon.net} {sentby:bopbeforesmtp 24.9.123.251 authed with apotheon.com} X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - box183.bluehost.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [737 12] / [47 12] X-AntiAbuse: Sender Address Domain - apotheon.com Subject: Re: Managing very large files X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Oct 2007 20:30:20 -0000 On Thu, Oct 04, 2007 at 04:16:29PM -0400, Steve Bertrand wrote: > >> man 1 split > >> > >> (esp. -l) > > > > That's probably the best option for a one-shot deal like this. On the > > other hand, Perl itself provides the ability to go through a file one > > line at a time, so you could just read a line, operate, write a line (to > > a new file) as needed, over and over, until you get through the whole > > file. > > > > The real problem would be reading the whole file into a variable (or even > > multiple variables) at once. > > This is what I am afraid of. Just out of curiosity, if I did try to read > the entire file into a Perl variable all at once, would the box panic, > or as the saying goes 'what could possibly go wrong'? Perl will happily load stuff into RAM until you run out of RAM. I imagine it would then keep loading stuff into "memory", and the box would start swapping. Eventually, you'd run out of swap space. Perl is known to some as the "Swiss Army chainsaw" for a reason: it'll cut limbs off trees about as quickly as you can put limbs in front of it. If you put one of your own limbs in front of it (say, a leg), it'll do exactly the same thing -- but with more bleeding and screaming. It's kinda like Unix, that way. -- CCD CopyWrite Chad Perrin [ http://ccd.apotheon.org ] Brian K. Reid: "In computer science, we stand on each other's feet."