From owner-freebsd-stable@FreeBSD.ORG  Thu Nov 30 10:32:26 2006
Return-Path: <owner-freebsd-stable@FreeBSD.ORG>
X-Original-To: freebsd-stable@freebsd.org
Delivered-To: freebsd-stable@freebsd.org
Received: from mx1.FreeBSD.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id D015F16A407
	for <freebsd-stable@freebsd.org>; Thu, 30 Nov 2006 10:32:26 +0000 (UTC)
	(envelope-from rwatson@FreeBSD.org)
Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42])
	by mx1.FreeBSD.org (Postfix) with ESMTP id DA80D43CB4
	for <freebsd-stable@freebsd.org>; Thu, 30 Nov 2006 10:32:11 +0000 (GMT)
	(envelope-from rwatson@FreeBSD.org)
Received: from fledge.watson.org (fledge.watson.org [209.31.154.41])
	by cyrus.watson.org (Postfix) with ESMTP id BB08A46D91;
	Thu, 30 Nov 2006 05:32:18 -0500 (EST)
Date: Thu, 30 Nov 2006 10:32:18 +0000 (GMT)
From: Robert Watson <rwatson@FreeBSD.org>
X-X-Sender: robert@fledge.watson.org
To: Kris Kennaway <kris@obsecurity.org>
In-Reply-To: <20061130044123.GA19761@xor.obsecurity.org>
Message-ID: <20061130103025.N95096@fledge.watson.org>
References: <7883F37393F1D44990888174C2E5E91222666A@TSV-EXCHANGE.ohhllp.com>
	<20061130044123.GA19761@xor.obsecurity.org>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
Cc: freebsd-stable@freebsd.org, "van Osnabrugge,
	Sean" <svanosnabrugge@osler.com>
Subject: Re: grep: memory exhausted
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>, 
	<mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable>
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
	<mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 30 Nov 2006 10:32:26 -0000


On Wed, 29 Nov 2006, Kris Kennaway wrote:

> On Wed, Nov 29, 2006 at 03:10:33PM -0500, van Osnabrugge, Sean wrote:
>>
>> I am running a fresh install of FreeBSD-6.1-Stable as a guest OS in VMWare 
>> 1.0.1 with 1 GB of RAM.
>>
>> Whenever I try to grep a large text file (400 MB+), grep terminates with 
>> "grep: memory exhausted"
>>
>> I have tried piping grep (cat "file" | grep "search term")
>>
>> I have tried it with -line-buffered
>>
>> ulimit -a show:
>>
>> core file size (blocks, -c) unlimited
>>
>> data seg size (kbytes, -d) 524288
>
> Try increasing this.  I think grep mmaps the file, so the large file could 
> be exceeding your limit.

I regularly grep multi-hundred-gigabyte files without a problem on boxes with 
very little memory, so simple file size is unlikely the cause.  However, it 
would be interesting to know how many lines "wc -l" thinks the text file has, 
and what the length of the longest line is.

Robert N M Watson
Computer Laboratory
University of Cambridge