From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 3 15:37:36 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DCCFC16A417 for ; Sun, 3 Feb 2008 15:37:36 +0000 (UTC) (envelope-from ws@au.dyndns.ws) Received: from ipmail05.adl2.internode.on.net (ipmail05.adl2.internode.on.net [203.16.214.145]) by mx1.freebsd.org (Postfix) with ESMTP id 78DB213C459 for ; Sun, 3 Feb 2008 15:37:35 +0000 (UTC) (envelope-from ws@au.dyndns.ws) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgAAAI9opUeWZWdv/2dsb2JhbAAIq3c X-IronPort-AV: E=Sophos;i="4.25,298,1199626200"; d="scan'208";a="47243121" Received: from ppp103-111.static.internode.on.net (HELO [192.168.1.131]) ([150.101.103.111]) by ipmail05.adl2.internode.on.net with ESMTP; 04 Feb 2008 01:52:17 +1030 From: Wayne Sierke To: Ed Schouten In-Reply-To: <20080203131322.GK1179@hoeg.nl> References: <8663x6mc2o.fsf@ds4.des.no> <20080203131322.GK1179@hoeg.nl> Content-Type: text/plain; charset=ISO-8859-1 Date: Mon, 04 Feb 2008 01:52:15 +1030 Message-Id: <1202052136.5350.137.camel@predator-ii.buffyverse> Mime-Version: 1.0 X-Mailer: Evolution 2.12.3 FreeBSD GNOME Team Port Content-Transfer-Encoding: 8bit Cc: Dag-Erling =?ISO-8859-1?Q?Sm=F8rgrav?= , hackers@freebsd.org Subject: Re: sort(1) memory usage X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 15:37:36 -0000 On Sun, 2008-02-03 at 14:13 +0100, Ed Schouten wrote: > * Dag-Erling Smørgrav wrote: > > I've been trying to figure out why some periodic scripts consume so much > > memory. I've narrowed it down to sort(1). > > > > At first, I thought the scripts were using it inefficiently, feeding it > > more data than was really needed. Then I discovered this: > > > > des@ds4 ~% (sleep 10 | sort) & (sleep 5 ; top -o res | grep sort) > > [1] 66024 > > 66024 des 1 -8 5 54796K 52680K piperd 1 0:00 0.88% sort > > > > That's right - sort(1) consumes 50+ MB of memory doing *nothing*. > > > > (roughly half that on a 32-bit box) > > > > Something is rotten in the state of GNU... > > On my i386 box it spends 27M, but when I replace `sort' with `sed', > without any arguments, it's only 1.4 MB. I tried this on RELENG_6. I can > also reproduce this on Linux. > %uname -vm FreeBSD 7.0-PRERELEASE #1: Fri Jan 25 01:08:47 CST 2008 root@freebsd7stable-1.vmware:/usr/obj/usr/src/sys/GENERIC-KTR-0x2000 i386 %( sleep 10 | sort ) & ( sleep 5 ; top -n 150 | grep sort ) [2] 38158 38158 ws 1 -8 0 29760K 736K piperd 0:00 0.00% sort %su - # ( sleep 10 | sort ) & ( sleep 5 ; top -n 150 | grep sort ) [2] 38165 38165 root 1 -8 0 29760K 732K piperd 0:00 0.00% sort $ uname -vm FreeBSD 6.3-PRERELEASE #1: Fri Dec 28 17:49:43 CST 2007 root@predator-ii.buffyverse:/usr/obj/usr/src/sys/LILLITH-IV i386 $ (sleep 10 | sort) & (sleep 5; top -n 150 | grep sort) 68953 ws 1 -8 0 26988K 660K piperd 0:00 0.00% sort $ su - # (sleep 10 | sort) & (sleep 5; top -n 150 | grep sort) [1] 68981 68981 root 1 -8 0 26988K 660K piperd 0:00 0.00% sort Next one is Ubuntu 7.04 $ uname -a Linux developer 2.6.20-16-generic #2 SMP Thu Jun 7 20:19:32 UTC 2007 i686 GNU/Linux $ (sleep 10 | sort) & (sleep 5; ps aux | grep -E "^USER|sort$") [9] 10523 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND ws 10526 0.0 0.0 28436 676 pts/4 S 01:29 0:00 sort (I had to change the 'top' incantation to pick up the 'sort' process on these busy boxes.) Wayne