From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 3 15:31:43 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C651316A46C for ; Sun, 3 Feb 2008 15:31:43 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 8C45613C507 for ; Sun, 3 Feb 2008 15:31:43 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id EB11E2082; Sun, 3 Feb 2008 16:31:34 +0100 (CET) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: -0.2/3.0 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on tim.des.no Received: from ds4.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id DCCE1207F; Sun, 3 Feb 2008 16:31:34 +0100 (CET) Received: by ds4.des.no (Postfix, from userid 1001) id C4688844B3; Sun, 3 Feb 2008 16:31:34 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Ed Schouten References: <8663x6mc2o.fsf@ds4.des.no> <20080203131322.GK1179@hoeg.nl> <20080203151550.GA67020@owl.midgard.homeip.net> <86prvekqs2.fsf@ds4.des.no> Date: Sun, 03 Feb 2008 16:31:34 +0100 In-Reply-To: <86prvekqs2.fsf@ds4.des.no> ("Dag-Erling =?utf-8?Q?Sm=C3=B8rg?= =?utf-8?Q?rav=22's?= message of "Sun\, 03 Feb 2008 16\:23\:25 +0100") Message-ID: <86lk62kqeh.fsf@ds4.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: hackers@freebsd.org Subject: Re: sort(1) memory usage X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 15:31:43 -0000 Dag-Erling Sm=C3=B8rgrav writes: > Erik Trulsson writes: > > Yep, it seems that GNU sort allocates a quite large buffer by default w= hen > > the size of the input is unknown (such as when it reads input from stdi= n.) > > A quick check in the source code indicates that it tries to size this b= uffer > > according to how much memory the system has (and according to any limit= s set > > on how much memory the process is allowed to use.) > Uh, OK. This scaling doesn't seem to work correctly. It seems to > allocate 27 MB on 32-bit machines and 54 MB on 64-bit machines, > regardless of memory size. Looking at the code, it seems to go to extreme lengths to get it absolutely wrong. For instance, if hw.physmem / 8 > hw.usermem, it will pick the former, which means it's pretty much guaranteed to either fail or hose your system (or both). In the immortal words of Blazing Star: YOU FAIL IT Count this as a vote for ditching GNU sort in favor of a BSD-licensed implementation (from {Net,Open}BSD for instance). DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no