From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 31 20:42:03 2012 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 20176518 for ; Wed, 31 Oct 2012 20:42:03 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vps.rulingia.com (host-122-100-2-194.octopus.com.au [122.100.2.194]) by mx1.freebsd.org (Postfix) with ESMTP id 894898FC0C for ; Wed, 31 Oct 2012 20:42:01 +0000 (UTC) Received: from server.rulingia.com (c220-239-241-202.belrs5.nsw.optusnet.com.au [220.239.241.202]) by vps.rulingia.com (8.14.5/8.14.5) with ESMTP id q9VKfwYL037281 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Thu, 1 Nov 2012 07:41:58 +1100 (EST) (envelope-from peter@rulingia.com) X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.14.5/8.14.5) with ESMTP id q9VKfqVF039592 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 1 Nov 2012 07:41:52 +1100 (EST) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.14.5/8.14.5/Submit) id q9VKfq1W039591; Thu, 1 Nov 2012 07:41:52 +1100 (EST) (envelope-from peter) Date: Thu, 1 Nov 2012 07:41:52 +1100 From: Peter Jeremy To: Alfred Perlstein Subject: Re: make -jN buildworld on < 512MB ram Message-ID: <20121031204152.GK3309@server.rulingia.com> References: <509182DA.8070303@mu.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="oJ71EGRlYNjSvfq7" Content-Disposition: inline In-Reply-To: <509182DA.8070303@mu.org> X-PGP-Key: http://www.rulingia.com/keys/peter.pgp User-Agent: Mutt/1.5.21 (2010-09-15) Cc: hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Oct 2012 20:42:03 -0000 --oJ71EGRlYNjSvfq7 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2012-Oct-31 12:58:18 -0700, Alfred Perlstein wrote: >It seems like the new compiler likes to get up to ~200+MB resident when=20 >building some basic things in our tree. The killer I found was the ctfmerge(1) on the kernel - which exceeds ~400MB on i386. Under low RAM, that fails _without_ reporting any errors back to make(1), resulting in a corrupt new kernel (it booted but had virtually no devices so it couldn't find root). >Doesn't our make(1) have some stuff to mitigate this? I would expect it= =20 >to be a bit smarter about detecting the number of swaps/pages/faults of=20 >its children and taking into account the machine's total ram before=20 >forking off new processes. The difficulty I see is that the make process can't tell anything about the memory requirements of the pipeline it is about to spawn. As a rule of thumb, C++ needs more memory than C but that depends on what is being compiled - I have a machine-generated C program that makes gcc bloat to ~12GB. >Any ideas? I mean a really simple algorithm could be devised that would= =20 >be better than what we appear to have (which is nothing). If you can afford to waste CPU, one approach would be for make(1) to setrlimit(2) child processes and if the child dies, it retries that child by itself - but that will generate unnecessary retries. Another, more involved, approach would be for the scheduler to manage groups of processes - if a group of processes is causing memory pressure as a whole then the scheduler just stops scheduling some of them until the pressure reduces (effectively swap them out). (Yes, that's vague and lots of hand-waving that might not be realisable). --=20 Peter Jeremy --oJ71EGRlYNjSvfq7 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlCRjRAACgkQ/opHv/APuIf8bACeKtbpNmaXSp2R6mEFNY16AeyK LiAAoKRUiIC+YlaaUmdMXdz27947sC98 =KqHt -----END PGP SIGNATURE----- --oJ71EGRlYNjSvfq7--