From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 31 20:24:20 2012 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D183BAFB for ; Wed, 31 Oct 2012 20:24:20 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-oa0-f54.google.com (mail-oa0-f54.google.com [209.85.219.54]) by mx1.freebsd.org (Postfix) with ESMTP id 938428FC0A for ; Wed, 31 Oct 2012 20:24:20 +0000 (UTC) Received: by mail-oa0-f54.google.com with SMTP id n9so2358064oag.13 for ; Wed, 31 Oct 2012 13:24:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=bJvTpIOjy+glawCt6Cg+vfLk8ebS7ksCivMjR0tpWck=; b=FyVH7Jba8R9pFz6/fFk34EN8uCewGSxV0c9Th8Wp1KAnTi7AXm8n7ZZiQiDJwkdRC0 Jtsv9xjxdxUtHREXKXCfkVLFlx4grjpUwWjEutJCv3sIIVYGcLgW4dTjlX24vDBdyvnZ c7VDhQISYOZHzQTn6T0+XtArskbzHOJ/R986BGvtRChtcOIjLS+qM5+p+cznD65QhExj /ekUD8dP79SNdxU3sfGQwISQ6eyPXwKMYxlCFL0b5dV1RTTMhcKI23QI3WtUxxx6S+s2 qmx1mEskKwZPrwXLdhwIscg9ARBY36hG+B8h4sJHLR59VHZH1hUa+qofpoWA23AtW6R1 vyew== MIME-Version: 1.0 Received: by 10.182.150.34 with SMTP id uf2mr31932678obb.66.1351715059841; Wed, 31 Oct 2012 13:24:19 -0700 (PDT) Received: by 10.76.143.33 with HTTP; Wed, 31 Oct 2012 13:24:19 -0700 (PDT) In-Reply-To: <509182DA.8070303@mu.org> References: <509182DA.8070303@mu.org> Date: Wed, 31 Oct 2012 13:24:19 -0700 Message-ID: Subject: Re: make -jN buildworld on < 512MB ram From: Garrett Cooper To: Alfred Perlstein Content-Type: text/plain; charset=ISO-8859-1 Cc: hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Oct 2012 20:24:20 -0000 On Wed, Oct 31, 2012 at 12:58 PM, Alfred Perlstein wrote: > It seems like the new compiler likes to get up to ~200+MB resident when > building some basic things in our tree. > > Unfortunately this causes smaller machines (VMs) to take days because of > swap thrashing. > > Doesn't our make(1) have some stuff to mitigate this? I would expect it to > be a bit smarter about detecting the number of swaps/pages/faults of its > children and taking into account the machine's total ram before forking off > new processes. I know gmake has some algorithms, although last I checked > they were very naive and didn't work well. > > Any ideas? I mean a really simple algorithm could be devised that would be > better than what we appear to have (which is nothing). > > Even if an algorithm can't be come up with, why not something just to > throttle the max number of c++/g++ processes thrown out. Maybe I'm missing > a trick I can pull off with some make.conf knobs? > > Idk, summer of code idea? Anyone mentoring someone they want to have a look > at this? FreeBSD make/bmake doesn't, but gmake sure does with --load [1]! I would ask sjg@ to be absolutely sure, but I don't see any getrlimit/setrlimit calls around the code, apart from RLIMIT_NOFILE. For now you have to tune your number of jobs appropriately.. HTH, -Garrett PS I feel your pain because I have a number of FreeBSD VMs with <=2GB RAM. 1. http://www.gnu.org/software/make/manual/make.html#index-g_t_0040code_007b_002d_002dload_002daverage_007d-746