Date: Thu, 1 Nov 2012 09:14:39 +1100 From: Peter Jeremy <peter@rulingia.com> To: Alfred Perlstein <bright@mu.org> Cc: hackers@freebsd.org Subject: Re: make -jN buildworld on < 512MB ram Message-ID: <20121031221439.GM3309@server.rulingia.com> In-Reply-To: <5091966F.6070706@mu.org> References: <509182DA.8070303@mu.org> <20121031204152.GK3309@server.rulingia.com> <5091966F.6070706@mu.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--3XA6nns4nE4KvaS/
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On 2012-Oct-31 14:21:51 -0700, Alfred Perlstein <bright@mu.org> wrote:
>Ah, but make(1) can delay spawning any new processes when it knows its=20
>children are paging.
That could work in some cases and may be worth implementing. Where it
won't work is when make(1) initially hits a parallelisable block of
"big" programs after a series of short, small tasks: System is OK so
the first big program is spawned. ~100msec later, the next small task
finishes. System in still OK (because the first big task is still
growing and hasn't achieved peak bloat[*]) so it spawns another big task.
Repeat a few times and you have a collection of big processes starting
to thrash the system.
>> Another, more involved, approach would be for the scheduler to manage
>> groups of processes - if a group of processes is causing memory
>> pressure as a whole then the scheduler just stops scheduling some of
>> them until the pressure reduces (effectively swap them out). (Yes,
>> that's vague and lots of hand-waving that might not be realisable).
>>
>I think that could be done, this is actually a very interesting idea.
>
>Another idea is for make(1) to start to kill -STOP a child when it=20
>detects a lot of child paging until other independent children complete=20
>running, which is basically what I do manually when my build explodes=20
>until it gets past some C++ bits.
This is roughly a userland variant of the scheduler change above. The
downside is that make(1) can no longer just wait(2) for a process to
exit and then decide what to do next. Instead, it needs to poll the
system's paging activity and take action on one of its children. Some
of the special cases it needs ta handle are:
1) The offending process isn't a direct child but a more distant=20
descendent - this will be the typical case: make(1) starts gcc(1)
which spawns cc1plus which bloats.
2) Multiple (potentially independent) make(1) processes all detect that
the system is too busy and stop their children. Soon after, the
system is free so they all SIGCONT their children. Repeat. (Note
that any scheduler changes also need to cope with this).
[*] Typical cc1/cc1plus behaviour is to steadily grow as the input is
processed. At higher optimisation levels, parse trees are not
freed at the end of a function to allow global inlining and
optimisation.
--=20
Peter Jeremy
--3XA6nns4nE4KvaS/
Content-Type: application/pgp-signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (FreeBSD)
iEYEARECAAYFAlCRos8ACgkQ/opHv/APuIf3kwCeMi5LYN1EUby33hKzzuN/htpI
gkYAni/+wCXieiDSKoPIOgmhQEH/5vTp
=oNfW
-----END PGP SIGNATURE-----
--3XA6nns4nE4KvaS/--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20121031221439.GM3309>