Date: Mon, 3 Apr 2017 11:38:50 -0700 From: Mark Millard <markmi@dsl-only.net> To: Brooks Davis <brooks@freebsd.org>, Dimitry Andric <dim@FreeBSD.org>, FreeBSD Ports <freebsd-ports@freebsd.org> Cc: FreeBSD Toolchain <freebsd-toolchain@freebsd.org>, FreeBSD Current <freebsd-current@freebsd.org>, FreeBSD PowerPC ML <freebsd-ppc@freebsd.org> Subject: Re: FYI: what it takes for RAM+swap to build devel/llvm40 with 4 processors or cores and WITH__DEBUG= (powerpc64 example) Message-ID: <543D47AE-7C0D-4F0E-83B4-BFEE5B802346@dsl-only.net> In-Reply-To: <FD405CB7-CF1C-4F81-A5A8-46FACF886576@dsl-only.net> References: <3EDEF0B7-59C5-4648-9737-6682E18645BC@dsl-only.net> <39C60316-F905-490D-B0AB-BC24D7F351A2@dsl-only.net> <7F94CE59-D2CC-4D6F-B1CD-FF3D1F8EDCE7@FreeBSD.org> <45E32F4F-A238-47AA-8E1E-7AD4D9E857D9@dsl-only.net> <20170329155316.GK59667@spindle.one-eyed-alien.net> <F6F4B285-5D72-4B14-A12E-3D789AED063C@dsl-only.net> <FDFF229D-B9DB-4798-8743-01CAF218BD09@dsl-only.net> <F315A080-0989-4521-A899-A0A06721934B@dsl-only.net> <FD405CB7-CF1C-4F81-A5A8-46FACF886576@dsl-only.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2017-Apr-1, at 3:51 AM, Mark Millard <markmi at dsl-only.net> wrote:
> On 2017-Mar-31, at 4:51 PM, Mark Millard <markmi at dsl-only.net> =
wrote:
>=20
>> On 2017-Mar-30, at 7:51 PM, Mark Millard <markmi at dsl-only.net> =
wrote:
>>=20
>>> On 2017-Mar-30, at 1:22 PM, Mark Millard <markmi at dsl-only.net> =
wrote:
>>>=20
>>>> Sounds like the ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG technique
>>>> would not change the "WITNESS and INVARIANTS"-like part of the
>>>> issue. In fact if WITH_DEBUG=3D causes the cmake debug-style
>>>> llvm40 build ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG might not
>>>> make any difference: separate enforcing of lack of optimization.
>>>>=20
>>>> But just to see what results I've done "pkg delete llvm40"
>>>> and am doing another build with ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG=3D=
>>>> and its supporting code in place in addition to using WITH_DEBUG=3D
>>>> as the type of build fro FreeBSD's viewpoint.
>>>>=20
>>>> If you know that the test is a waste of machine cycles, you can
>>>> let me know if you want.
>>>=20
>>> The experiment showed that ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG
>>> use made no difference for devel/llvm40 so devel/llvm40 itself
>>> has to change such as what Dimitry Andric reported separately
>>> as a working change to the Makefile .
>>>=20
>>> (ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG would still have its uses
>>> for various other ports.)
>>=20
>> I've now tried with both ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG and:
>=20
> I may have had a textual error that prevented
> ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG from even potentially
> contributing. So I'll re-run this test.
>=20
> For now I presume that what I reported was okay and so
> I continue to refer to these figures later below.
The retry got the same 42 GiByte and 102 GiByte sizes.
(And again I was not monitoring the swap space usage.)
So functionality like ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG
(keeping optimization flags in CFLAGS) does not contribute
to devel/llvm40's handling. Apparently the CMAKE_BUILD_TYPE
has full control over such and RelWithDebInfo still makes
for a massive build, though not as big as for DEBUG.
For my context I've chosen to go with:
#
# =46rom a local /usr/ports/Mk/bsd.port.mk extension:
ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG=3D
#
.if ${.CURDIR:M*/devel/llvm*}
#WITH_DEBUG=3D
.else
WITH_DEBUG=3D
.endif
WITH_DEBUG_FILES=3D
(where ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG is from a
local change to /usr/ports/Mk/bsd.port.mk but is
not justified via devel/llvm* ports but via behavior
for most other ports.)
>> # svnlite diff /usr/ports/devel/llvm40/
>> Index: /usr/ports/devel/llvm40/Makefile
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> --- /usr/ports/devel/llvm40/Makefile (revision 436747)
>> +++ /usr/ports/devel/llvm40/Makefile (working copy)
>> @@ -236,6 +236,11 @@
>>=20
>> .include <bsd.port.pre.mk>
>>=20
>> +.if defined(WITH_DEBUG)
>> +CMAKE_BUILD_TYPE=3D RelWithDebInfo
>> +STRIP=3D
>> +.endif
>> +
>> _CRTLIBDIR=3D =
${LLVM_PREFIX:S|${PREFIX}/||}/lib/clang/${LLVM_RELEASE}/lib/freebsd
>> .if ${ARCH} =3D=3D "amd64"
>> _COMPILER_RT_LIBS=3D \
>>=20
>>=20
>>=20
>> pkg delete after the build reports:
>>=20
>> Installed packages to be REMOVED:
>> llvm40-4.0.0
>>=20
>> Number of packages to be removed: 1
>>=20
>> The operation will free 42 GiB.
>>=20
>> So down by 7 GiBytes from 49 GiBytes.
>>=20
>> (I did not actually delete it.)
>>=20
>> Also:
>>=20
>> # du -sg /usr/obj/portswork/usr/ports/devel/llvm40
>> 102 /usr/obj/portswork/usr/ports/devel/llvm40
>>=20
>> which is down by 16 GiBytes from 118 GiBytes.
>>=20
>> Reminder: These are from portmaster -DK so no
>> cleanup after the build, which is what leaves
>> the source code and such around in case of
>> needing to look at a problem.
>>=20
>> (102+42) GiBytes =3D=3D 146 GiBytes.
>> vs.
>> (118+49) GiBytes =3D=3D 167 GiBytes.
>>=20
>> So a difference of 21 GiBytes (or so).
>>=20
>> But that is for everything in each case (and
>> WITH_DEBUG=3D in use):
>>=20
>> # more /var/db/ports/devel_llvm40/options
>> # This file is auto-generated by 'make config'.
>> # Options for llvm40-4.0.0.r4
>> _OPTIONS_READ=3Dllvm40-4.0.0.r4
>> _FILE_COMPLETE_OPTIONS_LIST=3DCLANG DOCS EXTRAS LIT LLD LLDB
>> OPTIONS_FILE_SET+=3DCLANG
>> OPTIONS_FILE_SET+=3DDOCS
>> OPTIONS_FILE_SET+=3DEXTRAS
>> OPTIONS_FILE_SET+=3DLIT
>> OPTIONS_FILE_SET+=3DLLD
>> OPTIONS_FILE_SET+=3DLLDB
>>=20
>> So avoiding WITH_DEBUG=3D and/or various build options
>> is still the major way of avoiding use of lots of space
>> if it is an issue.
>>=20
>>=20
>>=20
>> Why no RAM+SWAP total report this time:
>>=20
>> As far as I know FreeBSD does not track or report peak
>> swap-space usage since the last boot. And, unfortunately
>> I was not around to just sit and watch a top display this
>> time and I did not set up any periodic recording into a
>> file.
>>=20
>> That is why I've not reported on the RAM+SWAP total
>> this time. It will have to be another experiment
>> some other time.
>>=20
>> [I do wish FreeBSD had a way of reporting peak swap-space
>> usage.]
>=20
> I've also tried without WITH_DEBUG=3D and now. . .
>=20
>=20
> # pkg delete llvm40
> Checking integrity... done (0 conflicting)
> Deinstallation has been requested for the following 1 packages (of 0 =
packages in the universe):
>=20
> Installed packages to be REMOVED:
> llvm40-4.0.0
>=20
> Number of packages to be removed: 1
>=20
> The operation will free 1 GiB.
>=20
> Proceed with deinstalling packages? [y/N]: n
>=20
>=20
> # du -sg /usr/obj/portswork/usr/ports/devel/llvm40/
> 5 /usr/obj/portswork/usr/ports/devel/llvm40/
>=20
>=20
> So the alternatives (with everything built each time):
>=20
> (5+1) GiBytes =3D=3D 6 GiBytes. (without WITH_DEBUG=3D)
> vs.
> (102+42) GiBytes =3D=3D 146 GiBytes. (WITH_DEBUG=3D but the adjusted =
llvm40/Makefiele)
> vs.
> (118+49) GiBytes =3D=3D 167 GiBytes. (WITH_DEBUG=3D with Makefile =
adjustment)
>=20
>=20
> I'll likely end up having /etc/make.conf contain
> something like the following for most or all of
> my FreeBSD environments:
>=20
> #
> # =46rom a local /usr/ports/Mk/bsd.port.mk extension:
> ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG=3D
> #
> .if ${.CURDIR:M*/devel/llvm*}
> #WITH_DEBUG=3D
> .else
> WITH_DEBUG=3D
> .endif
> WITH_DEBUG_FILES=3D
>=20
>=20
> Along with using:
>=20
> # svnlite diff /usr/ports/Mk/
> Index: /usr/ports/Mk/bsd.port.mk
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> --- /usr/ports/Mk/bsd.port.mk (revision 436747)
> +++ /usr/ports/Mk/bsd.port.mk (working copy)
> @@ -1646,7 +1646,11 @@
> STRIP_CMD=3D ${TRUE}
> .endif
> DEBUG_FLAGS?=3D -g
> +.if defined(ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG)
> +CFLAGS:=3D ${CFLAGS} ${DEBUG_FLAGS}
> +.else
> CFLAGS:=3D ${CFLAGS:N-O*:N-fno-strict*} ${DEBUG_FLAGS}
> +.endif
> .if defined(INSTALL_TARGET)
> INSTALL_TARGET:=3D ${INSTALL_TARGET:S/^install-strip$/install/g}
> .endif
>=20
>=20
> [Although apparently the ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG
> use has no effect for devel/llvm40 .]
=3D=3D=3D
Mark Millard
markmi at dsl-only.net
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?543D47AE-7C0D-4F0E-83B4-BFEE5B802346>