Date: Fri, 25 Nov 2005 09:10:50 +1100 From: tridge@samba.org To: Jens Schweikhardt <schweikh@schweikhardt.net> Cc: "Andrew P." <infofarmer@gmail.com>, freebsd-ports@freebsd.org, sobomax@portaone.com Subject: Re: FreeBSD ccache port is wonderfiul! Message-ID: <17286.14954.340289.972852@samba.org> In-Reply-To: <20051124182645.GA1923@schweikhardt.net> References: <200511210625.16973.ringworm01@gmail.com> <cb5206420511210825v7b4dc852jf3f29f325d8ed7fd@mail.gmail.com> <20051124182645.GA1923@schweikhardt.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Jens,
> Note to Tridge: buildworld compiles a cc, installs a complete
> temporary build environment and uses that environment's compiler
> and headers from that point on. The interesting problem is how to
> make ccache find the same cc's. There are ways, but they're not
> entirely obvious.
interesting - that certainly makes things more complex, though ccache
is meant to cope with those sorts of games.
> Please see the thread "Using ccache for build{world, kernel}" on the
> current@ mailing list (only two weeks ago).
ahh, i can see a potential explanation. Looking at this:
http://lists.freebsd.org/pipermail/freebsd-current/2005-November/058052.html
The change of compiler during the build process would normally
invalidate the cache as the mtime/size of the compiler is used as part
of the cache hash index. What you are trying to do is find ways to
defeat that mechanism, as you want to be able to use cached object
files from an older compiler with a newer one.
I see that people have previously used CCACHE_PATH to do this which
overrides the 'find the compiler' code in ccache, and makes ccache use
the same compiler for both stages of the build, which of course is
incorrect, but fast :-)
Your CCACHE_NOHASH_SIZE_MTIME patch takes a different approach, and
just disables the 'detect the compiler has changed' code, allowing the
new compiler to be used, but still using the cached object files from
the old compiler. Strictly speaking that is also unsafe, but will at
least use the right compiler for uncached compiles, which is an
improvement over the CCACHE_PATH method.
So if I understand the problem correctly, ccache would build world in
freebsd fine if none of the overrides were used (CCACHE_PATH or
CCACHE_NOHASH_SIZE_MTIME), but it wouldn't gain nearly as much in
build speed as the 2nd stage build with the freshly installed compiler
would be uncached. Is that right? (I'm just making sure I haven't
missed a real ccache bug somewhere).
So, assuming this isn't a real ccache bug, then really what you are
doing is finding different ways to defeat the compiler bootstrapping
process in build world. Why not have an option in the build process to
not use compiler bootstrapping? If lots of people are defeating it
using ccache tricks anyway, then perhaps you need an option that
explicitly disables it? That might make the source of the problem a
little clearer when people get build failures.
I also noticed this patch from Maxim in the above thread:
http://www.portaone.com/~sobomax/ccache.buildworld
that one changes ccache to use a hash of the compiler binary instead
of the size/mtime. It also adds an extra cacheing layer, which caches
the compilers hash indexed by the hash of its size and mtime. I think
that extra cacheing layer is not really needed, and just adds a lot of
complexity. Hashing the cc binary should be fast enough that it won't
be noticed in the build (especially as it will be in memory), and some
quick tests here seem to confirm that its lost in the noise, at least
with gcc. Unless cc on freebsd is particularly large (gcc on my system
is just 90k) I don't think its a win.
I've committed a simpler patch:
http://build.samba.org/?function=diff;tree=ccache;date=1132869249;author=tridge
which should achieve the same thing. Maybe you could try it on the
freebsd build with CCACHE_HASH_COMPILER set and see if it helps?
Of course, hashing the compiler binary isn't a perfect way of
detecting that a compiler hasn't changed, as compilers often use
multiple stages with separate binaries and the 2nd stage might have
changed with no change in the main cc binary. That's why ccache
normally uses the size/mtime, as that tends to change when a new
compiler is installed, so its more conservative than hashing the
binary. Ideally ccache would have some way of asking the compiler for
some sort if ID that guarantees its the same, but that would be pretty
tricky to add to compilers I think.
Cheers, Tridge
PS: I've committed your CCACHE_NOHASH_SIZE_MTIME patch to cvs. See
http://build.samba.org/?function=diff;tree=ccache;date=1132866608;author=tridge
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?17286.14954.340289.972852>