Date: Sun, 21 Oct 2012 18:20:41 +1100 (EST) From: Bruce Evans <brde@optusnet.com.au> To: "Simon J. Gerraty" <sjg@juniper.net> Cc: "svn-src-head@FreeBSD.org" <svn-src-head@freebsd.org>, "svn-src-all@FreeBSD.org" <svn-src-all@freebsd.org>, "src-committers@FreeBSD.org" <src-committers@freebsd.org>, Bruce Evans <brde@optusnet.com.au>, Marcel Moolenaar <marcel@xcllnt.net> Subject: Re: svn commit: r241298 - in head: . gnu/usr.bin/cc/cc_int gnu/usr.bin/cc/include kerberos5 kerberos5/tools/asn1_compile kerberos5/tools/slc lib/clang/include share/mk tools/build/make_check usr.sbin/c... Message-ID: <20121021172847.N1824@besplex.bde.org> In-Reply-To: <20121020181822.A6C6C58094@chaos.jnpr.net> References: <201210062001.q96K16Or030755@svn.freebsd.org> <20121007161440.C7605@besplex.bde.org> <81CB9A3A-4BFF-41B2-A1F9-3721A40F6260@xcllnt.net> <20121008165228.F7997@besplex.bde.org> <965E2363-6866-4A10-A7B1-C428B3C78925@xcllnt.net> <20121020205008.C1781@besplex.bde.org> <20121020181822.A6C6C58094@chaos.jnpr.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, 20 Oct 2012, Simon J. Gerraty wrote:
> On Sat, 20 Oct 2012 21:13:36 +1100 (EST), Bruce Evans writes:
>> Back to the old thread. Followup to the new commit thread.
>
> I seem to have missed a bit of the conversation, so sorry if some of
> this has been done to death already.
>
> First some background:
>
> I believe the reason for the original change (which was wrong btw,
> I had stared at that diff many times and saw it as what I expected it to
> be - which I think Marcel has now committed),
> is that .ORDER does not work in jobs mode in the presence of explicit
> dependencies (see below).
>
> Since we auto collect dependencies during the build, we've learned to
> avoid .ORDER, it is often responsible for introducing cycles in the
> dependency graph.
That seems to be a bug in the dependency collection. .ORDER expresses
a requirement that is similar to a dependency requirement, and any
dependency that conflicts with that is a bug. In the yacc directives
it expresses the requirement more clearly than a fake dependency --
'.ORDER: a b' was replaced by 'b: a' (or perhaps the reverse, since the
order can be either), but there is no real dependency, only a sequential
requirement. Buggy auto-generation of depencies would cause essentially
the same cycle if it genrated 'a: b'. But old make didn't detect cycles
due to .ORDER.
> Given the makefile:
>
> --------------------8<--------------------
> .MAIN: all
>
> ALPHA=a b c d e f g h i j k l m n o p q r s t u v w x y z
>
> SECS?=1
>
> ${ALPHA}:
> @echo "`date +%s` $@..."; sleep ${SECS}
>
> .NOPATH: ${ALPHA}
> .PHONY: ${ALPHA}
>
> ALL=r a l p h i s g i t
>
> .ifdef ORDER
> x!= echo .ORDER: r a l p h >&2; echo
> .ORDER: r a l p h
> .endif
>
> all: ${ALL}
>
> .ifdef DEPS_A
> x!= echo a: p e >&2; echo
> a: p e
> .endif
> --------------------8<--------------------
>
> The following is using FreeBSD make from 10.x (though it doesn't make
> any difference):
>
> First compat mode:
>
> $ fmake -r -f /home/sjg/make-tests/order
> 1350755164 r...
> 1350755165 a...
> 1350755166 l...
> 1350755168 p...
> 1350755169 h...
> 1350755170 i...
> 1350755171 s...
> 1350755172 g...
> 1350755173 t...
> $
>
> Looks as you would expect, as does jobs mode:
>
> $ fmake -r -f /home/sjg/make-tests/order -j3
> 1350755240 r...
> 1350755240 l...
> 1350755240 a...
> 1350755241 p...
> 1350755241 h...
> 1350755241 i...
> 1350755242 s...
> 1350755242 g...
> 1350755242 t...
> $
>
> Ie. the order is no longer as listed, but we haven't introduce .ORDER yet.
> When we throw .ORDER in:
>
> $ fmake -r -f /home/sjg/make-tests/order -j3 -DORDER
> .ORDER: r a l p h
> 1350755270 s...
> 1350755270 i...
> 1350755270 r...
> 1350755271 g...
> 1350755271 t...
> 1350755271 a...
> 1350755272 l...
> 1350755273 p...
> 1350755275 h...
> $
>
> This is again what we should expect, the order is not as listed but
> the sequence r a l p h is correctly ordered.
>
> However, as soon as we thrown in an explicit dependency - not even one
> that would conflict:
>
> $ fmake -r -f /home/sjg/make-tests/order -j3 -DORDER -DDEPS_A
> .ORDER: r a l p h
> a: p e
> 1350755349 r...
> 1350755350 i...
> 1350755350 s...
> 1350755351 g...
> 1350755351 t...
> 1350755351 e...
> $
>
> That is simply incorrect.
The dependency conflicts with the ordering requirement, so it should be
about make like a cycle does. It says that p must be built before a,
but the ordering requirement says that a must be built before p.
> Even if we go back to compat mode:
>
> $ fmake -r -f /home/sjg/make-tests/order -DORDER -DDEPS_A
> .ORDER: r a l p h
> a: p e
> 1350755881 r...
> 1350755882 p...
> 1350755883 e...
> 1350755884 a...
> 1350755885 l...
> 1350755887 h...
> 1350755888 i...
> 1350755889 s...
> 1350755890 g...
> 1350755891 t...
> $
>
> This is not correct either, but you can see that the dependency takes
> precedence over .ORDER.
The opposite precedence would break things faster, but there is no correct
choice.
Everthing works unsurprisingly with the dependency "p: a" that is consistent
with the order.
> The results with bmake are more spectaclular since it detects that it is
> lost a spews its complaints:
>
> $ bmake -r -f /home/sjg/make-tests/order -DORDER -DDEPS_A -j3
> .ORDER: r a l p h
> a: p e
> --- r ---
> 1350756420 r...
> `p' was not built (made 1, flags 2009, type b030001)!
> `p' has .ORDER dependency against l (made 0, flags 9, type b030001)
> `e' was not built (made 0, flags 2009, type b030001)!
> `l' was not built (made 0, flags 2009, type b030001)!
> `l' has .ORDER dependency against a (made 1, flags 3019, type
> b030001)
> `h' was not built (made 0, flags 2009, type b030001)!
> `h' has .ORDER dependency against p (made 1, flags 2009, type
> b030001)
> `i' was not built (made 0, flags 2009, type b030001)!
> `s' was not built (made 0, flags 2009, type b030001)!
> `g' was not built (made 0, flags 2009, type b030001)!
> `t' was not built (made 0, flags 2009, type b030001)!
> $
>
> This is actually handy since you can check build logs for that sort of
> thing and know you have an issue.
It should detect the problem as an inconsistent makefile earlier (not its
problem).
The requirement for building yacc files would be even more clearly
expressed by a .SEQUENTIAL directive than a .ORDER one, since we
don't care about the precise order. Then there would be no pseudo-
dependency and make would have to choose an order permitted by the
real dependencies. make already has the .NOTPARALLEL directive,
but it applies to the whole makefile.
> The current logic does (for the interesting case - assuming yacc -d etc)
>
> parser.h: parser.y
> yacc etc
>
> parser.c: parser.h
> touch ${.TARGET}
>
> the only downside of this, is that if parser.c is removed
> but parser.h is left alone, and empty parser.c will be produced.
> This is actually a safe failure mode (build will fail), for what is
> essentially a corner case.
>
> In all other respects the above is preferable because it works reliably
> in jobs mode and it avoids any risk of running yacc multiple times to do
> the same thing, and will not conflict with any auto collected
> dependencies (assuming parser.c includes parser.h)
Any dependency of parser.c on parser.h is a bug, whether it is in the
makefile auto-collected. parser.c can't be compiled until parser.h is
created, but this isn't a dependency of parser.c on parser.h; it is
a dependency of the object file on both.
Was the original bug just this buggy dependency? The .ORDER statement
gives the opposite pseudo-dependency, so the bug might be detected
accidentally as a pseudo-cycle.
Bruce
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20121021172847.N1824>