Date: Fri, 25 Aug 2017 17:10:39 +0200 From: =?UTF-8?Q?Fernando_Apestegu=C3=ADa?= <fernando.apesteguia@gmail.com> To: Dimitry Andric <dim@freebsd.org> Cc: FreeBSD Hackers <freebsd-hackers@freebsd.org>, Ed Maste <emaste@freebsd.org>, Benjamin Kramer <benny.kra@gmail.com>, Hans Wennborg <hans@chromium.org> Subject: Re: Runaway process in -CURRENT Message-ID: <CAGwOe2Z2YPvFP0vzsixfWRAz1XHqDupc1mxmFvjuyocAf%2Bk5Dw@mail.gmail.com> In-Reply-To: <CAGwOe2ZBUDqSXzofkPESkv0q=cCqp%2BCGRMLXERUq7PYBA0znrg@mail.gmail.com> References: <CAGwOe2aCU7HigZJdx2YJCrDZsWUtRVApFzzvPTx%2BLE-AP_B4qA@mail.gmail.com> <71697463-4E52-4B46-B60B-00BDD87D8A2C@FreeBSD.org> <CAGwOe2aODZ3skAEVn9kF3ruKFQtQjFfUk_sa2_hJp99YYgnKsQ@mail.gmail.com> <CAGwOe2bkkNye_B4OjG9e0p4Dnp3EbRyP0=vh6yhfHBQ_6r-7yQ@mail.gmail.com> <CAGwOe2awSX9Svh9Y-3Ew_PMY1=srHLD01CKfqDzeuXNMA3KS5g@mail.gmail.com> <CAGwOe2Y1wFZSr0zEsugzPDoTA2b6oznG=ggZwfSTWoi5h173WA@mail.gmail.com> <EF12ACA3-3E86-4DCA-AC0A-E7D5E6D27438@FreeBSD.org> <CAGwOe2ZBUDqSXzofkPESkv0q=cCqp%2BCGRMLXERUq7PYBA0znrg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Aug 24, 2017 at 8:27 PM, Fernando Apestegu=C3=ADa <fernando.apesteguia@gmail.com> wrote: > On Thu, Aug 24, 2017 at 8:15 PM, Dimitry Andric <dim@freebsd.org> wrote: >> On 24 Aug 2017, at 16:42, Fernando Apestegu=C3=ADa <fernando.apesteguia@= gmail.com> wrote: >>> >>> On Thu, Aug 24, 2017 at 11:14 AM, Fernando Apestegu=C3=ADa >>> <fernando.apesteguia@gmail.com> wrote: >> ... >>> Just for the record, I got another email. Log URL: >>> >>> http://beefy11.nyi.freebsd.org/data/head-i386-default/p448640_s322824/l= ogs/stepcode-0.8_1.log >> ... >>> schemas/sdai_wip210e3/CMakeFiles/sdai_wip210e3.dir/SdaiAll.cc.o -MF >>> schemas/sdai_wip210e3/CMakeFiles/sdai_wip210e3.dir/SdaiAll.cc.o.d -o >>> schemas/sdai_wip210e3/CMakeFiles/sdai_wip210e3.dir/SdaiAll.cc.o -c >>> schemas/sdai_wip210e3/SdaiAll.cc >>> =3D=3D=3D=3D>> Killing runaway build after 7200 seconds with no output >> >> Hm, I had not noticed two facts, one being that there are a great many >> copies of SdaiAll.cc files, which all seem to be generated, and the >> other that this is occurring on i386. There are also other generated >> files, and specifically the compstructs.cc files can take extremely long >> to compile. >> >> On amd64, the longest compile time in the whole port is ~185 seconds, >> for a generated file of ~28000 lines: >> >> time lines filename >> =3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D >> 184.90 28345 schemas/sdai_wip210e3/compstructs.cc >> 115.08 103 schemas/sdai_ap239/SdaiAP239_PRODUCT_LIFE_CYCLE_SUPPORT_= ARM_LF_unity_types.cc >> 110.00 440 schemas/sdai_cd209/SdaiAP209_MULTIDISCIPLINARY_ANALYSIS_= AND_DESIGN_MIM_LF_unity_types.cc >> 106.70 292 schemas/sdai_wip210e3/SdaiAP210_ELECTRONIC_ASSEMBLY_INTE= RCONNECT_AND_PACKAGING_DESIGN_MIM_LF_unity_types.cc >> 90.73 290 schemas/sdai_cd242/SdaiAP242_MANAGED_MODEL_BASED_3D_ENGI= NEERING_MIM_LF_unity_types.cc >> 88.77 2232 schemas/sdai_cd209/SdaiAP209_MULTIDISCIPLINARY_ANALYSIS_= AND_DESIGN_MIM_LF_unity_entities.cc >> 81.87 2174 schemas/sdai_wip210e3/SdaiAP210_ELECTRONIC_ASSEMBLY_INTE= RCONNECT_AND_PACKAGING_DESIGN_MIM_LF_unity_entities.cc >> 73.92 19658 schemas/sdai_cd209/compstructs.cc >> 64.82 1734 schemas/sdai_cd242/SdaiAP242_MANAGED_MODEL_BASED_3D_ENGI= NEERING_MIM_LF_unity_entities.cc >> 61.42 189 schemas/sdai_ap210e2/SdaiAP210_ELECTRONIC_ASSEMBLY_INTER= CONNECT_AND_PACKAGING_DESIGN_MIM_LF_unity_types.cc >> >> On i386, this time is enormously increased, especially for the files >> with lots of lines: >> >> time lines filename >> =3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D >> 4809.23 28345 schemas/sdai_wip210e3/compstructs.cc >> 2164.37 19658 schemas/sdai_cd209/compstructs.cc >> 2056.54 15806 schemas/sdai_ap210e2/compstructs.cc >> 1545.19 16679 schemas/sdai_cd242/compstructs.cc >> 446.49 7667 schemas/sdai_ap203e2/compstructs.cc >> 273.96 6297 schemas/sdai_ap214e3/compstructs.cc >> 187.16 103 schemas/sdai_ap239/SdaiAP239_PRODUCT_LIFE_CYCLE_SUPPORT_= ARM_LF_unity_types.cc >> 163.39 440 schemas/sdai_cd209/SdaiAP209_MULTIDISCIPLINARY_ANALYSIS_= AND_DESIGN_MIM_LF_unity_types.cc >> 130.02 2232 schemas/sdai_cd209/SdaiAP209_MULTIDISCIPLINARY_ANALYSIS_= AND_DESIGN_MIM_LF_unity_entities.cc >> 122.15 290 schemas/sdai_cd242/SdaiAP242_MANAGED_MODEL_BASED_3D_ENGI= NEERING_MIM_LF_unity_types.cc >> > > I did some tests: > > -CURRENT (i326) in VirtualBox, 1 CPU, 8Gb RAM > > cd /usr/ports/cad/stepcode && time make > > * gcc 6.4.0 > > real 27m17.867s > user 50m.0.428s > sys 2m47.619 > > > * clang38 > > real 18m28.996s > user 33m22.790s > sys 1m22.711s > > * clang 5.0.0 > > It takes a lot of time compiling > schemas/sdai_{wip210e3,ap210e2}/SdaiAll.cc and other SdaAll.cc files. > It's still compiling after several hours. > > >> The compstructs.cc and SdaiAll.cc files contain *extremely* large >> generated functions, in most cases embodying the whole file, even. It >> seems that this is the cause for the long compile times. A test case >> tarball with the preprocessed version of the top compstructs.cc file, >> and a shell script to run it, is here: >> http://www.andric.com/freebsd/clang/stepcode.tar.xz >> >> With clang 4.0.0, the compile times are more at the level of the amd64 >> builds, even on i386. So this regressed somewhere along the way to >> 5.0.0. I will have to try to find out where, to see if there is an >> existing bug report or workaround. >> >> Note that in r321664, about three weeks ago, I already committed a >> couple of upstream fixes for some situations where compile times could >> get excessive (see also https://bugs.llvm.org/show_bug.cgi?id=3D33900), >> but it doesn't seem to have helped in this case. >> >> As a quick workaround, the port can be modified to compile these files >> using a lower optimization level. If -O1 is used for the longest case, >> e.g. schemas/sdai_wip210e3/compstructs.cc, the compile time drops down >> to ~30s. I don't think it will matter very much for the speed or size >> of the output. > > I'll have a look at this. Is this problem going to be reported upstream? > > Thanks! > >> >> -Dimitry >>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAGwOe2Z2YPvFP0vzsixfWRAz1XHqDupc1mxmFvjuyocAf%2Bk5Dw>