From owner-freebsd-hackers@freebsd.org Thu Aug 24 18:27:50 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 45D0DDE5962 for ; Thu, 24 Aug 2017 18:27:50 +0000 (UTC) (envelope-from fernando.apesteguia@gmail.com) Received: from mail-lf0-x22c.google.com (mail-lf0-x22c.google.com [IPv6:2a00:1450:4010:c07::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BC5692197; Thu, 24 Aug 2017 18:27:49 +0000 (UTC) (envelope-from fernando.apesteguia@gmail.com) Received: by mail-lf0-x22c.google.com with SMTP id z130so1107868lff.3; Thu, 24 Aug 2017 11:27:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=6/1DaqOxPXq0jaIkJzMADmYvuJgVQTZbLyOswHA5i1Y=; b=qhpxoUQUKeQf6KBMMqtfCIcJBDS0qJ/CCQa2/HwLWrewymc9bOYc+ErGnaQezRfuzJ RyBOjaFDcgnXkErTWFlTgpT9Qhh6MxIvVZiGipv719pGmf1I6dg6XhjiFCVp5B9FDgAh uJvQQ4VTydcXvkYBshzYDDufYvNx8kf66YbiHlsc0RLnx78PXriLGv7dOP4bROFnODTA oSrPeVN6MLGGkELBIR6B3EPyT81zCWt6fFxzv6L0xa7qEVYRCJ4PU8nMYpJVRDPANheX xQt/5Bl1V5Ac9IPBHn7Jyk8iQU74DYDytWRZrgOL5UFZ0jqWBbguEO+f1EfZOOZs3lrd MkiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=6/1DaqOxPXq0jaIkJzMADmYvuJgVQTZbLyOswHA5i1Y=; b=HG6t0WOZgYDojsSEfA6cpiJLBFZCR/sbCEltugIWCKZQmVKAL41+h5PUgQh8rMDVSP aZbTemm5oMJcQLOE5AfNudo1rcfN1Fs5e3MPk6dWqtqDTlCkG+bj74lTkHG5aIQjA02U Zaf/rJT5XYYsmaZKiFixOy6NadRzWYD5LDnzR0iy6LKpGVe7Ql8QSIBXkUfYEEwC3gH5 y6grLXChOTSmrIwzCcOdDPMRInWZUZsXG08ZzboQjOVY5PtxIZk3Dn0OYKcKmlSZWXjC x7rR890nQdtGfMhWvEu0r46cl5eGmJMqNK22PlEAw3CxGDn31fY0nvp3v6KL6NWLa0ai 37XQ== X-Gm-Message-State: AHYfb5hAw1IHRjz++9ZtwqzK2IYCNmNw+3Hlp2+xJHS2wo33qxTQxdzi 07VxNZ+LYPyJ0US6DuNQbGMERmerehXO X-Received: by 10.46.9.65 with SMTP id 62mr2846326ljj.177.1503599267557; Thu, 24 Aug 2017 11:27:47 -0700 (PDT) MIME-Version: 1.0 Received: by 10.25.21.225 with HTTP; Thu, 24 Aug 2017 11:27:46 -0700 (PDT) In-Reply-To: References: <71697463-4E52-4B46-B60B-00BDD87D8A2C@FreeBSD.org> From: =?UTF-8?Q?Fernando_Apestegu=C3=ADa?= Date: Thu, 24 Aug 2017 20:27:46 +0200 Message-ID: Subject: Re: Runaway process in -CURRENT To: Dimitry Andric Cc: FreeBSD Hackers , Ed Maste , Benjamin Kramer , Hans Wennborg Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Aug 2017 18:27:50 -0000 On Thu, Aug 24, 2017 at 8:15 PM, Dimitry Andric wrote: > On 24 Aug 2017, at 16:42, Fernando Apestegu=C3=ADa wrote: >> >> On Thu, Aug 24, 2017 at 11:14 AM, Fernando Apestegu=C3=ADa >> wrote: > ... >> Just for the record, I got another email. Log URL: >> >> http://beefy11.nyi.freebsd.org/data/head-i386-default/p448640_s322824/lo= gs/stepcode-0.8_1.log > ... >> schemas/sdai_wip210e3/CMakeFiles/sdai_wip210e3.dir/SdaiAll.cc.o -MF >> schemas/sdai_wip210e3/CMakeFiles/sdai_wip210e3.dir/SdaiAll.cc.o.d -o >> schemas/sdai_wip210e3/CMakeFiles/sdai_wip210e3.dir/SdaiAll.cc.o -c >> schemas/sdai_wip210e3/SdaiAll.cc >> =3D=3D=3D=3D>> Killing runaway build after 7200 seconds with no output > > Hm, I had not noticed two facts, one being that there are a great many > copies of SdaiAll.cc files, which all seem to be generated, and the > other that this is occurring on i386. There are also other generated > files, and specifically the compstructs.cc files can take extremely long > to compile. > > On amd64, the longest compile time in the whole port is ~185 seconds, > for a generated file of ~28000 lines: > > time lines filename > =3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D > 184.90 28345 schemas/sdai_wip210e3/compstructs.cc > 115.08 103 schemas/sdai_ap239/SdaiAP239_PRODUCT_LIFE_CYCLE_SUPPORT_A= RM_LF_unity_types.cc > 110.00 440 schemas/sdai_cd209/SdaiAP209_MULTIDISCIPLINARY_ANALYSIS_A= ND_DESIGN_MIM_LF_unity_types.cc > 106.70 292 schemas/sdai_wip210e3/SdaiAP210_ELECTRONIC_ASSEMBLY_INTER= CONNECT_AND_PACKAGING_DESIGN_MIM_LF_unity_types.cc > 90.73 290 schemas/sdai_cd242/SdaiAP242_MANAGED_MODEL_BASED_3D_ENGIN= EERING_MIM_LF_unity_types.cc > 88.77 2232 schemas/sdai_cd209/SdaiAP209_MULTIDISCIPLINARY_ANALYSIS_A= ND_DESIGN_MIM_LF_unity_entities.cc > 81.87 2174 schemas/sdai_wip210e3/SdaiAP210_ELECTRONIC_ASSEMBLY_INTER= CONNECT_AND_PACKAGING_DESIGN_MIM_LF_unity_entities.cc > 73.92 19658 schemas/sdai_cd209/compstructs.cc > 64.82 1734 schemas/sdai_cd242/SdaiAP242_MANAGED_MODEL_BASED_3D_ENGIN= EERING_MIM_LF_unity_entities.cc > 61.42 189 schemas/sdai_ap210e2/SdaiAP210_ELECTRONIC_ASSEMBLY_INTERC= ONNECT_AND_PACKAGING_DESIGN_MIM_LF_unity_types.cc > > On i386, this time is enormously increased, especially for the files > with lots of lines: > > time lines filename > =3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D > 4809.23 28345 schemas/sdai_wip210e3/compstructs.cc > 2164.37 19658 schemas/sdai_cd209/compstructs.cc > 2056.54 15806 schemas/sdai_ap210e2/compstructs.cc > 1545.19 16679 schemas/sdai_cd242/compstructs.cc > 446.49 7667 schemas/sdai_ap203e2/compstructs.cc > 273.96 6297 schemas/sdai_ap214e3/compstructs.cc > 187.16 103 schemas/sdai_ap239/SdaiAP239_PRODUCT_LIFE_CYCLE_SUPPORT_A= RM_LF_unity_types.cc > 163.39 440 schemas/sdai_cd209/SdaiAP209_MULTIDISCIPLINARY_ANALYSIS_A= ND_DESIGN_MIM_LF_unity_types.cc > 130.02 2232 schemas/sdai_cd209/SdaiAP209_MULTIDISCIPLINARY_ANALYSIS_A= ND_DESIGN_MIM_LF_unity_entities.cc > 122.15 290 schemas/sdai_cd242/SdaiAP242_MANAGED_MODEL_BASED_3D_ENGIN= EERING_MIM_LF_unity_types.cc > I did some tests: -CURRENT (i326) in VirtualBox, 1 CPU, 8Gb RAM cd /usr/ports/cad/stepcode && time make * gcc 6.4.0 real 27m17.867s user 50m.0.428s sys 2m47.619 * clang38 real 18m28.996s user 33m22.790s sys 1m22.711s * clang 5.0.0 It takes a lot of time compiling schemas/sdai_{wip210e3,ap210e2}/SdaiAll.cc and other SdaAll.cc files. It's still compiling after several hours. > The compstructs.cc and SdaiAll.cc files contain *extremely* large > generated functions, in most cases embodying the whole file, even. It > seems that this is the cause for the long compile times. A test case > tarball with the preprocessed version of the top compstructs.cc file, > and a shell script to run it, is here: > http://www.andric.com/freebsd/clang/stepcode.tar.xz > > With clang 4.0.0, the compile times are more at the level of the amd64 > builds, even on i386. So this regressed somewhere along the way to > 5.0.0. I will have to try to find out where, to see if there is an > existing bug report or workaround. > > Note that in r321664, about three weeks ago, I already committed a > couple of upstream fixes for some situations where compile times could > get excessive (see also https://bugs.llvm.org/show_bug.cgi?id=3D33900), > but it doesn't seem to have helped in this case. > > As a quick workaround, the port can be modified to compile these files > using a lower optimization level. If -O1 is used for the longest case, > e.g. schemas/sdai_wip210e3/compstructs.cc, the compile time drops down > to ~30s. I don't think it will matter very much for the speed or size > of the output. I'll have a look at this. Thanks! > > -Dimitry >