From owner-freebsd-hackers@freebsd.org Fri Aug 25 15:10:44 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 93347DDA706 for ; Fri, 25 Aug 2017 15:10:44 +0000 (UTC) (envelope-from fernando.apesteguia@gmail.com) Received: from mail-lf0-x22f.google.com (mail-lf0-x22f.google.com [IPv6:2a00:1450:4010:c07::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1279B63C00; Fri, 25 Aug 2017 15:10:44 +0000 (UTC) (envelope-from fernando.apesteguia@gmail.com) Received: by mail-lf0-x22f.google.com with SMTP id d202so229242lfd.5; Fri, 25 Aug 2017 08:10:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=4NvZlriYrt7fWlht9Ru4Y12EcqDsGieHkfLPiUM1wwM=; b=jxghPk7OlM6T2j5B1Yr6JNUAYf/OZvym8W8c+vTrOKDGZ/CPgXE1b30vaXYYCw+A+3 nLiMHEVOlGXWSoWO+qoCX/YJlAzFZxatrMiTIl35eG1OAl8NQp7uGvXshCj+YG/qYov2 BYTGBKCTVsV54m+tJm+nsSTbUBuk+0DbXvVucEjrZkRRoLB1HCoIJpusMCIYdkD0vzEC lH3EyLqkFIZGnCq+8sjoesv5E7f0920xVb1VwAK8P0ipa48iOIpPoVYdoKXpkVicM2IR l0vlJ/BmpFTN52ecNf/7CPtT+nna4rVdbaUQLuJvw5oaGklc+R2sFym5uEYP/UM2GPZD wVnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=4NvZlriYrt7fWlht9Ru4Y12EcqDsGieHkfLPiUM1wwM=; b=du/nXQGrSwVwBtNC8fVTTVR5f6OXsOB1ZPvDM44bIV+VN1fEwdXxa5tFVhqEIBuC0r 2PxS1wlBMZ4muZNMAOIcoK3uD4MVJ7WOfTR5kcJ5hvYiwLM8hPDYyKjoMD/74C+C7D26 rY4nGaSPE7bmQU5YTg3Lo4v4+jYc37Nj2oz7zkiV8HgNju1AxyFQDAxq2SxJ5CYeCYch YkeGCSR2TH6q3qScSky8j+IGtdIvdjYDv08c8LenyuCoaFW4BRPIhn9+RHRUCCUzy5th bwpqUm/GhwzIReWo8J++EjzAKIBvYV1Of5CQPsl9yXynpGqzJkC86UqCgfurfePOM0oe y3rw== X-Gm-Message-State: AHYfb5gK2jX7JU2yrlj44F3T8Luu7R5l7GjaHNsCTIFvBe8L5EjgN88k dpsckTwVOJYr8ie0Qzz4oaf+31xpKA== X-Received: by 10.25.209.148 with SMTP id i142mr3300190lfg.135.1503673840635; Fri, 25 Aug 2017 08:10:40 -0700 (PDT) MIME-Version: 1.0 Received: by 10.25.21.225 with HTTP; Fri, 25 Aug 2017 08:10:39 -0700 (PDT) In-Reply-To: References: <71697463-4E52-4B46-B60B-00BDD87D8A2C@FreeBSD.org> From: =?UTF-8?Q?Fernando_Apestegu=C3=ADa?= Date: Fri, 25 Aug 2017 17:10:39 +0200 Message-ID: Subject: Re: Runaway process in -CURRENT To: Dimitry Andric Cc: FreeBSD Hackers , Ed Maste , Benjamin Kramer , Hans Wennborg Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Aug 2017 15:10:44 -0000 On Thu, Aug 24, 2017 at 8:27 PM, Fernando Apestegu=C3=ADa wrote: > On Thu, Aug 24, 2017 at 8:15 PM, Dimitry Andric wrote: >> On 24 Aug 2017, at 16:42, Fernando Apestegu=C3=ADa wrote: >>> >>> On Thu, Aug 24, 2017 at 11:14 AM, Fernando Apestegu=C3=ADa >>> wrote: >> ... >>> Just for the record, I got another email. Log URL: >>> >>> http://beefy11.nyi.freebsd.org/data/head-i386-default/p448640_s322824/l= ogs/stepcode-0.8_1.log >> ... >>> schemas/sdai_wip210e3/CMakeFiles/sdai_wip210e3.dir/SdaiAll.cc.o -MF >>> schemas/sdai_wip210e3/CMakeFiles/sdai_wip210e3.dir/SdaiAll.cc.o.d -o >>> schemas/sdai_wip210e3/CMakeFiles/sdai_wip210e3.dir/SdaiAll.cc.o -c >>> schemas/sdai_wip210e3/SdaiAll.cc >>> =3D=3D=3D=3D>> Killing runaway build after 7200 seconds with no output >> >> Hm, I had not noticed two facts, one being that there are a great many >> copies of SdaiAll.cc files, which all seem to be generated, and the >> other that this is occurring on i386. There are also other generated >> files, and specifically the compstructs.cc files can take extremely long >> to compile. >> >> On amd64, the longest compile time in the whole port is ~185 seconds, >> for a generated file of ~28000 lines: >> >> time lines filename >> =3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D >> 184.90 28345 schemas/sdai_wip210e3/compstructs.cc >> 115.08 103 schemas/sdai_ap239/SdaiAP239_PRODUCT_LIFE_CYCLE_SUPPORT_= ARM_LF_unity_types.cc >> 110.00 440 schemas/sdai_cd209/SdaiAP209_MULTIDISCIPLINARY_ANALYSIS_= AND_DESIGN_MIM_LF_unity_types.cc >> 106.70 292 schemas/sdai_wip210e3/SdaiAP210_ELECTRONIC_ASSEMBLY_INTE= RCONNECT_AND_PACKAGING_DESIGN_MIM_LF_unity_types.cc >> 90.73 290 schemas/sdai_cd242/SdaiAP242_MANAGED_MODEL_BASED_3D_ENGI= NEERING_MIM_LF_unity_types.cc >> 88.77 2232 schemas/sdai_cd209/SdaiAP209_MULTIDISCIPLINARY_ANALYSIS_= AND_DESIGN_MIM_LF_unity_entities.cc >> 81.87 2174 schemas/sdai_wip210e3/SdaiAP210_ELECTRONIC_ASSEMBLY_INTE= RCONNECT_AND_PACKAGING_DESIGN_MIM_LF_unity_entities.cc >> 73.92 19658 schemas/sdai_cd209/compstructs.cc >> 64.82 1734 schemas/sdai_cd242/SdaiAP242_MANAGED_MODEL_BASED_3D_ENGI= NEERING_MIM_LF_unity_entities.cc >> 61.42 189 schemas/sdai_ap210e2/SdaiAP210_ELECTRONIC_ASSEMBLY_INTER= CONNECT_AND_PACKAGING_DESIGN_MIM_LF_unity_types.cc >> >> On i386, this time is enormously increased, especially for the files >> with lots of lines: >> >> time lines filename >> =3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D >> 4809.23 28345 schemas/sdai_wip210e3/compstructs.cc >> 2164.37 19658 schemas/sdai_cd209/compstructs.cc >> 2056.54 15806 schemas/sdai_ap210e2/compstructs.cc >> 1545.19 16679 schemas/sdai_cd242/compstructs.cc >> 446.49 7667 schemas/sdai_ap203e2/compstructs.cc >> 273.96 6297 schemas/sdai_ap214e3/compstructs.cc >> 187.16 103 schemas/sdai_ap239/SdaiAP239_PRODUCT_LIFE_CYCLE_SUPPORT_= ARM_LF_unity_types.cc >> 163.39 440 schemas/sdai_cd209/SdaiAP209_MULTIDISCIPLINARY_ANALYSIS_= AND_DESIGN_MIM_LF_unity_types.cc >> 130.02 2232 schemas/sdai_cd209/SdaiAP209_MULTIDISCIPLINARY_ANALYSIS_= AND_DESIGN_MIM_LF_unity_entities.cc >> 122.15 290 schemas/sdai_cd242/SdaiAP242_MANAGED_MODEL_BASED_3D_ENGI= NEERING_MIM_LF_unity_types.cc >> > > I did some tests: > > -CURRENT (i326) in VirtualBox, 1 CPU, 8Gb RAM > > cd /usr/ports/cad/stepcode && time make > > * gcc 6.4.0 > > real 27m17.867s > user 50m.0.428s > sys 2m47.619 > > > * clang38 > > real 18m28.996s > user 33m22.790s > sys 1m22.711s > > * clang 5.0.0 > > It takes a lot of time compiling > schemas/sdai_{wip210e3,ap210e2}/SdaiAll.cc and other SdaAll.cc files. > It's still compiling after several hours. > > >> The compstructs.cc and SdaiAll.cc files contain *extremely* large >> generated functions, in most cases embodying the whole file, even. It >> seems that this is the cause for the long compile times. A test case >> tarball with the preprocessed version of the top compstructs.cc file, >> and a shell script to run it, is here: >> http://www.andric.com/freebsd/clang/stepcode.tar.xz >> >> With clang 4.0.0, the compile times are more at the level of the amd64 >> builds, even on i386. So this regressed somewhere along the way to >> 5.0.0. I will have to try to find out where, to see if there is an >> existing bug report or workaround. >> >> Note that in r321664, about three weeks ago, I already committed a >> couple of upstream fixes for some situations where compile times could >> get excessive (see also https://bugs.llvm.org/show_bug.cgi?id=3D33900), >> but it doesn't seem to have helped in this case. >> >> As a quick workaround, the port can be modified to compile these files >> using a lower optimization level. If -O1 is used for the longest case, >> e.g. schemas/sdai_wip210e3/compstructs.cc, the compile time drops down >> to ~30s. I don't think it will matter very much for the speed or size >> of the output. > > I'll have a look at this. Is this problem going to be reported upstream? > > Thanks! > >> >> -Dimitry >>