Date: Thu, 11 Feb 2021 08:53:05 -0800 From: Mark Millard <marklmi@yahoo.com> To: Justin Hibbits <chmeeedalf@gmail.com> Cc: freebsd-ppc <freebsd-ppc@freebsd.org> Subject: Re: main (14-CURRENT) may be unstable on powerpc64 Message-ID: <AD35A6BF-198B-4846-9040-07D3C56C4B5D@yahoo.com> In-Reply-To: <6A4F7E52-B0FD-47A2-93EF-632BDBEAB252@yahoo.com> References: <28E64465-8A99-43CF-8B4F-044533EA03C4@yahoo.com> <20210205113544.349ee77e@ralga.knownspace> <52899C49-E0DF-4E19-A89F-8B9376B8F1F5@yahoo.com> <051FDA5B-C529-4805-89F1-3802D75FC7B7@yahoo.com> <E2F91111-2D87-4C1C-AA30-2666CDBFBF62@yahoo.com> <2009AE57-9720-4FF2-9EC9-8F5EFDCB9E78@yahoo.com> <6A4F7E52-B0FD-47A2-93EF-632BDBEAB252@yahoo.com>
index | next in thread | previous in thread | raw e-mail
On 2021-Feb-6, at 17:23, Mark Millard <marklmi at yahoo.com> wrote: > On 2021-Feb-6, at 12:55, Mark Millard <marklmi at yahoo.com> wrote: > >> On 2021-Feb-5, at 23:49, Mark Millard <marklmi at yahoo.com> wrote: >> >>> On 2021-Feb-5, at 21:34, Mark Millard <marklmi at yahoo.com> wrote: >>> >>>> On 2021-Feb-5, at 10:34, Mark Millard <marklmi at yahoo.com> wrote: >>>> >>>> On 2021-Feb-5, at 09:35, Justin Hibbits <chmeeedalf at gmail.com> wrote: >>>>> >>>>>> On Fri, 5 Feb 2021 04:05:55 -0800 >>>>>> Mark Millard via freebsd-ppc <freebsd-ppc@freebsd.org> wrote: >>>>>> >>>>>>> . . . >>>>>> >>>>>> This is probably fallout from 710e45c4b, which has since been reverted. >>>>>> 710e45c4b broke other things like swig as well, which caused a lot of >>>>>> poudriere fallout for me (devel/llvm* failed because swig crashed). >>>>>> >>>>>> Try updating past 33f0540b1 and testing again. >>>>>> >>>>> >>>>> The above is based on 847dfd2803f6, which is after 33f0540b1 . . . >>>>> >>>>> https://cgit.freebsd.org/src/log/?qt=range&q=33f0540b1~1..847dfd2803f6 >>>>> shows: >>>>> >>>>> Commit message (Expand) Author Age Files Lines >>>>> * readelf: do not trucate section name with -W Ed Maste 45 hours 1 -4/+9 >>>>> * readelf: decode LA48 and ASG_DISABLE feature flags Ed Maste 45 hours 1 -0/+2 >>>>> * Add a VM flag to prevent reclaim on a failed contig allocation Ryan Stone 45 hours 3 -2/+11 >>>>> * dwmmc: Multiple busdma fixes. Michal Meloun 46 hours 1 -15/+32 >>>>> * linux: remove locks around callout_drain in timerfd_close() shu 46 hours 1 -2/+0 >>>>> * Revert "Reimplement strlen" Mateusz Guzik 47 hours 2 -53/+108 >>>>> >>>>> (I Probably should have shown that in the original >>>>> message, given the difficulty in determining the >>>>> relative order of referenced commits.) >>>>> >>>>> Before updating to be 847dfd2803f6 based, I had also >>>>> previously hit the swig issue with the llvm10 build. >>>>> That problem failed reliably until after I'd updated >>>>> past the revert. (Not trusting the other things built >>>>> is why I did a -c poudriere bulk after updating to an >>>>> environment based on after the revert.) >>>>> >>>>> The variability in the pkg-static behavior this time >>>>> suggests race conditions are involved, though not >>>>> frequent failures. >>>>> >>>>> >>>> >>>> It still failed to build llvm10 but the error reporoted >>>> is rather different. At least there is a .tbz for me to >>>> expand. I may end up with more to report after that. >>>> >>>> Here is the log file's report: >>>> >>>> [2996/4558] /usr/bin/c++ -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools/clang/tools/extra/clang-tidy/plugin -I/wrkdirs/usr/ports/devel/llvm10/work/llvm-10. >>>> 0.1.src/tools/clang/tools/extra/clang-tidy/plugin -I/wrkdirs/usr/ports/devel/llvm10/work/llvm-10.0.1.src/tools/clang/include -Itools/clang/include -Iinclude -I/wrkdirs/usr/ports/devel/llvm10/work/llvm >>>> -10.0.1.src/include -O2 -pipe -DNDEBUG -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -DNDEBUG -isystem /usr/local/include -fPIC -fvisibility-inlines-hidden -Werror=date-ti >>>> me -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-defa >>>> ult -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -fno-common -Woverloaded-virtual -Wno-nested-anon-types >>>> -O2 -pipe -DNDEBUG -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -DNDEBUG -isystem /usr/local/include -fno-exceptions -std=c++14 -MD -MT tools/clang/tools/extra/clang-tid >>>> y/plugin/CMakeFiles/obj.clangTidyPlugin.dir/ClangTidyPlugin.cpp.o -MF tools/clang/tools/extra/clang-tidy/plugin/CMakeFiles/obj.clangTidyPlugin.dir/ClangTidyPlugin.cpp.o.d -o tools/clang/tools/extra/cl >>>> ang-tidy/plugin/CMakeFiles/obj.clangTidyPlugin.dir/ClangTidyPlugin.cpp.o -c /wrkdirs/usr/ports/devel/llvm10/work/llvm-10.0.1.src/tools/clang/tools/extra/clang-tidy/plugin/ClangTidyPlugin.cpp >>>> FAILED: tools/clang/tools/extra/clang-tidy/plugin/CMakeFiles/obj.clangTidyPlugin.dir/ClangTidyPlugin.cpp.o >>>> /usr/bin/c++ -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools/clang/tools/extra/clang-tidy/plugin -I/wrkdirs/usr/ports/devel/llvm10/work/llvm-10.0.1.src/tool >>>> s/clang/tools/extra/clang-tidy/plugin -I/wrkdirs/usr/ports/devel/llvm10/work/llvm-10.0.1.src/tools/clang/include -Itools/clang/include -Iinclude -I/wrkdirs/usr/ports/devel/llvm10/work/llvm-10.0.1.src/ >>>> include -O2 -pipe -DNDEBUG -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -DNDEBUG -isystem /usr/local/include -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=u >>>> nguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noe >>>> xcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -fno-common -Woverloaded-virtual -Wno-nested-anon-types -O2 -pipe -D >>>> NDEBUG -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -DNDEBUG -isystem /usr/local/include -fno-exceptions -std=c++14 -MD -MT tools/clang/tools/extra/clang-tidy/plugin/CMa >>>> keFiles/obj.clangTidyPlugin.dir/ClangTidyPlugin.cpp.o -MF tools/clang/tools/extra/clang-tidy/plugin/CMakeFiles/obj.clangTidyPlugin.dir/ClangTidyPlugin.cpp.o.d -o tools/clang/tools/extra/clang-tidy/plu >>>> gin/CMakeFiles/obj.clangTidyPlugin.dir/ClangTidyPlugin.cpp.o -c /wrkdirs/usr/ports/devel/llvm10/work/llvm-10.0.1.src/tools/clang/tools/extra/clang-tidy/plugin/ClangTidyPlugin.cpp >>>> PLEASE submit a bug report to https://bugs.freebsd.org/submit/ and include the crash backtrace, preprocessed source, and associated run script. >>>> Stack dump: >>>> 0. Program arguments: /usr/bin/c++ -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools/clang/tools/extra/clang-tidy/plugin -I/wrkdirs/usr/ports/devel/llvm1 >>>> 0/work/llvm-10.0.1.src/tools/clang/tools/extra/clang-tidy/plugin -I/wrkdirs/usr/ports/devel/llvm10/work/llvm-10.0.1.src/tools/clang/include -Itools/clang/include -Iinclude -I/wrkdirs/usr/ports/devel/llvm10/work/llvm-10.0.1.src/include -O2 -pipe -DNDEBUG -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -DNDEBUG -isystem /usr/local/include -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -fno-common -Woverloaded-virtual -Wno-nested-anon-types -O2 -pipe -DNDEBUG -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -DNDEBUG -isystem /usr/local/include -fno-exceptions -std=c++14 -MD -MT tools/clang/tools/extra/clang-tidy/plugin/CMakeFiles/obj.clangTidyPlugin.dir/ClangTidyPlugin.cpp.o -MF tools/clang/tools/extra/clang-tidy/plugin/CMakeFiles/obj.clangTidyPlugin.dir/ClangTidyPlugin.cpp.o.d -o tools/clang/tools/extra/clang-tidy/plugin/CMakeFiles/obj.clangTidyPlugin.dir/ClangTidyPlugin.cpp.o -c /wrkdirs/usr/ports/devel/llvm10/work/llvm-10.0.1.src/tools/clang/tools/extra/clang-tidy/plugin/ClangTidyPlugin.cpp >>>> 1. /wrkdirs/usr/ports/devel/llvm10/work/llvm-10.0.1.src/tools/clang/include/clang/Frontend/Utils.h:165:76: current parser token ')' >>>> 2. /wrkdirs/usr/ports/devel/llvm10/work/llvm-10.0.1.src/tools/clang/include/clang/Frontend/Utils.h:40:1: parsing namespace 'clang' >>>> 3. /wrkdirs/usr/ports/devel/llvm10/work/llvm-10.0.1.src/tools/clang/include/clang/Frontend/Utils.h:149:1: parsing struct/union/class body 'clang::ModuleDependencyCollector' >>>> 4. /wrkdirs/usr/ports/devel/llvm10/work/llvm-10.0.1.src/tools/clang/include/clang/Frontend/Utils.h:165:47: parsing function body 'clang::ModuleDependencyCollector::insertSeen' >>>> 5. /wrkdirs/usr/ports/devel/llvm10/work/llvm-10.0.1.src/tools/clang/include/clang/Frontend/Utils.h:165:47: in compound statement ('{}') >>>> 6. /usr/include/c++/v1/utility:297:29: instantiating class definition 'std::__1::pair<llvm::StringMapIterator<llvm::NoneType>, bool>' >>>> 7. /wrkdirs/usr/ports/devel/llvm10/work/llvm-10.0.1.src/include/llvm/ADT/StringMap.h:34:34: instantiating class definition 'llvm::StringMapIterator<llvm::NoneType>' >>>> 8. /wrkdirs/usr/ports/devel/llvm10/work/llvm-10.0.1.src/include/llvm/ADT/StringMap.h:488:7: instantiating class definition 'llvm::StringMapIterBase<llvm::StringMapIterator<llvm::NoneType>, llvm::StringMapEntry<llvm::NoneType>>' >>>> 9. /wrkdirs/usr/ports/devel/llvm10/work/llvm-10.0.1.src/include/llvm/ADT/iterator.h:67:7: instantiating class definition 'llvm::iterator_facade_base<llvm::StringMapIterator<llvm::NoneType>, std::__1::forward_iterator_tag, llvm::StringMapEntry<llvm::NoneType>, long, llvm::StringMapEntry<llvm::NoneType> *, llvm::StringMapEntry<llvm::NoneType> &>' >>>> #0 0x0000000012f52ed0 llvm::sys::PrintStackTrace(llvm::raw_ostream&) (/usr/bin/c+++0x12f52ed0) >>>> #1 0x0000000012f50770 llvm::sys::RunSignalHandlers() (/usr/bin/c+++0x12f50770) >>>> #2 0x0000000012ee4778 CrashRecoverySignalHandler(int) (/usr/bin/c+++0x12ee4778) >>>> #3 0x0000000813f2b2b4 (/lib/libthr.so.3+0x2f2b4) >>>> c++: error: clang frontend command failed due to signal (use -v to see invocation) >>>> FreeBSD clang version 11.0.1 (git@github.com:llvm/llvm-project.git llvmorg-11.0.1-0-g43ff75f2c3fe) >>>> Target: powerpc64-unknown-freebsd14.0 >>>> Thread model: posix >>>> InstalledDir: /usr/bin >>>> c++: note: diagnostic msg: >>>> ******************** >>>> >>>> PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT: >>>> Preprocessed source(s) and associated run script(s) are located at: >>>> c++: note: diagnostic msg: /tmp/ClangTidyPlugin-21aaa2.cpp >>>> c++: note: diagnostic msg: /tmp/ClangTidyPlugin-21aaa2.sh >>>> c++: note: diagnostic msg: >>>> >>>> >>>> Note: Unfortunately, poudriere does not capture those files >>>> from /tmp/ in the .tbz for the build failure. >>>> >>> >>> No *.core was left behind in the expanded .tbz . >>> /var/log/messages did not have a line reporting >>> any pid signal lines in the time frame. >>> >>> Repeating the failing command in the expansion of >>> the .tbz did not repeat the problem. >>> >>> It looks to be another example of there being one >>> or more not-readily-repeatable problems occurring. >>> >>> It will be many hours before the new poudriere bulk >>> can get back to that same point, but with just the >>> one builder and the one job for the builder. (Still >>> allowing use of an active process per cpu.) The >>> normal/prior bulk runs were allowing 2 jobs. >>> >>> >>> Note: /lib/libthr.so.3+0x2f2b4 seems to be in code >>> from the "static void handle_signal . . ." routine. >>> >> >> The devel/llvm10 rebuild attempt is well past the >> [2996/4558] point now: working on [3056/4558] . >> But it looks like it may have 6+ hrs to go to >> complete (if successful). So far: >> >> [FBSDpowerpc64-default] [2021-02-05_23h31m51s] [parallel_build:] Queued: 1 Built: 0 Failed: 0 Skipped: 0 Ignored: 0 Tobuild: 1 Time: 13:01:28 >> [01]: devel/llvm10 | llvm10-10.0.1_3 build (12:50:05 / 12:59:42) >> [13:02:12] Logs: /usr/local/poudriere/data/logs/bulk/FBSDpowerpc64-default/2021-02-05_23h31m51s >> >> I've no reasonable way to figure out if the parallel >> activity from letting 2 jobs run (so: higher load >> average) would have made a difference or not. Getting >> a very detailed repetition for such is difficult >> so any variation in what context it failed in makes >> useful comparison/contrast problematical. >> >> But, if this run completes successfully, there will >> be 101 more ports to let poudriere bulk try to build, >> one of which would be devel/llvm11 . >> > > It finished, apparently successfully: > > [17:36:24] Built ports: devel/llvm10 > [FBSDpowerpc64-default] [2021-02-05_23h31m51s] [committing:] Queued: 1 Built: 1 Failed: 0 Skipped: 0 Ignored: 0 Tobuild: 0 Time: 17:35:40 > [17:36:24] Logs: /usr/local/poudriere/data/logs/bulk/FBSDpowerpc64-default/2021-02-05_23h31m51s > > I've started the attempt to build what is still pending: > > . . . > [FBSDpowerpc64-default] [2021-02-06_17h13m39s] [balancing_pool:] Queued: 101 Built: 0 Failed: 0 Skipped: 0 Ignored: 0 Tobuild: 101 Time: 00:02:44 > [00:03:27] Recording filesystem state for prepkg... done > [00:03:27] Building 101 packages using 2 builders > [00:03:27] Starting/Cloning builders > [00:04:15] Hit CTRL+t at any time to see build progress and stats > [00:04:15] [01] [00:00:00] Building graphics/mesa-libs | mesa-libs-20.2.3 > [00:04:15] [02] [00:00:00] Building devel/llvm11 | llvm11-11.0.0 > . . . The 101 ports built fine. But after updating to be based on 3acea07c1873: # ~/fbsd-based-on-what-freebsd-main.sh merge-base: 3acea07c1873b1e4042f4a4fa8668745ee59f15b merge-base: CommitDate: 2021-02-08 19:15:21 +0000 c1845d00f818 (HEAD -> mm-src) mm-src snapshot for mm's patched build in git context. 3acea07c1873 (pure-src) Restore the augmented strlen commentary FreeBSD FBSDG5L2 14.0-CURRENT FreeBSD 14.0-CURRENT mm-src-n244686-c1845d00f818 GENERIC64vtsc-NODBG-dcons powerpc powerpc64 1400004 1400004 I updated /usr/ports and started a rebuild, it showing 44 to be rebuilt. The system crashed while doing (from before the crash): [05:12:43] [02] [00:18:39] Finished devel/git@lite | git-lite-2.30.1: Success load: 1.22 cmd: sh 46874 [runnable] 0.00r 0.00u 0.00s 0% 0k [FBSDpowerpc64-default] [2021-02-10_22h25m46s] [parallel_build:] Queued: 44 Built: 41 Failed: 0 Skipped: 0 Ignored: 0 Tobuild: 3 Time: 08:02:20 [01]: devel/boost-libs | boost-libs-1.72.0_3 stage (04:05:07 / 05:46:06) [08:03:03] Logs: /usr/local/poudriere/data/logs/bulk/FBSDpowerpc64-default/2021-02-10_22h25m46s [08:27:48] [01] [06:10:51] Finished devel/boost-libs | boost-libs-1.72.0_3: Success The crash had no messages, fans going by the time I discovered the status. No ddb access was available. So I again have no evidence beyond the crash itself, this time for the whole system. It is again trying to build the 3 ports that had not finished: [FBSDpowerpc64-default] [2021-02-11_08h31m57s] [balancing_pool:] Queued: 3 Built: 0 Failed: 0 Skipped: 0 Ignored: 0 Tobuild: 3 Time: 00:01:15 [00:02:29] Recording filesystem state for prepkg... done [00:02:29] Building 3 packages using 2 builders [00:02:29] Starting/Cloning builders [00:17:48] Hit CTRL+t at any time to see build progress and stats [00:17:48] [01] [00:00:00] Building devel/boost-libs | boost-libs-1.72.0_3 === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?AD35A6BF-198B-4846-9040-07D3C56C4B5D>