From owner-freebsd-current@freebsd.org Mon Aug 20 14:58:22 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7B93B1070ADC for ; Mon, 20 Aug 2018 14:58:22 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic302-3.consmr.mail.bf2.yahoo.com (sonic302-3.consmr.mail.bf2.yahoo.com [74.6.135.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 254C470F82 for ; Mon, 20 Aug 2018 14:58:22 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: HkooCzIVM1nUbGwjjs43NyMIK9lmckmLO_nlmsW81lr8l.B5Oo9ah0xBwpytXx5 6qOw9f2Y8lVikO5vtcrQoJXTkdLw6C1LRkyvk_3bkFsoLhCjwY8uk6cp5PNgZQXtwVqQ57H5rmjN .XcnPlBuDVvsT8DSYvley2pdLhX0HdJRTPidpEjtr18uqKESEnGlkfj2E0oMJxET3FbY.d7JvdZi 9ZJxD.ASLdQyxPrCtcneX9Gd4mM.iyf7PiYBOn_hvaERVw36aXsu4rtjpeXLdprgqtGoA9w2VG5n RbTAoAU8AnkfCAV_ZU1EAIRxc8ynQ0h89W26CMZaq6bkX3h_Yd8LEn3jWSRBV7c7r03.IOc2gU6U tm20msXdG0SjF11AMOq1Fwfh0zhSiyurriJu5wRx_.VVky5Djh91y4bBhh.aoONReyrxgR_T6SZ6 5yqtrz3ZUbp_wptEL3PFv.wiiCueFTTcx7lQKvQq2xPjYedqK_j3EZ6ueER3AmigucHW_vUUtFGG WEG1.Oq3TUpKQYiEc4ASyWvk_eqYCasQ.OK8oJO6Qgu_tFyqa_gNiITRT8BQW28vgqrlvRPcUTsd XLAgC0jgiZf60yTiWKiEbCHnK5exIWqRmZ7zNaFmPWVcQTOwuBGw9lastBuuxeSQo8zjlRJA460b dMKSO9Q6qGNdPx9Xd6WCL256QQn5DFF9WcCfoMbiruKyR8l0fkykKmMx2V.fsSsWpdxGjoxFjmsD wq_2sAJp6cZGpFN8hjIp6EGxuiDDgAD50Sq9uSZqb7A1gKCVF9jA9gs6sLy0S_SNUK5vJap10wAC dJ_fVI8Ulfo6NXrcWZ7AFVW1KLJi0_Qw1LDo2KMa_P4u9ICYB8iNr7C2.hOaLeNPBjGiqq0E5PJI S.swAgmP69oilIPqidt7vc02qsylxi.FOWgpwkpH70lzm0TVtCaAEt1.fjWJjMIXtzz5OZcRpNIH pC9gHtD_LdSTSSmF2Pq867th_3UZ.AN.PsaeBeYzDujQnaXwFzPDWZCmLtki2P6LWMpJ713Oh11D dtggrTESbdrhYCCpfHBh5sMzn9f0- Received: from sonic.gate.mail.ne1.yahoo.com by sonic302.consmr.mail.bf2.yahoo.com with HTTP; Mon, 20 Aug 2018 14:58:21 +0000 Received: from ip70-189-131-151.lv.lv.cox.net (EHLO [192.168.0.105]) ([70.189.131.151]) by smtp428.mail.bf1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 1e26d093bcee1be57370e1547a46f5e5; Mon, 20 Aug 2018 14:58:16 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: building LLVM threads gets killed Message-Id: <61A99917-931B-45C0-8F6E-6BC0BE5DEAFB@yahoo.com> Date: Mon, 20 Aug 2018 07:58:14 -0700 To: "Rodney W. Grimes" , FreeBSD Current X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2018 14:58:22 -0000 Rodney W. Grimes freebsd-rwg at pdx.rh.CN85.dnsmgr.net wrote on Mon Aug 20 14:26:55 UTC 2018 : > > On 20 Aug 2018, at 05:01, blubee blubeeme = wrote: > > >=20 > > > I am running current compiling LLVM60 and when it comes to linking > > > basically all the processes on my computer gets killed; Chrome, = Firefox and > > > some of the LLVM threads as well > > ... > > > llvm/build % ninja -j8 > > > [2408/2473] Building CXX object > > > lib/Passes/CMakeFiles/LLVMPasses.dir/PassBuilder.cpp.o > > > FAILED: lib/Passes/CMakeFiles/LLVMPasses.dir/PassBuilder.cpp.o > > > /usr/bin/c++ -DGTEST_HAS_RTTI=3D0 -D_DEBUG = -D__STDC_CONSTANT_MACROS > > > -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Ilib/Passes = -I../lib/Passes > > > -Iinclude -I../include -isystem /usr/local/include -fPIC > > > -fvisibility-inlines-hidden -Werror=3Ddate-time > > > -Werror=3Dunguarded-availability-new -std=3Dc++11 -Wall -Wextra > > > -Wno-unused-parameter -Wwrite-strings -Wcast-qual > > > -Wmissing-field-initializers -pedantic -Wno-long-long > > > -Wcovered-switch-default -Wnon-virtual-dtor = -Wdelete-non-virtual-dtor > > > -Wstring-conversion -fdiagnostics-color -g -fno-exceptions = -fno-rtti -MD > > > -MT lib/Passes/CMakeFiles/LLVMPasses.dir/PassBuilder.cpp.o -MF > > > lib/Passes/CMakeFiles/LLVMPasses.dir/PassBuilder.cpp.o.d -o > > > lib/Passes/CMakeFiles/LLVMPasses.dir/PassBuilder.cpp.o -c > > > ../lib/Passes/PassBuilder.cpp > > > c++: error: unable to execute command: Killed > >=20 > > It is running out of RAM while running multiple parallel link jobs. = If > > you are building using WITH_DEBUG, turn that off, it consumes large > > amounts of memory. If you must have debug info, try adding the > > following flag to the CMake command line: > >=20 > > -D LLVM_PARALLEL_LINK_JOBS:STRING=3D"1" > >=20 > > That will limit the amount of parallel link jobs to 1, even if you > > specify -j 8 to gmake or ninja. > >=20 > > Brooks, it would not be a bad idea to always use this CMake flag in = the > > llvm ports. :) >=20 > And this may also fix the issues that all the small > memory (aka, RPI*) buliders are facing when trying > to do -j4? It may help, but: Even just compiles with no links running can get the kills in such small system contexts. And going for a simpler context that can demonstrate the behavior . . . Taking a Pine64+ 2GB as an example (4 cores with 1 HW-thread per core, 2 GiBytes of RAM, USB device for root file system and swap partition): In another login: # stress -d 2 -m 4 --vm-keep --vm-bytes 536870912 That "4" and "536870912" total to the 2 GiBytes so swapping is induced for the context in question. (Scale --vm-bytes appropriately to context.) [Note: I had 3 GiBytes of swap space in a partition for the below.] [stress is from the port sysutils/stress .] I had left the default vm.pageout_oom_seq=3D12 in place for this, making the kills easier to get than the 120 figure would. It does not take very long generally for some sort of message to show up. Sometimes kills happen: My test environment has Mark Johnston patches to report things not normally reported: waited 9s for async swap write waited 9s for swap buffer waited 9s for async swap write waited 9s for async swap write waited 9s for async swap write v_free_count: 1357, v_inactive_count: 1 Aug 20 06:04:27 pine64 kernel: pid 1010 (stress), uid 0, was killed: out = of swap space waited 5s for async swap write waited 5s for swap buffer waited 5s for async swap write waited 5s for async swap write waited 5s for async swap write waited 13s for async swap write waited 12s for swap buffer waited 13s for async swap write waited 12s for async swap write waited 12s for async swap write swap_pager: indefinite wait buffer: bufobj: 0, blkno: 161177, size: = 131072 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 164766, size: = 65536 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 164064, size: = 12288 . . . (I made multiple runs, most manually stopped. None run for all that long.) (Warning: the swap space part of "killed: out of swap space" can be a misnomer. Killing is driven by having low free RAM for sufficiently long. vm.pageout_oom_seq controls how long. Swap space may be unused, little used, or actually be low. With 3 GiBytes of swap space in the partition, these runs were not low on swap space.) Even with fairly stable ms/w figures the queue depths can get to be large, implying long times before the last queued write completes. For example (for a USB storage device for the root file system and swap partition): dT: 1.006s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name 0 0 0 0 0.0 0 0 0.0 0 0 = 0.0 0.0| mmcsd0 56 312 0 0 0.0 312 19985 142.6 0 0 = 0.0 99.6| da0 Large quantities of bytes are being queued in a short time relative to the around 20 MiByte/sec figure that can be sustained. The quantity is spread over the queue entries. Note: The Pine46+ 2GB builds devel/llvm60 in 14.5hr just fine with vm.pageout_oom_seq=3D120 when using all 4 cores. But it has twice the RAM as a rpi3/rpi2 while having the same number of cores. It simply does not have as much to write out as often. (The same sort of point goes for buildworld buildkernel sorts of activities.) =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)