From owner-freebsd-current@freebsd.org Sun Jan 17 17:40:10 2021 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 88C064D159E; Sun, 17 Jan 2021 17:40:10 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (www.zefox.net [50.1.20.27]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "www.zefox.com", Issuer "www.zefox.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DJhyK4pvnz3lQY; Sun, 17 Jan 2021 17:40:09 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (localhost [127.0.0.1]) by www.zefox.net (8.16.1/8.15.2) with ESMTPS id 10HHe6Zg030969 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sun, 17 Jan 2021 09:40:07 -0800 (PST) (envelope-from fbsd@www.zefox.net) Received: (from fbsd@localhost) by www.zefox.net (8.16.1/8.15.2/Submit) id 10HHe6C5030968; Sun, 17 Jan 2021 09:40:06 -0800 (PST) (envelope-from fbsd) Date: Sun, 17 Jan 2021 09:40:06 -0800 From: bob prohaska To: bob prohaska Cc: Current FreeBSD , freebsd-arm@freebsd.org Subject: Silent hang in buildworld, was Re: Invoking -v for clang during buildworld Message-ID: <20210117174006.GA30728@www.zefox.net> References: <20210116043740.GA19523@www.zefox.net> <20210116155538.GA24259@www.zefox.net> <20210116220334.GA26756@www.zefox.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4DJhyK4pvnz3lQY X-Spamd-Bar: - X-Spamd-Result: default: False [-1.06 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; WWW_DOT_DOMAIN(0.50)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[zefox.net]; RBL_DBL_DONT_QUERY_IPS(0.00)[50.1.20.27:from]; AUTH_NA(1.00)[]; SPAMHAUS_ZRD(0.00)[50.1.20.27:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.96)[-0.963]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:7065, ipnet:50.1.16.0/20, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-current,freebsd-arm]; MID_RHS_WWW(0.50)[] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Jan 2021 17:40:10 -0000 On Sat, Jan 16, 2021 at 03:04:04PM -0800, Mark Millard wrote: > > Other than -j1 style builds (or equivalent), one pretty much > always needs to go looking around for a non-panic failure. It > is uncommon for all the material to be together in the build > log in such contexts. Running make cleandir twice and restarting -j4 buildworld brought the process full circle: A silent hang, no debugger response, no console warnings. That's what sent me down the rabbit hole of make without clean, which worked at least once... The residue of the top screen shows last pid: 63377; load averages: 4.29, 4.18, 4.15 up 1+07:11:07 04:46:46 60 processes: 5 running, 55 sleeping CPU: 70.7% user, 0.0% nice, 26.5% system, 2.8% interrupt, 0.0% idle Mem: 631M Active, 4932K Inact, 92M Laundry, 166M Wired, 98M Buf, 18M Free Swap: 2048M Total, 119M Used, 1928M Free, 5% Inuse, 16K In, 3180K Out packet_write_wait: Connection to 50.1.20.26 port 22: Broken pipe bob@raspberrypi:~ $ ssh www.zefox.com RES STATE C TIME WCPU COMMAND ssh: connect to host www.zefox.com port 22: Connection timed out86.17% c++ bob@raspberrypi:~ $ 1 99 0 277M 231M RUN 0 3:26 75.00% c++ 63245 bob 1 99 0 219M 173M CPU0 0 2:10 73.12% c++ 62690 bob 1 98 0 354M 234M RUN 3 9:42 47.06% c++ 63377 bob 1 30 0 5856K 2808K nanslp 0 0:00 3.13% gstat 38283 bob 1 24 0 5208K 608K wait 2 2:00 0.61% sh 995 bob 1 20 0 6668K 1184K CPU3 3 8:46 0.47% top 990 bob 1 20 0 12M 1060K select 2 0:48 0.05% sshd .... [apologies for typing over the remnants] I've put copies of the build and swap logs at http://www.zefox.net/~fbsd/rpi2/buildworld/ The last vmstat entry (10 second repeat time) reports: procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 sd0 in sy cs us sy id 4 0 14 969160 91960 685 2 2 1 707 304 0 0 11418 692 1273 45 5 50 Does that point to the memory exhaustion suggested earlier in the thread? At this point /boot/loader.conf contains vm.pfault_oom_attempts="-1", but that's a relic of long-ago attempts to use USB flash for root and swap. Might removing it stimulate more warning messages? Thanks for reading! bob prohaska