From owner-freebsd-current@freebsd.org Tue Jan 28 19:34:00 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A0DF0239746 for ; Tue, 28 Jan 2020 19:34:00 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from smtp-out-no.shaw.ca (smtp-out-no.shaw.ca [64.59.134.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 486cHW4Rqsz4RKS for ; Tue, 28 Jan 2020 19:33:59 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from spqr.komquats.com ([70.67.125.17]) by shaw.ca with ESMTPA id wWcUi7AwLnCigwWcWigSVy; Tue, 28 Jan 2020 12:33:57 -0700 X-Authority-Analysis: v=2.3 cv=cZisUULM c=1 sm=1 tr=0 a=VFtTW3WuZNDh6VkGe7fA3g==:117 a=VFtTW3WuZNDh6VkGe7fA3g==:17 a=jpOVt7BSZ2e4Z31A5e1TngXxSK0=:19 a=xqWC_Br6kY4A:10 a=IkcTkHD0fZMA:10 a=Jdjhy38mL1oA:10 a=CjxXgO3LAAAA:8 a=YxBL1-UpAAAA:8 a=YVOhz5M6AAAA:8 a=6I5d2MoRAAAA:8 a=btI72j7jPLSCBgcFqRgA:9 a=hcfIkKsiUbtcD7XP:21 a=y-KIT5yIWAFQ5dkf:21 a=QEXdDO2ut3YA:10 a=Ia-lj3WSrqcvXOmTRaiG:22 a=sbbTL3E6IKcx-RquDtO-:22 a=IjZwj45LgO3ly-622nXo:22 Received: from [IPv6:2605:8d80:401:570e:65af:2990:87df:6dc2] (unknown [72.143.222.28]) by spqr.komquats.com (Postfix) with ESMTPSA id CE07773C; Tue, 28 Jan 2020 11:33:53 -0800 (PST) Date: Tue, 28 Jan 2020 11:33:30 -0800 User-Agent: K-9 Mail for Android In-Reply-To: References: <202001261745.00QHjkuW044006@gndrsh.dnsmgr.net> <202001271309.00RD96nr005876@slippy.cwsent.com> <202001272048.00RKmiZs006726@slippy.cwsent.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Re: After update to r357104 build of poudriere jail fails with 'out of swap space' To: Mark Millard CC: "Rodney W. Grimes" , sgk@troutmask.apl.washington.edu, freebsd-current@freebsd.org, yasu@utahime.org From: Cy Schubert Message-ID: <4682F012-E3C5-4B49-8099-659EBCB7B585@cschubert.com> X-CMAE-Envelope: MS4wfH6qY8lRCa/fnEIo4O8Tb9sIdHkQogxA1X5UdtCy9/uUIoFaHLAv981fvkAJSm94wRtLexUJA5Hg/xd0bq1VWofJgczFmY4U18L/KJzVFw/VFheYCWo4 LnZSPVp+1pR0GPt6t3tnr25e77os0KTKDM7pTc1lw9PaRO0MpGkP9mxO5/ebsTX2KEHqH2NKUqE7f7wWjVWZznpFXkUjTMFE5BSINON6hSw6OxwpraG/zt1F VMwhs2ezFu3WFV8Va3+KMePtHOPRRLVp7IoLh1bgNKBMpjfW7Pvam7a/NjieQgnYqR79avvVmgJJ7A3zPa0fDScLw5HQmLtNSe+HCePTyX4= X-Rspamd-Queue-Id: 486cHW4Rqsz4RKS X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of cy.schubert@cschubert.com has no SPF policy when checking 64.59.134.13) smtp.mailfrom=cy.schubert@cschubert.com X-Spamd-Result: default: False [-4.66 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[28.222.143.72.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.11,17.125.67.70.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.11]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_NA(0.00)[]; FREEMAIL_TO(0.00)[yahoo.com]; RCVD_IN_DNSWL_LOW(-0.10)[13.134.59.64.list.dnswl.org : 127.0.5.1]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6327, ipnet:64.59.128.0/20, country:CA]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(-2.46)[ip: (-6.54), ipnet: 64.59.128.0/20(-3.18), asn: 6327(-2.48), country: CA(-0.09)]; FROM_EQ_ENVFROM(0.00)[] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Jan 2020 19:34:00 -0000 On January 27, 2020 2:25:59 PM PST, Mark Millard wrot= e: > > >On 2020-Jan-27, at 12:48, Cy Schubert >wrote: > >> In message , Mark >Millard=20 >> write >> s: >>>=20 >>>=20 >>>=20 >>> On 2020-Jan-27, at 10:20, Cy Schubert >wrote: >>>=20 >>>> On January 27, 2020 5:09:06 AM PST, Cy Schubert > >>> wrote: >>>>>> =2E =2E =2E=20 >>>>>=20 >>>>> Setting a lower arc_max at boot is unlikely to help=2E Rust was >building >>>>> on=20 >>>>> the 8 GB and 5 GB 4 core machines last night=2E It completed >successfully >>>>> on=20 >>>>> the 8 GB machine, while using 12 MB of swap=2E ARC was at 1307 MB=2E >>>>>=20 >>>>> On the 5 GB 4 core machine the rust build died of OOM=2E 328 KB swap >was=20 >>>>> used=2E ARC was reported at 941 MB=2E arc_min on this machine is 489= =2E2 >MB=2E >>>>=20 >>>> MAKE_JOBS_NUMBER=3D3 worked building rust on the 5 GB 4 core >machine=2E ARC is >>> at 534 MB with 12 MB swap used=2E >>>=20 >>> If you increase vm=2Epageout_oom_seq to, say, 10 times what you now >use, >>> does MAKE_JOBS_NUMBER=3D4 complete --or at least go notably longer >before >>> getting OOM behavior from the system? (The default is 12 last I >checked=2E >>> So that might be what you are now using=2E) >>=20 >> It's already 4096 (default is 12)=2E > >Wow=2E Then the count of tries to get free RAM above the threshold >does not seem likely to be the source of the OOM kills=2E > >>>=20 >>> Have you tried also having: vm=2Epfault_oom_attempts=3D"-1" (Presuming >>> you are not worried about actually running out of swap/page space, >>> or can tolerate a deadlock if it does run out=2E) This setting >presumes >>> head, not release or stable=2E (Last I checked anyway=2E) >>=20 >> Already there=2E > >Then page-out delay does not seem likely to be the source of the OOM >kills=2E > >> The box is a sandbox with remote serial console access so deadlocks >are ok=2E >>=20 >>>=20 >>> It would be interesting to know what difference those two settings >>> together might make for your context: it seems to be a good context >>> for testing in this area=2E (But you might already have set them=2E >>> If so, it would be good to report the figures in use=2E) >>>=20 >>> Of course, my experiment ideas need not be your actions=2E >>=20 >> It's a sandbox machine=2E We already know 8 GB works with 4 threads on >as=20 >> many cores=2E And, 5 GB works with 3 threads on 4 cores=2E > >It would be nice to find out what category of issue in the kernel >is driving the OOM kills for your 5GB context with MAKE_JOBS_NUMBER=3D4= =2E >Too bad the first kill does not report a backtrace spanning the >code choosing to do the kill (or otherwise report the type of issue >leading the the kill)=2E > >Your is consistent with the small arm board folks reporting that >recently >contexts that were doing buildworld and the like fine under somewhat >older kernels have started getting OOM kills, despite the two settings=2E > >At the moment I'm not sure how to find the category(s) of issue(s) that >is(are) driving these OOM kills=2E > >Thanks for reporting what settings you were using=2E > >=3D=3D=3D >Mark Millard >marklmi at yahoo=2Ecom >( dsl-only=2Enet went >away in early 2018-Mar) I've been able to reproduce the problem at $JOB in a Virtualbox VM with 1 = vCPU, 1=2E5 GB vRAM, and 2 GB swap building graphics/graphviz: cc killed ou= t of swap space=2E The killed cc had an address space of ~ 500 MB, using on= ly 43 MB of the 2 GB swap=2E Free space is exhausted but swap used never ex= ceeds tens of MB=2E Doubling the swap to 4 GB had no effect=2E The VM doesn= 't use ZFS=2E This appears recent=2E --=20 Pardon the typos and autocorrect, small keyboard in use=2E=20 Cy Schubert FreeBSD UNIX: Web: https://www=2EFreeBSD=2Eorg The need of the many outweighs the greed of the few=2E Sent from my Android device with K-9 Mail=2E Please excuse my brevity=2E