From owner-freebsd-current@freebsd.org Mon Jan 27 13:09:17 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 56E381F5D43 for ; Mon, 27 Jan 2020 13:09:17 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from smtp-out-no.shaw.ca (smtp-out-no.shaw.ca [64.59.134.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 485qp435Qrz4M80 for ; Mon, 27 Jan 2020 13:09:16 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from spqr.komquats.com ([70.67.125.17]) by shaw.ca with ESMTPA id w48di47BhkqGXw48fiHBM3; Mon, 27 Jan 2020 06:09:14 -0700 X-Authority-Analysis: v=2.3 cv=c/jVvi1l c=1 sm=1 tr=0 a=VFtTW3WuZNDh6VkGe7fA3g==:117 a=VFtTW3WuZNDh6VkGe7fA3g==:17 a=kj9zAlcOel0A:10 a=Jdjhy38mL1oA:10 a=iKhvJSA4AAAA:8 a=YxBL1-UpAAAA:8 a=6I5d2MoRAAAA:8 a=cU3w10EP4QF15qbrk4gA:9 a=jSGmALKL3TXUxhBw:21 a=Tr42tAgKKlysYeKq:21 a=CjuIK1q_8ugA:10 a=7FgKuMUr8FYA:10 a=odh9cflL3HIXMm4fY7Wr:22 a=Ia-lj3WSrqcvXOmTRaiG:22 a=IjZwj45LgO3ly-622nXo:22 Received: from slippy.cwsent.com (slippy [10.1.1.91]) by spqr.komquats.com (Postfix) with ESMTPS id BC1B7363; Mon, 27 Jan 2020 05:09:10 -0800 (PST) Received: from slippy.cwsent.com (localhost [127.0.0.1]) by slippy.cwsent.com (8.15.2/8.15.2) with ESMTP id 00RD9ANN005879; Mon, 27 Jan 2020 05:09:10 -0800 (PST) (envelope-from Cy.Schubert@cschubert.com) Received: from slippy (cy@localhost) by slippy.cwsent.com (8.15.2/8.15.2/Submit) with ESMTP id 00RD96nr005876; Mon, 27 Jan 2020 05:09:08 -0800 (PST) (envelope-from Cy.Schubert@cschubert.com) Message-Id: <202001271309.00RD96nr005876@slippy.cwsent.com> X-Authentication-Warning: slippy.cwsent.com: cy owned process doing -bs X-Mailer: exmh version 2.9.0 11/07/2018 with nmh-1.7.1 Reply-to: Cy Schubert From: Cy Schubert X-os: FreeBSD X-Sender: cy@cwsent.com X-URL: http://www.cschubert.com/ To: "Rodney W. Grimes" cc: Cy Schubert , sgk@troutmask.apl.washington.edu, freebsd-current@freebsd.org, Mark Millard , yasu@utahime.org Subject: Re: After update to r357104 build of poudriere jail fails with 'out of swap space' In-reply-to: <202001261745.00QHjkuW044006@gndrsh.dnsmgr.net> References: <202001261745.00QHjkuW044006@gndrsh.dnsmgr.net> Comments: In-reply-to "Rodney W. Grimes" message dated "Sun, 26 Jan 2020 09:45:46 -0800." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Mon, 27 Jan 2020 05:09:06 -0800 X-CMAE-Envelope: MS4wfBzW3U4SwHyQLGfQVmj/Jebr0eFpafaj1NAYteVAdTfB21P0Pdamnqw9bGdiA+FDF0lNJVbjcr7MMBSlgPef+TERZhY4dMsN1NsSKctwVgxgYzOgtskq gXwRS8DU6l27X3xIYeS1bSQBK6SdFVzF8JNHjJJ+qyfH9ojvoVvUKYmXk0yOpJCQ7/SmHpdLwnUF4dDslAz2AbZ463y5Ahg6Yq0e1XMoKHFoqlI5owDyXhm/ Xc8Rv/by2oEA3wcG686OIcjLNMSDCxRvlsRH43GywjJY/z4glen4t/N6RBpLhiCXfQv39c77tAdPP3wLG4q8rjfsTCo1vp7qpb5eospZLOg= X-Rspamd-Queue-Id: 485qp435Qrz4M80 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of cy.schubert@cschubert.com has no SPF policy when checking 64.59.134.13) smtp.mailfrom=cy.schubert@cschubert.com X-Spamd-Result: default: False [-4.15 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; HAS_REPLYTO(0.00)[Cy.Schubert@cschubert.com]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCPT_COUNT_FIVE(0.00)[6]; REPLYTO_EQ_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[17.125.67.70.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.11]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_NA(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[13.134.59.64.list.dnswl.org : 127.0.5.1]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6327, ipnet:64.59.128.0/20, country:CA]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(-2.45)[ip: (-6.53), ipnet: 64.59.128.0/20(-3.18), asn: 6327(-2.47), country: CA(-0.09)]; FROM_EQ_ENVFROM(0.00)[] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Jan 2020 13:09:17 -0000 In message <202001261745.00QHjkuW044006@gndrsh.dnsmgr.net>, "Rodney W. Grimes" writes: > > In message <20200125233116.GA49916@troutmask.apl.washington.edu>, Steve > > Kargl w > > rites: > > > On Sat, Jan 25, 2020 at 02:09:29PM -0800, Cy Schubert wrote: > > > > On January 25, 2020 1:52:03 PM PST, Steve Kargl ingt > > > on.edu> wrote: > > > > >On Sat, Jan 25, 2020 at 01:41:16PM -0800, Cy Schubert wrote: > > > > >> > > > > >> It's not just poudeiere. Standard port builds of chromium, rust > > > > >> and thunderbird also fail on my machines with less than 8 GB. > > > > >> > > > > > > > > > >Interesting. I routinely build chromium, rust, firefox, > > > > >llvm and few other resource-hunger ports on a i386-freebsd > > > > >laptop with 3.4 GB available memory. This is done with > > > > >chrome running with a few tabs swallowing a 1-1.5 GB of > > > > >memory. No issues. > > > > > > > > Number of threads makes a difference too. How many core/threads does yo > ur l > > > aptop have? > > > > > > 2 cores. > > > > This is why. > > > > > > > > > Reducing number of concurrent threads allowed my builds to complete > > > > on the 5 GB machine. My build machines have 4 cores, 1 thread per > > > > core. Reducing concurrent threads circumvented the issue. > > > > > > I use portmaster, and AFIACT, it uses 'make -j 2' for the build. > > > Laptop isn't doing too much, but an update and browsing. It does > > > take a long time especially if building llvm is required. > > > > I use portmaster as well (for quick incidental builds). It uses > > MAKE_JOBS_NUMBER=4 (which is equivalent to make -j 4). I suppose machines > > with not enough memory to support their cores with certain builds might > > have a better chance of having this problem. > > > > MAKE_JOBS_NUMBER_LIMIT to limit a 4 core machine with less than 2 GB per > > core might be an option. Looking at it this way, instead of an extra 3 GB, > > the extra 60% more memory in the other machine makes a big difference. A > > rule of thumb would probably be, have ~ 2 GB RAM for every core or thread > > when doing large parallel builds. > > Perhaps we need to redo some boot time calculations, for one the > ZFS arch cache, IMHO, is just silly at a fixed percent of total > memory. A high percentage at that. > > One idea based on what you just said might be: > > percore_memory_reserve = 2G (Your number, I personally would use 1G here) > arc_max = MAX(memory size - (Cores * percore_memory_reserve), 512mb) > > I think that simple change would go a long ways to cutting down the > number of OOM reports we see. ALSO IMHO there should be a way for > sub systems to easily tell zfs they are memory pigs too and need to > share the space. Ie, bhyve is horrible if you do not tune zfs arc > based on how much memory your using up for VM's. > > Another formulation might be > percore_memory_reserve = alpha * memory_zire / cores > > Alpha most likely falling in the 0.25 to 0.5 range, I think this one > would have better scalability, would need to run some numbers. > Probably needs to become non linear above some core count. Setting a lower arc_max at boot is unlikely to help. Rust was building on the 8 GB and 5 GB 4 core machines last night. It completed successfully on the 8 GB machine, while using 12 MB of swap. ARC was at 1307 MB. On the 5 GB 4 core machine the rust build died of OOM. 328 KB swap was used. ARC was reported at 941 MB. arc_min on this machine is 489.2 MB. -- Cheers, Cy Schubert FreeBSD UNIX: Web: http://www.FreeBSD.org The need of the many outweighs the greed of the few.