From owner-freebsd-current@freebsd.org Fri Sep 13 05:53:40 2019 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id F1E48E639F for ; Fri, 13 Sep 2019 05:53:40 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46V4ZD57rjz4BNT; Fri, 13 Sep 2019 05:53:40 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id x8D5rW4q048713 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Fri, 13 Sep 2019 08:53:35 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua x8D5rW4q048713 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id x8D5rWw9048712; Fri, 13 Sep 2019 08:53:32 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 13 Sep 2019 08:53:32 +0300 From: Konstantin Belousov To: Don Lewis Cc: Mark Johnston , FreeBSD Current , kib@freebsd.org Subject: Re: spurious out of swap kills Message-ID: <20190913055332.GN2559@kib.kiev.ua> References: <20190913000635.GG8397@raichu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.12.1 (2019-06-15) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-Rspamd-Queue-Id: 46V4ZD57rjz4BNT X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-5.99 / 15.00]; NEURAL_HAM_MEDIUM(-0.99)[-0.995,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Sep 2019 05:53:41 -0000 On Thu, Sep 12, 2019 at 05:42:00PM -0700, Don Lewis wrote: > On 12 Sep, Mark Johnston wrote: > > On Thu, Sep 12, 2019 at 04:00:17PM -0700, Don Lewis wrote: > >> My poudriere machine is running 13.0-CURRENT and gets updated to the > >> latest version of -CURRENT periodically. At least in the last week or > >> so, I've been seeing occasional port build failures when building my > >> default set of ports, and I finally had some time to do some > >> investigation. > >> > >> It's a 16-thread Ryzen machine, with 64 GB of RAM and 40 GB of swap. > >> Poudriere is configured with > >> USE_TMPFS="wrkdir data localbase" > >> and I have > >> .if ${.CURDIR:M*/www/chromium} > >> MAKE_JOBS_NUMBER=16 > >> .else > >> MAKE_JOBS_NUMBER=7 > >> .endif > >> in /usr/local/etc/poudriere.d/make.conf, since this gives me the best > >> overall build time for my set of ports. This hits memory pretty hard, > >> especially when chromium, firefox, libreoffice, and both versions of > >> openoffice are all building at the same time. During this time, the > >> amount of space consumed by tmpfs for /wrkdir gets large when building > >> these large ports. There is not enough RAM to hold it all, so some of > >> the older data spills over to swap. Swap usage peaks at about 10 GB, > >> leaving about 30 GB of free swap. Nevertheless, I see these errors, > >> with rustc being the usual victim: > >> > >> Sep 11 23:21:43 zipper kernel: pid 16581 (rustc), jid 43, uid 65534, was killed: out of swap space > >> Sep 12 02:48:23 zipper kernel: pid 1209 (rustc), jid 62, uid 65534, was killed: out of swap space > >> > >> Top shows the size of rustc being about 2 GB, so I doubt that it > >> suddenly needs an additional 30 GB of swap. > >> > >> I'm wondering if there might be a transient kmem shortage that is > >> causing a malloc(..., M_NOWAIT) failure in the swap allocation path > >> that is the cause of the problem. > > > > Perhaps this is a consequence of r351114? To confirm this, you might > > try increasing the value of vm.pfault_oom_wait to a larger value, like > > 20 or 30, and see if the OOM kills still occur. > > I wonder if increasing vm.pfault_oom_attempts might also be a good idea. If you are sure that you cannot exhaust your swap space, set attempts to -1 to disable this mechanism. Basically, page fault handler waits for vm.pfault_oom_wait * vm.pfault_oom_attempts for a page allocation before killing the process. Default is 30 secs, and if you cannot get a page for 30 secs, there is something very wrong with the machine.