From owner-freebsd-current@freebsd.org Fri Sep 13 00:06:40 2019 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8F528DEDDA for ; Fri, 13 Sep 2019 00:06:40 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-io1-xd35.google.com (mail-io1-xd35.google.com [IPv6:2607:f8b0:4864:20::d35]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46Twsr3H0fz3R57; Fri, 13 Sep 2019 00:06:40 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-io1-xd35.google.com with SMTP id n197so59154951iod.9; Thu, 12 Sep 2019 17:06:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=zXSDpPl0zFUKkD0PMalBOt4ILr01VMLqlspB62yRWIk=; b=A+wDDJ1H9RwWBg1mX4BM23Yut5RYAmvPixKV2uW3jbSNUXgB7qO3g+TXFPSo9nSwJ2 C93FUBylNy9m1wlE+aY72gg938XYPS+oho+NeJNAp+kRk/8xKXeMtRYSybls6GewaLZM tEu3vRDqZ5T7b0S521K60Wre3O4K1VUDt4hhiVDAI452fEQcT+Ls0AUtS/8T/oqIHpX1 FwT+9M96OymiRm0bYnIL/1XRVUOKeQDkDfBQfdvEfdZjlqUnhEZ+JeTFNku437tGF1Gy agF1BzR9Q3jZYybzKF/8gEQ+VSvNZz9UGler3miy9jgKhaUNEgstL7OntF/elkHGgtr2 tMpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=zXSDpPl0zFUKkD0PMalBOt4ILr01VMLqlspB62yRWIk=; b=dlhZucB5cUwHcG2NsKjjDdQICjwUaZp53zbnxROlPalLbt4tU7KVUf69GGgF5/lhgh rYakLrnSm+fsce/F6EChZk01DIMD8GRYB5wu2V7vS1EMg4hsEzRCqABk+TQWqUCHPRK9 BMk1PBOBC9VcGAPCQrrbGiEN8+4F+dXSAaYiPnl95OaPRTHlGd1G1r9goN+epEbMTXzo nPdYPWj3ryfAttRJqG7bkmNE1G4GuBRJpj54QbgwVqIeMr8LIcY/j7cHQa/9tbi0BaZb Bwj+BEYW9aXrI9ZqY7tj+4+KnW4Xn/DjHiJbIyBTh1lGbFIv2TKH67yLs7GjA/cwYhzr nGLQ== X-Gm-Message-State: APjAAAX5OKK/IkpElwkhiKC4dvPP5Sw9JFydAH63iPusaWeEJkKhwVno 1pqXNzA0ATw2wOkF+QR0+kiarVMN X-Google-Smtp-Source: APXvYqzybUUCiuxCOywauynyBxalm0VYYIqxWmO5db61Hi7pezxb06eSUAF5ckSLU9HwydAv1FIyJA== X-Received: by 2002:a6b:db0e:: with SMTP id t14mr7719780ioc.93.1568333198655; Thu, 12 Sep 2019 17:06:38 -0700 (PDT) Received: from raichu (toroon0560w-lp140-01-69-159-39-167.dsl.bell.ca. [69.159.39.167]) by smtp.gmail.com with ESMTPSA id x26sm20937946ioa.37.2019.09.12.17.06.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Sep 2019 17:06:37 -0700 (PDT) Sender: Mark Johnston Date: Thu, 12 Sep 2019 20:06:35 -0400 From: Mark Johnston To: Don Lewis Cc: FreeBSD Current , kib@freebsd.org Subject: Re: spurious out of swap kills Message-ID: <20190913000635.GG8397@raichu> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.12.1 (2019-06-15) X-Rspamd-Queue-Id: 46Twsr3H0fz3R57 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-6.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Sep 2019 00:06:40 -0000 On Thu, Sep 12, 2019 at 04:00:17PM -0700, Don Lewis wrote: > My poudriere machine is running 13.0-CURRENT and gets updated to the > latest version of -CURRENT periodically. At least in the last week or > so, I've been seeing occasional port build failures when building my > default set of ports, and I finally had some time to do some > investigation. > > It's a 16-thread Ryzen machine, with 64 GB of RAM and 40 GB of swap. > Poudriere is configured with > USE_TMPFS="wrkdir data localbase" > and I have > .if ${.CURDIR:M*/www/chromium} > MAKE_JOBS_NUMBER=16 > .else > MAKE_JOBS_NUMBER=7 > .endif > in /usr/local/etc/poudriere.d/make.conf, since this gives me the best > overall build time for my set of ports. This hits memory pretty hard, > especially when chromium, firefox, libreoffice, and both versions of > openoffice are all building at the same time. During this time, the > amount of space consumed by tmpfs for /wrkdir gets large when building > these large ports. There is not enough RAM to hold it all, so some of > the older data spills over to swap. Swap usage peaks at about 10 GB, > leaving about 30 GB of free swap. Nevertheless, I see these errors, > with rustc being the usual victim: > > Sep 11 23:21:43 zipper kernel: pid 16581 (rustc), jid 43, uid 65534, was killed: out of swap space > Sep 12 02:48:23 zipper kernel: pid 1209 (rustc), jid 62, uid 65534, was killed: out of swap space > > Top shows the size of rustc being about 2 GB, so I doubt that it > suddenly needs an additional 30 GB of swap. > > I'm wondering if there might be a transient kmem shortage that is > causing a malloc(..., M_NOWAIT) failure in the swap allocation path > that is the cause of the problem. Perhaps this is a consequence of r351114? To confirm this, you might try increasing the value of vm.pfault_oom_wait to a larger value, like 20 or 30, and see if the OOM kills still occur.