From owner-freebsd-current@freebsd.org Thu Sep 12 23:00:19 2019 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DEB06DCE84 for ; Thu, 12 Sep 2019 23:00:19 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46TvPH5brKz3NHS for ; Thu, 12 Sep 2019 23:00:19 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from mousie.catspoiler.org (unknown [76.212.85.177]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) (Authenticated sender: truckman) by smtp.freebsd.org (Postfix) with ESMTPSA id 53913627B for ; Thu, 12 Sep 2019 23:00:19 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Date: Thu, 12 Sep 2019 16:00:17 -0700 (PDT) From: Don Lewis Subject: spurious out of swap kills To: FreeBSD Current Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; CHARSET=us-ascii Content-Disposition: INLINE X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Sep 2019 23:00:19 -0000 My poudriere machine is running 13.0-CURRENT and gets updated to the latest version of -CURRENT periodically. At least in the last week or so, I've been seeing occasional port build failures when building my default set of ports, and I finally had some time to do some investigation. It's a 16-thread Ryzen machine, with 64 GB of RAM and 40 GB of swap. Poudriere is configured with USE_TMPFS="wrkdir data localbase" and I have .if ${.CURDIR:M*/www/chromium} MAKE_JOBS_NUMBER=16 .else MAKE_JOBS_NUMBER=7 .endif in /usr/local/etc/poudriere.d/make.conf, since this gives me the best overall build time for my set of ports. This hits memory pretty hard, especially when chromium, firefox, libreoffice, and both versions of openoffice are all building at the same time. During this time, the amount of space consumed by tmpfs for /wrkdir gets large when building these large ports. There is not enough RAM to hold it all, so some of the older data spills over to swap. Swap usage peaks at about 10 GB, leaving about 30 GB of free swap. Nevertheless, I see these errors, with rustc being the usual victim: Sep 11 23:21:43 zipper kernel: pid 16581 (rustc), jid 43, uid 65534, was killed: out of swap space Sep 12 02:48:23 zipper kernel: pid 1209 (rustc), jid 62, uid 65534, was killed: out of swap space Top shows the size of rustc being about 2 GB, so I doubt that it suddenly needs an additional 30 GB of swap. I'm wondering if there might be a transient kmem shortage that is causing a malloc(..., M_NOWAIT) failure in the swap allocation path that is the cause of the problem.